Understanding MP4 Edit Lists

MP4 files have an obscure feature called edit lists. In theory, they let you trim, delay, and speed up video playback without re-encoding. In practice, they’re mostly used to fix sync issues—and barely supported by modern players. Here’s why this elegant feature never quite took off.

The Orchestra Analogy

Think of an edit list as the conductor in an orchestra. In technical terms, the presentation timeline is the performance the audience sees, while media timelines are like the individual instruments in the orchestra. Edit lists are meant to use the media to orchestrate the presentation timeline according to creative intent.

But here's the catch—the edit list atom (edts) is optional. In its absence, it's like having the instruments play to the timing of the percussion section with only their sheet music to guide them. The media simply plays.

Perhaps my orchestra analogy is a bit romantic, but it captures the essence of what edit lists were designed to do.

The Reality: Fixing Sync Issues

In practice, edit lists are often used to hide the "sins" of media tracks. Most commonly, they're employed to adjust audio-video sync issues by delaying one track relative to another. Think of those moments when you watch a video and the lips don't match the words—edit lists can help fix that.

Diving into the Technical Structure

The edts atom, which is the container for the elst atom (edit list), resides in each trak atom. Each trak atom is in the moov atom. This makes it a bit tricky to work with.

Most videos have an edts atom with a single entry. You can manipulate this entry in place without too much trouble. But adding another entry to the elst atom means changing the size field of all parent atoms. This gets complicated because if the moov atom is positioned before the mdat atom (media data), any changes to moov will alter the absolute offset of chunk data referenced in the stss atom, causing playback to fail.

For manual explorations, it's best to work with videos where the moov atom comes after the mdat atom.

Edit List Structure

Here's what the edit list structure looks like in code:

                    aligned(8) class EditListBox extends FullBox('elst', version, 0) {  
                        unsigned int(32) entry_count;  
                    
                        for (i = 0; i < entry_count; i++) {  
                            if (version == 1) {  
                                unsigned int(64) segment_duration;  
                                signed int(64) media_time;  
                            } else {  // version == 0  
                                unsigned int(32) segment_duration;  
                                signed int(32) media_time;  
                            }  
                            
                            int(16) media_rate_integer;  
                            int(16) media_rate_fraction;  
                        }  
                    }                            
                

Key Components Explained

• segment_duration: Specifies how long this edit lasts (either 32-bit or 64-bit, depending on version).
• media_time: Specifies the starting point in the media. A value of -1 indicates an empty edit, creating a gap in playback.
• media_rate_integer: Integer part of the playback speed.
• media_rate_fraction: Fractional part of the playback speed. Together with media_rate_integer, they form a 16.16 fixed-point number (e.g., 1.0000 = normal speed, 0.5000 = half speed).

What Can You Do With Edit Lists?

Trimming Videos

Let's say you have a 30-second video, but you only want to play a segment from 5 seconds to 25 seconds. Instead of re-encoding the video, you can use edit lists to define exactly which portion plays.

To achieve this, you'd create an edit list with one entry where:

• segment_duration = 20 seconds (in the video's time scale)
• media_time = 5 seconds (in the video's time scale)
• media_rate_integer = 1
• media_rate_fraction = 0

Creating Complex Edits

You can get more creative by using multiple entries. For example, what if you want to play seconds 5-10 and then jump to seconds 15-25?

This would require two edit list entries:

1. First entry: segment_duration = 5s, media_time = 5s
2. Second entry: segment_duration = 10s, media_time = 15s

The beauty is that this works regardless of where the keyframes are in your video. The decoder handles the heavy lifting of decoding entry points, even if they fall between keyframes.

Creating Delays with Empty Edits

Another interesting use case: you can set media_time = -1 to create an empty edit—a gap in the presentation. This is particularly useful for syncing audio and video tracks.

For example, if your audio track is 5 seconds behind your video track, you could add an empty edit to the video track:

The edit list entry would look like:

• segment_duration = 5 seconds
• media_time = -1 (empty edit)
• Followed by a normal entry for the actual video content

Changing Playback Speed

You can also manipulate the media_rate_integer and media_rate_fraction fields to change playback speed. For example:

• Normal speed: media_rate_integer = 1, media_rate_fraction = 0
• Half speed: media_rate_integer = 0, media_rate_fraction = 32768 (0.5 in fixed-point)
• Double speed: media_rate_integer = 2, media_rate_fraction = 0

However, this is where we run into issues with implementation. While the MP4 standard supports varying playback speeds through edit lists, many players simply ignore this feature. QuickTime Player and FFplay might respect it, but most web browsers won't.

Try It Yourself

If you'd like to experiment with edit lists, here is hack where the video's elst atoms are manipulated according to what is given by the form.

There is no html trickery, just modifying the buffer directly and loaded to video element.

Why Implementation is Inconsistent

You might wonder why such a useful feature isn't universally supported. I believe one key reason is that MP4 specifications are often behind paywalls, making them less accessible to open-source developers. Many implementations are based on reverse engineering rather than official documentation, leading to inconsistent support.

This is evident when testing playback speed changes through edit lists. While QuickTime and FFplay might honor them, most web browsers won't. It's another case of "the standard allows it, but the players don't support it."

Edit lists represent a powerful but underutilized feature of the MP4 format. They allow for precise control over video playback without re-encoding—trimming content, adjusting synchronization, and even (theoretically) changing playback speed.

While implementation is inconsistent across players, understanding edit lists gives you deeper insight into how digital video works and opens up possibilities for lossless video editing through direct manipulation of MP4 metadata.