pyav read should remember the current index and avoid a seek if it can

Line 457 in c591a1c

decoder = self._seek(index, constant_framerate=constant_framerate)

Seeking videos is expensive since one can only seek to a keyframe.

This should "avoid" seeking if it is can detect that the codec context is already at the correct location.

Hm, what is the correct location? Are you thinking about the scenario of reading consecutive frames (i.e. index=old_index+1)?

This is "quite a common" scenario.

At the very least, it should remember:

If i continue to use the same generator I used previously, the next index is "previous index + 1". this means I do not need to seek! Otherwise, seek.

This little optimization is really helpful.

Otherwise, using a video like "random access" provides terrible performance.

I'm asking because the "read frame n+1" use case fits better with the iter paradigm in my opinion. Currently you can do this via:

import imageio.v3 as iio

for idx, frame in enumerate(iio.imiter("imageio:cockatoo.mp4", "r")):
    if idx < start_idx:
        continue

    ...  # do something useful with each frame

# alternatively, you can use iter outside a loop
# (but within an imopen context)
with iio.imopen("imageio:cockatoo.mp4", "r") as file:
    frame_gen = file.iter()
    for _ in range(start_index):
        next(frame_gen)

    # use the generator to get the next image when needed
    ...
    img = next(frame_gen)
    ...

That said, decoding the first start_idx many frames can be painful as well. I always wanted a start_idx and/or start_time kwarg for pyav's iter. This might be as simple as adding a call to self._seek (+- some edge cases with filters).

There is also the "read a frame at index where index > previous_idx" use case. This case is more interesting, in that we could be smart and check if the new index lies between previous_idx and the next keyframe, in which case it is indeed faster to keep decoding than to jump back and decode from there.

I understand the iter. "Iterators" work sometimes. But other times you just want to index it yourself.

I think the start_index issue is one of them that doesn't work well with the iterator context.

for data analysis, I think of the data as a volume, so sometimes it is really convenient to use the indexing syntax.

I think the start_index issue is one of them that doesn't work well with the iterator context.

Hm. Could you give an example of a scenario where this doesn't work well? I feel like I don't manage to follow your train of thought here.

Its more like code style.

Is the following (pseudocode) better

import numpy as np
import imageio
N_trials = 100
image_shape = (1024, 512)
my_data_describing_my_experiment = imageio.imopen("test.mp4", "w", plugin="pyav")
correction_factor = np.zeros((N_trials,), dtype='uint8')

results = np.zeros_like((N_trials,) + image_shape, dtype='uint8')
for i, (data, correction) in enumerate(zip(video, correction_factor)):
    results[i] = data * correction

than

import numpy as np
N_trials = 100
image_shape = (1024, 512)
my_data_describing_my_experiment = imageio.imopen("test.mp4", "w", plugin="pyav")
correction_factor = np.zeros((N_trials,), dtype='uint8')

results = np.zeros_like((N_trials,) + image_shape, dtype='uint8')
for i in range(N_trials):
    data = video.read(i)
    correction = correction_factor[i]
    results[i] = data * correction

I just know that numpy indexing has been found to be an immensely powerful tool so sometimes I really like to use it.

A few reasons:

When you have more than "2" inputs, zipping them becomes very annoying.
The second problem with zipping is that the failure isn't "loud". If you mismatch your axes, then you would likely never know.
Finally, if you are trying to zip arrays, you likely have to massage them to get them to iterate correctly.

its not that I'm asking for a behavior change in entirety. Mostly, I'm flagging a potential optimization that can be made to "memo" the next frame in the video. If it is the frame the user wants, no need for an expensive seek.

@hmaarrfk

Okay. I thought this over and settled for the following:

We will add a tracker for the current/next index to pyav. Tracking it is cheap (doesn't hurt single reads much) and enables several seek optimizations (beyond just avoiding seek when incrementally reading i+1).
While we will, ofc, try to be efficient where possible, we can not guarantee that syntax like [file.read(i) for i in range(N)] is read optimally.

To elaborate on the above: We can't guarantee this, because we don't know the next index a user is interested in when they call read. To make a video-specific example lets look at some scenarios: (1) A user wants to read exactly one frame. (2) A user wants to read multiple contiguous frames. (3) A user wants to read every n-th frame. (4) A user randomly indexes the video.

In the first case (1), any effort on making the next read more optimal is wasting time and memory because it will never happen.

In the second case (2) we can remember the current read index and reuse the current decoder state. We can do even more by choosing an optimized threading model (see tread_type). However, doing either hurts performance of use-case (1); especially when we change thread_type, because it trades better average time per frame for a (much) worse time until first frame. Compare this to a function like iter which communicates that "I want to read consecutive frames". For iter we can ignore scenario (1) and optimize for scenario (2) (which is btw why our current implementation is faster than OpenCV :) ).

In the third case (3), we can track the next keyframe and seek if the next frame lies beyond the next keyframe. This is fast compared to decoding n frames and only yielding one of them; however, crazy slow for the other two scenarios.

In scenario (4) all bets are off. We can only seek to the closest keyframe and decode from there. This will be sub-optimal if the desired index lies between the last read index and the next keyframe, but optimal otherwise. How well this works depends on the duration of the video and the average keyframe distance. For a typical video (2s keyframe distance) and at least 10s duration, this does the right thing more often than it doesn't. Since read provides random access into the image, this is the scenario that is currently being optimized for. At the same time, this is slow for scenario (2) and scenario (3), because we frequently read from the interval between the last read and the next keyframe in those scenarios.

Summing this up, it is a classic "the general implementation can't do optimizations that a specific one could" situation. No matter which scenario we optimize for, we shoot all the other scenarios in the foot. That said, tracking the index of the previous read has only a very small performance penalty in scenarios (1), and (4). At the same time, it makes scenario (2) optimal and the scenario of reading videos with a non-constant framerate improves, too. Thus, it's worthwhile to implement.

Commenting specifically on your two snippets @hmaarrfk : Implementing the above will improve the performance of the second snippet (using video.read(i)). That said, it will still not be on par with the first (video.iter()) due to optimizations like the choice of an optimized thread_type, which we can't do for video.read.

Navigation Menu

Repository navigation

pyav read should remember the current index and avoid a seek if it can #933

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

pyav read should remember the current index and avoid a seek if it can #933

Description

Activity

FirefoxMetzger commented on Jan 17, 2023

hmaarrfk commented on Jan 17, 2023

FirefoxMetzger commented on Jan 17, 2023

hmaarrfk commented on Jan 17, 2023

hmaarrfk commented on Jan 17, 2023

FirefoxMetzger commented on Jan 17, 2023

hmaarrfk commented on Jan 26, 2023

hmaarrfk commented on Jan 26, 2023

FirefoxMetzger commented on Feb 3, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions