Look at past and future frames with a buffer

Some decisions about a frame are easier with the frames around it: a count that jumps from one frame to the next, a detection that appears for a single frame. pf.Buffer keeps the last few frames with their results and hands each one back a few frames late, once the frames after it have arrived. For that frame, you have both the frames before it and the frames after it.

Python
import pixelflow as pf
from mayaku import from_pretrained

predictor = from_pretrained("mayaku-n-det")
person = predictor.class_names.index("Person")
buffer = pf.Buffer(frames=5)
counts = []


def process(people, frame):
    context = buffer.get_temporal_context()
    window = context["past_results"] + [people] + context["future_results"]
    counts.append((len(people), sum(len(d) for d in window) / len(window)))
    writer.write(pf.annotate.box(frame, people))


video = pf.VideoReader("crowd.mp4")
with pf.VideoWriter("crowd-buffered.mp4", like=video) as writer:
    for frame in video:
        result = predictor(frame[:, :, ::-1])  # Mayaku reads arrays as BGR
        detections = pf.from_mayaku(result, labels=predictor.class_names)
        people = detections.filter_by_confidence(0.5).filter_by_class_id(person)

        people, frame = buffer.update(people, frame)
        if frame is not None:
            process(people, frame)

    for people, frame in buffer.flush():
        process(people, frame)

print(writer.frames_written)
for index in (0, 100, 200, 340):
    count, average = counts[index]
    print(f"frame {index}: {count} people, {average:.1f} on average")
Text only
341
frame 0: 24 people, 23.7 on average
frame 100: 29 people, 28.8 on average
frame 200: 26 people, 26.2 on average
frame 340: 21 people, 21.6 on average

A frame of a crowded hall seen from above, with a box around each person

buffer.update(results, frame) stores the new frame and its results, and returns a frame and its results from earlier: with frames=5, the ones from 2 frames ago. They always belong together, so the boxes you draw are the right ones for the frame you draw on.

get_temporal_context() returns the buffer around that frame. Above, the count of each frame is averaged with the 2 frames before it and the 2 after. Because the window is centered on the frame, the average does not lag behind the crowd, as an average of past frames only would.

The first and last frames

For the first 2 calls, nothing is ready: frame 0 needs frames 1 and 2 after it. update returns (None, None), hence if frame is not None. From the third call on, it returns frames 0, 1, 2 and so on, in order.

When the video ends, the last 2 frames are still in the buffer, waiting for frames that will not come. flush() returns them. With both, every frame comes out once: above, 341 frames in, 341 out.

Near the start, a frame has fewer frames before it, and near the end, fewer after it: frame 0 is averaged over 3 frames, not 5. If your processing needs the full window, check the length of past_results and future_results. pf.smooth does this: it smooths only frames with a full window, and returns the others as they are.

buffer.delay is how many frames late the output is: frames // 2. A larger buffer gives more context and a longer delay. Use an odd number of frames, so the frame in the middle has the same number of frames on each side.

The context

get_temporal_context() describes the frame update or flush returned last. It returns None before the first one, then a dictionary:

Key
past_frames, past_results lists of the frames and results before it, oldest first
current_frame, current_results the frame and results update or flush returned
future_frames, future_results lists of the frames and results after it, oldest first

The results can be anything you pass to update: Detections, a count or a dictionary of your own.

get_buffer_contents() returns (results, frames) as two lists, oldest first. reset() empties the buffer. Calling update after flush() starts again with an empty buffer, so the frames of the next video do not mix with the last one.

Parameters

pf.Buffer(frames=5):

Parameter Default
frames 5 number of frames to keep, from 1 to 1000; an even number gives a warning
Attribute
delay how many frames late update returns: frames // 2
is_full whether the buffer holds frames frames
frames_processed number of update calls since the start or the last reset()
current_size number of frames in the buffer now

Every frame is copied when it enters the buffer: 5 frames of 1280×720 use 14 MB.