Detections: one format for every model's results

A detection model tells you what it found and where. Each framework returns that in its own objects, with its own field names and box formats. PixelFlow converts every result into a Detections object: a list of Detection objects, one per object found, each with the same fields whatever model produced it.

Python
import pixelflow as pf
from mayaku import from_pretrained

predictor = from_pretrained("mayaku-n-det")
result = predictor("street.jpg")
detections = pf.from_mayaku(result, labels=predictor.class_names)
detections = detections.filter_by_confidence(0.5)

print(len(detections))
for detection in detections:
    print(detection.class_name, detection.confidence, detection.bbox)
Text only
11
Person 0.835 [4.56, 588.91, 333.44, 1273.22]
Person 0.721 [275.62, 535.19, 469.88, 795.32]
Person 0.656 [624.09, 561.33, 743.54, 906.53]
Person 0.649 [944.12, 558.44, 1151.99, 874.55]
Umbrella 0.619 [716.73, 427.97, 957.37, 627.64]
...

Every function that comes after the model (filters, the tracker, zones, drawing) takes this object, so the rest of your code does not change when you change the model.

What a detection holds

A converter fills the fields the model provides and leaves the others empty (None, or an empty list). The last fields are filled later, by the PixelFlow step named in the table.

Field What it holds Filled by
bbox Box as [x1, y1, x2, y2]: the top-left and bottom-right corners, in pixels from the image's top-left corner, rounded to 2 decimals the converter
confidence The model's score, from 0 to 1, rounded to 3 decimals the converter
class_id The class number from the model the converter
class_name The class name, such as "Person", when the model or the labels you pass gives one the converter
masks Segmentation masks, from segmentation models the converter
segments The object's outline as one polygon, [[x, y], ...] the converter, or masks_to_segments()
keypoints Landmarks as KeyPoint objects (x, y, id, name, confidence), from pose models the converter
text Text the model produced for this object: what an OCR model read, or a caption the converter
tracker_id The same number for the same object in every frame of a video the tracker
zones, zone_names The zones the object is in Zones
line_crossings The lines the object crossed Crossings
first_seen_time, total_time When the object first appeared, and for how many seconds since TimeTracker
metadata Anything else, as a dictionary you

class_name and text are different things: class_name is one of the classes the model was trained on, and text is free text the model produced for this one object. A detection can have both.

Masks and outlines

A segmentation model fills masks with a list holding one mask per object: a boolean array the size of the image, True where the object is.

Python
import pixelflow as pf
from mayaku import from_pretrained

predictor = from_pretrained("mayaku-n-seg")
result = predictor("street.jpg")
detections = pf.from_mayaku(result, labels=predictor.class_names)
detections = detections.filter_by_confidence(0.5)

person = detections[0]
mask = person.masks[0]
print(mask.shape, mask.dtype, mask.sum())
Text only
(1280, 1920) bool 117697

A full-size mask per object takes memory and cannot be written to JSON as it is. When you need the outline instead, for example to store it or send it to a web page, masks_to_segments() traces each mask into a polygon and stores it in segments:

Python
detections.masks_to_segments()
print(len(person.segments), person.segments[:3])
Text only
105 [[15.0, 685.0], [15.0, 700.0], [19.0, 718.0]]

Pass keep_masks=False to drop the masks once they are traced, and tolerance (in pixels, default 1.5) to trade outline detail for fewer points.

Working with a Detections object

Make detections yourself

When your model has no converter, build the detections from your own values:

Python
import pixelflow as pf

detections = pf.Detections()
detections.add_detection(
    pf.Detection(bbox=[120, 80, 260, 400], class_id=0, class_name="person", confidence=0.91)
)
detections.add_detection(
    pf.Detection(bbox=[300, 50, 300, 90], class_name="person")
)
print(len(detections))
Text only
1

The second detection was not added: its box has no width. add_detection leaves out a detection whose box is missing, invalid or encloses no pixels, and one whose confidence is NaN. So every detection in a Detections has a usable box, and the code after it does not need to check.

When your model returns NumPy arrays of boxes, scores and class numbers, pf.from_arrays builds the whole Detections in one call. See Converters.