Detect small objects in large images

A detection model shrinks every image to its input size before it looks at it. In a large photo, a person at the back of the scene becomes a few pixels and is missed. SlicedInference cuts the image into overlapping slices, runs the model on each one at full resolution, and merges the results into one Detections for the whole image.

Python
import pixelflow as pf
from mayaku import from_pretrained

predictor = from_pretrained("mayaku-n-det")


def detect(tile):
    result = predictor(tile[:, :, ::-1])  # Mayaku reads arrays as BGR
    detections = pf.from_mayaku(result, labels=predictor.class_names)
    return detections.filter_by_confidence(0.5)


image = pf.read_image("street.jpg")
slicer = pf.SlicedInference(slice_width=640, slice_height=640)
detections = slicer.predict(image, detect)

print(len(detect(image)), len(detections))

image = pf.annotate.box(image, detections)
pf.save_image("street-sliced.jpg", image)
Text only
11 26

Boxes around many small people at the back of a crowded street, and a man in the front covered by several partial boxes

predict takes the image and a function that turns one slice into Detections. It calls that function once per slice, here 12 times for 4 columns and 3 rows, and moves each detection from slice to image coordinates: its box, and its mask, outline and keypoints if it has them. A mask is placed on an image-sized canvas, so with a segmentation model such as mayaku-n-seg, pf.annotate.mask draws the result on the whole image.

On the whole 1920×1280 image, the model finds 11 objects with a confidence of 0.5 or more, 8 of them people. On the slices it finds 26, 19 of them people, most at the back of the street.

What slicing costs

The model runs once per slice. Above, on the GPU of an Apple M1 Max, the 12 slices took 345 ms and one pass on the whole image 35 ms.

Large objects suffer. An object that crosses the edge of a slice is seen in pieces: in the image above, the man in the front gets two partial boxes, and the two motorcycles found on the whole image are not found at all, since no slice holds enough of either. A piece can also be mistaken for something else: the box at the bottom, labelled a person, is the back of a scooter. Use slicing when the objects you want are small compared to the image, such as people in a wide view, vehicles in an aerial photo or defects on a large scan.

The function must return Detections: convert the model's output inside it, as above. Anything else raises a TypeError.

Slice size and overlap

Make slices about the size the model works at: 640 pixels for most models. Smaller slices find smaller objects but take more runs and cut more objects.

Slices overlap by 20% of their size by default, so an object on the edge of one slice is whole in the next. Raise overlap_ratio_w and overlap_ratio_h when objects are cut too often; it also adds slices.

pf.auto_slice_size(height, width) gives a slice size with the image's shape: for a 1920×1280 image, 426 pixels high and 640 wide, which makes 16 slices.

Python
slice_height, slice_width = pf.auto_slice_size(*image.shape[:2])
slicer = pf.SlicedInference(slice_height=slice_height, slice_width=slice_width)

How duplicates are merged

An object inside an overlap is found twice, once in each slice. predict keeps one, going from the highest confidence down, and compares only boxes of the same class:

IoU is the area two boxes share divided by the area they cover together. With merge_mode="nmm", duplicates from neighbouring slices are replaced by their average box, weighted by confidence, instead of keeping only the best one.

Parameters

pf.SlicedInference(...):

Parameter Default
slice_height 640 slice height in pixels
slice_width 640 slice width in pixels
overlap_ratio_h 0.2 vertical overlap between slices, from 0 to 0.9
overlap_ratio_w 0.2 horizontal overlap between slices, from 0 to 0.9
iou_threshold 0.5 IoU above which two boxes are the same object
ios_threshold 0.9 share of a box inside another above which it is removed
merge_mode "nms" "nms" keeps the best box, "nmm" averages duplicates

slicer.predict(image, detector_func, verbose=False, **detector_kwargs):

Parameter Default
image RGB image
detector_func function that takes a slice (an RGB array) and returns Detections
verbose False print the number of detections per slice
**detector_kwargs passed on to detector_func