RF-DETR: detection, segmentation and keypoints

RF-DETR is Roboflow's real-time transformer detector. It finds objects of COCO's 80 classes and returns a box, a class and a confidence for each. The seg- variants add a mask to every box, and keypoint-preview returns people with 17 joints each. It needs no overlap threshold: the model predicts one box per object, so there is no non-maximum suppression to tune.

Quick start

Python
import mozo

model = mozo.get_model("rfdetr/medium")
found = model.predict("example.jpg", threshold=0.5)

for detection in found[:4]:
    print(detection.class_name, round(detection.confidence, 2), detection.bbox)
Text only
person 0.97 [0.61, 122.33, 761.01, 1268.59]
person 0.96 [1303.81, 16.69, 1917.26, 1270.06]
laptop 0.94 [685.36, 783.82, 1354.26, 1135.56]
cell phone 0.93 [1163.78, 1082.97, 1449.97, 1182.17]

Over HTTP, with mozo start running:

Bash
curl -X POST "http://localhost:8000/predict/rfdetr/medium?threshold=0.5" -F "file=@example.jpg"
JSON
[
  {"bbox": [0.61, 122.33, 761.01, 1268.59], "class_id": 1, "class_name": "person",
   "confidence": 0.97, "masks": null, "segments": null, "keypoints": null, "...": "..."}
]

Every detection carries every PixelFlow field, including the ones that are null.

Variants

ID Task Weights (torch) Runtimes Licence
rfdetr/nano detection 366 MB torch, onnx, coreml Apache-2.0
rfdetr/small detection 386 MB torch, onnx, coreml Apache-2.0
rfdetr/medium detection 405 MB torch, onnx, coreml Apache-2.0
rfdetr/large detection 136 MB torch, onnx, coreml Apache-2.0
rfdetr/seg-nano instance segmentation 135 MB torch, onnx, coreml Apache-2.0
rfdetr/seg-small instance segmentation 135 MB torch, onnx, coreml Apache-2.0
rfdetr/seg-medium instance segmentation 143 MB torch, onnx, coreml Apache-2.0
rfdetr/seg-large instance segmentation 145 MB torch, onnx, coreml Apache-2.0
rfdetr/keypoint-preview person keypoints 164 MB torch Apache-2.0

runtime="auto", the default, picks CoreML on Apple silicon (mps) and torch on CUDA and CPU. Each graph is checked against the torch model before it is published: the same detections, classes and count on real photographs, with boxes within 0.2 px and scores within 0.01. To choose one yourself, ask a ModelManager for it by name:

Python
from mozo import ModelManager

model = ModelManager().get_model("rfdetr", "small", runtime="onnx-fp32")   # pip install 'mozo[onnx]'
model.runtime      # 'onnx-fp32'

Inputs and outputs

Python
model.predict(image, threshold=0.5, labels=None)
Parameter Type Default Description
image str, bytes or numpy.ndarray A file path, encoded image bytes, or an HxWx3 RGB array.
threshold float 0.5 Minimum confidence to keep a detection.
labels list[str] None Class names for this call, replacing the checkpoint's own.

Returns a PixelFlow Detections, best first. Each detection has:

Over HTTP the same parameters are query parameters: threshold, and labels as a comma-separated list (labels=hardhat,vest).

How it behaves

Class names come from the weights

The names ship with each variant, so they are the vocabulary the checkpoint was trained on, not an assumption. A checkpoint of your own that carries no names returns class_id with class_name unset. Pass labels=[...] to name them: mozo never guesses a name.

Masks

The seg- variants answer the same call with a mask on every detection:

Python
found = mozo.get_model("rfdetr/seg-small").predict("example.jpg")
mask = found[0].masks[0]
mask.shape, mask.dtype      # ((1281, 1920), dtype('bool'))

Keypoints

keypoint-preview returns people, each with COCO's 17 joints in COCO's order (nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles). Each joint has x and y in pixels of your image, a name, and a confidence:

Python
found = mozo.get_model("rfdetr/keypoint-preview").predict("example.jpg", threshold=0.5)
nose = found[0].keypoints[0]
nose.name, nose.x, nose.y, round(nose.confidence, 3)    # ('nose', 1512.67, 299.17, 0.998)

A joint the model cannot see still has its slot. It comes back with a confidence near zero and coordinates that mean nothing, so filter on the confidence before you read a position. Upstream does the same, and mozo keeps it: dropping the hidden joints would renumber the rest, and a joint's index is what tells you which joint it is.

Two things differ from the other variants:

The model also predicts an uncertainty ellipse per joint. Mozo does not return it: a PixelFlow Detections has nowhere to carry one, and reducing it to a single number the model never predicted would be worse than leaving it out.

Roboflow reports 71.8 keypoint AP on COCO at 9.8 ms on a T4 under TensorRT FP16. That is their measurement, not mozo's. For pose on boxes you already have, from any detector, see ViTPose.

Weights and licence

Code and weights are Apache-2.0, from Roboflow's rf-detr release. Every variant ships with its licence and a NOTICE naming the exact upstream release, and mozo's extraction records the upstream commit it was built from in mozo/vendors/rfdetr_deploy/PROVENANCE.md.