Latency lies (a little): what a detector really costs per frame

Every detection library has a table with a latency column. YOLO26n: 1.7 ms. RF-DETR-N: 2.3 ms. D-FINE-N: 2.12 ms. All on a T4, all TensorRT FP16, all batch 1.

Then you put one of these models behind a camera and your frame takes 13 ms.

Nobody is lying in the strict sense. Every number in those tables is real. It's just the time of something, and that something is usually the part that makes the library look best. This post is the developer version of what's inside those numbers, what's left out, and what we measured when we stopped leaving things out.

Full disclosure first: I develop mayaku, one of the libraries below.

What the latency column usually measures

A detector in production does roughly this for every frame:

1. Pre-processing: take the camera frame (say 1920×1080), resize it to the model's input size, convert it to float, normalise it, move it to the GPU.
2. The network: the part that runs in TensorRT.
3. Post-processing: turn the raw output into boxes. For many models that means score threshold, top-k and NMS, then mapping boxes back to the original frame.

The latency column in most tables is step 2 only, with the input already sitting on the GPU. Steps 1 and 3 run on your time.

Straight from the source:

• Ultralytics reports the "inference" part of its speed counter. Its benchmark code reads results.speed["inference"], not pre-processing or post-processing (benchmarks.py).
• D-FINE's benchmark decodes and resizes images on the GPU outside the timer, then times the engine alone (tools/benchmark/trt_benchmark.py in the D-FINE repo).
• RF-DETR's benchmark (roboflow/single_artifact_benchmarking) does the pre-processing first, then times only the engine call, replayed as a CUDA graph.

None of this is secret. It's in the code. But nobody reads benchmark code before picking a model. I didn't either, for years.

NMS: the classic

Many detectors need NMS to remove duplicate boxes, and for years it was simply left out of the speed number. The RT-DETR authors said it plainly: "YOLO detectors typically report the model speed and exclude the NMS time" (RT-DETR).

How much does it matter? The YOLOv10 paper lists YOLOv8-N at 1.77 ms for the network alone and 6.16 ms with post-processing (YOLOv10, Table 1). Same model, same GPU, 3.5× slower once you count the part you can't skip.

NMS-free models (YOLOv10, YOLO26's end-to-end head, the DETR family, mayaku) mostly solved this one. The other hidden costs are still here.

The 200 ms nap

The RF-DETR paper found that latency numbers between papers didn't agree, and traced it to the GPU throttling under load. Their solution was "simply pausing for 200ms between consecutive forward passes" (RF-DETR, Section 4). In their benchmark code that is literally a sleep after every single image, "to let the GPU cool" (sab/evaluation.py).

Think about what that means. Each image gets a rested, cool GPU running at full boost clock. Then the GPU rests again, 100 times longer than the inference itself took. The paper itself says this setup "is not designed to measure sustained throughput".

No real workload looks like this. A video stream doesn't pause 200 ms between frames. Neither does a folder of images, or a server with eight cameras. In our own runs the RTX 3060 sat at its 170 W power limit during continuous inference and reported a power-cap throttle flag. That hot, power-limited GPU is the one you deploy on. Letting it sleep between frames doesn't make the measurement more realistic. It only makes the number smaller.

FP16 latency, but which accuracy?

Every latency table says "TensorRT FP16". So the FP16 engine had better give the same accuracy as the model in the accuracy column.

For a while in 2025, it didn't for RF-DETR's larger models. Users reported that FP16 TensorRT engines of RF-DETR Medium and Large "stopped detecting objects almost completely". One user measured mAP falling from 0.69 to 0.29 (#176, #352, and in DeepStream #445). The maintainers traced it to LayerNorm in FP16 and shipped a fix in October 2025, about six months after the first report. Meanwhile the published speed was the FP16 speed, the one fast mode that didn't work.

The RF-DETR paper itself calls out other papers for this: "prior work often reports latency using FP16 quantized models, but evaluates accuracy with FP32 models". Fair point. The same paper's Table 6 then reports two models, SAM2 and SigLIPv2, with "FP16 TensorRT latency with FP32 ONNX accuracy", because both "perform poorly when evaluated in FP16".

The lesson isn't that anyone is evil. It's that FP16 and INT8 accuracy is a real engineering problem 💪, and a speed number only means something if it comes from the same file that produced the accuracy number.

Why end to end is the only number I trust

Your users don't run "the network". They run a frame through your code and get boxes. So we measured exactly that:

• One machine: RTX 3060, i5-12400F, TensorRT, batch 1.
• Input: a 1920×1080 frame already decoded in RAM.
• Output: final boxes in original-image pixels, at a 0.25 confidence threshold.
• Each library's own pre- and post-processing, at its default settings: YOLO26 at 640 with the NMS-free head, RF-DETR at its default sizes (384 to 704), D-FINE at 640, mayaku at 800.
• 2000 different frames, back to back, no sleep, after 200 warm-up frames. We report the median.
• Three independent engine builds for every model, each measured separately. They agreed to within 1%.

We also measured the network alone, the same thing the tables report.

Look at the "pre-processing" column. RF-DETR-N's network takes 1.96 ms. Getting the frame into the network takes 10.5 ms. The network is 15% of the frame time.

Where those 10 ms come from

Those 10 ms aren't the network's fault. It's a 1080p frame being resized and normalised on the CPU in Python before it goes to the GPU:

• D-FINE's TensorRT inference example (tools/inference/trt_inf.py) resizes with PIL on the CPU. That's the code you get if you follow its README.
• RF-DETR's TensorRT benchmark script that ships inside the rfdetr package (rfdetr/export/benchmark.py, originally from LW-DETR) prepares images with torchvision transforms on the CPU, outside its timer.

Yes, a good engineer can move that to the GPU. But "can be fixed" is not "is fixed". Most people run a library the way its README tells them to, and they pay whatever that path costs. The latency table doesn't tell you that path costs 10 ms.

Where mayaku is great

Here's the short version, from the table above:

• At the nano size: mayaku n INT8 matches YOLO26n's speed (1.01 ms network, 2.30 vs 2.27 ms end to end) at +6.5 AP. It's more accurate than RF-DETR-N and D-FINE-N, and 5 to 6× faster end to end.
• At the medium size: mayaku m INT8 is faster than YOLO26m (2.80 vs 3.25 ms network, 4.14 vs 4.58 ms end to end) and more accurate than YOLO26l (60.1 vs 59.3). It ties RF-DETR-S and D-FINE-S on network time and accuracy, and is about 3.5× faster end to end.
• Its own pre-processing is 0.34 ms, the lowest of all four: the frame goes to the GPU as uint8 and everything else happens there.

You'll notice mayaku runs at 800 pixels while the others run at 640 or below. That's on purpose, and it's the whole idea. More pixels means small objects stay visible, and that's where a lot of real-world accuracy lives. We can afford 800 because the network is built to be cheap per pixel.

Why I like INT8

This is where I'm biased, so here's the short version.

I designed mayaku to run in INT8 from day one. The exported network uses only five operations: convolution, ReLU, add, max-pool and nearest-neighbour upsample. No attention, no softmax, no LayerNorm, no SiLU. These five are exactly what every INT8 engine handles well. The model is also trained with quantization in the loop. Across all 100 RF100-VL datasets, the INT8 model is within about ±0.1 AP of the float model. That's noise.

"Can be exported to INT8" is not the same thing. Ultralytics' own TensorRT docs show YOLOv8n going from 37 to 33 COCO mAP in INT8, and to 32 on a Jetson (Ultralytics TensorRT docs). Four or five points is roughly the gap between two model sizes, so at that point you're running a smaller model. On Apple, Ultralytics' quantize=8 Core ML export only compresses the weights; the math still runs in FP16 (we checked on an M4). RF-DETR's Core ML export has no INT8 option at all, and its FP16 Core ML export loses about 3 AP on COCO (rf-detr PR #1495, and we measured the same).

Apple is why this matters to me. The Neural Engine on the A17 Pro (iPhone 15 Pro) and M4 got much faster at int8 × int8 compute. In Apple's own Core ML numbers, full INT8 takes ResNet50 from 1.38 ms to 0.77 ms on an iPhone 15 Pro, while a vision transformer (MobileViTv2) only goes from 1.36 ms to 1.03 ms (coremltools docs). Plain convolutions get the full benefit. On an M4 Mac mini, INT8 cut mayaku n's network time by about 18% (2.92 → 2.40 ms at 800) for 0.13 AP.

What our numbers don't tell you

Our numbers aren't the final word either:

• One GPU. An RTX 3060, not a T4 or a Jetson. The CPU part also depends on your CPU.
• Accuracy. The competitors' accuracy in the table is what they publish (RF-DETR benchmark page). We also trained YOLO26n and RF-DETR-N ourselves on all 100 datasets. YOLO26n came out about 2.8 AP higher than published, and RF-DETR-N 0.3 lower (both on the validation split). On that same protocol mayaku n is still ahead of both: +4.1 over YOLO26n and +1.5 over RF-DETR-N.
• Decoding isn't counted. We start from a decoded frame in RAM, so camera capture and JPEG decoding aren't included. They're the same for everyone.
• Our code, our choices. We wrote the code that calls each library, following each library's own code as closely as we could. If we got something wrong, tell me and I'll rerun it.

What I'd ask from every library, including mine

• Publish an end-to-end number next to the network number, from a real-size frame to final boxes.
• Say what's inside the timer.
• Don't let the GPU sleep between frames.
• Report accuracy and latency from the same exported file.

The network number is still useful: it tells you how fast the model could be with a perfect pipeline. Just don't confuse it with what your users will see.

• Results, scripts and raw measurements: (link to the benchmark folder in the mayaku repo)
• RF-DETR paper · RT-DETR paper · YOLOv10 paper
• Roboflow single_artifact_benchmarking · D-FINE · Ultralytics TensorRT · Core ML quantization performance
• RF-DETR FP16 issues: #176 · #352 · #445