Twice the parameters, eight percent slower

We added one detection level to a small detector. The parameter count went from 5.3M to 10.8M. The latency went from 2.13 ms to 2.30 ms.

That's 2× the parameters for 8% more time. If you think of parameters as "how big the model is", and size as "how slow it is", that makes no sense. It made sense to me only after I measured where the time actually goes.

In Latency lies (a little) I looked at everything outside the network: pre-processing, NMS, GPUs taking naps. This post is about the inside, and why the number printed next to a model name (5M, 10M, 25M) tells you so little about it.

The change

The detector looked for objects at strides 8, 16 and 32. Every cell of the stride-8 grid looks at an 8×8 pixel patch, every cell of the stride-32 grid at a 32×32 patch. Large objects mostly live on the coarse grids.

We added a fourth level at stride 64. Nothing clever: copies of the widest layers we already had at stride 32. The numbers above are from an RTX 3060, TensorRT 10.13, FP16, batch 1, engine in a CUDA graph.

One honest footnote: part of that 8% is hidden by a canvas change from 800 to 768 (stride 64 needs a size that divides by 64). At the same 768 canvas, the level adds 63% more parameters and 13% more time. Still nowhere near 2×.

Parameters don't know how big the image is

Here's the whole trick. For a convolution with C_in input channels, C_out output channels and a k×k kernel:

Text
parameters = C_in × C_out × k × k
MACs       = parameters × H × W      (H × W = the grid it runs on)

The parameter count has no H × W in it. A 3×3 conv with 256 channels in and out holds 590k weights whether it runs on a 96×96 grid or a 12×12 one. The work is a different story: every weight is used once per grid cell.

And the grids in a detector shrink fast. On a 768 canvas:

The same 768x768 image covered by grids at stride 8, 16, 32 and 64
Each level has 4× fewer cells than the one before. Stride 64 has 144 cells. Stride 8 has 9,216.

Detectors put their widest layers on the coarsest grids, because that's where wide layers are cheap. So that's where the parameters pile up, and it's also where the compute almost disappears. Our stride-64 level holds 42% of the parameters, runs on 1.2% of the cells and does 6% of the MACs.

So the doubling is real, it's just doubling something that doesn't cost much time. But 6% of the MACs should cost a lot less than 13%. That gap is the interesting part.

How I measured

To understand it I built small test networks where I could change one thing at a time, and ran them on an idle RTX 3090:

  • GPU: RTX 3090 (24 GB), nothing else running on it.
  • Software: TensorRT 10.9, PyTorch 2.6 for export, ONNX opset 17.
  • Engine: FP16, batch 1, input already on the GPU.
  • Timing: CUDA events around the engine call. 50 warm-up runs, then 5 rounds of 200 timed runs, configurations interleaved so clock drift hits all of them alike. Median of the 1000 runs, inside a CUDA graph.

Same layer, five grids

First test: four 3×3 convolutions with 256 channels, 2.36M parameters. The exact same block, run on the grid of every pyramid level of a 768 canvas.

Grid (stride) Parameters GMACs Latency
192×192 (4) 2.36M 86.97 1.636 ms
96×96 (8) 2.36M 21.74 0.426 ms
48×48 (16) 2.36M 5.44 0.124 ms
24×24 (32) 2.36M 1.36 0.071 ms
12×12 (64) 2.36M 0.34 0.057 ms
Latency of the same conv block on five grid sizes, measured against what MACs would predict
On big grids, time follows MACs. On small grids it hits a floor.

Two things happen here.

On the big grids, latency follows MACs almost perfectly. Stride 4 has 4× the MACs of stride 8 and takes 3.8× the time. Parameters are the same in every row, so they explain nothing.

On the small grids, latency stops following MACs. From stride 8 to stride 64 the MACs drop 64×, the latency only 7.4×. If time followed MACs, the stride-64 block would take 0.007 ms. It takes 0.057 ms.

That floor is the second half of the story. A 12×12 output with 256 channels is 36,864 numbers. The 3090 has 82 streaming multiprocessors that can each keep around 1,500 threads in flight. The whole layer is a fraction of what the GPU can run at once, so most of it sits idle. You still pay to launch the kernel, schedule it and wait for it to finish before the next one starts. That cost doesn't care how many weights the layer has.

Same parameters, same MACs, 3× the time

If small layers have a fixed cost, then the number of layers should matter more than their size. So I took plain stacks of 3×3 convs on a 48×48 grid and picked four that land on exactly the same budget: 2.36M parameters and 5.44 GMACs.

Four conv stacks with the same parameters and MACs, from 1 layer of 512 channels to 64 layers of 64 channels
Same weights, same arithmetic. The deep, narrow one takes 3.2× longer.

The 64-layer version pays a few microseconds for every extra layer, and each of its layers is too thin to fill the GPU. This is why I stopped trusting parameter counts when comparing small models. A model can be "small" because it's narrow, and narrow and deep is exactly the expensive shape at batch 1.

Back to the stride-64 level

Now the level makes sense. It added 4.1M parameters, but those parameters run on 144 cells. What it really added was layers: a backbone stage, a level in each path of the neck (each with its own resize, add and convolutions), and the head on top. 129 more TensorRT layers in total.

On the 3090, at 768, the level adds 0.47 GMACs. The rest of the network gets through about 8.6 GMACs per millisecond, so at that rate 0.47 GMACs should cost about 0.05 ms. It costs 0.27 ms. That's 129 layers at about 2 µs each.

It isn't reading the weights either. In FP16 the level's 4.1M parameters are about 8 MB. The 3090 reads 8 MB in under 0.01 ms.

The TensorRT per-layer profiler agrees. Grouping every engine layer by the grid it runs on:

The stride-64 level holds 42% of the parameters, does 6% of the MACs and takes 20% of the GPU time
Lots of weights, little arithmetic, and a fifth of the time.

20% of the time matches the measured difference almost exactly (0.27 of 1.37 ms). So the most honest description of the stride-64 level is: lots of weights, little compute, and a lot of small kernels in a row.

What I take from this

  • Parameter count is a storage number. It tells you the file size and roughly how much the model can learn. It doesn't tell you the time.
  • MACs are a time number only on big grids. At stride 8 and 16 latency tracks MACs closely. At stride 32 and 64 it doesn't.
  • On small grids, count layers. At batch 1 every kernel has a fixed cost the GPU can't hide when the layer is too small to fill it.
  • Wide layers on coarse grids are cheap capacity. The weights cost disk, not time. What costs time is the fine grid (where the MACs are) and the number of layers (where the overhead is).

What these numbers don't tell you

  • Two GPUs, one family. Both are Ampere desktop cards. A Jetson, a T4 or a phone will have different floors.
  • FP16 only. In INT8 the arithmetic gets cheaper and the fixed cost doesn't, so the balance moves again.
  • Batch 1. At batch 32, small grids get 32× more cells per kernel and the per-layer cost mostly disappears. This post is about real time, one frame at a time.
  • CUDA graph on. Without one, the CPU launches every kernel separately and the per-layer cost gets bigger. Our models were 44 to 46% slower without a graph on the 3090.
  • The profiler isn't free. Per-layer profiling adds a small cost to every layer, which slightly favours small layers. Its stride-64 share still matched the measured difference.
  • Different TensorRT versions. The 3060 numbers are TensorRT 10.13, the 3090 numbers 10.9. I compare within a GPU, never across.

If you have a model that's "small" and still slow, count its layers and look at the grid each one runs on before you count its parameters.