Grounding DINO: detect what you describe

Grounding DINO finds objects you describe in words. It answers the same question as OWLv2, with the same call, the same endpoint and the same result type, so you can swap one for the other. Reach for it when your prompt is a description rather than a noun: it mixes the words into the image features six times and lets the decoder look back at them, so "the mug on the left" is read as a phrase, not as a bag of words.

Quick start

Python
import mozo

model = mozo.get_model("grounding_dino/tiny")
found = model.predict("example.jpg", ["a laptop", "a coffee cup", "a phone"])

for detection in found:
    print(detection.class_name, round(detection.confidence, 2), detection.bbox)
Text only
a phone 0.89 [1158.58, 1068.93, 1455.86, 1190.51]
a laptop 0.81 [680.65, 768.13, 1360.23, 1138.77]
a coffee cup 0.54 [908.0, 471.76, 1055.76, 608.64]

Over HTTP, with mozo start running. Repeat text for each phrase:

Bash
curl -X POST "http://localhost:8000/predict/grounding_dino/tiny?text=a%20laptop&text=a%20phone" \
  -F "file=@example.jpg"

Variants

ID Weights (torch) Runtimes Licence
grounding_dino/tiny 694 MB torch Apache-2.0
grounding_dino/base 938 MB torch Apache-2.0

Upstream publishes no others. tiny (Swin-T) scores 48.4 box AP on COCO zero-shot and base (Swin-B) 56.7, for 35% more weights, by the authors' numbers. tiny is 82% of upstream's own downloads, but 8.3 AP is a wide enough gap that base is worth trying.

Inputs and outputs

Python
model.predict(image, text, threshold=None)
Parameter Type Default Description
image str, bytes or numpy.ndarray A file path, encoded image bytes, or an HxWx3 RGB array.
text str or list[str] One phrase or several. No . or ? inside a phrase; about 60 phrases fit.
threshold float None Minimum score to keep. None applies upstream's default of 0.35.

Returns a PixelFlow Detections, best first. Each detection has a bbox in pixels of your image, a confidence, class_name set to the phrase you passed, and class_id set to its index in text.

Raises a ValueError if no phrase is given, a phrase is empty, a phrase contains . or ?, or the phrases together pass the model's 256-token budget.

Over HTTP: text (repeat it for several phrases) and threshold as query parameters.

How it behaves

Three rules for prompts

All your phrases are joined into one caption, so several phrases cost one forward pass.

The name you get back is the phrase you asked for

Upstream decodes the matching tokens back into a string, which can return "yellow school" for "a yellow school bus". Mozo reports which prompt matched instead, exactly, from the phrase map the tokenizer already builds. So class_name is your string and class_id its index. This is the one place mozo deliberately differs from upstream, and it changes a string, not a number. PROVENANCE.md has the reasoning.

Nothing is suppressed

As with OWLv2, two prompts describing the same thing can both find it, and overlapping boxes for one prompt are the model's answer, not a failure to deduplicate.

It costs the same every time you ask

Measured on the example photograph, mps, three prompts:

first look at an image asking the same image again
owlv2/base-ensemble 458 ms 13 ms
grounding_dino/tiny 472 ms 472 ms
grounding_dino/base 656 ms 656 ms

On a first look the two are comparable. Asking again, OWLv2 is about 40 times faster, and that gap comes from the architecture. OWLv2's image encode depends only on the image, so mozo caches it. Grounding DINO mixes the text into the image features, so its image encode depends on the prompt and there is nothing to keep. If you ask one image many questions, that is the whole comparison.

Torch only

The image is resized to a short side of 800, with the long side capped near 1333: the aspect ratio is preserved and nothing is padded. So unlike most families there is no fixed input size, and a graph exported at one shape cannot take another. No ONNX or CoreML artifact is published.

No transformers at run time

The text encoder is BERT, but the published checkpoint carries its own fine-tuned BERT weights. Mozo rebuilds that shape and ships the WordPiece vocabulary in the wheel, so tokenizing a prompt needs no network and no transformers.

Verified on every stage

Mozo's extraction is checked against the authors' implementation on the tokenizer, preprocessing, the BERT tower, all three Swin levels, the encoder, all six decoder layers and the final logits and boxes, with no tolerance: 138 comparisons across both variants, every one bit-identical.

Weights and licence

Code and weights are Apache-2.0. Mozo's code derives from IDEA-Research/GroundingDINO, the authors' own PyTorch release (© 2023 – present IDEA Research). The weights' licence is stated by the authors on the Hugging Face repositories that serve the files; the NOTICE published with each variant records exactly where. The tokenizer vocabulary is Google's bert-base-uncased (Apache-2.0). mozo/vendors/grounding_dino_deploy/PROVENANCE.md records the upstream commit.