CLIP: classify from words, embed images and text

CLIP is two networks trained together: an image tower and a text tower. Both end in a vector in one shared space, so the photo of a forklift and the phrase "a forklift" land near each other. Mozo gives you two things from it: a score for each phrase you name against an image, and the vectors themselves, to search a collection of images by words in a database of your own.

Quick start

Python
import mozo

model = mozo.get_model("clip/base")
scores = model.predict("example.jpg", ["a photo of people at a table",
                                       "a photo of a dog",
                                       "a photo of an empty room"])

for score in scores:
    print(score.class_name, round(score.confidence, 3))
Text only
a photo of people at a table 0.259
a photo of an empty room 0.181
a photo of a dog 0.165

The vectors, one row per image or phrase:

Python
images = model.encode_image("example.jpg")                    # (1, 512)
phrases = model.encode_text(["people working on a laptop",
                             "a dog on a beach"])              # (2, 512)
images @ phrases.T                                            # [[0.244, 0.1008]]

Over HTTP, with mozo start running. /predict scores; /encode returns vectors, for images or for phrases, not both in one call:

Bash
curl -X POST "http://localhost:8000/predict/clip/base?text=a%20photo%20of%20people%20at%20a%20table&text=a%20photo%20of%20a%20dog" \
  -F "file=@example.jpg"
JSON
[{"class_id": 0, "class_name": "a photo of people at a table", "confidence": 0.259, "metadata": null},
 {"class_id": 1, "class_name": "a photo of a dog", "confidence": 0.165, "metadata": null}]
Bash
curl -X POST "http://localhost:8000/encode/clip/base" -F "file=@example.jpg"
curl -X POST "http://localhost:8000/encode/clip/base?text=people%20working%20on%20a%20laptop"
JSON
{"model": "clip/base", "revision": "2026-08-23", "dim": 512, "embeddings": [[-0.00398, -0.00317, ...]]}

Variants

ID Image tower Vector Weights (torch) Runtimes Licence
clip/base ViT-B/32 512-d 605 MB torch MIT
clip/base-16 ViT-B/16 512-d 599 MB torch MIT
clip/large ViT-L/14 768-d 1,711 MB torch MIT
clip/large-336 ViT-L/14 at 336 px 768-d 1,712 MB torch MIT

Inputs and outputs

Python
model.predict(image, text, threshold=None)
model.encode_image(image)
model.encode_text(text)
Parameter Type Default Description
image str, bytes or numpy.ndarray A file path, encoded image bytes, or an HxWx3 RGB array. encode_image also takes a list of them.
text str or list[str] One phrase or several, up to 77 tokens each.
threshold float None Minimum score to keep. None returns every phrase, scored.

predict returns a PixelFlow Classifications, best first. class_name is your phrase, class_id its position in the list you passed, confidence the score. A blank phrase, no phrase, or one over 77 tokens raises ValueError.

encode_image and encode_text return a float32 array of shape (N, 512) or (N, 768), L2-normalised, so a dot product is a cosine similarity. Always two-dimensional, even for one input.

Over HTTP:

How it behaves

The scores are cosine similarities, not probabilities

They are not softmaxed. They do not sum to one, they can be negative, and they are compressed: a good match sits far below 1.0, so 0.259 does not mean "26% sure". Compare scores against each other, or against a threshold you calibrate on your own images.

A softmax would be worse here. It is relative to whichever phrases you passed, so one phrase always scores near 1 and adding a phrase moves every other number. Because mozo does not softmax, adding a phrase does not change the other phrases' scores, and that is what makes a threshold mean anything. Softmax the scores yourself if you want CLIP's published closed-set behaviour.

If you want a score that means something on its own, for a single yes-or-no question, use SigLIP 2.

Classify, or hand off the vectors

predict is the answer to "is there a person in this frame" when nobody trained a person detector for your cameras: name the classes in words, no training, no labelled data.

encode_image and encode_text are a handoff, not an answer. A vector is useful only next to other vectors, which means a store mozo does not provide. Embed your images once, keep the vectors in a vector database, then encode a query phrase and let the database find the nearest. Searching a million frames for "forklift near a person" is then one database query, not a million model runs. Mozo is the model in that pipeline and nothing else: it does not index, search or persist.

The towers load independently

Asking for phrases never builds the image tower, and the other way round. An ingest job that only embeds images holds the image tower alone, and a query service that only embeds phrases holds 63.4M parameters instead of 151.3M (base). There is nothing to configure: the first call decides.

An index is tied to the weights that built it

A vector from base means nothing against a vector from large, and nothing against a vector from a different revision of base. So a stored index is tied to the variant and the revision that built it, and changing either means embedding the whole collection again. /encode returns both with every response for that reason, and in Python the model has .revision ("2026-08-23").

Parity

Bit-identical to openai/CLIP at commit d05afc43 on all four variants: 104 comparisons across preprocessing, token ids, image features, text features, similarities and logits_per_image, with no tolerance.

Two traps a code review would miss and only a numeric check catches. The activation is QuickGELU (x * sigmoid(1.702x)), not nn.GELU; the standard one moves the features by 0.4 without any error. And upstream writes logit_scale * image_features @ text_features.t(), where * and @ share precedence, so the scale multiplies the features before the matrix product. Same arithmetic, different rounding, 1.9e-06 apart.

Weights and licence

Code and weights are MIT, © 2021 OpenAI, from the authors' own openai/CLIP repository. That repository publishes one LICENSE and no separate weights licence, so the weights are taken to be covered by it; the NOTICE beside every checkpoint says so. The five ResNet variants OpenAI also publishes use a different image tower and are not carried. The extraction and the upstream commit are recorded in mozo/vendors/clip_deploy/PROVENANCE.md.