SAM 2: segment what you click

SAM 2 segments the thing you point at. Give it a click, a box, or both, and it returns a mask with its box and the model's own estimate of how good the mask is. It does not know what it segmented: the mask is unnamed unless you name it. For masks of everything matching a phrase, see SAM 3; for a smaller, faster model with the same prompts, see EdgeTAM.

Quick start

A click on the laptop in the example photograph:

Python
import mozo
from mozo.image import load_image

image = load_image("example.jpg")
model = mozo.get_model("sam2/tiny")

found = model.predict(image, points=[[1020, 960]], labels=[1])

for candidate in found:
    print(round(candidate.confidence, 3), candidate.bbox, candidate.masks[0].shape)
Text only
0.982 [748.0, 777.0, 1353.0, 1132.0] (1281, 1920)
0.906 [513.0, 376.0, 1529.0, 1259.0] (1281, 1920)
0.886 [749.0, 791.0, 1347.0, 1133.0] (1281, 1920)

One click returns three candidate masks, best first: here the laptop, then a larger region around it, then the laptop again with a slightly different edge. A box is usually unambiguous, so ask for one mask:

Python
found = model.predict(image, boxes=[685, 783, 1355, 1136], multimask_output=False, name="laptop")
found[0].class_name, round(found[0].confidence, 3)     # ('laptop', 0.983)

Over HTTP, with mozo start running:

Bash
# A click: three candidate masks, best first
curl -X POST "http://localhost:8000/predict/sam2/tiny?point=1020,960&label=1" -F "file=@example.jpg"

# A box, one mask, named by you
curl -X POST "http://localhost:8000/predict/sam2/tiny?box=685,783,1355,1136&multimask=false&name=laptop" \
  -F "file=@example.jpg"

Variants

ID Weights (torch) Runtimes Licence
sam2/tiny 156 MB torch Apache-2.0
sam2/small 184 MB torch Apache-2.0
sam2/base_plus 324 MB torch Apache-2.0
sam2/large 898 MB torch Apache-2.0

SAM 2 also publishes ONNX graphs, and CoreML graphs for every size except base_plus. Mozo does not run them yet. A promptable model exports as several graphs and needs a runner that keeps the image encode and the prompt decode apart, so asking for one is refused rather than quietly answered with torch:

Text only
RuntimeError_: 'onnx-fp32' is published for this model, but this family can only execute torch. Available: torch-fp32

Inputs and outputs

Python
model.predict(image, points=None, labels=None, boxes=None, multimask_output=True, name=None)
model.prepare(image)
Parameter Type Default Description
image str, bytes or numpy.ndarray A file path, encoded image bytes, or an HxWx3 RGB array.
points (N, 2) or (B, N, 2) None Clicks as x, y in pixels of your image.
labels (N,) or (B, N) None 1 to include a point, 0 to exclude it. Required with points.
boxes (4,) or (B, 4) None Boxes as x1, y1, x2, y2 in pixels of your image.
multimask_output bool True Three candidate masks per prompt instead of one.
name str or list[str] None What to call what you pointed at: one name, or one per prompt.

Returns a PixelFlow Detections with one row per candidate mask. Each row has:

Raises a ValueError if no prompt is given, if points arrive without labels, or if name is a list whose length is not the number of prompts.

Over HTTP the same prompts are query parameters:

Parameter Meaning
point A click as x,y. Repeat it for several.
label 1 to include the matching point, 0 to exclude it. One per point, in the same order.
box A box as x1,y1,x2,y2. Can be combined with points.
multimask true (default) for three candidate masks, false for one.
name What to call what you pointed at.
sha256 The image kept by /prepare, sent instead of the file. See below.

How it behaves

Every prompt is a set of points

A click is a point with a label: 1 to include, 0 to exclude. Neither SAM 2 nor EdgeTAM has a separate box input: a box is written as its two corners carrying reserved labels, and mozo does that for you. Points and a box can be combined, for example a box plus a click to exclude a part:

Python
found = model.predict(image, boxes=[685, 783, 1355, 1136], points=[[900, 700]], labels=[0])

labels has no default. Guessing between include and exclude returns a confident mask of the wrong thing, so points without labels raise an error.

Three candidates for a click

With multimask_output=True, the default, each prompt returns three masks ranked by the model's predicted IoU. That is the right setting for a single click, which is genuinely ambiguous: did you mean the handle, the door or the car? Take the first row, or show all three and let the user choose. A box usually says what you meant, and multimask_output=False gives one tighter mask.

No name unless you give one

A click does not say what it clicked, and mozo will not invent a name: a name comes from the weights or from you. Pass name="laptop" if you know what you pointed at.

The image encode is cached

The image encoder is the expensive part, and it depends only on the image. Mozo caches its result by pixel content, so a second prompt on the same image pays only for the small decoder. On the example photograph on an Apple M1 Max (mps), sam2/tiny took 0.41 s for the first click and 0.045 s for the box after it. On CPU at 2 megapixels, sam2/tiny encodes in about 439 ms.

The cache holds five images per model. Preparing more drops the oldest, which is then encoded again when it is next used: slower, never wrong.

Prepare an image before the first prompt

prepare(image) runs the encode before the first prompt arrives. Call it when an image opens, on a background thread if you have one, and the first click costs what every later click does. On Apple silicon, sam2/large's first click falls from 522 ms to 16 ms, because the encode moved to when the picture appeared. A click that arrives while prepare is still running waits for that encode instead of starting a second one.

Python
model.prepare(image)                                          # when the image opens
found = model.predict(image, points=[[1020, 960]], labels=[1])

Pass the same array to both calls, so neither decodes the file again.

Over HTTP, POST /prepare/sam2/{variant} with the file encodes it, keeps the image, and answers with the sha256 of the file's bytes. Each click then sends ?sha256= and no file, a few hundred bytes instead of the picture:

Bash
curl -X POST "http://localhost:8000/prepare/sam2/large" -F "file=@example.jpg"     # {"sha256": "3f9a..."}
curl -X POST "http://localhost:8000/predict/sam2/large?sha256=3f9a...&point=1020,960&label=1"

If the server does not hold that image (another worker, a restart, or it was dropped), it answers 404. Send the file again with the same ?sha256= and it is kept for the clicks after. Only these two requests keep an image; an ordinary upload is not kept. MOZO_PREPARED_MB sets how much memory kept images may use, 512 MB by default, and the least recently used goes first.

This is for a client on the same machine or network as the server, clicking interactively: an annotation tool talking to a GPU machine down the hall, for example. Without it, every click uploads the picture again. Code running mozo in-process needs only prepare(image). A client that never sends ?sha256= is answered exactly as before.

EdgeTAM, SAM 3 and OWLv2 can be prepared too; GET /models says which families can.

Weights and licence

Code and weights are Apache-2.0. Mozo's code is an extraction of facebookresearch/sam2, reduced to the image path, and mozo/vendors/sam2_deploy/PROVENANCE.md records the upstream commit it was built from. Every variant ships with its licence and a NOTICE naming the exact upstream release.