Train an instance segmentation model

An instance segmentation model detects each object, giving a box, class and score, and also returns a pixel mask for it. In Mayaku it is a detector with a mask branch turned on (model.seg: true). Everything in Training applies.

Data

COCO JSON whose annotations have a segmentation field, either polygons or RLE:

JSON

{"id": 1, "image_id": 1, "category_id": 1, "bbox": [100, 120, 50, 80], "iscrowd": 0,
 "segmentation": [[100, 120, 150, 120, 150, 200, 100, 200]]}

See Datasets for the details.

Train

Start from the pretrained segmentation model:

Bash

mayaku train --weights mayaku-n-seg \
  --annotations data/train/_annotations.coco.json --images data/train \
  --val-annotations data/valid/_annotations.coco.json --val-images data/valid

Python

from mayaku import train

result = train(
    weights="mayaku-n-seg",
    train_annotations="data/train/_annotations.coco.json",
    train_images="data/train",
    val_annotations="data/valid/_annotations.coco.json",
    val_images="data/valid",
)

The -seg checkpoint already has the mask branch, so you don't need to set anything.

You can also add a mask branch to a detector's weights. The box part starts pretrained and the mask branch starts fresh:

Bash

mayaku train --weights mayaku-n-det --set model.seg=true \
  --annotations data/train/_annotations.coco.json --images data/train \
  --val-annotations data/valid/_annotations.coco.json --val-images data/valid

From scratch, with no weights: --set model.seg=true.

Mask settings

Field	Default	Meaning
`model.seg`	`false`	Turns the mask branch on.
`model.mask_stride`	`8`	Resolution of the mask features: `8` (1/8 of the input) or `4` (1/4, finer edges, more memory and compute). It changes the network, so it is fixed once trained.
`train.seg_gain`	`2.0`	Weight of the mask (Dice) loss.
`train.seg_cap`	`250`	Maximum number of positive samples per batch that train the mask loss, to bound memory.

Validation and metrics

Training scores both boxes and masks. The final metrics dict and eval/metrics.json hold the box metrics (AP, AP50, ...) and the mask metrics with a segm_ prefix (segm_AP, segm_AP50, segm_AP75, segm_AP-S, segm_AP-M, segm_AP-L, segm_AR1, ...). See Evaluation.

best.pt is chosen by box AP. Per-epoch mask AP is in train/log.jsonl.

Predict

Python

from mayaku import from_pretrained

predictor = from_pretrained("runs/mayaku-n-seg/train/best.pt")
det = predictor("photo.jpg")

det.masks          # torch.bool tensor (n, H, W): one full-resolution mask per detection
det.masks[0].sum() # pixel area of the first object's mask

det.masks[i] belongs to det.boxes[i], det.scores[i] and det.labels[i].
Masks have the original image's height and width, not the canvas size.
Full-resolution masks of many detections use a lot of memory on large images. Raise the score threshold to keep fewer, for example predictor("photo.jpg", conf=0.25).

Save a mask as an image:

Python

import cv2

cv2.imwrite("mask0.png", det.masks[0].numpy().astype("uint8") * 255)

Command-line output

mayaku predict writes each mask as COCO RLE under segmentation:

JSON

{
  "class_id": 0,
  "class": "widget",
  "score": 0.9132,
  "box_xyxy": [101.3, 119.8, 151.0, 200.4],
  "segmentation": {"size": [480, 640], "counts": "Xk`35n>..."}
}

size is [height, width]. Decode it with pycocotools, which is installed with Mayaku:

Python

from pycocotools import mask as mask_utils

binary = mask_utils.decode(detection["segmentation"])   # uint8 array (height, width), 1 inside

Export

Segmentation models export to every target like a detector. See Export and deployment. The exported graph outputs the mask features, and Mayaku's host-side decode assembles each object's mask. Running the artifact with from_pretrained("model.onnx") returns the same masks.