Train a keypoint detection model

A keypoint model detects each object, giving a box, class and score, and also locates a fixed set of K points on it: the joints of a person, the corners of a part, the landmarks of an animal. In Mayaku it is a detector with a keypoint branch turned on (model.keypoints). Everything in Training applies.

Data

COCO JSON whose annotations have keypoints, and optionally keypoint names on the categories:

JSON
{
  "annotations": [
    {"id": 10, "image_id": 1, "category_id": 1, "bbox": [100, 120, 50, 80], "iscrowd": 0,
     "keypoints": [110, 130, 2, 140, 130, 2, 125, 195, 1], "num_keypoints": 3}
  ],
  "categories": [
    {"id": 1, "name": "widget", "keypoints": ["left_top", "right_top", "bottom"]}
  ]
}

Name your keypoints left_* / right_*

Training flips images horizontally, so a point on an object's left must then be treated as its right. Mayaku pairs left_X with right_X using the names in categories[].keypoints. With names like left_eye and right_eye, flipping is handled correctly. Without names, a 17-point dataset is assumed to follow the COCO person skeleton, and any other count gets no pairs. If your points have a left/right meaning but different names, rename them, or turn flips off with --set train.aug.hflip=0.

Don't use vertical flips (train.aug.vflip) with keypoints. They would mislabel the pairs.

Train

Set the number of keypoints with model.keypoints.num, and start from the detector weights:

Bash
mayaku train --weights mayaku-n-det --set model.keypoints.num=3 \
  --annotations data/train/_annotations.coco.json --images data/train \
  --val-annotations data/valid/_annotations.coco.json --val-images data/valid
Python
from mayaku import train

result = train(
    weights="mayaku-n-det",
    overrides={"model": {"keypoints": {"num": 3}}},
    train_annotations="data/train/_annotations.coco.json",
    train_images="data/train",
    val_annotations="data/valid/_annotations.coco.json",
    val_images="data/valid",
)

The box part starts pretrained and the keypoint branch starts fresh. Keypoint names and left/right pairs are filled in from your categories (auto-config) and saved in the checkpoint.

Starting from a pretrained keypoint model

The mayaku-{n,s,m,l}-key models come with a pretrained keypoint branch for a fixed number of keypoints. They are the better start only when your data has the same keypoint count. Check it:

Python
from mayaku import from_pretrained

print(from_pretrained("mayaku-n-key").sidecar["keypoints"]["num"])

With a matching count:

Bash
mayaku train --weights mayaku-n-key \
  --annotations data/train/_annotations.coco.json --images data/train \
  --val-annotations data/valid/_annotations.coco.json --val-images data/valid

The keypoint count can't change during a warm start. Starting from a -key model with a different model.keypoints.num fails with state dict does not fit this model: ['aux.heat.weight', ...]. In that case start from mayaku-n-det, as shown above.

Validation and metrics

The final metrics dict holds the box metrics (AP, AP50, ...) and the keypoint metrics with a kpt_ prefix: kpt_AP, kpt_AP50, kpt_AP75, kpt_AP-M, kpt_AP-L, kpt_AR, kpt_AR50, kpt_AR75, kpt_AR-M, kpt_AR-L. COCO keypoint evaluation has no small-object bucket.

Keypoint AP uses COCO's OKS (object keypoint similarity). The per-keypoint tolerances are built into the model, not taken from your data.

best.pt is chosen by box AP. Per-epoch keypoint AP is in train/log.jsonl.

Predict

Python
from mayaku import from_pretrained

predictor = from_pretrained("runs/mayaku-n-det/train/best.pt")
det = predictor("photo.jpg")

det.keypoints        # float tensor (n, K, 3): x, y, visibility probability
names = predictor.sidecar["keypoints"]["names"]

for obj in det.keypoints:
    for name, (x, y, p) in zip(names, obj.tolist()):
        if p > 0.5:
            print(name, round(x), round(y))

mayaku predict writes "keypoints": [[x, y, p], ...], one triple per point, in each detection.

Export

Keypoint models export to every target like a detector. See Export and deployment.