Train a model with mayaku.train

mayaku.train (command line: mayaku train) trains every model type: detection, instance segmentation, keypoints, oriented boxes and classification. This page covers what all of them share. The task pages cover what differs: segmentation, keypoints, oriented boxes, classification.

The basic call

from mayaku import train

result = train(
    weights="mayaku-n-det",
    train_annotations="data/train/_annotations.coco.json",
    train_images="data/train",
    val_annotations="data/valid/_annotations.coco.json",
    val_images="data/valid",
)

mayaku train --weights mayaku-n-det \
  --annotations data/train/_annotations.coco.json --images data/train \
  --val-annotations data/valid/_annotations.coco.json --val-images data/valid

Where the model comes from

Training starts from pretrained weights, from a config, or from both:

weights is a hosted model name such as mayaku-n-det (see Models) or the path to any .pt or .pth checkpoint Mayaku wrote, including the best.pt of an earlier run.

Fine-tuning from pretrained weights is the recommended default. It converges in fewer epochs and reaches higher accuracy on small datasets than training from scratch.

What a warm start can and cannot change

When you start from a checkpoint, the weights are loaded into the model your config describes:

• Class count. It may differ from the checkpoint's. The class layer is re-initialized for your classes, and the log says classifier re-initialised for N classes.
• Heads. Masks, keypoints and oriented boxes may be added to or removed from a detector's weights. For example, mayaku-n-det with --set model.seg=true trains a segmentation model whose backbone and box head start pretrained and whose mask branch starts fresh.
• Size. It cannot change. mayaku-n-det loads only into an n model. Setting model.tier=s with n weights fails with state dict does not fit this model.
• Keypoint count. It cannot change from the checkpoint's. To train a different keypoint count, start from the detector weights (mayaku-n-det), not from mayaku-n-key. See Keypoint detection.

Training from scratch

Leave out weights:

mayaku train --set model.tier=s \
  --annotations data/train/_annotations.coco.json --images data/train \
  --val-annotations data/valid/_annotations.coco.json --val-images data/valid

A from-scratch run uses the full recipe: 125 epochs, with a longer warmup for the label assigner. Auto-config still sets the class count and input size, but it does not shorten the schedule.

Changing settings

Every setting is a field of the config (see Configuration for all of them). There are three ways to set a field, and they combine.

Command line, repeatable --set dotted.path=value (the value is parsed as YAML, so true, 0.005, null and [640, 640] get their types):

mayaku train --weights mayaku-n-det --set train.lr=0.005 --set train.batch=32 \
  --annotations data/train/_annotations.coco.json --images data/train

Python, with a nested overrides mapping:

train(weights="mayaku-n-det", overrides={"train": {"lr": 0.005, "batch": 32}},
      train_annotations="data/train/_annotations.coco.json", train_images="data/train")

A YAML file, passed as the first argument (config= in Python):

# my_run.yaml
model:
  tier: m
  seg: true
train:
  epochs: 150
  batch: 32

mayaku train my_run.yaml --weights mayaku-m-det \
  --annotations data/train/_annotations.coco.json --images data/train

Unknown keys are rejected at any level, so a typo fails immediately instead of being ignored.

Two shortcuts set common fields: num_epochs / --epochs sets train.epochs, and size_budget / --size-budget sets input.size_budget.

Input size

Every image is resized, preserving its aspect ratio, and padded onto one fixed rectangle called the canvas (letterboxing). The same canvas is used for training, evaluation, export and deployment.

• input.size_budget (default 800) is the compute dial. The canvas never has more than size_budget² pixels.
• The canvas shape follows your data. If nearly all training images share one aspect ratio, the canvas takes that ratio, so a 16:9 dataset at budget 800 trains on a 576×1024 canvas instead of 800×800 with padding bars. Mixed aspect ratios get a square canvas.
• Both sides are always multiples of 32.

(Height × width.)

A higher budget helps small objects and costs time and memory. A lower budget is faster. To fix the canvas exactly, set input.canvas_hw, for example --set input.canvas_hw=[640,640].

The canvas is saved in the checkpoint, so prediction and export always use the size the model was trained at.

Auto-config

At the start of training, Mayaku fills in the settings you left unset from your training annotations. Each change is logged:

[mayaku.train] auto-config train.epochs: 125 -> 100

Rules:

• Anything you set yourself wins. A field counts as set when it appears in your YAML, in overrides or --set, or through --epochs / --size-budget. Auto-config never touches it.
• Auto-config never changes the model size, the heads, the loss, the optimizer or the learning rate.
• --set auto_config.enabled=false turns it off. The class count and canvas are still filled in, because a model can't be built without them, but plainly: the data's class count and a square canvas at the size budget.
• If you set model.num_classes and it disagrees with the data, training stops with an error.

The run directory

By default a run writes to runs/<name>/, where <name> is the weights file's name (mayaku-n-det), your YAML file's name, or mayaku_run. Use output_dir / --output to choose another directory.

Note: two runs with the same name write to the same directory, and the second overwrites the first's files. Give each experiment its own --output.

runs/mayaku-n-det/
├── train/
│   ├── config.yaml      the full resolved config, auto-config included
│   ├── recipe.json      the training recipe and canvas, as used
│   ├── tier.json        the network's architecture
│   ├── log.jsonl        one JSON record per epoch: losses, learning rate, time, metrics
│   ├── best.pt          the model at its best validation score (only with a validation split)
│   ├── best.json        that epoch's number and metrics
│   ├── last.pt          the model after the final epoch
│   ├── state.pt         everything needed to resume (see Resuming)
│   └── metadata.json    a run summary: sizes, timings, metrics, versions, device
└── eval/
    └── metrics.json     the final evaluation of the final weights

• best.pt and last.pt are self-contained: each holds the weights, the full config, the class names, the canvas and the prediction thresholds. Pass either to from_pretrained, mayaku predict, mayaku eval, mayaku export, or back to train as weights.
• Both hold the EMA weights (an exponential moving average of the training weights), which is the model that is evaluated.
• best.pt is the epoch with the highest box AP (top-1 accuracy for a classifier). This is also true for segmentation and keypoint models.
• state.pt is much larger than the others because it also holds the optimizer state. You don't need it once training is done.

Validation

With val_annotations and val_images (--val-annotations and --val-images):

• The model is evaluated after every epoch, and the line is logged:
  Text onlyCopy

  epoch  41  box 0.812 cls 0.655 dfl 0.901  AP 0.6123  AP50 0.8540  AP75 0.6702  S 0.3120  M 0.5874  L 0.7012

• best.pt is kept, and after training it is scored once more through the same code path a deployment uses. The result goes to eval/metrics.json and result["metrics"].

Without a validation split, the run trains without scoring, writes no best.pt, and last.pt is the result. result["metrics"] is None.

S, M and L are AP on small, medium and large objects. -1.0 means the split has no objects of that size. See Evaluation.

The result

train returns a dict:

The training recipe

The defaults are the recipe Mayaku is tuned with. The fields you are most likely to change:

Batch size and gradient accumulation. Gradients are accumulated until about 64 images (train.lr_ref_batch) have been seen before each optimizer step, so train.lr means the same thing at any batch size. At batch: 16 each step accumulates 4 batches. Lowering train.batch to fit in memory does not require changing the learning rate.

The final stage. The last train.final_frac of the epochs (16% from scratch, 20% when fine-tuning) train without mosaic or scale and translate jitter, on images that look like the ones the model will see in use. The log announces it: epoch 80: mosaic off, final stage ....

Quantization-aware training (QAT) is on by default (model.qat: true). The model learns weights that keep their accuracy in int8, so the same checkpoint exports to float or int8 (see Export). After each evaluation, training also scores the first 500 validation images in simulated int8 and logs the difference:

epoch  41  ...  AP 0.6123 ...  int8 vs float -0.3 AP

It appends WARNING when int8 loses more than 1 AP point. Turn QAT off with --set model.qat=false if you will only ever deploy in float.

All recipe fields are listed in Configuration.

Resuming

After every epoch, train/state.pt holds everything needed to continue: weights, EMA weights, optimizer, loss scaler, best score so far, and the config. To continue an interrupted run, pass the state file or the run's train/ directory, with the same dataset paths:

mayaku train --resume runs/mayaku-n-det/train \
  --annotations data/train/_annotations.coco.json --images data/train \
  --val-annotations data/valid/_annotations.coco.json --val-images data/valid

train(resume="runs/mayaku-n-det/train",
      train_annotations="data/train/_annotations.coco.json", train_images="data/train",
      val_annotations="data/valid/_annotations.coco.json", val_images="data/valid")

• Training continues at the epoch after the last completed one.
• The config comes from the state. Don't pass config, weights, num_epochs, size_budget or overrides: that fails with resume restores the run's own config.
• The training split must have the same classes as the original run, or it fails with ... is not the dataset this run was training on.
• Output goes to the same run directory unless you pass --output.

Multiple GPUs

On one machine with several NVIDIA GPUs, set num_gpus / --num-gpus:

mayaku train --weights mayaku-m-det --num-gpus 4 \
  --annotations data/train/_annotations.coco.json --images data/train \
  --val-annotations data/valid/_annotations.coco.json --val-images data/valid

• One process runs per GPU.
• train.batch is the total batch across GPUs, so the recipe trains the same on 1 GPU or 8: with batch: 32 on 4 GPUs, each GPU takes 8 images per step. The batch must divide evenly by the GPU count, otherwise training stops with train.batch 16 does not split over 3 GPUs; use a multiple of 3.
• BatchNorm statistics are synchronized across GPUs (train.sync_bn: true), so the run matches its single-GPU equivalent.
• Only the first process evaluates, logs and writes files.
• MPS (Apple Silicon) does not support multiple devices.

For several machines, start Mayaku under torchrun, leave --num-gpus at 1, and torchrun sets the process count:

torchrun --nnodes 2 --nproc-per-node 8 --rdzv-backend c10d --rdzv-endpoint HOST:29500 \
  -m mayaku.cli train --weights mayaku-l-det \
  --annotations data/train/_annotations.coco.json --images data/train \
  --val-annotations data/valid/_annotations.coco.json --val-images data/valid

Every process must see the dataset at the same paths. From Python under torchrun, every process calls mayaku.train(...) with the same arguments, and every process gets the first process's result.

Logging from Python

train prints its log lines. Pass log= to send them somewhere else:

import logging
from mayaku import train

logger = logging.getLogger("mayaku")
train(weights="mayaku-n-det", log=logger.info,
      train_annotations="data/train/_annotations.coco.json", train_images="data/train")

The per-epoch records, with every loss term and metric, are in train/log.jsonl, one JSON object per line. Read it with pandas.read_json("runs/mayaku-n-det/train/log.jsonl", lines=True).

train() arguments