Troubleshooting: errors and fixes

Errors are listed by the text Mayaku prints, grouped by when they happen. Search this page for a fragment of your error message.

Installation

ImportError: cannot import name 'app' from 'mayaku.cli.__main__'

The mayaku command script was generated by an older install of a source checkout. Reinstall to regenerate it:

pip install -e .

Until then, python -m mayaku.cli ... runs the same program.

ModuleNotFoundError: No module named 'onnxruntime' (or onnx, coremltools, openvino, tensorrt, modelopt)

Export and exported-model runtimes are optional. Install the one you need:

Loading weights

weights 'NAME': not a local file and not in the manifest

The name is not a hosted model and no file has that path. Check the spelling against mayaku download --list. Model names have the form mayaku-<n|s|m|l>-<det|seg|key>.

weights file not found: PATH

The value contains / or \, so it was read as a file path, and no file exists there. Check the path relative to the current directory.

... was trained with mayaku 2.x (sidecar schema 1); run it with pip install 'mayaku<3'

The checkpoint or exported file comes from Mayaku 2, whose models are a different architecture. Mayaku 3 can't load it. Use pip install "mayaku<3" to run it, or retrain with Mayaku 3.

... carries no mayaku sidecar

The file was not written by Mayaku, or its metadata was removed. This happens, for example, with an ONNX file re-saved by another tool, a TensorRT engine built outside Mayaku, or a plain PyTorch state_dict. Mayaku needs its embedded description to run a model. Export again from the Mayaku checkpoint.

DownloadError: integrity check failed for NAME.pth after download

The downloaded file's size or SHA256 doesn't match the manifest. The partial file is deleted. Retry. If it keeps failing, the server or a proxy is serving a stale file.

Starting training

state dict does not fit this model: [...]

The warm-start weights don't fit the model the config describes. Common causes:

• Different size. --set model.tier=s with mayaku-n-det. Use the weights of the size you set, or drop the tier setting.
• Different keypoint count. Starting from a -key model with another model.keypoints.num, which shows keys such as aux.heat.weight. Start from mayaku-<size>-det instead. See Keypoints.

backbone does not fit this tier: [...]

When starting a classifier from a checkpoint, the checkpoint is a different size than model.tier. Use weights of the same size.

model.num_classes is N and the training annotations have M categories

You set model.num_classes and it disagrees with the training JSON. Remove the setting, since it is filled in automatically, or fix the data. Every entry in categories counts, including unused ones. See Datasets.

train_annotations (a COCO JSON) is required with train_images

Detector training needs --annotations. If you meant to train a classifier, add --set model.task=classify.

a classifier trains from class folders (images/<class>/...); pass no annotation files

Classifier training takes image folders only. Remove --annotations and --val-annotations.

val_annotations and val_images go together

For detectors, pass both validation options or neither.

annotation file not found: PATH / image directory not found: PATH

A dataset path is wrong. Paths are relative to the current directory.

DIR has classes the model does not know: [...]

The classification validation (or test) split has a class folder the training split doesn't have. Remove it, or add the class to the training data.

no images under DIR/<class>/

No image files were found in the class folders. Images must sit directly inside DIR/<class>/ with an extension of .jpg, .jpeg, .png, .bmp, .webp, .tif or .tiff.

oriented boxes are a task of their own: no masks or keypoints

model.obb can't be combined with model.seg or model.keypoints. Train separate models.

resume restores the run's own config; leave config, weights, size_budget, num_epochs and overrides unset

With --resume, pass only the dataset options (and optionally --output, --device, --num-gpus). The configuration comes from the saved state.

... is not the dataset this run was training on

The training split given to --resume has different classes from the original run. Pass the same dataset.

Configuration values

Extra inputs are not permitted / Unexpected keyword argument

A config field name is misspelled or doesn't exist. The error names the path, for example train.lerning_rate. Field names are listed in Configuration.

canvas (650, 650): both sides must be multiples of 32

input.size_budget and input.canvas_hw must be multiples of 32. Use, for example, 640 or 672.

Assertion failed, adam

train.optimizer accepts sgd, adamw or musgd.

Multiple GPUs

train.batch 16 does not split over 3 GPUs; use a multiple of 3

train.batch is the total across GPUs and must divide evenly. Change --set train.batch=... or the GPU count.

num_gpus=4 but only 2 CUDA device(s) are visible

Fewer GPUs are visible than requested. Check nvidia-smi and CUDA_VISIBLE_DEVICES.

MPS does not support multi-GPU training; use num_gpus=1

Apple Silicon has a single GPU device. Train with --num-gpus 1.

under torchrun the process count is torchrun's; leave num_gpus=1

Under torchrun, set the process count with --nproc-per-node and don't pass --num-gpus.

During training

CUDA out of memory

Lower train.batch (for example --set train.batch=8). The learning rate doesn't need to change, because gradients are accumulated to the same effective batch. If that isn't enough, lower --size-budget. Segmentation with model.mask_stride=4 needs more memory than the default 8.

Training is slow and the GPU is underused

Image loading is the bottleneck. Raise --set dataloader.num_workers=... if you have CPU cores to spare, and keep images on a fast local disk. Very large source images, such as 20-megapixel photos, are slow to decode. Resizing them once before training helps.

int8 vs float ... WARNING

The model loses more than 1 AP point when simulated in int8. This is common early in training and usually shrinks by the end. If you never deploy int8, train with --set model.qat=false. If you do deploy int8, measure the exported int8 file with mayaku eval.

eval image missing: PATH

A validation or test image listed in the JSON can't be read. Check that the file exists and is a valid image, and that --val-images points to the right directory.

AP is -1.0 for AP-S, AP-M or AP-L

The split has no ground-truth objects of that size. -1.0 means not applicable. See Evaluation.

Prediction

Hundreds of low-score detections

The default score threshold is 0.001, the value used for COCO AP. Pass a threshold: predictor(image, conf=0.25) or mayaku predict ... --conf 0.25. See Prediction.

Wrong colors or noticeably worse results on camera frames

NumPy images must be RGB. OpenCV frames are BGR. Pass frame[:, :, ::-1].

cannot read image PATH

OpenCV can't open the file. It is missing, not an image, or in a format OpenCV doesn't decode.

expected an (H, W, 3) RGB array or an image path; got shape ...

Grayscale and RGBA arrays aren't accepted. Convert them: np.stack([gray] * 3, -1) or rgba[:, :, :3].

graph input (...) disagrees with the sidecar canvas (...)

An exported file was modified after export so that its input size no longer matches its embedded description. Export again from the checkpoint.

Export

unknown target 'X'; available: onnx, coreml, openvino, tensorrt

Use one of the four target names.

onnx exports fp32, int8, not fp16 (and similar)

Each target supports certain precisions: ONNX fp32/int8, Core ML fp16/fp32/int8, OpenVINO fp32/fp16/int8, TensorRT fp16/int8/fp32.

int8 needs a quantization-aware model (train it with model.qat)

The model was trained with model.qat: false. Export in float, or retrain with QAT, which is the default.

TensorRT runs int8 with symmetric activation ranges only

The model was trained with model.qat_activations: affine, or is an older QAT checkpoint. Export TensorRT in fp16, or retrain with the default symmetric.

building a TensorRT engine needs a CUDA GPU

TensorRT engines can only be built on a machine with an NVIDIA GPU, ideally the one that will serve the model.

TensorRT 11 builds fp16 and int8 from an fp16 ONNX graph; install NVIDIA ModelOpt

Install it: pip install "nvidia-modelopt[onnx]", which mayaku[tensorrt] includes.

... outputs differ from the Torch model by X% RMS (tolerance Y%)

The exported file computes something measurably different from the PyTorch model, so export refused to report success. This usually points to a runtime version problem. Update the runtime package (onnxruntime, coremltools, openvino, tensorrt) and try again, or try another precision. If it persists, please open an issue with the full message and package versions.