# Troubleshooting: errors and fixes

Errors are listed by the text Mayaku prints, grouped by when they happen. Search this page for a fragment of your error message.

## Installation

### `ImportError: cannot import name 'app' from 'mayaku.cli.__main__'`

The `mayaku` command script was generated by an older install of a source checkout. Reinstall to regenerate it:

```bash
pip install -e .
```

Until then, `python -m mayaku.cli ...` runs the same program.

### `ModuleNotFoundError: No module named 'onnxruntime'` (or `onnx`, `coremltools`, `openvino`, `tensorrt`, `modelopt`)

Export and exported-model runtimes are optional. Install the one you need:

| Missing | Install |
|---|---|
| `onnx` | `pip install "mayaku[onnx]"` |
| `onnxruntime` | `pip install onnxruntime` (CPU) or `pip install onnxruntime-gpu` (CUDA). Needed both to export ONNX and to run it. |
| `coremltools` | `pip install "mayaku[coreml]"` |
| `openvino` | `pip install "mayaku[openvino]"` |
| `tensorrt`, `modelopt` | `pip install "mayaku[tensorrt]"` (Linux with CUDA) |

## Loading weights

### `weights 'NAME': not a local file and not in the manifest`

The name is not a hosted model and no file has that path. Check the spelling against `mayaku download --list`. Model names have the form `mayaku-<n|s|m|l>-<det|seg|key>`.

### `weights file not found: PATH`

The value contains `/` or `\`, so it was read as a file path, and no file exists there. Check the path relative to the current directory.

### `... was trained with mayaku 2.x (sidecar schema 1); run it with pip install 'mayaku<3'`

The checkpoint or exported file comes from Mayaku 2, whose models are a different architecture. Mayaku 3 can't load it. Use `pip install "mayaku<3"` to run it, or retrain with Mayaku 3.

### `... carries no mayaku sidecar`

The file was not written by Mayaku, or its metadata was removed. This happens, for example, with an ONNX file re-saved by another tool, a TensorRT engine built outside Mayaku, or a plain PyTorch `state_dict`. Mayaku needs its embedded description to run a model. Export again from the Mayaku checkpoint.

### `DownloadError: integrity check failed for NAME.pth after download`

The downloaded file's size or SHA256 doesn't match the manifest. The partial file is deleted. Retry. If it keeps failing, the server or a proxy is serving a stale file.

## Starting training

### `state dict does not fit this model: [...]`

The warm-start weights don't fit the model the config describes. Common causes:

- **Different size.** `--set model.tier=s` with `mayaku-n-det`. Use the weights of the size you set, or drop the `tier` setting.
- **Different keypoint count.** Starting from a `-key` model with another `model.keypoints.num`, which shows keys such as `aux.heat.weight`. Start from `mayaku-<size>-det` instead. See [Keypoints](/mayaku/more-tasks/keypoints/#starting-from-a-pretrained-keypoint-model).

### `backbone does not fit this tier: [...]`

When starting a classifier from a checkpoint, the checkpoint is a different size than `model.tier`. Use weights of the same size.

### `model.num_classes is N and the training annotations have M categories`

You set `model.num_classes` and it disagrees with the training JSON. Remove the setting, since it is filled in automatically, or fix the data. Every entry in `categories` counts, including unused ones. See [Datasets](/mayaku/data-and-training/datasets/#classes).

### `train_annotations (a COCO JSON) is required with train_images`

Detector training needs `--annotations`. If you meant to train a classifier, add `--set model.task=classify`.

### `a classifier trains from class folders (images/<class>/...); pass no annotation files`

Classifier training takes image folders only. Remove `--annotations` and `--val-annotations`.

### `val_annotations and val_images go together`

For detectors, pass both validation options or neither.

### `annotation file not found: PATH` / `image directory not found: PATH`

A dataset path is wrong. Paths are relative to the current directory.

### `DIR has classes the model does not know: [...]`

The classification validation (or test) split has a class folder the training split doesn't have. Remove it, or add the class to the training data.

### `no images under DIR/<class>/`

No image files were found in the class folders. Images must sit directly inside `DIR/<class>/` with an extension of `.jpg`, `.jpeg`, `.png`, `.bmp`, `.webp`, `.tif` or `.tiff`.

### `oriented boxes are a task of their own: no masks or keypoints`

`model.obb` can't be combined with `model.seg` or `model.keypoints`. Train separate models.

### `resume restores the run's own config; leave config, weights, size_budget, num_epochs and overrides unset`

With `--resume`, pass only the dataset options (and optionally `--output`, `--device`, `--num-gpus`). The configuration comes from the saved state.

### `... is not the dataset this run was training on`

The training split given to `--resume` has different classes from the original run. Pass the same dataset.

## Configuration values

### `Extra inputs are not permitted` / `Unexpected keyword argument`

A config field name is misspelled or doesn't exist. The error names the path, for example `train.lerning_rate`. Field names are listed in [Configuration](/mayaku/reference/configuration/).

### `canvas (650, 650): both sides must be multiples of 32`

`input.size_budget` and `input.canvas_hw` must be multiples of 32. Use, for example, 640 or 672.

### `Assertion failed, adam`

`train.optimizer` accepts `sgd`, `adamw` or `musgd`.

## Multiple GPUs

### `train.batch 16 does not split over 3 GPUs; use a multiple of 3`

`train.batch` is the total across GPUs and must divide evenly. Change `--set train.batch=...` or the GPU count.

### `num_gpus=4 but only 2 CUDA device(s) are visible`

Fewer GPUs are visible than requested. Check `nvidia-smi` and `CUDA_VISIBLE_DEVICES`.

### `MPS does not support multi-GPU training; use num_gpus=1`

Apple Silicon has a single GPU device. Train with `--num-gpus 1`.

### `under torchrun the process count is torchrun's; leave num_gpus=1`

Under `torchrun`, set the process count with `--nproc-per-node` and don't pass `--num-gpus`.

## During training

### CUDA out of memory

Lower `train.batch` (for example `--set train.batch=8`). The learning rate doesn't need to change, because gradients are accumulated to the same effective batch. If that isn't enough, lower `--size-budget`. Segmentation with `model.mask_stride=4` needs more memory than the default `8`.

### Training is slow and the GPU is underused

Image loading is the bottleneck. Raise `--set dataloader.num_workers=...` if you have CPU cores to spare, and keep images on a fast local disk. Very large source images, such as 20-megapixel photos, are slow to decode. Resizing them once before training helps.

### `int8 vs float ... WARNING`

The model loses more than 1 AP point when simulated in int8. This is common early in training and usually shrinks by the end. If you never deploy int8, train with `--set model.qat=false`. If you do deploy int8, measure the exported int8 file with `mayaku eval`.

### `eval image missing: PATH`

A validation or test image listed in the JSON can't be read. Check that the file exists and is a valid image, and that `--val-images` points to the right directory.

### AP is `-1.0` for `AP-S`, `AP-M` or `AP-L`

The split has no ground-truth objects of that size. `-1.0` means not applicable. See [Evaluation](/mayaku/deployment/evaluation/#detection-metrics).

## Prediction

### Hundreds of low-score detections

The default score threshold is `0.001`, the value used for COCO AP. Pass a threshold: `predictor(image, conf=0.25)` or `mayaku predict ... --conf 0.25`. See [Prediction](/mayaku/deployment/prediction/#the-score-threshold).

### Wrong colors or noticeably worse results on camera frames

NumPy images must be RGB. OpenCV frames are BGR. Pass `frame[:, :, ::-1]`.

### `cannot read image PATH`

OpenCV can't open the file. It is missing, not an image, or in a format OpenCV doesn't decode.

### `expected an (H, W, 3) RGB array or an image path; got shape ...`

Grayscale and RGBA arrays aren't accepted. Convert them: `np.stack([gray] * 3, -1)` or `rgba[:, :, :3]`.

### `graph input (...) disagrees with the sidecar canvas (...)`

An exported file was modified after export so that its input size no longer matches its embedded description. Export again from the checkpoint.

## Export

### `unknown target 'X'; available: onnx, coreml, openvino, tensorrt`

Use one of the four target names.

### `onnx exports fp32, int8, not fp16` (and similar)

Each target supports certain precisions: ONNX `fp32`/`int8`, Core ML `fp16`/`fp32`/`int8`, OpenVINO `fp32`/`fp16`/`int8`, TensorRT `fp16`/`int8`/`fp32`.

### `int8 needs a quantization-aware model (train it with model.qat)`

The model was trained with `model.qat: false`. Export in float, or retrain with QAT, which is the default.

### `TensorRT runs int8 with symmetric activation ranges only`

The model was trained with `model.qat_activations: affine`, or is an older QAT checkpoint. Export TensorRT in fp16, or retrain with the default `symmetric`.

### `building a TensorRT engine needs a CUDA GPU`

TensorRT engines can only be built on a machine with an NVIDIA GPU, ideally the one that will serve the model.

### `TensorRT 11 builds fp16 and int8 from an fp16 ONNX graph; install NVIDIA ModelOpt`

Install it: `pip install "nvidia-modelopt[onnx]"`, which `mayaku[tensorrt]` includes.

### `... outputs differ from the Torch model by X% RMS (tolerance Y%)`

The exported file computes something measurably different from the PyTorch model, so export refused to report success. This usually points to a runtime version problem. Update the runtime package (`onnxruntime`, `coremltools`, `openvino`, `tensorrt`) and try again, or try another precision. If it persists, please open an issue with the full message and package versions.