Files
syft/internal
Alex Goodman 50dbb33e66 fix(ai): bound model companion file reads and validate their contents (#5344)
Companion files next to AI models (`config.json`, `README.md`, and the OCI model.file, license and model config layers) are now read with named size limits, and are only used when they look like what they claim to be.

- JSON companions are read whole up to a limit and skipped when larger, rather than being fully buffered or parsed truncated. OCI companion layers are fetched one byte past the cap so oversize is detectable.
- README and license layers read a prefix. The frontmatter block is capped at 256 KB and list values at 32.
- dir and OCI scans share one 8 MB safetensors header cap. Directory scans used to allow 100 MB.
- `config.json` is only used when it has `architectures` or `_name_or_path`, and is looked up beside the model or in its parent directory, no longer all the way up to the scan root
- OCI model.file layers are processed in a stable order, so the chosen name no longer changes between runs
- README frontmatter:
  - a list-form `license:` is accepted and no longer drops `base_model`
  - `null` values are ignored
  - delimiters must be exact `---` lines
- model names from `_name_or_path` / `base_model`:
  - Windows paths are handled
  - `.`, `..` and `/` fall back to the next name source
  - names are capped at 256 characters

Behavior changes worth noting:
- a `config.json` two or more directories above the model no longer names it
- JSON `config.json` files without `architectures` or `_name_or_path` (e.g. `generation_config.json`, app configs) are ignored
- safetensors headers over 8 MB in directory scans are now rejected, matching OCI scans (real shard headers are well under 1 MB)

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
2026-09-29 09:33:57 -04:00
..
2023-08-29 15:52:26 -04:00
2020-11-17 12:37:13 -05:00
2022-02-22 21:45:12 -05:00