Decoding SPDX, CycloneDX and syft-json rebuilt nested archive chains from whatever the document said, with no checks on entry names, depth or length. A small crafted document could forge archive boundaries or cost memory quadratic in its size (a 3 MB SPDX allocated ~14 GB).
- decoders now hold names to the scan's `EntryName` rules, and chains to `MaxDepth` (16) and PATH_MAX in length
- a file within a refused chain is dropped, rather than moved to a top-level path or merged into its outer archive
- the scan holds itself to the same bounds, so its SBOMs always decode back: a configured `nested-archive-max-depth` past 16 stops at 16, and an entry whose chain would be too long is skipped and the archive reported as truncated
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
16.1.12 was never released, and 16.1.13 differed from it only in field descriptions, which per
`schema/json/README.md` don't get a version of their own.
- drop `schema-16.1.13.json` and set `JSONSchemaVersion` back to `16.1.12`
- regenerate `schema-16.1.12.json` and `schema-latest.json` so the `fullPath` descriptions are current
- the 16.1.12 changelog line now describes `fullPath` instead of the old `archivePath`
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
- java embeds the shared `cataloging.ArchiveSearchConfig` but only reads the two `Include*` booleans,
and setting depth or byte limits there fails `CreateSBOM`
- a `capabilities-fields:` annotation on an embedded struct now limits which of its fields are
listed, so the docs stop advertising settings that error
- building the java cataloger directly with those fields set now warns instead of debug logging
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Draining skipped entries still left a hole: a tar entry whose header doesn't read as a regular file
(a socket or directory per its mode bits, say) can carry a body, and the tar reader decompresses it
to reach the next header without anything being charged or ctx being checked.
- compressed tars are now read through a reader that charges the budget for every byte the decoder
produces and checks ctx per read, so no skip path inside the tar reader gets around it
- entry content from that stream isn't charged a second time, and the per-entry drain is gone
- zip and plain tar are unchanged
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
A recovered panic while cataloging an archive became an unknown whose reason carried the full
goroutine stack, so a crafted archive that panicked a decoder put kilobytes of stack and build paths
into the published SBOM.
- the reason is now just `panic while cataloging archive: <value>`
- the stack is logged at debug, matching `runSubTaskSafely`
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
The tree decompression budget was only spent as entry content was stored. A compressed tar reader
still decompresses whatever an entry leaves unread to reach the next header, so an excluded entry, a
file/directory clash, or the rest of one over the 2 GB per-entry cap decompressed with nothing charged
and no ctx check. A small `.tar.zst` could force hundreds of GB of work past the budget.
- in compressed tars the rest of every regular entry is now drained through the budget, with ctx
checked per chunk
- zip and plain tar are unchanged: they seek past skipped entries, so those cost nothing
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
When an archive held two entries at one path, the resolver kept the first, and this was documented as
intended. Everything that actually unpacks the archive keeps the last: `tar` (and `tar -r` appends
updates), OCI layer application, and the legacy java path's `ContentsFromZip`. So a jar could carry a
`pom.properties` reporting 2.17.1 ahead of one installing 2.14.0, and the SBOM would show the former.
- a later regular file, link or other non-directory entry now replaces the earlier one, refunding its
in-memory bytes
- a file/directory clash at one path still keeps the first, since swapping kinds would mean
re-parenting what is already under it
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
The `archiveEntries` refactor dropped `GlobMatch`'s case-insensitive flag, so with nested cataloging
off (the default) three lookups that main matched regardless of case became case-sensitive:
- license files under `/META-INF/*` and `/*` (`meta-inf/LICENSE` was missed)
- `/version.properties`
- the `**/*.jar`, `**/*.war` check behind the "nested archives not cataloged" unknown, so `LIB.JAR`
slipped past it
`archiveEntries` now has a `globFold` for those call sites. The zip side restores
`GlobMatch(true, ...)`, and the resolver side matches case-folded over every entry, since resolver
globs are case-sensitive.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
The conaninfo parser reads the package name and version out of the
`<name>/<ver>/<user>/<channel>/package/<id>/conaninfo.txt` layout. It used to read that from the file
reference's real path (the absolute host path) and fall back to the location, but the path sweep
switched it to `LocalPath()` only. A single file scan, or a directory scan rooted below
`.conan/data`, then failed to parse the path and dropped the package.
The reference's real path is used first again, falling back to `LocalPath()` (which is what nested
archive entries have, since they carry no reference).
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`indexedPaths` added each location's `RealPath`, which for a nested file is the outermost archive. So
`PackagesByPath("/opt/x.jar")` returned every package found inside it at any depth, a root OS package
owning that file got ownership-overlap edges to all of them, and
`ExcludeBinaryPackagesWithFileOwnershipOverlap` then deleted nested binary, ELF and JVM packages.
Packages are now indexed by full path and access path only. An archive's own package still sits at
root coordinates and is found by its path; what's inside it stays linked to it by CONTAINS edges
rather than by ownership of the archive file.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Nothing checked that a decoded `FullPath` actually extends `RealPath` with an archive chain. The SPDX
decoder sets `FullPath` to the file name even when the name doesn't end with the comment's chain, and
syft-json takes whatever is in the document. `{RealPath: "/x", FullPath: "/x"}` and `{RealPath: "/x"}`
share an ID but are different map keys, so decoded files, unknowns and locations could fail to line
up with the same coordinates built by a resolver.
The syft-json and SPDX decoders now clear a `FullPath` that equals `RealPath` or doesn't start with
`RealPath` + `!/`.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`Path()` fell back to `LocalPath()` when `AccessPath` was blank, so for a nested file it returned the
full chain from a resolver but the archive-local path from a hand-built or decoded location, and
callers couldn't tell which they had.
- `Path()` now falls back to `FullNestedPath()`. Root files are unchanged
- the maven parent pom lookup joins `relativePath` onto `LocalPath()`, since enough `../` on the full
path cleaned past the `!/` boundary into an outer path that the archive resolver then read as
archive-local
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`ProcessPathErrors` cut the `path="..."` it pulled out of an error at the first `!/` and built nested
coordinates from it, so a host file with `!/` in its name was reported as a file inside an archive.
- paths from error text are taken as plain real paths again, as on main
- the php extension reader already has the location in hand, so it records the unknown against that
instead of re-parsing the error (this is the one caller that could see a nested path). It also
no longer defers `Close` on a nil reader when the read fails
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`LocalPath` searched `FullNestedPath()` for the last `!/`, which is just `RealPath` for a file not in
an archive. A host file like `/data/Hello!/world.txt` came back as `/world.txt`, so lookups and
exclusions ran against the wrong path even with nested cataloging off. `ContainerPath` had the same
problem when `FullPath` was set to `RealPath`.
Both now cut `RealPath` off first and only look for a separator in the entry chain after it, which
can't contain `!/` since entry names with it are refused.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Adding an entry checked `HasPath` on every prefix of its path, so a name with thousands of
components did O(depth^2) hashing per entry before anything was charged.
- entry names over 256 path components are refused and reported like other unsafe names
- implied parent directories are found walking from the deepest parent up, stopping at the first
one that already exists
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`entryName` only rejected NUL and over-long names, so whatever an archive called an entry flowed
straight into `FullPath`, the file tree and every encoder.
- a name containing `!/` could pose as a nested archive boundary and collide with a real nested entry
- control characters and unicode line separators could add lines to SPDX tag-value output
Both are now refused, and refused names (over-long ones included) mark the resolver truncated with
the quoted name instead of being dropped silently.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: catalog GGUF models larger than the header
gguf-parser-go v0.26.3+ rejects a header whose tensors declare more data than follows it. Syft parses only a 50 MB copy of each model, so any larger model is dropped.
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
* fix: fix parser with rollback and verify with test
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
* fix: bound GGUF tensor info before the parser reads it
The header pre-scan stopped after the metadata KVs and left tensor infos to gguf-parser-go. Older releases of the library allocate whatever length a tensor name declares, so a tiny file could force a multi-GB allocation, or a fatal out of memory that panic recovery can't catch.
- tensor names are bounded like metadata keys (non-empty, length limit, shared string budget)
- dimension count is capped at `GGML_MAX_DIMS`
- a tensor count larger than the header now fails at EOF in the pre-scan
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: give the GGUF header copy the model's real size
The cataloger copies at most 50 MB of a model and parses that copy. Newer gguf-parser-go releases check the tensor data the header declares against the file size, so the truncated copy of any larger model was rejected.
- after the header is validated, the copy is extended to the size the resolver reports for the model, leaving the tail as a hole the parser never reads
- sources that report no size (OCI model artifacts today) keep the old behavior
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(deps): restore gguf-parser-go v0.26.4
The header copy now carries the model's real size, so the pin to v0.26.0 and its dependabot ignore are no longer needed. This also brings back upstream's string and array length bounds, and the library's own `IsNumeric`.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: keep the extended GGUF header copy sparse on Windows
NTFS reserves disk for the full length of a file extended with truncate unless the file is marked sparse, so cataloging a large model on Windows reserved the model's size in the temp dir until cleanup.
- the header copy is marked sparse (`FSCTL_SET_SPARSE`) before it is extended
- if the filesystem can't make it sparse (FAT32, exFAT), the copy is not extended, and models larger than it fail to parse as before
- `golang.org/x/sys` moves from an indirect to a direct dependency
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
The go-make module and its action need to move in lockstep, but dependabot bumps them in separate PRs that drift apart. Automation now bumps both together, so dependabot ignores go-make in both ecosystems.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Alex Goodman
Keith Zantow
Christopher Angelo PhillipsandAlex Goodman
dependabot[bot]
Weston Steimel
anchore-oss-update-botandanchore-oss-update-bot
Keith Zantow
Peter McConnell