16.1.12 was never released, and 16.1.13 differed from it only in field descriptions, which per
`schema/json/README.md` don't get a version of their own.
- drop `schema-16.1.13.json` and set `JSONSchemaVersion` back to `16.1.12`
- regenerate `schema-16.1.12.json` and `schema-latest.json` so the `fullPath` descriptions are current
- the 16.1.12 changelog line now describes `fullPath` instead of the old `archivePath`
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
- java embeds the shared `cataloging.ArchiveSearchConfig` but only reads the two `Include*` booleans,
and setting depth or byte limits there fails `CreateSBOM`
- a `capabilities-fields:` annotation on an embedded struct now limits which of its fields are
listed, so the docs stop advertising settings that error
- building the java cataloger directly with those fields set now warns instead of debug logging
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Draining skipped entries still left a hole: a tar entry whose header doesn't read as a regular file
(a socket or directory per its mode bits, say) can carry a body, and the tar reader decompresses it
to reach the next header without anything being charged or ctx being checked.
- compressed tars are now read through a reader that charges the budget for every byte the decoder
produces and checks ctx per read, so no skip path inside the tar reader gets around it
- entry content from that stream isn't charged a second time, and the per-entry drain is gone
- zip and plain tar are unchanged
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
A recovered panic while cataloging an archive became an unknown whose reason carried the full
goroutine stack, so a crafted archive that panicked a decoder put kilobytes of stack and build paths
into the published SBOM.
- the reason is now just `panic while cataloging archive: <value>`
- the stack is logged at debug, matching `runSubTaskSafely`
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
The tree decompression budget was only spent as entry content was stored. A compressed tar reader
still decompresses whatever an entry leaves unread to reach the next header, so an excluded entry, a
file/directory clash, or the rest of one over the 2 GB per-entry cap decompressed with nothing charged
and no ctx check. A small `.tar.zst` could force hundreds of GB of work past the budget.
- in compressed tars the rest of every regular entry is now drained through the budget, with ctx
checked per chunk
- zip and plain tar are unchanged: they seek past skipped entries, so those cost nothing
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
When an archive held two entries at one path, the resolver kept the first, and this was documented as
intended. Everything that actually unpacks the archive keeps the last: `tar` (and `tar -r` appends
updates), OCI layer application, and the legacy java path's `ContentsFromZip`. So a jar could carry a
`pom.properties` reporting 2.17.1 ahead of one installing 2.14.0, and the SBOM would show the former.
- a later regular file, link or other non-directory entry now replaces the earlier one, refunding its
in-memory bytes
- a file/directory clash at one path still keeps the first, since swapping kinds would mean
re-parenting what is already under it
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
The `archiveEntries` refactor dropped `GlobMatch`'s case-insensitive flag, so with nested cataloging
off (the default) three lookups that main matched regardless of case became case-sensitive:
- license files under `/META-INF/*` and `/*` (`meta-inf/LICENSE` was missed)
- `/version.properties`
- the `**/*.jar`, `**/*.war` check behind the "nested archives not cataloged" unknown, so `LIB.JAR`
slipped past it
`archiveEntries` now has a `globFold` for those call sites. The zip side restores
`GlobMatch(true, ...)`, and the resolver side matches case-folded over every entry, since resolver
globs are case-sensitive.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
The conaninfo parser reads the package name and version out of the
`<name>/<ver>/<user>/<channel>/package/<id>/conaninfo.txt` layout. It used to read that from the file
reference's real path (the absolute host path) and fall back to the location, but the path sweep
switched it to `LocalPath()` only. A single file scan, or a directory scan rooted below
`.conan/data`, then failed to parse the path and dropped the package.
The reference's real path is used first again, falling back to `LocalPath()` (which is what nested
archive entries have, since they carry no reference).
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`indexedPaths` added each location's `RealPath`, which for a nested file is the outermost archive. So
`PackagesByPath("/opt/x.jar")` returned every package found inside it at any depth, a root OS package
owning that file got ownership-overlap edges to all of them, and
`ExcludeBinaryPackagesWithFileOwnershipOverlap` then deleted nested binary, ELF and JVM packages.
Packages are now indexed by full path and access path only. An archive's own package still sits at
root coordinates and is found by its path; what's inside it stays linked to it by CONTAINS edges
rather than by ownership of the archive file.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Nothing checked that a decoded `FullPath` actually extends `RealPath` with an archive chain. The SPDX
decoder sets `FullPath` to the file name even when the name doesn't end with the comment's chain, and
syft-json takes whatever is in the document. `{RealPath: "/x", FullPath: "/x"}` and `{RealPath: "/x"}`
share an ID but are different map keys, so decoded files, unknowns and locations could fail to line
up with the same coordinates built by a resolver.
The syft-json and SPDX decoders now clear a `FullPath` that equals `RealPath` or doesn't start with
`RealPath` + `!/`.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`Path()` fell back to `LocalPath()` when `AccessPath` was blank, so for a nested file it returned the
full chain from a resolver but the archive-local path from a hand-built or decoded location, and
callers couldn't tell which they had.
- `Path()` now falls back to `FullNestedPath()`. Root files are unchanged
- the maven parent pom lookup joins `relativePath` onto `LocalPath()`, since enough `../` on the full
path cleaned past the `!/` boundary into an outer path that the archive resolver then read as
archive-local
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`ProcessPathErrors` cut the `path="..."` it pulled out of an error at the first `!/` and built nested
coordinates from it, so a host file with `!/` in its name was reported as a file inside an archive.
- paths from error text are taken as plain real paths again, as on main
- the php extension reader already has the location in hand, so it records the unknown against that
instead of re-parsing the error (this is the one caller that could see a nested path). It also
no longer defers `Close` on a nil reader when the read fails
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`LocalPath` searched `FullNestedPath()` for the last `!/`, which is just `RealPath` for a file not in
an archive. A host file like `/data/Hello!/world.txt` came back as `/world.txt`, so lookups and
exclusions ran against the wrong path even with nested cataloging off. `ContainerPath` had the same
problem when `FullPath` was set to `RealPath`.
Both now cut `RealPath` off first and only look for a separator in the entry chain after it, which
can't contain `!/` since entry names with it are refused.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Adding an entry checked `HasPath` on every prefix of its path, so a name with thousands of
components did O(depth^2) hashing per entry before anything was charged.
- entry names over 256 path components are refused and reported like other unsafe names
- implied parent directories are found walking from the deepest parent up, stopping at the first
one that already exists
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`entryName` only rejected NUL and over-long names, so whatever an archive called an entry flowed
straight into `FullPath`, the file tree and every encoder.
- a name containing `!/` could pose as a nested archive boundary and collide with a real nested entry
- control characters and unicode line separators could add lines to SPDX tag-value output
Both are now refused, and refused names (over-long ones included) mark the resolver truncated with
the quoted name instead of being dropped silently.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: catalog GGUF models larger than the header
gguf-parser-go v0.26.3+ rejects a header whose tensors declare more data than follows it. Syft parses only a 50 MB copy of each model, so any larger model is dropped.
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
* fix: fix parser with rollback and verify with test
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
* fix: bound GGUF tensor info before the parser reads it
The header pre-scan stopped after the metadata KVs and left tensor infos to gguf-parser-go. Older releases of the library allocate whatever length a tensor name declares, so a tiny file could force a multi-GB allocation, or a fatal out of memory that panic recovery can't catch.
- tensor names are bounded like metadata keys (non-empty, length limit, shared string budget)
- dimension count is capped at `GGML_MAX_DIMS`
- a tensor count larger than the header now fails at EOF in the pre-scan
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: give the GGUF header copy the model's real size
The cataloger copies at most 50 MB of a model and parses that copy. Newer gguf-parser-go releases check the tensor data the header declares against the file size, so the truncated copy of any larger model was rejected.
- after the header is validated, the copy is extended to the size the resolver reports for the model, leaving the tail as a hole the parser never reads
- sources that report no size (OCI model artifacts today) keep the old behavior
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(deps): restore gguf-parser-go v0.26.4
The header copy now carries the model's real size, so the pin to v0.26.0 and its dependabot ignore are no longer needed. This also brings back upstream's string and array length bounds, and the library's own `IsNumeric`.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: keep the extended GGUF header copy sparse on Windows
NTFS reserves disk for the full length of a file extended with truncate unless the file is marked sparse, so cataloging a large model on Windows reserved the model's size in the temp dir until cleanup.
- the header copy is marked sparse (`FSCTL_SET_SPARSE`) before it is extended
- if the filesystem can't make it sparse (FAT32, exFAT), the copy is not extended, and models larger than it fail to parse as before
- `golang.org/x/sys` moves from an indirect to a direct dependency
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
The go-make module and its action need to move in lockstep, but dependabot bumps them in separate PRs that drift apart. Automation now bumps both together, so dependabot ignores go-make in both ecosystems.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* Add support for scanning IPK packages using the deb-archive-cataloger
Signed-off-by: Silas Parker <silas@srp.me.uk>
* fix(debian): share License field parsing between opkg status and archive control files
- the control-file `License:` fallback now goes through the same helper as the opkg status DB path, so a package reports the same licenses whether found installed or as an archive
- bitbake `&` / `|` are rewritten to SPDX `AND` / `OR` and the whole value is validated as one expression instead of split by hand, which kept grouping like `(A & B) | C` intact
- non-SPDX values fall back to a deduped list of names with operators and parens dropped (previously the status DB path emitted `&`, `(BSD`, etc as licenses)
- fields over 4KB are ignored, bounding work per archive and keeping deep paren nesting away from the recursive expression parser
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(debian): split Yocto recipe filenames in the Source field
Yocto writes the bitbake recipe filename (`Source: zlib_1.3.2.bb`) where dpkg expects a source package name. This is now split into `Source: zlib` / `SourceVersion: 1.3.2` for both ipk archives and opkg status DB entries, instead of surfacing the `.bb` filename as the upstream package.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(debian): do not label ipk archives as Debian packages
- `.ipk` archives no longer get a `pkg:deb` PURL from the assumed debian distro; opkg-based distros (Yocto, OpenWrt) have no purl type, so these now match packages found in an opkg status DB, which already get no PURL
- docs now cover `.ipk` and note that only ar-framed ipks are supported (OpenWrt's older gzipped-tar framing is rejected as an invalid archive)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test(debian): assemble the ipk fixture at build time instead of committing it
The fixture Dockerfile now builds a Yocto-shaped `.ipk` from a checked-in `control/` directory (copied from the original poky-tiny build) using `tar`, `zstd` and binutils `ar`, the same way bitbake's `package_ipk` packs one. This drops the prebuilt binary and the 20-30 minute Yocto `Dockerfile.build` while keeping the same archive layout and control metadata.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Silas Parker <silas@srp.me.uk>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(redact): reset redaction store between command runs
cli.Command() may be invoked more than once in the same process (e.g.
when syft is embedded as a library). The clio initializer calls
internal/redact.Set on every command execution, but the redact store is
process-global and Set panics when a store already exists, so the second
invocation dies with "replace existing redaction store (probably
unintentional)".
Add redact.Reset() to clear the previous run's store and call it in the
initializer before Set. The double-Set guard in Set is kept: an
unexpected second Set within a single run still panics.
Add TestAppClioSetupConfigInitializerCanRunMultipleTimes which runs the
initializer twice; it panics with the exact reported message on the old
code and passes with the fix.
Fixes#2285
Signed-off-by: JasonMetal <935216773@qq.com>
* fix(redact): release the redact store at the end of each run
Clear the global redact store in the post-run hook instead of resetting it
before every `Set`. The double-`Set` panic keeps its meaning: a store is only
present while a run is in flight, so a second run starting mid-run still
panics rather than splitting secrets across two stores and leaking the ones
in the dropped store. Sequential runs in the same process work since the
previous run releases its store on the way out.
A run only releases the store it set, so it never clears a store owned by
someone else.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: JasonMetal <935216773@qq.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
Keith Zantow
Alex Goodman
Christopher Angelo PhillipsandAlex Goodman
dependabot[bot]
Weston Steimel
anchore-oss-update-botandanchore-oss-update-bot
Keith Zantow
Peter McConnell
Silas ParkerandAlex Goodman
MetalandAlex Goodman