The tree decompression budget was only spent as entry content was stored. A compressed tar reader
still decompresses whatever an entry leaves unread to reach the next header, so an excluded entry, a
file/directory clash, or the rest of one over the 2 GB per-entry cap decompressed with nothing charged
and no ctx check. A small `.tar.zst` could force hundreds of GB of work past the budget.
- in compressed tars the rest of every regular entry is now drained through the budget, with ctx
checked per chunk
- zip and plain tar are unchanged: they seek past skipped entries, so those cost nothing
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
When an archive held two entries at one path, the resolver kept the first, and this was documented as
intended. Everything that actually unpacks the archive keeps the last: `tar` (and `tar -r` appends
updates), OCI layer application, and the legacy java path's `ContentsFromZip`. So a jar could carry a
`pom.properties` reporting 2.17.1 ahead of one installing 2.14.0, and the SBOM would show the former.
- a later regular file, link or other non-directory entry now replaces the earlier one, refunding its
in-memory bytes
- a file/directory clash at one path still keeps the first, since swapping kinds would mean
re-parenting what is already under it
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
The `archiveEntries` refactor dropped `GlobMatch`'s case-insensitive flag, so with nested cataloging
off (the default) three lookups that main matched regardless of case became case-sensitive:
- license files under `/META-INF/*` and `/*` (`meta-inf/LICENSE` was missed)
- `/version.properties`
- the `**/*.jar`, `**/*.war` check behind the "nested archives not cataloged" unknown, so `LIB.JAR`
slipped past it
`archiveEntries` now has a `globFold` for those call sites. The zip side restores
`GlobMatch(true, ...)`, and the resolver side matches case-folded over every entry, since resolver
globs are case-sensitive.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
The conaninfo parser reads the package name and version out of the
`<name>/<ver>/<user>/<channel>/package/<id>/conaninfo.txt` layout. It used to read that from the file
reference's real path (the absolute host path) and fall back to the location, but the path sweep
switched it to `LocalPath()` only. A single file scan, or a directory scan rooted below
`.conan/data`, then failed to parse the path and dropped the package.
The reference's real path is used first again, falling back to `LocalPath()` (which is what nested
archive entries have, since they carry no reference).
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`indexedPaths` added each location's `RealPath`, which for a nested file is the outermost archive. So
`PackagesByPath("/opt/x.jar")` returned every package found inside it at any depth, a root OS package
owning that file got ownership-overlap edges to all of them, and
`ExcludeBinaryPackagesWithFileOwnershipOverlap` then deleted nested binary, ELF and JVM packages.
Packages are now indexed by full path and access path only. An archive's own package still sits at
root coordinates and is found by its path; what's inside it stays linked to it by CONTAINS edges
rather than by ownership of the archive file.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Nothing checked that a decoded `FullPath` actually extends `RealPath` with an archive chain. The SPDX
decoder sets `FullPath` to the file name even when the name doesn't end with the comment's chain, and
syft-json takes whatever is in the document. `{RealPath: "/x", FullPath: "/x"}` and `{RealPath: "/x"}`
share an ID but are different map keys, so decoded files, unknowns and locations could fail to line
up with the same coordinates built by a resolver.
The syft-json and SPDX decoders now clear a `FullPath` that equals `RealPath` or doesn't start with
`RealPath` + `!/`.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`Path()` fell back to `LocalPath()` when `AccessPath` was blank, so for a nested file it returned the
full chain from a resolver but the archive-local path from a hand-built or decoded location, and
callers couldn't tell which they had.
- `Path()` now falls back to `FullNestedPath()`. Root files are unchanged
- the maven parent pom lookup joins `relativePath` onto `LocalPath()`, since enough `../` on the full
path cleaned past the `!/` boundary into an outer path that the archive resolver then read as
archive-local
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`ProcessPathErrors` cut the `path="..."` it pulled out of an error at the first `!/` and built nested
coordinates from it, so a host file with `!/` in its name was reported as a file inside an archive.
- paths from error text are taken as plain real paths again, as on main
- the php extension reader already has the location in hand, so it records the unknown against that
instead of re-parsing the error (this is the one caller that could see a nested path). It also
no longer defers `Close` on a nil reader when the read fails
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`LocalPath` searched `FullNestedPath()` for the last `!/`, which is just `RealPath` for a file not in
an archive. A host file like `/data/Hello!/world.txt` came back as `/world.txt`, so lookups and
exclusions ran against the wrong path even with nested cataloging off. `ContainerPath` had the same
problem when `FullPath` was set to `RealPath`.
Both now cut `RealPath` off first and only look for a separator in the entry chain after it, which
can't contain `!/` since entry names with it are refused.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Adding an entry checked `HasPath` on every prefix of its path, so a name with thousands of
components did O(depth^2) hashing per entry before anything was charged.
- entry names over 256 path components are refused and reported like other unsafe names
- implied parent directories are found walking from the deepest parent up, stopping at the first
one that already exists
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`entryName` only rejected NUL and over-long names, so whatever an archive called an entry flowed
straight into `FullPath`, the file tree and every encoder.
- a name containing `!/` could pose as a nested archive boundary and collide with a real nested entry
- control characters and unicode line separators could add lines to SPDX tag-value output
Both are now refused, and refused names (over-long ones included) mark the resolver truncated with
the quoted name instead of being dropped silently.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: catalog GGUF models larger than the header
gguf-parser-go v0.26.3+ rejects a header whose tensors declare more data than follows it. Syft parses only a 50 MB copy of each model, so any larger model is dropped.
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
* fix: fix parser with rollback and verify with test
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
* fix: bound GGUF tensor info before the parser reads it
The header pre-scan stopped after the metadata KVs and left tensor infos to gguf-parser-go. Older releases of the library allocate whatever length a tensor name declares, so a tiny file could force a multi-GB allocation, or a fatal out of memory that panic recovery can't catch.
- tensor names are bounded like metadata keys (non-empty, length limit, shared string budget)
- dimension count is capped at `GGML_MAX_DIMS`
- a tensor count larger than the header now fails at EOF in the pre-scan
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: give the GGUF header copy the model's real size
The cataloger copies at most 50 MB of a model and parses that copy. Newer gguf-parser-go releases check the tensor data the header declares against the file size, so the truncated copy of any larger model was rejected.
- after the header is validated, the copy is extended to the size the resolver reports for the model, leaving the tail as a hole the parser never reads
- sources that report no size (OCI model artifacts today) keep the old behavior
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(deps): restore gguf-parser-go v0.26.4
The header copy now carries the model's real size, so the pin to v0.26.0 and its dependabot ignore are no longer needed. This also brings back upstream's string and array length bounds, and the library's own `IsNumeric`.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: keep the extended GGUF header copy sparse on Windows
NTFS reserves disk for the full length of a file extended with truncate unless the file is marked sparse, so cataloging a large model on Windows reserved the model's size in the temp dir until cleanup.
- the header copy is marked sparse (`FSCTL_SET_SPARSE`) before it is extended
- if the filesystem can't make it sparse (FAT32, exFAT), the copy is not extended, and models larger than it fail to parse as before
- `golang.org/x/sys` moves from an indirect to a direct dependency
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
The go-make module and its action need to move in lockstep, but dependabot bumps them in separate PRs that drift apart. Automation now bumps both together, so dependabot ignores go-make in both ecosystems.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* Add support for scanning IPK packages using the deb-archive-cataloger
Signed-off-by: Silas Parker <silas@srp.me.uk>
* fix(debian): share License field parsing between opkg status and archive control files
- the control-file `License:` fallback now goes through the same helper as the opkg status DB path, so a package reports the same licenses whether found installed or as an archive
- bitbake `&` / `|` are rewritten to SPDX `AND` / `OR` and the whole value is validated as one expression instead of split by hand, which kept grouping like `(A & B) | C` intact
- non-SPDX values fall back to a deduped list of names with operators and parens dropped (previously the status DB path emitted `&`, `(BSD`, etc as licenses)
- fields over 4KB are ignored, bounding work per archive and keeping deep paren nesting away from the recursive expression parser
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(debian): split Yocto recipe filenames in the Source field
Yocto writes the bitbake recipe filename (`Source: zlib_1.3.2.bb`) where dpkg expects a source package name. This is now split into `Source: zlib` / `SourceVersion: 1.3.2` for both ipk archives and opkg status DB entries, instead of surfacing the `.bb` filename as the upstream package.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(debian): do not label ipk archives as Debian packages
- `.ipk` archives no longer get a `pkg:deb` PURL from the assumed debian distro; opkg-based distros (Yocto, OpenWrt) have no purl type, so these now match packages found in an opkg status DB, which already get no PURL
- docs now cover `.ipk` and note that only ar-framed ipks are supported (OpenWrt's older gzipped-tar framing is rejected as an invalid archive)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test(debian): assemble the ipk fixture at build time instead of committing it
The fixture Dockerfile now builds a Yocto-shaped `.ipk` from a checked-in `control/` directory (copied from the original poky-tiny build) using `tar`, `zstd` and binutils `ar`, the same way bitbake's `package_ipk` packs one. This drops the prebuilt binary and the 20-30 minute Yocto `Dockerfile.build` while keeping the same archive layout and control metadata.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Silas Parker <silas@srp.me.uk>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(redact): reset redaction store between command runs
cli.Command() may be invoked more than once in the same process (e.g.
when syft is embedded as a library). The clio initializer calls
internal/redact.Set on every command execution, but the redact store is
process-global and Set panics when a store already exists, so the second
invocation dies with "replace existing redaction store (probably
unintentional)".
Add redact.Reset() to clear the previous run's store and call it in the
initializer before Set. The double-Set guard in Set is kept: an
unexpected second Set within a single run still panics.
Add TestAppClioSetupConfigInitializerCanRunMultipleTimes which runs the
initializer twice; it panics with the exact reported message on the old
code and passes with the fix.
Fixes#2285
Signed-off-by: JasonMetal <935216773@qq.com>
* fix(redact): release the redact store at the end of each run
Clear the global redact store in the post-run hook instead of resetting it
before every `Set`. The double-`Set` panic keeps its meaning: a store is only
present while a run is in flight, so a second run starting mid-run still
panics rather than splitting secrets across two stores and leaking the ones
in the dropped store. Sequential runs in the same process work since the
previous run releases its store on the way out.
A run only releases the store it set, so it never clears a store owned by
someone else.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: JasonMetal <935216773@qq.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* feat: add containers-storage support for Podman and Buildah, update tests and documentation
Signed-off-by: Bruce Clark <bruce.clark@mandg.com>
* test: update containers-storage tests to validate package.json and handle missing images
Signed-off-by: Bruce Clark <bruce.clark@mandg.com>
* test: fix rootless containers-storage test setup
Install uidmap for Podman and Buildah CI jobs.
Align temporary rootful and rootless storage paths.
Clean up rootless stores within the builder's user namespace.
Signed-off-by: Bruce Clark <bruce.clark@mandg.com>
* fix: only use containers-storage when explicitly requested
- containers-storage is no longer part of automatic image resolution; it's used only for `--from containers-storage`, the `containers-storage:` scheme, or `default-pull-source: containers-storage` (now an accepted value). Opening a store runs full graph driver init (mounts, locks, store writes), which is too invasive as a probe on every plain image reference.
- rootless stores must be read from inside the builder's user namespace (`podman unshare syft ...`), documented in the README.
- the `containers_image_openpgp` tag is now applied to unit/integration tests and lint, so dev tooling compiles the same provider that ships.
- containers-storage CI runs through `make containers-storage-test` and covers both vfs and overlay stores.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: exclude cgo btrfs driver and unmount overlay store in cleanup
- cgo-enabled linux builds (`-race` tests, lint) compile go.podman.io's btrfs graph driver, which needs libbtrfs headers. Add `exclude_graphdriver_btrfs` alongside `containers_image_openpgp` everywhere the tag is set; release builds are `CGO_ENABLED=0` and never included it.
- the overlay driver bind-mounts its graphroot inside the rootless user namespace, so the containers-storage test unmounts it before removing the store, and cleanup no longer fails the run.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Bruce Clark <bruce.clark@mandg.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Bruce Clark <bruce.clark@mandg.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* ci: run the full unit suite on windows via make unit
Drops the curated `unit:windows` subset; the windows job now bootstraps like linux and runs `make unit`. Exceptions get added as failures show up.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* ci: skip fixture refresh for windows unit tests
binny has no windows release, so the bootstrap action can't install the tools the fixture refresh needs.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: skip linux container fixtures when docker can't run them
Windows CI runners only run windows containers, so image fixtures go through `testutils.SkipWithoutLinuxContainers`. No behavior change on linux/mac.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: close leaked file handles and posix squashfs paths on windows
- output writers are now closed on early returns in `scan`/`convert` (open handles block cleanup on windows)
- `newSBOMMultiWriter` used `path.Dir` on host paths, so output subdirectories were never created on windows
- squashfs walk paths were built with `filepath.Join`
- skip a few posix-only test cases (dir perms, xdg fallback, go-rpmdb ndb handle leak upstream)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(golang): resolve go source module dir natively on windows
- `parseGoModFile` passed the resolver's posix (`/c/...`) path to the go toolchain, so go source resolution silently found nothing on windows
- skip container-built fixtures (go archs, java builds) and golden image tarballs on windows; stereoscope's layer cache filenames contain `:`, which windows rejects
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: keep posix paths posix on windows
- `ChrootContext.ToChrootPath` nested volumes (`D:\cwd\C:\root`) when the scan root was on another drive, so the prefix was never trimmed
- bitnami, python wheel/egg, and alpm parsers used `filepath` on posix location paths
- alpm backup-file lookups silently missed on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* ci: pull the test fixture cache for windows unit tests
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: don't map virtual file tree paths through the host on windows
`FiletreeResolver` without a chroot (snaps) ran request paths through `filepath.Abs`, rooting them on the current volume (`/payload.txt` -> `/d/payload.txt`), so snap scans on windows found nothing.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: match tar entry globs as posix on windows
- `matchesAnyGlob` used `doublestar.PathMatch` (host separator) on tar entry names, so tar-wrapped java archives silently produced nothing on windows
- test helpers now treat binary snippet keys and `FromFile` locations as posix
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* incorporate windows path fix
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* incorporate go-rpmdb close fix
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* close squashfs fixture writers before finalizing
- windows dir listings can report a zero size for files still open, so the manifest was packed empty
(also adds a temporary windows diagnostic for the binary cataloger)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* temp: surface windows binary cataloger diagnostics
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: keep directory symlink targets posix on windows
- the directory indexer joined symlink targets with native separators, so links in the posix file tree pointed nowhere and globs on link names found nothing
- binary fixture logical keys used native separators, so nested snippets were picked as the fixture dir on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* keep addSymlinkToIndex under the funlen limit
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* drop windows skips covered by the stereoscope and go-rpmdb bumps
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Alex Goodman
Keith Zantow
Christopher Angelo PhillipsandAlex Goodman
dependabot[bot]
Weston Steimel
anchore-oss-update-botandanchore-oss-update-bot
Keith Zantow
Peter McConnell
Silas ParkerandAlex Goodman
MetalandAlex Goodman
