* fix(redact): reset redaction store between command runs
cli.Command() may be invoked more than once in the same process (e.g.
when syft is embedded as a library). The clio initializer calls
internal/redact.Set on every command execution, but the redact store is
process-global and Set panics when a store already exists, so the second
invocation dies with "replace existing redaction store (probably
unintentional)".
Add redact.Reset() to clear the previous run's store and call it in the
initializer before Set. The double-Set guard in Set is kept: an
unexpected second Set within a single run still panics.
Add TestAppClioSetupConfigInitializerCanRunMultipleTimes which runs the
initializer twice; it panics with the exact reported message on the old
code and passes with the fix.
Fixes#2285
Signed-off-by: JasonMetal <935216773@qq.com>
* fix(redact): release the redact store at the end of each run
Clear the global redact store in the post-run hook instead of resetting it
before every `Set`. The double-`Set` panic keeps its meaning: a store is only
present while a run is in flight, so a second run starting mid-run still
panics rather than splitting secrets across two stores and leaking the ones
in the dropped store. Sequential runs in the same process work since the
previous run releases its store on the way out.
A run only releases the store it set, so it never clears a store owned by
someone else.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: JasonMetal <935216773@qq.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* feat: add containers-storage support for Podman and Buildah, update tests and documentation
Signed-off-by: Bruce Clark <bruce.clark@mandg.com>
* test: update containers-storage tests to validate package.json and handle missing images
Signed-off-by: Bruce Clark <bruce.clark@mandg.com>
* test: fix rootless containers-storage test setup
Install uidmap for Podman and Buildah CI jobs.
Align temporary rootful and rootless storage paths.
Clean up rootless stores within the builder's user namespace.
Signed-off-by: Bruce Clark <bruce.clark@mandg.com>
* fix: only use containers-storage when explicitly requested
- containers-storage is no longer part of automatic image resolution; it's used only for `--from containers-storage`, the `containers-storage:` scheme, or `default-pull-source: containers-storage` (now an accepted value). Opening a store runs full graph driver init (mounts, locks, store writes), which is too invasive as a probe on every plain image reference.
- rootless stores must be read from inside the builder's user namespace (`podman unshare syft ...`), documented in the README.
- the `containers_image_openpgp` tag is now applied to unit/integration tests and lint, so dev tooling compiles the same provider that ships.
- containers-storage CI runs through `make containers-storage-test` and covers both vfs and overlay stores.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: exclude cgo btrfs driver and unmount overlay store in cleanup
- cgo-enabled linux builds (`-race` tests, lint) compile go.podman.io's btrfs graph driver, which needs libbtrfs headers. Add `exclude_graphdriver_btrfs` alongside `containers_image_openpgp` everywhere the tag is set; release builds are `CGO_ENABLED=0` and never included it.
- the overlay driver bind-mounts its graphroot inside the rootless user namespace, so the containers-storage test unmounts it before removing the store, and cleanup no longer fails the run.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Bruce Clark <bruce.clark@mandg.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Bruce Clark <bruce.clark@mandg.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* ci: run the full unit suite on windows via make unit
Drops the curated `unit:windows` subset; the windows job now bootstraps like linux and runs `make unit`. Exceptions get added as failures show up.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* ci: skip fixture refresh for windows unit tests
binny has no windows release, so the bootstrap action can't install the tools the fixture refresh needs.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: skip linux container fixtures when docker can't run them
Windows CI runners only run windows containers, so image fixtures go through `testutils.SkipWithoutLinuxContainers`. No behavior change on linux/mac.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: close leaked file handles and posix squashfs paths on windows
- output writers are now closed on early returns in `scan`/`convert` (open handles block cleanup on windows)
- `newSBOMMultiWriter` used `path.Dir` on host paths, so output subdirectories were never created on windows
- squashfs walk paths were built with `filepath.Join`
- skip a few posix-only test cases (dir perms, xdg fallback, go-rpmdb ndb handle leak upstream)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(golang): resolve go source module dir natively on windows
- `parseGoModFile` passed the resolver's posix (`/c/...`) path to the go toolchain, so go source resolution silently found nothing on windows
- skip container-built fixtures (go archs, java builds) and golden image tarballs on windows; stereoscope's layer cache filenames contain `:`, which windows rejects
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: keep posix paths posix on windows
- `ChrootContext.ToChrootPath` nested volumes (`D:\cwd\C:\root`) when the scan root was on another drive, so the prefix was never trimmed
- bitnami, python wheel/egg, and alpm parsers used `filepath` on posix location paths
- alpm backup-file lookups silently missed on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* ci: pull the test fixture cache for windows unit tests
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: don't map virtual file tree paths through the host on windows
`FiletreeResolver` without a chroot (snaps) ran request paths through `filepath.Abs`, rooting them on the current volume (`/payload.txt` -> `/d/payload.txt`), so snap scans on windows found nothing.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: match tar entry globs as posix on windows
- `matchesAnyGlob` used `doublestar.PathMatch` (host separator) on tar entry names, so tar-wrapped java archives silently produced nothing on windows
- test helpers now treat binary snippet keys and `FromFile` locations as posix
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* incorporate windows path fix
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* incorporate go-rpmdb close fix
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* close squashfs fixture writers before finalizing
- windows dir listings can report a zero size for files still open, so the manifest was packed empty
(also adds a temporary windows diagnostic for the binary cataloger)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* temp: surface windows binary cataloger diagnostics
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: keep directory symlink targets posix on windows
- the directory indexer joined symlink targets with native separators, so links in the posix file tree pointed nowhere and globs on link names found nothing
- binary fixture logical keys used native separators, so nested snippets were picked as the fixture dir on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* keep addSymlinkToIndex under the funlen limit
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* drop windows skips covered by the stereoscope and go-rpmdb bumps
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* feat: catalog CPAN distributions installed by perl clients
Adds `perl` as a language and `cpan` as a package type, with two catalogers behind them.
- `perl-cpan-installed-cataloger` reads `.meta/*/install.json` (cpanm, cpm, carton) and `auto/**/.packlist` (anything installed through `ExtUtils::MakeMaker` or `Module::Build`, including CPAN.pm). Both globs are unanchored, so a local-lib or carton application tree is found the same as a system install.
- `perl-cpan-meta-cataloger` reads an unpacked release's own `META.json` or `META.yml`, gated on a sibling `MANIFEST` so source checkouts are ignored.
Packages are keyed on the **distribution**, not the module, because that is what CPAN, MetaCPAN and the advisory data all use. `LWP.pm` belongs to `libwww-perl` and reporting it as `LWP` would make every advisory for it unreachable. The distribution name comes from `install.json`'s `dist` field, and from the `auto/` path for packlists.
purls are `pkg:cpan/<distribution>@<version>`, with the PAUSE author added as an `author` qualifier when the evidence carries it. Metadata comes in two types: `cpan-distribution` for installed evidence and `cpan-unpacked-release` for a release sitting on disk. The tiers differ in more than a name, so a consumer should not have to string-match a cataloger name to tell them apart: an unpacked release has no PAUSE path and therefore no author and no file list, and "installed and loadable by the interpreter" is a materially stronger claim than "a source tree exists here".
A packlist's version comes from `perllocal.pod`. EUMM writes it and the packlist from the same variables in the same install target, so the `auto/` path segments and the perllocal `Module` name are the same key, and the recorded `VERSION` is what was evaluated at build time rather than what can be read back statically. Scraping `$VERSION` out of the main `.pm` is the fallback, since `NO_PERLLOCAL` suppresses the file and Module::Build never writes one. Three rules pick the stanza: dashed module name, longest `installed into` libdir that prefixes the packlist path, and last match wins because the file is append-only. Without the libdir rule a second perl on the image silently supplies the version.
Where scraping is the fallback, it reads more than a plain `our $VERSION = '...'`. A version declared in the package statement (`package Foo::Bar v1.0.0;`) counts, which matters because a distribution can declare it that way and carry no `$VERSION` at all: `CPAN::02Packages::Search` is one, and without this it reports no version and matches nothing. Fully qualified `$Foo::Bar::VERSION` counts too, which is what older Dist::Zilla emitted and what real `JSON::PP` 2.27300 still carries. `qv('1.2.3')` is handled. A version computed at runtime is not, and those are reported without one rather than guessed at.
Packlist entries are parsed the way `ExtUtils::Packlist` writes them: a line can carry space-separated metadata after the path (`/path/to/File.pm type=file`), which is what `installperl` produces, so the suffix is stripped rather than the line being cut at its first space.
`META.yml` is read alongside `META.json`, because about 43% of current CPAN releases ship no `META.json` and they skew old, which is where the advisories are. Where both sit in one directory the `META.json` wins. A `META.yml` a strict parser rejects skips that directory rather than failing the scan, which is routine rather than defensive for pre-spec releases.
A packlist is literally a file list, so its paths are surfaced through `pkg.FileOwner`. They are reported as recorded, unfiltered: a path the packlist claims and the filesystem lacks means the file was removed or overwritten out from under the installer, which is worth seeing rather than hiding.
Build leftovers under `~/.cpanm/work` and `~/.cpan/build` are skipped. They genuinely are unpacked release tarballs, `MANIFEST` included, so the `MANIFEST` gate admits them and a path exclusion is the only signal available. Without it every distribution on an image that did not clean up is reported twice. The cost is that a tarball deliberately kept under `~/.cpan/build` stops being reported.
Two behaviors that look wrong but are not:
- a distribution can be reported twice at different versions. `libwww-perl` 5.836 bundles `HTTP-Date`, `HTTP-Message` and `LWP-MediaTypes`, and installing it over the modern standalone releases overwrites their `.pm` files. Both versions are genuinely present, so the merge refuses to pair disagreeing versions rather than hiding one.
- a distribution with no readable version is reported without one rather than dropped, so it stays visible.
The coverage boundary is stated in full in the `package perl` doc comment. In short: modules installed from distro packages are out of scope, since packagers strip CPAN metadata with `NO_PACKLIST` and deb, rpm and apk already report them. Core and dual-life distributions bundled with the interpreter have no coverage, because nothing on disk carries their versions; the interpreter itself is reported separately. Vendored trees, `App::FatPacker` output and PAR archives are invisible, because what they carry is module identity with no offline map to a distribution. A packlist-derived name is the installer's `NAME` and may not be the distribution name, which is left to the vulnerability data to resolve.
One correction worth calling out, since it is easy to arrive at twice: a sibling `MANIFEST` is the only thing separating an unpacked release from a source checkout. An earlier version of the guard also rejected any tree containing a `dist.ini`, on the theory that it marked a Dist::Zilla source tree. dzil ships `dist.ini` *inside* the tarballs it builds and lists it in the generated `MANIFEST`, so that exclusion was silently skipping real releases, `URI` among them.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test(cli): bump coverage image package count for the new perl fixture
The CPAN fixture added to `image-pkg-coverage` (`URI-5.35`) is shared with the CLI suite through a symlink, so the squashed package count goes from 42 to 43.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(perl): take CPAN distribution name and version from install.json dist
cpanm writes the main module's `$VERSION` into `version` and the release distvname into `dist`, and the two often disagree (`Carp-Assert-More-2.9.0` records `2.009000`). Trimming `version` off `dist` failed on those, so the whole distvname became the package name, and the module version never paired with the packlist record for the same install, which reported the distribution twice.
- name and version are now split out of `dist` at the last `-` followed by a version, as CPAN::DistnameInfo does
- a `-TRIAL` suffix is dropped, so `Try-Tiny-0.26-TRIAL` reports as `Try-Tiny` `0.26`
- the PAUSE path fallback uses the same split
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: report scanner errors instead of returning a partial parse as complete
A line longer than `bufio.Scanner`'s 64KB buffer ends the scan with `ErrTooLong`, which only `Err()` reports. Parsers now return what they found along with the error (recorded as an unknown for the file), or log it where there is no error to return.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: allow lines up to 1MB in line-oriented package metadata parsers
Adds `internal.NewLineScanner`, a `bufio.Scanner` capped at 1MB per line rather than the 64KB default, the same cap the alpm parser already uses. Long single-line fields (descriptions, license text, dependency lists) now parse instead of ending the scan early.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: use io.ReadFull where a short read was silently accepted
- dotnet bundle 7-bit int decoding could read a zero-byte result as a zero byte
- wordpress plugin header parsing included the unread tail of its buffer
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: finish moving line scanners to the 1MB cap and checking scanner errors
- java manifest, pom.properties, Gemfile.lock, requirements.txt, homebrew formula, alpm and the purl/cpe decoders now use `internal.NewLineScanner`
- snap kernel changelog reports a scan error instead of silently dropping the base kernel package
- the attestation output reader keeps draining the pipe after a scan error so cosign can't block on a full pipe
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: stop logging the full setup.py line once per dependency match
The debug log ran once per regex match and included the whole line, so log volume grew with the square of line length. With 1MB lines that is hundreds of GB of debug output from a single crafted file.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(gentoo): don't dereference a nil location when a LICENSE scan fails
`extractLicenses` accepts a nil location, but the scan error log added earlier on this branch dereferenced it.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(gentoo): bound license group expansion on hostile license_groups
- dedup within a group uses a set instead of `slices.Contains`, which was quadratic in tokens per line and got much worse under the 1MB line cap
- each group is expanded once and memoized, so a chain of groups that reference each other repeatedly no longer expands exponentially
- repeated references to the same group within one line are skipped
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
- log lines shown under the TUI were glued onto the last UI row, and the last row could get erased (and overwritten by the report) when the TUI exits. Fixed in `bubbly` v0.2.2
- log colors were lost because the logger latched its TTY check on the first entry, which went to the TUI footer buffer. `go-logger` v0.2.0 checks per entry and supports `TerminalWriter`, which the footer writer now implements (`NO_COLOR` is still honored)
- log lines under the TUI get a dim `Logs (non-fatal, see -v for more):` title with a branch pointing at the first entry, so it's clear what the trailing lines are
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Docker Hub rebuilt and re-pushed `busybox:1.38.0`, which changed the per-platform config digests and broke `TestMultiPlatformOCIImageSelection` on uncached runs.
- pin the fixture to the busybox 1.38.0 index digest so future re-pushes can't shift it
- update the expected per-platform digests to match
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* build(deps): bump go-make to v0.8.1
Picks up golangci-lint 2.13.2 (built with go1.27) in place of 2.11.4 (go1.26).
The old build panics outright on a go1.27 GOROOT with "file requires newer Go
version go1.27", and an older mise-shim build fails the quieter way: it loses
type info and reports zero findings with exit 0, so a ruleguard run comes back
green without having evaluated anything.
Also folds in what the newer linter flags:
- `reflect.Ptr` -> `reflect.Pointer` in three files, applied by --fix
- `tw.CellFormatting{MergeMode: ...}` -> `tw.CellMerging{Mode: ...}` in the
cataloger info table, for a staticcheck SA1019 deprecation
Note the deprecation text names `CellConfig.CellMerging.Mode`, but the field is
`CellConfig.Merging`; `CellMerging` is the type. That command path has no test
coverage, so the swap was checked by rendering the same hierarchical-merge table
both ways and diffing: byte identical.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(lint): tune goconst for data tables
goconst got stricter in golangci-lint 2.13 and now counts strings in map
literals and struct fields, which lights up every lookup table in the repo. It
was already enabled, so this is main going red on a linter bump rather than
anything new in the tree: 101 findings, and the default threshold of 3 means a
vendor name appearing in three rows of a table reads as a magic constant.
Raising `min-occurrences` to 5 takes that to 14. The rest split by whether the
repeated string is data or an actual constant:
- `cpegenerate` is excluded by path. It maps artifact names to vendors,
products, and maven group ids, and repeats strings 35, 83, even 293 times, so
no threshold reaches it. Hoisting those would trade a readable table for
indirection.
- the two SBOM fixture builders in `format/internal/testutil` get a
function-level nolint each. The repeated names and versions are what the
golden files assert against.
- `mysql-cluster` and `php` were genuine repeated literals and are now
constants.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(test): extract measureAlloc into a shared helper
The allocation-budget tests added during the unbounded allocation work grew a
copy of the same helper in every package they touched: four identical
`measureAlloc` definitions plus two more spelled out inline against
`runtime.MemStats`.
Now `testutils.MeasureAlloc`, at the repo root so `internal/spillbuf` can reach
it too. The doc comment carries the two things that were only recorded in some
of the copies: TotalAlloc is process-wide so callers must not use t.Parallel,
and a budget assertion only means something if the fixture would really blow
past it unguarded, so measure the unguarded path as well.
No behavior change, the body was identical in all six.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(lint): add ruleguard rules for overflow and swallowed EOF
Two rules out of the recent unbounded allocation work (#5187, #5195, #5267,
#5291, #5292, #5293, #5294).
`noOverflowingBoundsCheck` flags `offset + size > limit` (and <, <=, >=) where
an operand is unsigned, neither addend is constant, and the operands are at
least 32 bits. Unsigned addition wraps, so an offset out of a file header can
carry the sum past the wrap point and pass a check it should have failed. The
fix is to subtract against the limit instead. Constant addends are excluded
because `pos+4 <= uint32(len(buf))` is the normal way to step a cursor, and the
32-bit floor drops nibbles widened for a comparison. Would have caught four
sites across #5195 and #5291.
`noSwallowedEOF` flags `return nil` / `return nil, nil` on an io.EOF check. The
caller gets a zero value it cannot tell from real data, and anything decoding a
fixed-size struct off it parses the padding as content. That is the shape behind
#5186.
Both decide from the expression alone, and writing the fix deletes the pattern,
so neither needs a suppression anywhere in the tree. That was the bar: rules for
the allocation and decompression classes were tried and dropped, because the
bound they look for lives in a preceding statement (or, for a decompressor
constructor, always downstream), so no correct implementation can silence them.
That would have meant ~15 permanent nolints on code that is already right. The
rules here that do work don't judge whether a bound exists, they route to a
wrapper that takes one: noDirectTempFiles, noDirectELFOpen.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: pin allocation budgets on the alpm, deb, and changelog bounds
The bounds added for #5292, #5293 and #5294 are asserted by the error they
return. That keeps passing if the bound is removed and something else rejects
the input later, which is the failure these were written for. Each of these
measures bytes instead, and proves the fixture is really a bomb first so a
fixture that quietly stops being one fails rather than passing.
- alpm mtree: 16MB of newlines, well inside the 64MB byte cap, costs 16.5GB
uncapped against 238MB with maxMtreeLines. That 16.5GB is the measurement the
line cap was added for and it was recorded only in the commit message. Note
what the cap actually buys: 300k entries still cost a few hundred MB, per
concurrent cataloger.
- deb members: 1.28GB against 392KB. Both arms read with io.ReadAll on purpose,
since the stream itself never holds much whatever the cap says; what the cap
protects is a consumer that buffers, and the real one is a tar reader building
entries.
- snap changelog: 1.28GB against 175KB. This one has no explicit cap at all. It
is held by bufio.Scanner refusing a token past MaxScanTokenSize, which is
incidental, so switching to io.ReadAll or a bufio.Reader to pick up long lines
would remove it silently.
Nothing added for the kernel module reader: its copy is already
io.Copy(tempFile, io.LimitReader(rc, max+1)), so peak disk is bounded by
construction and heap is streaming. A heap budget would assert the wrong
resource, and the three existing subtests cover the cap, at-cap fidelity, and
spill cleanup.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: pin allocation budgets on the ai and alpm byte caps
Follows the same pattern as the alpm line cap: measure bytes, and prove the
fixture is really a bomb first so one that quietly stops being a bomb fails
rather than passes.
- safetensors: 252MB against 2.6KB. The existing over-cap case deliberately
omits the body, since the guard rejects on the declared length before reading
anything. That is the right test for the guard but it would keep passing if
the check moved after the read, so this one delivers the bytes.
- gguf: 268MB against 168MB. copyHeader io.Copy's whatever it is handed, so
maxHeaderSize holds only because every caller wraps the reader first. This
pins that the limit is where the bound lives, so dropping the wrapper at a
call site fails here.
- alpm byte cap: 740MB against 132MB, the counterpart to the line cap budget.
Worth knowing for anyone writing the next one of these: a single enormous line
does not exercise the alpm byte cap. go-mtree runs its own bufio.Scanner and
refuses a token past 64KB, so that fixture fails as "token too long" before the
byte cap is consulted. The fixture here uses ordinary 1KB lines, enough of them
to pass 64MB while staying under the 300k line cap, so the byte cap is actually
the thing under test.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: assert bytes retained instead of allocated for gguf copyHeader
the bounded arm writes into a bytes.Buffer, whose doubling growth costs a
multiple of what it holds and lands on a different power of two depending on
the platform. it measured 168MB locally and 320MB on CI, past the budget.
buf.Len() states the same property deterministically.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* Update steroscope to include OCI platform metadata
Signed-off-by: Jason Paulos <jasonpaulos@users.noreply.github.com>
* chore(deps): point stereoscope at anchore/main
drops the fork `replace` directive and moves to a pseudo-version off stereoscope
main (`a22e5ae`, anchore/stereoscope#657). no tag carries it yet, so this should
move to a real version once one is cut.
`readImageMetadata` now fills os/architecture/variant for every provider, not
just oci. the docker tarball provider set neither before, so image metadata and
the image purl change for docker-archive input too:
- `"architecture": ""` -> `"amd64"`
- `"os": ""` -> `"linux"`
- `?arch=` -> `?arch=amd64`
the empty `arch=` qualifier was never right, but it is user-visible output.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: cover single-platform oci layouts
the pr only added assertions to the multiplatform fixture, which already
reported the right platform. the behavior that actually changed is single-image
layouts: they used to report empty os/arch and silently ignore `--platform`, and
now report what the image config says and fail with a platform mismatch.
also drops the non-linux skip on the default-platform test. the oci providers
default to `linux/<GOARCH>` regardless of host os, so it holds on macos too.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Jason Paulos <jasonpaulos@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* feat(convert): passthrough-exact-format copies matching syft-json input unchanged
When a syft-json input SBOM already exactly matches the requested output
format and schema version, `syft convert` can write the input bytes out
verbatim instead of decoding and re-encoding them. This avoids the lossy
round trip (and the associated cost) for callers that normalise SBOMs into
a target format without knowing up front whether the source already matches.
The behaviour is opt-in and configuration-only: it is enabled with
convert.passthrough-exact-format in the config file or the
SYFT_CONVERT_PASSTHROUGH_EXACT_FORMAT environment variable, and there is
deliberately no command-line flag, since it changes how every invocation in
a pipeline behaves rather than varying call by call. It defaults to off, so
existing behaviour is unchanged.
Pass-through applies only to syft-json input, and only when the document's
schema version is identical to the one this build of syft produces; any
other input is always converted. Outputs that do not match are still
converted, so a mixed set of `-o` values behaves as expected, and every
output is resolved and validated before anything is written. Pass-through
output is a byte-for-byte copy, so format options such as `format.pretty`
and `format.json.legacy` are not applied to it.
When the option is enabled a warning is logged stating so and noting the
syft-json-only scope, and a second warning is logged whenever an input is
found to match and the shortcut is actually taken.
The option lives under a `convert:` configuration section (options.Convert),
mirroring how attest scopes its own options. The description documents the
caveats above and notes that file-path input written with `-o <format>=<path>`
keeps memory minimal, whereas piped input or STDOUT output must hold the
whole document in memory.
Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* perf(convert): stream pass-through output instead of buffering the input
Identify, copy, and decode the input through a seekable reader rather than
loading the whole document into memory first. File inputs are read in place
and pass-through output is an io.Copy from the file. STDIN is used in place
when it is a redirected regular file. A pipe or terminal cannot seek, so its
contents are drained into an internal/spillbuf buffer: the first 1MB stays in
memory and the rest goes to a temp file under the user's TMPDIR (created
lazily, only when something actually spills), giving a seekable view without
ever holding the document in memory. The temp dir is rooted per invocation
via internal/tmpdir, honouring one already attached to the context, and is
removed when the command returns.
Writing pass-through output over the input file itself is now a no-op rather
than truncating the file before it is copied.
Measured on a 1GB syft-json document (Apple M1 Max), file in, file out:
origin/main full re-encode 86.3s wall 99s CPU 12.0GB max RSS
in-memory pass-through 7.4s wall 6.7s CPU 4.2GB max RSS
streaming pass-through 7.0s wall 6.9s CPU 3.1GB max RSS
and the same 1GB document piped over STDIN with pass-through enabled:
io.ReadAll into memory 1.3s wall 0.8s CPU 2.7GB max RSS
spill buffer 1.8s wall 1.0s CPU 66MB max RSS
The remaining cost of the streaming pass-through is entirely format.Identify,
which for syft-json decodes the whole document through a json.Decoder that
buffers the full value: measured alone it takes 7.1s and 3.1GB on the same
input. Reducing that further belongs to the format package.
Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* perf(syftjson): identify documents from their tail instead of a full parse
syft always writes the schema block as the last key of a syft-json document,
so the format and version can be read from the final bytes of the file. The
syft-json decoder's Identify now reads at most 2MB from the end of a seekable
input, finds the last "schema" key there, and verifies it sits directly within
the top-level object by scanning the remaining bytes for the single closing
brace of the document. Anything the tail cannot vouch for (non-seekable input,
a reordered document, a nested key) falls back to the existing full parse, so
behaviour is unchanged for documents the fast path declines.
The verification is position-independent: it also accepts documents whose
keys have been reordered so that other small keys follow schema. It rejects
nested schema objects, array-wrapped documents, truncated input, trailing
content, and schema text inside strings.
Supporting changes:
- the offset-aware reader the decoder collection hands to decoders now
supports io.SeekEnd, which the tail read requires
- model.Document documents that Schema must remain its last field, and a
test guards that the encoder writes it last in both compact and pretty
output
- the decoder list documents that syft-json must stay first
Since Decode calls Identify before parsing, this also removes one of the two
full parses every syft-json conversion used to perform.
Measured on a 1GB syft-json document (Apple M1 Max), file in, file out:
pass-through, previous commit 7.0s wall 6.9s CPU 3.1GB max RSS
pass-through, this commit 0.85s wall 0.05s CPU 66MB max RSS
full re-encode, previous commit 85.3s wall 99s CPU 11.5GB max RSS
full re-encode, this commit 75.2s wall 91s CPU 9.9GB max RSS
Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(syftjson): only identify from the tail when the document starts where the reader does
The tail scan vouched for the last 2MB, but nothing checked those bytes belonged to the
document the reader was pointed at. So `Identify` started claiming syft-json for files it
cannot decode: a document with a byte order mark or a stray log line in front of it, or a
syft document sitting after something else in the same stream, where `format.Decode` went
on to return an empty SBOM with a nil error.
Two O(1) checks:
- the first non-space byte at the reader's position has to be `{`
- `Decode` errors when the parsed document has no `schema.version`, instead of handing back
the empty document it decoded into
Streams holding several concatenated documents are deliberately not detected. Proving there
is exactly one top-level value means reading all of it, which is the cost the tail path
exists to avoid, so `Identify` can still name a later document's version on such a file.
`Decode` reports what it actually parsed.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(convert): recognize the input file by identity rather than by path
`convert - -o syft-json=sbom.json < sbom.json` truncated the file and exited 0. The
passthrough guard compared paths, and a shell redirect does not have one, so it was blind to
the one input mode where the truncation was silent.
`openConvertInput` now returns the input's `os.FileInfo`, which the STDIN branch already had
from its `Stat`, and `isSameFile` compares that instead.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
the base image is digest-pinned to a bullseye snapshot whose InRelease has
expired, so `apt-get update` now exits 100 on any cold fixture build. rather
than disable Check-Valid-Until (which just delays the next break), skip apt
entirely:
- `less` was never used and `unzip` already ships in the base image
- rezip with `jar cfM` from the base JDK instead of installing `zip`
the `-M` matters here: this fixture intentionally has no META-INF/MANIFEST.MF,
and plain `jar cf` would generate one.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`archive.ubuntu.com/ubuntu/pool/main` only keeps the current version of a
package, so the pinned `zlib1g 1:1.3.dfsg-3.1ubuntu2.1` URL started 404ing.
`curl` without `-f` wrote the error page into `zlib1g.deb` and exited 0, so
the fixture image built clean and just contained a non-deb, failing
`TestDpkgArchiveCataloger` only when the fixture cache was cold.
Now points at the launchpad `+files` URL, which is immutable. Also added `-f`
to the other fixture Dockerfiles that curl a pinned artifact so the next bit
of URL rot fails the build instead of quietly poisoning a fixture.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* feat(golang): add extended-stdlib scope and include patterns for symbol capture
`golang.capture-symbols` decides how much symbol data lands in the SBOM for grype's reachability analysis. It's `none`, `stdlib`, or `all` today, and the useful middle is missing: `stdlib` stops at the standard library, `all` multiplies SBOM size.
A new `extended-stdlib` configurable covers stdlib plus everything under `golang.org/x/`:
```yaml
golang:
capture-symbols: extended-stdlib
```
Also, a new `capture-symbols-include` configurable for modules that are noisy in your binaries but not everyone's. It's unioned with whatever the scope selects, so it only ever widens:
```yaml
golang:
capture-symbols: extended-stdlib
capture-symbols-include:
- github.com/klauspost/**
```
Patterns are standard doublestar globs, which matters because module paths carry `/v2`-style suffixes:
```yaml
golang:
capture-symbols-include:
- github.com/klauspost/* # compress, but not compress/v2
- github.com/klauspost/** # both
- k8s.io/client-go # exact match only
```
Ordering is `none` < `stdlib` < `extended-stdlib` < `all`. The existing three values
and the `none` default are unchanged, and the include list is inert under `none`.
Presets compile into glob lists internally, so a single matcher answers "does this
module get symbols" instead of a preset branch sitting next to a separate glob branch.
An unrecognized `capture-symbols` value still falls back to `none`, but warns now
instead of doing it silently. A malformed include pattern warns and gets skipped.
One thing worth a look beyond the feature: the `Symbols` field description in the JSON
schema was wrong after this (it claimed only `all` and `stdlib` populate anything), and
that description lives in the already-published `16.1.10`. Rather than bump a version for
a sentence, `16.1.10` is amended in place and `schema/json/README.md` grows an explicit
exception for description-only changes: descriptions only, no shape change of any kind,
`$id` unchanged. Anything else still needs a bump. Happy to split that into its own PR if
you'd rather review the policy separately.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* refactor(golang): rename capture-symbols-include to capture-symbols-modules
The key's entries are go module paths, and `-include` sitting next to `capture-symbols` reads as plausibly taking symbol or package names instead. Those spellings parse and match nothing, which is quieter than the confusion `-include` was picked to avoid, so the name now says what the list holds.
`golang.CatalogerConfig.CaptureSymbolsModules` and `WithCaptureSymbolsModules` rename with it. Nothing behavioral changes; the key is new in this PR so there is no compatibility surface.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* feat(golang): match capture-symbols-modules across major version suffixes
`github.com/anchore/*` covered `github.com/anchore/syft` and silently stopped covering it the day it became `github.com/anchore/syft/v2`. The config keeps parsing, nothing warns, and symbols quietly go missing from the SBOM. Exact paths had the same hole: `github.com/klauspost/compress` did not cover `compress/v2` either, so no spelling short of `**` survived a major bump.
A major version suffix is part of a module's path but not part of its identity, so patterns are now matched against the module path both with and without it, using `module.SplitPathVersion` from `golang.org/x/mod` (already a direct dep, already used in this package for `PseudoVersion`).
```yaml
golang:
capture-symbols-modules:
- github.com/klauspost/* # compress and compress/v2
- github.com/klauspost/compress # same module at every major version
- github.com/klauspost/compress/v2 # v2 alone
```
Only a trailing suffix is a version, which is Go's own rule. In `github.com/anchore/syft/v2/thing` the `v2` is an ordinary path element naming a major subdirectory a nested module lives in, so it stays literal and `github.com/anchore/**/thing` is how you reach it. `/v0` and `/v1` are not valid suffixes and are left alone.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Closes#4587
Adds a single info-level log line at the end of every `syft scan` run that reports the total scan wall-clock time, e.g.:
[0003] INFO scan completed in 2.017s
Per maintainer guidance on the issue, the line is logged at INFO so it surfaces with `-v` (alongside the existing per-cataloger timing) and stays out of the default TUI / stdout. Useful for users who leave long scans running and want the overall time at a glance, without having to wrap the invocation in `time`.
Signed-off-by: Chris (ChrisJr404) <11917633+ChrisJr404@users.noreply.github.com>
Co-authored-by: Chris (ChrisJr404) <11917633+ChrisJr404@users.noreply.github.com>
* chore: centralize temp files and prefer streaming IO
Catalogers that create temp files ad-hoc can easily forget cleanup,
leaking files on disk. Similarly, io.ReadAll is convenient but risks
OOM on large or malicious inputs.
Introduce internal/tmpdir to manage all cataloger temp storage under
a single root directory with automatic cleanup. Prefer streaming
parsers (bufio.Scanner, json/yaml.NewDecoder, io.LimitReader) over
buffering entire inputs into memory. Add ruleguard rules to enforce
both practices going forward.
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* chore: go back to old release parsing
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* simplify to limit reader in version check
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* chore: regex change postponed
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* simplify supplement release to limitreader
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
---------
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* chore(deps): update tools to latest versions
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* chore(lint): fix errors in new golangci-lint
Two fixes:
First, replace sb.WriteString(fmt.Sprintf(...)) with fmt.Fprintf(&sb, ...)
Second, suppress errors where we read from the local file system at a
user provided path. This is a CLI tool, and reads from user provided
paths on the local file system by design.
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
Co-authored-by: spiffcs <32073428+spiffcs@users.noreply.github.com>
Co-authored-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* migrate fixtures to testdata
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: correct broken symlinks after testdata migration
The migration from test-fixtures to testdata broke several symlinks:
- elf-test-fixtures symlinks pointed to old test-fixtures paths
- elf-test-fixtures needed to be renamed to elf-testdata
- image-pkg-coverage symlink pointed to test-fixtures instead of testdata
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: handle missing classifiers/bin directory in Makefile
The clean-fingerprint target was failing when classifiers/bin doesn't
exist (e.g., on fresh clone without downloaded binaries).
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: add gitignore negation for jar/zip fixtures in test/cli
The jar and zip files in test/cli/testdata/image-unknowns were being
gitignored by the root .gitignore patterns. This caused them to be
untracked and not included when building docker images in CI, resulting
in Test_Unknowns failures since the test expects errors from corrupt
archive files that weren't present.
Add a .gitignore in test/cli/testdata to negate the exclusions for
these specific test fixture files.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* switch fixture cache to v2
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: update expected versions for rebuilt fixtures
Update test expectations for packages that have been updated in
upstream repositories when docker images are rebuilt:
- glibc: 2.42-r4 → 2.43-r1 (wolfi)
- php: 8.2.29 → 8.2.30 (ubuntu/apache)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* upgrade go
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: add go-shlex dependency for testdata manager tool
The manager tool in syft/pkg/cataloger/binary/testdata/ imports
go-shlex, but since it's in a testdata directory, Go doesn't track
its dependencies. This caused CI failures when go.mod didn't
explicitly list the dependency.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* refactor: move binary classifier manager to internal/
Move the manager tool from testdata/manager to internal/manager so
that Go properly tracks its dependencies. Code in testdata directories
is ignored by Go for dependency tracking, which caused CI failures
when go.mod didn't explicitly list transitive dependencies.
This is a cleaner solution than manually adding dependencies to go.mod
for code that happens to live in testdata.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: add gitignore negations for test fixtures blocked by root patterns
Multiple test fixtures were being blocked by root-level gitignore patterns
like bin/, *.jar, *.tar, and *.exe. This adds targeted .gitignore files with
negation patterns to allow these specific test fixtures to be tracked:
- syft/linux/testdata/os/busybox/bin/busybox (blocked by bin/)
- syft/pkg/cataloger/java/testdata/corrupt/example.{jar,tar} (blocked by *.jar, *.tar)
- syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/**/bin/go (blocked by bin/)
- syft/pkg/cataloger/bitnami/testdata/no-rel/.../bin/redis-server (blocked by bin/)
Also updates the bitnami test expectation to include the newly required
.gitignore files in the test fixture.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: update glibc version expectation (2.43-r1 -> 2.43-r2)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* add capability drift check as unit step
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* dont clear test observations before drift detection
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* bump stereoscope commit to main
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fixed#4316 go mod with ver purl
Signed-off-by: Rez Moss <hi@rezmoss.com>
* go mod purl fixed, added func to handle go.mod
Signed-off-by: Rez Moss <hi@rezmoss.com>
* fix: use module name in PURL string everywhere
Signed-off-by: Keith Zantow <kzantow@gmail.com>
---------
Signed-off-by: Rez Moss <hi@rezmoss.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Co-authored-by: Keith Zantow <kzantow@gmail.com>
* feat: Add license enrichment from pypi to python packages
* Implement license caching and improve test coverage
---------
Signed-off-by: Tim Olshansky <456103+timols@users.noreply.github.com>
* ref: close source in test and examples
Signed-off-by: Kudryavcev Nikolay <kydry.nikolau@gmail.com>
* ref: pretty file/directory source resolver (make them more similar)
Signed-off-by: Kudryavcev Nikolay <kydry.nikolau@gmail.com>
* ref: move absoluteSymlinkFreePathToParent to file resolver
Signed-off-by: Kudryavcev Nikolay <kydry.nikolau@gmail.com>
* revert breaking change
Signed-off-by: Kudryavcev Nikolay <kydry.nikolau@gmail.com>
---------
Signed-off-by: Kudryavcev Nikolay <kydry.nikolau@gmail.com>
* feat: update integration test with correct package for httpd
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
* chore: update integration and cli tests with new upstream expectations
- php interpreter 8.3.21 => 8.3.22
- runCycloneDXInDocker update for local arm64 qemu emulation CycloneDX
- getSyftBinaryLocationByOS update to detect arm64 v8.0 artifact path
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
* chore: add snalshot to test command for fixture builds
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
* chore: update cdx in docker for all GOOS
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
---------
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
* feat: expose rpm signature information
This helps with more confident identification of an rpm.
In theory, two rpms can be built that have the same purl string, and
otherwise look identical in syft's output, but the PGP information
would distinguish them as signed either by different keys, or signed at
different times.
In practice, this usually makes no difference since rpms tend to have
unique name/version/release strings. This just gives increased
confidence about the identity of the rpm found in the db.
Signed-off-by: Ralph Bean <rbean@redhat.com>
* chore: generate json schema
Signed-off-by: Ralph Bean <rbean@redhat.com>
* re-generate json schema
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* rename to a more generic signature field
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* rename rpm.pgp to rpm.signatures
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* split out signature fields
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* bump json schema
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* include RPM archives
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* update json schema
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* dont fail on unknown signature type
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Ralph Bean <rbean@redhat.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
MetalandAlex Goodman
Jason PaulosandAlex Goodman
JuanandAlex Goodman
ChrisJr404andChris
Christopher Angelo PhillipsandAlex Goodman
gabandAlex Goodman
Rez MossandAlex Goodman
Archy
Keith Zantow
Will Murphy
Tim Olshansky
Kudryavcev Nikolay
n-bes
Simeon StoykovandChristopher Phillips
Ralph BeanandAlex Goodman