- apk origins and alpm pkgbases are build recipes, not published source packages, and their names are usually also real binary packages, so a `source-distribution` ref there would point at a binary package. They keep relying on the `upstream` PURL qualifier.
- document why this is an external reference rather than `pedigree.ancestors`, with links to the CycloneDX spec discussion (CycloneDX/specification#612)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
- the source PURL is built from package metadata instead of editing the binary PURL: only `distro` (plus `epoch` and `rpmmod` for rpm) is carried over, subpath is dropped, and the ref is skipped when the PURL type does not match the metadata ecosystem
- one emission rule for all ecosystems: only when the source name or version differs from the binary, so self-sourced rpm and alpm packages no longer get a redundant ref
- source RPM names must end in `.src.rpm` or `.nosrc.rpm`, and `nosrc` maps to `arch=nosrc`
- `source-distribution` refs are dropped when encoding below spec 1.6, since cyclonedx-go does not downgrade them (CycloneDX/cyclonedx-go#291)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
The traefik-binary classifier regex required a NUL byte or replacement
character immediately before the version string. On some architectures
(e.g. s390x), the real version string in the binary has no such prefix,
so the classifier instead matched an earlier, unrelated NUL-prefixed
numeric string elsewhere in the binary (e.g. '2.7.0' from an unrelated
printf-style format constant), producing an incorrect version.
This adds an additional pattern, tried first via MatchAny, that matches
version strings terminated by two NUL bytes without requiring a NUL
prefix. The original pattern remains as a fallback for older traefik
binaries (e.g. v1.x) that use single-NUL termination.
Fixes#4980
Signed-off-by: Mayank Sekhar <mayankshekharsingh@gmail.com>
* fix: scan a Java resource adapter (.rar) given as a file source
The file source picks an extractor for the scan target from its file name
alone. A Java resource adapter archive (JCA) uses the .rar extension but is
a zip file, so it was handed to the RAR extractor. With rardecode v2.2.0 the
RAR extractor does not return on non-RAR input (an infinite loop in its
signature search), so `syft scan --from file adapter.rar` never finished.
The same bytes named .jar scan in under a second.
When the name says RAR but the content is a zip archive, leave the file
unextracted, the same way .jar, .war and .ear scan targets are handled. The
java cataloger already reads .rar files (anchore/syft#4136) and now finds the
resource adapter and its nested jars. Real RAR archives are still extracted.
Signed-off-by: Ismael Donmez <ismael.donmez@chainguard.dev>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* chore(deps): bump github.com/nwaples/rardecode/v2 from 2.2.0 to 2.4.1
rardecode v2.2.0 loops forever in its RAR signature search when the input
does not start with a RAR signature (fixed upstream in nwaples/rardecode
f501ac8, "fix infinite loop in findSig", released in v2.2.2). syft reaches
that search for any file source whose name contains ".rar", so a scan of a
self-extracting RAR, or of any other file with such a name, never finished.
With v2.4.1 a self-extracting RAR is extracted, and a file that is not a RAR
archive fails fast with "rardecode: RAR signature not found", the same way a
file named .zip that is not a zip archive fails today.
github.com/mholt/archives v0.1.5 (the latest release) still requires v2.2.0,
so the newer version is selected here.
Signed-off-by: Ismael Donmez <ismael.donmez@chainguard.dev>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix: clarify the .rar zip log line and cover self-extracting RAR sources
- the debug line no longer calls every `.rar`-named zip a resource adapter
- a self-extracting RAR (stub before the signature) given as a file source is
still extracted
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Ismael Donmez <ismael.donmez@chainguard.dev>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
SPDX license list 3.29.0 rewrote every opensource.org seeAlso URL from
opensource.org/licenses/<ID> to opensource.org/license/<ID> (and e.g.
opensource.org/license/mit/ to opensource.org/license/MIT). Package metadata
in the wild (Maven POM <license><url>, npm, etc.) still uses the older forms,
so regenerating the list would stop LicenseByURL from resolving them.
Fall back to a normalized index of the opensource.org entries that ignores the
www. prefix, /licenses/ vs /license/, case, and a trailing slash. The index is
built from the generated urlToLicense map, so it works with both the current
and the updated license list without changes to the generator. URL forms that
would normalize to different licenses are left out rather than guessed.
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* ci: drop gen-5 instance families from the test runner
m5a runners run the unit suite ~2.3x slower than m8id (15m vs 6.5m for `go test`), and when spot capacity for newer families dries up runs-on falls back to them. Keeping gen-6 through gen-8 leaves a wide spot pool without the slow tail.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: shrink decompression bomb fixtures
The bomb and heap-bound tests pushed 100MB-16GB through gzip and parsers, which under `-race` made them the slowest tests in the unit suite. Each now uses the smallest fixture that still separates the bounded path from the unbounded one.
- arch mtree caps are measured at a small cap and scaled to the shipped constants, so raising a cap still fails the budget
- snap and debian bombs expand to 32MB instead of 512MB (4x their 8MB budgets)
- golang ELF bombs still declare 256MB but deliver 64MB
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: add windows unit test job and capture dir resolver path bug
- add a `Unit tests (Windows)` job running a new `make unit:windows` task, scoped to the packages that own host path handling (tests needing linux docker fixtures are skipped)
- `Unit tests` is now a gate job over the Linux and Windows unit jobs
- add a resolver test asserting that locations from `AllLocations` resolve again via `FilesByPath` and use posix paths. This is expected to fail on windows today, which is why file digests come back empty for `dir:` scans (#5325)
- fix windows-only test files that no longer compiled
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore: drop verbose output from integration tests
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: windows path handling outside of the file lookup bug
- chroot paths are always posix (`ToChrootPath` no longer returns backslash paths on windows), same for the directory source name derived from `base`
- `windows.FromPosix` no longer panics on input that isn't volume-encoded (e.g. relative or already native paths)
- skip tests on windows that rely on symlink fixtures, shell script fixtures, or emulate windows paths on a posix host
- disable `core.autocrlf` on the windows CI job so fixture digests match linux
This leaves only the `FilesByPath` lookup failures from #5325.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: skip posix-only fileresolver tests on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: split contained paths on posix separators on windows
- `allContainedPaths` cleaned with the native separator but split on `/`, so a root reached through a symlink never had its ancestors indexed on windows
- index assertions in tests now key on the posix path the index actually stores
- skip symlink fixture tests and a posix-only test helper on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore: revert unrelated go.sum churn and allContainedPaths change
- restore `go.sum` and `.make/go.sum` (accidentally populated by a local `-mod=mod` invocation)
- revert the `allContainedPaths` separator change: production passes native windows paths there, and splitting them on `/` produces drive-relative ancestors, so skip the posix-only test on windows instead
- annotate the remaining `filepath` usages that operate on native paths
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: search the file tree with posix paths on windows (#5325)
The directory and file resolvers key their file tree on posix paths (`/c/some/path`), but `FilesByPath`, `HasPath` and root-anchored `FilesByGlob` searched it with the native path from the chroot context (`C:\some\path`). On windows every lookup silently missed, so anything that resolves files by path found nothing. This is why file digests were empty for `dir:` scans.
Requests are now converted to posix before searching the tree.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(spdx): write a name for the root package when the source has none
A source read from another SBOM (a purl list, a CycloneDX document
without metadata.component, an SPDX document without a DESCRIBES
relationship) can have no name. The SPDX encoder then wrote the document
root package with an empty PackageName, which SPDX requires. In
tag-value output the PackageName line is omitted, so the root package's
fields land in the previous section and syft cannot read its own output
back ("received unknown tag SPDXID in CreationInfo section").
Fall back to the document name for the root package, and make the
document name fall back to "unknown" for every source type rather than
only for unrecognized ones.
Signed-off-by: Tony <tjagodka@gmail.com>
* fix(spdx): drop syft's root package on decode for every SPDX version
- the root package name now comes from one fallback chain (source name, image repo name, source ID, document name), so the root purl no longer gets an empty name for a source decoded from CycloneDX
- decoding strips the root syft writes for an unknown source, so `syft convert` round trips stop adding an `unknown` package each pass
- for SPDX 2.1 and 2.2, which have no `primaryPackagePurpose`, the purpose is taken from the source type in syft's `DocumentRoot-<type>-` ID
**behavior change:** decoding a syft-written SPDX 2.1/2.2 document no longer returns the root as a package or its `CONTAINS` relationships, matching 2.3.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Tony <tjagodka@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(python): link uv.lock dependents to the locked version they name
A forked uv.lock can lock several versions of one package, and uv records the version on each dependency entry in that case. The parser dropped that field, so every pandas version depended on every numpy version. Carry the locked version through provides and requires as name@version (bare name stays for unique locks) and keep marker text unchanged.
Fixes#5340
Signed-off-by: Dev M <devtechedge@gmail.com>
* fix(python): pair uv.lock versions without widening public metadata
- the locked version on a uv dependency entry is only needed to pair relationships, and the relationships already carry it, so it stays on the internal toml struct instead of `pkg.PythonUvLockDependencyEntry`. The previous approach hid it from JSON but still hashed it into package IDs.
- dependency specs are now built from the raw lock entries at parse time rather than read back off package metadata
- optional-dependencies get the same version pairing, so an extra that names one locked version no longer links to all of them
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Dev M <devtechedge@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
Motivation:
syft's PHP interpreter cataloger reports each bundled PHP C extension
(openssl, ldap, sqlite3, etc.) as its own package, using the PHP
interpreter's own version number as the package version (since these
extensions are compiled into a specific PHP release and have no
independent version of their own). The classifier already attaches an
explicit CPE like cpe:2.3:a:php-openssl:php-openssl:8.5.7:... to make
clear this refers to the bundled PHP extension, not upstream OpenSSL.
However, that CPE was marked with Source: cpe.GeneratedSource. Syft's
package-finalization pipeline (internal/task/package_task_factory.go)
only skips its generic, name-based CPE generation when a package
already has an "authoritative" CPE (any Source other than
GeneratedSource) - so it also generated and appended a second CPE,
cpe:2.3:a:openssl:openssl:8.5.7:..., derived from the bare package
name. That second CPE is indistinguishable from a real standalone
OpenSSL/LDAP/SQLite installation and can cause vulnerability scanners
that consume this SBOM to falsely flag PHP hosts as running vulnerable
upstream libraries at version "8.5.7" (a PHP version, not a real
OpenSSL/LDAP/SQLite release). See anchore/syft#5014 for a full repro.
Approach:
Mark the classifier's explicit CPE with Source: cpe.DeclaredSource
instead of cpe.GeneratedSource, so it is treated as authoritative and
the extra name-derived CPE is no longer generated. This mirrors the
existing pattern in
syft/pkg/cataloger/java/parse_jvm_release.go's newJvmCpe, which uses
the same DeclaredSource + comment convention to mark a
syft-synthesized (not literally file-declared) CPE as trustworthy and
suppress further generation. It also brings this classifier in line
with its sibling php-cli/php-fpm/php-apache classifiers in the same
file, which already mark their CPEs as authoritative (via
NVDDictionaryLookupSource, not applicable here since php-<ext> is not
a real NVD dictionary entry).
This change only suppresses the extra generated CPE; it does not
change the package name, type, version, or PURL, and the existing
php-<ext>:php-<ext> CPE is unaffected.
Validation:
go build ./...
go test ./syft/pkg/cataloger/php/... ./internal/task/... ./syft/pkg/cataloger/internal/cpegenerate/...
All pass except the two pre-existing docker-image-fixture subtests of
Test_InterpreterCataloger, which fail in this sandbox only because the
`docker` binary isn't available (exec: "docker": executable file not
found in $PATH); confirmed via `git stash` that these two subtests
fail identically on main without this change, i.e. this is a sandbox
limitation and not a regression. Added a new unit test,
Test_getClassifier_declaredCPESource, that directly asserts the
classifier's CPE now carries Source: cpe.DeclaredSource.
Fixes#5014
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Co-authored-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
* fix: report scanner errors instead of returning a partial parse as complete
A line longer than `bufio.Scanner`'s 64KB buffer ends the scan with `ErrTooLong`, which only `Err()` reports. Parsers now return what they found along with the error (recorded as an unknown for the file), or log it where there is no error to return.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: allow lines up to 1MB in line-oriented package metadata parsers
Adds `internal.NewLineScanner`, a `bufio.Scanner` capped at 1MB per line rather than the 64KB default, the same cap the alpm parser already uses. Long single-line fields (descriptions, license text, dependency lists) now parse instead of ending the scan early.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: use io.ReadFull where a short read was silently accepted
- dotnet bundle 7-bit int decoding could read a zero-byte result as a zero byte
- wordpress plugin header parsing included the unread tail of its buffer
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: finish moving line scanners to the 1MB cap and checking scanner errors
- java manifest, pom.properties, Gemfile.lock, requirements.txt, homebrew formula, alpm and the purl/cpe decoders now use `internal.NewLineScanner`
- snap kernel changelog reports a scan error instead of silently dropping the base kernel package
- the attestation output reader keeps draining the pipe after a scan error so cosign can't block on a full pipe
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: stop logging the full setup.py line once per dependency match
The debug log ran once per regex match and included the whole line, so log volume grew with the square of line length. With 1MB lines that is hundreds of GB of debug output from a single crafted file.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(gentoo): don't dereference a nil location when a LICENSE scan fails
`extractLicenses` accepts a nil location, but the scan error log added earlier on this branch dereferenced it.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(gentoo): bound license group expansion on hostile license_groups
- dedup within a group uses a set instead of `slices.Contains`, which was quadratic in tokens per line and got much worse under the 1MB line cap
- each group is expanded once and memoized, so a chain of groups that reference each other repeatedly no longer expands exponentially
- repeated references to the same group within one line are skipped
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(javascript): don't panic on a yarn v1 lockfile line missing its value
A v1 `yarn.lock` dependency line without a version constraint, or a
`version`/`resolved`/`integrity` line without a value, indexed past the end of
the split line. Such dependencies are now skipped and empty fields left empty.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(alpine): don't panic on an apk db ACL or checksum line before its entry
An `M:`, `a:` or `Z:` line that comes before any directory or file entry in
`lib/apk/db/installed` indexed `files[-1]`. Such a line is now skipped.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(erlang): don't panic on a truncated rebar.lock or .app file
The term parser read past the end of the input when a file ended mid-term:
whitespace only, an unclosed list, a lone `<`, or a comment ending in `\r`.
These now return an "unexpected end of input" error.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(lua): don't panic on a truncated .rockspec
The rockspec parser read past the end of the input when a file ended mid-statement:
a lone `[`, an unclosed `[[...]]` value or `[...]` index, or a `local` assignment cut off
before its `=`, value, or line end. These now return an "unexpected end of input" error.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(swift): don't panic on a Podfile.lock pod without a version
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(gentoo): don't panic on a short obj line in portage CONTENTS
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(spdx): don't panic on an opam checksum without an algorithm
The opam cataloger stores checksums verbatim, so a malformed one reached the SPDX encoder and panicked there, outside the cataloger's recover. Checksums without an `=` are now skipped.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* refactor: use strings.Cut where a split was only indexed after a Contains check
Also guards `windows.FromPosix` against a path with no separator.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: guard untyped package metadata fields
- new `any`-typed metadata fields must be allowlisted with a reason
- allowlisted untyped fields are pushed through `SetID` and every encoder with
non-finite, deeply nested and large values
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: fuzz every generic cataloger parser
One fuzz target over every glob-selected parser in the capabilities data, run
through its cataloger task. Each input must not panic, must finish in time and
must keep allocations proportional to its size. Seeds come from each parser's
testdata plus the string literals in its package.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: scan malformed inputs under memory and time limits
A directory of small malformed files (deep nesting, oversized declared counts,
truncation, type mismatches, non-finite floats) is scanned by the CLI with a
memory limit and an RSS ceiling. The scan must exit 0 and write an SBOM.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: stop a fuzz input once its live heap passes the allocation limit
The allocation check only ran after the parser returned, so an input on its way to exhausting memory ran the machine into swap first. The harness now polls the live heap while the parser runs and panics past the same limit, which the fuzz engine saves as a crasher.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: bound nesting depth and error context in the erlang and lua parsers
- a deeply nested rebar.lock or .rockspec overflowed the goroutine stack, which is fatal rather than a recoverable panic, so one file could take down the whole scan. Both parsers now stop at 1000 levels with an error.
- parse errors echoed whole input lines into the SBOM as unknowns, and lua appended another copy at every nesting level. `PrintError` now clips to a window around the error column, and lua decorates once at the top.
- the Podfile.lock error no longer echoes the pod entry.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: close ways the malformed-input checks could pass quietly
- a ctx-aware parser hitting the fuzz timeout now fails the input instead of racing the hard deadline
- the heap watchdog measures growth from a post-GC baseline rather than an absolute process-wide number
- a saved fuzz input naming a target that no longer exists fails instead of being skipped
- the CLI malformed-input scan fails on a recovered parser panic, not only on a non-zero exit
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(cpe): cap sub-selection prefixes to bound candidate cross-product
a name with thousands of `-`/`_` separators produced one vendor and product candidate per segment, and the vendor x product cross-product made CPE generation roughly cubic in time and memory (8KB name: ~250s, 4.8GB RSS). `generateSubSelections` now stops after 10 prefixes. The full name is still a candidate, so real package names are unaffected.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(cpe): bound candidate count and field length before the cross-product
the prefix cap only covered one way to inflate the vendor x product cross-product in `FromPackageAttributes`. Java groupIDs add a vendor (and, with no artifactID, a product) per dot segment, and .NET executables add one of each per file, so a 1000-segment `Bundle-SymbolicName` still produced ~1M CPEs and several GB.
- vendor and product candidates are each capped at 64 (shortest first, so truncation is deterministic)
- names, versions, and candidates over 512 bytes are skipped, since every byte is paid once per CPE in key building and regex validation
real packages produce around a dozen candidates with names well under 100 bytes, so output for real inputs is unchanged. Also corrects the earlier comment: prefixes only become vendor candidates and the cost was quadratic, not cubic.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
UPX pads to a 4 byte boundary before it writes the loader stub. The
image-small-upx build happened to need no padding, so the cataloger
tests over it passed with or without the fix in #5347.
- build the fixture with -X main.Version=1.0.11, which needs 2 bytes of
padding, so the existing cataloger tests fail without the fix
- add TestImageSmallUPXNeedsLoaderPadding, which fails if a change to the
fixture's inputs means it no longer needs padding
- run the crafted loader test on both sides of the boundary, so a skip
that always rounds up is caught too
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
UPX pads its output to a 4 byte boundary before it writes the loader
stub, and l_lsize counts from that boundary. The block chain resumed at
the end of the last PT_LOAD extent plus l_lsize, so whenever the
compressed extents ended off a 4 byte boundary it landed 1 to 3 bytes
early, read a misaligned b_info, and stopped. The padding between the
segments was never filled in, the reconstruction ended at that first
hole, and .go.buildinfo in the data segment went with it, so the binary
contributed no packages.
The loader skip now rounds up to 4 before adding l_lsize, which matches
the funpad4 in UPX's own unpacker. The crafted fixture for the tail
extents put its stub straight after the head extent, the layout the bug
assumed, so it now pads the way UPX does.
Signed-off-by: huuyafwww <huuya.yamauchi@3-shake.com>
Same issue as #5350: go1.27's compress/flate encodes this 512MB fixture to just over 1MB, so the fixture sanity check tripped before the bound was exercised. The line bound itself is fine.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
go1.27's compress/flate encodes 512MB of zeros to ~1MB (1052690 bytes) where go1.26 gives ~510KB, so the fixture sanity check tripped on newer toolchains before the bound was ever exercised. The decompression bound itself is fine (bounded read allocates ~300KB on both).
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
- pre-scan the GGUF metadata KV section and reject values gguf-parser-go would panic on, recurse on without bound, or overallocate for: wrong typed well-known keys, zero alignment, nested or oversized fully decoded arrays, an unreasonable `split.tensors.count`, and empty strings the parser would misread. Keys are matched the way the parser sees them. The KV count, fully read string bytes, and skipped string counts are bounded too. The file becomes an unknown instead of taking down the scan.
- `copyHeader` now bounds its own read to the max header size.
- non-finite floats in `header` (including inside arrays) become `"NaN"`, `"+Inf"` or `"-Inf"` so JSON encoding can't fail. `header` and `metadataHash` are otherwise unchanged.
- a `metadataHash` failure is now an error instead of an empty hash.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(ai): keep merging GGUF layers when one fails to parse
a single unparseable layer in an OCI GGUF artifact skipped the merge, so the
named model lost its parts and nameless layers were dropped. The error is
still returned.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(ai): only send GGUF layers to the GGUF cataloger
the media type glob `application/vnd.docker.ai*` also matched config,
model.file, license and safetensors layers, so every safetensors OCI model
reported spurious "invalid GGUF magic number" unknowns. Match
`application/vnd.docker.ai.gguf.v3` exactly.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(ai): don't wrap safetensors parameter counts
- tensors whose shape product overflows uint64 are skipped instead of wrapping
- per-file and per-shard parameter totals saturate instead of wrapping
- parameter count and dominant dtype are computed in one pass
- GGUF licenses are created with the scan context, and drop no-op unknown checks
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* refactor(ai): derive safetensors model info from the header in one place
the file and OCI layer parsers built identical metadata from a parsed header.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(ai): keep the GGUF media type with the GGUF cataloger
also corrects stale comments in the GGUF merge processor.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Companion files next to AI models (`config.json`, `README.md`, and the OCI model.file, license and model config layers) are now read with named size limits, and are only used when they look like what they claim to be.
- JSON companions are read whole up to a limit and skipped when larger, rather than being fully buffered or parsed truncated. OCI companion layers are fetched one byte past the cap so oversize is detectable.
- README and license layers read a prefix. The frontmatter block is capped at 256 KB and list values at 32.
- dir and OCI scans share one 8 MB safetensors header cap. Directory scans used to allow 100 MB.
- `config.json` is only used when it has `architectures` or `_name_or_path`, and is looked up beside the model or in its parent directory, no longer all the way up to the scan root
- OCI model.file layers are processed in a stable order, so the chosen name no longer changes between runs
- README frontmatter:
- a list-form `license:` is accepted and no longer drops `base_model`
- `null` values are ignored
- delimiters must be exact `---` lines
- model names from `_name_or_path` / `base_model`:
- Windows paths are handled
- `.`, `..` and `/` fall back to the next name source
- names are capped at 256 characters
Behavior changes worth noting:
- a `config.json` two or more directories above the model no longer names it
- JSON `config.json` files without `architectures` or `_name_or_path` (e.g. `generation_config.json`, app configs) are ignored
- safetensors headers over 8 MB in directory scans are now rejected, matching OCI scans (real shard headers are well under 1 MB)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(generic): recover from parser and processor panics
- a panic while parsing one file becomes an unknown for that location instead of failing the whole SBOM
- a panic in a package processor is logged as a warning and the unprocessed results are kept
- stacks go to the trace log, not into the unknown text
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(unknown): keep errors that unwrap to nil
`sync.PanicError` with a non-error panic value (e.g. `panic("boom")`) unwraps to nil, and was being dropped entirely from the error graph instead of being kept as a leaf.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(task): don't put the stack trace in task panic errors
Task panics are still fatal, but the error now reads `panic in task "<name>": <value>` and the stack goes to the debug log.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test(generic): cover chained processors when a later one panics
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(file): recover from panics in the executable and file digest catalogers
A panic while reading one file is now recorded as an unknown for that file instead of failing the whole SBOM, the same as generic catalogers.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(ai): use go.yaml.in/yaml/v3 to parse model card frontmatter
gopkg.in/yaml.v3 panics on some malformed merge-key payloads (for
example "<<:\r? 0:"), which crashed the gguf and safetensors catalogers
when a model README contained such frontmatter. Switch the Hugging Face
frontmatter parser to the maintained go.yaml.in/yaml/v3 fork, which is
already used by the other catalogers and returns an error instead of
panicking. The existing error path logs and skips the frontmatter.
Fixes#5327
Signed-off-by: tk1475 <118763009+tk1475@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* chore: finish moving off gopkg.in/yaml.v3
- move the snap catalogers and `internal/capabilities` to `go.yaml.in/yaml/v3`. The snap parsers decode yaml from scanned sources and hit the same merge-key panic as #5327
- enable `depguard` with a deny rule for `gopkg.in/yaml.v3` so it can't be reintroduced
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: tk1475 <118763009+tk1475@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
The package supplier is optional in SPDX, but fileSource dereferenced it
without the nil check containerSource already has, so a valid SPDX 2.3
document whose described package has primaryPackagePurpose FILE and no
supplier crashed syft convert and grype sbom: input.
A JSON null in files, a package's files or externalRefs (or an SPDX 2.1
package the version converter cannot map) decodes to a nil pointer that
the conversion then dereferences. Drop nil entries once at the start of
ToSyftModel instead of guarding each loop.
Signed-off-by: Tony <tjagodka@gmail.com>
- log lines shown under the TUI were glued onto the last UI row, and the last row could get erased (and overwritten by the report) when the TUI exits. Fixed in `bubbly` v0.2.2
- log colors were lost because the logger latched its TTY check on the first entry, which went to the TUI footer buffer. `go-logger` v0.2.0 checks per entry and supports `TerminalWriter`, which the footer writer now implements (`NO_COLOR` is still honored)
- log lines under the TUI get a dim `Logs (non-fatal, see -v for more):` title with a branch pointing at the first entry, so it's clear what the trailing lines are
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Docker Hub rebuilt and re-pushed `busybox:1.38.0`, which changed the per-platform config digests and broke `TestMultiPlatformOCIImageSelection` on uncached runs.
- pin the fixture to the busybox 1.38.0 index digest so future re-pushes can't shift it
- update the expected per-platform digests to match
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(java): resolve Groovy 4+ group IDs to org.apache.groovy
Groovy 4.0 moved from the Codehaus coordinates (org.codehaus.groovy) to the
Apache Software Foundation (org.apache.groovy). JARs that ship no pom
metadata fall through to the static artifact-to-group map, which carries the
3.x-era org.codehaus.groovy entries for the whole groovy family, so 4.x
artifacts were attributed to an obsolete group even though their manifest
identifies the Apache vendor.
Where the static map attributes an artifact to the historical group, a
major-version migration resolves groovy 4.x and newer to org.apache.groovy,
while 3.x and earlier keep their historical group. The Eclipse-hosted
groovy-eclipse-* artifacts, which share the prefix but kept their Codehaus
coordinates, are excluded.
Fixes#5311
Signed-off-by: texasich <101962694+texasich@users.noreply.github.com>
* refactor(java): share version-aware groovy group ID lookup with CPE generation
- move the artifact-to-group lookup behind `cpegenerate.ArtifactIDToGroupID(artifactID, version)` so the PURL and CPE paths agree; the exported mutable map is now unexported
- version-dependent entries are expressed as a `groupIDMigration{before, after, atMajor}` (groovy moved to `org.apache.groovy` at 4.0); unconditional entries, including `groovy-eclipse-*` and apache-only 4.0+ modules, stay in the plain map
- add apache-only modules (`groovy-ginq`, `groovy-contracts`, `groovy-callsite`, ...) that otherwise took the group from their per-module `Automatic-Module-Name`
This changes CPEs for pom-less groovy 4+ jars (all real ones): vendors are now `apache` (e.g. `cpe:2.3:a:apache:groovy:4.0.33`) instead of `codehaus`, matching the PURL, jars with pom metadata, and NVD.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: texasich <101962694+texasich@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
closes#5299: docker:// references are OCI images, not GitHub repos.
Returning empty string produces a package with no PURL, matching the
treatment for local composite actions.
Signed-off-by: Yunare Maia <yunare@gmail.com>
An extra-dep can be pinned as "@sha256:<hash>,<size>" or as "@rev:<n>". The hash
slice assumed the first form, so a rev pin left the end index at -1 and the
parser panicked with slice bounds out of range. A rev pin carries no package
hash, so only the sha256 form now yields one.
Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
* build(deps): bump go-make to v0.8.1
Picks up golangci-lint 2.13.2 (built with go1.27) in place of 2.11.4 (go1.26).
The old build panics outright on a go1.27 GOROOT with "file requires newer Go
version go1.27", and an older mise-shim build fails the quieter way: it loses
type info and reports zero findings with exit 0, so a ruleguard run comes back
green without having evaluated anything.
Also folds in what the newer linter flags:
- `reflect.Ptr` -> `reflect.Pointer` in three files, applied by --fix
- `tw.CellFormatting{MergeMode: ...}` -> `tw.CellMerging{Mode: ...}` in the
cataloger info table, for a staticcheck SA1019 deprecation
Note the deprecation text names `CellConfig.CellMerging.Mode`, but the field is
`CellConfig.Merging`; `CellMerging` is the type. That command path has no test
coverage, so the swap was checked by rendering the same hierarchical-merge table
both ways and diffing: byte identical.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(lint): tune goconst for data tables
goconst got stricter in golangci-lint 2.13 and now counts strings in map
literals and struct fields, which lights up every lookup table in the repo. It
was already enabled, so this is main going red on a linter bump rather than
anything new in the tree: 101 findings, and the default threshold of 3 means a
vendor name appearing in three rows of a table reads as a magic constant.
Raising `min-occurrences` to 5 takes that to 14. The rest split by whether the
repeated string is data or an actual constant:
- `cpegenerate` is excluded by path. It maps artifact names to vendors,
products, and maven group ids, and repeats strings 35, 83, even 293 times, so
no threshold reaches it. Hoisting those would trade a readable table for
indirection.
- the two SBOM fixture builders in `format/internal/testutil` get a
function-level nolint each. The repeated names and versions are what the
golden files assert against.
- `mysql-cluster` and `php` were genuine repeated literals and are now
constants.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(test): extract measureAlloc into a shared helper
The allocation-budget tests added during the unbounded allocation work grew a
copy of the same helper in every package they touched: four identical
`measureAlloc` definitions plus two more spelled out inline against
`runtime.MemStats`.
Now `testutils.MeasureAlloc`, at the repo root so `internal/spillbuf` can reach
it too. The doc comment carries the two things that were only recorded in some
of the copies: TotalAlloc is process-wide so callers must not use t.Parallel,
and a budget assertion only means something if the fixture would really blow
past it unguarded, so measure the unguarded path as well.
No behavior change, the body was identical in all six.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(lint): add ruleguard rules for overflow and swallowed EOF
Two rules out of the recent unbounded allocation work (#5187, #5195, #5267,
#5291, #5292, #5293, #5294).
`noOverflowingBoundsCheck` flags `offset + size > limit` (and <, <=, >=) where
an operand is unsigned, neither addend is constant, and the operands are at
least 32 bits. Unsigned addition wraps, so an offset out of a file header can
carry the sum past the wrap point and pass a check it should have failed. The
fix is to subtract against the limit instead. Constant addends are excluded
because `pos+4 <= uint32(len(buf))` is the normal way to step a cursor, and the
32-bit floor drops nibbles widened for a comparison. Would have caught four
sites across #5195 and #5291.
`noSwallowedEOF` flags `return nil` / `return nil, nil` on an io.EOF check. The
caller gets a zero value it cannot tell from real data, and anything decoding a
fixed-size struct off it parses the padding as content. That is the shape behind
#5186.
Both decide from the expression alone, and writing the fix deletes the pattern,
so neither needs a suppression anywhere in the tree. That was the bar: rules for
the allocation and decompression classes were tried and dropped, because the
bound they look for lives in a preceding statement (or, for a decompressor
constructor, always downstream), so no correct implementation can silence them.
That would have meant ~15 permanent nolints on code that is already right. The
rules here that do work don't judge whether a bound exists, they route to a
wrapper that takes one: noDirectTempFiles, noDirectELFOpen.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: pin allocation budgets on the alpm, deb, and changelog bounds
The bounds added for #5292, #5293 and #5294 are asserted by the error they
return. That keeps passing if the bound is removed and something else rejects
the input later, which is the failure these were written for. Each of these
measures bytes instead, and proves the fixture is really a bomb first so a
fixture that quietly stops being one fails rather than passing.
- alpm mtree: 16MB of newlines, well inside the 64MB byte cap, costs 16.5GB
uncapped against 238MB with maxMtreeLines. That 16.5GB is the measurement the
line cap was added for and it was recorded only in the commit message. Note
what the cap actually buys: 300k entries still cost a few hundred MB, per
concurrent cataloger.
- deb members: 1.28GB against 392KB. Both arms read with io.ReadAll on purpose,
since the stream itself never holds much whatever the cap says; what the cap
protects is a consumer that buffers, and the real one is a tar reader building
entries.
- snap changelog: 1.28GB against 175KB. This one has no explicit cap at all. It
is held by bufio.Scanner refusing a token past MaxScanTokenSize, which is
incidental, so switching to io.ReadAll or a bufio.Reader to pick up long lines
would remove it silently.
Nothing added for the kernel module reader: its copy is already
io.Copy(tempFile, io.LimitReader(rc, max+1)), so peak disk is bounded by
construction and heap is streaming. A heap budget would assert the wrong
resource, and the three existing subtests cover the cap, at-cap fidelity, and
spill cleanup.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: pin allocation budgets on the ai and alpm byte caps
Follows the same pattern as the alpm line cap: measure bytes, and prove the
fixture is really a bomb first so one that quietly stops being a bomb fails
rather than passes.
- safetensors: 252MB against 2.6KB. The existing over-cap case deliberately
omits the body, since the guard rejects on the declared length before reading
anything. That is the right test for the guard but it would keep passing if
the check moved after the read, so this one delivers the bytes.
- gguf: 268MB against 168MB. copyHeader io.Copy's whatever it is handed, so
maxHeaderSize holds only because every caller wraps the reader first. This
pins that the limit is where the bound lives, so dropping the wrapper at a
call site fails here.
- alpm byte cap: 740MB against 132MB, the counterpart to the line cap budget.
Worth knowing for anyone writing the next one of these: a single enormous line
does not exercise the alpm byte cap. go-mtree runs its own bufio.Scanner and
refuses a token past 64KB, so that fixture fails as "token too long" before the
byte cap is consulted. The fixture here uses ordinary 1KB lines, enough of them
to pass 64MB while staying under the 300k line cap, so the byte cap is actually
the thing under test.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: assert bytes retained instead of allocated for gguf copyHeader
the bounded arm writes into a bytes.Buffer, whose doubling growth costs a
multiple of what it holds and lands on a different power of two depending on
the platform. it measured 168MB locally and 320MB on CI, past the budget.
buf.Len() states the same property deterministically.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* Update steroscope to include OCI platform metadata
Signed-off-by: Jason Paulos <jasonpaulos@users.noreply.github.com>
* chore(deps): point stereoscope at anchore/main
drops the fork `replace` directive and moves to a pseudo-version off stereoscope
main (`a22e5ae`, anchore/stereoscope#657). no tag carries it yet, so this should
move to a real version once one is cut.
`readImageMetadata` now fills os/architecture/variant for every provider, not
just oci. the docker tarball provider set neither before, so image metadata and
the image purl change for docker-archive input too:
- `"architecture": ""` -> `"amd64"`
- `"os": ""` -> `"linux"`
- `?arch=` -> `?arch=amd64`
the empty `arch=` qualifier was never right, but it is user-visible output.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: cover single-platform oci layouts
the pr only added assertions to the multiplatform fixture, which already
reported the right platform. the behavior that actually changed is single-image
layouts: they used to report empty os/arch and silently ignore `--platform`, and
now report what the image config says and fail with a platform mismatch.
also drops the non-linux skip on the default-platform test. the oci providers
default to `linux/<GOARCH>` regardless of host os, so it holds on macos too.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Jason Paulos <jasonpaulos@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
When javascript.search-remote-licenses is enabled, the lock catalogers
query registry.npmjs.org once per package occurrence, on every run, with
nothing in front of the network call. The same name@version is fetched
again for every lockfile that mentions it, and again on the next scan.
This adds a javascriptLicenseResolver that resolves licenses through
syft's on-disk cache, keyed on "<name>/<version>" and namespaced as
javascript/v1, and routes the package-lock v1, package-lock v2, pnpm,
yarn and bun package constructors through it, leaving
getLicenseFromNpmRegistry itself unchanged. Failed lookups are not
cached: a transient registry failure or a name that 404s must not be able
to blank out licenses for the lifetime of the cache entry, since a
missing license silently weakens downstream license policy checks. A
successful lookup that returns an empty license is a real answer and is
still cached.
Signed-off-by: Anže Pečar <anze@pecar.me>
`.ko.gz`/`.ko.xz`/`.ko.zst` modules are spilled to a temp file so ELF parsing
can seek over them. The compressed size is bounded by the file in the image but
what it decompresses to is not, so a few KB of crafted zstd filled the temp
directory until the disk did.
`io.Copy` now stops one byte past a 256MB cap and rejects rather than
truncating, since a partial ELF reads as a corrupt module instead of a rejected
one. The partial spill is removed on the reject path.
The cap is a parameter on an inner function so the tests exercise the boundary
with a few hundred bytes instead of materializing 256MB, matching
`decompressSbomWithLimit` in the graalvm cataloger.
The cap bounds one spill, not the process: peak disk is 256MB times however many
modules are parsed at once, which is cataloger parallelism (`NumCPU*4` by
default, and unbounded when `--parallelism` is negative).
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(arch): bound the decompressed mtree listing
The ALPM cataloger handed a gzip stream straight to `mtree.ParseSpec`, which
materializes every entry before it returns any of them. Nothing capped the
decompressed size, so a crafted mtree in a scanned image expanded until the
process died. 64KB of input is enough to produce 64MB, and the ratio scales.
Reads through a limit now and rejects a listing that exceeds it. The read goes
one byte past the cap so that tripping it is distinguishable from a listing that
simply ends there, since silently truncating would produce a package missing
most of its files. The existing error path already surfaces this as an unknown.
The cap is well above any real package: an mtree names every file with its
digests, and even the largest packages land far under it.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(arch): bound mtree entry count, not just decompressed bytes
The byte cap alone does not bound what the parse retains. go-mtree keeps an
entry per line and appends it unconditionally, blank lines included, so short
lines cost about 1KB of heap each.
Measured on the previous commit: 15KB of gzipped newlines expanding to 16MB,
comfortably inside the 64MB byte cap, drove 8.1GB of peak heap over 74 seconds
and returned no records and a nil error. Same input now rejects in 0.05s at
51MB peak.
Counting lines before parsing is what makes the entry allocation bounded, since
the parser materializes all of them before returning. Lines the parser skips
only make the count high, never low.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(arch): apply the mtree bounds while streaming
Both caps were enforced against a fully buffered listing, so the peak cost
scaled with cataloger parallelism, which runs a goroutine per package. They now
apply as the stream flows: a limited reader for the byte cap and a counting
reader for the entry cap, which has to trip while the parser is still reading
since it materializes every entry before returning any of them.
Both limits are injectable so the boundary tests can be exact on both sides
without allocating their way up to the shipped 64MB. Each cap gets a listing
exactly at it, asserting the records parse whole rather than just that no error
came back, since quiet truncation is the failure a size check invites.
The byte limiter carries one byte of headroom, which is what makes an
overrunning listing distinguishable from one that simply ends at the cap.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(arch): refuse mtree line continuations
go-mtree collapses a line ending in a backslash into the next one with a
quadratic string concat, so the cost is lines times bytes and neither the byte
cap nor the line cap can see it. 199k continued lines sitting inside both caps
measured 4m34s and 6.1TB of allocation churn, and returned no error.
Also re-derived both constants against a real package rather than a guess.
`texlive-fontsextra` is 105,847 files and 1,978 directories, so roughly 108k
lines at 150-250 bytes each. 16MB would have falsely rejected it, so the byte
cap stays at 64MB and the line cap goes 200k -> 300k, which is ~2.8x a real
package instead of 1.85x.
Rejections are now sentinels so tests can name which bound tripped, and the
constants carry the measured ceiling they imply: a crafted listing of minimal
fields still reaches ~1.1GB at the byte cap, since go-mtree allocates per
whitespace-delimited field. Bounding that further means bounding fields rather
than bytes, and legitimate input is already within ~1.3x of the crafted worst
case, so there is little to win.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(arch): refuse a continuation spelled with a carriage return
The refusal matched the literal bytes backslash-newline, but go-mtree decides
"this is a continuation" from the scanner token, and `bufio.ScanLines` drops a
trailing carriage return before testing for the backslash suffix. So a listing
using CRLF ran the quadratic collapse with nothing watching.
Measured: 775KB of gzip expanding to 17MB across 150k lines, inside both other
bounds, took 1m33s and returned a nil error. Same input now rejects in 50ms.
The check is a single byte pass that counts newlines and looks for a backslash
before either line ending, which replaces the two scans and the straddle special
case it took to catch only one spelling. A tripped bound is also latched now:
`bufio.Scanner` stops at the first error, but nothing about the type said a
caller had to, and the refusal was recomputed per chunk.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test(arch): make the bombpkg fixture able to fail
The fixture was 310k bare newlines, which go-mtree turns into blank entries with
no name, and the parse loop already skips those. So the assertion that a rejected
listing leaves an empty file list held whether or not the bound existed. It is now
a repeated file line: 0 records with the bound, 310,000 without.
Dropped the two tests that asserted nothing as a result. The cataloger-level one
was a weaker restatement of what the golden comparison already pins exactly, and
the shipped-size-cap one drove a real 64MB listing through the parser to prove a
constant had not moved. Asserting the constants directly catches any change rather
than only a loosening.
Race-mode package runtime goes 15.7s -> 4.4s.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test(arch): make three mtree assertions able to fail
The truncated-gzip case asserted the failure was not misreported as the size
bound, but ran against the shipped 64MB cap on a 3.5KB listing, so the check it
guarded could never fire. It now runs against a cap just above the truncation
point, where the two genuinely race.
The bomb case asserted an allocation delta from `runtime.MemStats`, which is a
process-wide cumulative counter and measures the property indirectly. It now
counts bytes pulled through the reader, which is what "gave up while reading"
actually means and does not drift with GC or parallelism.
Added the one bound that had no test at all. A single line over 64KB is refused
by go-mtree's scanner, not by anything here, so a dependency bump that called
`scanner.Buffer` would remove it silently.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* docs(arch): correct the mtree bound comments and close the gzip reader
The comment said both bounds apply while the listing streams. Only the line cap
does. `io.LimitedReader` signals a drained budget with `io.EOF`, which the parser
reads as the end of a complete listing, so the byte cap is necessarily a check
after the fact. Peak cost is the same either way, since accepting a listing at
the cap costs what rejecting one just past it does, but the comment was claiming
a mechanism that isn't there.
Also trimmed the benchmark writeups down to the reasoning that outlives them.
The exact transient-heap and wall-clock figures are tied to a go-mtree version
with nothing to detect them going stale, and one of them described the CRLF case
as prevented when it wasn't. And `TestMtreeParse` was labelled the regression
guard for the continuation refusal, but its fixture has no backslashes in it.
`gzReader` now gets a `defer Close` to match the other gzip site in the tree.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(arch): stop the alpm database parser panicking on a malformed backup line
A `%BACKUP%` entry is `path\tmd5`, and the parser indexed the second field
without checking the split produced one. A line with no tab panics. The database
is attacker-controlled, and while the task executor recovers, it recovers at task
granularity, so one crafted line drops every Arch package from the SBOM.
The same loop also never checked the scanner's error. A field over the 1MB token
cap ends the scan, and the entry came back populated up to that field and empty
after it, with a nil error saying it was fine. That is the failure the mtree
bounds in this branch exist to avoid, two functions up in the same file.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* docs(arch): trim the comments added by this branch
the comments were carrying reasoning that belongs in the PR description rather
than the code. Kept the parts a future reader cannot recover from the code (the
cap sizing, why continuations are refused, why the byte check runs after the
parse) and cut the rest.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test(arch): generate the mtree bomb fixture instead of committing it
the fixture is a synthetic listing fully described by a line count and a
repeated line, but it landed as a committed gzip member, which shows neither and
leaves no way to rebuild it when maxMtreeLines moves. the two tests that walk
testdata/installed write it first now, and git ignores it.
the real pacman listings (gmp, tree-sitter) stay committed since those are
captured output rather than something a rule can reproduce.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* correct and simplify comments
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* feat(convert): passthrough-exact-format copies matching syft-json input unchanged
When a syft-json input SBOM already exactly matches the requested output
format and schema version, `syft convert` can write the input bytes out
verbatim instead of decoding and re-encoding them. This avoids the lossy
round trip (and the associated cost) for callers that normalise SBOMs into
a target format without knowing up front whether the source already matches.
The behaviour is opt-in and configuration-only: it is enabled with
convert.passthrough-exact-format in the config file or the
SYFT_CONVERT_PASSTHROUGH_EXACT_FORMAT environment variable, and there is
deliberately no command-line flag, since it changes how every invocation in
a pipeline behaves rather than varying call by call. It defaults to off, so
existing behaviour is unchanged.
Pass-through applies only to syft-json input, and only when the document's
schema version is identical to the one this build of syft produces; any
other input is always converted. Outputs that do not match are still
converted, so a mixed set of `-o` values behaves as expected, and every
output is resolved and validated before anything is written. Pass-through
output is a byte-for-byte copy, so format options such as `format.pretty`
and `format.json.legacy` are not applied to it.
When the option is enabled a warning is logged stating so and noting the
syft-json-only scope, and a second warning is logged whenever an input is
found to match and the shortcut is actually taken.
The option lives under a `convert:` configuration section (options.Convert),
mirroring how attest scopes its own options. The description documents the
caveats above and notes that file-path input written with `-o <format>=<path>`
keeps memory minimal, whereas piped input or STDOUT output must hold the
whole document in memory.
Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* perf(convert): stream pass-through output instead of buffering the input
Identify, copy, and decode the input through a seekable reader rather than
loading the whole document into memory first. File inputs are read in place
and pass-through output is an io.Copy from the file. STDIN is used in place
when it is a redirected regular file. A pipe or terminal cannot seek, so its
contents are drained into an internal/spillbuf buffer: the first 1MB stays in
memory and the rest goes to a temp file under the user's TMPDIR (created
lazily, only when something actually spills), giving a seekable view without
ever holding the document in memory. The temp dir is rooted per invocation
via internal/tmpdir, honouring one already attached to the context, and is
removed when the command returns.
Writing pass-through output over the input file itself is now a no-op rather
than truncating the file before it is copied.
Measured on a 1GB syft-json document (Apple M1 Max), file in, file out:
origin/main full re-encode 86.3s wall 99s CPU 12.0GB max RSS
in-memory pass-through 7.4s wall 6.7s CPU 4.2GB max RSS
streaming pass-through 7.0s wall 6.9s CPU 3.1GB max RSS
and the same 1GB document piped over STDIN with pass-through enabled:
io.ReadAll into memory 1.3s wall 0.8s CPU 2.7GB max RSS
spill buffer 1.8s wall 1.0s CPU 66MB max RSS
The remaining cost of the streaming pass-through is entirely format.Identify,
which for syft-json decodes the whole document through a json.Decoder that
buffers the full value: measured alone it takes 7.1s and 3.1GB on the same
input. Reducing that further belongs to the format package.
Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* perf(syftjson): identify documents from their tail instead of a full parse
syft always writes the schema block as the last key of a syft-json document,
so the format and version can be read from the final bytes of the file. The
syft-json decoder's Identify now reads at most 2MB from the end of a seekable
input, finds the last "schema" key there, and verifies it sits directly within
the top-level object by scanning the remaining bytes for the single closing
brace of the document. Anything the tail cannot vouch for (non-seekable input,
a reordered document, a nested key) falls back to the existing full parse, so
behaviour is unchanged for documents the fast path declines.
The verification is position-independent: it also accepts documents whose
keys have been reordered so that other small keys follow schema. It rejects
nested schema objects, array-wrapped documents, truncated input, trailing
content, and schema text inside strings.
Supporting changes:
- the offset-aware reader the decoder collection hands to decoders now
supports io.SeekEnd, which the tail read requires
- model.Document documents that Schema must remain its last field, and a
test guards that the encoder writes it last in both compact and pretty
output
- the decoder list documents that syft-json must stay first
Since Decode calls Identify before parsing, this also removes one of the two
full parses every syft-json conversion used to perform.
Measured on a 1GB syft-json document (Apple M1 Max), file in, file out:
pass-through, previous commit 7.0s wall 6.9s CPU 3.1GB max RSS
pass-through, this commit 0.85s wall 0.05s CPU 66MB max RSS
full re-encode, previous commit 85.3s wall 99s CPU 11.5GB max RSS
full re-encode, this commit 75.2s wall 91s CPU 9.9GB max RSS
Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(syftjson): only identify from the tail when the document starts where the reader does
The tail scan vouched for the last 2MB, but nothing checked those bytes belonged to the
document the reader was pointed at. So `Identify` started claiming syft-json for files it
cannot decode: a document with a byte order mark or a stray log line in front of it, or a
syft document sitting after something else in the same stream, where `format.Decode` went
on to return an empty SBOM with a nil error.
Two O(1) checks:
- the first non-space byte at the reader's position has to be `{`
- `Decode` errors when the parsed document has no `schema.version`, instead of handing back
the empty document it decoded into
Streams holding several concatenated documents are deliberately not detected. Proving there
is exactly one top-level value means reading all of it, which is the cost the tail path
exists to avoid, so `Identify` can still name a later document's version on such a file.
`Decode` reports what it actually parsed.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(convert): recognize the input file by identity rather than by path
`convert - -o syft-json=sbom.json < sbom.json` truncated the file and exited 0. The
passthrough guard compared paths, and a shell redirect does not have one, so it was blind to
the one input mode where the truncation was silent.
`openConvertInput` now returns the input's `os.FileInfo`, which the STDIN branch already had
from its `Stat`, and `isSameFile` compares that instead.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(java): bound user-controlled reads in the graalvm native-image cataloger
Sizes, offsets and lengths in a native-image binary all come from the file
being parsed, and several were acted on directly.
The PE export directory sized a `make([]byte, Size)` from a uint32 in the
optional header, so an 8KB file claiming 4GB reserved 4GB before reading a
byte. It now reads what the file actually holds, bounded by the declared size,
and still requires the whole directory to be present. This also fixes a latent
bug: `unionreader.readerAtAdapter.ReadAt` can return a short read with a nil
error, and the old code discarded the count, so squashfs-sourced binaries could
parse zero padding as export data.
The bounds checks in `decompressSbom` and the PE export walk validated only the
end of a range, computed by adding to a value out of the file. Those sums wrap,
and a wrapped sum compares as in range while the slice that follows it panics.
They now subtract from the known-good length instead. The three
`address - sectionBase` subtractions are unsigned and underflowed on an address
below the base; they share one guarded helper now.
The embedded SBOM decompresses through an `io.LimitedReader`, since the
compressed bytes are bounded by the file but what they expand to is not. The
limit is checked before the decoder's own error, which would otherwise report a
size problem as "not a cyclonedx json document". Hitting it is logged, since
the caller reports parse failures at trace level and a dropped SBOM would
otherwise go unnoticed.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(java): stop at the first format that parses a native image
The format loop tried ELF, Mach-O and PE against every reader and kept going
after one of them parsed, so a file that parses as more than one format
contributes its packages once per format.
No input parses as two formats today, so this is not a behavior change in
practice: `debug/pe` rejects an ELF on the optional-header magic, and both
`debug/elf` and `debug/macho` reject anything without their own magic. It
removes the possibility rather than a live bug.
Note this does not deduplicate across readers, which is the outer loop: a
universal Mach-O carrying the same native image for two architectures still
yields its packages twice.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(java): bound the graalvm export directory read against the file
Follow-up to the hardening earlier in this branch, from review.
The export directory read was still bounded only by file size, so a large
mostly-empty file (cheap to ship inside a compressed layer) was read into
memory in full and then discarded. It now weighs the declared size against an
absolute cap and against the bytes really present before allocating anything,
matching `readDataFromRVA` in `syft/pkg/cataloger/internal/dotnet/pe`. A 64MB
file declaring a 4GB export directory went from ~140MB allocated to ~3.6KB.
Also from the same review:
- dropped the decompressed-SBOM ceiling from 100MB to 16MB. Decoding costs
roughly 48x the JSON size in allocation and 6x retained, so the old ceiling
was worth several GB of churn per file, and catalogers run concurrently.
- `symbolOffsets` is now single-address `symbolOffset`. Two same-typed address
arguments in a row could be transposed at a call site and would produce a
silently wrong SBOM rather than an error.
- `newMachO` now returns an error on a real read failure. It was falling
through and dropping the file with no error and no log at all.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(java): reject a short read of the graalvm export directory
Second round of review on this branch.
The export directory read introduced earlier in the branch only consulted the
byte count when `ReadAt` also returned an error, so a reader handing back fewer
bytes with a nil error was accepted and the tail of the buffer stayed zeroed,
then got parsed as a valid export directory. The count decides now, whatever
the error says. `io.ReadFull` would also cover it but spins forever on a reader
that keeps answering `(0, nil)`, which is the same hostile input this is meant
to reject, so the single read stays.
Also in here:
- the decompressed SBOM is buffered and size-checked before the decoder is
invoked, instead of inferring the overrun from `io.LimitedReader.N` after the
fact. That check was only correct because the cyclonedx decoder happens to
drain its reader before parsing, and nothing would have caught it if that
changed.
- four bounds guards, the `newMachO` error path and the SBOM ceiling had no
test that failed when they were removed. They do now, verified by deleting
each one and watching a named test go red. The ceiling in particular could be
raised to 2GB with the suite still green, since every case derived its sizes
from the constant.
- `nativeImagePE.reader` is dead now that the read takes the reader as an
argument.
- three comments claimed guarantees the code does not make: that `symbolOffset`
cannot be called with transposed arguments, that a rejected file cannot go
unrecorded when it is only logged at trace level, and a set of measured
allocation multipliers that nothing pins.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(java): trim the graalvm cataloger comments
Comment pass over everything this branch added. Cut about a quarter of the
comment lines, mostly restatement and detail that was never going to stay true.
Dropped: a production comment naming a test, the measured allocation multiples
behind the two size ceilings, cross-references to the equivalent constants in
`dotnet/pe` and `elfutil`, and test docs that restated their own test name
before saying anything.
Kept the parts that fail silently if someone tidies them away: why the read is
a single `ReadAt` rather than `io.ReadFull`, the note that `readExportDirectory`
uses an RVA directly as a file offset, and the fixture notes explaining why a
`VirtualAddress` is deliberately huge or why no symbol sits at index 0.
Three that were stale rather than verbose:
- `nativeImagePE`'s doc still said it holds the underlying reader, which this
branch deleted.
- `TestFetchSbomSymbols_FindsAllThreeSymbols`' doc had ended up above
`TestFetchSbomSymbols_AddressBaseOutOfRangeGuard`, documenting the wrong
function.
- a comment referenced a commit sha that no longer exists on this branch.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(dotnet): bound the bundle signature search by the file, not the headers
The single-file bundle marker search sized its buffer from `calculatePEEndOffset` /
`calculateELFEndOffset`, both of which sum user-controlled header fields. A tiny file
declaring 4GB sections drove an 8.6GB allocation, and on the ELF side the sum could
overflow negative and panic in `make`.
The search now clamps its window to the real file length before sizing anything. The PE
and ELF paths had grown two copies of the same find-and-read sequence, so they collapse
into one `bundle.ExtractDepsJSON`, which lets the signature search and the header reader
both go unexported.
Also guards `read7BitEncodedInt` against returning a negative length, which on a 32-bit
`int` would seek a manifest walk backwards and re-read the same bytes per declared file.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(dotnet): stop a truncated version resource from spinning forever
`readIntoStruct` reported a truncated read as success and left the destination struct
zeroed. Its callers advance their loop counters by the offsets it updates, so the string
table loop consumed nothing, advanced nothing, and never reached its length bound: a
version resource whose header claims more content than the blob carries hung the parse.
A truncated read is now an error, and the two places where hitting the end of a blob is
normal say so explicitly. That distinction is what keeps a well-formed resource whose last
child is `VarFileInfo` from losing its `FileVersion` fallback. io.EOF and
io.ErrUnexpectedEOF mean the same thing here, so both take the same path.
The string table walk moves into its own function, since the two new arms put
`parseVersionResourceSection` over the funlen limit.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(dotnet): bound PE resource parsing by what the file actually holds
Resource sizes and RVAs are user-controlled uint32s that sized buffers before anything
checked the file held that many bytes, so a small crafted section could reserve up to 4GB.
The walk now carries the section's reader and its base RVA for its whole lifetime, so
`reader.Size()` is the single authoritative bound and one `offsetOf` helper rejects every
RVA the section does not hold. That replaces the per-node section value the recursion used
to synthesize, whose size argument was only correct by convention.
Bounding the allocations alone turns the OOM into a hang, though: nothing in the format
stops entries from aliasing, so thousands of them may name one fat blob and every offset
still checks out. A 256KB section built that way drove 24GB of allocation over two minutes
with peak memory flat. Capping entry *count* does not help since the cost is per byte read,
so the walk now charges every read against a budget of a few times the section size. Real
binaries come in at about 1x, since a well-formed section's entries partition it.
A resource tree that stops early no longer fails the whole file. It says nothing about the
CLR directory or an embedded deps.json, and dropping the package from the SBOM over one
malformed section is worse than reporting the fields we did get.
Also widens the entry counts before summing them (two uint16s can sum past 0xFFFF and wrap),
and bounds the name length and section header count against the reader for the same reason.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(dotnet): bound resource recursion depth and cap declared sizes absolutely
Review follow-ups on this branch. The headline one is a real crash: the resource
walk had no depth bound, and the `budget` comment claiming depth "falls out for
free" was wrong, since the budget scales with the section while the stack does
not. A chain of directories at distinct RVAs (which the `dirs` set does not
catch) recursed until the goroutine stack gave out. A 13.7MB crafted PE took the
whole process down with `fatal error: stack overflow`, which is unrecoverable,
so unlike every other bug on this branch no caller could log-and-continue past
it. Now capped at 32 levels, versus the 3 the format actually uses.
The other bounds moved from "the file" to a real ceiling. Clamping an allocation
to the bytes remaining in the file is not enough on its own: a mostly empty file
costs almost nothing inside a compressed layer, so a cheap artifact could still
authorize a multi-gigabyte read. Both the PE data directories and the bundle
marker search now cap the same way `elfutil` already does.
Also hardened in the bundle reader:
- `headerOffset` is validated against the file. It comes straight out of the
file bytes and everything downstream seeks to it, so it was the least
trustworthy value in there and the only one not being checked.
- a manifest entry count that could not fit in the remaining bytes is rejected
rather than walked for hundreds of millions of reads.
- a non-positive `searchLimit` falls back to the whole file instead of reading
nothing. It is only ever an optimization hint, so treating it as authoritative
let one bogus ELF `p_offset` hide a bundle from us entirely.
- a short read no longer aborts the search. `unionreader` can return fewer bytes
than the file reports, and the marker may well be in what we did get.
`pe.Read` now reports partial parses instead of choosing between dropping the
package and hiding the problem. It was inconsistent: a malformed resource tree
was non-fatal while an unreadable resource *section* one call away was fatal, and
the non-fatal path only reached a trace log, so the unknowns task could not see
that a file had been cataloged on incomplete evidence. Both now land on a new
`File.ParseErr`, which the dotnet and PE catalogers join into their unknowns and
still keep the package.
Worth noting for changelog purposes: the `isTruncated` guard fixes a
user-visible bug beyond the hardening. A version resource leaving 1 to 5
trailing bytes cuts the next struct in half, which reports as
`io.ErrUnexpectedEOF` rather than `io.EOF`; that used to propagate out and skip
the `VS_FIXEDFILEINFO` fallback, so those binaries lost their version entirely.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(dotnet): treat a zero bundle offset as unbundled, not malformed
The apphost ships the 32-byte bundle signature compiled in with its 8-byte header
offset left as zero, and only `HostWriter.SetAsBundle` fills that slot in when you
publish as a single file. So a zero offset means "never bundled", which is what
every framework-dependent .NET executable looks like.
The bounds work on this branch folded that case in with the hostile ones, so the
most common shape a .NET binary takes started reporting a parse failure. It
surfaces through the PE cataloger rather than the dotnet one, since an apphost
carries no CLR metadata of its own: the package still comes out, with a spurious
unknown attached to it. `main` had this right and said so in a comment; the
regression is on this branch.
Reproduces on a fixture already in the repo. Of the 53 PE files in
`image-net8-app`, `/app/dotnetapp.exe` was the only one reporting anything, and
the whole image is clean again now.
The three outcomes are now distinct, since collapsing any two of them loses
something the caller needs: zero means there is no bundle, an error means we could
not tell, and a positive offset is the answer.
`Test_Read_DotNetDetection` gains the apphost and, more usefully, an assertion
that a well-formed binary reports no partial parse at all. That one line is what
would have caught this, and it now covers all six fixtures. The unit case that
pinned the old behavior asserted the bug, so it flips.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(dotnet): measure readers with the shared helper and report refusals
Two things, both about not re-deriving what the campaign already has.
`internal/file.ReaderSize` exists on `main` for exactly this job, and `upx.go`
already uses it. Its doc comment is the problem statement here almost verbatim: a
declared size is worth nothing until you weigh it against a byte count the reader
can actually back, which it does by reading the last byte. This branch was
hand-rolling `Seek(0, io.SeekEnd)` in three places, one of them reimplementing the
helper's save-and-restore dance verbatim, and then absorbing the reader-lies case
downstream by tolerating short reads.
Adopting it means the bundle reader and `readDataFromRVA` now take the reader
types they were always being handed (`unionreader.UnionReader` and `io.ReaderAt`),
and the reads move to `ReadAt`, which leaves the caller's cursor alone and drops
two seeks. Net effect is a smaller diff than what it replaces, and the case the
`shortReader` fake was invented for is now caught up front instead of tolerated:
the fake was modeling a lie about length, which is not what a short squashfs read
actually does. It now models the real shape (honest length, read stops early) and
a second reader covers the lie.
Three places also still returned "there is nothing here" when what they meant was
"I could not tell", which is the thing `File.ParseErr` was added to stop:
- an ELF that `elfutil` refused to parse read as "not an ELF". `ErrDeclaredSizeExceeded`
is exported precisely so a refusal can be told apart from a broken file, and the
caller upstream already turns an error into an unknown.
- a `deps.json` past the 3MB cap returned "". That is the file's entire dependency
list, so this handed back an SBOM that looked complete and silently was not.
- the marker search clamped at 512MB and then reported no bundle. It cannot claim
there is no bundle when it declined to look everywhere one could be, so a miss
after clamping is now an error.
`readSizeRecorder` records `ReadAt` alongside `Read`, or the allocation-bound tests
would have quietly stopped measuring anything.
Also renames `findSignatureOffset` back to `findBundleHeaderOffset`, since it
returns a header offset and reading it as "no signature" is what the last bug was,
and fixes the stale `ExtractDepsJSONFromBundle` doc comment.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test(dotnet): assert allocation sizes directly in the bundle search guards
The PE and ELF search guards asserted the largest read the reader was asked for.
That happened to track the allocation, since the whole buffer goes to ReadAt, but
nothing said so. Both now measure the bytes allocated with the same `measureAlloc`
helper the resource guards use, and keep the read-size check as a separate claim.
Also names the reason a reader can report a size it cannot back: `debug/elf` and
`debug/pe` hand out `io.NewSectionReader(r, 0, 1<<63-1)`, whose `Size()` is nominal
rather than measured, so answering a size is not on its own evidence the bytes are
there. Renames `lyingReader` to `unbackedSizeReader` and regroups the two reader
implementations, which had their methods interleaved.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Decoding an SPDX document into the syft model was quadratic in document
size. findRootPackages (spdxhelpers/to_syft_model.go), which locates the
packages the document DESCRIBES, scanned every relationship for every
package. On a 119MB SPDX JSON document (70,701 packages, 153,301
relationships) that is 1.1e10 comparisons and 91% of all CPU samples; on a
238MB document it is 4.3e10 comparisons and 95%. A 1.3GB document did not
finish in 91 minutes. Everything else on the decode path is map-indexed and
linear, and spdx/tools-golang's own parse is linear (4.5s and 9.3s for the
two documents above). The scan dates from 9480f10c (top-level SPDX package
support, 2023).
The function now indexes the DESCRIBES / DESCRIBED_BY relationships once,
counting how many times each element is described by the DOCUMENT, then
walks the packages once. Semantics are unchanged: results follow package
order, an element described by more than one edge still appears once per
edge (so the existing "exactly one root" check behaves as before), only
edges touching DOCUMENT count, and nil entries are skipped.
Measured on an Apple M1 Max, `syft convert <spdx> -o <format>=<file>`:
input target before (main) after
119MB spdx-json@2.3 68.2s / 1.3GB 10.4s / 1.3GB
119MB spdx-json@2.2 70.2s / 1.3GB 12.6s / 1.3GB
238MB spdx-json@2.3 291.2s / 2.3GB 20.9s / 2.6GB
1.3GB spdx-json@2.3 killed at 91min 115.5s / 13.6GB
1.3GB spdx-json@2.2 not attempted 136.3s / 14.1GB
1.3GB syft-json not attempted 102.6s / 14.1GB
1.3GB cyclonedx-json not attempted 114.1s / 13.1GB
Isolating the conversion step (ToSyftModel) from the tools-golang parse:
119MB 72.3s -> 1.5s, 238MB 265.8s -> 3.1s. The remaining decode time and
all of the memory are in tools-golang, which parses the document several
times over; that is a separate, upstream concern.
Output equivalence was checked by converting 10 SPDX documents (the 8 SPDX
fixtures in this repository plus the 119MB and 238MB documents) to
syft-json, spdx-json@2.3, spdx-json@2.2, spdx-tag-value and
cyclonedx-json@1.6 with both binaries and comparing the results with
timestamps and generated ids removed: all 50 pairs are identical.
Tests pin the original semantics (including the duplicate-edge corner case)
and guard the complexity with a 50k x 100k document that must complete in
under 10s; the old implementation takes about a minute on it.
Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test(dotnet): speed up pe fixture tests
the pe package took ~90s under -race, nearly all of it loading docker archive
fixtures. two things:
the three bundle fixtures published with `SelfContained=true`, so each `hello.exe`
carried a full .NET runtime (~60MB, 33MB tar). the bundle format is what those
fixtures exercise and it is identical framework-dependent, so they now publish
with `SelfContained=false` (160KB exe, 2MB tar). the race detector multiplies the
layer inflate + diffID hashing by ~15x, which is why the size mattered so much.
`fixtureFile` also loaded the image and built a squashed resolver per table row,
so `image-net8-app` was loaded four times for four files in it. resolvers are now
built once per distinct fixture from the parent test.
87s -> 28.5s under -race, and ~60MB less fixture cache.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(dotnet): read the right file type from V1 single-file bundles
V1 bundles (.NET Core 3.x) predate the `unknown` member at the head of the
manifest file type enum, so their types are all one less than V2+: deps.json is
2, runtimeconfig.json is 3. we were matching on the V2+ value, which silently
matched runtimeconfig.json instead, so every .NET Core 3.x single-file app
reported its runtimeconfig as the embedded deps.json and its dependencies went
unseen.
the existing fixture test asserted the buggy output (it expected
`runtimeOptions`, which only ever appears in a runtimeconfig). the assertions
now pin the framework moniker out of each fixture's real deps.json, so reading
the neighboring file can't pass.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test(dotnet): drop the 64MB jruby binary from the net8-app fixture
the `image-net8-app` fixture wget'd a 64MB jruby release binary at build time to
serve as "a PE that is not a .NET binary", which was 64MB of the image's 65MB.
every test that loads that image pays for it, in both the dotnet cataloger and
the pe package, and it sits in the fixture cache twice.
the same jruby binary already has its own fixture in the binary cataloger
(`image-jruby`), asserted through the same `pe.Read`, so the coverage is not
lost. the negative case here now uses `/app/dotnetapp.exe` -- the native apphost
that is already in the image -- which pairs nicely with the single file case,
same apphost with the app bundled in and CLR evidence present.
also drops a github release download from the fixture build.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix oom when testing
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* right-size CI test runners
the `r` family and `ram=32+128` were sized for the tmpfs footprint rather
than the workload. with the caches back on disk the build+test peak fits in
16GB, so move to `m`/`c` at `ram=16+32`.
- ~27% cheaper per instance-hour on spot, and a much deeper pool to schedule
against (~20 x86 types vs 7)
- `cpu=4+8` with `ram=16+32` lets price-capacity-optimized opportunistically
land an 8 vCPU box when it's cheap
- plain `volume=100gb` replaces the provisioned throughput. a 4 vCPU instance
caps near 150MB/s of EBS and gp3's free baseline is already 125MB/s, so the
override bought nothing
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
the base image is digest-pinned to a bullseye snapshot whose InRelease has
expired, so `apt-get update` now exits 100 on any cold fixture build. rather
than disable Check-Valid-Until (which just delays the next break), skip apt
entirely:
- `less` was never used and `unzip` already ships in the base image
- rezip with `jar cfM` from the base JDK instead of installing `zip`
the `-M` matters here: this fixture intentionally has no META-INF/MANIFEST.MF,
and plain `jar cf` would generate one.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`archive.ubuntu.com/ubuntu/pool/main` only keeps the current version of a
package, so the pinned `zlib1g 1:1.3.dfsg-3.1ubuntu2.1` URL started 404ing.
`curl` without `-f` wrote the error page into `zlib1g.deb` and exited 0, so
the fixture image built clean and just contained a non-deb, failing
`TestDpkgArchiveCataloger` only when the fixture cache was cold.
Now points at the launchpad `+files` URL, which is immutable. Also added `-f`
to the other fixture Dockerfiles that curl a pinned artifact so the next bit
of URL rot fails the build instead of quietly poisoning a fixture.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(golang): bound UPX decompression by what the input can justify
A UPX block's `sz_unc` was passed straight to `make([]byte, n)`, so four bytes of
attacker input spanned the full uint32 range. A ~300KB file could drive multi-GB
resident memory, which Go reports as a fatal runtime OOM that no `recover` on this
path can contain. Syft parses binaries out of arbitrary images, so that is reachable
from any scan.
Two bounds do the work, and UPX's own invariants make them exact:
- all blocks together reconstruct `p_filesize`, so a running remainder caps the sum.
Bounding only per-block would leave the total at (block count x original size), so
the remainder is the part that actually closes it
- `p_filesize` sizes the output buffer, so it is capped both absolutely and against
the size of the file on disk. The absolute cap alone left a ~60 byte header able to
claim 500MB; the ratio alone cannot work either, since LZMA encodes a run of N
equal bytes in O(log N) and a `go:embed` of 120MB of zeros legitimately packs 209x
The output buffer is also allocated on the first block that survives validation
rather than up front, so a header claiming a large size with nothing decodable behind
it costs nothing.
Alongside that, several ways the block loop could be steered off its own buffer:
- `parseELFPTLoadOffsets` bounds-checked program headers with `phStart+phentsize >
len(buf)`, which overflows for a `p_offset` near 2^64 and lets an out-of-range entry
through into a slice index. Switched to the subtraction form, and a `phentsize`
shorter than an ELF64 program header is rejected since the reads use fixed offsets
- block placement had the same overflow shape, and on failure it skipped the copy and
fell through. `outputOffset` derives from the rejected offset, so a `p_offset` at the
uint64 ceiling wrapped it to 0 and the next block landed on the reconstructed ELF
header, still returning success. Out-of-range placement now ends the block chain,
keeping the blocks already placed since those often carry `.go.buildinfo`
- the UPX 2-byte LZMA header carries `lc`/`lp` as nibbles and `pb` as three bits, so
they can hold values LZMA does not permit. Folded into the props byte they wrap
(`lc=15, lp=15, pb=7` gives 465, which truncates to 209) and the stream mis-decodes
instead of failing
- blocks decode directly into a slice of the output buffer instead of a per-block
buffer that is then copied, so one block no longer doubles peak memory
- the block count is capped: the size budget alone still allows millions of 12-byte
blocks, each spinning up an LZMA reader
Verified against the `image-small-upx` fixture, a real `upx --best --lzma` binary:
four blocks, max `sz_unc` equal to `p_blocksize` exactly, blocks summing to 0.68 of
`p_filesize`, and a 1.96x expansion against the packed file, so every bound holds with
headroom on genuine output.
Note `p_blocksize` is deliberately not used as a ceiling on `sz_unc`. It adds nothing
the remainder does not already cover, and real output sits at exactly `p_blocksize`,
so it would run with zero headroom against a value the format does not promise.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(golang): report UPX binaries we cannot unpack as unknowns
A packed Go binary we fail to unpack means its packages are silently missing from the
SBOM. `getBuildInfo` checked the decompression error for nil and discarded it, so a
rejected file surfaced only the original `buildinfo.Read` error, and that one is
deliberately silenced because it is usually just "not a Go binary".
`decompressUPX` now marks the cases worth reporting with `errUPXDecompress`: it got
past the header and the method dispatch and still could not unpack the file. Everything
else stays quiet, which matters more than it sounds. `upx` defaults to NRV2B unless
`--lzma` is passed and only LZMA is implemented here, so treating any decompression
failure as reportable would attach a golang-cataloger unknown to every packed non-Go
binary in an image. Making the reportable case opt in rather than the quiet case a
growing list also means a guard added later is silent by default.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* refactor(golang): drop the vendored xcoff parser
The only thing the golang cataloger read from it was `TargetMachine`, which is the same
two magic bytes `getGOARCHFromBin` had already matched on to dispatch there. So a full
XCOFF walk (string table, symbol table, every relocation table) ran to recover a value
that was already in hand.
`getGOARCHFromBin` reads those two bytes directly now. That is also strictly more
correct: `xcoff.NewFile` bailed with "no symbol table" when `symptr == 0`, so a stripped
XCOFF binary reported no arch at all.
Fixtures move to `testdata/xcoff/`, since they are no longer a package's own testdata.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* feat(internal/file): report a reader's real size behind one helper
Bounds get written as "does this claim exceed what the reader actually holds", and the
subtlety is that a reader's own answer cannot be trusted: an `io.SectionReader` reports
the nominal length it was built with, so one constructed with `1<<63-1` will happily
claim to hold 8EB. Confirming the last byte is readable is what separates a real size
from a nominal one.
Returns `(int64, bool)` rather than a bare size, since there are five ways to not know
and collapsing them to 0 makes every caller reinvent "0 means stop bounding".
The doc also names the wrapper trap: a type embedding `io.ReaderAt` as an interface
promotes only `ReadAt`, so it answers no size at all however large the reader beneath it
is, and a bound written against it silently does nothing.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(golang): bound every section a reader can expand, not just the name table
`CheckSectionNameTable` bounds the one section `elf.NewFile` expands while parsing, which
is enough for a caller that only parses. It is not enough for one that goes on to read
sections: those are expanded lazily, so a 260KB ELF declaring a compressed `.symtab`
drove 1.3GB of allocation through `goversion`, which opens the file with `debug/elf`
inside its own package and cannot be routed through `elfutil.NewFile`.
`CheckAllSections` is that gate, and is deliberately named as the superset so the
relationship to the narrow one is structural rather than documented.
Also exports `ErrDeclaredSizeExceeded`. A refusal here costs the SBOM a package, so
callers report it as an unknown, and matching on an error string to decide that is not
something to build a reporting policy on.
A ruleguard rule now flags `buildinfo.Read` and `version.ReadExeFromReader` outside
`scan_binary.go`. Both open `debug/elf` internally and are only bounded there by the
wrappers that gate the reader first, which the existing rule on `elf.NewFile` could not
see through.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* feat(internal/spillbuf): sparse offset-addressed buffer that spills to disk
For output that arrives out of order, at offsets the input itself declares: a
decompressor placing extents, an archive rebuilding a file from chunks. A plain `[]byte`
cannot do that without either pre-sizing to a length the input claims or growing to the
furthest offset it names, and both hand a hostile input an allocation knob.
The load-bearing property is that it reports only what it actually stored. `Size` is the
contiguous run written from offset zero and a read past it is `io.EOF`, never the zeros
an unwritten region would hand back for free. Sparse storage serving holes as real
content is what lets a write offset stand in for output nothing produced, so anything
sizing an allocation against this reader (`saferio` in `debug/elf` does exactly that)
stays bounded by work actually done.
`FirstGap` answers where the next write fits, so callers filling holes do not need the
extent bookkeeping and cannot re-derive that rule wrongly, and it only reports gaps
lying entirely inside `[0, within)`. The extent type is unexported for the same reason.
The rest of the contract worth stating: reads and writes after `Close` fail with
`os.ErrClosed` rather than looking like an empty buffer, and a write that fails leaves
the buffer unchanged.
Memory is bounded by the limit rather than by how much is written: filling 1MB and
filling 64MB cost about the same. What lands on disk past that limit is the caller's
bound to set, not this package's. Growth is geometric and capped, because sizing the
tier to exactly what each write needs is quadratic once a caller streams through it in
small chunks.
The golang cataloger is the only consumer today, and nothing else in the tree has this
shape.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(golang): rebuild UPX binaries through spillbuf and read the reconstruction everywhere
Two amplification paths were still reachable. A 128 byte input allocated 16MB by
trusting `sz_cpr` before weighing it against the bytes the file actually holds, and the
reconstruction was assembled in a `[]byte` sized from the header's declared original
size, so a 301KB input allocated 300MB.
The reconstruction is a `spillbuf.Buffer` now, which is what makes the second one go
away: blocks are placed at the offsets the file declares, nothing is pre-sized from a
claim, and the reader's length is the contiguous run actually rebuilt. That last part is
the invariant everything downstream leans on. A reader reporting the declared size would
serve the holes between placed blocks as zeros it never stored, and a 64 byte block
parked at `p_filesize-64` then made a 300KB input hand out a 64MB reader: measured, that
drove 269MB through `getBuildInfo` and returned nothing, silently.
Placement and storage are separate concerns. `upx.go` places blocks at the offsets the
file declares and asks the buffer how much was rebuilt and where the next gap is; it
knows nothing about temp files or memory limits.
The remaining bounds are ratios against the input wherever they can be, since an
absolute ceiling only ever drops a large legitimate binary. The one absolute left is on
`p_filesize`, because padding an input is nearly free, and the LZMA dictionary is paid
for in its own block's `sz_cpr`.
Interface change: `scanFile` takes a context and hands back the reconstruction for each
binary that turned out to be packed, and the caller owns it. Everything after the build
info read (crypto settings, arch, symbols, the version scan) reads it, so a packed
binary no longer reports its packages with no symbols at all.
Also narrows what gets reported. This cataloger runs on every executable in an image, so
reporting every `getBuildInfo` error attached an unknown to every corrupt ELF, odd PE and
non-Go Mach-O slice in it. Now only four gaps, the ones that cost the SBOM something
syft could have had: a packed file we could not decode (`errUPXDecompress`), a header
the size bounds refused (`errUPXSizeRefused`), a reconstruction that came up short
(`errUPXPartial`), and an ELF expansion `elfutil` declined
(`ErrDeclaredSizeExceeded`). An unimplemented UPX method stays quiet on purpose: `upx`
defaults to NRV2B and only LZMA is implemented here, so reporting it would attach an
unknown to most packed non-Go binaries in an image.
That filtering is a new pattern. No other cataloger picks which of its errors become
unknowns, and the reason this one does is that it runs on every executable in an image
rather than on files a glob already selected.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore: ignore local spec/ planning notes
`/specs` was already ignored; `/spec` is the same thing under the singular name and was
not, so working notes landed in commits.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* refactor(golang): name what the unpack hands back, not where it stores it
the scan's signatures said `*spillbuf.Buffer` all the way down, which told
every reader below the scan that it cares whether the reconstruction lives in
memory or on disk. it does not. two interfaces instead, each naming only what
its side actually uses:
- `unpackedContents` for the consumers: `ReaderAt`, `Closer`, and a `Size` that
is the contiguous run rebuilt from zero
- `blockSink` for the decoder, which additionally needs `WriterAt` and
`FirstGap` to place blocks
this costs one thing worth flagging. `Close` on a nil `*spillbuf.Buffer` was
nil-safe, so callers just deferred it; widened into an interface, a nil pointer
becomes a non-nil interface and the same call panics. `closeUnpacked` absorbs
that, and it is now the only way an owner releases contents. `readerFor`,
`seekerFor` and `closeUnpacked` are the three places that resolve the nil.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* docs(elfutil): say that CheckAllSections is the decompression-bomb gate
the doc led with "bounds every section syft can drive debug/elf into
decompressing", which describes the mechanism without ever naming what it is
defending against. a reader landing on the call site could not tell that this
is the decompression-bomb check.
state the attack instead: a compressed section header declares its own
decompressed size, debug/elf believes that number and allocates it on open, and
nothing forces it to match what the compressed bytes actually yield. the 260KB
ELF that drove 1.3GB of allocation now lives in the doc rather than only at the
one call site that happened to mention it.
comments only.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* docs(golang): move the overflow note onto the bounds test it describes
the "subtraction form" note sat above the phStart assignment with ten lines of
wrap analysis between it and the `if` it was talking about, so the subtraction
it names reads as missing. it is `hdrLen-phStart < phentsize`.
move it onto that test and name the form it is avoiding (`phStart+phentsize >
hdrLen`, which can overflow past the buffer end) so the referent is not several
paragraphs away.
comments only.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(golang): drop the internal/ README left over from xcoff
that README existed to record where the vendored xcoff package came from and
that Go keeps it internal, which is a provenance note worth having while
third-party code sat there. dropping xcoff took the reason with it, and
repurposing it to point at gotestdata just made it redundant: gotestdata has
its own README covering why it is under internal/, why it is not testdata/, and
what belongs in each.
internal/ now holds one self-documenting directory.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
bullseye went EOL and deb.debian.org dropped it, so the elf executable
fixture image stopped building and took the whole unit test job with it
(fixture generation runs before any test does). main has been red on this
since 2026-09-08.
two separate breakages, both handled here:
- the base repo moved to `archive.debian.org`, whose Release files are
permanently expired, so `Acquire::Check-Valid-Until=false` is required
- `bullseye-security` is worse off: its index is still served but every
package in the pool behind it 404s, and it has not landed on the archive
yet. dropping it is the only option right now. these are throwaway
fixture build tools, so base bullseye versions are fine.
verified by rebuilding the image and running the fixture `verify` target,
which still reports `PASS: all security features accounted for`.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(deps): pin stereoscope to the hardlink-inode-adoption branch
temporary pin to anchore/stereoscope#670 so the hardlink changes can be
exercised end to end. needs re-pinning to a release tag before merge.
pulls transitive bumps along with it: docker/cli, docker/go-connections and
gabriel-vasile/mimetype.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: drop the image resolver hardlink workaround
stereoscope now adopts a hardlink's target at index time, so a hardlinked name
arrives already described as a regular file with the target's size, mime type
and content. the resolver-side fix from #5029 sat on top of that doing the same
job a second time, gated on a file type that no longer shows up, so both
`resolveHardLinkTarget` implementations and the extra tree walk they cost on
every `FilesByPath` are gone.
`file.NewVirtualLocationFromImage` stays as-is. it is exported and syft is v1,
so it keeps working for anyone using it, it just has no callers in syft now.
this picks up `FilesByMIMEType`, which #5029 explicitly could not fix, so
hardlinked names now reach mime-driven catalogers. on images built around
multi-call binaries (busybox and friends) that is a lot more file and executable
entries than before; distro and toolchain images move by about one entry.
one shape regresses: a hardlink stereoscope could not adopt (a link name that is
absent, empty, names a directory, names another un-adopted link, or points into
a lower layer) keeps `TypeHardLink`, and syft goes back to collapsing it onto
its target's path. no mainstream builder emits the cross-layer case.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* bump stereoscope to main
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
#5188 changed `pnpmV6LockYaml.Parse` to take a `*yaml.Node` and #5175 added
this test against the old `[]byte` signature. both were green on their own
branches, so main does not compile.
the v9 and v6 cases in the same test already go through the `yamlDocument`
helper, this is just the v5 one that was missed.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
parsePinnedVersion rejected any version constraint containing "!" to
skip the "!=" exclusion operator. PEP 440 epochs use "!" as well
(e.g. "1!2.0.0"), so an exact pin like "pkg == 1!2.0.0" was treated as
unparseable and returned "". parseRequirementsTxt drops a requirement
whose version resolves to "", so the package never made it into the
SBOM at all in either the pinned or guessed path.
Check for the "!=" operator as a substring instead of the bare "!"
character, which leaves the epoch separator alone.
Signed-off-by: Sueun Cho <sueun.dev@gmail.com>
pnpm v5 lockfiles encode resolved peer dependencies as an
underscore-delimited suffix of the dependency path, e.g.
/acorn-jsx/5.3.2_acorn@8.8.0 or a hashed form like
/webpack-cli/4.10.0_fzn43tb6bdtdxy2s3aqevve2su. Only the v6+
parenthesized form was stripped, so for v5 lockfiles the suffix leaked
into reported versions and PURLs, and peer variants of the same
name+version were not deduplicated.
Strip the suffix on the v5 code path (version < 6.0) wherever a version
is extracted: top-level dependencies values, packages keys, and
per-package dependencies map values. Only values with a leading digit
are stripped so link:/file:/git specifiers are unaffected.
Fixes#5174
Signed-off-by: codeAnqiang-ma <273298913+codeAnqiang-ma@users.noreply.github.com>
Co-authored-by: codeAnqiang-ma <273298913+codeAnqiang-ma@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(binary): detect multi-arch ingress-nginx
Add v1.9.6 ARM64, ARMv7, and s390x fixtures from the published controller image and match their version marker against the nearby Go build version.
Assisted-by: OpenAI Codex
Signed-off-by: Shurong Cao <170531907+CAOShurong@users.noreply.github.com>
* fix(binary): match ingress-nginx release marker on all arches
The release is injected with `-ldflags -X`, so it lands in its own aligned,
NUL-padded data symbol. Only that padding is portable -- the surrounding bytes
are an arch-specific float constant pool, which is why the existing matchers all
anchor on incidental junk like `$a` and `S=v<y5` and only ever worked on amd64.
Matching `v<version>` followed by two NULs finds exactly one hit in every v1.9.6
binary (amd64, arm, arm64, s390x) and in all nine amd64 releases already under
test. Two NULs matter -- with one, s390x matches a vendored `v1.19.0` earlier in
the file and reports the wrong version.
This also drops the ~8KB wildcard window the marker previously needed, so the new
fixtures are ordinary 369 byte snippets instead of 8.4KB ones.
Two things in the fixture tooling had to move for that to work:
- `config.yaml` pinned the multi-arch *index* digest for all four platforms, but
extraction runs `docker create <ref>` with no `--platform`, so every entry
resolved to whichever image was pulled last. `make download` failed outright on
`linux/arm/v7`, since the registry labels that platform `armv7` and docker
normalizes it to `arm`. Now pinned per-platform, matching the redis entry.
- `write-snippet` only recognized EM_X86_64 and EM_AARCH64, so anything else
landed in a `linux-unknown-<hex>` directory and had to be placed by hand.
Verified against the real binaries with `-must-use-original-binaries`.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(binary): put non-amd64 snippets where the tests look for them
`write-snippet` used to fall back to `unknown-<hex>` for any ELF machine it
didn't recognize, so three snippets were sitting in directories nothing reads:
helm/3.12.0/linux-unknown-454d5f53333930 (hex of "EM_S390")
helm/4.1.4/linux-unknown-454d5f53333930
redis-server/7.2.5/linux-unknown-454d5f333836 (hex of "EM_386")
The test table asks for `linux-s390x` and `linux-386`, so those three cases were
quietly falling through to downloading the full binaries instead. Renamed to
match, and taught the tool about EM_386 and EM_RISCV so it stops producing dead
directories (there is an existing `linux-riscv64` snippet it also couldn't have
written).
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Shurong Cao <170531907+CAOShurong@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Shurong Cao <170531907+CAOShurong@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(javascript): read every document of a pnpm-lock.yaml stream
pnpm-lock.yaml can be a multi-document YAML stream. pnpm keeps config
dependencies and the pinned package-manager version in a leading document
and the project's dependency graph in the next one. That layout has
shipped since pnpm 11 for projects using config dependencies, and pnpm 12
records the package-manager pin by default, so two documents is now the
common case.
The cataloger called yaml.Unmarshal, which reads the first document only.
On such a lockfile the SBOM contains pnpm's own release binaries and none
of the project's dependencies. Both documents declare the same
lockfileVersion and the result is well-formed and non-empty, so nothing
signals that it is wrong.
Split the stream and parse each document, merging the results. A
single-document lockfile yields exactly one document, so its behaviour is
unchanged. A document that omits lockfileVersion falls back to the
version from the first one rather than being dropped.
Fixes#5168
Signed-off-by: hamodywe <iosapk.org@gmail.com>
* refactor(javascript): stream pnpm-lock.yaml documents instead of buffering
The multi-document fix read the whole lockfile, decoded it to a `yaml.Node`
tree, re-marshaled every document back to `[]byte` and then re-parsed each one
twice. `yaml.Node` already has a `Decode` method, so the decoder can walk the
stream straight off the reader and hand each document's node to the parser.
On a 4.8MB single-document lockfile that takes total allocations from 1069MB to
250MB, which is also below where things sat before multi-document support. It
drops the `io.ReadAll` and its `noUnboundedReads` suppression, and puts pnpm in
line with every other YAML cataloger in the tree, all of which already stream.
`pnpmLockfileParser.Parse` now takes a `*yaml.Node` rather than `[]byte`. It is
package-private with two implementations and one production call site.
Documents accumulate into a single map keyed by `name@version`, so the stream
follows one collision rule rather than two: the last entry to appear wins,
which is what already happens within a document. That replaces the separate
first-wins-with-fill-in merge pass. A conflicting integrity between documents
is now traced rather than resolved silently.
Three behaviour fixes come with it:
- an empty or comment-only leading document no longer takes down the whole
file. It decodes to a null node, not a zero-kind one, so the old guard never
fired and the stream failed with `invalid lockfile version ""`.
- a malformed document no longer discards the documents that already parsed.
It is reported through `unknown` instead, matching how the rest of the
cataloger reports partial results.
- parse errors carry the document index, and their line numbers again point at
the user's file rather than into a re-marshaled buffer.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(javascript): trim the duplicated pnpm multi-document comment
The six-line explanation of why a pnpm lockfile can hold two documents sat
verbatim in both the parser and its test. Keep it next to the code, where
someone debugging will be, and leave the test pointing at it.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: hamodywe <iosapk.org@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
orderedIDSet.add scanned the entire existing slice for every inserted ID,
so building a Collection was quadratic in the number of packages sharing an
index key. This is most visible in idsByType: package types are few, so that
index accumulates nearly every package in the SBOM into a single set, and
each Add rescans it.
Keep the ordered slice (ordering and dedup semantics are unchanged) and add
a hash index for membership checks. The index is only built once a set grows
beyond a small threshold, so the many single-element sets held by idsByName
and idsByPath do not pay for a map, while the large idsByType sets get
constant-time lookups.
BenchmarkCollectionAdd (Apple M2 Pro), packages sharing one type:
packages before after speedup
1,000 1.37 ms 0.76 ms 1.8x
10,000 100.31 ms 9.80 ms 10.2x
50,000 2,423.79 ms 34.46 ms 70.3x
The tradeoff is memory: sets past the threshold allocate an index map,
about 14% more bytes at 50,000 packages, with allocation counts unchanged.
Signed-off-by: Oleksandr Vodotiiets <61548316+avodotiiets@users.noreply.github.com>
With search-remote-licenses enabled, every module name went to the proxy,
including toolchain binaries whose main module is synthesized from the
package path, such as cmd/cgo. The proxy answers 404, and the direct
fallback then treats the path as a repository host, producing requests
like https://cmd/cgo/info/refs?service=git-upload-pack.
Reuse isStandardImportPath, already in this package, to skip the remote
search for paths whose first element carries no dot. Those are never
publishable module paths, so neither a proxy nor a repository can resolve
them.
Fixes#3149
Signed-off-by: Luan Taraschi <130802253+luantaraschi@users.noreply.github.com>
* fix(elf): bound compressed ELF section reads
`debug/elf` takes a section's decompressed size from that section's own
compression header, and a highly compressible stream really does deliver the
bytes that header promises, so `internal/saferio` does not help: it faithfully
allocates every one of them. A 2MB input file drives `elf.NewFile` to allocate
over 10GB and return no error, which is a fatal OOM rather than a recoverable
panic.
Which sections get read is not up to the caller. `elf.NewFile` always reads the
section-name string table, and `File.Symbols` reads `.symtab` plus whatever
section its `Link` field points at, so being selective about sections is not
enough to avoid it.
New `elfutil.NewFile` is a drop-in for `elf.NewFile` that rejects a declared
decompressed size over 128MB. Every production call site goes through it, and a
ruleguard rule keeps the next one from going direct.
The check runs in two parts, since `debug/elf` expands sections at two different
times. The section-name string table is the only one `elf.NewFile` expands
itself, so it is checked against the raw bytes before the call; everything else
is expanded lazily by `(*Section).Open` and is checked after the parse, where
names, types and decompressed sizes are already resolved.
Only the sections syft can actually reach are bounded, which keeps the guard
from costing real binaries. DWARF is excluded since nothing calls `File.DWARF`,
so a large compressed `.debug_info` no longer skips the whole file, and sections
`debug/elf` will not decompress anyway (`SHF_ALLOC`, `SHT_NOBITS`) are left
alone. The legacy `.zdebug` form is matched on the section name the way
`debug/elf` gates it rather than on the `ZLIB` magic, so an ordinary section
starting with those four bytes is not mistaken for a compressed one.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(elf): gate debug/buildinfo behind the compressed-section check
`debug/buildinfo.Read` opens ELF files with `debug/elf` itself, and `elf.NewFile`
expands the section-name string table as it parses, so the golang cataloger was
still reachable by the same bomb `elfutil` exists to stop. A 261KB fixture drove
1.4GB of allocation through `buildinfo.Read` and returned no error.
`elfutil.CheckSectionNameTable` is now exported for that case: callers that cannot
use `NewFile` because the `debug/elf` call is made for them inside another package.
Both `buildinfo.Read` call sites go through it, including the UPX-decompressed one.
Also corrects claims that did not hold up:
- the package doc's 2MB-to-10GB figure is not reachable with zlib (~1000:1), so it
now carries the measured 510KB-to-2.6GB, and names zstd's 32767:1 since that is
what makes the small inputs possible
- `.go.buildinfo` was listed as a hot-path section elfutil covers, but it is read
through `debug/buildinfo` and never touches `Section.Data`
- the graalvm comment claimed routing size rejections away from `*elf.FormatError`
improved reporting; both branches are skipped by the caller and only the
FormatError branch logs, so it did the opposite
- `sharedLibraries` logged short and truncated files as real ELF failures, since
`debug/elf` returns a bare `io.EOF` rather than an `*elf.FormatError` for those
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* added test comments around the negative cases
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* additional tests
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* better decomposition and comments
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* use a less brittle constant for error detection
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`readerAtAdapter.ReadAt` seeks and then issues a single `Read`, which breaks the
`io.ReaderAt` contract in both directions against the squashfs reader it exists
to wrap:
- `squashfs.File.Read` copies against the decompressed block length but advances
its block cursor by the nominal block size, so a block that decompresses short
silently stops copying and returns fewer bytes with a nil error. `ReadAt`
forbids that, and callers rely on it: anything decoding a fixed-size structure
off the result gets zero padding it has no way to detect and parses it as real
data. The GraalVM PE export table and the UPX block reader both size a buffer
from a header field and then ignore `n` entirely, so a crafted image drives
them straight through the padding.
- a read landing exactly on the end of the file returns a *full* buffer paired
with `io.EOF`. `bytes.Reader.ReadAt` returns nil there, and the callers that
treat any error as fatal were written against that, so squashfs-resident
binaries sized near a read boundary were being skipped outright.
`io.ReadFull` normalizes both: it fills the buffer across short reads, and it
clears the error once the buffer is full. A genuinely short tail is reported as
`io.EOF`, which is what `ReadAt` implementations return at the end of a file, and
what the buffering branch of `GetUnionReader` already returns.
Affects squashfs-backed sources (snaps), so in practice the binary catalogers
reading structure out of executables.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
jackson-mapper-asl, jackson-core-asl, and their sibling artifacts
(jackson-jaxrs, jackson-xc, jackson-smile) predate the convention of
embedding META-INF/maven/.../pom.properties in the jar (they were
built with Ant before ~2014). With no POM metadata to read, syft's
groupIDFromJavaMetadata falls through to using the artifact name
itself as the group ID, e.g.
pkg:maven/jackson-mapper-asl/jackson-mapper-asl@1.9.13
instead of the correct
pkg:maven/org.codehaus.jackson/jackson-mapper-asl@1.9.13
(confirmed against the published POM on Maven Central for all five
artifacts). Because the generated purl's namespace doesn't match the
vulnerability database's namespace for these packages, this causes
false negatives in downstream scanning (e.g. Grype cannot match known
CVEs such as CVE-2019-10202 against jackson-mapper-asl).
Add the five artifacts to DefaultArtifactIDToGroupID, the same known-
package-list fallback already used for other jars with incomplete
metadata (e.g. the existing ant-*, spring-ldap* entries).
Fixes#4598
Signed-off-by: ankit090701 <ankitanku090701@gmail.com>
* feat(golang): add extended-stdlib scope and include patterns for symbol capture
`golang.capture-symbols` decides how much symbol data lands in the SBOM for grype's reachability analysis. It's `none`, `stdlib`, or `all` today, and the useful middle is missing: `stdlib` stops at the standard library, `all` multiplies SBOM size.
A new `extended-stdlib` configurable covers stdlib plus everything under `golang.org/x/`:
```yaml
golang:
capture-symbols: extended-stdlib
```
Also, a new `capture-symbols-include` configurable for modules that are noisy in your binaries but not everyone's. It's unioned with whatever the scope selects, so it only ever widens:
```yaml
golang:
capture-symbols: extended-stdlib
capture-symbols-include:
- github.com/klauspost/**
```
Patterns are standard doublestar globs, which matters because module paths carry `/v2`-style suffixes:
```yaml
golang:
capture-symbols-include:
- github.com/klauspost/* # compress, but not compress/v2
- github.com/klauspost/** # both
- k8s.io/client-go # exact match only
```
Ordering is `none` < `stdlib` < `extended-stdlib` < `all`. The existing three values
and the `none` default are unchanged, and the include list is inert under `none`.
Presets compile into glob lists internally, so a single matcher answers "does this
module get symbols" instead of a preset branch sitting next to a separate glob branch.
An unrecognized `capture-symbols` value still falls back to `none`, but warns now
instead of doing it silently. A malformed include pattern warns and gets skipped.
One thing worth a look beyond the feature: the `Symbols` field description in the JSON
schema was wrong after this (it claimed only `all` and `stdlib` populate anything), and
that description lives in the already-published `16.1.10`. Rather than bump a version for
a sentence, `16.1.10` is amended in place and `schema/json/README.md` grows an explicit
exception for description-only changes: descriptions only, no shape change of any kind,
`$id` unchanged. Anything else still needs a bump. Happy to split that into its own PR if
you'd rather review the policy separately.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* refactor(golang): rename capture-symbols-include to capture-symbols-modules
The key's entries are go module paths, and `-include` sitting next to `capture-symbols` reads as plausibly taking symbol or package names instead. Those spellings parse and match nothing, which is quieter than the confusion `-include` was picked to avoid, so the name now says what the list holds.
`golang.CatalogerConfig.CaptureSymbolsModules` and `WithCaptureSymbolsModules` rename with it. Nothing behavioral changes; the key is new in this PR so there is no compatibility surface.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* feat(golang): match capture-symbols-modules across major version suffixes
`github.com/anchore/*` covered `github.com/anchore/syft` and silently stopped covering it the day it became `github.com/anchore/syft/v2`. The config keeps parsing, nothing warns, and symbols quietly go missing from the SBOM. Exact paths had the same hole: `github.com/klauspost/compress` did not cover `compress/v2` either, so no spelling short of `**` survived a major bump.
A major version suffix is part of a module's path but not part of its identity, so patterns are now matched against the module path both with and without it, using `module.SplitPathVersion` from `golang.org/x/mod` (already a direct dep, already used in this package for `PseudoVersion`).
```yaml
golang:
capture-symbols-modules:
- github.com/klauspost/* # compress and compress/v2
- github.com/klauspost/compress # same module at every major version
- github.com/klauspost/compress/v2 # v2 alone
```
Only a trailing suffix is a version, which is Go's own rule. In `github.com/anchore/syft/v2/thing` the `v2` is an ordinary path element naming a major subdirectory a nested module lives in, so it stays literal and `github.com/anchore/**/thing` is how you reach it. `/v0` and `/v1` are not valid suffixes and are left alone.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
The two "RACE is unset" cases only skipped the t.Setenv call, so they
inherited whatever RACE was in the environment. Running `make test` with
RACE=false exported job-wide flipped the CI-default case and failed.
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
with a cold fixture cache the integration suite builds and saves 18 docker
images across 36 sequential tests, which walks past `go test`'s default 10m
timeout and takes the fixture cache rebuild down with it. that suite now runs
`go test` directly with `-timeout=30m` (gotest.Tasks() has no timeout option),
plus `-count=1` since the built fixtures are the side effect we're actually
after and a test cache hit would skip producing them.
also adds `RACE` as one switch for the race detector across every suite:
- `RACE=false make test` drops `-race` from unit + integration and skips the
race smoke, worth doing on a cache rebuild where the wall clock is all
docker builds anyway
- `RACE=true` forces it on locally
- unset behaves as before: on in CI, off locally and on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* report each hardlink as its own file when scanning images
image scans previously collapsed a set of hardlinks onto a single file, so only
one path per inode showed up in results. dir scans report every hardlink path,
which made image vs dir SBOMs of the same filesystem diverge (and produce
different SPDX `packageVerificationCode` values for packages that own hardlinked
files).
now both image resolvers (squash and all-layers) surface each hardlink at its
own path as a regular file bound to the target's content, matching dir scans.
user-facing impact:
- SBOMs for images containing hardlinks will list more `file` entries
- SPDX `packageVerificationCode` values change for affected packages, now
matching the equivalent `dir:` scan
- adds `file.NewVirtualLocationFromImage` to the public API
fixes#5019
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix busybox test assertion
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
parseErlangString advances past a backslash escape and then checks
len(data) >= *i before reading the escaped byte. That condition is
almost always true (it only turns false once *i runs off the end),
so the intended out-of-range guard fires on the very first escape
character it sees instead of only at EOF. Any rebar.lock or OTP
resource file containing a backslash in a quoted string (a Windows
git path, an escaped quote, anything) fails to parse and the whole
file, and every package in it, gets dropped.
Flip the comparison to *i >= len(data) so the guard only trips when
the escape is genuinely truncated, and add a regression test for a
string with an escaped quote.
Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
* Prevent duplicate Go packages after source replacement resolution
Source analysis already applies module replacements through go/packages. Avoid synthesizing the same replacement again from go.mod while retaining fallback synthesis for modules that source analysis did not resolve.
Constraint: Preserve unimported and local-path replacement cataloging.
Rejected: Deduplicate only during final assembly | That retains redundant license lookup and ambiguous metadata ownership.
Confidence: high
Scope-risk: narrow
Directive: Keep go.mod fallback packages limited to modules absent from source analysis.
Tested: Focused replacement regression, related Go module parser tests, go vet, gofmt, and diff checks.
Not-tested: Docker-backed full cataloger fixtures; local root storage was exhausted by image generation.
Signed-off-by: ychampion <ychampion@users.noreply.github.com>
* Keep replacement fixtures with the Go module test data
Constraint: The maintainer reserves internal/gotestdata for fixtures that need special Go tooling discovery.
Rejected: Leave this fixture in gotestdata | The regression opens its module explicitly and does not need the special location.
Confidence: high
Scope-risk: narrow
Directive: Use internal/gotestdata only when a fixture must avoid Go testdata discovery rules.
Tested: replacement regression repeated 10 times; Go module parser table; go vet for the Go cataloger; gofmt; diff checks.
Not-tested: Full cataloger package; three unrelated parser fixtures fail identically on exact prior head in this environment.
Signed-off-by: ychampion <ychampion@users.noreply.github.com>
---------
Signed-off-by: ychampion <ychampion@users.noreply.github.com>
Co-authored-by: ychampion <ychampion@users.noreply.github.com>
Closes#4587
Adds a single info-level log line at the end of every `syft scan` run that reports the total scan wall-clock time, e.g.:
[0003] INFO scan completed in 2.017s
Per maintainer guidance on the issue, the line is logged at INFO so it surfaces with `-v` (alongside the existing per-cataloger timing) and stays out of the default TUI / stdout. Useful for users who leave long scans running and want the overall time at a glance, without having to wrap the invocation in `time`.
Signed-off-by: Chris (ChrisJr404) <11917633+ChrisJr404@users.noreply.github.com>
Co-authored-by: Chris (ChrisJr404) <11917633+ChrisJr404@users.noreply.github.com>
The CycloneDX decoder was filtering on a hard-coded set of component
types (application/framework/library/machine-learning-model). Anything
else — including "firmware" — fell through silently, so a BOM
describing u-boot or other firmware ended up with no packages and
grype reported no matches (anchore/grype#2537).
@kzantow confirmed in the issue thread that firmware should be
included and gave this section as the spot to update.
Add ComponentTypeFirmware to the case list and a regression test that
constructs a single-firmware-component BOM and asserts the decoded
SBOM contains the package.
Closesanchore/grype#2537
Signed-off-by: Chris (ChrisJr404) <11917633+ChrisJr404@users.noreply.github.com>
Co-authored-by: Chris (ChrisJr404) <11917633+ChrisJr404@users.noreply.github.com>
Update the binary classifier cataloger developer README to match the
current workflow:
- Testing section used the non-existent flag -must-use-full-binaries
against .../testdata/...; the real flag is -must-use-original-binaries
and Test_Cataloger_PositiveCases lives in the binary package. Fix the
command and add a single-fixture variant.
- Point add-snippet references at the make target and note the (Y/n/q)
prompt requires y to write the snippet.
- Fix the test file name (classifier_cataloger_test.go) and document the
go generate ./internal/capabilities step for new classifiers.
Fixes#4510
Signed-off-by: alliasgher <alliasgher123@gmail.com>
GHC boot-lib constraints (any.base installed) have no ' ==', so fields[1] panicked with index out of range and the recover dropped all Haskell packages. Skip constraint lines without a version.
Signed-off-by: Synvoya <16019863+Synvoya@users.noreply.github.com>
Co-authored-by: Synvoya <16019863+Synvoya@users.noreply.github.com>
The hand-written rockspec parser reads a byte past the end of the buffer
in two spots when a comment runs right up to the end of the file.
In parseRockspecBlock, when a block starts with a leading comment that
consumes the rest of the file, the SkipWhitespace afterward leaves the
index at len(data) and the following `c = data[*i]` reads out of range.
In parseComment, `data[*i]` is read after the index is advanced to check
for a CR/LF pair, so a bare carriage return as the last byte reads past
the end.
Both cases show up with a rockspec whose final line is a comment ending
in a lone \r with no trailing newline. That is malformed but harmless
input, and the panic aborts the whole Lua cataloger, so every valid Lua
package in the same scan gets dropped. Guard both reads with a length
check and return cleanly at EOF. Added table cases covering a
comment-only file and a trailing comment, both ending in a bare CR.
Signed-off-by: arpitjain099 <arpitjain099@gmail.com>
deno.lock npm keys append resolved peer dependencies after the version,
separated by "_" (e.g. "typedoc@0.28.19_typescript@6.0.3" or the scoped
"@scope/name@1.2.3_@scope+peer@4.5.6"). parseDenoNpmNameVersion split on the
last "@", capturing a peer's "@" and producing a garbage package name plus
the peer's version (and a corrupt PURL that breaks vulnerability matching).
Strip the peer suffix (the first "_" after the version "@") before splitting
name@version; a semver version never contains "_".
Signed-off-by: Synvoya <16019863+Synvoya@users.noreply.github.com>
Co-authored-by: Synvoya <16019863+Synvoya@users.noreply.github.com>
* swift: emit canonical purls without .git suffix or repeated name
Reported in anchore/syft#3961: the purl syft emits for Swift packages
parsed out of Package.resolved keeps the repository URLs .git suffix
and then re-appends the package name, producing
pkg:swift/github.com/apple/swift-nio-ssl.git/swift-nio-ssl@2.0.0
NVD / Grype cannot match that purl against the known
swift-nio-ssl@2.0.0 CVE (GHSA-frg3-gpcx-968f), so every Swift SBOM
produced by syft silently loses vulnerability coverage. cdxgen and
the wider purl ecosystem use the shorter form
pkg:swift/github.com/apple/swift-nio-ssl@2.0.0
which Grype does match.
Replace the ad-hoc strings.Replace with swiftNamespaceFromSourceURL,
which trims the common URL schemes, strips the ".git" suffix, and
drops a trailing /<name> segment so the namespace is only the
organisation path (e.g. github.com/apple). Existing test expectations
are updated to the new purl shape.
Fixes#3961
Signed-off-by: Sai Asish Y <say.apm35@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* Revert unrelated consul binary classifier changes
The swift purl commit (ff5ffc37) accidentally bundled removal of the
consul GitDescribe and NUL-wrapped version matchers plus the 1.12.9 and
1.7.14 fixtures/test cases. That drops version detection for consul
binaries not carrying the CONSUL_VERSION string. Restore the binary
cataloger to its pre-commit state so this branch is swift-only.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* swift: strip repeated name segment case-insensitively
Package.resolved identities are lowercased while the repo URL path may be
mixed-case (e.g. github.com/Apple/Swift-NIO vs identity swift-nio). The
case-sensitive TrimSuffix left the repo segment in the namespace for such
repos, reintroducing the duplicated-name purl. Compare the trailing segment
with EqualFold and add unit coverage for swiftPackageManagerPackageURL.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Sai Asish Y <say.apm35@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
Mirrors the existing RHEL `ExtendedSupport` detector for Ubuntu so grype can
auto-enable its ESM distro channel on Pro/ESM-enabled scans.
`findUbuntuFeatures` sets `linux.Release.ExtendedSupport` when the scanned OS is
Ubuntu and any of these hold:
- an uncommented apt source (classic `.list` or DEB822 `.sources`, or the
ubuntu-advantage auth entry) referencing `esm.ubuntu.com`
- an enabled `esm-infra`/`esm-apps` service in the ubuntu-advantage
`status.json` cache
- an installed dpkg package whose version carries an `[~+]esm` pocket suffix
(the usual in-image signal, since containers are rarely Pro-attached)
The package fallback reads `s.Artifacts.Packages` from the `DetectFeatures`
SBOM callback, which runs after package cataloging, so no new plumbing was
needed. Conservative by design: plain Ubuntu and disabled/commented ESM
evidence stay false.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Debian's /etc/os-release declares only the major version in VERSION_ID (e.g. "13"), while /etc/debian_version carries the point release (e.g. "13.5"). The cataloger previously skipped debian_version whenever os-release already had a Version/VersionID, so point releases were lost.
Signed-off-by: Osama Elhindawy <25825931+OsamaSE@users.noreply.github.com>
The [requires] section parser set inRequirements=false when it
encountered a comment line (#) inside the section, because the comment
check was OR'd with the section-change check. Any dependency listed
after a comment was silently skipped.
Skip blank lines and comments before evaluating section state so they
never toggle inRequirements. Add a regression test fixture with a
comment inside [requires].
Fixes#5017
Signed-off-by: jeff <jfjrh2014@gmail.com>
parseMixLock read the version from a hex-only token position and hardcoded a pkg:hex PURL, so git/path deps got a URL-fragment version and a false hex.pm PURL (false CVE matches). Branch on the source atom: git uses the commit SHA, path an empty version, and pkg:hex PURLs are emitted only for hex deps.
Signed-off-by: Synvoya <16019863+Synvoya@users.noreply.github.com>
Co-authored-by: Synvoya <16019863+Synvoya@users.noreply.github.com>
the plain version pin drifted out from under us: debian mirrors only keep
the latest point release, so 8.2.31 became uninstallable once 8.2.32 shipped
and apt pulled the unpinned transitive php8.2-cli at 8.2.32. pointing apt at
snapshot.debian.org keeps the exact point release installable so the fixture
(and its test expectations) stay reproducible.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(template): expose sprig date functions in Go templates
Replace HermeticTxtFuncMap with TxtFuncMap to expose date/time
functions (now, date, dateInZone, etc.) while still excluding
security-sensitive env/expandenv functions.
Users can now use date functions in templates, e.g.:
{{ now | unixEpoch }}
{{ now | date "2006-01-02" }}
Fixes#2372
Signed-off-by: Sputnik-MAC <sputnik.mac.001@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* invert to add functions to the hermetic set, not the other way around
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Sputnik-MAC <sputnik.mac.001@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* ruby/gemspec: resolve simple #{s.name}/#{s.version} interpolation
Reported in anchore/syft#4720: scanning projects that depend on gems
like formatador leaks literal Ruby interpolation into the emitted
SBOM, e.g.
"externalReferences": [
{ "url": "https://github.com/geemus/#{s.name}", "type": "website" }
]
because formatador.gemspec uses
s.homepage = "https://github.com/geemus/#{s.name}"
and parseGemSpecEntries reads the file as plain text instead of
evaluating it. The interpolation leaks through the captured homepage
field and on into any externalReferences entry the cataloger produces.
Dependency Track then rejects the whole BOM because '{' and '}' are
not valid IRI-reference characters (RFC 3987).
Add a post-parse pass that substitutes the common interpolation forms
(#{s.name}, #{gem.name}, #{name}, and the matching #{*.version}
variants) in captured string fields using values already parsed from
the same gemspec. Anything still containing '#{' after best-effort
substitution is an unresolvable Ruby expression, and for URL-like
fields (currently just homepage) we drop the field entirely so the
SBOM is always schema-valid; callers would rather miss a homepage URL
than emit one that breaks downstream tools.
Adds testdata/formatador.gemspec, a minimal real-world gemspec using
the #{s.name} pattern, plus a new parser test asserting that the
homepage field comes out fully resolved.
Fixes#4720
Signed-off-by: Sai Asish Y <say.apm35@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* improve test cases
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Sai Asish Y <say.apm35@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
adds dependency-of relationships between elixir locked packages, matching how other
ecosystem catalogers (alpine, arch, debian, redhat, python) express the
dependency graph via the shared dependency.Processor/Specifier mechanism.
Signed-off-by: Chris Greeno <cgreeno@gmail.com>
Borland Package Library (.bpl) files are standard Windows PE/DLL files
used in Delphi and C++Builder ecosystems. This adds the .bpl glob
pattern to the PE file discovery so these files are cataloged alongside
.dll and .exe files.
This allows linting the imports to be grouped correctly, and provides
an auto-fix (`golangci-lint run --fix`).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
* fix(dpkg): extract License field for opkg/ipkg entries
opkg and ipkg use the dpkg cataloger but declare the package License
inline in the status DB (unlike Debian dpkg, where licenses live in
copyright files). The cataloger silently dropped the License field at
mapstructure decode time, so all opkg-managed packages reported empty
licenses.
This adds the field to the intermediate decode struct and the public
DpkgDBEntry, and populates licenses in newDpkgPackage using the alpine
cataloger's pattern: try license.ParseExpression first to keep valid
SPDX expressions whole, fall back to whitespace splitting for
space-separated lists.
Standard Debian dpkg status files never carry a License field per
Debian policy, so the new path is a no-op for them; the existing
copyright-file lookup in addLicenses is unaffected.
Closes#4940
Signed-off-by: David Dashti <47575784+Dashtid@users.noreply.github.com>
* remove license from dpkg metadata struct
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* restore format snapshot files
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* add additional tests
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: David Dashti <47575784+Dashtid@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
BPL (Borland Package Library) files are standard PE/DLL format used by
Delphi and C++Builder. Adding the extension to the glob list so syft
picks them up during directory scans without users needing to rename
to .dll first.
---------
Signed-off-by: jfjrh2014 <jfjrh2014@gmail.com>
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
Co-authored-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
* fix: detect compressed kernel modules (.ko.gz, .ko.xz, .ko.zst)
The linux-kernel-cataloger only matched plain *.ko files, missing
compressed modules produced when CONFIG_MODULE_COMPRESS is enabled
(common on Debian 13 / Ubuntu 24.04+). This resulted in near-zero
module packages being reported for such filesystems.
Changes:
- Add *.ko.gz, *.ko.xz, *.ko.zst glob patterns to both the cataloger
and capabilities.yaml so the file resolver picks up compressed modules
- Add decompressedModuleReader() which detects the extension and
transparently decompresses via compress/gzip, ulikunitz/xz, or
klauspost/compress/zstd before handing the ELF bytes to the existing
parseLinuxKernelModuleMetadata parser
- Promote github.com/klauspost/compress from indirect to direct dependency
- Add unit tests covering all three compression formats plus the
uncompressed baseline, using a programmatically generated minimal ELF
Fixes#4721
Signed-off-by: Will Bates <william.bates11@outlook.com>
* address reading archives into memory
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Will Bates <william.bates11@outlook.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Will Bates <william.bates11@outlook.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* Rework release workflow to canonical shape
Replace the custom quality-gate job with the reusable check-version-available
and check-gate workflows from anchore/workflows. Remove the phase
workflow_dispatch input; the install-script-only path is now a standalone
workflow (release-install-script.yaml) that can be triggered independently.
- add version-available and check-gate jobs using pinned anchore/workflows SHA
- remove phase input and quality-gate job
- release job now needs [check-gate, version-available]
- release-install-script job no longer conditionally skips based on phase
- add release-install-script.yaml for standalone install script runs
- set permissions: {} at workflow level (contents pushed to release job)
- add concurrency: group: release
Signed-off-by: wagoodman <wagoodman@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* Tighten workflow-level permissions to {}
Change top-level permissions from contents: read to {} in validations.yaml
and validate-github-actions.yaml, pushing the needed contents: read down
to each job that performs a checkout.
Signed-off-by: wagoodman <wagoodman@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* keep install script phase, remove workflow
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* remove schema detection workflow
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: wagoodman <wagoodman@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
The elixir-binary and elixir-library classifiers' regexes only matched
the bare semver triplet (and a single sub-segment for the library), so
release-candidate elixir images were either missed entirely or had
their version truncated:
$ syft -q elixir:1.12.0-rc | grep elixir # nothing
$ syft -q elixir:1.13.0-rc.0 | grep elixir
elixir 1.13.0 binary # truncated, "-rc.0" lost
Extend the version capture group to optionally include
"-<a-z0-9>+(\\.<digits>)?" so "1.12.0-rc.1", "1.13.0-rc.0", etc. match
exactly as the elixir.app and the binary's ELIXIR_VERSION line have
them.
Add a logical fixture under testdata/classifiers/snippets/elixir/
1.12.0-rc.1/linux-amd64 (cloned from the existing 1.19.1 fixture with
just the version strings changed) and register it in
Test_Cataloger_PositiveCases.
Closes#4819
Signed-off-by: Chris (ChrisJr404) <11917633+ChrisJr404@users.noreply.github.com>
Co-authored-by: Chris (ChrisJr404) <11917633+ChrisJr404@users.noreply.github.com>
A trailing slash on --exclude (e.g. './lib/') is dropped during pattern
normalization but doublestar.Match still requires an exact string match,
so the resulting pattern silently matches nothing and the directory is
not excluded. Strip a trailing slash so './lib/' and './lib' behave the
same.
Fixes#4839
Signed-off-by: ChrisJr404 <chris@hacknow.com>
* fix(debian): only parse machine-readable copyright files with Format header
Only parse debian/copyright files as machine-readable DEP-5 format when
they contain the mandatory Format header field pointing to the copyright
specification URI. Files without this header are free-form text and
should not have License: regex patterns applied to them, which previously
produced nonsensical results like "#", "Permission", "This", "see" for
non-machine-readable files.
The fallback license classifier in the debian cataloger will handle
non-machine-readable files by doing full-text license identification.
Closes#4708
Signed-off-by: Bahtya <bahtya@users.noreply.github.com>
Signed-off-by: Bahtya <bahtayr@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* decompose parseLicensesFromCopyright to address linting issues
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Bahtya <bahtya@users.noreply.github.com>
Signed-off-by: Bahtya <bahtayr@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Bahtya <bahtayr@gmail.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
Empty or whitespace-only .rockspec files cause parseRockspecBlock to
panic with "index out of range" because the existing end-of-data guard
requires len(out) > 0 before returning the "unexpected end of block"
error, letting the bare data[*i] access on the next line crash.
Split the guard so that:
- partial content at end of data still returns the existing error
- empty data (or whitespace-only) returns an empty block cleanly
Closes#4824.
Signed-off-by: Akihiko Komada <aki1770@gmail.com>
Ensures the correct distro id for AlmaLinux and Rocky Linux when falling
back to parsing distro information from the redhat-release file. Also
sets the idlike to `rhel` for these instances as that is necessary to
ensure correct vulnerability data matching.
Signed-off-by: Weston Steimel <author@code.w.steimel.me.uk>
The JRuby project migrated their downloads from S3 to GitHub Releases,
causing the old S3 URLs to return HTTP 403 Forbidden and breaking test
fixture image builds.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Passing '%q' to format strings for integer types is a go vet error in
recent go versions, and likely a bug.
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
The pattern 'type Alias *Document' does not strip methods under
encoding/json/v2 (GOEXPERIMENT=jsonv2), causing UnmarshalJSON to call
itself infinitely until the goroutine stack overflows (1GB limit).
Change to 'type Alias Document' with (*Alias)(d) cast — the standard
Go pattern that works correctly with both encoding/json v1 and v2.
Adds a regression test that uses debug.SetMaxStack to shrink the
goroutine stack limit to 8MB, making the overflow happen in milliseconds
rather than minutes if the recursion is reintroduced.
Ref: https://github.com/golang/go/issues/75361
Signed-off-by: Benjamin Grandfond <benjamin.grandfond@docker.com>
* chore: centralize temp files and prefer streaming IO
Catalogers that create temp files ad-hoc can easily forget cleanup,
leaking files on disk. Similarly, io.ReadAll is convenient but risks
OOM on large or malicious inputs.
Introduce internal/tmpdir to manage all cataloger temp storage under
a single root directory with automatic cleanup. Prefer streaming
parsers (bufio.Scanner, json/yaml.NewDecoder, io.LimitReader) over
buffering entire inputs into memory. Add ruleguard rules to enforce
both practices going forward.
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* chore: go back to old release parsing
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* simplify to limit reader in version check
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* chore: regex change postponed
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* simplify supplement release to limitreader
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
---------
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
BUILDPLATFORM is automatically set to the host's platform in new Docker,
so having it defined as an arg results in it being overridden by this
automatic value. Since it was always assigned to a literal string in the
test files, just use that string.
Additionally, image platform is better pulled from the manifest, not the
image config, in containerd store, so try that first.
Additionally, python3 is on PATH on new macs by default, but not python.
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* chore(deps): update tools to latest versions
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* chore(lint): fix errors in new golangci-lint
Two fixes:
First, replace sb.WriteString(fmt.Sprintf(...)) with fmt.Fprintf(&sb, ...)
Second, suppress errors where we read from the local file system at a
user provided path. This is a CLI tool, and reads from user provided
paths on the local file system by design.
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
Co-authored-by: spiffcs <32073428+spiffcs@users.noreply.github.com>
Co-authored-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* migrate fixtures to testdata
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: correct broken symlinks after testdata migration
The migration from test-fixtures to testdata broke several symlinks:
- elf-test-fixtures symlinks pointed to old test-fixtures paths
- elf-test-fixtures needed to be renamed to elf-testdata
- image-pkg-coverage symlink pointed to test-fixtures instead of testdata
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: handle missing classifiers/bin directory in Makefile
The clean-fingerprint target was failing when classifiers/bin doesn't
exist (e.g., on fresh clone without downloaded binaries).
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: add gitignore negation for jar/zip fixtures in test/cli
The jar and zip files in test/cli/testdata/image-unknowns were being
gitignored by the root .gitignore patterns. This caused them to be
untracked and not included when building docker images in CI, resulting
in Test_Unknowns failures since the test expects errors from corrupt
archive files that weren't present.
Add a .gitignore in test/cli/testdata to negate the exclusions for
these specific test fixture files.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* switch fixture cache to v2
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: update expected versions for rebuilt fixtures
Update test expectations for packages that have been updated in
upstream repositories when docker images are rebuilt:
- glibc: 2.42-r4 → 2.43-r1 (wolfi)
- php: 8.2.29 → 8.2.30 (ubuntu/apache)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* upgrade go
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: add go-shlex dependency for testdata manager tool
The manager tool in syft/pkg/cataloger/binary/testdata/ imports
go-shlex, but since it's in a testdata directory, Go doesn't track
its dependencies. This caused CI failures when go.mod didn't
explicitly list the dependency.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* refactor: move binary classifier manager to internal/
Move the manager tool from testdata/manager to internal/manager so
that Go properly tracks its dependencies. Code in testdata directories
is ignored by Go for dependency tracking, which caused CI failures
when go.mod didn't explicitly list transitive dependencies.
This is a cleaner solution than manually adding dependencies to go.mod
for code that happens to live in testdata.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: add gitignore negations for test fixtures blocked by root patterns
Multiple test fixtures were being blocked by root-level gitignore patterns
like bin/, *.jar, *.tar, and *.exe. This adds targeted .gitignore files with
negation patterns to allow these specific test fixtures to be tracked:
- syft/linux/testdata/os/busybox/bin/busybox (blocked by bin/)
- syft/pkg/cataloger/java/testdata/corrupt/example.{jar,tar} (blocked by *.jar, *.tar)
- syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/**/bin/go (blocked by bin/)
- syft/pkg/cataloger/bitnami/testdata/no-rel/.../bin/redis-server (blocked by bin/)
Also updates the bitnami test expectation to include the newly required
.gitignore files in the test fixture.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: update glibc version expectation (2.43-r1 -> 2.43-r2)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* add capability drift check as unit step
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* dont clear test observations before drift detection
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* bump stereoscope commit to main
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(deb and snaps): prevent excess reads
Previously, Syft could allocate excess memory or tempfile space if there
were highly compressed objects in deb archives, or at paths where the
kernel changelog was expected by the snap cataloger. Use io.LimitReaders
for extracting parts of deb archives, and refactor the snap cataloger's
reading of the kernel changelog to use a streaming parsing, eliminating
the possibility of excess allocation.
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* fix: always cleanup temp file from file source
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* use streaming strategy for deb archives
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
---------
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
Otherwise sometimes the test that reads will run during the test that
writes and fail because the yaml file is in a partially written state.
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* fix(cpe-generation): set start and end date
Previously, the update job was silently failing because the NVD API
returns a 404 with no body if a start date is specified but not an end
date. Further, the API returns an error if more than 120 days are in
range of the start and end date.
Update the API client to:
1. Return a non-nil error on http 404
2. Chunk the date range into 120 day chunks
3. Pass start and end date to avoid errors.
Also add more tolerant timestamp parsing since the previous update job
would fail with timestamp format errors.
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* refactor(cpe-generator): remove callbacks
Previously, this job had callbacks that were there to make sure that
incremental progress could be written to disk. However, incremental
progress was not being written to disk, and there were issues related to
the callbacks like double logging. Therefore, just remove the callbacks
and do simple imperative code to page through the API results.
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
---------
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
NVD uses "aomedia" as the vendor for libavif CVEs. This change adds
libavif to the APK package CPE candidate additions with "aomedia" as
an additional vendor, enabling Syft/Grype to match CVEs like
CVE-2025-48174 and CVE-2025-48175.
Signed-off-by: Peter Bücker <peter.buecker@gmail.com>
A CI failure was observed where a generated file was only partly written
when the CI job immediately tried to read it. Put in an fs.Sync call to
eliminate this flakiness.
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* fixed#4316 go mod with ver purl
Signed-off-by: Rez Moss <hi@rezmoss.com>
* go mod purl fixed, added func to handle go.mod
Signed-off-by: Rez Moss <hi@rezmoss.com>
* fix: use module name in PURL string everywhere
Signed-off-by: Keith Zantow <kzantow@gmail.com>
---------
Signed-off-by: Rez Moss <hi@rezmoss.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Co-authored-by: Keith Zantow <kzantow@gmail.com>
Removes an accidental `fmt.Println("error", err)` that was left in
the javascript dependency parser. This causes noisy output to stdout
when parsing npm package-lock.json files that contain dependency
specifiers that aren't valid PURLs.
Signed-off-by: Chris Greeno <chris@fresha.com>
* Adding a second function to validate/correct urls that are just github repositories
Signed-off-by: Kendrick <kmartinix@gmail.com>
* Adding test case to capture github repositories
Signed-off-by: Kendrick <kmartinix@gmail.com>
---------
Signed-off-by: Kendrick <kmartinix@gmail.com>
echo"Could not determine the next version to release. Exiting..."
exit1
fi
while true;do
read -p "${bold}Do you want to trigger a release for version '${NEXT_VERSION}'?${normal} [y/n] " yn
case$yn in
[Yy]* ) echo; break;;
[Nn]* ) echo;echo"Cancelling release..."; exit;;
* )echo"Please answer yes or no.";;
esac
done
echo"${bold}Kicking off release for ${NEXT_VERSION}${normal}..."
echo
$GH_CLI workflow run release.yaml -f version=${NEXT_VERSION}
echo
echo"${bold}Waiting for release to start...${normal}"
sleep 10
set +e
echo"${bold}Head to the release workflow to monitor the release:${normal}$($GH_CLI run list --workflow=release.yaml --limit=1 --json url --jq '.[].url')"
id=$($GH_CLI run list --workflow=release.yaml --limit=1 --json databaseId --jq '.[].databaseId')
$GH_CLI run watch $id --exit-status ||(echo;echo"${bold}Logs of failed step:${normal}"&&GH_PAGER=""$GH_CLI run view $id --log-failed)
All contributors for any Anchore project must follow the [Contributor Covenant Code of Conduct](https://oss.anchore.com/docs/contributing/code-of-conduct/).
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, religion, or sexual identity
and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the
overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or
advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email
address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
If you are looking to contribute to this project and want to open a GitHub pull request ("PR"), there are a few guidelines of what we are looking for in patches. Make sure you go through this document and ensure that your code proposal is aligned.
Thank you for your interest in contributing to Syft!
## Setting up your environment
Please see the [contribution guide](https://oss.anchore.com/docs/contributing/syft/) for development requirements and helpful tips to get started developing in the repo. For a deeper dive, please see the [architecture docs](https://oss.anchore.com/docs/architecture/syft/).
Before you can contribute to Syft, you need to configure your development environment.
**Have a question or need help?** Check out our [issues and discussions guide](https://oss.anchore.com/docs/contributing/issues-and-discussions/) to find the right place to start a conversation.
### Debian setup
**Ready to submit code?** Our [pull request guide](https://oss.anchore.com/docs/contributing/pull-requests/) covers everything from title conventions to the review process. Don't forget that ***all commits require a [sign-off](https://oss.anchore.com/docs/contributing/sign-off/)***.
You will need to install Go. The version on https://go.dev works best, using the system golang doesn't always work the way you might expect.
**Found a security issue?** Please do **not** open a public issue. Instead, see our [security policy](https://oss.anchore.com/docs/contributing/security/) for how to report vulnerabilities responsibly.
Refer to the go.mod file in the root of this repo for the recommended version of Go to install.
You will also need Docker. There's no reason the system packages shouldn't work, but we used the official Docker package. You can find instructions for installing Docker in Debian [here](https://docs.docker.com/engine/install/debian/).
You also need to install some Debian packages
```sh
sudo apt-get install build-essential zip bc libxml2-utils git
```
## Configuring Git
You will need to configure your git client with your name and email address. This is easily done from the command line.
This username and email address will matter later in this guide.
## Fork the repo
You should fork the Syft repo using the "Fork" button at the top right of the Syft GitHub [site](https://github.com/anchore/syft/). You will be doing your development in your fork, then submit a pull request to Syft. There are many resources how to use GitHub effectively, we will not cover those here.
## Adding a feature or fix
If you look at the Syft [Issue](https://github.com/anchore/syft/issues) there are plenty of bugs and feature requests. Maybe look at the [good first issue](https://github.com/anchore/syft/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) list if you're not sure where to start.
## Commit guidelines
In the Syft project we like commits and pull requests (PR) to be easy to understand and review. Open source thrives best when everything happening is over documented and small enough to be understood.
### Granular commits
Please try to make every commit as simple as possible, but no simpler. The idea is that each commit should be a logical unit of code. Try not to commit too many tiny changes, for example every line changed in a file as a separate commit. And also try not to make a commit enormous, for example committing all your work at the end of the day.
Rather than try to follow a strict guide on what is or is not best, we try to be flexible and simple in this space. Do what makes the most sense for the changes you are trying to include.
### Commit title and description
Remember that the message you leave for a commit is for the reviewer in the present, and for someone (maybe you) changing something in the future. Please make sure the title and description used is easy to understand and explains what was done. Jokes and clever comments generally don't age well in commit messages. Just the facts please.
## Sign off your work
The `sign-off` is an added line at the end of the explanation for the commit, certifying that you wrote it or otherwise have the right to submit it as an open-source patch. By submitting a contribution, you agree to be bound by the terms of the DCO Version 1.1 and Apache License Version 2.0.
Signing off a commit certifies the below Developer's Certificate of Origin (DCO):
```text
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
```
All contributions to this project are licensed under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/).
When committing your change, you can add the required line manually so that it looks like this:
```text
Signed-off-by: John Doe <john.doe@example.com>
```
Creating a signed-off commit is then possible with `-s` or `--signoff`:
```text
$ git commit -s -m "this is a commit message"
```
To double-check that the commit was signed-off, look at the log output:
This project has a `Makefile` which includes many helpers running both unit and integration tests. You can run `make help` to see all the options. Although PRs will have automatic checks for these, it is useful to run them locally, ensuring they pass before submitting changes. Ensure you've bootstrapped once before running tests:
```text
$ make bootstrap
```
You only need to bootstrap once. After the bootstrap process, you can run the tests as many times as needed:
```text
$ make unit
$ make integration
```
You can also run `make all` to run a more extensive test suite, but there is additional configuration that will be needed for those tests to run correctly. We will not cover the extra steps here.
## Pull Request
If you made it this far and all the tests are passing, it's time to submit a Pull Request (PR) for Syft. Submitting a PR is always a scary moment as what happens next can be an unknown. The Syft project strives to be easy to work with, we appreciate all contributions. Nobody is going to yell at you or try to make you feel bad. We love contributions and know how scary that first PR can be.
### PR Title and Description
Just like the commit title and description mentioned above, the PR title and description is very important for letting others know what's happening. Please include any details you think a reviewer will need to more properly review your PR.
A PR that is very large or poorly described has a higher likelihood of being pushed to the end of the list. Reviewers like PRs they can understand and quickly review.
### What to expect next
Please be patient with the project. We try to review PRs in a timely manner, but this is highly dependent on all the other tasks we have going on. It's OK to ask for a status update every week or two, it's not OK to ask for a status update every day.
It's very likely the reviewer will have questions and suggestions for changes to your PR. If your changes don't match the current style and flow of the other code, expect a request to change what you've done.
## Document your changes
And lastly, when proposed changes are modifying user-facing functionality or output, it is expected the PR will include updates to the documentation as well. Syft is not a project that is heavy on documentation. This will mostly be updating the README and help for the tool.
If nobody knows new features exist, they can't use them!
**Want to help improve the docs?** Check out the [anchore/oss-docs](https://github.com/anchore/oss-docs) repository.
In order to test and develop in this repo you will need the following dependencies installed:
- Golang
- docker
- make
- Python (>= 3.9)
### Docker settings for getting started
Make sure you've updated your docker settings so the default docker socket path is available.
Go to:
docker -> settings -> advanced
Make sure:
```
Allow the default Docker socket to be used
```
is checked.
Also double check that the docker context being used is the default context. If it is not, run:
`docker context use default`
After cloning, the following steps can help you get setup:
1. run `make bootstrap` to download go mod dependencies, create the `/.tmp` dir, and download helper utilities.
2. run `make` to view the selection of developer commands in the Makefile
3. run `make build` to build the release snapshot binaries and packages
4. for an even quicker start you can run `go run cmd/syft/main.go` to print the syft help.
- this command `go run cmd/syft/main.go alpine:latest` will compile and run syft against `alpine:latest`
5. view the README or syft help output for more output options
The main make tasks for common static analysis and testing are `lint`, `format`, `lint-fix`, `unit`, `integration`, and `cli`.
See `make help` for all the current make tasks.
### Internal Artifactory Settings
**Not always applicable**
Some companies have Artifactory setup internally as a solution for sourcing secure dependencies.
If you're seeing an issue where the unit tests won't run because of the below error then this section might be relevant for your use case.
```
[ERROR] [ERROR] Some problems were encountered while processing the POMs
```
If you're dealing with an issue where the unit tests will not pull/build certain java fixtures check some of these settings:
- a `settings.xml` file should be available to help you communicate with your internal artifactory deployment
- this can be moved to `syft/pkg/cataloger/java/test-fixtures/java-builds/example-jenkins-plugin/` to help build the unit test-fixtures
- you'll also want to modify the `build-example-jenkins-plugin.sh` to use `settings.xml`
For more information on this setup and troubleshooting see [issue 1895](https://github.com/anchore/syft/issues/1895#issuecomment-1610085319)
## Architecture
At a high level, this is the package structure of syft:
```
./cmd/syft/
│ ├── cli/
│ │ ├── cli.go // where all commands are wired up
│ │ ├── commands/ // all command implementations
│ │ ├── options/ // all command flags and configuration options
│ │ └── ui/ // all handlers for events that are shown on the UI
│ └── main.go // entrypoint for the application
└── syft/ // the "core" syft library
├── format/ // contains code to encode or decode to and from SBOM formats
├── pkg/ // contains code to catalog packages from a source
├── sbom/ // contains the definition of an SBOM
└── source/ // contains code to create a source object for some input type (e.g. container image, directory, etc)
```
Syft's core library is implemented in the `syft` package and subpackages, where the major packages are:
- the `syft/source` package produces a `source.Source` object that can be used to catalog a directory, container, and other source types.
- the `syft` package contains a single function that can take a `source.Source` object and catalog it, producing an `sbom.SBOM` object
- the `syft/format` package contains the ability to encode and decode SBOMs to and from different SBOM formats (such as SPDX and CycloneDX)
The `cmd` package at the highest level execution flow wires up [`spf13/cobra`](https://github.com/spf13/cobra) commands for execution in the main application:
```mermaid
sequenceDiagram
participant main as cmd/syft/main
participant cli as cli.New()
participant root as root.Execute()
participant cmd as <command>.Execute()
main->>+cli:
Note right of cli: wire ALL CLI commands
Note right of cli: add flags for ALL commands
cli-->>-main: root command
main->>+root:
root->>+cmd:
cmd-->>-root: (error)
root-->>-main: (error)
Note right of cmd: Execute SINGLE command from USER
```
The `packages` command uses the core library to generate an SBOM for the given user input:
```mermaid
sequenceDiagram
participant source as source.New(ubuntu:latest)
participant sbom as sbom.SBOM
participant catalog as syft.CatalogPackages(src)
participant encoder as syft.Encode(sbom, format)
Note right of source: use "ubuntu:latest" as SBOM input
source-->>+sbom: add source to SBOM struct
source-->>+catalog: pass src to generate catalog
catalog-->-sbom: add cataloging results onto SBOM
sbom-->>encoder: pass SBOM and format desired to syft encoder
encoder-->>source: return bytes that are the SBOM of the original input
Note right of catalog: cataloger configuration is done based on src
```
Additionally, here is a [gist of using syft as a library](https://gist.github.com/spiffcs/3027638b7ba904d07e482a712bc00d3d) to generate a SBOM for a docker image.
### `pkg.Package` object
The `pkg.Package` object is a core data structure that represents a software package. Fields like `name` and `version` probably don't need
a detailed explanation, but some of the other fields are worth a quick overview:
-`FoundBy`: the name of the cataloger that discovered this package (e.g. `python-pip-cataloger`).
-`Locations`: these are the set of paths and layer ids that were parsed to discover this package (e.g. `python-pip-cataloger`).
-`Language`: the language of the package (e.g. `python`).
-`Type`: this is a high-level categorization of the ecosystem the package resides in. For instance, even if the package is a egg, wheel, or requirements.txt reference, it is still logically a "python" package. Not all package types align with a language (e.g. `rpm`) but it is common.
-`Metadata`: specialized data for specific location(s) parsed. We should try and raise up as much raw information that seems useful. As a rule of thumb the object here should be as flat as possible and use the raw names and values from the underlying source material parsed.
When `pkg.Package` is serialized an additional `MetadataType` is shown. This is a label that helps consumers understand the datashape of the `Metadata` field.
By convention the `MetadataType` value should follow these rules of thumb:
- Only use lowercase letters, numbers, and hyphens. Use hyphens to separate words.
- **Try to anchor the name in the ecosystem, language, or packaging tooling it belongs to**. For a package manager for a language ecosystem the language, framework or runtime should be used as a prefix. For instance `pubspec-lock` is an OK name, but `dart-pubspec-lock` is better. For an OS package manager this is not necessary (e.g. `apk-db-entry` is a good name, but `alpine-apk-db-entry` is not since `alpine` and the `a` in `apk` is redundant).
- **Be as specific as possible to what the data represents**. For instance `ruby-gem` is NOT a good `MetadataType` value, but `ruby-gemspec` is. Why? Ruby gem information can come from a gemspec file or a Gemfile.lock, which are very different. The latter name provides more context as to what to expect.
- **Should describe WHAT the data is, NOT HOW it's used**. For instance `r-description-installed-file` is NOT a good `MetadataType` value since it's trying to convey that we use the DESCRIPTION file in the R ecosystem to detect installed packages. Instead simply describe what the DESCRIPTION file is itself without context of how it's used: `r-description`.
- **Use the `lock` suffix** to distinct between manifest files that loosely describe package version requirements vs files that strongly specify one and only one version of a package ("lock" files). These should only be used with respect to package managers that have the guide and lock distinction, but would not be appropriate otherwise (e.g. `rpm` does not have a guide vs lock, so `lock` should NOT be used to describe a db entry).
- **Use the `archive` suffix to indicate a package archive** (e.g. rpm file, apk file, etc) that describes the contents of the package. For example an RPM file that was cataloged would have a `rpm-archive` metadata type (not to be confused with an RPM DB record entry which would be `rpm-db-entry`).
- **Use the `entry` suffix** to indicate information about a package that was found as a single entry within file that has multiple package entries. If the entry was found within a DB or a flat-file store for an OS package manager, you should use `db-entry`.
- **Should NOT contain the phrase `package`**, though exceptions are allowed (say if the canonical name literally has the phrase package in it).
- **Should NOT contain have a `file` suffix** unless the canonical name has the term "file", such as a `pipfile` or `gemfile`. An example of a bad name for this rule is`ruby-gemspec-file`; a better name would be `ruby-gemspec`.
- **Should NOT contain the exact filename+extensions**. For instance `pipfile.lock` shouldn't really be in the name, instead try and describe what the file is: `python-pipfile-lock` (but shouldn't this be `python-pip-lock` you might ask? No, since the `pip` package manger is not related to the `pipfile` project).
- **Should NOT contain the phrase `metadata`**, unless the canonical name has this term.
- **Should represent a single use case**. For example, trying to describe Hackage metadata with a single `HackageMetadata` struct (and thus `MetadataType`) is not allowed since it represents 3 mutually exclusive use cases: representing a `stack.yaml`, `stack.lock`, or `cabal.project` file. Instead, each of these should have their own struct types and `MetadataType` values.
There are other cases that are not covered by these rules... and that's ok! The goal is to provide a consistent naming scheme that is easy to understand and use when it's applicable. If the rules do not exactly apply in your situation then just use your best judgement (or amend these rules as needed whe new common cases come up).
What if the underlying parsed data represents multiple files? There are two approaches to this:
- use the primary file to represent all the data. For instance, though the `dpkg-cataloger` looks at multiple files to get all information about a package, it's the `status` file that gets represented.
- nest each individual file's data under the `Metadata` field. For instance, the `java-archive-cataloger` may find information from on or all of the files: `pom.xml`, `pom.properties`, and `MANIFEST.MF`. However, the metadata is simply `java-metadata' with each possibility as a nested optional field.
### Syft Catalogers
Catalogers are the way in which syft is able to identify and construct packages given a set a targeted list of files.
For example, a cataloger can ask syft for all `package-lock.json` files in order to parse and raise up javascript packages
(see [how file globs](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/javascript/cataloger.go#L16-L21) and
[file parser functions](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/javascript/cataloger.go#L16-L21) are used
for a quick example).
From a high level catalogers have the following properties:
- _They are independent from one another_. The java cataloger has no idea of the processes, assumptions, or results of the python cataloger, for example.
- _They do not know what source is being analyzed_. Are we analyzing a local directory? an image? if so, the squashed representation or all layers? The catalogers do not know the answers to these questions. Only that there is an interface to query for file paths and contents from an underlying "source" being scanned.
- _Packages created by the cataloger should not be mutated after they are created_. There is one exception made for adding CPEs to a package after the cataloging phase, but that will most likely be moved back into the cataloger in the future.
Cataloger names should be unique and named with the following rules of thumb in mind:
- Must end with `-cataloger`
- Use lowercase letters, numbers, and hyphens only
- Use hyphens to separate words
- Catalogers for language ecosystems should start with the language name (e.g. `python-` for a cataloger that raises up python packages)
- Distinguish between when the cataloger is searching for evidence of installed packages vs declared packages. For example, there are currently two different gemspec-based catalogers, the `ruby-gemspec-cataloger` and `ruby-installed-gemspec-cataloger`, where the latter requires that the gemspec is found within a `specifications` directory (which means it was installed, not just at the root of a source repo).
#### Building a new Cataloger
Catalogers must fulfill the [`pkg.Cataloger` interface](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger.go) in order to add packages to the SBOM.
All catalogers should be added to:
- the [global list of catalogers](https://github.com/anchore/syft/blob/9995950c70e849f9921919faffbfcf46401f71f3/syft/pkg/cataloger/cataloger.go#L92-L125)
- at least one source-specific list, today the two lists are [directory catalogers and image catalogers](https://github.com/anchore/syft/blob/9995950c70e849f9921919faffbfcf46401f71f3/syft/pkg/cataloger/cataloger.go#L39-L89)
For reference, catalogers are [invoked within syft](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/catalog.go#L41-L100) one after the other, and can be invoked in parallel.
`generic.NewCataloger` is an abstraction syft used to make writing common components easier (see the [apkdb cataloger](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/apkdb/cataloger.go) for example usage).
It takes the following information as input:
- A `catalogerName` to identify the cataloger uniquely among all other catalogers.
- Pairs of file globs as well as parser functions to parse those files. These parser functions return a slice of [`pkg.Package`](https://github.com/anchore/syft/blob/9995950c70e849f9921919faffbfcf46401f71f3/syft/pkg/package.go#L19) as well as a slice of [`artifact.Relationship`](https://github.com/anchore/syft/blob/9995950c70e849f9921919faffbfcf46401f71f3/syft/artifact/relationship.go#L31) to describe how the returned packages are related. See this [the apkdb cataloger parser function](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/apkdb/parse_apk_db.go#L22-L102) as an example.
Identified packages share a common `pkg.Package` struct so be sure that when the new cataloger is constructing a new package it is using the [`Package` struct](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/package.go#L16-L31).
If you want to return more information than what is available on the `pkg.Package` struct then you can do so in the `pkg.Package.Metadata` section of the struct, which is unique for each [`pkg.Type`](https://github.com/anchore/syft/blob/v0.70.0/syft/pkg/type.go).
See [the `pkg` package](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg) for examples of the different metadata types that are supported today.
These are plugged into the `MetadataType` and `Metadata` fields in the above struct. `MetadataType` informs which type is being used. `Metadata` is an interface converted to that type.
Finally, here is an example of where the package construction is done within the apk cataloger:
- [Calling the APK package constructor from the parser function](https://github.com/anchore/syft/blob/v0.70.0/syft/pkg/cataloger/apkdb/parse_apk_db.go#L106)
Interested in building a new cataloger? Checkout the [list of issues with the `new-cataloger` label](https://github.com/anchore/syft/issues?q=is%3Aopen+is%3Aissue+label%3Anew-cataloger+no%3Aassignee)!
If you have questions about implementing a cataloger feel free to file an issue or reach out to us [on discourse](https://anchore.com/discourse)!
#### Searching for files
All catalogers are provided an instance of the [`file.Resolver`](https://github.com/anchore/syft/blob/v0.70.0/syft/source/file_resolver.go#L8) to interface with the image and search for files. The implementations for these
abstractions leverage [`stereoscope`](https://github.com/anchore/stereoscope) in order to perform searching. Here is a
rough outline how that works:
1. a stereoscope `file.Index` is searched based on the input given (a path, glob, or MIME type). The index is relatively fast to search, but requires results to be filtered down to the files that exist in the specific layer(s) of interest. This is done automatically by the `filetree.Searcher` abstraction. This abstraction will fallback to searching directly against the raw `filetree.FileTree` if the index does not contain the file(s) of interest. Note: the `filetree.Searcher` is used by the `file.Resolver` abstraction.
2. Once the set of files are returned from the `filetree.Searcher` the results are filtered down further to return the most unique file results. For example, you may have requested for files by a glob that returns multiple results. These results are filtered down to deduplicate by real files, so if a result contains two references to the same file, say one accessed via symlink and one accessed via the real path, then the real path reference is returned and the symlink reference is filtered out. If both were accessed by symlink then the first (by lexical order) is returned. This is done automatically by the `file.Resolver` abstraction.
3. By the time results reach the `pkg.Cataloger` you are guaranteed to have a set of unique files that exist in the layer(s) of interest (relative to what the resolver supports).
## Testing
### Testing commands
*`make help` shows a list of available commands
*`make unit`, `make integration`, `make cli`, and `make acceptance` run those test suites (see below)
*`make test` runs all those tests (and is therefore pretty slow)
*`make fixtures` clears and re-fetches all test fixtures.
*`go test ./syft/pkg/` for example can test particular packages, assuming fixtures are already made
*`make clean-cache` cleans all test cache. Note that subsequent test runs will be slower after this
### Levels of testing
-`unit`: The default level of test which is distributed throughout the repo are unit tests. Any `_test.go` file that
does not reside somewhere within the `/test` directory is a unit test. Other forms of testing should be organized in
the `/test` directory. These tests should focus on correctness of functionality in depth. % test coverage metrics
only considers unit tests and no other forms of testing.
-`integration`: located within `cmd/syft/internal/test/integration`, these tests focus on the behavior surfaced by the common library
entrypoints from the `syft` package and make light assertions about the results surfaced. Additionally, these tests
tend to make diversity assertions for enum-like objects, ensuring that as enum values are added to a definition
that integration tests will automatically fail if no test attempts to use that enum value. For more details see
the "Data diversity and freshness assertions" section below.
-`cli`: located with in `test/cli`, these are tests that test the correctness of application behavior from a
snapshot build. This should be used in cases where a unit or integration test will not do or if you are looking
for in-depth testing of code in the `cmd/` package (such as testing the proper behavior of application configuration,
CLI switches, and glue code before syft library calls).
-`acceptance`: located within `test/compare` and `test/install`, these are smoke-like tests that ensure that application
packaging and installation works as expected. For example, during release we provide RPM packages as a download
artifact. We also have an accompanying RPM acceptance test that installs the RPM from a snapshot build and ensures the
output of a syft invocation matches canned expected output. New acceptance tests should be added for each release artifact
and architecture supported (when possible).
### Data diversity and freshness assertions
It is important that tests against the codebase are flexible enough to begin failing when they do not cover "enough"
of the objects under test. "Cover" in this case does not mean that some percentage of the code has been executed
during testing, but instead that there is enough diversity of data input reflected in testing relative to the
definitions available.
For instance, consider an enum-like value like so:
```go
typeLanguagestring
const(
JavaLanguage="java"
JavaScriptLanguage="javascript"
PythonLanguage="python"
RubyLanguage="ruby"
GoLanguage="go"
)
```
Say we have a test that exercises all the languages defined today:
```go
funcTestCatalogPackages(t*testing.T){
testTable:=[]struct{
// ... the set of test cases that test all languages
}
for_,test:=rangecases{
t.Run(test.name,func(t*testing.T){
// use inputFixturePath and assert that syft.CatalogPackages() returns the set of expected Package objects
// ...
})
}
}
```
Where each test case has a `inputFixturePath` that would result with packages from each language. This test is
brittle since it does not assert that all languages were exercised directly and future modifications (such as
adding a new language) won't be covered by any test cases.
To address this the enum-like object should have a definition of all objects that can be used in testing:
```go
typeLanguagestring
// const( Java Language = ..., ... )
varAllLanguages=[]Language{
Java,
JavaScript,
Python,
Ruby,
Go,
Rust,
}
```
Allowing testing to automatically fail when adding a new language:
```go
funcTestCatalogPackages(t*testing.T){
testTable:=[]struct{
// ... the set of test cases that (hopefully) covers all languages
}
// new stuff...
observedLanguages:=strset.New()
for_,test:=rangecases{
t.Run(test.name,func(t*testing.T){
// use inputFixturePath and assert that syft.CatalogPackages() returns the set of expected Package objects
// ...
// new stuff...
for_,actualPkg:=rangeactual{
observedLanguages.Add(string(actualPkg.Language))
}
})
}
// new stuff...
for_,expectedLanguage:=rangepkg.AllLanguages{
if!observedLanguages.Contains(expectedLanguage){
t.Errorf("failed to test language=%q",expectedLanguage)
}
}
}
```
This is a better test since it will fail when someone adds a new language but fails to write a test case that should
exercise that new language. This method is ideal for integration-level testing, where testing correctness in depth
is not needed (that is what unit tests are for) but instead testing in breadth to ensure that units are well integrated.
A similar case can be made for data freshness; if the quality of the results will be diminished if the input data
is not kept up to date then a test should be written (when possible) to assert any input data is not stale.
An example of this is the static list of licenses that is stored in `internal/spdxlicense` for use by the SPDX
presenters. This list is updated and published periodically by an external group and syft can grab and update this
list by running `go generate ./...` from the root of the repo.
An integration test has been written to grabs the latest license list version externally and compares that version
with the version generated in the codebase. If they differ, the test fails, indicating to someone that there is an
action needed to update it.
**_The key takeaway is to try and write tests that fail when data assumptions change and not just when code changes.**_
### Snapshot tests
The format objects make a lot of use of "snapshot" testing, where you save the expected output bytes from a call into the
git repository and during testing make a comparison of the actual bytes from the subject under test with the golden
copy saved in the repo. The "golden" files are stored in the `test-fixtures/snapshot` directory relative to the go
package under test and should always be updated by invoking `go test` on the specific test file with a specific CLI
update flag provided.
Many of the `Format` tests make use of this approach, where the raw SBOM report is saved in the repo and the test
compares that SBOM with what is generated from the latest presenter code. The following command can be used to
update the golden files for the various snapshot tests:
```bash
make update-format-golden-files
```
These flags are defined at the top of the test files that have tests that use the snapshot files.
Snapshot testing is only as good as the manual verification of the golden snapshot file saved to the repo! Be careful
Syft is a powerful and easy-to-use open-source tool for generating Software Bill of Materials (SBOMs) for container images and filesystems. It provides detailed visibility into the packages and dependencies in your software, helping you manage vulnerabilities, license compliance, and software supply chain security.
Syft development is sponsored by [Anchore](https://anchore.com/), and is released under the [Apache-2.0 License](https://github.com/anchore/syft?tab=Apache-2.0-1-ov-file). For commercial support options with Syft or Grype, please [contact Anchore](https://get.anchore.com/contact/).
## Features
- Generates SBOMs for container images, filesystems, archives, and more to discover packages and libraries
-Supports OCI, Docker and [Singularity](https://github.com/sylabs/singularity) image formats
-Linux distribution identification
-Works seamlessly with [Grype](https://github.com/anchore/grype) (a fast, modern vulnerability scanner)
-Able to create signed SBOM attestations using the [in-toto specification](https://github.com/in-toto/attestation/blob/main/spec/README.md)
-Convert between SBOM formats, such as CycloneDX, SPDX, and Syft's own format.
-Generates SBOMs for **container images**, **filesystems**, **archives** (see the docs for a full list of [supported scan targets](https://oss.anchore.com/docs/guides/sbom/scan-targets/))
-Supports OCI, Docker, [Singularity](https://github.com/sylabs/singularity), and [more image formats](https://oss.anchore.com/docs/guides/sbom/scan-targets/)
-Works seamlessly with [Grype](https://github.com/anchore/grype) for vulnerability scanning
-Multiple output formats (**CycloneDX**, **SPDX**, **Syft JSON**, and [more](https://oss.anchore.com/docs/guides/sbom/formats/)) including the ability to [convert between SBOM formats](https://oss.anchore.com/docs/guides/sbom/conversion/)
- Create signed SBOM attestations using the [in-toto specification](https://github.com/in-toto/attestation/blob/main/spec/README.md)
> [!TIP]
> **New to Syft? Check out the [Getting Started guide](https://oss.anchore.com/docs/guides/sbom/getting-started/) for a walkthrough!**
## Installation
Syft binaries are provided for Linux, macOS and Windows.
-`-b`: Specify a custom installation directory (defaults to `./bin`)
-`-d`: More verbose logging levels (`-d` for debug, `-dd` for trace)
-`-v`: Verify the signature of the downloaded artifact before installation (requires [`cosign`](https://github.com/sigstore/cosign) to be installed)
### Homebrew
The quickest way to get up and going:
```bash
brew install syft
curl -sSfL https://get.anchore.io/syft | sudo sh -s -- -b /usr/local/bin
```
### Scoop
> [!TIP]
> **See [Installation docs](https://oss.anchore.com/docs/installation/syft/) for more ways to get Syft, including Homebrew, Docker, Scoop, Chocolatey, Nix, and more!**
```powershell
scoopinstallsyft
```
## The basics
### Chocolatey
The chocolatey distribution of Syft is community-maintained and not distributed by the Anchore team
```powershell
chocoinstallsyft-y
```
### Nix
**Note**: Nix packaging of Syft is [community maintained](https://github.com/NixOS/nixpkgs/blob/master/pkgs/by-name/sy/syft/package.nix). Syft is available in the [stable channel](https://wiki.nixos.org/wiki/Nix_channels#The_official_channels) since NixOS `22.05`.
See the packages within a container image or directory:
```bash
nix-env -i syft
# container image
syft alpine:latest
# directory
syft ./my-project
```
... or, just try it out in an ephemeral nix shell:
To get an SBOM, specify one or more output formats:
> **Check out the [Getting Started guide](https://oss.anchore.com/docs/guides/sbom/getting-started/)** to explore all of the capabilities and features.
>
> **Want to know all of the ins-and-outs of Syft?** Check out the [CLI docs](https://oss.anchore.com/docs/reference/syft/cli/), [configuration docs](https://oss.anchore.com/docs/reference/syft/configuration/), and [JSON schema](https://oss.anchore.com/docs/reference/syft/json/latest/).
To generate an SBOM for a container image:
```bash
syft <image>
```
The above output includes only software that is visible in the container (i.e., the squashed representation of the image). To include software from all image layers in the SBOM, regardless of its presence in the final image, provide `--scope all-layers`:
```bash
syft <image> --scope all-layers
```
### Output formats
The output format for Syft is configurable as well using the `-o` (or `--output`) option:
```
syft <image> -o <format>
```
Where the `formats` available are:
-`syft-json`: Use this to get as much information out of Syft as possible!
-`syft-text`: A row-oriented, human-and-machine-friendly output.
-`cyclonedx-xml`: An XML report conforming to the [CycloneDX 1.6 specification](https://cyclonedx.org/specification/overview/).
-`cyclonedx-xml@1.5`: An XML report conforming to the [CycloneDX 1.5 specification](https://cyclonedx.org/specification/overview/).
-`cyclonedx-json`: A JSON report conforming to the [CycloneDX 1.6 specification](https://cyclonedx.org/specification/overview/).
-`cyclonedx-json@1.5`: A JSON report conforming to the [CycloneDX 1.5 specification](https://cyclonedx.org/specification/overview/).
-`spdx-tag-value`: A tag-value formatted report conforming to the [SPDX 2.3 specification](https://spdx.github.io/spdx-spec/v2.3/).
-`spdx-tag-value@2.2`: A tag-value formatted report conforming to the [SPDX 2.2 specification](https://spdx.github.io/spdx-spec/v2.2.2/).
-`spdx-json`: A JSON report conforming to the [SPDX 2.3 JSON Schema](https://github.com/spdx/spdx-spec/blob/v2.3/schemas/spdx-schema.json).
-`spdx-json@2.2`: A JSON report conforming to the [SPDX 2.2 JSON Schema](https://github.com/spdx/spdx-spec/blob/v2.2/schemas/spdx-schema.json).
-`github-json`: A JSON report conforming to GitHub's dependency snapshot format.
-`syft-table`: A columnar summary (default).
-`template`: Lets the user specify the output format. See ["Using templates"](https://github.com/anchore/syft/wiki/using-templates) below.
Note that flags using the @<version> can be used for earlier versions of each specification as well.
### Supported Ecosystems
- Alpine (apk)
- Bitnami packages
- C (conan)
- C++ (conan)
- Dart (pubs)
- Debian (dpkg)
- Dotnet (deps.json)
- Objective-C (cocoapods)
- Elixir (mix)
- Erlang (rebar3)
- Go (go.mod, Go binaries)
- GitHub (workflows, actions)
- Haskell (cabal, stack)
- Java (jar, ear, war, par, sar, nar, rar, native-image)
* [Adding an SBOM to an image as an attestation using Syft](https://github.com/anchore/syft/wiki/attestation#adding-an-sbom-to-an-image-as-an-attestation-using-syft)
Check out our [contributing](/CONTRIBUTING.md) guide and [developer](/DEVELOPING.md) docs.
We encourage users to help make these tools better by [submitting issues](https://github.com/anchore/syft/issues) when you find a bug or want a new feature.
Check out our [contributing overview](https://oss.anchore.com/docs/contributing/) and [developer-specific documentation](https://oss.anchore.com/docs/contributing/syft/) if you are interested in providing code contributions.
## Syft Team Meetings
The Syft Team hold regular community meetings online. All are welcome to join to bring topics for discussion.
Syft development is sponsored by <a href="https://anchore.com/">Anchore</a>, and is released under the <a href="https://github.com/anchore/syft?tab=Apache-2.0-1-ov-file">Apache-2.0 License</a>.
The <a property="dct:title" rel="cc:attributionURL" href="https://anchore.com/wp-content/uploads/2024/11/syft-logo.svg">Syft logo</a> by <a rel="cc:attributionURL dct:creator" property="cc:attributionName" href="https://anchore.com/">Anchore</a> is licensed under <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank" rel="license noopener noreferrer" style="display:inline-block;">CC BY 4.0<img style="height:22px!important;margin-left:3px;vertical-align:text-bottom;" src="https://mirrors.creativecommons.org/presskit/icons/cc.svg" alt=""><img style="height:22px!important;margin-left:3px;vertical-align:text-bottom;" src="https://mirrors.creativecommons.org/presskit/icons/by.svg" alt=""></a>
</p>
For commercial support options with Syft or Grype, please [contact Anchore](https://get.anchore.com/contact/).
## Come talk to us!
The Syft Team holds regular community meetings online. All are welcome to join to bring topics for discussion.
- Check the [calendar](https://calendar.google.com/calendar/u/0/r?cid=Y182OTM4dGt0MjRtajI0NnNzOThiaGtnM29qNEBncm91cC5jYWxlbmRhci5nb29nbGUuY29t) for the next meeting date.
- Add items to the [agenda](https://docs.google.com/document/d/1ZtSAa6fj2a6KRWviTn3WoJm09edvrNUp4Iz_dOjjyY8/edit?usp=sharing) (join [this group](https://groups.google.com/g/anchore-oss-community) for write access to the [agenda](https://docs.google.com/document/d/1ZtSAa6fj2a6KRWviTn3WoJm09edvrNUp4Iz_dOjjyY8/edit?usp=sharing))
- See you there!
## Syft Logo
<p xmlns:cc="http://creativecommons.org/ns#" xmlns:dct="http://purl.org/dc/terms/"><a property="dct:title" rel="cc:attributionURL" href="https://anchore.com/wp-content/uploads/2024/11/syft-logo.svg">Syft Logo</a> by <a rel="cc:attributionURL dct:creator" property="cc:attributionName" href="https://anchore.com/">Anchore</a> is licensed under <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank" rel="license noopener noreferrer" style="display:inline-block;">CC BY 4.0<img style="height:22px!important;margin-left:3px;vertical-align:text-bottom;" src="https://mirrors.creativecommons.org/presskit/icons/cc.svg" alt=""><img style="height:22px!important;margin-left:3px;vertical-align:text-bottom;" src="https://mirrors.creativecommons.org/presskit/icons/by.svg" alt=""></a></p>
with a description of the issue, the steps you took to create the issue,
affected versions, and, if known, mitigations for the issue.
All support will be made on a best effort basis, so please indicate the "urgency level" of the vulnerability as Critical, High, Medium or Low.
For more details, see our [security policy documentation](https://oss.anchore.com/docs/contributing/security/).
## Trust Boundary
Syft is a tool to scan content and product an SBOM. Syft is not a tool designed to scan malicious content. Detecting and properly reporting on purposely malicious artifacts is outside the scope of Syft's expected operating environment.
There are many possible ways for malicious content to cause Syft to become confused or fail to include results in an SBOM. We do not consider this to be a security vulnerability.
**Examples**
- Removing or altering a package lock file
- Removing or altering an RPM or DEB database
- A malicious archive that Syft will skip but the runtime may not
- Self modifying systems that change state when running
We consider the security trust boundary for Syft to be anything that causes problems for the overall system running Syft, or Syft operating in a way that is dangerous to itself, the system, or the operator.
**Examples**
- Filling up temp space permanently
- Syft executing arbitrary code when scanning an artifact
- Syft leaking secrets from the environment or configuration files into logs or SBOMs
- Syft operating outside of the expected artifact or directory (directory traversal)
# snapshot builds load per-arch images rather than a multi-arch manifest, so the smoke
# test must run the host-native tag to avoid emulation (and red-local/green-CI splits).
DOCKER_ARCH:
sh:go env GOARCH
# note: the snapshot dir must be a relative path starting with ./
# e.g. when installing snapshot debs from a local path, ./ forces the deb to be installed in the current working directory instead of referencing a package name
Short:"Generate an SBOM as an attestation for the given [SOURCE] container image",
Long:"Generate a packaged-based Software Bill Of Materials (SBOM) from a container image as the predicate of an in-toto attestation that will be uploaded to the image registry",
returnfmt.Errorf("unable to generate cataloger info report: %w",err)
}
bus.Report(report)
bus.Notify("Note: the `cataloger info` command is experimental and may change or be removed without notice. Do not depend on its output in production systems.")
Long:"[Experimental] Convert SBOM files to, and from, SPDX, CycloneDX and Syft's format. For more info about data loss between formats see https://github.com/anchore/syft/wiki/format-conversion",
log.Warn("convert.passthrough-exact-format is enabled: syft-json input that already exactly matches a requested output format and schema version will be copied to that output unchanged (this does not apply to any other input format)")
ifuserInput=="-"{
// though os.Stdin is an os.File, it does not support seeking
// you will get errors such as "seek /dev/stdin: illegal seek".
log.WithFields("format",id,"version",version,"output",out).Warn("input already exactly matches the requested output format and schema version, copying it unchanged instead of converting it")
unchanged=append(unchanged,out)
continue
}
toConvert=append(toConvert,out)
}
returnunchanged,toConvert,nil
}
// writeUnchanged copies the SBOM document as-is to the destination described by the given "<format>[=<path>]"
// output value, falling back to the (deprecated) --file path, and finally to the report bus (STDOUT). When the
// destination is the input file itself there is nothing to do.
returnfmt.Errorf("cannot use both 'catalogers' and 'select-catalogers'/'default-catalogers' flags %q | %q %q",cfg.Catalogers,cfg.SelectCatalogers,cfg.DefaultCatalogers)
@@ -24,6 +26,7 @@ func (o *dotnetConfig) DescribeFields(descriptions clio.FieldDescriptionSet) {
descriptions.Add(&o.DepPackagesMustClaimDLL,`only keep dep.json packages which have a runtime/resource DLL claimed in the deps.json targets section (but not necessarily found on disk). The package is also included if any child package claims a DLL, even if the package itself does not claim a DLL.`)
descriptions.Add(&o.PropagateDLLClaimsToParents,`treat DLL claims or on-disk evidence for child packages as DLL claims or on-disk evidence for any parent package`)
descriptions.Add(&o.RelaxDLLClaimsWhenBundlingDetected,`show all packages from the deps.json if bundling tooling is present as a dependency (e.g. ILRepack)`)
descriptions.Add(&o.ExcludeProjectReferences,`exclude packages with type "project" from deps.json output (these are internal project references, not NuGet packages)`)
descriptions.Add(&o.UsePackagesLib,`use the golang.org/x/tools/go/packages library, which executes golang tooling found on the path in addition to potential network access to get the most accurate results`)
// note: descriptions must be static string literals; the app config discovery that generates the
// capability docs reads them straight out of the AST
# we should strive to not commit blobs to the repo and strive to keep the build process of how blobs are acquired in-repo.
# this blob is generated from syft/syft/catalogers/java/test-fixtures/java-builds , however, preserving the build process
# this blob is generated from syft/syft/catalogers/java/testdata/java-builds , however, preserving the build process
# twice in the repo seems redundant (even via symlink). Given that the fixture is a few kilobytes in size, the build process is already
# captured, and integration tests should only be testing if jars can be discovered (not necessarily depth in java detection
# functionality), committing it seems like an acceptable exception.
@@ -7,4 +7,6 @@
!image-pkg-coverage/pkgs/java/*.hpi
**/go.sum
!image-go-bin-arch-coverage/go.sum
!image-go-bin-arch-coverage/go.sum
cache/
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.