* fix(binary): detect multi-arch ingress-nginx
Add v1.9.6 ARM64, ARMv7, and s390x fixtures from the published controller image and match their version marker against the nearby Go build version.
Assisted-by: OpenAI Codex
Signed-off-by: Shurong Cao <170531907+CAOShurong@users.noreply.github.com>
* fix(binary): match ingress-nginx release marker on all arches
The release is injected with `-ldflags -X`, so it lands in its own aligned,
NUL-padded data symbol. Only that padding is portable -- the surrounding bytes
are an arch-specific float constant pool, which is why the existing matchers all
anchor on incidental junk like `$a` and `S=v<y5` and only ever worked on amd64.
Matching `v<version>` followed by two NULs finds exactly one hit in every v1.9.6
binary (amd64, arm, arm64, s390x) and in all nine amd64 releases already under
test. Two NULs matter -- with one, s390x matches a vendored `v1.19.0` earlier in
the file and reports the wrong version.
This also drops the ~8KB wildcard window the marker previously needed, so the new
fixtures are ordinary 369 byte snippets instead of 8.4KB ones.
Two things in the fixture tooling had to move for that to work:
- `config.yaml` pinned the multi-arch *index* digest for all four platforms, but
extraction runs `docker create <ref>` with no `--platform`, so every entry
resolved to whichever image was pulled last. `make download` failed outright on
`linux/arm/v7`, since the registry labels that platform `armv7` and docker
normalizes it to `arm`. Now pinned per-platform, matching the redis entry.
- `write-snippet` only recognized EM_X86_64 and EM_AARCH64, so anything else
landed in a `linux-unknown-<hex>` directory and had to be placed by hand.
Verified against the real binaries with `-must-use-original-binaries`.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(binary): put non-amd64 snippets where the tests look for them
`write-snippet` used to fall back to `unknown-<hex>` for any ELF machine it
didn't recognize, so three snippets were sitting in directories nothing reads:
helm/3.12.0/linux-unknown-454d5f53333930 (hex of "EM_S390")
helm/4.1.4/linux-unknown-454d5f53333930
redis-server/7.2.5/linux-unknown-454d5f333836 (hex of "EM_386")
The test table asks for `linux-s390x` and `linux-386`, so those three cases were
quietly falling through to downloading the full binaries instead. Renamed to
match, and taught the tool about EM_386 and EM_RISCV so it stops producing dead
directories (there is an existing `linux-riscv64` snippet it also couldn't have
written).
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Shurong Cao <170531907+CAOShurong@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Shurong Cao <170531907+CAOShurong@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(javascript): read every document of a pnpm-lock.yaml stream
pnpm-lock.yaml can be a multi-document YAML stream. pnpm keeps config
dependencies and the pinned package-manager version in a leading document
and the project's dependency graph in the next one. That layout has
shipped since pnpm 11 for projects using config dependencies, and pnpm 12
records the package-manager pin by default, so two documents is now the
common case.
The cataloger called yaml.Unmarshal, which reads the first document only.
On such a lockfile the SBOM contains pnpm's own release binaries and none
of the project's dependencies. Both documents declare the same
lockfileVersion and the result is well-formed and non-empty, so nothing
signals that it is wrong.
Split the stream and parse each document, merging the results. A
single-document lockfile yields exactly one document, so its behaviour is
unchanged. A document that omits lockfileVersion falls back to the
version from the first one rather than being dropped.
Fixes#5168
Signed-off-by: hamodywe <iosapk.org@gmail.com>
* refactor(javascript): stream pnpm-lock.yaml documents instead of buffering
The multi-document fix read the whole lockfile, decoded it to a `yaml.Node`
tree, re-marshaled every document back to `[]byte` and then re-parsed each one
twice. `yaml.Node` already has a `Decode` method, so the decoder can walk the
stream straight off the reader and hand each document's node to the parser.
On a 4.8MB single-document lockfile that takes total allocations from 1069MB to
250MB, which is also below where things sat before multi-document support. It
drops the `io.ReadAll` and its `noUnboundedReads` suppression, and puts pnpm in
line with every other YAML cataloger in the tree, all of which already stream.
`pnpmLockfileParser.Parse` now takes a `*yaml.Node` rather than `[]byte`. It is
package-private with two implementations and one production call site.
Documents accumulate into a single map keyed by `name@version`, so the stream
follows one collision rule rather than two: the last entry to appear wins,
which is what already happens within a document. That replaces the separate
first-wins-with-fill-in merge pass. A conflicting integrity between documents
is now traced rather than resolved silently.
Three behaviour fixes come with it:
- an empty or comment-only leading document no longer takes down the whole
file. It decodes to a null node, not a zero-kind one, so the old guard never
fired and the stream failed with `invalid lockfile version ""`.
- a malformed document no longer discards the documents that already parsed.
It is reported through `unknown` instead, matching how the rest of the
cataloger reports partial results.
- parse errors carry the document index, and their line numbers again point at
the user's file rather than into a re-marshaled buffer.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(javascript): trim the duplicated pnpm multi-document comment
The six-line explanation of why a pnpm lockfile can hold two documents sat
verbatim in both the parser and its test. Keep it next to the code, where
someone debugging will be, and leave the test pointing at it.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: hamodywe <iosapk.org@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
orderedIDSet.add scanned the entire existing slice for every inserted ID,
so building a Collection was quadratic in the number of packages sharing an
index key. This is most visible in idsByType: package types are few, so that
index accumulates nearly every package in the SBOM into a single set, and
each Add rescans it.
Keep the ordered slice (ordering and dedup semantics are unchanged) and add
a hash index for membership checks. The index is only built once a set grows
beyond a small threshold, so the many single-element sets held by idsByName
and idsByPath do not pay for a map, while the large idsByType sets get
constant-time lookups.
BenchmarkCollectionAdd (Apple M2 Pro), packages sharing one type:
packages before after speedup
1,000 1.37 ms 0.76 ms 1.8x
10,000 100.31 ms 9.80 ms 10.2x
50,000 2,423.79 ms 34.46 ms 70.3x
The tradeoff is memory: sets past the threshold allocate an index map,
about 14% more bytes at 50,000 packages, with allocation counts unchanged.
Signed-off-by: Oleksandr Vodotiiets <61548316+avodotiiets@users.noreply.github.com>
With search-remote-licenses enabled, every module name went to the proxy,
including toolchain binaries whose main module is synthesized from the
package path, such as cmd/cgo. The proxy answers 404, and the direct
fallback then treats the path as a repository host, producing requests
like https://cmd/cgo/info/refs?service=git-upload-pack.
Reuse isStandardImportPath, already in this package, to skip the remote
search for paths whose first element carries no dot. Those are never
publishable module paths, so neither a proxy nor a repository can resolve
them.
Fixes#3149
Signed-off-by: Luan Taraschi <130802253+luantaraschi@users.noreply.github.com>
* fix(elf): bound compressed ELF section reads
`debug/elf` takes a section's decompressed size from that section's own
compression header, and a highly compressible stream really does deliver the
bytes that header promises, so `internal/saferio` does not help: it faithfully
allocates every one of them. A 2MB input file drives `elf.NewFile` to allocate
over 10GB and return no error, which is a fatal OOM rather than a recoverable
panic.
Which sections get read is not up to the caller. `elf.NewFile` always reads the
section-name string table, and `File.Symbols` reads `.symtab` plus whatever
section its `Link` field points at, so being selective about sections is not
enough to avoid it.
New `elfutil.NewFile` is a drop-in for `elf.NewFile` that rejects a declared
decompressed size over 128MB. Every production call site goes through it, and a
ruleguard rule keeps the next one from going direct.
The check runs in two parts, since `debug/elf` expands sections at two different
times. The section-name string table is the only one `elf.NewFile` expands
itself, so it is checked against the raw bytes before the call; everything else
is expanded lazily by `(*Section).Open` and is checked after the parse, where
names, types and decompressed sizes are already resolved.
Only the sections syft can actually reach are bounded, which keeps the guard
from costing real binaries. DWARF is excluded since nothing calls `File.DWARF`,
so a large compressed `.debug_info` no longer skips the whole file, and sections
`debug/elf` will not decompress anyway (`SHF_ALLOC`, `SHT_NOBITS`) are left
alone. The legacy `.zdebug` form is matched on the section name the way
`debug/elf` gates it rather than on the `ZLIB` magic, so an ordinary section
starting with those four bytes is not mistaken for a compressed one.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(elf): gate debug/buildinfo behind the compressed-section check
`debug/buildinfo.Read` opens ELF files with `debug/elf` itself, and `elf.NewFile`
expands the section-name string table as it parses, so the golang cataloger was
still reachable by the same bomb `elfutil` exists to stop. A 261KB fixture drove
1.4GB of allocation through `buildinfo.Read` and returned no error.
`elfutil.CheckSectionNameTable` is now exported for that case: callers that cannot
use `NewFile` because the `debug/elf` call is made for them inside another package.
Both `buildinfo.Read` call sites go through it, including the UPX-decompressed one.
Also corrects claims that did not hold up:
- the package doc's 2MB-to-10GB figure is not reachable with zlib (~1000:1), so it
now carries the measured 510KB-to-2.6GB, and names zstd's 32767:1 since that is
what makes the small inputs possible
- `.go.buildinfo` was listed as a hot-path section elfutil covers, but it is read
through `debug/buildinfo` and never touches `Section.Data`
- the graalvm comment claimed routing size rejections away from `*elf.FormatError`
improved reporting; both branches are skipped by the caller and only the
FormatError branch logs, so it did the opposite
- `sharedLibraries` logged short and truncated files as real ELF failures, since
`debug/elf` returns a bare `io.EOF` rather than an `*elf.FormatError` for those
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* added test comments around the negative cases
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* additional tests
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* better decomposition and comments
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* use a less brittle constant for error detection
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`readerAtAdapter.ReadAt` seeks and then issues a single `Read`, which breaks the
`io.ReaderAt` contract in both directions against the squashfs reader it exists
to wrap:
- `squashfs.File.Read` copies against the decompressed block length but advances
its block cursor by the nominal block size, so a block that decompresses short
silently stops copying and returns fewer bytes with a nil error. `ReadAt`
forbids that, and callers rely on it: anything decoding a fixed-size structure
off the result gets zero padding it has no way to detect and parses it as real
data. The GraalVM PE export table and the UPX block reader both size a buffer
from a header field and then ignore `n` entirely, so a crafted image drives
them straight through the padding.
- a read landing exactly on the end of the file returns a *full* buffer paired
with `io.EOF`. `bytes.Reader.ReadAt` returns nil there, and the callers that
treat any error as fatal were written against that, so squashfs-resident
binaries sized near a read boundary were being skipped outright.
`io.ReadFull` normalizes both: it fills the buffer across short reads, and it
clears the error once the buffer is full. A genuinely short tail is reported as
`io.EOF`, which is what `ReadAt` implementations return at the end of a file, and
what the buffering branch of `GetUnionReader` already returns.
Affects squashfs-backed sources (snaps), so in practice the binary catalogers
reading structure out of executables.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
jackson-mapper-asl, jackson-core-asl, and their sibling artifacts
(jackson-jaxrs, jackson-xc, jackson-smile) predate the convention of
embedding META-INF/maven/.../pom.properties in the jar (they were
built with Ant before ~2014). With no POM metadata to read, syft's
groupIDFromJavaMetadata falls through to using the artifact name
itself as the group ID, e.g.
pkg:maven/jackson-mapper-asl/jackson-mapper-asl@1.9.13
instead of the correct
pkg:maven/org.codehaus.jackson/jackson-mapper-asl@1.9.13
(confirmed against the published POM on Maven Central for all five
artifacts). Because the generated purl's namespace doesn't match the
vulnerability database's namespace for these packages, this causes
false negatives in downstream scanning (e.g. Grype cannot match known
CVEs such as CVE-2019-10202 against jackson-mapper-asl).
Add the five artifacts to DefaultArtifactIDToGroupID, the same known-
package-list fallback already used for other jars with incomplete
metadata (e.g. the existing ant-*, spring-ldap* entries).
Fixes#4598
Signed-off-by: ankit090701 <ankitanku090701@gmail.com>
* feat(golang): add extended-stdlib scope and include patterns for symbol capture
`golang.capture-symbols` decides how much symbol data lands in the SBOM for grype's reachability analysis. It's `none`, `stdlib`, or `all` today, and the useful middle is missing: `stdlib` stops at the standard library, `all` multiplies SBOM size.
A new `extended-stdlib` configurable covers stdlib plus everything under `golang.org/x/`:
```yaml
golang:
capture-symbols: extended-stdlib
```
Also, a new `capture-symbols-include` configurable for modules that are noisy in your binaries but not everyone's. It's unioned with whatever the scope selects, so it only ever widens:
```yaml
golang:
capture-symbols: extended-stdlib
capture-symbols-include:
- github.com/klauspost/**
```
Patterns are standard doublestar globs, which matters because module paths carry `/v2`-style suffixes:
```yaml
golang:
capture-symbols-include:
- github.com/klauspost/* # compress, but not compress/v2
- github.com/klauspost/** # both
- k8s.io/client-go # exact match only
```
Ordering is `none` < `stdlib` < `extended-stdlib` < `all`. The existing three values
and the `none` default are unchanged, and the include list is inert under `none`.
Presets compile into glob lists internally, so a single matcher answers "does this
module get symbols" instead of a preset branch sitting next to a separate glob branch.
An unrecognized `capture-symbols` value still falls back to `none`, but warns now
instead of doing it silently. A malformed include pattern warns and gets skipped.
One thing worth a look beyond the feature: the `Symbols` field description in the JSON
schema was wrong after this (it claimed only `all` and `stdlib` populate anything), and
that description lives in the already-published `16.1.10`. Rather than bump a version for
a sentence, `16.1.10` is amended in place and `schema/json/README.md` grows an explicit
exception for description-only changes: descriptions only, no shape change of any kind,
`$id` unchanged. Anything else still needs a bump. Happy to split that into its own PR if
you'd rather review the policy separately.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* refactor(golang): rename capture-symbols-include to capture-symbols-modules
The key's entries are go module paths, and `-include` sitting next to `capture-symbols` reads as plausibly taking symbol or package names instead. Those spellings parse and match nothing, which is quieter than the confusion `-include` was picked to avoid, so the name now says what the list holds.
`golang.CatalogerConfig.CaptureSymbolsModules` and `WithCaptureSymbolsModules` rename with it. Nothing behavioral changes; the key is new in this PR so there is no compatibility surface.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* feat(golang): match capture-symbols-modules across major version suffixes
`github.com/anchore/*` covered `github.com/anchore/syft` and silently stopped covering it the day it became `github.com/anchore/syft/v2`. The config keeps parsing, nothing warns, and symbols quietly go missing from the SBOM. Exact paths had the same hole: `github.com/klauspost/compress` did not cover `compress/v2` either, so no spelling short of `**` survived a major bump.
A major version suffix is part of a module's path but not part of its identity, so patterns are now matched against the module path both with and without it, using `module.SplitPathVersion` from `golang.org/x/mod` (already a direct dep, already used in this package for `PseudoVersion`).
```yaml
golang:
capture-symbols-modules:
- github.com/klauspost/* # compress and compress/v2
- github.com/klauspost/compress # same module at every major version
- github.com/klauspost/compress/v2 # v2 alone
```
Only a trailing suffix is a version, which is Go's own rule. In `github.com/anchore/syft/v2/thing` the `v2` is an ordinary path element naming a major subdirectory a nested module lives in, so it stays literal and `github.com/anchore/**/thing` is how you reach it. `/v0` and `/v1` are not valid suffixes and are left alone.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
The two "RACE is unset" cases only skipped the t.Setenv call, so they
inherited whatever RACE was in the environment. Running `make test` with
RACE=false exported job-wide flipped the CI-default case and failed.
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
with a cold fixture cache the integration suite builds and saves 18 docker
images across 36 sequential tests, which walks past `go test`'s default 10m
timeout and takes the fixture cache rebuild down with it. that suite now runs
`go test` directly with `-timeout=30m` (gotest.Tasks() has no timeout option),
plus `-count=1` since the built fixtures are the side effect we're actually
after and a test cache hit would skip producing them.
also adds `RACE` as one switch for the race detector across every suite:
- `RACE=false make test` drops `-race` from unit + integration and skips the
race smoke, worth doing on a cache rebuild where the wall clock is all
docker builds anyway
- `RACE=true` forces it on locally
- unset behaves as before: on in CI, off locally and on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* report each hardlink as its own file when scanning images
image scans previously collapsed a set of hardlinks onto a single file, so only
one path per inode showed up in results. dir scans report every hardlink path,
which made image vs dir SBOMs of the same filesystem diverge (and produce
different SPDX `packageVerificationCode` values for packages that own hardlinked
files).
now both image resolvers (squash and all-layers) surface each hardlink at its
own path as a regular file bound to the target's content, matching dir scans.
user-facing impact:
- SBOMs for images containing hardlinks will list more `file` entries
- SPDX `packageVerificationCode` values change for affected packages, now
matching the equivalent `dir:` scan
- adds `file.NewVirtualLocationFromImage` to the public API
fixes#5019
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix busybox test assertion
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
parseErlangString advances past a backslash escape and then checks
len(data) >= *i before reading the escaped byte. That condition is
almost always true (it only turns false once *i runs off the end),
so the intended out-of-range guard fires on the very first escape
character it sees instead of only at EOF. Any rebar.lock or OTP
resource file containing a backslash in a quoted string (a Windows
git path, an escaped quote, anything) fails to parse and the whole
file, and every package in it, gets dropped.
Flip the comparison to *i >= len(data) so the guard only trips when
the escape is genuinely truncated, and add a regression test for a
string with an escaped quote.
Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
