* fix(golang): bound UPX decompression by what the input can justify
A UPX block's `sz_unc` was passed straight to `make([]byte, n)`, so four bytes of
attacker input spanned the full uint32 range. A ~300KB file could drive multi-GB
resident memory, which Go reports as a fatal runtime OOM that no `recover` on this
path can contain. Syft parses binaries out of arbitrary images, so that is reachable
from any scan.
Two bounds do the work, and UPX's own invariants make them exact:
- all blocks together reconstruct `p_filesize`, so a running remainder caps the sum.
Bounding only per-block would leave the total at (block count x original size), so
the remainder is the part that actually closes it
- `p_filesize` sizes the output buffer, so it is capped both absolutely and against
the size of the file on disk. The absolute cap alone left a ~60 byte header able to
claim 500MB; the ratio alone cannot work either, since LZMA encodes a run of N
equal bytes in O(log N) and a `go:embed` of 120MB of zeros legitimately packs 209x
The output buffer is also allocated on the first block that survives validation
rather than up front, so a header claiming a large size with nothing decodable behind
it costs nothing.
Alongside that, several ways the block loop could be steered off its own buffer:
- `parseELFPTLoadOffsets` bounds-checked program headers with `phStart+phentsize >
len(buf)`, which overflows for a `p_offset` near 2^64 and lets an out-of-range entry
through into a slice index. Switched to the subtraction form, and a `phentsize`
shorter than an ELF64 program header is rejected since the reads use fixed offsets
- block placement had the same overflow shape, and on failure it skipped the copy and
fell through. `outputOffset` derives from the rejected offset, so a `p_offset` at the
uint64 ceiling wrapped it to 0 and the next block landed on the reconstructed ELF
header, still returning success. Out-of-range placement now ends the block chain,
keeping the blocks already placed since those often carry `.go.buildinfo`
- the UPX 2-byte LZMA header carries `lc`/`lp` as nibbles and `pb` as three bits, so
they can hold values LZMA does not permit. Folded into the props byte they wrap
(`lc=15, lp=15, pb=7` gives 465, which truncates to 209) and the stream mis-decodes
instead of failing
- blocks decode directly into a slice of the output buffer instead of a per-block
buffer that is then copied, so one block no longer doubles peak memory
- the block count is capped: the size budget alone still allows millions of 12-byte
blocks, each spinning up an LZMA reader
Verified against the `image-small-upx` fixture, a real `upx --best --lzma` binary:
four blocks, max `sz_unc` equal to `p_blocksize` exactly, blocks summing to 0.68 of
`p_filesize`, and a 1.96x expansion against the packed file, so every bound holds with
headroom on genuine output.
Note `p_blocksize` is deliberately not used as a ceiling on `sz_unc`. It adds nothing
the remainder does not already cover, and real output sits at exactly `p_blocksize`,
so it would run with zero headroom against a value the format does not promise.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(golang): report UPX binaries we cannot unpack as unknowns
A packed Go binary we fail to unpack means its packages are silently missing from the
SBOM. `getBuildInfo` checked the decompression error for nil and discarded it, so a
rejected file surfaced only the original `buildinfo.Read` error, and that one is
deliberately silenced because it is usually just "not a Go binary".
`decompressUPX` now marks the cases worth reporting with `errUPXDecompress`: it got
past the header and the method dispatch and still could not unpack the file. Everything
else stays quiet, which matters more than it sounds. `upx` defaults to NRV2B unless
`--lzma` is passed and only LZMA is implemented here, so treating any decompression
failure as reportable would attach a golang-cataloger unknown to every packed non-Go
binary in an image. Making the reportable case opt in rather than the quiet case a
growing list also means a guard added later is silent by default.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* refactor(golang): drop the vendored xcoff parser
The only thing the golang cataloger read from it was `TargetMachine`, which is the same
two magic bytes `getGOARCHFromBin` had already matched on to dispatch there. So a full
XCOFF walk (string table, symbol table, every relocation table) ran to recover a value
that was already in hand.
`getGOARCHFromBin` reads those two bytes directly now. That is also strictly more
correct: `xcoff.NewFile` bailed with "no symbol table" when `symptr == 0`, so a stripped
XCOFF binary reported no arch at all.
Fixtures move to `testdata/xcoff/`, since they are no longer a package's own testdata.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* feat(internal/file): report a reader's real size behind one helper
Bounds get written as "does this claim exceed what the reader actually holds", and the
subtlety is that a reader's own answer cannot be trusted: an `io.SectionReader` reports
the nominal length it was built with, so one constructed with `1<<63-1` will happily
claim to hold 8EB. Confirming the last byte is readable is what separates a real size
from a nominal one.
Returns `(int64, bool)` rather than a bare size, since there are five ways to not know
and collapsing them to 0 makes every caller reinvent "0 means stop bounding".
The doc also names the wrapper trap: a type embedding `io.ReaderAt` as an interface
promotes only `ReadAt`, so it answers no size at all however large the reader beneath it
is, and a bound written against it silently does nothing.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(golang): bound every section a reader can expand, not just the name table
`CheckSectionNameTable` bounds the one section `elf.NewFile` expands while parsing, which
is enough for a caller that only parses. It is not enough for one that goes on to read
sections: those are expanded lazily, so a 260KB ELF declaring a compressed `.symtab`
drove 1.3GB of allocation through `goversion`, which opens the file with `debug/elf`
inside its own package and cannot be routed through `elfutil.NewFile`.
`CheckAllSections` is that gate, and is deliberately named as the superset so the
relationship to the narrow one is structural rather than documented.
Also exports `ErrDeclaredSizeExceeded`. A refusal here costs the SBOM a package, so
callers report it as an unknown, and matching on an error string to decide that is not
something to build a reporting policy on.
A ruleguard rule now flags `buildinfo.Read` and `version.ReadExeFromReader` outside
`scan_binary.go`. Both open `debug/elf` internally and are only bounded there by the
wrappers that gate the reader first, which the existing rule on `elf.NewFile` could not
see through.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* feat(internal/spillbuf): sparse offset-addressed buffer that spills to disk
For output that arrives out of order, at offsets the input itself declares: a
decompressor placing extents, an archive rebuilding a file from chunks. A plain `[]byte`
cannot do that without either pre-sizing to a length the input claims or growing to the
furthest offset it names, and both hand a hostile input an allocation knob.
The load-bearing property is that it reports only what it actually stored. `Size` is the
contiguous run written from offset zero and a read past it is `io.EOF`, never the zeros
an unwritten region would hand back for free. Sparse storage serving holes as real
content is what lets a write offset stand in for output nothing produced, so anything
sizing an allocation against this reader (`saferio` in `debug/elf` does exactly that)
stays bounded by work actually done.
`FirstGap` answers where the next write fits, so callers filling holes do not need the
extent bookkeeping and cannot re-derive that rule wrongly, and it only reports gaps
lying entirely inside `[0, within)`. The extent type is unexported for the same reason.
The rest of the contract worth stating: reads and writes after `Close` fail with
`os.ErrClosed` rather than looking like an empty buffer, and a write that fails leaves
the buffer unchanged.
Memory is bounded by the limit rather than by how much is written: filling 1MB and
filling 64MB cost about the same. What lands on disk past that limit is the caller's
bound to set, not this package's. Growth is geometric and capped, because sizing the
tier to exactly what each write needs is quadratic once a caller streams through it in
small chunks.
The golang cataloger is the only consumer today, and nothing else in the tree has this
shape.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(golang): rebuild UPX binaries through spillbuf and read the reconstruction everywhere
Two amplification paths were still reachable. A 128 byte input allocated 16MB by
trusting `sz_cpr` before weighing it against the bytes the file actually holds, and the
reconstruction was assembled in a `[]byte` sized from the header's declared original
size, so a 301KB input allocated 300MB.
The reconstruction is a `spillbuf.Buffer` now, which is what makes the second one go
away: blocks are placed at the offsets the file declares, nothing is pre-sized from a
claim, and the reader's length is the contiguous run actually rebuilt. That last part is
the invariant everything downstream leans on. A reader reporting the declared size would
serve the holes between placed blocks as zeros it never stored, and a 64 byte block
parked at `p_filesize-64` then made a 300KB input hand out a 64MB reader: measured, that
drove 269MB through `getBuildInfo` and returned nothing, silently.
Placement and storage are separate concerns. `upx.go` places blocks at the offsets the
file declares and asks the buffer how much was rebuilt and where the next gap is; it
knows nothing about temp files or memory limits.
The remaining bounds are ratios against the input wherever they can be, since an
absolute ceiling only ever drops a large legitimate binary. The one absolute left is on
`p_filesize`, because padding an input is nearly free, and the LZMA dictionary is paid
for in its own block's `sz_cpr`.
Interface change: `scanFile` takes a context and hands back the reconstruction for each
binary that turned out to be packed, and the caller owns it. Everything after the build
info read (crypto settings, arch, symbols, the version scan) reads it, so a packed
binary no longer reports its packages with no symbols at all.
Also narrows what gets reported. This cataloger runs on every executable in an image, so
reporting every `getBuildInfo` error attached an unknown to every corrupt ELF, odd PE and
non-Go Mach-O slice in it. Now only four gaps, the ones that cost the SBOM something
syft could have had: a packed file we could not decode (`errUPXDecompress`), a header
the size bounds refused (`errUPXSizeRefused`), a reconstruction that came up short
(`errUPXPartial`), and an ELF expansion `elfutil` declined
(`ErrDeclaredSizeExceeded`). An unimplemented UPX method stays quiet on purpose: `upx`
defaults to NRV2B and only LZMA is implemented here, so reporting it would attach an
unknown to most packed non-Go binaries in an image.
That filtering is a new pattern. No other cataloger picks which of its errors become
unknowns, and the reason this one does is that it runs on every executable in an image
rather than on files a glob already selected.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore: ignore local spec/ planning notes
`/specs` was already ignored; `/spec` is the same thing under the singular name and was
not, so working notes landed in commits.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* refactor(golang): name what the unpack hands back, not where it stores it
the scan's signatures said `*spillbuf.Buffer` all the way down, which told
every reader below the scan that it cares whether the reconstruction lives in
memory or on disk. it does not. two interfaces instead, each naming only what
its side actually uses:
- `unpackedContents` for the consumers: `ReaderAt`, `Closer`, and a `Size` that
is the contiguous run rebuilt from zero
- `blockSink` for the decoder, which additionally needs `WriterAt` and
`FirstGap` to place blocks
this costs one thing worth flagging. `Close` on a nil `*spillbuf.Buffer` was
nil-safe, so callers just deferred it; widened into an interface, a nil pointer
becomes a non-nil interface and the same call panics. `closeUnpacked` absorbs
that, and it is now the only way an owner releases contents. `readerFor`,
`seekerFor` and `closeUnpacked` are the three places that resolve the nil.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* docs(elfutil): say that CheckAllSections is the decompression-bomb gate
the doc led with "bounds every section syft can drive debug/elf into
decompressing", which describes the mechanism without ever naming what it is
defending against. a reader landing on the call site could not tell that this
is the decompression-bomb check.
state the attack instead: a compressed section header declares its own
decompressed size, debug/elf believes that number and allocates it on open, and
nothing forces it to match what the compressed bytes actually yield. the 260KB
ELF that drove 1.3GB of allocation now lives in the doc rather than only at the
one call site that happened to mention it.
comments only.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* docs(golang): move the overflow note onto the bounds test it describes
the "subtraction form" note sat above the phStart assignment with ten lines of
wrap analysis between it and the `if` it was talking about, so the subtraction
it names reads as missing. it is `hdrLen-phStart < phentsize`.
move it onto that test and name the form it is avoiding (`phStart+phentsize >
hdrLen`, which can overflow past the buffer end) so the referent is not several
paragraphs away.
comments only.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(golang): drop the internal/ README left over from xcoff
that README existed to record where the vendored xcoff package came from and
that Go keeps it internal, which is a provenance note worth having while
third-party code sat there. dropping xcoff took the reason with it, and
repurposing it to point at gotestdata just made it redundant: gotestdata has
its own README covering why it is under internal/, why it is not testdata/, and
what belongs in each.
internal/ now holds one self-documenting directory.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* feat(golang): add extended-stdlib scope and include patterns for symbol capture
`golang.capture-symbols` decides how much symbol data lands in the SBOM for grype's reachability analysis. It's `none`, `stdlib`, or `all` today, and the useful middle is missing: `stdlib` stops at the standard library, `all` multiplies SBOM size.
A new `extended-stdlib` configurable covers stdlib plus everything under `golang.org/x/`:
```yaml
golang:
capture-symbols: extended-stdlib
```
Also, a new `capture-symbols-include` configurable for modules that are noisy in your binaries but not everyone's. It's unioned with whatever the scope selects, so it only ever widens:
```yaml
golang:
capture-symbols: extended-stdlib
capture-symbols-include:
- github.com/klauspost/**
```
Patterns are standard doublestar globs, which matters because module paths carry `/v2`-style suffixes:
```yaml
golang:
capture-symbols-include:
- github.com/klauspost/* # compress, but not compress/v2
- github.com/klauspost/** # both
- k8s.io/client-go # exact match only
```
Ordering is `none` < `stdlib` < `extended-stdlib` < `all`. The existing three values
and the `none` default are unchanged, and the include list is inert under `none`.
Presets compile into glob lists internally, so a single matcher answers "does this
module get symbols" instead of a preset branch sitting next to a separate glob branch.
An unrecognized `capture-symbols` value still falls back to `none`, but warns now
instead of doing it silently. A malformed include pattern warns and gets skipped.
One thing worth a look beyond the feature: the `Symbols` field description in the JSON
schema was wrong after this (it claimed only `all` and `stdlib` populate anything), and
that description lives in the already-published `16.1.10`. Rather than bump a version for
a sentence, `16.1.10` is amended in place and `schema/json/README.md` grows an explicit
exception for description-only changes: descriptions only, no shape change of any kind,
`$id` unchanged. Anything else still needs a bump. Happy to split that into its own PR if
you'd rather review the policy separately.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* refactor(golang): rename capture-symbols-include to capture-symbols-modules
The key's entries are go module paths, and `-include` sitting next to `capture-symbols` reads as plausibly taking symbol or package names instead. Those spellings parse and match nothing, which is quieter than the confusion `-include` was picked to avoid, so the name now says what the list holds.
`golang.CatalogerConfig.CaptureSymbolsModules` and `WithCaptureSymbolsModules` rename with it. Nothing behavioral changes; the key is new in this PR so there is no compatibility surface.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* feat(golang): match capture-symbols-modules across major version suffixes
`github.com/anchore/*` covered `github.com/anchore/syft` and silently stopped covering it the day it became `github.com/anchore/syft/v2`. The config keeps parsing, nothing warns, and symbols quietly go missing from the SBOM. Exact paths had the same hole: `github.com/klauspost/compress` did not cover `compress/v2` either, so no spelling short of `**` survived a major bump.
A major version suffix is part of a module's path but not part of its identity, so patterns are now matched against the module path both with and without it, using `module.SplitPathVersion` from `golang.org/x/mod` (already a direct dep, already used in this package for `PseudoVersion`).
```yaml
golang:
capture-symbols-modules:
- github.com/klauspost/* # compress and compress/v2
- github.com/klauspost/compress # same module at every major version
- github.com/klauspost/compress/v2 # v2 alone
```
Only a trailing suffix is a version, which is Go's own rule. In `github.com/anchore/syft/v2/thing` the `v2` is an ordinary path element naming a major subdirectory a nested module lives in, so it stays literal and `github.com/anchore/**/thing` is how you reach it. `/v0` and `/v1` are not valid suffixes and are left alone.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Mirrors the existing RHEL `ExtendedSupport` detector for Ubuntu so grype can
auto-enable its ESM distro channel on Pro/ESM-enabled scans.
`findUbuntuFeatures` sets `linux.Release.ExtendedSupport` when the scanned OS is
Ubuntu and any of these hold:
- an uncommented apt source (classic `.list` or DEB822 `.sources`, or the
ubuntu-advantage auth entry) referencing `esm.ubuntu.com`
- an enabled `esm-infra`/`esm-apps` service in the ubuntu-advantage
`status.json` cache
- an installed dpkg package whose version carries an `[~+]esm` pocket suffix
(the usual in-image signal, since containers are rarely Pro-attached)
The package fallback reads `s.Artifacts.Packages` from the `DetectFeatures`
SBOM callback, which runs after package cataloging, so no new plumbing was
needed. Conservative by design: plain Ubuntu and disabled/commented ESM
evidence stay false.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
adds dependency-of relationships between elixir locked packages, matching how other
ecosystem catalogers (alpine, arch, debian, redhat, python) express the
dependency graph via the shared dependency.Processor/Specifier mechanism.
Signed-off-by: Chris Greeno <cgreeno@gmail.com>
BPL (Borland Package Library) files are standard PE/DLL format used by
Delphi and C++Builder. Adding the extension to the glob list so syft
picks them up during directory scans without users needing to rename
to .dll first.
---------
Signed-off-by: jfjrh2014 <jfjrh2014@gmail.com>
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
Co-authored-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
* chore: centralize temp files and prefer streaming IO
Catalogers that create temp files ad-hoc can easily forget cleanup,
leaking files on disk. Similarly, io.ReadAll is convenient but risks
OOM on large or malicious inputs.
Introduce internal/tmpdir to manage all cataloger temp storage under
a single root directory with automatic cleanup. Prefer streaming
parsers (bufio.Scanner, json/yaml.NewDecoder, io.LimitReader) over
buffering entire inputs into memory. Add ruleguard rules to enforce
both practices going forward.
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* chore: go back to old release parsing
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* simplify to limit reader in version check
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* chore: regex change postponed
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* simplify supplement release to limitreader
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
---------
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* migrate fixtures to testdata
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: correct broken symlinks after testdata migration
The migration from test-fixtures to testdata broke several symlinks:
- elf-test-fixtures symlinks pointed to old test-fixtures paths
- elf-test-fixtures needed to be renamed to elf-testdata
- image-pkg-coverage symlink pointed to test-fixtures instead of testdata
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: handle missing classifiers/bin directory in Makefile
The clean-fingerprint target was failing when classifiers/bin doesn't
exist (e.g., on fresh clone without downloaded binaries).
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: add gitignore negation for jar/zip fixtures in test/cli
The jar and zip files in test/cli/testdata/image-unknowns were being
gitignored by the root .gitignore patterns. This caused them to be
untracked and not included when building docker images in CI, resulting
in Test_Unknowns failures since the test expects errors from corrupt
archive files that weren't present.
Add a .gitignore in test/cli/testdata to negate the exclusions for
these specific test fixture files.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* switch fixture cache to v2
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: update expected versions for rebuilt fixtures
Update test expectations for packages that have been updated in
upstream repositories when docker images are rebuilt:
- glibc: 2.42-r4 → 2.43-r1 (wolfi)
- php: 8.2.29 → 8.2.30 (ubuntu/apache)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* upgrade go
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: add go-shlex dependency for testdata manager tool
The manager tool in syft/pkg/cataloger/binary/testdata/ imports
go-shlex, but since it's in a testdata directory, Go doesn't track
its dependencies. This caused CI failures when go.mod didn't
explicitly list the dependency.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* refactor: move binary classifier manager to internal/
Move the manager tool from testdata/manager to internal/manager so
that Go properly tracks its dependencies. Code in testdata directories
is ignored by Go for dependency tracking, which caused CI failures
when go.mod didn't explicitly list transitive dependencies.
This is a cleaner solution than manually adding dependencies to go.mod
for code that happens to live in testdata.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: add gitignore negations for test fixtures blocked by root patterns
Multiple test fixtures were being blocked by root-level gitignore patterns
like bin/, *.jar, *.tar, and *.exe. This adds targeted .gitignore files with
negation patterns to allow these specific test fixtures to be tracked:
- syft/linux/testdata/os/busybox/bin/busybox (blocked by bin/)
- syft/pkg/cataloger/java/testdata/corrupt/example.{jar,tar} (blocked by *.jar, *.tar)
- syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/**/bin/go (blocked by bin/)
- syft/pkg/cataloger/bitnami/testdata/no-rel/.../bin/redis-server (blocked by bin/)
Also updates the bitnami test expectation to include the newly required
.gitignore files in the test fixture.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: update glibc version expectation (2.43-r1 -> 2.43-r2)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* add capability drift check as unit step
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* dont clear test observations before drift detection
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* bump stereoscope commit to main
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
A CI failure was observed where a generated file was only partly written
when the CI job immediately tried to read it. Put in an fs.Sync call to
eliminate this flakiness.
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* feat: Add license enrichment from pypi to python packages
* Implement license caching and improve test coverage
---------
Signed-off-by: Tim Olshansky <456103+timols@users.noreply.github.com>
* ref: close source in test and examples
Signed-off-by: Kudryavcev Nikolay <kydry.nikolau@gmail.com>
* ref: pretty file/directory source resolver (make them more similar)
Signed-off-by: Kudryavcev Nikolay <kydry.nikolau@gmail.com>
* ref: move absoluteSymlinkFreePathToParent to file resolver
Signed-off-by: Kudryavcev Nikolay <kydry.nikolau@gmail.com>
* revert breaking change
Signed-off-by: Kudryavcev Nikolay <kydry.nikolau@gmail.com>
---------
Signed-off-by: Kudryavcev Nikolay <kydry.nikolau@gmail.com>