mirror of
https://github.com/anchore/syft.git
synced 2026-10-30 05:52:25 +01:00
main
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
   
|
4b8a3db5f3 |
Add nested archive cataloging (#5260)
Signed-off-by: Kudryavcev Nikolay <kydry.nikolau@gmail.com> Signed-off-by: Thomas Bechtold <thomasbechtold@jpberlin.de> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> Signed-off-by: Keith Zantow <kzantow@gmail.com> Co-authored-by: Kudryavcev Nikolay <kydry.nikolau@gmail.com> Co-authored-by: Thomas Bechtold <thomasbechtold@jpberlin.de> Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com> |
||
|
Alex Goodman
|
6c9bb732dd |
ci: run the full unit suite on windows (#5370)
* ci: run the full unit suite on windows via make unit Drops the curated `unit:windows` subset; the windows job now bootstraps like linux and runs `make unit`. Exceptions get added as failures show up. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * ci: skip fixture refresh for windows unit tests binny has no windows release, so the bootstrap action can't install the tools the fixture refresh needs. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * test: skip linux container fixtures when docker can't run them Windows CI runners only run windows containers, so image fixtures go through `testutils.SkipWithoutLinuxContainers`. No behavior change on linux/mac. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix: close leaked file handles and posix squashfs paths on windows - output writers are now closed on early returns in `scan`/`convert` (open handles block cleanup on windows) - `newSBOMMultiWriter` used `path.Dir` on host paths, so output subdirectories were never created on windows - squashfs walk paths were built with `filepath.Join` - skip a few posix-only test cases (dir perms, xdg fallback, go-rpmdb ndb handle leak upstream) Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix(golang): resolve go source module dir natively on windows - `parseGoModFile` passed the resolver's posix (`/c/...`) path to the go toolchain, so go source resolution silently found nothing on windows - skip container-built fixtures (go archs, java builds) and golden image tarballs on windows; stereoscope's layer cache filenames contain `:`, which windows rejects Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix: keep posix paths posix on windows - `ChrootContext.ToChrootPath` nested volumes (`D:\cwd\C:\root`) when the scan root was on another drive, so the prefix was never trimmed - bitnami, python wheel/egg, and alpm parsers used `filepath` on posix location paths - alpm backup-file lookups silently missed on windows Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * ci: pull the test fixture cache for windows unit tests Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix: don't map virtual file tree paths through the host on windows `FiletreeResolver` without a chroot (snaps) ran request paths through `filepath.Abs`, rooting them on the current volume (`/payload.txt` -> `/d/payload.txt`), so snap scans on windows found nothing. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix: match tar entry globs as posix on windows - `matchesAnyGlob` used `doublestar.PathMatch` (host separator) on tar entry names, so tar-wrapped java archives silently produced nothing on windows - test helpers now treat binary snippet keys and `FromFile` locations as posix Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * incorporate windows path fix Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * incorporate go-rpmdb close fix Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * close squashfs fixture writers before finalizing - windows dir listings can report a zero size for files still open, so the manifest was packed empty (also adds a temporary windows diagnostic for the binary cataloger) Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * temp: surface windows binary cataloger diagnostics Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix: keep directory symlink targets posix on windows - the directory indexer joined symlink targets with native separators, so links in the posix file tree pointed nowhere and globs on link names found nothing - binary fixture logical keys used native separators, so nested snippets were picked as the fixture dir on windows Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * keep addSymlinkToIndex under the funlen limit Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * drop windows skips covered by the stereoscope and go-rpmdb bumps Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> --------- Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> |
||
|
Alex Goodman
|
3325493831 |
chore: pin the bounds from the unbounded allocation work (#5298)
* build(deps): bump go-make to v0.8.1
Picks up golangci-lint 2.13.2 (built with go1.27) in place of 2.11.4 (go1.26).
The old build panics outright on a go1.27 GOROOT with "file requires newer Go
version go1.27", and an older mise-shim build fails the quieter way: it loses
type info and reports zero findings with exit 0, so a ruleguard run comes back
green without having evaluated anything.
Also folds in what the newer linter flags:
- `reflect.Ptr` -> `reflect.Pointer` in three files, applied by --fix
- `tw.CellFormatting{MergeMode: ...}` -> `tw.CellMerging{Mode: ...}` in the
cataloger info table, for a staticcheck SA1019 deprecation
Note the deprecation text names `CellConfig.CellMerging.Mode`, but the field is
`CellConfig.Merging`; `CellMerging` is the type. That command path has no test
coverage, so the swap was checked by rendering the same hierarchical-merge table
both ways and diffing: byte identical.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(lint): tune goconst for data tables
goconst got stricter in golangci-lint 2.13 and now counts strings in map
literals and struct fields, which lights up every lookup table in the repo. It
was already enabled, so this is main going red on a linter bump rather than
anything new in the tree: 101 findings, and the default threshold of 3 means a
vendor name appearing in three rows of a table reads as a magic constant.
Raising `min-occurrences` to 5 takes that to 14. The rest split by whether the
repeated string is data or an actual constant:
- `cpegenerate` is excluded by path. It maps artifact names to vendors,
products, and maven group ids, and repeats strings 35, 83, even 293 times, so
no threshold reaches it. Hoisting those would trade a readable table for
indirection.
- the two SBOM fixture builders in `format/internal/testutil` get a
function-level nolint each. The repeated names and versions are what the
golden files assert against.
- `mysql-cluster` and `php` were genuine repeated literals and are now
constants.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(test): extract measureAlloc into a shared helper
The allocation-budget tests added during the unbounded allocation work grew a
copy of the same helper in every package they touched: four identical
`measureAlloc` definitions plus two more spelled out inline against
`runtime.MemStats`.
Now `testutils.MeasureAlloc`, at the repo root so `internal/spillbuf` can reach
it too. The doc comment carries the two things that were only recorded in some
of the copies: TotalAlloc is process-wide so callers must not use t.Parallel,
and a budget assertion only means something if the fixture would really blow
past it unguarded, so measure the unguarded path as well.
No behavior change, the body was identical in all six.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore(lint): add ruleguard rules for overflow and swallowed EOF
Two rules out of the recent unbounded allocation work (#5187, #5195, #5267,
#5291, #5292, #5293, #5294).
`noOverflowingBoundsCheck` flags `offset + size > limit` (and <, <=, >=) where
an operand is unsigned, neither addend is constant, and the operands are at
least 32 bits. Unsigned addition wraps, so an offset out of a file header can
carry the sum past the wrap point and pass a check it should have failed. The
fix is to subtract against the limit instead. Constant addends are excluded
because `pos+4 <= uint32(len(buf))` is the normal way to step a cursor, and the
32-bit floor drops nibbles widened for a comparison. Would have caught four
sites across #5195 and #5291.
`noSwallowedEOF` flags `return nil` / `return nil, nil` on an io.EOF check. The
caller gets a zero value it cannot tell from real data, and anything decoding a
fixed-size struct off it parses the padding as content. That is the shape behind
#5186.
Both decide from the expression alone, and writing the fix deletes the pattern,
so neither needs a suppression anywhere in the tree. That was the bar: rules for
the allocation and decompression classes were tried and dropped, because the
bound they look for lives in a preceding statement (or, for a decompressor
constructor, always downstream), so no correct implementation can silence them.
That would have meant ~15 permanent nolints on code that is already right. The
rules here that do work don't judge whether a bound exists, they route to a
wrapper that takes one: noDirectTempFiles, noDirectELFOpen.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: pin allocation budgets on the alpm, deb, and changelog bounds
The bounds added for #5292, #5293 and #5294 are asserted by the error they
return. That keeps passing if the bound is removed and something else rejects
the input later, which is the failure these were written for. Each of these
measures bytes instead, and proves the fixture is really a bomb first so a
fixture that quietly stops being one fails rather than passing.
- alpm mtree: 16MB of newlines, well inside the 64MB byte cap, costs 16.5GB
uncapped against 238MB with maxMtreeLines. That 16.5GB is the measurement the
line cap was added for and it was recorded only in the commit message. Note
what the cap actually buys: 300k entries still cost a few hundred MB, per
concurrent cataloger.
- deb members: 1.28GB against 392KB. Both arms read with io.ReadAll on purpose,
since the stream itself never holds much whatever the cap says; what the cap
protects is a consumer that buffers, and the real one is a tar reader building
entries.
- snap changelog: 1.28GB against 175KB. This one has no explicit cap at all. It
is held by bufio.Scanner refusing a token past MaxScanTokenSize, which is
incidental, so switching to io.ReadAll or a bufio.Reader to pick up long lines
would remove it silently.
Nothing added for the kernel module reader: its copy is already
io.Copy(tempFile, io.LimitReader(rc, max+1)), so peak disk is bounded by
construction and heap is streaming. A heap budget would assert the wrong
resource, and the three existing subtests cover the cap, at-cap fidelity, and
spill cleanup.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: pin allocation budgets on the ai and alpm byte caps
Follows the same pattern as the alpm line cap: measure bytes, and prove the
fixture is really a bomb first so one that quietly stops being a bomb fails
rather than passes.
- safetensors: 252MB against 2.6KB. The existing over-cap case deliberately
omits the body, since the guard rejects on the declared length before reading
anything. That is the right test for the guard but it would keep passing if
the check moved after the read, so this one delivers the bytes.
- gguf: 268MB against 168MB. copyHeader io.Copy's whatever it is handed, so
maxHeaderSize holds only because every caller wraps the reader first. This
pins that the limit is where the bound lives, so dropping the wrapper at a
call site fails here.
- alpm byte cap: 740MB against 132MB, the counterpart to the line cap budget.
Worth knowing for anyone writing the next one of these: a single enormous line
does not exercise the alpm byte cap. go-mtree runs its own bufio.Scanner and
refuses a token past 64KB, so that fixture fails as "token too long" before the
byte cap is consulted. The fixture here uses ordinary 1KB lines, enough of them
to pass 64MB while staying under the 300k line cap, so the byte cap is actually
the thing under test.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: assert bytes retained instead of allocated for gguf copyHeader
the bounded arm writes into a bytes.Buffer, whose doubling growth costs a
multiple of what it holds and lands on a different power of two depending on
the platform. it measured 168MB locally and 320MB on CI, past the budget.
buf.Len() states the same property deterministically.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
|
||
 Will Murphy
|
e38851143e |
chore: centralize temp files and prefer streaming IO (#4668)
* chore: centralize temp files and prefer streaming IO Catalogers that create temp files ad-hoc can easily forget cleanup, leaking files on disk. Similarly, io.ReadAll is convenient but risks OOM on large or malicious inputs. Introduce internal/tmpdir to manage all cataloger temp storage under a single root directory with automatic cleanup. Prefer streaming parsers (bufio.Scanner, json/yaml.NewDecoder, io.LimitReader) over buffering entire inputs into memory. Add ruleguard rules to enforce both practices going forward. Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com> * chore: go back to old release parsing Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com> * simplify to limit reader in version check Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com> * chore: regex change postponed Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com> * simplify supplement release to limitreader Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com> --------- Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com> |
||
|
Alex Goodman
|
b5e85c3ea5 |
chore: migrate fixtures to testdata (#4651)
* migrate fixtures to testdata Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix: correct broken symlinks after testdata migration The migration from test-fixtures to testdata broke several symlinks: - elf-test-fixtures symlinks pointed to old test-fixtures paths - elf-test-fixtures needed to be renamed to elf-testdata - image-pkg-coverage symlink pointed to test-fixtures instead of testdata Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix: handle missing classifiers/bin directory in Makefile The clean-fingerprint target was failing when classifiers/bin doesn't exist (e.g., on fresh clone without downloaded binaries). Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix: add gitignore negation for jar/zip fixtures in test/cli The jar and zip files in test/cli/testdata/image-unknowns were being gitignored by the root .gitignore patterns. This caused them to be untracked and not included when building docker images in CI, resulting in Test_Unknowns failures since the test expects errors from corrupt archive files that weren't present. Add a .gitignore in test/cli/testdata to negate the exclusions for these specific test fixture files. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * switch fixture cache to v2 Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * test: update expected versions for rebuilt fixtures Update test expectations for packages that have been updated in upstream repositories when docker images are rebuilt: - glibc: 2.42-r4 → 2.43-r1 (wolfi) - php: 8.2.29 → 8.2.30 (ubuntu/apache) Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * upgrade go Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix: add go-shlex dependency for testdata manager tool The manager tool in syft/pkg/cataloger/binary/testdata/ imports go-shlex, but since it's in a testdata directory, Go doesn't track its dependencies. This caused CI failures when go.mod didn't explicitly list the dependency. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * refactor: move binary classifier manager to internal/ Move the manager tool from testdata/manager to internal/manager so that Go properly tracks its dependencies. Code in testdata directories is ignored by Go for dependency tracking, which caused CI failures when go.mod didn't explicitly list transitive dependencies. This is a cleaner solution than manually adding dependencies to go.mod for code that happens to live in testdata. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix: add gitignore negations for test fixtures blocked by root patterns Multiple test fixtures were being blocked by root-level gitignore patterns like bin/, *.jar, *.tar, and *.exe. This adds targeted .gitignore files with negation patterns to allow these specific test fixtures to be tracked: - syft/linux/testdata/os/busybox/bin/busybox (blocked by bin/) - syft/pkg/cataloger/java/testdata/corrupt/example.{jar,tar} (blocked by *.jar, *.tar) - syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/**/bin/go (blocked by bin/) - syft/pkg/cataloger/bitnami/testdata/no-rel/.../bin/redis-server (blocked by bin/) Also updates the bitnami test expectation to include the newly required .gitignore files in the test fixture. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * test: update glibc version expectation (2.43-r1 -> 2.43-r2) Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * add capability drift check as unit step Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * dont clear test observations before drift detection Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * bump stereoscope commit to main Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> --------- Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> |