* fix: report scanner errors instead of returning a partial parse as complete
A line longer than `bufio.Scanner`'s 64KB buffer ends the scan with `ErrTooLong`, which only `Err()` reports. Parsers now return what they found along with the error (recorded as an unknown for the file), or log it where there is no error to return.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: allow lines up to 1MB in line-oriented package metadata parsers
Adds `internal.NewLineScanner`, a `bufio.Scanner` capped at 1MB per line rather than the 64KB default, the same cap the alpm parser already uses. Long single-line fields (descriptions, license text, dependency lists) now parse instead of ending the scan early.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: use io.ReadFull where a short read was silently accepted
- dotnet bundle 7-bit int decoding could read a zero-byte result as a zero byte
- wordpress plugin header parsing included the unread tail of its buffer
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: finish moving line scanners to the 1MB cap and checking scanner errors
- java manifest, pom.properties, Gemfile.lock, requirements.txt, homebrew formula, alpm and the purl/cpe decoders now use `internal.NewLineScanner`
- snap kernel changelog reports a scan error instead of silently dropping the base kernel package
- the attestation output reader keeps draining the pipe after a scan error so cosign can't block on a full pipe
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: stop logging the full setup.py line once per dependency match
The debug log ran once per regex match and included the whole line, so log volume grew with the square of line length. With 1MB lines that is hundreds of GB of debug output from a single crafted file.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(gentoo): don't dereference a nil location when a LICENSE scan fails
`extractLicenses` accepts a nil location, but the scan error log added earlier on this branch dereferenced it.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(gentoo): bound license group expansion on hostile license_groups
- dedup within a group uses a set instead of `slices.Contains`, which was quadratic in tokens per line and got much worse under the 1MB line cap
- each group is expanded once and memoized, so a chain of groups that reference each other repeatedly no longer expands exponentially
- repeated references to the same group within one line are skipped
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* Prevent duplicate Go packages after source replacement resolution
Source analysis already applies module replacements through go/packages. Avoid synthesizing the same replacement again from go.mod while retaining fallback synthesis for modules that source analysis did not resolve.
Constraint: Preserve unimported and local-path replacement cataloging.
Rejected: Deduplicate only during final assembly | That retains redundant license lookup and ambiguous metadata ownership.
Confidence: high
Scope-risk: narrow
Directive: Keep go.mod fallback packages limited to modules absent from source analysis.
Tested: Focused replacement regression, related Go module parser tests, go vet, gofmt, and diff checks.
Not-tested: Docker-backed full cataloger fixtures; local root storage was exhausted by image generation.
Signed-off-by: ychampion <ychampion@users.noreply.github.com>
* Keep replacement fixtures with the Go module test data
Constraint: The maintainer reserves internal/gotestdata for fixtures that need special Go tooling discovery.
Rejected: Leave this fixture in gotestdata | The regression opens its module explicitly and does not need the special location.
Confidence: high
Scope-risk: narrow
Directive: Use internal/gotestdata only when a fixture must avoid Go testdata discovery rules.
Tested: replacement regression repeated 10 times; Go module parser table; go vet for the Go cataloger; gofmt; diff checks.
Not-tested: Full cataloger package; three unrelated parser fixtures fail identically on exact prior head in this environment.
Signed-off-by: ychampion <ychampion@users.noreply.github.com>
---------
Signed-off-by: ychampion <ychampion@users.noreply.github.com>
Co-authored-by: ychampion <ychampion@users.noreply.github.com>
* chore: centralize temp files and prefer streaming IO
Catalogers that create temp files ad-hoc can easily forget cleanup,
leaking files on disk. Similarly, io.ReadAll is convenient but risks
OOM on large or malicious inputs.
Introduce internal/tmpdir to manage all cataloger temp storage under
a single root directory with automatic cleanup. Prefer streaming
parsers (bufio.Scanner, json/yaml.NewDecoder, io.LimitReader) over
buffering entire inputs into memory. Add ruleguard rules to enforce
both practices going forward.
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* chore: go back to old release parsing
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* simplify to limit reader in version check
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* chore: regex change postponed
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* simplify supplement release to limitreader
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
---------
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
* migrate fixtures to testdata
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: correct broken symlinks after testdata migration
The migration from test-fixtures to testdata broke several symlinks:
- elf-test-fixtures symlinks pointed to old test-fixtures paths
- elf-test-fixtures needed to be renamed to elf-testdata
- image-pkg-coverage symlink pointed to test-fixtures instead of testdata
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: handle missing classifiers/bin directory in Makefile
The clean-fingerprint target was failing when classifiers/bin doesn't
exist (e.g., on fresh clone without downloaded binaries).
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: add gitignore negation for jar/zip fixtures in test/cli
The jar and zip files in test/cli/testdata/image-unknowns were being
gitignored by the root .gitignore patterns. This caused them to be
untracked and not included when building docker images in CI, resulting
in Test_Unknowns failures since the test expects errors from corrupt
archive files that weren't present.
Add a .gitignore in test/cli/testdata to negate the exclusions for
these specific test fixture files.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* switch fixture cache to v2
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: update expected versions for rebuilt fixtures
Update test expectations for packages that have been updated in
upstream repositories when docker images are rebuilt:
- glibc: 2.42-r4 → 2.43-r1 (wolfi)
- php: 8.2.29 → 8.2.30 (ubuntu/apache)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* upgrade go
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: add go-shlex dependency for testdata manager tool
The manager tool in syft/pkg/cataloger/binary/testdata/ imports
go-shlex, but since it's in a testdata directory, Go doesn't track
its dependencies. This caused CI failures when go.mod didn't
explicitly list the dependency.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* refactor: move binary classifier manager to internal/
Move the manager tool from testdata/manager to internal/manager so
that Go properly tracks its dependencies. Code in testdata directories
is ignored by Go for dependency tracking, which caused CI failures
when go.mod didn't explicitly list transitive dependencies.
This is a cleaner solution than manually adding dependencies to go.mod
for code that happens to live in testdata.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: add gitignore negations for test fixtures blocked by root patterns
Multiple test fixtures were being blocked by root-level gitignore patterns
like bin/, *.jar, *.tar, and *.exe. This adds targeted .gitignore files with
negation patterns to allow these specific test fixtures to be tracked:
- syft/linux/testdata/os/busybox/bin/busybox (blocked by bin/)
- syft/pkg/cataloger/java/testdata/corrupt/example.{jar,tar} (blocked by *.jar, *.tar)
- syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/**/bin/go (blocked by bin/)
- syft/pkg/cataloger/bitnami/testdata/no-rel/.../bin/redis-server (blocked by bin/)
Also updates the bitnami test expectation to include the newly required
.gitignore files in the test fixture.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: update glibc version expectation (2.43-r1 -> 2.43-r2)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* add capability drift check as unit step
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* dont clear test observations before drift detection
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* bump stereoscope commit to main
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fixed#4316 go mod with ver purl
Signed-off-by: Rez Moss <hi@rezmoss.com>
* go mod purl fixed, added func to handle go.mod
Signed-off-by: Rez Moss <hi@rezmoss.com>
* fix: use module name in PURL string everywhere
Signed-off-by: Keith Zantow <kzantow@gmail.com>
---------
Signed-off-by: Rez Moss <hi@rezmoss.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Co-authored-by: Keith Zantow <kzantow@gmail.com>
When the goModCataloger processes a Replace directive it currently adds the new
package to the resulting package list, but does not remove the old one unless
the path is unchanged.
Based on an existing comment in the code, removing the old one seems to be the
intended behavior, and results in a more expected end-result, so this does so.
Signed-off-by: Russell Haering <russellhaering@gmail.com>
* chore: ruleguard to enforce defer use
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* fix go.mod location
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* chore: defer close in linux release identifier
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* chore: better lint suggestion
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* chore: refactor binary classifier to defer close
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* chore: defer close readers in gentoo cataloger
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* chore: make go license parsing defer close readers
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* chore: defer closing readers in alpine apm parser
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* chore: defer close readers in graalvm parser
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* chore: defer close readers in debian package parser
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* chore: defer close readers in alpm parser
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* chore: defer close readers in executable file cataloger
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* chore: defer close readers in javascript license parser
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* chore: defer close readers in go mod parser
Signed-off-by: Will Murphy <will.murphy@anchore.com>
---------
Signed-off-by: Will Murphy <will.murphy@anchore.com>
this PR makes the following changes to update the underlying license model to have more expressive capabilities
it also provides some guarantee's surrounding the license values themselves
- Licenses are updated from string -> pkg.LicenseSet which contain pkg.License with the following fields:
- original `Value` read by syft
- If it's possible to construct licenses will always have a valid SPDX expression for downstream consumption
- the above is run against a generated list of SPDX license ID to try and find the correct ID
- SPDX concluded vs declared is added to the new struct
- URL source for license is added to the new struct
- Location source is added to the new struct to show where the expression was pulled from
* set package ID in catalogers and improve hashing performance
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* update setting ID + tests
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* migrate pkg.ID and pkg.Relationship to artifact package
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* return relationships from tasks
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix more tests
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add artifact.Identifiable by Identity() method
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix linting
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* remove catalog ID assignment
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* adjust spdx helpers to use copy of packages
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* stabilize package ID relative to encode-decode format cycles
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* rename Identity() to ID()
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* use zero value for nils in ID generation
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* enable source.Location to be identifiable
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* hoist up package relationship discovery to analysis stage
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* update ownership-by-file-overlap relationship description
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add test reminders to put new relationships under test
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* adjust PHP composer.lock parser function to return relationships
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Alex Goodman
Yashas Gunderiaandychampion
Will Murphy
Rez MossandKeith Zantow
Christopher Angelo PhillipsandAlex Goodman
VictorHuuandAlex Goodman
anchore-actions-token-generator[bot]andspiffcs
Russell Haering
William Murphy
Avi DeitcherandKeith Zantow
Alex Goodman