Compare commits

...
Author SHA1 Message Date
Alex Goodman ab69742d25 limit source-distribution refs to deb and rpm
- apk origins and alpm pkgbases are build recipes, not published source packages, and their names are usually also real binary packages, so a `source-distribution` ref there would point at a binary package. They keep relying on the `upstream` PURL qualifier.
- document why this is an external reference rather than `pedigree.ancestors`, with links to the CycloneDX spec discussion (CycloneDX/specification#612)

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
2026-09-30 12:56:05 -04:00
Alex Goodman bf8ded3cf8 build source PURL from metadata and gate it to CycloneDX 1.6+
- the source PURL is built from package metadata instead of editing the binary PURL: only `distro` (plus `epoch` and `rpmmod` for rpm) is carried over, subpath is dropped, and the ref is skipped when the PURL type does not match the metadata ecosystem
- one emission rule for all ecosystems: only when the source name or version differs from the binary, so self-sourced rpm and alpm packages no longer get a redundant ref
- source RPM names must end in `.src.rpm` or `.nosrc.rpm`, and `nosrc` maps to `arch=nosrc`
- `source-distribution` refs are dropped when encoding below spec 1.6, since cyclonedx-go does not downgrade them (CycloneDX/cyclonedx-go#291)

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
2026-09-30 12:56:05 -04:00
swadaptive d53a8d283f Adjust arch qualifier for source PURLs
Signed-off-by: swadaptive <268603322+swadaptive@users.noreply.github.com>
2026-09-30 12:56:05 -04:00
swadaptive 13c6591cb2 Golden test fixtures
Signed-off-by: swadaptive <268603322+swadaptive@users.noreply.github.com>
2026-09-30 12:56:05 -04:00
swadaptive 4ffde9f891 Add source-distribution external reference to components
Signed-off-by: swadaptive <268603322+swadaptive@users.noreply.github.com>
2026-09-30 12:56:05 -04:00
swadaptive 06451adc43 Source distribution external reference utilities and test coverage
Signed-off-by: swadaptive <268603322+swadaptive@users.noreply.github.com>
2026-09-30 12:56:05 -04:00
swadaptive ecf6b9eff5 Add tests for source packages in external references
Signed-off-by: swadaptive <268603322+swadaptive@users.noreply.github.com>
2026-09-30 12:56:05 -04:00
Mayank Sekhar d61c2cada3 fix(binary): correct traefik version detection on big-endian arches (#5281)
The traefik-binary classifier regex required a NUL byte or replacement
character immediately before the version string. On some architectures
(e.g. s390x), the real version string in the binary has no such prefix,
so the classifier instead matched an earlier, unrelated NUL-prefixed
numeric string elsewhere in the binary (e.g. '2.7.0' from an unrelated
printf-style format constant), producing an incorrect version.

This adds an additional pattern, tried first via MatchAny, that matches
version strings terminated by two NUL bytes without requiring a NUL
prefix. The original pattern remains as a fallback for older traefik
binaries (e.g. v1.x) that use single-NUL termination.

Fixes #4980

Signed-off-by: Mayank Sekhar <mayankshekharsingh@gmail.com>
2026-09-30 16:51:03 +00:00
0bbcff1ae3 Fix hang when scanning a Java resource adapter (.rar) file (#5348)
* fix: scan a Java resource adapter (.rar) given as a file source

The file source picks an extractor for the scan target from its file name
alone. A Java resource adapter archive (JCA) uses the .rar extension but is
a zip file, so it was handed to the RAR extractor. With rardecode v2.2.0 the
RAR extractor does not return on non-RAR input (an infinite loop in its
signature search), so `syft scan --from file adapter.rar` never finished.
The same bytes named .jar scan in under a second.

When the name says RAR but the content is a zip archive, leave the file
unextracted, the same way .jar, .war and .ear scan targets are handled. The
java cataloger already reads .rar files (anchore/syft#4136) and now finds the
resource adapter and its nested jars. Real RAR archives are still extracted.

Signed-off-by: Ismael Donmez <ismael.donmez@chainguard.dev>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* chore(deps): bump github.com/nwaples/rardecode/v2 from 2.2.0 to 2.4.1

rardecode v2.2.0 loops forever in its RAR signature search when the input
does not start with a RAR signature (fixed upstream in nwaples/rardecode
f501ac8, "fix infinite loop in findSig", released in v2.2.2). syft reaches
that search for any file source whose name contains ".rar", so a scan of a
self-extracting RAR, or of any other file with such a name, never finished.
With v2.4.1 a self-extracting RAR is extracted, and a file that is not a RAR
archive fails fast with "rardecode: RAR signature not found", the same way a
file named .zip that is not a zip archive fails today.

github.com/mholt/archives v0.1.5 (the latest release) still requires v2.2.0,
so the newer version is selected here.

Signed-off-by: Ismael Donmez <ismael.donmez@chainguard.dev>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: clarify the .rar zip log line and cover self-extracting RAR sources

- the debug line no longer calls every `.rar`-named zip a resource adapter
- a self-extracting RAR (stub before the signature) given as a file source is
  still extracted

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Ismael Donmez <ismael.donmez@chainguard.dev>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
2026-09-30 15:54:36 +00:00
anchore-oss-update-botandanchore-oss-update-bot f21e439a63 chore(deps): update SPDX license list (#5315)
Signed-off-by: anchore-oss-update-bot <anchore-oss-update-bot@users.noreply.github.com>
Co-authored-by: anchore-oss-update-bot <anchore-oss-update-bot@users.noreply.github.com>
2026-09-30 15:50:47 +00:00
Will Murphy c511d509ef fix: match opensource.org license URLs across SPDX URL forms (#5361)
SPDX license list 3.29.0 rewrote every opensource.org seeAlso URL from
opensource.org/licenses/<ID> to opensource.org/license/<ID> (and e.g.
opensource.org/license/mit/ to opensource.org/license/MIT). Package metadata
in the wild (Maven POM <license><url>, npm, etc.) still uses the older forms,
so regenerating the list would stop LicenseByURL from resolving them.

Fall back to a normalized index of the opensource.org entries that ignores the
www. prefix, /licenses/ vs /license/, case, and a trailing slash. The index is
built from the generated urlToLicense map, so it works with both the current
and the updated license list without changes to the generator. URL forms that
would normalize to different licenses are left out rather than guessed.

Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
2026-09-30 15:07:12 +00:00
Alex Goodman fdeb78fc19 Speed up unit tests in CI (#5360)
* ci: drop gen-5 instance families from the test runner

m5a runners run the unit suite ~2.3x slower than m8id (15m vs 6.5m for `go test`), and when spot capacity for newer families dries up runs-on falls back to them. Keeping gen-6 through gen-8 leaves a wide spot pool without the slow tail.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test: shrink decompression bomb fixtures

The bomb and heap-bound tests pushed 100MB-16GB through gzip and parsers, which under `-race` made them the slowest tests in the unit suite. Each now uses the smallest fixture that still separates the bounded path from the unbounded one.

- arch mtree caps are measured at a small cap and scaled to the shipped constants, so raising a cap still fails the budget
- snap and debian bombs expand to 32MB instead of 512MB (4x their 8MB budgets)
- golang ELF bombs still declare 256MB but deliver 64MB

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
2026-09-30 10:58:11 -04:00
Alex Goodman 6ac7afb439 Fix missing file hashes when running on windows system (#5341)
* test: add windows unit test job and capture dir resolver path bug

- add a `Unit tests (Windows)` job running a new `make unit:windows` task, scoped to the packages that own host path handling (tests needing linux docker fixtures are skipped)
- `Unit tests` is now a gate job over the Linux and Windows unit jobs
- add a resolver test asserting that locations from `AllLocations` resolve again via `FilesByPath` and use posix paths. This is expected to fail on windows today, which is why file digests come back empty for `dir:` scans (#5325)
- fix windows-only test files that no longer compiled

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* chore: drop verbose output from integration tests

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: windows path handling outside of the file lookup bug

- chroot paths are always posix (`ToChrootPath` no longer returns backslash paths on windows), same for the directory source name derived from `base`
- `windows.FromPosix` no longer panics on input that isn't volume-encoded (e.g. relative or already native paths)
- skip tests on windows that rely on symlink fixtures, shell script fixtures, or emulate windows paths on a posix host
- disable `core.autocrlf` on the windows CI job so fixture digests match linux

This leaves only the `FilesByPath` lookup failures from #5325.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test: skip posix-only fileresolver tests on windows

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: split contained paths on posix separators on windows

- `allContainedPaths` cleaned with the native separator but split on `/`, so a root reached through a symlink never had its ancestors indexed on windows
- index assertions in tests now key on the posix path the index actually stores
- skip symlink fixture tests and a posix-only test helper on windows

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* chore: revert unrelated go.sum churn and allContainedPaths change

- restore `go.sum` and `.make/go.sum` (accidentally populated by a local `-mod=mod` invocation)
- revert the `allContainedPaths` separator change: production passes native windows paths there, and splitting them on `/` produces drive-relative ancestors, so skip the posix-only test on windows instead
- annotate the remaining `filepath` usages that operate on native paths

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: search the file tree with posix paths on windows (#5325)

The directory and file resolvers key their file tree on posix paths (`/c/some/path`), but `FilesByPath`, `HasPath` and root-anchored `FilesByGlob` searched it with the native path from the chroot context (`C:\some\path`). On windows every lookup silently missed, so anything that resolves files by path found nothing. This is why file digests were empty for `dir:` scans.

Requests are now converted to posix before searching the tree.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
2026-09-30 09:08:13 -04:00
Darragh McGurk b23e461731 Stop reading the mount table for every single-file source (#5257)
Signed-off-by: Darragh McGurk <120114728+dmcg310@users.noreply.github.com>
2026-09-29 21:44:47 +00:00
KR RavindraandWill Murphy 8c63a2403c fix: conan expat CPE (#5259)
Add libexpat_project vendor and libexpat product candidates for Conan
packages named expat, so generated CPEs match the NVD entries used for
libexpat vulnerabilities.

Signed-off-by: KR Ravindra <42912207+KR-Ravindra@users.noreply.github.com>
Co-authored-by: Will Murphy <willmurphyscode@users.noreply.github.com>
2026-09-29 21:43:53 +00:00
Antoni (Tony) JagodkaandAlex Goodman e4ff7f5973 fix(spdx): write a name for the root package when the source has none (#5349)
* fix(spdx): write a name for the root package when the source has none

A source read from another SBOM (a purl list, a CycloneDX document
without metadata.component, an SPDX document without a DESCRIBES
relationship) can have no name. The SPDX encoder then wrote the document
root package with an empty PackageName, which SPDX requires. In
tag-value output the PackageName line is omitted, so the root package's
fields land in the previous section and syft cannot read its own output
back ("received unknown tag SPDXID in CreationInfo section").

Fall back to the document name for the root package, and make the
document name fall back to "unknown" for every source type rather than
only for unrecognized ones.

Signed-off-by: Tony <tjagodka@gmail.com>

* fix(spdx): drop syft's root package on decode for every SPDX version

- the root package name now comes from one fallback chain (source name, image repo name, source ID, document name), so the root purl no longer gets an empty name for a source decoded from CycloneDX
- decoding strips the root syft writes for an unknown source, so `syft convert` round trips stop adding an `unknown` package each pass
- for SPDX 2.1 and 2.2, which have no `primaryPackagePurpose`, the purpose is taken from the source type in syft's `DocumentRoot-<type>-` ID

**behavior change:** decoding a syft-written SPDX 2.1/2.2 document no longer returns the root as a package or its `CONTAINS` relationships, matching 2.3.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Tony <tjagodka@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
2026-09-29 21:30:52 +00:00
Dev MandAlex Goodman 37d45de9ff fix(python): link uv.lock dependents to the locked version they name (#5351)
* fix(python): link uv.lock dependents to the locked version they name

A forked uv.lock can lock several versions of one package, and uv records the version on each dependency entry in that case. The parser dropped that field, so every pandas version depended on every numpy version. Carry the locked version through provides and requires as name@version (bare name stays for unique locks) and keep marker text unchanged.

Fixes #5340

Signed-off-by: Dev M <devtechedge@gmail.com>

* fix(python): pair uv.lock versions without widening public metadata

- the locked version on a uv dependency entry is only needed to pair relationships, and the relationships already carry it, so it stays on the internal toml struct instead of `pkg.PythonUvLockDependencyEntry`. The previous approach hid it from JSON but still hashed it into package IDs.
- dependency specs are now built from the raw lock entries at parse time rather than read back off package metadata
- optional-dependencies get the same version pairing, so an extra that names one locked version no longer links to all of them

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Dev M <devtechedge@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
2026-09-29 17:23:22 -04:00
Pujitha PaladuguandPujitha Paladugu 26927eab02 fix: suppress misleading upstream CPE for PHP extension binaries (#5102)
Motivation:
syft's PHP interpreter cataloger reports each bundled PHP C extension
(openssl, ldap, sqlite3, etc.) as its own package, using the PHP
interpreter's own version number as the package version (since these
extensions are compiled into a specific PHP release and have no
independent version of their own). The classifier already attaches an
explicit CPE like cpe:2.3:a:php-openssl:php-openssl:8.5.7:... to make
clear this refers to the bundled PHP extension, not upstream OpenSSL.

However, that CPE was marked with Source: cpe.GeneratedSource. Syft's
package-finalization pipeline (internal/task/package_task_factory.go)
only skips its generic, name-based CPE generation when a package
already has an "authoritative" CPE (any Source other than
GeneratedSource) - so it also generated and appended a second CPE,
cpe:2.3:a:openssl:openssl:8.5.7:..., derived from the bare package
name. That second CPE is indistinguishable from a real standalone
OpenSSL/LDAP/SQLite installation and can cause vulnerability scanners
that consume this SBOM to falsely flag PHP hosts as running vulnerable
upstream libraries at version "8.5.7" (a PHP version, not a real
OpenSSL/LDAP/SQLite release). See anchore/syft#5014 for a full repro.

Approach:
Mark the classifier's explicit CPE with Source: cpe.DeclaredSource
instead of cpe.GeneratedSource, so it is treated as authoritative and
the extra name-derived CPE is no longer generated. This mirrors the
existing pattern in
syft/pkg/cataloger/java/parse_jvm_release.go's newJvmCpe, which uses
the same DeclaredSource + comment convention to mark a
syft-synthesized (not literally file-declared) CPE as trustworthy and
suppress further generation. It also brings this classifier in line
with its sibling php-cli/php-fpm/php-apache classifiers in the same
file, which already mark their CPEs as authoritative (via
NVDDictionaryLookupSource, not applicable here since php-<ext> is not
a real NVD dictionary entry).

This change only suppresses the extra generated CPE; it does not
change the package name, type, version, or PURL, and the existing
php-<ext>:php-<ext> CPE is unaffected.

Validation:
  go build ./...
  go test ./syft/pkg/cataloger/php/... ./internal/task/... ./syft/pkg/cataloger/internal/cpegenerate/...
All pass except the two pre-existing docker-image-fixture subtests of
Test_InterpreterCataloger, which fail in this sandbox only because the
`docker` binary isn't available (exec: "docker": executable file not
found in $PATH); confirmed via `git stash` that these two subtests
fail identically on main without this change, i.e. this is a sandbox
limitation and not a regression. Added a new unit test,
Test_getClassifier_declaredCPESource, that directly asserts the
classifier's CPE now carries Source: cpe.DeclaredSource.

Fixes #5014

Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Co-authored-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
2026-09-29 17:15:36 -04:00
Alex Goodman 1dc16fbac4 fix: report scanner errors and raise the line cap in metadata parsers (#5353)
* fix: report scanner errors instead of returning a partial parse as complete

A line longer than `bufio.Scanner`'s 64KB buffer ends the scan with `ErrTooLong`, which only `Err()` reports. Parsers now return what they found along with the error (recorded as an unknown for the file), or log it where there is no error to return.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: allow lines up to 1MB in line-oriented package metadata parsers

Adds `internal.NewLineScanner`, a `bufio.Scanner` capped at 1MB per line rather than the 64KB default, the same cap the alpm parser already uses. Long single-line fields (descriptions, license text, dependency lists) now parse instead of ending the scan early.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: use io.ReadFull where a short read was silently accepted

- dotnet bundle 7-bit int decoding could read a zero-byte result as a zero byte
- wordpress plugin header parsing included the unread tail of its buffer

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: finish moving line scanners to the 1MB cap and checking scanner errors

- java manifest, pom.properties, Gemfile.lock, requirements.txt, homebrew formula, alpm and the purl/cpe decoders now use `internal.NewLineScanner`
- snap kernel changelog reports a scan error instead of silently dropping the base kernel package
- the attestation output reader keeps draining the pipe after a scan error so cosign can't block on a full pipe

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: stop logging the full setup.py line once per dependency match

The debug log ran once per regex match and included the whole line, so log volume grew with the square of line length. With 1MB lines that is hundreds of GB of debug output from a single crafted file.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix(gentoo): don't dereference a nil location when a LICENSE scan fails

`extractLicenses` accepts a nil location, but the scan error log added earlier on this branch dereferenced it.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix(gentoo): bound license group expansion on hostile license_groups

- dedup within a group uses a set instead of `slices.Contains`, which was quadratic in tokens per line and got much worse under the 1MB line cap
- each group is expanded once and memoized, so a chain of groups that reference each other repeatedly no longer expands exponentially
- repeated references to the same group within one line are skipped

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
2026-09-29 16:52:58 -04:00
Alex Goodman d0e1e7be05 Harden cataloger parsers against truncated input (do not panic) (#5355)
* fix(javascript): don't panic on a yarn v1 lockfile line missing its value

A v1 `yarn.lock` dependency line without a version constraint, or a
`version`/`resolved`/`integrity` line without a value, indexed past the end of
the split line. Such dependencies are now skipped and empty fields left empty.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix(alpine): don't panic on an apk db ACL or checksum line before its entry

An `M:`, `a:` or `Z:` line that comes before any directory or file entry in
`lib/apk/db/installed` indexed `files[-1]`. Such a line is now skipped.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix(erlang): don't panic on a truncated rebar.lock or .app file

The term parser read past the end of the input when a file ended mid-term:
whitespace only, an unclosed list, a lone `<`, or a comment ending in `\r`.
These now return an "unexpected end of input" error.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix(lua): don't panic on a truncated .rockspec

The rockspec parser read past the end of the input when a file ended mid-statement:
a lone `[`, an unclosed `[[...]]` value or `[...]` index, or a `local` assignment cut off
before its `=`, value, or line end. These now return an "unexpected end of input" error.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix(swift): don't panic on a Podfile.lock pod without a version

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix(gentoo): don't panic on a short obj line in portage CONTENTS

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix(spdx): don't panic on an opam checksum without an algorithm

The opam cataloger stores checksums verbatim, so a malformed one reached the SPDX encoder and panicked there, outside the cataloger's recover. Checksums without an `=` are now skipped.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* refactor: use strings.Cut where a split was only indexed after a Contains check

Also guards `windows.FromPosix` against a path with no separator.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test: guard untyped package metadata fields

- new `any`-typed metadata fields must be allowlisted with a reason
- allowlisted untyped fields are pushed through `SetID` and every encoder with
  non-finite, deeply nested and large values

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test: fuzz every generic cataloger parser

One fuzz target over every glob-selected parser in the capabilities data, run
through its cataloger task. Each input must not panic, must finish in time and
must keep allocations proportional to its size. Seeds come from each parser's
testdata plus the string literals in its package.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test: scan malformed inputs under memory and time limits

A directory of small malformed files (deep nesting, oversized declared counts,
truncation, type mismatches, non-finite floats) is scanned by the CLI with a
memory limit and an RSS ceiling. The scan must exit 0 and write an SBOM.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test: stop a fuzz input once its live heap passes the allocation limit

The allocation check only ran after the parser returned, so an input on its way to exhausting memory ran the machine into swap first. The harness now polls the live heap while the parser runs and panics past the same limit, which the fuzz engine saves as a crasher.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: bound nesting depth and error context in the erlang and lua parsers

- a deeply nested rebar.lock or .rockspec overflowed the goroutine stack, which is fatal rather than a recoverable panic, so one file could take down the whole scan. Both parsers now stop at 1000 levels with an error.
- parse errors echoed whole input lines into the SBOM as unknowns, and lua appended another copy at every nesting level. `PrintError` now clips to a window around the error column, and lua decorates once at the top.
- the Podfile.lock error no longer echoes the pod entry.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test: close ways the malformed-input checks could pass quietly

- a ctx-aware parser hitting the fuzz timeout now fails the input instead of racing the hard deadline
- the heap watchdog measures growth from a post-GC baseline rather than an absolute process-wide number
- a saved fuzz input naming a target that no longer exists fails instead of being skipped
- the CLI malformed-input scan fails on a recovered parser panic, not only on a non-zero exit

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
2026-09-29 16:26:42 -04:00
Alex Goodman 57bd6a5f9c fix(cpe): bound CPE candidate generation for pathological package metadata (#5356)
* fix(cpe): cap sub-selection prefixes to bound candidate cross-product

a name with thousands of `-`/`_` separators produced one vendor and product candidate per segment, and the vendor x product cross-product made CPE generation roughly cubic in time and memory (8KB name: ~250s, 4.8GB RSS). `generateSubSelections` now stops after 10 prefixes. The full name is still a candidate, so real package names are unaffected.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix(cpe): bound candidate count and field length before the cross-product

the prefix cap only covered one way to inflate the vendor x product cross-product in `FromPackageAttributes`. Java groupIDs add a vendor (and, with no artifactID, a product) per dot segment, and .NET executables add one of each per file, so a 1000-segment `Bundle-SymbolicName` still produced ~1M CPEs and several GB.

- vendor and product candidates are each capped at 64 (shortest first, so truncation is deterministic)
- names, versions, and candidates over 512 bytes are skipped, since every byte is paid once per CPE in key building and regex validation

real packages produce around a dozen candidates with names well under 100 bytes, so output for real inputs is unchanged. Also corrects the earlier comment: prefixes only become vendor candidates and the cost was quadratic, not cubic.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
2026-09-29 16:01:21 -04:00
Will Murphy c60aabad37 test(golang): make the UPX fixture exercise loader padding (#5354)
UPX pads to a 4 byte boundary before it writes the loader stub. The
image-small-upx build happened to need no padding, so the cataloger
tests over it passed with or without the fix in #5347.

- build the fixture with -X main.Version=1.0.11, which needs 2 bytes of
  padding, so the existing cataloger tests fail without the fix
- add TestImageSmallUPXNeedsLoaderPadding, which fails if a change to the
  fixture's inputs means it no longer needs padding
- run the crafted loader test on both sides of the boundary, so a skip
  that always rounds up is caught too

Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
2026-09-29 19:00:42 +00:00
126 changed files with 4066 additions and 788 deletions
+7
View File
@@ -0,0 +1,7 @@
# check out text files with lf on every platform (regardless of core.autocrlf), letting git detect binary files
* text=auto eol=lf
# fixtures that must keep their crlf line endings byte for byte
syft/linux/testdata/unprintable -text
syft/pkg/cataloger/bitnami/testdata/no-rel/opt/bitnami/redis/.spdx-redis.spdx -text
syft/pkg/cataloger/java/testdata/jar-metadata/**/META-INF/MANIFEST.MF -text
+29 -1
View File
@@ -27,7 +27,7 @@ jobs:
# enough; a 4 vCPU instance can't pull much more than that anyway.
# note: s3-cache intentionally omitted -- PR runs are untrusted and must not write to the
# shared cache backend that the trusted release workflow reads from (cache poisoning).
runs-on: &test-runner "runs-on=${{ github.run_id }}/cpu=4+8/ram=16+32/family=m5+m6+m7+m8+c5+c6+c7+c8/spot=price-capacity-optimized/volume=100gb"
runs-on: &test-runner "runs-on=${{ github.run_id }}/cpu=4+8/ram=16+32/family=m6+m7+m8+c6+c7+c8/spot=price-capacity-optimized/volume=100gb"
permissions:
contents: read
steps:
@@ -49,6 +49,13 @@ jobs:
Unit-Test:
# Note: changing this job name requires making the same update in the .github/workflows/release.yaml pipeline
name: "Unit tests"
runs-on: ubuntu-latest
needs: [Unit-Test-Linux, Unit-Test-Windows]
steps:
- run: echo All unit tests passed!
Unit-Test-Linux:
name: "Unit tests (Linux)"
runs-on: *test-runner
permissions:
contents: read
@@ -68,6 +75,27 @@ jobs:
- name: Check for capability drift
run: make check-capability-drift
Unit-Test-Windows:
name: "Unit tests (Windows)"
runs-on: windows-latest
permissions:
contents: read
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #v7.0.1
with:
persist-credentials: false
# go-make's setup only (not ./.github/actions/bootstrap): no apt, binny tools, or fixture cache are needed
- name: Setup go + go-make tooling
uses: anchore/go-make/.github/actions/setup@fa5421b3bf24c9d18f11a736c5eb4172f2b7d683 # v0.8.1
with:
# keep in sync with the go-version default in ./.github/actions/bootstrap
go-version: "1.26.2"
# the full unit suite leans on linux docker fixtures, so this runs the windows-relevant subset
- name: Run unit tests
run: make unit:windows
Integration-Test:
# Note: changing this job name requires making the same update in the .github/workflows/release.yaml pipeline
name: "Integration tests"
+19 -1
View File
@@ -1,6 +1,7 @@
package main
import (
"os"
"path/filepath"
"runtime"
"strconv"
@@ -52,7 +53,7 @@ func main() {
raceFlag = " -race"
}
Run(
"go test -count=1 -timeout=30m -v"+raceFlag+" ./cmd/syft/internal/test/integration/...",
"go test -count=1 -timeout=30m"+raceFlag+" ./cmd/syft/internal/test/integration/...",
run.Env("GODEBUG", "dontfreezetheworld=1"),
)
},
@@ -96,6 +97,23 @@ func main() {
},
},
// windows unit tests: deliberately not hooked into "test" (no RunsOn). The full unit suite leans on
// linux docker fixture images and posix-only assumptions, so this covers the packages that own
// host path handling (where windows bugs live) and skips the tests that need docker image fixtures.
// ponytail: curated package list, widen it as more of the suite is made windows-safe.
Task{
Name: "unit:windows",
Description: "run the windows-relevant subset of unit tests",
Run: func() {
Run("go test -count=1"+
` -skip "Image|Squash|AllLayers|MixFileTypes|GivenCoordinates"`+
" ./syft/internal/windows/... ./syft/internal/fileresolver/... ./syft/source/directorysource/... ./syft/source/filesource/..."+
" ./syft/file ./syft/file/cataloger/filedigest/... ./syft/file/cataloger/filecontent/...",
run.Stdout(os.Stderr),
)
},
},
// default validation pipeline (replaces Taskfile `default`/`pr-validations`/`validations`).
Task{
Name: "default",
+5
View File
@@ -158,6 +158,11 @@ func (l *backgroundLineReader) read(reader io.Reader, stage *progress.Stage) {
l.lock.Unlock()
}
if err := s.Err(); err != nil {
log.WithFields("error", err).Debug("unable to read attestation output")
// keep draining so the writer never blocks on a full pipe
_, _ = io.Copy(io.Discard, reader)
}
}
func (l backgroundLineReader) Lines() []string {
+1 -1
View File
@@ -248,7 +248,7 @@ require (
github.com/muesli/cancelreader v0.2.2 // indirect
github.com/muesli/termenv v0.16.0 // indirect
github.com/ncruces/go-strftime v1.0.0 // indirect
github.com/nwaples/rardecode/v2 v2.2.0 // indirect
github.com/nwaples/rardecode/v2 v2.4.1 // indirect
github.com/olekukonko/cat v0.0.0-20250911104152-50322a0618f6 // indirect
github.com/olekukonko/errors v1.2.0 // indirect
github.com/olekukonko/ll v0.1.6 // indirect
+2 -2
View File
@@ -758,8 +758,8 @@ github.com/nix-community/go-nix v0.0.0-20250101154619-4bdde671e0a1 h1:kpt9ZfKcm+
github.com/nix-community/go-nix v0.0.0-20250101154619-4bdde671e0a1/go.mod h1:qgCw4bBKZX8qMgGeEZzGFVT3notl42dBjNqO2jut0M0=
github.com/nsf/jsondiff v0.0.0-20210926074059-1e845ec5d249 h1:NHrXEjTNQY7P0Zfx1aMrNhpgxHmow66XQtm0aQLY0AE=
github.com/nsf/jsondiff v0.0.0-20210926074059-1e845ec5d249/go.mod h1:mpRZBD8SJ55OIICQ3iWH0Yz3cjzA61JdqMLoWXeB2+8=
github.com/nwaples/rardecode/v2 v2.2.0 h1:4ufPGHiNe1rYJxYfehALLjup4Ls3ck42CWwjKiOqu0A=
github.com/nwaples/rardecode/v2 v2.2.0/go.mod h1:7uz379lSxPe6j9nvzxUZ+n7mnJNgjsRNb6IbvGVHRmw=
github.com/nwaples/rardecode/v2 v2.4.1 h1:F7zNW2LdAuuBThHWXQaiFUGVD/sef299NfWSB1nHAl4=
github.com/nwaples/rardecode/v2 v2.4.1/go.mod h1:7uz379lSxPe6j9nvzxUZ+n7mnJNgjsRNb6IbvGVHRmw=
github.com/olekukonko/cat v0.0.0-20250911104152-50322a0618f6 h1:zrbMGy9YXpIeTnGj4EljqMiZsIcE09mmF8XsD5AYOJc=
github.com/olekukonko/cat v0.0.0-20250911104152-50322a0618f6/go.mod h1:rEKTHC9roVVicUIfZK7DYrdIoM0EOr8mK1Hj5s3JjH0=
github.com/olekukonko/errors v1.2.0 h1:10Zcn4GeV59t/EGqJc8fUjtFT/FuUh5bTMzZ1XwmCRo=
+19
View File
@@ -0,0 +1,19 @@
package internal
import (
"bufio"
"io"
)
// maxScannedLineSize is the longest line NewLineScanner will return before Scan stops with bufio.ErrTooLong.
// The buffer only grows to this size when a line needs it.
const maxScannedLineSize = 1024 * 1024
// NewLineScanner returns a bufio.Scanner that accepts lines up to 1MB rather than the 64KB default, since
// package metadata files carry long single-line fields (descriptions, license text, dependency lists). Callers
// must still check Err() after scanning.
func NewLineScanner(r io.Reader) *bufio.Scanner {
s := bufio.NewScanner(r)
s.Buffer(nil, maxScannedLineSize)
return s
}
+24
View File
@@ -0,0 +1,24 @@
package internal
import (
"bufio"
"strings"
"testing"
"github.com/stretchr/testify/require"
)
func TestNewLineScanner(t *testing.T) {
long := strings.Repeat("a", 100*1024)
s := NewLineScanner(strings.NewReader(long + "\nb\n"))
require.True(t, s.Scan())
require.Equal(t, long, s.Text())
require.True(t, s.Scan())
require.Equal(t, "b", s.Text())
require.False(t, s.Scan())
require.NoError(t, s.Err())
s = NewLineScanner(strings.NewReader(strings.Repeat("a", maxScannedLineSize+1)))
require.False(t, s.Scan())
require.ErrorIs(t, s.Err(), bufio.ErrTooLong)
}
@@ -0,0 +1,74 @@
package packagemetadata
import (
"reflect"
"strings"
"testing"
)
// untypedFieldAllowlist holds package metadata fields that are allowed to carry untyped (interface) values.
// Values in these fields come straight from whatever a parser decoded, so they are not shaped by syft
// and can hold anything an encoder or hasher chokes on. Every entry is part of existing SBOM output and
// cannot be retyped without breaking it; new entries need a reason and a review.
var untypedFieldAllowlist = map[string]string{
"GGUFFileHeader.RemainingKeyValues": "raw GGUF header key-values; retyping changes the header output and metadataHash",
}
// TestNoUntypedMetadataFields fails when a package metadata type gains a field typed any, []any,
// map[string]any (or any other interface) that is not on the allowlist.
func TestNoUntypedMetadataFields(t *testing.T) {
var hits []string
for _, ty := range AllTypes() {
rt := reflect.TypeOf(ty)
collectUntyped(rt, rt.Name(), map[reflect.Type]bool{}, &hits)
}
seen := map[string]bool{}
for _, h := range hits {
seen[h] = true
if _, ok := untypedFieldAllowlist[h]; !ok {
t.Errorf("untyped field %s: use a concrete type, or add it to untypedFieldAllowlist with a reason", h)
}
}
for k := range untypedFieldAllowlist {
if !seen[k] {
t.Errorf("stale untypedFieldAllowlist entry %s: field no longer exists or is now typed", k)
}
}
}
// collectUntyped records "Struct.Field" for every field whose type reaches an interface, descending
// through pointers, slices, arrays, maps and nested syft-owned structs.
func collectUntyped(rt reflect.Type, path string, visited map[reflect.Type]bool, hits *[]string) {
switch rt.Kind() {
case reflect.Interface:
*hits = append(*hits, path)
case reflect.Pointer, reflect.Slice, reflect.Array:
collectUntyped(rt.Elem(), path, visited, hits)
case reflect.Map:
collectUntyped(rt.Key(), path, visited, hits)
collectUntyped(rt.Elem(), path, visited, hits)
case reflect.Struct:
if visited[rt] {
return
}
if rt.Name() != "" && !strings.HasPrefix(rt.PkgPath(), "github.com/anchore/syft/") {
// a third-party type in the model carries whatever shape the library gives it (the same leak as an
// interface field), so it counts as untyped. The standard library is left alone.
if strings.Contains(strings.Split(rt.PkgPath(), "/")[0], ".") {
*hits = append(*hits, path)
}
return
}
visited[rt] = true
for i := 0; i < rt.NumField(); i++ {
f := rt.Field(i)
// unexported embedded structs still promote their exported fields to the encoders and the hasher. A
// field is only out of reach when it is neither encoded nor hashed by SetID.
if (!f.IsExported() && !f.Anonymous) || (f.Tag.Get("json") == "-" && f.Tag.Get("hash") == "ignore") {
continue
}
collectUntyped(f.Type, rt.Name()+"."+f.Name, visited, hits)
}
}
}
+43
View File
@@ -44,6 +44,7 @@ type LicenseInfo struct {
// LicenseByURL returns the license ID and name for a given URL from the SPDX license list.
// The URL should match one of the URLs in the seeAlso field of an SPDX license.
// The scheme (http:// or https://) is stripped before lookup, so both schemes match.
// opensource.org URLs additionally match regardless of the form used (see normalizeOpenSourceOrgURL).
func LicenseByURL(url string) (LicenseInfo, bool) {
url = strings.TrimSpace(url)
url = stripScheme(url)
@@ -52,9 +53,51 @@ func LicenseByURL(url string) (LicenseInfo, bool) {
ID: id,
}, true
}
if normalized, ok := normalizeOpenSourceOrgURL(url); ok {
if id, exists := openSourceOrgURLToLicense[normalized]; exists {
return LicenseInfo{
ID: id,
}, true
}
}
return LicenseInfo{}, false
}
// openSourceOrgURLToLicense indexes the opensource.org entries of urlToLicense by normalized URL.
var openSourceOrgURLToLicense = buildOpenSourceOrgURLToLicense(urlToLicense)
func buildOpenSourceOrgURLToLicense(urls map[string]string) map[string]string {
index := make(map[string]string)
ambiguous := make(map[string]bool)
for url, id := range urls {
normalized, ok := normalizeOpenSourceOrgURL(url)
if !ok || ambiguous[normalized] {
continue
}
if existing, exists := index[normalized]; exists && existing != id {
// don't guess between licenses that only differ by URL form
delete(index, normalized)
ambiguous[normalized] = true
continue
}
index[normalized] = id
}
return index
}
// normalizeOpenSourceOrgURL reduces a scheme-less opensource.org URL to a canonical form.
// SPDX has changed these URLs over time (e.g. 3.29.0 replaced opensource.org/licenses/MIT and
// opensource.org/license/mit/ with opensource.org/license/MIT), while package metadata in the wild
// still uses the older forms. The www. prefix, /licenses/ vs /license/, case, and trailing slash are ignored.
func normalizeOpenSourceOrgURL(url string) (string, bool) {
url = strings.TrimPrefix(strings.ToLower(url), "www.")
if !strings.HasPrefix(url, "opensource.org/") {
return "", false
}
url = strings.Replace(url, "opensource.org/licenses/", "opensource.org/license/", 1)
return strings.TrimSuffix(url, "/"), true
}
// stripScheme removes http:// or https:// prefix from a URL.
// This allows a single map entry to match both schemes.
func stripScheme(url string) string {
+254 -199
View File
@@ -2,7 +2,7 @@
// This file was generated using data from https://spdx.org/licenses/licenses.json
package spdxlicense
const Version = "3.28.0"
const Version = "3.29.0"
var licenseIDs = map[string]string{
"0bsd": "0BSD",
@@ -116,6 +116,7 @@ var licenseIDs = map[string]string{
"aswfdigitalassets1.0.0": "ASWF-Digital-Assets-1.0",
"aswfdigitalassets1.1": "ASWF-Digital-Assets-1.1",
"aswfdigitalassets1.1.0": "ASWF-Digital-Assets-1.1",
"atcgame": "atc-game",
"baekmuk": "Baekmuk",
"bahyph": "Bahyph",
"barr": "Barr",
@@ -142,8 +143,11 @@ var licenseIDs = map[string]string{
"briangladman2.0clause": "Brian-Gladman-2-Clause",
"briangladman2clause": "Brian-Gladman-2-Clause",
"briangladman3.0.0clause": "Brian-Gladman-3-Clause",
"briangladman3.0.0clausenoconversion": "Brian-Gladman-3-Clause-no-conversion",
"briangladman3.0clause": "Brian-Gladman-3-Clause",
"briangladman3.0clausenoconversion": "Brian-Gladman-3-Clause-no-conversion",
"briangladman3clause": "Brian-Gladman-3-Clause",
"briangladman3clausenoconversion": "Brian-Gladman-3-Clause-no-conversion",
"bsd1.0.0clause": "BSD-1-Clause",
"bsd1.0clause": "BSD-1-Clause",
"bsd1clause": "BSD-1-Clause",
@@ -154,6 +158,7 @@ var licenseIDs = map[string]string{
"bsd2.0.0clausenetbsd": "BSD-2-Clause",
"bsd2.0.0clausepatent": "BSD-2-Clause-Patent",
"bsd2.0.0clausepkgconfdisclaimer": "BSD-2-Clause-pkgconf-disclaimer",
"bsd2.0.0clauseposunchanged": "BSD-2-Clause-pos-unchanged",
"bsd2.0.0clauseviews": "BSD-2-Clause-Views",
"bsd2.0clause": "BSD-2-Clause",
"bsd2.0clausedarwin": "BSD-2-Clause-Darwin",
@@ -162,6 +167,7 @@ var licenseIDs = map[string]string{
"bsd2.0clausenetbsd": "BSD-2-Clause",
"bsd2.0clausepatent": "BSD-2-Clause-Patent",
"bsd2.0clausepkgconfdisclaimer": "BSD-2-Clause-pkgconf-disclaimer",
"bsd2.0clauseposunchanged": "BSD-2-Clause-pos-unchanged",
"bsd2.0clauseviews": "BSD-2-Clause-Views",
"bsd2clause": "BSD-2-Clause",
"bsd2clausedarwin": "BSD-2-Clause-Darwin",
@@ -170,6 +176,7 @@ var licenseIDs = map[string]string{
"bsd2clausenetbsd": "BSD-2-Clause",
"bsd2clausepatent": "BSD-2-Clause-Patent",
"bsd2clausepkgconfdisclaimer": "BSD-2-Clause-pkgconf-disclaimer",
"bsd2clauseposunchanged": "BSD-2-Clause-pos-unchanged",
"bsd2clauseviews": "BSD-2-Clause-Views",
"bsd3.0.0clause": "BSD-3-Clause",
"bsd3.0.0clauseacpica": "BSD-3-Clause-acpica",
@@ -184,6 +191,7 @@ var licenseIDs = map[string]string{
"bsd3.0.0clausenonuclearlicense2014": "BSD-3-Clause-No-Nuclear-License-2014",
"bsd3.0.0clausenonuclearwarranty": "BSD-3-Clause-No-Nuclear-Warranty",
"bsd3.0.0clauseopenmpi": "BSD-3-Clause-Open-MPI",
"bsd3.0.0clauseopenwebui": "BSD-3-Clause-OpenWebUI",
"bsd3.0.0clausesun": "BSD-3-Clause-Sun",
"bsd3.0.0clausetso": "BSD-3-Clause-Tso",
"bsd3.0clause": "BSD-3-Clause",
@@ -199,6 +207,7 @@ var licenseIDs = map[string]string{
"bsd3.0clausenonuclearlicense2014": "BSD-3-Clause-No-Nuclear-License-2014",
"bsd3.0clausenonuclearwarranty": "BSD-3-Clause-No-Nuclear-Warranty",
"bsd3.0clauseopenmpi": "BSD-3-Clause-Open-MPI",
"bsd3.0clauseopenwebui": "BSD-3-Clause-OpenWebUI",
"bsd3.0clausesun": "BSD-3-Clause-Sun",
"bsd3.0clausetso": "BSD-3-Clause-Tso",
"bsd3clause": "BSD-3-Clause",
@@ -214,6 +223,7 @@ var licenseIDs = map[string]string{
"bsd3clausenonuclearlicense2014": "BSD-3-Clause-No-Nuclear-License-2014",
"bsd3clausenonuclearwarranty": "BSD-3-Clause-No-Nuclear-Warranty",
"bsd3clauseopenmpi": "BSD-3-Clause-Open-MPI",
"bsd3clauseopenwebui": "BSD-3-Clause-OpenWebUI",
"bsd3clausesun": "BSD-3-Clause-Sun",
"bsd3clausetso": "BSD-3-Clause-Tso",
"bsd4.0.0clause": "BSD-4-Clause",
@@ -232,12 +242,15 @@ var licenseIDs = map[string]string{
"bsd4reno": "BSD-4.3RENO",
"bsd4tahoe": "BSD-4.3TAHOE",
"bsdadvertisingacknowledgement": "BSD-Advertising-Acknowledgement",
"bsdasktoendorse": "BSD-ask-to-endorse",
"bsdattributionhpnddisclaimer": "BSD-Attribution-HPND-disclaimer",
"bsdinfernonettverk": "BSD-Inferno-Nettverk",
"bsdmarkmodifications": "BSD-Mark-Modifications",
"bsdprotection": "BSD-Protection",
"bsdsourcealtgpl": "BSD-Source-alt-GPL",
"bsdsourcebeginningfile": "BSD-Source-beginning-file",
"bsdsourcecode": "BSD-Source-Code",
"bsdsourcecodenodisclaimer": "BSD-Source-Code-no-disclaimer",
"bsdsystemics": "BSD-Systemics",
"bsdsystemicsw3.0.0works": "BSD-Systemics-W3Works",
"bsdsystemicsw3.0works": "BSD-Systemics-W3Works",
@@ -246,6 +259,7 @@ var licenseIDs = map[string]string{
"bsl1.0": "BSL-1.0",
"bsl1.0.0": "BSL-1.0",
"buddy": "Buddy",
"bugroff": "Bugroff",
"busl1": "BUSL-1.1",
"busl1.1": "BUSL-1.1",
"busl1.1.0": "BUSL-1.1",
@@ -316,8 +330,11 @@ var licenseIDs = map[string]string{
"ccbync3.0": "CC-BY-NC-3.0",
"ccbync3.0.0": "CC-BY-NC-3.0",
"ccbync3.0.0de": "CC-BY-NC-3.0-DE",
"ccbync3.0.0igo": "CC-BY-NC-3.0-IGO",
"ccbync3.0de": "CC-BY-NC-3.0-DE",
"ccbync3.0igo": "CC-BY-NC-3.0-IGO",
"ccbync3de": "CC-BY-NC-3.0-DE",
"ccbync3igo": "CC-BY-NC-3.0-IGO",
"ccbync4": "CC-BY-NC-4.0",
"ccbync4.0": "CC-BY-NC-4.0",
"ccbync4.0.0": "CC-BY-NC-4.0",
@@ -592,6 +609,7 @@ var licenseIDs = map[string]string{
"fair": "Fair",
"fbm": "FBM",
"fdkaac": "FDK-AAC",
"fdkmpegh": "FDK-MPEG-H",
"fergusontwofish": "Ferguson-Twofish",
"frameworx1": "Frameworx-1.0",
"frameworx1.0": "Frameworx-1.0",
@@ -767,6 +785,9 @@ var licenseIDs = map[string]string{
"hippocratic2": "Hippocratic-2.1",
"hippocratic2.1": "Hippocratic-2.1",
"hippocratic2.1.0": "Hippocratic-2.1",
"hippocratic3.0.0core": "Hippocratic-3.0-core",
"hippocratic3.0core": "Hippocratic-3.0-core",
"hippocratic3core": "Hippocratic-3.0-core",
"hp1986": "HP-1986",
"hp1986.0": "HP-1986",
"hp1986.0.0": "HP-1986",
@@ -813,6 +834,7 @@ var licenseIDs = map[string]string{
"imlib2": "Imlib2",
"imlib2.0": "Imlib2",
"imlib2.0.0": "Imlib2",
"informatica": "Informatica",
"infozip": "Info-ZIP",
"innernet2": "Inner-Net-2.0",
"innernet2.0": "Inner-Net-2.0",
@@ -1011,6 +1033,9 @@ var licenseIDs = map[string]string{
"mulanpsl2.0.0": "MulanPSL-2.0",
"multics": "Multics",
"mup": "Mup",
"mvt1": "MVT-1.1",
"mvt1.1": "MVT-1.1",
"mvt1.1.0": "MVT-1.1",
"naist2003": "NAIST-2003",
"naist2003.0": "NAIST-2003",
"naist2003.0.0": "NAIST-2003",
@@ -1489,6 +1514,8 @@ var urlToLicense = map[string]string{
"cdla.io/permissive-1-0": "CDLA-Permissive-1.0",
"cdla.io/sharing-1-0": "CDLA-Sharing-1.0",
"celestrak.org/publications/AIAA/2006-6753/faq.php": "SGP4",
"cgit.freebsd.org/src/plain/sys/dev/aic7xxx/aic7xxx_osm.c": "BSD-Source-alt-GPL",
"cgit.freebsd.org/src/tree/usr.bin/sockstat/sockstat.c": "BSD-2-Clause-pos-unchanged",
"cgit.freedesktop.org/libbsd/tree/man/setproctitle.3bsd": "libutil-David-Nugent",
"coil.apotheon.org/plaintext/01.0.txt": "COIL-1.0",
"core.tcl-lang.org/tk/file?name=compat/unistd.h": "HPND-UC",
@@ -1512,6 +1539,7 @@ var urlToLicense = map[string]string{
"creativecommons.org/licenses/by-nc/2.0/legalcode": "CC-BY-NC-2.0",
"creativecommons.org/licenses/by-nc/2.5/legalcode": "CC-BY-NC-2.5",
"creativecommons.org/licenses/by-nc/3.0/de/legalcode": "CC-BY-NC-3.0-DE",
"creativecommons.org/licenses/by-nc/3.0/igo/legalcode": "CC-BY-NC-3.0-IGO",
"creativecommons.org/licenses/by-nc/3.0/legalcode": "CC-BY-NC-3.0",
"creativecommons.org/licenses/by-nc/4.0/legalcode": "CC-BY-NC-4.0",
"creativecommons.org/licenses/by-nd-nc/1.0/legalcode": "CC-BY-NC-ND-1.0",
@@ -1564,6 +1592,7 @@ var urlToLicense = map[string]string{
"directory.fsf.org/wiki/License:Fdk": "FDK-AAC",
"doc.qt.io/archives/3.3/license.html": "QPL-1.0",
"doc.qt.nokia.com/3.3/license.html": "QPL-1.0",
"docs.mvt.re/en/latest/license/": "MVT-1.1",
"docs.omniverse.nvidia.com/install-guide/latest/common/licenses.html": "AML-glslang",
"docs.oracle.com/cd/E81115_01/html/E81116/licenses.html": "man2html",
"docs.python.org/3/license.html": "Python-2.0.1",
@@ -1669,6 +1698,7 @@ var urlToLicense = map[string]string{
"fedoraproject.org/wiki/Licensing:Mplus?rd=Licensing/mplus": "mplus",
"fedoraproject.org/wiki/Licensing:ThorPublicLicense": "TPL-1.0",
"firstdonoharm.dev/version/2/1/license.html": "Hippocratic-2.1",
"firstdonoharm.dev/version/3/0/core.txt": "Hippocratic-3.0-core",
"forge.gouv.qc.ca/licence/fr/liliq-v1-1/": "LiLiQ-P-1.1",
"forge.gouv.qc.ca/licence/liliq-p": "LiLiQ-R-1.1",
"forge.gouv.qc.ca/licence/liliq-p/": "LiLiQ-P-1.1",
@@ -1686,68 +1716,70 @@ var urlToLicense = map[string]string{
"ftp://ftp.tin.org/pub/news/utils/newsx/newsx-1.6.tar.gz": "Zeeff",
"gianluca.dellavedova.org/2011/01/03/clarified-artistic-license/": "ClArtistic",
"git.denx.de/?p=u-boot.git;a=blob;f=arch/powerpc/cpu/ppc4xx/miiphy.c;h=297155fdafa064b955e53e9832de93bfb0cfb85b;hb=9fab4bf4cc077c21e43941866f3f2c196f28670d": "IBM-pibs",
"git.enlightenment.org/legacy/imlib2.git/tree/COPYING": "Imlib2",
"git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/man2/getcpu.2#n4": "Linux-man-pages-1-para",
"git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/man2/migrate_pages.2#n8": "Linux-man-pages-copyleft-2-para",
"git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/man2/move_pages.2#n5": "Linux-man-pages-copyleft-2-para",
"git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/man2/set_mempolicy.2#n5": "Linux-man-pages-copyleft-var",
"git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/core/sa.h": "Linux-OpenIB",
"git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/sunrpc/auth_gss/gss_generic_token.c?h=v4.19": "HPND-sell-variant",
"git.openldap.org/openldap/openldap/-/blob/master/COPYRIGHT#L55-63": "BSD-4.3RENO",
"git.openldap.org/openldap/openldap/-/blob/master/COPYRIGHT?ref_type=heads#L39-51": "Furuseth",
"git.openldap.org/openldap/openldap/-/blob/master/libraries/libldap/os-local.c?ref_type=heads#L19-23": "PADL",
"git.savannah.gnu.org/cgit/freetype/freetype2.git/tree/docs/FTL.TXT": "FTL",
"git.savannah.gnu.org/cgit/gcl.git/tree/gcl/lsp/gcl_loop.lsp?h=Version_2_6_13pre": "LOOP",
"git.savannah.gnu.org/cgit/gnulib.git/tree/modules/COPYING?id=7b08932179d0d6b017f7df01a2ddf6e096b038e3": "FSFULLRSD",
"git.savannah.gnu.org/cgit/indent.git/tree/doc/indent.texi?id=a74c6b4ee49397cf330b333da1042bffa60ed14f#n1788": "BSD-4.3TAHOE",
"git.savannah.gnu.org/cgit/indent.git/tree/doc/indent.texi?id=a74c6b4ee49397cf330b333da1042bffa60ed14f#n74": "Latex2e-translated-notice",
"git.savannah.gnu.org/cgit/kazlib.git/tree/except.c?id=0062df360c2d17d57f6af19b0e444c51feb99036": "Kazlib",
"git.savannah.gnu.org/cgit/wget.git/tree/util/trunc.c?h=v1.21.3&id=40747a11e44ced5a8ac628a41f879ced3e2ebce9#n6": "FSFAP-no-warranty-disclaimer",
"git.sr.ht/~nabijaczleweli/ossp-var": "OSSP",
"github.com/389ds/389-ds-base/blob/main/ldap/include/sysexits-compat.h#L15": "BSD-4.3TAHOE",
"github.com/AcademySoftwareFoundation/foundation/blob/main/digital_assets/aswf_digital_assets_license_v1.0.txt": "ASWF-Digital-Assets-1.0",
"github.com/AcademySoftwareFoundation/foundation/blob/main/digital_assets/aswf_digital_assets_license_v1.1.txt": "ASWF-Digital-Assets-1.1",
"github.com/AdaCore/gnatcoll-core/blob/master/docs/index.rst": "AdaCore-doc",
"github.com/AdaCore/gnatcoll-db/blob/master/docs/index.rst": "AdaCore-doc",
"github.com/AdaCore/xmlada/blob/master/docs/index.rst": "AdaCore-doc",
"github.com/BuildCraft/BuildCraft/blob/623d323b1868712f29f4a8b0979a02e8d1835131/LICENSE": "MMPL-1.0.1",
"github.com/CommunitySpecification/1.0/blob/master/1._Community_Specification_License-v1.md": "Community-Spec-1.0",
"github.com/CorsixTH/deps/blob/fd339a9f526d1d9c9f01ccf39e438a015da50035/licences/libgsm.txt": "TU-Berlin-2.0",
"github.com/Cyan4973/xxHash/blob/dev/doc/xxhash_spec.md": "LPD-document",
"github.com/DISIC/politique-de-contribution-open-source/blob/master/LICENSE.pdf": "etalab-2.0",
"github.com/Distrotech/hdparm/blob/4517550db29a91420fb2b020349523b1b4512df2/LICENSE.TXT": "hdparm",
"github.com/Distrotech/transfig/blob/master/transfig/transfig.c": "Xfig",
"github.com/DrItanium/maya/blob/master/LICENSE.CLIPS": "Clips",
"github.com/Dual-Life/mime-base64/blob/master/Base64.xs#L12": "metamail",
"github.com/EthicalSource/hippocratic-license/blob/58c0e646d64ff6fbee275bfe2b9492f914e3ab2a/LICENSE.txt": "Hippocratic-2.1",
"github.com/FreeRADIUS/freeradius-client/blob/master/COPYRIGHT#L32": "HPND-Fenneberg-Livingston",
"github.com/GNOME/gcr/blob/master/docs/COPYING": "GCR-docs",
"github.com/GNOME/libxml2/blob/master/dict.c": "ISC-Veillard",
"github.com/HDFGroup/hdf5/?tab=License-1-ov-file#readme": "HDF5",
"github.com/KhronosGroup/SPIRV-Cross/blob/main/LICENSES/LicenseRef-KhronosFreeUse.txt": "MIT-Khronos-old",
"github.com/KhronosGroup/glslang/blob/main/LICENSE.txt#L949": "AML-glslang",
"github.com/MariaDB/server/blob/11.6/libmysqld/lib_sql.cc": "Boehm-GC-without-fee",
"github.com/Neo23x0/sigma/blob/master/LICENSE.Detection.Rules.md": "DRL-1.0",
"github.com/OGRECave/ogre/blob/master/OgreMain/include/OgreSingleton.h#L28C3-L35C46": "Game-Programming-Gems",
"github.com/Perl/perl5/blob/blead/util.c#L6136": "Martin-Birgmeier",
"github.com/PixarAnimationStudios/OpenSubdiv/blob/v3_5_0/opensubdiv/version.cpp#L2-L22": "Pixar",
"github.com/PixarAnimationStudios/OpenSubdiv/raw/v3_5_0/LICENSE.txt": "Pixar",
"github.com/RTimothyEdwards/magic/blob/master/LICENSE": "HPND-UC-export-US",
"github.com/SELinuxProject/selinux/blob/master/libselinux/LICENSE": "libselinux-1.0",
"github.com/SWI-Prolog/packages-clib/blob/master/sha1/brg_endian.h": "Brian-Gladman-3-Clause",
"github.com/SWI-Prolog/packages-xpce/blob/161a40cd82004f731ba48024f9d30af388a7edf5/src/img/gifwrite.c#L21-L26": "FBM",
"github.com/SWI-Prolog/swipl-devel/blob/master/src/os/dtoa.c": "dtoa",
"github.com/SigmaHQ/Detection-Rule-License/blob/6ec7fbde6101d101b5b5d1fcb8f9b69fbc76c04a/LICENSE.Detection.Rules.md": "DRL-1.1",
"github.com/Slicer/Slicer/blob/main/License.txt": "3D-Slicer-1.0",
"github.com/Unidata/UDUNITS-2/blob/master/COPYRIGHT": "UCAR",
"github.com/acpica/acpica/blob/master/source/common/acfileio.c#L119": "BSD-3-Clause-acpica",
"github.com/adaptivecomputing/torque/blob/master/PBS_License.txt": "OpenPBS-2.3",
"github.com/adversary-org/wtfnmf/raw/3f2cd8235a64350a57a51b9739715edaea63ec1a/COPYING.WTFNMFPL-utf8": "WTFNMFPL",
"github.com/adversary-org/wtfnmf/raw/refs/tags/1.0/COPYING.WTFNMFPL": "WTFNMFPL",
"github.com/ap/Text-Tabs/blob/master/lib.modern/Text/Tabs.pm#L148": "TTWL",
"github.com/apache/apr/blob/trunk/LICENSE#L298C6-L298C29": "Caldera-no-preamble",
"github.com/audacious-media-player/audacious/blob/master/COPYING": "BSD-2-Clause-pkgconf-disclaimer",
"github.com/audacious-media-player/audacious/blob/master/src/audacious/main.cc": "BSD-2-Clause-pkgconf-disclaimer",
"git.enlightenment.org/legacy/imlib2.git/tree/COPYING": "Imlib2",
"git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/man2/getcpu.2#n4": "Linux-man-pages-1-para",
"git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/man2/migrate_pages.2#n8": "Linux-man-pages-copyleft-2-para",
"git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/man2/move_pages.2#n5": "Linux-man-pages-copyleft-2-para",
"git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/man2/set_mempolicy.2#n5": "Linux-man-pages-copyleft-var",
"git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/core/sa.h": "Linux-OpenIB",
"git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/sunrpc/auth_gss/gss_generic_token.c?h=v4.19": "HPND-sell-variant",
"git.openldap.org/openldap/openldap/-/blob/master/COPYRIGHT#L55-63": "BSD-4.3RENO",
"git.openldap.org/openldap/openldap/-/blob/master/COPYRIGHT?ref_type=heads#L39-51": "Furuseth",
"git.openldap.org/openldap/openldap/-/blob/master/libraries/libldap/os-local.c?ref_type=heads#L19-23": "PADL",
"git.savannah.gnu.org/cgit/freetype/freetype2.git/tree/docs/FTL.TXT": "FTL",
"git.savannah.gnu.org/cgit/gcl.git/tree/gcl/lsp/gcl_loop.lsp?h=Version_2_6_13pre": "LOOP",
"git.savannah.gnu.org/cgit/gnulib.git/tree/modules/COPYING?id=7b08932179d0d6b017f7df01a2ddf6e096b038e3": "FSFULLRSD",
"git.savannah.gnu.org/cgit/indent.git/tree/doc/indent.texi?id=a74c6b4ee49397cf330b333da1042bffa60ed14f#n1788": "BSD-4.3TAHOE",
"git.savannah.gnu.org/cgit/indent.git/tree/doc/indent.texi?id=a74c6b4ee49397cf330b333da1042bffa60ed14f#n74": "Latex2e-translated-notice",
"git.savannah.gnu.org/cgit/kazlib.git/tree/except.c?id=0062df360c2d17d57f6af19b0e444c51feb99036": "Kazlib",
"git.savannah.gnu.org/cgit/wget.git/tree/util/trunc.c?h=v1.21.3&id=40747a11e44ced5a8ac628a41f879ced3e2ebce9#n6": "FSFAP-no-warranty-disclaimer",
"git.sr.ht/~nabijaczleweli/ossp-var": "OSSP",
"github.com/389ds/389-ds-base/blob/main/ldap/include/sysexits-compat.h#L15": "BSD-4.3TAHOE",
"github.com/AcademySoftwareFoundation/foundation/blob/main/digital_assets/aswf_digital_assets_license_v1.0.txt": "ASWF-Digital-Assets-1.0",
"github.com/AcademySoftwareFoundation/foundation/blob/main/digital_assets/aswf_digital_assets_license_v1.1.txt": "ASWF-Digital-Assets-1.1",
"github.com/AdaCore/gnatcoll-core/blob/master/docs/index.rst": "AdaCore-doc",
"github.com/AdaCore/gnatcoll-db/blob/master/docs/index.rst": "AdaCore-doc",
"github.com/AdaCore/xmlada/blob/master/docs/index.rst": "AdaCore-doc",
"github.com/BuildCraft/BuildCraft/blob/623d323b1868712f29f4a8b0979a02e8d1835131/LICENSE": "MMPL-1.0.1",
"github.com/CommunitySpecification/1.0/blob/master/1._Community_Specification_License-v1.md": "Community-Spec-1.0",
"github.com/CorsixTH/deps/blob/fd339a9f526d1d9c9f01ccf39e438a015da50035/licences/libgsm.txt": "TU-Berlin-2.0",
"github.com/Cyan4973/xxHash/blob/dev/doc/xxhash_spec.md": "LPD-document",
"github.com/DISIC/politique-de-contribution-open-source/blob/master/LICENSE.pdf": "etalab-2.0",
"github.com/Distrotech/hdparm/blob/4517550db29a91420fb2b020349523b1b4512df2/LICENSE.TXT": "hdparm",
"github.com/Distrotech/transfig/blob/master/transfig/transfig.c": "Xfig",
"github.com/DrItanium/maya/blob/master/LICENSE.CLIPS": "Clips",
"github.com/Dual-Life/mime-base64/blob/master/Base64.xs#L12": "metamail",
"github.com/EthicalSource/hippocratic-license/blob/58c0e646d64ff6fbee275bfe2b9492f914e3ab2a/LICENSE.txt": "Hippocratic-2.1",
"github.com/Fraunhofer-IIS/mpeghdec/blob/main/LICENSE.txt": "FDK-MPEG-H",
"github.com/FreeRADIUS/freeradius-client/blob/master/COPYRIGHT#L32": "HPND-Fenneberg-Livingston",
"github.com/GNOME/gcr/blob/master/docs/COPYING": "GCR-docs",
"github.com/GNOME/libxml2/blob/master/dict.c": "ISC-Veillard",
"github.com/HDFGroup/hdf5/?tab=License-1-ov-file#readme": "HDF5",
"github.com/KhronosGroup/SPIRV-Cross/blob/main/LICENSES/LicenseRef-KhronosFreeUse.txt": "MIT-Khronos-old",
"github.com/KhronosGroup/glslang/blob/main/LICENSE.txt#L949": "AML-glslang",
"github.com/MariaDB/server/blob/11.6/libmysqld/lib_sql.cc": "Boehm-GC-without-fee",
"github.com/Neo23x0/sigma/blob/master/LICENSE.Detection.Rules.md": "DRL-1.0",
"github.com/OGRECave/ogre/blob/master/OgreMain/include/OgreSingleton.h#L28C3-L35C46": "Game-Programming-Gems",
"github.com/Perl/perl5/blob/blead/util.c#L6136": "Martin-Birgmeier",
"github.com/PicoEFI/PicoEFI/blob/2263ac1a2bc98e5534d64a4896d3cfbbae00745a/LICENSES/LicenseRef-scancode-bsd-no-disclaimer-unmodified.txt": "BSD-Source-Code-no-disclaimer",
"github.com/PixarAnimationStudios/OpenSubdiv/blob/v3_5_0/opensubdiv/version.cpp#L2-L22": "Pixar",
"github.com/PixarAnimationStudios/OpenSubdiv/raw/v3_5_0/LICENSE.txt": "Pixar",
"github.com/RTimothyEdwards/magic/blob/master/LICENSE": "HPND-UC-export-US",
"github.com/SELinuxProject/selinux/blob/master/libselinux/LICENSE": "libselinux-1.0",
"github.com/SWI-Prolog/packages-clib/blob/master/sha1/brg_endian.h": "Brian-Gladman-3-Clause",
"github.com/SWI-Prolog/packages-xpce/blob/161a40cd82004f731ba48024f9d30af388a7edf5/src/img/gifwrite.c#L21-L26": "FBM",
"github.com/SWI-Prolog/swipl-devel/blob/master/src/os/dtoa.c": "dtoa",
"github.com/SigmaHQ/Detection-Rule-License/blob/6ec7fbde6101d101b5b5d1fcb8f9b69fbc76c04a/LICENSE.Detection.Rules.md": "DRL-1.1",
"github.com/Slicer/Slicer/blob/main/License.txt": "3D-Slicer-1.0",
"github.com/Unidata/UDUNITS-2/blob/master/COPYRIGHT": "UCAR",
"github.com/acpica/acpica/blob/master/source/common/acfileio.c#L119": "BSD-3-Clause-acpica",
"github.com/adaptivecomputing/torque/blob/master/PBS_License.txt": "OpenPBS-2.3",
"github.com/adversary-org/wtfnmf/raw/3f2cd8235a64350a57a51b9739715edaea63ec1a/COPYING.WTFNMFPL-utf8": "WTFNMFPL",
"github.com/adversary-org/wtfnmf/raw/refs/tags/1.0/COPYING.WTFNMFPL": "WTFNMFPL",
"github.com/ap/Text-Tabs/blob/master/lib.modern/Text/Tabs.pm#L148": "TTWL",
"github.com/apache/apr/blob/trunk/LICENSE#L298C6-L298C29": "Caldera-no-preamble",
"github.com/audacious-media-player/audacious/blob/master/COPYING": "BSD-2-Clause-pkgconf-disclaimer",
"github.com/audacious-media-player/audacious/blob/master/src/audacious/main.cc": "BSD-2-Clause-pkgconf-disclaimer",
"github.com/aws/mit-0": "MIT-0",
"github.com/awsdocs/aws-cloud9-user-guide/blob/master/LICENSE-SAMPLECODE": "MIT-0",
"github.com/bagder/curl/blob/master/COPYING": "curl",
@@ -1772,6 +1804,7 @@ var urlToLicense = map[string]string{
"github.com/file/file/blob/master/COPYING": "BSD-2-Clause-Darwin",
"github.com/freebsd/freebsd-src/blob/main/lib/msun/src/e_acosh.c": "SunPro",
"github.com/freebsd/freebsd-src/blob/main/lib/msun/src/e_lgammal.c": "SunPro",
"github.com/freebsd/pkg/blob/main/src/add.c": "BSD-2-Clause-pos-unchanged",
"github.com/gabegundy/atlserver/blob/master/License.txt": "MS-LPL",
"github.com/greymass/swift-eosio/blob/master/LICENSE": "BSD-3-Clause-No-Military-License",
"github.com/hamano/man2html/blob/master/man2html.c": "man2html",
@@ -1784,6 +1817,7 @@ var urlToLicense = map[string]string{
"github.com/jsommers/pytricia/blob/master/patricia.c#L33-L67": "BSD-4-Clause",
"github.com/kfish/xsel/blob/master/COPYING": "HPND-sell-variant",
"github.com/kohler/t1utils/blob/master/LICENSE": "MIT-Click",
"github.com/krb5/krb5/blob/kfw-4.0.1-beta1/src/lib/crypto/builtin/aes/aesopt.h": "Brian-Gladman-3-Clause-no-conversion",
"github.com/krb5/krb5/blob/krb5-1.21.2-final/NOTICE#L111-L133": "HPND-export2-US",
"github.com/krb5/krb5/blob/krb5-1.21.2-final/NOTICE#L1157-L1182": "HPND-export-US-modify",
"github.com/krb5/krb5/blob/krb5-1.21.2-final/NOTICE#L140-L156": "Brian-Gladman-2-Clause",
@@ -1793,6 +1827,7 @@ var urlToLicense = map[string]string{
"github.com/krb5/krb5/blob/krb5-1.21.2-final/NOTICE#L831-L852": "HPND-export-US-acknowledgement",
"github.com/krzyzanowskim/CryptoSwift/blob/main/LICENSE": "CryptoSwift",
"github.com/lattera/freebsd/blob/master/sys/cam/cam.c#L4": "BSD-Source-beginning-file",
"github.com/libarchive/libarchive/blob/master/libarchive_fe/line_reader.c": "BSD-2-Clause-pos-unchanged",
"github.com/libcheck/check/blob/master/checkmk/checkmk.in": "checkmk",
"github.com/libjpeg-turbo/libjpeg-turbo/blob/main/README.ijg#L117-L161": "IJG",
"github.com/lsof-org/lsof/blob/master/COPYING": "lsof",
@@ -1805,6 +1840,7 @@ var urlToLicense = map[string]string{
"github.com/mm2/Little-CMS/blob/master/src/cmssm.c#L207": "softSurfer",
"github.com/mruby/mruby/blob/83d12f8d52522cdb7c8cc46fad34821359f453e6/mrbgems/mruby-dir/src/Win/dirent.c#L127-L140": "HPND-Kevlin-Henney",
"github.com/mtoyoda/sl/blob/master/LICENSE": "SL",
"github.com/mvt-project/mvt/blob/main/LICENSE": "MVT-1.1",
"github.com/n8n-io/n8n/blob/master/LICENSE.md": "SUL-1.0",
"github.com/ncbi/datasets/blob/0ea4cd16b61e5b799d9cc55aecfa016d6c9bd2bf/LICENSE.md": "NCBI-PD",
"github.com/ncbi/datasets/blob/master/LICENSE.md": "NCBI-PD",
@@ -1815,6 +1851,7 @@ var urlToLicense = map[string]string{
"github.com/novnc/noVNC/blob/master/core/crypto/des.js#L24": "Widget-Workshop",
"github.com/oetiker/rrdtool-1.x/blob/master/src/plbasename.c#L8-L11": "gtkbook",
"github.com/open-quantum-safe/liboqs/blob/40b01fdbb270f8614fde30e65d30e9da18c02393/src/common/rand/rand_nist.c#L1-L15": "NIST-Software",
"github.com/open-webui/open-webui/blob/main/LICENSE": "BSD-3-Clause-OpenWebUI",
"github.com/openjdk/jdk/blob/6162e2c5213c5dd7c1127fd9616b543efa898962/LICENSE": "GPL-2.0-or-later",
"github.com/openssh/openssh-portable/blob/1b11ea7c58cd5c59838b5fa574cd456d6047b2d4/LICENCE#L10": "SSH-OpenSSH",
"github.com/openssh/openssh-portable/blob/1b11ea7c58cd5c59838b5fa574cd456d6047b2d4/pathnames.h": "SSH-short",
@@ -1846,6 +1883,7 @@ var urlToLicense = map[string]string{
"github.com/sigmavirus24/x11-ssh-askpass/blob/master/dynlist.c": "HPND-sell-variant-MIT-disclaimer-rev",
"github.com/signal11/hidapi/blob/master/LICENSE-orig.txt": "HIDAPI",
"github.com/slogan621/gtkbook": "gtkbook",
"github.com/sudo-project/sudo/blob/c1307ea9ff340ce0538779f8e456501461fc44b7/plugins/sudoers/redblack.c#L24-L43": "BSD-ask-to-endorse",
"github.com/swh/ladspa/blob/7bf6f3799fdba70fda297c2d8fd9f526803d9680/gsm/COPYRIGHT": "TU-Berlin-1.0",
"github.com/tats/w3m/blob/master/COPYING": "w3m",
"github.com/tcheneau/Routing/blob/f09f46fcfe636107f22f2c98348188a65a135d98/README.md": "NIST-PD",
@@ -1856,6 +1894,7 @@ var urlToLicense = map[string]string{
"github.com/uim/libgcroots/blob/master/COPYING": "Boehm-GC",
"github.com/usnistgov/fipy/blob/86aaa5c2ba2c6f1be19593c5986071cf6568cc34/LICENSE.rst": "NIST-PD-fallback",
"github.com/usnistgov/jsip/blob/59700e6926cbe96c5cdae897d9a7d2656b42abe3/LICENSE": "NIST-PD-fallback",
"github.com/vattam/BSDGames/blob/master/atc/extern.c#L36": "atc-game",
"github.com/vixie/cron/tree/545b3f5246824a9cda5905eeb7cf019c95e66995": "Vixie-Cron",
"github.com/vstroebel/jpeg-encoder/blob/main/src/fdct.rs#L1-L72": "IJG",
"github.com/wernerd/ZRTPCPP/blob/6b3cd8e6783642292bad0c21e3e5e5ce45ff3e03/cryptcommon/twofish.c#L113C3-L127": "Ferguson-Twofish",
@@ -1869,6 +1908,7 @@ var urlToLicense = map[string]string{
"gitlab.com/embeddable-common-lisp/ecl/-/blob/develop/src/lsp/loop.lsp": "LOOP",
"gitlab.com/libeigen/eigen/-/blob/master/COPYING.MINPACK": "Minpack",
"gitlab.common-lisp.net/cmucl/cmucl/-/blob/master/src/code/loop.lisp": "LOOP",
"gitlab.freedesktop.org/dbus/dbus/-/blob/789d97ad53044f94351a958e785c2923643b346a/COPYING#L11-207": "AFL-2.1",
"gitlab.freedesktop.org/freetype/freetype/-/raw/master/docs/FTL.TXT": "FTL",
"gitlab.freedesktop.org/libopenraw/libopenraw/blob/master/lib/ljpegdecompressor.cpp#L32": "Cornell-Lossless-JPEG",
"gitlab.freedesktop.org/mesa/glw/-/blob/master/README?ref_type=heads": "SGI-OpenGL",
@@ -1892,6 +1932,7 @@ var urlToLicense = map[string]string{
"gitlab.freedesktop.org/xorg/lib/libxinerama/-/blob/cc22c2f60c3862482562955116d5455263b443dc/COPYING#L44-66": "X11-no-permit-persons",
"gitlab.freedesktop.org/xorg/lib/libxtst/-/blob/master/COPYING?ref_type=heads#L108-117": "HPND-doc-sell",
"gitlab.freedesktop.org/xorg/lib/libxtst/-/blob/master/COPYING?ref_type=heads#L70-77": "HPND-doc",
"gitlab.freedesktop.org/xorg/xserver/-/blob/dd5c2595a42d3ff0c4f18d9b53d1f6c3fd934fd4/COPYING#L365-389": "MIT",
"gitlab.freedesktop.org/xorg/xserver/-/blob/master/COPYING?ref_type=heads#L1781": "HPND-sell-MIT-disclaimer-xserver",
"gitlab.freedesktop.org/xorg/xserver/-/blob/master/COPYING?ref_type=heads#L239": "DEC-3-Clause",
"gitlab.freedesktop.org/xorg/xserver/-/blob/master/COPYING?ref_type=heads#L752": "Adobe-Display-PostScript",
@@ -1906,6 +1947,7 @@ var urlToLicense = map[string]string{
"heasarc.gsfc.nasa.gov/docs/software/ftools/fv/doc/license.html": "CFITSIO",
"helixcommunity.org/content/rpsl": "RPSL-1.0",
"hstechdocs.helpsystems.com/manuals/globalscape/archive/cuteftp6/open_ssl_license_agreement.htm": "OpenSSL-standalone",
"interoperable-europe.ec.europa.eu/sites/default/files/inline-files/EUPL%20v1_2%20EN(1).txt": "EUPL-1.2",
"java.net/projects/javaeetutorial/pages/BerkeleyLicense": "BSD-3-Clause-No-Nuclear-License-2014",
"javaee.github.io/glassfish/LICENSE": "CDDL-1.1",
"jogamp.org/git/?p=gluegen.git;a=blob_plain;f=LICENSE.txt": "BSD-3-Clause-No-Nuclear-Warranty",
@@ -1924,6 +1966,7 @@ var urlToLicense = map[string]string{
"library.netapp.com/ecm/ecm_download_file/ECMP1196395": "OpenSSL-standalone",
"license.coscl.org.cn/MulanPSL/": "MulanPSL-1.0",
"license.coscl.org.cn/MulanPSL2": "MulanPSL-2.0",
"license.mvt.re/1.1/": "MVT-1.1",
"lists.gnu.org/archive/html/autoconf/2012-04/msg00061.html": "FSFULLRWD",
"lists.opensource.org/pipermail/license-discuss_lists.opensource.org/2002-November/006304.html": "HPND",
"mariadb.com/bsl11/": "BUSL-1.1",
@@ -1965,99 +2008,105 @@ var urlToLicense = map[string]string{
"opensource.linux-mirror.org/licenses/afl-1.1.txt": "AFL-1.1",
"opensource.linux-mirror.org/licenses/afl-1.2.txt": "AFL-1.2",
"opensource.linux-mirror.org/licenses/afl-2.1.txt": "AFL-2.1",
"opensource.org/license/apache-2-0": "Apache-2.0",
"opensource.org/license/mit/": "MIT",
"opensource.org/license/olfl-1-3/": "OLFL-1.3",
"opensource.org/license/osc-license-1-0": "OSC-1.0",
"opensource.org/licenses/0BSD": "0BSD",
"opensource.org/licenses/AGPL-3.0": "AGPL-3.0-or-later",
"opensource.org/licenses/APL-1.0": "APL-1.0",
"opensource.org/licenses/Apache-1.1": "Apache-1.1",
"opensource.org/licenses/Apache-2.0": "Apache-2.0",
"opensource.org/licenses/Artistic-1.0": "Artistic-1.0-cl8",
"opensource.org/licenses/BSD-2-Clause": "BSD-2-Clause",
"opensource.org/licenses/BSD-3-Clause": "BSD-3-Clause",
"opensource.org/licenses/BSDplusPatent": "BSD-2-Clause-Patent",
"opensource.org/licenses/BSL-1.0": "BSL-1.0",
"opensource.org/licenses/CAL-1.0": "CAL-1.0-Combined-Work-Exception",
"opensource.org/licenses/CATOSL-1.1": "CATOSL-1.1",
"opensource.org/licenses/CNRI-Python": "CNRI-Python",
"opensource.org/licenses/CPAL-1.0": "CPAL-1.0",
"opensource.org/licenses/CPL-1.0": "CPL-1.0",
"opensource.org/licenses/CUA-OPL-1.0": "CUA-OPL-1.0",
"opensource.org/licenses/ECL-1.0": "ECL-1.0",
"opensource.org/licenses/ECL-2.0": "ECL-2.0",
"opensource.org/licenses/EFL-1.0": "EFL-1.0",
"opensource.org/licenses/EFL-2.0": "EFL-2.0",
"opensource.org/licenses/EPL-1.0": "EPL-1.0",
"opensource.org/licenses/EUDatagrid": "EUDatagrid",
"opensource.org/licenses/EUPL-1.1": "EUPL-1.1",
"opensource.org/licenses/EUPL-1.2": "EUPL-1.2",
"opensource.org/licenses/Entessa": "Entessa",
"opensource.org/licenses/Fair": "Fair",
"opensource.org/licenses/Frameworx-1.0": "Frameworx-1.0",
"opensource.org/licenses/GPL-2.0": "GPL-2.0-or-later",
"opensource.org/licenses/GPL-3.0": "GPL-3.0-or-later",
"opensource.org/licenses/HPND": "HPND",
"opensource.org/licenses/IPA": "IPA",
"opensource.org/licenses/IPL-1.0": "IPL-1.0",
"opensource.org/licenses/ISC": "ISC",
"opensource.org/licenses/Intel": "Intel",
"opensource.org/licenses/LGPL-2.1": "LGPL-2.1-or-later",
"opensource.org/licenses/LGPL-3.0": "LGPL-3.0-or-later",
"opensource.org/licenses/LPL-1.0": "LPL-1.0",
"opensource.org/licenses/LPL-1.02": "LPL-1.02",
"opensource.org/licenses/LPPL-1.3c": "LPPL-1.3c",
"opensource.org/licenses/LiLiQ-P-1.1": "LiLiQ-P-1.1",
"opensource.org/licenses/LiLiQ-R-1.1": "LiLiQ-R-1.1",
"opensource.org/licenses/LiLiQ-Rplus-1.1": "LiLiQ-Rplus-1.1",
"opensource.org/licenses/MIT": "MIT",
"opensource.org/licenses/MPL-1.0": "MPL-1.0",
"opensource.org/licenses/MPL-1.1": "MPL-1.1",
"opensource.org/licenses/MPL-2.0": "MPL-2.0-no-copyleft-exception",
"opensource.org/licenses/MS-PL": "MS-PL",
"opensource.org/licenses/MS-RL": "MS-RL",
"opensource.org/licenses/MirOS": "MirOS",
"opensource.org/licenses/Motosoto": "Motosoto",
"opensource.org/licenses/Multics": "Multics",
"opensource.org/licenses/NASA-1.3": "NASA-1.3",
"opensource.org/licenses/NCSA": "NCSA",
"opensource.org/licenses/NGPL": "NGPL",
"opensource.org/licenses/NOSL3.0": "NPOSL-3.0",
"opensource.org/licenses/NTP": "NTP",
"opensource.org/licenses/Naumen": "Naumen",
"opensource.org/licenses/OCLC-2.0": "OCLC-2.0",
"opensource.org/licenses/OFL-1.1": "OFL-1.1-no-RFN",
"opensource.org/licenses/OGTSL": "OGTSL",
"opensource.org/licenses/OPL-2.1": "OSET-PL-2.1",
"opensource.org/licenses/OSL-1.0": "OSL-1.0",
"opensource.org/licenses/OSL-2.1": "OSL-2.1",
"opensource.org/licenses/OSL-3.0": "OSL-3.0",
"opensource.org/licenses/PHP-3.0": "PHP-3.0",
"opensource.org/licenses/PostgreSQL": "PostgreSQL",
"opensource.org/licenses/Python-2.0": "Python-2.0",
"opensource.org/licenses/QPL-1.0": "QPL-1.0",
"opensource.org/licenses/RPL-1.1": "RPL-1.1",
"opensource.org/licenses/RPL-1.5": "RPL-1.5",
"opensource.org/licenses/RPSL-1.0": "RPSL-1.0",
"opensource.org/licenses/RSCPL": "RSCPL",
"opensource.org/licenses/SISSL": "SISSL",
"opensource.org/licenses/SPL-1.0": "SPL-1.0",
"opensource.org/licenses/SimPL-2.0": "SimPL-2.0",
"opensource.org/licenses/Sleepycat": "Sleepycat",
"opensource.org/licenses/UCL-1.0": "UCL-1.0",
"opensource.org/licenses/UPL": "UPL-1.0",
"opensource.org/licenses/VSL-1.0": "VSL-1.0",
"opensource.org/licenses/W3C": "W3C",
"opensource.org/licenses/Watcom-1.0": "Watcom-1.0",
"opensource.org/licenses/Xnet": "Xnet",
"opensource.org/licenses/ZPL-2.0": "ZPL-2.0",
"opensource.org/licenses/Zlib": "Zlib",
"opensource.org/licenses/afl-3.0": "AFL-3.0",
"opensource.org/licenses/artistic-license-2.0": "Artistic-2.0",
"opensource.org/licenses/attribution": "AAL",
"opensource.org/licenses/cddl1": "CDDL-1.0",
"opensource.org/licenses/nokia": "Nokia",
"opensource.org/license/0BSD": "0BSD",
"opensource.org/license/AAL": "AAL",
"opensource.org/license/AFL-3.0": "AFL-3.0",
"opensource.org/license/AGPL-3.0": "AGPL-3.0-or-later",
"opensource.org/license/AGPL-3.0-only": "AGPL-3.0-only",
"opensource.org/license/APL-1.0": "APL-1.0",
"opensource.org/license/Artistic-1.0": "Artistic-1.0-cl8",
"opensource.org/license/Artistic-2.0": "Artistic-2.0",
"opensource.org/license/BSD-2-Clause": "BSD-2-Clause",
"opensource.org/license/BSD-2-Clause-Patent": "BSD-2-Clause-Patent",
"opensource.org/license/BSD-3-Clause": "BSD-3-Clause",
"opensource.org/license/BSL-1.0": "BSL-1.0",
"opensource.org/license/CAL-1.0": "CAL-1.0-Combined-Work-Exception",
"opensource.org/license/CATOSL-1.1": "CATOSL-1.1",
"opensource.org/license/CDDL-1.0": "CDDL-1.0",
"opensource.org/license/CNRI-Python": "CNRI-Python",
"opensource.org/license/CNRI-Python-GPL-Compatible": "CNRI-Python-GPL-Compatible",
"opensource.org/license/CPAL-1.0": "CPAL-1.0",
"opensource.org/license/CPL-1.0": "CPL-1.0",
"opensource.org/license/CUA-OPL-1.0": "CUA-OPL-1.0",
"opensource.org/license/ECL-1.0": "ECL-1.0",
"opensource.org/license/ECL-2.0": "ECL-2.0",
"opensource.org/license/EFL-1.0": "EFL-1.0",
"opensource.org/license/EFL-2.0": "EFL-2.0",
"opensource.org/license/EPL-1.0": "EPL-1.0",
"opensource.org/license/EUDatagrid": "EUDatagrid",
"opensource.org/license/EUPL-1.1": "EUPL-1.1",
"opensource.org/license/EUPL-1.2": "EUPL-1.2",
"opensource.org/license/Entessa": "Entessa",
"opensource.org/license/Fair": "Fair",
"opensource.org/license/Frameworx-1.0": "Frameworx-1.0",
"opensource.org/license/GPL-2.0": "GPL-2.0-or-later",
"opensource.org/license/GPL-2.0-only": "GPL-2.0-only",
"opensource.org/license/GPL-3.0": "GPL-3.0-or-later",
"opensource.org/license/GPL-3.0-only": "GPL-3.0-only",
"opensource.org/license/HPND": "HPND",
"opensource.org/license/IPA": "IPA",
"opensource.org/license/IPL-1.0": "IPL-1.0",
"opensource.org/license/ISC": "ISC",
"opensource.org/license/Intel": "Intel",
"opensource.org/license/LGPL-2.1": "LGPL-2.1-or-later",
"opensource.org/license/LGPL-2.1-only": "LGPL-2.1-only",
"opensource.org/license/LGPL-3.0": "LGPL-3.0-or-later",
"opensource.org/license/LPL-1.0": "LPL-1.0",
"opensource.org/license/LPL-1.02": "LPL-1.02",
"opensource.org/license/LPPL-1.3c": "LPPL-1.3c",
"opensource.org/license/LiLiQ-P-1.1": "LiLiQ-P-1.1",
"opensource.org/license/LiLiQ-R-1.1": "LiLiQ-R-1.1",
"opensource.org/license/LiLiQ-Rplus-1.1": "LiLiQ-Rplus-1.1",
"opensource.org/license/MIT": "MIT",
"opensource.org/license/MPL-1.0": "MPL-1.0",
"opensource.org/license/MPL-1.1": "MPL-1.1",
"opensource.org/license/MPL-2.0": "MPL-2.0-no-copyleft-exception",
"opensource.org/license/MS-PL": "MS-PL",
"opensource.org/license/MS-RL": "MS-RL",
"opensource.org/license/MirOS": "MirOS",
"opensource.org/license/Motosoto": "Motosoto",
"opensource.org/license/Multics": "Multics",
"opensource.org/license/NASA-1.3": "NASA-1.3",
"opensource.org/license/NCSA": "NCSA",
"opensource.org/license/NGPL": "NGPL",
"opensource.org/license/NPOSL-3.0": "NPOSL-3.0",
"opensource.org/license/NTP": "NTP",
"opensource.org/license/Naumen": "Naumen",
"opensource.org/license/Nokia": "Nokia",
"opensource.org/license/OCLC-2.0": "OCLC-2.0",
"opensource.org/license/OFL-1.1": "OFL-1.1-no-RFN",
"opensource.org/license/OGTSL": "OGTSL",
"opensource.org/license/OLFL-1.3": "OLFL-1.3",
"opensource.org/license/OSC-1.0": "OSC-1.0",
"opensource.org/license/OSET-PL-2.1": "OSET-PL-2.1",
"opensource.org/license/OSL-1.0": "OSL-1.0",
"opensource.org/license/OSL-2.1": "OSL-2.1",
"opensource.org/license/OSL-3.0": "OSL-3.0",
"opensource.org/license/PHP-3.0": "PHP-3.0",
"opensource.org/license/PSF-2-0": "PSF-2.0",
"opensource.org/license/PostgreSQL": "PostgreSQL",
"opensource.org/license/Python-2.0": "Python-2.0",
"opensource.org/license/Python-2.0.1": "Python-2.0.1",
"opensource.org/license/QPL-1.0": "QPL-1.0",
"opensource.org/license/RPL-1.1": "RPL-1.1",
"opensource.org/license/RPL-1.5": "RPL-1.5",
"opensource.org/license/RPSL-1.0": "RPSL-1.0",
"opensource.org/license/RSCPL": "RSCPL",
"opensource.org/license/SISSL": "SISSL",
"opensource.org/license/SPL-1.0": "SPL-1.0",
"opensource.org/license/SiMPL-2.0": "SimPL-2.0",
"opensource.org/license/Sleepycat": "Sleepycat",
"opensource.org/license/UCL-1.0": "UCL-1.0",
"opensource.org/license/UPL-1.0": "UPL-1.0",
"opensource.org/license/VSL-1.0": "VSL-1.0",
"opensource.org/license/W3C": "W3C",
"opensource.org/license/Watcom-1.0": "Watcom-1.0",
"opensource.org/license/Xnet": "Xnet",
"opensource.org/license/ZPL-2.0": "ZPL-2.0",
"opensource.org/license/Zlib": "Zlib",
"opensource.org/license/apache-1.1": "Apache-1.1",
"opensource.org/license/apache-2.0": "Apache-2.0",
"opensource.org/license/curl": "curl",
"oss.sgi.com/projects/FreeB/": "SGI-B-1.1",
"oss.sgi.com/projects/FreeB/SGIFreeSWLicB.1.0.html": "SGI-B-1.0",
"oss.sgi.com/projects/FreeB/SGIFreeSWLicB.2.0.pdf": "SGI-B-2.0",
@@ -2082,37 +2131,40 @@ var urlToLicense = map[string]string{
"raw.githubusercontent.com/NTIA/scos-sensor/refs/heads/master/LICENSE.md": "NTIA-PD",
"raw.githubusercontent.com/OpenMDW/OpenMDW/refs/heads/main/1.0/LICENSE.openmdw": "OpenMDW-1.0",
"raw.githubusercontent.com/blackhole89/notekit/master/data/fonts/Charter%20license.txt": "Bitstream-Charter",
"rc.quest.com/topics/openssh/license.php#tcpwrappers": "TCP-wrappers",
"research.cs.wisc.edu/condor/license.html#condor": "Condor-1.1",
"research.scea.com/scea_shared_source_license.html": "SCEA",
"romanrm.net/mit-zero": "MIT-0",
"sam.zoy.org/wtfpl/COPYING": "WTFPL",
"scripts.sil.org/cms/scripts/page.php?item_id=OFL10_web": "OFL-1.0-no-RFN",
"scripts.sil.org/cms/scripts/page.php?item_id=OFL_web": "OFL-1.1-no-RFN",
"slicer.org/LICENSE": "3D-Slicer-1.0",
"solderpad.org/licenses/SHL-0.5/": "SHL-0.5",
"solderpad.org/licenses/SHL-0.51/": "SHL-0.51",
"source.icu-project.org/repos/icu/icu/trunk/license.html": "ICU",
"sourceforge.net/p/buddy/gitcode/ci/master/tree/README": "Buddy",
"sourceforge.net/p/ctrio/git/ci/master/tree/README": "ISC-Veillard",
"sourceforge.net/p/mcj/xfig/ci/master/tree/src/Makefile.am": "Xfig",
"sourceforge.net/p/netpbm/code/HEAD/tree/super_stable/converter/other/pnmtorle.c": "URT-RLE",
"sourceforge.net/p/netpbm/code/HEAD/tree/super_stable/converter/other/rletopnm.c": "URT-RLE",
"sourceforge.net/p/netpbm/code/HEAD/tree/super_stable/converter/ppm/ppmtompeg/jrevdct.c#l1189": "MPEG-SSG",
"sourceforge.net/p/netpbm/code/HEAD/tree/super_stable/editor/pnmstitch.c#l2": "pnmstitch",
"sourceforge.net/p/netpbm/code/HEAD/tree/super_stable/netpbm.c#l8": "HPND-Pbmplus",
"sourceforge.net/p/sbcl/sbcl/ci/master/tree/src/code/loop.lisp": "LOOP",
"sourceforge.net/p/trustedqsl/tqsl/ci/master/tree/LICENSE.txt": "TrustedQSL",
"sourceforge.net/p/xmedcon/code/ci/master/tree/libs/dicom/README": "OFFIS",
"sourceforge.net/p/xmedcon/code/ci/master/tree/libs/ljpg/": "IJG-short",
"sourceforge.net/projects/sevenzip/files/LZMA%20SDK/": "LZMA-SDK-9.22",
"sources.debian.org/copyright/license/debianutils/4.11.2/": "SMAIL-GPL",
"sources.debian.org/src/openmpi/4.1.0-10/ompi/debuggers/msgq_interface.h/?hl=19#L19": "mpi-permissive",
"sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/licenses/BitTorrent?r1=1.1&r2=1.1.1.1&diff_format=s": "BitTorrent-1.0",
"sourceware.org/bzip2/1.0.5/bzip2-manual-1.0.5.html": "bzip2-1.0.6",
"sourceware.org/cgit/binutils-gdb/tree/include/coff/sym.h#n11": "MIPS",
"sourceware.org/cgit/binutils-gdb/tree/include/coff/symconst.h#n11": "ThirdEye",
"sourceware.org/cgit/valgrind/tree/mpi/libmpiwrap.c": "bzip2-1.0.6",
"raw.githubusercontent.com/landtuna/open-mtools/6c04c2882eca7c4313ec267766ea53ac9bfe129b/mdump.c": "Informatica",
"raw.githubusercontent.com/landtuna/open-mtools/6c04c2882eca7c4313ec267766ea53ac9bfe129b/msend.c": "Informatica",
"rc.quest.com/topics/openssh/license.php#tcpwrappers": "TCP-wrappers",
"research.cs.wisc.edu/condor/license.html#condor": "Condor-1.1",
"research.scea.com/scea_shared_source_license.html": "SCEA",
"romanrm.net/mit-zero": "MIT-0",
"sam.zoy.org/wtfpl/COPYING": "WTFPL",
"scancode-licensedb.aboutcode.org/bsd-unchanged": "BSD-2-Clause-pos-unchanged",
"scripts.sil.org/cms/scripts/page.php?item_id=OFL10_web": "OFL-1.0-no-RFN",
"scripts.sil.org/cms/scripts/page.php?item_id=OFL_web": "OFL-1.1-no-RFN",
"slicer.org/LICENSE": "3D-Slicer-1.0",
"solderpad.org/licenses/SHL-0.5/": "SHL-0.5",
"solderpad.org/licenses/SHL-0.51/": "SHL-0.51",
"source.icu-project.org/repos/icu/icu/trunk/license.html": "ICU",
"sourceforge.net/p/buddy/gitcode/ci/master/tree/README": "Buddy",
"sourceforge.net/p/ctrio/git/ci/master/tree/README": "ISC-Veillard",
"sourceforge.net/p/mcj/xfig/ci/master/tree/src/Makefile.am": "Xfig",
"sourceforge.net/p/netpbm/code/HEAD/tree/super_stable/converter/other/pnmtorle.c": "URT-RLE",
"sourceforge.net/p/netpbm/code/HEAD/tree/super_stable/converter/other/rletopnm.c": "URT-RLE",
"sourceforge.net/p/netpbm/code/HEAD/tree/super_stable/converter/ppm/ppmtompeg/jrevdct.c#l1189": "MPEG-SSG",
"sourceforge.net/p/netpbm/code/HEAD/tree/super_stable/editor/pnmstitch.c#l2": "pnmstitch",
"sourceforge.net/p/netpbm/code/HEAD/tree/super_stable/netpbm.c#l8": "HPND-Pbmplus",
"sourceforge.net/p/sbcl/sbcl/ci/master/tree/src/code/loop.lisp": "LOOP",
"sourceforge.net/p/trustedqsl/tqsl/ci/master/tree/LICENSE.txt": "TrustedQSL",
"sourceforge.net/p/xmedcon/code/ci/master/tree/libs/dicom/README": "OFFIS",
"sourceforge.net/p/xmedcon/code/ci/master/tree/libs/ljpg/": "IJG-short",
"sourceforge.net/projects/sevenzip/files/LZMA%20SDK/": "LZMA-SDK-9.22",
"sources.debian.org/copyright/license/debianutils/4.11.2/": "SMAIL-GPL",
"sources.debian.org/src/openmpi/4.1.0-10/ompi/debuggers/msgq_interface.h/?hl=19#L19": "mpi-permissive",
"sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/licenses/BitTorrent?r1=1.1&r2=1.1.1.1&diff_format=s": "BitTorrent-1.0",
"sourceware.org/bzip2/1.0.5/bzip2-manual-1.0.5.html": "bzip2-1.0.6",
"sourceware.org/cgit/binutils-gdb/tree/include/coff/sym.h#n11": "MIPS",
"sourceware.org/cgit/binutils-gdb/tree/include/coff/symconst.h#n11": "ThirdEye",
"sourceware.org/cgit/valgrind/tree/mpi/libmpiwrap.c": "bzip2-1.0.6",
"sourceware.org/git/?p=binutils-gdb.git;a=blob;f=libiberty/strcasecmp.c;h=131d81c2ce7881fa48c363dc5bf5fb302c61ce0b;hb=HEAD": "BSD-4.3RENO",
"sourceware.org/git/?p=binutils-gdb.git;a=blob;f=readline/readline/support/wcwidth.c;h=0f5ec995796f4813abbcf4972aec0378ab74722a;hb=HEAD#l55": "HPND-Markus-Kuhn",
"sourceware.org/git/?p=bzip2.git;a=blob;f=LICENSE;hb=bzip2-1.0.6": "bzip2-1.0.6",
@@ -2129,6 +2181,7 @@ var urlToLicense = map[string]string{
"tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/COPYING.TGPPL.rst": "TGPPL-1.0",
"ti.arc.nasa.gov/opensource/nosa/": "NASA-1.3",
"trac.enlightenment.org/e/browser/trunk/imlib2/COPYING": "Imlib2",
"tunes.org/legalese/bugroff.html": "Bugroff",
"ubuntu.com/legal/font-licence": "Ubuntu-font-1.0",
"unlicense.org/": "Unlicense",
"vimdoc.sourceforge.net/htmldoc/uganda.html": "Vim",
@@ -2139,6 +2192,7 @@ var urlToLicense = map[string]string{
"web.archive.org/web/20041020171434/http://www.rosenlaw.com/osl2.0.html": "OSL-2.0",
"web.archive.org/web/20050212003940/http://www.rosenlaw.com/osl21.htm": "OSL-2.1",
"web.archive.org/web/20060319014854/http://info.borland.com/devsupport/interbase/opensource/IPL.html": "Interbase-1.0",
"web.archive.org/web/20061013013632/http://opensource.linux-mirror.org/licenses/afl-2.1.txt": "AFL-2.1",
"web.archive.org/web/20080207013128/http://www.gnome.org/fonts/": "Bitstream-Vera",
"web.archive.org/web/20100302225219/http://www.zimbra.com/license/zimbra-public-license-1-3.html": "Zimbra-1.3",
"web.archive.org/web/20111123062036/http://research.cs.wisc.edu/condor/license.html#condor": "Condor-1.1",
@@ -2243,11 +2297,11 @@ var urlToLicense = map[string]string{
"www.jython.org/license.html": "CNRI-Jython",
"www.kermitproject.org/ck90.html#source": "HPND-export-US",
"www.kernel.org/doc/man-pages/licenses.html": "Linux-man-pages-copyleft",
"www.latex-project.org/lppl/LPPL-1.3c.txt": "LPPL-1.3c",
"www.latex-project.org/lppl/lppl-1-0.txt": "LPPL-1.0",
"www.latex-project.org/lppl/lppl-1-1.txt": "LPPL-1.1",
"www.latex-project.org/lppl/lppl-1-2.txt": "LPPL-1.2",
"www.latex-project.org/lppl/lppl-1-3a.txt": "LPPL-1.3a",
"www.latex-project.org/lppl/lppl-1-3c.txt": "LPPL-1.3c",
"www.lemis.com/grog/UNIX/ancient-source-all.pdf": "Caldera",
"www.libpng.org/pub/png/src/libpng-LICENSE.txt": "libpng-2.0",
"www.mail-archive.com/debian-bugs-rc@lists.debian.org/msg11494.html": "Symlinks",
@@ -2269,16 +2323,17 @@ var urlToLicense = map[string]string{
"www.netlib.org/lapack/LICENSE.txt": "BSD-3-Clause-Open-MPI",
"www.netlib.org/minpack/disclaimer": "Minpack",
"www.networkcultures.org/_uploads/%233notebook_telekommunist.pdf": "PPL",
"www.oclc.org/research/activities/software/license/v2final.htm": "OCLC-2.0",
"www.ogc.org/ogc/software/1.0": "OGC-1.0",
"www.ohwr.org/project/cernohl/wikis/Documents/CERN-OHL-version-2": "CERN-OHL-W-2.0",
"www.ohwr.org/project/licenses/wikis/cern-ohl-v1.1": "CERN-OHL-1.1",
"www.ohwr.org/project/licenses/wikis/cern-ohl-v1.2": "CERN-OHL-1.2",
"www.open-mpi.org/community/license.php": "BSD-3-Clause-Open-MPI",
"www.opencascade.com/content/occt-public-license": "OCCT-PL",
"www.opendatacommons.org/licenses/odbl/1.0/": "ODbL-1.0",
"www.opengroup.org/testing/downloads/The_Open_Group_TSL.txt": "OGTSL",
"www.openhub.net/licenses/mslpl": "MS-LPL",
"www.nist.gov/open/copyright-fair-use-and-licensing-statements-srd-data-software-and-technical-series-publications": "NIST-Software",
"www.oclc.org/research/activities/software/license/v2final.htm": "OCLC-2.0",
"www.ogc.org/ogc/software/1.0": "OGC-1.0",
"www.ohwr.org/project/cernohl/wikis/Documents/CERN-OHL-version-2": "CERN-OHL-W-2.0",
"www.ohwr.org/project/licenses/wikis/cern-ohl-v1.1": "CERN-OHL-1.1",
"www.ohwr.org/project/licenses/wikis/cern-ohl-v1.2": "CERN-OHL-1.2",
"www.open-mpi.org/community/license.php": "BSD-3-Clause-Open-MPI",
"www.opencascade.com/content/occt-public-license": "OCCT-PL",
"www.opendatacommons.org/licenses/odbl/1.0/": "ODbL-1.0",
"www.opengroup.org/testing/downloads/The_Open_Group_TSL.txt": "OGTSL",
"www.openhub.net/licenses/mslpl": "MS-LPL",
"www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=LICENSE;hb=1cae062821881f41b73012ba816434897abf4205": "OLDAP-2.6",
"www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=LICENSE;hb=37b4b3f6cc4bf34e1d3dec61e69914b9819d8894": "NBPL-1.0",
"www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=LICENSE;hb=42b0383c50c299977b5893ee695cf4e486fb0dc7": "OLDAP-1.2",
@@ -2301,7 +2356,7 @@ var urlToLicense = map[string]string{
"www.opensource.apple.com/license/apsl/": "APSL-2.0",
"www.opensource.apple.com/source/IOSerialFamily/IOSerialFamily-7/APPLE_LICENSE": "APSL-1.1",
"www.opensource.apple.com/source/tcl/tcl-5/tcl/generic/regfronts.c": "Spencer-99",
"www.opensource.org/licenses/EPL-2.0": "EPL-2.0",
"www.opensource.org/license/EPL-2.0": "EPL-2.0",
"www.openssl.org/source/license.html": "OpenSSL",
"www.osetfoundation.org/public-license": "OSET-PL-2.1",
"www.parliament.uk/site-information/copyright-parliament/open-parliament-licence/": "OPL-UK-3.0",
+90
View File
@@ -123,6 +123,96 @@ func TestLicenseByURL_AlternateScheme(t *testing.T) {
}
}
func TestLicenseByURL_OpenSourceOrgVariants(t *testing.T) {
// SPDX has changed the form of opensource.org URLs across list versions (e.g. 3.29.0 moved
// opensource.org/licenses/MIT to opensource.org/license/MIT); every form should keep resolving.
tests := []struct {
name string
url string
wantID string
}{
{name: "old plural path", url: "http://opensource.org/licenses/MIT", wantID: "MIT"},
{name: "new singular path", url: "https://opensource.org/license/MIT", wantID: "MIT"},
{name: "lowercase with trailing slash", url: "https://opensource.org/license/mit/", wantID: "MIT"},
{name: "plural path with trailing slash", url: "https://opensource.org/licenses/MIT/", wantID: "MIT"},
{name: "www prefix", url: "https://www.opensource.org/licenses/MIT", wantID: "MIT"},
{name: "uppercase host", url: "https://WWW.OpenSource.org/licenses/MIT", wantID: "MIT"},
{name: "old plural path BSD-3-Clause", url: "https://opensource.org/licenses/BSD-3-Clause", wantID: "BSD-3-Clause"},
{name: "new singular path BSD-3-Clause", url: "https://opensource.org/license/BSD-3-Clause", wantID: "BSD-3-Clause"},
{name: "lowercase BSD-3-Clause", url: "https://opensource.org/licenses/bsd-3-clause", wantID: "BSD-3-Clause"},
{name: "old plural path Apache-2.0", url: "http://opensource.org/licenses/Apache-2.0", wantID: "Apache-2.0"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
info, found := LicenseByURL(tt.url)
if !found {
t.Fatalf("LicenseByURL(%q) found = false, want true", tt.url)
}
if info.ID != tt.wantID {
t.Errorf("LicenseByURL(%q) ID = %v, want %v", tt.url, info.ID, tt.wantID)
}
})
}
}
func TestLicenseByURL_NonOpenSourceOrgStaysExact(t *testing.T) {
// the relaxed matching is scoped to opensource.org; other hosts may have case-sensitive paths
if _, found := LicenseByURL("https://www.apache.org/licenses/license-2.0"); found {
t.Errorf("LicenseByURL() found = true for case-altered non-opensource.org URL, want false")
}
if _, found := LicenseByURL("https://opensource.org/licenses/not-a-real-license"); found {
t.Errorf("LicenseByURL() found = true for unknown opensource.org URL, want false")
}
}
func TestNormalizeOpenSourceOrgURL(t *testing.T) {
tests := []struct {
url string
want string
wantOK bool
}{
{url: "opensource.org/licenses/MIT", want: "opensource.org/license/mit", wantOK: true},
{url: "opensource.org/license/mit/", want: "opensource.org/license/mit", wantOK: true},
{url: "www.opensource.org/licenses/EPL-2.0", want: "opensource.org/license/epl-2.0", wantOK: true},
{url: "lists.opensource.org/pipermail/license-discuss", wantOK: false},
{url: "www.apache.org/licenses/LICENSE-2.0", wantOK: false},
}
for _, tt := range tests {
t.Run(tt.url, func(t *testing.T) {
got, ok := normalizeOpenSourceOrgURL(tt.url)
if ok != tt.wantOK {
t.Fatalf("normalizeOpenSourceOrgURL(%q) ok = %v, want %v", tt.url, ok, tt.wantOK)
}
if got != tt.want {
t.Errorf("normalizeOpenSourceOrgURL(%q) = %q, want %q", tt.url, got, tt.want)
}
})
}
}
func TestBuildOpenSourceOrgURLToLicense(t *testing.T) {
index := buildOpenSourceOrgURLToLicense(map[string]string{
"opensource.org/licenses/MIT": "MIT",
"opensource.org/license/mit/": "MIT",
"opensource.org/licenses/Foo": "Foo-1.0",
"opensource.org/license/foo": "Foo-2.0",
"opensource.org/license/FOO/": "Foo-1.0",
"www.apache.org/licenses/Bar": "Bar",
})
if got := index["opensource.org/license/mit"]; got != "MIT" {
t.Errorf("index[mit] = %q, want MIT", got)
}
if got, exists := index["opensource.org/license/foo"]; exists {
t.Errorf("index[foo] = %q, want no entry for URL forms mapping to different licenses", got)
}
if len(index) != 1 {
t.Errorf("len(index) = %d, want 1 (non-opensource.org URLs excluded): %v", len(index), index)
}
}
func TestStripScheme(t *testing.T) {
tests := []struct {
name string
@@ -2,6 +2,7 @@
package spdxhelpers
import (
"cmp"
"crypto/sha1"
"fmt"
"path"
@@ -179,7 +180,8 @@ func toRootRelationships(rootPackage *spdx.Package, packages []*spdx.Package) (o
func toRootPackage(s source.Description) *spdx.Package {
var prefix string
name := s.Name
// the SPDX package name is mandatory, and a source decoded from another SBOM may not carry one
name := cmp.Or(s.Name, imageRepoName(s.Metadata), s.ID, helpers.DocumentName(s))
version := s.Version
var purl *packageurl.PackageURL
@@ -210,7 +212,7 @@ func toRootPackage(s source.Description) *spdx.Package {
checksums = append(checksums, *c)
purl = &packageurl.PackageURL{
Type: "oci",
Name: s.Name,
Name: name,
Version: m.ManifestDigest,
Qualifiers: qualifiers,
}
@@ -240,7 +242,7 @@ func toRootPackage(s source.Description) *spdx.Package {
checksums = append(checksums, *c)
purl = &packageurl.PackageURL{
Type: "oci",
Name: s.Name,
Name: name,
Version: m.ManifestDigest,
Qualifiers: qualifiers,
}
@@ -275,10 +277,6 @@ func toRootPackage(s source.Description) *spdx.Package {
default:
prefix = prefixUnknown
purpose = spdxPrimaryPurposeOther
if name == "" {
name = s.ID
}
}
p := &spdx.Package{
@@ -308,6 +306,25 @@ func toRootPackage(s source.Description) *spdx.Package {
return p
}
// imageRepoName returns the repository name from an image or model user input, the same way the image source names
// itself, so a source decoded from another SBOM (which may only carry the user input) gets a name without the tag.
func imageRepoName(metadata any) string {
var userInput string
switch m := metadata.(type) {
case source.ImageMetadata:
userInput = m.UserInput
case source.OCIModelMetadata:
userInput = m.UserInput
default:
return ""
}
ref, _ := reference.Parse(userInput)
if named, ok := ref.(reference.Named); ok {
return named.Name()
}
return ""
}
func toSPDXID(identifiable artifact.Identifiable) spdx.ElementID {
id := string(identifiable.ID())
if after, ok := strings.CutPrefix(id, "SPDXRef-"); ok {
@@ -581,10 +598,13 @@ func toPackageChecksums(p pkg.Package) ([]spdx.Checksum, bool) {
})
case pkg.OpamPackage:
for _, checksum := range meta.Checksums {
parts := strings.Split(checksum, "=")
algorithm, value, ok := strings.Cut(checksum, "=")
if !ok {
continue
}
checksums = append(checksums, spdx.Checksum{
Algorithm: spdx.ChecksumAlgorithm(strings.ToUpper(parts[0])),
Value: parts[1],
Algorithm: spdx.ChecksumAlgorithm(strings.ToUpper(algorithm)),
Value: value,
})
}
}
@@ -516,6 +516,20 @@ func Test_toPackageChecksums(t *testing.T) {
},
filesAnalyzed: false,
},
{
name: "Opam Package with a checksum missing its algorithm separator",
pkg: pkg.Package{
Name: "test",
Version: "1.0.0",
Metadata: pkg.OpamPackage{
Checksums: []string{"f5f1c0b4", "md5=abc"},
},
},
expected: []spdx.Checksum{
{Algorithm: "MD5", Value: "abc"},
},
filesAnalyzed: false,
},
{
name: "Package with no metadata type",
pkg: pkg.Package{
@@ -1131,3 +1145,49 @@ func Test_otherLicenses(t *testing.T) {
})
}
}
func Test_toRootPackage_name(t *testing.T) {
tests := []struct {
name string
src source.Description
wantName string
wantPurl string
}{
{
name: "source name wins",
src: source.Description{Name: "named", ID: "id", Metadata: source.ImageMetadata{UserInput: "alpine:3.18"}},
wantName: "named",
},
{
// a CycloneDX container component decodes to an image source with only the user input set
name: "image user input without a source name",
src: source.Description{Metadata: source.ImageMetadata{
UserInput: "alpine:3.18",
ManifestDigest: "sha256:0e4b2e0d6b2f3c2b1d3bb3a1c8f1c4d9e6a9b1e8c6f4d2b3a1f0e9d8c7b6a5f4",
}},
wantName: "alpine",
wantPurl: "pkg:oci/alpine@sha256%3A0e4b2e0d6b2f3c2b1d3bb3a1c8f1c4d9e6a9b1e8c6f4d2b3a1f0e9d8c7b6a5f4?arch=&tag=3.18",
},
{
name: "id without a source name",
src: source.Description{ID: "some-id"},
wantName: "some-id",
},
{
name: "nothing to name it by",
src: source.Description{Metadata: source.DirectoryMetadata{}},
wantName: "unknown",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
p := toRootPackage(tt.src)
assert.Equal(t, tt.wantName, p.PackageName)
var purl string
if len(p.PackageExternalReferences) > 0 {
purl = p.PackageExternalReferences[0].Locator
}
assert.Equal(t, tt.wantPurl, purl)
})
}
}
@@ -147,12 +147,24 @@ func extractSource(spdxIDMap map[string]any, doc *spdx.Document) source.Descript
}
p := rootPackages[0]
rootType := syftRootType(p.PackageSPDXIdentifier)
switch p.PrimaryPackagePurpose {
purpose := p.PrimaryPackagePurpose
if purpose == "" {
// SPDX 2.1 and 2.2 have no primary package purpose, so fall back to the source type syft puts in the root's ID
purpose = syftRootPurpose(rootType)
}
switch purpose {
case spdxPrimaryPurposeContainer:
src = containerSource(p)
case spdxPrimaryPurposeFile:
src = fileSource(p)
case spdxPrimaryPurposeOther:
// syft writes a root package even for a source it knows nothing about, which is not a package in the SBOM
if rootType != prefixUnknown {
return src
}
default:
return src
}
@@ -165,6 +177,30 @@ func extractSource(spdxIDMap map[string]any, doc *spdx.Document) source.Descript
return src
}
// syftRootType returns the source type syft encodes in a root package ID ("DocumentRoot-<type>-<name>"), or "" for
// any other ID.
func syftRootType(id spdx.ElementID) string {
rest, ok := strings.CutPrefix(string(id), "DocumentRoot-")
if !ok {
return ""
}
rootType, _, _ := strings.Cut(rest, "-")
return rootType
}
// syftRootPurpose returns the primary package purpose syft writes for a root package of the given source type.
func syftRootPurpose(rootType string) string {
switch rootType {
case prefixImage, prefixOCIModel, prefixSnap:
return spdxPrimaryPurposeContainer
case prefixDirectory, prefixFile:
return spdxPrimaryPurposeFile
case prefixUnknown:
return spdxPrimaryPurposeOther
}
return ""
}
func containerSource(p *spdx.Package) source.Description {
id := string(p.PackageSPDXIdentifier)
+5 -1
View File
@@ -7,6 +7,7 @@ import (
"io"
"strings"
syftinternal "github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/cpe"
"github.com/anchore/syft/syft/format/internal"
@@ -61,7 +62,7 @@ func toSyftModel(r io.Reader) (*sbom.SBOM, error) {
var errs []error
pkgs := pkg.NewCollection()
scanner := bufio.NewScanner(r)
scanner := syftinternal.NewLineScanner(r)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
@@ -86,6 +87,9 @@ func toSyftModel(r io.Reader) (*sbom.SBOM, error) {
p.SetID()
pkgs.Add(p)
}
if err := scanner.Err(); err != nil {
errs = append(errs, err)
}
return &sbom.SBOM{
Artifacts: sbom.Artifacts{
+39
View File
@@ -114,6 +114,45 @@ func TestCycloneDxImageEncoder(t *testing.T) {
)
}
func TestCycloneDxSourcePackageEncoder(t *testing.T) {
dir := t.TempDir()
testutil.AssertEncoderAgainstGoldenSnapshot(t,
testutil.EncoderSnapshotTestConfig{
Subject: testutil.SourcePackagesInput(t, dir),
Format: getEncoder(t),
UpdateSnapshot: *updateSnapshot,
PersistRedactionsInSnapshot: true,
IsJSON: true,
Redactor: redactor(dir),
},
)
}
func TestCycloneDxSourcePackageEncoder_SpecVersions(t *testing.T) {
s := testutil.SourcePackagesInput(t, t.TempDir())
tests := []struct {
version string
want bool
}{
{version: "1.4", want: false},
{version: "1.5", want: false},
{version: "1.6", want: true},
{version: cyclonedxutil.DefaultVersion, want: true},
}
for _, tt := range tests {
t.Run(tt.version, func(t *testing.T) {
enc, err := NewFormatEncoderWithConfig(EncoderConfig{Version: tt.version})
require.NoError(t, err)
var buf bytes.Buffer
require.NoError(t, enc.Encode(&buf, s))
// the source-distribution reference type only exists from spec 1.6
assert.Equal(t, tt.want, strings.Contains(buf.String(), `"source-distribution"`))
})
}
}
func redactor(values ...string) testutil.Redactor {
return testutil.NewRedactions().
WithValuesRedacted(values...).
@@ -0,0 +1,275 @@
{
"$schema": "http://cyclonedx.org/schema/bom-1.7.schema.json",
"bomFormat": "CycloneDX",
"specVersion": "1.7",
"serialNumber": "urn:uuid:redacted",
"version": 1,
"metadata": {
"timestamp": "timestamp:redacted",
"tools": {
"components": [
{
"type": "application",
"author": "anchore",
"name": "syft",
"version": "v0.42.0-bogus"
}
]
},
"component": {
"bom-ref": "redacted",
"type": "file",
"name": "some/path"
}
},
"components": [
{
"bom-ref": "pkg:deb/debian/attr@1%3A2.4.47-2%2Bb1?arch=amd64&distro=debian-12&package-id=5cc0e8979d7f5115&upstream=attr%401%3A2.4.47-2",
"type": "library",
"name": "attr",
"version": "1:2.4.47-2+b1",
"purl": "pkg:deb/debian/attr@1%3A2.4.47-2%2Bb1?arch=amd64&distro=debian-12&upstream=attr%401%3A2.4.47-2",
"externalReferences": [
{
"url": "pkg:deb/debian/attr@1%3A2.4.47-2?arch=source&distro=debian-12",
"type": "source-distribution"
}
],
"properties": [
{
"name": "syft:package:foundBy",
"value": "dpkg-db-cataloger"
},
{
"name": "syft:package:type",
"value": "deb"
},
{
"name": "syft:package:metadataType",
"value": "dpkg-db-entry"
},
{
"name": "syft:location:0:path",
"value": "/var/lib/dpkg/status"
},
{
"name": "syft:metadata:installedSize",
"value": "0"
},
{
"name": "syft:metadata:source",
"value": "attr"
},
{
"name": "syft:metadata:sourceVersion",
"value": "1:2.4.47-2"
}
]
},
{
"bom-ref": "pkg:rpm/centos/bind-export-libs@9.11.13-3.el8?arch=x86_64&distro=centos-8&epoch=32&package-id=6008fca05ca7b418&upstream=bind-9.11.13-3.el8.src.rpm",
"type": "library",
"name": "bind-export-libs",
"version": "32:9.11.13-3.el8",
"purl": "pkg:rpm/centos/bind-export-libs@9.11.13-3.el8?arch=x86_64&distro=centos-8&epoch=32&upstream=bind-9.11.13-3.el8.src.rpm",
"externalReferences": [
{
"url": "pkg:rpm/centos/bind@9.11.13-3.el8?arch=src&distro=centos-8&epoch=32",
"type": "source-distribution"
}
],
"properties": [
{
"name": "syft:package:foundBy",
"value": "rpm-db-cataloger"
},
{
"name": "syft:package:type",
"value": "rpm"
},
{
"name": "syft:package:metadataType",
"value": "rpm-db-entry"
},
{
"name": "syft:location:0:path",
"value": "/var/lib/rpm/Packages"
},
{
"name": "syft:metadata:epoch",
"value": "32"
},
{
"name": "syft:metadata:release",
"value": "3.el8"
},
{
"name": "syft:metadata:size",
"value": "0"
},
{
"name": "syft:metadata:sourceRpm",
"value": "bind-9.11.13-3.el8.src.rpm"
}
]
},
{
"bom-ref": "pkg:apk/alpine/busybox@1.35.0-r17?arch=x86_64&distro=alpine-3.16.3&package-id=aa74af1190de1c40",
"type": "library",
"name": "busybox",
"version": "1.35.0-r17",
"purl": "pkg:apk/alpine/busybox@1.35.0-r17?arch=x86_64&distro=alpine-3.16.3",
"properties": [
{
"name": "syft:package:foundBy",
"value": "apk-db-cataloger"
},
{
"name": "syft:package:type",
"value": "apk"
},
{
"name": "syft:package:metadataType",
"value": "apk-db-entry"
},
{
"name": "syft:location:0:path",
"value": "/lib/apk/db/installed"
},
{
"name": "syft:metadata:installedSize",
"value": "0"
},
{
"name": "syft:metadata:originPackage",
"value": "busybox"
},
{
"name": "syft:metadata:size",
"value": "0"
}
]
},
{
"bom-ref": "pkg:alpm/arch/gcc-libs@13.2.1-3?arch=x86_64&distro=arch-rolling&package-id=58a2fcf749a9a3cc&upstream=gcc",
"type": "library",
"name": "gcc-libs",
"version": "13.2.1-3",
"purl": "pkg:alpm/arch/gcc-libs@13.2.1-3?arch=x86_64&distro=arch-rolling&upstream=gcc",
"properties": [
{
"name": "syft:package:foundBy",
"value": "alpm-db-cataloger"
},
{
"name": "syft:package:type",
"value": "alpm"
},
{
"name": "syft:package:metadataType",
"value": "alpm-db-entry"
},
{
"name": "syft:location:0:path",
"value": "/var/lib/pacman/local/gcc-libs-13.2.1-3/desc"
},
{
"name": "syft:metadata:architecture",
"value": "x86_64"
},
{
"name": "syft:metadata:basepackage",
"value": "gcc"
},
{
"name": "syft:metadata:package",
"value": "gcc-libs"
},
{
"name": "syft:metadata:size",
"value": "0"
},
{
"name": "syft:metadata:version",
"value": "13.2.1-3"
}
]
},
{
"bom-ref": "pkg:apk/alpine/libc-utils@0.7.2-r3?arch=x86_64&distro=alpine-3.16.3&package-id=16a8c7b19fde6f4f&upstream=libc-dev",
"type": "library",
"name": "libc-utils",
"version": "0.7.2-r3",
"purl": "pkg:apk/alpine/libc-utils@0.7.2-r3?arch=x86_64&distro=alpine-3.16.3&upstream=libc-dev",
"properties": [
{
"name": "syft:package:foundBy",
"value": "apk-db-cataloger"
},
{
"name": "syft:package:type",
"value": "apk"
},
{
"name": "syft:package:metadataType",
"value": "apk-db-entry"
},
{
"name": "syft:location:0:path",
"value": "/lib/apk/db/installed"
},
{
"name": "syft:metadata:installedSize",
"value": "0"
},
{
"name": "syft:metadata:originPackage",
"value": "libc-dev"
},
{
"name": "syft:metadata:size",
"value": "0"
}
]
},
{
"bom-ref": "pkg:deb/debian/libpam-runtime@1.5.2-6%2Bdeb12u1?arch=all&distro=debian-12&package-id=7b4ddc97df57aec4&upstream=pam",
"type": "library",
"name": "libpam-runtime",
"version": "1.5.2-6+deb12u1",
"purl": "pkg:deb/debian/libpam-runtime@1.5.2-6%2Bdeb12u1?arch=all&distro=debian-12&upstream=pam",
"externalReferences": [
{
"url": "pkg:deb/debian/pam@1.5.2-6%2Bdeb12u1?arch=source&distro=debian-12",
"type": "source-distribution"
}
],
"properties": [
{
"name": "syft:package:foundBy",
"value": "dpkg-db-cataloger"
},
{
"name": "syft:package:type",
"value": "deb"
},
{
"name": "syft:package:metadataType",
"value": "dpkg-db-entry"
},
{
"name": "syft:location:0:path",
"value": "/var/lib/dpkg/status"
},
{
"name": "syft:metadata:installedSize",
"value": "0"
},
{
"name": "syft:metadata:source",
"value": "pam"
}
]
}
]
}
+14
View File
@@ -87,6 +87,20 @@ func TestCycloneDxImageEncoder(t *testing.T) {
)
}
func TestCycloneDxSourcePackageEncoder(t *testing.T) {
dir := t.TempDir()
testutil.AssertEncoderAgainstGoldenSnapshot(t,
testutil.EncoderSnapshotTestConfig{
Subject: testutil.SourcePackagesInput(t, dir),
Format: getEncoder(t),
UpdateSnapshot: *updateSnapshot,
PersistRedactionsInSnapshot: true,
IsJSON: false,
Redactor: redactor(dir),
},
)
}
func redactor(values ...string) testutil.Redactor {
return testutil.NewRedactions().
WithValuesRedacted(values...).
@@ -0,0 +1,121 @@
<?xml version="1.0" encoding="UTF-8"?>
<bom xmlns="http://cyclonedx.org/schema/bom/1.7" serialNumber="redacted" version="1">
<metadata>
<timestamp>redacted</timestamp>
<tools>
<components>
<component type="application">
<author>anchore</author>
<name>syft</name>
<version>v0.42.0-bogus</version>
</component>
</components>
</tools>
<component bom-ref="redacted" type="file">
<name>some/path</name>
</component>
</metadata>
<components>
<component bom-ref="pkg:deb/debian/attr@1%3A2.4.47-2%2Bb1?arch=amd64&amp;distro=debian-12&amp;package-id=5cc0e8979d7f5115&amp;upstream=attr%401%3A2.4.47-2" type="library">
<name>attr</name>
<version>1:2.4.47-2+b1</version>
<purl>pkg:deb/debian/attr@1%3A2.4.47-2%2Bb1?arch=amd64&amp;distro=debian-12&amp;upstream=attr%401%3A2.4.47-2</purl>
<externalReferences>
<reference type="source-distribution">
<url>pkg:deb/debian/attr@1%3A2.4.47-2?arch=source&amp;distro=debian-12</url>
</reference>
</externalReferences>
<properties>
<property name="syft:package:foundBy">dpkg-db-cataloger</property>
<property name="syft:package:type">deb</property>
<property name="syft:package:metadataType">dpkg-db-entry</property>
<property name="syft:location:0:path">/var/lib/dpkg/status</property>
<property name="syft:metadata:installedSize">0</property>
<property name="syft:metadata:source">attr</property>
<property name="syft:metadata:sourceVersion">1:2.4.47-2</property>
</properties>
</component>
<component bom-ref="pkg:rpm/centos/bind-export-libs@9.11.13-3.el8?arch=x86_64&amp;distro=centos-8&amp;epoch=32&amp;package-id=6008fca05ca7b418&amp;upstream=bind-9.11.13-3.el8.src.rpm" type="library">
<name>bind-export-libs</name>
<version>32:9.11.13-3.el8</version>
<purl>pkg:rpm/centos/bind-export-libs@9.11.13-3.el8?arch=x86_64&amp;distro=centos-8&amp;epoch=32&amp;upstream=bind-9.11.13-3.el8.src.rpm</purl>
<externalReferences>
<reference type="source-distribution">
<url>pkg:rpm/centos/bind@9.11.13-3.el8?arch=src&amp;distro=centos-8&amp;epoch=32</url>
</reference>
</externalReferences>
<properties>
<property name="syft:package:foundBy">rpm-db-cataloger</property>
<property name="syft:package:type">rpm</property>
<property name="syft:package:metadataType">rpm-db-entry</property>
<property name="syft:location:0:path">/var/lib/rpm/Packages</property>
<property name="syft:metadata:epoch">32</property>
<property name="syft:metadata:release">3.el8</property>
<property name="syft:metadata:size">0</property>
<property name="syft:metadata:sourceRpm">bind-9.11.13-3.el8.src.rpm</property>
</properties>
</component>
<component bom-ref="pkg:apk/alpine/busybox@1.35.0-r17?arch=x86_64&amp;distro=alpine-3.16.3&amp;package-id=aa74af1190de1c40" type="library">
<name>busybox</name>
<version>1.35.0-r17</version>
<purl>pkg:apk/alpine/busybox@1.35.0-r17?arch=x86_64&amp;distro=alpine-3.16.3</purl>
<properties>
<property name="syft:package:foundBy">apk-db-cataloger</property>
<property name="syft:package:type">apk</property>
<property name="syft:package:metadataType">apk-db-entry</property>
<property name="syft:location:0:path">/lib/apk/db/installed</property>
<property name="syft:metadata:installedSize">0</property>
<property name="syft:metadata:originPackage">busybox</property>
<property name="syft:metadata:size">0</property>
</properties>
</component>
<component bom-ref="pkg:alpm/arch/gcc-libs@13.2.1-3?arch=x86_64&amp;distro=arch-rolling&amp;package-id=58a2fcf749a9a3cc&amp;upstream=gcc" type="library">
<name>gcc-libs</name>
<version>13.2.1-3</version>
<purl>pkg:alpm/arch/gcc-libs@13.2.1-3?arch=x86_64&amp;distro=arch-rolling&amp;upstream=gcc</purl>
<properties>
<property name="syft:package:foundBy">alpm-db-cataloger</property>
<property name="syft:package:type">alpm</property>
<property name="syft:package:metadataType">alpm-db-entry</property>
<property name="syft:location:0:path">/var/lib/pacman/local/gcc-libs-13.2.1-3/desc</property>
<property name="syft:metadata:architecture">x86_64</property>
<property name="syft:metadata:basepackage">gcc</property>
<property name="syft:metadata:package">gcc-libs</property>
<property name="syft:metadata:size">0</property>
<property name="syft:metadata:version">13.2.1-3</property>
</properties>
</component>
<component bom-ref="pkg:apk/alpine/libc-utils@0.7.2-r3?arch=x86_64&amp;distro=alpine-3.16.3&amp;package-id=16a8c7b19fde6f4f&amp;upstream=libc-dev" type="library">
<name>libc-utils</name>
<version>0.7.2-r3</version>
<purl>pkg:apk/alpine/libc-utils@0.7.2-r3?arch=x86_64&amp;distro=alpine-3.16.3&amp;upstream=libc-dev</purl>
<properties>
<property name="syft:package:foundBy">apk-db-cataloger</property>
<property name="syft:package:type">apk</property>
<property name="syft:package:metadataType">apk-db-entry</property>
<property name="syft:location:0:path">/lib/apk/db/installed</property>
<property name="syft:metadata:installedSize">0</property>
<property name="syft:metadata:originPackage">libc-dev</property>
<property name="syft:metadata:size">0</property>
</properties>
</component>
<component bom-ref="pkg:deb/debian/libpam-runtime@1.5.2-6%2Bdeb12u1?arch=all&amp;distro=debian-12&amp;package-id=7b4ddc97df57aec4&amp;upstream=pam" type="library">
<name>libpam-runtime</name>
<version>1.5.2-6+deb12u1</version>
<purl>pkg:deb/debian/libpam-runtime@1.5.2-6%2Bdeb12u1?arch=all&amp;distro=debian-12&amp;upstream=pam</purl>
<externalReferences>
<reference type="source-distribution">
<url>pkg:deb/debian/pam@1.5.2-6%2Bdeb12u1?arch=source&amp;distro=debian-12</url>
</reference>
</externalReferences>
<properties>
<property name="syft:package:foundBy">dpkg-db-cataloger</property>
<property name="syft:package:type">deb</property>
<property name="syft:package:metadataType">dpkg-db-entry</property>
<property name="syft:location:0:path">/var/lib/dpkg/status</property>
<property name="syft:metadata:installedSize">0</property>
<property name="syft:metadata:source">pam</property>
</properties>
</component>
</components>
</bom>
@@ -2,6 +2,7 @@ package cyclonedxutil
import (
"io"
"slices"
"github.com/CycloneDX/cyclonedx-go"
@@ -29,9 +30,35 @@ func NewEncoder(version string, format cyclonedx.BOMFileFormat, pretty bool) (En
func (e Encoder) Encode(writer io.Writer, s sbom.SBOM) error {
bom := cyclonedxhelpers.ToFormatModel(s)
if e.version < cyclonedx.SpecVersion1_6 {
// cyclonedx-go does not downgrade this 1.6+ reference type for older spec versions
// (see https://github.com/CycloneDX/cyclonedx-go/issues/291), so drop it rather than emit an invalid document
dropExternalReferenceType(bom.Components, cyclonedx.ERTypeSourceDistribution)
}
enc := cyclonedx.NewBOMEncoder(writer, e.format)
enc.SetPretty(e.pretty)
enc.SetEscapeHTML(false)
return enc.EncodeVersion(bom, e.version)
}
func dropExternalReferenceType(components *[]cyclonedx.Component, refType cyclonedx.ExternalReferenceType) {
if components == nil {
return
}
for i := range *components {
c := &(*components)[i]
dropExternalReferenceType(c.Components, refType)
if c.ExternalReferences == nil {
continue
}
refs := slices.DeleteFunc(*c.ExternalReferences, func(r cyclonedx.ExternalReference) bool {
return r.Type == refType
})
if len(refs) == 0 {
c.ExternalReferences = nil
continue
}
c.ExternalReferences = &refs
}
}
@@ -7,6 +7,7 @@ import (
"github.com/CycloneDX/cyclonedx-go"
"github.com/anchore/packageurl-go"
"github.com/anchore/syft/internal/file"
syftFile "github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/pkg"
@@ -79,12 +80,162 @@ func encodeExternalReferences(p pkg.Package) *[]cyclonedx.ExternalReference {
}
}
}
if srcRef := encodeSourcePackageExternalReference(p); srcRef != nil {
refs = append(refs, *srcRef)
}
if len(refs) > 0 {
return &refs
}
return nil
}
// encodeSourcePackageExternalReference returns a "source-distribution" reference holding the PURL of the distro
// source package that a binary OS package was built from (e.g. libpam-runtime -> pkg:deb/debian/pam@...?arch=source).
// It is only emitted when the source differs from the binary package (by name or version) and when the package PURL
// is of the type implied by the metadata. The main consumer is vulnerability matching, since distros tend to report
// vulnerabilities against the source package rather than each binary package built from it.
//
// why an external reference and not pedigree: `component.pedigree.ancestors` looks like the obvious spot for "this
// binary came from that source", but pedigree describes code lineage (forks, patched variants, and the patches that
// separate them). A binary built unmodified from its own distro source package is the same code in a different form,
// not a fork, and CycloneDX components are deliberately agnostic to source vs binary form (see the comments from the
// CycloneDX maintainers in https://github.com/CycloneDX/specification/issues/612#issuecomment-2958800363). The
// ancestor slot is better left for the upstream project a distro patched (e.g. Linux-PAM for Debian's pam), which
// syft does not know today.
//
// why "source-distribution": the spec defines it as "the location where the source code distributable can be
// obtained", and a deb/rpm source package is exactly that distributable (a .dsc and tarballs in the Debian source
// pool, a .src.rpm in an SRPMS repo). The value is a PURL (an identifier) rather than a download URL, which the
// schema allows since external reference URLs are URIs of any scheme, and a CycloneDX maintainer confirmed this exact
// shape is valid in https://github.com/CycloneDX/specification/issues/612#issuecomment-5428057874. A first-class
// "source" component type was discussed in that issue and deferred to CycloneDX 2.0.
func encodeSourcePackageExternalReference(p pkg.Package) *cyclonedx.ExternalReference {
if p.PURL == "" {
return nil
}
src := sourcePackageOf(p)
if src == nil {
return nil
}
// the binary PURL is only consulted for the namespace and a small set of qualifiers (e.g. distro), everything
// else describes the binary package and must not be restated as a fact about the source package.
binPurl, err := packageurl.FromString(p.PURL)
if err != nil || binPurl.Type != src.purlType {
return nil
}
qualifiers := map[string]string{}
for _, q := range binPurl.Qualifiers {
if src.keepQualifier(q.Key) {
qualifiers[q.Key] = q.Value
}
}
if src.arch != "" {
qualifiers[pkg.PURLQualifierArch] = src.arch
}
srcPurl := packageurl.NewPackageURL(
src.purlType,
binPurl.Namespace,
src.name,
src.version,
packageurl.QualifiersFromMap(qualifiers),
"",
)
return &cyclonedx.ExternalReference{
Type: cyclonedx.ERTypeSourceDistribution,
URL: srcPurl.ToString(),
}
}
// sourcePackage is the identity of the source package a binary OS package was built from.
type sourcePackage struct {
purlType string
name string
version string
// arch is the ecosystem's source architecture marker ("source" for deb, "src" or "nosrc" for rpm)
arch string
}
func (s sourcePackage) keepQualifier(key string) bool {
switch key {
case pkg.PURLQualifierDistro:
return true
case pkg.PURLQualifierEpoch, pkg.PURLQualifierRpmModularity:
return s.purlType == pkg.RpmPkg.PackageURLType()
}
return false
}
// sourcePackageOf returns the source package for p, or nil when there is none or it is the same as the binary.
//
// only deb and rpm are supported, since those are the ecosystems that publish real source packages with their own
// PURL identity (arch=source / arch=src). apk (origin) and alpm (pkgbase) are intentionally left out: there the
// "source" is a build recipe rather than a published source distributable, and its name is usually also the name of
// a real binary package (e.g. pkg:apk/alpine/libc-dev@0.7.2-r3 is an installable package, not a source archive), so
// a "source-distribution" reference would point at a binary package. Whether those ecosystems have a meaningful
// source PURL is still an open question, and until then the `upstream` PURL qualifier continues to carry that info.
func sourcePackageOf(p pkg.Package) *sourcePackage {
var src sourcePackage
var binName, binVersion string
switch m := p.Metadata.(type) {
case pkg.DpkgDBEntry:
src = dpkgSourcePackage(m)
binName, binVersion = m.Package, m.Version
case pkg.DpkgArchiveEntry:
src = dpkgSourcePackage(pkg.DpkgDBEntry(m))
binName, binVersion = m.Package, m.Version
case pkg.RpmDBEntry:
src = rpmSourcePackage(m.SourceRpm)
binName, binVersion = m.Name, m.Version+"-"+m.Release
case pkg.RpmArchive:
src = rpmSourcePackage(m.SourceRpm)
binName, binVersion = m.Name, m.Version+"-"+m.Release
default:
return nil
}
if src.name == "" || src.version == "" {
return nil
}
if src.name == binName && src.version == binVersion {
return nil
}
return &src
}
func dpkgSourcePackage(entry pkg.DpkgDBEntry) sourcePackage {
version := entry.SourceVersion
if version == "" {
version = entry.Version
}
return sourcePackage{purlType: pkg.DebPkg.PackageURLType(), name: entry.Source, version: version, arch: "source"}
}
// rpmSourcePackage parses a source RPM filename of the form <name>-<version>-<release>.(no)src.rpm.
func rpmSourcePackage(sourceRpm string) sourcePackage {
var arch string
switch {
case strings.HasSuffix(sourceRpm, ".src.rpm"):
arch = "src"
case strings.HasSuffix(sourceRpm, ".nosrc.rpm"):
arch = "nosrc"
default:
return sourcePackage{}
}
nvr := strings.TrimSuffix(sourceRpm, "."+arch+".rpm")
release := strings.LastIndex(nvr, "-")
if release < 0 {
return sourcePackage{}
}
ver := strings.LastIndex(nvr[:release], "-")
if ver < 0 {
return sourcePackage{}
}
return sourcePackage{purlType: pkg.RpmPkg.PackageURLType(), name: nvr[:ver], version: nvr[ver+1:], arch: arch}
}
// supported algorithm in cycloneDX as of 1.4
// "MD5", "SHA-1", "SHA-256", "SHA-384", "SHA-512",
// "SHA3-256", "SHA3-384", "SHA3-512", "BLAKE2b-256", "BLAKE2b-384", "BLAKE2b-512", "BLAKE3"
@@ -144,6 +144,342 @@ func Test_encodeExternalReferences(t *testing.T) {
}
}
func Test_encodeSourcePackage(t *testing.T) {
rpmEpoch := 32
tests := []struct {
name string
input pkg.Package
expected *cyclonedx.ExternalReference
}{
{
name: "no metadata",
input: pkg.Package{},
expected: nil,
},
{
name: "no PURL",
input: pkg.Package{
Name: "libpam-runtime",
Version: "1.5.2-6+deb12u1",
Type: pkg.DebPkg,
Metadata: pkg.DpkgDBEntry{
Package: "libpam-runtime",
Version: "1.5.2-6+deb12u1",
Source: "pam",
},
},
expected: nil,
},
{
// apk origins are build recipes, not published source packages, so they are not supported
name: "from apk is not supported",
input: pkg.Package{
Name: "libc-utils",
Version: "0.7.2-r3",
Type: pkg.ApkPkg,
PURL: "pkg:apk/alpine/libc-utils@0.7.2-r3?arch=x86_64&distro=alpine-3.16.3&upstream=libc-dev",
Metadata: pkg.ApkDBEntry{
Package: "libc-utils",
OriginPackage: "libc-dev",
Version: "0.7.2-r3",
Architecture: "x86_64",
},
},
expected: nil,
},
{
name: "from deb",
input: pkg.Package{
Name: "libpam-runtime",
Version: "1.5.2-6+deb12u1",
Type: pkg.DebPkg,
PURL: "pkg:deb/debian/libpam-runtime@1.5.2-6%2Bdeb12u1?arch=all&distro=debian-12&upstream=pam",
Metadata: pkg.DpkgDBEntry{
Package: "libpam-runtime",
Version: "1.5.2-6+deb12u1",
Source: "pam",
Architecture: "all",
},
},
expected: &cyclonedx.ExternalReference{
Type: cyclonedx.ERTypeSourceDistribution, URL: "pkg:deb/debian/pam@1.5.2-6%2Bdeb12u1?arch=source&distro=debian-12",
},
},
{
name: "from deb with source version",
input: pkg.Package{
Name: "attr",
Version: "1:2.4.47-2+b1",
Type: pkg.DebPkg,
PURL: "pkg:deb/debian/attr@1%3A2.4.47-2%2Bb1?arch=amd64&distro=debian-12&upstream=attr%401%3A2.4.47-2",
Metadata: pkg.DpkgDBEntry{
Package: "attr",
Version: "1:2.4.47-2+b1",
Source: "attr",
SourceVersion: "1:2.4.47-2",
Architecture: "amd64",
},
},
expected: &cyclonedx.ExternalReference{
Type: cyclonedx.ERTypeSourceDistribution, URL: "pkg:deb/debian/attr@1%3A2.4.47-2?arch=source&distro=debian-12",
},
},
{
name: "from deb with empty source",
input: pkg.Package{
Name: "some-package",
Version: "1.0.0",
Type: pkg.DebPkg,
PURL: "pkg:deb/debian/some-package@1.0.0?arch=all&distro=debian-12",
Metadata: pkg.DpkgDBEntry{
Package: "some-package",
Version: "1.0.0",
Architecture: "all",
},
},
expected: nil,
},
{
name: "from deb archive",
input: pkg.Package{
Name: "libpam-runtime",
Version: "1.5.2-6+deb12u1",
Type: pkg.DebPkg,
PURL: "pkg:deb/libpam-runtime@1.5.2-6%2Bdeb12u1?arch=all&upstream=pam",
Metadata: pkg.DpkgArchiveEntry{
Package: "libpam-runtime",
Version: "1.5.2-6+deb12u1",
Source: "pam",
Architecture: "all",
},
},
expected: &cyclonedx.ExternalReference{
Type: cyclonedx.ERTypeSourceDistribution, URL: "pkg:deb/pam@1.5.2-6%2Bdeb12u1?arch=source",
},
},
{
name: "from rpm",
input: pkg.Package{
Name: "bind-export-libs",
Version: "32:9.11.13-3.el8",
Type: pkg.RpmPkg,
PURL: "pkg:rpm/centos/bind-export-libs@9.11.13-3.el8?arch=x86_64&distro=centos-8&epoch=32&upstream=bind-9.11.13-3.el8.src.rpm",
Metadata: pkg.RpmDBEntry{
Name: "bind-export-libs",
Version: "9.11.13",
Release: "3.el8",
Epoch: &rpmEpoch,
Arch: "x86_64",
SourceRpm: "bind-9.11.13-3.el8.src.rpm",
},
},
expected: &cyclonedx.ExternalReference{
Type: cyclonedx.ERTypeSourceDistribution, URL: "pkg:rpm/centos/bind@9.11.13-3.el8?arch=src&distro=centos-8&epoch=32",
},
},
{
name: "from rpm with source = bin",
input: pkg.Package{
Name: "some-package",
Version: "32:1.0.0-1.el8",
Type: pkg.RpmPkg,
PURL: "pkg:rpm/centos/some-package@1.0.0-1.el8?arch=x86_64&distro=centos-8&epoch=32&upstream=some-package-1.0.0-1.el8.src.rpm",
Metadata: pkg.RpmDBEntry{
Name: "some-package",
Version: "1.0.0",
Release: "1.el8",
Epoch: &rpmEpoch,
Arch: "x86_64",
SourceRpm: "some-package-1.0.0-1.el8.src.rpm",
},
},
expected: nil,
},
{
name: "from rpm archive",
input: pkg.Package{
Name: "bind-export-libs",
Version: "32:9.11.13-3.el8",
Type: pkg.RpmPkg,
PURL: "pkg:rpm/bind-export-libs@9.11.13-3.el8?arch=x86_64&epoch=32&upstream=bind-9.11.13-3.el8.src.rpm",
Metadata: pkg.RpmArchive{
Name: "bind-export-libs",
Version: "9.11.13",
Release: "3.el8",
Epoch: &rpmEpoch,
Arch: "x86_64",
SourceRpm: "bind-9.11.13-3.el8.src.rpm",
},
},
expected: &cyclonedx.ExternalReference{
Type: cyclonedx.ERTypeSourceDistribution, URL: "pkg:rpm/bind@9.11.13-3.el8?arch=src&epoch=32",
},
},
{
// alpm pkgbases are build recipes, not published source packages, so they are not supported
name: "from alpm is not supported",
input: pkg.Package{
Name: "gcc-libs",
Version: "13.2.1-3",
Type: pkg.AlpmPkg,
PURL: "pkg:alpm/arch/gcc-libs@13.2.1-3?arch=x86_64&distro=arch-rolling&upstream=gcc",
Metadata: pkg.AlpmDBEntry{
Package: "gcc-libs",
Version: "13.2.1-3",
BasePackage: "gcc",
Architecture: "x86_64",
},
},
expected: nil,
},
{
name: "from rpm with dashes in the source name",
input: pkg.Package{
Name: "perl-Foo-Bar-libs",
Type: pkg.RpmPkg,
PURL: "pkg:rpm/redhat/perl-Foo-Bar-libs@1.0-1.el8?arch=x86_64&distro=rhel-8",
Metadata: pkg.RpmDBEntry{
Name: "perl-Foo-Bar-libs",
Version: "1.0",
Release: "1.el8",
SourceRpm: "perl-Foo-Bar-1.0-1.el8.src.rpm",
},
},
expected: &cyclonedx.ExternalReference{
Type: cyclonedx.ERTypeSourceDistribution, URL: "pkg:rpm/redhat/perl-Foo-Bar@1.0-1.el8?arch=src&distro=rhel-8",
},
},
{
name: "from rpm nosrc",
input: pkg.Package{
Name: "some-firmware",
Type: pkg.RpmPkg,
PURL: "pkg:rpm/redhat/some-firmware@1.0-1.el8?arch=noarch&distro=rhel-8",
Metadata: pkg.RpmDBEntry{
Name: "some-firmware",
Version: "1.0",
Release: "1.el8",
SourceRpm: "firmware-1.0-1.el8.nosrc.rpm",
},
},
expected: &cyclonedx.ExternalReference{
Type: cyclonedx.ERTypeSourceDistribution, URL: "pkg:rpm/redhat/firmware@1.0-1.el8?arch=nosrc&distro=rhel-8",
},
},
{
name: "from rpm keeps rpmmod",
input: pkg.Package{
Name: "nodejs-libs",
Type: pkg.RpmPkg,
PURL: "pkg:rpm/redhat/nodejs-libs@18.0-1.el8?arch=x86_64&distro=rhel-8&rpmmod=nodejs%3A18",
Metadata: pkg.RpmDBEntry{
Name: "nodejs-libs",
Version: "18.0",
Release: "1.el8",
SourceRpm: "nodejs-18.0-1.el8.src.rpm",
},
},
expected: &cyclonedx.ExternalReference{
Type: cyclonedx.ERTypeSourceDistribution, URL: "pkg:rpm/redhat/nodejs@18.0-1.el8?arch=src&distro=rhel-8&rpmmod=nodejs%3A18",
},
},
{
name: "from rpm with malformed source rpm",
input: pkg.Package{
Name: "gpg-pubkey",
Type: pkg.RpmPkg,
PURL: "pkg:rpm/redhat/gpg-pubkey@1-1?distro=rhel-8",
Metadata: pkg.RpmDBEntry{
Name: "gpg-pubkey",
Version: "1",
Release: "1",
SourceRpm: "(none)",
},
},
expected: nil,
},
{
name: "from rpm with non-source rpm filename",
input: pkg.Package{
Name: "bash",
Type: pkg.RpmPkg,
PURL: "pkg:rpm/redhat/bash@5.1-2?arch=x86_64&distro=rhel-8",
Metadata: pkg.RpmDBEntry{
Name: "bash",
Version: "5.1",
Release: "2",
SourceRpm: "bash-5.1-2.x86_64.rpm",
},
},
expected: nil,
},
{
name: "from rpm with too few dashes",
input: pkg.Package{
Name: "bash",
Type: pkg.RpmPkg,
PURL: "pkg:rpm/redhat/bash@5.1-2?arch=x86_64&distro=rhel-8",
Metadata: pkg.RpmDBEntry{
Name: "bash",
Version: "5.1",
Release: "2",
SourceRpm: "bash-5.1.src.rpm",
},
},
expected: nil,
},
{
name: "binary qualifiers and subpath are not carried over",
input: pkg.Package{
Name: "libssl3",
Type: pkg.DebPkg,
PURL: "pkg:deb/debian/libssl3@3.0.0-1?arch=amd64&distro=debian-12&checksum=sha256:dead&download_url=https://example.com/x.deb&cpes=cpe:2.3:a:x:x:1:*:*:*:*:*:*:*#sub/dir",
Metadata: pkg.DpkgDBEntry{
Package: "libssl3",
Version: "3.0.0-1",
Source: "openssl",
},
},
expected: &cyclonedx.ExternalReference{
Type: cyclonedx.ERTypeSourceDistribution, URL: "pkg:deb/debian/openssl@3.0.0-1?arch=source&distro=debian-12",
},
},
{
name: "PURL type does not match metadata",
input: pkg.Package{
Name: "x",
Type: pkg.DebPkg,
PURL: "pkg:npm/x@1",
Metadata: pkg.DpkgDBEntry{
Package: "x",
Version: "1",
Source: "lodash",
},
},
expected: nil,
},
{
name: "invalid PURL",
input: pkg.Package{
Name: "x",
Type: pkg.DebPkg,
PURL: "not-a-purl",
Metadata: pkg.DpkgDBEntry{
Package: "x",
Version: "1",
Source: "y",
},
},
expected: nil,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
assert.Equal(t, test.expected, encodeSourcePackageExternalReference(test.input))
})
}
}
func Test_isValidExternalRef(t *testing.T) {
tests := []struct {
name string
@@ -9,16 +9,21 @@ func DocumentName(src source.Description) string {
return src.Name
}
var name string
switch metadata := src.Metadata.(type) {
case source.ImageMetadata:
return metadata.UserInput
name = metadata.UserInput
case source.OCIModelMetadata:
return metadata.UserInput
name = metadata.UserInput
case source.DirectoryMetadata:
return metadata.Path
name = metadata.Path
case source.FileMetadata:
return metadata.Path
default:
name = metadata.Path
}
// the SPDX document name is mandatory, and a source read from another SBOM may not have one
if name == "" {
return "unknown"
}
return name
}
@@ -1,8 +1,6 @@
package helpers
import (
"fmt"
"strings"
"testing"
"github.com/stretchr/testify/assert"
@@ -73,11 +71,23 @@ func Test_DocumentName(t *testing.T) {
},
expected: "some/name",
},
{
name: "no name or metadata",
srcMetadata: source.Description{},
expected: "unknown",
},
{
name: "metadata without a name",
srcMetadata: source.Description{
Metadata: source.ImageMetadata{},
},
expected: "unknown",
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
actual := DocumentName(test.srcMetadata)
assert.True(t, strings.HasPrefix(actual, test.expected), fmt.Sprintf("actual name %q", actual))
assert.Equal(t, test.expected, actual)
// track each scheme tested (passed or not)
tracker.Tested(t, test.srcMetadata.Metadata)
@@ -0,0 +1,146 @@
package testutil
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/require"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/sbom"
"github.com/anchore/syft/syft/source/directorysource"
)
func SourcePackagesInput(t testing.TB, dir string) sbom.SBOM {
path := filepath.Join(dir, "some", "path")
require.NoError(t, os.MkdirAll(path, 0755))
src, err := directorysource.New(
directorysource.Config{
Path: path,
Base: dir,
},
)
require.NoError(t, err)
return sbom.SBOM{
Artifacts: sbom.Artifacts{
Packages: newSourcePackageCatalog(),
},
Source: src.Describe(),
Descriptor: sbom.Descriptor{
Name: "syft",
Version: "v0.42.0-bogus",
},
}
}
//nolint:funlen
func newSourcePackageCatalog() *pkg.Collection {
epoch := 32
catalog := pkg.NewCollection()
// apk: the origin differs from the package name, but apk is not supported so no source reference is expected
catalog.Add(pkg.Package{
Name: "libc-utils",
Version: "0.7.2-r3",
Type: pkg.ApkPkg,
FoundBy: "apk-db-cataloger",
Locations: file.NewLocationSet(file.NewLocation("/lib/apk/db/installed")),
PURL: "pkg:apk/alpine/libc-utils@0.7.2-r3?arch=x86_64&distro=alpine-3.16.3&upstream=libc-dev",
Metadata: pkg.ApkDBEntry{
Package: "libc-utils",
OriginPackage: "libc-dev",
Version: "0.7.2-r3",
Architecture: "x86_64",
},
})
// apk: negative control -- the package is its own origin
catalog.Add(pkg.Package{
Name: "busybox",
Version: "1.35.0-r17",
Type: pkg.ApkPkg,
FoundBy: "apk-db-cataloger",
Locations: file.NewLocationSet(file.NewLocation("/lib/apk/db/installed")),
PURL: "pkg:apk/alpine/busybox@1.35.0-r17?arch=x86_64&distro=alpine-3.16.3",
Metadata: pkg.ApkDBEntry{
Package: "busybox",
OriginPackage: "busybox",
Version: "1.35.0-r17",
Architecture: "x86_64",
},
})
// deb: source name differs, and the source version is implied by the binary version
catalog.Add(pkg.Package{
Name: "libpam-runtime",
Version: "1.5.2-6+deb12u1",
Type: pkg.DebPkg,
FoundBy: "dpkg-db-cataloger",
Locations: file.NewLocationSet(file.NewLocation("/var/lib/dpkg/status")),
PURL: "pkg:deb/debian/libpam-runtime@1.5.2-6%2Bdeb12u1?arch=all&distro=debian-12&upstream=pam",
Metadata: pkg.DpkgDBEntry{
Package: "libpam-runtime",
Version: "1.5.2-6+deb12u1",
Source: "pam",
Architecture: "all",
},
})
// deb: the source version differs from the binary version
catalog.Add(pkg.Package{
Name: "attr",
Version: "1:2.4.47-2+b1",
Type: pkg.DebPkg,
FoundBy: "dpkg-db-cataloger",
Locations: file.NewLocationSet(file.NewLocation("/var/lib/dpkg/status")),
PURL: "pkg:deb/debian/attr@1%3A2.4.47-2%2Bb1?arch=amd64&distro=debian-12&upstream=attr%401%3A2.4.47-2",
Metadata: pkg.DpkgDBEntry{
Package: "attr",
Version: "1:2.4.47-2+b1",
Source: "attr",
SourceVersion: "1:2.4.47-2",
Architecture: "amd64",
},
})
// rpm: name and version both come out of the source RPM filename, and the epoch is carried over
catalog.Add(pkg.Package{
Name: "bind-export-libs",
Version: "32:9.11.13-3.el8",
Type: pkg.RpmPkg,
FoundBy: "rpm-db-cataloger",
Locations: file.NewLocationSet(file.NewLocation("/var/lib/rpm/Packages")),
PURL: "pkg:rpm/centos/bind-export-libs@9.11.13-3.el8?arch=x86_64&distro=centos-8&epoch=32&upstream=bind-9.11.13-3.el8.src.rpm",
Metadata: pkg.RpmDBEntry{
Name: "bind-export-libs",
Version: "9.11.13",
Release: "3.el8",
Epoch: &epoch,
Arch: "x86_64",
SourceRpm: "bind-9.11.13-3.el8.src.rpm",
},
})
// alpm: the base package differs from the package name, but alpm is not supported so no source reference is expected
catalog.Add(pkg.Package{
Name: "gcc-libs",
Version: "13.2.1-3",
Type: pkg.AlpmPkg,
FoundBy: "alpm-db-cataloger",
Locations: file.NewLocationSet(file.NewLocation("/var/lib/pacman/local/gcc-libs-13.2.1-3/desc")),
PURL: "pkg:alpm/arch/gcc-libs@13.2.1-3?arch=x86_64&distro=arch-rolling&upstream=gcc",
Metadata: pkg.AlpmDBEntry{
Package: "gcc-libs",
Version: "13.2.1-3",
BasePackage: "gcc",
Architecture: "x86_64",
},
})
return catalog
}
+5 -1
View File
@@ -8,6 +8,7 @@ import (
"strings"
"github.com/anchore/packageurl-go"
syftinternal "github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/format/internal"
"github.com/anchore/syft/syft/pkg"
@@ -61,7 +62,7 @@ func toSyftModel(r io.Reader) (*sbom.SBOM, error) {
var errs []error
pkgs := pkg.NewCollection()
scanner := bufio.NewScanner(r)
scanner := syftinternal.NewLineScanner(r)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
@@ -84,6 +85,9 @@ func toSyftModel(r io.Reader) (*sbom.SBOM, error) {
p.SetID()
pkgs.Add(p)
}
if err := scanner.Err(); err != nil {
errs = append(errs, err)
}
return &sbom.SBOM{
Artifacts: sbom.Artifacts{
+2 -23
View File
@@ -236,23 +236,6 @@ func TestSupportedVersions(t *testing.T) {
subject := testutil.DirectoryInput(t, t.TempDir())
dec := NewFormatDecoder()
relationshipOffsetPerVersion := map[string]int{
// the package representing the source gets a relationship from the source package to all other packages found
// these relationships cannot be removed until the primaryPackagePurpose info is available in 2.3
"2.1": 2,
"2.2": 2,
// the source-to-package relationships can be removed since the primaryPackagePurpose info is available in 2.3
"2.3": 0,
"3.0": 0,
}
pkgCountOffsetPerVersion := map[string]int{
"2.1": 1, // the source is mapped as a package, but cannot distinguish it since the primaryPackagePurpose info is not available until 2.3
"2.2": 1, // the source is mapped as a package, but cannot distinguish it since the primaryPackagePurpose info is not available until 2.3
"2.3": 0, // the source package can be removed since the primaryPackagePurpose info is available
"3.0": 0,
}
for _, enc := range encs {
t.Run(enc.Version(), func(t *testing.T) {
require.Contains(t, versions, enc.Version())
@@ -274,13 +257,9 @@ func TestSupportedVersions(t *testing.T) {
require.NotEmpty(t, s.Artifacts.Packages.PackageCount())
offset := relationshipOffsetPerVersion[enc.Version()]
assert.Equal(t, len(subject.Relationships), len(s.Relationships), "mismatched relationship count")
assert.Equal(t, len(subject.Relationships)+offset, len(s.Relationships), "mismatched relationship count")
offset = pkgCountOffsetPerVersion[enc.Version()]
if !assert.Equal(t, subject.Artifacts.Packages.PackageCount()+offset, s.Artifacts.Packages.PackageCount(), "mismatched package count") {
if !assert.Equal(t, subject.Artifacts.Packages.PackageCount(), s.Artifacts.Packages.PackageCount(), "mismatched package count") {
t.Logf("expected: %d", subject.Artifacts.Packages.PackageCount())
for _, p := range subject.Artifacts.Packages.Sorted() {
t.Logf(" - %s", p.String())
@@ -47,13 +47,13 @@
"copyrightText": "NOASSERTION"
},
{
"SPDXID": "SPDXRef-DocumentRoot-Unknown-",
"SPDXID": "SPDXRef-DocumentRoot-Unknown-unknown",
"copyrightText": "NOASSERTION",
"downloadLocation": "NOASSERTION",
"filesAnalyzed": false,
"licenseConcluded": "NOASSERTION",
"licenseDeclared": "NOASSERTION",
"name": "",
"name": "unknown",
"supplier": "NOASSERTION"
}
],
@@ -82,18 +82,18 @@
"relationshipType": "CONTAINS"
},
{
"spdxElementId": "SPDXRef-DocumentRoot-Unknown-",
"spdxElementId": "SPDXRef-DocumentRoot-Unknown-unknown",
"relatedSpdxElement": "SPDXRef-Package-files-analyzed-false-7d37ba9d2f7c574b",
"relationshipType": "CONTAINS"
},
{
"spdxElementId": "SPDXRef-DocumentRoot-Unknown-",
"spdxElementId": "SPDXRef-DocumentRoot-Unknown-unknown",
"relatedSpdxElement": "SPDXRef-Package-files-analyzed-true-035066c2086b8bb4",
"relationshipType": "CONTAINS"
},
{
"spdxElementId": "SPDXRef-DOCUMENT",
"relatedSpdxElement": "SPDXRef-DocumentRoot-Unknown-",
"relatedSpdxElement": "SPDXRef-DocumentRoot-Unknown-unknown",
"relationshipType": "DESCRIBES"
}
]
+39 -21
View File
@@ -3,6 +3,7 @@ package spdxtagvalue
import (
"bytes"
"flag"
"fmt"
"testing"
"github.com/stretchr/testify/assert"
@@ -143,21 +144,6 @@ func TestSupportedVersions(t *testing.T) {
subject := testutil.DirectoryInput(t, t.TempDir())
dec := NewFormatDecoder()
relationshipOffsetPerVersion := map[string]int{
// the package representing the source gets a relationship from the source package to all other packages found
// these relationships cannot be removed until the primaryPackagePurpose info is available in 2.3
"2.1": 2,
"2.2": 2,
// the source-to-package relationships can be removed since the primaryPackagePurpose info is available in 2.3
"2.3": 0,
}
pkgCountOffsetPerVersion := map[string]int{
"2.1": 1, // the source is mapped as a package, but cannot distinguish it since the primaryPackagePurpose info is not available until 2.3
"2.2": 1, // the source is mapped as a package, but cannot distinguish it since the primaryPackagePurpose info is not available until 2.3
"2.3": 0, // the source package can be removed since the primaryPackagePurpose info is available
}
for _, enc := range encs {
t.Run(enc.Version(), func(t *testing.T) {
require.Contains(t, versions, enc.Version())
@@ -178,13 +164,9 @@ func TestSupportedVersions(t *testing.T) {
require.NotEmpty(t, s.Artifacts.Packages.PackageCount())
offset := relationshipOffsetPerVersion[enc.Version()]
assert.Equal(t, len(subject.Relationships), len(s.Relationships), "mismatched relationship count")
assert.Equal(t, len(subject.Relationships)+offset, len(s.Relationships), "mismatched relationship count")
offset = pkgCountOffsetPerVersion[enc.Version()]
if !assert.Equal(t, subject.Artifacts.Packages.PackageCount()+offset, s.Artifacts.Packages.PackageCount(), "mismatched package count") {
if !assert.Equal(t, subject.Artifacts.Packages.PackageCount(), s.Artifacts.Packages.PackageCount(), "mismatched package count") {
t.Logf("expected: %d", subject.Artifacts.Packages.PackageCount())
for _, p := range subject.Artifacts.Packages.Sorted() {
t.Logf(" - %s", p.String())
@@ -209,3 +191,39 @@ func defaultFormatEncoders() []sbom.FormatEncoder {
}
return encs
}
func TestEncodeDecodeSourceWithoutName(t *testing.T) {
// a source read from another SBOM (a purl list, a CycloneDX document without metadata.component) may have no name,
// but the SPDX package name is mandatory and the tag-value decoder needs it to start the root package section
p := pkg.Package{Name: "left-pad", Version: "1.3.0", Type: pkg.NpmPkg}
p.SetID()
for _, src := range []source.Description{
{},
{Metadata: source.FileMetadata{}},
{Metadata: source.DirectoryMetadata{}},
{Metadata: source.ImageMetadata{}},
{Metadata: source.OCIModelMetadata{}},
{Metadata: source.SnapMetadata{}},
} {
for _, enc := range defaultFormatEncoders() {
t.Run(fmt.Sprintf("%T/%s", src.Metadata, enc.Version()), func(t *testing.T) {
var buf bytes.Buffer
require.NoError(t, enc.Encode(&buf, sbom.SBOM{
Source: src,
Artifacts: sbom.Artifacts{Packages: pkg.NewCollection(p)},
}))
s, _, _, err := NewFormatDecoder().Decode(bytes.NewReader(buf.Bytes()))
require.NoError(t, err, "encoded document:\n%s", buf.String())
var names []string
for _, decoded := range s.Artifacts.Packages.Sorted() {
names = append(names, decoded.Name)
}
// the root package describes the source, so it must not come back as a package of the SBOM
assert.Equal(t, []string{"left-pad"}, names)
})
}
}
}
+94
View File
@@ -0,0 +1,94 @@
package format
import (
"bytes"
"fmt"
"math"
"testing"
"github.com/anchore/syft/syft/format/syftjson"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/sbom"
"github.com/anchore/syft/syft/source"
)
// TestEncodersHandleUntypedMetadataValues pushes awkward values through the untyped metadata fields
// allowlisted in internal/packagemetadata (see untypedFieldAllowlist) and asserts that SetID and every
// encoder neither panic nor fail. A new allowlist entry should get a case here too.
func TestEncodersHandleUntypedMetadataValues(t *testing.T) {
cases := []struct {
name string
value any
// encoding/json rejects non-finite floats, so syft-json returns an error for them. Producers
// have to replace these before they reach the model (the GGUF cataloger does); the case stays
// here so the expectation flips visibly if syft-json ever learns to encode them.
syftJSONRejects bool
}{
{name: "NaN", value: math.NaN(), syftJSONRejects: true},
{name: "+Inf float32", value: float32(math.Inf(1)), syftJSONRejects: true},
{name: "-Inf", value: math.Inf(-1), syftJSONRejects: true},
{name: "NaN in array", value: []any{1.0, math.NaN()}, syftJSONRejects: true},
{name: "invalid utf-8", value: "\xff\xfe"},
{name: "NUL and control characters", value: "a\x00b\x1b"},
{name: "max uint64", value: uint64(math.MaxUint64)},
{name: "deep array", value: nested(2000, func(v any) any { return []any{v} })},
{name: "deep map", value: nested(2000, func(v any) any { return map[string]any{"k": v} })},
{name: "large array", value: make([]any, 100_000)},
}
for _, tt := range cases {
t.Run(tt.name, func(t *testing.T) {
p := pkg.Package{
Name: "model",
Version: "1.0",
Type: pkg.ModelPkg,
Metadata: pkg.GGUFFileHeader{
RemainingKeyValues: map[string]any{"general.value": tt.value},
},
}
if !noPanic(t, "SetID", p.SetID) {
return
}
s := sbom.SBOM{
Artifacts: sbom.Artifacts{Packages: pkg.NewCollection(p)},
Source: source.Description{Metadata: source.DirectoryMetadata{Path: "/models"}},
}
for _, enc := range Encoders() {
id := fmt.Sprintf("%s@%s", enc.ID(), enc.Version())
var buf bytes.Buffer
noPanic(t, id, func() {
err := enc.Encode(&buf, s)
switch {
case tt.syftJSONRejects && enc.ID() == syftjson.ID:
if err == nil {
t.Errorf("%s: expected an encode error for a non-finite float", id)
}
case err != nil:
t.Errorf("%s: encode failed: %v", id, err)
}
})
}
})
}
}
func nested(depth int, wrap func(any) any) any {
var v any = "leaf"
for range depth {
v = wrap(v)
}
return v
}
func noPanic(t *testing.T, what string, fn func()) (ok bool) {
t.Helper()
defer func() {
if r := recover(); r != nil {
t.Errorf("%s: panic: %v", what, r)
ok = false
}
}()
fn()
return true
}
+4 -1
View File
@@ -216,5 +216,8 @@ func (r ChrootContext) ToChrootPath(nativePath string) string {
responsePath = strings.TrimPrefix(responsePath, prefix)
}
return responsePath
// chroot paths are posix regardless of host (these end up as location paths in the SBOM). This uses
// filepath (not path) since responsePath is native here (FromPosix above, then trimmed of a native prefix),
// and only filepath.ToSlash knows the host separator to convert.
return filepath.ToSlash(responsePath)
}
@@ -3,6 +3,7 @@ package fileresolver
import (
"os"
"path/filepath"
"runtime"
"testing"
"github.com/stretchr/testify/assert"
@@ -10,6 +11,9 @@ import (
)
func Test_ChrootContext_RequestResponse(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("fixture relies on symlinks, which git does not check out as links on windows by default")
}
// /
// somewhere/
// outside.txt
@@ -563,6 +567,9 @@ func TestNewChrootContext_BaseValidation(t *testing.T) {
}
func TestToNativeGlob(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("chroot context and expectations are posix paths")
}
tests := []struct {
name string
chrootContext ChrootContext
@@ -4,7 +4,6 @@ import (
"fmt"
"io/fs"
"os"
"path"
"path/filepath"
"runtime"
"sort"
@@ -18,6 +17,7 @@ import (
"github.com/wagoodman/go-progress"
"github.com/anchore/stereoscope/pkg/file"
"github.com/anchore/syft/syft/internal/windows"
)
type indexerMock struct {
@@ -138,6 +138,9 @@ func TestDirectoryIndexer_handleFileAccessErr(t *testing.T) {
}
func TestDirectoryIndexer_IncludeRootPathInIndex(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("indexes the posix filesystem root")
}
filterFn := func(_, path string, _ os.FileInfo, _ error) error {
if path != "/" {
return fs.SkipDir
@@ -206,11 +209,7 @@ func TestDirectoryIndexer_index(t *testing.T) {
info, err := os.Stat(test.path)
assert.NoError(t, err)
// note: the index uses absolute paths, so assertions MUST keep this in mind
cwd, err := os.Getwd()
require.NoError(t, err)
p := file.Path(path.Join(cwd, test.path))
p := indexedPath(t, test.path)
assert.Equal(t, true, tree.HasPath(p))
exists, ref, err := tree.File(p)
assert.Equal(t, true, exists)
@@ -226,6 +225,9 @@ func TestDirectoryIndexer_index(t *testing.T) {
}
func TestDirectoryIndexer_index_for_AncestorSymlinks(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("fixture relies on symlinks, which git does not check out as links on windows by default")
}
// note: this test is testing the effects from NewFromDirectory, indexTree, and addPathToIndex
_, filename, _, ok := runtime.Caller(0)
require.True(t, ok)
@@ -256,11 +258,7 @@ func TestDirectoryIndexer_index_for_AncestorSymlinks(t *testing.T) {
info, err := os.Stat(test.path)
assert.NoError(t, err)
// note: the index uses absolute paths, so assertions MUST keep this in mind
cwd, err := os.Getwd()
require.NoError(t, err)
p := file.Path(path.Join(cwd, test.path))
p := indexedPath(t, test.path)
assert.Equal(t, true, tree.HasPath(p))
exists, ref, err := tree.File(p)
assert.Equal(t, true, exists)
@@ -287,6 +285,9 @@ func TestDirectoryIndexer_index_survive_badSymlink(t *testing.T) {
}
func TestDirectoryIndexer_index_survive_inaccessibleSymlinkTarget(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("relies on symlinks and posix permissions")
}
// a symlink that resolves into a directory we don't have permission to traverse should be skipped
// with a warning like any other inaccessible path, not abort the entire scan (see #3286).
if os.Geteuid() == 0 {
@@ -323,6 +324,9 @@ func TestDirectoryIndexer_index_survive_inaccessibleSymlinkTarget(t *testing.T)
}
func TestDirectoryIndexer_SkipsAlreadyVisitedLinkDestinations(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("fixture relies on symlinks, which git does not check out as links on windows by default")
}
var observedPaths []string
pathObserver := func(_, p string, _ os.FileInfo, _ error) error {
fields := strings.Split(p, "testdata/symlinks-prune-indexing")
@@ -430,6 +434,11 @@ func TestDirectoryIndexer_IndexesAllTypes(t *testing.T) {
}
func Test_allContainedPaths(t *testing.T) {
if runtime.GOOS == "windows" {
// ponytail: production passes native windows paths here, which these posix cases don't represent.
// Ancestor indexing for a root reached through a symlink is likely broken on windows; fix separately.
t.Skip("cases are posix paths")
}
tests := []struct {
name string
@@ -485,6 +494,9 @@ func Test_allContainedPaths(t *testing.T) {
}
func Test_relativePath(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("exercises a posix-only test helper")
}
tests := []struct {
name string
basePath string
@@ -548,3 +560,17 @@ func relativePath(basePath, givenPath string) string {
return relPath
}
// indexedPath returns how the given cwd-relative path is keyed in the index (absolute and always posix)
func indexedPath(t *testing.T, rel string) file.Path {
t.Helper()
cwd, err := os.Getwd()
require.NoError(t, err)
// filepath (not path): cwd is a native host path, so it must be joined with the host separator before
// being converted to the posix form the index is keyed on
p := filepath.Join(cwd, rel)
if windows.HostRunningOnWindows() {
p = windows.ToPosix(p)
}
return file.Path(p)
}
@@ -0,0 +1,48 @@
package fileresolver
import (
"context"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
stereoscopeFile "github.com/anchore/stereoscope/pkg/file"
)
// a location handed out by the resolver should be resolvable again by its own path. This is what the file
// digest cataloger relies on, and it silently found nothing on windows (see #5325).
func TestDirectory_LocationPathsRoundTrip(t *testing.T) {
// resolve symlinks up front (e.g. /var -> /private/var on macOS) so the cwd prefix matches the indexed paths
root, err := filepath.EvalSymlinks(t.TempDir())
require.NoError(t, err)
require.NoError(t, os.MkdirAll(filepath.Join(root, "sub", "dir"), 0o755))
for _, p := range []string{"top.txt", filepath.Join("sub", "mid.txt"), filepath.Join("sub", "dir", "deep.txt")} {
require.NoError(t, os.WriteFile(filepath.Join(root, p), []byte(p), 0o600))
}
// mirror `syft dir:.` from within the scanned directory
t.Chdir(root)
resolver, err := NewFromDirectory(".", "")
require.NoError(t, err)
var regular int
for loc := range resolver.AllLocations(context.Background()) {
md, err := resolver.FileMetadataByLocation(loc)
require.NoError(t, err)
if md.Type != stereoscopeFile.TypeRegular {
continue
}
regular++
assert.NotContains(t, loc.RealPath, `\`, "location paths should be posix")
found, err := resolver.FilesByPath(loc.RealPath)
require.NoError(t, err)
assert.Len(t, found, 1, "unable to resolve %q by its own path", loc.RealPath)
}
assert.Equal(t, 3, regular)
}
@@ -1,126 +0,0 @@
package fileresolver
import "testing"
func Test_windowsToPosix(t *testing.T) {
type args struct {
windowsPath string
}
tests := []struct {
name string
args args
wantPosixPath string
}{
{
name: "basic case",
args: args{
windowsPath: `C:\some\windows\place`,
},
wantPosixPath: "/c/some/windows/place",
},
{
name: "escaped case",
args: args{
windowsPath: `C:\\some\\windows\\place`,
},
wantPosixPath: "/c/some/windows/place",
},
{
name: "forward slash",
args: args{
windowsPath: `C:/foo/bar`,
},
wantPosixPath: "/c/foo/bar",
},
{
name: "mix slash",
args: args{
windowsPath: `C:\foo/bar\`,
},
wantPosixPath: "/c/foo/bar",
},
{
name: "case sensitive case",
args: args{
windowsPath: `C:\Foo/bAr\`,
},
wantPosixPath: "/c/Foo/bAr",
},
{
name: "special char case",
args: args{
windowsPath: `C:\ふー\バー`,
},
wantPosixPath: "/c/ふー/バー",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if gotPosixPath := windowsToPosix(tt.args.windowsPath); gotPosixPath != tt.wantPosixPath {
t.Errorf("windowsToPosix() = %v, want %v", gotPosixPath, tt.wantPosixPath)
}
})
}
}
func Test_posixToWindows(t *testing.T) {
type args struct {
posixPath string
}
tests := []struct {
name string
args args
wantWindowsPath string
}{
{
name: "basic case",
args: args{
posixPath: "/c/some/windows/place",
},
wantWindowsPath: `C:\some\windows\place`,
},
{
name: "escaped case",
args: args{
posixPath: "/c/some/windows/place",
},
wantWindowsPath: `C:\\some\\windows\\place`,
},
{
name: "forward slash",
args: args{
posixPath: "/c/foo/bar",
},
wantWindowsPath: `C:/foo/bar`,
},
{
name: "mix slash",
args: args{
posixPath: "/c/foo/bar",
},
wantWindowsPath: `C:\foo/bar\`,
},
{
name: "case sensitive case",
args: args{
posixPath: "/c/Foo/bAr",
},
wantWindowsPath: `C:\Foo/bAr\`,
},
{
name: "special char case",
args: args{
posixPath: "/c/ふー/バー",
},
wantWindowsPath: `C:\ふー\バー`,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if gotWindowsPath := posixToWindows(tt.args.posixPath); gotWindowsPath != tt.wantWindowsPath {
t.Errorf("posixToWindows() = %v, want %v", gotWindowsPath, tt.wantWindowsPath)
}
})
}
}
+5 -1
View File
@@ -29,11 +29,15 @@ func newFileIndexer(path, base string, visitors ...PathIndexVisitor) *fileIndexe
base: base,
tree: filetree.New(),
index: filetree.NewIndex(),
// Unlike the directory indexer there is no skipPathsByMountTypeAndName here: it exists
// to keep a directory walk out of /proc, /sys and similar, and to do so it reads the
// whole mount table on construction. A file source indexes exactly one file and its
// parent, so there is nothing to walk into, and the mount table read is paid for
// every file source with nothing in return.
pathIndexVisitors: append(
[]PathIndexVisitor{
requireFileInfo,
disallowByFileType,
skipPathsByMountTypeAndName(path),
},
visitors...,
),
@@ -0,0 +1,51 @@
package fileresolver
import (
"os"
"path/filepath"
"testing"
"github.com/moby/sys/mountinfo"
)
// BenchmarkNewFromFile is the cost of building a resolver for a single-file source, which is
// paid once per file by callers that catalog files individually. The default case is
// NewFromFile as shipped. The with-mount-skipper case attaches skipPathsByMountTypeAndName
// the way the file indexer used to, so the difference is what reading the mount table for
// every file source cost. That cost scales with the mount table, so its size is reported as
// the mounts metric: expect a bigger gap on a container host than on a laptop.
//
// go test ./syft/internal/fileresolver/ -run '^$' -bench NewFromFile -benchmem -count 6 | tee bench.txt
// benchstat -col /variant bench.txt
func BenchmarkNewFromFile(b *testing.B) {
path := filepath.Join(b.TempDir(), "file.txt")
if err := os.WriteFile(path, []byte("hello\n"), 0o644); err != nil {
b.Fatal(err)
}
mounts, err := mountinfo.GetMounts(nil)
if err != nil {
b.Fatal(err)
}
b.Run("variant=default", func(b *testing.B) {
for b.Loop() {
if _, err := NewFromFile(path); err != nil {
b.Fatal(err)
}
}
b.ReportMetric(float64(len(mounts)), "mounts")
})
b.Run("variant=with-mount-skipper", func(b *testing.B) {
for b.Loop() {
r, err := newFromFileWithoutIndex(path, skipPathsByMountTypeAndName(path))
if err != nil {
b.Fatal(err)
}
if err := r.buildIndex(); err != nil {
b.Fatal(err)
}
}
b.ReportMetric(float64(len(mounts)), "mounts")
})
}
@@ -3,13 +3,11 @@ package fileresolver
import (
"io/fs"
"os"
"path"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/anchore/stereoscope/pkg/file"
)
// - Verify that both the parent and the path are indexed
@@ -38,11 +36,7 @@ func Test_index(t *testing.T) {
info, err := os.Stat(test.path)
assert.NoError(t, err)
// note: the index uses absolute paths, so assertions MUST keep this in mind
cwd, err := os.Getwd()
require.NoError(t, err)
p := file.Path(path.Join(cwd, test.path))
p := indexedPath(t, test.path)
assert.Equal(t, true, tree.HasPath(p))
exists, ref, err := tree.File(p)
assert.Equal(t, true, exists)
@@ -71,7 +65,9 @@ func Test_ignoresPathIfFiltered(t *testing.T) {
testPath := "testdata/system_paths/target/home/place"
cwd, cwdErr := os.Getwd()
require.NoError(t, cwdErr)
ignorePath := path.Join(cwd, testPath)
// filepath (not path): filters see the native absolute path (before posix normalization), so the
// expected value must be joined with the host separator to match on windows
ignorePath := filepath.Join(cwd, testPath)
filterFn := func(_, path string, _ os.FileInfo, _ error) error {
if path == ignorePath {
return ErrSkipPath
@@ -91,7 +87,8 @@ func Test_ignoresPathIfParentFiltered(t *testing.T) {
cwd, cwdErr := os.Getwd()
require.NoError(t, cwdErr)
ignorePath := path.Join(cwd, parentPath)
// filepath (not path): see Test_ignoresPathIfFiltered
ignorePath := filepath.Join(cwd, parentPath)
filterFn := func(_, path string, _ os.FileInfo, _ error) error {
if path == ignorePath {
return fs.SkipDir
@@ -6,6 +6,7 @@ import (
"fmt"
"io"
"os"
"path/filepath"
stereoscopeFile "github.com/anchore/stereoscope/pkg/file"
"github.com/anchore/stereoscope/pkg/filetree"
@@ -34,8 +35,17 @@ func nativeOSFileOpener(ref stereoscopeFile.Reference) (io.ReadCloser, error) {
return stereoscopeFile.NewLazyReadCloser(filePath), nil
}
// requestPath converts a user (chroot) path into the path the file tree is keyed on. The tree is always posix
// (windows paths are volume-encoded, e.g. /c/some/path), while ToNativePath yields a native host path.
func (r *FiletreeResolver) requestPath(userPath string) (string, error) {
return r.Chroot.ToNativePath(userPath)
nativePath, err := r.Chroot.ToNativePath(userPath)
if err != nil {
return "", err
}
if windows.HostRunningOnWindows() {
return windows.ToPosix(nativePath), nil
}
return nativePath, nil
}
// responsePath takes a path from the underlying fs domain and converts it to a path that is relative to the root of the file resolver.
@@ -85,10 +95,6 @@ func (r FiletreeResolver) FilesByPath(userPaths ...string) ([]file.Location, err
continue
}
if windows.HostRunningOnWindows() {
userStrPath = windows.ToPosix(userStrPath)
}
if ref.HasReference() {
references = append(references,
file.NewVirtualLocationFromDirectory(
@@ -103,8 +109,19 @@ func (r FiletreeResolver) FilesByPath(userPaths ...string) ([]file.Location, err
return references, nil
}
// requestGlob is the glob equivalent of requestPath.
func (r FiletreeResolver) requestGlob(pattern string) (string, error) {
return r.Chroot.ToNativeGlob(pattern)
glob, err := r.Chroot.ToNativeGlob(pattern)
if err != nil {
return "", err
}
// only a glob that was anchored to the root comes back native. Globs that start with a wildcard
// (e.g. **/foo) are returned untouched and are already posix, and ToPosix would root them at "/".
// This uses filepath (not path) since we are asking whether this is a native absolute path.
if windows.HostRunningOnWindows() && filepath.IsAbs(glob) {
return windows.ToPosix(glob), nil
}
return glob, nil
}
// FilesByGlob returns all file.References that match the given path glob pattern from any layer in the image.
+24 -7
View File
@@ -22,6 +22,15 @@ func SkipWhitespace(data []byte, i *int) {
}
}
// MaxDepth bounds nesting in the recursive hand-written parsers. Real files nest a handful of levels; without a
// bound, deep input overflows the goroutine stack, which is fatal rather than a recoverable panic.
const MaxDepth = 1000
// errorContext is how many bytes of each echoed line PrintError keeps around the error column, so an error in a
// huge single-line input stays small (and does not copy the file into the SBOM as an unknown).
const errorContext = 80
// PrintError describes where in data offset i is, echoing up to two lines of context clipped around the column.
func PrintError(data []byte, i int) string {
line := 1
char := 1
@@ -57,15 +66,23 @@ func PrintError(data []byte, i int) string {
sep := ": "
lines := ""
if len(prev) > 1 {
lines += fmt.Sprintf("%s%s%s\n", l1, sep, prev[len(prev)-2])
}
if len(prev) > 0 {
lines += fmt.Sprintf("%s%s%s\n", l2, sep, prev[len(prev)-1])
start := max(0, char-1-errorContext/2)
clip := func(s string) string {
if start >= len(s) {
return ""
}
return s[start:min(len(s), start+errorContext)]
}
pointer := strings.Repeat(" ", len(l2)+len(sep)+char-1) + "^"
lines := ""
if len(prev) > 1 {
lines += fmt.Sprintf("%s%s%s\n", l1, sep, clip(prev[len(prev)-2]))
}
if len(prev) > 0 {
lines += fmt.Sprintf("%s%s%s\n", l2, sep, clip(prev[len(prev)-1]))
}
pointer := strings.Repeat(" ", len(l2)+len(sep)+char-1-start) + "^"
return fmt.Sprintf("line: %v, char: %v\n%s%s", line, char, lines, pointer)
}
+8
View File
@@ -35,7 +35,15 @@ func ToPosix(windowsPath string) (posixPath string) {
func FromPosix(posixPath string) (windowsPath string) {
// decode the volume (e.g. /c/<path> --> C:\\) - There should always be a volume name.
// The volume may be a UNC path (e.g. /\\localhost\C$\ --> \\localhost\C$\)
if !strings.HasPrefix(posixPath, "/") {
// not a volume-encoded posix path (e.g. relative or already native), so there is nothing to decode
return posixPath
}
pathFields := strings.Split(posixPath, "/")
if len(pathFields) < 2 {
// no separator means no volume to decode
return filepath.Clean(posixPath)
}
rootPath := strings.ToUpper(pathFields[1])
volumeName := AppendRootTerminator(rootPath)
@@ -0,0 +1,35 @@
package windows
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestToPosix(t *testing.T) {
tests := map[string]string{
`C:\some\windows\place`: "/c/some/windows/place",
`C:\\some\\windows\\place`: "/c/some/windows/place",
`C:/foo/bar`: "/c/foo/bar",
`C:\foo/bar\`: "/c/foo/bar",
`C:\Foo/bAr\`: "/c/Foo/bAr",
`C:\ふー\バー`: "/c/ふー/バー",
}
for in, want := range tests {
assert.Equal(t, want, ToPosix(in), in)
}
}
func TestFromPosix(t *testing.T) {
tests := map[string]string{
"/c/some/windows/place": `C:\some\windows\place`,
"/c/Foo/bAr": `C:\Foo\bAr`,
"/c/ふー/バー": `C:\ふー\バー`,
// non volume-encoded input is left alone
"relative/path": "relative/path",
`D:\already\native`: `D:\already\native`,
}
for in, want := range tests {
assert.Equal(t, want, FromPosix(in), in)
}
}
+2 -2
View File
@@ -156,8 +156,8 @@ var busyboxVersionMatcher = regexp.MustCompile(`BusyBox v[\d.]+`)
func parseBusyBox(contents string) (*Release, error) {
matches := busyboxVersionMatcher.FindAllString(contents, -1)
for _, match := range matches {
parts := strings.Split(match, " ")
version := strings.ReplaceAll(parts[1], "v", "")
_, version, _ := strings.Cut(match, " ")
version = strings.ReplaceAll(version, "v", "")
return simpleRelease(match, "busybox", version, ""), nil
}
+12
View File
@@ -270,6 +270,10 @@ func (f apkField) apply(p *parsedData, ctx *apkFileParsingContext) {
ctx.indexOfLatestDirectory = len(ctx.files) - 1
case "M":
i := ctx.indexOfLatestDirectory
if i < 0 {
log.Debugf("APK field %q appears before any directory entry", f.name)
return
}
latest := ctx.files[i]
var ok bool
@@ -296,6 +300,10 @@ func (f apkField) apply(p *parsedData, ctx *apkFileParsingContext) {
ctx.indexOfLatestRegularFile = len(ctx.files) - 1
case "a":
i := ctx.indexOfLatestRegularFile
if i < 0 {
log.Debugf("APK field %q appears before any file entry", f.name)
return
}
latest := ctx.files[i]
var ok bool
@@ -309,6 +317,10 @@ func (f apkField) apply(p *parsedData, ctx *apkFileParsingContext) {
ctx.files[i] = latest
case "Z":
i := ctx.indexOfLatestRegularFile
if i < 0 {
log.Debugf("APK field %q appears before any file entry", f.name)
return
}
latest := ctx.files[i]
latest.Digest = processChecksum(f.value)
@@ -869,3 +869,29 @@ https://foo.them.org/alpine/v3.14/community`,
})
}
}
func TestParseApkDB_attributeBeforeItsEntry(t *testing.T) {
// ACL (M:, a:) and checksum (Z:) lines describe the most recent directory or file entry, and used to index
// files[-1] when no entry had been seen yet
tests := []struct {
name string
line string
}{
{name: "directory ACL before any directory", line: "M:0:0:755"},
{name: "file ACL before any file", line: "a:0:0:755"},
{name: "checksum before any file", line: "Z:Q1Kz6TXnPVJeDQUfO0IJVxD3HnYmU="},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
contents := strings.Join([]string{"P:foo", "V:1.0-r0", tt.line, ""}, "\n")
reader := file.NewLocationReadCloser(file.NewLocation("installed"), io.NopCloser(strings.NewReader(contents)))
pkgs, _, err := parseApkDB(context.Background(), nil, new(generic.Environment), reader)
require.NoError(t, err)
require.Len(t, pkgs, 1)
assert.Equal(t, "foo", pkgs[0].Name)
assert.Empty(t, pkgs[0].Metadata.(pkg.ApkDBEntry).Files, "an orphaned attribute line should not invent a file")
})
}
}
+1 -6
View File
@@ -148,12 +148,7 @@ func parseAlpmDBEntry(reader io.Reader) (*parsedData, error) {
}
func newScanner(reader io.Reader) *bufio.Scanner {
// This is taken from the apk parser
// https://github.com/anchore/syft/blob/v0.47.0/syft/pkg/cataloger/apkdb/parse_apk_db.go#L37
const maxScannerCapacity = 1024 * 1024
bufScan := make([]byte, maxScannerCapacity)
scanner := bufio.NewScanner(reader)
scanner.Buffer(bufScan, maxScannerCapacity)
scanner := internal.NewLineScanner(reader)
onDoubleLF := func(data []byte, atEOF bool) (advance int, token []byte, err error) {
for i := range data {
if i > 0 && data[i-1] == '\n' && data[i] == '\n' {
+38 -19
View File
@@ -492,12 +492,16 @@ func Test_parseMtree_malformedInput(t *testing.T) {
// bytes still costs multiples of its own size in heap. Asserting the error would keep passing if the
// line cap were removed, because the byte cap would still reject a large enough input eventually.
func Test_parseMtree_lineCapBoundsHeap(t *testing.T) {
// newline-only lines are the worst case: minimum bytes per entry, so the most entries per byte
const payloadSize = 16 * intFile.MB
spec := bytes.Repeat([]byte{'\n'}, payloadSize)
// the property is linear in the cap, so it is measured at a small cap and scaled to the shipped
// one. Running it at 300k lines means an uncapped parse costing ~16GB, which under -race is most
// of a CI minute for a test with no concurrency in it.
const (
lineCap = 4096
payloadSize = 64 * lineCap
)
// well inside the byte cap, so only the line cap can stop this
require.Less(t, int64(payloadSize), int64(maxMtreeSize))
// newline-only lines are the worst case: minimum bytes per entry, so the most entries per byte
spec := bytes.Repeat([]byte{'\n'}, payloadSize)
unbounded := testutils.MeasureAlloc(t, func() {
// the same listing with the line cap lifted, to prove the fixture is really a bomb
@@ -507,19 +511,29 @@ func Test_parseMtree_lineCapBoundsHeap(t *testing.T) {
require.Greater(t, unbounded, uint64(payloadSize),
"fixture did not actually cost more than its own size; it is no longer a bomb")
bounded := testutils.MeasureAlloc(t, func() {
_, err := parseMtreeWithLimits(gzipOf(t, spec), maxMtreeSize, maxMtreeLines)
require.ErrorIs(t, err, errTooManyMtreeLines)
})
boundedAt := func(limit int) uint64 {
gz := gzipOf(t, spec)
return testutils.MeasureAlloc(t, func() {
_, err := parseMtreeWithLimits(gz, maxMtreeSize, limit)
require.ErrorIs(t, err, errTooManyMtreeLines)
})
}
bounded := boundedAt(lineCap)
t.Logf("unbounded allocated %d bytes, bounded allocated %d bytes", unbounded, bounded)
// the marginal cost between two caps, so fixed setup (gzip reader, parser) does not get scaled
perLine := (boundedAt(2*lineCap) - bounded) / lineCap
atShippedCap := perLine * maxMtreeLines
t.Logf("unbounded allocated %d bytes, bounded allocated %d bytes (%d per line, ~%d at the shipped cap)",
unbounded, bounded, perLine, atShippedCap)
// measured at ~238MB against ~16.5GB uncapped, so the cap is doing its job, but note what it
// actually buys: 300k entries still cost a few hundred MB, and that is per concurrent cataloger.
// The budget is set to catch the cap being removed or raised by an order of magnitude, not to pin
// the exact figure.
assert.Less(t, bounded, uint64(512*intFile.MB),
assert.Less(t, bounded, unbounded/10,
"the line cap has to stop the parse before it builds an entry per line")
// measured at ~238MB for the shipped 300k lines, so the cap is doing its job, but note what it
// actually buys: that is still a few hundred MB, per concurrent cataloger. The budget is set to
// catch the cap being raised by an order of magnitude, not to pin the exact figure.
assert.Less(t, atShippedCap, uint64(512*intFile.MB),
"the shipped line cap allows more heap than budgeted")
}
// Test_parseMtree_sizeCapBoundsHeap is the byte-cap counterpart to the line-cap budget above.
@@ -532,9 +546,12 @@ func Test_parseMtree_lineCapBoundsHeap(t *testing.T) {
// be ordinary for the byte cap to be the thing under test, and there have to be few enough of them
// that the line cap does not fire first.
func Test_parseMtree_sizeCapBoundsHeap(t *testing.T) {
// same reasoning as the line cap: measured at a small byte cap, since the shipped 64MB cap means
// gzipping and parsing ~200MB of fixture, which is slow under -race and proves nothing more.
const (
sizeCap = intFile.MB
lineLen = 1024
numLines = 96 * 1024 // ~96MB across ~98k lines: past the 64MB cap, well under the 300k line cap
numLines = 2 * sizeCap / lineLen // twice the cap, and far under the line cap
)
var raw bytes.Buffer
@@ -542,7 +559,7 @@ func Test_parseMtree_sizeCapBoundsHeap(t *testing.T) {
for range numLines {
raw.Write(line)
}
require.Greater(t, int64(raw.Len()), int64(maxMtreeSize), "fixture must exceed the byte cap")
require.Greater(t, int64(raw.Len()), int64(sizeCap), "fixture must exceed the byte cap")
require.Less(t, numLines, maxMtreeLines, "and must not trip the line cap first")
unbounded := testutils.MeasureAlloc(t, func() {
@@ -554,11 +571,13 @@ func Test_parseMtree_sizeCapBoundsHeap(t *testing.T) {
"fixture did not actually cost more than its own size; it is no longer a bomb")
bounded := testutils.MeasureAlloc(t, func() {
_, err := parseMtreeWithLimits(gzipOf(t, raw.Bytes()), maxMtreeSize, maxMtreeLines)
_, err := parseMtreeWithLimits(gzipOf(t, raw.Bytes()), sizeCap, maxMtreeLines)
require.ErrorIs(t, err, errMtreeTooLarge)
})
t.Logf("unbounded allocated %d bytes, bounded allocated %d bytes", unbounded, bounded)
assert.Less(t, bounded, uint64(512*intFile.MB),
// the old budget was 512MB against the 64MB cap, so 8x the cap
assert.Less(t, bounded, uint64(8*sizeCap),
"the size cap has to stop the read at the cap, not after the member is drained")
}
@@ -407,6 +407,30 @@ func Test_Cataloger_PositiveCases(t *testing.T) {
Metadata: metadata("traefik-binary"),
},
},
{
// regression test: big-endian architectures (e.g. s390x) previously matched an unrelated
// single-NUL-terminated numeric string elsewhere in the binary instead of the real version
logicalFixture: "traefik/3.6.8/linux-s390x",
expected: pkg.Package{
Name: "traefik",
Version: "3.6.8",
Type: "binary",
PURL: "pkg:generic/traefik@3.6.8",
Locations: locations("traefik"),
Metadata: metadata("traefik-binary"),
},
},
{
logicalFixture: "traefik/2.11.1/linux-arm",
expected: pkg.Package{
Name: "traefik",
Version: "2.11.1",
Type: "binary",
PURL: "pkg:generic/traefik@2.11.1",
Locations: locations("traefik"),
Metadata: metadata("traefik-binary"),
},
},
{
logicalFixture: "memcached/1.6.18/linux-amd64",
expected: pkg.Package{
+9 -2
View File
@@ -284,11 +284,18 @@ func DefaultClassifiers() []binutils.Classifier {
{
Class: "traefik-binary",
FileGlob: "**/traefik",
EvidenceMatcher: m.FileContentsVersionMatcher(
EvidenceMatcher: binutils.MatchAny(
// on some architectures (e.g. s390x) the real version string isn't NUL-prefixed
// and is instead terminated by two NUL bytes; checking this pattern first avoids
// matching an unrelated single-NUL-terminated numeric string elsewhere in the binary
m.FileContentsVersionMatcher(
`(?m)(?P<version>[0-9]+\.[0-9]+\.[0-9]+(-alpha[0-9]|-beta[0-9]|-rc[0-9])?)\x00\x00`),
// [NUL]v1.7.34[NUL]
// [NUL]2.9.6[NUL]
// 3.0.4[NUL]
`(?m)(\x00v?|\x{FFFD}.?)(?P<version>[0-9]+\.[0-9]+\.[0-9]+(-alpha[0-9]|-beta[0-9]|-rc[0-9])?)\x00`),
m.FileContentsVersionMatcher(
`(?m)(\x00v?|\x{FFFD}.?)(?P<version>[0-9]+\.[0-9]+\.[0-9]+(-alpha[0-9]|-beta[0-9]|-rc[0-9])?)\x00`),
),
Package: "traefik",
PURL: mustPURL("pkg:generic/traefik@version"),
CPEs: singleCPE("cpe:2.3:a:traefik:traefik:*:*:*:*:*:*:*:*", cpe.NVDDictionaryLookupSource),
@@ -102,10 +102,10 @@ func (c Application) Validate() error {
}
func (c Application) GetBinaryFromImage(name, version string) *BinaryFromImage {
if strings.Contains(name, "@") && version == "" {
parts := strings.Split(name, "@")
name = parts[0]
version = parts[1]
if version == "" {
if n, v, ok := strings.Cut(name, "@"); ok {
name, version = n, v
}
}
for _, entry := range c.FromImages {
if entry.Name() == name && entry.Version == version {
+5 -2
View File
@@ -1,7 +1,6 @@
package debian
import (
"bufio"
"io"
"regexp"
"sort"
@@ -10,6 +9,7 @@ import (
"github.com/scylladb/go-set/strset"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/log"
)
// For more information see: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/#license-syntax
@@ -34,7 +34,7 @@ const (
func parseLicensesFromCopyright(reader io.Reader) []string {
findings := strset.New()
scanner := bufio.NewScanner(reader)
scanner := internal.NewLineScanner(reader)
// per the DEP-5 spec, machine-readable copyright files MUST have a
// Format field whose value is a URI for the specification. Only files
@@ -69,6 +69,9 @@ func parseLicensesFromCopyright(reader io.Reader) []string {
findings.Add(found)
}
}
if err := scanner.Err(); err != nil {
log.WithFields("error", err).Debug("failed to fully read debian copyright file")
}
if !formatVerified {
return nil
@@ -373,9 +373,11 @@ func Test_processDataTar_capsCopyrightFiles(t *testing.T) {
// than after the fact. The existing bounds test uses a payload small enough that buffering the whole
// thing would pass anyway; this one is large enough that the difference shows up as bytes.
func Test_decompressionStream_failsEarlyInsteadOfBuffering(t *testing.T) {
const expanded = 512 * intFile.MB
// 4x the 8MB budget below is plenty to tell bounded from unbounded. Going bigger only makes the
// fixture slower to build and inflate, which is most of this test's time under -race.
const expanded = 32 * intFile.MB
// well under 1% of input expanding to 512MB, so heap is what separates a streaming bound from a
// well under 1% of input expanding to 32MB, so heap is what separates a streaming bound from a
// bound checked after the read completes
var gz bytes.Buffer
gw := gzip.NewWriter(&gz)
+25 -6
View File
@@ -15,6 +15,10 @@ type erlangNode struct {
var errSkipComments = errors.New("")
var errUnexpectedEnd = errors.New("unexpected end of input")
var errTooDeep = fmt.Errorf("nesting deeper than %d levels", parsing.MaxDepth)
func (e erlangNode) Slice() []erlangNode {
out, ok := e.value.([]erlangNode)
if ok {
@@ -79,7 +83,7 @@ func parseErlang(reader io.Reader) (erlangNode, error) {
}
func parseErlangBlock(data []byte, i *int) (erlangNode, error) {
block, err := parseErlangNode(data, i)
block, err := parseErlangNode(data, i, 0)
if err != nil {
return node(nil), err
}
@@ -89,13 +93,19 @@ func parseErlangBlock(data []byte, i *int) (erlangNode, error) {
return block, nil
}
func parseErlangNode(data []byte, i *int) (erlangNode, error) {
func parseErlangNode(data []byte, i *int, depth int) (erlangNode, error) {
parsing.SkipWhitespace(data, i)
if *i >= len(data) {
return node(nil), errUnexpectedEnd
}
c := data[*i]
switch c {
case '[', '{':
offset := *i + 1
parsing.SkipWhitespace(data, &offset)
if offset >= len(data) {
return node(nil), errUnexpectedEnd
}
c2 := data[offset]
// Add support for empty lists
@@ -104,7 +114,7 @@ func parseErlangNode(data []byte, i *int) (erlangNode, error) {
return node(nil), nil
}
return parseErlangList(data, i)
return parseErlangList(data, i, depth+1)
case '"':
fallthrough
case '\'':
@@ -145,6 +155,9 @@ func parseErlangAngleString(data []byte, i *int) (erlangNode, error) {
}
func parseErlangString(data []byte, i *int) (erlangNode, error) {
if *i >= len(data) {
return node(nil), errUnexpectedEnd
}
delim := data[*i]
*i++
var buf bytes.Buffer
@@ -167,13 +180,16 @@ func parseErlangString(data []byte, i *int) (erlangNode, error) {
return node(nil), fmt.Errorf("unterminated string at %d", *i)
}
func parseErlangList(data []byte, i *int) (erlangNode, error) {
func parseErlangList(data []byte, i *int, depth int) (erlangNode, error) {
if depth > parsing.MaxDepth {
return node(nil), errTooDeep
}
*i++
out := erlangNode{
value: []erlangNode{},
}
for *i < len(data) {
item, err := parseErlangNode(data, i)
item, err := parseErlangNode(data, i, depth)
if err != nil {
if err == errSkipComments {
parsing.SkipWhitespace(data, i)
@@ -183,6 +199,9 @@ func parseErlangList(data []byte, i *int) (erlangNode, error) {
}
out.value = append(out.value.([]erlangNode), item)
parsing.SkipWhitespace(data, i)
if *i >= len(data) {
return node(nil), errUnexpectedEnd
}
c := data[*i]
switch c {
case ',':
@@ -210,7 +229,7 @@ func parseErlangComment(data []byte, i *int) {
// Rest of a line is a comment. Deals with CR, LF and CR/LF
if c == '\n' {
break
} else if c == '\r' && data[*i] == '\n' {
} else if c == '\r' && *i < len(data) && data[*i] == '\n' {
*i++
break
}
@@ -2,6 +2,7 @@ package erlang
import (
"bytes"
"strings"
"testing"
"github.com/stretchr/testify/assert"
@@ -128,3 +129,35 @@ func Test_parseErlangString_escapedByte(t *testing.T) {
require.NoError(t, err)
assert.Equal(t, `a"b`, got.String())
}
func TestParseErlang_deepNesting(t *testing.T) {
// 4MB of '[' overflowed the goroutine stack (fatal, not recoverable) before nesting was bounded
_, err := parseErlang(strings.NewReader(strings.Repeat("[a,", 4<<20)))
require.ErrorIs(t, err, errTooDeep)
// the error echoes a clipped window of the single-line input, not the whole thing
assert.Less(t, len(err.Error()), 1024)
}
func Test_parseErlang_truncated(t *testing.T) {
// each of these read past the end of the input before end-of-input was checked
tests := []struct {
name string
content string
wantErr require.ErrorAssertionFunc
}{
{name: "whitespace only", content: " ", wantErr: require.Error},
{name: "unterminated list", content: "[a", wantErr: require.Error},
{name: "unterminated empty list", content: "[ ", wantErr: require.Error},
{name: "lone binary string opener", content: "<", wantErr: require.Error},
{name: "comment ending in CR", content: "%\r", wantErr: require.NoError},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var err error
require.NotPanics(t, func() {
_, err = parseErlang(strings.NewReader(tt.content))
})
tt.wantErr(t, err)
})
}
}
@@ -3,6 +3,8 @@ package gentoo
import (
"testing"
"github.com/stretchr/testify/require"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/pkg"
@@ -170,3 +172,16 @@ func TestCataloger_Globs(t *testing.T) {
})
}
}
func TestPortageCataloger_ShortObjLine(t *testing.T) {
// obj lines missing their path or digest are skipped rather than indexed past their end
pkgtest.NewCatalogTester().
FromDirectory(t, "testdata/short-obj-line").
ExpectsAssertion(func(t *testing.T, pkgs []pkg.Package, _ []artifact.Relationship) {
require.Len(t, pkgs, 1)
m := pkgs[0].Metadata.(pkg.PortageEntry)
require.Len(t, m.Files, 1)
require.Equal(t, "/usr/bin/bar", m.Files[0].Path)
}).
TestCataloger(t, NewPortageCataloger())
}
+79 -72
View File
@@ -5,7 +5,6 @@ import (
"bytes"
"fmt"
"io"
"slices"
"strings"
"github.com/scylladb/go-set/strset"
@@ -35,7 +34,7 @@ import (
func extractLicenses(resolver file.Resolver, closestLocation *file.Location, reader io.Reader) (string, string) {
findings := strset.New()
contentsWriter := bytes.Buffer{}
scanner := bufio.NewScanner(io.TeeReader(reader, &contentsWriter))
scanner := internal.NewLineScanner(io.TeeReader(reader, &contentsWriter))
scanner.Split(bufio.ScanWords)
var (
mandatoryLicenses, conditionalLicenses, useflagLicenses []string
@@ -69,6 +68,13 @@ func extractLicenses(resolver file.Resolver, closestLocation *file.Location, rea
}
}
}
if err := scanner.Err(); err != nil {
fields := []any{"error", err}
if closestLocation != nil {
fields = append(fields, "path", closestLocation.RealPath)
}
log.WithFields(fields...).Debug("failed to fully read portage LICENSE")
}
var licenseGroups map[string][]string
if usesGroups {
@@ -80,31 +86,22 @@ func extractLicenses(resolver file.Resolver, closestLocation *file.Location, rea
findings.Add(conditionalLicenses...)
findings.Add(useflagLicenses...)
var mandatoryStatement, conditionalStatement string
return strings.TrimSpace(contentsWriter.String()), licenseExpression(mandatoryLicenses, conditionalLicenses)
}
// attempt to build valid SPDX license expression
if len(mandatoryLicenses) > 0 {
mandatoryStatement = strings.Join(mandatoryLicenses, " AND ")
// licenseExpression attempts to build a valid SPDX license expression
func licenseExpression(mandatoryLicenses, conditionalLicenses []string) string {
mandatoryStatement := strings.Join(mandatoryLicenses, " AND ")
conditionalStatement := strings.Join(conditionalLicenses, " OR ")
switch {
case mandatoryStatement != "" && conditionalStatement != "":
return mandatoryStatement + " AND (" + conditionalStatement + ")"
case mandatoryStatement != "":
return mandatoryStatement
default:
return conditionalStatement
}
if len(conditionalLicenses) > 0 {
conditionalStatement = strings.Join(conditionalLicenses, " OR ")
}
contents := strings.TrimSpace(contentsWriter.String())
if mandatoryStatement != "" && conditionalStatement != "" {
return contents, mandatoryStatement + " AND (" + conditionalStatement + ")"
}
if mandatoryStatement != "" {
return contents, mandatoryStatement
}
if conditionalStatement != "" {
return contents, conditionalStatement
}
return contents, ""
}
func readLicenseGroups(resolver file.Resolver, closestLocation *file.Location) map[string][]string {
@@ -161,10 +158,9 @@ func replaceLicenseGroups(licenses []string, groups map[string][]string) []strin
}
func parseLicenseGroups(reader io.Reader) (map[string][]string, error) {
result := make(map[string][]string)
rawGroups := make(map[string][]string)
scanner := bufio.NewScanner(reader)
scanner := internal.NewLineScanner(reader)
// first collect all raw groups
for scanner.Scan() {
@@ -191,58 +187,69 @@ func parseLicenseGroups(reader io.Reader) (map[string][]string, error) {
}
// next process each group to expand nested references
for groupName, licenses := range rawGroups {
expanded, err := expandLicenses(groupName, licenses, rawGroups, make(map[string]bool))
expanded := make(map[string][]string)
for groupName := range rawGroups {
if _, err := expandLicenses(groupName, rawGroups, expanded, make(map[string]bool)); err != nil {
return nil, err
}
}
return expanded, nil
}
// expandLicenses handles the recursive expansion of license groups. Each group is expanded once and memoized in
// 'expanded' (the file is untrusted, so re-expanding on every reference would be exponential), and 'visiting' holds
// the groups on the current path to detect cycles. We are always in terms of slices instead of sets to ensure
// original ordering is preserved.
func expandLicenses(currentGroup string, rawGroups, expanded map[string][]string, visiting map[string]bool) ([]string, error) {
if result, ok := expanded[currentGroup]; ok {
return result, nil
}
if visiting[currentGroup] {
return nil, fmt.Errorf("cycle detected in license group definitions for group: %s", currentGroup)
}
visiting[currentGroup] = true
defer delete(visiting, currentGroup)
result := make([]string, 0)
// sets keep dedup linear, since a single group line can hold hundreds of thousands of tokens
seen := strset.New()
seenRefs := strset.New()
for _, item := range rawGroups[currentGroup] {
refGroupName, isRef := strings.CutPrefix(item, "@")
if !isRef {
// this is a regular license
if !seen.Has(item) {
seen.Add(item)
result = append(result, item)
}
continue
}
// this is a reference to another group
if seenRefs.Has(refGroupName) {
continue
}
seenRefs.Add(refGroupName)
if _, exists := rawGroups[refGroupName]; !exists {
return nil, fmt.Errorf("referenced group not found: %s", refGroupName)
}
refLicenses, err := expandLicenses(refGroupName, rawGroups, expanded, visiting)
if err != nil {
return nil, err
}
result[groupName] = expanded
}
return result, nil
}
// expandLicenses handles the recursive expansion of license groups, 'visited' is used to detect cycles. We are always
// in terms of slices instead of sets to ensure original ordering is preserved.
func expandLicenses(currentGroup string, licenses []string, rawGroups map[string][]string, visited map[string]bool) ([]string, error) {
if visited[currentGroup] {
return nil, fmt.Errorf("cycle detected in license group definitions for group: %s", currentGroup)
}
visited[currentGroup] = true
result := make([]string, 0)
for _, item := range licenses {
if strings.HasPrefix(item, "@") {
// this is a reference to another group
refGroupName := item[1:] // remove '@' prefix
refLicenses, exists := rawGroups[refGroupName]
if !exists {
return nil, fmt.Errorf("referenced group not found: %s", refGroupName)
for _, license := range refLicenses {
if !seen.Has(license) {
seen.Add(license)
result = append(result, license)
}
newVisited := make(map[string]bool)
for k, v := range visited {
newVisited[k] = v
}
expanded, err := expandLicenses(refGroupName, refLicenses, rawGroups, newVisited)
if err != nil {
return nil, err
}
for _, license := range expanded {
if !slices.Contains(result, license) {
result = append(result, license)
}
}
} else if !slices.Contains(result, item) {
// ...this is a regular license
result = append(result, item)
}
}
expanded[currentGroup] = result
return result, nil
}
+29
View File
@@ -2,6 +2,7 @@ package gentoo
import (
"bytes"
"fmt"
"os"
"strings"
"testing"
@@ -181,3 +182,31 @@ func TestReplaceLicenseGroups(t *testing.T) {
})
}
}
func Test_extractLicenses_tokenTooLong(t *testing.T) {
// a nil location must not panic when the scan fails partway through
license := "MIT " + strings.Repeat("a", 2<<20)
_, expression := extractLicenses(nil, nil, strings.NewReader(license))
assert.Equal(t, "MIT", expression)
}
func TestParseLicenseGroups_hostileInput(t *testing.T) {
// a single wide group was quadratic to dedup, and a chain where each group references the next
// twice was exponential to expand; both must now finish quickly
var wide strings.Builder
wide.WriteString("WIDE")
for i := range 120_000 {
fmt.Fprintf(&wide, " L%d", i)
}
var chain strings.Builder
chain.WriteString("G0 MIT\n")
for i := 1; i <= 64; i++ {
fmt.Fprintf(&chain, "G%d @G%d @G%d\n", i, i-1, i-1)
}
groups, err := parseLicenseGroups(strings.NewReader(wide.String() + "\n" + chain.String()))
require.NoError(t, err)
assert.Len(t, groups["WIDE"], 120_000)
assert.Equal(t, []string{"MIT"}, groups["G64"])
}
@@ -1,7 +1,6 @@
package gentoo
import (
"bufio"
"context"
"fmt"
"path"
@@ -70,12 +69,13 @@ func addFiles(resolver file.Resolver, dbLocation file.Location, entry *pkg.Porta
}
defer internal.CloseAndLogError(contentsReader, dbLocation.RealPath)
scanner := bufio.NewScanner(contentsReader)
scanner := internal.NewLineScanner(contentsReader)
for scanner.Scan() {
line := strings.Trim(scanner.Text(), "\n")
fields := strings.Split(line, " ")
if fields[0] == "obj" {
// an obj line is "obj <path> <md5> <mtime>"
if len(fields) >= 3 && fields[0] == "obj" {
record := pkg.PortageFileRecord{
Path: fields[1],
}
@@ -86,6 +86,9 @@ func addFiles(resolver file.Resolver, dbLocation file.Location, entry *pkg.Porta
entry.Files = append(entry.Files, record)
}
}
if err := scanner.Err(); err != nil {
log.WithFields("path", dbLocation.RealPath, "error", err).Debug("failed to fully read portage contents")
}
}
func addLicenses(ctx context.Context, resolver file.Resolver, dbLocation file.Location, entry *pkg.PortageEntry) (pkg.LicenseSet, []file.Location) {
@@ -127,7 +130,7 @@ func addSize(resolver file.Resolver, dbLocation file.Location, entry *pkg.Portag
}
defer internal.CloseAndLogError(sizeReader, location.RealPath)
scanner := bufio.NewScanner(sizeReader)
scanner := internal.NewLineScanner(sizeReader)
for scanner.Scan() {
line := strings.Trim(scanner.Text(), "\n")
size, err := strconv.Atoi(line)
@@ -135,6 +138,9 @@ func addSize(resolver file.Resolver, dbLocation file.Location, entry *pkg.Portag
entry.InstalledSize = size
}
}
if err := scanner.Err(); err != nil {
log.WithFields("path", location.RealPath, "error", err).Debug("failed to fully read portage SIZE")
}
return []file.Location{location.WithAnnotation(pkg.EvidenceAnnotationKey, pkg.SupportingEvidenceAnnotation)}
}
@@ -0,0 +1,4 @@
dir /usr
obj
obj /usr/bin/foo
obj /usr/bin/bar abc123 1600000000
+13 -12
View File
@@ -59,9 +59,10 @@ func Test_PackageCataloger_Binary(t *testing.T) {
{
name: "upx compressed binary",
fixture: "image-small-upx",
// v1.0.11, not v1.0.0: see the fixture's Dockerfile
wantErr: require.NoError,
expectedPkgs: []string{
"anchore.io/not/real @ v1.0.0 (/run-me)",
"anchore.io/not/real @ v1.0.11 (/run-me)",
"github.com/andybalholm/brotli @ v1.1.1 (/run-me)",
"github.com/dsnet/compress @ v0.0.2-0.20210315054119-f66993602bf5 (/run-me)",
"github.com/golang/snappy @ v0.0.4 (/run-me)",
@@ -75,17 +76,17 @@ func Test_PackageCataloger_Binary(t *testing.T) {
"github.com/anchore/archiver/v3 @ v3.5.3-0.20241210171143-5b1d8d1c7c51 (/run-me)",
},
expectedRels: []string{
"github.com/andybalholm/brotli @ v1.1.1 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.0 (/run-me)",
"github.com/dsnet/compress @ v0.0.2-0.20210315054119-f66993602bf5 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.0 (/run-me)",
"github.com/golang/snappy @ v0.0.4 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.0 (/run-me)",
"github.com/klauspost/compress @ v1.17.11 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.0 (/run-me)",
"github.com/klauspost/pgzip @ v1.2.6 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.0 (/run-me)",
"github.com/anchore/archiver/v3 @ v3.5.3-0.20241210171143-5b1d8d1c7c51 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.0 (/run-me)",
"github.com/nwaples/rardecode @ v1.1.3 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.0 (/run-me)",
"github.com/pierrec/lz4/v4 @ v4.1.21 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.0 (/run-me)",
"github.com/ulikunitz/xz @ v0.5.12 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.0 (/run-me)",
"github.com/xi2/xz @ v0.0.0-20171230120015-48954b6210f8 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.0 (/run-me)",
"stdlib @ go1.23.2 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.0 (/run-me)",
"github.com/andybalholm/brotli @ v1.1.1 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.11 (/run-me)",
"github.com/dsnet/compress @ v0.0.2-0.20210315054119-f66993602bf5 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.11 (/run-me)",
"github.com/golang/snappy @ v0.0.4 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.11 (/run-me)",
"github.com/klauspost/compress @ v1.17.11 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.11 (/run-me)",
"github.com/klauspost/pgzip @ v1.2.6 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.11 (/run-me)",
"github.com/anchore/archiver/v3 @ v3.5.3-0.20241210171143-5b1d8d1c7c51 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.11 (/run-me)",
"github.com/nwaples/rardecode @ v1.1.3 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.11 (/run-me)",
"github.com/pierrec/lz4/v4 @ v4.1.21 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.11 (/run-me)",
"github.com/ulikunitz/xz @ v0.5.12 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.11 (/run-me)",
"github.com/xi2/xz @ v0.0.0-20171230120015-48954b6210f8 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.11 (/run-me)",
"stdlib @ go1.23.2 (/run-me) [dependency-of] anchore.io/not/real @ v1.0.11 (/run-me)",
},
},
{
+4 -2
View File
@@ -1,7 +1,6 @@
package golang
import (
"bufio"
"context"
"fmt"
"go/build"
@@ -437,7 +436,7 @@ func parseGoSumFile(resolver file.Resolver, reader file.LocationReadCloser) (map
// github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
// github.com/BurntSushi/toml v0.4.1 h1:GaI7EiDXDRfa8VshkTj7Fym7ha+y8/XxIgD2okUIjLw=
// github.com/BurntSushi/toml v0.4.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ=
scanner := bufio.NewScanner(contents)
scanner := internal.NewLineScanner(contents)
// optionally, resize scanner's capacity for lines over 64K, see next example
for scanner.Scan() {
line := scanner.Text()
@@ -449,6 +448,9 @@ func parseGoSumFile(resolver file.Resolver, reader file.LocationReadCloser) (map
hash := parts[2]
out[nameVersion] = hash
}
if err := scanner.Err(); err != nil {
return out, fmt.Errorf("unable to read go.sum: %w", err)
}
return out, nil
}
@@ -20,19 +20,25 @@ import (
// Test_getBuildInfo_compressedSectionBomb covers the reason readBuildInfo exists: debug/buildinfo opens
// ELF files with debug/elf itself, and elf.NewFile expands the section-name string table as it parses,
// so an oversized compression header there is an unbounded allocation on a path elfutil.NewFile never
// sees. The fixture is a real zlib stream, so it delivers every byte its header promises.
// sees. The fixture is a real zlib stream, but it stops short of what its header promises: delivering
// the full 256MB only made it slower to build and inflate, mostly under -race.
func Test_getBuildInfo_compressedSectionBomb(t *testing.T) {
const declared = 256 << 20 // comfortably over elfutil's bound, small enough to allocate in a test
const (
declared = 256 << 20 // comfortably over elfutil's bound, which only reads the header
// debug/elf allocates the declared size up front today, so delivery barely matters. Twice the
// guarded budget keeps the fixture a bomb if it ever switches to reading in chunks.
delivered = 64 << 20
)
bomb := elfWithCompressedNameTable(t, declared)
t.Logf("%d byte fixture declares a %d byte section name table", len(bomb), declared)
bomb := elfWithCompressedNameTable(t, declared, delivered)
t.Logf("%d byte fixture declares a %d byte section name table and delivers %d", len(bomb), declared, delivered)
// the unguarded path is the thing being defended against: prove the fixture really is a bomb
unguarded := testutils.MeasureAlloc(t, func() {
_, err := buildinfo.Read(bytes.NewReader(bomb))
t.Logf("buildinfo.Read err: %v", err)
})
assert.Greater(t, unguarded, uint64(declared), "fixture did not actually deliver the declared bytes")
assert.Greater(t, unguarded, uint64(delivered), "fixture did not actually deliver its bytes")
guarded := testutils.MeasureAlloc(t, func() {
_, err := getBuildInfo(bytes.NewReader(bomb))
@@ -47,17 +53,22 @@ func Test_getBuildInfo_compressedSectionBomb(t *testing.T) {
// goversion opens the file itself and reads .symtab plus the string table it links, and those are
// expanded lazily, so the section-name table bound getBuildInfo applies never reaches them.
func Test_getCryptoInformation_compressedSymtabBomb(t *testing.T) {
const declared = 256 << 20 // comfortably over elfutil's bound, small enough to allocate in a test
const (
declared = 256 << 20 // comfortably over elfutil's bound, which only reads the header
// debug/elf allocates the declared size up front today, so delivery barely matters. Twice the
// guarded budget keeps the fixture a bomb if it ever switches to reading in chunks.
delivered = 64 << 20
)
bomb := elfWithCompressedSymtab(t, declared)
t.Logf("%d byte fixture declares a %d byte symbol table", len(bomb), declared)
bomb := elfWithCompressedSymtab(t, declared, delivered)
t.Logf("%d byte fixture declares a %d byte symbol table and delivers %d", len(bomb), declared, delivered)
// the unguarded path is the thing being defended against: prove the fixture really is a bomb
unguarded := testutils.MeasureAlloc(t, func() {
_, err := version.ReadExeFromReader(bytes.NewReader(bomb))
t.Logf("goversion err: %v", err)
})
assert.Greater(t, unguarded, uint64(declared), "fixture did not actually deliver the declared bytes")
assert.Greater(t, unguarded, uint64(delivered), "fixture did not actually deliver its bytes")
guarded := testutils.MeasureAlloc(t, func() {
_, err := getCryptoInformation(bytes.NewReader(bomb))
@@ -85,12 +96,12 @@ func Test_getCryptoInformation_passesThroughNonELF(t *testing.T) {
}
// elfWithCompressedSymtab builds a minimal ELF64 whose .symtab is SHF_COMPRESSED, declaring `declared`
// decompressed bytes and genuinely delivering them. The name table is left uncompressed so the file gets
// decompressed bytes and genuinely delivering `delivered` of them. The name table is left uncompressed so the file gets
// past the bound getBuildInfo already applies, which is the point: this is the section that bound misses.
func elfWithCompressedSymtab(t *testing.T, declared uint64) []byte {
func elfWithCompressedSymtab(t *testing.T, declared, delivered uint64) []byte {
t.Helper()
payload := make([]byte, declared)
payload := make([]byte, delivered)
var compressed bytes.Buffer
zw := zlib.NewWriter(&compressed)
_, err := zw.Write(payload)
@@ -150,12 +161,12 @@ func elfWithCompressedSymtab(t *testing.T, declared uint64) []byte {
}
// elfWithCompressedNameTable builds a minimal ELF64 whose only real section is a SHF_COMPRESSED .shstrtab
// declaring `declared` decompressed bytes and genuinely delivering them. Only the test that runs the
// declaring `declared` decompressed bytes and genuinely delivering `delivered` of them. Only the test that runs the
// unguarded path needs delivery; use elfDeclaringNameTable everywhere else, since building this one costs
// `declared` bytes of allocation plus a zlib compress of them.
func elfWithCompressedNameTable(t *testing.T, declared uint64) []byte {
// `delivered` bytes of allocation plus a zlib compress of them.
func elfWithCompressedNameTable(t *testing.T, declared, delivered uint64) []byte {
t.Helper()
return elfNameTableFixture(t, declared, true)
return elfNameTableFixture(t, declared, delivered)
}
// elfDeclaringNameTable declares `declared` bytes without delivering them. CheckSectionNameTable reads the
@@ -163,19 +174,15 @@ func elfWithCompressedNameTable(t *testing.T, declared uint64) []byte {
// bound itself never needs the stream to be real.
func elfDeclaringNameTable(t *testing.T, declared uint64) []byte {
t.Helper()
return elfNameTableFixture(t, declared, false)
return elfNameTableFixture(t, declared, 64)
}
func elfNameTableFixture(t *testing.T, declared uint64, deliver bool) []byte {
func elfNameTableFixture(t *testing.T, declared, delivered uint64) []byte {
t.Helper()
// the decompressed name table only has to start with the section names; the rest is padding that
// exists purely to make the declared size real
size := declared
if !deliver {
size = 64
}
payload := make([]byte, size)
payload := make([]byte, delivered)
copy(payload, "\x00.shstrtab\x00")
var compressed bytes.Buffer
@@ -9,7 +9,9 @@ COPY go.mod go.sum ./
RUN go mod download
COPY main.go main.go
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags "-X main.Version=1.0.0" -o run-me .
# 1.0.11 is chosen so UPX has to pad before its loader stub (1.0.0 needed none, which hid a bug).
# TestImageSmallUPXNeedsLoaderPadding checks the padding is still there.
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags "-X main.Version=1.0.11" -o run-me .
RUN upx --best --lzma --exact run-me
FROM scratch
+42 -18
View File
@@ -718,33 +718,57 @@ func TestDecompressUPX_TailExtentsArePlacedPastTheLoader(t *testing.T) {
// front of it. Every other crafted fixture here has l_lsize == 0, so without this skipLoader was only
// reachable through the Docker-backed fixture.
//
// UPX pads to a 4 byte boundary before it writes the stub and l_lsize counts from there, so the head
// extent here deliberately ends off that boundary: its end plus l_lsize lands inside the stub.
// UPX pads to a 4 byte boundary before it writes the stub and l_lsize counts from there, so each case
// ends the head extent the given number of bytes short of a boundary. The head is a stored block so its
// length alone decides where it ends; a compressed one ends wherever the encoder happens to stop.
//
// note: block 1 here is not an ELF, so there are no PT_LOAD offsets and the tail lands sequentially.
// The hole-filling path this feeds in production is covered by TestFirstHole and by the sparse
// fixtures in upx_bounds_regression_test.go.
const loaderSize = 64
head := bytes.Repeat([]byte("H"), 128)
tail := bytes.Repeat([]byte("T"), 32)
data := buildUPXHeaderWithLoader(4096, 4096, loaderSize)
data = append(data, blockFor(t, head)...)
require.NotZero(t, len(data)%4, "the head extent has to end off a 4 byte boundary for the padding to exist")
data = append(data, make([]byte, 4-len(data)%4)...) // the padding UPX writes in front of the stub
data = append(data, make([]byte, 12)...) // end marker: closes the first run of extents
data = append(data, make([]byte, loaderSize-12)...) // the loader stub the chain skips over
data = append(data, blockFor(t, tail)...)
data = append(data, make([]byte, 12)...) // end marker for the tail run
data = padTo(data, 1024)
tests := []struct {
name string
padding int
}{
{name: "no padding", padding: 0},
{name: "1 byte of padding", padding: 1},
{name: "2 bytes of padding", padding: 2},
{name: "3 bytes of padding", padding: 3},
}
out, err := unpack(t, data)
// 160 bytes of a declared 4096, short by construction like every other crafted fixture here
require.ErrorIs(t, err, errUPXPartial)
got := readAll(t, out)
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
data := buildUPXHeaderWithLoader(4096, 4096, loaderSize)
assert.Equal(t, head, got[:len(head)], "the head extent stays where it was placed")
assert.Contains(t, string(got), string(tail), "the extent behind the loader must be placed too")
// trim a 128 byte head so that it, plus its b_info, ends tt.padding bytes short of a boundary
headLen := 128 - (len(data)+12+128+tt.padding)%4
head := bytes.Repeat([]byte("H"), headLen)
stored := make([]byte, 12)
binary.LittleEndian.PutUint32(stored[0:4], uint32(headLen)) // sz_unc
binary.LittleEndian.PutUint32(stored[4:8], uint32(headLen)) // sz_cpr, equal for a stored block
stored[8] = upxMethodStored
data = append(data, stored...)
data = append(data, head...)
require.Equal(t, tt.padding, (4-len(data)%4)%4, "the head should end tt.padding bytes short of a boundary")
data = append(data, make([]byte, tt.padding)...) // the padding UPX writes in front of the stub
data = append(data, make([]byte, 12)...) // end marker: closes the first run of extents
data = append(data, make([]byte, loaderSize-12)...) // the loader stub the chain skips over
data = append(data, blockFor(t, tail)...)
data = append(data, make([]byte, 12)...) // end marker for the tail run
data = padTo(data, 1024)
out, err := unpack(t, data)
// a few hundred bytes of a declared 4096, short by construction like every other crafted fixture here
require.ErrorIs(t, err, errUPXPartial)
got := readAll(t, out)
assert.Equal(t, head, got[:len(head)], "the head extent stays where it was placed")
assert.Contains(t, string(got), string(tail), "the extent behind the loader must be placed too")
})
}
}
func TestDecompressUPX_UnsupportedMethodIsNotReportable(t *testing.T) {
+34
View File
@@ -2,10 +2,14 @@ package golang
import (
"bytes"
"io"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
stereoscopeFile "github.com/anchore/stereoscope/pkg/file"
"github.com/anchore/stereoscope/pkg/imagetest"
)
// every case here is one the magic scan must reject, asserted on parseUPXInfo's errNotUPX result.
@@ -104,3 +108,33 @@ func TestParseUPXInfo_ValidHeader(t *testing.T) {
assert.Equal(t, uint8(22), info.format)
assert.Equal(t, uint32(0x100000), info.originalSize)
}
// TestImageSmallUPXNeedsLoaderPadding checks the fixture rather than the code. UPX pads to a 4 byte
// boundary before its loader stub, and the cataloger tests over image-small-upx only catch an unpacker that
// ignores that padding if there is some. A build that needs none passes either way, which is how the bug
// went unnoticed. The case without padding is covered by TestDecompressUPX_TailExtentsArePlacedPastTheLoader.
func TestImageSmallUPXNeedsLoaderPadding(t *testing.T) {
img := imagetest.GetFixtureImage(t, "docker-archive", "image-small-upx")
rc, err := img.OpenPathFromSquash(stereoscopeFile.Path("/run-me"))
require.NoError(t, err)
data, err := io.ReadAll(rc)
require.NoError(t, rc.Close())
require.NoError(t, err)
r := bytes.NewReader(data)
info, err := parseUPXInfo(r, int64(len(data)))
require.NoError(t, err)
require.NotZero(t, info.loaderSize, "real UPX output carries a loader stub")
// walk the first run of extents to the end marker in front of the stub
extentsEnd := info.firstBlockOff
for blockNum := 0; ; blockNum++ {
block, _, err := readChainBlock(r, extentsEnd, blockNum, info.inputLen)
require.NoError(t, err)
if block == nil {
break
}
extentsEnd = block.dataOffset + int64(block.compressedSize)
}
require.NotZero(t, extentsEnd%4, "the fixture no longer needs padding: change the version string in its Dockerfile")
}
@@ -1,11 +1,11 @@
package homebrew
import (
"bufio"
"context"
"path"
"strings"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/file"
@@ -46,7 +46,7 @@ func parseHomebrewFormula(ctx context.Context, resolver file.Resolver, _ *generi
func parseFormulaFile(reader file.LocationReadCloser) (*parsedHomebrewData, error) {
pd := parsedHomebrewData{}
scanner := bufio.NewScanner(reader)
scanner := internal.NewLineScanner(reader)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if strings.Contains(line, "class ") && strings.Contains(line, " < Formula") {
@@ -518,6 +518,11 @@ var defaultCandidateAdditions = buildCandidateLookup(
candidateKey{PkgName: "libxml2"},
candidateAddition{AdditionalVendors: []string{"xmlsoft"}},
},
{
pkg.ConanPkg,
candidateKey{PkgName: "expat"},
candidateAddition{AdditionalVendors: []string{"libexpat_project"}, AdditionalProducts: []string{"libexpat"}},
},
})
var defaultCandidateRemovals = buildCandidateRemovalLookup(
@@ -245,3 +245,10 @@ func Test_defaultCandidateAdditions_conan_libxml2(t *testing.T) {
vendors := findAdditionalVendors(defaultCandidateAdditions, pkg.ConanPkg, "libxml2", "")
assert.Contains(t, vendors, "xmlsoft")
}
func Test_defaultCandidateAdditions_conan_expat(t *testing.T) {
vendors := findAdditionalVendors(defaultCandidateAdditions, pkg.ConanPkg, "expat", "")
assert.Contains(t, vendors, "libexpat_project")
products := findAdditionalProducts(defaultCandidateAdditions, pkg.ConanPkg, "expat")
assert.Contains(t, products, "libexpat")
}
@@ -3,10 +3,12 @@ package cpegenerate
import (
"bufio"
"bytes"
"cmp"
_ "embed"
"encoding/json"
"fmt"
"regexp"
"slices"
"sort"
"strings"
"sync"
@@ -145,8 +147,14 @@ func FromPackageAttributes(p pkg.Package) []cpe.CPE {
return nil
}
vendors := candidateVendors(p)
products := candidateProducts(p)
// real CPE fields are short, and every byte of a field is paid once per generated CPE (key building and regex
// validation), so a hostile name or version is not worth generating candidates for at all.
if len(p.Name) > maxCandidateFieldLength || len(p.Version) > maxCandidateFieldLength {
return nil
}
vendors := boundCandidates(candidateVendors(p))
products := boundCandidates(candidateProducts(p))
targetSWs := candidateTargetSw(p)
if len(products) == 0 {
return nil
@@ -183,6 +191,32 @@ func FromPackageAttributes(p pkg.Package) []cpe.CPE {
return result
}
// maxCandidateFieldLength bounds the name, version, and each vendor and product candidate that reaches the
// cross-product in FromPackageAttributes. Real values are well under this.
const maxCandidateFieldLength = 512
// maxCandidates bounds how many vendor and product candidates each feed the cross-product in FromPackageAttributes,
// so a single package yields at most maxCandidates² CPEs. Some sources add a candidate per input element with no bound
// of their own (e.g. one vendor per java groupID segment, one product per .NET executable), and real packages produce
// a dozen or so candidates at most.
const maxCandidates = 64
// boundCandidates drops candidates longer than maxCandidateFieldLength and keeps at most maxCandidates of the rest.
// Candidates are ordered shortest first (then lexically) so that truncation is deterministic and keeps the short,
// generic values that tend to be real vendors and products.
func boundCandidates(candidates []string) []string {
candidates = slices.DeleteFunc(candidates, func(c string) bool {
return len(c) > maxCandidateFieldLength
})
if len(candidates) <= maxCandidates {
return candidates
}
slices.SortFunc(candidates, func(a, b string) int {
return cmp.Or(cmp.Compare(len(a), len(b)), strings.Compare(a, b))
})
return candidates[:maxCandidates]
}
func candidateTargetSw(p pkg.Package) []string {
switch p.Type {
case pkg.WordpressPluginPkg:
@@ -359,13 +393,19 @@ func addAllSubSelections(fields fieldCandidateSet) {
}
}
// maxSubSelections bounds how many prefixes generateSubSelections returns. Each prefix becomes a vendor candidate, so
// without a bound a name with N separators yields O(N) vendors of O(N) length each (quadratic in time and memory).
// Real package names rarely exceed a handful of segments.
const maxSubSelections = 10
// generateSubSelections attempts to split a field by hyphens and underscores and return a list of sensible sub-selections
// that can be used as product or vendor candidates. E.g. jenkins-ci-tools -> [jenkins-ci-tools, jenkins-ci, jenkins].
// that can be used as product or vendor candidates. E.g. jenkins-ci-tools -> [jenkins, jenkins-ci, jenkins-ci-tools].
// At most maxSubSelections prefixes are returned, shortest first.
func generateSubSelections(field string) (results []string) {
scanner := bufio.NewScanner(strings.NewReader(field))
scanner.Split(scanByHyphenOrUnderscore)
var lastToken uint8
for scanner.Scan() {
for len(results) < maxSubSelections && scanner.Scan() {
rawCandidate := scanner.Text()
if len(rawCandidate) == 0 {
break
@@ -892,6 +892,22 @@ func TestGeneratePackageCPEs(t *testing.T) {
"cpe:2.3:a:rust_package:rust_package:0.5.0:*:*:*:*:rust:*:*",
},
},
{
name: "conan expat: libexpat_project vendor and libexpat product",
p: pkg.Package{
Name: "expat",
Version: "2.5.0",
Type: pkg.ConanPkg,
},
expected: []string{
"cpe:2.3:a:expat:expat:2.5.0:*:*:*:*:*:*:*",
"cpe:2.3:a:expat:libexpat:2.5.0:*:*:*:*:*:*:*",
"cpe:2.3:a:libexpat:expat:2.5.0:*:*:*:*:*:*:*",
"cpe:2.3:a:libexpat:libexpat:2.5.0:*:*:*:*:*:*:*",
"cpe:2.3:a:libexpat_project:expat:2.5.0:*:*:*:*:*:*:*",
"cpe:2.3:a:libexpat_project:libexpat:2.5.0:*:*:*:*:*:*:*",
},
},
}
for _, test := range tests {
@@ -1121,6 +1137,11 @@ func Test_generateSubSelections(t *testing.T) {
field: "_",
expected: nil,
},
{
// capped at maxSubSelections, keeping the shortest prefixes (the full name is not returned)
field: "a_b_c_d_e_f_g_h_i_j_k_l",
expected: []string{"a", "a_b", "a_b_c", "a_b_c_d", "a_b_c_d_e", "a_b_c_d_e_f", "a_b_c_d_e_f_g", "a_b_c_d_e_f_g_h", "a_b_c_d_e_f_g_h_i", "a_b_c_d_e_f_g_h_i_j"},
},
}
for _, test := range tests {
t.Run(test.field, func(t *testing.T) {
@@ -1265,3 +1286,59 @@ func TestAddBinaryPackageDigitVariations(t *testing.T) {
})
}
}
func TestFromPackageAttributes_boundsSubSelectionBlowup(t *testing.T) {
// a name with many separators used to produce a vendor candidate per segment, each O(name length), which was
// quadratic in time and memory.
name := strings.Repeat("a_", 200) + "a"
p := pkg.Package{Name: name, Version: "1.0", Type: pkg.PythonPkg, Language: pkg.Python}
cpes := FromPackageAttributes(p)
assert.NotEmpty(t, cpes)
assert.LessOrEqual(t, len(cpes), 1000)
}
func TestFromPackageAttributes_boundsCandidateCrossProduct(t *testing.T) {
// every groupID segment becomes a vendor, and with no artifactID every segment past the second also becomes a
// product, so the vendor x product cross-product is quadratic in segment count without a bound.
var segments []string
for i := range 1000 {
segments = append(segments, fmt.Sprintf("s%d", i))
}
p := pkg.Package{
Name: "thing",
Version: "1.0",
Type: pkg.JavaPkg,
Language: pkg.Java,
Metadata: pkg.JavaArchive{
Manifest: &pkg.JavaManifest{
Main: pkg.KeyValues{{Key: "Bundle-SymbolicName", Value: "com.x." + strings.Join(segments, ".")}},
},
},
}
cpes := FromPackageAttributes(p)
assert.NotEmpty(t, cpes)
assert.LessOrEqual(t, len(cpes), maxCandidates*maxCandidates)
}
func TestFromPackageAttributes_skipsOversizedFields(t *testing.T) {
long := strings.Repeat("a", maxCandidateFieldLength+1)
assert.Empty(t, FromPackageAttributes(pkg.Package{Name: long, Version: "1.0", Type: pkg.PythonPkg}))
assert.Empty(t, FromPackageAttributes(pkg.Package{Name: "thing", Version: long, Type: pkg.PythonPkg}))
assert.NotEmpty(t, FromPackageAttributes(pkg.Package{Name: "thing", Version: "1.0", Type: pkg.PythonPkg}))
}
func Test_boundCandidates(t *testing.T) {
long := strings.Repeat("a", maxCandidateFieldLength+1)
assert.Equal(t, []string{"b", "a"}, boundCandidates([]string{"b", long, "a"}), "under the cap order is untouched")
var many []string
for i := range maxCandidates + 10 {
many = append(many, strings.Repeat("x", maxCandidates+10-i))
}
got := boundCandidates(many)
assert.Len(t, got, maxCandidates)
assert.Equal(t, "x", got[0], "shortest candidates are kept")
}
@@ -209,7 +209,7 @@ func read7BitEncodedInt(r io.Reader) (int, error) {
shift := 0
for {
var b [1]byte
if _, err := r.Read(b[:]); err != nil {
if _, err := io.ReadFull(r, b[:]); err != nil {
return 0, err
}
result |= int(b[0]&0x7F) << shift
@@ -1,10 +1,11 @@
package java
import (
"bufio"
"context"
"fmt"
"strings"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/pkg"
@@ -22,7 +23,7 @@ func parseGradleLockfile(_ context.Context, _ file.Resolver, _ *generic.Environm
var pkgs []pkg.Package
// Create a new scanner to read the file
scanner := bufio.NewScanner(reader)
scanner := internal.NewLineScanner(reader)
// Create slices to hold the dependencies and plugins
dependencies := []lockfileDependency{}
@@ -45,6 +46,7 @@ func parseGradleLockfile(_ context.Context, _ file.Resolver, _ *generic.Environm
dependencies = append(dependencies, dep)
}
}
scanErr := scanner.Err()
// map the dependencies
for _, dep := range dependencies {
@@ -72,5 +74,8 @@ func parseGradleLockfile(_ context.Context, _ file.Resolver, _ *generic.Environm
pkgs = append(pkgs, mappedPkg)
}
if scanErr != nil {
return pkgs, nil, fmt.Errorf("unable to read gradle lockfile: %w", scanErr)
}
return pkgs, nil, nil
}
@@ -1,8 +1,15 @@
package java
import (
"bufio"
"context"
"io"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
@@ -69,3 +76,14 @@ func Test_parserGradleLockfile(t *testing.T) {
})
}
}
func Test_parserGradleLockfile_lineTooLong(t *testing.T) {
// packages before an oversized line are kept, and the scan error is still reported
contents := "org.example:before:1.0=compileClasspath\n" + strings.Repeat("a", 2<<20) + "\norg.example:after:2.0=compileClasspath\n"
reader := file.NewLocationReadCloser(file.NewLocation("gradle.lockfile"), io.NopCloser(strings.NewReader(contents)))
pkgs, _, err := parseGradleLockfile(context.Background(), nil, nil, reader)
require.ErrorIs(t, err, bufio.ErrTooLong)
require.Len(t, pkgs, 1)
assert.Equal(t, "before", pkgs[0].Name)
}
@@ -1,12 +1,12 @@
package java
import (
"bufio"
"fmt"
"io"
"strings"
"unicode"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/pkg"
)
@@ -26,7 +26,7 @@ func parseJavaManifest(path string, reader io.Reader) (*pkg.JavaManifest, error)
}
var lastKey string
scanner := bufio.NewScanner(reader)
scanner := internal.NewLineScanner(reader)
for scanner.Scan() {
line := scanner.Text()
@@ -1,13 +1,13 @@
package java
import (
"bufio"
"fmt"
"io"
"strings"
"github.com/go-viper/mapstructure/v2"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/syft/pkg"
)
@@ -16,7 +16,7 @@ const pomPropertiesGlob = "**/*pom.properties"
func parsePomProperties(path string, reader io.Reader) (*pkg.JavaPomProperties, error) {
var props pkg.JavaPomProperties
propMap := make(map[string]string)
scanner := bufio.NewScanner(reader)
scanner := internal.NewLineScanner(reader)
for scanner.Scan() {
line := scanner.Text()
@@ -206,28 +206,16 @@ func parseYarnV1LockFile(reader io.ReadCloser) ([]yarnPackage, error) {
continue
}
if strings.HasPrefix(line, " ") && !strings.HasPrefix(line, " ") {
line = strings.Trim(line, " ")
array := strings.Split(line, " ")
switch array[0] {
case "version":
pkg.Version = strings.Trim(array[1], "\"")
case "resolved":
name, version, resolved := findResolvedPackageAndVersion(line)
if name != "" && version != "" && resolved != "" {
pkg.Name = name
pkg.Version = version
pkg.Resolved = resolved
} else {
pkg.Resolved = strings.Trim(array[1], "\"")
}
case "integrity":
pkg.Integrity = strings.Trim(array[1], "\"")
}
applyYarnV1Field(&pkg, strings.Trim(line, " "))
continue
}
if strings.HasPrefix(line, " ") {
line = strings.Trim(line, " ")
array := strings.Split(line, " ")
if len(array) < 2 {
// a dependency without a version constraint is malformed; skip it rather than guess
continue
}
dependencyName := strings.Trim(array[0], "\"")
dependencyVersion := strings.Trim(array[1], "\"")
dependencies[dependencyName] = dependencyVersion
@@ -244,6 +232,31 @@ func parseYarnV1LockFile(reader io.ReadCloser) ([]yarnPackage, error) {
return pkgs, nil
}
// applyYarnV1Field sets the package field described by a single (already trimmed) yarn v1 field line.
func applyYarnV1Field(pkg *yarnPackage, line string) {
array := strings.Split(line, " ")
// a field line may be missing its value in a malformed lockfile
var value string
if len(array) > 1 {
value = strings.Trim(array[1], "\"")
}
switch array[0] {
case "version":
pkg.Version = value
case "resolved":
name, version, resolved := findResolvedPackageAndVersion(line)
if name != "" && version != "" && resolved != "" {
pkg.Name = name
pkg.Version = version
pkg.Resolved = resolved
} else {
pkg.Resolved = value
}
case "integrity":
pkg.Integrity = value
}
}
func parseYarnLockYaml(reader io.ReadCloser) ([]yarnPackage, error) {
var lockfile = map[string]yarnV2PackageEntry{}
if err := yaml.NewDecoder(reader, yaml.AllowDuplicateMapKey()).Decode(&lockfile); err != nil {
@@ -6,9 +6,11 @@ import (
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/file"
@@ -1091,3 +1093,35 @@ func setupYarnRegistry() (mux *http.ServeMux, serverURL string, teardown func())
return mux, server.URL, server.Close
}
func TestParseYarnV1LockFile_missingValues(t *testing.T) {
tests := []struct {
name string
content string
want []yarnPackage
}{
{
// a dependency line without a constraint is dropped rather than indexed past its end
name: "dependency without version",
content: "# yarn lockfile v1\n\na@1:\n version \"1\"\n dependencies:\n b\n\n",
want: []yarnPackage{{Name: "a", Version: "1", Dependencies: map[string]string{}}},
},
{
// a field line without a value leaves the field empty
name: "field without value",
content: "# yarn lockfile v1\n\na@1:\n version\n\n",
want: []yarnPackage{{Name: "a", Dependencies: map[string]string{}}},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var got []yarnPackage
var err error
require.NotPanics(t, func() {
got, err = parseYarnV1LockFile(io.NopCloser(strings.NewReader(tt.content)))
})
require.NoError(t, err)
assert.Equal(t, tt.want, got)
})
}
}
+76 -42
View File
@@ -2,6 +2,7 @@ package lua
import (
"bytes"
"errors"
"fmt"
"io"
@@ -33,6 +34,10 @@ func (r rockspecNode) String() string {
return ""
}
var errUnexpectedEnd = errors.New("unexpected end of input")
var errTooDeep = fmt.Errorf("nesting deeper than %d levels", parsing.MaxDepth)
var noReturn = rockspec{
value: nil,
}
@@ -46,10 +51,11 @@ func parseRockspecData(reader io.Reader) (rockspec, error) {
i := 0
locals := make(map[string]string)
blocks, err := parseRockspecBlock(data, &i, locals)
blocks, err := parseRockspecBlock(data, &i, locals, 0)
if err != nil {
return noReturn, err
// decorate once here rather than at every nesting level, which copied the input once per level
return noReturn, fmt.Errorf("%w\n%s", err, parsing.PrintError(data, i))
}
return rockspec{
@@ -57,9 +63,12 @@ func parseRockspecData(reader io.Reader) (rockspec, error) {
}, nil
}
func parseRockspecBlock(data []byte, i *int, locals map[string]string) ([]rockspecNode, error) {
func parseRockspecBlock(data []byte, i *int, locals map[string]string, depth int) ([]rockspecNode, error) {
if depth > parsing.MaxDepth {
return nil, errTooDeep
}
var out []rockspecNode
var iterator func(data []byte, i *int, locals map[string]string) (*rockspecNode, error)
var iterator func(data []byte, i *int, locals map[string]string, depth int) (*rockspecNode, error)
parsing.SkipWhitespace(data, i)
@@ -92,9 +101,9 @@ func parseRockspecBlock(data []byte, i *int, locals map[string]string) ([]rocksp
}
for *i < len(data) {
item, err := iterator(data, i, locals)
item, err := iterator(data, i, locals, depth)
if err != nil {
return nil, fmt.Errorf("%w\n%s", err, parsing.PrintError(data, *i))
return nil, err
}
parsing.SkipWhitespace(data, i)
@@ -114,7 +123,7 @@ func parseRockspecBlock(data []byte, i *int, locals map[string]string) ([]rocksp
}
//nolint:funlen, gocognit
func parseRockspecNode(data []byte, i *int, locals map[string]string) (*rockspecNode, error) {
func parseRockspecNode(data []byte, i *int, locals map[string]string, depth int) (*rockspecNode, error) {
parsing.SkipWhitespace(data, i)
if *i >= len(data) {
@@ -212,7 +221,7 @@ func parseRockspecNode(data []byte, i *int, locals map[string]string) (*rockspec
*i = offset
parsing.SkipWhitespace(data, i)
obj, err := parseRockspecBlock(data, i, locals)
obj, err := parseRockspecBlock(data, i, locals, depth+1)
if err != nil {
return nil, err
@@ -247,6 +256,9 @@ func parseRockspecNode(data []byte, i *int, locals map[string]string) (*rockspec
}
value := str.String()
if *i >= len(data) {
return nil, errUnexpectedEnd
}
c = data[*i]
if c != ']' {
@@ -271,7 +283,7 @@ func parseRockspecNode(data []byte, i *int, locals map[string]string) (*rockspec
}, nil
}
func parseRockspecListItem(data []byte, i *int, locals map[string]string) (*rockspecNode, error) {
func parseRockspecListItem(data []byte, i *int, locals map[string]string, _ int) (*rockspecNode, error) {
parsing.SkipWhitespace(data, i)
if *i >= len(data) {
@@ -378,6 +390,9 @@ out:
if err != nil {
return "", err
}
if *i >= len(data) {
return "", errUnexpectedEnd
}
c = data[*i]
if c != ']' {
return "", fmt.Errorf("unterminated literal at %d", *i)
@@ -394,6 +409,9 @@ out:
}
func parseRockspecString(data []byte, i *int, _ map[string]string) (*rockspecNode, error) {
if *i >= len(data) {
return nil, errUnexpectedEnd
}
delim := data[*i]
var endDelim byte
switch delim {
@@ -434,6 +452,39 @@ func parseComment(data []byte, i *int) {
}
}
// parseLocalValue parses a single value on the right-hand side of a local assignment. Expressions
// resolve to an empty string, and bare references resolve through locals. The caller must ensure *i < len(data).
func parseLocalValue(data []byte, i *int, locals map[string]string) (string, error) {
c := data[*i]
switch c {
case '"', '\'':
value, err := parseRockspecString(data, i, locals)
if err != nil {
return "", err
}
return value.value.(string), nil
default:
ref, err := parseRockspecLiteral(data, i, locals)
if err != nil {
return "", err
}
// skip if it's an expression
skipWhitespaceNoNewLine(data, i)
if *i >= len(data) {
return "", errUnexpectedEnd
}
c := data[*i]
if c != '\n' && c != '\r' {
skipExpression(data, i)
return "", nil
}
return locals[ref], nil
}
}
//nolint:funlen
func parseLocal(data []byte, i *int, locals map[string]string) error {
keys := []string{}
@@ -459,6 +510,9 @@ keys:
keys = append(keys, key)
parsing.SkipWhitespace(data, i)
if *i >= len(data) {
return errUnexpectedEnd
}
c := data[*i]
@@ -477,42 +531,22 @@ keys:
values:
for {
skipWhitespaceNoNewLine(data, i)
c := data[*i]
switch c {
case '"', '\'':
value, err := parseRockspecString(data, i, locals)
if err != nil {
return err
}
values = append(values, value.value.(string))
default:
ref, err := parseRockspecLiteral(data, i, locals)
if err != nil {
return err
}
// Skip if it's an expression
skipWhitespaceNoNewLine(data, i)
c := data[*i]
var value string
if c != '\n' && c != '\r' {
skipExpression(data, i)
value = ""
} else {
value = locals[ref]
}
values = append(values, value)
if *i >= len(data) {
return errUnexpectedEnd
}
skipWhitespaceNoNewLine(data, i)
value, err := parseLocalValue(data, i, locals)
if err != nil {
return err
}
values = append(values, value)
c = data[*i]
skipWhitespaceNoNewLine(data, i)
if *i >= len(data) {
return errUnexpectedEnd
}
c := data[*i]
switch c {
case ',':
@@ -2,6 +2,7 @@ package lua
import (
"bytes"
"strings"
"testing"
"github.com/stretchr/testify/assert"
@@ -282,3 +283,35 @@ hello = foo.. `,
})
}
}
func TestParseRockspec_deepNesting(t *testing.T) {
// deep input overflowed the goroutine stack (fatal, not recoverable) before nesting was bounded, and each level
// used to append its own copy of the input to the error
_, err := parseRockspecData(strings.NewReader(strings.Repeat("a={", 3<<20)))
require.ErrorIs(t, err, errTooDeep)
assert.Less(t, len(err.Error()), 1024)
}
func Test_parseRockspecData_truncated(t *testing.T) {
// each of these ends mid-statement, and used to read past the end of the input
tests := []struct {
name string
content string
}{
{name: "unterminated long string value", content: "a=[[x]"},
{name: "unterminated index literal", content: `a["x"`},
{name: "local missing equals", content: "local a"},
{name: "local missing value", content: "local a="},
{name: "local reference at end of input", content: "local a=b"},
{name: "local string at end of input", content: `local a="x"`},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var err error
require.NotPanics(t, func() {
_, err = parseRockspecData(strings.NewReader(tt.content))
})
require.Error(t, err)
})
}
}
+1 -1
View File
@@ -95,7 +95,7 @@ func (c dbCataloger) selectDBParser(dbLocation file.Location, resolver file.Reso
scanner := bufio.NewScanner(schemaContents)
if !scanner.Scan() {
log.WithFields("path", loc.RealPath).Tracef("failed to read Nix database schema file, assuming %d", defaultSchema)
log.WithFields("path", loc.RealPath, "error", scanner.Err()).Tracef("failed to read Nix database schema file, assuming %d", defaultSchema)
return c.schemaProcessor[defaultSchema], 0
}
+400
View File
@@ -0,0 +1,400 @@
package cataloger_test
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"go/ast"
"go/parser"
"go/token"
"io/fs"
"os"
"path/filepath"
"regexp"
"runtime"
"runtime/debug"
"runtime/metrics"
"runtime/pprof"
"sort"
"strconv"
"strings"
"testing"
"time"
"github.com/bmatcuk/doublestar/v4"
gosync "github.com/anchore/go-sync"
"github.com/anchore/syft/internal/capabilities"
"github.com/anchore/syft/internal/sbomsync"
"github.com/anchore/syft/internal/task"
"github.com/anchore/syft/internal/unknown"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
"github.com/anchore/syft/syft/sbom"
"github.com/anchore/syft/syft/source"
"github.com/anchore/syft/syft/source/directorysource"
)
const (
// per-input limits. These are generous on purpose: the net is for inputs that take seconds or
// hundreds of MB, not for small inefficiencies.
fuzzTimeout = 10 * time.Second
// fuzzHangSlack is how long past fuzzTimeout a parser that ignores ctx gets before the process is taken down,
// so a ctx-aware parser always reports its timeout through the normal path first
fuzzHangSlack = 5 * time.Second
fuzzAllocBase = 256 << 20
fuzzAllocPerByte = 1024
fuzzMaxSeedBytes = 256 << 10
fuzzMaxSeedsPerFn = 8
)
type fuzzTarget struct {
cataloger string
glob string
path string // a concrete relative path that matches glob
dir string // the cataloger package directory, relative to this package, for seeds
task task.Task
}
func (ft fuzzTarget) String() string { return ft.cataloger + " " + ft.glob }
// FuzzGenericParsers feeds one input at a time to each glob-selected parser of every generic cataloger, by writing
// it to a path the parser's glob matches and running the cataloger task over that directory. Catalogers that are not
// generic, and processor panics (which the generic cataloger only logs), are not covered.
//
// # What fuzzing is
//
// A fuzz test is a test whose input is generated. Go's fuzzer starts from a set of example inputs (the seed corpus),
// mutates them (flipping bytes, splicing pieces together, inserting values) and keeps any mutation that reaches code
// it has not reached before, so over time it works its way into parser branches that hand-written tests never try.
// Each input is one parser and one file's worth of bytes. An input fails when the parser:
//
// - panics (including a panic the generic cataloger recovers and reports as an unknown)
// - takes longer than fuzzTimeout
// - allocates much more than the input size warrants (see fuzzAllocBase and fuzzAllocPerByte)
//
// A parser returning an error is fine: rejecting bad input is the correct behavior.
//
// # Plain go test (no fuzzing)
//
// Without -fuzz, go test does not generate anything. It runs every seed once, like a table test, and reports them
// as subtests named seed#0, seed#1, ... (go names seeds by position; the failure message says which parser it was).
// The seeds are each parser's own testdata files plus a few inputs built from the string literals in its package,
// which give the parser its own key names and prefixes to trip over. This is what runs in CI, and takes seconds.
// It also replays every saved failure in testdata/fuzz/FuzzGenericParsers/ (see below).
//
// # Running the fuzzer
//
// Fuzzing is only done on demand, locally. Pick one parser with SYFT_FUZZ_TARGET, a regexp over
// "<cataloger> <glob>" (cataloger names are listed by `syft cataloger list`), and give it a time budget:
//
// SYFT_FUZZ_TARGET='^apk-db-cataloger ' go test ./syft/pkg/cataloger -run '^$' -fuzz FuzzGenericParsers -fuzztime 60s
//
// -run '^$' skips the ordinary tests, and -fuzztime bounds the run (without it the fuzzer runs until stopped).
// Without SYFT_FUZZ_TARGET every parser shares one run, so each gets a small slice of the time. The fuzzer's
// "new interesting" inputs are kept in the go build cache, not the repo, so later runs pick up where earlier ones
// left off.
//
// # When the fuzzer finds a failure
//
// It shrinks the input to a small version that still fails, prints the failure, and writes it to
// testdata/fuzz/FuzzGenericParsers/<hash>. That file is two lines of text, the parser name and the input bytes, so
// it can be read in review. From then on plain go test replays it as the subtest FuzzGenericParsers/<hash>, and it
// can be run alone:
//
// go test ./syft/pkg/cataloger -run 'FuzzGenericParsers/<hash>'
//
// The lifecycle of a failure:
//
// 1. fix the parser so the input is rejected with an error (or parsed correctly) rather than panicking or hanging
// 2. commit the saved file in the same change as the fix, as the regression test for it; do not commit one without
// its fix, since it fails every test run until the fix lands
// 3. fuzz that parser again, since one fix often uncovers the next failure behind it
//
// Saved files are kept indefinitely, like any other regression test. Delete one only when it no longer means
// anything (for example, the parser it names was removed). A test case in the parser's own _test.go is an equally
// good home for the input, and reads better there.
func FuzzGenericParsers(f *testing.F) {
targets := fuzzTargets(f)
if len(targets) == 0 {
f.Skip("no targets selected")
}
// inputs name their target rather than index it, so a saved crasher keeps pointing at the same parser when
// the target list changes or is filtered
byName := map[string]fuzzTarget{}
for _, ft := range targets {
byName[ft.String()] = ft
for _, seed := range fuzzSeeds(ft) {
f.Add(ft.String(), seed)
}
}
f.Fuzz(func(t *testing.T, name string, data []byte) {
ft, ok := byName[name]
if !ok {
// while fuzzing, the mutator rewrites names too, and a filter deliberately drops targets. Otherwise this
// is a saved input whose target was renamed or removed, and skipping it would silently drop a regression.
if os.Getenv("SYFT_FUZZ_TARGET") == "" && flag.Lookup("test.fuzz").Value.String() == "" {
t.Fatalf("saved fuzz input names unknown target %q: update or remove it", name)
}
t.Skip("not a selected target")
}
runFuzzTarget(t, ft, data)
})
}
func runFuzzTarget(t *testing.T, ft fuzzTarget, data []byte) {
root := t.TempDir()
p := filepath.Join(root, filepath.FromSlash(ft.path))
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, data, 0o600); err != nil {
t.Fatal(err)
}
src, err := directorysource.NewFromPath(root)
if err != nil {
t.Fatal(err)
}
resolver, err := src.FileResolver(source.AllLayersScope)
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(pkgtest.Context(t), fuzzTimeout)
defer cancel()
var before, after runtime.MemStats
runtime.ReadMemStats(&before)
// the heap metric is process-wide and counts unswept garbage, so collect first and measure growth from there
heap := []metrics.Sample{{Name: "/memory/classes/heap/objects:bytes"}}
runtime.GC()
metrics.Read(heap)
heapBase := heap[0].Value.Uint64()
done := make(chan error, 1)
go func() {
defer func() {
if r := recover(); r != nil {
done <- fmt.Errorf("panic: %v\n%s", r, debug.Stack())
}
}()
done <- ft.task.Execute(ctx, resolver, sbomsync.NewBuilder(&sbom.SBOM{Artifacts: sbom.Artifacts{Packages: pkg.NewCollection()}}))
}()
limit := uint64(fuzzAllocBase + fuzzAllocPerByte*len(data))
tick := time.NewTicker(50 * time.Millisecond)
defer tick.Stop()
deadline := time.After(fuzzTimeout + fuzzHangSlack)
wait:
for {
select {
case err = <-done:
break wait
case <-tick.C:
// the after-the-fact TotalAlloc check below can't stop a parser that is on its way to exhausting the
// machine, so watch heap growth while it runs and take the process down once it passes the limit
metrics.Read(heap)
if grown := heap[0].Value.Uint64() - min(heapBase, heap[0].Value.Uint64()); grown > limit {
_ = pprof.Lookup("goroutine").WriteTo(os.Stderr, 2)
panic(fmt.Sprintf("%s: input of %d bytes grew the heap by %d MB (limit %d MB)", ft, len(data), grown>>20, limit>>20))
}
case <-deadline:
// not every parser watches ctx, so a hung one keeps running and would skew every later input in this
// process. Dump where it is stuck and take the process down; the fuzz engine records that as a crasher.
_ = pprof.Lookup("goroutine").WriteTo(os.Stderr, 2)
panic(fmt.Sprintf("%s: input of %d bytes did not finish within %s", ft, len(data), fuzzTimeout+fuzzHangSlack))
}
}
if ctx.Err() != nil {
t.Fatalf("%s: input of %d bytes did not finish within %s", ft, len(data), fuzzTimeout)
}
// a panic recovered further down (a go-sync PanicError, or the generic per-file recover) still surfaces as an error
if errors.As(err, &gosync.PanicError{}) || hasRecoveredPanic(err) {
t.Fatalf("%s: %v", ft, err)
}
// TotalAlloc is cumulative, so this bounds churn rather than peak, and it is read after the fact: a single huge
// allocation kills the process before this check, which the fuzz engine still records as a crasher
runtime.ReadMemStats(&after)
if alloc := after.TotalAlloc - before.TotalAlloc; alloc > limit {
t.Fatalf("%s: input of %d bytes allocated %d MB (limit %d MB)", ft, len(data), alloc>>20, limit>>20)
}
}
// hasRecoveredPanic finds an error anywhere in the tree that starts with "recovered from panic", the wording of the
// generic per-file recover (and of the golang and license recovers). Matching on the start of a node keeps parse
// errors that echo the input from counting.
func hasRecoveredPanic(err error) bool {
if err == nil {
return false
}
if strings.HasPrefix(err.Error(), "recovered from panic") {
return true
}
switch u := err.(type) {
case *unknown.CoordinateError:
// per-file errors arrive wrapped with their location, and CoordinateError does not unwrap
return hasRecoveredPanic(u.Reason)
case interface{ Unwrap() []error }:
for _, e := range u.Unwrap() {
if hasRecoveredPanic(e) {
return true
}
}
case interface{ Unwrap() error }:
return hasRecoveredPanic(u.Unwrap())
}
return false
}
// fuzzTargets pairs every glob-selected parser in the capabilities data with the task that runs its cataloger.
func fuzzTargets(tb testing.TB) []fuzzTarget {
tasks, err := task.DefaultPackageTaskFactories().Tasks(task.DefaultCatalogingFactoryConfig())
if err != nil {
tb.Fatal(err)
}
byName := map[string]task.Task{}
for _, tsk := range tasks {
byName[tsk.Name()] = tsk
}
entries, err := capabilities.Packages()
if err != nil {
tb.Fatal(err)
}
var filter *regexp.Regexp
if v := os.Getenv("SYFT_FUZZ_TARGET"); v != "" {
if filter, err = regexp.Compile(v); err != nil {
tb.Fatalf("SYFT_FUZZ_TARGET: %v", err)
}
}
var targets []fuzzTarget
for _, e := range entries {
tsk, ok := byName[e.Name]
if e.Type != "generic" || !ok {
continue
}
for _, p := range e.Parsers {
// glob parsers only. The mimetype parsers (binary classifiers) need a real executable
// header to be selected at all; add them if a mutating seed turns out to be worth it.
if p.Detector.Method != capabilities.GlobDetection {
continue
}
for _, g := range p.Detector.Criteria {
ft := fuzzTarget{
cataloger: e.Name,
glob: g,
path: concretePath(g),
dir: strings.TrimPrefix(filepath.Dir(e.Source.File), "syft/pkg/cataloger/"),
task: tsk,
}
if ok, _ := doublestar.Match(g, "/"+ft.path); !ok || !filepath.IsLocal(ft.path) {
tb.Logf("skipping %s: no concrete path for glob", ft)
continue
}
if filter != nil && !filter.MatchString(ft.String()) {
continue
}
targets = append(targets, ft)
}
}
}
if filter != nil && len(targets) == 0 {
tb.Fatalf("SYFT_FUZZ_TARGET %q matches no target", filter)
}
return targets
}
var braceAlt = regexp.MustCompile(`\{([^,}]*)[^}]*\}`)
// concretePath turns a glob into one relative path it matches, e.g. `**/*.gguf` -> `x.gguf`.
func concretePath(glob string) string {
s := strings.TrimPrefix(strings.TrimPrefix(glob, "**/"), "/")
s = braceAlt.ReplaceAllString(s, "$1")
s = strings.ReplaceAll(s, "**", "x")
return strings.ReplaceAll(s, "*", "x")
}
// fuzzSeeds is the parser's own testdata that matches its glob, plus a few inputs built from the string literals in
// its package. Random bytes rarely produce the exact key names a parser branches on, the literals give the mutator
// something to splice.
func fuzzSeeds(ft fuzzTarget) [][]byte {
var seeds [][]byte
// the CLI's malformed fixtures are deliberately not seeds here: before their fixes land some of them allocate
// gigabytes, which in-process means swapping the machine rather than failing a check. The CLI test runs them in
// a subprocess under a memory limit instead.
_ = filepath.WalkDir(ft.dir, func(p string, d fs.DirEntry, err error) error {
if err != nil || len(seeds) >= fuzzMaxSeedsPerFn {
return fs.SkipAll
}
if d.IsDir() || !d.Type().IsRegular() {
return nil
}
if ok, _ := doublestar.Match(ft.glob, "/"+filepath.ToSlash(p)); !ok {
return nil
}
if info, err := d.Info(); err != nil || info.Size() > fuzzMaxSeedBytes {
return nil
}
if b, err := os.ReadFile(p); err == nil {
seeds = append(seeds, b)
}
return nil
})
lits := stringLiterals(ft.dir)
if len(lits) == 0 {
return append(seeds, []byte{})
}
obj := map[string]string{}
var lines, kv strings.Builder
for _, l := range lits {
obj[l] = l
lines.WriteString(l + "\n")
kv.WriteString(strconv.Quote(l) + ": " + strconv.Quote(l) + "\n")
}
js, _ := json.Marshal(obj)
return append(seeds, []byte(lines.String()), []byte(kv.String()), js)
}
// stringLiterals returns the distinct short string literals in the non-test go files of a package directory.
func stringLiterals(dir string) []string {
entries, err := os.ReadDir(dir)
if err != nil {
return nil
}
set := map[string]struct{}{}
fset := token.NewFileSet()
for _, e := range entries {
name := e.Name()
if e.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") {
continue
}
f, err := parser.ParseFile(fset, filepath.Join(dir, name), nil, 0)
if err != nil {
continue
}
ast.Inspect(f, func(n ast.Node) bool {
if bl, ok := n.(*ast.BasicLit); ok && bl.Kind == token.STRING {
if s, err := strconv.Unquote(bl.Value); err == nil && len(s) > 1 && len(s) <= 64 {
set[s] = struct{}{}
}
}
return true
})
}
out := make([]string, 0, len(set))
for s := range set {
out = append(out, s)
}
sort.Strings(out)
return out
}
@@ -236,7 +236,10 @@ func (p interpreterCataloger) getClassifier(realPath string) (string, *binutils.
Vendor: fmt.Sprintf("php-%s", name),
Product: fmt.Sprintf("php-%s", name),
},
Source: cpe.GeneratedSource,
// note: we must use a declared source here so that this CPE is treated as authoritative and no
// further CPEs are generated from the (misleading) package name alone, e.g. cpe:...:openssl:openssl
// for the openssl PHP extension, which would imply this is the upstream OpenSSL project.
Source: cpe.DeclaredSource,
},
},
}
@@ -3,6 +3,10 @@ package php
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/anchore/syft/syft/cpe"
"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
)
@@ -147,3 +151,16 @@ func Test_InterpreterCataloger(t *testing.T) {
})
}
}
func Test_getClassifier_declaredCPESource(t *testing.T) {
// regression test for https://github.com/anchore/syft/issues/5014
// the CPE assigned to a PHP extension must be declared (not generated) so that downstream CPE
// generation does not also add a misleading CPE derived from the bare extension name alone
// (e.g. cpe:2.3:a:openssl:openssl:... for the openssl extension, implying the upstream OpenSSL project).
c := interpreterCataloger{name: "php-interpreter-cataloger"}
name, cls := c.getClassifier("/usr/lib/php85/modules/openssl.so")
require.Equal(t, "openssl", name)
require.NotNil(t, cls)
require.Len(t, cls.CPEs, 1)
assert.Equal(t, cpe.DeclaredSource, cls.CPEs[0].Source)
}
+1 -1
View File
@@ -15,7 +15,7 @@ import (
"github.com/anchore/syft/syft/pkg/cataloger/internal/dependency"
)
func poetryLockDependencySpecifier(p pkg.Package) dependency.Specification { //nolint:dupl // this is very similar to the uv lock dependency specifier, but should remain separate
func poetryLockDependencySpecifier(p pkg.Package) dependency.Specification {
meta, ok := p.Metadata.(pkg.PythonPoetryLockEntry)
if !ok {
log.Tracef("cataloger failed to extract poetry lock metadata for package %+v", p.Name)
@@ -1,7 +1,6 @@
package python
import (
"bufio"
"context"
"fmt"
"regexp"
@@ -105,7 +104,7 @@ func (rp requirementsParser) parseRequirementsTxt(ctx context.Context, _ file.Re
var errs error
var packages []pkg.Package
scanner := bufio.NewScanner(reader)
scanner := internal.NewLineScanner(reader)
var lastLine string
for scanner.Scan() {
line := trimRequirementsTxtLine(scanner.Text())
@@ -214,9 +213,8 @@ func guessVersion(constraint string) string {
constraints := strings.Split(constraint, ",")
filteredVersions := map[string]struct{}{}
for _, part := range constraints {
if strings.Contains(part, "!=") {
parts := strings.Split(part, "!=")
filteredVersions[strings.TrimSpace(parts[1])] = struct{}{}
if _, excluded, ok := strings.Cut(part, "!="); ok {
filteredVersions[strings.TrimSpace(excluded)] = struct{}{}
}
}
@@ -236,9 +234,8 @@ func guessVersion(constraint string) string {
continue
}
if strings.Contains(part, "==") {
parts := strings.Split(part, "==")
return strings.TrimSpace(parts[1])
if _, pinned, ok := strings.Cut(part, "=="); ok {
return strings.TrimSpace(pinned)
}
if closestVersion == nil || version.GreaterThan(*closestVersion) {
+13 -8
View File
@@ -1,11 +1,12 @@
package python
import (
"bufio"
"context"
"fmt"
"regexp"
"strings"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/file"
@@ -36,7 +37,7 @@ var unquotedPinnedDependency = regexp.MustCompile(`^\s*(\w+)\s*==\s*([\w\.\-]+)`
func (sp setupFileParser) parseSetupFile(ctx context.Context, _ file.Resolver, _ *generic.Environment, reader file.LocationReadCloser) ([]pkg.Package, []artifact.Relationship, error) {
var packages []pkg.Package
scanner := bufio.NewScanner(reader)
scanner := internal.NewLineScanner(reader)
for scanner.Scan() {
line := scanner.Text()
@@ -45,20 +46,23 @@ func (sp setupFileParser) parseSetupFile(ctx context.Context, _ file.Resolver, _
packages = sp.processQuotedDependencies(ctx, line, reader, packages)
packages = sp.processUnquotedDependency(ctx, line, reader, packages)
}
if err := scanner.Err(); err != nil {
return packages, nil, fmt.Errorf("unable to read setup.py: %w", err)
}
return packages, nil, nil
}
func (sp setupFileParser) processQuotedDependencies(ctx context.Context, line string, reader file.LocationReadCloser, packages []pkg.Package) []pkg.Package {
for _, match := range pinnedDependency.FindAllString(line, -1) {
if p, ok := sp.parseQuotedDependency(ctx, match, line, reader); ok {
if p, ok := sp.parseQuotedDependency(ctx, match, reader); ok {
packages = append(packages, p)
}
}
return packages
}
func (sp setupFileParser) parseQuotedDependency(ctx context.Context, match, line string, reader file.LocationReadCloser) (pkg.Package, bool) {
func (sp setupFileParser) parseQuotedDependency(ctx context.Context, match string, reader file.LocationReadCloser) (pkg.Package, bool) {
parts := strings.Split(match, "==")
if len(parts) != 2 {
return pkg.Package{}, false
@@ -67,7 +71,7 @@ func (sp setupFileParser) parseQuotedDependency(ctx context.Context, match, line
name := cleanDependencyString(parts[0])
version := cleanDependencyString(parts[len(parts)-1])
return sp.validateAndCreatePackage(ctx, name, version, line, reader)
return sp.validateAndCreatePackage(ctx, name, version, reader)
}
// processUnquotedDependency extracts and processes an unquoted dependency from a line
@@ -80,7 +84,7 @@ func (sp setupFileParser) processUnquotedDependency(ctx context.Context, line st
name := strings.TrimSpace(matches[1])
version := strings.TrimSpace(matches[2])
if p, ok := sp.validateAndCreatePackage(ctx, name, version, line, reader); ok {
if p, ok := sp.validateAndCreatePackage(ctx, name, version, reader); ok {
if !isDuplicatePackage(p, packages) {
packages = append(packages, p)
}
@@ -96,14 +100,15 @@ func cleanDependencyString(s string) string {
return s
}
func (sp setupFileParser) validateAndCreatePackage(ctx context.Context, name, version, line string, reader file.LocationReadCloser) (pkg.Package, bool) {
func (sp setupFileParser) validateAndCreatePackage(ctx context.Context, name, version string, reader file.LocationReadCloser) (pkg.Package, bool) {
if hasTemplateDirective(name) || hasTemplateDirective(version) {
// this can happen in more dynamic setup.py where there is templating
return pkg.Package{}, false
}
if name == "" || version == "" {
log.WithFields("path", reader.RealPath).Debugf("unable to parse package in setup.py line: %q", line)
// the raw line is not logged since this runs once per match, which is quadratic on a long line
log.WithFields("path", reader.RealPath, "name", name, "version", version).Debug("unable to parse package in setup.py")
return pkg.Package{}, false
}
+72 -54
View File
@@ -5,12 +5,13 @@ import (
"context"
"fmt"
"io"
"maps"
"slices"
"sort"
"strings"
"github.com/BurntSushi/toml"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/internal/unknown"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/file"
@@ -45,8 +46,12 @@ type uvPackage struct {
Metadata uvMetadata `toml:"metadata"`
}
type uvDependencies []struct {
Name string `toml:"name"`
type uvDependencies []uvDependency
type uvDependency struct {
Name string `toml:"name"`
// Version is only written by uv when more than one version of this name is locked
Version string `toml:"version"`
Extras []string `toml:"extra"`
Markers string `toml:"marker"`
}
@@ -83,12 +88,16 @@ func newUvLockParser(cfg CatalogerConfig) uvLockParser {
// parseUvLock is a parser function for uv.lock contents, returning all the pakcages discovered
func (ulp uvLockParser) parseUvLock(ctx context.Context, _ file.Resolver, _ *generic.Environment, reader file.LocationReadCloser) ([]pkg.Package, []artifact.Relationship, error) {
pkgs, err := ulp.uvLockPackages(ctx, reader)
pkgs, specs, err := ulp.uvLockPackages(ctx, reader)
if err != nil {
return nil, nil, err
}
return pkgs, dependency.Resolve(uvLockDependencySpecifier, pkgs), err
specifier := func(p pkg.Package) dependency.Specification {
return specs[p.ID()]
}
return pkgs, dependency.Resolve(specifier, pkgs), err
}
func extractUvIndex(p uvPackage) string {
@@ -139,105 +148,114 @@ func newPythonUvLockEntry(p uvPackage) pkg.PythonUvLockEntry {
}
}
func (ulp uvLockParser) uvLockPackages(ctx context.Context, reader file.LocationReadCloser) ([]pkg.Package, error) {
// uvLockPackages returns the packages in the lock along with the dependency specification for each (by package ID).
func (ulp uvLockParser) uvLockPackages(ctx context.Context, reader file.LocationReadCloser) ([]pkg.Package, map[artifact.ID]dependency.Specification, error) {
var parsedLockFileVersion uvLockFileVersion
// we cannot use the reader twice, so we read the contents first --uv.lock files tend to be small enough
contents, err := io.ReadAll(reader) //nolint:gocritic // multi-pass parse requires []byte
if err != nil {
return nil, unknown.New(reader.Location, fmt.Errorf("failed to read uv lock file: %w", err))
return nil, nil, unknown.New(reader.Location, fmt.Errorf("failed to read uv lock file: %w", err))
}
_, err = toml.NewDecoder(bytes.NewReader(contents)).Decode(&parsedLockFileVersion)
if err != nil {
return nil, fmt.Errorf("failed to read uv lock version: %w", err)
return nil, nil, fmt.Errorf("failed to read uv lock version: %w", err)
}
// We will need to add some logic to parse and branch on different
// lock file versions should they arise, but this gets us
// started down this road for now.
if parsedLockFileVersion.Version > 1 {
return nil, fmt.Errorf("could not parse uv lock file version %d", parsedLockFileVersion.Version)
return nil, nil, fmt.Errorf("could not parse uv lock file version %d", parsedLockFileVersion.Version)
}
var parsedLockFile uvLockFile
_, err = toml.NewDecoder(bytes.NewReader(contents)).Decode(&parsedLockFile)
if err != nil {
return nil, fmt.Errorf("failed to parse uv lock packages: %w", err)
}
// The uv lock file doesn't store the dependency version in the dependency structure.
// Thus, we need a name -> version map for invoking extractUvDependencies.
// We then, of course, have to pass it down the call stack.
var pkgVerMap = make(map[string]string)
for _, p := range parsedLockFile.Packages {
pkgVerMap[p.Name] = p.Version
return nil, nil, fmt.Errorf("failed to parse uv lock packages: %w", err)
}
var pkgs []pkg.Package
specs := make(map[artifact.ID]dependency.Specification)
for _, p := range parsedLockFile.Packages {
pkgs = append(pkgs,
newPackageForIndexWithMetadata(
ctx,
ulp.licenseResolver,
p.Name,
p.Version,
newPythonUvLockEntry(p),
reader.WithAnnotation(pkg.EvidenceAnnotationKey, pkg.PrimaryEvidenceAnnotation),
),
np := newPackageForIndexWithMetadata(
ctx,
ulp.licenseResolver,
p.Name,
p.Version,
newPythonUvLockEntry(p),
reader.WithAnnotation(pkg.EvidenceAnnotationKey, pkg.PrimaryEvidenceAnnotation),
)
pkgs = append(pkgs, np)
specs[np.ID()] = uvLockDependencySpecification(p)
}
return pkgs, unknown.IfEmptyf(pkgs, "unable to determine packages")
return pkgs, specs, unknown.IfEmptyf(pkgs, "unable to determine packages")
}
func isDependencyForUvExtra(dep pkg.PythonUvLockDependencyEntry) bool {
func isDependencyForUvExtra(dep uvDependency) bool {
return strings.Contains(dep.Markers, "extra ==")
}
// This is identical to poetryLockDependencySpecifier since it operates on identical
// data structures. Keeping it separate for now since it's always possible for data
// structures to change down the line.
// It *is* possible we may be able to merge the Uv and Poetry data structures
func uvLockDependencySpecifier(p pkg.Package) dependency.Specification { //nolint:dupl // this is very similar to the poetry lock dependency specifier, but should remain separate
meta, ok := p.Metadata.(pkg.PythonUvLockEntry)
if !ok {
log.Tracef("cataloger failed to extract UV lock metadata for package %+v", p.Name)
return dependency.Specification{}
}
provides := []string{packageRef(p.Name, "")}
// uvLockDependencySpecification is built from the raw lock entry rather than the package metadata. When a lock
// holds several versions of one name (a forked resolution), uv records which version each dependency entry means,
// and that is the only way to pair a dependent with the right one. The pairing is carried by the resulting
// relationships, so the version does not need to live on the metadata.
func uvLockDependencySpecification(p uvPackage) dependency.Specification {
var requires []string
for _, dep := range meta.Dependencies {
for _, dep := range p.Dependencies {
if isDependencyForUvExtra(dep) {
continue
}
requires = append(requires, packageRef(dep.Name, ""))
for _, extra := range dep.Extras {
requires = append(requires, packageRef(dep.Name, extra))
}
requires = append(requires, uvLockRequires(dep)...)
}
var variants []dependency.ProvidesRequires
for _, extra := range meta.Extras {
for _, extra := range slices.Sorted(maps.Keys(p.OptionalDependencies)) {
var extraRequires []string
for _, dep := range p.OptionalDependencies[extra] {
extraRequires = append(extraRequires, uvLockRequires(dep)...)
}
variants = append(variants,
dependency.ProvidesRequires{
Provides: []string{packageRef(p.Name, extra.Name)},
Requires: extractPackageNames(extra.Dependencies),
Provides: uvLockProvides(p, extra),
Requires: extraRequires,
},
)
}
return dependency.Specification{
ProvidesRequires: dependency.ProvidesRequires{
Provides: provides,
Provides: uvLockProvides(p, ""),
Requires: requires,
},
Variants: variants,
}
}
// uvLockProvides offers both the bare ref (for dependency entries without a version) and the versioned ref (for
// dependency entries uv pinned to this version).
func uvLockProvides(p uvPackage, extra string) []string {
return []string{packageRef(p.Name, extra), uvLockPackageRef(p.Name, p.Version, extra)}
}
// uvLockRequires always requires the base package, plus each extra individually (name[extra1] and name[extra2],
// never name[extra1,extra2]).
func uvLockRequires(dep uvDependency) []string {
refs := []string{uvLockPackageRef(dep.Name, dep.Version, "")}
for _, extra := range dep.Extras {
refs = append(refs, uvLockPackageRef(dep.Name, dep.Version, extra))
}
return refs
}
func uvLockPackageRef(name, version, extra string) string {
ref := packageRef(name, extra)
version = strings.TrimSpace(version)
if version == "" {
return ref
}
return ref + "@" + version
}
@@ -129,6 +129,79 @@ func TestParseUvLock(t *testing.T) {
pkgtest.TestFileParser(t, fixture, uvLockParser.parseUvLock, expectedPkgs, expectedRelationships)
}
func TestParseUvLockForkedVersions(t *testing.T) {
fixture := "testdata/uv/forked-versions/uv.lock"
locations := file.NewLocationSet(file.NewLocation(fixture))
index := "https://pypi.org/simple"
newPkg := func(name, version string, meta pkg.PythonUvLockEntry) pkg.Package {
return pkg.Package{
Name: name,
Version: version,
Locations: locations,
PURL: "pkg:pypi/" + name + "@" + version,
Language: pkg.Python,
Type: pkg.PythonPkg,
Metadata: meta,
}
}
forkdemo := newPkg("forkdemo", "0.1.0", pkg.PythonUvLockEntry{
Index: ".",
Dependencies: []pkg.PythonUvLockDependencyEntry{
{Name: "pandas", Markers: "python_full_version < '3.11'"},
{Name: "pandas", Markers: "python_full_version >= '3.11'"},
},
Extras: []pkg.PythonUvLockExtraEntry{
{Name: "legacy", Dependencies: []string{"numpy"}},
},
})
numpy1264 := newPkg("numpy", "1.26.4", pkg.PythonUvLockEntry{Index: index})
numpy226 := newPkg("numpy", "2.2.6", pkg.PythonUvLockEntry{Index: index})
numpy246 := newPkg("numpy", "2.4.6", pkg.PythonUvLockEntry{Index: index})
numpy253 := newPkg("numpy", "2.5.3", pkg.PythonUvLockEntry{Index: index})
pandas233 := newPkg("pandas", "2.3.3", pkg.PythonUvLockEntry{
Index: index,
Dependencies: []pkg.PythonUvLockDependencyEntry{
{Name: "numpy", Markers: "python_full_version < '3.10'"},
{Name: "numpy", Markers: "python_full_version == '3.10.*'"},
},
})
pandas306 := newPkg("pandas", "3.0.6", pkg.PythonUvLockEntry{
Index: index,
Dependencies: []pkg.PythonUvLockDependencyEntry{
{Name: "numpy", Markers: "python_full_version == '3.11.*'"},
{Name: "numpy", Markers: "python_full_version >= '3.12'"},
},
})
dependencyOf := func(from, to pkg.Package) artifact.Relationship {
return artifact.Relationship{From: from, To: to, Type: artifact.DependencyOfRelationship}
}
// each dependent links only to the versions its entries name, including through an optional dependency
expectedRelationships := []artifact.Relationship{
dependencyOf(pandas233, forkdemo),
dependencyOf(pandas306, forkdemo),
dependencyOf(numpy1264, forkdemo),
dependencyOf(numpy1264, pandas233),
dependencyOf(numpy226, pandas233),
dependencyOf(numpy246, pandas306),
dependencyOf(numpy253, pandas306),
}
uvLockParser := newUvLockParser(DefaultCatalogerConfig())
pkgtest.TestFileParser(t, fixture, uvLockParser.parseUvLock, []pkg.Package{
forkdemo,
numpy1264,
numpy226,
numpy246,
numpy253,
pandas233,
pandas306,
}, expectedRelationships)
}
func TestParseUvLockWithLicenseEnrichment(t *testing.T) {
ctx := context.TODO()
fixture := "testdata/pypi-remote/uv.lock"
@@ -1,7 +1,6 @@
package python
import (
"bufio"
"fmt"
"path/filepath"
"strings"
@@ -9,6 +8,7 @@ import (
"github.com/bmatcuk/doublestar/v4"
"github.com/go-viper/mapstructure/v2"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/pkg"
@@ -68,7 +68,7 @@ func extractRFC5322Fields(locationReader file.LocationReadCloser) (map[string]an
// though this spec is governed by RFC 5322 (mail message), the metadata files are not guaranteed to be compliant.
// We must survive parsing as much info as possible without failing and dropping the data.
scanner := bufio.NewScanner(locationReader)
scanner := internal.NewLineScanner(locationReader)
for scanner.Scan() {
line := scanner.Text()
line = strings.TrimRight(line, "\n")
@@ -108,6 +108,10 @@ func extractRFC5322Fields(locationReader file.LocationReadCloser) (map[string]an
}
}
}
if err := scanner.Err(); err != nil {
// the caller discards everything on error, so keep the partial fields and log instead
log.WithFields("path", locationReader.Path(), "error", err).Debug("failed to fully read python wheel/egg metadata")
}
return fields, nil
}
+54
View File
@@ -0,0 +1,54 @@
version = 1
requires-python = ">=3.9"
[[package]]
name = "forkdemo"
version = "0.1.0"
source = { virtual = "." }
dependencies = [
{ name = "pandas", version = "2.3.3", marker = "python_full_version < '3.11'" },
{ name = "pandas", version = "3.0.6", marker = "python_full_version >= '3.11'" },
]
[package.optional-dependencies]
legacy = [
{ name = "numpy", version = "1.26.4" },
]
[[package]]
name = "numpy"
version = "1.26.4"
source = { registry = "https://pypi.org/simple" }
[[package]]
name = "numpy"
version = "2.2.6"
source = { registry = "https://pypi.org/simple" }
[[package]]
name = "numpy"
version = "2.4.6"
source = { registry = "https://pypi.org/simple" }
[[package]]
name = "numpy"
version = "2.5.3"
source = { registry = "https://pypi.org/simple" }
[[package]]
name = "pandas"
version = "2.3.3"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "numpy", version = "1.26.4", marker = "python_full_version < '3.10'" },
{ name = "numpy", version = "2.2.6", marker = "python_full_version == '3.10.*'" },
]
[[package]]
name = "pandas"
version = "3.0.6"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "numpy", version = "2.4.6", marker = "python_full_version == '3.11.*'" },
{ name = "numpy", version = "2.5.3", marker = "python_full_version >= '3.12'" },
]
+6 -2
View File
@@ -1,12 +1,13 @@
package r
import (
"bufio"
"context"
"io"
"regexp"
"strings"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/pkg"
@@ -105,7 +106,7 @@ func extractFieldsFromDescriptionFile(reader io.Reader) map[string]string {
result := make(map[string]string)
key := ""
var valueFragment strings.Builder
scanner := bufio.NewScanner(reader)
scanner := internal.NewLineScanner(reader)
for scanner.Scan() {
line := scanner.Text()
@@ -139,6 +140,9 @@ func extractFieldsFromDescriptionFile(reader io.Reader) map[string]string {
if key != "" {
result[key] = valueFragment.String()
}
if err := scanner.Err(); err != nil {
log.WithFields("error", err).Debug("failed to fully read R DESCRIPTION file")
}
return result
}
@@ -1,12 +1,12 @@
package ruby
import (
"bufio"
"context"
"strings"
"github.com/scylladb/go-set/strset"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/unknown"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/file"
@@ -21,7 +21,7 @@ var sectionsOfInterest = strset.New("GEM", "GIT", "PATH", "PLUGIN SOURCE")
// parseGemFileLockEntries is a parser function for Gemfile.lock contents, returning all Gems discovered.
func parseGemFileLockEntries(_ context.Context, _ file.Resolver, _ *generic.Environment, reader file.LocationReadCloser) ([]pkg.Package, []artifact.Relationship, error) {
var pkgs []pkg.Package
scanner := bufio.NewScanner(reader)
scanner := internal.NewLineScanner(reader)
var currentSection string
+5 -2
View File
@@ -1,7 +1,6 @@
package ruby
import (
"bufio"
"context"
"encoding/json"
"fmt"
@@ -75,7 +74,7 @@ func processList(s string) []string {
func parseGemSpecEntries(ctx context.Context, resolver file.Resolver, _ *generic.Environment, reader file.LocationReadCloser) ([]pkg.Package, []artifact.Relationship, error) {
var pkgs []pkg.Package
var fields = make(map[string]any)
scanner := bufio.NewScanner(reader)
scanner := internal.NewLineScanner(reader)
for scanner.Scan() {
line := scanner.Text()
@@ -101,6 +100,7 @@ func parseGemSpecEntries(ctx context.Context, resolver file.Resolver, _ *generic
}
}
}
scanErr := scanner.Err()
resolveRubyInterpolationsInFields(fields)
@@ -121,6 +121,9 @@ func parseGemSpecEntries(ctx context.Context, resolver file.Resolver, _ *generic
)
}
if scanErr != nil {
return pkgs, nil, fmt.Errorf("unable to read gemspec: %w", scanErr)
}
return pkgs, nil, nil
}
+3 -4
View File
@@ -60,10 +60,9 @@ func parseBaseDpkgYaml(_ context.Context, _ file.Resolver, _ *generic.Environmen
}
// Handle architecture suffixes (e.g., "libssl1.1:amd64")
if strings.Contains(name, ":") {
archParts := strings.SplitN(name, ":", 2)
name = archParts[0]
snapMetadata.Architecture = archParts[1]
if n, arch, ok := strings.Cut(name, ":"); ok {
name = n
snapMetadata.Architecture = arch
}
debPkg := newDebianPackageFromSnap(
@@ -1,13 +1,13 @@
package snap
import (
"bufio"
"compress/gzip"
"context"
"fmt"
"regexp"
"strings"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/pkg"
@@ -32,7 +32,7 @@ func parseKernelChangelog(_ context.Context, _ file.Resolver, _ *generic.Environ
}
defer gzReader.Close()
scanner := bufio.NewScanner(gzReader)
scanner := internal.NewLineScanner(gzReader)
// read the first line to extract kernel version
// Format: "linux (5.4.0-195.215) focal; urgency=medium"
@@ -65,6 +65,9 @@ func parseKernelChangelog(_ context.Context, _ file.Resolver, _ *generic.Environ
break
}
}
if err := scanner.Err(); err != nil {
return packages, nil, fmt.Errorf("failed to read changelog content: %w", err)
}
return packages, nil, nil
}
@@ -318,7 +318,9 @@ func TestParseKernelChangelog(t *testing.T) {
// thing holding this path, and it is incidental: switching to io.ReadAll or a bufio.Reader to pick up
// long lines would remove it silently, which is what this test is here to catch.
func TestParseKernelChangelog_gzipBombStaysBounded(t *testing.T) {
const expanded = 512 * 1024 * 1024
// 4x the 8MB budget below is plenty to tell bounded from unbounded. Going bigger only makes the
// fixture slower to build and inflate, which is most of this test's time under -race.
const expanded = 32 * 1024 * 1024
// one enormous line with no newline, the worst case for a line scanner
var gz bytes.Buffer
@@ -82,10 +82,9 @@ func parseSystemManifest(_ context.Context, _ file.Resolver, _ *generic.Environm
// Handle architecture suffixes if present
currentMetadata := snapMetadata
if strings.Contains(name, ":") {
archParts := strings.SplitN(name, ":", 2)
name = archParts[0]
currentMetadata.Architecture = archParts[1]
if n, arch, ok := strings.Cut(name, ":"); ok {
name = n
currentMetadata.Architecture = arch
}
debPkg := newDebianPackageFromSnap(

Some files were not shown because too many files have changed in this diff Show More