mirror of
https://github.com/anchore/syft.git
synced 2026-10-30 14:02:26 +01:00
679027251852f07dd80f2bde260f321cb637548b
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  JuanandAlex Goodman
|
e6d364437a |
feat(convert): passthrough-exact-format copies matching syft-json input unchanged (#5279)
* feat(convert): passthrough-exact-format copies matching syft-json input unchanged When a syft-json input SBOM already exactly matches the requested output format and schema version, `syft convert` can write the input bytes out verbatim instead of decoding and re-encoding them. This avoids the lossy round trip (and the associated cost) for callers that normalise SBOMs into a target format without knowing up front whether the source already matches. The behaviour is opt-in and configuration-only: it is enabled with convert.passthrough-exact-format in the config file or the SYFT_CONVERT_PASSTHROUGH_EXACT_FORMAT environment variable, and there is deliberately no command-line flag, since it changes how every invocation in a pipeline behaves rather than varying call by call. It defaults to off, so existing behaviour is unchanged. Pass-through applies only to syft-json input, and only when the document's schema version is identical to the one this build of syft produces; any other input is always converted. Outputs that do not match are still converted, so a mixed set of `-o` values behaves as expected, and every output is resolved and validated before anything is written. Pass-through output is a byte-for-byte copy, so format options such as `format.pretty` and `format.json.legacy` are not applied to it. When the option is enabled a warning is logged stating so and noting the syft-json-only scope, and a second warning is logged whenever an input is found to match and the shortcut is actually taken. The option lives under a `convert:` configuration section (options.Convert), mirroring how attest scopes its own options. The description documents the caveats above and notes that file-path input written with `-o <format>=<path>` keeps memory minimal, whereas piped input or STDOUT output must hold the whole document in memory. Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * perf(convert): stream pass-through output instead of buffering the input Identify, copy, and decode the input through a seekable reader rather than loading the whole document into memory first. File inputs are read in place and pass-through output is an io.Copy from the file. STDIN is used in place when it is a redirected regular file. A pipe or terminal cannot seek, so its contents are drained into an internal/spillbuf buffer: the first 1MB stays in memory and the rest goes to a temp file under the user's TMPDIR (created lazily, only when something actually spills), giving a seekable view without ever holding the document in memory. The temp dir is rooted per invocation via internal/tmpdir, honouring one already attached to the context, and is removed when the command returns. Writing pass-through output over the input file itself is now a no-op rather than truncating the file before it is copied. Measured on a 1GB syft-json document (Apple M1 Max), file in, file out: origin/main full re-encode 86.3s wall 99s CPU 12.0GB max RSS in-memory pass-through 7.4s wall 6.7s CPU 4.2GB max RSS streaming pass-through 7.0s wall 6.9s CPU 3.1GB max RSS and the same 1GB document piped over STDIN with pass-through enabled: io.ReadAll into memory 1.3s wall 0.8s CPU 2.7GB max RSS spill buffer 1.8s wall 1.0s CPU 66MB max RSS The remaining cost of the streaming pass-through is entirely format.Identify, which for syft-json decodes the whole document through a json.Decoder that buffers the full value: measured alone it takes 7.1s and 3.1GB on the same input. Reducing that further belongs to the format package. Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * perf(syftjson): identify documents from their tail instead of a full parse syft always writes the schema block as the last key of a syft-json document, so the format and version can be read from the final bytes of the file. The syft-json decoder's Identify now reads at most 2MB from the end of a seekable input, finds the last "schema" key there, and verifies it sits directly within the top-level object by scanning the remaining bytes for the single closing brace of the document. Anything the tail cannot vouch for (non-seekable input, a reordered document, a nested key) falls back to the existing full parse, so behaviour is unchanged for documents the fast path declines. The verification is position-independent: it also accepts documents whose keys have been reordered so that other small keys follow schema. It rejects nested schema objects, array-wrapped documents, truncated input, trailing content, and schema text inside strings. Supporting changes: - the offset-aware reader the decoder collection hands to decoders now supports io.SeekEnd, which the tail read requires - model.Document documents that Schema must remain its last field, and a test guards that the encoder writes it last in both compact and pretty output - the decoder list documents that syft-json must stay first Since Decode calls Identify before parsing, this also removes one of the two full parses every syft-json conversion used to perform. Measured on a 1GB syft-json document (Apple M1 Max), file in, file out: pass-through, previous commit 7.0s wall 6.9s CPU 3.1GB max RSS pass-through, this commit 0.85s wall 0.05s CPU 66MB max RSS full re-encode, previous commit 85.3s wall 99s CPU 11.5GB max RSS full re-encode, this commit 75.2s wall 91s CPU 9.9GB max RSS Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix(syftjson): only identify from the tail when the document starts where the reader does The tail scan vouched for the last 2MB, but nothing checked those bytes belonged to the document the reader was pointed at. So `Identify` started claiming syft-json for files it cannot decode: a document with a byte order mark or a stray log line in front of it, or a syft document sitting after something else in the same stream, where `format.Decode` went on to return an empty SBOM with a nil error. Two O(1) checks: - the first non-space byte at the reader's position has to be `{` - `Decode` errors when the parsed document has no `schema.version`, instead of handing back the empty document it decoded into Streams holding several concatenated documents are deliberately not detected. Proving there is exactly one top-level value means reading all of it, which is the cost the tail path exists to avoid, so `Identify` can still name a later document's version on such a file. `Decode` reports what it actually parsed. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix(convert): recognize the input file by identity rather than by path `convert - -o syft-json=sbom.json < sbom.json` truncated the file and exited 0. The passthrough guard compared paths, and a shell redirect does not have one, so it was blind to the one input mode where the truncation was silent. `openConvertInput` now returns the input's `os.FileInfo`, which the STDIN branch already had from its `Stat`, and `isSameFile` compares that instead. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> --------- Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com> |
||
 Keith Zantow
|
9c321691d4 |
feat: SPDX 3 (#4269)
Signed-off-by: Keith Zantow <kzantow@gmail.com> |
||
|
Alex Goodman
|
b5e85c3ea5 |
chore: migrate fixtures to testdata (#4651)
* migrate fixtures to testdata Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix: correct broken symlinks after testdata migration The migration from test-fixtures to testdata broke several symlinks: - elf-test-fixtures symlinks pointed to old test-fixtures paths - elf-test-fixtures needed to be renamed to elf-testdata - image-pkg-coverage symlink pointed to test-fixtures instead of testdata Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix: handle missing classifiers/bin directory in Makefile The clean-fingerprint target was failing when classifiers/bin doesn't exist (e.g., on fresh clone without downloaded binaries). Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix: add gitignore negation for jar/zip fixtures in test/cli The jar and zip files in test/cli/testdata/image-unknowns were being gitignored by the root .gitignore patterns. This caused them to be untracked and not included when building docker images in CI, resulting in Test_Unknowns failures since the test expects errors from corrupt archive files that weren't present. Add a .gitignore in test/cli/testdata to negate the exclusions for these specific test fixture files. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * switch fixture cache to v2 Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * test: update expected versions for rebuilt fixtures Update test expectations for packages that have been updated in upstream repositories when docker images are rebuilt: - glibc: 2.42-r4 → 2.43-r1 (wolfi) - php: 8.2.29 → 8.2.30 (ubuntu/apache) Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * upgrade go Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix: add go-shlex dependency for testdata manager tool The manager tool in syft/pkg/cataloger/binary/testdata/ imports go-shlex, but since it's in a testdata directory, Go doesn't track its dependencies. This caused CI failures when go.mod didn't explicitly list the dependency. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * refactor: move binary classifier manager to internal/ Move the manager tool from testdata/manager to internal/manager so that Go properly tracks its dependencies. Code in testdata directories is ignored by Go for dependency tracking, which caused CI failures when go.mod didn't explicitly list transitive dependencies. This is a cleaner solution than manually adding dependencies to go.mod for code that happens to live in testdata. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix: add gitignore negations for test fixtures blocked by root patterns Multiple test fixtures were being blocked by root-level gitignore patterns like bin/, *.jar, *.tar, and *.exe. This adds targeted .gitignore files with negation patterns to allow these specific test fixtures to be tracked: - syft/linux/testdata/os/busybox/bin/busybox (blocked by bin/) - syft/pkg/cataloger/java/testdata/corrupt/example.{jar,tar} (blocked by *.jar, *.tar) - syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/**/bin/go (blocked by bin/) - syft/pkg/cataloger/bitnami/testdata/no-rel/.../bin/redis-server (blocked by bin/) Also updates the bitnami test expectation to include the newly required .gitignore files in the test fixture. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * test: update glibc version expectation (2.43-r1 -> 2.43-r2) Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * add capability drift check as unit step Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * dont clear test observations before drift detection Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * bump stereoscope commit to main Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> --------- Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> |
||
|
Alex Goodman
|
e0e1c4ba0a |
Internalize majority of cmd package (#2533)
* internalize majority of cmd package and migrate integration tests Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * add internal api encoder Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * create internal representation of all formats Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * export capability to get default encoders Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * restore test fixtures Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> --------- Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> |
||
  Avi DeitcherandAlex Goodman
|
cc731c7b19 |
Add Linux Kernel cataloger (#1694)
* add kernel handler Signed-off-by: Avi Deitcher <avi@deitcher.net> * [wip] combine kernel and kernel module cataloging Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * [wip] combine kernel and kernel module cataloging Signed-off-by: Alex Goodman <alex.goodman@anchore.com> Signed-off-by: Avi Deitcher <avi@deitcher.net> * rename Kernel package to LinuxKernel package Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * split kernel and module packages within cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * wire up application configuration with kernel cataloger options Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * dont use references for packages on relationships Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix linting and tests Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * kernel cataloger should be resistent to partial failure Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * log upon kernel module metadata missing Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add tests for linux kernel cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * update integration tests Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * update cli package test counts Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add evidence annotations for kernel packages Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * reduce noise in cli test output Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * missed cli test to reduce noise for Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix package counts Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * update docs with linux kernel cataloging refs Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * bump json schema with new metadata fields Signed-off-by: Alex Goodman <alex.goodman@anchore.com> --------- Signed-off-by: Avi Deitcher <avi@deitcher.net> Signed-off-by: Alex Goodman <alex.goodman@anchore.com> Signed-off-by: <> Co-authored-by: Alex Goodman <alex.goodman@anchore.com> |
||
|
Alex Goodman
|
9b9a7d6c98 |
allow convert to take stdin (#1570)
Signed-off-by: Alex Goodman <alex.goodman@anchore.com> |