mirror of
https://github.com/anchore/syft.git
synced 2026-10-11 21:57:22 +02:00
JuanandAlex Goodman
e6d364437a
* feat(convert): passthrough-exact-format copies matching syft-json input unchanged When a syft-json input SBOM already exactly matches the requested output format and schema version, `syft convert` can write the input bytes out verbatim instead of decoding and re-encoding them. This avoids the lossy round trip (and the associated cost) for callers that normalise SBOMs into a target format without knowing up front whether the source already matches. The behaviour is opt-in and configuration-only: it is enabled with convert.passthrough-exact-format in the config file or the SYFT_CONVERT_PASSTHROUGH_EXACT_FORMAT environment variable, and there is deliberately no command-line flag, since it changes how every invocation in a pipeline behaves rather than varying call by call. It defaults to off, so existing behaviour is unchanged. Pass-through applies only to syft-json input, and only when the document's schema version is identical to the one this build of syft produces; any other input is always converted. Outputs that do not match are still converted, so a mixed set of `-o` values behaves as expected, and every output is resolved and validated before anything is written. Pass-through output is a byte-for-byte copy, so format options such as `format.pretty` and `format.json.legacy` are not applied to it. When the option is enabled a warning is logged stating so and noting the syft-json-only scope, and a second warning is logged whenever an input is found to match and the shortcut is actually taken. The option lives under a `convert:` configuration section (options.Convert), mirroring how attest scopes its own options. The description documents the caveats above and notes that file-path input written with `-o <format>=<path>` keeps memory minimal, whereas piped input or STDOUT output must hold the whole document in memory. Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * perf(convert): stream pass-through output instead of buffering the input Identify, copy, and decode the input through a seekable reader rather than loading the whole document into memory first. File inputs are read in place and pass-through output is an io.Copy from the file. STDIN is used in place when it is a redirected regular file. A pipe or terminal cannot seek, so its contents are drained into an internal/spillbuf buffer: the first 1MB stays in memory and the rest goes to a temp file under the user's TMPDIR (created lazily, only when something actually spills), giving a seekable view without ever holding the document in memory. The temp dir is rooted per invocation via internal/tmpdir, honouring one already attached to the context, and is removed when the command returns. Writing pass-through output over the input file itself is now a no-op rather than truncating the file before it is copied. Measured on a 1GB syft-json document (Apple M1 Max), file in, file out: origin/main full re-encode 86.3s wall 99s CPU 12.0GB max RSS in-memory pass-through 7.4s wall 6.7s CPU 4.2GB max RSS streaming pass-through 7.0s wall 6.9s CPU 3.1GB max RSS and the same 1GB document piped over STDIN with pass-through enabled: io.ReadAll into memory 1.3s wall 0.8s CPU 2.7GB max RSS spill buffer 1.8s wall 1.0s CPU 66MB max RSS The remaining cost of the streaming pass-through is entirely format.Identify, which for syft-json decodes the whole document through a json.Decoder that buffers the full value: measured alone it takes 7.1s and 3.1GB on the same input. Reducing that further belongs to the format package. Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * perf(syftjson): identify documents from their tail instead of a full parse syft always writes the schema block as the last key of a syft-json document, so the format and version can be read from the final bytes of the file. The syft-json decoder's Identify now reads at most 2MB from the end of a seekable input, finds the last "schema" key there, and verifies it sits directly within the top-level object by scanning the remaining bytes for the single closing brace of the document. Anything the tail cannot vouch for (non-seekable input, a reordered document, a nested key) falls back to the existing full parse, so behaviour is unchanged for documents the fast path declines. The verification is position-independent: it also accepts documents whose keys have been reordered so that other small keys follow schema. It rejects nested schema objects, array-wrapped documents, truncated input, trailing content, and schema text inside strings. Supporting changes: - the offset-aware reader the decoder collection hands to decoders now supports io.SeekEnd, which the tail read requires - model.Document documents that Schema must remain its last field, and a test guards that the encoder writes it last in both compact and pretty output - the decoder list documents that syft-json must stay first Since Decode calls Identify before parsing, this also removes one of the two full parses every syft-json conversion used to perform. Measured on a 1GB syft-json document (Apple M1 Max), file in, file out: pass-through, previous commit 7.0s wall 6.9s CPU 3.1GB max RSS pass-through, this commit 0.85s wall 0.05s CPU 66MB max RSS full re-encode, previous commit 85.3s wall 99s CPU 11.5GB max RSS full re-encode, this commit 75.2s wall 91s CPU 9.9GB max RSS Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix(syftjson): only identify from the tail when the document starts where the reader does The tail scan vouched for the last 2MB, but nothing checked those bytes belonged to the document the reader was pointed at. So `Identify` started claiming syft-json for files it cannot decode: a document with a byte order mark or a stray log line in front of it, or a syft document sitting after something else in the same stream, where `format.Decode` went on to return an empty SBOM with a nil error. Two O(1) checks: - the first non-space byte at the reader's position has to be `{` - `Decode` errors when the parsed document has no `schema.version`, instead of handing back the empty document it decoded into Streams holding several concatenated documents are deliberately not detected. Proving there is exactly one top-level value means reading all of it, which is the cost the tail path exists to avoid, so `Identify` can still name a later document's version on such a file. `Decode` reports what it actually parsed. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix(convert): recognize the input file by identity rather than by path `convert - -o syft-json=sbom.json < sbom.json` truncated the file and exited 0. The passthrough guard compared paths, and a shell redirect does not have one, so it was blind to the one input mode where the truncation was silent. `openConvertInput` now returns the input's `os.FileInfo`, which the STDIN branch already had from its `Stat`, and `isSameFile` compares that instead. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> --------- Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>