mirror of
https://github.com/anchore/syft.git
synced 2026-10-30 14:02:26 +01:00
feat(convert): passthrough-exact-format copies matching syft-json input unchanged (#5279)
* feat(convert): passthrough-exact-format copies matching syft-json input unchanged When a syft-json input SBOM already exactly matches the requested output format and schema version, `syft convert` can write the input bytes out verbatim instead of decoding and re-encoding them. This avoids the lossy round trip (and the associated cost) for callers that normalise SBOMs into a target format without knowing up front whether the source already matches. The behaviour is opt-in and configuration-only: it is enabled with convert.passthrough-exact-format in the config file or the SYFT_CONVERT_PASSTHROUGH_EXACT_FORMAT environment variable, and there is deliberately no command-line flag, since it changes how every invocation in a pipeline behaves rather than varying call by call. It defaults to off, so existing behaviour is unchanged. Pass-through applies only to syft-json input, and only when the document's schema version is identical to the one this build of syft produces; any other input is always converted. Outputs that do not match are still converted, so a mixed set of `-o` values behaves as expected, and every output is resolved and validated before anything is written. Pass-through output is a byte-for-byte copy, so format options such as `format.pretty` and `format.json.legacy` are not applied to it. When the option is enabled a warning is logged stating so and noting the syft-json-only scope, and a second warning is logged whenever an input is found to match and the shortcut is actually taken. The option lives under a `convert:` configuration section (options.Convert), mirroring how attest scopes its own options. The description documents the caveats above and notes that file-path input written with `-o <format>=<path>` keeps memory minimal, whereas piped input or STDOUT output must hold the whole document in memory. Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * perf(convert): stream pass-through output instead of buffering the input Identify, copy, and decode the input through a seekable reader rather than loading the whole document into memory first. File inputs are read in place and pass-through output is an io.Copy from the file. STDIN is used in place when it is a redirected regular file. A pipe or terminal cannot seek, so its contents are drained into an internal/spillbuf buffer: the first 1MB stays in memory and the rest goes to a temp file under the user's TMPDIR (created lazily, only when something actually spills), giving a seekable view without ever holding the document in memory. The temp dir is rooted per invocation via internal/tmpdir, honouring one already attached to the context, and is removed when the command returns. Writing pass-through output over the input file itself is now a no-op rather than truncating the file before it is copied. Measured on a 1GB syft-json document (Apple M1 Max), file in, file out: origin/main full re-encode 86.3s wall 99s CPU 12.0GB max RSS in-memory pass-through 7.4s wall 6.7s CPU 4.2GB max RSS streaming pass-through 7.0s wall 6.9s CPU 3.1GB max RSS and the same 1GB document piped over STDIN with pass-through enabled: io.ReadAll into memory 1.3s wall 0.8s CPU 2.7GB max RSS spill buffer 1.8s wall 1.0s CPU 66MB max RSS The remaining cost of the streaming pass-through is entirely format.Identify, which for syft-json decodes the whole document through a json.Decoder that buffers the full value: measured alone it takes 7.1s and 3.1GB on the same input. Reducing that further belongs to the format package. Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * perf(syftjson): identify documents from their tail instead of a full parse syft always writes the schema block as the last key of a syft-json document, so the format and version can be read from the final bytes of the file. The syft-json decoder's Identify now reads at most 2MB from the end of a seekable input, finds the last "schema" key there, and verifies it sits directly within the top-level object by scanning the remaining bytes for the single closing brace of the document. Anything the tail cannot vouch for (non-seekable input, a reordered document, a nested key) falls back to the existing full parse, so behaviour is unchanged for documents the fast path declines. The verification is position-independent: it also accepts documents whose keys have been reordered so that other small keys follow schema. It rejects nested schema objects, array-wrapped documents, truncated input, trailing content, and schema text inside strings. Supporting changes: - the offset-aware reader the decoder collection hands to decoders now supports io.SeekEnd, which the tail read requires - model.Document documents that Schema must remain its last field, and a test guards that the encoder writes it last in both compact and pretty output - the decoder list documents that syft-json must stay first Since Decode calls Identify before parsing, this also removes one of the two full parses every syft-json conversion used to perform. Measured on a 1GB syft-json document (Apple M1 Max), file in, file out: pass-through, previous commit 7.0s wall 6.9s CPU 3.1GB max RSS pass-through, this commit 0.85s wall 0.05s CPU 66MB max RSS full re-encode, previous commit 85.3s wall 99s CPU 11.5GB max RSS full re-encode, this commit 75.2s wall 91s CPU 9.9GB max RSS Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix(syftjson): only identify from the tail when the document starts where the reader does The tail scan vouched for the last 2MB, but nothing checked those bytes belonged to the document the reader was pointed at. So `Identify` started claiming syft-json for files it cannot decode: a document with a byte order mark or a stray log line in front of it, or a syft document sitting after something else in the same stream, where `format.Decode` went on to return an empty SBOM with a nil error. Two O(1) checks: - the first non-space byte at the reader's position has to be `{` - `Decode` errors when the parsed document has no `schema.version`, instead of handing back the empty document it decoded into Streams holding several concatenated documents are deliberately not detected. Proving there is exactly one top-level value means reading all of it, which is the cost the tail path exists to avoid, so `Identify` can still name a later document's version on such a file. `Decode` reports what it actually parsed. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix(convert): recognize the input file by identity rather than by path `convert - -o syft-json=sbom.json < sbom.json` truncated the file and exited 0. The passthrough guard compared paths, and a shell redirect does not have one, so it was blind to the one input mode where the truncation was silent. `openConvertInput` now returns the input's `os.FileInfo`, which the STDIN branch already had from its `Stat`, and `isSameFile` compares that instead. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> --------- Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
This commit is contained in:
@@ -1,24 +1,34 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"github.com/anchore/clio"
|
||||
"github.com/anchore/go-homedir"
|
||||
"github.com/anchore/syft/cmd/syft/internal/options"
|
||||
"github.com/anchore/syft/cmd/syft/internal/ui"
|
||||
"github.com/anchore/syft/internal"
|
||||
"github.com/anchore/syft/internal/bus"
|
||||
"github.com/anchore/syft/internal/log"
|
||||
"github.com/anchore/syft/internal/spillbuf"
|
||||
"github.com/anchore/syft/internal/tmpdir"
|
||||
"github.com/anchore/syft/syft/format"
|
||||
"github.com/anchore/syft/syft/format/syftjson"
|
||||
"github.com/anchore/syft/syft/sbom"
|
||||
)
|
||||
|
||||
const (
|
||||
convertExample = ` {{.appName}} {{.command}} img.syft.json -o spdx-json convert a syft SBOM to spdx-json, output goes to stdout
|
||||
{{.appName}} {{.command}} img.syft.json -o cyclonedx-json=img.cdx.json convert a syft SBOM to CycloneDX, output is written to the file "img.cdx.json"
|
||||
{{.appName}} {{.command}} - -o spdx-json convert an SBOM from STDIN to spdx-json
|
||||
SYFT_CONVERT_PASSTHROUGH_EXACT_FORMAT=true {{.appName}} {{.command}} img.syft.json -o syft-json=out.syft.json copy the input to "out.syft.json" unchanged if it is already syft-json at the current schema version
|
||||
`
|
||||
)
|
||||
|
||||
@@ -26,6 +36,7 @@ type ConvertOptions struct {
|
||||
options.Config `yaml:",inline" mapstructure:",squash"`
|
||||
options.Output `yaml:",inline" mapstructure:",squash"`
|
||||
options.UpdateCheck `yaml:",inline" mapstructure:",squash"`
|
||||
Convert options.Convert `yaml:"convert" json:"convert" mapstructure:"convert"`
|
||||
}
|
||||
|
||||
func Convert(app clio.Application) *cobra.Command {
|
||||
@@ -34,6 +45,7 @@ func Convert(app clio.Application) *cobra.Command {
|
||||
opts := &ConvertOptions{
|
||||
UpdateCheck: options.DefaultUpdateCheck(),
|
||||
Output: options.DefaultOutput(),
|
||||
Convert: options.DefaultConvert(),
|
||||
}
|
||||
|
||||
return app.SetupCommand(&cobra.Command{
|
||||
@@ -46,11 +58,11 @@ func Convert(app clio.Application) *cobra.Command {
|
||||
}),
|
||||
Args: validateConvertArgs,
|
||||
PreRunE: applicationUpdateCheck(id, &opts.UpdateCheck),
|
||||
RunE: func(_ *cobra.Command, args []string) error {
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
restoreStdout := ui.CaptureStdoutToTraceLog()
|
||||
defer restoreStdout()
|
||||
|
||||
return RunConvert(opts, args[0])
|
||||
return RunConvert(cmd.Context(), opts, args[0])
|
||||
},
|
||||
}, opts)
|
||||
}
|
||||
@@ -59,30 +71,63 @@ func validateConvertArgs(cmd *cobra.Command, args []string) error {
|
||||
return validateArgs(cmd, args, "an SBOM argument is required")
|
||||
}
|
||||
|
||||
func RunConvert(opts *ConvertOptions, userInput string) error {
|
||||
func RunConvert(ctx context.Context, opts *ConvertOptions, userInput string) error {
|
||||
log.Warn("convert is an experimental feature, run `syft convert -h` for help")
|
||||
|
||||
writer, err := opts.SBOMWriter()
|
||||
// a temp dir is only needed when STDIN is a pipe and has to be spilled to disk; it is created lazily on
|
||||
// first use (under the user's TMPDIR), so a file input never touches it. Mirrors syft.CreateSBOM, which
|
||||
// also honors a TempDir already attached to the context.
|
||||
td := tmpdir.FromContext(ctx)
|
||||
if td == nil {
|
||||
_, td = tmpdir.Root(ctx, "syft-convert")
|
||||
defer func() {
|
||||
_ = td.Cleanup()
|
||||
}()
|
||||
}
|
||||
|
||||
reader, inputInfo, err := openConvertInput(userInput, td)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() {
|
||||
_ = reader.Close()
|
||||
}()
|
||||
|
||||
var reader io.ReadSeekCloser
|
||||
outputs := opts.Outputs
|
||||
var unchanged []string
|
||||
if opts.Convert.PassthroughExactFormat {
|
||||
log.Warn("convert.passthrough-exact-format is enabled: syft-json input that already exactly matches a requested output format and schema version will be copied to that output unchanged (this does not apply to any other input format)")
|
||||
|
||||
if userInput == "-" {
|
||||
// though os.Stdin is an os.File, it does not support seeking
|
||||
// you will get errors such as "seek /dev/stdin: illegal seek".
|
||||
// We need to buffer what we read.
|
||||
reader = internal.NewBufferedSeeker(os.Stdin)
|
||||
} else {
|
||||
f, err := os.Open(userInput)
|
||||
unchanged, outputs, err = partitionOutputsBySourceFormat(opts.Output, reader)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to open SBOM file: %w", err)
|
||||
return err
|
||||
}
|
||||
defer func() {
|
||||
_ = f.Close()
|
||||
}()
|
||||
reader = f
|
||||
}
|
||||
|
||||
// resolve (and validate) every output that needs converting before writing anything, so that
|
||||
// a bad -o value fails the command without leaving partial results behind
|
||||
var writer sbom.Writer
|
||||
if len(outputs) > 0 {
|
||||
outputOpts := opts.Output
|
||||
outputOpts.Outputs = outputs
|
||||
writer, err = outputOpts.SBOMWriter()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
for _, output := range unchanged {
|
||||
if err := writeUnchanged(output, opts.LegacyFile, inputInfo, reader); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if writer == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
if _, err := reader.Seek(0, io.SeekStart); err != nil {
|
||||
return fmt.Errorf("unable to seek to start of SBOM: %w", err)
|
||||
}
|
||||
|
||||
s, _, _, err := format.Decode(reader)
|
||||
@@ -100,3 +145,171 @@ func RunConvert(opts *ConvertOptions, userInput string) error {
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// openConvertInput opens the SBOM document at the given path, or STDIN when "-". The returned reader is seekable so
|
||||
// the document can be identified, copied, and decoded in turn without being loaded into memory. The returned
|
||||
// os.FileInfo describes the file the document is being read from, so that writing back over it can be recognized;
|
||||
// it is nil when the input is not a file (a pipe or a terminal on STDIN).
|
||||
func openConvertInput(userInput string, td *tmpdir.TempDir) (io.ReadSeekCloser, os.FileInfo, error) {
|
||||
if userInput == "-" {
|
||||
return openStdin(td)
|
||||
}
|
||||
|
||||
f, err := os.Open(userInput)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("failed to open SBOM file: %w", err)
|
||||
}
|
||||
|
||||
info, err := f.Stat()
|
||||
if err != nil {
|
||||
// the document is readable, we just cannot recognize it as an output destination
|
||||
log.WithFields("error", err, "path", userInput).Debug("unable to stat SBOM file")
|
||||
}
|
||||
return f, info, nil
|
||||
}
|
||||
|
||||
// openStdin returns a seekable view of STDIN. When STDIN is a regular file (such as a shell redirect) it is read in
|
||||
// place. A pipe or terminal cannot seek (you will get errors such as "seek /dev/stdin: illegal seek"), so its
|
||||
// contents are drained into a spill buffer: a small head stays in memory and everything beyond it goes to a temp
|
||||
// file under td, so an arbitrarily large piped SBOM never has to be held in memory to be made seekable.
|
||||
func openStdin(td *tmpdir.TempDir) (io.ReadSeekCloser, os.FileInfo, error) {
|
||||
if info, err := os.Stdin.Stat(); err == nil && info.Mode().IsRegular() {
|
||||
if start, err := os.Stdin.Seek(0, io.SeekCurrent); err == nil {
|
||||
// a redirect reads the file in place, so it is just as much an input file as a path argument is
|
||||
return readSeekNopCloser{io.NewSectionReader(os.Stdin, start, info.Size()-start)}, info, nil
|
||||
}
|
||||
}
|
||||
|
||||
buf := spillbuf.New(td)
|
||||
if _, err := io.Copy(io.NewOffsetWriter(buf, 0), os.Stdin); err != nil {
|
||||
_ = buf.Close()
|
||||
return nil, nil, fmt.Errorf("failed to read SBOM from STDIN: %w", err)
|
||||
}
|
||||
|
||||
return &spilledReader{
|
||||
SectionReader: io.NewSectionReader(buf, 0, buf.Size()),
|
||||
buf: buf,
|
||||
}, nil, nil
|
||||
}
|
||||
|
||||
type readSeekNopCloser struct {
|
||||
io.ReadSeeker
|
||||
}
|
||||
|
||||
func (readSeekNopCloser) Close() error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// spilledReader is a seekable view over a spill buffer; closing it releases the buffer's memory and spill file.
|
||||
type spilledReader struct {
|
||||
*io.SectionReader
|
||||
buf *spillbuf.Buffer
|
||||
}
|
||||
|
||||
func (r *spilledReader) Close() error {
|
||||
return r.buf.Close()
|
||||
}
|
||||
|
||||
// partitionOutputsBySourceFormat splits the requested outputs (in "<format>[@<version>][=<path>]" form) into those
|
||||
// that exactly match the format and schema version of a syft-json input document (which can be written out
|
||||
// unchanged) and those that still require conversion. Input in any other format is always converted. Outputs that
|
||||
// cannot be resolved to an encoder are left for the writer to report on.
|
||||
func partitionOutputsBySourceFormat(output options.Output, reader io.ReadSeeker) (unchanged []string, toConvert []string, err error) {
|
||||
if _, err := reader.Seek(0, io.SeekStart); err != nil {
|
||||
return nil, nil, fmt.Errorf("unable to seek to start of SBOM: %w", err)
|
||||
}
|
||||
|
||||
id, version := format.Identify(reader)
|
||||
if id == "" || version == "" {
|
||||
// let decoding surface the appropriate error
|
||||
return nil, output.Outputs, nil
|
||||
}
|
||||
|
||||
if id != syftjson.ID {
|
||||
log.WithFields("format", id, "version", version).Debug("input is not syft-json, passthrough does not apply")
|
||||
return nil, output.Outputs, nil
|
||||
}
|
||||
|
||||
encoders, err := output.Encoders()
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
collection := format.NewEncoderCollection(encoders...)
|
||||
|
||||
for _, out := range output.Outputs {
|
||||
name, _, _ := strings.Cut(strings.TrimSpace(out), "=")
|
||||
enc := collection.GetByString(name)
|
||||
if enc != nil && enc.ID() == id && enc.Version() == version {
|
||||
log.WithFields("format", id, "version", version, "output", out).Warn("input already exactly matches the requested output format and schema version, copying it unchanged instead of converting it")
|
||||
unchanged = append(unchanged, out)
|
||||
continue
|
||||
}
|
||||
toConvert = append(toConvert, out)
|
||||
}
|
||||
|
||||
return unchanged, toConvert, nil
|
||||
}
|
||||
|
||||
// writeUnchanged copies the SBOM document as-is to the destination described by the given "<format>[=<path>]"
|
||||
// output value, falling back to the (deprecated) --file path, and finally to the report bus (STDOUT). When the
|
||||
// destination is the input file itself there is nothing to do.
|
||||
func writeUnchanged(output string, defaultFile string, inputInfo os.FileInfo, reader io.ReadSeeker) error {
|
||||
if _, err := reader.Seek(0, io.SeekStart); err != nil {
|
||||
return fmt.Errorf("unable to seek to start of SBOM: %w", err)
|
||||
}
|
||||
|
||||
_, path, _ := strings.Cut(strings.TrimSpace(output), "=")
|
||||
if path == "" {
|
||||
path = defaultFile
|
||||
}
|
||||
|
||||
if path == "" {
|
||||
// the report bus only carries strings, so STDOUT output must be materialized in memory
|
||||
content, err := io.ReadAll(reader) //nolint:gocritic // the whole SBOM is the report; there is no meaningful bound
|
||||
if err != nil {
|
||||
return fmt.Errorf("unable to read SBOM: %w", err)
|
||||
}
|
||||
bus.Report(string(content))
|
||||
return nil
|
||||
}
|
||||
|
||||
expandedPath, err := homedir.Expand(path)
|
||||
if err != nil {
|
||||
log.Warnf("could not expand given writer output path=%q: %w", path, err)
|
||||
expandedPath = path
|
||||
}
|
||||
|
||||
if isSameFile(inputInfo, expandedPath) {
|
||||
log.WithFields("path", expandedPath).Info("output is the input file, leaving it as-is")
|
||||
return nil
|
||||
}
|
||||
|
||||
if err := os.MkdirAll(filepath.Dir(expandedPath), 0755); err != nil {
|
||||
return fmt.Errorf("unable to create report directory: %w", err)
|
||||
}
|
||||
|
||||
f, err := os.OpenFile(expandedPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
|
||||
if err != nil {
|
||||
return fmt.Errorf("unable to create report file: %w", err)
|
||||
}
|
||||
|
||||
if _, err := io.Copy(f, reader); err != nil {
|
||||
_ = f.Close()
|
||||
return fmt.Errorf("unable to write report file: %w", err)
|
||||
}
|
||||
|
||||
return f.Close()
|
||||
}
|
||||
|
||||
// isSameFile reports whether the given path names the file the input is being read from. Comparing the input's
|
||||
// os.FileInfo rather than its path covers a shell redirect onto STDIN, which has no path to compare.
|
||||
func isSameFile(inputInfo os.FileInfo, path string) bool {
|
||||
if inputInfo == nil {
|
||||
return false
|
||||
}
|
||||
pathInfo, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
return os.SameFile(inputInfo, pathInfo)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,448 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"github.com/wagoodman/go-partybus"
|
||||
|
||||
"github.com/anchore/syft/cmd/syft/internal/options"
|
||||
"github.com/anchore/syft/internal/bus"
|
||||
"github.com/anchore/syft/internal/spillbuf"
|
||||
"github.com/anchore/syft/syft/event"
|
||||
"github.com/anchore/syft/syft/file"
|
||||
"github.com/anchore/syft/syft/format"
|
||||
"github.com/anchore/syft/syft/format/cyclonedxjson"
|
||||
"github.com/anchore/syft/syft/format/spdxjson"
|
||||
"github.com/anchore/syft/syft/format/syftjson"
|
||||
"github.com/anchore/syft/syft/pkg"
|
||||
"github.com/anchore/syft/syft/sbom"
|
||||
"github.com/anchore/syft/syft/source"
|
||||
)
|
||||
|
||||
func convertTestSBOM() sbom.SBOM {
|
||||
catalog := pkg.NewCollection()
|
||||
catalog.Add(pkg.Package{
|
||||
Name: "musl-utils",
|
||||
Version: "1.2.3-r4",
|
||||
Type: pkg.ApkPkg,
|
||||
FoundBy: "apk-db-cataloger",
|
||||
Locations: file.NewLocationSet(
|
||||
file.NewLocation("/lib/apk/db/installed"),
|
||||
),
|
||||
PURL: "pkg:apk/alpine/musl-utils@1.2.3-r4",
|
||||
})
|
||||
|
||||
return sbom.SBOM{
|
||||
Artifacts: sbom.Artifacts{
|
||||
Packages: catalog,
|
||||
},
|
||||
Source: source.Description{
|
||||
ID: "some-id",
|
||||
Name: "some-dir",
|
||||
Version: "latest",
|
||||
Metadata: source.DirectoryMetadata{
|
||||
Path: "/some/path",
|
||||
},
|
||||
},
|
||||
Descriptor: sbom.Descriptor{
|
||||
Name: "syft",
|
||||
Version: "v0.0.0-test",
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func mustConvertEncoder(enc sbom.FormatEncoder, err error) sbom.FormatEncoder {
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return enc
|
||||
}
|
||||
|
||||
func encodeConvertTestSBOM(t *testing.T, enc sbom.FormatEncoder) []byte {
|
||||
buf := &bytes.Buffer{}
|
||||
require.NoError(t, enc.Encode(buf, convertTestSBOM()))
|
||||
return buf.Bytes()
|
||||
}
|
||||
|
||||
// prettySyftJSON returns the test SBOM as pretty-printed syft-json. Pretty input makes a verbatim copy
|
||||
// distinguishable from a re-encode, since re-encoding always produces compact output here.
|
||||
func prettySyftJSON(t *testing.T) []byte {
|
||||
cfg := syftjson.DefaultEncoderConfig()
|
||||
cfg.Pretty = true
|
||||
return encodeConvertTestSBOM(t, mustConvertEncoder(syftjson.NewFormatEncoderWithConfig(cfg)))
|
||||
}
|
||||
|
||||
func Test_partitionOutputsBySourceFormat(t *testing.T) {
|
||||
syftEncoder := mustConvertEncoder(syftjson.NewFormatEncoderWithConfig(syftjson.DefaultEncoderConfig()))
|
||||
syftVersion := syftEncoder.Version()
|
||||
syftContent := encodeConvertTestSBOM(t, syftEncoder)
|
||||
|
||||
// the same document claiming an older schema version
|
||||
olderSyftContent := bytes.ReplaceAll(syftContent, []byte(`"version":"`+syftVersion+`"`), []byte(`"version":"16.0.0"`))
|
||||
require.NotEqual(t, syftContent, olderSyftContent)
|
||||
|
||||
cdxContent := encodeConvertTestSBOM(t, mustConvertEncoder(cyclonedxjson.NewFormatEncoderWithConfig(cyclonedxjson.DefaultEncoderConfig())))
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
content []byte
|
||||
outputs []string
|
||||
wantUnchanged []string
|
||||
wantToConvert []string
|
||||
}{
|
||||
{
|
||||
name: "syft-json matches the bare format name",
|
||||
content: syftContent,
|
||||
outputs: []string{"syft-json"},
|
||||
wantUnchanged: []string{"syft-json"},
|
||||
},
|
||||
{
|
||||
name: "syft-json matches its alias",
|
||||
content: syftContent,
|
||||
outputs: []string{"json"},
|
||||
wantUnchanged: []string{"json"},
|
||||
},
|
||||
{
|
||||
name: "syft-json matches an explicit version and file path",
|
||||
content: syftContent,
|
||||
outputs: []string{"syft-json@" + syftVersion + "=out/some.syft.json"},
|
||||
wantUnchanged: []string{"syft-json@" + syftVersion + "=out/some.syft.json"},
|
||||
},
|
||||
{
|
||||
name: "syft-json at an older schema version requires conversion",
|
||||
content: olderSyftContent,
|
||||
outputs: []string{"syft-json"},
|
||||
wantToConvert: []string{"syft-json"},
|
||||
},
|
||||
{
|
||||
name: "a different output format requires conversion",
|
||||
content: syftContent,
|
||||
outputs: []string{"spdx-json"},
|
||||
wantToConvert: []string{"spdx-json"},
|
||||
},
|
||||
{
|
||||
name: "mixed outputs are partitioned",
|
||||
content: syftContent,
|
||||
outputs: []string{"syft-json=a.json", "spdx-json=b.json", "cyclonedx-xml"},
|
||||
wantUnchanged: []string{"syft-json=a.json"},
|
||||
wantToConvert: []string{"spdx-json=b.json", "cyclonedx-xml"},
|
||||
},
|
||||
{
|
||||
name: "unknown format is left for the writer to report",
|
||||
content: syftContent,
|
||||
outputs: []string{"bogus"},
|
||||
wantToConvert: []string{"bogus"},
|
||||
},
|
||||
{
|
||||
name: "non syft-json input is always converted, even when it matches the output",
|
||||
content: cdxContent,
|
||||
outputs: []string{"cyclonedx-json"},
|
||||
wantToConvert: []string{"cyclonedx-json"},
|
||||
},
|
||||
{
|
||||
name: "unidentifiable content is left for the decoder to report",
|
||||
content: []byte("definitely not an sbom"),
|
||||
outputs: []string{"syft-json"},
|
||||
wantToConvert: []string{"syft-json"},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
output := options.DefaultOutput()
|
||||
output.Outputs = tt.outputs
|
||||
|
||||
unchanged, toConvert, err := partitionOutputsBySourceFormat(output, bytes.NewReader(tt.content))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, tt.wantUnchanged, unchanged)
|
||||
assert.Equal(t, tt.wantToConvert, toConvert)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func Test_RunConvert_passthroughExactFormat(t *testing.T) {
|
||||
syftContent := prettySyftJSON(t)
|
||||
|
||||
newOpts := func(passthrough bool, outputs ...string) *ConvertOptions {
|
||||
opts := &ConvertOptions{
|
||||
Output: options.DefaultOutput(),
|
||||
Convert: options.Convert{
|
||||
PassthroughExactFormat: passthrough,
|
||||
},
|
||||
}
|
||||
opts.Outputs = outputs
|
||||
return opts
|
||||
}
|
||||
|
||||
writeInput := func(t *testing.T, content []byte) string {
|
||||
input := filepath.Join(t.TempDir(), "input.json")
|
||||
require.NoError(t, os.WriteFile(input, content, 0600))
|
||||
return input
|
||||
}
|
||||
|
||||
t.Run("matching output is copied unchanged while others are converted", func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
syftOut := filepath.Join(dir, "out.syft.json")
|
||||
spdxOut := filepath.Join(dir, "out.spdx.json")
|
||||
|
||||
opts := newOpts(true, "syft-json="+syftOut, "spdx-json="+spdxOut)
|
||||
require.NoError(t, RunConvert(context.Background(), opts, writeInput(t, syftContent)))
|
||||
|
||||
got, err := os.ReadFile(syftOut)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, syftContent, got)
|
||||
|
||||
spdx, err := os.ReadFile(spdxOut)
|
||||
require.NoError(t, err)
|
||||
id, _ := format.Identify(bytes.NewReader(spdx))
|
||||
assert.Equal(t, spdxjson.ID, id)
|
||||
})
|
||||
|
||||
t.Run("STDIN input is copied unchanged while others are converted", func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
syftOut := filepath.Join(dir, "out.syft.json")
|
||||
spdxOut := filepath.Join(dir, "out.spdx.json")
|
||||
|
||||
stdin, err := os.Open(writeInput(t, syftContent))
|
||||
require.NoError(t, err)
|
||||
defer stdin.Close()
|
||||
|
||||
originalStdin := os.Stdin
|
||||
os.Stdin = stdin
|
||||
t.Cleanup(func() { os.Stdin = originalStdin })
|
||||
|
||||
opts := newOpts(true, "syft-json="+syftOut, "spdx-json="+spdxOut)
|
||||
require.NoError(t, RunConvert(context.Background(), opts, "-"))
|
||||
|
||||
got, err := os.ReadFile(syftOut)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, syftContent, got)
|
||||
|
||||
spdx, err := os.ReadFile(spdxOut)
|
||||
require.NoError(t, err)
|
||||
id, _ := format.Identify(bytes.NewReader(spdx))
|
||||
assert.Equal(t, spdxjson.ID, id)
|
||||
})
|
||||
|
||||
t.Run("piped STDIN input is copied unchanged while others are converted", func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
syftOut := filepath.Join(dir, "out.syft.json")
|
||||
spdxOut := filepath.Join(dir, "out.spdx.json")
|
||||
|
||||
pipeReader, pipeWriter, err := os.Pipe()
|
||||
require.NoError(t, err)
|
||||
defer pipeReader.Close()
|
||||
go func() {
|
||||
defer pipeWriter.Close()
|
||||
_, _ = pipeWriter.Write(syftContent)
|
||||
}()
|
||||
|
||||
originalStdin := os.Stdin
|
||||
os.Stdin = pipeReader
|
||||
t.Cleanup(func() { os.Stdin = originalStdin })
|
||||
|
||||
opts := newOpts(true, "syft-json="+syftOut, "spdx-json="+spdxOut)
|
||||
require.NoError(t, RunConvert(context.Background(), opts, "-"))
|
||||
|
||||
got, err := os.ReadFile(syftOut)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, syftContent, got)
|
||||
|
||||
spdx, err := os.ReadFile(spdxOut)
|
||||
require.NoError(t, err)
|
||||
id, _ := format.Identify(bytes.NewReader(spdx))
|
||||
assert.Equal(t, spdxjson.ID, id)
|
||||
})
|
||||
|
||||
t.Run("without the option the matching output is re-encoded", func(t *testing.T) {
|
||||
syftOut := filepath.Join(t.TempDir(), "out.syft.json")
|
||||
|
||||
opts := newOpts(false, "syft-json="+syftOut)
|
||||
require.NoError(t, RunConvert(context.Background(), opts, writeInput(t, syftContent)))
|
||||
|
||||
got, err := os.ReadFile(syftOut)
|
||||
require.NoError(t, err)
|
||||
id, _ := format.Identify(bytes.NewReader(got))
|
||||
assert.Equal(t, syftjson.ID, id)
|
||||
// the input was pretty-printed and the encoder is not configured to be, so a re-encode cannot reproduce it
|
||||
assert.NotEqual(t, syftContent, got)
|
||||
})
|
||||
|
||||
t.Run("non syft-json input is re-encoded even when it matches the output", func(t *testing.T) {
|
||||
cdxContent := encodeConvertTestSBOM(t, mustConvertEncoder(cyclonedxjson.NewFormatEncoderWithConfig(cyclonedxjson.DefaultEncoderConfig())))
|
||||
cdxOut := filepath.Join(t.TempDir(), "out.cdx.json")
|
||||
|
||||
opts := newOpts(true, "cyclonedx-json="+cdxOut)
|
||||
require.NoError(t, RunConvert(context.Background(), opts, writeInput(t, cdxContent)))
|
||||
|
||||
got, err := os.ReadFile(cdxOut)
|
||||
require.NoError(t, err)
|
||||
id, _ := format.Identify(bytes.NewReader(got))
|
||||
assert.Equal(t, cyclonedxjson.ID, id)
|
||||
// a fresh serial number is minted on every encode, so a round trip never reproduces the input
|
||||
assert.NotEqual(t, cdxContent, got)
|
||||
})
|
||||
|
||||
t.Run("the deprecated --file path is honored", func(t *testing.T) {
|
||||
syftOut := filepath.Join(t.TempDir(), "out.syft.json")
|
||||
|
||||
opts := newOpts(true, "syft-json")
|
||||
opts.LegacyFile = syftOut
|
||||
require.NoError(t, RunConvert(context.Background(), opts, writeInput(t, syftContent)))
|
||||
|
||||
got, err := os.ReadFile(syftOut)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, syftContent, got)
|
||||
})
|
||||
|
||||
t.Run("writing over the input file leaves it intact", func(t *testing.T) {
|
||||
input := writeInput(t, syftContent)
|
||||
|
||||
opts := newOpts(true, "syft-json="+input)
|
||||
require.NoError(t, RunConvert(context.Background(), opts, input))
|
||||
|
||||
got, err := os.ReadFile(input)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, syftContent, got)
|
||||
})
|
||||
|
||||
t.Run("writing over a redirected STDIN input leaves it intact", func(t *testing.T) {
|
||||
input := writeInput(t, syftContent)
|
||||
|
||||
stdin, err := os.Open(input)
|
||||
require.NoError(t, err)
|
||||
defer stdin.Close()
|
||||
|
||||
originalStdin := os.Stdin
|
||||
os.Stdin = stdin
|
||||
t.Cleanup(func() { os.Stdin = originalStdin })
|
||||
|
||||
opts := newOpts(true, "syft-json="+input)
|
||||
require.NoError(t, RunConvert(context.Background(), opts, "-"))
|
||||
|
||||
got, err := os.ReadFile(input)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, syftContent, got)
|
||||
})
|
||||
|
||||
t.Run("an invalid output fails before anything is written", func(t *testing.T) {
|
||||
syftOut := filepath.Join(t.TempDir(), "out.syft.json")
|
||||
|
||||
opts := newOpts(true, "syft-json="+syftOut, "bogus")
|
||||
require.Error(t, RunConvert(context.Background(), opts, writeInput(t, syftContent)))
|
||||
|
||||
assert.NoFileExists(t, syftOut)
|
||||
})
|
||||
|
||||
t.Run("a missing input file is reported", func(t *testing.T) {
|
||||
syftOut := filepath.Join(t.TempDir(), "out.syft.json")
|
||||
|
||||
opts := newOpts(true, "syft-json="+syftOut)
|
||||
err := RunConvert(context.Background(), opts, filepath.Join(t.TempDir(), "does-not-exist.json"))
|
||||
require.ErrorContains(t, err, "failed to open SBOM file")
|
||||
|
||||
assert.NoFileExists(t, syftOut)
|
||||
})
|
||||
|
||||
t.Run("unrecognized input is left to the decoder to reject", func(t *testing.T) {
|
||||
syftOut := filepath.Join(t.TempDir(), "out.syft.json")
|
||||
|
||||
opts := newOpts(true, "syft-json="+syftOut)
|
||||
err := RunConvert(context.Background(), opts, writeInput(t, []byte("definitely not an sbom")))
|
||||
require.ErrorContains(t, err, "failed to decode SBOM")
|
||||
})
|
||||
}
|
||||
|
||||
// largeSyftJSON returns a pretty-printed syft-json document of at least minBytes, built from many small packages.
|
||||
func largeSyftJSON(t *testing.T, minBytes int64) []byte {
|
||||
catalog := pkg.NewCollection()
|
||||
for i := 0; i < int(minBytes/100); i++ {
|
||||
catalog.Add(pkg.Package{
|
||||
Name: fmt.Sprintf("pkg-%d", i),
|
||||
Version: "1.0.0",
|
||||
Type: pkg.ApkPkg,
|
||||
PURL: fmt.Sprintf("pkg:apk/alpine/pkg-%d@1.0.0", i),
|
||||
})
|
||||
}
|
||||
s := convertTestSBOM()
|
||||
s.Artifacts.Packages = catalog
|
||||
|
||||
cfg := syftjson.DefaultEncoderConfig()
|
||||
cfg.Pretty = true
|
||||
enc := mustConvertEncoder(syftjson.NewFormatEncoderWithConfig(cfg))
|
||||
|
||||
buf := &bytes.Buffer{}
|
||||
require.NoError(t, enc.Encode(buf, s))
|
||||
require.Greater(t, int64(buf.Len()), minBytes)
|
||||
return buf.Bytes()
|
||||
}
|
||||
|
||||
func Test_RunConvert_pipedInputLargerThanMemoryLimitSpillsToDisk(t *testing.T) {
|
||||
// a piped document bigger than the spill buffer's in-memory tier is written to a temp file under TMPDIR;
|
||||
// the conversion must identify, copy and clean up from there rather than holding the document in memory
|
||||
dir := t.TempDir()
|
||||
syftOut := filepath.Join(dir, "out.syft.json")
|
||||
spillRoot := filepath.Join(dir, "spill")
|
||||
require.NoError(t, os.Mkdir(spillRoot, 0o755))
|
||||
t.Setenv("TMPDIR", spillRoot)
|
||||
|
||||
content := largeSyftJSON(t, 2*spillbuf.DefaultMemLimit)
|
||||
|
||||
pipeReader, pipeWriter, err := os.Pipe()
|
||||
require.NoError(t, err)
|
||||
defer pipeReader.Close()
|
||||
go func() {
|
||||
defer pipeWriter.Close()
|
||||
_, _ = pipeWriter.Write(content)
|
||||
}()
|
||||
|
||||
originalStdin := os.Stdin
|
||||
os.Stdin = pipeReader
|
||||
t.Cleanup(func() { os.Stdin = originalStdin })
|
||||
|
||||
opts := &ConvertOptions{
|
||||
Output: options.DefaultOutput(),
|
||||
Convert: options.Convert{
|
||||
PassthroughExactFormat: true,
|
||||
},
|
||||
}
|
||||
opts.Outputs = []string{"syft-json=" + syftOut}
|
||||
require.NoError(t, RunConvert(context.Background(), opts, "-"))
|
||||
|
||||
got, err := os.ReadFile(syftOut)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, content, got)
|
||||
|
||||
entries, err := os.ReadDir(spillRoot)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, entries, "the spill directory must be removed once the conversion finishes")
|
||||
}
|
||||
|
||||
func Test_writeUnchanged_stdout(t *testing.T) {
|
||||
// with no output path the document is published on the report bus, which is how the CLI writes to STDOUT
|
||||
b := partybus.NewBus()
|
||||
subscription := b.Subscribe()
|
||||
bus.Set(b)
|
||||
t.Cleanup(func() {
|
||||
bus.Set(nil)
|
||||
})
|
||||
|
||||
content := []byte(`{"some":"document"}`)
|
||||
require.NoError(t, writeUnchanged("syft-json", "", nil, bytes.NewReader(content)))
|
||||
|
||||
select {
|
||||
case e := <-subscription.Events():
|
||||
assert.Equal(t, event.CLIReport, e.Type)
|
||||
assert.Equal(t, string(content), e.Value)
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("no report was published")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
package options
|
||||
|
||||
import (
|
||||
"github.com/anchore/clio"
|
||||
)
|
||||
|
||||
var _ clio.FieldDescriber = (*Convert)(nil)
|
||||
|
||||
// Convert holds the options specific to the convert command. These are deliberately configuration-only (config file
|
||||
// or environment) rather than flags: they change how the command behaves for every invocation in a pipeline and are
|
||||
// not expected to vary from one call to the next.
|
||||
type Convert struct {
|
||||
PassthroughExactFormat bool `yaml:"passthrough-exact-format" json:"passthrough-exact-format" mapstructure:"passthrough-exact-format"`
|
||||
}
|
||||
|
||||
func DefaultConvert() Convert {
|
||||
return Convert{
|
||||
PassthroughExactFormat: false,
|
||||
}
|
||||
}
|
||||
|
||||
func (o *Convert) DescribeFields(descriptions clio.FieldDescriptionSet) {
|
||||
descriptions.Add(&o.PassthroughExactFormat, `when a syft-json input SBOM already exactly matches the requested output format and schema version, copy it to that output unchanged instead of decoding and re-encoding it.
|
||||
Note: this only applies to syft-json input, and only when the document's schema version is identical to the schema version this version of syft produces. Any other input is always converted.
|
||||
Note: output written this way is a byte-for-byte copy of the input, so format options (such as 'format.pretty' or 'format.json.legacy') are not applied to it.
|
||||
Note: to keep memory use minimal on large documents, give the input as a file path (or a shell redirect) and write the output with '-o <format>=<path>'. Piped input, or output to STDOUT, requires the whole document to be held in memory.`)
|
||||
}
|
||||
@@ -21,6 +21,9 @@ func init() {
|
||||
|
||||
func Decoders() []sbom.FormatDecoder {
|
||||
return []sbom.FormatDecoder{
|
||||
// syft-json must remain the first decoder tried: it is by far the most common input, and its decoder can
|
||||
// identify a syft document from the tail of the file alone. Trying anything else first would pay for a full
|
||||
// parse of a large document by another decoder before reaching the common path.
|
||||
syftjson.NewFormatDecoder(),
|
||||
cyclonedxxml.NewFormatDecoder(),
|
||||
cyclonedxjson.NewFormatDecoder(),
|
||||
|
||||
@@ -72,8 +72,21 @@ func (o *offsetReadSeeker) Seek(offset int64, whence int) (int64, error) {
|
||||
}
|
||||
newOffset, err := o.rdr.Seek(offset, io.SeekCurrent)
|
||||
return newOffset - o.offset, err
|
||||
case io.SeekEnd:
|
||||
newOffset, err := o.rdr.Seek(offset, io.SeekEnd)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("cannot seek end: %w", err)
|
||||
}
|
||||
if newOffset < o.offset {
|
||||
// landed before the start of this view; move back to the start so the position stays valid
|
||||
if _, err := o.rdr.Seek(o.offset, io.SeekStart); err != nil {
|
||||
return 0, fmt.Errorf("cannot seek end: %w", err)
|
||||
}
|
||||
return 0, fmt.Errorf("cannot seek < 0")
|
||||
}
|
||||
return newOffset - o.offset, nil
|
||||
}
|
||||
return 0, fmt.Errorf("only SeekStart and SeekCurrent supported")
|
||||
return 0, fmt.Errorf("only SeekStart, SeekCurrent, and SeekEnd supported")
|
||||
}
|
||||
|
||||
var _ io.ReadSeeker = (*offsetReadSeeker)(nil)
|
||||
|
||||
@@ -179,11 +179,25 @@ func Test_offsetReadSeeker(t *testing.T) {
|
||||
whence: io.SeekCurrent,
|
||||
expected: "4",
|
||||
},
|
||||
{
|
||||
name: "seek end",
|
||||
input: abcd1234offset(2),
|
||||
seek: -4,
|
||||
whence: io.SeekEnd,
|
||||
expected: "1234",
|
||||
},
|
||||
{
|
||||
name: "seek end before the start of the view",
|
||||
input: abcd1234offset(2),
|
||||
seek: -7,
|
||||
whence: io.SeekEnd,
|
||||
wantErr: require.Error, // would be < offset, which is an error
|
||||
},
|
||||
{
|
||||
name: "bad whence",
|
||||
input: abcd1234,
|
||||
seek: 1,
|
||||
whence: io.SeekEnd,
|
||||
whence: 99,
|
||||
wantErr: require.Error,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
package syftjson
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
@@ -15,6 +17,12 @@ import (
|
||||
"github.com/anchore/syft/syft/sbom"
|
||||
)
|
||||
|
||||
// identifyTailSize is how much of the end of a document is inspected when identifying it from its trailing schema
|
||||
// block, before falling back to parsing the whole document.
|
||||
const identifyTailSize = 2 * 1024 * 1024
|
||||
|
||||
var schemaKey = []byte(`"schema"`)
|
||||
|
||||
var _ sbom.FormatDecoder = (*decoder)(nil)
|
||||
|
||||
type decoder struct{}
|
||||
@@ -45,6 +53,12 @@ func (d decoder) Decode(r io.Reader) (*sbom.SBOM, sbom.FormatID, string, error)
|
||||
return nil, "", "", fmt.Errorf("unable to decode syft-json document: %w", err)
|
||||
}
|
||||
|
||||
if doc.Schema.Version == "" {
|
||||
// Identify found a schema block that the parsed document does not have, so it was describing something
|
||||
// else in the stream; returning the empty document it decoded into would be a silently wrong answer
|
||||
return nil, "", "", fmt.Errorf("not a syft-json document")
|
||||
}
|
||||
|
||||
if err := checkSupportedSchema(doc.Schema.Version, internal.JSONSchemaVersion); err != nil {
|
||||
log.Warn(err)
|
||||
}
|
||||
@@ -57,6 +71,15 @@ func (d decoder) Identify(r io.Reader) (sbom.FormatID, string) {
|
||||
return "", ""
|
||||
}
|
||||
|
||||
// fast path: syft writes the schema block as the last key of the document (see model.Document), so for seekable
|
||||
// input the format and version can be read from the tail without parsing a potentially very large document.
|
||||
// Anything the tail cannot vouch for falls through to the full parse below.
|
||||
if rs, ok := r.(io.ReadSeeker); ok {
|
||||
if id, version, ok := identifyFromTail(rs); ok {
|
||||
return id, version
|
||||
}
|
||||
}
|
||||
|
||||
type Document struct {
|
||||
Schema model.Schema `json:"schema"`
|
||||
}
|
||||
@@ -78,6 +101,160 @@ func (d decoder) Identify(r io.Reader) (sbom.FormatID, string) {
|
||||
return ID, doc.Schema.Version
|
||||
}
|
||||
|
||||
// identifyFromTail reads at most identifyTailSize bytes from the end of the reader and attempts to identify the
|
||||
// document from a top-level schema block found there. The reader is returned to its original position afterwards.
|
||||
func identifyFromTail(rs io.ReadSeeker) (sbom.FormatID, string, bool) {
|
||||
start, err := rs.Seek(0, io.SeekCurrent)
|
||||
if err != nil {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
id, version, ok := readTail(rs, start)
|
||||
|
||||
if _, err := rs.Seek(start, io.SeekStart); err != nil {
|
||||
// without the original position the caller cannot safely parse the document afterwards
|
||||
log.WithFields("error", err).Debug("unable to restore reader position after inspecting document tail")
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
return id, version, ok
|
||||
}
|
||||
|
||||
// readTail does the seeking and reading for identifyFromTail, leaving the reader wherever it ends up.
|
||||
func readTail(rs io.ReadSeeker, start int64) (sbom.FormatID, string, bool) {
|
||||
end, err := rs.Seek(0, io.SeekEnd)
|
||||
if err != nil || end <= start {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
// the tail only describes the document the reader is pointed at if that document starts where the reader is,
|
||||
// so check the first byte: bytes ahead of it (a byte order mark, a log line prepended by a wrapper) would
|
||||
// otherwise be accepted silently, leaving Identify claiming a document that cannot be decoded
|
||||
if !opensObject(rs, start) {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
tailLen := min(end-start, int64(identifyTailSize))
|
||||
tail := make([]byte, tailLen)
|
||||
if _, err := rs.Seek(end-tailLen, io.SeekStart); err != nil {
|
||||
return "", "", false
|
||||
}
|
||||
if _, err := io.ReadFull(rs, tail); err != nil {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
return identifyFromTailBytes(tail)
|
||||
}
|
||||
|
||||
// opensObject reports whether the document at the given offset begins with a JSON object.
|
||||
func opensObject(rs io.ReadSeeker, start int64) bool {
|
||||
if _, err := rs.Seek(start, io.SeekStart); err != nil {
|
||||
return false
|
||||
}
|
||||
|
||||
head := make([]byte, 64)
|
||||
n, err := io.ReadFull(rs, head)
|
||||
if err != nil && !errors.Is(err, io.ErrUnexpectedEOF) {
|
||||
return false
|
||||
}
|
||||
|
||||
head = bytes.TrimLeft(head[:n], " \t\r\n")
|
||||
return len(head) > 0 && head[0] == '{'
|
||||
}
|
||||
|
||||
// identifyFromTailBytes looks for a syft schema block among the trailing bytes of a JSON document and, when one is
|
||||
// found directly within the top-level object, returns the format and schema version it declares. The bytes may
|
||||
// begin anywhere within the document, but must run to its very end.
|
||||
func identifyFromTailBytes(tail []byte) (sbom.FormatID, string, bool) {
|
||||
// find the last "schema" key. An unescaped quote cannot occur inside a JSON string, so a match whose opening
|
||||
// quote is not preceded by a backslash is a genuine string token (and a key if a colon follows it).
|
||||
idx := len(tail)
|
||||
for {
|
||||
idx = bytes.LastIndex(tail[:idx], schemaKey)
|
||||
if idx <= 0 {
|
||||
// not found, or found at the very start of the tail where we cannot tell whether it is escaped
|
||||
return "", "", false
|
||||
}
|
||||
if tail[idx-1] != '\\' {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
rest := bytes.TrimLeft(tail[idx+len(schemaKey):], " \t\r\n")
|
||||
if len(rest) == 0 || rest[0] != ':' {
|
||||
// a string value that happens to read "schema", not a key
|
||||
return "", "", false
|
||||
}
|
||||
rest = rest[1:]
|
||||
|
||||
var schema model.Schema
|
||||
dec := json.NewDecoder(bytes.NewReader(rest))
|
||||
if err := dec.Decode(&schema); err != nil {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
// the schema block only identifies the document when it belongs to the top-level object
|
||||
if !closesTopLevelObject(rest[dec.InputOffset():]) {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
if schema.Version == "" || !strings.Contains(schema.URL, "anchore/syft") {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
return ID, schema.Version, true
|
||||
}
|
||||
|
||||
// closesTopLevelObject reports whether the given bytes, which follow a value somewhere within a JSON document, run
|
||||
// to the end of that document closing exactly one enclosing object (and nothing else) at the very end. When true,
|
||||
// the value they follow sits directly within the top-level object. Braces inside strings are ignored.
|
||||
func closesTopLevelObject(b []byte) bool {
|
||||
var (
|
||||
depth int
|
||||
inString bool
|
||||
escaped bool
|
||||
closed bool
|
||||
)
|
||||
for _, c := range b {
|
||||
if closed {
|
||||
// only whitespace may follow the top-level object
|
||||
if c == ' ' || c == '\t' || c == '\r' || c == '\n' {
|
||||
continue
|
||||
}
|
||||
return false
|
||||
}
|
||||
if inString {
|
||||
switch {
|
||||
case escaped:
|
||||
escaped = false
|
||||
case c == '\\':
|
||||
escaped = true
|
||||
case c == '"':
|
||||
inString = false
|
||||
}
|
||||
continue
|
||||
}
|
||||
switch c {
|
||||
case '"':
|
||||
inString = true
|
||||
case '{', '[':
|
||||
depth++
|
||||
case '}':
|
||||
depth--
|
||||
if depth == -1 {
|
||||
closed = true
|
||||
}
|
||||
case ']':
|
||||
depth--
|
||||
if depth == -1 {
|
||||
// the enclosing container is an array, not the top-level object
|
||||
return false
|
||||
}
|
||||
}
|
||||
}
|
||||
return closed
|
||||
}
|
||||
|
||||
func checkSupportedSchema(documentVersion string, parserVersion string) error {
|
||||
documentV, err := semver.NewVersion(documentVersion)
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,340 @@
|
||||
package syftjson
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/anchore/syft/internal"
|
||||
"github.com/anchore/syft/syft/format/internal/stream"
|
||||
"github.com/anchore/syft/syft/format/internal/testutil"
|
||||
)
|
||||
|
||||
const (
|
||||
testIdentifySchemaURL = "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-16.0.0.json"
|
||||
testIdentifySchemaBlock = `"schema":{"version":"16.0.0","url":"` + testIdentifySchemaURL + `"}`
|
||||
)
|
||||
|
||||
func Test_identifyFromTailBytes(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
doc string
|
||||
wantVersion string
|
||||
wantOK bool
|
||||
}{
|
||||
{
|
||||
name: "syft key order (schema is the last key)",
|
||||
doc: `{"artifacts":[],"descriptor":{"name":"syft"},` + testIdentifySchemaBlock + `}`,
|
||||
wantVersion: "16.0.0",
|
||||
wantOK: true,
|
||||
},
|
||||
{
|
||||
name: "alphabetical key order (schema is followed by source)",
|
||||
doc: `{"artifacts":[],` + testIdentifySchemaBlock + `,"source":{"id":"x","name":"img","type":"image","metadata":{"tags":["a","b"],"layers":[{"digest":"sha256:1"}]}}}`,
|
||||
wantVersion: "16.0.0",
|
||||
wantOK: true,
|
||||
},
|
||||
{
|
||||
name: "pretty printed with a trailing newline",
|
||||
doc: "{\n \"artifacts\": [],\n \"schema\": {\n \"version\": \"16.0.0\",\n \"url\": \"" + testIdentifySchemaURL + "\"\n }\n}\n",
|
||||
wantVersion: "16.0.0",
|
||||
wantOK: true,
|
||||
},
|
||||
{
|
||||
name: "nested schema key is not mistaken for the document schema",
|
||||
doc: `{"artifacts":[{"metadata":{` + testIdentifySchemaBlock + `}}],"descriptor":{}}`,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "top-level schema wins over an earlier nested one",
|
||||
doc: `{"artifacts":[{"metadata":{"schema":{"version":"1","url":"anchore/syft"}}}],` + testIdentifySchemaBlock + `}`,
|
||||
wantVersion: "16.0.0",
|
||||
wantOK: true,
|
||||
},
|
||||
{
|
||||
name: "escaped schema text (with braces) inside a trailing string is ignored",
|
||||
doc: `{"artifacts":[],` + testIdentifySchemaBlock + `,"source":{"name":"contains \"schema\":{\"version\":\"9\"} text"}}`,
|
||||
wantVersion: "16.0.0",
|
||||
wantOK: true,
|
||||
},
|
||||
{
|
||||
name: "array wrapped document",
|
||||
doc: `[{"artifacts":[],` + testIdentifySchemaBlock + `}]`,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "truncated document",
|
||||
doc: `{"artifacts":[],` + testIdentifySchemaBlock,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "trailing content after the document",
|
||||
doc: `{"artifacts":[],` + testIdentifySchemaBlock + `}{"another":"document"}`,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "schema value is not an object",
|
||||
doc: `{"artifacts":[],"schema":"16.0.0"}`,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "schema is a string value rather than a key",
|
||||
doc: `{"artifacts":[],"kind":"schema"}`,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "not a syft schema url",
|
||||
doc: `{"artifacts":[],"schema":{"version":"1.0","url":"https://example.com/schema.json"}}`,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "no schema key at all",
|
||||
doc: `{"bomFormat":"CycloneDX","specVersion":"1.6","components":[]}`,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "schema key at the very start of the tail cannot be vouched for",
|
||||
doc: testIdentifySchemaBlock + `}`,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "empty",
|
||||
doc: "",
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
// the key's text is: he said "schema
|
||||
name: "an escaped quote does not make a string into the schema key",
|
||||
doc: `{"he said \"schema":{"version":"1.0","url":"https://example.com/anchore/syft"}}`,
|
||||
wantOK: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
id, version, ok := identifyFromTailBytes([]byte(tt.doc))
|
||||
assert.Equal(t, tt.wantOK, ok)
|
||||
assert.Equal(t, tt.wantVersion, version)
|
||||
if tt.wantOK {
|
||||
assert.Equal(t, ID, id)
|
||||
} else {
|
||||
assert.Empty(t, id)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func Test_Identify_largeDocument(t *testing.T) {
|
||||
// build a document comfortably larger than the tail window so only the tail path could avoid a full parse
|
||||
unit := `{"name":"pkg","version":"1.0"},`
|
||||
padding := strings.Repeat(unit, (identifyTailSize/len(unit))+1024)
|
||||
doc := `{"artifacts":[` + padding + `{"name":"last"}],` + testIdentifySchemaBlock + `}`
|
||||
require.Greater(t, len(doc), identifyTailSize)
|
||||
|
||||
t.Run("seekable reader", func(t *testing.T) {
|
||||
r := strings.NewReader(doc)
|
||||
|
||||
id, version := NewFormatDecoder().Identify(r)
|
||||
assert.Equal(t, ID, id)
|
||||
assert.Equal(t, "16.0.0", version)
|
||||
|
||||
// the reader is left where it started so callers can go on to read the document
|
||||
pos, err := r.Seek(0, io.SeekCurrent)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int64(0), pos)
|
||||
})
|
||||
|
||||
t.Run("non-seekable reader falls back to a full parse", func(t *testing.T) {
|
||||
r := struct{ io.Reader }{strings.NewReader(doc)}
|
||||
|
||||
id, version := NewFormatDecoder().Identify(r)
|
||||
assert.Equal(t, ID, id)
|
||||
assert.Equal(t, "16.0.0", version)
|
||||
})
|
||||
|
||||
t.Run("reader positioned at a later document", func(t *testing.T) {
|
||||
first := `{"artifacts":[],"schema":{"version":"1.0.0","url":"` + testIdentifySchemaURL + `"}}`
|
||||
r := strings.NewReader(first + "\n" + doc)
|
||||
_, err := r.Seek(int64(len(first)+1), io.SeekStart)
|
||||
require.NoError(t, err)
|
||||
|
||||
// mirror what the decoder collection does: seeks become relative to the current position
|
||||
sr, err := stream.SeekableReader(r)
|
||||
require.NoError(t, err)
|
||||
|
||||
id, version := NewFormatDecoder().Identify(sr)
|
||||
assert.Equal(t, ID, id)
|
||||
assert.Equal(t, "16.0.0", version)
|
||||
|
||||
pos, err := sr.Seek(0, io.SeekCurrent)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int64(0), pos)
|
||||
})
|
||||
}
|
||||
|
||||
func Test_Identify_agreesWithFullParse(t *testing.T) {
|
||||
// the tail is only a shortcut: whatever it decides must match what parsing the whole document says, including
|
||||
// for documents it has to decline.
|
||||
//
|
||||
// note: a stream holding several concatenated documents is deliberately not covered. Identifying the document
|
||||
// at the reader's position would mean proving no other document ends before the stream does, which means
|
||||
// reading all of it, which is the cost the tail path exists to avoid. Decode still reports the document it
|
||||
// actually parsed (see Test_Decode_rejectsADocumentTheTailDidNotDescribe).
|
||||
syftDoc := `{"artifacts":[],` + testIdentifySchemaBlock + `}`
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
doc string
|
||||
wantVersion string
|
||||
}{
|
||||
{name: "plain document", doc: syftDoc, wantVersion: "16.0.0"},
|
||||
{name: "byte order mark before the document", doc: "\xef\xbb\xbf" + syftDoc},
|
||||
{name: "content before the document", doc: "WARN something happened\n" + syftDoc},
|
||||
{name: "not a syft document", doc: `{"bomFormat":"CycloneDX","specVersion":"1.6","components":[]}`},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
d := NewFormatDecoder()
|
||||
|
||||
// a bytes.Reader is seekable, so this takes the tail path
|
||||
fastID, fastVersion := d.Identify(bytes.NewReader([]byte(tt.doc)))
|
||||
|
||||
// readerOnly hides Seek, so this takes the full parse
|
||||
slowID, slowVersion := d.Identify(readerOnly{strings.NewReader(tt.doc)})
|
||||
|
||||
assert.Equal(t, slowID, fastID, "fast and slow paths disagree on the format")
|
||||
assert.Equal(t, slowVersion, fastVersion, "fast and slow paths disagree on the version")
|
||||
assert.Equal(t, tt.wantVersion, fastVersion)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// readerOnly hides any Seek method the underlying reader has, forcing the full parse.
|
||||
type readerOnly struct {
|
||||
io.Reader
|
||||
}
|
||||
|
||||
func Test_Decode_rejectsADocumentTheTailDidNotDescribe(t *testing.T) {
|
||||
// the tail can find a schema block belonging to a later document in the stream; whatever it decides, Decode
|
||||
// must not hand back the empty SBOM that parsing the first document into model.Document would produce
|
||||
doc := `{"bomFormat":"CycloneDX","specVersion":"1.6","components":[{"type":"library","name":"pkg-a"}]}` +
|
||||
"\n" + `{"artifacts":[],` + testIdentifySchemaBlock + `}`
|
||||
|
||||
s, _, _, err := NewFormatDecoder().Decode(bytes.NewReader([]byte(doc)))
|
||||
require.Error(t, err)
|
||||
assert.Nil(t, s)
|
||||
}
|
||||
|
||||
func Test_encoderWritesSchemaLast(t *testing.T) {
|
||||
// identification reads the schema block from the tail of the document (see model.Document), which depends on
|
||||
// the encoder always writing it as the final key
|
||||
s := testutil.DirectoryInput(t, t.TempDir())
|
||||
|
||||
for _, pretty := range []bool{false, true} {
|
||||
cfg := DefaultEncoderConfig()
|
||||
cfg.Pretty = pretty
|
||||
|
||||
enc, err := NewFormatEncoderWithConfig(cfg)
|
||||
require.NoError(t, err)
|
||||
|
||||
var buf bytes.Buffer
|
||||
require.NoError(t, enc.Encode(&buf, s))
|
||||
|
||||
keys := topLevelKeys(t, buf.Bytes())
|
||||
require.NotEmpty(t, keys)
|
||||
assert.Equal(t, "schema", keys[len(keys)-1], "pretty=%v", pretty)
|
||||
|
||||
id, version, ok := identifyFromTailBytes(buf.Bytes())
|
||||
assert.True(t, ok, "pretty=%v", pretty)
|
||||
assert.Equal(t, ID, id)
|
||||
assert.Equal(t, internal.JSONSchemaVersion, version)
|
||||
}
|
||||
}
|
||||
|
||||
// noEndSeeker behaves like a reader that buffers a stream as it goes: it can seek within what it has, but cannot
|
||||
// seek relative to the end.
|
||||
type noEndSeeker struct {
|
||||
io.ReadSeeker
|
||||
}
|
||||
|
||||
func (s noEndSeeker) Seek(offset int64, whence int) (int64, error) {
|
||||
if whence == io.SeekEnd {
|
||||
return 0, errors.New("SeekEnd is not supported")
|
||||
}
|
||||
return s.ReadSeeker.Seek(offset, whence)
|
||||
}
|
||||
|
||||
func Test_Identify_fallsBackWhenTailIsUnavailable(t *testing.T) {
|
||||
doc := `{"artifacts":[],` + testIdentifySchemaBlock + `}`
|
||||
r := noEndSeeker{strings.NewReader(doc)}
|
||||
|
||||
id, version := NewFormatDecoder().Identify(r)
|
||||
assert.Equal(t, ID, id)
|
||||
assert.Equal(t, "16.0.0", version)
|
||||
}
|
||||
|
||||
func Test_identifyFromTail_readerEdgeCases(t *testing.T) {
|
||||
doc := `{"artifacts":[],` + testIdentifySchemaBlock + `}`
|
||||
|
||||
t.Run("empty reader", func(t *testing.T) {
|
||||
r := strings.NewReader("")
|
||||
|
||||
_, _, ok := identifyFromTail(r)
|
||||
assert.False(t, ok)
|
||||
})
|
||||
|
||||
t.Run("reader already at the end has nothing to inspect and keeps its position", func(t *testing.T) {
|
||||
r := strings.NewReader(doc)
|
||||
_, err := r.Seek(0, io.SeekEnd)
|
||||
require.NoError(t, err)
|
||||
|
||||
_, _, ok := identifyFromTail(r)
|
||||
assert.False(t, ok)
|
||||
|
||||
pos, err := r.Seek(0, io.SeekCurrent)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int64(len(doc)), pos)
|
||||
})
|
||||
|
||||
t.Run("whitespace around the colon and CRLF line endings", func(t *testing.T) {
|
||||
crlf := "{\r\n \"artifacts\": [],\r\n \"schema\" : {\r\n \"version\": \"16.0.0\",\r\n \"url\": \"" + testIdentifySchemaURL + "\"\r\n }\r\n}\r\n"
|
||||
|
||||
id, version, ok := identifyFromTailBytes([]byte(crlf))
|
||||
require.True(t, ok)
|
||||
assert.Equal(t, ID, id)
|
||||
assert.Equal(t, "16.0.0", version)
|
||||
})
|
||||
|
||||
t.Run("document ending inside a string is not trusted", func(t *testing.T) {
|
||||
truncated := `{"artifacts":[],` + testIdentifySchemaBlock + `,"source":{"name":"unterminated`
|
||||
|
||||
_, _, ok := identifyFromTailBytes([]byte(truncated))
|
||||
assert.False(t, ok)
|
||||
})
|
||||
}
|
||||
|
||||
func topLevelKeys(t *testing.T, doc []byte) []string {
|
||||
dec := json.NewDecoder(bytes.NewReader(doc))
|
||||
|
||||
tok, err := dec.Token()
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, json.Delim('{'), tok)
|
||||
|
||||
var keys []string
|
||||
for dec.More() {
|
||||
tok, err := dec.Token()
|
||||
require.NoError(t, err)
|
||||
keys = append(keys, tok.(string))
|
||||
|
||||
var value json.RawMessage
|
||||
require.NoError(t, dec.Decode(&value))
|
||||
}
|
||||
return keys
|
||||
}
|
||||
@@ -5,6 +5,14 @@ import (
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// Field order matters: encoding/json writes struct fields in declaration order, and Schema must remain the last
|
||||
// field so that the schema block is always the final key of the encoded document. Format identification relies on
|
||||
// finding the schema block by reading only the tail of a (potentially very large) document; moving Schema earlier
|
||||
// would not break decoding but would silently force a full parse of the whole document to identify it.
|
||||
//
|
||||
// (This block is deliberately separated from the doc comment below: the JSON schema generator lifts doc comments
|
||||
// into schema descriptions, and changing the description would alter the published schema.)
|
||||
|
||||
// Document represents the syft cataloging findings as a JSON document
|
||||
type Document struct {
|
||||
Artifacts []Package `json:"artifacts"` // Artifacts is the list of packages discovered and placed into the catalog
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"github.com/anchore/syft/syft/format/cyclonedxxml"
|
||||
"github.com/anchore/syft/syft/format/spdxjson"
|
||||
"github.com/anchore/syft/syft/format/spdxtagvalue"
|
||||
"github.com/anchore/syft/syft/format/syftjson"
|
||||
"github.com/anchore/syft/syft/sbom"
|
||||
)
|
||||
|
||||
@@ -64,6 +65,93 @@ func TestConvertCmd(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestConvertCmd_PassthroughExactFormat(t *testing.T) {
|
||||
// pretty-printed input makes a verbatim copy distinguishable from a re-encode (which is compact)
|
||||
sbomArgs := []string{"dir:./testdata/image-pkg-coverage", "-o", "syft-json"}
|
||||
cmd, input, stderr := runSyft(t, map[string]string{"SYFT_FORMAT_PRETTY": "true"}, sbomArgs...)
|
||||
if cmd.ProcessState.ExitCode() != 0 {
|
||||
t.Log("STDOUT:\n", input)
|
||||
t.Log("STDERR:\n", stderr)
|
||||
t.Log("COMMAND:", strings.Join(cmd.Args, " "))
|
||||
t.Fatalf("failure executing syft creating an sbom")
|
||||
return
|
||||
}
|
||||
|
||||
inputID, inputVersion := format.Identify(strings.NewReader(input))
|
||||
require.Equal(t, syftjson.ID, inputID)
|
||||
require.NotEmpty(t, inputVersion)
|
||||
|
||||
const (
|
||||
enabledWarning = "convert.passthrough-exact-format is enabled"
|
||||
copiedWarning = "copying it unchanged instead of converting it"
|
||||
)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
env map[string]string
|
||||
to string
|
||||
wantUnchanged bool
|
||||
wantID sbom.FormatID
|
||||
wantVersion string
|
||||
wantInStderr []string
|
||||
wantNotStderr []string
|
||||
}{
|
||||
{
|
||||
name: "matching syft-json is passed through unchanged",
|
||||
env: map[string]string{"SYFT_CONVERT_PASSTHROUGH_EXACT_FORMAT": "true"},
|
||||
to: "syft-json",
|
||||
wantUnchanged: true,
|
||||
wantID: syftjson.ID,
|
||||
wantVersion: inputVersion,
|
||||
wantInStderr: []string{enabledWarning, copiedWarning},
|
||||
},
|
||||
{
|
||||
name: "different format is converted",
|
||||
env: map[string]string{"SYFT_CONVERT_PASSTHROUGH_EXACT_FORMAT": "true"},
|
||||
to: "cyclonedx-json",
|
||||
wantID: cyclonedxjson.ID,
|
||||
wantVersion: mustEncoder(cyclonedxjson.NewFormatEncoderWithConfig(cyclonedxjson.DefaultEncoderConfig())).Version(),
|
||||
wantInStderr: []string{enabledWarning},
|
||||
wantNotStderr: []string{copiedWarning},
|
||||
},
|
||||
{
|
||||
name: "option off by default, matching syft-json is re-encoded",
|
||||
to: "syft-json",
|
||||
wantID: syftjson.ID,
|
||||
wantVersion: inputVersion,
|
||||
wantNotStderr: []string{enabledWarning, copiedWarning},
|
||||
},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
cmd := getSyftCommand(t, "convert", "-", "-o", test.to)
|
||||
cmd.Stdin = strings.NewReader(input)
|
||||
stdout, stderr := runCommandObj(t, cmd, test.env, false)
|
||||
|
||||
assertSuccessfulReturnCode(t, stdout, stderr, cmd.ProcessState.ExitCode())
|
||||
logOutputOnFailure(t, cmd, stdout, stderr)
|
||||
|
||||
foundID, foundVersion := format.Identify(strings.NewReader(stdout))
|
||||
require.Equal(t, test.wantID, foundID)
|
||||
require.Equal(t, test.wantVersion, foundVersion)
|
||||
|
||||
if test.wantUnchanged {
|
||||
require.Equal(t, strings.TrimSpace(input), strings.TrimSpace(stdout))
|
||||
} else {
|
||||
require.NotEqual(t, strings.TrimSpace(input), strings.TrimSpace(stdout))
|
||||
}
|
||||
|
||||
for _, want := range test.wantInStderr {
|
||||
require.Contains(t, stderr, want)
|
||||
}
|
||||
for _, notWant := range test.wantNotStderr {
|
||||
require.NotContains(t, stderr, notWant)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func mustEncoder(enc sbom.FormatEncoder, err error) sbom.FormatEncoder {
|
||||
if err != nil {
|
||||
panic(err)
|
||||
|
||||
Reference in New Issue
Block a user