mirror of
https://github.com/anchore/syft.git
synced 2026-11-01 23:16:30 +01:00
feat(convert): passthrough-exact-format copies matching syft-json input unchanged (#5279)
* feat(convert): passthrough-exact-format copies matching syft-json input unchanged When a syft-json input SBOM already exactly matches the requested output format and schema version, `syft convert` can write the input bytes out verbatim instead of decoding and re-encoding them. This avoids the lossy round trip (and the associated cost) for callers that normalise SBOMs into a target format without knowing up front whether the source already matches. The behaviour is opt-in and configuration-only: it is enabled with convert.passthrough-exact-format in the config file or the SYFT_CONVERT_PASSTHROUGH_EXACT_FORMAT environment variable, and there is deliberately no command-line flag, since it changes how every invocation in a pipeline behaves rather than varying call by call. It defaults to off, so existing behaviour is unchanged. Pass-through applies only to syft-json input, and only when the document's schema version is identical to the one this build of syft produces; any other input is always converted. Outputs that do not match are still converted, so a mixed set of `-o` values behaves as expected, and every output is resolved and validated before anything is written. Pass-through output is a byte-for-byte copy, so format options such as `format.pretty` and `format.json.legacy` are not applied to it. When the option is enabled a warning is logged stating so and noting the syft-json-only scope, and a second warning is logged whenever an input is found to match and the shortcut is actually taken. The option lives under a `convert:` configuration section (options.Convert), mirroring how attest scopes its own options. The description documents the caveats above and notes that file-path input written with `-o <format>=<path>` keeps memory minimal, whereas piped input or STDOUT output must hold the whole document in memory. Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * perf(convert): stream pass-through output instead of buffering the input Identify, copy, and decode the input through a seekable reader rather than loading the whole document into memory first. File inputs are read in place and pass-through output is an io.Copy from the file. STDIN is used in place when it is a redirected regular file. A pipe or terminal cannot seek, so its contents are drained into an internal/spillbuf buffer: the first 1MB stays in memory and the rest goes to a temp file under the user's TMPDIR (created lazily, only when something actually spills), giving a seekable view without ever holding the document in memory. The temp dir is rooted per invocation via internal/tmpdir, honouring one already attached to the context, and is removed when the command returns. Writing pass-through output over the input file itself is now a no-op rather than truncating the file before it is copied. Measured on a 1GB syft-json document (Apple M1 Max), file in, file out: origin/main full re-encode 86.3s wall 99s CPU 12.0GB max RSS in-memory pass-through 7.4s wall 6.7s CPU 4.2GB max RSS streaming pass-through 7.0s wall 6.9s CPU 3.1GB max RSS and the same 1GB document piped over STDIN with pass-through enabled: io.ReadAll into memory 1.3s wall 0.8s CPU 2.7GB max RSS spill buffer 1.8s wall 1.0s CPU 66MB max RSS The remaining cost of the streaming pass-through is entirely format.Identify, which for syft-json decodes the whole document through a json.Decoder that buffers the full value: measured alone it takes 7.1s and 3.1GB on the same input. Reducing that further belongs to the format package. Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * perf(syftjson): identify documents from their tail instead of a full parse syft always writes the schema block as the last key of a syft-json document, so the format and version can be read from the final bytes of the file. The syft-json decoder's Identify now reads at most 2MB from the end of a seekable input, finds the last "schema" key there, and verifies it sits directly within the top-level object by scanning the remaining bytes for the single closing brace of the document. Anything the tail cannot vouch for (non-seekable input, a reordered document, a nested key) falls back to the existing full parse, so behaviour is unchanged for documents the fast path declines. The verification is position-independent: it also accepts documents whose keys have been reordered so that other small keys follow schema. It rejects nested schema objects, array-wrapped documents, truncated input, trailing content, and schema text inside strings. Supporting changes: - the offset-aware reader the decoder collection hands to decoders now supports io.SeekEnd, which the tail read requires - model.Document documents that Schema must remain its last field, and a test guards that the encoder writes it last in both compact and pretty output - the decoder list documents that syft-json must stay first Since Decode calls Identify before parsing, this also removes one of the two full parses every syft-json conversion used to perform. Measured on a 1GB syft-json document (Apple M1 Max), file in, file out: pass-through, previous commit 7.0s wall 6.9s CPU 3.1GB max RSS pass-through, this commit 0.85s wall 0.05s CPU 66MB max RSS full re-encode, previous commit 85.3s wall 99s CPU 11.5GB max RSS full re-encode, this commit 75.2s wall 91s CPU 9.9GB max RSS Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix(syftjson): only identify from the tail when the document starts where the reader does The tail scan vouched for the last 2MB, but nothing checked those bytes belonged to the document the reader was pointed at. So `Identify` started claiming syft-json for files it cannot decode: a document with a byte order mark or a stray log line in front of it, or a syft document sitting after something else in the same stream, where `format.Decode` went on to return an empty SBOM with a nil error. Two O(1) checks: - the first non-space byte at the reader's position has to be `{` - `Decode` errors when the parsed document has no `schema.version`, instead of handing back the empty document it decoded into Streams holding several concatenated documents are deliberately not detected. Proving there is exactly one top-level value means reading all of it, which is the cost the tail path exists to avoid, so `Identify` can still name a later document's version on such a file. `Decode` reports what it actually parsed. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix(convert): recognize the input file by identity rather than by path `convert - -o syft-json=sbom.json < sbom.json` truncated the file and exited 0. The passthrough guard compared paths, and a shell redirect does not have one, so it was blind to the one input mode where the truncation was silent. `openConvertInput` now returns the input's `os.FileInfo`, which the STDIN branch already had from its `Stat`, and `isSameFile` compares that instead. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> --------- Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
This commit is contained in:
Juan
Alex Goodman
@@ -21,6 +21,9 @@ func init() {
|
||||
|
||||
func Decoders() []sbom.FormatDecoder {
|
||||
return []sbom.FormatDecoder{
|
||||
// syft-json must remain the first decoder tried: it is by far the most common input, and its decoder can
|
||||
// identify a syft document from the tail of the file alone. Trying anything else first would pay for a full
|
||||
// parse of a large document by another decoder before reaching the common path.
|
||||
syftjson.NewFormatDecoder(),
|
||||
cyclonedxxml.NewFormatDecoder(),
|
||||
cyclonedxjson.NewFormatDecoder(),
|
||||
|
||||
@@ -72,8 +72,21 @@ func (o *offsetReadSeeker) Seek(offset int64, whence int) (int64, error) {
|
||||
}
|
||||
newOffset, err := o.rdr.Seek(offset, io.SeekCurrent)
|
||||
return newOffset - o.offset, err
|
||||
case io.SeekEnd:
|
||||
newOffset, err := o.rdr.Seek(offset, io.SeekEnd)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("cannot seek end: %w", err)
|
||||
}
|
||||
if newOffset < o.offset {
|
||||
// landed before the start of this view; move back to the start so the position stays valid
|
||||
if _, err := o.rdr.Seek(o.offset, io.SeekStart); err != nil {
|
||||
return 0, fmt.Errorf("cannot seek end: %w", err)
|
||||
}
|
||||
return 0, fmt.Errorf("cannot seek < 0")
|
||||
}
|
||||
return newOffset - o.offset, nil
|
||||
}
|
||||
return 0, fmt.Errorf("only SeekStart and SeekCurrent supported")
|
||||
return 0, fmt.Errorf("only SeekStart, SeekCurrent, and SeekEnd supported")
|
||||
}
|
||||
|
||||
var _ io.ReadSeeker = (*offsetReadSeeker)(nil)
|
||||
|
||||
@@ -179,11 +179,25 @@ func Test_offsetReadSeeker(t *testing.T) {
|
||||
whence: io.SeekCurrent,
|
||||
expected: "4",
|
||||
},
|
||||
{
|
||||
name: "seek end",
|
||||
input: abcd1234offset(2),
|
||||
seek: -4,
|
||||
whence: io.SeekEnd,
|
||||
expected: "1234",
|
||||
},
|
||||
{
|
||||
name: "seek end before the start of the view",
|
||||
input: abcd1234offset(2),
|
||||
seek: -7,
|
||||
whence: io.SeekEnd,
|
||||
wantErr: require.Error, // would be < offset, which is an error
|
||||
},
|
||||
{
|
||||
name: "bad whence",
|
||||
input: abcd1234,
|
||||
seek: 1,
|
||||
whence: io.SeekEnd,
|
||||
whence: 99,
|
||||
wantErr: require.Error,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
package syftjson
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
@@ -15,6 +17,12 @@ import (
|
||||
"github.com/anchore/syft/syft/sbom"
|
||||
)
|
||||
|
||||
// identifyTailSize is how much of the end of a document is inspected when identifying it from its trailing schema
|
||||
// block, before falling back to parsing the whole document.
|
||||
const identifyTailSize = 2 * 1024 * 1024
|
||||
|
||||
var schemaKey = []byte(`"schema"`)
|
||||
|
||||
var _ sbom.FormatDecoder = (*decoder)(nil)
|
||||
|
||||
type decoder struct{}
|
||||
@@ -45,6 +53,12 @@ func (d decoder) Decode(r io.Reader) (*sbom.SBOM, sbom.FormatID, string, error)
|
||||
return nil, "", "", fmt.Errorf("unable to decode syft-json document: %w", err)
|
||||
}
|
||||
|
||||
if doc.Schema.Version == "" {
|
||||
// Identify found a schema block that the parsed document does not have, so it was describing something
|
||||
// else in the stream; returning the empty document it decoded into would be a silently wrong answer
|
||||
return nil, "", "", fmt.Errorf("not a syft-json document")
|
||||
}
|
||||
|
||||
if err := checkSupportedSchema(doc.Schema.Version, internal.JSONSchemaVersion); err != nil {
|
||||
log.Warn(err)
|
||||
}
|
||||
@@ -57,6 +71,15 @@ func (d decoder) Identify(r io.Reader) (sbom.FormatID, string) {
|
||||
return "", ""
|
||||
}
|
||||
|
||||
// fast path: syft writes the schema block as the last key of the document (see model.Document), so for seekable
|
||||
// input the format and version can be read from the tail without parsing a potentially very large document.
|
||||
// Anything the tail cannot vouch for falls through to the full parse below.
|
||||
if rs, ok := r.(io.ReadSeeker); ok {
|
||||
if id, version, ok := identifyFromTail(rs); ok {
|
||||
return id, version
|
||||
}
|
||||
}
|
||||
|
||||
type Document struct {
|
||||
Schema model.Schema `json:"schema"`
|
||||
}
|
||||
@@ -78,6 +101,160 @@ func (d decoder) Identify(r io.Reader) (sbom.FormatID, string) {
|
||||
return ID, doc.Schema.Version
|
||||
}
|
||||
|
||||
// identifyFromTail reads at most identifyTailSize bytes from the end of the reader and attempts to identify the
|
||||
// document from a top-level schema block found there. The reader is returned to its original position afterwards.
|
||||
func identifyFromTail(rs io.ReadSeeker) (sbom.FormatID, string, bool) {
|
||||
start, err := rs.Seek(0, io.SeekCurrent)
|
||||
if err != nil {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
id, version, ok := readTail(rs, start)
|
||||
|
||||
if _, err := rs.Seek(start, io.SeekStart); err != nil {
|
||||
// without the original position the caller cannot safely parse the document afterwards
|
||||
log.WithFields("error", err).Debug("unable to restore reader position after inspecting document tail")
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
return id, version, ok
|
||||
}
|
||||
|
||||
// readTail does the seeking and reading for identifyFromTail, leaving the reader wherever it ends up.
|
||||
func readTail(rs io.ReadSeeker, start int64) (sbom.FormatID, string, bool) {
|
||||
end, err := rs.Seek(0, io.SeekEnd)
|
||||
if err != nil || end <= start {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
// the tail only describes the document the reader is pointed at if that document starts where the reader is,
|
||||
// so check the first byte: bytes ahead of it (a byte order mark, a log line prepended by a wrapper) would
|
||||
// otherwise be accepted silently, leaving Identify claiming a document that cannot be decoded
|
||||
if !opensObject(rs, start) {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
tailLen := min(end-start, int64(identifyTailSize))
|
||||
tail := make([]byte, tailLen)
|
||||
if _, err := rs.Seek(end-tailLen, io.SeekStart); err != nil {
|
||||
return "", "", false
|
||||
}
|
||||
if _, err := io.ReadFull(rs, tail); err != nil {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
return identifyFromTailBytes(tail)
|
||||
}
|
||||
|
||||
// opensObject reports whether the document at the given offset begins with a JSON object.
|
||||
func opensObject(rs io.ReadSeeker, start int64) bool {
|
||||
if _, err := rs.Seek(start, io.SeekStart); err != nil {
|
||||
return false
|
||||
}
|
||||
|
||||
head := make([]byte, 64)
|
||||
n, err := io.ReadFull(rs, head)
|
||||
if err != nil && !errors.Is(err, io.ErrUnexpectedEOF) {
|
||||
return false
|
||||
}
|
||||
|
||||
head = bytes.TrimLeft(head[:n], " \t\r\n")
|
||||
return len(head) > 0 && head[0] == '{'
|
||||
}
|
||||
|
||||
// identifyFromTailBytes looks for a syft schema block among the trailing bytes of a JSON document and, when one is
|
||||
// found directly within the top-level object, returns the format and schema version it declares. The bytes may
|
||||
// begin anywhere within the document, but must run to its very end.
|
||||
func identifyFromTailBytes(tail []byte) (sbom.FormatID, string, bool) {
|
||||
// find the last "schema" key. An unescaped quote cannot occur inside a JSON string, so a match whose opening
|
||||
// quote is not preceded by a backslash is a genuine string token (and a key if a colon follows it).
|
||||
idx := len(tail)
|
||||
for {
|
||||
idx = bytes.LastIndex(tail[:idx], schemaKey)
|
||||
if idx <= 0 {
|
||||
// not found, or found at the very start of the tail where we cannot tell whether it is escaped
|
||||
return "", "", false
|
||||
}
|
||||
if tail[idx-1] != '\\' {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
rest := bytes.TrimLeft(tail[idx+len(schemaKey):], " \t\r\n")
|
||||
if len(rest) == 0 || rest[0] != ':' {
|
||||
// a string value that happens to read "schema", not a key
|
||||
return "", "", false
|
||||
}
|
||||
rest = rest[1:]
|
||||
|
||||
var schema model.Schema
|
||||
dec := json.NewDecoder(bytes.NewReader(rest))
|
||||
if err := dec.Decode(&schema); err != nil {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
// the schema block only identifies the document when it belongs to the top-level object
|
||||
if !closesTopLevelObject(rest[dec.InputOffset():]) {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
if schema.Version == "" || !strings.Contains(schema.URL, "anchore/syft") {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
return ID, schema.Version, true
|
||||
}
|
||||
|
||||
// closesTopLevelObject reports whether the given bytes, which follow a value somewhere within a JSON document, run
|
||||
// to the end of that document closing exactly one enclosing object (and nothing else) at the very end. When true,
|
||||
// the value they follow sits directly within the top-level object. Braces inside strings are ignored.
|
||||
func closesTopLevelObject(b []byte) bool {
|
||||
var (
|
||||
depth int
|
||||
inString bool
|
||||
escaped bool
|
||||
closed bool
|
||||
)
|
||||
for _, c := range b {
|
||||
if closed {
|
||||
// only whitespace may follow the top-level object
|
||||
if c == ' ' || c == '\t' || c == '\r' || c == '\n' {
|
||||
continue
|
||||
}
|
||||
return false
|
||||
}
|
||||
if inString {
|
||||
switch {
|
||||
case escaped:
|
||||
escaped = false
|
||||
case c == '\\':
|
||||
escaped = true
|
||||
case c == '"':
|
||||
inString = false
|
||||
}
|
||||
continue
|
||||
}
|
||||
switch c {
|
||||
case '"':
|
||||
inString = true
|
||||
case '{', '[':
|
||||
depth++
|
||||
case '}':
|
||||
depth--
|
||||
if depth == -1 {
|
||||
closed = true
|
||||
}
|
||||
case ']':
|
||||
depth--
|
||||
if depth == -1 {
|
||||
// the enclosing container is an array, not the top-level object
|
||||
return false
|
||||
}
|
||||
}
|
||||
}
|
||||
return closed
|
||||
}
|
||||
|
||||
func checkSupportedSchema(documentVersion string, parserVersion string) error {
|
||||
documentV, err := semver.NewVersion(documentVersion)
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,340 @@
|
||||
package syftjson
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/anchore/syft/internal"
|
||||
"github.com/anchore/syft/syft/format/internal/stream"
|
||||
"github.com/anchore/syft/syft/format/internal/testutil"
|
||||
)
|
||||
|
||||
const (
|
||||
testIdentifySchemaURL = "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-16.0.0.json"
|
||||
testIdentifySchemaBlock = `"schema":{"version":"16.0.0","url":"` + testIdentifySchemaURL + `"}`
|
||||
)
|
||||
|
||||
func Test_identifyFromTailBytes(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
doc string
|
||||
wantVersion string
|
||||
wantOK bool
|
||||
}{
|
||||
{
|
||||
name: "syft key order (schema is the last key)",
|
||||
doc: `{"artifacts":[],"descriptor":{"name":"syft"},` + testIdentifySchemaBlock + `}`,
|
||||
wantVersion: "16.0.0",
|
||||
wantOK: true,
|
||||
},
|
||||
{
|
||||
name: "alphabetical key order (schema is followed by source)",
|
||||
doc: `{"artifacts":[],` + testIdentifySchemaBlock + `,"source":{"id":"x","name":"img","type":"image","metadata":{"tags":["a","b"],"layers":[{"digest":"sha256:1"}]}}}`,
|
||||
wantVersion: "16.0.0",
|
||||
wantOK: true,
|
||||
},
|
||||
{
|
||||
name: "pretty printed with a trailing newline",
|
||||
doc: "{\n \"artifacts\": [],\n \"schema\": {\n \"version\": \"16.0.0\",\n \"url\": \"" + testIdentifySchemaURL + "\"\n }\n}\n",
|
||||
wantVersion: "16.0.0",
|
||||
wantOK: true,
|
||||
},
|
||||
{
|
||||
name: "nested schema key is not mistaken for the document schema",
|
||||
doc: `{"artifacts":[{"metadata":{` + testIdentifySchemaBlock + `}}],"descriptor":{}}`,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "top-level schema wins over an earlier nested one",
|
||||
doc: `{"artifacts":[{"metadata":{"schema":{"version":"1","url":"anchore/syft"}}}],` + testIdentifySchemaBlock + `}`,
|
||||
wantVersion: "16.0.0",
|
||||
wantOK: true,
|
||||
},
|
||||
{
|
||||
name: "escaped schema text (with braces) inside a trailing string is ignored",
|
||||
doc: `{"artifacts":[],` + testIdentifySchemaBlock + `,"source":{"name":"contains \"schema\":{\"version\":\"9\"} text"}}`,
|
||||
wantVersion: "16.0.0",
|
||||
wantOK: true,
|
||||
},
|
||||
{
|
||||
name: "array wrapped document",
|
||||
doc: `[{"artifacts":[],` + testIdentifySchemaBlock + `}]`,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "truncated document",
|
||||
doc: `{"artifacts":[],` + testIdentifySchemaBlock,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "trailing content after the document",
|
||||
doc: `{"artifacts":[],` + testIdentifySchemaBlock + `}{"another":"document"}`,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "schema value is not an object",
|
||||
doc: `{"artifacts":[],"schema":"16.0.0"}`,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "schema is a string value rather than a key",
|
||||
doc: `{"artifacts":[],"kind":"schema"}`,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "not a syft schema url",
|
||||
doc: `{"artifacts":[],"schema":{"version":"1.0","url":"https://example.com/schema.json"}}`,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "no schema key at all",
|
||||
doc: `{"bomFormat":"CycloneDX","specVersion":"1.6","components":[]}`,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "schema key at the very start of the tail cannot be vouched for",
|
||||
doc: testIdentifySchemaBlock + `}`,
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
name: "empty",
|
||||
doc: "",
|
||||
wantOK: false,
|
||||
},
|
||||
{
|
||||
// the key's text is: he said "schema
|
||||
name: "an escaped quote does not make a string into the schema key",
|
||||
doc: `{"he said \"schema":{"version":"1.0","url":"https://example.com/anchore/syft"}}`,
|
||||
wantOK: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
id, version, ok := identifyFromTailBytes([]byte(tt.doc))
|
||||
assert.Equal(t, tt.wantOK, ok)
|
||||
assert.Equal(t, tt.wantVersion, version)
|
||||
if tt.wantOK {
|
||||
assert.Equal(t, ID, id)
|
||||
} else {
|
||||
assert.Empty(t, id)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func Test_Identify_largeDocument(t *testing.T) {
|
||||
// build a document comfortably larger than the tail window so only the tail path could avoid a full parse
|
||||
unit := `{"name":"pkg","version":"1.0"},`
|
||||
padding := strings.Repeat(unit, (identifyTailSize/len(unit))+1024)
|
||||
doc := `{"artifacts":[` + padding + `{"name":"last"}],` + testIdentifySchemaBlock + `}`
|
||||
require.Greater(t, len(doc), identifyTailSize)
|
||||
|
||||
t.Run("seekable reader", func(t *testing.T) {
|
||||
r := strings.NewReader(doc)
|
||||
|
||||
id, version := NewFormatDecoder().Identify(r)
|
||||
assert.Equal(t, ID, id)
|
||||
assert.Equal(t, "16.0.0", version)
|
||||
|
||||
// the reader is left where it started so callers can go on to read the document
|
||||
pos, err := r.Seek(0, io.SeekCurrent)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int64(0), pos)
|
||||
})
|
||||
|
||||
t.Run("non-seekable reader falls back to a full parse", func(t *testing.T) {
|
||||
r := struct{ io.Reader }{strings.NewReader(doc)}
|
||||
|
||||
id, version := NewFormatDecoder().Identify(r)
|
||||
assert.Equal(t, ID, id)
|
||||
assert.Equal(t, "16.0.0", version)
|
||||
})
|
||||
|
||||
t.Run("reader positioned at a later document", func(t *testing.T) {
|
||||
first := `{"artifacts":[],"schema":{"version":"1.0.0","url":"` + testIdentifySchemaURL + `"}}`
|
||||
r := strings.NewReader(first + "\n" + doc)
|
||||
_, err := r.Seek(int64(len(first)+1), io.SeekStart)
|
||||
require.NoError(t, err)
|
||||
|
||||
// mirror what the decoder collection does: seeks become relative to the current position
|
||||
sr, err := stream.SeekableReader(r)
|
||||
require.NoError(t, err)
|
||||
|
||||
id, version := NewFormatDecoder().Identify(sr)
|
||||
assert.Equal(t, ID, id)
|
||||
assert.Equal(t, "16.0.0", version)
|
||||
|
||||
pos, err := sr.Seek(0, io.SeekCurrent)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int64(0), pos)
|
||||
})
|
||||
}
|
||||
|
||||
func Test_Identify_agreesWithFullParse(t *testing.T) {
|
||||
// the tail is only a shortcut: whatever it decides must match what parsing the whole document says, including
|
||||
// for documents it has to decline.
|
||||
//
|
||||
// note: a stream holding several concatenated documents is deliberately not covered. Identifying the document
|
||||
// at the reader's position would mean proving no other document ends before the stream does, which means
|
||||
// reading all of it, which is the cost the tail path exists to avoid. Decode still reports the document it
|
||||
// actually parsed (see Test_Decode_rejectsADocumentTheTailDidNotDescribe).
|
||||
syftDoc := `{"artifacts":[],` + testIdentifySchemaBlock + `}`
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
doc string
|
||||
wantVersion string
|
||||
}{
|
||||
{name: "plain document", doc: syftDoc, wantVersion: "16.0.0"},
|
||||
{name: "byte order mark before the document", doc: "\xef\xbb\xbf" + syftDoc},
|
||||
{name: "content before the document", doc: "WARN something happened\n" + syftDoc},
|
||||
{name: "not a syft document", doc: `{"bomFormat":"CycloneDX","specVersion":"1.6","components":[]}`},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
d := NewFormatDecoder()
|
||||
|
||||
// a bytes.Reader is seekable, so this takes the tail path
|
||||
fastID, fastVersion := d.Identify(bytes.NewReader([]byte(tt.doc)))
|
||||
|
||||
// readerOnly hides Seek, so this takes the full parse
|
||||
slowID, slowVersion := d.Identify(readerOnly{strings.NewReader(tt.doc)})
|
||||
|
||||
assert.Equal(t, slowID, fastID, "fast and slow paths disagree on the format")
|
||||
assert.Equal(t, slowVersion, fastVersion, "fast and slow paths disagree on the version")
|
||||
assert.Equal(t, tt.wantVersion, fastVersion)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// readerOnly hides any Seek method the underlying reader has, forcing the full parse.
|
||||
type readerOnly struct {
|
||||
io.Reader
|
||||
}
|
||||
|
||||
func Test_Decode_rejectsADocumentTheTailDidNotDescribe(t *testing.T) {
|
||||
// the tail can find a schema block belonging to a later document in the stream; whatever it decides, Decode
|
||||
// must not hand back the empty SBOM that parsing the first document into model.Document would produce
|
||||
doc := `{"bomFormat":"CycloneDX","specVersion":"1.6","components":[{"type":"library","name":"pkg-a"}]}` +
|
||||
"\n" + `{"artifacts":[],` + testIdentifySchemaBlock + `}`
|
||||
|
||||
s, _, _, err := NewFormatDecoder().Decode(bytes.NewReader([]byte(doc)))
|
||||
require.Error(t, err)
|
||||
assert.Nil(t, s)
|
||||
}
|
||||
|
||||
func Test_encoderWritesSchemaLast(t *testing.T) {
|
||||
// identification reads the schema block from the tail of the document (see model.Document), which depends on
|
||||
// the encoder always writing it as the final key
|
||||
s := testutil.DirectoryInput(t, t.TempDir())
|
||||
|
||||
for _, pretty := range []bool{false, true} {
|
||||
cfg := DefaultEncoderConfig()
|
||||
cfg.Pretty = pretty
|
||||
|
||||
enc, err := NewFormatEncoderWithConfig(cfg)
|
||||
require.NoError(t, err)
|
||||
|
||||
var buf bytes.Buffer
|
||||
require.NoError(t, enc.Encode(&buf, s))
|
||||
|
||||
keys := topLevelKeys(t, buf.Bytes())
|
||||
require.NotEmpty(t, keys)
|
||||
assert.Equal(t, "schema", keys[len(keys)-1], "pretty=%v", pretty)
|
||||
|
||||
id, version, ok := identifyFromTailBytes(buf.Bytes())
|
||||
assert.True(t, ok, "pretty=%v", pretty)
|
||||
assert.Equal(t, ID, id)
|
||||
assert.Equal(t, internal.JSONSchemaVersion, version)
|
||||
}
|
||||
}
|
||||
|
||||
// noEndSeeker behaves like a reader that buffers a stream as it goes: it can seek within what it has, but cannot
|
||||
// seek relative to the end.
|
||||
type noEndSeeker struct {
|
||||
io.ReadSeeker
|
||||
}
|
||||
|
||||
func (s noEndSeeker) Seek(offset int64, whence int) (int64, error) {
|
||||
if whence == io.SeekEnd {
|
||||
return 0, errors.New("SeekEnd is not supported")
|
||||
}
|
||||
return s.ReadSeeker.Seek(offset, whence)
|
||||
}
|
||||
|
||||
func Test_Identify_fallsBackWhenTailIsUnavailable(t *testing.T) {
|
||||
doc := `{"artifacts":[],` + testIdentifySchemaBlock + `}`
|
||||
r := noEndSeeker{strings.NewReader(doc)}
|
||||
|
||||
id, version := NewFormatDecoder().Identify(r)
|
||||
assert.Equal(t, ID, id)
|
||||
assert.Equal(t, "16.0.0", version)
|
||||
}
|
||||
|
||||
func Test_identifyFromTail_readerEdgeCases(t *testing.T) {
|
||||
doc := `{"artifacts":[],` + testIdentifySchemaBlock + `}`
|
||||
|
||||
t.Run("empty reader", func(t *testing.T) {
|
||||
r := strings.NewReader("")
|
||||
|
||||
_, _, ok := identifyFromTail(r)
|
||||
assert.False(t, ok)
|
||||
})
|
||||
|
||||
t.Run("reader already at the end has nothing to inspect and keeps its position", func(t *testing.T) {
|
||||
r := strings.NewReader(doc)
|
||||
_, err := r.Seek(0, io.SeekEnd)
|
||||
require.NoError(t, err)
|
||||
|
||||
_, _, ok := identifyFromTail(r)
|
||||
assert.False(t, ok)
|
||||
|
||||
pos, err := r.Seek(0, io.SeekCurrent)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int64(len(doc)), pos)
|
||||
})
|
||||
|
||||
t.Run("whitespace around the colon and CRLF line endings", func(t *testing.T) {
|
||||
crlf := "{\r\n \"artifacts\": [],\r\n \"schema\" : {\r\n \"version\": \"16.0.0\",\r\n \"url\": \"" + testIdentifySchemaURL + "\"\r\n }\r\n}\r\n"
|
||||
|
||||
id, version, ok := identifyFromTailBytes([]byte(crlf))
|
||||
require.True(t, ok)
|
||||
assert.Equal(t, ID, id)
|
||||
assert.Equal(t, "16.0.0", version)
|
||||
})
|
||||
|
||||
t.Run("document ending inside a string is not trusted", func(t *testing.T) {
|
||||
truncated := `{"artifacts":[],` + testIdentifySchemaBlock + `,"source":{"name":"unterminated`
|
||||
|
||||
_, _, ok := identifyFromTailBytes([]byte(truncated))
|
||||
assert.False(t, ok)
|
||||
})
|
||||
}
|
||||
|
||||
func topLevelKeys(t *testing.T, doc []byte) []string {
|
||||
dec := json.NewDecoder(bytes.NewReader(doc))
|
||||
|
||||
tok, err := dec.Token()
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, json.Delim('{'), tok)
|
||||
|
||||
var keys []string
|
||||
for dec.More() {
|
||||
tok, err := dec.Token()
|
||||
require.NoError(t, err)
|
||||
keys = append(keys, tok.(string))
|
||||
|
||||
var value json.RawMessage
|
||||
require.NoError(t, dec.Decode(&value))
|
||||
}
|
||||
return keys
|
||||
}
|
||||
@@ -5,6 +5,14 @@ import (
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// Field order matters: encoding/json writes struct fields in declaration order, and Schema must remain the last
|
||||
// field so that the schema block is always the final key of the encoded document. Format identification relies on
|
||||
// finding the schema block by reading only the tail of a (potentially very large) document; moving Schema earlier
|
||||
// would not break decoding but would silently force a full parse of the whole document to identify it.
|
||||
//
|
||||
// (This block is deliberately separated from the doc comment below: the JSON schema generator lifts doc comments
|
||||
// into schema descriptions, and changing the description would alter the published schema.)
|
||||
|
||||
// Document represents the syft cataloging findings as a JSON document
|
||||
type Document struct {
|
||||
Artifacts []Package `json:"artifacts"` // Artifacts is the list of packages discovered and placed into the catalog
|
||||
|
||||
Reference in New Issue
Block a user