Juan
fb59c4e65c
Decoding an SPDX document into the syft model was quadratic in document
size. findRootPackages (spdxhelpers/to_syft_model.go), which locates the
packages the document DESCRIBES, scanned every relationship for every
package. On a 119MB SPDX JSON document (70,701 packages, 153,301
relationships) that is 1.1e10 comparisons and 91% of all CPU samples; on a
238MB document it is 4.3e10 comparisons and 95%. A 1.3GB document did not
finish in 91 minutes. Everything else on the decode path is map-indexed and
linear, and spdx/tools-golang's own parse is linear (4.5s and 9.3s for the
two documents above). The scan dates from 9480f10c (top-level SPDX package
support, 2023).
The function now indexes the DESCRIBES / DESCRIBED_BY relationships once,
counting how many times each element is described by the DOCUMENT, then
walks the packages once. Semantics are unchanged: results follow package
order, an element described by more than one edge still appears once per
edge (so the existing "exactly one root" check behaves as before), only
edges touching DOCUMENT count, and nil entries are skipped.
Measured on an Apple M1 Max, `syft convert <spdx> -o <format>=<file>`:
input target before (main) after
119MB spdx-json@2.3 68.2s / 1.3GB 10.4s / 1.3GB
119MB spdx-json@2.2 70.2s / 1.3GB 12.6s / 1.3GB
238MB spdx-json@2.3 291.2s / 2.3GB 20.9s / 2.6GB
1.3GB spdx-json@2.3 killed at 91min 115.5s / 13.6GB
1.3GB spdx-json@2.2 not attempted 136.3s / 14.1GB
1.3GB syft-json not attempted 102.6s / 14.1GB
1.3GB cyclonedx-json not attempted 114.1s / 13.1GB
Isolating the conversion step (ToSyftModel) from the tools-golang parse:
119MB 72.3s -> 1.5s, 238MB 265.8s -> 3.1s. The remaining decode time and
all of the memory are in tools-golang, which parses the document several
times over; that is a separate, upstream concern.
Output equivalence was checked by converting 10 SPDX documents (the 8 SPDX
fixtures in this repository plus the 119MB and 238MB documents) to
syft-json, spdx-json@2.3, spdx-json@2.2, spdx-tag-value and
cyclonedx-json@1.6 with both binaries and comparing the results with
timestamps and generated ids removed: all 50 pairs are identical.
Tests pin the original semantics (including the duplicate-edge corner case)
and guard the complexity with a 50k x 100k document that must complete in
under 10s; the old implementation takes about a minute on it.
Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Syft
A CLI tool and Go library for generating a Software Bill of Materials (SBOM) from container images and filesystems. Exceptional for vulnerability detection when used with a scanner like Grype.
Features
- Generates SBOMs for container images, filesystems, archives (see the docs for a full list of supported scan targets)
- Supports dozens of packaging ecosystems (e.g. Alpine (apk), Debian (dpkg), RPM, Go, Python, Java, JavaScript, Ruby, Rust, PHP, .NET, and many more)
- Supports OCI, Docker, Singularity, and more image formats
- Works seamlessly with Grype for vulnerability scanning
- Multiple output formats (CycloneDX, SPDX, Syft JSON, and more) including the ability to convert between SBOM formats
- Create signed SBOM attestations using the in-toto specification
Tip
New to Syft? Check out the Getting Started guide for a walkthrough!
Installation
The quickest way to get up and going:
curl -sSfL https://get.anchore.io/syft | sudo sh -s -- -b /usr/local/bin
Tip
See Installation docs for more ways to get Syft, including Homebrew, Docker, Scoop, Chocolatey, Nix, and more!
The basics
See the packages within a container image or directory:
# container image
syft alpine:latest
# directory
syft ./my-project
To get an SBOM, specify one or more output formats:
# SBOM to stdout
syft <image> -o cyclonedx-json
# Multiple SBOMs to files
syft <image> -o spdx-json=./spdx.json -o cyclonedx-json=./cdx.json
Tip
Check out the Getting Started guide to explore all of the capabilities and features.
Want to know all of the ins-and-outs of Syft? Check out the CLI docs, configuration docs, and JSON schema.
Contributing
We encourage users to help make these tools better by submitting issues when you find a bug or want a new feature. Check out our contributing overview and developer-specific documentation if you are interested in providing code contributions.
Syft development is sponsored by Anchore, and is released under the Apache-2.0 License.
The Syft logo by Anchore is licensed under CC BY 4.0
For commercial support options with Syft or Grype, please contact Anchore.
Come talk to us!
The Syft Team holds regular community meetings online. All are welcome to join to bring topics for discussion.
- Check the calendar for the next meeting date.
- Add items to the agenda (join this group for write access to the agenda)
- See you there!