oss/syft
2
0
Fork 0
mirror of https://github.com/anchore/syft.git synced 2026-11-04 16:36:40 +01:00
Code Issues Packages Projects Releases Wiki Activity

perf(spdx): make findRootPackages linear instead of packages x relationships (#5280)

Browse Source 
Decoding an SPDX document into the syft model was quadratic in document
size. findRootPackages (spdxhelpers/to_syft_model.go), which locates the
packages the document DESCRIBES, scanned every relationship for every
package. On a 119MB SPDX JSON document (70,701 packages, 153,301
relationships) that is 1.1e10 comparisons and 91% of all CPU samples; on a
238MB document it is 4.3e10 comparisons and 95%. A 1.3GB document did not
finish in 91 minutes. Everything else on the decode path is map-indexed and
linear, and spdx/tools-golang's own parse is linear (4.5s and 9.3s for the
two documents above). The scan dates from 9480f10c (top-level SPDX package
support, 2023).

The function now indexes the DESCRIBES / DESCRIBED_BY relationships once,
counting how many times each element is described by the DOCUMENT, then
walks the packages once. Semantics are unchanged: results follow package
order, an element described by more than one edge still appears once per
edge (so the existing "exactly one root" check behaves as before), only
edges touching DOCUMENT count, and nil entries are skipped.

Measured on an Apple M1 Max, `syft convert <spdx> -o <format>=<file>`:

  input       target          before (main)      after
  119MB       spdx-json@2.3   68.2s / 1.3GB      10.4s / 1.3GB
  119MB       spdx-json@2.2   70.2s / 1.3GB      12.6s / 1.3GB
  238MB       spdx-json@2.3   291.2s / 2.3GB     20.9s / 2.6GB
  1.3GB       spdx-json@2.3   killed at 91min    115.5s / 13.6GB
  1.3GB       spdx-json@2.2   not attempted      136.3s / 14.1GB
  1.3GB       syft-json       not attempted      102.6s / 14.1GB
  1.3GB       cyclonedx-json  not attempted      114.1s / 13.1GB

Isolating the conversion step (ToSyftModel) from the tools-golang parse:
119MB 72.3s -> 1.5s, 238MB 265.8s -> 3.1s. The remaining decode time and
all of the memory are in tools-golang, which parses the document several
times over; that is a separate, upstream concern.

Output equivalence was checked by converting 10 SPDX documents (the 8 SPDX
fixtures in this repository plus the 119MB and 238MB documents) to
syft-json, spdx-json@2.3, spdx-json@2.2, spdx-tag-value and
cyclonedx-json@1.6 with both binaries and comparing the results with
timestamps and generated ids removed: all 50 pairs are identical.

Tests pin the original semantics (including the duplicate-edge corner case)
and guard the complexity with a 50k x 100k document that must complete in
under 10s; the old implementation takes about a minute on it.

Signed-off-by: juanjsebgarcia <github@juangarcia.co.uk>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
This commit is contained in:
Juan
2026-09-11 21:11:21 +00:00
committed by GitHub
parent c4800b2393
commit fb59c4e65c
2 changed files with 204 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
syft/format/common/spdxhelpers/to_syft_model.go
+23 -13
View File
@@ -83,20 +83,30 @@ func removeRelationships(relationships []*spdx.Relationship, spdxID spdx.Element
}
func findRootPackages(doc *spdx.Document) (out []*spdx.Package) {
// index the DESCRIBES / DESCRIBED_BY relationships once, counting how many
// times each element is described by the document, so that the result
// matches the previous packages x relationships scan exactly (one entry per
// matching relationship) without the quadratic cost.
describedCount := make(map[spdx.ElementID]int)
for _, r := range doc.Relationships {
if r == nil {
continue
}
switch {
case r.RefA.ElementRefID == "DOCUMENT" && r.Relationship == spdx.RelationshipDescribes:
describedCount[r.RefB.ElementRefID]++
case r.RefB.ElementRefID == "DOCUMENT" && r.Relationship == spdx.RelationshipDescribedBy:
describedCount[r.RefA.ElementRefID]++
}
}
if len(describedCount) == 0 {
return nil
}
for _, p := range doc.Packages {
for _, r := range doc.Relationships {
describes := r.RefA.ElementRefID == "DOCUMENT" &&
r.Relationship == spdx.RelationshipDescribes &&
r.RefB.ElementRefID == p.PackageSPDXIdentifier
describedBy := r.RefB.ElementRefID == "DOCUMENT" &&
r.Relationship == spdx.RelationshipDescribedBy &&
r.RefA.ElementRefID == p.PackageSPDXIdentifier
if !describes && !describedBy {
continue
}
if p == nil {
continue
}
for i := 0; i < describedCount[p.PackageSPDXIdentifier]; i++ {
out = append(out, p)
}
}