Compare commits

..
Author SHA1 Message Date
Alex Goodman f2aa4caf55 [wip] git-root config search concept
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-11-30 16:43:41 -05:00
Alex Goodman fe616acd98 directory resolver should account for the proc cwd relative to the root path (#644)
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-11-30 15:34:37 -05:00
Blaize KayeandAlex Goodman 6af132e088 Adds composer installed.json parser (#643)
* Adds installed.json functionality and tests

Signed-off-by: Blaize Kaye <blaize.kaye@amazee.com>

* Adds php-installed-cataloger

Signed-off-by: Blaize Kaye <blaize.kaye@amazee.com>

* Changes fallback logic

Signed-off-by: Blaize Kaye <blaize.kaye@amazee.com>

* Adds image tests for installed.json composer packages

Signed-off-by: Blaize Kaye <blaize.kaye@amazee.com>

* tweak PHP cataloger names

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* unexport PHP types and fix CLI tests

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* rename PHP cataloger file

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

Co-authored-by: Alex Goodman <alex.goodman@anchore.com>
2021-11-30 11:36:08 -05:00
Alex Goodman 00206233e6 use smaller jotframe event buffer (#647)
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-11-30 11:23:16 -05:00
Alex Goodman bd9007fc0e Migrate SPDX-JSON relationships to SBOM model (#634)
* remove power-user document shape

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add power-user specific fields to syft-json format

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* port remaining spdx-json relationships to sbom model

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add coordinate set

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add SBOM file path helper

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* use internal mimetype helper in go binary cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add new package-of relationship

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update json schema to v2

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* replace power-user presenter with syft-json format

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix tests and linting

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* remove "package-of" relationship (in favor of "contains")

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add tests for spdx22json format encoding enhancements

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update TODO and log entries

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* introduce sbom.Descriptor

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-11-23 14:54:17 -05:00
Alex Goodman e3b34813d7 ignore partybus unsubscribe errors (#641)
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-11-22 16:07:55 -05:00
Dan Luhring f0b2d81599 Announce meetup on 2021-12-01 (#640)
Signed-off-by: Dan Luhring <dan+github@luhrings.com>
2021-11-22 10:13:18 -05:00
Christopher Angelo Phillips 0849539729 Update concurrency primitives for relationships in command layer (#639)
* update concurrency primitives for relationships in command layer

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
2021-11-20 03:44:17 -05:00
Christopher Angelo Phillips 4f0099583a promote catalog task pattern to all commands (#636)
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
2021-11-19 13:26:23 -05:00
Alex Goodman d76c868481 Catalog archive contents for single-file input (#637)
* add first-level archive processing when input is a file

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add license exception for github.com/xi2/xz

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* always return cleanup function

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* change source.NewFromFile log entry to warn

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* ensure file source always has cleanup function

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* ensure we are always preferring the unarchive cleanup function for source

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-11-19 14:16:25 +00:00
Alex GoodmanandDan Luhring e38cde35ed Introduce minimal source coordinates (#623)
* split source.Location and create source.Coordinates for minimal path addressing

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* move coordinates into separate file

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* Update syft/source/coordinates.go

Co-authored-by: Dan Luhring <luhring@users.noreply.github.com>
2021-11-18 18:13:22 +00:00
Alex Goodman 9090c3a772 bump stereoscope, docker, and GGCR (#633)
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-11-16 19:43:46 +00:00
Alex Goodman ef627d82ef Introduce relationships as first-class objects (#607)
* migrate pkg.ID and pkg.Relationship to artifact package

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* return relationships from tasks

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix more tests

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add artifact.Identifiable by Identity() method

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix linting

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* remove catalog ID assignment

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* adjust spdx helpers to use copy of packages

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* stabilize package ID relative to encode-decode format cycles

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* rename Identity() to ID()

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* use zero value for nils in ID generation

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* enable source.Location to be identifiable

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* hoist up package relationship discovery to analysis stage

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update ownership-by-file-overlap relationship description

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add test reminders to put new relationships under test

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* adjust PHP composer.lock parser function to return relationships

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-11-16 14:14:13 -05:00
164 changed files with 4909 additions and 2185 deletions
+5
View File
@@ -15,3 +15,8 @@ ignore-packages:
# * GNU General Public License, version 2.0 or later (GPL-2.0-or-later).
# (we choose Apache-2.0)
- github.com/spdx/tools-golang
# from: https://github.com/xi2/xz/blob/master/LICENSE
# All these files have been put into the public domain.
# You can do whatever you want with these files.
- github.com/xi2/xz
+1 -1
View File
@@ -135,7 +135,7 @@ lint-fix: ## Auto-format all source code + run golangci lint fixers
go mod tidy
.PHONY: check-licenses
check-licenses:
check-licenses: ## Ensure transitive dependencies are compliant with the current license policy
$(TEMPDIR)/bouncer check
check-go-mod-tidy:
+12
View File
@@ -11,6 +11,16 @@
A CLI tool and Go library for generating a Software Bill of Materials (SBOM) from container images and filesystems. Exceptional for vulnerability detection when used with a scanner tool like [Grype](https://github.com/anchore/grype).
### Join us at the virtual OSS meetup!
When: December 1, 2021 @ 11am-noon PT (2-3pm ET)
We’ve got security experts from VMware and Anchore who will share their tips, tricks, and lessons learned on securing containers using [Syft](https://github.com/anchore/syft), [Grype](https://github.com/anchore/grype), and [Cartographer](https://github.com/vmware-tanzu/cartographer).
Register now: https://get.anchore.com/anchore-oss-meetup-dec-1-2021/
---
![syft-demo](https://user-images.githubusercontent.com/590471/90277200-2a253000-de33-11ea-893f-32c219eea11a.gif)
## Features
@@ -191,6 +201,8 @@ Configuration search paths:
- `.syft.yaml`
- `.syft/config.yaml`
- `<git repo root>/.syft.yaml`
- `<git repo root>/.syft/config.yaml`
- `~/.syft.yaml`
- `<XDG_CONFIG_HOME>/syft/config.yaml`
+8 -2
View File
@@ -48,7 +48,13 @@ func initCmdAliasBindings() {
panic(err)
}
if activeCmd == packagesCmd || activeCmd == rootCmd {
// enable all cataloger by default if power-user command is run
if activeCmd == powerUserCmd {
config.PowerUserCatalogerEnabledDefault()
}
switch activeCmd {
case packagesCmd, rootCmd:
// note: we need to lazily bind config options since they are shared between both the root command
// and the packages command. Otherwise there will be global viper state that is in contention.
// See for more details: https://github.com/spf13/viper/issues/233 . Additionally, the bindings must occur BEFORE
@@ -58,7 +64,7 @@ func initCmdAliasBindings() {
if err = bindPackagesConfigOptions(activeCmd.Flags()); err != nil {
panic(err)
}
} else {
default:
// even though the root command or packages command is NOT being run, we still need default bindings
// such that application config parsing passes.
if err = bindPackagesConfigOptions(packagesCmd.Flags()); err != nil {
+2 -4
View File
@@ -14,7 +14,7 @@ import (
// eventLoop listens to worker errors (from execution path), worker events (from a partybus subscription), and
// signal interrupts. Is responsible for handling each event relative to a given UI an to coordinate eventing until
// an eventual graceful exit.
// nolint:gocognit,funlen
// nolint:funlen
func eventLoop(workerErrs <-chan error, signals <-chan os.Signal, subscription *partybus.Subscription, cleanupFn func(), uxs ...ui.UI) error {
defer cleanupFn()
events := subscription.Events()
@@ -41,9 +41,7 @@ func eventLoop(workerErrs <-chan error, signals <-chan os.Signal, subscription *
if err != nil {
// capture the error from the worker and unsubscribe to complete a graceful shutdown
retErr = multierror.Append(retErr, err)
if err := subscription.Unsubscribe(); err != nil {
retErr = multierror.Append(retErr, err)
}
_ = subscription.Unsubscribe()
// the worker has exited, we may have been mid-handling events for the UI which should now be
// ignored, in which case forcing a teardown of the UI irregardless of the state is required.
forceTeardown = true
+38 -14
View File
@@ -13,7 +13,8 @@ import (
"github.com/anchore/syft/internal/formats"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/internal/ui"
"github.com/anchore/syft/syft"
"github.com/anchore/syft/internal/version"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/event"
"github.com/anchore/syft/syft/format"
"github.com/anchore/syft/syft/sbom"
@@ -238,6 +239,12 @@ func packagesExecWorker(userInput string) <-chan error {
go func() {
defer close(errs)
tasks, err := tasks()
if err != nil {
errs <- err
return
}
f := formats.ByOption(packagesPresenterOpt)
if f == nil {
errs <- fmt.Errorf("unknown format: %s", packagesPresenterOpt)
@@ -251,24 +258,30 @@ func packagesExecWorker(userInput string) <-chan error {
errs <- fmt.Errorf("failed to determine image source: %w", err)
return
}
defer cleanup()
catalog, d, err := syft.CatalogPackages(src, appConfig.Package.Cataloger.ScopeOpt)
if err != nil {
errs <- fmt.Errorf("failed to catalog input: %w", err)
return
if cleanup != nil {
defer cleanup()
}
sbomResult := sbom.SBOM{
Artifacts: sbom.Artifacts{
PackageCatalog: catalog,
Distro: d,
},
s := sbom.SBOM{
Source: src.Metadata,
Descriptor: sbom.Descriptor{
Name: internal.ApplicationName,
Version: version.FromBuild().Version,
Configuration: appConfig,
},
}
var relationships []<-chan artifact.Relationship
for _, task := range tasks {
c := make(chan artifact.Relationship)
relationships = append(relationships, c)
go runTask(task, &s.Artifacts, src, c, errs)
}
s.Relationships = append(s.Relationships, mergeRelationships(relationships...)...)
if appConfig.Anchore.Host != "" {
if err := runPackageSbomUpload(src, sbomResult); err != nil {
if err := runPackageSbomUpload(src, s); err != nil {
errs <- err
return
}
@@ -276,12 +289,22 @@ func packagesExecWorker(userInput string) <-chan error {
bus.Publish(partybus.Event{
Type: event.PresenterReady,
Value: f.Presenter(sbomResult),
Value: f.Presenter(s),
})
}()
return errs
}
func mergeRelationships(cs ...<-chan artifact.Relationship) (relationships []artifact.Relationship) {
for _, c := range cs {
for n := range c {
relationships = append(relationships, n)
}
}
return relationships
}
func runPackageSbomUpload(src *source.Source, s sbom.SBOM) error {
log.Infof("uploading results to %s", appConfig.Anchore.Host)
@@ -326,5 +349,6 @@ func runPackageSbomUpload(src *source.Source, s sbom.SBOM) error {
if err := c.Import(context.Background(), importCfg); err != nil {
return fmt.Errorf("failed to upload results to host=%s: %+v", appConfig.Anchore.Host, err)
}
return nil
}
+34 -19
View File
@@ -2,18 +2,20 @@ package cmd
import (
"fmt"
"sync"
"github.com/anchore/syft/syft/sbom"
"os"
"github.com/anchore/stereoscope"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/bus"
"github.com/anchore/syft/internal/formats/syftjson"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/internal/presenter/poweruser"
"github.com/anchore/syft/internal/ui"
"github.com/anchore/syft/internal/version"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/event"
"github.com/anchore/syft/syft/sbom"
"github.com/anchore/syft/syft/source"
"github.com/gookit/color"
"github.com/pkg/profile"
"github.com/spf13/cobra"
"github.com/wagoodman/go-partybus"
@@ -21,6 +23,8 @@ import (
const powerUserExample = ` {{.appName}} {{.command}} <image>
DEPRECATED - THIS COMMAND WILL BE REMOVED in v1.0.0
Only image sources are supported (e.g. docker: , docker-archive: , oci: , etc.), the directory source (dir:) is not supported.
All behavior is controlled via application configuration and environment variables (see https://github.com/anchore/syft#configuration)
@@ -74,6 +78,10 @@ func powerUserExec(_ *cobra.Command, args []string) error {
if err := closer(); err != nil {
log.Warnf("unable to write to report destination: %+v", err)
}
// inform user at end of run that command will be removed
deprecated := color.Style{color.Red, color.OpBold}.Sprint("DEPRECATED: This command will be removed in v1.0.0")
fmt.Fprintln(os.Stderr, deprecated)
}()
if err != nil {
@@ -88,13 +96,16 @@ func powerUserExec(_ *cobra.Command, args []string) error {
ui.Select(isVerbose(), appConfig.Quiet, reporter)...,
)
}
func powerUserExecWorker(userInput string) <-chan error {
errs := make(chan error)
go func() {
defer close(errs)
tasks, err := powerUserTasks()
appConfig.Secrets.Cataloger.Enabled = true
appConfig.FileMetadata.Cataloger.Enabled = true
appConfig.FileContents.Cataloger.Enabled = true
appConfig.FileClassification.Cataloger.Enabled = true
tasks, err := tasks()
if err != nil {
errs <- err
return
@@ -107,30 +118,34 @@ func powerUserExecWorker(userInput string) <-chan error {
errs <- err
return
}
defer cleanup()
if cleanup != nil {
defer cleanup()
}
analysisResults := sbom.SBOM{
s := sbom.SBOM{
Source: src.Metadata,
Descriptor: sbom.Descriptor{
Name: internal.ApplicationName,
Version: version.FromBuild().Version,
Configuration: appConfig,
},
}
wg := &sync.WaitGroup{}
var relationships []<-chan artifact.Relationship
for _, task := range tasks {
wg.Add(1)
go func(task powerUserTask) {
defer wg.Done()
if err = task(&analysisResults.Artifacts, src); err != nil {
errs <- err
return
}
}(task)
c := make(chan artifact.Relationship)
relationships = append(relationships, c)
go runTask(task, &s.Artifacts, src, c, errs)
}
wg.Wait()
s.Relationships = append(s.Relationships, mergeRelationships(relationships...)...)
bus.Publish(partybus.Event{
Type: event.PresenterReady,
Value: poweruser.NewJSONPresenter(analysisResults, *appConfig),
Value: syftjson.Format().Presenter(s),
})
}()
return errs
}
+57 -42
View File
@@ -4,25 +4,25 @@ import (
"crypto"
"fmt"
"github.com/anchore/syft/syft/sbom"
"github.com/anchore/syft/syft"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/sbom"
"github.com/anchore/syft/syft/source"
)
type powerUserTask func(*sbom.Artifacts, *source.Source) error
type task func(*sbom.Artifacts, *source.Source) ([]artifact.Relationship, error)
func powerUserTasks() ([]powerUserTask, error) {
var tasks []powerUserTask
func tasks() ([]task, error) {
var tasks []task
generators := []func() (powerUserTask, error){
catalogPackagesTask,
catalogFileMetadataTask,
catalogFileDigestsTask,
catalogSecretsTask,
catalogFileClassificationsTask,
catalogContentsTask,
generators := []func() (task, error){
generateCatalogPackagesTask,
generateCatalogFileMetadataTask,
generateCatalogFileDigestsTask,
generateCatalogSecretsTask,
generateCatalogFileClassificationsTask,
generateCatalogContentsTask,
}
for _, generator := range generators {
@@ -30,6 +30,7 @@ func powerUserTasks() ([]powerUserTask, error) {
if err != nil {
return nil, err
}
if task != nil {
tasks = append(tasks, task)
}
@@ -38,51 +39,51 @@ func powerUserTasks() ([]powerUserTask, error) {
return tasks, nil
}
func catalogPackagesTask() (powerUserTask, error) {
func generateCatalogPackagesTask() (task, error) {
if !appConfig.Package.Cataloger.Enabled {
return nil, nil
}
task := func(results *sbom.Artifacts, src *source.Source) error {
packageCatalog, theDistro, err := syft.CatalogPackages(src, appConfig.Package.Cataloger.ScopeOpt)
task := func(results *sbom.Artifacts, src *source.Source) ([]artifact.Relationship, error) {
packageCatalog, relationships, theDistro, err := syft.CatalogPackages(src, appConfig.Package.Cataloger.ScopeOpt)
if err != nil {
return err
return nil, err
}
results.PackageCatalog = packageCatalog
results.Distro = theDistro
return nil
return relationships, nil
}
return task, nil
}
func catalogFileMetadataTask() (powerUserTask, error) {
func generateCatalogFileMetadataTask() (task, error) {
if !appConfig.FileMetadata.Cataloger.Enabled {
return nil, nil
}
metadataCataloger := file.NewMetadataCataloger()
task := func(results *sbom.Artifacts, src *source.Source) error {
task := func(results *sbom.Artifacts, src *source.Source) ([]artifact.Relationship, error) {
resolver, err := src.FileResolver(appConfig.FileMetadata.Cataloger.ScopeOpt)
if err != nil {
return err
return nil, err
}
result, err := metadataCataloger.Catalog(resolver)
if err != nil {
return err
return nil, err
}
results.FileMetadata = result
return nil
return nil, nil
}
return task, nil
}
func catalogFileDigestsTask() (powerUserTask, error) {
func generateCatalogFileDigestsTask() (task, error) {
if !appConfig.FileMetadata.Cataloger.Enabled {
return nil, nil
}
@@ -111,24 +112,24 @@ func catalogFileDigestsTask() (powerUserTask, error) {
return nil, err
}
task := func(results *sbom.Artifacts, src *source.Source) error {
task := func(results *sbom.Artifacts, src *source.Source) ([]artifact.Relationship, error) {
resolver, err := src.FileResolver(appConfig.FileMetadata.Cataloger.ScopeOpt)
if err != nil {
return err
return nil, err
}
result, err := digestsCataloger.Catalog(resolver)
if err != nil {
return err
return nil, err
}
results.FileDigests = result
return nil
return nil, nil
}
return task, nil
}
func catalogSecretsTask() (powerUserTask, error) {
func generateCatalogSecretsTask() (task, error) {
if !appConfig.Secrets.Cataloger.Enabled {
return nil, nil
}
@@ -143,24 +144,24 @@ func catalogSecretsTask() (powerUserTask, error) {
return nil, err
}
task := func(results *sbom.Artifacts, src *source.Source) error {
task := func(results *sbom.Artifacts, src *source.Source) ([]artifact.Relationship, error) {
resolver, err := src.FileResolver(appConfig.Secrets.Cataloger.ScopeOpt)
if err != nil {
return err
return nil, err
}
result, err := secretsCataloger.Catalog(resolver)
if err != nil {
return err
return nil, err
}
results.Secrets = result
return nil
return nil, nil
}
return task, nil
}
func catalogFileClassificationsTask() (powerUserTask, error) {
func generateCatalogFileClassificationsTask() (task, error) {
if !appConfig.FileClassification.Cataloger.Enabled {
return nil, nil
}
@@ -171,24 +172,24 @@ func catalogFileClassificationsTask() (powerUserTask, error) {
return nil, err
}
task := func(results *sbom.Artifacts, src *source.Source) error {
task := func(results *sbom.Artifacts, src *source.Source) ([]artifact.Relationship, error) {
resolver, err := src.FileResolver(appConfig.FileClassification.Cataloger.ScopeOpt)
if err != nil {
return err
return nil, err
}
result, err := classifierCataloger.Catalog(resolver)
if err != nil {
return err
return nil, err
}
results.FileClassifications = result
return nil
return nil, nil
}
return task, nil
}
func catalogContentsTask() (powerUserTask, error) {
func generateCatalogContentsTask() (task, error) {
if !appConfig.FileContents.Cataloger.Enabled {
return nil, nil
}
@@ -198,19 +199,33 @@ func catalogContentsTask() (powerUserTask, error) {
return nil, err
}
task := func(results *sbom.Artifacts, src *source.Source) error {
task := func(results *sbom.Artifacts, src *source.Source) ([]artifact.Relationship, error) {
resolver, err := src.FileResolver(appConfig.FileContents.Cataloger.ScopeOpt)
if err != nil {
return err
return nil, err
}
result, err := contentsCataloger.Catalog(resolver)
if err != nil {
return err
return nil, err
}
results.FileContents = result
return nil
return nil, nil
}
return task, nil
}
func runTask(t task, a *sbom.Artifacts, src *source.Source, c chan<- artifact.Relationship, errs chan<- error) {
defer close(c)
relationships, err := t(a, src)
if err != nil {
errs <- err
return
}
for _, relationship := range relationships {
c <- relationship
}
}
+19 -13
View File
@@ -12,41 +12,47 @@ require (
github.com/anchore/go-testutils v0.0.0-20200925183923-d5f45b0d3c04
github.com/anchore/go-version v1.2.2-0.20200701162849-18adb9c92b9b
github.com/anchore/packageurl-go v0.0.0-20210922164639-b3fa992ebd29
github.com/anchore/stereoscope v0.0.0-20211024152658-003132a67c10
github.com/anchore/stereoscope v0.0.0-20211116152349-7e4e1b56a15d
// we are hinting brotli to latest due to warning when installing archiver v3:
// go: warning: github.com/andybalholm/brotli@v1.0.1: retracted by module author: occasional panics and data corruption
github.com/andybalholm/brotli v1.0.4 // indirect
github.com/antihax/optional v1.0.0
github.com/bmatcuk/doublestar/v2 v2.0.4
github.com/docker/docker v17.12.0-ce-rc1.0.20200309214505-aa6a9891b09c+incompatible
github.com/docker/docker v20.10.10+incompatible
github.com/dustin/go-humanize v1.0.0
github.com/facebookincubator/nvdtools v0.1.4
github.com/go-test/deep v1.0.7
github.com/google/go-cmp v0.4.1
github.com/google/go-cmp v0.5.6
github.com/google/uuid v1.2.0
github.com/gookit/color v1.2.7
github.com/hashicorp/go-multierror v1.1.0
github.com/hashicorp/go-version v1.2.0
github.com/jinzhu/copier v0.3.2
github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d // indirect
github.com/mholt/archiver/v3 v3.5.1
github.com/mitchellh/go-homedir v1.1.0
github.com/mitchellh/hashstructure v1.1.0
github.com/mitchellh/mapstructure v1.3.1
github.com/mitchellh/hashstructure/v2 v2.0.2
github.com/mitchellh/mapstructure v1.4.1
github.com/olekukonko/tablewriter v0.0.4
github.com/pelletier/go-toml v1.8.1
github.com/pelletier/go-toml v1.9.3
github.com/pkg/profile v1.5.0
github.com/scylladb/go-set v1.0.2
github.com/sergi/go-diff v1.1.0
github.com/sirupsen/logrus v1.8.1
github.com/spdx/tools-golang v0.1.0
github.com/spf13/afero v1.2.2
github.com/spf13/cobra v1.0.1-0.20200909172742-8a63648dd905
github.com/spf13/afero v1.6.0
github.com/spf13/cobra v1.2.1
github.com/spf13/pflag v1.0.5
github.com/spf13/viper v1.7.0
github.com/spf13/viper v1.8.1
github.com/stretchr/testify v1.7.0
github.com/vifraa/gopom v0.1.0
github.com/wagoodman/go-partybus v0.0.0-20210627031916-db1f5573bbc5
github.com/wagoodman/go-progress v0.0.0-20200731105512-1020f39e6240
github.com/wagoodman/jotframe v0.0.0-20200730190914-3517092dd163
github.com/wagoodman/jotframe v0.0.0-20211129225309-56b0d0a4aebb
github.com/x-cray/logrus-prefixed-formatter v0.5.2
github.com/xeipuuv/gojsonschema v1.2.0
golang.org/x/mod v0.3.0
golang.org/x/net v0.0.0-20210813160813-60bc85c4be6d
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1
golang.org/x/mod v0.4.2
golang.org/x/net v0.0.0-20211111160137-58aab5ef257a
golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b
gopkg.in/yaml.v2 v2.4.0
)
+661 -449
View File
File diff suppressed because it is too large Load Diff
+8 -4
View File
@@ -59,8 +59,10 @@ func TestPackageSbomToModel(t *testing.T) {
FoundBy: "foundBy",
Locations: []source.Location{
{
RealPath: "path",
FileSystemID: "layerID",
Coordinates: source.Coordinates{
RealPath: "path",
FileSystemID: "layerID",
},
},
},
Licenses: []string{"license"},
@@ -157,8 +159,10 @@ func TestPackageSbomImport(t *testing.T) {
FoundBy: "foundBy",
Locations: []source.Location{
{
RealPath: "path",
FileSystemID: "layerID",
Coordinates: source.Coordinates{
RealPath: "path",
FileSystemID: "layerID",
},
},
},
Licenses: []string{"license"},
+68 -18
View File
@@ -3,7 +3,10 @@ package config
import (
"errors"
"fmt"
"os"
"os/exec"
"path"
"path/filepath"
"reflect"
"strings"
@@ -17,6 +20,8 @@ import (
var ErrApplicationConfigNotFound = fmt.Errorf("application config not found")
var catalogerEnabledDefault = false
type defaultValueLoader interface {
loadDefaultValues(*viper.Viper)
}
@@ -44,6 +49,11 @@ type Application struct {
Registry registry `yaml:"registry" json:"registry" mapstructure:"registry"`
}
// PowerUserCatalogerEnabledDefault switches all catalogers to be enabled when running power-user command
func PowerUserCatalogerEnabledDefault() {
catalogerEnabledDefault = true
}
func newApplicationConfig(v *viper.Viper, cliOpts CliOnlyOptions) *Application {
config := &Application{
CliOptions: cliOpts,
@@ -156,6 +166,10 @@ func (cfg *Application) parseLogLevelOption() error {
}
}
if cfg.Log.Level == "" {
cfg.Log.Level = cfg.Log.LevelOpt.String()
}
return nil
}
@@ -195,44 +209,80 @@ func readConfig(v *viper.Viper, configPath string) error {
// 1. look for .<appname>.yaml (in the current directory)
v.AddConfigPath(".")
v.SetConfigName("." + internal.ApplicationName)
if err = v.ReadInConfig(); err == nil {
return nil
} else if !errors.As(err, &viper.ConfigFileNotFoundError{}) {
return fmt.Errorf("unable to parse config=%q: %w", v.ConfigFileUsed(), err)
if exists, err := readViperConfig(v); exists || err != nil {
return err
}
// 2. look for .<appname>/config.yaml (in the current directory)
v.AddConfigPath("." + internal.ApplicationName)
v.SetConfigName("config")
if err = v.ReadInConfig(); err == nil {
return nil
} else if !errors.As(err, &viper.ConfigFileNotFoundError{}) {
return fmt.Errorf("unable to parse config=%q: %w", v.ConfigFileUsed(), err)
if exists, err := readViperConfig(v); exists || err != nil {
return err
}
// 3. look for ~/.<appname>.yaml
if root := gitRepoRoot(); root != "" {
// 3. look for <git repo root>/.<appname>.yaml, where the repo root is defined by "git rev-parse --show-toplevel"
v.AddConfigPath(root)
v.SetConfigName("." + internal.ApplicationName)
if exists, err := readViperConfig(v); exists {
if err != nil {
return err
}
return os.Chdir(root)
}
// 4. look for <git repo root>/.<appname>/config.yaml, where the repo root is defined by "git rev-parse --show-toplevel"
v.AddConfigPath(filepath.Join(root, "."+internal.ApplicationName))
v.SetConfigName("config")
if exists, err := readViperConfig(v); exists {
if err != nil {
return err
}
return os.Chdir(root)
}
}
// 5. look for ~/.<appname>.yaml
home, err := homedir.Dir()
if err == nil {
v.AddConfigPath(home)
v.SetConfigName("." + internal.ApplicationName)
if err = v.ReadInConfig(); err == nil {
return nil
} else if !errors.As(err, &viper.ConfigFileNotFoundError{}) {
return fmt.Errorf("unable to parse config=%q: %w", v.ConfigFileUsed(), err)
if exists, err := readViperConfig(v); exists || err != nil {
return err
}
}
// 4. look for <appname>/config.yaml in xdg locations (starting with xdg home config dir, then moving upwards)
// 6. look for <appname>/config.yaml in xdg locations (starting with xdg home config dir, then moving upwards)
v.AddConfigPath(path.Join(xdg.ConfigHome, internal.ApplicationName))
for _, dir := range xdg.ConfigDirs {
v.AddConfigPath(path.Join(dir, internal.ApplicationName))
}
v.SetConfigName("config")
if err = v.ReadInConfig(); err == nil {
return nil
} else if !errors.As(err, &viper.ConfigFileNotFoundError{}) {
return fmt.Errorf("unable to parse config=%q: %w", v.ConfigFileUsed(), err)
if exists, err := readViperConfig(v); exists || err != nil {
return err
}
return ErrApplicationConfigNotFound
}
func readViperConfig(v *viper.Viper) (bool, error) {
err := v.ReadInConfig()
if err == nil {
return true, nil
} else if errors.As(err, &viper.ConfigFileNotFoundError{}) {
return false, nil
}
return true, fmt.Errorf("unable to parse config=%q: %w", v.ConfigFileUsed(), err)
}
func gitRepoRoot() string {
root, err := exec.Command("git", "rev-parse", "--show-toplevel").Output()
if err != nil {
return ""
}
absRepoRoot, err := filepath.Abs(strings.TrimSpace(string(root)))
if err != nil {
return ""
}
return absRepoRoot
}
+1 -1
View File
@@ -10,7 +10,7 @@ type fileClassification struct {
}
func (cfg fileClassification) loadDefaultValues(v *viper.Viper) {
v.SetDefault("file-classification.cataloger.enabled", true)
v.SetDefault("file-classification.cataloger.enabled", catalogerEnabledDefault)
v.SetDefault("file-classification.cataloger.scope", source.SquashedScope)
}
+1 -1
View File
@@ -13,7 +13,7 @@ type fileContents struct {
}
func (cfg fileContents) loadDefaultValues(v *viper.Viper) {
v.SetDefault("file-contents.cataloger.enabled", true)
v.SetDefault("file-contents.cataloger.enabled", catalogerEnabledDefault)
v.SetDefault("file-contents.cataloger.scope", source.SquashedScope)
v.SetDefault("file-contents.skip-files-above-size", 1*file.MB)
v.SetDefault("file-contents.globs", []string{})
+1 -1
View File
@@ -11,7 +11,7 @@ type FileMetadata struct {
}
func (cfg FileMetadata) loadDefaultValues(v *viper.Viper) {
v.SetDefault("file-metadata.cataloger.enabled", true)
v.SetDefault("file-metadata.cataloger.enabled", catalogerEnabledDefault)
v.SetDefault("file-metadata.cataloger.scope", source.SquashedScope)
v.SetDefault("file-metadata.digests", []string{"sha256"})
}
+1 -1
View File
@@ -15,7 +15,7 @@ type secrets struct {
}
func (cfg secrets) loadDefaultValues(v *viper.Viper) {
v.SetDefault("secrets.cataloger.enabled", true)
v.SetDefault("secrets.cataloger.enabled", catalogerEnabledDefault)
v.SetDefault("secrets.cataloger.scope", source.AllLayersScope)
v.SetDefault("secrets.reveal-values", false)
v.SetDefault("secrets.skip-files-above-size", 1*file.MB)
+1 -1
View File
@@ -6,5 +6,5 @@ const (
// JSONSchemaVersion is the current schema version output by the JSON presenter
// This is roughly following the "SchemaVer" guidelines for versioning the JSON schema. Please see schema/json/README.md for details on how to increment.
JSONSchemaVersion = "1.1.0"
JSONSchemaVersion = "2.0.0"
)
@@ -2,7 +2,7 @@ package spdxhelpers
import "github.com/anchore/syft/syft/pkg"
func Description(p *pkg.Package) string {
func Description(p pkg.Package) string {
if hasMetadata(p) {
switch metadata := p.Metadata.(type) {
case pkg.ApkMetadata:
@@ -14,10 +14,6 @@ func Description(p *pkg.Package) string {
return ""
}
func packageExists(p *pkg.Package) bool {
return p != nil
}
func hasMetadata(p *pkg.Package) bool {
return packageExists(p) && p.Metadata != nil
func hasMetadata(p pkg.Package) bool {
return p.Metadata != nil
}
@@ -50,7 +50,7 @@ func Test_Description(t *testing.T) {
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
assert.Equal(t, test.expected, Description(&test.input))
assert.Equal(t, test.expected, Description(test.input))
})
}
}
@@ -2,7 +2,7 @@ package spdxhelpers
import "github.com/anchore/syft/syft/pkg"
func DownloadLocation(p *pkg.Package) string {
func DownloadLocation(p pkg.Package) string {
// 3.7: Package Download Location
// Cardinality: mandatory, one
// NONE if there is no download location whatsoever.
@@ -48,7 +48,7 @@ func Test_DownloadLocation(t *testing.T) {
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
assert.Equal(t, test.expected, DownloadLocation(&test.input))
assert.Equal(t, test.expected, DownloadLocation(test.input))
})
}
}
@@ -6,13 +6,9 @@ import (
"github.com/anchore/syft/syft/pkg"
)
func ExternalRefs(p *pkg.Package) (externalRefs []model.ExternalRef) {
func ExternalRefs(p pkg.Package) (externalRefs []model.ExternalRef) {
externalRefs = make([]model.ExternalRef, 0)
if !packageExists(p) {
return externalRefs
}
for _, c := range p.CPEs {
externalRefs = append(externalRefs, model.ExternalRef{
ReferenceCategory: model.SecurityReferenceCategory,
@@ -39,7 +39,7 @@ func Test_ExternalRefs(t *testing.T) {
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
assert.ElementsMatch(t, test.expected, ExternalRefs(&test.input))
assert.ElementsMatch(t, test.expected, ExternalRefs(test.input))
})
}
}
@@ -1,51 +0,0 @@
package spdxhelpers
import (
"crypto/sha256"
"fmt"
"path/filepath"
"github.com/anchore/syft/internal/formats/spdx22json/model"
"github.com/anchore/syft/syft/pkg"
)
func Files(packageSpdxID string, p *pkg.Package) (files []model.File, fileIDs []string, relationships []model.Relationship) {
files = make([]model.File, 0)
fileIDs = make([]string, 0)
relationships = make([]model.Relationship, 0)
if !hasMetadata(p) {
return files, fileIDs, relationships
}
pkgFileOwner, ok := p.Metadata.(pkg.FileOwner)
if !ok {
return files, fileIDs, relationships
}
for _, ownedFilePath := range pkgFileOwner.OwnedFiles() {
baseFileName := filepath.Base(ownedFilePath)
pathHash := sha256.Sum256([]byte(ownedFilePath))
fileSpdxID := model.ElementID(fmt.Sprintf("File-%s-%x", p.Name, pathHash)).String()
fileIDs = append(fileIDs, fileSpdxID)
files = append(files, model.File{
FileName: ownedFilePath,
Item: model.Item{
Element: model.Element{
SPDXID: fileSpdxID,
Name: baseFileName,
},
},
})
relationships = append(relationships, model.Relationship{
SpdxElementID: packageSpdxID,
RelationshipType: model.ContainsRelationship,
RelatedSpdxElement: fileSpdxID,
})
}
return files, fileIDs, relationships
}
@@ -2,7 +2,7 @@ package spdxhelpers
import "github.com/anchore/syft/syft/pkg"
func Homepage(p *pkg.Package) string {
func Homepage(p pkg.Package) string {
if hasMetadata(p) {
switch metadata := p.Metadata.(type) {
case pkg.GemMetadata:
@@ -50,7 +50,7 @@ func Test_Homepage(t *testing.T) {
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
assert.Equal(t, test.expected, Homepage(&test.input))
assert.Equal(t, test.expected, Homepage(test.input))
})
}
}
@@ -7,7 +7,7 @@ import (
"github.com/anchore/syft/syft/pkg"
)
func License(p *pkg.Package) string {
func License(p pkg.Package) string {
// source: https://spdx.github.io/spdx-spec/3-package-information/#313-concluded-license
// The options to populate this field are limited to:
// A valid SPDX License Expression as defined in Appendix IV;
@@ -17,7 +17,7 @@ func License(p *pkg.Package) string {
// (ii) the SPDX file creator has made no attempt to determine this field; or
// (iii) the SPDX file creator has intentionally provided no information (no meaning should be implied by doing so).
if !packageExists(p) || len(p.Licenses) == 0 {
if len(p.Licenses) == 0 {
return "NONE"
}
@@ -67,7 +67,7 @@ func Test_License(t *testing.T) {
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
assert.Equal(t, test.expected, License(&test.input))
assert.Equal(t, test.expected, License(test.input))
})
}
}
@@ -108,7 +108,7 @@ func Test_Originator(t *testing.T) {
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
assert.Equal(t, test.expected, Originator(&test.input))
assert.Equal(t, test.expected, Originator(test.input))
})
}
}
@@ -6,7 +6,7 @@ import (
"github.com/anchore/syft/syft/pkg"
)
func Originator(p *pkg.Package) string {
func Originator(p pkg.Package) string {
if hasMetadata(p) {
switch metadata := p.Metadata.(type) {
case pkg.ApkMetadata:
@@ -6,11 +6,7 @@ import (
"github.com/anchore/syft/syft/pkg"
)
func SourceInfo(p *pkg.Package) string {
if !packageExists(p) {
return ""
}
func SourceInfo(p pkg.Package) string {
answer := ""
switch p.Type {
case pkg.RpmPkg:
@@ -19,14 +19,8 @@ func Test_SourceInfo(t *testing.T) {
input: pkg.Package{
// note: no type given
Locations: []source.Location{
{
RealPath: "/a-place",
VirtualPath: "/b-place",
},
{
RealPath: "/c-place",
VirtualPath: "/d-place",
},
source.NewVirtualLocation("/a-place", "/b-place"),
source.NewVirtualLocation("/c-place", "/d-place"),
},
},
expected: []string{
@@ -139,7 +133,7 @@ func Test_SourceInfo(t *testing.T) {
if test.input.Type != "" {
pkgTypes = append(pkgTypes, test.input.Type)
}
actual := SourceInfo(&test.input)
actual := SourceInfo(test.input)
for _, expected := range test.expected {
assert.Contains(t, actual, expected)
}
+20 -6
View File
@@ -124,6 +124,15 @@ func ImageInput(t testing.TB, testImage string, options ...ImageOption) sbom.SBO
Distro: &dist,
},
Source: src.Metadata,
Descriptor: sbom.Descriptor{
Name: "syft",
Version: "v0.42.0-bogus",
// the application configuration should be persisted here, however, we do not want to import
// the application configuration in this package (it's reserved only for ingestion by the cmd package)
Configuration: map[string]string{
"config-key": "config-value",
},
},
}
}
@@ -133,7 +142,6 @@ func populateImageCatalog(catalog *pkg.Catalog, img *image.Image) {
// populate catalog with test data
catalog.Add(pkg.Package{
ID: "package-1-id",
Name: "package-1",
Version: "1.0.1",
Locations: []source.Location{
@@ -154,7 +162,6 @@ func populateImageCatalog(catalog *pkg.Catalog, img *image.Image) {
},
})
catalog.Add(pkg.Package{
ID: "package-2-id",
Name: "package-2",
Version: "2.0.1",
Locations: []source.Location{
@@ -189,6 +196,15 @@ func DirectoryInput(t testing.TB) sbom.SBOM {
Distro: &dist,
},
Source: src.Metadata,
Descriptor: sbom.Descriptor{
Name: "syft",
Version: "v0.42.0-bogus",
// the application configuration should be persisted here, however, we do not want to import
// the application configuration in this package (it's reserved only for ingestion by the cmd package)
Configuration: map[string]string{
"config-key": "config-value",
},
},
}
}
@@ -197,13 +213,12 @@ func newDirectoryCatalog() *pkg.Catalog {
// populate catalog with test data
catalog.Add(pkg.Package{
ID: "package-1-id",
Name: "package-1",
Version: "1.0.1",
Type: pkg.PythonPkg,
FoundBy: "the-cataloger-1",
Locations: []source.Location{
{RealPath: "/some/path/pkg1"},
source.NewLocation("/some/path/pkg1"),
},
Language: pkg.Python,
MetadataType: pkg.PythonPackageMetadataType,
@@ -223,13 +238,12 @@ func newDirectoryCatalog() *pkg.Catalog {
},
})
catalog.Add(pkg.Package{
ID: "package-2-id",
Name: "package-2",
Version: "2.0.1",
Type: pkg.DebPkg,
FoundBy: "the-cataloger-2",
Locations: []source.Location{
{RealPath: "/some/path/pkg1"},
source.NewLocation("/some/path/pkg1"),
},
MetadataType: pkg.DpkgMetadataType,
Metadata: pkg.DpkgMetadata{
@@ -33,7 +33,7 @@ func toFormatModel(s sbom.SBOM) model.Document {
return doc
}
func toComponent(p *pkg.Package) model.Component {
func toComponent(p pkg.Package) model.Component {
return model.Component{
Type: "library", // TODO: this is not accurate
Name: p.Name,
+12 -12
View File
@@ -3,17 +3,17 @@ package model
type FileType string
const (
DocumentationFileType FileType = "DOCUMENTATION"
ImageFileType FileType = "IMAGE"
VideoFileType FileType = "VIDEO"
ArchiveFileType FileType = "ARCHIVE"
SpdxFileType FileType = "SPDX"
ApplicationFileType FileType = "APPLICATION"
SourceFileType FileType = "SOURCE"
BinaryFileType FileType = "BINARY"
TextFileType FileType = "TEXT"
AudioFileType FileType = "AUDIO"
OtherFileType FileType = "OTHER"
DocumentationFileType FileType = "DOCUMENTATION" // if the file serves as documentation
ImageFileType FileType = "IMAGE" // if the file is associated with a picture image file (MIME type of image/*, e.g., .jpg, .gif)
VideoFileType FileType = "VIDEO" // if the file is associated with a video file type (MIME type of video/*)
ArchiveFileType FileType = "ARCHIVE" // if the file represents an archive (.tar, .jar, etc.)
SpdxFileType FileType = "SPDX" // if the file is an SPDX document
ApplicationFileType FileType = "APPLICATION" // if the file is associated with a specific application type (MIME type of application/*)
SourceFileType FileType = "SOURCE" // if the file is human readable source code (.c, .html, etc.)
BinaryFileType FileType = "BINARY" // if the file is a compiled object, target image or binary executable (.o, .a, etc.)
TextFileType FileType = "TEXT" // if the file is human readable text file (MIME type of text/*)
AudioFileType FileType = "AUDIO" // if the file is associated with an audio file (MIME type of audio/* , e.g. .mp3)
OtherFileType FileType = "OTHER" // if the file doesn't fit into the above categories (generated artifacts, data files, etc.)
)
type File struct {
@@ -36,6 +36,6 @@ type File struct {
// Indicates the project in which the SpdxElement originated. Tools must preserve doap:homepage and doap:name
// properties and the URI (if one is known) of doap:Project resources that are values of this property. All other
// properties of doap:Projects are not directly supported by SPDX and may be dropped when translating to or
// from some SPDX formats(deprecated).
// from some SPDX formats (deprecated).
ArtifactOf []string `json:"artifactOf,omitempty"`
}
@@ -3,18 +3,18 @@
"name": "/some/path",
"spdxVersion": "SPDX-2.2",
"creationInfo": {
"created": "2021-10-29T16:26:08.995826Z",
"created": "2021-11-17T19:35:54.834877Z",
"creators": [
"Organization: Anchore, Inc",
"Tool: syft-[not provided]"
],
"licenseListVersion": "3.14"
"licenseListVersion": "3.15"
},
"dataLicense": "CC0-1.0",
"documentNamespace": "https:/anchore.com/syft/dir/some/path-5362d380-914a-458f-b059-d8d27899574c",
"documentNamespace": "https:/anchore.com/syft/dir/some/path-65e2226e-a61e-4ed1-81bb-56022e1ff1eb",
"packages": [
{
"SPDXID": "SPDXRef-Package-python-package-1-1.0.1",
"SPDXID": "SPDXRef-2a115ac97d018a0e",
"name": "package-1",
"licenseConcluded": "MIT",
"downloadLocation": "NOASSERTION",
@@ -31,15 +31,12 @@
}
],
"filesAnalyzed": false,
"hasFiles": [
"SPDXRef-File-package-1-efae7fecc76ca25da40f79d7ef5b8933510434914835832c7976f3e866aa756a"
],
"licenseDeclared": "MIT",
"sourceInfo": "acquired package info from installed python package manifest file: /some/path/pkg1",
"versionInfo": "1.0.1"
},
{
"SPDXID": "SPDXRef-Package-deb-package-2-2.0.1",
"SPDXID": "SPDXRef-5e920b2bece2c3ae",
"name": "package-2",
"licenseConcluded": "NONE",
"downloadLocation": "NOASSERTION",
@@ -60,20 +57,5 @@
"sourceInfo": "acquired package info from DPKG DB: /some/path/pkg1",
"versionInfo": "2.0.1"
}
],
"files": [
{
"SPDXID": "SPDXRef-File-package-1-efae7fecc76ca25da40f79d7ef5b8933510434914835832c7976f3e866aa756a",
"name": "foo",
"licenseConcluded": "",
"fileName": "/some/path/pkg1/dependencies/foo"
}
],
"relationships": [
{
"spdxElementId": "SPDXRef-Package-python-package-1-1.0.1",
"relationshipType": "CONTAINS",
"relatedSpdxElement": "SPDXRef-File-package-1-efae7fecc76ca25da40f79d7ef5b8933510434914835832c7976f3e866aa756a"
}
]
}
@@ -3,18 +3,18 @@
"name": "user-image-input",
"spdxVersion": "SPDX-2.2",
"creationInfo": {
"created": "2021-10-29T16:26:09.001799Z",
"created": "2021-11-17T19:35:57.761372Z",
"creators": [
"Organization: Anchore, Inc",
"Tool: syft-[not provided]"
],
"licenseListVersion": "3.14"
"licenseListVersion": "3.15"
},
"dataLicense": "CC0-1.0",
"documentNamespace": "https:/anchore.com/syft/image/user-image-input-3ad8571c-513f-4fce-944e-5125353c3186",
"documentNamespace": "https:/anchore.com/syft/image/user-image-input-5383918f-ec96-4aa9-b756-ad16e1ada31e",
"packages": [
{
"SPDXID": "SPDXRef-Package-python-package-1-1.0.1",
"SPDXID": "SPDXRef-888661d4f0362f02",
"name": "package-1",
"licenseConcluded": "MIT",
"downloadLocation": "NOASSERTION",
@@ -36,7 +36,7 @@
"versionInfo": "1.0.1"
},
{
"SPDXID": "SPDXRef-Package-deb-package-2-2.0.1",
"SPDXID": "SPDXRef-4068ff5e8926b305",
"name": "package-2",
"licenseConcluded": "NONE",
"downloadLocation": "NOASSERTION",
+167 -29
View File
@@ -3,17 +3,21 @@ package spdx22json
import (
"fmt"
"path"
"path/filepath"
"sort"
"strings"
"time"
"github.com/anchore/syft/syft/sbom"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/formats/common/spdxhelpers"
"github.com/anchore/syft/internal/formats/spdx22json/model"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/internal/spdxlicense"
"github.com/anchore/syft/internal/version"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/sbom"
"github.com/anchore/syft/syft/source"
"github.com/google/uuid"
)
@@ -21,7 +25,6 @@ import (
// toFormatModel creates and populates a new JSON document struct that follows the SPDX 2.2 spec from the given cataloging results.
func toFormatModel(s sbom.SBOM) model.Document {
name := documentName(s.Source)
packages, files, relationships := extractFromCatalog(s.Artifacts.PackageCatalog)
return model.Document{
Element: model.Element{
@@ -40,9 +43,9 @@ func toFormatModel(s sbom.SBOM) model.Document {
},
DataLicense: "CC0-1.0",
DocumentNamespace: documentNamespace(name, s.Source),
Packages: packages,
Files: files,
Relationships: relationships,
Packages: toPackages(s.Artifacts.PackageCatalog, s.Relationships),
Files: toFiles(s),
Relationships: toRelationships(s.Relationships),
}
}
@@ -58,6 +61,17 @@ func documentName(srcMetadata source.Metadata) string {
return uuid.Must(uuid.NewRandom()).String()
}
func cleanSPDXName(name string) string {
// remove # according to specification
name = strings.ReplaceAll(name, "#", "-")
// remove : for url construction
name = strings.ReplaceAll(name, ":", "-")
// clean relative pathing
return path.Clean(name)
}
func documentNamespace(name string, srcMetadata source.Metadata) string {
input := "unknown-source-type"
switch srcMetadata.Scheme {
@@ -76,19 +90,12 @@ func documentNamespace(name string, srcMetadata source.Metadata) string {
return path.Join(anchoreNamespace, identifier)
}
func extractFromCatalog(catalog *pkg.Catalog) ([]model.Package, []model.File, []model.Relationship) {
func toPackages(catalog *pkg.Catalog, relationships []artifact.Relationship) []model.Package {
packages := make([]model.Package, 0)
relationships := make([]model.Relationship, 0)
files := make([]model.File, 0)
for _, p := range catalog.Sorted() {
license := spdxhelpers.License(p)
packageSpdxID := model.ElementID(fmt.Sprintf("Package-%+v-%s-%s", p.Type, p.Name, p.Version)).String()
packageFiles, fileIDs, packageFileRelationships := spdxhelpers.Files(packageSpdxID, p)
files = append(files, packageFiles...)
relationships = append(relationships, packageFileRelationships...)
packageSpdxID := model.ElementID(p.ID()).String()
// note: the license concluded and declared should be the same since we are collecting license information
// from the project data itself (the installed package files).
@@ -97,14 +104,16 @@ func extractFromCatalog(catalog *pkg.Catalog) ([]model.Package, []model.File, []
DownloadLocation: spdxhelpers.DownloadLocation(p),
ExternalRefs: spdxhelpers.ExternalRefs(p),
FilesAnalyzed: false,
HasFiles: fileIDs,
HasFiles: fileIDsForPackage(packageSpdxID, relationships),
Homepage: spdxhelpers.Homepage(p),
LicenseDeclared: license, // The Declared License is what the authors of a project believe govern the package
Originator: spdxhelpers.Originator(p),
SourceInfo: spdxhelpers.SourceInfo(p),
VersionInfo: p.Version,
// The Declared License is what the authors of a project believe govern the package
LicenseDeclared: license,
Originator: spdxhelpers.Originator(p),
SourceInfo: spdxhelpers.SourceInfo(p),
VersionInfo: p.Version,
Item: model.Item{
LicenseConcluded: license, // The Concluded License field is the license the SPDX file creator believes governs the package
// The Concluded License field is the license the SPDX file creator believes governs the package
LicenseConcluded: license,
Element: model.Element{
SPDXID: packageSpdxID,
Name: p.Name,
@@ -113,16 +122,145 @@ func extractFromCatalog(catalog *pkg.Catalog) ([]model.Package, []model.File, []
})
}
return packages, files, relationships
return packages
}
func cleanSPDXName(name string) string {
// remove # according to specification
name = strings.ReplaceAll(name, "#", "-")
func fileIDsForPackage(packageSpdxID string, relationships []artifact.Relationship) (fileIDs []string) {
for _, relationship := range relationships {
if relationship.Type != artifact.ContainsRelationship {
continue
}
// remove : for url construction
name = strings.ReplaceAll(name, ":", "-")
if _, ok := relationship.From.(pkg.Package); !ok {
continue
}
// clean relative pathing
return path.Clean(name)
if _, ok := relationship.To.(source.Coordinates); !ok {
continue
}
if string(relationship.From.ID()) == packageSpdxID {
fileIDs = append(fileIDs, string(relationship.To.ID()))
}
}
return fileIDs
}
func toFiles(s sbom.SBOM) []model.File {
results := make([]model.File, 0)
artifacts := s.Artifacts
for _, coordinates := range sbom.AllCoordinates(s) {
var metadata *source.FileMetadata
if metadataForLocation, exists := artifacts.FileMetadata[coordinates]; exists {
metadata = &metadataForLocation
}
var digests []file.Digest
if digestsForLocation, exists := artifacts.FileDigests[coordinates]; exists {
digests = digestsForLocation
}
// TODO: add file classifications (?) and content as a snippet
var comment string
if coordinates.FileSystemID != "" {
comment = fmt.Sprintf("layerID: %s", coordinates.FileSystemID)
}
results = append(results, model.File{
Item: model.Item{
Element: model.Element{
SPDXID: string(coordinates.ID()),
Name: filepath.Base(coordinates.RealPath),
Comment: comment,
},
// required, no attempt made to determine license information
LicenseConcluded: "NOASSERTION",
},
Checksums: toFileChecksums(digests),
FileName: coordinates.RealPath,
FileTypes: toFileTypes(metadata),
})
}
// sort by real path then virtual path to ensure the result is stable across multiple runs
sort.SliceStable(results, func(i, j int) bool {
return results[i].FileName < results[j].FileName
})
return results
}
func toFileChecksums(digests []file.Digest) (checksums []model.Checksum) {
for _, digest := range digests {
checksums = append(checksums, model.Checksum{
Algorithm: digest.Algorithm,
ChecksumValue: digest.Value,
})
}
return checksums
}
func toFileTypes(metadata *source.FileMetadata) (ty []string) {
if metadata == nil {
return nil
}
mimeTypePrefix := strings.Split(metadata.MIMEType, "/")[0]
switch mimeTypePrefix {
case "image":
ty = append(ty, string(model.ImageFileType))
case "video":
ty = append(ty, string(model.VideoFileType))
case "application":
ty = append(ty, string(model.ApplicationFileType))
case "text":
ty = append(ty, string(model.TextFileType))
case "audio":
ty = append(ty, string(model.AudioFileType))
}
if internal.IsExecutable(metadata.MIMEType) {
ty = append(ty, string(model.BinaryFileType))
}
if internal.IsArchive(metadata.MIMEType) {
ty = append(ty, string(model.ArchiveFileType))
}
// TODO: add support for source, spdx, and documentation file types
if len(ty) == 0 {
ty = append(ty, string(model.OtherFileType))
}
return ty
}
func toRelationships(relationships []artifact.Relationship) (result []model.Relationship) {
for _, r := range relationships {
exists, relationshipType, comment := lookupRelationship(r.Type)
if !exists {
log.Warnf("unable to convert relationship from SPDX 2.2 JSON, dropping: %+v", r)
continue
}
result = append(result, model.Relationship{
SpdxElementID: string(r.From.ID()),
RelationshipType: relationshipType,
RelatedSpdxElement: string(r.To.ID()),
Comment: comment,
})
}
return result
}
func lookupRelationship(ty artifact.RelationshipType) (bool, model.RelationshipType, string) {
switch ty {
case artifact.ContainsRelationship:
return true, model.ContainsRelationship, ""
case artifact.OwnershipByFileOverlapRelationship:
return true, model.OtherRelationship, fmt.Sprintf("%s: indicates that the parent package claims ownership of a child package since the parent metadata indicates overlap with a location that a cataloger found the child package by", ty)
}
return false, "", ""
}
@@ -0,0 +1,256 @@
package spdx22json
import (
"testing"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/internal/formats/spdx22json/model"
"github.com/anchore/syft/syft/source"
"github.com/stretchr/testify/assert"
)
func Test_toFileTypes(t *testing.T) {
tests := []struct {
name string
metadata source.FileMetadata
expected []string
}{
{
name: "application",
metadata: source.FileMetadata{
MIMEType: "application/vnd.unknown",
},
expected: []string{
string(model.ApplicationFileType),
},
},
{
name: "archive",
metadata: source.FileMetadata{
MIMEType: "application/zip",
},
expected: []string{
string(model.ApplicationFileType),
string(model.ArchiveFileType),
},
},
{
name: "audio",
metadata: source.FileMetadata{
MIMEType: "audio/ogg",
},
expected: []string{
string(model.AudioFileType),
},
},
{
name: "video",
metadata: source.FileMetadata{
MIMEType: "video/3gpp",
},
expected: []string{
string(model.VideoFileType),
},
},
{
name: "text",
metadata: source.FileMetadata{
MIMEType: "text/html",
},
expected: []string{
string(model.TextFileType),
},
},
{
name: "image",
metadata: source.FileMetadata{
MIMEType: "image/png",
},
expected: []string{
string(model.ImageFileType),
},
},
{
name: "binary",
metadata: source.FileMetadata{
MIMEType: "application/x-sharedlib",
},
expected: []string{
string(model.ApplicationFileType),
string(model.BinaryFileType),
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
assert.ElementsMatch(t, test.expected, toFileTypes(&test.metadata))
})
}
}
func Test_lookupRelationship(t *testing.T) {
tests := []struct {
input artifact.RelationshipType
exists bool
ty model.RelationshipType
comment string
}{
{
input: artifact.ContainsRelationship,
exists: true,
ty: model.ContainsRelationship,
},
{
input: artifact.OwnershipByFileOverlapRelationship,
exists: true,
ty: model.OtherRelationship,
comment: "ownership-by-file-overlap: indicates that the parent package claims ownership of a child package since the parent metadata indicates overlap with a location that a cataloger found the child package by",
},
{
input: "made-up",
exists: false,
},
}
for _, test := range tests {
t.Run(string(test.input), func(t *testing.T) {
exists, ty, comment := lookupRelationship(test.input)
assert.Equal(t, exists, test.exists)
assert.Equal(t, ty, test.ty)
assert.Equal(t, comment, test.comment)
})
}
}
func Test_toFileChecksums(t *testing.T) {
tests := []struct {
name string
digests []file.Digest
expected []model.Checksum
}{
{
name: "empty",
},
{
name: "has digests",
digests: []file.Digest{
{
Algorithm: "sha256",
Value: "deadbeefcafe",
},
{
Algorithm: "md5",
Value: "meh",
},
},
expected: []model.Checksum{
{
Algorithm: "sha256",
ChecksumValue: "deadbeefcafe",
},
{
Algorithm: "md5",
ChecksumValue: "meh",
},
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
assert.ElementsMatch(t, test.expected, toFileChecksums(test.digests))
})
}
}
func Test_fileIDsForPackage(t *testing.T) {
p := pkg.Package{
Name: "bogus",
}
c := source.Coordinates{
RealPath: "/path",
FileSystemID: "nowhere",
}
tests := []struct {
name string
id string
relationships []artifact.Relationship
expected []string
}{
{
name: "find file IDs for packages with package-file relationships",
id: string(p.ID()),
relationships: []artifact.Relationship{
{
From: p,
To: c,
Type: artifact.ContainsRelationship,
},
},
expected: []string{
string(c.ID()),
},
},
{
name: "ignore package-to-package",
id: string(p.ID()),
relationships: []artifact.Relationship{
{
From: p,
To: p,
Type: artifact.ContainsRelationship,
},
},
expected: []string{},
},
{
name: "ignore file-to-file",
id: string(p.ID()),
relationships: []artifact.Relationship{
{
From: c,
To: c,
Type: artifact.ContainsRelationship,
},
},
expected: []string{},
},
{
name: "ignore file-to-package",
id: string(p.ID()),
relationships: []artifact.Relationship{
{
From: c,
To: p,
Type: artifact.ContainsRelationship,
},
},
expected: []string{},
},
{
name: "filter by relationship type",
id: string(p.ID()),
relationships: []artifact.Relationship{
{
From: p,
To: c,
Type: artifact.OwnershipByFileOverlapRelationship,
},
},
expected: []string{},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
assert.ElementsMatch(t, test.expected, fileIDsForPackage(test.id, test.relationships))
})
}
}
@@ -256,7 +256,7 @@ func toFormatPackages(catalog *pkg.Catalog) map[spdx.ElementID]*spdx.Package2_2
return results
}
func formatSPDXExternalRefs(p *pkg.Package) (refs []*spdx.PackageExternalReference2_2) {
func formatSPDXExternalRefs(p pkg.Package) (refs []*spdx.PackageExternalReference2_2) {
for _, ref := range spdxhelpers.ExternalRefs(p) {
refs = append(refs, &spdx.PackageExternalReference2_2{
Category: string(ref.ReferenceCategory),
+1 -5
View File
@@ -31,11 +31,7 @@ func TestEncodeDecodeCycle(t *testing.T) {
continue
}
// ids will never be equal
p.ID = ""
actualPackages[idx].ID = ""
for _, d := range deep.Equal(*p, *actualPackages[idx]) {
for _, d := range deep.Equal(p, actualPackages[idx]) {
if strings.Contains(d, ".VirtualPath: ") {
// location.Virtual path is not exposed in the json output
continue
+1 -2
View File
@@ -8,8 +8,7 @@ import (
)
func encoder(output io.Writer, s sbom.SBOM) error {
// TODO: application config not available yet
doc := ToFormatModel(s, nil)
doc := ToFormatModel(s)
enc := json.NewEncoder(output)
// prevent > and < from being escaped in the payload
+173
View File
@@ -4,6 +4,15 @@ import (
"flag"
"testing"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/distro"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/sbom"
"github.com/anchore/syft/syft/source"
"github.com/anchore/syft/internal/formats/common/testutils"
)
@@ -24,3 +33,167 @@ func TestImagePresenter(t *testing.T) {
*updateJson,
)
}
func TestEncodeFullJSONDocument(t *testing.T) {
catalog := pkg.NewCatalog()
p1 := pkg.Package{
Name: "package-1",
Version: "1.0.1",
Locations: []source.Location{
{
Coordinates: source.Coordinates{
RealPath: "/a/place/a",
},
},
},
Type: pkg.PythonPkg,
FoundBy: "the-cataloger-1",
Language: pkg.Python,
MetadataType: pkg.PythonPackageMetadataType,
Licenses: []string{"MIT"},
Metadata: pkg.PythonPackageMetadata{
Name: "package-1",
Version: "1.0.1",
Files: []pkg.PythonFileRecord{},
},
PURL: "a-purl-1",
CPEs: []pkg.CPE{
pkg.MustCPE("cpe:2.3:*:some:package:1:*:*:*:*:*:*:*"),
},
}
p2 := pkg.Package{
Name: "package-2",
Version: "2.0.1",
Locations: []source.Location{
{
Coordinates: source.Coordinates{
RealPath: "/b/place/b",
},
},
},
Type: pkg.DebPkg,
FoundBy: "the-cataloger-2",
MetadataType: pkg.DpkgMetadataType,
Metadata: pkg.DpkgMetadata{
Package: "package-2",
Version: "2.0.1",
Files: []pkg.DpkgFileRecord{},
},
PURL: "a-purl-2",
CPEs: []pkg.CPE{
pkg.MustCPE("cpe:2.3:*:some:package:2:*:*:*:*:*:*:*"),
},
}
catalog.Add(p1)
catalog.Add(p2)
s := sbom.SBOM{
Artifacts: sbom.Artifacts{
PackageCatalog: catalog,
FileMetadata: map[source.Coordinates]source.FileMetadata{
source.NewLocation("/a/place").Coordinates: {
Mode: 0775,
Type: "directory",
UserID: 0,
GroupID: 0,
},
source.NewLocation("/a/place/a").Coordinates: {
Mode: 0775,
Type: "regularFile",
UserID: 0,
GroupID: 0,
},
source.NewLocation("/b").Coordinates: {
Mode: 0775,
Type: "symbolicLink",
LinkDestination: "/c",
UserID: 0,
GroupID: 0,
},
source.NewLocation("/b/place/b").Coordinates: {
Mode: 0644,
Type: "regularFile",
UserID: 1,
GroupID: 2,
},
},
FileDigests: map[source.Coordinates][]file.Digest{
source.NewLocation("/a/place/a").Coordinates: {
{
Algorithm: "sha256",
Value: "366a3f5653e34673b875891b021647440d0127c2ef041e3b1a22da2a7d4f3703",
},
},
source.NewLocation("/b/place/b").Coordinates: {
{
Algorithm: "sha256",
Value: "1b3722da2a7d90d033b87581a2a3f12021647445653e34666ef041e3b4f3707c",
},
},
},
FileContents: map[source.Coordinates]string{
source.NewLocation("/a/place/a").Coordinates: "the-contents",
},
Distro: &distro.Distro{
Type: distro.RedHat,
RawVersion: "7",
IDLike: "rhel",
},
},
Relationships: []artifact.Relationship{
{
From: p1,
To: p2,
Type: artifact.OwnershipByFileOverlapRelationship,
Data: map[string]string{
"file": "path",
},
},
},
Source: source.Metadata{
Scheme: source.ImageScheme,
ImageMetadata: source.ImageMetadata{
UserInput: "user-image-input",
ID: "sha256:c2b46b4eb06296933b7cf0722683964e9ecbd93265b9ef6ae9642e3952afbba0",
ManifestDigest: "sha256:2731251dc34951c0e50fcc643b4c5f74922dad1a5d98f302b504cf46cd5d9368",
MediaType: "application/vnd.docker.distribution.manifest.v2+json",
Tags: []string{
"stereoscope-fixture-image-simple:85066c51088bdd274f7a89e99e00490f666c49e72ffc955707cd6e18f0e22c5b",
},
Size: 38,
Layers: []source.LayerMetadata{
{
MediaType: "application/vnd.docker.image.rootfs.diff.tar.gzip",
Digest: "sha256:3de16c5b8659a2e8d888b8ded8427be7a5686a3c8c4e4dd30de20f362827285b",
Size: 22,
},
{
MediaType: "application/vnd.docker.image.rootfs.diff.tar.gzip",
Digest: "sha256:366a3f5653e34673b875891b021647440d0127c2ef041e3b1a22da2a7d4f3703",
Size: 16,
},
},
RawManifest: []byte("eyJzY2hlbWFWZXJzaW9uIjoyLCJtZWRpYVR5cGUiOiJh..."),
RawConfig: []byte("eyJhcmNoaXRlY3R1cmUiOiJhbWQ2NCIsImNvbmZp..."),
RepoDigests: []string{},
},
},
Descriptor: sbom.Descriptor{
Name: "syft",
Version: "v0.42.0-bogus",
// the application configuration should be persisted here, however, we do not want to import
// the application configuration in this package (it's reserved only for ingestion by the cmd package)
Configuration: map[string]string{
"config-key": "config-value",
},
},
}
testutils.AssertPresenterAgainstGoldenSnapshot(t,
Format().Presenter(s),
*updateJson,
)
}
+6 -4
View File
@@ -4,10 +4,12 @@ package model
type Document struct {
Artifacts []Package `json:"artifacts"` // Artifacts is the list of packages discovered and placed into the catalog
ArtifactRelationships []Relationship `json:"artifactRelationships"`
Source Source `json:"source"` // Source represents the original object that was cataloged
Distro Distro `json:"distro"` // Distro represents the Linux distribution that was detected from the source
Descriptor Descriptor `json:"descriptor"` // Descriptor is a block containing self-describing information about syft
Schema Schema `json:"schema"` // Schema is a block reserved for defining the version for the shape of this JSON document and where to find the schema document to validate the shape
Files []File `json:"files,omitempty"` // note: must have omitempty
Secrets []Secrets `json:"secrets,omitempty"` // note: must have omitempty
Source Source `json:"source"` // Source represents the original object that was cataloged
Distro Distro `json:"distro"` // Distro represents the Linux distribution that was detected from the source
Descriptor Descriptor `json:"descriptor"` // Descriptor is a block containing self-describing information about syft
Schema Schema `json:"schema"` // Schema is a block reserved for defining the version for the shape of this JSON document and where to find the schema document to validate the shape
}
// Descriptor describes what created the document as well as surrounding metadata
+25
View File
@@ -0,0 +1,25 @@
package model
import (
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/source"
)
type File struct {
ID string `json:"id"`
Location source.Coordinates `json:"location"`
Metadata *FileMetadataEntry `json:"metadata,omitempty"`
Contents string `json:"contents,omitempty"`
Digests []file.Digest `json:"digests,omitempty"`
Classifications []file.Classification `json:"classifications,omitempty"`
}
type FileMetadataEntry struct {
Mode int `json:"mode"`
Type source.FileType `json:"type"`
LinkDestination string `json:"linkDestination,omitempty"`
UserID int `json:"userID"`
GroupID int `json:"groupID"`
MIMEType string `json:"mimeType"`
}
+11 -11
View File
@@ -9,7 +9,7 @@ import (
"github.com/anchore/syft/syft/source"
)
// Package represents a pkg.Package object specialized for JSON marshaling and unmarshaling.
// Package represents a pkg.Package object specialized for JSON marshaling and unmarshalling.
type Package struct {
PackageBasicData
PackageCustomData
@@ -17,16 +17,16 @@ type Package struct {
// PackageBasicData contains non-ambiguous values (type-wise) from pkg.Package.
type PackageBasicData struct {
ID string `json:"id"`
Name string `json:"name"`
Version string `json:"version"`
Type pkg.Type `json:"type"`
FoundBy string `json:"foundBy"`
Locations []source.Location `json:"locations"`
Licenses []string `json:"licenses"`
Language pkg.Language `json:"language"`
CPEs []string `json:"cpes"`
PURL string `json:"purl"`
ID string `json:"id"`
Name string `json:"name"`
Version string `json:"version"`
Type pkg.Type `json:"type"`
FoundBy string `json:"foundBy"`
Locations []source.Coordinates `json:"locations"`
Licenses []string `json:"licenses"`
Language pkg.Language `json:"language"`
CPEs []string `json:"cpes"`
PURL string `json:"purl"`
}
// PackageCustomData contains ambiguous values (type-wise) from pkg.Package.
@@ -4,5 +4,5 @@ type Relationship struct {
Parent string `json:"parent"`
Child string `json:"child"`
Type string `json:"type"`
Metadata interface{} `json:"metadata"`
Metadata interface{} `json:"metadata,omitempty"`
}
@@ -0,0 +1,11 @@
package model
import (
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/source"
)
type Secrets struct {
Location source.Coordinates `json:"location"`
Secrets []file.SearchResult `json:"secrets"`
}
@@ -1,7 +1,7 @@
{
"artifacts": [
{
"id": "package-1-id",
"id": "2a115ac97d018a0e",
"name": "package-1",
"version": "1.0.1",
"type": "python",
@@ -36,7 +36,7 @@
}
},
{
"id": "package-2-id",
"id": "5e920b2bece2c3ae",
"name": "package-2",
"version": "2.0.1",
"type": "deb",
@@ -77,10 +77,13 @@
},
"descriptor": {
"name": "syft",
"version": "[not provided]"
"version": "v0.42.0-bogus",
"configuration": {
"config-key": "config-value"
}
},
"schema": {
"version": "1.1.0",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-1.1.0.json"
"version": "2.0.0",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-2.0.0.json"
}
}
@@ -1,78 +1,7 @@
{
"fileContents": [
{
"location": {
"path": "/a/place/a"
},
"contents": "the-contents"
}
],
"fileMetadata": [
{
"location": {
"path": "/a/place"
},
"metadata": {
"mode": 775,
"type": "directory",
"userID": 0,
"groupID": 0,
"mimeType": ""
}
},
{
"location": {
"path": "/a/place/a"
},
"metadata": {
"mode": 775,
"type": "regularFile",
"userID": 0,
"groupID": 0,
"digests": [
{
"algorithm": "sha256",
"value": "366a3f5653e34673b875891b021647440d0127c2ef041e3b1a22da2a7d4f3703"
}
],
"mimeType": ""
}
},
{
"location": {
"path": "/b"
},
"metadata": {
"mode": 775,
"type": "symbolicLink",
"linkDestination": "/c",
"userID": 0,
"groupID": 0,
"mimeType": ""
}
},
{
"location": {
"path": "/b/place/b"
},
"metadata": {
"mode": 644,
"type": "regularFile",
"userID": 1,
"groupID": 2,
"digests": [
{
"algorithm": "sha256",
"value": "1b3722da2a7d90d033b87581a2a3f12021647445653e34666ef041e3b4f3707c"
}
],
"mimeType": ""
}
}
],
"artifacts": [
{
"id": "package-1-id",
"id": "962403cfb7be50d7",
"name": "package-1",
"version": "1.0.1",
"type": "python",
@@ -102,7 +31,7 @@
}
},
{
"id": "package-2-id",
"id": "b11f44847bba0ed1",
"name": "package-2",
"version": "2.0.1",
"type": "deb",
@@ -131,7 +60,84 @@
}
}
],
"artifactRelationships": [],
"artifactRelationships": [
{
"parent": "962403cfb7be50d7",
"child": "b11f44847bba0ed1",
"type": "ownership-by-file-overlap",
"metadata": {
"file": "path"
}
}
],
"files": [
{
"id": "913b4592e2c2ebdf",
"location": {
"path": "/a/place"
},
"metadata": {
"mode": 775,
"type": "directory",
"userID": 0,
"groupID": 0,
"mimeType": ""
}
},
{
"id": "e7c88bd18e11b0b",
"location": {
"path": "/a/place/a"
},
"metadata": {
"mode": 775,
"type": "regularFile",
"userID": 0,
"groupID": 0,
"mimeType": ""
},
"contents": "the-contents",
"digests": [
{
"algorithm": "sha256",
"value": "366a3f5653e34673b875891b021647440d0127c2ef041e3b1a22da2a7d4f3703"
}
]
},
{
"id": "5c3dc6885f48b5a1",
"location": {
"path": "/b"
},
"metadata": {
"mode": 775,
"type": "symbolicLink",
"linkDestination": "/c",
"userID": 0,
"groupID": 0,
"mimeType": ""
}
},
{
"id": "799d2f12da0bcec4",
"location": {
"path": "/b/place/b"
},
"metadata": {
"mode": 644,
"type": "regularFile",
"userID": 1,
"groupID": 2,
"mimeType": ""
},
"digests": [
{
"algorithm": "sha256",
"value": "1b3722da2a7d90d033b87581a2a3f12021647445653e34666ef041e3b4f3707c"
}
]
}
],
"source": {
"type": "image",
"target": {
@@ -167,77 +173,13 @@
},
"descriptor": {
"name": "syft",
"version": "[not provided]",
"version": "v0.42.0-bogus",
"configuration": {
"configPath": "",
"output": "",
"file": "",
"quiet": false,
"check-for-app-update": false,
"anchore": {
"host": "",
"path": "",
"dockerfile": "",
"overwrite-existing-image": false,
"import-timeout": 0
},
"dev": {
"profile-cpu": false,
"profile-mem": false
},
"log": {
"structured": false,
"level": "",
"file-location": ""
},
"package": {
"cataloger": {
"enabled": false,
"scope": ""
}
},
"file-metadata": {
"cataloger": {
"enabled": false,
"scope": ""
},
"digests": [
"sha256"
]
},
"file-classification": {
"cataloger": {
"enabled": false,
"scope": ""
}
},
"file-contents": {
"cataloger": {
"enabled": false,
"scope": ""
},
"skip-files-above-size": 0,
"globs": null
},
"secrets": {
"cataloger": {
"enabled": false,
"scope": ""
},
"additional-patterns": null,
"exclude-pattern-names": null,
"reveal-values": false,
"skip-files-above-size": 0
},
"registry": {
"insecure-skip-tls-verify": false,
"insecure-use-http": false,
"auth": null
}
"config-key": "config-value"
}
},
"schema": {
"version": "1.1.0",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-1.1.0.json"
"version": "2.0.0",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-2.0.0.json"
}
}
@@ -1,7 +1,7 @@
{
"artifacts": [
{
"id": "package-1-id",
"id": "888661d4f0362f02",
"name": "package-1",
"version": "1.0.1",
"type": "python",
@@ -32,7 +32,7 @@
}
},
{
"id": "package-2-id",
"id": "4068ff5e8926b305",
"name": "package-2",
"version": "2.0.1",
"type": "deb",
@@ -98,10 +98,13 @@
},
"descriptor": {
"name": "syft",
"version": "[not provided]"
"version": "v0.42.0-bogus",
"configuration": {
"config-key": "config-value"
}
},
"schema": {
"version": "1.1.0",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-1.1.0.json"
"version": "2.0.0",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-2.0.0.json"
}
}
+111 -22
View File
@@ -2,20 +2,25 @@ package syftjson
import (
"fmt"
"sort"
"strconv"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/sbom"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/formats/syftjson/model"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/internal/version"
"github.com/anchore/syft/syft/distro"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/source"
)
// TODO: this is export4ed for the use of the power-user command (temp)
func ToFormatModel(s sbom.SBOM, applicationConfig interface{}) model.Document {
// TODO: this is exported for the use of the power-user command (temp)
func ToFormatModel(s sbom.SBOM) model.Document {
src, err := toSourceModel(s.Source)
if err != nil {
log.Warnf("unable to create syft-json source object: %+v", err)
@@ -23,14 +28,12 @@ func ToFormatModel(s sbom.SBOM, applicationConfig interface{}) model.Document {
return model.Document{
Artifacts: toPackageModels(s.Artifacts.PackageCatalog),
ArtifactRelationships: toRelationshipModel(pkg.NewRelationships(s.Artifacts.PackageCatalog)),
ArtifactRelationships: toRelationshipModel(s.Relationships),
Files: toFile(s),
Secrets: toSecrets(s.Artifacts.Secrets),
Source: src,
Distro: toDistroModel(s.Artifacts.Distro),
Descriptor: model.Descriptor{
Name: internal.ApplicationName,
Version: version.FromBuild().Version,
Configuration: applicationConfig,
},
Descriptor: toDescriptor(s.Descriptor),
Schema: model.Schema{
Version: internal.JSONSchemaVersion,
URL: fmt.Sprintf("https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-%s.json", internal.JSONSchemaVersion),
@@ -38,6 +41,93 @@ func ToFormatModel(s sbom.SBOM, applicationConfig interface{}) model.Document {
}
}
func toDescriptor(d sbom.Descriptor) model.Descriptor {
return model.Descriptor{
Name: d.Name,
Version: d.Version,
Configuration: d.Configuration,
}
}
func toSecrets(data map[source.Coordinates][]file.SearchResult) []model.Secrets {
results := make([]model.Secrets, 0)
for coordinates, secrets := range data {
results = append(results, model.Secrets{
Location: coordinates,
Secrets: secrets,
})
}
// sort by real path then virtual path to ensure the result is stable across multiple runs
sort.SliceStable(results, func(i, j int) bool {
return results[i].Location.RealPath < results[j].Location.RealPath
})
return results
}
func toFile(s sbom.SBOM) []model.File {
results := make([]model.File, 0)
artifacts := s.Artifacts
for _, coordinates := range sbom.AllCoordinates(s) {
var metadata *source.FileMetadata
if metadataForLocation, exists := artifacts.FileMetadata[coordinates]; exists {
metadata = &metadataForLocation
}
var digests []file.Digest
if digestsForLocation, exists := artifacts.FileDigests[coordinates]; exists {
digests = digestsForLocation
}
var classifications []file.Classification
if classificationsForLocation, exists := artifacts.FileClassifications[coordinates]; exists {
classifications = classificationsForLocation
}
var contents string
if contentsForLocation, exists := artifacts.FileContents[coordinates]; exists {
contents = contentsForLocation
}
results = append(results, model.File{
ID: string(coordinates.ID()),
Location: coordinates,
Metadata: toFileMetadataEntry(coordinates, metadata),
Digests: digests,
Classifications: classifications,
Contents: contents,
})
}
// sort by real path then virtual path to ensure the result is stable across multiple runs
sort.SliceStable(results, func(i, j int) bool {
return results[i].Location.RealPath < results[j].Location.RealPath
})
return results
}
func toFileMetadataEntry(coordinates source.Coordinates, metadata *source.FileMetadata) *model.FileMetadataEntry {
if metadata == nil {
return nil
}
mode, err := strconv.Atoi(fmt.Sprintf("%o", metadata.Mode))
if err != nil {
log.Warnf("invalid mode found in file catalog @ location=%+v mode=%q: %+v", coordinates, metadata.Mode, err)
mode = 0
}
return &model.FileMetadataEntry{
Mode: mode,
Type: metadata.Type,
LinkDestination: metadata.LinkDestination,
UserID: metadata.UserID,
GroupID: metadata.GroupID,
MIMEType: metadata.MIMEType,
}
}
func toPackageModels(catalog *pkg.Catalog) []model.Package {
artifacts := make([]model.Package, 0)
if catalog == nil {
@@ -50,31 +140,30 @@ func toPackageModels(catalog *pkg.Catalog) []model.Package {
}
// toPackageModel crates a new Package from the given pkg.Package.
func toPackageModel(p *pkg.Package) model.Package {
func toPackageModel(p pkg.Package) model.Package {
var cpes = make([]string, len(p.CPEs))
for i, c := range p.CPEs {
cpes[i] = c.BindToFmtString()
}
// ensure collections are never nil for presentation reasons
var locations = make([]source.Location, 0)
if p.Locations != nil {
locations = p.Locations
}
var licenses = make([]string, 0)
if p.Licenses != nil {
licenses = p.Licenses
}
var coordinates = make([]source.Coordinates, len(p.Locations))
for i, l := range p.Locations {
coordinates[i] = l.Coordinates
}
return model.Package{
PackageBasicData: model.PackageBasicData{
ID: string(p.ID),
ID: string(p.ID()),
Name: p.Name,
Version: p.Version,
Type: p.Type,
FoundBy: p.FoundBy,
Locations: locations,
Locations: coordinates,
Licenses: licenses,
Language: p.Language,
CPEs: cpes,
@@ -87,14 +176,14 @@ func toPackageModel(p *pkg.Package) model.Package {
}
}
func toRelationshipModel(relationships []pkg.Relationship) []model.Relationship {
func toRelationshipModel(relationships []artifact.Relationship) []model.Relationship {
result := make([]model.Relationship, len(relationships))
for i, r := range relationships {
result[i] = model.Relationship{
Parent: string(r.Parent),
Child: string(r.Child),
Parent: string(r.From.ID()),
Child: string(r.To.ID()),
Type: string(r.Type),
Metadata: r.Metadata,
Metadata: r.Data,
}
}
return result
+16 -3
View File
@@ -20,10 +20,19 @@ func toSyftModel(doc model.Document) (*sbom.SBOM, error) {
PackageCatalog: toSyftCatalog(doc.Artifacts),
Distro: &dist,
},
Source: *toSyftSourceData(doc.Source),
Source: *toSyftSourceData(doc.Source),
Descriptor: toSyftDescriptor(doc.Descriptor),
}, nil
}
func toSyftDescriptor(d model.Descriptor) sbom.Descriptor {
return sbom.Descriptor{
Name: d.Name,
Version: d.Version,
Configuration: d.Configuration,
}
}
func toSyftSourceData(s model.Source) *source.Metadata {
switch s.Type {
case "directory":
@@ -60,12 +69,16 @@ func toSyftPackage(p model.Package) pkg.Package {
cpes = append(cpes, value)
}
var locations = make([]source.Location, len(p.Locations))
for i, c := range p.Locations {
locations[i] = source.NewLocationFromCoordinates(c)
}
return pkg.Package{
ID: pkg.ID(p.ID),
Name: p.Name,
Version: p.Version,
FoundBy: p.FoundBy,
Locations: p.Locations,
Locations: locations,
Licenses: p.Licenses,
Language: p.Language,
Type: p.Type,
+72
View File
@@ -0,0 +1,72 @@
package internal
import "github.com/scylladb/go-set/strset"
var (
ArchiveMIMETypeSet = strset.New(
// derived from https://en.wikipedia.org/wiki/List_of_archive_formats
[]string{
// archive only
"application/x-archive",
"application/x-cpio",
"application/x-shar",
"application/x-iso9660-image",
"application/x-sbx",
"application/x-tar",
// compression only
"application/x-bzip2",
"application/gzip",
"application/x-lzip",
"application/x-lzma",
"application/x-lzop",
"application/x-snappy-framed",
"application/x-xz",
"application/x-compress",
"application/zstd",
// archiving and compression
"application/x-7z-compressed",
"application/x-ace-compressed",
"application/x-astrotite-afa",
"application/x-alz-compressed",
"application/vnd.android.package-archive",
"application/x-freearc",
"application/x-arj",
"application/x-b1",
"application/vnd.ms-cab-compressed",
"application/x-cfs-compressed",
"application/x-dar",
"application/x-dgc-compressed",
"application/x-apple-diskimage",
"application/x-gca-compressed",
"application/java-archive",
"application/x-lzh",
"application/x-lzx",
"application/x-rar-compressed",
"application/x-stuffit",
"application/x-stuffitx",
"application/x-gtar",
"application/x-ms-wim",
"application/x-xar",
"application/zip",
"application/x-zoo",
}...,
)
ExecutableMIMETypeSet = strset.New(
[]string{
"application/x-executable",
"application/x-mach-binary",
"application/x-elf",
"application/x-sharedlib",
"application/vnd.microsoft.portable-executable",
}...,
)
)
func IsArchive(mimeType string) bool {
return ArchiveMIMETypeSet.Has(mimeType)
}
func IsExecutable(mimeType string) bool {
return ExecutableMIMETypeSet.Has(mimeType)
}
+57
View File
@@ -0,0 +1,57 @@
package internal
import (
"testing"
"github.com/stretchr/testify/assert"
)
func Test_IsArchive(t *testing.T) {
tests := []struct {
name string
mimeType string
expected bool
}{
{
name: "not an archive",
mimeType: "application/vnd.unknown",
expected: false,
},
{
name: "archive",
mimeType: "application/x-rar-compressed",
expected: true,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
assert.Equal(t, test.expected, IsArchive(test.mimeType))
})
}
}
func Test_IsExecutable(t *testing.T) {
tests := []struct {
name string
mimeType string
expected bool
}{
{
name: "not an executable",
mimeType: "application/vnd.unknown",
expected: false,
},
{
name: "executable",
mimeType: "application/x-mach-binary",
expected: true,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
assert.Equal(t, test.expected, IsExecutable(test.mimeType))
})
}
}
@@ -1,35 +0,0 @@
package poweruser
import (
"github.com/anchore/syft/internal/formats/syftjson"
"github.com/anchore/syft/internal/formats/syftjson/model"
"github.com/anchore/syft/syft/sbom"
)
type JSONDocument struct {
// note: poweruser.JSONDocument is meant to always be a superset of packages.JSONDocument, any additional fields
// here should be optional by supplying "omitempty" on these fields hint to the jsonschema generator to not
// require these fields. As an accepted rule in this repo all collections should still be initialized in the
// context of being used in a JSON document.
FileClassifications []JSONFileClassifications `json:"fileClassifications,omitempty"` // note: must have omitempty
FileContents []JSONFileContents `json:"fileContents,omitempty"` // note: must have omitempty
FileMetadata []JSONFileMetadata `json:"fileMetadata,omitempty"` // note: must have omitempty
Secrets []JSONSecrets `json:"secrets,omitempty"` // note: must have omitempty
model.Document
}
// NewJSONDocument creates and populates a new JSON document struct from the given cataloging results.
func NewJSONDocument(s sbom.SBOM, appConfig interface{}) (JSONDocument, error) {
fileMetadata, err := NewJSONFileMetadata(s.Artifacts.FileMetadata, s.Artifacts.FileDigests)
if err != nil {
return JSONDocument{}, err
}
return JSONDocument{
FileClassifications: NewJSONFileClassifications(s.Artifacts.FileClassifications),
FileContents: NewJSONFileContents(s.Artifacts.FileContents),
FileMetadata: fileMetadata,
Secrets: NewJSONSecrets(s.Artifacts.Secrets),
Document: syftjson.ToFormatModel(s, appConfig),
}, nil
}
@@ -1,34 +0,0 @@
package poweruser
import (
"sort"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/source"
)
type JSONFileClassifications struct {
Location source.Location `json:"location"`
Classification file.Classification `json:"classification"`
}
func NewJSONFileClassifications(data map[source.Location][]file.Classification) []JSONFileClassifications {
results := make([]JSONFileClassifications, 0)
for location, classifications := range data {
for _, classification := range classifications {
results = append(results, JSONFileClassifications{
Location: location,
Classification: classification,
})
}
}
// sort by real path then virtual path to ensure the result is stable across multiple runs
sort.SliceStable(results, func(i, j int) bool {
if results[i].Location.RealPath == results[j].Location.RealPath {
return results[i].Location.VirtualPath < results[j].Location.VirtualPath
}
return results[i].Location.RealPath < results[j].Location.RealPath
})
return results
}
@@ -1,31 +0,0 @@
package poweruser
import (
"sort"
"github.com/anchore/syft/syft/source"
)
type JSONFileContents struct {
Location source.Location `json:"location"`
Contents string `json:"contents"`
}
func NewJSONFileContents(data map[source.Location]string) []JSONFileContents {
results := make([]JSONFileContents, 0)
for location, contents := range data {
results = append(results, JSONFileContents{
Location: location,
Contents: contents,
})
}
// sort by real path then virtual path to ensure the result is stable across multiple runs
sort.SliceStable(results, func(i, j int) bool {
if results[i].Location.RealPath == results[j].Location.RealPath {
return results[i].Location.VirtualPath < results[j].Location.VirtualPath
}
return results[i].Location.RealPath < results[j].Location.RealPath
})
return results
}
@@ -1,63 +0,0 @@
package poweruser
import (
"fmt"
"sort"
"strconv"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/source"
)
type JSONFileMetadata struct {
Location source.Location `json:"location"`
Metadata JSONFileMetadataEntry `json:"metadata"`
}
type JSONFileMetadataEntry struct {
Mode int `json:"mode"`
Type source.FileType `json:"type"`
LinkDestination string `json:"linkDestination,omitempty"`
UserID int `json:"userID"`
GroupID int `json:"groupID"`
Digests []file.Digest `json:"digests,omitempty"`
MIMEType string `json:"mimeType"`
}
func NewJSONFileMetadata(data map[source.Location]source.FileMetadata, digests map[source.Location][]file.Digest) ([]JSONFileMetadata, error) {
results := make([]JSONFileMetadata, 0)
for location, metadata := range data {
mode, err := strconv.Atoi(fmt.Sprintf("%o", metadata.Mode))
if err != nil {
return nil, fmt.Errorf("invalid mode found in file catalog @ location=%+v mode=%q: %w", location, metadata.Mode, err)
}
var digestResults []file.Digest
if digestsForLocation, exists := digests[location]; exists {
digestResults = digestsForLocation
}
results = append(results, JSONFileMetadata{
Location: location,
Metadata: JSONFileMetadataEntry{
Mode: mode,
Type: metadata.Type,
LinkDestination: metadata.LinkDestination,
UserID: metadata.UserID,
GroupID: metadata.GroupID,
Digests: digestResults,
MIMEType: metadata.MIMEType,
},
})
}
// sort by real path then virtual path to ensure the result is stable across multiple runs
sort.SliceStable(results, func(i, j int) bool {
if results[i].Location.RealPath == results[j].Location.RealPath {
return results[i].Location.VirtualPath < results[j].Location.VirtualPath
}
return results[i].Location.RealPath < results[j].Location.RealPath
})
return results, nil
}
@@ -1,36 +0,0 @@
package poweruser
import (
"encoding/json"
"io"
"github.com/anchore/syft/syft/sbom"
)
// JSONPresenter is a JSON presentation object for the syft results
type JSONPresenter struct {
sbom sbom.SBOM
config interface{}
}
// NewJSONPresenter creates a new JSON presenter object for the given cataloging results.
func NewJSONPresenter(s sbom.SBOM, appConfig interface{}) *JSONPresenter {
return &JSONPresenter{
sbom: s,
config: appConfig,
}
}
// Present the PackageCatalog results to the given writer.
func (p *JSONPresenter) Present(output io.Writer) error {
doc, err := NewJSONDocument(p.sbom, p.config)
if err != nil {
return err
}
enc := json.NewEncoder(output)
// prevent > and < from being escaped in the payload
enc.SetEscapeHTML(false)
enc.SetIndent("", " ")
return enc.Encode(&doc)
}
@@ -1,187 +0,0 @@
package poweruser
import (
"bytes"
"flag"
"testing"
"github.com/anchore/syft/syft/sbom"
"github.com/sergi/go-diff/diffmatchpatch"
"github.com/anchore/syft/syft/file"
"github.com/anchore/go-testutils"
"github.com/anchore/syft/internal/config"
"github.com/anchore/syft/syft/distro"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/source"
)
var updateJSONGoldenFiles = flag.Bool("update-json", false, "update the *.golden files for json presenters")
func must(c pkg.CPE, e error) pkg.CPE {
if e != nil {
panic(e)
}
return c
}
func TestJSONPresenter(t *testing.T) {
var buffer bytes.Buffer
catalog := pkg.NewCatalog()
catalog.Add(pkg.Package{
ID: "package-1-id",
Name: "package-1",
Version: "1.0.1",
Locations: []source.Location{
{
RealPath: "/a/place/a",
},
},
Type: pkg.PythonPkg,
FoundBy: "the-cataloger-1",
Language: pkg.Python,
MetadataType: pkg.PythonPackageMetadataType,
Licenses: []string{"MIT"},
Metadata: pkg.PythonPackageMetadata{
Name: "package-1",
Version: "1.0.1",
Files: []pkg.PythonFileRecord{},
},
PURL: "a-purl-1",
CPEs: []pkg.CPE{
must(pkg.NewCPE("cpe:2.3:*:some:package:1:*:*:*:*:*:*:*")),
},
})
catalog.Add(pkg.Package{
ID: "package-2-id",
Name: "package-2",
Version: "2.0.1",
Locations: []source.Location{
{
RealPath: "/b/place/b",
},
},
Type: pkg.DebPkg,
FoundBy: "the-cataloger-2",
MetadataType: pkg.DpkgMetadataType,
Metadata: pkg.DpkgMetadata{
Package: "package-2",
Version: "2.0.1",
Files: []pkg.DpkgFileRecord{},
},
PURL: "a-purl-2",
CPEs: []pkg.CPE{
must(pkg.NewCPE("cpe:2.3:*:some:package:2:*:*:*:*:*:*:*")),
},
})
appConfig := config.Application{
FileMetadata: config.FileMetadata{
Digests: []string{"sha256"},
},
}
cfg := sbom.SBOM{
Artifacts: sbom.Artifacts{
PackageCatalog: catalog,
FileMetadata: map[source.Location]source.FileMetadata{
source.NewLocation("/a/place"): {
Mode: 0775,
Type: "directory",
UserID: 0,
GroupID: 0,
},
source.NewLocation("/a/place/a"): {
Mode: 0775,
Type: "regularFile",
UserID: 0,
GroupID: 0,
},
source.NewLocation("/b"): {
Mode: 0775,
Type: "symbolicLink",
LinkDestination: "/c",
UserID: 0,
GroupID: 0,
},
source.NewLocation("/b/place/b"): {
Mode: 0644,
Type: "regularFile",
UserID: 1,
GroupID: 2,
},
},
FileDigests: map[source.Location][]file.Digest{
source.NewLocation("/a/place/a"): {
{
Algorithm: "sha256",
Value: "366a3f5653e34673b875891b021647440d0127c2ef041e3b1a22da2a7d4f3703",
},
},
source.NewLocation("/b/place/b"): {
{
Algorithm: "sha256",
Value: "1b3722da2a7d90d033b87581a2a3f12021647445653e34666ef041e3b4f3707c",
},
},
},
FileContents: map[source.Location]string{
source.NewLocation("/a/place/a"): "the-contents",
},
Distro: &distro.Distro{
Type: distro.RedHat,
RawVersion: "7",
IDLike: "rhel",
},
},
Source: source.Metadata{
Scheme: source.ImageScheme,
ImageMetadata: source.ImageMetadata{
UserInput: "user-image-input",
ID: "sha256:c2b46b4eb06296933b7cf0722683964e9ecbd93265b9ef6ae9642e3952afbba0",
ManifestDigest: "sha256:2731251dc34951c0e50fcc643b4c5f74922dad1a5d98f302b504cf46cd5d9368",
MediaType: "application/vnd.docker.distribution.manifest.v2+json",
Tags: []string{
"stereoscope-fixture-image-simple:85066c51088bdd274f7a89e99e00490f666c49e72ffc955707cd6e18f0e22c5b",
},
Size: 38,
Layers: []source.LayerMetadata{
{
MediaType: "application/vnd.docker.image.rootfs.diff.tar.gzip",
Digest: "sha256:3de16c5b8659a2e8d888b8ded8427be7a5686a3c8c4e4dd30de20f362827285b",
Size: 22,
},
{
MediaType: "application/vnd.docker.image.rootfs.diff.tar.gzip",
Digest: "sha256:366a3f5653e34673b875891b021647440d0127c2ef041e3b1a22da2a7d4f3703",
Size: 16,
},
},
RawManifest: []byte("eyJzY2hlbWFWZXJzaW9uIjoyLCJtZWRpYVR5cGUiOiJh..."),
RawConfig: []byte("eyJhcmNoaXRlY3R1cmUiOiJhbWQ2NCIsImNvbmZp..."),
RepoDigests: []string{},
},
},
}
if err := NewJSONPresenter(cfg, appConfig).Present(&buffer); err != nil {
t.Fatal(err)
}
actual := buffer.Bytes()
if *updateJSONGoldenFiles {
testutils.UpdateGoldenFileContents(t, actual)
}
expected := testutils.GetGoldenFileContents(t)
if !bytes.Equal(expected, actual) {
dmp := diffmatchpatch.New()
diffs := dmp.DiffMain(string(expected), string(actual), true)
t.Errorf("mismatched output:\n%s", dmp.DiffPrettyText(diffs))
}
}
@@ -1,32 +0,0 @@
package poweruser
import (
"sort"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/source"
)
type JSONSecrets struct {
Location source.Location `json:"location"`
Secrets []file.SearchResult `json:"secrets"`
}
func NewJSONSecrets(data map[source.Location][]file.SearchResult) []JSONSecrets {
results := make([]JSONSecrets, 0)
for location, secrets := range data {
results = append(results, JSONSecrets{
Location: location,
Secrets: secrets,
})
}
// sort by real path then virtual path to ensure the result is stable across multiple runs
sort.SliceStable(results, func(i, j int) bool {
if results[i].Location.RealPath != results[j].Location.RealPath {
return results[i].Location.VirtualPath < results[j].Location.VirtualPath
}
return false
})
return results
}
-2
View File
@@ -11,8 +11,6 @@ import (
"golang.org/x/term"
)
// TODO: build tags to exclude options from windows
// Select is responsible for determining the specific UI function given select user option, the current platform
// config values, and environment status (such as a TTY being present). The first UI in the returned slice of UIs
// is intended to be used and the UIs that follow are meant to be attempted only in a fallback posture when there
+2 -2
View File
@@ -12,7 +12,7 @@ import (
"github.com/alecthomas/jsonschema"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/presenter/poweruser"
syftjsonModel "github.com/anchore/syft/internal/formats/syftjson/model"
"github.com/anchore/syft/syft/pkg"
)
@@ -48,7 +48,7 @@ func build() *jsonschema.Schema {
return strings.TrimPrefix(r.Name(), "JSON")
},
}
documentSchema := reflector.ReflectFromType(reflect.TypeOf(&poweruser.JSONDocument{}))
documentSchema := reflector.ReflectFromType(reflect.TypeOf(&syftjsonModel.Document{}))
metadataSchema := reflector.ReflectFromType(reflect.TypeOf(&artifactMetadataContainer{}))
// TODO: inject source definitions
File diff suppressed because it is too large Load Diff
+26
View File
@@ -0,0 +1,26 @@
package artifact
import (
"fmt"
"github.com/mitchellh/hashstructure/v2"
)
// ID represents a unique value for each package added to a package catalog.
type ID string
type Identifiable interface {
ID() ID
}
func IDFromHash(obj interface{}) (ID, error) {
f, err := hashstructure.Hash(obj, hashstructure.FormatV2, &hashstructure.HashOptions{
ZeroNil: true,
SlicesAsSets: true,
})
if err != nil {
return "", fmt.Errorf("could not build ID for object=%+v: %+v", obj, err)
}
return ID(fmt.Sprintf("%x", f)), nil
}
+21
View File
@@ -0,0 +1,21 @@
package artifact
const (
// OwnershipByFileOverlapRelationship (supports package-to-package linkages) indicates that the parent package
// claims ownership of a child package since the parent metadata indicates overlap with a location that a
// cataloger found the child package by. This relationship must be created only after all package cataloging
// has been completed.
OwnershipByFileOverlapRelationship RelationshipType = "ownership-by-file-overlap"
// ContainsRelationship (supports any-to-any linkages) is a proxy for the SPDX 2.2 CONTAINS relationship.
ContainsRelationship RelationshipType = "contains"
)
type RelationshipType string
type Relationship struct {
From Identifiable
To Identifiable
Type RelationshipType
Data interface{}
}
+1 -1
View File
@@ -60,7 +60,7 @@ identifyLoop:
}
if len(locations) == 0 {
log.Debugf("No Refs found from path: %s", entry.path)
log.Debugf("path not found: %s", entry.path)
continue
}
+3 -3
View File
@@ -15,8 +15,8 @@ func NewClassificationCataloger(classifiers []Classifier) (*ClassificationCatalo
}, nil
}
func (i *ClassificationCataloger) Catalog(resolver source.FileResolver) (map[source.Location][]Classification, error) {
results := make(map[source.Location][]Classification)
func (i *ClassificationCataloger) Catalog(resolver source.FileResolver) (map[source.Coordinates][]Classification, error) {
results := make(map[source.Coordinates][]Classification)
numResults := 0
for location := range resolver.AllLocations() {
@@ -26,7 +26,7 @@ func (i *ClassificationCataloger) Catalog(resolver source.FileResolver) (map[sou
return nil, err
}
if result != nil {
results[location] = append(results[location], *result)
results[location.Coordinates] = append(results[location.Coordinates], *result)
numResults++
}
}
+9 -3
View File
@@ -19,7 +19,9 @@ func TestFilepathMatches(t *testing.T) {
{
name: "simple-filename-match",
location: source.Location{
RealPath: "python2.7",
Coordinates: source.Coordinates{
RealPath: "python2.7",
},
},
patterns: []string{
`python([0-9]+\.[0-9]+)$`,
@@ -29,7 +31,9 @@ func TestFilepathMatches(t *testing.T) {
{
name: "filepath-match",
location: source.Location{
RealPath: "/usr/bin/python2.7",
Coordinates: source.Coordinates{
RealPath: "/usr/bin/python2.7",
},
},
patterns: []string{
`python([0-9]+\.[0-9]+)$`,
@@ -59,7 +63,9 @@ func TestFilepathMatches(t *testing.T) {
{
name: "anchored-filename-match-FAILS",
location: source.Location{
RealPath: "/usr/bin/python2.7",
Coordinates: source.Coordinates{
RealPath: "/usr/bin/python2.7",
},
},
patterns: []string{
`^python([0-9]+\.[0-9]+)$`,
+3 -3
View File
@@ -23,8 +23,8 @@ func NewContentsCataloger(globs []string, skipFilesAboveSize int64) (*ContentsCa
}, nil
}
func (i *ContentsCataloger) Catalog(resolver source.FileResolver) (map[source.Location]string, error) {
results := make(map[source.Location]string)
func (i *ContentsCataloger) Catalog(resolver source.FileResolver) (map[source.Coordinates]string, error) {
results := make(map[source.Coordinates]string)
var locations []source.Location
locations, err := resolver.FilesByGlob(i.globs...)
@@ -49,7 +49,7 @@ func (i *ContentsCataloger) Catalog(resolver source.FileResolver) (map[source.Lo
if err != nil {
return nil, err
}
results[location] = result
results[location.Coordinates] = result
}
log.Debugf("file contents cataloger processed %d files", len(results))
+16 -16
View File
@@ -15,41 +15,41 @@ func TestContentsCataloger(t *testing.T) {
globs []string
maxSize int64
files []string
expected map[source.Location]string
expected map[source.Coordinates]string
}{
{
name: "multi-pattern",
globs: []string{"test-fixtures/last/*.txt", "test-fixtures/*.txt"},
files: allFiles,
expected: map[source.Location]string{
source.NewLocation("test-fixtures/last/path.txt"): "dGVzdC1maXh0dXJlcy9sYXN0L3BhdGgudHh0IGZpbGUgY29udGVudHMh",
source.NewLocation("test-fixtures/another-path.txt"): "dGVzdC1maXh0dXJlcy9hbm90aGVyLXBhdGgudHh0IGZpbGUgY29udGVudHMh",
source.NewLocation("test-fixtures/a-path.txt"): "dGVzdC1maXh0dXJlcy9hLXBhdGgudHh0IGZpbGUgY29udGVudHMh",
expected: map[source.Coordinates]string{
source.NewLocation("test-fixtures/last/path.txt").Coordinates: "dGVzdC1maXh0dXJlcy9sYXN0L3BhdGgudHh0IGZpbGUgY29udGVudHMh",
source.NewLocation("test-fixtures/another-path.txt").Coordinates: "dGVzdC1maXh0dXJlcy9hbm90aGVyLXBhdGgudHh0IGZpbGUgY29udGVudHMh",
source.NewLocation("test-fixtures/a-path.txt").Coordinates: "dGVzdC1maXh0dXJlcy9hLXBhdGgudHh0IGZpbGUgY29udGVudHMh",
},
},
{
name: "no-patterns",
globs: []string{},
files: []string{"test-fixtures/last/path.txt", "test-fixtures/another-path.txt", "test-fixtures/a-path.txt"},
expected: map[source.Location]string{},
expected: map[source.Coordinates]string{},
},
{
name: "all-txt",
globs: []string{"**/*.txt"},
files: allFiles,
expected: map[source.Location]string{
source.NewLocation("test-fixtures/last/path.txt"): "dGVzdC1maXh0dXJlcy9sYXN0L3BhdGgudHh0IGZpbGUgY29udGVudHMh",
source.NewLocation("test-fixtures/another-path.txt"): "dGVzdC1maXh0dXJlcy9hbm90aGVyLXBhdGgudHh0IGZpbGUgY29udGVudHMh",
source.NewLocation("test-fixtures/a-path.txt"): "dGVzdC1maXh0dXJlcy9hLXBhdGgudHh0IGZpbGUgY29udGVudHMh",
expected: map[source.Coordinates]string{
source.NewLocation("test-fixtures/last/path.txt").Coordinates: "dGVzdC1maXh0dXJlcy9sYXN0L3BhdGgudHh0IGZpbGUgY29udGVudHMh",
source.NewLocation("test-fixtures/another-path.txt").Coordinates: "dGVzdC1maXh0dXJlcy9hbm90aGVyLXBhdGgudHh0IGZpbGUgY29udGVudHMh",
source.NewLocation("test-fixtures/a-path.txt").Coordinates: "dGVzdC1maXh0dXJlcy9hLXBhdGgudHh0IGZpbGUgY29udGVudHMh",
},
},
{
name: "subpath",
globs: []string{"test-fixtures/*.txt"},
files: allFiles,
expected: map[source.Location]string{
source.NewLocation("test-fixtures/another-path.txt"): "dGVzdC1maXh0dXJlcy9hbm90aGVyLXBhdGgudHh0IGZpbGUgY29udGVudHMh",
source.NewLocation("test-fixtures/a-path.txt"): "dGVzdC1maXh0dXJlcy9hLXBhdGgudHh0IGZpbGUgY29udGVudHMh",
expected: map[source.Coordinates]string{
source.NewLocation("test-fixtures/another-path.txt").Coordinates: "dGVzdC1maXh0dXJlcy9hbm90aGVyLXBhdGgudHh0IGZpbGUgY29udGVudHMh",
source.NewLocation("test-fixtures/a-path.txt").Coordinates: "dGVzdC1maXh0dXJlcy9hLXBhdGgudHh0IGZpbGUgY29udGVudHMh",
},
},
{
@@ -57,9 +57,9 @@ func TestContentsCataloger(t *testing.T) {
maxSize: 42,
globs: []string{"**/*.txt"},
files: allFiles,
expected: map[source.Location]string{
source.NewLocation("test-fixtures/last/path.txt"): "dGVzdC1maXh0dXJlcy9sYXN0L3BhdGgudHh0IGZpbGUgY29udGVudHMh",
source.NewLocation("test-fixtures/a-path.txt"): "dGVzdC1maXh0dXJlcy9hLXBhdGgudHh0IGZpbGUgY29udGVudHMh",
expected: map[source.Coordinates]string{
source.NewLocation("test-fixtures/last/path.txt").Coordinates: "dGVzdC1maXh0dXJlcy9sYXN0L3BhdGgudHh0IGZpbGUgY29udGVudHMh",
source.NewLocation("test-fixtures/a-path.txt").Coordinates: "dGVzdC1maXh0dXJlcy9hLXBhdGgudHh0IGZpbGUgY29udGVudHMh",
},
},
}
+3 -3
View File
@@ -29,8 +29,8 @@ func NewDigestsCataloger(hashes []crypto.Hash) (*DigestsCataloger, error) {
}, nil
}
func (i *DigestsCataloger) Catalog(resolver source.FileResolver) (map[source.Location][]Digest, error) {
results := make(map[source.Location][]Digest)
func (i *DigestsCataloger) Catalog(resolver source.FileResolver) (map[source.Coordinates][]Digest, error) {
results := make(map[source.Coordinates][]Digest)
var locations []source.Location
for location := range resolver.AllLocations() {
locations = append(locations, location)
@@ -48,7 +48,7 @@ func (i *DigestsCataloger) Catalog(resolver source.FileResolver) (map[source.Loc
return nil, err
}
prog.N++
results[location] = result
results[location.Coordinates] = result
}
log.Debugf("file digests cataloger processed %d files", prog.N)
prog.SetCompleted()
+6 -6
View File
@@ -16,8 +16,8 @@ import (
"github.com/anchore/syft/syft/source"
)
func testDigests(t testing.TB, files []string, hashes ...crypto.Hash) map[source.Location][]Digest {
digests := make(map[source.Location][]Digest)
func testDigests(t testing.TB, files []string, hashes ...crypto.Hash) map[source.Coordinates][]Digest {
digests := make(map[source.Coordinates][]Digest)
for _, f := range files {
fh, err := os.Open(f)
@@ -32,7 +32,7 @@ func testDigests(t testing.TB, files []string, hashes ...crypto.Hash) map[source
for _, hash := range hashes {
h := hash.New()
h.Write(b)
digests[source.NewLocation(f)] = append(digests[source.NewLocation(f)], Digest{
digests[source.NewLocation(f).Coordinates] = append(digests[source.NewLocation(f).Coordinates], Digest{
Algorithm: CleanDigestAlgorithmName(hash.String()),
Value: fmt.Sprintf("%x", h.Sum(nil)),
})
@@ -49,7 +49,7 @@ func TestDigestsCataloger_SimpleContents(t *testing.T) {
name string
digests []crypto.Hash
files []string
expected map[source.Location][]Digest
expected map[source.Coordinates][]Digest
catalogErr bool
}{
{
@@ -160,13 +160,13 @@ func TestDigestsCataloger_MixFileTypes(t *testing.T) {
}
l := source.NewLocationFromImage(test.path, *ref, img)
if len(actual[l]) == 0 {
if len(actual[l.Coordinates]) == 0 {
if test.expected != "" {
t.Fatalf("no digest found, but expected one")
}
} else {
assert.Equal(t, actual[l][0].Value, test.expected, "mismatched digests")
assert.Equal(t, actual[l.Coordinates][0].Value, test.expected, "mismatched digests")
}
})
}
+3 -3
View File
@@ -16,8 +16,8 @@ func NewMetadataCataloger() *MetadataCataloger {
return &MetadataCataloger{}
}
func (i *MetadataCataloger) Catalog(resolver source.FileResolver) (map[source.Location]source.FileMetadata, error) {
results := make(map[source.Location]source.FileMetadata)
func (i *MetadataCataloger) Catalog(resolver source.FileResolver) (map[source.Coordinates]source.FileMetadata, error) {
results := make(map[source.Coordinates]source.FileMetadata)
var locations []source.Location
for location := range resolver.AllLocations() {
locations = append(locations, location)
@@ -30,7 +30,7 @@ func (i *MetadataCataloger) Catalog(resolver source.FileResolver) (map[source.Lo
return nil, err
}
results[location] = metadata
results[location.Coordinates] = metadata
prog.N++
}
log.Debugf("file metadata cataloger processed %d files", prog.N)
+1 -1
View File
@@ -136,7 +136,7 @@ func TestFileMetadataCataloger(t *testing.T) {
l := source.NewLocationFromImage(test.path, *ref, img)
assert.Equal(t, test.expected, actual[l], "mismatched metadata")
assert.Equal(t, test.expected, actual[l.Coordinates], "mismatched metadata")
})
}
+3 -3
View File
@@ -40,8 +40,8 @@ func NewSecretsCataloger(patterns map[string]*regexp.Regexp, revealValues bool,
}, nil
}
func (i *SecretsCataloger) Catalog(resolver source.FileResolver) (map[source.Location][]SearchResult, error) {
results := make(map[source.Location][]SearchResult)
func (i *SecretsCataloger) Catalog(resolver source.FileResolver) (map[source.Coordinates][]SearchResult, error) {
results := make(map[source.Coordinates][]SearchResult)
var locations []source.Location
for location := range resolver.AllLocations() {
locations = append(locations, location)
@@ -60,7 +60,7 @@ func (i *SecretsCataloger) Catalog(resolver source.FileResolver) (map[source.Loc
}
if len(result) > 0 {
secretsDiscovered.N += int64(len(result))
results[location] = result
results[location.Coordinates] = result
}
prog.N++
}
+4 -4
View File
@@ -198,11 +198,11 @@ func TestSecretsCataloger(t *testing.T) {
}
loc := source.NewLocation(test.fixture)
if _, exists := actualResults[loc]; !exists {
if _, exists := actualResults[loc.Coordinates]; !exists {
t.Fatalf("could not find location=%q in results", loc)
}
assert.Equal(t, test.expected, actualResults[loc], "mismatched secrets")
assert.Equal(t, test.expected, actualResults[loc.Coordinates], "mismatched secrets")
})
}
}
@@ -432,13 +432,13 @@ j4f668YfhUbKdRF6S6734856
}
loc := source.NewLocation(test.fixture)
if _, exists := actualResults[loc]; !exists && test.expected != nil {
if _, exists := actualResults[loc.Coordinates]; !exists && test.expected != nil {
t.Fatalf("could not find location=%q in results", loc)
} else if !exists && test.expected == nil {
return
}
assert.Equal(t, test.expected, actualResults[loc], "mismatched secrets")
assert.Equal(t, test.expected, actualResults[loc.Coordinates], "mismatched secrets")
})
}
}
+8 -6
View File
@@ -19,6 +19,8 @@ package syft
import (
"fmt"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/internal/bus"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/distro"
@@ -32,10 +34,10 @@ import (
// CatalogPackages takes an inventory of packages from the given image from a particular perspective
// (e.g. squashed source, all-layers source). Returns the discovered set of packages, the identified Linux
// distribution, and the source object used to wrap the data source.
func CatalogPackages(src *source.Source, scope source.Scope) (*pkg.Catalog, *distro.Distro, error) {
func CatalogPackages(src *source.Source, scope source.Scope) (*pkg.Catalog, []artifact.Relationship, *distro.Distro, error) {
resolver, err := src.FileResolver(scope)
if err != nil {
return nil, nil, fmt.Errorf("unable to determine resolver while cataloging packages: %w", err)
return nil, nil, nil, fmt.Errorf("unable to determine resolver while cataloging packages: %w", err)
}
// find the distro
@@ -59,15 +61,15 @@ func CatalogPackages(src *source.Source, scope source.Scope) (*pkg.Catalog, *dis
log.Info("cataloging directory")
catalogers = cataloger.DirectoryCatalogers()
default:
return nil, nil, fmt.Errorf("unable to determine cataloger set from scheme=%+v", src.Metadata.Scheme)
return nil, nil, nil, fmt.Errorf("unable to determine cataloger set from scheme=%+v", src.Metadata.Scheme)
}
catalog, err := cataloger.Catalog(resolver, theDistro, catalogers...)
catalog, relationships, err := cataloger.Catalog(resolver, theDistro, catalogers...)
if err != nil {
return nil, nil, err
return nil, nil, nil, err
}
return catalog, theDistro, nil
return catalog, relationships, theDistro, nil
}
// SetLogger sets the logger object used for all syft logging calls.
+31 -66
View File
@@ -4,6 +4,9 @@ import (
"sort"
"sync"
"github.com/anchore/syft/syft/artifact"
"github.com/jinzhu/copier"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/log"
@@ -11,18 +14,18 @@ import (
// Catalog represents a collection of Packages.
type Catalog struct {
byID map[ID]*Package
idsByType map[Type][]ID
idsByPath map[string][]ID // note: this is real path or virtual path
byID map[artifact.ID]Package
idsByType map[Type][]artifact.ID
idsByPath map[string][]artifact.ID // note: this is real path or virtual path
lock sync.RWMutex
}
// NewCatalog returns a new empty Catalog
func NewCatalog(pkgs ...Package) *Catalog {
catalog := Catalog{
byID: make(map[ID]*Package),
idsByType: make(map[Type][]ID),
idsByPath: make(map[string][]ID),
byID: make(map[artifact.ID]Package),
idsByType: make(map[Type][]artifact.ID),
idsByPath: make(map[string][]artifact.ID),
}
for _, p := range pkgs {
@@ -38,21 +41,26 @@ func (c *Catalog) PackageCount() int {
}
// Package returns the package with the given ID.
func (c *Catalog) Package(id ID) *Package {
func (c *Catalog) Package(id artifact.ID) *Package {
v, exists := c.byID[id]
if !exists {
return nil
}
return v
var p Package
if err := copier.Copy(&p, &v); err != nil {
log.Warnf("unable to copy package id=%q name=%q: %+v", id, v.Name, err)
return nil
}
return &p
}
// PackagesByPath returns all packages that were discovered from the given path.
func (c *Catalog) PackagesByPath(path string) []*Package {
func (c *Catalog) PackagesByPath(path string) []Package {
return c.Packages(c.idsByPath[path])
}
// Packages returns all packages for the given ID.
func (c *Catalog) Packages(ids []ID) (result []*Package) {
func (c *Catalog) Packages(ids []artifact.ID) (result []Package) {
for _, i := range ids {
p, exists := c.byID[i]
if exists {
@@ -67,68 +75,32 @@ func (c *Catalog) Add(p Package) {
c.lock.Lock()
defer c.lock.Unlock()
if p.ID == "" {
fingerprint, err := p.Fingerprint()
if err != nil {
log.Warnf("failed to add package to catalog: %w", err)
return
}
p.ID = ID(fingerprint)
}
// note: since we are capturing the ID, we cannot modify the package being added from this point forward
id := p.ID()
// store by package ID
c.byID[p.ID] = &p
c.byID[id] = p
// store by package type
c.idsByType[p.Type] = append(c.idsByType[p.Type], p.ID)
c.idsByType[p.Type] = append(c.idsByType[p.Type], id)
// store by file location paths
observedPaths := internal.NewStringSet()
for _, l := range p.Locations {
if l.RealPath != "" && !observedPaths.Contains(l.RealPath) {
c.idsByPath[l.RealPath] = append(c.idsByPath[l.RealPath], p.ID)
c.idsByPath[l.RealPath] = append(c.idsByPath[l.RealPath], id)
observedPaths.Add(l.RealPath)
}
if l.VirtualPath != "" && l.RealPath != l.VirtualPath && !observedPaths.Contains(l.VirtualPath) {
c.idsByPath[l.VirtualPath] = append(c.idsByPath[l.VirtualPath], p.ID)
c.idsByPath[l.VirtualPath] = append(c.idsByPath[l.VirtualPath], id)
observedPaths.Add(l.VirtualPath)
}
}
}
func (c *Catalog) Remove(id ID) {
c.lock.Lock()
defer c.lock.Unlock()
_, exists := c.byID[id]
if !exists {
log.Errorf("package ID does not exist in the catalog : id=%+v", id)
return
}
// Remove all index references to this package ID
for t, ids := range c.idsByType {
c.idsByType[t] = removeID(id, ids)
if len(c.idsByType[t]) == 0 {
delete(c.idsByType, t)
}
}
for p, ids := range c.idsByPath {
c.idsByPath[p] = removeID(id, ids)
if len(c.idsByPath[p]) == 0 {
delete(c.idsByPath, p)
}
}
// Remove package
delete(c.byID, id)
}
// Enumerate all packages for the given type(s), enumerating all packages if no type is specified.
func (c *Catalog) Enumerate(types ...Type) <-chan *Package {
channel := make(chan *Package)
func (c *Catalog) Enumerate(types ...Type) <-chan Package {
channel := make(chan Package)
go func() {
defer close(channel)
for ty, ids := range c.idsByType {
@@ -146,7 +118,10 @@ func (c *Catalog) Enumerate(types ...Type) <-chan *Package {
}
}
for _, id := range ids {
channel <- c.Package(id)
p := c.Package(id)
if p != nil {
channel <- *p
}
}
}
}()
@@ -155,8 +130,7 @@ func (c *Catalog) Enumerate(types ...Type) <-chan *Package {
// Sorted enumerates all packages for the given types sorted by package name. Enumerates all packages if no type
// is specified.
func (c *Catalog) Sorted(types ...Type) []*Package {
pkgs := make([]*Package, 0)
func (c *Catalog) Sorted(types ...Type) (pkgs []Package) {
for p := range c.Enumerate(types...) {
pkgs = append(pkgs, p)
}
@@ -176,12 +150,3 @@ func (c *Catalog) Sorted(types ...Type) []*Package {
return pkgs
}
func removeID(id ID, target []ID) (result []ID) {
for _, value := range target {
if value != id {
result = append(result, value)
}
}
return result
}
+37 -96
View File
@@ -10,30 +10,16 @@ import (
var catalogAddAndRemoveTestPkgs = []Package{
{
ID: "my-id",
Locations: []source.Location{
{
RealPath: "/a/path",
VirtualPath: "/another/path",
},
{
RealPath: "/b/path",
VirtualPath: "/bee/path",
},
source.NewVirtualLocation("/a/path", "/another/path"),
source.NewVirtualLocation("/b/path", "/bee/path"),
},
Type: RpmPkg,
},
{
ID: "my-other-id",
Locations: []source.Location{
{
RealPath: "/c/path",
VirtualPath: "/another/path",
},
{
RealPath: "/d/path",
VirtualPath: "/another/path",
},
source.NewVirtualLocation("/c/path", "/another/path"),
source.NewVirtualLocation("/d/path", "/another/path"),
},
Type: NpmPkg,
},
@@ -45,6 +31,11 @@ type expectedIndexes struct {
}
func TestCatalogAddPopulatesIndex(t *testing.T) {
fixtureID := func(i int) string {
return string(catalogAddAndRemoveTestPkgs[i].ID())
}
tests := []struct {
name string
pkgs []Package
@@ -55,16 +46,16 @@ func TestCatalogAddPopulatesIndex(t *testing.T) {
pkgs: catalogAddAndRemoveTestPkgs,
expectedIndexes: expectedIndexes{
byType: map[Type]*strset.Set{
RpmPkg: strset.New("my-id"),
NpmPkg: strset.New("my-other-id"),
RpmPkg: strset.New(fixtureID(0)),
NpmPkg: strset.New(fixtureID(1)),
},
byPath: map[string]*strset.Set{
"/another/path": strset.New("my-id", "my-other-id"),
"/a/path": strset.New("my-id"),
"/b/path": strset.New("my-id"),
"/bee/path": strset.New("my-id"),
"/c/path": strset.New("my-other-id"),
"/d/path": strset.New("my-other-id"),
"/another/path": strset.New(fixtureID(0), fixtureID(1)),
"/a/path": strset.New(fixtureID(0)),
"/b/path": strset.New(fixtureID(0)),
"/bee/path": strset.New(fixtureID(0)),
"/c/path": strset.New(fixtureID(1)),
"/d/path": strset.New(fixtureID(1)),
},
},
},
@@ -80,50 +71,6 @@ func TestCatalogAddPopulatesIndex(t *testing.T) {
}
}
func TestCatalogRemove(t *testing.T) {
tests := []struct {
name string
pkgs []Package
removeId ID
expectedIndexes expectedIndexes
}{
{
name: "vanilla-add",
removeId: "my-other-id",
pkgs: catalogAddAndRemoveTestPkgs,
expectedIndexes: expectedIndexes{
byType: map[Type]*strset.Set{
RpmPkg: strset.New("my-id"),
},
byPath: map[string]*strset.Set{
"/another/path": strset.New("my-id"),
"/a/path": strset.New("my-id"),
"/b/path": strset.New("my-id"),
"/bee/path": strset.New("my-id"),
},
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
c := NewCatalog(test.pkgs...)
c.Remove(test.removeId)
assertIndexes(t, c, test.expectedIndexes)
if c.Package(test.removeId) != nil {
t.Errorf("expected package to be removed, but was found!")
}
if c.PackageCount() != len(test.pkgs)-1 {
t.Errorf("expected count to be affected but was not")
}
})
}
}
func assertIndexes(t *testing.T, c *Catalog, expectedIndexes expectedIndexes) {
// assert path index
if len(c.idsByPath) != len(expectedIndexes.byPath) {
@@ -132,7 +79,7 @@ func assertIndexes(t *testing.T, c *Catalog, expectedIndexes expectedIndexes) {
for path, expectedIds := range expectedIndexes.byPath {
actualIds := strset.New()
for _, p := range c.PackagesByPath(path) {
actualIds.Add(string(p.ID))
actualIds.Add(string(p.ID()))
}
if !expectedIds.IsEqual(actualIds) {
@@ -147,7 +94,7 @@ func assertIndexes(t *testing.T, c *Catalog, expectedIndexes expectedIndexes) {
for ty, expectedIds := range expectedIndexes.byType {
actualIds := strset.New()
for p := range c.Enumerate(ty) {
actualIds.Add(string(p.ID))
actualIds.Add(string(p.ID()))
}
if !expectedIds.IsEqual(actualIds) {
@@ -157,39 +104,33 @@ func assertIndexes(t *testing.T, c *Catalog, expectedIndexes expectedIndexes) {
}
func TestCatalog_PathIndexDeduplicatesRealVsVirtualPaths(t *testing.T) {
p1 := Package{
Locations: []source.Location{
source.NewVirtualLocation("/b/path", "/another/path"),
source.NewVirtualLocation("/b/path", "/b/path"),
},
Type: RpmPkg,
Name: "Package-1",
}
p2 := Package{
Locations: []source.Location{
source.NewVirtualLocation("/b/path", "/b/path"),
},
Type: RpmPkg,
Name: "Package-2",
}
tests := []struct {
name string
pkg Package
}{
{
name: "multiple locations with shared path",
pkg: Package{
ID: "my-id",
Locations: []source.Location{
{
RealPath: "/b/path",
VirtualPath: "/another/path",
},
{
RealPath: "/b/path",
VirtualPath: "/b/path",
},
},
Type: RpmPkg,
},
pkg: p1,
},
{
name: "one location with shared path",
pkg: Package{
ID: "my-id",
Locations: []source.Location{
{
RealPath: "/b/path",
VirtualPath: "/b/path",
},
},
Type: RpmPkg,
},
pkg: p2,
},
}
+6 -4
View File
@@ -8,6 +8,8 @@ import (
"strconv"
"strings"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/internal/log"
@@ -21,7 +23,7 @@ var _ common.ParserFn = parseApkDB
// parseApkDb parses individual packages from a given Alpine DB file. For more information on specific fields
// see https://wiki.alpinelinux.org/wiki/Apk_spec .
func parseApkDB(_ string, reader io.Reader) ([]pkg.Package, error) {
func parseApkDB(_ string, reader io.Reader) ([]pkg.Package, []artifact.Relationship, error) {
// larger capacity for the scanner.
const maxScannerCapacity = 1024 * 1024
// a new larger buffer for the scanner
@@ -47,7 +49,7 @@ func parseApkDB(_ string, reader io.Reader) ([]pkg.Package, error) {
for scanner.Scan() {
metadata, err := parseApkDBEntry(strings.NewReader(scanner.Text()))
if err != nil {
return nil, err
return nil, nil, err
}
if metadata != nil {
packages = append(packages, pkg.Package{
@@ -62,10 +64,10 @@ func parseApkDB(_ string, reader io.Reader) ([]pkg.Package, error) {
}
if err := scanner.Err(); err != nil {
return nil, fmt.Errorf("failed to parse APK DB file: %w", err)
return nil, nil, fmt.Errorf("failed to parse APK DB file: %w", err)
}
return packages, nil
return packages, nil, nil
}
// nolint:funlen
@@ -775,7 +775,8 @@ func TestMultiplePackages(t *testing.T) {
}
}()
pkgs, err := parseApkDB(file.Name(), file)
// TODO: no relationships are under test yet
pkgs, _, err := parseApkDB(file.Name(), file)
if err != nil {
t.Fatal("Unable to read file contents: ", err)
}
+52 -4
View File
@@ -1,8 +1,11 @@
package cataloger
import (
"fmt"
"github.com/anchore/syft/internal/bus"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/distro"
"github.com/anchore/syft/syft/event"
"github.com/anchore/syft/syft/pkg"
@@ -38,8 +41,9 @@ func newMonitor() (*progress.Manual, *progress.Manual) {
// In order to efficiently retrieve contents from a underlying container image the content fetch requests are
// done in bulk. Specifically, all files of interest are collected from each catalogers and accumulated into a single
// request.
func Catalog(resolver source.FileResolver, theDistro *distro.Distro, catalogers ...Cataloger) (*pkg.Catalog, error) {
func Catalog(resolver source.FileResolver, theDistro *distro.Distro, catalogers ...Cataloger) (*pkg.Catalog, []artifact.Relationship, error) {
catalog := pkg.NewCatalog()
var allRelationships []artifact.Relationship
filesProcessed, packagesDiscovered := newMonitor()
@@ -47,7 +51,7 @@ func Catalog(resolver source.FileResolver, theDistro *distro.Distro, catalogers
var errs error
for _, theCataloger := range catalogers {
// find packages from the underlying raw data
packages, err := theCataloger.Catalog(resolver)
packages, relationships, err := theCataloger.Catalog(resolver)
if err != nil {
errs = multierror.Append(errs, err)
continue
@@ -65,17 +69,61 @@ func Catalog(resolver source.FileResolver, theDistro *distro.Distro, catalogers
// generate PURL
p.PURL = generatePackageURL(p, theDistro)
// create file-to-package relationships for files owned by the package
owningRelationships, err := packageFileOwnershipRelationships(p, resolver)
if err != nil {
log.Warnf("unable to create any package-file relationships for package name=%q: %w", p.Name, err)
} else {
allRelationships = append(allRelationships, owningRelationships...)
}
// add to catalog
catalog.Add(p)
}
allRelationships = append(allRelationships, relationships...)
}
allRelationships = append(allRelationships, pkg.NewRelationships(catalog)...)
if errs != nil {
return nil, errs
return nil, nil, errs
}
filesProcessed.SetCompleted()
packagesDiscovered.SetCompleted()
return catalog, nil
return catalog, allRelationships, nil
}
func packageFileOwnershipRelationships(p pkg.Package, resolver source.FilePathResolver) ([]artifact.Relationship, error) {
fileOwner, ok := p.Metadata.(pkg.FileOwner)
if !ok {
return nil, nil
}
var relationships []artifact.Relationship
for _, path := range fileOwner.OwnedFiles() {
locations, err := resolver.FilesByPath(path)
if err != nil {
return nil, fmt.Errorf("unable to find path for path=%q: %w", path, err)
}
if len(locations) == 0 {
// TODO: this is a known-unknown that could later be persisted in the SBOM (or as a validation failure)
log.Warnf("unable to find location which a package claims ownership of: %s", path)
continue
}
for _, l := range locations {
relationships = append(relationships, artifact.Relationship{
From: p,
To: l.Coordinates,
Type: artifact.ContainsRelationship,
})
}
}
return relationships, nil
}
+4 -2
View File
@@ -6,6 +6,7 @@ catalogers defined in child packages as well as the interface definition to impl
package cataloger
import (
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/pkg/cataloger/apkdb"
"github.com/anchore/syft/syft/pkg/cataloger/deb"
@@ -27,7 +28,7 @@ type Cataloger interface {
// Name returns a string that uniquely describes a cataloger
Name() string
// Catalog is given an object to resolve file references and content, this function returns any discovered Packages after analyzing the catalog source.
Catalog(resolver source.FileResolver) ([]pkg.Package, error)
Catalog(resolver source.FileResolver) ([]pkg.Package, []artifact.Relationship, error)
}
// ImageCatalogers returns a slice of locally implemented catalogers that are fit for detecting installations of packages.
@@ -35,6 +36,7 @@ func ImageCatalogers() []Cataloger {
return []Cataloger{
ruby.NewGemSpecCataloger(),
python.NewPythonPackageCataloger(),
php.NewPHPComposerInstalledCataloger(),
javascript.NewJavascriptPackageCataloger(),
deb.NewDpkgdbCataloger(),
rpmdb.NewRpmdbCataloger(),
@@ -50,7 +52,7 @@ func DirectoryCatalogers() []Cataloger {
ruby.NewGemFileLockCataloger(),
python.NewPythonIndexCataloger(),
python.NewPythonPackageCataloger(),
php.NewPHPIndexCataloger(),
php.NewPHPComposerLockCataloger(),
javascript.NewJavascriptLockCataloger(),
deb.NewDpkgdbCataloger(),
rpmdb.NewRpmdbCataloger(),
+1 -1
View File
@@ -83,7 +83,7 @@ func candidateVendors(p pkg.Package) []string {
// allow * as a candidate. Note: do NOT allow Java packages to have * vendors.
switch p.Language {
case pkg.Ruby, pkg.JavaScript:
vendors.addValue("*")
vendors.addValue(wfn.Any)
}
switch p.MetadataType {
@@ -637,7 +637,7 @@ func TestCandidateProducts(t *testing.T) {
}
for _, test := range tests {
t.Run(fmt.Sprintf("%+v %+v", test.p, test.expected), func(t *testing.T) {
t.Run(test.name, func(t *testing.T) {
assert.ElementsMatch(t, test.expected, candidateProducts(test.p))
})
}
+14 -10
View File
@@ -6,6 +6,8 @@ package common
import (
"fmt"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/pkg"
@@ -35,18 +37,18 @@ func (c *GenericCataloger) Name() string {
}
// Catalog is given an object to resolve file references and content, this function returns any discovered Packages after analyzing the catalog source.
func (c *GenericCataloger) Catalog(resolver source.FileResolver) ([]pkg.Package, error) {
func (c *GenericCataloger) Catalog(resolver source.FileResolver) ([]pkg.Package, []artifact.Relationship, error) {
var packages []pkg.Package
parserByLocation := c.selectFiles(resolver)
var relationships []artifact.Relationship
for location, parser := range parserByLocation {
for location, parser := range c.selectFiles(resolver) {
contentReader, err := resolver.FileContentsByLocation(location)
if err != nil {
// TODO: fail or log?
return nil, fmt.Errorf("unable to fetch contents for location=%v : %w", location, err)
return nil, nil, fmt.Errorf("unable to fetch contents for location=%v : %w", location, err)
}
entries, err := parser(location.RealPath, contentReader)
discoveredPackages, discoveredRelationships, err := parser(location.RealPath, contentReader)
internal.CloseAndLogError(contentReader, location.VirtualPath)
if err != nil {
// TODO: should we fail? or only log?
@@ -54,14 +56,16 @@ func (c *GenericCataloger) Catalog(resolver source.FileResolver) ([]pkg.Package,
continue
}
for _, entry := range entries {
entry.FoundBy = c.upstreamCataloger
entry.Locations = []source.Location{location}
for _, p := range discoveredPackages {
p.FoundBy = c.upstreamCataloger
p.Locations = append(p.Locations, location)
packages = append(packages, entry)
packages = append(packages, p)
}
relationships = append(relationships, discoveredRelationships...)
}
return packages, nil
return packages, relationships, nil
}
// SelectFiles takes a set of file trees and resolves and file references of interest for future cataloging
@@ -8,11 +8,12 @@ import (
"github.com/stretchr/testify/assert"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/source"
)
func parser(_ string, reader io.Reader) ([]pkg.Package, error) {
func parser(_ string, reader io.Reader) ([]pkg.Package, []artifact.Relationship, error) {
contents, err := ioutil.ReadAll(reader)
if err != nil {
panic(err)
@@ -21,7 +22,7 @@ func parser(_ string, reader io.Reader) ([]pkg.Package, error) {
{
Name: string(contents),
},
}, nil
}, nil, nil
}
func TestGenericCataloger(t *testing.T) {
@@ -47,7 +48,7 @@ func TestGenericCataloger(t *testing.T) {
}
}
actualPkgs, err := cataloger.Catalog(resolver)
actualPkgs, _, err := cataloger.Catalog(resolver)
assert.NoError(t, err)
assert.Len(t, actualPkgs, len(expectedPkgs))
+2 -1
View File
@@ -3,8 +3,9 @@ package common
import (
"io"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/pkg"
)
// ParserFn standardizes a function signature for parser functions that accept the virtual file path (not usable for file reads) and contents and return any discovered packages from that file
type ParserFn func(string, io.Reader) ([]pkg.Package, error)
type ParserFn func(string, io.Reader) ([]pkg.Package, []artifact.Relationship, error)
+9 -9
View File
@@ -13,6 +13,7 @@ import (
"github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/source"
)
@@ -36,24 +37,23 @@ func (c *Cataloger) Name() string {
}
// Catalog is given an object to resolve file references and content, this function returns any discovered Packages after analyzing dpkg support files.
func (c *Cataloger) Catalog(resolver source.FileResolver) ([]pkg.Package, error) {
func (c *Cataloger) Catalog(resolver source.FileResolver) ([]pkg.Package, []artifact.Relationship, error) {
dbFileMatches, err := resolver.FilesByGlob(pkg.DpkgDBGlob)
if err != nil {
return nil, fmt.Errorf("failed to find dpkg status files's by glob: %w", err)
return nil, nil, fmt.Errorf("failed to find dpkg status files's by glob: %w", err)
}
var results []pkg.Package
var pkgs []pkg.Package
var allPackages []pkg.Package
for _, dbLocation := range dbFileMatches {
dbContents, err := resolver.FileContentsByLocation(dbLocation)
if err != nil {
return nil, err
return nil, nil, err
}
pkgs, err = parseDpkgStatus(dbContents)
pkgs, err := parseDpkgStatus(dbContents)
internal.CloseAndLogError(dbContents, dbLocation.VirtualPath)
if err != nil {
return nil, fmt.Errorf("unable to catalog dpkg package=%+v: %w", dbLocation.RealPath, err)
return nil, nil, fmt.Errorf("unable to catalog dpkg package=%+v: %w", dbLocation.RealPath, err)
}
for i := range pkgs {
@@ -70,9 +70,9 @@ func (c *Cataloger) Catalog(resolver source.FileResolver) ([]pkg.Package, error)
addLicenses(resolver, dbLocation, p)
}
results = append(results, pkgs...)
allPackages = append(allPackages, pkgs...)
}
return results, nil
return allPackages, nil, nil
}
func addLicenses(resolver source.FileResolver, dbLocation source.Location, p *pkg.Package) {
+1 -1
View File
@@ -100,7 +100,7 @@ func TestDpkgCataloger(t *testing.T) {
t.Errorf("could not get resolver error: %+v", err)
}
actual, err := c.Catalog(resolver)
actual, _, err := c.Catalog(resolver)
if err != nil {
t.Fatalf("failed to catalog: %+v", err)
}
+1 -1
View File
@@ -23,7 +23,7 @@ var (
// parseDpkgStatus is a parser function for Debian DB status contents, returning all Debian packages listed.
func parseDpkgStatus(reader io.Reader) ([]pkg.Package, error) {
buffedReader := bufio.NewReader(reader)
var packages = make([]pkg.Package, 0)
var packages []pkg.Package
continueProcessing := true
for continueProcessing {
@@ -6,22 +6,16 @@ package golang
import (
"fmt"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/source"
)
const catalogerName = "go-module-binary-cataloger"
// current mime types to search by to discover go binaries
var mimeTypes = []string{
"application/x-executable",
"application/x-mach-binary",
"application/x-elf",
"application/x-sharedlib",
"application/vnd.microsoft.portable-executable",
}
type Cataloger struct{}
// NewGoModuleBinaryCataloger returns a new Golang cataloger object.
@@ -35,17 +29,18 @@ func (c *Cataloger) Name() string {
}
// Catalog is given an object to resolve file references and content, this function returns any discovered Packages after analyzing rpm db installation.
func (c *Cataloger) Catalog(resolver source.FileResolver) ([]pkg.Package, error) {
pkgs := make([]pkg.Package, 0)
fileMatches, err := resolver.FilesByMIMEType(mimeTypes...)
func (c *Cataloger) Catalog(resolver source.FileResolver) ([]pkg.Package, []artifact.Relationship, error) {
var pkgs []pkg.Package
fileMatches, err := resolver.FilesByMIMEType(internal.ExecutableMIMETypeSet.List()...)
if err != nil {
return pkgs, fmt.Errorf("failed to find bin by mime types: %w", err)
return pkgs, nil, fmt.Errorf("failed to find bin by mime types: %w", err)
}
for _, location := range fileMatches {
r, err := resolver.FileContentsByLocation(location)
if err != nil {
return pkgs, fmt.Errorf("failed to resolve file contents by location: %w", err)
return pkgs, nil, fmt.Errorf("failed to resolve file contents by location: %w", err)
}
goPkgs, err := parseGoBin(location, r)
@@ -53,9 +48,9 @@ func (c *Cataloger) Catalog(resolver source.FileResolver) ([]pkg.Package, error)
log.Warnf("could not parse possible go binary: %+v", err)
}
r.Close()
internal.CloseAndLogError(r, location.RealPath)
pkgs = append(pkgs, goPkgs...)
}
return pkgs, nil
return pkgs, nil, nil
}
+1 -3
View File
@@ -23,9 +23,7 @@ func parseGoBin(location source.Location, reader io.ReadCloser) ([]pkg.Package,
goVersion, mod := findVers(x)
pkgs := buildGoPkgInfo(location, mod, goVersion)
return pkgs, nil
return buildGoPkgInfo(location, mod, goVersion), nil
}
func buildGoPkgInfo(location source.Location, mod, goVersion string) []pkg.Package {
+26 -11
View File
@@ -34,8 +34,10 @@ func TestBuildGoPkgInfo(t *testing.T) {
Type: pkg.GoModulePkg,
Locations: []source.Location{
{
RealPath: "/a-path",
FileSystemID: "layer-id",
Coordinates: source.Coordinates{
RealPath: "/a-path",
FileSystemID: "layer-id",
},
},
},
MetadataType: pkg.GolangBinMetadataType,
@@ -51,8 +53,10 @@ func TestBuildGoPkgInfo(t *testing.T) {
Type: pkg.GoModulePkg,
Locations: []source.Location{
{
RealPath: "/a-path",
FileSystemID: "layer-id",
Coordinates: source.Coordinates{
RealPath: "/a-path",
FileSystemID: "layer-id",
},
},
},
MetadataType: pkg.GolangBinMetadataType,
@@ -79,8 +83,10 @@ func TestBuildGoPkgInfo(t *testing.T) {
Type: pkg.GoModulePkg,
Locations: []source.Location{
{
RealPath: "/a-path",
FileSystemID: "layer-id",
Coordinates: source.Coordinates{
RealPath: "/a-path",
FileSystemID: "layer-id",
},
},
},
MetadataType: pkg.GolangBinMetadataType,
@@ -96,8 +102,10 @@ func TestBuildGoPkgInfo(t *testing.T) {
Type: pkg.GoModulePkg,
Locations: []source.Location{
{
RealPath: "/a-path",
FileSystemID: "layer-id",
Coordinates: source.Coordinates{
RealPath: "/a-path",
FileSystemID: "layer-id",
},
},
},
MetadataType: pkg.GolangBinMetadataType,
@@ -113,8 +121,10 @@ func TestBuildGoPkgInfo(t *testing.T) {
Type: pkg.GoModulePkg,
Locations: []source.Location{
{
RealPath: "/a-path",
FileSystemID: "layer-id",
Coordinates: source.Coordinates{
RealPath: "/a-path",
FileSystemID: "layer-id",
},
},
},
MetadataType: pkg.GolangBinMetadataType,
@@ -130,7 +140,12 @@ func TestBuildGoPkgInfo(t *testing.T) {
for _, tt := range tests {
tt := tt
t.Run(tt.name, func(t *testing.T) {
location := source.Location{RealPath: "/a-path", FileSystemID: "layer-id"}
location := source.Location{
Coordinates: source.Coordinates{
RealPath: "/a-path",
FileSystemID: "layer-id",
},
}
pkgs := buildGoPkgInfo(location, tt.mod, goCompiledVersion)
assert.Equal(t, tt.expected, pkgs)
})
+5 -4
View File
@@ -6,22 +6,23 @@ import (
"io/ioutil"
"sort"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/pkg"
"golang.org/x/mod/modfile"
)
// parseGoMod takes a go.mod and lists all packages discovered.
func parseGoMod(path string, reader io.Reader) ([]pkg.Package, error) {
func parseGoMod(path string, reader io.Reader) ([]pkg.Package, []artifact.Relationship, error) {
packages := make(map[string]pkg.Package)
contents, err := ioutil.ReadAll(reader)
if err != nil {
return nil, fmt.Errorf("failed to read go module: %w", err)
return nil, nil, fmt.Errorf("failed to read go module: %w", err)
}
file, err := modfile.Parse(path, contents, nil)
if err != nil {
return nil, fmt.Errorf("failed to parse go module: %w", err)
return nil, nil, fmt.Errorf("failed to parse go module: %w", err)
}
for _, m := range file.Require {
@@ -59,5 +60,5 @@ func parseGoMod(path string, reader io.Reader) ([]pkg.Package, error) {
return pkgsSlice[i].Name < pkgsSlice[j].Name
})
return pkgsSlice, nil
return pkgsSlice, nil, nil
}
@@ -70,7 +70,8 @@ func TestParseGoMod(t *testing.T) {
t.Fatalf(err.Error())
}
actual, err := parseGoMod(test.fixture, f)
// TODO: no relationships are under test yet
actual, _, err := parseGoMod(test.fixture, f)
if err != nil {
t.Fatalf(err.Error())
}
+23 -17
View File
@@ -9,6 +9,7 @@ import (
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/internal/file"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/pkg/cataloger/common"
)
@@ -34,12 +35,12 @@ type archiveParser struct {
}
// parseJavaArchive is a parser function for java archive contents, returning all Java libraries and nested archives.
func parseJavaArchive(virtualPath string, reader io.Reader) ([]pkg.Package, error) {
func parseJavaArchive(virtualPath string, reader io.Reader) ([]pkg.Package, []artifact.Relationship, error) {
parser, cleanupFn, err := newJavaArchiveParser(virtualPath, reader, true)
// note: even on error, we should always run cleanup functions
defer cleanupFn()
if err != nil {
return nil, err
return nil, nil, err
}
return parser.parse()
}
@@ -80,29 +81,31 @@ func newJavaArchiveParser(virtualPath string, reader io.Reader, detectNested boo
}
// parse the loaded archive and return all packages found.
func (j *archiveParser) parse() ([]pkg.Package, error) {
var pkgs = make([]pkg.Package, 0)
func (j *archiveParser) parse() ([]pkg.Package, []artifact.Relationship, error) {
var pkgs []pkg.Package
var relationships []artifact.Relationship
// find the parent package from the java manifest
parentPkg, err := j.discoverMainPackage()
if err != nil {
return nil, fmt.Errorf("could not generate package from %s: %w", j.virtualPath, err)
return nil, nil, fmt.Errorf("could not generate package from %s: %w", j.virtualPath, err)
}
// find aux packages from pom.properties/pom.xml and potentially modify the existing parentPkg
auxPkgs, err := j.discoverPkgsFromAllMavenFiles(parentPkg)
if err != nil {
return nil, err
return nil, nil, err
}
pkgs = append(pkgs, auxPkgs...)
if j.detectNested {
// find nested java archive packages
nestedPkgs, err := j.discoverPkgsFromNestedArchives(parentPkg)
nestedPkgs, nestedRelationships, err := j.discoverPkgsFromNestedArchives(parentPkg)
if err != nil {
return nil, err
return nil, nil, err
}
pkgs = append(pkgs, nestedPkgs...)
relationships = append(relationships, nestedRelationships...)
}
// lastly, add the parent package to the list (assuming the parent exists)
@@ -110,7 +113,7 @@ func (j *archiveParser) parse() ([]pkg.Package, error) {
pkgs = append([]pkg.Package{*parentPkg}, pkgs...)
}
return pkgs, nil
return pkgs, relationships, nil
}
// discoverMainPackage parses the root Java manifest used as the parent package to all discovered nested packages.
@@ -189,31 +192,32 @@ func (j *archiveParser) discoverPkgsFromAllMavenFiles(parentPkg *pkg.Package) ([
// discoverPkgsFromNestedArchives finds Java archives within Java archives, returning all listed Java packages found and
// associating each discovered package to the given parent package.
func (j *archiveParser) discoverPkgsFromNestedArchives(parentPkg *pkg.Package) ([]pkg.Package, error) {
var pkgs = make([]pkg.Package, 0)
func (j *archiveParser) discoverPkgsFromNestedArchives(parentPkg *pkg.Package) ([]pkg.Package, []artifact.Relationship, error) {
var pkgs []pkg.Package
var relationships []artifact.Relationship
// search and parse pom.properties files & fetch the contents
openers, err := file.ExtractFromZipToUniqueTempFile(j.archivePath, j.contentPath, j.fileManifest.GlobMatch(archiveFormatGlobs...)...)
if err != nil {
return nil, fmt.Errorf("unable to extract files from zip: %w", err)
return nil, nil, fmt.Errorf("unable to extract files from zip: %w", err)
}
// discover nested artifacts
for archivePath, archiveOpener := range openers {
archiveReadCloser, err := archiveOpener.Open()
if err != nil {
return nil, fmt.Errorf("unable to open archived file from tempdir: %w", err)
return nil, nil, fmt.Errorf("unable to open archived file from tempdir: %w", err)
}
nestedPath := fmt.Sprintf("%s:%s", j.virtualPath, archivePath)
nestedPkgs, err := parseJavaArchive(nestedPath, archiveReadCloser)
nestedPkgs, nestedRelationships, err := parseJavaArchive(nestedPath, archiveReadCloser)
if err != nil {
if closeErr := archiveReadCloser.Close(); closeErr != nil {
log.Warnf("unable to close archived file from tempdir: %+v", closeErr)
}
return nil, fmt.Errorf("unable to process nested java archive (%s): %w", archivePath, err)
return nil, nil, fmt.Errorf("unable to process nested java archive (%s): %w", archivePath, err)
}
if err = archiveReadCloser.Close(); err != nil {
return nil, fmt.Errorf("unable to close archived file from tempdir: %w", err)
return nil, nil, fmt.Errorf("unable to close archived file from tempdir: %w", err)
}
// attach the parent package to all discovered packages that are not already associated with a java archive
@@ -226,9 +230,11 @@ func (j *archiveParser) discoverPkgsFromNestedArchives(parentPkg *pkg.Package) (
}
pkgs = append(pkgs, p)
}
relationships = append(relationships, nestedRelationships...)
}
return pkgs, nil
return pkgs, relationships, nil
}
func pomPropertiesByParentPath(archivePath string, extractPaths []string, virtualPath string) (map[string]pkg.PomProperties, error) {
@@ -242,7 +242,7 @@ func TestParseJar(t *testing.T) {
t.Fatalf("should not have filed... %+v", err)
}
actual, err := parser.parse()
actual, _, err := parser.parse()
if err != nil {
t.Fatalf("failed to parse java archive: %+v", err)
}
@@ -507,7 +507,7 @@ func TestParseNestedJar(t *testing.T) {
t.Fatalf("failed to open fixture: %+v", err)
}
actual, err := parseJavaArchive(fixture.Name(), fixture)
actual, _, err := parseJavaArchive(fixture.Name(), fixture)
if err != nil {
t.Fatalf("failed to parse java archive: %+v", err)
}

Some files were not shown because too many files have changed in this diff Show More