Compare commits

...
9 Commits
Author SHA1 Message Date
MikiandChristopher Phillips d5e12ff89c Updates parsing of yarn.lock to use resolved URLs that are pulled from yarn and npm registries (#926)
Co-authored-by: Christopher Phillips <christopher.phillips@anchore.com>
2022-06-24 11:05:25 -04:00
Christopher Angelo Phillips bafc66a5e7 remove OSS Meetup message (#1057) 2022-06-23 18:07:23 +00:00
Jonas XavierandChristopher Phillips 1d14f22e45 add pom.xml cataloger (#1055)
Co-authored-by: Christopher Phillips <christopher.phillips@anchore.com>
2022-06-22 14:19:10 -04:00
Tom Fay 3f6afd572a Add support for CBL-Mariner distroless images (#1045) 2022-06-21 13:27:03 -04:00
Alex Goodmanandramanan-ravi ea611dab5f Add catalogers configuration (#1038)
* Option to enable specific language or ecosystem cataloger

Signed-off-by: ramanan-ravi <ramanan@deepfence.io>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* Disable dotnet cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* Option to enable specific language or ecosystem cataloger

Signed-off-by: Ramanan Ravikumar <ramanan@deepfence.io>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* rename "enable-cataloger" option to "catalogers"

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add cli test for --catalogers option

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update readme with latest cataloger names

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* enable dotnet cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix linting

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix cataloger imports

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update readme with alpmdb cataloger config example

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

Co-authored-by: ramanan-ravi <ramanan@deepfence.io>
2022-06-21 13:06:25 +00:00
Jonas Xavier aed1599c4d add template output (#1051)
* add template output

Signed-off-by: Jonas Xavier <jonasx@anchore.com>

* remove dead code

Signed-off-by: Jonas Xavier <jonasx@anchore.com>

* fix template cli flag

Signed-off-by: Jonas Xavier <jonasx@anchore.com>

* implement template's own format type

Signed-off-by: Jonas Xavier <jonasx@anchore.com>

* simpler code

Signed-off-by: Jonas Xavier <jonasx@anchore.com>

* fix readme link to Go template

Signed-off-by: Jonas Xavier <jonasx@anchore.com>

* feedback changes

Signed-off-by: Jonas Xavier <jonasx@anchore.com>

* simpler func signature patter

Signed-off-by: Jonas Xavier <jonasx@anchore.com>

* nit

Signed-off-by: Jonas Xavier <jonasx@anchore.com>

* fix linter error

Signed-off-by: Jonas Xavier <jonasx@anchore.com>
2022-06-17 14:04:31 -04:00
Christopher Angelo Phillips 03e37044d4 update stereoscope to latest version (#1052) 2022-06-16 14:56:33 -04:00
Christopher Angelo Phillips 9e72771b85 update zip_read_closer to incorporate zip64 support (#1041) 2022-06-16 10:43:18 -04:00
Morten LinderudandChristopher Phillips e72d68b0c6 Add pacman (alpm) parser support (#943)
Co-authored-by: Christopher Phillips <christopher.phillips@anchore.com>
2022-06-13 18:51:37 +00:00
175 changed files with 4162 additions and 12086 deletions
+57 -8
View File
@@ -11,14 +11,6 @@
A CLI tool and Go library for generating a Software Bill of Materials (SBOM) from container images and filesystems. Exceptional for vulnerability detection when used with a scanner like [Grype](https://github.com/anchore/grype).
### Join our Virtual OSS Meetup!
You are invited to join us on June 15th, 11AM-Noon PT for our virtual open source meetup.
Hosts Amy Bass from Docker Desktop and Christopher Phillips from Anchore OSS will explore how Docker Extensions for Docker Desktop is supporting open source projects and we’ll have the latest update on Syft: in-toto attestations.
[Register here ->](https://get.anchore.com/anchore-oss-meetup-jun-15-2022/)
### Join our community meetings!
- Calendar: https://calendar.google.com/calendar/u/0/r?cid=Y182OTM4dGt0MjRtajI0NnNzOThiaGtnM29qNEBncm91cC5jYWxlbmRhci5nb29nbGUuY29t
@@ -230,6 +222,40 @@ Where the `formats` available are:
- `spdx-json`: A JSON report conforming to the [SPDX 2.2 JSON Schema](https://github.com/spdx/spdx-spec/blob/v2.2/schemas/spdx-schema.json).
- `github`: A JSON report conforming to GitHub's dependency snapshot format.
- `table`: A columnar summary (default).
- `template`: Lets the user specify the output format. See ["Using templates"](#using-templates) below.
#### Using templates
Syft lets you define custom output formats, using [Go templates](https://pkg.go.dev/text/template). Here's how it works:
- Define your format as a Go template, and save this template as a file.
- Set the output format to "template" (`-o template`).
- Specify the path to the template file (`-t ./path/to/custom.template`).
- Syft's template processing uses the same data models as the `json` output format — so if you're wondering what data is available as you author a template, you can use the output from `syft <image> -o json` as a reference.
**Example:** You could make Syft output data in CSV format by writing a Go template that renders CSV data and then running `syft <image> -o template -t ~/path/to/csv.tmpl`.
Here's what the `csv.tmpl` file might look like:
```gotemplate
"Package","Version Installed","Found by"
{{- range .Artifacts}}
"{{.Name}}","{{.Version}}","{{.FoundBy}}"
{{- end}}
```
Which would produce output like:
```text
"Package","Version Installed","Found by"
"alpine-baselayout","3.2.0-r20","apkdb-cataloger"
"alpine-baselayout-data","3.2.0-r20","apkdb-cataloger"
"alpine-keys","2.4-r1","apkdb-cataloger"
...
```
Syft also includes a vast array of utility templating functions from [sprig](http://masterminds.github.io/sprig/) apart from the default Golang [text/template](https://pkg.go.dev/text/template#hdr-Functions) to allow users to customize the output format.
#### Multiple outputs
@@ -360,6 +386,29 @@ exclude: []
# same as --platform; SYFT_PLATFORM env var
platform: ""
# set the list of package catalogers to use when generating the SBOM
# default = empty (cataloger set determined automatically by the source type [image or file/directory])
# catalogers:
# - ruby-gemfile
# - ruby-gemspec
# - python-index
# - python-package
# - javascript-lock
# - javascript-package
# - php-composer-installed
# - php-composer-lock
# - alpmdb
# - dpkgdb
# - rpmdb
# - java
# - apkdb
# - go-module-binary
# - go-mod-file
# - dartlang-lock
# - rust
# - dotnet-deps
catalogers:
# cataloging packages is exposed through the packages and power-user subcommands
package:
+1 -1
View File
@@ -13,7 +13,7 @@ import (
func Run(ctx context.Context, app *config.Application, args []string) error {
log.Warn("convert is an experimental feature, run `syft convert -h` for help")
writer, err := options.MakeWriter(app.Outputs, app.File)
writer, err := options.MakeWriter(app.Outputs, app.File, "")
if err != nil {
return err
}
+1 -1
View File
@@ -46,7 +46,7 @@ func generateCatalogPackagesTask(app *config.Application) (Task, error) {
}
task := func(results *sbom.Artifacts, src *source.Source) ([]artifact.Relationship, error) {
packageCatalog, relationships, theDistro, err := syft.CatalogPackages(src, app.Package.ToConfig())
packageCatalog, relationships, theDistro, err := syft.CatalogPackages(src, app.ToCatalogerConfig())
if err != nil {
return nil, err
}
+28 -12
View File
@@ -15,6 +15,7 @@ import (
type PackagesOptions struct {
Scope string
Output []string
OutputTemplatePath string
File string
Platform string
Host string
@@ -24,45 +25,52 @@ type PackagesOptions struct {
Exclude []string
OverwriteExistingImage bool
ImportTimeout uint
Catalogers []string
}
var _ Interface = (*PackagesOptions)(nil)
func (o *PackagesOptions) AddFlags(cmd *cobra.Command, v *viper.Viper) error {
cmd.PersistentFlags().StringVarP(&o.Scope, "scope", "s", cataloger.DefaultSearchConfig().Scope.String(),
cmd.Flags().StringVarP(&o.Scope, "scope", "s", cataloger.DefaultSearchConfig().Scope.String(),
fmt.Sprintf("selection of layers to catalog, options=%v", source.AllScopes))
cmd.PersistentFlags().StringArrayVarP(&o.Output, "output", "o", FormatAliases(table.ID),
cmd.Flags().StringArrayVarP(&o.Output, "output", "o", FormatAliases(table.ID),
fmt.Sprintf("report output format, options=%v", FormatAliases(syft.FormatIDs()...)))
cmd.PersistentFlags().StringVarP(&o.File, "file", "", "",
cmd.Flags().StringVarP(&o.File, "file", "", "",
"file to write the default report output to (default is STDOUT)")
cmd.PersistentFlags().StringVarP(&o.Platform, "platform", "", "",
cmd.Flags().StringVarP(&o.OutputTemplatePath, "template", "t", "",
"specify the path to a Go template file")
cmd.Flags().StringVarP(&o.Platform, "platform", "", "",
"an optional platform specifier for container image sources (e.g. 'linux/arm64', 'linux/arm64/v8', 'arm64', 'linux')")
cmd.PersistentFlags().StringVarP(&o.Host, "host", "H", "",
cmd.Flags().StringVarP(&o.Host, "host", "H", "",
"the hostname or URL of the Anchore Enterprise instance to upload to")
cmd.PersistentFlags().StringVarP(&o.Username, "username", "u", "",
cmd.Flags().StringVarP(&o.Username, "username", "u", "",
"the username to authenticate against Anchore Enterprise")
cmd.PersistentFlags().StringVarP(&o.Password, "password", "p", "",
cmd.Flags().StringVarP(&o.Password, "password", "p", "",
"the password to authenticate against Anchore Enterprise")
cmd.PersistentFlags().StringVarP(&o.Dockerfile, "dockerfile", "d", "",
cmd.Flags().StringVarP(&o.Dockerfile, "dockerfile", "d", "",
"include dockerfile for upload to Anchore Enterprise")
cmd.PersistentFlags().StringArrayVarP(&o.Exclude, "exclude", "", nil,
cmd.Flags().StringArrayVarP(&o.Exclude, "exclude", "", nil,
"exclude paths from being scanned using a glob expression")
cmd.PersistentFlags().BoolVarP(&o.OverwriteExistingImage, "overwrite-existing-image", "", false,
cmd.Flags().StringArrayVarP(&o.Catalogers, "catalogers", "", nil,
"enable one or more package catalogers")
cmd.Flags().BoolVarP(&o.OverwriteExistingImage, "overwrite-existing-image", "", false,
"overwrite an existing image during the upload to Anchore Enterprise")
cmd.PersistentFlags().UintVarP(&o.ImportTimeout, "import-timeout", "", 30,
cmd.Flags().UintVarP(&o.ImportTimeout, "import-timeout", "", 30,
"set a timeout duration (in seconds) for the upload to Anchore Enterprise")
return bindPackageConfigOptions(cmd.PersistentFlags(), v)
return bindPackageConfigOptions(cmd.Flags(), v)
}
func bindPackageConfigOptions(flags *pflag.FlagSet, v *viper.Viper) error {
@@ -80,10 +88,18 @@ func bindPackageConfigOptions(flags *pflag.FlagSet, v *viper.Viper) error {
return err
}
if err := v.BindPFlag("catalogers", flags.Lookup("catalogers")); err != nil {
return err
}
if err := v.BindPFlag("output", flags.Lookup("output")); err != nil {
return err
}
if err := v.BindPFlag("output-template-path", flags.Lookup("template")); err != nil {
return err
}
if err := v.BindPFlag("platform", flags.Lookup("platform")); err != nil {
return err
}
+9 -3
View File
@@ -5,6 +5,7 @@ import (
"strings"
"github.com/anchore/syft/internal/formats/table"
"github.com/anchore/syft/internal/formats/template"
"github.com/anchore/syft/syft"
"github.com/anchore/syft/syft/sbom"
"github.com/hashicorp/go-multierror"
@@ -12,8 +13,8 @@ import (
// makeWriter creates a sbom.Writer for output or returns an error. this will either return a valid writer
// or an error but neither both and if there is no error, sbom.Writer.Close() should be called
func MakeWriter(outputs []string, defaultFile string) (sbom.Writer, error) {
outputOptions, err := parseOutputs(outputs, defaultFile)
func MakeWriter(outputs []string, defaultFile, templateFilePath string) (sbom.Writer, error) {
outputOptions, err := parseOutputs(outputs, defaultFile, templateFilePath)
if err != nil {
return nil, err
}
@@ -27,7 +28,7 @@ func MakeWriter(outputs []string, defaultFile string) (sbom.Writer, error) {
}
// parseOptions utility to parse command-line option strings and retain the existing behavior of default format and file
func parseOutputs(outputs []string, defaultFile string) (out []sbom.WriterOption, errs error) {
func parseOutputs(outputs []string, defaultFile, templateFilePath string) (out []sbom.WriterOption, errs error) {
// always should have one option -- we generally get the default of "table", but just make sure
if len(outputs) == 0 {
outputs = append(outputs, string(table.ID))
@@ -56,6 +57,11 @@ func parseOutputs(outputs []string, defaultFile string) (out []sbom.WriterOption
continue
}
if tmpl, ok := format.(template.OutputFormat); ok {
tmpl.SetTemplatePath(templateFilePath)
format = tmpl
}
out = append(out, sbom.NewWriterOption(format, file))
}
return out, errs
+1 -1
View File
@@ -28,7 +28,7 @@ func TestIsSupportedFormat(t *testing.T) {
}
for _, tt := range tests {
_, err := MakeWriter(tt.outputs, "")
_, err := MakeWriter(tt.outputs, "", "")
tt.wantErr(t, err)
}
}
+7 -6
View File
@@ -14,12 +14,13 @@ import (
const (
packagesExample = ` {{.appName}} {{.command}} alpine:latest a summary of discovered packages
{{.appName}} {{.command}} alpine:latest -o json show all possible cataloging details
{{.appName}} {{.command}} alpine:latest -o cyclonedx show a CycloneDX formatted SBOM
{{.appName}} {{.command}} alpine:latest -o cyclonedx-json show a CycloneDX JSON formatted SBOM
{{.appName}} {{.command}} alpine:latest -o spdx show a SPDX 2.2 Tag-Value formatted SBOM
{{.appName}} {{.command}} alpine:latest -o spdx-json show a SPDX 2.2 JSON formatted SBOM
{{.appName}} {{.command}} alpine:latest -vv show verbose debug information
{{.appName}} {{.command}} alpine:latest -o json show all possible cataloging details
{{.appName}} {{.command}} alpine:latest -o cyclonedx show a CycloneDX formatted SBOM
{{.appName}} {{.command}} alpine:latest -o cyclonedx-json show a CycloneDX JSON formatted SBOM
{{.appName}} {{.command}} alpine:latest -o spdx show a SPDX 2.2 Tag-Value formatted SBOM
{{.appName}} {{.command}} alpine:latest -o spdx-json show a SPDX 2.2 JSON formatted SBOM
{{.appName}} {{.command}} alpine:latest -vv show verbose debug information
{{.appName}} {{.command}} alpine:latest -o template -t my_format.tmpl show a SBOM formatted according to given template file
Supports the following image sources:
{{.appName}} {{.command}} yourrepo/yourimage:tag defaults to using images from a Docker daemon. If Docker is not present, the image is pulled directly from the registry.
+23 -1
View File
@@ -13,6 +13,7 @@ import (
"github.com/anchore/syft/internal/anchore"
"github.com/anchore/syft/internal/bus"
"github.com/anchore/syft/internal/config"
"github.com/anchore/syft/internal/formats/template"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/internal/ui"
"github.com/anchore/syft/internal/version"
@@ -25,7 +26,12 @@ import (
)
func Run(ctx context.Context, app *config.Application, args []string) error {
writer, err := options.MakeWriter(app.Outputs, app.File)
err := validateOutputOptions(app)
if err != nil {
return err
}
writer, err := options.MakeWriter(app.Outputs, app.File, app.OutputTemplatePath)
if err != nil {
return err
}
@@ -185,3 +191,19 @@ func runPackageSbomUpload(src *source.Source, s sbom.SBOM, app *config.Applicati
return nil
}
func validateOutputOptions(app *config.Application) error {
var usesTemplateOutput bool
for _, o := range app.Outputs {
if o == template.ID.String() {
usesTemplateOutput = true
break
}
}
if usesTemplateOutput && app.OutputTemplatePath == "" {
return fmt.Errorf(`must specify path to template file when using "template" output format`)
}
return nil
}
+1 -1
View File
@@ -13,7 +13,7 @@ import (
const powerUserExample = ` {{.appName}} {{.command}} <image>
DEPRECATED - THIS COMMAND WILL BE REMOVED in v1.0.0
Only image sources are supported (e.g. docker: , podman: , docker-archive: , oci: , etc.), the directory source (dir:) is not supported.
Only image sources are supported (e.g. docker: , podman: , docker-archive: , oci: , etc.), the directory source (dir:) is not supported, template outputs are not supported.
All behavior is controlled via application configuration and environment variables (see https://github.com/anchore/syft#configuration)
`
+10 -2
View File
@@ -13,7 +13,7 @@ require (
github.com/anchore/go-testutils v0.0.0-20200925183923-d5f45b0d3c04
github.com/anchore/go-version v1.2.2-0.20200701162849-18adb9c92b9b
github.com/anchore/packageurl-go v0.1.1-0.20220428202044-a072fa3cb6d7
github.com/anchore/stereoscope v0.0.0-20220518185348-c97a3c6ffc67
github.com/anchore/stereoscope v0.0.0-20220616165231-b0fd10fdee06
github.com/antihax/optional v1.0.0
github.com/bmatcuk/doublestar/v4 v4.0.2
github.com/dustin/go-humanize v1.0.0
@@ -54,6 +54,7 @@ require (
)
require (
github.com/Masterminds/sprig/v3 v3.2.2
github.com/docker/docker v20.10.12+incompatible
github.com/google/go-containerregistry v0.8.1-0.20220209165246-a44adc326839
github.com/in-toto/in-toto-golang v0.3.4-0.20211211042327-af1f9fb822bf
@@ -61,6 +62,7 @@ require (
github.com/sigstore/cosign v1.9.0
github.com/sigstore/rekor v0.4.1-0.20220114213500-23f583409af3
github.com/sigstore/sigstore v1.2.1-0.20220424143412-3d41663116d5
github.com/vbatts/go-mtree v0.5.0
)
require (
@@ -78,6 +80,8 @@ require (
github.com/Azure/go-autorest/autorest/date v0.3.0 // indirect
github.com/Azure/go-autorest/logger v0.2.1 // indirect
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/Masterminds/goutils v1.1.1 // indirect
github.com/Masterminds/semver/v3 v3.1.1 // indirect
github.com/Microsoft/go-winio v0.5.2 // indirect
github.com/PaesslerAG/gval v1.0.0 // indirect
github.com/PaesslerAG/jsonpath v0.1.1 // indirect
@@ -109,7 +113,7 @@ require (
github.com/cncf/udpa/go v0.0.0-20210930031921-04548b0d99d4 // indirect
github.com/cncf/xds/go v0.0.0-20211130200136-a8f946100490 // indirect
github.com/common-nighthawk/go-figure v0.0.0-20210622060536-734e95fb86be // indirect
github.com/containerd/containerd v1.5.10 // indirect
github.com/containerd/containerd v1.5.13 // indirect
github.com/containerd/stargz-snapshotter/estargz v0.10.1 // indirect
github.com/coreos/go-oidc/v3 v3.1.0 // indirect
github.com/coreos/go-semver v0.3.0 // indirect
@@ -174,6 +178,7 @@ require (
github.com/hashicorp/go-retryablehttp v0.7.1 // indirect
github.com/hashicorp/golang-lru v0.5.4 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/huandu/xstrings v1.3.2 // indirect
github.com/iancoleman/orderedmap v0.0.0-20190318233801-ac98e3ecb4b0 // indirect
github.com/imdario/mergo v0.3.12 // indirect
github.com/inconshreveable/mousetrap v1.0.0 // indirect
@@ -196,6 +201,8 @@ require (
github.com/mattn/go-runewidth v0.0.13 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
github.com/miekg/pkcs11 v1.1.1 // indirect
github.com/mitchellh/copystructure v1.2.0 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/nwaples/rardecode v1.1.0 // indirect
@@ -217,6 +224,7 @@ require (
github.com/secure-systems-lab/go-securesystemslib v0.4.0 // indirect
github.com/segmentio/ksuid v1.0.4 // indirect
github.com/shibumi/go-pathspec v1.3.0 // indirect
github.com/shopspring/decimal v1.2.0 // indirect
github.com/sigstore/fulcio v0.1.2-0.20220114150912-86a2036f9bc7 // indirect
github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966 // indirect
github.com/soheilhy/cmux v0.1.5 // indirect
+14 -4
View File
@@ -186,14 +186,17 @@ github.com/GoogleCloudPlatform/cloudsql-proxy v0.0.0-20191009163259-e802c2cb94ae
github.com/GoogleCloudPlatform/cloudsql-proxy v1.27.0/go.mod h1:bn9iHmAjogMoIPkqBGyJ9R1m9cXGCjBE/cuhBs3oEsQ=
github.com/Knetic/govaluate v3.0.1-0.20171022003610-9aa49832a739+incompatible/go.mod h1:r7JcOSlj0wfOMncg0iLm8Leh48TZaKVeNIfJntJ2wa0=
github.com/Masterminds/goutils v1.1.0/go.mod h1:8cTjp+g8YejhMuvIA5y2vz3BpJxksy863GQaJW2MFNU=
github.com/Masterminds/goutils v1.1.1 h1:5nUrii3FMTL5diU80unEVvNevw1nH4+ZV4DSLVJLSYI=
github.com/Masterminds/goutils v1.1.1/go.mod h1:8cTjp+g8YejhMuvIA5y2vz3BpJxksy863GQaJW2MFNU=
github.com/Masterminds/semver v1.4.2/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF078ddwwvV3Y=
github.com/Masterminds/semver v1.5.0/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF078ddwwvV3Y=
github.com/Masterminds/semver/v3 v3.0.3/go.mod h1:VPu/7SZ7ePZ3QOrcuXROw5FAcLl4a0cBrbBpGY/8hQs=
github.com/Masterminds/semver/v3 v3.1.0/go.mod h1:VPu/7SZ7ePZ3QOrcuXROw5FAcLl4a0cBrbBpGY/8hQs=
github.com/Masterminds/semver/v3 v3.1.1 h1:hLg3sBzpNErnxhQtUy/mmLR2I9foDujNK030IGemrRc=
github.com/Masterminds/semver/v3 v3.1.1/go.mod h1:VPu/7SZ7ePZ3QOrcuXROw5FAcLl4a0cBrbBpGY/8hQs=
github.com/Masterminds/sprig v2.15.0+incompatible/go.mod h1:y6hNFY5UBTIWBxnzTeuNhlNS5hqE0NB0E6fgfo2Br3o=
github.com/Masterminds/sprig v2.22.0+incompatible/go.mod h1:y6hNFY5UBTIWBxnzTeuNhlNS5hqE0NB0E6fgfo2Br3o=
github.com/Masterminds/sprig/v3 v3.2.2 h1:17jRggJu518dr3QaafizSXOjKYp94wKfABxUmyxvxX8=
github.com/Masterminds/sprig/v3 v3.2.2/go.mod h1:UoaO7Yp8KlPnJIYWTFkMaqPUYKTfGFPhxNuwnnxkKlk=
github.com/Microsoft/go-winio v0.4.11/go.mod h1:VhR8bwka0BXejwEJY73c50VrPtXAaKcyvVC4A4RozmA=
github.com/Microsoft/go-winio v0.4.14/go.mod h1:qXqCSQ3Xa7+6tgxaGTIe4Kpcdsi+P8jBhyzoq1bpyYA=
@@ -215,6 +218,7 @@ github.com/Microsoft/hcsshim v0.8.15/go.mod h1:x38A4YbHbdxJtc0sF6oIz+RG0npwSCAvn
github.com/Microsoft/hcsshim v0.8.16/go.mod h1:o5/SZqmR7x9JNKsW3pu+nqHm0MF8vbA+VxGOoXdC600=
github.com/Microsoft/hcsshim v0.8.21/go.mod h1:+w2gRZ5ReXQhFOrvSQeNfhrYB/dg3oDwTOcER2fw4I4=
github.com/Microsoft/hcsshim v0.8.23/go.mod h1:4zegtUJth7lAvFyc6cH2gGQ5B3OFQim01nnU2M8jKDg=
github.com/Microsoft/hcsshim v0.8.24/go.mod h1:4zegtUJth7lAvFyc6cH2gGQ5B3OFQim01nnU2M8jKDg=
github.com/Microsoft/hcsshim/test v0.0.0-20201218223536-d3e5debf77da/go.mod h1:5hlzMzRKMLyo42nCZ9oml8AdTlq/0cvIaBv6tK1RehU=
github.com/Microsoft/hcsshim/test v0.0.0-20210227013316-43a75bb4edd3/go.mod h1:mw7qgWloBUl75W/gVH3cQszUg1+gUITj7D6NY7ywVnY=
github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ=
@@ -269,8 +273,8 @@ github.com/anchore/go-version v1.2.2-0.20200701162849-18adb9c92b9b h1:e1bmaoJfZV
github.com/anchore/go-version v1.2.2-0.20200701162849-18adb9c92b9b/go.mod h1:Bkc+JYWjMCF8OyZ340IMSIi2Ebf3uwByOk6ho4wne1E=
github.com/anchore/packageurl-go v0.1.1-0.20220428202044-a072fa3cb6d7 h1:kDrYkTSM9uIxaX/P9s0F4nKYNM+hnSgLJdLpqvsaQ/g=
github.com/anchore/packageurl-go v0.1.1-0.20220428202044-a072fa3cb6d7/go.mod h1:Blo6OgJNiYF41ufcgHKkbCKF2MDOMlrqhXv/ij6ocR4=
github.com/anchore/stereoscope v0.0.0-20220518185348-c97a3c6ffc67 h1:nbcYgEEv9CLnuKg/8ExvXDiEpCA9pwZcyyraZyBE+aw=
github.com/anchore/stereoscope v0.0.0-20220518185348-c97a3c6ffc67/go.mod h1:yoCLUZY0k/pYLNIy0L80p2Ko0PKVNXm8rHtgxp4OiSc=
github.com/anchore/stereoscope v0.0.0-20220616165231-b0fd10fdee06 h1:TSRA7gtuia3eyleTO3t7iPU+9xHbdSaufoUFNQUwUXo=
github.com/anchore/stereoscope v0.0.0-20220616165231-b0fd10fdee06/go.mod h1:sai2ZjAtT/y1GRQBDRbynhdhnQcGWBvVcv8CN3hTWmI=
github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8=
github.com/andybalholm/brotli v1.0.1/go.mod h1:loMXtMfwqflxFJPmdbJO0a3KNoPuLBgiu3qAvBg8x/Y=
github.com/andybalholm/brotli v1.0.2/go.mod h1:loMXtMfwqflxFJPmdbJO0a3KNoPuLBgiu3qAvBg8x/Y=
@@ -505,6 +509,7 @@ github.com/containerd/cgroups v0.0.0-20200710171044-318312a37340/go.mod h1:s5q4S
github.com/containerd/cgroups v0.0.0-20200824123100-0b889c03f102/go.mod h1:s5q4SojHctfxANBDvMeIaIovkq29IP48TKAxnhYRxvo=
github.com/containerd/cgroups v0.0.0-20210114181951-8a68de567b68/go.mod h1:ZJeTFisyysqgcCdecO57Dj79RfL0LNeGiFUqLYQRYLE=
github.com/containerd/cgroups v1.0.1/go.mod h1:0SJrPIenamHDcZhEcJMNBB85rHcUsw4f25ZfBiPYRkU=
github.com/containerd/cgroups v1.0.3/go.mod h1:/ofk34relqNjSGyqPrmEULrO4Sc8LJhvJmWbUCUKqj8=
github.com/containerd/console v0.0.0-20180822173158-c12b1e7919c1/go.mod h1:Tj/on1eG8kiEhd0+fhSDzsPAFESxzBBvdyEgyryXffw=
github.com/containerd/console v0.0.0-20181022165439-0650fd9eeb50/go.mod h1:Tj/on1eG8kiEhd0+fhSDzsPAFESxzBBvdyEgyryXffw=
github.com/containerd/console v0.0.0-20191206165004-02ecf6a7291e/go.mod h1:8Pf4gM6VEbTNRIT26AyyU7hxdQU3MvAvxVI0sc00XBE=
@@ -527,8 +532,8 @@ github.com/containerd/containerd v1.5.1/go.mod h1:0DOxVqwDy2iZvrZp2JUx/E+hS0UNTV
github.com/containerd/containerd v1.5.7/go.mod h1:gyvv6+ugqY25TiXxcZC3L5yOeYgEw0QMhscqVp1AR9c=
github.com/containerd/containerd v1.5.8/go.mod h1:YdFSv5bTFLpG2HIYmfqDpSYYTDX+mc5qtSuYx1YUb/s=
github.com/containerd/containerd v1.5.9/go.mod h1:fvQqCfadDGga5HZyn3j4+dx56qj2I9YwBrlSdalvJYQ=
github.com/containerd/containerd v1.5.10 h1:3cQ2uRVCkJVcx5VombsE7105Gl9Wrl7ORAO3+4+ogf4=
github.com/containerd/containerd v1.5.10/go.mod h1:fvQqCfadDGga5HZyn3j4+dx56qj2I9YwBrlSdalvJYQ=
github.com/containerd/containerd v1.5.13 h1:XqvKw9i4P7/mFrC3TSM7yV5cwFZ9avXe6M3YANKnzEE=
github.com/containerd/containerd v1.5.13/go.mod h1:3AlCrzKROjIuP3JALsY14n8YtntaUDBu7vek+rPN5Vc=
github.com/containerd/continuity v0.0.0-20190426062206-aaeac12a7ffc/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y=
github.com/containerd/continuity v0.0.0-20190815185530-f2a389ac0a02/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y=
github.com/containerd/continuity v0.0.0-20191127005431-f65d91d395eb/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y=
@@ -1300,6 +1305,7 @@ github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpO
github.com/huandu/xstrings v1.0.0/go.mod h1:4qWG/gcEcfX4z/mBDHJ++3ReCw9ibxbsNJbcucJdbSo=
github.com/huandu/xstrings v1.2.0/go.mod h1:DvyZB1rfVYsBIigL8HwpZgxHwXozlTgGqn63UyNX5k4=
github.com/huandu/xstrings v1.3.1/go.mod h1:y5/lhBue+AyNmUVz9RLU9xbLR0o4KIIExikq4ovT0aE=
github.com/huandu/xstrings v1.3.2 h1:L18LIDzqlW6xN2rEkpdV8+oL/IXWJ1APd+vsdYy4Wdw=
github.com/huandu/xstrings v1.3.2/go.mod h1:y5/lhBue+AyNmUVz9RLU9xbLR0o4KIIExikq4ovT0aE=
github.com/hudl/fargo v1.3.0/go.mod h1:y3CKSmjA+wD2gak7sUSXTAoopbhU08POFhmITJgmKTg=
github.com/iancoleman/orderedmap v0.0.0-20190318233801-ac98e3ecb4b0 h1:i462o439ZjprVSFSZLZxcsoAe592sZB1rci2Z8j4wdk=
@@ -1854,6 +1860,7 @@ github.com/shazow/go-diff v0.0.0-20160112020656-b6b7b6733b8c/go.mod h1:/PevMnwAx
github.com/shibumi/go-pathspec v1.3.0 h1:QUyMZhFo0Md5B8zV8x2tesohbb5kfbpTi9rBnKh5dkI=
github.com/shibumi/go-pathspec v1.3.0/go.mod h1:Xutfslp817l2I1cZvgcfeMQJG5QnU2lh5tVaaMCl3jE=
github.com/shirou/gopsutil/v3 v3.21.10/go.mod h1:t75NhzCZ/dYyPQjyQmrAYP6c8+LCdFANeBMdLPCNnew=
github.com/shopspring/decimal v1.2.0 h1:abSATXmQEYyShuxI4/vyW3tV1MrKAJzCZ/0zLUXYbsQ=
github.com/shopspring/decimal v1.2.0/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o=
github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e/go.mod h1:TDJrrUr11Vxrven61rcy3hJMUqaf/CLWYhHNPmT14Lk=
github.com/shurcooL/go-goon v0.0.0-20170922171312-37c2f522c041/go.mod h1:N5mDOmsrJOB+vfqUK+7DmDyjhSLIIBnXo9lvZJj3MWQ=
@@ -1871,6 +1878,7 @@ github.com/sigstore/sigstore v1.2.1-0.20220424143412-3d41663116d5/go.mod h1:OvpZ
github.com/sirupsen/logrus v1.0.4-0.20170822132746-89742aefa4b2/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
github.com/sirupsen/logrus v1.0.6/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.3.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
@@ -2040,6 +2048,8 @@ github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyC
github.com/valyala/fasthttp v1.30.0/go.mod h1:2rsYD01CKFrjjsvFxx75KlEUNpWNBY9JWD3K/7o2Cus=
github.com/valyala/quicktemplate v1.7.0/go.mod h1:sqKJnoaOF88V07vkO+9FL8fb9uZg/VPSJnLYn+LmLk8=
github.com/valyala/tcplisten v1.0.0/go.mod h1:T0xQ8SeCZGxckz9qRXTfG43PvQ/mcWh7FwZEA7Ioqkc=
github.com/vbatts/go-mtree v0.5.0 h1:dM+5XZdqH0j9CSZeerhoN/tAySdwnmevaZHO1XGW2Vc=
github.com/vbatts/go-mtree v0.5.0/go.mod h1:7JbaNHyBMng+RP8C3Q4E+4Ca8JnGQA2R/MB+jb4tSOk=
github.com/vbatts/tar-split v0.11.2 h1:Via6XqJr0hceW4wff3QRzD5gAk/tatMw/4ZA7cTlIME=
github.com/vbatts/tar-split v0.11.2/go.mod h1:vV3ZuO2yWSVsz+pfFzDG/upWH1JhjOiEaWq6kXyQ3VI=
github.com/vektah/gqlparser v1.1.2/go.mod h1:1ycwN7Ij5njmMkPPAOaRFY4rET2Enx7IkVv3vaXspKw=
+27
View File
@@ -5,8 +5,11 @@ import (
"fmt"
"path"
"reflect"
"sort"
"strings"
"github.com/anchore/syft/syft/pkg/cataloger"
"github.com/sirupsen/logrus"
"github.com/adrg/xdg"
@@ -38,11 +41,13 @@ type Application struct {
// -q, indicates to not show any status output to stderr (ETUI or logging UI)
Quiet bool `yaml:"quiet" json:"quiet" mapstructure:"quiet"`
Outputs []string `yaml:"output" json:"output" mapstructure:"output"` // -o, the format to use for output
OutputTemplatePath string `yaml:"output-template-path" json:"output-template-path" mapstructure:"output-template-path"` // -t template file to use for output
File string `yaml:"file" json:"file" mapstructure:"file"` // --file, the file to write report output to
CheckForAppUpdate bool `yaml:"check-for-app-update" json:"check-for-app-update" mapstructure:"check-for-app-update"` // whether to check for an application update on start up or not
Anchore anchore `yaml:"anchore" json:"anchore" mapstructure:"anchore"` // options for interacting with Anchore Engine/Enterprise
Dev development `yaml:"dev" json:"dev" mapstructure:"dev"`
Log logging `yaml:"log" json:"log" mapstructure:"log"` // all logging-related options
Catalogers []string `yaml:"catalogers" json:"catalogers" mapstructure:"catalogers"`
Package pkg `yaml:"package" json:"package" mapstructure:"package"`
FileMetadata FileMetadata `yaml:"file-metadata" json:"file-metadata" mapstructure:"file-metadata"`
FileClassification fileClassification `yaml:"file-classification" json:"file-classification" mapstructure:"file-classification"`
@@ -54,6 +59,17 @@ type Application struct {
Platform string `yaml:"platform" json:"platform" mapstructure:"platform"`
}
func (cfg Application) ToCatalogerConfig() cataloger.Config {
return cataloger.Config{
Search: cataloger.SearchConfig{
IncludeIndexedArchives: cfg.Package.SearchIndexedArchives,
IncludeUnindexedArchives: cfg.Package.SearchUnindexedArchives,
Scope: cfg.Package.Cataloger.ScopeOpt,
},
Catalogers: cfg.Catalogers,
}
}
func (cfg *Application) LoadAllValues(v *viper.Viper, configPath string) error {
// priority order: viper.Set, flag, env, config, kv, defaults
// flags have already been loaded into viper by command construction
@@ -85,6 +101,16 @@ func (cfg *Application) LoadAllValues(v *viper.Viper, configPath string) error {
}
func (cfg *Application) parseConfigValues() error {
// parse options on this struct
var catalogers []string
for _, c := range cfg.Catalogers {
for _, f := range strings.Split(c, ",") {
catalogers = append(catalogers, strings.TrimSpace(f))
}
}
sort.Strings(catalogers)
cfg.Catalogers = catalogers
// parse application config options
for _, optionFn := range []func() error{
cfg.parseUploadOptions,
@@ -172,6 +198,7 @@ func loadDefaultValues(v *viper.Viper) {
// set the default values for primitive fields in this struct
v.SetDefault("quiet", false)
v.SetDefault("check-for-app-update", true)
v.SetDefault("catalogers", nil)
// for each field in the configuration struct, see if the field implements the defaultValueLoader interface and invoke it if it does
value := reflect.ValueOf(Application{})
-10
View File
@@ -21,13 +21,3 @@ func (cfg pkg) loadDefaultValues(v *viper.Viper) {
func (cfg *pkg) parseConfigValues() error {
return cfg.Cataloger.parseConfigValues()
}
func (cfg pkg) ToConfig() cataloger.Config {
return cataloger.Config{
Search: cataloger.SearchConfig{
IncludeIndexedArchives: cfg.SearchIndexedArchives,
IncludeUnindexedArchives: cfg.SearchUnindexedArchives,
Scope: cfg.Cataloger.ScopeOpt,
},
}
}
+1 -1
View File
@@ -6,5 +6,5 @@ const (
// JSONSchemaVersion is the current schema version output by the JSON encoder
// This is roughly following the "SchemaVer" guidelines for versioning the JSON schema. Please see schema/json/README.md for details on how to increment.
JSONSchemaVersion = "3.2.4"
JSONSchemaVersion = "3.3.0"
)
@@ -3,5 +3,10 @@ set -eux
# $1 —— absolute path to destination file, should end with .zip, ideally
# $2 —— absolute path to directory from which to add entries to the archive
# $3 —— if files should be zip64 or not
pushd "$2" && find . -print | zip "$1" -@ && popd
if [[$3]]; then
pushd "$2" && find . -print | zip -fz "$1" -@ && popd
else
pushd "$2" && find . -print | zip "$1" -@ && popd
fi
+10 -6
View File
@@ -20,16 +20,20 @@ var expectedZipArchiveEntries = []string{
}
// createZipArchive creates a new ZIP archive file at destinationArchivePath based on the directory found at
// sourceDirPath.
func createZipArchive(t testing.TB, sourceDirPath, destinationArchivePath string) {
// sourceDirPath. It forces a zip64 archive if zip64 is "0".
func createZipArchive(t testing.TB, sourceDirPath, destinationArchivePath string, zip64 bool) {
t.Helper()
cwd, err := os.Getwd()
if err != nil {
t.Fatalf("unable to get cwd: %+v", err)
}
zip64Arg := "0"
if zip64 {
zip64Arg = "1"
}
cmd := exec.Command("./generate-zip-fixture-from-source-dir.sh", destinationArchivePath, path.Base(sourceDirPath))
cmd := exec.Command("./generate-zip-fixture-from-source-dir.sh", destinationArchivePath, path.Base(sourceDirPath), zip64Arg)
cmd.Dir = filepath.Join(cwd, "test-fixtures")
if err := cmd.Start(); err != nil {
@@ -66,7 +70,7 @@ func assertNoError(t testing.TB, fn func() error) func() {
// which should be called (typically deferred) by the caller, the path of the created zip archive, and an error,
// which should trigger a fatal test failure in the consuming test. The returned cleanup function will never be nil
// (even if there's an error), and it should always be called.
func setupZipFileTest(t testing.TB, sourceDirPath string) string {
func setupZipFileTest(t testing.TB, sourceDirPath string, zip64 bool) string {
t.Helper()
archivePrefix, err := ioutil.TempFile("", "syft-ziputil-archive-TEST-")
@@ -84,7 +88,7 @@ func setupZipFileTest(t testing.TB, sourceDirPath string) string {
destinationArchiveFilePath := archivePrefix.Name() + ".zip"
t.Logf("archive path: %s", destinationArchiveFilePath)
createZipArchive(t, sourceDirPath, destinationArchiveFilePath)
createZipArchive(t, sourceDirPath, destinationArchiveFilePath, zip64)
t.Cleanup(
assertNoError(t,
@@ -109,7 +113,7 @@ func ensureNestedZipExists(t *testing.T, sourceDirPath string) error {
t.Helper()
nestedArchiveFilePath := path.Join(sourceDirPath, "nested.zip")
createZipArchive(t, sourceDirPath, nestedArchiveFilePath)
createZipArchive(t, sourceDirPath, nestedArchiveFilePath, false)
return nil
}
+37 -2
View File
@@ -22,7 +22,42 @@ func TestNewZipFileManifest(t *testing.T) {
t.Fatal(err)
}
archiveFilePath := setupZipFileTest(t, sourceDirPath)
archiveFilePath := setupZipFileTest(t, sourceDirPath, false)
actual, err := NewZipFileManifest(archiveFilePath)
if err != nil {
t.Fatalf("unable to extract from unzip archive: %+v", err)
}
if len(expectedZipArchiveEntries) != len(actual) {
t.Fatalf("mismatched manifest: %d != %d", len(actual), len(expectedZipArchiveEntries))
}
for _, e := range expectedZipArchiveEntries {
_, ok := actual[e]
if !ok {
t.Errorf("missing path: %s", e)
}
}
if t.Failed() {
b, err := json.MarshalIndent(actual, "", " ")
if err != nil {
t.Fatalf("can't show results: %+v", err)
}
t.Errorf("full result: %s", string(b))
}
}
func TestNewZip64FileManifest(t *testing.T) {
cwd, err := os.Getwd()
if err != nil {
t.Fatal(err)
}
sourceDirPath := path.Join(cwd, "test-fixtures", "zip-source")
archiveFilePath := setupZipFileTest(t, sourceDirPath, true)
actual, err := NewZipFileManifest(archiveFilePath)
if err != nil {
@@ -62,7 +97,7 @@ func TestZipFileManifest_GlobMatch(t *testing.T) {
t.Fatal(err)
}
archiveFilePath := setupZipFileTest(t, sourceDirPath)
archiveFilePath := setupZipFileTest(t, sourceDirPath, false)
z, err := NewZipFileManifest(archiveFilePath)
if err != nil {
+2 -2
View File
@@ -47,7 +47,7 @@ func TestUnzipToDir(t *testing.T) {
goldenRootDir := filepath.Join(cwd, "test-fixtures")
sourceDirPath := path.Join(goldenRootDir, "zip-source")
archiveFilePath := setupZipFileTest(t, sourceDirPath)
archiveFilePath := setupZipFileTest(t, sourceDirPath, false)
unzipDestinationDir, err := ioutil.TempDir("", "syft-ziputil-contents-TEST-")
t.Cleanup(assertNoError(t, func() error {
@@ -227,7 +227,7 @@ func prepZipSourceFixture(t testing.TB) string {
t.Logf("archive path: %s", archivePath)
createZipArchive(t, "zip-source", archivePrefix.Name())
createZipArchive(t, "zip-source", archivePrefix.Name(), false)
return archivePath
}
+75 -2
View File
@@ -3,6 +3,7 @@ package file
import (
"archive/zip"
"encoding/binary"
"errors"
"fmt"
"io"
"os"
@@ -13,7 +14,13 @@ import (
// - https://github.com/golang/go/blob/go1.16.4/src/archive/zip/reader.go
// findArchiveStartOffset is derived from the same stdlib utils, specifically the readDirectoryEnd function.
const directoryEndLen = 22
const (
directoryEndLen = 22
directory64LocLen = 20
directory64EndLen = 56
directory64LocSignature = 0x07064b50
directory64EndSignature = 0x06064b50
)
// ZipReadCloser is a drop-in replacement for zip.ReadCloser (from zip.OpenReader) that additionally considers zips
// that have bytes prefixed to the front of the archive (common with self-extracting jars).
@@ -72,6 +79,12 @@ func (b *readBuf) uint32() uint32 {
return v
}
func (b *readBuf) uint64() uint64 {
v := binary.LittleEndian.Uint64(*b)
*b = (*b)[8:]
return v
}
type directoryEnd struct {
diskNbr uint32 // unused
dirDiskNbr uint32 // unused
@@ -82,6 +95,7 @@ type directoryEnd struct {
}
// note: this is derived from readDirectoryEnd within the archive/zip package
// nolint:gocognit
func findArchiveStartOffset(r io.ReaderAt, size int64) (startOfArchive uint64, err error) {
// look for directoryEndSignature in the last 1k, then in the last 65k
var buf []byte
@@ -120,13 +134,22 @@ func findArchiveStartOffset(r io.ReaderAt, size int64) (startOfArchive uint64, e
directoryOffset: uint64(b.uint32()),
}
// Calculate where the zip data actually begins
startOfArchive = uint64(directoryEndOffset) - d.directorySize - d.directoryOffset
// These values mean that the file can be a zip64 file
if d.directoryRecords == 0xffff || d.directorySize == 0xffff || d.directoryOffset == 0xffffffff {
p, err := findDirectory64End(r, directoryEndOffset)
if err == nil && p >= 0 {
directoryEndOffset = p
err = readDirectory64End(r, p, d)
}
if err != nil {
return 0, err
}
startOfArchive = 0 // Prefixed data not supported
}
startOfArchive = uint64(directoryEndOffset) - d.directorySize - d.directoryOffset
// Make sure directoryOffset points to somewhere in our file.
if o := int64(d.directoryOffset); o < 0 || o >= size {
return 0, zip.ErrFormat
@@ -134,6 +157,56 @@ func findArchiveStartOffset(r io.ReaderAt, size int64) (startOfArchive uint64, e
return startOfArchive, nil
}
// findDirectory64End tries to read the zip64 locator just before the
// directory end and returns the offset of the zip64 directory end if
// found.
func findDirectory64End(r io.ReaderAt, directoryEndOffset int64) (int64, error) {
locOffset := directoryEndOffset - directory64LocLen
if locOffset < 0 {
return -1, nil // no need to look for a header outside the file
}
buf := make([]byte, directory64LocLen)
if _, err := r.ReadAt(buf, locOffset); err != nil {
return -1, err
}
b := readBuf(buf)
if sig := b.uint32(); sig != directory64LocSignature {
return -1, nil
}
if b.uint32() != 0 { // number of the disk with the start of the zip64 end of central directory
return -1, nil // the file is not a valid zip64-file
}
p := b.uint64() // relative offset of the zip64 end of central directory record
if b.uint32() != 1 { // total number of disks
return -1, nil // the file is not a valid zip64-file
}
return int64(p), nil
}
// readDirectory64End reads the zip64 directory end and updates the
// directory end with the zip64 directory end values.
func readDirectory64End(r io.ReaderAt, offset int64, d *directoryEnd) (err error) {
buf := make([]byte, directory64EndLen)
if _, err := r.ReadAt(buf, offset); err != nil {
return err
}
b := readBuf(buf)
if sig := b.uint32(); sig != directory64EndSignature {
return errors.New("could not read directory64End")
}
b = b[12:] // skip dir size, version and version needed (uint64 + 2x uint16)
d.diskNbr = b.uint32() // number of this disk
d.dirDiskNbr = b.uint32() // number of the disk with the start of the central directory
d.dirRecordsThisDisk = b.uint64() // total number of entries in the central directory on this disk
d.directoryRecords = b.uint64() // total number of entries in the central directory
d.directorySize = b.uint64() // size of the central directory
d.directoryOffset = b.uint64() // offset of start of central directory with respect to the starting disk number
return nil
}
func findSignatureInBlock(b []byte) int {
for i := len(b) - directoryEndLen; i >= 0; i-- {
// defined from directoryEndSignature
@@ -9,6 +9,8 @@ import (
func SourceInfo(p pkg.Package) string {
answer := ""
switch p.Type {
case pkg.AlpmPkg:
answer = "acquired package info from ALPM DB"
case pkg.RpmPkg:
answer = "acquired package info from RPM DB"
case pkg.ApkPkg:
@@ -142,6 +142,14 @@ func Test_SourceInfo(t *testing.T) {
"from dotnet project assets file",
},
},
{
input: pkg.Package{
Type: pkg.AlpmPkg,
},
expected: []string{
"from ALPM DB",
},
},
}
var pkgTypes []pkg.Type
for _, test := range tests {
@@ -61,7 +61,7 @@ func TestSPDXJSONSPDXIDs(t *testing.T) {
},
},
},
true,
*updateSpdxTagValue,
spdxTagValueRedactor,
)
}
@@ -2,11 +2,11 @@ SPDXVersion: SPDX-2.2
DataLicense: CC0-1.0
SPDXID: SPDXRef-DOCUMENT
DocumentName: .
DocumentNamespace: https://anchore.com/syft/dir/422d92b9-57e8-44ee-8039-f75c1d19be87
DocumentNamespace: https://anchore.com/syft/dir/bdb67358-651c-4dd8-b5ee-5318936eb16a
LicenseListVersion: 3.17
Creator: Organization: Anchore, Inc
Creator: Tool: syft-v0.42.0-bogus
Created: 2022-05-24T22:52:02Z
Created: 2022-06-07T19:33:39Z
##### Package: @at-sign
@@ -13,6 +13,10 @@ type LinuxRelease struct {
IDLike IDLikes `json:"idLike,omitempty"`
Version string `json:"version,omitempty"`
VersionID string `json:"versionID,omitempty"`
VersionCodename string `json:"versionCodename,omitempty"`
BuildID string `json:"buildID,omitempty"`
ImageID string `json:"imageID,omitempty"`
ImageVersion string `json:"imageVersion,omitempty"`
Variant string `json:"variant,omitempty"`
VariantID string `json:"variantID,omitempty"`
HomeURL string `json:"homeURL,omitempty"`
+11 -2
View File
@@ -47,7 +47,6 @@ func (p *packageMetadataUnpacker) String() string {
}
// UnmarshalJSON is a custom unmarshaller for handling basic values and values with ambiguous types.
// nolint:funlen
func (p *Package) UnmarshalJSON(b []byte) error {
var basic PackageBasicData
if err := json.Unmarshal(b, &basic); err != nil {
@@ -61,9 +60,19 @@ func (p *Package) UnmarshalJSON(b []byte) error {
return err
}
p.MetadataType = unpacker.MetadataType
return unpackMetadata(p, unpacker)
}
// nolint:funlen
func unpackMetadata(p *Package, unpacker packageMetadataUnpacker) error {
p.MetadataType = unpacker.MetadataType
switch p.MetadataType {
case pkg.AlpmMetadataType:
var payload pkg.AlpmMetadata
if err := json.Unmarshal(unpacker.Metadata, &payload); err != nil {
return err
}
p.Metadata = payload
case pkg.ApkMetadataType:
var payload pkg.ApkMetadata
if err := json.Unmarshal(unpacker.Metadata, &payload); err != nil {
@@ -88,7 +88,7 @@
}
},
"schema": {
"version": "3.2.4",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-3.2.4.json"
"version": "3.3.0",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-3.3.0.json"
}
}
@@ -184,7 +184,7 @@
}
},
"schema": {
"version": "3.2.4",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-3.2.4.json"
"version": "3.3.0",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-3.3.0.json"
}
}
@@ -9,7 +9,7 @@
"locations": [
{
"path": "/somefile-1.txt",
"layerID": "sha256:fb6beecb75b39f4bb813dbf177e501edd5ddb3e69bb45cedeb78c676ee1b7a59"
"layerID": "sha256:7ef28e9c2d56471ee090b578a678bdf28c3b5a311ca7b2e28c2a4185e5bb34c0"
}
],
"licenses": [
@@ -40,7 +40,7 @@
"locations": [
{
"path": "/somefile-2.txt",
"layerID": "sha256:319b588ce64253a87b533c8ed01cf0025e0eac98e7b516e12532957e1244fdec"
"layerID": "sha256:86da8aee621161bea2efaf27a2709ddab5e7d44e30ecdfda728b02c03a28fd98"
}
],
"licenses": [],
@@ -67,7 +67,7 @@
"type": "image",
"target": {
"userInput": "user-image-input",
"imageID": "sha256:2480160b55bec40c44d3b145c7b2c1c47160db8575c3dcae086d76b9370ae7ca",
"imageID": "sha256:5dd5f5f4247e4e946f555f0de7681a631a5240b614e52717d0aed04808e8c65f",
"manifestDigest": "sha256:2731251dc34951c0e50fcc643b4c5f74922dad1a5d98f302b504cf46cd5d9368",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"tags": [
@@ -77,17 +77,17 @@
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:fb6beecb75b39f4bb813dbf177e501edd5ddb3e69bb45cedeb78c676ee1b7a59",
"digest": "sha256:7ef28e9c2d56471ee090b578a678bdf28c3b5a311ca7b2e28c2a4185e5bb34c0",
"size": 22
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:319b588ce64253a87b533c8ed01cf0025e0eac98e7b516e12532957e1244fdec",
"digest": "sha256:86da8aee621161bea2efaf27a2709ddab5e7d44e30ecdfda728b02c03a28fd98",
"size": 16
}
],
"manifest": "eyJzY2hlbWFWZXJzaW9uIjoyLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi92bmQuZG9ja2VyLmRpc3RyaWJ1dGlvbi5tYW5pZmVzdC52Mitqc29uIiwiY29uZmlnIjp7Im1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL3ZuZC5kb2NrZXIuY29udGFpbmVyLmltYWdlLnYxK2pzb24iLCJzaXplIjo2NjcsImRpZ2VzdCI6InNoYTI1NjoyNDgwMTYwYjU1YmVjNDBjNDRkM2IxNDVjN2IyYzFjNDcxNjBkYjg1NzVjM2RjYWUwODZkNzZiOTM3MGFlN2NhIn0sImxheWVycyI6W3sibWVkaWFUeXBlIjoiYXBwbGljYXRpb24vdm5kLmRvY2tlci5pbWFnZS5yb290ZnMuZGlmZi50YXIuZ3ppcCIsInNpemUiOjIwNDgsImRpZ2VzdCI6InNoYTI1NjpmYjZiZWVjYjc1YjM5ZjRiYjgxM2RiZjE3N2U1MDFlZGQ1ZGRiM2U2OWJiNDVjZWRlYjc4YzY3NmVlMWI3YTU5In0seyJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi92bmQuZG9ja2VyLmltYWdlLnJvb3Rmcy5kaWZmLnRhci5nemlwIiwic2l6ZSI6MjA0OCwiZGlnZXN0Ijoic2hhMjU2OjMxOWI1ODhjZTY0MjUzYTg3YjUzM2M4ZWQwMWNmMDAyNWUwZWFjOThlN2I1MTZlMTI1MzI5NTdlMTI0NGZkZWMifV19",
"config": "eyJhcmNoaXRlY3R1cmUiOiJhbWQ2NCIsImNvbmZpZyI6eyJFbnYiOlsiUEFUSD0vdXNyL2xvY2FsL3NiaW46L3Vzci9sb2NhbC9iaW46L3Vzci9zYmluOi91c3IvYmluOi9zYmluOi9iaW4iXSwiV29ya2luZ0RpciI6Ii8iLCJPbkJ1aWxkIjpudWxsfSwiY3JlYXRlZCI6IjIwMjEtMTAtMDRUMTE6NDA6MDAuNjM4Mzk0NVoiLCJoaXN0b3J5IjpbeyJjcmVhdGVkIjoiMjAyMS0xMC0wNFQxMTo0MDowMC41OTA3MzE2WiIsImNyZWF0ZWRfYnkiOiJBREQgZmlsZS0xLnR4dCAvc29tZWZpbGUtMS50eHQgIyBidWlsZGtpdCIsImNvbW1lbnQiOiJidWlsZGtpdC5kb2NrZXJmaWxlLnYwIn0seyJjcmVhdGVkIjoiMjAyMS0xMC0wNFQxMTo0MDowMC42MzgzOTQ1WiIsImNyZWF0ZWRfYnkiOiJBREQgZmlsZS0yLnR4dCAvc29tZWZpbGUtMi50eHQgIyBidWlsZGtpdCIsImNvbW1lbnQiOiJidWlsZGtpdC5kb2NrZXJmaWxlLnYwIn1dLCJvcyI6ImxpbnV4Iiwicm9vdGZzIjp7InR5cGUiOiJsYXllcnMiLCJkaWZmX2lkcyI6WyJzaGEyNTY6ZmI2YmVlY2I3NWIzOWY0YmI4MTNkYmYxNzdlNTAxZWRkNWRkYjNlNjliYjQ1Y2VkZWI3OGM2NzZlZTFiN2E1OSIsInNoYTI1NjozMTliNTg4Y2U2NDI1M2E4N2I1MzNjOGVkMDFjZjAwMjVlMGVhYzk4ZTdiNTE2ZTEyNTMyOTU3ZTEyNDRmZGVjIl19fQ==",
"manifest": "eyJzY2hlbWFWZXJzaW9uIjoyLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi92bmQuZG9ja2VyLmRpc3RyaWJ1dGlvbi5tYW5pZmVzdC52Mitqc29uIiwiY29uZmlnIjp7Im1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL3ZuZC5kb2NrZXIuY29udGFpbmVyLmltYWdlLnYxK2pzb24iLCJzaXplIjo2NzMsImRpZ2VzdCI6InNoYTI1Njo1ZGQ1ZjVmNDI0N2U0ZTk0NmY1NTVmMGRlNzY4MWE2MzFhNTI0MGI2MTRlNTI3MTdkMGFlZDA0ODA4ZThjNjVmIn0sImxheWVycyI6W3sibWVkaWFUeXBlIjoiYXBwbGljYXRpb24vdm5kLmRvY2tlci5pbWFnZS5yb290ZnMuZGlmZi50YXIuZ3ppcCIsInNpemUiOjIwNDgsImRpZ2VzdCI6InNoYTI1Njo3ZWYyOGU5YzJkNTY0NzFlZTA5MGI1NzhhNjc4YmRmMjhjM2I1YTMxMWNhN2IyZTI4YzJhNDE4NWU1YmIzNGMwIn0seyJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi92bmQuZG9ja2VyLmltYWdlLnJvb3Rmcy5kaWZmLnRhci5nemlwIiwic2l6ZSI6MjA0OCwiZGlnZXN0Ijoic2hhMjU2Ojg2ZGE4YWVlNjIxMTYxYmVhMmVmYWYyN2EyNzA5ZGRhYjVlN2Q0NGUzMGVjZGZkYTcyOGIwMmMwM2EyOGZkOTgifV19",
"config": "eyJhcmNoaXRlY3R1cmUiOiJhbWQ2NCIsImNvbmZpZyI6eyJFbnYiOlsiUEFUSD0vdXNyL2xvY2FsL3NiaW46L3Vzci9sb2NhbC9iaW46L3Vzci9zYmluOi91c3IvYmluOi9zYmluOi9iaW4iXSwiV29ya2luZ0RpciI6Ii8iLCJPbkJ1aWxkIjpudWxsfSwiY3JlYXRlZCI6IjIwMjItMDYtMDJUMTQ6MzQ6MzQuNzE5MTM1MTc0WiIsImhpc3RvcnkiOlt7ImNyZWF0ZWQiOiIyMDIyLTA2LTAyVDE0OjM0OjM0LjY4NjkzMzI2M1oiLCJjcmVhdGVkX2J5IjoiQUREIGZpbGUtMS50eHQgL3NvbWVmaWxlLTEudHh0ICMgYnVpbGRraXQiLCJjb21tZW50IjoiYnVpbGRraXQuZG9ja2VyZmlsZS52MCJ9LHsiY3JlYXRlZCI6IjIwMjItMDYtMDJUMTQ6MzQ6MzQuNzE5MTM1MTc0WiIsImNyZWF0ZWRfYnkiOiJBREQgZmlsZS0yLnR4dCAvc29tZWZpbGUtMi50eHQgIyBidWlsZGtpdCIsImNvbW1lbnQiOiJidWlsZGtpdC5kb2NrZXJmaWxlLnYwIn1dLCJvcyI6ImxpbnV4Iiwicm9vdGZzIjp7InR5cGUiOiJsYXllcnMiLCJkaWZmX2lkcyI6WyJzaGEyNTY6N2VmMjhlOWMyZDU2NDcxZWUwOTBiNTc4YTY3OGJkZjI4YzNiNWEzMTFjYTdiMmUyOGMyYTQxODVlNWJiMzRjMCIsInNoYTI1Njo4NmRhOGFlZTYyMTE2MWJlYTJlZmFmMjdhMjcwOWRkYWI1ZTdkNDRlMzBlY2RmZGE3MjhiMDJjMDNhMjhmZDk4Il19fQ==",
"repoDigests": [],
"architecture": "",
"os": ""
@@ -111,7 +111,7 @@
}
},
"schema": {
"version": "3.2.4",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-3.2.4.json"
"version": "3.3.0",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-3.3.0.json"
}
}
@@ -55,6 +55,10 @@ func toLinuxReleaser(d *linux.Release) model.LinuxRelease {
IDLike: d.IDLike,
Version: d.Version,
VersionID: d.VersionID,
VersionCodename: d.VersionCodename,
BuildID: d.BuildID,
ImageID: d.ImageID,
ImageVersion: d.ImageVersion,
Variant: d.Variant,
VariantID: d.VariantID,
HomeURL: d.HomeURL,
@@ -38,6 +38,10 @@ func toSyftLinuxRelease(d model.LinuxRelease) *linux.Release {
IDLike: d.IDLike,
Version: d.Version,
VersionID: d.VersionID,
VersionCodename: d.VersionCodename,
BuildID: d.BuildID,
ImageID: d.ImageID,
ImageVersion: d.ImageVersion,
Variant: d.Variant,
VariantID: d.VariantID,
HomeURL: d.HomeURL,
+49
View File
@@ -0,0 +1,49 @@
package template
import (
"errors"
"fmt"
"os"
"reflect"
"text/template"
"github.com/Masterminds/sprig/v3"
"github.com/mitchellh/go-homedir"
)
func makeTemplateExecutor(templateFilePath string) (*template.Template, error) {
if templateFilePath == "" {
return nil, errors.New("no template file: please provide a template path")
}
expandedPathToTemplateFile, err := homedir.Expand(templateFilePath)
if err != nil {
return nil, fmt.Errorf("unable to expand path %s", templateFilePath)
}
templateContents, err := os.ReadFile(expandedPathToTemplateFile)
if err != nil {
return nil, fmt.Errorf("unable to get template content: %w", err)
}
templateName := expandedPathToTemplateFile
tmpl, err := template.New(templateName).Funcs(funcMap).Parse(string(templateContents))
if err != nil {
return nil, fmt.Errorf("unable to parse template: %w", err)
}
return tmpl, nil
}
// These are custom functions available to template authors.
var funcMap = func() template.FuncMap {
f := sprig.HermeticTxtFuncMap()
f["getLastIndex"] = func(collection interface{}) int {
if v := reflect.ValueOf(collection); v.Kind() == reflect.Slice {
return v.Len() - 1
}
return 0
}
return f
}()
+29
View File
@@ -0,0 +1,29 @@
package template
import (
"flag"
"testing"
"github.com/anchore/syft/internal/formats/common/testutils"
"github.com/stretchr/testify/assert"
)
var updateTmpl = flag.Bool("update-tmpl", false, "update the *.golden files for json encoders")
func TestFormatWithOption(t *testing.T) {
f := OutputFormat{}
f.SetTemplatePath("test-fixtures/csv.template")
testutils.AssertEncoderAgainstGoldenSnapshot(t,
f,
testutils.DirectoryInput(t),
*updateTmpl,
)
}
func TestFormatWithoutOptions(t *testing.T) {
f := Format()
err := f.Encode(nil, testutils.DirectoryInput(t))
assert.ErrorContains(t, err, "no template file: please provide a template path")
}
+47
View File
@@ -0,0 +1,47 @@
package template
import (
"io"
"github.com/anchore/syft/internal/formats/syftjson"
"github.com/anchore/syft/syft/sbom"
)
const ID sbom.FormatID = "template"
func Format() sbom.Format {
return OutputFormat{}
}
// implementation of sbom.Format interface
// to make use of format options
type OutputFormat struct {
templateFilePath string
}
func (f OutputFormat) ID() sbom.FormatID {
return ID
}
func (f OutputFormat) Decode(reader io.Reader) (*sbom.SBOM, error) {
return nil, sbom.ErrDecodingNotSupported
}
func (f OutputFormat) Encode(output io.Writer, s sbom.SBOM) error {
tmpl, err := makeTemplateExecutor(f.templateFilePath)
if err != nil {
return err
}
doc := syftjson.ToFormatModel(s)
return tmpl.Execute(output, doc)
}
func (f OutputFormat) Validate(reader io.Reader) error {
return sbom.ErrValidationNotSupported
}
// SetTemplatePath sets path for template file
func (f *OutputFormat) SetTemplatePath(filePath string) {
f.templateFilePath = filePath
}
@@ -0,0 +1,4 @@
"Package","Version Installed", "Found by"
{{- range .Artifacts}}
"{{.Name}}","{{.Version}}","{{.FoundBy}}"
{{- end}}
@@ -0,0 +1,3 @@
"Package","Version Installed", "Found by"
"package-1","1.0.1","the-cataloger-1"
"package-2","2.0.1","the-cataloger-2"
+1
View File
@@ -28,6 +28,7 @@ can be extended to include specific package metadata struct shapes in the future
// not matter as long as it is exported.
type artifactMetadataContainer struct {
Apk pkg.ApkMetadata
Alpm pkg.AlpmMetadata
Dpkg pkg.DpkgMetadata
Gem pkg.GemMetadata
Java pkg.JavaMetadata
File diff suppressed because it is too large Load Diff
+5
View File
@@ -11,6 +11,7 @@ import (
"github.com/anchore/syft/internal/formats/spdx22tagvalue"
"github.com/anchore/syft/internal/formats/syftjson"
"github.com/anchore/syft/internal/formats/table"
"github.com/anchore/syft/internal/formats/template"
"github.com/anchore/syft/internal/formats/text"
"github.com/anchore/syft/syft/sbom"
)
@@ -25,6 +26,7 @@ const (
GitHubID = github.ID
SPDXTagValueFormatID = spdx22tagvalue.ID
SPDXJSONFormatID = spdx22json.ID
TemplateFormatID = template.ID
)
var formats []sbom.Format
@@ -39,6 +41,7 @@ func init() {
spdx22json.Format(),
table.Format(),
text.Format(),
template.Format(),
}
}
@@ -84,6 +87,8 @@ func FormatByName(name string) sbom.Format {
return FormatByID(table.ID)
case "text":
return FormatByID(text.ID)
case "template":
FormatByID(template.ID)
}
return nil
+6
View File
@@ -13,6 +13,7 @@ import (
"github.com/anchore/syft/internal/formats/spdx22tagvalue"
"github.com/anchore/syft/internal/formats/syftjson"
"github.com/anchore/syft/internal/formats/table"
"github.com/anchore/syft/internal/formats/template"
"github.com/anchore/syft/internal/formats/text"
"github.com/anchore/syft/syft/sbom"
"github.com/stretchr/testify/require"
@@ -181,6 +182,11 @@ func TestFormatByName(t *testing.T) {
name: "github-json",
want: github.ID,
},
{
name: "template",
want: template.ID,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
+4
View File
@@ -111,6 +111,10 @@ func parseOsRelease(contents string) (*Release, error) {
IDLike: idLike,
Version: values["VERSION"],
VersionID: values["VERSION_ID"],
VersionCodename: values["VERSION_CODENAME"],
BuildID: values["BUILD_ID"],
ImageID: values["IMAGE_ID"],
ImageVersion: values["IMAGE_VERSION"],
Variant: values["VARIANT"],
VariantID: values["VARIANT_ID"],
HomeURL: values["HOME_URL"],
+3
View File
@@ -125,6 +125,7 @@ func TestIdentifyRelease(t *testing.T) {
ID: "ubuntu",
IDLike: []string{"debian"},
Version: "20.04 LTS (Focal Fossa)",
VersionCodename: "focal",
VersionID: "20.04",
HomeURL: "https://www.ubuntu.com/",
SupportURL: "https://help.ubuntu.com/",
@@ -217,6 +218,7 @@ func TestIdentifyRelease(t *testing.T) {
Name: "Arch Linux",
ID: "arch",
IDLike: nil,
BuildID: "rolling",
HomeURL: "https://www.archlinux.org/",
SupportURL: "https://bbs.archlinux.org/",
BugReportURL: "https://bugs.archlinux.org/",
@@ -349,6 +351,7 @@ func TestParseOsRelease(t *testing.T) {
IDLike: []string{"debian"},
Version: "20.04 LTS (Focal Fossa)",
VersionID: "20.04",
VersionCodename: "focal",
HomeURL: "https://www.ubuntu.com/",
SupportURL: "https://help.ubuntu.com/",
BugReportURL: "https://bugs.launchpad.net/ubuntu/",
+8 -1
View File
@@ -8,6 +8,10 @@ type Release struct {
IDLike []string `cyclonedx:"idLike"` // list of operating system identifiers in the same syntax as the ID= setting. It should list identifiers of operating systems that are closely related to the local operating system in regards to packaging and programming interfaces.
Version string // identifies the operating system version, excluding any OS name information, possibly including a release code name, and suitable for presentation to the user.
VersionID string `cyclonedx:"versionID"` // identifies the operating system version, excluding any OS name information or release code name, and suitable for processing by scripts or usage in generated filenames.
VersionCodename string `cyclonedx:"versionCodename"`
BuildID string `cyclonedx:"buildID"` // A string uniquely identifying the system image originally used as the installation base.
ImageID string `cyclonedx:"imageID"`
ImageVersion string `cyclonedx:"imageVersion"`
Variant string `cyclonedx:"variant"` // identifies a specific variant or edition of the operating system suitable for presentation to the user.
VariantID string `cyclonedx:"variantID"` // identifies a specific variant or edition of the operating system. This may be interpreted by other packages in order to determine a divergent default configuration.
HomeURL string
@@ -30,6 +34,9 @@ func (r *Release) String() string {
if r.Version != "" {
return r.ID + " " + r.Version
}
if r.VersionID != "" {
return r.ID + " " + r.VersionID
}
return r.ID + " " + r.VersionID
return r.ID + " " + r.BuildID
}
+80
View File
@@ -0,0 +1,80 @@
package pkg
import (
"sort"
"time"
"github.com/anchore/packageurl-go"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/linux"
"github.com/scylladb/go-set/strset"
)
const AlpmDBGlob = "**/var/lib/pacman/local/**/desc"
type AlpmMetadata struct {
BasePackage string `mapstructure:"base" json:"basepackage"`
Package string `mapstructure:"name" json:"package"`
Version string `mapstructure:"version" json:"version"`
Description string `mapstructure:"desc" json:"description"`
Architecture string `mapstructure:"arch" json:"architecture"`
Size int `mapstructure:"size" json:"size" cyclonedx:"size"`
Packager string `mapstructure:"packager" json:"packager"`
License string `mapstructure:"license" json:"license"`
URL string `mapstructure:"url" json:"url"`
Validation string `mapstructure:"validation" json:"validation"`
Reason int `mapstructure:"reason" json:"reason"`
Files []AlpmFileRecord `mapstructure:"files" json:"files"`
Backup []AlpmFileRecord `mapstructure:"backup" json:"backup"`
}
type AlpmFileRecord struct {
Path string `mapstruture:"path" json:"path,omitempty"`
Type string `mapstructure:"type" json:"type,omitempty"`
UID string `mapstructure:"uid" json:"uid,omitempty"`
GID string `mapstructure:"gid" json:"gid,omitempty"`
Time time.Time `mapstructure:"time" json:"time,omitempty"`
Size string `mapstructure:"size" json:"size,omitempty"`
Link string `mapstructure:"link" json:"link,omitempty"`
Digests []file.Digest `mapstructure:"digests" json:"digest,omitempty"`
}
// PackageURL returns the PURL for the specific Arch Linux package (see https://github.com/package-url/purl-spec)
func (m AlpmMetadata) PackageURL(distro *linux.Release) string {
qualifiers := map[string]string{
PURLQualifierArch: m.Architecture,
}
if m.BasePackage != "" {
qualifiers[PURLQualifierUpstream] = m.BasePackage
}
distroID := ""
if distro != nil {
distroID = distro.ID
}
return packageurl.NewPackageURL(
"alpm",
distroID,
m.Package,
m.Version,
purlQualifiers(
qualifiers,
distro,
),
"",
).ToString()
}
func (m AlpmMetadata) OwnedFiles() (result []string) {
s := strset.New()
for _, f := range m.Files {
if f.Path != "" {
s.Add(f.Path)
}
}
result = s.List()
sort.Strings(result)
return result
}
+112
View File
@@ -0,0 +1,112 @@
package pkg
import (
"testing"
"github.com/anchore/packageurl-go"
"github.com/anchore/syft/syft/linux"
"github.com/sergi/go-diff/diffmatchpatch"
)
func TestAlpmMetadata_pURL(t *testing.T) {
tests := []struct {
name string
metadata AlpmMetadata
distro linux.Release
expected string
}{
{
name: "gocase",
metadata: AlpmMetadata{
Package: "p",
Version: "v",
Architecture: "a",
},
distro: linux.Release{
ID: "arch",
BuildID: "rolling",
},
expected: "pkg:alpm/arch/p@v?arch=a&distro=arch-rolling",
},
{
name: "missing architecture",
metadata: AlpmMetadata{
Package: "p",
Version: "v",
},
distro: linux.Release{
ID: "arch",
},
expected: "pkg:alpm/arch/p@v?distro=arch",
},
{
metadata: AlpmMetadata{
Package: "python",
Version: "3.10.0",
Architecture: "any",
},
distro: linux.Release{
ID: "arch",
BuildID: "rolling",
},
expected: "pkg:alpm/arch/python@3.10.0?arch=any&distro=arch-rolling",
},
{
metadata: AlpmMetadata{
Package: "g plus plus",
Version: "v84",
Architecture: "x86_64",
},
distro: linux.Release{
ID: "arch",
BuildID: "rolling",
},
expected: "pkg:alpm/arch/g%20plus%20plus@v84?arch=x86_64&distro=arch-rolling",
},
{
name: "add source information as qualifier",
metadata: AlpmMetadata{
Package: "p",
Version: "v",
Architecture: "a",
BasePackage: "origin",
},
distro: linux.Release{
ID: "arch",
BuildID: "rolling",
},
expected: "pkg:alpm/arch/p@v?arch=a&upstream=origin&distro=arch-rolling",
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
actual := test.metadata.PackageURL(&test.distro)
if actual != test.expected {
dmp := diffmatchpatch.New()
diffs := dmp.DiffMain(test.expected, actual, true)
t.Errorf("diff: %s", dmp.DiffPrettyText(diffs))
}
// verify packageurl can parse
purl, err := packageurl.FromString(actual)
if err != nil {
t.Errorf("cannot re-parse purl: %s", actual)
}
if purl.Name != test.metadata.Package {
dmp := diffmatchpatch.New()
diffs := dmp.DiffMain(test.metadata.Package, purl.Name, true)
t.Errorf("invalid purl name: %s", dmp.DiffPrettyText(diffs))
}
if purl.Version != test.metadata.Version {
dmp := diffmatchpatch.New()
diffs := dmp.DiffMain(test.metadata.Version, purl.Version, true)
t.Errorf("invalid purl version: %s", dmp.DiffPrettyText(diffs))
}
if purl.Qualifiers.Map()["arch"] != test.metadata.Architecture {
dmp := diffmatchpatch.New()
diffs := dmp.DiffMain(test.metadata.Architecture, purl.Qualifiers.Map()["arch"], true)
t.Errorf("invalid purl architecture: %s", dmp.DiffPrettyText(diffs))
}
})
}
}
+2 -1
View File
@@ -1,10 +1,11 @@
package pkg
import (
"github.com/anchore/syft/syft/linux"
"strings"
"testing"
"github.com/anchore/syft/syft/linux"
"github.com/anchore/packageurl-go"
"github.com/go-test/deep"
"github.com/sergi/go-diff/diffmatchpatch"
+48
View File
@@ -0,0 +1,48 @@
package alpm
import (
"fmt"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/source"
)
const catalogerName = "alpmdb-cataloger"
type Cataloger struct{}
// NewAlpmdbCataloger returns a new ALPM DB cataloger object.
func NewAlpmdbCataloger() *Cataloger {
return &Cataloger{}
}
// Name returns a string that uniquely describes a cataloger
func (c *Cataloger) Name() string {
return catalogerName
}
// Catalog is given an object to resolve file references and content, this function returns any discovered Packages after analyzing rpm db installation.
func (c *Cataloger) Catalog(resolver source.FileResolver) ([]pkg.Package, []artifact.Relationship, error) {
fileMatches, err := resolver.FilesByGlob(pkg.AlpmDBGlob)
if err != nil {
return nil, nil, fmt.Errorf("failed to find rpmdb's by glob: %w", err)
}
var pkgs []pkg.Package
for _, location := range fileMatches {
dbContentReader, err := resolver.FileContentsByLocation(location)
if err != nil {
return nil, nil, err
}
discoveredPkgs, err := parseAlpmDB(resolver, location.RealPath, dbContentReader)
internal.CloseAndLogError(dbContentReader, location.VirtualPath)
if err != nil {
return nil, nil, fmt.Errorf("unable to catalog package=%+v: %w", location.RealPath, err)
}
pkgs = append(pkgs, discoveredPkgs...)
}
return pkgs, nil, nil
}
+245
View File
@@ -0,0 +1,245 @@
package alpm
import (
"bufio"
"compress/gzip"
"fmt"
"io"
"path/filepath"
"strconv"
"strings"
"time"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/source"
"github.com/mitchellh/mapstructure"
"github.com/vbatts/go-mtree"
)
var (
ignoredFiles = map[string]bool{
"/set": true,
".BUILDINFO": true,
".PKGINFO": true,
"": true,
}
)
func newAlpmDBPackage(d *pkg.AlpmMetadata) *pkg.Package {
return &pkg.Package{
Name: d.Package,
Version: d.Version,
FoundBy: catalogerName,
Type: "alpm",
Licenses: strings.Split(d.License, " "),
MetadataType: pkg.AlpmMetadataType,
Metadata: *d,
}
}
func newScanner(reader io.Reader) *bufio.Scanner {
// This is taken from the apk parser
// https://github.com/anchore/syft/blob/v0.47.0/syft/pkg/cataloger/apkdb/parse_apk_db.go#L37
const maxScannerCapacity = 1024 * 1024
bufScan := make([]byte, maxScannerCapacity)
scanner := bufio.NewScanner(reader)
scanner.Buffer(bufScan, maxScannerCapacity)
onDoubleLF := func(data []byte, atEOF bool) (advance int, token []byte, err error) {
for i := 0; i < len(data); i++ {
if i > 0 && data[i-1] == '\n' && data[i] == '\n' {
return i + 1, data[:i-1], nil
}
}
if !atEOF {
return 0, nil, nil
}
// deliver the last token (which could be an empty string)
return 0, data, bufio.ErrFinalToken
}
scanner.Split(onDoubleLF)
return scanner
}
func getFileReader(path string, resolver source.FileResolver) (io.Reader, error) {
locs, err := resolver.FilesByPath(path)
if err != nil {
return nil, err
}
// TODO: Should we maybe check if we found the file
dbContentReader, err := resolver.FileContentsByLocation(locs[0])
if err != nil {
return nil, err
}
return dbContentReader, nil
}
// nolint:funlen
func parseDatabase(b *bufio.Scanner) (*pkg.AlpmMetadata, error) {
var entry pkg.AlpmMetadata
var err error
pkgFields := make(map[string]interface{})
for b.Scan() {
fields := strings.SplitN(b.Text(), "\n", 2)
// End of File
if len(fields) == 1 {
break
}
// The alpm database surrounds the keys with %.
key := strings.ReplaceAll(fields[0], "%", "")
key = strings.ToLower(key)
value := strings.TrimSpace(fields[1])
switch key {
case "files":
var files []map[string]string
for _, f := range strings.Split(value, "\n") {
path := fmt.Sprintf("/%s", f)
if ok := ignoredFiles[path]; !ok {
files = append(files, map[string]string{"path": path})
}
}
pkgFields[key] = files
case "backup":
var backup []map[string]interface{}
for _, f := range strings.Split(value, "\n") {
fields := strings.SplitN(f, "\t", 2)
path := fmt.Sprintf("/%s", fields[0])
if ok := ignoredFiles[path]; !ok {
backup = append(backup, map[string]interface{}{
"path": path,
"digests": []file.Digest{{
Algorithm: "md5",
Value: fields[1],
}}})
}
}
pkgFields[key] = backup
case "reason":
fallthrough
case "size":
pkgFields[key], err = strconv.ParseInt(value, 10, 64)
if err != nil {
return nil, fmt.Errorf("failed to parse %s to integer", value)
}
default:
pkgFields[key] = value
}
}
if err := mapstructure.Decode(pkgFields, &entry); err != nil {
return nil, fmt.Errorf("unable to parse ALPM metadata: %w", err)
}
if entry.Package == "" && len(entry.Files) == 0 && len(entry.Backup) == 0 {
return nil, nil
}
if entry.Backup == nil {
entry.Backup = make([]pkg.AlpmFileRecord, 0)
}
return &entry, nil
}
func parseMtree(r io.Reader) ([]pkg.AlpmFileRecord, error) {
var err error
var entries []pkg.AlpmFileRecord
r, err = gzip.NewReader(r)
if err != nil {
return nil, err
}
specDh, err := mtree.ParseSpec(r)
if err != nil {
return nil, err
}
for _, f := range specDh.Entries {
var entry pkg.AlpmFileRecord
entry.Digests = make([]file.Digest, 0)
fileFields := make(map[string]interface{})
if ok := ignoredFiles[f.Name]; ok {
continue
}
path := fmt.Sprintf("/%s", f.Name)
fileFields["path"] = path
for _, kv := range f.Keywords {
kw := string(kv.Keyword())
switch kw {
case "time":
// All unix timestamps have a .0 suffixs.
v := strings.Split(kv.Value(), ".")
i, _ := strconv.ParseInt(v[0], 10, 64)
tm := time.Unix(i, 0)
fileFields[kw] = tm
case "sha256digest":
entry.Digests = append(entry.Digests, file.Digest{
Algorithm: "sha256",
Value: kv.Value(),
})
case "md5digest":
entry.Digests = append(entry.Digests, file.Digest{
Algorithm: "md5digest",
Value: kv.Value(),
})
default:
fileFields[kw] = kv.Value()
}
}
if err := mapstructure.Decode(fileFields, &entry); err != nil {
return nil, fmt.Errorf("unable to parse ALPM mtree data: %w", err)
}
entries = append(entries, entry)
}
return entries, nil
}
func parseAlpmDBEntry(reader io.Reader) (*pkg.AlpmMetadata, error) {
scanner := newScanner(reader)
metadata, err := parseDatabase(scanner)
if err != nil {
return nil, err
}
if metadata == nil {
return nil, nil
}
return metadata, nil
}
func parseAlpmDB(resolver source.FileResolver, desc string, reader io.Reader) ([]pkg.Package, error) {
metadata, err := parseAlpmDBEntry(reader)
if err != nil {
return nil, err
}
base := filepath.Dir(desc)
mtree := filepath.Join(base, "mtree")
r, err := getFileReader(mtree, resolver)
if err != nil {
return nil, err
}
pkgFiles, err := parseMtree(r)
if err != nil {
return nil, err
}
// The replace the files found the the pacman database with the files from the mtree These contain more metadata and
// thus more useful.
metadata.Files = pkgFiles
// We only really do this to get any backup database entries from the files database
files := filepath.Join(base, "files")
_, err = getFileReader(files, resolver)
if err != nil {
return nil, err
}
filesMetadata, err := parseAlpmDBEntry(reader)
if err != nil {
return nil, err
} else if filesMetadata != nil {
metadata.Backup = filesMetadata.Backup
}
p := *newAlpmDBPackage(metadata)
p.SetID()
return []pkg.Package{p}, nil
}
@@ -0,0 +1,195 @@
package alpm
import (
"bufio"
"os"
"testing"
"time"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/pkg"
"github.com/go-test/deep"
)
func TestDatabaseParser(t *testing.T) {
tests := []struct {
name string
expected pkg.AlpmMetadata
}{
{
name: "test alpm database parsing",
expected: pkg.AlpmMetadata{
Backup: []pkg.AlpmFileRecord{
{
Path: "/etc/pacman.conf",
Digests: []file.Digest{{
Algorithm: "md5",
Value: "de541390e52468165b96511c4665bff4",
}},
},
{
Path: "/etc/makepkg.conf",
Digests: []file.Digest{{
Algorithm: "md5",
Value: "79fce043df7dfc676ae5ecb903762d8b",
}},
},
},
Files: []pkg.AlpmFileRecord{
{
Path: "/etc/",
},
{
Path: "/etc/makepkg.conf",
},
{
Path: "/etc/pacman.conf",
},
{
Path: "/usr/",
},
{
Path: "/usr/bin/",
},
{
Path: "/usr/bin/makepkg",
},
{
Path: "/usr/bin/makepkg-template",
},
{
Path: "/usr/bin/pacman",
},
{
Path: "/usr/bin/pacman-conf",
},
{
Path: "/var/",
},
{
Path: "/var/cache/",
},
{
Path: "/var/cache/pacman/",
},
{
Path: "/var/cache/pacman/pkg/",
},
{
Path: "/var/lib/",
},
{
Path: "/var/lib/pacman/",
},
},
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
file, err := os.Open("test-fixtures/files")
if err != nil {
t.Fatal("Unable to read test-fixtures/file: ", err)
}
defer func() {
err := file.Close()
if err != nil {
t.Fatal("closing file failed:", err)
}
}()
reader := bufio.NewReader(file)
entry, err := parseAlpmDBEntry(reader)
if err != nil {
t.Fatal("Unable to read file contents: ", err)
}
if diff := deep.Equal(entry.Files, test.expected.Files); diff != nil {
for _, d := range diff {
t.Errorf("files diff: %+v", d)
}
}
if diff := deep.Equal(entry.Backup, test.expected.Backup); diff != nil {
for _, d := range diff {
t.Errorf("backup diff: %+v", d)
}
}
})
}
}
func parseTime(stime string) time.Time {
t, _ := time.Parse(time.RFC3339, stime)
return t
}
func TestMtreeParse(t *testing.T) {
tests := []struct {
name string
expected []pkg.AlpmFileRecord
}{
{
name: "test mtree parsing",
expected: []pkg.AlpmFileRecord{
{
Path: "/etc",
Type: "dir",
Time: parseTime("2022-04-10T14:59:52+02:00"),
Digests: make([]file.Digest, 0),
},
{
Path: "/etc/pacman.d",
Type: "dir",
Time: parseTime("2022-04-10T14:59:52+02:00"),
Digests: make([]file.Digest, 0),
},
{
Path: "/etc/pacman.d/mirrorlist",
Size: "44683",
Time: parseTime("2022-04-10T14:59:52+02:00"),
Digests: []file.Digest{
{
Algorithm: "md5digest",
Value: "81c39827e38c759d7e847f05db62c233",
},
{
Algorithm: "sha256",
Value: "fc135ab26f2a227b9599b66a2f1ba325c445acb914d60e7ecf6e5997a87abe1e",
},
},
},
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
file, err := os.Open("test-fixtures/mtree")
if err != nil {
t.Fatal("Unable to read test-fixtures/mtree: ", err)
}
defer func() {
err := file.Close()
if err != nil {
t.Fatal("closing file failed:", err)
}
}()
reader := bufio.NewReader(file)
entry, err := parseMtree(reader)
if err != nil {
t.Fatal("Unable to read file contents: ", err)
}
if diff := deep.Equal(entry, test.expected); diff != nil {
for _, d := range diff {
t.Errorf("files diff: %+v", d)
}
}
})
}
}
@@ -0,0 +1,20 @@
%FILES%
etc/
etc/makepkg.conf
etc/pacman.conf
usr/
usr/bin/
usr/bin/makepkg
usr/bin/makepkg-template
usr/bin/pacman
usr/bin/pacman-conf
var/
var/cache/
var/cache/pacman/
var/cache/pacman/pkg/
var/lib/
var/lib/pacman/
%BACKUP%
etc/pacman.conf de541390e52468165b96511c4665bff4
etc/makepkg.conf 79fce043df7dfc676ae5ecb903762d8b
Binary file not shown.
+44 -5
View File
@@ -6,8 +6,12 @@ catalogers defined in child packages as well as the interface definition to impl
package cataloger
import (
"strings"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/pkg/cataloger/alpm"
"github.com/anchore/syft/syft/pkg/cataloger/apkdb"
"github.com/anchore/syft/syft/pkg/cataloger/dart"
"github.com/anchore/syft/syft/pkg/cataloger/deb"
@@ -35,7 +39,8 @@ type Cataloger interface {
// ImageCatalogers returns a slice of locally implemented catalogers that are fit for detecting installations of packages.
func ImageCatalogers(cfg Config) []Cataloger {
return []Cataloger{
return filterCatalogers([]Cataloger{
alpm.NewAlpmdbCataloger(),
ruby.NewGemSpecCataloger(),
python.NewPythonPackageCataloger(),
php.NewPHPComposerInstalledCataloger(),
@@ -46,12 +51,13 @@ func ImageCatalogers(cfg Config) []Cataloger {
apkdb.NewApkdbCataloger(),
golang.NewGoModuleBinaryCataloger(),
dotnet.NewDotnetDepsCataloger(),
}
}, cfg.Catalogers)
}
// DirectoryCatalogers returns a slice of locally implemented catalogers that are fit for detecting packages from index files (and select installations)
func DirectoryCatalogers(cfg Config) []Cataloger {
return []Cataloger{
return filterCatalogers([]Cataloger{
alpm.NewAlpmdbCataloger(),
ruby.NewGemFileLockCataloger(),
python.NewPythonIndexCataloger(),
python.NewPythonPackageCataloger(),
@@ -60,18 +66,20 @@ func DirectoryCatalogers(cfg Config) []Cataloger {
deb.NewDpkgdbCataloger(),
rpmdb.NewRpmdbCataloger(),
java.NewJavaCataloger(cfg.Java()),
java.NewJavaPomCataloger(),
apkdb.NewApkdbCataloger(),
golang.NewGoModuleBinaryCataloger(),
golang.NewGoModFileCataloger(),
rust.NewCargoLockCataloger(),
dart.NewPubspecLockCataloger(),
dotnet.NewDotnetDepsCataloger(),
}
}, cfg.Catalogers)
}
// AllCatalogers returns all implemented catalogers
func AllCatalogers(cfg Config) []Cataloger {
return []Cataloger{
return filterCatalogers([]Cataloger{
alpm.NewAlpmdbCataloger(),
ruby.NewGemFileLockCataloger(),
ruby.NewGemSpecCataloger(),
python.NewPythonIndexCataloger(),
@@ -81,11 +89,42 @@ func AllCatalogers(cfg Config) []Cataloger {
deb.NewDpkgdbCataloger(),
rpmdb.NewRpmdbCataloger(),
java.NewJavaCataloger(cfg.Java()),
java.NewJavaPomCataloger(),
apkdb.NewApkdbCataloger(),
golang.NewGoModuleBinaryCataloger(),
golang.NewGoModFileCataloger(),
rust.NewCargoLockCataloger(),
dart.NewPubspecLockCataloger(),
dotnet.NewDotnetDepsCataloger(),
}, cfg.Catalogers)
}
func filterCatalogers(catalogers []Cataloger, enabledCatalogerPatterns []string) []Cataloger {
// if cataloger is not set, all applicable catalogers are enabled by default
if len(enabledCatalogerPatterns) == 0 {
return catalogers
}
var keepCatalogers []Cataloger
for _, cataloger := range catalogers {
if contains(enabledCatalogerPatterns, cataloger.Name()) {
keepCatalogers = append(keepCatalogers, cataloger)
continue
}
log.Infof("skipping cataloger %q", cataloger.Name())
}
return keepCatalogers
}
func contains(enabledPartial []string, catalogerName string) bool {
catalogerName = strings.TrimSuffix(catalogerName, "-cataloger")
for _, partial := range enabledPartial {
partial = strings.TrimSuffix(partial, "-cataloger")
if partial == "" {
continue
}
if strings.Contains(catalogerName, partial) {
return true
}
}
return false
}
+201
View File
@@ -0,0 +1,201 @@
package cataloger
import (
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/source"
"github.com/stretchr/testify/assert"
"testing"
)
var _ Cataloger = (*dummy)(nil)
type dummy struct {
name string
}
func (d dummy) Name() string {
return d.name
}
func (d dummy) Catalog(_ source.FileResolver) ([]pkg.Package, []artifact.Relationship, error) {
panic("not implemented")
}
func Test_filterCatalogers(t *testing.T) {
tests := []struct {
name string
patterns []string
catalogers []string
want []string
}{
{
name: "no filtering",
patterns: nil,
catalogers: []string{
"ruby-gemspec-cataloger",
"python-package-cataloger",
"php-composer-installed-cataloger",
"javascript-package-cataloger",
"dpkgdb-cataloger",
"rpmdb-cataloger",
"java-cataloger",
"apkdb-cataloger",
"go-module-binary-cataloger",
},
want: []string{
"ruby-gemspec-cataloger",
"python-package-cataloger",
"php-composer-installed-cataloger",
"javascript-package-cataloger",
"dpkgdb-cataloger",
"rpmdb-cataloger",
"java-cataloger",
"apkdb-cataloger",
"go-module-binary-cataloger",
},
},
{
name: "exact name match",
patterns: []string{
"rpmdb-cataloger",
"javascript-package-cataloger",
},
catalogers: []string{
"ruby-gemspec-cataloger",
"python-package-cataloger",
"php-composer-installed-cataloger",
"javascript-package-cataloger",
"dpkgdb-cataloger",
"rpmdb-cataloger",
"java-cataloger",
"apkdb-cataloger",
"go-module-binary-cataloger",
},
want: []string{
"javascript-package-cataloger",
"rpmdb-cataloger",
},
},
{
name: "partial name match",
patterns: []string{
"ruby",
"installed",
},
catalogers: []string{
"ruby-gemspec-cataloger",
"ruby-gemfile-cataloger",
"python-package-cataloger",
"php-composer-installed-cataloger",
"javascript-package-cataloger",
"dpkgdb-cataloger",
"rpmdb-cataloger",
"java-cataloger",
"apkdb-cataloger",
"go-module-binary-cataloger",
},
want: []string{
"php-composer-installed-cataloger",
"ruby-gemspec-cataloger",
"ruby-gemfile-cataloger",
},
},
{
name: "ignore 'cataloger' keyword",
patterns: []string{
"cataloger",
},
catalogers: []string{
"ruby-gemspec-cataloger",
"ruby-gemfile-cataloger",
"python-package-cataloger",
"php-composer-installed-cataloger",
"javascript-package-cataloger",
"dpkgdb-cataloger",
"rpmdb-cataloger",
"java-cataloger",
"apkdb-cataloger",
"go-module-binary-cataloger",
},
want: []string{},
},
{
name: "only some patterns match",
patterns: []string{
"cataloger",
"go-module",
},
catalogers: []string{
"ruby-gemspec-cataloger",
"ruby-gemfile-cataloger",
"python-package-cataloger",
"php-composer-installed-cataloger",
"javascript-package-cataloger",
"dpkgdb-cataloger",
"rpmdb-cataloger",
"java-cataloger",
"apkdb-cataloger",
"go-module-binary-cataloger",
},
want: []string{
"go-module-binary-cataloger",
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var catalogers []Cataloger
for _, n := range tt.catalogers {
catalogers = append(catalogers, dummy{name: n})
}
got := filterCatalogers(catalogers, tt.patterns)
var gotNames []string
for _, g := range got {
gotNames = append(gotNames, g.Name())
}
assert.ElementsMatch(t, tt.want, gotNames)
})
}
}
func Test_contains(t *testing.T) {
type args struct {
}
tests := []struct {
name string
enabledCatalogers []string
catalogerName string
want bool
}{
{
name: "keep exact match",
enabledCatalogers: []string{
"php-composer-installed-cataloger",
},
catalogerName: "php-composer-installed-cataloger",
want: true,
},
{
name: "match substring",
enabledCatalogers: []string{
"python",
},
catalogerName: "python-package-cataloger",
want: true,
},
{
name: "dont match on 'cataloger'",
enabledCatalogers: []string{
"cataloger",
},
catalogerName: "python-package-cataloger",
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.want, contains(tt.enabledCatalogers, tt.catalogerName))
})
}
}
+2 -1
View File
@@ -5,7 +5,8 @@ import (
)
type Config struct {
Search SearchConfig
Search SearchConfig
Catalogers []string
}
func DefaultConfig() Config {
+1 -1
View File
@@ -335,7 +335,7 @@ func pomProjectByParentPath(archivePath, virtualPath string, extractPaths []stri
projectByParentPath := make(map[string]pkg.PomProject)
for filePath, fileContents := range contentsOfMavenProjectFiles {
pomProject, err := parsePomXML(filePath, strings.NewReader(fileContents))
pomProject, err := parsePomXMLProject(filePath, strings.NewReader(fileContents))
if err != nil {
log.Warnf("failed to parse pom.xml virtualPath=%q path=%q: %+v", virtualPath, filePath, err)
continue
+62 -17
View File
@@ -6,34 +6,79 @@ import (
"io"
"strings"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/pkg"
"github.com/vifraa/gopom"
"golang.org/x/net/html/charset"
)
const pomXMLGlob = "*pom.xml"
const pomXMLDirGlob = "**/pom.xml"
func parsePomXML(path string, reader io.Reader) (*pkg.PomProject, error) {
var project gopom.Project
decoder := xml.NewDecoder(reader)
// prevent against warnings for "xml: encoding "iso-8859-1" declared but Decoder.CharsetReader is nil"
decoder.CharsetReader = charset.NewReaderLabel
if err := decoder.Decode(&project); err != nil {
return nil, fmt.Errorf("unable to unmarshal pom.xml: %w", err)
func parserPomXML(path string, content io.Reader) ([]*pkg.Package, []artifact.Relationship, error) {
pom, err := decodePomXML(content)
if err != nil {
return nil, nil, err
}
var pkgs []*pkg.Package
for _, dep := range pom.Dependencies {
p := newPackageFromPom(dep)
if p.Name == "" {
continue
}
pkgs = append(pkgs, p)
}
return pkgs, nil, nil
}
func parsePomXMLProject(path string, reader io.Reader) (*pkg.PomProject, error) {
project, err := decodePomXML(reader)
if err != nil {
return nil, err
}
return newPomProject(path, project), nil
}
func newPomProject(path string, p gopom.Project) *pkg.PomProject {
return &pkg.PomProject{
Path: path,
Parent: pomParent(project.Parent),
GroupID: project.GroupID,
ArtifactID: project.ArtifactID,
Version: project.Version,
Name: project.Name,
Description: cleanDescription(project.Description),
URL: project.URL,
}, nil
Parent: pomParent(p.Parent),
GroupID: p.GroupID,
ArtifactID: p.ArtifactID,
Version: p.Version,
Name: p.Name,
Description: cleanDescription(p.Description),
URL: p.URL,
}
}
func newPackageFromPom(dep gopom.Dependency) *pkg.Package {
p := &pkg.Package{
Name: dep.ArtifactID,
Version: dep.Version,
Language: pkg.Java,
Type: pkg.JavaPkg, // TODO: should we differentiate between packages from jar/war/zip versus packages from a pom.xml that were not installed yet?
MetadataType: pkg.JavaMetadataType,
FoundBy: javaPomCataloger,
}
p.Metadata = pkg.JavaMetadata{PURL: packageURL(*p)}
return p
}
func decodePomXML(content io.Reader) (project gopom.Project, err error) {
decoder := xml.NewDecoder(content)
// prevent against warnings for "xml: encoding "iso-8859-1" declared but Decoder.CharsetReader is nil"
decoder.CharsetReader = charset.NewReaderLabel
if err := decoder.Decode(&project); err != nil {
return project, fmt.Errorf("unable to unmarshal pom.xml: %w", err)
}
return project, nil
}
func pomParent(parent gopom.Parent) (result *pkg.PomParent) {
+49 -2
View File
@@ -10,7 +10,54 @@ import (
"github.com/stretchr/testify/assert"
)
func Test_parsePomXML(t *testing.T) {
func Test_parserPomXML(t *testing.T) {
tests := []struct {
input string
expected []*pkg.Package
}{
{
input: "test-fixtures/pom/pom.xml",
expected: []*pkg.Package{
{
Name: "joda-time",
Version: "2.9.2",
FoundBy: javaPomCataloger,
Language: pkg.Java,
Type: pkg.JavaPkg,
MetadataType: pkg.JavaMetadataType,
Metadata: pkg.JavaMetadata{
PURL: "pkg:maven/joda-time/joda-time@2.9.2",
},
},
{
Name: "junit",
Version: "4.12",
FoundBy: "java-pom-cataloger",
Language: pkg.Java,
Type: pkg.JavaPkg,
MetadataType: pkg.JavaMetadataType,
Metadata: pkg.JavaMetadata{
PURL: "pkg:maven/junit/junit@4.12",
},
},
},
},
}
for _, test := range tests {
t.Run(test.input, func(t *testing.T) {
fixture, err := os.Open(test.input)
assert.NoError(t, err)
actual, relationships, err := parserPomXML(fixture.Name(), fixture)
assert.NoError(t, err)
assert.Nil(t, relationships)
assert.Equal(t, test.expected, actual)
})
}
}
func Test_parsePomXMLProject(t *testing.T) {
tests := []struct {
expected pkg.PomProject
}{
@@ -37,7 +84,7 @@ func Test_parsePomXML(t *testing.T) {
fixture, err := os.Open(test.expected.Path)
assert.NoError(t, err)
actual, err := parsePomXML(fixture.Name(), fixture)
actual, err := parsePomXMLProject(fixture.Name(), fixture)
assert.NoError(t, err)
assert.Equal(t, &test.expected, actual)
+17
View File
@@ -0,0 +1,17 @@
package java
import "github.com/anchore/syft/syft/pkg/cataloger/common"
const javaPomCataloger = "java-pom-cataloger"
// NewJavaPomCataloger returns a cataloger capable of parsing
// dependencies from a pom.xml file.
// Pom files list dependencies that maybe not be locally installed yet.
func NewJavaPomCataloger() *common.GenericCataloger {
globParsers := make(map[string]common.ParserFn)
// java project files
globParsers[pomXMLDirGlob] = parserPomXML
return common.NewGenericCataloger(nil, globParsers, javaPomCataloger)
}
@@ -0,0 +1,59 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.anchore</groupId>
<artifactId>example-java-app-maven</artifactId>
<packaging>jar</packaging>
<version>0.1.0</version>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<!-- tag::joda[] -->
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.9.2</version>
</dependency>
<!-- end::joda[] -->
<!-- tag::junit[] -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<!-- end::junit[] -->
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>hello.HelloWorld</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
@@ -25,6 +25,16 @@ var (
// versionExp matches the "version" line of a yarn.lock entry and captures the version value.
// For example: version "4.10.1" (...and the value "4.10.1" is captured)
versionExp = regexp.MustCompile(`^\W+version(?:\W+"|:\W+)([\w-_.]+)"?`)
// packageURLExp matches the name and version of the dependency in yarn.lock
// from the resolved URL, including scope/namespace prefix if any.
// For example:
// `resolved "https://registry.yarnpkg.com/async/-/async-3.2.3.tgz#ac53dafd3f4720ee9e8a160628f18ea91df196c9"`
// would return "async" and "3.2.3"
//
// `resolved "https://registry.yarnpkg.com/@4lolo/resize-observer-polyfill/-/resize-observer-polyfill-1.5.2.tgz#58868fc7224506236b5550d0c68357f0a874b84b"`
// would return "@4lolo/resize-observer-polyfill" and "1.5.2"
packageURLExp = regexp.MustCompile(`^\s+resolved\s+"https://registry\.(?:yarnpkg\.com|npmjs\.org)/(.+?)/-/(?:.+?)-(\d+\..+?)\.tgz`)
)
const (
@@ -43,39 +53,37 @@ func parseYarnLock(path string, reader io.Reader) ([]*pkg.Package, []artifact.Re
scanner := bufio.NewScanner(reader)
parsedPackages := internal.NewStringSet()
currentPackage := noPackage
currentVersion := noVersion
for scanner.Scan() {
line := scanner.Text()
if currentPackage == noPackage {
// Scan until we find the next package
packageName := findPackageName(line)
if packageName == noPackage {
continue
}
if parsedPackages.Contains(packageName) {
// We don't parse repeated package declarations.
continue
if packageName := findPackageName(line); packageName != noPackage {
// When we find a new package, check if we have unsaved identifiers
if currentPackage != noPackage && currentVersion != noVersion && !parsedPackages.Contains(currentPackage+"@"+currentVersion) {
packages = append(packages, newYarnLockPackage(currentPackage, currentVersion))
parsedPackages.Add(currentPackage + "@" + currentVersion)
}
currentPackage = packageName
parsedPackages.Add(currentPackage)
} else if version := findPackageVersion(line); version != noVersion {
currentVersion = version
} else if packageName, version := findPackageAndVersion(line); packageName != noPackage && version != noVersion && !parsedPackages.Contains(packageName+"@"+version) {
packages = append(packages, newYarnLockPackage(packageName, version))
parsedPackages.Add(packageName + "@" + version)
continue
}
// We've found the package entry, now we just need the version
if version := findPackageVersion(line); version != noVersion {
packages = append(packages, newYarnLockPackage(currentPackage, version))
// Cleanup to indicate no unsaved identifiers
currentPackage = noPackage
continue
currentVersion = noVersion
}
}
// check if we have valid unsaved data after end-of-file has reached
if currentPackage != noPackage && currentVersion != noVersion && !parsedPackages.Contains(currentPackage+"@"+currentVersion) {
packages = append(packages, newYarnLockPackage(currentPackage, currentVersion))
parsedPackages.Add(currentPackage + "@" + currentVersion)
}
if err := scanner.Err(); err != nil {
return nil, nil, fmt.Errorf("failed to parse yarn.lock file: %w", err)
}
@@ -99,6 +107,14 @@ func findPackageVersion(line string) string {
return noVersion
}
func findPackageAndVersion(line string) (string, string) {
if matches := packageURLExp.FindStringSubmatch(line); len(matches) >= 2 {
return matches[1], matches[2]
}
return noPackage, noVersion
}
func newYarnLockPackage(name, version string) *pkg.Package {
return &pkg.Package{
Name: name,
@@ -9,7 +9,7 @@ import (
"github.com/stretchr/testify/require"
)
func TestParseYarnLock(t *testing.T) {
func TestParseYarnBerry(t *testing.T) {
expected := map[string]pkg.Package{
"@babel/code-frame": {
Name: "@babel/code-frame",
@@ -66,10 +66,87 @@ func TestParseYarnLock(t *testing.T) {
Type: pkg.NpmPkg,
},
}
testFixtures := []string{
"test-fixtures/yarn-berry/yarn.lock",
}
for _, file := range testFixtures {
file := file
t.Run(file, func(t *testing.T) {
t.Parallel()
fixture, err := os.Open(file)
require.NoError(t, err)
// TODO: no relationships are under test yet
actual, _, err := parseYarnLock(fixture.Name(), fixture)
require.NoError(t, err)
assertPkgsEqual(t, actual, expected)
})
}
}
func TestParseYarnLock(t *testing.T) {
expected := map[string]pkg.Package{
"@babel/code-frame": {
Name: "@babel/code-frame",
Version: "7.10.4",
Language: pkg.JavaScript,
Type: pkg.NpmPkg,
},
"@types/minimatch": {
Name: "@types/minimatch",
Version: "3.0.3",
Language: pkg.JavaScript,
Type: pkg.NpmPkg,
},
"@types/qs": {
Name: "@types/qs",
Version: "6.9.4",
Language: pkg.JavaScript,
Type: pkg.NpmPkg,
},
"ajv": {
Name: "ajv",
Version: "6.12.3",
Language: pkg.JavaScript,
Type: pkg.NpmPkg,
},
"atob": {
Name: "atob",
Version: "2.1.2",
Language: pkg.JavaScript,
Type: pkg.NpmPkg,
},
"aws-sdk": {
Name: "aws-sdk",
Version: "2.706.0",
Language: pkg.JavaScript,
Type: pkg.NpmPkg,
},
"jhipster-core": {
Name: "jhipster-core",
Version: "7.3.4",
Language: pkg.JavaScript,
Type: pkg.NpmPkg,
},
"asn1.js": {
Name: "asn1.js",
Version: "4.10.1",
Language: pkg.JavaScript,
Type: pkg.NpmPkg,
},
"something-i-made-up": {
Name: "something-i-made-up",
Version: "7.7.7",
Language: pkg.JavaScript,
Type: pkg.NpmPkg,
},
}
testFixtures := []string{
"test-fixtures/yarn/yarn.lock",
"test-fixtures/yarn-berry/yarn.lock",
}
for _, file := range testFixtures {
+22
View File
@@ -49,5 +49,27 @@ func (c *Cataloger) Catalog(resolver source.FileResolver) ([]pkg.Package, []arti
pkgs = append(pkgs, discoveredPkgs...)
}
// Additionally look for RPM manifest files to detect packages in CBL-Mariner distroless images
manifestFileMatches, err := resolver.FilesByGlob(pkg.RpmManifestGlob)
if err != nil {
return nil, nil, fmt.Errorf("failed to find rpm manifests by glob: %w", err)
}
for _, location := range manifestFileMatches {
reader, err := resolver.FileContentsByLocation(location)
if err != nil {
return nil, nil, err
}
discoveredPkgs, err := parseRpmManifest(location, reader)
internal.CloseAndLogError(reader, location.VirtualPath)
if err != nil {
return nil, nil, fmt.Errorf("unable to catalog rpm manifest=%+v: %w", location.RealPath, err)
}
pkgs = append(pkgs, discoveredPkgs...)
}
return pkgs, nil, nil
}
+1 -1
View File
@@ -14,7 +14,7 @@ import (
rpmdb "github.com/knqyf263/go-rpmdb/pkg"
)
// parseApkDb parses an "Packages" RPM DB and returns the Packages listed within it.
// parseRpmDb parses an "Packages" RPM DB and returns the Packages listed within it.
func parseRpmDB(resolver source.FilePathResolver, dbLocation source.Location, reader io.Reader) ([]pkg.Package, error) {
f, err := ioutil.TempFile("", internal.ApplicationName+"-rpmdb")
if err != nil {
@@ -0,0 +1,104 @@
package rpmdb
import (
"bufio"
"fmt"
"io"
"strconv"
"strings"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/source"
)
// Parses an entry in an RPM manifest file as used in Mariner distroless containers
// Each line is the output of :
// rpm --query --all --query-format "%{NAME}\t%{VERSION}-%{RELEASE}\t%{INSTALLTIME}\t%{BUILDTIME}\t%{VENDOR}\t%{EPOCH}\t%{SIZE}\t%{ARCH}\t%{EPOCHNUM}\t%{SOURCERPM}\n"
// https://github.com/microsoft/CBL-Mariner/blob/3df18fac373aba13a54bd02466e64969574f13af/toolkit/docs/how_it_works/5_misc.md?plain=1#L150
func parseRpmManifestEntry(entry string, location source.Location) (*pkg.Package, error) {
parts := strings.Split(entry, "\t")
if len(parts) < 10 {
return nil, fmt.Errorf("unexpected number of fields in line: %s", entry)
}
versionParts := strings.Split(parts[1], "-")
if len(versionParts) != 2 {
return nil, fmt.Errorf("unexpected version field: %s", parts[1])
}
version := versionParts[0]
release := versionParts[1]
converted, err := strconv.Atoi(parts[8])
var epoch *int
if err != nil || parts[5] == "(none)" {
epoch = nil
} else {
epoch = &converted
}
converted, err = strconv.Atoi(parts[6])
var size int
if err == nil {
size = converted
}
metadata := pkg.RpmdbMetadata{
Name: parts[0],
Version: version,
Epoch: epoch,
Arch: parts[7],
Release: release,
SourceRpm: parts[9],
Vendor: parts[4],
Size: size,
}
p := pkg.Package{
Name: parts[0],
Version: toELVersion(metadata),
Locations: source.NewLocationSet(location),
FoundBy: catalogerName,
Type: pkg.RpmPkg,
MetadataType: pkg.RpmdbMetadataType,
Metadata: metadata,
}
p.SetID()
return &p, nil
}
// Parses an RPM manifest file, as used in Mariner distroless containers, and returns the Packages listed
func parseRpmManifest(dbLocation source.Location, reader io.Reader) ([]pkg.Package, error) {
r := bufio.NewReader(reader)
allPkgs := make([]pkg.Package, 0)
for {
line, err := r.ReadString('\n')
if err != nil {
if err == io.EOF {
break
}
return nil, err
}
if line == "" {
continue
}
p, err := parseRpmManifestEntry(strings.TrimSuffix(line, "\n"), dbLocation)
if err != nil {
log.Warnf("unable to parse RPM manifest entry: %w", err)
continue
}
if !pkg.IsValid(p) {
continue
}
p.SetID()
allPkgs = append(allPkgs, *p)
}
return allPkgs, nil
}
@@ -0,0 +1,117 @@
package rpmdb
import (
"os"
"testing"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/source"
"github.com/go-test/deep"
)
func TestParseRpmManifest(t *testing.T) {
location := source.NewLocation("test-path")
fixture_path := "test-fixtures/container-manifest-2"
expected := map[string]pkg.Package{
"mariner-release": {
Name: "mariner-release",
Version: "2.0-12.cm2",
Locations: source.NewLocationSet(location),
FoundBy: catalogerName,
Type: pkg.RpmPkg,
MetadataType: pkg.RpmdbMetadataType,
Metadata: pkg.RpmdbMetadata{
Name: "mariner-release",
Epoch: nil,
Arch: "noarch",
Release: "12.cm2",
Version: "2.0",
SourceRpm: "mariner-release-2.0-12.cm2.src.rpm",
Size: 580,
Vendor: "Microsoft Corporation",
},
},
"filesystem": {
Name: "filesystem",
Version: "1.1-9.cm2",
Locations: source.NewLocationSet(location),
FoundBy: catalogerName,
Type: pkg.RpmPkg,
MetadataType: pkg.RpmdbMetadataType,
Metadata: pkg.RpmdbMetadata{
Name: "filesystem",
Epoch: nil,
Arch: "x86_64",
Release: "9.cm2",
Version: "1.1",
SourceRpm: "filesystem-1.1-9.cm2.src.rpm",
Size: 7596,
Vendor: "Microsoft Corporation",
},
},
"glibc": {
Name: "glibc",
Version: "2.35-2.cm2",
Locations: source.NewLocationSet(location),
FoundBy: catalogerName,
Type: pkg.RpmPkg,
MetadataType: pkg.RpmdbMetadataType,
Metadata: pkg.RpmdbMetadata{
Name: "glibc",
Epoch: nil,
Arch: "x86_64",
Release: "2.cm2",
Version: "2.35",
SourceRpm: "glibc-2.35-2.cm2.src.rpm",
Size: 10855265,
Vendor: "Microsoft Corporation",
},
},
"openssl-libs": {
Name: "openssl-libs",
Version: "1.1.1k-15.cm2",
Locations: source.NewLocationSet(location),
FoundBy: catalogerName,
Type: pkg.RpmPkg,
MetadataType: pkg.RpmdbMetadataType,
Metadata: pkg.RpmdbMetadata{
Name: "openssl-libs",
Epoch: nil,
Arch: "x86_64",
Release: "15.cm2",
Version: "1.1.1k",
SourceRpm: "openssl-1.1.1k-15.cm2.src.rpm",
Size: 4365048,
Vendor: "Microsoft Corporation",
},
},
}
fixture, err := os.Open(fixture_path)
if err != nil {
t.Fatalf("failed to open fixture: %+v", err)
}
actual, err := parseRpmManifest(location, fixture)
if err != nil {
t.Fatalf("failed to parse rpm manifest: %+v", err)
}
if len(actual) != 12 {
for _, a := range actual {
t.Log(" ", a)
}
t.Fatalf("unexpected package count: %d!=%d", len(actual), len(expected))
}
for _, a := range actual[0:4] {
e := expected[a.Name]
diffs := deep.Equal(a, e)
if len(diffs) > 0 {
for _, d := range diffs {
t.Errorf("diff: %+v", d)
}
}
}
}
@@ -0,0 +1,12 @@
mariner-release 2.0-12.cm2 1653816591 1653753130 Microsoft Corporation (none) 580 noarch 0 mariner-release-2.0-12.cm2.src.rpm
filesystem 1.1-9.cm2 1653816591 1653628924 Microsoft Corporation (none) 7596 x86_64 0 filesystem-1.1-9.cm2.src.rpm
glibc 2.35-2.cm2 1653816591 1653628955 Microsoft Corporation (none) 10855265 x86_64 0 glibc-2.35-2.cm2.src.rpm
openssl-libs 1.1.1k-15.cm2 1653816591 1653631609 Microsoft Corporation (none) 4365048 x86_64 0 openssl-1.1.1k-15.cm2.src.rpm
libgcc 11.2.0-2.cm2 1653816591 1650702349 Microsoft Corporation (none) 103960 x86_64 0 gcc-11.2.0-2.cm2.src.rpm
openssl 1.1.1k-15.cm2 1653816591 1653631609 Microsoft Corporation (none) 1286337 x86_64 0 openssl-1.1.1k-15.cm2.src.rpm
glibc-iconv 2.35-2.cm2 1653816591 1653628955 Microsoft Corporation (none) 8397230 x86_64 0 glibc-2.35-2.cm2.src.rpm
iana-etc 20211115-1.cm2 1653816591 1650711959 Microsoft Corporation (none) 4380680 noarch 0 iana-etc-20211115-1.cm2.src.rpm
tzdata 2022a-1.cm2 1653816591 1653752882 Microsoft Corporation (none) 1535764 noarch 0 tzdata-2022a-1.cm2.src.rpm
prebuilt-ca-certificates-base 2.0.0-3.cm2 1653816591 1653771776 Microsoft Corporation (none) 65684 noarch 1 prebuilt-ca-certificates-base-2.0.0-3.cm2.src.rpm
distroless-packages-minimal 0.1-2.cm2 1653816591 1650712132 Microsoft Corporation (none) 0 x86_64 0 distroless-packages-0.1-2.cm2.src.rpm
distroless-packages-base 0.1-2.cm2 1653816591 1650712132 Microsoft Corporation (none) 0 x86_64 0 distroless-packages-0.1-2.cm2.src.rpm
@@ -21,3 +21,9 @@ docker exec -i --tty=false generate-rpmdb-fixture bash <<-EOF
EOF
docker cp generate-rpmdb-fixture:/scratch/Packages .
docker build -o . - <<EOF
FROM mcr.microsoft.com/cbl-mariner/distroless/base:2.0 as base
FROM scratch
COPY --from=base /var/lib/rpmmanifest/container-manifest-2 .
EOF
+3
View File
@@ -12,6 +12,7 @@ const (
UnknownMetadataType MetadataType = "UnknownMetadata"
ApkMetadataType MetadataType = "ApkMetadata"
AlpmMetadataType MetadataType = "AlpmMetadata"
DpkgMetadataType MetadataType = "DpkgMetadata"
GemMetadataType MetadataType = "GemMetadata"
JavaMetadataType MetadataType = "JavaMetadata"
@@ -28,6 +29,7 @@ const (
var AllMetadataTypes = []MetadataType{
ApkMetadataType,
AlpmMetadataType,
DpkgMetadataType,
GemMetadataType,
JavaMetadataType,
@@ -44,6 +46,7 @@ var AllMetadataTypes = []MetadataType{
var MetadataTypeByName = map[MetadataType]reflect.Type{
ApkMetadataType: reflect.TypeOf(ApkMetadata{}),
AlpmMetadataType: reflect.TypeOf(AlpmMetadata{}),
DpkgMetadataType: reflect.TypeOf(DpkgMetadata{}),
GemMetadataType: reflect.TypeOf(GemMetadata{}),
JavaMetadataType: reflect.TypeOf(JavaMetadata{}),
+3
View File
@@ -16,6 +16,9 @@ import (
// rpmdb.sqlite is the sqlite format used in fedora + derivates
const RpmDBGlob = "**/var/lib/rpm/{Packages,Packages.db,rpmdb.sqlite}"
// Used in CBL-Mariner distroless images
const RpmManifestGlob = "**/var/lib/rpmmanifest/container-manifest-2"
var (
_ FileOwner = (*RpmdbMetadata)(nil)
_ urlIdentifier = (*RpmdbMetadata)(nil)
+6
View File
@@ -9,6 +9,7 @@ const (
// the full set of supported packages
UnknownPkg Type = "UnknownPackage"
ApkPkg Type = "apk"
AlpmPkg Type = "alpm"
GemPkg Type = "gem"
DebPkg Type = "deb"
RpmPkg Type = "rpm"
@@ -27,6 +28,7 @@ const (
// AllPkgs represents all supported package types
var AllPkgs = []Type{
ApkPkg,
AlpmPkg,
GemPkg,
DebPkg,
RpmPkg,
@@ -47,6 +49,8 @@ func (t Type) PackageURLType() string {
switch t {
case ApkPkg:
return "alpine"
case AlpmPkg:
return "alpm"
case GemPkg:
return packageurl.TypeGem
case DebPkg:
@@ -90,6 +94,8 @@ func TypeByName(name string) Type {
return DebPkg
case packageurl.TypeRPM:
return RpmPkg
case "alpm":
return AlpmPkg
case "alpine":
return ApkPkg
case packageurl.TypeMaven:
+4
View File
@@ -64,6 +64,10 @@ func TestTypeFromPURL(t *testing.T) {
purl: "pkg:maven/org.apache.xmlgraphics/batik-anim@1.9.1?type=zip&classifier=dist",
expected: JavaPkg,
},
{
purl: "pkg:alpm/arch/linux@5.10.0?arch=x86_64&distro=arch",
expected: AlpmPkg,
},
}
var pkgTypes []string
+19 -6
View File
@@ -1,7 +1,6 @@
package pkg
import (
"fmt"
"regexp"
"sort"
"strings"
@@ -86,12 +85,26 @@ func purlQualifiers(vars map[string]string, release *linux.Release) (q packageur
})
}
if release != nil && release.ID != "" && release.VersionID != "" {
q = append(q, packageurl.Qualifier{
Key: PURLQualifierDistro,
Value: fmt.Sprintf("%s-%s", release.ID, release.VersionID),
})
distroQualifiers := []string{}
if release == nil {
return q
}
if release.ID != "" {
distroQualifiers = append(distroQualifiers, release.ID)
}
if release.VersionID != "" {
distroQualifiers = append(distroQualifiers, release.VersionID)
} else if release.BuildID != "" {
distroQualifiers = append(distroQualifiers, release.BuildID)
}
q = append(q, packageurl.Qualifier{
Key: PURLQualifierDistro,
Value: strings.Join(distroQualifiers, "-"),
})
return q
}
+18
View File
@@ -190,6 +190,24 @@ func TestPackageURL(t *testing.T) {
expected: "pkg:maven/g.id/a@v",
},
{
name: "alpm",
distro: &linux.Release{
ID: "arch",
BuildID: "rolling",
},
pkg: Package{
Name: "linux",
Version: "5.10.0",
Type: AlpmPkg,
Metadata: AlpmMetadata{
Package: "linux",
Version: "5.10.0",
},
},
expected: "pkg:alpm/arch/linux@5.10.0?distro=arch-rolling",
},
}
var pkgTypes []string
+9 -2
View File
@@ -2,10 +2,12 @@ package cli
import (
"fmt"
"github.com/stretchr/testify/require"
"strings"
"testing"
"github.com/stretchr/testify/require"
"github.com/anchore/syft/internal/formats/template"
"github.com/anchore/syft/syft"
)
@@ -23,7 +25,12 @@ func TestAllFormatsExpressible(t *testing.T) {
require.NotEmpty(t, formats)
for _, o := range formats {
t.Run(fmt.Sprintf("format:%s", o), func(t *testing.T) {
cmd, stdout, stderr := runSyft(t, nil, "dir:./test-fixtures/image-pkg-coverage", "-o", string(o))
args := []string{"dir:./test-fixtures/image-pkg-coverage", "-o", string(o)}
if o == template.ID {
args = append(args, "-t", "test-fixtures/csv.template")
}
cmd, stdout, stderr := runSyft(t, nil, args...)
for _, traitFn := range commonAssertions {
traitFn(t, stdout, stderr, cmd.ProcessState.ExitCode())
}
+9 -1
View File
@@ -96,7 +96,7 @@ func TestPackagesCmdFlags(t *testing.T) {
name: "squashed-scope-flag",
args: []string{"packages", "-o", "json", "-s", "squashed", coverageImage},
assertions: []traitAssertion{
assertPackageCount(32),
assertPackageCount(33),
assertSuccessfulReturnCode,
},
},
@@ -227,6 +227,14 @@ func TestPackagesCmdFlags(t *testing.T) {
),
},
},
{
name: "catalogers-option",
args: []string{"packages", "-o", "json", "--catalogers", "python,ruby-gemspec", coverageImage},
assertions: []traitAssertion{
assertPackageCount(6),
assertSuccessfulReturnCode,
},
},
}
for _, test := range tests {
+4
View File
@@ -0,0 +1,4 @@
"Package","Version Installed", "Found by"
{{- range .Artifacts}}
"{{.Name}}","{{.Version}}","{{.FoundBy}}"
{{- end}}
@@ -60,6 +60,15 @@ var imageOnlyTestCases = []testCase{
"libc-utils": "0.7.2-r0",
},
},
{
name: "find java packages excluding pom.xml", // image scans can not include packages that have yet to be installed
pkgType: pkg.JavaPkg,
pkgLanguage: pkg.Java,
pkgInfo: map[string]string{
"example-java-app-maven": "0.1.0",
"joda-time": "2.9.2",
},
},
}
var dirOnlyTestCases = []testCase{
@@ -218,9 +227,27 @@ var dirOnlyTestCases = []testCase{
"System.Runtime.CompilerServices.Unsafe": "6.0.0",
},
},
{
name: "find java packages including pom.xml", // directory scans can include packages that have yet to be installed
pkgType: pkg.JavaPkg,
pkgLanguage: pkg.Java,
duplicates: 1, // joda-time is included in both pom.xml AND the .jar collection
pkgInfo: map[string]string{
"example-java-app-maven": "0.1.0",
"joda-time": "2.9.2",
"junit": "4.12",
},
},
}
var commonTestCases = []testCase{
{
name: "find alpm packages",
pkgType: pkg.AlpmPkg,
pkgInfo: map[string]string{
"pacman": "6.0.1-5",
},
},
{
name: "find rpmdb packages",
pkgType: pkg.RpmPkg,
@@ -237,15 +264,7 @@ var commonTestCases = []testCase{
"netbase": "5.4",
},
},
{
name: "find java packages",
pkgType: pkg.JavaPkg,
pkgLanguage: pkg.Java,
pkgInfo: map[string]string{
"example-java-app-maven": "0.1.0",
"joda-time": "2.9.2",
},
},
{
name: "find jenkins plugins",
pkgType: pkg.JenkinsPluginPkg,
@@ -90,7 +90,6 @@ func TestPkgCoverageImage(t *testing.T) {
pkgCount := 0
for a := range sbom.Artifacts.PackageCatalog.Enumerate(c.pkgType) {
if a.Language.String() != "" {
observedLanguages.Add(a.Language.String())
}
@@ -167,7 +166,6 @@ func TestPkgCoverageDirectory(t *testing.T) {
actualPkgCount := 0
for actualPkg := range sbom.Artifacts.PackageCatalog.Enumerate(test.pkgType) {
observedLanguages.Add(actualPkg.Language.String())
observedPkgs.Add(string(actualPkg.Type))
+9 -13
View File
@@ -42,32 +42,28 @@ func TestConvertCmd(t *testing.T) {
f, err := ioutil.TempFile("", "test-convert-sbom-")
require.NoError(t, err)
defer func() {
err := f.Close()
require.NoError(t, err)
os.Remove(f.Name())
}()
err = format.Encode(f, sbom)
require.NoError(t, err)
stdr, stdw, err := os.Pipe()
require.NoError(t, err)
originalStdout := os.Stdout
os.Stdout = stdw
ctx := context.Background()
app := &config.Application{Outputs: []string{format.ID().String()}}
// stdout reduction of test noise
rescue := os.Stdout // keep backup of the real stdout
os.Stdout, _ = os.OpenFile(os.DevNull, os.O_APPEND|os.O_WRONLY, os.ModeAppend)
defer func() {
os.Stdout = rescue
}()
err = convert.Run(ctx, app, []string{f.Name()})
require.NoError(t, err)
stdw.Close()
out, err := ioutil.ReadAll(stdr)
file, err := ioutil.ReadFile(f.Name())
require.NoError(t, err)
os.Stdout = originalStdout
formatFound := syft.IdentifyFormat(out)
formatFound := syft.IdentifyFormat(file)
if format.ID() == table.ID {
require.Nil(t, formatFound)
return
@@ -0,0 +1,22 @@
package integration
import (
"testing"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/source"
)
func TestMarinerDistroless(t *testing.T) {
sbom, _ := catalogFixtureImage(t, "image-mariner-distroless", source.SquashedScope)
expectedPkgs := 12
actualPkgs := 0
for range sbom.Artifacts.PackageCatalog.Enumerate(pkg.RpmPkg) {
actualPkgs += 1
}
if actualPkgs != expectedPkgs {
t.Errorf("unexpected number of RPM packages: %d != %d", expectedPkgs, actualPkgs)
}
}
+7 -4
View File
@@ -1,6 +1,7 @@
package integration
import (
"reflect"
"strings"
"testing"
@@ -33,6 +34,7 @@ func TestYarnPackageLockDirectory(t *testing.T) {
sbom, _ := catalogDirectory(t, "test-fixtures/yarn-lock")
foundPackages := internal.NewStringSet()
expectedPackages := internal.NewStringSet("async@0.9.2", "async@3.2.3", "merge-objects@1.0.5", "should-type@1.3.0", "@4lolo/resize-observer-polyfill@1.5.2")
for actualPkg := range sbom.Artifacts.PackageCatalog.Enumerate(pkg.NpmPkg) {
for _, actualLocation := range actualPkg.Locations.ToSlice() {
@@ -40,12 +42,13 @@ func TestYarnPackageLockDirectory(t *testing.T) {
t.Errorf("found packages from yarn.lock in node_modules: %s", actualLocation)
}
}
foundPackages.Add(actualPkg.Name)
foundPackages.Add(actualPkg.Name + "@" + actualPkg.Version)
}
// ensure that integration test commonTestCases stay in sync with the available catalogers
const expectedPackageCount = 5
if len(foundPackages) != expectedPackageCount {
t.Errorf("found the wrong set of yarn.lock packages (expected: %d, actual: %d)", expectedPackageCount, len(foundPackages))
if len(foundPackages) != len(expectedPackages) {
t.Errorf("found the wrong set of yarn.lock packages (expected: %d, actual: %d)", len(expectedPackages), len(foundPackages))
} else if !reflect.DeepEqual(foundPackages, expectedPackages) {
t.Errorf("found the wrong set of yarn.lock packages (expected: %+q, actual: %+q)", expectedPackages.ToSlice(), foundPackages.ToSlice())
}
}
@@ -0,0 +1 @@
FROM mcr.microsoft.com/cbl-mariner/distroless/base:2.0.202205275@sha256:f550c5428df17b145851ad75983aca6d613ad4b51ca7983b2a83e67d0ac91a5d
@@ -0,0 +1,59 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.anchore</groupId>
<artifactId>example-java-app-maven</artifactId>
<packaging>jar</packaging>
<version>0.1.0</version>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<!-- tag::joda[] -->
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.9.2</version>
</dependency>
<!-- end::joda[] -->
<!-- tag::junit[] -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<!-- end::junit[] -->
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>hello.HelloWorld</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
@@ -0,0 +1,57 @@
%NAME%
pacman
%VERSION%
6.0.1-5
%BASE%
pacman
%DESC%
A library-based package manager with dependency support
%URL%
https://www.archlinux.org/pacman/
%ARCH%
x86_64
%BUILDDATE%
1652116331
%INSTALLDATE%
1654074247
%PACKAGER%
Morten Linderud <foxboron@archlinux.org>
%SIZE%
4925474
%REASON%
1
%GROUPS%
base-devel
%LICENSE%
GPL
%VALIDATION%
pgp
%DEPENDS%
bash
glibc
libarchive
curl
gpgme
pacman-mirrorlist
archlinux-keyring
%OPTDEPENDS%
perl-locale-gettext: translation support in makepkg-template
%PROVIDES%
libalpm.so=13-64
@@ -0,0 +1,30 @@
%FILES%
etc/
etc/makepkg.conf
etc/pacman.conf
usr/
usr/bin/
usr/bin/makepkg
usr/bin/makepkg-template
usr/bin/pacman
usr/bin/pacman-conf
usr/bin/pacman-db-upgrade
usr/bin/pacman-key
usr/bin/repo-add
usr/bin/repo-elephant
usr/bin/repo-remove
usr/bin/testpkg
usr/bin/vercmp
usr/include/
usr/include/alpm.h
usr/include/alpm_list.h
usr/lib/
usr/lib/libalpm.so
usr/lib/libalpm.so.13
usr/lib/libalpm.so.13.0.1
usr/lib/pkgconfig/
usr/lib/pkgconfig/libalpm.pc
%BACKUP%
etc/pacman.conf de541390e52468165b96511c4665bff4
etc/makepkg.conf 79fce043df7dfc676ae5ecb903762d8b
-1
View File
@@ -1 +0,0 @@
../semver/bin/semver.js
-22
View File
@@ -1,22 +0,0 @@
{
"systemParams": "darwin-x64-83",
"modulesFolders": [
"node_modules"
],
"flags": [],
"linkedModules": [],
"topLevelPatterns": [
"collapse-white-space@^2.0.0",
"merge-objects@^1.0.5",
"semver@^7.3.5"
],
"lockfileEntries": {
"collapse-white-space@^2.0.0": "https://registry.yarnpkg.com/collapse-white-space/-/collapse-white-space-2.0.0.tgz#37d8521344cdd36635db180a7c83e9d515ac281b",
"lru-cache@^6.0.0": "https://registry.yarnpkg.com/lru-cache/-/lru-cache-6.0.0.tgz#6d6fe6570ebd96aaf90fcad1dafa3b2566db3a94",
"merge-objects@^1.0.5": "https://registry.yarnpkg.com/merge-objects/-/merge-objects-1.0.5.tgz#ad923ff3910091acc1438f53eb75b8f37d862a86",
"semver@^7.3.5": "https://registry.yarnpkg.com/semver/-/semver-7.3.5.tgz#0b621c879348d8998e4b0e4be94b3f12e6018ef7",
"yallist@^4.0.0": "https://registry.yarnpkg.com/yallist/-/yallist-4.0.0.tgz#9bb92790d9c0effec63be73519e11a35019a3a72"
},
"files": [],
"artifacts": {}
}
+19
View File
@@ -0,0 +1,19 @@
Copyright (c) 2010-2018 Caolan McMahon
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
+17
View File
@@ -0,0 +1,17 @@
{
"name": "async",
"main": "dist/async.js",
"ignore": [
"bower_components",
"lib",
"test",
"node_modules",
"perf",
"support",
"**/.*",
"*.config.js",
"*.json",
"index.js",
"Makefile"
]
}
@@ -0,0 +1,80 @@
{
"name": "async",
"description": "Higher-order functions and common patterns for asynchronous code",
"version": "3.2.3",
"main": "dist/async.js",
"author": "Caolan McMahon",
"homepage": "https://caolan.github.io/async/",
"repository": {
"type": "git",
"url": "https://github.com/caolan/async.git"
},
"bugs": {
"url": "https://github.com/caolan/async/issues"
},
"keywords": [
"async",
"callback",
"module",
"utility"
],
"devDependencies": {
"babel-core": "^6.26.3",
"babel-eslint": "^8.2.6",
"babel-minify": "^0.5.0",
"babel-plugin-add-module-exports": "^0.2.1",
"babel-plugin-istanbul": "^5.1.4",
"babel-plugin-syntax-async-generators": "^6.13.0",
"babel-plugin-transform-es2015-modules-commonjs": "^6.26.2",
"babel-preset-es2015": "^6.3.13",
"babel-preset-es2017": "^6.22.0",
"babel-register": "^6.26.0",
"babelify": "^8.0.0",
"benchmark": "^2.1.1",
"bluebird": "^3.4.6",
"browserify": "^16.2.3",
"chai": "^4.2.0",
"cheerio": "^0.22.0",
"coveralls": "^3.0.4",
"es6-promise": "^2.3.0",
"eslint": "^6.0.1",
"eslint-plugin-prefer-arrow": "^1.1.5",
"fs-extra": "^0.26.7",
"jsdoc": "^3.6.2",
"karma": "^4.1.0",
"karma-browserify": "^5.3.0",
"karma-edge-launcher": "^0.4.2",
"karma-firefox-launcher": "^1.1.0",
"karma-junit-reporter": "^1.2.0",
"karma-mocha": "^1.2.0",
"karma-mocha-reporter": "^2.2.0",
"karma-safari-launcher": "^1.0.0",
"mocha": "^6.1.4",
"mocha-junit-reporter": "^1.18.0",
"native-promise-only": "^0.8.0-a",
"nyc": "^14.1.1",
"rollup": "^0.63.4",
"rollup-plugin-node-resolve": "^2.0.0",
"rollup-plugin-npm": "^2.0.0",
"rsvp": "^3.0.18",
"semver": "^5.5.0",
"yargs": "^11.0.0"
},
"scripts": {
"coverage": "nyc npm run mocha-node-test -- --grep @nycinvalid --invert",
"coveralls": "npm run coverage && nyc report --reporter=text-lcov | coveralls",
"jsdoc": "jsdoc -c ./support/jsdoc/jsdoc.json && node support/jsdoc/jsdoc-fix-html.js",
"lint": "eslint --fix lib/ test/ perf/memory.js perf/suites.js perf/benchmark.js support/build/ support/*.js karma.conf.js",
"mocha-browser-test": "karma start",
"mocha-node-test": "mocha",
"mocha-test": "npm run mocha-node-test && npm run mocha-browser-test",
"test": "npm run lint && npm run mocha-node-test"
},
"license": "MIT",
"nyc": {
"exclude": [
"test"
]
},
"module": "dist/async.mjs"
}
@@ -1,7 +0,0 @@
/**
* Collapse whitespace to a single space.
*
* @param {string} value
* @returns {string}
*/
export function collapseWhiteSpace(value: string): string;
@@ -1,9 +0,0 @@
/**
* Collapse whitespace to a single space.
*
* @param {string} value
* @returns {string}
*/
export function collapseWhiteSpace(value) {
return String(value).replace(/\s+/g, ' ')
}
@@ -1,22 +0,0 @@
(The MIT License)
Copyright (c) 2015 Titus Wormer <tituswormer@gmail.com>
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -1,69 +0,0 @@
{
"name": "collapse-white-space",
"version": "2.0.0",
"description": "Replace multiple white space characters with a single space",
"license": "MIT",
"keywords": [
"collapse",
"white",
"space"
],
"repository": "wooorm/collapse-white-space",
"bugs": "https://github.com/wooorm/collapse-white-space/issues",
"funding": {
"type": "github",
"url": "https://github.com/sponsors/wooorm"
},
"author": "Titus Wormer <tituswormer@gmail.com> (https://wooorm.com)",
"contributors": [
"Titus Wormer <tituswormer@gmail.com> (https://wooorm.com)"
],
"sideEffects": false,
"type": "module",
"main": "index.js",
"types": "index.d.ts",
"files": [
"index.d.ts",
"index.js"
],
"devDependencies": {
"@types/tape": "^4.0.0",
"c8": "^7.0.0",
"prettier": "^2.0.0",
"remark-cli": "^9.0.0",
"remark-preset-wooorm": "^8.0.0",
"rimraf": "^3.0.0",
"tape": "^5.0.0",
"typescript": "^4.0.0",
"xo": "^0.38.0"
},
"scripts": {
"prepublishOnly": "npm run build",
"format": "remark . -qfo && prettier . -w --loglevel warn && xo --fix",
"prebuild": "rimraf \"*.d.ts\"",
"build": "tsc",
"test-api": "node test",
"test-coverage": "c8 --check-coverage --branches 100 --functions 100 --lines 100 --statements 100 --reporter lcov node test.js",
"test": "npm run format && npm run build && npm run test-coverage"
},
"prettier": {
"tabWidth": 2,
"useTabs": false,
"singleQuote": true,
"bracketSpacing": false,
"semi": false,
"trailingComma": "none"
},
"xo": {
"prettier": true,
"rules": {
"no-var": "off",
"prefer-arrow-callback": "off"
}
},
"remarkConfig": {
"plugins": [
"preset-wooorm"
]
}
}
@@ -1,65 +0,0 @@
# collapse-white-space
[![Build][build-badge]][build]
[![Coverage][coverage-badge]][coverage]
[![Downloads][downloads-badge]][downloads]
[![Size][size-badge]][size]
Replace multiple whitespace characters with a single space.
## Install
This package is ESM only: Node 12+ is needed to use it and it must be `import`ed
instead of `require`d.
[npm][]:
```sh
npm install collapse-white-space
```
## Use
```js
import {collapseWhiteSpace} from 'collapse-white-space'
collapseWhiteSpace('\tfoo \n\tbar \t\r\nbaz') //=> ' foo bar baz'
```
## API
This package exports the following identifiers: `collapseWhiteSpace`.
There is no default export.
### `collapseWhiteSpace(value)`
Replace multiple whitespace characters in `value` (`string`) with a single
space.
## License
[MIT][license] © [Titus Wormer][author]
<!-- Definitions -->
[build-badge]: https://github.com/wooorm/collapse-white-space/workflows/main/badge.svg
[build]: https://github.com/wooorm/collapse-white-space/actions
[coverage-badge]: https://img.shields.io/codecov/c/github/wooorm/collapse-white-space.svg
[coverage]: https://codecov.io/github/wooorm/collapse-white-space
[downloads-badge]: https://img.shields.io/npm/dm/collapse-white-space.svg
[downloads]: https://www.npmjs.com/package/collapse-white-space
[size-badge]: https://img.shields.io/bundlephobia/minzip/collapse-white-space.svg
[size]: https://bundlephobia.com/result?p=collapse-white-space
[npm]: https://docs.npmjs.com/cli/install
[license]: license
[author]: https://wooorm.com
File diff suppressed because it is too large Load Diff
@@ -1,15 +0,0 @@
The ISC License
Copyright (c) Isaac Z. Schlueter and Contributors
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
@@ -1,166 +0,0 @@
# lru cache
A cache object that deletes the least-recently-used items.
[![Build Status](https://travis-ci.org/isaacs/node-lru-cache.svg?branch=master)](https://travis-ci.org/isaacs/node-lru-cache) [![Coverage Status](https://coveralls.io/repos/isaacs/node-lru-cache/badge.svg?service=github)](https://coveralls.io/github/isaacs/node-lru-cache)
## Installation:
```javascript
npm install lru-cache --save
```
## Usage:
```javascript
var LRU = require("lru-cache")
, options = { max: 500
, length: function (n, key) { return n * 2 + key.length }
, dispose: function (key, n) { n.close() }
, maxAge: 1000 * 60 * 60 }
, cache = new LRU(options)
, otherCache = new LRU(50) // sets just the max size
cache.set("key", "value")
cache.get("key") // "value"
// non-string keys ARE fully supported
// but note that it must be THE SAME object, not
// just a JSON-equivalent object.
var someObject = { a: 1 }
cache.set(someObject, 'a value')
// Object keys are not toString()-ed
cache.set('[object Object]', 'a different value')
assert.equal(cache.get(someObject), 'a value')
// A similar object with same keys/values won't work,
// because it's a different object identity
assert.equal(cache.get({ a: 1 }), undefined)
cache.reset() // empty the cache
```
If you put more stuff in it, then items will fall out.
If you try to put an oversized thing in it, then it'll fall out right
away.
## Options
* `max` The maximum size of the cache, checked by applying the length
function to all values in the cache. Not setting this is kind of
silly, since that's the whole purpose of this lib, but it defaults
to `Infinity`. Setting it to a non-number or negative number will
throw a `TypeError`. Setting it to 0 makes it be `Infinity`.
* `maxAge` Maximum age in ms. Items are not pro-actively pruned out
as they age, but if you try to get an item that is too old, it'll
drop it and return undefined instead of giving it to you.
Setting this to a negative value will make everything seem old!
Setting it to a non-number will throw a `TypeError`.
* `length` Function that is used to calculate the length of stored
items. If you're storing strings or buffers, then you probably want
to do something like `function(n, key){return n.length}`. The default is
`function(){return 1}`, which is fine if you want to store `max`
like-sized things. The item is passed as the first argument, and
the key is passed as the second argumnet.
* `dispose` Function that is called on items when they are dropped
from the cache. This can be handy if you want to close file
descriptors or do other cleanup tasks when items are no longer
accessible. Called with `key, value`. It's called *before*
actually removing the item from the internal cache, so if you want
to immediately put it back in, you'll have to do that in a
`nextTick` or `setTimeout` callback or it won't do anything.
* `stale` By default, if you set a `maxAge`, it'll only actually pull
stale items out of the cache when you `get(key)`. (That is, it's
not pre-emptively doing a `setTimeout` or anything.) If you set
`stale:true`, it'll return the stale value before deleting it. If
you don't set this, then it'll return `undefined` when you try to
get a stale entry, as if it had already been deleted.
* `noDisposeOnSet` By default, if you set a `dispose()` method, then
it'll be called whenever a `set()` operation overwrites an existing
key. If you set this option, `dispose()` will only be called when a
key falls out of the cache, not when it is overwritten.
* `updateAgeOnGet` When using time-expiring entries with `maxAge`,
setting this to `true` will make each item's effective time update
to the current time whenever it is retrieved from cache, causing it
to not expire. (It can still fall out of cache based on recency of
use, of course.)
## API
* `set(key, value, maxAge)`
* `get(key) => value`
Both of these will update the "recently used"-ness of the key.
They do what you think. `maxAge` is optional and overrides the
cache `maxAge` option if provided.
If the key is not found, `get()` will return `undefined`.
The key and val can be any value.
* `peek(key)`
Returns the key value (or `undefined` if not found) without
updating the "recently used"-ness of the key.
(If you find yourself using this a lot, you *might* be using the
wrong sort of data structure, but there are some use cases where
it's handy.)
* `del(key)`
Deletes a key out of the cache.
* `reset()`
Clear the cache entirely, throwing away all values.
* `has(key)`
Check if a key is in the cache, without updating the recent-ness
or deleting it for being stale.
* `forEach(function(value,key,cache), [thisp])`
Just like `Array.prototype.forEach`. Iterates over all the keys
in the cache, in order of recent-ness. (Ie, more recently used
items are iterated over first.)
* `rforEach(function(value,key,cache), [thisp])`
The same as `cache.forEach(...)` but items are iterated over in
reverse order. (ie, less recently used items are iterated over
first.)
* `keys()`
Return an array of the keys in the cache.
* `values()`
Return an array of the values in the cache.
* `length`
Return total length of objects in cache taking into account
`length` options function.
* `itemCount`
Return total quantity of objects currently in cache. Note, that
`stale` (see options) items are returned as part of this item
count.
* `dump()`
Return an array of the cache entries ready for serialization and usage
with 'destinationCache.load(arr)`.
* `load(cacheEntriesArray)`
Loads another cache entries array, obtained with `sourceCache.dump()`,
into the cache. The destination cache is reset before loading new entries
* `prune()`
Manually iterates over the entire cache proactively pruning old entries

Some files were not shown because too many files have changed in this diff Show More