Compare commits

...
6 Commits
Author SHA1 Message Date
Alex Goodman f24bbc1838 Deduplicate packages across multiple container image layers (#930) 2022-03-31 15:45:51 -04:00
Eric LarssenandAlex Goodman cb3e73e308 Add dart support (#919)
Co-authored-by: Alex Goodman <alex.goodman@anchore.com>
2022-03-31 15:44:55 -04:00
Alex Goodman f157d7a862 Pull from DockerHub fails for public images when using SSO (#928) 2022-03-30 17:32:49 +00:00
Dan Luhring 028cd9e27e Fix nil pointer dereference in directory resolver's indexPath method (#924)
* Add failing test for dir resolver panic

Signed-off-by: Dan Luhring <dan+github@luhrings.com>

* Fix panic

Signed-off-by: Dan Luhring <dan+github@luhrings.com>
2022-03-28 13:15:09 -04:00
Dan Luhring 5549939cc6 Fixups and clarifications in README (#920)
Signed-off-by: Dan Luhring <dan+github@luhrings.com>
2022-03-25 11:36:43 -04:00
briankoe741 47ea910868 Remove announcement for OSS Meetup (#915)
Proposing changes to remove our 3/23 meetup

Signed-off-by: Dan Luhring <dan+github@luhrings.com>
2022-03-25 00:17:14 +00:00
87 changed files with 3142 additions and 466 deletions
+57 -67
View File
@@ -9,15 +9,7 @@
[![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/anchore/syft/blob/main/LICENSE)
[![Slack Invite](https://img.shields.io/badge/Slack-Join-blue?logo=slack)](https://anchore.com/slack)
A CLI tool and Go library for generating a Software Bill of Materials (SBOM) from container images and filesystems. Exceptional for vulnerability detection when used with a scanner tool like [Grype](https://github.com/anchore/grype).
### Join our Virtual OSS Meetup!
You are invited to join us on March 23rd, 11AM-Noon PT for our virtual open source meetup.
Hosts Priyanka “Pinky” Ravi of Weaveworks and Dan Luhring from Anchore OSS will explore how open source project Flux is utilizing Syft-generated SBOMs to secure the software supply chain.
[Register here!](https://get.anchore.com/anchore-oss-meetup-mar-23-2022/)
A CLI tool and Go library for generating a Software Bill of Materials (SBOM) from container images and filesystems. Exceptional for vulnerability detection when used with a scanner like [Grype](https://github.com/anchore/grype).
### Join our community meetings!
@@ -28,15 +20,26 @@ Hosts Priyanka “Pinky” Ravi of Weaveworks and Dan Luhring from Anchore OSS w
![syft-demo](https://user-images.githubusercontent.com/590471/90277200-2a253000-de33-11ea-893f-32c219eea11a.gif)
## Features
- Catalog container images and filesystems to discover packages and libraries.
- Generate in-toto attestations where an SBOM is included as the payload.
- Supports packages and libraries from various ecosystems (APK, DEB, RPM, Ruby Bundles, Python Wheel/Egg/requirements.txt, JavaScript NPM/Yarn, Java JAR/EAR/WAR/PAR/SAR, Jenkins plugins JPI/HPI, Go modules, PHP Composer)
- Linux distribution identification (supports Alpine, BusyBox, CentOS/RedHat, Debian/Ubuntu flavored distributions)
- Supports Docker and OCI image formats
- Direct support for [Grype](https://github.com/anchore/grype), a fast and powerful vulnerability matcher.
- Generates SBOMs for container images, filesystems, archives, and more to discover packages and libraries
- Supports OCI and Docker image formats
- Linux distribution identification
- Works seamlessly with [Grype](https://github.com/anchore/grype) (a fast, modern vulnerability scanner)
- Able to create signed SBOM attestations using the [in-toto specification](https://github.com/in-toto/attestation/blob/main/spec/README.md)
### Supported Ecosystems
If you encounter an issue, please [let us know using the issue tracker](https://github.com/anchore/syft/issues).
- Alpine (apk)
- Dart (pubs)
- Debian (dpkg)
- Go (go.mod, Go binaries)
- Java (jar, ear, war, par, sar)
- JavaScript (npm, yarn)
- Jenkins Plugins (jpi, hpi)
- PHP (composer)
- Python (wheel, egg, poetry, requirements.txt)
- Red Hat (rpm)
- Ruby (gem)
- Rust (cargo.lock)
## Installation
@@ -61,15 +64,13 @@ brew install syft
### Nix
**Note**: nix packaging of Syft is [community maintained](https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/admin/syft/default.nix)
Also syft is currently only in the [unstable channel](https://nixos.wiki/wiki/Nix_channels#The_official_channels) awaiting the `22.05` release
**Note**: Nix packaging of Syft is [community maintained](https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/admin/syft/default.nix). Syft is currently only in the [unstable channel](https://nixos.wiki/wiki/Nix_channels#The_official_channels) awaiting the `22.05` release
```bash
nix-env -i syft
```
... or, just try it out in an ephemeral nix shell
... or, just try it out in an ephemeral nix shell:
```bash
nix-shell -p syft
@@ -78,33 +79,28 @@ nix-shell -p syft
## Getting started
#### SBOM
To generate an SBOM for an OCI image:
To generate an SBOM for a container image:
```
syft <image>
```
**Note**: This is equivalent to specifying the `packages` subcommand:
```
syft packages <image>
```
The above output includes only software that is visible in the container (i.e., the squashed representation of the image).
To include software from all image layers in the SBOM, regardless of its presence in the final image, provide `--scope all-layers`:
The above output includes only software that is visible in the container (i.e., the squashed representation of the image). To include software from all image layers in the SBOM, regardless of its presence in the final image, provide `--scope all-layers`:
```
syft packages <image> --scope all-layers
syft <image> --scope all-layers
```
#### SBOM Attestation
To generate an attested SBOM for an OCI image as the predicate of an in-toto attestation
#### SBOM attestation
To generate an SBOM attestation for a container image:
```
syft attest --output [FORMAT] --key [KEY] [SOURCE] [flags]
```
The above output is in the form of the [DSSE envelope](https://github.com/secure-systems-lab/dsse/blob/master/envelope.md#dsse-envelope).
The payload is a base64 encoded `in-toto` statement with the SBOM as the predicate, the payload type is `application/vnd.in-toto+json`, and the signatures array is populated
with the contents needed for public key verification. For details on workflows using this command see [here](#adding-an-sbom-to-an-image-as-an-attestation-using-syft).
The payload is a base64 encoded `in-toto` statement with the generated SBOM as the predicate. For details on workflows using this command see [here](#adding-an-sbom-to-an-image-as-an-attestation-using-syft).
### Supported sources
@@ -112,10 +108,10 @@ Syft can generate a SBOM from a variety of sources:
```
# catalog a container image archive (from the result of `docker image save ...`, `podman save ...`, or `skopeo copy` commands)
syft packages path/to/image.tar
syft path/to/image.tar
# catalog a directory
syft packages path/to/dir
syft path/to/dir
```
Sources can be explicitly provided with a scheme:
@@ -154,7 +150,7 @@ The output format for Syft is configurable as well using the
`-o` (or `--output`) option:
```
syft packages <image> -o <format>
syft <image> -o <format>
```
Where the `formats` available are:
@@ -172,20 +168,16 @@ Syft can also output _multiple_ files in differing formats by appending
`=<file>` to the option, for example to output Syft JSON and SPDX JSON:
```shell
syft packages <image> -o json=sbom.syft.json -o spdx-json=sbom.spdx.json
syft <image> -o json=sbom.syft.json -o spdx-json=sbom.spdx.json
```
## Private Registry Authentication
### Local Docker Credentials
When a container runtime is not present, Syft can still utilize credentials configured in common credential sources (such as `~/.docker/config.json`).
It will pull images from private registries using these credentials. The config file is where your credentials are stored when authenticating with private registries via some command like `docker login`.
For more information see the `go-containerregistry` [documentation](https://github.com/google/go-containerregistry/tree/main/pkg/authn).
When a container runtime is not present, Syft can still utilize credentials configured in common credential sources (such as `~/.docker/config.json`). It will pull images from private registries using these credentials. The config file is where your credentials are stored when authenticating with private registries via some command like `docker login`. For more information see the `go-containerregistry` [documentation](https://github.com/google/go-containerregistry/tree/main/pkg/authn).
An example `config.json` looks something like this:
```
// config.json
```json
{
"auths": {
"registry.example.com": {
@@ -198,15 +190,17 @@ An example `config.json` looks something like this:
You can run the following command as an example. It details the mount/environment configuration a container needs to access a private registry:
`docker run -v ./config.json:/config/config.json -e "DOCKER_CONFIG=/config" anchore/syft:latest <private_image>`
```
docker run -v ./config.json:/config/config.json -e "DOCKER_CONFIG=/config" anchore/syft:latest <private_image>
```
### Docker Credentials in Kubernetes
The below section shows a simple workflow on how to mount this config file as a secret into a container on kubernetes.
1. Create a secret. The value of `config.json` is important. It refers to the specification detailed [here](https://github.com/google/go-containerregistry/tree/main/pkg/authn#the-config-file).
Below this section is the `secret.yaml` file that the pod configuration will consume as a volume.
The key `config.json` is important. It will end up being the name of the file when mounted into the pod.
```
Here's a simple workflow to mount this config file as a secret into a container on Kubernetes.
1. Create a secret. The value of `config.json` is important. It refers to the specification detailed [here](https://github.com/google/go-containerregistry/tree/main/pkg/authn#the-config-file). Below this section is the `secret.yaml` file that the pod configuration will consume as a volume. The key `config.json` is important. It will end up being the name of the file when mounted into the pod.
```yaml
# secret.yaml
apiVersion: v1
@@ -221,11 +215,9 @@ The key `config.json` is important. It will end up being the name of the file wh
`kubectl apply -f secret.yaml`
2. Create your pod running syft. The env `DOCKER_CONFIG` is important because it advertises where to look for the credential file.
In the below example, setting `DOCKER_CONFIG=/config` informs syft that credentials can be found at `/config/config.json`.
This is why we used `config.json` as the key for our secret. When mounted into containers the secrets' key is used as the filename.
The `volumeMounts` section mounts our secret to `/config`. The `volumes` section names our volume and leverages the secret we created in step one.
```
2. Create your pod running syft. The env `DOCKER_CONFIG` is important because it advertises where to look for the credential file. In the below example, setting `DOCKER_CONFIG=/config` informs syft that credentials can be found at `/config/config.json`. This is why we used `config.json` as the key for our secret. When mounted into containers the secrets' key is used as the filename. The `volumeMounts` section mounts our secret to `/config`. The `volumes` section names our volume and leverages the secret we created in step one.
```yaml
# pod.yaml
apiVersion: v1
@@ -254,10 +246,9 @@ The `volumeMounts` section mounts our secret to `/config`. The `volumes` section
`kubectl apply -f pod.yaml`
3. The user can now run `kubectl logs syft-private-registry-demo`. The logs should show the syft analysis for the `<private_image>` provided in the pod configuration.
3. The user can now run `kubectl logs syft-private-registry-demo`. The logs should show the Syft analysis for the `<private_image>` provided in the pod configuration.
Using the above information, users should be able to configure private registry access without having to do so in the `grype` or `syft` configuration files.
They will also not be dependent on a docker daemon, (or some other runtime software) for registry configuration and access.
Using the above information, users should be able to configure private registry access without having to do so in the `grype` or `syft` configuration files. They will also not be dependent on a Docker daemon, (or some other runtime software) for registry configuration and access.
## Configuration
@@ -472,26 +463,25 @@ anchore:
`syft attest --output [FORMAT] --key [KEY] [SOURCE] [flags]`
SBOMs themselves can serve as input to different analysis tools. The Anchore organization offers the vulnerability scanner
[grype](https://github.com/anchore/grype) as one such tool.
One of the foundational approaches to "trust" between tools is for producers to use the artifacts generated by syft as attestations to their images.
The DSSE output of `syft attest` can be used with the [cosign](https://github.com/sigstore/cosign) tool to attach an attestation to an image.
SBOMs themselves can serve as input to different analysis tools. [Grype](https://github.com/anchore/grype), a vulnerability scanner CLI tool from Anchore, is one such tool. Publishers of container images can use attestations to enable their consumers to trust Syft-generated SBOM descriptions of those container images. To create and provide these attestations, image publishers can run `syft attest` in conjunction with the [cosign](https://github.com/sigstore/cosign) tool to attach SBOM attestations to their images.
#### Example attest
#### Example attestation
Note for the following example replace `docker.io/image:latest` with an image you own. You should also have push access to
its remote reference. Replace $MY_PRIVATE_KEY with a private key you own or have generated with cosign.
its remote reference. Replace `$MY_PRIVATE_KEY` with a private key you own or have generated with cosign.
```bash
syft attest --key $MY_PRIVATE_KEY docker.io/image:latest > image_latest_sbom_attestation.json
cosign attach attestation --attestation image_latest_sbom_attestation.json docker.io/image:latest
```
Verify the new attestation exists on your image
Verify the new attestation exists on your image.
```bash
cosign verify-attestation -key $MY_PUBLIC_KEY docker.io/image:latest | jq '.payload | @base64d | .payload | fromjson | .predicate'
```
You should see this output along with the attached SBOM.
You should see this output along with the attached SBOM:
```
Verification for docker.io/image:latest --
The following checks were performed on each of these signatures:
+1 -1
View File
@@ -14,7 +14,7 @@ require (
github.com/anchore/go-testutils v0.0.0-20200925183923-d5f45b0d3c04
github.com/anchore/go-version v1.2.2-0.20200701162849-18adb9c92b9b
github.com/anchore/packageurl-go v0.1.1-0.20220314153042-1bcd40e5206b
github.com/anchore/stereoscope v0.0.0-20220323180633-4bdc70599cde
github.com/anchore/stereoscope v0.0.0-20220330165332-7fc73ee7b0f0
github.com/antihax/optional v1.0.0
github.com/bmatcuk/doublestar/v4 v4.0.2
github.com/dustin/go-humanize v1.0.0
+2 -3
View File
@@ -280,8 +280,8 @@ github.com/anchore/go-version v1.2.2-0.20200701162849-18adb9c92b9b h1:e1bmaoJfZV
github.com/anchore/go-version v1.2.2-0.20200701162849-18adb9c92b9b/go.mod h1:Bkc+JYWjMCF8OyZ340IMSIi2Ebf3uwByOk6ho4wne1E=
github.com/anchore/packageurl-go v0.1.1-0.20220314153042-1bcd40e5206b h1:YJWYt/6KQXR9JR46lLHrTTYi8rcye42tKcyjREA/hvA=
github.com/anchore/packageurl-go v0.1.1-0.20220314153042-1bcd40e5206b/go.mod h1:Blo6OgJNiYF41ufcgHKkbCKF2MDOMlrqhXv/ij6ocR4=
github.com/anchore/stereoscope v0.0.0-20220323180633-4bdc70599cde h1:yOi6g1D2q3WwzID0QQyF+x9VkzUyGSJy3cEYbp7ZOyk=
github.com/anchore/stereoscope v0.0.0-20220323180633-4bdc70599cde/go.mod h1:OVVJ4y/L26pspeeHNvNa7GeQfoLPWxJe13iODl9x5l4=
github.com/anchore/stereoscope v0.0.0-20220330165332-7fc73ee7b0f0 h1:mObz7bepZ6DtbIrsB2mxuOE9XEYmVtA/P/EDlKwfbjs=
github.com/anchore/stereoscope v0.0.0-20220330165332-7fc73ee7b0f0/go.mod h1:yoCLUZY0k/pYLNIy0L80p2Ko0PKVNXm8rHtgxp4OiSc=
github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8=
github.com/andybalholm/brotli v1.0.0/go.mod h1:loMXtMfwqflxFJPmdbJO0a3KNoPuLBgiu3qAvBg8x/Y=
github.com/andybalholm/brotli v1.0.1/go.mod h1:loMXtMfwqflxFJPmdbJO0a3KNoPuLBgiu3qAvBg8x/Y=
@@ -1877,7 +1877,6 @@ github.com/sassoftware/relic v0.0.0-20210427151427-dfb082b79b74 h1:sUNzanSKA9z/h
github.com/sassoftware/relic v0.0.0-20210427151427-dfb082b79b74/go.mod h1:YlB8wFIZmFLZ1JllNBfSURzz52fBxbliNgYALk1UDmk=
github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0=
github.com/sclevine/spec v1.4.0/go.mod h1:LvpgJaFyvQzRvc1kaDs0bulYwzC70PbiYjC4QnFHkOM=
github.com/scylladb/go-set v1.0.2/go.mod h1:DkpGd78rljTxKAnTDPFqXSGxvETQnJyuSOQwsHycqfs=
github.com/scylladb/go-set v1.0.3-0.20200225121959-cc7b2070d91e h1:7q6NSFZDeGfvvtIRwBrU/aegEYJYmvev0cHAwo17zZQ=
github.com/scylladb/go-set v1.0.3-0.20200225121959-cc7b2070d91e/go.mod h1:DkpGd78rljTxKAnTDPFqXSGxvETQnJyuSOQwsHycqfs=
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
+3 -3
View File
@@ -56,14 +56,14 @@ func sbomFixture() sbom.SBOM {
Name: "name",
Version: "version",
FoundBy: "foundBy",
Locations: []source.Location{
{
Locations: source.NewLocationSet(
source.Location{
Coordinates: source.Coordinates{
RealPath: "path",
FileSystemID: "layerID",
},
},
},
),
Licenses: []string{"license"},
Language: pkg.Python,
Type: pkg.PythonPkg,
+1 -1
View File
@@ -6,5 +6,5 @@ const (
// JSONSchemaVersion is the current schema version output by the JSON encoder
// This is roughly following the "SchemaVer" guidelines for versioning the JSON schema. Please see schema/json/README.md for details on how to increment.
JSONSchemaVersion = "3.2.1"
JSONSchemaVersion = "3.2.2"
)
@@ -13,8 +13,9 @@ import (
func encodeComponent(p pkg.Package) cyclonedx.Component {
props := encodeProperties(p, "syft:package")
props = append(props, encodeCPEs(p)...)
if len(p.Locations) > 0 {
props = append(props, encodeProperties(p.Locations, "syft:location")...)
locations := p.Locations.ToSlice()
if len(locations) > 0 {
props = append(props, encodeProperties(locations, "syft:location")...)
}
if hasMetadata(p) {
props = append(props, encodeProperties(p.Metadata, "syft:metadata")...)
@@ -73,10 +74,13 @@ func decodeComponent(c *cyclonedx.Component) *pkg.Package {
return p
}
func decodeLocations(vals map[string]string) []source.Location {
func decodeLocations(vals map[string]string) source.LocationSet {
v := common.Decode(reflect.TypeOf([]source.Location{}), vals, "syft:location", CycloneDXFields)
out, _ := v.([]source.Location)
return out
out, ok := v.([]source.Location)
if !ok {
out = nil
}
return source.NewLocationSet(out...)
}
func decodePackageMetadata(vals map[string]string, c *cyclonedx.Component, typ pkg.MetadataType) interface{} {
@@ -26,9 +26,9 @@ func Test_encodeComponentProperties(t *testing.T) {
name: "from apk",
input: pkg.Package{
FoundBy: "cataloger",
Locations: []source.Location{
{Coordinates: source.Coordinates{RealPath: "test"}},
},
Locations: source.NewLocationSet(
source.Location{Coordinates: source.Coordinates{RealPath: "test"}},
),
Metadata: pkg.ApkMetadata{
Package: "libc-utils",
OriginPackage: "libc-dev",
@@ -13,6 +13,8 @@ func SourceInfo(p pkg.Package) string {
answer = "acquired package info from RPM DB"
case pkg.ApkPkg:
answer = "acquired package info from APK DB"
case pkg.DartPubPkg:
answer = "acquired package info from pubspec manifest"
case pkg.DebPkg:
answer = "acquired package info from DPKG DB"
case pkg.NpmPkg:
@@ -33,7 +35,7 @@ func SourceInfo(p pkg.Package) string {
answer = "acquired package info from the following paths"
}
var paths []string
for _, l := range p.Locations {
for _, l := range p.Locations.ToSlice() {
paths = append(paths, l.RealPath)
}
@@ -18,10 +18,10 @@ func Test_SourceInfo(t *testing.T) {
name: "locations are captured",
input: pkg.Package{
// note: no type given
Locations: []source.Location{
Locations: source.NewLocationSet(
source.NewVirtualLocation("/a-place", "/b-place"),
source.NewVirtualLocation("/c-place", "/d-place"),
},
),
},
expected: []string{
"from the following paths",
@@ -126,6 +126,14 @@ func Test_SourceInfo(t *testing.T) {
"from PHP composer manifest",
},
},
{
input: pkg.Package{
Type: pkg.DartPubPkg,
},
expected: []string{
"from pubspec manifest",
},
},
}
var pkgTypes []pkg.Type
for _, test := range tests {
+8 -8
View File
@@ -157,9 +157,9 @@ func populateImageCatalog(catalog *pkg.Catalog, img *image.Image) {
catalog.Add(pkg.Package{
Name: "package-1",
Version: "1.0.1",
Locations: []source.Location{
Locations: source.NewLocationSet(
source.NewLocationFromImage(string(ref1.RealPath), *ref1, img),
},
),
Type: pkg.PythonPkg,
FoundBy: "the-cataloger-1",
Language: pkg.Python,
@@ -177,9 +177,9 @@ func populateImageCatalog(catalog *pkg.Catalog, img *image.Image) {
catalog.Add(pkg.Package{
Name: "package-2",
Version: "2.0.1",
Locations: []source.Location{
Locations: source.NewLocationSet(
source.NewLocationFromImage(string(ref2.RealPath), *ref2, img),
},
),
Type: pkg.DebPkg,
FoundBy: "the-cataloger-2",
MetadataType: pkg.DpkgMetadataType,
@@ -234,9 +234,9 @@ func newDirectoryCatalog() *pkg.Catalog {
Version: "1.0.1",
Type: pkg.PythonPkg,
FoundBy: "the-cataloger-1",
Locations: []source.Location{
Locations: source.NewLocationSet(
source.NewLocation("/some/path/pkg1"),
},
),
Language: pkg.Python,
MetadataType: pkg.PythonPackageMetadataType,
Licenses: []string{"MIT"},
@@ -259,9 +259,9 @@ func newDirectoryCatalog() *pkg.Catalog {
Version: "2.0.1",
Type: pkg.DebPkg,
FoundBy: "the-cataloger-2",
Locations: []source.Location{
Locations: source.NewLocationSet(
source.NewLocation("/some/path/pkg1"),
},
),
MetadataType: pkg.DpkgMetadataType,
Metadata: pkg.DpkgMetadata{
Package: "package-2",
+6 -4
View File
@@ -58,8 +58,9 @@ func toSnapshotMetadata(s *sbom.SBOM) Metadata {
}
func filesystem(p pkg.Package) string {
if len(p.Locations) > 0 {
return p.Locations[0].FileSystemID
locations := p.Locations.ToSlice()
if len(locations) > 0 {
return locations[0].FileSystemID
}
return ""
}
@@ -76,8 +77,9 @@ func toPath(s source.Metadata, p pkg.Package) string {
if inputPath == "." {
inputPath = ""
}
if len(p.Locations) > 0 {
location := p.Locations[0]
locations := p.Locations.ToSlice()
if len(locations) > 0 {
location := locations[0]
packagePath := location.RealPath
if location.VirtualPath != "" {
packagePath = location.VirtualPath
+21 -15
View File
@@ -35,32 +35,38 @@ func Test_toGithubModel(t *testing.T) {
{
Name: "pkg-1",
Version: "1.0.1",
Locations: []source.Location{{
Coordinates: source.Coordinates{
RealPath: "/usr/lib",
FileSystemID: "fsid-1",
Locations: source.NewLocationSet(
source.Location{
Coordinates: source.Coordinates{
RealPath: "/usr/lib",
FileSystemID: "fsid-1",
},
},
}},
),
},
{
Name: "pkg-2",
Version: "2.0.2",
Locations: []source.Location{{
Coordinates: source.Coordinates{
RealPath: "/usr/lib",
FileSystemID: "fsid-1",
Locations: source.NewLocationSet(
source.Location{
Coordinates: source.Coordinates{
RealPath: "/usr/lib",
FileSystemID: "fsid-1",
},
},
}},
),
},
{
Name: "pkg-3",
Version: "3.0.3",
Locations: []source.Location{{
Coordinates: source.Coordinates{
RealPath: "/etc",
FileSystemID: "fsid-1",
Locations: source.NewLocationSet(
source.Location{
Coordinates: source.Coordinates{
RealPath: "/etc",
FileSystemID: "fsid-1",
},
},
}},
),
},
} {
p.PURL = packageurl.NewPackageURL(
@@ -3,18 +3,18 @@
"name": "/some/path",
"spdxVersion": "SPDX-2.2",
"creationInfo": {
"created": "2021-12-20T19:12:47.869816Z",
"created": "2022-03-30T21:48:28.297464Z",
"creators": [
"Organization: Anchore, Inc",
"Tool: syft-[not provided]"
],
"licenseListVersion": "3.15"
"licenseListVersion": "3.16"
},
"dataLicense": "CC0-1.0",
"documentNamespace": "https://anchore.com/syft/dir/some/path-4b896ded-7852-4e31-b764-136b53bdf346",
"documentNamespace": "https://anchore.com/syft/dir/some/path-e188d59b-76f6-4c7f-a9f2-1ae7d0577781",
"packages": [
{
"SPDXID": "SPDXRef-1d97af55efe9512f",
"SPDXID": "SPDXRef-b85dbb4e6ece5082",
"name": "package-1",
"licenseConcluded": "MIT",
"downloadLocation": "NOASSERTION",
@@ -36,7 +36,7 @@
"versionInfo": "1.0.1"
},
{
"SPDXID": "SPDXRef-ad3d1c4abd84bf75",
"SPDXID": "SPDXRef-ceda99598967ae8d",
"name": "package-2",
"licenseConcluded": "NONE",
"downloadLocation": "NOASSERTION",
@@ -3,18 +3,18 @@
"name": "user-image-input",
"spdxVersion": "SPDX-2.2",
"creationInfo": {
"created": "2021-12-20T19:13:07.647486Z",
"created": "2022-03-30T21:48:28.303986Z",
"creators": [
"Organization: Anchore, Inc",
"Tool: syft-[not provided]"
],
"licenseListVersion": "3.15"
"licenseListVersion": "3.16"
},
"dataLicense": "CC0-1.0",
"documentNamespace": "https://anchore.com/syft/image/user-image-input-174da656-1824-4bd3-8604-28919f8a65bc",
"documentNamespace": "https://anchore.com/syft/image/user-image-input-9e4f4190-c5ae-4e31-a852-d1ab71357516",
"packages": [
{
"SPDXID": "SPDXRef-d16127444133b5c1",
"SPDXID": "SPDXRef-2a46171f91c8d4bc",
"name": "package-1",
"licenseConcluded": "MIT",
"downloadLocation": "NOASSERTION",
@@ -36,7 +36,7 @@
"versionInfo": "1.0.1"
},
{
"SPDXID": "SPDXRef-24907357f3705420",
"SPDXID": "SPDXRef-ae77680e9b1d087e",
"name": "package-2",
"licenseConcluded": "NONE",
"downloadLocation": "NOASSERTION",
@@ -2,16 +2,16 @@ SPDXVersion: SPDX-2.2
DataLicense: CC0-1.0
SPDXID: SPDXRef-DOCUMENT
DocumentName: /some/path
DocumentNamespace: https://anchore.com/syft/dir/some/path-4b90f56d-d596-4ad8-b6a5-17f7d801350d
DocumentNamespace: https://anchore.com/syft/dir/some/path-71aa3553-1a73-405f-9f1f-6347d6d4593b
LicenseListVersion: 3.16
Creator: Organization: Anchore, Inc
Creator: Tool: syft-[not provided]
Created: 2022-02-10T21:09:27Z
Created: 2022-03-30T21:48:22Z
##### Package: package-2
PackageName: package-2
SPDXID: SPDXRef-Package-deb-package-2-ad3d1c4abd84bf75
SPDXID: SPDXRef-Package-deb-package-2-ceda99598967ae8d
PackageVersion: 2.0.1
PackageDownloadLocation: NOASSERTION
FilesAnalyzed: false
@@ -24,7 +24,7 @@ ExternalRef: PACKAGE_MANAGER purl a-purl-2
##### Package: package-1
PackageName: package-1
SPDXID: SPDXRef-Package-python-package-1-1d97af55efe9512f
SPDXID: SPDXRef-Package-python-package-1-b85dbb4e6ece5082
PackageVersion: 1.0.1
PackageDownloadLocation: NOASSERTION
FilesAnalyzed: false
@@ -2,16 +2,16 @@ SPDXVersion: SPDX-2.2
DataLicense: CC0-1.0
SPDXID: SPDXRef-DOCUMENT
DocumentName: user-image-input
DocumentNamespace: https://anchore.com/syft/image/user-image-input-26a2def6-53d0-4504-b99a-a046832508ac
DocumentNamespace: https://anchore.com/syft/image/user-image-input-e46e20f4-43a4-40e7-9f82-fd55b8a89e5f
LicenseListVersion: 3.16
Creator: Organization: Anchore, Inc
Creator: Tool: syft-[not provided]
Created: 2022-02-10T21:09:27Z
Created: 2022-03-30T21:48:22Z
##### Package: package-2
PackageName: package-2
SPDXID: SPDXRef-Package-deb-package-2-73f796c846875b9e
SPDXID: SPDXRef-Package-deb-package-2-ae77680e9b1d087e
PackageVersion: 2.0.1
PackageDownloadLocation: NOASSERTION
FilesAnalyzed: false
@@ -24,7 +24,7 @@ ExternalRef: PACKAGE_MANAGER purl a-purl-2
##### Package: package-1
PackageName: package-1
SPDXID: SPDXRef-Package-python-package-1-d9527e708c11f8b9
SPDXID: SPDXRef-Package-python-package-1-2a46171f91c8d4bc
PackageVersion: 1.0.1
PackageDownloadLocation: NOASSERTION
FilesAnalyzed: false
+6 -6
View File
@@ -42,13 +42,13 @@ func TestEncodeFullJSONDocument(t *testing.T) {
p1 := pkg.Package{
Name: "package-1",
Version: "1.0.1",
Locations: []source.Location{
{
Locations: source.NewLocationSet(
source.Location{
Coordinates: source.Coordinates{
RealPath: "/a/place/a",
},
},
},
),
Type: pkg.PythonPkg,
FoundBy: "the-cataloger-1",
Language: pkg.Python,
@@ -68,13 +68,13 @@ func TestEncodeFullJSONDocument(t *testing.T) {
p2 := pkg.Package{
Name: "package-2",
Version: "2.0.1",
Locations: []source.Location{
{
Locations: source.NewLocationSet(
source.Location{
Coordinates: source.Coordinates{
RealPath: "/b/place/b",
},
},
},
),
Type: pkg.DebPkg,
FoundBy: "the-cataloger-2",
MetadataType: pkg.DpkgMetadataType,
@@ -130,6 +130,12 @@ func (p *Package) UnmarshalJSON(b []byte) error {
return err
}
p.Metadata = payload
case pkg.DartPubMetadataType:
var payload pkg.DartPubMetadata
if err := json.Unmarshal(unpacker.Metadata, &payload); err != nil {
return err
}
p.Metadata = payload
default:
log.Warnf("unknown package metadata type=%q for packageID=%q", p.MetadataType, p.ID)
}
@@ -1,7 +1,7 @@
{
"artifacts": [
{
"id": "1d97af55efe9512f",
"id": "b85dbb4e6ece5082",
"name": "package-1",
"version": "1.0.1",
"type": "python",
@@ -36,7 +36,7 @@
}
},
{
"id": "ad3d1c4abd84bf75",
"id": "ceda99598967ae8d",
"name": "package-2",
"version": "2.0.1",
"type": "deb",
@@ -88,7 +88,7 @@
}
},
"schema": {
"version": "3.2.1",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-3.2.1.json"
"version": "3.2.2",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-3.2.2.json"
}
}
@@ -1,7 +1,7 @@
{
"artifacts": [
{
"id": "d9a7c58726ab4bef",
"id": "b3fa3ee64756b0c6",
"name": "package-1",
"version": "1.0.1",
"type": "python",
@@ -31,7 +31,7 @@
}
},
{
"id": "ac462e450060da2c",
"id": "b324f4d9ee5413fe",
"name": "package-2",
"version": "2.0.1",
"type": "deb",
@@ -184,7 +184,7 @@
}
},
"schema": {
"version": "3.2.1",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-3.2.1.json"
"version": "3.2.2",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-3.2.2.json"
}
}
@@ -1,7 +1,7 @@
{
"artifacts": [
{
"id": "d9527e708c11f8b9",
"id": "2a46171f91c8d4bc",
"name": "package-1",
"version": "1.0.1",
"type": "python",
@@ -32,7 +32,7 @@
}
},
{
"id": "73f796c846875b9e",
"id": "ae77680e9b1d087e",
"name": "package-2",
"version": "2.0.1",
"type": "deb",
@@ -111,7 +111,7 @@
}
},
"schema": {
"version": "3.2.1",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-3.2.1.json"
"version": "3.2.2",
"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-3.2.2.json"
}
}
+3 -2
View File
@@ -175,8 +175,9 @@ func toPackageModel(p pkg.Package) model.Package {
licenses = p.Licenses
}
var coordinates = make([]source.Coordinates, len(p.Locations))
for i, l := range p.Locations {
locations := p.Locations.ToSlice()
var coordinates = make([]source.Coordinates, len(locations))
for i, l := range locations {
coordinates[i] = l.Coordinates
}
+3 -2
View File
@@ -53,7 +53,8 @@ func toSyftRelationships(doc *model.Document, catalog *pkg.Catalog, relationship
for _, p := range catalog.Sorted() {
idMap[string(p.ID())] = p
for _, l := range p.Locations {
locations := p.Locations.ToSlice()
for _, l := range locations {
idMap[string(l.Coordinates.ID())] = l.Coordinates
}
}
@@ -166,7 +167,7 @@ func toSyftPackage(p model.Package, idAliases map[string]string) pkg.Package {
Name: p.Name,
Version: p.Version,
FoundBy: p.FoundBy,
Locations: locations,
Locations: source.NewLocationSet(locations...),
Licenses: p.Licenses,
Language: p.Language,
Type: p.Type,
+1
View File
@@ -37,6 +37,7 @@ type artifactMetadataContainer struct {
Cargo pkg.CargoPackageMetadata
Go pkg.GolangBinMetadata
Php pkg.PhpComposerJSONMetadata
Dart pkg.DartPubMetadata
}
func main() {
File diff suppressed because it is too large Load Diff
+73 -16
View File
@@ -10,11 +10,28 @@ import (
"github.com/jinzhu/copier"
)
type orderedIDSet struct {
slice []artifact.ID
}
func (s *orderedIDSet) add(ids ...artifact.ID) {
loopNewIDs:
for _, newID := range ids {
for _, existingID := range s.slice {
if existingID == newID {
continue loopNewIDs
}
}
s.slice = append(s.slice, newID)
}
}
// Catalog represents a collection of Packages.
type Catalog struct {
byID map[artifact.ID]Package
idsByType map[Type][]artifact.ID
idsByPath map[string][]artifact.ID // note: this is real path or virtual path
idsByName map[string]orderedIDSet
idsByType map[Type]orderedIDSet
idsByPath map[string]orderedIDSet // note: this is real path or virtual path
lock sync.RWMutex
}
@@ -22,8 +39,9 @@ type Catalog struct {
func NewCatalog(pkgs ...Package) *Catalog {
catalog := Catalog{
byID: make(map[artifact.ID]Package),
idsByType: make(map[Type][]artifact.ID),
idsByPath: make(map[string][]artifact.ID),
idsByName: make(map[string]orderedIDSet),
idsByType: make(map[Type]orderedIDSet),
idsByPath: make(map[string]orderedIDSet),
}
for _, p := range pkgs {
@@ -55,7 +73,12 @@ func (c *Catalog) Package(id artifact.ID) *Package {
// PackagesByPath returns all packages that were discovered from the given path.
func (c *Catalog) PackagesByPath(path string) []Package {
return c.Packages(c.idsByPath[path])
return c.Packages(c.idsByPath[path].slice)
}
// PackagesByName returns all packages that were discovered with a matching name.
func (c *Catalog) PackagesByName(name string) []Package {
return c.Packages(c.idsByName[name].slice)
}
// Packages returns all packages for the given ID.
@@ -81,26 +104,58 @@ func (c *Catalog) Add(p Package) {
id = p.ID()
}
// store by package ID
c.byID[id] = p
if existing, exists := c.byID[id]; exists {
// there is already a package with this fingerprint merge the existing record with the new one
if err := existing.merge(p); err != nil {
log.Warnf("failed to merge packages: %+v", err)
} else {
c.addPathsToIndex(p)
}
return
}
// store by package type
c.idsByType[p.Type] = append(c.idsByType[p.Type], id)
c.addToIndex(p)
}
// store by file location paths
func (c *Catalog) addToIndex(p Package) {
c.byID[p.id] = p
c.addNameToIndex(p)
c.addTypeToIndex(p)
c.addPathsToIndex(p)
}
func (c *Catalog) addNameToIndex(p Package) {
nameIndex := c.idsByName[p.Name]
nameIndex.add(p.id)
c.idsByName[p.Name] = nameIndex
}
func (c *Catalog) addTypeToIndex(p Package) {
typeIndex := c.idsByType[p.Type]
typeIndex.add(p.id)
c.idsByType[p.Type] = typeIndex
}
func (c *Catalog) addPathsToIndex(p Package) {
observedPaths := internal.NewStringSet()
for _, l := range p.Locations {
for _, l := range p.Locations.ToSlice() {
if l.RealPath != "" && !observedPaths.Contains(l.RealPath) {
c.idsByPath[l.RealPath] = append(c.idsByPath[l.RealPath], id)
c.addPathToIndex(p.id, l.RealPath)
observedPaths.Add(l.RealPath)
}
if l.VirtualPath != "" && l.RealPath != l.VirtualPath && !observedPaths.Contains(l.VirtualPath) {
c.idsByPath[l.VirtualPath] = append(c.idsByPath[l.VirtualPath], id)
c.addPathToIndex(p.id, l.VirtualPath)
observedPaths.Add(l.VirtualPath)
}
}
}
func (c *Catalog) addPathToIndex(id artifact.ID, path string) {
pathIndex := c.idsByPath[path]
pathIndex.add(id)
c.idsByPath[path] = pathIndex
}
// Enumerate all packages for the given type(s), enumerating all packages if no type is specified.
func (c *Catalog) Enumerate(types ...Type) <-chan Package {
channel := make(chan Package)
@@ -124,7 +179,7 @@ func (c *Catalog) Enumerate(types ...Type) <-chan Package {
continue
}
}
for _, id := range ids {
for _, id := range ids.slice {
p := c.Package(id)
if p != nil {
channel <- *p
@@ -145,8 +200,10 @@ func (c *Catalog) Sorted(types ...Type) (pkgs []Package) {
sort.SliceStable(pkgs, func(i, j int) bool {
if pkgs[i].Name == pkgs[j].Name {
if pkgs[i].Version == pkgs[j].Version {
if pkgs[i].Type == pkgs[j].Type && len(pkgs[i].Locations) > 0 && len(pkgs[j].Locations) > 0 {
return pkgs[i].Locations[0].String() < pkgs[j].Locations[0].String()
iLocations := pkgs[i].Locations.ToSlice()
jLocations := pkgs[j].Locations.ToSlice()
if pkgs[i].Type == pkgs[j].Type && len(iLocations) > 0 && len(jLocations) > 0 {
return iLocations[0].String() < jLocations[0].String()
}
return pkgs[i].Type < pkgs[j].Type
}
+141 -19
View File
@@ -3,11 +3,11 @@ package pkg
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/scylladb/go-set/strset"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/source"
"github.com/scylladb/go-set/strset"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
type expectedIndexes struct {
@@ -19,17 +19,17 @@ func TestCatalogAddPopulatesIndex(t *testing.T) {
var pkgs = []Package{
{
Locations: []source.Location{
Locations: source.NewLocationSet(
source.NewVirtualLocation("/a/path", "/another/path"),
source.NewVirtualLocation("/b/path", "/bee/path"),
},
),
Type: RpmPkg,
},
{
Locations: []source.Location{
Locations: source.NewLocationSet(
source.NewVirtualLocation("/c/path", "/another/path"),
source.NewVirtualLocation("/d/path", "/another/path"),
},
),
Type: NpmPkg,
},
}
@@ -106,47 +106,169 @@ func assertIndexes(t *testing.T, c *Catalog, expectedIndexes expectedIndexes) {
func TestCatalog_PathIndexDeduplicatesRealVsVirtualPaths(t *testing.T) {
p1 := Package{
Locations: []source.Location{
Locations: source.NewLocationSet(
source.NewVirtualLocation("/b/path", "/another/path"),
source.NewVirtualLocation("/b/path", "/b/path"),
},
),
Type: RpmPkg,
Name: "Package-1",
}
p2 := Package{
Locations: []source.Location{
Locations: source.NewLocationSet(
source.NewVirtualLocation("/b/path", "/b/path"),
},
),
Type: RpmPkg,
Name: "Package-2",
}
p2Dup := Package{
Locations: source.NewLocationSet(
source.NewVirtualLocation("/b/path", "/another/path"),
source.NewVirtualLocation("/b/path", "/c/path/b/dup"),
),
Type: RpmPkg,
Name: "Package-2",
}
tests := []struct {
name string
pkg Package
name string
pkgs []Package
paths []string
}{
{
name: "multiple locations with shared path",
pkg: p1,
pkgs: []Package{p1},
paths: []string{
"/b/path",
"/another/path",
},
},
{
name: "one location with shared path",
pkg: p2,
pkgs: []Package{p2},
paths: []string{
"/b/path",
},
},
{
name: "two instances with similar locations",
pkgs: []Package{p2, p2Dup},
paths: []string{
"/b/path",
"/another/path",
"/c/path/b/dup", // this updated the path index on merge
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
actual := NewCatalog(test.pkg).PackagesByPath("/b/path")
if len(actual) != 1 {
t.Errorf("expected exactly one package path, got %d", len(actual))
for _, path := range test.paths {
actualPackages := NewCatalog(test.pkgs...).PackagesByPath(path)
require.Len(t, actualPackages, 1)
}
})
}
}
func TestCatalog_MergeRecords(t *testing.T) {
var tests = []struct {
name string
pkgs []Package
expectedLocations []source.Location
}{
{
name: "multiple Locations with shared path",
pkgs: []Package{
{
Locations: source.NewLocationSet(
source.Location{
Coordinates: source.Coordinates{
RealPath: "/b/path",
FileSystemID: "a",
},
VirtualPath: "/another/path",
},
),
Type: RpmPkg,
},
{
Locations: source.NewLocationSet(
source.Location{
Coordinates: source.Coordinates{
RealPath: "/b/path",
FileSystemID: "b",
},
VirtualPath: "/another/path",
},
),
Type: RpmPkg,
},
},
expectedLocations: []source.Location{
{
Coordinates: source.Coordinates{
RealPath: "/b/path",
FileSystemID: "a",
},
VirtualPath: "/another/path",
},
{
Coordinates: source.Coordinates{
RealPath: "/b/path",
FileSystemID: "b",
},
VirtualPath: "/another/path",
},
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
actual := NewCatalog(tt.pkgs...).PackagesByPath("/b/path")
require.Len(t, actual, 1)
assert.Equal(t, tt.expectedLocations, actual[0].Locations.ToSlice())
})
}
}
func TestCatalog_EnumerateNilCatalog(t *testing.T) {
var c *Catalog
assert.Empty(t, c.Enumerate())
}
func Test_idOrderedSet_add(t *testing.T) {
tests := []struct {
name string
input []artifact.ID
expected []artifact.ID
}{
{
name: "elements deduplicated when added",
input: []artifact.ID{
"1", "2", "3", "4", "1", "2", "3", "4", "1", "2", "3", "4",
},
expected: []artifact.ID{
"1", "2", "3", "4",
},
},
{
name: "elements retain ordering when added",
input: []artifact.ID{
"4", "3", "2", "1",
},
expected: []artifact.ID{
"4", "3", "2", "1",
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var s orderedIDSet
s.add(tt.input...)
assert.Equal(t, tt.expected, s.slice)
})
}
}
+3
View File
@@ -9,6 +9,7 @@ import (
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/pkg/cataloger/apkdb"
"github.com/anchore/syft/syft/pkg/cataloger/dart"
"github.com/anchore/syft/syft/pkg/cataloger/deb"
"github.com/anchore/syft/syft/pkg/cataloger/golang"
"github.com/anchore/syft/syft/pkg/cataloger/java"
@@ -61,6 +62,7 @@ func DirectoryCatalogers(cfg Config) []Cataloger {
golang.NewGoModuleBinaryCataloger(),
golang.NewGoModFileCataloger(),
rust.NewCargoLockCataloger(),
dart.NewPubspecLockCataloger(),
}
}
@@ -80,5 +82,6 @@ func AllCatalogers(cfg Config) []Cataloger {
golang.NewGoModuleBinaryCataloger(),
golang.NewGoModFileCataloger(),
rust.NewCargoLockCataloger(),
dart.NewPubspecLockCataloger(),
}
}
+13 -13
View File
@@ -16,7 +16,7 @@ func Test_disallowJenkinsServerCPEForPluginPackage(t *testing.T) {
}{
{
name: "go case (filter out)",
cpe: mustCPE("cpe:2.3:a:name:jenkins:3.2:*:*:*:*:*:*:*"),
cpe: pkg.MustCPE("cpe:2.3:a:name:jenkins:3.2:*:*:*:*:*:*:*"),
pkg: pkg.Package{
Type: pkg.JenkinsPluginPkg,
},
@@ -24,7 +24,7 @@ func Test_disallowJenkinsServerCPEForPluginPackage(t *testing.T) {
},
{
name: "ignore jenkins plugins with unique name",
cpe: mustCPE("cpe:2.3:a:name:ci-jenkins:3.2:*:*:*:*:*:*:*"),
cpe: pkg.MustCPE("cpe:2.3:a:name:ci-jenkins:3.2:*:*:*:*:*:*:*"),
pkg: pkg.Package{
Type: pkg.JenkinsPluginPkg,
},
@@ -32,7 +32,7 @@ func Test_disallowJenkinsServerCPEForPluginPackage(t *testing.T) {
},
{
name: "ignore java packages",
cpe: mustCPE("cpe:2.3:a:name:jenkins:3.2:*:*:*:*:*:*:*"),
cpe: pkg.MustCPE("cpe:2.3:a:name:jenkins:3.2:*:*:*:*:*:*:*"),
pkg: pkg.Package{
Type: pkg.JavaPkg,
},
@@ -55,7 +55,7 @@ func Test_disallowJenkinsCPEsNotAssociatedWithJenkins(t *testing.T) {
}{
{
name: "filter out mismatched name (cloudbees vendor)",
cpe: mustCPE("cpe:2.3:a:cloudbees:jenkins:3.2:*:*:*:*:*:*:*"),
cpe: pkg.MustCPE("cpe:2.3:a:cloudbees:jenkins:3.2:*:*:*:*:*:*:*"),
pkg: pkg.Package{
Name: "not-j*nkins",
Type: pkg.JavaPkg,
@@ -64,7 +64,7 @@ func Test_disallowJenkinsCPEsNotAssociatedWithJenkins(t *testing.T) {
},
{
name: "filter out mismatched name (jenkins vendor)",
cpe: mustCPE("cpe:2.3:a:jenkins:jenkins:3.2:*:*:*:*:*:*:*"),
cpe: pkg.MustCPE("cpe:2.3:a:jenkins:jenkins:3.2:*:*:*:*:*:*:*"),
pkg: pkg.Package{
Name: "not-j*nkins",
Type: pkg.JavaPkg,
@@ -73,7 +73,7 @@ func Test_disallowJenkinsCPEsNotAssociatedWithJenkins(t *testing.T) {
},
{
name: "filter out mismatched name (any vendor)",
cpe: mustCPE("cpe:2.3:a:*:jenkins:3.2:*:*:*:*:*:*:*"),
cpe: pkg.MustCPE("cpe:2.3:a:*:jenkins:3.2:*:*:*:*:*:*:*"),
pkg: pkg.Package{
Name: "not-j*nkins",
Type: pkg.JavaPkg,
@@ -82,7 +82,7 @@ func Test_disallowJenkinsCPEsNotAssociatedWithJenkins(t *testing.T) {
},
{
name: "ignore packages with the name jenkins",
cpe: mustCPE("cpe:2.3:a:*:jenkins:3.2:*:*:*:*:*:*:*"),
cpe: pkg.MustCPE("cpe:2.3:a:*:jenkins:3.2:*:*:*:*:*:*:*"),
pkg: pkg.Package{
Name: "jenkins-thing",
Type: pkg.JavaPkg,
@@ -91,7 +91,7 @@ func Test_disallowJenkinsCPEsNotAssociatedWithJenkins(t *testing.T) {
},
{
name: "ignore product names that are not exactly 'jenkins'",
cpe: mustCPE("cpe:2.3:a:*:jenkins-something-else:3.2:*:*:*:*:*:*:*"),
cpe: pkg.MustCPE("cpe:2.3:a:*:jenkins-something-else:3.2:*:*:*:*:*:*:*"),
pkg: pkg.Package{
Name: "not-j*nkins",
Type: pkg.JavaPkg,
@@ -115,7 +115,7 @@ func Test_disallowJiraClientServerMismatch(t *testing.T) {
}{
{
name: "filter out mismatched name (atlassian vendor)",
cpe: mustCPE("cpe:2.3:a:atlassian:jira:3.2:*:*:*:*:*:*:*"),
cpe: pkg.MustCPE("cpe:2.3:a:atlassian:jira:3.2:*:*:*:*:*:*:*"),
pkg: pkg.Package{
Name: "something-client",
Type: pkg.JavaPkg,
@@ -124,7 +124,7 @@ func Test_disallowJiraClientServerMismatch(t *testing.T) {
},
{
name: "filter out mismatched name (jira vendor)",
cpe: mustCPE("cpe:2.3:a:jira:jira:3.2:*:*:*:*:*:*:*"),
cpe: pkg.MustCPE("cpe:2.3:a:jira:jira:3.2:*:*:*:*:*:*:*"),
pkg: pkg.Package{
Name: "something-client",
Type: pkg.JavaPkg,
@@ -133,7 +133,7 @@ func Test_disallowJiraClientServerMismatch(t *testing.T) {
},
{
name: "filter out mismatched name (any vendor)",
cpe: mustCPE("cpe:2.3:a:*:jira:3.2:*:*:*:*:*:*:*"),
cpe: pkg.MustCPE("cpe:2.3:a:*:jira:3.2:*:*:*:*:*:*:*"),
pkg: pkg.Package{
Name: "something-client",
Type: pkg.JavaPkg,
@@ -142,7 +142,7 @@ func Test_disallowJiraClientServerMismatch(t *testing.T) {
},
{
name: "ignore package names that do not have 'client'",
cpe: mustCPE("cpe:2.3:a:*:jira:3.2:*:*:*:*:*:*:*"),
cpe: pkg.MustCPE("cpe:2.3:a:*:jira:3.2:*:*:*:*:*:*:*"),
pkg: pkg.Package{
Name: "jira-thing",
Type: pkg.JavaPkg,
@@ -151,7 +151,7 @@ func Test_disallowJiraClientServerMismatch(t *testing.T) {
},
{
name: "ignore product names that are not exactly 'jira'",
cpe: mustCPE("cpe:2.3:a:*:jira-something-else:3.2:*:*:*:*:*:*:*"),
cpe: pkg.MustCPE("cpe:2.3:a:*:jira-something-else:3.2:*:*:*:*:*:*:*"),
pkg: pkg.Package{
Name: "not-j*ra",
Type: pkg.JavaPkg,
+1 -1
View File
@@ -55,7 +55,7 @@ func Generate(p pkg.Package) []pkg.CPE {
// filter out any known combinations that don't accurately represent this package
cpes = filter(cpes, p, cpeFilters...)
sort.Sort(BySpecificity(cpes))
sort.Sort(pkg.CPEBySpecificity(cpes))
return cpes
}
@@ -58,9 +58,8 @@ func (c *GenericCataloger) Catalog(resolver source.FileResolver) ([]pkg.Package,
for _, p := range discoveredPackages {
p.FoundBy = c.upstreamCataloger
p.Locations = append(p.Locations, location)
p.Locations.Add(location)
p.SetID()
packages = append(packages, *p)
}
@@ -53,7 +53,7 @@ func TestGenericCataloger(t *testing.T) {
assert.Len(t, actualPkgs, len(expectedPkgs))
for _, p := range actualPkgs {
ref := p.Locations[0]
ref := p.Locations.ToSlice()[0]
exP, ok := expectedPkgs[ref.RealPath]
if !ok {
t.Errorf("missing expected pkg: ref=%+v", ref)
+14
View File
@@ -0,0 +1,14 @@
package dart
import (
"github.com/anchore/syft/syft/pkg/cataloger/common"
)
// NewPubspecLockCataloger returns a new Dartlang cataloger object base on pubspec lock files.
func NewPubspecLockCataloger() *common.GenericCataloger {
globParsers := map[string]common.ParserFn{
"**/pubspec.lock": parsePubspecLock,
}
return common.NewGenericCataloger(nil, globParsers, "dartlang-lock-cataloger")
}
@@ -0,0 +1,96 @@
package dart
import (
"fmt"
"io"
"net/url"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/pkg/cataloger/common"
"gopkg.in/yaml.v2"
)
// integrity check
var _ common.ParserFn = parsePubspecLock
const defaultPubRegistry string = "https://pub.dartlang.org"
type pubspecLock struct {
Packages map[string]pubspecLockPackage `yaml:"packages"`
Sdks map[string]string `yaml:"sdks"`
}
type pubspecLockPackage struct {
Dependency string `yaml:"dependency" mapstructure:"dependency"`
Description pubspecLockDescription `yaml:"description" mapstructure:"description"`
Source string `yaml:"source" mapstructure:"source"`
Version string `yaml:"version" mapstructure:"version"`
}
type pubspecLockDescription struct {
Name string `yaml:"name" mapstructure:"name"`
URL string `yaml:"url" mapstructure:"url"`
Path string `yaml:"path" mapstructure:"path"`
Ref string `yaml:"ref" mapstructure:"ref"`
ResolvedRef string `yaml:"resolved-ref" mapstructure:"resolved-ref"`
}
func parsePubspecLock(path string, reader io.Reader) ([]*pkg.Package, []artifact.Relationship, error) {
var packages []*pkg.Package
dec := yaml.NewDecoder(reader)
var p pubspecLock
if err := dec.Decode(&p); err != nil {
return nil, nil, fmt.Errorf("failed to parse pubspec.lock file: %w", err)
}
for name, pubPkg := range p.Packages {
packages = append(packages, newPubspecLockPackage(name, pubPkg))
}
return packages, nil, nil
}
func newPubspecLockPackage(name string, p pubspecLockPackage) *pkg.Package {
return &pkg.Package{
Name: name,
Version: p.Version,
Language: pkg.Dart,
Type: pkg.DartPubPkg,
MetadataType: pkg.DartPubMetadataType,
Metadata: &pkg.DartPubMetadata{
Name: name,
Version: p.Version,
HostedURL: p.getHostedURL(),
VcsURL: p.getVcsURL(),
},
}
}
func (p *pubspecLockPackage) getVcsURL() string {
if p.Source == "git" {
if p.Description.Path == "." {
return fmt.Sprintf("%s@%s", p.Description.URL, p.Description.ResolvedRef)
}
return fmt.Sprintf("%s@%s#%s", p.Description.URL, p.Description.ResolvedRef, p.Description.Path)
}
return ""
}
func (p *pubspecLockPackage) getHostedURL() string {
if p.Source == "hosted" && p.Description.URL != defaultPubRegistry {
u, err := url.Parse(p.Description.URL)
if err != nil {
log.Debugf("Unable to parse registry url %w", err)
return p.Description.URL
}
return u.Host
}
return ""
}
@@ -0,0 +1,99 @@
package dart
import (
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/anchore/syft/syft/pkg"
)
func assertPackagesEqual(t *testing.T, actual []*pkg.Package, expected map[string]*pkg.Package) {
assert.Len(t, actual, len(expected))
}
func TestParsePubspecLock(t *testing.T) {
expected := map[string]*pkg.Package{
"ale": {
Name: "ale",
Version: "3.3.0",
Language: pkg.Dart,
Type: pkg.DartPubPkg,
MetadataType: pkg.DartPubMetadataType,
Metadata: pkg.DartPubMetadata{
Name: "ale",
Version: "3.3.0",
HostedURL: "pub.hosted.org",
},
},
"analyzer": {
Name: "analyzer",
Version: "0.40.7",
Language: pkg.Dart,
Type: pkg.DartPubPkg,
MetadataType: pkg.DartPubMetadataType,
Metadata: pkg.DartPubMetadata{
Name: "analyzer",
Version: "0.40.7",
},
},
"ansicolor": {
Name: "ansicolor",
Version: "1.1.1",
Language: pkg.Dart,
Type: pkg.DartPubPkg,
MetadataType: pkg.DartPubMetadataType,
Metadata: pkg.DartPubMetadata{
Name: "ansicolor",
Version: "1.1.1",
},
},
"archive": {
Name: "archive",
Version: "2.0.13",
Language: pkg.Dart,
Type: pkg.DartPubPkg,
MetadataType: pkg.DartPubMetadataType,
Metadata: pkg.DartPubMetadata{
Name: "archive",
Version: "2.0.13",
},
},
"args": {
Name: "args",
Version: "1.6.0",
Language: pkg.Dart,
Type: pkg.DartPubPkg,
MetadataType: pkg.DartPubMetadataType,
Metadata: pkg.DartPubMetadata{
Name: "args",
Version: "1.6.0",
},
},
"key_binder": {
Name: "key_binder",
Version: "1.11.20",
Language: pkg.Dart,
Type: pkg.DartPubPkg,
MetadataType: pkg.DartPubMetadataType,
Metadata: pkg.DartPubMetadata{
Name: "key_binder",
Version: "1.11.20",
VcsURL: "git@github.com:Workiva/key_binder.git#3f7b3a6350e73c7dcac45301c0e18fbd42af02f7",
},
},
}
fixture, err := os.Open("test-fixtures/pubspec.lock")
if err != nil {
t.Fatalf("failed to open fixture: %+v", err)
}
actual, _, err := parsePubspecLock(fixture.Name(), fixture)
if err != nil {
t.Fatalf("failed to parse pubspec.lock: %+v", err)
}
assertPackagesEqual(t, actual, expected)
}
@@ -0,0 +1,49 @@
# Generated by pub
# See https://dart.dev/tools/pub/glossary#lockfile
packages:
ale:
dependency: transitive
description:
name: ale
url: "https://pub.hosted.org"
source: hosted
version: "3.3.0"
analyzer:
dependency: transitive
description:
name: analyzer
url: "https://pub.dartlang.org"
source: hosted
version: "0.40.7"
ansicolor:
dependency: transitive
description:
name: ansicolor
url: "https://pub.dartlang.org"
source: hosted
version: "1.1.1"
archive:
dependency: transitive
description:
name: archive
url: "https://pub.dartlang.org"
source: hosted
version: "2.0.13"
args:
dependency: transitive
description:
name: args
url: "https://pub.dartlang.org"
source: hosted
version: "1.6.0"
key_binder:
dependency: "direct main"
description:
path: "."
ref: HEAD
resolved-ref: "3f7b3a6350e73c7dcac45301c0e18fbd42af02f7"
url: "git@github.com:Workiva/key_binder.git"
source: git
version: "1.11.20"
sdks:
dart: ">=2.12.0 <3.0.0"
+3 -3
View File
@@ -60,7 +60,7 @@ func (c *Cataloger) Catalog(resolver source.FileResolver) ([]pkg.Package, []arti
for i := range pkgs {
p := &pkgs[i]
p.FoundBy = c.Name()
p.Locations = []source.Location{dbLocation}
p.Locations.Add(dbLocation)
// the current entry only has what may have been listed in the status file, however, there are additional
// files that are listed in multiple other locations. We should retrieve them all and merge the file lists
@@ -88,7 +88,7 @@ func addLicenses(resolver source.FileResolver, dbLocation source.Location, p *pk
p.Licenses = parseLicensesFromCopyright(copyrightReader)
// keep a record of the file where this was discovered
p.Locations = append(p.Locations, *copyrightLocation)
p.Locations.Add(*copyrightLocation)
}
}
@@ -117,7 +117,7 @@ loopNewFiles:
p.Metadata = metadata
// persist location information from each new source of information
p.Locations = append(p.Locations, infoLocations...)
p.Locations.Add(infoLocations...)
}
func getAdditionalFileListing(resolver source.FileResolver, dbLocation source.Location, p *pkg.Package) ([]pkg.DpkgFileRecord, []source.Location) {
+5 -6
View File
@@ -1,6 +1,7 @@
package deb
import (
"github.com/stretchr/testify/assert"
"testing"
"github.com/anchore/syft/syft/file"
@@ -115,15 +116,13 @@ func TestDpkgCataloger(t *testing.T) {
for idx := range actual {
a := &actual[idx]
// we will test the sources separately
var sourcesList = make([]string, len(a.Locations))
for i, s := range a.Locations {
var sourcesList = make([]string, len(a.Locations.ToSlice()))
for i, s := range a.Locations.ToSlice() {
sourcesList[i] = s.RealPath
}
a.Locations = nil
a.Locations = source.NewLocationSet()
for _, d := range deep.Equal(sourcesList, test.sources[a.Name]) {
t.Errorf("diff: %+v", d)
}
assert.ElementsMatch(t, sourcesList, test.sources[a.Name])
}
// test remaining fields...
+6 -8
View File
@@ -44,14 +44,12 @@ func newGoBinaryPackage(dep *debug.Module, goVersion, architecture string, locat
}
p := pkg.Package{
FoundBy: catalogerName,
Name: dep.Path,
Version: dep.Version,
Language: pkg.Go,
Type: pkg.GoModulePkg,
Locations: []source.Location{
location,
},
FoundBy: catalogerName,
Name: dep.Path,
Version: dep.Version,
Language: pkg.Go,
Type: pkg.GoModulePkg,
Locations: source.NewLocationSet(location),
MetadataType: pkg.GolangBinMetadataType,
Metadata: pkg.GolangBinMetadata{
GoCompiledVersion: goVersion,
+20 -18
View File
@@ -133,14 +133,14 @@ func TestBuildGoPkgInfo(t *testing.T) {
FoundBy: catalogerName,
Language: pkg.Go,
Type: pkg.GoModulePkg,
Locations: []source.Location{
{
Locations: source.NewLocationSet(
source.Location{
Coordinates: source.Coordinates{
RealPath: "/a-path",
FileSystemID: "layer-id",
},
},
},
),
MetadataType: pkg.GolangBinMetadataType,
Metadata: pkg.GolangBinMetadata{
GoCompiledVersion: goCompiledVersion,
@@ -190,14 +190,14 @@ func TestBuildGoPkgInfo(t *testing.T) {
Version: "v0.2.1",
Language: pkg.Go,
Type: pkg.GoModulePkg,
Locations: []source.Location{
{
Locations: source.NewLocationSet(
source.Location{
Coordinates: source.Coordinates{
RealPath: "/a-path",
FileSystemID: "layer-id",
},
},
},
),
MetadataType: pkg.GolangBinMetadataType,
Metadata: pkg.GolangBinMetadata{
GoCompiledVersion: goCompiledVersion,
@@ -252,14 +252,14 @@ func TestBuildGoPkgInfo(t *testing.T) {
Version: "v0.2.1",
Language: pkg.Go,
Type: pkg.GoModulePkg,
Locations: []source.Location{
{
Locations: source.NewLocationSet(
source.Location{
Coordinates: source.Coordinates{
RealPath: "/a-path",
FileSystemID: "layer-id",
},
},
},
),
MetadataType: pkg.GolangBinMetadataType,
Metadata: pkg.GolangBinMetadata{
GoCompiledVersion: goCompiledVersion,
@@ -273,14 +273,14 @@ func TestBuildGoPkgInfo(t *testing.T) {
Version: "v0.0.0-20210222170800-9c70f9b80bcf",
Language: pkg.Go,
Type: pkg.GoModulePkg,
Locations: []source.Location{
{
Locations: source.NewLocationSet(
source.Location{
Coordinates: source.Coordinates{
RealPath: "/a-path",
FileSystemID: "layer-id",
},
},
},
),
MetadataType: pkg.GolangBinMetadataType,
Metadata: pkg.GolangBinMetadata{
GoCompiledVersion: goCompiledVersion,
@@ -327,12 +327,14 @@ func TestBuildGoPkgInfo(t *testing.T) {
Version: "v0.0.0-20211006194710-c8a6f5223071",
Language: pkg.Go,
Type: pkg.GoModulePkg,
Locations: []source.Location{
{
Locations: source.NewLocationSet(
source.Location{
Coordinates: source.Coordinates{
RealPath: "/a-path",
FileSystemID: "layer-id",
}}},
},
},
),
MetadataType: pkg.GolangBinMetadataType,
Metadata: pkg.GolangBinMetadata{
GoCompiledVersion: goCompiledVersion,
@@ -344,14 +346,14 @@ func TestBuildGoPkgInfo(t *testing.T) {
Version: "v0.0.0-20210916214954-140adaaadfaf",
Language: pkg.Go,
Type: pkg.GoModulePkg,
Locations: []source.Location{
{
Locations: source.NewLocationSet(
source.Location{
Coordinates: source.Coordinates{
RealPath: "/a-path",
FileSystemID: "layer-id",
},
},
},
),
MetadataType: pkg.GolangBinMetadataType,
Metadata: pkg.GolangBinMetadata{
GoCompiledVersion: goCompiledVersion,
@@ -80,7 +80,7 @@ func (c *PackageCataloger) catalogEggOrWheel(resolver source.FileResolver, metad
Name: metadata.Name,
Version: metadata.Version,
FoundBy: c.Name(),
Locations: sources,
Locations: source.NewLocationSet(sources...),
Licenses: licenses,
Language: pkg.Python,
Type: pkg.PythonPkg,
@@ -144,7 +144,7 @@ func TestPythonPackageWheelCataloger(t *testing.T) {
t.Fatal(err)
}
test.expectedPackage.Locations = locations
test.expectedPackage.Locations = source.NewLocationSet(locations...)
actual, _, err := NewPythonPackageCataloger().Catalog(resolver)
if err != nil {
@@ -1,6 +1,7 @@
package python
import (
"github.com/anchore/syft/syft/source"
"os"
"testing"
@@ -48,7 +49,14 @@ func TestParsePipFileLock(t *testing.T) {
t.Fatalf("failed to parse requirements: %+v", err)
}
if diff := cmp.Diff(expected, actual, cmp.AllowUnexported(pkg.Package{})); diff != "" {
if diff := cmp.Diff(expected, actual,
cmp.AllowUnexported(pkg.Package{}),
cmp.Comparer(
func(x, y source.LocationSet) bool {
return cmp.Equal(x.ToSlice(), y.ToSlice())
},
),
); diff != "" {
t.Errorf("unexpected result from parsing (-expected +actual)\n%s", diff)
}
}
@@ -1,6 +1,7 @@
package python
import (
"github.com/anchore/syft/syft/source"
"os"
"testing"
@@ -42,7 +43,14 @@ func TestParseRequirementsTxt(t *testing.T) {
t.Fatalf("failed to parse requirements: %+v", err)
}
if diff := cmp.Diff(expected, actual, cmp.AllowUnexported(pkg.Package{})); diff != "" {
if diff := cmp.Diff(expected, actual,
cmp.AllowUnexported(pkg.Package{}),
cmp.Comparer(
func(x, y source.LocationSet) bool {
return cmp.Equal(x.ToSlice(), y.ToSlice())
},
),
); diff != "" {
t.Errorf("unexpected result from parsing (-expected +actual)\n%s", diff)
}
}
@@ -1,6 +1,7 @@
package python
import (
"github.com/anchore/syft/syft/source"
"os"
"testing"
@@ -54,7 +55,14 @@ func TestParseSetup(t *testing.T) {
t.Fatalf("failed to parse requirements: %+v", err)
}
if diff := cmp.Diff(expected, actual, cmp.AllowUnexported(pkg.Package{})); diff != "" {
if diff := cmp.Diff(expected, actual,
cmp.AllowUnexported(pkg.Package{}),
cmp.Comparer(
func(x, y source.LocationSet) bool {
return cmp.Equal(x.ToSlice(), y.ToSlice())
},
),
); diff != "" {
t.Errorf("unexpected result from parsing (-expected +actual)\n%s", diff)
}
}
+1 -1
View File
@@ -63,7 +63,7 @@ func parseRpmDB(resolver source.FilePathResolver, dbLocation source.Location, re
p := pkg.Package{
Name: entry.Name,
Version: toELVersion(metadata),
Locations: []source.Location{dbLocation},
Locations: source.NewLocationSet(dbLocation),
FoundBy: catalogerName,
Type: pkg.RpmPkg,
MetadataType: pkg.RpmdbMetadataType,
+2 -2
View File
@@ -71,7 +71,7 @@ func TestParseRpmDB(t *testing.T) {
"dive": {
Name: "dive",
Version: "0.9.2-1",
Locations: []source.Location{dbLocation},
Locations: source.NewLocationSet(dbLocation),
FoundBy: catalogerName,
Type: pkg.RpmPkg,
MetadataType: pkg.RpmdbMetadataType,
@@ -98,7 +98,7 @@ func TestParseRpmDB(t *testing.T) {
"dive": {
Name: "dive",
Version: "0.9.2-1",
Locations: []source.Location{dbLocation},
Locations: source.NewLocationSet(dbLocation),
FoundBy: catalogerName,
Type: pkg.RpmPkg,
MetadataType: pkg.RpmdbMetadataType,
@@ -1,4 +1,4 @@
package cpe
package pkg
import (
"sort"
@@ -6,15 +6,15 @@ import (
"github.com/facebookincubator/nvdtools/wfn"
)
var _ sort.Interface = (*BySpecificity)(nil)
var _ sort.Interface = (*CPEBySpecificity)(nil)
type BySpecificity []wfn.Attributes
type CPEBySpecificity []wfn.Attributes
func (c BySpecificity) Len() int { return len(c) }
func (c CPEBySpecificity) Len() int { return len(c) }
func (c BySpecificity) Swap(i, j int) { c[i], c[j] = c[j], c[i] }
func (c CPEBySpecificity) Swap(i, j int) { c[i], c[j] = c[j], c[i] }
func (c BySpecificity) Less(i, j int) bool {
func (c CPEBySpecificity) Less(i, j int) bool {
iScore := weightedCountForSpecifiedFields(c[i])
jScore := weightedCountForSpecifiedFields(c[j])
@@ -1,40 +1,32 @@
package cpe
package pkg
import (
"sort"
"testing"
"github.com/anchore/syft/syft/pkg"
"github.com/stretchr/testify/assert"
)
func mustCPE(c string) pkg.CPE {
return must(pkg.NewCPE(c))
}
func must(c pkg.CPE, e error) pkg.CPE {
if e != nil {
panic(e)
}
return c
func mustCPE(c string) CPE {
return must(NewCPE(c))
}
func TestCPESpecificity(t *testing.T) {
tests := []struct {
name string
input []pkg.CPE
expected []pkg.CPE
input []CPE
expected []CPE
}{
{
name: "sort strictly by wfn *",
input: []pkg.CPE{
input: []CPE{
mustCPE("cpe:2.3:a:*:package:1:*:*:*:*:*:*:*"),
mustCPE("cpe:2.3:a:some:package:1:*:*:*:*:*:*:*"),
mustCPE("cpe:2.3:a:*:package:1:*:*:*:*:some:*:*"),
mustCPE("cpe:2.3:a:some:package:1:*:*:*:*:some:*:*"),
mustCPE("cpe:2.3:a:some:package:*:*:*:*:*:*:*:*"),
},
expected: []pkg.CPE{
expected: []CPE{
mustCPE("cpe:2.3:a:some:package:1:*:*:*:*:some:*:*"),
mustCPE("cpe:2.3:a:some:package:1:*:*:*:*:*:*:*"),
mustCPE("cpe:2.3:a:some:package:*:*:*:*:*:*:*:*"),
@@ -44,7 +36,7 @@ func TestCPESpecificity(t *testing.T) {
},
{
name: "sort strictly by field length",
input: []pkg.CPE{
input: []CPE{
mustCPE("cpe:2.3:a:1:22:1:*:*:*:*:1:*:*"),
mustCPE("cpe:2.3:a:55555:1:1:*:*:*:*:1:*:*"),
mustCPE("cpe:2.3:a:1:1:333:*:*:*:*:1:*:*"),
@@ -52,7 +44,7 @@ func TestCPESpecificity(t *testing.T) {
mustCPE("cpe:2.3:a:1:1:1:*:*:*:*:1:*:*"),
mustCPE("cpe:2.3:a:1:1:1:*:*:*:*:4444:*:*"),
},
expected: []pkg.CPE{
expected: []CPE{
mustCPE("cpe:2.3:a:1:666666:1:*:*:*:*:1:*:*"),
mustCPE("cpe:2.3:a:55555:1:1:*:*:*:*:1:*:*"),
mustCPE("cpe:2.3:a:1:1:1:*:*:*:*:4444:*:*"),
@@ -63,7 +55,7 @@ func TestCPESpecificity(t *testing.T) {
},
{
name: "sort by mix of field length and specificity",
input: []pkg.CPE{
input: []CPE{
mustCPE("cpe:2.3:a:1:666666:*:*:*:*:*:1:*:*"),
mustCPE("cpe:2.3:a:*:1:1:*:*:*:*:4444:*:*"),
mustCPE("cpe:2.3:a:1:*:333:*:*:*:*:*:*:*"),
@@ -71,7 +63,7 @@ func TestCPESpecificity(t *testing.T) {
mustCPE("cpe:2.3:a:1:22:1:*:*:*:*:1:*:*"),
mustCPE("cpe:2.3:a:55555:1:1:*:*:*:*:1:*:*"),
},
expected: []pkg.CPE{
expected: []CPE{
mustCPE("cpe:2.3:a:55555:1:1:*:*:*:*:1:*:*"),
mustCPE("cpe:2.3:a:1:22:1:*:*:*:*:1:*:*"),
mustCPE("cpe:2.3:a:1:1:1:*:*:*:*:1:*:*"),
@@ -82,7 +74,7 @@ func TestCPESpecificity(t *testing.T) {
},
{
name: "sort by mix of field length, specificity, dash",
input: []pkg.CPE{
input: []CPE{
mustCPE("cpe:2.3:a:alpine:alpine_keys:2.3-r1:*:*:*:*:*:*:*"),
mustCPE("cpe:2.3:a:alpine_keys:alpine_keys:2.3-r1:*:*:*:*:*:*:*"),
mustCPE("cpe:2.3:a:alpine-keys:alpine_keys:2.3-r1:*:*:*:*:*:*:*"),
@@ -90,7 +82,7 @@ func TestCPESpecificity(t *testing.T) {
mustCPE("cpe:2.3:a:alpine-keys:alpine-keys:2.3-r1:*:*:*:*:*:*:*"),
mustCPE("cpe:2.3:a:alpine_keys:alpine-keys:2.3-r1:*:*:*:*:*:*:*"),
},
expected: []pkg.CPE{
expected: []CPE{
mustCPE("cpe:2.3:a:alpine-keys:alpine-keys:2.3-r1:*:*:*:*:*:*:*"),
mustCPE("cpe:2.3:a:alpine-keys:alpine_keys:2.3-r1:*:*:*:*:*:*:*"),
mustCPE("cpe:2.3:a:alpine_keys:alpine-keys:2.3-r1:*:*:*:*:*:*:*"),
@@ -103,7 +95,7 @@ func TestCPESpecificity(t *testing.T) {
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
sort.Sort(BySpecificity(test.input))
sort.Sort(CPEBySpecificity(test.input))
assert.Equal(t, test.expected, test.input)
})
}
+38
View File
@@ -0,0 +1,38 @@
package pkg
import (
"github.com/anchore/packageurl-go"
"github.com/anchore/syft/syft/linux"
)
type DartPubMetadata struct {
Name string `mapstructure:"name" json:"name"`
Version string `mapstructure:"version" json:"version"`
HostedURL string `mapstructure:"hosted_url" json:"hosted_url,omitempty"`
VcsURL string `mapstructure:"vcs_url" json:"vcs_url,omitempty"`
}
func (m DartPubMetadata) PackageURL(_ *linux.Release) string {
var qualifiers packageurl.Qualifiers
if m.HostedURL != "" {
qualifiers = append(qualifiers, packageurl.Qualifier{
Key: "hosted_url",
Value: m.HostedURL,
})
} else if m.VcsURL != "" { // Default to using Hosted if somehow both are provided
qualifiers = append(qualifiers, packageurl.Qualifier{
Key: "vcs_url",
Value: m.VcsURL,
})
}
return packageurl.NewPackageURL(
packageurl.TypePub,
"",
m.Name,
m.Version,
qualifiers,
"",
).ToString()
}
+1 -1
View File
@@ -20,7 +20,7 @@ var jenkinsPluginPomPropertiesGroupIDs = []string{
// JavaMetadata encapsulates all Java ecosystem metadata for a package as well as an (optional) parent relationship.
type JavaMetadata struct {
VirtualPath string `json:"virtualPath"`
VirtualPath string `json:"virtualPath" cyclonedx:"virtualPath"` // we need to include the virtual path in cyclonedx documents to prevent deduplication of jars within jars
Manifest *JavaManifest `mapstructure:"Manifest" json:"manifest,omitempty"`
PomProperties *PomProperties `mapstructure:"PomProperties" json:"pomProperties,omitempty" cyclonedx:"-"`
PomProject *PomProject `mapstructure:"PomProject" json:"pomProject,omitempty"`
+4
View File
@@ -19,6 +19,7 @@ const (
Ruby Language = "ruby"
Go Language = "go"
Rust Language = "rust"
Dart Language = "dart"
)
// AllLanguages is a set of all programming languages detected by syft.
@@ -30,6 +31,7 @@ var AllLanguages = []Language{
Ruby,
Go,
Rust,
Dart,
}
// String returns the string representation of the language.
@@ -62,6 +64,8 @@ func LanguageByName(name string) Language {
return Ruby
case purlCargoPkgType:
return Rust
case packageurl.TypePub, string(Dart):
return Dart
default:
return UnknownLanguage
}
+6 -1
View File
@@ -1,9 +1,10 @@
package pkg
import (
"testing"
"github.com/scylladb/go-set/strset"
"github.com/stretchr/testify/assert"
"testing"
)
func TestLanguageFromPURL(t *testing.T) {
@@ -29,6 +30,10 @@ func TestLanguageFromPURL(t *testing.T) {
purl: "pkg:golang/github.com/gorilla/context@234fd47e07d1004f0aed9c",
want: Go,
},
{
purl: "pkg:pub/util@1.2.34",
want: Dart,
},
{
purl: "pkg:cargo/clap@2.33.0",
want: Rust,
+25
View File
@@ -0,0 +1,25 @@
package pkg
import (
"sort"
)
func mergeCPEs(a, b []CPE) (result []CPE) {
aCPEs := make(map[string]CPE)
// keep all CPEs from a and create a quick string-based lookup
for _, aCPE := range a {
aCPEs[aCPE.BindToFmtString()] = aCPE
result = append(result, aCPE)
}
// keep all unique CPEs from b
for _, bCPE := range b {
if _, exists := aCPEs[bCPE.BindToFmtString()]; !exists {
result = append(result, bCPE)
}
}
sort.Sort(CPEBySpecificity(result))
return result
}
+3
View File
@@ -17,6 +17,7 @@ const (
JavaMetadataType MetadataType = "JavaMetadata"
NpmPackageJSONMetadataType MetadataType = "NpmPackageJsonMetadata"
RpmdbMetadataType MetadataType = "RpmdbMetadata"
DartPubMetadataType MetadataType = "DartPubMetadata"
PythonPackageMetadataType MetadataType = "PythonPackageMetadata"
RustCargoPackageMetadataType MetadataType = "RustCargoPackageMetadata"
KbPackageMetadataType MetadataType = "KbPackageMetadata"
@@ -31,6 +32,7 @@ var AllMetadataTypes = []MetadataType{
JavaMetadataType,
NpmPackageJSONMetadataType,
RpmdbMetadataType,
DartPubMetadataType,
PythonPackageMetadataType,
RustCargoPackageMetadataType,
KbPackageMetadataType,
@@ -45,6 +47,7 @@ var MetadataTypeByName = map[MetadataType]reflect.Type{
JavaMetadataType: reflect.TypeOf(JavaMetadata{}),
NpmPackageJSONMetadataType: reflect.TypeOf(NpmPackageJSONMetadata{}),
RpmdbMetadataType: reflect.TypeOf(RpmdbMetadata{}),
DartPubMetadataType: reflect.TypeOf(DartPubMetadata{}),
PythonPackageMetadataType: reflect.TypeOf(PythonPackageMetadata{}),
RustCargoPackageMetadataType: reflect.TypeOf(CargoMetadata{}),
KbPackageMetadataType: reflect.TypeOf(KbPackageMetadata{}),
+30 -12
View File
@@ -14,18 +14,18 @@ import (
// Package represents an application or library that has been bundled into a distributable format.
// TODO: if we ignore FoundBy for ID generation should we merge the field to show it was found in two places?
type Package struct {
id artifact.ID `hash:"ignore"`
Name string // the package name
Version string // the version of the package
FoundBy string `cyclonedx:"foundBy"` // the specific cataloger that discovered this package
Locations []source.Location // the locations that lead to the discovery of this package (note: this is not necessarily the locations that make up this package)
Licenses []string // licenses discovered with the package metadata
Language Language `cyclonedx:"language"` // the language ecosystem this package belongs to (e.g. JavaScript, Python, etc)
Type Type `cyclonedx:"type"` // the package type (e.g. Npm, Yarn, Python, Rpm, Deb, etc)
CPEs []CPE `hash:"ignore"` // all possible Common Platform Enumerators (note: this is NOT included in the definition of the ID since all fields on a CPE are derived from other fields)
PURL string `hash:"ignore"` // the Package URL (see https://github.com/package-url/purl-spec) (note: this is NOT included in the definition of the ID since all fields on a pURL are derived from other fields)
MetadataType MetadataType `cyclonedx:"metadataType"` // the shape of the additional data in the "metadata" field
Metadata interface{} // additional data found while parsing the package source
id artifact.ID `hash:"ignore"`
Name string // the package name
Version string // the version of the package
FoundBy string `cyclonedx:"foundBy"` // the specific cataloger that discovered this package
Locations source.LocationSet // the locations that lead to the discovery of this package (note: this is not necessarily the locations that make up this package)
Licenses []string // licenses discovered with the package metadata
Language Language `cyclonedx:"language"` // the language ecosystem this package belongs to (e.g. JavaScript, Python, etc)
Type Type `cyclonedx:"type"` // the package type (e.g. Npm, Yarn, Python, Rpm, Deb, etc)
CPEs []CPE `hash:"ignore"` // all possible Common Platform Enumerators (note: this is NOT included in the definition of the ID since all fields on a CPE are derived from other fields)
PURL string `hash:"ignore"` // the Package URL (see https://github.com/package-url/purl-spec)
MetadataType MetadataType `cyclonedx:"metadataType"` // the shape of the additional data in the "metadata" field
Metadata interface{} // additional data found while parsing the package source
}
func (p *Package) SetID() {
@@ -46,3 +46,21 @@ func (p Package) ID() artifact.ID {
func (p Package) String() string {
return fmt.Sprintf("Pkg(name=%q version=%q type=%q id=%q)", p.Name, p.Version, p.Type, p.id)
}
func (p *Package) merge(other Package) error {
if p.id != other.id {
return fmt.Errorf("cannot merge packages with different IDs: %q vs %q", p.id, other.id)
}
if p.PURL != other.PURL {
log.Warnf("merging packages have with different pURLs: %q=%q vs %q=%q", p.id, p.PURL, other.id, other.PURL)
}
p.Locations.Add(other.Locations.ToSlice()...)
p.CPEs = mergeCPEs(p.CPEs, other.CPEs)
if p.PURL == "" {
p.PURL = other.PURL
}
return nil
}
+266 -32
View File
@@ -3,24 +3,28 @@ package pkg
import (
"testing"
"github.com/anchore/stereoscope/pkg/file"
"github.com/anchore/syft/syft/source"
"github.com/google/go-cmp/cmp"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestFingerprint(t *testing.T) {
func TestIDUniqueness(t *testing.T) {
originalLocation := source.Location{
Coordinates: source.Coordinates{
RealPath: "39.0742° N, 21.8243° E",
FileSystemID: "Earth",
},
VirtualPath: "/Ancient-Greece",
}
originalPkg := Package{
Name: "pi",
Version: "3.14",
FoundBy: "Archimedes",
Locations: []source.Location{
{
Coordinates: source.Coordinates{
RealPath: "39.0742° N, 21.8243° E",
FileSystemID: "Earth",
},
VirtualPath: "/Ancient-Greece",
},
},
Locations: source.NewLocationSet(
originalLocation,
),
Licenses: []string{
"cc0-1.0",
"MIT",
@@ -45,9 +49,9 @@ func TestFingerprint(t *testing.T) {
// this is a set of differential tests, ensuring that select mutations are reflected in the fingerprint (or not)
tests := []struct {
name string
transform func(pkg Package) Package
expectIdentical bool
name string
transform func(pkg Package) Package
expectedIDComparison assert.ComparisonAssertionFunc
}{
{
name: "go case (no transform)",
@@ -55,7 +59,7 @@ func TestFingerprint(t *testing.T) {
// do nothing!
return pkg
},
expectIdentical: true,
expectedIDComparison: assert.Equal,
},
{
name: "same metadata is ignored",
@@ -72,7 +76,7 @@ func TestFingerprint(t *testing.T) {
}
return pkg
},
expectIdentical: true,
expectedIDComparison: assert.Equal,
},
{
name: "licenses order is ignored",
@@ -84,7 +88,7 @@ func TestFingerprint(t *testing.T) {
}
return pkg
},
expectIdentical: true,
expectedIDComparison: assert.Equal,
},
{
name: "name is reflected",
@@ -92,7 +96,42 @@ func TestFingerprint(t *testing.T) {
pkg.Name = "new!"
return pkg
},
expectIdentical: false,
expectedIDComparison: assert.NotEqual,
},
{
name: "location is reflected",
transform: func(pkg Package) Package {
locations := source.NewLocationSet(pkg.Locations.ToSlice()...)
locations.Add(source.NewLocation("/somewhere/new"))
pkg.Locations = locations
return pkg
},
expectedIDComparison: assert.NotEqual,
},
{
name: "same path for different filesystem is NOT reflected",
transform: func(pkg Package) Package {
newLocation := originalLocation
newLocation.FileSystemID = "Mars"
pkg.Locations = source.NewLocationSet(newLocation)
return pkg
},
expectedIDComparison: assert.Equal,
},
{
name: "multiple equivalent paths for different filesystem is NOT reflected",
transform: func(pkg Package) Package {
newLocation := originalLocation
newLocation.FileSystemID = "Mars"
locations := source.NewLocationSet(pkg.Locations.ToSlice()...)
locations.Add(newLocation, originalLocation)
pkg.Locations = locations
return pkg
},
expectedIDComparison: assert.Equal,
},
{
name: "version is reflected",
@@ -100,7 +139,7 @@ func TestFingerprint(t *testing.T) {
pkg.Version = "new!"
return pkg
},
expectIdentical: false,
expectedIDComparison: assert.NotEqual,
},
{
name: "licenses is reflected",
@@ -108,7 +147,7 @@ func TestFingerprint(t *testing.T) {
pkg.Licenses = []string{"new!"}
return pkg
},
expectIdentical: false,
expectedIDComparison: assert.NotEqual,
},
{
name: "type is reflected",
@@ -116,7 +155,7 @@ func TestFingerprint(t *testing.T) {
pkg.Type = RustPkg
return pkg
},
expectIdentical: false,
expectedIDComparison: assert.NotEqual,
},
{
name: "metadata type is reflected",
@@ -124,7 +163,7 @@ func TestFingerprint(t *testing.T) {
pkg.MetadataType = RustCargoPackageMetadataType
return pkg
},
expectIdentical: false,
expectedIDComparison: assert.NotEqual,
},
{
name: "CPEs is ignored",
@@ -132,7 +171,7 @@ func TestFingerprint(t *testing.T) {
pkg.CPEs = []CPE{}
return pkg
},
expectIdentical: true,
expectedIDComparison: assert.Equal,
},
{
name: "pURL is ignored",
@@ -140,7 +179,7 @@ func TestFingerprint(t *testing.T) {
pkg.PURL = "new!"
return pkg
},
expectIdentical: true,
expectedIDComparison: assert.Equal,
},
{
name: "language is reflected",
@@ -148,7 +187,7 @@ func TestFingerprint(t *testing.T) {
pkg.Language = Rust
return pkg
},
expectIdentical: false,
expectedIDComparison: assert.NotEqual,
},
{
name: "foundBy is reflected",
@@ -156,7 +195,7 @@ func TestFingerprint(t *testing.T) {
pkg.FoundBy = "new!"
return pkg
},
expectIdentical: false,
expectedIDComparison: assert.NotEqual,
},
{
name: "metadata mutation is reflected",
@@ -166,7 +205,7 @@ func TestFingerprint(t *testing.T) {
pkg.Metadata = metadata
return pkg
},
expectIdentical: false,
expectedIDComparison: assert.NotEqual,
},
{
name: "new metadata is reflected",
@@ -176,7 +215,7 @@ func TestFingerprint(t *testing.T) {
}
return pkg
},
expectIdentical: false,
expectedIDComparison: assert.NotEqual,
},
{
name: "nil metadata is reflected",
@@ -184,7 +223,7 @@ func TestFingerprint(t *testing.T) {
pkg.Metadata = nil
return pkg
},
expectIdentical: false,
expectedIDComparison: assert.NotEqual,
},
}
@@ -199,12 +238,207 @@ func TestFingerprint(t *testing.T) {
transformedFingerprint := transformedPkg.ID()
assert.NotEmpty(t, transformedFingerprint)
if test.expectIdentical {
assert.Equal(t, originalFingerprint, transformedFingerprint)
} else {
assert.NotEqual(t, originalFingerprint, transformedFingerprint)
test.expectedIDComparison(t, originalFingerprint, transformedFingerprint)
})
}
}
func TestPackage_Merge(t *testing.T) {
originalLocation := source.Location{
Coordinates: source.Coordinates{
RealPath: "39.0742° N, 21.8243° E",
FileSystemID: "Earth",
},
VirtualPath: "/Ancient-Greece",
}
similarLocation := originalLocation
similarLocation.FileSystemID = "Mars"
tests := []struct {
name string
subject Package
other Package
expected *Package
}{
{
name: "merge two packages (different cpes + locations)",
subject: Package{
Name: "pi",
Version: "3.14",
FoundBy: "Archimedes",
Locations: source.NewLocationSet(
originalLocation,
),
Licenses: []string{
"cc0-1.0",
"MIT",
},
Language: "math",
Type: PythonPkg,
CPEs: []CPE{
must(NewCPE(`cpe:2.3:a:Archimedes:pi:3.14:*:*:*:*:math:*:*`)),
},
PURL: "pkg:pypi/pi@3.14",
MetadataType: PythonPackageMetadataType,
Metadata: PythonPackageMetadata{
Name: "pi",
Version: "3.14",
License: "cc0-1.0",
Author: "Archimedes",
AuthorEmail: "Archimedes@circles.io",
Platform: "universe",
SitePackagesRootPath: "Pi",
},
},
other: Package{
Name: "pi",
Version: "3.14",
FoundBy: "Archimedes",
Locations: source.NewLocationSet(
similarLocation, // NOTE: difference; we have a different layer but the same path
),
Licenses: []string{
"cc0-1.0",
"MIT",
},
Language: "math",
Type: PythonPkg,
CPEs: []CPE{
must(NewCPE(`cpe:2.3:a:DIFFERENT:pi:3.14:*:*:*:*:math:*:*`)), // NOTE: difference
},
PURL: "pkg:pypi/pi@3.14",
MetadataType: PythonPackageMetadataType,
Metadata: PythonPackageMetadata{
Name: "pi",
Version: "3.14",
License: "cc0-1.0",
Author: "Archimedes",
AuthorEmail: "Archimedes@circles.io",
Platform: "universe",
SitePackagesRootPath: "Pi",
},
},
expected: &Package{
Name: "pi",
Version: "3.14",
FoundBy: "Archimedes",
Locations: source.NewLocationSet(
originalLocation,
similarLocation, // NOTE: merge!
),
Licenses: []string{
"cc0-1.0",
"MIT",
},
Language: "math",
Type: PythonPkg,
CPEs: []CPE{
must(NewCPE(`cpe:2.3:a:Archimedes:pi:3.14:*:*:*:*:math:*:*`)),
must(NewCPE(`cpe:2.3:a:DIFFERENT:pi:3.14:*:*:*:*:math:*:*`)), // NOTE: merge!
},
PURL: "pkg:pypi/pi@3.14",
MetadataType: PythonPackageMetadataType,
Metadata: PythonPackageMetadata{
Name: "pi",
Version: "3.14",
License: "cc0-1.0",
Author: "Archimedes",
AuthorEmail: "Archimedes@circles.io",
Platform: "universe",
SitePackagesRootPath: "Pi",
},
},
},
{
name: "error when there are different IDs",
subject: Package{
Name: "pi",
Version: "3.14",
FoundBy: "Archimedes",
Locations: source.NewLocationSet(
originalLocation,
),
Licenses: []string{
"cc0-1.0",
"MIT",
},
Language: "math",
Type: PythonPkg,
CPEs: []CPE{
must(NewCPE(`cpe:2.3:a:Archimedes:pi:3.14:*:*:*:*:math:*:*`)),
},
PURL: "pkg:pypi/pi@3.14",
MetadataType: PythonPackageMetadataType,
Metadata: PythonPackageMetadata{
Name: "pi",
Version: "3.14",
License: "cc0-1.0",
Author: "Archimedes",
AuthorEmail: "Archimedes@circles.io",
Platform: "universe",
SitePackagesRootPath: "Pi",
},
},
other: Package{
Name: "pi-DIFFERENT", // difference
Version: "3.14",
FoundBy: "Archimedes",
Locations: source.NewLocationSet(
originalLocation,
),
Licenses: []string{
"cc0-1.0",
"MIT",
},
Language: "math",
Type: PythonPkg,
CPEs: []CPE{
must(NewCPE(`cpe:2.3:a:Archimedes:pi:3.14:*:*:*:*:math:*:*`)),
},
PURL: "pkg:pypi/pi@3.14",
MetadataType: PythonPackageMetadataType,
Metadata: PythonPackageMetadata{
Name: "pi",
Version: "3.14",
License: "cc0-1.0",
Author: "Archimedes",
AuthorEmail: "Archimedes@circles.io",
Platform: "universe",
SitePackagesRootPath: "Pi",
},
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
tt.subject.SetID()
tt.other.SetID()
err := tt.subject.merge(tt.other)
if tt.expected == nil {
require.Error(t, err)
return
}
require.NoError(t, err)
tt.expected.SetID()
require.Equal(t, tt.expected.id, tt.subject.id)
if diff := cmp.Diff(*tt.expected, tt.subject,
cmp.AllowUnexported(Package{}),
cmp.Comparer(
func(x, y source.LocationSet) bool {
return cmp.Equal(
x.ToSlice(), y.ToSlice(),
cmp.AllowUnexported(source.Location{}),
cmp.AllowUnexported(file.Reference{}),
)
},
),
); diff != "" {
t.Errorf("unexpected result from parsing (-expected +actual)\n%s", diff)
}
})
}
}
@@ -18,10 +18,10 @@ func TestOwnershipByFilesRelationship(t *testing.T) {
name: "owns-by-real-path",
setup: func(t testing.TB) ([]Package, []artifact.Relationship) {
parent := Package{
Locations: []source.Location{
Locations: source.NewLocationSet(
source.NewVirtualLocation("/a/path", "/another/path"),
source.NewVirtualLocation("/b/path", "/bee/path"),
},
),
Type: RpmPkg,
MetadataType: RpmdbMetadataType,
Metadata: RpmdbMetadata{
@@ -35,10 +35,10 @@ func TestOwnershipByFilesRelationship(t *testing.T) {
parent.SetID()
child := Package{
Locations: []source.Location{
Locations: source.NewLocationSet(
source.NewVirtualLocation("/c/path", "/another/path"),
source.NewVirtualLocation("/d/path", "/another/path"),
},
),
Type: NpmPkg,
}
child.SetID()
@@ -61,10 +61,10 @@ func TestOwnershipByFilesRelationship(t *testing.T) {
name: "owns-by-virtual-path",
setup: func(t testing.TB) ([]Package, []artifact.Relationship) {
parent := Package{
Locations: []source.Location{
Locations: source.NewLocationSet(
source.NewVirtualLocation("/a/path", "/some/other/path"),
source.NewVirtualLocation("/b/path", "/bee/path"),
},
),
Type: RpmPkg,
MetadataType: RpmdbMetadataType,
Metadata: RpmdbMetadata{
@@ -78,10 +78,10 @@ func TestOwnershipByFilesRelationship(t *testing.T) {
parent.SetID()
child := Package{
Locations: []source.Location{
Locations: source.NewLocationSet(
source.NewVirtualLocation("/c/path", "/another/path"),
source.NewLocation("/d/path"),
},
),
Type: NpmPkg,
}
child.SetID()
@@ -103,10 +103,10 @@ func TestOwnershipByFilesRelationship(t *testing.T) {
name: "ignore-empty-path",
setup: func(t testing.TB) ([]Package, []artifact.Relationship) {
parent := Package{
Locations: []source.Location{
Locations: source.NewLocationSet(
source.NewVirtualLocation("/a/path", "/some/other/path"),
source.NewVirtualLocation("/b/path", "/bee/path"),
},
),
Type: RpmPkg,
MetadataType: RpmdbMetadataType,
Metadata: RpmdbMetadata{
@@ -121,10 +121,10 @@ func TestOwnershipByFilesRelationship(t *testing.T) {
parent.SetID()
child := Package{
Locations: []source.Location{
Locations: source.NewLocationSet(
source.NewVirtualLocation("/c/path", "/another/path"),
source.NewLocation("/d/path"),
},
),
Type: NpmPkg,
}
+6
View File
@@ -20,6 +20,7 @@ const (
GoModulePkg Type = "go-module"
RustPkg Type = "rust-crate"
KbPkg Type = "msrc-kb"
DartPubPkg Type = "dart-pub"
)
// AllPkgs represents all supported package types
@@ -36,6 +37,7 @@ var AllPkgs = []Type{
GoModulePkg,
RustPkg,
KbPkg,
DartPubPkg,
}
// PackageURLType returns the PURL package type for the current package.
@@ -61,6 +63,8 @@ func (t Type) PackageURLType() string {
return packageurl.TypeGolang
case RustPkg:
return "cargo"
case DartPubPkg:
return packageurl.TypePub
default:
// TODO: should this be a "generic" purl type instead?
return ""
@@ -98,6 +102,8 @@ func TypeByName(name string) Type {
return GemPkg
case "cargo", "crate":
return RustPkg
case packageurl.TypePub:
return DartPubPkg
default:
return UnknownPkg
}
+6 -1
View File
@@ -1,9 +1,10 @@
package pkg
import (
"github.com/scylladb/go-set/strset"
"testing"
"github.com/scylladb/go-set/strset"
"github.com/stretchr/testify/assert"
)
@@ -46,6 +47,10 @@ func TestTypeFromPURL(t *testing.T) {
purl: "pkg:cargo/clap@2.33.0",
expected: RustPkg,
},
{
purl: "pkg:pub/util@1.2.34?hosted_url=pub.hosted.org",
expected: DartPubPkg,
},
{
purl: "pkg:composer/laravel/laravel@5.5.0",
expected: PhpComposerPkg,
+14
View File
@@ -25,6 +25,20 @@ func TestPackageURL(t *testing.T) {
},
expected: "pkg:golang/github.com/anchore/syft@v0.1.0",
},
{
name: "pub",
pkg: Package{
Name: "bad-name",
Version: "0.1.0",
Type: DartPubPkg,
Metadata: DartPubMetadata{
Name: "name",
Version: "0.2.0",
HostedURL: "pub.hosted.org",
},
},
expected: "pkg:pub/name@0.2.0?hosted_url=pub.hosted.org",
},
{
name: "python",
pkg: Package{
+86
View File
@@ -0,0 +1,86 @@
package source
import (
"sort"
"github.com/mitchellh/hashstructure/v2"
"github.com/scylladb/go-set/strset"
)
type CoordinateSet struct {
set map[Coordinates]struct{}
}
func NewCoordinateSet(coordinates ...Coordinates) (s CoordinateSet) {
for _, l := range coordinates {
s.Add(l)
}
return s
}
func (s *CoordinateSet) Add(coordinates ...Coordinates) {
if s.set == nil {
s.set = make(map[Coordinates]struct{})
}
for _, l := range coordinates {
s.set[l] = struct{}{}
}
}
func (s CoordinateSet) Remove(coordinates ...Coordinates) {
if s.set == nil {
return
}
for _, l := range coordinates {
delete(s.set, l)
}
}
func (s CoordinateSet) Contains(l Coordinates) bool {
if s.set == nil {
return false
}
_, ok := s.set[l]
return ok
}
func (s CoordinateSet) Paths() []string {
if s.set == nil {
return nil
}
paths := strset.New()
for _, c := range s.ToSlice() {
paths.Add(c.RealPath)
}
pathSlice := paths.List()
sort.Strings(pathSlice)
return pathSlice
}
func (s CoordinateSet) ToSlice() []Coordinates {
if s.set == nil {
return nil
}
coordinates := make([]Coordinates, len(s.set))
idx := 0
for v := range s.set {
coordinates[idx] = v
idx++
}
sort.SliceStable(coordinates, func(i, j int) bool {
if coordinates[i].RealPath == coordinates[j].RealPath {
return coordinates[i].FileSystemID < coordinates[j].FileSystemID
}
return coordinates[i].RealPath < coordinates[j].RealPath
})
return coordinates
}
func (s CoordinateSet) Hash() (uint64, error) {
return hashstructure.Hash(s.ToSlice(), hashstructure.FormatV2, &hashstructure.HashOptions{
ZeroNil: true,
SlicesAsSets: true,
})
}
+117
View File
@@ -0,0 +1,117 @@
package source
import (
"github.com/anchore/syft/syft/artifact"
"github.com/stretchr/testify/require"
"testing"
"github.com/stretchr/testify/assert"
)
func TestCoordinatesSet(t *testing.T) {
binA := Coordinates{
RealPath: "/bin",
FileSystemID: "a",
}
binB := Coordinates{
RealPath: "/bin",
FileSystemID: "b",
}
tests := []struct {
name string
input []Coordinates
expected []Coordinates
}{
{
name: "de-dup same location",
input: []Coordinates{
binA, binA, binA,
},
expected: []Coordinates{
binA,
},
},
{
name: "dont de-dup different filesystem",
input: []Coordinates{
binB, binA,
},
expected: []Coordinates{
binA, binB,
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
set := NewCoordinateSet(test.input...)
assert.Equal(t, test.expected, set.ToSlice())
})
}
}
func TestCoordinateSet_Hash(t *testing.T) {
etcA := Coordinates{
RealPath: "/etc",
FileSystemID: "a",
}
etcB := Coordinates{
RealPath: "/etc",
FileSystemID: "b",
}
binA := Coordinates{
RealPath: "/bin",
FileSystemID: "a",
}
binB := Coordinates{
RealPath: "/bin",
FileSystemID: "b",
}
tests := []struct {
name string
setA CoordinateSet
setB CoordinateSet
want assert.ComparisonAssertionFunc
}{
{
name: "empty sets have the same hash",
setA: NewCoordinateSet(),
setB: NewCoordinateSet(),
want: assert.Equal,
},
{
name: "sets with same elements have the same hash",
setA: NewCoordinateSet(binA, etcA),
setB: NewCoordinateSet(etcA, binA),
want: assert.Equal,
},
{
name: "sets with different elements have different hashes",
setA: NewCoordinateSet(binA, etcA),
setB: NewCoordinateSet(binA),
want: assert.NotEqual,
},
{
name: "sets with same paths but different FS IDs have different hashes",
setA: NewCoordinateSet(etcA, binA),
setB: NewCoordinateSet(etcB, binB),
want: assert.NotEqual,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
gotA, err := artifact.IDByHash(tt.setA)
require.NoError(t, err)
gotB, err := artifact.IDByHash(tt.setB)
require.NoError(t, err)
tt.want(t, gotA, gotB)
})
}
}
-47
View File
@@ -2,7 +2,6 @@ package source
import (
"fmt"
"sort"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/artifact"
@@ -14,18 +13,6 @@ type Coordinates struct {
FileSystemID string `json:"layerID,omitempty" cyclonedx:"layerID"` // An ID representing the filesystem. For container images, this is a layer digest. For directories or a root filesystem, this is blank.
}
// CoordinateSet represents a set of string types.
type CoordinateSet map[Coordinates]struct{}
// NewCoordinateSet creates a CoordinateSet populated with values from the given slice.
func NewCoordinateSet(start ...Coordinates) CoordinateSet {
ret := make(CoordinateSet)
for _, s := range start {
ret.Add(s)
}
return ret
}
func (c Coordinates) ID() artifact.ID {
f, err := artifact.IDByHash(c)
if err != nil {
@@ -45,37 +32,3 @@ func (c Coordinates) String() string {
}
return fmt.Sprintf("Location<%s>", str)
}
// Add a string to the set.
func (s CoordinateSet) Add(i Coordinates) {
s[i] = struct{}{}
}
// Remove a string from the set.
func (s CoordinateSet) Remove(i Coordinates) {
delete(s, i)
}
// Contains indicates if the given string is contained within the set.
func (s CoordinateSet) Contains(i Coordinates) bool {
_, ok := s[i]
return ok
}
// ToSlice returns a sorted slice of Locations that are contained within the set.
func (s CoordinateSet) ToSlice() []Coordinates {
ret := make([]Coordinates, len(s))
idx := 0
for v := range s {
ret[idx] = v
idx++
}
sort.SliceStable(ret, func(i, j int) bool {
if ret[i].RealPath == ret[j].RealPath {
return ret[i].FileSystemID < ret[j].FileSystemID
}
return ret[i].RealPath < ret[j].RealPath
})
return ret
}
-51
View File
@@ -1,51 +0,0 @@
package source
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestCoordinateSet(t *testing.T) {
binA := Coordinates{
RealPath: "/bin",
FileSystemID: "a",
}
binB := Coordinates{
RealPath: "/bin",
FileSystemID: "b",
}
tests := []struct {
name string
input []Coordinates
expected []Coordinates
}{
{
name: "de-dup same location",
input: []Coordinates{
binA, binA, binA,
},
expected: []Coordinates{
binA,
},
},
{
name: "dont de-dup different filesystem",
input: []Coordinates{
binB, binA,
},
expected: []Coordinates{
binA, binB,
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
assert.Equal(t, test.expected, NewCoordinateSet(test.input...).ToSlice())
})
}
}
+6 -6
View File
@@ -132,10 +132,15 @@ func (r *directoryResolver) indexTree(root string, stager *progress.Stage) ([]st
}
func (r *directoryResolver) indexPath(path string, info os.FileInfo, err error) (string, error) {
// link cycles could cause a revisit --we should not allow this
if r.hasBeenIndexed(path) {
return "", nil
}
// ignore any path which a filter function returns true
for _, filterFn := range r.pathFilterFns {
if filterFn != nil && filterFn(path, info) {
if info.IsDir() {
if info != nil && info.IsDir() {
return "", fs.SkipDir
}
return "", nil
@@ -146,11 +151,6 @@ func (r *directoryResolver) indexPath(path string, info os.FileInfo, err error)
return "", nil
}
// link cycles could cause a revisit --we should not allow this
if r.hasBeenIndexed(path) {
return "", nil
}
if info == nil {
// walk may not be able to provide a FileInfo object, don't allow for this to stop indexing; keep track of the paths and continue.
r.errPaths[path] = fmt.Errorf("no file info observable at path=%q", path)
+21
View File
@@ -865,3 +865,24 @@ func Test_IncludeRootPathInIndex(t *testing.T) {
_, exists = resolver.metadata[ref.ID()]
require.True(t, exists)
}
func TestDirectoryResolver_indexPath(t *testing.T) {
// TODO: Ideally we can use an OS abstraction, which would obviate the need for real FS setup.
tempFile, err := os.CreateTemp("", "")
require.NoError(t, err)
resolver, err := newDirectoryResolver(tempFile.Name())
require.NoError(t, err)
t.Run("filtering path with nil os.FileInfo", func(t *testing.T) {
// We use one of these prefixes in order to trigger a pathFilterFn
filteredPath := unixSystemRuntimePrefixes[0]
var fileInfo os.FileInfo = nil
assert.NotPanics(t, func() {
_, err := resolver.indexPath(filteredPath, fileInfo, nil)
assert.NoError(t, err)
})
})
}
+6
View File
@@ -109,3 +109,9 @@ func (l Location) String() string {
}
return fmt.Sprintf("Location<%s>", str)
}
func (l Location) Equals(other Location) bool {
return l.RealPath == other.RealPath &&
l.VirtualPath == other.VirtualPath &&
l.FileSystemID == other.FileSystemID
}
+78
View File
@@ -0,0 +1,78 @@
package source
import (
"sort"
"github.com/mitchellh/hashstructure/v2"
)
type LocationSet struct {
set map[Location]struct{}
}
func NewLocationSet(locations ...Location) (s LocationSet) {
for _, l := range locations {
s.Add(l)
}
return s
}
func (s *LocationSet) Add(locations ...Location) {
if s.set == nil {
s.set = make(map[Location]struct{})
}
for _, l := range locations {
s.set[l] = struct{}{}
}
}
func (s LocationSet) Remove(locations ...Location) {
if s.set == nil {
return
}
for _, l := range locations {
delete(s.set, l)
}
}
func (s LocationSet) Contains(l Location) bool {
if s.set == nil {
return false
}
_, ok := s.set[l]
return ok
}
func (s LocationSet) ToSlice() []Location {
if s.set == nil {
return nil
}
locations := make([]Location, len(s.set))
idx := 0
for v := range s.set {
locations[idx] = v
idx++
}
sort.Sort(Locations(locations))
return locations
}
func (s *LocationSet) CoordinateSet() CoordinateSet {
if s.set == nil {
return NewCoordinateSet()
}
set := NewCoordinateSet()
for l := range s.set {
set.Add(l.Coordinates)
}
return set
}
func (s LocationSet) Hash() (uint64, error) {
// access paths and filesystem IDs are not considered when hashing a location set, only the real paths
return hashstructure.Hash(s.CoordinateSet().Paths(), hashstructure.FormatV2, &hashstructure.HashOptions{
ZeroNil: true,
SlicesAsSets: true,
})
}
+178
View File
@@ -0,0 +1,178 @@
package source
import (
"github.com/anchore/syft/syft/artifact"
"github.com/stretchr/testify/require"
"testing"
"github.com/stretchr/testify/assert"
)
func TestLocationSet(t *testing.T) {
etcHostsLinkVar := Location{
Coordinates: Coordinates{
RealPath: "/etc/hosts",
FileSystemID: "a",
},
VirtualPath: "/var/etc/hosts",
}
etcHostsLinkHome := Location{
Coordinates: Coordinates{
RealPath: "/etc/hosts",
FileSystemID: "a",
},
VirtualPath: "/home/wagoodman/hosts",
}
binA := Location{
Coordinates: Coordinates{
RealPath: "/bin",
FileSystemID: "a",
},
VirtualPath: "/usr/bin",
}
binB := Location{
Coordinates: Coordinates{
RealPath: "/bin",
FileSystemID: "b",
},
VirtualPath: "/usr/bin",
}
tests := []struct {
name string
input []Location
expected []Location
}{
{
name: "de-dup same location",
input: []Location{
binA, binA, binA,
},
expected: []Location{
binA,
},
},
{
name: "dont de-dup different filesystem",
input: []Location{
binB, binA,
},
expected: []Location{
binA, binB,
},
},
{
name: "dont de-dup different virtual paths",
input: []Location{
etcHostsLinkVar, etcHostsLinkHome,
},
expected: []Location{
etcHostsLinkHome, etcHostsLinkVar,
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
set := NewLocationSet(test.input...)
assert.Equal(t, test.expected, set.ToSlice())
})
}
}
func TestLocationSet_Hash(t *testing.T) {
etcAlink := Location{
Coordinates: Coordinates{
RealPath: "/etc/hosts",
FileSystemID: "a",
},
VirtualPath: "/var/etc/hosts",
}
etcA := Location{
Coordinates: Coordinates{
RealPath: "/etc/hosts",
FileSystemID: "a",
},
}
etcB := Location{
Coordinates: Coordinates{
RealPath: "/etc/hosts",
FileSystemID: "b",
},
}
binA := Location{
Coordinates: Coordinates{
RealPath: "/bin",
FileSystemID: "a",
},
VirtualPath: "/usr/bin",
}
binB := Location{
Coordinates: Coordinates{
RealPath: "/bin",
FileSystemID: "b",
},
VirtualPath: "/usr/bin",
}
tests := []struct {
name string
setA LocationSet
setB LocationSet
want assert.ComparisonAssertionFunc
}{
{
name: "empty sets have the same hash",
setA: NewLocationSet(),
setB: NewLocationSet(),
want: assert.Equal,
},
{
name: "sets with same elements accessed through different paths have the same hash",
setA: NewLocationSet(binA, etcA),
setB: NewLocationSet(etcAlink, binA),
want: assert.Equal,
},
{
name: "sets with same elements have the same hash",
setA: NewLocationSet(binA, etcA),
setB: NewLocationSet(etcA, binA),
want: assert.Equal,
},
{
name: "sets with different element counts have different hashes",
setA: NewLocationSet(binA, etcA),
setB: NewLocationSet(binA),
want: assert.NotEqual,
},
{
name: "sets with same path but different FS IDs have the same hash",
setA: NewLocationSet(binA),
setB: NewLocationSet(binB),
want: assert.Equal,
},
{
name: "sets with same paths but different FS IDs have the same hash",
setA: NewLocationSet(etcA, binA),
setB: NewLocationSet(binB, etcB),
want: assert.Equal,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
gotA, err := artifact.IDByHash(tt.setA)
require.NoError(t, err)
gotB, err := artifact.IDByHash(tt.setB)
require.NoError(t, err)
tt.want(t, gotA, gotB)
})
}
}
+21
View File
@@ -0,0 +1,21 @@
package source
type Locations []Location
func (l Locations) Len() int {
return len(l)
}
func (l Locations) Less(i, j int) bool {
if l[i].RealPath == l[j].RealPath {
if l[i].VirtualPath == l[j].VirtualPath {
return l[i].FileSystemID < l[j].FileSystemID
}
return l[i].VirtualPath < l[j].VirtualPath
}
return l[i].RealPath < l[j].RealPath
}
func (l Locations) Swap(i, j int) {
l[i], l[j] = l[j], l[i]
}
+19 -5
View File
@@ -8,6 +8,7 @@ import (
)
func TestPackagesCmdFlags(t *testing.T) {
hiddenPackagesImage := "docker-archive:" + getFixtureImage(t, "image-hidden-packages")
coverageImage := "docker-archive:" + getFixtureImage(t, "image-pkg-coverage")
//badBinariesImage := "docker-archive:" + getFixtureImage(t, "image-bad-binaries")
tmp := t.TempDir() + "/"
@@ -100,21 +101,34 @@ func TestPackagesCmdFlags(t *testing.T) {
},
},
{
name: "all-layers-scope-flag",
args: []string{"packages", "-o", "json", "-s", "all-layers", coverageImage},
name: "squashed-scope-flag-hidden-packages",
args: []string{"packages", "-o", "json", "-s", "squashed", hiddenPackagesImage},
assertions: []traitAssertion{
assertPackageCount(22),
assertPackageCount(162),
assertNotInOutput("vsftpd"), // hidden package
assertSuccessfulReturnCode,
},
},
{
name: "all-layers-scope-flag",
args: []string{"packages", "-o", "json", "-s", "all-layers", hiddenPackagesImage},
assertions: []traitAssertion{
assertPackageCount(163), // packages are now deduplicated for this case
assertInOutput("all-layers"),
assertInOutput("vsftpd"), // hidden package
assertSuccessfulReturnCode,
},
},
{
name: "all-layers-scope-flag-by-env",
args: []string{"packages", "-o", "json", coverageImage},
args: []string{"packages", "-o", "json", hiddenPackagesImage},
env: map[string]string{
"SYFT_PACKAGE_CATALOGER_SCOPE": "all-layers",
},
assertions: []traitAssertion{
assertPackageCount(22),
assertPackageCount(163), // packages are now deduplicated for this case
assertInOutput("all-layers"),
assertInOutput("vsftpd"), // hidden package
assertSuccessfulReturnCode,
},
},
@@ -0,0 +1,4 @@
FROM centos:7.9.2009
# all-layers scope should pickup on vsftpd
RUN yum install -y vsftpd
RUN yum remove -y vsftpd
@@ -186,6 +186,19 @@ var dirOnlyTestCases = []testCase{
"alcaeus/mongo-php-adapter": "1.1.11",
},
},
{
name: "find pubspec lock packages",
pkgType: pkg.DartPubPkg,
pkgLanguage: pkg.Dart,
pkgInfo: map[string]string{
"ansicolor": "1.1.1",
"archive": "2.0.13",
"args": "1.6.0",
"key_binder": "1.11.20",
"ale": "3.3.0",
"analyzer": "0.40.7",
},
},
}
var commonTestCases = []testCase{
+5 -2
View File
@@ -1,9 +1,10 @@
package integration
import (
"github.com/stretchr/testify/require"
"testing"
"github.com/stretchr/testify/require"
"github.com/anchore/syft/syft/linux"
"github.com/anchore/syft/syft/pkg/cataloger"
"github.com/google/go-cmp/cmp"
@@ -53,7 +54,7 @@ func BenchmarkImagePackageCatalogers(b *testing.B) {
}
func TestPkgCoverageImage(t *testing.T) {
sbom, _ := catalogFixtureImage(t, "image-pkg-coverage")
sbom, _ := catalogFixtureImage(t, "image-pkg-coverage", source.SquashedScope)
observedLanguages := internal.NewStringSet()
definedLanguages := internal.NewStringSet()
@@ -64,6 +65,7 @@ func TestPkgCoverageImage(t *testing.T) {
// for image scans we should not expect to see any of the following package types
definedLanguages.Remove(pkg.Go.String())
definedLanguages.Remove(pkg.Rust.String())
definedLanguages.Remove(pkg.Dart.String())
observedPkgs := internal.NewStringSet()
definedPkgs := internal.NewStringSet()
@@ -75,6 +77,7 @@ func TestPkgCoverageImage(t *testing.T) {
definedPkgs.Remove(string(pkg.KbPkg))
definedPkgs.Remove(string(pkg.GoModulePkg))
definedPkgs.Remove(string(pkg.RustPkg))
definedPkgs.Remove(string(pkg.DartPubPkg))
var cases []testCase
cases = append(cases, commonTestCases...)
+2 -1
View File
@@ -1,6 +1,7 @@
package integration
import (
"github.com/anchore/syft/syft/source"
"testing"
"github.com/stretchr/testify/assert"
@@ -9,7 +10,7 @@ import (
)
func TestDistroImage(t *testing.T) {
sbom, _ := catalogFixtureImage(t, "image-distro-id")
sbom, _ := catalogFixtureImage(t, "image-distro-id", source.SquashedScope)
expected := &linux.Release{
PrettyName: "BusyBox v1.31.1",
+12 -10
View File
@@ -2,12 +2,14 @@ package integration
import (
"bytes"
"regexp"
"testing"
"fmt"
"github.com/anchore/syft/internal/formats/cyclonedxjson"
"github.com/anchore/syft/internal/formats/cyclonedxxml"
"github.com/anchore/syft/internal/formats/syftjson"
"github.com/anchore/syft/syft/source"
"regexp"
"testing"
"github.com/anchore/syft/syft/sbom"
"github.com/stretchr/testify/require"
@@ -51,12 +53,12 @@ func TestEncodeDecodeEncodeCycleComparison(t *testing.T) {
},
},
}
for _, test := range tests {
t.Run(string(test.formatOption), func(t *testing.T) {
// use second image for relationships
for _, image := range []string{"image-pkg-coverage", "image-owning-package"} {
originalSBOM, _ := catalogFixtureImage(t, image)
for _, test := range tests {
// use second image for relationships
for _, image := range []string{"image-pkg-coverage", "image-owning-package"} {
t.Run(fmt.Sprintf("%s/%s", test.formatOption, image), func(t *testing.T) {
originalSBOM, _ := catalogFixtureImage(t, image, source.SquashedScope)
format := syft.FormatByID(test.formatOption)
require.NotNil(t, format)
@@ -87,7 +89,7 @@ func TestEncodeDecodeEncodeCycleComparison(t *testing.T) {
t.Errorf("diff: %s", dmp.DiffPrettyText(diffs))
}
}
}
})
})
}
}
}
+2 -2
View File
@@ -14,7 +14,7 @@ func TestNpmPackageLockDirectory(t *testing.T) {
foundPackages := internal.NewStringSet()
for actualPkg := range sbom.Artifacts.PackageCatalog.Enumerate(pkg.NpmPkg) {
for _, actualLocation := range actualPkg.Locations {
for _, actualLocation := range actualPkg.Locations.ToSlice() {
if strings.Contains(actualLocation.RealPath, "node_modules") {
t.Errorf("found packages from package-lock.json in node_modules: %s", actualLocation)
}
@@ -35,7 +35,7 @@ func TestYarnPackageLockDirectory(t *testing.T) {
foundPackages := internal.NewStringSet()
for actualPkg := range sbom.Artifacts.PackageCatalog.Enumerate(pkg.NpmPkg) {
for _, actualLocation := range actualPkg.Locations {
for _, actualLocation := range actualPkg.Locations.ToSlice() {
if strings.Contains(actualLocation.RealPath, "node_modules") {
t.Errorf("found packages from yarn.lock in node_modules: %s", actualLocation)
}
@@ -0,0 +1,84 @@
package integration
import (
"fmt"
"github.com/anchore/syft/syft/source"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"testing"
)
func TestPackageDeduplication(t *testing.T) {
tests := []struct {
scope source.Scope
packageCount int
instanceCount map[string]int
locationCount map[string]int
}{
{
scope: source.AllLayersScope,
packageCount: 172, // without deduplication this would be 618
instanceCount: map[string]int{
"basesystem": 1,
"wget": 1,
"curl": 2, // upgraded in the image
"vsftpd": 1,
"httpd": 1,
},
locationCount: map[string]int{
"basesystem-10.0-7.el7.centos": 4,
"curl-7.29.0-59.el7": 1, // from base image
"curl-7.29.0-59.el7_9.1": 3, // upgrade
"wget-1.14-18.el7_6.1": 3,
"vsftpd-3.0.2-29.el7_9": 2,
"httpd-2.4.6-97.el7.centos.5": 1,
},
},
{
scope: source.SquashedScope,
packageCount: 170,
instanceCount: map[string]int{
"basesystem": 1,
"wget": 1,
"curl": 1, // upgraded, but the most recent
"vsftpd": 1,
"httpd": 1,
},
locationCount: map[string]int{
"basesystem-10.0-7.el7.centos": 1,
"curl-7.29.0-59.el7_9.1": 1, // upgrade
"wget-1.14-18.el7_6.1": 1,
"vsftpd-3.0.2-29.el7_9": 1,
"httpd-2.4.6-97.el7.centos.5": 1,
},
},
}
for _, tt := range tests {
t.Run(string(tt.scope), func(t *testing.T) {
sbom, _ := catalogFixtureImage(t, "image-vertical-package-dups", tt.scope)
assert.Equal(t, tt.packageCount, sbom.Artifacts.PackageCatalog.PackageCount())
for name, expectedInstanceCount := range tt.instanceCount {
pkgs := sbom.Artifacts.PackageCatalog.PackagesByName(name)
// with multiple packages with the same name, something is wrong (or this is the wrong fixture)
require.Len(t, pkgs, expectedInstanceCount)
for _, p := range pkgs {
nameVersion := fmt.Sprintf("%s-%s", name, p.Version)
expectedLocationCount, ok := tt.locationCount[nameVersion]
if !ok {
t.Fatalf("missing name-version: %s", nameVersion)
}
// we should see merged locations (assumption, there was 1 location for each package)
assert.Len(t, p.Locations.ToSlice(), expectedLocationCount)
// all paths should match
assert.Len(t, p.Locations.CoordinateSet().Paths(), 1)
}
}
})
}
}
@@ -3,6 +3,7 @@ package integration
import (
"bytes"
"encoding/json"
"github.com/anchore/syft/syft/source"
"testing"
"github.com/anchore/syft/internal/formats/syftjson"
@@ -22,7 +23,7 @@ func TestPackageOwnershipRelationships(t *testing.T) {
for _, test := range tests {
t.Run(test.fixture, func(t *testing.T) {
sbom, _ := catalogFixtureImage(t, test.fixture)
sbom, _ := catalogFixtureImage(t, test.fixture, source.SquashedScope)
output := bytes.NewBufferString("")
err := syftjson.Format().Encode(output, sbom)
@@ -1,6 +1,7 @@
package integration
import (
"github.com/anchore/syft/syft/source"
"testing"
"github.com/anchore/syft/syft/pkg"
@@ -9,7 +10,7 @@ import (
func TestRegression212ApkBufferSize(t *testing.T) {
// This is a regression test for issue #212 (https://github.com/anchore/syft/issues/212) in which the apk db could
// not be processed due to a scanner buffer that was too small
sbom, _ := catalogFixtureImage(t, "image-large-apk-data")
sbom, _ := catalogFixtureImage(t, "image-large-apk-data", source.SquashedScope)
expectedPkgs := 58
actualPkgs := 0
@@ -1,6 +1,7 @@
package integration
import (
"github.com/anchore/syft/syft/source"
"strings"
"testing"
@@ -15,12 +16,12 @@ func TestRegressionGoArchDiscovery(t *testing.T) {
)
// This is a regression test to make sure the way we detect go binary packages
// stays consistent and reproducible as the tool chain evolves
sbom, _ := catalogFixtureImage(t, "image-go-bin-arch-coverage")
sbom, _ := catalogFixtureImage(t, "image-go-bin-arch-coverage", source.SquashedScope)
var actualELF, actualWIN, actualMACOS int
for p := range sbom.Artifacts.PackageCatalog.Enumerate(pkg.GoModulePkg) {
for _, l := range p.Locations {
for _, l := range p.Locations.ToSlice() {
switch {
case strings.Contains(l.RealPath, "elf"):
actualELF++
@@ -1,9 +1,10 @@
package integration
import (
"github.com/anchore/syft/syft/source"
"testing"
)
func TestRegressionJavaNoMainPackage(t *testing.T) { // Regression: https://github.com/anchore/syft/issues/252
catalogFixtureImage(t, "image-java-no-main-package")
catalogFixtureImage(t, "image-java-no-main-package", source.SquashedScope)
}
@@ -0,0 +1,49 @@
# Generated by pub
# See https://dart.dev/tools/pub/glossary#lockfile
packages:
ale:
dependency: transitive
description:
name: ale
url: "https://pub.hosted.org"
source: hosted
version: "3.3.0"
analyzer:
dependency: transitive
description:
name: analyzer
url: "https://pub.dartlang.org"
source: hosted
version: "0.40.7"
ansicolor:
dependency: transitive
description:
name: ansicolor
url: "https://pub.dartlang.org"
source: hosted
version: "1.1.1"
archive:
dependency: transitive
description:
name: archive
url: "https://pub.dartlang.org"
source: hosted
version: "2.0.13"
args:
dependency: transitive
description:
name: args
url: "https://pub.dartlang.org"
source: hosted
version: "1.6.0"
key_binder:
dependency: "direct main"
description:
path: "."
ref: HEAD
resolved-ref: "3f7b3a6350e73c7dcac45301c0e18fbd42af02f7"
url: "git@github.com:Workiva/key_binder.git"
source: git
version: "1.11.20"
sdks:
dart: ">=2.12.0 <3.0.0"
@@ -0,0 +1,6 @@
FROM centos:7.9.2009
# modifying the RPM DB multiple times will result in duplicate packages when using all-layers (if there was no de-dup logic)
# curl is tricky, it already exists in the image and is being upgraded
RUN yum install -y wget curl
RUN yum install -y vsftpd
RUN yum install -y httpd
+2 -2
View File
@@ -13,7 +13,7 @@ import (
"github.com/anchore/syft/syft/source"
)
func catalogFixtureImage(t *testing.T, fixtureImageName string) (sbom.SBOM, *source.Source) {
func catalogFixtureImage(t *testing.T, fixtureImageName string, scope source.Scope) (sbom.SBOM, *source.Source) {
imagetest.GetFixtureImage(t, "docker-archive", fixtureImageName)
tarPath := imagetest.GetFixtureImageTarPath(t, fixtureImageName)
userInput := "docker-archive:" + tarPath
@@ -25,7 +25,7 @@ func catalogFixtureImage(t *testing.T, fixtureImageName string) (sbom.SBOM, *sou
// TODO: this would be better with functional options (after/during API refactor)
c := cataloger.DefaultConfig()
c.Search.Scope = source.SquashedScope
c.Search.Scope = scope
pkgCatalog, relationships, actualDistro, err := syft.CatalogPackages(theSource, c)
if err != nil {
t.Fatalf("failed to catalog image: %+v", err)