Fix passing "--key" to the attest command. Additionally, pull in an update to
the clio CLI library to permit unit testing that flags and env vars are parsed
to the correct field on command options structs. This testing strategy was
needed here because testing attestation in an end to end test requires a
prohibitive amount of setup.
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Will Murphy <will.murphy@anchore.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
This fixes an issue where filenames containing a period that aren't a
group ID, such as some-jar.12.jar, would be mistakenly be reported as
having the name "12" by syft, instead of the name "some-jar.12".
It works by testing whether the parts of the filename split on "."
are all valid Java identifiers.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* with release signature of checksums file
* attach SBOMs to the release
* update acceptance tests
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore: stop re-exporting wfn.Attributes
Previously, Syft re-exported wfn.Attributes from the nvdtools package as
a member of the Package struct. However, Syft doesn't own this struct,
and so after Syft 1.0, might be forced to bump a semver major version
due to a breaking change in wfn.Attributes. Rather than incur this risk
going into 1.0, instead replace Syft's use of wfn.Attributes with Syft's
own cpe.CPE type. That type has some pass-through calls to
wfn.Attributes, but hides the dependency from the rest of the
application.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* chore: make cpe.CPE type a Stringer
Previously, the cpe.CPE type was an alias for wfn.Attributes from
nvdtools. Now that it is a type we control, make the String method take
the CPE as a receiver, rather than as a normal parameter, so that Syft's
cpe.CPE type implements Stringer.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
---------
Signed-off-by: Will Murphy <will.murphy@anchore.com>
The previous implementation would leak a goroutine if the caller of
AllLocations stopped iterating early. Now, accept a context so that the
caller can cancel the AllLocations iterator rather than leak the
goroutine.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
To reduce toil in this repo, enable dependabot PRs to be automatically
approved, but not merged. They are not automatically merged because if
the default GitHub token is used to automatically merge a PR, the
resulting commit will not trigger workflows on main. Rather than
generate a more potent token, just automatically review them, which
reduces toil by eliminating several clicks and page loads for
maintainers who are trying to merge dependabot PRs.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* ERLang parser support for empty lists
* ERLang add support for single quote strings
* ERLang parser support for comments
---------
Signed-off-by: Laurent Goderre <laurent.goderre@docker.com>
Fixes a bug where previously the haskell cataloger would panic
when parsing a stack.yaml.lock file that had an entry with an empty
hackage string.
Signed-off-by: houdini91 <mdstrauss91@gmail.com>
Signed-off-by: Will Murphy <will.murphy@anchore.com>
Co-authored-by: houdini91 <mdstrauss91@gmail.com>
Previously, pointers to primitive types in config summarization could be
printed literally (like "0x123aefef"). Pull in fangs to get the fix for
this.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
According to maven central, the package called "xalan" should just have
the group ID xalan, but currently syft isn't able to find that.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
Previously, building the index of the filesystem when source was file
would fail if part of the path syft was passed to the file included a
symlinked directory, resulting in cataloging misses.
---------
Signed-off-by: Will Murphy <will.murphy@anchore.com>
Allow existing matcher to match host identifiers longer than 12
characters. The binaries distributed by redis have the version before
payload, so add a matcher for that. Add test fixtures covering these
scenarios.
Signed-off-by: David Dooling <david.dooling@docker.com>
* fix: update parsing logic to remove empty space
* tests: update with test cases provided by community
---------
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
* Add ruby.NewGemSpecCataloger to DirectoryCatalogers.
Signed-off-by: Evan <chaol@vmware.com>
* fixed tests
Signed-off-by: Evan <chaol@vmware.com>
* Addressed review comment
Signed-off-by: Evan <chaol@vmware.com>
* Remove NewInstalledGemSpecCataloger from default dir catalogers
Because the files that the installed gemspec cataloger work off of are a
subset of the files that the more general gemspec cataloger will work
off of, we shouldn't have both of them on by default, since this could
result in finding the same package twice.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
---------
Signed-off-by: Evan <chaol@vmware.com>
Signed-off-by: Will Murphy <will.murphy@anchore.com>
Co-authored-by: Will Murphy <will.murphy@anchore.com>
adds a unique synthetic package to the SBOM output that represents the go compiler when it is detected as a part of a package discovered by the go binary cataloger.
When using an SBOM generated by syft - downstream vulnerability scanners now have the opportunity to detect/report on the PURL/CPEs attached to the new stdlib package.
---------
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
Previously, which PURL was generated depended on the order of key iteration
in maps. Also update an integ test that was apparently only passing because
of the previous issue.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
Now that the test fixture pins to a particular digest, there's no need
for platform specific architecture switches in this test.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
Many of these images have a slightly different sets of packages when the
arm64 variant is pulled, so that leaving this digest unpinned causes the
tests to fail on arm64 hosts. Pin the FROM lines to force stable
platform values regardless of host architecture.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
Previously, there were some data races in syft. Right now, none are
detected, so check for data races on the overall command, and on unit
tests. (Checking for races on integration tests triples the time needed
for those tests, from ~1 minute to ~3 minutes on my workstation, so that
was not done at this time.)
Signed-off-by: Will Murphy <will.murphy@anchore.com>
Otherwise, small renames like 'hudson-war-2.2.1.war' to 'hudson.war', would cause
syft to incorrectly catolog the archive.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
If crypto settings or arch cannot be determined, still attempt to catalog packages from
the build info, rather than panicking.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
A recent update to gopom changed many fields to be omitted when empty,
resulting in a number of nil pointers inside the nested structure of the
pom that previously didn't exist. Defend against these in the search for
the property value.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
This reflect code occasionally throws an obscure panic, but not enough
information is logged before the panic to know why it panicked. Log
enough to tell what property and package are being analyzed when the
panic occurs.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
Separate the logic for CPE and PURL generation.
PURL generation needs a single answer for groupID based on a priority of discovering the field.
CPE generation still uses multiple potential groupID to populate the candidate cpe.
Improve GroupID detection.
Currently syft does not use any hierarchy for GroupID detection and treats all sources as equal.
It treats fields from the manifest file with priority. This change adds a hierarchy to the fields and returns a single answer based on that hierarchy.
---------
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Co-authored-by: Keith Zantow <kzantow@gmail.com>
Fixes#931
PR #1948 introduces a new implicit exclusion for binary packages that overlap by file ownership and have certain characteristics:
1) the relationship between packages is OwnershipByFileOverlap
2) the parent package is an "os" package - see changelog for included catalogers
3) the child is a synthetic package generated by the binary cataloger - see changelog for included catalogers
4) the package names are identical
---------
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
* Add opkg info directory and status file to deb cataloger
opkg uses the same or nearly the same metadata and structure as Debian:
**/lib/opkg/status lists status information for all packages
**/lib/opkg/info/opkg.conffiles is a list of configuration files
**/lib/opkg/info/*.list contains files and directories installed by the package
**/lib/opkg/info/*.preinst are scripts to run before installation
**/lib/opkg/info/*.postinst are scripts to run after installation
**/lib/opkg/info/*.postrm are scripts to run after package removal
**/lib/opkg/info/*.control provides package metadata
Signed-off-by: Nicholas R. Smith <nicholas_smith@selinc.com>
---------
Signed-off-by: Nicholas R. Smith <nicholas_smith@selinc.com>
Co-authored-by: Nicholas R. Smith <nicholas_smith@selinc.com>
* fix: default image source name to user input
Signed-off-by: Keith Zantow <kzantow@gmail.com>
* chore: add test
Signed-off-by: Keith Zantow <kzantow@gmail.com>
---------
Signed-off-by: Keith Zantow <kzantow@gmail.com>
* Add support for parsing .NET assemblies
Signed-off-by: Dan Luhring <dluhring@chainguard.dev>
Former-commit-id: 69c33fe4d77357d843c11590f3b07825bc6249ac
* Add dll and exe files
Signed-off-by: Dan Luhring <dluhring@chainguard.dev>
Former-commit-id: b9d204efa6d2ef385b5fbb7a59a3474ecabea641
* Add PE cataloger to directory catalogers
Signed-off-by: Dan Luhring <dluhring@chainguard.dev>
Former-commit-id: 9711c00d9da92e2887e0c1f92edd740ea5345849
* Don't set language to dotnet for PEs
Signed-off-by: Dan Luhring <dluhring@chainguard.dev>
Former-commit-id: 368313fddac9160d8a06a01ebe8c5ac7990232f5
* Fix spelling of cataloger in constructor
Signed-off-by: Dan Luhring <dluhring@chainguard.dev>
Former-commit-id: e42fd77b2f8b6d42e076a84f6cce386861260941
* Adjust which cases in PE parsing return errors
Signed-off-by: Dan Luhring <dluhring@chainguard.dev>
Former-commit-id: 95b25f8fc3a7d4e18fe30e489b09851f316795ff
* remove build binary from branch
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Former-commit-id: fa54c0d0aef0998d5520e9f44cae51f5f9cd38a2
* Fix failing CLI tests
Signed-off-by: Dan Luhring <dluhring@chainguard.dev>
---------
Signed-off-by: Dan Luhring <dluhring@chainguard.dev>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* docs: update docs with new docker specific instructions
---------
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
Former-commit-id: c67c76e84df84e3e24aa307637d884ca8b7e3eea
* fix: or-later suffix has been updated to consider deprecated +
If a given license has the suffix "or-later" it previously could have
been considered or represented with a "+". Example "GFDL-1.0-or-later"
could have been represented as "GFDL-1.0+". This PR allows the license
list generation to consider "or-later" as == to "+" when generating
permutations for upgrading deprecated licenses.
---------
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
Add some explanation around why there are different default sets of catalogers for image scans versus directory scans. Hopefully clarify questions related to #1776.
Signed-off-by: Timothy Gerla <tim@gerla.net>
Track a chronicle config file that causes chronicle to bump minor
version instead of major version in response to the "breaking-change"
label for pre-1.0 releases.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
* [wip] try to reflect metadata types... probably wont work
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* refactor to add unit test to ensure there is coverage in the schema
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* [wip] generate metadata container
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add generation of metadata container struct for JSON schema generation
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix linting
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* update linter script to account for code generation
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
---------
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
to keep things consistent across the schema we want Locations and URLs to be plural fields now that they are fields on the License struct
---------
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
this PR makes the following changes to update the underlying license model to have more expressive capabilities
it also provides some guarantee's surrounding the license values themselves
- Licenses are updated from string -> pkg.LicenseSet which contain pkg.License with the following fields:
- original `Value` read by syft
- If it's possible to construct licenses will always have a valid SPDX expression for downstream consumption
- the above is run against a generated list of SPDX license ID to try and find the correct ID
- SPDX concluded vs declared is added to the new struct
- URL source for license is added to the new struct
- Location source is added to the new struct to show where the expression was pulled from
If syft is asked to parse an SBOM that was written by a newer version of
syft, emit a warning, since the current version of syft doesn't know about
fields that may be added in the future.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
Add a cataloger that detects installed R packages by looking for DESCRIPTION
files. The base R package is now picked up in coverageImage tests in
test/cli/packages_cmd_test.go, so increment expected package counts for the
tests that use that image.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
Rather than log a warning for every instance of an unknown relationship type,
or similar error, log a count of how many times each of these errors is
raised.
---------
Signed-off-by: Will Murphy <will.murphy@anchore.com>
The tests in test/cli/packages_cmd_test.go are hard to debug when different
packages are found in different environments. For example, CI runs and M1 macs
have been observed to have different package counts. Therefore, if the test is
about to fail, log a sorted list of the packages that were found, so that it is
easy to compare failures of these tests.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
- Create new metadata struct and type for python requirements.
- Update parsing of python requirements to use python requirements metadata.
- Remove extras and url from line. Add them to metadata instead.
- Add unit test to test that extras are removed from package name.
- Update test to look at requirements metadata.
- Will need updated in future to support more than just == for the version constraint.
- Update JSON schema data
Closesanchore/grype#1246Closesanchore/grype#1251
Signed-off-by: Shane Dell <shanedell100@gmail.com>
* Create consul binary classifier
Closes#1590
Signed-off-by: Shane Dell <shanedell100@gmail.com>
* Create test for consul binary classifier
Signed-off-by: Shane Dell <shanedell100@gmail.com>
* Update version for consul. Add note that about consul version matcher is brittle
Signed-off-by: Shane Dell <shanedell100@gmail.com>
---------
Signed-off-by: Shane Dell <shanedell100@gmail.com>
- Update README.md to show that nar is now supported.
- Created a java-archives/example.nar so that the tests wouldn't break.
- Add nar glob and as an option for pkgType.
Closes#1701
Signed-off-by: Shane Dell <shanedell100@gmail.com>
Update the license_list.go to have more permissible inputs for greater SPDXID matching.
EX:
GPL3 gpl3 gpl-3 and GPL-3 can all map to GPL-3.0-only
By moving all strings to lower and removing the "-" we're able to return valid SPDX license ID for a greater diversity of input strings.
---------
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
* add argocd, helm, kustomize and kubectl binary classifiers
* update golang PURL
* address PR faceback about binary/test-fixtures/Makefile
* remove the /v[n] suffix from the PURL in both argocd and helm
---------
Signed-off-by: y12studio <y12studio@gmail.com>
Adding APK OriginPackage CPE candidates to the child package
results in false positives in grype because it can't associate
CPE-based findings to the corresponding OriginPackage APK fixes.
This reverts changing the `upstream` in the PURL for APK packages
as the logic in Grype that uses it expects it to be an APK package
name. This also allows refactoring to unexport and move the APK
CPE candidate generation logic closer to where CPE generation occurs
Signed-off-by: Weston Steimel <weston.steimel@anchore.com>
This fixes some instances where the improved APK CPE generation
logic caused regressions for older alpine package APK metadata.
It now generates multiple "upstream" candidates with both name
and package type which reduces the amount of duplicated code in
the apk cpe gen logic. This also improves the handling of stream
version packages, so now we can correctly identify packages such
as ruby3.2-rexml as the rexml ruby gem.
Signed-off-by: Weston Steimel <weston.steimel@anchore.com>
The apk purl spec allows for vendor-specific namespace. I noticed
in the embedded SBOMs from wolfi that the purls are of the form
`pkg:apk/wolfi/curl@7.83.0-r0?arch=x86`, but the current logic in
syft actually prevents purl generation entirely if the distro isn't
alpine, so this corrects that behaviour.
Signed-off-by: Weston Steimel <weston.steimel@anchore.com>
* refactor: move apk upstream logic to apk metadata
Export the logic for parsing upstream APK package names
so it can be accessed from apk metadata objects directly.
This also tightens the upstream regex pattern as several
edge cases were being missed.
Signed-off-by: Weston Steimel <weston.steimel@anchore.com>
* fix: ensure correct handling for apk packages beginning with digits
Signed-off-by: Weston Steimel <weston.steimel@anchore.com>
* fix: upstream generation for ruby
Signed-off-by: Weston Steimel <weston.steimel@anchore.com>
---------
Signed-off-by: Weston Steimel <weston.steimel@anchore.com>
* replace raw globs with index equivelent operations
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add cataloger test for alpm cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix import sorting for binary cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix linting for mock resolver
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* separate portage cataloger parser impl from cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* enhance cataloger pkgtest utils to account for resolver responses
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for alpm cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for apkdb cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for dpkg cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for cpp cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for dart cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for dotnet cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for elixir cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for erlang cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for golang cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for haskell cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for java cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for javascript cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for php cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for portage cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for python cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for rpm cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for rust cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for sbom cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for swift cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* allow generic catloger to run all mimetype searches at once
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* remove stutter from php and javascript cataloger constructors
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* bump stereoscope
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add tests for generic.Search
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add exceptions for java archive git ignore entries
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* enhance basename and extension resolver methods to be variadic
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* dont allow * prefix on extension searches
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add glob-based cataloger tests for ruby cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* remove unnecessary string casting
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* incorporate surfacing of leaf link resolitions from stereoscope results
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* [wip] switch to stereoscope file metadata
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* [wip + failing] revert to old globs but keep new resolvers
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* index files, links, and dirs within the directory resolver
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix several resolver bugs and inconsistencies
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* move format testutils to internal package
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* update syft json to account for file type string normalization
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* split up directory resolver from indexing
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* update docs to include details about searching
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* [wip] bump stereoscope to development version
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix linting
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* adjust symlinks fixture to be fixed to digest
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix all-locations resolver tests
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix test fixture reference
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* rename file.Type
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* bump stereoscope
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix PR comment to exclude extra *
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* bump to dev version of stereoscope
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* bump to final version of stereoscope
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* move observing resolver to pkgtest
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
---------
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add release trigger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* deduplicate version and changelog calls + add gh checks
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add more chronicle verbosity, but not when triggering releases
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* bump chronicle version to get --version-file feature
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* update bootstrap tool workflow to include glow
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add version prefix check on tags in release quality gate
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
---------
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
When base is set, it should appear identically to when we scan the root
filesystem - and as a result, the path should begin with the path
separator.
E.g. when scanning the root `./target/` with the same base,
`target/bin/busybox` should appear in the output as `/bin/busybox`, not
as previously as `bin/busybox`.
Signed-off-by: Justin Chadwell <me@jedevc.com>
* fix: update config struct to not decode password/key
* test: update tests to confirm no secrets in output
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
---------
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
* source: avoid second-step of symlink resolution in directory resolver
We can use the already existing file tree to peform symlink resolution
for FilesByPath, instead of traversing the symlinks again.
This moves all of the symlink logic into the indexing code, and then we
can rely on syft's resolution algorithm over the index in this part of
the codebase.
Signed-off-by: Justin Chadwell <me@jedevc.com>
* source: add base parameter to directory resolver
The new base parameter is an optional parameter for the directory
resolver that resolves all symlinks relative to this root. There are two
intended use cases:
- base = "/". The previous behavior, symlinks are resolved relative to
the root filesystem.
- base = path. Symlinks are resolved relative to the target filesystem,
allowing correct behavior when scanning unpacked container filesystems
on disk.
Signed-off-by: Justin Chadwell <me@jedevc.com>
* source: add tests for new base parameter
Signed-off-by: Justin Chadwell <me@jedevc.com>
---------
Signed-off-by: Justin Chadwell <me@jedevc.com>
* fix AssertEncoderAgainstGoldenSnapshot calls to conditionally update
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* chore: redact schema versions from syftjson comparison
Signed-off-by: Keith Zantow <kzantow@gmail.com>
* chore: does not need a multiline expression
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Co-authored-by: Keith Zantow <kzantow@gmail.com>
* feat: prefer known CPE vendors over other candidates
All ASF projects will be under the `apache` vendor in CPE, and
indeed this is already one of the candidates, but the logic
for selecting the 'most specific' CPE string would select for
example `apache_software_foundation` or `commons-text`.
This is not necessarily 'wrong' in the CPE candidate selection
logic: there is no way to reliably determine the right candidate.
I think it makes sense to use specific data around the vendor
candidate generation, somewhat similar to
'defaultCandidateAdditions'.
Unfortunately there are still a few CVE's for old (pre-5.x,
long unsupported) tomcat versions that are actually tagged with
`apache_software_foundation`, but I'm not sure those are worth
spending time on.
Signed-off-by: Arnout Engelen <arnout@bzzt.net>
* chore: swap out array of vendors for set data structure
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Arnout Engelen <arnout@bzzt.net>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Co-authored-by: Alex Goodman <alex.goodman@anchore.com>
The original fix b125ea83ba didn't catch
all the excessive warnings, it seems like getArches can also be called
on binaries that aren't neccessarily go binaries, so the messages from
this should also be Trace instead of Warn.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This PR adds support for npm lockfile version 3, which drops the
"dependencies" key and uses "packages" instead. I've refactored the
lockfile parser to make the distinction between the versions explicit
rather than the implicit behaviour before. It _might_ be worth splitting
into separate files at some point, but the logic is so minimal that I
haven't done it.
Fixes#1203
Signed-off-by: Rob Cresswell <robcresswell@users.noreply.github.com>
* add second generation of generic cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* upgrade aplm cataloger to use generic.Cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* remove pacakge found-by attribute from the definition of a package ID
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Previously, extracting relationships between packages and files was not
completing correctly, as SPDXRef- ElementIDs were being compared to raw
IDs, and so never matched. This patch ensures that we always compare
ElementIDs, to ensure that the hasFiles field is correctly populated.
Signed-off-by: Justin Chadwell <me@jedevc.com>
add debug distroless image to published release
Debian was chosen based on the fact that it is the smallest available distroless image
The new tag is `anchore/syft:debug`
Closes#833
* add main module field to go bin metadata
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* udpate json ouput schema to 3.2.4
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* clean up fixture
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* Add filters to package cataloger
This PR adds filters so a package without name or version doesn't go in
the list of all discovered packages.
Integration and cli tests were added to validate the feature.
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* add nolint:funlen to cataloger/catalog.go
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* don't require package version
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* add package filtering to generic and python cataloger
also removes cli tests in favor of integration and unit tests
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* drop nolint:funlen
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* check for no-removal operation
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* remove unused fixtures
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* rename no-version file to hide semantic version
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* drop integration tests and add pkg func for validation
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* python cataloger use global pkg validation
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* check for valid packages on deb/go/rpm catalogers
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* update rpm cataloger after rebase
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* nit with pointers
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* simpler use of package validation
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* remmove double pkg validations
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* rename func param to artifactsToExclude
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* add test for relationships and bug fix
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* feedback changes
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* Support RPM distros with newer db formats
Recent RPM distros (Fedora 33+, CBL-Mariner 2.0+, amazonlinux 2022+)
use an sqlite package database in /var/lib/rpm/rpmdb.sqlite, or
"ndb" format (SUSE).
Remove anchore's fork in favour of the upstream,
https://github.com/knqyf263/go-rpmdb, to gain support for
these formats.
Signed-off-by: Tom Fay <tomfay@microsoft.com>
* add exception for modernc.org repos
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* shorten rpmdb helper function
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Co-authored-by: Alex Goodman <alex.goodman@anchore.com>
* Use SBOM descriptor version
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* Update tests
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* CycloneDX extract tools metadata in decoding stage
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* add descriptor to spdx tag-value test
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* remove comment
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* golang module CPE with full path
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* add note on longer Golang CPEs
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* bump stereoscope version to include source path fix
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* go mod tidy
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* add convert command
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* mvp
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
* fix hanging bug
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
* validate SBOM formats for conversion
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
* move convert cmd to new structure
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
* remove bin
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
* drop event loop from convert cmd
extract SBOM type from document namespace
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
* validate SPDX in tests
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
* documenting convert cmd
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
* support output format=file.json notation
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
* test convertible formats
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
* fix typo
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
* clean up
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
* more clean up and docs
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
* feedback changes
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
* nit
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* feedback changes
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* re-use more code
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* undo encode-decode cycle test
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* remove unnecessary test constraint
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* fix readme
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* try verbose
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* cleaner README and no table conversion
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* simpler conversion
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* feedback changes and cleanup
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* nit space fix
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* use defer
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* feedback changes
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
Co-authored-by: Keith Zantow <kzantow@gmail.com>
* Fix "bad output format" for `github-json` output
Signed-off-by: Steven Maude <git@stevenmaude.co.uk>
* Update formats in README
Signed-off-by: Steven Maude <git@stevenmaude.co.uk>
* Run `make lint-fix`
Signed-off-by: Steven Maude <git@stevenmaude.co.uk>
* read Go main module version as is - (devel)
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
* fix package test with default (devel) main module
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
The Official CPE dictionary currently contains entries for springframework with three different vendors: springsource, vmware, and pivotal_software. This appears to be because ownership has changed over time.
Signed-off-by: Weston Steimel <weston.steimel@anchore.com>
* Implement fmt.Stringer with format.ID
Signed-off-by: Dan Luhring <dan+github@luhrings.com>
* Add failing test for formats processing empty SBOMs
Signed-off-by: Dan Luhring <dan+github@luhrings.com>
* Account for nil SPDX document during Syft model conversion
Signed-off-by: Dan Luhring <dan+github@luhrings.com>
* Less verbose logging in Golang Cataloger
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* debug for known gray errors
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* only show warnings when a binary is not a go executable
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Co-authored-by: Alex Goodman <alex.goodman@anchore.com>
Since syft now depends on debug/buildinfo go 1.18 is required to build
syft and as such go.mod needs updating
Signed-off-by: 06kellyjac <jack@control-plane.io>
* initial working version
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* added build settings to pkg metadata
wip - unit tests
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* handle mach-O FatFiles
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* add support to mod replace
fixed golang catalger tests
trying GH Actions with go 1.18rc1
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* log error
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* use go-macholibre for extraction
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* cleaner tests
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* add version to main module
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* check macho file with macholibre
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* run golangci in its own workflow
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* wip - golangci workflow
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* fix golangci wf yml
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* fix golangci wf yml
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* wip - golangci wf
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* wip - golangci wf
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* get arch from bin file headers
upgrade macholibre
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* go mod tidy
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* test new stereoscope lazy reader interface
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* go mod tidy
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* remove devel version from golang cataloger
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* go mod tidy
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* switch github workflows to go1.18 stable
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* add union reader interface in golang cataloger
update stereoscope
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* go mod tidy
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* simpler golangci validation
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* fix makefile
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* get archs refactor
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* nolint for golang version
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* fix go bin tests
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* feedback changes
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* golangci nolint needs a \n before package
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* cleanup
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* move golangci-lint to its own jobs again
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* fix ci yaml
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* add support for xcoff files
add arch assets to test bin file types
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* clean up golangci-lint config
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* nolint for xcoff
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* explain nolints
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* remove unused xcoff testdata assets
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* make go bin test-fixtures in docker
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* fix make clean with -f
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* update json output schema
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* update schema version in test fixture
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* feedback changes
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* explain possible empty main module
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
# note: set the partition list for this certificate's private key to include "apple-tool:" and "apple:" allows the codesign command to access this keychain item without an interactive user prompt.
commentary "...cert has already been imported onto the new keychain"
fi
commentary "make certain there are identities that can be used for code signing"
security find-identity -p codesigning "$KEYCHAIN_PATH"| grep -C 30"$identity"|| exit_with_error "could not find identity that can be used with codesign"
title "add the new keychain to the search path for codesign"
add_keychain "$KEYCHAIN_NAME"
commentary "verify the new keychain can be found by the security sub-system"
security list-keychains | grep "$KEYCHAIN_NAME"|| exit_with_error "could not find new keychain"
printf"WARNING! It looks like this isn't the CI environment. This script modifies the macOS Keychain setup in ways you probably wouldn't want for your own machine. It also requires an Apple Developer ID Certificate that you shouldn't have outside of the CI environment.\n\nExiting early to make sure nothing bad happens.\n"
log_crit "uname_os_check '$(uname -s)' got converted to '$os' which is not a GOOS value. Please file bug at https://github.com/client9/shlib"
return1
}
uname_arch_check(){
arch=$(uname_arch)
case"$arch" in
386)return0;;
amd64)return0;;
arm64)return0;;
armv5)return0;;
armv6)return0;;
armv7)return0;;
ppc64)return0;;
ppc64le)return0;;
mips)return0;;
mipsle)return0;;
mips64)return0;;
mips64le)return0;;
s390x)return0;;
amd64p32)return0;;
esac
log_crit "uname_arch_check '$(uname -m)' got converted to '$arch' which is not a GOARCH value. Please file bug report at https://github.com/client9/shlib"
return1
}
untar(){
tarball=$1
case"${tarball}" in
*.tar.gz | *.tgz) tar --no-same-owner -xzf "${tarball}";;
*.tar) tar --no-same-owner -xf "${tarball}";;
*.zip) unzip "${tarball}";;
*)
log_err "untar unknown archive format for ${tarball}"
If you are looking to contribute to this project and want to open a GitHub pull request ("PR"), there are a few guidelines of what we are looking for in patches. Make sure you go through this document and ensure that your code proposal is aligned.
## Setting up your environment
Before you can contribute to Syft, you need to configure your development environment.
### Debian setup
You will need to install Go. The version on https://go.dev works best, using the system golang doesn't always work the way you might expect.
Refer to the go.mod file in the root of this repo for the recommended version of Go to install.
You will also need Docker. There's no reason the system packages shouldn't work, but we used the official Docker package. You can find instructions for installing Docker in Debian [here](https://docs.docker.com/engine/install/debian/).
You also need to install some Debian packages
```sh
sudo apt-get install build-essential zip bc libxml2-utils git
```
## Configuring Git
You will need to configure your git client with your name and email address. This is easily done from the command line.
This username and email address will matter later in this guide.
## Fork the repo
You should fork the Syft repo using the "Fork" button at the top right of the Syft GitHub [site](https://github.com/anchore/syft/). You will be doing your development in your fork, then submit a pull request to Syft. There are many resources how to use GitHub effectively, we will not cover those here.
## Adding a feature or fix
If you look at the Syft [Issue](https://github.com/anchore/syft/issues) there are plenty of bugs and feature requests. Maybe look at the [good first issue](https://github.com/anchore/syft/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) list if you're not sure where to start.
## Commit guidelines
In the Syft project we like commits and pull requests (PR) to be easy to understand and review. Open source thrives best when everything happening is over documented and small enough to be understood.
### Granular commits
Please try to make every commit as simple as possible, but no simpler. The idea is that each commit should be a logical unit of code. Try not to commit too many tiny changes, for example every line changed in a file as a separate commit. And also try not to make a commit enormous, for example committing all your work at the end of the day.
Rather than try to follow a strict guide on what is or is not best, we try to be flexible and simple in this space. Do what makes the most sense for the changes you are trying to include.
### Commit title and description
Remember that the message you leave for a commit is for the reviewer in the present, and for someone (maybe you) changing something in the future. Please make sure the title and description used is easy to understand and explains what was done. Jokes and clever comments generally don't age well in commit messages. Just the facts please.
## Sign off your work
The `sign-off` is an added line at the end of the explanation for the commit, certifying that you wrote it or otherwise have the right to submit it as an open-source patch. By submitting a contribution, you agree to be bound by the terms of the DCO Version 1.1 and Apache License Version 2.0.
@@ -44,13 +95,6 @@ When committing your change, you can add the required line manually so that it l
Signed-off-by: John Doe <john.doe@example.com>
```
Alternatively, configure your Git client with your name and email to use the `-s` flag when creating a commit:
[//]: # (TODO: Commit guidelines, commit title, extra body description)
[//]: # (TODO: PR title and description)
## Sign your commits
To ensure the authenticity and integrity of code contributions, **we require that all commits are signed**. Signing commits proves that your commits were truly created by you, as the holder of a private key.
Configuring git to sign your commits is a straightforward process. To get set up, see [GitHub's documentation on signing commits](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits).
## Test your changes
This project has a `Makefile` which includes many helpers running both unit and integration tests. Although PRs will have automatic checks for these, it is useful to run them locally, ensuring they pass before submitting changes. Ensure you've bootstrapped once before running tests:
This project has a `Makefile` which includes many helpers running both unit and integration tests. You can run `make help` to see all the options. Although PRs will have automatic checks for these, it is useful to run them locally, ensuring they pass before submitting changes. Ensure you've bootstrapped once before running tests:
```text
$ make bootstrap
@@ -103,6 +129,26 @@ $ make unit
$ make integration
```
You can also run `make all` to run a more extensive test suite, but there is additional configuration that will be needed for those tests to run correctly. We will not cover the extra steps here.
## Pull Request
If you made it this far and all the tests are passing, it's time to submit a Pull Request (PR) for Syft. Submitting a PR is always a scary moment as what happens next can be an unknown. The Syft project strives to be easy to work with, we appreciate all contributions. Nobody is going to yell at you or try to make you feel bad. We love contributions and know how scary that first PR can be.
### PR Title and Description
Just like the commit title and description mentioned above, the PR title and description is very important for letting others know what's happening. Please include any details you think a reviewer will need to more properly review your PR.
A PR that is very large or poorly described has a higher likelihood of being pushed to the end of the list. Reviewers like PRs they can understand and quickly review.
### What to expect next
Please be patient with the project. We try to review PRs in a timely manner, but this is highly dependent on all the other tasks we have going on. It's OK to ask for a status update every week or two, it's not OK to ask for a status update every day.
It's very likely the reviewer will have questions and suggestions for changes to your PR. If your changes don't match the current style and flow of the other code, expect a request to change what you've done.
## Document your changes
When proposed changes are modifying user-facing functionality or output, it is expected the PR will include updates to the documentation as well.
And lastly, when proposed changes are modifying user-facing functionality or output, it is expected the PR will include updates to the documentation as well. Syft is not a project that is heavy on documentation. This will mostly be updating the README and help for the tool.
If nobody knows new features exist, they can't use them!
In order to test and develop in this repo you will need the following dependencies installed:
- Golang
- docker
- make
- Python (>= 3.9)
After cloning do the following:
### Docker settings for getting started
Make sure you've updated your docker settings so the default docker socket path is available.
Go to:
docker -> settings -> advanced
Make sure:
```
Allow the default Docker socket to be used
```
is checked.
Also double check that the docker context being used is the default context. If it is not, run:
`docker context use default`
After cloning the following step can help you get setup:
1. run `make bootstrap` to download go mod dependencies, create the `/.tmp` dir, and download helper utilities.
2. run `make` to run linting, tests, and other verifications to make certain everything is working alright.
2. run `make` to view the selection of developer commands in the Makefile
3. run `make build` to build the release snapshot binaries and packages
4. for an even quicker start you can run `go run cmd/syft/main.go` to print the syft help.
- this command `go run cmd/syft/main.go alpine:latest` will compile and run syft against `alpine:latest`
5. view the README or syft help output for more output options
Checkout `make help` to see what other actions you can take.
The main make tasks for common static analysis and testing are `lint`, `format`, `lint-fix`, `unit`, `integration`, and `cli`.
The main make tasks for common static analysis and testing are `lint`, `lint-fix`, `unit`, `integration`, and `cli`.
See `make help` for all the current make tasks.
## Levels of testing
### Internal Artifactory Settings
**Not always applicable**
Some companies have Artifactory setup internally as a solution for sourcing secure dependencies.
If you're seeing an issue where the unit tests won't run because of the below error then this section might be relevant for your use case.
```
[ERROR] [ERROR] Some problems were encountered while processing the POMs
```
If you're dealing with an issue where the unit tests will not pull/build certain java fixtures check some of these settings:
- a `settings.xml` file should be available to help you communicate with your internal artifactory deployment
- this can be moved to `syft/pkg/cataloger/java/test-fixtures/java-builds/example-jenkins-plugin/` to help build the unit test-fixtures
- you'll also want to modify the `build-example-jenkins-plugin.sh` to use `settings.xml`
For more information on this setup and troubleshooting see [issue 1895](https://github.com/anchore/syft/issues/1895#issuecomment-1610085319)
## Architecture
At a high level, this is the package structure of syft:
```
./cmd/syft/
│ ├── cli/
│ │ ├── cli.go // where all commands are wired up
│ │ ├── commands/ // all command implementations
│ │ ├── options/ // all command flags and configuration options
│ │ └── ui/ // all handlers for events that are shown on the UI
│ └── main.go // entrypoint for the application
└── syft/ // the "core" syft library
├── format/ // contains code to encode or decode to and from SBOM formats
├── pkg/ // contains code to catalog packages from a source
├── sbom/ // contains the definition of an SBOM
└── source/ // contains code to create a source object for some input type (e.g. container image, directory, etc)
```
Syft's core library is implemented in the `syft` package and subpackages, where the major packages are:
- the `syft/source` package produces a `source.Source` object that can be used to catalog a directory, container, and other source types.
- the `syft` package contains a single function that can take a `source.Source` object and catalog it, producing an `sbom.SBOM` object
- the `syft/format` package contains the ability to encode and decode SBOMs to and from different SBOM formats (such as SPDX and CycloneDX)
The `cmd` pacakge at the highest level execution flow wires up [`spf13/cobra`](https://github.com/spf13/cobra) commands for execution in the main application:
```mermaid
sequenceDiagram
participant main as cmd/syft/main
participant cli as cli.New()
participant root as root.Execute()
participant cmd as <command>.Execute()
main->>+cli:
Note right of cli: wire ALL CLI commands
Note right of cli: add flags for ALL commands
cli-->>-main: root command
main->>+root:
root->>+cmd:
cmd-->>-root: (error)
root-->>-main: (error)
Note right of cmd: Execute SINGLE command from USER
```
The `packages` command uses the core library to generate an SBOM for the given user input:
```mermaid
sequenceDiagram
participant source as source.New(ubuntu:latest)
participant sbom as sbom.SBOM
participant catalog as syft.CatalogPackages(src)
participant encoder as syft.Encode(sbom, format)
Note right of source: use "ubuntu:latest" as SBOM input
source-->>+sbom: add source to SBOM struct
source-->>+catalog: pass src to generate catalog
catalog-->-sbom: add cataloging results onto SBOM
sbom-->>encoder: pass SBOM and format desiered to syft encoder
encoder-->>source: return bytes that are the SBOM of the original input
Note right of catalog: cataloger configuration is done based on src
```
Additionally, here is a [gist of using syft as a library](https://gist.github.com/spiffcs/3027638b7ba904d07e482a712bc00d3d) to generate a SBOM for a docker image.
### `pkg.Package` object
The `pkg.Package` object is a core data structure that represents a software package. Fields like `name` and `version` probably don't need
a detailed explanation, but some of the other fields are worth a quick overview:
-`FoundBy`: the name of the cataloger that discovered this package (e.g. `python-pip-cataloger`).
-`Locations`: these are the set of paths and layer ids that were parsed to discover this package (e.g. `python-pip-cataloger`).
-`Language`: the language of the package (e.g. `python`).
-`Type`: this is a high-level categorization of the ecosystem the package resides in. For instance, even if the package is a egg, wheel, or requirements.txt reference, it is still logically a "python" package. Not all package types align with a language (e.g. `rpm`) but it is common.
-`Metadata`: specialized data for specific location(s) parsed. We should try and raise up as much raw information that seems useful. As a rule of thumb the object here should be as flat as possible and use the raw names and values from the underlying source material parsed.
When `pkg.Package` is serialized an additional `MetadataType` is shown. This is a label that helps consumers understand the datashape of the `Metadata` field.
By convention the `MetadataType` value should follow these rules of thumb:
- Only use lowercase letters, numbers, and hyphens. Use hyphens to separate words.
- **Try to anchor the name in the ecosystem, language, or packaging tooling it belongs to**. For a package manager for a language ecosystem the language, framework or runtime should be used as a prefix. For instance `pubspec-lock` is an OK name, but `dart-pubspec-lock` is better. For an OS package manager this is not necessary (e.g. `apk-db-entry` is a good name, but `alpine-apk-db-entry` is not since `alpine` and the `a` in `apk` is redundant).
- **Be as specific as possible to what the data represents**. For instance `ruby-gem` is NOT a good `MetadataType` value, but `ruby-gemspec` is. Why? Ruby gem information can come from a gemspec file or a Gemfile.lock, which are very different. The latter name provides more context as to what to expect.
- **Should describe WHAT the data is, NOT HOW it's used**. For instance `r-description-installed-file` is NOT a good `MetadataType` value since it's trying to convey that we use the DESCRIPTION file in the R ecosystem to detect installed packages. Instead simply describe what the DESCRIPTION file is itself without context of how it's used: `r-description`.
- **Use the `lock` suffix** to distinct between manifest files that loosely describe package version requirements vs files that strongly specify one and only one version of a package ("lock" files). These should only be used with respect to package managers that have the guide and lock distinction, but would not be appropriate otherwise (e.g. `rpm` does not have a guide vs lock, so `lock` should NOT be used to describe a db entry).
- **Use the `archive` suffix to indicate a package archive** (e.g. rpm file, apk file, etc) that describes the contents of the package. For example an RPM file that was cataloged would have a `rpm-archive` metadata type (not to be confused with an RPM DB record entry which would be `rpm-db-entry`).
- **Use the `entry` suffix** to indicate information about a package that was found as a single entry within file that has multiple package entries. If the entry was found within a DB or a flat-file store for an OS package manager, you should use `db-entry`.
- **Should NOT contain the phrase `package`**, though exceptions are allowed (say if the canonical name literally has the phrase package in it).
- **Should NOT contain have a `file` suffix** unless the canonical name has the term "file", such as a `pipfile` or `gemfile`. An example of a bad name for this rule is`ruby-gemspec-file`; a better name would be `ruby-gemspec`.
- **Should NOT contain the exact filename+extensions**. For instance `pipfile.lock` shouldn't really be in the name, instead try and describe what the file is: `python-pipfile-lock` (but shouldn't this be `python-pip-lock` you might ask? No, since the `pip` package manger is not related to the `pipfile` project).
- **Should NOT contain the phrase `metadata`**, unless the canonical name has this term.
- **Should represent a single use case**. For example, trying to describe Hackage metadata with a single `HackageMetadata` struct (and thus `MetadataType`) is not allowed since it represents 3 mutually exclusive use cases: representing a `stack.yaml`, `stack.lock`, or `cabal.project` file. Instead, each of these should have their own struct types and `MetadataType` values.
There are other cases that are not covered by these rules... and that's ok! The goal is to provide a consistent naming scheme that is easy to understand and use when it's applicable. If the rules do not exactly apply in your situation then just use your best judgement (or amend these rules as needed whe new common cases come up).
What if the underlying parsed data represents multiple files? There are two approaches to this:
- use the primary file to represent all the data. For instance, though the `dpkg-cataloger` looks at multiple files to get all information about a package, it's the `status` file that gets represented.
- nest each individual file's data under the `Metadata` field. For instance, the `java-archive-cataloger` may find information from on or all of the files: `pom.xml`, `pom.properties`, and `MANIFEST.MF`. However, the metadata is simply `java-metadata' with each possibility as a nested optional field.
### Syft Catalogers
Catalogers are the way in which syft is able to identify and construct packages given a set a targeted list of files.
For example, a cataloger can ask syft for all `package-lock.json` files in order to parse and raise up javascript packages
(see [how file globs](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/javascript/cataloger.go#L16-L21) and
[file parser functions](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/javascript/cataloger.go#L16-L21) are used
for a quick example).
From a high level catalogers have the following properties:
- _They are independent from one another_. The java cataloger has no idea of the processes, assumptions, or results of the python cataloger, for example.
- _They do not know what source is being analyzed_. Are we analyzing a local directory? an image? if so, the squashed representation or all layers? The catalogers do not know the answers to these questions. Only that there is an interface to query for file paths and contents from an underlying "source" being scanned.
- _Packages created by the cataloger should not be mutated after they are created_. There is one exception made for adding CPEs to a package after the cataloging phase, but that will most likely be moved back into the cataloger in the future.
Cataloger names should be unique and named with the following rules of thumb in mind:
- Must end with `-cataloger`
- Use lowercase letters, numbers, and hyphens only
- Use hyphens to separate words
- Catalogers for language ecosystems should start with the language name (e.g. `python-` for a cataloger that raises up python packages)
- Distinct between when the cataloger is searching for evidence of installed packages vs declared packages. For example, there are currently two different gemspec-based catalogers, the `ruby-gemspec-cataloger` and `ruby-installed-gemspec-cataloger`, where the latter requires that the gemspec is found within a `specifications` directory (which means it was installed, not just at the root of a source repo).
#### Building a new Cataloger
Catalogers must fulfill the [`pkg.Cataloger` interface](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger.go) in order to add packages to the SBOM.
All catalogers should be added to:
- the [global list of catalogers](https://github.com/anchore/syft/blob/9995950c70e849f9921919faffbfcf46401f71f3/syft/pkg/cataloger/cataloger.go#L92-L125)
- at least one source-specific list, today the two lists are [directory catalogers and image catalogers](https://github.com/anchore/syft/blob/9995950c70e849f9921919faffbfcf46401f71f3/syft/pkg/cataloger/cataloger.go#L39-L89)
For reference, catalogers are [invoked within syft](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/catalog.go#L41-L100) one after the other, and can be invoked in parallel.
`generic.NewCataloger` is an abstraction syft used to make writing common components easier (see the [apkdb cataloger](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/apkdb/cataloger.go) for example usage).
It takes the following information as input:
- A `catalogerName` to identify the cataloger uniquely among all other catalogers.
- Pairs of file globs as well as parser functions to parse those files. These parser functions return a slice of [`pkg.Package`](https://github.com/anchore/syft/blob/9995950c70e849f9921919faffbfcf46401f71f3/syft/pkg/package.go#L19) as well as a slice of [`artifact.Relationship`](https://github.com/anchore/syft/blob/9995950c70e849f9921919faffbfcf46401f71f3/syft/artifact/relationship.go#L31) to describe how the returned packages are related. See this [the apkdb cataloger parser function](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/apkdb/parse_apk_db.go#L22-L102) as an example.
Identified packages share a common `pkg.Package` struct so be sure that when the new cataloger is constructing a new package it is using the [`Package` struct](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/package.go#L16-L31).
If you want to return more information than what is available on the `pkg.Package` struct then you can do so in the `pkg.Package.Metadata` section of the struct, which is unique for each [`pkg.Type`](https://github.com/anchore/syft/blob/v0.70.0/syft/pkg/type.go).
See [the `pkg` package](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg) for examples of the different metadata types that are supported today.
These are plugged into the `MetadataType` and `Metadata` fields in the above struct. `MetadataType` informs which type is being used. `Metadata` is an interface converted to that type.
Finally, here is an example of where the package construction is done within the apk cataloger:
- [Calling the APK package constructor from the parser function](https://github.com/anchore/syft/blob/v0.70.0/syft/pkg/cataloger/apkdb/parse_apk_db.go#L106)
Interested in building a new cataloger? Checkout the [list of issues with the `new-cataloger` label](https://github.com/anchore/syft/issues?q=is%3Aopen+is%3Aissue+label%3Anew-cataloger+no%3Aassignee)!
If you have questions about implementing a cataloger feel free to file an issue or reach out to us [on slack](https://anchore.com/slack)!
#### Searching for files
All catalogers are provided an instance of the [`file.Resolver`](https://github.com/anchore/syft/blob/v0.70.0/syft/source/file_resolver.go#L8) to interface with the image and search for files. The implementations for these
abstractions leverage [`stereoscope`](https://github.com/anchore/stereoscope) in order to perform searching. Here is a
rough outline how that works:
1. a stereoscope `file.Index` is searched based on the input given (a path, glob, or MIME type). The index is relatively fast to search, but requires results to be filtered down to the files that exist in the specific layer(s) of interest. This is done automatically by the `filetree.Searcher` abstraction. This abstraction will fallback to searching directly against the raw `filetree.FileTree` if the index does not contain the file(s) of interest. Note: the `filetree.Searcher` is used by the `file.Resolver` abstraction.
2. Once the set of files are returned from the `filetree.Searcher` the results are filtered down further to return the most unique file results. For example, you may have requested for files by a glob that returns multiple results. These results are filtered down to deduplicate by real files, so if a result contains two references to the same file, say one accessed via symlink and one accessed via the real path, then the real path reference is returned and the symlink reference is filtered out. If both were accessed by symlink then the first (by lexical order) is returned. This is done automatically by the `file.Resolver` abstraction.
3. By the time results reach the `pkg.Cataloger` you are guaranteed to have a set of unique files that exist in the layer(s) of interest (relative to what the resolver supports).
## Testing
### Levels of testing
-`unit`: The default level of test which is distributed throughout the repo are unit tests. Any `_test.go` file that
does not reside somewhere within the `/test` directory is a unit test. Other forms of testing should be organized in
the `/test` directory. These tests should focus on correctness of functionality in depth. % Test coverage metrics
the `/test` directory. These tests should focus on correctness of functionality in depth. % test coverage metrics
only considers unit tests and no other forms of testing.
-`integration`: located within `test/integration`, these tests focus on the behavior surfaced by the common library
-`integration`: located within `cmd/syft/internal/test/integration`, these tests focus on the behavior surfaced by the common library
entrypoints from the `syft` package and make light assertions about the results surfaced. Additionally, these tests
tend to make diversity assertions for enum-like objects, ensuring that as enum values are added to a definition
that integration tests will automatically fail if no test attempts to use that enum value. For more details see
@@ -166,14 +379,4 @@ These flags are defined at the top of the test files that have tests that use th
Snapshot testing is only as good as the manual verification of the golden snapshot file saved to the repo! Be careful
and diligent when updating these files.
## Architecture
TODO: outline:
- analysis creates a static SBOM which can be encoded and decoded.
- format objects, should strive to not add or enrich data in encoding that could otherwise be done during analysis
A CLI tool and Go library for generating a Software Bill of Materials (SBOM) from container images and filesystems. Exceptional for vulnerability detection when used with a scanner tool like [Grype](https://github.com/anchore/grype).
A CLI tool and Go library for generating a Software Bill of Materials (SBOM) from container images and filesystems. Exceptional for vulnerability detection when used with a scanner like [Grype](https://github.com/anchore/grype).
### Join our community meetings!
@@ -17,18 +17,43 @@ A CLI tool and Go library for generating a Software Bill of Materials (SBOM) fro
- Agenda: https://docs.google.com/document/d/1ZtSAa6fj2a6KRWviTn3WoJm09edvrNUp4Iz_dOjjyY8/edit?usp=sharing (join [this group](https://groups.google.com/g/anchore-oss-community) for write access)
- All are welcome!
For commercial support options with Syft or Grype, please [contact Anchore](https://get.anchore.com/contact/)
-Catalog container images and filesystems to discover packages and libraries.
-Generate in-toto attestations where an SBOM is included as the payload.
-Supports packages and libraries from various ecosystems (APK, DEB, RPM, Ruby Bundles, Python Wheel/Egg/requirements.txt, JavaScript NPM/Yarn, Java JAR/EAR/WAR/PAR/SAR, Jenkins plugins JPI/HPI, Go modules, PHP Composer)
-Linux distribution identification (supports Alpine, BusyBox, CentOS/RedHat, Debian/Ubuntu flavored distributions)
-Supports Docker and OCI image formats
-Direct support for [Grype](https://github.com/anchore/grype), a fast and powerful vulnerability matcher.
-Generates SBOMs for container images, filesystems, archives, and more to discover packages and libraries
-Supports OCI, Docker and [Singularity](https://github.com/sylabs/singularity) image formats
-Linux distribution identification
-Works seamlessly with [Grype](https://github.com/anchore/grype) (a fast, modern vulnerability scanner)
-Able to create signed SBOM attestations using the [in-toto specification](https://github.com/in-toto/attestation/blob/main/spec/README.md)
-Convert between SBOM formats, such as CycloneDX, SPDX, and Syft's own format.
### Supported Ecosystems
If you encounter an issue, please [let us know using the issue tracker](https://github.com/anchore/syft/issues).
- Alpine (apk)
- C (conan)
- C++ (conan)
- Dart (pubs)
- Debian (dpkg)
- Dotnet (deps.json)
- Objective-C (cocoapods)
- Elixir (mix)
- Erlang (rebar3)
- Go (go.mod, Go binaries)
- Haskell (cabal, stack)
- Java (jar, ear, war, par, sar, nar, native-image)
- JavaScript (npm, yarn)
- Jenkins Plugins (jpi, hpi)
- Linux kernel archives (vmlinz)
- Linux kernel modules (ko)
- Nix (outputs in /nix/store)
- PHP (composer)
- Python (wheel, egg, poetry, requirements.txt)
- Red Hat (rpm)
- Ruby (gem)
- Rust (cargo.lock)
- Swift (cocoapods, swift-package-manager)
## Installation
@@ -45,23 +70,34 @@ curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b <DESTINATION_DIR> <RELEASE_VERSION>
```
### Chocolatey
The chocolatey distribution of syft is community maintained and not distributed by the anchore team
```powershell
chocoinstallsyft-y
```
### Scoop
```powershell
scoopinstallsyft
```
### Homebrew
```bash
brew tap anchore/syft
brew install syft
```
### Nix
**Note**: nix packaging of Syft is [community maintained](https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/admin/syft/default.nix)
Also syft is currently only in the [unstable channel](https://nixos.wiki/wiki/Nix_channels#The_official_channels) awaiting the `22.05` release
**Note**: Nix packaging of Syft is [community maintained](https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/admin/syft/default.nix). Syft is available in the [stable channel](https://nixos.wiki/wiki/Nix_channels#The_official_channels) since NixOS `22.05`.
```bash
nix-env -i syft
```
... or, just try it out in an ephemeral nix shell
... or, just try it out in an ephemeral nix shell:
```bash
nix-shell -p syft
@@ -69,60 +105,165 @@ nix-shell -p syft
## Getting started
#### SBOM
To generate an SBOM for an OCI image:
```
### SBOM
To generate an SBOM for a container image:
```bash
syft <image>
```
**Note**: This is equivalent to specifying the `packages` subcommand:
```
syft packages <image>
```
The above output includes only software that is visible in the container (i.e., the squashed representation of the image).
To include software from all image layers in the SBOM, regardless of its presence in the final image, provide `--scope all-layers`:
The above output includes only software that is visible in the container (i.e., the squashed representation of the image). To include software from all image layers in the SBOM, regardless of its presence in the final image, provide `--scope all-layers`:
```bash
syft <image> --scope all-layers
```
syft packages <image> --scope all-layers
```
#### SBOM Attestation
To generate an attested SBOM for an OCI image as the predicate of an in-toto attestation
The above output is in the form of the [DSSE envelope](https://github.com/secure-systems-lab/dsse/blob/master/envelope.md#dsse-envelope).
The payload is a base64 encoded `in-toto` statement with the SBOM as the predicate, the payload type is `application/vnd.in-toto+json`, and the signatures array is populated
with the contents needed for public key verification. For details on workflows using this command see [here](#adding-an-sbom-to-an-image-as-an-attestation-using-syft).
### Supported sources
Syft can generate a SBOM from a variety of sources:
Syft can generate an SBOM from a variety of sources:
```
```bash
# catalog a container image archive (from the result of `docker image save ...`, `podman save ...`, or `skopeo copy` commands)
syft packages path/to/image.tar
syft path/to/image.tar
# catalog a Singularity Image Format (SIF) container
syft path/to/image.sif
# catalog a directory
syft packages path/to/dir
syft path/to/dir
```
Sources can be explicitly provided with a scheme:
```
docker:yourrepo/yourimage:tag use images from the Docker daemon
podman:yourrepo/yourimage:tag use images from the Podman daemon
docker-archive:path/to/yourimage.tar use a tarball from disk for archives created from "docker save"
oci-archive:path/to/yourimage.tar use a tarball from disk for OCI archives (from Skopeo or otherwise)
oci-dir:path/to/yourimage read directly from a path on disk for OCI layout directories (from Skopeo or otherwise)
dir:path/to/yourproject read directly from a path on disk (any directory)
file:path/to/yourproject/file read directly from a path on disk (any single file)
registry:yourrepo/yourimage:tag pull image directly from a registry (no container runtime required)
docker:yourrepo/yourimage:tag use images from the Docker daemon
podman:yourrepo/yourimage:tag use images from the Podman daemon
containerd:yourrepo/yourimage:tag use images from the Containerd daemon
docker-archive:path/to/yourimage.tar use a tarball from disk for archives created from "docker save"
oci-archive:path/to/yourimage.taruse a tarball from disk for OCI archives (from Skopeo or otherwise)
oci-dir:path/to/yourimage read directly from a path on disk for OCI layout directories (from Skopeo or otherwise)
singularity:path/to/yourimage.sif read directly from a Singularity Image Format (SIF) container on disk
dir:path/to/yourproject read directly from a path on disk (any directory)
file:path/to/yourproject/file read directly from a path on disk (any single file)
registry:yourrepo/yourimage:tag pull image directly from a registry (no container runtime required)
```
If an image source is not provided and cannot be detected from the given reference it is assumed the image should be pulled from the Docker daemon.
If docker is not present, then the Podman daemon is attempted next, followed by reaching out directly to the image registry last.
This default behavior can be overridden with the `default-image-pull-source` configuration option (See [Configuration](https://github.com/anchore/syft#configuration) for more details).
### File selection
By default, Syft will catalog file details and digests for files that are owned by discovered packages. You can change this behavior by using the `SYFT_FILE_METADATA_SELECTION` environment variable or the `file.metadata.selection` configuration option. The options are:
-`all`: capture all files from the search space
-`owned-by-package`: capture only files owned by packages (default)
-`none`: disable capturing any file information
### Package cataloger selection
#### Concepts
> [!IMPORTANT]
> Syft uses a different set of catalogers by default when scanning files directly than it does when scanning images
The catalogers for an image scan assumes that package installation steps have already been completed. For example, Syft will identify Python packages that have egg or wheel metadata files under a `site-packages` directory, since this is how the canonical tooling `pip` installs python packages.
The catalogers for a directory scan will look for installed software as well as declared dependencies that are not necessarily installed. For example, dependencies listed in a Python `requirements.txt`.
This default set of catalogers being dynamic is critical as this allows Syft to be used in a variety of contexts while still generating accurate SBOMs.
Overriding the set of default catalogers is not recommended for most purposes, however, is possible if needed.
Catalogers can be referenced in two different ways:
- *by name*: the exact cataloger name (e.g. `java-pom-cataloger` or `java-archive-cataloger`)
- *by tag*: a tag that is associated with a cataloger (e.g. `java`)
Syft can take lists of references on the CLI or in the application configuration to define which catalogers to use.
You can **set** the list of catalogers explicitly to use with the `--override-default-catalogers` CLI flag, accepting a comma-separated list of cataloger names or tags.
You can also **add** to, **remove** from, or **sub-select** catalogers to use within the default set of catalogers by using the `--select-catalogers` CLI flag.
- To **sub-select** catalogers simply provide a tag (e.g. `--select-catalogers TAG`). Catalogers will always be selected from the default set of catalogers (e.g. `--select-catalogers java,go` will select all the `java` catalogers in the default set and all the `go` catalogers in the default set).
- To **add** a cataloger prefix the cataloger name with `+` (e.g. `--select-catalogers +NAME`). Added catalogers will _always be added_ regardless of removals, filtering, or other defaults.
- To **remove** a cataloger prefix the cataloger name or tag with `-` (e.g. `--select-catalogers -NAME_OR_TAG`). Catalogers are removed from the set of default catalogers after processing any sub-selections.
These rules and the dynamic default cataloger sets approximates to the following logic:
```
image_catalogers = all_catalogers AND catalogers_tagged("image")
directory_catalogers = all_catalogers AND catalogers_tagged("directory")
default_catalogers = image_catalogers OR directory_catalogers
sub_selected_catalogers = default_catalogers INTERSECT catalogers_tagged(TAG) [ UNION sub_selected_catalogers ... ]
base_catalogers = default_catalogers OR sub_selected_catalogers
final_set = (base_catalogers SUBTRACT removed_catalogers) UNION added_catalogers
```
#### Examples
Only scan for python related packages with catalogers appropriate for the source type (image or directory):
Syft can exclude files and paths from being scanned within a source by using glob expressions
@@ -146,38 +287,73 @@ The output format for Syft is configurable as well using the
`-o` (or `--output`) option:
```
syft packages <image> -o <format>
syft <image> -o <format>
```
Where the `formats` available are:
-`json`: Use this to get as much information out of Syft as possible!
-`text`: A row-oriented, human-and-machine-friendly output.
-`cyclonedx`: A XML report conforming to the [CycloneDX 1.3 specification](https://cyclonedx.org/specification/overview/).
-`cyclonedx-json`: A JSON report conforming to the [CycloneDX 1.3 specification](https://cyclonedx.org/specification/overview/).
-`spdx`: A tag-value formatted report conforming to the [SPDX 2.2 specification](https://spdx.github.io/spdx-spec/).
-`spdx-json`: A JSON report conforming to the [SPDX 2.2 JSON Schema](https://github.com/spdx/spdx-spec/blob/v2.2/schemas/spdx-schema.json).
-`table`: A columnar summary (default).
-`syft-json`: Use this to get as much information out of Syft as possible!
-`syft-text`: A row-oriented, human-and-machine-friendly output.
-`cyclonedx-xml`: A XML report conforming to the [CycloneDX 1.4 specification](https://cyclonedx.org/specification/overview/).
-`cyclonedx-json`: A JSON report conforming to the [CycloneDX 1.4 specification](https://cyclonedx.org/specification/overview/).
-`spdx-tag-value`: A tag-value formatted report conforming to the [SPDX 2.3 specification](https://spdx.github.io/spdx-spec/v2.3/).
-`spdx-tag-value@2.2`: A tag-value formatted report conforming to the [SPDX 2.2 specification](https://spdx.github.io/spdx-spec/v2.2.2/).
-`spdx-json`: A JSON report conforming to the [SPDX 2.3 JSON Schema](https://github.com/spdx/spdx-spec/blob/v2.3/schemas/spdx-schema.json).
-`spdx-json@2.2`: A JSON report conforming to the [SPDX 2.2 JSON Schema](https://github.com/spdx/spdx-spec/blob/v2.2/schemas/spdx-schema.json).
-`github-json`: A JSON report conforming to GitHub's dependency snapshot format.
-`syft-table`: A columnar summary (default).
-`template`: Lets the user specify the output format. See ["Using templates"](#using-templates) below.
#### Multiple outputs
## Using templates
Syft lets you define custom output formats, using [Go templates](https://pkg.go.dev/text/template). Here's how it works:
- Define your format as a Go template, and save this template as a file.
- Set the output format to "template" (`-o template`).
- Specify the path to the template file (`-t ./path/to/custom.template`).
- Syft's template processing uses the same data models as the `syft-json` output format — so if you're wondering what data is available as you author a template, you can use the output from `syft <image> -o syft-json` as a reference.
**Example:** You could make Syft output data in CSV format by writing a Go template that renders CSV data and then running `syft <image> -o template -t ~/path/to/csv.tmpl`.
Syft also includes a vast array of utility templating functions from [sprig](http://masterminds.github.io/sprig/) apart from the default Golang [text/template](https://pkg.go.dev/text/template#hdr-Functions) to allow users to customize the output format.
Lastly, Syft has custom templating functions defined in `./syft/format/template/encoder.go` to help parse the passed-in JSON structs.
## Multiple outputs
Syft can also output _multiple_ files in differing formats by appending
`=<file>` to the option, for example to output Syft JSON and SPDX JSON:
When a container runtime is not present, Syft can still utilize credentials configured in common credential sources (such as `~/.docker/config.json`).
It will pull images from private registries using these credentials. The config file is where your credentials are stored when authenticating with private registries via some command like `docker login`.
For more information see the `go-containerregistry` [documentation](https://github.com/google/go-containerregistry/tree/main/pkg/authn).
When a container runtime is not present, Syft can still utilize credentials configured in common credential sources (such as `~/.docker/config.json`). It will pull images from private registries using these credentials. The config file is where your credentials are stored when authenticating with private registries via some command like `docker login`. For more information see the `go-containerregistry` [documentation](https://github.com/google/go-containerregistry/tree/main/pkg/authn).
An example `config.json` looks something like this:
```
// config.json
```json
{
"auths":{
"registry.example.com":{
@@ -190,17 +366,19 @@ An example `config.json` looks something like this:
You can run the following command as an example. It details the mount/environment configuration a container needs to access a private registry:
`docker run -v ./config.json:/config/config.json -e "DOCKER_CONFIG=/config" anchore/syft:latest <private_image>`
```
docker run -v ./config.json:/config/config.json -e "DOCKER_CONFIG=/config" anchore/syft:latest <private_image>
```
### Docker Credentials in Kubernetes
The below section shows a simple workflow on how to mount this config file as a secret into a container on kubernetes.
1. Create a secret. The value of `config.json` is important. It refers to the specification detailed [here](https://github.com/google/go-containerregistry/tree/main/pkg/authn#the-config-file).
Below this section is the `secret.yaml` file that the pod configuration will consume as a volume.
The key `config.json` is important. It will end up being the name of the file when mounted into the pod.
```
Here's a simple workflow to mount this config file as a secret into a container on Kubernetes.
1. Create a secret. The value of `config.json` is important. It refers to the specification detailed [here](https://github.com/google/go-containerregistry/tree/main/pkg/authn#the-config-file). Below this section is the `secret.yaml` file that the pod configuration will consume as a volume. The key `config.json` is important. It will end up being the name of the file when mounted into the pod.
```yaml
# secret.yaml
apiVersion: v1
kind: Secret
metadata:
@@ -210,16 +388,14 @@ The key `config.json` is important. It will end up being the name of the file wh
config.json: <base64 encoded config.json>
```
`kubectl apply -f secret.yaml`
`kubectl apply -f secret.yaml`
2. Create your pod running syft. The env `DOCKER_CONFIG` is important because it advertises where to look for the credential file.
In the below example, setting `DOCKER_CONFIG=/config` informs syft that credentials can be found at `/config/config.json`.
This is why we used `config.json` as the key for our secret. When mounted into containers the secrets' key is used as the filename.
The `volumeMounts` section mounts our secret to `/config`. The `volumes` section names our volume and leverages the secret we created in step one.
```
2. Create your pod running syft. The env `DOCKER_CONFIG` is important because it advertises where to look for the credential file. In the below example, setting `DOCKER_CONFIG=/config` informs syft that credentials can be found at `/config/config.json`. This is why we used `config.json` as the key for our secret. When mounted into containers the secrets' key is used as the filename. The `volumeMounts` section mounts our secret to `/config`. The `volumes` section names our volume and leverages the secret we created in step one.
```yaml
# pod.yaml
apiVersion: v1
kind: Pod
metadata:
@@ -243,13 +419,75 @@ The `volumeMounts` section mounts our secret to `/config`. The `volumes` section
secretName: registry-config
```
`kubectl apply -f pod.yaml`
`kubectl apply -f pod.yaml`
3. The user can now run `kubectl logs syft-private-registry-demo`. The logs should show the syft analysis for the `<private_image>` provided in the pod configuration.
3. The user can now run `kubectl logs syft-private-registry-demo`. The logs should show the Syft analysis for the `<private_image>` provided in the pod configuration.
Using the above information, users should be able to configure private registry access without having to do so in the `grype` or `syft` configuration files. They will also not be dependent on a Docker daemon, (or some other runtime software) for registry configuration and access.
## Format conversion (experimental)
The ability to convert existing SBOMs means you can create SBOMs in different formats quickly, without the need to regenerate the SBOM from scratch, which may take significantly more time.
This feature is experimental and data might be lost when converting formats. Packages are the main SBOM component easily transferable across formats, whereas files and relationships, as well as other information Syft doesn't support, are more likely to be lost.
We support formats with wide community usage AND good encode/decode support by Syft. The supported formats are:
- Syft JSON (```-o syft-json```)
- SPDX 2.2 JSON (```-o spdx-json```)
- SPDX 2.2 tag-value (```-o spdx-tag-value```)
- CycloneDX 1.4 JSON (```-o cyclonedx-json```)
- CycloneDX 1.4 XML (```-o cyclonedx-xml```)
Conversion example:
```sh
syft alpine:latest -o syft-json=sbom.syft.json # generate a syft SBOM
syft convert sbom.syft.json -o cyclonedx-json=sbom.cdx.json # convert it to CycloneDX
```
## Attestation (experimental)
### Keyless support
Syft supports generating attestations using cosign's [keyless](https://github.com/sigstore/cosign/blob/main/KEYLESS.md) signatures.
Note: users need to have >= v1.12.0 of cosign installed for this command to function
To use this feature with a format like CycloneDX json simply run:
```
syft attest --output cyclonedx-json <IMAGE WITH OCI WRITE ACCESS>
```
This command will open a web browser and allow the user to authenticate their OIDC identity as the root of trust for the attestation (Github, Google, Microsoft).
After authenticating, Syft will upload the attestation to the OCI registry specified by the image that the user has write access to.
You will need to make sure your credentials are configured for the OCI registry you are uploading to so that the attestation can write successfully.
Users can then verify the attestation(or any image with attestations) by running:
Users should see that the uploaded attestation claims are validated, the claims exist within the transparency log, and certificates on the attestations were verified against [fulcio](https://github.com/SigStore/fulcio).
There will also be a printout of the certificates subject `<user identity>` and the certificate issuer URL: `<provider of user identity (Github, Google, Microsoft)>`:
The above output is in the form of the [DSSE envelope](https://github.com/secure-systems-lab/dsse/blob/master/envelope.md#dsse-envelope).
The payload is a base64 encoded `in-toto` statement with the generated SBOM as the predicate. For details on workflows using this command see [here](#adding-an-sbom-to-an-image-as-an-attestation-using-syft).
Using the above information, users should be able to configure private registry access without having to do so in the `grype` or `syft` configuration files.
They will also not be dependent on a docker daemon, (or some other runtime software) for registry configuration and access.
SBOMs themselves can serve as input to different analysis tools. The Anchore organization offers the vulnerability scanner
[grype](https://github.com/anchore/grype) as one such tool.
One of the foundational approaches to "trust" between tools is for producers to use the artifacts generated by syft as attestations to their images.
The DSSE output of `syft attest` can be used with the [cosign](https://github.com/sigstore/cosign) tool to attach an attestation to an image.
SBOMs themselves can serve as input to different analysis tools. [Grype](https://github.com/anchore/grype), a vulnerability scanner CLI tool from Anchore, is one such tool. Publishers of container images can use attestations to enable their consumers to trust Syft-generated SBOM descriptions of those container images. To create and provide these attestations, image publishers can run `syft attest` in conjunction with the [cosign](https://github.com/sigstore/cosign) tool to attach SBOM attestations to their images.
#### Example attest
Note for the following example replace `docker.io/image:latest` with an image you own. You should also have push access to
its remote reference. Replace $MY_PRIVATE_KEY with a private key you own or have generated with cosign.
#### Example attestation
Note for the following example replace `docker.io/image:latest` with an image you own. You should also have push access to
its remote reference. Replace `$MY_PRIVATE_KEY` with a private key you own or have generated with cosign.
{{.appName}}{{.command}} --key [KEY] yourrepo/yourimage:tag defaults to using images from a Docker daemon. If Docker is not present, the image is pulled directly from the registry.
{{.appName}}{{.command}} --key [KEY] path/to/a/file/or/dir only for OCI tar or OCI directory
returnfmt.Errorf("could not generate source input for attest command: %w",err)
}
switchsi.Scheme{
casesource.ImageScheme,source.UnknownScheme:
// at this point we know that it cannot be dir: or file: schemes, so we will assume that the unknown scheme could represent an image
si.Scheme=source.ImageScheme
default:
returnfmt.Errorf("attest command can only be used with image sources but discovered %q when given %q",si.Scheme,userInput)
}
// if the original detection was from a local daemon we want to short circuit
// that and attempt to generate the image source from a registry source instead
switchsi.ImageSource{
caseimage.UnknownSource,image.OciRegistrySource:
si.ImageSource=image.OciRegistrySource
default:
returnfmt.Errorf("attest command can only be used with image sources fetch directly from the registry, but discovered an image source of %q when given %q",si.ImageSource,userInput)
}
iflen(appConfig.Outputs)>1{
returnfmt.Errorf("unable to generate attestation for more than one output")
}
format:=syft.FormatByName(appConfig.Outputs[0])
predicateType:=formatPredicateType(format)
ifpredicateType==""{
returnfmt.Errorf("could not produce attestation predicate for given format: %q. Available formats: %+v",formatAliases(format.ID()),formatAliases(attestFormats...))
returnfmt.Errorf("cannot generate attestation since no repo digests were found; make sure you're passing an OCI registry source for the attest command")
case1:
default:
returnfmt.Errorf("cannot generate attestation since multiple repo digests were found for the image: %+v",src.Image.Metadata.RepoDigests)
// NOTE(alex): Write a helper for the binding operation, which can be used to perform the binding but also double check that the intended effect was had or else return an error. Another thought is to somehow provide zero-valued defaults for all values in our config struct (maybe with reflection?). There may be a mechanism that already exists in viper that protects against this that I'm not aware of. ref: https://github.com/anchore/syft/pull/805#discussion_r801931192
Short:"Generate an SBOM as an attestation for the given [SOURCE] container image",
Long:"Generate a packaged-based Software Bill Of Materials (SBOM) from a container image as the predicate of an in-toto attestation that will be uploaded to the image registry",
Long:"[Experimental] Convert SBOM files to, and from, SPDX, CycloneDX and Syft's format. For more info about data loss between formats see https://github.com/anchore/syft#format-conversion-experimental",
err=multierror.Append(err,task.ErrInvalidExpression{Expression:"bar",Operation:task.SubSelectOperation,Err:errors.New("explanation")})// this is what makes this test different...
returnfmt.Errorf("unable to read config file during validations %q: %w",o.Config.ConfigFile,err)
}
varlegacylegacyConfig
iferr:=yaml.Unmarshal(by,&legacy);err!=nil{
returnfmt.Errorf("unable to parse config file during validations %q: %w",o.Config.ConfigFile,err)
}
iflegacy.DefaultImagePullSource!=nil{
returnfmt.Errorf("the config file option 'default-image-pull-source' has been removed, please use 'source.image.default-pull-source' instead")
}
iflegacy.ExcludeBinaryOverlapByOwnership!=nil{
returnfmt.Errorf("the config file option 'exclude-binary-overlap-by-ownership' has been removed, please use 'package.exclude-binary-overlap-by-ownership' instead")
}
iflegacy.BasePath!=nil{
returnfmt.Errorf("the config file option 'base-path' has been removed, please use 'source.base-path' instead")
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.