* Option to enable specific language or ecosystem cataloger
Signed-off-by: ramanan-ravi <ramanan@deepfence.io>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* Disable dotnet cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* Option to enable specific language or ecosystem cataloger
Signed-off-by: Ramanan Ravikumar <ramanan@deepfence.io>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* rename "enable-cataloger" option to "catalogers"
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add cli test for --catalogers option
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* update readme with latest cataloger names
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* enable dotnet cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix linting
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix cataloger imports
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* update readme with alpmdb cataloger config example
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Co-authored-by: ramanan-ravi <ramanan@deepfence.io>
* add main module field to go bin metadata
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* udpate json ouput schema to 3.2.4
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* clean up fixture
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* Add filters to package cataloger
This PR adds filters so a package without name or version doesn't go in
the list of all discovered packages.
Integration and cli tests were added to validate the feature.
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* add nolint:funlen to cataloger/catalog.go
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* don't require package version
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* add package filtering to generic and python cataloger
also removes cli tests in favor of integration and unit tests
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* drop nolint:funlen
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* check for no-removal operation
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* remove unused fixtures
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* rename no-version file to hide semantic version
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* drop integration tests and add pkg func for validation
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* python cataloger use global pkg validation
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* check for valid packages on deb/go/rpm catalogers
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* update rpm cataloger after rebase
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* nit with pointers
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* simpler use of package validation
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* remmove double pkg validations
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* rename func param to artifactsToExclude
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* add test for relationships and bug fix
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* feedback changes
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* Support RPM distros with newer db formats
Recent RPM distros (Fedora 33+, CBL-Mariner 2.0+, amazonlinux 2022+)
use an sqlite package database in /var/lib/rpm/rpmdb.sqlite, or
"ndb" format (SUSE).
Remove anchore's fork in favour of the upstream,
https://github.com/knqyf263/go-rpmdb, to gain support for
these formats.
Signed-off-by: Tom Fay <tomfay@microsoft.com>
* add exception for modernc.org repos
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* shorten rpmdb helper function
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Co-authored-by: Alex Goodman <alex.goodman@anchore.com>
* golang module CPE with full path
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* add note on longer Golang CPEs
Signed-off-by: Jonas Xavier <jonasx@anchore.com>
* read Go main module version as is - (devel)
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
* fix package test with default (devel) main module
Signed-off-by: Jonas Galvão Xavier <jonasx@anchore.com>
The Official CPE dictionary currently contains entries for springframework with three different vendors: springsource, vmware, and pivotal_software. This appears to be because ownership has changed over time.
Signed-off-by: Weston Steimel <weston.steimel@anchore.com>
* Less verbose logging in Golang Cataloger
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* debug for known gray errors
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* only show warnings when a binary is not a go executable
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Co-authored-by: Alex Goodman <alex.goodman@anchore.com>
* initial working version
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* added build settings to pkg metadata
wip - unit tests
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* handle mach-O FatFiles
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* add support to mod replace
fixed golang catalger tests
trying GH Actions with go 1.18rc1
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* log error
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* use go-macholibre for extraction
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* cleaner tests
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* add version to main module
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* check macho file with macholibre
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* run golangci in its own workflow
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* wip - golangci workflow
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* fix golangci wf yml
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* fix golangci wf yml
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* wip - golangci wf
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* wip - golangci wf
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* get arch from bin file headers
upgrade macholibre
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* go mod tidy
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* test new stereoscope lazy reader interface
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* go mod tidy
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* remove devel version from golang cataloger
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* go mod tidy
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* switch github workflows to go1.18 stable
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* add union reader interface in golang cataloger
update stereoscope
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* go mod tidy
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* simpler golangci validation
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* fix makefile
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* get archs refactor
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* nolint for golang version
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* fix go bin tests
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* feedback changes
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* golangci nolint needs a \n before package
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* cleanup
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* move golangci-lint to its own jobs again
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* fix ci yaml
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* add support for xcoff files
add arch assets to test bin file types
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* clean up golangci-lint config
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* nolint for xcoff
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* explain nolints
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* remove unused xcoff testdata assets
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* make go bin test-fixtures in docker
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* fix make clean with -f
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* update json output schema
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* update schema version in test fixture
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* feedback changes
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* explain possible empty main module
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* add test cases for yarn parser regex
Signed-off-by: Patrick Glass <patrickglass@gmail.com>
* update yarn.lock parser to support yarn berry
Add support for Yarn v3 (berry) which changes the output
Collapse regex for parsing scoped and non-scoped packages
Add tests for the regex to ensure backwards compatability
and to catch issues with future changes.
Signed-off-by: Patrick Glass <patrickglass@gmail.com>
* simplify yarn test expressions
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Co-authored-by: Patrick Glass <patrickglass@gmail.com>
Co-authored-by: Alex Goodman <alex.goodman@anchore.com>
* ignore minor parsing error when reading dpkg status files
helps with https://github.com/anchore/syft/issues/733
Question: should we add a smarter parser to guess approximate installed-size
value?
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* add datasize lib to help dpkg parsing
added unit tests to expand coverage of dpkg parsing
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* drop parse error
added unit tests to handleNewKeyValue
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* don't return parsing errors from dpkg
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* go mod tidy
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* test higher level functions
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* return parsing err to let cataloger handle it
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* feedback changes
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* ignore key parsing error
log warning with relevant context
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* go mod tidy
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* add context info to log lines
simpler error assertion
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* use error.As to assert error in chain
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* [wip] get assets based on gh api
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* put install.sh download_asset fn under test
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* put install.sh install_asset fn under test
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* use zip for darwin installs
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix install.sh negative test cases
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* allow errors to propagate in install.sh
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* remove exit on error from install.sh tests
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add more docs around install.sh helpers
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add integration tests for install.sh
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add install.sh testing to pipeline
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add install test cache to CI
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* make colors globally available
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* test download against github release
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* always test release-based install against latest release
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* use better install.sh test names
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add language detection from pURLs
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add package type detection from pURLs
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add cargo and npm pURL support
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix npm tests and linting
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add php related metadata
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* enable decoding of php metadata for syftjson format
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add php metadata to json schema
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* remove strong distro type
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* bump json schema to v3 (breaking distro shape)
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix linting
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* allow for v2 decoding of distro idLikes field in v3 json decoder
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix casing in simple linux release name
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* use discovered name as pretty name in simple linux release
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* Fix CPE generation when the generated CPE contains invalid characters
Currently syft seems to generate invalid CPEs which do not
conform with the official CPE spec. This is because the underlying
nvdtools library is not a completely spec compliant implementation
and has some interesting bugs/issues.
The following are the list of issues I have encountered with nvdtools:
1. It parses strings which are not CPEs incorrectly as valid CPEs. This
messes up our filter function which is supposed to filter out any
incorrect CPEs we generate. In order to fix this, I have introduced
a new regex in the NewCPE function which follows the upstream spec and
filters out any incorrect CPEs.
2. Introduce wfn.WFNize for any cpe attributes we infer from packages.
This ensures that we are escaping and quoting any special characters
before putting them into CPEs. Note that nvdtools has yet another bug
in the WFNize function, specifically the "addSlashesAt" part of the
function which stops the loop as soon as it encounters ":" a valid
character for a WFN attribute after quoting, but the way nvdtools
handles it causes it to truncate strings that container ":". As a result
strings like "prefix:1.2" which would have been quoted as "prefix\:1.2"
end up becoming "prefix" instead causing loss of information and
incorrect CPEs being generated. As a result in such cases, we remove out
strings containing ":" in any part entirely for now. This is similar
to the way we were handling CPE filtering in the past with http urls as
vendor strings
3. Add special handling for version which contain ":" due to epochs in
debian and rpm. In this case, we strip out the parts before ":" i.e.
the epoch and only output the actual function. This ensures we are not
discarding valid version strings due to pt #.2.
In the future we should look at moving to a more spec compliant cpe
parsing library to avoid such shenanigans.
Signed-off-by: Sambhav Kothari <skothari44@bloomberg.net>
* Remove WFNize for input strings
WFNize seems to not be part of the standard as per
https://pkg.go.dev/github.com/facebookincubator/nvdtools@v0.1.4/wfn#WFNize
and seems to have bugs/issues with encode/decode cycles, so I am
just removing it at this point and relying on the CPE regex to filter
out invalid CPEs for now.
Signed-off-by: Sambhav Kothari <skothari44@bloomberg.net>
* Quote the string on decode to ensure consistent CPE string generation
Signed-off-by: Sambhav Kothari <skothari44@bloomberg.net>
* Add test cases for round-tripping the CPE and fix strip slashes
Signed-off-by: Sambhav Kothari <skothari44@bloomberg.net>
* Add comprehensive tests for cpe parsing
Signed-off-by: Sambhav Kothari <skothari44@bloomberg.net>
* Use strings.Builder instead of byte buffer
Signed-off-by: Sambhav Kothari <skothari44@bloomberg.net>
* add lpkg support to java cataloger
linter clean up
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* fix comment formatting
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* add filename test for lpkg
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* commment on lpkg file extension tests
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* fix comment typo
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* fix import format
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* simpler test validation
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* add direct_url.json fields to python metadata
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
* rename DirectURLOrigin struct; add stub for file
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
* add detection for direct_url.json
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
* Add tests for direct-url information and add it to the output purl
Signed-off-by: Sambhav Kothari <skothari44@bloomberg.net>
* Update golden snapshot ids after adding new python package metadata field
Signed-off-by: Sambhav Kothari <skothari44@bloomberg.net>
* Add test names for packageurl tests
Signed-off-by: Sambhav Kothari <skothari44@bloomberg.net>
Co-authored-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
* set package ID in catalogers and improve hashing performance
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* update setting ID + tests
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* Add failing test for extra empty lines in manifest
Signed-off-by: Dan Luhring <dan+github@luhrings.com>
* Handle extra empty lines in Java manifests
Signed-off-by: Dan Luhring <dan+github@luhrings.com>
* ignore target link files based on path
log when files are actually indexed
add test for sym link resolution
golang test nits
Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
* nil catalog should act like an empty catalog
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* remove dir path filtering in favor of file type filtering
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* split out addPathToIndex into specialized functions
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add test for nul catalog enumeration
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* conditionally discover MIME types for file based on file resolver index
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* change logging around cataloging
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add tests to cover possible infinite symlink loop for resolver
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Co-authored-by: Alex Goodman <alex.goodman@anchore.com>
