* Fix CPE generation when the generated CPE contains invalid characters
Currently syft seems to generate invalid CPEs which do not
conform with the official CPE spec. This is because the underlying
nvdtools library is not a completely spec compliant implementation
and has some interesting bugs/issues.
The following are the list of issues I have encountered with nvdtools:
1. It parses strings which are not CPEs incorrectly as valid CPEs. This
messes up our filter function which is supposed to filter out any
incorrect CPEs we generate. In order to fix this, I have introduced
a new regex in the NewCPE function which follows the upstream spec and
filters out any incorrect CPEs.
2. Introduce wfn.WFNize for any cpe attributes we infer from packages.
This ensures that we are escaping and quoting any special characters
before putting them into CPEs. Note that nvdtools has yet another bug
in the WFNize function, specifically the "addSlashesAt" part of the
function which stops the loop as soon as it encounters ":" a valid
character for a WFN attribute after quoting, but the way nvdtools
handles it causes it to truncate strings that container ":". As a result
strings like "prefix:1.2" which would have been quoted as "prefix\:1.2"
end up becoming "prefix" instead causing loss of information and
incorrect CPEs being generated. As a result in such cases, we remove out
strings containing ":" in any part entirely for now. This is similar
to the way we were handling CPE filtering in the past with http urls as
vendor strings
3. Add special handling for version which contain ":" due to epochs in
debian and rpm. In this case, we strip out the parts before ":" i.e.
the epoch and only output the actual function. This ensures we are not
discarding valid version strings due to pt #.2.
In the future we should look at moving to a more spec compliant cpe
parsing library to avoid such shenanigans.
Signed-off-by: Sambhav Kothari <skothari44@bloomberg.net>
* Remove WFNize for input strings
WFNize seems to not be part of the standard as per
https://pkg.go.dev/github.com/facebookincubator/nvdtools@v0.1.4/wfn#WFNize
and seems to have bugs/issues with encode/decode cycles, so I am
just removing it at this point and relying on the CPE regex to filter
out invalid CPEs for now.
Signed-off-by: Sambhav Kothari <skothari44@bloomberg.net>
* Quote the string on decode to ensure consistent CPE string generation
Signed-off-by: Sambhav Kothari <skothari44@bloomberg.net>
* Add test cases for round-tripping the CPE and fix strip slashes
Signed-off-by: Sambhav Kothari <skothari44@bloomberg.net>
* Add comprehensive tests for cpe parsing
Signed-off-by: Sambhav Kothari <skothari44@bloomberg.net>
* Use strings.Builder instead of byte buffer
Signed-off-by: Sambhav Kothari <skothari44@bloomberg.net>
* migrate pkg.ID and pkg.Relationship to artifact package
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* return relationships from tasks
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix more tests
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add artifact.Identifiable by Identity() method
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* fix linting
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* remove catalog ID assignment
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* adjust spdx helpers to use copy of packages
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* stabilize package ID relative to encode-decode format cycles
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* rename Identity() to ID()
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* use zero value for nils in ID generation
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* enable source.Location to be identifiable
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* hoist up package relationship discovery to analysis stage
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* update ownership-by-file-overlap relationship description
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add test reminders to put new relationships under test
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* adjust PHP composer.lock parser function to return relationships
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add query by MIME type to source.FileResolver
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* import stereoscope lib changes to find mime type
- add bin cataloger
- add bin parser
- add mime type go utils
- import new resolver
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
* add go std library code to unpack bin
- keep them in their own (original) files
- add note for "this code was copied from"
- comment the lines the required changing
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
Co-authored-by: Alex Goodman <alex.goodman@anchore.com>
* enhance cpe generation for group id and filtering
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* rename group id const + add doc comment for HasAnyOfPrefixes
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add registry image source
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* use explicit source for fetching image + add scheme and registry tests
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* adjust test variable name and add credential helper function
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add marking package relations by file ownership
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* correct json schema version; ensure fileOwners dont return dups; pin test pkg versions
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* extract package relationships into separate section
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* pull in client-go features for import of PackageRelationships
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* move unit test for ownership by files relationship further down
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* rename relationship to "ownership-by-file-overlap"
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* bump stereoscope to pull in content API refactors
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* incorporate symlink fixes
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* with filetree.File() adjustments
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* regress all-layers scope to not include dead-links + default tests to squashed scope
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* restore all layers resolver glob behavior (custom + lazy link resolution)
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* incorporate filetree link resolution options and restore no-follow dead link option for resolvers
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* removed path from lower-level FileTree.File() calls
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* bump stereoscope to pull in latest link resolution fixes
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* bump doublestar to v2 for directory resolver
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add support to upload results to enterprise
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add package sbom upload
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add dockerfile support
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add manifest, index, and dockerfile import functions
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* schema version to json output + enhance json schema generation
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* modify package SBOM shape to be entire syft document + add etui updates
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add import image config and manifest support
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* add config options for import to enterprise
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* incorporate final stereoscope and client-go deps
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
* Expand matching of requirements.txt file to include any prefixes or suffixes
Signed-off-by: Samuel Dacanay <sam.dacanay@anchore.com>
* Add some test cases to integration test (ensure syft can pick up multiple requirements files)
Signed-off-by: Samuel Dacanay <sam.dacanay@anchore.com>
* Run lint-fix
Signed-off-by: Samuel Dacanay <sam.dacanay@anchore.com>
Alex Goodman
Sambhav Kothari
Keith Zantow
Blaize KayeandAlex Goodman
NikitaandAlex Goodman
Christopher Angelo Phillips
houdini91
Samuel Dacanay
Alfredo Deza
Weston Steimel
Dan Luhring
Toure Dunnon