99 Commits

Author SHA1 Message Date
Alex Goodman
da62387545
Fix SPDX namespace value (#649)
* fix spdx namespace and add scheme range assertions

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* validate SPDX document name from source metadata

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* comment why namespace tests only check prefix

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-12-02 14:10:40 +00:00
Alex Goodman
21d1738b27
fix directory content fetching (#651)
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-12-02 09:02:54 -05:00
Alex Goodman
fe616acd98
directory resolver should account for the proc cwd relative to the root path (#644)
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-11-30 15:34:37 -05:00
Alex Goodman
bd9007fc0e
Migrate SPDX-JSON relationships to SBOM model (#634)
* remove power-user document shape

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add power-user specific fields to syft-json format

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* port remaining spdx-json relationships to sbom model

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add coordinate set

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add SBOM file path helper

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* use internal mimetype helper in go binary cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add new package-of relationship

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update json schema to v2

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* replace power-user presenter with syft-json format

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix tests and linting

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* remove "package-of" relationship (in favor of "contains")

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add tests for spdx22json format encoding enhancements

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update TODO and log entries

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* introduce sbom.Descriptor

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-11-23 14:54:17 -05:00
Alex Goodman
d76c868481
Catalog archive contents for single-file input (#637)
* add first-level archive processing when input is a file

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add license exception for github.com/xi2/xz

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* always return cleanup function

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* change source.NewFromFile log entry to warn

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* ensure file source always has cleanup function

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* ensure we are always preferring the unarchive cleanup function for source

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-11-19 14:16:25 +00:00
Alex Goodman
e38cde35ed
Introduce minimal source coordinates (#623)
* split source.Location and create source.Coordinates for minimal path addressing

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* move coordinates into separate file

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* Update syft/source/coordinates.go

Co-authored-by: Dan Luhring <luhring@users.noreply.github.com>
2021-11-18 18:13:22 +00:00
Alex Goodman
ef627d82ef
Introduce relationships as first-class objects (#607)
* migrate pkg.ID and pkg.Relationship to artifact package

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* return relationships from tasks

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix more tests

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add artifact.Identifiable by Identity() method

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix linting

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* remove catalog ID assignment

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* adjust spdx helpers to use copy of packages

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* stabilize package ID relative to encode-decode format cycles

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* rename Identity() to ID()

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* use zero value for nils in ID generation

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* enable source.Location to be identifiable

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* hoist up package relationship discovery to analysis stage

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update ownership-by-file-overlap relationship description

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add test reminders to put new relationships under test

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* adjust PHP composer.lock parser function to return relationships

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-11-16 14:14:13 -05:00
Christopher Angelo Phillips
fc39710d06
287 - Add retry for image fetch using user's input (#626)
* add fallback to user input if source hint fails

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* refactor for smaller functions

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
2021-11-12 17:57:04 -05:00
Christopher Angelo Phillips
9bbc9ff633
Add layer information for go-module-binary-cataloger (#620)
* update functions to pass Location

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* update unit tests to pass new locations

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* fix image source.FileResolvers to include layer info

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add non-empty location in golang binary cataloger testing

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

Co-authored-by: Alex Goodman <alex.goodman@anchore.com>
2021-11-08 16:05:53 -05:00
Jonas Galvão Xavier
6d0ee326d8
ignore irregular files while indexing directory (#616)
* stop hidding  command from help doc

Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>

* do not index irregular files

Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>

* fix fixture dir and err name

Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>

* improve the description of irregular files

Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>

* explicitly check indexed file name

Signed-off-by: Jonas Galvão Xavier <jonas.agx@gmail.com>
2021-11-04 11:24:05 -07:00
Alex Goodman
3ac95ac4f6
Allow for cataloging a single file (#608)
* allow for cataloging a single file

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* use all catalogers for file schemes

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-11-02 12:09:06 -04:00
Christopher Angelo Phillips
10fa8dc7c9
Add windows support (#548)
* update  build tags, ui support, and stereoscope, and release for windows support

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
2021-10-21 12:49:36 -04:00
Christopher Angelo Phillips
3462e18af3
478 identify go binaries and extract mod information (#534)
* add query by MIME type to source.FileResolver

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* import stereoscope lib changes to find mime type

- add bin cataloger
- add bin parser
- add mime type go utils
- import new resolver

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* add go std library code to unpack bin

- keep them in their own (original) files
- add note for "this code was copied from"
- comment the lines the required changing

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
Co-authored-by: Alex Goodman <alex.goodman@anchore.com>
2021-10-07 12:16:38 -04:00
Alex Goodman
a000a2926b
update linter + fix whitespace (#536)
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-10-06 13:46:55 +00:00
Alex Goodman
9189ed68df
Add query by MIME type to source.FileResolver (#529)
* add query by MIME type to source.FileResolver

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* pull in stereoscope MIME type feature

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix tests

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-10-05 20:50:53 -04:00
houdini91
2f99a35f51
Power user command support for directory scans (#467)
* Power-user directory source support
Signed-off-by: Mikey Strauss <mikey@scribe-security.com>

Signed-off-by: houdini91 <mdstrauss91@gmail.com>

* Remove newline

Signed-off-by: houdini91 <mdstrauss91@gmail.com>

* Shared filetree (#1)

* Shared directory resolver filetree

Signed-off-by: houdini91 <mdstrauss91@gmail.com>

* PR - change error ErrObserve to ErrPath

Signed-off-by: houdini91 <mdstrauss91@gmail.com>

* PR - share directory resolver
* Use pointer to source struct

Signed-off-by: houdini91 <mdstrauss91@gmail.com>

* Fix Lint

Signed-off-by: houdini91 <mdstrauss91@gmail.com>
2021-09-08 09:18:53 -04:00
Alex Goodman
6f2b52971d
directory resolver indexer should report one progressable (#457)
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-07-02 10:18:41 -04:00
Alex Goodman
fb0857ff93
Add support for indexing root filesystem (#442)
* change directory resolver to ignore system runtime paths + drive by index

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add event/etui support for filesystem indexing (for dir resolver)

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add warnings for path indexing problems

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add directory resolver index tests

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* improve testing around directory resolver

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* renamed p var to path when not conflicting with import

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* pull docker image in CLI dir scan timeout test

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* ensure file not exist errors do not stop directory resolver indexing

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-06-29 22:06:47 +00:00
Alex Goodman
962e82297c
Split UI from event handling (#448)
* split UI from event handling

Signed-off-by: Alex Goodman <wagoodman@gmail.com>

* add event loop tests

Signed-off-by: Alex Goodman <wagoodman@gmail.com>

* use stereoscope cleanup function during signal handling

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* correct error wrapping in packages cmd

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* migrate ui event handlers to ui package

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* clarify command worker input var + remove dead comments

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-06-29 18:28:09 +00:00
Alex Goodman
c363b2b532
Add ability to pull images directly from a registry (#378)
* add registry image source

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* use explicit source for fetching image + add scheme and registry tests

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* adjust test variable name and add credential helper function

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-04-13 13:30:57 +00:00
Alex Goodman
c1551a03c5
add file contents cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-04-12 17:30:57 -04:00
Alex Goodman
9ec09add67
Add secrets search capability (#367)
* add initial secrets cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update ETUI elements with new catalogers (file metadata, digests, and secrets)

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update secrets cataloger to read full contents into memory for searching

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* quick prototype of parallelization secret regex search

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* quick prototype with single aggregated regex

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* quick prototype for secret search line-by-line

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* quick prototype hybrid secrets search

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add secrets cataloger with line strategy

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* adjust verbiage towards SearchResults instead of Secrets + add tests

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update json schema with secrets cataloger results

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* address PR comments

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update readme with secrets config options

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* ensure file catalogers call AllLocations once

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-04-01 21:34:15 +00:00
Alex Goodman
db35186c7d
allow file metadata digests to be optional + add link destination
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-03-29 12:07:42 -04:00
Alex Goodman
8551168702
capitalize file type strings
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-03-29 12:07:41 -04:00
Haim Helman
13a066ea18 Add repo digests if available from Stereoscope to the image metadata
Signed-off-by: Haim Helman <hhelman@vmware.com>
2021-03-25 05:57:23 -07:00
Alex Goodman
1b7c755536
add file-type mix as golden image
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-03-23 16:50:57 -04:00
Alex Goodman
d420368ba9
add tests around new file metadata cataloger
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-03-23 11:00:59 -04:00
Alex Goodman
a6cba5d9db
remove multi* content fetching from resolvers
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-03-22 10:53:51 -04:00
Alex Goodman
2f8a568d4f
check for multiple delimiters when parsing pom properties
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-03-20 08:00:08 -04:00
Alex Goodman
6a49717314
add VirtualPath to source.Location
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-01-05 21:16:42 -05:00
Alex Goodman
06641cfda2
prefer real paths for glob results
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-01-05 18:48:28 -05:00
Alex Goodman
fc8b431ea6
duplicate reference readers for duplicate location resolutions
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-01-05 13:54:39 -05:00
Alex Goodman
33c27c4f3d
add HasPath() to Resolver interface
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-01-04 19:39:49 -05:00
Alex Goodman
7f4e8ab97d
Fix symlink resolutions for constituent paths (#304)
* bump stereoscope to pull in content API refactors

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* incorporate symlink fixes

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* with filetree.File() adjustments

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* regress all-layers scope to not include dead-links + default tests to squashed scope

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* restore all layers resolver glob behavior (custom + lazy link resolution)

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* incorporate filetree link resolution options and restore no-follow dead link option for resolvers

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* removed path from lower-level FileTree.File() calls

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* bump stereoscope to pull in latest link resolution fixes

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* bump doublestar to v2 for directory resolver

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-01-04 16:41:45 -05:00
Dan Luhring
c2799b35d8
Fix mock resolver interface implementation
Signed-off-by: Dan Luhring <dan.luhring@anchore.com>
2021-01-04 14:18:51 -05:00
Alex Goodman
d475e6280a
bump stereoscope to pull in content API refactors
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2021-01-04 14:07:14 -05:00
Dan Luhring
359212e8ee
Disable lint rule prealloc
Signed-off-by: Dan Luhring <dan.luhring@anchore.com>
2020-12-23 11:35:49 -05:00
Dan Luhring
52e719dcb8
Create MockResolver and use to improve python cataloger tests
Signed-off-by: Dan Luhring <dan.luhring@anchore.com>
2020-12-23 11:24:49 -05:00
Alex Goodman
d94d7a7d80
add tests for content requester object
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2020-12-15 10:59:15 -05:00
Alex Goodman
e4a3e433b6
add content requested and refactor python cataloger to use it
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2020-12-15 10:59:14 -05:00
Alex Goodman
52bac6e2fd
Add enterprise upload capability (#285)
* add support to upload results to enterprise

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add package sbom upload

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add dockerfile support

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add manifest, index, and dockerfile import functions

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* schema version to json output + enhance json schema generation

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* modify package SBOM shape to be entire syft document + add etui updates

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add import image config and manifest support

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add config options for import to enterprise

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* incorporate final stereoscope and client-go deps

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2020-12-09 22:20:53 -05:00
Alex Goodman
030427bb33
rename json artifact to package + update resolver integrity checks
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2020-11-17 12:37:13 -05:00
Alex Goodman
c892c3609e
improve doc comments
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2020-11-17 12:37:13 -05:00
Alex Goodman
f46de19c6b
migrate scope option to image metadata (from source)
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2020-11-17 12:37:12 -05:00
Alex Goodman
91baabe5a1
add image metadata as catalogFromJSON return
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2020-11-17 12:37:12 -05:00
Alex Goodman
6f7a4fd3e4
move source metadata upstream and fix tests
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2020-11-17 12:37:12 -05:00
Alex Goodman
aa0d444fd4
fix tests to use location instead of file.Reference
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2020-11-17 12:37:12 -05:00
Alex Goodman
b694dacb21
add source.Location + reorient Resolvers to use it
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2020-11-17 12:37:11 -05:00
Alex Goodman
9668341a14
rename scope to source
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
2020-11-17 12:37:11 -05:00