oss/syft
2
0
Fork 0
mirror of https://github.com/anchore/syft.git synced 2026-11-02 07:26:34 +01:00
Code Issues Packages Projects Releases Wiki Activity

Speed up cataloging by replacing globs searching with index lookups (#1510)

Browse Source 
* replace raw globs with index equivelent operations

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add cataloger test for alpm cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix import sorting for binary cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix linting for mock resolver

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* separate portage cataloger parser impl from cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* enhance cataloger pkgtest utils to account for resolver responses

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for alpm cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for apkdb cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for dpkg cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for cpp cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for dart cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for dotnet cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for elixir cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for erlang cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for golang cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for haskell cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for java cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for javascript cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for php cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for portage cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for python cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for rpm cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for rust cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for sbom cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for swift cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* allow generic catloger to run all mimetype searches at once

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* remove stutter from php and javascript cataloger constructors

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* bump stereoscope

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add tests for generic.Search

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add exceptions for java archive git ignore entries

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* enhance basename and extension resolver methods to be variadic

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* dont allow * prefix on extension searches

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* add glob-based cataloger tests for ruby cataloger

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* remove unnecessary string casting

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* incorporate surfacing of leaf link resolitions from stereoscope results

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* [wip] switch to stereoscope file metadata

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* [wip + failing] revert to old globs but keep new resolvers

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* index files, links, and dirs within the directory resolver

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix several resolver bugs and inconsistencies

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* move format testutils to internal package

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update syft json to account for file type string normalization

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* split up directory resolver from indexing

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update docs to include details about searching

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* [wip] bump stereoscope to development version

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix linting

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* adjust symlinks fixture to be fixed to digest

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix all-locations resolver tests

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix test fixture reference

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* rename file.Type

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* bump stereoscope

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix PR comment to exclude extra *

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* bump to dev version of stereoscope

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* bump to final version of stereoscope

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* move observing resolver to pkgtest

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

---------

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
This commit is contained in:
Alex Goodman
2023-02-09 16:19:47 +00:00
committed by GitHub
parent 550e2fc7c3
commit 988041ba6d
201 changed files with 4166 additions and 1370 deletions
Download Patch File Download Diff File Expand all files Collapse all files
syft/file/all_regular_files.go
+2 -1
View File
@@ -1,6 +1,7 @@
package file
import (
"github.com/anchore/stereoscope/pkg/file"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/source"
)
@@ -20,7 +21,7 @@ func allRegularFiles(resolver source.FileResolver) (locations []source.Location)
continue
}
if metadata.Type != source.RegularFile {
if metadata.Type != file.TypeRegular {
continue
}
locations = append(locations, resolvedLocation)
syft/file/all_regular_files_test.go
+3 -2
View File
@@ -3,6 +3,7 @@ package file
import (
"testing"
"github.com/google/go-cmp/cmp"
"github.com/scylladb/go-set/strset"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -69,8 +70,8 @@ func Test_allRegularFiles(t *testing.T) {
virtualLocations.Add(l.VirtualPath)
}
}
assert.ElementsMatch(t, tt.wantRealPaths.List(), realLocations.List(), "mismatched real paths")
assert.ElementsMatch(t, tt.wantVirtualPaths.List(), virtualLocations.List(), "mismatched virtual paths")
assert.ElementsMatch(t, tt.wantRealPaths.List(), realLocations.List(), "real paths differ: "+cmp.Diff(tt.wantRealPaths.List(), realLocations.List()))
assert.ElementsMatch(t, tt.wantVirtualPaths.List(), virtualLocations.List(), "virtual paths differ: "+cmp.Diff(tt.wantVirtualPaths.List(), virtualLocations.List()))
})
}
}
syft/file/digest_cataloger.go
+2 -1
View File
@@ -11,6 +11,7 @@ import (
"github.com/wagoodman/go-partybus"
"github.com/wagoodman/go-progress"
"github.com/anchore/stereoscope/pkg/file"
"github.com/anchore/syft/internal"
"github.com/anchore/syft/internal/bus"
"github.com/anchore/syft/internal/log"
@@ -65,7 +66,7 @@ func (i *DigestsCataloger) catalogLocation(resolver source.FileResolver, locatio
}
// we should only attempt to report digests for files that are regular files (don't attempt to resolve links)
if meta.Type != source.RegularFile {
if meta.Type != file.TypeRegular {
return nil, errUndigestableFile
}
syft/file/digest_cataloger_test.go
+1 -1
View File
@@ -146,7 +146,7 @@ func TestDigestsCataloger_MixFileTypes(t *testing.T) {
if err != nil {
t.Fatalf("unable to get file=%q : %+v", test.path, err)
}
l := source.NewLocationFromImage(test.path, *ref, img)
l := source.NewLocationFromImage(test.path, *ref.Reference, img)
if len(actual[l.Coordinates]) == 0 {
if test.expected != "" {
syft/file/metadata_cataloger_test.go
+18 -11
View File
@@ -6,6 +6,7 @@ import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/anchore/stereoscope/pkg/file"
"github.com/anchore/stereoscope/pkg/imagetest"
@@ -50,8 +51,9 @@ func TestFileMetadataCataloger(t *testing.T) {
path: "/file-1.txt",
exists: true,
expected: source.FileMetadata{
Path: "/file-1.txt",
Mode: 0644,
Type: "RegularFile",
Type: file.TypeRegular,
UserID: 1,
GroupID: 2,
Size: 7,
@@ -62,8 +64,9 @@ func TestFileMetadataCataloger(t *testing.T) {
path: "/hardlink-1",
exists: true,
expected: source.FileMetadata{
Path: "/hardlink-1",
Mode: 0644,
Type: "HardLink",
Type: file.TypeHardLink,
LinkDestination: "file-1.txt",
UserID: 1,
GroupID: 2,
@@ -74,8 +77,9 @@ func TestFileMetadataCataloger(t *testing.T) {
path: "/symlink-1",
exists: true,
expected: source.FileMetadata{
Path: "/symlink-1",
Mode: 0777 | os.ModeSymlink,
Type: "SymbolicLink",
Type: file.TypeSymLink,
LinkDestination: "file-1.txt",
UserID: 0,
GroupID: 0,
@@ -86,8 +90,9 @@ func TestFileMetadataCataloger(t *testing.T) {
path: "/char-device-1",
exists: true,
expected: source.FileMetadata{
Path: "/char-device-1",
Mode: 0644 | os.ModeDevice | os.ModeCharDevice,
Type: "CharacterDevice",
Type: file.TypeCharacterDevice,
UserID: 0,
GroupID: 0,
MIMEType: "",
@@ -97,8 +102,9 @@ func TestFileMetadataCataloger(t *testing.T) {
path: "/block-device-1",
exists: true,
expected: source.FileMetadata{
Path: "/block-device-1",
Mode: 0644 | os.ModeDevice,
Type: "BlockDevice",
Type: file.TypeBlockDevice,
UserID: 0,
GroupID: 0,
MIMEType: "",
@@ -108,8 +114,9 @@ func TestFileMetadataCataloger(t *testing.T) {
path: "/fifo-1",
exists: true,
expected: source.FileMetadata{
Path: "/fifo-1",
Mode: 0644 | os.ModeNamedPipe,
Type: "FIFONode",
Type: file.TypeFIFO,
UserID: 0,
GroupID: 0,
MIMEType: "",
@@ -119,11 +126,13 @@ func TestFileMetadataCataloger(t *testing.T) {
path: "/bin",
exists: true,
expected: source.FileMetadata{
Path: "/bin",
Mode: 0755 | os.ModeDir,
Type: "Directory",
Type: file.TypeDirectory,
UserID: 0,
GroupID: 0,
MIMEType: "",
IsDir: true,
},
},
}
@@ -131,11 +140,9 @@ func TestFileMetadataCataloger(t *testing.T) {
for _, test := range tests {
t.Run(test.path, func(t *testing.T) {
_, ref, err := img.SquashedTree().File(file.Path(test.path))
if err != nil {
t.Fatalf("unable to get file: %+v", err)
}
require.NoError(t, err)
l := source.NewLocationFromImage(test.path, *ref, img)
l := source.NewLocationFromImage(test.path, *ref.Reference, img)
assert.Equal(t, test.expected, actual[l.Coordinates], "mismatched metadata")