chore: migrate fixtures to testdata (#4651)

* migrate fixtures to testdata

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: correct broken symlinks after testdata migration

The migration from test-fixtures to testdata broke several symlinks:
- elf-test-fixtures symlinks pointed to old test-fixtures paths
- elf-test-fixtures needed to be renamed to elf-testdata
- image-pkg-coverage symlink pointed to test-fixtures instead of testdata

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: handle missing classifiers/bin directory in Makefile

The clean-fingerprint target was failing when classifiers/bin doesn't
exist (e.g., on fresh clone without downloaded binaries).

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: add gitignore negation for jar/zip fixtures in test/cli

The jar and zip files in test/cli/testdata/image-unknowns were being
gitignored by the root .gitignore patterns. This caused them to be
untracked and not included when building docker images in CI, resulting
in Test_Unknowns failures since the test expects errors from corrupt
archive files that weren't present.

Add a .gitignore in test/cli/testdata to negate the exclusions for
these specific test fixture files.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* switch fixture cache to v2

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test: update expected versions for rebuilt fixtures

Update test expectations for packages that have been updated in
upstream repositories when docker images are rebuilt:
- glibc: 2.42-r4 → 2.43-r1 (wolfi)
- php: 8.2.29 → 8.2.30 (ubuntu/apache)

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* upgrade go

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: add go-shlex dependency for testdata manager tool

The manager tool in syft/pkg/cataloger/binary/testdata/ imports
go-shlex, but since it's in a testdata directory, Go doesn't track
its dependencies. This caused CI failures when go.mod didn't
explicitly list the dependency.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* refactor: move binary classifier manager to internal/

Move the manager tool from testdata/manager to internal/manager so
that Go properly tracks its dependencies. Code in testdata directories
is ignored by Go for dependency tracking, which caused CI failures
when go.mod didn't explicitly list transitive dependencies.

This is a cleaner solution than manually adding dependencies to go.mod
for code that happens to live in testdata.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: add gitignore negations for test fixtures blocked by root patterns

Multiple test fixtures were being blocked by root-level gitignore patterns
like bin/, *.jar, *.tar, and *.exe. This adds targeted .gitignore files with
negation patterns to allow these specific test fixtures to be tracked:

- syft/linux/testdata/os/busybox/bin/busybox (blocked by bin/)
- syft/pkg/cataloger/java/testdata/corrupt/example.{jar,tar} (blocked by *.jar, *.tar)
- syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/**/bin/go (blocked by bin/)
- syft/pkg/cataloger/bitnami/testdata/no-rel/.../bin/redis-server (blocked by bin/)

Also updates the bitnami test expectation to include the newly required
.gitignore files in the test fixture.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test: update glibc version expectation (2.43-r1 -> 2.43-r2)

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add capability drift check as unit step

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* dont clear test observations before drift detection

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* bump stereoscope commit to main

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

syft/pkg/cataloger/binary/README.md

@@ -2,7 +2,7 @@
 
 > [!TIP]
 > **TL;DR** to add a test for a new classifier:
->  1. head to the correct directory: `cd test-fixtures`
+>  1. head to the correct directory: `cd testdata`
 >  2. add a new entry to `config.yaml` to track where to get the binary from (verify the entry with `make list`)
 >  3. run `make download` to get the binary
 >  4. run `make add-snippet` and follow the prompts (use `/` to search)
@@ -20,7 +20,7 @@ The upside with full binaries is that they are the "Real McCoy" and allows the b
 
 You can find the test fixtures at the following locations:
 ```
-syft/pkg/cataloger/binary/test-fixtures/
+syft/pkg/cataloger/binary/testdata/
 └── classifiers/
     ├── bin/        # full binaries
     ├── ...
@@ -77,7 +77,7 @@ The test cases have been setup to allow testing against full binaries or a mix o
 To force running only against full binaries run with:
 
 ```bash
-go test -must-use-full-binaries ./syft/pkg/cataloger/binary/test-fixtures/...
+go test -must-use-full-binaries ./syft/pkg/cataloger/binary/testdata/...
 ```
 
 ## Adding a new test fixture

syft/pkg/cataloger/binary/classifier_cataloger_test.go

@@ -20,7 +20,7 @@ import (
 	"github.com/anchore/syft/syft/cpe"
 	"github.com/anchore/syft/syft/file"
 	"github.com/anchore/syft/syft/pkg"
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/testutil"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/testutil"
 	"github.com/anchore/syft/syft/pkg/cataloger/internal/binutils"
 	"github.com/anchore/syft/syft/source"
 	"github.com/anchore/syft/syft/source/directorysource"
@@ -32,8 +32,8 @@ var mustUseOriginalBinaries = flag.Bool("must-use-original-binaries", false, "fo
 func Test_Cataloger_PositiveCases(t *testing.T) {
 	tests := []struct {
 		name string
-		// logicalFixture is the logical path to the full binary or snippet. This is relative to the test-fixtures/classifiers/snippets
-		// or test-fixtures/classifiers/bin directory . Snippets are searched for first, and if not found, then existing binaries are
+		// logicalFixture is the logical path to the full binary or snippet. This is relative to the testdata/classifiers/snippets
+		// or testdata/classifiers/bin directory . Snippets are searched for first, and if not found, then existing binaries are
 		// used. If no binary or snippet is found the test will fail. If '-must-use-original-binaries' is used the only
 		// full binaries are tested (no snippets), and if no binary is found the test will be skipped.
 		logicalFixture string
@@ -1948,8 +1948,8 @@ func Test_Cataloger_PositiveCases(t *testing.T) {
 		t.Run(test.logicalFixture, func(t *testing.T) {
 			c := NewClassifierCataloger(DefaultClassifierCatalogerConfig())
 
-			// logicalFixture is the logical path to the full binary or snippet. This is relative to the test-fixtures/classifiers/snippets
-			// or test-fixtures/classifiers/bin directory . Snippets are searched for first, and if not found, then existing binaries are
+			// logicalFixture is the logical path to the full binary or snippet. This is relative to the testdata/classifiers/snippets
+			// or testdata/classifiers/bin directory . Snippets are searched for first, and if not found, then existing binaries are
 			// used. If no binary or snippet is found the test will fail. If '-must-use-original-binaries' is used the only
 			// full binaries are tested (no snippets), and if no binary is found the test will be skipped.
 			path := testutil.SnippetOrBinary(t, test.logicalFixture, *mustUseOriginalBinaries)
@@ -2014,7 +2014,7 @@ func Test_Cataloger_DefaultClassifiers_PositiveCases_Image(t *testing.T) {
 func TestClassifierCataloger_DefaultClassifiers_NegativeCases(t *testing.T) {
 	c := NewClassifierCataloger(DefaultClassifierCatalogerConfig())
 
-	src, err := directorysource.NewFromPath("test-fixtures/classifiers/negative")
+	src, err := directorysource.NewFromPath("testdata/classifiers/negative")
 	assert.NoError(t, err)
 
 	resolver, err := src.FileResolver(source.SquashedScope)
@@ -2065,7 +2065,7 @@ func Test_Cataloger_CustomClassifiers(t *testing.T) {
 			config: ClassifierCatalogerConfig{
 				Classifiers: []binutils.Classifier{},
 			},
-			fixtureDir: "test-fixtures/custom/go-1.14",
+			fixtureDir: "testdata/custom/go-1.14",
 			expected:   nil,
 		},
 		{
@@ -2073,7 +2073,7 @@ func Test_Cataloger_CustomClassifiers(t *testing.T) {
 			config: ClassifierCatalogerConfig{
 				Classifiers: defaultClassifers,
 			},
-			fixtureDir: "test-fixtures/custom/go-1.14",
+			fixtureDir: "testdata/custom/go-1.14",
 			expected:   &golangExpected,
 		},
 		{
@@ -2081,7 +2081,7 @@ func Test_Cataloger_CustomClassifiers(t *testing.T) {
 			config: ClassifierCatalogerConfig{
 				Classifiers: []binutils.Classifier{fooClassifier},
 			},
-			fixtureDir: "test-fixtures/custom/go-1.14",
+			fixtureDir: "testdata/custom/go-1.14",
 			expected:   nil,
 		},
 		{
@@ -2092,7 +2092,7 @@ func Test_Cataloger_CustomClassifiers(t *testing.T) {
 					fooClassifier,
 				),
 			},
-			fixtureDir: "test-fixtures/custom/go-1.14",
+			fixtureDir: "testdata/custom/go-1.14",
 			expected:   &golangExpected,
 		},
 		{
@@ -2111,7 +2111,7 @@ func Test_Cataloger_CustomClassifiers(t *testing.T) {
 					},
 				),
 			},
-			fixtureDir: "test-fixtures/custom/extra",
+			fixtureDir: "testdata/custom/extra",
 			expected:   nil,
 		},
 		{
@@ -2122,7 +2122,7 @@ func Test_Cataloger_CustomClassifiers(t *testing.T) {
 					fooClassifier,
 				),
 			},
-			fixtureDir: "test-fixtures/custom/extra",
+			fixtureDir: "testdata/custom/extra",
 			expected:   &customExpected,
 		},
 	}

syft/pkg/cataloger/binary/elf_package_cataloger_test.go

@@ -21,7 +21,7 @@ func Test_ELFPackageCataloger(t *testing.T) {
 	}{
 		{
 			name:    "go case",
-			fixture: "elf-test-fixtures",
+			fixture: "elf-testdata",
 			expected: []pkg.Package{
 				{
 					Name:    "libhello_world.so",
@@ -119,8 +119,8 @@ func Test_ELFPackageCataloger(t *testing.T) {
 			expected: []pkg.Package{
 				{
 					Name:    "glibc",
-					Version: "2.42-r4",
-					PURL:    "pkg:apk/wolfi/glibc@2.42-r4?distro=wolfi",
+					Version: "2.43-r2",
+					PURL:    "pkg:apk/wolfi/glibc@2.43-r2?distro=wolfi",
 					Locations: file.NewLocationSet(
 						file.NewLocationFromDirectory("/lib/libBrokenLocale.so.1",
 							"sha256:559eaef4e501b8e7a150661a94ee8b9ebc63bfca3256953a703f9f82053346f2",

syft/pkg/cataloger/binary/internal/manager/internal/cli/cli.go

@@ -3,8 +3,8 @@ package cli
 import (
 	"github.com/spf13/cobra"
 
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal/cli/commands"
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal/config"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal/cli/commands"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal/config"
 )
 
 // list all managed binaries (in ./bin, organized by 'name-version/platform/binary')

syft/pkg/cataloger/binary/internal/manager/internal/cli/commands/add_snippet.go

@@ -9,9 +9,9 @@ import (
 	"github.com/anmitsu/go-shlex"
 	"github.com/spf13/cobra"
 
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal"
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal/config"
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal/ui"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal/config"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal/ui"
 )
 
 func AddSnippet(appConfig config.Application) *cobra.Command {

syft/pkg/cataloger/binary/internal/manager/internal/cli/commands/download.go

@@ -5,8 +5,8 @@ import (
 
 	"github.com/spf13/cobra"
 
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal"
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal/config"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal/config"
 )
 
 func Download(appConfig config.Application) *cobra.Command {

syft/pkg/cataloger/binary/internal/manager/internal/cli/commands/list.go

@@ -8,8 +8,8 @@ import (
 	"github.com/jedib0t/go-pretty/v6/table"
 	"github.com/spf13/cobra"
 
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal"
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal/config"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal/config"
 )
 
 func List(appConfig config.Application) *cobra.Command {

syft/pkg/cataloger/binary/internal/manager/internal/cli/commands/root.go

@@ -3,7 +3,7 @@ package commands
 import (
 	"github.com/spf13/cobra"
 
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal/config"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal/config"
 )
 
 func Root(_ config.Application) *cobra.Command {

syft/pkg/cataloger/binary/internal/manager/internal/cli/commands/write_snippet.go

@@ -13,8 +13,8 @@ import (
 	"github.com/spf13/cobra"
 	"go.yaml.in/yaml/v3"
 
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal"
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal/config"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal/config"
 )
 
 func WriteSnippet(appConfig config.Application) *cobra.Command {

syft/pkg/cataloger/binary/internal/manager/internal/download_from_image.go

@@ -11,8 +11,8 @@ import (
 
 	"github.com/google/uuid"
 
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal/config"
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal/ui"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal/config"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal/ui"
 )
 
 const digestFileSuffix = ".xxh64"

syft/pkg/cataloger/binary/internal/manager/internal/download_from_image_test.go

@@ -8,7 +8,7 @@ import (
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal/config"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal/config"
 )
 
 func TestIsDownloadStale(t *testing.T) {

syft/pkg/cataloger/binary/internal/manager/internal/list_entries.go

@@ -11,7 +11,7 @@ import (
 
 	"golang.org/x/exp/maps"
 
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal/config"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal/config"
 )
 
 type Entries map[LogicalEntryKey]EntryInfo

syft/pkg/cataloger/binary/internal/manager/internal/list_entries_test.go

@@ -8,7 +8,7 @@ import (
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal/config"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal/config"
 )
 
 func TestSortingLogicalEntryKeys(t *testing.T) {

syft/pkg/cataloger/binary/internal/manager/main.go

@@ -4,8 +4,8 @@ import (
 	"fmt"
 	"os"
 
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal/cli"
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal/ui"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal/cli"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal/ui"
 )
 
 func main() {

syft/pkg/cataloger/binary/internal/manager/testutil/snippet_or_binary.go

@@ -8,13 +8,13 @@ import (
 
 	"github.com/stretchr/testify/require"
 
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal"
-	"github.com/anchore/syft/syft/pkg/cataloger/binary/test-fixtures/manager/internal/config"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal"
+	"github.com/anchore/syft/syft/pkg/cataloger/binary/internal/manager/internal/config"
 )
 
 // SnippetOrBinary returns the path to either the binary or the snippet for the given logical entry key.
 // Note: this is intended to be used only within the context of the binary cataloger test fixtures. Any other
-// use is unsupported. Path should be a logical path relative to the test-fixtures/classifiers directory (but does
+// use is unsupported. Path should be a logical path relative to the testdata/classifiers directory (but does
 // not specify the "bin" or "snippets" parent path... this is determined logically [snippets > binary unless told
 // otherwise]). Path should also be to the directory containing the binary or snippets of interest (not the binaries
 // or snippets itself).
@@ -23,12 +23,12 @@ func SnippetOrBinary(t *testing.T, path string, requireBinary bool) string {
 
 	require.Len(t, internal.SplitFilepath(path), 3, "path must be a in the form <name>/<version>/<arch>")
 
-	// cd to test-fixtures directory and load the config
+	// cd to testdata directory and load the config
 
 	cwd, err := os.Getwd()
 	require.NoError(t, err)
 
-	require.NoError(t, os.Chdir("test-fixtures"))
+	require.NoError(t, os.Chdir("testdata"))
 	defer func() {
 		require.NoError(t, os.Chdir(cwd))
 	}()
@@ -65,7 +65,7 @@ func SnippetOrBinary(t *testing.T, path string, requireBinary bool) string {
 			}
 
 			if v.SnippetPath != "" && !v.IsConfigured {
-				t.Skip("no binary found, but is covered by a snippet. Please add this case to the 'binary/test-fixtures/config.yaml' and recreate the snippet")
+				t.Skip("no binary found, but is covered by a snippet. Please add this case to the 'binary/testdata/config.yaml' and recreate the snippet")
 			}
 
 			t.Fatalf("no binary found for %q", path)
@@ -78,7 +78,7 @@ func SnippetOrBinary(t *testing.T, path string, requireBinary bool) string {
 
 	// this should be relative to the tests-fixtures directory and should be the directory containing the binary or
 	// snippet of interest (not the path to the binary or snippet itself)
-	return filepath.Join("test-fixtures", filepath.Dir(fixturePath))
+	return filepath.Join("testdata", filepath.Dir(fixturePath))
 }
 
 func validateSnippet(binaryPath, snippetPath string) error {

syft/pkg/cataloger/binary/testdata/.gitignore

@@ -5,5 +5,7 @@ classifiers/bin
 !lib*.so
 !lib*.dylib
 
-# allow for go-hint file
-!VERSION*
+# allow for go-hint files and binaries
+!VERSION*
+!classifiers/snippets/**/bin/
+!*.exe

syft/pkg/cataloger/binary/testdata/Makefile

@@ -11,16 +11,16 @@ fixtures: download
 fingerprint: clean-fingerprint $(FINGERPRINT_FILE)
 
 list: ## list all managed binaries and snippets
-	go run ./manager list
+	go run ../internal/manager list
 
 download: ## download only binaries that are not covered by a snippet
-	go run ./manager download $(name) --skip-if-covered-by-snippet
+	go run ../internal/manager download $(name) --skip-if-covered-by-snippet
 
 download-all: ## download all managed binaries
-	go run ./manager download
+	go run ../internal/manager download
 
 add-snippet: ## add a new snippet from an existing binary
-	go run ./manager add-snippet
+	go run ../internal/manager add-snippet
 
 # requirement 3: we always need to recalculate the fingerprint based on source regardless of any existing fingerprint
 .PHONY: $(FINGERPRINT_FILE)
@@ -32,7 +32,7 @@ clean:  ## clean up all downloaded binaries
 	rm -rf $(BIN)
 
 clean-fingerprint: ## clean up all legacy fingerprint files
-	@find $(BIN) -name '*.fingerprint' -delete
+	@if [ -d $(BIN) ]; then find $(BIN) -name '*.fingerprint' -delete; fi
 
 
 ## Halp! #################################