chore: migrate fixtures to testdata (#4651)

* migrate fixtures to testdata

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: correct broken symlinks after testdata migration

The migration from test-fixtures to testdata broke several symlinks:
- elf-test-fixtures symlinks pointed to old test-fixtures paths
- elf-test-fixtures needed to be renamed to elf-testdata
- image-pkg-coverage symlink pointed to test-fixtures instead of testdata

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: handle missing classifiers/bin directory in Makefile

The clean-fingerprint target was failing when classifiers/bin doesn't
exist (e.g., on fresh clone without downloaded binaries).

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: add gitignore negation for jar/zip fixtures in test/cli

The jar and zip files in test/cli/testdata/image-unknowns were being
gitignored by the root .gitignore patterns. This caused them to be
untracked and not included when building docker images in CI, resulting
in Test_Unknowns failures since the test expects errors from corrupt
archive files that weren't present.

Add a .gitignore in test/cli/testdata to negate the exclusions for
these specific test fixture files.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* switch fixture cache to v2

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test: update expected versions for rebuilt fixtures

Update test expectations for packages that have been updated in
upstream repositories when docker images are rebuilt:
- glibc: 2.42-r4 → 2.43-r1 (wolfi)
- php: 8.2.29 → 8.2.30 (ubuntu/apache)

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* upgrade go

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: add go-shlex dependency for testdata manager tool

The manager tool in syft/pkg/cataloger/binary/testdata/ imports
go-shlex, but since it's in a testdata directory, Go doesn't track
its dependencies. This caused CI failures when go.mod didn't
explicitly list the dependency.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* refactor: move binary classifier manager to internal/

Move the manager tool from testdata/manager to internal/manager so
that Go properly tracks its dependencies. Code in testdata directories
is ignored by Go for dependency tracking, which caused CI failures
when go.mod didn't explicitly list transitive dependencies.

This is a cleaner solution than manually adding dependencies to go.mod
for code that happens to live in testdata.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: add gitignore negations for test fixtures blocked by root patterns

Multiple test fixtures were being blocked by root-level gitignore patterns
like bin/, *.jar, *.tar, and *.exe. This adds targeted .gitignore files with
negation patterns to allow these specific test fixtures to be tracked:

- syft/linux/testdata/os/busybox/bin/busybox (blocked by bin/)
- syft/pkg/cataloger/java/testdata/corrupt/example.{jar,tar} (blocked by *.jar, *.tar)
- syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/**/bin/go (blocked by bin/)
- syft/pkg/cataloger/bitnami/testdata/no-rel/.../bin/redis-server (blocked by bin/)

Also updates the bitnami test expectation to include the newly required
.gitignore files in the test fixture.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test: update glibc version expectation (2.43-r1 -> 2.43-r2)

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add capability drift check as unit step

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* dont clear test observations before drift detection

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* bump stereoscope commit to main

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

.github/actions/bootstrap/action.yaml

@@ -5,7 +5,7 @@ inputs:
   go-version:
     description: "Go version to install"
     required: true
-    default: "1.25.x"
+    default: "1.26.x"
   go-dependencies:
     description: "Download go dependencies"
     required: true

.github/scripts/capability-drift-check.sh

@@ -6,7 +6,7 @@ if [ "$(git status --porcelain | wc -l)" -ne "0" ]; then
   exit 1
 fi
 
-if ! make generate-capabilities; then
+if ! make generate-capabilities REFRESH=false; then
   echo "Generating capability descriptions failed"
   exit 1
 fi

.github/scripts/fingerprint_docker_fixtures.py

@@ -33,9 +33,9 @@ def is_git_tracked_or_untracked(directory):
 
 
 def find_test_fixture_dirs_with_images(base_dir):
-    """Find directories that contain 'test-fixtures' and at least one 'image-*' directory."""
+    """Find directories that contain 'testdata' and at least one 'image-*' directory."""
     for root, dirs, files in os.walk(base_dir):
-        if 'test-fixtures' in root:
+        if 'testdata' in root:
             image_dirs = [d for d in dirs if d.startswith('image-')]
             if image_dirs:
                 yield os.path.realpath(root)

.github/workflows/validations.yaml

@@ -58,6 +58,9 @@ jobs:
       - name: Run unit tests
         run: make unit
 
+      - name: Check for capability drift
+        run: make check-capability-drift
+
   Integration-Test:
     # Note: changing this job name requires making the same update in the .github/workflows/release.yaml pipeline
     name: "Integration tests"

.gitignore

@@ -37,6 +37,7 @@ VERSION
 coverage.txt
 *.log
 **/test-fixtures/test-observations.json
+**/testdata/test-observations.json
 
 # probable archives
 .images

Taskfile.yaml

@@ -8,7 +8,9 @@ vars:
   OWNER: anchore
   PROJECT: syft
 
-  CACHE_IMAGE: ghcr.io/{{ .OWNER }}/{{ .PROJECT }}/test-fixture-cache:latest
+  # v1: when fixtures were located at test-fixtures dirs
+  # v2: migration to testdata dirs
+  CACHE_IMAGE: ghcr.io/{{ .OWNER }}/{{ .PROJECT }}/test-fixture-cache:v2
 
   # static file dirs
   TOOL_DIR: .tool
@@ -73,7 +75,6 @@ tasks:
       - task: check-licenses
       - task: lint
       - task: check-json-schema-drift
-      - task: check-capability-drift
       - task: check-binary-fixture-size
 
   test:
@@ -199,7 +200,7 @@ tasks:
   check-binary-fixture-size:
     desc: Ensure that the binary test fixtures are not too large
     cmds:
-      - .github/scripts/check_binary_fixture_size.sh syft/pkg/cataloger/binary/test-fixtures/classifiers/snippets
+      - .github/scripts/check_binary_fixture_size.sh syft/pkg/cataloger/binary/testdata/classifiers/snippets
 
 
   ## Testing tasks #################################
@@ -266,9 +267,9 @@ tasks:
   fingerprints:
     desc: Generate fingerprints for all non-docker test fixture
     silent: true
-    # this will look for `test-fixtures/Makefile` and invoke the `fingerprint` target to calculate all cache input fingerprint files
+    # this will look for `testdata/Makefile` and invoke the `fingerprint` target to calculate all cache input fingerprint files
     generates:
-      - '**/test-fixtures/**/*.fingerprint'
+      - '**/testdata/**/*.fingerprint'
       - test/install/cache.fingerprint
     cmds:
       - |
@@ -277,7 +278,7 @@ tasks:
         RESET='\033[0m'
 
         echo -e "${YELLOW}creating fingerprint files for non-docker fixtures...${RESET}"
-        for dir in $(find . -type d -name 'test-fixtures'); do
+        for dir in $(find . -type d -name 'testdata'); do
           if [ -f "$dir/Makefile" ]; then
             # for debugging...
             #echo -e "${YELLOW}• calculating fingerprints in $dir... ${RESET}"
@@ -370,7 +371,7 @@ tasks:
   build-fixtures:
     desc: Generate all non-docker test fixtures
     silent: true
-    # this will look for `test-fixtures/Makefile` and invoke the `fixtures` target to generate any and all test fixtures
+    # this will look for `testdata/Makefile` and invoke the `fixtures` target to generate any and all test fixtures
     cmds:
       - |
         # we want to stop on the first build error
@@ -381,7 +382,7 @@ tasks:
         RESET='\033[0m'
 
         # Use a for loop with command substitution to avoid subshell issues
-        for dir in $(find . -type d -name 'test-fixtures'); do
+        for dir in $(find . -type d -name 'testdata'); do
           if [ -f "$dir/Makefile" ]; then
             echo -e "${YELLOW}${BOLD}generating fixtures in $dir${RESET}"
             make -C "$dir" fixtures
@@ -435,7 +436,7 @@ tasks:
       - "echo 'Docker daemon cache:'"
       - "docker images --format '{{`{{.ID}}`}} {{`{{.Repository}}`}}:{{`{{.Tag}}`}}' | grep stereoscope-fixture- | sort"
       - "echo '\nTar cache:'"
-      - 'find . -type f -wholename "**/test-fixtures/cache/stereoscope-fixture-*.tar" | sort'
+      - 'find . -type f -wholename "**/testdata/cache/stereoscope-fixture-*.tar" | sort'
 
   check-docker-cache:
     desc: Ensure docker caches aren't using too much disk space
@@ -469,7 +470,7 @@ tasks:
       - "cd test/install && make ci-test-mac"
 
   generate-compare-file:
-    cmd: "go run ./cmd/syft {{ .COMPARE_TEST_IMAGE }} -o json > {{ .COMPARE_DIR }}/test-fixtures/acceptance-{{ .COMPARE_TEST_IMAGE }}.json"
+    cmd: "go run ./cmd/syft {{ .COMPARE_TEST_IMAGE }} -o json > {{ .COMPARE_DIR }}/testdata/acceptance-{{ .COMPARE_TEST_IMAGE }}.json"
 
   compare-mac:
     deps: [tmpdir]
@@ -537,11 +538,16 @@ tasks:
     deps:
       - tmpdir
       - fixtures
+    vars:
+      # set REFRESH=true to run package tests first and refresh test observations (default: true)
+      REFRESH: '{{ .REFRESH | default "true" }}'
     cmds:
       # remove all test observations prior to regenerating
       - task: clean-test-observations
+        if: '{{ eq .REFRESH "true" }}'
       # this is required to update test observations; such evidence is used to update the packages/*.yaml
-      - "go test ./syft/pkg/... -count=1"
+      - cmd: "go test ./syft/pkg/... -count=1"
+        if: '{{ eq .REFRESH "true" }}'
       - "go generate ./internal/capabilities/..."
       - "gofmt -s -w ./internal/capabilities"
       # now that we have the latest capabilities, run completeness tests to ensure this is self-consistent
@@ -639,6 +645,15 @@ tasks:
 
   ## Cleanup targets #################################
 
+  clean:
+    desc: Remove all cache files and old builds
+    cmds:
+      - task: clean-snapshot
+      - task: clean-cache
+      - task: clean-test-observations
+      - task: clean-docker-cache
+      - task: clean-oras-cache
+
   clean-snapshot:
     desc: Remove any snapshot builds
     cmds:
@@ -648,7 +663,7 @@ tasks:
   clean-docker-cache:
     desc: Remove all docker cache tars and images from the daemon
     cmds:
-      - find . -type d -wholename "**/test-fixtures/cache" | xargs rm -rf
+      - find . -type d -wholename "**/testdata/cache" | xargs rm -rf
       - docker images --format '{{`{{.ID}}`}} {{`{{.Repository}}`}}' | grep stereoscope-fixture- | awk '{print $1}' | uniq | xargs -r docker rmi --force
 
   clean-oras-cache:
@@ -665,7 +680,7 @@ tasks:
         RESET='\033[0m'
 
         # Use a for loop with command substitution to avoid subshell issues
-        for dir in $(find . -type d -name 'test-fixtures'); do
+        for dir in $(find . -type d -name 'testdata'); do
         if [ -f "$dir/Makefile" ]; then
           echo -e "${YELLOW}${BOLD}deleting ephemeral test fixtures in $dir${RESET}"
           (make -C "$dir" clean)
@@ -675,6 +690,6 @@ tasks:
       - rm -f {{ .LAST_CACHE_PULL_FILE }} {{ .CACHE_PATHS_FILE }}
 
   clean-test-observations:
-    desc: Remove all test observations (i.e. test-fixtures/test-observations.json)
+    desc: Remove all test observations (i.e. testdata/test-observations.json)
     cmds:
-      - find . -type f -wholename "**/test-fixtures/test-observations.json" | xargs rm -f
+      - find . -type f -wholename "**/testdata/test-observations.json" | xargs rm -f

cmd/syft/internal/clio_setup_config.go

@@ -49,7 +49,7 @@ func AppClioSetupConfig(id clio.Identification, out io.Writer) *clio.SetupConfig
 			},
 		).
 		WithPostRuns(func(_ *clio.State, _ error) {
-			stereoscope.Cleanup()
+			stereoscope.Cleanup() //nolint:staticcheck // we don't have access to the image object here
 		})
 	return clioCfg
 }

cmd/syft/internal/commands/scan_test.go

@@ -19,30 +19,30 @@ func Test_scanOptions_validateLegacyOptionsNotUsed(t *testing.T) {
 		},
 		{
 			name: "config file with no legacy options",
-			cfg:  "test-fixtures/scan-configs/no-legacy-options.yaml",
+			cfg:  "testdata/scan-configs/no-legacy-options.yaml",
 		},
 		{
 			name:    "config file with default image pull source legacy option",
-			cfg:     "test-fixtures/scan-configs/with-default-pull-source.yaml",
+			cfg:     "testdata/scan-configs/with-default-pull-source.yaml",
 			wantErr: assertErrorContains("source.image.default-pull-source"),
 		},
 		{
 			name:    "config file with exclude-binary-overlap-by-ownership legacy option",
-			cfg:     "test-fixtures/scan-configs/with-exclude-binary-overlap-by-ownership.yaml",
+			cfg:     "testdata/scan-configs/with-exclude-binary-overlap-by-ownership.yaml",
 			wantErr: assertErrorContains("package.exclude-binary-overlap-by-ownership"),
 		},
 		{
 			name:    "config file with file string legacy option",
-			cfg:     "test-fixtures/scan-configs/with-file-string.yaml",
+			cfg:     "testdata/scan-configs/with-file-string.yaml",
 			wantErr: assertErrorContains("outputs"),
 		},
 		{
 			name: "config file with file section",
-			cfg:  "test-fixtures/scan-configs/with-file-section.yaml",
+			cfg:  "testdata/scan-configs/with-file-section.yaml",
 		},
 		{
 			name:    "config file with base-path legacy option",
-			cfg:     "test-fixtures/scan-configs/with-base-path.yaml",
+			cfg:     "testdata/scan-configs/with-base-path.yaml",
 			wantErr: assertErrorContains("source.base-path"),
 		},
 	}

cmd/syft/internal/test/integration/catalog_packages_test.go

@@ -125,7 +125,7 @@ func TestPkgCoverageImage(t *testing.T) {
 }
 
 func TestPkgCoverageDirectory(t *testing.T) {
-	sbom, _ := catalogDirectory(t, "test-fixtures/image-pkg-coverage")
+	sbom, _ := catalogDirectory(t, "testdata/image-pkg-coverage")
 
 	observedLanguages := strset.New()
 	definedLanguages := strset.New()
@@ -261,7 +261,7 @@ func TestPkgCoverageImage_HasEvidence(t *testing.T) {
 }
 
 func TestPkgCoverageDirectory_HasEvidence(t *testing.T) {
-	sbom, _ := catalogDirectory(t, "test-fixtures/image-pkg-coverage")
+	sbom, _ := catalogDirectory(t, "testdata/image-pkg-coverage")
 
 	var cases []testCase
 	cases = append(cases, commonTestCases...)

cmd/syft/internal/test/integration/files_test.go

@@ -22,7 +22,7 @@ import (
 func TestFileCataloging_Default(t *testing.T) {
 	cfg := options.DefaultCatalog().ToSBOMConfig(clio.Identification{})
 	cfg = cfg.WithFilesConfig(filecataloging.DefaultConfig())
-	sbom, _ := catalogDirectoryWithConfig(t, "test-fixtures/files", cfg)
+	sbom, _ := catalogDirectoryWithConfig(t, "testdata/files", cfg)
 
 	var metadata map[file.Coordinates]file.Metadata
 
@@ -48,13 +48,13 @@ func TestFileCataloging_AllFiles(t *testing.T) {
 			SkipFilesAboveSize: 30,
 		},
 	})
-	sbom, _ := catalogDirectoryWithConfig(t, "test-fixtures/files", cfg)
+	sbom, _ := catalogDirectoryWithConfig(t, "testdata/files", cfg)
 
 	pwd, err := os.Getwd()
 	require.NoError(t, err)
 
 	testPath := func(path string) string {
-		return filepath.Join(pwd, "test-fixtures/files", path)
+		return filepath.Join(pwd, "testdata/files", path)
 	}
 
 	metadata := map[file.Coordinates]file.Metadata{

cmd/syft/internal/test/integration/node_packages_test.go

@@ -11,7 +11,7 @@ import (
 )
 
 func TestNpmPackageLockDirectory(t *testing.T) {
-	sbom, _ := catalogDirectory(t, "test-fixtures/npm-lock")
+	sbom, _ := catalogDirectory(t, "testdata/npm-lock")
 
 	foundPackages := strset.New()
 
@@ -32,7 +32,7 @@ func TestNpmPackageLockDirectory(t *testing.T) {
 }
 
 func TestYarnPackageLockDirectory(t *testing.T) {
-	sbom, _ := catalogDirectory(t, "test-fixtures/yarn-lock")
+	sbom, _ := catalogDirectory(t, "testdata/yarn-lock")
 
 	foundPackages := strset.New()
 	// merge-objects and should-type are devDependencies in package.json and are excluded by default

cmd/syft/internal/test/integration/package_binary_elf_relationships_test.go

@@ -26,7 +26,7 @@ func TestBinaryElfRelationships(t *testing.T) {
 	}
 
 	// run the test...
-	sbom, _ := catalogFixtureImage(t, "elf-test-fixtures", source.SquashedScope)
+	sbom, _ := catalogFixtureImage(t, "elf-testdata", source.SquashedScope)
 
 	// get a mapping of package names to their IDs
 	nameToId := map[string]artifact.ID{}

cmd/syft/internal/test/integration/package_cataloger_convention_test.go

@@ -278,7 +278,7 @@ func packageCatalogerExports(t *testing.T) map[string]exportTokenSet {
 		if info.IsDir() ||
 			!strings.HasSuffix(info.Name(), ".go") ||
 			strings.HasSuffix(info.Name(), "_test.go") ||
-			strings.Contains(path, "test-fixtures") ||
+			strings.Contains(path, "testdata") ||
 			strings.Contains(path, "internal") {
 			return nil
 		}

cmd/syft/internal/test/integration/test-fixtures/elf-test-fixtures

@@ -1 +0,0 @@
-../../../../../../syft/pkg/cataloger/binary/test-fixtures/elf-test-fixtures

cmd/syft/internal/test/integration/test-fixtures/image-pkg-coverage/pkgs/java/generate-fixtures.md

@@ -1 +0,0 @@
-See the syft/cataloger/java/test-fixtures/java-builds dir to generate test fixtures and copy to here manually.

cmd/syft/internal/test/integration/testdata/.gitignore

@@ -1,5 +1,5 @@
 # we should strive to not commit blobs to the repo and strive to keep the build process of how blobs are acquired in-repo.
-# this blob is generated from syft/syft/catalogers/java/test-fixtures/java-builds , however, preserving the build process
+# this blob is generated from syft/syft/catalogers/java/testdata/java-builds , however, preserving the build process
 # twice in the repo seems redundant (even via symlink). Given that the fixture is a few kilobytes in size, the build process is already
 # captured, and integration tests should only be testing if jars can be discovered (not necessarily depth in java detection
 # functionality), committing it seems like an acceptable exception.

cmd/syft/internal/test/integration/testdata/elf-testdata

@@ -0,0 +1 @@
+../../../../../../syft/pkg/cataloger/binary/testdata/elf-testdata

cmd/syft/internal/test/integration/testdata/image-pkg-coverage/pkgs/java/generate-fixtures.md

@@ -0,0 +1 @@
+See the syft/cataloger/java/testdata/java-builds dir to generate test fixtures and copy to here manually.