chore: migrate fixtures to testdata (#4651)

* migrate fixtures to testdata

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: correct broken symlinks after testdata migration

The migration from test-fixtures to testdata broke several symlinks:
- elf-test-fixtures symlinks pointed to old test-fixtures paths
- elf-test-fixtures needed to be renamed to elf-testdata
- image-pkg-coverage symlink pointed to test-fixtures instead of testdata

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: handle missing classifiers/bin directory in Makefile

The clean-fingerprint target was failing when classifiers/bin doesn't
exist (e.g., on fresh clone without downloaded binaries).

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: add gitignore negation for jar/zip fixtures in test/cli

The jar and zip files in test/cli/testdata/image-unknowns were being
gitignored by the root .gitignore patterns. This caused them to be
untracked and not included when building docker images in CI, resulting
in Test_Unknowns failures since the test expects errors from corrupt
archive files that weren't present.

Add a .gitignore in test/cli/testdata to negate the exclusions for
these specific test fixture files.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* switch fixture cache to v2

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test: update expected versions for rebuilt fixtures

Update test expectations for packages that have been updated in
upstream repositories when docker images are rebuilt:
- glibc: 2.42-r4 → 2.43-r1 (wolfi)
- php: 8.2.29 → 8.2.30 (ubuntu/apache)

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* upgrade go

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: add go-shlex dependency for testdata manager tool

The manager tool in syft/pkg/cataloger/binary/testdata/ imports
go-shlex, but since it's in a testdata directory, Go doesn't track
its dependencies. This caused CI failures when go.mod didn't
explicitly list the dependency.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* refactor: move binary classifier manager to internal/

Move the manager tool from testdata/manager to internal/manager so
that Go properly tracks its dependencies. Code in testdata directories
is ignored by Go for dependency tracking, which caused CI failures
when go.mod didn't explicitly list transitive dependencies.

This is a cleaner solution than manually adding dependencies to go.mod
for code that happens to live in testdata.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix: add gitignore negations for test fixtures blocked by root patterns

Multiple test fixtures were being blocked by root-level gitignore patterns
like bin/, *.jar, *.tar, and *.exe. This adds targeted .gitignore files with
negation patterns to allow these specific test fixtures to be tracked:

- syft/linux/testdata/os/busybox/bin/busybox (blocked by bin/)
- syft/pkg/cataloger/java/testdata/corrupt/example.{jar,tar} (blocked by *.jar, *.tar)
- syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/**/bin/go (blocked by bin/)
- syft/pkg/cataloger/bitnami/testdata/no-rel/.../bin/redis-server (blocked by bin/)

Also updates the bitnami test expectation to include the newly required
.gitignore files in the test fixture.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test: update glibc version expectation (2.43-r1 -> 2.43-r2)

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add capability drift check as unit step

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* dont clear test observations before drift detection

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* bump stereoscope commit to main

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

syft/pkg/cataloger/binary/testdata/.gitignore

@@ -0,0 +1,11 @@
+classifiers/dynamic
+classifiers/bin
+
+# allow for lb patterns (rust, pytho, php and more)
+!lib*.so
+!lib*.dylib
+
+# allow for go-hint files and binaries
+!VERSION*
+!classifiers/snippets/**/bin/
+!*.exe

syft/pkg/cataloger/binary/testdata/Makefile

@@ -0,0 +1,44 @@
+BIN=classifiers/bin
+FINGERPRINT_FILE=$(BIN).fingerprint
+
+
+.DEFAULT_GOAL := fixtures
+
+# requirement 1: 'fixtures' goal to generate any and all test fixtures
+fixtures: download
+
+# requirement 2: 'fingerprint' goal to determine if the fixture input that indicates any existing cache should be busted
+fingerprint: clean-fingerprint $(FINGERPRINT_FILE)
+
+list: ## list all managed binaries and snippets
+	go run ../internal/manager list
+
+download: ## download only binaries that are not covered by a snippet
+	go run ../internal/manager download $(name) --skip-if-covered-by-snippet
+
+download-all: ## download all managed binaries
+	go run ../internal/manager download
+
+add-snippet: ## add a new snippet from an existing binary
+	go run ../internal/manager add-snippet
+
+# requirement 3: we always need to recalculate the fingerprint based on source regardless of any existing fingerprint
+.PHONY: $(FINGERPRINT_FILE)
+$(FINGERPRINT_FILE): ## prints the sha256sum of the any input to the download command (to determine if there is a cache miss)
+	@sha256sum ./config.yaml > $(FINGERPRINT_FILE)
+
+# requirement 4: 'clean' goal to remove all generated test fixtures
+clean:  ## clean up all downloaded binaries
+	rm -rf $(BIN)
+
+clean-fingerprint: ## clean up all legacy fingerprint files
+	@if [ -d $(BIN) ]; then find $(BIN) -name '*.fingerprint' -delete; fi
+
+
+## Halp! #################################
+
+.PHONY: help
+help:
+	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "$(BOLD)$(CYAN)%-25s$(RESET)%s\n", $$1, $$2}'
+
+.PHONY: default list download download-all clean clean-fingerprint add-snippet fingerprint

syft/pkg/cataloger/binary/testdata/capture-snippet.sh

@@ -0,0 +1,154 @@
+#!/bin/bash
+
+# Default values for length and prefix length
+LENGTH=100
+PREFIX_LENGTH=20
+SEARCH_FOR=''
+GROUP_NAME=''
+
+# Function to show usage
+usage() {
+    echo "Usage: $0 <path-to-binary> <version> [--search-for <pattern>] [--length <length>] [--prefix-length <prefix_length>] [--group <name>]"
+    exit 1
+}
+
+# Parse arguments
+while [[ $# -gt 0 ]]; do
+    key="$1"
+
+    case $key in
+	--search-for)
+	    SEARCH_FOR="$2"
+	    shift # past argument
+            shift # past value
+	    ;;
+        --length)
+            LENGTH="$2"
+            shift # past argument
+            shift # past value
+            ;;
+        --group)
+            GROUP_NAME="$2"
+            shift # past argument
+            shift # past value
+            ;;
+        --prefix-length)
+            PREFIX_LENGTH="$2"
+            shift # past argument
+            shift # past value
+            ;;
+        *)
+            if [ -z "$BINARY_FILE" ]; then
+                BINARY_FILE="$1"
+            elif [ -z "$VERSION" ]; then
+                VERSION="$1"
+            else
+                echo "Unknown option: $1"
+                usage
+            fi
+            shift # past argument
+            ;;
+    esac
+done
+
+LENGTH=$(expr "$LENGTH" + "$PREFIX_LENGTH")
+
+# check if binary file and pattern are provided
+if [ -z "$BINARY_FILE" ] || [ -z "$VERSION" ]; then
+    usage
+fi
+
+# if group name is empty use the binary filename
+if [ -z "$GROUP_NAME" ]; then
+    GROUP_NAME=$(basename "$BINARY_FILE")
+fi
+
+# check if xxd is even installed
+if ! command -v xxd &> /dev/null; then
+    echo "xxd not found. Please install xxd."
+    exit 1
+fi
+
+# check if xargs is even installed
+if ! command -v xargs &> /dev/null; then
+    echo "xargs not found. Please install xargs."
+    exit 1
+fi
+
+PATTERN=${SEARCH_FOR:-$VERSION}
+
+echo "Using binary file:      $BINARY_FILE"
+echo "Searching for pattern:  $PATTERN"
+echo "Capture length:         $LENGTH bytes"
+echo "Capture prefix length:  $PREFIX_LENGTH bytes"
+
+PATTERN_RESULTS=$(strings -t d "$BINARY_FILE" | grep "$PATTERN")
+RESULT_COUNT=$(echo "$PATTERN_RESULTS" | wc -l)
+
+CONTINUE_LOOP=true
+
+while $CONTINUE_LOOP; do
+
+  # if there are multiple matches, prompt the user to select one
+  if [ $RESULT_COUNT -gt 1 ]; then
+      echo "Multiple string matches found in the binary:"
+      echo ""
+
+      # show result lines one at a time (in a numbered list)
+      # but only show everything after the first field (not the offset)
+      echo "$PATTERN_RESULTS" | cut -d ' ' -f 2- | nl -w 1 -s ') '
+
+
+      echo ""
+      read -p "Please select a match: " SELECTION
+
+      # if the selection is not a number, exit
+      if ! [[ "$SELECTION" =~ ^[0-9]+$ ]]; then
+          echo "Invalid selection."
+          exit 1
+      fi
+
+      # if the selection is out of bounds, exit
+      if [ "$SELECTION" -gt $(echo "$PATTERN_RESULTS" | wc -l) ]; then
+          echo "Invalid selection."
+          exit 1
+      fi
+
+      # select the line from the results
+      SELECTED_RESULT=$(echo "$PATTERN_RESULTS" | sed -n "${SELECTION}p")
+  else
+    SELECTED_RESULT="$PATTERN_RESULTS"
+  fi
+
+  # search for the pattern in the binary file and capture the offset
+  OFFSET=$(echo "${SELECTED_RESULT}" | xargs | cut -d ' ' -f 1)
+
+  if [ -z "$OFFSET" ]; then
+      echo "Pattern not found."
+      exit 1
+  fi
+
+  # adjust the offset to capture prefix length before the match
+  OFFSET=$(expr "$OFFSET" - "$PREFIX_LENGTH")
+
+  # use xxd to capture the specified length from the calculated offset
+  SNIPPET=$(xxd -l "$LENGTH" -s "$OFFSET" "$BINARY_FILE")
+
+  # display the output and prompt the user
+  echo ""
+  echo "$SNIPPET"
+  echo ""
+  read -p "Does this snippet capture what you need? (Y/n/q) " RESPONSE
+
+  RESPONSE=${RESPONSE:-y}
+
+  if [ "$RESPONSE" == "y" ]; then
+    CONTINUE_LOOP=false
+  elif [ $RESULT_COUNT -eq 1 ] || [ "$RESPONSE" == "q" ]; then
+    echo "Exiting with no action taken."
+    exit 0
+  fi
+
+done
+
+go run ./manager write-snippet "$BINARY_FILE" --offset "$OFFSET" --length "$LENGTH" --name "$GROUP_NAME" --version "$VERSION"

syft/pkg/cataloger/binary/testdata/classifiers/negative/.gitignore

@@ -0,0 +1 @@
+!libpython2.7.so

syft/pkg/cataloger/binary/testdata/classifiers/negative/busybox

@@ -0,0 +1 @@
+another bad binary

syft/pkg/cataloger/binary/testdata/classifiers/negative/go

@@ -0,0 +1 @@
+a bad go binary

syft/pkg/cataloger/binary/testdata/classifiers/negative/libpython2.7.so

@@ -0,0 +1,3 @@
+# note: this should NOT match
+
+DO NOT DETECT

syft/pkg/cataloger/binary/testdata/classifiers/negative/python2.6

@@ -0,0 +1,3 @@
+# note: this should NOT match
+
+just some noise

syft/pkg/cataloger/binary/testdata/classifiers/snippets/busybox/1.36.1/linux-amd64/busybox

@@ -0,0 +1 @@
+./[

syft/pkg/cataloger/binary/testdata/classifiers/snippets/consul/1.15.2/linux-amd64/consul

@@ -0,0 +1,12 @@
+name: consul
+offset: 57272433
+length: 100
+snippetSha256: a4a955b180d6df471797a9f17f5ebf6f23b92d688d40532712683922e119dac0
+fileSha256: 5c5fed218247eaf43c3b54008b6a4c5d5cfa1b38539d6e7bfc09ac04623389fc
+
+### byte snippet to follow ###
+e%7D" />
+
+  <!-- CONSUL_VERSION: 1.15.2
+ -->
+  <link rel="icon" href="{{.ContentPath}}assets/favicon

syft/pkg/cataloger/binary/testdata/classifiers/snippets/elixir/1.19.1/linux-amd64/elixir

@@ -0,0 +1,20 @@
+#!/bin/sh
+
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: 2021 The Elixir Team
+# SPDX-FileCopyrightText: 2012 Plataformatec
+
+set -e
+
+ELIXIR_VERSION=1.19.1
+
+if [ $# -eq 0 ] || { [ $# -eq 1 ] && { [ "$1" = "--help" ] || [ "$1" = "-h" ]; }; }; then
+  cat <<USAGE >&2
+Usage: $(basename "$0") [options] [.exs file] [data]
+
+## General options
+
+  -e "COMMAND"                 Evaluates the given command (*)
+  -h, --help                   Prints this message (standalone)
+  -r "FILE"                    Requires the given files/patterns (*)
+  -S SCRIPT                    Finds and executes the given script in \$PATH

syft/pkg/cataloger/binary/testdata/classifiers/snippets/elixir/1.19.1/linux-amd64/lib/elixir/ebin/elixir.app

@@ -0,0 +1,19 @@
+{application,elixir,
+    [{description,"elixir"},
+     {vsn,"1.19.1"},
+     {modules,
+         ['Elixir.Access','Elixir.Agent.Server','Elixir.Agent',
+          'Elixir.Application','Elixir.ArgumentError',
+          elixir_overridable,elixir_parser,elixir_quote,elixir_rewrite,
+          elixir_sup,elixir_tokenizer,elixir_utils,iex]},
+     {registered,[elixir_sup,elixir_config,elixir_code_server]},
+     {applications,[kernel,stdlib,compiler]},
+     {mod,{elixir,[]}},
+     {env,
+         [{ansi_syntax_colors,
+              [{atom,cyan},
+               {binary,default_color},
+               {operator,default_color}]},
+          {check_endianness,true},
+          {dbg_callback,{'Elixir.Macro',dbg,[]}},
+          {time_zone_database,'Elixir.Calendar.UTCOnlyTimeZoneDatabase'}]}]}.

syft/pkg/cataloger/binary/testdata/classifiers/snippets/erlang/27.0/linux-amd64/beam.smp

@@ -0,0 +1,9 @@
+name: beam.smp
+offset: 6378906
+length: 140
+snippetSha256: 186c882decc4160bfcfbdddaee08e8364675c795d3e85697422fbd7218452ee2
+fileSha256: f84f0ffc2e397abcf5f081b865e241017b1793c2ec38f5136bff4e5e21ffdad7
+
+### byte snippet to follow ###
+ied by configure */
+#define ERTS_EMU_CMDLINE_FLAGS " -fno-strict-aliasing -fno-common -g -O2 -I/usr/src/otp_src_27.0/erts/x86_64-pc-linux-mu

syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/1.15/any/VERSION

@@ -0,0 +1 @@
+go1.15 Fri 2003

syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/1.15/any/bin/go

@@ -0,0 +1 @@
+no version in this binary

syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/1.15w/any/VERSION

@@ -0,0 +1 @@
+go1.15 Fri 2003

syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/1.15w/any/bin/go.exe

@@ -0,0 +1 @@
+no version in this binary

syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/1.21/any/VERSION

@@ -0,0 +1 @@
+go1.21 Fri 2003

syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/1.21/any/go

@@ -0,0 +1 @@
+no version in this binary

syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/1.25/any/VERSION.cache

@@ -0,0 +1,2 @@
+devel go1.25-d524e1e Sun Feb 16 07:23:13 2025 -0800
+3devel go1.25-d524e1e Sun Feb 16 07:23:13 2025 -0800

syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/1.25/any/bin/go

@@ -0,0 +1 @@
+no version in this binary

syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/1.25w/any/VERSION

@@ -0,0 +1,2 @@
+devel go1.25-d524e1e Sun Feb 16 07:23:13 2025 -0800
+3devel go1.25-d524e1e Sun Feb 16 07:23:13 2025 -0800

syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/1.25w/any/go.exe

@@ -0,0 +1 @@
+no version in this binary