* feat: add containers-storage support for Podman and Buildah, update tests and documentation
Signed-off-by: Bruce Clark <bruce.clark@mandg.com>
* test: update containers-storage tests to validate package.json and handle missing images
Signed-off-by: Bruce Clark <bruce.clark@mandg.com>
* test: fix rootless containers-storage test setup
Install uidmap for Podman and Buildah CI jobs.
Align temporary rootful and rootless storage paths.
Clean up rootless stores within the builder's user namespace.
Signed-off-by: Bruce Clark <bruce.clark@mandg.com>
* fix: only use containers-storage when explicitly requested
- containers-storage is no longer part of automatic image resolution; it's used only for `--from containers-storage`, the `containers-storage:` scheme, or `default-pull-source: containers-storage` (now an accepted value). Opening a store runs full graph driver init (mounts, locks, store writes), which is too invasive as a probe on every plain image reference.
- rootless stores must be read from inside the builder's user namespace (`podman unshare syft ...`), documented in the README.
- the `containers_image_openpgp` tag is now applied to unit/integration tests and lint, so dev tooling compiles the same provider that ships.
- containers-storage CI runs through `make containers-storage-test` and covers both vfs and overlay stores.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: exclude cgo btrfs driver and unmount overlay store in cleanup
- cgo-enabled linux builds (`-race` tests, lint) compile go.podman.io's btrfs graph driver, which needs libbtrfs headers. Add `exclude_graphdriver_btrfs` alongside `containers_image_openpgp` everywhere the tag is set; release builds are `CGO_ENABLED=0` and never included it.
- the overlay driver bind-mounts its graphroot inside the rootless user namespace, so the containers-storage test unmounts it before removing the store, and cleanup no longer fails the run.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Bruce Clark <bruce.clark@mandg.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Bruce Clark <bruce.clark@mandg.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* ci: run the full unit suite on windows via make unit
Drops the curated `unit:windows` subset; the windows job now bootstraps like linux and runs `make unit`. Exceptions get added as failures show up.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* ci: skip fixture refresh for windows unit tests
binny has no windows release, so the bootstrap action can't install the tools the fixture refresh needs.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: skip linux container fixtures when docker can't run them
Windows CI runners only run windows containers, so image fixtures go through `testutils.SkipWithoutLinuxContainers`. No behavior change on linux/mac.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: close leaked file handles and posix squashfs paths on windows
- output writers are now closed on early returns in `scan`/`convert` (open handles block cleanup on windows)
- `newSBOMMultiWriter` used `path.Dir` on host paths, so output subdirectories were never created on windows
- squashfs walk paths were built with `filepath.Join`
- skip a few posix-only test cases (dir perms, xdg fallback, go-rpmdb ndb handle leak upstream)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(golang): resolve go source module dir natively on windows
- `parseGoModFile` passed the resolver's posix (`/c/...`) path to the go toolchain, so go source resolution silently found nothing on windows
- skip container-built fixtures (go archs, java builds) and golden image tarballs on windows; stereoscope's layer cache filenames contain `:`, which windows rejects
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: keep posix paths posix on windows
- `ChrootContext.ToChrootPath` nested volumes (`D:\cwd\C:\root`) when the scan root was on another drive, so the prefix was never trimmed
- bitnami, python wheel/egg, and alpm parsers used `filepath` on posix location paths
- alpm backup-file lookups silently missed on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* ci: pull the test fixture cache for windows unit tests
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: don't map virtual file tree paths through the host on windows
`FiletreeResolver` without a chroot (snaps) ran request paths through `filepath.Abs`, rooting them on the current volume (`/payload.txt` -> `/d/payload.txt`), so snap scans on windows found nothing.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: match tar entry globs as posix on windows
- `matchesAnyGlob` used `doublestar.PathMatch` (host separator) on tar entry names, so tar-wrapped java archives silently produced nothing on windows
- test helpers now treat binary snippet keys and `FromFile` locations as posix
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* incorporate windows path fix
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* incorporate go-rpmdb close fix
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* close squashfs fixture writers before finalizing
- windows dir listings can report a zero size for files still open, so the manifest was packed empty
(also adds a temporary windows diagnostic for the binary cataloger)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* temp: surface windows binary cataloger diagnostics
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: keep directory symlink targets posix on windows
- the directory indexer joined symlink targets with native separators, so links in the posix file tree pointed nowhere and globs on link names found nothing
- binary fixture logical keys used native separators, so nested snippets were picked as the fixture dir on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* keep addSymlinkToIndex under the funlen limit
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* drop windows skips covered by the stereoscope and go-rpmdb bumps
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: revert hashstructure to v0.6.0 to keep artifact IDs stable
hashstructure v1 changed `SlicesAsSets` hashing (every slice now gets a finalizing hash pass, and duplicate elements are deduplicated), which changed nearly every artifact ID syft produces.
- pin `github.com/gohugoio/hashstructure` back to v0.6.0 and have dependabot ignore major bumps
- make the aggregate `Unit tests` job fail when a dependency fails; previously it was skipped, which satisfies the required check and let #5277 auto-merge with failing unit tests
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* ci: fail unit test gate on any failed, cancelled, or skipped dependency
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* ci: require every unit test dependency to succeed
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* ci: drop gen-5 instance families from the test runner
m5a runners run the unit suite ~2.3x slower than m8id (15m vs 6.5m for `go test`), and when spot capacity for newer families dries up runs-on falls back to them. Keeping gen-6 through gen-8 leaves a wide spot pool without the slow tail.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: shrink decompression bomb fixtures
The bomb and heap-bound tests pushed 100MB-16GB through gzip and parsers, which under `-race` made them the slowest tests in the unit suite. Each now uses the smallest fixture that still separates the bounded path from the unbounded one.
- arch mtree caps are measured at a small cap and scaled to the shipped constants, so raising a cap still fails the budget
- snap and debian bombs expand to 32MB instead of 512MB (4x their 8MB budgets)
- golang ELF bombs still declare 256MB but deliver 64MB
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: add windows unit test job and capture dir resolver path bug
- add a `Unit tests (Windows)` job running a new `make unit:windows` task, scoped to the packages that own host path handling (tests needing linux docker fixtures are skipped)
- `Unit tests` is now a gate job over the Linux and Windows unit jobs
- add a resolver test asserting that locations from `AllLocations` resolve again via `FilesByPath` and use posix paths. This is expected to fail on windows today, which is why file digests come back empty for `dir:` scans (#5325)
- fix windows-only test files that no longer compiled
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore: drop verbose output from integration tests
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: windows path handling outside of the file lookup bug
- chroot paths are always posix (`ToChrootPath` no longer returns backslash paths on windows), same for the directory source name derived from `base`
- `windows.FromPosix` no longer panics on input that isn't volume-encoded (e.g. relative or already native paths)
- skip tests on windows that rely on symlink fixtures, shell script fixtures, or emulate windows paths on a posix host
- disable `core.autocrlf` on the windows CI job so fixture digests match linux
This leaves only the `FilesByPath` lookup failures from #5325.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: skip posix-only fileresolver tests on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: split contained paths on posix separators on windows
- `allContainedPaths` cleaned with the native separator but split on `/`, so a root reached through a symlink never had its ancestors indexed on windows
- index assertions in tests now key on the posix path the index actually stores
- skip symlink fixture tests and a posix-only test helper on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore: revert unrelated go.sum churn and allContainedPaths change
- restore `go.sum` and `.make/go.sum` (accidentally populated by a local `-mod=mod` invocation)
- revert the `allContainedPaths` separator change: production passes native windows paths there, and splitting them on `/` produces drive-relative ancestors, so skip the posix-only test on windows instead
- annotate the remaining `filepath` usages that operate on native paths
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: search the file tree with posix paths on windows (#5325)
The directory and file resolvers key their file tree on posix paths (`/c/some/path`), but `FilesByPath`, `HasPath` and root-anchored `FilesByGlob` searched it with the native path from the chroot context (`C:\some\path`). On windows every lookup silently missed, so anything that resolves files by path found nothing. This is why file digests were empty for `dir:` scans.
Requests are now converted to posix before searching the tree.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix oom when testing
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* right-size CI test runners
the `r` family and `ram=32+128` were sized for the tmpfs footprint rather
than the workload. with the caches back on disk the build+test peak fits in
16GB, so move to `m`/`c` at `ram=16+32`.
- ~27% cheaper per instance-hour on spot, and a much deeper pool to schedule
against (~20 x86 types vs 7)
- `cpu=4+8` with `ram=16+32` lets price-capacity-optimized opportunistically
land an 8 vCPU box when it's cheap
- plain `volume=100gb` replaces the provisioned throughput. a 4 vCPU instance
caps near 150MB/s of EBS and gp3's free baseline is already 125MB/s, so the
override bought nothing
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* Rework release workflow to canonical shape
Replace the custom quality-gate job with the reusable check-version-available
and check-gate workflows from anchore/workflows. Remove the phase
workflow_dispatch input; the install-script-only path is now a standalone
workflow (release-install-script.yaml) that can be triggered independently.
- add version-available and check-gate jobs using pinned anchore/workflows SHA
- remove phase input and quality-gate job
- release job now needs [check-gate, version-available]
- release-install-script job no longer conditionally skips based on phase
- add release-install-script.yaml for standalone install script runs
- set permissions: {} at workflow level (contents pushed to release job)
- add concurrency: group: release
Signed-off-by: wagoodman <wagoodman@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* Tighten workflow-level permissions to {}
Change top-level permissions from contents: read to {} in validations.yaml
and validate-github-actions.yaml, pushing the needed contents: read down
to each job that performs a checkout.
Signed-off-by: wagoodman <wagoodman@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* keep install script phase, remove workflow
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* remove schema detection workflow
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: wagoodman <wagoodman@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* migrate fixtures to testdata
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: correct broken symlinks after testdata migration
The migration from test-fixtures to testdata broke several symlinks:
- elf-test-fixtures symlinks pointed to old test-fixtures paths
- elf-test-fixtures needed to be renamed to elf-testdata
- image-pkg-coverage symlink pointed to test-fixtures instead of testdata
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: handle missing classifiers/bin directory in Makefile
The clean-fingerprint target was failing when classifiers/bin doesn't
exist (e.g., on fresh clone without downloaded binaries).
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: add gitignore negation for jar/zip fixtures in test/cli
The jar and zip files in test/cli/testdata/image-unknowns were being
gitignored by the root .gitignore patterns. This caused them to be
untracked and not included when building docker images in CI, resulting
in Test_Unknowns failures since the test expects errors from corrupt
archive files that weren't present.
Add a .gitignore in test/cli/testdata to negate the exclusions for
these specific test fixture files.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* switch fixture cache to v2
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: update expected versions for rebuilt fixtures
Update test expectations for packages that have been updated in
upstream repositories when docker images are rebuilt:
- glibc: 2.42-r4 → 2.43-r1 (wolfi)
- php: 8.2.29 → 8.2.30 (ubuntu/apache)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* upgrade go
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: add go-shlex dependency for testdata manager tool
The manager tool in syft/pkg/cataloger/binary/testdata/ imports
go-shlex, but since it's in a testdata directory, Go doesn't track
its dependencies. This caused CI failures when go.mod didn't
explicitly list the dependency.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* refactor: move binary classifier manager to internal/
Move the manager tool from testdata/manager to internal/manager so
that Go properly tracks its dependencies. Code in testdata directories
is ignored by Go for dependency tracking, which caused CI failures
when go.mod didn't explicitly list transitive dependencies.
This is a cleaner solution than manually adding dependencies to go.mod
for code that happens to live in testdata.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: add gitignore negations for test fixtures blocked by root patterns
Multiple test fixtures were being blocked by root-level gitignore patterns
like bin/, *.jar, *.tar, and *.exe. This adds targeted .gitignore files with
negation patterns to allow these specific test fixtures to be tracked:
- syft/linux/testdata/os/busybox/bin/busybox (blocked by bin/)
- syft/pkg/cataloger/java/testdata/corrupt/example.{jar,tar} (blocked by *.jar, *.tar)
- syft/pkg/cataloger/binary/testdata/classifiers/snippets/go-version-hint/**/bin/go (blocked by bin/)
- syft/pkg/cataloger/bitnami/testdata/no-rel/.../bin/redis-server (blocked by bin/)
Also updates the bitnami test expectation to include the newly required
.gitignore files in the test fixture.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: update glibc version expectation (2.43-r1 -> 2.43-r2)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* add capability drift check as unit step
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* dont clear test observations before drift detection
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* bump stereoscope commit to main
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
dependabot[bot]
Christopher Angelo Phillips
VictorHuu
Michael BrileyandAlex Goodman