* feat: add containers-storage support for Podman and Buildah, update tests and documentation
Signed-off-by: Bruce Clark <bruce.clark@mandg.com>
* test: update containers-storage tests to validate package.json and handle missing images
Signed-off-by: Bruce Clark <bruce.clark@mandg.com>
* test: fix rootless containers-storage test setup
Install uidmap for Podman and Buildah CI jobs.
Align temporary rootful and rootless storage paths.
Clean up rootless stores within the builder's user namespace.
Signed-off-by: Bruce Clark <bruce.clark@mandg.com>
* fix: only use containers-storage when explicitly requested
- containers-storage is no longer part of automatic image resolution; it's used only for `--from containers-storage`, the `containers-storage:` scheme, or `default-pull-source: containers-storage` (now an accepted value). Opening a store runs full graph driver init (mounts, locks, store writes), which is too invasive as a probe on every plain image reference.
- rootless stores must be read from inside the builder's user namespace (`podman unshare syft ...`), documented in the README.
- the `containers_image_openpgp` tag is now applied to unit/integration tests and lint, so dev tooling compiles the same provider that ships.
- containers-storage CI runs through `make containers-storage-test` and covers both vfs and overlay stores.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: exclude cgo btrfs driver and unmount overlay store in cleanup
- cgo-enabled linux builds (`-race` tests, lint) compile go.podman.io's btrfs graph driver, which needs libbtrfs headers. Add `exclude_graphdriver_btrfs` alongside `containers_image_openpgp` everywhere the tag is set; release builds are `CGO_ENABLED=0` and never included it.
- the overlay driver bind-mounts its graphroot inside the rootless user namespace, so the containers-storage test unmounts it before removing the store, and cleanup no longer fails the run.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Bruce Clark <bruce.clark@mandg.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Bruce Clark <bruce.clark@mandg.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
* ci: run the full unit suite on windows via make unit
Drops the curated `unit:windows` subset; the windows job now bootstraps like linux and runs `make unit`. Exceptions get added as failures show up.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* ci: skip fixture refresh for windows unit tests
binny has no windows release, so the bootstrap action can't install the tools the fixture refresh needs.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: skip linux container fixtures when docker can't run them
Windows CI runners only run windows containers, so image fixtures go through `testutils.SkipWithoutLinuxContainers`. No behavior change on linux/mac.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: close leaked file handles and posix squashfs paths on windows
- output writers are now closed on early returns in `scan`/`convert` (open handles block cleanup on windows)
- `newSBOMMultiWriter` used `path.Dir` on host paths, so output subdirectories were never created on windows
- squashfs walk paths were built with `filepath.Join`
- skip a few posix-only test cases (dir perms, xdg fallback, go-rpmdb ndb handle leak upstream)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix(golang): resolve go source module dir natively on windows
- `parseGoModFile` passed the resolver's posix (`/c/...`) path to the go toolchain, so go source resolution silently found nothing on windows
- skip container-built fixtures (go archs, java builds) and golden image tarballs on windows; stereoscope's layer cache filenames contain `:`, which windows rejects
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: keep posix paths posix on windows
- `ChrootContext.ToChrootPath` nested volumes (`D:\cwd\C:\root`) when the scan root was on another drive, so the prefix was never trimmed
- bitnami, python wheel/egg, and alpm parsers used `filepath` on posix location paths
- alpm backup-file lookups silently missed on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* ci: pull the test fixture cache for windows unit tests
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: don't map virtual file tree paths through the host on windows
`FiletreeResolver` without a chroot (snaps) ran request paths through `filepath.Abs`, rooting them on the current volume (`/payload.txt` -> `/d/payload.txt`), so snap scans on windows found nothing.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: match tar entry globs as posix on windows
- `matchesAnyGlob` used `doublestar.PathMatch` (host separator) on tar entry names, so tar-wrapped java archives silently produced nothing on windows
- test helpers now treat binary snippet keys and `FromFile` locations as posix
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* incorporate windows path fix
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* incorporate go-rpmdb close fix
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* close squashfs fixture writers before finalizing
- windows dir listings can report a zero size for files still open, so the manifest was packed empty
(also adds a temporary windows diagnostic for the binary cataloger)
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* temp: surface windows binary cataloger diagnostics
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: keep directory symlink targets posix on windows
- the directory indexer joined symlink targets with native separators, so links in the posix file tree pointed nowhere and globs on link names found nothing
- binary fixture logical keys used native separators, so nested snippets were picked as the fixture dir on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* keep addSymlinkToIndex under the funlen limit
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* drop windows skips covered by the stereoscope and go-rpmdb bumps
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: add windows unit test job and capture dir resolver path bug
- add a `Unit tests (Windows)` job running a new `make unit:windows` task, scoped to the packages that own host path handling (tests needing linux docker fixtures are skipped)
- `Unit tests` is now a gate job over the Linux and Windows unit jobs
- add a resolver test asserting that locations from `AllLocations` resolve again via `FilesByPath` and use posix paths. This is expected to fail on windows today, which is why file digests come back empty for `dir:` scans (#5325)
- fix windows-only test files that no longer compiled
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore: drop verbose output from integration tests
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: windows path handling outside of the file lookup bug
- chroot paths are always posix (`ToChrootPath` no longer returns backslash paths on windows), same for the directory source name derived from `base`
- `windows.FromPosix` no longer panics on input that isn't volume-encoded (e.g. relative or already native paths)
- skip tests on windows that rely on symlink fixtures, shell script fixtures, or emulate windows paths on a posix host
- disable `core.autocrlf` on the windows CI job so fixture digests match linux
This leaves only the `FilesByPath` lookup failures from #5325.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* test: skip posix-only fileresolver tests on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: split contained paths on posix separators on windows
- `allContainedPaths` cleaned with the native separator but split on `/`, so a root reached through a symlink never had its ancestors indexed on windows
- index assertions in tests now key on the posix path the index actually stores
- skip symlink fixture tests and a posix-only test helper on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* chore: revert unrelated go.sum churn and allContainedPaths change
- restore `go.sum` and `.make/go.sum` (accidentally populated by a local `-mod=mod` invocation)
- revert the `allContainedPaths` separator change: production passes native windows paths there, and splitting them on `/` produces drive-relative ancestors, so skip the posix-only test on windows instead
- annotate the remaining `filepath` usages that operate on native paths
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* fix: search the file tree with posix paths on windows (#5325)
The directory and file resolvers key their file tree on posix paths (`/c/some/path`), but `FilesByPath`, `HasPath` and root-anchored `FilesByGlob` searched it with the native path from the chroot context (`C:\some\path`). On windows every lookup silently missed, so anything that resolves files by path found nothing. This is why file digests were empty for `dir:` scans.
Requests are now converted to posix before searching the tree.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
---------
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
with a cold fixture cache the integration suite builds and saves 18 docker
images across 36 sequential tests, which walks past `go test`'s default 10m
timeout and takes the fixture cache rebuild down with it. that suite now runs
`go test` directly with `-timeout=30m` (gotest.Tasks() has no timeout option),
plus `-count=1` since the built fixtures are the side effect we're actually
after and a test cache hit would skip producing them.
also adds `RACE` as one switch for the race detector across every suite:
- `RACE=false make test` drops `-race` from unit + integration and skips the
race smoke, worth doing on a cache rebuild where the wall clock is all
docker builds anyway
- `RACE=true` forces it on locally
- unset behaves as before: on in CI, off locally and on windows
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
