mirror of
https://github.com/anchore/syft.git
synced 2026-10-31 14:36:33 +01:00
Signed-off-by: Keith Zantow <keith.zantow@anchore.com>
This commit is contained in:
Keith Zantow
@@ -0,0 +1,94 @@
|
||||
# Recursive Archive Cataloging — Design
|
||||
|
||||
Status: **approved, in implementation**
|
||||
Branch: `feat/recursive-archive-combined`
|
||||
|
||||
## Background
|
||||
|
||||
Syft can scan inside Java archives (JAR/WAR/EAR, including shaded JARs) but ignores the
|
||||
contents of generic archives (`.tar`, `.tar.gz`/`.tgz`, `.zip`, `.apk`, `.gem`, `.iso`, …)
|
||||
found in a scanned directory or image. Tracked by issues
|
||||
[#246](https://github.com/anchore/syft/issues/246),
|
||||
[#1379](https://github.com/anchore/syft/issues/1379),
|
||||
[#2407](https://github.com/anchore/syft/issues/2407).
|
||||
|
||||
Two community PRs attempted this and are **superseded** by this design:
|
||||
|
||||
- **#4044** (@Rupikz) — recursion at the `fileresolver` layer.
|
||||
- **#4761** (@toabctl) — a composite resolver + orchestrator wired into `CreateSBOM`.
|
||||
|
||||
Both author's commits are preserved in this branch's history for credit; their
|
||||
approaches are replaced for the reasons below.
|
||||
|
||||
## Why the prior approaches were rejected
|
||||
|
||||
Search in syft *is* an index lookup. `FilesByPath`/`FilesByGlob`
|
||||
(`syft/internal/fileresolver/filetree_resolver.go`) call
|
||||
`SearchContext.SearchByPath(...)` and then `r.Index.Get(ref)`; a file that is not in
|
||||
*that resolver's own* `filetree.Index` causes the lookup to error and the file is
|
||||
**silently skipped**. A composite/wrapped resolver that merges archive contents without
|
||||
merging a real index therefore excludes files from indexing and breaks search. Pushing
|
||||
recursion into the directory resolver also smears archive contents into the parent's
|
||||
tree/chroot and forces a signature change on every `NewFromDirectory` caller.
|
||||
|
||||
## Core principle
|
||||
|
||||
**Each archive is treated as its own fully-indexed, standalone filesystem** — exactly how
|
||||
syft already treats container image layers. Extract the archive to a temp directory and
|
||||
build a resolver with `fileresolver.NewFromDirectory(tempDir, …)`, which produces a
|
||||
complete `Tree` + `Index` + `SearchContext`. No resolver merging, no shared index.
|
||||
|
||||
## Design decisions (locked)
|
||||
|
||||
1. **Detection is content-based** via `internal/file.IdentifyArchive` (catches `.tgz`,
|
||||
renamed archives, etc.).
|
||||
2. **A new `archive-cataloger` Task** (`internal/task/`) drives
|
||||
detect → extract → run the cataloger sub-pipeline → recurse. It is added to the
|
||||
package-cataloger task group, **before** relationship/unknowns tasks.
|
||||
3. **Provenance is expressed only via file-level `CONTAINS` relationships** in the first
|
||||
iteration: `From` = the archive file's `Coordinates`, `To` = each nested file/package.
|
||||
No changes to `Location`/`Coordinates`/metadata schema.
|
||||
4. **`Coordinates` is never modified** — it must remain a comparable two-string struct
|
||||
(used as `map[file.Coordinates]…` keys throughout `sbom.Artifacts` and as the
|
||||
`LocationData` key in `LocationSet`).
|
||||
5. **Node identity uses a per-archive `FileSystemID`** (an *existing* `Coordinates`
|
||||
field — no struct change). The chain rule is:
|
||||
|
||||
```
|
||||
childFSID = parentFSID == "" ? archiveAccessPath
|
||||
: parentFSID + "/" + archiveAccessPath
|
||||
```
|
||||
|
||||
- Uses the archive Location's **AccessPath**.
|
||||
- For non-image sources the root FSID is empty, so the chain starts with the relative
|
||||
archive path (e.g. `some/path/to.zip/some/nested/path.tar.gz`). Image sources start
|
||||
from the layer digest (e.g. `<layerDigest>/some/path/to.zip/…`).
|
||||
- The `FiletreeResolver` carries a `fileSystemID` and stamps it onto every returned
|
||||
`Location`, the same pattern image-layer resolvers already use. This keeps
|
||||
same-named files in different archives from colliding in the coordinate-keyed tables,
|
||||
with no post-processing in catalogers.
|
||||
6. **Java is left as-is in phase 1.** The generic archive cataloger recurses only into
|
||||
**non-JAR** types; detection excludes the formats the Java cataloger owns
|
||||
(`.jar/.war/.ear/.par/.sar/.nar/.jpi/.hpi/.kar/.far/.lpkg/.rar/.zap`). Unifying JARs
|
||||
into the generic mechanism is deferred.
|
||||
7. **Disabled by default.** Opt-in via `--archive-max-depth` / `SYFT_ARCHIVE_MAX_DEPTH`
|
||||
(default `0`), plus the extraction safety limits (max extraction size, file count,
|
||||
total bytes) carried over from the prior work.
|
||||
|
||||
## Reused from the prior PRs
|
||||
|
||||
- `internal/archive/extractor.go` — archive extractor + zip-bomb / size-limit guards.
|
||||
- `syft/cataloging/archive_search.go` — the `ArchiveSearchConfig` (depth + limits).
|
||||
- `syft/pkg/cataloger/java/parse_pom_xml.go` — skip `META-INF/maven/*/pom.xml` phantom
|
||||
packages.
|
||||
|
||||
## Implementation phases
|
||||
|
||||
| Phase | Work |
|
||||
|------|------|
|
||||
| 0 | Remove PR-4044's resolver recursion and PR-4761's composite/orchestrator; restore the 2-arg `NewFromDirectory`. Keep the items listed under "Reused" above. |
|
||||
| 1 | `FiletreeResolver.fileSystemID` + stamp on all Location-returning call sites; add `NewFromDirectoryWithFS(root, base, fileSystemID, …)` (existing `NewFromDirectory` delegates with `""`). |
|
||||
| 2 | `extractToResolver(...)` — extract to a temp dir and build a standalone indexed resolver with the computed `childFSID`. |
|
||||
| 3 | The `archive-cataloger` Task — detect (non-JAR) archives, run the package/file cataloger sub-pipeline into a scratch builder, merge into the shared builder, emit `CONTAINS` edges, and recurse to `MaxDepth`. |
|
||||
| 4 | Wire the task into `makeTaskGroups`; add `--archive-max-depth` (default 0) and limits. |
|
||||
| 5 | Tests: FSID stamping, extractor limits/zip-slip, nested-fixture FSID/relationship/dedup assertions, CLI integration. |
|
||||
Reference in New Issue
Block a user