Add nested archive cataloging (#5260)

Signed-off-by: Kudryavcev Nikolay <kydry.nikolau@gmail.com>
Signed-off-by: Thomas Bechtold <thomasbechtold@jpberlin.de>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Co-authored-by: Kudryavcev Nikolay <kydry.nikolau@gmail.com>
Co-authored-by: Thomas Bechtold <thomasbechtold@jpberlin.de>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>

internal/archive/README.md

@@ -0,0 +1,106 @@
+# internal/archive
+
+Opens one archive as its own filesystem so the ordinary catalogers can run against its contents.
+The archive cataloger task (`internal/task/archive_tasks.go`) drives the recursion; this package
+does everything for a single archive.
+
+## Flow
+
+```
+Extract(ctx, reader, archiveCoordinates, limiter, exclusions) -> *Resolver
+  acquire          read in place if the reader is already seekable, else hold in memory / spill to disk
+  identifyFormat   internal/file.IdentifyArchive first, end-of-central-directory fallback for stub-prefixed zips
+  Resolver.extract every entry -> entryHeader -> Resolver.add (excluded entries are never stored)
+  digestsOf        SHA-1 of the archive itself
+```
+
+`Extract` returns `nil` for content that is not an archive (a bare compression format such as a
+gzipped file that is not a tar counts as not an archive), `ErrDiskLimitReached` when the archive's
+own bytes cannot be placed, and otherwise a `Resolver` whose `Cleanup` releases everything.
+
+## Resolver
+
+One type does the work: `Resolver` is fed entries by `Extract`, holds their content, indexes them,
+and implements `file.Resolver` over the result. Each entry is a `node` carrying its header and a
+`blob`; the archive's own bytes, when they have to be held at all, are one more blob.
+
+Every location it returns extends the archive's coordinates: `RealPath` and `FileSystemID` stay
+those of the outermost archive on the scanned filesystem, and `FullPath` is the `!/`-delimited
+chain down to the entry (`/app.war!/WEB-INF/lib/dep.jar!/META-INF/MANIFEST.MF`). `LocalPath` is the
+entry's path within this archive.
+
+## Where bytes live
+
+Blobs are held in memory while the memory limit admits them and in one spill file per archive
+(`archive-spill-*`) once it does not; when a chunk is refused, everything held in memory moves to
+the file first. The file is created on first use, so most archives never touch the filesystem, and
+blobs are located in it by offset with no framing of their own. It goes under the scan's temp root
+when the context carries one (`internal/tmpdir`), otherwise the system temp directory.
+
+Anything already random-access (an `*os.File`, or a nested archive read out of its parent's
+resolver) is used in place and costs nothing, so only a top-level stream is ever copied. Its memory
+is refunded once it is extracted and digested; bytes it spilled wait for `Cleanup` with the entries.
+
+## Limits
+
+`Limits` bound what one scan holds *at once*; each falls as archives are released. For both limits,
+positive is the bound, zero forbids the resource, negative is unbounded.
+
+- `Limiter` is scan-wide; a `charge` is one archive's draw on it. Bytes are charged as they land,
+  never from declared sizes.
+- Memory refused → content moves to disk. Disk refused → the archive's own bytes cannot be placed
+  (`ErrDiskLimitReached`, archive skipped) or an entry cannot be stored (`Resolver.Truncated`, the
+  entries stored so far are still cataloged).
+- Index bookkeeping is approximated as `approxIndexBytesPerEntry` (2 KiB) per node, listed or
+  implied by an entry's path, plus name length. It lives in memory, so it is charged to the memory
+  limit, spilling held content to make room; only a zero memory limit lets it fall back to the disk
+  budget so that configuration still indexes. An entry whose index cost is refused is dropped like
+  one whose content is, and marks the resolver `Truncated`.
+- `Budget` bounds the bytes decompressed under one top-level archive, summed over everything nested
+  in it. Unlike `Limits` it never refills, so a recursive bomb whose every level fits the limits
+  still ends (`errBudgetSpent`, the resolver is `Truncated`).
+- `Limiter.Peak` reports the most held at once over the scan, for reporting.
+
+## Hostile archives
+
+Nothing an archive declares is trusted. Content is charged chunk by chunk as it decompresses, so a
+deflate or gzip bomb stops at the disk limit with the entry dropped whole. Every node is charged
+against the memory limit, including directories implied by deep paths, so an archive of millions of
+tiny entries or of a few entries with 4 KiB names stops at the memory limit. A recursive bomb stops
+at the `Budget`. Names longer than PATH_MAX are refused, and nesting stops at the task's `MaxDepth`.
+See the bomb tests in `extract_test.go`.
+
+## Lookup
+
+Entries are indexed in a stereoscope `filetree.FileTree` + `filetree.Index` and searched with
+`filetree.SearchContext`, the same index the directory and image resolvers use. A query may name an
+entry by its full path or by its `LocalPath` (leading slash optional); a full path into another
+archive in the scan names nothing. Directories implied by entry paths exist for `HasPath` but are
+never files. The last entry at a path wins over earlier
+duplicates, as it does on extraction; a file and a directory at one path keep the first.
+
+Links resolve inside the archive only, as the file tree resolves them: a hard link names its target
+from the archive root. A lookup matching a link and its target returns one location, with
+`AccessPath` the full path matched and `FullPath` that of the file read. Dangling and cyclic links are not
+files. `AllLocations` enumerates every non-directory path, links included. Answers are sorted by path.
+
+## Other pieces
+
+- `Exclusions`: the scan's exclusion patterns, of which only the `**/` shape means the same thing
+  re-rooted at an archive. Matched against each entry path and its parent directories at extraction
+  time. `syft.CreateSBOM` passes the source's patterns to the task, which builds one `Exclusions`
+  for the scan.
+- `Traversal`: put on the context by the task so a cataloger (java) knows which archive it is inside,
+  what its digests are, and which packages describe its parent. `VirtualPath(location)` converts a
+  location's `FullPath` to the java cataloger's colon-delimited `app.war:WEB-INF/lib/inner.jar`
+  form, escaping `%` and `:` in entry names. `Container` and `EntryCoordinates` split and build the
+  coordinates of files within archives (the SPDX and CycloneDX formats use them too); `OwnerContainers` and
+  `OwnedCoordinates` map package-owned file paths to coordinates in the right archive.
+- `WithNestedCataloging` / `NestedCatalogingEnabled`: `syft.CreateSBOM` marks the context when the
+  archive task is running, so the java cataloger leaves nested archives to the task instead of
+  opening them itself.
+- `entryHeader`: keeps regular files, directories and links only (no devices or fifos), cleans entry names to
+  archive-relative paths, refuses names longer than PATH_MAX or containing control characters, line
+  separators or the `!/` archive path separator (marking the resolver truncated), and drops header
+  fields nothing reads (PAX records, user and group names, sub-second and access times) since the
+  header is held for the life of the archive.