Draining skipped entries still left a hole: a tar entry whose header doesn't read as a regular file
(a socket or directory per its mode bits, say) can carry a body, and the tar reader decompresses it
to reach the next header without anything being charged or ctx being checked.
- compressed tars are now read through a reader that charges the budget for every byte the decoder
produces and checks ctx per read, so no skip path inside the tar reader gets around it
- entry content from that stream isn't charged a second time, and the per-entry drain is gone
- zip and plain tar are unchanged
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
The tree decompression budget was only spent as entry content was stored. A compressed tar reader
still decompresses whatever an entry leaves unread to reach the next header, so an excluded entry, a
file/directory clash, or the rest of one over the 2 GB per-entry cap decompressed with nothing charged
and no ctx check. A small `.tar.zst` could force hundreds of GB of work past the budget.
- in compressed tars the rest of every regular entry is now drained through the budget, with ctx
checked per chunk
- zip and plain tar are unchanged: they seek past skipped entries, so those cost nothing
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
When an archive held two entries at one path, the resolver kept the first, and this was documented as
intended. Everything that actually unpacks the archive keeps the last: `tar` (and `tar -r` appends
updates), OCI layer application, and the legacy java path's `ContentsFromZip`. So a jar could carry a
`pom.properties` reporting 2.17.1 ahead of one installing 2.14.0, and the SBOM would show the former.
- a later regular file, link or other non-directory entry now replaces the earlier one, refunding its
in-memory bytes
- a file/directory clash at one path still keeps the first, since swapping kinds would mean
re-parenting what is already under it
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`LocalPath` searched `FullNestedPath()` for the last `!/`, which is just `RealPath` for a file not in
an archive. A host file like `/data/Hello!/world.txt` came back as `/world.txt`, so lookups and
exclusions ran against the wrong path even with nested cataloging off. `ContainerPath` had the same
problem when `FullPath` was set to `RealPath`.
Both now cut `RealPath` off first and only look for a separator in the entry chain after it, which
can't contain `!/` since entry names with it are refused.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Adding an entry checked `HasPath` on every prefix of its path, so a name with thousands of
components did O(depth^2) hashing per entry before anything was charged.
- entry names over 256 path components are refused and reported like other unsafe names
- implied parent directories are found walking from the deepest parent up, stopping at the first
one that already exists
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`entryName` only rejected NUL and over-long names, so whatever an archive called an entry flowed
straight into `FullPath`, the file tree and every encoder.
- a name containing `!/` could pose as a nested archive boundary and collide with a real nested entry
- control characters and unicode line separators could add lines to SPDX tag-value output
Both are now refused, and refused names (over-long ones included) mark the resolver truncated with
the quoted name instead of being dropped silently.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
SPDX wrote `FileName` from `RealPath` only, so two archives' `META-INF/MANIFEST.MF` became two SPDX files with the same name and nothing saying which archive each came from. Worse, a consumer reading just the name would take a nested file for one at that path in the scan root.
- a file inside an archive is now named by its full chain, using the same `:` convention and escaping as `ArchivePath` / java's `VirtualPath`: `app.war:WEB-INF/lib/a.jar:META-INF/MANIFEST.MF`
- `FileComment` gains an `archivePath: ...` line (next to the existing `layerID: ...`, one per line), so decoding recovers `ArchivePath` and the entry's own `RealPath` without guessing where the chain ends
- files outside archives are unchanged
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
the legacy recursion links a nested jar to the jar that holds it: a `DEPENDENCY_OF` edge from inner to outer, and `JavaArchive.Parent` on the inner package. Inside an extracted archive java called `parse(ctx, nil)`, so both went away as soon as the feature was on. What replaced them was `file CONTAINS package`, which SPDX and CycloneDX drop, so those formats lost all war-to-jar structure.
- `archive.Traversal` gains `Parent` and `Packages` (the packages found describing that archive, filled in by the task after its sub-pipeline runs)
- java picks the enclosing jar's main package from the parent traversal (the one whose virtual path is the archive's own; pom-derived aux packages carry a suffixed one) and passes it to `parse`, then sets `Parent` the same way the legacy path does
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
with `FilesOwnedByPackageSelection` the file catalogers take every package location in the SBOM and look it up by `RealPath` in the resolver they were given. Inside an archive, the java package's location is the archive itself *in its parent's frame* (`WEB-INF/lib/dep.jar`), so the sub-pipeline looked that path up inside `dep.jar`'s own resolver: a miss, or an unrelated same-named entry by coincidence. At the root, coordinates from inside archives were looked up against the scan root the same way.
`coordinatesForSelection` now keeps only coordinates whose `ArchivePath` matches the current frame (the traversal's `ContentsArchivePath()`, or blank at the root).
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
a read error while copying one entry (a zip CRC mismatch, a corrupt deflate stream) was returned from `add`. That aborted the walk, and `Extract` threw away every entry already stored. So a large war with one damaged, irrelevant entry produced zero packages with the feature on, while java's legacy path (which reads only the entries it needs) still cataloged it.
- a read error on one entry now drops that entry and marks the resolver truncated (`entry "x" could not be read: ...`), and the walk continues
- if the stream itself breaks partway (a truncated `.tar.gz`), whatever was read before the break is kept and the archive is truncated. With nothing read it is still an error, as before
- disk limit and decompression budget errors still stop the walk
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
the limiter bounds what's held *at once*, and each archive gives its bytes back when it's released. So a recursive bomb (42.zip shape) never trips a limit: every level fits on its own, and the scan just runs for effectively forever. Capping each archive at N× its own size wouldn't help either, since every level of such a bomb is high-ratio, so the work still grows about N-fold per level.
- each top-level archive gets one decompression budget of `max(100 × its size, 1 GiB)`, spent by every archive nested under it at any depth
- the budget is attached to the context when the walk enters a top-level archive, so `Extract` picks it up without a signature change
- once it's spent, the archive being extracted is truncated with a "possible decompression bomb" unknown and the walk moves on. The next top-level archive starts fresh
real trees (a war of jars, a layer of tarballs) expand a few times over, well under the ratio.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`put` had no per-entry cap, unlike the legacy java path which capped every file at 2 GB through `intFile.SafeCopy`. At depth 1, one entry of an overlapping-entry zip bomb could decompress all the way to the 100 GB disk limit before anything stopped it.
- entries are now read through the same 2 GB cap (`intFile.PerFileReadLimit`, now exported). An entry over it is dropped after cap+1 bytes and the rest of the archive is still extracted
- `Resolver` gains `TruncatedReason`, so the truncation unknown says what happened ("an entry larger than 2.0 GiB was skipped", or the existing disk-limit text) instead of always blaming the limits
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`identifyFormat` accepted any extractor mholt identified, which put the 7z and rar decoders (`bodgit/sevenzip`, `rardecode`) on untrusted input with no test coverage. Only zip, tar and compressed tar are opened now, until other formats are deliberately supported.
This also fixes java resource adapters: a `.rar` that is really a zip used to be identified by name as RAR and fail to extract. An unsupported identification now falls through to the zip end-of-central-directory check, so it's read as the zip it is.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
sub-pipeline catalogers were already run under `runTaskSafely`, but extraction wasn't: the zip, tar and compressor decoders run on untrusted bytes, and a panic in one unwound the whole walk and failed `CreateSBOM` with "failed to run tasks".
- `processArchive` recovers and returns the panic (with its stack) as an error at the archive's coordinates, so it becomes an unknown and the sibling archives are still cataloged
- `Extract` now defers `Cleanup` on failure instead of calling it on each error path, so a panic no longer leaks the limiter charge or the spill file
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
each `ArchivePath` segment came from `loc.Path()`, so it was the link path when an archive was reached through a symlink and the real path otherwise. That depended on how discovery happened to find the archive. The package location said `store/3f9a1c` while the files inside the same jar were recorded under `...:app/lib/json-simple-1.1.1.jar`, so a consumer couldn't join a package to its files by prefix.
The chain for extracted files is now built from `RealPath`. `Traversal.Location` still carries the access path, so java's name check and `VirtualPath` metadata behave the same as before. The field comment and schema now say every segment has links resolved.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
`:` was escaped as `%3A`, but `%` itself wasn't, so an entry named `a:b.jar` and one named `a%3Ab.jar` got the same `ArchivePath`. Their contents then collide in `FileMetadata` / `FileDigests` (last writer wins) and share artifact IDs.
`%` is now escaped as `%25` before `:`, which makes the encoding injective. The encoding is documented on `Coordinates.ArchivePath` and in the schema.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Archives may contain directory entries with read-only permissions
(e.g., META-INF/ with mode 0555 in JARs). When extracted to temp
directories, these restricted permissions prevented writing files
into those directories, causing "permission denied" errors for
subsequent file entries like META-INF/MANIFEST.MF.
Always use 0755 for extracted directories since they are temporary
working directories owned by the extraction process.
Signed-off-by: Thomas Bechtold <thomasbechtold@jpberlin.de>
Add integration tests covering the full archive extraction pipeline:
- tar.gz containing Python package metadata (single-level)
- zip containing a JAR with MANIFEST.MF (two-level, same format)
- Mixed archive types (zip + tar.gz at same level)
- Depth limit enforcement (3-level zip nesting, limited to 1)
- Extension exclusion filtering
- 3-level mixed format nesting: tar.gz -> zip -> tar.gz -> files
verifying cross-format extraction and full access path chains
- Reverse format crossing: tar.gz -> zip -> tar.gz verifying
content readability at all levels and correct relationship count
Signed-off-by: Thomas Bechtold <thomasbechtold@jpberlin.de>
Add tests for important orchestrator edge cases:
- Nested archives (zip-inside-zip): verifies multi-round extraction
discovers archives within previously-extracted archives
- Context cancellation: graceful handling without hanging
- Multiple archives at same level: both extracted in one round
- Corrupt archive: skipped without error, doesn't block other work
- Empty archive: not counted as extracted
- Per-archive file limit: partial extraction handled correctly
- Relationship structure: contains relationships emit the archive's
coordinates as the source side
Signed-off-by: Thomas Bechtold <thomasbechtold@jpberlin.de>
Implement the Orchestrator that manages the iterative
extract-catalog loop for recursive archive discovery:
- Scans the composite resolver for candidate archive locations,
detecting format by peeking 512 bytes via FileContentsByLocation
rather than copying every non-archive file to a probe-temp file
- Extracts identified archives to per-archive temp directories
- Builds child resolvers via an injectable ResolverFactory (kept
pluggable so tests can substitute a mock resolver without
importing syft/internal/fileresolver)
- Tracks extraction state to avoid reprocessing
- Enforces safety limits (depth, per-archive size, file count,
total extraction budget)
- Generates ContainsRelationship entries linking each archive to
its extracted-contents filesystem
- Manages temp directory lifecycle with Cleanup()
Signed-off-by: Thomas Bechtold <thomasbechtold@jpberlin.de>
Add tests for important edge cases:
- Nested children (child-in-child): verifies that archives within
archives produce correct multi-level AccessPath chains like
"/archive1.tar.gz:/inner-archive.zip:/deep-file.txt"
- Empty child resolver: no locations leak from empty archives
- Same file path in parent and child: both versions accessible
with distinct FileSystemID routing
- Context cancellation in AllLocations: no hanging
Signed-off-by: Thomas Bechtold <thomasbechtold@jpberlin.de>
Implement a file.Resolver decorator that composites a parent resolver
with child resolvers created from extracted archive contents. Files from
child resolvers appear transparently alongside parent files.
Key behaviors:
- FilesByPath/FilesByGlob/FilesByMIMEType delegate to parent AND children
- FileContentsByLocation routes to the correct resolver via FileSystemID
- Locations from children get archive-prefixed AccessPath (using ":"
separator matching Java cataloger convention) and unique FileSystemID
- Thread-safe via RWMutex for concurrent access
Signed-off-by: Thomas Bechtold <thomasbechtold@jpberlin.de>
Introduce the internal/archive package with a generic Extractor
interface for extracting archive contents to a destination directory.
Includes two implementations:
- ZipExtractor: handles zip-based archives (.zip, .jar, .war, etc.)
- TarExtractor: handles tar-based archives (.tar, .tar.gz, .tar.bz2, etc.)
Both extractors enforce configurable safety limits and protect against
common malicious-archive vectors:
- Per-archive extraction size limit
- Per-archive file count limit
- Zip-slip path traversal prevention via existing SafeJoin
- Symlink entries are written as real symlinks only when their target
resolves inside destDir; symlinks pointing outside are dropped to
avoid escape via symlink-then-write
- Non-regular non-symlink entries (devices, fifos, sockets) are skipped
because their contents can't safely round-trip to disk
Also provides FindExtractor for auto-detecting archive format, and
IsExcludedExtension for filtering by extension.
Signed-off-by: Thomas Bechtold <thomasbechtold@jpberlin.de>
Alex Goodman
Keith Zantow
Keith Zantow
Thomas Bechtold