Fix: alpm mtree gzip unbounded processing (#5292)

* fix(arch): bound the decompressed mtree listing

The ALPM cataloger handed a gzip stream straight to `mtree.ParseSpec`, which
materializes every entry before it returns any of them. Nothing capped the
decompressed size, so a crafted mtree in a scanned image expanded until the
process died. 64KB of input is enough to produce 64MB, and the ratio scales.

Reads through a limit now and rejects a listing that exceeds it. The read goes
one byte past the cap so that tripping it is distinguishable from a listing that
simply ends there, since silently truncating would produce a package missing
most of its files. The existing error path already surfaces this as an unknown.

The cap is well above any real package: an mtree names every file with its
digests, and even the largest packages land far under it.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix(arch): bound mtree entry count, not just decompressed bytes

The byte cap alone does not bound what the parse retains. go-mtree keeps an
entry per line and appends it unconditionally, blank lines included, so short
lines cost about 1KB of heap each.

Measured on the previous commit: 15KB of gzipped newlines expanding to 16MB,
comfortably inside the 64MB byte cap, drove 8.1GB of peak heap over 74 seconds
and returned no records and a nil error. Same input now rejects in 0.05s at
51MB peak.

Counting lines before parsing is what makes the entry allocation bounded, since
the parser materializes all of them before returning. Lines the parser skips
only make the count high, never low.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix(arch): apply the mtree bounds while streaming

Both caps were enforced against a fully buffered listing, so the peak cost
scaled with cataloger parallelism, which runs a goroutine per package. They now
apply as the stream flows: a limited reader for the byte cap and a counting
reader for the entry cap, which has to trip while the parser is still reading
since it materializes every entry before returning any of them.

Both limits are injectable so the boundary tests can be exact on both sides
without allocating their way up to the shipped 64MB. Each cap gets a listing
exactly at it, asserting the records parse whole rather than just that no error
came back, since quiet truncation is the failure a size check invites.

The byte limiter carries one byte of headroom, which is what makes an
overrunning listing distinguishable from one that simply ends at the cap.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix(arch): refuse mtree line continuations

go-mtree collapses a line ending in a backslash into the next one with a
quadratic string concat, so the cost is lines times bytes and neither the byte
cap nor the line cap can see it. 199k continued lines sitting inside both caps
measured 4m34s and 6.1TB of allocation churn, and returned no error.

Also re-derived both constants against a real package rather than a guess.
`texlive-fontsextra` is 105,847 files and 1,978 directories, so roughly 108k
lines at 150-250 bytes each. 16MB would have falsely rejected it, so the byte
cap stays at 64MB and the line cap goes 200k -> 300k, which is ~2.8x a real
package instead of 1.85x.

Rejections are now sentinels so tests can name which bound tripped, and the
constants carry the measured ceiling they imply: a crafted listing of minimal
fields still reaches ~1.1GB at the byte cap, since go-mtree allocates per
whitespace-delimited field. Bounding that further means bounding fields rather
than bytes, and legitimate input is already within ~1.3x of the crafted worst
case, so there is little to win.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix(arch): refuse a continuation spelled with a carriage return

The refusal matched the literal bytes backslash-newline, but go-mtree decides
"this is a continuation" from the scanner token, and `bufio.ScanLines` drops a
trailing carriage return before testing for the backslash suffix. So a listing
using CRLF ran the quadratic collapse with nothing watching.

Measured: 775KB of gzip expanding to 17MB across 150k lines, inside both other
bounds, took 1m33s and returned a nil error. Same input now rejects in 50ms.

The check is a single byte pass that counts newlines and looks for a backslash
before either line ending, which replaces the two scans and the straddle special
case it took to catch only one spelling. A tripped bound is also latched now:
`bufio.Scanner` stops at the first error, but nothing about the type said a
caller had to, and the refusal was recomputed per chunk.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test(arch): make the bombpkg fixture able to fail

The fixture was 310k bare newlines, which go-mtree turns into blank entries with
no name, and the parse loop already skips those. So the assertion that a rejected
listing leaves an empty file list held whether or not the bound existed. It is now
a repeated file line: 0 records with the bound, 310,000 without.

Dropped the two tests that asserted nothing as a result. The cataloger-level one
was a weaker restatement of what the golden comparison already pins exactly, and
the shipped-size-cap one drove a real 64MB listing through the parser to prove a
constant had not moved. Asserting the constants directly catches any change rather
than only a loosening.

Race-mode package runtime goes 15.7s -> 4.4s.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test(arch): make three mtree assertions able to fail

The truncated-gzip case asserted the failure was not misreported as the size
bound, but ran against the shipped 64MB cap on a 3.5KB listing, so the check it
guarded could never fire. It now runs against a cap just above the truncation
point, where the two genuinely race.

The bomb case asserted an allocation delta from `runtime.MemStats`, which is a
process-wide cumulative counter and measures the property indirectly. It now
counts bytes pulled through the reader, which is what "gave up while reading"
actually means and does not drift with GC or parallelism.

Added the one bound that had no test at all. A single line over 64KB is refused
by go-mtree's scanner, not by anything here, so a dependency bump that called
`scanner.Buffer` would remove it silently.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* docs(arch): correct the mtree bound comments and close the gzip reader

The comment said both bounds apply while the listing streams. Only the line cap
does. `io.LimitedReader` signals a drained budget with `io.EOF`, which the parser
reads as the end of a complete listing, so the byte cap is necessarily a check
after the fact. Peak cost is the same either way, since accepting a listing at
the cap costs what rejecting one just past it does, but the comment was claiming
a mechanism that isn't there.

Also trimmed the benchmark writeups down to the reasoning that outlives them.
The exact transient-heap and wall-clock figures are tied to a go-mtree version
with nothing to detect them going stale, and one of them described the CRLF case
as prevented when it wasn't. And `TestMtreeParse` was labelled the regression
guard for the continuation refusal, but its fixture has no backslashes in it.

`gzReader` now gets a `defer Close` to match the other gzip site in the tree.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix(arch): stop the alpm database parser panicking on a malformed backup line

A `%BACKUP%` entry is `path\tmd5`, and the parser indexed the second field
without checking the split produced one. A line with no tab panics. The database
is attacker-controlled, and while the task executor recovers, it recovers at task
granularity, so one crafted line drops every Arch package from the SBOM.

The same loop also never checked the scanner's error. A field over the 1MB token
cap ends the scan, and the entry came back populated up to that field and empty
after it, with a nil error saying it was fine. That is the failure the mtree
bounds in this branch exist to avoid, two functions up in the same file.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* docs(arch): trim the comments added by this branch

the comments were carrying reasoning that belongs in the PR description rather
than the code. Kept the parts a future reader cannot recover from the code (the
cap sizing, why continuations are refused, why the byte check runs after the
parse) and cut the rest.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test(arch): generate the mtree bomb fixture instead of committing it

the fixture is a synthetic listing fully described by a line count and a
repeated line, but it landed as a committed gzip member, which shows neither and
leaves no way to rebuild it when maxMtreeLines moves. the two tests that walk
testdata/installed write it first now, and git ignores it.

the real pacman listings (gmp, tree-sitter) stay committed since those are
captured output rather than something a rule can reproduce.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* correct and simplify comments

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
This commit is contained in:
Alex Goodman
2026-09-14 19:59:20 +00:00
committed by GitHub
parent e6d364437a
commit 30e9e65608
5 changed files with 440 additions and 3 deletions
+44
View File
@@ -1,10 +1,14 @@
package arch
import (
"bytes"
"context"
"io"
"os"
"testing"
"github.com/google/go-cmp/cmp/cmpopts"
"github.com/stretchr/testify/require"
"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/file"
@@ -12,7 +16,23 @@ import (
"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
)
const bombFixture = "testdata/installed/var/lib/pacman/local/bombpkg-1.0-1/mtree"
// writeBombFixture generates bombpkg's mtree: one line over maxMtreeLines, so the cataloger rejects
// it while walking testdata/installed. Generated rather than committed so the size that makes it a
// bomb tracks the constant instead of hiding in a gzip blob.
func writeBombFixture(t *testing.T) {
t.Helper()
listing := append([]byte("#mtree\n"), bytes.Repeat([]byte("./f\n"), maxMtreeLines+1)...)
compressed, err := io.ReadAll(gzipOf(t, listing))
require.NoError(t, err)
require.NoError(t, os.WriteFile(bombFixture, compressed, 0o600))
}
func TestAlpmUnknowns(t *testing.T) {
writeBombFixture(t)
pkgtest.NewCatalogTester().
FromDirectory(t, "testdata/installed").
WithCompareOptions(cmpopts.IgnoreFields(pkg.AlpmFileRecord{}, "Time")).
@@ -21,11 +41,14 @@ func TestAlpmUnknowns(t *testing.T) {
}
func TestAlpmCataloger(t *testing.T) {
writeBombFixture(t)
gmpDbLocation := file.NewLocation("var/lib/pacman/local/gmp-6.2.1-2/desc")
treeSitterDbLocation := file.NewLocation("var/lib/pacman/local/tree-sitter-0.22.6-1/desc")
emacsDbLocation := file.NewLocation("var/lib/pacman/local/emacs-29.3-3/desc")
fuzzyDbLocation := file.NewLocation("var/lib/pacman/local/fuzzy-1.2-3/desc")
madeupDbLocation := file.NewLocation("var/lib/pacman/local/madeup-20.30-4/desc")
bombDbLocation := file.NewLocation("var/lib/pacman/local/bombpkg-1.0-1/desc")
ctx := context.TODO()
treeSitterPkg := pkg.Package{
@@ -119,6 +142,26 @@ func TestAlpmCataloger(t *testing.T) {
},
}
// bombPkg's listing exceeds maxMtreeLines, so it emits with an empty file list, not dropped
bombPkg := pkg.Package{
Name: "bombpkg",
Version: "1.0-1",
Type: pkg.AlpmPkg,
FoundBy: "alpm-db-cataloger",
Locations: file.NewLocationSet(bombDbLocation),
Metadata: pkg.AlpmDBEntry{
BasePackage: "bombpkg",
Package: "bombpkg",
Version: "1.0-1",
Description: "Package whose mtree listing exceeds the parser's line bound",
Architecture: "x86_64",
Size: 1,
Reason: 1,
Files: []pkg.AlpmFileRecord{},
Backup: []pkg.AlpmFileRecord{},
},
}
gmpPkg := pkg.Package{
Name: "gmp",
Version: "6.2.1-2",
@@ -283,6 +326,7 @@ func TestAlpmCataloger(t *testing.T) {
fuzzyPkg,
madeupPkg,
gmpPkg,
bombPkg,
}
expectedRelationships := []artifact.Relationship{
+107 -3
View File
@@ -4,6 +4,7 @@ import (
"bufio"
"compress/gzip"
"context"
"errors"
"fmt"
"io"
"path"
@@ -16,6 +17,7 @@ import (
"github.com/vbatts/go-mtree"
"github.com/anchore/syft/internal"
intFile "github.com/anchore/syft/internal/file"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/internal/unknown"
"github.com/anchore/syft/syft/artifact"
@@ -216,6 +218,10 @@ func parseDatabase(b *bufio.Scanner) (*parsedData, error) {
var backup []map[string]any
for _, f := range strings.Split(value, "\n") {
fields := strings.SplitN(f, "\t", 2)
// without the tab there is no digest, and fields[1] would panic
if len(fields) < 2 {
continue
}
p := fmt.Sprintf("/%s", fields[0])
if ok := ignoredFiles[p]; !ok {
backup = append(backup, map[string]any{
@@ -241,6 +247,12 @@ func parseDatabase(b *bufio.Scanner) (*parsedData, error) {
}
}
// a field over the scanner's 1MB token cap ends the scan early, which would otherwise look
// like a complete package
if err := b.Err(); err != nil {
return nil, err
}
return parsePkgFiles(pkgFields)
}
@@ -277,15 +289,107 @@ func parsePkgFiles(pkgFields map[string]any) (*parsedData, error) {
return &entry, nil
}
// maxMtreeSize bounds the decompressed listing. texlive-fontsextra, the largest package Arch
// ships, runs 16 to 27MB.
//
// this caps bytes, not memory: go-mtree allocates per field, so a crafted listing can still cost
// about a gigabyte, times cataloger parallelism.
const maxMtreeSize = 64 * intFile.MB
// maxMtreeLines bounds entry count, which the byte cap does not: go-mtree keeps a ~128 byte entry
// per line, blanks included. 300k is ~2.8x the largest package Arch ships.
const maxMtreeLines = 300_000
var (
errMtreeTooLarge = errors.New("mtree file is larger than the max allowed size")
errTooManyMtreeLines = errors.New("mtree file has more lines than allowed")
// errMtreeLineContinued rejects a line ending in a backslash, which readers treat as "join with
// the next line". We refuse the file instead, for two reasons.
//
// cost: go-mtree joins with a quadratic string concat, so work grows as lines squared. 100k
// continued lines is a 1.2MB listing costing ~3.7s and ~52GB of churn. The bounds above are
// linear and never see it, and no line cap helps, since real packages reach ~107k lines.
//
// spec: mtree(5) says "Each line is interpreted independently" and documents no continuation,
// nor does Arch's ALPM-MTREEv2(5). Readers accept them anyway (BSD mtree via fparseln(3),
// libarchive in readline()), but no writer emits one: libarchive escapes a literal backslash as
// \134, and Arch builds .MTREE with it. So a continuation here is crafted, not a package.
errMtreeLineContinued = errors.New("mtree file uses line continuations")
)
// lineLimitedReader fails the read once the stream carries more than max newlines, or a line
// continuation. Counting while the bytes flow is what bounds entries: the parser builds one per
// line before returning any of them.
//
// continuations are refused rather than joined, unlike every other mtree reader. See
// errMtreeLineContinued. Both \n and \r\n count, since go-mtree strips a trailing carriage return
// before testing for the backslash.
//
// a tripped bound is latched, so reading past the first error does not come back clean.
type lineLimitedReader struct {
reader io.Reader
lines int
max int
lastByte byte
err error
}
func (l *lineLimitedReader) Read(p []byte) (int, error) {
if l.err != nil {
return 0, l.err
}
n, err := l.reader.Read(p)
// prev carries over the last byte of the previous read, so a backslash split from its line
// ending is still caught
prev := l.lastByte
for _, c := range p[:n] {
if (c == '\n' || c == '\r') && prev == '\\' {
l.err = errMtreeLineContinued
return n, l.err
}
if c == '\n' {
l.lines++
}
prev = c
}
l.lastByte = prev
if l.lines > l.max {
l.err = fmt.Errorf("%w (max %d)", errTooManyMtreeLines, l.max)
return n, l.err
}
return n, err
}
func parseMtree(r io.Reader) ([]pkg.AlpmFileRecord, error) {
var err error
return parseMtreeWithLimits(r, maxMtreeSize, maxMtreeLines)
}
// parseMtreeWithLimits lets the boundary tests use small exact limits instead of the shipped caps.
// maxSize must be below math.MaxInt64, since the limiter carries a byte of headroom.
func parseMtreeWithLimits(r io.Reader, maxSize int64, maxLines int) ([]pkg.AlpmFileRecord, error) {
var entries []pkg.AlpmFileRecord
r, err = gzip.NewReader(r)
gzReader, err := gzip.NewReader(r)
if err != nil {
return nil, err
}
specDh, err := mtree.ParseSpec(r)
defer internal.CloseAndLogError(gzReader, "mtree")
// the line cap trips mid-stream, the byte cap cannot: io.LimitedReader reports a drained budget
// as io.EOF, which the parser reads as a complete listing, so it is checked after the parse. The
// extra byte tells a drained budget from a listing ending exactly at the cap.
sizeLimited := &io.LimitedReader{R: gzReader, N: maxSize + 1}
specDh, err := mtree.ParseSpec(&lineLimitedReader{reader: sizeLimited, max: maxLines})
// before err: a drained budget means the parser saw only part of the listing
if sizeLimited.N <= 0 {
return nil, fmt.Errorf("%w (%d bytes)", errMtreeTooLarge, maxSize)
}
if err != nil {
return nil, err
}
@@ -2,13 +2,19 @@ package arch
import (
"bufio"
"bytes"
"compress/gzip"
"fmt"
"io"
"os"
"strings"
"testing"
"time"
"github.com/google/go-cmp/cmp"
"github.com/stretchr/testify/require"
intFile "github.com/anchore/syft/internal/file"
"github.com/anchore/syft/syft/file"
"github.com/anchore/syft/syft/pkg"
)
@@ -155,6 +161,26 @@ func TestDatabaseParser(t *testing.T) {
}
}
func Test_parseAlpmDBEntry_malformedDatabase(t *testing.T) {
t.Run("backup line with no tab", func(t *testing.T) {
// this used to panic, which drops every package rather than the one
entry, err := parseAlpmDBEntry(strings.NewReader("%NAME%\nfoo\n\n%BACKUP%\nnotabhere\n\n"))
require.NoError(t, err)
require.NotNil(t, entry)
require.Empty(t, entry.Backup)
})
t.Run("field over the scanner's token cap", func(t *testing.T) {
// everything after the oversize field is lost, so this has to error rather than look complete
db := "%NAME%\nfoo\n\n%DESC%\n" + strings.Repeat("x", 2*intFile.MB) + "\n\n%ARCH%\nx86_64\n\n"
_, err := parseAlpmDBEntry(strings.NewReader(db))
require.ErrorIs(t, err, bufio.ErrTooLong)
})
}
func parseTime(stime string) time.Time {
t, _ := time.Parse(time.RFC3339, stime)
return t
@@ -207,6 +233,7 @@ func TestMtreeParse(t *testing.T) {
reader := bufio.NewReader(f)
// a normal listing must not trip any of the bounds
entry, err := parseMtree(reader)
require.NoError(t, err)
@@ -217,3 +244,243 @@ func TestMtreeParse(t *testing.T) {
}
}
// mtreeSpec builds a valid listing of n files. Boundary tests need the parser to reach the end of
// it, which filler bytes never do.
func mtreeSpec(n int) []byte {
var buf bytes.Buffer
buf.WriteString("#mtree\n")
buf.WriteString("/set type=file uid=0 gid=0 mode=644\n")
for i := range n {
fmt.Fprintf(&buf, "./file%d time=1649595592.0 size=10 sha256digest=%064x\n", i, i)
}
return buf.Bytes()
}
func gzipOf(t *testing.T, data []byte) io.Reader {
t.Helper()
var buf bytes.Buffer
w := gzip.NewWriter(&buf)
_, err := w.Write(data)
require.NoError(t, err)
require.NoError(t, w.Close())
return bytes.NewReader(buf.Bytes())
}
// gzipOfRepeated builds a gzip member decompressing to n bytes of one repeated byte, so a few KB of
// input expands to whatever size the caller asks for.
func gzipOfRepeated(t *testing.T, payload byte, n int64) *bytes.Reader {
t.Helper()
var buf bytes.Buffer
w := gzip.NewWriter(&buf)
chunk := bytes.Repeat([]byte{payload}, 32*1024)
for remaining := n; remaining > 0; {
size := min(remaining, int64(len(chunk)))
written, err := w.Write(chunk[:size])
require.NoError(t, err)
remaining -= int64(written)
}
require.NoError(t, w.Close())
return bytes.NewReader(buf.Bytes())
}
func Test_parseMtree_boundsDecompressedSize(t *testing.T) {
// limits come from the fixture, so both sides of the boundary are exact
spec := mtreeSpec(50)
t.Run("a listing at the cap parses whole", func(t *testing.T) {
records, err := parseMtreeWithLimits(gzipOf(t, spec), int64(len(spec)), maxMtreeLines)
// records, not just no error: landing exactly on the cap must not truncate
require.NoError(t, err)
require.Len(t, records, 50)
require.Equal(t, "/file0", records[0].Path)
require.Equal(t, "/file49", records[49].Path)
})
t.Run("rejects a listing one byte past the cap", func(t *testing.T) {
_, err := parseMtreeWithLimits(gzipOf(t, spec), int64(len(spec))-1, maxMtreeLines)
require.ErrorIs(t, err, errMtreeTooLarge)
})
}
// Test_parseMtree_shippedConstants pins the production values. The boundary tests parameterize the
// limits, so they would pass unchanged if a constant regressed. Driving a real 64MB listing through
// the parser instead cost 12s of a 16s race run.
func Test_parseMtree_shippedConstants(t *testing.T) {
require.Equal(t, int64(64*intFile.MB), int64(maxMtreeSize))
require.Equal(t, 300_000, maxMtreeLines)
}
func Test_parseMtree_boundsLineCount(t *testing.T) {
// the byte cap does not bound memory: the parser keeps an entry per line, blanks included
spec := mtreeSpec(50)
lines := bytes.Count(spec, []byte("\n")) // the two header lines get entries of their own
t.Run("a listing at the cap parses whole", func(t *testing.T) {
records, err := parseMtreeWithLimits(gzipOf(t, spec), maxMtreeSize, lines)
require.NoError(t, err)
require.Len(t, records, 50)
})
t.Run("rejects a listing one line past the cap", func(t *testing.T) {
_, err := parseMtreeWithLimits(gzipOf(t, spec), maxMtreeSize, lines-1)
require.ErrorIs(t, err, errTooManyMtreeLines)
})
t.Run("rejects a bomb at the production limits without reading all of it", func(t *testing.T) {
// what the cap exists for: a few KB expanding past the line cap while inside the byte cap
bomb := gzipOfRepeated(t, '\n', 4*1024*1024)
require.Less(t, bomb.Size(), int64(64*1024), "payload should be small enough to be worth rejecting")
counted := &countingReader{reader: bomb}
_, err := parseMtree(counted)
require.ErrorIs(t, err, errTooManyMtreeLines)
// what matters is giving up mid-read, not after building the whole listing. Bytes consumed is
// the stable way to check that; an allocation delta drifts with GC and parallelism.
require.Less(t, counted.n, bomb.Size(),
"the whole bomb was read, so the cap tripped after the parse rather than during it")
})
}
// countingReader records how many bytes were pulled through it.
type countingReader struct {
reader io.Reader
n int64
}
func (c *countingReader) Read(p []byte) (int, error) {
n, err := c.reader.Read(p)
c.n += int64(n)
return n, err
}
// Test_parseMtree_sizeCapTakesPrecedenceOverLineCap pins which error wins when one read exceeds both
// bounds. No other test exceeds both at once, so a reorder would go unnoticed.
func Test_parseMtree_sizeCapTakesPrecedenceOverLineCap(t *testing.T) {
spec := mtreeSpec(50)
// maxLines=0 trips on the first newline; maxSize=20 is drained by that same read
_, err := parseMtreeWithLimits(gzipOf(t, spec), 20, 0)
require.ErrorIs(t, err, errMtreeTooLarge)
require.NotErrorIs(t, err, errTooManyMtreeLines)
}
func Test_parseMtree_rejectsLineContinuations(t *testing.T) {
// go-mtree strips a trailing carriage return before testing for the backslash, so CRLF continues
// a line too
for _, lineEnding := range []string{"\\\n", "\\\r\n"} {
t.Run(fmt.Sprintf("%q", lineEnding), func(t *testing.T) {
spec := mtreeSpec(5)
spec = bytes.Replace(spec, []byte("size=10 sha256digest="), []byte("size=10 "+lineEnding+"sha256digest="), 1)
_, err := parseMtree(gzipOf(t, spec))
require.ErrorIs(t, err, errMtreeLineContinued)
})
}
}
// scriptedReader replays a fixed sequence of Read results, driving the straddling case directly.
type scriptedReader struct {
chunks []string
}
func (r *scriptedReader) Read(p []byte) (int, error) {
if len(r.chunks) == 0 {
return 0, io.EOF
}
n := copy(p, r.chunks[0])
// keep what did not fit, so a short buffer still obeys io.Reader
if n < len(r.chunks[0]) {
r.chunks[0] = r.chunks[0][n:]
} else {
r.chunks = r.chunks[1:]
}
return n, nil
}
func Test_lineLimitedReader_rejectsStraddlingContinuation(t *testing.T) {
// the backslash and its line ending land in separate Read calls, which happens for real: the
// io.LimitedReader truncates the buffer and flate returns short reads.
tests := []struct {
name string
chunks []string
}{
{name: "lf", chunks: []string{"...\\", "\n..."}},
{name: "crlf", chunks: []string{"...\\", "\r\n..."}},
{name: "crlf split at the carriage return", chunks: []string{"...\\\r", "\n..."}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
lr := &lineLimitedReader{reader: &scriptedReader{chunks: tt.chunks}, max: 100}
var err error
buf := make([]byte, 16)
for err == nil {
_, err = lr.Read(buf)
}
require.ErrorIs(t, err, errMtreeLineContinued)
})
}
}
func Test_lineLimitedReader_latchesTheError(t *testing.T) {
// a tripped bound must not come back clean on the next read
lr := &lineLimitedReader{reader: &scriptedReader{chunks: []string{"a\\\nb", "clean"}}, max: 100}
buf := make([]byte, 16)
_, first := lr.Read(buf)
require.ErrorIs(t, first, errMtreeLineContinued)
n, second := lr.Read(buf)
require.ErrorIs(t, second, errMtreeLineContinued)
require.Zero(t, n)
}
func Test_parseMtree_malformedInput(t *testing.T) {
t.Run("not gzip", func(t *testing.T) {
_, err := parseMtree(bytes.NewReader(mtreeSpec(2)))
require.ErrorIs(t, err, gzip.ErrHeader)
})
t.Run("truncated gzip", func(t *testing.T) {
spec := mtreeSpec(50)
var buf bytes.Buffer
_, err := io.Copy(&buf, gzipOf(t, spec))
require.NoError(t, err)
// the cap sits just above what the truncated stream delivers, so the size check and the
// truncation actually race. The shipped 64MB cap could never fire here.
_, err = parseMtreeWithLimits(bytes.NewReader(buf.Bytes()[:buf.Len()/2]), int64(len(spec)), maxMtreeLines)
// must fail, but not as a size-bound trip
require.Error(t, err)
require.NotErrorIs(t, err, errMtreeTooLarge)
})
t.Run("single line over the scanner's token cap", func(t *testing.T) {
// the third bound is go-mtree's scanner, at the default 64KB token limit. Nothing else here
// would catch a dependency bump that calls scanner.Buffer.
spec := append([]byte("#mtree\n./big "), bytes.Repeat([]byte("a"), 128*1024)...)
_, err := parseMtree(gzipOf(t, append(spec, '\n')))
require.ErrorContains(t, err, "token too long")
})
}
@@ -0,0 +1,2 @@
# generated by writeBombFixture, see cataloger_test.go
/mtree
@@ -0,0 +1,20 @@
%NAME%
bombpkg
%VERSION%
1.0-1
%BASE%
bombpkg
%DESC%
Package whose mtree listing exceeds the parser's line bound
%ARCH%
x86_64
%SIZE%
1
%REASON%
1