oss/syft
2
0
Fork 0
mirror of https://github.com/anchore/syft.git synced 2026-10-11 21:57:22 +02:00
Code Issues Packages Projects Releases Wiki Activity

Fix slow pe tests (#5285)

Browse Source 
* test(dotnet): speed up pe fixture tests

the pe package took ~90s under -race, nearly all of it loading docker archive
fixtures. two things:

the three bundle fixtures published with `SelfContained=true`, so each `hello.exe`
carried a full .NET runtime (~60MB, 33MB tar). the bundle format is what those
fixtures exercise and it is identical framework-dependent, so they now publish
with `SelfContained=false` (160KB exe, 2MB tar). the race detector multiplies the
layer inflate + diffID hashing by ~15x, which is why the size mattered so much.

`fixtureFile` also loaded the image and built a squashed resolver per table row,
so `image-net8-app` was loaded four times for four files in it. resolvers are now
built once per distinct fixture from the parent test.

87s -> 28.5s under -race, and ~60MB less fixture cache.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix(dotnet): read the right file type from V1 single-file bundles

V1 bundles (.NET Core 3.x) predate the `unknown` member at the head of the
manifest file type enum, so their types are all one less than V2+: deps.json is
2, runtimeconfig.json is 3. we were matching on the V2+ value, which silently
matched runtimeconfig.json instead, so every .NET Core 3.x single-file app
reported its runtimeconfig as the embedded deps.json and its dependencies went
unseen.

the existing fixture test asserted the buggy output (it expected
`runtimeOptions`, which only ever appears in a runtimeconfig). the assertions
now pin the framework moniker out of each fixture's real deps.json, so reading
the neighboring file can't pass.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test(dotnet): drop the 64MB jruby binary from the net8-app fixture

the `image-net8-app` fixture wget'd a 64MB jruby release binary at build time to
serve as "a PE that is not a .NET binary", which was 64MB of the image's 65MB.
every test that loads that image pays for it, in both the dotnet cataloger and
the pe package, and it sits in the fixture cache twice.

the same jruby binary already has its own fixture in the binary cataloger
(`image-jruby`), asserted through the same `pe.Read`, so the coverage is not
lost. the negative case here now uses `/app/dotnetapp.exe` -- the native apphost
that is already in the image -- which pairs nicely with the single file case,
same apphost with the app bundled in and CLR evidence present.

also drops a github release download from the fixture build.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
This commit is contained in:
Alex Goodman
2026-09-11 15:27:20 -04:00
committed by GitHub
parent eb58f698bd
commit c4800b2393
7 changed files with 97 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
syft/pkg/cataloger/dotnet/testdata/image-net8-app/Dockerfile
Vendored
-3
View File
@@ -72,9 +72,6 @@ RUN dotnet publish -r $RUNTIME --no-restore -o /app
# > Humanizer.Core.zh-Hans 2.14.1
# > Humanizer.Core.zh-Hant 2.14.1
# lets pull in a file that is not related at all and in fact is not a .NET binary either (this should be ignored)
RUN wget -O /app/jruby_windows_9_3_15_0.exe https://github.com/jruby/jruby/releases/download/9.3.15.0/jruby_windows_9_3_15_0.exe
FROM busybox
WORKDIR /app
COPY --from=build /app .
syft/pkg/cataloger/internal/dotnet/bundle/bundle.go
+16 -2
View File
@@ -31,7 +31,9 @@ type dotNetBundleHeaderV2 struct {
Flags uint64
}
// dotNetFileType represents the type of bundled file in the manifest
// dotNetFileType represents the type of bundled file in the manifest, as of V2 bundles (.NET 5+).
// note: V1 bundles (.NET Core 3.x) predate the unknown member at the head of this enum, so every type there is
// one less than the values below (e.g. deps.json is 2, not 3). see depsJSONFileType().
type dotNetFileType uint8
const (
@@ -121,8 +123,20 @@ func readDepsJSONAtOffset(r io.ReadSeeker, offset, size int64) (string, error) {
return string(data), nil
}
// depsJSONFileType returns the manifest file type that marks deps.json for the given bundle version. V1 bundles
// (.NET Core 3.x) have no unknown member in the file type enum, so all of their types are shifted down by one --
// reading them with the V2+ values silently matches runtimeconfig.json instead.
func depsJSONFileType(majorVersion uint32) dotNetFileType {
if majorVersion < 2 {
return dotNetFileTypeDepsJSON - 1
}
return dotNetFileTypeDepsJSON
}
// findDepsJSONInManifest parses manifest entries to find deps.json (for V1 bundles or fallback)
func findDepsJSONInManifest(r io.ReadSeeker, numFiles int32, majorVersion uint32) (string, error) {
depsJSONType := depsJSONFileType(majorVersion)
for i := int32(0); i < numFiles; i++ {
var offset, size int64
@@ -151,7 +165,7 @@ func findDepsJSONInManifest(r io.ReadSeeker, numFiles int32, majorVersion uint32
return "", err
}
if fileType == dotNetFileTypeDepsJSON && size > 0 {
if fileType == depsJSONType && size > 0 {
// save current position to resume manifest parsing if needed
currentPos, err := r.Seek(0, io.SeekCurrent)
if err != nil {
syft/pkg/cataloger/internal/dotnet/pe/bundle_test.go
+26 -17
View File
@@ -13,34 +13,43 @@ func Test_extractDepsJSONFromBundle_Versions(t *testing.T) {
fixture string
path string
wantDepsJSON bool // true if deps.json should be found
wantJSONContain string // string that should be in the JSON (varies by .NET version)
wantJSONContain string // a string unique to the deps.json (and not the runtimeconfig.json) of this fixture
}{
{
name: "V1 bundle (.NET Core 3.1)",
fixture: "image-dotnet31-single-file",
path: "/app/hello.exe",
wantDepsJSON: true,
wantJSONContain: "runtimeOptions", // .NET Core 3.1 uses runtimeOptions
name: "V1 bundle (.NET Core 3.1)",
fixture: "image-dotnet31-single-file",
path: "/app/hello.exe",
wantDepsJSON: true,
wantJSONContain: `"runtimeTarget": {
"name": ".NETCoreApp,Version=v3.1/win-x64"`,
},
{
name: "V2 bundle (.NET 5)",
fixture: "image-dotnet5-single-file",
path: "/app/hello.exe",
wantDepsJSON: true,
wantJSONContain: "runtimeTarget", // .NET 5+ uses runtimeTarget
name: "V2 bundle (.NET 5)",
fixture: "image-dotnet5-single-file",
path: "/app/hello.exe",
wantDepsJSON: true,
wantJSONContain: `"runtimeTarget": {
"name": ".NETCoreApp,Version=v5.0/win-x64"`,
},
{
name: "V6 bundle (.NET 6)",
fixture: "image-dotnet6-single-file",
path: "/app/hello.exe",
wantDepsJSON: true,
wantJSONContain: "runtimeTarget", // .NET 6+ uses runtimeTarget
name: "V6 bundle (.NET 6)",
fixture: "image-dotnet6-single-file",
path: "/app/hello.exe",
wantDepsJSON: true,
wantJSONContain: `"runtimeTarget": {
"name": ".NETCoreApp,Version=v6.0/win-x64"`,
},
}
var fixtures []string
for _, tt := range tests {
fixtures = append(fixtures, tt.fixture)
}
resolvers := fixtureResolvers(t, fixtures...)
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
reader := fixtureFile(t, tt.fixture, tt.path)
reader := fixtureFile(t, resolvers[tt.fixture], tt.path)
defer reader.Close()
got, err := Read(reader)
syft/pkg/cataloger/internal/dotnet/pe/pe_test.go
+43 -18
View File
@@ -86,19 +86,23 @@ func Test_Read_DotNetDetection(t *testing.T) {
wantErr: require.NoError,
},
{
name: "jruby",
path: "/app/jruby_windows_9_3_15_0.exe",
// the apphost is a native launcher, not a managed assembly, so there is no CLR evidence to find in it.
// note the contrast with the single file deployment case below, which is the same apphost with the app
// (and the CLRDEBUGINFO resource) bundled into it.
name: "apphost",
path: "/app/dotnetapp.exe",
fixture: "image-net8-app",
wantCLR: false, // important!
wantVR: map[string]string{
"CompanyName": "JRuby Dev Team",
"FileDescription": "JRuby",
"FileVersion": "9.3.15.0",
"InternalName": "jruby",
"LegalCopyright": "JRuby Dev Team",
"OriginalFilename": "jruby_windows-x32_9_3_15_0.exe",
"ProductName": "JRuby",
"ProductVersion": "9.3.15.0",
"CompanyName": "dotnetapp",
"FileDescription": "dotnetapp",
"FileVersion": "1.0.0.0",
"InternalName": "dotnetapp.dll",
"LegalCopyright": " ",
"OriginalFilename": "dotnetapp.dll",
"ProductName": "dotnetapp",
"ProductVersion": "1.0.0",
"Assembly Version": "1.0.0.0",
},
wantErr: require.NoError,
},
@@ -125,13 +129,19 @@ func Test_Read_DotNetDetection(t *testing.T) {
},
}
var fixtures []string
for _, tt := range tests {
fixtures = append(fixtures, tt.fixture)
}
resolvers := fixtureResolvers(t, fixtures...)
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if tt.wantErr == nil {
tt.wantErr = require.NoError
}
reader := fixtureFile(t, tt.fixture, tt.path)
reader := fixtureFile(t, resolvers[tt.fixture], tt.path)
got, err := Read(reader)
tt.wantErr(t, err)
@@ -153,16 +163,31 @@ func Test_Read_DotNetDetection(t *testing.T) {
}
}
func fixtureFile(t *testing.T, fixture, path string) file.LocationReadCloser {
img := imagetest.GetFixtureImage(t, "docker-archive", fixture)
// fixtureResolvers loads each distinct image fixture once, from the parent test, so that subtests reading multiple
// files out of the same image don't each pay for loading the docker archive again (which dominates the runtime of
// these tests, especially under -race).
func fixtureResolvers(t *testing.T, fixtures ...string) map[string]file.Resolver {
resolvers := make(map[string]file.Resolver)
for _, fixture := range fixtures {
if _, ok := resolvers[fixture]; ok {
continue
}
s := stereoscopesource.New(img, stereoscopesource.ImageConfig{
Reference: fixture,
})
img := imagetest.GetFixtureImage(t, "docker-archive", fixture)
r, err := s.FileResolver(source.SquashedScope)
require.NoError(t, err)
s := stereoscopesource.New(img, stereoscopesource.ImageConfig{
Reference: fixture,
})
r, err := s.FileResolver(source.SquashedScope)
require.NoError(t, err)
resolvers[fixture] = r
}
return resolvers
}
func fixtureFile(t *testing.T, r file.Resolver, path string) file.LocationReadCloser {
locs, err := r.FilesByPath(path)
require.NoError(t, err)
syft/pkg/cataloger/internal/dotnet/pe/testdata/image-dotnet31-single-file/Dockerfile
Vendored
+4 -1
View File
@@ -1,9 +1,12 @@
# note: this is published framework-dependent on purpose: the bundle format (what these fixtures exercise) is
# identical either way, but bundling the runtime makes the fixture ~60MB instead of ~160KB, which dominates
# the runtime of the tests that load this image.
FROM --platform=linux/amd64 mcr.microsoft.com/dotnet/core/sdk:3.1 AS build
WORKDIR /src
COPY src .
RUN dotnet publish -c Release -r win-x64 \
-p:PublishSingleFile=true \
-p:SelfContained=true \
-p:SelfContained=false \
-o /app
FROM busybox
syft/pkg/cataloger/internal/dotnet/pe/testdata/image-dotnet5-single-file/Dockerfile
Vendored
+4 -1
View File
@@ -1,9 +1,12 @@
# note: this is published framework-dependent on purpose: the bundle format (what these fixtures exercise) is
# identical either way, but bundling the runtime makes the fixture ~60MB instead of ~160KB, which dominates
# the runtime of the tests that load this image.
FROM --platform=linux/amd64 mcr.microsoft.com/dotnet/sdk:5.0 AS build
WORKDIR /src
COPY src .
RUN dotnet publish -c Release -r win-x64 \
-p:PublishSingleFile=true \
-p:SelfContained=true \
-p:SelfContained=false \
-o /app
FROM busybox
syft/pkg/cataloger/internal/dotnet/pe/testdata/image-dotnet6-single-file/Dockerfile
Vendored
+4 -1
View File
@@ -1,9 +1,12 @@
# note: this is published framework-dependent on purpose: the bundle format (what these fixtures exercise) is
# identical either way, but bundling the runtime makes the fixture ~60MB instead of ~160KB, which dominates
# the runtime of the tests that load this image.
FROM --platform=linux/amd64 mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY src .
RUN dotnet publish -c Release -r win-x64 \
-p:PublishSingleFile=true \
-p:SelfContained=true \
-p:SelfContained=false \
-o /app
FROM busybox