Commit Graph
17 Commits
Author SHA1 Message Date
Alex Goodman 8d5567c423 feat: catalog CPAN distributions installed by perl clients (#5131)
* feat: catalog CPAN distributions installed by perl clients

Adds `perl` as a language and `cpan` as a package type, with two catalogers behind them.

- `perl-cpan-installed-cataloger` reads `.meta/*/install.json` (cpanm, cpm, carton) and `auto/**/.packlist` (anything installed through `ExtUtils::MakeMaker` or `Module::Build`, including CPAN.pm). Both globs are unanchored, so a local-lib or carton application tree is found the same as a system install.
- `perl-cpan-meta-cataloger` reads an unpacked release's own `META.json` or `META.yml`, gated on a sibling `MANIFEST` so source checkouts are ignored.

Packages are keyed on the **distribution**, not the module, because that is what CPAN, MetaCPAN and the advisory data all use. `LWP.pm` belongs to `libwww-perl` and reporting it as `LWP` would make every advisory for it unreachable. The distribution name comes from `install.json`'s `dist` field, and from the `auto/` path for packlists.

purls are `pkg:cpan/<distribution>@<version>`, with the PAUSE author added as an `author` qualifier when the evidence carries it. Metadata comes in two types: `cpan-distribution` for installed evidence and `cpan-unpacked-release` for a release sitting on disk. The tiers differ in more than a name, so a consumer should not have to string-match a cataloger name to tell them apart: an unpacked release has no PAUSE path and therefore no author and no file list, and "installed and loadable by the interpreter" is a materially stronger claim than "a source tree exists here".

A packlist's version comes from `perllocal.pod`. EUMM writes it and the packlist from the same variables in the same install target, so the `auto/` path segments and the perllocal `Module` name are the same key, and the recorded `VERSION` is what was evaluated at build time rather than what can be read back statically. Scraping `$VERSION` out of the main `.pm` is the fallback, since `NO_PERLLOCAL` suppresses the file and Module::Build never writes one. Three rules pick the stanza: dashed module name, longest `installed into` libdir that prefixes the packlist path, and last match wins because the file is append-only. Without the libdir rule a second perl on the image silently supplies the version.

Where scraping is the fallback, it reads more than a plain `our $VERSION = '...'`. A version declared in the package statement (`package Foo::Bar v1.0.0;`) counts, which matters because a distribution can declare it that way and carry no `$VERSION` at all: `CPAN::02Packages::Search` is one, and without this it reports no version and matches nothing. Fully qualified `$Foo::Bar::VERSION` counts too, which is what older Dist::Zilla emitted and what real `JSON::PP` 2.27300 still carries. `qv('1.2.3')` is handled. A version computed at runtime is not, and those are reported without one rather than guessed at.

Packlist entries are parsed the way `ExtUtils::Packlist` writes them: a line can carry space-separated metadata after the path (`/path/to/File.pm type=file`), which is what `installperl` produces, so the suffix is stripped rather than the line being cut at its first space.

`META.yml` is read alongside `META.json`, because about 43% of current CPAN releases ship no `META.json` and they skew old, which is where the advisories are. Where both sit in one directory the `META.json` wins. A `META.yml` a strict parser rejects skips that directory rather than failing the scan, which is routine rather than defensive for pre-spec releases.

A packlist is literally a file list, so its paths are surfaced through `pkg.FileOwner`. They are reported as recorded, unfiltered: a path the packlist claims and the filesystem lacks means the file was removed or overwritten out from under the installer, which is worth seeing rather than hiding.

Build leftovers under `~/.cpanm/work` and `~/.cpan/build` are skipped. They genuinely are unpacked release tarballs, `MANIFEST` included, so the `MANIFEST` gate admits them and a path exclusion is the only signal available. Without it every distribution on an image that did not clean up is reported twice. The cost is that a tarball deliberately kept under `~/.cpan/build` stops being reported.

Two behaviors that look wrong but are not:

- a distribution can be reported twice at different versions. `libwww-perl` 5.836 bundles `HTTP-Date`, `HTTP-Message` and `LWP-MediaTypes`, and installing it over the modern standalone releases overwrites their `.pm` files. Both versions are genuinely present, so the merge refuses to pair disagreeing versions rather than hiding one.
- a distribution with no readable version is reported without one rather than dropped, so it stays visible.

The coverage boundary is stated in full in the `package perl` doc comment. In short: modules installed from distro packages are out of scope, since packagers strip CPAN metadata with `NO_PACKLIST` and deb, rpm and apk already report them. Core and dual-life distributions bundled with the interpreter have no coverage, because nothing on disk carries their versions; the interpreter itself is reported separately. Vendored trees, `App::FatPacker` output and PAR archives are invisible, because what they carry is module identity with no offline map to a distribution. A packlist-derived name is the installer's `NAME` and may not be the distribution name, which is left to the vulnerability data to resolve.

One correction worth calling out, since it is easy to arrive at twice: a sibling `MANIFEST` is the only thing separating an unpacked release from a source checkout. An earlier version of the guard also rejected any tree containing a `dist.ini`, on the theory that it marked a Dist::Zilla source tree. dzil ships `dist.ini` *inside* the tarballs it builds and lists it in the generated `MANIFEST`, so that exclusion was silently skipping real releases, `URI` among them.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test(cli): bump coverage image package count for the new perl fixture

The CPAN fixture added to `image-pkg-coverage` (`URI-5.35`) is shared with the CLI suite through a symlink, so the squashed package count goes from 42 to 43.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix(perl): take CPAN distribution name and version from install.json dist

cpanm writes the main module's `$VERSION` into `version` and the release distvname into `dist`, and the two often disagree (`Carp-Assert-More-2.9.0` records `2.009000`). Trimming `version` off `dist` failed on those, so the whole distvname became the package name, and the module version never paired with the packlist record for the same install, which reported the distribution twice.

- name and version are now split out of `dist` at the last `-` followed by a version, as CPAN::DistnameInfo does
- a `-TRIAL` suffix is dropped, so `Try-Tiny-0.26-TRIAL` reports as `Try-Tiny` `0.26`
- the PAUSE path fallback uses the same split

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
2026-09-30 14:15:05 -04:00
Rez MossandChristopher Phillips 74c9380248 fix: 4423 dotnet-deps cataloger skips project type by def
---------
Signed-off-by: Rez Moss <hi@rezmoss.com>
Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
Signed-off-by: Christopher Angelo Phillips <32073428+spiffcs@users.noreply.github.com>
Co-authored-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
2025-12-18 13:38:47 -05:00
Christopher Angelo Phillips 6b48bd4b5e feat: add package supplier flag (#4131)
---------

Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>
2025-08-12 14:49:41 -04:00
Rez MossandAlex Goodman 12d91f47dc Add a homebrew cataloger (#3724)
* Cataloger homebrew (#4)

* homebrew cataloger

* uptd

* fixed test

* fixed test

* fixed tests

* fixed lint

* inc schema ver

* upt schema

* fixed integration test

* fixed integration tst

* fixed test

Signed-off-by: Rez Moss <hi@rezmoss.com>

* Update parse_homebrew_test.go

Signed-off-by: Rez Moss <hi@rezmoss.com>

* Update parse_homebrew_test.go

fixed DCO

Signed-off-by: Rez Moss <hi@rezmoss.com>



Signed-off-by: Rez Moss <hi@rezmoss.com>

* Update parse_homebrew_test.go

add evd anno to test

Signed-off-by: Rez Moss <hi@rezmoss.com>

* lint

Signed-off-by: Rez Moss <hi@rezmoss.com>

* fixed test

Signed-off-by: Rez Moss <hi@rezmoss.com>

* with PR refactors

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* regenerate json schema

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix tests

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* regenerate jsonschema

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* refactor homebrew parser + add tests

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* more resiliant variable extraction

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Rez Moss <hi@rezmoss.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
2025-05-13 13:01:41 -04:00
Alex Goodman ad9928cb2a Merge the .NET deps.json and PE binary catalogers (#3563)
* add combined deps.json + pe binary cataloger

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* deprecate pe and deps standalone catalogers

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* parse resource names + add tests

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix integration and CLI tests

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add some helpful code comments

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* allow for dropping Dep packages that are missing DLLs

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* migrate json schema changes to 24

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* keep application configuration

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* correct config help

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* [wip] detect claims of dlls within deps.json

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* [wip] fix tests

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add assembly repack detection

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* .net package count is lower due to dll claim requirement

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
2025-03-27 14:38:16 -04:00
Keith Zantow 4a9437808e feat: parallelize catalogers per-file and hash contents in parallel (#3636)
Signed-off-by: Keith Zantow <kzantow@gmail.com>
2025-03-26 11:10:08 -04:00
Alex Goodman 684b6e3f98 Add file catalogers to selection configuration (#3505)
* add file catalogers to selection configuration

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix typos

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* warn when there is conflicting file cataloging configuration

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* allow for explicit removal of all package and file tasks

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* address PR feedback

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
2025-02-03 19:10:17 +00:00
Alex Goodman 0a3f513f92 Slim down docker cache size (#3190)
* slim down docker cache size

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* remove old centos images

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* troubleshoot test failure

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix wget version ref

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* refactor caching mechanisms

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add cache cleanup steps

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* simplify deleting cache

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix first clone issue

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add tool dep

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
2024-09-09 11:15:13 -04:00
Laurent Goderre b51252dbdb feat: Add Lua cataloger (#2613)
Signed-off-by: Laurent Goderre <laurent.goderre@docker.com>
2024-05-24 16:56:34 -04:00
Laurent Goderre e0233625cb feat: cataloger for PHP Pecl and PEAR packages (#2604)
Signed-off-by: Laurent Goderre <laurent.goderre@docker.com>
2024-04-02 11:55:56 -04:00
Keith Zantow a978966cad feat: add --from flag, refactor source providers (#2610)
Signed-off-by: Keith Zantow <kzantow@gmail.com>
2024-02-27 16:44:37 -05:00
Alexandr HacicheantandAlex Goodman 96ee2db875 feat: Add Wordpress cataloger (#2218)
* Closes #1911 Wordpress cataloger

Signed-off-by: disc <a.hacicheant@gmail.com>

* Fixed a few unit tests and static analizer notices

Signed-off-by: disc <a.hacicheant@gmail.com>

* Updated `README.md`

Signed-off-by: disc <a.hacicheant@gmail.com>

* Fixed `golangci-lint` notices
Added integration test for `wordpress-plugin`

Signed-off-by: disc <a.hacicheant@gmail.com>

* Fixed `gosimports` notices

Signed-off-by: disc <a.hacicheant@gmail.com>

* Updated `json schema` version

Signed-off-by: disc <a.hacicheant@gmail.com>

* Fixed CLI tests, increased expected package count

Signed-off-by: disc <a.hacicheant@gmail.com>

* Read first 4Kb of a plugins file's content

Signed-off-by: disc <a.hacicheant@gmail.com>

* replace JSON schema version

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* change wording on source info for wordpress packages

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* Minor changes after a huge refactoring

Signed-off-by: disc <a.hacicheant@gmail.com>

* Removed unused files

Signed-off-by: disc <a.hacicheant@gmail.com>

* Updated schema

Signed-off-by: disc <a.hacicheant@gmail.com>

* Fixed integration tests

Signed-off-by: disc <a.hacicheant@gmail.com>

* fix integration tests

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* Renamed `metadata.Name` to `metadata.PluginInstallDirectory`

Signed-off-by: disc <a.hacicheant@gmail.com>

* rename fields to be compliant with json conventions

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: disc <a.hacicheant@gmail.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
2024-02-14 16:03:25 +00:00
Keith Zantow d26a5c4d30 fix: ensure version output to stdout (#2621) 2024-02-09 20:59:25 +00:00
William MurphyandAlex Goodman bbddac1f9d Fix attest with --key (#2551)
Fix passing "--key" to the attest command. Additionally, pull in an update to
the clio CLI library to permit unit testing that flags and env vars are parsed
to the correct field on command options structs. This testing strategy was
needed here because testing attestation in an end to end test requires a
prohibitive amount of setup.

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Will Murphy <will.murphy@anchore.com>
Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
2024-01-31 16:39:17 -05:00
Alex Goodman 4c77783461 Re-introduce linux kernel cataloger (#2526)
* re-add linux kernel cataloger

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* ensure there is at least a directory or image tag on each task

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix CLI tests to account for kernel finding (+2)

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
2024-01-22 16:31:41 +00:00
Alex Goodman b0ab75fd89 Replace core SBOM-creation API with builder pattern (#1383)
* remove existing cataloging API

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add file cataloging config

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add package cataloging config

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add configs for cross-cutting concerns

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* rename CLI option configs to not require import aliases later

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* update all nested structs for the Catalog struct

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* update Catalog cli options

- add new cataloger selection options (selection and default)
- remove the excludeBinaryOverlapByOwnership
- deprecate "catalogers" flag
- add new javascript configuration

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* migrate relationship capabilities to separate internal package

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* refactor golang cataloger to use configuration options when creating packages

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* create internal object to facilitate reading from and writing to an SBOM

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* create a command-like object (task) to facilitate partial SBOM creation

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add cataloger selection capability

- be able to parse string expressions into a set of resolved actions against sets
- be able to use expressions to select/add/remove tasks to/from the final set of tasks to run

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add package, file, and environment related tasks

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* update existing file catalogers to use nested UI elements

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add CreateSBOMConfig that drives the SBOM creation process

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* capture SBOM creation info as a struct

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add CreateSBOM() function

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix tests

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* update docs with SBOM selection help + breaking changes

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix multiple override default inputs

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix deprecation flag printing to stdout

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* refactor cataloger selection description to separate object

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* address review comments

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* keep expression errors and show specific suggestions only

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* address additional review feedback

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* address more review comments

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* addressed additional PR review feedback

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix file selection references

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* remove guess language data generation option

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add tests for coordinatesForSelection

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* rename relationship attributes

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add descriptions to relationships config fields

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* improve documentation around configuration options

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add explicit errors around legacy config entries

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
2024-01-12 17:39:13 -05:00
Alex Goodman 4c20a74d2f Replace packages command with scan (#2446)
* replace packages command with scan

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add tests for packages alias

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* update comments with referenes to the packages command

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* rename valiadte args function

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
2024-01-04 16:56:57 +00:00