oss/syft
2
0
Fork 0
mirror of https://github.com/anchore/syft.git synced 2026-02-12 02:26:42 +01:00
Code Issues Packages Projects Releases Wiki Activity

Migrate CI to runs-on (#4351)

Browse Source 
* migrate to runs-on runners

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* keep validations on x64

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* correct ubuntu arm refs

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* bust cache

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* use artifacts api between jobs

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* download individual artifacts

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* troubleshoot artifact upload

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* disable magic cache

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fix deps

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* attempt to replicate layout

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* change asset names and remove extras

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* better artifact filters

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* use action to get artifacts working

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add sboms

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* simpler artifacts

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* add logging

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* remove logging

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* make artifacts executable

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* improve workflow dispatch calls

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* attempt to bring down ci times

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* update repo path

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* use local config instead of shared one

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* use runner labels instead of config

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* use valid alias

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* use compute instances for build

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* small comment on concurrency

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* fixes from review

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* remove parallelism

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
This commit is contained in:
Alex Goodman 2026-01-05 09:53:06 -05:00 committed by GitHub
parent c8184bdb4c
commit 11fed90075
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 66 additions and 147 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.github/actions/bootstrap/action.yaml vendored
View File

@ -13,7 +13,7 @@ inputs:
cache-key-prefix:
description: "Prefix all cache keys with this value"
required: true
default: "181053ac82"
default: "53ac821810"
download-test-fixture-cache:
description: "Download test fixture cache from OCI and github actions"
required: true

13
.github/workflows/release.yaml vendored
View File

@ -21,8 +21,8 @@ on:
jobs:
quality-gate:
environment: release
runs-on: ubuntu-24.04
if: ${{ github.event.inputs.phase == 'all' }}
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 #v6.0.1
with:
@ -127,9 +127,14 @@ jobs:
false
release:
needs: [quality-gate]
runs-on: ubuntu-24.04
needs: [ quality-gate ]
if: ${{ github.event.inputs.phase == 'all' }}
# runs-on.com: compute instances for parallel builds
# spot disabled: reliability for build workflows (used for releases too)
# goreleaser uses parallelism of 12, so we need more CPUs
# s3-cache: faster actions cache
# tmpfs: faster io-intensive workflows
runs-on: runs-on=${{ github.run_id }}/cpu=16+32/ram=32+128/family=c5+c6+c7+c8/spot=false/extras=s3-cache+tmpfs
permissions:
contents: write
packages: write
@ -198,7 +203,7 @@ jobs:
if: ${{ success() }}
release-install-script:
needs: [release]
needs: [ release ]
if: ${{ always() && (needs.release.result == 'success' || github.event.inputs.phase == 'install-script-only') }}
uses: "anchore/workflows/.github/workflows/release-install-script.yaml@main"
with:

8
.github/workflows/test-fixture-cache-publish.yaml vendored
View File

@ -13,8 +13,12 @@ jobs:
Publish:
name: "Publish test fixture image cache"
# we use this runner to get enough storage space for docker images and fixture cache
runs-on: ubuntu-22.04-4core-16gb
# runs-on.com: memory-optimized instances for building test fixture cache
# spot disabled: can run up to an hour, avoid interruptions
# disk=large: need more storage for docker images and fixture cache
# s3-cache: faster actions cache
# tmpfs: faster io-intensive workflows
runs-on: runs-on=${{ github.run_id }}/cpu=4+8/ram=64+128/family=r5+r6+r7+r8/spot=false/extras=s3-cache+tmpfs
if: github.repository == 'anchore/syft' # only run for main repo
permissions:
packages: write

190
.github/workflows/validations.yaml vendored
View File

@ -1,5 +1,7 @@
name: "Validations"
# we should cancel any in-progress runs for the same workflow + PR/ref
# so that we can avoid redundant work / save on CI minutes
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
@ -12,14 +14,17 @@ on:
- main
permissions:
contents: read
contents: read
jobs:
Static-Analysis:
# Note: changing this job name requires making the same update in the .github/workflows/release.yaml pipeline
name: "Static analysis"
runs-on: ubuntu-24.04
# runs-on.com: memory & general purpose instances for testing
# spot enabled: ok to interrupt non-production workloads
# s3-cache: faster actions cache
# tmpfs: faster io-intensive workflows
runs-on: &test-runner "runs-on=${{ github.run_id }}/cpu=4+8/ram=32+128/family=r5+r6+r7+r8+m4+m5+m6+m7+m8/spot=price-capacity-optimized/extras=s3-cache+tmpfs"
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 #v6.0.1
with:
@ -36,12 +41,10 @@ jobs:
- name: Run static analysis
run: make static-analysis
Unit-Test:
# Note: changing this job name requires making the same update in the .github/workflows/release.yaml pipeline
name: "Unit tests"
# we need more storage than what's on the default runner
runs-on: ubuntu-22.04-4core-16gb
runs-on: *test-runner
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 #v6.0.1
with:
@ -55,11 +58,10 @@ jobs:
- name: Run unit tests
run: make unit
Integration-Test:
# Note: changing this job name requires making the same update in the .github/workflows/release.yaml pipeline
name: "Integration tests"
runs-on: ubuntu-24.04
runs-on: *test-runner
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 #v6.0.1
with:
@ -76,11 +78,18 @@ jobs:
- name: Run integration tests
run: make integration
Build-Snapshot-Artifacts:
name: "Build snapshot artifacts"
runs-on: ubuntu-24.04
# runs-on.com: compute instances for parallel builds
# spot disabled: reliability for build workflows (used for releases too)
# goreleaser uses parallelism of 12, so we need more CPUs
# s3-cache: faster actions cache
# tmpfs: faster io-intensive workflows
runs-on: "runs-on=${{ github.run_id }}/cpu=16+32/ram=32+128/family=c5+c6+c7+c8/spot=false/extras=s3-cache+tmpfs"
steps:
# required for magic-cache from runs-on to function with artifact upload/download (see https://runs-on.com/caching/magic-cache/#actionsupload-artifact-compatibility)
- uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # v2.0.3
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 #v6.0.1
with:
persist-credentials: false
@ -96,83 +105,22 @@ jobs:
- name: Smoke test snapshot build
run: make snapshot-smoke-test
# why not use actions/upload-artifact? It is very slow (3 minutes to upload ~600MB of data, vs 10 seconds with this approach).
# see https://github.com/actions/upload-artifact/issues/199 for more info
- name: Upload snapshot artifacts
uses: actions/cache/save@9255dc7a253b0ccc959486e2bca901246202afeb #v5.0.1
uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 #v6.0.0
with:
# we need to preserve the snapshot data itself as well as the task data that confirms if the
# snapshot build is stale or not. Otherwise the downstream jobs will attempt to rebuild the snapshot
# even though it already exists.
path: |
snapshot
.task
key: snapshot-build-${{ github.run_id }}
Upload-Snapshot-Artifacts:
# Note: changing this job name requires making the same update in the .github/workflows/release.yaml pipeline
name: "Upload snapshot artifacts"
needs: [Build-Snapshot-Artifacts]
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 #v6.0.1
with:
persist-credentials: false
- name: Bootstrap environment
uses: ./.github/actions/bootstrap
with:
download-test-fixture-cache: true
- name: Download snapshot build
id: snapshot-cache
uses: actions/cache/restore@9255dc7a253b0ccc959486e2bca901246202afeb #v5.0.1
with:
path: |
snapshot
.task
fail-on-cache-miss: true
key: snapshot-build-${{ github.run_id }}
# workaround for https://github.com/actions/cache/issues/1265
- name: (cache-miss) Snapshot build missing
if: steps.snapshot-cache.outputs.cache-hit != 'true'
run: echo "unable to download snapshots from previous job" && false
- run: npm install @actions/artifact@2.2.2
- uses: actions/github-script@v8
with:
script: |
const { readdirSync } = require('fs')
const { DefaultArtifactClient } = require('@actions/artifact')
const artifact = new DefaultArtifactClient()
const ls = d => readdirSync(d, { withFileTypes: true })
const baseDir = "./snapshot"
const dirs = ls(baseDir).filter(f => f.isDirectory()).map(f => f.name)
const uploads = []
for (const dir of dirs) {
// uploadArtifact returns Promise<{id, size}>
uploads.push(artifact.uploadArtifact(
// name of the archive:
`${dir}`,
// array of all files to include:
ls(`${baseDir}/${dir}`).map(f => `${baseDir}/${dir}/${f.name}`),
// base directory to trim from entries:
`${baseDir}/${dir}`,
{ retentionDays: 30 }
))
}
// wait for all uploads to finish
Promise.all(uploads)
name: snapshot
path: snapshot/
retention-days: 30
Acceptance-Linux:
# Note: changing this job name requires making the same update in the .github/workflows/release.yaml pipeline
name: "Acceptance tests (Linux)"
needs: [Build-Snapshot-Artifacts]
runs-on: ubuntu-24.04
runs-on: *test-runner
steps:
# required for magic-cache from runs-on to function with artifact upload/download (see https://runs-on.com/caching/magic-cache/#actionsupload-artifact-compatibility)
- uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # v2.0.3
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 #v6.0.1
with:
persist-credentials: false
@ -182,20 +130,14 @@ jobs:
with:
download-test-fixture-cache: true
- name: Download snapshot build
id: snapshot-cache
uses: actions/cache/restore@9255dc7a253b0ccc959486e2bca901246202afeb #v5.0.1
- name: Download snapshot artifacts
uses: actions/download-artifact@018cc2cf5baa6db3ef3c5f8a56943fffe632ef53 #v6.0.0
with:
path: |
snapshot
.task
fail-on-cache-miss: true
key: snapshot-build-${{ github.run_id }}
name: snapshot
path: snapshot
# workaround for https://github.com/actions/cache/issues/1265
- name: (cache-miss) Snapshot build missing
if: steps.snapshot-cache.outputs.cache-hit != 'true'
run: echo "unable to download snapshots from previous job" && false
- name: Restore binary permissions
run: chmod +x snapshot/*/syft snapshot/*/*.exe 2>/dev/null || true
- name: Run comparison tests (Linux)
run: make compare-linux
@ -211,11 +153,11 @@ jobs:
if: steps.install-test-image-cache.outputs.cache-hit != 'true'
run: make install-test-cache-save
Acceptance-Mac:
# Note: changing this job name requires making the same update in the .github/workflows/release.yaml pipeline
name: "Acceptance tests (Mac)"
needs: [Build-Snapshot-Artifacts]
# note: macos runners aren't supported yet for runs-on managed runners.
runs-on: macos-latest
steps:
- name: Install Cosign
@ -232,20 +174,14 @@ jobs:
go-dependencies: false
download-test-fixture-cache: true
- name: Download snapshot build
id: snapshot-cache
uses: actions/cache/restore@9255dc7a253b0ccc959486e2bca901246202afeb #v5.0.1
- name: Download snapshot artifacts
uses: actions/download-artifact@018cc2cf5baa6db3ef3c5f8a56943fffe632ef53 #v6.0.0
with:
path: |
snapshot
.task
fail-on-cache-miss: true
key: snapshot-build-${{ github.run_id }}
name: snapshot
path: snapshot
# workaround for https://github.com/actions/cache/issues/1265
- name: (cache-miss) Snapshot build missing
if: steps.snapshot-cache.outputs.cache-hit != 'true'
run: echo "unable to download snapshots from previous job" && false
- name: Restore binary permissions
run: chmod +x snapshot/*/syft 2>/dev/null || true
- name: Run comparison tests (Mac)
run: make compare-mac
@ -253,13 +189,15 @@ jobs:
- name: Run install.sh tests (Mac)
run: make install-test-ci-mac
Cli-Linux:
# Note: changing this job name requires making the same update in the .github/workflows/release.yaml pipeline
name: "CLI tests (Linux)"
needs: [Build-Snapshot-Artifacts]
runs-on: ubuntu-24.04
runs-on: *test-runner
steps:
# required for magic-cache from runs-on to function with artifact upload/download (see https://runs-on.com/caching/magic-cache/#actionsupload-artifact-compatibility)
- uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # v2.0.3
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 #v6.0.1
with:
persist-credentials: false
@ -269,42 +207,14 @@ jobs:
with:
download-test-fixture-cache: true
- name: Download snapshot build
id: snapshot-cache
uses: actions/cache/restore@9255dc7a253b0ccc959486e2bca901246202afeb #v5.0.1
- name: Download snapshot artifacts
uses: actions/download-artifact@018cc2cf5baa6db3ef3c5f8a56943fffe632ef53 #v6.0.0
with:
path: |
snapshot
.task
fail-on-cache-miss: true
key: snapshot-build-${{ github.run_id }}
name: snapshot
path: snapshot
# workaround for https://github.com/actions/cache/issues/1265
- name: (cache-miss) Snapshot build missing
if: steps.snapshot-cache.outputs.cache-hit != 'true'
run: echo "unable to download snapshots from previous job" && false
- name: Restore binary permissions
run: chmod +x snapshot/*/syft snapshot/*/*.exe 2>/dev/null || true
- name: Run CLI Tests (Linux)
run: make cli
Cleanup-Cache:
name: "Cleanup snapshot cache"
if: github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-24.04
permissions:
actions: write
needs:
- Acceptance-Linux
- Acceptance-Mac
- Cli-Linux
- Upload-Snapshot-Artifacts
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 #v6.0.1
with:
persist-credentials: false
- name: Delete snapshot cache
run: gh cache delete "snapshot-build-${{ github.run_id }}"
env:
GH_TOKEN: ${{ github.token }}