Skip to content

Commit ec8aaf3

Browse files
JavagedesJavagedes
authored
Add github workflows for stuart CI Builds (#545)
This pull request adds two new workflows and a github action that are reusable across repositories that use the Stuart CI system. ## New Actions ### `stuart-ci` action This action is an OS agnostic action that will setup the stuart environment (`stuart_setup`, `stuart_ci_setup`, `stuart_update`) and run `stuart_ci_build` with the specified configurations. It provides a single output, `log-path` which is a path to all logs generated by this action. It has some performance increases like caching the .git/modules folder. ### `PackageCi` worfklow This workflow is an OS agnostic workflow that will setup the entire runner environment and run CI on the specified packages. This workflow will generate a build matrix using the provided configuration input to test multiple packages under multiple conditions. The `package-config` input is a json string that contains all matrix configuration for the job (e.g. the input is passed directly to `strategy.matrix` (Examples below). This input string can be hardcoded or automatically generated. It has some performance increases like caching the python version and python dependencies. ### `PackageMatrix` worfklow This workflow is used to generate a matrix configuration that is consumed by the `PackageCi` worfklow. This workflow works by discovering all packages in a repository and generating a configuration for it. This workflow has two main inputs: (1) `default-config` which is the default list of matrix configurations to apply for each discovered package in the workspace and (2) `package-config` which is a way to define more specific configuration for a specific package, which will replace the `default-config`. `PackageMatrix` also supports running `stuart_ci_build`, which will filter the discovered packages in the workspace to only package's that actually need to be tested. This is useful as it prevents generating a runner for a package that does not need tested. ## Usage Examples The three actions / workflows above are not sync'd to repositories. It is expected that a repository maintains its own final github workflow that consumes these packages with it's specific configuration. An example can be seen below <details> <summary> Example Leaf node worfklow </summary> ```yaml # A workflow to build EDKII packages using the CLANGPDB toolchain in windows and ubuntu environments. ## # Copyright (c) Microsoft Corporation. # # SPDX-License-Identifier: BSD-2-Clause-Patent ## name: CLANGPDB Package CI on: workflow_dispatch: push: branches: - release/202511 pull_request: branches: - release/202511 jobs: package-matrix: name: Gather Repository Packages uses: microsoft/mu_devops/.github/workflows/PackageMatrix.yml@personal/joeyvagedes/ci-workflow with: ci-config: '.pytool/CISettings.py' # Manually declared package configurations. package-config: | - packages: MdeModulePkg targets: RELEASE toolchain: CLANGPDB - packages: MdeModulePkg targets: DEBUG toolchain: CLANGPDB # Default config applied to any undeclared packages in the repo default-config: | targets: ["DEBUG,RELEASE"] toolchain: ["CLANGPDB"] python-version: '3.12' windows-ci: name: Windows CLANGPDB CI if: ${{ needs.package-matrix.outputs.matrix != '' }} needs: package-matrix uses: microsoft/mu_devops/.github/workflows/PackageCi.yml@personal/joeyvagedes/ci-workflow with: package-config: ${{ needs.package-matrix.outputs.matrix }} runner: windows-latest python-version: '3.12' setup-cmd: setup ubuntu-ci: name: Ubuntu CLANGPDB CI if: ${{ needs.package-matrix.outputs.matrix != '' }} needs: package-matrix uses: microsoft/mu_devops/.github/workflows/PackageCi.yml@personal/joeyvagedes/ci-workflow with: package-config: ${{ needs.package-matrix.outputs.matrix }} runner: ubuntu-latest container: ghcr.io/microsoft/mu_devops/ubuntu-24-test:latest python-version: '3.12' setup-cmd: setup ``` </details> ## Example CI runs Below is an example of a pull request that generates a large amount of runners because it changes file that is not specific to a package, which results in all packages needing tested: <img width="706" height="385" alt="image" src="https://github.com/user-attachments/assets/0401d480-2e50-4b59-8656-5b9577227326" /> Below is an example of a pull request that generates only a few runners, as stuart_pr_eval decided only a few packages required testing: <img width="695" height="380" alt="image" src="https://github.com/user-attachments/assets/686a5229-1114-4964-bd8c-857f2fa23e70" />
1 parent cd2668a commit ec8aaf3

File tree

6 files changed

+632
-1
lines changed

6 files changed

+632
-1
lines changed

‎.github/actions/stuart-ci/action.yml‎

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
# A callable Github Action to run CI tests on EDK II package(s) using stuart.
2+
#
3+
# Produces an output, `log-path`, which is the path to the folder containing all logs produced.
4+
#
5+
##
6+
# Copyright (c) Microsoft Corporation.
7+
#
8+
# SPDX-License-Identifier: BSD-2-Clause-Patent
9+
##
10+
name: EDK II Package CI
11+
description: Prepare and run CI tests on EDK II package(s) using stuart.
12+
13+
inputs:
14+
ci-config:
15+
description: Path to the CI configuration file (relative to the repository root). No value means '.pytool/CISettings.py'.
16+
required: false
17+
default: '.pytool/CISettings.py'
18+
packages:
19+
description: Comma-separated list of EDK II packages to run CI tests on. No value means all packages. Do not include spaces between package names.
20+
required: false
21+
default: ''
22+
targets:
23+
description: Comma-separated list of build targets to run CI tests on. No value means all targets. Do not include spaces between target names.
24+
required: false
25+
default: ''
26+
architectures:
27+
description: Comma-separated list of architectures to run CI tests on. No value means all architectures. Do not include spaces between architecture names.
28+
required: false
29+
default: ''
30+
toolchain:
31+
description: The toolchain to use for the CI build.
32+
required: false
33+
default: 'CLANGPDB'
34+
stuart-args:
35+
description: Additional arguments to pass to `stuart_ci_build`. Should be space-separated. For example, `--test-filter MyTest*`.
36+
required: false
37+
default: ''
38+
stuart-setup:
39+
description: Whether to run `stuart_setup` before running CI tests ('true' or 'false').
40+
required: false
41+
default: 'false'
42+
stuart-ci-setup:
43+
description: Whether to run `stuart_ci_setup` before running CI tests ('true' or 'false').
44+
required: false
45+
default: 'true'
46+
47+
outputs:
48+
log-path:
49+
description: 'Path to the folder containing the build logs'
50+
value: ${{ steps.tempdir.outputs.path }}
51+
52+
runs:
53+
using: composite
54+
55+
steps:
56+
- name: Create temporary directory for log files
57+
id: tempdir
58+
shell: bash
59+
env:
60+
PACKAGES: ${{ inputs.packages }}
61+
run: |
62+
TEMP_DIR="ci-logs"
63+
if [ -n "${PACKAGES}" ]; then
64+
TEMP_DIR="${TEMP_DIR}-$(echo "${PACKAGES}" | tr ',' '-')"
65+
fi
66+
mkdir -p "$RUNNER_TEMP/$TEMP_DIR"
67+
echo "path=$RUNNER_TEMP/$TEMP_DIR" >> $GITHUB_OUTPUT
68+
69+
- name: Gather submodule hashes
70+
if: ${{ fromJson(inputs.stuart-setup) }}
71+
id: submodules-hash
72+
shell: bash
73+
run: |
74+
HASH=$(git submodule foreach --quiet 'echo $sha1' | sha256sum | awk '{print $1}')
75+
echo "hash=$HASH" >> $GITHUB_OUTPUT
76+
77+
- name: Cache submodules
78+
if: ${{ fromJson(inputs.stuart-setup) }}
79+
uses: actions/cache@v5
80+
with:
81+
path: .git/modules
82+
key: ${{ runner.os }}-submodules-${{ steps.submodules-hash.outputs.hash }}
83+
84+
- name: Download required submodules (stuart_setup)
85+
if: ${{ fromJson(inputs.stuart-setup) }}
86+
env:
87+
CI_CONFIG: ${{ inputs.ci-config }}
88+
shell: bash
89+
run: |
90+
stuart_setup -c "${CI_CONFIG}"
91+
92+
- name: Move setup log
93+
if: always() && fromJson(inputs.stuart-setup)
94+
shell: bash
95+
env:
96+
TEMP_DIR: ${{ steps.tempdir.outputs.path }}
97+
run: |
98+
shopt -s globstar nullglob
99+
files=(Build/SETUP*.txt)
100+
101+
if ((${#files[@]})); then
102+
echo "Moving ${#files[@]} log files"
103+
mv "${files[@]}" "$TEMP_DIR"
104+
else
105+
echo "No matching log files found"
106+
fi
107+
108+
- name: Download CI dependencies (stuart_ci_setup)
109+
if: ${{ fromJson(inputs.stuart-ci-setup) }}
110+
shell: bash
111+
env:
112+
CI_CONFIG: ${{ inputs.ci-config }}
113+
run: |
114+
stuart_ci_setup -c "${CI_CONFIG}"
115+
116+
- name: Move CI setup log
117+
if: always() && fromJson(inputs.stuart-ci-setup)
118+
shell: bash
119+
env:
120+
TEMP_DIR: ${{ steps.tempdir.outputs.path }}
121+
run: |
122+
shopt -s globstar nullglob
123+
files=(Build/CI_SETUP*.txt)
124+
125+
if ((${#files[@]})); then
126+
echo "Moving ${#files[@]} log files"
127+
mv "${files[@]}" "$TEMP_DIR"
128+
else
129+
echo "No matching log files found"
130+
fi
131+
132+
- name: Download external dependencies (stuart_update)
133+
shell: bash
134+
env:
135+
CI_CONFIG: ${{ inputs.ci-config }}
136+
run: |
137+
stuart_update -c "${CI_CONFIG}"
138+
139+
- name: Move update log
140+
if: always()
141+
shell: bash
142+
env:
143+
TEMP_DIR: ${{ steps.tempdir.outputs.path }}
144+
run: |
145+
shopt -s globstar nullglob
146+
files=(Build/UPDATE*.txt)
147+
148+
if ((${#files[@]})); then
149+
echo "Moving ${#files[@]} log files"
150+
mv "${files[@]}" "$TEMP_DIR"
151+
else
152+
echo "No matching log files found"
153+
fi
154+
155+
- name: Run CI tests
156+
shell: bash
157+
env:
158+
CI_CONFIG: ${{ inputs.ci-config }}
159+
PACKAGES: ${{ inputs.packages }}
160+
TARGETS: ${{ inputs.targets }}
161+
ARCHITECTURES: ${{ inputs.architectures }}
162+
TOOLCHAIN: ${{ inputs.toolchain }}
163+
STUART_ARGS: ${{ inputs.stuart-args }}
164+
run: |
165+
ARGS=(-c "${CI_CONFIG}")
166+
if [ -n "${PACKAGES}" ]; then
167+
ARGS+=(-p "${PACKAGES}")
168+
fi
169+
if [ -n "${TARGETS}" ]; then
170+
ARGS+=(-t "${TARGETS}")
171+
fi
172+
if [ -n "${ARCHITECTURES}" ]; then
173+
ARGS+=(-a "${ARCHITECTURES}")
174+
fi
175+
if [ -n "${TOOLCHAIN}" ]; then
176+
ARGS+=(TOOL_CHAIN_TAG="${TOOLCHAIN}")
177+
fi
178+
echo "stuart_ci_build ${ARGS[*]} ${STUART_ARGS}"
179+
stuart_ci_build "${ARGS[@]}" ${STUART_ARGS}
180+
181+
- name: Move build log
182+
if: always()
183+
shell: bash
184+
env:
185+
TEMP_DIR: ${{ steps.tempdir.outputs.path }}
186+
run: |
187+
shopt -s globstar nullglob
188+
files=(Build/**/CI_BUILDLOG*.txt Build/**/BUILDLOG_*.txt)
189+
190+
if ((${#files[@]})); then
191+
echo "Moving ${#files[@]} log files"
192+
mv "${files[@]}" "$TEMP_DIR"
193+
else
194+
echo "No matching log files found"
195+
fi

‎.github/workflows/PackageCi.yml‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
# A workflow to build EDK II packages in this repository using certain configurations.
2+
#
3+
# NOTE: This file is automatically synchronized from Mu DevOps to keep the version of the
4+
# workflow up to date. Update the original file there instead of the file in this repo.
5+
#
6+
# - Mu DevOps Repo: https://github.com/microsoft/mu_devops
7+
# - File Sync Settings: https://github.com/microsoft/mu_devops/blob/main/.sync/Files.yml
8+
#
9+
# Copyright (c) Microsoft Corporation.
10+
#
11+
# SPDX-License-Identifier: BSD-2-Clause-Patent
12+
#
13+
name: Package CI
14+
15+
on:
16+
workflow_call:
17+
inputs:
18+
ci-config:
19+
description: >
20+
Path to the CI configuration file used to run the CI build. Default is
21+
'.pytool/CISettings.py'.
22+
type: string
23+
required: false
24+
default: '.pytool/CISettings.py'
25+
container:
26+
description: A container image to run the build in when using a linux runner.
27+
type: string
28+
required: false
29+
default: ''
30+
package-config:
31+
description: >
32+
A YAML array of objects. Each object must have a "packages" key.
33+
All other keys are passed through to the output.
34+
Example:
35+
- packages: 'MdePkg,UefiCpuPkg'
36+
targets: 'DEBUG,RELEASE,NO-TARGET,NOOPT'
37+
architectures: 'X64,AARCH64'
38+
- packages: 'ShellPkg'
39+
targets: 'DEBUG'
40+
type: string
41+
required: true
42+
python-version:
43+
description: Python version to use when not running in a container
44+
type: string
45+
required: false
46+
default: '3.12'
47+
runner:
48+
description: >
49+
The type of runner to use for the CI build.
50+
type: string
51+
required: false
52+
default: 'ubuntu-latest'
53+
setup-cmd:
54+
description: >
55+
The stuart setup command to run, e.g. `setup`, `ci-setup`, or `both`.
56+
Any other value will result in no setup command being run. Default is `both`.
57+
type: string
58+
required: false
59+
default: 'both'
60+
stuart-args:
61+
description: Additional arguments to pass to `stuart_ci_build`. Should be space-separated. For example, `--test-filter MyTest*`.
62+
type: string
63+
required: false
64+
default: ''
65+
66+
jobs:
67+
package-ci:
68+
69+
name: "${{ matrix.packages }} ${{ matrix.architectures }} ${{ matrix.targets }}"
70+
71+
runs-on: ${{ inputs.runner }}
72+
73+
container:
74+
image: ${{ inputs.container || null }}
75+
options: --security-opt seccomp=unconfined
76+
77+
strategy:
78+
fail-fast: false
79+
matrix: ${{ fromJson(inputs.package-config) }}
80+
81+
steps:
82+
- name: Checkout repository
83+
uses: actions/checkout@v6
84+
85+
- name: Setup Python ${{ inputs.python-version }}
86+
if: ${{ inputs.container == null }}
87+
uses: actions/setup-python@v6
88+
with:
89+
python-version: ${{ inputs.python-version }}
90+
cache: 'pip'
91+
cache-dependency-path: pip-requirements.txt
92+
93+
- name: Install Python dependencies
94+
run: pip install -r pip-requirements.txt
95+
96+
- name: Container Configuration
97+
if: ${{ inputs.container != null }}
98+
shell: bash
99+
run: |
100+
git config --global --add safe.directory '*'
101+
git config --global user.name "GitHub Actions"
102+
git config --global user.email "actions@github.com"
103+
104+
- name: EDK II Package CI
105+
id: package-ci
106+
uses: microsoft/mu_devops/.github/actions/stuart-ci@v18.0.4
107+
with:
108+
ci-config: ${{ inputs.ci-config }}
109+
packages: ${{ matrix.packages }}
110+
targets: ${{ matrix.targets }}
111+
architectures: ${{ matrix.architectures }}
112+
toolchain: ${{ matrix.toolchain }}
113+
stuart-setup: ${{ inputs.setup-cmd == 'setup' || inputs.setup-cmd == 'both' }}
114+
stuart-ci-setup: ${{ inputs.setup-cmd == 'ci-setup' || inputs.setup-cmd == 'both' }}
115+
stuart-args: ${{ inputs.stuart-args }}
116+
117+
- name: Upload CI build logs
118+
if: always()
119+
uses: actions/upload-artifact@v7
120+
with:
121+
name: ci-logs-${{ matrix.packages }}-${{ matrix.architectures }}-${{ matrix.targets }}-${{ inputs.runner }}
122+
path: ${{ steps.package-ci.outputs.log-path }}

0 commit comments

Comments
 (0)