Building Better Code Together: The Key to Success with GitHub Workflow Testing

The Key: Testing Reusable Workflows

As the use of reusable workflows becomes increasingly prevalent across an organization, ensuring their robustness and reliability is of paramount importance. These workflows are used by many teams and projects, making it imperative to subject them to rigorous testing procedures to identify and rectify potential flaws and inconsistencies.

Automated testing offers a scalable and efficient approach to validate the functionality and performance of these workflows. By executing them with predetermined sets of inputs and verifying the output against expected results, DevOps teams can gain confidence in their correctness and repeatability.

Implementation: Using Workflows to Test Workflows

Testing a reusable workflow using another workflow is the logical step in ensuring its reliability and effectiveness. By crafting a comprehensive testing codebase, DevOps engineers can validate the reusable workflow's performance with different sets of inputs.

The process involves providing the testing workflow with both good quality code and bad quality code to simulate various scenarios. For the good quality code, the testing workflow evaluates whether the reusable workflow processes it successfully, generating the expected outputs and outcomes. On the other hand, the bad quality code is deliberately designed to trigger specific quality-gates within the reusable workflow, simulating potential issues or errors.

By incorporating these test cases, the testing workflow effectively exposes vulnerabilities or shortcomings, enabling DevOps engineers to identify and address potential pitfalls. Ultimately, this iterative approach to testing empowers DevOps teams to fine-tune the reusable workflow, enhancing its overall robustness and adaptability in supporting diverse projects and user needs.

Step 1: Reusable Workflow

The reusable-workflows repository has the following structure:

reusable-workflows
├── .github
│   └── workflows
│       └── python-flake8.yaml
└── README.md

The reusable-workflow looks like this: (some parts were shortened for brevity)

# .github/workflows/python-flake8.yaml
name: python-flake8

on:
  workflow_call: # trigger for reusable-workflows
    inputs:
      flake8_options: ...
      working_directory: ...
      continue_on_error: ... # this input is required for testing quality-gates

    outputs:
      flake8: # this output is required to check the flake8 quality-gate
        description: result of running flake8
        value: ${{ jobs.flake8.outputs.result }}

The continue_on_error input and flake8 output are essential additions to enable the automated testing.

jobs:
  flake8:
    runs-on: ubuntu-latest
    # continue-on-error is set to true when the quality-gate is under test
    continue-on-error: ${{ inputs.continue_on_error }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v4
      - name: install flake8
        run: pip install flake8

Above you can see continue-on-error (docs) [1] is configured. Setting it to true when triggering the quality-gate means that the overall workflow execution is not aborted when the quality-gate fails.

      - name: run flak8e 
        id: flake8
        working-directory: ${{ inputs.working_directory }}
        run: flake8 . ${{ inputs.flake8_options }}

    outputs:
      # this output records the success of the flake8 execution
      result: ${{ steps.flake8.outcome }}

The last step executes the flake8 quality-gate and the result outcome (docs) [2] is passed through to the workflow output.

Step 2: Testing Workflow

The reusable-workflows-testing repository has the following structure:

reusable-workflows-testing
├── .github
│   └── workflows
│       └── test-python-flake8.yaml
├── README.md
└── python-flake8
    ├── fail
    │   └── fail.py
    └── success
        └── success.py

This repository contains the testing workflow (.github/workflows/test-python-flake8.yaml) and additional files inside the python-flake8 folder, that are designed to pass the quality-gate (inside success/) or trigger the quality-gate (inside fail/).

This testing workflow has been configured to execute nightly, enabling the DevOps team to promptly receive notifications in the event of any issues:

# .github/workflows/test-python-flake8.yaml
name: test-python-flake8

on:
  # this workflow is triggered once every night
  schedule:
    - cron: "0 3 * * *"

jobs:
  flake8-success:
    uses: WoodmarkConsultingAG/reusable-workflows/.github/workflows/python-flake8.yaml@main
    with:
      # the workflow is executed using the success folder
      working_directory: python-flake8/success/
      # continue_on_error is false since we expect a successful run
      continue_on_error: false

  flake8-fail:
    uses: WoodmarkConsultingAG/reusable-workflows/.github/workflows/python-flake8.yaml@main
    with:
      # the workflow is executed using the fail folder
      working_directory: python-flake8/fail/
      # continue_on_error is true since we expect to trigger flake8
      continue_on_error: true

The python-flake8.yaml reusable-workflow is called twice in the tests above, once with code that should pass successfully, and once with code that should trigger the quality-gate. Note that the continue_on_error input is true when the quality-gate is expected to be triggered.

  check-results:
    runs-on: ubuntu-latest
    if: always()
    needs:
      - flake8-success
      - flake8-fail
    steps:
      - name: ensure expected outcome of flake8-fail
        run: |
          if ${{ needs.flake8-fail.outputs.flake8 != 'failure' }}; then
            echo "error: flake8 did not fail, even though it was expected to"
            exit 1
          else
            echo "flake8-fail ist wie erwartet fehlgeschlagen"
            exit 0
          fi

The last job is to check the results of the previous tests. The check works by inspecting the outcome of triggering the quality-gate.

• If it was not triggered (meaning the output is not failure), then our quality-gate is not working as expected and the overall test result is failure.
• If it was triggered (meaning the output is failure), then our quality-gate is working as expected and the overall test result is success.

Step 3: Good Code, Bad Code

The last piece of the puzzle is the code that is used to execute the tests with. In this case very simple one line examples were chosen, but you might need more elaborate examples depending on your use-case and quality-gate.

# python-flake8/success/success.py
print("success!")

The code above should cause no issues with flake8.

# python-flake8/fail/fail.py
example = lambda: 'example'

This code specifically triggers "flake8 error E731" [3]. This is used to ensure flake8 is working as intended.

Results

The image provided above demonstrates the GitHub Actions UI during the execution of our testing workflow. The "flake8-success" job is shown in green, while the "flake8-fail" job is correctly indicated in red, aligning with our expectations. As a result, the "check-results" job and the overall status are displayed as green, affirming the successful performance of our testing.

These test outcomes provide us with a high level of confidence in the functionality of our reusable workflow. Should the need arise, we can incorporate additional tests to further validate its effectiveness.

Summary

Automated Testing of Reusable Workflows in GitHub

As the use of reusable workflows becomes increasingly prevalent in DevOps teams, ensuring their robustness and reliability is of paramount importance. These workflows are employed by multiple users and projects, making it imperative to subject them to rigorous testing procedures.

Reusable workflows, being utilized by others, demand a comprehensive testing strategy to identify and rectify potential flaws and inconsistencies. Automated testing offers a scalable and efficient approach to validate the functionality and performance of these workflows. By executing them with predetermined sets of inputs and verifying the output against expected results, developers can gain confidence in the correctness and repeatability of these essential components.

Quality gates, which are critical checkpoints in reusable workflows, necessitate special attention during testing. To ensure their reliability, automated tests include the triggering of these gates and subsequent verification that they perform as intended. This process ensures that the quality of the outputs meets predefined criteria and maintains the overall integrity of the workflow.

In conclusion, the adoption of automated testing in the assessment of reusable workflows within GitHub repositories is a crucial step toward enhancing the overall software development process. By assuring the accuracy of outputs and the effectiveness of quality gates, developers can establish a robust foundation for sharing and implementing reusable workflows across diverse projects and user bases.