How to use ClearRun in CI without blocking every pull request

Score a small, fixed set of sample answers, run the job only when prompts or model settings change, and start with the job set to report. Turn on blocking once you trust your threshold.

Last updated

Steps

  1. Run only when it matters

    on:
  pull_request:
    paths:
      - "prompts/**"
      - "responses/**"

  2. Report first

    - run: npx -y @clearrun/cli@1.2.0 score --json > clearrun.json
  continue-on-error: true
  env:
    CLEARRUN_API_KEY: ${{ secrets.CLEARRUN_API_KEY }}

    The job shows results but doesn't fail the pull request.

  3. Block when ready

    Remove continue-on-error and add a threshold in clearrun.config.json.

Keep the set small

Every file is scored on every run and uses credits. A few dozen well-chosen answers usually tell you more than hundreds.