How to choose a ClearRun score threshold

Score answers you already know are well argued and answers you know are not, then set the line between them. ClearRun does not pick a threshold for you, and the number depends on your use.

Last updated

Steps

  1. Collect two small sets

    10 to 20 answers you would trust as written, and 10 to 20 you would not.

  2. Score both

    clearrun score good/ --json > good.json
clearrun score bad/ --json > bad.json

  3. Pick the line

    Choose a number that most good answers clear and most bad ones don't. Lean higher where a wrong answer costs more.

  4. Review it

    Check misses every few weeks, and after you change models or prompts. Note scoringVersion too.

Gray zone

Instead of one line, you can use two: auto-accept above the top one, send to a person between them, and retry below the bottom one. The confidenceInterval shows how wide the uncertainty is.