# Flaw families

> The kinds of flaw an audit finds. Some flaws inflate a score and some deflate it. Only a material flaw makes a task flawed.

Source: https://audit-docs.horizonanalyticslabs.com/docs/reference/flaw-families

A task is flawed when its score does not measure the work. A flaw either inflates a score, so that an agent is paid for work it did not do, or deflates a score, so that an agent is not paid for work it did.

## Flaws that inflate a score

| Kind of flaw                              | What it means                                                                          |
| ----------------------------------------- | -------------------------------------------------------------------------------------- |
| The score can be had without the work     | An agent that does nothing, or that fakes its output, gets full reward.                |
| The answer is in the environment          | The agent can read the answer, for example from the history or from the image.         |
| The tests check less than the task asks   | A wrong answer passes, because the grader does not test what the instruction requires. |
| The task does not test the skill it names | An agent can pass without the skill that the task claims to measure.                   |

## Flaws that deflate a score

| Kind of flaw                                | What it means                                                                         |
| ------------------------------------------- | ------------------------------------------------------------------------------------- |
| The tests and the instruction disagree      | A right answer fails, because the grader asks for something the instruction does not. |
| The task cannot be run or reviewed          | A part is missing, the environment does not build, or a requirement cannot be met.    |
| Grading depends on something that can break | The result rests on something outside the task that can change or stop.               |

A task that misses a part is not counted as flawed, and it is not counted as clean. It comes back as "Needs from you".

## Proved flaws and flags

A proved flaw is shown in a run. It comes with the saved output of that run. A flaw that is not measured counts as proved only after an engineer has confirmed it.

A flag is something worth a look. A flag never decides a verdict.

## What material means

A flaw is material when it can change a realistic agent's score. Only a material flaw makes a task flawed. A flaw is material in these cases:

- A realistic answer is scored the wrong way, whether the answer is right or wrong.
- The instruction contradicts the grader.
- The answer key is wrong.
- The reference solution fails.
- The answer leaks to the agent.
- A requirement cannot be met.

A defect that exists but cannot change a realistic score is not material. An engineer decides this when the engineer reads the finding of the defect. A finding of a defect that is not material is not confirmed, and it is not published. The task can still be clean. A defect that is not confirmed does not change the verdict.

## Who stands behind a verdict

A verdict rests on what was measured. What is not measured counts toward a verdict only after an engineer has confirmed it. An engineer confirms every flawed verdict before you see it. You never see a draft.

## What you get for a flaw

Each flaw of a flawed task has a finding. A finding has three parts:

- where the flaw is, with the lines quoted
- how it was established, as numbered steps you can repeat
- what the finding does not claim

A finding does not say how to change the task. See [read a verdict](/docs/guide/read-a-verdict).