6.3 Hallucination, inconsistency and bias

In everyday life. A friend says a restaurant "won an award" (no such award exists), says it is cheap and later says it is expensive, and calls it bad because it is "a chain."

A result can fail in three different ways. Each needs its own test, so name the one you are looking for. The examples come from Brightline's AP Worker, which handles the bills the company owes. It sent Dave, the controller, a package for Friday's payment run. To hold an invoice means to keep it back from payment.

Failure	What it is	The test	In Friday's package
Hallucination	Invented or false content presented as fact: a fact, citation, section or event	Trace it. If nothing supports it, it is unsupported. More checking decides whether it is false	"Bank change verified." "Policy 3.4"
Inconsistency	The output disagrees with itself, with its inputs, or with another version of itself	Compare. Line up the same fact from every place it appears	The memo holds Tri-County. The CSV pays it
Bias	A judgment that leans one way without evidence. It can rest on an attribute, such as size, on one-sided evidence, or on the asker's view taken as fact	Ask what evidence the judgment rests on. Swap the attribute and see if the judgment stays the same	"Small, family-run" and "less reliable"

A hallucination often reads more smoothly than anything else on the page, because no source limited what it could say. The three can happen together. An invented attribute, such as "family-run," is a hallucination that also leads to a biased judgment.

The memo's note about Lakeshore Janitorial, a supplier, shows how bias works. The only evidence was an out-of-date line on two invoices: their payment terms, the number of days Brightline has to pay. The vendor record, where Brightline keeps each supplier's details, shows Brightline changed Lakeshore's terms on August 14, and Lakeshore's invoices still show the old ones. Nothing in any file says the business is small or family-run. Now swap the attribute: picture a large supplier with the same out-of-date line. The worker would most likely have written "ask them to update their template." The judgment came from the attribute, not the evidence. It also answered a question nobody asked.

Bias can come from you too. Chapter 5 warned that a worker shown the conclusion you hope for tends to find it. Check for that kind of bias as well: did the output simply agree with the conclusion the brief hoped for?

For a closer check, this book's first edition has an Error Taxonomy. It lists six types of error to look for one at a time: factual error, logical gap, false confidence, missing context, fabricated source and stale fact.¹

Figure 6.3. Three failures, three tests.

6.3 Hallucination, inconsistency and bias

Check yourself

A row says 'bank change verified.' No input or record supports it, and Maria says no call was made. Which failure?

Sources