CHEATSHEET catalogues {LP_STATS.usg} incidents of unattributed specification gaming — where the exploit looks like a property of the model rather than the setup — across {LP_STATS.papers} research papers published between {LP_STATS.years}. Every entry is classified against a public rubric, anchored to its source, and browsable by category, year, model family, or paper.
{hero.intended}
{hero.actual}
Specification gaming happens when an AI system does well measured narrowly by the letter of its objective while missing what the people who built it actually wanted. It may be that a model finds a shortcut, exploits a quirk in how it is scored, or satisfies the stated goal in a way nobody intended. Some of these cases are harmless. Others, however, may point to behaviour that is harder to explain away as a banal system quirk.
The incidents are drawn from the research literature, each entry recording what the system was asked to do, what it actually did, the model involved, and a link back to the source paper. See more about how we did this{" "} here.
CheatSheet is a catalogue of these cases.
The{" "} incidents page{" "} holds the full catalogue, browsable as a tree or a dashboard and filterable by category, year, model family, and source paper. Every entry links to the paper that reported it. How the corpus was built and classified is described on the{" "} methods page, and the{" "} rubric{" "} is published in full under project files, so any entry's classification can be checked against it.
Every incident is sorted against a published rubric classifying the incidents into two major categories: attributed specification gaming and unattributed specification gaming. When going through much of the research literature, incidents called specification gaming often didn't disambiguate themselves between behaviour that was likely produced due to the experimental set-up itself (and thus may not generalise as easily to behaviour in the wild) and those that were harder to explain.
Attributed Specification Gaming (ASG) — when something in the setup, such as the training data, the environment, or the way the system was evaluated, made the exploit foreseeable, so the behaviour can be traced to how the experiment was built.
Unattributed Specification Gaming (USG) — when nothing in the setup obviously pointed to the exploit, which leaves the behaviour looking more like a property of the model than of the test.
Some may notice that these categories may be over-inclusive in some circumstances, in that they include similar and overlapping distinctions found in the literature that we have flattened in our taxonomy. This includes reward hacking, reward gaming, reward misspecification, reward tampering, goal misgeneralization, etc. Similarly, we flattened the cause-based labels from{" "} Krakovna's list of specification gaming, from which this work takes inspiration, as it split incidents by cause where ours splits them only by attributability.
We have also kept cases that we did not count as specification gaming proper. We grouped these under a set of out-of-scope categories: negative side effects, benign competence, benign incompetence, and a catch-all for incidents the rubric could not be applied to yet that pops up in searches for specification gaming. We've included these in an effort of transparency on where lines were drawn, and due to the fact that much of the classifying was produced through an AI-guided process, using different models of Claude 4.x (mainly Opus and Sonnet). We have tried to verify the data through a number of sweeps, but there may still be mistakes, misattributions, and likely plenty of places where one might disagree. {" "}We welcome your feedback.