How the AI Bug Hunt works

CodevArena tests one specific skill: whether you can tell working code from code that only looks like it works. Every snippet carries a defect class that AI coding assistants produce in real use. It takes about ninety seconds, needs no account, and gives you a shareable scorecard with a real percentile at the end.

What is the AI Bug Hunt?

A free 90-second benchmark that measures whether you can spot the kinds of defect AI coding assistants actually produce. You are shown four short snippets. Some contain a real bug, some are clean, and you are never told how many of each. For each one you either flag the broken line and say what is wrong with it, or ship it as correct.

Why does it matter whether you can review AI-generated code?

Coding assistants now write a large share of the code that reaches production, and reviewing a diff is a different skill from writing the code yourself. The common failure is accepting a plausible-looking suggestion without verifying it. The second, less discussed failure is the opposite: blocking correct code because it looks unfamiliar. This benchmark measures both.

Which languages does it cover?

The pool covers Python, TypeScript, JavaScript, Go, SQL, and Java, but a single run is four snippets drawn across three or four of them — you cannot pick, and you will not see all six. That is deliberate. Every snippet is chosen to be readable in under fifteen seconds by a developer who does not write that language, so the test measures code review rather than language expertise. Not recognising a language is not an excuse for missing its bug; the defects used are ones you can see without knowing the idiom.

What kinds of bugs does it use?

Real defect classes that survive code review: off-by-one errors, async callbacks passed to forEach so the function returns before the work finishes, reference equality where value equality was meant, inverted cache expiry checks, a WHERE clause that silently converts a LEFT JOIN to an INNER JOIN, NOT IN against a nullable subquery, a defer registered before its error check, mutable default arguments, and sort comparators that return a boolean instead of a number.

Where do the snippets come from?

Two ways, and each snippet says which above the code. Some are written by a language model: it is given a real implementation task with a missable edge case and told to write it the way someone under time pressure would, correct about half the time. We do not ask it to insert a bug — an injected bug reads like a puzzle, while a mistake that happens on its own reads like code. Those are badged AI-WRITTEN and record which model and prompt produced them. The rest are written by a person in the style of a documented AI failure mode, and are badged only UNREVIEWED. Either way the defect classes are drawn from mistakes coding assistants make in real use.

How are generated snippets checked before they appear?

Every candidate passes an automated gate and then a human read. The gate rejects anything that does not fit the format: more than one defect, a bug that does not localize to exactly one line, a line too wide to read, or a decoy set that would let you identify clean code by elimination. Rejected candidates are discarded rather than patched — editing one would put a person back into the authorship chain and make the AI-written badge untrue. Roughly half of what is generated does not survive, and a person reads every survivor before it reaches the pool, because the gate can check shape but not whether a decoy is secretly also correct.

How is the score calculated?

Your rank is based on how many of the four you got right, from Vibe Coder at zero to Principal AI Auditor at four. Speed contributes a smaller amount, so a correct careful answer always beats a fast wrong one. The percentile is a real comparison against every previous run, not an estimate — while the sample is still small it is labelled provisional rather than presented as a measurement.

Do I need an account?

No. There is no login, no payment, and no cookies. Grading happens on the server, so the answers are never sent to your browser.