Measurement

Unknown CAPTCHA outcomes belong in the report

Keep timeouts, lost responses, and unfinished observations visible instead of forcing every attempt into success or failure.

By the easycaptcha team · Published April 10, 2026

A client timeout can happen after a solver accepted work. A browser can close after receiving a result but before the final application outcome is observed. Those cases are not equivalent to a definite rejected proof. Calling all of them failures makes the dataset simple at the expense of its meaning.

Missing evidence

Use a third state for unknown observations. Specify which stage is unknown: admission, solver completion, verification acceptance, or final action. An unknown final action can coexist with a known completed solver job.

Make outcome coverage explicit

Suppose an illustrative trial starts 100 attempts, observes 70 accepted proofs and 10 rejected proofs, and loses the outcome for 20. Acceptance among observed outcomes is 70/80, or 87.5%. That does not establish 87.5% acceptance across all 100 attempts. Coverage is 80/100, or 80%.

Report both figures and the reasons observations disappeared. If the lost cases disproportionately include slow or difficult tasks, dropping them can materially improve the headline while hiding the operational problem.

Recover known work before creating more

Preserve task IDs as soon as creation returns. A later worker can retrieve accepted jobs whose observers stopped. A lost creation response without an ID is harder: the caller cannot conclude the server never accepted it.

Do not automatically submit another paid job to resolve uncertainty. That creates new evidence about new work, not the original attempt. Keep a reconciliation record and an application-owned policy for the next permitted action.

Reconcile the report after its observation window

Choose an initial reporting cutoff and a later reconciliation pass. Record changes from unknown to known without silently overwriting the original cutoff result. This helps distinguish a truly missing outcome from a merely delayed one.

The same discipline applies to cost. Charges and adjustments can become known after the client stopped waiting. A report that excludes unknown attempts from both denominator and cost can look artificially efficient.

Success-rate definitions explain stage denominators. Task recovery covers ID persistence, and cost per outcome keeps financial observations tied to the same cohort instead of whatever happened to arrive on time.