Evaluation design

Designing a representative CAPTCHA evaluation cohort

Choose task families, site configurations, variants, and observation windows before comparing completion or acceptance rates.

By the easycaptcha team · Published September 22, 2026

An evaluation needs to specify which attempts belong in it. A cohort containing only easy demonstrations does not represent every production flow. A cohort that changes halfway through a comparison can make performance shifts impossible to interpret.

The measured population

List the task families, public configuration attribution, environment, mode, and intended application outcome. Include how unknown outcomes are handled and how long observation continues.

Preserve meaningful strata

Separate binary recognition, area selection, drag placement, and token operations when they require different input or results. If enterprise work is relevant, evaluate its actual required data and entitlement rather than substituting a regular-token demonstration.

A sitekey can provide useful context, but it is not proof that every challenge served under it has the same question or difficulty. Keep recognizable variants visible when the study can legitimately record them.

Control comparisons and report their limits

Use equivalent intended actions and stable collection procedures. Live challenges may change between attempts, so a sequential test is not always the same as two services seeing identical input. Explain what was matched and what was merely sampled from a similar population.

Do not replace failed or unknown attempts with easier ones while retaining the original denominator. Keep inclusion and exclusion rules reviewable.

Observe long enough to classify outcomes

A local deadline can stop a user-facing wait while remote work continues. Record the cutoff outcome and a later reconciliation result separately. Report missing coverage rather than guessing the final state.

Include actual billed work associated with the cohort, selected modes, and relevant adjustments. Recognition groups and tokens have different units; a mixed cohort needs more than one unit price.

This article proposes an evaluation method, not a benchmark result for easycaptcha. Fair comparisons covers reporting, variant drift explains changing populations, and recognition families supplies the typed distinction that makes otherwise pooled results easier to interpret.