How to Bench Test Email Verifiers Fairly, Side by Side

To bench test email verifiers fairly, run every tool on the same file, on the same day, and score them against addresses whose true status you already know. Mix seeded known-good, known-bad and known catch-all rows into a real sample of your list. Map each vendor's labels into the same four buckets. Score false valids and false invalids separately. Then settle the disagreements on your real rows with a small controlled send and the bounce codes it returns.

Most vendor comparisons skip at least one of these steps. The result is a test that rewards whichever tool is most willing to say "valid". This guide gives you a protocol that holds up, whichever vendor wins, including us.

Why most verifier bench tests mislead

A bench test is only as good as its answer key. The common mistakes all come from not having one.

Mistake	What goes wrong
Treating vendor A's "valid" as the truth	You measure agreement with A, not accuracy
Counting "risky" or "catch-all" as an error	You punish the honest answer for domains that cannot be checked
Running tools days apart	Mailboxes close and servers change in between
Using 20 or 30 addresses	One mistake swings the error rate by several points
Only testing Gmail and Outlook addresses	Consumer providers behave very differently from company domains
Judging by the size of the valid file	A tool can grow its valid file by relabelling catch-alls
Re-testing a list one vendor has seen	Cached results can return faster and look more confident

Our guide to what accuracy claims actually measure explains why there is no public ground truth for email verification. A bench test fixes that for your own data by building one.

Step 1: Build a seeded answer key

Seed rows are addresses whose status you know before any verifier sees them. You need three kinds.

Known-good addresses

Use addresses you have proof of delivery for, recently:

• Mailboxes you control, at several providers and company domains.
• Colleagues and partners who agree to be included.
• Contacts who replied to you in the last 30 days.

Avoid "opened" as proof. Open tracking is unreliable, and a recent open does not prove the mailbox still exists today.

Known-bad addresses

Use addresses that cannot receive mail:

• Recent hard bounces with a mailbox-not-found code. Gmail's text for this is "The email account that you tried to reach does not exist" with code 550 5.1.1, per Google's SMTP error reference. In RFC 3463, X.1.1 means "bad destination mailbox address".
• Addresses at domains with no MX and no A record, or at a domain that publishes a null MX.
• Clearly made-up local parts at domains you have confirmed reject unknown recipients.

Use bounces from the last two weeks. An address that bounced a year ago may have been recreated.

Known catch-all addresses

Pick a few company domains that you have confirmed accept mail for any recipient. You can check a domain with the catch-all checker. Seed both a real person you know at that domain and a made-up address. The true answer for both is "cannot be confirmed by SMTP". Our explainer on catch-all domains covers why.

The real sample

Add a random sample from the list you actually plan to send. Seeds measure errors. The real sample measures how each tool behaves on your mix of domains. Take at least 1,000 rows if the list is large.

Shuffle seeds into the real sample, so no tool can treat them differently. Keep a private copy with a truth column.

Step 2: Map every vendor's labels to four buckets

Vendors do not share a vocabulary. One says "deliverable", another "valid", another "ok". "Risky" at one vendor includes catch-alls, role addresses and timeouts. At another, catch-all is its own label.

Before scoring, map each label to one of four buckets:

Bucket	Meaning	Typical vendor labels
Confirmed	The receiving server accepted this mailbox and rejected a made-up one	valid, deliverable, ok
Undeliverable	Mailbox or domain does not take mail	invalid, undeliverable, bad, does not exist
Catch-all	Domain accepts any recipient, so the mailbox cannot be confirmed	catch-all, accept-all, risky (accept all)
Unknown	The check did not finish: timeout, greylisting, provider that hides status	unknown, unverifiable, risky (server would not confirm)

Keep flags such as role, disposable and free provider as separate columns. They describe the address, not whether the mailbox exists. On SimpleVerifier, the API docs list each status and the reasons behind it, so the mapping is direct: valid to confirmed, invalid to undeliverable, and risky split by its reason into catch-all or unknown.

If a vendor's documentation does not tell you what a label means, ask. Our 12 questions for choosing a verifier include this one.

Step 3: Run every tool on the same file, the same hour

Upload the identical file to each tool within the same hour. Record for each:

• Start and finish time.
• Count per bucket.
• Whether the tool offered a retry for unknowns, and whether you used it.

Timing matters for two reasons. First, the world changes. Mailboxes close and domains change mail hosts. Second, servers respond differently over time. Greylisting, described in RFC 6647, deliberately answers a first attempt from an unfamiliar sender with a temporary failure. A verifier that retries later may turn an unknown into a definite answer. One that does not will leave it unknown. Neither is wrong. They are making a different trade between speed and coverage.

Speed is worth recording, but read it carefully. A large file that finishes in seconds may mean the tool used cached results or skipped the mailbox check for some domains. Ask which.

Step 4: Score the seeded rows

Score each tool on the seeds only. Four numbers matter.

Measure	How to compute it	Why it matters
False-valid rate	Known-bad rows labelled confirmed, divided by known-bad rows	These will bounce if you trust the tool
False-invalid rate	Known-good rows labelled undeliverable, divided by known-good rows	These are real people you would delete
Catch-all honesty	Known catch-all rows labelled confirmed	A confirmed result here is not backed by evidence
Unknown rate	Seeds labelled unknown, divided by all seeds	Unknowns cost review time, not bounces

The two error types cost different things. A false valid costs a bounce, and bounces count against your sender reputation. Google's sender guidelines tell senders to unsubscribe recipients with multiple bounced messages and to slow down when bounces start. A false invalid costs a lost prospect. Weigh them by your own use case. Our false positives explainer covers where each error comes from.

Worked example (illustrative numbers)

The numbers below are made up to show the method. They are not results from any real vendor.

Seed group	Rows	Tool A	Tool B
Known-bad	100	2 confirmed, 95 undeliverable, 3 unknown	0 confirmed, 91 undeliverable, 9 unknown
Known-good	100	97 confirmed, 1 undeliverable, 2 unknown	92 confirmed, 0 undeliverable, 8 unknown
Known catch-all	20	14 confirmed, 6 catch-all	0 confirmed, 20 catch-all

Tool A looks better at a glance: more confirmed and fewer unknowns. But it marked 2 known-bad rows and 14 catch-all rows as confirmed. Tool B made no false valids and labelled every catch-all honestly, at the cost of more unknowns. If bounces are your main risk, as they are in cold email, Tool B is the safer tool. If losing reachable contacts costs you more, the answer may change. The point of the test is to make that trade visible.

How many seeds you need

Small samples hide error rates. The NIST/SEMATECH e-Handbook recommends the Wilson interval for a proportion across most sample sizes. When a tool makes zero errors on n rows, the Wilson 95% upper bound simplifies to about 3.84 / (n + 3.84).

Known-bad seeds	Errors seen	95% upper bound on true false-valid rate
20	0	about 16%
50	0	about 7.1%
100	0	about 3.7%
200	0	about 1.9%

So a clean result on 20 seeds tells you very little. If you need to tell a 1% tool from a 3% tool, you need a few hundred seeds per group.

Step 5: Build a disagreement matrix on the real sample

Seeds tell you about errors. The real sample tells you where tools part ways on your data. Cross-tabulate the buckets of two tools:

	B: confirmed	B: undeliverable	B: catch-all	B: unknown
A: confirmed	agree	conflict	A is more confident	A is more confident
A: undeliverable	conflict	agree	conflict	A is more confident
A: catch-all	B is more confident	conflict	agree	different label
A: unknown	B is more confident	B is more confident	different label	agree

Two cells deserve attention:

1. Hard conflicts: one tool says confirmed, the other undeliverable. These are rare when both tools work, and they are where errors live.
2. Confidence gaps: one tool says confirmed, the other catch-all. Look at the domain. If your seed test showed it is catch-all, the "confirmed" is not backed by a mailbox check.

Group conflicts by domain and by mail host. Many disagreements cluster on a handful of providers or security gateways. Our guide on gateways like Proofpoint and Mimecast explains why those domains produce so many.

Step 6: Settle conflicts with a small controlled send

When two tools conflict, only the receiving server can settle it. Do this carefully:

1. Take a small random sample of the conflict rows, not all of them. Twenty to fifty is usually enough to see a pattern.
2. Send from a domain and inbox you use for testing, at low volume, with a plain, relevant message.
3. Record every bounce with its full SMTP reply and enhanced code.
4. Wait at least 72 hours. Gateways that accept mail first can bounce it later. RFC 5321 requires a server that accepts a message and then cannot deliver it to send a failure notice back.
5. Count a 5.1.1-style "mailbox does not exist" bounce as undeliverable. Do not count policy or reputation blocks, such as Gmail's 550 5.7.26 for unauthenticated mail, as proof about the mailbox.

No bounce after 72 hours is evidence the address accepted mail. It is not proof a person reads it.

Reading the results honestly

A fair bench test usually ends with a trade-off, not a clean winner:

• Low false valids, more unknowns or catch-alls: safer for cold email and new domains.
• Few unknowns, some false valids: more contacts to send to, more bounce risk.
• Many catch-alls marked confirmed: the tool reports the server's setting, not the mailbox.

Do not judge a tool by the single list that annoyed you. Judge it on the seeded error rates and on the conflict rows you settled with real bounces.

We ran a version of this test on ourselves. Our post on catching a verifier that marks made-up addresses valid describes the time our own tool failed it, and what we changed. If a fair test on your data shows SimpleVerifier getting something wrong, we want the row and the bounce code.

What it costs to run

A bench test of a few thousand rows is cheap. On SimpleVerifier, a pay-as-you-go pack of $9 for 2,500 verifications covers a seeded set plus a real sample. Credits never expire. Larger tests fit the $19 pack for 10,000. If you are testing the API or plan to verify every month, Pro is $29.99 a month. See pricing for the details.

Run the same file through each vendor's paid tier, not a limited sample tool, so you compare like with like.

The short version

Build an answer key before you test. Seed known-good, known-bad and known catch-all addresses into a real sample. Map every vendor's labels into four buckets. Run all tools on the same file in the same hour. Score false valids and false invalids separately, with enough seeds to make the rates meaningful. Then settle real disagreements with a small, careful send and read the bounce codes. The tool that wins is the one whose mistakes you can afford.