Skip to the content
VERMA

Shalini Verma · AI Marketing Engineer · Toronto, Canada

Enquiries

A/B test calculator

Two calculators in one: test whether the difference between a control and a variant is real, or work out how many visitors you need before you start. Built for landing pages, emails and holdout groups.

Significant at 95% confidence

Rate A
5.00%
Rate B
6.00%
Relative lift
+20.0%
p-value (two-sided)
0.0283

95% confidence interval for B − A: 0.11% to 1.89% (percentage points). Two-proportion z-test; decide your sample size before you start and do not stop the test the moment it turns significant.

How to use it

  1. Pick “Is my result significant?” or “How many visitors do I need?”.
  2. Enter visitors and conversions for each group, or your baseline rate and the smallest lift worth detecting.
  3. Read the verdict, p-value and confidence interval, or the visitors needed per group.

Questions

What does statistically significant mean in an A/B test?

It means a difference as large as the one you saw would be unlikely if the two versions truly performed the same. At 95% confidence the p-value is below 0.05. It does not tell you the difference is large or that it will last.

Which test does this calculator use?

A two-sided two-proportion z-test with a pooled standard error for the p-value, and an unpooled 95% interval for the difference between the rates. Sample size uses the standard formula for two proportions at 5% significance and 80% power.

Can I stop the test as soon as it is significant?

No. Checking repeatedly and stopping at the first significant reading inflates false positives. Decide the sample size before you start, run to that size, then read the result once.

How is a holdout test different from an A/B test?

A holdout keeps a share of people from receiving the campaign at all, so you measure the campaign’s true lift instead of comparing two versions. The maths is the same; the smaller group decides how long you need to run.

Read the method

Field notes behind this tool.

Measurement4 min readAd Creative Testing Against a Control: A Practical SystemTest ad creative the repeatable way: one variable at a time, every variant against a control, a fixed sample, and a log so each batch starts from a winner.
Lifecycle4 min readHoldout Tests for Lifecycle Marketing: Prove the LiftRun a holdout test on lifecycle email: pick the size, randomise, measure lift against people who got nothing, and report it with a confidence interval.
AI agents4 min readLLM-as-Judge for Marketing Copy: Rubric and Golden SetScore AI-written ads and emails with an LLM judge: the rubric to write, the golden set to test against, and the regression checks that catch drift.

More free tools

AI share of voicePaste which brands ChatGPT, Perplexity and Gemini cite for your prompts and get each brand’s share of AI-answer citations, overall and per engine.
CAC and LTV calculatorCalculate customer acquisition cost, lifetime value, the LTV:CAC ratio and CAC payback months from spend, customers, revenue, margin and churn.
SPF and DMARC generatorCreate correct SPF and DMARC TXT records for your domain, with policy, reports and alignment options, a lookup check and a plain-English explanation.
Hook checkerCheck a headline, subject line or ad hook against five persuasion patterns, from loss framing to social proof, with the research behind each.

Introductions

By appointment. Full-time, contract or advisory.

Currently leading AI marketing at Imagera AI.Open to a select full-time role leading AI marketing or GTM engineering.Advisory and contract work by arrangement.

What would we explore?
Start a conversationRésumé (PDF)LinkedIn

Roles considered: AI Marketing Engineer · Lead · Head of Growth · AI products · GTM Engineering Lead

shaliniworks24@gmail.com · Toronto, Canada · Remote or hybrid across Canada and the US