Build an A/B Test Plan for Cold Email Subject Lines

You are a sales operations lead at a digital marketing agency. You design outbound experiments that produce signal, not vanity. You know enough stats to avoid declaring winners on 50 opens.
Agency: [AGENCY_NAME] — [SERVICES] | Current control subject: [CURRENT_SUBJECT] | Current control open rate: [CURRENT_OPEN_RATE] | Current control reply rate: [CURRENT_REPLY_RATE] | Sending tool: [SENDING_TOOL] | Weekly sending capacity: [WEEKLY_SENDS] | Hypothesis to test: [HYPOTHESIS]
Design a complete A/B test plan for subject lines. Include 2 challenger subjects, sample size, duration, primary + guardrail metrics, kill criteria, and the winner decision rule.

- Primary metric: reply rate, not open rate (opens lie post-MPP)
- Guardrail metrics: unsubscribe rate, bounce rate, spam complaint rate
- Sample size must be calculated to detect a minimum lift of 2 percentage points at 90% confidence
- Each variant gets equal volume from the same mailbox pool
- Duration capped at 14 days max
- Kill criteria if guardrail breached
- All subjects ≤6 words, no spam triggers

Output in sections: 1) Hypothesis (clean restatement). 2) Variants (Control / Challenger A / Challenger B, with the lever each tests). 3) Sample size & duration (with the math). 4) Primary + guardrail metrics with thresholds. 5) Kill criteria. 6) Winner decision rule. 7) What to do with the loser.

• [AGENCY_NAME] — Your agency
• [SERVICES] — Services sold
• [CURRENT_SUBJECT] — Current control subject line
• [CURRENT_OPEN_RATE] — Current open rate, e.g. 28%
• [CURRENT_REPLY_RATE] — Current reply rate, e.g. 3.1%
• [SENDING_TOOL] — Instantly, Smartlead, Apollo, etc.
• [WEEKLY_SENDS] — Total sends per week across all mailboxes
• [HYPOTHESIS] — What you believe will lift reply rate and why

Agency: ClickReef — Google Ads for ecom | Control: 'pmax audit for [company]' / 26% / 2.4% | Tool: Smartlead | Weekly sends: 2,000 | Hypothesis: naming a specific ROAS number in the subject will lift replies because DTC founders audit by numbers

1. Hypothesis: A subject naming a concrete ROAS number lifts reply rate vs a generic audit offer.
2. Variants — Control: pmax audit for [company] (generic). A: pmax roas under 2? (question + number). B: [company] branded roas at 1.4 (specificity).
3. Sample size: ~1,150 sends per variant to detect +2pt reply lift at 90% conf (baseline 2.4%). At 2,000/wk total = ~667/variant/wk = 12 days to power.
4. Primary: reply rate. Guardrails: unsubscribe <0.8%, bounce <2%, spam <0.1%.
5. Kill: any guardrail breached for 2 consecutive days = pause that variant.
6. Winner: variant with highest reply rate at day 12 AND a confidence interval not overlapping control's.
7. Loser: archive subject + lever notes in playbook; don't recycle for 90 days.

• Open rates are noise post-Apple MPP — judge on replies and positive replies only.
• Never test more than 2 challengers at once — you'll never reach significance on agency volume.
• Save losing subjects + the lever they tested in a 'do not retry' doc so you stop repeating dead bets.