Most email tests are inconclusive because the sample is too small, the change is too minor or the wrong metric picks the winner. A simple discipline fixes all three.
1. Start with a hypothesis
Write one sentence: "If we do X, then Y will improve because Z." For example: "If we send the replenishment reminder 7 days before run-out instead of on run-out day, the second-order rate will rise because customers have time to act."
2. Test one variable
Change a single thing at a time: timing, offer, headline, call to action, layout. If you change several, you will not know what caused the result.
3. Pick a metric that matters
- Prefer clicks, conversion or revenue per recipient to opens, which are distorted (why).
- For flows, the real outcome is often the second-order rate.
4. Mind sample size
With a small audience, small differences are noise. Either run the test longer, accumulate results across sends, or test bolder changes whose effect is large enough to see.
5. Test in campaigns and in flows
Campaign tests give quick answers about subject lines and creative. Flow tests pay off for longer because every future subscriber passes through the winner. Good flow tests to start with: reminder timing in the replenishment flow, the number of emails in a welcome series, and the point at which you introduce an offer.
6. Log every test
Keep a simple table: date, hypothesis, variants, audience, metric, result, decision. After a few months it becomes your best playbook.
Common mistakes
- Declaring a winner after a few hours or a handful of clicks.
- Testing tiny cosmetic changes on a small list.
- Choosing the winner by open rate.
- Rolling out a winner without checking unsubscribes and complaints.
Quick answers
What should I A/B test in email marketing?
Start with elements that change behavior: the offer, timing, the call to action and the main message. Subject lines and send times are fine too, but judge them on clicks or revenue, not opens.
How long should an email A/B test run?
Long enough to reach a meaningful number of recipients and conversions in each variant. Small lists need longer tests or bigger changes to show a real difference.
Can I A/B test flows in Klaviyo?
Yes. You can split a flow into variants and compare performance over time, which suits tests of timing, content or offers.
Tehseen Arbab
Retention and lifecycle marketer for DTC supplement and coffee brands. I led retention for a supplement brand whose monthly run-rate grew from about $500K to $700K, and I work in Klaviyo, Skio, Recharge, Meta Ads and Shopify.