What is A/B Split Testing
Posted: Sun Jan 05, 2025 9:57 am
In marketing, A/B split testing is often applied to things like email subject lines, landing pages, calls-to-action and more.
For example, you could send out two identical emails with mom database different subject lines to a randomized list of contacts in your database to learn which subject line encourages the most users to open the email.
You should document every split test you execute to keep track of how each is performing. Different documentation systems will work better for different marketers, but ideally, your documented split test should look something like this:
Hypothesis: If I add the word “free” to my email subject line, I’ll boost open rates by 10 percent. (Side note: This one’s debunked! Subject lines with the word “free” in them tend to get significantly lower engagement rates than those without.)
Variable A: Subject line “Get your e-book on Inbound Marketing”
Variable B: Subject line “Get your free e-book on Inbound Marketing”
Sample size: 500 random blog subscribers
Duration: 1 email send
In this example, variable A is the “control” variable, whereas variable B is the “challenger.” The sample size is an audience of email contacts with similar attributes — they’re all subscribed to your blog — but to ensure an unbiased result, you’ll need to send out the control and the challenger email to random contacts within that audience.
HubSpot Enterprise includes an A/B split testing tool that helps you determine your sample size and automatically splits your audience for you. If you’re testing something that doesn’t necessarily have a fixed audience such as a landing page, just make sure you define the right length of time to run your test.
Typically, we’d recommend running a split test for at least three to six months. The more data you can gather, the more confident and accurate a conclusion you can draw from the final result.
For example, you could send out two identical emails with mom database different subject lines to a randomized list of contacts in your database to learn which subject line encourages the most users to open the email.
You should document every split test you execute to keep track of how each is performing. Different documentation systems will work better for different marketers, but ideally, your documented split test should look something like this:
Hypothesis: If I add the word “free” to my email subject line, I’ll boost open rates by 10 percent. (Side note: This one’s debunked! Subject lines with the word “free” in them tend to get significantly lower engagement rates than those without.)
Variable A: Subject line “Get your e-book on Inbound Marketing”
Variable B: Subject line “Get your free e-book on Inbound Marketing”
Sample size: 500 random blog subscribers
Duration: 1 email send
In this example, variable A is the “control” variable, whereas variable B is the “challenger.” The sample size is an audience of email contacts with similar attributes — they’re all subscribed to your blog — but to ensure an unbiased result, you’ll need to send out the control and the challenger email to random contacts within that audience.
HubSpot Enterprise includes an A/B split testing tool that helps you determine your sample size and automatically splits your audience for you. If you’re testing something that doesn’t necessarily have a fixed audience such as a landing page, just make sure you define the right length of time to run your test.
Typically, we’d recommend running a split test for at least three to six months. The more data you can gather, the more confident and accurate a conclusion you can draw from the final result.