How to create an A/B subject line test

beehiiv offers you the ability to test your subject line with a small subset of your audience before sending your email to all your subscribers. This allows you to send the best subject line to your audience, to help boost your open rate. Watch this video to learn how. 

 

 

To set up a test within a new post, navigate to Configure —> Email Settings. Add in the first version of your subject line that you want to test, and then click into Create A/B Test:

A new field will appear for a second subject line. Once you've added the second version of your subject line to test, you'll be able to add another test version.

Once you've created the different versions of your subject line, click into settings to set up the details of the test. 

 

 

Here you can view your Duration and Sample Size.

Duration: This is how long your test will run for. The duration defaults to 180 minutes, but you can choose any length of time between 5-240 minutes. We will run the test for this duration to determine the best subject line. Once the test is complete, we will send the winning subject line to the remainder of your audience. 

Note: Once you set up your duration and start your test, you are not able to end your test early. 

Example: If you set the duration to 180 minutes, we will send the test versions of your subject line to the subscribers in your sample size. After 180 minutes, we will determine which subject line performed best. That winning subject line will be sent to all of your other subscribers at that time. 

 

Sample Size: This is how many subscribers will be include in your test. We default to 20%, but you can choose any size between 1-100%. Based upon the percent you choose, we will automatically show you how many users fall into that test group. We will then equally split that full test group into smaller groups that will each receive one version of your subject line. 

Example: You have 100K subscribers total, and you set the sample size to 20%. We will run the A/B subject line test with 20K subscribers. If you have 2 versions of the subject line, we will send each version to 10K users. After your test duration is complete, the remaining 80K subscribers will receive the winning subject line. 

Once you have completed setting up the test details, you can click Publish Now. We will send the test versions to the specified audience size. Once the test is complete, we will determine if there is a statistically significant outcome for the winning subject line. If not, we will simply choose the subject line with the higher opens. This winning subject line will be sent to the remaining of your subscribers.