Elke e-mail-marketeer heeft ervaring met A/B-testen. Maar wat is een A / B-test? Waarom test je eigenlijk? En wat moet een A/B-test doen om succesvol te zijn? In dit artikel laten we zien hoe een A/B-test in 5 stappen kan worden opgezet.

De definitie van een A / B-test is:

“Een vorm van gesplitste tests waarbij twee varianten van bijvoorbeeld een (opt-in) pagina of advertentie worden getest ten opzichte van elkaar om de conversieratio te verhogen.”

In dit artikel gaat het specifiek over het A / B testen van e-mails. Zoals de definitie aangeeft, is het doel van een A / B-test om je resultaat te verbeteren. In het geval van e-mail kan dit tot verschillende resultaten leiden, zoals een toename van het aantal openingen, klikken of verkopen.

Stap # 1: Bepaal waarom en wat je gaat A / B testen.

Het is belangrijk om van tevoren te bepalen waarom je wilt testen. Wil je bijvoorbeeld de omzet verhogen? Kies vervolgens een e-mail die het meest bijdraagt aan je commerciële doel of gewoon een e-mail die nog steeds veel te krijgen is.
Natuurlijk kunt u alles in alle soorten e-mails testen, maar de ene heeft een grotere impact dan de andere. Daarom is het belangrijk om te bepalen welk aspect u wilt testen. Stel dat u het aantal klikken op de e-mailknop wilt verbeteren. Vervolgens kunt u de call-to-action testen, maar ook de grootte of kleur van de knop. Daarom moet je van tevoren weten wat je wilt testen.

Step # 2: Consider which item(s) you will test you at the same time.

At PI marketing we always choose to limit ourselves to testing one item: an A / B test and no multivariate test. We do this because you need many more recipients for multiple items, so it will take longer until your test is significant.

However, you can have multiple results at once. For example: you can test the subjectline and the button. You can also choose to split this into two different tests: you first test the subject and then the button.

Step # 3: Make a plan for your test

Before you begin your A / B test, it’s useful to write a plan which processes your foundings of steps #1 and #2: Why you are going to test? What to test? And what items to test? You will also write down your expected result: the hypothesis.

Your hypothesis could be: “When adjusting the color of the button from orange to red, I expect 20% more clicks.”.

Step # 4: Set up the A / B test and monitor it accurately

Especially in the first days of an automatic campaign it is important to check your campaign regularly. How quick are you booking results? Is everything still going smooth? This of course depends on the size of your test.

For example, you can keep track of results in an Excel file where you set the formulas to calculate significance, conversion, etc. Or you can use an online tool like Google Optimize or VWO.

Step # 5: Review and document the results

In the previous step, you have tracked your results from the A / B test. With this step, you will review the results. By keeping track of the results of the test, keep an overview of what has already been tested and what not.

It’s useful to sort these results on: tested subjects, layouts or call-to-actions. You write down what has been accurately tested and what was the result and of course the winner! But also write down striking results you did not expect. When something is significant depends on your hypothesis. Usually we speak of a successful test at 90% significance, however, you can go for 95% or even 99%. When you pick 99% significance you have a reliable test and the result of the test is no coincidence.

Examples: What can you test?

We can be brief, everything can be tested. However, some elements will have more impact than others and therefore step #2 is so important. We A / B continuously test for our customers. For instance: buttons, colors, subjectlines or sending time. Below are some examples of the elements we test for customers.

– Shipping Time After Shopping Cart *

At this A / B test, we wanted to test whether the email sent earlier had more impact. The test consisted of the original shipment time of 30 minutes (A) after leaving the page opposite 20 minutes (B) after leaving the page. B came here as a winner: 93% at open and 99% on the number of transactions (based on the number of shipments)

* This is the email you receive after leaving the shopping cart

– Less choice in supply

In the e-mail in which we display the last consumer searches (After Search Mail *), we showed the last 5 by default. For this we have used an A / B test by adjusting the offer to 3 searches. This showed that reducing the number of choices from 5 to 3 resulted in a conversion increase of 66% with a significance of 99.12%. That’s how you see that more choices are not always better.

* This is the email that one receives after making a search

– Subjectline

A subjectline is of course something that can be infinitely tested and what is happening continuously. An example of what has won 99% significance is to name the discount in the subjectline. Another example is, “These are the most special destinations,” versus “see the most beautiful destinations”: the last subject has won 68% more open.


At PI marketing, the A / B testing of the emails is a top priority to continue to improve our email campaigns and achieve better results. How do you test A / B tests and what outstanding results have you encountered? Let us know below!