The simple guide to A/B Testing in Episerver/Optimizely

Here at Niteco, we know that the development of a website doesn’t end when said site goes live. With offerings like our 24/7 Application Managed Services (AMS), we take care of our customers’ projects even after launch. However, after launch is also when the website editors’ real work begins. And one of their main tasks is the same as a developer’s main task: Optimize, optimize, optimize.

A website editor on an Episerver/Optimizely website, unlike a developer, doesn’t need to be proficient in HTML. To do their jobs well, however, they need to be proficient in the possibilities that Episerver/Optimizely’s backend offers. One of those possibilities is A/B Testing.

What is A/B Testing?

The principle behind A/B Testing is simple and probably known to those of you familiar with Digital Marketing and Website Content Management. The idea is to find out which variation of a piece of content, a call to action and other elements users respond to the most – basically, which option will lead to the most conversions. The variation in question could be a different design for a button, different visuals, different copy or any other change that you think might affect conversion numbers.

Within the Episerver/Optimizely ecosystem, A/B Testing is implemented through an add-on for the Digital Experience Platform. That means you have to separately install the functionality, but it requires no additional or extended license.

The test itself pitches a piece of content, the control, often a piece that has been on your site for a while, against a new piece, called the challenger. Half of the visitors coming to the page that contains this content will get presented with the control, while the other half will be shown the challenger. The two are then put in a competition to see which option will lead to more desired actions by the users.

To kick things off, start with a published page within your site. This page will be your control.

Now, make the change you want to test, creating a draft page in the CMS. Then click on Publish and select A/B Test changes. The A/B test page will appear, showing thumbnails of both options next to each other. This is where you can configure your test.

What do you need to A/B test?

In the Test goal box, you can leave a short description of what you are trying to achieve with this test. This is just for the information of yourself and other editors within the system. Next, you have to tell the system what the desired action for visitors to the page is. Should they click a link and navigate to a specific other page? Should they download a file? Should they fill out a form? Should they stay on the page for a set period of time? The possibilities are vast. If a user takes said action while the test is running, they will count as a conversion and add their numbers to the test total. You can also add more than one goal to a single A/B test and prioritize their importance to your purposes.

After setting these criteria, you can choose how many visitors you want to involve in this test. Of course, if you choose 100 percent, all users will take part and you will receive more data. Unless you are very unsure about the performance of your challenger and suspect it may significantly decrease conversions, there is barely any reason to only limit yourself to using smaller amounts of visitors.

Finally, choose the duration of your test and the confidence level of statistical significance. You can now start your test. However, remember that you can’t change any content or any test configurations while the test is running. If you need to make a change, you have to cancel the test and start a new one.

Viewing your A/B Test and choosing a winner

While the test is running, you can view its numbers and performance by clicking on View test in the yellow notification bar. It will show you exactly what data has been collected so far. However, don’t forget that statistical significance is only applied at the end of the test.

After the time period you set for the test’s duration, you will be presented with the final results. In the test view, the winner – i.e. the option with the highest conversion rate – will be highlighted for you. A system admin can set the winner to be published automatically after a test is finished. If that is not the case, you can publish yourself by choosing Pick the winner. You can even do so before the test is finished if it becomes clear that one option is the obvious winner.

Congratulations, you’ve just taken an important step in optimizing your site’s content. However, don’t rest on your laurels. You should keep continually improving in order to achieve the highest conversion rates you can.

If you need help with the continuous maintenance and improvement of all aspects of your site, come to Niteco. Not only does our AMS offering perfectly complement Episerver/Optimizely DXC support, we also offer content management and governance services. Contact us now to find out more!

Frequently Asked Questions about A/B Testing in Episerver/Optimizely

What is server-side AB testing? 

Server-side A/B testing is a kind of experimentation where the different versions of a test are implemented directly on the web server itself. 

What is server-side vs client side A B testing? 

Client-side a/b testing is best used when you are deploying an internal team of marketers to run experiments without the need for them to touch coding. Server-side a/b testing is best used when the scope of an experiment is bigger, for example when changes need to be made at the product level. 

If you want to run experiments without requiring your internal marketing team to use code, client-side A/B testing is ideal. However, if you need to implement larger-scale changes, such as modifications at the product level, server-side A/B testing is the preferable option. 

How to Conduct A/B Testing in General? 

• Choose a single variable to test. 
• Define your objective. 
• Develop a "control" and a "challenger" version. 
• Randomly divide your sample groups equally. 
• Determine the required sample size (if applicable). 
• Decide on the level of significance you need in your findings. 
• Ensure that you are conducting only one test per campaign at a time. 

What statistical test is used for AB testing? 

An A/B test is a type of statistical hypothesis testing, in which a hypothesis is formulated regarding the correlation between two data sets. The data sets are then compared to each other to ascertain if there is a statistically significant correlation or not. 

What is A and B user testing? 

In A and B user testing or A/B testing, users are exposed to one of two versions of a design, namely A or B. Subsequently, the conversion rates for each version are measured to determine which one is more effective. Another form of testing is multivariate testing, where three or more variations of a design are examined.