How to get 23% more turnover with a multivariate test (study case)

Better Trustpilot, Free Shipping or both?

Together with the customer, we wondered if the inclusion of a Trustpilot badge and / or the indication of free shipping on the product page could improve the conversion rate and average revenue per user.

Although they may seem harmless changes , we at Hoculus know very well that this is not the case ... over time we have seen hundreds of times how even the simplest changes are capable of upsetting the performance of an e-commerce for better or for worse , and this case study is a clear example of this.

Test mode

To get the most information, we have chosen to perform a Multivariate Test.

This typology consists of creating basic variants (modified versions of the page to be tested) and then mix them together in order to evaluate all possible cases.

In particular, in this case, we have created two basic variants :

• A1 - Trustbox Trustpilot, of type Microreview, under the product title
• B1 - " Free Shipping " text next to the product price

Given the type of test, four overall variants are obtained:

• A0-B0 - Original unmodified version of the page (the landmark)
• A1-B0 - Trustpilot Trustbox only
• A0-B1 - "Free Sedition" text only
• A1-B1 - Trustbox Trustpilot + Free Shipping

Technical considerations

We use Google Optimize to carry out this test.

Although the changes in question are very simple to make, we avoid implementing them using the graphical interface of this Tool.

By modifying the pages with the integrated editor on Google Optimize, bugs may occur due to the technical ways in which the changes are applied.

For this reason we prefer to apply the changes directly on the customer's site, with the code written ad hoc, using Google Optimize and Google Analytics exclusively for the collection and analysis of the experiment data .

This gives us more control over the data and we have no technical limitations on the complexity of variant changes .

Analysis of the results

Once the set-up is complete and testing has started, we wait for each variant to get enough sessions and conversions for the data to be statistically relevant.

At the end of the data collection phase, (in this case about 17 days ) it is therefore possible to analyze the data :

The table above shows the real values ​​obtained from the variants during the experiment, thanks to which we can finally identify the winning variant.

In particular, the last column shows the Revenue values ​​calculated per session (EPV or EPS), or rather the Revenue obtained from the single variant divided by the number of corresponding sessions; by doing so you get the average value that each session generates on the site.

The reference data is that of the A0-B0 combination which represents the original version of the site without any changes.

Looking at the last column, it is therefore evident that the A1-B0 combination has generated more revenue than all the others, realizing a + 23% on the EPV of the original version and that, once implemented definitively, it will be reflected in the overall turnover.

By comparing the Revenue Per Session of the original variant and those of A1-B0, it can be deduced that the latter will make about € 1400 more revenue every 17 days than the unmodified version.

Equally interesting is the data recorded by A0-B1 ... Who could have ever imagined that the text "Free shipping" could worsen the performance?

Yet the data speak for themselves : both variants that contain that sentence, respectively A0-B1 and A1-B1 , recorded a more or less incisive worsening compared to the reference version.

Conclusions

We have chosen to publish this test as a case study not so much for the percentage improvement, nor for the technical complexity; our aim is to demonstrate how, at times, even the most innocuous changes can cause permanent and unexpected loss of revenue while other changes can exceed expectations .

For this reason we often advise our customers to allow us to test the changes through A / B Tests or Mutivariate Tests , so as to be sure that the interventions performed actually have a positive impact on e-commerce.

PS. It took 30 minutes of our Shopify Ecommerce Enhancement service to create the variants, implement this test, and analyze the data.