How to Get 23% More Revenue with a Multivariate Test (Case Study)

Trustpilot, Free Shipping or Both?

Together with the client, we wondered whether inserting a Trustpilot badge and/or indicating free shipping on the product page could improve the conversion rate and average revenue per user.

Although they may seem like harmless changes , we at Hoculus know very well that this is not the case... over time we have seen hundreds of times how even the simplest changes are capable of distorting the performance of an e-commerce for better or worse , and this case study is a clear example.

Test mode

To obtain the greatest amount of information, we chose to perform a Multivariate Test.

This typology consists of creating basic variants (modified versions of the page to be tested) and then mix them together to evaluate all possible cases.

In particular, in this case, we have created two basic variants :

• A1 - Trustpilot Trustbox, Microreview type, under the product title
• B1 - " Free Shipping " text next to the product price

Given the type of test, four overall variants are obtained:

• A0-B0 - Original unmodified version of the page (the reference point)
• A1-B0 - Trustpilot Trustbox only
• A0-B1 - Text only "Free Sedition"
• A1-B1 - Trustbox Trustpilot + Free Shipping

Technical considerations

We use Google Optimize to run this test.

Although the changes in question are very simple to make, we avoid implementing them using the graphical interface of this Tool.

When editing pages with the integrated editor on Google Optimize, bugs may occur due to the technical methods with which the changes are applied.

For this reason, we prefer to apply the changes directly on the client's site, with code written specifically, using Google Optimize and Google Analytics exclusively for the collection and analysis of the experiment data .

This way we gain greater control over the data and have no technical limitations on the complexity of variant changes .

Analysis of Results

Once the setup is complete and the test is running, we wait for each variant to get enough sessions and conversions for the data to be statistically significant.

Once the data collection phase is complete (in this case approximately 17 days ), it is possible to analyse the data :

The table above shows the actual values ​​obtained by the variants during the experiment, thanks to which we can finally identify the winning variant.

In particular, the last column reports the values ​​of Revenue calculated per session (EPV or EPS), that is, the Revenue obtained from the single variant divided by the number of corresponding sessions; in this way, the average value that each session generates on the site is obtained.

The reference data is the combination A0-B0 which represents the original version of the site without any modifications.

Looking at the last column it is therefore evident that the A1-B0 combination generated more revenue than all the others, achieving a +23% on the EPV of the original version and that, once definitively implemented, it will be reflected in the overall turnover.

Comparing the Revenue Per Session of the original variant and those of A1-B0, we can see that the latter will generate approximately €1400 more revenue every 17 days than the unmodified version.

Equally interesting is the data recorded by A0-B1 ... Who could have ever imagined that the text "Free Shipping" could worsen performance?

Yet the data speak clearly : both variants containing that sentence, A0-B1 and A1-B1 respectively, have recorded a more or less significant worsening compared to the reference version.

Conclusions

We chose to publish this test as a case study not so much for the percentage improvement, nor for the technical complexity; our aim is to demonstrate how, sometimes, even the most apparently innocuous changes can cause permanent and unexpected losses of revenue while other changes can exceed expectations .

For this reason, we often recommend our customers to allow us to test the changes through A/B Tests or Multivariate Tests , so as to be sure that the interventions carried out actually have a positive impact on e-commerce.

PS. Creating the variants, implementing this test and analyzing the data took 30 minutes of our Shopify Ecommerce Boost service.