Unfortunately, you didn’t get it right. Variation 1 was the winner.

Winner: Variation 1
Start: 19/04/22
End: 03/05/22
Duration: 15 days
Control Visitors: 152,407
V1 Visitors: 150,259

Highlights

• We observed an uplift in the primary metric of Overview page visits of 11.9% vs. control at full statistical significance. There is clear evidence that the new homepage layout encourages a higher proportion of visitors to navigate to overview pages for key products including PCA, Loans, and Credit Cards.
• The uplift was higher on mobile devices, with a 21.7% uplift in overview page visits, compared with a 5.3% uplift on desktop. We believe most of the uplift was driven by the new quick links content inserted below the hero banner. 
• The other key metric for this test was homepage exit rate. We observed a -15% reduction in overall unique exits from the homepage, -12% on mobile and -8.5% on desktop. This indicates that the new layout was much “stickier” and encouraged users to explore more of the content on the page than the control experience.
• We observed a high level of interaction with the newly introduced “Quick links” content slot below the hero banner on the homepage. On mobile, total click rate for this section was 2.46%, with the most popular links being “current accounts” (1.03%)  and “savings accounts” (0.44%). We also saw a significant level of interest in the “Help & support” link (0.42%) which indicates that users are looking for content other than products within this content slot.

Conclusion & recommendations

There is clear evidence that the new homepage layout has a positive impact on navigation to key product overview pages ( 11.9%) as well as help & support content compared to the control experience, driven primarily by the introduction of quick links content.

A couple of usability issues were observed such as the non-clickable hero banner on mobile and placement of quick links content on desktop, which can be tweaked in follow up variations of the design. There is also an opportunity to personalise key elements of the homepage, primarily the quick links which appear to be a key lever in terms of navigation.

We can therefore recommend that the new layout is deployed to 100% of visitors across both desktop and mobile devices, at which point we can begin follow up experiments to improve usability of the page and introduce various levels of personalisation for both logged in and logged out visitor segments.  Further follow up activity should also be considered when the Trustpilot content is ready to be introduced to identify the optimal placement for this content on the homepage and whether we can establish a correlation between displaying reviews content and higher app start rate for key products.