Optimizing User Experience with Real-Time A/B Testing

A Case Study from Siemens and Groundfog

Join us as we explore a compelling case study of collaboration between Siemens and Groundfog. Discover the imperative for real-time A/B testing in optimizing user experiences and learn about the technical intricacies of our AWS-based solution, leveraging services like Kinesis and Athena to capture and visualize user interactions in real time.

Robin Schoenmaeckers - January 23, 2024

In today's fast-paced digital landscape, understanding user behavior isn't just a luxury — it's a necessity. Every interaction, every click, holds valuable insights that can drive decisions and shape strategies. Siemens, a long-standing partner of Groundfog, recognized this early on. The quest for continuous improvement and adaptation led them, alongside our team, to explore the dynamic world of A/B testing.

With Groundfog's foundation of trust with Siemens and our robust affiliation with Amazon Web Services (AWS), an opportunity unfolded. Siemens envisioned a real-time A/B testing solution that leverages AWS's serverless architecture, benefiting from our advanced AWS partnership. This isn't just about presenting two versions of a webpage; it's about harnessing real-time user feedback to create more impactful digital experiences.

As we unpack this solution, we'll shed light on its technical components and emphasize the role of AWS's Quicksight in refining Siemens' data-driven approach.

The Need for Real-Time A/B Testing

In the age of instant gratification, where users expect seamless experiences tailored to their preferences, businesses cannot afford to wait. Traditionally, decisions around website changes or digital strategies might have been based on past data or extensive market research. However, as digital interactions multiply and user behaviors evolve, there's a growing need for real-time insights. This is where real-time A/B testing steps in.

User Experience is King:
A/B testing isn't just about choosing between two color schemes or headlines; it's about optimizing the user journey. Businesses like Siemens can pinpoint what resonates best with their audience by comparing different versions of a webpage or application feature. Real-time feedback ensures that any adjustments to improve the user experience can be made swiftly, avoiding prolonged exposure to less-effective content.

Staying Ahead of the Curve:
The digital landscape is fiercely competitive. To maintain an edge, businesses must be agile, making informed decisions as quickly as challenges or opportunities arise. Real-time A/B testing allows Siemens to do just that. By receiving immediate data on user interactions, Siemens can adapt and refine its strategies on the fly, ensuring they're always one step ahead of the competition.

Reducing Guesswork:
Decision-making backed by real-time data reduces reliance on assumptions or outdated information. With a real-time A/B testing solution, Siemens can take actionable insights directly from their audience's current interactions, leading to more informed and accurate strategies.

Real-time A/B testing isn't just a tool—it's an evolving strategy. It empowers businesses like Siemens to remain dynamic, responsive and ever-focused on delivering the best possible experience to their users.

The Groundfog Solution

A/B testing can either be done on the client side or on the server side. While client side testing can have some advantages like easier implementation or better flexibility, it comes with significant tradeoffs like flickering (momentary display of the original version of a webpage before the test version is shown to the user), negative user performance impact and complications when dealing with SPAs (Single Page Applications). For these reasons, we opted for server-side A/B testing to deliver a smoother, more consistent user experience, which minimizes disruptions that can be off-putting for users.

Configuring an A/B Tests

The first step of the solution is to empower the editors to configure the A/B test themselves conveniently using the content management system they are used to. This feature must integrate seamlessly into their existing tools rather than adding an additional tool to their set. In this case, we enabled them to define the test and the traffic routes to each page in Adobe Experience Manager.

After finalizing and publishing the test setup, we developed an integration with AWS that automatically stores the A/B test configuration in a DynamoDB table as a reliable and scalable storage solution. This integration isn't exclusive to AEM; we've designed it to be adaptable, meaning it can easily be layered onto other customer CMS platforms.

Assigning and Loading A/B Tests

In the browser, each user gets assigned a unique and consistent user ID stored in the cookies. In our case, this is managed by our tracking solution, Adobe Analytics, but this could also be any arbitrary cookie solution.

When the user requests a page, the request goes through AWS CloudFront, which triggers Lambda@Edge functions.

We then leverage the power of Lambda@Edge to determine to which test group a user belongs, load the A/B testing configuration for that page, and serve him the correct page. This approach is based on the excellent blog post “A/B testing on AWS CloudFront with Lambda@Edge”, where you can find a more in-depth technical explanation of how this works. In a nutshell:

By implementing A/B testing server-side using CloudFront, we get the following benefits:

Capturing Real-Time User Behavior

Configuring an A/B test is insufficient, as we also need to know how the different pages perform. On top of this, our client wanted to get real-time insights. Our approach is loosely based on AWS' blog post “Create real-time clickstream sessions and run analytics with Amazon Kinesis Data Analytics, AWS Glue, and Amazon Athena” and is the following:

This approach can scale very well for high-traffic websites due to the serverless nature of each individual component. Thanks to Amazon Athena serverless query engine, we can quickly get insights from big amounts of data using SQL.

Bringing Data to Life with Amazon QuickSight

Collecting data is a pivotal process, but the real essence is in its access, utilization, and interpretation to derive actionable insights. Selecting the appropriate Business Intelligence (BI) tool was a fundamental segment of our strategy. Our choice was Amazon QuickSight, and here's why:

Connecting Athena to QuickSight was super straightforward and in a very short amount of time we were able to create QuickSight dashboards that give a real-time view of performance metrics for two versions of the same page, showing insightful metrics like bounce rate, conversion rate, clickthrough rate, or average session length.

Innovation Meets Determination: A Collaborative Journey

In a world driven by digital interactions, having the tools to swiftly decipher user behavior can shape the future of businesses. What began as a collaborative vision between Groundfog and Siemens has now materialized into a state-of-the-art cloud-native A/B testing mechanism, ensuring that the digital experiences we provide are not only aligned with user expectations but also continually refined based on real-time feedback.
By leveraging AWS's robust suite of tools, our journey has proven the scalability and efficiency of serverless architectures. From configuring the tests seamlessly in the CMS, capturing user interactions in real-time, and ultimately visualizing the gathered data through QuickSight, every step was about ensuring simplicity without compromising precision and scalability.

Beyond the technology and tools, this journey underscores the power of collaboration and shared vision. Our alliance with Siemens, strengthened by our expertise and affiliation with AWS, showcases what's possible when innovation meets determination.

As we move forward, we remain committed to exploring and implementing solutions that push the boundaries of what's possible in the digital world, ensuring that our clients are always at the forefront of digital evolution.

Missing out on real-time insights for your digital strategy?

Don't gamble with your strategy. Gain real-time insights to boost conversions and elevate user experiences.



  • Source: https://aws.amazon.com/de/quicksight/