In today's fast-paced digital landscape, understanding user behavior isn't just a luxury — it's a necessity. Every interaction, every click, holds valuable insights that can drive decisions and shape strategies. Siemens, a long-standing partner of Groundfog, recognized this early on. The quest for continuous improvement and adaptation led them, alongside our team, to explore the dynamic world of A/B testing.

With Groundfog's foundation of trust with Siemens and our robust affiliation with Amazon Web Services (AWS), an opportunity unfolded. Siemens envisioned a real-time A/B testing solution that leverages AWS's serverless architecture, benefiting from our advanced AWS partnership. This isn't just about presenting two versions of a webpage; it's about harnessing real-time user feedback to create more impactful digital experiences.

As we unpack this solution, we'll shed light on its technical components and emphasize the role of AWS's Quicksight in refining Siemens' data-driven approach.

The Need for Real-Time A/B Testing

In the age of instant gratification, where users expect seamless experiences tailored to their preferences, businesses cannot afford to wait. Traditionally, decisions around website changes or digital strategies might have been based on past data or extensive market research. However, as digital interactions multiply and user behaviors evolve, there's a growing need for real-time insights. This is where real-time A/B testing steps in.

User Experience is King:
A/B testing isn't just about choosing between two color schemes or headlines; it's about optimizing the user journey. Businesses like Siemens can pinpoint what resonates best with their audience by comparing different versions of a webpage or application feature. Real-time feedback ensures that any adjustments to improve the user experience can be made swiftly, avoiding prolonged exposure to less-effective content.

Staying Ahead of the Curve:
The digital landscape is fiercely competitive. To maintain an edge, businesses must be agile, making informed decisions as quickly as challenges or opportunities arise. Real-time A/B testing allows Siemens to do just that. By receiving immediate data on user interactions, Siemens can adapt and refine its strategies on the fly, ensuring they're always one step ahead of the competition.

Reducing Guesswork:
Decision-making backed by real-time data reduces reliance on assumptions or outdated information. With a real-time A/B testing solution, Siemens can take actionable insights directly from their audience's current interactions, leading to more informed and accurate strategies.

Real-time A/B testing isn't just a tool—it's an evolving strategy. It empowers businesses like Siemens to remain dynamic, responsive and ever-focused on delivering the best possible experience to their users.

The Groundfog Solution

A/B testing can either be done on the client side or on the server side. While client side testing can have some advantages like easier implementation or better flexibility, it comes with significant tradeoffs like flickering (momentary display of the original version of a webpage before the test version is shown to the user), negative user performance impact and complications when dealing with SPAs (Single Page Applications). For these reasons, we opted for server-side A/B testing to deliver a smoother, more consistent user experience, which minimizes disruptions that can be off-putting for users.

Configuring an A/B Tests

The first step of the solution is to empower the editors to configure the A/B test themselves conveniently using the content management system they are used to. This feature must integrate seamlessly into their existing tools rather than adding an additional tool to their set. In this case, we enabled them to define the test and the traffic routes to each page in Adobe Experience Manager.

After finalizing and publishing the test setup, we developed an integration with AWS that automatically stores the A/B test configuration in a DynamoDB table as a reliable and scalable storage solution. This integration isn't exclusive to AEM; we've designed it to be adaptable, meaning it can easily be layered onto other customer CMS platforms.

Assigning and Loading A/B Tests

In the browser, each user gets assigned a unique and consistent user ID stored in the cookies. In our case, this is managed by our tracking solution, Adobe Analytics, but this could also be any arbitrary cookie solution.

When the user requests a page, the request goes through AWS CloudFront, which triggers Lambda@Edge functions.

We then leverage the power of Lambda@Edge to determine to which test group a user belongs, load the A/B testing configuration for that page, and serve him the correct page. This approach is based on the excellent blog post “A/B testing on AWS CloudFront with Lambda@Edge”, where you can find a more in-depth technical explanation of how this works. In a nutshell:

• In the viewer request function, we use a hashing function to determine if the user will get assigned page A or page B and store this information in the request header. If the user declines cookies, we assign him to group A.
• This new request header is part of the cache key in CloudFront.
• If it’s a cache hit, the cached page is returned immediately.
• If it’s a cache miss, the origin request function is triggered. It loads the A/B configuration from DynamoDB and determines which page to serve to the user accordingly.

By implementing A/B testing server-side using CloudFront, we get the following benefits:

• This solution scales very well and is suitable for high-traffic websites like Siemens global Website.
• We fully use CloudFront’s caching mechanism even with A/B testing, resulting in an optimized caching solution that significantly emphasizes user performance. This is a significant benefit compared to many client-side integrations, which may impact performance due to loading multiple page versions.
• Authors can seamlessly manage A/B testing configurations from their CMS.

Capturing Real-Time User Behavior

Configuring an A/B test is insufficient, as we also need to know how the different pages perform. On top of this, our client wanted to get real-time insights. Our approach is loosely based on AWS' blog post “Create real-time clickstream sessions and run analytics with Amazon Kinesis Data Analytics, AWS Glue, and Amazon Athena” and is the following:

• In the frontend, a data layer contains information about the A/B test, and user interactions (clicking, scrolling, viewing, etc.) about this test are captured and sent to the backend.
• In the backend (we leverage lambda in our case), these interactions are sent to a Kinesis Data Stream. By using Kinesis Data Streams On-Demand, our infrastructure can serve gigabytes of write and read throughput per minute without capacity planning. Any other consumer in our infrastructure can also plug in this stream to create other use cases with the analytics data.
• This data stream gets consumed by a Kinesis Data Firehose. We use an AWS Lambda function in conjunction to transform the data according to the requirements and then store it in a S3 bucket. Kinesis Data Firehose automatically provisions, scales, and partitions streaming data in our storage solution.
• The data is stored in S3 and automatically partitioned by Kinesis Firehose. We create an AWS Glue table in the Glue data catalog to store the schema and metadata.
• The data can then be queried via SQL using Athena.

This approach can scale very well for high-traffic websites due to the serverless nature of each individual component. Thanks to Amazon Athena serverless query engine, we can quickly get insights from big amounts of data using SQL.

Bringing Data to Life with Amazon QuickSight

Collecting data is a pivotal process, but the real essence is in its access, utilization, and interpretation to derive actionable insights. Selecting the appropriate Business Intelligence (BI) tool was a fundamental segment of our strategy. Our choice was Amazon QuickSight, and here's why:

• Its pricing model is both scalable and usage-based, allowing us to test on a limited user base before full-scale deployment efficiently.
• The infrastructure is entirely serverless and auto-scales.
• It boasts seamless integration with other AWS services.
• QuickSight is enhanced with Machine Learning features such as auto-narratives, anomaly detection, forecasting, and the innovative QuickSight Q.
• It offers the ability to manage users effortlessly and content programmatically and embed its dashboards into our applications.
• Linking Athena with QuickSight proved straightforward, enabling us to craft dashboards that rapidly portray real-time performance metrics.

Connecting Athena to QuickSight was super straightforward and in a very short amount of time we were able to create QuickSight dashboards that give a real-time view of performance metrics for two versions of the same page, showing insightful metrics like bounce rate, conversion rate, clickthrough rate, or average session length.

• The dashboards are crafted from real-time data, ensuring current and accurate insights.
• QuickSight's compatibility with numerous data sources lets us effortlessly merge analytics data with other datasets.
• Dashboards are fully parameterized, allowing users to filter data based on specific criteria like user region, device type, or a given time span.
• With QuickSight Q, users can intuitively pose questions, such as “What's the trend in bounce rate for mobile users in the last 24 hours?” and receive instant answers.
• QuickSight's comprehensive capabilities enabled us to dedicate less time to solution management and more to generating valuable insights for our clientele.

Innovation Meets Determination: A Collaborative Journey

In a world driven by digital interactions, having the tools to swiftly decipher user behavior can shape the future of businesses. What began as a collaborative vision between Groundfog and Siemens has now materialized into a state-of-the-art cloud-native A/B testing mechanism, ensuring that the digital experiences we provide are not only aligned with user expectations but also continually refined based on real-time feedback.
By leveraging AWS's robust suite of tools, our journey has proven the scalability and efficiency of serverless architectures. From configuring the tests seamlessly in the CMS, capturing user interactions in real-time, and ultimately visualizing the gathered data through QuickSight, every step was about ensuring simplicity without compromising precision and scalability.

Beyond the technology and tools, this journey underscores the power of collaboration and shared vision. Our alliance with Siemens, strengthened by our expertise and affiliation with AWS, showcases what's possible when innovation meets determination.

As we move forward, we remain committed to exploring and implementing solutions that push the boundaries of what's possible in the digital world, ensuring that our clients are always at the forefront of digital evolution.