How to Read Your A/B Test Results

The Algolia A/B testing feature helps you set up an A/B test to assess whether your search strategy is successful. Reading the results and trends from your test might seem straightforward, but there are several things to bear in mind when interpreting results.

Understanding how A/B testing works

How results are computed

A/B test results are computed from click-through and conversion rates based on the events you send to the Insights API. This computation can lead to different results from what you have on your business data.

Looking at your business data

As Algolia A/B testing is specific to search, the focus is on search data. It’s best to look at the A/B test results in light of your business data to assess your test’s impact. For example, you can cross-reference search and revenue data to compute your interpretation of conversion or to look at custom metrics.

When to interpret what

The A/B testing interface in the Algolia dashboard displays data starting from the second day of your test. Even if some numbers are showing, it doesn’t mean it’s time for you to draw conclusions. Here’s what you should look at before interpreting the A/B test results.

Is the data significant?

Before you interpret results and draw conclusions on what steps to take next, ensure that the numbers are significant. The significance score helps you assess whether the results are the product of chance or an accurate representation of how your users behave.

When the significance is higher than 95%, you can be confident that the results will happen in the future. You shouldn’t draw any conclusion if the significance is below 95%.

Is there enough data?

The larger your sample size, the better. You should run A/B tests for at least two business cycles and wait for the end of the test to draw conclusions. Experience proves the best results happen when the traffic represented by both variants is over 100,000 searches. If your overall A/B test traffic is below this, you might want to take its results with a grain of salt.

Is the split off?

When setting your A/B test, you assign a percentage of search traffic to each variant (by default, 50/50). The expected traffic split should be reflected in the displayed user count. For most A/B test configurations, the search count for each variant should match the traffic split. If there’s a noticeable discrepancy, there’s probably an issue.

For example, you could have a 50/50 split, ending up with 800,000 searches on one side and 120,000 on the other. The results are unreliable if you see a difference higher than 20% from the expected number. In such scenarios, you should investigate your A/B test setup to understand what’s happening. It could be due to outliers like internal IP addresses or external bots skewing the results, or a back-end implementation not sending all the data.

Troubleshooting

If you believe something’s wrong with your A/B test results, there are some checks to identify what could be the root cause.

Analytics and events implementation

Since A/B tests rely on Click and Conversion events, ensure you’ve properly implemented them.

For example, you might want to check the following:

• Are you catching both click-through and conversion rates?
• Are there enough events on popular searches?
• Are there any errors in the Insights API Logs within the Monitoring section of the Algolia dashboard?

Seasonality

Sales or holidays (such as Black Friday) can affect your test, with more out-of-stock items, for example. If you see unexpected results, check whether you’ve been conducting the A/B test during a special period.

A/B test in a Dynamic Re-ranking context

If you have launched an A/B test to evaluate Dynamic Re-ranking’s effect, there are a few additional considerations:

• Make sure to launch the A/B test through the Dynamic Re-ranking interface. If it’s not the case, Dynamic Re-ranking must be opted-in for the index being tested.
• If using a replica, ensure re-ranking does have an impact: you should have queries re-ranked for this index. If not, change its events source index to the primary index for Dynamic Re-ranking.
• Check whether Personalization is also enabled for the index you’re testing. The analysis isn’t optimal when Personalization is enabled: it counts traffic towards Dynamic Re-ranking when it probably doesn’t matter for a given percentage of the traffic. Indeed, as soon as a Personalization user profile is detected, Dynamic Re-Ranking no longer has any effect.
• Dynamic Re-ranking isn’t optimized for particular use cases like marketplaces with short-lived items, for instance. If this is your use case, you may see A/B test results that aren’t satisfactory.

Are you using distinct? If so, items are regrouped so Dynamic Re-ranking has a lesser impact.

The data is no longer significant

The significance computation involves ratios, and there’s no guarantee that it will increase. After a certain point gathering more data may affect the outcome and make it less trustworthy. Typically, seasonality effects could change the conversion rate at the last minute. Your data could be significant at some point, and no longer significant some days later.

Click-through rate is going up and conversion rate is going down

This probably means that top search results don’t convert. Potential causes include out-of-stock products, items with the wrong picture, misleading descriptions, or unavailable sizes. In such cases, you can use filtering to eliminate unavailable products, fix the relevance implementation to prioritize certain items or clean up your data.

If possible, you should also look at your own business intelligence metrics to confirm/deny the results you’re seeing. If you’re looking at revenue, for instance, what’s the impact of the A/B test?

Click-through and conversion rates are both going down

You might reach the point where you have been through all the possible checks to see if there’s something wrong with your A/B test and Insights API implementation, and nothing seems to explain the results. Perhaps the settings you’re testing aren’t a good strategy for your search: keep iterating to find a successful strategy.

Contact Algolia’s support team for further help.