AB Testing

What is AB Testing?

AB Testing is a way to test different versions of a page to see which one performs better.

In a case of our search engine, it allows to try out different search configurations at the same time and see which one performs better.

How AB Testing works?

In LupaSearch Console, or using an API, you can create AB Tests.

Each AB Test has:

• Name - a name of the test;

• Triggering query - a query that triggers the test. (If user uses this query key in a public query request, ab test will be triggered);

• Queries - a list of queries that will be used in the test. (Each query has a trigger chance percentage, which defines how often it will be used);

• Active From / Active To - a time period when the test is active.

• Enabled - a flag that defines if the test is enabled. (If the test is not enabled, it will not be triggered);

If AB test is active, LupaSearch tracks various stats related to AB ab test query engagement (clicks, CTR, no results rate), which allows you to determine which query performs better.

What are examples of AB Testing?

You can use AB Testing to test different search configurations, like:

• Determine if additional query fields decrease zero results rate;

• Determine if enabled Personalization or AI Synonymyms help with increasing CTR;

• Check which default sorting or boosting rules result in better click-through rate;

• Determine if different facets or facet ordering rules result in higher estimated Add To Cart value.

AB Testing recommendations

When creating AB Tests, consider the following recommendations:

• Target queries should only be used in AB tests - do not use queries that are used in AB tests in other parts of the search configuration. This will help to avoid confusion and make it easier to analyze the results. Only refer to it with the trigger query (searchQueryId).

• Include sessionId and userId in both /query and /event requests - this will help to track user behavior and analyze the results. If userId and sessionId is not included in either of the requests (at least one of these should be included), AB test might not be triggered.

Configuring AB Testing using Dashboard

To configure AB Testing using Dashboard, go to AB Testing section in LupaSearch Console:

Click Add AB Test button to create a new AB Test:

Fill in the form, add trigger query and queries and click Save Changes button. You can optionally change query trigger percentages manually (by default, they all will be equal):

Let test to run for a few days and then check the results using the "Eye" icon:

You will see a page with AB Test results summary.

The page includes the following information:

Main ab test details - name, active period, enabled flag, and trigger query.

Sumamry - summary of the test results, including: total number of each query executions, CTR, no results rate and estimated add to cart value.

Query Charts - charts with daily stats for each query, including: number of executions, CTR, no results rate and click events.

High and Low CTR Terms - list of terms that have the highest and lowest (among TOP 100 most popular terms) CTR for each query.

Configuring AB Testing using API

To configure AB Testing using API, you can use the following endpoints:

List all AB Tests

API Reference: Get AB Tests.

HTTP GET /v1/indices/{indexId}/abTests

Response example:

[
  {
    "id": "string",
    "name": "Test A/B 1",
    "searchQueryId": "string",
    "activeFrom": "2023-07-01",
    "activeTo": "2023-08-01",
    "enabled": true,
    "createdAt": "2023-05-01",
    "updatedAt": "2023-06-01"
  }
]

Create new AB Test

API Reference: Create AB Test.

HTTP POST /v1/indices/{indexId}/abTests

Request body:

{
  "name": "Test A/B 1",
  "searchQueryId": "string",
  "activeFrom": "2023-07-01",
  "activeTo": "2023-08-01",
  "queries": [
    {
      "searchQueryId": "string",
      "triggerChancePercent": 50
    },
    {
      "searchQueryId": "string",
      "triggerChancePercent": 50
    }
  ]
}

Edit existing AB Test

API Reference: Update AB Test.

HTTP PUT /v1/indices/{indexId}/abTests/{abTestId}

Request body:

{
  "name": "Test A/B 1",
  "searchQueryId": "string",
  "activeFrom": "2023-07-01",
  "activeTo": "2023-08-01",
  "queries": [
    {
      "searchQueryId": "string",
      "triggerChancePercent": 50
    },
    {
      "searchQueryId": "string",
      "triggerChancePercent": 50
    }
  ]
}

Delete existing AB Test

API Reference: Delete AB Test.

HTTP DELETE /v1/indices/{indexId}/abTests/{abTestId}

View AB Test details

API Reference: Get AB Test Details.

HTTP GET /v1/indices/{indexId}/abTests/{abTestId}

Resposne example:

{
  "id": "string",
  "name": "Test A/B 1",
  "searchQueryId": "string",
  "activeFrom": "2023-07-01",
  "activeTo": "2023-08-01",
  "enabled": true,
  "queries": [
    {
      "searchQueryId": "id1",
      "triggerChancePercent": 33
    },
    {
      "searchQueryId": "id2",
      "triggerChancePercent": 33
    },
    {
      "searchQueryId": "id3",
      "triggerChancePercent": 33
    }
  ],
  "createdAt": "2023-05-01",
  "updatedAt": "2023-06-01"
}

View AB Test results

To view AB Test results, use Analytics API endpoints with respective query key filters (see Analytics API).