How to review new test variants and fine-tune their configuration

Owned by Sylvain Paillard
Last updated: Nov 30, 2020
11 min read

Overview

In this document, we describe how to review the functioning of a test variant in order to confirm that it can go live. The document covers how to use the testing view to simulate end-user requests, analyse their response and understand why the results are selected in that order, to adjust the weighting of the underlying strategies causing the scoring and to test again, until the results are satisfactory and can go live.

This process can be done to verify the change resulting of new test variants in an existing widget, but they can also be used to review and confirm the base functioning (without test variants) of a new widget.

How to simulate an End-User request

First access the Boxalino Intelligence Admin on https://intelligence-stage.bx-cloud.com and login with your credentials.

Then go to the Deployment > Testing view.

You will see the Test request panel which enables you to recreate any end-user request.

First, select the widget which provides the results on your web-site (e.g.: for a search result page, select the widget 'search').

Second, depending on the case, you will need to indicate an extra information (e.g.: for 'search': the user search query, for navigation: the category id of the page, for similar recommendations: the product id of Product Detail Page, for basket: the list of product ids of the basket, etc.)

Finally, select the mode 'live mode with live data' because you want to test what is already live on your web-site (and not the currently unpublished changes).

If you have prepared several test variants (which you should do, a/b testing is key :)), you can select the test variant in the left main Widget tab in the test variant drop down (which appears automatically after you select your widget).

How to simulate a Search Query

Select the widget “search” and enter your search in the appearing text box next to it (that’s it folks )

If you use a widget for search which is not called “search”, then select it and go to the Parameters view and indicate your search query in the Search Query input field.

How to simulate a Navigation (Category) Query

Select the widget “navigation” and select the property which defines the navigation page (typically it is “Category”, but it could be something else (e.g. : “brand” if it is a brand page).

Then you can enter a category id or a part of the category path and simply select it in the autocomplete list.

How to simulate a Home Page Recommendation Query

Nothing special to do here, just select “home” as Widget. But you might be interested to check : How to simulate any Query with Personalization (below) for this case.

How to simulate a Product Detail Page Recommendation Query

Select the widget you want (similar, accessories, complementary or related).

Then you can either enter a product id (this is the specific product id of a specific buyable product (not the product group id) or, if you don’t know it, click on the search button on the right.

Then make a search for the product you want, select it in the list and then click on “ADD”.

How to simulate a Basket Recommendation Query

The logic is very similar than the Product Detail Page Recommendation Query before, except that you should select the widget “basket” and provide the list of the products in the basket of the user as Context Items.

How to simulate a Recommendation Query with Selected Facets

To simulate a specific Facet Selection by the user, go to the Parameters tab and set your facets (as parameter and values in a url) in the Facets text field next to it (e.g.: facet_1=value_1&facet_2=_value2)

In some cases, Facets are set as soft facets (for example, in the product finder), these facets should be provided as Context Parameters and not as facets, for example:

bx_bxi_relation=freundin&bxrpw_products_enrichment_Type_haseverything=100

How to simulate any Query with Personalization

Some visitors see something different than others? Well, that's the goal of personalization. How can I diagnose a specific case?

When you widget strategy includes real-time personalization, it means it can return different products for different visitors.

When your strategy includes personalization based on customer data (purchase history, age, etc.), it means it can return different products for different identified clients.

Boxalino automatically creates a browser cookie named 'cemv' which will store the Visitor id. So if you want to reproduce a specific personalization situation for this visitor, you need to copy this cookie value (press F12 and go to Application -> Cookies in your browser while you are on your live web-site) and paste it in the Left Section in the tab 'Visitor / Customer ID'. Then the results will be provided for this particular visitor.

Boxalino automatically connects the visitors to their account when they log in or when they come from a identified source such as the newsletter.

If your visitor has already logged in, the personalization based on the customer data (transaction history, customer properties, etc.) will automatically work with the Visitor id alone.

If this visitor didn't log in, or if you want to test the personaliztion for a customer without having a Visitor id, then type the customer id in the 'Visitor / Customer ID' tab.

Now we see results

When you are ready, click on the 'Get results' button an the list of results will appear on the right. To refresh the result with other parameters, just simply click on the same button again . If you want to compare different results, you can click on the column icon on the left side of the 'Get results' button.

This button will trigger the compare mode. A dialog box will appear which will give you a simple and short explanation of how this mode works. If you want to save the dialog box for another time, in case you forget how the compare mode works, then click on the ‘KEEP IT FOR LATER’ button. But If you are already familiar with the compare mode, you can simply click on the ‘DON'T SHOW THIS AGAIN’ and the dialog box will never show up again.

 

Here is the dialog box with the explanation:

 

 

 

On the right, you will see the pictures and names of your products as they appear on the web-site as well as their scores in relative %. You can see more information about the product attributes in the Details tab (to add additional attributes, select the "fields" you want in the left and refresh).

In case the results don’t match or you don’t have pictures, please first check that you have set all the parameters of your request correctly (as described in this document) and, if it still doesn’t match, safe your request (with the save button on the right, in the Request drop-down menu on the left side of the screen.) and indicate to Boxalino the name of your request (which you can set by your own, by replacing the red ‘Enter your Title’ text on the top of the Test Results on the right. Boxalino will be able to reproduce your query and let you know why it doesn’t work.

On the top-right of the results, you will see 4 icons:

[extra info] [debug info] [upload] [delete]

 

Extra Info: This button opens a dialog which gives provides additional information about your request.

Debug Info: Loads the data used to get the results in this column back into the request form.

Upload: When you change the results (by drag&drop of an item, or adding a rule on a result) refreshing is not enough, you will need to upload your configuration (which can take a few minutes), which is the same as doing save & test on a widget strategy.

Delete: Remove this column of results (this one is easy to understand )

Understanding the score

One of the first question seeing the results is: why does this product appear in this position?

The selection of the product is done through 2 mechanisms:

  1. Filtering - removing anything that should not be there

  2. Scoring - ordering what should be there in the right way

The score is indicated directly in the results with a % sign. The first result is always normalize at 100% and the other scores are represented as a relative percentage compared to the first result. This way, you can easily see visually how the scores drop throughout the results.

Why does this product has this score?

To understand the score, go to the Explain Tab and check which one of your Use Cases contributes to which part of the total ranking score. Normally, your use-case should always be configured with easy to understand labels, so (as on the left) it should be easy to understand what are the constituents of the score.

The scoring is easy: it sums the value generated by each of the pie chart elements and the total is indicated above the pie-chart. The product with the highest score appears first.

 

A much more complex to understand tree (with all the technical details) appears below the pie-chart for the ones who want to understand the formula exactly.

I need to see more product attributes to understand why it has this score!

The detailed tab will show you all the fields which are involved in the scoring and filtering of your products.

If you want to see other product attributes, you can add them on the left in the “Returned Fields” and refresh the view.

Then they will appear as well in the list of attributes of the Details tab of the products.

 

Why is there not another product before? I would expect it to have a higher score!

If you don't understand why a product doesn't appear in the top results, click on the compare button in the explain tab of any of the products and indicate the product in the dialog box.

This will open a new column with only this product.

You can check the scoring factors of this product compared to the top appearing products (in the explain tabs) to understand what Use-Case causes the difference in scores.

How to change the scoring if I know what is wrong?

When you have been able to identify (thanks to the prior steps) what Use-Case should be adjusted, then you can adjust its weight. Open a new tab (believe me, it will be good to have two different types open and to keep one open with the test view as you have it now ).

Open there the Main View (Widget & Page Optimizer on the first menu) and click on the widget you want to adjust).

We will not describe here all the details of how to set-up your Use-Cases in general and what are the parameters of each Use-Case (this is another Use-Case) but to make it short, you have on this view a list of Use-Cases (on the right) with their labels and you need to click on the one you want to adjust.

Look for the “boost weight” of your use-case (either in the Values tab like in the screen-shot above, or in the Advanced tab) and change the value to what you think would be a better value.

How do the “boost weight” work?

There are two different cases with distinct behaviors:

  1. Filter Match Boost Weights

  2. Scorer Fields Boost Weights

Filter Match Boost Weights

This case applies to most Use-Case which are defining a match (e.g.: a Use Case boosting products matching a certain condition on a product property and value, like boost products with color=red). The boost weight is then directly what affects the scoring. If your Use Case can create several matches, it might be a good idea to defined a Max Weight (see info box above).

Scorer Fields Boost Weights

This case applies to Use-Cases connected to one (or several) numeric fields which value should be used for the scoring (for example: the number of sales a product has). In this case, the boost weight is the multiplication factor to be used (so in this example of the number of sales, the factor will be the number of sales the product has multiplied by the boost weight).

Save & Test

If you want to test changes you did in widget strategies, first make sure you clicked on 'Save & Test' in any widget configuration (it will anyway update the test mode of all your widgets). Then, please check the help 1. Why does it appear live, as all these instructions will be helpful for this case as well.

After you get the confirmation that the “Save & Test” is finished, you can refresh your results and see the effect of your change. Make sure you have the right parameters selected.