Isolating a single variable - DataMilk

In order to measure the increase in revenue / profit / conversion rate or any other metric due to DataMilk we must create an AB test to compare the behavior of users on your site/app with DataMilk to the original site without DataMilk.

In order to do so we're going to assign each session at their beginning (same as visits) to a control group that will see the original site (let's call this “original”) and a treatment group which may see the optimized (“smart”) site.

Once the sessions are split we can observe the users behavior and compare how users interact with the original site Vs the smart site with DataMilk.

How is the decision made to assign the session into the “original” or “smart” group?

The DataMilk JS code picks the “original” or “smart” group at the very beginning of a new session as soon as the first page loads before the user interacts with the site and reports the choice made to the DataMilk servers as well as to the Google Analytics installed on the domain.

The decision for a client to be part of the “original” group or the “smart” group is stored for seven days unless the user deletes the data from local storage or is using incognito mode which prevents the data from being kept. If a user has been assigned to a group in the last seven days and returns to create another session they will remain in their group.

Any session in the “original” group will see the original site without any modifications. Any session in the ”smart” group may see Smart Components but is not guaranteed to.

Comparing the ”original” and “smart” sites by performance

Now that we have some sessions seeing the “original” site without DataMilk changing the experience and some sessions seeing the “smart” site with DataMilk we can start comparing the behavior of the users in those sessions. We measure the following metrics and more and display them in the customers’ dashboard.

Conversion rate uplift

This metric describes the difference in conversion rate between the “original” and “smart” sessions.

Example:

Conversion rate uplift = (1.2% - 1%) / 1% = 20%

Revenue per session uplift

This metric describes the difference in revenue between the “original” and “smart” sessions.

Example:

Revenue per session uplift = ($130 -$100) /$100 =30%

Profit per session uplift

This metric describes the difference in profit per session in the “original” and “smart” sessions.

Example:

Profit per session uplift = ($140 -$100) /$100 =40%

You can find a full list of metrics we collect here.