Use Cases

Variant Experiment Server Sample Use Cases

1Online Retail / Free Shipping Offer

Offer free shipment on orders over a threshold amount to non-promo customers. Run multiple variants of the offer to determine the most cost-effective threshold amount.

This experiment involves two critical steps: qualification and targeting. If a user session originated as a click or touch on an on-line ad or an email promo, we want to exclude it from the experiment, which is to say make it ineligible for the subsequent step: targeting. Those sessions that came to the online store directly are qualified for the experiment and will be targeted for one of the five experiences.

The host application is responsible for figuring out if the user session is a promo, and must set a session attribute which will make it available to the server side code:

The server side implements a custom qualification hook, which uses the value of the session attribute to qualify the session for the experiment.

The sessions disqualified by this hook will be shown the existing experience and will not be triggering Variant events, which is to say that these sessions will not be considered control for the purposes of this experiment.

The sessions qualified by this event will be targeted randomly to either control or one of four variant experiences according to their weights, as declared in the experiment schema:

According to this experiment schema, all user traffic qualified for the experiment will be targeted 96% to control and 1% to each of the four variants.

2Online Lending / Synchronize Online and Support Call Center Experiences

When a customer calls the support center, ensure that the automated telephone menus he hears reflect his online experience.

Frequently, when you roll out a new feature on your fintech online application you also must make corresponding changes to your support center’s inbound telephone menus. If you want to instrument the new feature as an online experiment, you must include the call center software in the experiment, so that if a customer calls the support center, he hears the phone menus appropriate for his online experience.

The core issue here is experiment scoped targeting stability, or preservation of targeting information between user sessions. (The narrower session scoped targeting stability guarantees such preservation for the duration of user session and is provided by Variant server automatically). In this use case even if the customer calls support while his online session is still active, it will be inaccessible to the call center software, so a new session will be created, making it the case of experiment-scoped targeting stability.

Variant cannot provide experience-scoped targeting stability automatically, but it gives you the mechanism to implement it in an elegant and reusable fashion: the targeting user hook, when posted, will have the opportunity to override the default randomized targeting and direct the user’s telephone session into the right experience:

Here we expect the experience to already have been placed in session by the qualifying hook, which fires first. The reason we need the qualifying hook is because we don’t want to include in the experiment users who are simply calling the call center and have not been targeted for the experiment online. These users will be disqualified from the experiment for the duration of their phone call:

Here we first check if the call center software sent the phone number. If there’s no phone number, we assume that the foreground session is something other than the phone call to customer support and do nothing. If we have the phone number, we have to figure out the last recorded experience for this test, if any. This call is likely to involve the operational database, so if such an experience exists we want to save it in session for later use by the targeting hook.

Finally, we need to configure both hooks in the experiment schema: