Variant Experiment Server Overview
Variant Experiment Server is deployed on the same network as the host application(s), either on premises or in the cloud, in proximity to operational data. Variant server can directly access live operational data via server side extensions API, which provides a powerful framework for enrichment of the server’s default behavior with highly specific semantics, such as “disqualify customers who have been offered a similar promo in the last 10 days.”
You can leverage your established technology stack to access live operational data securely and with very low latency. Direct integration with live data ensures that your experiment will not be compromised by stale data extracts.
Variant’s technology offers a uniquely generalized approach to online controlled experiments. The core model operates on abstract states, which, depending on the host application, may be HTML pages, Android activities, telephone menus, etc. You may even instrument completely headless interactions, such as an API call/response.
This degree of abstraction liberates the developer to think of an experiment as an arbitrary application code delta. This delta may be a simple isolated change, such as the color of a button or a tweak to an algorithm behind an API call, or span multiple services of a distributed application. Variant enables and encourages you to experiment with an entire product feature, however complex.
We’ve organized definitions of all your experiments in a single human readable file called experiment schema. It uses a set of powerful abstractions, collectively known as the Experiment Definition Model, to encompass definitions of all experiments running on an application domain. Experiment schemata are managed by Variant server, leaving client code completely free from “experimentation smell.”
An experiment schema is a regular development artifact, like any other source files, and subject to regular revision control procedures. You can run your automated tests with the experiments on, against a single instance of Variant server, thanks to its support multiple experiment schemata.
The sophistication with which Variant lets you manage concurrent experiments is unequaled. Unlike other A/B tools, which force you to flatten all concurrent tests into a single multivariate test (which crams unrelated concerns into one experiment), Variant enables the test designer to operate with the entire variant space and keeps concurrent experiments independent. By default, Variant avoids targeting user sessions for a combination of test experiences that has no backing implementation. When this approach causes downstream experiments traffic starvation, or to simply speed things up, you may implement hybrid variants and let Variant server know about them.
In a regular use case, you will deploy a new code path to coexist with the current one in the form of a randomized controlled experiment in order to compare their performance. Typically, you will send only a small fraction of the user traffic into the new code path for two reasons: 1) to avoid publication of the new experience until it is validated; and 2) to minimize the impact of possible defects in the new code.
If you are not really interested in 1), but only in 2), Variant is still the best tool to help with the gradual roll-out of a new feature. You can use Variant’ sophisticated targeting and qualification logic to manage which user population will be sent into the new code path, throw the switch off on an experiment, should a defect be discovered, without interrupting the host application, or use Variant’s custom events to trace the new code path.
Whenever a user’s experience is a matter of a dice roll, the concern is what will happen when the same user returns to the application the next day. It is frequently critical and even a legal requirement that a recognized returning user always see the same experience as he saw last — the feature we call targeting stability.
Other A/B tools cannot help you provide stable targeting because it requires real-time access to live operational data. Variant’s direct integration with operational data makes targeting stability a standard feature. As a result, your user may start an online loan application on a work desktop, continue on a phone on her ride home and finish on a home laptop: Variant will make sure that the experiment she is going through will not play a trick on her.
Variant Experiment Server is made to be extended via the server-side Extension API. The central idea in ExtAPI is user hooks, which are callback functions, posted by various server life cycle events. For example, the session qualification life cycle event is triggered when a user session first comes in contact with an experiment, and Variant server must decide whether the user is qualified for it. You can create a user hook for this event, which will augment the default functionality with specific qualification semantics based on live operational data, such as “only include organic search traffic.”
User hooks are defined in the experiment schema and can have scope of the entire schema, a particular test or a particular state. Hooks can also be chained, to help you modularize and reuse your code.
Variant Java Client
Variant Java client is a Java library that allows a Java host application to communicate with Variant server. It exposes all of its functionality via rich native Java objects, hiding much of the complexity from the application developer. The Java client has minimal third party dependencies and can be consumed by any host application running on a Java Virtual Machine.
In addition to the bare Java client, Variant also offers a number of open-sourced client adapters, which wrap the bare client in a higher level APIs, suitable for particular Java-based frameworks, such as the servlet adapter.