This is a basic example of how to set up an A/B test for an ASP.NET MVC4 application
If you dont know what A/B testing is check out the Wikipedia Article on A/B testing
Having read The Lean Start Up I decided it was about time to get my head around A/B testing and to try and bake it into some of the projects I work on.
I evaluated a number of excellent third party products however they all seemed to target the marketeer persona, who wants to test the change of some copy/banners/images on their site.
I was looking for something that we could build into our application so that content/functions etc could be surfaced to the user in an A/B test, these are the results of my findings.
You can break down an A/B testing into 4 steps
- Segment - Divide your uses into separate groups for analysis
- Augment - Change the compositon of your application based on the users segment profile
- Report - Track your users behaviour
- Analyse - Determine if you test was successful
The first part of a test requires that we segment our user base into a control group and a test group. The proportion of segmentation is only important in that you need enough traffic into the test group to make a confident judgement The example in this project uses a simple 50/50 segmentation, however this algorithim could be changed to create a smaller test group. The way that we are going to segment the traffic initially is with a session cookie. This cookie is then used in both the Augmentation and Reporting steps.
This example uses a simple 50/50 "flip a coin" to decide to give a user a "test" cookie during the BeginRequest event of the Global.asax file
Everything I build uses IoC container to manage the composition of an components. Therefore it is a natural extension for me to lean on my IoC container to Augment the application based on a users test group
My IoC of choice is the excellent Castle Windsor, which had a natural extension point called the IHandlerSelector that you can use to select a particular component based on a give critieria.
Both Ayende Rahien and Mike Hadlow have excellent articles talking about how you can use the IHandlerSelector interface so I wont go into detail about how to use it here.
In the example we are going to use it to select a given IAdvert based on the cookie that was set up during the Segmentation phase. However this methodology could be extended to select controllers, databases etc, just about anything that you can think of and has the added benefit of nothaving to pollute your code with "if this then that" statements
In an attempt to try and keep things simple the easiest way to report our A/B tests is to use custom variables in Google Analytics. By using the values set in our cookie we can plug then into a custom variable which we can use as an advanced segment later on. I would also suggest that its worth setting up a Goal in Google Analytics to report on the experiment as it then becomes very easy to track and report on the success.
Google Analytics gives aggregate values and conversion rates which is great, but unfortunately it does not generate the statistical information required to be confident that the result we are seeing is correct. To determine the confidence of a test there is a calculator that you can use on the page http://localhost/confidence of the example application. All you need to do to get the confidence of the test is to plug in the values from your Google Analytics account.