Skip to main content


When Warhol Meets the Web: How A/B Testing Revolutionises Product Development and Maximises Impact

When Warhol Meets the Web: How A/B Testing Revolutionises Product Development and Maximises Impact

Testing two products or marketing offers with different sections of your customer base can yield a winning approach.

Andy Warhol’s famous representation of Marilyn Monroe has numerous colour combinations and contrasts. The original, made after her death in 1962, though, was on a gold background, with a boldly coloured pink face, yellow hair, turquoise eye shadow and shaded black patches to illustrate her features.

If Warhol and the current generation of creative professionals and marketing executives had anything in common, it is that a massive variety of visual possibilities could create an impact, but some more than others.

What if, prior to painting, Warhol had wanted to figure out which combination of feature (her hair, her teeth, her lips) and colour (green and red) would maximise the viewers’ experience? To determine this, he would need to create six small sketches (3x2) of the same painting, only differing from one another in one aspect (i.e., either one colour or one feature), and ask viewers how much they liked each version, before moving to producing the final painting. 

For over four centuries, experimentation has proven itself a central tool to the advance of sciences. Through experimentation, researchers compare two or more outcomes and control for as much “noise” as possible, gaining insight into which conditions are most favourable. This methodology can be applied to myriad subjects, from the growth of plants, to product design and consumer psychology.

For businesses, successfully engaging in experimentation by testing the success of one product or service against another, widely known as A/B testing, can provide valuable information about customers and thus genuine competitive advantage.

What is A/B Testing?

A/B testing is the practice of randomly splitting a population of unknowing users into groups, and offering each group a different option (for example, different version of a webpage, an ad, or even a newsletter) in order to measure which option is driving the most desirable behaviour.

Because of the low cost associated with changing online content, digital channels offer a fertile ground for A/B testing with various applications. To illustrate, the 2012 Obama for President campaign raised an additional US$500 million using A/B testing to optimise the website. The campaign’s organisers tested many aspects of the online campaign from the visual design of the campaign’s email communications to smaller changes like colors, fonts, layouts, subject lines, and time of day. This revealed that some website variations that were staff favourites actually performed poorly with the public in comparison to other variations that were tested and eventually rolled out.

For executives and managers, A/B testing represents a triple benefit:

- Speed: Testing interfaces and functionality before a full rollout can save time by helping businesses reach their most effective approach before deploying it on a larger scale.

- Accuracy: Testing an experience and gaining feedback from real users can help a brand gauge customer interest and target potential customers more easily.

- Effectiveness: Overall effectiveness is improved when resources are directed quickly and accurately.

The path to successful A/B testing

Key to success in A/B testing, having a rigorously controlled test is crucial to generating reliably accurate results. For instance, offering customers an option one week and another option the next one may lead to biased results if customer behaviour is likely to change weekly.

Instead, variations should be offered over simultaneous timeframes and at random to control for changes in audience composition over time. It is also important to run the test for long enough to establish a clear winner.

A successful approach to A/B testing is also dependent on the type of company and product offered. For mega-traffic companies like Google or Amazon, these kinds of tests are worth the cost of testing because a sub-1 percent lift still contributes substantially to their bottom line. But for small and medium businesses, ‘shallow’ A/B tests of a button colour or call to action will often yield inconclusive results.  For these companies, testing deeper changes to the product, user interface layouts or entire experience workflows are what moves the needle.

Of importance, small and large companies alike should remember to think about the end-user during the test design process. Referred to as “empathic A/B testing”, this will help you ask the questions; what changes can I make to motivate action? What are they looking for? What do they care about?

The winner is?

Finally, companies should make sure to measure performance related to the final goal. To illustrate, Silicon Valley-based startup recently shared an A/B test in which they ran two variations of messaging for a signup button on a website: “Get Started” and “Create Free Account”. Although the latter version resulted in a 21 percent higher click rate, users who clicked “Get Free Account” were both less likely to complete the sign up form and less likely to sign up for paid services than those who clicked “Get Started”.

The former variation’s higher conversion rate was moot, because the greater goal of registering paid users superseded it.

A/B testing can be used at different points of the consumer journey: for instance, A/B tests of bottom-funnel campaigns (for example, testing of ad copy, landing pages, and keyword combinations) can help attract the most qualified visitors. A/B testing can increase the effectiveness of mid-funnel campaigns by showing which argument of sales or type of content is most popular. Finally, A/B tests which give a variety of campaign elements across search engines and social networks near the top of the sales funnel can show both which media and distribution channels lead to the greatest level of sales.

Many companies continually produce and test iterations of their mobile apps and websites in order to increase their adoption by users. Facebook shared some information on the A/B testing process they developed for their mobile app that provides some good insight into how some large enterprises approach A/B testing.

Recently, a growing ecosystem emerged to help companies design and systematically implement A/B testing on a large scale. Companies like Splitforce, Leanplum and Apptimize offer the necessary tools for A/B test mobile apps built for iPhone, Android or even mobile games. These tools make it possible to A/B test game mechanics, level difficulties, and other elements of an online strategy to drive more in-app revenue and engagement.

Towards a culture of A/B

A/B testing is also not limited to the online world. A culture of testing and optimisation is spilling into physical product development. Y-Combinator-backed startup Crowdery aims to allow companies to A/B test fashion and retail offerings among select consumers before they go into production. In this model, customers get a chance to acquire hot new goods at a discount, and companies get valuable insights into consumer preferences. Taking things even further into the real world is Predictive Technologies. The ‘Big Data’ software firm works with many of the world’s largest companies to run small-scale programmes in their physical and digital storefronts. Testing before large-scale rollouts can help companies save time, improve customer service, and mitigate the risk associated with new product launches.

A/B and multivariate testing is a fast-moving field with new providers, technologies, and techniques appearing regularly. As these tools continue to become smarter, they will be leveraged by forward-thinking enterprises. In the end, those companies that have successfully made A/B testing part of their DNA will have significant competitive edge in tomorrow’s business environment.

David Dubois is an assistant professor of marketing at INSEAD. You can follow him on Twitter @d1Dubois

Zac Aghion is the CEO & Co-Founder of Splitforce, a web application for A/B testing mobile apps. You can follow him on Twitter @zacaghion


View Comments

Anonymous User

22/07/2014, 01.01 am

interesting post, guys. thanks for sharing.

Leave a Comment
Please log in or sign up to comment.