Most ecommerce companies who do A/B testing don’t actually have a CRO strategy. They just run one off tests, one after another, based on the problem of the day:
- Our PDP photos aren’t as good as competitors, let’s test it!
- The CEO likes our competitor’s checkout process, let’s test that!
- The personalization platform’s sales rep says we need to move their container up the page, let’s try that!
If a test wins, you slap high fives and move on to the next one. If it loses, you shrug and move on to the next one.
This is not a strategy. This is random one-off testing.
In fact, we have a name for it. We call it “tunnel vision testing”.
The problem with tunnel vision testing is that there is little to no accumulated learning. This month you think of 4 things to test. Next month you think of another 4 things to test. A year from now, you think 4 more things to test.
But none of these tests are connected.
If someone asks you “What works in CRO for your site?” People who do tunnel vision testing respond by listing off a few tests, like:
“Well this filters tests worked. And then this test on checkout worked.”
They can’t connect the dots between tests and paint a larger picture of what’s going on in their site.
A strategy, on the other hand, means you view the conversion rate optimization (CRO) of your site holistically.
To do this, you need a framework that ties every test together into a larger story about what does and (equally important) doesn’t compel your customers to act, to buy, to convert.
After running hundreds of A/B tests on ecommerce sites over the last 4 years, we’ve developed such a framework. We call it the Purpose Framework for ecommerce CRO because it categorizes every A/B test on an ecommerce site as having one of 8 Purposes.
These purposes connect every test together.
Like brush strokes on a painting, they work together to form a picture of what moves the needle and what doesn’t; what your customers care about and what they don’t; what affects conversion rate and what doesn’t.
So when everyone else is only focused on the day-to-day, test-by-test minutiae (the trees) at the ground level, you have a system to see the 10,000 ft view at the strategy level (the forest).
Doing ecommerce CRO like this leads to a much richer understanding of what moves the needle, so we can focus on what matters and stop wasting time on what doesn’t.
This article will walk through our Purpose Framework in detail and show it applied on real client case studies. But first let’s dig further into why Tunnel-Vision Testing is so common and the negative consequences it leads to.
Note: You can see all of the A/B tests we’ve ever run on ecommerce sites , organized by Purpose in our Growth Rock live database of A/B tests.
If you’re interested in working with us to help your ecommerce brand increase online conversion rate, you can learn more and reach out here.
Tunnel-Vision Testing: How Most Companies Do CRO
From what we’ve seen, every ecommerce A/B test has a story for why it’s being run.
- Someone on merchandising wanted to test this new product video
- Blogs say ecommerce websites need to focus on cart abandonment, so let’s run tests on the cart page
- Marketing ran a heatmap on the checkout page and now wants to test it
- The CEO says Amazon does it this way, so she wants to test it on our site too
Each of these tests may have a legitimate reason it’s being run, that’s why tunnel-vision testing is so prevalent. That’s not the problem. The problem is that the testing is done in an unconnected, one off way: what you learn from some copywriting test is thought to be unrelated to the product video, the CEO’s Amazon copycat test is unrelated to marketing’s checkout page test, and so on.
Note that this is a problem even if “conversion research” or “analytics” was used to motivate a test, because often those research activities are done in the same one-off, ad hoc way as A/B tests. Someone happened to be looking in Google Analytics, and noticed the bounce rate on a page was high and now wants to add some social proof, testimonials, or move the CTA around.
That is still ad hoc, even though it involved analytics.
As a general rule: If you’re not connecting an A/B test to the big-picture CRO strategy of the site’s entire user experience, you’re still just doing tunnel vision testing.
When you do this, you can spend years testing and have barely improved your understanding of what really moves the needle for your customers and conversion rate. You know a lot about individual elements (“We tested that!”), but little about the big picture.
Litmus Test: Answering “What Conversion Optimization Have You Already Done?”
When we ask ecommerce managers what they’ve already done in conversion optimization, almost all of them just rattle of a few tests that come to mind:
“Well, one test made entering a credit card easier, and that worked.”
“Another added new product images, and that also worked.”
They don’t say things like:
“Our customer has no problem finding products or checking out, it’s all about trust in the brand, that’s the lever where we get the most impact”.
That is strategy-level knowledge!
See the difference?
Think about what’s required to have the strategy-level of understanding of your ecommerce store. You need to know how a series of tests, together, create a larger story about your customers and what matters to them.
How do you get that level of understanding? Where you can say “this matters”, and just as important, “these other things don’t matter”?
Our Purposes Framework for Ecommerce Conversion Rate Optimization
Years ago we identified the problem (tunnel vision testing) but had yet to find a solution. We tried various solutions (like our old framework Usability vs. Desirability) but they weren’t enough.
The breakthrough came when we realized that all ecommerce A/B tests we’ve ever done (Here’s a live database of all of them) have one of 8 purposes:
- Brand – Increase trust, credibility, or appeal of the overall brand.
- Discovery – Make it easier to find or discover the right product
- Product Appeal – Make individual products more appealing with messaging, positioning, or imagery.
- Product Detail – Highlight or specify details of products (like ingredients in a lotion or specs of a car part) that help customers choose
- Price & Value – Make the price to value ratio better
- Usability – Reduce UX friction (e.g. reduce form fields)
- Quantity – Increase average order value or cart size
- Scarcity – Give a sense of urgency by highlighting limited time or quantity to purchase
For example, here are the purposes behind a few A/B test case studies we’ve published:
- Review Star Rating on Product Page – Product Appeal and Brand as the star rating makes both a specific product and the overall brand more appealing.
- Free Shipping Messaging – This is Price & Value.
- Sticky Add to Cart – This is pure Usability as it simply changes some UX on the product page
- Moving the Size Guide Link – This is Product Detail as size is a detail that can affect purchase decisions for apparel.
Clarity around these eight purposes developed slowly in the course of our work. First we started calling some tests “Brand” tests, others “Discovery” and others “Product”. Those were the only three buckets we had.
Then, we tied A/B tests to the page on the site where the test ran (e.g. Listing page tests were “Discovery”) but realized later that page is less relevant. A test on the cart that shows “You may also like” products is about Discovery just the same as tests on the listing page.
Slowly we added more and more purposes until we got to those eight. They seem to sufficiently capture all tests we run for ecommerce clients without getting overly specific.
As this picture got clearer, we noticed the power of these purposes: they could tie tests together that were done over months or years of work to give us a bigger picture understanding of what customers want.
For example, we could get “aha” moments like these:
- “Wow, tests about site search, filters, or finding products in some way are winning often” (Discovery)
- “Interesting, for this client, free shipping, discounts and promo codes work really well, but for this other client, they don’t make any different” (Price & Value)
Eventually, we got our act together and started carefully labeling and counting tests by purpose. As we’ll show below, it helped produce insightful graphs like this:
You can see with the blue bars how many tests we’re running in each category and ask: Why aren’t we doing Brand or Product Detail tests? Does that make sense? Can we defend that decision?
You can see with the red bars how many are winning and ask: Which purposes win more? How can we use that to do more of what works?
Once you start seeing this 10,000 ft view of what affects the conversion funnel on your site, everything changes. Every A/B test you do from that moment on is connected. It feeds into the larger picture of which purposes move the needle and which don’t.
You develop a CRO strategy.
How to Implement The Purposes Framework
Implementing this framework requires labeling each test with its main purpose(s), tracking the wins and losses, and analyzing and graphing the data.
Step 1 – Track Purposes – Track every test you’ve done and label them with one or two purposes each. We do this in a spreadsheet. Here’s a recent month for a client:
Allow for 2 purpose labels because many tests touch on more than one.
For example, the first test in the screenshot above is on some UX changes we made on the filters in the left navigation area of their listing page. We labeled this with Discovery, because filters on listing pages help the customer find and discover products, and Usability is because it was a UX optimization with the left nav, meant to make it easier to use.
Two purposes at most per test has been enough. More than that seems to overcomplicate things (if tests are labeled with too many purposes, you’ll wash out trends in the data).
Step 2 – Lifts – Track if each test showed a statistically significant change in conversion rate.
That’s done in our “Lift” column above. Importantly, we mark a test as having a difference even if the “control” won. For this analysis we want to know which purposes affect conversion rate. So we’re less interested in which variation won but rather that a test, and its purposes, resulted in a measurable difference in conversion rate. If it moves the needle, we count it.
Step 3 – Page – Track on which page of the site the test was run. We use 6 page categories: home, navigation, listing, PDP (product detail page), checkout, and sitewide. Just as we’ll count and monitor which Purposes we’re testing more of, we’ll also want to monitor which pages we’re testing heavily on and which we’re ignoring.
Step 4 – Analyze Once you have an entire spreadsheet full of these for your site, you can then count:
- How many tests you’ve been running within each purpose
- How many of each resulted in statistically significant conversion rate or revenue differences
For example here is a table of these stats for one of the first clients we ever implemented this framework on:
Seeing the data is only the beginning. Graphing it, analyzing it, and building a strategy with it is where it gets interesting.
For that let’s use some case studies.
Case Studies: How The Purpose Framework Revealed Impactful Insights About CRO Strategy
We first implemented the purpose framework for a client in the food space with about 30 – 50 products in the store. At the time, we had been working together for over a year, running multiple A/B tests every month.
We’d had many winners, a great relationship, and felt we were doing CRO strategically, but with the benefit of hindsight, we were doing tunnel vision testing. The purpose framework, as we’ll show here, revealed impactful insights about CRO strategy for this client.
Tier 1 Analysis: Can We Defend Where We’re Spending Testing Time?
The Tier 1 question to ask in this process is: Are we testing in the Purpose buckets that make the most sense? Is it defensible?
To answer this, we graphed how many tests had been run on each purpose.
Seeing our test history grouped by Purpose like this was striking. The armor of tunnel vision testing started to crack. We started to view the site holistically (the original goal!). Strategy–level insights and questions arose.
Why had we run so few brand tests? And so many Usability tests?
This client has an awesome brand. They’re one of the first players in their niche, and they grew their brand with content for years before selling anything. So they have authority. People in this space respect them. They were also one of the first to sell the flagship product that all competitors in this niche are now selling.
So why had we not tested doubling down on these brand strengths?
We had no good answer.
Instead, the graph shows a lot of usability tests. Why? We think doing a lot of usability tests is common, and a byproduct of tunnel vision testing. When you don’t think strategically about what you want to test and why, you default to usability tests because they’re easier to think about. Big picture strategy is hard. Testing UX minutiae is easy: “Should we add our phone number on this page?” “Should move the product video in the PDP to this other spot?” “Should we hide the shipping costs or show them?”
But there was so much more we could have tested on brand. What about homepage tests that talk about how they are the original, dominant brand in this space? What about testing different value propositions and brand messaging? What about mentioning this on the listing page? What about showing the mass number of total positive reviews accumulated to date?
This simple graph got us to ask tough questions like these for each of the eight Purposes.
The 8 Purposes Are Like Bricks on a Wall. You Tap on Them With A/B Tests.
The eight purposes are like bricks on a wall you’re trying to take down. Each A/B test is a tap on a brick.
“Let’s tap on Brand with this homepage messaging test.”
“Let’s tap on Quantity and Discovery with different upsells on the shopping cart.”
You’re trying to see which bricks move. To do this you need to tap multiple times on each brick. If you don’t tap at all on a brick, or you only tap, lazily, once or twice, you rob yourself of the opportunity to see if moves. There could be an entire world of tests behind that purpose that you’re leaving on the table because you didn’t have a system to tell you: Hey, these bricks haven’t been touched.
Tier 2 Analysis: Contrast With Win Rate
Next, we plotted the win rate as well:
This adds the next layer of understanding. For this client, for example:
- Discovery and Quantity tests are winning at a high percentage (more than half the time).
- Usability tests aren’t winning as much but we’re doing a bunch of them.
Those are great insights, but stopping there doesn’t give you enough information to really know why certain purposes are winning. You need to understand the “why” to know how to do more of what works. To do this, you need to dig into each Purpose bucket and analyze which tests are winning in that bucket and why. That’s where you’ll find the real patterns that you can exploit to string together multiple winning tests.
Case Study: Stringing Together Winning Tests
For example, it turns out for this food client that Discovery and Quantity have a high win rate. Why?
We dug in and found that it was a series of tests around upsells. Most of these tests were labeled both Discovery and Quantity because the desired action from presenting product upsells is to increase the quantity of items purchased (aka “cart size”) and we do that by helping the online shopper Discover new products.
Once we learned about this pattern we kept poking at these purposes (Discovery and Quantity via upsells) with more A/B tests and they kept moving the needle:
- Early on, we tested emphasizing product bundles on the homepage with a quiz: +7.2% increase in revenue per visitor with 99% stat sig
- Later we added an upsell to the cart: +4% increase in orders, 98% stat sig
- Then we did another cross sell of similar products: +16% average order value (AOV) with 98% statistical significance. It was only for subscription orders not one time orders, but a nice win nonetheless.
- Then we tested a price threshold after which we’d give a free gift (another product): +7% increase in AOV with 95% stat sig
- Then we offered a product bundle as an upsell of a single item product: +3% increase in revenue per visitor with 93% statistical significance
Case Study: Learning That Luxury Shoppers Are Still Moved By Price
As a second example, let’s look at a client in the women’s luxury apparel space. They have a ton of products (1000+ skus). This analysis paints a different picture than the food client above:
Here Discovery and Price & Value are winning at the highest rate, why?
Insight #1: Huge store has product discovery issues
In the Discovery category, we found tests winning that were about (1) site search and (2) product recommendation containers.
Why are those tests winning?
This client had 1000+ products spread across many departments (shirts, pants, jackets, bags, shoes, lotion and beauty, jewelry). It’s a massive store. So product discovery is a huge pain point (in particular for new customers). Search obviously helped a lot. But so did recommending products at the right part of the page.
They are different ways of attacking the product discovery problem. This is a currently active client and we’re continuing to investigate both, but having this big picture (10,000 ft) view helps focus our work and give us confidence that we are solving important conversion optimization problems, not just throwing darts at a board.
Insight #2: Luxury shoppers are still affected by price
The win rate of the Price & Value category was interesting because this a luxury ecommerce business. Many of their products are designer: e.g. $2000 handbags, $4000 coats. So, everyone’s thinking was that these customers aren’t price constrained and therefore typical department store style Price & Value tactics to increase conversions like promotions and coupon codes shouldn’t affect these luxury shoppers.
Fortunately, we didn’t avoid tapping on the Price & Value brick because of this assumption. You can see in the graph that we’ve only done five tests with a Price & Value purpose, but 2 out of 5 have already shown conversion lifts, for an intriguing 40% win rate.
Those tests included changing some promotion copy on the promo bar at the header, so clearly their “luxury shoppers” do notice those things and are affected by them (+22% in conversion rate, 95% stat sig). Second, we tested a financing option (like Affirm, or AfterPay) and found that including these payment options increased conversion rate significantly (+15% with 94% stat sig).
These early tests shape strategy by telling us, “Ah, this shopper, although on a luxury apparel site, may be price sensitive.”
Insight #3: There is nuance inside each purpose
This is not to say CRO is easy. When we tapped on this Price & Value brick further by putting free shipping and free returns” copy in the cart drawer, or near the add to cart button on the product page, conversion rate decreased.
The lesson is: there is nuance inside each purpose. Offering financing seems important. Sometimes changes in the promo bar copy are important. But maybe “Free shipping and Free Returns” copy is distracting (in particular when it’s near call to action buttons). Or maybe it cheapens the shopping experience and thereby the brand because potential customers expect everything to be free in the shipping and return policies of luxury online retailers? Again, we will continue to investigate. But look at how, with the purposes framework, we can do so knowing that we are following a larger framework and systematically improving the site, instead of just bouncing around from one idea to the next with no larger strategy in place (e.g. “What about the cart abandonment rate?!” “Let’s add some trust signals!”)
Checklist to Implement This Yourself
Here is a checklist of implementing the Purposes Framework at your ecommerce company to build a CRO strategy and escape from tunnel vision testing:
- Start keeping track of every A/B test you’ve run in a spreadsheet
- Next to each test, note the purposes (at most two per test) and whether the test caused a statistically significant change in conversion rate or revenue/AOV
- Add up how many tests you’re running in each purpose bucket. Can you defend the distribution?
- Next, plot the number of tests run along with the win rate for each purpose bucket. Dig into each purpose bucket and look for patterns in why tests are winning or losing.
If you do this, you’ll defeat tunnel vision testing and begin to build an actual CRO strategy for your ecommerce store.
You can see our live database of every ecommerce A/B test we’ve run, labeled, of course, by purpose here.
If you’re interested in working with us, you can learn more and reach out here.
Finally, you can join our email newsletter to get articles like this and AB test breakdowns periodically.