The ROI of AB Testing: When is AB testing worth it?


I have a simple 2-step criteria for determining if AB testing has a high enough ROI to be seriously considered for your marketing team:

  1. Criteria #1: Your business makes $2,000,000 or more in revenue
  2. Criteria #2: You get 100,000 monthly unique visitors or more to your site

Yes, like any numerical “cutoff”, these numbers are somewhat arbitrary (e.g. In the U.S. at 15.5 years old, you are too young to drive, but at 16.1 years, you’re not).

But, when you zoom out, cutoffs have reasoning behind them: You wouldn’t trust an 8 year old to drive, and waiting until people are 30 to drive is also ridiculous.

Similarly, instead of debating the numbers themselves, let’s discuss the reasoning so you can adjust the above numbers as needed for your business. (Fortunately, unlike a driver’s license, our 2 step criteria above aren’t hard rules).

Finally, I’ll also discuss a 3rd corollary rule that falls out of the reasoning behind the first two: If you fulfill the first two criteria and begin investing in AB testing but aren’t getting routine lifts in orders or sales of 5% – 10% or more, the ROI on your testing investment may also not be worth it.

Why a Revenue a Cutoff? Because AB testing isn’t free.

AB testing isn’t free. Even if you include software costs, the majority of AB testing costs are for the people needed to run the operation. This can come in two forms:

  1. An outside agency, that charges you a monthly rate. From my un-scientific survey of other CRO agencies, they typically charge between $2,000/m – $15,000/m, with most experienced ones being typically towards the middle/high end of that range: $5,000 – $10,000/month.
  2. Using internal employees, which also burn cash (almost always more than an agency).

I’ll soon have an entire article that dives into these costs and discusses how to choose between using in-house resources vs. outside. But for now, as an example, say your monthly AB testing costs are $5,000.

Deciding if a dedicated CRO program is “worth it” at $5,000/month is a matter of estimating:

  1. How much of a revenue lift you’re likely to achieve
  2. How long you’ll have to spend to achieve it

Before you get worked up about this, let me make this clear: it’s impossible to know the answers to these questions a priori.

But luckily, you’re not the first company to do AB testing, so we can look at what’s typical, conservative, aggressive, etc., based on past experience.

Because this is all speculation, however, I’m going to just boil things down to this general rule: assume you can get (1) a 10% increase in revenue in (2) 6 months of testing.

So, that means you’d spend $30,000 ($5,000/m for 6 months) and, if you were making $2 million in revenue before (bare minimum of the criteria above), you’ve increased annual revenues by $200,000/year.

That’s a 567% ROI from the first year of revenues alone. Pretty good.

At this point, let me re-iterate a point I made above about arbitrariness in different words: You may not get a 10% lift in 6 months, or, you may get way more than a 10% lift in revenue in 6 months, or you may get the lift in 2 months.

I can’t say.

You can’t say.

No one can predict this.

But, I know that 10% revenue lift in 6 months is reasonable and achievable — our agency has achieved this multiple times for multiple businesses.

If you want to discuss whether this is achievable for your business and possibly get a few initial ideas to test from us, click here.

Now, similar to the driver’s license example, instead of arguing about whether you you should get a 10% or 25% or 7.72% lift in this time period, let’s zoom out and look at some extremes.

If you told me “For a 6 month AB testing campaign, we’ll need to see a 150% increase in sitewide revenue for it to be worth it.” I’d respond with “Maybe you should focus on something else.”

I’m not saying that’s impossible.

Anything is possible.

It’s possible you could do some user research for a month or two, discover some immense barrier to purchasing that was due to on-site elements, UI/UX, or product related issues, fix them, and see an increase in revenue of 150%.

That is possible.

But it’s just not common, and I’d go as far as to say it’s not even reasonable to expect 150% increase in revenue in a few months.

But 10%? That’s reasonable.

Examining Different Revenue Ranges

Revenue of $500,000 – $1,000,000

Let’s look at lower revenue numbers to see why I put our arbitrary criteria at $2 million.

At $500,000, 10% is $50,000 a year. That’s awfully close to the $30,000 you’d spend doing this for 6 months. Sure maybe you’d see that 10% lift in 3 months, but would you then stop your testing spend just to make sure your ROI was good? No, you’d keep going.

Or, you may only see 5% lift, which is only $25,000.

Either way, it’s not likely that at $500,000 sitewide revenue you’re going to get an outstanding, no-brainer ROI on AB testing.

After months of testing, you may only see an increase in revenue of between $50,000 – $150,000, and spend about $60,000 a year in AB testing costs. So although the ROI could be positive, it’s not a no brainer.

More importantly, for businesses in this revenue range, there are often bigger wins.

In my experience, these bigger wins are usually SEO or paid media optimization. Those two drivers of traffic can move the needle significantly.

I’ve seen in multiple client’s analytics, traffic double from one year to the next. If that fruit is still hanging for your business, pick it first. Don’t worry about trying to eek out 10% or even 35% lifts via CRO.

(Yes, it’d be ideal to do both, but resources are finite.)

Or, your business may be hitting the $500,000 or $1,000,000 revenue mark with some initial paid ad spend that is by no means saturated. If that’s the case, can you double paid ad spend and still maintain sufficient profitability?

The answer may be yes, the answer may be no, but that question should be asked and explored in great detail before starting up a CRO program from scratch.

Revenue Less than $500,000

Startups that are just starting to make money often ask us about AB testing.

If your company is making less than $500,000 in revenue, regardless of how much traffic you have, it’s hard to make the math work for CRO. If you make say $250,000/year, you could easily spend 6 months and $30,000 to get a 10% lift and you wouldn’t be making your money back.

Aside: If you’re thinking, “I’m the founder and I’m barely paying myself,” stop. Your time is the most valuable, and almost always at this stage your focus should be on getting better product market fit.

More importantly, even if you did make your money back (or had a positive ROI) from CRO, it’s not likely to produce a step change in growth. The increase will be incremental, and your business is likely at a stage where you’re looking to find significant growth — so focusing there makes more sense. (Typically that opportunity is better product/market fit or traffic generation.)

Revenues of More than $10,000,000

Now, let’s look at 8 figure businesses and above with the same criteria.

A 10% lift, achieved in 3 – 6 months (reasonable), would yield over a $1,000,000 in annual revenue.

You’re not likely going to spend anywhere near $1,000,000 for 3 – 6 months of testing. Even if you paid $10,000 a month to an agency for 6 months, your ROI is significant: 1567% on a $60,000 investment.

Alternatively, even if you increased revenues by just 5%, that’s $500,000 of extra annual revenue — a 733% ROI.

This math, combined with our observation that 8 figure businesses have more often than not spent years optimizing traffic and paid advertising (and thus either don’t have super low hanging fruit or already have a healthy operation focused on cranking out consistent wins on those two fronts), means that starting up a CRO program for an 8-figure+ business is a no brainer.

Note: Even understanding “what is reasonable?” is easier for an established company. They have a traffic history and a paid media history that they can look to. For example: Has your SEO traffic grown at 20% – 30% for the last 3 years? Great, you can probably expect the same this year. 

The word “starting” in the previous sentence is particularly important because like any marketing initiative, the easiest wins are there for the taking at the beginning, so a site that has never before been formally “optimized” likely has some easy wins that can be realized by CRO (user research + hypothesis generation + ab testing) in the first 3 – 6 months.

Corollary Rule #3: You also need to be seeing consistent conversion lifts to get a good ROI on testing

So we see from the above analysis that a certain revenue range is required for reasonable lifts in conversion rate to yield increases in revenue that make testing “worth it”. But that means the inverse is also true: you need to achieve reasonable lifts in conversion rate for testing to be worth it! 

If you check the boxes on the first two criteria:

  1. We have more than $2MM in revenue
  2. We have more than 100,000 monthly uniques

And you start investing in AB testing, great. That means the potential of seeing a great ROI on testing are there. But it doesn’t mean you will achieve that potential.

Continuing with the numbers in my examples above, you’d need to achieve a 10% lift in orders or sales on a consistent basis to realize your ROI potential.

If you are testing with a typical ad hoc or “conference room” approach (everyone sits around a conference room and throws up their favorite pet ideas to test), then it’s quite likely you won’t achieve that result. Feel free to reach out to us at the link below if you want to discuss was to solve this problem in your organization.

Final thoughts

Again, before you start drafting your comment arguing that my numbers are arbitrary, I want to emphasize (for the 15th time) that I’m fully aware they are arbitrary. Don’t use them to the exact digit. Adjust them for your business, and use the guiding principle, which is:

Before investing in CRO, first compare reasonably achievable revenue increases that CRO could produce versus the equivalent increases you could possibly get from other channels or initiatives (e.g. Improving the product, increasing traffic via SEO, paid ads, content marketing, etc.)

Have a 7 or 8-figure business and want to a evaluation of conversion optimization opportunities for your business? Contact us.

Want updates when articles like this come out? Join our email list (at the top of this page).

The U-D Conversion Rate Optimization Framework: Usability and Desirability

Truth be told, optimizing a website is a daunting task.

The same anxiety that makes writers stare at a blank page for hours or a musician futz around with their instrument for an afternoon is what makes it so hard. It’s this question that gnaws at you:

Where should we start? 

There are big things: Let’s redesign the whole site! We need a new ecommerce platform!

There are little things: We need to reword this. What if we put the logos above the diagram?

And a million options in between.

Now, you can read case study after case study online (and they can often be great for inspiration and sparking creativity), but in my experience, random guesses only take you so far. Having a framework for tackling large projects like this can be transformative in turning a situation from “the loudest people’s opinions get tested first” into a systematic process for improving conversion rates and thus revenue.

This article isn’t going to lay out an entire framework for conversion optimization, but it will outline a foundational piece. (more…)

Checkout Form Optimization: Try Asking for Zip Code First

If you have a payment or checkout flow on your site, you should test asking for Zip Code first, and fill in the buyer’s City and State for them.

This should make it easier for them to fill in the form and, ideally, increase checkout rate and thus revenue. Doing this is not as hard since the brains behind it (filing in city/state from zip) has been done by other people.

Let me explain… (more…)

Conversion Teardown: How Dealflicks Can Sell More Tickets with Less Distractions

CRO_teardown_graphics

Note: This is part of a new series of posts I’m calling Conversion Teardowns, where I walk through how to optimize sites from various business types for conversions. I outline my process in the first one. To get updates of future ones, Click Here.

This week, we’re breaking down Dealficks, a movie ticket and concession deals site that honestly seems to be a pretty cool way to score movie tickets for cheap.

Showtimes__Movie_Tickets__Theaters____Coupons___Dealflicks

What they sell is pretty straightforward: tickets and food at a discount. But their site presents some interesting challenges and questions that would make for good test hypotheses:

  • Do users want to search by location or by movie?
  • They have display ads, is the revenue worth the distraction?
  • Are testimonials relevant for this product?
  • Are app store download CTAs important on the web version?

(more…)

Conversion Teardowns: Cratejoy (Subscription Box Marketplace)

Banners_and_Alerts_and_deveshdesign_canvas

Note: I’m trying a new style of post called Conversion Teardowns. I’m going to pick a site and break down how I would approach optimizing their conversions rate. We’ll cover SaaS, ecommerce, content marketing, and whatever else readers ask for.

Free Bonus Video: Get a user testing video where I walk through the site with my friend Justin Simon, ask questions along the way, and get his insight into what works and what doesn’t. Click here to get the link to the video, free.

Conversion case studies are great. They show real results from real sites (instead of cliche top 10 lists). But one issue with them is that you only see the nice, shiny, glorious end result.

You don’t often see the thinking and strategy that led to those results. That’s what I’m going to be presenting here (and in future conversion teardowns).

I’ll take a site that I haven’t optimized, haven’t seen the analytics of, and don’t have an affiliation to, and I’ll break down how I would think about optimizing their conversion rate.

Here’s my goal: This will show you how to start thinking about your own site’s conversion strategy — from the beginning. Not just “try a red button color”, but actual processes for doing user research, learning about customers, finding good test hypotheses, and where to prioritize your efforts.

Let’s get started. (more…)