Should you AB test large site redesigns, and if so how?

An issue that’s come up with more than one client for us recently is when a large site redesign is “in the pipeline” and people in the company disagree about whether or not they should test the large redesigns or feature release.

The arguments for not testing large site redesigns is usually some flavor of the following:

These changes have been things we’ve wanted to do for a long time, we know we’re going to implement them, so why test them?

Sometimes I’ve heard it veiled in phrases like “This change is too important.” or “This will be too difficult to test.” But in the end the real reason is that some people in the company just don’t want to test it.

Don’t buy the “this is too hard to test” argument. As I explain below, almost any site change can be AB tested with even the most basic AB testing tool.

You shouldn’t just agree blindly to this idea of not testing large redesigns, though, because if your site is bringing in enough revenue, the ROI of continuous AB testing can be significant.

Why large or inevitable site changes should still be AB tested

I’ll list my arguments for why you should still test large site changes in response to the various objections I listed above that we often here in our work.

My hope is, if you’re the person in the organization arguing for testing, you can use these arguments to help with your battle.

“This change is going to happen regardless of the test outcome”

I have a couple responses to this, one is nicer than the other.

The nicer response: That’s okay if the change will be implemented regardless, but don’t we want to know what the effect on revenue will be?

For example (and this is common): Say an 8-figure ecommerce company is finally updating their entire site design. Their site was made 6 years ago, parts were added piecemeal over time, sales have grown tremendously, but the site hasn’t caught up. It doesn’t have modern design, the checkout process is definitely not optimal, and it’s not mobile friendly. The company knows it needs to update the site. So they hired an expensive agency to totally redesign it.

Anyone who has AB tested elements of an ecommerce site knows that seemingly simple changes to single pages can swing orders by 10%. A 10% change in orders is worth millions for an 8-figure revenue company.

Even if you’re going to implement the large change regardless of the outcome, don’t you want to know if it will reduce revenues by over a million dollars?

If it does hurt sales, you can delay the launch a bit, hypothesize what parts of the new design could be causing the decrease, retest just those elements, and isolate the problem.

The not as nice response: Rolling out a large change without testing it first is irresponsible. See arguments above for why.

Warning: Be careful if the web design agency is arguing that you should just roll out the changes and not test them. It’s not in their interest to see if their new design performs better or not. It’s in their interest to just tell you their design is fantastic and have you love it. Unfortunately customers and their wallets are the true (and ruthless) judge of whether the site is “better”.

The change is too big to test. AB testing is for front end changes, and this changes a lot more.

You don’t have to code the test in Optimizely or whatever testing platform you use. In fact, it’s not recommended that you do this for large tests. Instead, you should code and deploy the new design on your own servers with slightly modified URLs (,, etc) and have the AB testing program simply redirect users to both versions of the site. Most programs can do this.

This method can handle large changes as well.

It will be confusing if customers see two different experiences at the same time

First, every AB testing platform I’ve heard of cookies users so they’ll always see the same variation unless they clear cookies or use a different browser or device.

Second, to protect against confusion if they use separate devices (checking something at work vs. at home, etc.), on the new variation, you can always install a soft popup or bottom of page slider that says “Hey, we’re testing a new site design and would love your feedback. Let us know how you like it by…”.

Lastly, it’s just not that big of a deal. Modern companies test sites all the time now. Sites get updated, they change. Frankly, this idea that if you test a new site for a limited amount of time, and a fraction of customers see an inconsistent site experience for a while will “hurt your brand” is old fashioned marketing thinking.

This thinking relies more on “gut instinct” and opinions of people in conference rooms rather than data from the only opinions that matter: your customers.

Heard any other objections to testing large redesigns or features? Ask away in the comments.

If your business brings in 7 to 8-figures of annual revenue online and you’re interested in getting a conversion audit of your site to being increasing its conversion rate, email us or fill out the form on our homepage.

The ROI of AB Testing: When is AB testing worth it?

I have a simple 2-step criteria for determining if AB testing has a high enough ROI to be seriously considered for your marketing team:

  1. Criteria #1: Your business makes $2,000,000 or more in revenue
  2. Criteria #2: You get 100,000 monthly unique visitors or more to your site

Yes, like any numerical “cutoff”, these numbers are somewhat arbitrary (e.g. In the U.S. at 15.5 years old, you are too young to drive, but at 16.1 years, you’re not).

But, when you zoom out, cutoffs have reasoning behind them: You wouldn’t trust an 8 year old to drive, and waiting until people are 30 to drive is also ridiculous.

Similarly, instead of debating the numbers themselves, let’s discuss the reasoning so you can adjust the above numbers as needed for your business. (Fortunately, unlike a driver’s license, our 2 step criteria above aren’t hard rules).

Finally, I’ll also discuss a 3rd corollary rule that falls out of the reasoning behind the first two: If you fulfill the first two criteria and begin investing in AB testing but aren’t getting routine lifts in orders or sales of 5% – 10% or more, the ROI on your testing investment may also not be worth it.

Why a Revenue a Cutoff? Because AB testing isn’t free.

AB testing isn’t free. Even if you include software costs, the majority of AB testing costs are for the people needed to run the operation. This can come in two forms:

  1. An outside agency, that charges you a monthly rate. From my un-scientific survey of other CRO agencies, they typically charge between $2,000/m – $15,000/m, with most experienced ones being typically towards the middle/high end of that range: $5,000 – $10,000/month.
  2. Using internal employees, which also burn cash (almost always more than an agency).

I’ll soon have an entire article that dives into these costs and discusses how to choose between using in-house resources vs. outside. But for now, as an example, say your monthly AB testing costs are $5,000.

Deciding if a dedicated CRO program is “worth it” at $5,000/month is a matter of estimating:

  1. How much of a revenue lift you’re likely to achieve
  2. How long you’ll have to spend to achieve it

Before you get worked up about this, let me make this clear: it’s impossible to know the answers to these questions a priori.

But luckily, you’re not the first company to do AB testing, so we can look at what’s typical, conservative, aggressive, etc., based on past experience.

Because this is all speculation, however, I’m going to just boil things down to this general rule: assume you can get (1) a 10% increase in revenue in (2) 6 months of testing.

So, that means you’d spend $30,000 ($5,000/m for 6 months) and, if you were making $2 million in revenue before (bare minimum of the criteria above), you’ve increased annual revenues by $200,000/year.

That’s a 567% ROI from the first year of revenues alone. Pretty good.

At this point, let me re-iterate a point I made above about arbitrariness in different words: You may not get a 10% lift in 6 months, or, you may get way more than a 10% lift in revenue in 6 months, or you may get the lift in 2 months.

I can’t say.

You can’t say.

No one can predict this.

But, I know that 10% revenue lift in 6 months is reasonable and achievable — our agency has achieved this multiple times for multiple businesses.

If you want to discuss whether this is achievable for your business and possibly get a few initial ideas to test from us, click here.

Now, similar to the driver’s license example, instead of arguing about whether you you should get a 10% or 25% or 7.72% lift in this time period, let’s zoom out and look at some extremes.

If you told me “For a 6 month AB testing campaign, we’ll need to see a 150% increase in sitewide revenue for it to be worth it.” I’d respond with “Maybe you should focus on something else.”

I’m not saying that’s impossible.

Anything is possible.

It’s possible you could do some user research for a month or two, discover some immense barrier to purchasing that was due to on-site elements, UI/UX, or product related issues, fix them, and see an increase in revenue of 150%.

That is possible.

But it’s just not common, and I’d go as far as to say it’s not even reasonable to expect 150% increase in revenue in a few months.

But 10%? That’s reasonable.

Examining Different Revenue Ranges

Revenue of $500,000 – $1,000,000

Let’s look at lower revenue numbers to see why I put our arbitrary criteria at $2 million.

At $500,000, 10% is $50,000 a year. That’s awfully close to the $30,000 you’d spend doing this for 6 months. Sure maybe you’d see that 10% lift in 3 months, but would you then stop your testing spend just to make sure your ROI was good? No, you’d keep going.

Or, you may only see 5% lift, which is only $25,000.

Either way, it’s not likely that at $500,000 sitewide revenue you’re going to get an outstanding, no-brainer ROI on AB testing.

After months of testing, you may only see an increase in revenue of between $50,000 – $150,000, and spend about $60,000 a year in AB testing costs. So although the ROI could be positive, it’s not a no brainer.

More importantly, for businesses in this revenue range, there are often bigger wins.

In my experience, these bigger wins are usually SEO or paid media optimization. Those two drivers of traffic can move the needle significantly.

I’ve seen in multiple client’s analytics, traffic double from one year to the next. If that fruit is still hanging for your business, pick it first. Don’t worry about trying to eek out 10% or even 35% lifts via CRO.

(Yes, it’d be ideal to do both, but resources are finite.)

Or, your business may be hitting the $500,000 or $1,000,000 revenue mark with some initial paid ad spend that is by no means saturated. If that’s the case, can you double paid ad spend and still maintain sufficient profitability?

The answer may be yes, the answer may be no, but that question should be asked and explored in great detail before starting up a CRO program from scratch.

Revenue Less than $500,000

Startups that are just starting to make money often ask us about AB testing.

If your company is making less than $500,000 in revenue, regardless of how much traffic you have, it’s hard to make the math work for CRO. If you make say $250,000/year, you could easily spend 6 months and $30,000 to get a 10% lift and you wouldn’t be making your money back.

Aside: If you’re thinking, “I’m the founder and I’m barely paying myself,” stop. Your time is the most valuable, and almost always at this stage your focus should be on getting better product market fit.

More importantly, even if you did make your money back (or had a positive ROI) from CRO, it’s not likely to produce a step change in growth. The increase will be incremental, and your business is likely at a stage where you’re looking to find significant growth — so focusing there makes more sense. (Typically that opportunity is better product/market fit or traffic generation.)

Revenues of More than $10,000,000

Now, let’s look at 8 figure businesses and above with the same criteria.

A 10% lift, achieved in 3 – 6 months (reasonable), would yield over a $1,000,000 in annual revenue.

You’re not likely going to spend anywhere near $1,000,000 for 3 – 6 months of testing. Even if you paid $10,000 a month to an agency for 6 months, your ROI is significant: 1567% on a $60,000 investment.

Alternatively, even if you increased revenues by just 5%, that’s $500,000 of extra annual revenue — a 733% ROI.

This math, combined with our observation that 8 figure businesses have more often than not spent years optimizing traffic and paid advertising (and thus either don’t have super low hanging fruit or already have a healthy operation focused on cranking out consistent wins on those two fronts), means that starting up a CRO program for an 8-figure+ business is a no brainer.

Note: Even understanding “what is reasonable?” is easier for an established company. They have a traffic history and a paid media history that they can look to. For example: Has your SEO traffic grown at 20% – 30% for the last 3 years? Great, you can probably expect the same this year. 

The word “starting” in the previous sentence is particularly important because like any marketing initiative, the easiest wins are there for the taking at the beginning, so a site that has never before been formally “optimized” likely has some easy wins that can be realized by CRO (user research + hypothesis generation + ab testing) in the first 3 – 6 months.

Corollary Rule #3: You also need to be seeing consistent conversion lifts to get a good ROI on testing

So we see from the above analysis that a certain revenue range is required for reasonable lifts in conversion rate to yield increases in revenue that make testing “worth it”. But that means the inverse is also true: you need to achieve reasonable lifts in conversion rate for testing to be worth it! 

If you check the boxes on the first two criteria:

  1. We have more than $2MM in revenue
  2. We have more than 100,000 monthly uniques

And you start investing in AB testing, great. That means the potential of seeing a great ROI on testing are there. But it doesn’t mean you will achieve that potential.

Continuing with the numbers in my examples above, you’d need to achieve a 10% lift in orders or sales on a consistent basis to realize your ROI potential.

If you are testing with a typical ad hoc or “conference room” approach (everyone sits around a conference room and throws up their favorite pet ideas to test), then it’s quite likely you won’t achieve that result. Feel free to reach out to us at the link below if you want to discuss was to solve this problem in your organization.

Final thoughts

Again, before you start drafting your comment arguing that my numbers are arbitrary, I want to emphasize (for the 15th time) that I’m fully aware they are arbitrary. Don’t use them to the exact digit. Adjust them for your business, and use the guiding principle, which is:

Before investing in CRO, first compare reasonably achievable revenue increases that CRO could produce versus the equivalent increases you could possibly get from other channels or initiatives (e.g. Improving the product, increasing traffic via SEO, paid ads, content marketing, etc.)

Have a 7 or 8-figure business and want to a evaluation of conversion optimization opportunities for your business? Contact us.

Want updates when articles like this come out? Join our email list (at the top of this page).

The Usability vs. Desirability Framework for Conversion Rate Optimization

Truth be told, optimizing a website is a daunting task.

The same anxiety that makes writers stare at a blank page for hours or a musician futz around with their instrument for an afternoon is what makes it so hard. It’s this question that gnaws at you:

Where should we start? 

There are big things: Let’s redesign the whole site! We need a new ecommerce platform!

There are little things: We need to reword this. What if we put the logos above the diagram?

And a million options in between.

Now, you can read case study after case study online (and they can often be great for inspiration and sparking creativity), but in my experience, random guesses only take you so far. Having a framework for tackling large projects like this can be transformative in turning a situation from “the loudest people’s opinions get tested first” into a systematic process for improving conversion rates and thus revenue.

This article isn’t going to lay out an entire framework for conversion optimization, but it will outline a foundational piece. (more…)

Checkout Form Optimization: Try Asking for Zip Code First

If you have a payment or checkout flow on your site, you should test asking for Zip Code first, and fill in the buyer’s City and State for them.

This should make it easier for them to fill in the form and, ideally, increase checkout rate and thus revenue. Doing this is not as hard since the brains behind it (filing in city/state from zip) has been done by other people.

Let me explain… (more…)

Conversion Teardown: How Dealflicks Can Sell More Tickets with Less Distractions


Note: This is part of a new series of posts I’m calling Conversion Teardowns, where I walk through how to optimize sites from various business types for conversions. I outline my process in the first one. To get updates of future ones, Click Here.

This week, we’re breaking down Dealficks, a movie ticket and concession deals site that honestly seems to be a pretty cool way to score movie tickets for cheap.


What they sell is pretty straightforward: tickets and food at a discount. But their site presents some interesting challenges and questions that would make for good test hypotheses:

  • Do users want to search by location or by movie?
  • They have display ads, is the revenue worth the distraction?
  • Are testimonials relevant for this product?
  • Are app store download CTAs important on the web version?


Conversion Teardowns: Cratejoy (Subscription Box Marketplace)


Note: I’m trying a new style of post called Conversion Teardowns. I’m going to pick a site and break down how I would approach optimizing their conversions rate. We’ll cover SaaS, ecommerce, content marketing, and whatever else readers ask for.

Free Bonus Video: Get a user testing video where I walk through the site with my friend Justin Simon, ask questions along the way, and get his insight into what works and what doesn’t. Click here to get the link to the video, free.

Conversion case studies are great. They show real results from real sites (instead of cliche top 10 lists). But one issue with them is that you only see the nice, shiny, glorious end result.

You don’t often see the thinking and strategy that led to those results. That’s what I’m going to be presenting here (and in future conversion teardowns).

I’ll take a site that I haven’t optimized, haven’t seen the analytics of, and don’t have an affiliation to, and I’ll break down how I would think about optimizing their conversion rate.

Here’s my goal: This will show you how to start thinking about your own site’s conversion strategy — from the beginning. Not just “try a red button color”, but actual processes for doing user research, learning about customers, finding good test hypotheses, and where to prioritize your efforts.

Let’s get started. (more…)

Ecommerce Marketing Case Study: How an Online Shoe Retailer Retargets Customers for Less than $0.03 per Click with Email

Today I’m going to show you an ecommerce marketing case study of how a French online shoe retailer used product bonuses to retarget visitors for one one-hundredth the cost of Google or Facebook retargeting.

Retargeting by email also gets him click through rates in the 14%+ range — a massive improvement from typical PPC retargeting click through rates.

He did this without spending hours on social media.

And he did this without fancy web design tricks.

He also didn’t have to spend tons of time creating new content — he just used what he already had.

Let’s meet him. (more…)

Exit PopUp Optimization: How I Increased Email Subscribers By 216% in One Month

Today, I’m going to show you how I increased a news site’s email opt-in rate by 216% in 30 days.

We did this…

Without making popups re-appear every few seconds (and annoying readers)

And without having to make a unique lead magnet (content upgrade) for every post.

Instead, we used what I call the Topicbox Technique. I’ll walk you through exactly how we did it, step by step, in this case study. (more…)

Social Media Statistics: Email Still Has the Highest Engagement

“I want more traffic”. Have you said that?

Here’s the reality: You don’t really want more traffic, you want more sales. And sales require more than just traffic.

You want that traffic to convert, to do something: click, comment, opt-in, take a credit card out, buy, refer friends, rinse, repeat.

What kind of people do all these things?


That’s what you need: fans.

People who keep coming back to your store, people who read everything you write, people who leave comments on your articles, and people who buy your products and share your stuff with their friends.

Regular ol’ traffic needs to be replenished. Fans come back. And they tell their friends. That’s why fans are more important than anything else — especially if your business is young.

I’m not the only one that agrees, various, famous people have said this before.

But how do you get fans?

Most people think “Ok, I need Twitter followers”, or “a Facebook page.”

But is that the best way to get the fans you so desperately need?

Sidebar and End of Post Email Opt-in Forms: Don’t Waste Your Time

A lot of clients ask me, “Can you help us grow our email list?”

So I ask, “Ok, what are you doing to capture subscribers now?”

If they have a blog, their answers inevitably involve this word: “sidebar”.

“Well, we have a form in the sidebar.”

“I’m giving away this ebook in the sidebar…”


I take a deep breath, squirt some more Mio in my water, and ask “Ok, what else are you doing?”

Of course, 99% of the time, I’ve already looked at their blog — as I’ve looked at hundreds and hundreds of other blogs — and the vast vast vast majority of them have two opt-in forms that convert extremely poorly: (more…)