Ready for Cyber Monday? Here’s One Critical Thing Entrepreneurs Forget About the Biggest Online Shopping Day

The fall equinox is behind us now.

Our kids are back in school, and the weather outside is slowly getting crisp and cool.

It's autumn. And you know what that means:

The holidays are just around the corner.

And for entrepreneurs, one of the most important days in the holiday season is Cyber Monday.

It's Black Friday's hip young millennial cousin, the biggest online shopping day of the entire year.

Everyone's going to be looking for great deals — and most importantly, they're guaranteed to be spending money.

You want them spending that money with you, not your competitor.

And one of the best ways to make sure your Cyber Monday is a resounding success is to do some A/B testing ahead of time to make sure your marketing is on point.

But A/B testing for Cyber Monday is unique, and it has some of its own little quirks.

In a recent blog post, Shopify explains how to go about A/B testing in anticipation of the Cyber Monday shopping rush.

Why is testing during Black Friday Cyber Monday different?

You might be wondering, at this point, why you can’t just read The Beginner's Guide to Simple A/B Testing and be on your way.

Well, testing during BFCM (or any other exceptional time, like Father’s Day or Boxing Day) is a bit different.

There’s a sneaky something called sample pollution, which comes into play during BFCM.

I remember listening to Bart Schutz of Online Dialogue presenting at CTA Conference a few years ago.

He said something that’s always stuck with me: “If you don’t know about sample pollution, stop testing.”

Sample pollution is essentially any external factor that impacts your tests and leads to invalid results.

Unfortunately, sample pollution is often present without the experimenter’s knowledge.

Here’s how it creeps in:

Running your test for too long. 

There’s no universal answer to how long you should let a test run, but the rule of thumb is two business cycles, which usually works out to two to four weeks.

If you stop a test before the pre-calculated sample size is reached, you’ve polluted the data.

If you stop a test too late, you run the risk of anomalies (your co-founder started a PPC campaigns, it’s BFCM, etc.)

Not testing on mobile and desktop separately. 

We all use multiple devices, right?

Let’s say Daniel visits your store and sees variation A on his iPhone. A few days later, he visits your store from his laptop.

There’s a 50% chance he’s going to see a different variation than he did the first time.

To mitigate this, you should be testing mobile and desktop traffic separately.

Visitors deleting their cookies.

How often do you delete your cookies?

Other people delete their browsing history, too.

Let’s say Susan visits your store and then deletes her cookies. When she returns to your store, there’s a 50% chance she’s going to see a different variation than she did the first time.

And that’s just to name a few.

The longer your test runs, the more likely it is that sample pollution will sneak its way in.

So, as a general rule, you need to be able to reach your calculated sample size in four weeks or less.

Though, you can never truly eliminate sample pollution—you can only limit it.

On top of all this, you want to have a representative sample.

That means you want to make sure the people who are involved in your test are representative of your typical audience.

BFCM makes that difficult because you might: see a spike in traffic, see a spike in purchases, offer deep discounts for a limited time, see people spending more than they usually would, see more bargain shoppers abandoning their carts, etc.

What happens during BFCM is not typical, it’s not representative of your usual audience.

So how can you apply insights learned during an unusual time to your usual audience? You can’t, which makes BFCM testing different.

You can find more tips for this year's Cyber Monday over at Shopify.