6 use cases for random sample branching—an underrated A/B testing approach

If you’re like most marketers, you have the basics of A/B testing down to a science. You develop a hypothesis about your tactic, decide which metric will determine a winner, choose your variable to test, and set the test live.

Brands can gain a lot from traditional A/B testing, but once you’ve built a bigger audience, you can do more advanced kinds of testing.

In this blog, we’ll delve into random sample branching: when you should use it, why it’s useful, and which specific use cases it can be most powerful for.

What is random sample branching?

Random sample branching is an A/B testing technique in which you split an automated flow into branches and test certain tactics on different segments within the automated flow.

When should you start using random sample branching?

Typically, you want to explore random sample branching once you have over 100,000 people on your email list. A large list is crucial because you need a decent sample size for each tactic you test.

“Any brand chasing giant changes should try random sample branch testing,” says Hani Rifai, head of engineering and CRO at StepOne, a sustainable activewear brand based in Sydney, Australia. “If you’re looking for a huge increase in revenue, this is a good way to get there.”

Rifai offers a few caveats, too. “In addition to list size, brands need to make sure they have someone on their team who can aggregate all the metrics,” he says.

But if you’re not swimming in cash just yet, the approach may still be worth a try. Christian Nørberg Enger, CPO at Segmento, a fast-growing full-service performance marketing agency in Denmark, says he uses random sample branch testing with brands as small as those who bring in $100,000 a month.

“It can be helpful to get started with this kind of testing early, then double down on the approaches that work as your list grows,” he says. “The smaller your business, the fewer your branches are relative to the volume that the brand gets per metric.”

“Of course” Enger adds, “bigger brands can test faster and get faster results relative to their volume.”

6 use cases for random sample branching

As chief marketer at Retention Commerce, I help brands scale their businesses through their owned channels. A/B testing is a crucial part of every marketing strategy, but lots of brands just aren’t that familiar with the benefits of random sample branching specifically.

Here are 6 ways we use it to help brands make sure their content resonates with their audience.

1. Test a series of messages with a specific content approach

Let’s say you have 500,000 decently engaged people on your email list, along with thoughtful flows that are tried and tested. It’s not broken, so why fix it?

But you’re a creative thinker, always looking for growth, so you’d love to try a harder sell approach to see what results it produces. And not just in your welcome flow, but throughout the entire library of automated flows:

• Welcome series
• Abandoned cart
• Browse abandonment
• Post-purchase
• Win-back
• Sunset

With such a large list, you have room to experiment. Random sample branching is a great tactic to use when you want to test out a few different approaches across your entire customer journey.

Let’s say you want to compare a more aggressive approach to a softer and more nurturing tone, but you don’t want to sacrifice your existing journey that has worked well for you up to this point.

By using a conditional branch to group your audience with a custom property, you can randomly select 12.5% of subscribers and give them the harder-sell approach, including more messages, extra SMS, and even push notifications throughout the customer journey, for an entire year.

Then, take another 12.5% of subscribers and give them a much softer approach in all their messaging over the same length of time. This group may get longer messages that focus on your brand’s story, or the emails may have fewer CTAs with less transactional language.

You can then sit back and let your automations work their magic, all while the other 75% of your subscribers are getting your tried-and-true messaging—meaning, you don’t stand to risk very much if one of your approaches tests poorly.

This is also an excellent way to test:

• Which unique selling point (USP) should be in the forefront
• Different design philosophies across an entire automated flow
• Which message in the flow is ideal for conversion

“When you create very different experiences for your subscribers, it’s a risk, but there’s a possibility of seeing a giant uplift,” Rifai says. “At Step One, we rarely test incremental changes. Small changes may give you a 3–4% increase in revenue, but the kinds of big swings you take when testing an entire approach can produce 20–50% increases.”

2. Test the number of messages you’re sending

To continue the example above, imagine you’re sending 15% of your subscribers the harder-sell approach. Instead of 3 emails, you want to send 5—plus an SMS. You know this is risky—it might be too much—but you’re curious to see what effect it produces.

Random sample branching allows you to have Klaviyo select a random set of subscribers who will get those two extra emails and an extra SMS. You can drag a split to the end of your automated flow, add a time delay, and then add the additional messages.

Using random sample branching is an excellent way to look at your core automations holistically. Often, brands want to understand which exact email converted someone. But the reality is that maybe the first email laid the groundwork, while email number two convinced the recipient to buy with some social proof—and email number three simply reminded them to buy, going in with a strong offer.

If you send an automated flow that only includes the converting email, you’d never know how much work the first two were contributing to prime the conversion. Sometimes, prior messages are an important part to get someone to convert.

Random sample branching allows you to compare entire automated flows against each other, giving you a clear picture of what’s working and where you’ve got room to improve.

3. Test your SMS strategy—without sending too many messages

If you’re in the beginning stages of your SMS marketing program, you may want to try adding a marketing text to your check-out abandonment flow. But sending an SMS to everyone who has opted into SMS and gets the check-out abandonment flow would be expensive. Plus, it might bother even those who’ve opted in.

And, what message should you even try?

Random sample branching is a great way to test this addition without blowing your budget or risking annoying all your subscribers.

Here, in this example of an abandoned check-out flow where we only want to test sending SMS to just 20% of eligible SMS subscribers. We still want everyone to receive the email version, regardless if they receive SMS or not.

To do this:

1. Split on users that can receive SMS.
2. Decide what percentage of your eligible subscribers will get the SMS.
3. Drag a conditional split to the point in the flow where you want the SMS to send.
4. Add a SMS message to the “Yes” path.
5. Route the rest of the folks on the “No” path around the SMS.

4. Test different time delays

Sometimes it’s not the content of your messaging, but the timing. Random sample branching allows you to test a new time delay on whichever percentage of your audience feels right.

To do this:

1. Create a conditional split based on what percentage of people you want to test.
2. On the “Yes” path, drag in a time delay and set the timing parameters.
3. On the “No” path, drag in the other time delay and set the timing parameter.
4. Clone the message so that both paths have the same message after the timing delays.
5. Join the path back to the flow.

Through random sample branching, Enger and his team at Segmento have discovered that, with older customers, a slower, less pushy cadence performs better.

“We do ‘dynamic testing,’” he says. “We set up our flow to split upon time delays in different branches at first to find the best possible time to send the email to different customer segments. This is one of the ways we test repeatedly to find the best fit.”

Rifai and his team use random sample branching to switch up the timing of SMS messages. “It makes it feel less predictable,” he says.

5. Test the order of messages

Random sample branching is a great way to see if the order you’re sending a series of emails in is working. Switch up the order for a small percentage of your audience and see if it moves any of the important needles.

One frequent test we like to evaluate is to look at the whole automated flow and see which messages have higher engagement or conversion rates. We may want to try moving the higher-performing message earlier in the flow to see if we can get the benefit of that conversion sooner.

Using a random sample branch for 15–20% of subscribers, we simply duplicate the flow in this new branch and re-order the high converting message. This tells us if moving the message earlier in the flow will still yield similar or higher conversion rate, but much sooner.

6. Test a discount—on a small percentage

The final use case for random sample branching is when you want to test a discount sequence—without going all in at once. A typical discount sequence starts with a message that contains a specific offer, has small reminders in the messages that follows, and ends with a discount expiration reminder.

By using a random sample branch, we can send that discount code in a message that only goes to a small percentage of your subscribers. We can also follow up in the last reminder message with the same offer.

Using a branch instead of a message lets you test the entire flow with the same offer so that the subscriber receives a coherent series of messages with the same discount.

A/B testing is hopefully already a regular part of your marketing strategy. Consider random sample branching for that extra, holistic view.