5 examples of offline CRO to help shape your online A/B tests

BY STU BOWKER | 13th August 2019 |

Conversion rate optimisation isn't just for websites. For years, companies have been improving their user experiences, offline.

I was first made aware of offline CRO after a discussion with Craig Sullivan. We've become so blinkered by our online testing, we don't notice the experiments going on all around us. For example, Craig described how UPS drivers don't turn left. This saves the company both time and money (see more on this further down the page).

In March 2018 I gave a talk at BrightonCRO on offline CRO. Here I bring you a collection of stories of how companies are experimenting offline.

1. Baggage claim

Image of an airport baggage claim carousel with people waiting for their bags.

Photo: hiphen.net

No doubt this is a familiar sight for many of us. Not just baggage claim itself, but the endless wait that ensues for your case to arrive. At Houston Airport in Texas, they received a lot of complaints highlighting this exact problem, the wait for baggage was too long.

Experiment 1: Add more baggage handlers. This worked and the average wait time came down to 7 minutes.

But this wasn’t good enough, the complaints kept coming. So they decided to look at the issue in more detail.

Upon further investigation, they noticed the average traveller had to walk 1 minute from the arrival gate to baggage claim. Then they waited a further 7 minutes for their bags to turn up. Total time: 8 minutes. Management were unable to add more baggage handlers but they could control the walk to the baggage claim area.

Experiment 2: No they didn’t make people walk slower. Instead, they moved the arrival gates as far away from the main terminal as possible and used the outermost baggage carousels. This resulted in a 7 minute walk and a 1 minute wait at the carousel. Total time: 8 minutes. There were no longer any complaints about waiting times. But why was this? Although the total time remained the same, the travellers were occupied with walking. They were unaware of the time as their goal was to reach baggage claim.

Lesson: Our perception of time can be a cause of frustration if we're left waiting. Keep us occupied in that time and there's no frustration. That's why time flies when we're having fun, we're not concious of time.

2. Coinstar

Image of a Coinstar machine.

Photo: family-budgeting.co.uk

Coinstar, these are the machines you often see in supermarkets that convert your loose change into cash. They too had complaints of waiting time. In this case though, the waiting times weren’t long enough.

The machine counts the coins almost instantly. During pre-launch user testing, people felt this speed meant it wasn't accurate.

Of course people don’t want to be short changed and trust in the service is paramount if it is to succeed. So, what did Coinstar do about it?

Experiment: The coins were counted in the same amount of time, but the results shown to the customer were delayed. During this delay, the noise of coins running through the machine could continue to be heard thanks to a recording through a speaker.

The test worked. Despite the experience being completely fabricated, the experience now matched user expectations. Trust was instilled.

Lesson: If they hadn’t conducted user testing and not addressed this issue, then the success of the service may have had a very different outcome.

3. All you can eat buffet

A truck stop with an all you can eat buffet improves customer health with some subtle changes. Watch the video to see what they are:

The size and colour of the plates can have a drastic effect on our eating habits without us even thinking about it. I always find it amazing how we can be manipulated with cues that are hiding in plain sight. But with that we must also be careful not to abuse this power at the expense of customers. This is where 'dark patterns' come into play, but that's whole other blog post.

Lesson: Size matters apparently.

4. Fire exits

What's the quickest way to get people out of a burning building? Put something in their way, surely not? Press play to find out.

Lesson: Even the fire marshal was surprised by the results. This goes to show, even the most seemingly counter-intuitive ideas can be life-saving. When you're brainstorming solutions, there's no such thing as a bad idea.

5. UPS

Image of a UPS truck.

Photo: youtube.com

Another experiment that seems counter-intuitive is this one from UPS. To optimise their delivery routes, UPS decided to stop looking at the shortest route and find other ways to improve efficiency.

Experiment: Avoid turning left through oncoming traffic at a junction. Despite potentially having to travel in the opposite direction of their destination.

This had a significant impact on the business:

Reduced the chances of an accident
Cut delivery delays caused by waiting for a gap in the traffic
Saved millions of gallons of fuel each year
Less CO₂ emissions
Reduced costs
Increased capacity for more deliveries

This infographic sums up some of the figures from this experiment:

Infographic showing UPS save 10 million gallons of fuel per year, 100,000 metric tons of CO2 emissions and 6 to 8 fewer miles driven per route. The equivelent to 21,000 cars taken off the road.

Photo: cnn.com

Lesson: Subtle changes can have a major impact on your business. But you don't know that until you start to experiment.

BONUS: 6 more examples of offline CRO

Playing stereotypical French music in a supermarket's wine aisle boosted sales of French wine
A study showed that contactless card payments are perceived to be cheaper than cash
Removing the dollar sign from restaurant menus improved sales by 8%
Placing a 'This week's best selling ale' sign in a bar increased sales of that ale by 2.5 times, even if it was false
Car dealers offering 'employee discounts' to everyone, resulting in a surge of sales. Despite previous offers being the same or even better
If a recipe is written in a difficult to read font, the recipe will be perceived to be harder to execute

Summary

These are all great examples of problems originating from user research, the business and a desire to increase sales. The problems are not necessarily unfamiliar, but the approach taken to arrive at the solutions are creative to say the least. If it's occupying our time or filling our plates, it's fundamentally all psychological tweaks that made the difference.

The same applies to websites. Use great content design and copy to make your users happier.

If you know of any other offline experiments I'd love to hear about them. Get in touch on twitter @StuBowker.

Slides, resources and further reading:

* denotes affiliate links (for awesome books you need to read)