Amazon makes over $800 million a month in extra sales by continuously improving their search results - and all of their competitors (and probably every other online store) could nearly double their total online sales if they followed suit.
I know what you’re thinking: “Amazon has built-in advantages like shipping and logistics, and Prime -- they just have so much scale that competitors can’t match.”
You’re right to be skeptical. But, stay with me, and I’ll prove to you that Amazon’s “trick” isn’t one of those types of advantages.
In fact, it can be applied to every business with a search bar - right now, today.
I didn’t set out in search of this. I stumbled across this in my research for a presentation on an entirely different topic.
My presentation was supposed to be about “How people search when they’re not on Google.” Amazon is the third largest search engine, and so I wanted to compare it to others on the list. I was looking to see what percent of people that visit Amazon actually do a search -- turns out it’s about 1 in 3. That’s when I noticed this:
Amazon’s conversion rate shoots up 6x when visitors do a search (2% → 12%)
Amazon's conversions average 2.17% when someone visits the site. For visitors that run a search, that conversion jumps to 12.29%. That’s nearly a 6x increase!
Holy Sh**! Is that normal?
Because that seems like a pretty big jump. I’m not talking about the base 2% or the 12% that it goes to, or even the 10% jump. Although those are pretty impressive themselves, the thing that seems really amazing to me is the 6x increase.
A quick note on methodology: Before I go too much further, I should explain how I’m getting this data. I’m using Nacho Analytics, which loads the browsing behavior of millions and millions of users into Google Analytics (GA) in real-time. I didn’t hack into Amazon’s Google Analytics account - in fact, to my knowledge, they don’t even use it.
Amazon is obviously enormous, logging billions of page views per month. GA only allows up to 10 million page views for free (before you have to upgrade to GA360).
So, to load all the traffic of a site that size into a single free GA account, we do it in thousands. So, all of the traffic numbers you see - pageviews, sessions, users, etc. - those are in thousands.
For example, in the image above, we’re not looking at 567,790 sessions - that’s 567,790,000. But, each of those 567,790 sessions map directly to 1 precise underlying user. They just represent 999 more.
Not all sites loaded into Nacho Analytics are loaded in thousands. When a site gets hundreds of millions, it is loaded in 100s; if it’s tens of millions, it is loaded in 10s.
If a website gets less than 10M page views, there’s no multiplier at all. In this article, I’m going to be looking at four sites: Amazon (1000x), Walmart (100x), Etsy (100x), and Bestbuy (10x).
The sales numbers you see do not need to be multiplied by 1000, I’ve already done that by setting the “Goal Value” to 1000x the average sale amount. I derived the average sale amount ($77.30) by examining what products were purchased by a random sample of 40 sessions (on Amazon, you can look at the products in the cart, but not the amount).
So, I took a look at the second biggest online retailer: Walmart.
Okay. About the same percent - about ⅓ of Walmart visitors do a search. I 100% expected Amazon to have higher conversion rates. Walmart’s 1.1% conversion rate goes to 2.9% with a search, a 2.4x boost. Not even close to the 6x Amazon is seeing.
So, then I looked at the number 3 retailer, Etsy. (Actually Ebay comes before both Walmart and Etsy, but I don’t really think it’s as good of a comparison, y’know because of the whole auction thing)
Etsy’s got a bit higher base conversion rate than Walmart, and a bit more of boost at 3x. But, still nothing like Amazon’s.
I looked at one more site, this time a bit more tech-heavy just so I was covering all the bases. So, I’ve got - Broad retail: Walmart, Deep-Niche: Etsy, and Tech-heavy: Best Buy. And across the board, Amazon gets a 2-3x more boost when people do a search.
How are Amazon’s users different from everyone else’s?
Causation vs. Correlation
Every click a visitor takes into a website tends to see an increase in conversion rate.
Nobody ever does that, but I think it illustrates the sort of classic recurring issue in conversion rate optimization (CRO) - is what we’re seeing the behavior of buyers, or are we causing people to buy?
At some point, while I was struggling through this problem, I reached out to my friend Brian Massey (the Founder of Conversion Sciences). I asked him if he had any ideas, and he rightly asked: “Do people who are ready to buy use site search more, or does site search make people buy?”
And so, let me be clear about that. I’m not claiming causation - not yet anyway. I run across this kind of thing all the time. And there's ALWAYS an explanation. Finding causation is extremely (and somewhat depressingly) rare. My disposition is that it is not causation unless I can eliminate every alternative explanation.
That’s not to say that finding the reason for the correlation - the reason buyers behave differently isn’t interesting. But, I do try to destroy the notion of causation as quickly as possible.
But what if it IS causal? This could be a HUGE deal. It really is worth chasing down no matter how unlikely.
Because if it’s real, it means $800 million more per month to Amazon - a $10 Billion per year “growth hack.”
If Walmart could learn from them, it’d mean a +80% increase in total online revenue ($37.5 million / month). And the same for Etsy, Best Buy, and probably every other online store.
So, let’s burn that precious thought to the ground before it has a chance to infect you with inspiration. ;)
Maybe it’s a specific channel that’s over performing
My first thought was that perhaps it’s a specific channel that’s vastly over performing.
Often enough, actually, sites have a ridiculously high-converting referral channel.
When you drill into it, you see that the “referrer” that’s driving all the conversions is Paypal. Basically, the shopping cart sends buyers over to Paypal, and then they return to the see the receipt page. Thus, Paypal’s conversion rate seems to be like 100%.
So, let’s look at Amazon.
Amazon doesn’t have anything crazy going on. While their base conversion rates vary a bit by channel, the boost they get with a search stays pretty consistent, and it’s still better than the other sites we’ve seen.
Amazon’s most significant source of traffic is Organic search from Google -- it accounts for about half of all of Amazon’s traffic. And actually, Google does a better than baseline job of putting them on the right page. But, when people come in from Google, and do a second search - they have a 13.26% conversion rate - which is almost 6x higher.
As a consequence - despite the fact that over twice as many people search on Google, land on Amazon, and don’t do the search on Amazon -- Amazon still gets 2 and a half times more sales from the people that use their search rather than Google’s.
Let’s look at Walmart and Best Buy.
Nothing stands out in either of these. The ratios are consistent across all major channels. Keep in mind, I’m looking for a 2-3x change here, and any small differences aren’t really going to make a meaningful impact.
Etsy is a bit more interesting. Only about 20% of visitors to Etsy do a search, compared to about 30-35% for the other three. All the other sites had Direct as one of their top 3 channels - Etsy doesn’t.
Also, none of the other sites has Social in their top 3 channels - Social is Etsy’s second largest channel. And - Etsy’s social channel has a shallow base conversion rate and a pretty high multiplier (~5.5x) when social visitors do a search.
But, what’s Etsy’s top social network?
Yep, Pinterest. But, it’s a terrible converter - a 0.18% conversion rate. It’s worse than any other channel and worse than any other social referrer. Weird right?
The narrative that I’ve always heard is that Pinterest drives the crap out of Etsy sales.
Like the two are made for each other. Pinterest drives a lot of traffic, for sure. But in terms of sales - Facebook and YouTube combine for the same as Pinterest. But, Google Organic drives 10x more sales. It’s by far the largest channel.
The proportionate difference between Etsy’s social channel and Amazon is something to keep in mind, but it certainly doesn’t account for our 2-3x boost in search conversion rate.
Here’s are all the websites broken down by key channels side by side compared to Amazon:
So, Amazon consistently gets a more significant conversion boost than all of its competitors across channels. The only exception to the rule is Etsy’s direct channel - but remember, Etsy’s direct channel is their fifth largest. It’s only about 8% of their traffic.
Maybe it’s Amazon Prime.
Are return visitors to Amazon just super-efficient buyers? I know I am - I literally bought shampoo from inside my shower once. Also if you Google “Amazon Conversion Rate,” there’s a study based on surveying Amazon customers that makes some bold claims about Prime. So, maybe that’s it.
Sure enough - those returning users are way more likely to convert: 1.3% for new users vs. 5.7% for returning. But, for those new users - when they do a search, their conversion rate goes up 8.4x. So, if anything search has more of an impact on new users than returning.
Let’s compare that to Walmart:
Again, returning users have a higher conversion rate than new users. But, Walmart’s ratios are pretty much the same as before: 2-3x improvement when a search happens. Nothing close to the 5-8x that Amazon sees.
So, that’s not it.
Maybe Amazon users are just more sophisticated searchers.
Maybe they use longer, more precise searches. Perhaps this is like Google searchers vs. AOL searchers, y’know? As a power-user of Amazon, and a realistically almost never-user of Walmart - I could get behind that narrative. I use precise searches. If you described me as sophisticated, I wouldn’t correct with you. Maybe that’s it.
As expected conversion rate does increase as we add more words to the search - by about 15% per word. But, the real question is, how does this compare?
And, unfortunately for us sophisticate-hopefuls, Walmart has basically the same exact distribution of search length.
Side note: This slide inspired me in a completely surprising way. I didn’t even make the connection to a previous slide about thirty slides earlier until I was off stage and back in my hotel room. What I realized, I wrote up in an article entitled: Search Autocomplete Boosts Sales by 24%, and I Can Prove it. In it, I give quite a few more examples and detail on the length of searches and how it changes conversion rates.
Here are the custom segments I used to create these reports, btw.
Site searches – 1 word (Custom Segment)
Site search – 2 word (Custom Segment)
Site search – 3 word (Custom Segment)
Site search – 4 word (Custom Segment)
Anyway, this isn’t it.
Amazon has ads. Maybe that’s it.
Amazon has ads, and the other guys don’t. Well, maybe Etsy does. But, others definitely don’t. And perhaps this is a case of the wisdom of the betting market - people betting their money that their product will sell.
Marketers are improving their titles and descriptions and their price to create offers that outperform the default unpaid results. This could be it.
If you look closely at the URLs for Amazon’s ads, they differ slightly from the standard “organic” product results.
They go all go through a tracking URL that begins with “slredirect” - and they also have a “sspa” modifier to an extremely useful query string parameter that I’ll talk more about later. I can use the slredirect to create a custom segment that will let me see all ad clicks.
So, what I did here is I compared people that did a search on Amazon and then clicked on an ad vs. people that clicked on a sponsored ad vs. All users.
Amazon ads work, and they work pretty well. When someone searches and clicks a sponsored result, their conversion rate goes up to 26% which is 6% higher than an organic result. Unfortunately, that only accounts for 2.3% of users, so it doesn’t move the needle for our 2-3x gap.
Note: It might be premature to say that Amazon ads work well because we’d really need to control for the position of the ad. We’d want to compare an ad in position 1 to an organic result in position 1, and so on. I kind of want to go into that right now, because I’m curious, but then I’d never finish this article. Maybe I’ll just write another article about that.
It’s gotta be search personalization
They’re leveraging what they know about me to show me things they know I’ll buy.
Kind of like their “other people that bought that also bought this” - thing, but infused into the search results.
But, first I needed to prove that they were personalizing the results. I hadn’t ever seen it for sure, but I was pretty sure it’d be there. So, I started to experiment.
I used a Chrome browser running in Incognito through a proxy as my control, and my normal browser, logged in to my account as my test. Then, I ran the same search in both windows. I ran lots of different types of searches:
- “Boosted board” to see if it would filter out the product it knew I owned, or if it would prioritize replacement parts. It didn’t.
- “Shampoo” since I’d bought some from the shower recently - maybe it’ll recommend the product I’d recently bought. It didn’t.
- I thought maybe it would know that I’m less price conscious than average, so I searched for several types of products where there’s luxury and cheap stuff to see if it changed the order. “iWatch band,” for example. There was no difference.
Finally, I got a hit! I searched for a board game that I bought from Amazon: “Catan.” I expected that it’d offer me expansion packs before trying to sell me the main game again.
It didn’t do that. But, it DID show me a Game of Thrones extension in position 6 instead of 8. And I’d JUST WATCHED GoT THE NIGHT BEFORE. They must be pulling in my Prime Video data and marrying it with search results!!
But, sigh ... I didn’t watch Game of Thrones on Amazon. I watched it on HBONow on my Roku. This is probably just random. I’m hitting two different servers that are returning insignificantly different Search Engine Result Pages (SERPs).
I don’t think Amazon is doing a meaningful amount of personalized search. I couldn’t detect it. I’ll bet they do in the future. But, as of April 2019, I declare that they are not.
And even if they’re doing it a little, it’s definitely not enough to explain our 2-3x difference.
So, that’s not it.
Maybe their search results are just more relevant
But, how could I test that?
Remember that “ref” parameter with the “sr_1_x” stuff? Well, it turns out that the second number is the position the product was in the SERP. So, the first product would be “sr_1_1”, the second “sr_1_2”, the third “sr_1_3” and so on.
So, I can create a custom segment for each position in the SERP. And I’ll exclude the ads, just so we’re comparing apples to apples.
And with that, I can see the conversion rates for each position on the SERP.
Okay, well, that was unexpected. I definitely expected the number one result to have the highest conversion rate. I thought maybe the problem was that my “sr_1_1” also matched “sr_1_10”, “sr_1_11”, etc. But, that’s not it.
I’m going to have to make a spreadsheet for this, to see how it plays out.
Okay. So… I hypothesized that I could prove that their secret. Maybe they have really good relevancy. And the way that I’d do that is by showing how the number one result is so great. And that their ranking is perfectly aligned with what people want. I mean, this is very interesting, but it doesn’t look so good for my hypothesis.
Let’s compare this to Etsy:
So, it does kind of the same thing. Weird. This is not what I expected.
I guess I’ll plot it all out and compare the curves:
This is some pretty glorious data. I don’t think anyone has ever seen Amazon’s click-through curve outside of Amazon, much fewer conversions by SERP position. Same thing for Etsy.
So, this is really interesting. But. Dang. There’s not really a meaningful difference here.
I mean, it captures searcher behavior. You can see that people on both of these sites are much more likely to click on the top result. But, that they’re definitely not more likely to buy the first result. And if anything the conversion rate across all the positions is actually surprisingly consistent.
It’s interesting to me that both sites have a very similar conversion rate curve. The conversion rates sort of go up as you go down the page at first, but then go down as you get toward the bottom.
It’s also interesting that Amazon’s curve is much steeper. But, I can’t say for sure what exactly that means, or why that is, or if it’s a good thing. What it says to me is that Amazon might be better at getting you to buy what they want you to buy. But, it could also be a signal of search relevance.
But, as compelling as this data is - it doesn’t in any way explain the 2-3x difference we’re looking for.
Which was kind of deflating, tbh. Building a never-seen-before click through curve, and having that be the coup d’ gras of the whole presentation would have been really cool. Plus, I was out of ideas.
I was stumped.
And I was like - I’m going to end this presentation with a like, “and it’s all still a mystery” I mean, damn.
Seriously. I was rationalizing my defeat. I was like, “it’s okay. I mean, if I could just sit down and in 8 or 12 hours figure out exactly what makes the most successful company in the world more successful than everyone else…”
And then I started bargaining: “I mean, it’s too much to ask. If I could do that, I’d be absurdly rich. Too much for anyone to expect of me. The audience will be fine.
Seriously, I’ve figured out some fascinating stuff already. Some good insights. And plus, uncertainty is part of the game.”
It was like the seven stages of grief.
In a sort of lucky coincidence, I happened to have a book called Search Relevance sitting on my desk. I started flipping through it in frustration.
And then it hit me.
THEY’RE ALL THE SAME!
All the results on the SERP are basically the same. Their conversion rates differ a little bit, but not much.
What matters is whether or not the search returns a click-worthy result.
THAT’s the right way to measure relevance. How likely is it that the SERP gives you the thing you’re looking for?
This method is also super convenient because it can be done on any site. I don’t have to rely on the secret “SERP position” parameters to be there.
BTW, I didn’t absolutely need to have them start at Home. But, it eliminates variables and keeps the experiment clean.
I just needed to create two custom segments. One for Home → Search, and one for Home → Search → Click. And then, I’m going to look at the percent of searchers that click, relative to the ones that didn’t (maybe they search again, perhaps they click back, perhaps they leave).
In this timeframe, we’ve got 20.8M searches, and 8.6M that resulted in a click.
So, 8.6 / 20.8 = 42%
42% of Amazon searches result in a click!
But wait. That seems like maybe not very good. More than half of people don’t find what they want on the first search. I can’t tell whether this is good or bad. I need some context.
Fortunately, this is actually probably the easiest thing to benchmark so far.
So, 500k / 3.8M = 13%.
At that moment, I took a short quick breath. I could feel my eyelids widening as I focused more intently on the screen. It felt like I was re-reading a winning lottery ticket.
And then, without control, I made bird noises. Actually...not noises birds make in real life, but like crazy, exaggerated sounds - as if I were mocking the birds. KCAW-kk-CAAaawww-CKAwww! And then, I started to dance...explosive, weird, like interpretive dance. Not good dancing - <I don’t dance> - but definitely innovative.
People started to come out of their offices because I was standing at a table in one of the open spaces where all of our office doors open out. They probably thought I was being attacked by a chicken.
It took me a minute to realize that they were all staring at me.
“THAT’S OUR F%*&ING 2-3X!!!” I screamed, “CC-CKAaaaaw!”
After I stopped flapping my wings, I thought - “Oh, crap. I hope that holds up. Otherwise, that’d be embarrassing.”
Confession: as an entrepreneurial-optimist, I sometimes celebrate prematurely.
So, shuffled back to my laptop and ran the same analysis on Best Buy and Walmart.
IT HOLDS UP ACROSS THE BOARD!
And remember where we started: I noticed that when people search on Amazon, their conversion rate goes up by about 6x. That’s between 1.9 and 2.5x better than Walmart, Etsy, and Best Buy.
Amazon has the same - actually slightly bigger - advantage in search relevance - what percentage of searchers click on a SERP. Or, the other side of the coin: what percent of SERPs deliver a clickable result.
So, these things line up. They tie together. And it makes perfect sense. Amazon’s results are better. They’re giving their customers what they are looking for, the first time 2 - 3.5x more often.
And as a consequence, they’re selling 2-3x more stuff for every search.
So, are you still skeptical that this is causal?
Good. Because I was too.
What if relevance isn’t anything Amazon is doing, but they just have more products?
Right. I think I’d argue (with my skeptical self) that that part of the equation is reflected in the “base” conversion rate. But, I wonder if there’s anything I can do to PROVE whether they’re DOING stuff or not.
Relevance Engineering is a thing
It turns out, there’s a niche specialty in software development called Relevance Engineering. They might call themselves a “relevance engineer,” or “software developer - search relevance,” or be on the “relevance team.” “Relevance” is kind of the keyword.
So, if you search on LinkedIn for the relevance experts at Amazon:
Amazon has 858! And the top results are for managers - which means they have teams (multiple) dedicated to improving relevance.
Guess how many Etsy and Best Buy have? Zero.
Walmart has a decent number of relevance engineers - mostly at Walmart Labs in San Francisco. They actually have 153.
And they seem to be focused on precisely what we’d want them to be focused on. One guy claims responsibility for improving search conversions by 3%. Which is basically the difference between Walmart’s search relevance (16%) and Etsy/BestBuy (12-13%).
It seems to be a fairly recent initiative, though. There appeared to have been a surge in hiring about 20 months ago. And 153, while significant is a long way from Amazon’s 858. But, that might not be a fair comparison - Amazon has a lot more software developers than Walmart.
About 1.5% of Amazon’s development team is involved in relevance. By comparison, Walmart is at 1.3% - so, they’re on the right track, and over time they’ll probably close the relevance gap.
Not that the “gap” matters necessarily. Looking at your competitors isn’t necessarily about “closing the gap.” It’s about finding out what’s possible. The only metrics that should matter to Walmart is improving their own.
And the same is true for Amazon - they’re not going to stop improving search relevance because they are destroying the crap out of everyone else. They’ll keep improving because it moves their needle.
That’s the real lesson here. Amazon didn’t implement a search feature on their website and clap their hands together and move on. They understand that building a search box is the first step, but that they need to continually and methodically improve it.
They can look at their data and see what the most searched keywords that resulted in a click <42% of the time (below their average) are. Or they could look for searches that fall way below average - like 10%.
Then, they can run that search themselves and think about what could make that SERP better - more clickable. Then, they’d test their assumptions and repeat. No matter how good their results get, they’ll always have plenty that falls below the average.
So, it’s a process of continuous improvement. A “better than yesterday” mentality. No doubt, when Amazon first started, they had search results exactly on par with the industry average. But, every day they’ve improved - and built their team, and executed.
So today, they earn an extra >$10 Billion per year because of it, and that flywheel keeps turning. Next year, they’ll probably make $11 or $12 Billion dollars more.
If Walmart keeps at it, they’ll make an extra $37.5 million per month ($450M /yr). And Etsy, Best Buy, and probably every other online store focuses on improving their search results and continuously improves them like Amazon, they could nearly double their total (including non-search) revenue.
Let me know if you’re still skeptical. I’ll do my best to provide more proof.
Want to dig deeper? Check out any website you like - with all the power of GA.