Recently I sat in on a presentation by Ronny Kohavi, GM for Microsoft’s Experimentation Platform. The talk was called Top Seven Testing Pitfalls. Among other things, it gave me some good ammo to share with you on the subject of why constant testing is so critical.
As a lifelong marketer, I am thrilled – thrilled – that we can now track, test and measure visitor movement through a website and use that data to improve how well a site turns visitors into paying customers or sales-ready leads. And we get to do this without even once having to sit through a 3-hour meeting to discuss the color of the logo or where it should go on the webpage.
But then I’ve spent most of my career working with engineers, scientists and other logical thinkers for whom statistically significant data ranks right up there with oxygen. The need to quantify decisions rather than going with mere opinions has rubbed off on me. I also like being able to prove that our work produces good results.
I admit, though, when faced with the opinions of a room full of “experts” – especially if you’re one of them – it’s tempting to make decisions based on intuition, instinct or experience. That’s okay as long as it’s just a starting point. If there’s one thing I’ve learned in my decades of marketing it’s that you just can’t be sure what people will do. No matter how positive you are you are right – people will regularly prove you wrong with their actions.
How often are we wrong?
Ronny Kohavi had some very good data (yes, data) on that subject from some of the world’s great testing companies. These are companies that test EVERYTHING and each test clears a number of hurdles including web designers, usability experts, copywriters, marketing geniuses and executives before it is run. Everyone believes the challenger page in the experiment has a good chance to deliver better results than the existing page.
Amazon finds that half of the experiments they try fail to show any statistically significant improvement.
Microsoft finds that about one third of their experiments have a positive effect while one third have negative effects and one third have no significant effect at all.
Why test if most tests won’t give you positive results?
Because you don’t know where the positive results lie until you test! If the experts gathered by the likes of Amazon and Microsoft guess wrong most of the time, far be it from me to be able to tell what works by just looking at it.
And I prove that to be true over and over again with our own experiments.
Keep testing.
[tags] testing, A/B testing, website, conversion optimization [/tags]
Recently I sat in on a presentation by Ronny Kohavi, GM for Microsoft’s Experimentation Platform. The talk was called Top Seven Testing Pitfalls. Among other things, it gave me some good ammo to share with you on the subject of why constant testing is so critical.
As a lifelong marketer, I am thrilled – thrilled – that we can now track, test and measure visitor movement through a website and use that data to improve how well a site turns visitors into paying customers or sales-ready leads. And we get to do this without even once having to sit through a 3-hour meeting to discuss the color of the logo or where it should go on the webpage.
But then I’ve spent most of my career working with engineers, scientists and other logical thinkers for whom statistically significant data ranks right up there with oxygen. The need to quantify decisions rather than going with mere opinions has rubbed off on me. I also like being able to prove that our work produces good results.
I admit, though, when faced with the opinions of a room full of “experts” – especially if you’re one of them – it’s tempting to make decisions based on intuition, instinct or experience. That’s okay as long as it’s just a starting point. If there’s one thing I’ve learned in my decades of marketing it’s that you just can’t be sure what people will do. No matter how positive you are you are right – people will regularly prove you wrong with their actions.
How often are we wrong?
Ronny Kohavi had some very good data (yes, data) on that subject from some of the world’s great testing companies. These are companies that test EVERYTHING and each test clears a number of hurdles including web designers, usability experts, copywriters, marketing geniuses and executives before it is run. Everyone believes the challenger page in the experiment has a good chance to deliver better results than the existing page.
Amazon finds that half of the experiments they try fail to show any statistically significant improvement.
Microsoft finds that about one third of their experiments have a positive effect while one third have negative effects and one third have no significant effect at all.
Why test if most tests won’t give you positive results?
Because you don’t know where the positive results lie until you test! If the experts gathered by the likes of Amazon and Microsoft guess wrong most of the time, far be it from me to be able to tell what works by just looking at it.
And I prove that to be true over and over again with our own experiments.
Keep testing.
December 24th, 2009 at 10:59 am
Hi, what blog platform is this? Can I download it for free or..? I would really love it if you could answer this question! Thanks in advance!