Thursday, 13 August 2009

Look Inside a 1,024 Recipe Multivariate Experiment

Do small changes make a big difference? In the world of landing page
testing, they certainly can. A few weeks ago, we ran one of the largest
multivariate experiments ever: a 1,024 recipe experiment on 100% of our
US-English homepage. Utilising Google Website Optimizer,
we made small changes to three sections on our homepage (see below),
with the goal of increasing the number of people who signed up for an
account. The results were impressive: the new page performed 15.7% better than the original, resulting in thousands more sign-ups and personalised views to the homepage every day.

Below is a screenshot of the homepage before the test and the three sections we focused on:

And here are examples of various fonts, messages, and colors we tested:

Section 1: (The words "Sign Up" were put in all caps)

Section 2: (We placed these bubble messages to the left of the word "Sign Up")

Section 3: (We experimented with these different banners in that area)

Can you guess which combination of elements performed the best?

If you guessed the Original "Sign Up" Font (section 1) + Red Bubble "Sign
Up" (section 2) + White "Don't Just Watch, Participate" box (section
3), you win!

Winning Page:

While we could have hypothesised which elements result in greater conversions
(for example, the colour red is more eye-catching), multivariate testing
reveals and proves the combinatorial impact of different
configurations. Running tests like this also help guide our design
process: instead of relying on our own ideas and intuition, you have a
big part in steering us in the right direction. In fact, we plan on
incorporating many of these elements in future evolutions of our

Curtis Lee, Product Marketing Manager, recently watched "KFC Freestyle Delonte West."

P.S. If you need assistance running tests on your website, there are many Google Website Optimizer authorised consultants who can help. Special thanks to VKI Studios who assisted on ours.


Wedo said...

problem is, these changes are rubbish. But then again u dont care about the people that made youtube big would u?

illage2 said...

Wait .... so we all had to sit though a personal message glitch for this crap?

Again you fixed something that isn't clearly broken I liked the old look better, and you only did tests in the US? Before you implement any system you should have it tested internationally first. Just because the US likes it doesn't mean the rest of the world will, seriously ... Youtube you can do better than this.

GAMES said...

Good work. I have picked up some interesting pointers.

Xylokaine PR said...

Once again, YouTube fails to make any beneficial changes to the site whilst succeeding in fixing something that isn't at all broken. Congratulations, you now have 15.7% more users signing up to a site that is crumbling more and more by the second. A site where big movie companies are allowed to move in on territory they don't need and crush the small-timers - which is all just fine as long as YouTube continues to roll in the sea of cash provided by said movie companies. Absolutely ridiculous. Users made YouTube famous, but now those users are being thrown away in favour of companies shoving wads of cash in YouTube's pocket and telling it what to do. Fuck you, YouTube.

TheHonestTheist said...

I don't want to have to click twice to get to my inbox for comments! Don't make it so the envelope shortcut disappears! That sucks!

TheHonestTheist said...



I don't know where else to mention this, so I will do so here...


You want people to watch videos while in someone's channel? Then let us maximize while in channel! If you don't add this feature, then people will have to leave channel. Which defeats the point of this entire re-design!


illage2 said...

Youtube, instead of spendimg time trying to fill your pockets with money, listen to what the comunity has to say, most of the changes you have made since being brought by google, is for lack of a better word, as disgrace.

Just because Google wants you to optimise thier page doesn't mean you have to, grow a pair and show some backbone and make changes that YOU feel are needed not what google thinks, as it seems that every change and update has had something to do with Google.

Also another thing, is remove the "Featured videos" section as they are all cooperate shite, and I've yet to see any videos from the comunity appear there. You should let the comunuty decide what's good and what's not, not Google.

Also, it would be nice to see some sort of "reaport user" feature as there are many Youtube users that seem to want to post hate comments towards other users(which is in violation of the comunity guidelines), and if you want people to come to this site you have to make sure that people think they are safe on the site, other wise they will go elsewhere.

Next, while the MP4 download is a good idea, you need to make sure it's downloaded at a good resolution as when I downloaded one of my HD videos the resolution of the downloaded video was 320x240 and that is an untter disgrace.

Also when you do big public events try doing it elsewhere for a change, because it seems that every big event "such as the 789 gathering) was in america only and I found this a bit stupid as it means that americans get the spotlight whilst the rest of the world don't, I am aware that your HQ is in america but you should at least consider this option.

I know I have wasted my time here as you will not even bother to read this comment, so I will let the comunity read it and see what they have to say about it.

Rossco said...

Xylokaine and illage2 are right. Making more users sign up means nothing. What ever happened to quality over quantity? I would much rather have an intelligent community of a thousand or so members and a website that had half the features people expect over a drooling mass of millions and a website that seems to be getting worse instead of better by the day.

If you're encouraging people to sign up that are so stupid that they can't find a "Sign up" button without this much help, I'm honestly starting to worry about what this "community" is going to become.

mt_pockets_3 said...

This is all great but there are MORE serious matter in this world like X FACTOR 2009.
I'm not joking you, if you want to win some money, pop down to your local betting shop and place a bet on this guy winning. DANYL JOHNSON. If ya don't believe me then watch his first audition

DiSCo said...
This comment has been removed by the author.
DiSCo said...

Really trustworthy blog. sesli Please keep updating with great posts like this one. sesli sohbet I have booked marked your site and am about to email it

to a few friends of mine that I know would enjoy reading.. sesli chat

DiSCo said...

Really trustworthy blog. sesli Please keep updating with great posts like this one. sesli sohbet I have booked marked your site and am about to email it

to a few friends of mine that I know would enjoy reading.. sesli chat

DiSCo said...

Thank you for sharing a nice article.

Post a Comment