Hey, before we get started, I want to show you that I just got new business cards from PrintPrintPrint.biz. Here’s let’s open them up. There they are. I asked for very plain cards, just my name and contact information. If you need business cards, go to PrintPrintPrint.biz and use the discount code Mixergy. So PrintPrintPrint.biz. They came within a couple of days and they look great.
I also want to tell you about my second sponsor, Grasshopper.com. It’s the virtual phone system that entrepreneurs love. And I say that because by virtual phone system I mean you don’t need heavy equipment, you could just manage the whole thing online. You can get extensions, call forwarding, voicemails to email, you can get toll free or local number, and so many other features. Go to Grasshopper.com.
Finally, you need to know that if you’re an entrepreneur or working at a tech startup, the lawyer that you want to work with is Scott Edward Walker of Walker Corporate Law. He’s in the space, he understands the space, he represents many of the company’s that you know and admire in the space. And if you want to talk to him either about just getting yourself started or what he really does, which is transactional work like raising money or selling your company, he’s the lawyer that you go to. Scott Edward Walker of Walker Corporate law.
Here’s the program.
Andrew: Hi, everyone. My name is Andrew Warner. I’m the founder of mixergy.com, home of the ambitious upstart. How can optimizing what you have turn your whole business around? Joining me is Dan Siroker, founder of Optimizely. As Barack Obama’s director of analytics during his first presidential campaign, Dan helped grow email signups, voter registrations, and donations. And he helped Obama win. Today, through Optimizely, he’s doing the same for companies like yours by making optimization dead easy. Dan, welcome.
Dan: Thanks Andrew. It’s a pleasure to be here.
Andrew: So, can you give me a concrete example of how optimizing a website can increase, how about the bottom line. I know it could increase registrations and all kinds of other numbers, but I care about the bottom line. Do you have an example of that?
Dan: Sure. So at the end of the day, two percent of people who come to your website on average turn into a customer. Either they donate to your website, they add an item to a shopping cart and convert, or they sign up for an email list. On average, that’s two percent. That means 98 percent of the people who come to your website end up leaving either because they couldn’t find what they’re looking for or because they weren’t able to convert across your funnel. And so a concrete example, back in the campaign, we would run several experiments to try and get people further along down the funnel. One of the first experiments we ran, back in December 2007, was on our splash page, which was basically targeted at trying to get as many people who came to our website to sign up for our email list, because once they’re on our email list, we did a great job of raising money from them. Back then, we tried a couple videos, we tried a couple images, and a couple of buttons and through just the simple process of coming up with the most compelling visual and the most compelling call to action, we’re able to improve the effectiveness of the splash page by 40 percent. And if you look at sort of improvement over time, that means a lot of dollars and a lot of additional voters, a lot of additional volunteers, and a lot of people on our email list.
Andrew: OK. You said a lot of dollars. Any sense of how many dollars?
Dan: Yeah. I’ll talk about that specific example. And this was back in December of 2007, before the Iowa caucus. So we were working on trying to figure out what is the ideal initial call to action. We ran an experiment, three images, three videos, and in four different buttons and we looked at sort of the improvement. We said OK I think the original sign up rate was something like 8.6, and we were able to improve that by 40 percent. So you look at that 40 percent increase in the number of people on our email list and just sort of assume throughout the campaign that that would’ve held up, that would mean an additional million or two people on our email list, and because on average for every person on our email list we were able to extract two or three donations from them, that would turn into something like $57 million in raised money through email.
So the fact that we did this early, which is a great lesson to learn, and the fact that this improvement sort of held up throughout time, really turned into quite a few more donations.
Andrew: I see. So that’s $57 million just from that one change, not from all the other changes that you made throughout the site?
Dan: Yeah, yeah. I mean this is the one that we did the most analysis on after the fact to try to see what the impact was. And we did the experiments across the board from our donation page to our home page. But because this was the first page people saw when they came to BarackObama.com, this was a great opportunity for us to get a lot of traffic, figure out the best possible way to convert that traffic into volunteers, into donations, eventually into voters. And that’s, for this one experiment, what we saw.
Andrew: All right, great. So here’s my agenda for this call. I want to know how my audience can use some of what you learned on their businesses. So I want to get as specific as possible with the ideas and with the examples that we’ll be using to illustrate those ideas. And then second, since you and I met because I sent you a fan letter, I saw your new website, Optimizely, and I sent you an email and then you emailed me back and you said I’m a fan of Mixergy, I like your work. I said all right, let’s invite him on. So because I’m such a big fan of your site, I want to make sure that other people understand why I think it’s so great, why I think it’s going to be so useful to them. So no commercial but just heartfelt fan letter here via video Skype. But let’s start off with the information we’re going to teach people. First thing that you told me in our pre-interview is that we need to define quantifiable success metrics. What do you mean by that?
Dan: It’s actually really simple. Basically, it means come up with a number that you’re trying to improve. On the surface, most businesses who we’ve worked with and talked to they know what they’re trying to do. If they’re an e-commerce shop, they’re trying to sell things. If they’re a non-profit, they’re trying to raise money. If they’re a blogger, maybe they’re trying to get people to sign up on their email list. And they have a vague sense of that, but when it comes to try and run an experiment, they start moving all these levers at the same time. They change everything around. They say let’s run an experiment here, let’s do an experiment there, and they don’t really focus in on a specific number they’re trying to improve.
The reason why that’s so important is because without knowing a sort of measurable goal, any change you make could have detrimental effects on other parts of your site that you had no idea about. I can give a concrete example here. We had a quantifiable success metric during the campaign, which was the average dollars per page view of people who came to our home page. How much money did we get from them on average the length of the campaign? And by running a simple experiment, you could say well, okay, let’s improve the click through rate on the donation button, then improve the effectiveness of the donation page for people who come from the home page. And we saw, okay, we can improve that by ten percent, we could get ten percent more people to donate. Everyone’s like oh, this is great.
But by looking at this one number that we were trying to optimize dollars per page view, we also noticed that 50 percent fewer people were signing up for our email list. Holistically looking at that sort of as a big picture, it turned out to be really bad optimization, because in the short term it made a lot more money but in the long term we need a great job of converting people on our email list into donors. If we had a lot fewer people signing up on our email list, in the end we would’ve made less money. So focusing on specific numbers — things like dollars per page view, sign up rate, and any kind of quantifiable metric — you can make sure that when you’re starting to move levers around, you’re actually focusing on your goals. You’re not just arbitrarily changing things in the short term looks good but maybe in the long term hurts you.
Andrew: All right. I’m thinking about the typical company, and their website would probably have multiple goals. They want people to read the blog and find out what interesting information they have to share with the world, because they might share those blog posts with others. They have a product that they’re trying to sell, so they might want to push that. They have an email list that they want people to join. They maybe want them to join the RSS. Are you saying eliminate all those and focus in on one goal and that’s what each page needs to optimize for?
Dan: No, not necessarily. I think it’s important to quantify, that’s the important part. To have multiple goals is okay, because going into a particular experiment you may have multiple things you’re trying to achieve. And only after the experiment’s done will you be able to weigh the trade offs between maybe slightly hurting your sign up rate but dramatically increasing your time on site for example. And I think the important part is to measure that. It’s something that seems really simple and very trivial, but without actually coming up with a number and beforehand knowing what that number is and afterwards knowing what the difference is in that number, you can’t really know what you’re affecting.
Andrew: I see. You’re not saying have one goal for the business or one goal for the site or one goal even for a specific page on the site. You’re saying having one goal for the test that you’re about to perform.
Dan: And be able to measure that. Yeah, and actually I would even argue that one goal is actually sort of this misnomer that a lot of existing tools force you to have. You have a single conversion page. But we’ve noticed over time, as web pages have become more dynamic, there’s a lot of things you can do, having a singular conversion goal is actually really bad because you might run an experiment and you’re actually thinking okay, our real goal here is to get more people to sign up. But if you go to Mixergy.com and you see a banner at the top, your goal for that banner is to get people to sign up with their email list. But it may turn out that that actually has a positive impact on other parts of your business. So at the end of the day, I think it’s really important to understand the holistic impact of the changes you’re making, and oftentimes improving one thing will hurt another. And without knowing what both those numbers are, it’s very hard to optimize your site.
Andrew: Okay, all right. The next point that you made is question your assumptions. What do you mean by that?
Dan: Time and time again, the areas where there are the biggest improvements are areas in which somebody questioned the core assumptions that went into that particular part of the page or that particular part of the website. I’ll give you a really concrete example. I mentioned the splash page that we had at the beginning of the campaign back in 2007 in December. We had a really gung ho volunteer who made this amazing video that brought them to tears. And if you were a supporter, this is great. It’s got mashed up all these great clips from Barack Obama. And we were like this is great. Let’s throw that on the splash page. And everyone at the time fell in love with this video. I mean it was hard not to love this video. But it turned out, after we did an experiment that we compared all of, we tried three images and three videos. It turned out that every single video did worse than every single image. Even the simplest or dumbest image we could come up with the videos did far worse. And you could attribute this after the fact to things like maybe it didn’t load on people’s computers or they just wanted to skip through to the next page. But again, that core assumption would’ve really hurt us if we had gone and said oh yeah, this video, we all love this video. We just want to see this on our site. If we had gone with that assumption initially, we would’ve really hurt ourselves. So questioning that was very beneficial to us. And I think for lots of businesses sort of finding those core assumptions and questioning them can really lead to the big benefits.
Andrew: That seems a little basic actually. If I look at the page, I would assume that I would make no assumptions. I would say I’ve got to change everything, right? Especially with a site like yours, with Optimizely, where I can change buttons and banners and pictures. If anything, I would think that people wouldn’t have any assumptions at all but would start playing around with everything. Is this really a problem for the entrepreneur or businessperson listening to us right now?
Dan: Yeah, I mean that’s a great question. I think one of the things, and this is something that really, Avinash came on your show a month or two ago, mentioned this, the unknown unknowns, the things you don’t know you don’t know. And that’s really where sort of questioning your assumptions comes into play, because in reality the things that you don’t know that you assumed are just sort of core to your business, like that you should have a free trial on your site or that you should have three options for how to donate instead of two or one or even any, those are the core assumptions that you assume going into designing your website are actually the right decisions. And when it comes to optimizing, you type in your URL into our product and the first thing you think about is oh, let’s maybe change this color or change this layout. But it’s those core assumptions that are actually the things where there’s the biggest value.
So, yeah, if you talk about it in the abstract, people will say oh, yeah, sure the pictures would all do well and screw the videos. But at the campaign, everyone was gung ho for the video.
Andrew: Hmm. How do we do that? Because you know what? I really am thinking here that I know that this is important. You’re not the first person to say question your assumptions. In fact, I was just going through Hiten Shah’s transcript recently, and he said that he made huge mistakes early on in his career by starting businesses and building products where he didn’t recognize you made certain assumptions about the industries and boom, it turns out that no, people didn’t want the hosting plan that he was building and that he assumed they all wanted. People didn’t want the podcasting system that he was building and he assumed every podcaster would need. But I’m wondering how do we even know it? How do we even recognize these assumptions that we’re not aware of? How do we give people something that would, that they could use right now, that they could look at their business and change it because it of this understanding.
Dan: Yeah, well, that’s a great question. And I think it’s hard to do that. And one of the biggest places where you have an opportunity is any page where you have a high bounce rate. Those are the pages where, we’re talking not just business assumption, questioning business assumption is sort of a different topic. I’m talking about the assumption of what does the user want when they come to this particular page? And understanding how do we improve this page by sort of taking the true reality of what is it they’re looking for and improving that. So particularly if you look at any landing page and you say okay, what do I actually want people to do here? What is the one thing I want them to click on? And sort of focus in on is that actually what I want them to do? Are people coming to this page and doing that? If you look at Google Analytics or if you use KISSmetrics or any other product, you can see okay, well how many people are coming to this page and not actually taking this goal? What is the action I want them to take? And if you start there and say okay, this is the page that had the highest drop off of all of the pages in my funnel, that’s where I think you can, you’re likely to find some assumption that you made at the very beginning that just may be completely wrong.
An example for Optimizely, very early on we wanted to make sure that people could actually get through the entire process of entering their website URL to trying the entire product without any sign up, the least amount of information required. We found that that works really great [inaudible 14:36] people to sign up and give us their credit card at some point. We need to start experimenting. Initially, a couple of weeks ago, we had this model. You just type in a URL to get started. Crazier experiment, only if you want to save the experiment do you actually ever need to give us an email address, and only if you want to start the experiment do you give us a credit card. Even that core assumption was very flawed, because we had a lot of people who did this, went through all this effort. A thousand people in a day will start an experiment, and when it comes to actually entering their credit card, obviously they don’t want to do it. So that’s one of the hardest things to get somebody to do is pull out their credit card.
So just by questioning that we said okay, let’s push that even later. So now if you come to our website, you can actually do everything on our website, including starting the experiment and only when you actually want to see the results do we actually ask you for the credit card. You know, because from our perspective, we want Optimizely to be a return on investment. We want people to see value that they’re getting out of this. If they’re paying X dollars a month, we want them to be making hundreds of dollars or thousands of dollars more back to them in their business. And further, that’s sort of the thing that we really want to push and by arbitrarily making this, oh yeah, let’s get them the credit card right when they’re most excited in this experiment, I think we were sort of hurting ourselves. And lots of people would drop off.
Andrew: I see. How about in the page itself? I’m not looking at Optimizely.com, and what I see is on the left side of the page I see a video. On the right side, I see a text description of it. Underneath that is a bar where I could enter my website address, hit get started, and then immediately I can start playing around with Optimizely on my website, experimenting with what my site could look like as I was using your tool. How did you change this, or what assumptions did you make about this specific page that you may not have realized but helped you increase conversions on it?
Dan: Well, we still have a long ways to go, and right now on that particular page only 25 percent of people actually enter a website URL to get started, which to me seems abysmally low. Like I hope that at some point we’ll get to the stage where people will come to our website and if they’re interested in it, they’ll just enter their website and get started.
On that particular page, I think we have, one core assumption that we early on thought would be really cool is to put a lot of features, talk about the benefits. And we had this big explanation what is A/B testing. We had a big diagram. But in the end, actually, the most important part here is to let people kick the tires. We just happened to be in a product where people want to just try it out. And by sort of removing all the other explanations and text, which in reality I wouldn’t even really read either, I just want to give it a shot and see if this works. And actually adding really only testimonials is if you look at it the only content on that page are testimonials, we were able to improve the likelihood somebody’s actually going to give us a shot and try the product.
Andrew: I see. So if I’m understanding you right, you’re saying that even though you have a lot to change on the site and a lot to learn about how to improve your conversions, but even with that in mind, one of the early assumptions that you made was that people want to know all the features before they try it out. And what you’re realizing is no, they really want to try it out, and once they get to try it out, everything else falls into place.
Dan: Yeah, yeah. They want to know, they want to know what it costs and then they want to try it out. We found tons of people click the pricing page and then just out of curiosity before they get into it to know if it’s worth their time and investment to check it out.
Andrew: Okay. All right. Do you have anyone that you can talk to, to help you be aware of the assumptions that you’re making or to help slap you in the face and say hey, no you’re absolutely wrong, you need to collect email addresses before donations or anything like that?
Dan: Not really. The best person who slaps me in the face and tells me I’m all wrong is my co-founder. So that’s always helpful to have somebody who can call your crap when he sees it and make you fix the basic stuff. We look at this business as an opportunity to rethink any of the core assumptions anyone else before us has come up with. There’s another product that allows you to do A/B testing that Google offers, which is a free product and that’s something that we used a lot during the campaign. And part of why we’re even doing Optimizely came out of the experience we had doing all these A/B tests and realizing the real core pain points we had.
I think going back to well we always have to ask ourselves, is this really solving the pain points that we had back during the campaign? And if it’s not, then it’s probably a waste of time and we shouldn’t focus on it.
Andrew: Okay. Next point that you and I talked about before the interview was divide and conquer you said.
Dan: Yeah. So divide and conquer is a computer science term, but it basically means to try to understand a little bit about the people who are going through your experiment and then to use that as a way of segmenting your results. And it doesn’t mean you need to run different experiments on different groups of people. It just means that when you’re looking at your results try to understand whether or not one of the winners in your experiment works better for one particular segment or another.
And I’ll give you a really concrete example. During the campaign, we ran an experiment on the home page on the button that’s in the upper right-hand corner that said donate now. And we actually tried a couple of different variations of that button. We tried donate now. We tried why donate? We tried contribute. We tried donate and get a gift and please donate. What we found was that the one size fits all solution, the one that works on average the best was actually a very, very sub-optimal outcome. It really mattered who the person was that came to your website in terms of what button was most effective. And so if you had previously donated, contribute did the best and that did it best by a large margin. If you had signed up for an email list but never donated, then I think it was donate now or something did very, very well. And if you had never donated or never signed up, donate and get a gift did really well. But we would’ve never known that had we not divided sort of our audience into these three different groups.
I think it’s easy to sort of just run an experiment, look what’s the best winner and then just go with that. But without truly understanding a little bit about your audience, I think you can really miss out on a lot of potential gains by dividing and conquering.
Andrew: Does Optimizely allow me to do that, allow me to say paid members get to see this variation and unpaid members get to see this other variation?
Dan: We do in some ways right now. You can run an experiment on specific URLs, and you can define that either as a regular expression or structuring and so it requires you to do a little bit of hacking to come up with sort of unique URLs for these different audiences. We’re working on more advanced targeting and segmentation to be able to let you do sort of arbitrary targeting to say I want to target people who have been here before with a cookie. Or I want to target people who came from Facebook or I want to target people who came from AdWords. And understanding for each of those audiences what’s the most effective way to convert those people into customers.
Andrew: Okay, I see. You seem to be using a lot of examples from the campaign, and I admire what you guys were able to do in the campaign. You brought technology to a world that wasn’t ready to accept it, and you showed that it works and you proved that there are numbers that show that it works. And you proved it with numbers is what I’m trying to say. But I’m seeing a lot of logos on your website, Café Press, Kiva, Shopify, Drop.IO, so many others, Bread Pig. Why aren’t you using some examples from the companies that have you used your site?
Dan: I’m glad to share any examples. One really interesting more recent example in the last week or two was a Facebook app that was made called the Commit to Vote app, which was basically an app to get people to vote. And they used Optimizely to increase their viral coefficient. It was kind of a unique way of looking at website optimization, because it was all one landing page. There wasn’t a conversion page and a test page. All they were trying to do was get more people to share this on Facebook. And so by running sort of a series of A/B tests after A/B tests, they’re able to sort of take a page and say okay, what is the most effective way to get somebody who lands on this page to tell their friends about it? And by tracking the number of times and the number of people they contacted, they measured very clearly okay, the viral coefficient is now 2.2, now it’s 2.3. That’s a pretty cool use of website optimization.
Earlier this year the Clinton Bush Haiti Fund used Optimizely to optimize their donation page. And I think that was . . .
Andrew: Actually, I’m sorry, can you tell me more about how they were able to increase their viral coefficient? What kind of assumptions did they make that they suddenly were able to identify and adjust and test and discover conversion rate? What I’m trying to say is can you give me specifics of what they did differently because they were testing?
Dan: Sure. So they looked at every part of the funnel. So the app was really simple. It was basically a are you on Facebook yes or no. And if you are, you connect through Facebook connect and it imports all the people on your friend list that haven’t commit to vote yet. And they tested things like what’s the most compelling message that they could suggest to a user to share with somebody else to get them to actually commit to vote. And by being able to track that, to say okay, is it “I’m committing to vote, will you too?” Or try to get somebody to say yeah, I’ve gotten the most people to commit to vote of my friendship circle, can you do that too? By focusing on either of those goals, they can say what’s the most effective messaging that we can sort of encourage our users to use to try to convince their friends? And so they looked at every step of the funnel. First was are you on Facebook yes or no? The second step was okay, now you commit to vote if you haven’t already. And once you’ve committed to vote, try to get more people to commit to vote. So they looked at things like how many people should you recommend to initially message to? Should you say the first five people, the first ten people? I think it was like over maybe a week or two they ran like 52 experiments one right after the other, and because they were able to get real time results and because this was going viral, it was a great sort of virtuous cycle because as more people used the app, they were statistically getting results sooner and they could run another experiment.
Andrew: I see. Okay. The other question that I had as you were giving some of the examples is how many people, how much traffic does a website need to have in order for these tests to be useful?
Dan: So, in general, you need around 100 conversions per variation. And that’s a very rough number. And the two biggest factors on determining this is how much traffic you have obviously, and how different the variations are that you’re testing. One of the things I really encourage people to start off with is a pretty big change. Try to make a big change on your website. Don’t just make sort of a subtle change like one word now has an exclamation mark at the end or not. Try to make a pretty big change, because what you’ll find is the bigger changes are going to get statistical significance sooner. Either it helps or it hurts. But by making a really big change, you can learn whether or not it actually had an impact.
Roughly you need about 100 conversions. So that means if you have a 10 percent conversion rate, you need about 1,000 visitors per variation per period of time.
Andrew: I see. Okay. All right. Next point that you had was avoid local maximums.
Dan: Yeah. So avoiding local maximums, this is a debate people have been having online, on the Internet for a long time. It basically boils down to this idea that A/B testing and website optimization are all about sort of eking out that extra couple percentage points on your website. And I think that’s kind of not really true. You actually use, you can use website optimization to test really dramatic differences in your website. I think there’s a temptation to try to take your website and sort of start where it is today and sort of move in the right direction from where we are today. But the reality is you might hit a local maximum. Like maybe you’ll take your website now, improve it by five percent or six percent and be like this is great. But the truth is you can take a completely different website, remove everything from your page. Like for example on Optimizely, we can remove all the testimonials, remove the video, just have one big bar right in the middle. And taking a dramatic difference and sort of understanding whether or not that has an improvement is actually more valuable than sort of eking out those little small things, because you might find out there’s this big hill over down the road that we can get way, way higher conversion rates by focusing on these sort of big holistic changes.
Andrew: I see. Yeah, you know what, I can see that. That if I had a landing page that I wanted to improve conversion rates on, I might change the text of the button, I might change the placement of the registration form, I might add a video or replace the video with a picture, but I wouldn’t think start fresh and have a brand new site or brand new landing page to compete with the one that I’m tweaking. That’s what you’re saying?
Dan: Yeah, exactly. And it’s also a question of going back to Avinash’s point of the unknown unknowns, that you have a landing page that exists today and the truth is it’s probably not nearly as good as a completely different page. But you have this temptation to say okay, well let’s just try a small tweak and a small tweak there. Reality is there’s this huge space of potential pages you could be testing, and the chances are a completely different page will be a great place to start. And then once you’re there, once you have tried three or four just completely radical different pages, then it makes sense to go in and try to optimize locally around that. But starting locally and then optimizing that and then later on doing the big test is a waste of time.
Andrew: You know I got some complaining emails about the headline that I gave his interview, Avinash I think used the phrase “minor orgasm of analytics.” And so that’s what I used in the headline to get people’s attention. It pissed some people off, but it is true when you look at him, you can see that he’s having this real visceral, excited feeling for response to analytics. And I see something similar in you. What is it about analytics? It’s numbers. What is it about those numbers that gets you so freaking excited?
Dan: It’s a good question. So I guess I, and thinking back now to the campaign and I used to work at Google as a product manager, and I remember coming to the campaign and being shocked at how many decisions were made by what Avinash and all those other people call the HIPPO syndrome — the highest paid person’s opinion. You’re in a room with whoever’s getting paid the most and the guy that just says what we’re going to do and it goes from there. It was such a gratifying feeling to come into those meetings in this political organization where I’m just like this Google guy that knows technology but nothing about politics and to come in with data to be armed with data and to be able to just completely depants somebody and be like, “Okay, no actually your opinion is interesting and that might work, but the data shows that this is the best answer.” I think it’s a very empowering feeling to be able to have data to back up your decisions. Time and time again we found that if you go with some of your gut instincts, if you go sort of Don Draper in the room, you know who has this really compelling idea, the video on the splash page is a perfect example. Everyone felt like this was a great solution, but having data really just splits the opinions from the facts, and you can use entirely facts to make better decisions.
Andrew: Have you seen an example of a company that was made or broken because of use of optimization like this? Or are we just talking about the tool that gives you an extra few percentage points in growth in your business?
Dan: I don’t know of one in particular that was either made or broken. I know that, especially this Facebook app that I talked about, whether it would’ve gone viral or not without Optimizely, I’m not sure. But I know that had they never done any optimization, they would’ve had a viral coefficient that for every one person that used the product maybe 1.1 would grow from there. And I guarantee you that if they hadn’t done this optimization, they wouldn’t have ramped up to where they ended up because they just didn’t have the time. I mean the election was going to happen whether they wanted it to or not, and their success was very much predicated on being able to maximize the amount of people that would tell other people about this app in the amount of time they had. So, yeah, I mean in that particular case, it’s very possible if they weren’t as viral as they ended up being, then they would’ve half as many people use the app. In the end, I think they had over a million active people using the app or more.
Andrew: Okay. Final point that you made before we started was start today.
Dan: Yeah. This is a point that is really predicated on the fact that there is a slew of free tools, there’s pay tools. The tools aren’t the problem. I think the problem is people feel like this is too time consuming or too painful or too confusing. I sort of want to eliminate that notion that the idea of taking your website and running an experiment on it actually is not as hard as it may sound. And so I think people will disagree with me about whether to start today with something trivial. I think just on the KISSmetrics blog somebody said don’t just do any little test. But I would say just start anything, like start a little. Once you’ve tasted a little bit of this optimization glory, you can’t stop. And I would encourage you to try, if you want a free tool, Google Website Optimizer will work. If you want, we try to make it very, very easy. You just enter your website with no commitment to try out optimization. You can go to Optimizely.com right now and try it. I’m convinced if you try it and use it, you’ll see that actually there’s this idea of having to talk to IT and get them bought in, talk to engineering, it’s really not that difficult. And so I want to emphasize that you really should start today. It’s not that hard.
Andrew: All right. Let’s see if I can understand where people could start. Because if we just say start today, I don’t think we’re giving them enough information that they could use. Let’s see where they might be in the audience and how they might be able to start optimizing.
Dan: Sure.
Andrew: Let’s suppose somebody has an idea, hasn’t launched a single damn thing yet, they have a domain and nothing else. They could put up a WordPress blog. Do they even need to do that?
Dan: So okay, that’s a harder audience because you need traffic in order to do optimization.
Andrew: So you’re not even in business yet and it’s true, you do need traffic, but we’ll come to that in a moment. Let’s say they just want to test an idea like Christian Owens, comes to my site, he says that he’s got something called Mac something bundle, shit, I can’t even think of the name. But it’s basically bundling Mac software, and he’s going to sell it at a deep discount. You’ve seen businesses like this before. He said the first thing that he did was he put up a form on his website asking people to submit their email addresses if they were interested in his business when he launched it. And the way he got traffic to the site was he started promoting it to blogs that cared about Mac software. So he gets the traffic to the site. Now he might want to start experimenting with either emphasizing the discount or making the text on the button different or emphasizing different product elements that he plans to have on there. At that point if he wants to do a test before he’s even launched the site, when he knows he just wants to experiment, he wants to just see what features get people interested, how does he start optimizing? How does he start?
Dan: So a great place to start would be to create a WordPress blog or create a Weebly page and put the content that your best guess is what people want. And if you take that page and you have sort of a starting point, the hypothesis, these are the features I think people want. And I’d even just list it out, but say these are exactly what’s going to be provided if you sign up and if you’re interested enter your email address. That’s great because you have a clear call to action, like enter your email address, you can measure that. Then if you improve, you’ll know whether you’re improving the effectiveness of your landing page. If you then go and you can type in that URL into the home page on Optimizely.com or you can go to Google Website Optimizer and say start experiment. When you go to the experiment editor and start clicking on the parts of the page you want to change, it’s a great opportunity to try different features. You can say okay, you can bundle these ten different products together or the first month’s free or whatever core features that you want to test, it gives you a great opportunity just to completely try different text. And that’s just a question it’s just sort of like PowerPoint, you just point, click, change the text you want, create four, five variations, hit save. Then when you run the experiment, we just split traffic to the different people that come to that site. Because you have that really clear conversion goal getting people to sign up on your email list, you can then actually measure what are the features people are most excited about.
And if you’re not getting a ton of traffic, I’d recommend doing very simple like three or four features you really, you think are great and then two or three variations of those features that may be some crazy feature you never even thought people would want, throw it up there and see does that impact conversion rate.
Andrew: Okay. All right. So that’s somebody who doesn’t have a business yet and is about to launch something. What about someone who does have a business, does have traffic coming in and has an idea for a new product that they want to launch to their existing customers? How would they experiment? What could they do next?
Dan: So for people who have an existing business and they’re exploring a new product, a great place to test is upsells. Try to figure out where, look at your funnel and say where do we have enough traffic where we’re going to be able to get [inaudible 35:54] and then try different upsells. Just try simple sentences that say like you’re interested in this bug tracker, well you can also get integration with your support system today. And then have a link and that link can just go to coming soon, enter your email address. But you can actually just track the number of people that click that link. That’s one of the simplest experiments you can do is just add a piece of text to your existing website or your existing product, change the copy a couple different ways, maybe have an image or two and try those different images, and then measure the clicks on that particular link.
Andrew: Okay. Who else might want to use this? How about somebody who’s trying to collect email addresses? Got a blogger, he’s linking over to a page where he’s hopefully going to collect email addresses for his newsletter, that page, that landing page, that’s something that they could start experimenting with.
Dan: Yeah. And we did actually a lot of experimentation on email signups and on email page landing page optimization. On email signups, that’s again it’s something that we constantly were surprised with the results that we found, that the people, somebody’s willingness to enter their email address is very much dependent on their comfort with the page. In that experiment I talked about earlier, the page that ended up doing really well on the Barackobama.com splash page was an image of Barack Obama and his family, a very informal picture of his daughters on his lap and a button that said learn more. And neither of those two things anybody thought would’ve worked really well. But because getting somebody to enter their email address is sort of this you need to sort of take it up a level of comfort or of familiarity with the product, try those kind of things. Learn more is a great call to action for signing up an email, because nobody likes to sign up and nobody likes to be on a mailing list. They do want to learn more, everyone wants to learn more. And so trying experiments like that, trying to figure out and even just going to a couple of people, showing them the different landing pages and try to get people comfortable with that idea I think is a great place to start.
Andrew: Okay. All right. How about finally somebody listening to us who’s working at a company like Zappos? They don’t have control of the whole site, they just want to show that they could optimize one product and maybe convince somebody in the company that they should take those ideas to the rest of the business. Can they take one product and optimize just the way that that’s sold?
Dan: Yes. Yes. You can definitely take one product. The thing I would love, we’re working with a couple of partners to be able to provide this in a very easy way, but I think that being able to track the value of your experiment is really important, because it’s very easy to, like again I said earlier, you need to know the holistic impact of what you’re trying to do. It’s very easy to over optimize for add to cart or over optimize for even go look at my cart or even click on an image of the product. But at the end of the day, if you’re an e-commerce business, what you care about is value. How much money did I actually get? So one of the things, you can do this today we have Google Analytics Integration where you run an experiment on a product page, you’re either trying different copy, different calls to action, or different discounts, you can actually go and see, if you have e-commerce tracking set up in Google Analytics, you can see the difference in dollars per page view. And in some ways you’re really lucky. If you have a business where you’re making money from people landing on your page, you can actually measure the average amount of money I’m going to get per eyeball on this page. And that’s a great thing to optimize, because the amount of traffic you’re getting you can’t really change. You can pay for more. But getting more, more money for every eyeball on your page is a great metric to try to optimize.
Andrew: All right. One of the things that I’m excited about with your site as opposed to Website Optimizer, Google’s product, is that Google’s products tend to be just too confusing. Even Google Analytics has so much freaking stuff in it that I’ll interview people who will be amazed at one feature and I want to go and play with it and I’ll realize it’s going to take me forever to just figure out how to use that feature. You, on your site, I’ll do a little bit of fan boy talk about this. Here’s what I love about it. I entered my name in, my website, I hit get started and then I see my site exactly as it, my page exactly as it looks on my site. Then I want to just play around just to get a feel for it, for the tool. I can highlight text, I can bold that text, I can move a picture around, I can change the button, so easy to adjust. Then I could hit save and save it as a variation and then create another variation. One with maybe with no bold text and no buttons at all and save that and then do another variation. It’s all WSYIWYG, what I see is what I get and it’s so easy to use. And then I think do you give me a little bit of code that I put on my website?
Dan: Correct. Yep.
Andrew: Dead simple.
Dan: Yeah, yeah, yeah. I mean we, the original spark for what we’re working on today came from frustrations we had with other products. And I think it’s very easy to build a product in sort of a vacuum and be like oh, yeah, this product, the goal for this product is to run an experiment on your website. And if that’s what you do, if you go into it that the goal for this product is running an experiment on your website, you design and you build a completely different product than if your goal is to say I want to make it really easy for an online marketer to take their website and change something on it and then run an experiment. And if you have this idea, if you have an online marketer in mind and I want to make it so that they’re not technical, they can easily just point and click, they can treat their website basically like a PowerPoint presentation, they can drag things around. Then I think, using that mental model, you can come up with a completely different product. At the end of the day, the technical implementation of sort of how the experiment runs is very similar to what a lot of other products do. But because you can think of it in terms as an online marketer or as somebody who is struggling with IT or engineering to help them improve the effectiveness of their website and they want to be free of those shackles and they want to just do it on their own without actually having to sign up or without having to get the IT, the CTO approval, I think the end product comes out very differently. And I hope that’s what we’ve been able to do.
Andrew: Why didn’t you offer a free version?
Dan: We thought about it. You know we thought about offering a free version, and I think we decided to price the product the way we did based off the feedback we got from people we talked to in our early beta. It looked to us like people saw the value in A/B testing. People knew that by doing an A/B test on their website that they would make more money. And what we really needed to do was make it clear to them that by paying us $19 a month or $79 dollars a month that they’re going to get a lot more value back on the other end. And so we thought if it’s an ROI argument, if it’s a return on investment and they know that what they’re paying us is just a small percentage of what we’re giving to them, that we’ll be successful. And we want to actually, we want to work with those kind of companies, because we want to make that true as well. We offer a money back guarantee if you’re not satisfied for any reason, because we want to make sure that people find value in our product without finding it as like a tax on their business. So we decided not to offer a free plan because we wanted to focus in on those customers where that ROI argument was really, really true and understanding what we need to do to make the product better to fulfill that promise.
Andrew: Okay. Earlier you were going to give me an example from the Clinton Bush Haiti Fund. I wrote a note down here since I interrupted you when you started bringing that up I wrote a note to come back to it. What was the example you were going to give us there?
Dan: So this is a cool example because it shows how in just a couple of days if you have enough traffic you can really make a big difference on your page. Back in January, after the earthquake in Haiti, they ran a couple of experiments using Optimizely, and they did really simple things. They did things like they added an image of relief workers and victims to the page. They changed the submit button from saying submit to saying support Haiti at the bottom of the page. They slightly increased the font size. The cool thing was all of these changes were very, very difficult to do with the CMS they had. We found over and over again businesses and especially non-profits are really in painful situations with their CMS. CMS sort of restricts the things they can do. It kind of treats them like they’re not allowed to make changes. And by just pointing, clicking, making these changes you can very, very simply improve the effectiveness of their site. In this particular case, they ran four experiments over a couple of days, and the net result, the net incremental benefit was ten percent. And since running the experiments, they raised $10 million, which ten percent increase is about a $1 million additional raised because of the optimizations they’re able to do. Even simple things like it shocked me how well just changing submit from the bottom of the page to saying support Haiti had an impact.
Andrew: Someone did a post on that? Who’s the woman who runs a website where she shows two variations and she asks you to guess which ones going to be the right one? Do you know that website?
Dan: WhichTestWon.com.
Andrew: WhichTestWon.com, and so she showed two and she said obviously the one that had the call to action at the top right is going to do better than the one that had the call to action on the bottom. Are there other lessons like that that are pretty consistent, like having that call to action on the upper right or what else? The text that you gave for the button learn more instead of saying sign up for my newsletter. What else are you seeing that’s consistent?
Dan: I’m wary of answering that question, because what we found more often than not are the things that work for one business very rarely work for another business. And so I hate to give you sort of false truisms of like you should always say learn more or you should always have a red button, because I think it focuses people on this mentality of creating a recipe for their website and following a bunch of best practices. I think the things that you can say that are sort of true across all websites are things like that if you want somebody to click on one particular link, reduce all the other options they have and make that the one focus of that page. If you want somebody to fill out a form, have fewer form fields so that they finish the form sooner. If you want somebody to actually like, if there’s a call to action that you want somebody to actually click on, make sure it showed up on the top of part of the page. So those are the kind of truisms I think are generally applicable. But those are the kind of truisms I think everybody knows. I’m hesitant to say that there’s a sort of a secret magic, that there’s two or three things that you should just do for sure on your website, because more often than not . . . well, first of all, it’s very easy to test that. You can test any of these things. If you hear on a blog post somebody says, oh, green buttons always work, green’s great. Just test it. Take your website, enter it into Optimizer.com at our home page, change the button to green, and you can see does that actually improve your conversion rate. It’s a lot better to do that than sort of take these recipes and try to apply it to a website.
Andrew: Here’s why recipes help though. The problem that we often find with a blank slate or even a slate that we’re familiar with because we put all the information on it is we don’t know where to get started. Even going back to the Hiten Shah interview that I referenced earlier, he said that he wanted to create a survey site, a site that enabled anyone to create surveys. And he said one of the problems that he realized was that people don’t know what to even ask their customers. So he was going to create a series of surveys that were predefined, prewritten and you could pick one out based on your situation. Of course, you could then customize it. A lot of what we’re talking about here, people don’t know where to get started. We don’t know should we . . . we see our page every day. We don’t know where to get started. So are there any best practices, any templates, any samples out there that we can experiment, that we can at least test and then we could start tweaking?
Dan: Yeah. I totally get that. I mean one of the biggest questions we get is, “What should I be testing? What is the first thing I need to test?”
Andrew: And I don’t think somebody who asks that is just looking for you to walk them through their business. They’re just saying I got a problem. I understand what you’re saying. I just don’t know where to get started.
Dan: Yeah. Well, the first simplest thing I’ll say focus on the one thing you want somebody to do on that page. Once you’ve identified that one thing, remove almost everything else. Try a variation of your page, which is focused entirely on that one thing you want them to do, on our home page it would be enter your URL, click get started. On your page, it might be entering, clicking the banner at the top of the page and getting people to sign up. Whatever that one thing that you really want them to do, focus in on that and remove almost everything else, remove especially copy, remove images, remove text and start from that. Start with a very dramatically sparse page, and that’s a great starting point because what that’ll show you is the impact of all the stuff you think helps, all of these great screen shots or links to show that you were on the New York Times and all that stuff, maybe that’s actually distracting people from what they should really be focusing on, which is the one call to action you have on your page. I would say that’s my best recommendation for a good place to get started. And once you’ve done that, you’ve traversed this local maximum problem, you’re now actually at a point where that’s a great part to say, “Now that we have the sparse with the core beginning essentials of what are needed on our landing page, now let’s start adding things to it and see what actually will improve the conversion rate.”
Andrew: All right. That’s great advice. Thank you. Thanks for doing the interview. So if people want to check out the website, Optimizely.com, Thanks.
Dan: Thanks, Andrew, appreciate your time.
Andrew: You bet.
This transcript brought to you by www.SpeechPad.com.
