Web 2.0 Podcast: Disruption Opportunity - Harnessing the Collective Intelligence

by Daniel H. Steinberg

Even before he coined the term 'Web 2.0', Web 2.0 Summit program chair Tim O'Reilly was considering the notion of harnessing the collective intelligence of the web. In this podcast he moderates a panel that explores the idea that everything is connected. Panelists include craigslist CEO Jim Buckmaster, former chairman of MySpace Richard Rosenblatt, Toni Schnieder CEO of Automattic, the makers of the WordPress, and FaceBook CEO Owen Van Natta.

You can download the audio as an mp3 or download the video as an mp4, or you can subscribe to the audio podcast or to the video podcast. Check out the entire set of Web 2.0 Summit podcasts.

Intel Software Network Intel Software Partner Program

This episode is sponsored by the Intel Software Network.

Transcript created by Casting Words.

Announcer: Before coining the term "Web 2.0", Web 2.0 program chair Tim O'Reilly was considering the notion of harnessing the collective intelligence of the web.

In this podcast he moderates a panel that explores the idea that everything is connected.
Analysts include: Craigslist CEO Jim Buckmaster; Demand Media CEO and former chairman of MySpace, Richard Rosenblat; Tony Schneider,CEO of Automatic -- the makers of Wordpress -- and Facebook CEO, Owen Van Natta.

Here is the panel on harnessing collective intelligence, with Web 2.0 Summit program chair Tim O'Reilly.

Tim O'Reilly: The fundamental idea that came to me that led to this whole discussion of Web 2.0 was really the idea that as we move to the network as a platform, one of the things that distinguishes the applications that we can build is that they are able to harness network effects in a way that you can't do with stand-alone apps.

And this is why when I talk about Web 2.0, one of the most important principles that I try to get people to focus is the idea of harnessing collective intelligence, and to really get people to think broadly about all of the ways that that works.

We are just at the beginning of a world in which everything is going to be connected. And Web 2.0, as we are currently exploring it and excited about it, is the sort of froth at the beginning of an enormous wave. And a lot of it thinking about all the data that is collected, all the data that is connected. And starting to think about, how do we harness this data in ways that are going to make our applications more powerful, in ways that are going to make our applications learn from their users so that they get better the more that people use them? That is a fundamental idea in Web 2.0.

We put together, I think, an interesting panel. I showed you earlier -- yesterday or the day before -- the slide that Greg showed, which actually originally was done by Jim Buckmaster, who is the CEO of Craigslist, one of our panelists. Showing how they are one of the top sites on the net with only twenty-odd employees.

By the way, if you guys are ready to come out, you can come out at any point and I'll introduce you as you come out. But we've also got Richard Rosenblat, who was the chairman Myspace, which had something to do with at least one aspect of this harnessing collective intelligence idea, namely getting a lot of user-generated content.

This is Owen Van Natta who was formerly of and is now the CEO of Facebook, which I think is also in that user-generated content space.

So basically the lineup -- in case you don't recognize the people here -- we've got Owen, Richard to Owen's left and Tony Schneider, who was at Yahoo in charge of Yahoo developer services, now the CEO of Automatic, the creator of the Wordpress blogging software and and Akismet the sort of blog, trackback, spam fighting plug-in. And of course Jim Buckmaster, the CEO of Craigslist.

So let me just sort of start actually with Jim. Because, for me, Craigslist is one of those sites that just sort of slaps you upside the head and makes you say "Whoah. How did they do that?" I mean how do you manage to do so much with only twenty employees? What are some of the ways that you put the users to work for you?
Jim Buckmaster: Well, it's just about every way we have been able to think of, or rather, every way our users have been able to think of that they can do the heavy lifting. Whether it's our flagging system, where users take responsibility to police the site and keep bad listings of, or right down to our business model, kind of from the get go as to whether we should raise financing and take a kind of venture route.

Or whether we should have a sales force, whether we should do marketing, whether we should for anything and if so how much we should charge. So really, our users have kept us out of a lot of trouble. The site is sort of the net result of millions of user's suggestions. We don't have a logo for instance.
Tim: But how do you even pay attention to millions of user's suggestions with 23 employees?
Jim: Well, at this point, we always try and get as much out of the way as possible. At this point, the way we do it is we have feedback forums where users come in and discuss amongst themselves the existing problems of the site and how we should be doing things differently, and we just kind of watch the conversation. You have to have a thick skin sometimes, because it can be kind of embarrassing actually. But that's our current approach.
Tim: The other thing that somewhat struck me, you sent some an email when we were talking this about how you actually use user feedback in your business decisions. It's not just that they post content on the site, but they also give you feedback. Can you say a little bit more about that? I mean, how do you instrument things so that you're...?
Jim: I guess one example is we're approached fairly often about running text ads on the site. It's been pointed out to us that we could be making many tens of millions of dollars each year from just running relatively unobtrusive text ads on the margins of pages. But so far our users are not asking us to run those, so it really hasn't become something we would consider.

Tim: So you have some unique characteristics, in that you're not venture-funded and you're not actually out there trying to maximize your profitability. What's the roots of that?
Jim: Well, certainly the roots would certainly be Craig. Making money and even having a business was never really part of his thinking, he just kind of as a hobby put together an email list which friends told friends about. And originally, it was just events in San Francisco, but people kept contributing all different kinds of things and converted it to a website.

And to this day he really doesn't have any interest in Craigslist as a business and all. Of course there are a lot business elements that you have to pay attention to whether you want to or not, and we do pay attention to those. But that is certainly the roots of it.
Tim: So, jumping over to this end. Richard, you obviously did some pretty interesting things at MySpace. But your new venture, at least I would imagine, is from your perspective even more interesting. So tell us a little bit about what you are doing as CEO of Demand Media and how that plays in this whole user generated content space?
Richard Rosenblat: Great. Can you hear me? OK. So what we learned from MySpace was that... It was only two years ago when I remember talking to people and saying "User generated content could be a tremendous opportunity because the users produce the content, the users invite their friends. And you can build an entire media company with very low cost."

And that was the whole concept. But really even up to eighteen months ago nobody really believed that that would occur. What has occurred though is that the phenomenon of a MySpace or a Facebook is that these companies are doing great but they appeal to a certain demographic. They appeal to eighteen or college type of student. But if you see the demographic, it is actually growing. You are seeing an enormous amount of growth in the 35+, and you are also seeing that the entire web universe wants to be creative. They want to actually get involved in the web.

What we are doing with Demand Media is we are focused on enthusiastic verticals. So, we think that there is a lot of opportunity for social networking, and instead of making it one big place, is to have it actually focused on a very targeted area. Just by way of example, we have acquired a company called We will be adding in the next 30 days a full suite of user publishing tools and social networking tools that will let people that are enthusiastic about hiking and outdoors sports actually meet others to share ideas about it. It is much less about sharing pictures and much more about sharing knowledge. So our entire platform allows us to -- whether we take millions of domains, or whether we take thousands of vertical websites -- allow us to add these kinds of social networking.
Tim: So, your point is Demand Media is actually a platform for bringing MySpace-like functionality into various kinds of vertical sites.
Jim: It is. What we really think is, it is a bottoms-up media company. We let the users determine what verticals they are enthusiastic about and then they coalesce around those verticals and that content moves to the top and that becomes the programming. It is whatever the users decide.
Tim: So, how do you actually do that?
Jim: The way we actually do it is through a suite of tools. We have three systems that we "snap" into things. So our system allows users to automatically publish. We just launched a site called weHow, which allows anybody that wants, to become an expert on anything. So you can publish. So you can publish an article on how to make a great margarita, or you can publish an article on how to fix a leaky pipe.
Tim: Something like what Instructibles is doing. They were in the Launchpad the other day.
Jim: Yes. What we did was we started with a sister site called eHow, which we acquired, which has six million unique visitors per month, 25 thousand professional articles. So publishers can now share knowledge. It is the whole concept of, "Hey, you are an expert on something. Why don't you write an article about it? Get listed in weHow which is part of eHow and let users talk about it." We will be launching in two weeks the ability to actually get paid. So you would actually be publishing and get paid for producing this type of content.
Tim: So, you are actually also kind of an illustration of something came up in the panel with Ram and Roger earlier about VC. You have actually raised a whole lot of money. There is a lot of talk going around Web 2.0, that it doesn't take much money to start something, but you have raised $220 million.
Jim: Well we thought, either go big, or don't go at all. That was our theory.
Tim: So, it is really the roll-out that is costing you the money, rather than the technology platform per se.
Jim: We actually like to call it a roll-on. What we did was we formed a company in May. We raised $220 million in two different rounds and we bought nine companies. The idea was that we rolled these companies on to this platform. Everything that we bought either was strategic in that it filled up this new media platform or it was a vertical, in an area that we think of as enthusiastic categories, where users can build their content across.
Tim: So, it is kind of a mix of bottom-up and strategic top-down in terms of what you are targeting.
Jim: The content sites we buy, like Trails. The way in which Trails works is there is 50,000 professional hiking trails. That's what we consider professional content. So, that is kind of the bottoms-down. What we then add is all the user-generated tools that let the user decide how they want to use that content. Do they want to break into kayaking trails, water trails as compared to just hiking trails?
Tim: Right. So, jumping to superior user-generated content with Automatic, you've just kind of given them the tools and turned them loose. Can you talk a little bit more about your vision and strategy of how the users are going to build your business?
Tony Schneider: Absolutely. I should say sitting between $220 million and no VC money, we took a little bit of VC money. Our approach is actually in some ways very close to what you guys are doing at Craigslist, in that we are extremely focused on our users and listening to them telling us what to do and even more so not just listening to them but turning them into contributors and collaborators.

Part of that is because Wrox Press started as an open source project. With open source, of course, it's all about somehow bringing people in and letting them contribute. I think Wrox Press has been very successful in open source because of that, and really opening up as much as possible and sort of moving almost into what you would call the libertarian model of software development where we have a little bit of centralized control to avoid chaos and vandalism and everything else gets really opened up to the users.

So, we have not just people developing software but people documenting and writing plug-ins and designing new themes and supporting each other and the list goes on. Literally hundreds of thousands of people at any given time are working on our product. So, that's the roots of where we came from.
Tim: I am really glad you bring in the connection between open source and Web 2.0. That's how I got there. I was thinking about what happened at the open source, which is also the first example of network collaboration as that started moving up the stack and other tools and activities followed.
Tony: I think what we have learned is that we can take those principles from open source and apply them to a company, which is what Automatic is trying to do. Also apply them not just to the technology development but to business decisions like Jim mentioned. That is really the entire way of how we run the company.
Tim: Can you also talk a bit about Akismet? Because it is really easy to fall into the trap to think that collective intelligence is about user generated content. I want to distinguish those two.
Tony: Absolutely.
Tim: So what is the way to do that?
Tony: The term "user-generated content" is actually a term I dislike strongly, and I wish we could stop using it. It is for a couple of reasons. One is, it's too narrow a term. It doesn't describe a lot of phenomena that really are under the umbrella of collective intelligence, and Akismet is an example of that. Akismet is our anti-spam service and it's a centralized service that keeps spam, initially, off of blogs. It is also now used to keep spam off of wikis and social networks and all other kinds of sites that are open to user-contributed content, so they are also open to spammers.

It is really becoming a big problem. I was just mentioning backstage that we have seen a doubling of spam in our blogs in just the last three weeks. It's just going up like crazy. The solution we built is a centralized service that sees all the spam across hundreds of thousands of blogs.
Tim: When I mark it as spam on my blog, you know that it is...
Tony: Exactly. It is very good at catching things. If something goes through, somebody can mark it. It gets sent back to the centralized service and it learns and it adapts itself. This is an example of harnessing collective intelligence. You have got hundreds of thousands of bloggers working together to keep the spam away. It is not user-generated content though. It doesn't fall into that bucket at all.
Tim: So Owen -- I am sorry we are taking probably a little too long, kind of working through each of you, I would love to get you talking amongst yourselves so to speak, but -- Owen, talk to me just about what you are doing at Facebook, what you learned from Amazon that you brought to Facebook and just your whole take on this question of your application as an example of harnessing collective intelligence.
Owen: Sure. We think about Facebook as a website that helps increase information flow among people. We have taken a little bit of a different approach than I think a lot of folks have. Then how it is that we have done that? We view what it is we do as, we build tools and platforms/vessels, that we put into a structure that includes who knows whom and we let the users basically use those tools and platforms for communicating and sharing information. It's highly effective, and that's essentially what we do as a company. In terms of things I've learned at Amazon or other places, I didn't necessarily bring these things to Facebook.

Mark, who is our founder and CEO, has a very user-centric view of the world, and of our product, and our business and what it is we're trying to do. That's very consistent with how you've heard Jeff Bezos talking about wanting to be the customer-centric company in the world as an online retailer. And I think a lot of those values just mapped very well and that's one of the reasons I think why we've been able to be a very good.
Tim: So a question, word on the street is that a lot of people are moving from MySpace to Facebook, do you have any thoughts on why that might be? Or you might have thoughts on whether you think that's true.
Owen: I don't really know if that's true or not. What we've definitely seen at Facebook is we have improved the tools we offer our users and we've certainly reacted to their feedbacks. People in the audience, you may have heard about this product we launched called News Feed and our users got pretty upset that we made all the changes we made, and that we didn't control some of the controls that they wanted over how it is that their information is shared through News Feed. And in fact, our tools and our platform ended up allowing them to come together with a collective voice that they really weren't able to do otherwise.
Tim: Well, that kind of brings up an aspect of just how responsive you can be to your customers. And it's part of, "How do you listen?" I mentioned backstage, Kathy Sierra, on her Creative Passion user's blog had this great post about MySpace and what was so great about it and her daughter said, "Everything you want is there as soon as you want it! They keep changing and adding new features just when I want them." So how do you do that, Richard?
Richard: That's absolutely true, I would say one of the main differences is we looked at product features as marketing. Because instead of spending dollars on marketing, because we spend none, we'd spend it on trying to roll out as many features as we could but as quickly as we could. So the concept was...
Tim: And then if people use them, they'd stay, and if people didn't use them, they'd go away?
Richard: Exactly. And we started out with Tom actually listening to every email and trying to understand what the product was that people wanted, and then a team of people working with Tom -- and Tom today still determines every single product that goes on the website. And we really said that, we said rolling out features is our form of marketing and we invested most of our dollars in building out the product and then we believed the users would take it from there.
Tim: So that's a great segue into a kind of broad question I'd like to throw out to each of you. What is the biggest thing you've learned that if you're an entrepreneur out in the audience and you said, "Gee, wow, what can I learn from these guys?" What would that be? You know, your single biggest lesson. And then if people have questions after that, why don't we get you guys to come up to the microphones, or mic -- I think there's only one out there.
Tony: I'll give it a try. It's probably not the biggest one -- but certainly something that I think would be useful, having been at a startup at a big company like Yahoo back at the startup and working at Yahoo, at the sort of opening up and trying to bring in more users and really turn them more into contributors and collaborators -- what I found is that there's really a sort of an initial step of realizing that you need to open up and I think more and more companies are realizing that. And that has to do with opening up content and data and making it easy to connect to you.

And there's another step that I think is really hard, especially for big companies, and that's that you have to give up a lot of control. And I see these big companies who still think that the user's out there, and we're the creators and the closest I'll ever get to my users is sort of behind the mirrored glass and, you know, watching them use the software.

And there's also a mindset of, "I'll watch what people do, I'll analyze a click-stream and I'll try to learn and harness some of this intelligence out there." But it's always removed. And what you need to be able to do -- what I think I learned, again, through watching things like open source projects -- is you can give up a lot more control than that. You can hand over control to your users and you can let them do things.

As an example -- and we're not the first company to do this -- but we've handed over translation of to our users and not only can they translate it but what they translate goes onto the site live and this is a really scary thing for a big company. But for example, one of the first translations that went live was German. The German users were very active and, literally, it was like 24 hours later and it went up there.

And then a few weeks later we went actually through and sort of checked to see how good that translation is. And what was great was there were some mistakes in there and things we tweaked, but generally it was very high quality. And more importantly, there was only one string in the entire translation of thousands of strings that was wrong, just kind of clearly someone had put in something goofy, but it wasn't even obscene or rude, it was a guy announcing his wedding. [laughter]

And so we completely gave over control, which is now resulting in people in places like Latvia and Uzbekistan and Iran to put up our blogging solution and bring it to those markets. And it didn't really hurt us.
Tim: And anyone else have any...
Richard: I would just say, there's two things. One, which I think is for everybody here is, don't believe everybody that tells you about it being a bad idea. I mean, I have never had a business -- which is probably a bad omen -- except this one, which anybody thought was a good idea.

So, you know, when I was telling people that small businesses, back in 1994, were going to come online and we kind of created the first online shopping mall, which became the Commerce Solution. Nobody believed that made any sense, that any business would come online. And with MySpace and the rest of the company, nobody believed the users would generate content.

And I think the second thing is, give the power to the users. I mean, again, we let them build their own online stores in 1994 and 1995 and we had a thousand stores where people figured out how to do e-commerce, how to hook in credit cards, and how to do payment processing. So if you just focus on what you believe in and give the powers to the users, then in the end, the company will probably work out just fine.
Jim: I guess I want to amplify some of the same sentiments that defer to users, early and often. We only found one area where we can't really do that, and that's in the legal regulatory realm. I mean users... generally Americans don't believe in the Cuban embargo but to just freely allow the sale of Cubans cigars on the site, you can be jailed for that. So that's really the only area where we haven't been able defer to what our users want. [laughter]
Owen: My spin on this would kind of be: it's OK to try do things differently. I think one the things that's been a real learning experience for me at Facebook -- coming from Amazon, other startups, and other large companies -- is that. Taking that traditional path of how it is you build out your business, how it is you decide to manage your business, who you decide to put in charge of different things -- even though they may not have had that exact experience before, how you leverage experience, and how you leverage the fact that not having experience is kind of a liberating thing and it actually lets people think really big and do things that you couldn't even imaging you could do when you're encumbered with, "Yeah, we tried that, it's not going to work."

And I think we've done quite a bit of that at Facebook, just in the culture, how it is that Mark has decided to manage the product and the evolution of that and it has paid massive dividends. And if I had just come in and really pushed hard to do all the things that traditionally I would have done in certain situations, instead of really start to defer to that and be really much more open to trying new things, we wouldn't have made nearly as much progress as we have.
Jim: It strikes me that one interesting thing that is happening is that things that didn't use to work sometimes now do work as the internet progresses and moves along. So that idea of sometimes forgetting the lessons of the past is actually the right thing to do.
Tim: OK. Good point. I can't quite see who is back there but why don't you introduce yourself and.
David Forres: Hi, David Forres from the Motley Fool. Arthur from the New York Times the other day talked about knowledge versus opinion and the cost associated with knowledge versus opinion or popularity versus relevance. I'm wondering if each of you have a different definition of intelligence and how well do you think we are delivering on intelligence and places where we may not be delivering so well on intelligence.
Richard: Well I have a very quick answer to that. Anybody here in the audience ever heard of Sturgeon's Law? OK. Well he was a science fiction writer and somebody once came up to him and said, "95 percent of science fiction is crap," and he said, "Well, 95 percent of everything is crap" and that has come down as Sturgeon's Law and I think what it comes down is whether it's collective intelligence or maybe even professional intelligence from people like the CIA, 95 percent of it does turn out to be crap. But it's the five percent that matters and I think we are actually pretty good at getting to the five percent often through just sort of user review.
Tony: Yeah. I would agree with that. I doesn't mean that just because there is a lot more content that it's all good. But I think thee are a lot examples out there now of things that bubble up much more quickly and that we see what people want and what is interesting to them much more rapidly. And we build platforms where that can happen rather than just sort of top-down gatekeeper model where somebody has to decide it.

Real quick example is last week on our service we saw several blogs pop-up about the new Nintendo Wii device and they very quickly got popular. And it was literally things like people got the first boxes in, and they would take pictures of just unwrapping the thing and getting it out and putting them up there, and they have tens of thousands of visitors to their blogs within hours. And something like that can just pop up and get popular very quickly and it just spontaneously emerged. And it's clearly popular, I don't know if it's great content or intelligent, but it is clearly useful to people and it didn't have to go through big long stages of somebody deciding that maybe that would be interesting for someone.
Tim: So, it's the rapid feedback we iterate through error potentially. Next question.
Craig Kaplan: I'm Craig Kaplan from I'd like to pick up on something that was said earlier, that distinction between user-generated content and collective intelligence. It seems to me that's a very important distinction. That there is, something that a lot of people are doing, which is just putting more content out and having users put out content and having everybody just sort of say what they think. And then there is more sophisticated ways of combining the intelligence of the users, sort of the example of processing spam and so forth. I think there is a lot of potential in sort of cognitive communities where you have people working together and coming up with intelligent results that were otherwise possible. So, I'd just like the panelists to comment, if they could, on what sort of things you might be doing that go beyond just user- generated content and sort of touch on more collaborative or collective intelligence really.
Owen: I'll take that. One of the things that Facebook has launched in the last couple of months is a program that we call "Share" which effectively gives people the ability to both within Facebook and then also on third-party websites to share information with their Facebook friends. It packages that information, it basically makes it part of the user that they are sharing it with's Facebook experience. And these are the types of tools that allow for collaboration, commentary, the ability for someone to have something shared with them and then for them to take it and share it with many others. The whole viral nature of it is a big aspect of all of our businesses here. As we think about it, enabling people to have more control over their information and be able to more easily share information in a targeted way with their friends is at the core of how it is that we try to facilitate that.
Tim: I think that we are running out of time but we can maybe just squeeze in one more.
Tony Conrad: Great. I'm Tony Conrad from Sphere and one of the common threads I see in your businesses is the amazing communities that you have built. And the common thread of heavy lifting that those communities do and promoting the services but also in helping you shape the products. I'd like to understand and hear from you how do you view your role in that community as a participant or as a leader and nurturing it?
Tony Schnieder: Should I take that? I think that is a great question. I think that we have two roles. I think our number one role is to provide the best tools and technologies to let people express themselves. And one of the things that we are very focused on is to try and provide a structured format. Because it's like starting with a blank piece of paper, it's very difficult to think what you want write about, and for a lot of people what they want to potentially blog about. So if you have a more structured format and you know that somebody is enthusiastic about a category like home and garden and you provide them the ability to see other articles related to home and garden, and one of the things we talk about is professional content that will then spawn their own user generated content.

I think that is one thing that you can do to really kick start the ability because I truly believe that everybody wants to be creative, everybody wants to publish. They just may not know how and they may not know what format.

So I think it's our role to, 1) Focus on the tools and 2) We need to police the community if it gets out of hand.

I think it's really important to let the community police itself, but if there are some users that are just ruining the experience for everybody or talking about stuff that's -- it's OK if it's not relative -- but if it's destructive to the collective then as moderators, not us as executives but users, need to have the power to remove those people so that they don't stop that whole experience.
Tim: So, in other words, we still have a long way until we really learn how to manage collective action on the net and manage groups. We've got some good tools but it's still an evolving story.
Tony: It really is the first inning.
Richard: The key thing in my mind to that is giving the users the control. It's the fine-grained controls over their information that ultimately allows the policing to occur in a much more organic way within the community. I think it's the emphasis on giving people control over information, both control to share with a large number of people and control over exactly who gets to see it that is really going to allow these things to continue to flourish in big way. Without turning into the type of place that you just don't want to be, where the bad stuff starts to outweigh all the value that people get out of it today.
Jim: I think one more quick point on that is, and maybe what Tony was trying to get at a little bit to, is that, I think an additional effect of opening up to users and a collective intelligence is not just that they're starting to participate in a product but that they are starting to participate in your company and especially the very active users are starting to feel almost an ownership in the company.

I'm sure this happens with Craigslist a lot, where people very much feel part of not just their content that they create but the overall company and I think that's part of the role in the kinds of companies that we are building, is to understand that and respect and provide ways for people to feel rewarded for it.
Tim: I think that is a great note on which to end. Thanks a lot, I really appreciate you guys coming out and sharing your experience with everyone.

Announcer: Jim Buckmaster, Richard Rosenblat, Tony Schneider and Owen Van Natta at the Web 2.0 Summit 2006.

Daniel H. Steinberg is the editor for the new series of Mac Developer titles for the Pragmatic Programmers. He writes feature articles for Apple's ADC web site and is a regular contributor to Mac Devcenter. He has presented at Apple's Worldwide Developer Conference, MacWorld, MacHack and other Mac developer conferences.