Wikia Search issues community-based challenge to Google and Yahoo

Yesterday brought the alpha release of Wikia Search, a new engine built on free software and free culture values. For Wikia co-founder Jimmy Wales, the release is a milestone in the realization of a long-held dream. However, as he tells Linux.com, Wikia Search is still a couple of years from maturity, with many of the details still to work out.

“Philosophically, I’m a big supporter of free software,” says Wales. “I’ve been interested in a streamlined, transparent search engine for a long time.” Not only are free software technologies powering Wikia Search — including Grub, the distributed Web crawler that Wikia bought last summer and whose code it immediately released — but Wales is determined that “at every point where we find that there’s an editorial decision to be made, we want to push that decision outside the company and into the community.”

He talks about a social networking profile that will double as a people-finder; mini-articles for features such as disambiguation, pictures, and definitions and “whatever the community thinks sensible”; a five-point rating for each search result; and white and black lists to help the community improve the quality of results. With the alpha release, only some of these features are implemented (see sidebar), but, as he talks, Wales’ excitement is obvious — even though he has clearly said the same thing many times in the year since work on the search engine began.

“It’s a project to make a search engine,” he says simply.

History and influences

Despite this philosophical orientation, Wikia Search is supported by hard-headed business interests. A project of Wikia — a company that its publicists are at pains to note has no direct connection with Wikipedia — Wikia Search was first announced 13 months ago. It is funded by a $4 million investment by each of Amazon.com and Bessemer Venture Partners, and its development has received a boost by the acquisition in December 2006 of ArmchairGM, a sports-oriented wiki, and in July 2007 of Grub, a distributed search engine whose code Wikia immediately released. Just as importantly, the generally unspoken assumption behind the project is that community-based features can improve the quality of Internet searches to the extent that Wikia Search can rival long-established companies such as Google and Yahoo.

Technically, Wikia Search is as separate from Wikia’s wiki communities, where users are attempting to write articles about their areas of interest, as it is from Wikipedia, the project for which Wales is best known. However, the search engine shares a common login with the wiki communities, and, at least so far, many of the top search results point to articles written by the communities. So far, Wikia has about 4,000 communities, and, according to Wales, “A lot of those members may be involved in the search community. We’re not sure yet.”

Wales cites the development of the Open Directory Project (ODP) as crucial to the development of his thinking. “The Open Directory Project started with a similar concept,” he says. “But, unfortunately, since AOL bought them, they’ve been basically ignored.” According to Wales, the problem was that, “when the ODP faced problems with spam and editing, they did what everybody thought was natural, [and] became a very closed and insular community. It’s very hard to join or participate.”

By contrast, he says, “What Wikipedia has shown is that there’s another way, [making] everything open and transparent. Whatever actions that people take, they can be seen, so that they can be scrutinized by the community, so that people can be blocked or banned when misbehaving.”

Looking ahead

In a news release, Wiki Search is described as being dedicated to transparency, community, quality, and privacy. Many of these values are already observable in the alpha release, such as a three-tier privacy setting on user profiles.

However, many of the details are still evolving. Perhaps the least developed is the rating system, which is intended as one of the main sources for quality assurance. “We’re leaving it very open-ended to start with,” Wales says. “When users are rating things, we want to feed that information back into improvement of the search results. But we don’t yet know how to do that, because we don’t know yet what the data will look like. So once we begin to have data flowing in, we can begin experimenting with different algorithms and different ways of looking at that.

“We want to engage academics, hackers, and anybody who wants to participate in how you get from user data back to quality results,” he says. One possibility, he suggests, is having results by creating a network of trusted relationships and slanting results received by logged-in members according to comments made by their friends.

Such details will not be worked out overnight. With the front page currently admitting that “the quality of the search results is low,” Wales suggests that Wikia Search will take “a minimum of two years before we have industry-quality results.”

Still, Wales is clearly in the project for the long term. Asked why he is taking on a project that represents a direct challenge to two near monopolies, Wales replies, “Why not? Since you’re from a Linux site, you know the answer to this one. It’s, ‘Gee, there’s Microsoft, and its pretty dominant. Why are you wasting your time on this crazy open source thing?’ Because it’s fun, that’s why.”