Back in the old days – or the ’90s as some call them – we utilized the Internet as an information resource. What’s that phone number, where is that address, where can I buy that product – you had concrete questions and were no longer required to speak to a human to get answers. Sure, there were bulletin boards and Usenet forums for discussion but they primarily involved coding arguments and game walkthroughs. The Internet wasn’t truly upended into a community, and all that that entails, until just a couple of years ago. It was then that the inundation of bloggers collided with social networking and lifestreaming to produce a perfect storm of content. (And when I say lifestreaming, I mean the trend of putting as many pieces of our life online as possible – books we’re reading, music we like, etc.) We’ve now backed ourselves into a corner online, raging against the indundation of content even as we scroll through our fifth page of FriendFeed updates. We recommend well-written articles about navigating through the noise, right after sharing 25 items in Google Reader.

The logical next step in this technological journey is to therefore prune, to make our time online more meaningful and relevent, no matter how small the nugget of information. Whether I’m setting out to qualify findings in a drug discovery experiment or wondering when Amy Winehouse was last arrested, I want the most reliable, relevant answer in the shortest amount of time. The problem is no longer whether the information is out there but rather how we can get to it quickly and accurately.

It’s against this background that I’m seeing a gradual evolution of the semantic search market. What was once viewed as a sector attempting to “kill” Google has become something different in recent months. It turns out that what Google does is 1) pretty hard to do well and 2) pretty sufficient for most people. Creating an all-purpose search engine that answers any query in any form and delivers a more targeted result will take time to get right. I don’t think we’re going to wake up one day to find GoogleKiller.com suddenly filling all our search needs.

Several companies I’ve met with lately have innovative takes on making the Web more meaningful for users. But I’ve noticed that my reaction has gone from, “Wow, I’ve never seen that before” to “Oh, that’s an interesting take on what so-and-so is doing.” In other words, I believe smarter search and discovery have reached a new stage in their evolution. The problem has been well defined and the angles from which it can be approached have been established (at least for the foreseeable future.) It’s now time to focus on the niggling details: how to identify and amass relevant information, present it to users in an easily consumable yet rich visual format, and, perhaps most importantly, establish a user base that will consistently turn to your product as a valued information resource.

I’ve written previously about companies that I keep a close eye on in the smarter search space. These products, along with a couple of new entrants, bear repeating, as they’re molding a new shape for the Internet.

Powerset raised the bar for UI with its Wikipedia search engine, and, perhaps more importantly, sent a tacit message to the tech world that a smarter search engine isn’t going to develop overnight. Regular readers of The Guidewire know I won’t shutup about Silobreaker, a current affairs search engine that really shines in visualization and content extraction. Its entity maps are among the best I’ve seen, providing instant insight into relevant connections between people, places and things. Evri is a newer entrant in the field, with a product that combines the visual maps of Silobreaker with the subject and verb extraction of Powerset to give users a deeper dive into news articles. Semantinet brings to mind Yoono‘s discovery feature, but on steroids, recognizing entities on the page you’re viewing and bringing in related info from other sites. It makes discovery more intriguing by adding a personal layer, i.e., while viewing a page about Italy, receiving notification that a Facebook friend just added photos from Italy. True Knowledge is Freebase meets Cyc with natural language search thrown in for good measure.

As I was writing this post, a couple of good articles popped up that are worth reading. ReadWriteWeb took a broad look at trends it believes will shake up search in the near future. And Epicenter’s Betsy Schiffman had an interesting take on Google’s dominance of the market. I have to respectfully disagree with Chris Sherman of Search Engine Land, who said in Schiffman’s piece:

“I certainly think the game’s over in terms of building a world-class search engine. It’s too expensive — Google and others are literally spending hundreds of millions of dollars, in terms of research, development and infrastructure. Start-ups don’t have access to those sorts of resources.”

I see his point about cost but think we’re far from declaring game over. Though GoogleKiller.com may not appear any time soon, the smaller engines are working other avenues to build adoption and user base. The game has changed from pouring millions into a destination site to instead channeling resources into multiple iterations of a product to gain traction. Powerset, for example, is testing a new Facebook app that finally brings capable search into the social network, in addition to its just-announced iPhone app, which generated 46% of its total traffic yesterday.

Sherman is right in one aspect: with such a complex problem and so many angles from which to attack, going it alone as a destination engine makes for a tougher, longer and more expensive fight. Smart companies are now shifting focus, integrating their algorithms into existing communities that are sorely in need of sophisticated search. Look for this integration and coalescing to continue on multiple levels in the next year. It will no longer be about killing Google but about making search, discovery and recommenation a more seamless and intuitive part of our existing online lives.

Chris and I engaged in a lively discussion via Skype this morning regarding the merits of Rushmore Drive, a new search engine targeted to African-Americans. I heard about it from SheGeeks, who stated quite clearly how she feels about the service. Especially after hearing Rushmore Drive is under the same corporate umbrella as Ask.com, my immediate reaction was also one of dismissal and “what the hell are they thinking?” You may remember my own rant a couple of months ago about Ask’s development of a search engine targeted to suburban women in the Midwest. My point then – that the path to search success lies in broadening rather than narrowing your audience – holds true with Rushmore Drive. Assuming a group of people wants results from a limited pool denegrates the audience and simply doesn’t hold water.

Chris isn’t necessarily a fan of these sites either but she can’t help but put her experienced analyst hat on and deliver some opposing points. Her argument is that engines like Rushmore are serving a viable subset within a demographic that vehemently holds on to that demographic as their identity. There are enough of those subsets in any demographic to create a business; the question of how big that business is remains to be answered. She concluded by allowing that there are some issues she might turn to a women’s site over a general one, assuming they’ll have better information, i.e., health-related such as breast cancer, pregnancy, or menopause.

It’s easy to deliver emotional responses to such a model, as it’s inherently personal. That, after all, is the intended effect of the engines. Unable to differentiate algorithmically from the Google way of search, these companies are instead aiming to add a personal layer. If I can’t necessarily deliver a better search result to you, dear user, I’ll try appealing to your gut. Who are you and with whom do you identify? It’s a philosophical/psychological approach and it’s risky. To work effectively, the engine must excise some results and/or bring others to the fore. Who’s making that determination? Can one possibly write an algorithm to home in on female or African-American search results? I doubt it and I think that’s the point.

Technology should be blind to race, gender and creed. If you want to appeal to a demographic, create a destination site. Pack it to the hilt with what you think are appropriate links and material and let it be sourced by a general search engine. But the very nature of search is and should be egalitarian. Attempting to attract certain groups of people by rearranging their search results is, at best, touting a product for what it doesn’t do. And that to me, seems bad business.

What do you think?

The recent hubbub over Wikia Search had many questioning the wisdom of tech stalwarts Jimmy Wales and Gil Penchina. What were they thinking, introing a rudimentary search app that delivered poor results? At varying times called a “practice run,” “doom [for] Wikia as a business,” and my personal favorite “weapons grade fail,” seasoned tech bloggers were unrelenting in their criticisms. After talking with Gil earlier this week, though, I think the most interesting angle on this story has been MIA.

Wikia could have stemmed some of the criticism upfront with more effective messaging. Though coverage in the BBC and New York Times was more even-handed, Wales still trumpeted to them the message of community participation in search result creation. I don’t think that’s the real story here. According to Penchina, while the goal of Wikia Search is indeed to create a community-created search engine, it’s also to open up the algorithm of search on a larger scale, allowing people to create their own engines. (To download the crawler and create your own engine, visit Grub.org) Bringing the power of open source to search, a heretofore secretive and highly guarded tech sector.

This is not a search engine to compete with Google. Reviewers who tackled the search function itself, testing certain phrases to see which results came back, were set up to fail. Wikia set out to bring a Web-2.0-level of open innovation to search and, in the process, launched an engine of their own. Whether Wikia Search can direct you to relevant results on a topic isn’t the point, at least not now. Wales and Penchina hit a nerve with Wikipedia and saw how strongly the Internet community wanted involvement in its technology.

It’s what more and more companies are doing these days: sharing powerful technology with the community at large to do with as they see fit. MetaWeb (Freebase), Faraday Media (APML), Franz (AllegroGraph), and now Wikia – all developed sophisticated technologies that are now available for the greater good. It’s a bit of technology socialism, if you will.

Gil ended our conversation with an excellent analogy. The open source world is a relay race. Some will work on a technology for a while, burn out and drop the baton. There will always be someone else running by, willing to pick it up and contribute their own effort. Wikia Search isn’t a single product to be evaluated in the now. It’s a new attitude toward building search engines, one that will bear fruits down the road, each to stand or fall on its own merits.

Powered by ScribeFire.

There’s a lot of anger in the air these days. It’s the nature of blogging, I suppose – dash some vitriol off and you’re guaranteed to start a conversation. That’s not necessarily a bad thing; better to generate emotion of any kind than none at all. But it can also be damaging to the very essence of entrepreneurship. If ideas are met with too much resistance, will the tap eventually dry up? Will innovators become too afraid to put themselves out there, lest they anger the blog gods? The answer depends on who you are.

I talked with two companies recently that have been on the receiving end of some anger. Wikia Search rolled out its alpha to a chorus of “What the hell is this?” Company execs responded, “It is what is,” able to do so because of a supportive community and a general agreement that the search sector needs innovation. At the other end of the spectrum lies Story2Oh, an intriguing little project started by screenwriter Jill Golick, who wanted to dip her toe into Web 2.0 and got smacked down harshly in return. I’ll cover specifics on both companies in future posts but wanted to comment on the larger issue here. Golick is the very definition of an “outsider,” working in a field far removed from Silicon Valley (penning dramas for Canadian television). She was intrigued by the possibilities of the social graph, taking an experimental attitude of “let’s go play in the new frontier.” Only two weeks in the making, Story2Oh was awarded the label of “stupidest idea ever” by a popular blog, leaving Golick to wonder what she was thinking in the first place.

The merits of the Story2Oh idea are still to be determined. I’m not posting about it because I necessarily think it’s going to succeed. What gets my goat is the implication that she doesn’t have the right to try. Wikia Search is also the definition of a work in progress but because they’re Valley insiders (and admittedly have a huge success in their pocket), the anger was somewhat tempered with a “let’s wait and see what happens.” I’d bet a pile of cash that if Jimmy Wales had come up with the idea of social storytelling, the response would have been quite different.

As tech insiders, we should think of ourselves as hosts; guides to a world that welcomes innovation no matter whence it comes. Let’s take the “community” that is so important to Web 2.0 and share it with everyone on the Web, not just the ones who show up at cocktail launch parties. Otherwise, we may critique ourselves right into oblivion.