The logical next step in this technological journey is to therefore prune, to make our time online more meaningful and relevent, no matter how small the nugget of information. Whether I’m setting out to qualify findings in a drug discovery experiment or wondering when Amy Winehouse was last arrested, I want the most reliable, relevant answer in the shortest amount of time. The problem is no longer whether the information is out there but rather how we can get to it quickly and accurately.

It’s against this background that I’m seeing a gradual evolution of the semantic search market. What was once viewed as a sector attempting to “kill” Google has become something different in recent months. It turns out that what Google does is 1) pretty hard to do well and 2) pretty sufficient for most people. Creating an all-purpose search engine that answers any query in any form and delivers a more targeted result will take time to get right. I don’t think we’re going to wake up one day to find GoogleKiller.com suddenly filling all our search needs.

Several companies I’ve met with lately have innovative takes on making the Web more meaningful for users. But I’ve noticed that my reaction has gone from, “Wow, I’ve never seen that before” to “Oh, that’s an interesting take on what so-and-so is doing.” In other words, I believe smarter search and discovery have reached a new stage in their evolution. The problem has been well defined and the angles from which it can be approached have been established (at least for the foreseeable future.) It’s now time to focus on the niggling details: how to identify and amass relevant information, present it to users in an easily consumable yet rich visual format, and, perhaps most importantly, establish a user base that will consistently turn to your product as a valued information resource.

I’ve written previously about companies that I keep a close eye on in the smarter search space. These products, along with a couple of new entrants, bear repeating, as they’re molding a new shape for the Internet.

Powerset raised the bar for UI with its Wikipedia search engine, and, perhaps more importantly, sent a tacit message to the tech world that a smarter search engine isn’t going to develop overnight. Regular readers of The Guidewire know I won’t shutup about Silobreaker, a current affairs search engine that really shines in visualization and content extraction. Its entity maps are among the best I’ve seen, providing instant insight into relevant connections between people, places and things. Evri is a newer entrant in the field, with a product that combines the visual maps of Silobreaker with the subject and verb extraction of Powerset to give users a deeper dive into news articles. Semantinet brings to mind Yoono‘s discovery feature, but on steroids, recognizing entities on the page you’re viewing and bringing in related info from other sites. It makes discovery more intriguing by adding a personal layer, i.e., while viewing a page about Italy, receiving notification that a Facebook friend just added photos from Italy. True Knowledge is Freebase meets Cyc with natural language search thrown in for good measure.

As I was writing this post, a couple of good articles popped up that are worth reading. ReadWriteWeb took a broad look at trends it believes will shake up search in the near future. And Epicenter’s Betsy Schiffman had an interesting take on Google’s dominance of the market. I have to respectfully disagree with Chris Sherman of Search Engine Land, who said in Schiffman’s piece:

“I certainly think the game’s over in terms of building a world-class search engine. It’s too expensive — Google and others are literally spending hundreds of millions of dollars, in terms of research, development and infrastructure. Start-ups don’t have access to those sorts of resources.”

I see his point about cost but think we’re far from declaring game over. Though GoogleKiller.com may not appear any time soon, the smaller engines are working other avenues to build adoption and user base. The game has changed from pouring millions into a destination site to instead channeling resources into multiple iterations of a product to gain traction. Powerset, for example, is testing a new Facebook app that finally brings capable search into the social network, in addition to its just-announced iPhone app, which generated 46% of its total traffic yesterday.

Sherman is right in one aspect: with such a complex problem and so many angles from which to attack, going it alone as a destination engine makes for a tougher, longer and more expensive fight. Smart companies are now shifting focus, integrating their algorithms into existing communities that are sorely in need of sophisticated search. Look for this integration and coalescing to continue on multiple levels in the next year. It will no longer be about killing Google but about making search, discovery and recommenation a more seamless and intuitive part of our existing online lives.

Powerset’s introduction today of its new Wikipedia search, which also integrates data from Freebase, could have a significant impact on the tech market overall, in that it changes the rules of the search game. Users who experience the incredibly deep, interactive, and intuitive nature of the Powerset search will be even more frustrated with the standard string of result pages delivered by traditional keyword search. Once you’ve dug into the meat of a Wikipedia article with just a couple of clicks, zeroing in on precise actions and entities and going directly to their citations in the article, paging through flat hyperlinks just ain’t going to cut it.

Powerset’s changing of the rules is evidenced by one key statement made by the company: a page of search results, no matter how targeted, is just the beginning of the effort required by the user. Once you’ve found relevant links, you still have to click through to new pages and scour the text for usable information. Powerset’s new way of searching attempts to do some of that work for you; with the scouring and drilling down already complete, you arrive at what you need much quicker.

The Outline feature of the Powerset search is a real gem and I expect will set a new standard for UI in search technology. Having a constant window beside the text as you browse provides an incredibly simple way to jump back and forth between concepts and facts. It could make the browser’s back button obsolete.

What I don’t love about the new search is that it’s currently only on Wikipedia. There are many searches I typed in that can’t take advantage of all this whiz-bang semantic technology. More nebulous concepts aren’t Wikipedia’s strong suit, so Powerset only returns standard results. Example: “Can Hillary win the democratic nomination” returned relevant results but no Wikipedia entry to plumb. So my big “if” with this announcement is whether Powerset can pursue a successful content partnership strategy. If the right publishers, and enough of them, integrate Powerset search into their sites, the long-anticipated threat to Google could finally take shape. No matter the long-term outcome, though, Powerset has raised the bar for search interaction and usability.

Wikia could have stemmed some of the criticism upfront with more effective messaging. Though coverage in the BBC and New York Times was more even-handed, Wales still trumpeted to them the message of community participation in search result creation. I don’t think that’s the real story here. According to Penchina, while the goal of Wikia Search is indeed to create a community-created search engine, it’s also to open up the algorithm of search on a larger scale, allowing people to create their own engines. (To download the crawler and create your own engine, visit Grub.org) Bringing the power of open source to search, a heretofore secretive and highly guarded tech sector.

This is not a search engine to compete with Google. Reviewers who tackled the search function itself, testing certain phrases to see which results came back, were set up to fail. Wikia set out to bring a Web-2.0-level of open innovation to search and, in the process, launched an engine of their own. Whether Wikia Search can direct you to relevant results on a topic isn’t the point, at least not now. Wales and Penchina hit a nerve with Wikipedia and saw how strongly the Internet community wanted involvement in its technology.

It’s what more and more companies are doing these days: sharing powerful technology with the community at large to do with as they see fit. MetaWeb (Freebase), Faraday Media (APML), Franz (AllegroGraph), and now Wikia – all developed sophisticated technologies that are now available for the greater good. It’s a bit of technology socialism, if you will.

Gil ended our conversation with an excellent analogy. The open source world is a relay race. Some will work on a technology for a while, burn out and drop the baton. There will always be someone else running by, willing to pick it up and contribute their own effort. Wikia Search isn’t a single product to be evaluated in the now. It’s a new attitude toward building search engines, one that will bear fruits down the road, each to stand or fall on its own merits.

Powered by ScribeFire.