Back in the old days – or the ’90s as some call them – we utilized the Internet as an information resource. What’s that phone number, where is that address, where can I buy that product – you had concrete questions and were no longer required to speak to a human to get answers. Sure, there were bulletin boards and Usenet forums for discussion but they primarily involved coding arguments and game walkthroughs. The Internet wasn’t truly upended into a community, and all that that entails, until just a couple of years ago. It was then that the inundation of bloggers collided with social networking and lifestreaming to produce a perfect storm of content. (And when I say lifestreaming, I mean the trend of putting as many pieces of our life online as possible – books we’re reading, music we like, etc.) We’ve now backed ourselves into a corner online, raging against the indundation of content even as we scroll through our fifth page of FriendFeed updates. We recommend well-written articles about navigating through the noise, right after sharing 25 items in Google Reader.

The logical next step in this technological journey is to therefore prune, to make our time online more meaningful and relevent, no matter how small the nugget of information. Whether I’m setting out to qualify findings in a drug discovery experiment or wondering when Amy Winehouse was last arrested, I want the most reliable, relevant answer in the shortest amount of time. The problem is no longer whether the information is out there but rather how we can get to it quickly and accurately.

It’s against this background that I’m seeing a gradual evolution of the semantic search market. What was once viewed as a sector attempting to “kill” Google has become something different in recent months. It turns out that what Google does is 1) pretty hard to do well and 2) pretty sufficient for most people. Creating an all-purpose search engine that answers any query in any form and delivers a more targeted result will take time to get right. I don’t think we’re going to wake up one day to find GoogleKiller.com suddenly filling all our search needs.

Several companies I’ve met with lately have innovative takes on making the Web more meaningful for users. But I’ve noticed that my reaction has gone from, “Wow, I’ve never seen that before” to “Oh, that’s an interesting take on what so-and-so is doing.” In other words, I believe smarter search and discovery have reached a new stage in their evolution. The problem has been well defined and the angles from which it can be approached have been established (at least for the foreseeable future.) It’s now time to focus on the niggling details: how to identify and amass relevant information, present it to users in an easily consumable yet rich visual format, and, perhaps most importantly, establish a user base that will consistently turn to your product as a valued information resource.

I’ve written previously about companies that I keep a close eye on in the smarter search space. These products, along with a couple of new entrants, bear repeating, as they’re molding a new shape for the Internet.

Powerset raised the bar for UI with its Wikipedia search engine, and, perhaps more importantly, sent a tacit message to the tech world that a smarter search engine isn’t going to develop overnight. Regular readers of The Guidewire know I won’t shutup about Silobreaker, a current affairs search engine that really shines in visualization and content extraction. Its entity maps are among the best I’ve seen, providing instant insight into relevant connections between people, places and things. Evri is a newer entrant in the field, with a product that combines the visual maps of Silobreaker with the subject and verb extraction of Powerset to give users a deeper dive into news articles. Semantinet brings to mind Yoono‘s discovery feature, but on steroids, recognizing entities on the page you’re viewing and bringing in related info from other sites. It makes discovery more intriguing by adding a personal layer, i.e., while viewing a page about Italy, receiving notification that a Facebook friend just added photos from Italy. True Knowledge is Freebase meets Cyc with natural language search thrown in for good measure.

As I was writing this post, a couple of good articles popped up that are worth reading. ReadWriteWeb took a broad look at trends it believes will shake up search in the near future. And Epicenter’s Betsy Schiffman had an interesting take on Google’s dominance of the market. I have to respectfully disagree with Chris Sherman of Search Engine Land, who said in Schiffman’s piece:

“I certainly think the game’s over in terms of building a world-class search engine. It’s too expensive — Google and others are literally spending hundreds of millions of dollars, in terms of research, development and infrastructure. Start-ups don’t have access to those sorts of resources.”

I see his point about cost but think we’re far from declaring game over. Though GoogleKiller.com may not appear any time soon, the smaller engines are working other avenues to build adoption and user base. The game has changed from pouring millions into a destination site to instead channeling resources into multiple iterations of a product to gain traction. Powerset, for example, is testing a new Facebook app that finally brings capable search into the social network, in addition to its just-announced iPhone app, which generated 46% of its total traffic yesterday.

Sherman is right in one aspect: with such a complex problem and so many angles from which to attack, going it alone as a destination engine makes for a tougher, longer and more expensive fight. Smart companies are now shifting focus, integrating their algorithms into existing communities that are sorely in need of sophisticated search. Look for this integration and coalescing to continue on multiple levels in the next year. It will no longer be about killing Google but about making search, discovery and recommenation a more seamless and intuitive part of our existing online lives.

I’ve been playing around in Persai‘s beta for a while now and wasn’t initially sure what to make of it. It’s a content recommendation site, pure and simple, and I have so much other content flying at me that I didn’t see how the site could fit into my daily grind. But the algorithms behind it prove more intriguing as the days go by.

There’s no learning curve with Persai: enter terms you’re interested in and it returns articles on a regular basis. No social networks, no friend lists, no ratings. Just news. So simple, it’s almost jarring at first glance. Once the engine returns relevant articles, you have two choices – read or reject. Both actions help the engine learn, yes, but Persai’s creators – Matt Kent, Kyle Shank and Ted Dziuba – don’t want users to set out with the aim of teaching the engine. “Just use the thing,” they say in the company blog. So that’s precisely what I did. Among the several interests I set up – including emerging tech, election 2008, independent film and semantics – Persai hits closest to the mark for election ’08 and semantics. I attribute the former to a large pool from which to draw and the latter to the specificity of the term. The independent film category has been the toughest for the engine to nail; for some reason, it keeps giving me articles about the Indian film industry. Not that there’s anything wrong with that.

I’ve never been in a beta quite like Persai before. Its creators don’t give interviews, there’s no whiz-bang design to pretty up the site, and I don’t think I’ve ever received an email from them, a la, “Here are our new features! Tell us how it’s going! Are you happy? Let us know if you’re not!” They’re not marketers, in other words. All that’s left to contend with is the technology and I find that profoundly interesting. Users are essentially thrown into the engine with little background or direction as to how to use it. I suspect this is what a beta looked like way back before it became a marketing term: wrestle with the tech for a bit and here’s a tiny Feedback link if you have suggestions.

Persai is in a profoundly tough space, considering the breadth and depth of companies playing in recommendation, machine learning, and news filtering. Its revenue model currently consists of contextual ads, which I’ve found to be quite close to the mark when filtered through their engine. In short, Persai has wormed its way into my daily grind, whether I wanted it there or not. I find articles that I’m not seeing anywhere else. The company is clearly smart about its algorithms. But how they push those out to the market, make them friendly for mass consumption and position themselves against weighter entities will be just as much, if not more, of a test.