Nearly a decade ago I spent a few wonderful months living in San Francisco. I got a short contract position at a web company in SoMa. It was an exciting time to be in the industry, still before the Tech Wreck. For a wide-eyed young graduate from Brisbane it was the funkiest place to work I could imagine. There was a games room with a pool table. You didn’t have to put coins in the softdrink machine. There were free massages for staff on Tuesdays. But my favourite part, geek that I am, was every Friday when people would bring their lunch, sit around on beanbags and listen to one of the team deliver a lecture or presentation on some aspect of technology or culture that interested them. That’s where I first heard about collaborative filtering.

They’ve been talking about collaborative filtering, a subset of recommendation engines, over at ReadWriteWeb.

In January 2000, recommendation engines were still building up a head of steam, but of course now they’re everywhere. Our daily experience of the internet is so completely overwhelmed by data – email, websites, youTube videos, music, widgets – that it’s almost impossible to self-impose any kind of filter. That’s where recommendation engines come in – they help people to find content they should (in theory) like based on their own past behaviour, or someone else’s past behaviour, or the nature of the content they are seeking.

The second one – recommendations based on someone else’s past behaviour , or collaborative filtering – is used by retailers like Amazon. 67% of people who bought that book also bought these other four books.

It makes sense to apply this to consumer models. Retailers want you to buy more things, so they make it easy for you to find content you’ll like.

But it occurs to me that the algorithms of recommendation engines could be used by publishers as well. How could, for example, this modelling provide a clearer picture for book publishers sifting through piles of unsolicited manuscripts? Publishers are always searching for the next big hit and often only have the last couple of successful books as a benchmark. But what if they could turn the data into more meaningful decision-making tools? Would it enable them to make more commercially accurate calls on manuscripts they are considering taking to market?

I guess unpublished authors try (most often unsuccessfully) to do this in their queries. “My young adult fantasy would sit comfortably next to Harry Potter…” 

But it is an interesting thought-experiment to consider how recommendation engines could guide content publishing decisions not just consumer purchasing decisions. It may require manuscripts to be tagged or scored in some way, which ultimately could be as onerous as simply assessing them subjectively or qualitatively based on an editor’s experience and instinct.  Perhaps it’s something that could be combined with a crowd sourcing platform to harness the power of a pool of readers or fans.