Posts Tagged ‘netflix’

R.I.P. Movie Rating Prediction

R.I.P. Movie Rating Prediction

There is no longer any reason to bother researching new ways of predicting the ratings users will give to movies.  It’s time to move on to more interesting things.  But seriously, given the fact that the last few miles of the Netflix competition were hard-fought by combining hundreds of different algorithms, is there much value in trying to improve recommender systems in this way, anymore?

I expect that the Netflix Prize data set, if left open to the public, will still be useful for a number of machine learning tasks where the goal is not necessarily improving recommender systems.  So predicting movie ratings may never be really dead.  But it is my hope that that as a goal for research will diminish and the focus will start moving towards other aspects of recommender systems still greatly lacking.  Like building systems that facilitate discovery of new items.

Factoring in the temporal dimension was a big deal in the latter part of the competition.  Sometimes you’re just in the mood for something gloomy.  Or something funny.  Or something ridiculous.  The same movie may totally turn you off a week later.  No machine (biological or mechanical) can predict these swings of emotions in the near future, so why bother?  Flip that around and let’s find ways of improving the search for items matching our mood at the time.

A system that interactively elicits your mood and guides you to matching items would be incredibly useful, don’t you think?

Image representing Netflix as depicted in Crun...
Image via CrunchBase

It looks like some of the top players in the Netflix Prize competition have teamed up and finally broke the 10% improvement barrier.  I know I’m a few days late on this, though not because I didn’t see when it happened.  I’ve been battling an ear infection all week and it has left me dizzy, in pain, and with no energy when I get home from work.  I hesitated before even posting anything about this, since there is little I can add at this point that hasn’t already been said. I’ll just share a few thoughts and experiences for posterity and leave it at that.  I’m also going to eventually make the point that recommender systems are operating under a false assumption, if you read this all the way through. :)

I competed for the prize for a bit, trying out a few ideas with support vector machines and maximum margin matrix factorization [pdf] that never panned out.  We were getting about a 4% improvement over Cinematch, which put us way down the list.  Going further would mean investing a lot of effort into implementing other algorithms, working out the ensemble, etc., unless we came up with some novel algorithm that bridged the gap.  That didn’t seem likely, so I stopped working on it just after leaving school.  I learned a lot about machine learning, matrix factorization, and scaling thanks to the competition, so it was hardly a net loss for me.

The one thing I regret is that the prize encouraged me and my advisor to spend more effort on the competition than we should have, which in turn meant we didn’t spend more time working on something tangibly productive for research.  Bluntly put, I think if we hadn’t wasted so much time on the competition, we could have worked on a different research problem more likely to produce a paper.  The lack of published research on my CV was the main reason I didn’t move on to get my PhD at CMU (at least, that’s what I was told by those close to the decision).  Hindsight is 20/20, and at the time, the shining glory of winning a million bucks and fame was delicious.  It also seemed like we had ideas that “maybe kinda sorta” were going somewhere.  That turned out to not be the case, but when admissions committees look at research experience, negative results = no results.

Many people have lauded the competition by saying that it has encouraged research in collaborative filtering and brought public attention to the field.  I was one of those people.  Others have criticized it for not focusing more on what people actually care about when using recommender systems — getting something useful and having a good experience!  And yes, Daniel Lemire, I’m thinking of you. :)  But I’m convinced that Daniel is right.  I remember reading in the literature that a 10% improvement is about what’s needed for someone to actually be able to notice a difference in recommender systems.  So maybe people will notice a slight improvement in the Netflix recommendations if these ideas are ever implemented.  Which is another problem — most of the stuff that led to winning the prize is so computationally expensive, it’s not really feasible for production.  Netflix recently released some improvements, and I didn’t notice a damned thing.  They still recommended me Daft Punk’s Electroma, which was a mind-numbing screen-turd.  And I must have seen every good sci-fi movie ever made, because there are no more recommendations for me in that category.  I have trouble believing that.

The point of a recommender system really shouldn’t be just to guess what I might happen to rate something at a given time.  The fact that introducing time makes such a big difference in improving performance in the competition seems like a ginormous red flag to me.  Sure I can look back in time and say “on day X, people liked movies about killing terrorists.”  The qualifying set in the competition asked you to predict the rating for a movie by a user on a given date in the past.  Remember what I said about hindsight being 20/20?  How about you predict what I will rate a movie this coming weekend.  See the problem?

I will sound the HCIR trumpets and say that what recommender systems should really be looking at is improving exploration.  When I go looking for a movie to a watch, or a pair of shoes to buy, I already know what I like in general.  Let me pick a starting point and then show me useful ways of narrowing down my search to the cool thing I really want.  Clerk dogs is a good first step on this path, though I think we’re going to have to move away from curated knowledge before this is going to catch fire.

Maybe I have this all wrong.  Maybe we need to discard the notion of recommender systems, since they are operating under the wrong premise.  We don’t need a machine to recommend something it thinks we’ll like.  We need a machine that will help us discover something we’ll like.  We need to be making discovery engines.  (Replace recommender system with search engine in most of what I just said and you’ll find that I have really been sounding the HCIR trumpets.)

Reblog this post [with Zemanta]

Netflix UI Fail

Posted: 31 January 2009 in Uncategorized
Tags: , , ,

A simple if-check would take care of this.

Netflix user interface fail.

Netflix user interface fail.

I happened on clerk dogs, a new movie recommender, the other day.  They are still in beta and are missing data in many key areas of film, but they are definitely worth checking out.  Like Pandora, clerk dogs uses human editors to classify movies along several dimensions.  Indeed, the founder Stuart Skorman (also founder of calls it the movie genome project.  Of course, another movie recommender (also, still in beta) is using that term.  Stuart goes on to say:

We have designed this innovative search engine for the movie buffs who have seen so many movies that they’re having a hard time finding new ones (or old ones) that they will really love. I hope you find hundreds of great movies!



This is a problem I’ve been noticing with Netflix lately.  I would be pretty sure I’ve seen every sci-fi movie worth seeing that has been released if all I had to go on was Netflix’s recommendations.  I gave clerk dogs a shot, starting with my favorite movie.  They seem to have done a decent job with classifying Brazil and a number of the similar movies they have listed are indeed similar in many ways to it.  When I first visited the site, they showed the similar movies on a grid and said whether it was “more dark”, “less disturbing”, “more violent”, and so on.  If that functionality still exists, I can’t find it.

However, you can “Mash it” to find movies that fit your mood.  Pick your base movie and mash it.  Then change the sliding scale to decide what sort of differences you are looking for.  Can you say kickass?

I applaud clerk dogs for a job well done.  I’ve already found a number of movies that Netflix was hiding from me.  I added them to my Netflix queue though so I guess they are still benefitting.


Posted: 20 June 2008 in Uncategorized
Tags: , , , ,

This post contains no spoilers.

I rewatched Primer this week. I had seen it a couple years ago as one of the first movies I got from Netflix the first time I signed up. It was a successful recommendation. Since I was a kid, I have been totally intrigued with time travel and time travel movies. Time travel movies rank among my favorite films, like 12 Monkeys, Time Bandits, the Butterfly Effect, etc. Time travel books are great too, like The Time Traveler’s Wife. Thinking about the implications of being able to change things — and what happens when you do — filled many teenage hours.  An important part of my fascination then is resolving the conflicts inherent in time travel.  What happens if you change something in the past?  What are the rules in the movie or book?  Does the movie/book adhere to its own rules or do they screw up?

Primer is a time travel movie in a league of its own.  I think it’s pretty much impossible to fully grasp the first time through.  It is probably the most confusing movie I have ever seen (that is not “absurd” anyway).  It’s been bumping around in my head for the past couple years, driving me to see it again.  Mike D’Angelo in Esquire said it’s like “following the path of one blade on a high-speed ceiling fan.”  That’s a fairly accurate description.


This post is spoiler free.

I finally got to see Juno tonight. It’s been sitting at the top of my Netflix queue for nearly two months with a long wait. What a great movie! One of my favorite parts was the soundtrack. There were several great songs by Kimya Dawson (of the Moldy Peaches) and then a performance by the two leads of the Moldy Peaches song “Anyone Else But You.” The version sung in the movie is missing a few stanzas. My favorite of the missing ones is below (sung by Kimya):

“Up up down down left right left right B A start
Just because we use cheats
Doesn’t mean we’re not smart
I don’t see what anyone can see in anyone else
But you…”

Go geek references (and Thundercats)! And speaking of cheats, trying using that cheat code in Google Reader (minus the start button at the end of course).

And returning to Netflix: they are removing individual profiles from accounts as of September 1st. What a boneheaded, retardafreakin’ idea. Supposedly it will help them make the website better. I hope it’s a lot better since this change has me pissed.

Netflix Friends

Posted: 26 January 2008 in Uncategorized
Tags: , ,

Become my friend on Netflix.  I think it helps that you are actually already on Netflix.  :P

Greg Linden and Daniel Lemire have both written a little about the Netflix Prize and whether the systems that are doing the best are really worth anything. The KorBell system that recently won the Progress Prize consists of 107 different parts in an ensemble system (Note: the team of Bob Bell and Yehuda Koren at AT&T goes by BellKor and KorBell on the Netflix leaderboard). The paper is interesting for two reasons: the ensemble method being used and the fact that only about 3 or 4 of those components are doing the heavy lifting. Actually, I have no idea whether the actual ensemble algorithm they use would be especially interesting to anyone else, but as I have no experience with ensembles in this context, it was interesting to me. (more…)