It was a great homework assignment — one of the more interesting I’ve had in a long while. Kudos!

I tried some normalization, but there were definitely more things to try (and a lack of time to do it). I completely agree about the problems of raters using different mental scales and selectively rating certain movies. I have also heard of people taking time into account, but I can’t remember any details or anything that stood out to me as particularly cool. Time-based normalizations are not a bad idea at all, I wonder how much they would help..

What you would do if you were submitting to Netflix is this. Hold out some test cases, like the probe set they include. For each of your models, generate a prediction for test cases in the probe set. Let’s call these {M1, M2, …, Mn} for n models. Each model consists of the predictions for the test cases in the probe set. In a grid search, you would cycle through all possible values for the weights and find the best RMSE for the ensemble. So if you have a weight set W = {w1, w2, …, wn}, you would basically have n loops over the possible values of w (which you can probably restrict to the interval [-1,1], with the constraint that they all sum 1). So if you were combining two models:

for i = -1 to 1 step 0.1
  for j = 1 to size(probe_set)
    p[j] = i * M1[j] + (1 – i) * M2[j]

As you can see, each model you add increases the complexity by a factor O(ms) (where m is the number size of the probe set and s is the number of parameter values you are checking), so if you have 100 models, it will take forever this way: O(m*s^99). But you can apply some heuristics and limit the space you search.

One final note: You would only need to do this step once to find the weights to apply to each model. So as long as your models remain the same you don’t have to repeat this step. If you add or change a model, you have to reconstruct this.

There’s a lot you can do in scaling & normalizing the ratings — some users are more generous on average than others, for example, and taking that into account when you do predictions helps quite a bit. A 5 from someone who gives all 4′s and 5′s means something different than a 5 from a different user.

One thing I didn’t look into at all, but is likely a promising direction, is taking into account time. I would guess ratings come in bursts & someone’s mood at the moment heavily influences their rating generosity. I recently read a blog post on someone who’s doing exactly this with the netflix dataset with much success. (the link escapes me now…)

Good luck with your assignment.

3.47, 4.09, 2.75

With the weighting scheme I mentioned, where the weights are {0.45, 0.35, 0.2}, the blended rating would be 3.47 * 0.45 + 4.09 * 0.35 + 2.75 * 0.2 = 3.54. The weights have to sum to one, or else you would need to normalize the final score in some way.

This is just one way of doing it, but I think it’s fairly typical since it works well and is really easy to do a grid search over the weights on development data to optimize your parameters. It does however make your final model less explainable. Compare this odd blending of models to simple kNN where you can say, well I predicted this rating because k closest users said it should be this. In the real-life case above, a model that performs worse is given a higher weight than the best model. Why?