http://mendicantbug.com/2007/12/14/netflix-prize-good-science-or-not/ Wanderings into computational linguistics, science, social media and life... Fri, 09 Jan 2009 13:22:08 +0000 http://wordpress.org/?v=MU hourly 1 http://mendicantbug.com/2007/12/14/netflix-prize-good-science-or-not/#comment-815 MT Eval with Binary Comparisons « The Mendicant Bug Tue, 13 May 2008 04:53:56 +0000 http://mendicantbug.com/2007/12/14/netflix-prize-good-science-or-not/#comment-815 [...] 12 May 2008 in collaborative filtering, computational linguistics, machine learning, machine translation, machine translation evaluation, mt, mt eval, rankings, recommender systems The standard way of doing human evaluations of machine translation (MT) quality for the past few years has been to have human judges grade each sentence of MT output against a reference translation on measures of adequacy and fluency.  Adequacy is the level at which the translation conveys the information contained in the original (source language) sentence.  Fluency is the level at which the translation conforms to the standards of the target language (in most cases, English).  The judges give each sentence a score for both in the range of 1-5, similar to a movie rating.   It became apparent early on that not even humans correlate well with each other.  One judge may be sparing with the number of 5’s he gives out, while another may give them freely.  The same problem crops up in recommender systems, which I have talked about in the past. [...] [...] 12 May 2008 in collaborative filtering, computational linguistics, machine learning, machine translation, machine translation evaluation, mt, mt eval, rankings, recommender systems The standard way of doing human evaluations of machine translation (MT) quality for the past few years has been to have human judges grade each sentence of MT output against a reference translation on measures of adequacy and fluency.  Adequacy is the level at which the translation conveys the information contained in the original (source language) sentence.  Fluency is the level at which the translation conforms to the standards of the target language (in most cases, English).  The judges give each sentence a score for both in the range of 1-5, similar to a movie rating.   It became apparent early on that not even humans correlate well with each other.  One judge may be sparing with the number of 5’s he gives out, while another may give them freely.  The same problem crops up in recommender systems, which I have talked about in the past. [...]

]]> http://mendicantbug.com/2007/12/14/netflix-prize-good-science-or-not/#comment-373 Ian’s Blog » Netflix Prize - is RMSE a good measurement? Mon, 17 Dec 2007 18:20:06 +0000 http://mendicantbug.com/2007/12/14/netflix-prize-good-science-or-not/#comment-373 [...] I think a user might have a hard time noticing such a difference in performance. I’m not the first to express concerns of this [...] [...] I think a user might have a hard time noticing such a difference in performance. I’m not the first to express concerns of this [...]

]]> http://mendicantbug.com/2007/12/14/netflix-prize-good-science-or-not/#comment-367 Chris Sat, 15 Dec 2007 16:50:55 +0000 http://mendicantbug.com/2007/12/14/netflix-prize-good-science-or-not/#comment-367 Bob Carpenter at Alia-i and Hal Daume at his NLPer blog also blogged about this. I'll leave the quality assessment up to you guys since you're far better qualified, but I'll say this: rarely does anything involving NLP get this kind of press. There's no such thing as bad press, right? Bob Carpenter at Alia-i and Hal Daume at his NLPer blog also blogged about this. I’ll leave the quality assessment up to you guys since you’re far better qualified, but I’ll say this: rarely does anything involving NLP get this kind of press. There’s no such thing as bad press, right?

]]>