Comments on: MT Eval with Binary Comparisons http://mendicantbug.com/2008/05/12/mt-eval-binary-system-comps/ Wanderings into computational linguistics, science, social media and life... Mon, 23 Apr 2012 02:02:52 +0000 hourly 1 http://wordpress.com/ By: Top 25 Memes « The Mendicant Bug http://mendicantbug.com/2008/05/12/mt-eval-binary-system-comps/#comment-1279 Sat, 28 Feb 2009 05:33:20 +0000 http://ealdent.wordpress.com/?p=613#comment-1279 [...] me with this very problem and he brought up using binary comparisons. I have talked about binary comparisons when dealing with machine translation evaluation before, and many in the academic community have [...]

]]> By: Jason Adams http://mendicantbug.com/2008/05/12/mt-eval-binary-system-comps/#comment-818 Tue, 13 May 2008 12:01:59 +0000 http://ealdent.wordpress.com/?p=613#comment-818 Cool, thanks for the link, I wasn’t aware of it. It might turn into another blog post.. :)

The link was giving me an “Access Forbidden” error from my comp at home, but here is an alternate link: http://research.microsoft.com/~sdumais/ecir08-carteretteetal-final.pdf.

]]> By: Jon Elsas http://mendicantbug.com/2008/05/12/mt-eval-binary-system-comps/#comment-817 Tue, 13 May 2008 10:46:30 +0000 http://ealdent.wordpress.com/?p=613#comment-817 Jason – some recent work on this type of evaluation for IR was presented at ECIR: B. Carterette, P.N. Bennett, D.M. Chickering, S.T. Dumais (2008). Here or There: Preference Judgments for Relevance . (http://ciir.cs.umass.edu/~carteret/ecir08.pdf)

These pairwise evaluations tend to look a lot like using something like Kendell’s tau as an objective function instead of squared error. The interesting things, at least from the IR standpoint, is when the mis-ranked instance pairs should be weighted higher for items near the top of the ranked list.

]]>