[logon] The highest accuracy on the JHPSTG corpus is 40.6. Is this a bug?

Stephan Oepen oe at ifi.uio.no
Tue Apr 21 19:58:20 CEST 2009


hi bill,

> I ran a set of parse ranking experiments on the JHPSTG corpus.  I chose a
> reasonable set of learning parameters and ranged over the machine learning
> priors.  The highest accuracy I got was 40.6.  This seems very low, and way
> beneath the state of the art.  Do I have a bug?

not sure, but that number does seem suspiciously low.  i do not have
comparison figures for that exact configuration, but i ran one quick
experiment for the configuration we used for the IWPT 2007 paper; it
arrived at 52.5% exact match accuracy (for the configuration recorded
in `train.lisp': three-level grandparenting, no `active edges', and no
n-grams).  this is still four percent points below what we saw at the
time of the IWPT paper, but i am willing to attribute that to changes
in the grammar (more coverage always brings more ambiguity).

from the configuration you sent, it seems you only varied estimation
hyper-parameters, but not grandparenting levels, n-gram size, et al.
it is, after all, quite possible that the many extra features in your
configuration degrade performance a lot, though i admit i would find
that surprising from earlier rounds of experiments.

                                                        best  -  oe

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++ Universitetet i Oslo (IFI); Boks 1080 Blindern; 0316 Oslo; (+47) 2284 0125
+++     CSLI Stanford; Ventura Hall; Stanford, CA 94305; (+1 650) 723 0515
+++       --- oe at ifi.uio.no; oe at csli.stanford.edu; stephan at oepen.net ---
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++



More information about the logon mailing list