[logon] The highest accuracy on the JHPSTG corpus is 40.6. Is this a bug?
Stephan Oepen
oe at ifi.uio.no
Tue Apr 21 19:58:20 CEST 2009
hi bill,
> I ran a set of parse ranking experiments on the JHPSTG corpus. I chose a
> reasonable set of learning parameters and ranged over the machine learning
> priors. The highest accuracy I got was 40.6. This seems very low, and way
> beneath the state of the art. Do I have a bug?
not sure, but that number does seem suspiciously low. i do not have
comparison figures for that exact configuration, but i ran one quick
experiment for the configuration we used for the IWPT 2007 paper; it
arrived at 52.5% exact match accuracy (for the configuration recorded
in `train.lisp': three-level grandparenting, no `active edges', and no
n-grams). this is still four percent points below what we saw at the
time of the IWPT paper, but i am willing to attribute that to changes
in the grammar (more coverage always brings more ambiguity).
from the configuration you sent, it seems you only varied estimation
hyper-parameters, but not grandparenting levels, n-gram size, et al.
it is, after all, quite possible that the many extra features in your
configuration degrade performance a lot, though i admit i would find
that surprising from earlier rounds of experiments.
best - oe
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++ Universitetet i Oslo (IFI); Boks 1080 Blindern; 0316 Oslo; (+47) 2284 0125
+++ CSLI Stanford; Ventura Hall; Stanford, CA 94305; (+1 650) 723 0515
+++ --- oe at ifi.uio.no; oe at csli.stanford.edu; stephan at oepen.net ---
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
More information about the logon
mailing list