Just so I'm sure, the IWPT paper is Zhang et al. 2007 "Efficiency in Unification-Based N-Best Parsing," correct?<div><br></div><div>I'll try my experiment with the parameters in $logon/lingo/redwoods/train.lisp as a sanity check.</div> <div><br><div class="gmail_quote">On Tue, Apr 21, 2009 at 10:58 AM, Stephan Oepen <span dir="ltr"><<a href="mailto:oe@ifi.uio.no">oe@ifi.uio.no</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"> hi bill,<br> <div class="im"><br> > I ran a set of parse ranking experiments on the JHPSTG corpus. I chose a<br> > reasonable set of learning parameters and ranged over the machine learning<br> > priors. The highest accuracy I got was 40.6. This seems very low, and way<br> > beneath the state of the art. Do I have a bug?<br> <br> </div>not sure, but that number does seem suspiciously low. i do not have<br> comparison figures for that exact configuration, but i ran one quick<br> experiment for the configuration we used for the IWPT 2007 paper; it<br> arrived at 52.5% exact match accuracy (for the configuration recorded<br> in `train.lisp': three-level grandparenting, no `active edges', and no<br> n-grams). this is still four percent points below what we saw at the<br> time of the IWPT paper, but i am willing to attribute that to changes<br> in the grammar (more coverage always brings more ambiguity).<br> <br> from the configuration you sent, it seems you only varied estimation<br> hyper-parameters, but not grandparenting levels, n-gram size, et al.<br> it is, after all, quite possible that the many extra features in your<br> configuration degrade performance a lot, though i admit i would find<br> that surprising from earlier rounds of experiments.<br> <br> best - oe<br> <br> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br> +++ Universitetet i Oslo (IFI); Boks 1080 Blindern; 0316 Oslo; (+47) 2284 0125<br> +++ CSLI Stanford; Ventura Hall; Stanford, CA 94305; (+1 650) 723 0515<br> +++ --- <a href="mailto:oe@ifi.uio.no">oe@ifi.uio.no</a>; <a href="mailto:oe@csli.stanford.edu">oe@csli.stanford.edu</a>; <a href="mailto:stephan@oepen.net">stephan@oepen.net</a> ---<br> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br> </blockquote></div><br><br clear="all"><br>-- <br>W.P. McNeill<br><a href="http://staff.washington.edu/billmcn/index.shtml">http://staff.washington.edu/billmcn/index.shtml</a><br>Sent from Seattle, WA, United States </div>