[logon] The highest accuracy on the JHPSTG corpus is 40.6. Is this a bug?

W.P. McNeill (UW) billmcn at u.washington.edu
Wed Apr 22 00:27:25 CEST 2009


Just so I'm sure, the IWPT paper is Zhang et al. 2007 "Efficiency in
Unification-Based N-Best Parsing," correct?
I'll try my experiment with the parameters in
$logon/lingo/redwoods/train.lisp as a sanity check.

On Tue, Apr 21, 2009 at 10:58 AM, Stephan Oepen <oe at ifi.uio.no> wrote:

> hi bill,
>
> > I ran a set of parse ranking experiments on the JHPSTG corpus.  I chose a
> > reasonable set of learning parameters and ranged over the machine
> learning
> > priors.  The highest accuracy I got was 40.6.  This seems very low, and
> way
> > beneath the state of the art.  Do I have a bug?
>
> not sure, but that number does seem suspiciously low.  i do not have
> comparison figures for that exact configuration, but i ran one quick
> experiment for the configuration we used for the IWPT 2007 paper; it
> arrived at 52.5% exact match accuracy (for the configuration recorded
> in `train.lisp': three-level grandparenting, no `active edges', and no
> n-grams).  this is still four percent points below what we saw at the
> time of the IWPT paper, but i am willing to attribute that to changes
> in the grammar (more coverage always brings more ambiguity).
>
> from the configuration you sent, it seems you only varied estimation
> hyper-parameters, but not grandparenting levels, n-gram size, et al.
> it is, after all, quite possible that the many extra features in your
> configuration degrade performance a lot, though i admit i would find
> that surprising from earlier rounds of experiments.
>
>                                                        best  -  oe
>
>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> +++ Universitetet i Oslo (IFI); Boks 1080 Blindern; 0316 Oslo; (+47) 2284
> 0125
> +++     CSLI Stanford; Ventura Hall; Stanford, CA 94305; (+1 650) 723 0515
> +++       --- oe at ifi.uio.no; oe at csli.stanford.edu; stephan at oepen.net ---
>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>



-- 
W.P. McNeill
http://staff.washington.edu/billmcn/index.shtml
Sent from Seattle, WA, United States
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.emmtee.net/archives/logon/attachments/20090421/f99772dc/attachment.html>


More information about the logon mailing list