[developers] PET regression testing

[Emily M. Bender]

Hi Bernd,

In the Grammar Matrix project we have a set of regression
tests that call [incr tsdb()] in a way similar to what it sounds
like you're doing (but through python), and then use the
fine system itself to compare on the dimension we're interested
in (MRSs).  I doubt our actual tests would be of interest
(for one thing, we're parsing with LKB right now, but hope
to transition to PET at some point soon!), but if the overall
structure is potentially informative, I can make it available to you.

I would be interested in providing a snapshot of the Wambaya
grammar and its associated (small) test suite (804 examples
only) as a test for you, if that would be useful.  It has some
interesting morphological complexity :)  The only issue I can
think of is how to keep the copyright notice/license file together
with the Wambaya data.  I guess it could be added to the
profiles/skeletons?

Emily

On Tue, Jul 19, 2011 at 2:56 AM, Bernd Kiefer <kiefer at dfki.de> wrote:
> Hi Francis,
>
> thanks for your support. I'd like to have a tool, maybe building on
> tsdb profiles, that fully automatically checks if PET still behaves
> "the same way" on a fixed set of grammar/corpora pairs. That means
> that there a lot of configurations that could potentially be tested,
> which ones are meaningful or subsumed by others has to be found out.
>
> Dimensions i can think of are:
>
> - packing / no packing (or even different variants)?
> - input modes? (including character conversion)
> - morphology
> - chart mapping
> - mrs construction
> - partial/full unpacking
>
> It would also be nice to have synthetic grammars to run more unit-like
> tests for, e.g., type hierarchy & unification, FS inheritance &
> unification, etc.
>
> Are there things i have missed?
>
> I've started with some Java code (because that's something that is
> available free-of-charge on every modern computer) that gives me
> efficient access to tsdb skeletons. What i need now is a set of
> grammars and correlated input items that i can process without
> additional machinery, the more, the better. For the first version, i'd
> suggest to just check the number of passive edges for full parsing
> without any packing and resource restrictions, which means that test items
> have to be of moderate size.
>
> The rest could then be added step by step (maybe some help out there?).
>
> Best,
>        Bernd
>
>
> On 08.07.2011 14:07, Francis Bond wrote:
>>
>> G'day,
>>
>> I am afraid I don't have a test harness in place, but would be willing
>> to invest some time in helping set one up.  I don't think I can be the
>> lead on this, but can definitely contribute test data.
>>
>>> sorry for coming back with the same things again and again, but i'm
>>> in desperate need for a means of doing regression tests with PET.
>>
>> What kinds of tests do you think we need?
>>
>>> In my PET source tree, there are quite a lot of changes lying around
>>> (among them an updated version of -tdsbdump, which might be interesting
>>> to Francis) which i don't dare checking in because i'm not sure if
>>> the old functionality is preserved.
>>>
>>> I'd be happy if somebody would give me an *easy* way to do that, or
>>> would be willing to do the tests, given she/he has already set up
>>> something, maybe for a different purpose.
>>>
>>> I hate to say that, but with the very limited amount of time i can
>>> invest in the development of PET, not having an automatic test harness
>>> that does free more human resources than it requires, brings my
>>> efforts almost to a halt.
>>>
>>> Any volunteers to work on that (maybe collaborately)?
>>>
>>> Enough whining for today,
>>
>> s/whining/pointing out the need for important infrastructure/
>>
>> and it won't be enough until we've done it.
>>
>
> --
> ----------------------------------------------------------------------
> Bernd Kiefer     DFKI GmbH,  Stuhlsatzenhausweg,  D-66123 Saarbruecken
> kiefer at dfki.de   +49-681/85775-5301 (phone)   +49-681/85775-5338 (fax)
> ----------------------------------------------------------------------
> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
> Trippstadter Strasse 122, D-67663 Kaiserslautern, Germany
> Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vor-
>                    sitzender), Dr. Walter Olthoff
> Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
> Amtsgericht Kaiserslautern, HRB 2313
>



-- 
Emily M. Bender
Associate Professor
Department of Linguistics
Check out CLMA on facebook! http://www.facebook.com/uwclma