[developers] PET regression testing

[Bernd Kiefer]

Hi Francis,

thanks for your support. I'd like to have a tool, maybe building on
tsdb profiles, that fully automatically checks if PET still behaves
"the same way" on a fixed set of grammar/corpora pairs. That means
that there a lot of configurations that could potentially be tested,
which ones are meaningful or subsumed by others has to be found out.

Dimensions i can think of are:

- packing / no packing (or even different variants)?
- input modes? (including character conversion)
- morphology
- chart mapping
- mrs construction
- partial/full unpacking

It would also be nice to have synthetic grammars to run more unit-like
tests for, e.g., type hierarchy & unification, FS inheritance &
unification, etc.

Are there things i have missed?

I've started with some Java code (because that's something that is
available free-of-charge on every modern computer) that gives me
efficient access to tsdb skeletons. What i need now is a set of
grammars and correlated input items that i can process without
additional machinery, the more, the better. For the first version, i'd
suggest to just check the number of passive edges for full parsing
without any packing and resource restrictions, which means that test 
items have to be of moderate size.

The rest could then be added step by step (maybe some help out there?).

Best,
	Bernd


On 08.07.2011 14:07, Francis Bond wrote:
> G'day,
>
> I am afraid I don't have a test harness in place, but would be willing
> to invest some time in helping set one up.  I don't think I can be the
> lead on this, but can definitely contribute test data.
>
>> sorry for coming back with the same things again and again, but i'm
>> in desperate need for a means of doing regression tests with PET.
>
> What kinds of tests do you think we need?
>
>> In my PET source tree, there are quite a lot of changes lying around
>> (among them an updated version of -tdsbdump, which might be interesting
>> to Francis) which i don't dare checking in because i'm not sure if
>> the old functionality is preserved.
>>
>> I'd be happy if somebody would give me an *easy* way to do that, or
>> would be willing to do the tests, given she/he has already set up
>> something, maybe for a different purpose.
>>
>> I hate to say that, but with the very limited amount of time i can
>> invest in the development of PET, not having an automatic test harness
>> that does free more human resources than it requires, brings my
>> efforts almost to a halt.
>>
>> Any volunteers to work on that (maybe collaborately)?
>>
>> Enough whining for today,
>
> s/whining/pointing out the need for important infrastructure/
>
> and it won't be enough until we've done it.
>

-- 
----------------------------------------------------------------------
Bernd Kiefer     DFKI GmbH,  Stuhlsatzenhausweg,  D-66123 Saarbruecken
kiefer at dfki.de   +49-681/85775-5301 (phone)   +49-681/85775-5338 (fax)
----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Trippstadter Strasse 122, D-67663 Kaiserslautern, Germany
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vor-
                     sitzender), Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313