Google Summer of Code 2021 proposal draft
Name: Daniel Swanson
This project is really 3 projects UNIfied only by the fact that they all start with UNI:
This portion of the project is Ideas for Google Summer of Code/Robust tokenisation. All string processing would be transferred to ICU and all inputs would be normalized. A program would be inserted at the beginning of the pipeline which normalizes incoming text and compilers would emit warnings for non-normalized input.
The normalization will likely be a modified version of NFC. See https://github.com/apertium/organisation/issues/24 for discussion.
Our regression tests are few and far between and almost none of them are run automatically. This part of the project would be the creation of a Grand Unified Unit-Testing Framework™, probably mainly derived from https://github.com/TinoDidriksen/regtest (though I have included time in the schedule for bikeshedding the exact feature set).
I would also plan to enable this testing on all existing repositories unless the maintainers really didn't want it. This will entail converting any existing tests from whatever state they are currently in.
UNIversal Dependencies-Based Structural Transfer
I have worked out in broad outlines how to get apertium-recursive's rtx-proc to parse a dependency grammar. This third phase would be implementing that concept and seeing if it's actually useful.
If it works, we'll have a way of creating UD parsers (though I suppose we already have that with CG) though I think the more useful thing is that I have some ideas for a UD generator uses the rules for the parser and uses them to reorder a set of relations. The result of that would be a semi-monolingualized syntactic transfer system, where translation pairs could immediately start using the UD parsers in the monolingual repos and would only need rules for particular problematic constructions.
There's probably also cool stuff we could do with treebanks.
I am a senior at Swarthmore College studying math and linguistics. I have a lot of experience with Python and a decent knowledge of C++. I am a native speaker of English and can read Spanish and Biblical Hebrew.
Repositories I maintain:
Repositories I have been very involved in:
Preliminary UD work: https://github.com/mr-martian/ud-experiments/tree/master
Status as of Feb 9: working Python compiler from minimal formalism to trx.
Feb 23: the coding challenge listed at Ideas_for_Google_Summer_of_Code/Robust_tokenisation can be found here and a draft of the file structure for the unit-testing framework can be found at User:Popcorndude/Unit-Testing.
Mar 2: Unit-testing draft is probably nearly ready to send to the mailing list for discussion.
The following work plan is written assuming the number of hours that GSoC recommends (17 or so per week), but it is very likely that having fun or not having much else to do will result in me putting in substantially more time than that. Either this will balance out my inevitable underestimate of the complexity of the task, or I'll end up way ahead and start inventing new tasks for myself.
|Community Bonding Period
May 17-June 5
|Finalize implementation plan for Unicode support||
||Draft of string-handling guidelines (= updated version of Code style)|
||PR on lttoolbox repository|
||PR on apertium repository|
|Convert remaining tools||
||PRs on apertium-separable, apertium-anaphora, apertium-recursive, and apertium-lex-tools|
June 27-July 3
||Unicode transition branches ready for merging|
|Propose Grand Unified Unit-Testing Framework™||
||Nothing terribly concrete|
|midterm evaluation||Full Unicode support||Full Unicode support with proper tokenization and normalization in all native pipeline modules and coding guidelines for future contributors|
|Finish initial UD compiler||
||Preliminary UD-based transfer system|
|Implement Grand Unified Unit-Testing Framework™||
||Working unit-testing framework|
||Test reports everywhere|
|Further UD experimentation||
||Preliminary evaluation of the effectiveness of UD transfer|
|Unknown||What I write here will have little bearing on what actually happens and also by not making a plan for this week, I ensure that my overall plan can be at most 90% wrong.||Unknown|
|final evaluation||Project done||Unicode support with accompanying developer documentation, unit-testing framework for all language and pair repositories, and some potentially useful messing around with UD-based transfer|
I have no other commitments this summer and would be able to work on this project full-time.