Difference between revisions of "User:Pmodi/GSOC 2020 proposal: Hindi-Punjabi/progress"

From Apertium
Jump to navigation Jump to search
(Created page with "Progress on Automatic_blank_handling ==Current task== ===lttoolbox=== * Make lt-proc correctly disperse inline bl...")
 
Line 1: Line 1:
  +
Progress on [[https://wiki.apertium.org/wiki/User:Pmodi/GSOC_2020_proposal:_Hindi-Punjabi]]
−
Progress on [[Ideas_for_Google_Summer_of_Code/Automatic_blank_handling|Automatic_blank_handling]]
  
   
−
==Current task==
 +
==Current task(Community bonding Week 1)==
  +
===Finalising Frequency lists & Learning Persian script===
−
===lttoolbox===
  
  +
* DONE - Frequency lists using WikiExtractor on latest dump.
−
* Make lt-proc correctly disperse inline blanks onto each lexical unit until the next <code><nowiki>[</nowiki></code>
  
  +
* DONE - Read wiki on generating freq lists and monolingual building dictionaries.
−
* Task: Create a pull request to https://github.com/unhammer/lttoolbox/ with tests in https://github.com/unhammer/lttoolbox/tree/master/tests/lt_proc
  
  +
* IN PROGRESS - Finish scraping more texts and merge results with the current freq. list.
  +
* IN PROGRESS - Practicing basic understanding of Persian script.
   
   
  +
==TODO(Next week - Community Bonding Week 2)==
−
==TODO==
  
  +
* Fix bidirectional translation i.e. pan->hin
−
===hfst===
  
  +
* Fix transfer rules for postpositions.
−
* Make hfst-proc correctly disperse inline blanks onto each lexical unit until the next <code><nowiki>[</nowiki></code>
  
  +
* Start work on function words addition to dictionary.
−
* Task: Create a pull request to https://github.com/hfst/hfst/ with tests in https://github.com/hfst/hfst/tree/master/test/tools/
  
−  
−
===transfer (non-chunking)===
  
−
* Test if current transfer.cc handles non-chunking/single-stage transfer correctly, if not, fix
  
−
* Task: PR to https://github.com/unhammer/apertium/ with tests showing working transfer.cc for single-stage/non-chunking transfer, with inline vs block-level blank handling and test that rules using misnumbered/missing b-elements should not mess up formatting
  
−  
−
===postchunk===
  
−
(Should be done after interchunk is complete)
  
−  
−
* Task: PR to https://github.com/unhammer/apertium/ including tests showing working postchunk blank handling – test that rules using misnumbered/missing b-elements should not mess up formatting
  
−  
−
===etc===
  
−
* Ensure all other modules are fine with the new format for inline blanks (e.g. cg-proc)
  
−
* Work on other deformatters (mediawiki? latex?)
  
−  
−
==Done==
  
−
(Some of these are from coding challenges)
  
−  
−
===deformatting prototypes===
  
−
# Make the HTML format handler <code>apertium-deshtml</code> turn "&lt;i&gt;foo &lt;b&gt;bar&lt;/b&gt;&lt;/i&gt;" into "<nowiki>[{&lt;i&gt;}]foo [{&lt;i&gt;&lt;b&gt;}]bar</nowiki>"
  
−
#* Code at https://github.com/SilentFlame/apertium/blob/master/challenge-1.cpp
  
−
#* make <code>apertium-deshtml</code> *not* wrap tags like {{tag|p}} or {{tag|div}} in <code>{}</code> (ie. only for inline tags)
  
−
#* Code at https://github.com/SilentFlame/apertium/blob/master/challenge-2.cpp
  
−  
−
===pretransfer===
  
−
* Make pretransfer disperse tags when splitting lexical units https://github.com/unhammer/apertium/commit/39bd7d9fa45c64586d3a9b0f1a7df89e7d007c1a , code cleanup:
  
−
#* Fork https://github.com/unhammer/apertium and check out and compile the <code>master</code> branch
  
−
#* then in a different folder, do <code>git clone -b blank-handling https://github.com/junaidiiith/apertium</code>
  
−
#* from junaidiiith/blank-handling, copy over the changes that were made there to apertium_pretransfer.cc into your fork of unhammer/apertium, along with the pretransfer tests
  
−
#* ensure tests pass
  
−
#* PR at https://github.com/unhammer/apertium/pull/4
  
−  
−
===transfer (chunker)===
  
−
# Fix a memory bug
  
−
#* uncommenting apertium/transfer.cc:1259 <code> // delete[] format;</code> in the blank handling branch leads to a double-free – find out why and ensure we're correctly releasing memory
  
−
#* Install valgrind from your package manager or http://valgrind.org/, then compile your program with -O0 -g3, then run <code>valgrind -v --leak-check=full apertium/apertium-transfer</code> and read the output
  
−  
−
===Interchunk===
  
−
Interchunk needs to ignore the "pos" argument to b elements, and output each superblank exactly once, preferably where the rule has a b element (if there are not enough b's, output the rest at the end of the rule).
  
−
Interchunk shouldn't have to deal with wordblanks, since we can't look inside chunks when in interchunk.
  
−  
−
# Apply changes to transfer.cc to interchunk.cc
  
−
#* Check <code>git clone -b blank-handling https://github.com/unhammer/apertium</code>
  
−
#* Apply the <code>git diff 4c7c4f8f1b..2025182991</code> from transfer.cc to interchunk.cc
  
−
#* Try to make it compile and run – report things that didn't seem to have a 1-1 correspondence
  
−
#* Write tests for interchunk, like those for transfer at https://github.com/unhammer/apertium/tree/blank-handling/tests
  
−  
−
===Deformatters===
  
−
* Complete prototype HTML deformatters
  
−
** Current prototype code at https://github.com/junaidiiith/apertium / https://github.com/junaidiiith/Apertium_Code and https://github.com/SilentFlame/apertium/
  
−
** Task: Create a clean pull request to https://github.com/unhammer with HTML deformatter and reformatter, including tests
  
−  
−
===Reformatters===
  
−  
−
* Make reformat turn inline-blanks back into real tags
 
−
** <nowiki>[{&lt;i&gt;}]foo [{&lt;i&gt;&lt;b&gt;}]bar</nowiki> should become &lt;i&gt;foo&lt;/i&gt; &lt;i&gt;&lt;b&gt;bar&lt;/b&gt;&lt;/i&gt;
  
−
** prototypes exist for this in https://github.com/junaidiiith/apertium / https://github.com/junaidiiith/Apertium_Code
  

Revision as of 20:46, 9 May 2020

Progress on [[1]]

Current task(Community bonding Week 1)

Finalising Frequency lists & Learning Persian script

  • DONE - Frequency lists using WikiExtractor on latest dump.
  • DONE - Read wiki on generating freq lists and monolingual building dictionaries.
  • IN PROGRESS - Finish scraping more texts and merge results with the current freq. list.
  • IN PROGRESS - Practicing basic understanding of Persian script.


TODO(Next week - Community Bonding Week 2)

  • Fix bidirectional translation i.e. pan->hin
  • Fix transfer rules for postpositions.
  • Start work on function words addition to dictionary.
Retrieved from "https://wiki.apertium.org/w/index.php?title=User:Pmodi/GSOC_2020_proposal:_Hindi-Punjabi/progress&oldid=72170"