User:Ggregori - Revision history

Francis Tyers: /* Final report */

2011-08-27T05:10:25Z

Final report

Ggregori at 14:57, 26 August 2011

2011-08-26T14:57:36Z

Ggregori at 14:56, 26 August 2011

2011-08-26T14:56:12Z

Ggregori at 09:10, 22 August 2011

2011-08-22T09:10:55Z

Ggregori at 09:57, 15 August 2011

2011-08-15T09:57:12Z

Ggregori at 09:22, 8 August 2011

2011-08-08T09:22:21Z

Ggregori at 16:14, 2 August 2011

2011-08-02T16:14:53Z

Ggregori at 11:29, 2 August 2011

2011-08-02T11:29:32Z

Ggregori: /* GSoC 2011 */

2011-07-28T19:31:08Z

GSoC 2011

Ggregori at 11:21, 25 July 2011

2011-07-25T11:21:21Z

@@ Line 148: / Line 148: @@
 This project aimed at creating a transfer system with two main components: a compiler and a virtual machine. The compiler generates a pseudo-assembly representation of a transfer rule file (.t1x, .t2x or .t3x) and, then, the stack-based virtual machine is able to read and execute the instructions generated.
-The project also pretended to create two versions of this new transfer system. One which would become the foundation of an easy to modify and experiment with system and would be implemented in Python. And another one which implementation would be more focused on performance and would be implemented in C++. Also, a lot of last year's work was intended to be reused and refined to some extent, for example, I ended up making the instruction set simpler and more consistent<ref>http://wiki.apertium.org/wiki/VM_for_transfer</ref>.
+The project also aimed to create two versions of this new transfer system. One which would become the foundation of an easy to modify and experiment with system and would be implemented in Python. And another one which implementation would be more focused on performance and would be implemented in C++. Also, a lot of last year's work was intended to be reused and refined to some extent, for example, I ended up making the instruction set simpler and more consistent<ref>http://wiki.apertium.org/wiki/VM_for_transfer</ref>.
 After three months of hard work, both versions are completed and can be downloaded from their respective repositories<ref>https://github.com/ggm/vm-for-transfer</ref><ref>https://github.com/ggm/vm-for-transfer-cpp</ref>. In addition, a small new feature, not initially planed, was added during the development of the Python version and consists of a gdb-like debugger for the virtual machine. In hindsight, most of the difficulties of the project weren't found on the programming side but on discovering exactly how each part of the transfer system works and, although a bit frustrating at some points, it has been a rewarding experience.

@@ Line 142: / Line 142: @@
 | run-tests-vm.sh || <code> 37.337s </code> || <code> 11.702s </code> || 3.2x
 |-
+|}
 === Final report ===

@@ Line 142: / Line 142: @@
 | run-tests-vm.sh || <code> 37.337s </code> || <code> 11.702s </code> || 3.2x
 |-
+=== Final report ===
+This project aimed at creating a transfer system with two main components: a compiler and a virtual machine. The compiler generates a pseudo-assembly representation of a transfer rule file (.t1x, .t2x or .t3x) and, then, the stack-based virtual machine is able to read and execute the instructions generated.
+The project also pretended to create two versions of this new transfer system. One which would become the foundation of an easy to modify and experiment with system and would be implemented in Python. And another one which implementation would be more focused on performance and would be implemented in C++. Also, a lot of last year's work was intended to be reused and refined to some extent, for example, I ended up making the instruction set simpler and more consistent<ref>http://wiki.apertium.org/wiki/VM_for_transfer</ref>.
+After three months of hard work, both versions are completed and can be downloaded from their respective repositories<ref>https://github.com/ggm/vm-for-transfer</ref><ref>https://github.com/ggm/vm-for-transfer-cpp</ref>. In addition, a small new feature, not initially planed, was added during the development of the Python version and consists of a gdb-like debugger for the virtual machine. In hindsight, most of the difficulties of the project weren't found on the programming side but on discovering exactly how each part of the transfer system works and, although a bit frustrating at some points, it has been a rewarding experience.
+Finally, regarding performance the C++ version is faster than the Python one, as was expected<ref>Weekly reports (week 13), http://wiki.apertium.org/wiki/User:Ggregori#Coding_Period</ref>. Unfortunately, it is too soon to compare the vm with Apertium's transfer for several reasons but, mainly, because Apertium's one has a lot of optimizations which are lacking, for now, in the virtual machine (preprocessed and binarised trie, stringCache...). Therefore, currently, Apertium's transfer is faster but I plan to continue improving and following the path this project has started to see if better performance can be achieved.
+==== References ====
+<references/>
 ----

@@ Line 15: / Line 15: @@
 === TODO list ===
-*Port modules: <s>callstack</s>, debugger, interpreter, <s>systemtrie</s> and <s>transferword</s>.
+*Port modules: debugger.
-*Pass all the tests of the Python version to make sure it behaves likewise.
 === Weekly reports ===
@@ Line 127: / Line 126: @@
 *Implemented the callstack module extracting a method to better suit C++.
-'''Week 13''' - (15/08 - 22/08):
+'''Week 13''' - (15/08 - 22/08): This week I finished porting the VM to C++, meeting the project deadline and its four deliverables requirement.
+*Implemented all the interpreter module.
+*Fixed the locale setup memory leak.
+*Some refactoring: created a class hierarchy for lexical units with a common base.
+*Ported the remaining methods in the vm.cc involving pattern matching and such.
+*Fixed everything until all vm tests passed exactly as in the Python version.
+*As it was expected the C++ version is faster than the Python one, not only due to the language choice but because the Python version has been programmed more to be an easy to understand reference than a fast one. In addition, there is a lot of room for performance improvement, for example implementing the binary format for the vm.
+Just to show some numbers:
+{| class="wikitable" border="1"
+! Test !! Python version !! C++ version !! Improvement
+|-
+| run-tests-compiler.sh || <code> 5.372s </code> || <code> 0.968s </code> || 5.5x
+|-
+| run-tests-vm.sh || <code> 37.337s </code> || <code> 11.702s </code> || 3.2x
+|-
 ----

@@ Line 15: / Line 15: @@
 === TODO list ===
-*Port modules: callstack, debugger, interpreter, systemtrie and transferword.
+*Port modules: <s>callstack</s>, debugger, interpreter, <s>systemtrie</s> and <s>transferword</s>.
 *Pass all the tests of the Python version to make sure it behaves likewise.
@@ Line 122: / Line 122: @@
 *I have also been thinking a lot about the design and how to improve some components like the transferword module.
+'''Week 12''' - (08/08 - 14/08): A lot of time went on redesigning the transfer words module, but as I need time to measure things and make decisions based on data and there isn’t much time left, I am going to just port the left code and when everything is finished I’ll start improving it. Only the interpreter and some vm methods left to have a fully working vm in C++.
-'''Week 12''' - (08/08 - 14/08):
+*Implemented the entire transferwords module changing it to only parse the content of words when strictly necessary, therefore improving performance.
+*Implemented all the systemtrie module as in the Python version.
+*Implemented the callstack module extracting a method to better suit C++.
+'''Week 13''' - (15/08 - 22/08):
 ----

@@ Line 15: / Line 15: @@
 === TODO list ===
+*Port modules: callstack, debugger, interpreter, systemtrie and transferword.
-*Start porting the VM to C++.
 *Pass all the tests of the Python version to make sure it behaves likewise.
@@ Line 115: / Line 115: @@
 *Adapted the test script, test data and test expected output and passed all tests.
+'''Week 11''' - (01/08 - 07/08): This week I started porting the vm to c++, as with the compiler the start was slow but at the end I have ported several classes and more importantly have reimplemented and plan to change several components to improve their performance.
-'''Week 11''' - (01/08 - 07/08):
+*Solved the problem with the case policy in the postchunk.
+*Ported several classes:
+**Main program, VM, scope, loader, exceptions...
+*Added lazy-loading to the loader, so that rules and macros are only preloaded. This means that the first time, and not until then, a rule or a macro is called, the loader will convert them to the internal vm representation and process them properly.
+*I have also been thinking a lot about the design and how to improve some components like the transferword module.
+'''Week 12''' - (08/08 - 14/08):
 ----

@@ Line 15: / Line 15: @@
 === TODO list ===
-*Sergio has to explain me how the case policy is applied in the postchunk.
+*Find out if there are rules interfering with how the case policy is applied in the postchunk.
+*Start porting the VM to C++.
-*Create the C++ repository and start with the basic structure.
+*Pass all the tests of the Python version to make sure it behaves likewise.
-*Start porting the Compiler to C++.
-*Pass all the tests of the Python version to make sure it likewise.
 === Weekly reports ===
@@ Line 112: / Line 111: @@
 *Added messages “Path to rule blocked” and tested them with existing transfer files.
+'''Week 10''' - (25/07 - 31/07): This week I ported the compiler to C++, although there are some small things to refine, it works exactly the same as the Python one and passes all the tests.
-'''Week 10''' - (25/07 - 01/08):
+*Fixed and refactored some small things in the Python compiler.
+*Ported the entire compiler to C++, including adapting some things to the xml parser used in Apertium which worked slightly different than the Python one.
+*Adapted the test script, test data and test expected output and passed all tests.
+'''Week 11''' - (01/08 - 07/08):
 ----

@@ Line 10: / Line 10: @@
 [[User:Ggregori/Application|VM for the transfer module - Application]]
-'''Github repository:''' [https://github.com/ggm/vm-for-transfer]
+'''Github repository (python):''' [https://github.com/ggm/vm-for-transfer]
+'''Github repository (c++):''' [https://github.com/ggm/vm-for-transfer-cpp]
 === TODO list ===

@@ Line 13: / Line 13: @@
 === TODO list ===
+*Sergio has to explain me how the case policy is applied in the postchunk.
-*Implement blanks as talked with Sergio.
+*Create the C++ repository and start with the basic structure.
-*Try implementing the trie as an external "ROM".
+*Start porting the Compiler to C++.
-*Implement a script to test the vm vs Apertium with large articles.
+*Pass all the tests of the Python version to make sure it likewise.
 === Weekly reports ===
@@ Line 101: / Line 102: @@
 *Updated the wiki with more explanation and examples.
+'''Week 9''' - (18/07 - 24/07): After evaluation I had to finish some things Sergio told me and experimented some other. The bad news are that the experiments weren't successful, on the other hand I implemented blanks and fixed what was different from Apertium (I spent a lot of hours debugging some corner cases).
-'''Week 9''' - (18/07 - 24/07):
+*Finally, implemented blanks and superblanks, although it isn't the exact same behaviour as Apertium, it has some advantages and maybe some drawbacks. We'll have to look more into it, when gsoc finishes.
+*Tried to dump the trie to a file with pickle using every protocol available and the c extension but is still too slow and too much memory. Sergio wanted to expend a weekend improving the trie’s data structure, so I’ll continue with my project for now.
+*Did some testing with full articles, fixing all discrepancies I found (mainly blanks and some corner cases like passing the pos=0 as a macro parameter, in the postchunk stage, and then using an storecl instruction).
+*Fixed the trie for patterns like "to<pr>" and "*<pr>" with input "in order to<pr>" which has to be accepted only by the second one.
+*Added the remaining features to the debugger, only remaining linking to the transfer file right now.
+*Added messages “Path to rule blocked” and tested them with existing transfer files.
+'''Week 10''' - (25/07 - 01/08):
 ----