Enhancing Translation Quality through Hybrid Machine Translation Techniques

Searching for the Best Translation Combination Mat ss Rikters Darba vad t ja: Dr. Dat., prof. Inguna Skadi a Doktorant ras semin rs R ga, Latvija 12. oktobris 2016

Contents Hybrid Machine Translation Multi-System Hybrid MT Simple combining of translations Combining full whole translations Combining translations of sentence chunks Combining translations of linguistically motivated chunks Searching for the best translation combination Other work Future plans

Hybrid Machine Translation Statistical rule generation Rules for RBMT systems are generated from training corpora Multi-pass Process data through RBMT first, and then through SMT Multi-System hybrid MT Multiple MT systems run in parallel

Multi-System Hybrid MT Related work: SMT + RBMT (Ahsan and Kolachina, 2010) Confusion Networks (Barrault, 2010) + Neural Network Model (Freitag et al., 2015) SMT + EBMT + TM + NE (Santanu et al., 2014) Recursive sentence decomposition (Mellebeek et al., 2006)

Combining Translations Combining full whole translations Translate the full input sentence with multiple MT systems Choose the best translation as the output

Combining Translations Combining full whole translations Translate the full input sentence with multiple MT systems Choose the best translation as the output Combining translations of sentence chunks Split the sentence into smaller chunks The chunks are the top level subtrees of the syntax tree of the sentence Translate each chunk with multiple MT systems Choose the best translated chunks and combine them http://lrec2016.lrec-conf.org/media/filer_public/2013/05/30/elra.gif

Whole translations Teikumu dal ana tekstvien b s Sentence tokenization Tulko ana ar tie saistes MT API Translation with online MT Google Translate Bing Translator LetsMT Selection of Lab k tulkojuma izv le the best translation Tulkojuma izvade Output

Chunks Teikumu dal ana tekstvien b s Sentence tokenization Syntactic analysis Sintaktisk anal ze Teikumu sadal ana fragmentos Sentence chunking Tulko ana ar tie saistes MT API Translation with online MT Google Translate Bing LetsMT Selection of Translator Lab ko fragmentu izv le the best chunks Sentence Teikumu apvieno ana recomposition http://lrec2016.lrec-conf.org/media/filer_public/2013/05/30/elra.gif Tulkojumu izvade Output

Choosing the best Choosing the best translation: KenLM (Heafield, 2011) calculates probabilities based on the observed entry with longest matching history ?? ?: ? 1 ? 1 ? 1) ? 1= ? ?? ?? ? ?? ?1 ?(?? ?=1 ? 1and backoff penalties ?(?? ? 1) are where the probability ? ?? ?? given by an already-estimated language model. Perplexity is then calculated using this probability: unknown probability distribution p anda proposed probability model q, it is evaluated by determining how well it predicts a separate test sample x1, x2... xN drawn from p. where given an

Linguistically motivated chunks An advanced approach to chunking Traverse the syntax tree bottom up, from right to left Add a word to the current chunk if The current chunk is not too long (sentence word count / 4) The word is non-alphabetic or only one symbol long The word begins with a genitive phrase ( of ) Otherwise, initialize a new chunk with the word In case when chunking results in too many chunks, repeat the process, allowing more (than sentence word count / 4) words in a chunk CICLing 2016

Linguistically motivated chunks Selection of the best translation: 12-gram LM trained with KenLM DGT-Translation Memory corpus (Steinberger, 2011) 3.1 million Latvian legal domain sentences Sentences scored with the query program from KenLM CICLing 2016

Linguistically motivated chunks Selection of the best translation: 12-gram LM trained with KenLM DGT-Translation Memory corpus (Steinberger, 2011) 3.1 million Latvian legal domain sentences Sentences scored with the query program from KenLM Test data 1581 random sentences from the JRC-Acquis corpus ACCURAT balanced evaluation corpus CICLing 2016

Linguistically motivated chunks CICLing 2016

Linguistically motivated chunks Simple chunks Linguistically motivated chunks Recently Recently there has been an increased interest there in the automated discovery of equivalent expressions has been an increased interest in the automated discovery expressions in different languages of equivalent in different languages . . CICLing 2016

Linguistically motivated chunks

Searching for the best The main differences: the manner of scoring chunks with the LM and selecting the best translation utilisation of multi-threaded computing that allows to run the process on all available CPU cores in parallel still very slow http://hlt2016.tilde.eu/sites/hlt2016.tilde.eu/themes/hlt2016/img/logo.png

Searching for the best Legal domain General domain 3 4 5 4 3 6 2 7 7 1 6 8 9 5 2 1 8 9 http://hlt2016.tilde.eu/sites/hlt2016.tilde.eu/themes/hlt2016/img/logo.png

Whole translations May 2015 results (Rikters 2015) Hybrid selection System BLEU Google Bing LetsMT Equal 16.92 100 % - - - Google Translate 17.16 - 100 % - - Bing Translator 28.27 - - 100 % - LetsMT 50.09 % 45.03 % - 4.88 % Hybrid Google + Bing 17.28 22.89 46.17 % - 48.39 % 5.44 % Hybrid Google + LetsMT 22.83 - 45.35 % 49.84 % 4.81 % Hybrid LetsMT + Bing 21.08 28.93 % 34.31 % 33.98 % 2.78 % Hybrid Google + Bing + LetsMT

Simple chunks September 2015 (Rikters and Skadi a 2016(1)) BLEU Hybrid selection System Whole translations Simple chunks Google Bing LetsMT 18.09 100% - - Google Translate 18.87 - 100% - Bing Translator 30.28 - - 100% LetsMT 18.73 74% 26% - Hybrid Google + Bing 21.27 24.50 25% - 75% Hybrid Google + LetsMT 26.24 24.66 - 24% 76% Hybrid LetsMT + Bing 26.63 22.69 17% 18% 65% Hybrid Google + Bing + LetsMT 24.72

Linguistically motivated chunks January 2016 (Rikters and Skadi a 2016(2)) System BLEU Equal Bing Google Hugo Yandex - - 17.43 17.73 17.14 16.04 BLEU 17.70 7.25% 43.85% 48.90% - - Whole translations G+B 17.63 3.55% 33.71% 30.76% 31.98% - Whole translations G+B+H 17.95 4.11% 19.46% 76.43% - - Simple Chunks G+B 17.30 3.88% 15.23% 19.48% 61.41% - Simple Chunks G+B+H 22.75% 39.10% 38.15% - - Linguistic Chunks G+B 18.29 7.36% 30.01% 19.47% 32.25% 10.91% Linguistic Chunks G+B+H+Y 19.21

Searching for the best May 2016 (Rikters 2016 (2)) BLEU System Legal General 14.40 Full-search 23.61 20.00 17.27 Linguistic chunks 16.99 17.43 Bing 16.19 17.72 Google 20.27 17.13 Hugo 19.75 16.03 Yandex

Interactive MS MT Start page Translate with online systems Input translations to combine Settings Input source sentence Input source sentence Input translated chunks Translation results http://www.dbis.lu.lv/fileadmin/_processed_/csm_BalticDBIS2016_logo_800x160_f3714c8b29.png

Publications Mat ss Rikters "Multi-system machine translation using online APIs for English-Latvian" ACL-IJCNLP 2015 Mat ss Rikters and Inguna Skadi a "Syntax-based multi-system machine translation" LREC 2016 Mat ss Rikters and Inguna Skadi a "Combining machine translated sentence chunks from multiple MT systems" CICLing 2016 Mat ss Rikters "K-translate interactive multi-system machine translation" Baltic DB&IS 2016 Mat ss Rikters Searching for the Best Translation Combination Across All Possible Variants Baltic HLT 2016 CICLing 2016 http://hlt2016.tilde.eu/sites/hlt2016.tilde.eu/themes/hlt2016/img/logo.png http://lrec2016.lrec-conf.org/media/filer_public/2013/05/30/elra.gif http://www.dbis.lu.lv/fileadmin/_processed_/csm_BalticDBIS2016_logo_800x160_f3714c8b29.png

Publications in progress Mat ss Rikters "Interactive Multi-system Machine Translation With Neural Language Models" IOS Press Mat ss Rikters Neural Network Language Models for Candidate Scoring in Hybrid Multi- System Machine Translation CoLing 2016

Neural Language Models 50.00 25.00 50.00 24.00 16.30 45.00 45.00 23.00 15.80 40.00 40.00 22.00 15.30 Perplexity Perplexity 35.00 35.00 21.00 BLEU BLEU 20.00 14.80 30.00 30.00 19.00 14.30 25.00 25.00 18.00 13.80 20.00 20.00 17.00 15.00 16.00 15.00 13.30 0.11 0.20 0.32 0.41 0.50 0.61 0.70 0.79 0.88 Epoch 1.00 1.09 1.20 1.29 1.40 1.47 1.56 1.67 1.74 1.77 0.11 0.20 0.32 0.41 0.50 0.61 0.70 0.79 0.88 Epoch 1.00 1.09 1.20 1.29 1.40 1.47 1.56 1.67 1.74 1.77 Perplexity BLEU-HY Linear (BLEU-HY) Perplexity BLEU Linear (BLEU)

Code on GitHub Code on GitHub http://ej.uz/ChunkMT http://ej.uz/SyMHyT http://ej.uz/MSMT http://ej.uz/chunker http://ej.uz/NeuralLM

Future work More enhancements for the chunking step Add special processing of multi-word expressions (MWEs) Try out other types of LMs POS tag + lemma Recurrent Neural Network Language Model (Mikolov et al., 2010) Continuous Space Language Model (Schwenk et al., 2006) Character-Aware Neural Language Model (Kim et al., 2015) Choose the best translation candidate with MT quality estimation QuEst++ (Specia et al., 2015) SHEF-NN (Shah et al., 2015) Handling MWEs in neural machine translation systems Experiments on English Estonian language pair

Citi darbi Pedago iskie darbi Vad ti vair ki kursa un kvalifik cijas darbi Vid j atz me 8.67 Studentu kurators Vasaras / ziemas skolas Deep Learning For Machine Translation ParseME 2nd Training School Neural Machine Translation Marathon

References References Ahsan, A., and P. Kolachina. "Coupling Statistical Machine Translation with Rule-based Transfer and Generation, AMTA-The Ninth Conference of the Association for Machine Translation in the Americas." Denver, Colorado (2010). Barrault, Lo c. "MANY: Open source machine translation system combination." The Prague Bulletin of Mathematical Linguistics 93 (2010): 147-155. Heafield, Kenneth. "KenLM: Faster and smaller language model queries." Proceedings of the Sixth Workshop on Statistical Machine Translation. Association for Computational Linguistics, 2011. Kim, Yoon, et al. "Character-aware neural language models." arXiv preprint arXiv:1508.06615 (2015). Mellebeek, Bart, et al. "Multi-engine machine translation by recursive sentence decomposition." (2006). Mikolov, Tomas, et al. "Recurrent neural network based language model." INTERSPEECH. Vol. 2. 2010. Petrov, Slav, et al. "Learning accurate, compact, and interpretable tree annotation." Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2006. Raivis Skadi ,K rlis Goba, Valters ics. 2010. Improving SMT for Baltic Languages with Factored Models. Proceedings of the Fourth International Conference Baltic HLT 2010, Frontiers in Artificial Intelligence and Applications, Vol. 2192. , 125-132. Rikters, M., Skadi a, I.: Syntax-based multi-system machine translation. LREC 2016. (2016) Rikters, M., Skadi a, I.: Combining machine translated sentence chunks from multiple MT systems. CICLing 2016. (2016) Santanu, Pal, et al. "USAAR-DCU Hybrid Machine Translation System for ICON 2014" The Eleventh International Conference on Natural Language Processing. , 2014. Schwenk, Holger, Daniel Dchelotte, and Jean-Luc Gauvain. "Continuous space language models for statistical machine translation." Proceedings of the COLING/ACL on Main conference poster sessions. Association for Computational Linguistics, 2006. Shah, Kashif, et al. "SHEF-NN: Translation Quality Estimation with Neural Networks." Proceedings of the Tenth Workshop on Statistical Machine Translation. 2015. Specia, Lucia, G. Paetzold, and Carolina Scarton. "Multi-level Translation Quality Prediction with QuEst++." 53rd Annual Meeting of the Association for Computational Linguistics and Seventh International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing: System Demonstrations. 2015. Steinberger, Ralf, et al. "Dgt-tm: A freely available translation memory in 22 languages." arXiv preprint arXiv:1309.5226 (2013). Steinberger, Ralf, et al. "The JRC-Acquis: A multilingual aligned parallel corpus with 20+ languages." arXiv preprint cs/0609058 (2006).

Paldies!