Results of the second year of the project were presented in Luxembourg on November 25, 2013. The following slides give details about the progress.
- Project Overview
- WP1: User Interface Studies
- WP2: Interactive Translation Predictions
- WP3: Interactive Editing
- WP4: Advanced Translation Models
- WP5: Integration - Tool Development
- WP6: Evaluation - Field Trial
Scientific Overview: Year 2
In the second year progress was made on all objectives of the project – research into new methods of assistance and translator behaviour, efficient implementation of methods into the workbench, testing in lab and field tests, and outreach to external users.
Work progress and achievements during the period
The CASMACAT project spans the entire breadth of
- research: development of novel methods and scientific studies of translator behaviour
- development: efficient implementation of methods in a common workbench
- testing: validation of methods in lab and field tests
- dissemination: promotion of the workbench and collaboration with external partners
Transfer of methods through these successive stages takes time, and in the “middle year” of the project we work on all these aspects: entirely novel methods and models were researched, some of these and earlier researched methods were properly integrated into the workbench, some of these were tested in lab and field tests, and fully validated methods were made available in a beta version of the workbench that was released and promoted during the year, which enabled us to build useful connections with external partners for further real world test cases.
WP5: Integration – Workbench
The Workbench consists of a web-based editor which connects to a server that communicates with the MT engine and a database. Additional components are a management tool, handwriting recognition server, and the integration of the eye tracker.
The modular design allows integration of CASMACAT functionality into existing CAT tools. The workbench was jointly developed with the MATECAT project.
The collaboration with the MATECAT project in the development of a common workbench created both challenges and opportunities for the project. The main focus of the MATECAT project is to build an efficient and reliable tool with more conservative functionality. Hence, the collaboration currently takes the form of MATECAT developing the core GUI tool, and CASMACAT enhancing it with additional GUI and back-end capability. While the transfer of advances from CASMACAT back to the core MATECAT branch (besides changes to support CASMACAT’s richer capabilities) has been somewhat slow, there are some instances of transfer (for instance, the integration of a bilingual concordancer), and truly common development (Machine translation server). The benefits of the collaboration for the goals of the CASMACAT project is to be able to develop novel capabilities in a robust and well-tested user interface which leads to more uptake by external users, even though this carries significant overhead in co-ordinating of project objectives and merging of code bases.
The beta version CASMACAT workbench was released in month 18, with significant successful outreach activity (conference publications, demonstration sessions, large interest from commercial and governmental organizations). More detail on these activities is provided in the dissemination section of this report.
The main capabilities of the beta release are:
Modular design: The workbench decomposes into clearly separated components GUI/web server, CAT server, and MT server, with clearly defined API calls between them. This allows for partial use of components and replacement with alternative implementations.
Basic functionality: Documents are uploaded as XLIFF files and can be worked on either by post-editing output from machine translation or translation memory servers, or by using interactive translation prediction.
Advanced CAT features: Besides the “auto-completion” functionality of interactive translation prediction with the option of setting the number of visible predicted words and requesting alternative translations, the tool offers word-level confidence measures which highlight less reliable words in the automatic translation, visualization of word alignment, and a document-wide search and replace function.
Logging and Replay: All user interactions (via mouse or keyboard) are logged to the degree that the entire session, as it appeared in the browser, can be replayed at a later stage.
Eye tracking: With a special browser plug-in and eye tracking hardware, the logging and replay may also include the exact gaze position (in terms of x/y co-ordinates) and word in focus (computed with automatic gaze-to-word mapping).
E-Pen Interaction: In addition to keyboard and mouse, an e-pen may be used to edit translations, with specifically designed gestures for edit actions such as delete, insert, substitute, reject, merge, undo, redo, etc. This requires an additional component, a hand-written text recognition (HTR) server component. Additional capability is being added at the time of this writing, and will be throughout the remainder of the project and in the future life of the workbench.
WP6: Evaluation – Field Trial
The large set of functionality added to the CASMACAT workbench creates many opportunities to explore how each functionality, language pair, text domain, and individual translator impacts productivity and behaviour. Since the main goal of the CASMACAT project is to gather sufficient evidence to assess the novel capabilities, we limited ourselves in other aspects: The field trial was carried out only on English-Spanish in the news domain, using a phrase-based system developed by UEDIN for the WMT 2013 evaluation campaign.
We compare three different tool settings: simple post-editing, interactive machine translation and an extended form of interactive machine translation. Some of the specific choices for the tool settings were made based on results of preliminary studies and the experience from the first field trial.
As already the data gathered from the first field trial provided valuable training data for various methods, we expect many benefits from this new set of data. Data and models from the field trial are publicly available.
WP1: Cognitive Modeling
The field trials and lab experiments, with additional user activity data collections gives the CASMACAT project the basis to study translator behaviour and then ground novel types of assistance to translators in actual usage studies. There is also a clear scientific benefit to gain a better understanding of how professional translators translate, post-edit and use new interactive translation tools.
Our collection of post-editing user activity data as well as interactive translation user activity data allows us to gain a better understanding of the core efficiencies of these approaches and help us to build more in-depth models of human behaviour in this context.
Recognizing individual differences, we were able to identify different translator types post-editing styles, which leads directly to recommendations on how translation tools should be built.
Cognitive modeling connects our work to psycho-linguistic research. We carried out eye tracking studies to shed light on how translators respond to different types of machine translation errors in post-editing.
WP2: Interactive Machine Translation
A core goal of the CASMACAT project is the development of novel types human-computer interactions for translation. To a large degree this is explored as extension to the interactive machine translation paradigm, but also includes multi-modality.
'Search and machine learning criteria for prediction goes beyond the search graph string edit distance matching approach of traditional interactive machine translation and explores the integration of additional features, error metrics, and search algorithms. Work includes the reframing of the optimization problem for the best extension using Bayesian decision theory instead of simple maximum-a-posteriori optimization, more efficient algorithms based on greedy search and the use of error correcting technology, and finally the reformulation of the optimization problem as a supervised classification problem.
Multi-modality for interactive translation prediction explores the use of an e-pen as an input device, which has some advantages over keyboard and mouse when editing text. It is worth noting, that in a related project building on the CASMACAT workbench, we integrated speech as additional input modality.
Prediction from active interaction integrates work on confidence estimation (more on that in WP3 below) into the interactive machine translation approach.
Prediction from parse forest extends the interactive machine translation approach from traditional phrase-based models to hierarchical and syntax-based translation models which have show to deliver better machine translation quality for language pairs such as Chinese-English and German-English. We explored two broad approaches for matching user prefixes against parse forests (the search graphs of such models): firstly exact search for partial translation, extended to use an error-correction interpretation of string edit distance, and secondly two approximate string matching algorithms that search the parse forest top-down and bottom-up.
WP 3: Interactive Editing
The CASMACAT project investigates a new methods to assist the editing of translations. In year 2, one main focus is the development of confidence measures that informs the user about the reliability of machine translation output, and another is the display of alternative translations.
Confidence measures are currently also referred to as quality estimates have become a larger research topic in the field of statistical machine translation. The CASMACAT project contributed to the organization of a shared task on this subject, which was co-organized with other EU-funded research efforts (MATECAT, QualityLauchPad, MosesCore), and hosted by the Workshop on Statistical Machine Translation (WMT), co-located with the Annual Meeting of the Association for Cormputational Linguistics (ACL). The main contribution by CASMACAT was the provision of training and testing data derived from the first field trial.
We also developed novel methods for sentence-level and word-level confidence measures. Sentence level confidence measures were developed using dimensionally reduction (which performs better than feature selection) and a number of different machine learning methods (with support vector machines showing the best performance). Word-level confidence measures attempt classify words in the machine translation output as either correct or wrong. We developed a new classifier based on partial least square discriminative analysis (PLS-DA).
We worked on two approaches for the display of multiple translation options. The first idea is to build on the work in the Caitra tool, which showed possible translations for input words and phrases in a translation option array. Here, we are working on increasing the diversity of the displayed options. The second idea is the integration of a bilingual concordancer, which shows possible translations for input words and phrases in context. We completed the implementation of this functionality and collaborated with the MATECAT on alternative integrations into the workbench.
A novel form of authoring assistance is the capability to request alternative translations for part of the output sentence on demand. This task is related to automatic paraphrasing, except that it is mainly informed by the source. We developed a method based on the search graph of the decoder with a strong focus on diversity and additional interactivity.
WP4: Advanced Translation Models – Online Updating
Machine translation system perform best if they are adapted for the task at hand. Besides exploring general adaptation methods, the CASMACAT also utilizes a unique property of the computer aided translation process: When the professional translator translates a new sentence, it and its translation can be exploited as new sentence pair by the machine translation model.
Online learning for interactive translation prediction addresses exactly this insight. Extending work from year 1, we refined and clarified some of the already developed approaches, created efficient implementations, and were able to carry out experiments with larger corpora.
Active learning in interactive translation prediction optimizes the use of human contribution to the process of translating a document by focusing on sentences that benefit most from such human supervision.
Domain and user adaptation is a more general field of research in statistical machine translation, which is important beyond the application of computer aided translation. We apply the Bayesian learning paradigm to optimize the weights in the log-linear model.