Improving the post-editing experience using translation recommendation: a user study

We report findings from a user study with professional post-editors using a translation recommendation framework (He et al., 2010) to integrate Statistical Machine Translation (SMT) output with Translation Memory (TM) systems. The framework recommends SMT outputs to a TM user when it predicts that SMT outputs are more suitable for post-editing than the hits provided by the TM. We analyze the effectiveness of the model as well as the reaction of potential users. Based on the performance statistics and the users’comments, we find that translation recommendation can reduce the workload of professional post-editors and improve the acceptance of MT in the localization industry

Classification accuracy comparison: hypothesis tests and the use of confidence intervals in evaluations of difference, equivalence and non-inferiority

The comparison of classification accuracy statements has generally been based upon tests of difference or inequality when other scenarios and approaches may be more appropriate. Procedures for evaluating two scenarios with interest focused on the similarity in accuracy values, non-inferiority and equivalence, are outlined following a discussion of tests of difference (inequality). It is also suggested that the confidence interval of the difference in classification accuracy may be used as well as or instead of conventional hypothesis testing to reveal more information about the disparity in the classification accuracy values compared

Classification accuracy comparison: hypothesis tests and the use of confidence intervals in evaluations of difference, equivalence and non-inferiority

The comparison of classification accuracy statements has generally been based upon tests of difference or inequality when other scenarios and approaches may be more appropriate. Procedures for evaluating two scenarios with interest focused on the similarity in accuracy values, non-inferiority and equivalence, are outlined following a discussion of tests of difference (inequality). It is also suggested that the confidence interval of the difference in classification accuracy may be used as well as or instead of conventional hypothesis testing to reveal more information about the disparity in the classification accuracy values compared

Selection of Learning Algorithm for Musical Tone Stimulated Wavelet De-Noised EEG Signal Classification

The task of classifying EEG signals pose a challenge in the selection of which learning algorithm is best to provide higher classification accuracy. In this study, five wellknown learning algorithms used in data mining were utilized. The task is to classify musical tone stimulated wavelet de-noised EEG signals. Classification tasks include whether the EEG signal is tone stimulated or not, and whether the EEG signal is stimulated by either the C, F or G tone. Results show higher correct classification instances (CCI) percentages and accuracies in the first classification task using the J48 decision tree as the learning algorithm. For the second classification task, the k-nn learning algorithm outruns the other classifiers but gave low accuracy and low correct classification percentage. The possibility of increasing the performance was explored by increasing the k (number of neighbors). With the increment, its produced directly proportionate in accuracy and correct classification percentage within a certain value of k. A larger k value will reduce the accuracy and the correct classification percentages

QADI as a New Method and Alternative to Kappa for Accuracy Assessment of Remote Sensing-Based Image Classification

Classification is a very common image processing task. The accuracy of the classified map is typically assessed through a comparison with real-world situations or with available reference data to estimate the reliability of the classification results. Common accuracy assessment approaches are based on an error matrix and provide a measure for the overall accuracy. A frequently used index is the Kappa index. As the Kappa index has increasingly been criticized, various alternative measures have been investigated with minimal success in practice. In this article, we introduce a novel index that overcomes the limitations. Unlike Kappa, it is not sensitive to asymmetric distributions. The quantity and allocation disagreement index (QADI) index computes the degree of disagreement between the classification results and reference maps by counting wrongly labeled pixels as A and quantifying the difference in the pixel count for each class between the classified map and reference data as Q. These values are then used to determine a quantitative QADI index value, which indicates the value of disagreement and difference between a classification result and training data. It can also be used to generate a graph that indicates the degree to which each factor contributes to the disagreement. The efficiency of Kappa and QADI were compared in six use cases. The results indicate that the QADI index generates more reliable classification accuracy assessments than the traditional Kappa can do. We also developed a toolbox in a GIS software environment.University of Tabriz, International and Academic Cooperation DirectionAlexander Von Humboldt FoundationPeer Reviewe

A Quantitative Methodology for Vetting Dark Network Intelligence Sources for Social Network Analysis

Social network analysis (SNA) is used by the DoD to describe and analyze social networks, leading to recommendations for operational decisions. However, social network models are constructed from various information sources of indeterminate reliability. Inclusion of unreliable information can lead to incorrect models resulting in flawed analysis and decisions. This research develops a methodology to assist the analyst by quantitatively identifying and categorizing information sources so that determinations on including or excluding provided data can be made. This research pursued three main thrusts. It consolidated binary similarity measures to determine social network information sources\u27 concordance and developed a methodology to select suitable measures dependent upon application considerations. A methodology was developed to assess the validity of individual sources of social network data. This methodology utilized source pairwise comparisons to measure information sources\u27 concordance and a weighting schema to account for sources\u27 unique perspectives of the underlying social network. Finally, the developed methodology was tested over a variety of generated networks with varying parameters in a design of experiments paradigm (DOE). Various factors relevant to conditions faced by SNA analysts potentially employing this methodology were examined. The DOE was comprised of a 24 full factorial design augmented with a nearly orthogonal Latin hypercube. A linear model was constructed using quantile regression to mitigate the non-normality of the error terms

The integration of machine translation and translation memory

We design and evaluate several models for integrating Machine Translation (MT) output into a Translation Memory (TM) environment to facilitate the adoption of MT technology in the localization industry. We begin with the integration on the segment level via translation recommendation and translation reranking. Given an input to be translated, our translation recommendation model compares the output from the MT and the TMsystems, and presents the better one to the post-editor. Our translation reranking model combines k-best lists from both systems, and generates a new list according to estimated post-editing effort. We perform both automatic and human evaluation on these models. When measured against the consensus of human judgement, the recommendation model obtains 0.91 precision at 0.93 recall, and the reranking model obtains 0.86 precision at 0.59 recall. The high precision of these models indicates that they can be integrated into TM environments without the risk of deteriorating the quality of the post-editing candidate, and can thereby preserve TM assets and established cost estimation methods associated with TMs. We then explore methods for a deeper integration of translation memory and machine translation on the sub-segment level. We predict whether phrase pairs derived from fuzzy matches could be used to constrain the translation of an input segment. Using a series of novel linguistically-motivated features, our constraints lead both to more consistent translation output, and to improved translation quality, reflected by a 1.2 improvement in BLEU score and a 0.72 reduction in TER score, both of statistical significance (p < 0.01). In sum, we present our work in three aspects: 1) translation recommendation and translation reranking models that can access high quality MT outputs in the TMenvironment, 2) a sub-segment translation memory and machine translation integration model that improves both translation consistency and translation quality, and 3) a human evaluation pipeline to validate the effectiveness of our models with human judgements

Key issues in rigorous accuracy assessment of land cover products

© 2019 Accuracy assessment and land cover mapping have been inexorably linked throughout the first 50 years of publication of Remote Sensing of Environment. The earliest developers of land-cover maps recognized the importance of evaluating the quality of their maps, and the methods and reporting format of these early accuracy assessments included features that would be familiar to practitioners today. Specifically, practitioners have consistently recognized the importance of obtaining high quality reference data to which the map is compared, the need for sampling to collect these reference data, and the role of an error matrix and accuracy measures derived from the error matrix to summarize the accuracy information. Over the past half century these techniques have undergone refinements to place accuracy assessment on a more scientifically credible footing. We describe the current status of accuracy assessment that has emerged from nearly 50 years of practice and identify opportunities for future advances. The article is organized by the three major components of accuracy assessment, the sampling design, response design, and analysis, focusing on good practice methodology that contributes to a rigorous, informative, and honest assessment. The long history of research and applications underlying the current practice of accuracy assessment has advanced the field to a mature state. However, documentation of accuracy assessment methods needs to be improved to enhance reproducibility and transparency, and improved methods are required to address new challenges created by advanced technology that has expanded the capacity to map land cover extensively in space and intensively in time

Robust outlier detection in high-density surface electromyographic signals

Conf Proc IEEE Eng Med Biol Soc. 2010;2010:4850-3. PubMed PMID: 2109664