Reevaluating Assembly Evaluations with Feature Response Curves: GAGE and Assemblathons

In just the last decade, a multitude of bio-technologies and software pipelines have emerged to revolutionize genomics. To further their central goal, they aim to accelerate and improve the quality of de novo whole-genome assembly starting from short DNA reads. However, the performance of each of these tools is contingent on the length and quality of the sequencing data, the structure and complexity of the genome sequence, and the resolution and quality of long-range information. Furthermore, in the absence of any metric that captures the most fundamental "features" of a high-quality assembly, there is no obvious recipe for users to select the most desirable assembler/assembly. International competitions such as Assemblathons or GAGE tried to identify the best assembler(s) and their features. Some what circuitously, the only available approach to gauge de novo assemblies and assemblers relies solely on the availability of a high-quality fully assembled reference genome sequence. Still worse, reference-guided evaluations are often both difficult to analyze, leading to conclusions that are difficult to interpret. In this paper, we circumvent many of these issues by relying upon a tool, dubbed FRCbam, which is capable of evaluating de novo assemblies from the read-layouts even when no reference exists. We extend the FRCurve approach to cases where lay-out information may have been obscured, as is true in many deBruijn-graph-based algorithms. As a by-product, FRCurve now expands its applicability to a much wider class of assemblers -- thus, identifying higher-quality members of this group, their inter-relations as well as sensitivity to carefully selected features, with or without the support of a reference sequence or layout for the reads. The paper concludes by reevaluating several recently conducted assembly competitions and the datasets that have resulted from them.Comment: Submitted to PLoS One. Supplementary material available at http://www.nada.kth.se/~vezzi/publications/supplementary.pdf and http://cs.nyu.edu/mishra/PUBLICATIONS/12.supplementaryFRC.pd

A proposal for the evaluation of adaptive information retrieval systems using simulated interaction

The Centre for Next Generation Localisation (CNGL) is involved in building interactive adaptive systems which combine Information Retrieval (IR), Adaptive Hypermedia (AH) and adaptive web techniques and technologies. The complex functionality of these systems coupled with the variety of potential users means that the experiments necessary to evaluate such systems are difficult to plan, implement and execute. This evaluation requires both component-level scientific evaluation and user-based evaluation. Automated replication of experiments and simulation of user interaction would be hugely beneficial in the evaluation of adaptive information retrieval systems (AIRS). This paper proposes a methodology for the evaluation of AIRS which leverages simulated interaction. The hybrid approach detailed combines: (i) user-centred methods for simulating interaction and personalisation; (ii) evaluation metrics that combine Human Computer Interaction (HCI), AH and IR techniques; and (iii) the use of qualitative and quantitative evaluations. The benefits and limitations of evaluations based on user simulations are also discussed

Reliability assessment of microgrid with renewable generation and prioritized loads

With the increase in awareness about the climate change, there has been a tremendous shift towards utilizing renewable energy sources (RES). In this regard, smart grid technologies have been presented to facilitate higher penetration of RES. Microgrids are the key components of the smart grids. Microgrids allow integration of various distributed energy resources (DER) such as the distributed generation (DGs) and energy storage systems (ESSs) into the distribution system and hence remove or delay the need for distribution expansion. One of the crucial requirements for utilities is to ensure that the system reliability is maintained with the inclusion of microgrid topology. Therefore, this paper evaluates the reliability of a microgrid containing prioritized loads and distributed RES through a hybrid analytical-simulation method. The stochasticity of RES introduces complexity to the reliability evaluation. The method takes into account the variability of RES through Monte- Carlo state sampling simulation. The results indicate the reliability enhancement of the overall system in the presence of the microgrid topology. In particular, the highest priority load has the largest improvement in the reliability indices. Furthermore, sensitivity analysis is performed to understand the effects of the failure of microgrid islanding in the case of a fault in the upstream network

MARINE: Man-in-the-middle attack resistant trust model IN connEcted vehicles

Vehicular Ad-hoc NETwork (VANET), a novel technology holds a paramount importance within the transportation domain due to its abilities to increase traffic efficiency and safety. Connected vehicles propagate sensitive information which must be shared with the neighbors in a secure environment. However, VANET may also include dishonest nodes such as Man-in-the-Middle (MiTM) attackers aiming to distribute and share malicious content with the vehicles, thus polluting the network with compromised information. In this regard, establishing trust among connected vehicles can increase security as every participating vehicle will generate and propagate authentic, accurate and trusted content within the network. In this paper, we propose a novel trust model, namely, Man-in-the-middle Attack Resistance trust model IN connEcted vehicles (MARINE), which identifies dishonest nodes performing MiTM attacks in an efficient way as well as revokes their credentials. Every node running MARINE system first establishes trust for the sender by performing multi-dimensional plausibility checks. Once the receiver verifies the trustworthiness of the sender, the received data is then evaluated both directly and indirectly. Extensive simulations are carried out to evaluate the performance and accuracy of MARINE rigorously across three MiTM attacker models and the bench-marked trust model. Simulation results show that for a network containing 35% MiTM attackers, MARINE outperforms the state of the art trust model by 15%, 18%, and 17% improvements in precision, recall and F-score, respectively.N/A

Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems

Natural language generation (NLG) is a critical component of spoken dialogue and it has a significant impact both on usability and perceived quality. Most NLG systems in common use employ rules and heuristics and tend to generate rigid and stylised responses without the natural variation of human language. They are also not easily scaled to systems covering multiple domains and languages. This paper presents a statistical language generator based on a semantically controlled Long Short-term Memory (LSTM) structure. The LSTM generator can learn from unaligned data by jointly optimising sentence planning and surface realisation using a simple cross entropy training criterion, and language variation can be easily achieved by sampling from output candidates. With fewer heuristics, an objective evaluation in two differing test domains showed the proposed method improved performance compared to previous methods. Human judges scored the LSTM system higher on informativeness and naturalness and overall preferred it to the other systems.Comment: To be appear in EMNLP 201