Inverted index compression based on term and document identifier reassignment

Ankara : The Department of Computer Engineering and the Institute of Engineering and Science of Bilkent University, 2008.Thesis (Master's) -- Bilkent University, 2008.Includes bibliographical references leaves 43-46.Compression of inverted indexes received great attention in recent years. An inverted index consists of lists of document identifiers, also referred as posting lists, for each term. Compressing an inverted index reduces the size of the index, which also improves the query performance due to the reduction on disk access times. In recent studies, it is shown that reassigning document identifiers has great effect in compression of an inverted index. In this work, we propose a novel technique that reassigns both term and document identifiers of an inverted index by transforming the matrix representation of the index into a block-diagonal form, which improves the compression ratio dramatically. We adapted row-net hypergraph-partitioning model for the transformation into block-diagonal form, which improves the compression ratio by as much as 50%. To the best of our knowledge, this method performs more effectively than previous inverted index compression techniques.Baykan, İzzet ÇağrıM.S

Index compression for information retrielval systems

[Abstract] Given the increasing amount of information that is available today, there is a clear need for Information Retrieval (IR) systems that can process this information in an efficient and effective way. Efficient processing means minimising the amount of time and space required to process data, whereas effective processing means identifying accurately which information is relevant to the user and which is not. Traditionally, efficiency and effectiveness are at opposite ends (what is beneficial to efficiency is usually harmful to effectiveness, and vice versa), so the challenge of IR systems is to find a compromise between efficient and effective data processing. This thesis investigates the efficiency of IR systems. It suggests several novel strategies that can render IR systems more efficient by reducing the index size of IR systems, referred to as index compression. The index is the data structure that stores the information handled in the retrieval process. Two different approaches are proposed for index compression, namely document reordering and static index pruning. Both of these approaches exploit document collection characteristics in order to reduce the size of indexes, either by reassigning the document identifiers in the collection in the index, or by selectively discarding information that is less relevant to the retrieval process by pruning the index. The index compression strategies proposed in this thesis can be grouped into two categories: (i) Strategies which extend state of the art in the field of efficiency methods in novel ways. (ii) Strategies which are derived from properties pertaining to the effectiveness of IR systems; these are novel strategies, because they are derived from effectiveness as opposed to efficiency principles, and also because they show that efficiency and effectiveness can be successfully combined for retrieval. The main contributions of this work are in indicating principled extensions of state of the art in index compression, and also in suggesting novel theoretically-driven index compression techniques which are derived from principles of IR effectiveness. All these techniques are evaluated extensively, in thorough experiments involving established datasets and baselines, which allow for a straight-forward comparison with state of the art. Moreover, the optimality of the proposed approaches is addressed from a theoretical perspective.[Resumen] Dada la creciente cantidad de información disponible hoy en día, existe una clara necesidad de sistemas de Recuperación de Información (RI) que sean capaces de procesar esa información de una manera efectiva y eficiente. En este contexto, eficiente significa cantidad de tiempo y espacio requeridos para procesar datos, mientras que efectivo significa identificar de una manera precisa qué información es relevante para el usuario y cual no lo es. Tradicionalmente, eficiencia y efectividad se encuentran en polos opuestos - lo que es beneficioso para la eficiencia, normalmente perjudica la efectividad y viceversa - así que un reto para los sistemas de RI es encontrar un compromiso adecuado entre el procesamiento efectivo y eficiente de los datos. Esta tesis investiga el problema de la eficiencia de los sistemas de RI. Sugiere diferentes estrategias novedosas que pueden permitir la reducción de los índices de los sistemas de RI, enmarcadas dentro da las técnicas conocidas como compresión de índices. El índice es la estructura de datos que almacena la información utilizada en el proceso de recuperación. Se presentan dos aproximaciones diferentes para la compresión de los índices, referidas como reordenación de documentos y pruneado estático del índice. Ambas aproximaciones explotan características de colecciones de documentos para reducir el tamaño final de los índices, mediante la reasignación de los identificadores de los documentos de la colección o bien descartando selectivamente la información que es "menos relevante" para el proceso de recuperación. Las estrategias de compresión propuestas en este tesis se pueden agrupar en dos categorías: (i) estrategias que extienden el estado del arte en la eficiencia de una manera novedosa y (ii) estrategias derivadas de propiedades relacionadas con los principios de la efectividad en los sistemas de RI; estas estrategias son novedosas porque son derivadas desde principios de la efectividad como contraposición a los de la eficiencia, e porque revelan como la eficiencia y la efectividad pueden ser combinadas de una manera efectiva para la recuperación de información. Las contribuciones de esta tesis abarcan la elaboración de técnicas del estado del arte en compresión de índices y también en la derivación de técnicas de compresión basadas en fundamentos teóricos derivados de los principios de la efectividad de los sistemas de RI. Todas estas técnicas han sido evaluadas extensamente con numerosos experimentos que involucran conjuntos de datos y técnicas de referencia bien establecidas en el campo, las cuales permiten una comparación directa con el estado del arte. Finalmente, la optimalidad de las aproximaciones presentadas es tratada desde una perspectiva teórica

Integrating Skips and Bitvectors for List Intersection

This thesis examines space-time optimizations of in-memory search engines. Search engines can answer queries quickly, but this is accomplished using significant resources in the form of multiple machines running concurrently. Improving the performance of search engines means reducing the resource costs, such as hardware, energy, and cooling. These saved resources can then be used to improve the effectiveness of the search engine or provide additional added value to the system. We improve the space-time performance for search engines in the context of in-memory conjunctive intersection of ordered document identifier lists. We show that reordering of document identifiers can produce dense regions in these lists, where bitvectors can be used to improve the efficiency of conjunctive list intersection. Since the process of list intersection is a fundamental building block and a major performance bottleneck for search engines, this work will be important for all search engine researchers and developers. Our results are presented in three stages. First, we show how to combine multiple existing techniques for list intersection to improve space-time performance. We combine bitvectors for large lists with skips over compressed values for the other lists. When the skips are large and overlaid on the compressed lists, space-time performance is superior to existing techniques, such as using skips or bitvectors separately. Second, we show that grouping documents by size and ordering by URL within groups combines the skewed clustering that results from document size ordering with the tight clustering that results from URL ordering. We propose a new semi-bitvector data structure that encodes the front of a list, including groups with large documents, as a bitvector and the rest of the list as skips over compressed values. This combination produces significant space-time performance gains on top of the gains from the first stage. Third, we show how partitioning by document size into separate indexes can also produce high density regions that can be exploited by bitvectors, resulting in benefits similar to grouping by document size within one index. This partitioning technique requires no modification of the intersection algorithms, and it is therefore broadly applicable. We further show that any of our partitioning approaches can be combined with semi-bitvectors and grouping within each partition to effectively exploit skewed clustering and tight clustering in our dataset. A hierarchy of partitioning approaches may be required to exploit clustering in very large document collections

Manoeuvre Planning Architecture for the Optimisation of Spacecraft Formation Flying Reconfiguration Manoeuvres

Formation flying of multiple spacecraft collaborating toward the same goal is fast becoming a reality for space mission designers. Often the missions require the spacecraft to perform translational manoeuvres relative to each other to achieve some mission objective. These manoeuvres need to be planned to ensure the safety of the spacecraft in the formation and to optimise fuel management throughout the fleet. In addition to these requirements is it desirable for this manoeuvre planning to occur autonomously within the fleet to reduce operations cost and provide greater planning flexibility for the mission. One such mission that would benefit from this type of manoeuvre planning is the European Space Agency’s DARWIN mission, designed to search for extra-solar Earth-like planets using separated spacecraft interferometry. This thesis presents a Manoeuvre Planning Architecture for the DARWIN mission. The design of the Architecture involves identifying and conceptualising all factors affecting the execution of formation flying manoeuvres at the Sun/Earth libration point L2. A systematic trade-off analysis of these factors is performed and results in a modularised Manoeuvre Planning Architecture for the optimisation of formation flying reconfiguration manoeuvres. The Architecture provides a means for DARWIN to autonomously plan manoeuvres during the reconfiguration mode of the mission. The Architecture consists of a Science Operations Module, a Position Assignment Module, a Trajectory Design Module and a Station-keeping Module that represents a multiple multi-variable optimisation approach to the formation flying manoeuvre planning problem. The manoeuvres are planned to incorporate target selection for maximum science returns, collision avoidance, thruster plume avoidance, manoeuvre duration minimisation and manoeuvre fuel management (including fuel consumption minimisation and formation fuel balancing). With many customisable variables the Architecture can be tuned to give the best performance throughout the mission duration. The implementation of the Architecture highlights the importance of planning formation flying reconfiguration manoeuvres. When compared with a benchmark manoeuvre planning strategy the Architecture demonstrates a performance increase of 27% for manoeuvre scheduling and fuel savings of 40% over a fifty target observation tour. The Architecture designed in this thesis contributes to the field of spacecraft formation flying analysis on various levels. First, the manoeuvre planning is designed at the mission level with considerations for mission operations and station-keeping included in the design. Secondly, the requirements analysis and implementation of Science Operation Module represent a unique insight into the complexity of observation scheduling for exo-planet analysis missions and presents a robust method for autonomously optimising that scheduling. Thirdly, in-depth analyses are performed on DARWIN-based modifications of existing manoeuvre optimisation strategies identifying their strengths and weaknesses and ways to improve them. Finally, though not implemented in this thesis, the design of a Station-keeping Module is provided to add station-keeping optimisation functionality to the Architecture

Recent Developments in Smart Healthcare

Medicine is undergoing a sector-wide transformation thanks to the advances in computing and networking technologies. Healthcare is changing from reactive and hospital-centered to preventive and personalized, from disease focused to well-being centered. In essence, the healthcare systems, as well as fundamental medicine research, are becoming smarter. We anticipate significant improvements in areas ranging from molecular genomics and proteomics to decision support for healthcare professionals through big data analytics, to support behavior changes through technology-enabled self-management, and social and motivational support. Furthermore, with smart technologies, healthcare delivery could also be made more efficient, higher quality, and lower cost. In this special issue, we received a total 45 submissions and accepted 19 outstanding papers that roughly span across several interesting topics on smart healthcare, including public health, health information technology (Health IT), and smart medicine

LIPIcs, Volume 244, ESA 2022, Complete Volume

LIPIcs, Volume 244, ESA 2022, Complete Volum