Optimal column layout for hybrid workloads

Data-intensive analytical applications need to support both efficient reads and writes. However, what is usually a good data layout for an update-heavy workload, is not well-suited for a read-mostly one and vice versa. Modern analytical data systems rely on columnar layouts and employ delta stores to inject new data and updates. We show that for hybrid workloads we can achieve close to one order of magnitude better performance by tailoring the column layout design to the data and query workload. Our approach navigates the possible design space of the physical layout: it organizes each column’s data by determining the number of partitions, their corresponding sizes and ranges, and the amount of buffer space and how it is allocated. We frame these design decisions as an optimization problem that, given workload knowledge and performance requirements, provides an optimal physical layout for the workload at hand. To evaluate this work, we build an in-memory storage engine, Casper, and we show that it outperforms state-of-the-art data layouts of analytical systems for hybrid workloads. Casper delivers up to 2.32x higher throughput for update-intensive workloads and up to 2.14x higher throughput for hybrid workloads. We further show how to make data layout decisions robust to workload variation by carefully selecting the input of the optimization.http://www.vldb.org/pvldb/vol12/p2393-athanassoulis.pdfPublished versionPublished versio

GNSS-R as a source of opportunity for remote sensing of the cryosphere

This work evaluates the potential use of signals from the Global Navigation Satellite Systems (GNSS) that scatter off the Earth surface for the retrieval of geophysical information from the cryosphere. For this purpose, the present study is based on data collected with a dedicated reflectometry GNSS receiver during two field campaigns, which were focused on two types of characteristic surfaces of the cryosphere: thin sea ice covers and thick dry snow accumulations. During the first experiment, the complete process of formation, evolution and melting of sea ice was monitorized for more than seven months in a bay located in Greenland. This type of ice is typically characterized by its thickness, concentration and roughness. Different observables from GNSS reflections are analyzed to try to infer these properties. The ice thickness is linked to the free-board level, defined as the height of the sea ice surface. Accurate phase altimetry is achieved, showing good agreement with an Arctic tide model. In addition, the long term results of ellipsoidal height retrievals are consistent with the evolution of the ice surface temperature product given by MODIS, which is a key parameter in the rate of growth of sea ice. On the other hand, the presence of salinity in the sea ice modifies its dielectric properties, resulting in different amplitude and phase for the co- and cross-polar components of the complex Fresnel coefficients. The polarimetric measurements obtained show good agreement with visual inspections of ice concentration from an Arctic weather station. Finally, the shape of the reflected signals and its phase dispersion are tested as potential signatures of surface roughness. For comparison, ice charts of the experimental area are employed. In particular, maximums in roughness given by the GNSS observables coincide with fast ice events. Fast ice is defined as ice anchored to the coast, where the tidal movements contribute to the development of strange patterns, cracks, and fissures on its surface, thus consistent with the GNSS-R roughness retrievals. The second experiment took place on Antarctica, monitoring a pristine snow area which is well-known for the calibration of remote sensing instruments. Due to the relative stability of the snow layers, the data acquisition was limited to ten continuous days. Interferometric beats were found after a first analysis of the amplitude from the collected signals, which were consistent with a multipath model where the reflector lies below the surface level. Motivated by these results, a forward model is developed that reconstructs the complex received signal as a sum of a finite number of reflections, coming from different snow layers (a snow density profile obtained from in-situ measurements). The interferometric information is then retrieved from the spectral analysis applied to time series from both real and modeled signals (lag-holograms). We find that the frequency bands predicted by the model are in general consistent with the data and the lag-holograms show repeatability for different days. Then, we attempt a proper inversion of the collected data to determine the dominant layers of the dry snow profile that contribute to L-band reflections, which are related to significant gradients of snow density/permittivity.Aquest treball avalua el possible ús dels senyals dels sistemes mundials de navegació per satèl lit (GNSS) que es reflecteixen a la superfície terrestre, per a l’extracció de la informació geofísica de la criosfera. Amb aquest propòsit, el present estudi es basa en dades recollides amb un reflectòmetre GNSS durant dues campanyes experimentals, centrades en dos tipus de superfícies característiques de la criosfera: cobertes de gel marí i gruixudes acumulacions de neu seca. En el primer experiment, el procés complet de formació, evolució i fusió del gel marí va ser monitoritzat durant més de set mesos a una badia situada a Groenlàndia. Aquest tipus de gel es caracteritza típicament amb el seu gruix, concentració i rugositat. Diferents observables de les reflexions GNSS són analitzats per tractar de fer una estimació d’aquestes propietats. El gruix de gel està relacionat amb el nivell de francbord, que a la seva vegada està relacionat amb l’alçada de la superfície de gel marí. S’ha aconseguit altimetria de fase precisa, que mostra correlació amb un model de marea de l’Àrtic. A més, els resultats a llarg termini de l’alçada elipsoidal segueixen l’evolució de les mesures de temperatura de superfície de gel donades per MODIS. La temperatura és un paràmetre clau en el ritme de creixement del gel marí. Per altra banda, la presència de sal a aquest tipus de gel modifica les seves propietats dielèctriques, el que implica variacions d’amplitud i fase per als coeficients de Fresnel complexos amb polaritzacions oposades. Les mesures polarimètriques obtingudes mostren concordança amb els valors de concentració de gel obtinguts des d’una estació meteorològica propera. Finalment, la forma de la senyal reflectida i la dispersió de la seva fase s’evaluen com a potencials indicadors de la rugositat de superfície. Per a la seva comparació, es fan servir mapes del gel de la zona experimental. En concret, els valors màxims a la rugositat estimada a partir pels observables GNSS coincideixen amb el gel fixe, que es refereix a gel ancorat a la costa, on els moviments de les marees contribueixen al desenvolupament de patrons estranys, esquerdes i fissures en la seva superfície. El segon experiment es va dur a terme a l’Antàrtida, monitoritzant una àrea de neu pristina que és ben coneguda per al calibratge d’instruments de teledetecció. A causa de la relativa estabilitat de les capes de neu, l’adquisició de dades es va limitar a deu dies consecutius. Es van trobar pulsacions interferomètriques a partir d’un primer anàlisi de l’amplitud de les senyals recollides, les quals eren compatibles amb un model de propagació multicamí a on el reflector es troba per sota del nivell de superfície. Com a conseqüència d’aquests resultats, s’ha desenvolupat un model que reconstrueix el senyal complexe rebut com la suma d’un nombre finit de reflexions, procedents de diferents capes de neu (determinat per mesures locals). La informació interferomètrica es recupera després de l’anàlisi espectral aplicat a les sèries temporals tant de les senyals reals, com de les modelades (lag-hologrames). Trobem que les bandes de freqüències predites pel model són en general consistents amb les dades i que els lag-hologrames mostren repetibilitat per dies diferents. Posteriorment, es realitza un anàlisi de les dades recollides per determinar les capes dominants del perfil de neu seca que contribueixen a les reflexions en banda L, i que a la seva vegada, estan relacionades amb gradents significatius de densitat/permitivitat.Este trabajo evalúa el posible uso de las señales de los sistemas globales de navegación por satélite (GNSS) que se reflejan en la superficie terrestre para la extracción de información geofísica de la criosfera. Con este propósito, el presente estudio se basa en datos recogidos con un reflectómetro GNSS durante dos campañas experimentales, centradas en dos tipos de superficies características de la criosfera: capas de hielo marino y gruesas acumulaciones de nieve seca. Durante el primer experimento, el proceso completo de formación, evolución y fusión del hielo marino fue monitorizado durante más de siete meses en una bahía ubicada en Groenlandia. Este tipo de hielo se caracteriza típicamente por su grosor, concentración y rugosidad. Diferentes observables de las reflexiones GNSS son analizados para tratar de estimar dichas propiedades. El espesor de hielo está relacionado con el nivel de francobordo o borda libre, que a su vez está relacionado con la altura de la superficie de hielo marino. Se ha logrado altimetría de fase precisa, mostrando correlación con un modelo de marea del Ártico. Además, los resultados a largo plazo de la altura elipsoidal siguen la evolución de las mediciones de temperatura de superficie de hielo proporcionadas por MODIS. La temperatura es un parámetro clave en el ritmo de crecimiento del hielo marino. Por otro lado, la presencia de sal en este tipo de hielo modifica sus propiedades dieléctricas, lo que implica variaciones en las amplitudes y fases de los coeficientes complejos de Fresnel con polarizaciones opuestas. Los resultados polarimétricos concuerdan con los valores de concentración de hielo obtenidos mediante inspección visual desde una estación meteorológica cercana. Por último, la forma de la señal reflejada y la dispersión de su fase son evaluadas como potenciales indicadores de la rugosidad de superficie. Para su comparación, se emplean mapas del hielo de la zona experimental. En particular, valores máximos de rugosidad estimada por los observables GNSS coinciden con hielo fijo, que se refiere al hielo anclado a la costa, donde los movimientos de las mareas contribuyen al desarrollo de patrones extraños, grietas y fisuras en su superficie. El segundo experimento se llevó a cabo en la Antártida, monitorizando una área de nieve pristina que es bien conocida para la calibración de instrumentos de teledetección. Debido a la relativa estabilidad de las capas de nieve, la adquisición de datos se limitó a diez días consecutivos. Se encontraron pulsaciones interferométricas a partir de un primer análisis de la amplitud de las señales recibidas, las cuales eran compatibles con un modelo de propagación multicamino donde el reflector se encuentra por debajo del nivel de la superficie. Como consecuencia de estos resultados, se ha desarrollado un modelo que reconstruye la señal recibida como la suma de un número finito de reflexiones, procedentes de diferentes capas de nieve (caracterizados por mediciones locales). La información interferométrica se recupera después del análisis espectral aplicado a las series temporales tanto de las señales reales, como de las modeladas (lag-hologramas). Encontramos que las bandas de frecuencias predichas por el modelo son en general consistentes con los datos y que los lag-hologramas muestran repetibilidad para días diferentes. Posteriormente, se realiza un análisis de los datos recogidos para determinar las capas dominantes del perfil de nieve seca que contribuyen a las reflexiones en banda L, y que a su vez, están relacionadas con gradientes significativos de densidad/permitivida

GNSS reflectometry for land remote sensing applications

Soil moisture and vegetation biomass are two essential parameters from a scienti c and economical point of view. On one hand, they are key for the understanding of the hydrological and carbon cycle. On the other hand, soil moisture is essential for agricultural applications and water management, and vegetation biomass is crucial for regional development programs. Several remote sensing techniques have been used to measure these two parameters. However, retrieving soil moisture and vegetation biomass with the required accuracy, and the appropriate spatial and temporal resolutions still remains a major challenge. The use of Global Navigation Satellite Systems (GNSS) reflected signals as sources of opportunity for measuring soil moisture and vegetation biomass is assessed in this PhD Thesis. This technique, commonly known as GNSS-Reflectometry (GNSS-R), has gained increasing interest among the scienti c community during the last two decades due to its unique characteristics. Previous experimental works have already shown the capabilities of GNSS-R to sense small reflectivity changes on the surface. The use of the co- and cross-polarized reflected signals was also proposed to mitigate nuisance parameters, such as soil surface roughness, in the determination of soil moisture. However, experimental evidence of the suitability of that technique could not be demonstrated. This work analyses from a theoretical and an experimental point of view the capabilities of polarimetric observations of GNSS reflected signals for monitoring soil moisture and vegetation biomass. The Thesis is structured in four main parts. The fi rst part examines the fundamental aspects of the technique and provides a detailed review of the GNSS-R state of the art for soil moisture and vegetation monitoring. The second part deals with the scattering models from land surfaces. A comprehensive description of the formation of scattered signals from rough surfaces is provided. Simulations with current state of the art models for bare and vegetated soils were performed in order to analyze the scattering components of GNSS reflected signals. A simpli ed scattering model was also developed in order to relate in a straightforward way experimental measurements to soil bio-geophysical parameters. The third part reviews the experimental work performed within this research. The development of a GNSS-R instrument for land applications is described, together with the three experimental campaigns carried out in the frame of this PhD Thesis. The analysis of the GNSS-R and ground truth data is also discussed within this part. As predicted by models, it was observed that GNSS scattered signals from natural surfaces are a combination of a coherent and an incoherent scattering components. A data analysis technique was proposed to separate both scattering contributions. The use of polarimetric observations for the determination of soil moisture was demonstrated to be useful under most soil conditions. It was also observed that forests with high levels of biomass could be observed with GNSS reflected signals. The fourth and last part of the Thesis provides an analysis of the technology perspectives. A GNSS-R End-to-End simulator was used to determine the capabilities of the technique to observe di erent soil reflectivity conditions from a low Earth orbiting satellite. It was determined that high accuracy in the estimation of reflectivity could be achieved within reasonable on-ground resolution, as the coherent scattering component is expected to be the predominant one in a spaceborne scenario. The results obtained in this PhD Thesis show the promising potential of GNSS-R measurements for land remote sensing applications, which could represent an excellent complementary observation for a wide range of Earth Observation missions such as SMOS, SMAP, and the recently approved ESA Earth Explorer Mission Biomass.La humedad del suelo y la biomasa de la vegetaci on son dos parametros clave desde un punto de vista tanto cient co como econ omico. Por una parte son esenciales para el estudio del ciclo del agua y del carbono. Por otra parte, la humedad del suelo es esencial para la gesti on de las cosechas y los recursos h dricos, mientras que la biomasa es un par ametro fundamental para ciertos programas de desarrollo. Varias formas de teledetección se han utilizado para la observaci on remota de estos par ametros, sin embargo, su monitorizaci on con la precisi on y resoluci on necesarias es todav a un importante reto tecnol ogico. Esta Tesis evalua la capacidad de medir humedad del suelo y biomasa de la vegetaci on con señales de Sistemas Satelitales de Posicionamiento Global (GNSS, en sus siglas en ingl es) reflejadas sobre la Tierra. La t ecnica se conoce como Reflectometr í a GNSS (GNSS-R), la cual ha ganado un creciente inter es dentro de la comunidad científ ca durante las dos ultimas d ecadas. Experimentos previos a este trabajo ya demostraron la capacidad de observar cambios en la reflectividad del terreno con GNSS-R. El uso de la componente copolar y contrapolar de la señal reflejada fue propuesto para independizar la medida de humedad del suelo de otros par ametros como la rugosidad del terreno. Sin embargo, no se pudo demostrar una evidencia experimental de la viabilidad de la t ecnica. En este trabajo se analiza desde un punto de vista te orico y experimental el uso de la informaci on polarim etrica de la señales GNSS reflejadas sobre el suelo para la determinaci on de humedad y biomasa de la vegetaci on. La Tesis se estructura en cuatro partes principales. En la primera parte se eval uan los aspectos fundamentales de la t ecnica y se da una revisi on detallada del estado del arte para la observaci on de humedad y vegetaci on. En la segunda parte se discuten los modelos de dispersi on electromagn etica sobre el suelo. Simulaciones con estos modelos fueron realizadas para analizar las componentes coherente e incoherente de la dispersi on de la señal reflejada sobre distintos tipos de terreno. Durante este trabajo se desarroll o un modelo de reflexi on simpli cado para poder relacionar de forma directa las observaciones con los par ametros geof sicos del suelo. La tercera parte describe las campañas experimentales realizadas durante este trabajo y discute el an alisis y la comparaci on de los datos GNSS-R con las mediciones in-situ. Como se predice por los modelos, se comprob o experimentalmente que la señal reflejada est a formada por una componente coherente y otra incoherente. Una t ecnica de an alisis de datos se propuso para la separacióon de estas dos contribuciones. Con los datos de las campañas experimentales se demonstr o el bene cio del uso de la informaci on polarim etrica en las señales GNSS reflejadas para la medici on de humedad del suelo, para la mayor a de las condiciones de rugosidad observadas. Tambi en se demostr o la capacidad de este tipo de observaciones para medir zonas boscosas densamente pobladas. La cuarta parte de la tesis analiza la capacidad de la t ecnica para observar cambios en la reflectividad del suelo desde un sat elite en orbita baja. Los resultados obtenidos muestran que la reflectividad del terreno podr a medirse con gran precisi on ya que la componente coherente del scattering ser a la predominante en ese tipo de escenarios. En este trabajo de doctorado se muestran la potencialidades de la t ecnica GNSS-R para observar remotamente par ametros del suelo tan importantes como la humedad del suelo y la biomasa de la vegetaci on. Este tipo de medidas pueden complementar un amplio rango de misiones de observaci on de la Tierra como SMOS, SMAP, y Biomass, esta ultima recientemente aprobada para la siguiente misi on Earth Explorer de la ESA

Optimal column layout for hybrid workloads (VLDB 2020 talk)

Data-intensive analytical applications need to support both efficient reads and writes. However, what is usually a good data layout for an update-heavy workload, is not well-suited for a read-mostly one and vice versa. Modern analytical data systems rely on columnar layouts and employ delta stores to inject new data and updates. We show that for hybrid workloads we can achieve close to one order of magnitude better performance by tailoring the column layout design to the data and query workload. Our approach navigates the possible design space of the physical layout: it organizes each column’s data by determining the number of partitions, their corresponding sizes and ranges, and the amount of buffer space and how it is allocated. We frame these design decisions as an optimization problem that, given workload knowledge and performance requirements, provides an optimal physical layout for the workload at hand. To evaluate this work, we build an in-memory storage engine, Casper, and we show that it outperforms state-of-the-art data layouts of analytical systems for hybrid workloads. Casper delivers up to 2:32 higher throughput for update-intensive workloads and up to 2:14 higher throughput for hybrid workloads. We further show how to make data layout decisions robust to workload variation by carefully selecting the input of the optimization.http://www.vldb.org/pvldb/vol12/p2393-athanassoulis.pdfPublished versio

Digital photo album management techniques: from one dimension to multi-dimension.

Lu Yang.Thesis submitted in: November 2004.Thesis (M.Phil.)--Chinese University of Hong Kong, 2005.Includes bibliographical references (leaves 96-103).Abstracts in English and Chinese.Abstract --- p.iAcknowledgement --- p.ivChapter 1 --- Introduction --- p.1Chapter 1.1 --- Motivation --- p.1Chapter 1.2 --- Our Contributions --- p.3Chapter 1.3 --- Thesis Outline --- p.5Chapter 2 --- Background Study --- p.7Chapter 2.1 --- MPEG-7 Introduction --- p.8Chapter 2.2 --- Image Analysis in CBIR Systems --- p.11Chapter 2.2.1 --- Color Information --- p.13Chapter 2.2.2 --- Color Layout --- p.19Chapter 2.2.3 --- Texture Information --- p.20Chapter 2.2.4 --- Shape Information --- p.24Chapter 2.2.5 --- CBIR Systems --- p.26Chapter 2.3 --- Image Processing in JPEG Frequency Domain --- p.30Chapter 2.4 --- Photo Album Clustering --- p.33Chapter 3 --- Feature Extraction and Similarity Analysis --- p.38Chapter 3.1 --- Feature Set in Frequency Domain --- p.38Chapter 3.1.1 --- JPEG Frequency Data --- p.39Chapter 3.1.2 --- Our Feature Set --- p.42Chapter 3.2 --- Digital Photo Similarity Analysis --- p.43Chapter 3.2.1 --- Energy Histogram --- p.43Chapter 3.2.2 --- Photo Distance --- p.45Chapter 4 --- 1-Dimensional Photo Album Management Techniques --- p.49Chapter 4.1 --- Photo Album Sorting --- p.50Chapter 4.2 --- Photo Album Clustering --- p.52Chapter 4.3 --- Photo Album Compression --- p.56Chapter 4.3.1 --- Variable IBP frames --- p.56Chapter 4.3.2 --- Adaptive Search Window --- p.57Chapter 4.3.3 --- Compression Flow --- p.59Chapter 4.4 --- Experiments and Performance Evaluations --- p.60Chapter 5 --- High Dimensional Photo Clustering --- p.67Chapter 5.1 --- Traditional Clustering Techniques --- p.67Chapter 5.1.1 --- Hierarchical Clustering --- p.68Chapter 5.1.2 --- Traditional K-means --- p.71Chapter 5.2 --- Multidimensional Scaling --- p.74Chapter 5.2.1 --- Introduction --- p.75Chapter 5.2.2 --- Classical Scaling --- p.77Chapter 5.3 --- Our Interactive MDS-based Clustering --- p.80Chapter 5.3.1 --- Principal Coordinates from MDS --- p.81Chapter 5.3.2 --- Clustering Scheme --- p.82Chapter 5.3.3 --- Layout Scheme --- p.84Chapter 5.4 --- Experiments and Results --- p.87Chapter 6 --- Conclusions --- p.94Bibliography --- p.9

Scene Segmentation and Object Classification for Place Recognition

This dissertation tries to solve the place recognition and loop closing problem in a way similar to human visual system. First, a novel image segmentation algorithm is developed. The image segmentation algorithm is based on a Perceptual Organization model, which allows the image segmentation algorithm to ‘perceive’ the special structural relations among the constituent parts of an unknown object and hence to group them together without object-specific knowledge. Then a new object recognition method is developed. Based on the fairly accurate segmentations generated by the image segmentation algorithm, an informative object description that includes not only the appearance (colors and textures), but also the parts layout and shape information is built. Then a novel feature selection algorithm is developed. The feature selection method can select a subset of features that best describes the characteristics of an object class. Classifiers trained with the selected features can classify objects with high accuracy. In next step, a subset of the salient objects in a scene is selected as landmark objects to label the place. The landmark objects are highly distinctive and widely visible. Each landmark object is represented by a list of SIFT descriptors extracted from the object surface. This object representation allows us to reliably recognize an object under certain viewpoint changes. To achieve efficient scene-matching, an indexing structure is developed. Both texture feature and color feature of objects are used as indexing features. The texture feature and the color feature are viewpoint-invariant and hence can be used to effectively find the candidate objects with similar surface characteristics to a query object. Experimental results show that the object-based place recognition and loop detection method can efficiently recognize a place in a large complex outdoor environment

LANDSAT menhaden and thread herring resources investigation

The author has identified the following significant results. The most significant achievement realized thus far has been the successful completion of the data acquisition phase. This success must be attributed to the interest, support, and competency of the participants. The apparent consistency of water color and turbidity condition over time and between test sites at sites of menhaden capture is significant especially since color is readily measured with satellite and aircraft sensors and a LANDSAT MSS based computer model for inferring tubidity has been developed

Intra- and Intersystem Interference in GNSS: Performance Models and Signal Design

The European Galileo, the American Global Positioning System (GPS), and other global navigation satellite systems (GNSSs) transmit direct-sequence spread spectrum (DSSS) signals from space, allowing receivers on Earth to compute their position, velocity, and time (PVT) based on the principle of pseudorange trilateration. However, as multiple satellites and systems transmit signals simultaneously within shared frequency bands, multiple access interference (MAI) in the form of intra- and intersystem interference can affect signal processing at the receiver. To compute a pseudorange, the receiver must estimate synchronization parameters of the respective signal with high resolution. This synchronization is performed in a two-step approach, consisting of signal acquisition (detection) and fine parameter estimation. Most GNSSs rely on asynchronous direct-sequence code-division multiple access (DS-CDMA), assigning different pseudorandom noise (PRN) code to each satellite. This multiple access scheme involves a controlled level of MAI degrading acquisition and parameter estimation performance, which needs to be carefully modeled before launching new signals or raising transmit power levels. The International Telecommunications Union (ITU) regulates that radio frequency compatibility (RFC) of systems, satellites and signals within the radionavigation frequency bands must be ensured, meaning that receiver performance must not be harmed significantly. Conventional receiver performance models are based on the spectral separation coefficient (SSC) between desired and interfering signal, and mostly rely on the idealization that GNSS signals are wide-sense stationary (WSS), circularly-symmetric Gaussian (CSG) random processes. In this work, we propose refined models for performance of coarse and fine estimation of synchronization parameters, taking into account the signals’ wide-sense cyclostationary (WSCS) property and their non-circularity. This is of particular interest in light of the recent signal design trend towards novel coarse/acquisition (C/A) signals with short PRN codes, which are especially vulnerable to MAI but very attractive for the group of mass-market GNSS-enabled electronic devices. Ultimately, our performance model enables the C/A signal designer to minimize the PRN code length while ensuring a given acquisition performance constraint. Moreover, with regard to RFC of an increasing number of navigation systems, satellites, and signals, our detailed models for interference effects on user equipment will allow to make more efficient use of the available radio frequency spectrum