28 research outputs found
Revisiting pre-trained remote sensing model benchmarks: resizing and normalization matters
Research in self-supervised learning (SSL) with natural images has progressed
rapidly in recent years and is now increasingly being applied to and
benchmarked with datasets containing remotely sensed imagery. A common
benchmark case is to evaluate SSL pre-trained model embeddings on datasets of
remotely sensed imagery with small patch sizes, e.g., 32x32 pixels, whereas
standard SSL pre-training takes place with larger patch sizes, e.g., 224x224.
Furthermore, pre-training methods tend to use different image normalization
preprocessing steps depending on the dataset. In this paper, we show, across
seven satellite and aerial imagery datasets of varying resolution, that by
simply following the preprocessing steps used in pre-training (precisely, image
sizing and normalization methods), one can achieve significant performance
improvements when evaluating the extracted features on downstream tasks -- an
important detail overlooked in previous work in this space. We show that by
following these steps, ImageNet pre-training remains a competitive baseline for
satellite imagery based transfer learning tasks -- for example we find that
these steps give +32.28 to overall accuracy on the So2Sat random split dataset
and +11.16 on the EuroSAT dataset. Finally, we report comprehensive benchmark
results with a variety of simple baseline methods for each of the seven
datasets, forming an initial benchmark suite for remote sensing imagery
Navigating the Web of Misinformation: A Framework for Misinformation Domain Detection Using Browser Traffic
The proliferation of misinformation and propaganda is a global challenge,
with profound effects during major crises such as the COVID-19 pandemic and the
Russian invasion of Ukraine. Understanding the spread of misinformation and its
social impacts requires identifying the news sources spreading false
information. While machine learning (ML) techniques have been proposed to
address this issue, ML models have failed to provide an efficient
implementation scenario that yields useful results. In prior research, the
precision of deployment in real traffic deteriorates significantly,
experiencing a decrement up to ten times compared to the results derived from
benchmark data sets. Our research addresses this gap by proposing a graph-based
approach to capture navigational patterns and generate traffic-based features
which are used to train a classification model. These navigational and
traffic-based features result in classifiers that present outstanding
performance when evaluated against real traffic. Moreover, we also propose
graph-based filtering techniques to filter out models to be classified by our
framework. These filtering techniques increase the signal-to-noise ratio of the
models to be classified, greatly reducing false positives and the computational
cost of deploying the model. Our proposed framework for the detection of
misinformation domains achieves a precision of 0.78 when evaluated in real
traffic. This outcome represents an improvement factor of over ten times over
those achieved in previous studies
Assessment of Differentially Private Synthetic Data for Utility and Fairness in End-to-End Machine Learning Pipelines for Tabular Data
Differentially private (DP) synthetic data sets are a solution for sharing
data while preserving the privacy of individual data providers. Understanding
the effects of utilizing DP synthetic data in end-to-end machine learning
pipelines impacts areas such as health care and humanitarian action, where data
is scarce and regulated by restrictive privacy laws. In this work, we
investigate the extent to which synthetic data can replace real, tabular data
in machine learning pipelines and identify the most effective synthetic data
generation techniques for training and evaluating machine learning models. We
investigate the impacts of differentially private synthetic data on downstream
classification tasks from the point of view of utility as well as fairness. Our
analysis is comprehensive and includes representatives of the two main types of
synthetic data generation algorithms: marginal-based and GAN-based. To the best
of our knowledge, our work is the first that: (i) proposes a training and
evaluation framework that does not assume that real data is available for
testing the utility and fairness of machine learning models trained on
synthetic data; (ii) presents the most extensive analysis of synthetic data set
generation algorithms in terms of utility and fairness when used for training
machine learning models; and (iii) encompasses several different definitions of
fairness. Our findings demonstrate that marginal-based synthetic data
generators surpass GAN-based ones regarding model training utility for tabular
data. Indeed, we show that models trained using data generated by
marginal-based algorithms can exhibit similar utility to models trained using
real data. Our analysis also reveals that the marginal-based synthetic data
generator MWEM PGM can train models that simultaneously achieve utility and
fairness characteristics close to those obtained by models trained with real
data.Comment: arXiv admin note: text overlap with arXiv:2106.1024
Poverty rate prediction using multi-modal survey and earth observation data
This work presents an approach for combining household demographic and living
standards survey questions with features derived from satellite imagery to
predict the poverty rate of a region. Our approach utilizes visual features
obtained from a single-step featurization method applied to freely available
10m/px Sentinel-2 surface reflectance satellite imagery. These visual features
are combined with ten survey questions in a proxy means test (PMT) to estimate
whether a household is below the poverty line. We show that the inclusion of
visual features reduces the mean error in poverty rate estimates from 4.09% to
3.88% over a nationally representative out-of-sample test set. In addition to
including satellite imagery features in proxy means tests, we propose an
approach for selecting a subset of survey questions that are complementary to
the visual features extracted from satellite imagery. Specifically, we design a
survey variable selection approach guided by the full survey and image features
and use the approach to determine the most relevant set of small survey
questions to include in a PMT. We validate the choice of small survey questions
in a downstream task of predicting the poverty rate using the small set of
questions. This approach results in the best performance -- errors in poverty
rate decrease from 4.09% to 3.71%. We show that extracted visual features
encode geographic and urbanization differences between regions.Comment: In 2023 ACM SIGCAS/SIGCHI Conference on Computing and Sustainable
Societies (COMPASS 23) Short Papers Trac
Dwelling Type Classification for Disaster Risk Assessment Using Satellite Imagery
Vulnerability and risk assessment of neighborhoods is essential for effective
disaster preparedness. Existing traditional systems, due to dependency on
time-consuming and cost-intensive field surveying, do not provide a scalable
way to decipher warnings and assess the precise extent of the risk at a
hyper-local level. In this work, machine learning was used to automate the
process of identifying dwellings and their type to build a potentially more
effective disaster vulnerability assessment system. First, satellite imageries
of low-income settlements and vulnerable areas in India were used to identify 7
different dwelling types. Specifically, we formulated the dwelling type
classification as a semantic segmentation task and trained a U-net based neural
network model, namely TernausNet, with the data we collected. Then a risk score
assessment model was employed, using the determined dwelling type along with an
inundation model of the regions. The entire pipeline was deployed to multiple
locations prior to natural hazards in India in 2020. Post hoc ground-truth data
from those regions was collected to validate the efficacy of this model which
showed promising performance. This work can aid disaster response organizations
and communities at risk by providing household-level risk information that can
inform preemptive actions.Comment: Accepted for presentation in AI+HADR workshop, Neurips 202
Rapid building damage assessment workflow: An implementation for the 2023 Rolling Fork, Mississippi tornado event
Rapid and accurate building damage assessments from high-resolution satellite
imagery following a natural disaster is essential to inform and optimize first
responder efforts. However, performing such building damage assessments in an
automated manner is non-trivial due to the challenges posed by variations in
disaster-specific damage, diversity in satellite imagery, and the dearth of
extensive, labeled datasets. To circumvent these issues, this paper introduces
a human-in-the-loop workflow for rapidly training building damage assessment
models after a natural disaster. This article details a case study using this
workflow, executed in partnership with the American Red Cross during a tornado
event in Rolling Fork, Mississippi in March, 2023. The output from our
human-in-the-loop modeling process achieved a precision of 0.86 and recall of
0.80 for damaged buildings when compared to ground truth data collected
post-disaster. This workflow was implemented end-to-end in under 2 hours per
satellite imagery scene, highlighting its potential for real-time deployment.Comment: In submission to the 2023 ICCV Humanitarian Assistance and Disaster
Response Worksho
Multimodal Foundation Models for Zero-shot Animal Species Recognition in Camera Trap Images
Due to deteriorating environmental conditions and increasing human activity,
conservation efforts directed towards wildlife is crucial. Motion-activated
camera traps constitute an efficient tool for tracking and monitoring wildlife
populations across the globe. Supervised learning techniques have been
successfully deployed to analyze such imagery, however training such techniques
requires annotations from experts. Reducing the reliance on costly labelled
data therefore has immense potential in developing large-scale wildlife
tracking solutions with markedly less human labor. In this work we propose
WildMatch, a novel zero-shot species classification framework that leverages
multimodal foundation models. In particular, we instruction tune
vision-language models to generate detailed visual descriptions of camera trap
images using similar terminology to experts. Then, we match the generated
caption to an external knowledge base of descriptions in order to determine the
species in a zero-shot manner. We investigate techniques to build instruction
tuning datasets for detailed animal description generation and propose a novel
knowledge augmentation technique to enhance caption quality. We demonstrate the
performance of WildMatch on a new camera trap dataset collected in the
Magdalena Medio region of Colombia.Comment: 18 pages, 9 figure