The Text Retrieval Conference (TREC) Incident Streams track is a new initiative that aims to mature social
media-based emergency response technology. This initiative advances the state of the art in this area through an
evaluation challenge, which attracts researchers and developers from across the globe. The 2018 edition of the track
provides a standardized evaluation methodology, an ontology of emergency-relevant social media information types,
proposes a scale for information criticality, and releases a dataset containing fifteen test events and approximately
20,000 labeled tweets. Analysis of this dataset reveals a significant amount of actionable information on social
media during emergencies (> 10%). While this data is valuable for emergency response efforts, analysis of the
39 state-of-the-art systems demonstrate a performance gap in identifying this data. We therefore find the current
state-of-the-art is insufficient for emergency responders’ requirements, particularly for rare actionable information
for which there is little prior training data available
