1 research outputs found
Performance Bounds for Pairwise Entity Resolution
One significant challenge to scaling entity resolution algorithms to massive
datasets is understanding how performance changes after moving beyond the realm
of small, manually labeled reference datasets. Unlike traditional machine
learning tasks, when an entity resolution algorithm performs well on small
hold-out datasets, there is no guarantee this performance holds on larger
hold-out datasets. We prove simple bounding properties between the performance
of a match function on a small validation set and the performance of a pairwise
entity resolution algorithm on arbitrarily sized datasets. Thus, our approach
enables optimization of pairwise entity resolution algorithms for large
datasets, using a small set of labeled data