51 research outputs found
Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks
Data contamination has become prevalent and challenging with the rise of
models pretrained on large automatically-crawled corpora. For closed models,
the training data becomes a trade secret, and even for open models, it is not
trivial to detect contamination. Strategies such as leaderboards with hidden
answers, or using test data which is guaranteed to be unseen, are expensive and
become fragile with time. Assuming that all relevant actors value clean test
data and will cooperate to mitigate data contamination, what can be done? We
propose three strategies that can make a difference: (1) Test data made public
should be encrypted with a public key and licensed to disallow derivative
distribution; (2) demand training exclusion controls from closed API holders,
and protect your test data by refusing to evaluate without them; (3) avoid data
which appears with its solution on the internet, and release the web-page
context of internet-derived data along with the data. These strategies are
practical and can be effective in preventing data contamination.Comment: Accepted to EMNLP 202
+Spaces: Intelligent Virtual Spaces for eGovernment
Intelligent Environments most commonly take a physical form such as homes, offices, hotels, restaurants, shops, that are equipped with advanced networked computer based systems, which enable better or new lifestyles for people. However, Intelligent Environments can also take the form of virtual online spaces such as SecondLife, which can both mimic the real world and provide functionalities which could not be provided in reality, such as advanced simulations and movement. There is the growing trend for people to spend more time in such virtual environments and, to these ends, this work in progress paper reports on a new project, +Spaces which is developing a range of virtual world tools for e-government applications, and presents some of the concepts and technical challenges involved in creating these intelligent virtual spaces for e-government. © 2010 IEEE
- …