Connected vehicles are becoming commonplace. A constant connection between
vehicles and a central server enables new features and services. This added
connectivity raises the likelihood of exposure to attackers and risks
unauthorized access. A possible countermeasure to this issue are intrusion
detection systems (IDS), which aim at detecting these intrusions during or
after their occurrence. The problem with IDS is the large variety of possible
approaches with no sensible option for comparing them. Our contribution to this
problem comprises the conceptualization and implementation of a testbed for an
automotive real-world scenario. That amounts to a server-side IDS detecting
intrusions into vehicles remotely. To verify the validity of our approach, we
evaluate the testbed from multiple perspectives, including its fitness for
purpose and the quality of the data it generates. Our evaluation shows that the
testbed makes the effective assessment of various IDS possible. It solves
multiple problems of existing approaches, including class imbalance.
Additionally, it enables reproducibility and generating data of varying
detection difficulties. This allows for comprehensive evaluation of real-time,
remote IDS.Comment: Peer-reviewed version accepted for publication in the proceedings of
the 34th ACM/SIGAPP Symposium On Applied Computing (SAC'19