The automated recognition of vehicle interaction is crucial for self-driving, collision avoidance and security surveillance applications. In this paper, we present a novel Long-Short Term Memory Neural Network (LSTM) based method for vehicle trajectory classification. We use Qualitative Trajectory Calculus (QTC) to represent the relative motion between a pair of vehicles. The spatio-temporal features of the interacting vehicles are captured as a sequence of QTC states and then encoded using one hot vector representation. Then, we develop an LSTM network to classify QTC trajectories that represent vehicle pairwise activities. Most of the high performing LSTM models are manually designed and require expertise in hyperparameter configuration. We adapt Bayesian Optimisation method to find an optimal LSTM architecture for classifying QTC trajectories of vehicle interaction. We evaluated our method on three different datasets comprising 7257 trajectories of 9 unique vehicle activities in different traffic scenarios. We demonstrate that our proposed method outperforms the state-of-the-art techniques. Further, we evaluated our approach with a combined dataset of the three datasets and achieved an error rate of no more than 1.79%. Though, our work mainly focuses on vehicle trajectories, the proposed method is generic and can be used on pairwise analysis of other interacting objects
