Realizing Context-Aware Services through Intelligent Mobile Data Analysis

In recent years there has been a rapid development of mobile technologies, Internet of Things (IoT) and cellular network infrastructures. In combination with the fast evolution of data analysis, and particularly within machine learning, this has led to new unprecedented opportunities for building smart environments. Central to smart environments lie context-aware systems. These systems sense the users environment, analyse the sensed data to deduce new insights and, from this, provide content, experiences, help and suggestions to the users. Accessing and providing such services have never been easier, since more than 6.5 billion people in the world own smartphones, a platform prime for supporting context-aware services. A topical form of context-aware services, when using smartphones as sensing equipment, are so-called location-aware services. Their particular characteristic is the utilization of the user’s location to provide useful suggestions related to their surroundings. Location-aware services, in particular, services used in retail (shopping) and public transport, are the main focus of this thesis. That are two domains, people generally interact with on a daily basis. In retail and shopping, so-called mobile recommender systems (MRS) are used to perform recommendations such as points of interest or products in the user’s proximity. Crucial to these services are the accurate real-time locations of both, the user and the products they suggest. The location of the user can be provided using a real-time location system (RTLS) tracking the user’s smartphone. However, an accurate, efficient solution to the location of the product is not as readily available. In public transport on the other hand, the detection of the mobile context of vehicles and their passengers is key to realize intelligent transportation systems (ITS). A topical example of this is the in-vehicle presence of a passenger, essential for context-aware services such as automated-ticketing. In order for these systems to work, the accuracy of the solution needs to be incredibly high, something, the state-of-the-art solutions available today still lack. To support these next-generation location-aware services, advanced context reasoning techniques, i.e., algorithms extracting useful information from the data sensed from the user’s environment, are of utmost importance. Machine learning algorithms form a very promising category of such contextreasoning techniques. In consequence, most of the work done in this Ph.D. project centers on them. Particularly, machine learning technology is used in this thesis to address challenges regarding context-aware services within the above mentioned fields retail and public transport. Altogether, we provided the following four contributions: The first contribution introduced in this thesis, is an automatic product localisation algorithm. The product locator proposed in this thesis infers the location of the products in a store by accumulating the locations at which customers stop when picking up products as well as the list of purchased products. A simulation-based environment shows that 99.9% out of 8, 000 products in a typical large Norwegian grocery store can be correctly located by aggregating data from customers over a 12-day period. The second contribution presented in this thesis is DeepMatch, a highly accurate in-vehicle presence detection algorithm. The approach is utilizing the smartphone of a passenger to analyze and match the sensor event streams of the device against the streams of sensors embedded in an on-board reference unit installed in public transportation vehicles. The matching is facilitated by a new deep learning model employed in a distributed fashion, where the feature extraction and dimensionality reduction is offloaded to the smartphones and the reference unit, while the matching is performed on a remote server. The approach achieved an invehicle prediction accuracy of 0.9781 on a dataset consisting of real data gathered by volunteers. The third contribution builds on the second one. It consists of the method DeepMatch2, the successor of Deepmatch, that increases its accuracy from 0.9781 to 0.9851. In addition, the algorithm improves its efficiency, effectively reducing the amount of data needed by the model by a factor of four. Furthermore, we propose a travelling user inference system based on Deep- Match2 with the ability to infer if and for which period of time a passenger makes a trip in a public transport vehicle with a very low error rate. The fourth contribution of this thesis is Ataraxis, a solution to hardwareless in-vehicle presence prediction. Through the collaboration with Public Transportation Authorities, we learned that some PTAs do not have the control to decide autonomously about the hardware that is installed in the vehicles they use. Therefore, the hardware-based solution proposed in Deep- Match and DeepMatch2, i.e., the installation of additional hardware in the vehicles, is not always suitable. Ataraxis addresses this challenge. A deep convolutional neural network to detect the transport mode of users from the sensor events generated by ordinary smartphones was developed. The user mode is used in combination with a GPS trace of the user and nearby public transport vehicles in order to infer the in-vehicle presence of the user. The deep learning model created for Ataraxis achieved an F1 Score of 98.69% when classifying the four user modes driving a car, riding a bike walking and using public transport. To summarize, in this thesis, we contribute applied research through several learning algorithms, system designs, and software solutions in order to enhance and improve the intelligence and quality of location-based contextaware services within retail and public transport. Furthermore, we show through extensive empirical experiments that the proposed approaches can be used in practice without negatively impacting the users smartphones. It answered the main research objective as well as provided several business critical patents to the industrial partner

Automated Product Localization through Mobile Data Analysis

Recent developments in the field of indoor RealTime Locating Systems (RTLS) using mobile devices stimulate decision support for users. For instance, smartphone-based navigation in shops can enable location-aware recommendations of certain products to customers. An impeding factor to realize such systems is that they need the exact position of products. Existing product localization solutions, however, are based on tagging or manual location registering which tend to be quite costly and laborious. In this paper, we propose an automated product localization approach solving this problem. Our system infers the location of products based on the results of accumulating two sets of customer data, i.e., the locations at which the customers stop for picking up items as well as the list of the items, they purchase. These two data sets are accumulated for a large number of users, making it possible to build correct mappings between the products and their positions. We introduce a basic version of our localization algorithm and two extensions. One helps to improve calculating the position of relocated products while the other one fosters a faster localization using a smaller number of user data sets. We discuss the results of various simulation runs which give evidence that our system has a good potential to work in practice

DEEPMATCH2: A comprehensive deep learning-based approach for in-vehicle presence detection

The accurate detection of the mobile context information of public transportation vehicles and their passengers is a key feature to realize intelligent transportation systems. A topical example is in-vehicle presence detection that can, e.g., be used to ticket passengers automatically. Unfortunately, most existing solutions in this field suffer from low spatiotemporal accuracy which impedes their use in practice. In previous work, we addressed this challenge through a deep learning-based framework, called DeepMatch, that allows us to detect in-vehicle presence with a high degree of accuracy. DeepMatch utilizes the smartphone of a passenger to analyse and match the event streams of its own sensors with the event streams of counterpart sensors provided by a reference unit that is installed inside the vehicle. This is achieved through a new learning model architecture using Stacked Convolutional Autoencoders to compress sensor input streams by feature extraction and dimensionality reduction as well as a deep convolutional neural network to match the streams of the user phone and the reference device. The sensor stream compression is offloaded to the smartphone, while the matching is performed in a server. In this paper, we introduce DeepMatch2. It is an amended version of DeepMatch that reduces the amount of data to be transferred from the user and reference devices to the server by the factor of four. Further, DeepMatch2 improves the already good accuracy of DeepMatch from 97.81% to 98.51%. Moreover, we propose a travel inference algorithm, based on DeepMatch2, to detect the duration of whole passenger trips in public transport vehicles with a high degree of precision. This is needed to create intelligent and highly reliable auto-ticketing systems. Thanks to the high accuracy of 98.51% by DeepMatch2, the inferences can be carried out with a negligible error rate

DEEPMATCH2: A comprehensive deep learning-based approach for in-vehicle presence detection

The accurate detection of the mobile context information of public transportation vehicles and their passengers is a key feature to realize intelligent transportation systems. A topical example is in-vehicle presence detection that can, e.g., be used to ticket passengers automatically. Unfortunately, most existing solutions in this field suffer from low spatiotemporal accuracy which impedes their use in practice. In previous work, we addressed this challenge through a deep learning-based framework, called DeepMatch, that allows us to detect in-vehicle presence with a high degree of accuracy. DeepMatch utilizes the smartphone of a passenger to analyse and match the event streams of its own sensors with the event streams of counterpart sensors provided by a reference unit that is installed inside the vehicle. This is achieved through a new learning model architecture using Stacked Convolutional Autoencoders to compress sensor input streams by feature extraction and dimensionality reduction as well as a deep convolutional neural network to match the streams of the user phone and the reference device. The sensor stream compression is offloaded to the smartphone, while the matching is performed in a server. In this paper, we introduce DeepMatch2. It is an amended version of DeepMatch that reduces the amount of data to be transferred from the user and reference devices to the server by the factor of four. Further, DeepMatch2 improves the already good accuracy of DeepMatch from 97.81% to 98.51%. Moreover, we propose a travel inference algorithm, based on DeepMatch2, to detect the duration of whole passenger trips in public transport vehicles with a high degree of precision. This is needed to create intelligent and highly reliable auto-ticketing systems. Thanks to the high accuracy of 98.51% by DeepMatch2, the inferences can be carried out with a negligible error rate

Automated Product Localization through Mobile Data Analysis

Recent developments in the field of indoor RealTime Locating Systems (RTLS) using mobile devices stimulate decision support for users. For instance, smartphone-based navigation in shops can enable location-aware recommendations of certain products to customers. An impeding factor to realize such systems is that they need the exact position of products. Existing product localization solutions, however, are based on tagging or manual location registering which tend to be quite costly and laborious. In this paper, we propose an automated product localization approach solving this problem. Our system infers the location of products based on the results of accumulating two sets of customer data, i.e., the locations at which the customers stop for picking up items as well as the list of the items, they purchase. These two data sets are accumulated for a large number of users, making it possible to build correct mappings between the products and their positions. We introduce a basic version of our localization algorithm and two extensions. One helps to improve calculating the position of relocated products while the other one fosters a faster localization using a smaller number of user data sets. We discuss the results of various simulation runs which give evidence that our system has a good potential to work in practice