Reinforcement Learning

Brains rule the world, and brain-like computation is increasingly used in computers and electronic devices. Brain-like computation is about processing and interpreting data or directly putting forward and performing actions. Learning is a very important aspect. This book is on reinforcement learning which involves performing actions to achieve a goal. The first 11 chapters of this book describe and extend the scope of reinforcement learning. The remaining 11 chapters show that there is already wide usage in numerous fields. Reinforcement learning can tackle control tasks that are too complex for traditional, hand-designed, non-learning controllers. As learning computers can deal with technical complexities, the tasks of human operators remain to specify goals on increasingly higher levels. This book shows that reinforcement learning is a very dynamic area in terms of theory and applications and it shall stimulate and encourage new research in this field

Spatio-temporal patterns act as computational mechanisms governing emergent behavior in robotic swarms

Our goal is to control a robotic swarm without removing its swarm-like nature. In other words, we aim to intrinsically control a robotic swarm emergent behavior. Past attempts at governing robotic swarms or their self-coordinating emergent behavior, has proven ineffective, largely due to the swarm's inherent randomness (making it difficult to predict) and utter simplicity (they lack a leader, any kind of centralized control, long-range communication, global knowledge, complex internal models and only operate on a couple of basic, reactive rules). The main problem is that emergent phenomena itself is not fully understood, despite being at the forefront of current research. Research into 1D and 2D Cellular Automata has uncovered a hidden computational layer which bridges the micro-macro gap (i.e., how individual behaviors at the micro-level influence the global behaviors on the macro-level). We hypothesize that there also lie embedded computational mechanisms at the heart of a robotic swarm's emergent behavior. To test this theory, we proceeded to simulate robotic swarms (represented as both particles and dynamic networks) and then designed local rules to induce various types of intelligent, emergent behaviors (as well as designing genetic algorithms to evolve robotic swarms with emergent behaviors). Finally, we analysed these robotic swarms and successfully confirmed our hypothesis; analyzing their developments and interactions over time revealed various forms of embedded spatiotemporal patterns which store, propagate and parallel process information across the swarm according to some internal, collision-based logic (solving the mystery of how simple robots are able to self-coordinate and allow global behaviors to emerge across the swarm)

Temporal search in document streams

In this thesis, we address major challenges in searching temporal document collections. In such collections, documents are created and/or edited over time. Examples of temporal document collections are web archives, news archives, blogs, personal emails and enterprise documents. Unfortunately, traditional IR approaches based on termmatching only can give unsatisfactory results when searching temporal document collections. The reason for this is twofold: the contents of documents are strongly time-dependent, i.e., documents are about events happened at particular time periods, and a query representing an information need can be time-dependent as well, i.e., a temporal query. On the other hand, time-only-based methods fall short when it comes to reasoning about events in social media. During the last few years users create chronologically ordered documents about topics that draw their attention in an ever increasing pace. However, with the vast adoption of social media, new types of marketing campaigns have been developed in order to promote content, i.e. brands, products, celebrities, etc

Analyzing of Gender Behaviors from Paths Using Process Mining: A Shopping Mall Application

[EN] The study presents some results of customer paths¿ analysis in a shopping mall. Bluetooth-based technology is used to collect data. The event log containing spatiotemporal information is analyzed with process mining. Process mining is a technique that enables one to see the whole process contrary to data-centric methods. The use of process mining can provide a readily-understandable view of the customer paths. We installed iBeacon devices, a Bluetooth-based positioning system, in the shopping mall. During December 2017 and January and February 2018, close to 8000 customer data were captured. We aim to investigate customer behaviors regarding gender by using their paths. We can determine the gender of customers if they go to the men¿s bathroom or women¿s bathroom. Since the study has a comprehensive scope, we focused on male and female customers¿ behaviors. This study shows that male and female customers have different behaviors. Their duration and paths, in general, are not similar. In addition, the study shows that the process mining technique is a viable way to analyze customer behavior using Bluetooth-based technology.Dogan, O.; Bayo-Monton, JL.; Fernández Llatas, C.; Oztaysi, B. (2019). Analyzing of Gender Behaviors from Paths Using Process Mining: A Shopping Mall Application. Sensors. 19(3):1-20. https://doi.org/10.3390/s19030557S120193Oosterlinck, D., Benoit, D. F., Baecke, P., & Van de Weghe, N. (2017). Bluetooth tracking of humans in an indoor environment: An application to shopping mall visits. Applied Geography, 78, 55-65. doi:10.1016/j.apgeog.2016.11.005Merad, D., Aziz, K.-E., Iguernaissi, R., Fertil, B., & Drap, P. (2016). Tracking multiple persons under partial and global occlusions: Application to customers’ behavior analysis. Pattern Recognition Letters, 81, 11-20. doi:10.1016/j.patrec.2016.04.011Wu, Y., Wang, H.-C., Chang, L.-C., & Chou, S.-C. (2015). Customer’s Flow Analysis in Physical Retail Store. Procedia Manufacturing, 3, 3506-3513. doi:10.1016/j.promfg.2015.07.672Dogan, O., & Öztaysi, B. (2018). In-store behavioral analytics technology selection using fuzzy decision making. Journal of Enterprise Information Management, 31(4), 612-630. doi:10.1108/jeim-02-2018-0035Hwang, I., & Jang, Y. J. (2017). Process Mining to Discover Shoppers’ Pathways at a Fashion Retail Store Using a WiFi-Base Indoor Positioning System. IEEE Transactions on Automation Science and Engineering, 14(4), 1786-1792. doi:10.1109/tase.2017.2692961Abedi, N., Bhaskar, A., Chung, E., & Miska, M. (2015). Assessment of antenna characteristic effects on pedestrian and cyclists travel-time estimation based on Bluetooth and WiFi MAC addresses. Transportation Research Part C: Emerging Technologies, 60, 124-141. doi:10.1016/j.trc.2015.08.010Mou, S., Robb, D. J., & DeHoratius, N. (2018). Retail store operations: Literature review and research directions. European Journal of Operational Research, 265(2), 399-422. doi:10.1016/j.ejor.2017.07.003Fernandez-Llatas, C., Lizondo, A., Monton, E., Benedi, J.-M., & Traver, V. (2015). Process Mining Methodology for Health Process Tracking Using Real-Time Indoor Location Systems. Sensors, 15(12), 29821-29840. doi:10.3390/s151229769Van der Aalst, W. M. P., van Dongen, B. F., Herbst, J., Maruster, L., Schimm, G., & Weijters, A. J. M. M. (2003). Workflow mining: A survey of issues and approaches. Data & Knowledge Engineering, 47(2), 237-267. doi:10.1016/s0169-023x(03)00066-1Ou-Yang, C., & Winarjo, H. (2011). Petri-net integration – An approach to support multi-agent process mining. Expert Systems with Applications, 38(4), 4039-4051. doi:10.1016/j.eswa.2010.09.066Partington, A., Wynn, M., Suriadi, S., Ouyang, C., & Karnon, J. (2015). Process Mining for Clinical Processes. ACM Transactions on Management Information Systems, 5(4), 1-18. doi:10.1145/2629446Yoo, S., Cho, M., Kim, E., Kim, S., Sim, Y., Yoo, D., … Song, M. (2016). Assessment of hospital processes using a process mining technique: Outpatient process analysis at a tertiary hospital. International Journal of Medical Informatics, 88, 34-43. doi:10.1016/j.ijmedinf.2015.12.018Funkner, A. A., Yakovlev, A. N., & Kovalchuk, S. V. (2017). Towards evolutionary discovery of typical clinical pathways in electronic health records. Procedia Computer Science, 119, 234-244. doi:10.1016/j.procs.2017.11.181Jans, M., Alles, M., & Vasarhelyi, M. (2013). The case for process mining in auditing: Sources of value added and areas of application. International Journal of Accounting Information Systems, 14(1), 1-20. doi:10.1016/j.accinf.2012.06.015Yoshimura, Y., Sobolevsky, S., Ratti, C., Girardin, F., Carrascal, J. P., Blat, J., & Sinatra, R. (2014). An Analysis of Visitors’ Behavior in the Louvre Museum: A Study Using Bluetooth Data. Environment and Planning B: Planning and Design, 41(6), 1113-1131. doi:10.1068/b130047pDe Leoni, M., van der Aalst, W. M. P., & Dees, M. (2016). A general process mining framework for correlating, predicting and clustering dynamic behavior based on event logs. Information Systems, 56, 235-257. doi:10.1016/j.is.2015.07.003Rebuge, Á., & Ferreira, D. R. (2012). Business process analysis in healthcare environments: A methodology based on process mining. Information Systems, 37(2), 99-116. doi:10.1016/j.is.2011.01.003Arroyo, R., Yebes, J. J., Bergasa, L. M., Daza, I. G., & Almazán, J. (2015). Expert video-surveillance system for real-time detection of suspicious behaviors in shopping malls. Expert Systems with Applications, 42(21), 7991-8005. doi:10.1016/j.eswa.2015.06.016Popa, M. C., Rothkrantz, L. J. M., Shan, C., Gritti, T., & Wiggers, P. (2013). Semantic assessment of shopping behavior using trajectories, shopping related actions, and context information. Pattern Recognition Letters, 34(7), 809-819. doi:10.1016/j.patrec.2012.04.015Kang, L., & Hansen, M. (2017). Behavioral analysis of airline scheduled block time adjustment. Transportation Research Part E: Logistics and Transportation Review, 103, 56-68. doi:10.1016/j.tre.2017.04.004Rovani, M., Maggi, F. M., de Leoni, M., & van der Aalst, W. M. P. (2015). Declarative process mining in healthcare. Expert Systems with Applications, 42(23), 9236-9251. doi:10.1016/j.eswa.2015.07.040Fernández-Llatas, C., Benedi, J.-M., García-Gómez, J., & Traver, V. (2013). Process Mining for Individualized Behavior Modeling Using Wireless Tracking in Nursing Homes. Sensors, 13(11), 15434-15451. doi:10.3390/s131115434Van der Aalst, W. M. P., Reijers, H. A., Weijters, A. J. M. M., van Dongen, B. F., Alves de Medeiros, A. K., Song, M., & Verbeek, H. M. W. (2007). Business process mining: An industrial application. Information Systems, 32(5), 713-732. doi:10.1016/j.is.2006.05.003M. Valle, A., A.P. Santos, E., & R. Loures, E. (2017). Applying process mining techniques in software process appraisals. Information and Software Technology, 87, 19-31. doi:10.1016/j.infsof.2017.01.004Juhaňák, L., Zounek, J., & Rohlíková, L. (2019). Using process mining to analyze students’ quiz-taking behavior patterns in a learning management system. Computers in Human Behavior, 92, 496-506. doi:10.1016/j.chb.2017.12.015Sedrakyan, G., De Weerdt, J., & Snoeck, M. (2016). Process-mining enabled feedback: «Tell me what I did wrong» vs. «tell me how to do it right». Computers in Human Behavior, 57, 352-376. doi:10.1016/j.chb.2015.12.040Schoor, C., & Bannert, M. (2012). Exploring regulatory processes during a computer-supported collaborative learning task using process mining. Computers in Human Behavior, 28(4), 1321-1331. doi:10.1016/j.chb.2012.02.016Werner, M., & Gehrke, N. (2015). Multilevel Process Mining for Financial Audits. IEEE Transactions on Services Computing, 8(6), 820-832. doi:10.1109/tsc.2015.2457907De Weerdt, J., Schupp, A., Vanderloock, A., & Baesens, B. (2013). Process Mining for the multi-faceted analysis of business processes—A case study in a financial services organization. Computers in Industry, 64(1), 57-67. doi:10.1016/j.compind.2012.09.010Herbert, L., Hansen, Z. N. L., Jacobsen, P., & Cunha, P. (2014). Evolutionary Optimization of Production Materials Workflow Processes. Procedia CIRP, 25, 53-60. doi:10.1016/j.procir.2014.10.010Yim, J., Jeong, S., Gwon, K., & Joo, J. (2010). Improvement of Kalman filters for WLAN based indoor tracking. Expert Systems with Applications, 37(1), 426-433. doi:10.1016/j.eswa.2009.05.047Delafontaine, M., Versichele, M., Neutens, T., & Van de Weghe, N. (2012). Analysing spatiotemporal sequences in Bluetooth tracking data. Applied Geography, 34, 659-668. doi:10.1016/j.apgeog.2012.04.003Frisby, J., Smith, V., Traub, S., & Patel, V. L. (2017). Contextual Computing : A Bluetooth based approach for tracking healthcare providers in the emergency room. Journal of Biomedical Informatics, 65, 97-104. doi:10.1016/j.jbi.2016.11.008Yoshimura, Y., Krebs, A., & Ratti, C. (2017). Noninvasive Bluetooth Monitoring of Visitors’ Length of Stay at the Louvre. IEEE Pervasive Computing, 16(2), 26-34. doi:10.1109/mprv.2017.33Cao, Q., Jones, D. R., & Sheng, H. (2014). Contained nomadic information environments: Technology, organization, and environment influences on adoption of hospital RFID patient tracking. Information & Management, 51(2), 225-239. doi:10.1016/j.im.2013.11.007Larson, J. S., Bradlow, E. T., & Fader, P. S. (2005). An exploratory look at supermarket shopping paths. International Journal of Research in Marketing, 22(4), 395-414. doi:10.1016/j.ijresmar.2005.09.005Fernandez-Llatas, C., Martinez-Millana, A., Martinez-Romero, A., Benedi, J. M., & Traver, V. (2015). Diabetes care related process modelling using Process Mining techniques. Lessons learned in the application of Interactive Pattern Recognition: coping with the Spaghetti Effect. 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). doi:10.1109/embc.2015.7318809Conca, T., Saint-Pierre, C., Herskovic, V., Sepúlveda, M., Capurro, D., Prieto, F., & Fernandez-Llatas, C. (2018). Multidisciplinary Collaboration in the Treatment of Patients With Type 2 Diabetes in Primary Care: Analysis Using Process Mining. Journal of Medical Internet Research, 20(4), e127. doi:10.2196/jmir.8884De Medeiros, A. K. A., Weijters, A. J. M. M., & van der Aalst, W. M. P. (2007). Genetic process mining: an experimental evaluation. Data Mining and Knowledge Discovery, 14(2), 245-304. doi:10.1007/s10618-006-0061-7Heyer, L. J. (1999). Exploring Expression Data: Identification and Analysis of Coexpressed Genes. Genome Research, 9(11), 1106-1115. doi:10.1101/gr.9.11.1106Yang, W.-S., & Hwang, S.-Y. (2006). A process-mining framework for the detection of healthcare fraud and abuse. Expert Systems with Applications, 31(1), 56-68. doi:10.1016/j.eswa.2005.09.00

Modeling, Predicting and Capturing Human Mobility

Realistic models of human mobility are critical for modern day applications, specifically for recommendation systems, resource planning and process optimization domains. Given the rapid proliferation of mobile devices equipped with Internet connectivity and GPS functionality today, aggregating large sums of individual geolocation data is feasible. The thesis focuses on methodologies to facilitate data-driven mobility modeling by drawing parallels between the inherent nature of mobility trajectories, statistical physics and information theory. On the applied side, the thesis contributions lie in leveraging the formulated mobility models to construct prediction workflows by adopting a privacy-by-design perspective. This enables end users to derive utility from location-based services while preserving their location privacy. Finally, the thesis presents several approaches to generate large-scale synthetic mobility datasets by applying machine learning approaches to facilitate experimental reproducibility