Integration of Computer Vision and Natural Language Processing in Multimedia Robotics Application

Computer vision and natural language processing (NLP) are two active machine learning research areas. However, the integration of these two areas gives rise to a new interdisciplinary field, which is currently attracting more attention of researchers. Research has been carried out to extract the text associated with an image or a video that can assist in making computer vision effective. Moreover, researchers focus on utilizing NLP to extract the meaning of words through the use of computer vision. This concept is widely used in robotics. Although robots should observe the surroundings from different ways of interactions, natural gestures and spoken languages are the most convenient way for humans to interact with the robots. This would be possible only if the robots can understand such types of interactions. In the present paper, the proposed integrated application is utilized for guiding vision-impaired people. As vision is the most essential in the life of a human being, an alternative source that helps in guiding the blind in their movements is highly important. For this purpose, the current paper uses a smartphone with the capabilities of vision, language, and intelligence which has been attached to the blind person to capture the images of their surroundings, and it is associated with a Faster Region Convolutional Neural Network (F-RCNN) based central server to detect the objects in the image to inform the person about them and avoid obstacles in their way. These results are passed to the smartphone which produces a speech output for the guidance of the blinds

Simulations of embodied evolving semiosis: Emergent semantics in artificial environments

As we enter this amazing new world of artificial and virtual systems and environments in the context of human communities, we are interested in the development of systems and environments which have the capacity to grow and evolve their own meanings in the context of this community of interaction. In this paper we analyze the necessary conditions to achieve systems and environments with these properties: 1) a coupled interaction between a system and its environment; 2) an environment with sufficient initial richness and structure to allow for; 3) embodied emergent classification of that environment-system coupling; 4) which is subject to pragmatic selection

A Visualization Tool for the Mini-Robot Khepera: Behaviour Analysis and Optimization

Löffler A, Klahold J, Hußmann M, Rückert U. A Visualization Tool for the Mini-Robot Khepera: Behaviour Analysis and Optimization. In: Floreano D, Nicoud J-D, Mondada F, eds. Proceedings of the 5th International European Conference on Artificial Life (ECAL99). Vol 1674. Lausanne, Switzerland: Springer-Verlag; 1999: 329-333.The design of behavior generating control structures for real robots acting autonomously in a real and changing environment is a complex task. This is in particular true with respect to the debugging process, the documentation of the encountered behavior, its quantitative analysis and the final evaluation. To successfully implement such a behavior, it is vital to couple the synthesis on a simulator and the experiment on a real robot with a thorough analysis. The available simulator tools in general only allow behavioral snapshots and do not provide the option of online interference. In order to cure these shortcomings, a visualization tool for aposteriori graphical analysis of recorded data sets which gives access to all relevant internal states and parameters of the system is presented. The mini-robot Khepera has been chosen as experimentatory platform

Logic programming for deliberative robotic task planning

Over the last decade, the use of robots in production and daily life has increased. With increasingly complex tasks and interaction in different environments including humans, robots are required a higher level of autonomy for efficient deliberation. Task planning is a key element of deliberation. It combines elementary operations into a structured plan to satisfy a prescribed goal, given specifications on the robot and the environment. In this manuscript, we present a survey on recent advances in the application of logic programming to the problem of task planning. Logic programming offers several advantages compared to other approaches, including greater expressivity and interpretability which may aid in the development of safe and reliable robots. We analyze different planners and their suitability for specific robotic applications, based on expressivity in domain representation, computational efficiency and software implementation. In this way, we support the robotic designer in choosing the best tool for his application

Freedom, reason, and holding responsible

In this thesis, I present a novel compatibilist solution to the problem of free will. The presented solution rests on three strategic pillars. The first pillar: It is widely accepted that the justifiability of our common practice of holding others morally responsible is seriously undermined should it turn out that free will is impossible. This threat to the practice is what motivates the discussion of the problem of free will. However, the motivation is not, I will argue, theoretically innocent. It comes part and parcel with certain deeply misleading constraints. In Chapter 1, I challenge the widely accepted background motivation and the associated constraints. The insight generated by the discussion in Chapter 1 allows me to approach the problem of free will from a different angle. The second pillar: The vast majority of philosophers agree that free will can be attributed only to agents that are, in some suitably robust sense, rational; that is, they see rationality as a necessary condition of attributability of free will. In Chapter 2, I formulate an original argument in support of this view. In Chapter 3, I present a thought experiment designed to show that, and how, rationality can be understood as not only a necessary condition but a condition that is, in fact, sufficient for the attributability. I conclude that acting rationally is all there is to acting freely. The third pillar: The logic of my account of how rationality can be understood as a sufficient condition of the attributability of free will is such that it won`t help neutralize the threat that universal causal determinism poses to the justifiability of our practice of holding others morally responsible. This theoretical impotence will be perceived as a weakness of the account. The last two chapters of the thesis are meant to neutralize the perceived weakness. It is generally, and mostly implicitly, assumed that once it has been established that we have free will, there won`t be any other principle reason that undermines the justifiability of our practice of holding others morally responsible. I challenge the assumption. I argue that there is another such principal reason in virtue of which the practice is irremediably unjustifiable. To be able to argue so, I first identify the kind of holding morally responsible that calls for the free will assumption. This is done in Chapter 4. In Chapter 5, I argue that the kind of holding morally responsible that calls for the free will assumption is in principle unjustifiable due to the morally corrupted essence of man. I conclude that the theoretical impotence of my account of free will regarding the issue of the justifiability of our practice of holding others morally responsible cannot be taken as a weakness of the account because the practice is, for a separate reason, irremediably unjustifiable anyway. In Appendix, I offer a separate argument for the claim that man`s moral standing is essentially corrupted. The argument relies on an interpretation of a passage from the New Testament and as such its appeal is somewhat limited. The target audience of the argument is a broadly Christian reader. Because of the limited appeal of the argument, it hasn`t been included in the main body of the thesis

The Mini-Robot Khepera as a Foraging Animate: Synthesis and Analysis of Behaviour

Löffler A, Klahold J, Rückert U. The Mini-Robot Khepera as a Foraging Animate: Synthesis and Analysis of Behaviour. In: Rückert U, Sitte J, Witkowski U, eds. Proceedings of the 5th International Heinz Nixdorf Symposium: Autonomous Minirobots for Research and Edutainment (AMiRE01). Vol 97. Paderborn, Germany: Heinz Nixdorf Institut, Universität Paderborn; 2001: 93-130.The work presented in this paper deals with the development of a methodology for resource-efficient behaviour synthesis on autonomous systems. In this context, a definition of a maximal problem with respect to the resources of a given system is introduced. It is elucidated by means of an exemplary implementation of the solution to such a problem using the mini-robot Khepera as the experimental platform. The described task consists of exploring an unknown and dynamically changing environment, collecting and transporting objects, which are associated with light-sources, and navigating to a home-base. The critical point is represented by the accumulated positioning errors in odometrical path-integration due to slippage. Therefore, adaptive sensor calibration using a specific variant of Kohonen’s algorithm is applied in two cases to extract symbolic, e.g. geometric, information from the sub-symbolic sensor data, which is used to enhance position control by landmark mapping and orientation. In order to successfully handle the arising complex interactions, a heterogeneous control-architecture based on a parallel implementation of basic behaviours coupled by a rule-based central unit is proposed

Foundations and Recent Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Multimodal machine learning is a vibrant multi-disciplinary research field that aims to design computer agents with intelligent capabilities such as understanding, reasoning, and learning through integrating multiple communicative modalities, including linguistic, acoustic, visual, tactile, and physiological messages. With the recent interest in video understanding, embodied autonomous agents, text-to-image generation, and multisensor fusion in application domains such as healthcare and robotics, multimodal machine learning has brought unique computational and theoretical challenges to the machine learning community given the heterogeneity of data sources and the interconnections often found between modalities. However, the breadth of progress in multimodal research has made it difficult to identify the common themes and open questions in the field. By synthesizing a broad range of application domains and theoretical frameworks from both historical and recent perspectives, this paper is designed to provide an overview of the computational and theoretical foundations of multimodal machine learning. We start by defining two key principles of modality heterogeneity and interconnections that have driven subsequent innovations, and propose a taxonomy of 6 core technical challenges: representation, alignment, reasoning, generation, transference, and quantification covering historical and recent trends. Recent technical achievements will be presented through the lens of this taxonomy, allowing researchers to understand the similarities and differences across new approaches. We end by motivating several open problems for future research as identified by our taxonomy