Learning Object Placements For Relational Instructions by Hallucinating Scene Representations

Robots coexisting with humans in their environment and performing services for them need the ability to interact with them. One particular requirement for such robots is that they are able to understand spatial relations and can place objects in accordance with the spatial relations expressed by their user. In this work, we present a convolutional neural network for estimating pixelwise object placement probabilities for a set of spatial relations from a single input image. During training, our network receives the learning signal by classifying hallucinated high-level scene representations as an auxiliary task. Unlike previous approaches, our method does not require ground truth data for the pixelwise relational probabilities or 3D models of the objects, which significantly expands the applicability in practical applications. Our results obtained using real-world data and human-robot experiments demonstrate the effectiveness of our method in reasoning about the best way to place objects to reproduce a spatial relation. Videos of our experiments can be found at https://youtu.be/zaZkHTWFMKMComment: Accepted at the 2020 IEEE International Conference on Robotics and Automation (ICRA). Video at https://www.youtube.com/watch?v=zaZkHTWFMK

Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placement

We present a new method, PARsing And visual GrOuNding (ParaGon), for grounding natural language in object placement tasks. Natural language generally describes objects and spatial relations with compositionality and ambiguity, two major obstacles to effective language grounding. For compositionality, ParaGon parses a language instruction into an object-centric graph representation to ground objects individually. For ambiguity, ParaGon uses a novel particle-based graph neural network to reason about object placements with uncertainty. Essentially, ParaGon integrates a parsing algorithm into a probabilistic, data-driven learning framework. It is fully differentiable and trained end-to-end from data for robustness against complex, ambiguous language input.Comment: To appear in ICRA 202

Grounded Semantic Reasoning for Robotic Interaction with Real-World Objects

Robots are increasingly transitioning from specialized, single-task machines to general-purpose systems that operate in unstructured environments, such as homes, offices, and warehouses. In these real-world domains, robots need to manipulate novel objects while adapting to changes in environments and goals. Semantic knowledge, which concisely describes target domains with symbols, can potentially reveal the meaningful patterns shared between problems and environments. However, existing robots are yet to effectively reason about semantic data encoding complex relational knowledge or jointly reason about symbolic semantic data and multimodal data pertinent to robotic manipulation (e.g., object point clouds, 6-DoF poses, and attributes detected with multimodal sensing). This dissertation develops semantic reasoning frameworks capable of modeling complex semantic knowledge grounded in robot perception and action. We show that grounded semantic reasoning enables robots to more effectively perceive, model, and interact with objects in real-world environments. Specifically, this dissertation makes the following contributions: (1) a survey providing a unified view for the diversity of works in the field by formulating semantic reasoning as the integration of knowledge sources, computational frameworks, and world representations; (2) a method for predicting missing relations in large-scale knowledge graphs by leveraging type hierarchies of entities, effectively avoiding ambiguity while maintaining generalization of multi-hop reasoning patterns; (3) a method for predicting unknown properties of objects in various environmental contexts, outperforming prior knowledge graph and statistical relational learning methods due to the use of n-ary relations for modeling object properties; (4) a method for purposeful robotic grasping that accounts for a broad range of contexts (including object visual affordance, material, state, and task constraint), outperforming existing approaches in novel contexts and for unknown objects; (5) a systematic investigation into the generalization of task-oriented grasping that includes a benchmark dataset of 250k grasps, and a novel graph neural network that incorporates semantic relations into end-to-end learning of 6-DoF grasps; (6) a method for rearranging novel objects into semantically meaningful spatial structures based on high-level language instructions, more effectively capturing multi-object spatial constraints than existing pairwise spatial representations; (7) a novel planning-inspired approach that iteratively optimizes placements of partially observed objects subject to both physical constraints and semantic constraints inferred from language instructions.Ph.D

Semantic Robot Programming for Taskable Goal-Directed Manipulation

Autonomous robots have the potential to assist people to be more productive in factories, homes, hospitals, and similar environments. Unlike traditional industrial robots that are pre-programmed for particular tasks in controlled environments, modern autonomous robots should be able to perform arbitrary user-desired tasks. Thus, it is beneficial to provide pathways to enable users to program an arbitrary robot to perform an arbitrary task in an arbitrary world. Advances in robot Programming by Demonstration (PbD) has made it possible for end-users to program robot behavior for performing desired tasks through demonstrations. However, it still remains a challenge for users to program robot behavior in a generalizable, performant, scalable, and intuitive manner. In this dissertation, we address the problem of robot programming by demonstration in a declarative manner by introducing the concept of Semantic Robot Programming (SRP). In SRP, we focus on addressing the following challenges for robot PbD: 1) generalization across robots, tasks, and worlds, 2) robustness under partial observations of cluttered scenes, 3) efficiency in task performance as the workspace scales up, and 4) feasibly intuitive modalities of interaction for end-users to demonstrate tasks to robots. Through SRP, our objective is to enable an end-user to intuitively program a mobile manipulator by providing a workspace demonstration of the desired goal scene. We use a scene graph to semantically represent conditions on the current and goal states of the world. To estimate the scene graph given raw sensor observations, we bring together discriminative object detection and generative state estimation for the inference of object classes and poses. The proposed scene estimation method outperformed the state of the art in cluttered scenes. With SRP, we successfully enabled users to program a Fetch robot to set up a kitchen tray on a cluttered tabletop in 10 different start and goal settings. In order to scale up SRP from tabletop to large scale, we propose Contextual-Temporal Mapping (CT-Map) for semantic mapping of large scale scenes given streaming sensor observations. We model the semantic mapping problem via a Conditional Random Field (CRF), which accounts for spatial dependencies between objects. Over time, object poses and inter-object spatial relations can vary due to human activities. To deal with such dynamics, CT-Map maintains the belief over object classes and poses across an observed environment. We present CT-Map semantically mapping cluttered rooms with robustness to perceptual ambiguities, demonstrating higher accuracy on object detection and 6 DoF pose estimation compared to state-of-the-art neural network-based object detector and commonly adopted 3D registration methods. Towards SRP at the building scale, we explore notions of Generalized Object Permanence (GOP) for robots to search for objects efficiently. We state the GOP problem as the prediction of where an object can be located when it is not being directly observed by a robot. We model object permanence via a factor graph inference model, with factors representing long-term memory, short-term memory, and common sense knowledge over inter-object spatial relations. We propose the Semantic Linking Maps (SLiM) model to maintain the belief over object locations while accounting for object permanence through a CRF. Based on the belief maintained by SLiM, we present a hybrid object search strategy that enables the Fetch robot to actively search for objects on a large scale, with a higher search success rate and less search time compared to state-of-the-art search methods.PHDElectrical and Computer EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/155073/1/zengzhen_1.pd

The Art of Suicide - The Pain in Paintings

This research projects deals with the question of whether the paintings of artists who have committed suicide is reflective of their mental states both in terms of content and form. It specifically attempts to answer whether the deterioration in mental state from a time of better mental health to the time of their suicide is expressed graphically in the paintings of those artists and whether this can be reliably observed. It was discovered that paintings in the absence of contact with or interpretation by the artists, provided enough information to enable non-expert judges to make reliable global content-related judgements (e.g. destructiveness and hopelessness) as well as form-specific ratings (e.g. lack of detail) that distinguished between paintings created near the time of artists’ suicides and their paintings created at a time of better mental health as well as paintings from artists who were suffering from depression. It was also found that non-expert judges were able to correctly identify paintings that were created just before artists’ suicides as reflecting serious mental health problems. Furthermore, it was discovered that there was a general preference for paintings from depressed artists over the last paintings by artists who have taken their own lives. The implications of these findings for clinical work both in terms of assessment and treatment were discussed. Furthermore, several limitations of this research project were noted and suggestions for future research were provided

FACING EXPERIENCE: A PAINTER’S CANVAS IN VIRTUAL REALITY

Full version unavailable due to 3rd party copyright restrictions.This research investigates how shifts in perception might be brought about through the development of visual imagery created by the use of virtual environment technology. Through a discussion of historical uses of immersion in art, this thesis will explore how immersion functions and why immersion has been a goal for artists throughout history. It begins with a discussion of ancient cave drawings and the relevance of Plato’s Allegory of the Cave. Next it examines the biological origins of “making special.” The research will discuss how this concept, combined with the ideas of “action” and “reaction,” has reinforced the view that art is fundamentally experiential rather than static. The research emphasizes how present-day virtual environment art, in providing a space that engages visitors in computer graphics, expands on previous immersive artistic practices. The thesis examines the technical context in which the research occurs by briefly describing the use of computer science technologies, the fundamentals of visual arts practices, and the importance of aesthetics in new media and provides a description of my artistic practice. The aim is to investigate how combining these approaches can enhance virtual environments as artworks. The computer science of virtual environments includes both hardware and software programming. The resultant virtual environment experiences are technologically dependent on the types of visual displays being used, including screens and monitors, and their subsequent viewing affordances. Virtual environments fill the field of view and can be experienced with a head mounted display (HMD) or a large screen display. The sense of immersion gained through the experience depends on how tracking devices and related peripheral devices are used to facilitate interaction. The thesis discusses visual arts practices with a focus on how illusions shift our cognition and perception in the visual modalities. This discussion includes how perceptual thinking is the foundation of art experiences, how analogies are the foundation of cognitive experiences and how the two intertwine in art experiences for virtual environments. An examination of the aesthetic strategies used by artists and new media critics are presented to discuss new media art. This thesis investigates the visual elements used in virtual environments and prescribes strategies for creating art for virtual environments. Methods constituting a unique virtual environment practice that focuses on visual analogies are discussed. The artistic practice that is discussed as the basis for this research also concentrates on experiential moments and shifts in perception and cognition and references Douglas Hofstadter, Rudolf Arnheim and John Dewey. iv Virtual environments provide for experiences in which the imagery generated updates in real time. Following an analysis of existing artwork and critical writing relative to the field, the process of inquiry has required the creation of artworks that involve tracking systems, projection displays, sound work, and an understanding of the importance of the visitor. In practice, the research has shown that the visitor should be seen as an interlocutor, interacting from a first-person perspective with virtual environment events, where avatars or other instrumental intermediaries, such as guns, vehicles, or menu systems, do not to occlude the view. The aesthetic outcomes of this research are the result of combining visual analogies, real time interactive animation, and operatic performance in immersive space. The environments designed in this research were informed initially by paintings created with imagery generated in a hypnopompic state or during the moments of transitioning from sleeping to waking. The drawings often emphasize emotional moments as caricatures and/or elements of the face as seen from a number of perspectives simultaneously, in the way of some cartoons, primitive artwork or Cubist imagery. In the imagery, the faces indicate situations, emotions and confrontations which can offer moments of humour and reflective exploration. At times, the faces usurp the space and stand in representation as both face and figure. The power of the placement of the caricatures in the paintings become apparent as the imagery stages the expressive moment. The placement of faces sets the scene, establishes relationships and promotes the honesty and emotions that develop over time as the paintings are scrutinized. The development process of creating virtual environment imagery starts with hand drawn sketches of characters, develops further as paintings on “digital canvas”, are built as animated, three-dimensional models and finally incorporated into a virtual environment. The imagery is generated while drawing, typically with paper and pencil, in a stream of consciousness during the hypnopompic state. This method became an aesthetic strategy for producing a snappy straightforward sketch. The sketches are explored further as they are worked up as paintings. During the painting process, the figures become fleshed out and their placement on the page, in essence brings them to life. These characters inhabit a world that I explore even further by building them into three dimensional models and placing them in computer generated virtual environments. The methodology of developing and placing the faces/figures became an operational strategy for building virtual environments. In order to open up the range of art virtual environments, and develop operational strategies for visitors’ experience, the characters and their facial features are used as navigational strategies, signposts and methods of wayfinding in order to sustain a stream of consciousness type of navigation. Faces and characters were designed to represent those intimate moments of self-reflection and confrontation that occur daily within ourselves and with others. They sought to reflect moments of wonderment, hurt, curiosity and humour that could subsequently be relinquished for more practical or purposeful endeavours. They were intended to create conditions in which visitors might reflect upon their emotional state, v enabling their understanding and trust of their personal space, in which decisions are made and the nature of world is determined. In order to extend the split-second, frozen moment of recognition that a painting affords, the caricatures and their scenes are given new dimensions as they become characters in a performative virtual reality. Emotables, distinct from avatars, are characters confronting visitors in the virtual environment to engage them in an interactive, stream of consciousness, non-linear dialogue. Visitors are also situated with a role in a virtual world, where they were required to adapt to the language of the environment in order to progress through the dynamics of a drama. The research showed that imagery created in a context of whimsy and fantasy could bring ontological meaning and aesthetic experience into the interactive environment, such that emotables or facially expressive computer graphic characters could be seen as another brushstroke in painting a world of virtual reality

Spaces of Appearance: Writings on Contemporary Theatre and Performance

This thesis, a collection of previously published materials, reflects a plural and evolving engagement with theatre and performance over the past fifteen years or so: as researcher, writer, editor, teacher, practitioner, spectator. These have rarely been discreet categories for me, but rather different modalities of exploration and enquiry, interrelated spaces encouraging dynamic connectivities, flows and further questions. Section 1 offers critical accounts of the practices of four contemporary theatre directors: Jerzy Grotowski, Robert Wilson, Peter Brook and Ariane Mnouchkine. Section 2 draws on elements of contemporary philosophy and critical thinking to explore the mutable parameters of performance. lt proposes performative mappings of certain unpredictable, energetic events 'in proximity of performance', to borrow Matthew Goulish's phrase: contact, fire, animals, alterity, place. Section 3 contains examples of documentation of performance practices, including a thick description of a mise en scene of a major international theatre production, reflections on process, training and dramaturgy, a performance text with a framing dramaturgical statement, and personal perspectives on particular collaborations. The external Appendix comprises a recently published collection of edited and translated materials concerning five core collaborative projects realised by Ariane Mnouchkine and the Theatre du Soleil at their base in the Cartoucherie de Vincennes, Paris. The core concerns of this thesis include attempts to think through: • the working regimes, poetics and pedagogies of certain directors, usually in collaborative devising contexts within which the creative agency of performers is privileged; • the processes and micro-politics of collaboration, devising, and dramaturgical composition; the dramaturgical implications of trainings, narrative structures, spaces, mise en scene, and of images as multi-layered, dynamic 'fields'; • the predicament and agency of spectators in diverse performance contexts, and the ways in which spectatorial roles are posited or constructed by dramaturgies; • the imbrication of embodiment, movement and perception in performance, and the plurality of modes of perception; • the critical and political functions of theatre and theatre criticism as cultural/social practice and 'art of memory' (de Certeau), of dramaturgies as critical historiographies, and of theatre cultures (and identities) as plural, dynamic, and contested; • performance as concentrated space for inter-subjectivity and the flaring into appearance of the 'face-to-face' (Levinas); the possibility of ethical, 'response-able' encounter and exchange with another; identity as relational and in-process, alterity as productive event, the inter-personal as political; • the poetics and politics of what seems an unthinkable surplus (and constitutive 'outside') to the cognitive reach of many conventional frames and maps in theatre criticism and historiography; an exploration of acts of writing as performative propositions and provocations ('critical fictions') to think the event of meanings at/of the limits of knowledge and subjectivity. This partial listing of recurrent and evolving concerns within the thesis traces a trajectory in my evolution as a writer and thinker, a gradual displacement from the relatively 'solid ground' of theatre studies and theatre history towards more fluid and tentative articulations of the shifting 'lie of the land' in contemporary performance and philosophy. This trajectory reflects a growing fascination with present process, conditions, practices, perceptions 'in the middle', and ways of writing (about) performance as interactive and ephemeral event

Proceedings of the 9th Arab Society for Computer Aided Architectural Design (ASCAAD) international conference 2021 (ASCAAD 2021): architecture in the age of disruptive technologies: transformation and challenges.

The ASCAAD 2021 conference theme is Architecture in the age of disruptive technologies: transformation and challenges. The theme addresses the gradual shift in computational design from prototypical morphogenetic-centered associations in the architectural discourse. This imminent shift of focus is increasingly stirring a debate in the architectural community and is provoking a much needed critical questioning of the role of computation in architecture as a sole embodiment and enactment of technical dimensions, into one that rather deliberately pursues and embraces the humanities as an ultimate aspiration

Geographies of Care and Posthuman Relationality in North American Fiction by Women

Cette thèse met en relief comment la primauté de la relationalité est représentée dans sept romans nord-américains contemporains écrits par des femmes. Pour y arriver, je montre, d’un point de vue critique, comment les notions de « géographies du care » et de « care posthumain » favorisent l’identification de pratiques et d’attitudes d’un « prendre soin » qui facilitent, non sans obstacle, l’appropriation de structures sociales et intimes par le développement d’espaces et de relations de solidarité. Cette étude fait ainsi interagir les pratiques du care et les pratiques discursives afin de mieux cerner « les inégalités structurelles et les enjeux de la domination qui touchent les sujets marginalisés » (Bourgault & Perreault 11). Le premier chapitre déploie le tissage conceptuel de la thèse à l’intersection de la géographie émotionnelle (Davidson, Bondi & Smith; Anderson & Smith), de théories féministes sur l’espace (Shands, Miranne & Young, Massey), des éthiques du care (Laugier, Tronto) et du discours sur le posthumain (Braidotti, Hayles). Situant d’abord les ancrages entre l’espace vécu et le care, je propose un déplacement de la notion de « chez soi » vers celle de « géographies du care » afin de mieux circonscrire les expériences relationnelles imaginées dans les romans. Puis j’introduis le concept du « care posthumain » comme outil critique afin de mieux identifier les nouvelles subjectivités représentées et d’approfondir les apports du care lorsque les relations intersubjectives mettent en scène des figures non humaines et non vivantes. Le deuxième chapitre explore les pratiques soucieuses et spatiales de préservation et de protection dans les romans Housekeeping et Room en portant attention à comment chacun des textes montre les difficultés de recevoir et de donner différentes formes de care en contextes d’oppression patriarcale, de marginalisation sociale et de tensions familiales. Je pose aussi certaines balises théoriques et méthodologiques quant à la lecture et à la configuration, en tant que lectrice privilégiée, des représentations de subjectivités fragiles et de lieux de dominations dans les textes. Le troisième chapitre pose un regard critique sur deux romans qui imaginent un espace domestique marqué par l’exclusion, les dynamiques de pouvoir et le contrôle des corps : The Birth House et Sous béton. Les géographies du care dans ces deux romans montrent les liens complexes entre les notions de proximité relationnelle, d’appartenance et d’autonomie alors que le quotidien des personnages est inscrit dans une dynamique oppressive articulée par des conventions morales, sociales et scientifiques qui tendent à déshumaniser ceux et celles qui ne se conforment pas. Le quatrième chapitre analyse comment le fardeau du trauma et les figures fantomatiques affectent l’expérience relationnelle des personnages ainsi que leur rapport à l’hospitalité et au processus de guérison. Les romans Home et Le ciel de Bay City montrent comment ces figures fantomatiques symbolisent les liens entre mémoire, trauma, et responsabilité, des liens entre passé et présent que le care illumine. Finalement, le cinquième chapitre aborde la notion de « care posthumain » directement, par un retour à Sous béton et à Room, dans lesquels les protagonistes évoluent au fil de relations avec des éléments non humains. J’analyse aussi le roman post-apocalyptique The Year of the Flood, dans lequel les protagonistes usent de stratégies de résistance qui favorisent la solidarité, la guérison et l’adaptation à des débordements technoscientifiques.This dissertation explores how seven contemporary North-American novels written by women illustrate the primacy of relationality. To achieve this goal, I use the notions of “geographies of care” and “posthuman care” critically to uncover, in the texts, gestures, and attitudes of care that facilitate, despite obstacles, the appropriation of social and intimate structures through the development of spaces and relationships of solidarity. This study places caring and discursive practices into dialogue to circumscribe “les inégalités structurelles et les enjeux de domination qui touchent les sujets marginalisés” (Bourgault & Perreault 11). The first chapter consists of a theoretical discussion at the intersection of emotional geography (Davidson, Bondi & Smith, Anderson & Smith), feminist space theory (Shands, Miranne & Young, Massey), care ethics (Laugier, Tronto, DeFalco), and critical posthumanism (Braidotti, Hayle). I expose the interconnections between care and relational space before showing the relevance of geographies of care over the notion of home. Finally, I introduce the idea of posthuman care as a critical tool for reading new subjectivities and for complicating the input of care when intersubjective relations involve the nonhuman. Chapter two explores caring and spatial preservation and protection practices in the novels Housekeeping and Room, by looking at how each text illustrates difficulties of caregiving and care receiving in contexts of patriarchal oppression, social marginalization, and familial tensions. It also sets certain theoretical and methodological beacons regarding the reading and the configuring, as a privileged reader, of representations of fragile subjectivities and spaces of domination in the texts. The third chapter investigates two novels that dramatize domestic spaces marked by exclusion, power dynamics, and control of the body: The Birth House and Sous béton. In both novels the geographies of care expose complex links between notions of relational proximity, belonging and autonomy as the characters’ everyday struggle is characterized by constraining social, moral and scientific conventions that tend to dehumanize those who do not fit. Chapter four analyzes how the burden of trauma and ghostly figures affect the relational experiences of characters, their sense of hospitality and ability to heal. The novels Home and Le ciel de Bay City illustrate how these ghostly figures symbolize and testify to the interconnections between memory, trauma, and responsibility and uncover links between past and present that care illuminates. And finally, Chapter five addresses the notion of “posthuman care” directly by returning to Sous béton and Room, in which the characters evolve through interactions with the nonhuman. I also address the post-apocalyptic novel The Year of the Flood, in which the protagonists make use of strategies of resistance that foster solidarity, healing, and easier adaptation to techno-scientific excesses