K. Tanaka-Ishii, Semiotics of Programming.

Unknow

Peirce's sign theory as an open-source R package.

Throughout Peirce’s writing, we witness his developing vision of a machine that scientists will eventually be able to create. Nadin (2010) raised the question:Why do computer scientists continue to ignore Peirce’s sign theory? A review of the literature on Peirce’s theory and the semiotics machine reveals that many authors discussed the machine;however, they donot differentiate between a physical computer machine and its software. This paper discusses the problematic issues involved in converting Peirce’s theory into a programming language, machine and software application. We demonstrate this challenge by introducing Peirce’s sign theory as a software application that runs under an open-source R environmen

A tangible programming environment model informed by principles of perception and meaning

It is a fundamental Human-Computer Interaction problem to design a tangible programming environment for use by multiple persons that can also be individualised. This problem has its origin in the phenomenon that the meaning an object holds can vary across individuals. The Semiotics Research Domain studies the meaning objects hold. This research investigated a solution based on the user designing aspects of the environment at a time after it has been made operational and when the development team is no longer available to implement the user’s design requirements. Also considered is how objects can be positioned so that the collection of objects is interpreted as a program. I therefore explored how some of the principles of relative positioning of objects, as researched in the domains of Psychology and Art, could be applied to tangible programming environments. This study applied the Gestalt principle of perceptual grouping by proximity to the design of tangible programming environments to determine if a tangible programming environment is possible in which the relative positions of personally meaningful objects define the program. I did this by applying the Design Science Research methodology with five iterations and evaluations involving children. The outcome is a model of a Tangible Programming Environment that includes Gestalt principles and Semiotic theory; Semiotic theory explains that the user can choose a physical representation of the program element that carries personal meaning whereas the Gestalt principle of grouping by proximity predicts that objects can be arranged to appear as if linked to each other.School of ComputingPh. D. (Computer Science

A Semiotic Analysis of If we are holding hands, whose hand am I holding?

This paper presents a system which asserts understanding of complex utterances by semiotic analysis. The question in the title was chosen because: it enquires about a conceptual state of affairs, referring to in two ways; and reaches an answer by subtracting I from we to get the unspoken you. The system introduces concepts, that about which we can speak, defined by repertoires in natural language, including phrasal analysis patterns. The repertoire conceptualization grounds it on a state of affairs against which reasoning can be performed. On matching an utterance, associated intentions are internally vocalized; thus, concepts are informed by more specific concepts. The repertoire conceptualism allows hypothetical states of affairs to be constructed, such as in the title question

Semiotic Aggregation in Deep Learning

Convolutional neural networks utilize a hierarchy of neural network layers. The statistical aspects of information concentration in successive layers can bring an insight into the feature abstraction process. We analyze the saliency maps of these layers from the perspective of semiotics, also known as the study of signs and sign-using behavior. In computational semiotics, this aggregation operation (known as superization) is accompanied by a decrease of spatial entropy: signs are aggregated into supersign. Using spatial entropy, we compute the information content of the saliency maps and study the superization processes which take place between successive layers of the network. In our experiments, we visualize the superization process and show how the obtained knowledge can be used to explain the neural decision model. In addition, we attempt to optimize the architecture of the neural model employing a semiotic greedy technique. To the extent of our knowledge, this is the first application of computational semiotics in the analysis and interpretation of deep neural networks

Critical Programming: Toward a Philosophy of Computing

Beliefs about the relationship between human beings and computing machines and their destinies have alternated from heroic counterparts to conspirators of automated genocide, from apocalyptic extinction events to evolutionary cyborg convergences. Many fear that people are losing key intellectual and social abilities as tasks are offloaded to the everywhere of the built environment, which is developing a mind of its own. If digital technologies have contributed to forming a dumbest generation and ushering in a robotic moment, we all have a stake in addressing this collective intelligence problem. While digital humanities continue to flourish and introduce new uses for computer technologies, the basic modes of philosophical inquiry remain in the grip of print media, and default philosophies of computing prevail, or experimental ones propagate false hopes. I cast this as-is situation as the post-postmodern network dividual cyborg, recognizing that the rational enlightenment of modernism and regressive subjectivity of postmodernism now operate in an empire of extended mind cybernetics combined with techno-capitalist networks forming societies of control. Recent critical theorists identify a justificatory scheme foregrounding participation in projects, valorizing social network linkages over heroic individualism, and commending flexibility and adaptability through life long learning over stable career paths. It seems to reify one possible, contingent configuration of global capitalism as if it was the reflection of a deterministic evolution of commingled technogenesis and synaptogenesis. To counter this trend I offer a theoretical framework to focus on the phenomenology of software and code, joining social critiques with textuality and media studies, the former proposing that theory be done through practice, and the latter seeking to understand their schematism of perceptibility by taking into account engineering techniques like time axis manipulation. The social construction of technology makes additional theoretical contributions dispelling closed world, deterministic historical narratives and requiring voices be given to the engineers and technologists that best know their subject area. This theoretical slate has been recently deployed to produce rich histories of computing, networking, and software, inform the nascent disciplines of software studies and code studies, as well as guide ethnographers of software development communities. I call my syncretism of these approaches the procedural rhetoric of diachrony in synchrony, recognizing that multiple explanatory layers operating in their individual temporal and physical orders of magnitude simultaneously undergird post-postmodern network phenomena. Its touchstone is that the human-machine situation is best contemplated by doing, which as a methodology for digital humanities research I call critical programming. Philosophers of computing explore working code places by designing, coding, and executing complex software projects as an integral part of their intellectual activity, reflecting on how developing theoretical understanding necessitates iterative development of code as it does other texts, and how resolving coding dilemmas may clarify or modify provisional theories as our minds struggle to intuit the alien temporalities of machine processes

Interaction Tangible sur Table, définitions et modèles

International audienceIn recent years, tangible user interfaces, which imply interactions performed with one or several objects, gain more and more interest in research in Human-Computer Interaction (HCI). The tangible object represents a subject or an action. It acts on the system, as an action in classical user interfaces (e.g,. GUI). Interaction on a table, which is a common furniture in everyday life and used in multiple activities (desktop, coffee table, kitchen table, etc.), opens a new way for research and development in HCI. In this article, we present definitions, models, and key issues elicited from the literature that enable understanding and reasoning about the couple within an interactive system. Then, we propose a framework that allows to characterize applications supported by the couple in a domain-independent manner.Depuis quelques années les interfaces tangibles impliquant des interactions réalisées via un objet (ou plusieurs) prennent de plus en plus d’importance dans les recherches en interaction homme-machine. L’objet tangible représente un sujet ou une action ; l’objet agit sur le système, telle une action sur une interface « classique ». L’interaction sur table, c’est-à-dire sur un meuble présent dans la vie courante et utilisé à diverses fins (bureau, table à manger, table de salon, table bar, etc.), ouvre un champ nouveau de recherche et de développement. La mise en exergue, issue de l’état de l’art, des définitions, modèles et problématiques, permet d’abord d’appréhender le couple (table, objet tangible) au sein d’un système interactif. Puis, nous proposons un cadre qui permet de positionner des applications mettant en oeuvre le couple (table, objet tangible). Le cadre est décrit de manière à être utilisé pour positionner des applications indépendamment du domaine

Semiotics & Syringe Pumps

Semiotics can be a useful paradigm in HCI research and yet the cognitive process of semiosis is difficult to uncover and empirically study. Thus in the domain of HCI semiotics has largely remained a descriptive theory, able to provide a theoretical basis for the study of interfaces and interaction, but unable to produce empirical data and generative research. This work, made up of several studies, aims to investigates human errors motivated by the problems of medical interfaces. It takes an empirical approach to investigate the interplay between semiotic signs and human error, attempting to uncover how signs in the interface may affect the use of interactive devices. Interfaces are created from signs, collections of symbols, icons and indices which form a semiotic scene, a meaningful whole through which the user may interact with the underlying system. Therefore interaction with an interface relies heavily on the process of semiosis. The first study in this thesis was a questionnaire study looking at number pads as indurate signs for calcu- lators and telephones. The questionnaire was designed to ascertain how users interpreted number-pads and what features of the number-pad influenced this interpretation. We found that the layout of the numerical buttons on a number-pad had little to do with how the number-pad was perceived, and that the users based their assumptions about the use of the interface based entirely upon the extra contextualizing non-numerical buttons. The wish to use a semiotic paradigm in an empirical study demanded the exploration of a novel experimental methodology. The next set of studies were experiments to see whether the interpretation of indurate signs could be overcome under pressure. Thus we used a computer game based experiments as it was thought that they would allow for the complete control and manipulation of signs within the experimental environment, and encourage more natural semiosis that one might expect from participants in a real life task based explicit ex- periment. In these studies it was found that under pressure participants fell back upon the culturally fossilized meanings of the indurate signs they encountered, suggesting that indurate signs may cause misinterpretation in human-machine interaction if used ineffectively. Overall this thesis makes a contribution to semiotics by exploring the notion of indurate signs and how they are interpreted, by investigating what features of common interfaces affect semiosis, and by attempting to further the course of empirical semiotic studies. This thesis also contributes towards the use of computer games as a research tool by charting the evolution of the game based experimental methodology over the course of this thesis

A Framework For Abstracting, Designing And Building Tangible Gesture Interactive Systems

This thesis discusses tangible gesture interaction, a novel paradigm for interacting with computer that blends concepts from the more popular fields of tangible interaction and gesture interaction. Taking advantage of the human innate abilities to manipulate physical objects and to communicate through gestures, tangible gesture interaction is particularly interesting for interacting in smart environments, bringing the interaction with computer beyond the screen, back to the real world. Since tangible gesture interaction is a relatively new field of research, this thesis presents a conceptual framework that aims at supporting future work in this field. The Tangible Gesture Interaction Framework provides support on three levels. First, it helps reflecting from a theoretical point of view on the different types of tangible gestures that can be designed, physically, through a taxonomy based on three components (move, hold and touch) and additional attributes, and semantically, through a taxonomy of the semantic constructs that can be used to associate meaning to tangible gestures. Second, it helps conceiving new tangible gesture interactive systems and designing new interactions based on gestures with objects, through dedicated guidelines for tangible gesture definition and common practices for different application domains. Third, it helps building new tangible gesture interactive systems supporting the choice between four different technological approaches (embedded and embodied, wearable, environmental or hybrid) and providing general guidance for the different approaches. As an application of this framework, this thesis presents also seven tangible gesture interactive systems for three different application domains, i.e., interacting with the In-Vehicle Infotainment System (IVIS) of the car, the emotional and interpersonal communication, and the interaction in a smart home. For the first application domain, four different systems that use gestures on the steering wheel as interaction means with the IVIS have been designed, developed and evaluated. For the second application domain, an anthropomorphic lamp able to recognize gestures that humans typically perform for interpersonal communication has been conceived and developed. A second system, based on smart t-shirts, recognizes when two people hug and reward the gesture with an exchange of digital information. Finally, a smart watch for recognizing gestures performed with objects held in the hand in the context of the smart home has been investigated. The analysis of existing systems found in literature and of the system developed during this thesis shows that the framework has a good descriptive and evaluative power. The applications developed during this thesis show that the proposed framework has also a good generative power.Questa tesi discute l’interazione gestuale tangibile, un nuovo paradigma per interagire con il computer che unisce i principi dei più comuni campi di studio dell’interazione tangibile e dell’interazione gestuale. Sfruttando le abilità innate dell’uomo di manipolare oggetti fisici e di comunicare con i gesti, l’interazione gestuale tangibile si rivela particolarmente interessante per interagire negli ambienti intelligenti, riportando l’attenzione sul nostro mondo reale, al di là dello schermo dei computer o degli smartphone. Poiché l’interazione gestuale tangibile è un campo di studio relativamente recente, questa tesi presenta un framework (quadro teorico) che ha lo scopo di assistere lavori futuri in questo campo. Il Framework per l’Interazione Gestuale Tangibile fornisce supporto su tre livelli. Per prima cosa, aiuta a riflettere da un punto di vista teorico sui diversi tipi di gesti tangibili che possono essere eseguiti fisicamente, grazie a una tassonomia basata su tre componenti (muovere, tenere, toccare) e attributi addizionali, e che possono essere concepiti semanticamente, grazie a una tassonomia di tutti i costrutti semantici che permettono di associare dei significati ai gesti tangibili. In secondo luogo, il framework proposto aiuta a concepire nuovi sistemi interattivi basati su gesti tangibili e a ideare nuove interazioni basate su gesti con gli oggetti, attraverso linee guida per la definizione di gesti tangibili e una selezione delle migliore pratiche per i differenti campi di applicazione. Infine, il framework aiuta a implementare nuovi sistemi interattivi basati su gesti tangibili, permettendo di scegliere tra quattro differenti approcci tecnologici (incarnato e integrato negli oggetti, indossabile, distribuito nell’ambiente, o ibrido) e fornendo una guida generale per la scelta tra questi differenti approcci. Come applicazione di questo framework, questa tesi presenta anche sette sistemi interattivi basati su gesti tangibili, realizzati per tre differenti campi di applicazione: l’interazione con i sistemi di infotainment degli autoveicoli, la comunicazione interpersonale delle emozioni, e l’interazione nella casa intelligente. Per il primo campo di applicazione, sono stati progettati, sviluppati e testati quattro differenti sistemi che usano gesti tangibili effettuati sul volante come modalità di interazione con il sistema di infotainment. Per il secondo campo di applicazione, è stata concepita e sviluppata una lampada antropomorfica in grado di riconoscere i gesti tipici dell’interazione interpersonale. Per lo stesso campo di applicazione, un secondo sistema, basato su una maglietta intelligente, riconosce quando due persone si abbracciano e ricompensa questo gesto con uno scambio di informazioni digitali. Infine, per l’interazione nella casa intelligente, è stata investigata la realizzazione di uno smart watch per il riconoscimento di gesti eseguiti con oggetti tenuti nella mano. L’analisi dei sistemi interattivi esistenti basati su gesti tangibili permette di dimostrare che il framework ha un buon potere descrittivo e valutativo. Le applicazioni sviluppate durante la tesi mostrano che il framework proposto ha anche un valido potere generativo