A data-driven robotic Chinese calligraphy system using convolutional auto-encoder and differential evolution

The Chinese stroke evaluation and generation systems required in an autonomous calligraphy robot play a crucial role in producing high-quality writing results with good diversity. These systems often suffer from inefficiency and non-optima despite of intensive research effort investment by the robotic community. This paper proposes a new learning system to allow a robot to automatically learn to write Chinese calligraphy effectively. In the proposed system, the writing quality evaluation subsystem assesses written strokes using a convolutional auto-encoder network (CAE), which enables the generation of aesthetic strokes with various writing styles. The trained CAE network effectively excludes poorly written strokes through stroke reconstruction, but guarantees the inheritance of information from well-written ones. With the support of the evaluation subsystem, the writing trajectory model generation subsystem is realized by multivariate normal distributions optimized by differential evolution (DE), a type of heuristic optimization search algorithm. The proposed approach was validated and evaluated using a dataset of nine stroke categories; high-quality written strokes have been resulted with good diversity which shows the robustness and efficacy of the proposed approach and its potential in autonomous action-state space exploration for other real-world applications

Solving Robotic Trajectory Sequential Writing Problem via Learning Character’s Structural and Sequential Information

The writing sequence of numerals or letters often affects aesthetic aspects of the writing outcomes. As such, it remains a challenge for robotic calligraphy systems to perform, mimicking human writers’ implicit intention. This article presents a new robot calligraphy system that is able to learn writing sequences with limited sequential information, producing writing results compatible to human writers with good diversity. In particular, the system innovatively applies a gated recurrent unit (GRU) network to generate robotic writing actions with the support of a prelabeled trajectory sequence vector. Also, a new evaluation method is proposed that considers the shape, trajectory sequence, and structural information of the writing outcome, thereby helping ensure the writing quality. A swarm optimization algorithm is exploited to create an optimal set of parameters of the proposed system. The proposed approach is evaluated using Arabic numerals, and the experimental results demonstrate the competitive writing performance of the system against state-of-the-art approaches regarding multiple criteria (including FID, MAE, PSNR, SSIM, and PerLoss), as well as diversity performance concerning variance and entropy. Importantly, the proposed GRU-based robotic motion planning system, supported with swarm optimization can learn from a small dataset, while producing calligraphy writing with diverse and aesthetically pleasing outcomes

A Robotic Writing Framework-Learning Human Aesthetic Preferences via Human-Machine Interactions

Intelligent robots are required to fully understand human intentions and operations in order to support or collaborate with humans to complete complicated tasks, which is typically implemented by employing human-machine interaction techniques. This paper proposes a new robotic learning framework to perform numeral writing tasks by investigating human-machine interactions with human preferences. In particular, the framework implements a trajectory generative module using a generative adversarial network (GAN)-based method and develops a human preference feedback system to enable the robot to learn human preferences. In addition, a convolutional neural network, acting as a discriminative network, classifies numeral images to support the development of the basic numeral writing ability, and another convolutional neural network, acting as a human preference network, learns a human user’s aesthetic preference by taking the feedback on two written numerical images during the training process. The experimental results show that the written numerals based on the preferences of ten users were different from those of the training data set and that the writing models with the preferences from different users generate numerals in different styles, as evidenced by the Fréchet inception distance (FID) scores. The FID scores of the proposed framework with a preference network were noticeably greater than those of the framework without a preference network. This phenomenon indicates that the human-machine interactions effectively guided the robotic system to learn different writing styles. These results prove that the proposed approach is able to enable the calligraphy robot to successfully write numerals in accordance with the preferences of a human user

An LSTM Based Generative Adversarial Architecture for Robotic Calligraphy Learning System

Robotic calligraphy is a very challenging task for the robotic manipulators, which can sustain industrial manufacturing. The active mechanism of writing robots require a large sized training set including sequence information of the writing trajectory. However, manual labelling work on those training data may cause the time wasting for researchers. This paper proposes a machine calligraphy learning system using a Long Short-Term Memory (LSTM) network and a generative adversarial network (GAN), which enables the robots to learn and generate the sequences of Chinese character stroke (i.e., writing trajectory). In order to reduce the size of the training set, a generative adversarial architecture combining an LSTM network and a discrimination network is established for a robotic manipulator to learn the Chinese calligraphy regarding its strokes. In particular, this learning system converts Chinese character stroke image into the trajectory sequences in the absence of the stroke trajectory writing sequence information. Due to its powerful learning ability in handling motion sequences, the LSTM network is used to explore the trajectory point writing sequences. Each generation process of the generative adversarial architecture contains a number of loops of LSTM. In each loop, the robot continues to write by following a new trajectory point, which is generated by LSTM according to the previously written strokes. The written stroke in an image format is taken as input to the next loop of the LSTM network until the complete stroke is finally written. Then, the final output of the LSTM network is evaluated by the discriminative network. In addition, a policy gradient algorithm based on reinforcement learning is employed to aid the robot to find the best policy. The experimental results show that the proposed learning system can effectively produce a variety of high-quality Chinese stroke writing

Solving Robotic Trajectory Sequential Writing Problem via Learning Character’s Structural and Sequential Information

The writing sequence of numerals or letters often affects aesthetic aspects of the writing outcomes. As such, it remains a challenge for robotic calligraphy systems to perform, mimicking human writers’ implicit intention. This article presents a new robot calligraphy system that is able to learn writing sequences with limited sequential information, producing writing results compatible to human writers with good diversity. In particular, the system innovatively applies a gated recurrent unit (GRU) network to generate robotic writing actions with the support of a prelabeled trajectory sequence vector. Also, a new evaluation method is proposed that considers the shape, trajectory sequence, and structural information of the writing outcome, thereby helping ensure the writing quality. A swarm optimization algorithm is exploited to create an optimal set of parameters of the proposed system. The proposed approach is evaluated using Arabic numerals, and the experimental results demonstrate the competitive writing performance of the system against state-of-the-art approaches regarding multiple criteria (including FID, MAE, PSNR, SSIM, and PerLoss), as well as diversity performance concerning variance and entropy. Importantly, the proposed GRU-based robotic motion planning system, supported with swarm optimization can learn from a small dataset, while producing calligraphy writing with diverse and aesthetically pleasing outcomes

Intelligent Process Monitoring of Laser-Induced Graphene Production with Deep Transfer Learning

Three-dimensional graphene has been increasingly used in many applications due to its superior properties. The laser-induced graphene (LIG) technique is an effective way to produce 3-D graphene by combining graphene preparation and patterning into a single step using direct laser writing. However, the variation in process parameters and environment could largely affect the formation and crystallization quality of 3-D graphene. This article develops a vision and deep transfer learning-based processing monitoring system for LIG production. To solve the problem of limited labeled data, novel convolutional de-noising auto-encoder (CDAE)-based unsupervised learning is developed to utilize the available unlabeled images. The learned weights from CDAE are then transferred to a Gaussian convolutional deep belief network (GCDBN) model for further fine-tuning with a very small amount of labeled images. The experimental results show that the proposed method can achieve the state-of-art performance of precise and robust monitoring for the quality of the LIG formation

The Problem of Expression in Generative and Human Drawing Agents

Drawing is considered one of the most accessible forms of expression. Children as young as four years of age as well as artists with more advanced technical skills can use simple drawing lines to effectively communicate abstract concepts and express emotional states. According to recent theories of embodiment rooted in the field of empirical aesthetics, an audience’s aesthetic experience is conditioned by the artists’ emotional states and intentions expressed through their works. This is evident especially when the art work involves brush strokes and drawing lines. Within this paradigm, I argue that investigating the relation between the artist and the audience becomes particularly relevant when the drawing agent is artificial. In that case, what emotional state and what agency is the audience relating to? This thesis examines drawings as a means to express emotions by evaluating how a naive audience experienced drawings made by human and artificial drawing agents. In particular, it asks if there are substantial differences between the way we “perceive” and “feel” about human made strokes as opposed to artificially generated ones. For this purpose, four studies were conducted combining methods from empirical aesthetics and Machine Learning field. The studies conducted with human participants showed that naive sketchers unintentionally conveyed information about their individual differences in emotional states (i.e., anxiety level) to a naive crowd by using figurative drawings. Similar effects were observed using abstract drawing lines made by naive sketchers whose mood changed while listening to music. Further analysis conducted at the stroke level provided insight into the interdependencies between the sketchers’ emotional states, the music, and the drawing process. Finally, the thesis investigated people’s ability to distinguish between generative and human-made strokes within theories of dual-processing reasoning. The thesis contributes to the art made by machines debate by evaluating the extent to which a generative model could learn from the human drawing process and generate expressive drawing strokes. The datasets and the models produced during this work add further to the field of the computational arts and Human Computer Interaction, particularly focusing on non-verbal expression