GANCCRobot:Generative Adversarial Nets based Chinese Calligraphy Robot

Robotic calligraphy, as a typical application of robot movement planning, is of great significance for the inheritance and education of calligraphy culture. The existing implementations of such robots often suffer from its limited ability for font generation and evaluation, leading to poor writing style diversity and writing quality. This paper proposes a calligraphic robotic framework based on the generative adversarial nets (GAN) to address such limitation. The robot implemented using such framework is able to learn to write fundamental Chinese character strokes with rich diversities and good quality that is close to the human level, without the requirement of specifically designed evaluation functions thanks to the employment of the revised GAN. In particular, the type information of the stroke is introduced as condition information, and the latent codes are applied to maximize the style quality of the generated strokes. Experimental results demonstrate that the proposed model enables a calligraphic robot to successfully write fundamental Chinese strokes based on a given type and style, with overall good quality. Although the proposed model was evaluated in this report using calligraphy writing, the underpinning research is readily applicable to many other applications, such as robotic graffiti and character style conversion

A data-driven robotic Chinese calligraphy system using convolutional auto-encoder and differential evolution

The Chinese stroke evaluation and generation systems required in an autonomous calligraphy robot play a crucial role in producing high-quality writing results with good diversity. These systems often suffer from inefficiency and non-optima despite of intensive research effort investment by the robotic community. This paper proposes a new learning system to allow a robot to automatically learn to write Chinese calligraphy effectively. In the proposed system, the writing quality evaluation subsystem assesses written strokes using a convolutional auto-encoder network (CAE), which enables the generation of aesthetic strokes with various writing styles. The trained CAE network effectively excludes poorly written strokes through stroke reconstruction, but guarantees the inheritance of information from well-written ones. With the support of the evaluation subsystem, the writing trajectory model generation subsystem is realized by multivariate normal distributions optimized by differential evolution (DE), a type of heuristic optimization search algorithm. The proposed approach was validated and evaluated using a dataset of nine stroke categories; high-quality written strokes have been resulted with good diversity which shows the robustness and efficacy of the proposed approach and its potential in autonomous action-state space exploration for other real-world applications

Integration of an actor-critic model and generative adversarial networks for a Chinese calligraphy robot

As a combination of robotic motion planning and Chinese calligraphy culture, robotic calligraphy plays a significant role in the inheritance and education of Chinese calligraphy culture. Most existing calligraphy robots focus on enabling the robots to learn writing through human participation, such as human–robot interactions and manually designed evaluation functions. However, because of the subjectivity of art aesthetics, these existing methods require a large amount of implementation work from human engineers. In addition, the written results cannot be accurately evaluated. To overcome these limitations, in this paper, we propose a robotic calligraphy model that combines a generative adversarial network (GAN) and deep reinforcement learning to enable a calligraphy robot to learn to write Chinese character strokes directly from images captured from Chinese calligraphic textbooks. In our proposed model, to automatically establish an aesthetic evaluation system for Chinese calligraphy, a GAN is first trained to understand and reconstruct stroke images. Then, the discriminator network is independently extracted from the trained GAN and embedded into a variant of the reinforcement learning method, the “actor-critic model”, as a reward function. Thus, a calligraphy robot adopts the improved actor-critic model to learn to write multiple character strokes. The experimental results demonstrate that the proposed model successfully allows a calligraphy robot to write Chinese character strokes based on input stroke images. The performance of our model, compared with the state-of-the-art deep reinforcement learning method, shows the efficacy of the combination approach. In addition, the key technology in this work shows promise as a solution for robotic autonomous assembly

Use of Automatic Chinese Character Decomposition and Human Gestures for Chinese Calligraphy Robots

Conventional Chinese calligraphy robots often suffer from the limited sizes of predefined font databases, which prevent the robots from writing new characters. This paper presents a robotic handwriting system to address such limitations, which extracts Chinese characters from textbooks and uses a robot’s manipulator to write the characters in a different style. The key technologies of the proposed approach include the following: (1) automatically decomposing Chinese characters into strokes using Harris corner detection technology and (2) matching the decomposed strokes to robotic writing trajectories learned from human gestures. Briefly, the system first decomposes a given Chinese character into a set of strokes and obtains the stroke trajectory writing ability by following the gestures performed by a human demonstrator. Then, it applies a stroke classification method that recognizes the decomposed strokes as robotic writing trajectories. Finally, the robot arm is driven to follow the trajectories and thus write the Chinese character. Seven common Chinese characters have been used in an experiment for system validation and evaluation. The experimental results demonstrate the power of the proposed system, given that the robot successfully wrote all the testing characters in the given Chinese calligraphic style

Evaluating Brush Movements for Chinese Calligraphy:A Computer Vision Based Approach

Chinese calligraphy is a popular, highly esteemed art form in the Chinese cultural sphere and worldwide. Ink brushes are the traditional writing tool for Chinese calligraphy and the subtle nuances of brush movements have a great impact on the aesthetics of the written characters. However, mastering the brush movement is a challenging task for many calligraphy learners as it requires many years’ practice and expert supervision. This paper presents a novel approach to help Chinese calligraphy learners to quantify the quality of brush movements without expert involvement. Our approach extracts the brush trajectories from a video stream; it then compares them with example templates of reputed calligraphers to produce a score for the writing quality. We achieve this by first developing a novel neural network to extract the spatial and temporal movement features from the video stream. We then employ methods developed in the computer vision and signal processing domains to track the brush movement trajectory and calculate the score. We conducted extensive experiments and user studies to evaluate our approach. Experimental results show that our approach is highly accurate in identifying brush movements, yielding an average accuracy of 90%, and the generated score is within 3% of errors when compared to the one given by human experts

A Developmental Evolutionary Learning Framework for Robotic Chinese Stroke Writing

The ability of robots to write Chinese strokes, which is recognized as a sophisticated task, involves complicated kinematic control algorithms. The conventional approaches for robotic writing of Chinese strokes often suffer from limited font generation methods, which limits the ability of robots to perform high-quality writing. This paper instead proposes a developmental evolutionary learning framework that enables a robot to learn to write fundamental Chinese strokes. The framework first considers the learning process of robotic writing as an evolutionary easy-to-difficult procedure. Then, a developmental learning mechanism called “Lift-constraint, act and saturate” that stems from developmental robotics is used to determine how the robot learns tasks ranging from simple to difficult by building on the learning results from the easy tasks. The developmental constraints, which include altitude adjustments, number of mutation points, and stroke trajectory points, determine the learning complexity of robot writing. The developmental algorithm divides the evolutionary procedure into three developmental learning stages. In each stage, the stroke trajectory points gradually increase, while the number of mutation points and adjustment altitudes gradually decrease, allowing the learning difficulties involved in these three stages to be categorized as easy, medium, and difficult. Our robot starts with an easy learning task and then gradually progresses to the medium and difficult tasks. Under various developmental constraint setups in each stage, the robot applies an evolutionary algorithm to handle the basic shapes of the Chinese strokes and eventually acquires the ability to write with good quality. The experimental results demonstrate that the proposed framework allows a calligraphic robot to gradually learn to write five fundamental Chinese strokes and also reveal a developmental pattern similar to that of humans. Compared to an evolutionary algorithm without the developmental mechanism, the proposed framework achieves good writing quality more rapidly

Visual-based decision for iterative quality enhancement in robot drawing.

Kwok, Ka Wai.Thesis (M.Phil.)--Chinese University of Hong Kong, 2005.Includes bibliographical references (leaves 113-116).Abstracts in English and Chinese.ABSTRACT --- p.iChapter 1. --- INTRODUCTION --- p.1Chapter 1.1 --- Artistic robot in western art --- p.1Chapter 1.2 --- Chinese calligraphy robot --- p.2Chapter 1.3 --- Our robot drawing system --- p.3Chapter 1.4 --- Thesis outline --- p.3Chapter 2. --- ROBOT DRAWING SYSTEM --- p.5Chapter 2.1 --- Robot drawing manipulation --- p.5Chapter 2.2 --- Input modes --- p.6Chapter 2.3 --- Visual-feedback system --- p.8Chapter 2.4 --- Footprint study setup --- p.8Chapter 2.5 --- Chapter summary --- p.10Chapter 3. --- LINE STROKE EXTRACTION AND ORDER ASSIGNMENT --- p.11Chapter 3.1 --- Skeleton-based line trajectory generation --- p.12Chapter 3.2 --- Line stroke vectorization --- p.15Chapter 3.3 --- Skeleton tangential slope evaluation using MIC --- p.16Chapter 3.4 --- Skeleton-based vectorization using Bezier curve interpolation --- p.21Chapter 3.5 --- Line stroke extraction --- p.25Chapter 3.6 --- Line stroke order assignment --- p.30Chapter 3.7 --- Chapter summary --- p.33Chapter 4. --- PROJECTIVE RECTIFICATION AND VISION-BASED CORRECTION --- p.34Chapter 4.1 --- Projective rectification --- p.34Chapter 4.2 --- Homography transformation by selected correspondences --- p.35Chapter 4.3 --- Homography transformation using GA --- p.39Chapter 4.4 --- Visual-based iterative correction example --- p.45Chapter 4.5 --- Chapter summary --- p.49Chapter 5. --- ITERATIVE ENHANCEMENT ON OFFSET EFFECT AND BRUSH THICKNESS --- p.52Chapter 5.1 --- Offset painting effect by Chinese brush pen --- p.52Chapter 5.2 --- Iterative robot drawing process --- p.53Chapter 5.3 --- Iterative line drawing experimental results --- p.56Chapter 5.4 --- Chapter summary --- p.67Chapter 6. --- GA-BASED BRUSH STROKE GENERATION --- p.68Chapter 6.1 --- Brush trajectory representation --- p.69Chapter 6.2 --- Brush stroke modeling --- p.70Chapter 6.3 --- Stroke simulation using GA --- p.72Chapter 6.4 --- Evolutionary computing results --- p.77Chapter 6.5 --- Chapter summary --- p.95Chapter 7. --- BRUSH STROKE FOOTPRINT CHARACTERIZATION --- p.96Chapter 7.1 --- Footprint video capturing --- p.97Chapter 7.2 --- Footprint image property --- p.98Chapter 7.3 --- Experimental results --- p.102Chapter 7.4 --- Chapter summary --- p.109Chapter 8. --- CONCLUSIONS AND FUTURE WORKS --- p.111BIBLIOGRAPHY --- p.11

Stroke trajectory generation for a robotic Chinese calligrapher.

Lam, Hiu Man.Thesis (M.Phil.)--Chinese University of Hong Kong, 2008.Includes bibliographical references (leaves 84-89).Abstracts in English and Chinese.Chapter Chapter 1: --- Introduction --- p.1Chapter 1.1. --- Overview on Robotics --- p.1Chapter 1.2. --- Literture Review on Art-Robot --- p.1Chapter 1.3. --- Robot artist for Chinese Calligraphy and Paintings --- p.3Chapter 1.4. --- Motivation and Research Objective --- p.4Chapter 1.5. --- Thesis Outline --- p.5Chapter Chapter 2: --- Intelligent Robotic Art System --- p.6Chapter 2.1. --- Previous Configuration --- p.6Chapter 2.1.1. --- 3 DOF Manipulator --- p.7Chapter 2.1.2. --- Digital Image Input System --- p.7Chapter 2.2. --- Hardware Modification --- p.8Chapter 2.2.1. --- Additional Degree of Freedoms --- p.8Chapter 2.2.2. --- Infra-red Sensing System for Manipulator Positioning --- p.9Chapter 2.2.3. --- Axial-rotary Brush --- p.11Chapter 2.2.4. --- Interface program --- p.13Chapter 2.2.5. --- Vibration Reduction --- p.16Chapter Chapter 3: --- Skeletonization Based on Delaunay Triangulation and Bezier Interpolation --- p.18Chapter 3.1. --- Background Theory --- p.20Chapter 3.1.1. --- Smoothed Local Symmetry --- p.20Chapter 3.1.2. --- Delaunay Triangulation --- p.21Chapter 3.1.3. --- Bezier Curve --- p.23Chapter 3.2. --- Algorithm --- p.24Chapter 3.2.1. --- Edge Sampling --- p.24Chapter 3.2.2. --- Triangle Modification --- p.26Chapter 3.2.3. --- Triangle Filtering and Replacement --- p.28Chapter 3.2.4. --- Internal Edge Refinement --- p.30Chapter 3.2.5. --- Skeletal Interpolation --- p.31Chapter 3.3. --- Experiments --- p.32Chapter 3.4. --- Chapter Summary --- p.36Chapter Chapter 4: --- Stroke Segmentation for Chinese Words --- p.37Chapter 4.1. --- Rule-based Spurious Branches Removal --- p.38Chapter 4.1.1. --- Spurious Branch in Stroke Terminal --- p.40Chapter 4.1.2. --- Spurious Branch Caused by Turning Stroke --- p.42Chapter 4.2. --- Stroke Connectivity Determination --- p.44Chapter 4.2.1. --- Gradient of Medial Axis --- p.45Chapter 4.2.2. --- Gradient of Branch Boundary --- p.47Chapter 4.2.3. --- Branch Width --- p.49Chapter 4.2.4. --- Combined Objective Function --- p.50Chapter 4.3. --- Stroke Generation --- p.51Chapter 4.3.1. --- Stroke Connection between Branches --- p.52Chapter 4.3.2. --- Stroke Generation in Stroke Terminal --- p.53Chapter 4.4. --- Experiment Using Intelligent Robotic Art System --- p.54Chapter 4.5. --- Discussion --- p.59Chapter Chapter 5: --- Experimental Acquisition of Brush Footprints --- p.61Chapter 5.1. --- Brush Footprint Extraction --- p.62Chapter 5.2. --- Graphical Interface for Inputting Sample Points of Brush Footprints --- p.64Chapter 5.3. --- Curve Fitting for Brush Footprint Sample Points --- p.70Chapter 5.3.1. --- Curve Fitting Using Genetic Algorithm --- p.70Chapter 5.3.2. --- Curve Fitting by Least Squares Regression --- p.72Chapter 5.4. --- Discussion --- p.74Chapter Chapter 6: --- Trajectory Generation for Robotic Chinese Calligraphy --- p.75Chapter 6.1. --- Stroke Trajectory Searching with According Stroke Width --- p.75Chapter 6.2. --- Improvement in Stroke Trajectory --- p.77Chapter 6.3. --- Experiment --- p.80Conclusion and Future Work --- p.82References --- p.84Appendix --- p.90Chapter 9.1. --- Segmented Strokes of Bada Shanren's Calligraphy --- p.9

A Reduced Classifier Ensemble Approach to Human Gesture Classification for Robotic Chinese Handwriting

The paper presents an approach to applying a classifier ensemble to identify human body gestures, so as to control a robot to write Chinese characters. Robotic handwriting ability requires complicated robotic control algorithms. In particular, the Chinese handwriting needs to consider the relative positions of a character’s strokes. This approach derives the font information from human gestures by using a motion sensing input device. Five elementary strokes are used to form Chinese characters, and each elementary stroke is assigned to a type of human gestures. Then, a classifier ensemble is applied to identify each gesture so as to recognize the characters that gestured by the human demonstrator. The classier ensemble’s size is reduced by feature selection techniques and harmony search algorithm, thereby achieving higher accuracy and smaller ensemble size. The inverse kinematics algorithm converts each stroke’s trajectory to the robot’s motor values that are executed by a robotic arm to draw the entire character. Experimental analysis shows that the proposed approach can allow a human to naturally and conveniently control the robot in order to write many Chinese characters