A data-driven robotic Chinese calligraphy system using convolutional auto-encoder and differential evolution

The Chinese stroke evaluation and generation systems required in an autonomous calligraphy robot play a crucial role in producing high-quality writing results with good diversity. These systems often suffer from inefficiency and non-optima despite of intensive research effort investment by the robotic community. This paper proposes a new learning system to allow a robot to automatically learn to write Chinese calligraphy effectively. In the proposed system, the writing quality evaluation subsystem assesses written strokes using a convolutional auto-encoder network (CAE), which enables the generation of aesthetic strokes with various writing styles. The trained CAE network effectively excludes poorly written strokes through stroke reconstruction, but guarantees the inheritance of information from well-written ones. With the support of the evaluation subsystem, the writing trajectory model generation subsystem is realized by multivariate normal distributions optimized by differential evolution (DE), a type of heuristic optimization search algorithm. The proposed approach was validated and evaluated using a dataset of nine stroke categories; high-quality written strokes have been resulted with good diversity which shows the robustness and efficacy of the proposed approach and its potential in autonomous action-state space exploration for other real-world applications

Use of Automatic Chinese Character Decomposition and Human Gestures for Chinese Calligraphy Robots

Conventional Chinese calligraphy robots often suffer from the limited sizes of predefined font databases, which prevent the robots from writing new characters. This paper presents a robotic handwriting system to address such limitations, which extracts Chinese characters from textbooks and uses a robot’s manipulator to write the characters in a different style. The key technologies of the proposed approach include the following: (1) automatically decomposing Chinese characters into strokes using Harris corner detection technology and (2) matching the decomposed strokes to robotic writing trajectories learned from human gestures. Briefly, the system first decomposes a given Chinese character into a set of strokes and obtains the stroke trajectory writing ability by following the gestures performed by a human demonstrator. Then, it applies a stroke classification method that recognizes the decomposed strokes as robotic writing trajectories. Finally, the robot arm is driven to follow the trajectories and thus write the Chinese character. Seven common Chinese characters have been used in an experiment for system validation and evaluation. The experimental results demonstrate the power of the proposed system, given that the robot successfully wrote all the testing characters in the given Chinese calligraphic style

Integration of an actor-critic model and generative adversarial networks for a Chinese calligraphy robot

As a combination of robotic motion planning and Chinese calligraphy culture, robotic calligraphy plays a significant role in the inheritance and education of Chinese calligraphy culture. Most existing calligraphy robots focus on enabling the robots to learn writing through human participation, such as human–robot interactions and manually designed evaluation functions. However, because of the subjectivity of art aesthetics, these existing methods require a large amount of implementation work from human engineers. In addition, the written results cannot be accurately evaluated. To overcome these limitations, in this paper, we propose a robotic calligraphy model that combines a generative adversarial network (GAN) and deep reinforcement learning to enable a calligraphy robot to learn to write Chinese character strokes directly from images captured from Chinese calligraphic textbooks. In our proposed model, to automatically establish an aesthetic evaluation system for Chinese calligraphy, a GAN is first trained to understand and reconstruct stroke images. Then, the discriminator network is independently extracted from the trained GAN and embedded into a variant of the reinforcement learning method, the “actor-critic model”, as a reward function. Thus, a calligraphy robot adopts the improved actor-critic model to learn to write multiple character strokes. The experimental results demonstrate that the proposed model successfully allows a calligraphy robot to write Chinese character strokes based on input stroke images. The performance of our model, compared with the state-of-the-art deep reinforcement learning method, shows the efficacy of the combination approach. In addition, the key technology in this work shows promise as a solution for robotic autonomous assembly

Stroke Extraction of Chinese Character Based on Deep Structure Deformable Image Registration

Stroke extraction of Chinese characters plays an important role in the field of character recognition and generation. The most existing character stroke extraction methods focus on image morphological features. These methods usually lead to errors of cross strokes extraction and stroke matching due to rarely using stroke semantics and prior information. In this paper, we propose a deep learning-based character stroke extraction method that takes semantic features and prior information of strokes into consideration. This method consists of three parts: image registration-based stroke registration that establishes the rough registration of the reference strokes and the target as prior information; image semantic segmentation-based stroke segmentation that preliminarily separates target strokes into seven categories; and high-precision extraction of single strokes. In the stroke registration, we propose a structure deformable image registration network to achieve structure-deformable transformation while maintaining the stable morphology of single strokes for character images with complex structures. In order to verify the effectiveness of the method, we construct two datasets respectively for calligraphy characters and regular handwriting characters. The experimental results show that our method strongly outperforms the baselines. Code is available at https://github.com/MengLi-l1/StrokeExtraction.Comment: 10 pages, 8 figures, published to AAAI-23 (oral

Visual-based decision for iterative quality enhancement in robot drawing.

Kwok, Ka Wai.Thesis (M.Phil.)--Chinese University of Hong Kong, 2005.Includes bibliographical references (leaves 113-116).Abstracts in English and Chinese.ABSTRACT --- p.iChapter 1. --- INTRODUCTION --- p.1Chapter 1.1 --- Artistic robot in western art --- p.1Chapter 1.2 --- Chinese calligraphy robot --- p.2Chapter 1.3 --- Our robot drawing system --- p.3Chapter 1.4 --- Thesis outline --- p.3Chapter 2. --- ROBOT DRAWING SYSTEM --- p.5Chapter 2.1 --- Robot drawing manipulation --- p.5Chapter 2.2 --- Input modes --- p.6Chapter 2.3 --- Visual-feedback system --- p.8Chapter 2.4 --- Footprint study setup --- p.8Chapter 2.5 --- Chapter summary --- p.10Chapter 3. --- LINE STROKE EXTRACTION AND ORDER ASSIGNMENT --- p.11Chapter 3.1 --- Skeleton-based line trajectory generation --- p.12Chapter 3.2 --- Line stroke vectorization --- p.15Chapter 3.3 --- Skeleton tangential slope evaluation using MIC --- p.16Chapter 3.4 --- Skeleton-based vectorization using Bezier curve interpolation --- p.21Chapter 3.5 --- Line stroke extraction --- p.25Chapter 3.6 --- Line stroke order assignment --- p.30Chapter 3.7 --- Chapter summary --- p.33Chapter 4. --- PROJECTIVE RECTIFICATION AND VISION-BASED CORRECTION --- p.34Chapter 4.1 --- Projective rectification --- p.34Chapter 4.2 --- Homography transformation by selected correspondences --- p.35Chapter 4.3 --- Homography transformation using GA --- p.39Chapter 4.4 --- Visual-based iterative correction example --- p.45Chapter 4.5 --- Chapter summary --- p.49Chapter 5. --- ITERATIVE ENHANCEMENT ON OFFSET EFFECT AND BRUSH THICKNESS --- p.52Chapter 5.1 --- Offset painting effect by Chinese brush pen --- p.52Chapter 5.2 --- Iterative robot drawing process --- p.53Chapter 5.3 --- Iterative line drawing experimental results --- p.56Chapter 5.4 --- Chapter summary --- p.67Chapter 6. --- GA-BASED BRUSH STROKE GENERATION --- p.68Chapter 6.1 --- Brush trajectory representation --- p.69Chapter 6.2 --- Brush stroke modeling --- p.70Chapter 6.3 --- Stroke simulation using GA --- p.72Chapter 6.4 --- Evolutionary computing results --- p.77Chapter 6.5 --- Chapter summary --- p.95Chapter 7. --- BRUSH STROKE FOOTPRINT CHARACTERIZATION --- p.96Chapter 7.1 --- Footprint video capturing --- p.97Chapter 7.2 --- Footprint image property --- p.98Chapter 7.3 --- Experimental results --- p.102Chapter 7.4 --- Chapter summary --- p.109Chapter 8. --- CONCLUSIONS AND FUTURE WORKS --- p.111BIBLIOGRAPHY --- p.11

AutoGraff: towards a computational understanding of graffiti writing and related art forms.

The aim of this thesis is to develop a system that generates letters and pictures with a style that is immediately recognizable as graffiti art or calligraphy. The proposed system can be used similarly to, and in tight integration with, conventional computer-aided geometric design tools and can be used to generate synthetic graffiti content for urban environments in games and in movies, and to guide robotic or fabrication systems that can materialise the output of the system with physical drawing media. The thesis is divided into two main parts. The first part describes a set of stroke primitives, building blocks that can be combined to generate different designs that resemble graffiti or calligraphy. These primitives mimic the process typically used to design graffiti letters and exploit well known principles of motor control to model the way in which an artist moves when incrementally tracing stylised letter forms. The second part demonstrates how these stroke primitives can be automatically recovered from input geometry defined in vector form, such as the digitised traces of writing made by a user, or the glyph outlines in a font. This procedure converts the input geometry into a seed that can be transformed into a variety of calligraphic and graffiti stylisations, which depend on parametric variations of the strokes

StrokeStyles: Stroke-based Segmentation and Stylization of Fonts

We develop a method to automatically segment a font’s glyphs into a set of overlapping and intersecting strokes with the aim of generating artistic stylizations. The segmentation method relies on a geometric analysis of the glyph’s outline, its interior, and the surrounding areas and is grounded in perceptually informed principles and measures. Our method does not require training data or templates and applies to glyphs in a large variety of input languages, writing systems, and styles. It uses the medial axis, curvilinear shape features that specify convex and concave outline parts, links that connect concavities, and seven junction types. We show that the resulting decomposition in strokes can be used to create variations, stylizations, and animations in different artistic or design-oriented styles while remaining recognizably similar to the input font

Research on hybrid manufacturing using industrial robot

The applications of using industrial robots in hybrid manufacturing overcome many restrictions of the conventional manufacturing methods, such as small part building size, long building period, and limited material choices. However, some problems such as the uneven distribution of motion accuracy within robot working volume, the acceleration impact of robot under heavy external loads, few methods and facilities for increasing the efficiency of hybrid manufacturing process are still challenging. This dissertation aims to improve the applications of using industrial robot in hybrid manufacturing by addressing following three categories research issues. The first research issue proposed a novel concept view on robot accuracy and stiffness problem, for making the maximum usage of current manufacturing capability of robot system. Based on analyzing the robot forward/inverse kinematic, the angle error sensitivity of different joint and the stiffness matrix properties of robot, new evaluation formulations are established to help finding the best position and orientation to perform a specific trajectory within the robot\u27s working volume. The second research issue focus on the engineering improvements of robotic hybrid manufacturing. By adopting stereo vision, laser scanning technology and curved surface compensation algorithm, it enhances the automation level and adaptiveness of hybrid manufacturing process. The third research issue extends the robotic hybrid manufacturing process to the broader application area. A mini extruder with a variable pitch and progressive diameter screw is developed for large scale robotic deposition. The proposed robotic deposition system could increase the building efficiency and quality for large-size parts. Moreover, the research results of this dissertation can benefit a wide range of industries, such as automation manufacturing, robot design and 3D printing --Abstract, page iv

ディープラーニングを用いた視覚運動学習による適応的な描画行為

早大学位記番号:新7963早稲田大