Augmented Reality (AR) enhances a user's perception of the real world by enhancing it with specific objects that are over-imposed to the real objects. The virtual objects display information that the user may use. The information conveyed by the virtual objects helps a user perform real-world tasks. An augmented reality application is the assembly, maintenance, and repair of complex machinery or common goods. Instructions might be easier to understand if they were available, not as manuals with text and pictures, but rather as 3-D drawings superimposed upon the actual equipment, showing step-by-step the tasks that need to be done and how to do them. These superimposed 3-D drawings can be animated, making the directions even more explicit. This paper describes a proof-of-concept AR application to be used for office printer service operation

Ana-Maria IONESCU

Florin Bogdan MARIN

Ionel PETREA

The Annals of “Dunarea de Jos” University of Galati Fascicle IX Metallurgy and Materials Science

Directory of Open Access Journals

FAC
UL
TA
TE
A
D
E
M
E
TA
LU
RG
IE, ŞTIINŢA MATER
IA
L
EL
O
R
ŞIM
EDIU
FONDATĂ
1976
 
 
THE ANNALS OF “DUNAREA DE JOS” UNIVERSITY OF GALATI. 
FASCICLE IX. METALLURGY AND MATERIALS SCIENCE 
N0. 4 – 2012, ISSN 1453 – 083X 
 
 
 
 
AUGMENTED REALITY–APPLIED INFORMATICS TECHNOLOGY 
TO PRODUCTS MAINTENANCE, REPAIR AND SERVICING  
 
Ana-Maria IONESCU1, Ionel PETREA2, Florin Bogdan MARIN2 
1Student IPMI of Faculty of Metallurgy and Materials Science 
2Dr.eng. of Faculty of Metallurgy and Materials Science 
University of “Dunărea de Jos” of Galati 
email:flmarin@ugal.ro 
 
ABSTRACT 
 
Augmented Reality (AR) enhances a user's perception of the real world by 
enhancing it with specific objects that are over-imposed to the real objects. The 
virtual objects display information that the user may use. The information conveyed 
by the virtual objects helps a user perform real-world tasks. An augmented reality 
application is the assembly, maintenance, and repair of complex machinery or 
common goods. Instructions might be easier to understand if they were available, 
not as manuals with text and pictures, but rather as 3-D drawings superimposed 
upon the actual equipment, showing step-by-step the tasks that need to be done and 
how to do them.  These superimposed 3-D drawings can be animated, making the 
directions even more explicit. This paper describes a proof-of-concept AR 
application to be used for office printer service operation. 
 
KEYWORDS: augumented reality, informatics, maintenance, repair 
 
1. Introduction 
 
Augmented Reality enhances a user's perception 
of the real world by enhancing it with specific objects 
that are over-imposed to the real objects. 
The virtual objects display information that the 
user may use. The information conveyed by the 
virtual objects helps a user perform real-world tasks.  
There are several AR applications. such as 
medical visualization, maintenance and repair, 
annotation, robot path planning, entertainment, and 
military aircraft navigation and targeting.  
The next section describes only little research in 
the AR field. An AR system is defined as a system 
which combines real and virtual objects in a real 
environment and run interactively and in real time. 
In the medical field, doctors could use 
Augmented Reality as a training aid for surgery using 
3-D datasets of a patient in real time, using non-
invasive sensors like Magnetic Resonance Imaging 
(MRI), Computed Tomography scans (CT), or 
ultrasound imaging, the AR might be a solution to 
simulate operation [2] [3] [4].  
At the moment many research groups are 
aiming at developing augmented reality algorithms 
and applications [5] [6] [7] [8], but no commercial 
robust products are available on the market. World 
known company BMW A.G. is developing in their 
research labs an augmented reality application to help 
mechanics as shown in Fig. 1. By adding more 
information to the reality, the mechanic will be 
assisted to service the car. The mechanic is using 
special goggles to access a computer by wireless 
communication and to access the exactly information 
they need in workshop, such as order of assembly, 
torque, etc. 
Another category of Augmented Reality 
applications is the assembly, maintenance, and repair 
of complex machinery. Instructions might be easier to 
understand if they were available, not as manuals 
with text and pictures, but rather as 3-D drawings 
superimposed upon the actual equipment, showing 
step-by-step the tasks that need to be done and how to 
do them. These superimposed 3-D instructions can be 
animated, making the directions even more explicit. 
Current maintenance systems require even the 
most experienced user to spend time to search 
manuals, or search through manuals for specific 
procedures and component data.  
Besides that, the stress induced by this search 
within manuals can affect work for certain 
maintenance and repair operations. 
-  64  -
FAC
UL
TA
TE
A
D
E
M
E
TA
LU
RG
IE, ŞTIINŢA MATER
IA
L
EL
O
R
ŞIM
EDIU
FONDATĂ
1976
 
 
THE ANNALS OF “DUNAREA DE JOS” UNIVERSITY OF GALATI. 
FASCICLE IX. METALLURGY AND MATERIALS SCIENCE 
N0. 4 – 2012, ISSN 1453 – 083X 
 
 
 
 
Fig. 1. AR application developed by BMW for workshop repair 
 
During the maintaining process, the user must 
interpret the information in the manual and apply it 
spatially and conceptually to a particular repair 
procedure. This means filtering relevant information 
to matching components in the manual to real world 
components. 
In the early 1990s, Caudell and Mizell invented 
the term “augmented reality,” to define the idea of 
using an AR system to replace the complex manual 
documentation used for airplanes, such as the Boeing 
747 [11]. 
In servicing such system provides several 
advantages, such as to be able to synthesize additional 
information and complex sequences relatively hard to 
be explained especially to an inexperienced user (in 
case of common goods products such as office 
printers). The information in the manual must 
synthesise information and it is very hard to place and 
describe a specific context of servicing.  
Feiner and colleagues introduced the idea of 
Knowledge-based Augmented Reality for 
Maintenance Assistance (KARMA), demonstrating a 
system to aid a user in servicing an office laser printer 
[15]. This system interactively generated  instructions 
by tracking user position and orientation and printer 
position components in order to inform current 
servicing operation by over-imposing a virtual reality 
to real one (Fig. 2). 
 
 
 
Fig. 2. Augmented reality for servicing an office printer [9] 
 
2. Problem formulation 
 
We propose the following problem statement 
to guide our work: providing maintenance and repair 
assistance to actively assisting in their performance, 
using augmented reality technology and incorporate 
instructions and assistance with specific software. We 
aim to develop an AR application for office printer 
service. 
We found several challenges in order to 
develop robust augmented reality algorithm: 
-real-time data acquisition and processing; in 
order to be effective and usable, the augmented 
reality algorithm needs fast execution and low 
processing resources. 
-consistent data models; in order to identify 
parts we need data models. Some components are 
quite easy to be identified whereas others, with 
-  65  -
FAC
UL
TA
TE
A
D
E
M
E
TA
LU
RG
IE, ŞTIINŢA MATER
IA
L
EL
O
R
ŞIM
EDIU
FONDATĂ
1976
 
 
THE ANNALS OF “DUNAREA DE JOS” UNIVERSITY OF GALATI. 
FASCICLE IX. METALLURGY AND MATERIALS SCIENCE 
N0. 4 – 2012, ISSN 1453 – 083X 
 
 
complex and similar shapes with other are difficult to 
be identified in 3D space. 
 
3. Problem solution 
  
The hardware used is a Genius Hd Camera and 
a PC with Pentium D 3 GHz processor processing. 
The frame’s resolution to be processed is 1200X800 
pixels. The proposed algorithm is shown in Fig. 3. 
The object recognition, which is the most challenging 
part of the algorithm, is represented by SURF 
(Speeded Up Robust Feature). SURF (Speeded Up 
Robust Feature) is a robust local feature detector 
presented by Herbert Bay in 2006, and is based on 
identifying remarkable features in the image. These 
features are compared between the model image to 
the one to be processed and identified.  
Using known distance between identified 
features of the model image, we can approximate 
position in space relative to the camera of the object.  
Once the component is identified the relative 
positions of features are calculated and the actual 
position of the component in space is approximated. 
Using OPenGL library the cartridge model is 
displayed, taking into account necessary 
transformation concerning rotation, scale, and point 
of view. The 3d cartridge model is over-imposed in 
real image at the correct coordinates. 
 
 
 
Fig. 3. Proposed algorithm 
 
In Fig. 4 some of the features detected by SURF 
algorithm are displayed.  
The features are previously selected in the 
model image. In Fig. 5 the real image as taken by 
camera is displayed and in Fig. 6 it is shown 
augmented reality with the 3D model of the cartridge 
in place and instructions to remove it. 
 
 
 
Fig. 4. Several SURF features identified in the image 
 
 
 
  
Fig. 5. Reality as taken by web camera Fig. 6. Augmented reality with instructions for servicing 
 
-  66  -
FAC
UL
TA
TE
A
D
E
M
E
TA
LU
RG
IE, ŞTIINŢA MATER
IA
L
EL
O
R
ŞIM
EDIU
FONDATĂ
1976
 
 
THE ANNALS OF “DUNAREA DE JOS” UNIVERSITY OF GALATI. 
FASCICLE IX. METALLURGY AND MATERIALS SCIENCE 
N0. 4 – 2012, ISSN 1453 – 083X 
 
 
We used Visual C 8 IDE environment along 
with Open CV as computer vision library and 
OpenGL libraries for 3D graphics display. 
The algorithm can be successfully used for 
those components with enough image variations to be 
identified as features for SURF algorithm.  
However, in case of complex surface with no 
image variations such as casing of the printer, the 
algorithm presented fails to identify the surface. 
Further algorithm and techniques need to be done to 
solve advanced identification image issues. 
 
4. Conclusion 
 
Augmented Reality is a beyond maturity at the 
moment.  
Today AR systems are primarily found in 
academic and industrial research laboratories. Several 
manufacturers are researching currently to release 
commercial products.  
The paper describes a proof-of-concept 
algorithm. The algorithm successfully can be used for 
those components with enough image variations to be 
identified as features for SURF algorithm.  
However, in case of complex surface with no 
image variations such as casing of the printer, the 
algorithm presented fails to identify the surface.   
References 
 
[1]. Andreas Dünser, Hannes Kaufmann, Karin Steinbügl, 
Judith Glück - Virtual and Augmented Reality as Spatial Ability 
Training Tools. 
[2]. R. Beichel. -Virtual Liver Surgery Planning: Segmentation of 
CT Data. PhD thesis, Graz University of Technology, (2005). 
[3]. Henry Fuchs, Mark A. Livingston, Ramesh Raskar - 
D‘nardo Colucci, Kurtis Keller, Andrei State, Jessica R. 
Crawford, Paul Rademacher. 
[4]. Samuel H. Drake, and Anthony A. Meyer - Augmented 
reality visualization for laparoscopic surgery. In MICCAI ’98: 
Proceedings of the First International Conference on Medical 
Image Computing and Computer-Assisted. 
[5]. Wacker FK, Vogt S, Khamene A, Jesberger JA, Nour SG, 
Elgort DR, Sauer F, Duerk JL, and Lewin JS. - An augmented 
reality system for mr image-guided needle biopsy: initial results in 
a swine model. Radiology, 2(238):497–504, February (2006). 
[6]. T. P. Caudell and D. W. Mizell - Augmented Reality: An 
Application of Heads-up Display Technology to Manual 
Manufacturing Processes," Proceedings of the Twenty-Fifth 
International Conference on System Sciences, Hawaii, (1992), pp. 
659-669 vol. 2. 
[7]. Dangelmaier, W., Franke, W., Mueck, B.; Mahajan, K. - 
Augmented Reality Applications in Warehouse Logistics. In 
Proceedings of the 7th International Conference on Production 
Engineering and Logistics, Aim-Shams University, Cairo. Dede, C. 
(2006): How mediated immersion shapes learning. Online: 
http://csdl2.computer.org/comp/proceedings/vr/2006/ 
0224/00/022xiii.pdf, IEE.  
[8]. Doil, F., Schreiber, W., Alt, T., Patron, C. - Augmented 
Reality for manufacturing planning. In Proceedings of the 7th 
International Immersive Projection Technologies Workshop, ETH 
Zurich, Zurich (2003).  
[9]. Friedrich, W. ARVIKA – Augmented reality for development, 
production and service. Publicis Corporate Publishing, Erlangen 
(2004). 
[10]. S. Feiner, B. Macintyre, and D. Seligmann - Knowledge-
based Augmented Reality, Communications of the ACM, 36(7), 
(1993), pp. 53-62. 
 
-  67  -


Augmented Reality-Applied Informatics Technology to Products Maintenance, Repair and Servicing

https://www.gup.ugal.ro/ugaljournals/index.php/mms/article/download/2868/2553

Augmented Reality-Applied Informatics Technology to Products Maintenance, Repair and Servicing

Abstract

Similar works

Full text

Available Versions

Directory of Open Access Journals