In this paper, the 3D face hallucination system is proposed on both 2D training face images as well as respective 3D training
face models with grey-level. The proposed method hallucinates the 3D high-resolution model patch using same position of
each image patches of interpolated 2D training image and 3D high-resolution training face model for low-resolution input
image. Firstly, the optimal weights of the 2D input low-resolution image position-patches are estimated with the corresponding
2D low-resolution training image patches. The canonical correlation analysis (CCA) is used to learn the mapping between the
2D interpolated face training image and the 3D face model with respect to their weights. Secondly, the corresponding 3D face
model patch with weight by matching high score among the 2D interpolated training image patches and 3D training face
model is selected. Finally, the 3D high-resolution facial model is formed by integrating the hallucinated 3D patches which are
obtained through mapping patches with respective weights. In order to evaluate the performances of the above approaches, we
used example based learning methods to obtain the high-resolution output for a low-resolution input. In this approach, we used
the available frontal data sets such as FERET, CAS-PERL and CMU to analysis the performance and some parameters are also
considered, which may affect the results from the above proposed method

Naleer, H.M.M

Yao, LU

English

IR South Eastern University of Sri Lanka

Sciences 
Proceedings, 04th International Symposium, SEUSL 
Page 501 
Hallucinated 3D Face Model from A Single 2D Low-Resolution  
Face Using Machine Learning  
H.M.M Naleer1 and  Yao LU2 
1Faculty of Applied Sciences,  
South Eastern University of Sri Lanka, Oluvil, Sri Lanka. 
2Laboratory of Intelligent Information Technology,  
Beijing Institute of Technology, Beijing 100081, China. 
-mail: hmmnaleer@gmail.com,vis_yl@bit.edu.cn 
Abstract 
In this paper, the 3D face hallucination system is proposed on both 2D training face images as well as respective 3D training 
face models with grey-level. The proposed method hallucinates the 3D high-resolution model patch using same position of 
each image patches of interpolated 2D training image and 3D high-resolution training face model for low-resolution input 
image. Firstly, the optimal weights of the 2D input low-resolution image position-patches are estimated with the corresponding 
2D low-resolution training image patches. The canonical correlation analysis (CCA) is used to learn the mapping between the 
2D interpolated face training image and the 3D face model with respect to their weights. Secondly, the corresponding 3D face 
model patch with weight by matching high score among the 2D interpolated training image patches and 3D training face 
model is selected. Finally, the 3D high-resolution facial model is formed by integrating the hallucinated 3D patches which are 
obtained through mapping patches with respective weights. In order to evaluate the performances of the above approaches, we 
used example based learning methods to obtain the high-resolution output for a low-resolution input. In this approach, we used 
the available frontal data sets such as FERET, CAS-PERL and CMU to analysis the performance and some parameters are also 
considered, which may affect the results from the above proposed method. 
Keywords: face hallucination, 3D face model, 2D low-resolution, CCA 
Introduction 
The modeling face is used in several computer vision applications, which are presented in (Blanz, et al., 2003, Edwards et al., 
1998, Blanz & Vetter, 2003, Ramanathan, et. al., 2006, Baker, et al., 2004). The 3D models become useful universally by face 
enhancement of 3D models. In several aspects, the 3D face model super-resolution is employed. The capture of 3D data is 
limited according to the distance of the situation and objects. The availability of 3D scanners and capturing instruments are 
acquired typically low-resolution 3D data. Therefore, the hallucination of 3D face models is very helpful while the 3D data 
models are low-resolution. However, the time-consuming 3D scanner (Eisert & Girod 1998) is not accepted for such a face 
hallucination system. The method of using two orthogonal photos (Goto, et al., 2002) cannot provide real-time processing 
either. The single image is used to obtain the super-resolution of face (Feng & Yuen 2000), where the head rotation parameters 
require measuring via other images. In addition, it gives the inaccurate results for 3D face models. On the other hand, the 
multiple images are used to obtain 3D super-resolution, is presented (Morris, et al., 1999). This process is involved with 2D 
image and 3D model. The shape from stereo and shading are studied in (Morris, et al., 1999, Zhang, et al.,1995, Hom & 
Brooks, 1989) for 3D super-resolution. Though, few methods are presented for 3D super-resolution with an input 2D image. 
Presently, 3D data has become more advance while rapid process in 3D technology. This kind of 3D super-resolution is great 
needs in practice, since occasionally the 3D acquirement system does not provide good enough data due to 3D face recognition 
and lengthy distance (Gang & Zhaohui 2005).  
Sciences
Proceedings, 04th International Symposium, SEUSL 
Page 502 
The 3D position values of head shape are obtained by scanner technology which is the main approach to create 3D face model 
currently. It needs additional cost due to need of specific instrument. Meanwhile, the 3D model creation can be obtained from 
2D images. Conversely, in this paper, a new method based on learning-method is focused to provide the 3D hallucinated face 
model for 2D low-resolution input image. Recently, very few works studied on 3D face hallucination. However, the super-
resolution of 3D face is presented (Pan et al., 2006), which is pursued by the work (Baker & Kanade 2000). Firstly, 3D shape 
is acquired by progressive resolution chain (PRC). Secondly, MAP approach is used to obtain the 3D face super-resolution 
(Yang et al., 2007).   
 
Furthermore, human may be used 2D image and it can be visualize into 3D image for bettervisualization that is bring new 
change for eye with brain role. Thus 3D image is may not be precise.  If the 2D image is low-resolution, the human brain has 
very important role in acquiring the 3D image and it may not be precise.  To overcome this problem, we proposed the 
hallucinated 3D face model from a 2D low-resolution input image. We can divide our whole system into two phases. We 
consider as matching phase between 2D interpolated training image and 3D training face model and hallucination phase. 
Output of proposed method is shown in Fig. 1.1. The rest of work is organized as follows: Section 3 is the fundamental idea of 
CCA. Section 4 describes our proposed hallucinated 3D face model via image patches and section 5 presents experimental 
results on construction of 3D hallucinated face model from a 2D low-resolution image. Finally, the conclusion is given in 
section 6. 
1.  Basic Idea of CCA 
The dominant multivariate analysis is, Canonical Correlation Analysis (CCA) (Hardoon, et al., 2004), which has been used in 
several applications such as post estimation and face matching (Melzer, et al., 2003 & Naleer, et al., 2013). To maximize the 
correlation between two sets, the CCA is constructing the subspace.  
Given N pairs of sample ),( ji yx  of ),( YX , Ni ,......,1 , where 
nRX and nRY . The mean of both X and Y is 
zero. The goal of CCA is to learn a pair of direction xw and yw to maximize the correlation between the two projections Txw
and Tyw , where T denotes the transpose, which is to maximize: 
y
TT
yx
TT
x
y
TT
x
wYYwEwXXwE
wXYwE
,   (3.1) 
where yxfE , - the experimental anticipation of the function yxf , . The covariance matrix of YX , is 
T
YY
YX
XY
XX
T
C
C
C
C
E
Y
X
Y
X
EYXC , ,  
where XXC and YYC are within sets covariance matrices. Hence, can be written as 
 
yyy
T
yxxx
T
x
yxy
T
x
wCwwCw
wCw
      (3.2) 
Sciences 
Proceedings, 04th International Symposium, SEUSL 
Page 503 
Let
0
0
YX
XY
C
C
A ,
YY
XX
C
C
B
0
0
, we can show that the solution 
TT
y
T
x wwW , amounts to the extreme points 
of the Rayleigh quotient: 
     
BWW
AWWr T
T
      (3.3) 
The solution xw and yw can be obtained as solutions of the generalized eigen problem: 
     BWAW       (3.4) 
In case of the sample size is comparatively small; the CCA has a tendency to over-fit to the training image [17]. If 
0x and 0y are regularization parameters, addition of Ix and Iy to xxC and yyC respectively, gives encouraging 
avoided outputs. Let yx , the object function of regularized CCA is to maximize , 
where 
yyy
T
xxxx
T
x
yxy
T
x
wICwwICw
wCw
. 
 
2. Hallucinated 3D Face Model via Image Patches 
2.1 CCA Regression for 2D-3D Matching Phase 
The following section used CCA to find out the results of mapping between 3D and 2D face models. In the learning phase, the 
2D (interpolated 2D training dataset )3224( )-3D (3D training face model )12896( ) mapping is learnt from the training 
set which consists of K pairs of 2D-3D face image face model respectively. In the mapping phase, the most correlative 3D 
face shape to the 2D (2D low-resolution input image )3224( ) is found by 2D-3D mapping. In the learning stage, N pairs 
of 2D-3D face are given as )...........2,1(,),(),( NkyxYX kk  , where ),( kk yx is a corresponding pair of 2D and 3D.  
Furthermore, to reduce the computational complex, PCA is applied firstly to transform kx and ky into the lower dimensional 
spaces (known as dimension reduction). PCA transform matrices xP  and yP are learnt from 2D and 3D training sets 
respectively. The PCA projections are computed as )(' XXPX Tx and )(
' YYPY Ty , where X andY are the mean 
faces of 2D face image and 3D face model respectively. On the other hand, two further projection directions xw and yw are 
learnt for linear CCA by performing CCA on 'X and 'Y respectively (known as CCA regression). Also, let 'XwTx and 
'YwTy
are best correlated. In the mapping phase, also need to project new pair of images tsX and tsY into PCA firstly, that is 
)(' XXPX ts
T
xts and )(
' XYPY ts
T
yts respectively. For linear CCA, 
'
tsX and 
'
tsY are projected into CCA sup-space, 
that is  
 'ts
T
xout XwX , 
'
ts
T
yout YwY       (4.1) 
Finally, the matching score )(So can be calculated as 
  
)(
),(
outout
outout
outout YX
YXYXSo       (4.2) 
2.2 3D Face Model Hallucination via Learning Method 
Sciences 
Proceedings, 04th International Symposium, SEUSL 
Page 504 
In this section, the two phases as mapping phase and hallucination phase are measured. As mentioned in section 4.1, the 3D 
face model patch corresponding to 2D input low-resolution image patch can be found. The description of weights calculation 
on 2D low-resolution training image patches with input 2D low-resolution image is described in this section firstly. Secondly, 
hallucination phase is considered. 
4.2.1Calculation of Mapping Scores 
We adopted the CCA to get the exact model of 2D-3D face mapping by patch based CCA, the following steps are considered 
in mapping stage.    
 A large number of patches are created from 2D face image and 3D face models.  
 CCA is used to learn the performance between 2D image and 3D face model for every patch. 
 In the mapping phase, the testing images are also departed into several parts in the same way. Then we can get a score 
for each patch. 
 The final score is obtained by combining the individual scores of all participating patches. 
 Finally, selected the 3D patches with weights on 3D training face model set using mapping score to the corresponding 
2D patches on interpolated 2D training set.  
The weights are calculated as follows and the style of the patch is shown in Fig. 4.1In the patch based weights in the particular 
location ),( ji is calculated by the following Eq. (4.3), which is described in (Naleer, et al., 2013) Algorithm-I, where S - 2D 
column vector of low-resolution training images
n
q
q
LI 1 .If ji, is indicated as coordinate of the image patch in the 
particular location, the weight is calculated by Eq. (4.3) 
 
 
 
 
 
 
.  
 
 
 
 
 
   
 
 
 
 
 
 
 
),( jiwi RjiIRjiLjiIRjiLR
jiIRjiLjiIRjiL
P
i
i
L
TP
iI
TP
i
i
L
TP
iI
T
P
i
i
L
TP
iI
P
i
i
L
TP
iI
1
1111
11
1
11
]})),(()),([(])),(()),({[(
]),(()),([(])),(()),([(
(4.3) 
The selection of patch size is an important issue. It is too large or it is too small, in both cases the local mapping of whole face 
information will be difficult. Hence the scores of each patch are combined to such a decisive level so that it is simplified. 
Various combining schemes include Median Rule and, Sum Rule, Majority Voting, Max Rule, Product Rule and Min Rule [6]. 
Eq. (4.4) gives the results of final matching score where, iSo and iw are the output score and corresponding weight for 
thi
patch respectively. 
P
i
ii SowF
1
     (4.4) 
n
q
q
LI 1
Sciences 
Proceedings, 04th International Symposium, SEUSL 
Page 505 
4.2.2. 3D Hallucination Phase 
Once we found the 2D face training image patch position with respective weights to corresponding 2D low-resolution image 
patch in the same position, according to the final matching score using CCA regression mapping between 3D training face 
model and 2D interpolated training images using weights, the following process are used to construct the final output as 
hallucinated 3D face model.According to the original position, the final 3D hallucinated face model is obtained by combining 
the 3D face model patches. Average of the pixels values in the overlapping areas among two adjacent hallucinated patches is 
used to acquire the final result for pixel of the overlapping areas. The proposed 3D face hallucination model has been 
summarized below and the entire framework is given in Fig. 4.2. 
Step1:Find the weights between 2D low-resolution input and 2D low-resolution training images in the same positions using 
Eq. (4.3). 
Step2:Find the fusion score between 2D training image and 3D face image model using Eq. (4.4). 
Step3:Once selected the 3D face mode patch with respective weights according to the fusion score, the hallucinated 3D face 
model PHI can be obtained by following Eq. (4.5).      
     
n
q
i
q
H
P
H jiwjiIjiI
1
),()),((),(      (4.5) 
Sciences 
Proceedings, 04th International Symposium, SEUSL 
Page 506 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
F u s io n  S co re  o f  E ach  P a tch  w ith  W e ig h t  
F in a l D ec is io n  w ith  3 D  P atch  an d  
W eig h t
 
3 D  S u b sp aces
 
 
 
C o n s tru c ted  
3 D F ace M o d e l 
 
3 D  T rain in g  F ace  
M o d e l D a ta se t 
 
 
3224
12896
3224
12896
12896
C C A  
R egression
C C A  
R egression
C C A  
R egression
C C A  
R egression
C C A  
R egression
C C A  
R egression
n
q
q
LI 1
n
q
q
HI 1
P
HI
IL
Sciences 
Proceedings, 04th International Symposium, SEUSL 
Page 507 
 
Experiments and Results 
Our experiments are conducted with the CAS-PEAL and FERET face datasets consisting of 200 individuals. The training face 
data sets are formed as 2D face images and 3D face models pairs for respective each individual.  The laser scanner is used to 
obtain 3D training face model, which is offer range images and the faces are essentially 2.7D data. In the 3D face model 
training set, have a little facial expression and head pose varying for corresponding 2D training data set of each individual. All 
3D training face models ( 12896  ) and 2D low-resolution face images ( 3224 ), and all training images are manually 
aligned based on eyeballs, mouths positions with centers of left and right eyeballs and center of the mouth, and the input low-
resolution image is also  manually aligned. Our proposed approach is evaluated on 80 test low-resolution inputs. Some output 
of the test images on CASE-PERL are presented in Fig. 5.1.  
 
 
The comparative results for input low-resolution face image for each patch )33( are given in Fig. 5.2 in terms of cumulative 
match curves. Meanwhile the performance with training samples is also considered the results by cubic interpolation method 
and it is shown in Fig 5.3. To exhibit the performance of our proposed method, the amount of training images with RMS 
values has been given in Fig. 5.3. In fact, the cubic interpolation is not depending on the number of training images. 
Meanwhile, the performance of the proposed method is very poor while less than about 73 training images. It means, our 
methods is enclosed most of the variance of the 3D face model.  
 
3224
)12896(
Sciences 
Proceedings, 04th International Symposium, SEUSL 
Page 508 
 
 
 
 
Sciences
Proceedings, 04th International Symposium, SEUSL 
Page 509 
Conclusion 
This work has presented a learning-based 3D face hallucination method for single 2D low-resolution face image with 2D low-
resolution face image and 3D face model training image pairs based on matching scores. The experiments show that, it could 
be applicable to generate a 3D hallucinated face image version of the 2D low-resolution input face image in absence of 3D 
high-resolution face model in the training sets.  
 
References 
In Computer Graphics Forum, 
vol. 22(3), pp. 641-650. 
European Conference on 
Computer Vision, vol. II (ECCV) [C], pp. 581-595. 
IEEE Trans. on Pattern Analysis 
and Machine Intelligence, vol. 25(9), pp. 1063-1074. 
Ramanathan, S, Kassim, A, Venkatesh 
Inter. Conf. on Image Processing (ICIP) [C], pp. 661-664. 
 
IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 26 (10), pp. 1380-1384. 
IEEE Computer Graphics and 
Application, vol. 18(5), pp. 70 78.  
Goto, T, Le Signal Process and Image 
Communication, vol. 17(3), pp. 243 259. 
-&-Shoulder Face Image Using Virtual Frontal- IEEE Trans. 
Systems Man and Cybernetics Part A, vol. 30(6), pp. 871 883. 
IEEE Signal Processing Workshop [C], pp. 140-143. 
Artificial Intelligent, vol. 78(1), pp. 87-119. 
0 20 40 60 80 100 120
4
6
8
10
12
14
Number of Training Image Samples Pairs
 
 
Cubic Interpolation
Proposed Method
Sciences 
Proceedings, 04th International Symposium, SEUSL 
Page 510 
MIT Press, pp. 123-171. 
Inter. Journal of Image and Graphics, vol. 5(3) pp. 1-
21. 
- In Proc. European Conf. on Computer Vision 
(ECCV) [C], vol. 3952, pp. 389-401. 
In Proc. Inter. Conf. on Automatic Face and Gesture Recognition. (FG) 
[C], pp. 83-88.  
-Depth Super In Proc. IEEE Conf. 
Computer Vision Pattern Recognition (CVPR) [C], pp. 1-8. 
Hardoon, DR, Szedmak SR, & Shawe-
Neural Comput, vol. 16(12), pp. 2639  2664. 
Pattern 
Recognition, vol. 36(9), pp. 1961 1971. 
-Step Method for Face Hallucination Based on Sparse Compensation 
IET image processing, vol. 7(2), pp. 624-632, 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 


Hallucinated 3d face model from a single 2d low-resolution face using machine learning

http://ir.lib.seu.ac.lk/bitstream/123456789/1447/1/4%20th%20Int%20Symp_2014_Article_65_Pages%20from%20501-510.pdf

Hallucinated 3d face model from a single 2d low-resolution face using machine learning

Abstract

Similar works

Full text

Available Versions

IR South Eastern University of Sri Lanka