Much of the progress made in image processing in the past decades can be attributed to better modeling of image content, and a wise deployment of these models in relevant applications. In this paper, we review the role of this recent model in image processing, its rationale, and models related to it. As it turns out, the field of image processing is one of the main beneficiaries from the recent progress made in the theory and practice of sparse and redundant representations. Sparse coding is a key principle that underlies wavelet representation of images. Sparse representation based classification has led to interesting image recognition results, while the dictionary used for sparse coding plays a key role in it. In general, the choice of a proper dictionary can be done using one of two ways: i) building asparsifying  dictionary based on a mathematical model of the data, or ii) learning a dictionary to perform best on a training set

Correia, Rachel

Dmonti, Primeya

D’mello, Shonal

Figer, Jerina

English

Gyandhara International Academic Publication (GIAP): Journals

International Journal of Students Research in Technology & Management 
Vol 3 (05), May 2015,  ISSN 2321-2543, pg 373-376  
 
www.giapjournals.org/ijsrtm.html  Page 373 
 
MULTIPLE DICTIONARY FOR SPARSE 
MODELING 
 Rachel Correia1 , Shonal D’mello2, Primeya Dmonti3, Jerina Figer4
 
Department of Electronics and Telecommunication Engineering, Xavier Institute of Engineering 
Mahim, Mumbai, India 
1
rechu.correia@gmail.com  
                                     
2
shona190194@gmail.com  
                                     
3
dmontiprimeya@gmail.com   
                                     
4
jerryfig25@gmail.com                                                                                           
Abstract— Much of the progress made in image processing in 
the past decades can be attributed to better modeling of image 
content, and a wise deployment of these models in relevant 
applications. In this paper, we review the role of this recent 
model in image processing, its rationale, and models related to it. 
As it turns out, the field of image processing is one of the main 
beneficiaries from the recent progress made in the theory and 
practice of sparse and redundant representations. Sparse coding 
is a key principle that underlies wavelet representation of images. 
Sparse representation based classification has led to interesting 
image recognition results, while the dictionary used for sparse 
coding plays a key role in it. In general, the choice of a proper 
dictionary can be done using one of two ways: i) building as 
parsifying  dictionary based on a mathematical model of the data, 
or ii) learning a dictionary to perform best on a training set.  
 
I. INTRODUCTION 
The process of digitally sampling a  natural  signal  leads to its 
representation as the sum of Delta functions in space or  time. 
This representation, while convenient for the purposes of 
display or playback, is mostly inefficient for analysis tasks. 
Signal processing techniques commonly require more 
meaningful representations which capture the useful 
characteristics of the signal for recognition, the representation 
should highlight salient features; for denoising, the 
representation should efficiently separate signal and noise; and 
for compression, the representation should capture a large part 
of the signal with only a few coefficients. Interestingly, in 
many cases these seemingly different goals align, sharing a 
core desire for simplification. Representing a signal involves 
the choice of a dictionary, which is the set of elementary 
signals or atoms used to decompose the signal. When the 
dictionary forms a basis, every signal is uniquely represented 
as the linear combination of the dictionary atoms. In the 
simplest case the dictionary is orthogonal, and the 
representation coefficients can be computed as inner products 
of the signal and the atoms; in the non-orthogonal case, the 
coefficients are the inner products of the signal and the 
dictionary inverse, also referred to as the bi-orthogonal 
dictionary. 
In the last decade, sparsity has emerged as one of the leading 
concepts in a wide range of signal-processing applications 
(restoration, feature extraction, compression, to name only a 
few applications). Sparsity has long been an attractive 
theoretical and practical signal property in many areas of 
applied mathematics (such as computational harmonic 
analysis, statistical estimation, and theoretical signal 
processing).The sparse representation theory has shown that 
sparse signals can be exactly reconstructed from a small 
number of elementary signals (or atoms).The sparse 
representation of natural signals can be achieved by exploiting 
its sparsity or compressibility. A natural signal is said to be 
sparse signal if that can be compactly expressed as a linear 
combination of a few small number of basis vectors .Sparse 
representation has become an invaluable tool as compared to 
direct time-domain and transform-domain signal processing 
methods.Sparse and redundant representation modeling of 
data assumes an ability to describe signals as linear 
combinations of a few atoms from a pre-specified dictionary. 
As such, the choice of the dictionary that sparsifies the signals 
is crucial for the success of this model. Unlike to DCT, DWT 
and PCA analysis and their variations sparse models do not 
impose any condition of orthogonallity on the basis vectors, 
hence allowing more flexibility to adapt the representation.  
 
II.  PROPOSED SOLUTION 
A.  Dictionary Learning: 
In its simplest form a linear generative model can be 
written algebraically as: 
  

i
n
i
isdDSX
0
              (1) 
The input vector 
1 mRX is a vector of data 
derived from the image block under study, which is to be 
encoded. The observation is modeled as a linear 
combination of atoms id , which are the elementary 
structural representations establishing the 
dictionary
kmD  . The vector S defines the multiplicative 
weights of these features. The vector  represents the 
error due to the inability of the model to represent the 
input exactly.  If the requirement is to have zero 
observation error, i.e. 0 and a square known 
orthogonal matrix D, the problem of finding S for any 
International Journal of Students Research in Technology & Management 
Vol 3 (05), May 2015,  ISSN 2321-2543, pg 373-376  
 
www.giapjournals.org/ijsrtm.html  Page 374 
 
input X becomes a standard orthogonal transform. For 
cases where both D and S are unknown for km   , 
PCA, ICA and similar variations are the established tools, 
which guarantee to have the elements of S uncorrelated 
and independent. The central problem in applying PCA 
and ICA is in finding the matrix D while minimizing  . 
Unlike to this sparse modeling deals with km  .Apart 
from estimating the dictionary D the need is also to find a 
method to compute S for a given estimate.  When a 
sufficient amount of training data is available, the 
dictionary can be adapted to the data itself. The joint 
optimization problem of dictionary learning and sparse 
coding is then expressed as: 
1,
,min
2
2
,


k
piSD
dktosubject
sDSX 
……………(2) 
where
pi
s is the pl norm and 10  p measures the 
sparsity of the vector is . Sparse approximation refers to 
an representation of X with fewer non-zero coefficients 
than the dimension m of X. A measure of the sparsity of a 
representation is counting the zeros or L0 norm. This 
norm is easy to compute, however, the problem of finding 
the minimum of equation (2) with this norm is NP-hard . 
Different approximations to this norm have therefore been 
discussed. 
A wide range of dictionary learning algorithms have 
been proposed in the literature . Standard unsupervised 
methods like K-means, K-SVD [1], and LLC [25] train 
visual dictionaries based on only information of images. 
A widely acceptable dictionary learning method for image 
restoration is the KSVD algorithm, which learns an over-
complete dictionary from a training dataset of natural 
image patches. Statistical learning theory has proved that 
augmenting the amount of training samples dramatically 
enhances the generalization of dictionary. However, 
addition of training samples may also lead to enlarge the 
variation in pattern with appearance of many types of 
textures and edges. Apparently natural images being 
greatly flushed with smooth blocks, use of single 
dictionary don’t efficiently preserve the high order 
statistical manifolds.   Based on KSVD, Mairal added a 
discriminative reconstruction constraint in the DL model 
to gain the discriminative ability. Algorithms that 
incorporate class-specific information when learning the 
dictionary have also been developed, and successfully 
applied for digit recognition and image classification. 
 
 K-SVD Algorithm 
1. Initialize Dictionary 
 Select atoms from input 
 Atoms can be patches from the image 
 Patches are overlapping  
2. Sparse Coding(OMP) 
 Use OMP or any other fast method 
 Output gives sparse code for all signals 
 Minimize error in representation  
3. Update Dictionary-One atom at a time  
 Replace unused atom with minimally represented  
signal  
 Identify signals that use k-th atom (non zero entries in 
rows of X)  
 Minimize this error matrix with rank-1 approx from 
SVD 
 [U,S,V] = svd(Ek) 
 Replace coeffiecient of atom dk in X with entries of 
s1v1 
 dk = u1/||u1||2 
This paper addresses about learning of multiple decoupled 
dictionaries in a framework to use Discrete Wavelet 
Transform for pattern classification. 
Sparse coding is the process of computing the representation 
coefficients X based on the given signal Y and the dictionary 
D. This process, commonly referred to as ―atom 
decomposition,‖ requires solving above equations , and this is 
typically done by a ―pursuit algorithm‖ that finds an 
approximate solution. Exact determination of sparsest 
representations proves to be an NP-hard problem. Thus, 
approximate solutions are considered instead, and in the past 
decade or so several efficient pursuit algorithms have been 
proposed. The simplest ones are the matching pursuit (MP) 
and the orthogonal matching pursuit (OMP) algorithms. These 
are greedy algorithms that select the dictionary atoms 
sequentially. These methods are very simple, involving the 
computation of inner products between the signal and 
dictionary columns, and possibly deploying some least squares 
solvers. Both above equations are easily addressed by 
changing the stopping rule of the algorithm. 
A second well-known pursuit approach is the basis pursuit 
(BP). It suggests a convexification of the problem sposed in 
the above equations by replacing the lo-norm with and  l1-
norm. The focal underdetermined system solver (FOCUSS) is 
very similar, using the l p-norm with  p≤ 1 as a replacement 
for the lo –norm in the above equations. Sparse representation 
of signals have drawn considerable interest in recent years. 
Sparse coding is a key principle that underlies wavelet 
representation of images. In this paper, we explain the effort 
of seeking a common wavelet sparse coding of images from 
same object category leads to an active basis model called 
Sparse-land model, where the images share the same set of 
selected wavelet elements, which forms a linea rbasis for 
restoring the blurred image. The aim of image restoration is 
the removal of noise (sensor noise, blur etc) from images. The 
simplest approach for noise removal is based on various types 
of filters such as low-pass filters or median filters. In this 
International Journal of Students Research in Technology & Management 
Vol 3 (05), May 2015,  ISSN 2321-2543, pg 373-376  
 
www.giapjournals.org/ijsrtm.html  Page 375 
 
paper, we used Haar wavelet transform based Weiner filter 
followed by PCD algorithm. We observed that the outcomes 
does not show any tendency to be sparse. In this case, we start 
by defining a random generator of sparse vector and it can be 
modified in various ways. One among them is 
MMSE(Minimum mean square error) estimator and it is used 
in this paper. This analysis gives a more solid foundation for 
the sparsest representation of deblurred image, which is said to 
be Sparse-land model. 
 
B.   Proposed Sparse Modeling Using DWT Classification And 
Decoupled Dictionary 
 
I) Dictionary Training 
 
 
 
 
II) Sparse Coding  
 
 
 
Fig:Sparse Coding 
 
     C.   Block Diagram 
 
FIG1 
 
 
 
 
FIG2 
 
III. DESCRIPTION  
 
―Discrete Wavelet Transform‖, transforms discrete signal 
from time domain into time-frequency domain. The 
transformation product is set of coefficients organized in the 
way that enables not only spectrum analyses of the signal, but 
also spectral behavior of the signal in time .This is achieved 
by decomposing signal, breaking it into two components, each 
caring information about source signal. Filters from the filter 
bank used for decomposition come in pairs: low pass and high 
pass. The filtering is succeeded by down sampling (obtained 
filtering result Is "re-sampled" so that every second coefficient 
is kept).Low pass filtered signal contains information about 
slow changing component of the signal, looking very similar 
to the original signal, only two times shorter in term of 
number of samples. High pass filtered signal contains 
information about fast changing component of the signal. In 
International Journal of Students Research in Technology & Management 
Vol 3 (05), May 2015,  ISSN 2321-2543, pg 373-376  
 
www.giapjournals.org/ijsrtm.html  Page 376 
 
most cases high pass component is not so rich with data 
offering good property for compression. In some cases ,such 
as audio or video signal, it is possible to discard some of the 
samples of the high pass component without noticing any 
significant changes in signal. Filters from the filter bank are 
called "wavelets". 
IV.IMPLEMENTATION 
1-level decomposition  of original image 
 
3-level decomposition of original image 
 
 
A.  Dictionary Training 
 
Dictionary training is a much more recent approach to 
dictionary design, and as such, has been strongly 
influenced by the latest advances in sparse representation 
theory and algorithms. The most recent training methods 
focus on and sparsity measures, which lead to simple 
formulations and enable the use of recently developed 
efficient sparse coding techniques The main advantage of 
trained dictionaries is that they lead to state-of-the-art 
results in many practical signal processing applications. 
The cost Vas in the case of the KLT is a dictionary with 
no known inner structure or fast implementation. Thus, 
the most recent contributions to the field employ 
parametric models in the training process, which produce 
structured dictionaries, and offer several advantages. A 
different development, which we do not discuss here, is 
the recent advancement in online dictionary learning 
which allows training dictionaries from very large sets of 
examples, and is found to accelerate convergence and 
improve the trained result. 
 
4.1 Results So Far For Dictionary Training: 
 
 
 
IV. CONCLUSION 
In this paper, we have briefly reviewed sparse and redundant 
representations as a new model that harnesses the local low-
dimensional structure of natural images. Dictionary learning is 
a central step in employing a sparsity based model for various 
data processing tasks. Therefore, the speed of such learning 
algorithms is key in making many algorithms more efficient 
and thus more practical. In this paper we have also propose 
two simple yet effective modifications for the KSVD learning 
algorithm and an image watermarking technique based on a 3-
level discrete wavelet transform has been implemented. 
 
REFERENCES 
[1] M. Aharon, M. Elad, and A.M. Bruckstein, ―The K-
SVD: An algorithm for designing of 
overcomplete dictionaries for sparse 
representation‖,IEEE Transactions On Signal 
Processing, vol. 54, pp. 4311–4322,November 
2006.http://en.wikipedia.org/wiki/Electrocardio
graphy 
[2] Gonzalez, Woods, ―Digital Image Processing‖ 
Prentice Hall of India, second edition, 813 pages. 
[3] F. Rodriguez and G. Sapiro. Sparse representations 
for image classification: Learning discriminative 
andreconstructive non-parametric dictionaries. 
IMA Preprint 2213, 2007.  
 


MULTIPLE DICTIONARY FOR SPARSE MODELING

https://giapjournals.com/index.php/ijsrtm/article/download/180/174

MULTIPLE DICTIONARY FOR SPARSE MODELING

Abstract

Similar works

Full text

Available Versions

Gyandhara International Academic Publication (GIAP): Journals