SR from digital video is a relatively new field, in only its third decade of existence. There is no doubt that as imaging sensor technologies, optical fabrication techniques, and computational algorithms mature, SR will find its way into digital video products such as cameras and digital cable set-top boxes. These papers on the fundamental SR topics of image registration, regularization, photometric diversity, detector nonuniformity, compression, optical design, and performance metrics serve as pioneers in the dynamic and evolving field of SR image reconstruction research and development. We are proud to present them to the image and video processing research community. (Refers to papers appearing in the same issue of the EURASIP Journal on Advances in Signal Processing in which this editorial appeared.

Barner, Kenneth E.

Hardie, Russell C

Schultz, Richard R.

English

University of Dayton

University of DaytoneCommonsElectrical and Computer Engineering FacultyPublicationsDepartment of Electrical and ComputerEngineering2007Super-Resolution Enhancement of Digital VideoRussell C. HardieUniversity of Dayton, rhardie1@udayton.eduRichard R. SchultzUniversity of North DakotaKenneth E. BarnerUniversity of DelawareFollow this and additional works at: https://ecommons.udayton.edu/ece_fac_pubPart of the Optics Commons, and the Signal Processing CommonsThis Editorial is brought to you for free and open access by the Department of Electrical and Computer Engineering at eCommons. It has beenaccepted for inclusion in Electrical and Computer Engineering Faculty Publications by an authorized administrator of eCommons. For moreinformation, please contact frice1@udayton.edu, mschlangen1@udayton.edu.eCommons CitationHardie, Russell C.; Schultz, Richard R.; and Barner, Kenneth E., "Super-Resolution Enhancement of Digital Video" (2007). Electricaland Computer Engineering Faculty Publications. 9.https://ecommons.udayton.edu/ece_fac_pub/9Hindawi Publishing CorporationEURASIP Journal on Advances in Signal ProcessingVolume 2007, Article ID 20984, 3 pagesdoi:10.1155/2007/20984EditorialSuper-Resolution Enhancement of Digital VideoRussell C. Hardie,1 Richard R. Schultz,2 and Kenneth E. Barner31 Department of Electrical and Computer Engineering, University of Dayton, 300 College Park, Dayton,OH 45469-0026, USA2 Department of Electrical Engineering, University of North Dakota, Upson II Room 160, P.O. Box 7165,Grand Forks, ND 58202-7165, USA3 Department of Electrical and Computer Engineering, University of Delaware, 140 Evans Hall, Newark,DE 19716-3130, USAReceived 10 June 2007; Accepted 10 June 2007Copyright © 2007 Russell C. Hardie et al. This is an open access article distributed under the Creative Commons AttributionLicense, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properlycited.When designing a system for image acquisition, there is gen-erally a desire for high spatial resolution and a wide fieldof view. To satisfy these competing requirements, a camerasystem typically must employ small f -number optics, whichproduces an image with very high spatial-frequency band-width at the focal plane. To avoid aliasing caused by under-sampling, the corresponding focal plane array (FPA) must besufficiently dense. However, cost and fabrication complexi-ties may make this impractical. More fundamentally, smallerdetectors capture fewer photons, which can lead to poten-tially severe noise levels in the acquired imagery. Consider-ing these factors, one may choose to accept a certain level ofundersampling or to sacrifice some optical resolution and/orfield of view.In super-resolution (SR) image reconstruction, postpro-cessing is used to obtain images with resolutions that go be-yond the conventional limits of the uncompensated imag-ing system. In some systems, the primary limiting factoris the optical resolution of the image in the focal planeas defined by the cut-off frequency of the optics. We usethe term “optical SR” to refer to SR methods that aim tocreate an image with valid spatial-frequency content thatgoes beyond this cut-off frequency. Such techniques typi-cally must rely on extensive a priori information. In otherimage acquisition systems, the limiting factor may be thedensity of the FPA, subsequent postprocessing requirements,or transmission bit rate constraints that require data com-pression. We refer to the process of overcoming the lim-itations of the FPA in order to obtain the full resolu-tion afforded by the selected optics as “detector SR.” Somemethods may seek to perform both optical and detectorSR.Detector SR algorithms generally process a set of low-resolution aliased frames from a video sequence to producea high-resolution frame. When subpixel relative motion ispresent between the objects in the scene and the detector ar-ray, a unique set of scene samples is acquired for each frame.This provides the mechanism for effectively increasing thespatial sampling rate of the imaging system without reduc-ing the physical size of the detectors.With the proliferation of digital imaging and video, SRhas become a rapidly growing field. Recent advances in SR in-clude innovative algorithms, generalized methods, real-timeimplementations, and novel applications. The purpose ofthis special issue is to present leading research and develop-ment in the area of super-resolution image reconstruction asapplied to digital video. Specific topics addressed in this spe-cial issue include image registration, regularization, photo-metric diversity, detector nonuniformity, compression, opti-cal design, and performance metrics.Two of the papers in this special issue address topicsrelated to subpixel image registration for SR. In particu-lar, the paper by L. C. Pickup et al. presents a joint regis-tration and SR approach in a Bayesian framework, as wellas an alternative approach using Bayesian marginalizationover unknown registration parameters. The paper by M. DasGupta et al. presents a supervised learning based approachto subpixel motion estimation for video super-resolution.Two other papers in this issue explore various cost functionsand regularization methods for SR. V. Patanavijit and S. Ji-tapunkul use a Lorentzian error norm for measuring the dif-ference between the projected estimate of the high-resolutionimage and each low-resolution image. They also employboth Tikhonov and Lorentzian-Tikhonov regularization to2 EURASIP Journal on Advances in Signal Processingremove artifacts from the SR estimate and improve the con-vergence rate of the corresponding iterative algorithms. Thepaper by M. Ng et al. employs total variation regularizationin the reconstruction model, as a variant on the more tradi-tional regularized least-squares-based techniques.Images used for SR may be affected by factors otherthan the traditional aliasing, blur, and noise that mustbe addressed. For example, multiple images may be cap-tured under photometric variation. This may include chang-ing external illumination conditions and/or camera cali-bration differences. M. Gevrekci and B. K. Gunturk pro-pose a novel nonlinear technique for addressing these vari-ations. Another problem often encountered, particularly forinfrared imagers, is detector nonuniformity. R. C. Hardieand D. R. Droege present a Bayesian approach for simultane-ously estimating detector nonuniformity parameters and SRimagery.SR with image compression is another important areaaddressed in this issue. L. Shao presents an adaptive reso-lution up-conversion algorithm that is robust to compres-sion artifacts. The method is based on classification of localimage patterns and uses a least-mean-square training proce-dure. The paper by N. Maor et al. considers compression atthe source for digital camcorders. Their system is based on avariable spatiotemporal sampling strategy, which combinesinfrequent high-resolution frames with more frequent low-resolution frames.As SR techniques becomemore prevalent, optical designsare starting to be influenced by the potential for SR postpro-cessing. The paper by N. A. Ahuja and N. K. Bose exploresthe design of an optical system with a large field of view us-ing a lenslet array, applying multiframe SR techniques to theresulting images. The proposed system is capable of realizingboth a specified resolution and specified field of view. Finally,the paper by A. van Eekeren et al. explores performance met-rics for SR on real-world data. Their proposedmetric is basedon an observer task referred to as triangle orientation dis-crimination. This provides a useful means for quantitativelycomparing the performance of SR reconstruction methodsunder varying conditions.We sincerely hope that you enjoy and learn from thiscollection of SR papers. SR from digital video is a relativelynew field, in only its third decade of existence. There is nodoubt that as imaging sensor technologies, optical fabri-cation techniques, and computational algorithms mature,SR will find its way into digital video products such ascameras and digital cable set-top boxes. These papers onthe fundamental SR topics of image registration, regu-larization, photometric diversity, detector nonuniformity,compression, optical design, and performance metrics serveas pioneers in the dynamic and evolving field of SR imagereconstruction research and development. We are proud topresent them to the image and video processing researchcommunity.Russell C. HardieRichard R. SchultzKenneth E. BarnerRussell C. Hardie graduated magna cumlaude from Loyola College in BaltimoreMaryland in 1988 with a B.S. degree inengineering science. He obtained his M.S.and Ph.D. degrees in electrical engineeringfrom the University of Delaware in 1990and 1992, respectively. Dr. Hardie served asa Senior Scientist at Earth Satellite Corpo-ration in Maryland prior to his appoint-ment at the University of Dayton in 1993.He is currently a Full Professor in the Department of Electricaland Computer Engineering and holds a joint appointment withthe Electro-Optics Program. Along with several collaborators, Dr.Hardie received the Rudolf Kingslake Medal and Prize from SPIEin 1998 for work on multiframe image resolution enhancement al-gorithms. Dr. Hardie recently received the University of Dayton’sTop University-Wide Teaching Award, the 2006 Alumni Award inTeaching. In 1999, he received the School of Engineering Award ofExcellence in Teaching at the University of Dayton and was the re-cipient of the first Professor of the Year Award in 2002 from thestudent chapter of the IEEE at the University of Dayton. His re-search interests include a wide variety of topics in the area of digitalsignal and image processing. His research work has focused on im-age enhancement and restoration, pattern recognition, andmedicalimage processing.Richard R. Schultz serves as an AssociateProfessor and Chair of electrical engineer-ing at the University of North Dakota(UND) in Grand Forks. Dr. Schultz receivedthe B.S.E.E. degree from UND in 1990 andthe M.S.E.E. and Ph.D. degrees from theUniversity of Notre Dame in 1992 and 1995,respectively. He joined the UND faculty in1995, and his teaching and research inter-ests are in aerospace payload and sensor de-velopment, control systems, digital image and video processing,embedded systems design, student-generated intellectual property,systems engineering, and technology entrepreneurship. Prior to hisappointment as Department Chair, Dr. Schultz served as a Copro-ject Director of the North Dakota Experimental Program to Stim-ulate Competitive Research (EPSCoR), from January 2004 throughAugust 2005. In this role, he had the unique opportunity of foster-ing interdisciplinary campus and statewide research, development,and commercialization partnerships. Dr. Schultz is currently in-volved in the Unmanned Aircraft Systems Center of Excellence atthe University of North Dakota, an interdisciplinary research cen-ter that includes the John D. Odegard School of Aerospace Sciences,the School of Engineering and Mines, the College of Nursing, andthe Department of Psychology.Within this center, Dr. Schultz helpssupervise the design of unmanned aerial vehicle payloads and sen-sors, conducts research on the enhancement of surveillance digitalvideo for automatic target recognition and tracking, and guides thedevelopment of sense and avoid technologies for unmanned air-craft.Kenneth E. Barner received his B.S.E.E.degree from Lehigh University, Bethle-hem, Pa, USA in 1987. He received theM.S.E.E. and Ph.D. degrees from the Uni-versity of Delaware, Newark, Delaware, in1989 and 1992, respectively. He is cur-rently a Professor in the Department ofElectrical and Computer Engineering atthe University of Delaware. He was theRussell C. Hardie et al. 3cochair of the 2001 IEEE-EURASIP Nonlinear Signal and ImageProcessing (NSIP) Workshop and a Guest Editor for a special is-sue of the EURASIP Journal of Applied Signal Processing on Non-linear Signal and Image Processing. He is a Member of the Non-linear Signal and Image Processing Board and is coeditor of thebook Nonlinear Signal and Image Processing: Theory, Methods,and Applications, CRC Press, 2004. He was the Technical Programcochair for ICASSP 2005 and is currently serving on the IEEE SignalProcessing Theory and Methods (SPTM) and IEEE Bio-Imagingand Signal Processing (BISP) technical committees. Dr. Barner hasserved as an Associate Editor of the IEEE Transactions on SignalProcessing, the IEEE Transaction on Neural Systems and Rehabil-itation Engineering, and the IEEE Signal Processing Magazine. Heis currently the Editor-in-Chief of the journal Advances in Human-Computer Interaction, a Member of the Editorial Board of theEURASIP Journal of Applied Signal Processing, and is serving asa Guest Editor for that journal on the Empirical Mode Decompo-sition and the Hilbert-Huang Transform special issue. His researchinterests include signal and image processing, robust signal pro-cessing, nonlinear systems, communications, human-computer in-teraction, haptic and tactile methods, and universal access.

Super-Resolution Enhancement of Digital Video

https://ecommons.udayton.edu/cgi/viewcontent.cgi?article=1007&amp;context=ece_fac_pub

Super-Resolution Enhancement of Digital Video

Abstract

Similar works

Full text

Available Versions

University of Dayton