Haneishi, Hideaki

Miyake, Yoichi

Kyoto University Research Information Repository

TitleComputer Simulation for Improvement of Color Reproductionin Electronic Endoscopes(Domain Decomposition Methods andRelated Topics)Author(s)Miyake, Yoichi; Haneishi, HideakiCitation数理解析研究所講究録 (1997), 989: 122-131Issue Date1997-04URL http://hdl.handle.net/2433/61067RightType Departmental Bulletin PaperTextversionpublisherKyoto UniversityComputer Simulation for Improvement of Color Reproductionin Electronic EndoscopesYoichi Miyake and Hideaki HaneishiAbstractImage quality of the electronic endoscopes is poor compared with that from conventionaloptical endoscopes that use color film. Our goal is to improve the image quality, particularlycolor reproduction of electronic endoscopes. To achieve our goal, we have developed theendoscopic spectrophotometer to measure the spectral reflectance of gastric mucous membrane.Three hundred and ten spectra have been analyzed by principal component analysis. The resultsindicate that the reflectance $\mathrm{s}\mathrm{p}\overline{\mathrm{e}}\mathrm{C}\mathrm{t}\mathrm{r}\mathrm{a}$ can be adequately described using only three principalcomponents. Based on the above experimental result, it is shown that the spectral reflectance ofgastric mucous membranes can be calculated from the $\mathrm{R},$ $\mathrm{G},$ $\mathrm{B}$ signals of the conventionalelectronic endoscopes. Therefore, spectral reflectance of all pixels in gastric mucous membranetaken by electronic endoscopes can be estimated within less than the average color difference$\Delta E_{\iota l\mathcal{V}}^{*}=2.66$ in the $\mathrm{L}^{*}\mathrm{u}^{*}\mathrm{V}^{*}$ color space. Computer simulations of the color reproduction ofelectronic endoscopes were performed, and the resulting spectral characteristics under variousilluminants are described and analyzed.1. IntroductionIn recent years, electronic endoscopes with CCD area sensor have been developed andwidely used instead of the conventional optical endoscopes that use color film. Electronicendoscopes can be easily applied to telemedicine, image filing, image processing and PictureArchiving and Communication Systems(PACS).Color information is important for the early stage of diagnosis in various kinds of stomachdiseases. However, color reproduction- of the images taken $.\mathrm{b}\mathrm{y}$ electronic endoscopes is poorcompared with the optical endoscope images because the present electronic endoscopes use theNTSC system.To improve the color reproduction of electronic endoscopes, we have developed an989 1997 122-131 122endoscopic spectrophotometer to measure the spectral reflectance of gastric mucous membrane.1We have measured about 1000 spectral reflectance using this spectrophotometer and reportedthe capability of automatic diagnosis using spectral data and chromaticity.23 We have alsoreported a color correction method in the electronic endoscope’s input system, using a simple$3\mathrm{x}3$ matrix to reproduce a colorimetrically correct color based on the analysis of the spectralreflectance.4These color correction methods are based on the analysis of pixel to pixel, because theendoscopic spectrophotometer can only measure the spectral reflectance of gastric mucousmembrane at $\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{f}\mathrm{i}\dot{\mathrm{c}}$ points and cannot measure the spectra in entire viewing fields. Tomeasure the spectral reflectance of $\mathrm{M}\mathrm{x}\mathrm{N}$ sampled points, normally, a multi-band method hasbeen used, but the method is difficult to apply to the electronic endoscope.In this study, we analyzed 310 spectral reflectance of gastric mucous membranes byprincipal component analysis, and it became clear that those measured reflectance spectra canbe adequately described using only thuee principal components. Therefore, the spectralreflectance of all pixels in gastric mucous membranes can be estimated by $\mathrm{R},$ $\mathrm{G},$ $\mathrm{B}$ signals takenby electronic endoscopes. On the basis of the experimental results, the computer simulation ofthe color reproduction of electronic endoscopes was performed.2. Principal Component Analysis of the Spectral Reflectance of Gastric MucousMembraneFigure 1 shows the block diagram of developed endoscopic spectrophotometer. The instrumentconsist of $1\mathrm{i}_{\mathrm{o}}\sigma \mathrm{h}\mathrm{t}$ source with xenon lamp; a conventional spectrophotometer; an opticalmultichannel analyzer with 1024 silicon photodiodes; and a personal computer. We measuredabout thousand reflectance spectra of gastric mucous membrane. Figure 2 (a) and (b) areexamples of spectral reflectance of gastric mucous membrane. Thuee hundred and ten spectralreflectance of gastric mucous membranes excluding blurred reflectance spectra, have beenanalyzed by the principal component analysis. Figure 3 shows the cumulative contribution ratioof the pnincipal components. The result shows that the contribution ratio from first to thirdcomponents is about 99%; in other words, the reflectance spectra of gastric mucous membranescan be represented approximately 99% of the time using a linear combination of three principalcomponents $\mathrm{u}_{1},$ $\mathrm{u}_{2}$ and $\mathrm{u}_{3}$ . Figure 4 shows these three principal components or eigenvectorsobtained from the covariance matrix of the reflectance spectra.Figure 5 shows the plan of an electronic endoscopes that utilizes $\mathrm{R},$ $\mathrm{G},$ $\mathrm{B}$ sequencing,incorporating a rotating color wheel comprising of $\mathrm{R},$ $\mathrm{G},$ $\mathrm{B}$ filters used in the simulation. Thevalue $E(\lambda)$ is the spectral radiance distnibution of light source, $f_{R}(\lambda),$ $f_{G}(\lambda),$ $f_{B}(\lambda)$ arerespectively, spectral transmittance of the $\mathrm{R},$ $\mathrm{G},$ $\mathrm{B}$ filters, $S(\lambda)$ is spectral sensitivity of CCD123$..\vee\wedge\backslash _{\mathrm{o}_{\underline{\mathrm{O}}}}^{\mathrm{o}}\vee\varpi$$.\cdot\vee 0\underline{\mathrm{o}\subset}\supset$$\frac{=}{\mathrm{o}\mathrm{c},0}$$. \frac{\sim\varpi\geq\omega}{0=\in\supset}$$\mathrm{P}\mathrm{r}\mathrm{l}\mathrm{n}\mathrm{C}\mathfrak{l}\mathrm{P}\mathrm{a}\mathrm{I}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{p}\circ \mathrm{r}\iota 6l\iota 1$Fi.gure 3. Cumulative contribution ratio of theprincipal components.Figure 1. Diagram of the endoscopic spectrophotometer.$\angle_{\mathrm{k}\cup\cup}$ $3\mathrm{U}\mathrm{U}$ OUU $/\mathrm{U}\mathrm{U}$wavelength $(\mathrm{n}\mathrm{m})$ wavelcngm $(\mathrm{n}\mathrm{m})$(a). (b)Figure 2. Examples of reflectance spectra of gastoric mucous membranes.124area sensor, $L(\lambda)$ is combined spectral transmittance of the light guide fiber and the imaginglens, and $o(\lambda)$ is the spectral reflectance of gastric mucous membranes. Then the output $\mathrm{R},$ $\mathrm{G}$and $\mathrm{B}$ values of the electronic endoscopes are calculated by Eq. (1).$v_{R}= \int f_{R}(\lambda)E(\lambda)L(\lambda)s(\lambda)_{o()d\lambda}\lambda$$v_{G}= \int f_{G}(\lambda)E(\lambda)L(\lambda)S(\lambda)_{o()d\lambda}\lambda$ (1)$v_{B}= \int f_{B}(\lambda)E(\lambda)L(\lambda)s(\lambda)_{o()d\lambda}\lambda$Although the spectral characteristics of the systems are inherently continuous, for theconvenience the discrete vector notation is introduced as $E(\lambda):\mathrm{E}$ , $f_{R}(\lambda)$ , $f_{G}(\lambda)$ ,$f_{B}(\lambda):\mathrm{f}\mathrm{i}(\mathrm{i}=\mathrm{R},\mathrm{G},\mathrm{B}),$ $L(\lambda):\mathrm{L},$ $S(\lambda):\mathrm{s},$ $o(\lambda):\mathrm{O}$ .Then the $\mathrm{R},$ $\mathrm{G}$ and $\mathrm{B}$ values of an object that are obtained by an endoscope are expressed as$v_{i}=\mathrm{f}_{i}f\mathrm{E}\mathrm{L}\mathrm{S}\mathrm{o}$$=\mathrm{F}_{i}^{t}\mathrm{o}$ (2)$(i=R,G,B)$where $\mathrm{F}_{i}^{t}(i=R,G,B)$ is an overall spectral sensitivities of the electronic endoscopes, and $[\bullet]^{f}$represents transposition.Because the orthonormal vectors with $\mathrm{n}$-dimensional space were obtained by the principalcomponent analysis, the reflectance spectra of gastric mucous membranes $0$ can be calculatedby$0= \sum_{i=1}\alpha_{i}n\mathrm{u}_{i^{+}}\mathrm{m}$ (3)where $\alpha_{i}$ is an expansion coefficient and $\mathrm{m}$ is the mean vector of 310 reflectance spectra. Fromthe result of principal component analysis, $0$ can be approximated by the linear $\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{b}\mathrm{i}\mathrm{n}\mathrm{a}\dot{\mathfrak{a}}_{\mathrm{O}}\mathrm{n}$ ofthe three eigenvectors $\mathrm{u}_{1},$ $\mathrm{u}_{2}$ and $\mathrm{u}_{3}$ with the mean vector $\mathrm{m}$ .$0 \equiv\sum_{i=1}^{3}-\alpha_{i}\mathrm{u}i+\mathrm{m}=[\mathrm{u}_{1} \mathrm{u}_{2} \mathrm{u}_{3}]+\mathrm{m}$ (4)From Eqs.(2) and (4)125$4\mathrm{U}\mathrm{U}$ $3\mathrm{U}\mathrm{U}$ OUU $/\mathrm{U}\mathrm{U}$wavelength $(\mathrm{n}\mathrm{m})$Figure 5. Simulation model of an electronic endoscope.(a) (b)Figure 6. Examples of estimated reflectance spectra.126$[_{\mathcal{V}_{B}}^{\mathcal{V}}\mathcal{V}_{G}]R=[_{\mathrm{F}_{B}^{t}}^{\mathrm{F}_{R}^{t}}\mathrm{F}_{c]\{[}^{t}\mathrm{u}1$$\mathrm{u}_{2}$$\mathrm{u}_{3}][_{\alpha}\alpha 2]_{-}\alpha_{3}1+\mathrm{m}$ (5)Then, the constant value $\alpha_{1},$ $\alpha_{2}$ and $\alpha_{3}$ is given by$=$$.\cdot-\}$ , (6)where,$=\mathrm{m}$ .Because $\mathrm{u}_{i}(i=1,2,3)$ and $\mathrm{m}$ are known and $\mathrm{F}_{i}^{t}(i=R,G,B),$ $v_{i}(i=R,G,B)$ and $v_{im}(i=R,G,B)$are obtained from the practical electronic endoscopes, the coefficients $\alpha_{1},$ $\alpha_{2}$ and $\alpha_{3}$ can be$\mathrm{c}\mathrm{a}1_{\mathrm{C}\mathrm{u}}\mathrm{I}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{d}$ by $\mathrm{F}\mathrm{q}.(6)$ .The reflectance spectra of gastnic mucous membrane can be estimated by Eq. (4); therefore itbecomes possible to calculate the reflectance spectra of all pixels $O(x,y,\lambda)$ from the $\mathrm{R},$ $\mathrm{G},$ $\mathrm{B}$output signals obtained by a general endoscope system.We estimated 310 reflectance spectra of gastric mucous membranes by the above method.Examples of $\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{e}\dot{\grave{\mathrm{d}}}$ reflectance spectra are shown in Fig. 6 (a) and (b). Estimated spectralreflectance are approximately equal to the measured spectral reflectance. On the other hand,color differences of $CIEL^{*}\mathcal{U}^{**}\mathcal{V}$ (under illuminant D65) between measured and estimatedspectra are shown in Fig. 7. The average color difference $\Delta E_{uv}^{*}$ of three hundred and tensamples was 2.66, the minimum color difference $\Delta E_{u\nu}^{*}$ was 0.64 and maximum color difference$\Delta E_{u\nu}^{*}$ was 9.14. As shown in the next section, we simulated color reproduction of endoscopeimages under 12 different illuminants. The color reproduction of 310 measured and estimatedreflectance spectra under those illuminants was also evaluated in $CIEL^{\mathrm{s}}uv\wedge*\mathrm{C}\mathrm{o}\mathrm{l}\mathrm{o}\mathrm{r}$ space. Theresultant $\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{a}_{\mathrm{o}}\sigma \mathrm{e}$ color difference $\Delta E_{uv}^{*}$ between measured and estimated ranged from 2.20 to3.29. These results indicate that the proposed method has sufficient accuracy to estimate thereflectance spectra of gastric mucous membrane and that the slight estimation error does notaffect the simulation of color reproduction of endoscope images.3. Simulation of Color Reproduction in Electronic EndoscopesThe method was applied to improve color reproduction of practical eight kinds of endoscopic127images with $270\mathrm{x}270$ pixels. Namely gastnic membranes with (a) cancer, (b) early stage ofchronic gastritis, (c) chronic gastritis $\mathrm{I}\mathrm{I}\mathrm{b},$ $(\mathrm{d})$ advanced stage of chronic gastritis, (e) gastriculcer, ($\mathrm{t}\gamma$ normal membrane, (g) polyp and (h) xanthoma were used for computer simulation. Inthe simulation, first the spectral reflectance of gastric mucous membrane of all pixels wascalculated from $\mathrm{R},$ $\mathrm{G},$ $\mathrm{B}$ signals, then the color reproduction of electronic endoscopes undertwelve illuminants was estinated by computer simulation. The illuminants used in thesimulation are CIE standard illuminant $\mathrm{A}$ ; D65; CIE fluorescent lamps Fl, F2, and F3, xenonlamp, metal halide lamp; fluorescent lamps for home use Ml, M2, M3, M4, M5. The densitybalance for each iuninant was chosen so that the integral of the spectral radiance of eachilluminant weighted by the spectral sensitivity of human visual system was constant.The color temperature of the illuminants is shown in Table 1. Figure 8 shows thedistribution of $\mathrm{u}’ \mathrm{v}$ ’ chromaticities for the images under illuminant CIE standard A and xenonlamp. The distribution shifts depending on the illuminants. To estimate the optimum illuminant,those obtained images were displayed on a CRT and evaluated by two physicians who arespecialist of endoscopic diagnosis. Images by xenon lamp was selected as most preferred.Because the xenon lamp has been used as light source in conventional electronic endoscopes,we consider that the color memory of doctors may be fixed to the color reproduction by xenonlamp illumination. However, the doctors said that in the diagnosis of the membrane,particularly in the vascular pattem of polyp, image by fluorescent M2 is more significant thanthe image with other illuminants.As the physical measure of the illuminant, we assumed that the illuminant with wide colorgamut in reproduced images may be better for electronic endoscopes. Then, we defined theevaluation value $E_{V}$ as Eq. (7).$E_{V}=\underline{\sum_{i=1j=}^{m-}\sum_{1}1mi+\{(L_{j}^{*}-L_{j})2(^{\wedge*}+\mathcal{U}-u)^{2}+(*\mathcal{V}iji*-\mathcal{V})^{2}j\mathrm{I}*/_{\sim}1_{\circ}}$(7)$m2C$where $\mathrm{m}$ is the number ofmeasured spectra. The equation means that as $E_{V}$ becomes larger,the color differences between pixels becomes large.Figure 9 shows the results of the calculated evaluation values. The evaluation values of theiuuminant M3, M4, M2, A and Fl are larger than that of the xenon lamp. Simulation imageswith large $E_{V}$ enhanced the color difference between pixels. Thus, for example, the vascularpattems of the polyp and chronic gastritis became more clear although the color reproduction ispoor.128$\iota’.|$ $\iota’..r$ $\mathrm{u}.\supset$$\mathrm{u}$’(a) C1E Asample no. $().()$Figure 7. Color differences of $\mathrm{C}\mathrm{I}\mathrm{E}\mathrm{L}^{*}\mathrm{u}*_{\mathrm{V}}*$between measured and estimated reflectancespectra. $\mathrm{v}^{\iota}\mathrm{t}$ ) $.4$$\{)_{\vee}^{\gamma}$.{ $).|$ $()..1$ 0.5$\mathrm{t}1$‘(b) xenon lampFigure 8. The $\mathrm{u}’ \mathrm{v}$ ’ chromaticity of a polyp imageunder two kinds of illuminants:(a)CIE $\mathrm{A},$ $(\mathrm{b})$ xenonlamp.Table 1. The color temperatures of the 12. illuminants used{llumlnant Color temperatureC1E standard A 2856 $\mathrm{K}$$\mathrm{C}\mathrm{l}\mathrm{E}$ standa $r\mathrm{d}$ D65 6500 $\mathrm{K}$C1E fluorescent Fl about 3000 $\mathrm{K}$Figure 9. Results of the calculated C1E $\mathrm{t}\downarrow \mathrm{u}\mathrm{o}\prime \mathrm{e}S\mathrm{o}\mathrm{e}\mathrm{n}\downarrow \mathrm{F}.2$ about 4000 $\mathrm{K}$$\mathrm{e}\mathrm{V}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}.\mathrm{n}$ valu.es.$-$C1E fluorescent $\mathrm{F}3|$ about 6500 $\mathrm{K}$Xenon about 5100 $\mathrm{K}$Metal Halogen 4274 $\mathrm{K}$FIuorescent lamp Ml 6245 $\mathrm{K}$$\mathrm{F}\mathrm{l}\cup 0\Gamma \mathrm{e}\mathrm{S}\mathrm{C}\mathrm{e}\mathrm{n}\mathrm{t}$ lamp M2 7763 $\mathrm{K}$FIuorescent lamp M3 2766 $\mathrm{K}$$\mathrm{F}\mathrm{l}\mathrm{u}0r\mathrm{e}\epsilon \mathrm{C}\mathrm{e}\mathrm{n}\dagger$ lamp M4 5150 $\mathrm{K}$$\mathrm{F}\mathrm{l}\mathrm{u}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{c}\mathrm{e}\mathrm{n}\iota$lamp M5 6758 $\mathrm{K}$1294. Conclusion and DiscussionIn this paper, we showed that the spectral reflectance of gastric mucous membranes can beestimated $\mathrm{w}\mathrm{i}\mathrm{t}\acute{\mathrm{h}}\dot{\mathrm{h}}\mathrm{i}\mathrm{g}\dot{\mathrm{h}}$ accuracy using only three $\vee \mathrm{p}\dot{\mathrm{m}}$cipalcomponents-. On the basis of theanalysis, the spectral reflectance of 270 $\mathrm{x}270$ pixels were estimated from the $\mathrm{R},$ $\mathrm{G},$ $\mathrm{B}$ signals ofgastric mucous membrane taken by $\mathrm{p}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{i}\dot{\mathrm{C}}\mathrm{a}\mathrm{l}$ electronic endoscopes. The color reproductioncharacteristics of eight kinds of membranes with different disorders under the 12 differentilluminants were estimated by computer simulation. $\mathrm{T}\mathrm{h}\mathrm{o}$se- images were rated by physicians,and the color reproduction of the image taken by xenon lamp was most preferred, however, inthe diagnosis of vascular pattem, another illuminant was better than the xenon.In this paper, we only considered the improVe..ment of color reproduction of electronicendoscopes based on colorimetric color reproduction under various illuminants. The perceivedcolors, however, are different in practice from the colorimetric color reproduction, that arecaused by the chromatic adaptation of human vision. The color appearance can be predictedusing chromatic adaptation $\mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}\mathrm{S}6,78$ such as the von Kries, Fairchild, RLAB , Hunt, etc..Therefore, furthermore research considering color appearance models for the optimum colorreproduction should be performed for each of the various diagnosis and viewing condition.AcknowledgmentAuthor wishes to thank Dr. Kunihiko Nagahara and Dr. Takama Maekawa, Kyoto NationalHospital for their cooperation for image evaluation and their useful comments.References(*)Presented in part at the J. Imaging Science and Technology, 40(6), 494(1996)1. Y. Miyake, T. Sekiya, T. Yano, S. Kubo and T. Hara; “ A New $\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{c}\iota \mathrm{r}\mathrm{Q}..\mathrm{p}\mathrm{h}_{0}\mathrm{t}\mathrm{o}\mathrm{m}\mathrm{e}\mathrm{t}\mathrm{e}\mathrm{r}$forMeasuning the Spectral Reflectance of Gastric Mucous Membrane “, J. Pho-togr. Sci, ,37, 13(1989).2. T. Sekiya, T. Hara, H. Yaguchi and Y. Miyake ; “ Evaluation and Analysis of $\mathrm{s}_{\mathrm{P}^{\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{r}\mathrm{a}}}!$Reflectance of Gastric Mucous Membranes (I) “, J. SPSTJ, 54 (2), 125 (199i) in Japanese.3. T. Sekiya, T. Hara, H. Haneishi, H. Yaguchi and Y. Miyake; “ Evaluation and Analysis ofSpectral Reflectance of Gastric Mucous Membranes (II) “, J. SPSTJ, 54 (3), 346 (1991) inJapanese.4. H. Haneishi, T. Shiobara and Y. Miyake; “ Color correction for colorimetric colorreproduction in an electronic endoscope “, Optics Communications, 114 (1), 57 (1995).5. N. Ojima, H. Haneishi and Y. Miyake; “ The Appearance of Skin with Make-up (III)(Estimation for spectral reflectance of skin with the HDTV color image) “, J. SPSTJ, 57 (2),78 (1994) in Japanese.1306. $\mathrm{M}.\mathrm{D}$ .Fairchild, $\mathrm{R}.\mathrm{S}$ .Bems: Image Color Appearance Specification Through Extension ofCIELAB, Color Res. and Appl., 18, 178(1993)7. Francisco H. Imai, N.Tsumura, H.Haneishi and $\mathrm{Y}.\mathrm{M}\mathrm{i}\mathrm{y}\mathrm{a}\mathrm{k}\mathrm{e}:\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{C}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}$ of $\mathrm{C}\mathrm{o}1.\mathrm{o}\mathrm{r}$Reproduction for Skin Color Under Different Illuminants Based on the Color AppearanceModels, J.Imaging Science and Technology (in print)8. $\mathrm{R}.\mathrm{W}.\mathrm{G}.\mathrm{H}\mathrm{u}\mathrm{n}\mathrm{t}:\mathrm{A}$ Model of Colour Vision for Predicting Colour Appearance in VariousViewing Conditions, Color Res. Appl., 12, 297(1987)Y.MiyakeDepartment ofInformation and Computer Sciences, Chiba University1-33 Yayoi-cho, Inage-ku, Chiba 263, Japan$E$-rnail: miyake@ics. $tj$. chiba-u. $ac$.jp131

Computer Simulation for Improvement of Color Reproduction in Electronic Endoscopes(Domain Decomposition Methods and Related Topics)

http://repository.kulib.kyoto-u.ac.jp/dspace/bitstream/2433/61067/1/0989-11.pdf

Computer Simulation for Improvement of Color Reproduction in Electronic Endoscopes(Domain Decomposition Methods and Related Topics)

Abstract

Similar works

Full text

Available Versions

Kyoto University Research Information Repository