A mobile augmented reality application for supporting real-time skin lesion analysis based on deep learning

AbstractMelanoma is considered the deadliest skin cancer and when it is in an advanced state it is difficult to treat. Diagnoses are visually performed by dermatologists, by naked-eye observation. This paper proposes an augmented reality smartphone application for supporting the dermatologist in the real-time analysis of a skin lesion. The app augments the camera view with information related to the lesion features generally measured by the dermatologist for formulating the diagnosis. The lesion is also classified by a deep learning approach for identifying melanoma. The real-time process adopted for generating the augmented content is described. The real-time performances are also evaluated and a user study is also conducted. Results revealed that the real-time process may be entirely executed on the Smartphone and that the support provided is well judged by the target users

Designing a Tablet-based Intervention to Support Self-checking for Melanoma

Postprin

Skin Lesion Extraction And Its Application

In this thesis, I study skin lesion detection and its applications to skin cancer diagnosis. A skin lesion detection algorithm is proposed. The proposed algorithm is based color information and threshold. For the proposed algorithm, several color spaces are studied and the detection results are compared. Experimental results show that YUV color space can achieve the best performance. Besides, I develop a distance histogram based threshold selection method and the method is proven to be better than other adaptive threshold selection methods for color detection. Besides the detection algorithms, I also investigate GPU speed-up techniques for skin lesion extraction and the results show that GPU has potential applications in speeding-up skin lesion extraction. Based on the skin lesion detection algorithms proposed, I developed a mobile-based skin cancer diagnosis application. In this application, the user with an iPhone installed with the proposed application can use the iPhone as a diagnosis tool to find the potential skin lesions in a persons\u27 skin and compare the skin lesions detected by the iPhone with the skin lesions stored in a database in a remote server

Multimedia sensors embedded in smartphones for ambient assisted living and e-health

The final publication is available at link.springer.com[EN] Nowadays, it is widely extended the use of smartphones to make human life more comfortable. Moreover, there is a special interest on Ambient Assisted Living (AAL) and e-Health applications. The sensor technology is growing and amount of embedded sensors in the smartphones can be very useful for AAL and e-Health. While some sensors like the accelerometer, gyroscope or light sensor are very used in applications such as motion detection or light meter, there are other ones, like the microphone and camera which can be used as multimedia sensors. This paper reviews the published papers focused on showing proposals, designs and deployments of that make use of multimedia sensors for AAL and e-health. We have classified them as a function of their main use. They are the sound gathered by the microphone and image recorded by the camera. We also include a comparative table and analyze the gathered information.Parra-Boronat, L.; Sendra, S.; Jimenez, JM.; Lloret, J. (2016). Multimedia sensors embedded in smartphones for ambient assisted living and e-health. Multimedia Tools and Applications. 75(21):13271-13297. doi:10.1007/s11042-015-2745-8S13271132977521Acampora G, Cook DJ, Rashidi P, Vasilakos AV (2013) A survey on ambient intelligence in healthcare. Proc IEEE 101(12):2470–2494Al-Attas R, Yassine A, Shirmohammadi S (2012) Tele-Medical Applications in Home-Based Health Care. In proceeding of the 2012 I.E. International Conference on Multimedia and Expo Workshops (ICMEW 2012). Jul. 9–13, 2012. Melbourne, Australia. (pp. 441–446)Alemdar H, Ersoy C (2010) Wireless sensor networks for healthcare: a survey. Comput Netw 54(15):2688–2710Alqassim S, Ganesh M, Khoja S, Zaidi M, Aloul F, Sagahyroon A (2012) Sleep apnea monitoring using mobile phones. In proceedings of the 14th International Conference on e-Health Networking, Applications and Services (Healthcom 2012). Oct. 10 – 13, 2012. Beijing, China. (pp. 443–446)Anderson G, Horvath J (2004) The growing burden of chronic disease in America. Public Health Rep 119(3):263–270Aquilano M, Cavallo F, Bonaccorsi M, Esposito R, Rovini E, Filippi M, Carrozza MC (2012) Ambient assisted living and ageing: Preliminary results of RITA project. In proceedings of 34th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2012), Aug. 28-Sept. 1, 2012. San Diego USA. (pp. 5823–5826)Bellini P, Bruno I, Cenni D, Fuzier A, Nesi P, Paolucci M (2012) Mobile Medicine: semantic computing management for health care applications on desktop and mobile devices. Multimed Tools Appl 58(1):41–79Boulos MN, Wheeler S, Tavares C, Jones R (2011) How smartphones are changing the face of mobile and participatory healthcare: an overview, with example from eCAALYX. Biomed Eng Online 10(1):24Bourouis A, Feham M, Hossain MA, Zhang L (2014) An intelligent mobile based decision support system for retinal disease diagnosis. Decis Support Syst 59(2014):341–350Bourouis A, Zerdazi A, Feham M, Bouchachia A (2013) M-health: skin disease analysis system using Smartphone’s camera. Procedia Comput Sci 19(2013):1116–1120M.W. Brault, (2010). Americans With Disabilities: 2010. Household Economic Studies. In United States Census Bureau website. Available at: www.census.gov/prod/2012pubs/p70-131.pdf Last Access 16 Dec 2014Breath Counter App. In Google Play website. Available at: https://play.google.com/store/apps/details?id=com.softrove.app.bc Last Access 30 Nov 2014Cardinaux F, Bhowmik D, Abhayaratne C, Hawley MS (2011) Video based technology for ambient assisted living: a review of the literature. J Ambient Intell Smart Environ 3(3):253–269Cardiograph App. In Google Play website. Available at: https://play.google.com/store/apps/details?id=com.macropinch.hydra.android . Last Access 30 Nov 2014Chaaraoui AA, Climent-Pérez P, Flórez-Revuelta F (2012) A review on vision techniques applied to human behaviour analysis for ambient-assisted living. Expert Syst Appl 39(12):10873–10888Chen NC, Wang KC, Chu HH (2012) Listen-to-nose: a low-cost system to record nasal symptoms in daily life. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing (UBIComp 2012). Sep. 05–08, 2012. Pittsburgh, USA. (pp. 590–591)Chiarini G, Ray P, Akter S, Masella C, Ganz A (2013) mHealth technologies for chronic diseases and elders: a systematic review. IEEE J Sel Areas Commun 31(9):6–18Color Detector App In Google Play website. Available at: //play.google.com/store/apps/details?id = com.mobialia.colordetector. Last Access 30 Nov 2014Colorblind Assitant App. In Google Play website. Available at: https://play.google.com/store/apps/details?id=com.unclechromedome.colorblindassistant . Last Access 30 Nov 2014Dale O, Solheim I, Halbach T, Schulz T, Spiru L, Turcu I (2013) What seniors want in a mobile Help-On-Demand service. In proceedings of the Fifth International Conference on eHealth, Telemedicine, and Social Medicine (eTELEMED 2013). Feb. 24 – Mar. 1, 2013. Nice, France. (pp. 96–101)Estepa AJ, Estepa R, Vozmediano J, Carrillo P (2014) Dynamic VoIP codec selection on smartphones. Netw Protoc Algoritm 6(2):22–37Falk TH, Maier M (2013) Context awareness in WBANs: a survey on medical and non-medical applications. IEEE Wirel Commun 20(4):30–37Franco C, Fleury A, Guméry PY, Diot B, Demongeot J, Vuillerme N (2013) iBalance-ABF: a smartphone-based audio-biofeedback balance system. IEEE Trans Biomed Eng 60(1):211–215García M, Lloret J, Bellver I, Tomás J (2013) Intelligent IPTV Distribution for Smart Phones (Book Chapter 13). In Intelligent Multimedia Technologies for Networking Applications. IGI GlobalGregoski MJ, Mueller M, Vertegel A, Shaporev A, Jackson BB, Frenzel RM, Treiber FA (2012) Development and validation of a smartphone heart rate acquisition application for health promotion and wellness telehealth applications. Int J Telemed Appl 2012, 1. Article ID 696324Grimaldi D, Kurylyak Y, Lamonaca F, Nastro A (2011) Photoplethysmography detection by smartphone’s videocamera. In proceedings of the 6th International Conference on Intelligent Data Acquisition and Advanced Computing Systems (IEEE IDAACS 2011), Sep. 15–17, 2011. Prague, Czech Republic. (Vol. 1, pp. 488–491)Gurrin C, Qiu Z, Hughes M, Caprani N, Doherty AR, Hodges SE, Smeaton AF (2013) The smartphone as a platform for wearable cameras in health research. Am J Prev Med 44(3):308–313Haché G, Lemaire ED, Baddour N (2011) Wearable mobility monitoring using a multimedia smartphone platform. IEEE Trans Instrum Meas 60(9):3153–3161Heathers JA (2013) Smartphone-enabled pulse rate variability: an alternative methodology for the collection of heart rate variability in psychophysiological research. Int J Psychophysiol 89(3):297–304Hoseini-Tabatabaei SA, Gluhak A, Tafazolli R (2013) A survey on smartphone-based systems for opportunistic user context recognition. ACM Comput Surv (CSUR) 45(3):1–51, Paper No. 27Illiger K, Hupka M, von Jan U, Wichelhaus D, Albrecht UV (2014) Mobile technologies: expectancy, usage, and acceptance of clinical staff and patients at a University Medical Center. JMIR mHealth uHealth 2(4), e42Kanjo E (2012) Tools and architectural support for mobile phones based crowd control systems. Netw Protoc Algoritm 4(3):4–14Kawano Y, Yanai K (2014) FoodCam: a real-time food recognition system on a smartphone. Multimedia Tools and Applications,Published online:April 2014: 1–25Khan FH, Khan ZH (2010) A systematic approach for developing mobile information system based on location based services. Netw Protoc Algoritm 2(2):54–65Kochanov D, Jonas S, Hamadeh N, Yalvac E, Slijp H, Deserno TM (2014) Urban Positioning Using Smartphone-Based Imaging. In Bildverarbeitung für die Medizin, 2014: 186–191Kurniawan S (2008) Older people and mobile phones: a multi-method investigation. Int J Human-Comput Stud 66(12):889–901Lacuesta R, Lloret J, Sendra S, Peñalver L (2014) Spontaneous Ad Hoc mobile cloud computing network. Sci World J 2014:1–19Lakens D (2013) Using a Smartphone to measure heart rate changes during relived happiness and anger. IEEE Trans Affect Comput 5(3):217–226Larson EC, Goel M, Boriello G, Heltshe S, Rosenfeld M, Patel SN (2012) Spirosmart: using a microphone to measure lung function on a mobile phone, In proceedings of the 2012 ACM Conference on Ubiquitous Computing (UBIComp 2012). Sep. 05–08, 2012. Pittsburgh, USA. (pp. 280–289)Lee J, Reyes BA, McManus DD, Mathias O, Chon KH (2012) Atrial fibrillation detection using a smart phone. In proceedings of the 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2012). Aug.28-Sep.1, 2012. San Diego, (pp. 1177–1180)Lloret J, Garcia M, Bri D, Diaz JR (2009) A cluster-based architecture to structure the topology of parallel wireless sensor networks. Sensors (Basel) 9(12):10513–10544Lu H, Frauendorfer D, Rabbi M, Mast MS, Chittaranjan GT, Campbell AT, Gatica-Perez D, Choudhury T (2012) StressSense: detecting stress in unconstrained acoustic environments using smartphones. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing (UBIComp 2012). Sep. 05–08, 2012. Pittsburgh, USA. (pp. 351–360)Macías E, Abdelfatah H, Suárez A, Cánovas A (2011) Full geo-localized mobile video in Android mobile telephones. Netw Protoc Algoritm 3(1):64–81Macias E, Lloret J, Suarez A, Garcia M (2012) Architecture and protocol of a semantic system designed for video tagging with sensor data in mobile devices. Sensors 12(2):2062–2087Macias E, Suarez A, Lloret J (2013) Mobile sensing systems. Sensors 13(12):17292–17321MedCam App. In Google Play website. Available at: https://play.google.com/store/apps/details?id=com.cupel.MedCam . Last Access 30 Nov 2014Monteiro DM, Rodrigues JJ, Lloret J, Sendra S (2014) A hybrid NFC–Bluetooth secure protocol for Credit Transfer among mobile phones. Secur Commun Netw 7(2):325–337Mosa ASM, Yoo I, Sheets L (2012) A systematic review of healthcare applications for smartphones. BMC Med Inform Decis Mak 12(1):67MyEarDroid App. In Google Play website. Available at: https://play.google.com/store/apps/details?id=com.tecnalia.health.myeardroid . Last Access 30 Nov 2014O’Grady MJ, Muldoon C, Dragone M, Tynan R, O’Hare GM (2010) Towards evolutionary ambient assisted living systems. J Ambient Intell Humaniz Comput 1(1):15–29Poppe R (2010) A survey on vision-based human action recognition. Image Vis Comput 28(6):976–990Quit Snoring App. In Google Play website. Available at: https://play.google.com/store/apps/details?id=com.ptech_hm.qs . Last Access 30 Nov 2014Rahman MA, Hossain MS, El Saddik A (2013) Context-aware multimedia services modeling: an e-Health perspective. Multimed Tools Appl 73(3):1147–1176Sendra S, Granell E, Lloret J, Rodrigues JJPC (2014) Smart collaborative mobile system for taking care of disabled and elderly people. Mob Netw Appl 19(3):287–302Smartphone Milestone: Half of Mobile Subscribers Ages 55+ Own Smartphones Mobile. Online report.(April 22,2014). In the Nielsen Company website. Available at: http://www.nielsen.com/us/en/insights/news/2014/smartphone-milestone-half-of-americans-ages-55-own-smartphones.html Last Access 25 Nov 2014Smith A (2013) Smartphone Ownership 2013. On-line Report June 5, 2013. In Pew Research Center’s Internet & American Life Project website. Available at: http://www.pewinternet.org/2013/06/05/smartphone-ownership-2013/ Last Access 25 Nov 2014SnoreClock App. In Google Play website. Available at: https://play.google.com/store/apps/details?id=de.ralphsapps.snorecontrol Last Access 30 Nov 2014Storf H, Kleinberger T, Becker M, Schmitt M, Bomarius F, Prueckner S (2009) An event-driven approach to activity recognition in ambient assisted living. Lect Notes Comput Sci 5859:123–132Su X, Tong H, Ji P (2014) Activity recognition with smartphone sensors. Tsinghua Sci Technol 19(3):235–249Tapu R, Mocanu B, Bursuc A, Zaharia T (2013) A smartphone-based obstacle detection and classification system for assisting visually impaired people. In proceedings of the 2013 I.E. International Conference on Computer Vision Workshops (ICCVW 2013). Dec. 2–8, 2013. Sydney, Australia. (pp. 444–451)The vOICe for Android App. In Google Play website. Available at: https://play.google.com/store/apps/details?id=vOICe.vOICe . Last Access 30 Nov 2014Tudzarov A, Janevski T (2011) Protocols and algorithms for the next generation 5G mobile systems. Netw Protoc Algoritm 3(1):94–114Tyagi A, Miller K, Cockburn M (2012) e-Health tools for targeting and improving melanoma screening: a review. J Skin Cancer 2012, Article ID 437502Voice Cam for Blind App. In Google Play website. Available at: https://play.google.com/store/apps/details?id=com.prod.voice.cam Last Access 30 Nov 2014Wadhawan T, Situ N, Rui H, Lancaster K, Yuan X, Zouridakis G (2011) Implementation of the 7-point checklist for melanoma detection on smart handheld devices. In proceedings of the 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, (EMBC 2011). Aug. 30- Sep 03, 2011. Boston, MA, USA (pp. 3180–3183)Xiong H, Zhang D, Zhang D, Gauthier V (2012) Predicting mobile phone user locations by exploiting collective behavioral patterns. In proceedings of the 9th International Conference on Ubiquitous Intelligence & Computing and 9th International Conference on Autonomic & Trusted Computing (UIC/ATC). 4–7 Sept. 2012. Fukuoka, Japan. (pp. 164–171)Xu X, Shu L, Guizani M, Liu M, Lu J (2014) A survey on energy harvesting and integrated data sharing in wireless body area networks. Int J Distrib Sens Netw. Article ID 438695Yu W, Su X, Hansen J (2012) A smartphone design approach to user communication interface for administering storage system network. Netw Protoc Algoritm 4(4):126–155Zhang D, Vasilakos AV, Xiong H (2012) Predicting location using mobile phone calls. ACM SIGCOMM Comput Commun Rev 42(4):295–296Zhang D, Xiong H, Yang L, Gauither V (2013) NextCell: predicting location using social interplay from cell phone traces. EEE Trans Comput 64(2):452–46

Position statement of the EADV Artificial Intelligence (AI) Task Force on AI‐assisted smartphone apps and web‐based services for skin disease

Background: As the use of smartphones continues to surge globally, mobile applications (apps) have become a powerful tool for healthcare engagement. Prominent among these are dermatology apps powered by Artificial Intelligence (AI), which provide immediate diagnostic guidance and educational resources for skin diseases, including skin cancer. Objective: This article, authored by the EADV AI Task Force, seeks to offer insights and recommendations for the present and future deployment of AI‐assisted smartphone applications (apps) and web‐based services for skin diseases with emphasis on skin cancer detection.MethodsAn initial position statement was drafted on a comprehensive literature review, which was subsequently refined through two rounds of digital discussions and meticulous feedback by the EADV AI Task Force, ensuring its accuracy, clarity and relevance. Results: Eight key considerations were identified, including risks associated with inaccuracy and improper user education, a decline in professional skills, the influence of non‐medical commercial interests, data security, direct and indirect costs, regulatory approval and the necessity of multidisciplinary implementation. Following these considerations, three main recommendations were formulated: (1) to ensure user trust, app developers should prioritize transparency in data quality, accuracy, intended use, privacy and costs; (2) Apps and web‐based services should ensure a uniform user experience for diverse groups of patients; (3) European authorities should adopt a rigorous and consistent regulatory framework for dermatology apps to ensure their safety and accuracy for users. Conclusions: The utilisation of AI‐assisted smartphone apps and web‐based services in diagnosing and treating skin diseases has the potential to greatly benefit patients in their dermatology journeys. By prioritising innovation, fostering collaboration and implementing effective regulations, we can ensure the successful integration of these apps into clinical practice

Automatic Extraction of Dermatological Parameters from Nevi Using an Inexpensive Smartphone Microscope: A Proof of Concept

The evolution of smartphone technology has made their use more common in dermatological applications. Here we studied the feasibility of using an inexpensive smartphone microscope for the extraction of dermatological parameters and compared the results obtained with a portable dermoscope, commonly used in clinical practice. Forty-two skin lesions were imaged with both devices and visually analyzed by an expert dermatologist. The presence of a reticular pattern was observed in 22 dermoscopic images, but only in 10 smartphone images. The proposed paradigm segments the image and extracts texture features which are used to train and validate a neural network to classify the presence of a reticular pattern. Using 5-fold cross-validation, an accuracy of 100% and 95% was obtained with the dermoscopic and smartphone images, respectively. This approach can be useful for general practitioners and as a triage tool for skin lesion analysis

Smartphone applications for triaging adults with skin lesions that are suspicious for melanoma

Background Melanoma accounts for a small proportion of all skin cancer cases but is responsible for the majority of skin cancer-related deaths. Early detection and treatment can improve survival. Smartphone applications are readily accessible and potentially offer an instant risk assessment of the likelihood of malignancy, so that the right people seek further medical attention from a clinician for more detailed assessment of the lesion. There is, however, a risk that melanomas will be missed and treatment delayed if the application reassures the user that their lesion is low risk. Objectives To determine the diagnostic accuracy of smartphone applications to rule out cutaneous invasive melanoma and intraepidermal melanocytic variants in adults with concerns about suspicious skin lesions.NIH

Smartphone as a Portable Detector, Analytical Device, or Instrument Interface

The Encyclopedia Britannia defines a smartphone as a mobile telephone with a display screen, at the same time serves as a pocket watch, calendar, addresses book and calculator and uses its own operating system (OS). A smartphone is considered as a mobile telephone integrated to a handheld computer. As the market matured, solid-state computer memory and integrated circuits became less expensive over the following decade, smartphone became more computer-like, and more more-advanced services, and became ubiquitous with the introduction of mobile phone networks. The communication takes place for sending and receiving photographs, music, video clips, e-mails and more. The growing capabilities of handheld devices and transmission protocols have enabled a growing number of applications. The integration of camera, access Wi-Fi, payments, augmented reality or the global position system (GPS) are features that have been used for science because the users of smartphone have risen all over the world. This chapter deals with the importance of one of the most common communication channels, the smartphone and how it impregnates in the science. The technological characteristics of this device make it a useful tool in social sciences, medicine, chemistry, detections of contaminants, pesticides, drugs or others, like so detection of signals or image