From dermoscopy to mobile teledermatology

Chapter 1 From Dermoscopy to Mobile Teledermatology LU´ IS ROSADO, MARIA JO˜ AO M. VASCONCELOS, RUI CASTRO Fraunhofer Portugal AICOS Porto, Portugal Email: {luis.rosado, maria.vasconcelos, rui.castro}@fraunhofer.pt JO˜ AO MANUEL R. S. TAVARES Instituto de Engenharia Mecˆ anica e Gest˜ ao Industrial, Departamento de Engenharia Mecˆ anica, Faculdade de Engenharia, Universidade do Porto Porto, Portugal Email: [email protected] 1.1 Introduction Skin cancer constitutes nowadays the most common malignancies in the Caucasian population, with incidences that are reaching epidemic proportions [1]. According to the American Cancer Society, one in every three cancers diagnosed is a skin cancer. Although Malignant Melanoma (MM) accounts for only a small percentage of skin cancer, it is far more 2Dermoscopy Image Analysis dangerous than other skin cancers and causes most skin cancer deaths. If detected during the early stages of its development, the success rates of recovery are very high, so early diagnosis of MM is extremely important. According to the World Health Organization [1], the global annual occurrences of non-melanoma and melanoma skin cancers are estimated to be between 2 and 3 million and 132,000, respectively. Malignant melanoma is the fastest growing form of cancer and, if not detected early, is the deadliest form, accounting for nearly 37,000 annual deaths. Despite the huge number of visual inspections (60 to 70 million) performed annually, melanoma-based mortality rates are as high as 23%, mainly due to missed or late diagnosed melanomas. This could be attributed to the limitations of the traditional visual inspection, which gives subjective results and are prone to uncertain diagnosis. In addition to missing out the melanomas, it may also gives rise to false positives resulting in unnecessary biopsies. It is estimated that only 3%out of the 6 to 7 million excisions performed annually turn out to be malignant melanoma. Also, the treatment for melanoma alone costs e1.18 billion a year. This explains the criticality of the need for early detection of melanoma. As an example, the skin cancer corresponds approximately to 25%of all malignant tumors detected each year in Portugal, affecting 1 in every 5 persons throughout life. MM represents 10%of all skin cancer but it is responsible for around 80%of all skin cancer-related deaths registered in Portugal [2]. Each year there are about 700 new cases of MM in Portugal. A recent study presented by the Portuguese Health Central Administration [3] shows an inadequate distribution of dermatologists along the country. The clearly inadequate distribution of the human resources of the Dermatology services comes from two main factors: (1) clear uneven regional distribution of dermatologists, with overallocation of specialists near the big urban centers; (2) the number of dermatologists currently working on the healthcare system represents only 60% of the required resources estimated. With increasing aging population globally, there is a growing incidence of skin cancer. But the percentage of population participating in skin cancer screening versus the incidence is alarmingly low. Late detection leads to rise in skin cancer mortalities, especially melanomas. As such, there is a need for complementing existing technologies in order to check out the malignancy level of a mole. In addition, given the current need to decrease the costs of the healthcare providers and the usage of new lightweight monitoring systems that can be carried around easily and used regularly by the patients it is considered crucial to find new ways of making better decisions on treatment without having to meet the patients face-to-face. The search for new “Personal Health Systems” is, in fact, one of the major priorities of the European Union current eHealth program [4]. In this context, Teledermatology has the potential to improve efficiency and quality aspects of care at lower costs and has proven to have similar accuracy and reliability as face-to-face dermatology [5,6]. Moreover, Mobile Teledermatology (MT) appears as a promising tool for personal dermatology data ac- From Dermoscopy to Mobile Teledermatology 3 quisition [7, 8], with the potential of not only becoming an easy applicable tool that empowers patients to adopt an active role in managing their own health status and facilitates the early diagnosis of skin cancers, but also offering the opportunity to make available consultations with experts in critical areas. Besides, an automated MT triage framework would not only have the purpose of delivering dermatologic expertise to those critical zones, but would also be important to prevent the overloading of the already scarce resources. A recent study [9] focused on the importance of MT in the developing world, which confirmed the added value of using a system that amplifies the access to dermatologic expertise in underserved regions. A detailed review about the computerized analysis of pigmented skin lesions and skin cancer images can be seen in [10, 11], where the authors present an extensive review of this research topic to microscopic (dermoscopic) and macroscopic (clinical) images. More recently, in [12] a state of the art in the utilization of computer vision techniques in the diagnosis of skin cancer is given covering also microscopic and macroscopic images. In the next section, a summary about the available dermatological databases and atlas is presented. Afterwards, over the third section, ones discuss the stages related with medical imaging applied in melanoma diagnosis: starting with image acquisition types used on, passing through the preprocessing inherent challenges, such as uneven illumination, color correction, contrast enhancement, hair removal or image restoration, giving a brief review about these challenges and also presenting self developed methodologies used to overcome problems like reflection or blur detection. In the fourth section, it is proposed a physical and information architectures suitable for a patient-oriented system of skin lesion analysis using smart devices. In the fifth section, a review about the existing smart device-adaptable dermoscopes, together with a discussion about their differences is given, in terms of color reproduction, image area and distortion, illumination, sharpness and differential structures visibility. This chapter ends with the conclusions about the topics discussed. 1.2 Dermatological Databases To be able to develop a reliable and robust system for melanoma detection it is crucial to have a complete set of images as diverse as possible and correctly annotated. The available dermatological databases and atlas are indicated in Table 1.1, where details like the total number of images or melanoma images, the score information of asymmetry, border, color and differential structure (ABCD) [13], and the type of images are specified. Although there already exist enough dermoscopic databases images with medical annotation, there is only one medical annotated database with images acquired via mobile device. The construction of a complete dataset with different image types (acquiered using dermoscopic, macro- 4Dermoscopy Image Analysis scopic or mobile imaging devices like mobile phones), conveniently annotated by experts for research and benchmarking purposes, would be of extreme importance in order to allow comparative studies in the near future. It would not only facilitate the development of computer-aided diagnosis systems but also be useful for the patient knowledge about this subject. This also confirmed by [12] who states that the absence of benchmark datasets for standardized algorithm evaluation is a barrier to a more dynamic development in this area. In fact, according to a recent study [14] about the impact of visual images on patient skin selfexamination (SSE) knowledge, attitudes and accuracy, images positively affect knowledge and self-efficacy related to SSE. Table 1.1: Dermatological Datasets (Type: D - Dermoscopic; M - Macroscopic; C - Clinical; MP - Mobile Phone; NA - Not available). Databases Total Images Melanoma Images ABCD score Type IPO Mobile [15] 90 NA Yes MP IPO [15] 217 12 Yes D / M Interactive Dermoscopy Atlas [16] 729 219 Yes D / M PH2[17] 200 40 Yes D Menzies Atlas [18] 320 NA NA D Dermnet Skin Disease Atlas [19] 23000 190 No M Danderm [20] 3000 49 No M / C MED M Heenen [21] 1207 51 No M / C Dermatology Atlas [22] 8084 80 No M / C Dermatlas net [23] 1000 32 No M / C Dermoscopy Atlas [24] NA 153 No D / M / C DermIS [25] NA 300 No D / M / C DermQuest [26] NA 312 No D / M / C 1.3 Dermatology Digital Imaging Along the years different imaging techniques were used for melanoma diagnosis, which is detailed next. In the following subsection, a review of existing techniques is presented and, in the last subsection, self developed methodologies to overcome some of the problems of dealing with these kind of images are also described. 1.3.1 Image Acquisition In the 1960s and 1970s the diagnosis of melanoma was simply based on symptoms, such as bleeding, itching and ulceration, and at the time of diagnosis the prognosis was poor. Later, the introduction of the asymmetry, From Dermoscopy to Mobile Teledermatology 5 border, color and diameter, so called ’ABCD’ rule by Friedman et. al [27] allowed the early detection of a high number of melanomas, and it has been adopted worldwide since then. This rule is based on simple clinical morphological features of melanoma and the inclusion of a fifth criterion E, for evolution concerning morphological changes of the lesion over time, brought improvement to the existent rule. As the clinical diagnosis based on the ABCD rule fail to recognize small melanomas and some benign melanocytic nevi may mimic melanoma from a clinical point of view, new imaging techniques were developed to overcome these problems. More specifically, in [13] Stolz et al. addresses the ABCD Dermoscopy rule that quantifies if the selected melanocytic lesion is benign, suspicious or malignant according to the score information of asymmetry, border, color and differential structures. A number of studies had shown that medium resolution microscopic views using skin surface microscopy (dermoscopy) provided a new level of clinical morphology linking clinical morphology and histopathology. Dermoscopy (epiluminescence microscopy, dermatoscopy, skin-surface, incident light microscopy) has been estabilished in the last three decades as the preferred imaging method for improving early detection of MM and for reducing unnecessary excision of benign naevi. It is a non-invasive, in vivo examination with a microscope that uses incident light and oil immersion to make subsurface structures of the skin accessible to visual examination. This method allows the observer to look not only onto but also into the superficial skin layers, permitting a more detailed inspection of pigmented skin lesions. More recently, new hand-held devices using polarized light have been introduced, which renders the epidermis translucent, turning unnecessary the use of oil for visualizing the subsurface structures [28]. Moreover, it is important to refer that some of these hand-held dermoscopes are already capable of adapting to smartphones, as is explored in section 5. According to [29], dermoscopy improves the diagnostic accuracy for melanoma in comparison with inspection by the unaided eye, however only for experienced examiners. In addition, the study [30] indicates that analysis either by a trained dermatologist or an artificial neural network-trained can improve the diagnostic accuracy of melanoma compared with that of an inexperienced clinician and that computational diagnosis might represent a useful tool for the screening of melanoma, particularly at centers not experienced in dermoscopy. Medium resolution clinical images of lesions (macros) are usually acquired using oblique modeling lighting and at close distances to best represent the view that a physician would see under a detailed skin inspection regime in an ideal clinic setting. These macroscopic views could be the best indicators of suspicious lesions [31]. Other possibility, less expensive and easier to spread, is to obtain images from mobile devices such as smartphones. This alternative allows both clinicians in general and patients to obtain several images of suspicious moles to be further analyzed by experienced examiners. The same procedure can be adapted using smartphone-adaptable dermoscopes, obtaining the same image quality than regular dermoscopes with the adding value of image storage and the associated benefits, like being possible to discuss the high-quality 6Dermoscopy Image Analysis image with specialists worldwide at short-term or analyzing the mole evolution at a long-term. Finally, clinical images consist of general imaging of the body with the intent to show the skin condition, the number of lesions, the amount of sun damage, and other clinical identifiers that are also important for the relative assessment of the overall risk of the patient. In [31] a comparative study between a personal device such as a smartphone and clinical photography in monitoring skin lesions is made. Although the quality of images produced using clinical photography is superior, personal devices technology still provides useful clinical information plus offering a relatively inexpensive alternative. When professional applications for skin imaging rely on high quality image acquisition devices, the resulting image quality is supposed to be optimal for skin cancer detection purposes. And, when compared to dermoscopy or smartphone-adaptable dermoscopes, these images may contain several additional artifacts which could have impact in terms of image quality. Due to that, new challenges appear regarding preprocessing of macroscopic images acquired with cameras of mobile devices. 1.3.2 Image Preprocessing Challenges Color spaces The Commission Internationale de L’ ´ Eclairage (CIE) has defined a system that classifies color according to the human visual system. The color spaces CIE RGB (Red-Green-Blue) and CIE XYZ (Y matches closely to luminance, while X and Z gives color information) were the first mathematically defined color spaces, where the first (CIE RGB) is a set of CIE color-matching curves based on many experiments with average observers and based on pure light sources at specific RGB wavelengths where the resulting spectral curves are called the CIE standard primaries, and the later (CIE XYZ) is based on and derived from the first, where XYZ are extrapolations of RGB created to avoid negative numbers and are called tristimulus values [32]. In an attempt to linearize the perceptibility of color differences, the CIE proposed two other color spaces, CIE L*a*b* and CIE L*u*v*, where: L* represents the lightness of the color; a* and b* represents the color differences in terms of redness-greenness and yellowness-blueness, respectively; u* and v* also represents differences in the chromaticity coordinates, but are not associated to color names as a* and b*. These color spaces are device independent, since they are based on the CIE system, however they suffer from being quite unintuitive [32, 33]. Since CIE L*u*v* and CIE L*a*b* are absolute color spaces, it defines color exactly and thus includes more color than other color spaces (even more than the human eye can see). The most known color space is the RGB which are three primary additive colors and are represented by a three-dimensional, Cartesian coordinate system [32–34]. This system is commonly used in computational applications since no transform is required to display information on common computer screens. RGB is a device dependent color space because the values depend on the specific sensitivity function of the imaging From Dermoscopy to Mobile Teledermatology 7 acquisition device. The main disadvantage of this color space in applications with natural images is a high correlation between its components. The chromaticity variables (rgb) are the normalized RGB color variables in order to reduce the dependence on changing in space lighting intensities [32,34]. The hue, saturation and lightness color space (HSL) or similar ones like HSI (intensity), HSV (value) or HCI (chroma) are based on linear transformations from RGB and, also, device dependent [32–35]. The advantage of these systems lies in the intuitive manner of specifying color making them a good choice in user interfaces and, more importantly, the small correlation between the three components [36]. The Luminance – Chrominance color spaces like YCbCr(Y is the luma component; Cband Cris the blue-difference and red-difference chroma components, respectively) separate RGB into luminance and chrominance information and are used in compression applications and digital video encoding [32, 34]. The advantage of converting an original RGB image into luminance-chrominance color space is that the components are pairwise uncorrelated. Furthermore, the chrominance channels contain much redundant information and can easily be subsampled without sacrificing any visual quality for the reconstructed image. In order to overcome sensitivity against various imaging conditions, Gever and Smeulders proposed c1c2c3, l1l2l3and m1m2m3color spaces [37]. c1c2c3was proposed to achieve independency of color illumination and discount the object’s geometry, l1l2l3was proposed to determine the direction of the triangular color plane in RGB space and m1m2m3achieved a constant color model considering a change in spectral power distribution of the illumination [33, 34, 38]. Nevertheless, it should be noted that the former color spaces were built assuming dichromatic reflectance and white illumination. Because of separation of luminance and chrominance components, the YCbCrcolor space is one of the most popular choices for skin detection. In [39], the authors use YCbCrto model skin and further segment human face in color from images, other examples that use this color space are [40–43]. Recently, in [44] a comparison of skin color segmentation results using the YCbCrand CIE L*a*b* color spaces, experimental results show that CIE L*a*b* performs better because it gives more information than the other color space model. Different color spaces are better for different applications, since some colors are perceptually linear or just more intuitive to use. Also some color spaces are tied to a specific imaging equipment while other are equally valid on whatever the device used. Taking into account studies referred, the resume Table 1.2 was built considering different color spaces and their sensitivity to color illumminant, illumination intensity, highligth and shadows. In the table “+” means invariance and “-” no variance to the criteria. Illumination The correction of uneven illumination is considered crucial to prevent segmentation errors in several skin lesion analysis methodologies. The authors in [45] realized that the uneven illumination corresponded to 8Dermoscopy Image Analysis Table 1.2: Sensitivity of different color spaces (adapted from [37]). Color Space Color of illumninant Illumination intensity Highlight Shadow RGB - - - - rgb - + - + H - + + + S - + - + YCbCr+ (only Y) - - - c1c2c3- + - + l1l2l3+ + + + m1m2m3+ + - + a low frequency spatial component of the image, while the information about the skin texture and the pigmented lesion were enclosed in the high spatial frequency component of the image. Thus, they proposed a correction of the illumination by simply removing the low frequent spatial component of the image. Although this method can be efficient for some images, it requires specific parameters that are not unique for different images and the authors do not detail how to obtain them automatically. New techniques to improve the processing of skin images acquired with standard imaging cameras were also proposed in [46], such as a data-driven shading attenuation stage to improve the robustness of the skin lesion segmentation. It starts by converting the input image from the original RGB color space to the HSV color space and retain V channel value; then a pixel set of the four corners of the image is extracted in order to estimate a quadratic function and this information is used to relight the image itself. This method is adequate to model and attenuate the global illumination variation; however, tends to have limited effect on local cast shadows and also tends to fail on surface shapes that are not locally smooth, since the quadratic function is not able to capture the local illumination variation. The approach of [47] follows the previous method and uses the entropy minimization technique, succeeding in removing or strongly attenuating shading and intensity falloffs. Homomorphic filtering (HF) [48] is a generalized technique for non linear image enhancement and normalizes the brightness of a dermoscopic image. A later improvement of the former technique to human perception is presented in [49], where HF is performed in both spatial and frequency domains, by processing the J plane of the JCh color space followed by a contrast adjustment in this color space. Wang et al. [39], by opposite, use a three step brightness adjustment procedure to minimize the vignetting effect (darkened image corners due to position dependent loss of brightness in the output of an optical system) in dermoscopic images. After defining a set of concentric circular regions, the brightness of the next circular region, starting at the image center, is adjusted so that the average intensity is equal as the center. From Dermoscopy to Mobile Teledermatology 9 Recently, a multistage illumination modeling algorithm [50] was proposed to correct illumination variation in dermatological skin lesion images. It first determines a nonparametric model of illumination using Monte Carlo sampling and a parametric model assuming a quadratic surface model is used to determine the final illumination estimate based on a subset of pixels from the first step. Finally, by using the final illuminate estimate, the reflectance component of the image is calculated and a new image is built which is corrected for illumination. Color correction Different illumination or different imaging devices will lead to distinct image colors of the same lesion and so can also compromise further steps. In [51], an algorithm for automatic color correction of digital skin images in teledermatology was proposed. For that, the widely known principle that skin color is one of the “basic colors” of human color perception was considered. A different method to assess skin tones and retrieve color information from uncalibrated images consists on imaging a skin region with a calibration target and extracting its color values to compute a color correction transform [52]. Another approach capable of addressing this problem is based on automatic color equalization technique [53] which consists of two main stages: chromatic/spatial adjustment and dynamic tone reproduction scaling. Iyatomi et al. [54] also described a color correction method for dermoscopy images based on HSV color model where a multiple linear regression model is built for each channel using low-level features extracted from a training image set. Then, through the use of these regression models, the method automatically adjusts the hue and saturation of a previously unseen image. In [55], the authors describe a two-stage color normalization scheme where color variations are removed and the contrast of the images are enhanced by combining Grayworld and MaxRGB normalization techniques. The authors [56] suggest selecting judicious colors from image database to design a customized pattern before applying usual color correction. Also, a comparative study is driven concluding that the approach ensures a stronger constancy of the color of interest and provides a more robust automatic classification of skin tissues. Contrast enhancement Histogram equalization is commonly used contrast adjustment by allowing for areas with lower contrast to gain a higher contrast by spreading out the most frequent intensity values. Generalizations of this method use multiple histograms to emphasize local contrast, rather than overall contrast. Examples of such methods include Adaptive Histogram Equalization (AHE) [57] and Contrast Limited Adaptive Histogram Equalization (CLAHE) [58]. Another method to perform histogram analysis in order to enhance contrast was 16 Dermoscopy Image Analysis Table 1.3: Summary of the features extracted for blur detection in [96]. Group Name Attributes Gradient Energy Image Gradient Sum, mean, std, max based Squared Gradient Sum, mean, std, max Tenengrad Sum, mean, std, max, var Laplacian Energy of Laplacian Sum, mean, std, max based Sum Modified Laplacian Sum, mean, std, max Diagonal Laplacian Sum, mean, std, max Variance of Laplacian Mean, std, max, var Laplacian and Gaussian Sum, mean, std, max Statistical Gray Level Variance Sum, mean, std, min, max, var based Norm. Gray L. Variance Normalized variance Histogram Entropy Sum (R, G, B, gray) Histogram Range Range (R, G, B, gray) DCT/ DCT Sum, mean, std, min, max DFT DFT Sum, mean, std, min, max Other Brenner’s Measure Sum, mean, std principles Image Curvature Sum, mean, std, min, max Spatial Freq. Measure Sum, mean, std, max Vollath’s autocorrelation Sum, mean, std, max Perceptual blur Count and mean (horizontal and vertical) described as a technique capable of improving the diagnostic accuracy [29]. In [8], it is presented the first study performing MT using cellular phones with in-built cameras. In this work, the authors used a close up clinical image and a dermoscopic image by applying the mobile phone on a pocket dermoscopy device to investigate the feasibility of teleconsultation using a mobile phone. The images were acquired with a Sony Ericcson K 750i with a built-in 2 megapixel camera with autofocus, macro mode and zoom and the dermoscopy device was a DermLite II Pro HR with a 25mm 10x lens. The images were reviewed by two teleconsultants and compared to face-to-face diagnosed obtaining a correct score of 89%and 91.5%for clinical and dermoscopic images, respectively. However, face-to-face consultation is considered superior to SFTD as a clinical assessment method due to its benefits in terms of lesion palpation, additional enquiry and examination [105,106]. From Dermoscopy to Mobile Teledermatology 17 Summarizing, there are strong evidences that skin cancer triage services should be integrated with a community of dermatological expertises, simultaneously ensuring that Teleconsultants (TC) must always feel able to invite patients to attend for face-to-face assessment whenever it could be necessary [108]. So far, the great majority of the proposed methodologies on the literature are based on dermoscopic image analysis, usually aiming for dermatology specialist’s usage as decision support systems. From a patient perspective, instead, one can identify different needs and implementation strategies. Patients with clinical or family history of skin cancer should consult regularly their dermatologists for physical skin examinations, for instance once a year. Between these appointments the patients are usually advised by their doctors to check for relevant changes in their skin moles, and to anticipate the consultation in case of detecting something suspicious. Unfortunately, the patients typically do not have enough dermatological expertise to perform this kind of risk assessment. Due to that, patient-oriented approaches are new paradigms for skin lesion analysis, trying not only to motivate and educate the patients, but also to empower them in terms of dermatological expertise, which can lead to a significant impact in the early diagnosis of skin moles malignancies. In this section, it is proposed a physical and information architecture suitable for a patient-oriented system of skin lesion analysis based on smart mobile devices with image cameras [109]. 1.4.1 A Patient-oriented system Physical architecture A physical architecture (Figure 1.3) for a patient-oriented system of skin lesion analysis formed by three main blocks: 1) Front-end Device (The user’s smart device): used to acquire or load an image of a skin lesion and send it to the server for analysis. The application that allows the communication between the smartphone and the server communicates is via the HTTP communication protocol. 2) Server: The server side consists on a RESTFul webservice implemented in Java using the Jersey library [110] and deployed on an Apache Tomcat 7.0 web server. The Image Processing Module (IPM) can be implemented in C++ using the OpenCv library [111], which will be executed as an external program when an image is received in the server. The IPM will receive as input the original skin lesion image and return a quantitative analysis of the image in addition to a visual output of the segmentation and features extraction steps. 3) Back-end Device: The device that will receive the outputs generated by the IPM analysis. From the user’s perspective, the results can be directly returned to one own smart device. Moreover, the IPM analysis information can also be provided to dermatology specialists through a web interface. 18 Dermoscopy Image Analysis Figure 1.3: Physical diagram architecture. Information architecture In the proposed patient-oriented solution, the information flows along three main groups: users, skin lesions and check-ups (see Figure 1.4). The application may have several users; and one user may have several skin lesions; one skin lesion has one or more check-ups. Each user has the following information: 1) name; 2) gender; and 3) age. Each skin lesion has the following information: 1) location on the body; and 2) list of check-ups. Each check-up has the following information: 1) skin lesion image; 2) check-up date; 3) four ABCD features scores (generated by the IPM); and 4) skin lesion size (in mm). Figure 1.4: Diagram of the information architecture proposed. 1.5 Smart Device-Adaptable Dermoscopes This section gives a review about the existing smart device-adaptable dermoscopes, together with a study regarding their differences in terms of color reproduction, image area and distortion, illumination, sharpness and differential structures visibility. Here, we used 2 different smartphones: the Samsung Galaxy S4 and the iPhone 5, which are currently top range devices of the operating systems Android and iOS, respectively. From Dermoscopy to Mobile Teledermatology 19 Table 1.4 presents the camera specifications of each smartphone used. By the time of our study, there were two different brands of dermoscopes with adaptability to smartphones on the market: the pocket Dermlite line [112] of 3Gen from USA and the Handyscope dermoscope, of FotoFinder’s from Germany [113]. The pocket Dermlite line allows standalone usability, meaning that it can work without a camera and the adaptation of the DL1, DL2 and DL3 dermoscopes; also to test this line, we selected two dermoscopes from different ranges: the Dermlite DL1 (bottom range) and the Dermlite DL3 (top range). Regarding the Fotofinder line, we selected the Handyscope for iPhone 5. The prices of the considered adaptable dermoscopes ranges between 400eand 800eat the time of this study, and we used a magnification of 10x in all dermoscopes. Table 1.5 shows the specifications of the three considered smart devices-adaptable dermoscopes. Table 1.4: Camera specifications of the two considered smartphones. Samsung Galaxy S4 iPhone 5 Camera 13 megapixels 8 megapixels Flash LED LED Aperture size F2.2 F2.4 Focal length 31mm - Sensor size 1/3.06” 1/3.2” Table 1.5: Specifications of the three considered smart devices-adaptable dermoscopes. Dermlite DL1 Dermlite DL3 Handyscope Number of - 4 - 21 (Cross-polarization) - 6 (Cross-polarization) LEDS - 7 (Non-polarization) - 6 (Non-polarization) Illumination - Cross polarized light - Cross-polarized light - Cross-polarized light Modes (sliding clip) - Non-polarized light - Non-polarized light - Non-polarized light (sliding spacer unit) Magnification 10x 10x Up to 20x Supported - iPhone 4/4S/5/5S/5C - iPhone 4/4S/5/5S/5C - iPhone 4/4S/5/5S Smart devices - iPad 3/4/Air/mini - iPad 3/4/Air/mini - iPod Touch - Samsung Galaxy S3/S4 - Samsung Galaxy S3/S4 Battery life 1 hour 5 hours 6-8 hours Standalone usability Yes Yes No We tested 5 different combinations of the considered smartphones and dermoscopes: 1) Samsung Galaxy S4 20 Dermoscopy Image Analysis with Dermlite DL1; 2) Samsung Galaxy S4 with Dermlite DL3; 3) iPhone 5 with Dermlite DL1;4) iPhone 5 with Dermlite DL3; and 5) iPhone 5 with Handyscope. Regarding dermoscopes comparison, the case reports and reviews available on the literature were pretty scarce for handheld dermoscopes [114, 115], and nonexistent if we only consider smartphone-adaptable dermoscopes. Therefore, we analyzed the 5 different combinations of smartphones and mobile-adaptable dermoscopes in terms of color reproduction, image area, image distortion, illumination, sharpness and differential structures visibility. 1.5.1 Color Reproduction To test the color reproducibility of the combinations of the considered smartphones and dermoscopes, we have chosen 12 different reference colors from the ColorChecker chart [116]: 6 grayscale colors, 3 primary colors and 3 secondary colors. The reference colors were printed in standard office paper, and all the images were acquired using cross-polarized light with 10x magnification. For comparison purposes, only a square of 1000 x 1000 pixels at the center of the acquired images was considered, since in dermoscopic images the best illumination conditions are usually achieved at the center of the image. The obtained results using all the combinations are depicted in Table 1.6. From the analysis of Table 1.6, one can observe that the color reproducibility of the Handyscope with the iPhone 5 seems to be the most accurate, principally for the primary and secondary color patches. The Dermlite DL3 had a similar performance in terms of the primary and secondary colors, however, the images acquired with the Dermlite DL3 for the grayscale colors have a significant bluish tint. This clear tendency towards the blue color was present on both smartphones, which might indicate that it could be related with intrinsic characteristics of the Dermlite DL3. Comparatively, the images acquired with the Dermlite DL1 look considerably washed out and with subdued colors. These types of artifacts are commonly present in overexposed images, so in the case of the Dermlite DL1 probably more light than the required was acquired by the camera. 1.5.2 Image Area and Distortion To compare the visible area and image distortion of the acquired images, we used millimeter paper and images were acquired using cross-polarized illumination. The width and length of the visible area was calculated by counting the lines of the millimeter paper grid (see Table 1.7). Moreover, to analyze the image distortion of the captured images, a 10 x 10 mm2square was centered in the image and the images were qualitatively compared (see Table 1.8). In terms of visible area, the Dermlite DL3 dermoscope presents the biggest area, followed by the Handyscope, and finally the Dermlite DL1. Comparing the dermoscopes of the Dermlite line on both smartphones, it is From Dermoscopy to Mobile Teledermatology 21 Table 1.6: Color Reproduction. Reference Samsung Galaxy S4 i Phone 5 Color DL1 DL3 DL1 DL3 Handyscope worth noting that the iPhone 5 guarantees a wider visible area. Regarding the image distortion, one can verify that all tested combinations produced Pincushion distortion 22 Dermoscopy Image Analysis Table 1.7: Visible area comparison. Samsung Galaxy S4 i Phone 5 DL1 DL3 DL1 DL3 Handyscope 17 x 13 mm224 x 15 mm218 x 18 mm224 x 18 mm221 x 18 mm2 Table 1.8: Image distortion comparison. Samsung Galaxy S4 i Phone 5 DL1 DL3 DL1 DL3 Handyscope on the acquired images, i.e. lines that did not pass through the center of the image were curved inwards, towards the center of the image. The minimal Pincushion distortion was obtained using iPhone 5 with Handyscope, followed by Dermlite DL1 with both smartphones. On the other hand, the images with most Pincushion distortion were acquired with Dermlite DL3 for both smartphones. 1.5.3 Illumination In order to compare possible differences in light spreading and brightness, images of the same skin mole were acquired using the 5 different considered combinations of smartphones and dermoscopes, Table 1.9. The Dermlite DL1 appears to have the worst performance in terms of light spreading, with a brighter vertical area in the center of the image (more visible when using iPhone 5). Moreover, comparatively with the other dermoscopes, the Dermlite DL1 led to the biggest shadow area near the black border of the dermoscopic image. Regarding the light spreading, the best performance was achieved by Dermlite DL3 and Handyscope using iPhone 5. Comparing the Dermlite line for both smartphones, the images acquired with iPhone 5 appeared to be brighter and with less shadow areas near the border of the dermoscopic images. However, color reproducibility seemed to be better for Samsung Galaxy S4, whilst being excessively whitish for Handyscope and Dermlite DL1 when using iPhone 5. From Dermoscopy to Mobile Teledermatology 23 Table 1.9: Light spreading comparison. Samsung Galaxy S4 i Phone 5 DL1 DL3 DL1 DL3 Handyscope 1.5.4 Sharpness and Differential Structures To compare the sharpness and visibility of the differential structures, images of 4 different skin lesions were acquired using the 5 considered combinations of smartphones and dermoscopes. For each lesion, 3 different types of images were acquired: a dermoscopic image with cross-polarized light, for each smartphonedermoscope combination (see Table 1.10); a non-dermoscopic image using only the built-in smartphone camera, with and without flash (see Table 1.11). Considering the dermoscopic images with cross-polarized light, the sharpest images seem to be obtained with Dermlite DL3 and iPhone 5, being simultaneously the images where the differential structures were most clearly visible. This combination also seemed to be the one that guarantees best color reproducibility and contrast. Handyscope and Dermlite DL3 with Samsung Galaxy S4 closely followed the image quality delivered by Dermlite DL3 in terms of sharpness and visibility of differential structures. The worst results were with Dermlite DL1, with images that comparatively looked considerably washed out and with a lower contrast. Moreover, the border of the skin lesions on Dermlite DL1 were less marked and the differential structures more difficult to see. Finally, to analyze the impact of the flash light in the mobile image acquisition process of skin lesions, images of the same skin moles were also acquired using both smartphones. As we can see on Table 1.11, the images acquired using the flash light are sharper and the inner structures of the skin lesions are more visible. The acquisition process without flash light was significantly more difficult, being inclusively not possible to obtain focused images even after several attempts, as it was the case for Lesions #3 and #4 using the 24 Dermoscopy Image Analysis Samsung Galaxy S4. However, when using the flash light, it should be taken into consideration that the likely appearance of highlight artifacts caused by the reflections of the skin, might considerably reduce the skin lesions visibility. Table 1.10: Dermoscopic skin lesion images acquired using cross-polarized light. Samsung Galaxy S4 i Phone 5 DL1 DL3 DL1 DL3 Handyscope Lesion #1 Lesion #2 Lesion #3 Lesion #4 1.6 Final Remarks Mobile Teledermatology has indeed several potential applicabilities for both doctors and patients, being an important tool for skin cancer prevention. Statistics say that skin cancer is the most common malignancy in Caucasian population and the indicators show that each year this number is alarmingly growing. Therefore the development of prevention measures is essential and MT can significantly add value in this case. Here, it was presented a survey about the new trends of MT, covering topics such as: the available dermatological databases that can be used for developing robust detection methods; the topics related with medical imaging, including image acquisition types, to preprocessing challenges of the acquired images and some solutions to overcome them; proposing a new MT patient-oriented design approach; and finalizing with a discussion about the existing smart devices-adaptable dermoscopes. Yet, being this a recent area of investigation, several directions may be fruitful to explore in order to advance in MT field: From Dermoscopy to Mobile Teledermatology 25 Table 1.11: Skin lesion images acquired using the smartphones’ built-in cameras. Samsung Galaxy S4 i Phone 5 With Flash No Flash With Flash No Flash Lesion #1 Lesion #2 Lesion #3N.A. Lesion #4N.A. •Preprocessing tasks : it is important to develop robust methodologies focused on the normalization of images acquired using different mobile devices under uncontrolled conditions, both in terms of illumination and color calibration. Moreover, the reliable image quality assessment should also be considered one of the key aspects for the successful deployment of these type of systems. •System design : the patient-oriented approach opens up new possibilities for an active role of the patient in managing one skin health status, as well as simultaneously improving the patient-doctor relationship. Thus, the design of these type of systems should be rethought and improved in order to meet both personal/experts requirements. •Smart devices-adaptable dermoscopes : The recent appearance of these devices on the market can bring significant advantages to dermatology specialists in terms of data acquisition, transmission and storage. Taking also into account the current global penetration of smart devices, if the price range of smart device-adaptable dermoscopes considerably lowers down in the following years, it is expectable that these devices start spreading among the general population. Thus, it is important to start investigating the advantages and limitations of these devices in terms of image quality, so that the image processing and analysis techniques proposed to date for standard dermoscopic images can be improved accordingly. 32 Dermoscopy Image Analysis [75] Z. She, P. J. Fish, and A. W. Duller, “Improved approaches to hair removal from skin image,” in Proceedings of SPIE, vol. 4322, p. 492, 2001. [76] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, “Image inpainting,” in Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p. 417–424, ACM Press/Addison-Wesley Publishing Co., 2000. [77] P. Wighton, T. K. Lee, and M. S. Atkins, “Dermascopic hair disocclusion using inpainting,” Medical Imaging 2008, Proceedings SPIE 6914, vol. 6914, p. 691427–6914278, 2008. [78] F.-Y. Xie, S.-Y. Qin, Z.-G. Jiang, and R.-S. Meng, “PDE-based unsupervised repair of hair-occluded information in dermoscopy images of melanoma,” Computerized Medical Imaging and Graphics, vol. 33, no. 4, p. 275–282, 2009. [79] N. H. Nguyen, T. K. Lee, and M. S. Atkins, “Segmentation of light and dark hair in dermoscopic images: a hybrid approach using a universal kernel,” in Proceedings of SPIE, vol. 7623, pp. 76234N–1, 2010. [80] G. Capdehourat, A. Corez, A. Bazzano, R. Alonso, and P. Mus´ e, “Toward a combined tool to assist dermatologists in melanoma detection from dermoscopic images of pigmented skin lesions,” Pattern Recognition Letters, vol. 32, no. 16, p. 2187–2196, 2011. [81] A. Criminisi, P. Perez, and K. Toyama, “Object removal by exemplar-based inpainting,” in Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. II–721, IEEE, 2003. [82] M. Fiorese, E. Peserico, and A. Silletti, “VirtualShave: automated hair removal from digital dermatoscopic images,” in 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society EMBC, p. 5145–5148, IEEE, 2011. [83] K. Kiani and A. R. Sharafat, “E-shaver: An improved DullRazor for digitally removing dark and light-colored hairs in dermoscopic images,” Computers in Biology and Medicine, vol. 41, no. 3, p. 139–145, 2011. [84] R. Nathan, “Digital video-data handling,” Technical Report No. 32-877, Jet Propulsion Laboratory, Pasadena, 1966. [85] B. R. Hunt, “The application of constrained least squares estimation to image restoration by digital computer,” IEEE Transactions on Computers, vol. 100, no. 9, p. 805–812, 1973. [86] B. Caputo, V. Panichelli, and G. E. Gigante, “Toward a quantitative analysis of skin lesion images,” Studies in Health Technology and Informatics, p. 509–513, 2002. [87] M. K. A. Mahmoud and A. Al-Jumaily, “Segmentation of skin cancer images based on gradient vector flow (GVF) snake,” in 2011 International Conference on Mechatronics and Automation (ICMA), p. 216–220, IEEE, 2011. [88] M. Jiang and G. Wang, “Development of blind image deconvolution and its applications,” Journal of X-ray Science and Technology, vol. 11, no. 1, p. 13–19, 2003. From Dermoscopy to Mobile Teledermatology 33 [89] P. Campisi and K. Egiazarian, Blind image deconvolution: theory and applications. CRC press, 2007. [90] P. A. Patil and R. B. Wagh, “Review of blind image restoration methods,” World Journal of Science and Technology, vol. 2, no. 3, p. 168–170, 2012. [91] R. Nakagaki and A. K. Katsaggelos, “A VQ-based blind image restoration algorithm,” IEEE Transactions on Image Processing, vol. 12, no. 9, p. 1044–1053, 2003. [92] A. C. Likas and N. P. Galatsanos, “A variational approach for bayesian blind image deconvolution,” IEEE Transactions on Signal Processing, vol. 52, no. 8, p. 2222–2233, 2004. [93] S. D. Babacan, R. Molina, and A. K. Katsaggelos, “Variational bayesian blind deconvolution using a total variation prior,” IEEE Transactions on Image Processing, vol. 18, no. 1, p. 12–26, 2009. [94] B. Amizic, R. Molina, and A. K. Katsaggelos, “Sparse bayesian blind image deconvolution with parameter estimation,” EURASIP Journal on Image and Video Processing, vol. 2012, no. 1, p. 1–15, 2012. [95] M. J. M. Vasconcelos and L. Rosado, “No-reference blur assessment of dermatological images acquired via mobile devices,” in Accepted in International Conference on Image and Signal Processing 2014, 2014. [96] M. J. M. Vasconcelos and L. Rosado, “Automatic reflection detection algorithm of dermatological images acquired via mobile devices,” in Accepted in 18th Annual Conference in Medical Image Understanding and Analysis 2014, 2014. [97] H.-H. Tsai, Y.-P. Pong, C.-C. Liang, P.-Y. Lin, and C.-H. Hsieh, “Teleconsultation by using the mobile camera phone for remote management of the extremity wound: a pilot study,” Annals of plastic surgery, vol. 53, no. 6, p. 584–587, 2004. [98] C.-H. Hsieh, S.-F. Jeng, C.-Y. Chen, J.-W. Yin, J. C.-S. Yang, H.-H. Tsai, and M.-C. Yeh, “Teleconsultation with mobile camera-phone in remote evaluation of replantation potential,” The Journal of Trauma and Acute Care Surgery, vol. 58, no. 6, p. 1208–1212, 2005. [99] R. P. Braun, J. L. Vecchietti, L. Thomas, C. Prins, L. E. French, A. J. Gewirtzman, J.-H. Saurat, and D. Salomon, “Telemedical wound care using a new generation of mobile telephones: a feasibility study,” Archives of dermatology, vol. 141, no. 2, p. 254, 2005. [100] P. Chung, T. Yu, and N. Scheinfeld, “Using cellphones for teledermatology, a preliminary study.,” Dermatology online journal, vol. 13, no. 3, p. 2, 2007. [101] C. May, L. Giles, and G. Gupta, “Prospective observational comparative study assessing the role of store and forward teledermatology triage in skin cancer,” Clinical and experimental dermatology, vol. 33, no. 6, p. 736–739, 2008. [102] C. Massone, A. Di Stefani, and H. P. Soyer, “Dermoscopy for skin cancer detection,” Current opinion in oncology, vol. 17, no. 2, p. 147–153, 2005. [103] C. Massone, A. M. G. Brunasso, T. M. Campbell, and H. P. Soyer, “Mobile teledermoscopy—melanoma diagnosis by one click?,” in Seminars in cutaneous medicine and surgery, vol. 28, p. 203–205, Elsevier, 2009. 34 Dermoscopy Image Analysis [104] J. D. Whited, “Economic analysis of telemedicine and the teledermatology paradigm,” Telemedicine and eHealth, vol. 16, no. 2, p. 223–228, 2010. [105] E. M. Warshaw, Y. J. Hillman, N. L. Greer, E. M. Hagel, R. MacDonald, I. R. Rutks, and T. J. Wilt, “Teledermatology for diagnosis and management of skin conditions: a systematic review,” Journal of the American Academy of Dermatology, vol. 64, no. 4, p. 759–772, 2011. [106] C. A. Morton, F. Downie, S. Auld, B. Smith, M. Van Der Pol, P. Baughan, J. Wells, and R. Wootton, “Community photo-triage for skin cancer referrals: an aid to service delivery,” Clinical and experimental dermatology, vol. 36, no. 3, p. 248–254, 2011. [107] C. M. Grin, A. W. Kopf, B. Welkovich, R. S. Bart, and M. J. Levenstein, “Accuracy in the clinical diagnosis of malignant melanoma,” Archives of dermatology, vol. 126, no. 6, p. 763, 1990. [108] M. Gray and A. Bowling, “Considerations for a successful teledermatology application,” Health Informatics New Zealand, 2010. [109] L. Rosado, M. J. Vasconcelos, and M. Ferreira, “A mobile-based prototype for skin lesion analysis: Towards a patient-oriented design approach,” International Journal of Online Engineering (iJOE), vol. 9, pp. pp. 27–29, Dec. 2013. [110] Jersey, “The open source JAX-RS reference implementation for building RESTful web services..” URL: https://jersey.java.net/, 2014. [111] G. Bradski and A. Kaehler, Learning OpenCV: Computer Vision with the OpenCV Library. ”O’Reilly Media, Inc.”, Sept. 2008. [112] 3Gen, “Dermlite.” URL: http://dermlite.com/collections/pocket-dermoscopy-devices, 2014. [113] Fotofinder, “Handyscope.” URL: http://www.handyscope.net/, 2014. [114] A. Blum and S. Jaworski, “Clear differences in hand-held dermoscopes,” Journal der Deutschen Dermatologischen Gesellschaft JDDG, vol. 4, pp. 1054–1057, Dec. 2006. PMID: 17176414. [115] A. Blum, S. Jaworski, H. L¨ udtke, and U. Ellwanger, “Systematic comparison of five hand-held dermoscopes reveals clear differences,” Aktuelle Dermatologie, vol. 34, no. 01/02, pp. 9–15, 2008. [116] “ColorChecker.” URL: http://en.wikipedia.org/w/index.php?title=ColorChecker&oldid=565716177, Feb. 2014.