Framework for Planing the Aesthetic Result After Breast Surgery

FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO Framework for Planing the Aesthetic Result after Breast Surgery Sílvia da Conceição Neto Bessa Integrated Master in Bioengineering Supervisor: Hélder Filipe Pinto de Oliveira, PhD Co-supervisor: Eduardo José Marques Pereira, Eng. April 15, 2016 c Sílvia da Conceição Neto Bessa, 2016 Framework for Planing the Aesthetic Result after Breast Surgery Sílvia da Conceição Neto Bessa Integrated Master in Bioengineering April 15, 2016 Resumo Deformações da mama, causadas por tratamentos contra o cancro da mama, podem afetar a aparência física dos pacientes, levando a problemas emocionais e psicológicos significativos. Esta situação é agravada, em particular, no caso do cancro de mama feminino, considerando quer as altas taxas de incidência de cancro de mama em mulheres, em todo o mundo, quer a importância do tamanho e forma da mama na imagem do corpo feminino e sentimento de femininidade. Quando as mulheres são diagnosticadas com cancro de mama, é provável que a cirurgia seja parte do seu tratamento. Felizmente, enquanto algumas décadas atrás a mastectomia era o procedimento padrão, hoje em dia técnicas menos invasivas, como o tratamento conservador do cancro da mama são possíveis, minimizando as mudanças físicas da mama. No entanto, embora as técnicas mais recentes visem essencialmente a obtenção de um melhor resultado estético, práticas de trabalho heterogêneas têm contribuído para resultados estético diferentes. Como consequência, o planeamento cirúrgico está a ganhar importância na cirurgia do cancro de mama. A capacidade de visualizar os potenciais resultados da cirurgia, e tomar decisões sobre opções cirúrgicas é muito importante para pacientes e cirurgiões. Além disso, estudos mostram que quando as mulheres são envolvidos nas decisões relativas ao tratamento, são mais propensas a aceitar os resultados. Atualmente, várias ferramentas de planeamento de cirurgia da mama estão disponíveis, mas modeladores da mama para a cirurgia de mama que não seja o aumento da mama são menos comuns, ou não são específicas para cada paciente. A ideia deste trabalho é criar uma ferramenta para planear a cirurgia de mama baseada na imagem tri-dimensional do corpo do paciente, apropriada quer para a educação do paciente, quer para planeamento cirúrgico. Assim, o ajustamento dos pontos que consitituem a nuvem de pontos da mama, a um modelo paramêtrico adequado para o planeamento cirúrgico é investigado nesta dissertação. O objetivo é desenvolver modelos deformáveis para prever deformações da mama resultantes da cirurgia de cancro da mama, utilizando parâmetros ajustáveis facilmente compreendidos pelos utilizadores. Os resultados são promissores, mostrando que os modelos deformáveis podem ser aprendidos a partir de dados de exemplos, com erros baixos, utilizando regressões baseadas em Redes Neuronais. Três deformações mamárias comuns foram modeladas com sucesso, usando modelos de regressão que permitem ajustes por tipo e grau de deformação, e as adaptações necessárias para descrever outros tipos de deformações são exaustivamente discutidas. Os resultados preliminares sobre o uso de modelos de Forma Livre Deformação (Free Form Deformation) combinados com regressões baseadas em Redes Neurais para modelar deformidades da mama, sem depender de qualquer equação física conhecida, também são apresentados. Estes evidenciam que modelos de regressão baseados em Redes Neurais, se forem devidamente treinados, com dados de exemplos adequados, podem produzir modelos deformáveis estatísticos, utilizando um número reduzido de pontos de controlo. i ii Abstract Beast deformities caused by breast cancer treatments can impact the physical appearance of patients, leading to significant emotional and psychological problems. This is particular aggravated in the the case of female breast cancer, considering both, the high incidence rates of breast cancer in women, worldwide, and the importance of breast size and shape on the female body image and sense of femininity. When women are diagnosed with breast cancer, surgery is likely to be part of their treatment. Fortunately, while a few decades ago mastectomy was the standard procedure, nowadays less invasive techniques such as breast cancer conserving surgery are possible, minimizing the physical changes of the breast. Nonetheless, although the newer techniques aim essentially at attaining a better cosmetic outcome, heterogeneous working practices have contributed to different aesthetic results. As a consequence, surgical planning is gaining importance in breast cancer surgery. The ability to visualise the potential outcomes of the surgery and make decisions on their surgical options is very important, for both patients and surgeons. Moreover, studies show that when women are involved in treatment decisions, they are more likely to accept the resulting outcomes. Several breast surgery planing tools are currently available, but breast modelers for breast surgery other than breast augmentation are less common, or fail to be patient-specific. The idea of this work is to create a tool for three-dimensional breast surgery planing based on patient’s body image, appropriate either for individual patient education, or surgical planning. Therefore, the fitting of a tree-dimensional point cloud of the breast to a parametric model suitable for surgery planning is investigated. The goal is to develop deformable models to predict common breast deformities resulting from breast cancer surgery, using adjustable parameters easily understood by the users. Results are promising, showing that deformable models can be learnt from exemplar data using Neural Networks regressions with low errors. Three common breast deformations were successfully modelled using regression models that allow adjustment by type and degree of deformation, and the adaptions necessary to describe other types of deformations are thoroughly discussed. Preliminary results on the use of Free Form Deformation models of breast deformities, combined with Neural Networks regressions to model breast deformities without depending on any known physical equation, are also shown. These evidentiate that a careful training of Neural Networks regression models with proper exemplar data can produce statistical deformable models, while using a reduced number of control points. Keywords: Breast Cancer; Breast Deformations; Surgical Planning; 3D Modelling; Parametric Models; Regression Models; iii iv Agradecimentos As jornadas difíceis são as mais satisfatórias quando terminadas com sucesso. Tornar-me engenheira foi, certamente, uma delas! Engenharia está-me no coração, mas o curso de Bioengenharia, em particular, fez-me enfrentar situações que só foram possíveis de suportar e ultrapassar com o apoio incansável e incondicional dos que me são próximos. Embora o nome que aparece na dissertação seja o meu, esta não teria sido possível sem apoio daqueles a quem aqui agradeço: Hélder, Eduardo, meus orientadores, obrigada por me proporem percorrer esta jornada convosco, por serem incansáveis a tentarem perceber o que por vezes só estava claro na minha cabeça. Obrigada por não me darem desproporcionalmente na cabeça, mesmo quando o podiam fazer, mas acima de tudo, obrigada por terem sido uns companheiros de viagem, uns amigos que tornaram a repetição deste trilho, o mais divertido e interessante que alguém poderia fazer. Hooshiar, obrigada pela partilha dos teus conhecimentos na área, por me ajudares a perceber detalhes importantes de alguns algoritmos, e por partilhares, sem hesitação, algumas das tuas rotinas comigo. Kelwin, obrigada por te teres juntado a algumas das nossas discussões, e pelas tuas sugestões, algumas delas implementadas nesta dissertação. Ao professor Jaime Cardoso, por me ter ensinado tanto durante o tempo em que trabalhámos juntos, e por me ter integrado, pela primeira vez, no grupo VCMI. Aos membros do VCMI, obrigada por me receberem sempre bem, por todo o carinho, e pelas palavras de incentivo. Célia, minha querida, obrigada por todo o apoio, principalmente aquele que mais me ninguém deu no ano de caloira. Foste sempre um porto de abrigo, e sempre levaste com as minhas frustrações devolvendo-me um sorriso e uma palavra de apoio. Sem ti, definitivamente, não teria acabado Bioengenharia. Rute, Rita, Levindo, Rui, Nuno Sousa, ao resto d’Os Manjericos, obrigada por me mostrarem que pudemos ser esquesitóides juntos, por serem a família que eu escolhi. A faculdade teria sido muito mais chata e aborrecida sem vos ter por perto. O Quartel General está sempre aberto para vós. A nossa amizade é, de longe, o melhor que a FEUP me deu. Pai, meu heroí, obrigada pelo sorriso do teu olhar em todos os momentos, mesmo aqueles em que podias não perceber o que se estava a passar, ou podias estar desiludido. Obrigada por acordares todos os dias com a preocupação de garantir que nunca me falta nada, e por me ensinares que persistência e cabeça fria são, muitas vezes, a única solução. Mãe, minha querida mãe, obrigada por me ensinares a ser uma guerreira, a não virar as costas à luta, e por me teres dado o apoio fundamental para enfrentar aqueles que deliberadamente foram um entrave aos meus sucessos. Salomé, minha princesa, ma sistra, obrigada por compreenderes que o tempo é coisa limitada quando há uma dissertação para fazer. Obrigada por me dares perspetivas alternativas, pelos abraços silenciosos que muitas vezes dizem tudo o que eu preciso, e por seres tão especial para mim. Nuno, my soon to be husband, meu mais que tudo que me faz acreditar que tenho poderes transcedentes e que consigo levar tudo à frente, obrigada! Obrigada por escolheres e assemblares a minha máquina de trabalho e por confirmares os meus raciocínios e teorias a horas impensáveis. v LIST OF FIGURES 4.12 Diagram of a NN with activation, bias and weights nomenclature examples [35]. . 42 4.13 Block diagram of the proposed methodology for statistical models of breast deformations........................................ 47 4.14 Intermediate results of fitting FFD to 3D raw data of breast - raw data in skin color, models in red and CP box in green. . . . . . . . . . . . . . . . . . . . . . . . . . 49 5.1 Residual analysis of ptosis parameters predictions on datasets containing variable breast sizes of the same generic breast shape. . . . . . . . . . . . . . . . . . . . 57 5.2 Examples for the best regression model predicting ptosis parameters b0-NN regression model - original breasts (black) and model breast (skin color) superimposed. The best, average and worst prediction results are shown in each column, respectively...................................... 59 5.3 Examples for the best regression model predicting ptosis parameters b1-NN regression - original breasts (black) and model breast (skin color) superimposed. The best, average and worst prediction results are shown in each column, respectively. ........................................ 60 5.4 Examples for the best regression model predicting ptosis parameters b0>b1NN regression - original breasts (black) and model breast (skin color) superimposed. The best, average and worst prediction results are shown in each column, respectively...................................... 61 5.5 Examples for the best regression model predicting ptosis parameters b1≥b0NN regression - original breasts (black) and model breast (skin color) superimposed. The best, average and worst prediction results are shown in each column, respectively...................................... 62 5.6 Residual analysis of ptosis parameters predictions on datasets containing variable breastshapesandsizes................................ 64 5.7 Examples for ptosis parameters b0>b1predictions on datasets containing variable breast shapes and sizes. - original breasts (black) and model breast (skin color) superimposed. The best, average and worst prediction results are shown in eachcolumn,respectively. ............................. 64 5.8 Examples for ptosis parameters b1≥b0predictions on datasets containing variable breast shapes and sizes. - original breasts (black) and model breast (skin color) superimposed. The best, average and worst prediction results are shown in eachcolumn,respectively. ............................. 65 5.9 Residual analysis of turn parameters predictions on datasets containing variable breastshapesandsizes................................ 66 5.10 Examples for turn parameters c0>c1predictions on datasets containing variable breast shapes and sizes - original breasts (black) and model breast (skin color) superimposed. The best, average and worst prediction results are shown in each column,respectively. ................................ 67 5.11 Examples for turn parameters c1>c0predictions on datasets containing variable breast shapes and sizes - original breasts (black) and model breast (skin color) superimposed. The best, average and worst prediction results are shown in each column,respectively. ................................ 68 5.12 Distribution of the extent of degree of 8 top shape deformations caused by varying s0and s1, and with t0=t1=0 - with blue and red representing train and test breasts, respectively, and dashed lines limit the margin the examples included in thedatasets...................................... 69 xii LIST OF FIGURES 5.13 Effect of increasing the absolute value of top shape slope parameters, on the top breast profile: deformations obtained with s− 0>s+ 1, and t0=t1=0. Shows the points that area on which the points are more affected by the deformation. . . . . 70 5.14 Residual analysis of top shape parameters predictions on datasets containing variable breast shapes and sizes. Two conditions are shown, with different values and signs of s0and s1, and fixed values of t0=t1=0. Inequations compare the absolute value of the parameters, regardless of their signs. . . . . . . . . . . . . . 71 5.15 Examples for top parameters parameters c1>c0predictions on datasets containing variable breast shapes and sizes. Two conditions are shown, with different values and signs of s0and s1, and fixed values of t0=t1=0. Inequations compare the absolute value of the parameters, regardless their signs - original breasts (black) and model breast (skin color) superimposed. The best, average and worst prediction results are shown in each column, respectively. . . . . . . . . . . . . . 71 5.16 b0>b1....................................... 74 5.17 Residual analysis plots of the best statistical model - NN regression with an hiddenlayer of 125 nodes, and FFD models with 125 CP. . . . . . . . . . . . . . . . . . 74 5.18 Examples of ptosis deformation predictions using statistical models and FFD models with 125 CPoriginal breasts (black) and model breast (skin color) superimposed......................................... 75 xiii LIST OF FIGURES xiv List of Tables 2.1 Breast cancer treatments according to type and stage. . . . . . . . . . . . . . . . 9 4.1 Deformation parameters used to create the synthetic breasts composing database Vand S(parameters’ nomenclature as used in [16])................. 30 4.2 Minimum and maximum values of the range of deformations parameters used to create examples of ptosis, turn and top shape deformations. . . . . . . . . . . . . 35 4.3 Characterization of the feature sets compared in the ptosis study: DD stands for degree of deformation, capital letters refer to vectors and lowercase are associated tosinglevalues.................................... 43 5.1 Regression models results for two different conditions of ptosis (b0and b1) - performances with 8 degrees of deformation and optimized number of PCA features. 53 5.2 Regression models results for two different conditions of ptosis (b0and b1) - performances with 8 degrees of deformation and without PCA. ........... 54 5.3 NN regression models results with 8 degrees of deformation for multiple output parametersinptosis. ................................ 56 5.4 Indirect performance metrics for the best regression models predicting ptosis parameters. All models are applied without PCA, using feature set FS5 with 8 degrees of deformation. Relative distances are show in percentage. . . . . . . . . 58 5.5 Indirect performance metrics of NN regression models predicting ptosis parameters in datasets derived from database S. ................... 63 5.6 Indirect performance metrics of NN regression models predicting turn parameters in datasets derived from database S. ..................... 66 5.7 Indirect performance metrics of NN regression models predicting top shape parameters in datasets derived from database S. Inequations compare the absolute value of the parameters, regardless their signs. . . . . . . . . . . . . . . . . . . . 70 5.8 Indirect performance metrics ptosis deformation predictions using statistical models and FFD models with 125 and 216 CP. . . . . . . . . . . . . . . . . . . . 73 A.1 Regression models results for two different conditions of ptosis (b0and b1) - performances with 8 degrees of deformation and optimized number of PCA features. 79 A.2 Regression models results for two different conditions of ptosis (b0and b1) - performances with 8 degrees of deformation and without PCA. ........... 80 A.3 NN regression models results with 4 degrees of deformation for multiple output parametersinptosis. ................................ 80 xv LIST OF TABLES xvi Abbreviations and Symbols 2D 2-Dimensional 3D 3-Dimensional DCIS Ductal Carcinoma in Situ IDC Invasive Ductal Carcinoma ILC Invasive Lobular Carcinoma BCS Breast Conserving Surgery QoL Quality-of-Life MRI Magnetic Resonance Image CT Computed Tomography PCA Principal Component Analysis SVD Singular Value Decomposition RBF Radial Basis Functions FFD Free Form Deformation FEM Finite Element Model LR Linear Regression SVM Support Vector Machine NN Neural Networks FS Feature Set RMSE Relative Mean Squared Error RMPE Relative Mean Percentage Error CP Control Points xvii Chapter 1 Introduction In a world where perception of body image takes an important role in the self-esteem of most women, the high incidence rates of breast cancer have been jeopardizing the sense of femininity and quality-of-life (QoL) of women worldwide. After being diagnosed with breast cancer, women not only face the fear of death, but also the fear of breast disfigurement, specially when breast cancer surgery is still the primary treatment for breast cancer. With the improvements in surgical procedures and oncological treatments, the outcome results have become less dramatic. Treatments have evolved from radical mastectomay, where the whole breast has to be removed, to more sparing procedures, such as Breast Cancer Conserving Treatment where only the tumour and a soft margin of health tissue around it are removed, minimizing the breast shape changes. These strategies reduce the physical impact of cancer-related breast treatment, but the multitude of surgical options allied with heterogeneous practices still contribute to different aesthetic results. While a few decades ago the primary goal of breast cancer treatments was to eliminate cancer, with newer techniques, the aesthetic results now play a special role in the treatment decision process. This is particular important given that approximately 90% of breast cancers are curable if detected in an early stage, meaning that an increasing number of women have to live with the consequences of treatments for many years. The involvement of women in the treatment decision process has been proven benefit to accept the resulting outcomes, highlighting the necessity of creating tools that predict the outcomes of each possible option, providing patients with visual clues of the expected results for more conscientious decisions. 1.1 Motivation Taking into account that the acceptance of the outcomes of breast cancer treatments are improved when women are involved in the decisions, it is interesting to create a breast surgery planning tool, on which data from the patient’s breast is used to model the possible breast deformities, caused by different breast cancer treatment options. A tool with this specifications can be used as a communication and planing tool for both surgeons and patients. Surgeons will be able to 1 Introduction educate patients better, patients will be able to communicate their expectations in a clearer way, and surgical decisions will take in consideration both clinical limitations and patients desires. The use of surgery planning tools is already well adopted in plastic surgery of different parts of the body, and breast surgery is no exception, with several breast augmentation planning tools at disposal. These tools range from simple volumetric models applied to female generic torsos, to patient-specific high fidelity three-dimensional (3D) models of the patient’s own breast acquired with proprietary systems. However, the existence of breast modelers for other breast surgeries than breast augmentation is limited. The few examples, though, usually fail to be patient specific, rely on adjustable parameters that are not shape related or well understandable by the common user, or even depend on technically complex acquisitions of 3D images involving the positioning of landmarks to map the acquired point cloud to 3D models or template meshes. The use of patient specific models is of major importance, taking in consideration that women presented with outcome results modelled in generic torsos find it hard to project themselves in the models, which hardens their choices and jeopardizes their expectations. 1.2 Objectives In the pursuit of circumventing these problems, this work is focused on developing 3D deformable models of the breast with adjustable parameters that are easily understand by the user. The ultimate goal is to create a 3D planning tool to predict the aesthetic results after breast surgery, that is patient specific and support well informed decisions. In the process of creating such a tool, the 3D raw point cloud of the patient’s breast has to be fitted with parametric model, that can be deformed by adjusting simple and breast related parameters to obtain the desired shape of breasts. A smaller goal is to develop models that can learn breast deformations functions from exemplar data of real cased of deformations caused by breast cancer treatments. 1.3 Contributions The main contributions of this dissertation are listed bellow. In this dissertation: •The fitting of 3D parametric models to three common breast deformations was successfully accomplished, using regression models to predict the inputs of the physical equations describing the deformations; •A simple parameter model comprehensible for both surgeons and patients is proposed and used to adjust the breast to the desired shape; •A statistical deformable model of breast ptosis was tested, which is able to model the deformity with a reduced number of parameters, and learn its deformation function from exemplar data. 2 Introduction 1.4 Structure This dissertation is organized in six major chapters. In chapter 2, the fundamentals of breast cancer types, stages and treatments, as well as the psychological impact of breast cancer surgery in women are explained. In chapter 3, on overview of 3D modelling methodologies is presented. The chapter starts by listing breast surgery planning methodologies based on 3D parametric models, stating their limitations. Then, the mathematical formulations of parametric, deformable and statistical models are detailed, supported by examples of their applications. Chapter 4summarizes the proposed methodology to develop parametric models of the breast deformations, and explains the approach followed to derive statistical models of deformations from exemplar data. The Chapter starts by giving an insight on the regression models that will be used, followed by a section describing the used databases and the proposed approaches. The results of the proposed methodologies are presented in chapter 5, including their discussion. The first Section shows the results of the parametric models of breast deformations, while the second discusses the results of statistical models. In the end, some final considerations are summarized. Finally, conclusions and some insight on future work and research are provided in Chapter 6. 3 Breast Cancer radical mastectomy, simple (or total) mastectomy, skin-sparing mastectomy, ordered from the less to more tissue sparing procedure8 10. •Breast Conserving Surgery (BCS) (also known as lumpectomy,partial mastectomy - or wide excision) in which only the part of the breast containing the cancer is removed. A BCS usually removes the least amount of breast tissue possible. The surgeon removes the cancer and a small portion or margin of the surrounding tissue, but not the breast itself. How much of the breast is removed depends on the size and location of the tumor and other factors. Sometimes a larger portion of the breast — perhaps a whole segment or quadrant of tissue — has to be removed in order to completely eliminate the cancer [1]11. (a) Breast Conserving Surgery (b) Mastectomy Figure 2.4: Breast cancer surgery11. Lumpectomy is the least invasive breast cancer surgery, but it can still be very effective, without the need of further surgery: for most women with stage 1 or 2 breast cancer, breast-conserving surgery plus radiation therapy is as effective as mastectomy. Survival rates of women treated with these two approaches are the same [19]. Some women still fear that a less extensive surgery such as BCS may rise the risk of cancer coming back, facing mastectomy instead. In alternative, an increasing number of women are option for BCS when it is presented by their doctors as a reasonable option. In fact, studies have shown that when BCS can be done, mastectomy does not provide any better chance of cancer survival [1]. In spite of being a fair alternative in early stage cancers, women having BCS should be informed how the shape and appearance of the intervened breast can change and know the options. There is a wide range of defects which can result from BCS. These are affected by variables such the orientation of the skin incision, the pre-operative breast size, the percentage of breast parenchyma resected, the location of that resection, the intensity and method of delivery of radiotherapy and a patient’s response to radiotherapy. 12 The larger the portion of breast tissue removed, the more likely is the shape of the breast to change afterwards. In fact, a change in the 10http://www.webmd.com/breast-cancer/mastectomy?page=2 11https://ww5.komen.org/BreastCancer/Surgery.html 12http://www.beautifulabc.com/en/breast-conserving-surgery-general 10 Breast Cancer breast shape is always to be expected, as might be the formation of a hard scar tissue in the surgical site13. Therefore, surgery options might be discussed between surgeons and patients so that the final results pleases the intervened woman in the first place. Nevertheless, breast reconstruction can still be considered, either during the surgery or later on to improve the aesthetical results [1]. 2.3 Psychosocial impact of breast cancer surgery Breast size and shape are a significant part of female body image and sense of femininity. Breast deformities can occur due to congenital and traumatic causes, natural changes associated with aging and, most common, arise from cancer treatment [16]. After confirmation of the diagnosis, the woman feels that her female identity is being questioned because the breast is a symbol of body beauty, fertility, and health in all the stages of a woman’s life [25]. The majority of evidence shows that women with breast cancer experience a range of negative emotions, including depression and anxiety, concerns about disease recurrence, as well as feelings of sexual unattractiveness and alteration of femininity [2]. Despite efforts have been made to preserve breasts, treatment for breast cancer frequently results in marked changes to the physical appearance of patients that can result in significant emotional and psychosocial morbidity. The surgical treatment is required in almost all cases and provokes changes in the self-concept and body image. The distortion in the body image occurs mainly in women undergoing mastectomy and begins with an aversion toward themselves, manifested, for example, as difficulty in looking in the mirror and in resuming their sex lives [25]. In fact, patient self-reported body image scale has significant impact on their QoL outcomes. Breastspecific concerns, such as altered sense of femininity, feelings of decreased attractiveness and changes in body image and sexuality can affect general QoL [29]. Most studies have shown that body image can be impaired after breast cancer surgery, with women who performed a mastectomy having more concerns about their body image than women receiving BCS [2]. Furthermore, is has been shown that patients’ body image perception might be distinct from that of a panel of experts and software measurements, pinpointing that women view their own breast differently from observers. This emphasizes the need to educate the patient during surgery planing and increase its involvement in the decisions. A review from 2015 [46] on the patients involvement in the decision making has come to interesting conclusions: •Active participation is more common on breast cancer than in other malignancies. •The patient’s husband has a significant role in the decision making. •The risk of psychological comorbidity is considerably reduced if the patient is highly involved and takes responsibility for the decision. 13http://www.breastcancer.org/treatment/surgery 11 Breast Cancer •Older patients put greater emphasis on daily functioning, self-care, and QoL than younger patients, who tend to be more interested in physical and sexual attractiveness and preserving their body image. •Several studies have concluded that great involvement in the decision leads to higher mastectomy rates, although BCS rates still remain higher than mastectomy ones. Authors are not consensual why this happens, but this is commonly related with the fear of reocurrence and reoperation which might rise when the patient fully understand the risks. However, this is not the case among young women (who give physical appearance a high weight in the decision), if the physician holds a higher academic degree or the surgeon is more experienced in BCS. In a final note, the impact of the surgical treatment on sexual functioning, was associated with the type of surgical treatment that women with breast cancer received: women who underwent BCS reported a lower impact on their sexual life, and less concerns about sexual attractiveness, than women who underwent a mastectomy. [2] 2.4 Summary The evolution of screening programs to detect breast cancer in its early stages of development, as well as improvements in neoadjuvant and adjuvant therapies helped increase the survival rates of women with breast cancer, meaning that more than 90% of them will live with the consequences of the treatments for many years. However, studies have shown that when patients are involved in treatments decisions, it results in better acceptance of the outcome results [46]. The communication between surgeon and patients is extremely important, and should be adapted to patient’s personality, and age-group. So a surgical planning tool that simplifies the communication is important, specially when distinct aesthetic outcomes can occur due to heterogeneous practices in BCS and ontological treatments. The ability to visualize the potential outcomes of surgery, offered by surgical planing tools, is key in the decision making process. Nowadays, surgeon and patient communication about aesthetic outcomes of surgery still relies in 2D drawings or photo manipulation or, at most, resorts to 3D tools limited to generic female torsos. In either way, patients can not project themselves in the models, which hardens their choices and jeopardizes their expectations. For these reasons, pre-operative simulation software can be used as a communication and planing tool for both the patient and the surgeon. The advantages are multiple: surgeons will be able to educate patients better, patients will be able to communicate their expectations in a clearer way, and surgical procedures choices will take in consideration clinical limitations and patients desires [4,16]. In a further extension, simulation tools can be used to: (1) minimize doctor–patient misunderstandings, (2) find the optimal way of operation, (3) diminish the patient’s fear about the operation and (4) select the most desired figure. Furthermore, they may help the surgeon plan specific aspects of the procedure to achieve the agreed upon goals, and they facilitate surgical training by allowing trainees to design procedures and understand the 12 Breast Cancer results prior to actually operating on the patient [30]. Hence, efforts have been made to deliver a breast surgery simulation tool which is patient-specific. Some technical limitations need to be solved, and they comprise the need to fit a 3D point cloud of the breast to a usable model for simulating breast deformities. The next chapter details on the related work and breast models developments nowadays. 13 Breast Cancer 14 Chapter 3 3D Modelling There is still a significant number of women intervened for breast conserving surgery that are not happy with the result, which leads to self-image issues and emotional overload [44]1. The ability to visualise the potential outcomes of the surgery and make decisions on their surgical options is, therefore, very important for patients and surgeons. Currently, surgeons rely mostly on 2D visualization tools for discussing several cosmetic results with patients, such as 2D drawings, images and simple computing morphing capabilities. 3D modelling options have been addressed in later years, but usually not in a patient-specific manner or demanding expensive 3D scanners, landmarks positioned in women torsos or complicated procedures to obtain 3D data of the patient. To develop a breast surgery planning tool, it is necessary to create 3D models of the patient’s breasts. In this chapter, the related work of 3D modeling is reviewed, with special focus on parametric models, the fitting of superquadrics as primitive functions, deformable models and non-rigid registration. Statistical shape models and modelling by example are also discussed. 3.1 Overview Surface reconstruction, and precise representation of anatomical structures from 3D data using a small set of numerical parameters, has applications ranging from better ergonomics design of human spaces [3], design of clothing [32], modeling of realistic human characters in animation, or medical purposes. The description of human body shape by a small set of parameters has a long history, but considerably less work has been done and published regarding specific organs. In the particular case of breast, the work is even more sparse [22]. The main drawbacks in modelling the breasts are related to their large variability in shapes and the lack of unambiguous anatomical landmarks which remain intact after breast interventions. Female breast is a complex 3D object lying on 1http://www.beautifulabc.com 15 3D Modelling a curved surface (the chest wall). Its boundaries are rather fuzzily defined, and few anatomical landmarks are easily identifiable in the study area [4,13]. Attempts to model the 3D surface of the breast include the use of Magnetic Resonance Image (MRI), Computed Tomography (CT), 3D surface imaging systems (laser scan, stereophotogrammetry, fringe light projection etc.) or reconstructions of the 3D shape from 2D photographs [23]. However, these have been predominantly used to obtain 3D representations of the breast to objectively evaluate aesthetic outcomes of plastic surgery (mainly augmentation) and only to a lower extent in reconstructive surgery results. The goal is to extract volumetric measures or characterize the shape, instead of modelling or simulating breast modifications for visualization purposes. A technological review of 3D techniques used in the assessment of aesthetic outcomes from breast surgery can be found in [36]. Here, special reference is made to the Breast Shape Analysis project from IPLAB. Within the scope of this project, several studies were developed [13,14,18] in pursuit of a suitable tool for objective outcome evaluation in breast reconstructive surgery, from which resulted the Breast Shape Analyzer. This group also researched parametric representations of the breast, making use of MRI scans and Principal Component Analysis (PCA) [21,22], which might be the ground for surgery planing and breast modelling. Despite this generalized use of 3D data of the breast for characterizing the shape and evaluating its aesthetic profile, breast modelers or simulators are less common, at least for oncologic breast surgery. In fact, there is a multitude of breast simulators for breast augmentation surgery, which rise from modelers based on 2D images of the patient such as Crisalix c 2, to more complex and proprietary systems to collect 3D images and obtain 3D patient specific models of the subject and simulate results before plastic surgery, such as 3DBreastSculptor c 3and AxisThree c 4.Crisalix c is a web based 3D simulation application for plastic procedures including breast augmentations. The simulation procedure is based on standard 2D photographs, and a range of implant types, sizes and surgery techniques can be selected during the 3D consultation. 3DBreastSculptor c is a proprietary platform (Vectra3Dc ) for breast augmentation surgery. It uses six cameras to take simultaneous photos of the patient and create a virtual model. Simulations of breast augmentation surgery are generated by varying the size of implants, as well by selecting different implants manufactures . AxisThree c also uses a proprietary scanner to capture 3D images of the patient torso used to simulate breast surgery outcomes by selecting implants from the major manufacturer catalogs, positioning the implant above or under the muscle and adjusting tissue elasticity with their Tissue Behavior Simulation software. Breast modelers for breast surgery other than breast augmentation are less common, but some works are presented here. These usually require the mapping of 3D data to a model which is posteriorly modified to describe breast deformities. Kim et. al [30] developed a 3D virtual simulator for breast plastic surgery using image-based modeling and morphing the shape of the breast by an example-based approach. First, the subject’s 2http://www.crisalix.com/en.VtoN6jYbCV4 3http://www.sculptmydream.com 4http://www.axisthree.com/products/breast-surgery-simulation 16 3D Modelling 3D torso data is obtained from 2D orthogonal photographs (one front-view plus two side-view photographs). Several feature points are marked in the images, which are used to deform a 3D model template to fit the breast: an affine transformation matrix is obtained with Singular Value Decomposition (SVD) to match the template model feature points to the feature points calculated from the 2D images, and local deformation of the model is further accomplished by interpolating the remaining vertices of the template mesh with Radial Basis Functions (RBF). The simulation of the surgery outcome is based on the idea that each subject can be expressed as a linear combination of the exemplar data. Hence, a weighing matrix of the feature points can be obtained from the pre-operative data of the subject, which is then applied to the displacement vectors between the exemplar data feature points before and after surgery. A morphing tool is also available for further adjustments by the user. Figure 3.1 shows the overview of the virtual simulator, as well as the defined feature points used to reconstruct the breast. Figure 3.1: Overview of the 3D virtual simulator for breast plastic surgery and the feature points used in the recostruction of the breast model [30]. Seo et. al [41] created a breast modeler based on user-intuitive attributes. For this, a set of 28 3D scans obtained with landmarks were mapped to a template mesh in a coarse-to-fine manner. The process for mapping the template mesh to the scanned data mesh are described in Figure 3.2. In the coarse level, a feature-based method is applied to match a low level mesh (comprised of triangle patches connecting landmark points on the template), to the data mesh. A higher level mesh is obtained by subdividing each triangle of the template mesh into a fixed number of finer triangles. The fine fitting is accomplished by non-rigid registration of this fine mesh to data scans. In this manner, each data mesh has the same set of points and triangles which constitute shape vectors. For the model generation, the shape vectors dimensionality are reduced by PCA and a linear model that relates intuitive user-supplied parameters to the shape of the breast is built. Finally, RBF interpolation is used to map the user-supplied attributes to the shape vectors. Breast models are generated accordingly to user-supplied parameters. 17 3D Modelling Figure 3.2: Mapping of the template mesh to data mesh in [41]: (a) Template mesh prior to the pre-processing, (b) target scanned mesh, (c) deformation of the lower level template mesh by relocating feature points, (d) higher level mesh is obtained by subdividing each patch in the mesh, (e) deformation of the higher level mesh. The works from Gallo et. al [21,22] present a technique to describe the shapes of women breasts in a low dimensional space. This is an example of a parametrization of the breast, which is accomplished by applying PCA to a dataset of 40 Nuclear Magnetic Resonances. The resulting principal modes are manipulated to generate new models of the breast. Although direct mapping between the proposed modes and common properties like volume, roundness or concavity is not possible, some clinical interpretation can be made. As seen on the examples above, the fitting of template models or parametric representations of the breast in low dimensional spaces are usually the first step in the simulation of surgery results or modelling of breast deformities. Since 3D raw data cannot be easily manipulated for planning, it is often needed to fit smooth surfaces to 3D data in order to obtain deformable models. The purpose is to have a compact representation of the breast, usually with a reduced number of parameters, while still providing a good global approximation of the shape. 3.2 Parametric Models In 1995, Bardinet et al. [7] published a work on how to fit a parametric deformable model to unstructured 3D data, with applications in medical images of the heart. Later, Chen et. al [16] developed the idea of fitting superquadrics to the breast, and more recently, Pernes et. al [38] updated on Chen’s work by trying to improve the fitting process of the proposed parametric model of the breast. In this section, these works are reviewed in detail, but first superquadrics are described. Although there is a multitude of geometric parametric entities commonly employed in computer vision for 3D object modeling (i.e. generalized cylinders, implicit polynomials, blob models), superquadrics are extensively used. The reasons for their choice are several: •their parameters have an intuitive meaning which make their handling straightforward; •superquadrics parameters have a large expressiveness power for natural shapes with rounded edges and corners, such the case of the rounded shape of breasts; 18 3D Modelling •the fitting of superquadrics to 3D data is a problem thoroughly investigated, robust and fast methods have been developed for this purpose. The term superquadrics was first defined by Barr [8], and refers to a family of shapes that includes not only superellipsoids, but also superhyperboloids of one or two pieces, as well as supertoroids. (Figure 3.3). Superquadrics are also often used to refer to superellipsoids instead of the family of the implicit surfaces obtained by extension of quadrics. This is the case of the referred works, so the term superquadrics will also be used here as a synonym for superellipsoids. Figure 3.3: Superquadrics are a family of shapes that includes (a) superellipsoids, (b) superhyperboloids of one, and (c) of two pieces, and (d) supertoroids [28]. 3.2.1 Fitting of superquadrics Accordingly to Barr [8], a 3D surface can be obtained by the spherical product of 2D curves. A unit sphere (3.3), for example, is produced when half circle in a plane orthogonal to (x,y)(3.1) is crossed with the full circle in (x,y)(3.2): m(η) = "cosη sinη#,−π/2≤η≤π/2 (3.1) h(ω) = "cosω sinω#,−π≤ω<π(3.2) r(η,ω) = m(η)⊗h(ω) =    x y z   =   cosηcosω cosηsinω sinη   ,−π/2≤η≤π/2 −π≤ω<π(3.3) 19 3D Modelling (a) REM mesh (b) Breast modeled using the REM at different resolutions. chosen Figure 3.6: Radial Elements Method for modelling breast deformations [6]. 3.4 Statistical models Some approaches to model anatomical parts of the body are based on statistical models. These models are derived from a training set of exemplar images, manually annotated or with automatic landmarks. The resulting models mimic the variations in shape of the training set, and are used to synthesize shapes similar to those in a training set. The fundamental idea is that any new shape can be modelled as combination (i.e, linear combination) of the exemplar data. Example-based models are extensively used to capture the shape of human body and synthesize human avatars [17]. The generation of these models depends on suitable landmarks, i.e., the identification of points which can be consistently located from one image to another. The position of these points form the shape vector of each image, and combining all shape vectors of each image we obtain the training set. Before extracting statistics from this set, it is necessary to align it, meaning that all shapes are represented in the same coordinate frame. The next step is to model the distribution of the training set to derive a parameterized model of the shape and restrict the range of plausible. 6 In particular, any new data Xnew can be generated as a combination of the nexemplar shapes in the dataset X: XXnew = n ∑ i=1 αiXi,with n ∑ i=1 αi=1.(3.15) To simplify the model, data compression methodologies as PCA are applied to data. PCA computes the main axes of variation in the dataset, so the model can be obtained with fewer parameters. The number of selected modes define the explainability of the model. The application of PCA is as follows: 6http://www.cmlab.csie.ntu.edu.tw/ cyy/learning/papers/PCAASM.pd f 26 3D Modelling •Compute the mean shape: X=1 n n ∑ i=1 Xi(3.16) •Compute the covariance of the shapes: S=1 n−1 n ∑ i=1 (Xi−X)(Xi−X)T(3.17) •Compute the eigenvector, φiand corresponding eigenvalues λiof X(sorted in descendent order). If Φcontains the teigenvectores corresponding to the largest eigenvalues, the training set, X can be approximated using: X∼X+Φb,(3.18) where vector bdefines a set of parameters of the model, and is a tdimensional vector given by b=Φt(X−X). As consequence, the equation 3.15 can be simplified to: Xnew =X+ n−1 ∑ i=1 αixi,(3.19) which means that the model can be fitted to a new shape data by adjusting the weighting factors αiassociated with each mode xi, and deformations of the shape can be simulated by varying the modes of the model. Statistical modelling has been extensively used to design morphable models of the human body [3,42], or particular parts of the body such as faces [11]. In the last case, the model can even be used in tasks of facial recognition. Additionally, statistical shape models are a promising approach to improve the segmentation of anatomical shapes in medical images, as the example of the liver segmentation from CT datasets in [31]. Moreover, this example-based strategy is the one followed in the previously described works [29,41]. However, while in the case of the [29] the methodology, for generating patient specific simulations of breast deformations, is described, in [41] new shapes can be modelled from the exemplar data by interpolation and user-supplied attributes, but the simulation of breast deformations on patient specific data is not covered by the authors. Finally, statistical models can also be combined with other deformation models, such as the Finite Element Models (FEMs)6. In fact, this is the strategy exploit in [45] to circumvent the need of spatial correspondences between breasts. In this work, a 3D statistical deformation model of the breast is built from biomechanical simulations. Breast were globally aligned in their compressed form, using FEMs. The displacement fields from FEMs were then mapped to a common space (one selected atient) and normalized by breast size. Variability is modelled based on the principal components of the FEM displacements between all examples and the common space. The goal of the work was to capture the average breast motion and its variability due to compressing it between 27 3D Modelling two plates as performed in mammography. Two statistical models were built which could prove useful for mammograms registration. The first model captures the motion between compressed and uncompressed breasts, and a second one describes the deformation difference due to variations in patient positioning and compression magnitude. 3.5 Summary The use of models to describe the human body has a long history, and they have been successfully used in creating human avatars with applications ranging from medical to entertainment, such as the use of human models in movies and games. Models can be more rigid, such as parametric models whose deformations are limited by the primitive function, or more flexible as in the case of the FFD. Therefore, the former is particular useful in model coarse fitting, while the fine fitting is accomplished employing the latter. Additionally, physical models based on physical equilibrium equations are at disposal, and their combination with statistical models can expand their usability and adequacy to reality. Some examples have been shown on which these models are used to model the human breast and simulate their deformations. However, their practical applications are focused on plastic surgery or image registration, supporting the lack of work in simulating breast reconstruction surgery results. 28 Chapter 4 Proposed Methodology The development of a planning tool for breast surgery demands the use of breast models that can be interactively adjusted to describe the desired breast shape. However, the usability of a surgery planning tool is potentiated if the models used to simulate the results are specific for each patient, and the parameters required to create deformations are interpretable by the surgeon. In this chapter, two approaches are described for planing breast deformations, but first, the methodology used to create the synthetic breast databases used in this work is detailed. 4.1 Databases The development of deformable models for planing breast deformations depends on the availability of train and test examples of breast deformations. However, to the best of our knowledge, no public database of breast deformation exists. Therefore, two synthetic databases of breasts were created to develop the methodologies in this dissertation: •Database V, consisting of breasts with the exact same shape, but varying volumes (V). There are 65 synthetic breasts, divided into train (50) and test (15). Breast volumes are normally distributed, as would estimated if real breasts were used [27]. •Database S, with breasts of different volumes and 4 different shapes (S). 12 (train) and 2 (test) breast of different volumes were generated for each shape of breasts, resulting in 48 breasts are used for train, and 12 for test. The breasts composing each database were created using the methodology described in [16]; breasts are generated following the steps: 1. design a superquadric by setting the axis values (a1,a2,a3,a4,a5), with (ε1,ε2) = [1,1]; 2. define the 12 deformation parameters inputs of the 5 global deformation functions; 29 Proposed Methodology 3. deform the superquadric with the 5 deformation functions (ptosis, turn, top-shape, flattenside and turn-top). Breasts with different sizes and volumes were obtained by varying the size of the superquadric primitive, controlled by the axis values. Note, however, that the coordinate system associated to the breasts is different than the one proposed in [16]. In here, the z axis protrudes from the chest wall outward through the nipple (anterior-posterior), y axis goes up (inferior-superior), and the x axis goes from right to left (lateral-medial), as shown in Figure 4.1. The deformations described in the paper were all adapted for this system. The deformation parameters used to create the breasts of the synthetic database are listed in Table 4.1, and examples are shown in Figures 4.2 and 4.3. The histogram of the breast volumes in databse V is also shown in Figure 4.4. All breast were created with 10 000 points. Figure 4.1: Side and Front Views of the quadratic primitive used to model synthetic breasts with the associated coordinate system (adapted from [16]). Ptosis Turn Top Shape Flatten Side Turn Top (b0,b1)(c0,c1)(s0,s1,t0,t1)(h0,h1)(g0,g1) Database V Generic Shape N/A (0.38, 0) N/A N/A (0.41,0) Database S Shape 1 (-0.2, 0) (0.25, 0) (0.5, -1, 0, -1) (0, 0.01) (0, 0.2) Shape 2 (-0.1, 0) (0, 0.05) (1.5, -7, 0, -5) (0 , 0.05) (0.05 , 0.05) Shape 3 (0, -0.05) (0.3 , 0.05) (1, -0.5, 0.5, -0.5) ( 0.1 , 0) (0 , 0.3) Shape 4 (-0.1, -0.05) (0.38 , 0) (1, -0.5, 0.5, -0.5) N/A (0.41 , 0) Table 4.1: Deformation parameters used to create the synthetic breasts composing database V and S(parameters’ nomenclature as used in [16]). 30 Proposed Methodology (a) 3D view (b) Front view (c) Top view (d) Side View Figure 4.2: Generic breast shape used in Database V. (a) 3D view (b) Front View (c) Top View (d) Side View Figure 4.3: Four different types of breast shape used in Database S (all breasts shown were created with the same axis values, for demonstration purposes). 31 Proposed Methodology Figure 4.4: Histogram of database Vbreast volumes in train (blue) and test (brownn). Volumes shown in number of pixels. 4.1.1 Parametrization of Breast Deformations In this dissertation, three breast deformations were modelled, namely the ptosis, turn and top shape deformations proposed by [16]. For each type of deformation, a range of values is defined for the deformation parameters, and subsequently applied to each breast of a selected database, to obtain examples of breasts deformations with different degrees. This means that the number of examples in a dataset will depend on the number of degrees of deformation (#dd), and the number of breasts (#breasts) in the database, to each the deformation is applied: each dataset will have #dd ×#breasts examples. Depending on the number of degrees of deformation, a matrix of deformation parameters is created for each type of deformation, with as many rows as #dd, and the column number defined by the number of input parameters of the corresponding deformation function. Regardless the number of degrees of deformation, the parameter values of the first and last rows will remain constant. Next, the extent of deformation, Di j, caused by the parameters of a deformation matrix row j applied to breast iof the database, is quantified by the normalized euclidean distance between the points of the original, Pi0and deformed Pi j point clouds of the breast, with respect to a reference breast shape P00, and its deformed point cloud P0jobtained with the same deformation parameters: Di j =kPi0−P00k+ Pi j −P0j ,(4.1) where k.kis the normalized euclidean distance defined in equation: ko−mk=p(ox−mx)2+(oy−my)2+(oz−mz)2 qo2 x+o2 y+o2 z .(4.2) 32 Proposed Methodology (4.2). Finally, the degree of deformation is encountered by a discretization function which takes the extent of deformation as input, and outputs the degree of deformation for deformed point cloud. This function has as many conditions as the number of degrees of deformation, and the outputs are integer numbers between 1 and NDD, where NDD stands for the highest degree of deformation. The degrees of deformation 1 and NDD are always attributed to the same extent of deformation, regardless the type and number of degrees of the studied breast deformations. The reference breast shape, P00 has the generic shape used in database V, and was obtained with all axis values set to 10. In this thesis, three types of deformation were modelled, namely ptosis, turn and top-shape deformations as defined in [16]. Ptosis Ptosis is a deformation that describes the sagging that affects the breast with aging. In the defined coordinated system, this means that points will change towards the negative side of the y axis (Figure 4.1). The amount of change is conditioned, though, by their zposition, meaning that points near the chest wall will be less deformed. Using the notation of [16], the breast shape is defined by s(u)=[ex,ey,ez], where u= (u,v) with the material coordinates as shown in Figure 4.1.sis obtained as a result of a global transformation function T= (e;qT), which takes as input the vector of primitive parameters, e= (a1,a2,a3,a4,a5,ε1,ε2), and a vector of deformation parameters, qT. For ptosis, qt= [b0,b1]T, where b0and b1are coefficients of a quadratic function, and the breast shape s=Tp(e;b0,b1)is given by: s(u) = [ex(u),ey(u)−b0ez(u)−b1ez(u)2,ez(u)]T.(4.3) Figure ( 4.5) shows a comparison between the original breast shape, and the shape of the deformed breast after a ptosis deformation. (a) 3D view (b) Front view (c) Side view Figure 4.5: Example of ptosis deformation with original breasts (black) and deformed breast (skin color) superimposed (b0,b1= (0.7393,0.1643). Turn Turn deformation causes the shape of the breast to turn to left or right; in the breast coordinate system, points will change along the xaxis, toward the negative or positive side of the axis 33 Proposed Methodology (Figure 4.1), depending if the deformation causes the nipple to point towards left or right. Similarly to ptosis, the amount by which the points xcoordinates change is modelled by a quadratic equation with coefficients (c0,c1). The parameters of turn are qt= [c0,c1]T, and the breast shape s=Tc(e;c0,c1)is given by: s(u) = [ex(u)−c0ez(u)−c1ez(u)2,ey(u),ez(u)]T(4.4) Figure( 4.6) shows a comparison between the original breast shape, and the shape of the deformed breast after a turn deformation. (a) 3D view (b) Front view (c) Top view Figure 4.6: Example of turn deformation with original breasts (black) and deformed breast (skin color) superimposed (c0,c1) = (1,0.87140.3714). Top Shape The top shape deformation models the concavity/convexity of the profile of the top breast, meaning that only points on the top half of the breast (0≤v≤π)will be affected. Ideally, the top breast profile is concave, but it can be change due to surgery or congenital deformations, for instance. In this deformation, points will change along the zaxis, as function as their angular position uin relation to the nipple (Figure 4.1). The zcoordinate of points will be scaled by a polynomial function of u=u2 pi , where uspans the range [0,1]. The parameters of top shape deformation are qt= [s0,s1,t0,t1]T, and the breast shape s=Ts(e;s0,s1,t0,t1)is given by:    s(u) = [ex(u),ey(u),ez(u)β(u)]T,if u∈[0,π 2]×[0,π] e(u),otherwise (4.5) β(u) = A(u)5+B(u)4+C(u)3+D(u)2+E(u)+ F,(4.6) with A=1 2t0−3s0−3s1+1 2t1;B=3 2t0+8s0+7s1−1 2t1;C=−3 2t0−6s0−4s1+1 2t1;D= 1 2t0;E=s0; and F=1. For detailed information about the effect of the deformation parameters and their role in the scaling polynomial function β(u), please refer to the original paper [16]. Nonetheless, sparameters define slopes, and tparameters define curvatures; 0 indicates that the parameters change the top profile near the chest wall, while 1 indicates changes near the nipple. Figure 4.7 shows the effect of varying the slope parameters ssignal on the top breast profile, for fixed values of curvature. 34 Proposed Methodology (a) (s0,s1) = (−1,−1)(b) (s0,s1) = (1,1) (c) (s0,s1) = (−1,1)(d) (s0,s1) = (1,−1) Figure 4.7: Effect of varying the slope parameters (s0,s1)signal on the top breast profile in top shape deformations; original breasts (black) and deformed breast (skin color) superimposed. Table 4.2 lists the minimum and maximum values of each range of deformations parameters used to create the examples of breast deformations used to train and test regression models. Type of Deformation Parameters Min Max Ptosis b00.45 2.25 b10.1 0.5 b0>b1(0.225 , 0.55) (1.125 , 0.25) b1≥b0(0.05 , 01) (0.5 , 0.5) Turn c0 > c1 (0.3 , 0.05) (1.1 , 0.5) c1>c0(0.05 , 0.125) (0.5 , 0.625) Top Shape s− 0>s+ 1(-0.5 , 0.25) (-2.5 , 2) s− 1>s+ 0(0.25 , -0.5) (1.75,- 2 ) Table 4.2: Minimum and maximum values of the range of deformations parameters used to create examples of ptosis, turn and top shape deformations. 4.2 Parametric Models for Planning Breast Deformations In this section, a methodology focused on validating the use of machine learning techniques to learn the parameters associated with each major breast deformation is presented. The proposed approach is detailed in the diagram of Figure 4.8. 35 Proposed Methodology which means that the NN outputs are obtained evaluating this equation on all neurons of the output layer [35]. Figure 4.12: Diagram of a NN with activation, bias and weights nomenclature examples [35]. The use of NN is particularly advised when a multiple output regression model with correlated outputs is needed because the weights and activation functions of the NN are trained taking data correlation in consideration. This is not the case when LR or SVM multiple output regression models are created. These methods address the multiple output problem by creating as many independent regression models as outputs, which does not take in consideration the relationships between the multiple outputs [10]. 4.2.2 Feature Extraction and Selection In the use case of a tool for planning breast cancer surgery results, the inputs of the system would be the point cloud of the patient’s breast, along with pairs of types and degrees of deformations that the doctor would expect to result from surgery. Therefore, the feature set used to train the regression models must depend solely on the original breast, and necessarily include the degree of deformation. For the ptosis study, five feature sets plus a reference were designed, as described in Table 4.3. 42 Proposed Methodology Features Acronym DD v0Z0Y0z0y0Ydisplacement REF X X X X FS1 X X X FS2 X X X X FS3 X X X FS4 X X FS5 X X X X Table 4.3: Characterization of the feature sets compared in the ptosis study: DD stands for degree of deformation, capital letters refer to vectors and lowercase are associated to single values. Taking in consideration that ptosis changes the points ycoordinates as function of their z coordinates, the xcoordinates are not important to predict the parameters. So, a simple feature set would include a vector of the ycoordinates of all points in the original point cloud, (Y0), as well as a vector of their zcoordinates, (Z0). The synthetic breasts in databases were generated with 10000 points, so each vector would have the same size. The extent of deformation caused by ptosis varies with the size of the breast, so the original volume, (v0), was also considered in every feature set. On the other hand, the points in each vector of coordinates are well correlated, so a feature set was defined on which only the coordinates (z0) and (y0) of one point (the average point cloud) was used. Variations of these datasets include no point coordinates, as in FS4; or only vectors of a specific coordinate, Z0or Y0, as in FS1 and FS3, respectively. Considering that feature sets containing coordinates vectors result in large datasets, with the number of features going up to 20005 features, the use of PCA to reduce datasets dimensionality was explored. Moreover, the effect of the amount of variability explained by PCA eigenvectors on the regression model performance was also considered. In other words, the number of PCA features were optimized up to a maximum number of features describing 100% of the dataset variability. Later, the performance of datasets with the optimized number of PCA features were compared with the performance of their correspondents without the application of PCA. Finally, a reference feature set, REF, was defined on which the displacement in ypoints coordinates, Ydisplacement , was included. This is the solution used to have a comparison baseline between feature sets and types of models, regarding the limitations of the literature methods to model breast deformations in a patient-specific way. The exception would be the method proposed by Kim et al. [29]. Nevertheless, this was limited by the definition of invariant landmarks on the 2D breast images, and the use of a 3D template model with fixed number of vertices where the relationship of the mapped landmark points was constant. Therefore, to implement this method on the generated point clouds, fixed landmarks would have to be manually annotated on every 3D point cloud of breasts in both databases. These would be extremely laborious, time consuming and prone to human errors itself, jeopardizing the ability of using this method as baseline. On the other hand, the use of displacements as features would result in the best performance possible for 43 Proposed Methodology each regression model, because the regression target values would have caused the displacements themselves. All features were normalized to lie between 0 and 1, and the feature set associated with the best regression performance on the ptosis study is next adapted to the remaining deformations, and used in the subsequent tests. 4.2.3 Model Optimization For each feature set of ptosis, the three types of regression models described in Section 4.2.1 (LR, SVM and NN regression models) were explored and their relative performances compared. Once more, the model with the best performance on the ptosis study was selected, and afterwards used to predict the deformation parameters of ptosis, turn and top shape deformations using datasets with breasts varying in size and shapes (datasets obtained from database S). Despite the known limitations of LR to deal with large datasets and adapt to some non-linear relationships resultant from some deformation functions, this method was tested due to its simplicity of analysis. The simple linear regression was considered, and polynomial basis functions up to the fourth degree were tested, either with or without intercept term. The fitting of linear regression models was conducted using Matlab R Statistics and Machine Learning Toolbox (version 10.1), which uses the Least Squares method to minimize the sum of the squares of the regression errors [33]. Alternatively, SVM regressions were explored due to their popularity in solving regression problems, and their ability to describe non-linear relationships using the kernel trick and with sparse solutions, which decreases the computation times of predictions as compared to other models such as NN. Linear, polynomial and RBF kernels were tested, and a grid-search optimization with 4-fold cross validation strategy [26] was used to select the best parametrization, with the Relative Mean Squared Error (RMSE), defined by equation 4.23. In particular, exponentially growing sequences of Cand γ(only for RBF kernel) were tested: C=2−5,2−3,...,27, and γ=2−5,2−3,...,23. For the polynomial kernel, the searching values for the degree parameter were {1,2,3,4}. SVM regression models were implement using the Matlab R version 3.21 of the LIBSVM library [15], with the Least Squares learning algorithm. Although initially the regression problem was decomposed in simple regression problems of predicting a single parameter of the ptosis function at time, in practice, the selected regression model has to be capable of predicting multiple outputs: all deformation functions have at least two inputs. Both LR and SVM regressions approach the problem of multiple output regression in the same way: by modelling each output individually. However, this might lead to erroneous predictions due to the correlation nature of the outputs in our problem, which is disregarded in these type of approaches. Hence, NN regressions were introduced to predict multiple deformation parameters due to their capability o modelling the outputs’ correlation in the trained regression model. Feed-forward NNs with one hidden layer were trained with a varying number of neurons, {5,10,25,50,75}, using the Matlab R Neural Network Toolbox (version 8.4) and parallel computing provided by the Parallel Computing Toolbox (version 6.7), used to reduce the time that 44 Proposed Methodology took to train the NN, by using the multi-core, and multithreading, capabilities of the processor. The retraining strategy proposed in the Neural Network Toolbox 1to improve NN generalization and avoid overfitting was also followed. All NN were trained using the Scaled Conjugate Gradient Backpropagation [34]. LR, SVM and NN regression models were optimized using 4-fold cross validation in the train datasets, and the best parametrization was blindly tested on test datasets. Features were scaled and normalized to be between 0 and 1, and the same normalization was used in both train and test datasets. The RMSE, detailed in next section, was used as the evaluation metric in all optimizations. All code, routines and results were obtained in a machine with the following specifications: •Processor: Intel Core i7 4790k •RAM: 16 GB 1600 MHz DDR3 •Hard drive: 120 GB Solid State SATA Drive •Operating System: El Capitan (10.11.2) •Matlab: 8.6.0.267246 (R2015b) 4.2.4 Evaluation Metrics Let x1,...,xnbe the testing data and f(x1),..., f(xN)the target values for regression. If the true target values of testing data are known and denoted as yi,...,yN, the prediction results are directly evaluated by the relative mean squared error (RMSE, Eq. 4.23) and the mean percentage error (MPE, Eq. 4.24). MSE and MPE are normalized by the target values to provide relative metrics. This is convenient to compare the predictions results of different subsets of deformation parameters. RMSE = 1 N∑N i=1(f(xi)−yi))2 1 N∑N i=1yi ×100 (%)(4.23) MPE =1 N N ∑ i=1 f(xi)−yi yi ×100 (%)(4.24) These metrics are suitable for evaluating the ability of the model to correctly predict the parameters used as inputs for the deformation function. However, they are limited in the sense of describing the resulting differences between the original (O)and modelled (M)breasts. Therefore, the performance of the regression models was further evaluated using Hausdorff (4.25) and average Euclidean distances (4.2) as indirect performance metrics. These distances are computed between original and modelled breasts, and in both directions. The Hausdorff distance is used because it describes the worst case scenario. To provide a fairly comparison between breasts with 1http://www.mathworks.com/help/nnet/ug/improve-neural-network-generalization-and-avoid-overfitting.html 45 Proposed Methodology different sizes and resolutions within the same dataset, distances are always normalized by the distance of the original breast point to origin. h(O,M) = max o∈Omin m∈M||o−m||,(4.25) where Oand Mare the matrix of points (Npoints x3 dimensions X,Yand Z), and k.kis the normalized euclidean distance, previously defined in Eq. 4.2: 4.2.5 Residual Analysis In any regression problem is important to examine the residual plots in order to validate the model. In fact, this is a very useful way of verifying whether the regression has achieved its goal to explain as much variation as possible in a dependent variable. Residuals are estimates of the experimental error obtained by subtracting the target values from the predicted ones. They can be thought of as elements of variation unexplained by the fitted model. Since this is a form of error, one expects them to be (roughly) normal and (approximately) independently distributed with mean of 0 and some constant variance. 2A histogram plot of the residuals should exhibit a symmetric bell-shaped distribution, indicating that the normality assumption is likely to be true. 3 Ideally, all residuals should be small and unstructured. The regression model is expected to fail in predicting the target values in a random fashion: the model should predict values higher than the target value and lower than target value with equal probability, and should be independent of the size of the target variable. As consequence, a scatter plot of the residuals will be disordered if the regression is good 2 3. However, if residuals exhibit a structure or present any special aspect that does not seem random, the regression model is failing to describe the structure of the data and it should be revisited, perhaps adding additional terms, transforming data or even changing the model itself. 4 To sum up, the analysis of residuals is essential to verify whether some of the underlying assumptions of regression have been violated. A simple way to verify these assumptions is to use a classic 6-plot 5, which include: •Scatter plot of predicted versus target values; •Scatter plot of residuals versus target values; •Scatter plot of residuals versus predicted values; •Lag plot of residuals; •Histogram of residuals; 2http://www.itl.nist.gov/div898/handbook/pri/section2/pri24.htm 3http://www.originlab.com/doc/Origin-Help/Residual-Plot-AnalysisImprovingtheregressionmodelusingresidualsplots 4http://docs.statwing.com/interpreting-residual-plots-to-improve-your-regression/x-unbalanced-header 5http://www.itl.nist.gov/div898/handbook/eda/section3/eda3333.htm 46 Proposed Methodology •Normal probability plot of residuals (ordered residuals versus theoretical values from a normal distribution N(0,1)for ordered residuals. 4.2.6 Statistical Analysis Hypothesis tests, also known as significance tests, are used to compare and decide upon methodologies strategies. Independent-samples t-test with a significance level of 5% are conducted to find out whether the results of a specific strategy are better than its alternative. 4.3 Statistical Models for Planing Breast Deformations In parametric models of breast deformations (Section 4.2), the use of regression models are explored to learn the parameters of deformation functions associated with specific breast deformations, and within a range of degrees of deformation. These models are limited by the knowledge of the physical equation associated with each deformation, which might be unknown in practice, particularly in the case of breast deformations caused by breast cancer surgery. To overcome this limitation, a different strategy is also proposed, on which breast deformation models will be learnt solely from exemplar data, without relying on a deformation function to make predictions. To do so, NN models will be used to predict the points’ displacements between original and deformed breasts, as schematized in the diagram of Figure 4.13. However, predicting points’ displacements instead of singular parameters exponentially increases the number of outputs in the model. This is particularly problematic if data points from the original point cloud are used, considering that the number of points in a point cloud can vary greatly and includes thousands of points. Alternatively, a reduced number of points can be used if a model is fitted to the original point cloud, and used instead of the original raw data. Based on the fitting properties of the models described on Chapter 3, the FFD is used with a superquadric primitive to model the breast, following the works of [7] and methodologies developed in the scope of the PICTURE project 6. Figure 4.13: Block diagram of the proposed methodology for statistical models of breast deformations. 6http://www.vph-picture.eu 47 Proposed Methodology Similarly to the methodology described in Section 4.2, there will be as many NN regression models as the number of types of deformations to be modelled, and deformations are created in function of a degree of deformation used as input. However, instead of predicting deformation parameters, these models will predict the displacement of each point’s coordinates. Given a new point cloud describing the original shape of the breast, data will be fitted to a FFD model with a defined number of Control Points (CP). The coordinates of all CP, as well as features extracted from the fitted model, are then used to feed a set of regressions models, that output a matrix of CP displacements according to the desired degree of deformation. The matrix of CP displacements is used to modify the original matrix of CP, P, (by adding displacements to the original coordinates), which is then multiplied by the models’ transformation matrix, B, to obtain the deformed model, X, using Eq. 3.13. 4.3.1 Modelling 3D Breast Data with Free Form Deformation The method of fitting a FFD to 3D data is described in detail in Section 3.3.1, and follows the approach proposed by Bardinet et al. [7]. The main steps of the fitting methodology are enumerated as follows, and the intermediate results are shown in Figure 4.14: 1. Initialize the parametric surface from the 3D data of the breast: •A superellipsoid, with parameters (ε1,ε2)=(1,1), is centered at the center of gravity of the data, oriented with the moments of inertia of data, and the size of the ellipsoid axis are computed. 2. A uniform volumetric box of CP, P, is defined to embed the ellipsoide fitted in 1: •The ellipsoid is projected to a 2D plane, which is converted to local coordinates within the frame of CP; •The transformation matrix, B, that links the plane to the grid of CP is computed. 3. The box of CP is deformed based on the displacements field between the model and the 3D data of the breast, using the iterative two-step algorithm detailed in [7], which minimizes the error between model and data using the Least Squares Method. •The iterative model is repeated until an error bellow 0.0015 is obtained, or a maximum number of 1000 iterations is reached. 48 Proposed Methodology (a) 3D Data of Breast (b) Initial Superellipsoid (c) 3D Data of Breast embedded on the box of CP (d) Fitting plane on the box of CP (e) Final FFD model and box of CP (f) Final FFD and box of CP, superimposed on the original data Figure 4.14: Intermediate results of fitting FFD to 3D raw data of breast - raw data in skin color, models in red and CP box in green. 4.3.2 Model Optimization In the scope of this dissertation, the fitting of a NN regression model to predict the displacement of the CP, will only be optimized for one condition of ptosis deformation, namely the deformations caused by b0>b1, applied with 8 degrees of deformation to the database containing breasts of different shapes and sizes (database S)). This is due to the high computational cost of fitting the FFD to the 3D data of each breast in the resultant dataset using Matlab c , which takes several hours to complete, plus the computational cost of training and optimizing NN regression models with a large number of input and output nodes. Taking this into consideration, NN regression models used to statistically model breast deformations, were trained for a limited number of optimization scenarios: only the number of nodes in the hidden-layer, as well as the effect of the number of CPs used to model the 3D raw data of the data, were tested. In addition, a train-validation-test approach was used, instead of the 4-fold cross validation methodology followed in the optimization of parametric models. NN regression models were trained using the default values suggested by the Neural Network Fitting Interface of Matlab c , with the exception of the training algorithm (which was selected to be Scaled Gradient Backpropagation, the same used in the parametric models training), and the number of nodes of the hidden layer. The feature set was composed by the coordinates of CPs, as well as the volume of the FFD model of the breasts, with all features normalized between 0 and 1, and the performance of each model was compared using the RMSE (Eq.4.23), and the indirect performance metrics described in Section 4.2.4. 49 Proposed Methodology 4.4 Summary In this chapter, the creation of the synthetic databases is detailed, and the parametrizations of ptosis, turn and top shape deformations were defined, in order to use them as reference in this study. In addition, three regression models were reviewed, resulting in two proposed methodologies to evaluate the applicability of machine learning techniques for modelling breast deformations; the first proposal is based on the knowledge of the deformation function, with the goals of predicting the deformation parameters that cause a particular degree of deformation, while the second is based on learning from exemplar data, to create statistical models capable of describing breast deformations solely from real data examples. The next Chapter presents the results of these methodologies, and discusses the advantages and limitations of each proposed approach. 50 Chapter 5 Results and Discussion In the previous Chapter, parametric and statistical models were suggested as two possible approaches of using machine learning techniques to describe and predict breast deformations. In this Chapter, the results of such methodologies are presented and discussed, to determine the more adequate method to be integrated in a 3D planing tool for breast surgery. In the first Section, 5.1, the performances of several regression models optimized to predict the parameters of ptosis deformation are compared, and the model with the best results is adapted to create parametric models of other deformations. The results of turn and top shape deformations with the adapted model are also shown. In Section 5.2, the results of using regression models to predict breast deformations by learning from exemplar data are discussed. In the last Section, some final considerations are made. 5.1 Parametric Models for Planing Breast Deformations Following the two steps strategy described in the proposed methodology to obtain parametric models of breast deformations, results are divided into two subsections: the first subsection, 5.1.1, presents the results of the optimization process using the ptosis deformation as reference, while the fourth subsection, 5.1.4, presents the results of the selected model and features predicting the three breast deformations: ptosis, turn and top shape deformations. 5.1.1 Feature Set Selection and Model Optimization - a study on Ptosis In order to obtain the best predictions of breast deformation parameters, different combinations of feature sets and regression models were tested, using datasets derived from database V, which contains breast with variable size but with the same generic shape (Figure 4.1). This assures that the models performances will only be conditioned by the volume differences between the breasts in the dataset, which simplifies the comparison because the other components affecting the 51 Results and Discussion of each original point of the breast, to the origin, so values are shown in percentage. The modelled examples with the lowest, average and higher distances of each condition, are also shown in Figures 5.2,5.3,5.4 and 5.5. An analysis of the indirect performance metrics suggest that models are neither systematically bigger, or smaller than the original breast, because the distances computed in different directions (Modelled ⇒Original and Original ⇒Mordelled) have similar values. For all outputs, the average Euclidean, and Hausdorff distances, are lower than 2% and 7%, respectively. The maximum Euclidean distances are lower than 9%, and there are cases, on the worst case scenario, in which the original and breast model points can dist from each other up to a distance of 27.14% of their coordinates values, as suggested by the maximum Hausdorff distance for b1predictions. Nonetheless, results are on average acceptable, as shown in next figures. Modelled ⇒Original Original ⇒Modelled Output (s) Statistics Euclidean Hausdorff Euclidean Hausdorff b0 µ1.56 5.39 1.58 5.35 σ0.84 2.68 0.83 2.62 Min 0.01 0.02 0.01 0.02 Max 4.90 12.87 4.78 12.88 b1 µ1.68 6.70 1.82 6.99 σ1.23 4.23 1.22 4.22 Min 0.15 0.46 0.15 0.46 Max 8.60 27.14 5.73 20.11 b0>b1 µ1.71 6.16 1.74 6.18 σ0.94 3.12 0.91 3.18 Min 0.06 0.12 0.06 0.12 Max 5.78 19.10 4.59 17.71 b1≥b0 µ1.67 6.74 1.86 7.20 σ0.98 3.27 1.28 4.18 Min 0.19 0.74 0.19 0.74 Max 5.16 17.94 6.63 21.85 Table 5.4: Indirect performance metrics for the best regression models predicting ptosis parameters. All models are applied without PCA, using feature set FS5 with 8 degrees of deformation. Relative distances are show in percentage. 58 Results and Discussion (a) 3D view (b) Front view (c) Side view Figure 5.2: Examples for the best regression model predicting ptosis parameters b0-NN regression model - original breasts (black) and model breast (skin color) superimposed. The best, average and worst prediction results are shown in each column, respectively. 59 Results and Discussion (a) 3D view (b) Front view (c) Side view Figure 5.3: Examples for the best regression model predicting ptosis parameters b1-NN regression - original breasts (black) and model breast (skin color) superimposed. The best, average and worst prediction results are shown in each column, respectively. 60 Results and Discussion (a) 3D view (b) Front view (c) Side view Figure 5.4: Examples for the best regression model predicting ptosis parameters b0>b1-NN regression - original breasts (black) and model breast (skin color) superimposed. The best, average and worst prediction results are shown in each column, respectively. 61 Results and Discussion (a) 3D view (b) Front view (c) Side view Figure 5.5: Examples for the best regression model predicting ptosis parameters b1≥b0-NN regression - original breasts (black) and model breast (skin color) superimposed. The best, average and worst prediction results are shown in each column, respectively. 5.1.4 Ptosis, Turn and Top Shape Deformations Although the results of the ptosis study are encouraging, they were obtained with datasets consisting of breasts with the same generic breast shape, but with different sizes. However, breasts shapes can vary greatly among women, which is why further testing is imposed, using datasets consisting of breasts with variable sizes and shapes. Besides, other breast deformations than ptosis, have to be modelled in order to build a surgical planning tool for the breast, so the generalization of the proposed methodology to model other types of deformations is tested, with attempts to model turn and top shapes deformations. In this section, the performances of such models are presented, with all results obtained using datasets derived from database S. Models are evaluated in terms of indirect performance metrics; residual analysis plots and examples of the deformable models are shown for supplementary assessment of the models quality. 62 Results and Discussion 5.1.4.1 Ptosis NN regressions trained with feature set FS5 provided the best results when predicting ptosis parameters on datasets derived from database V. Now, these results are replicated using datasets containing different breast shapes and sizes, and the performance of NN regression predicting b0>b1, or b1≥b0are listed in Table 5.5. In line with to the results obtained in Section 5.1.3, the distances computed between Original ⇒ Modelled, or Modelled ⇒Original deformations are similar, meaning that the models are not systematically bigger, or smaller than the original breasts. The results for b0>b1have nearly the same performance of the regression model tested on datasets containing breasts with the same shape, but b1≥b0models have sightly higher differences than their counterparts. However, this can be caused by an outlier, considering that the maximum Euclidean and Hausdorff distances for b0>b1, computed in Modelled ⇒Original direction, are considerably higher then the same distances obtained in the datasets derived from database V. In fact, the residual analysis of these models ( Figure 5.6) confirmed the existence of an outlier ( signaled by a red arrow). The outlier is an example of ptosis deformations caused by high deformation parameters, whose degree of deformation was badly assigned by the mapping function described in Section 4.1.1. In spit of the existence of an outlier, residual analysis of both b0>b1and b1≥b0models suggest goodness of fitting: residuals are randomly dispersed in scatter plots of target, or predicted values, versus residuals; no structure is clearly identifiable in Lag plots, and residuals distributions are approximately normal, as implied by residuals’ histogram and normal plots. Figures 5.7 and 5.8 show examples of models with the best and average performance, including the model of the outlier as the worst example of modelling. Modelled ⇒Original Original ⇒Modelled Outputs Statistics Euclidean Hausdorff Euclidean Hausdorff b0>b1 µ1.66 5.69 2.10 6.52 σ1.38 4.60 2.51 6.66 Min 0.10 0.18 0.10 0.18 Max 6.71 22.22 11.76 30.97 b1≥b0 µ2.98 9.64 2.08 7.25 σ5.56 14.44 1.95 5.10 Min 0.16 0.46 0.16 0.46 Max 37.94 95.87 11.90 25.79 Table 5.5: Indirect performance metrics of NN regression models predicting ptosis parameters in datasets derived from database S. 63 Results and Discussion (a) b0>b1(b) b1≥b0 Figure 5.6: Residual analysis of ptosis parameters predictions on datasets containing variable breast shapes and sizes. (a) 3D view (b) Front view (c) Side view Figure 5.7: Examples for ptosis parameters b0>b1predictions on datasets containing variable breast shapes and sizes. - original breasts (black) and model breast (skin color) superimposed. The best, average and worst prediction results are shown in each column, respectively. 64 Results and Discussion (a) 3D view (b) Front view (c) Side view Figure 5.8: Examples for ptosis parameters b1≥b0predictions on datasets containing variable breast shapes and sizes. - original breasts (black) and model breast (skin color) superimposed. The best, average and worst prediction results are shown in each column, respectively. 65 Results and Discussion 5.1.4.2 Turn Turn deformation has a deformation function (Eq. 4.4) similar to ptosis, having two deformation parameters, (c0and c1), which control the amount by which the xcoordinates of breast points change, either by a linear - c0or quadratic - c1function of the points’ zcoordinates. In this way, ptosis approach is easily adapted if models are trained with feature sets containing xcoordinates instead of y. Turn deformations with c0>c1, or c1>c0were modelled using NN regressions, and the adapted version of FS5, without applying PCA. The indirect performances of turn models are listed in Table 5.6, and the correspondent residual analysis plots are shown in Figure 5.9. Modelled ⇒Original Original ⇒Modelled Outputs Statistics Euclidean Hausdorff Euclidean Hausdorff c0>c1 µ2.21 6.77 2.12 6.53 σ2.22 6.30 1.86 5.69 Min 0.19 0.41 0.18 0.41 Max 11.09 29.62 8.36 26.63 c1>c0 µ1.67 6.04 1.86 6.54 σ1.31 4.24 1.60 4.90 Min 0.16 0.38 0.16 0.38 Max 7.16 21.81 6.72 20.22 Table 5.6: Indirect performance metrics of NN regression models predicting turn parameters in datasets derived from database S. (a) c0>c1(b) c1>c0 Figure 5.9: Residual analysis of turn parameters predictions on datasets containing variable breast shapes and sizes. As expected, the performances of turn deformation models are identical to ptosis models’ performances, and the same outlier (signaled by a red arrow) was also identifiable in the residuals 66 Results and Discussion analysis, i.e, the application of the highest turn deformation parameters to the breast that caused the ptosis outlier, also resulted in a bad assignment of degree of deformation in turn deformation. The analysis of the turn models residuals also lead to the same conclusions of the ptosis, confirming the goodness of fitting of the regressions. Figures 5.10 and 5.11 show examples of models with the best and average performance, including the model of the outlier as the worst example of modelling. (a) 3D view (b) Front view (c) Top view Figure 5.10: Examples for turn parameters c0>c1predictions on datasets containing variable breast shapes and sizes - original breasts (black) and model breast (skin color) superimposed. The best, average and worst prediction results are shown in each column, respectively. 67 Results and Discussion that statistical models obtained using 125 CP have statistically significantly better performances (lower differences) than models using 216 CP: p−values are lower than 0.0001 obtained for each one of the four distances computed ( Euclidean and Hausdorff distances computed in both Original ⇒Model and Model ⇒Original directions). Although this result is unexpected, the differences between statistical models using 125 and 216 CP are particular evident when comparing the Hausdorff distances between Modelled ⇒Original which are 210.76% and 341.13%, respectively. Therefore, further analysis of statistical models was made using statistical models obtained with FFD with 125 CP, and trained with NN regression models with as many nodes as number of CP in the hidden-layer. The residual analysis plots of this model are presented in Figure 5.17: residuals analysis suggest a linear relationship between predicted and actual CP displacements, but scatter and lag plots of residuals suggest that there is some structure in the residuals distribution, and residuals are not normally distributed, as confirmed by their histogram and the normal plot. This means that the NN regression model used to predict CP displacements do not successfully describes all variation in the underlying function governing CP displacements. Figure 5.17: Residual analysis plots of the best statistical model - NN regression with an hiddenlayer of 125 nodes, and FFD models with 125 CP. The visual quality of ptosis deformations predictions obtained with this statiscal model, shown in Figure 5.18, further confirm the inability of the statistical model to to learn the underlying function of ptosis deformations. This figure shows examples of good, common and unappropriate predictions, and the examples suggest that small errors in CP displacement predictions can cause significant errors in the ptosis deformation model, even causing unnatural results (Figure 5.18c). Moreover, errors tend to be higher for displacements of CP situated on the extremes of the grid of CP ( CP that have an inferior number of CP neighbours). This causes common predictions to have 74 Results and Discussion local areas on which the differences between modelled and original deformations are higher, due to local sharp displacement of model points, explaining the high distance values of the Hausdorff distance (Figure 5.18b). (a) Good prediction (b) Common prediction (c) Unapropriate prediction Figure 5.18: Examples of ptosis deformation predictions using statistical models and FFD models with 125 CPoriginal breasts (black) and model breast (skin color) superimposed. 5.3 Final Considerations In this chapter, regression models were explored as a solution to predict parameters of breast deformations described by known equations, that would latter be used as inputs, in global deforming function to model the breast to a desired shape. In the absence of a database of real deformations, synthetic breasts and deformations were created to train and test the regression models, and a mapping function was designed to associate a measure of deformation to each example. Results show that regression models can be trained to successfully model breast deformations, such as ptosis or turn deformations, but they require that exemplar breast is properly classified in terms of type and scale of deformations. This was particular evident when trying to adapt the model optimized 75 Results and Discussion on the ptosis study to model top shapes deformations. The results obtained also provide valuable insights on the utility of machine learning to model breast deformations. The results of statiscal models, however, have to be further explored in future, before conclusions can be safely made about their suitability to learn deformations from examples. Perhaps, more complex NN regression models, with multiple hidden-layer should be optimized to eliminate the observable structure in residuals, and the goodness of fitting of the underlying FFD should be considered, by comparing the performance of models using other primitives in the coarse fitting step of the FFD methodology, The effect of the number of CP should also be confirmed by enlarging the range of different number of CP used. Additionally, a pre-processing stage should be implemented in any regression methodology to eliminate unwanted results caused by outliers. The negative influence of including outliers in the training and test steps of regression models was evidentiated when generalizing the ptosis approach to describe top shapes deformation, and the presence of outliers in the train set can partially explain the unsatisfactory fitiing of NN regression models in the statiscal model methodology. 76 Chapter 6 Conclusions The development of a 3D planing tool for breast surgery requires the existence of proper 3D deformable models of the breast with easy to manipulate parameters. While 3D modelling is a vast area of research with known applications in medical sciences or entertainment, 3D models of the breast are less common, due to the lack of physical landmarks that remain unchanged after deformation. Approaches to model the breast include: parametric models with deformable superquadrics to fit the 3D point cloud of the breast; non-physical deformable models among which FFD plays an important role; and physical models based on the mechanical properties of the breast, which usually resort to the use of mass-spring and finite element methods. In addition, statistical models obtained from exemplar data of the breast have been explored, on which a new breast is described as weighted combination of the exemplar data on the database. However, few studies are focused on modelling deformities of the breast, and the few examples that actual do it either require the positioning of landmarks on the patient’s body during image acquisition [29], are dependent on a limited number of mathematical equations that describe particular breast deformities [16], or fail to be patient-specific and use adjustable parameters easy to manipulate by the common user. The work of this dissertation was focused on developing 3D parametric models of the breast that enable the adjustment of the breast shape by manipulating the degree of deformation that is expected after breast surgery. At first, a complete study was carried on ptosis, to prove the usefulness of regression models to predict parameters of know deformation functions, and the type and degree of deformation were suggested as adaptable parameters to create the desired breast shape. Next, the best model and features were tested in the prediction of other types of deformation, namely turn and top shape deformations. Models on ptosis showed good fittings, being able to describe with acceptable errors different degrees and shapes of ptosis deformations. Turn models had similar results, but the generalization of these models to describe top-shape deformations questioned the suitability of the degree of deformation mapping function. So far, varying the absolute value of the deformation parameters caused different extents of deformation that could be measured by some distance metric between the original and deformed shapes, and subsequently be mapped to common degrees of deformation between the distinct breasts. While this strategy was 77 Conclusions adequate to ptosis and turn deformations, varying the values of the top-shape parameters do not cause distances between deformed and original shapes to overlap between distinct breasts. Instead, geometric different shapes resulted by varying either the absolute value or the sign of parameters, enlightening the necessity of a shape related metric to assess the degree of deformation. Facing this methodology pitfall, the nature of the underlying metric to determine the degree of deformation was adapted to successfully model top shape deformations, but the necessity of involving surgeons in the determination of the type and adjustable parameters that should be implemented on a planning tool for aesthetic results of breast cancer is fundamental. However, the proposed deformable models still depended on the knowledge of a physical deformation functions, which deeply limited their generalization for any type of deformation. In this sense, a preliminary study on the development of purely statistical models using exemplar data and NN regressions was also conducted. The obtained results are still unsatisfactory for practical uses, because some local strange deformations can occur if the displacement of a single point is wrongly predicted, but suggest that further research on this type of models can solve the problem of predicting any type of breast deformities, as long as breast shape data is available to train the models, and constraints are applied to deal with unnatural local deformations caused by errors affecting a particular control point. 6.1 Future Work The results obtained on this dissertation are encouraging and show that machine learning techniques are valuable techniques to bring innovation to the field of prediciting breast deformations, but they still require further development and improvement, before a proper breast surgery planning tool can be deployed and used in surgeon/patient communication. These are baseline results proving that 3D deformable models can be fitted to breasts, and used to predict the breast deformations in a patient-specific way, using simple adjustable parameters. However, these results need to be validated using 3D point clouds of real breasts and deformations. This can be challenging due to the lack of databases containing 3D point clouds of breasts, prior and after breast surgery, properly annotated with the type and degree of deformation. In fact, a primordial task in any future development in this area would involve the creation of such database. Furthermore, surgeons should be consulted to define the type of breast deformations that need to be modelled and validate the adequacy of the degree of deformation as the adjustable parameter. Finally, the preliminary results on statistical models are worth of further, and careful, exploration due to their adaptability and generalization characteristics. These models are generated using only exemplar data along with the type, and degree, of deformation, so the underlying methodology of these models can, in theory, be adaptable to any type of breast deformation, provided that a large number of exemplar data is available to be modeled. 78 Appendix A Ptosis Study A.1 Results of Regression Models Obtained using 4Degress of Deformation MPE Model Outputs Features #Features Fraction Kernel / # Nodes RMSE µ σ Max Min SVM b0 REF 4 100 Linear 0.64 9.09 10.36 50.88 0.63 FS1 3 100 Polynomial 7.33 13.79 14.14 75.57 1.30 FS2 5 100 Polynomial 7.22 14.06 14.19 75.57 1.04 FS3 4 100 Linear 7.59 13.44 14.66 78.38 0.20 FS4 2 100 Polynomial 7.39 13.69 14.38 75.49 0.53 FS5 3 97.69 RBF 6.84 13.76 13.34 73.96 0.27 b1 REF 4 100 Linear 0.64 16.37 19.54 93.87 0.13 FS1 3 100 Linear 2.66 31.46 34.02 159.53 0.10 FS2 5 100 Polynomial 2.60 32.59 37.55 172.30 0.39 FS3 4 100 Linear 2.52 33.94 38.43 152.09 0.19 FS4 2 100 RBF 2.57 37.31 33.51 102.10 1.09 FS5 3 97.73 RBF 2.32 29.69 35.53 179.80 0.48 NN b0 REF 3 99.999 5 0.00 0.61 0.91 5.12 0.00 FS1 3 100 5 7.72 24.44 23.02 92.66 2.82 FS2 4 100 5 8.41 26.82 28.16 109.96 0.42 FS3 4 100 5 7.30 23.15 20.87 83.87 0.55 FS4 2 100.000 5 8.41 23.77 23.75 93.84 0.74 FS5 3 97.41 5 7.36 23.19 21.28 95.70 0.16 b1 REF 2 99.997 10 0.00 0.96 1.13 6.50 0.01 FS1 2 99.996 5 1.678 15.80 16.36 72.99 0.06 FS2 5 100.0000 5 1.679 19.14 18.48 98.53 0.82 FS3 4 100 5 1.88 20.51 19.04 90.64 0.12 FS4 2 100 5 2.30 22.66 19.18 82.62 0.27 FS5 3 97.51 5 1.76 20.93 19.99 117.10 0.04 Table A.1: Regression models results for two different conditions of ptosis (b0and b1) - performances with 8 degrees of deformation and optimized number of PCA features. 79 Ptosis Study MPE Model Outputs Features Kernel/ # Nodes RMSE µ σ Max Min SVM b0 REF Linear 0.64 9.09 10.36 50.88 0.63 FS1 Linear 7.40 13.11 14.57 74.35 0.20 FS2 Linear 7.36 13.93 14.42 76.57 0.10 FS3 Linear 7.59 13.44 14.66 78.38 0.20 FS4 Polynomial 7.39 13.69 14.38 75.49 0.53 FS5 RBF 6.90 13.62 13.33 74.24 0.48 b1 REF Linear 0.64 16.37 19.54 93.87 0.13 FS1 Linear 2.71 32.82 38.21 179.49 1.80 FS2 Linear 2.70 33.43 37.14 175.46 0.69 FS3 Liner 2.53 33.79 38.06 150.01 0.12 FS4 RBF 2.57 37.31 33.51 102.10 1.09 FS5 RBF 2.32 30.46 36.37 169.76 1.24 NN b0 REF 50 0.16 2.93 2.78 10.64 0.02 FS1 10 8.55 26.32 31.12 105.37 2.11 FS2 5 12.58 32.33 45.14 285.91 0.19 FS3 50 10.17 29.07 36.65 134.00 0.29 FS4 5 8.54 21.88 26.16 109.48 0.15 FS5 5 8.56 23.20 22.62 83.89 1.02 b1 REF 50 0.05 4.46 4.04 18.95 0.22 FS1 5 2.08 24.09 26.49 128.28 1.39 FS2 5 7.46 72.28 76.52 220.78 12.51 FS3 50 2.78 30.38 27.98 134.30 0.53 FS4 5 2.16 19.74 18.26 95.29 2.35 FS5 5 1.97 17.66 22.87 149.35 0.21 Table A.2: Regression models results for two different conditions of ptosis (b0and b1) - performances with 8 degrees of deformation and without PCA. MPE PCA Outputs Features #Features Fraction # Nodes RMSE µMax Min With PCA b0>b1 REF 1 100 75 0.03 6.78 7.74 64.23 0.03 FS1 1 100 25 2.82 48.11 33.26 309.66 0.15 FS2 1 100 5 4.72 87.92 65.70 450.84 0.38 FS3 1 100 50 2.86 40.46 29.42 304.18 0.06 FS4 1 100 5 1.87 20.55 9.42 83.76 1.12 FS5 1 100 5 1.51 17.32 6.96 91.79 0.05 Without PCA b1≥b0 REF 3 99.997 5 0.00 0.34 0.17 1.90 0.00 FS1 2 99.996 5 1.71 19.24 7.00 111.92 0.09 FS2 5 100 5 1.73 16.50 2.30 79.41 0.14 FS3 4 100 5 1.78 25.92 17.37 157.63 0.49 FS4 2 100 5 2.17 20.79 3.30 79.73 0.21 FS5 3 97.44 5 1.67 18.07 2.70 75.48 0.29 Table A.3: NN regression models results with 4 degrees of deformation for multiple output parameters in ptosis. 80 References [1] ACS, American Cancer Society. Breast Cancer Detailed Guide. Technical report, 2014. [2] L Aerts, MR Christiaens, Paul Enzlin, Patrick Neven, and Frédéric Amant. Sexual functioning in women after mastectomy versus breast conserving therapy for early-stage breast cancer: A prospective controlled study. The Breast, 23(5):629–636, 2014. [3] Brett Allen, Brian Curless, and Zoran Popovi´ c. The space of human body shapes: reconstruction and parameterization from range scans. In ACM Transactions on Graphics (TOG), volume 22, pages 587–594. ACM, 2003. [4] Remis Balaniuk, Ivan Costa, and Jairo Melo. Cosmetic breast surgery simulation. In VIII symposium on virtual reality, pages 387–396, 2006. [5] Remis Balaniuk, Ivan Costa, and Jairo Melo. Cosmetic breast surgery simulation. In VIII symposium on virtual reality, pages 387–396, 2006. [6] Remis Balaniuk and Kenneth Salisbury. Soft-tissue simulation using the radial elements method. In Surgery Simulation and Soft Tissue Modeling, pages 48–58. Springer, 2003. [7] Eric Bardinet, Laurent D Cohen, and Nicholas Ayache. A parametric deformable model to fit unstructured 3d data. Computer vision and image understanding, 71(1):39–54, 1998. [8] Alan H Barr. Superquadrics and angle-preserving transformations. IEEE Computer graphics and Applications, 1(1):11–23, 1981. [9] Asa Ben-Hur and Jason Weston. A user’s guide to support vector machines. Data mining techniques for the life sciences, pages 223–239, 2010. [10] CM Bishop. Bishop Pattern Recognition and Machine Learning. Springer, New York, 2001. [11] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 187–194. ACM Press/Addison-Wesley Publishing Co., 1999. [12] Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik. A training algorithm for optimal margin classifiers. In Proceedings of the fifth annual workshop on Computational learning theory, pages 144–152. ACM, 1992. [13] Authors G Catanuto, G Gallo, G M Farinella, G Impoco, A Pennati, A Spano, and M B Nava. Breast shape analysis on three dimensional models. In proceedings of Plastic and Reconstructive Surgery of the Breast: Third European Conference, 2005. 81 REFERENCES [14] Giuseppe Catanuto, Andrea Spano, Angela Pennati, E Riggio, Giovanni Maria Farinella, Gaetano Impoco, Salvatore Spoto, Giovanni Gallo, and Maurizio B Nava. Experimental methodology for digital breast shape analysis and objective surgical outcome evaluation. Journal of Plastic, Reconstructive & Aesthetic Surgery, 61(3):314–318, 2008. [15] Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27, 2011. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm. [16] David T Chen, Ioannis A Kakadiaris, Michael J Miller, R Bowen Loftin, and Charles Patrick. Modeling for plastic and reconstructive breast surgery. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2000, pages 1040–1050. Springer, 2000. [17] Rhodri Huw Davies. Learning shape: optimal models for analysing natural variability. University of Manchester, 2002. [18] Giovanni Maria Farinella, Gaetano Impoco, Giovanni Gallo, Salvatore Spoto, and Giuseppe Catanuto. Unambiguous analysis of woman breast shape for plastic surgery outcome evaluation. In eurographics Italian chapter conference, pages 255–261, 2006. [19] Bernard Fisher, Stewart Anderson, John Bryant, Richard G Margolese, Melvin Deutsch, Edwin R Fisher, Jong-Hyeon Jeong, and Norman Wolmark. Twenty-year follow-up of a randomized trial comparing total mastectomy, lumpectomy, and lumpectomy plus irradiation for the treatment of invasive breast cancer. New England Journal of Medicine, 347(16):1233– 1241, 2002. [20] Tristan Fletcher. Support vector machines explained. Online]. http://sutikno. blog. undip. ac. id/files/2011/11/SVM-Explained. pdf.[Accessed 06 06 2013], 2009. [21] Giovanni Gallo, Giuseppe Claudio Guarnera, and Giuseppe Catanuto. Human breast shape analysis using pca. In BIOSIGNALS, pages 163–167. Citeseer, 2010. [22] Giovanni Gallo, Giuseppe Claudio Guarnera, Francesco Milanese, Davide Modica, Giuseppe Catanuto, and Francesco Pane. Parametric representation of human breast shapes. 2009. [23] Joachim Georgii, Martin Eder, Kai Burger, Sebastian Klotz, Florian Ferstl, Levente Kovacs, and Rudiger Westermann. A computational tool for preoperative breast augmentation planning in aesthetic plastic surgery. Biomedical and Health Informatics, IEEE Journal of, 18(3):907–919, 2014. [24] Sarah FF Gibson and Brian Mirtich. A survey of deformable modeling in computer graphics. Technical report, Citeseer, 1997. [25] Nathália Silva Gomes and Sueli Riul da Silva. Evaluation of the self-esteem of women who had undergone breast cancer surgery. Texto & Contexto-Enfermagem, 22(2):509–516, 2013. [26] Chih-Wei Hsu, Chih-Chung Chang, Chih-Jen Lin, et al. A practical guide to support vector classification. 2003. [27] Shih-Ying Huang, John M Boone, Kai Yang, Nathan J Packard, Sarah E McKenney, Nicolas D Prionas, Karen K Lindfors, and Martin J Yaffe. The characterization of breast anatomical metrics using dedicated breast ct. Medical physics, 38(4):2180–2191, 2011. 82 REFERENCES [28] Aleš Jakliˇ c, Aleš Leonardis, and Franc Solina. Superquadrics and their geometric properties. In Segmentation and Recovery of Superquadrics, pages 13–39. Springer, 2000. [29] MK Kim, Taeryung Kim, Hyeong-Gon Moon, Ung Sik Jin, K Kim, J Kim, JW Lee, Eunshin Lee, Tae-Kyung Yoo, D-Y Noh, et al. Effect of cosmetic outcome on quality of life after breast cancer surgery. European Journal of Surgical Oncology (EJSO), 41(3):426–432, 2015. [30] Youngjun Kim, Kunwoo Lee, and Wontae Kim. 3d virtual simulator for breast plastic surgery. Computer Animation and Virtual Worlds, 19(3-4):515–526, 2008. [31] Hans Lamecker, Thomas Lange, and Martin Seebass. A statistical shape model for the liver. In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2002, pages 421–427. Springer, 2002. [32] Hyun-Young Lee, Kyunghi Hong, and Eun Ae Kim. Measurement protocol of women’s nude breasts using a 3d scanning technique. Applied Ergonomics, 35(4):353–359, 2004. [33] Steven J Miller. The method of least squares. Mathematics Department Brown University, pages 1–7, 2006. [34] Martin Fodslette Møller. A scaled conjugate gradient algorithm for fast supervised learning. Neural networks, 6(4):525–533, 1993. [35] Michael A Nielsen. Neural networks and deep learning. 2015. [36] Hélder P Oliveira, Jaime S Cardoso, André Magalhães, and Maria J Cardoso. Methods for the aesthetic evaluation of breast cancer conservation treatment: a technological review. Current Medical Imaging Reviews, 9(1):32–46, 2013. [37] Paolo Patete, Maria Ida Iacono, Maria Francesca Spadea, Giovanna Trecate, Daniele Vergnaghi, Luca Tommaso Mainardi, and Guido Baroni. A multi-tissue mass-spring model for computer assisted breast surgery. Medical engineering & physics, 35(1):47–53, 2013. [38] Diogo Pernes, Jaime S Cardoso, and Helder P Oliveira. Fitting of superquadrics for breast modelling by geometric distance minimization. In Bioinformatics and Biomedicine (BIBM), 2014 IEEE International Conference on, pages 293–296. IEEE, 2014. [39] Romain Raffin. Free form deformations or deformations non-constrained by geometries or topologies. In Deformation Models, pages 49–74. Springer, 2013. [40] Frank Rosenblatt. Principles of neurodynamics. perceptrons and the theory of brain mechanisms. Technical report, DTIC Document, 1961. [41] Hyewon Seo, Frederic Cordier, and Kyunghi Hong. A breast modeler based on analysis of breast scans. Computer Animation and Virtual Worlds, 18(2):141–151, 2007. [42] Hyewon Seo and Nadia Magnenat-Thalmann. An example-based approach to human body manipulation. Graphical Models, 66(1):1–23, 2004. [43] Alex J Smola and Bernhard Schölkopf. A tutorial on support vector regression. Statistics and computing, 14(3):199–222, 2004. [44] Demetris Stavrou, Oren Weissman, Anna Polyniki, Neofytos Papageorgiou, Joseph Haik, Nimrod Farber, and Eyal Winkler. Quality of life after breast cancer surgery with or without reconstruction. ePlasty: Open Access Journal of Plastic Surgery, 9, 2009. 83