An approach to characterize and evaluate the quality of Product Lifecycle Management Software Systems

An approach to characterize and evaluate the quality of Product Lifecycle Management Software Systems J.G. Enríquez, J.M. Sánchez-Begines, F.J. Domínguez-Mayo, J.A. García-García, M.J. Escalona Web Engineering and Early Testing research group, Computer Languages and System Department, University of Seville, Seville, Spain Keywords: Product Lifecycle Management Software quality assurance ABSTRAC T PLM (Product Lifecycle Management) is an information management system that can integrate data, processes, business systems and staffin a company, in general. PLM allows managing efficiently and economically the information that all these elements generate from the initial idea to design, manufacture, maintenance and elimination phases of the product lifecycle. PLM has to include processes and tools to assure the quality of the final products. This way, it is difficult for PLM experts (from aeronautical or automation organizations, among others) to find an environment that suggests which is the best PLM solution that copes with their necessities. A number of PLM solutions are available for this purpose, but experts require a suitable mechanism to select the most appropriate one for the specific context of each organization. For this purpose, this paper presents a quality model, based on QuEF (Quality Evaluation Framework), that aims at helping organizations choose the most useful PLM solution for their particular environments. This model supports both static and dynamic aspects that may be customized for any kind of organization and taken as reference model. Particularly, our approach has been validated in the context of large enterprises in the aeronautical industry within a real R&D project carried out between our research group and Airbus. 1. Introduction The market is being progressively globalized in the last decade. This process has been powered by strong technology development as well as transport infrastructure and telecommunications improvement all over the world. This evolution of the market has caused a revolution in the behavior patterns of consumers of goods and services. In fact, these consumers are demanding increasingly complex products with better design that suit their needs and expectations. Additionally, these products are manufactured in shorter periods of time and under cost constraints. In this context, companies should accelerate their product development and manufacture, always taking into account the updated legislative regulation on safety, health and environmental pollution. Moreover, companies should consider distributed consumers in a global market, who are characterized by elements like cultural diversity and language barriers, among other features. These aspects add new challenges in production models and products management, as follows: 1. Integrating within a single framework the partners that make up the company (e.g. suppliers, outsourcing, partnerships and multi-site activities). 2. Reducing manufacturing costs of goods. In consequence, it is necessary to reuse parts of previously developed products, as well as to implement tools that can optimize the time of design, prototyping or production, for instance. 3. Managing large volumes of data that often appear in a disorganized manner and without security access or manipulation. These challenges can cause a reduction in productivity and quality, which can provoke great economic losses. There are phases in which companies must continue to keep some control over their products, for example, by interacting with users to identify new needs in future products, by evaluating possible deficiencies of their products or by providing services for recycling or removing their products. All these factors make a need arise within organizations. It is related to having absolute control of products throughout their lifecycle, that is to say, during the product lifetime, from its conception as an idea to the stages of design, manufacture, support and recycling. The PLM (Product Lifecycle Management) paradigm was born to support these needs [1–3]. PLM is defined as a strategic business approach that applies a T 2. Related work We have hardly found out formal research papers that evaluate and compare PLM systems in a methodological way. However, despite the existence of whitepapers and techniques references about this topic, it must be stated that there are not are too many. In [6], authors develop a web application to compare only nine PLM systems that users can assess taking into account subjective features (such as ease of administration, ease of installation, ease of use or support evaluation) and objective ones (such as cost, deployment time and user assumption). These evaluations are based on real users’reviews and they are updated in real time. Nevertheless, Aras displays in [7] a characterization scheme of its own tool (Aras Innovator) and four other different tools (Siemens, Dassault, PTC and Oracle). This scheme is composed of 20 features, but there may be a bias in this comparison, since Aras Innovator supports all features. In [7], the authors present guidelines on how a comparative study should be performed to compare PLM systems in terms of the needs and strategies of the company. This paper does not provide a characterization scheme. Basically, the authors in [9] propose some methodological assistance to select the most appropriate PLM system for SMEs (Small and Medium-sized Enterprises). For this aim, they suggest an approach for a selection process based on determining the need, organizing the evaluation, identifying management, functional, technical and integration requirements, and evaluating a potential vendor partner. In [8], the authors analyze the PLM concept in 2005, study its trend in business environment and propose a full scenario of PLM technology solutions focused on the complete analysis of business drivers, industry requirements, limit of current solution and recent state-of-the-art review in the PLM domain. However, this paper does not provide any characterization scheme to compare PLM technology solutions, either. Finally, we have also covered some related works to propose and implement our feature model. On the one hand, we consider Camba et al.’s paper [9] that explains the importance of synchronous communication in PLM environments because, despite significant advances in the area of PLM, most tools are used as separate services disconnected from existing development environments. The authors propose a solution based on annotated CAD (Computer Aided Design) models. We understand that this situation may generate main information difficult to access during a communication session, where elements of CAD models are rarely linked to the context of discussion. Therefore, we will consider this feature in our model. On the other hand, we also follow the standard ISO 16,792-2015 [10], which specifies requirements for the preparation, revision and presentation of digital product definition data, hereafter referred to as data sets. Additionally, this standard defines guidelines for IT engineers to improve the annotation modeling in CAD models. These guidelines are associated with the previous related work. This reinforces our concern to include a feature to evaluate the creation of annotations in CAD models. 3. Background: QuEF methodology At the beginning, QuEF [5] was used to manage quality in Model- Driven Web Development methodologies, but, at present, QuEF manages quality requirements of entities (such as products, processes, services or organizations) in any context and domain (e.g. PLM solutions). As described in [5], QuEF is defined taking into account different quality standards. Some of these standards are: (i) ISO 9000 [11–13], which means a basis for performance improvement and organization excellence, since it determines the aspects to improve the methodology and QuEF itself; (ii) ISO/IEC 9126 [14] and ISO/IEC 25000:2005 (SQuaRE) [15], which provide quality characteristics to evaluate a product and lay the foundation for the definition of the Quality metamodel defined by QuEF; and (iii) ISO/IEC 20000 [16–20] and ITIL [21], which define best practices dealing with improving service quality based on quality continuous improvement of the service lifecycle. This framework describes templates to define a specific quality consistent set of business solutions in order to create, manage, control and disseminate all information generated along the lifecycle of a product. This approach allows integrating people involved in product manufacture, manufacturing processes, business systems and information systems. Thus, PLM can be conceived as a business strategy that takes advantage of the latest Information and Telecommunication Technologies with the aim to manage data, processes, methods of work, staff and information systems that take part in the entire product lifecycle. Hence, it is key to provide software solutions to support this management. These software solutions are characterized by centralizing and organizing all data related to product development, providing security mechanisms for having access to information, integrating design processes with manufacturing processes, reusing the know-how among departments of the same company and incorporating software tools that support the PLM strategy (Computer Aided Design, Computer Aided Engineering or Product Data Management, for instance) with other enterprise systems (Enterprise Resource Planning, Customer Relationship Management or Supply Chain Management, among others). This paper aims to study and understand the concept of PLM systems through an in-depth analysis of these systems that will allow defining its scope, domain, functionalities or benefits, among other aspects. It is important to mention that PLM is a broad concept that is sometimes understood as a container of some more sectorial solutions. Thus, it is very complicated to define a standardized feature model due to the heterogeneity it involves. This may be the reason why no standard model was found in the literature. In this context, this paper focuses on defining a solution that allows characterizing PLM systems, in terms of specific features, in large enterprises categorized into the aeronautical industry. Although this solution is framed into the aeronautical context, it is important to mention that it was designed to be as much flexible as possible, thus it is considered to be easily applicable to any other field, while a formal validation of this guess belongs to a future work, according to a company's own nature and enterprise needs (background, internal organization or relative importance that a company gives to each aspect), which are essential. This is mentioned in [4], where authors argue about the key PLM functionalities for the particular context of a Collaborative Ceramic Tile Design Chain. This adaptability is possible because our solution defines a quality model conformed to the Quality Evaluation Framework (QuEF) [5], which establishes methodological guidelines to compare and evaluate entities (e.g., PLM commercial systems). There are many definitions in the literature that try to explain the concept of quality model, but QuEF refers to a set of characteristics and its relationships, which constitutes the base to identify quality requirements and evaluate them. This paper also presents a comparative evaluation of some of the most widespread PLM systems. For this purpose, thanks to the development of a real R&D project carried out between our research group and Airbus, we have defined a quality model based on a set of functional features. Finally, a subject for study and possible future research work is proposed with the aim of defining our quality model following the philosophy of Quality Continuous Improvement proposed by QuEF. The remainder of this paper is organized as follows: after this introduction, Section 2 summarizes some of the most recent work related to PLM systems. Section 3 describes the QuEF in detail and its theoretical foundations. Then, Section 4 explains how the QuEF has been applied to define our quality model and Section 5 defines how this quality model has been instanced to evaluate and analyze each PLM systems. Finally, Sections 6 and 7 state our ongoing work and conclusions, respectively. model for the domain under study. It also offers a method to instantiate the quality model, evaluate it and calculate the preferences of the elements that form it. Besides, the framework includes the definition of a set of phases to enforce quality continuous improvement in the quality model. The most important aspect is that quality management is centred on the quality model. Furthermore, a tool support is also implemented in order to promote this solution in real environments. Therefore, we can have quality management in an automatic way using QuEF, by automating quality management of entities to reduce cost and time, and improve quality in the quality management process. As Fig. 1 shows, the framework can be used from two points of view: (i) providers, who need to analyze, control, evaluate and improve entities; and (ii) consumers, who decide on selecting the most suitable one for them. It differs from other frameworks since it focuses on the quality model and defines a lifecycle where all phases turn around such quality model, as Fig. 2 shows. Moreover, QuEF provides an agile, flexible and efficient solution based on ITIL (Information Technology Infrastructure Library) v3 [21], but with a difference: it does not focus on services, but on a quality model. Similarly to ITIL v3, QuEF is composed of five phases, in order to guarantee quality continuous improvement of the quality model. The aim is to address all quality management efforts to the quality model. As mentioned, it comprises several phases including different objectives and artifacts (Fig. 2): •Quality Model Strategy phase: It is a strategic active that looks at the definition of a quality management strategy. Past, present and future view elements of the quality model in the domain under study are essential to achieve an effective and efficient quality management process. •Quality Model Design phase: This is the phase where the quality model is finally designed in terms of all strategic actives from the previous phase. It is the model used in the following phase for quality management performance. •Quality Model Operation phase: In this phase the quality model is used to carry out the quality management process. Consequently, the analysis and evaluation management processes are performed within this phase. •Quality Model Transition phase: This phase describes the processes that execute changes in the quality model in cases where the domain or the context change due to the appearance of new trends, but without affecting the Operation phase. •Quality Continuous Improvement phase: This phase gathers all mechanisms to improve quality in all processes of the lifecycle and the quality model. An effective and efficient quality management essentially demands that the domain under study be defined. In this context, QuEF does not only stand for ensuring a clear strategy for quality management, but also an automatic quality continuous improvement by means of generating checklists and documentation, as well as automatic evaluations and plans that control and improve quality and thus, automatically, reduce effort and time. For this purpose, the quality management of QuEF is based on a quality metamodel (Fig. 3), which refers to a set of characteristics and its relationships and constitutes the base both to specify quality requirements and evaluate them. The quality model represents the core and quality management revolves around it. We propose a Quality Model metamodel consisting in a simplification and adaptation of ISO/IEC 15939 [22] so that the model instantiation can be more adaptable and practical. The main objective concludes that quality management becomes strategically active. Therefore, all the strategic assets have to be identified and it is necessary to capture, define and validate the quality model that will be used for quality management. A quality model contains Features and Sub- Features (both are categories of an entity's properties). A Feature is a higher-level category of the domain description of an entity, while a Fig. 1. Conceptual scheme representing the goals to be achieved with QuEF. Fig. 2. Quality Management based on the quality model lifecycle [5]. Sub-Feature is a lower-level category. A Property points out the degree to which a Sub-Feature is measured, that means that a Property is used for measuring Sub-Features. Below, different levels for Properties are explained. Then, this metamodel has been instantiated with a set of characteristics that describes all properties a PLM system contains. This system has lots of functions and each function can be a Property. Below, different levels of basic characteristics are explained: •Feature (FT- < Level 1 > ): It is a general concept that involves a set of Properties. It is a higher-level concept of a PLM system that broadly describes it. A Feature includes a set of Sub-Features. •Sub-Feature (FT- < Level 0 > ): It constitutes a specific concept of an entity. It is a set of properties, but a lower-level concept of an entity's characterization. It is used to classify the properties of the entity in two levels (Feature and Sub-Feature), as well as to categorize PLM systems in two levels (Feature and Sub-Feature). •Property: It indicates the degree to which a Sub-Feature is measured by the use of a Metric. Particularly, a Property is used for describing and analyzing the Sub-Features of an entity. It is an element of a PLM system. In simple words, a Property is used for describing and analyzing Sub-Features. Finally, it is important to mention that QuEF shares many principles and values from other methodologies like TQM (Total Quality Management) [23] [23], Six Sigma [24] or CMMI (Capability Maturity Model Integration) [25]. It mainly differs in defining quality only through models. The idea consists in quality management based on model-driven quality that can automate quality management by means of generating artifacts. Another issue to take into account deals with improving quality from quality continuous improvement of a quality model lifecycle defining different phases. QuEF regards quality model management, but not services management. Table 1 shows the relationship among other standards. It also represents the relation among these standards as well as best practices and approaches that have been applied QuEF in order to define it [5]. 4. An approach to characterize and evaluate the quality of PLM systems The PLM system to implement in the company depends on the PLM strategy and plan initially defined according to the objectives to achieve. This implies that each PLM system is specific to each particular business case and its context. In this section, we analyze the components and functionalities that some researchers, consultants in the field and providers of commercial PLM systems catalog as key. We also present some other elements that are emerging and consolidating as part of the PLM spectrum. The lack of a common definition of the PLM concept together with the fact that the processes involved in PLM do not follow a standard imply that these functional characteristics have not been standardized and therefore, each system presents its own functions. Although there are common processes in PLM, such as State-Gate for the development of new products or CMII (Configuration Management II) standards for change management, there are no deeply standardized processes related to innovation and product development, as they are, for example, the accounting ones. Anyway, as mentioned above, it is important to point out that the following features have been chosen in relation to the needs of a large enterprise that belongs to the aeronautical industry suppliers. In addition, it is worth highlighting that each characteristic of our quality model has been assessed in a balanced way, giving the same level of importance to each of them. 4.1. Basic features for PLM systems Table 2 describes a number of functions for the creation and use of product data (product definition) that are the bases of most commercial software tools or PLM systems that support the core business processes of the company [26]. 1. Field state control: The PLM system can automatically monitor both the status of files or attachments and their status in the lifecycle. 2. Creation of objects: The creation of a new document, a component part of a product or the approval of a purchase order for a particular component is typically performed in manufacturing enterprises. 3. Distribution management: It is implemented in different situations, for example, when the approved documents are distributed in a process using workflows. The PLM system systematically distributes them according to the workflow process and the principles defined in the software. Extra information necessary for a specific document can be attached to the structure of the product, so it is available when required as a reference. This allows users of PLM systems to administer a lot of information in a simple way. 4. Information search and retrieval: These are some of the main functions of PLM systems. According to Kenneth McIntosh, in production industries engineers invest between 15 and 40% of their days searching and rescuing routines from independent information systems. Information searches are possible thanks to the classification of the same attributes, and also by creating sources of aid or by describing each element and allowing the system to analyze the information of the element of each system. This system lets the user study the contents of the documents that have been classified as the same type, although the content of each document does not exactly match the search criteria. 5. Product structure management and maintenance: It is one of the most important features in the PLM system because it lays the foundation for other core functions. Some properties of version management, structural presentation of information and change management, as well as configuration management are generally based on the management of the product structure. 6. Change management: This is another key issue for PLM. Companies need to classify the definition of each update as changes that are made to product design. Managing product changes ensures that they are carried out in a clearly defined, documented and controlled manner throughout the product lifecycle forms. 7. File transfer and conversion: File transfer and conversion among applications of the system are arranged so that the developer, user or reader does not need to know the current location since the usage environment can be a LAN (Local Area Network), WAN (Wide Area Network) network or the Internet. The file is retrieved, automatically converted and opened in the appropriate application. 8. Communication and messages, and tasks management: They represent a cornerstone that concurrent engineering PLM systems must support. The system manages all the messaging so that the relevant information that may affect their work can reach all users Fig. 3. Quality Metamodel on QuEF. in case they are required an action. Moreover, the system provides a communication forum for daily work. 9. Physical documentation management: It involves the ability to digitalize documents through scanning so that they can be treated by the system. 4.2. Description of functionalities / features of PLM systems in relation to their applications or software tools This section presents a description of the different features or modules of PLM software environment. •Data/ Document management. These applications let the company store and make data available along the product lifecycle by means of a security access controlled in a distributed environment. It allows the version management, review management, classification, searches, analysis and reports [3]. •Part/ Product/ Configuration management. These applications allow the company to manage products, structures and attributes of the product along the lifecycle by means of a security access controlled in a distributed environment. It enables improvement of designs and components as well as modules reuse [3]. •Process/ Workflow management. According to the Workflow Management Coalition (WfMC), workflow represents the automation of a business process, in whole or part, during which documents, information or tasks are passed from one participant to another for action, according to a set of procedural rules [27]. In this sense, applications for PLM workflows management enable the company to automate simple workflows and ensure compliance with the legal requirements of organizations, such as ISO. •Program/ Project management. These applications allow the company to plan, direct and control projects and programs. They enable the supervision of stages, gates, milestones and deliverables, making it possible to learn about the state of a project in terms of progress. •Collaboration management. These applications allow dispersing geographically teams and other users to work in an integrated, secure and structured way within a virtual workspace where the information of the product is already updated. •Visualization. These applications enable the visualization of different kind of documents and files, regardless of the tool, version or operative system where they have been created for the user in an abstract way. •Integration applications. These applications allow integration among tools that give support to the different lifecycle phases of the product, both physical and non-physical. Some standards, like ISO 10303 and STEP (Standard for the Exchange of Product Data), are available in order to achieve this goal [28]. •Infrastructure management. These applications allow the Work context Standard model, best practices and approaches Work application QuEF and Specific domain (WA*) ISO 9000 standards This standard represents a basis for performance improvement and organization excellence. It determines the aspects to improve MDWE (Model-Driven Web Engineering) methodologies and QuEF. Specific domain (WA*) ISO/IEC 9126, ISO/IEC 25000:2005 (SQuaRE) These standards provide Quality Characteristics to evaluate a product. These ISO standards define aspects to be evaluated. They lay the foundation for the definition of Quality Characteristics so as to evaluate MDWE methodologies in QuEF. These methodologies are assessed as self-products, although the stakeholders’community must agree on Quality Characteristics of MDWE in the Strategy Phase of QuEF. ISO/IEC 15504 MDWE methodologies define processes and techniques to develop Web Applications. This standard determines the computer software development process and related business management functions. QuEF ISO/IEC 20000, ITIL ISO/IEC 2000 standard and ITIL best practices deal with improving service quality based on quality continuous improvement of the service lifecycle. For instance, ITIL defines a Strategy phase, a Design phase, an Operation phase, a Transition phase and a Quality Continuous Improvement phase. QuEF covers the same idea with a different goal, since the QuEF framework manages quality based on quality continuous improvement of the quality model lifecycle. TQM, Six Sigma, CMMI, Planguage, C- INCAMI or CTQ, among others. QuEF defines different phases with artifacts, methods and tools for each phase. Most of these approaches could be adapted and applied to some phases of QuEF. They cover the similar aspects between QuEF and quality management Strategy and Operation phases. For instance: - TQM is a management integrative philosophy that aims at continuously improving quality of products and processes. It could be applied in the Strategy and Quality Continuous Improvement phases within QuEF. - Six Sigma is a business process management strategy very similar to TQM working with many established quality-management tools. Most of them could be used in the Strategy and Operation phases of QuEF. - CMMI is a process improvement approach that intends to help organizations implement their performance. Therefore, it could be applied to the Strategy and Quality Continuous Improvement phases. - Planguage could be applied to the Strategy phase of QuEF for defining quality. - C-INCAMI provides a domain (ontological) model defining all the concepts and relationships needed to design and implement processes. Hence, it could be used in the Strategy and Operation phases. CTQ could also be applied to the Strategy phase, the Design Phase and the Operation Phase of QuEF for defining project context, non-functional requirements, measurement, evaluation and analysis. ⁎ WA (Methodologies for modeling web applications). Table 2 Basic features of a PLM system. Functions of a PLM system 1. Field state control 2. Creation of objects 3. Distribution management 4. Information search and retrieval 5. Product structure management and maintenance 6. Change management 7. File transfer and conversion 8. Communication and messages and tasks management 9. Physical documentation management Table 1 Related work application and context in QuEF [5]. management of services that support infrastructures like networks, databases and servers. •Idea management. These applications favor the capture of ideas for analysis, taking appropriate actions and monitoring the progress of the design phase of a product. •After sales/ Product feedback management. These applications enable to capture client's data or feedback for a further analysis that will help us identify needs, business opportunities and products improvement. •Product portfolio management. These applications assist the review, analysis, simulation and evaluation of the company products portfolio, consisting of existing products as well as development of new lines of products. This functionality can show estimation of sales and reuse, at the same time that it assesses the impact of decisions, such as introducing new technologies, acquisitions and launch of joint ventures. •Idea generation management. These tools support the generation of ideas, defined as the process of creating, developing and communicating abstract, concrete or visual ideas. The process comprises building the concept from the idea, self-innovation and the actual realization of the concept. •Requirements and specifications management. These applications allow the company's systematic processes to collect, analyze, communicate and manage product requirements that describe the needs of the market and customers. They provide the company with the way to systematically manage product specifications in a standardized manner. •Collaborative product definition management. These applications allow the definition of products for people who are members of different teams, who are in different geographical locations or who Table 3 Common features in the definitions of PLM. Table 4 Table for QuEF. Basic/ core components of a PLM system Available characteristics Yes Partially No Extension components of a PLM System Available characteristics Yes Partially No Concepts Design Production Support/ Use General work in different companies. •Supplier and sourcing management. These applications enable procurement teams to collaborate with other teams, partners and external suppliers for various activities, such as revision, selection and purchasing of parts and components. •Manufacturing management/ Manufacturing process management. These applications allow production equipment to simulate, optimize and define processes as well as understand the relationships among product, plan and manufacturing processes [3]. •Digital manufacturing. These applications contribute to analyze how the maintenance of the product is and how it happens, helping manufacturing engineers create the complete definition of a manufacturing process in a virtual environment (tools, assembly lines, work centers, design facilities, ergonomics and resources). •Maintenance manufacturing. These applications give assistance to support and maintenance teams to streamline processes, gain better customer feedback and carry out activities more efficiently, with a better management of inventories of parts, components and equipment. •Compliance/ Environment, health and safety management. These applications allow the development and management of business processes to comply with the regulation on environmental pollution standards, such as EFSA (European Food Safety Authority) and ISO (International Organization for Standardization), in addition to ensure that these regulations will be followed throughout the product lifecycle. •Intellectual property management. These applications allow the assessment and management of the Intellectual Property (IP) of the company represented by products and services. Some of their functions include the management of inventions and patents through an updated control of their states, administration of trademarks, trade names, contracts and agreements related to IP with monitoring compliance obligations, traceability or associated expenses, for instance. •CAD (Computer aided design). It is a comprehensive term of graphical tools that helps the transformation of requirements or conceptual model into a design. The core of these applications are 2D drawing and 3D modeling tools that support the four stages of design in CAD geometric modeling; analysis and design optimization, design review and evaluation, documentation and drawing (drafting). •CAE (Computer aided engineering). It includes all the tools to analyze and simulate engineering designs made with computers, or otherwise designs created and entered into the computer to assess their characteristics, properties, feasibility and profitability. Most of these tools are modules or extensions of CAD applications. •CAM (Computer aided manufacturing). It includes all applications that are used in the activities of production engineering. They provide a bridge between CAD and the programming language of machines / tools, by seeking maximum automation with minimal Basic/ core components of a PLM System Dassault Enovia V6 Siemens TeamCenter V9 PTC WindChill 10 Trace One CATIA PLM ARAS Oracle Agile PLM Requirements and specifications management Yes Yes Yes Yes Yes Yes Yes Data/ document management Yes Yes Yes Yes Yes Yes Yes Part/ production / configuration management Yes Yes Yes Yes Yes Yes Yes Process/ workflow management Yes Yes Yes Yes Yes Yes Yes Program/ project management Yes Yes Yes Yes Yes Yes Yes Collaboration management Yes Yes Yes Partially Yes Yes Yes Visualization Yes Yes Yes No No Yes Yes Integration applications Yes Yes Yes No No Yes Yes Product portfolio management Yes Yes Yes No Yes Yes Yes Collaborative product definition management Yes Yes Yes Partially Yes Yes Yes Supplier and sourcing management Yes Yes Yes No Yes No Yes Manufacturing management DelmiaV6 TecnoMatrix Yes Yes Yes Yes Yes Maintenance/ service management Yes Yes Yes No Yes Yes No Extension components of a PLM system Dassault Enovia V6 Siemens TeamCenter V9 PTC WindChill 10 Trace One CATIA PLM ARAS Oracle Agile PLM Concepts Idea management No No No No No No Yes Idea generation management No No No No No No Yes Design Compliance, environment, health and safety management Yes Yes Yes Partially Yes Yes Yes Product analysis, validation and simulation SimuliaV6 TecnoMatrix PTC Creo No Yes Yes Yes Authoring Tools: CAD, CAE, CAM, ECAD, CASE…CATIA/SW NX/SEdge PTC Creo No Yes Yes Yes MultiCAD management Yes Yes Yes No Yes Yes No Software development Partially Partially No No Yes Yes Yes Technical documentation Yes Yes Yes No Yes Yes Yes Technology planning Partially Partially No No No No No Production Digital manufacturing SimuliaV6 TecnoMatrix No No Yes Yes No PLC programming Delmia TecnoMatrix No No Yes No No Support/ Use After sales management Partially Partially Partially No No Yes No Marketing 3DExcite Mar No No No No No No General Intellectual Property management Yes Yes Yes Yes Yes Yes Yes Quality lifecycle management Yes Yes Yes Yes Yes Yes Yes Communities of practice Yes Yes Yes Yes Yes Yes Yes Infrastructure management Yes Yes Yes Yes Yes Yes Yes Distribution Partially Partially No Yes No Yes AVM ⁎⁎ *In some cases, it is known by the name of the tool or application of the provider that renders support to the field, but it is not a part of the analyzed suite. Therefore, the value for the study will be “No”in the specific characteristics. ⁎⁎ Agile variant management. Table 5 Evaluation of PLM systems of different companies from the proposed model. operator intervention. Numeric Control (NC) Programming and Programmable Logic Controller (PLC), for programming in machine code of the tools, and Computer Aided Production Engineering (CAPE), for helping Production engineering, among others, are included in these applications. •Components supplier management / CSM. These applications are designed to identify and address parts and components that can be reused to reduce costs, once the list of suppliers and approved manufacturers is defined, and also to develop a more accurate list of the company's materials. •Technology planning. According to the Product development management association, technology planning is defined as: "the process of acquiring knowledge that can then be used in developing new products to meet market needs. This process results in a concrete action plan to take advantage of new and existing technologies in line with the company strategy and customer needs" [29]. •PLC programming. A Programmable Logic Controller (PLC) is a computer used in the automated engineering or industrial automation to mechanize electromechanical processes, such as control of machinery on factory assembly lines processes [30]. These applications can debug PLC code in a virtual environment before downloading the actual equipment. By simulating and validating the automation equipment, it can almost be confirmed that it will work as expected, significantly reducing the start-up time. •Product analysis, validation and simulation. Applications of simulation and validation are used to study the performance of the product before they physically occur. Normally, simulation involves the development of a product model and the development of a model of the environment where the simulation is operated to check the performance of the product or component. This will identify and correct errors before it is released to production and recommendations for improvements are made. These applications reduce development time, because they avoid construction activities of both physical models of the product/component as well as the environment. •Technical documentation/ Technical publication. These applications allow users to create, manage, publish and deliver technical publications, such as a User Manual [3]. •Multi-CAD management. The management of CAD data from different tools or disciplines, also called Multi-CAD management, assumes that design teams can generate a unique definition of the product - a single source of truth - clear enough to be released to production with confidence. •Quality / CAPA management. These applications support the systematic management of quality, reliability and product risk, using methods that are integrated into the product development phase and are visible to all parties interested in quality. •Communities of practice. They are named social groups that are formed in order to develop specialized knowledge, sharing learning reflections based on practical experiences. Fig. 4. Graphical representation of Design assessment features. •Distribution. These applications are used for the design, development and manufacture of packaging or promotional displays, for instance, and they mainly focus on the field of consumer packaged goods. Aspects like building a brand, value through packaging or sustainable distribution, among others, are managed with these applications [30]. •Marketing. This is one of the characteristics or functions present in a less mature PLM environment, where environmental agents have recently detected an increased demand interaction between the area of design and the area of marketing [31]. These applications enable design and marketing departments share information about developing products at an early stage. Thus, global marketing teams have access to products display when they are in the development stage, allowing adjusting them to local needs. These applications also favor planning and launching marketing campaigns before the product is actually built (e.g. digital technology eliminates the need for real photos and videos). Interactive marketing campaigns can be launched to engage the customer at an early stage by means of this technology. •PDM (Product data management). PDM applications support activities along the product lifecycle: design, dissemination of information among multiple users, traceability of engineering changing orders, management of alternative designs and control of product configurations. 4.3. Proposing a model of characterization for PLM systems This section proposes the quality model for the characterization of PLM systems based on QuEF. Table 3 summarizes all the different PLM system characterizations that have been analyzed according to some of the largest companies of PLM (such as Start, PTC, ARC, AMR, Atos, Techguide and CIMData) with the objective of having a visual guide for the analysis of the main and secondary components. It also highlights the common constituents that are listed in most of the definitions above and are understood as recurring elements in most deployments of PLM. Therefore, we can observe a first group of applications that comprise part of PDM: Data Management, Configuration Management, Workflow Management Data Management, Configuration Management and Workflow Management (a set that implicitly consists in Change Management). Furthermore, it is important to include: Requirements and Specifications, Collaborative Product Definition, Supplier and Sourcing, Manufacturing and Maintenance Management. Besides, the integration with other applications must be specified. It does not explicitly appear in other definitions, but it is a fundamental element of PLM, as it was mentioned before. Finally, in Table 4, the proposed quality model for PLM systems characterization is introduced based on the information studied, which allows the evaluation of different PLM software solutions. 5. Evaluation and analysis In this section, the first iteration of the QuEF methodology is carried out. In this line, we have analyzed different PLM system providers: Enovia V6 of Dassault Systèmes, Siemens TeamCenter V9, PTC WindChill 10, Trace One PLM, CATIA PLM, ARAS and Oracle Agile PLM. Table 5 shows the results of this evaluation according to the characterization model defined in the previous section. The fields of Manufacturing Management, Product Analysis, Simulation and Validation, Digital Manufacturing, PLC Programming and Marketing stand for the name of the suite or application where the provider supports this particular feature, although the evaluation of the field in this suite does not have such a feature. Fig. 5. Graphical representation of Production and Support assessment features.