Accelerating a technology commercialization: With a discussion on the relation between technology transfer efficiency and open innovation

Sutopo, Wahyudi; Astuti, Rina Wiji; Suryandari, Retno Tanding Article Accelerating a technology commercialization: With a discussion on the relation between technology transfer efficiency and open innovation Journal of Open Innovation: Technology, Market, and Complexity Provided in Cooperation with: Society of Open Innovation: Technology, Market, and Complexity (SOItmC) Suggested Citation: Sutopo, Wahyudi; Astuti, Rina Wiji; Suryandari, Retno Tanding (2019) : Accelerating a technology commercialization: With a discussion on the relation between technology transfer efficiency and open innovation, Journal of Open Innovation: Technology, Market, and Complexity, ISSN 2199-8531, MDPI, Basel, Vol. 5, Iss. 4, pp. 1-28, https://doi.org/10.3390/joitmc5040095 This Version is available at: https://hdl.handle.net/10419/241362 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ Journal of Open Innovation: Technology, Market, and Complexity Article Accelerating a Technology Commercialization; with a Discussion on the Relation between Technology Transfer Efficiency and Open Innovation Wahyudi Sutopo 1,* , Rina Wiji Astuti 2and Retno Tanding Suryandari 3 1Research Group of Industrial Engineering and Techno-Economic, Department of Industrial Engineering, Universitas Sebelas Maret, Surakarta 57125, Indonesia 2 Teaching Factory of LFP Battery Universitas Sebelas Maret, Surakarta 57125, Indonesia; [email protected] 3Department of Management, Universitas Sebelas Maret, Surakarta 57125, Indonesia; [email protected] *Correspondence: [email protected] Received: 19 September 2019; Accepted: 19 November 2019; Published: 22 November 2019   Abstract: Commercialization strategy is an all-encompassing plan that organizes technology transfer office goals to commercialize a university’s technologies. Measurement strategy requires feasible variables that make up those goals. This strategy also ensures that all variables that are important in measuring contribute to the larger goals. A useful way to assess and explain the effectiveness of the technology transfer office (TTO) of universities is to model this within a production function/frontier framework. Such a production function is typically estimated econometrically. This study presents evidence on the relative efficiency of research commercialization in the university through the data envelopment analysis (DEA) model. The implication of the DEA efficiency result is to derive the efficiency level of the TTO’s strategy from the observed performance. It also helps in identifying the benchmarking of other TTOs, which would be valuable information for improving their new technology commercialization strategy. In detail, a benchmark is provided to improve the weakness of strategy and resource allocation of a poorly performing TTO. The proposed matrix of indicators is an exploit of how performance could be measured within the decision-making units that have been chosen. By introducing the measure to commercialization strategy framework the development of technology transfer offices policies are considered. Keywords: data envelopment analysis (DEA); efficiency strategy; performance measurement; technology commercialization; technology transfer office (TTO) 1. Introduction The shifting paradigm in university can be spotted from the difference between the goal achievement that should be taken then and now [ 1 ]. In the past, universities were aiming to achieve goals compiled in the Three Pillars of Higher Education, which are education and teaching, research and development, and public services. On the other hand, the university’s new paradigm is recently dynamically developing. In addition to obligating the Three Pillars of Higher Education, universities are also obligated to conduct some actions in autonomy status enhancement, economic development, and research output commercialization in order to improve the quality and competitiveness of a university in both national and international scope [ 2 ]. Every university has its vision, mission, and financial management in conducting an obligation to commercialize their research output. Universities are aiming to take a more active role in developing domestic economy power by participating in the development of science and technology-based business and industry to accelerate the commercialization of new technologies and promote economic development. J. Open Innov. Technol. Mark. Complex. 2019,5, 95; doi:10.3390/joitmc5040095 www.mdpi.com/journal/joitmc J. Open Innov. Technol. Mark. Complex. 2019,5, 95 2 of 25 Technology commercialization is a means to exploit technology resulted from research in either production or consumption activity so that the researcher can gain profit from the activity [ 3 , 4 ]. In most cases, many technology products resulting from research happened to fail to be launched to the market due to the valley of death [ 5 ]. This obstacle usually occurs in the transition process between technology development and technology commercialization. Hence, critical action is needed to accelerate the technology commercialization in order to ensure the commercialization potency of research output does not fall into the valley of death. A university is expected to take part in this process in order to accelerate the transfer of new technology products to the market [6,7]. As a result of this legislation, almost all research universities in the world established technology transfer offices (TTOs) to manage new technology commercialization [ 8 – 14 ]. A technology transfer office (TTO) is a kind of organization to assist research organizations in managing their intellectual assets in ways that facilitate their transformation into a benefit for society [ 15 ]. The general roles of TTO include establishing relationship with firms and community actors, generating new funding support from sponsored research or consulting opportunities, providing assistance on all areas related to entrepreneurship and intellectual property (IP), facilitating the formation of university-connected companies utilizing university’s technology (start-up) and/or university resource (spin-off) to enhance prospect or further development and generating net royalties for the university’s technology and collaborating partner. Strategies for commercializing university technology are formulated by the technology transfer office (TTO). Strategies that can be done by the TTO in carrying out its role, among others, are to have the physical facilities to support technology commercialization, mentoring and coaching activities, marketing, and business networking, financial support, and internal university regulation itself. The efficiency of strategies that have been executed by the technology transfer office (TTO) in each university needs to be measured. As an attempt, a performance measurement method that can provide university efficiency information is required. The efficiency measurement result can be later used as a reference for other higher education institutions to formulate strategies regarding the commercialization strategy of research output. Universitas Sebelas Maret (UNS) is one of the universities in Indonesia that has established a technology transfer office (TTO) to manage research products produced by academics from universities. One technology product that has been developed by the university is a lithium LiFePO 4 (LFP battery). Some of the research that has been produced include References [ 16 – 18 ]. The university’s TTO has a business unit to produce and commercialize LFP batteries. The business unit has not been pioneered recently, therefore the TTO is still making continuous improvements to commercialize the product optimally so that it can compete in the market, and the technology product does not fall into the valley of death. In this study, we developed a framework to measure the efficiency of technology commercialization strategies from TTOs at four universities in Indonesia. The measurement results can be used as a benchmark to develop the best strategy for commercializing LFP batteries as a result of university research. A measurement strategy is an all-encompassing plan that organizes technology transfer office goals to commercialize university’s technologies and how it will be measured (Figure 1). Measurement strategy requires feasible variables that make up those goals. This strategy also ensures that all-important variables in measuring contribute to the larger goals. A useful way to assess and explain the effectiveness of a TTO is to model this within a production function/frontier framework. Such a production function is typically estimated econometrically. Production frontiers are also estimated using nonparametric models, which offer some advantages, relative to the parametric approach. For instance, these methods obviate the need to specify a functional form for the production frontier and also enable us to identify “best practice” technology transfer offices. Nonparametric techniques can also handle multiple outputs. Performance measurement is a structured process through which a technology transfer office identifies, measures, and monitors essential programs, systems, and processes [ 19 ]. A commercialization strategy is the strategy revolving around the commercialization function of the technology transfer J. Open Innov. Technol. Mark. Complex. 2019,5, 95 3 of 25 office and is connected to the overall university strategy. The performance measurement system for the commercialization strategy begins with the university strategy input and the grouping of roles. The roles grouping is based on the goals that exist for a unit within the university. Based on the grouping and the requirements of stakeholders, two types of objectives can be distinguished, namely business objectives and objectives regarding the key performance drivers [ 20 ]. Objectives regarding the key performance drivers address how the university objectives are to be met in more detail. These objectives can also address changes in the value creation process or the overall technology transfer office set-up. Once the objectives are defined, the actual definition of performance measures and performance measurement itself can begin [21]. J. Open Innov. Technol. Mark. Complex. 2019, 5, x FOR PEER REVIEW 3 of 28 Figure 1. Measurement strategy. Measurement strategy requires feasible variables that make up those goals. This strategy also ensures that all-important variables in measuring contribute to the larger goals. A useful way to assess and explain the effectiveness of a TTO is to model this within a production function/frontier framework. Such a production function is typically estimated econometrically. Production frontiers are also estimated using nonparametric models, which offer some advantages, relative to the parametric approach. For instance, these methods obviate the need to specify a functional form for the production frontier and also enable us to identify “best practice” technology transfer offices. Nonparametric techniques can also handle multiple outputs. Performance measurement is a structured process through which a technology transfer office identifies, measures, and monitors essential programs, systems, and processes [19]. A commercialization strategy is the strategy revolving around the commercialization function of the technology transfer office and is connected to the overall university strategy. The performance measurement system for the commercialization strategy begins with the university strategy input and the grouping of roles. The roles grouping is based on the goals that exist for a unit within the university. Based on the grouping and the requirements of stakeholders, two types of objectives can be distinguished, namely business objectives and objectives regarding the key performance drivers [20]. Objectives regarding the key performance drivers address how the university objectives are to be met in more detail. These objectives can also address changes in the value creation process or the overall technology transfer office set-up. Once the objectives are defined, the actual definition of performance measures and performance measurement itself can begin [21]. Strategy, in general, is “a pattern in a stream of actions.” A strategy is understood as a statement and realization of pre-defined actions as well as involved consistencies in action [22]. A strategy needs to be deliberately implemented before it becomes a realized strategy. The research focus is the intended strategy and its translation is the realization. The development of performance measurement with its various sub-streams and aspects has grown to be increasingly complex. It will continue to grow in complexity as the scope of performance becomes increasingly diverse [23]. The result of performance measurement should be focused on learning and understanding rather than sole control and depends on an understanding of the own era of the university and its stakeholders [24]. Performance measurement gives recommendations about building blocks for commercialization strategy. They can be split up into recommendations for performance measures and recommendations for performance measurement framework and system design [25]. The development and implementation of measuring the efficiency of strategies for commercializing university technology have provided guidelines for efficiency judgment measurement; they are the commercialized product, technology-based start-up, joint venture, license, and increased employment. Figure 1. Measurement strategy. Strategy, in general, is “a pattern in a stream of actions”. A strategy is understood as a statement and realization of pre-defined actions as well as involved consistencies in action [ 22 ]. A strategy needs to be deliberately implemented before it becomes a realized strategy. The research focus is the intended strategy and its translation is the realization. The development of performance measurement with its various sub-streams and aspects has grown to be increasingly complex. It will continue to grow in complexity as the scope of performance becomes increasingly diverse [ 23 ]. The result of performance measurement should be focused on learning and understanding rather than sole control and depends on an understanding of the own era of the university and its stakeholders [ 24 ]. Performance measurement gives recommendations about building blocks for commercialization strategy. They can be split up into recommendations for performance measures and recommendations for performance measurement framework and system design [ 25 ]. The development and implementation of measuring the efficiency of strategies for commercializing university technology have provided guidelines for efficiency judgment measurement; they are the commercialized product, technology-based start-up, joint venture, license, and increased employment. Regarding commercialization strategy, goals for efficiency and the entire commercialization process are derived from the overall strategy and the commercialization function environment. Therefore, the performance measurement of commercialization strategy is defined as the degree of fulfillment of the commercialization strategy set for the technology transfer office in the university while considering the influence of contextual output measurement. Methods dealing with the efficiency evaluation are generally based upon the estimation of a production frontier. They can be broadly classified into two primary groups as parametric and non-parametric approaches [ 26 ]. Parametric frontiers rely on specific functional form and can be either deterministic or stochastic [ 27 ]. Data envelopment analysis (DEA) is an example of a performance measurement method which can represent relative efficiency from several decision-making units J. Open Innov. Technol. Mark. Complex. 2019,5, 95 4 of 25 (DMUs) based on multi-criteria (input and output). Decision-making units (DMUs) are a group of entities that will be analyzed. Due to the comparison principle, a DMU must have the same goal and target as well as universal input and output [ 28 ]. This method works by determining an efficient DMU according to input and output criteria, then calculating relative efficiency from a respective DMU towards an efficient DMU. Several research models have been developed for performance measurement by using data envelopment analysis (DEA) as used by Jyoti, Banwet and Deshmukh [ 29 ] to measure relative efficiency from a public research organization. A similar study has been conducted by An-Yuan et al. [ 30 ] discussing the approach of production flexibility measurement. A study has been performed about the utilization of DEA in the electronic industry performance measurement in Taiwan [ 31 ]. The data envelopment analysis (DEA) method is also used in a study conducted by Joseph, Sandra, and Haiyan [ 32 ] related to bank merger efficiency in the national scope. While Chen-Ta et al. [ 33 ] conduct performance measurement research using DEA application to measure stock performance, Jelena and Alemka [ 34 ] have another study discussing government performance measurement in the Republic of Croatia by using data envelopment analysis. Meanwhile, Tanase and Morar [ 35 ] use DEA to analyze performance in the machinery industry. A study regarding DEA utilization in efficiency and effectiveness measurement of supplier selection was also conducted by Tavassoli, Faramarzi and Saen [ 36 ]. Another study is efficiency measurement using time-series by Silva et al. [ 37 ]. Research by Jian and Dai [ 38 ] use the DEA method to measure the efficiency with output uncertainty. A comparison study was conducted by Sahney [ 39 ], which talks about the performance measurement of a university in India while Huang, Nin Ho, and Chen [ 40 ] measure the efficiency of marketing strategy in Taiwan. Thore [ 41 ] study the use of DEA analysis methods in the process of innovation and commercialization in the scope of industry, universities, and countries. Eilat, Golani and Shtub [ 42 ] propose and demonstrate an efficient, effective, and balanced methodology for the development and analysis of R&D projects. Wang and Huang [ 43 ] evaluated the relative efficiency of R&D activities in several countries. Sueyoshi and Goto [ 44 ] integrate DEA and discriminant analysis to test whether R&D expenditures affect industry financial performance, and years later, they assess the importance of R&D expenditures in the information technology industry in Japan [ 45 ]. Liu and Lu [ 46 ] introduce a network-based approach, which is a new method to increase discriminants in DEA. Zhong et al. [ 47 ] evaluate the relative efficiency of 30 regional R&D investments, and Shirouyehzad et al. [ 48 ] employ the DEA method to measure labor efficiency while Chun et al. [ 49 ] analyze the productivity of R&D and commercialization activities at the company level. This paper presents a framework for measuring the efficiency of strategy for commercializing a university’s technology. We linked strategy to commercialize the university’s new technology and strategic conditions facing a university. The framework gives a contribution to how commercialization strategy in a university that is supported by the technology transfer office must be measured. Our analysis has suggested how to choose an efficient strategy for the technology transfer office to commercialize a university’s technology. 2. Literature Review In Section 1, we have discussed the background of this study and the factors that might affect commercializing research outputs in university. In Section 2, we give a literature review. In Section 3, we discuss an approach to develop, and then in Section 4results and in Section 5discussion. In the final section, we describe the Conclusions and Implications. 2.1. University Paradigm Three aspects indicate the changing paradigm of the university. The first aspect is the evolution of the university itself, then a comparison between the current and past higher education roles, and the last indication is the definition of innovation and research. The evolution of higher education can be divided into five categories, which are Middle Ages European and Chinese Imperial Era, where classically J. Open Innov. Technol. Mark. Complex. 2019,5, 95 5 of 25 university can be seen either elitist, royal, aristocrats, or bureaucrats [ 50 ]. In the 1890s, university took a role as a studying place for the public, then shifted in the 1930s, and eventually, after World War II became a research center [ 1 ]. In the 1990s–2000s universities added entrepreneurial value inside, and nowadays, higher education in Korea, India, and China call themselves agents of change [2]. The new university paradigm is seen from three roles of a university, which are education—teaching and learning activity which produce educated human resources, research, and public service-transfer of science and technology from university to community interest [ 51 ]. Meanwhile, the future paradigm in a university that is dynamically shifting possesses not only the Three Pillars of University but also autonomy status, economic development, and commercialization of research output in a university. By fulfilling those aspects, universities are expected to increase cooperation with industry, technology transfer as a university revenue source, and development of entrepreneurs, which is purposing to the downstream research output of the university. 2.2. Technology Commercialization Nlemvo [ 52 ] stated that commercialization consists of design, development, manufacturing, startup marketing phase, and everything regarding product development. While Siegel and Marconi [ 53 ] argued that commercialization is an activity to transform and set technology to the beneficial or profitable point. Shane and Stuart [ 54 ] also define that commercialization involves identification of consumer need, product concept design, product design, and prototyping process until the manufacturing process in the end. Parker and Mainelli [ 55 ] mention two points where the technology may generate profit. The first commercialization encounter is when technology invented from scientific research succeeds in creating a license. The second phase happens when technology is transformed or applied in a particular product through product development activity in a company so that it can be used in advanced production or consumption activity. 2.3. How Does Research Output Commercialization Become So Urgent? Commercialization of technology or research output can be described as an activity to bring new technology from the inventor’s institution to market (in product or service form). New technology commercialization is defined as “change the idea to money”. The transfer of new technology to market can be executed by the institution/company itself or other party or the inventor’s institution cooperating with another party. Money earned from technology commercialization is intended to ensure the sustainability of the product and service production, which exploit the technology. In other words, the utilization of a particular technology that is not beneficial will not be able to produce a sustainable product and service. As a result, various subsidy programs provided for a particular program or activity will not be sustainable. However, new technology commercialization aims are much broader than to generate money or profit; instead, it leverages industry competitiveness, which ends in an increase in public wealth. This new technology commercialization brings positive impact through the creation of several aspects such as job opportunity, economic development, global and national competitiveness enhancement, increasing of company revenue and profit, tax for the country, and development funds for public wealth. Commercialization of research output is a form of technology transfer from universities to markets. Technology transfer has become a pillar of open innovation with the acceleration in the economic and the digital industrial era as a result of the current industry 4.0. There have been several studies regarding open innovation in the current fourth industrial revolution. Yun et al. [ 56 ] propose a sustainable open innovation strategy and an approach to sustainable serial entrepreneurship. Besides, Yun et al. [ 57 ] examine the relationship between collective intelligence crowd innovation and open innovation. Yun and Liu [ 58 ] identify microand macro-dynamics of open innovation in addition to the dynamic roles of industry, government, university, and society. Yun et al. [ 59 ] analyze the difference between the role of a business model in the converted industry and the emerging industry. Yun, Won and Park [ 60 ] address entrepreneurial cyclical dynamics of open innovation. Yun et al. [ 61 ] study the effect of open innovation of technology value and technology transfer in the automotive, robotics, and aviation industries. J. Open Innov. Technol. Mark. Complex. 2019,5, 95 6 of 25 2.4. Technology Transfer Office (TTO) Role in Commercialization Understanding of technology commercialization route can be beneficial for an organization with the primary function of commercializing new technology, also known as a technology transfer office (TTO). For instance, that scheme can be a planned financial power source and human resource in the future strategy. The intellectual properties (IP) commercialization process of R&D institutions, as illustrated in Figure 2, shows the presence of an institutionalized commercialization process or activity scheme. Through these TTOs, IP can be given a commercialization facility by receiving complete help from the institution. This commercialization process also necessitates researchers to reveal their potential research output so that its IP will be protected and will leverage the institution’s IP portfolio. The portfolio will be evaluated afterward by a team in a technology transfer or commercialization office in order to program the commercialization plan and required budget or investment allocation. The commercialization route indicates that the establishment of a new commercialized technology-based startup can be one of the commercialization targets itself besides the number of licenses. J. Open Innov. Technol. Mark. Complex. 2019, 5, x FOR PEER REVIEW 7 of 28 Figure 2. Science–technology transfer and innovation diffusion process. 2.5. Performance Measurement Performance measurement is a quantifying process of efficiency and effectiveness of a particular action [62]. Effectiveness represents how much the targeted goals have been achieved. Meanwhile, efficiency refers to the speed of doing a task or how a budget can adequately cover the task. Poister [63] defines performance measurement as a process to define, monitor, and use an objective indicator of organization performance. Following the definition, the performance measurement process involves activity in determining, defining, and using indicators to monitor an organization’s performance. Monitoring is used to figure out how effective and efficient an organization is in achieving its goals. Performance measurement is a medium to improve organization performance. An organization applying a performance measurement is likely to possess better performance rather than an organization which is not [64]. Due to performance measurement, organizations can monitor improvement and motivate themselves to obtain their goals. According to Moxham [65], performance measurement is seen as a tool to get improvement in the public service sector and an essential factor in the reformation of that sector. 2.6. Data Envelopment Analysis (DEA) Data envelopment analysis (DEA) is a method of linear-based programming to evaluate the efficiency of organization unit performance, also known as decision-making units (DMUs) [66]. A DMU can be a group of firms, departments, divisions, or administrative units with a common goal and target as well as the input and output [28]. DEA aims to measures how efficient a DMU is using its resources in producing several outputs [67]. Data envelopment analysis (DEA) has two basic models, which are the Charnes, Cooper, and Rhodes (CCR) model and Banker, Charnes, and Cooper (BCC) model. By its name, the CCR model is developed by Charnes, Cooper, and Rhodes [67], while the BCC model is constructed by Banker, Charnes, and Cooper [68]. The two models differ in assumption. The CCR model assumes that increasing the input value will impact the output value proportionally, in other words: constant return to scale. Therefore, the output and input ratio will always be constant. Unlike the CCR model, the BCC model assumes that the increase of input value will impact in increasing an output value disproportionately or variably. Hence, increasing the input of “x” times does not always mean Figure 2. Science–technology transfer and innovation diffusion process. There is a need for intermediation between a technology owner and user or other parties that tend to produce a product and service from the technology, as seen in Figure 2. The role of a TTO in the technology transfer process is very determining. The TTO must audit the new technology as an attempt to spot the commercial prospect. Whenever a technology requires proof of concept activity so that the product can be accepted in the market, the TTO provides accompaniment and mentoring suited to the TTO and experts specializing in the scope of its network. 2.5. Performance Measurement Performance measurement is a quantifying process of efficiency and effectiveness of a particular action [ 62 ]. Effectiveness represents how much the targeted goals have been achieved. Meanwhile, efficiency refers to the speed of doing a task or how a budget can adequately cover the task. Poister [ 63 ] defines performance measurement as a process to define, monitor, and use an objective indicator of organization performance. Following the definition, the performance measurement process involves activity in determining, defining, and using indicators to monitor an organization’s performance. Monitoring is used to figure out how effective and efficient an organization is in achieving its goals. J. Open Innov. Technol. Mark. Complex. 2019,5, 95 7 of 25 Performance measurement is a medium to improve organization performance. An organization applying a performance measurement is likely to possess better performance rather than an organization which is not [ 64 ]. Due to performance measurement, organizations can monitor improvement and motivate themselves to obtain their goals. According to Moxham [ 65 ], performance measurement is seen as a tool to get improvement in the public service sector and an essential factor in the reformation of that sector. 2.6. Data Envelopment Analysis (DEA) Data envelopment analysis (DEA) is a method of linear-based programming to evaluate the efficiency of organization unit performance, also known as decision-making units (DMUs) [ 66 ]. A DMU can be a group of firms, departments, divisions, or administrative units with a common goal and target as well as the input and output [ 28 ]. DEA aims to measures how efficient a DMU is using its resources in producing several outputs [67]. Data envelopment analysis (DEA) has two basic models, which are the Charnes, Cooper, and Rhodes (CCR) model and Banker, Charnes, and Cooper (BCC) model. By its name, the CCR model is developed by Charnes, Cooper, and Rhodes [ 67 ], while the BCC model is constructed by Banker, Charnes, and Cooper [ 68 ]. The two models differ in assumption. The CCR model assumes that increasing the input value will impact the output value proportionally, in other words: constant return to scale. Therefore, the output and input ratio will always be constant. Unlike the CCR model, the BCC model assumes that the increase of input value will impact in increasing an output value disproportionately or variably. Hence, increasing the input of “x” times does not always mean increasing output by the same amount but may be larger or smaller. This assumption called variable return to scale (VRS). Both the DEA model can be used for either input or output orientation. Table 1presents 20 studies that use the DEA method to measure relative efficiency. The study collation was carried out by author-based mapping. The object of the studies include manufacturing systems, manufacturing industries, government organizations, national (state), financial services systems, and universities. Then the mapping was based on the type of DMU used in the study, the DEA model and orientation used in the study, the number of inputs and outputs studied, and the type of data scale used in the study. Based on those studies, no research measured the efficiency of technology transfer service offices, and no research used ordinal data measurement scales in the form of a Likert scale. The Likert scale used in this study aimed to determine the perception of the efficiency of technology commercialization strategies in universities. 3. Approach to Develop Framework Performance evaluation is a necessary part of management control. Not only can it be used as a reference in decision making but it also the basis of any improvement. Hence, how to measure performance becomes essential. Stakeholders and other researchers have tried to accurately measure the performance at the individual, organizational, and national levels for many decades. This paper illustrates how to use a data envelopment analysis (DEA) to give relative efficiency value for every commercialization strategy that was used by the university. These issues are discussed in this paper: 1. What is the framework of the university efficiency strategy measurement in new technology commercialization? 2. What is the relative strategic efficiency of new technology commercialization performance, and which technology transfer office has an improvement in strategic performance? The criteria for measuring the university strategy efficiency in new technology commercialization have to be defined first to answer the questions mentioned earlier. Based on the literatures, indicators used include physical facilities and shared business service and equipment, monitoring and networking, funding and support, marketing assistance, professional business service, and business etiquette, management, and human resource assistance, and university regulation. J. Open Innov. Technol. Mark. Complex. 2019,5, 95 8 of 25 Table 1. State of the art of this research. Author Object of Study Decision-Making Unit (DMU) DEA Model Orientation Number of Inputs Number of Outputs Measurement Scale CCR BCC Input Output An-Yuan et al. [30] Manufacturing System Production machine √ √ 4 2 Ratio Thore [41] University, Industry, Country Eilat, Golani, and Shtub [42] Government Organization National R&D organization √ √ 6 6 Ratio Shiuh-Nan [31] Manufacturing System Companies in the electronics industry √ √ 7 3 Ratio Wang and Huang [43] National (State) National R&D organization √ √ 4 4 Ratio Jyoti, Banwet, and Deshmukh [29] Government Organization National R&D organization √ √ 6 4 Ratio Joseph, Sandra, and Haiyan [32] Financial Services System Financial companies in Canada √ √ 5 5 Ratio Chen Ta et al. [33] Manufacturing Industry US companies registered online √ √ 2 2 Ratio Sueyoshi and Goto [44] Financial Services System Regional industrial finance √ √ 3 3 Ratio Liu and Lu [46] Government Organization National R&D organization √ √ √ 4 3 Ratio Zhong et al. [47] Government Organization National R&D organization √7 5 Ratio Jelena and Alemka [34] Government Organization e-Business governance √ √ 8 3 Ratio Shirouyehzad et al. [48] Manufacturing Industry Labor √ √ 4 4 Ordinal (Likert) Tanase and Morar [35] Manufacturing Industry Construction machinery √ √ 4 4 Ratio Sueyoshi and Goto [45] Manufacturing Industry National R&D organization √4 2 Ratio Tavassoli, Faramarzi, and Saen [ 36 ] Manufacturing System Company suppliers √ √ 3 3 Ratio Chun et al. [49] Manufacturing System A manufacturing company in Korea √ √ 3 2 Ratio Huang, Nin Ho, and Chen [40] Service Company Hotel business √ √ 4 3 Ratio Sahney [39] University Higher education institutions √ √ 5 5 Ratio This research University Technology Transfer Services Office √ √ 5 5 Ordinal (Likert) J. Open Innov. Technol. Mark. Complex. 2019,5, 95 15 of 25 the relative efficiency of each technology transfer service unit to be able to determine an efficient strategy. An efficient TTO was the one which relative efficiency value was equal to one. The relative efficiency was calculated using a mathematical model based on the DEA variable returns to scale (VRS) output-oriented to properly evaluate the efficiency of the production scales of the unit’s best technology transfer services. The VRS primal was used to determine which technology transfer services unit was efficient (=1) and inefficient (<1) and to investigate the value of variable weights whereas the dual VRS was used to find the value scale efficiency (SE). SE grades indicated whether the technology transfer services unit operates optimally or not. It operated in optimal condition if the value of the VRS >SE and did not operate in optimal condition when the value of the VRS <SE. J. Open Innov. Technol. Mark. Complex. 2019, 5, x FOR PEER REVIEW 17 of 28 sold in the market, the establishment of technology-based startup companies, joint ventures, licensing, and employment. Those dimensions were considered to be the input and output of the unit service technology transfer. Then, we calculated the relative efficiency of each technology transfer service unit to be able to determine an efficient strategy. An efficient TTO was the one which relative efficiency value was equal to one. The relative efficiency was calculated using a mathematical model based on the DEA variable returns to scale (VRS) output-oriented to properly evaluate the efficiency of the production scales of the unit’s best technology transfer services. The VRS primal was used to determine which technology transfer services unit was efficient (=1) and inefficient (<1) and to investigate the value of variable weights whereas the dual VRS was used to find the value scale efficiency (SE). SE grades indicated whether the technology transfer services unit operates optimally or not. It operated in optimal condition if the value of the VRS > SE and did not operate in optimal condition when the value of the VRS < SE. Figure 4. Performance measurement framework of commercialization strategy. 4.2. Relative Efficiency Value Figure 4. Performance measurement framework of commercialization strategy. 4.2. Relative Efficiency Value We processed the collected data using Max DEA in order to figure out the value of relative efficiency for each respective DMU. Before the data was processed by Max DEA software, data recapitulation was J. Open Innov. Technol. Mark. Complex. 2019,5, 95 16 of 25 created first to arrange them with the format required by Max DEA. Table 6presents the recapitulation result by finding the average value of every indicator data from the questionnaire. Table 6. Strategy assessment data for respective variable. DMU I1 I2 I3 I4 I5 O1 O2 O3 O4 O5 TTO1 5.00 4.89 4.67 4.60 3.83 5 5 3 4 4 TTO2 5.00 4.89 4.50 4.60 3.83 5 4 3 4 4 TTO3 4.50 4.00 4.33 4.40 4.50 4 2 3 4 3 TTO4 4.00 3.78 3.83 3.80 4.17 3 1 3 3 3 TTO5 3.25 3.56 3.33 3.60 4.17 3 1 2 3 3 TTO6 4.75 4.33 4.17 4.40 4.67 4 4 4 3 4 TTO7 4.50 4.11 4.17 4.40 4.67 3 3 4 3 3 TTO8 4.00 3.78 3.67 4.00 4.33 3 3 4 3 3 TTO9 3.50 3.44 3.50 3.80 4.00 3 3 3 3 3 TTO10 3.50 3.22 3.00 3.60 4.00 3 3 3 3 3 TTO11 5.00 4.56 4.17 4.20 4.83 4 4 4 5 4 TTO12 4.25 4.56 4.17 4.20 4.83 4 3 4 4 3 TTO13 3.75 4.33 3.83 4.00 4.33 3 3 4 3 3 TTO14 3.50 3.78 3.67 3.40 4.00 3 3 3 3 3 TTO15 3.00 3.67 3.50 3.20 4.00 3 3 3 3 3 TTO16 4.75 4.33 4.17 4.20 4.33 3 4 4 4 4 TTO17 4.75 4.22 4.00 4.00 4.33 3 3 4 4 3 TTO18 4.00 4.00 3.67 3.80 3.83 3 3 4 4 3 TTO19 3.75 3.67 3.33 3.40 3.67 3 3 3 3 3 TTO20 3.75 3.67 3.33 3.40 3.50 3 3 3 3 3 TTO21 4.00 4.11 3.50 3.60 4.67 3 4 3 4 5 TTO22 4.00 4.00 3.50 3.60 4.67 3 3 4 4 4 TTO23 3.50 3.78 3.33 3.60 4.17 3 3 4 4 4 TTO24 3.25 3.56 3.17 3.20 3.83 3 3 3 3 3 TTO25 3.25 3.44 3.17 3.00 3.50 3 3 3 3 3 The mathematical model below is an example of a mathematic model of BCC with input orientation (BCC-O) for DMU1. The model was constructed by using Formula (3) and data from Table 6. Max θm subject to: 5λ1+4.75λ2+5λ3+4.75λ4+4λ5≥5 4.89λ1+4.33λ2+4.56λ3+3.33λ4+3.11λ5≥4.89 4.67λ1+4.17λ2+4.17λ3+4.17λ4+3.5λ5≥4.67 4.6λ1+4.4λ2+4.2λ3+4.2λ4+3.6λ5≥4.6 3.83λ1+3.67λ2+4.83λ3+4.33λ4+4.67λ5≥3.83 5λ1+4λ2+4λ3+3λ4+3λ5≤5θ1 5λ1+4λ2+4λ3+4λ4+4λ5≤5θ2 3λ1+4λ2+4λ3+4λ4+3λ5≤3θ3 4λ1+3λ2+5λ3+4λ4+4λ5≤4θ4 4λ1+4λ2+4λ3+4λ4+5λ5≤4θ5 λ1+λ2+λ3+λ4+λ5=1 λ1,λ2,λ3,λ4,λ5≥0 J. Open Innov. Technol. Mark. Complex. 2019,5, 95 17 of 25 θm(independent) infinite. Then, the model was solved with help from Max DEA to discover the DMU relative efficiency value. The other DMU relative efficiency values were calculated using a similar formula. Table 7presents the results of DMU relative efficiency calculation in commercializing university research products. The score column represents DMU relative efficiency value, and the rank column shows DMU rank based on relative efficiency value. Meanwhile, the reference set (lambda) column provides information about efficient DMUs, which could be used as a benchmark reference for inefficient DMU along with the intensity. Table 7. Decision-making units (DMU) relative efficiency value. DMU Score Rank Reference Set Benchmark (Lambda) TTO1 1 1 TTO1 1 TTO2 1 1 TTO2 1 TTO3 1 1 TTO3 1 TTO4 0.876436 6 TTO2 0.057896 TTO6 0.182529 TTO10 0.387987 TTO11 0.124633 TTO23 0.115791 TTO25 0.131164 TTO5 0.978437 2 TTO2 0.033058 TTO15 0.264463 TTO23 0.033058 TTO25 0.669421 TTO6 1 1 TTO6 1 TTO7 1 1 TTO7 1 TTO8 1 1 TTO8 1 TTO9 0.965544 3 TTO1 0.053528 TTO10 0.518248 TTO23 0.107056 TTO25 0.321168 TTO10 1 1 TTO10 1 TTO11 1 1 TTO11 1 TTO12 1 1 TTO12 1 TTO13 1 1 TTO17 0.200000 TTO23 0.800000 TTO14 0.92 5 TTO1 0.130435 TTO15 0.173913 TTO23 0.260870 TTO25 0.434783 TTO15 1 1 TTO15 1 TTO16 1 1 TTO16 1 TTO17 1 1 TTO12 0.242424 TTO23 0.757576 TTO18 1 1 TTO18 1 TTO19 0.942677 4 TTO1 0.091214 TTO10 0.035346 TTO23 0.182427 TTO25 0.691013 TTO20 1 1 TTO25 1 TTO21 1 1 TTO21 1 TTO22 1 1 TTO22 1 TTO23 1 1 TTO23 1 TTO24 1 1 TTO24 1 TTO25 1 1 TTO25 1 Data processing using Max DEA produced efficiency values that indicated the relative efficiency values of DMU. This value is relative, so if there was a change in a DMU, the efficiency score may have changed. Efficiency scores range from 0 to 1 (0 to 100%). DMU with an efficiency score equal to 1 was classified into an efficient DMU. This meant that there was no other DMU that could use input with a smaller amount than the DMU based on the same amount of output. An efficient DMU was a DMU that could optimize the strategies used in achieving technology commercialization output by predetermined targets. The performance of DMUs was assessed in DEA using the concept of efficiency or productivity, which was the ratio of virtual outputs to virtual inputs. Therefore, all outcomes generated from the strategy, which were the profits from the results of DMU operations, were expressed as virtual outputs. In contrast, all resources used by DMU or conditions that affected DMU performance were expressed as virtual inputs. If the ratio value between virtual output and virtual input =1, the DMU was declared efficient. Conversely, if the ratio value was less than 1, the DMU was declared inefficient. Based on Table 7, we determined which DMUs were classified as efficient and inefficient. There were 20 DMUs that were classified as efficient, namely TTO1, TTO2, TTO3, TTO6, TTO7, TTO8, TTO10, TTO11, TTO12, TTO13, TTO15, TTO16, TTO17, TTO18, TTO20, TTO21, TTO22, TTO23, TTO24, and TTO25, whereas five DMUs, namely TTO4, TTO5, TTO9, TTO14, and TTO19, had efficiency values less than one, so they were classified as inefficient DMUs. 5. Discussion 5.1. Strategy Analysis for Improving Research Commercialization in University We considered five technology transfer offices TTO1, TTO2, TTO3, TTO4, TTO5 that produce the same level of a single output O, from two inputs I1 and I2, shown in Figure 5. TTO1 and TTO3 were efficient. They represented the best practice. This implied that no other firm nor linear combination J. Open Innov. Technol. Mark. Complex. 2019,5, 95 18 of 25 of technology transfer offices could be identified, which produced the same level of output for less than either or both the inputs. Figure 5represents the efficiency frontier of five TTOs consuming two inputs. TTO1 held an efficient practice for input I1, whereas TTO3 held the efficient practice input I2. The straight-line representing TTO2 represents the best achievable target performance (corresponding to point A), which was actually a linear combination of the best practice TTO1 and TTO3. TTO4 was an inefficient DMU. The value of input I1 on TTO4 was higher than the value of input I2. Thus, TTO4 referred to TTO1, which was the best practice in input I1. This is illustrated through a straight line that intersects TTO1 at point B. Therefore, to achieve an increase in commercialized output, TTO4 must be able to increase the input values of I1 and I2 by point B. J. Open Innov. Technol. Mark. Complex. 2019, 5, x FOR PEER REVIEW 21 of 28 5. Discussion 5.1. Strategy Analysis for Improving Research Commercialization in University We considered five technology transfer offices TTO1, TTO2, TTO3, TTO4, TTO5 that produce the same level of a single output O, from two inputs I1 and I2, shown in Figure 5. TTO1 and TTO3 were efficient. They represented the best practice. This implied that no other firm nor linear combination of technology transfer offices could be identified, which produced the same level of output for less than either or both the inputs. Figure 5 represents the efficiency frontier of five TTOs consuming two inputs. TTO1 held an efficient practice for input I1, whereas TTO3 held the efficient practice input I2. The straight-line representing TTO2 represents the best achievable target performance (corresponding to point A), which was actually a linear combination of the best practice TTO1 and TTO3. TTO4 was an inefficient DMU. The value of input I1 on TTO4 was higher than the value of input I2. Thus, TTO4 referred to TTO1, which was the best practice in input I1. This is illustrated through a straight line that intersects TTO1 at point B. Therefore, to achieve an increase in commercialized output, TTO4 must be able to increase the input values of I1 and I2 by point B. Figure 5. Efficiency frontier of five technology transfer offices consuming two inputs. The correlation test between variables for 20 efficient DMUs was conducted to see the value of the correlation between input variables and output variables, which would later be used to help determine the strategy formulation. Correlation test was done using SPSS software and using the Kendall Tau correlation test. Table 8 shows the correlation values between variables for 20 efficient DMUs from the Max DEA processing results. It could be derived from the table that all input variables to the output variable had a positive correlation. These results could later be used to design a commercialization strategy based on the output the technology transfer service office at the university focuses on achieving. Figure 5. Efficiency frontier of five technology transfer offices consuming two inputs. The correlation test between variables for 20 efficient DMUs was conducted to see the value of the correlation between input variables and output variables, which would later be used to help determine the strategy formulation. Correlation test was done using SPSS software and using the Kendall Tau correlation test. Table 8shows the correlation values between variables for 20 efficient DMUs from the Max DEA processing results. It could be derived from the table that all input variables to the output variable had a positive correlation. These results could later be used to design a commercialization strategy based on the output the technology transfer service office at the university focuses on achieving. Table 8. Correlation value between efficient DMU variables. Strategy The Commercial Product (O1) PPBT (O2) Joint Venture (O3) License (O4) Employment (O5) Physical Facilities (I1) 0.609 0.480 0.251 0.523 0.260 Mentoring and Coaching (I2) 0.608 0.525 0.300 0.487 0.412 Marketing and Business Networking (I3) 0.680 0.360 0.198 0.399 0.347 Financial Aspect (I4) 0.606 0.354 0.237 0.335 0.255 University Internal Regulation (I5) 0.671 0.690 0.503 0.302 0.343 J. Open Innov. Technol. Mark. Complex. 2019,5, 95 19 of 25 For universities, this model could be used as a model to measure the efficiency of a strategy that has been carried out then determine the alternative improvement that must be chosen among the five input strategies that exist to achieve the target five outputs that have been determined. This model is also able to provide recommendations for technology transfer service offices in strengthening the strategies that must be taken. Based on Table 8, we could underline which input variables needed to be improved by the university to increase output, for example, if the university wants to focus on improving commercial products what needs to be done is to increase marketing assistance strategies and expand business networks including increasing research dissemination, conducting market research on products to be commercialized, adding to promotional media, holding business meetings and exhibitions, expanding access to various industrial networks and potential consumers, and strengthening synergies with the business ecosystem network. In addition to improving marketing strategies and business networks, it is also necessary to prepare internal college regulations, including regulations on product readiness evaluation systems, commercialization of intellectual property rights, policies, and rules regarding the encouragement of innovation and product development, policies, and or incentive rules. The strategy to increase physical facilities also needs to be improved, including the availability of office space, workshops, meeting rooms, telephone and internet networks as well as available business and showroom spaces. Whereas if it is more focused on increasing joint ventures, it is necessary to improve the university’s internal regulatory strategy, mentoring and coaching including the preparation of business plans and business models, coaching on management of company design and business risk, contracting systems in government, management of intellectual property rights, introduction to law and business ethics, organizational management, tax systems and mentoring regarding pricing strategies, sales systems and product distribution, and improvement of physical facilities. Likewise, if they want to increase licenses, it is necessary to improve the physical facilities strategy, mentoring and coaching as well as marketing and business networking assistance. In order to increase employment, it is necessary to improve mentoring and coaching strategies, marketing assistance, and business networks as well as formulating internal regulations of the university. This model is also able to be used for evaluating strategy feedback for the development of technology-based startup companies, including how to improve the university’s internal regulatory strategy which significantly affects the success of technology-based startups in the future, namely the need for support from universities in the formulation of policies on the commercialization of intellectual property rights. In full, the drive for innovation and intensive product development as well as the encouragement of industrial cooperation, then it also needs policy regarding the rules of incentives, as well as an evaluation system for product readiness to be commercialized. Mentoring and coaching strategies also need to be improved in the development of technology-based startup companies, including mentoring and coaching activities in the preparation of business plans and business models, management of company design and business risk, procurement processes, the introduction of legal and business ethics, and tax systems. The state of physical facilities is also very influential in the development of technology-based startup companies, at least available office space, workshops, laboratories, meeting rooms, telephone, and internet networks, as well as available business space and showrooms of research results to be commercialized. 5.2. The Relation between Technology Transfer Efficiency and Open Innovation The relationship between technology transfer efficiency and open innovation can be illustrated through the open innovation paradigm in Figure 6. Many new technologies resulted from university research. However, not all of these technology products can develop in the market. Therefore, technology insourcing is needed to accelerate the commercialization of technology, both through internal and external technology bases. In this case, the university plays a role as a technology spinoff, in which there is a TTO in charge of carrying out technology incubation. An efficient DMU will increase the probability of the success of a technology product. Therefore, in our model, we used the DEA model to find J. Open Innov. Technol. Mark. Complex. 2019,5, 95 20 of 25 efficient DMUs in transferring technology to synthesize effective strategies. Thus, we could generate a variety of decisions that could be used for technological interventions in terms of physical facilities, mentoring and coaching, marketing and business networking, financial aspects, and university’s internal regulation. The more efficient DMUs, the more best practices we could implement in commercializing new technologies. J. Open Innov. Technol. Mark. Complex. 2019, 5, x FOR PEER REVIEW 23 of 28 introduction of legal and business ethics, and tax systems. The state of physical facilities is also very influential in the development of technology-based startup companies, at least available office space, workshops, laboratories, meeting rooms, telephone, and internet networks, as well as available business space and showrooms of research results to be commercialized. 5.2. The Relation between Technology Transfer Efficiency and Open Innovation The relationship between technology transfer efficiency and open innovation can be illustrated through the open innovation paradigm in Figure 6. Many new technologies resulted from university research. However, not all of these technology products can develop in the market. Therefore, technology insourcing is needed to accelerate the commercialization of technology, both through internal and external technology bases. In this case, the university plays a role as a technology spinoff, in which there is a TTO in charge of carrying out technology incubation. An efficient DMU will increase the probability of the success of a technology product. Therefore, in our model, we used the DEA model to find efficient DMUs in transferring technology to synthesize effective strategies. Thus, we could generate a variety of decisions that could be used for technological interventions in terms of physical facilities, mentoring and coaching, marketing and business networking, financial aspects, and university’s internal regulation. The more efficient DMUs, the more best practices we could implement in commercializing new technologies. Figure 6. The open innovation paradigm (sources from Reference [90]). 6. Conclusions and Implications This paper constructed a framework of university efficiency strategy measurement in new technology commercialization. This framework employed DEA analysis to assess and examine the efficiency of a direct and precise method for relative strategical efficiency of new technology commercialization performance. The efficiency evaluation technique employed in this paper could Figure 6. The open innovation paradigm (sources from Reference [90]). 6. Conclusions and Implications This paper constructed a framework of university efficiency strategy measurement in new technology commercialization. This framework employed DEA analysis to assess and examine the efficiency of a direct and precise method for relative strategical efficiency of new technology commercialization performance. The efficiency evaluation technique employed in this paper could provide another insight to analyze the performance of new technology commercialization in universities. The implication of the DEA efficiency result was to drive the efficiency level of the university’s strategy from the observed performance. It also helped to identify the benchmarking of other universities, which would be valuable information for improving their new technology commercialization strategy performance. In detail, a benchmark was provided to improve their weakness of strategy and resource allocation of poorly performing universities. As TTOs can evaluate the commercialization strategy, TTOs can accelerate commercialization by using policy choices that have high-efficiency frontier value. In line with down streaming and commercialization of new technologies, the role of TTOs in the commercialization of new technologies is deemed necessary to be developed at the university. ATTO is expected to be the main actor for the bridging system of technology commercialization to ensure that the potential product does not fall into the valley of death-scourge in the technology commercialization process. J. Open Innov. Technol. Mark. Complex. 2019,5, 95 21 of 25 This model could be used as a benchmark by UNS to develop a strategy for commercializing LFP battery products. To accelerate LFP battery commercialization, UNS needs to determine its strategic focus in which output needs to be improved. Thus, the university can determine which input performance should be improved. Thus, the university can formulate strategies to improve the commercialization of LFP batteries. The three main priorities to focus on improving commercial products that need to be done are improving marketing assistance strategies and expanding business networks, internal university regulations, and improving physical facilities whereas the three priorities for improving technology-based startup companies can be making improvements to the internal university strategy, mentoring and coaching as well as physical facilities. Meanwhile, to focus on increasing the joint venture it is necessary to improve the internal regulation strategy of universities, mentoring and coaching as well as improving physical facilities, likewise, if the university wants to increase licenses it is necessary to improve the physical facilities strategy, mentoring and coaching as well as marketing and business networking assistance. In order to increase employment, it is necessary to improve mentoring and coaching strategies, marketing assistance, and business networks as well as internal regulations of the university. Theoretically, this research has supplemented literature in performance measurement of commercialization of research output from higher education in Indonesia. While practically, this study provided information about the relative efficiency value of university strategy to commercialize their research output, which could be used in formulating the strategy of performance enhancement. Author Contributions: Conceptualization, W.S., R.W.A., and R.T.S.; methodology, W.S. and R.W.A.; software, R.W.A.; validation, W.S., R.W.A., and R.T.S.; formal analysis, W.S.; resources, W.S. and R.T.S.; data curation, W.S.; writing—original draft preparation, W.S. and R.W.A.; writing—review and editing, W.S. and R.W.A.; supervision, W.S.; project administration, W.S. Funding: Ministry of Research, Technology, and Higher Education, Indonesia. Acknowledgments: This research was partially funded by the Indonesian Ministry of Research, Technology and Higher Education under WCU Program managed by Institut Teknologi Bandung. Conflicts of Interest: The authors declare no conflict of interest. References 1. Govender, V.; Rampersad, R. Change management in the higher education landscape: A case of the transition process at a South African University. J. Risk Gov. Control Financ. Mark. Inst. 2016,6, 43–51. [CrossRef] 2. Bramwell, A.; Wolfe, D.A. Universities and regional economic development: The entrepreneurial University of Waterloo. Res. Policy 2008,37, 1175–1187. [CrossRef] 3. Siegel, D.S.; Waldman, D.; Link, A. Assessing the impact of organizational practices on the relative productivity of university technology transfer offices: An exploratory study. Res. Policy 2003,32, 27–48. [CrossRef] 4. Wicaksana, D.E.P.; Yunaristanto, Y.; Sutopo, W. Identification of Incubation Scheme by Incubator in University Innovation Center to Develop Indonesian Economy. In Proceedings of the Joint International Conference on Electric Vehicular Technology and Industrial, Mechanical, Electrical and Chemical Engineering (ICEVT & IMECE), Surakarta, Indonesia, 4–5 November 2015. 5. Kusuma, C.; Sutopo, W.; Yuniaristanto, Y.; Hadiyono, S.; Nizam, M. Incubation Scheme of the University Spin Offto Commercialize the Invention in Sebelas Maret University. In Proceedings of the International MultiConference of Engineer and Computer Scientist, Hong Kong, China, 18–20 March 2015. 6. Sutopo, W. Book Review: Technopreneurship, unpublished; Surakarta, Indonesia, 2015. 7. Sutopo, W. The Roles of Industrial Engineering Education for Promoting Innovations and Technology Commercialization in the Digital Era; IOP Conference Series: Materials Science and Engineering; IOP Publishing Ltd.: Bristol, UK, 2019. [CrossRef] 8. Siegel, D.S.; Veugelers, R.; Wright, M. Technology transfer offices and commercialization of university intellectual property: Performance and policy implications. Oxf. Rev. Econ. Policy 2007,23, 640–660. [CrossRef] 9. Dalmarco, G.; Dewes, M.D.F.; Zawislak, P.A.; Padula, A.D. Universities’ Intellectual Property: Path for Innovation or Patent Competition? J. Technol. Manag. Innov. 2011,6, 159–170. [CrossRef] J. Open Innov. Technol. Mark. Complex. 2019,5, 95 22 of 25 10. Vinig, T.; Lips, D. Measuring the performance of university technology transfer using meta data approach: The case of Dutch universities. J. Technol. Transf. 2015,40, 1034–1049. [CrossRef] 11. Rogers, E.M.; Yin, J.; Hoffmann, J. Assessing the effectiveness of technology transfer offices at US research universities. J. Assoc. Univ. Technol. Manag. 2000,12, 47–80. 12. Phan, P.H.; Siegel, D.S. The effectiveness of university technology transfer. Found. Trends ® Entrep. 2006 ,2, 77–144. [CrossRef] 13. Harlow, D. Knowledge transfer to industry at selected R1 research universities in North Carolina. Electron. J. Knowl. Manag. 2017,15, 3–16. 14. DeVol, R.; Lee, J.; Ratnatunga, M. Concept to Commercialization: The Best Universities for Technology Transfer; Milken Institute: Santa Monica, CA, USA, 2017. 15. Carlsson, B.; Fridh, A. Technology Transfer in United States Universities: A Survey and Statistical Analysis. J. Evol. Econ. 2002,12, 199–232. [CrossRef] 16. Sutopo, W.; Nizam, M.; Purwanto, A.; Atikah, N.; Putri, A.S. A Cost Estimation Application for Determining Feasib i lity Assessment of Li - Ion Battery in Mini Plant Scale. Int. J. Electr. Eng. Inform. 2016 ,8, 189–199. [CrossRef] 17. Sutopo, W.; Kadir, E.A. An Indonesian Standard of Lithium-ion Battery Cell Ferro Phospate for Electric Vehicle Applications. Telkomnika 2017,15, 584–589. [CrossRef] 18. Sutopo, W.; Kadir, E.A. Designing Framework for Standardization Case Study: Lithium-Ion Battery Module in Electric Vehicle Application. Int. J. Electr. Comput. Eng. 2018,8, 220–226. [CrossRef] 19. Neely, A. The Evolution of Performance Measurement Research Development in the last Decade and a Research Agenda for The Next. Int. J. Oper. Prod. Manag. 2008,25, 1264–1277. [CrossRef] 20. Medori, D.; Steeple, D. A Framework for Auditing and Enhancing Performance Measurement Systems. Int. J. Oper. Prod. Manag. 2000,20, 520–533. [CrossRef] 21. Lohman, C.; Fortuin, L.; Wouters, M. Designing a Performance Measurement System. A Case Study. Eur. J. Oper. Res. 2004,156, 267–286. [CrossRef] 22. Mintzberg, H. Patterns in Strategy Formation. Manag. Sci. 1978,24, 934–948. [CrossRef] 23. Hill, T. Teaching Manufacturing Strategy. Int. J. Oper. Prod. Manag. 1986,6, 10–20. [CrossRef] 24. Bititci, U.; Garengo, P.; Dofler, V.; Nudurupati, S. Performance Measurement Chalenges for Tomorrow. Int. J. Manag. Rev. 2012,14, 305–327. [CrossRef] 25. Folan, P.; Browne, J. A Review of Performance Measurement Towards Performance Management. Comput. Ind. 2005,56, 663–680. [CrossRef] 26. Word, P.; McCreery, J.; Ritzman, L. Competitive Priorities in Operating Management. Decis. Sci. 1998 ,29, 1035–1046. [CrossRef] 27. Thiam, A.; Bravo-Ureta, B.E.; Rivas, T. Technical Efficiency in Developing Country Agriculture: A meta-analysis. Agric. Econ. 2001,25, 235–243. [CrossRef] 28. Al-Shammari, M. A Multi-Criteria data Envelopment Analysis Model for Measuring The productive Efficiency of hospitals. Int. J. Oper. Prod. Manag. 1999,19, 879–891. [CrossRef] 29. Yoti, J.; Banwet, D.K.; Deshmukh, S.G. Evaluating performance of national R&D organizations using integrated DEA-AHP technique. Int. J. Product. Perform. Manag. 2008,58, 370–388. 30. Chang, A.Y.; Whitehouse, D.J.; Chang, S.L.; Hsieh, Y.C. An Approach to The Measurement of Single-Machine Flexibility. Int. J. Prod. Res. 2001,39, 24. [CrossRef] 31. Hwang, S.N. An Application of Data Envelopment Analysis to Measure the Managerial Performance of Electronics Industry in taiwan. Int. J. Technol. Manag. 2007,40, 215–221. [CrossRef] 32. Joseph, C.; Sandra, A.; Haiyan, Z. Adjusting for Cultural Differences, a New DEA Model Applied to A Merged bank. J. Prod. Anal. 2010,33, 109–123. 33. Chen-Ta, B.; Desheng, D.; Chris, C.; David, L. A Risk Scoring Model and Application to Measuring Internet Stock Performance. Int. J. Inf. Technol. Decis. Mak. 2009,8, 133–149. 34. Jelena, J.; Alemka, S. Measuring the Performance of Local E-Goverment in The Republic of Croatia Using Data Envelopment Analysis. Probl. Perspect. Manag. 2012,10, 35–44. 35. Tanase, L.; Morar, L. Productive Performance Analysis in Machinery Industry Using mamquist Index. Appl. Mec. Mater. 2012,245, 220–226. [CrossRef] 36. Tavassoli, M.; Faramarzi, G.R.; Saen, R.F. Efficiency and effectiveness in airline performance using a SBM-NDEA model in the presence of shared input. J. Air Transp. Manag. 2014,34, 146–153. [CrossRef] J. Open Innov. Technol. Mark. Complex. 2019,5, 95 23 of 25 37. Silva, D.; Alves, G.; Neto, P.; Ferreira, T. Measurement of Fitness Function Efficiency Using Data Envelopment Analysis. J. Expert Syst. Appl. 2014,41, 7147–7160. [CrossRef] 38. Jian, Y.; Dai, L. Statistical Inference of DEA Model of Environtmental Efficiency Considering Undesirable Outputs. In Proceedings of the 2014 Tenth International Conference on Computational Intelligence and Security, Kunming, China, 15–16 November 2014. 39. Sahney, S.; Thakkar, J. A comparative assessment of the performance of select higher education institutes in India. Qual. Assur. Educ. 2016,24, 278–302. [CrossRef] 40. Huang, C.W.; Nin Ho, F.; Chen, Y.C. Assessing the Effectiveness of Marketing Strategies in Tourist Hotels: An Illustration Using a Multi-Method Approach in Taiwan. J. Travel Tour. Mark. 2015,32, 1–15. [CrossRef] 41. Thore, S.A.O. DEA and Related Analytical Methods for Evaluating the Use and Implementation of Technical Innovation; Springer: Berlin/Heidelberg, Germany, 2002. 42. Eilat, H.; Golany, B.; Shtub, A. Constructing and Evaluating Balanced Portfolios of RD Projects with Interactions: A DEA Based Methodology. Eur. J. Oper. Res. 2006,172, 1018–1039. [CrossRef] 43. Wang, E.; Huang, W. Relative Efficiency Of R&D Activities: A Cross-Country Study Accounting for Environmental Factors In The DEA Approach. Res. Policy 2007,36, 260–273. 44. Sueyoshi, T.; Goto, M. Can R&D expenditure avoid corporate bankruptcy?: Comparison between Japanese machinery and electric equipment industries using DEA–discriminant analysis. Eur. J. Oper. Res. 2009 ,196, 289–311. 45. Sueyoshi, T.; Goto, M. A use of DEA–DA to measure importance of R&D expenditure in Japanese information technology industry. Decis. Support Syst. 2013,54, 941–952. 46. Liu, J.S.; Lu, W.-M. DEA and Ranking with the Network-Based Approach: A Case of R&D performance. Omega 2011,38, 453–464. 47. Zhong, W.; Yuan, W.; Li, S.; Huang, Z. The Performance Evaluation of Regional R&D Investments in China: An Application of DEA Based on The First Official China Economic Census Data. Omega 2011,39, 447–455. 48. Shirouyehzad, H.; Lotfi, F.H.; Aryanezhad, M.; Dabestani, R. A data envelopment analysis approach for measuring the efficiency of employees: A case study. S. Afr. J. Ind. Eng. 2012,23, 191–201. [CrossRef] 49. Chun, D.; Chung, Y.; Woo, C.; Seo, H.; Ko, H. Labor Union Effects on Innovation and Commercialization Productivity: An Integrated Propensity Score Matching and Two-Stage Data Envelopment Analyis. Sustainability 2015,7, 5120–5138. [CrossRef] 50. Carayannis, E.G.; Rogers, E.M.; Kurihara, K.; Albritton, M. High-Technology Spin-Offs from Goverment R&D Laboratories and Research Universitries. Technovation 1998,18, 1–11. 51. Baldini, N.; Grimaldi, R.; Sobrero, M. Instutional Changes and The Commercialization of Academic Knowledge: A Study of Italian Universities Patenting Activities. Res. Policy 2006,35, 518–532. [CrossRef] 52. Nlemvo, N.; Pirnay, F.; Surlemont, B. A Stage Model of Academic Spin OffCreation. Technovation 2002 ,5, 281–289. 53. Siegel, G.; Marconi, H.R. Behavioral Accounting; Thomson South-Western: Mason, OH, USA, 1989. 54. Shane, S.; Stuart, T. Organizational Endowments and The Performance of University Start-Ups. Manag. Sci. 2002,48, 154–170. [CrossRef] 55. Mainelli, P. Managing Innovation and Entrepreneurship in Technology Based Firm; John Wiley&Sons Inc.: New York, NY, USA, 2001. 56. Yun, J.J.; Lee, M.; Park, K.; Zhao, X. Open Innovation and Serial Entrepreneurs. Sustainability 2019 ,11, 5055. [CrossRef] 57. Yun, J.J.; Jeong, E.; Zhao, X.; Hahm, S.D.; Kim, K. Collective Intelligence: An Emerging World in Open Innovation. Sustainability 2019,11, 4495. [CrossRef] 58. Yun, J.J.; Liu, Z. Microand Macro-Dynamics of Open Innovation with a Quadruple-Helix Model. Sustainability 2019,11, 3301. [CrossRef] 59. Yun, J.J.; Won, D.; Park, K.; Jeong, E.; Zhao, X. The role of a business model in market growth: The difference between the converted industry and the emerging industry. Technol. Forecast. Soc. Chang. 2019 ,146, 534–562. [CrossRef] 60. Yun, J.J.; Won, D.; Park, K. Entrepreneurial cyclical dynamics of open innovation. J. Evol. Econ. 2018 ,28, 1151–1174. [CrossRef] J. Open Innov. Technol. Mark. Complex. 2019,5, 95 24 of 25 61. Yun, J.J.; Jeong, E.; Lee, Y.; Kim, K. The Effect of Open Innovation on Technology Value and Technology Transfer: A Comparative Analysis of the Automotive, Robotics, and Aviation Industries of Korea. Sustainability 2018 , 10, 2459. [CrossRef] 62. Neely, A.; Gregory, M.; Platts, K. Performance Measurement System Design: A Literature Review and Research Agenda. Int. J. Oper. Prod. Manag. 2005,25, 1228–1263. [CrossRef] 63. Poister, T. Measuring Performance in Public and Nonprofit Organization; Jossey-Bass: San Francisco, CA, USA, 2003. 64. De Waal, A.; Geodegebuure, R.; Geradts, P. The Impact of performance Management on the Result of a Non-Profit Organization. Int. J. Product. Perform. Manag. 2011,60, 778–796. [CrossRef] 65. Moxham, C. Performance Measurement; Examining the Applicability of the Existing Body of Knowledge to Nonprofit Organizations. Int. J. Oper. Prod. Manag. 2009,29, 740–763. [CrossRef] 66. Ramanathan, R. An Introduction to Data Envelopment Analysis: A Tool for Performance Measurement; Sage Publication Pvt Ltd.: New Delhi, India, 2003. 67. Charnes, A.; Cooper, W.; Rhodes, E. Measuring the Efficiency of Decision Making Units. Eur. J. Oper. Res. 1978,2, 429–444. [CrossRef] 68. Banker, R.; Charnes, A.; Cooper, W. Models for Estimating technical and Scale Inefficiencies in data Envelopment Analysis. Manag. Sci. 1984,30, 1078–1092. [CrossRef] 69. Scholten, V.E.; Elfring, T.; Omta, S.W.F. The Parent Connection: The Effect of the Parent–Spin-OffRelationship on the Spin-Off’s Successes. In Proceedings of the Babson Kauffman Entrepreneurship Research Conference, Strathclyde, Scotland, 3 June 2004. 70. Egeln, J.; Gottschalk, S.; Rammer, C.; Spielkamp, A. Public Research Spin-Offs in Germany, Summary Report, ZEW Documentation 03-04; Centre for European Economic Research: Manheim, Germany, 2003. 71. Scholten, E. The Early Growth of Academic Spin-Offs. Ph.D. Thesis, Wageningen University, Wageningen, The Netherlands, 2006. 72. Kriegesmann, B. Unternehmensgründungen aus der Wissenschaft. Eine empirische Analyse zu Stand, Entwicklungen und institutionellen Rahmenbedingungen in außeruniversitären Forschung-seinrichtungen. Z Betr. 2000,70, 397–414. 73. Vohara, A.; Wright, M.; Lockett, A. Critical Junctures in The Development of University High-Tech Spinout Companies. Res. Policy 2004,33, 147–175. [CrossRef] 74. Cooper, S.; Hetherington, L. Facilitating Academic Entrepreneurship: Encouraging Technology Commercialisation through the Development of Entrepreneurial Capability; Babson College: Wellesley, MA, USA, 2003. 75. Lockett, A.; Wright, M. Resources, Capabilities, Risk Capital and The Creation of University Spin-Out Companies. Res. Policy 2005,34, 1043–1057. [CrossRef] 76. Markman, G.D.; Phan, P.H.; Balkin, D.B.; Gianiodis, P.T. Innovation speed: Transferring University Technology to Market. Res. Policy 2005,34, 1058–1075. [CrossRef] 77. Smilor, W.; Matthews, J. University Venturing: Technology Transfer and Commercialisation in Higher Education. Int. J. Technol. Transf. Commer. 2004,3, 111–128. [CrossRef] 78. Lockett, A.; Wright, M.; Franklin, S. Technology Transfer and Universities’spin-out Atrategies. Small Bus. Econ. 2003,20, 185–200. [CrossRef] 79. Schmelter, A. Entwicklungsverläufe forschungsnaher Unternehmensgru ndungen und deren Determinanten. Die Betr. 2004,64, 471–486. 80. Grandi,A.; Grimaldi,R.Exploringthenetworkingcharacteristics of newventurefoundingteams. Small Bus. Econ. 2003,21, 329–341. [CrossRef] 81. Lendner, C. How University Business Incubators Help Start-Ups to Succeed: An International Study. In Babson Kauffmann Entrepreneurship Research Conference; Babson College: Wellesley, MA, USA, 2003. 82. O’Shea, R.; Allen, T.; Chevalier, A. Entrepreneurial Orientation, Technology Transfer, and Spin-Off Performance of U.S. Universities. Res. Policy 2005,34, 994–1009. [CrossRef] 83. Trune, D.R.; Goslin, L.N. University Technology Transfer Programs: A Profit/Loss Analysis. Technol. Forecast. Soc. Chang. 1998,57, 197–204. [CrossRef] 84. Jensen, R.A.; Thursby, M.C. Patent Licensing and the Research University. NBER Working Paper No. w10758. Available online: https://ssrn.com/abstract=590773 (accessed on 15 August 2019). 85. Thursby, J.G.; Thursby, M.C. University Licensing (Winter 2007). Oxf. Rev. Econ. Policy 2007 ,23, 620–639. [CrossRef]