Crowdsourcing in patent examination: overcoming patent examiners' local search bias

Lampe, Hannes W. Article — Published Version Crowdsourcing in patent examination: overcoming patent examiners' local search bias R&D Management Provided in Cooperation with: John Wiley & Sons Suggested Citation: Lampe, Hannes W. (2023) : Crowdsourcing in patent examination: overcoming patent examiners' local search bias, R&D Management, ISSN 1467-9310, Wiley, Hoboken, NJ, Vol. 53, Iss. 5, pp. 764-777, https://doi.org/10.1111/radm.12597 This Version is available at: https://hdl.handle.net/10419/288112 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. http://creativecommons.org/licenses/by/4.0/ © 2023 The Author. R&D Management published by RADMA and John Wiley & Sons Ltd. 764 This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. Crowdsourcing in patent examination: overcoming patent examiners’ local search bias Hannes W. Lampe1,2,* 1 Institute of Entrepreneurship,Hamburg University of Technology, Hamburg, 21073, Germany. 2 Capgemini Invent, Berlin, 10785, Germany. [email protected] This article investigates how crowdsourcing for knowledge creation in a crucial knowledgeintense task – patent application examination – informs decisionmaking. It is hypothesized that patent examiners’ views underly a local search bias (i.e., they rely on locally preferred and conveniently available local information), which may be overcome through crowdsourcing. To analyze this potential effect of crowdsourcing, this study analyzes USPTO’s Peer To Patent initiative, opening the patent examination process to public participation for the first time. The data from this initiative is further enhanced with data from the PatentsView database and the Patent Examination Research Database. The study results provide the first empirical evidence that crowdsourcing aids a patent examination process in overcoming the examiner’s local search bias – their overreliance on internal knowledge. In particular, it is found that crowdsourcing in patent examination increases examiners’ reliance on atypical and less formalized knowledge. Overall, these findings enable several theoretical and practical recommendations. 1. Introduction In February 2018, Waymo and Uber settled a lawsuit, with Waymo being awarded Uber’s shares worth $245 million. The litigation was based on two accusations: First, as predominantly communicated in the news, Waymo accused its engineers of taking more than 14,000 technical confidential files over to Uber. Second, Waymo claimed that Uber infringed four of its patents concerning Uber’s laserranging lidar devices (BBC,2018). However, the most surprising aspect about this litigation is that, in the same year, a notinvolved engineer, Eric Swilden, filed a reexamination request with the United States Patent and Trademark Office (USPTO) (Wired, 2017). Swilden argued that cited references to the prior art (any evidence that an invention is already known1), which are already part of the patent were not taken into consideration by the examiner. In March 2018, the reexamination resulted in the rejection of 53 out of 56 challenged claims (USPTO,2018). This example elucidates the potential advantages of externals providing knowledge. Organizations’ levering the increasingly distributed knowledge of marginalized actors at low cost (Jeppesen and Lakhani, 2010) is called crowdsourcing. However, the above example further taps into a different direction, namely that individuals focus too much on their own capabilities and the solution space known to them, called local search bias (March, 1991; Helfat,1994; Hippel,1994). Past research has established that the local search bias leads to inefficient and presumably worse decisions in various contexts that involve the evaluation of innovative activity and/ or output (Martin and Mitchell, 1998; Poetz and Prügl,2010). Using the example of the Peer- to- Patent © 2023 The Author. R&D Management published by RADMA and John Wiley & Sons Ltd. Crowdsourcing in patent examination R&D Management 53, 5, 2023 765 (PTP) initiative and its impact on patent examination processes, my work provides insights into how the local search bias can be overcome by crowdsourcing, i.e., through an open call for nonpatent examiners (externals) to participate (Howe,2006; Agerfalk and Fitzgerald, 2008; Afuha and Tucci, 2012). While crowdsourcing per se does not influence the likelihood of a patent being granted, it does indeed provide more atypical and less formal material to inform decisionmaking. By combining crowdsourcing’s democracy of open participation with the legitimacy and effectiveness of administrative decisionmaking, this additional information may in turn serve patent examiners in making a wellthought- out decision. The study results indicate that policymakers and public officials, as well as patent examiners, are advised to consider the additional benefits that crowdsourcing, or any other form of external participation, may have on their difficult and consequential decisions alongside knowledgeintense tasks to evaluate innovative outcomes, such as, for example, the patent examination process. As such, they should wherever possible strive to increase the amount of more atypical and less formalized knowledge available. For patent applicants, being individuals or firms, it may also prove to be useful to enrich their patent submission with more, but less formalized and atypical knowledge when making their case. While previous research has identified various benefits of crowdsourcing (Lüthje et al.,2006; Boudreau and Lakhani,2009; Poetz and Schreier,2012), little is known about how crowdsourcing might affect the patent application examination process. This is surprising to be noted that as a joint initiative between the USPTO and the New York Law School, PTP, which integrates externals into the patent examination process (Allen et al., 2009) has been implemented. The second pilot project has been completed, and 520 patent applications constituted this initiative. There is a serious dearth of research into the benefits of crowdsourced prior art searches, given that the first PTP initiative from the U.S. is used as a push to make Peer- to- Patent an international phenomenon; for instance, PTP has diffused to Australia, where the Queensland University of Technology collaborated with IP Australia and the New York Law School (Fitzgeral et al., 2010). Other examples of these include the Japan Patent Office (JPO), the Korean Intellectual Property Office (KIPO), and the UK Intellectual Property Office. Further, there have been repeated calls to expand the use of crowdsourcing in the patent review process (Bestor and Hamp,2010; Ghafaele and Gibert,2011). Although there have been some annual progress reports for these pilot projects, this study extends these reports by analyzing results beyond the first examiner action and summary statistics (Allen et al.,2008, 2009, 2012). The remainder of this article proceeds as follows. Section2 provides an overview of the USPTO’s Peer To Patent initiative – this study’s empirical focus – as well as the key weaknesses of the traditional patent application examination process. Section 3 derives the hypotheses. Next, the data are explained and models are derived to test the elaborated benefits of crowdsourcing on the patent examination process. In Section5, the results are presented. The last section discusses the findings, outlines potential limitations and some avenues for future research, and concludes. 2. Patent examination and the Peer To Patent initiative To analyze the PTP initiative’s potential benefits, one must primarily understand the patent examination process and its downturns in its traditional form, which will be explained briefly next. An assigned examiner, usually a civil servant with a scientific or engineering background, reviews a patent application to determine whether or not the claimed invention should be granted. A patent examiner’s most important task is to review the disclosure in an application and to compare it to the prior art (similar to the scientific review process). This includes reading and understanding a patent application as well as searching for prior art (in databases on granted patents, patent applications, the scientific literature, etc.) to identify whether the potential invention can contribute any further when compared to the prior art. Thus, a patent application needs to be novel and nonobvious in addition to having industrial applicability/utility for it to be granted. A patent examiner must substantially review whether a patent application complies with the legal requirements for granting a patent. In practice, this process is effortful – for instance, there may be several office actions2 that require a response from inventors and their patent attorneys (USPTO,2019). 2.1. The shortcomings of the patent examination process Before elaborating on crowdsourcing’s potential benefits in deriving the hypotheses, the author will first explain the key shortcomings of the traditional patent examination process. This emphasizes the importance of this study’s setting and further clarifies the potential advantages of crowdsourcing in this public sector task. First, the inventors are not required to conduct prior art searches or supply the © 2023 The Author. R&D Management published by RADMA and John Wiley & Sons Ltd. Hannes W. Lampe 766 R&D Management 53, 5, 2023 patent office with the prior art of which they lack awareness (Allen et al., 2012). Second, the patent examiners have a limited database to conduct their research: conference presentations are not available in a comprehensive form, and software code is often poorly documented and cumbersome to detect (Allen et al.,2012). In contrast to prior patents, patent examiners have restricted access to nonpatent knowledge sources. Third, patent examiners have limited time to invest in a patent application. Examiners have roughly 20 hr to examine a patent and decide whether or not to grant it. Such a decision can have a huge impact, since, for instance, a 20- year grant of monopoly rights is likely to shape the future of both fundamental research and industry (Allen et al.,2012). In their examination process, examiners are expected to both digest the content of the application and prior art mentioned in this application in addition to the prior art not emphasized or cited in the application. USPTO is aware of patent examiners’ lack of access to adequate information and their consequent inability to make the best decisions (USPTO Media Release,2007). This grievance is exacerbated by examiners having limited time to conduct their prior art searches (USPTO Media Release, 2007). These downsides hamper USPTO, which is struggling with a massive backlog, as shown by an unexamined patent application inventory of more than 570,000 in January 2020 (USPTO, 2020). The next subchapter will now explain what the Peer To Patent initiative is and how it might counteract the downturns of the patent examination process. 2.2. USPTO’s Peer To Patent (PTP) initiative The PTP Initiative was launched in June 2007 and consists of an online system using Web 2.0 technology to integrate external experts into the examination process – crowdsourcing – helping examiners to identify prior art and thus support them in their prior art search. PTP’s crowdsourcing process may be distinguished into five steps (displayed on the righthand side of Figure1): Step 1 consists of reviewing and discussing patent applications that were voluntarily submitted to PTP. Step 2 is research and find prior art, where members of the public conduct their research to identify reasonable prior art references which have to be mentioned or question the justification of a patent application. In step 3, prior art that is relevant to a patent application’s claims is uploaded. In step 4, participants can annotate and evaluate the entirety of the submitted prior art. In the final step, the top 10 evaluated prior art references are forwarded to the USPTO to enable the examiner of an application to gain access. The patent examiner makes the final decision based on legal standards. This process combines the democracy of open participation with the legitimacy and effectiveness of administrative decisionmaking as displayed in Figure1. The next chapter will now elaborate on the Figure 1. USPTO’s peer- to- patent process. Patent application filing Patent examiner: ■ Reviews the disclosure in the application and compare it with prior art. ■ Search for prior art –a patent needs to be novel and non-obvious ■ Inventors are not required to conduct their own prior art searches or to supply the patent office with prior art they are not immediately aware of. ■ During the examination of an application or reexamination of apatent, theexaminer should cite appropriate prior artthat is nearest to the subject matter defined in the claims. Decision: ■ When an application complies with the legal requirements a patent is granted. Step 1: Review and discuss posted patent application Step 2: Research and find prior art Step 3: Upload prior art relevant to claims Step 4: Annotate and evaluate all submitted prior art Step 5: „Top ten“ prior art references forwarded to USPTO The Peer To Patent Process © 2023 The Author. R&D Management published by RADMA and John Wiley & Sons Ltd. Crowdsourcing in patent examination R&D Management 53, 5, 2023 767 theoretical constructs that affect this procedure and how the public participation, in the form of crowdsourcing, of PTP is likely to influence the information basis of patent examiners’ decisionmaking. 3. Prior research and hypotheses 3.1. Prior research into patent examination Although previous research has revealed the importance of examination quality, surprisingly, there have been very few studies on the inclusion of externals in the patent examination process. Especially, in the aforementioned example, a single engineer found evidence of prior art that was not considered by the official examiner. Three studies that have analyzed the patent examination process and the inclusion of externals stand out. First, Yamauchi and Nagaoka(2015) analyzed and found that outsourcing prior art searches lower the examination duration and likely leads to fewer appeals against examiner decisions (rejecting or granting). Similarly, Kim and Oh(2017) found that outsourcing prior art searches lowered the propensity to grant a patent and the likelihood of the reversal of an invalidation trial. However, both studies analyzed the outsourcing of prior art searches to third parties, in contrast to crowdsourcing, which enables an examiner to avoid the initial search and is depicted by a paid function subject to budgetary considerations. Crowdsourcing was analyzed in prior art searches in the form of an additional voluntary inclusion of individuals, enabling access to the wisdom of the crowd as an unpaid procedure. The third study, which closely relates to this research, was that of Kim and Mitra- Kahn(2020), which analyzed the potential effects of crowdsourcing in prior art searches. The authors focused on this procedure’s unintended outcomes, showing that crowdsourcing in prior art searches led to more requests for continued examination and increased forward citations of treated patents. The authors further showed that crowdsourcing of prior art searches increased examiners’ search efforts in the form of both increases in the number of search reports and the number of references added by an examiner following the first office action. Followed by elaborating on previous studies, taking the patent examination process into account, the next subsection derives this study’s hypotheses. 3.2. The effect of crowdsourcing on patent examiners’ knowledge The current research has identified that individuals residing within a single organization overrely on internal knowledge when searching for solutions to innovationrelated problems (March,1991; Helfat, 1994; Hippel, 1994; Martin and Mitchell, 1998), further called local search bias. Furthermore, local search bias is a widely accepted issue in innovationrelated outputs such as patents (Stuart and Podolny, 1996; Rosenkopf and Almeida, 2003). In order to overcome this bias, previous research has argued for the use of crowdsourcing. Crowdsourcing may be understood as the outsourcing of idea generation to a potentially large crowd through an open call (Howe,2006; Agerfalk and Fitzgerald, 2008; Afuha and Tucci, 2012). Organizations that engage in crowdsourcing seek to leverage the increasingly distributed knowledge of marginalized actors at convincingly low costs (Jeppesen and Lakhani, 2010). The construct wisdom of the crowd (Kremer et al.,2014) is key in this case. The research has analyzed crowdsourcing in the forms of innovation contests and collaborative communities (Boudreau and Lakhani,2009). Crowdsourcing is an accepted tool to overcome local search bias, and thus to look beyond existing knowledge sources and tap into external sources of innovation (Lüthje et al.,2006). Some of the associated benefits include problem resolution and the capturing of value from open innovation. Here, the argumentation of tapping into external knowledge sources is in the foreground. In line with this argument, the research has shown that solutions provided by crowdsourcing outperform those developed by company experts (Poetz and Schreier,2012). Further, Franke et al.(2014), have shown that problemsolvers from analogous markets generate more novel and less feasible solutions than solvers from the innovation task’s focus market. To conclude, research has found that aggregated group solutions outperform those individuals (including experts) involved in various tasks (Budescu and Chen,2015). In sum, the inclusion of externals into the patent examination process – in particular, prior art searches – via crowdsourcing is likely to provide better outcomes, compared to internal experts’ solutions (Poetz and Schreier,2012; Franke et al.,2014) resulting in a lower likelihood of accepting patent applications. Thus, when examiners are provided with information coming from crowdsourcing, they are more likely to identify unjustified patent applications leading to higher rejection rates. Patent applications undergoing the PTP procedure are thus less likely to become granted patents. Hypothesis 1 Crowdsourcing decreases the likelihood of patent examiners’ accepting a patent application. © 2023 The Author. R&D Management published by RADMA and John Wiley & Sons Ltd. Hannes W. Lampe 768 R&D Management 53, 5, 2023 As argued above, local search bias might be an obstacle to the patent examination process. Furthermore, research has disentangled two causes for the occurrence of local search bias. Lakhani (2006, p. 2450) argued that “Problemsolvers residing within a single organization will still face some level of solution myopia due to the impact of locallypreferred solution algorithms and convenientlyavailable local information.” Thus, according to Lakhani(2006), local search bias may occur due to two separate restrictions: (1) conveniently available information and (2) locally preferred information. The author will now elaborate on why these restrictions and, thus, a local search bias are likely among patent examiners, as well as how crowdsourcing is likely to counteract these effects. First, the conveniently available information may be a reason for local search bias in the patent examination process. Prior research has argued that knowledge might be divided into formalized knowledge, expressed in specifications, objects as well as textbooks, and unformalized knowledge, depicted via theoretical models or models of behavior and perspectives based on empirical data and experience (Nonaka and Takeuchi,1995). Stewart(1997) argues that formalized knowledge consists of elements such as intellectual property and databases, for instance in form of patents. Contrarily less formalized knowledge is understood as published articles or conference presentations, here depicted as nonpatent references. As noted, patent examiners are restricted in accessing such less formalized knowledge (USPTO Media Release,2007; Allen et al.,2012). For instance, examiners have access to “some nonpatent literature, they do not have the same degree of access to much of the nonpatent prior art literature that exists, such as published articles, software code, and conference presentations” (Allen et al.,2012, p. 4). Their search effort is further restricted by examiners’ limited time to conduct their prior art searches (USPTO Media Release,2007). Research has further identified that individuals and organizations are not aware of the stock of less formalized knowledge available, and have no formalized way to access it (Du Plessis,2007). Pyka(2002) argues that innovators seek the needed information and knowledge from professional colleagues through informal networks as valuable knowledge is often available only in less formalized formats and collaboration is a quick and efficient way to access this knowledge. Thus, patent examiners likely have a local search bias owing to conventionally available local information in the form that less formalized knowledge is harder to access for them. As pointed out above, patents are easy to access for examiners, contrary to less formalized knowledge, in form of nonpatent prior art knowledge, which is more difficult (Allen et al.,2012). Here, this research sets off the argument that crowdsourcing in patent examination enables the integration of less formalized knowledge. Thus, integrating externals into prior art searches via crowdsourcing enables access to additional knowledge sources that are likely not as formalized (e.g., nonpatent literature). Externals may have access to additional knowledge sources (Lüthje et al.,2006) in form of less formalized knowledge for instance via informal networks (Pyka, 2002). By using crowdsourcing, the patent examination process, and consequently the patent examiner is likely to overcome the inherent local search bias due to conveniently available information depicted via formalized knowledge as patents. Hypothesis 2 Crowdsourcing induces decisionmakers to rely more on less formalized knowledge. Second, local search bias is understood to lead individuals to overly rely on internal expertise, decreasing the probability to find alternative solutions (Lakhani,2006). This point is crucial since examiners are likely to be specialists in a certain technological domain. For instance, Righi and Simcoe (2019) studied examiner specialization and found that examiners handle more applications from a given technology subclass or assignee than expected under a random allocation. However, specialization also means a narrower focus on certain technological subclasses – another reason for local search bias (Lakhani, 2006). Similarly, Fleming(2001) argues that individuals inevitably become narrower in their expertise as the body of knowledge expands, especially given the difficulty of searching unfamiliar domains. Again, this might be exacerbated by the time examiners have for their prior art searches (USPTO Media Release, 2007; Allen et al., 2012), which likely increases the potential effects of their local search bias. The innovation research has argued that the narrow recombination of similar knowledge can be seen as a local search (Gavetti and Levinthal,2000; Fleming,2001; Ethiraj and Levinthal,2004; Kaplan and Vakili, 2015). Thus, examiners seem well aware of prior art in the same technology class as the examined patent. Further, examiners are likely to be aware of the prior art typically referenced in their specialized domain. However, examiners are less aware of atypically referenced technology © 2023 The Author. R&D Management published by RADMA and John Wiley & Sons Ltd. Crowdsourcing in patent examination R&D Management 53, 5, 2023 769 classes, resulting in a local search bias that favors references typically associated with technology classes. Via crowdsourcing, examiners are likely to overcome their locally preferred solutions by including more atypical knowledge. Having a broader audience identifying prior art is likely to include a wider range of prior knowledge, which is likely to come from technological domains typically not associated with the focal patent’s technology class. Thus, via the PTP, examiners’ references’ technology classes will become more atypical: Hypothesis 3 Crowdsourcing induces decisionmakers to rely more on atypical knowledge. 4. Method and data The above Hypotheses focus on two different aspects of the examiner’s decisionmaking: Hypothesis 1 focuses on the overall likelihood to grant a patent, whereas Hypothesis2 and 3 proposed two information characteristics: formalized and atypical knowledge. Thus, two separate datasets had to be built to test these hypotheses. Dataset 1 tested for a higher likelihood to grant a patent application (H1). To examine if PTP- treated patent applications have a higher likelihood to become granted patents, this dataset took patent applications into account. The Patent Examination (PatEx) Research Database, sourcing data from the Public Patent Application Information Retrieval system (Public PAIR) (Graham et al., 2015) was used to build dataset 1. The original Public PAIR dataset contained information on 11,125,755 patent applications, which was then matched with the PTP- treated applications (this information was downloaded from the PTP homepage – https:// www.peert opate nt.org – and included 520 patent applications that underwent the PTP). Using the United States Patent Classification (USPC) technology classes (36) and the associated filing year (2005 to 2011) of each application in the PTP sample, all applications were further subsetted, matching these filing year/technology class combinations, which resulted in 375,532 patent applications. Again, the dataset comprises patent applications with the same technology classes and filing years as the PTP- treated patent applications, not blurring the results with effects associated due to different acceptance rates over the years or technology classes. Deleting applications without a final decision status (granted or not granted) resulted in 338,051 patent applications (of which 520 were part of the PTP initiative). This dataset was then used to test H1. To build the second dataset, testing H2 and H3, which focus on information characteristics – formalized and atypical knowledge – of patent examiners, additional information on references made by the examiner was needed. This information is given for granted patents by the PatentsView database (Leydesdorff et al.,2017). Similar to the building of dataset 1, here granted patents with the same technology class and filing year as the PTP- treated patents were included. Owing to focusing only on granted patents, the observations dropped to 221,148.3 In addition – as the focus here lies on the examiner references – the dataset was further subsetted to granted patents examined by the same examiners as those that underwent the PTP treatment, which resulted in 6806 observations. 4.1. Case– control samples Case– control samples are used to restrict or subset the dataset based on a certain restriction. In addition to the base case (explained above for each dataset) information on patent’s abstracts and patent citations was used. Unfortunately, these were only available for granted patents, and thus only the case– control of the second dataset was altered. In addition to base restrictions (technology/year classification), only those patents with the same number of forwarded citations (as the PTP- treated ones) were kept to allow for differences due to a patent’s value and technological success (Fleming, 2001; Jung and Lee,2016). Further, the topic modeling (Blei et al., 2003; Kaplan and Vakili,2015) was used to build another case– control sample. Topic modeling allows one to uncover automatically identified topics4 and themes that are latent in a collection of documents and to detect which theme composition best accounts for each document. The algorithm then uses this information to detect document similarities. This information was then used to subset the dataset to those patents that underwent the PTP procedure and their most similar (based on their abstracts) counterparts (not part of the PTP program) enabling a robust control sample (Arts et al.,2018). This sample consisted of 582 granted patents. 4.2. Dependent and independent variables To test the hypotheses, three models were built, each with a different dependent variable. Hypothesis 1 (based on dataset 1) focused on the likelihood of a patent application becoming a granted patent. To © 2023 The Author. R&D Management published by RADMA and John Wiley & Sons Ltd. Hannes W. Lampe 770 R&D Management 53, 5, 2023 test whether a PTP- treated patent application is less likely to be granted, a dummy variable, indicating if a patent application was granted or not, was used. Hypothesis2 assumed that crowdsourcing induces decisionmakers to rely more strongly on less formalized knowledge. To measure if the PTP initiative led to higher use of less formalized knowledge, the number of nonpatent references in a granted patent was used. In particular, the total nonpatent references as a proxy for those made by the examiner were used, since no further information on these reference types was given. However, when the nonpatent references increase, the share of nonpatent references made by an examiner may also increase. Hypothesis 3 proposed that PTP- treated patents show more atypical references made by the examiner. To test this hypothesis, the author constructed a variable that measures how atypical references made by the examiner were. Following the previous research and arguing that examiners are specialized and experienced in certain technology classes (Righi and Simcoe, 2019), to measure the atypicality of examiner references based on the referenced patent’s technology class compared to the focal patent’s technology class. Atypical technology class recombinations (comparison of cited and focal patent technology class) by examiner added references of granted patents were used as the third dependent variable. This measure was built by adopting and modifying a measure from Lo and Kennedy(2015) in two steps. First, the proximity index for each pair of technology classes i and j was computed. This index measures how typical two technology classes are in their combination. c depicts the focal patent’s primary technology class, while j refers to the primary technology class of a prior art patent reference added by an examiner. The proximity index Pij for two classes i and j is the average of two proportions; thus, it is calculated by dividing Cij , by the total number of times a technology class is mentioned in the entire sample ( Ci ) and the reference’s technology class is referenced ( Cj ) to, respectively. Cij is the number of referenced (by examiner) patents from technology class j in technology class i patents and thus depicts the number of times one technology class is cited in another technology class (aggregated over patents). This can be written as follows: where Cij is the number of times a patent of classes i had an examiner reference a patent with class j. Ci are the total number of times that class i appeared as the primary technology class in patents, and Cj depicts the total number of examiner references to a patent with primary technology class j in a certain time span. For the base case, the focal patent’s filing year as the time span was used. For additional robustness testing, this measure was built using 1- year, 3- year, and 5- year time windows. For instance, a 3- year time window takes not only citations of patents in the focal year into account, but further includes citations from patents applied for in the previous three years to measure the typicality of examiner references. Second, to measure how typical an examiner reference is, compared to the entirety of all examiner references, the proximity indices for every pair of classes were averaged. Larger scores imply that a class combination occurs together more often (and is thus more typical), while smaller scores mean a more unusual or atypical blend of technology class combinations. Typicality was measured as follows: Here, T is a patent examiner referencing typicality concerning the focal patent’s technology class. Pij are the measures of proximity for each pair of classes, and thus, the focal patent’s technology class and the examinerreferenced patent’s technology class. L is the number of the focal patent’s references coming from the examiner. To measure atypicality, this variable was then subtracted from 1, resulting in the dependent variable atypicality of examiner references. A more extensive derivation of this atypicality measure can be found in Lo and Kennedy(2015). 4.3. Independent and control variables The unique feature of the PTP initiative is that this treatment enables the comparison of the patents that underwent the PTP procedure with those that did not. To test this study’s hypotheses, the focal independent variable is therefore a dummy variable, indicating one if a patent (application) was part of the PTP procedure and zero otherwise. All three models included dummies for the filing year and the technology class of an application or patent to control for differences in the dependent variable due to these two factors. Since recent research has identified the key role of examiner specialization and experience (Righi and Simcoe,2019), a measure for examiner experience was included in all models. This measure depicts the number of years an examiner has been examining patent applications related to the filing year of the focal patent or patent application. In Models 2 and 3, which considered granted patents, the total number of claims of (1) P ij =1 2( C ij C i + C ij C j) (2) T = ∑ Pij [L(L− 1 )]∕2 © 2023 The Author. R&D Management published by RADMA and John Wiley & Sons Ltd. Crowdsourcing in patent examination R&D Management 53, 5, 2023 771 a patent was further used (Kaplan and Vakili,2015; Jung and Lee, 2016) (since this information was only available for patents, not applications). Model 3 further included controls for the total number of references (Fleming,2001) and the number of nonpatent references (Jung and Lee,2016). 4.4. Estimation methods To test the hypotheses, three dependent variables were used. All of these variables differ in their characteristics and distributions, and these differences were taken into account by using different estimation approaches. Model 1, which tests H1, used a dummy variable as a dependent variable, leading to the use of logistic regression (Maddala and Lahiri,1992). To test H2, this study incorporated the number of nonpatent references, a typical count variable that cannot assume values smaller than zero and corresponds to counts (Lampe and Reerink,2021). Following Hausman et al.(1984), the obvious approach would be to use a Poisson model. However, additional tests showed that this variable is overdispersed. This led to the use of a negative binomial model. Third, the testing of a patent’s reference atypicality was expressed via a continuous variable following a normal distribution, which led to an ordinary least squares (OLS) regression to test the predicted effect. The literature has identified the examiners’ key role in several aspects of patents (see Lemley and Sampat,2012; Righi and Simcoe,2019). To account for the importance of examiners and to allow for serial autocorrelation, clustered standard errors were used, treating each examiner as a cluster in all models. Table1 presents the descriptive statistics and a correlation matrix for the variables used in Models 2 and 3. 5. Results Table2 presents the final regression results. Model 1 shows the results of the logistic regression in combination with the patent application dataset to test for H1. Models 2 and 3 used the final dataset on granted patents. Model 2 used a negative binomial regression model. Model 3 tested H3 and used an OLS regression. Model 1 tested for H1, predicting that a PTP- treated patent application is less likely to be granted. The PTP treatment’s effect was insignificant; thus, H1 was not supported. Previous research was also unable to show a significant relationship between crowdsourced prior art searches and the likelihood of a patent application is granted (Kim and Mitra- Kahn,2020). H2 predicted that a PTP- treated patent is likely to have more nonpatent references, and thus a higher use of less formalized knowledge. Model 2 showed that this effect was significant ( 𝛽 = 0.342, P < 0.01); thus, H2 was supported. Model 3 tested H3. The proposed positive effect of the PTP treatment on the atypicality of examiner prior art references ( 𝛽 = 0.061, P < 0.05) was supported. To challenge the current study’s findings’ robustness, several robustness tests were conducted. First, to test H3, the dependent variable was determined not only with respect to the focal patent’s filing year, but further rebuilt the variable using 1- year, 3- year, and 5- year time windows. The results were qualitatively consistent. Further, the matching procedure was challenged to build the case– control sample. Then, PTP- treated patents (again building on the filing year/technology class combinations) were matched with those with the same number of forwarded citations to allow for differences due to a patent’s value and technological success (Fleming,2001; Jung and Lee,2016). The results based on this matching procedure appear in Table3 (models 1 and 2). Further, I used topic modeling (Blei et al.,2003; Kaplan and Vakili, 2015) to build another case– control sample, consisting of 582 granted patents. This dataset was then used to test H2 and H3. The results are displayed in models 3 and 4 in Table3. Overall, the results remained the same. Only for model 4, which tested H3, was a drop in the significance level observed. This may be due to the drop in the number of observations. Table 1. The descriptive statistics Mean Mean SD 1 2 3 4 5 1Nonpatent references 8.67 26.54 2 Atypicality of examiner references 0.76 0.40 0.03 3PTP- treated (dummy) 0.05 0.21 0.04 0.02 4 Examiner experience (years) 11.94 10.85 −0.01 0.07 −0.05 5 Number of claims 18.26 10.25 0.14 0.03 −0.03 0.01 6 Number of references 17.50 38.32 0.61 0.10 0.01 0.05 0.11