Methodological issues related to the use of online labour market data

Fabo, Brian; Kureková, Lucia M´ytna Working Paper Methodological issues related to the use of online labour market data ILO Working Paper, No. 68 Provided in Cooperation with: International Labour Organization (ILO), Geneva Suggested Citation: Fabo, Brian; Kureková, Lucia M´ytna (2022) : Methodological issues related to the use of online labour market data, ILO Working Paper, No. 68, ISBN 978-92-2-037282-1, International Labour Organization (ILO), Geneva, https://doi.org/10.54394/ZZBC8484 This Version is available at: https://hdl.handle.net/10419/263129 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/3.0/igo/ XMethodological issues related to the use of online labour market data Authors / Brian Fabo, Lucia Mýtna Kureková June / 2022 ILO Working Paper 68 Copyright © International Labour Organization 2022 This is an open access work distributed under the Creative Commons Attribution 3.0 IGO License (http:// creativecommons.org/licenses/by/3.0/igo). Users can reuse, share, adapt and build upon the original work, even for commercial purposes, as detailed in the License. The ILO must be clearly credited as the owner of the original work. The use of the emblem of the ILO is not permitted in connection with users’ work. Translations – In case of a translation of this work, the following disclaimer must be added along with the attribution: This translation was not created by the International Labour Office (ILO) and should not be considered an official ILO translation. The ILO is not responsible for the content or accuracy of this translation. Adaptations – In case of an adaptation of this work, the following disclaimer must be added along with the attribution: This is an adaptation of an original work by the International Labour Office (ILO). Responsibility for the views and opinions expressed in the adaptation rests solely with the author or authors of the adaptation and are not endorsed by the ILO. All queries on rights and licensing should be addressed to ILO Publications (Rights and Licensing), CH-1211 Geneva 22, Switzerland, or by email to [email protected]. ISBN: 9789220372814 (print) ISBN: 9789220372821 (web-pdf) ISBN: 9789220372838 (epub) ISBN: 9789220372845 (mobi) ISBN: 9789220372852 (html) ISSN: 2708-3446 https://doi.org/10.54394/ZZBC8484 The designations employed in ILO publications, which are in conformity with United Nations practice, and the presentation of material therein do not imply the expression of any opinion whatsoever on the part of the International Labour Office concerning the legal status of any country, area or territory or of its authorities, or concerning the delimitation of its frontiers. The responsibility for opinions expressed in signed articles, studies and other contributions rests solely with their authors, and publication does not constitute an endorsement by the International Labour Office of the opinions expressed in them. Reference to names of firms and commercial products and processes does not imply their endorsement by the International Labour Office, and any failure to mention a particular firm, commercial product or process is not a sign of disapproval. ILO Working Papers summarize the results of ILO research in progress, and seek to stimulate discussion of a range of issues related to the world of work. Comments on this ILO Working Paper are welcome and can be sent to [email protected], [email protected]. Authorization for publication: Richard Samans, Director RESEARCH ILO Working Papers can be found at: www.ilo.org/global/publications/working-papers Suggested citation: Fabo, B., Mýtna Kureková , L. 2022. Methodological issues related to the use of online labour market data, ILO Working Paper 68 (Geneva, ILO). 01 ILO Working Paper 68 Abstract This report provides a mapping of existing research that employs online labour market data, covering both online job vacancies (demand side) and online applicant data (CVs) (supply side). We discuss and assess a variety of tools and empirical methods that have been used to address specific disadvantages of this data, such as non-representativeness or fluctuations in data quantity and structure; these may be due to external shocks, such as the COVID-19 pandemic. We find that while this research field has expanded rapidly, including with respect to geographical coverage, many empirical studies do not engage with the methodological aspects and weaknesses of online labour market data and take them at face value. We highlight that there are legitimate research approaches, which are inductive in nature, focused on discovering patterns and trends in underlying data. These are by definition less concerned with generalizability of findings, as they have different objectives. For this body of research, online labour market data open new avenues for understanding developments in labour markets. We also argue that biases in online labour market data emerge due to multiple factors. With respect to the order of discrepancies between online labour market data and representative data sources, these are typically not paramount. Different techniques have been adopted to deal with the non-representativeness problem, such as statistical techniques; adapting the research questions and research focus to the quality of data; and use of mixed methods, including qualitative methods, to increase the robustness of results. About the authors Brian Fabo (PhD) is a lecturer in Economics at the Department of Public Policy, Comenius University in Bratislava, Slovakia, and a Senior Economist at the National Bank of Slovakia. His research focuses on the application of novel data sources in social science research, digitalization, and bias in research. Lucia Mýtna Kureková (PhD) works as a senior researcher at the Slovak Academy of Sciences, Centre for Social and Psychological Sciences. She is a labour market researcher focusing on skills demand and skill changes, big data in labour market research, labour migration and migrant integration, and labour market inequalities and social inclusion. 02 ILO Working Paper 68 Abstract 01 About the authors 01 XIntroduction 05 X1 Methodology 07 X2 Online data in labour market research: Trends and characteristics 08 X3 Advantages of online labour market data 11 X4 Sources of biases in online labour market data 13 X5 Methodological aspects of online labour market data 16 A. Describing data processing techniques 16 B. Key conceptual starting points: Deductive versus inductive science 16 C. Mapping the degree of discrepancies between online data and representative data 17 D. Fluctuations in online labour market data 21 E. Techniques and approaches to address non-representativeness and other biases 22 i. Statistical techniques 22 ii. Research design approach: Adapting the research questions and research focus 24 iii. Multimethod research 24 XConclusion and implications 25 Annex 27 References 30 Acknowledgements 40 Table of contents 03 ILO Working Paper 68 List of Tables Table 1: Nature of discrepancies between online data and representative data: selected studies focusing on skills analysis 19 Table 2: Overview of studies using online labour market data 27 04 ILO Working Paper 68 List of Boxes Box 1: Alternative online data sources relevant for labour market analysis 10 Box 2: Skills, tasks, occupations: What do we see in online vacancies? 11 Box 3: Policy initiatives using online labour market data 12 Box 4: Fluctuations in online labour market data during the COVID-19 pandemic: Selected findings  22 05 ILO Working Paper 68 XIntroduction Mismatches between workforce skills and employers’ demands represent a key obstacle to economic growth; they are closely associated with factors such as productivity, unemployment, labour force participation and informality (Acemoglu and Autor 2010; Beblavý, Maselli, and Veselková 2014; 2015; CEDEFOP 2014; Ernesto and Francesco 2016). Skill and task demand is changing due to rapid technological advancement, automation and digitalization – processes that are relevant not only to developed economies, but also to emerging and developing ones (Comyn and Strietska-Ilina 2019). The ability to respond to the changing skills demand is considered key to successful economic transitions that are inevitably sought by economies and individuals throughout the world, and for developmental catch-up between higherand lower-income countries. The improved understanding of how required skills and work tasks are changing, and the quest to better align skill supply to employer demand, have inspired efforts to create more demand-driven labour market policies. Timely and reliable data are a key prerequisite for these efforts. The online data on labour markets have in the past years become an important source of information for better understanding how labour markets function. This process has been affected by the spread of the Internet and emergence of online labour market intermediary platforms (e.g. Babajobs in India, Glassdoor in the United States (US), Profesia in Slovakia), online vacancy aggregators (e.g. Burning Glass Technologies), and professional websites and social media (e.g. LinkedIn, Twitter, Facebook). These types of labour market data currently provide a source of timely, granular and often comprehensive data that has been increasingly used by academics and policy makers to analyse labour markets around the world. The use of such data has been driven throughout the world by the fact that traditional representative surveys might not be available, or do not cover more specific aspects of labour markets in sufficient detail and frequency. The absence of high-frequency, detailed survey data has led researchers to revert to a second-best solution of using online data to study diverse questions. More traditional micro-economic questions (focusing on the behaviour of firms and individuals, skill supply, skill demand, matching, and skill-biased technological change) or macro-economic questions (such as predictions of the unemployment rate, aggregate demand, broader phenomena, and changes at the national, regional or local levels) have been analysed with the use of online data (Boselli, Cesarini, Marrara, et al. 2018). Furthermore, new questions or approaches to a structured understanding of labour market characteristics or changes have also emerged in relation to online data availability (for instance, building skill or task taxonomies, building curricula based on identified demand, a deeper understanding of job changes, etc.). In general, research using online data to study labour market issues has been organized around five related aspects of research: labour market monitoring and analysis; studying demand for workforce skills; observing job search behaviour and improving skill matching; predictive analysis of skill demand; and experimental studies (Nomura et al. 2017). Due to the granularity of the data, research in these areas has been conducted also at sub-national levels, examining regional or local labour markets (e.g. Azar et al. 2019; 2020; Marinescu and Rathelot 2018). The use of online labour market data, however, is not without disadvantages. The key concern is the non-representativeness of online data, and the implications this has for various aspects of research and policy making. Other data quality issues relate to data validity, reliability, scalability, generalizability, integrity or privacy, and legal issues (Blazquez and Domenech 2018). This paper situates itself within the debate on the methodological appropriateness of using online labour market data for academic and policy purpose, and provides a systematic review and discussion of: (1) the types and forms of biases present in online labour market data, and ways in which these are understood, discussed and addressed by research; and (2) measures and tools – statistical and other that have been used in past academic and policy research to remedy biases of online labour market data, with a particular focus on two dimensions: representativeness, and fluctuations in online labour market data. 06 ILO Working Paper 68 We build on previous studies that have discussed methodological aspects of big data more generally, including implications for the development of new analytical approaches and tools (Blazquez and Domenech 2018; Einav and Levin 2014; Mezzanzanica and Mercorio 2019; Varian 2014). We differ from these studies by providing a narrower focus on online labour market data, such as job vacancies and applicant data. Nevertheless, in particular parts of this paper we refer to broader conceptual issues of different motivations for research: for instance, deductive versus inductive approaches to gathering and analysing information. 13 ILO Working Paper 68 X4 Sources of biases in online labour market data Notwithstanding the evident advantages discussed in the previous chapter, the online labour market data suffer from various biases, particularly a lack of representativeness. Non-representativeness in applicant data might have different causes to that affecting vacancy data. With respect to job applications and job searches, the main source of non-representativeness is linked to the fact that the universe of jobs intermediated online is not equal to the universe of new jobs that exist. Internet access is an important driver of this, as individuals’ ability to access the Internet remains unequal across and within countries, and varies by socioeconomic status, age or skills. Other aspects that intervene in decisions about online job seeking include a sector’s level of informality, as well as the level of social capital that sustains referrals, which are more widely used in lower-skilled jobs and in smaller enterprises. Regarding vacancy data, the individual labour market segments are unlikely to advertise open positions to an equal extent. While Internet access has become less of an issue for firms, factors such as the intensity of labour demand, nature of work, level of informality in a given sector, or aspects such as firm size, all affect the likelihood of a vacancy being published online in the first place (Sostero and Fernandez-Macias 2021). Sectors such as construction or agriculture are in some countries less amenable to the use of online labour intermediation platforms, while micro and small enterprises are more likely to rely on informal and non-advertised hiring processes. We now turn to discussing these aspects in greater detail. The extent to which the population is connected to the Web varies greatly between different countries, but also within them. As evident from Figure 1, Internet access is close to universal in high-income countries, and is available to a majority in most upper-middle income countries and some lower-middle income countries. However, the majority of the population in low-income countries and many lower-middle income countries are still without access to the Internet. The poorest, least educated and the most distant from the labour market, even in high-income countries, are typically digitally disconnected (Warschauer 2003; van Dijk 2006; 2020; Scheerder, van Deursen, and van Dijk 2017). Furthermore, other specific groups such as females, older workers and rural populations are likely to be unable to go online in countries where Internet access is not widespread (Birba and Diagne 2012). Hence, information about online labour market matching is likely to have biases in less developed and emerging economies, due to limited or skewed Internet access. 14 ILO Working Paper 68 XFigure 1: Share of Internet users per population, correlated with GDP in 2017 on a log scale. Data source: World Bank: World Development Indicators (extracted 25 February 2022) In countries where Internet access is close to universal, the bias of analysis using Internet labour market data might be less significant (Askitas and Zimmermann 2015); however, there is still an observable bias in the data, leading to an over-representation of tertiary educated workers and job opportunities for better-educated workers (Muller and Safir 2019; Štefánik 2012). For example, Carnevale, Jayasundera and Repnikov (2014) studied Burning Glass Technologies data for the US labour market, and estimated that 80–90 per cent of job vacancies requiring a tertiary degree (bachelor’s and higher) are posted online, compared to about 40–60 per cent of job advertisements requiring a high-school diploma. In spite of this limitation, some researchers have used online labour market data to understand demand in low-skilled occupations or in unstable, typically less skilled jobs (Beblavý, Kureková, and Haita 2016; Kureková and Žilinčíková 2016). Another important dimension to consider is that of informal labour. From the existing ILO analyses, we know that six workers out of ten work in the informal economy. Unlike some past predictions, we know that this number is not necessarily decreasing. The issue is not limited to emerging countries, and particularly affects vulnerable populations such as women, uneducated people, or migrants (ILO 2021). The reasons for informal employment vary; for instance, enterprises might opt to operate informally to avoid regulations applying to a formal employment relationship. Additionally, even formal firms might employ workers informally; in some cases this reflects the preference of workers, as in the case of online crowdworkers preferring to make some quick money on the side. Thus, informal employment might be associated with lower numbers of vacancies being published either online or offline. Nevertheless, we see that some online job portals also cover the informal labour market, such as Babajobs in India (see Nomura et al. 2017). Next, particularly in the developing and emerging markets, a major part of the workforce finds itself in a self-employment arrangement, due to necessity or choice; even though their work is similar to that performed by employed workers (ILO 2016; Poschke 2019). Self-employed work does not typically generate vacancies (Dunlop 1966), and people might be particularly prone to being inaccurate when describing their self-employment experience in CVs (Jones 1984). 15 ILO Working Paper 68 In the formal economy, there are also reasons for not advertising jobs publicly. Enterprises and job seekers might opt for an informal (internal) approach for multiple reasons, including lower search costs, ability to avoid initial screening, and because seeking workers or work through informal networks is likely to result in opportunities and applicants located in the near vicinity (van Ours 1989). The sheer size of the online job markets demonstrates that there are many situations where a formal job search is nonetheless initiated; but we need to be mindful of the limited generalizability of any patterns identified in the job postings, even in countries with a high share of Internet users and an insignificant informal economy. That being said, the relatively low cost of advertising job vacancies (or of finding a worker via a CV posted online) might empower actors who would not have initiated formal recruitment in the pre-Internet era (Sodhi and Son 2010). In addition to non-representativeness, validity and reliability might also be of concern when using online data. Both vacancy and CV data are self-reported, and there are no tools embedded to check the validity and reliability of information provided. For example, Internet job boards can be flooded by resumés that in fact no longer correspond to people who are searching for a job – known as “stale” resumés (Kuhn 2014) – while the same might be true in the case of vacancies. Nonetheless, some researchers consider online information about job applicants to be more truthful and accurate (van Loo and Pouliakas 2020). Moreover, vacancies might be posted online even after the position has been filled, or one posted vacancy can in practice mean more job openings. These specificities warn of a measurement error due to duplicates and the lifetime of a vacancy. Research has also identified that firms might use vacancies as an advertising or company branding tool, which is likely to affect the choice of vacancy content (Winzenried, 2020). Particular concerns might also arise for cross-country comparative research. Existing studies have shown that employers in different countries seem to use very different strategies in terms of their expressed expectations for skills or education; this might be due to underlying differences in the functioning and institutional underpinning of national labour markets across Europe (Kureková et al. 2016). Similar differences have been identified among formally identical jobs advertised in different sectors, occupations and skill levels (Beblavý, Kureková, and Haita 2016; Brandas, Panzaru, and Filip 2016; van Loo and Pouliakas 2020). Brandas et al. (2016), who studied academic vacancies worldwide, also pointed out a lack of semantic and structural compatibility of data mined from different sources. Winzenried (2020) provided examples of job vacancies that greatly vary in the “density” of skills they require, and emphasized the importance of “implicit” knowledge in vacancy posting, which can be country-, sectoror occupation-specific (such as the significance of education or experience). 16 ILO Working Paper 68 X5 Methodological aspects of online labour market data In this chapter we present how the methodological weaknesses of online data have been addressed in existing research, by covering a range of statistical and other approaches. First, we describe data processing techniques, and explore more conceptual questions regarding the philosophy of research and its aims; we then discuss these in light of the research objectives, using online labour market data. A. Describing data processing techniques Online job vacancy data research is heavily focused on text classification, with the aim of making sense of the content of vacancies in order to identify skills, tasks or education requirements. Relatedly, research has tried to advance label classification, through matching vacancies to existing occupational standards (ISCO, ESCO), or national standards such as CGCO, the Chinese occupational classification (Kotu and Deshpande 2019; Xu et al. 2017). Finally, research has attempted to advance skill or task classification. Refer again to Box 2 for a list of findings based on skills analysis using online labour market data. Evidently, skills are also analysed from the perspective of job seekers, and the skill sets they attain. A typical processing strategy is to systematize data from CVs into respective categories. For some parameters, data can be easily turned into tabular and numerical data (such as education level), while textual analysis can be applied to process other parts of CVs. In essence, CVs include work histories and can be transformed into longitudinal data. Information in the CV can also be used to derive variables not directly present in the CV, such as foreign work experience (Kureková and Žilinčíková, 2018). Automated processes using machine learning techniques have also been developed to identify patterns in CVs. Within the typology of big data, online labour market data – vacancies and applicant data – belong to semi-structured data (Gandomi and Haider, 2015; Blasquez and Domenech, 2018). Across job portals or social media platforms, information that can be found with respect to a vacancy or a CV includes predictable and similar categories (such as education, sector, experience), which can then be organized into a structured format by Web scraping. Commercial websites often organize their content in a structured way, which indirectly supports and facilitates potential analytics on the basis of such data (for instance, the online job portal Profesia.sk in Slovakia; or the EURES portal which aggregates public employment services (PES) vacancies across the EU). B. Key conceptual starting points: Deductive versus inductive science Prior to outlining the dimensions of non-representativeness and fluctuations in online labour market data, we will highlight several conceptual points. These are based on discussions in existing studies that have theoretically (rather than empirically) engaged with online data. They also partly stem from observations derived from our review of various empirical studies using online labour market data. First, the character and quality of online data and big data generally, not just with respect to the labour market, have influenced the methodologies that researchers use to analyse them (Blazquez and Domenech 2018; Einav and Levin 2014; Mezzanzanica and Mercorio 2019; Varian 2014). Importantly, methodological developments are linked to different stages of data processing. Among the principal newer methods for accessing data are Web data mining and machine learning. Given the large amount of text, Natural Language Processing has progressed; this includes techniques such as Sentiment Analysis, Latent Semantic Analysis and Word 17 ILO Working Paper 68 Embedding (Blazquez and Domenech 2018). It is beyond the scope of this paper to discuss the respective methodological advances in relation to big data analytics extensively, and we refer the reader to other studies that have engaged at length with this topic (also more generally, beyond the online labour market data) (e.g. Blazquez and Domenech 2018; Einav and Levin 2014; Gandomi and Haider 2015; Kotu and Deshpande 2019; Mezzanzanica and Mercorio 2019; Varian 2014). Second, we consider the distinction between the different motivations and techniques of research, which can broadly be categorized as deductive and inductive. For the most part, empirical research using online labour market data is predominantly inductive and bottom-up, and often exploratory; rather than deductive and top-down, in the sense of aiming to test existing theories, concepts or relationships.1 Specifically, data are used to understand the underlying qualities of labour markets, skill characteristics, or trends identified through the longitudinal collection of online labour market data. This is also reflected in the analytical methods applied, and implicitly in a lesser concern with data characteristics – in particular, their representativeness, and whether there is a normal distribution. We find the inductive approach reflected in the aims and methodology of many studies that we covered in our unrestricted search of diverse studies (Phase 1). Table 1A summarizes key features from the studies reviewed in the first round of the analysis. It is evident that descriptive statistics, frequencies and correlations are frequently used methods, irrespective of the studied country. Furthermore, many studies do not discuss any aspect of bias of their data, and are not concerned with broader generalizability, beyond briefly acknowledging the issue in footnote or a short remark. In summary, unlike theory-testing and theoretically driven research that relies on probabilistic and inferential statistics, based on the assumption of normal distribution and requiring representativeness of data, inductive research is less concerned with representativeness and generalizability. Furthermore, inductive research using online labour market data also takes account of the fact that online data have no intrinsic value; rather, data value is extrinsic, given by the analyst who applies her/his knowledge in designing research with the respective dataset (Mezzanzanica and Mercorio, 2019; Gandomi and Haider, 2015). An example of such research is what has been termed the “KDD process – knowledge discovery in databases”, which inductively studies underlying features of large datasets to create patterns (Kotu and Deshpande 2019). Another example are studies that have used online labour market data to create frameworks or systems. For example, a study by Xu et al. (2017) used online vacancy data from Chinese labour recruitment websites to develop a framework for systematizing vacancies into Chinese occupational categories (CGCO). Likewise, Brandas et al. (2017) exploited global academic jobsites to set up a “Labour Market Decisions Support System”. C. Mapping the degree of discrepancies between online data and representative data Several recent studies have attempted to map out the scope of discrepancies between online data and representative data by comparing the sectoral and/or occupational distribution in the online data to an alternative source. An important observation is the fact that representativeness adjustments on the basis of representative data appear more appropriate for online applicant data than for online vacancy data, as we explain below. Firstly, in advanced economies, representative sources to understand the structure of the labour market are collected on an annual basis; these include the Labour Force Survey (LFS) and its alternatives (e.g. German Socio-Economic Panel (SOEP) data, Current Population Survey (CPS) in the US). From the perspective of labour supply, requiring the analysis of online applicant data, representative surveys such as LFS provide a 1 Blasquez and Domenech (2018) present these different approaches to research as Supervised Learning and Non-supervised Learning. 18 ILO Working Paper 68 good source to compare biases, and potentially then employ weighting on the basis of an underlying representative structure. For example, in their study of young return migrants (below 35 years of age), Kureková and Žilinčíková (2018) compared online CV data with LFS data in Slovakia, and found that the online sample had an unbiased gender and age distribution, but a bias towards people with tertiary education. Štefánik (2012), who studied representativeness and skill demand for graduates in the Slovak labour market, compared online CVs data with the structure of university graduates, and found a surprisingly good fit of online data and representative data. From the perspective of labour demand, however, labour force surveys capture the stock of jobs that exist in an economy at any given moment, while the online labour market data are a source for understanding the flows in the labour market. Online job vacancies do not capture the stock of matched or unmatched jobs, and represent only the labour market demand.2 In other words, online vacancy data depend on turnover rates, whereas survey data often represent a cross-section of workers. To illustrate this, there might be many civil servants in a country, but far fewer civil service openings, because public service workers tend to stay in their job for a long time. Occupations with a high turnover, such as odd jobs, tend to be advertised very often, because job holders in these occupations tend to move on to more lucrative jobs as soon as they can. Sostero and Fernandez-Macias (2021) showed that the ratio between the number of job holders and job vacancies can range from nearly 1:1 to 1:100, and even 1:1,000. We therefore do not consider labour force survey data to be an appropriate source for making adjustments to online job vacancy data in particular (Kureková et al. 2015). We failed to find papers which appeared to use firm-level data to adjust online job vacancies, but there are examples of research in which online CV data (work histories) were linked with representative business data: for instance, to study interfirm mobility and innovation (Masso et al. 2012) Secondly, making adjustments to online vacancy data is a strenuous task, because the population of vacancies is inherently unknown in most countries (Kureková et al. 2015). This is due to the reasons previously described – because hiring processes in firms have different underlying motivations, and hiring often takes places internally or informally. Moreover, an enterprise will advertise vacancies (online or offline) not only when it requires labour due to growth, but also due to replacement needs, such as in response to workforce turnover or retirement. It is therefore important to differentiate between the stock of demand for skills (a company hiring a replacement worker to compensate for attrition) and a flow of skill demand (a company reacting to IT innovations by hiring ICT specialists, creating demand for new skills). Furthermore, because filling a vacancy takes time, enterprises are likely to post vacancies when they anticipate requiring workers with a certain skill some time before they are actually needed (Ferber and Ford 1966). Finally, a need for new skill can be addressed by hiring (which will create a vacancy) or by retraining existing staff (which will not create a vacancy). Therefore, assessing skill demand only on the basis of vacancies, without considering the training investments in the companies, will not provide full information on skill demand (Holt and David 1966). To describe the degree of discrepancies between online data and representative sources, we have identified several studies that compare the properties of online job data to some measure of labour market flows, typically vacancies (Table 1). The availability of appropriate comparator data varies between countries, as some surveys have been used quite extensively, such as the Job Openings and Labour Turnover Survey (JOLTS) in the US, or Office for National Statistics (ONS) Vacancy Survey in the UK. Nonetheless, JOLTS has its own representativeness issues,3 as do vacancy databases maintained by public employment agencies in Europe (Drahokoupil and Fabo 2022; Hershbein and Kahn 2016). In most countries, however, firms’ reporting of vacancies to public employment services is voluntary, and as such, there is no readily available source for comparing the structure of labour demand. In summary, the key challenge is the problematic 2Matching can to some extent be proxied by, for example, the number of clicks on a particular vacancy, which suggests interest in a given position; or alternatively, number of views of a particular CV (Kureková and Žilinčíková 2018). 3Robust evidence exists that many hires (perhaps as high as 20 per cent), and thus the job openings, are not mediated through vacancies, which seems to result in systematic under-reporting of vacancies in JOLTS (Davis, Faberman, and Haltiwanger 2013) 19 ILO Working Paper 68 accessibility of such data – which is also one of the reasons for using the online labour market data in the first place. This is particularly the case for developing countries. In Table 1 we review several studies, and provide the basic conclusions of the comparison. The key finding is that while there are some discrepancies in the structure of vacancies, the general picture painted by the online job vacancies largely corresponds to other data sources. Importantly, some papers that have explored the biases of widely used online sources, such as Burning Glass data, are used by subsequent studies as a reference to understand the nature of discrepancies. For example, reference to the paper by Hershbein and Kahn (2016) appears widely in papers that study US labour market with Burning Glass data. Other studies refer to past methodological discussions (such as Kureková et al. 2015) in their brief acknowledgement of online data’s limitations (see Table 2). With respect to biases, the published studies largely concur that the share of online vacancies is over-represented in sectors such as ICT or finance; while those in hospitality, food service, manufacturing, and particularly public service, tend to be under-represented. Interestingly, some difference is observable between variants of capitalism – the healthcare sector tends to be over-represented in the US vacancies but under-represented in Europe, possibly due to the public sector’s much stronger role in Europe than in the US. Furthermore, white-collar, skilled jobs tend to be over-represented, while trades and manual positions tend to be under-represented. However, there is no clear line between white-collar vacancies being more posted online than blue-collar openings, as public jobs are prevailingly white-collar, but tend to be less advertised online. In addition to the type of work, hiring practices and turnover of jobs can be additional factors that shape the probability of vacancies and their volume appearing online. Overall, while some types of jobs might be under-represented in respective online labour markets, sample sizes nevertheless tend to be sufficient to conduct an effective analysis. Moreover, as explained above, for certain (typically exploratory) questions, these biases are of secondary importance and do not prohibit further analyses. XTable 1: Nature of discrepancies between online data and representative data: selected studies focusing on skills analysis Study (reference) Location of relevant information Source of online job vacancies (OJVs) Alternative data source Discrepancies identified Hershbein and Kahn (2016) Section A.1 of the Internet appendix contains a lengthy discussion of representativeness of the OJV data. This source is cited by many empirical papers. Burning Glass US data Sectoral structure comparison with JOLTS Occupation distribution comparison with CPS New Jobs and OES, including in time Sectors match reasonably well; Burning Glass (BG) is over-represented in healthcare and social assistance (+2 pp), finance and insurance, and education. It is under-represented in accommodation and food services (-5 pp), public admin/ government, and construction. BG has a much larger representation of computer and mathematical occupations (four times higher than shares in OES and CPS), as well as management, healthcare practitioners, and business and financial operations. On the other hand, BG data are under-represented in transportation, food preparation and serving, production and construction, among others. Representativeness in terms of occupation structure is found to be largely constant across time. 20 ILO Working Paper 68 Study (reference) Location of relevant information Source of online job vacancies (OJVs) Alternative data source Discrepancies identified Burke et al. (2020) Section A.1.2. of the Internet appendix Burning Glass US data Sectoral structure, including over time. Occupation based on Minnesota Job Vacancy Survey The BG data are over-represented in finance and insurance (+7 pp), healthcare and social assistance, education services. Meanwhile, they are under-represented in food services and accommodation (-7 pp), public administration and government, and construction. Changes over time are very small. Occupation-wise, BG data are over-represented in mathematical and computer professions (+9 pp), management (+7 pp), and business and financial (+6 pp). They are under-represented in food preparation and serving (-7 pp), healthcare (-5 pp), and transportation (-4 pp). Acemoglu et al. (2020) Figure A1 Burning Glass US data Sectoral and occupation comparison with JOLTS and OES Findings closely match Hershbein and Kahn (2016) and Burke et al. (2020). Turell et al. (2019) Section 3.2.”Coverage and representativeness bias” Reed co.uk – a leading UK job site Sectoral breakdown based on ONS Vacancy Survey The mean annual ratios of the individual sectors in the two data sources are compared. For professional and scientific activities, ICT and administration, the Reed data are of comparable magnitude to the ONS estimates of vacancies. The largest differences were identified for public administration and manufacturing. Sostero and FernandezMacias (2021) Section 5 Online job ads as representation of the labour market Burning Glass UK data Economic survey by ONS (job stocks, not vacancies) Professional, associate professional and administrative occupations show a ratio approaching one advertisement for one person employed. Whereas in elementary occupations and skilled trades, there is one vacancy for 100 or even 1,000 people employed. Drahokoupil and Fabo (2022) Tables 3 and 4 Profesia.sk – a leading job site in Slovakia Sectoral and occupational comparison with vacancies registered by the public employment agency (UPSVAR), including over time Professionals over-represented in OJVs (+8 pp to +15 pp), crafts and trades under-represented {-3 pp to -10 pp). Occupations-wise, ICT is very over-represented (+12 pp to +17pp), while manufacturing are under-represented {-6 pp to -12 pp), transportation (-9 pp to -10 pp), and education, healthcare and culture (-6 pp to -14 pp). The pattern is largely stable over time. Marinescu (2017) Discussion in text, p.17 CareerBuilder.com Sectoral structure comparison with JOLTS Geographical distribution of vacancies with JOLTS IT, finance and insurance, real estate, rental and leasing are over-represented, while local government, accommodation and food services, other services, and construction are under-represented. CareerBuilder has the same geographic distribution as JOLTS. Similar overtime trends are also identified, with a correlation of 0.57 (statistically significant at 90 per cent) Kureková and Žilinčíková (2018) Table 2, descriptive statistics Profesia.sk – a leading job site in Slovakia Comparing online CVs to Labour Force Survey data, examining demographic characteristics Online data are unbiased by gender and age, but are biased towards individuals with higher education. Analysis is restricted to young people aged up to 35 years. 21 ILO Working Paper 68 D. Fluctuations in online labour market data The fluctuations in online labour market data intake have been studied extensively, in particular for online job vacancies. It appears that the online labour vacancy intake fluctuates for a variety of reasons. Fluctuations as such are unproblematic when they reflect actual changes in the labour market, but they might be a methodological concern if they include changes in biases over time. However, we are not aware of any literature discussing the representativeness implications of flows changing over time. Most studies we identified used fluctuations in online job vacancy data to identify and measure actual labour market changes, and found online job vacancy data to be an accurate representation of real shifts. Fluctuations can reflect economic cycles and macroeconomic trends, while they might also be indicative of structural changes within or between occupations. De Pedraza and his co-authors (2019), building on standard labour economics literature, decomposed the variation into three components: trend cycle, seasonal, and irregular (reflecting macroeconomic shocks). They identified similar patterns in all three components, in online data and a National Statistical Office dataset. They also found an underlying trend of increase in the number of online job vacancies over time, which is likely to reflect the increased importance of the Internet as the “labour market matchmaker”. Importantly, the online job vacancies data are capable of capturing fluctuations not just in the number of vacancies but also in their structure, which is possibly relevant for understanding skill demand. For example, Beblavý, Kureková and Haita (2016) pooled job vacancy data for a number of years (2007–2011) to enlarge the underlying sample. They noted that lowand medium-skilled vacancies grew in their relative share among all vacancies in 2010 and 2011, and interpreted this as a (structural) rise in demand for lessskilled jobs in the Slovak labour market, during the post-2008 recovery phase. They focused their analysis on identifying skill intensities of traditional jobs (electrician, cook, driver), as well as “new” occupations in the context of structural and technological advances, such as courier or porter, to identify how these are seen in the Slovak labour market. Hence, in their case, shifts across occupations were a focus of their analysis, and not a problem of the data. Nonetheless, there is a precedent for an online data source being found unreliable, that was once thought to be robust,. Google Flu Trends is a good example: for a considerable time, it predicted the actual doctor visits fairly well, but strongly overestimated the growth of infections, in reaction to a flurry of media reports about a flu pandemic causing people to search for flu symptoms even when not feeling sick (Lazer et al. 2014). Overall, the general picture appears to be that the differences between online job vacancies and alternative vacancy data, in terms of sectoral and occupational structure, remain quite stable over time (Burke et al. 2020; Drahokoupil and Fabo 2022; Hershbein and Kahn 2016; Lovaglio, Mezzanzanica, and Colombo 2020). This is an important consideration from the analytical perspective, and some researchers (e.g. Hershbein and Kahn, 2016) have used this longitudinal stability as a justification for research using online data. Long-term trends notwithstanding, the important strength of online labour market data lies in the ability to identify the “irregular” movements in skills demand caused by macroeconomic shocks. For example, the COVID-19 pandemic represented a major shock for the labour market that had a very uneven impact in different sectors: while some segments of the labour market, such as the ICT industry, seamlessly shifted to working from home and saw demand for their services increase, areas such as hospitality or tourism were devastated (Kahn, Lange, and Wiczer 2020). Box 4 summarizes some of the key findings from this literature regarding changes in online data. The overall observation is that labour market shock was widespread across sectors, and that online labour market data well described real trends in the respective economies. Discussions about fluctuations in online data related to the COVID-19 pandemic further stress that online data need to be interpreted in context, and with knowledge of the particularities of specific labour markets. 22 ILO Working Paper 68 XBox4:FluctuationsinonlinelabourmarketdataduringtheCOVID-19pandemic:Selectedfindings ●The OECD used the Burning Glass dataset to observe the impact across industries. The report also shows a strong correlation between the stringency of the pandemic measures and the decline in posted online job vacancies in the US (OECD 2021). ●A study using data from a popular Swedish private online job board found that together with a sharp decrease of vacancies, the average number of clicks per vacancy also declined because fewer people were actually looking for jobs. The decline in demand was most severe in the sectors particularly affected by the pandemic (Hensvik, Le Barbanchon, and Rathelot 2021). ●A similar trend was observed in Slovakia with the dominant online job portal Profesia.sk, with a marked dip in both demand and supply for labour after March 2020. The labour market recovered over the next months, and growth fully resumed in 2022, overtaking the 2019 levels. The greatest demand in March 2022 existed in occupations which were hit the most by the lockdown measures, specifically hospitality and catering (Profesia 2022). ●Another study focusing on the US labour market, based on Burning Glass data, found evidence of down-skilling caused by the pandemic, observing that firms cut back on hiring for high-skilled more than low-skilled jobs (Campello, Kankanhalli, and Muthukrishnan 2020). Nonetheless, another study using LinkedIn data found that workers from affected industries tended to apply for jobs in sectors such as ICT or healthcare, resulting in reallocation of labour and, potentially, changes in skills development (Bauer et al. 2021). ●A study which combined online data (Burning Glass) with representative data (unemployment claims and employment statistics data) in the US found that a decline in economic activity in the online labour market was reflected in representative sources of data. No difference between “essential” and “non-essential” sectors was found, in terms of a decline in online job postings (with the exception of essential retail) (Forsythe et al. 2020) As regards the supply side more broadly, compared to job vacancies, we do not find the same evidence that fluctuations in online CVs availability match the trends existing in other data sources. Job search intensity is an important predictor of labour market developments (Mukoyama, Patterson, and Şahin 2018), which would make such an indicator very useful; but we did not discover any published research that attempted to estimate the fluctuations of CVs being posted on job portals over time. Some job portals publish raw trends in their data. For example, according to Profesia.sk data, while job applicant data fell abruptly in response to the first lockdown in March 2020, applicant data fairly quickly recovered. This is most likely due to a structural shift in demand for labour (Profesia 2022). E. Techniques and approaches to address non-representativeness and other biases While many empirical studies are not concerned with the issues of representativeness or other related biases, we identified a set of studies that take a rigorous approach in their use of online labour market data. In the process of our mapping exercise, we found a variety of approaches to address biases of the data. In this section we discuss these, providing examples of concrete studies which have adopted a particular approach. As we came across far fewer studies using online applicant data, most of the discussion is based on studies using online job vacancies. The approaches identified in the literature can be broadly divided into three categories: (1) statistical techniques, such as weighting and data cleaning techniques; (2) the research design approach, which involves adapting research questions and the research focus to issues well covered by online labour market data; and (3) the mixed-methods approach, where online job market data are complemented by other research strategies, including qualitative methods. In some cases, these approaches are used in combination and are seen as mutually exclusive. i. Statistical techniques Statistical techniques at two stages of the analytical process are relevant: (1) the data cleaning and data preparation stage; and (2) the data analysis phase. In some instances, to avoid bias in online job vacancies (such as from duplications), data cleaning techniques employ job vacancy aggregators, which then provide 29 ILO Working Paper 68 Short reference Country and time coverage Type of online data (CV or vacancy) Source of data (specific website) Methods applied (single or multiple; statistical method) Focus of the analysis/ research question Biases identified Approaches to address biases Representativeness discussed (Y/N, and conclusion) Generalizability of findings discussed (Y/N, and conclusion) Skhvediani et al. 2021 Russia Vacancy 100 job adverts on Headhunters Descriptives Skill requirements for data scientists in Russia None None None None Debortoli et al. 2014 US, Canada, Australia, UK Vacancy Monster.com Singular value decompos-ition Comparing business intelligence and big data skills None None None None Muller and Safir 2019 Ukraine Vacancy 2500 job vacancies from headhunter Descriptives Job requirements in Ukraine None None None Mentioned that jobs published on the website mostly target high-skilled workers Nomura et al. 2017 India Vacancy data and job applicant data (job search behaviour) Babajobs – one of the leading job portals Econometric analysis Text/content analysis Econometric analysis/ probit model Predictive analysis with longitudinal data Randomized control trials Gender differences in wage offers Skills demand Job search behaviour Wage trends – real wage offers by occupation and location Reduce information asymmetries Informal labour market in India, high youth un-employment Self-selection and sample error Babajobs covers less skilled and informal markets better than other portals (phonebased access to job seekers) Yes, reference to Kurekova et al. 2015; external data that could be used to counter some of the biases are very limited in India Yes, reference to Kurekova et al. 2015 30 ILO Working Paper 68 References Abraham, Katharine G. 1983. “Structural/Frictional vs. Deficient Demand Unemployment: Some New Evidence.” The American Economic Review 73 (4): 708–24. Acemoglu, Daron, and David Autor. 2010. “Skills, Tasks and Technologies: Implications for Employment and Earnings.” Working Paper 16082. Working Paper Series. National Bureau of Economic Research. https:// doi.org/10.3386/w16082. Acemoglu, Daron, David Autor, Jonathon Hazell, and Pascual Restrepo. 2020. “AI and Jobs: Evidence from Online Vacancies.” Working Paper 28257. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/w28257. Álvarez, Andrés, and Marc Hofstetter. 2014. “Job Vacancies in Colombia: 1976–2012.” IZA Journal of Labor & Development 3 (1): 15. https://doi.org/10.1186/2193-9020-3-15. Anastasopoulos, L. Jason, George J. Borjas, Gavin G. Cook, and Michael Lachanski. 2021. “Job Vacancies and Immigration: Evidence from the Mariel Supply Shock.” Journal of Human Capital 15 (1): 1–33. https://doi. org/10.1086/713041. Antenucci, Dolan, Michael Cafarella, Margaret Levenstein, Christopher Ré, and Matthew D. Shapiro. 2014. “Using Social Media to Measure Labor Market Flows.” Working Paper 20010. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/w20010. Apaza, Honorio, Américo Vidal, and Josimar Chire. 2021. “Job Recommendation Based on Curriculum Vitae Using Text Mining.” In , 1051–59. https://doi.org/10.1007/978-3-030-73100-7_72. Askitas, Nikolaos, and Klaus F. Zimmermann. 2009. “Google Econometrics and Unemployment Forecasting.” Applied Economics Quarterly 55 (2): 107–21. ———. 2015. “The Internet as a Data Source for Advancement in Social Sciences.” International Journal of Manpower 36 (1): 2–12. https://doi.org/10.1108/IJM-02-2015-0029. Autor, David H. 2001. “Wiring the Labor Market.” Journal of Economic Perspectives 15 (1): 25–40. https:// doi.org/10.1257/jep.15.1.25. Azar, José, Emiliano Huet-Vaughn, Ioana Marinescu, Bledi Taska, and Till von Wachter. 2019. “Minimum Wage Employment Effects and Labor Market Concentration.” Working Paper 26101. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/w26101. Azar, José, Ioana Marinescu, Marshall Steinbaum, and Bledi Taska. 2020. “Concentration in US Labor Markets: Evidence from Online Vacancy Data.” Labour Economics 66 (October): 101886. https://doi.org/10.1016/j. labeco.2020.101886. Barron, John M., and John Bishop. 1985. “Extensive Search, Intensive Search, and Hiring Costs: New Evidence on Employer Hiring Activity.” Economic Inquiry 23 (3): 363–82. https://doi.org/10.1111/j.1465-7295.1985. tb01773.x. Bauer, Anja, Tobias Hartl, Christian Hutter, and Enzo Weber. 2021. “Search Processes on the Labor Market during the Covid-19 Pandemic.” CESifo Forum 22 (04): 15–19. 31 ILO Working Paper 68 Beblavý, Miroslav, Mehtap Akgüc, Brian Fabo, and Karolien Lenaerts. 2016. “Occupations ObservatoryMethodological Note.” CEPS Special Report, no. 144. Beblavý, Miroslav, Lucia Kureková, and Corina Haita. 2016. “The Surprisingly Exclusive Nature of Mediumand Low-Skilled Jobs: Evidence from a Slovak Job Portal.” Personnel Review 45 (2): 255–73. https://doi.org/10.1108/ PR-12-2014-0276. Beblavý, Miroslav, Ilaria Maselli, and Marcela Veselková, eds. 2014. Let’s Get to Work!: The Future of Labour in Europe. Brussels: Centre for European Policy Studies. ———, eds. 2015. Green, Pink and Silver?: The Future of Labour in Europe. Brussels: Centre for European Policy Studies. Bilal, Muhammad, Nadia Malik, Maham Khalid, and M. Ikram Ullah Lali. 2017. “Exploring Industrial Demand Trends in Pakistan Software Industry Using Online Job Portal Data.” University of Sindh Journal of Information and Communication Technology 1 (1): 17–24. Birba, Ousmane, and Abdoulaye Diagne. 2012. “Determinants of Adoption of Internet in Africa: Case of 17 Sub-Saharan Countries.” Structural Change and Economic Dynamics 23 (4): 463–72. https://doi.org/10.1016/j. strueco.2012.06.003. Blair, Peter Q., and David J. Deming. 2020. “Structural Increases in Skill Demand after the Great Recession.” Working Paper 26680. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/ w26680. Blazquez, Desamparados, and Josep Domenech. 2018. “Big Data Sources and Methods for Social and Economic Analyses.” Technological Forecasting and Social Change 130 (May): 99–113. https://doi.org/10.1016/j.techfore.2017.07.027. Bloom, David E., Richard B. Freeman, and Sanders D. Korenman. 1988. “The Labour-Market Consequences of Generational Crowding.” European Journal of Population / Revue Européenne de Démographie 3 (2): 131–76. Boselli, Roberto, Mirko Cesarini, Stefania Marrara, Fabio Mercorio, Mario Mezzanzanica, Gabriella Pasi, and Marco Viviani. 2018. “WoLMIS: A Labor Market Intelligence System for Classifying Web Job Vacancies.” Journal of Intelligent Information Systems 51 (3): 477–502. https://doi.org/10.1007/s10844-017-0488-x. Boselli, Roberto, Mirko Cesarini, Fabio Mercorio, and Mario Mezzanzanica. 2018. “Classifying Online Job Advertisements through Machine Learning.” Future Generation Computer Systems 86 (September): 319– 28. https://doi.org/10.1016/j.future.2018.03.035. Brancatelli, Calogero, Alicia Marguerie, and Stefanie Brodmann. 2020. “Job Creation and Demand for Skills in Kosovo: What Can We Learn from Job Portal Data?” Working Paper. Washington, DC: World Bank. https:// doi.org/10.1596/1813-9450-9266. Brandas, Claudiu, Ciprian Panzaru, and Florin Gheorghe Filip. 2016. “Data Driven Decision Support Systems: An Application Case in Labour Market Analysis.” Romanian Journal of Information Science and Technology 19 (1–2): 65–77. Buchs, Helen, and Laura Alexandra Helbling. 2016. “Job Opportunities and School-to-Work Transitions in Occupational Labour Markets. Are Occupational Change and Unskilled Employment after Vocational Education Interrelated?” Empirical Research in Vocational Education and Training 8 (1): 17. https://doi.org/10.1186/ s40461-016-0044-x. 32 ILO Working Paper 68 Burke, Mary A., Alicia Sasser, Shahriar Sadighi, Rachel B. Sederberg, and Bledi Taska. 2020. “No Longer Qualified? Changes in the Supply and Demand for Skills within Occupations.” Working Paper 20–3. Working Papers. https://doi.org/10.29412/res.wp.2020.03. Cammeraat, Emile, and Mariagrazia Squicciarini. 2021. “Burning Glass Technologies’ Data Use in PolicyRelevant Analysis: An Occupation-Level Assessment.” Paris: OECD. https://doi.org/10.1787/cd75c3e7-en. Campello, Murillo, Gaurav Kankanhalli, and Pradeep Muthukrishnan. 2020. “Corporate Hiring under COVID-19: Labor Market Concentration, Downskilling, and Income Inequality.” Working Paper 27208. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/w27208. Campos-Vazquez, Raymundo M., Gerardo Esquivel, and Raquel Y. Badillo. 2021. “How Has Labor Demand Been Affected by the COVID-19 Pandemic? Evidence from Job Ads in Mexico.” Latin American Economic Review 30 (May): 1–42. Capiluppi, Andrea, and Andres Baravalle. 2010. “Matching Demand and Offer in On-Line Provision: A Longitudinal Study of Monster.Com.” In 2010 12th IEEE International Symposium on Web Systems Evolution (WSE), 13–21. https://doi.org/10.1109/WSE.2010.5623576. CEDEFOP. 2014. “Briefing Note - Skill Mismatch: More than Meets the Eye.” March 19, 2014. https://www. cedefop.europa.eu/en/publications/9087. ———. 2019. “Online Job Vacancies and Skills Analysis: A Cedefop Pan-European Approach | VOCEDplus, the International Tertiary Education and Research Database.” https://www.voced.edu.au/content/ngv:82496. Chowdhury, Afra Rahman, Ana Carolina Areias, Saori Imaizumi, Shinsaku Nomura, and Futoshi Yamauchi. 2018. “Reflections of Employers’Gender Preferences in Job Ads in India: An Analysis of Online Job Portal Data.” SSRN Scholarly Paper ID 3150092. Rochester, NY: Social Science Research Network. https://papers. ssrn.com/abstract=3150092. Colombo, Emilio, Fabio Mercorio, and Mario Mezzanzanica. 2018. “Applying Machine Learning Tools on Web Vacancies for Labour Market and Skill Analysis.” Comyn, Paul, and Olga Strietska-Ilina. 2019. Skills and Jobs Mismatches in Lowand Middle-Income Countries. Geneva, Switzerland: ILO. https://www.semanticscholar.org/paper/Skills-and-jobs-mismatches-in-low-andmiddle-income-Comyn/1a2c34a7e6c337b46fc0b004291c2c8d27f5c85f. Das, Jishnu, and Quy-Toan Do. 2014. “US and Them: The Geography of Academic Research.” VoxEU.Org (blog). February 11, 2014. https://voxeu.org/article/geographical-bias-top-journal-publication. Das, Marcel, Peter Ester, and Lars Kaczmirek. 2018. Social and Behavioral Research and the Internet: Advances in Applied Methods and Research Strategies. Routledge. Davis, Steven J., R. Jason Faberman, and John C. Haltiwanger. 2013. “The Establishment-Level Behavior of Vacancies and Hiring *.” The Quarterly Journal of Economics 128 (2): 581–622. https://doi.org/10.1093/qje/ qjt002. Debortoli, Stefan, Oliver Müller, and Jan vom Brocke. 2014. “Comparing Business Intelligence and Big Data Skills.” Business & Information Systems Engineering 6 (5): 289–300. https://doi.org/10.1007/s12599-0140344-2. Deming, David, and Lisa B. Kahn. 2018. “Skill Requirements across Firms and Labor Markets: Evidence from Job Postings for Professionals.” Journal of Labor Economics 36 (S1): S337–69. https://doi.org/10.1086/694106. 33 ILO Working Paper 68 Dijk, Jan van. 2006. “Digital Divide Research, Achievements and Shortcomings.” Poetics, The digital divide in the twenty-first century, 34 (4): 221–35. https://doi.org/10.1016/j.poetic.2006.05.004. ———. 2020. The Digital Divide. 1st edition. Cambridge, UK ; Medford, MA: Polity. Dörfler, Laura, and Herman van de Werfhorst. 2009. “Employers’ Demand for Qualifications and Skills.” European Societies 11 (5): 697–721. https://doi.org/10.1080/14616690802474374. Drahokoupil, Jan, and Brian Fabo. 2016. “The Platform Economy and the Disruption of the Employment Relationship.” ETUI Research Paper-Policy Brief 5. ———. 2022. “The Limits of Foreign-Led Growth: Demand for Skills by Foreign and Domestic Firms.” Review of International Political Economy 29 (1): 152–74. Dunlop, John T. 1966. “Job Vacancy Measures and Economic Analysis.” In The Measurement and Interpretation of Job Vacancies, 27–47. NBER. https://www.nber.org/books-and-chapters/measurement-and-interpretation-job-vacancies/job-vacancy-measures-and-economic-analysis. Ecleo, Jerina Jean, and Adrian Galido. 2017. “Surveying LinkedIn Profiles of Data Scientists: The Case of the Philippines.” Procedia Computer Science, 4th Information Systems International Conference 2017, ISICO 2017, 6-8 November 2017, Bali, Indonesia, 124 (January): 53–60. https://doi.org/10.1016/j.procs.2017.12.129. Einav, Liran, and Jonathan Levin. 2014. “Economics in the Age of Big Data.” Science 346 (6210): 1243089. https://doi.org/10.1126/science.1243089. Ernesto, Caroleo Floro, and Pastore Francesco. 2016. “Overeducation: A Disease of the School-to-Work Transition System.” In Youth and the Crisis. Routledge. ESCO. 2015. “European Skills, Competences, Qualifications and Occupations.” (https://ec.europa.eu/esco/ portal/home#modal-one. Fabo, Brian, Miroslav Beblavý, and Karolien Lenaerts. 2017. “The Importance of Foreign Language Skills in the Labour Markets of Central and Eastern Europe: Assessment Based on Data from Online Job Portals.” Empirica 44 (3): 487–508. https://doi.org/10.1007/s10663-017-9374-6. Fabo, Brian, and Martin Kahanec. 2018. “Can a Voluntary Web Survey Be Useful beyond Explorative Research?” International Journal of Social Research Methodology. ———. 2020. “The Role of Computer Skills on the Occupation Level.” European Journal of Business Science and Technology 6 (2): 87–99. https://doi.org/10.11118/ejobsat.2020.006. Fang, Hanming, Chunmian Ge, Hanwei Huang, and Hongbin Li. 2020. “Pandemics, Global Supply Chains, and Local Labor Demand: Evidence from 100 Million Posted Jobs in China.” Working Paper 28072. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/w28072. Ferber, Robert, and Neil Ford. 1966. “The Time Dimension in the Collection of Job Vacancy Data.” In The Measurement and Interpretation of Job Vacancies, 447–61. NBER. https://www.nber.org/books-and-chapters/measurement-and-interpretation-job-vacancies/time-dimension-collection-job-vacancy-data. Fondeur, Y., and Frédéric Karamé. 2013. “Can Google Data Help Predict French Youth Unemployment?” Economic Modelling 30 (C): 117–25. 34 ILO Working Paper 68 Forsythe, Eliza, Lisa B. Kahn, Fabian Lange, and David Wiczer. 2020. “Labor Demand in the Time of COVID-19: Evidence from Vacancy Postings and UI Claims.” Journal of Public Economics 189 (September): 104238. https://doi.org/10.1016/j.jpubeco.2020.104238. Gandomi, Amir, and Murtaza Haider. 2015. “Beyond the Hype: Big Data Concepts, Methods, and Analytics.” International Journal of Information Management 35 (2): 137–44. https://doi.org/10.1016/j.ijinfomgt.2014.10.007. Gortmaker, Jeff, Jessica Jeffers, and Michael Lee. 2021. “Labor Reactions to Credit Deterioration: Evidence from LinkedIn Activity.” SSRN Scholarly Paper ID 3456285. Rochester, NY: Social Science Research Network. https://doi.org/10.2139/ssrn.3456285. Haddad, Rabih, and Eunika Mercier-Laurent. 2021. “Curriculum Vitae (CVs) Evaluation Using Machine Learning Approach.” In Artificial Intelligence for Knowledge Management, edited by Eunika Mercier-Laurent, M. Özgür Kayalica, and Mieczyslaw Lech Owoc, 48–65. IFIP Advances in Information and Communication Technology. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-80847-1_4. Hensvik, Lena, Thomas Le Barbanchon, and Roland Rathelot. 2021. “Job Search during the COVID-19 Crisis.” Journal of Public Economics 194 (February): 104349. https://doi.org/10.1016/j.jpubeco.2020.104349. Hershbein, Brad. 2016. “Is College the New High School? Evidence from Vacancy Postings.” https://research. upjohn.org/projects/169. Hershbein, Brad, and Lisa B. Kahn. 2016. “Do Recessions Accelerate Routine-Biased Technological Change? Evidence from Vacancy Postings.” Working Paper 22762. National Bureau of Economic Research. https:// doi.org/10.3386/w22762. Holt, Charles, and Martin David. 1966. “The Concept of Job Vacancies in a Dynamic Theory of the Labor Market.” In The Measurement and Interpretation of Job Vacancies, 73–110. NBER. http://www.nber.org/ chapters/c1599.pdf. Horton, John J. 2011. “The Condition of the Turking Class: Are Online Employers Fair and Honest?” Economics Letters 111 (1): 10–12. https://doi.org/10.1016/j.econlet.2010.12.007. Huang, Hayan, Lynette Kvasny, K.D. Joshi, Eileen Trauth, and Jan Mahar. 2009. “Synthesizing IT Job Skills Identified in Academic Studies, Practitioner Publications and Job Ads.” In Proceedings of the Special Interest Group on Management Information System’s 47th Annual Conference on Computer Personnel Research. Ireland: ACM. ILO. 2016. “Disguised Employment / Dependent Self-Employment.” Document. November 11, 2016. http:// www.ilo.org/global/topics/non-standard-employment/WCMS_534833/lang--en/index.htm. ———. 2021. “Transition from the Informal to the Formal Economy - Theory of Change.” Briefing note. January 29, 2021. http://www.ilo.org/global/topics/employment-promotion/informal-economy/publications/ WCMS_768807/lang--en/index.htm. Jackson, Michelle. 2007. “How Far Merit Selection? Social Stratification and the Labour Market1.” The British Journal of Sociology 58 (3): 367–90. https://doi.org/10.1111/j.1468-4446.2007.00156.x. Japec, Lilli, and Lars Lyberg. 2020. “Big Data Initiatives in Official Statistics.” In Big Data Meets Survey Science, 273–302. John Wiley & Sons, Ltd. https://doi.org/10.1002/9781118976357.ch9. 35 ILO Working Paper 68 Jones, Lyndon. 1984. “Lies, Damned Lies, and CVs.” Education + Training 26 (4): 124–26. https://doi.org/10.1108/ eb002125. Kahn, Lisa B., Fabian Lange, and David Wiczer. 2020. “Labor Supply in the Time of COVID19.” 06–2020. Cahiers de Recherche. Cahiers de Recherche. Centre interuniversitaire de recherche en économie quantitative, CIREQ. https://ideas.repec.org/p/mtl/montec/06-2020.html. Khaouja, Imane, Ismail Kassou, and Mounir Ghogho. 2021. “A Survey on Skill Identification From Online Job Ads.” IEEE Access 9: 118134–53. https://doi.org/10.1109/ACCESS.2021.3106120. Kobayashi, Vladimer, Stefan T. Mol, Gábor Kismihók, and Maria Hesterberg. 2016. “Automatic Extraction of Nursing Tasks from Online Job Vacancies.” In Professional Education and Training through Knowledge, Technology and Innovation, 51. Siegen, Germany: Universitätsverlag Siegen. Kotu, Vijay, and Bala Deshpande. 2019. “Chapter 9 - Text Mining.” In Data Science (Second Edition), edited by Vijay Kotu and Bala Deshpande, 281–305. Morgan Kaufmann. https://doi.org/10.1016/B978-0-12814761-0.00009-5. Kotzab, Herbert, Christoph Teller, Michael Bourlakis, and Sebastian Wünsche. 2018. “Key Competences of Logistics and SCM Professionals – the Lifelong Learning Perspective.” Supply Chain Management: An International Journal 23 (1): 50–64. https://doi.org/10.1108/SCM-02-2017-0079. Kuhn, Peter. 2014. “The Internet as a Labor Market Matchmaker.” IZA World of Labor, May. https://doi. org/10.15185/izawol.18. Kuhn, Peter, and Hani Mansour. 2014. “Is Internet Job Search Still Ineffective?” The Economic Journal 124 (581): 1213–33. https://doi.org/10.1111/ecoj.12119. Kuhn, Peter, and Kailing Shen. 2013. “Gender Discrimination in Job Ads: Evidence from China.” The Quarterly Journal of Economics 128 (1): 287–336. https://doi.org/10.1093/qje/qjs046. Kureková, Lucia, Miroslav Beblavý, Corina Haita, and Anna Elisabeth Thum Thysen. 2016. “Employers’ Skill Preferences across Europe: Between Cognitive and Non-Cognitive Skills.” Journal of Education and Work 29 (6): 662–87. https://doi.org/10.1080/13639080.2015.1024641. Kureková, Lucia, Miroslav Beblavý, and Anna Elisabeth Thum Thysen. 2015. “Using Online Vacancies and Web Surveys to Analyse the Labour Market: A Methodological Inquiry.” IZA Journal of Labor Economics 4 (1): 18. https://doi.org/10.1186/s40172-015-0034-4. Kureková, Lucia, and Zuzana Žilinčíková. 2016. “Are Student Jobs Flexible Jobs? Using Online Data to Study Employers’ Preferences in Slovakia.” IZA Journal of European Labor Studies 5 (1): 20. https://doi.org/10.1186/ s40174-016-0070-5. ———. 2018. “What Is the Value of Foreign Work Experience for Young Return Migrants?” International Journal of Manpower 39 (1): 71–92. https://doi.org/10.1108/IJM-04-2016-0091. Lazer, David, Ryan Kennedy, Gary King, and Alessandro Vespignani. 2014. “The Parable of Google Flu: Traps in Big Data Analysis.” Science 343 (6176): 1203–5. https://doi.org/10.1126/science.1248506. Leitner, Sandra M., and Oliver Reiter. 2020. “Employers’ Skills Requirements in the Austrian Labour Market: On the Relative Importance of ICT, Cognitive and Non-Cognitive Skills over the Past 15 Years.” Working Paper 190. wiiw Working Paper. https://www.econstor.eu/handle/10419/240633. 36 ILO Working Paper 68 Lenaerts, Karolien, Miroslav Beblavý, and Brian Fabo. 2016. “Prospects for Utilisation of Non-Vacancy Internet Data in Labour Market Analysis—an Overview.” IZA Journal of Labor Economics 5 (1): 1. https://doi.org/10.1186/ s40172-016-0042-z. Lewis, Phil, and Jennifer Norton. 2016. “Identification of ‘Hot Technologies’ within the O*NET® System.” https://www.onetcenter.org/reports/Hot_Technologies.html. Loo, Jasper van, and Konstantinos Pouliakas. 2020. “Cedefop and the Analysis of European Online Job Vacancies.” In The Feasibility of Using Big Data in Anticipating and Matching Skills Needs. Geneva, Switzerland: ILO. Lovaglio, Pietro Giorgio, Mario Mezzanzanica, and Emilio Colombo. 2020. “Comparing Time Series Characteristics of Official and Web Job Vacancy Data.” Quality & Quantity 54 (1): 85–98. https://doi.org/10.1007/ s11135-019-00940-3. Mamertino, Mariano, and Tara M. Sinclair. 2019. “Migration and Online Job Search: A Gravity Model Approach.” Economics Letters 181 (August): 51–53. https://doi.org/10.1016/j.econlet.2019.05.005. Marconi, Gabriele. 2022. “Content Removal Bias in Web Scraped Data: A Solution Applied to Real Estate Ads.” SSRN Scholarly Paper ID 4031466. Rochester, NY: Social Science Research Network. https://papers. ssrn.com/abstract=4031466. Marinescu, Ioana. 2017. “The General Equilibrium Impacts of Unemployment Insurance: Evidence from a Large Online Job Board.” Journal of Public Economics 150 (June): 14–29. https://doi.org/10.1016/j.jpubeco.2017.02.012. Marinescu, Ioana, and Roland Rathelot. 2018. “Mismatch Unemployment and the Geography of Job Search.” American Economic Journal: Macroeconomics 10 (3): 42–70. https://doi.org/10.1257/mac.20160312. Marinescu, Ioana, and Ronald Wolthoff. 2020. “Opening the Black Box of the Matching Function: The Power of Words.” Journal of Labor Economics 38 (2): 535–68. https://doi.org/10.1086/705903. Marrara, Stefania, Gabriella Pasi, Marco Viviani, Mirko Cesarini, Fabio Mercorio, Mario Mezzanzanica, and Marco Pappagallo. 2017. “A Language Modelling Approach for Discovering Novel Labour Market Occupations from the Web.” In Proceedings of the International Conference on Web Intelligence, 1026–34. WI ’17. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3106426.3109035. Masso, Jaan, Raul Eamets, Pille Mõtsmees, and Kaia Philips. 2012. “The Impact of Interfirm Labor Mobility on Innovation: Evidence from Job Search Portal Data.” In Innovation Systems in Small Catching-Up Economies: New Perspectives on Practice and Policy, edited by Elias G. Carayannis, Urmas Varblane, and Tõnu Roolaht, 297–321. Innovation, Technology, and Knowledge Management. New York, NY: Springer. https://doi. org/10.1007/978-1-4614-1548-0_16. Masso, Jaan, Lucia Kureková, Maryna Tverdostup, and Zuzana Žilinčíková. 2016. “Return Migration to CEE after the Crisis: Estonia and Slovakia.” https://style-handbook.eu/contents-list/migration-and-mobility/return-migration-to-cee-after-the-crisis-estonia-and-slovakia/. Matsuda, Norihiko, Tutan Ahmed, and Shinsaku Nomura. 2019. “Labor Market Analysis Using Big Data: The Case of a Pakistani Online Job Portal.” SSRN Scholarly Paper ID 3491253. Rochester, NY: Social Science Research Network. https://papers.ssrn.com/abstract=3491253. 37 ILO Working Paper 68 Maurer-Fazio, Margaret, and Lei Lei. 2015. “‘As Rare as a Panda’: How Facial Attractiveness, Gender, and Occupation Affect Interview Callbacks at Chinese Firms.” International Journal of Manpower 36 (1): 68–85. https://doi.org/10.1108/IJM-12-2014-0258. McLaren, Nick, and Rachana Shanbhogue. 2011. “Using Internet Search Data as Economic Indicators.” Bank of England Quarterly Bulletin 51 (2): 134–40. Mezzanzanica, Mario, and Fabio Mercorio. 2019. “Big Data for Labour Market Intelligence: An Introductory Guide.” European Training Foundation. https://www.etf.europa.eu/en/publications-and-resources/publications/big-data-labour-market-intelligence-introductory-guide. Modestino, Alicia Sasser, Daniel Shoag, and Joshua Ballance. 2020. “Upskilling: Do Employers Demand Greater Skill When Workers Are Plentiful?” The Review of Economics and Statistics 102 (4): 793–805. https:// doi.org/10.1162/rest_a_00835. Mukoyama, Toshihiko, Christina Patterson, and Ayşegül Şahin. 2018. “Job Search Behavior over the Business Cycle.” American Economic Journal: Macroeconomics 10 (1): 190–215. https://doi.org/10.1257/mac.20160202. Muller, Noël, and Abla Safir. 2019. “What Employers Actually Want: Skills in Demand in Online Job Vacancies in Ukraine.” Working Paper. Washington, DC: World Bank. https://doi.org/10.1596/31884. Nomura, Shinsaku, Saori Imaizumi, Ana Carolina Areias, and Futoshi Yamauchi. 2017. “Toward Labor Market Policy 2.0: The Potential for Using Online Job-Portal Big Data to Inform Labor Market Policies in India.” Working Paper. Washington, DC: World Bank. https://doi.org/10.1596/1813-9450-7966. OECD. 2021. “OECD Skills Outlook 2021: Learning for Life.” https://www.oecd.org/education/oecd-skillsoutlook-e11c1c2d-en.htm. Ours, Jan van. 1989. “Durations of Dutch Job Vacancies.” De Economist 137 (3): 309–27. https://doi.org/10.1007/ BF02115697. Ours, Jan van, and Geert Ridder. 1992. “Vacancies and the Recruitment of New Employees.” Journal of Labor Economics 10 (2): 138–55. Pedraza, Pablo de, Stefano Visintin, Kea Tijdens, and Gábor Kismihók. 2019. “Survey vs Scraped Data: Comparing Time Series Properties of Web and Survey Vacancy Data.” IZA Journal of Labor Economics 8 (1). https://doi.org/10.2478/izajole-2019-0004. Pejic-Bach, Mirjana, Tine Bertoncel, Maja Meško, and Živko Krstić. 2020. “Text Mining of Industry 4.0 Job Advertisements.” International Journal of Information Management 50 (February): 416–31. https://doi. org/10.1016/j.ijinfomgt.2019.07.014. Piróg, Danuta. 2016. “Job Search Strategies of Recent University Graduates in Poland: Plans and Effectiveness.” Higher Education 71 (4): 557–73. Pitukhin, Eugene, Marina Astafyeva, and Irina Astafyeva. 2020. “Methodology for Job Advertisements Analysis in the Labor Market in Metropolitan Cities: The Case Study of the Capital of Russia.” In Intelligent Algorithms in Software Engineering, edited by Radek Silhavy, 413–29. Advances in Intelligent Systems and Computing. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-51965-0_37. Poschke, Markus. 2019. “Wage Employment, Unemployment and Self-Employment across Countries.” https:// www.iza.org/publications/dp/12367/wage-employment-unemployment-and-self-employment-across-countries. 38 ILO Working Paper 68 Profesia. 2022. “Covid19 Recruitment Report.” 2022. https://public.tableau.com/app/profile/profesia.analytics4840/viz/ProfesiaReport_V2/Covid?publish=yes. Rios, Joseph A., Guangming Ling, Robert Pugh, Dovid Becker, and Adam Bacall. 2020. “Identifying Critical 21st-Century Skills for Workplace Success: A Content Analysis of Job Advertisements.” Educational Researcher 49 (2): 80–89. https://doi.org/10.3102/0013189X19890600. Scheerder, Anique, Alexander van Deursen, and Jan van Dijk. 2017. “Determinants of Internet Skills, Uses and Outcomes. A Systematic Review of the Secondand Third-Level Digital Divide.” Telematics and Informatics 34 (8): 1607–24. https://doi.org/10.1016/j.tele.2017.07.007. Simionescu, Mihaela, and Klaus F. Zimmermann. 2017. “Big Data and Unemployment Analysis.” 81. GLO Discussion Paper Series. GLO Discussion Paper Series. Global Labor Organization (GLO). https://ideas.repec.org/p/zbw/glodps/81.html. Skhvediani, Angi, Sergey Sosnovskikh, Irina Rudskaia, and Tatiana Kudryavtseva. 2021. “Identification and Comparative Analysis of the Skills Structure of the Data Analyst Profession in Russia.” Journal of Education for Business 0 (0): 1–10. https://doi.org/10.1080/08832323.2021.1937018. Smyk, Magdalena, Joanna Tyrowicz, and Lucas van der Velde. 2018. “A Cautionary Note on the Reliability of the Online Survey Data: The Case of Wage Indicator.” Sociological Methods & Research, July, 0049124118782538. https://doi.org/10.1177/0049124118782538. Sodhi, M S, and B-G Son. 2010. “Content Analysis of OR Job Advertisements to Infer Required Skills.” Journal of the Operational Research Society 61 (9): 1315–27. https://doi.org/10.1057/jors.2009.80. Sostero, Matteo, and Enrique Fernandez-Macias. 2021. “The Professional Lens: What Online Job Advertisements Can Say About Occupational Task Profiles.” JRC Working Papers on Labour, Education and Technology 2021– 13. Joint Research Centre (Seville site). https://econpapers.repec.org/paper/iptlaedte/202113.htm. Štefánik, Miroslav. 2012. “Internet Job Search Data as a Possible Source of Information on Skills Demand (with Results for Slovak University Graduates.” In , 258–72. Thessaloniki: CEDEFOP. Štefánik, Miroslav, Štefan Lyócsa, and Matúš Bilka. 2022. “Using Online Job Vacancies to Predict Key Labour Market Indicators.” Social Science Computer Review. DOI: https://doi.org/10.1177/08944393221085705 Su, Zhi. 2014. “Chinese Online Unemployment-Related Searches and Macroeconomic Indicators.” Frontiers of Economics in China 9 (4): 573–605. https://doi.org/10.3868/s060-003-014-0027-3. Tambe, Prasanna. 2014. “Big Data Investment, Skills, and Firm Value.” Management Science 60 (6): 1452–69. Tambe, Prasanna, Lorin Hitt, Daniel Rock, and Erik Brynjolfsson. 2020. “Digital Capital and Superstar Firms.” Working Paper 28285. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/ w28285. Tijdens, Kea. 2010. “Measuring Occupations in Web-Surveys: The WISCO Database of Occupations.” https:// pure.uva.nl/ws/files/1491455/118626_1000_WP86_Tijdens_Measuring_occupations_WISCO_database.pdf. Tijdens, Kea, and Stephanie Steinmetz. 2016. “Is the Web a Promising Tool for Data Collection in Developing Countries? An Analysis of the Sample Bias of 10 Web and Face-to-Face Surveys from Africa, Asia, and South America.” International Journal of Social Research Methodology 19 (4): 461–79. https://doi.org/10.1080/13 645579.2015.1035875.