Desenvolvimento de um modelo e metodologia para implementação de um sistema de medição de desempenho

Universidade do Minho Escola de Engenharia Flávio Avelino Vilaça da Cunha Desenvolvimento de um modelo e metodologia para a implementação de um sistema de medição de desempenho Abril de 2025 UMinho | 2025Flávio Avelino Vilaça da Cunha Desenvolvimento de um modelo e metodologia para a implementação de um sistema de medição de desempenho Universidade do Minho Escola de Engenharia Flávio Avelino Vilaça da Cunha Desenvolvimento de um modelo e metodologia para implementação de um sistema de medição de desempenho Tese do Programa Doutoral em Engenharia Industrial e Sistemas Trabalho efetuado sob a orientação do Professor Doutor Rui Manuel Alves da Silva e Sousa Professor José Dinis Araújo Carvalho Abril de 2025 ii DIREITOS DE AUTOR E CONDIÇÕES DE UTILIZAÇÃO DO TRABALHO POR TERCEIROS Este é um trabalho académico que pode ser utilizado por terceiros desde que respeitadas as regras e boas práticas internacionalmente aceites, no que concerne aos direitos de autor e direitos conexos. Assim, o presente trabalho pode ser utilizado nos termos previstos na licença abaixo indicada. Caso o utilizador necessite de permissão para poder fazer um uso do trabalho em condições não previstas no licenciamento indicado, deverá contactar o autor, através do RepositóriUM da Universidade do Minho. Atribuição-NãoComercial CC BY-NC https://creativecommons.org/licenses/by-nc/4.0/ iii AGRADECIMENTOS A conclusão desta tese não seria possível sem a contribuição, direta ou indireta, de algumas pessoas. A essas pessoas quero deixar o meu sincero agradecimento. Aos meus orientadores, professor Rui Sousa e professor Dinis Carvalho, pelos seus contributos, disponibilidade e orientação ao longo destes 3 anos. À minha família e amigos, especialmente aos meus pais e irmã por todo o apoio. Por fim, às empresas onde trabalhei nestes 3 anos, Stokvis e Borgwarner, por toda a disponibilidade e flexibilidade que me proporcionaram. iv DECLARAÇÃO DE INTEGRIDADE Declaro ter atuado com integridade na elaboração do presente trabalho académico e confirmo que não recorri à prática de plágio nem a qualquer forma de utilização indevida ou falsificação de informações ou resultados em nenhuma das etapas conducente à sua elaboração. Mais declaro que conheço e que respeitei o Código de Conduta Ética da Universidade do Minho. v Desenvolvimento de um modelo e metodologia para implementação de um sistema de medição de desempenho RESUMO A medição e monitorização de desempenho são fatores fundamentais para a melhoria contínua e gestão nas organizações. A recolha e análise de dados, incluindo os dados de desempenho, sobre o estado atual permitem descrever e representar o ponto de partida para a melhoria ajudando assim a identificar onde se deve concentrar o foco inicial. A medição de desempenho tem sido uma prioridade para muitas organizações, com o objetivo de identificar os segmentos dos sistemas industriais que necessitam de melhorias. Os dados de desempenho são cruciais para orientar e gerir a melhoria do desempenho. Contudo, a utilização e manutenção eficazes de um Sistema de Medição de Desempenho (PMS) não são tarefas simples e podem representar desafios consideráveis para as organizações. Embora vários modelos de PMS sejam conhecidos, a sua implementação, utilização e manutenção continuam a apresentar falhas. Este projeto de investigação tem como objetivo responder às seguintes perguntas de investigação: “ Quais os principais fatores que dificultam a implementação, uso e manutenção de um sistema de medição de desempenho numa organização? ” e “ Como pode ser alcançada a melhoria da implementação, uso e manutenção de um sistema de medição de desempenho? ”. Para isso, este trabalho foi desenvolvido em 5 fases principais, resultando cada uma num artigo publicado ou em processo de publicação. Na primeira fase foram identificadas as principais barreiras à eficácia dos PMS, através de uma revisão de literatura. Na segunda foram estudadas as perceções da existência dessas barreiras numa organização, através da realização de entrevistas e questionários. Na terceira fase foram identificados os modelos PMS existentes na literatura e foram classificados de acordo com a sua capacidade para mitigar ou eliminar as barreiras à eficácia dos PMS. Na quarta fase foram identificadas as principais características para a eficácia dos PMS, através de uma revisão de literatura. Na quinta fase foi desenvolvido um modelo de PMS eficaz, bem como a respetiva metodologia de implementação, através do conhecimento obtido nas 4 fases anteriores. Deste projeto resultou uma proposta de modelo e metodologia para o desenvolvimento, implementação e uso de um PMS eficaz. PALAVRAS-CHAVE Indicadores Chave de Desempenho; Medição de desempenho; Sistema de Medição de Desempenho. vi Development of a framework and methodology for the deployment of a performance measurement system ABSTRACT Performance measurement and monitoring are fundamental factors for continuous improvement and management in organizations. The collection and analysis of data, including performance data, on the current state allows describing and representing the starting point for improvement and helps identify where to initially focus. Performance measurement has been a priority for many organizations, aiming to identify segments within industrial systems that require improvement. Performance data is crucial for guiding and managing performance improvement. However, the effective use and maintenance of a Performance Measurement System (PMS) are not simple tasks and can pose significant challenges for organizations. Although various PMS models are known, their implementation, use, and maintenance continue to exhibit flaws. This research project aims to answer the following research questions: " What are the main factors that hinder the implementation, use, and maintenance of a performance measurement system in an organization? " and " How can the improvement of the implementation, use, and maintenance of a performance measurement system be achieved? " To answer these research questions, this work was developed in 5 main phases, each resulting in a published article or in the process of publication. In the first phase, the main barriers to the effectiveness of a PMS were identified through a literature review. In the second phase, the perceptions of the existence of these barriers in an organization were studied through interviews and questionnaires. In the third phase, existing PMS models in the literature were identified and classified according to their ability to mitigate or eliminate barriers to PMS effectiveness. In the fourth phase, the main characteristics for PMS effectiveness were identified through a literature review. In the fifth phase, a model of an effective PMS was developed, as well as the corresponding methodology for implementation, based on the knowledge obtained from the previous four phases. This project resulted in a proposal of a model and methodology for the development, implementation, and use of an effective PMS. KEYWORDS Key Performance Indicators; Performance Measurement; Performance Measurement System. vii ÍNDICE Agradecimentos .................................................................................................................................. iii Resumo............................................................................................................................................... v Abstract.............................................................................................................................................. vi Índice ................................................................................................................................................ vii Índice de Figuras ................................................................................................................................. x Índice de Tabelas ............................................................................................................................... xi Lista de Abreviaturas, Siglas e Acrónimos .......................................................................................... xii 1. Introdução .................................................................................................................................. 1 1.1 Enquadramento ................................................................................................................. 1 1.2 Objetivos ........................................................................................................................... 5 1.3 Metodologia ....................................................................................................................... 5 1.4 Estrutura da tese ............................................................................................................. 10 2. Sistemas de Medição de Desempenho em Ambientes de Melhoria Contínua: Barreiras à Sua Eficácia 12 2.1 Abstract and Keywords..................................................................................................... 12 2.2 Introduction ..................................................................................................................... 12 2.3 Methods .......................................................................................................................... 14 2.4 Results ............................................................................................................................ 16 2.5 Discussion ....................................................................................................................... 18 2.6 Conclusion....................................................................................................................... 23 2.7 References ...................................................................................................................... 25 3. Análise das Barreiras à Eficácia de um Sistema de Medição de Desempenho numa Empresa: Perceções Através dos Níveis Hierárquicos ....................................................................................... 29 3.1 Abstract and Keywords..................................................................................................... 29 3.2 Introduction ..................................................................................................................... 30 3.3 Literature review .............................................................................................................. 31 3.4 Methods .......................................................................................................................... 32 3.5 Results ............................................................................................................................ 35 3.6 Discussion ....................................................................................................................... 41 2 acordo com regras (Stevens, 1946). A definição de medição está ligada a 2 conceitos fundamentais: empirismo e objetividade. O empirismo surge quando uma representação é o resultado de uma observação e não, por exemplo, de uma experiência, ou seja, ocorre quando o tipo de relação é observável. A objetividade existe quando há independência dos sujeitos e resultado fornece apenas informações sobre a propriedade medida, ou seja, a mesma medição efetuado por sujeitos diferentes obterá o mesmo resultado (Franceschini et al., 2007). Medição permite, avaliar o progresso, tomar medidas corretivas baseadas nessa avaliação e avaliar o impacto dessas mesmas ações (Basili, 1994). De acordo com Basili (1994) para ser eficaz uma medição deve ser: x Focada em objetivos específicos; x Aplicada a todos os produtos, processos e recursos; x Interpretada com base na caracterização e compreensão do contexto organizacional, ambiente e objetivos. De modo a que a redução de desperdícios possa ser feita de forma eficiente é necessário identificar onde, e de que forma, os desperdícios estão presentes numa organização, e, quantificá-los. Um Sistema de Medição de Desempenho / Performance Measurement System (PMS) de uma organização é o sistema que, através da medição, permite identificar onde se deve intervir para eliminar desperdícios e dessa forma melhorar o desempenho dessa organização. A primeira condição para melhorar, e alcançar a excelência é desenvolver e implementar o PMS (Kanji, 2002). PMS pode ser definido como o conjunto de métricas utilizadas para quantificar a eficácia e eficiência de ações (Neely et al., 1995). Bititci (2015) define a criação e utilização de um PMS como o processo de definir objetivos, desenvolver um conjunto de métricas de desempenho, recolher, analisar, reportar, interpretar, rever e agir sobre dados de desempenho. De acordo com Franco-Santos et al. (2007), os PMS desempenham os seguintes papéis numa organização: 1. Medir o desempenho: monitorizar o progresso e medir/avaliar o desempenho; 2. Auxiliar a gestão da estratégia: planear, formular, implementar e executar a estratégia, focar a atenção e fornecer alinhamento; 3. Comunicação: comunicação interna e externa, benchmarking e conformidade com normas; 4. Influenciar comportamentos: recompensar e compensar comportamentos, gerir e controlar relacionamentos; 5. Aprender e melhorar: feedback , aprendizagem e melhoria do desempenho. 3 De acordo com Fitzgerald & Moon (1996) não existe um PMS perfeito, isto é, aplicável em todos os contextos. Os mesmos autores abordam 5 aspetos que consideram ser as melhores práticas: 1. Alinhamento estratégico - de acordo com a estratégia organizacional; 2. Adotar um conjunto de medidas financeiras e não financeiras; 3. Extrair medidas comparativas - tem de existir uma comparação de desempenho; 4. Comunicar/reportar os resultados regularmente - promove conhecimento e ação; 5. Dirigir o sistema do topo – gestores de topo tem de utilizar o sistema. De acordo com Zairi (1994), um PMS deve ser capaz de responder a várias perguntas como: x O que medir? x Como medir? x Onde medir? x Quando medir? x Porquê medir? Como se pode verificar, o PMS é um sistema complexo que engloba várias dimensões como a medição de desempenho, gestão estratégica, comunicação, influência comportamental e estímulo e melhoria contínua e aprendizagem (Franco-Santos et al., 2007). Esta complexidade faz com que o desenvolvimento e a implementação de um PMS efetivo sejam um desafio para as organizações. Apesar de existirem vários modelos PMS, a sua implementação, uso e manutenção continua a falhar. Apesar do PMS ser essencial numa organização, a medição de desempenho é frequentemente discutida mas raramente é definida (Neely et al., 1995). Alguns estudos feitos em PME (Pequenas e Médias Empresas) mostram que a maioria das tentativas de implementar um PMS falham (Langwerden, 2015; Malagueño et al., 2018; Neely, 2004; Todorut et al., 2013). Outros estudos mostram que cerca de 70% das tentativas falham (McCunn, 1998). O problema, que cria o maior desafio às organizações, parece ser a falta de uma metodologia para implementar indicadores de desempenho, integrá-los com a cultura organizacional e usá-los para a melhoria contínua (Zairi, 1994). Projetar um PMS, identificar os Key Performance Indicators (KPI) adequados e implementar um sistema de monitorização, representa atualmente um desafio para as unidades industriais. A medição é o verdadeiro desafio porque se pode errar de várias formas. Por exemplo, os indicadores medidos podem ser em excesso ou insuficientes, podem ser demasiado complexos ou demasiado simples. Além disso, são frequentemente parciais e desequilibrados, e a sua frequência de monitorização 4 pode ser inadequada. Podem ainda não ser compreendidos, interpretados ou utilizados corretamente, perdendo o significado para os seus utilizadores. De acordo com Franceschini et al. (2007) os principais e mais comuns problemas na implementação de um PMS são: x Acumular dados em excesso ou em quantidade insuficiente. Consequentemente, há dados que podem ser ignorados ou usados ineficientemente; x Foco em indicadores a curto prazo. A maior parte das organizações apenas recolhe dados financeiros e operacionais, ignorando indicadores que se focam no longo prazo; x Recolher dados inconsistentes, conflituosos e desnecessários; x Indicadores que não estão ligados aos objetivos estratégicos da organização; x Desequilíbrio no foco do desempenho da organização; x Medição do progresso feita muito frequentemente ou muito raramente. Zairi (1994) aponta 10 motivos pelos quais os PMS falham: x Falha na definição de desempenho operacional; x Falha em relacionar desempenho ao processo; x Falha na definição de fronteiras do processo; x Indicadores não são compreendidos ou são mal usados; x Falha na distinção entre medidas de controlo e medidas de melhoria; x Medir as coisas erradas; x Má compreensão ou mau uso de informação por parte dos gestores; x Medo de distorcer prioridades de desempenho; x Medo de expor o mau desempenho; x Medo de redução de autonomia. Um PMS ineficaz é prejudicial para a organização porque suporta decisões incorretas que levam à alocação incorreta de recursos escassos a iniciativas que irão falhar na entrega de resultados (Van Camp & Braet, 2016). Por isso existe uma necessidade evidente de um modelo de medição, uma forma de senso comum de medir o que é relevante (Napier & McDaniel, 2006). Existem várias investigações relacionadas com PMS. Ghalayini & Noble (1996) apresentam limitações dos PMS existentes. Outros, como Kennerley & Neely (2002) e Gabris (1986), selecionam e apresentam o que consideram ser as principais barreiras na implementação de um PMS. Existem ainda autores que ao invés de apresentarem as principais dificuldades ou problemas, identificam áreas críticas de um PMS, 5 como é o caso de Zairi (1994), ou apontam as características que um PMS deve ter, como é caso de Fitzgerald & Moon (1996). Este projeto de investigação diferencia-se dos anteriores porque identifica as principais barreiras à eficácia de um PMS e apresenta um modelo PMS com capacidade de mitigar ou eliminar essas barreiras, podendo-se observar os seus objetivos no capítulo seguinte. 1.2 Objetivos A pergunta de investigação está no centro do projeto de investigação. Influencia a escolha da literatura a ser revista, o projeto de investigação e a seleção dos métodos de recolha e de análise de dados (Saunders et al., 2019). Este projeto de investigação assenta em 2 perguntas de investigação: 1. Quais os principais fatores que dificultam a implementação, uso e manutenção de um sistema de medição de desempenho numa organização? 2. Como pode ser alcançada a melhoria da implementação, uso e manutenção de um sistema de medição de desempenho? As perguntas de investigação são a base para os objetivos de investigação, sendo estes os que permitem a operacionalização do tema de investigação (Saunders et al., 2019). Relativamente à primeira pergunta de investigação o objetivo é identificar quais os principais fatores que dificultam a implementação, uso e manutenção de um sistema de medição de desempenho numa organização. Quais são os principais problemas que as organizações enfrentam na implementação de um sistema de medição e controlo de desempenho. Relativamente à segunda pergunta de investigação o objetivo passa pelo desenvolvimento de um modelo PMS eficaz, estratégias e metodologias que permitam mitigar, ou eliminar, o impacto dos fatores que dificultam a implementação do sistema de medição de desempenho. 1.3 Metodologia De forma a alcançar os objetivos propostos anteriormente foram desenvolvidas investigações, com diferentes metodologias de investigação, seguindo o fluxo descrito na Figura 1. Os resultados dessas investigações foram publicados ou estão em processo de publicação em revistas indexadas. De seguida, são detalhadas as metodologias de investigação adotadas em cada um dos artigos. 6 Figura 1 – Fluxo da metodologia de investigação. O artigo “ Performance Measurement Systems in Continuous Improvement Environments: Obstacles to their Effectiveness ” (Cunha et al., 2023) seguiu uma metodologia de revisão sistemática de literatura, utilizando o método PRISMA ( Preferred Reporting Items for Systematic reviews and Meta-Analysis ) (Moher et al., 2009). Este método é caracterizado pelas fases de identificação, triagem, elegibilidade e inclusão de publicações. Na fase de identificação foram utilizadas as bases de dados Scopus e Web of Science . A pesquisa nestas 2 bases de dados foi feita com as seguintes restrições: “medição de desempenho” nas palavras-chave; publicações com 5 ou mais citações; e das áreas de Engenharia, Gestão, Negócio, Ciências Sociais ou Ciências de Computadores. Foram identificadas 1787 publicações na Scopus e 1728 na Web of Science . Inicialmente foram removidas as publicações duplicadas, resultando num total de 2808 publicações. Na fase de triagem, foram analisados os títulos e as palavras-chave das publicações, resultando na exclusão de 2550 publicações. Na fase de elegibilidade, foram analisados os resumos das 258 publicações restantes, resultando na exclusão de 176 publicações. Na fase de inclusão, foram analisadas as 82 publicações restantes na sua integridade, sendo 31 publicações incluídas no estudo. O resultado da revisão de literatura permitiu identificar as principais barreiras à eficácia de um sistema 7 de medição de desempenho, bem como as suas relações de interdependência. Foram identificadas 19 barreiras que foram agrupadas em 6 categorias. No artigo “ Analysis of barriers for performance measurement system effectiveness in a company: perceptions across hierarchical levels ” (Cunha et al., 2024d) foram exploradas as perceções da existência das 19 barreiras identificadas no estudo anterior nos diferentes níveis hierárquicos de uma organização. Para isso foram recolhidos dados primários (Saunders et al., 2019) através de entrevistas semiestruturadas e através de um questionário com uma classificação de escala de Likert de 5 pontos (Joshi et al., 2015), de 1 a 5 (1 – discordo totalmente; 2 – discordo; 3 – indiferente; 4 – concordo; 5 – concordo totalmente). Foram criados 3 tipos de entrevista para avaliar as perceções dos colaboradores operacionais, da gestão intermédia e da gestão de topo. A entrevista do nível operacional foi realizada a colaboradores operacionais de diferentes departamentos. Esta entrevista foi composta pelas seguintes 10 perguntas: 1. Conhece os objetivos da sua equipa/secção? 2. Quais as medidas/indicadores (que conhece) utilizadas para medir o desempenho da sua equipa/secção? 3. Existem alguma medida/indicador que não faz sentido para si? 4. Existem outras medidas/indicadores que acharia importante adotar? 5. Confia na fiabilidade das medições dos diferentes indicadores da sua equipa/secção? 6. De que forma os objetivos da sua equipa/secção contribuem para os objetivos da empresa? 7. Quando uma medida/indicador não está a atingir o objetivo é feita alguma coisa para corrigir e melhorar o desempenho? 8. Como se sente relativamente à utilização de indicadores na sua equipa? (se vê com bons olhos do ponto de vista individual) 9. Os indicadores são utilizados como forma de culpabilização das pessoas quando os objetivos não são atingidos? 10. Quais são as principais dificuldades (as principais barreiras) ao funcionamento do sistema de medição de desempenho? A entrevista da gestão intermédia foi feita aos chefes de turno e aos chefes dos diferentes departamentos. Foi composta pelas 9 perguntas seguintes: 1. Conhece os objetivos estratégicos da empresa? 8 2. Existe informação se os objetivos estão a ser atingidos ou não? Mantém-se informado sobre o estado dos objetivos estratégicos? 3. Todas as medidas/indicadores são úteis e fazem sentido? 4. Identifica a necessidade de outras medidas/indicadores que não existem atualmente? 5. As métricas/indicadores são utilizadas como forma de identificar onde se deve melhorar? São utilizadas como a base para a melhoria? 6. Confia na fiabilidade das medições dos diferentes indicadores do seu departamento? 7. Os indicadores são utilizados como forma de culpabilização do responsável do departamento quando os objetivos não são atingidos? 8. Considera que a estrutura de indicadores do seu departamento é apropriada? 9. Quais são as principais dificuldades (as principais barreiras) ao funcionamento do sistema de medição de desempenho? A entrevista da gestão de topo foi feita ao diretor geral, diretor financeiro e diretor de produção. As 7 questões que a compõem são as seguintes: 1. Os objetivos estratégicos estão definidos de acordo com a visão e valores da organização? 2. São comunicados de forma eficaz? (De que forma?) 3. Todos os indicadores estão alinhados com os objetivos estratégicos da organização? 4. Identifica a necessidade de outras medidas/indicadores que não existem atualmente? (Quais?) 5. Confia na fiabilidade das medições dos diferentes indicadores da sua organização? 6. Considera que a medição de desempenho é eficaz? (utilizado como ponto de partida para a melhoria) 7. Quais são os principais motivos que dificultam o funcionamento do sistema de medição de desempenho nesta empresa? O questionário foi aplicado a todas as pessoas entrevistadas e foi-lhes pedido que classificassem, de acordo com a escala de 5 pontos de Likert , a sua perceção da existência de cada uma das 19 barreiras à eficácia do PMS na organização. Para as entrevistas e para o questionário foi realizada uma análise estatística descritiva (Boone & Boone, 2012). Para a entrevista dos colaboradores operacionais e para os questionários foi feito o teste do quiquadrado (Pandis, 2016), para verificar a dependência das respostas do grau de escolaridade, género, turno, departamento ou nível hierárquico. As hipóteses testadas com o teste do chi-quadrado foram: 9 x H0: As variáveis são independentes; não existe nenhuma relação entre as variáveis; x H1: As variáveis são dependentes; existe uma relação entre as variáveis. Para testar as hipóteses, é necessário calcular as frequências esperadas para cada variável. Depois, a diferença entre frequências observadas e esperadas é avaliada através do teste do chi-quadrado, onde o p-value é obtido. Se o p-value for maior que 0,05, a hipótese nula (H0) é aceite. Se o p-value for menor que 0,05, a hipótese nula é rejeitada, o que significa que existe uma diferença significativa entre os valores observados e os valores esperados (Pandis, 2016). O artigo “ Assessment of Performance Measurement Systems Ability to Mitigate or Eliminate Typical Barriers Compromising Organisational Sustainability ” (Cunha et al., 2024a) seguiu uma metodologia de revisão sistemática de literatura, que foi feita segundo a metodologia PRISMA, com objetivo de identificar os principais modelos PMS existentes na literatura. A pesquisa de publicações foi conduzida nas bases de dados Scopus e Web of Science , utilizando as seguintes restrições: as palavras-chave “ performance measurement framework ” ou “ performance measurement model ” e as áreas de Engenharia, Negócios e Sociologia. Identificaram-se 2098 publicações na Scopus e 3784 na Web of Science . Inicialmente, foram removidos os itens duplicados, totalizando 5857 publicações exclusivas. Na fase de triagem, realizou-se uma análise dos títulos e das palavras-chave, resultando na exclusão de 5725 publicações. Em seguida, na fase de elegibilidade, foram analisados os resumos das 132 publicações restantes, das quais 60 foram excluídas. Por fim, na fase de inclusão, as 72 publicações restantes foram revistas na sua totalidade, resultando na inclusão de 39 publicações no estudo final. De forma a avaliar a capacidade dos 28 modelos identificados mitigarem ou eliminarem cada uma das 19 barreiras à eficácia de um PMS, estes foram classificados de acordo com a escala da Tabela 1. Tabela 1 - Escala de classificação da capacidade de um PMS eliminar ou mitigar uma barreira Valor Símbolo Significado 0 ▯ Fraca capacidade para eliminar ou mitigar 1 ▯ Alguma capacidade para eliminar ou mitigar 2 ▯ Forte capacidade para eliminar ou mitigar Esta investigação permitiu identificar os principais modelos PMS existentes, classificar a sua capacidade de eliminar ou mitigar cada uma das 19 barreiras à eficácia dos PMS e identificar o motivo pelo qual são capazes ou não de eliminar ou mitigar essas barreiras. O artigo “ Key Characteristics of Effective Performance Measurement Systems ” (Cunha et al., 2024b) seguiu uma metodologia de revisão sistemática de literatura, recorrendo à metodologia PRISMA. A pesquisa foi realizada com as seguintes restrições: a presença dos termos “ performance measurement ” e “ characteristics ” nas palavras-chave, no título ou no resumo, e publicações com pelo menos uma 10 citação. Como resultado, foram identificadas 2647 publicações na Scopus e 3905 na Web of Science . Após a remoção de publicações duplicadas, restaram 5762 publicações. A partir da análise dos títulos e palavras-chave, foram excluídas 5611 publicações. As 151 publicações restantes foram então submetidas a uma análise dos resumos, o que resultou na exclusão de 63 publicações. As 88 publicações restantes foram analisadas integralmente, sendo que em 27 delas foram identificadas características que um sistema eficaz de medição de desempenho (PMS) deve apresentar. As características para a eficácia do PMS descritas nessas 27 publicações foram inicialmente listadas e, em seguida, agrupadas com base nos seus significados compartilhados, resultando em 16 características. Este artigo permitiu identificar as principais características que um PMS deve ter para ser eficaz, explorando também as relações entre estas características e as principais barreiras a essa eficácia. O artigo “ Performance Measurements Systems: An Effective Model ” (Cunha et al., 2024c) é o artigo final deste estudo, onde é desenvolvida uma proposta de modelo e metodologia para um PMS eficaz, baseando-se no conhecimento e resultados obtidos nos 4 artigos anteriores. O modelo obtido foi classificado de acordo com a sua capacidade de eliminar ou mitigar as principais barreiras à eficácia de um PMS, de acordo com a metodologia apresentada anteriormente na Tabela 1. Foi também realizado um diagnóstico comparando o modelo obtido com um PMS existente numa organização. Para isso foi criado um questionário onde foi pedido aos respondentes que classificassem, de acordo com uma classificação de escala de Likert de 5 pontos (Joshi et al., 2015), de 1 a 5 (1 – discordo totalmente; 2 – discordo; 3 – indiferente; 4 – concordo; 5 – concordo totalmente), a existência das características do modelo obtido no PMS da organização. Os dados obtidos através do questionário foram analisados através de uma análise estatística e descritiva, sendo feito também o teste do qui-quadrado para verificar se as respostas obtidas eram independentes do grau de educação, género, turno, departamento, nível hierárquico na organização (operacional, gestão intermédia, gestão de topo). Através deste artigo foi possível obter uma proposta de um modelo e metodologia para projetar, implementar e utilizar um PMS de forma eficaz. 1.4 Estrutura da tese Esta tese é elaborada através de artigos publicados, ou em processo de publicação, em revistas indexadas, num total de 5 artigos. A estrutura a tese é composta por 7 capítulos. O primeiro capítulo é o da introdução, que é composto pelo enquadramento, objetivos, metodologia e pela descrição da estrutura da tese. O segundo capítulo, com o título de “Sistemas de Medição de Desempenho em Ambientes de Melhoria Contínua: Barreiras 11 à Sua Eficácia” apresenta o primeiro artigo desta investigação. O terceiro capítulo, com o título de “Análise das Barreiras à Eficácia de um Sistema de Medição de Desempenho numa Empresa: Perceções Através dos Níveis Hierárquicos” apresenta o segundo artigo desta investigação. O quarto capítulo, com o título de “Avaliação da Capacidade dos Sistemas de Medição de Desempenho para Mitigar ou Eliminar as Barreiras Típicas que Comprometem a Sustentabilidade Organizacional” apresenta o terceiro artigo desta investigação. O quinto capítulo, com o título de “Características-Chave de Sistemas de Medição de Desempenho Eficazes” apresenta o quarto artigo desta investigação. O sexto capítulo, com o título de “Sistemas de Medição de Desempenho: um Modelo Eficaz” apresenta o quinto artigo desta investigação. O sétimo e último capítulo apresenta as conclusões e o trabalho futuro, sendo abordadas as contribuições e limitações da investigação, bem como recomendações para investigações futuras. 18 In the first five positions, only the System category occurs twice, although its accumulated number of mentions (30) is lower than that of the most mentioned type of obstacle: inappropriate indicators. In general terms, it can be seen that slightly more than 30% of the types of obstacles correspond to 62% of the total number of mentions. The findings concerning the likely links between the types of obstacles to PMS effectiveness, as well as possible conditions required to eliminate or mitigate their impact, are discussed in the next section. 2.5 Discussion This section presents, for each of the identified categories and for each type of obstacle, the obstacles identified in the literature review. It also attempts to identify how a type of obstacle can emerge and how different types of obstacles can influence each other. 4.1. System Within the System category, six types of obstacles were identified (Figure 3): 1. Lack of connection to the strategy: This can be an obstacle to the effective functioning of a PMS and can originate in three ways: a. Failure to define strategic objectives: undeveloped or poorly developed strategic objectives do not allow the creation of the alignment necessary to effectively implement a PMS [14,17–22]; b. Failure to link the indicators to the strategic objectives: even with well-defined strategic objectives, there may be a failure in the link between these and the indicators in an ineffective PMS [18,23–26]; c. PMS does not keep up with changes in organizational strategy: a PMS that is not able to keep up with changes in organizational strategy could mean that, when any change in organizational strategy occurs, the link between Key Success Factors (KSF) and Key Performance Indicators (KPI) fails [17,20]. 2. Lack of use for improvement: Not using the PMS for continuous improvement makes it useless; it should be used as a support tool for the daily management of the organization [22]. The PMS alone will not translate into automatic improvements; it only allows identifying where improvements can and should be made [20,27]. If it does not have an effective improvement process associated with it, it will become irrelevant to the people in the organization [28–31]. 3. Issues on target definitions: This factor arises from the difficulty in defining targets and comparing performance with them [32]. Failure to define targets will impact people’s motivation and the 19 ability of the PMS to be used for the continuous improvement of the organization. This failure may occur when targets are not based on stakeholder interests, process boundaries and process improvement resources [29]. It may also occur if there is not a correct deployment of objectives from the top level of the organization to the level where the real improvement activities reside [29]. 4. Unclear system: vagueness in the performance measurement system can lead to a different use of the PMS from what was intended, dooming it to failure. This lack of clarity can arise in several aspects of the PMS: a. Failure to define measurement frequency: performance measurement occurs too often or too rarely [23]; b. Static system: the system is inflexible [28] and cannot be continuously revised and improved [19]. c. Failures in the definition of the PMS: the system has not yet reached the maturity (full definition) required to be implemented [22,33] and there may be failures in the definition of operational performance, in relating performance to the process, in defining the boundaries of the process [13]. There may also be vagueness related to the hierarchical structure and its deployment in the PMS [25,31,33,34] causing uncertainty of responsibility on performance measurement [35]. One of the causes mentioned is the direct use of another existing PMS model [14] which results in a PMS that is not adjusted to the organization [19]. 5. Communication system: Communication of performance measurement to employees plays an important role in involving employees in the PMS and maintaining its relevance. It is essential to ensure good communication between those who report and those who use the metrics [36]. This communication fails when it is not clear, simple, periodic and formal [19]. In order to be simpler, it must be visual [31]. Equally acting as an obstacle to the implementation and maintenance of a PMS is the fact that new processes and their impacts are not explained to employees [35], which can lead to a lack of commitment and lack of awareness. 6. Complexity: The more complex a system is, the more difficult it is to manage, the more resources and effort it takes to maintain it [14]. The complexity can also make it more difficult to communicate the system and its processes to employees, making their involvement more difficult. 20 4.2. Indicators Within the Indicators category, three types of obstacles were identified (Figure 3): 1. Inappropriate indicators: Performance indicators can be one of the factors that hinder the implementation, use and maintenance of a PMS, being pointed out as main reasons: a. Lack of long-term indicators: use of indicators with only a short-term focus, namely financial indicators [23,24,37]; b. Measuring the wrong things: using a set of indicators that has no relevance to the organization [13,18,19,21,24,26,30,38]; c. Outdated indicators: historical indicators with dated and irrelevant information [39]; d. Indicators that promote wrong behavior: indicators that promote wrong performance, indicators of courtesy instead of indicators of performance, indicators of behavior instead of indicators of achievement, and indicators that encourage competition rather than teamwork [37]; e. Poorly developed indicators: poorly defined indicators [29], confusing and complex [19]; f. Competing indicators: conflicts between indicators where the dependence and influence between them is not clearly defined [29,32,36,40]; g. Difficulty in developing indicators: uncertainty about what to measure, with difficulty in defining new performance indicators [22,24,35,36,39,41]. It can be caused by a failure to define the organization’s strategic objectives; h. Disaggregation of indicators: disaggregated in different dimensions, in different time periods [40], local and isolated indicators [36] not being properly integrated in the PMS. 2. Excess of indicators: the use of a high number of indicators will increase the complexity of the system, making it more difficult to maintain [19,26,32,39]. The more complex the system, the more resources are needed to maintain it and the more difficult it becomes to understand and use. 3. Lack of balance of indicators: failure to balance indicators causes an imbalance between different perspectives of the business, which can cause an imbalance in the organization’s performance [19,23,29,31]. 4.3. People Within the People category, four types of obstacles were identified (Figure 3): 21 1. False expectations: the expectations created by people regarding the PMS can represent an obstacle to maintaining it because they can be disappointed [42]. The organization will not improve just because the PMS has been implemented. An effective improvement process must be associated with it. If only performance is measured and nothing is achieved to improve it, the PMS can be abandoned, as it will not respond to false expectations of automatic improvement. 2. Lack of resources or capacity: for an effective implementation and maintenance of a PMS, it is essential that employees are educated and trained, with all the necessary skills, to understand and use the PMS correctly. The lack of training or understanding of the PMS represents an obstacle to its implementation and maintenance [14,21], [28,38,41,43] as it can lead to an incorrect use of the PMS, leading to its distortion and consequent abandonment. 3. Employee Commitment/Involvement: this involvement can fail when there is fear of performance measurement [13], which can result in increased resistance to the implementation and use of the PMS [20,39] and/or manipulation of the performance data [24,44]. Failure to motivate employees to use the PMS means that there is no commitment to change [34], and if the PMS is not relevant to people [19,30] resistance to its use increases [22,38], condemning it to failure. Conflicts and friction between employees may also arise as a result of performance measurement [35]. 4. Lack of indicator understanding: the non-understanding of performance indicators by employees may result from indicators that are not relevant to people [24], lack of training of employees to use the PMS [19] or high complexity in communicating information [35]. This can lead to a misuse of performance indicators through an incorrect interpretation of the meaning of the indicators [13]. Poor understanding of indicators can also lead to increase the resistance to use them [26,31]. 4.4 Culture Within the Culture category, three types of obstacles were identified (Figure 3): 1. Blame culture: using a PMS as a tool to coerce employees is referred thirteen times. Using performance measurement as a way of control and to put pressure on employees will create a blame culture [13,14,28,33,35,42], that will make the employees feel threatened [27,30,31,40,45]. This is one of the causes for the resistance of the employees to the PMS and for their lack of involvement in these practices [31]. In organizations with a blame culture, the PMS will not be used as tool to enable continuous improvement, and it may become a tool for punishing errors [19]. 22 2. Lack of commitment from top management: the lack of commitment from top management with the PMS [14,20,21,31,34,41], or the fact that it is considered a low priority [19,22] can convey the message to other employees that the PMS is not important. Additionally, a wrong comprehension or utilization of information by the top management [13,38] can pass the message to the rest of the organization that the top management is not fully committed to the PMS. 3. Lack of rewards: The lack of incentives, rewards or recognition for achieving goals is considered an obstacle [18,19] as it can result in the lack of motivation of employees, progressing to resistance to the PMS. 4.5. Technology Within the Technology category, two types of obstacles were identified (Figure 3): 1. Inadequate IT tools: the lack of adequate IT tools represents an obstacle to the implementation and maintenance of a PMS [33,35,36] because it can lead to increased difficulty in collecting, analyzing and presenting data. This difficulty causes an increase in the time and resources required to implement and maintain the PMS. 2. Time and resources required: the time and resources required to implement and maintain a PMS can represent an important obstacle to its effectiveness [17,20,45]. Required resources can be underestimated by top management causing a lack of resources allocated to the PMS [14,34]. The organization may be limited in terms of the costs and resources it can allocate to the PMS [21,22,24,39,41–43]. Due to the lack of resources allocated to the PMS, this can be seen as a burden for the organization because it removes employees from their real responsibilities [42]. 4.6. Data As for the Data category, just one type of obstacle was identified: 1. Difficulty in collecting, analyzing and presenting data: can occur for the following reasons: a. Too much data: accumulating too much data [23,37] can make it difficult to transform it into usable knowledge [19]; b. Insufficient data: not having enough data for what is intended to be measured [23], [37]; c. Difficulty in accessing data: technical complexity [35] due to the inadequacy of information systems and/or data dispersion [17,20,21,26,41]; 23 d. Data reliability: there are doubts regarding the reliability of the data [19,23] due to the fact that the available information is not appropriate [20] or due to the risk of data having been manipulated due to the existence of pressure to achieve goals [44]. In Figure 5, it is possible to identify the 19 types of obstacles and the interactions between them. Each circle represents one type of obstacle: in red are the obstacles from the category Culture, in blue from System, in green from Technology, in light blue from Data, in yellow from People and in dark blue from Indicators. The arrows express the probable cause–effect relationships between different obstacles. It is possible to observe, for example, that eight of these obstacles can be a cause for the lack of employee involvement. Figure 5 – Interaction between the types of obstacles. 2.6 Conclusion This study was centered on the research question “What are the main obstacles to effective performance measurement systems in organizations?”. The identification of those obstacles allows the establishment of the basis for future work that aims to explore how to create conditions to eliminate/mitigate them, allowing to implement, use and maintain a PMS effectively. 24 In the systematic literature review conducted, 175 references of obstacles to PMS effectiveness were identified. Those 175 referred obstacles were grouped, according to their similar meaning, into 19 types of obstacles. They were also classified into six different categories, namely: System, Indicators, People, Culture, Technology and Data. The identification of the types of obstacles to the effectiveness of a PMS should be the starting point to the process of eliminating or at least mitigating their impact. As discussed previously, the relation between the obstacles identified can be complex. The failure of a PMS can result from one or from the combination of several of the obstacles identified. Accordingly, in order to maximize the odds of achieving an effective PMS, methodologies and techniques that eliminate or at least mitigate the impact of the obstacles must be developed and used. I4.0 can help mitigate some of the obstacles to the effectiveness of a PMS, especially those related with the collection, analysis and reporting of data, through the digital transformation that enables an automatized real-time collection, analysis and reporting of data. On the other hand, for an organization to be able to move toward I4.0, an effective PMS already implemented is required, since performance measurement makes it possible to create the visibility required in a matured I4.0 organization. The first requirement to implement a PMS is related with the organizational culture. The culture of the organization is the foundation for the implementation and continuous use of a PMS. The organization must have a culture of respect for people and continuous improvement instead of a culture of punishment, and there must be commitment from the top management to performance measurement and improvement. These cultural characteristics are also mentioned by Amaro [46] as requirements for a sustainable Lean implementation, where behaviors of learning, improvement, adaptability, innovation, striving for new challenges, being open-minded and not blaming others are essential. Moreover, as way for motivating and encouraging improvement, a robust reward system must be in place. This reward system must issue rewards when targets are achieved and indicate when they are not. The reward system can have a financial component, but most important is to recognize where and when improvement is being achieved and where and when it is failing. It is essential that the system is robust and well-defined. When developing and defining a PMS, one must be sure that the organization’s strategy deployment is robust. One way of keeping all organizations aligned with the strategy is through the use of Hoshin Kanri, a process for strategy deployment. If the PMS is properly linked with the organizational strategy, it is easier to define what is essential to measure and what are the goals. It also allows keeping the PMS simple, only measuring what matters to the organization. 25 The deployment of both indicators and targets through the different levels of the organization is essential to involve and motivate all the employees. The PMS should be made a tool to help people with their work instead of being an extra task or bureaucracy. The performance indicators should be appropriate to all the stakeholders; for that, the recommendations of the ISO 22400:2014 can be followed, an international standard related with key performance indicators for manufacturing operations management. The set of indicators should be balanced and the influences that each one has on another should be explicit and well documented as way to avoid problems with concurrent indicators. The collection, analysis and reporting of data can be managed by the use of appropriate IT tools. That will depend on the complexity of the system and the amount of data that needs to be managed. In some cases, a set of automated and interconnected spreadsheets, with dashboards, can be the appropriate tool. As further investigation on this subject, recommendations that emerge as ways to eliminate or mitigate the obstacles to PMS effectiveness should be made. Those recommendations must be associated with methodologies and techniques that can be grouped and integrated in a model, resulting in a step-by-step guide that makes it possible to implement, use and maintain a PMS successfully. The main limitations of this study are the size and type of sample used in the systematic literature review. The number of publications is limited to only 31 from where 175 references of the obstacles are drawn. Moreover, the type of sample did not take into account publications cited less than five times. This could have kept valid publications on this subject out of this study, as may have been the case with newer publications that may have not reached five citations at the time when the systematic literature review was performed. 2.7 References 1. Imai, M. Gemba Kaizen, 2nd ed.; McGraw-Hill: New York, NY, USA, 1997. 2. Shin, W.S.; Dahlgaard, J.J.; Dahlgaard-Park, S.M.; Kim, M.G. A Quality Scorecard for the era of Industry 4.0. Total Qual. Manag. Bus. Excell. 2018, 29, 959–976. 3. Ante, G.; Facchini, F.; Mossa, G.; Digiesi, S. Developing a key performance indicators tree for lean and smart production systems. IFAC-PapersOnLine 2018, 51, 13–18. 4. Muhammad, U.; Ferrer, B.R.; Mohammed, W.M.; Lastra, J.L.M. An approach for implementing key performance indicators of a discrete manufacturing simulator based on the ISO 22400 standard. In Proceedings of the 2018 IEEE Industrial Cyber-Physical Systems (ICPS), Saint Petersburg, Russia, 15–18 May 2018; pp. 629–636. https://doi.org/10.1109/ICPHYS.2018.8390779. 26 5. Womack, J.; Jones, D. Lean Thinking: Banish Waste and Create Wealth in Your Corporation; Simon & Schuster; New York, NY, USA, 1996. 6. Dinis-Carvalho, J.; Macedo, H. Toyota Inspired Excellence Models. IFIP Adv. Inf. Commun. Technol. 2021, 610, 235246. https://doi.org/10.1007/978-3-030-92934-3_24. 7. Gupta, S.; Jain, S.K. A literature review of lean manufacturing. Int. J. Manag. Sci. Eng. Manag. 2013, 8, 241–249. https://doi.org/10.1080/17509653.2013.825074. 8. Kaplan, R.S.; Norton, D.P. The Balanced Scorecard Translating Strategy into Action; Harvard Business School Press: Boston, MA, USA, 1996. 9. Rossi, A.H.G.; Marcondes, G.B.; Pontes, J.; Leitão, P.; Treinta, F.T.; De Resende, L.M.M.; Mosconi, E.; Yoshino, R.T. Lean Tools in the Context of Industry 4.0: Literature Review, Implementation and Trends. Sustainability 2022, 14, 12295. https://doi.org/10.3390/su141912295. 10. Nagy, J.; Oláh, J.; Erdei, E.; Máté, D.; Popp, J. The Role and Impact of Industry 4.0 and the Internet of Things on the Business Strategy of the Value Chain—The Case of Hungary. Sustainability 2018, 10, 3491. https://doi.org/10.3390/su10103491. 11. Neely, A.; Gregory, M.; Platts, K. Performance measurement system design: A literature review and research agenda. Int. J. Oper. Prod. Manag. 1995, 15, 80–116. https://doi.org/10.1108/01443579510083622. 12. Bititci, U.S. Managing Business Performance; Wiley: New York, NY, USA, 2015. 13. Zairi, M. Measuring Performance for Business Results; Springer Science & Business Media: Berlin/Heidelberg, Germany, 1994. 14. McCunn, P. The Balanced Scorecard...the Eleventh Commandment. Manag. Account. 1998, 76, 34–36. https://www.proquest.com/trade-journals/balanced-scorecard-eleventh-commandment/docview/195676560/se2?accountid=39260 (accessed on 31 May 2022). 15. Moher, D.; Liberati, A.; Tetzlaff, J.; Altman, D.G.; Group, T.P. Preferred Reporting Items for Systematic Reviews and Me-ta-Analyses: The PRISMA Statement. PLoS Med. 2009, 6, 2-3. https://doi.org/10.1371/journal.pmed.1000097. 16. Liliana, L. A new model of Ishikawa diagram for quality assessment. In IOP Conference Series: Materials Science and Engineering; IOP Publishing: Bristol, UK, 2016; Volume 161. https://doi.org/10.1088/1757-899X/161/1/012099. 17. Bourne, M.; Neely, A.; Platts, K.; Mills, J. The success and failure of performance measurement initiatives: Perceptions of participating managers. Int. J. Oper. Prod. Manag. 2002, 22, 1288–1310. https://doi.org/10.1108/01443570210450329. 18. Watts, T.; McNair‐Connolly, C.J. New performance measurement and management control systems. J. Appl. Account. Res. 2012, 13, 226–241. https://doi.org/10.1108/09675421211281308. 19. Franco, M.; Bourne, M. Factors that play a role in ‘managing through measures. Manag. Decis. 2003, 41, 698–710. https://doi.org/10.1108/00251740310496215. 20. Bourne, M. Handbook of Performance Measurement; GEE Publishing: London, UK, 2004. 21. Charan, P.; Shankar, R.; Baisya, R.K. Modelling the barriers of supply chain performance measurement system implementation in the Indian automobile supply chain. Int. J. Logist. Syst. Manag. 2009, 5, 614–630. https://doi.org/10.1504/IJLSM.2009.024794. 22. de Waal, A.A.; Counet, H. Lessons learned from performance management systems implementations. Int. J. Product. Perform. Manag. 2009, 58, 367–390. https://doi.org/10.1108/17410400910951026. 27 23. Franceschini, F.; Galetto, M.; Maisano, D. Management by Measurement: Designing Key Indicators and Performance Measurement Systems; Spinger Science & Business Media: Berlin/Heidelberg, Germany, 2007. 24. Ghalayini, A.M.; Noble, J.S. The changing basis of performance measurement. Int. J. Oper. Prod. Manag. 1996, 16, 63–80. https://doi.org/10.1108/01443579610125787. 25. Dixon, J.R.; Nanni, A.J.; Vollmann, T.E. The New Performance Challenge; Dow Jones-Irwin: New York, NY, USA, 1990. 26. Hatten, K.J.; Rosenthal, S.R. Why‐and How‐to Systematize Performance Measurement. J. Organ. Excell. 2001, 20, 59–73. https://doi.org/10.1002/npr.1108. 27. Bourne, M.; Neely, A.; Mills, J.; Platts, K. Why some performance measurement initiatives fail: Lessons from the change management literature. Int. J. Bus. Perform. Manag. 2003, 5, 245–269. https://doi.org/10.1504/ijbpm.2003.003250. 28. Kennerley, M.; Neely, A. A framework of the factors affecting the evolution of performance measurement systems. Int. J. Oper. Prod. Manag. 2002, 22, 1222–1245. https://doi.org/10.1108/01443570210450293. 29. Schneiderman, A.M. Why balanced scorecards fail. J. Strateg. Perform. Meas. 1999, 2, 6–11. 30. Neely, A.; Bourne, M. Why Measurement Initiatives Fail. Meas. Bus. Excell. 2000, 4, 3–7. https://doi.org/10.1108/13683040010362283. 31. Meekings, A. Unlocking the potential of performance measurement: A practical implementation guide. Public Money Manag. 1995, 15, 5–12. https://doi.org/10.1080/09540969509387888. 32. Giovannoni, E.; Maraghini, M.P. The challenges of integrated performance measurement systems: Integrating mechanisms for integrated measures. Account. Audit. Account. J. 2013, 26, 978–1008. https://doi.org/10.1108/AAAJ-04-2013-1312. 33. Bititci, U.; Garengo, P.; Dörfler, V.; Nudurupati, S. Performance Measurement: Challenges for Tomorrow. Int. J. Manag. Rev. 2012, 14, 305–327. https://doi.org/10.1111/j.1468-2370.2011.00318.x. 34. Townley, B.; Cooper, D.J.; Oakes, L. Performance Measures and the Rationalization of Organizations. Organ. Stud. 2003, 24, 1045–1071. https://doi.org/10.1177/01708406030247003. 35. Okwir, S.; Nudurupati, S.S.; Ginieis, M.; Angelis, J. Performance measurement and management systems: A perspective from complexity theory. Int. J. Manag. Rev. 2018, 20, 731–754. https://doi.org/10.1111/ijmr.12184. 36. Lohman, C.; Fortuin, L.; Wouters, M. Designing a performance measurement system: A case study. Eur. J. Oper. Res. 2004, 156, 267–286. https://doi.org/10.1016/S0377-2217(02)00918-9. 37. Brown, M.G. Keeping Score: Using the Right Metrics to Drive World-Class Performance; CRC Press: Boca Raton, FL, USA,: 1996. 38. Radu-Alexandru, Ș.; Mihaela, H. Performance Management Systems—Proposing and Testing a Conceptual Model. Stud. Bus. Econ. 2019, 14, 231–244. https://doi.org/10.2478/sbe-2019-0018. 39. Nudurupati, S.S.; Bititci, U.S.; Kumar, V.; Chan, F.T.S. State of the art literature review on performance measurement. Comput. Ind. Eng. 2011, 60, 279–290. https://doi.org/10.1016/j.cie.2010.11.010. 40. Wouters, M.; Wilderom, C. Developing performance-measurement systems as enabling formalization: A longitudinal field study of a logistics department. Account. Organ. Soc. 2008, 33, 488–516. https://doi.org/10.1016/j.aos.2007.05.002. 41. Sousa, S.D.; Aspinwall, E.M.; Rodrigues, A.G. Performance measures in English small and medium enterprises: Survey results. Benchmarking Int. J. 2006, 13, 120–134. https://doi.org/10.1108/14635770610644628. 34 6. Are metrics/indicators used as a way to identify where to improve? Are they used as the basis for improvement? 7. Do you trust the reliability of measurements of the different indicators in your department? 8. Are indicators used as a way of blaming the department head when objectives are not achieved? 9. Do you think your department’s indicator structure is appropriate? 10. What are the main difficulties (the main barriers) to the effectiveness of the PMS? The above list is the authors’ own creation. The top management interview was deployed to the general manager, production manager and financial manager of the organization. The seven questions that composed this interview are presented in the following list. Questions of the top management interview are as follows: 1. Are the strategic objectives defined in accordance with the organization’s vision and values? 2. Are they communicated effectively? (In what way?) 3. Are all indicators aligned with the organization’s strategic objectives? 4. Do you identify the need for other measures/indicators that currently do not exist? (which?) 5. Do you trust the reliability of measurements of the different indicators of your organization? 6. Do you think performance measurement is effective? (used as a starting point for improvement) 7. What are the main reasons that make it difficult for the PMS to work in this company? The above list is the authors’ own creation. Thirty-two interviews were performed, 23 for employees, 6 for middle management and 3 for top management. The interviews were recorded and then transcribed. After being transcribed, the answers with similar meanings were organized into groups where the answers had the same meaning to allow a statistical analysis of the answers given in the interviews. The short questionnaire was deployed to the same individuals who were interviewed right after the interview. Here, the questionnaire was the same independent of the role in the company. In this questionnaire, individuals were asked to classify, using a Likert scale, their perception of the presence in the organization of commonly known barriers to PMS effectiveness. A five-point Likert scale was used (Joshi et al.,2015), ranging from 1 to 5 (1 - Strongly disagree, 2 - Disagree, 3 - Indifferent, 4 - Agree and 5 - Strongly agree). The barriers that respondents were asked to classify were: blame culture, unclear system, high complexity, lack of rewards, excess of indicators, inappropriate indicators, lack of connection to strategy, 35 lack of top management involvement, lack of use for improvement, false expectations, lack of trained resources, lack of indicators understanding, issues on target definition, poor communication system, lack of balance of indicators, inappropriate information technology (IT) tools, lack of employee involvement, time and resources required and difficulties in collecting, analyzing and presenting data (Cunha et al., 2023). For the interviews and the questionnaire, a descriptive statistical analysis was performed (Boone and Boone, 2012). Also, for the data from the employees interviews and questionnaires, a chi-square test (Pandis, 2016) was performed to ascertain whether the answers depend on the respondents degree of education, gender, shift or department. For the questionnaire it was also tested if the answers depend on the role in the company (employees, middle management and top management). The hypothesis tested with the chi-square test were: x H0. The variables are independent; there is no relationship between the categorical variables. x H1. The variables are dependent; there is a relationship between the categorical variables. To test these hypotheses, first, it is required to calculate the expected frequencies for each variable. Then, the difference between observed and expected frequencies is assessed through the chi-square test, where the p-value is obtained. If the p-value is greater than 0.05, the null hypothesis (H0) is accepted. This means that there is a small difference between the observed and the expected values. If the p-value is smaller than 0.05, the null hypothesis is rejected, meaning that there is a large difference between the observed and the expected values (Pandis, 2016). The survey results and the interview findings were analyzed, establishing a connection between the survey outcomes and the responses provided during the interviews. 3.5 Results This section presents, for each type of interview deployed, as well as for the questionnaire, the main results drawn from the data treatment and statistical analysis. Twenty-three employees’ interviews were performed. Of those interviewed, the population is characterized by: x Gender: 16 (70%) were female and 7 (30%) were male; x Education: 21 (91%) had basic education and 2 (9%) had higher education; x Department: 20 (87%) were from the production department and 3 (13%) from other departments; and 36 x Shift: 13 (57%) were from Shift A (from 6:00 am to 2:30 pm), 8 (35%) from Shift B (from 2:30 pm to 11:00 pm) and 2 (8%) from the normal shift (from 8:30 am to 5:30 pm). Regarding the answers of employees’ interviews: x 91% of the respondents did not know the objectives of their team or section. Through the chisquare test, it was possible to verify that the answers given depended on the degree of education, gender, shift and the department of the respondents; x 30% of the respondents did not know any indicators, 61% knew one indicator and only 9% knew the indicators. The answers given depended on the degree of education, shift and the department of the respondents. The answers were not dependent on the gender of the respondents; x 30% of the respondents identified indicators that did not make sense to them. The answers were not dependent on the degree of education, gender, shift or department of the respondents; x 74% of the respondents identified the need to adopt new indicators. The answers given depended on the degree of education, shift and the department of the respondents. The answers were not dependent on the gender of the respondents; x 65% of the respondents did not trust the reliability of the indicators. The answers were dependent on the degree of education and department of the respondents but not dependent on the gender and shift of the respondents; x 83% of the respondents did not know how their team goals contribute to the company goals. The answers given depended on the degree of education, shift and the department of the respondents. The answers were not dependent on the gender of the respondents; x 83% of the respondents did not feel an improvement culture. The answers were not dependent on the degree of education, gender, shift or department of the respondents; x 87% of the respondents feel good about the use of performance indicators in their team. The answers were not dependent on the degree of education, gender, shift or department of the respondents; x 43% of the respondents feel a blame culture when the objectives are not achieved. The answers were not dependent on the degree of education, gender, shift or department of the respondents; x 52% of the respondents identified the communication system as a barrier to the PMS effectiveness in the organization. Also, 22% identified a lack of employee involvement as a barrier to PMS effectiveness. Other barriers referred to were: blame culture, issues on target definition, unclear system, lack of rewards and lack of use for improvement. The answers were not dependent on the degree of education, gender, shift or department of the respondents. 37 The chi-square test was performed to verify if the answers, given by the respondents, are independent of the variables: degree of education, gender, shift and department. The hypothesis accepted and the pvalue are presented in Table 2. Table 2 – Employees interview chi-square hypothesis results and p-value. (Notes: Data is the p-value of the chi-square test. The significance level is 0,05.) (Source: Authors’ own creation). Degree of Education Gender Shift Department Do you know the objectives of your team/section? H1 (0,000) H1 (0,025) H1 (0,000) H1 (0,008) What are the measures/indicators (that you know) used to measure the performance of your team/section? H1 (0,000) H0 (0,077) H1 (0,000) H1 (0,000) Are there any measures/indicators that do not make sense to you? H0 (0,082) H0 (0,668) H0 (0,242) H0 (0,241) Are there any other measures/indicators that you would find important to adopt? H1 (0,000) H0 (0,062) H1 (0,001) H1 (0,001) Do you trust the reliability of the measurements of the different indicators of your team/section? H1 (0,019) H0 (0,848) H0 (0,056) H1 (0,038) How do your team/section goals contribute to the company's goals? H1 (0,000) H0 (0,061) H1 (0,000) H1 (0,001) When a measure/indicator is not achieving the objective, is anything done to correct and improve performance? H0 (0,203) H0 (0,349) H0 (0,438) H0 (0,435) How do you feel about the use of indicators in your team? H0 (0,567) H0 (0,219) H0 (0,833) H0 (0,472) Are indicators used as a way of blaming people when objectives are not achieved? H0 (0,194) H0 (0,340) H0 (0,424) H0 (0,103) What are the main difficulties (the main barriers) to the effectiveness of the performance measurement system? H0 (0,977) H0 (0,582) H0 (0,851) H0 (0,974) Six middle management interviews were performed. Of those interviewed, the population is characterized by: x Gender: three (50%) were female and three (50%) were male; x Education: four (67%) had basic education and two (33%) had higher education; x Department: three (50%) were from the production department and three (50%) were from other departments; and x Shift: two (33%) were from Shift A and four (67%) were from the normal shift. Regarding the middle management interview: x None of the respondents were able to identify the company’s strategic objectives; 38 x 83% of the respondents consider that there is no information available on whether the objectives are being achieved or not; x 50% of the respondents considered that there are some indicators that are not useful or do not make sense; x 83% of the respondents identify the need for other indicators that currently do not exist; x 50% of the respondents did not feel an improvement culture; x 33% of the respondents did not trust the reliability of the different indicators in their department; x 33% of the respondents felt that there was a blame culture in the organization; x 50% felt that their department indicator structure was not appropriate; and x 50% of the respondents identified the communication system as a barrier to the PMS effectiveness in the organization. Other barriers to the PMS effectiveness identified were inappropriate indicators, lack of employee involvement, lack of top management involvement, lack of use for improvement, time and resources required and unclear system. For the middle management interview, the chi-square test was not performed because the size of the sample (six) was too small. Three top management interviews were performed. Of those interviewed, the population is characterized by: x Gender: two (67%) were female and one (33%) were male; x Education: three (100%) had higher education; x Department: one (33%) was from production department and two (67) were from other departments; and x Shift: three (100%) were from the normal. Regarding the top management interview: x All the respondents considered that the strategic objectives were in accordance with the organization’s vision and values. x All the respondents considered that the communication of the objectives is not yet made effectively throughout the organization. x All the respondents considered the indicators aligned with the organization’s strategic objectives. x 67% of the respondents identified the need for other indicators that currently do not exist. x 67% of the respondents did not fully trust the reliability of measurement of the different indicators of the organization. 39 x All the respondents did not feel an improvement culture in the organization. x 67% of the respondents identified the lack of use for improvement as a barrier to the PMS effectiveness in the organization. Other barriers to the PMS effectiveness identified were communication system, lack of indicators understanding and unclear system. For the top management interview the chi-square test was not performed because the size of the sample (three) was too small. Thirty-two questionnaires were performed. Those who answered the questionnaire are characterized by the following: x Gender: 21 (66%) were female and 11 (34%) were male; x Education: 25 (78%) had basic education and 7 (22%) had higher education; x Department: 24 (75%) were from the production department and 8 (25%) were from other departments; and x Shift: 15 (47%) were from Shift A, 8 (25%) were from Shift B and 9 (28%) were from the normal shift. Participants were asked to classify if they strongly agree, agree, are indifferent, disagree or completely disagree, with the existence inside the organization, of each of the 19 individual barriers to the effectiveness of its PMS. The results of the classification are presented in Figure 1. Figure 1 – Likert scale classification of the existence of the barriers in the organization. (Source: Authors’ own creation) 91% 81% 75% 72% 72% 72% 69% 69% 69% 66% 66% 53% 53% 47% 44% 31% 31% 19% 3% 3% 3% 0% 3% 3% 25% 3% 0% 0% 6% 6% 25% 3% 6% 22% 22% 22% 15% 6% 6% 16% 25% 25% 25% 3% 28% 31% 31% 28% 28% 22% 44% 47% 34% 47% 47% 66% 91% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Agree/ Strongly Agree Indifferent Disagree /Strongly disagree 40 It is possible to verify that 91% of the respondents agreed or strongly agreed that the communication system is currently a barrier to the PMS effectiveness in the organization. The second barrier mostly agreed by the respondents was the lack of trained resources, with 81%. The third barrier mostly agreed was issues in defining targets, with 75%. Following with 72% were lack of employee involvement, lack of understanding the indicators and lack of use for improvement. The barriers least agreed were: excess of indicators (3%), lack of connection to strategy (19%), inappropriate IT tools (31%) and inappropriate indicators (31%). The chi-square test was performed to verify if the classification, made by the respondents, is independent of the variables: degree of education, gender, shift, department or role in the company. The results are presented in Table 3. Regarding the poor communication system barrier, the chi-square test revealed that the answers are dependent on the gender of the respondents. This happens because only two males disagreed that communication system is a barrier to the current PMS effectiveness, while all females classified it with agree or strongly agree. Only the two respondents of the engineering department know the objectives, probably justifying the results of the chi-square test that identified that the answers given were dependent on the degree of education, gender, shift and department of the respondents. The two interviewed from the engineering department were both male, higher educated and from the normal shift. Also, 91% of the respondents either did not know any indicators or only knew one indicator, and 83% did not know how their team objectives contributed to the company objectives. The answers to both these questions were dependent on the degree of education, shift and department of the respondents. Here, again, this dependency might be due to the answers given by both members of the engineering department. Regarding lack of training and lack of understanding of the PMS, through the chi-square test is possible to verify that the classification of these barriers is independent of the degree of education, gender, shift, department and role in the company of the respondents. The perception of the barriers lack of employee involvement and lack of use for improvement are only dependent on the gender of the respondents. The lack of rewards or recognition for achieving goals is dependent on the degree of education, shift and department. This barrier is mostly perceived by people with basic education, from Shifts A and B and from the production department. The lack of top management commitment with the PMS is dependent on the degree of education, gender, shift, department and role in the company. This barrier is perceived by 74% of the employees and by 67% 41 of the intermediate management. On the other hand, 100% of the top management disagree or strongly disagree that there is a lack of top management commitment. It is possible to verify that the perception of the top management is significantly different from the other roles in the company. 3.6 Discussion This first part of the discussion is about statistically relevant differences in perceptions among different groups in the company as shown in Table 3. The authors believe it is a valuable contribution to the communities of managers and academics since these different perceptions identified in this case study may be also observed in other companies, even with different PMS approaches and maturity levels. The second part of this discussion is dedicated to the overall results from the questionnaire in this particular case. Table 3 – Questionnaire chi-square hypothesis results and p-value. (Notes: Data is the p-value of the chi-square test. The significance level is 0,05.) (Source: Authors’ own creation). Degree of Education Gender Shift Department Role in the company Poor communication system H0 (0,541) H1 (0,042) H0 (0,720) H0 (0,603) H0 (0,269) Lack of trained resources H0 (0,302) H0 (0,205) H0 (0,054) H0 (0,054) H0 (0,673) Issues on target definition H0 (0,805) H1 (0,005) H0 (0,766) H0 (0,059) H0 (0,539) Lack of employee involvement H0 (0,143) H1 (0,042) H0 (0,487) H0 (0,065) H0 (0,940) Lack of indicators understanding H0 (0,850) H0 (0,587) H0 (0,563) H0 (0,568) H0 (0,940) Lack of use for improvement H0 (0,850) H1 (0,018) H0 (0,435) H0 (0,156) H0 (0,481) Lack of rewards H1 (0,001) H0 (0,246) H1 (0,001) H1 (0,003) H0 (0,055) Unclear system H0 (0,094) H1 (0,004) H0 (0,173) H1 (0,028) H0 (0,359) Time and resources required H0 (0,094) H0 (0,210) H0 (0,600) H0 (0,660) H0 (0,308) Lack of top management involvement H1 (0,015) H1 (0,038) H1 (0,046) H1 (0,003) H1 (0,046) Lack of balance of indicators H0 (0,509) H0 (0,218) H0 (0,279) H0 (0,602) H0 (0,486) Difficulties in collecting, analyzing and presenting data H0 (0,545) H0 (0,222) H0 (0,350) H0 (0,068) H0 (0,199) False expectations H0 (0,857) H0 (0,349) H0 (0,345) H0 (0,167) H0 (0,281) Blame Culture H0 (0,610) H0 (0,273) H0 (0,753) H0 (0,315) H0 (0,872) High complexity H0 (0,809) H0 (0,179) H0 (0,652) H0 (0,203) H0 (0,538) Inappropriate indicators H0 (0,209) H0 (0,819) H0 (0,313) H0 (0,221) H0 (0,247) Inappropriate IT tools H0 (0,059) H0 (0,102) H0 (0,149) H1 (0,026) H0 (0,327) Lack of connection to strategy H0 (0,752) H0 (0,073) H0 (0,405) H0 (0,806) H0 (0,192) Excess of indicators H0 (0,125) H0 (0,420) H0 (0,456) H0 (0,159) H0 (0,269) 42 Regarding the first part of this discussion, the cells with the text in italics in Table 3 are the ones showing the statistically relevant differences in the perception between different classes of people. It is possible to verify that the perception is dependent on the degree of education for the barriers “lack of rewards” and “lack of top management involvement.” For these barriers, the classification agree/strongly agree was made significantly more by people with basic education as can be observed in Table 4. It is interesting to notice that people with basic education pay more attention to rewards. This can be discussed based on the argument that people with less intellectually challenging work may need more extrinsic motivation (rewards) than people with more challenging work assuming that people with only basic education are more involved in less intellectually challenging jobs. The differences in perception regarding “Lack of top management involvement” could be explained by the fact that top management people do not consider themselves as not being involved, and they are normally more educated than most of the workers. Table 4 – Percentage of people by degree of education that classified with agree or strongly agree (Source: Authors’ own creation). Higher Education Basic Education Lack of rewards 14,3% 84,0% Lack of top management involvement 28,5% 76,0% The classification is dependable of the variable gender for the barriers presented in Table 5. The female respondents agreed or strongly agreed with the existence of these barriers significantly more than the male respondents. The effect here can be explained by the fact that, in this company, senior and middle management are predominantly male while the workers are predominantly female. Since middle and top management are more involved in the PMS they tend to be more tolerant of its faults. Table 5 – Percentage of people by gender that classified with agree or strongly agree (Source: Authors’ own creation). Male Female Poor communication system 72,7% 100,0% Issues on target definition 45,5% 90,4% Lack of employee involvement 45,5% 85,7% Lack of use for improvement 45,5% 85,7% Unclear system 36,4% 85,7% Lack of top management involvement 36,4% 81,0% Shift is a variable that makes the classification dependable for the barriers “lack of rewards” and “lack of top management involvement”. Respondents from the normal shift did not classify the existence of these barriers with agree or strongly agree as much as did the respondents from Shifts A and B (Table 43 6). The possible reason for this difference in perception could be that there are many more middle and top managers on the normal shift than on the other shifts. Table 6 – Percentage of people that classified with agree or strongly agree by shift (Source: Authors’ own creation). Shift A Shift B Normal Shift Lack of rewards 86,7% 85,7% 22,2% Lack of top management involvement 80,0% 75,0% 33,3% The classification of the barriers listed in Table 7 depends on the variable department. Respondents from the production department classified the existence of these barriers, with agree or strongly agree, more than respondents from other departments (Table 7). This difference may be due to the fact that it is in the production department where there are more workers with less training and who are less involved in developing the PMS. Table 7 – Percentage of people by department that classified with agree or strongly agree (Source: Authors’ own creation). Production Department Other Lack of rewards 83,3% 25,0% Unclear system 79,2% 37,5% Lack of top management involvement 79,2% 25,0% Inappropriate IT tools 37,5% 12,5% The classification of the barrier “top management involvement” is dependable of the variable role in the company. No respondents from top management agreed or strongly agreed with the existence of this barrier, while the majority of respondents from middle management and employees agreed or strongly agreed with the existence of this barrier (Table 8). This effect is expected since top management is normally involved in the PMS implementation and for that reason they normally have the perception that they are involved. Middle management and workers do often feel it in a different way in companies with no real culture of continuous improvement operational excellence. Table 8 – Percentage of people by role in the company that classified with agree or strongly agree (Source: Authors’ own creation). Top management Middle management Employees Lack of top management involvement 0,0% 66,7% 73,9% This second part of the discussion is dedicated to the overall results from the questionnaires. The main barrier perceived by people in the organization was the ineffective communication system. The communication system is ineffective when it is not simple, clear, periodical and formal (Franco and Bourne, 2003). Not only 91% of the respondents agree or strongly agree that this is a barrier to the PMS 50 4. AVALIAÇÃO DA CAPACIDADE DOS SISTEMAS DE MEDIÇÃO DE DESEMPENHO PARA MITIGAR OU ELIMINAR AS BARREIRAS TÍPICAS QUE COMPROMETEM A SUSTENTABILIDADE ORGANIZACIONAL Este capítulo apresenta o artigo “ Assessment of Performance Measurement Systems’ Ability to Mitigate or Eliminate Typical Barriers Compromising Organisational Sustainability ” (Cunha et al., 2024a), que foi publicado na revista Sustainability a 03/03/2024. Este estudo identifica, através de uma revisão sistemática da literatura, os modelos PMS existentes e classifica-os de acordo com a sua capacidade de mitigar ou eliminar as barreiras que comprometem a eficácia de um PMS. 4.1 Abstract and Keywords Abstract: This paper aims to identify the main performance measurement systems (PMSs) documented in the literature and assess their ability to overcome/mitigate a set of 19 specific barriers (identified in a previous paper) to their effectiveness. It also aims to understand what makes each PMS capable of or not capable of dealing with these barriers (i.e., what traits it has) and to explore their connection to some sustainable development goals (SDG). The PRISMA methodology was used to identify the relevant publications, which were then subjected to a detailed content analysis with statistical treatment, followed by the assessment of the potential of each PMS to deal with the barriers. The results made it possible to identify the PMSs most referred to in the literature (ordered list), quantitatively classify the PMSs according to their ability to overcome/mitigate barriers, and identify the barriers most and least addressed by the PMSs. While no single PMS offers a comprehensive solution, certain common traits contribute significantly to overcoming prevalent barriers. The complex interplay between barriers means that some traits can effectively address multiple barriers either directly or indirectly. Regarding implications, these findings provide important inputs (e.g., key recommendations) for developing or improving PMS frameworks that are able to comprehensively address the barriers, thus contributing to organisational effectiveness and, consequently, to the achievement of the SDGs. This constitutes the innovative contribution of this paper. As for limitations, this work is based on the analysis of 28 PMSs resulting from the systematic literature review in two databases (Scopus and Web of Science). Keywords: performance measurement system; key performance indicators; continuous improvement; operational excellence; lean production; sustainable development goals. 51 4.2 Introduction In today’s dynamic business environment, maintaining success requires organisations to adopt a flexible approach to evolving conditions, forcing them to adopt strategies to be more competitive and sustainable [1]. The notion of sustainable development is grounded in the principles of development (socio-economic progress within ecological limits), needs (equitable resource distribution to uphold the quality of life for all), and future generations (ensuring resources are utilized responsibly for the sustained well-being of future generations) [2]. All United Nations Member States endorsed the 2030 Agenda for Sustainable Development, a broad initiative centred on 17 Sustainable Development Goals (SDGs), which emphasise the importance of broader societal issues [3]. This endorsement established a developmental framework prioritising the eradication of poverty and deprivations, the enhancement of health and education, and the stimulation of economic growth. Simultaneously, the agenda addresses the critical issue of environmental resource degradation on a global scale [4]. Within this main framework, some goals are dedicated to improving production methods with a particular focus on SDG8 and SDG9. These goals endeavour to champion full and productive employment and sustainable industrialisation, innovation, and infrastructure, playing a pivotal role in advancing technological progress and advocating for sustainable production processes. The ultimate aim is to mitigate the adverse environmental impacts associated with industrialisation. Another pertinent goal within this context is SDG 12, which emphasises responsible consumption and production [5]. SDG 12 aims to reduce waste, enhance resource efficiency, and foster sustainable practices in both production and consumption, contributing to the broader goal of sustainable development. For an organisation that aims to be competitive and sustainable, it becomes imperative to regularly monitor and evaluate its performance and to make appropriate decisions and actions. So, it is only natural that the performance of an organisation has become the preferred subject of interest of managers since the end of the 20th century [6]. The performance and measurement system (PMS) plays a pivotal role in guiding an organisation’s journey toward continuous improvement and sustainability. It plays a crucial role in revealing the current status of an organisation by providing a foundational starting point for improvement by identifying specific areas where enhancements are needed and should be undertaken. The main functions of a PMS are to create organisational alignment and to translate strategy into action [7]. Bititci (2015) [8] defines the development and use of a PMS as the process of defining objectives; developing a set of performance metrics; and collecting, analysing, reporting, interpreting, reviewing, and acting on performance data. The extensive and intricate nature of that process makes the development 52 and implementation of a PMS a significant challenge for organisations [7], hindering those organisations competitiveness and sustainability. These challenges often give rise to barriers that can impede the overall effectiveness of a PMS. In a prior study conducted by Cunha et al. (2023) [9], a comprehensive examination was undertaken through a systematic literature review. This investigation enabled the identification and categorisation of the primary obstacles that have the potential to hinder the effectiveness of a PMS. The comprehensive list of these barriers (19) is presented in Table 1. Notably, this systematic literature review followed the PRISMA methodology (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) and was specifically tailored to pinpoint the prevailing barriers frequently cited in the literature as impediments to the effectiveness of a PMS. Table 1 – Main barriers to PMS effectiveness 1. blame culture 11. false expectations 2. lack of connection to strategy 12. inappropriate IT tools 3. issues on target definition 13. excess of indicators 4. unclear system 14. lack of trained resources 5. lack of top management involvement 15. lack of employee involvement 6. poor communication system 16. inappropriate indicators 7. high complexity 17. lack of indicator understanding 8. lack of use for improvement 18. time and resources required 9. lack of balance of indicators 19. difficulties in collecting, analysing, and presenting data 10. Lack of rewards Since the barriers have already been identified, it is now crucial to explore the existing PMS frameworks (i.e., not only the PMS software/application itself but also the inherent design and development processes) and their capacity to address these barriers (namely, the tools and resources utilised to either eliminate or mitigate the barriers). Therefore, the primary objectives of this study are to systematically identify the spectrum of PMSs documented in the existing literature and to scrutinise the capabilities of these systems in addressing and mitigating or eliminating the several barriers that may hinder their effectiveness. There are several studies in the literature that compile and analyse the existing PMSs; however, they have different objectives than the present study. Taticchi et al. (2012) [10] focused on identifying PMSs to offer research guidelines for building a PMS, pinpointing some design challenges. Yadav et al. (2014) [11], on the other hand, explored the existing PMSs mainly to assess the state of the art and establish connections with strategic management theories. In another study, Yadav et al. (2013) [12] centred on compiling and analysing PMSs, specifically aiming to scrutinise the research trends over the last two decades. Folan and Browne (2005) [13] contributed to the literature by outlining the evolution of performance 53 measurement and providing recommendations, frameworks, systems, and insights into interorganisational performance measurement. Thus, the authors argue that a research gap exists, as the ability of PMSs to deal with barriers to their effectiveness is not adequately addressed by the existing literature. The scientific novelty of this article lies precisely in its contribution to filling this research gap. This paper is structured according to the flowchart in Figure 1. Figure 1 – Structure of the paper 4.3 Methods As a way of identifying and understanding the PMSs that exist in the literature, a systematic literature review (SLR) was conducted. The SLR was performed following the steps of the PRISMA methodology, which is composed of four phases: identification, screening, eligibility, and included publications. In the first phase, a search for scientific articles, books, and book chapters was executed on the databases Scopus and Web of Science. The search on the databases was performed with the following restrictions: “performance measurement framework” or “performance measurement model” as keywords and from the subject areas of Engineering, Business, and Sociology. As a result of the search, we identified 2098 publications from the Scopus database and 3784 publications from the Web of Science database. The steps followed during the application of the PRISMA methodology are described in Figure 2. After identifying the publications according to the restrictions, the first step was to remove the duplicate results found in the two databases. In this step, the 25 identified duplicate results were removed. The remaining 5857 publications were screened by analysing their title and keywords, and 5725 were removed during this step. The remaining 132 publication abstracts were analysed, and 60 publications were excluded. The remaining 72 publications were fully analysed, and in 39 of them, PMS frameworks or models were identified. 54 Figure 2 – Results of PRISMA methodology application The references and/or descriptions of the PMSs found in the 39 publications were recorded in a table (not presented in this paper) containing the name of the PMS and the title of the publication that mentions it. Some publications refer to several PMSs, and some PMSs are referred to in several publications (manyto-many relationship). The frequency with which each PMS is mentioned was determined, and it was decided to analyse only those that are mentioned more than once (to limit the size of the study due to the high number of PMSs found). To analyse these PMSs, a detailed reading of the 39 publications identified by PRISMA was carried out. This rigorous scrutiny made it possible, for each PMS, to perceive its ability to deal with each of the 19 barriers identified in Table 1. Based on this perception, each PMS was classified in relation to each barrier according to the values shown in Table 2. The total score for each PMS is obtained by adding up the values obtained by the PMS with regard to its ability to deal with each barrier. Table 2 – Scale of values to classify the capacity of a PMS to mitigate or eliminate a barrier. Value Symbol Meaning 0 ▯ Weak capacity to mitigate or eliminate 1 ▯ Some capacity to mitigate or eliminate 2 ▯ Strong capacity to mitigate or eliminate The results of that analysis are described in Section 3 and discussed in Section 4, leading to the outline of conclusions in Section 5. 55 4.4 Results This section presents the outcomes of the systematic literature review and the findings related to the classification of the PMSs’ capacity to mitigate or eliminate the most common barriers that hinder their effectiveness. In the 39 publications included in this study, 217 mentions of PMSs were found, which resulted in the identification of 95 different PMSs. Of these, only 28 were considered for a detailed analysis because, as explained in the Methods section, it was decided to only consider PMSs that are mentioned in more than one publication. These PMSs are listed in Table 3, which also indicates the number of publications and the ones each PMS is referred to. Table 3 – PMS to be analysed. PMS Number of References References Balanced Scorecard 28 [10–37] Performance prism 15 [10–13,16,17,19,21,23–25,28–30,32] Performance pyramid (SMART) 14 [10,11,13,17,21,25–31,38,39] SCOR MODEL 12 [14,18,19,22,25,28,29,31,32,34,36,40] Activity based costing 7 [10,27,28,31,32,34,38] Results and determinants framework 7 [10–13,17,26,38] Integrated Performance and Measurement framework (IPMF) 6 [10–14,41] EFQM model 6 [10–13,25,26] Performance Measurement Questionnaire (PMQ) 5 [10,11,15,27,32] Economic Value-Added Model (EVA) 4 [10–12,34] Performance Measurement Matrix 4 [11,17,21,26] Integrated Performance Measurement System (IPMS) 3 [10–12] Performance Planning Value Chain (PPVC) 3 [10–12] ECOGRAI 3 [15,31,32] Theory of constraints (TOC) measurement system 3 [25,33,34] Action-Profit Linkage Model (APL) 3 [10–12] Dynamic Performance Measurement System (DPMS) 3 [10–12] Quantitative model for performance measurement system (QMPMS) 3 [12,27,42] Analytic Hierarchical Performance Model 2 [43,44] Dynamic multidimensional performance framework 2 [11,12] Flexible strategy game-card 2 [11,12] AMBITE 2 [13,38] Kanji’s business scorecard 2 [11,12] Cambridge Performance Measurement Framework (CPMF) 2 [10,32] Brown’s framework 2 [13,17] DuPont model 2 [11,33] Sustainability performance measurement system 2 [11,12] Holistic performance management framework 2 [11,12] The in-depth analysis of the 28 PMSs allowed for each of them to be classified using the values indicated in Table 2. That classification can be observed in Table 4. 56 Table 4 – Classification of PMSs according to their capacity to mitigate or eliminate each barrier. PMS 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Balanced Scorecard ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Performance prism ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Performance pyramid (SMART) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Supply Chain Operations Reference (SCOR) Model ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Activity based costing (ABC) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Results and determinants framework ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Integrated Performance and Measurement framework (IPMF) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● European Foundation for Quality Management (EFQM) Model ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Performance Measurement Questionnaire (PMQ) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Economic Value-Added Model (EVA) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Performance Measurement Matrix ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Integrated Performance Measurement System (IPMS) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Performance Planning Value Chain (PPVC) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ECOGRAI ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Theory of constraints (TOC) measurement system ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Action-Profit Linkage Model (APL) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Dynamic Performance Measurement System (DPMS) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Quantitative model for performance measurement system (QMPMS) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Analytic Hierarchical Performance Model ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Dynamic multidimensional performance framework ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Flexible strategy game-card ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● AMBITE ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Kanji’s business scorecard ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Cambridge Performance Measurement Framework (CPMF) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Brown’s framework ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● DuPont model ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Sustainability performance measurement system ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Holistic performance management framework ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● After the classification process was completed (Figure 3), it became evident that only four out of the 28 PMSs assessed had achieved a score exceeding half of the highest possible classification (38 points). These distinguished PMSs are (i) ECOGRAI, (ii) Performance Pyramid, (iii) Integrated Performance and Measurement System (IPMS), and (iv) European Foundation for Quality Management (EFQM) model with scores of 66%, 61%, 58%, and 53%, respectively. On the other hand, seven PMSs failed to attain even one-third of the highest possible classification, as can be observed in Figure 3. 57 Figure 3 – Total score of each PMS. For the PMSs that were classified as the most complete regarding their ability to mitigate or eliminate barriers to their effectiveness, the following was found: x ECOGRAI is capable of mitigating or eliminating eight barriers and has some capacity to mitigate or eliminate another nine. It is classified as not being able to mitigate or eliminate only two barriers. x Performance Pyramid is capable of mitigating or eliminating seven barriers and has some capacity to mitigate or eliminate another nine. It is classified as not being able to mitigate or eliminate three barriers. x IPMS is capable of mitigating or eliminating five barriers and has some capacity to mitigate or eliminate another twelve. It is classified as not being able to mitigate or eliminate only two barriers. x EFQM model is capable of mitigating or eliminating two barriers and has some capacity to mitigate or eliminate another sixteen. It is classified as not being able to mitigate or eliminate only one barrier. This classification, for each of the 29 PMSs, can be observed in Figure 4. 58 Figure 4 – Number of classifications of each type per PMS. The barriers (Table 1) with the highest scores in terms of the capacity of the PMSs to address them were the lack of connection to strategy, high complexity, and excess of indicators, registering percentages of 75%, 70%, and 68%, respectively. On the other hand, the barriers with the lowest scores were poor communication system, blame culture, lack of trained resources, and lack of rewards, as can be observed in Figure 5. Figure 5 – Total score of each barrier. 59 Regarding the barriers on which more PMSs were classified as having the capacity to mitigate or eliminate, the following was found: x Lack of connection to strategy: A total of 16 PMSs have capacity to eliminate or mitigate this barrier, and 10 have some capacity. Only two PMSs have weak or no capacity. x High complexity: A total of 14 PMSs have capacity to eliminate or mitigate this barrier, and 11 have some capacity. Three PMSs have weak or no capacity. x Excess of indicators: A total of 11 PMSs have capacity to eliminate or mitigate this barrier, and 16 have some capacity. Only one PMS has weak or no capacity. The number of PMSs classified according to their capacity to mitigate or eliminate each of the 19 barriers can be observed in Figure 6. Figure 6 – Number of systems of each category of mitigation/elimination capacity per barrier. The next section explores a discussion of the most robust PMSs for mitigating and eliminating each specific barrier along with the tools and mechanisms that empower them to excel in this regard. 4.5 Discussion This section presents, for each barrier, the PMSs and why they have been classified as having capacity to mitigate or eliminate that barrier. At the end, there is a discussion about the relationship between PMSs and sustainable development, starting with the connection between some barriers to the effectiveness of PMSs and the difficulties in achieving sustainable development. 66 19. Difficulties in collecting, analysing, and presenting data In the context of this specific barrier, none of the assessed PMSs achieved a strong classification. However, 26 PMSs demonstrated some capacity to address it, as the existence of this barrier is closely intertwined with the complexity of the system. Conversely, two PMSs were classified as weak. The first is due to its omission of any mention regarding data collection, analysis, or presentation; and the second is because it dealt with substantial volumes of data (as observed in the case of PPVC). These two characteristics rendered the mitigation or elimination of this barrier difficult. Generally, the majority of PMSs, unless exceptionally complex, should be equipped to manage data effectively through readily available IT tools in contemporary organisations, such as spreadsheets and dashboard software. 4.2. Relationship between PMSs and Sustainable Development In general terms, it can be speculated that some barriers to the effectiveness of PMSs are related to difficulties in the process of transitioning to sustainable practices. For example, the barrier lack of use for improvement is related to the difficulty in developing better production processes (an aspect directly mentioned in SDG8 and SDG9, see next) and is, therefore, also an obstacle to sustainable development. In fact, better PMSs contribute to more efficient monitoring of companies’ performance, which can cover various aspects (e.g., economic, environmental, operational, and social), thus allowing for the development of pertinent and focused improvement actions. These actions may address, for example, the rational use of resources, which is directly related to SDG9 and SDG12 [5], more specifically to target 9.4 (“… increased resource-use efficiency…”) and target 12.2 (“… efficient use of natural resources…”), the efficiency of production processes (SDG8, target 8.2 “…technological upgrading and innovation…” and SDG9, target 9.4 “… upgrade infrastructure and retrofit industries to make them sustainable…”), including waste production (SDG12, target 12.5 “… substantially reduce waste generation…”), and employee satisfaction/motivation (SDG8 “…full and productive employment and decent work for all…”), thus contributing to increased productivity and profit. Therefore, the authors argue that PMSs can be designed to be aligned with the principles of sustainable development referred to in the introduction (development, needs, and future generations) and, from this perspective, are instrumental for companies’ sustainability. 67 4.6 Conclusion The analysis showed how PMSs relate to overcoming the studied barriers. While no perfect solution emerged, many share traits vital for tackling common barriers. PMS traits can effectively address multiple barriers due to their interconnectedness. First and foremost, a PMS must prioritise simplicity and clarity to avoid unclear systems and high complexity. Without these traits, other barriers, e.g., difficulties in collecting, analysing, and presenting data, alongside inappropriate IT tools and time and resources required can be exacerbated. Simplicity and clarity can be attained through a schematic visual representation of the PMS coupled with a step-bystep implementation guide. Additionally, a successful PMS should foster a culture of continuous improvement, which is crucial for addressing barriers like blame culture, false expectations, lack of employee involvement, and lack of use for improvement. Establishing a direct link between actions and the key performance indicators (KPIs) that they positively impact is essential to sustain this culture effectively. The key to a strong PMS is its alignment with a well-defined organisational strategy established by top management, guiding the derivation of objectives and selection of indicators, targets, and actions. This helps mitigate issues like lack of connection to the strategy, lack of top management involvement, and inappropriate indicators. The objectives derived from the strategy should cascade throughout the organisation, being customised to each hierarchical level. This helps mitigate issues related to a blame culture, lack of employee and top management involvement, and poor communication systems. Subsequently, indicators must be carefully derived from objectives and tailored to each organisational level to ensure a balanced and relevant set of metrics. This step is pivotal in addressing an excess of indicators and a lack of balance of indicators. Thorough documentation of the PMS is crucial, ensuring information accessibility to all and facilitating human resource training. This contributes to mitigating issues such as lack of understanding of indicators and lack of trained resources. Part of this documentation should delve into the factors that can influence indicators positively or negatively, aiding in goal definition and achieving a balanced indicator framework. While certain barriers are addressed by many PMSs, others, such as blame culture, lack of trained resources, lack of rewards, and poor communication systems, remain largely unaddressed. Moreover, issues like inadequate IT tools, false expectations, lack of use for improvement, issues on target definition, and lack of indicator understanding are only lightly addressed or addressed by a limited number of PMSs. Notably, none of the analysed PMSs demonstrated a comprehensive ability to address all these barriers. 68 Thus, in conclusion, future research is needed to develop or enhance PMS frameworks that can effectively mitigate or eliminate all barriers. This study lays the groundwork for such research by identifying the existing PMSs and their tools to address each barrier as well as highlighting areas where new tools or methodologies are needed. As shown in Section 4.2, better PMSs contribute to organisations’ sustainable development. As for limitations, this study analysed 28 PMSs (systematic literature review in Scopus and Web of Science databases), focusing solely on the methodologies and tools within these selected systems. We acknowledge the possibility of additional PMSs in the literature that were not included in this study with the capacity to mitigate or eliminate common barriers. 4.7 References 1. Dinis-Carvalho, J.; Sousa, R.M.; Moniz, I.; Macedo, H.; Lima, R.M. Improving the Performance of a SME in the Cutlery Sector Using Lean Thinking and Digital Transformation. Sustainability 2023, 15, 8302. https://doi.org/10.3390/su15108302. 2. Klarin, T. The Concept of Sustainable Development: From its Beginning to the Contemporary Issues. Zagreb Int. Rev. Econ. Bus. 2018, 21, 67–94. https://doi.org/10.2478/zireb-2018-0005. 3. Manioudis, M.; Meramveliotakis, G. Broad strokes towards a grand theory in the analysis of sustainable development: A return to the classical political economy. New Polit. Econ. 2022, 27, 866–878. https://doi.org/10.1080/13563467.2022.2038114. 4. United Nations. Sustainable Development Goals—Teaching Guide and Resources; United Nations: New York, NY, USA, 2018. 5. United Nations. Transforming Our World: The 2030 Agenda for Sustainable Development; United Nations: New York, NY, USA, 2016; p. 36. 6. Papulová, Z.; Gažová, A.; Šlenker, M.; Papula, J. Performance measurement system: Implementation process in smes. Sustainability 2021, 13, 4794. https://doi.org/10.3390/su13094794. 7. Fantozzi, I.C.; Di Luozzo, S.; Schiraldi, M.M. Industrial Performance Measurement Systems Coherence: A Comparative Analysis of Current Methodologies, Validation and Introduction to Key Activity Indicators. Appl. Sci. 2023, 13, 235. https://doi.org/10.3390/app13010235. 8. Bititci, U.S. Managing Business Performance; Wiley: Hoboken, NJ, USA, 2015. 9. Cunha, F.; Dinis-Carvalho, J.; Sousa, R.M. Performance Measurement Systems in Continuous Improvement Environments: Obstacles to Their Effectiveness. Sustainability 2023, 15, 867. https://doi.org/10.3390/su15010867. 10. Taticchi, P.; Balachandran, K.; Tonelli, F. Performance measurement and management systems: State of the art, guidelines for design and challenges. Meas. Bus. Excell. 2012, 16, 41–54. https://doi.org/10.1108/13683041211230311. 69 11. Yadav, N.; Sushil; Sagar, M. Revisiting performance measurement and management: Deriving linkages with strategic management theories. Int. J. Bus. Perform. Manag. 2014, 15, 87–105. https://doi.org/10.1504/IJBPM.2014.060146. 12. Yadav, N.; Sushil; Sagar, M. Performance measurement and management frameworks: Research trends of the last two decades. Bus. Process Manag. J. 2013, 19, 947–971. https://doi.org/10.1108/BPMJ-01-2013-0003. 13. Folan, P.; Browne, J. A review of performance measurement: Towards performance management. Comput. Ind. 2005, 56, 663–680. https://doi.org/10.1016/j.compind.2005.03.001. 14. Chorfi, Z.; Benabbou, L.; Berrado, A. An integrated performance measurement framework for enhancing public health care supply chains. Supply Chain Forum 2018, 19, 191–203. https://doi.org/10.1080/16258312.2018.1465796. 15. Moreira, M.; Tjahjono, B. Applying performance measures to support decision-making in supply chain operations: A case of beverage industry. Int. J. Prod. Res. 2016, 54, 2345–2365. https://doi.org/10.1080/00207543.2015.1076944. 16. Franco-Santos, M.; Lucianetti, L.; Bourne, M. Contemporary performance measurement systems: A review of their consequences and a framework for research. Manag. Account. Res. 2012, 23, 79–119. https://doi.org/10.1016/j.mar.2012.04.001. 17. Atkinson, M. Developing and using a performance management framework: A case study. Meas. Bus. Excell. 2012, 16, 47–56. https://doi.org/10.1108/13683041211257402. 18. Sousa, S.; Aspinwall, E. Development of a performance measurement framework for SMEs. Total Qual. Manag. Bus. Excell. 2010, 21, 475–501. https://doi.org/10.1080/14783363.2010.481510. 19. Ferreira, P.S.; Shamsuzzoha, A.H.M.; Toscano, C.; Cunha, P. Framework for performance measurement and management in a collaborative business environment. Int. J. Product. Perform. Manag. 2012, 61, 672–690. https://doi.org/10.1108/17410401211249210. 20. Gambelli, D.; Solfanelli, F.; Orsini, S.; Zanoli, R. Measuring the economic performance of small ruminant farms using balanced scorecard and importance‐performance analysis: A european case study. Sustainability 2021, 13, 3321. https://doi.org/10.3390/su13063321. 21. Zanon, L.G.; Ulhoa, T.F.; Esposto, K.F. Performance measurement and lean maturity: Congruence for improvement. Prod. Plan. Control 2021, 32, 760–774. https://doi.org/10.1080/09537287.2020.1762136. 22. Frederico, G.F.; Garza-Reyes, J.A.; Kumar, A.; Kumar, V. Performance measurement for supply chains in the Industry 4.0 era: A balanced scorecard approach. Int. J. Product. Perform. Manag. 2021, 70, 789–807. https://doi.org/10.1108/IJPPM-08-2019-0400. 23. Pei, Y.L.; Amekudzi, A.A.; Meyer, M.D.; Barrella, E.M.; Ross, C.L. Performance measurement frameworks and development of effective sustainable transport strategies and indicators. Transp. Res. Rec. 2010, 2163, 73–80. https://doi.org/10.3141/2163-08. 24. Micheli, P.; Kennerley, M. Performance measurement frameworks in public and non-profit sectors. Prod. Plan. Control 2005, 16, 125–134. https://doi.org/10.1080/09537280512331333039. 25. Öz, H.H.; Özyörük, B. Performance measurement in-fourth party reverse logistics. Meas. Bus. Excell. 2021, 27, 549– 578. https://doi.org/10.1108/MBE-05-2019-0048. 26. Gaiardelli, P.; Saccani, N.; Songini, L. Performance measurement systems in after-sales service: An integrated framework. Int. J. Bus. Perform. Manag. 2007, 9, 145–171. https://doi.org/10.1504/IJBPM.2007.011860. 70 27. Suwignjo, P.; Bititci, U.S.; Carrie, A.S. Quantitative Models for Performance Measurement System. Int. J. Prod. Econ. 2000, 64, 231–241. https://doi.org/10.1016/S0925-5273(99)00061-4. 28. Saleheen, F.; Habib, M.M.; Hanafi, Z. Supply chain performance measurement model: A literature review. Int. J. Supply Chain Manag. 2018, 7, 70–78. 29. Bai, C.; Sarkis, J. Supply-chain performance-measurement system management using neighbourhood rough sets. Int. J. Prod. Res. 2012, 50, 2484–2500. https://doi.org/10.1080/00207543.2011.581010. 30. Gimbert, X.; Bisbe, J.; Mendoza, X. The role of performance measurement systems in strategy formulation processes. Long Range Plann. 2010, 43, 477–497. https://doi.org/10.1016/j.lrp.2010.01.001. 31. Berrah, L.; Clivillé, V. Towards an aggregation performance measurement system model in a supply chain context. Comput. Ind. 2007, 58, 709–719. https://doi.org/10.1016/j.compind.2007.05.012. 32. Lauras, M.; Lamothe, J.; Pingaud, H. A business process oriented method to design supply chain performance measurement systems. Int. J. Bus. Perform. Manag. 2011, 12, 354–376. https://doi.org/10.1504/IJBPM.2011.042013. 33. Melnyk, S.A.; Stewart, D.M.; Swink, M. Metrics and performance measurement in operations management: Dealing with the metrics maze. J. Oper. Manag. 2004, 22, 209–218. https://doi.org/10.1016/j.jom.2004.01.004. 34. Agami, N.; Saleh, M.; Rasmy, M. A hybrid dynamic framework for supply chain performance improvement. IEEE Syst. J. 2012, 6, 469–478. https://doi.org/10.1109/JSYST.2011.2177109. 35. Garengo, P. A performance measurement system for SMEs taking part in Quality Award Programmes. Total Qual. Manag. Bus. Excell. 2009, 20, 91–105. https://doi.org/10.1080/14783360802614307. 36. Dweekat, A.J.; Hwang, G.; Park, J. A supply chain performance measurement approach using the internet of things: Toward more practical SCPMS. Ind. Manag. Data Syst. 2017, 117, 267–286. https://doi.org/10.1108/IMDS-032016-0096. 37. Bhagwat, R.; Sharma, M.K. An application of the integrated AHP-PGP model for performance measurement of supply chain management. Prod. Plan. Control 2009, 20, 678–690. https://doi.org/10.1080/09537280903069897. 38. Tan, W.A.; Shen, W.; Xu, L.; Zhou, B.; Li, L. A business process intelligence system for enterprise process performance management. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 2008, 38, 745–756. https://doi.org/10.1109/TSMCC.2008.2001571. 39. Laitinen, E.K.; Länsiluoto, A.; Rautiainen, I. Extracting appropriate scope for information systems: A case study. Ind. Manag. Data Syst. 2009, 109, 305–321. https://doi.org/10.1108/02635570910939353. 40. Taticchi, P.; Garengo, P.; Nudurupati, S.S.; Tonelli, F.; Pasqualino, R. A review of decision-support tools and performance measurement and sustainable supply chain management. Int. J. Prod. Res. 2015, 53, 6473–6494. https://doi.org/10.1080/00207543.2014.939239. 41. Amir-Heidari, P.; Maknoon, R.; Taheri, B.; Bazyari, M. A new framework for HSE performance measurement and monitoring. Saf. Sci. 2017, 100, 157–167. https://doi.org/10.1016/j.ssci.2016.11.001. 42. Bititci, U.S.; Suwignjo, P.; Carrie, A.S. Strategy management through quantitative modelling of performance measurement systems. Int. J. Prod. Econ. 2001, 69, 15–22. https://doi.org/10.1016/S0925-5273(99)00113-9. 43. Lee, H.; Han, I.; Kwak, W. Developing a business performance evaluation system: An analytic hierarchical model. Eng. Econ. 1995, 40, 343–357. https://doi.org/10.1080/00137919508903159. 71 44. Yurdakul, M. Measuring a manufacturing system’s performance using Saaty’s system with feedback approach. Integr. Manuf. Syst. 2002, 13, 25–34. https://doi.org/10.1108/09576060210411486. 45. ISO 22400; Automation systems and integration. International Organization for Standardization: Geneva, Switzerland, 2014. 72 5. CARACTERÍSTICAS-CHAVE DE SISTEMAS DE MEDIÇÃO DE DESEMPENHO EFICAZES Este capítulo apresenta o artigo “ Key Characteristics of Effective Performance Measurement Systems ” (Cunha et al., 2024b), que foi submetido para publicação na revista South African Journal of Industrial Engineering a 17/07/2024. Este artigo identifica as principais características para a eficácia de um PMS, através de uma revisão sistemática de literatura, explorando também as relações entre estas características e as barreiras à eficácia de um PMS. 5.1 Abstract Abstract: This paper aims to identify the main characteristics that Performance Measurement Systems (PMS) must have to be effective, linking them to the most common barriers to the effectiveness of these same PMS. Furthermore, it delves into the relationships between these characteristics. This study is one of the components of a broader doctoral research project that aims to develop a framework for creating effective PMS. The PRISMA methodology was used to identify relevant publications, that were then analysed, and from which were extracted effective PMS’s characteristics or traits. These were registered and grouped according to the similarity of meaning, resulting in the identification of 16 types of characteristics, which were represented visually through a diagram with the purpose of exploring the connections between them. Two types of connections were identified: “requires” and “enables”. These findings provide critical inputs for the development or improvement of PMS frameworks, specifically with information related to the characteristics that enable PMS effectiveness. In terms of limitations, this study is based on 27 publications resulting from the systematic literature review in two data bases (Scopus and Web of Science). 5.2 Introduction Performance measurement and monitoring stand as foundational elements for driving continuous improvement, effective management, and sustainability within an organization [1]. At the core of this endeavour lies the Performance Measurement System (PMS) which acts as a compass, guiding an organization’s journey towards continuous improvement and sustainability. It plays a crucial role in revealing the current status of an organization by providing a foundational starting point for improvement by identifying specific areas where enhancements are needed and should be undertaken [2]. An ineffective PMS poses substantial risks to an organization because it supports incorrect decisions that lead to an incorrect allocation of scarce resources to initiatives that will fail to deliver results [3]. Thus, 73 there arises a pressing need to identify and comprehend the characteristics underlying PMS effectiveness, enabling their incorporation into the development and implementation phases of such systems. This strategic integration of key characteristics maximizes the likelihood of PMS effectiveness. It is also imperative to understand the links between these characteristics and the most common barriers to PMS effectiveness. Understanding these characteristics and their connections to the barriers is crucial for developing an effective PMS model in future work. 5.3 Theoretical Framework The main functions of a PMS are to create organizational alignment and translate strategy into action [4]. The development and use of a PMS can be defined as the process of defining objectives, developing a set of performance metrics, and collecting, analysing, reporting, interpreting, reviewing and acting on performance data [5]. However, the extensive and intricate nature of that process makes the development and implementation of a PMS a significant challenge for organizations [4]. This study is part of a broader investigation addressing key challenges in a PMS. Initially, the main barriers to PMS effectiveness were identified [1] (Table 1). Table 1 – Main barriers to PMS effectiveness [1] 1. blame culture 11. false expectations 2. lack of connection to strategy 12. inappropriate IT tools 3. issues on target definition 13. excess of indicators 4. unclear system 14. lack of trained resources 5. lack of top management involvement 15. lack of employee involvement 6. poor communication system 16. inappropriate indicators 7. high complexity 17. lack of indicator understanding 8. lack of use for improvement 18. time and resources required 9. lack of balance of indicators 19. difficulties in collecting, analysing, and presenting data 10. Lack of rewards Next, perceptions of these barriers within an organization were explored [6]. Interviews and questionnaires were administered to 32 respondents, including 23 employees, 6 middle managers, and 3 top managers. The data collected indicated that the primary perceived barriers were: poor communication system, lack of trained resources, issues with target definition, lack of employee involvement, lack of indicators understanding and lack of use for improvement. The subsequent phase examined the ability of existing PMS to mitigate or eliminate the identified barriers [2]. A systematic literature review (SLR) was conducted to identify the most frequently mentioned PMS. These PMS were then quantitatively classified based on their ability to overcome or mitigate barriers. The 74 review identified the barriers most and least addressed by each PMS. Although no single PMS offers a comprehensive solution, certain common traits significantly contribute to overcoming prevalent barriers. Finally, this publication focus on identifying the essential characteristics of an effective PMS. 5.4 Methods A PMS must comprise some essential characteristics in order to be effective. To identify what are those characteristics a Systematic Literature Review (SLR) was performed. The SLR was performed following the steps of the PRISMA methodology, which is composed of four phases: identification, screening, eligibility and included publications [7]. In the identification phase a search for scientific articles, books and book chapters was performed on Scopus and Web of Science databases. The search was made with the following restrictions: “performance measurement” and “characteristics” as being present in the keywords, title or abstract; and publications with one or more citations. As a result, 2647 publications were identified in Scopus and 3905 were identified in Web of science. The steps followed during the application of the PRISMA methodology are described in Figure 1. After removing duplicates, 5762 publications remained. From the analysis of titles and keywords, 5611 publications were excluded. The remaining 151 publications were examined through an abstract analysis, resulting in the exclusion of 63 of them. The 88 remaining publications were analysed in their full extension and in 27 publications were identified characteristics an effective PMS should have. Figure 1 – Results of PRISMA methodology application. 75 The characteristics to PMS effectiveness described in the 27 publications were initially listed and subsequently grouped into distinct types based on their shared attributes. Recognizing the presence of several characteristics with varying names but converging meanings, the notion of 'types of characteristics' was introduced to streamline their classification under unified labels. The Results section delineates the outcomes of this classification process, elucidating the categorization of characteristics into their respective types. Following this, a comprehensive discussion ensues, exploring the implications and significance of these categorized attributes. This discussion sets the stage for the formulation of conclusions, which summarizes the insights derived from the study's findings. 5.5 Results The characteristics found in the literature were recorded in a table, with the corresponding reference where they were found, and classified into types of characteristics according with their similar meaning. This means that could be identified in the same literature reference several characteristics that would be classified under the same type of characteristic. This resulted in 16 different types of characteristics (Table 2) from the 221 mentions of characteristics in the literature. For the sake of simplicity, from here onwards the types of characteristics will be referred to simply as characteristics. 82 The lack of appropriate IT tools represents an obstacle to the implementation, use and maintenance of a PMS [42], [43], [54] because it can hinder the capacity of data management and can also impact the amount of resources necessary. The connections between these characteristics will be explored in the discussion section. 5.6 Discussion To systematically identify and map the primary connections among the identified characteristics, a visual diagram was developed (Figure 2). Within this diagrammatic representation, each circle embodies a distinct characteristic, while the interconnecting arrows signify the nature of their relationships. Two fundamental types of connections emerged: "requires” and "enables." Through graphic representation, the complexities of the relationships among characteristics are succinctly explored, offering a comprehensive understanding of their relationships or connections. Figure 2 – Interaction between the characteristics of a PMS The organizational culture should stimulate continuous improvement to enhance the effectiveness of the PMS and mitigate the detrimental impact of a blame culture. Without such a culture, the PMS can be underutilized, failing to fulfil its intended purpose of translating performance measurement into 83 performance improvement. When an organization lacks a culture that prioritizes continuous improvement, the PMS may become useless, merely serving as a tool for measurement without translating its insights into actionable improvements. Consequently, this disconnect can lead to disengagement from stakeholders, as they perceive the system as ineffective in driving meaningful change. The stakeholders of an organization form a diverse spectrum, including suppliers, customers, employees, top managers, owners, and authorities. However, when considering the effectiveness of a PMS, the pivotal focus narrows to two key groups: employees and top managers. These stakeholders are at the forefront of this discussion since their roles are essential in shaping the system's efficacy. Therefore, by actively stimulating a culture of continuous improvement, organizations enable a meaningful involvement of the stakeholders with the PMS. This involvement facilitates a collaborative approach to performance improvement, aligning organizational objectives with individual contributions and fostering a sense of ownership and accountability among stakeholders. The stakeholders’ involvement requires a correct deployment through the organization of objectives, indicators, and targets across all hierarchical levels and functions. To effectively engage stakeholders, it is imperative that they possess a clear understanding of and commitment to their strategic objectives. This includes awareness of how the attainment of these objectives is measured, how they are translated into measurable targets, and what specific actions are required to meet the defined performance goals. The correct deployment of the PMS through the organization requires a link to the strategy to ensure that it is only being measured what matters to the organization and that the objectives, measures, and targets deployed for each stakeholder are aligned with the organizations’ strategy, and therefore are relevant and will be an active part of the organization’s continual effort of performance improvement. This deployment requires an effective communication through the organization. It is essential that the objectives, measures and targets are understood and assimilated by everyone in all hierarchical levels. This can only be achieved through clear and widespread communication means. An effective communication is of pivotal importance as it acts as an enabler of the stakeholders involvement in the PMS, as stated above. When the communication is not effective it contributes to lack of awareness and commitment from employees. This can occur either due to a misunderstanding of communicated information or, in more severe cases, due to a complete absence of communication regarding the significance of the PMS. Effective communication requires data management, as it is imperative to promptly update individuals on the current status of performance indicators. Data management encompasses various stages, including data acquisition, collection, sorting, analysis, 84 interpretation, and dissemination. It is this comprehensive process that facilitates the calculation of performance indicators. Data management requires appropriate IT tools, resources and training, and documented procedures: x Appropriate IT tools: Data management relies on appropriate IT tools to maximize effectiveness. Automation facilitated by these tools reduces time and resource requirements, thereby minimizing the operational costs associated with running the PMS. On the other hand, inadequate IT tools are less capable of providing an automatization of the data management process, potentially increasing the resources need to be to keep the indicators updated. x Resources and training: The data management process requires dedicated resources and welltrained personnel to execute tasks such as data acquisition, collection, sorting, analysis, interpretation, and dissemination throughout the organization. Insufficient resources and training may result in delays in updating indicators, risking a gradual erosion of PMS effectiveness due to outdated data. x Documented procedures: The data management process requires documented procedures because they provide clear instructions on how to calculate indicators and specifying the type of data required and its collection methods. These procedures ensure consistency and clarity in data management, fostering accurate and reliable performance assessments. The failure to define targets with clarity can lead stakeholders to question the purpose or relevance of their efforts. Without a clear understanding of how their contributions align with organizational objectives, stakeholders may struggle to connect their actions to strategic goals. This lack of alignment can erode motivation and diminish the involvement of stakeholders in the effort to achieve set targets. Moreover, when targets seem unrealistic or beyond reach, individuals may become disengaged, believing their efforts will ultimately be futile. This can significantly dampen enthusiasm and hinder collaborative efforts towards organizational performance improvement. Therefore, ensuring that targets are appropriately defined and clearly communicated is essential for maintaining stakeholder motivation and involvement. Clear, attainable targets provide a sense of direction and purpose, empowering stakeholders to channel their efforts effectively towards shared goals. An appropriate target definition requires appropriate indicators and effective deployment through the organization. Firstly, appropriate indicators serve as the foundation for setting targets. Targets are derived directly from these indicators, making their relevance and appropriateness vital. If the chosen indicators lack relevance or appropriateness, any subsequent target definition will lack meaning and efficacy. Secondly, effective deployment through the organization ensures that objectives, indicators, and targets 85 permeate throughout the organization, fostering alignment and relevance at all levels. This dissemination process is crucial for ensuring that targets resonate with individuals and contribute to the organization's strategic objectives. Without such deployment, targets set for specific functions or hierarchical levels may risk misalignment with their respective objectives or fail to consider contextual nuances, undermining their effectiveness. Link performance to rewards is another crucial characteristic that enables stakeholders involvement. The existence of rewards/recognition for achieving performance targets is an impeller of the employees motivation and involvement. It is essential that this rewards system extends across all hierarchical levels and is directly tied to actual performance outcomes derived from relevant objectives, indicators, and targets specific to each level. Furthermore, this rewards system should be transparent, consistent, and maintained throughout the evaluation period. Link performance to rewards requires an appropriate target definition, because performance will measured against a previously defined target. A well-defined target is essential to ensure fairness, when targets are unattainable or irrelevant, it can breed feelings of injustice among employees who struggle to achieve them. Moreover, targets must be aligned with both individual and organizational objectives to avoid misallocation of resources towards goals that do not contribute to overall performance improvement. In essence, a clear and appropriate target definition serves as the foundation for linking performance to rewards, promoting fairness, alignment, and efficient resource utilization within the organization. A clear system is another characteristic that enables stakeholders involvement. When the PMS is clearly defined with explicit objectives, roles, and responsibilities, it becomes more accessible and understandable to all stakeholders. This clarity not only enhances comprehension but also encourages their involvement with the PMS. A clear system provides stakeholders with a clear roadmap, delineating their contributions and highlighting how they fit into the broader organizational objectives. By understanding their roles and responsibilities within the PMS, stakeholders feel a greater sense of ownership and accountability, which motivates them to actively participate in its implementation and execution. When stakeholders have a clear understanding of the purpose and processes of the PMS, they are more likely to trust its outcomes and contribute meaningfully towards its success. This fosters a culture of collaboration and collective responsibility, where stakeholders work together towards shared goals, driving organizational performance and success. A clear system also acts as an enabler of an effective communication. A clear and properly defined system is easier to communicate to people because it minimizes the vagueness of the system reducing the 86 chances of misalignments and misunderstandings. This clarity fosters a sense of trust and confidence among employees, promoting a culture of involvement and continuous improvement. Appropriate indicators enable the existence of a clear system. The appropriateness of indicators is essential to keep the system clear, properly defined indicators with recognized relevance through all the organization will contribute to make clear what is being measured, where it is being measured, when it is being measured, and why it is being measured. By ensuring that these fundamental questions can be clearly answered, the system becomes more transparent and understandable for everyone involved. A requirement for appropriate indicators is that they are linked to the organization’s strategy, are balanced (multidimensional). They need to be linked to the strategy because indicators should be derived from it. Strategic objectives serve as the foundation from which indicators are derived, enabling the measurement of progress toward these objectives. It is essential that these objectives and corresponding indicators are effectively deployed and adjusted across all functions and hierarchies of the organization to ensure alignment with the organization’s overall strategy. To be appropriate, indicators need to be balanced, they should be multidimensional allowing the measurement of different perspectives of the business. Without this balance, there is a risk of the organization overly focusing on just one or two aspects of its operations, resulting in an imbalance in overall performance. Additionally, indicators need to be balanced to prevent the occurrence of concurrent indicators, which occur when performance improvement in one indicator automatically affects negatively other indicator. This kind of behaviour between indicators is negative for the organization as it will foster negative competition between departments, functions and other hierarchies, that will cause an imbalance in the organization’s performance. On the other hand, appropriate indicators require an effective data management process because, according with ISO 22400:2014 [58], among other characteristics an indicators should be: x Quantitative: Their value must be expressed numerically to facilitate measurement and comparison. x Precise: The measured value of the indicator should be as close as possible to the reality; x In real time: Is calculated and accessible in real time, enabling timely decision-making; x Correct: The calculation of indicators should obey closely to the standard definition, minimizing deviations. x Automatized: Calculation and communication of the indicator should be as automatized as possible; 87 x Economic: The cost of measuring, calculating and reporting the indicator should be as low as possible. These attributes are achieved through the implementation of an effective data management process, which plays a pivotal role in ensuring the reliability and utility of performance indicators. Appropriate indicators are an enabler of a simple system, as their relevance directly influences the number required to drive organizational performance improvement. When indicators are appropriate, only a few are necessary to effectively monitor and improve organization’s performance. On the contrary, if the indicators are not appropriate, there may be a tendency to continuously add new ones in pursuit of those that drive improvement. This systematic addition of new indicators can lead to an excess of indicators that will increase the complexity of the system, making it more difficult and costly to maintain. A simple system is an enabler of an effective communication and of the stakeholders involvement. The simplicity of the system directly correlates with the ease of conveying information to employees and their comprehension of it. As the system becomes simpler, communication becomes more straightforward, enabling employees to grasp it with greater ease and facilitating their active involvement. A dynamic system is an enabler of the PMS being linked to the strategy. According with Bititci et.al. [59], to be dynamic a PMS should: x Be sensible to changes in both the internal and external environments of the organization; x Review and reprioritize the internal objectives when there are significant changes in the internal or external environments; x Implement changes to the internal objectives and priorities of the organization, assuring a constant alignment; x Assure that the performance improvements achieved are sustained over time. The dynamism of the system allows for regular review and updating in line with changes occurring in the organization's strategy, thereby ensuring ongoing alignment of the PMS with the strategic objectives. 5.7 Conclusion This study was centred on the research question “What are the main characteristics of effective Performance Measurement Systems?”. Additionally, the relationship between those characteristics is also explored and analysed, to get a better grasp of how those characteristics can interact with each other when they compose a system. The identification of those characteristics is part of an ongoing work where it aims to identify why PMS fail and provide a PMS model that maximizes the chances of effectiveness of 88 that system. Initially, where identified the main barriers to PMS effectiveness [1], then the perception of those barriers in an organization [60], then the ability of the existing PMS model to eliminate or mitigate those barriers [2]. The identification of the characteristics is the fourth step in this expanded study, with the final purpose of providing a PMS model that encompasses the knowledge collected previously. In the systematic literature review conducted, 221 references of characteristics to PMS effectiveness were identified. Those 221 references were grouped, according to their similar meaning, into 16 types of characteristics. The effectiveness of PMS hinges upon those 16 key characteristics and their interconnections. Through a systematic approach, those characteristics were identified, mapped and visually represented, revealing their intricate relationships. The development of a visual diagram highlighted two fundamental types of connections – “requires” and “enables”. Organizational culture emerges as a corner stone in stimulating continuous improvement and stakeholder involvement within the PMS. A culture that prioritizes continuous improvement serves as an enabler for stakeholder involvement, empowering individuals to actively participate in performance improvement. Furthermore, effective communication and deployment of the PMS throughout the organization are essential for ensuring alignment with strategic objectives and fostering stakeholder involvement. Appropriate indicators play a pivotal role in maintaining clarity and simplicity within the system. They must be linked to the organization's strategy and need to be balanced across multiple dimensions to accurately measure performance and drive improvement. Additionally, an effective data management process is crucial for ensuring the reliability and utility of performance indicators, enabling timely decisionmaking and informed action. Appropriate target definition paired with a linkage of rewards to performance also act as an enabler of the stakeholders involvement, by defining achievable and stimulating performance targets and rewarding the achievement of the proposed performance targets. Ultimately, a dynamic system that adapts to changes in the internal and external environment is essential for ensuring ongoing alignment with the organization's strategy. By continuously reviewing and reprioritizing objectives, implementing necessary changes, and sustaining performance improvements over time, the PMS remains responsive and relevant in driving organizational success. In essence, understanding these characteristics and the complex interconnections among them, provides critical information to design an effective PMS model, that not only measures performance but also drives continuous improvement, fosters stakeholder engagement, and ultimately, enables organizational success. 89 The primary limitations of this study reside in the size and composition of the sample utilized for the systematic literature review. The study was restricted to 27 publications, from which 221 references to characteristics were extracted. Additionally, the selection criteria did not encompass publications with zero citations. Consequently, potentially valuable contributions to the subject matter may have been overlooked, particularly newer publications that had not yet got citations at the time of the systematic literature review. 5.8 References [1] F. Cunha, J. Dinis-Carvalho, and R. M. Sousa, “Performance Measurement Systems in Continuous Improvement Environments: Obstacles to Their Effectiveness,” Sustain., vol. 15, no. 1, p. 867, 2023, doi: 10.3390/su15010867. [2] F. Cunha, J. Dinis-Carvalho, and R. M. Sousa, “Assessment of Performance Measurement Systems’ Ability to Mitigate or Eliminate Typical Barriers Compromising Organisational Sustainability,” Sustain., vol. 16, no. 5, p. 2173, 2024, doi: 10.3390/su16052173. [3] J. Van Camp and J. Braet, “Taxonomizing performance measurement systems’ failures,” Int. J. Product. Perform. Manag., vol. 65, no. 5, pp. 672–693, 2016, doi: 10.1108/IJPPM-03-2015-0054. [4] I. C. Fantozzi, S. Di Luozzo, and M. M. Schiraldi, “Industrial Performance Measurement Systems Coherence: A Comparative Analysis of Current Methodologies, Validation and Introduction to Key Activity Indicators,” Appl. Sci., vol. 13, no. 1, 2023, doi: 10.3390/app13010235. [5] U. Bititci, Managing business performance: The science and the art. John Wiley & Sons, 2015. [6] F. Cunha, J. Dinis-Carvalho, and R. M. Sousa, “Analysis of barriers for performance measurement system effectiveness in a company: perceptions across hierarchical levels,” Int. J. Lean Six Sigma, vol. Ahead of p, no. Ahead of print, 2024, doi: 10.1108/IJLSS-12-2023-0205. [7] D. Moher, A. Liberati, J. Tetzlaff, D. G. Altman, and T. P. Group, “Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement,” PLoS Med, vol. 6, no. 7, 2009, doi: 10.1371/journal.pmed.1000097. [8] P. Cocca and M. Alberti, “A framework to assess performance measurement systems in SMEs,” Int. J. Product. Perform. Manag., vol. 59, no. 2, pp. 186–200, 2010, doi: 10.1108/17410401011014258. [9] D. Naslund and A. Norrman, “A performance measurement system for change initiatives: An action research study from design to evaluation,” Bus. Process Manag. J., vol. 25, no. 7, pp. 1647–1672, 2019, doi: 10.1108/BPMJ-112017-0309. [10] M. Hudson Smith and D. Smith, “Implementing strategically aligned performance measurement in small firms,” Int. J. Prod. Econ., vol. 106, no. 2, pp. 393–408, 2007, doi: 10.1016/j.ijpe.2006.07.011. [11] J. Lehtinen and T. Ahola, “Is performance measurement suitable for an extended enterprise?,” Int. J. Oper. Prod. Manag., vol. 30, no. 2, pp. 181–204, 2010, doi: 10.1108/01443571011018707. [12] M. Stříteská, D. Zapletal, and L. Jelínková, “Performance management systems in Czech companies: Findings from a questionnaire survey,” E a M Ekon. a Manag., vol. 19, no. 4, pp. 44–55, 2016, doi: 10.15240/tul/001/2016-4-004. 90 [13] M. Bahri, J. St-Pierre, and O. Sakka, “Performance measurement and management for manufacturing SMEs: a financial statement-based system,” Meas. Bus. Excell., vol. 21, no. 1, pp. 17–36, 2017, doi: 10.1108/MBE-06-20150034. [14] A. Neely, J. Mills, K. Platts, H. Richards, M. Gregory, and M. Kennerley, “Performance measurement system design : developing and testing a process-based approach,” Int. J. Oper. Prod. Manag., vol. 20, no. 10, pp. 1119–1145, 2000. [15] A. De Toni and S. Tonchia, “Performance measurement systems Models, characteristics and measures,” Int. J. Oper. Prod. Manag., vol. 21, no. 1–2, pp. 46–70, 2001, doi: 10.1108/01443570110358459. [16] P. Micheli and J. F. Manzoni, “Strategic performance measurement: Benefits, limitations and paradoxes,” Long Range Plann., vol. 43, no. 4, pp. 465–476, 2010, doi: 10.1016/j.lrp.2009.12.004. [17] M. Striteska and O. Svoboda, “Survey of performance measurement systems in Czech companies,” E a M Ekon. a Manag., vol. 15, no. 2, 2012. [18] A. M. Ghalayini and J. S. Noble, “The changing basis of performance measurement,” Int. J. Oper. Prod. Manag., vol. 16, no. 8, pp. 63–80, 1996, doi: 10.1108/01443579610125787. [19] K. Baird, “The effectiveness of strategic performance measurement systems,” Int. J. Product. Perform. Manag., vol. 66, no. 1, pp. 3–21, 2017, doi: 10.1108/IJPPM-06-2014-0086. [20] M. Behery, F. Jabeen, and M. Parakandi, “Adopting a contemporary performance management system: A fast-growth small-to-medium enterprise (FGSME) in the UAE,” Int. J. Product. Perform. Manag., vol. 63, no. 1, pp. 22–43, 2014, doi: 10.1108/IJPPM-07-2012-0076. [21] M. Franco-Santos et al., “Towards a definition of a business performance measurement system,” Int. J. Oper. Prod. Manag., vol. 27, no. 8, pp. 784–801, 2007, doi: 10.1108/01443570710763778. [22] J. Ukko and M. Saunila, “Understanding the practice of performance measurement in industrial collaboration: From design to implementation,” J. Purch. Supply Manag., vol. 26, no. 1, 2020, doi: 10.1016/j.pursup.2019.02.001. [23] E. R. G. Pedersen and F. Sudzina, “Which firms use measures?: Internal and external factors shaping the adoption of performance measurement systems in Danish firms,” Int. J. Oper. Prod. Manag., vol. 32, no. 1, pp. 4–27, 2012, doi: 10.1108/01443571211195718. [24] R. Munir, K. Baird, and Z. Si, “Associations with the use of multidimensional performance measures and the effectiveness of performance measurement systems,” Asia-Pacific Manag. Account. J., vol. 7, no. 2, 2012. [25] M. Franco-Santos, L. Lucianetti, and M. Bourne, “Contemporary performance measurement systems: A review of their consequences and a framework for research,” Manag. Account. Res., vol. 23, no. 2, pp. 79–119, 2012, doi: 10.1016/j.mar.2012.04.001. [26] P. Mettänen, “Design and implementation of a performance measurement system for a research organization,” Prod. Plan. Control, vol. 16, no. 2, pp. 178–188, 2005, doi: 10.1080/09537280512331333075. [27] G. Yang, A. Macnab, L. Yang, and C. Fan, “Developing performance measures and setting their targets for national research institutes based on strategy maps,” J. Sci. Technol. Policy Manag., vol. 6, no. 2, pp. 165–186, 2015, doi: 10.1108/JSTPM-12-2014-0042. [28] K. Lee and S. Lee, “Enhancing R&D Performance Management: A Case of R&D Projects in South Korea,” Sustain., vol. 15, no. 15, pp. 1–14, 2023, doi: 10.3390/su151511752. 91 [29] P. Jonsson and M. Lesshammar, “Evaluation and improvement of manufacturing performance measurement systems - the role of OEE Patrik Jonsson Magnus Lesshammar Article,” Int. J. Oper. Prod. Manag., vol. 19, no. 1, pp. 55–78, 1999. [30] K. K. Choong, “Has this large number of performance measurement publications contributed to its better understanding? A systematic review for research and applications,” Int. J. Prod. Res., vol. 52, no. 14, pp. 4174–4197, 2014, doi: 10.1080/00207543.2013.866285. [31] A. Tung, K. Baird, and H. P. Schoch, “Factors influencing the effectiveness of performance measurement systems,” Int. J. Oper. Prod. Manag., vol. 31, no. 12, pp. 1287–1310, 2011, doi: 10.1108/01443571111187457. [32] P. Pekkanen and P. Niemi, “Process performance improvement in justice organizations - Pitfalls of performance measurement,” Int. J. Prod. Econ., vol. 143, no. 2, pp. 605–611, 2013, doi: 10.1016/j.ijpe.2012.08.009. [33] J. Couturier and N. Sklavounos, “Performance dialogue: A framework to enhance the effectiveness of performance measurement systems,” Int. J. Product. Perform. Manag., vol. 68, no. 4, pp. 699–720, 2019, doi: 10.1108/IJPPM09-2017-0238. [34] M. Wouters and C. Wilderom, “Developing performance-measurement systems as enabling formalization: A longitudinal field study of a logistics department,” Accounting, Organ. Soc., vol. 33, no. 4–5, pp. 488–516, 2008, doi: 10.1016/j.aos.2007.05.002. [35] Șerban Radu-Alexandru and H. Mihaela, “Performance Management Systems – Proposing and Testing a Conceptual Model,” Stud. Bus. Econ., vol. 14, no. 1, pp. 231–244, 2019, doi: 10.2478/sbe-2019-0018. [36] S. S. Nudurupati, U. S. Bititci, V. Kumar, and F. T. S. Chan, “State of the art literature review on performance measurement,” Comput. Ind. Eng., vol. 60, no. 2, pp. 279–290, 2011, doi: 10.1016/j.cie.2010.11.010. [37] M. Franco and M. Bourne, “Factors that play a role in ‘managing through measures,’” Manag. Decis., vol. 41, no. 8, pp. 698–710, 2003, doi: 10.1108/00251740310496215. [38] F. Franceschini, M. Galetto, and D. Maisano, Management by Measurement: Designing key indicators and performance measurement systems. SpingerScience & Business Media, 2007. [39] A. Meekings, “Unlocking the potential of performance measurement: A practical implementation guide,” Public money Manag., vol. 15, no. 4, pp. 5–12, 1995, doi: 10.1080/09540969509387888. [40] A. M. Schneiderman, “Why balanced scorecards fail,” J. Strateg. Perform. Meas., vol. 2, no. 11, pp. 6–11, 1999. [41] A. A. de Waal and H. Counet, “Lessons learned from performance management systems implementations,” Int. J. Product. Perform. Manag., vol. 58, no. 4, pp. 367–390, 2009, doi: 10.1108/17410400910951026. [42] S. Okwir, S. S. Nudurupati, M. Ginieis, and J. Angelis, “Performance measurement and management systems: a perspective from complexity theory,” Int. J. Manag. Rev., vol. 20, no. 3, pp. 731–754, 2018, doi: 10.1111/ijmr.12184. [43] C. Lohman, L. Fortuin, and M. Wouters, “Designing a performance measurement system: A case study,” Eur. J. Oper. Res., vol. 156, no. 2, pp. 267–286, 2004, doi: 10.1016/S0377-2217(02)00918-9. [44] S. D. Sousa, E. M. Aspinwall, and A. Guimarães Rodrigues, “Performance measures in English small and medium enterprises: survey results,” Benchmarking An Int. J., vol. 13, no. 1/2, pp. 120–134, Jan. 2006, doi: 10.1108/14635770610644628. [45] M. Kennerley and A. Neely, “A framework of the factors affecting the evolution of performance measurement systems,” Int. J. Oper. Prod. Manag., vol. 22, no. 11, pp. 1222–1245, 2002, doi: 10.1108/01443570210450293. 98 10. Documented procedures 11. Appropriate target definition 12. Simple system 13. Data management 14. Resources and training 15. Clear system 16. Appropriate IT tools The objective of this study is to develop an effective PMS framework derived from the insights garnered in the preceding four studies outlined briefly above. The paper is structured into the following sections: Experimental, Results and Discussion, and Summary and Conclusion. The Methods section details the methodologies employed to develop the framework. In the Results and Discussion section, the framework is introduced and explained, engaging in an analysis of the strengths and weaknesses of the framework. Finally, the concluding section encapsulates the findings of the study and offers directions for future research in this domain. 6.4 Experimental To develop an effective PMS model, the methodology outlined in Fig. 4 was followed. Several steps within this methodology led to the publication of articles, which collectively contributed to achieving the final step of this methodology and culminated in this publication. 99 Fig. 4 – Methodology adopted (Authors’ own work) (Symbols legend in Appendix A) Firstly, was investigated why PMS models fail by identifying the main barriers to their effectiveness. This was achieved through a systematic literature review of scientific articles from two databases: Scopus and Web of Science. The second step involved identifying perceptions of these barriers across different hierarchical levels within an organization. This was accomplished through interviews and questionnaires administered to individuals at all levels of the organization. The third step involved identifying existing PMS models in the literature and classifying them based on their ability to mitigate or eliminate each of the barriers to PMS effectiveness. This identification was conducted through a systematic literature review. To evaluate each PMS model's capacity to address the barriers, they were assessed and classified by using a three-level scale: 0 for weak capacity, 1 for some capacity, and 2 for strong capacity to mitigate or eliminate the barrier. Since no existing PMS model was found to effectively mitigate or eliminate all the identified barriers, the next step was to develop solutions for each barrier to PMS effectiveness. To achieve this, the key characteristics of effective PMS were identified through a systematic literature review. Then the relationships between these key characteristics were explored and mapped, as well as their connections to the barriers. 100 Next, these characteristics were grouped into categories based on their similarities. These groups formed the foundation for developing a new, effective PMS model. The resulting PMS model and methodology, developed through this methodology, are presented in detail in the Results and Discussion section. The PMS model is classified according with its ability to mitigate or eliminate the most common barriers to PMS effectiveness, using the same methodology used by Cunha et al. (2024a). The model is also evaluated by being compared to an implemented PMS in an organization. For this, a questionnaire was performed. In this questionnaire, individuals were asked to classify, using a Likert scale, their perception of their organization PMS effectiveness and the presence in the organization’ PMS of the key characteristics of the PMS model presented in this study (Appendix B). A five-point Likert scale was used (Joshi et al., 2015), ranging from 1 to 5 (1– Strongly disagree, 2– Disagree, 3– Indifferent, 4– Agree and 5– Strongly agree). For the data extracted from the questionnaire a descriptive statistical analysis (Boone & Boone, 2012) and a chi-square test (Pandis, 2016) were performed to ascertain whether the answers depend on the respondents degree of education, gender, shift, department or role in the company (employees, middle management and top management). The hypothesis tested with the chi-square test were: x H0. The variables are independent; there is no relationship between the categorical variables. x H1. The variables are dependent; there is a relationship between the categorical variables. To test these hypotheses, the expected frequencies for each variable were first calculated. Then, the difference between observed and expected frequencies is assessed using the chi-square test, from which the p-value was obtained. If the p-value is greater than 0.05, the null hypothesis (H0) is accepted, indicating a small difference between the observed and expected values. If the p-value is less than 0.05, the null hypothesis is rejected, indicating a significant difference between the observed and expected values (Pandis, 2016). 6.5 Results and discussion The 16 key characteristics of effective PMS, identified by Cunha et al, (2024b), were grouped into four dimensions: Culture, People, Data and Tools, and Indicators. These groupings were based on the meaning of each characteristic. Additionally, three characteristics were identified as common to all dimensions and were not assigned to any single dimension but were considered as overarching characteristics. 101 6.5.1 The proposed model The proposed effective PMS model, incorporating these dimensions and characteristics, is represented in Fig. 5. Fig. 5 – Effective PMS model (Authors’ own work) The characteristics in the model are not represented by their importance but rather by their hierarchy of precedence. According to the proposed model, an effective PMS should encompass four dimensions: Culture, People, Data and Tools, and Indicators. The first dimension to consider is Culture. The organization's culture forms the foundation of the PMS and should instill key characteristics into the system. It should stimulate continuous improvement, assure that the PMS is linked to the organization’s strategy, that there is an effective communication system linked to the PMS and that there is an effective deployment of the PMS through the organization. Next in the hierarchy is the People dimension. This dimension should assure the stakeholders involvement in the PMS (mainly top management and employees). It should also assure that there are appropriate trained resources to implement, use and maintain the PMS. It should also ensure that appropriately trained resources are available to implement, use, and maintain the PMS, and establish a link between performance and rewards (both financial and non-financial). The third dimension, Data and Tools, should ensure the presence of an appropriate data management process that allows an effective collection, analysis and presentation of data. It should provide appropriate IT tools to support this process and include documented procedures on how to collect, analyse, and present data, as well as other key aspects of the PMS. The final dimension is Indicators. This dimension should ensure the existence of appropriate, balanced (multidimensional) indicators and the proper definition of targets for these indicators. 102 Three characteristics are inherently associated with all the dimensions of the PMS presented above. Those characteristics are simple, clear and dynamic. All aspects of the PMS should be as simple and clear as possible. Additionally, the PMS should be dynamic, allowing for updates as needed to keep the system current. 6.5.2 Methodology for model implementation The operationalization of this model, or the methodology of implementation of a PMS according with the model presented above can be described according with Fig. 6. Fig. 6 – Methodology to design and implement a PMS (Authors’ own work) (Symbols legend in Appendix A) The first steps, or the pre requirements to design and implement a PMS according with the presented model is to have a culture that stimulates continuous improvement in the organization. It is also necessary for the organization to have a properly defined strategy. Next, top management should derive objectives from the established strategy. Based on these objectives, they need to define appropriate and balanced indicators to monitor performance effectively. It's important 103 to consider the data management required to calculate these indicators, the necessary IT tools, and the documentation of procedures related to both indicators and data management. Actions that can positively influence the indicators and contribute to achieving the objectives should be clearly defined. Targets associated with the indicators should be set for specific time periods. The attainment of these targets should be directly linked to the evaluation of the individuals responsible, thereby connecting performance to rewards. Based on the objectives defined by top management, a top-down deployment should occur across every department and hierarchical level. In this process, objectives, indicators, actions, and targets should be established in the same manner as they were for top management. This model was also classified according to the method used by Cunha et. Al (Cunha et al., 2024a), where the PMS is evaluated based on its ability to mitigate or eliminate each barrier. The classification scale is as follows: 0 indicates a weak capacity, 1 indicates some capacity, and 2 indicates a strong capacity. The total score is calculated by summing the values assigned to the PMS for its effectiveness in addressing each barrier. The detailed classification can be observed in Table 1. Table 1 – Classification of the PMS model according to his capacity to mitigate or eliminate each barrier (Authors’ own work). Barrier Classification 1. Blame culture ● 2. Lack of connection to strategy ● 3. Issues on target definition ● 4. Unclear system ● 5. Lack of top management involvement ● 6. Poor communication system ● 7. High complexity ● 8. Lack of use for improvement ● 9. Lack of balance of indicators ● 10. Lack of rewards ● 11. False expectations ● 12. Inappropriate IT tools ● 13. Excess of indicators ● 14. Lack of trained resources ● 15. Lack of employee involvement ● 16. Inappropriate indicators ● 17. Lack of indicator understanding ● 18. Time and resources required ● 19. Difficulties in collecting, analysing, and presenting data ● Total 35 (92%) 104 When comparing the classification obtained by this model with that obtained by the PMS classified by Cunha et al. (2024a), it is evident that this model achieves a significantly higher classification. This model achieves a score of 92%, whereas the highest score obtained by any PMS in Cunha et al. (2024a) study was 66%. 6.5.3 Model characteristics Each key characteristic represented in the model and methodology is explored below: 1. Stimulate continuous improvement To successfully design and implement a PMS, fostering a culture of continuous improvement is essential. This culture is a key enabler of stakeholder involvement and plays a crucial role in overcoming common barriers such as lack of use for improvement, blame culture, and false expectations. By emphasizing continuous improvement, organizations can encourage proactive actions aimed at genuine progress, rather than falling into the trap of believing that mere measurement leads to improvement. Focusing on improvement rather than assigning blame helps to avoid the development of a blame culture. A PMS that fails to drive performance improvement risks becoming irrelevant to stakeholders, ultimately leading to its failure (Kennerley & Neely, 2002; Meekings, 1995; Neely & Bourne, 2000; Schneiderman, 2013). Cultivating a culture of continuous improvement is fundamentally an organizational challenge that extends beyond the PMS itself. However, this culture can and should be integrated into the PMS. While the PMS should serve as a tool for accountability, it must never become a vehicle for blame. To align with this culture, improvement targets should be set at every organizational level, and specific actions that can positively impact performance indicators should be identified and implemented (Ghalayini & Noble, 1996; Jonsson & Lesshammar, 1999). This process can be effectively managed through the application of PDCA (Plan, Do, Check, Act) cycles (Carvalho, 2021). 2. Linked to the strategy The PMS must be closely aligned with the organization's strategy. This alignment is critical, as it directly impacts other key aspects of the PMS. For instance, indicators must be strategically linked to ensure their relevance and appropriateness, as they are intended to measure the performance of objectives derived from the organization's strategy. Effective deployment of the PMS throughout the organization hinges on maintaining this strategic connection. Top management should derive objectives directly from the strategy, and subsequent 105 hierarchical levels should then derive their objectives from those set by top management. If this strategic link is broken, the PMS may still be implemented, but it will likely focus on measuring and improving objectives that do not align with the organization's strategic goals. A dynamic PMS is essential to maintaining this strategic alignment over time. It must be adaptable, allowing for updates when the organization's strategy or focus changes, ensuring that the PMS remains a relevant and effective tool for driving performance in line with the organization's evolving priorities. 3. Effective communication Effective communication is crucial for ensuring stakeholder involvement and successfully deploying a PMS across an organization. A well-designed PMS, characterized by clarity and simplicity, facilitates efficient communication, which in turn depends on appropriate data management. Inadequate communication can lead to poor employee involvement and a lack of understanding of indicators. This issue is often exacerbated by the system's complexity and difficulties in data collection, analysis, and presentation. The more complex the PMS, the harder it becomes to maintain effective communication. Conversely, communication becomes more straightforward when reliable and clear data is available, enabling effective feedback to stakeholders. To achieve effective communication within the PMS, objectives and indicators should be clearly and consistently deployed throughout the organization, accompanied by regular updates on the status of these indicators. Employees need to be aware of their performance metrics and understand how their actions influence these indicators, either positively or negatively. Communication should be clear, simple, regular, and formal (Franco & Bourne, 2003). Strategies and objectives should be formally communicated to all relevant parties and can also be displayed within the organization through signs and posters. Performance indicators should be communicated as frequently as possible, utilizing methods such as team performance boards or digital displays to keep everyone informed. 4. Deployment through the organization Effective deployment of a PMS throughout the organization is crucial for securing stakeholder involvement and establishing appropriate targets. This deployment must be closely linked to the organization's strategy and supported by effective communication. Successful deployment is key to mitigating or eliminating the barrier of lack of employee involvement. This is achieved by systematically cascading objectives, indicators, targets, and actions across every level and function of the organization (Behery et al., 2014; Franco-Santos et al., 2007; Stříteská et al., 2016). 106 By following a top-down approach, you ensure that all levels remain aligned with the organization's strategic goals (Mettänen, 2005). In the proposed model and methodology, this process is represented by a cycle in which the definition of objectives, indicators, actions, and targets is repeated across all hierarchical levels. This cyclical approach ensures that the deployment is comprehensive, reaching every part of the organization and remaining firmly grounded in the organization's strategy. 5. Stakeholders involvement Stakeholders encompass a wide range of parties with an interest in the organization, but for the PMS, the primary focus is on top management and employees. Ensuring their involvement is critical to avoiding disengagement from both groups. Stakeholder involvement requires an effective deployment throughout the organization. A simple and clear PMS is essential for fostering this involvement, as increasing complexity and ambiguity can lead to a sense of disconnection from the system. Additionally, a culture of continuous improvement is a significant enabler, encouraging stakeholders to engage with the system by focusing on performance enhancement rather than assigning blame. Linking performance to rewards is another powerful motivator for stakeholder involvement. When stakeholders see a direct connection between their performance and tangible rewards, whether financial or otherwise, they are more likely to understand and actively influence their performance indicators to achieve the desired targets. In the proposed model and methodology, stakeholder involvement is integrated through the deployment cycle mentioned earlier, the connection between performance and rewards, and the emphasis on maintaining a culture of continuous improvement. These elements work together to ensure that stakeholders remain engaged and committed to the PMS. 6. Resources and training Allocating the necessary resources and providing adequate training are critical for the effective implementation and maintenance of a PMS. While this applies to all aspects of the PMS, it is particularly crucial for data management. To manage data effectively and develop appropriate indicators, it is essential to have personnel with the right training and expertise. This approach helps eliminate two major barriers to PMS effectiveness: the lack of trained resources and the time and resources required for proper implementation. 107 Implementing and sustaining a PMS demands significant financial and human resources (Bahri et al., 2017; Hudson Smith & Smith, 2007). It is essential that these resources are not only available but also well-trained in the specific skills needed to implement, use, and maintain the PMS effectively (Tung et al., 2011). The operationalization of resource allocation and training must be tailored to the unique needs and complexities of each organization. While the proposed model and methodology highlight the importance of providing resources and training, they do not prescribe a specific approach, recognizing that the best methods will vary depending on the organization's context. 7. Link performance to rewards A PMS should link performance to rewards (Cocca & Alberti, 2010; Franco-Santos et al., 2007; Lee & Lee, 2023; Lehtinen & Ahola, 2010; Stříteská et al., 2016; Tung et al., 2011) as a way of ensuring employee involvement and motivation to continuously improve and achieve the established targets. This connection between performance and rewards serves as a powerful motivator, encouraging employees to engage actively with the PMS and strive toward the organization's goals. For this approach to be effective, however, it requires the careful and appropriate definition of targets. The success of linking rewards to performance hinges on setting clear, achievable, and strategically aligned targets for each hierarchical level and function within the organization. The proposed model and methodology incorporate these principles, recommending that the linkage between performance and rewards be established only after performance targets have been defined for each level and function. This ensures that rewards are directly tied to meaningful and relevant performance outcomes, reinforcing the overall effectiveness of the PMS. 8. Data management Effective data management is essential for establishing accurate and relevant indicators within a PMS. As a critical support infrastructure for the PMS (Franco-Santos et al., 2012), data management encompasses the entire process of data acquisition, collection, sorting, analysis, interpretation, and dissemination (Franco-Santos et al., 2007). These steps are crucial for calculating performance indicators both accurately and in a timely manner. To ensure the success of this process, it is necessary to have documented procedures, appropriate IT tools, and well-trained resources. These elements work together to create a robust data management system that underpins the reliability of the performance indicators. 114 communication—prerequisites for top management setting clear objectives. Following this, appropriate and balanced performance indicators should be defined, supported by an efficient data management process that includes appropriate IT tools, resources and training, all documented in formal procedures. Once indicators are established, actions that positively influence these indicators should be identified, and performance targets should be set for specific time periods. These targets should be linked to rewards, acting as an enabler of stakeholders involvement. This process should be repeated across all hierarchical levels, starting with the definition of objectives. Additionally, the entire process should be revisited whenever there are significant changes in the internal or external environment, ensuring that the PMS remains updated and aligned with the organization's strategy. When evaluated using the methodology of Cunha et al. (2024a), the proposed model achieved a classification of 92%, significantly outperforming other PMS models previously assessed in the study by Cunha et al. (2024a). Furthermore, by applying the proposed model and methodology for an effective PMS to assess an existing system within an organization, using a questionnaire distributed to multiple employees, it was possible to identify critical gaps in the organization’s PMS. Several essential features of the proposed PMS model were missing in the organization’s current system, rendering it ineffective in the eyes of its own employees. In conclusion, the presented model and methodology offer valuable insights for managers and employees involved in designing, implementing, and maintaining an effective PMS. However, further real-world testing in organizations is necessary to fully validate the proposed solution. 6.7 References Bahri, M., St-Pierre, J., & Sakka, O. (2017). Performance measurement and management for manufacturing SMEs: a financial statement-based system. Measuring Business Excellence, 21(1), 17–36. https://doi.org/10.1108/MBE-06-20150034 Baird, K. (2017). The effectiveness of strategic performance measurement systems. International Journal of Productivity and Performance Management, 66(1), 3–21. https://doi.org/10.1108/IJPPM-06-2014-0086 Behery, M., Jabeen, F., & Parakandi, M. (2014). Adopting a contemporary performance management system: A fast-growth small-to-medium enterprise (FGSME) in the UAE. International Journal of Productivity and Performance Management, 63(1), 22–43. https://doi.org/10.1108/IJPPM-07-2012-0076 Bititci, U. (2015). Managing business performance: The science and the art. John Wiley & Sons. Boone, H. N., & Boone, D. A. (2012). Analyzing likert data. Journal of Extension, 50(2), 1–5. Carvalho, J. D. (2021). Melhoria Contínua nas Organizações. Lidel. 115 Choong, K. K. (2014). Has this large number of performance measurement publications contributed to its better understanding? A systematic review for research and applications. International Journal of Production Research, 52(14), 4174–4197. https://doi.org/10.1080/00207543.2013.866285 Cocca, P., & Alberti, M. (2010). A framework to assess performance measurement systems in SMEs. International Journal of Productivity and Performance Management, 59(2), 186–200. https://doi.org/10.1108/17410401011014258 Cunha, F., Dinis-Carvalho, J., & Sousa, R. M. (2023). Performance Measurement Systems in Continuous Improvement Environments: Obstacles to Their Effectiveness. Sustainability (Switzerland), 15(1). https://doi.org/10.3390/su15010867 Cunha, F., Dinis-Carvalho, J., & Sousa, R. M. (2024a). Assessment of Performance Measurement Systems’ Ability to Mitigate or Eliminate Typical Barriers Compromising Organisational Sustainability. Sustainability (Switzerland) , 16(5). https://doi.org/10.3390/su16052173 Cunha, F., Dinis-Carvalho, J., & Sousa, R. M. (2024b). Key Characteristics of effective Performance Measurement Systems. [Manuscript Submitted for Publication]. Cunha, F., Dinis-Carvalho, J., & Sousa, R. M. (2024c). Analysis of barriers for performance measurement system effectiveness in a company: perceptions across hierarchical levels. International Journal of Lean Six Sigma, 2040(4166). https://doi.org/10.1108/IJLSS-12-2023-0205 Franco-Santos, M., Kennerley, M., Micheli, P., Martinez, V., Mason, S., Marr, B., Gray, D., & Neely, A. (2007). Towards a definition of a business performance measurement system. International Journal of Operations and Production Management, 27(8), 784–801. https://doi.org/10.1108/01443570710763778 Franco-Santos, M., Lucianetti, L., & Bourne, M. (2012). Contemporary performance measurement systems: A review of their consequences and a framework for research. Management Accounting Research, 23(2), 79–119. https://doi.org/10.1016/j.mar.2012.04.001 Franco, M., & Bourne, M. (2003). Factors that play a role in “managing through measures.” Management Decision, 41(8), 698–710. https://doi.org/10.1108/00251740310496215 Ghalayini, A. M., & Noble, J. S. (1996). The changing basis of performance measurement. International Journal of Operations and Production Management, 16(8), 63–80. https://doi.org/10.1108/01443579610125787 Gupta, S., & Jain, S. K. (2013). A literature review of lean manufacturing. International Journal of Management Science and Engineering Management, 8(4), 241–249. https://doi.org/10.1080/17509653.2013.825074 Hudson Smith, M., & Smith, D. (2007). Implementing strategically aligned performance measurement in small firms. International Journal of Production Economics, 106(2), 393–408. https://doi.org/10.1016/j.ijpe.2006.07.011 Ingaldi, M., Ulewicz, R. (2020). Problems with the Implementation of Industry 4.0 in Enterprises from the SME Sector. Sustainability, 12, 217. DOI: 10.3390/su12010217 ISO/TC 184/SC5. (2014). ISO 22400-1:2014 Automation systems and integration — Key performance indicators (KPIs) for manufacturing operations management (Vol. 2014). Jonsson, P., & Lesshammar, M. (1999). Evaluation and improvement of manufacturing performance measurement systems - the role of OEE Patrik Jonsson Magnus Lesshammar Article. International Journal of Operations & Production Management, 19(1), 55–78. http://dx.doi.org/10.1108/01443579910244223 Joshi, A., Kale, S., Chandel, S., & Pal, D. (2015). Likert Scale: Explored and Explained. British Journal of Applied Science & Technology, 7(4), 396–403. https://doi.org/10.9734/bjast/2015/14975 116 Kennerley, M., & Neely, A. (2002). A framework of the factors affecting the evolution of performance measurement systems. International Journal of Operations and Production Management, 22(11), 1222–1245. https://doi.org/10.1108/01443570210450293 Kerzner, H. (2017). Project management metrics, KPIs and D dashboards: a guide to measuring and monitoring project performance (Third Edit). John Wiley & Sons. Langwerden, E. F. (2015). Performance Measurement System Development in SMEs : Testing & Refining The Circular Methodology. In 5Th IBA Bachelor Thesis Conference, July 2015, Enschede, Netherlands. Lee, K., & Lee, S. (2023). Enhancing R&D Performance Management: A Case of R&D Projects in South Korea. Sustainability (Switzerland), 15(15), 1–14. https://doi.org/10.3390/su151511752 Lehtinen, J., & Ahola, T. (2010). Is performance measurement suitable for an extended enterprise? International Journal of Operations and Production Management, 30(2), 181–204. https://doi.org/10.1108/01443571011018707 Malagueño, R., Lopez-Valeiras, E., & Gomez-Conde, J. (2018). Balanced scorecard in SMEs: effects on innovation and financial performance. Small Business Economics, 51(1). https://doi.org/10.1007/s11187-017-9921-3 McCunn, P. (1998). The Balanced Scorecard... The eleventh commandment. Management Accounting: Magazine for Chartered Management Accountants, 76(11), 34. https://doi.org/Available: https://www.proquest.com/tradejournals/balanced-scorecard-eleventh-commandment/docview/195676560/se-2?accountid=39260. Meekings, A. (1995). Unlocking the potential of performance measurement: A practical implementation guide. Public Money & Management, 15(4), 5–12. https://doi.org/10.1080/09540969509387888 Mettänen, P. (2005). Design and implementation of a performance measurement system for a research organization. Production Planning and Control, 16(2), 178–188. https://doi.org/10.1080/09537280512331333075 Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & Group, T. P. (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med, 6(7). https://doi.org/10.1371/journal.pmed.1000097 Munir, R., Baird, K., & Si, Z. (2012). Associations with the use of multidimensional performance measures and the effectiveness of performance measurement systems. Asia-Pacific Management Accounting Journal, 7(2). Naslund, D., & Norrman, A. (2019). A performance measurement system for change initiatives: An action research study from design to evaluation. Business Process Management Journal, 25(7), 1647–1672. https://doi.org/10.1108/BPMJ11-2017-0309 Neely, A., & Bourne, M. (2000). Why Measurement Initiatives Fail. Measuring Business Excellence, 4, 3–7. https://doi.org/10.1108/13683040010362283 Neely, A., Mills, J., Platts, K., Richards, H., Gregory, M., & Kennerley, M. (2000). Performance measurement system design : developing and testing a process-based approach. International Journal of Operations & Production Management, 20(10), 1119–1145. Pandis, N. (2016). The chi-square test. American Journal of Orthodontics and Dentofacial Orthopedics, 150(5), 898–899. https://doi.org/10.1016/j.ajodo.2016.08.009 Pedersen, E. R. G., & Sudzina, F. (2012). Which firms use measures?: Internal and external factors shaping the adoption of performance measurement systems in Danish firms. International Journal of Operations and Production Management, 32(1), 4–27. https://doi.org/10.1108/01443571211195718 117 Sá, J., Carvalho, A., Fonseca, L., Santos, G., Dinis-Carvalho, J. (2023). Science Based Targets and the factors contributing to the sustainable development of an organisation from a Literature review to a conceptual model. Production Engineering Archives, 29(3), 241-253. DOI: 10.30657/pea.2023.29.28 Schneiderman, A. M. (2013). Why Balanced Scorecards Fail. Journal of Strategic Performance Measurement. Striteska, M., & Svoboda, O. (2012). Survey of performance measurement systems in Czech companies. E a M: Ekonomie a Management, 15(2). Stříteská, M., Zapletal, D., & Jelínková, L. (2016). Performance management systems in Czech companies: Findings from a questionnaire survey. E a M: Ekonomie a Management, 19(4), 44–55. https://doi.org/10.15240/tul/001/2016-4004 Suwignjo, P., Bititci, U. S., & Carrie, A. S. (2000). Quantitative Models for Performance Measurement System. International Journal of Production Economics, 64(1), 231–241. https://doi.org/10.1016/S0925-5273(99)00061-4 Todorut, A. V., Bojinca, M., & Tselentis, V. (2013). Using balanced scorecard for measuring excellence in SMEs. Recent Researches in Law Science and Finances. Tung, A., Baird, K., & Schoch, H. P. (2011). Factors influencing the effectiveness of performance measurement systems. International Journal of Operations and Production Management, 31(12), 1287–1310. https://doi.org/10.1108/01443571111187457 Ukko, J., & Saunila, M. (2020). Understanding the practice of performance measurement in industrial collaboration: From design to implementation. Journal of Purchasing and Supply Management, 26(1). https://doi.org/10.1016/j.pursup.2019.02.001 Wouters, M., & Wilderom, C. (2008). Developing performance-measurement systems as enabling formalization: A longitudinal field study of a logistics department. Accounting, Organizations and Society, 33(4–5), 488–516. https://doi.org/10.1016/j.aos.2007.05.002 Yang, G., Macnab, A., Yang, L., & Fan, C. (2015). Developing performance measures and setting their targets for national research institutes based on strategy maps. Journal of Science and Technology Policy Management, 6(2), 165–186. https://doi.org/10.1108/JSTPM-12-2014-0042 Zairi, M. (1994). Measuring performance for business results. Springer Science & Business Media. https://books.google.pt/books?id=VfD7CAAAQBAJ Voss, C.A., Tsikiktsis, N., Frohlich, M. (2002). Case research in operations management, International Journal of Operations & Production Management, 22, 195-219. Xu, X., Lu, Y., Vogel-Heuser, B., Wang, L. (2021). Industry 4.0 and Industry 5.0-Inception, conception and perception, Journal of Manufacturing Systems, 61, 530-535, DOI: 10.1016/j.jmsy.2021.10.006 118 6.8 Appendix Appendix A Table 1. Legend of the flowchart symbols Symbol Meaning Input/Output Start/End Process Sub Process Document Annotation Decision Appendix B Table 2. Questions in the questionnaire (Authors’ own work) Question The PMS of the organization is effective? The PMS improved since the last questionnaire? The company culture stimulates continuous improvement? The PMS is linked to the organization strategy (objectives, indicators, targets)? The communication system is effective? The deployment of the PMS through the organization is effective? The stakeholders are involved (top management and employees)? The resources and training necessary exist? The performance is linked to rewards? The appropriate IT tools exist? The procedures associated with the PMS are documented? The indicators are appropriate? The indicators are balanced? The system is simple? The system is clear? The system is dynamic? 119 Appendix C Fig. 1. Likert classification of the existence of the characteristics in the organization (Authors’ own work) Appendix D Table 3. Questionnaire chi-square hypothesis results and p-value (Authors’ own work) Question Education Gender Shift Department Role The PMS of the organization is effective? H1(0,007) H0(0,590) H1(0,034) H1(0,033) H0(0,177) The PMS improved since the last questionnaire? H0(0,502) H0(0,960) H0(0,722) H0(0,378) H0(0,258) The company culture stimulates continuous improvement? H0(0,356) H0(0,955) H0(0,304) H0(0,987) H0(0,468) The PMS is linked to the organization strategy? H0(0,659) H0(0,527) H0(0,538) H0(0,616) H0(0,934) The communication system is effective? H1(0,022) H0(0,720) H0(0,250) H0(0,237) H0(0,210) The deployment of the PMS through the organization is effective? H1(0,049) H0(0,927) H0(0,242) H0(0,354) H0(0,319) The stakeholders are involved (top management and employees)? H1(0,031) H0(0,772) H0(0,059) H0(0,133) H0(0,469) The resources and training necessary exist? H0(0,066) H0(0,442) H0(0,241) H0(0,548) H0(0,119) The performance is linked to rewards? H0(0,068) H0(0,513) H0(0,283) H0(0,268) H0(0,551) The data management system is effective? H1(0,010) H0(0,488) H1(0,033) H1(0,040) H0(0,131) The appropriate IT tools exist? H0(0,124) H0(0,527) H0(0335) H0(0,452) H0(0,516) The procedures associated with the PMS are documented? H1(0,040) H0(0,867) H0(0,100) H0(0,418) H0(0,872) The indicators are appropriate? H0(0,052) H0(0,426) H0(0,087) H0(0,123) H0(0,768) The indicators are balanced? H1(0,039) H0(0,476) H0(0,099) H0(0,104) H0(0,355) The target definition is appropriate? H0(0,090) H0(0,948) H1(0,033) H0(0,600) H1(0,049) The system is simple? H1(0,023) H0(0,931) H0(0,107) H0(0,299) H0(0,568) The system is clear? H1(0,008) H0(0,857) H1(0,018) H0(0,314) H0(0,365) The system is dynamic? H1(0,008) H0(0,931) H1(0,046) H0(0,808) H0(0,243) 35% 35% 65% 91% 39% 35% 39% 30% 52% 35% 74% 43% 39% 43% 30% 57% 52% 39% 52% 13% 26% 4% 48% 43% 43% 48% 35% 35% 13% 9% 26% 13% 39% 13% 17% 22% 13% 52% 9% 4% 13% 22% 17% 22% 13% 30% 13% 48% 35% 43% 30% 30% 30% 39% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% The PMS of the organization is effective The PMS improved since the last questionnaire The company culture stimulates continuous… The PMS is linked to the organization startegy… The communciation system is effective The deployment of the PMS through the organization is… The stakeholders are involved (top management… The resources and training necessary exist The performance is linked to rewards The data management system is effective The appropriate IT tools exist The procedures associated with the PMS are… The indicators are appropriate The indicators are balanced The target definition is appropriate The system is simple The system is clear The system is dynamic 12345678910 11 12 13 14 15 16 17 18 Agree/Strongly Agree Disagree/Strongly disagree Indiferent 120 7. CONCLUSÕES E TRABALHO FUTURO Neste capítulo são apresentadas as conclusões relativamente ao trabalho desenvolvido, bem como oportunidades para investigações futuras. 7.1 Conclusões A medição de desempenho é um fator fundamental para a melhoria contínua das organizações, pois permite identificar áreas que necessitam ser melhoradas, criando uma base de comparação e avaliação para a melhoria. O Sistema de Medição de Desempenho / Performance Measurement System (PMS) de uma organização tem de ser eficaz para permitir uma adequada medição e monitorização de desempenho e consequentemente servir como base para a melhoria contínua. O objetivo desta investigação foi o desenvolvimento de um modelo e metodologia para a implementação de um sistema de medição de desempenho, apoiando-se em 2 perguntas de investigação: 1. Quais os principais fatores que dificultam a implementação, uso e manutenção de um sistema de medição de desempenho numa organização? 2. Como pode ser alcançada a melhoria da implementação, uso e manutenção de um sistema de medição de desempenho? De forma a dar resposta à primeira pergunta de investigação, foram identificadas 19 barreiras à eficácia de um PMS através de uma revisão sistemática de literatura, conforme apresentado no capítulo 2. Essas barreiras são: cultura de culpabilização; sistema pouco claro; sistema complexo; falta de recompensas de desempenho; excesso de indicadores; indicadores inapropriados; falta de ligação à estratégia organizacional; falta de envolvimento da gestão de topo; falta de uso para a melhoria; falsas expectativas; falta de recursos com formação; falta de compreensão dos indicadores; problemas na definição de objetivos; sistema de comunicação ineficaz; falta de equilíbrio de indicadores; ferramentas TI (Tecnologia da Informação) inapropriadas; falta de envolvimento dos colaboradores; tempo e recursos requeridos; dificuldades em recolher, analisar e apresentar dados. Foi também possível verificar que as relações entre estas barreiras são complexas, existindo vários relacionamentos causa-efeito, podendo a falha de um PMS ser causada por uma ou por uma combinação de várias destas barreiras. Adicionalmente, foi possível verificar as perceções da existência destas barreiras numa organização, conforme apresentado no capítulo 3. Foi possível identificar quais as barreiras para as quais existia uma maior perceção da sua existência, sendo também possível verificar que essas perceções variam consideravelmente de acordo com as categorias a que as pessoas pertencem, nomeadamente o seu 121 nível hierárquico. Isto permite aferir que a eficácia de um PMS não se traduz da mesma forma para todos os níveis hierárquicos de uma organização, sendo algumas barreiras mais evidentes para uns níveis hierárquicos do que para outros. Para responder à segunda pergunta de investigação, foram identificados os modelos PMS existentes e foi analisada a sua capacidade para mitigar ou eliminar as barreiras à eficácia de um PMS, conforme apresentado no capítulo 4. Apesar de não ter sido encontrado nenhum PMS que solucionasse todas as barreiras, foi possível identificar características comuns de diferentes PMS com capacidade de combater barreiras comuns. Enquanto foram identificadas barreiras que os PMS existentes são capazes de eliminar ou mitigar, outras, como cultura de culpabilização, falta de recursos com formação, falta de recompensas de desempenho, e sistema de comunicação ineficaz, não foram abordadas de maneira eficaz por nenhum dos PMS analisados. Desta forma foram identificadas lacunas nos PMS existentes, que foram tidas em conta no desenvolvimento de um novo modelo. De modo a responder à segunda pergunta de investigação foram também identificadas, através de uma revisão sistemática de literatura, as principais características para um PMS eficaz (capítulo 5), sendo organizadas em 4 dimensões: x Cultura: estimula melhoria contínua; ligação à estratégia da organização; comunicação eficaz; desdobramento pela organização. x Pessoas: envolvimento das partes interessadas; recursos e formação; ligação de desempenho a recompensas. x Dados e ferramentas: gestão de dados; ferramentas TI apropriadas; procedimentos documentados. x Indicadores: indicadores apropriados; indicadores balanceados; definição apropriada de objetivos. Certas características como, simplicidade, clareza e dinamismo foram identificadas como essenciais para todas as dimensões de um PMS. Essas características possuem relações de interdependência complexas, podendo ser de “habilitação”, quando uma característica possibilita outra, ou de “requisito”, quando uma característica é necessária para que outra exista. Com base no reconhecimento obtido e de forma a concluir a resposta à segunda pergunta de investigação foi possível desenvolver um modelo de PMS potencialmente mais eficaz dos que os existentes na literatura, bem como a respetiva metodologia de implementação, que são apresentadas no capítulo 6. O processo de design e implementação do modelo começa com a promoção de uma cultura de melhoria contínua, o estabelecimento de uma estratégia bem definida e a garantia de uma comunicação eficaz – 122 pré-requisitos para que a gestão de topo defina objetivos claros. Em seguida, é fundamental definir indicadores de desempenho adequados e equilibrados, apoiados por um processo eficiente de gestão de dados que inclua ferramentas de TI apropriadas, recursos e formação, todos documentados em procedimentos formais. Uma vez estabelecidos os indicadores, devem-se identificar ações que influenciem positivamente esses indicadores, e definir metas de desempenho para períodos específicos. Essas metas devem estar vinculadas a recompensas, atuando como facilitadoras do envolvimento das partes interessadas. Esse processo deve ser repetido em todos os níveis hierárquicos, começando pela definição de objetivos. Além disso, é importante revisitar o processo sempre que houver mudanças significativas no ambiente interno ou externo, garantindo que o sistema de medição de desempenho permaneça atualizado e alinhado com a estratégia organizacional. O trabalho apresentado permitiu o desenvolvimento de um modelo de PMS potencialmente mais eficaz do que os existentes na literatura, assim como da respetiva metodologia de implementação, que deverão facilitar a criação, utilização e manutenção de um PMS, evitando as barreiras mais comuns à eficácia. A principal limitação deste estudo reside na falta de validação do modelo e da metodologia em organizações reais. 7.2 Trabalho futuro Como recomendações de investigação futura, sugere-se a validação do modelo e metodologia apresentados em contexto real. Para isso sugere-se a aplicação do modelo e metodologia no desenvolvimento, uso e manutenção de um PMS em diferentes organizações. Antes dessa etapa, é essencial realizar um diagnóstico ao PMS da organização, bem como das perceções das pessoas sobre a existência das principais barreiras à eficácia do PMS na organização. Para esse fim, poderão ser utilizadas as metodologias de classificação da capacidade de um PMS eliminar ou mitigar as barreiras, conforme apresentado no terceiro artigo (Cunha et al., 2024a), bem como os métodos de entrevistas e questionários descritos no segundo artigo (Cunha et al., 2024d), respetivamente. O diagnóstico destas duas componentes deverá ser repetido após o desenvolvimento, utilização e manutenção do PMS de acordo com o modelo apresentado, de forma a aferir a eficácia do modelo e metodologia em ambiente real. 123 REFERÊNCIAS BIBLIOGRÁFICAS Basili, V. R. (1994). The Goal Question Metric Paradigm. Encyclopedia of Software Engineering , 528–532. https://www.cs.umd.edu/~basili/publications/technical/T89.pdf Bititci, U. S. (2015). Managing Business Performance. In 1st . Wiley. https://doi.org/10.1002/9781119166542.app2 Boone, H. N., & Boone, D. A. (2012). Analyzing likert data. Journal of Extension , 50 (2), 1–5. Cunha, F., Dinis-Carvalho, J., & Sousa, R. M. (2023). Performance Measurement Systems in Continuous Improvement Environments: Obstacles to Their Effectiveness. Sustainability (Switzerland) , 15 (1), 867. https://doi.org/10.3390/su15010867 Cunha, F., Dinis-Carvalho, J., & Sousa, R. M. (2024a). Assessment of Performance Measurement Systems’ Ability to Mitigate or Eliminate Typical Barriers Compromising Organisational Sustainability. Sustainability (Switzerland) , 16 (5), 2173. https://doi.org/10.3390/su16052173 Cunha, F., Dinis-Carvalho, J., & Sousa, R. M. (2024b). Key Characteristics of effective Performance Measurement Systems. Manuscript Submitted for Publication . Cunha, F., Dinis-Carvalho, J., & Sousa, R. M. (2024c). Performance Measurement Systems: an effective model. Manuscript Submitted for Publication . Cunha, F., Dinis-Carvalho, J., & Sousa, R. M. (2024d). Analysis of barriers for performance measurement system effectiveness in a company: perceptions across hierarchical levels. International Journal of Lean Six Sigma , Ahead of p (Ahead of print). https://doi.org/10.1108/IJLSS-12-2023-0205 Ferrer, B. R., Muhammad, U., Mohammed, W. M., & Lastra, J. L. M. (2018). Implementing and visualizing ISO 22400 key performance indicators for monitoring discrete manufacturing systems. Machines , 6 (3). https://doi.org/10.3390/MACHINES6030039 Finkelstein, L. (2003). Widely, strongly and weakly defined measurement. Measurement: Journal of the International Measurement Confederation , 34 (1), 39–48. https://doi.org/10.1016/S0263-2241(03)00018-6 Fitzgerald, L., & Moon, P. (1996). Performance Measurement in Service Industries: Making It Work . CIMA Publishing. Franceschini, F., Galetto, M., & Maisano, D. (2007). Management by Measurement: Designing key indicators and performance measurement systems . SpingerScience & Business Media. Franco-Santos, M., Kennerley, M., Micheli, P., Martinez, V., Mason, S., Marr, B., Gray, D., & Neely, A. (2007). Towards a definition of a business performance measurement system. International Journal of Operations and Production Management , 27 (8), 784–801. https://doi.org/10.1108/01443570710763778 Gabris, G. T. (1986). Recognizing Management Technique Dysfunctions : How Management Tools Often Create More Problems than They Solve. Public Productivity Review , 10 (2), 3–19. https://doi.org/10.2307/3380448 Ghalayini, A. M., & Noble, J. S. (1996). The changing basis of performance measurement. International Journal of Operations and Production Management , 16 (8), 63–80. https://doi.org/10.1108/01443579610125787 Imai, M. (1997). Gemba Kaizen (2nd Editio). McGraw-Hill. Joshi, A., Kale, S., Chandel, S., & Pal, D. (2015). Likert Scale: Explored and Explained. British Journal of Applied Science & Technology , 7 (4), 396–403. https://doi.org/10.9734/bjast/2015/14975 Kanji, G. K. (2002). Performance measurement system. Total Quality Management , 13 (5), 715–728.