Leveraging LLMs for Smart Cities Qualitative Data Analysis

Leveraging LLMs for Smart Cities Qualitative Data Analysis Elisa Covato1,*, Kamran Soomro1, Zaheer Khan1, Muhammad Bilal2, Sam Kirby3, and Tom Yiangou3 1 School of Computing and Creative Technologies, University of the West of England, Bristol, UK 2 Business School, Birmingham City University, Birmingham, UK 3 Bristol City Council, Bristol, UK * [email protected] * corresponding author Abstract. Public authorities frequently conduct surveys and analyse data from citizens, a process that is often labour-intensive when performed manually. This paper explores how Generative Artificial Intelligence (GAI) can assist in automating data analysis for public authorities. In this respect, we investigate the potential of Large Language Models (LLMs) to perform sentiment analysis and summarisation of unstructured data as smart services. Using data from the East Bristol Liveable Neighbourhood (EBLN) as a case study, we assess the accuracy and precision of these models and validate the results against ground truth data and expert evaluations. Our findings indicate that sentiment classification achieved over 90% accuracy. In contrast, the expert validation rated the summarisation without context accuracy highly satisfactory, however, the satisfaction was low when contextual summarisation was evaluated. These results suggest that LLMs offer a promising approach to improving the efficiency of qualitative analysis, though further research is required to enhance their accuracy and usefulness. Keywords: sentiment analysis, text summarisation, smart cities, generative Artificial Intelligence. 1 Introduction Cities are undergoing significant transformations as they seek to harness the full potential of urban governance driven by Information and Communication Technology[1]. Several benefits are sought to manage the complexity of urban systems and gain insights from public participation in planning processes [6,18]. An important driving force in this transformation concerns the information overload from urban systems and local initiatives such as citizen observatories. The recent policy shift towards more emphasis on citizen engagement in data gathering, planning and decision-making 2 E. Covato et al. processes e.g., European Bauhaus 1 , Built4People 2 and EU Mission Cities 3 , requires further technological support to manage and analyse such data. In particular, much of the data is gathered through surveys or comments provided by citizens on specific plans or topics via social media or other participation platforms. Since these data are typically unstructured, analysing it can be time-consuming and requires specific skills. This problem can be exacerbated by the growing number of surveys and high levels of public participation. Over the past decade, data analytics has emerged as an important innovative technology in the domain of smart cities [1, 22]. Various statistical, machine learning and data mining techniques have proven to be revolutionary in the context of Smart City Data Analytics. Among these, deep learning models have become one of the most cutting-edge technologies, offering significant contributions across a range of applications for smart city services [15, 26]. Recently, disruptive technologies such as Generative Artificial Intelligence (GAI) and Large Language Models (LLMs) are revolutionising the way data analysis can be performed. Their benefits cut across domains such as transportation, urban planning, and smart buildings. Their ability to analyse and evaluate unstructured qualitative data with high accuracy and precision makes them ideal candidates to help public authorities streamline their processes. In this respect, we investigate the suitability of LLMs for two automated services for public authorities: (i) sentiment analysis and (ii) summarisation for survey data. We applied selected models to real, unstructured textual data collected through the East Bristol Liveable Neighbourhood (EBLN) 4 initiative in Bristol, United Kingdom. The EBLN is one of the pilots under the Horizon Europe GREENGAGE project 5 . The GREENGAGE project aims to develop citizen observatories to facilitate European GREEN Deal policy objectives across different pilots in EU and UK. By leveraging citizen science, the project will collect extensive unstructured qualitative data through surveys, discussions and comments gathered through the GREENGAGE GREEN Engine, such as mobile apps. Additionally, the project will combine sensor data from traffic monitoring and pollution levels, with citizens' feedback to raise public awareness and engagement in environmental issues, as well as to gather their opinions and needs on these topics. The remainder of this paper is structured as follows: Section 2 presents background information on LLMs and their suitability for data analysis tasks. Section 3 introduces the EBLN dataset, detailing our pre-processing steps, model architectures, and experimental design of the two key services: sentiment analysis and text summarisation. We discuss our findings and their implications in Section 4, followed by conclusive remarks in Section 5. 1 https://new-european-bauhaus.europa.eu/about/about-initiative\_en 2 https://built4people.eu/ 3 https://netzerocities.eu/mission-cities/ 4 https://eastbristolliveableneighbourhoods.commonplace.is/ 5 https://www.greengage-project.eu/ Leveraging LLMs for Smart Cities Qualitative Data Analysis 3 2 Background 2.1 Large Language Models for Qualitative Analytics Traditional approaches to qualitative data analysis rely on understanding vast amounts of unstructured data (text, video, or audio) through manual processes [8]. These approaches typically require significant time investments, and are heavily dependent on human expertise, inevitably incorporating individual biases into the analytical process. The rise of Large Language Models has encouraged a shift towards developing more sophisticated, automated systems for qualitative data analysis. These LLMs, comprising billions of parameters and trained on vast datasets, have demonstrated remarkable capabilities in various language processing tasks [16]. Various studies have explored the application of LLMs in qualitative analysis, focusing on specific tasks such as sentiment analysis [13], text summarisation [2,27], theory-driven categorisation [25], or holistic solutions [19,23]. However, it remains a gap in understanding their practical benefits in real-world scenarios. Current research often lacks evaluation from end-users, or applications in real smart cities case studies. This study aims to address this gap by applying LLMs on recent survey data, and incorporating feedback from actual practitioners. Our final goal is to set the foundation for future research directions, and assess the true potential of LLMs in qualitative data analysis within practical contexts. 2.2 Zero-shot learning and LLMs Training machine learning models is a resource-intensive task, requiring substantial data, time, and computational power [20]. However, real-world scenarios often impose limitations on these resources. Zero-shot learning is a machine learning paradigm that enables models to make predictions for classes they have not encountered during training [3], reducing computational costs and the need for large amounts of annotated datasets. While this has been in the past mainly used for computer vision, it has recently gained attraction with the surge of LLMs [24]. LLMs are pre-trained on vast amounts of data, capable of performing a wide range of language tasks effectively. Zero-shot learning leverages this extensive knowledge base, making LLMs flexible and adaptable across various scenarios. Given the ability of LLMs to understand and generate human-like text, LLMs can be directed to perform zero-shot learning simply through natural language instructions or ‘prompts’ [14]. This conversational manner of task description allows these models to adapt to new contexts without the need for task-specific training data, time-consuming data preparation steps or extensive model fine-tuning. Moreover, this approach is particularly beneficial in providing non-experts with a tool to perform language analysis. In this study, we have created suitable prompts to perform zero-shot learning for sentiment classification and summarisation. Creating effective prompts involves a trialand-error approach, which can be challenging [17]. While exploring the nuances of prompt creation is outside the scope of this initial study, we will focus on discussing the results obtained. It is important to note that varying prompts may yield different results in this context. 4 E. Covato et al. 2.3 Sentiment Classification using LLMs Understanding public opinion, citizens’ experiences and satisfaction is crucial in the context of Smart Cities. This enables authorities to assess initiative reception, pinpoint concerns, improve quality of life, and ensure responsive governance. To accomplish these objectives effectively, sentiment classification emerges as an invaluable analytical tool [7]. This natural language processing technique aims to determine the emotional tone or attitude expressed in a piece of text. It typically involves categorising text into binary sentiments - positive or negative - although more nuanced classifications are also feasible. Traditional sentiment classification techniques often rely on lexicon-based methods, which use predefined dictionaries of sentiment-bearing words, or machine learning approaches such as Naive Bayes, Support Vector Machines and Recurrent Neural Networks (RNNs) which learn from labelled data [21]. These techniques, while effective, can struggle with context-dependent sentiments, require large amounts of labelled data for training, demand extensive feature engineering and dataset preparation, which can be time-consuming and costly to perform. In contrast, LLMs excel in understanding context and nuanced language, require minimal task-specific training due to their pretrained nature, and can perform well with limited or no labelled data [13,21]. LLMs’ versatility in processing sentiments across a range of domains, with superior accuracy, makes them ideal for sentiment analysis, especially in the context of survey data collection, characterised by large quantitative of unstructured data covering different topics. 2.4 Flan-T5: model selected for sentiment classification For the sentiment classification task in this study, we have tested Flan-T5 language model, developed by Google AI in 2022 [4]. Flan-T5 is an open-source, improved version of the original T5 (Text-to-Text Transfer Transformer) model, treating all natural language processing tasks as text-to-text problems. This model has been fine-tuned on a diverse range of tasks using instruction-based learning, making it suitable to be used in zero-shot learning context. This versatile approach adapts well to sentiment classification, with the model’s instruction following ability making it an easy tool for language analysis, even for non-technical users. Flan-T5 is comparable in performance to many heavyweight, proprietary models while being open-source [28], reducing cost constraints and allowing customisation. These features make it an excellent candidate for this investigation, balancing advanced capabilities, flexibility, and cost-effectiveness. Google has released multiple variants of Flan-T5, ranging from compact models with 80 million parameters and 300MB in size, to expansive versions with 11 billion parameters requiring 80GB of memory. For this study, we evaluated the following checkpoints available on Hugging Face 6 : 1. Flan-T5-small: 80 million parameters, 300MB 2. Flan-T5-base: 250 million parameters, 990MB 6 https://huggingface.co/docs/transformers/main/en/model\char`_doc/flan-t5 Leveraging LLMs for Smart Cities Qualitative Data Analysis 5 3. Flan-T5-large: 780 million parameters, 1GB The choice of these model sizes was motivated by the aim to assess the performance on common laptops with limited resources, evaluate the trade-off between execution time and accuracy across different model sizes, and make the approach accessible to a wide range of users. 2.5 Summarisation using LLMs In the digital age, where information is abundant and readily available, the ability to condense and summarise text is of high importance. Text summarisation serves as a powerful tool that aids in the efficient extraction and comprehension of key information from large volumes of data. It enables individuals and businesses to sift through vast amounts of content and quickly identify the most relevant and critical information. By transforming lengthy documents into concise summaries, it saves time, enhances productivity, and supports informed decision-making processes. As such, text summarisation plays a crucial role in navigating information overload in today’s data-driven world. Text summarisation approaches have been around for a while. For example, frequency-based, graph-based and topic-based approaches aid in extractive summarisation techniques. On the other hand, Sequence-to-Sequence models such as RNNs and LSTMs aid abstractive summarisation. The emergence of LLMs promises to usher in a new era of summarisation techniques. However, being relatively recent, not much work has been performed to investigate the performance of LLMs for summarisation in various domains. Although there is limited research, it suggests that traditional approaches still perform reasonably well for generic tasks, though LLMs might outperform them when fine-tuned for specific domains [9,10]. Therefore, in this paper, we explore how LLMs perform in summarisation of survey data. 2.6 Llama3: model selected for summarisation For the summarisation service, we have chosen the Llama3 models. This model has been recently released by Meta in 8B and 70B parameter scales in both instructiontuned and fine-tuned varieties 7 . We opted to test Llama3 because it is a new model that has shown promising results in various tasks, including summarisation, with comparable or superior performance to other leading LLMs. The LLama3 models have been trained using a variety of techniques to optimise both the pretraining and post-training steps. In particular, a dataset containing 15 trillion tokens was specifically curated from publicly available sources to train the model. Rigorous quality control measures, including heuristic filters and eliminating semantic duplicates, have been implemented to improve data reliability compared to previous Llama versions. Taking into account these factors, along with the model’s evaluation of 1,800 prompts across 12 essential tasks, including summarisation, we have selected 7 https://ai.meta.com/blog/meta-llama-3/ 6 E. Covato et al. this model specifically for our data summarisation task with the aim to achieve robust, efficient, and practical summarisation results. 3 Methodology and Implementation 3.1 Case study and data collection The East Bristol Liveable Neighbourhood (EBLN) project is a community-driven initiative to transform East Bristol's streets. The main goal is to collaborate with local residents, workers, students and commuters to design streets that are liveable and people-friendly. This project is part of a co-design phase aimed at developing long-term solutions with the help of the local community. The EBLN is also a pilot in the Horizon Europe GREENGAGE project, which promotes inclusive citizen observatories by using digital solutions. The survey data used in this study were collected between January and March 2022 by Bristol City Council, UK. Participants, including those living, working, or travelling to the area, shared their opinions either using an interactive online map 8 , or by engaging with members of Bristol City Council through door-to-door interviews or community discussions. Face-to-face interviews were conducted and recorded manually. Citizens were asked about their thoughts on what they appreciate about their local area, and how they would like to improve it. They had the option to respond to either or both questions, resulting in over 400 individual contributions and a total of 845 comments of varying length. While the complete dataset cannot be disclosed due to confidentiality constraints, Table 1 presents a representative sample of responses collected. Online, citizens could express their feedback by: 1. Marking specific locations on a map 2. Linking a feeling, chosen from a range of five sentiments, spanning from negative to positive, with the selected area 3. Providing optional comments in a free-text box 4. Additionally, selecting one or more topics related to their comments, reasons for their expressed feeling, and suggestions for improving the area An example of response from the online survey is shown in Table 2. The dataset collected through the online tool comprises 540 geo-located entries, of which 90% contains both textual comments and sentiment labels. A preliminary study conducted on this dataset [5], which involved analysing the textual comments alongside the provided sentiment and topic labels, showed that most participants lived within the area and tended to emphasise negative aspects of their neighbourhood, particularly in relation to main roads and junctions. Interestingly, the word "road" frequently appeared in negative contexts, while "street" was associated with positive sentiments, reflecting nuanced perceptions of different urban spaces. Conversely, areas with green spaces generally 8 https://eastbristolliveableneighbourhoods.commonplace.is/map/map Leveraging LLMs for Smart Cities Qualitative Data Analysis 7 received more positive feedback. Sentiment analysis showed that negative statements were more prevalent around safety and environment-related comments, while texts linked to community aspects displayed a more balanced distribution of sentiments. Table 1. Example of responses from the EBLN interviews Q1: What do you like about your local area? A1: Good sense of community A2: It’s fairly diverse - there’s a real sense of community, it feels very friendly A3: Park, shops Q2: How would you like to make it better? A4: More activities for children and improve parks. Homes are too old and damaged and we need repairs A5: Traffic A6: Wider roads Table 2. Example of responses from the online EBLN map. Sentiment Positive Negative Subjects Trees and greenery on street, Street trees and planting Walking, Crossings Reasons Pleasant Not pedestrian friendly, Difficult to cross the street Suggestions Slow down traffic Add crossing, Safer junction for walking and cycling Comments The street planters have reduced traffic speed and made the street ’greener’. Something similar could be done in other locations within the project area and in traffic displacement areas outside the project area Hard to cross here - there is a traffic island slightly above this point but often want to cross lower down and it’s hard to do so as the road is busy with two lanes of fast traffic. The preliminary study [5] demonstrated the potential of analysing public feedback to inform urban planning initiatives. However, it also highlights the need for scalable and efficient methodologies. While the EBLN dataset has labelled textual data, most urban datasets lack such structured annotations, resulting in the need for extensive time to analyse and generate comprehensive summaries to extract insights and actionable steps. In this study, we have tested our LLM-based methods against the EBLN datasets. For sentiment classification, we have focused on the online EBLN dataset, leveraging its pre-existing sentiment labels to assess our approach's accuracy and effectiveness. For the summarisation tasks, we have worked with both the online and in-person 8 E. Covato et al. collected data, and results have been evaluated by members of Bristol City Council directly involved in the EBLN and GREENGAGE project. 3.2 Data preprocessing Our work centers exclusively on the free-text comments, and their corresponding sentiment labels in the case of the sentiment classification task, therefore all other features in our datasets were excluded. In line with our aim of utilising LLMs, our preprocessing was intentionally minimal. This is because LLMs are designed to handle natural language inputs, including variations in spelling, grammar, and informal expressions commonly found in public feedback. Therefore, for both tasks, we retained all comments regardless of length to preserve the full range of citizen feedback, since even brief comments can provide insight into urban perceptions. For the summarisation tasks, two files were generated that included all the text comments. These comments were either placed in one column for the online dataset or in two columns for the two questions asked during the interviews. In the case of the sentiment classification task, we performed a sentiment simplification. The original dataset included five sentiment categories: positive, mostly positive, neutral, mostly negative, negative. Following the approach in [5] we have excluded the neutral comments, and grouped the remaining labels into two polar groups, positive and negative. We simplified the sentiments into binary classification to facilitate a clearer, more straightforward interpretation of public opinions, and avoid the complexities of neutral or fine-grained sentiment analysis, which demands more nuanced approaches. As a result of our preprocessing, our method was tested on 446 textual comments from the EBLN survey, with 52 comments labelled as positive. 3.3 Service 1: Sentiment Classification For the sentiment classification task in this study, we have tested Flan-T5 language model in three different model sizes (see Section 2.4). To provide a comprehensive comparison, we tested these models both on a laptop device with an Intel Core i5 processor, and on an NVIDIA Titan RTX GPU-equipped computer. This additional test helped us quantify the reduction in execution time achievable with more powerful hardware, providing insights into the scalability of our approach. All the Flan-T5 model variants mentioned were tested using identical prompt instructions, to ensure a consistent and fair comparison across different model sizes. We aimed to use a clear and straightforward prompt, including only the essential components, namely the task name, task definition, and output format. Prompt format and wording were taken from [28]. Leveraging LLMs for Smart Cities Qualitative Data Analysis 9 Table 3. Sentiment classification prompt In the prompt, we specify the name of the task (Sentiment Classification), and provide instructions on the task itself (assign a label to a given sentence). We also specify the output format we require after the final word 'Label'. We have executed this prompt for every data point in the dataset, replacing {input_text} with the appropriate text content from the dataset. 3.4 Service 2: Summarisation For the purposes of this use case, we used Llama 3 with 8 billion parameters, and ran the model on an NVIDIA Titan RTX GPU-equipped computer. To test the efficacy of the prompt engineering approach in the summarisation service, we experimented with two types of prompts; with context and without context. Prompts with context included additional information or requests provided by users. Prompts without context only included the actual system instructions, without any additional user instructions. Furthermore, we also experimented with two different system prompts. System Prompt 1 specified a more elaborate prompt with instructions to the LLM to be helpful. System Prompt 2 was more bare bones and only instructed the LLM to provide a summary. The two prompts used are shown below. Table 4. Summarisation System Prompt 1 Provide a detailed summary of the following document using the given context. Be as helpful as possible. context: "{context}" document: "{text}" Please perform Sentiment Classification task. Given the sentence, assign a sentiment label from ['negative','positive']. Return label only. Sentence: {input_text} Label: 16 E. Covato et al. 14. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 55(9) (2023). https://doi.org/10.1145/3560815 15. Mohammadi, M., Al-Fuqaha, A., Sorour, S., Guizani, M.: Deep learning for iot big data and streaming analytics: A survey. IEEE Communications Surveys & Tutorials 20(4), 2923– 2960 (2018). https://doi.org/10.1109/COMST.2018.2844341 16. Naveed, H., Khan, A.U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., Mian, A.: A comprehensive overview of large language models (2024), https://arxiv.org/abs/2307.06435 17. Orlanski, G.: Evaluating prompts across multiple choice tasks in a zero-shot setting (2022), https://arxiv.org/abs/2203.15754 18. Poplin, A.: Playful public participation in urban planning: A case study for online serious games. Computers, Environment and Urban Systems 36(3), 195–206 (2012). https://doi.org/10.1016/j.compenvurbsys.2011.10.003 19. Rasheed, Z., Waseem, M., Ahmad, A., Kemell, K.K., Xiaofeng, W., Duc, A.N., Abrahamsson, P.: Can large language models serve as data analysts? a multi-agent assisted approach for qualitative data analysis (2024), https://arxiv.org/abs/ 2402.01386 20. Sarker, I.H.: Machine learning: Algorithms, real-world applications and research directions (2021). https://doi.org/10.1007/s42979-021-00592-x 21. Sharma, N.A., Ali, A.B.M.S., Kabir, M.A.: A review of sentiment analysis: tasks, applications, and deep learning techniques. International Journal of Data Science and Analysis (2024). https://doi.org/10.1007/s41060-024-00594-x 22. Soomro, K., Bhutta, M.N.M., Khan, Z., Tahir, M.A.: Smart city big data analytics: An advanced review. WIREs Data Mining and Knowledge Discovery 9(5), e1319 (2019). https://doi.org/10.1002/widm.1319 23. Tai, R.H., Bentley, L.R., Xia, X., Sitt, J.M., Fankhauser, S.C., Chicas-Mosier, A.M., Monteith, B.G.: An examination of the use of large language models to aid analysis of textual data. International Journal of Qualitative Methods 23, 16094069241231168 (2024). https://doi.org/10.1177/16094069241231168 24. Wang, Z., Pang, Y., Lin, Y.: Large language models are zero-shot text classifiers (2023), https://arxiv.org/abs/2312.01044 25. Xiao, Z., Yuan, X., Liao, Q.V., Abdelghani, R., Oudeyer, P.Y.: Supporting qualitative analysis with large language models: Combining codebook with gpt-3 for deductive coding. In: Companion Proceedings of the 28th International Conference on Intelligent User Interfaces. p. 75–78. IUI ’23 Companion, Association for Computing Machinery, New York, NY, USA (2023). https://doi.org/10.1145/ 3581754.3584136 26. Zaouali, K., Rekik, R., Bouallegue, R.: Deep learning forecasting based on autolstm model for home solar power systems. In: 2018 IEEE 20th International Conference on High Performance Computing and Communications; IEEE 16th International Conference on Smart City; IEEE 4th International Conference on Data Science and Systems (HPCC/SmartCity/DSS). pp. 235–242 (2018). https: //doi.org/10.1109/HPCC/SmartCity/DSS.2018.00062 27. Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., Hashimoto, T.B.: Benchmarking Large Language Models for News Summarization. Transactions of the Association for Computational Linguistics 12, 39–57 (01 2024). https://doi. org/10.1162/tacl_a_00632 28. Zhang, W., Deng, Y., Liu, B., Pan, S.J., Bing, L.: Sentiment analysis in the era of large language models: A reality check (2023), https://arxiv.org/abs/2305. 15005