StudMap 3.0 : an interoperable web-based platform for geospatial data offers in academic life

STUDMAP 3.0 - AN INTEROPERABLE WEB-BASED PLATFORM FOR GEOSPATIAL DATA OFFERS IN ACADEMIC LIFE Ștefana Cioban i STUDMAP 3.0 - AN INTEROPERABLE WEB-BASED PLATFORM FOR GEOSPATIAL DATA OFFERS IN ACADEMIC LIFE Dissertation supervised by Prof. Doutor Vitor Manuel Pereira Duarte dos Santos, Information Management School, NOVA Lisbon Dr. Torsten Prinz, Institute for Geoinformatics, WWU Münster Dr. Andrés Muñoz, Geotech Institute, UJI, Castelló February 2019 ii ACKNOWLEDGEMENTS I would first like to thank all my supervisors for their continuous support and guidance. It was a real pleasure to work with you and to learn from your critique. Thank you, Dr. Torsten Prinz, for enabling me to think out of my old box and for giving me an insight into the beauty of daring! Thank you, Prof. Vitor Santos, for always pushing this thesis closer to successful research! And, thank you, Dr. Andrés Muñoz for allowing me to test all my ideas on a working environment! I would like to thank all my colleagues from the Geospatial Technologies Masters for your expertise always inspired me. You are truly professional, and your advice was a good contribution to the output of my research. A special word goes for Diamantino Correia, from the ESRI remote support team of Lisbon, and for Sven Oliver Pagel, from the IVV department in the Institute for Geoinformatics, WWU Münster, for analyzing all technical problems so meticulously and helping me to bring this project to reality. I would also like to acknowledge all experts and students involved in the validation process of my research. Without your passionate participation and comments, the final evaluation of the portal could not have been successfully conducted. Finally, I must express my gratitude to my partner and my family for their support, patience and restless encouragement. iii STUDMAP 3.0 - AN INTEROPERABLE WEB-BASED PLATFORM FOR GEOSPATIAL DATA OFFERS IN ACADEMIC LIFE ABSTRACT Geographic Information Systems has now entered the realm of web and yields for feasible solutions to balance the technology offers with the users’ needs to share, access and explore the massive amounts of geodata available. Challenges occur when moving forward from old 2D platforms towards innovative and integrated webGIS systems that align functionality with the necessity to grant a complete understanding of the surrounding reality. 3D space responds to this but, however, stands only at the beginning of its era and cannot yet reach the development of 2D web integration. Research is now aiming at possible webGIS solutions to adapt to the special structure imposed by 3D data. In this context, this thesis focuses on designing an architecture for 2D and 3D geospatial data integration on a studentoriented web platform. This concept was further delivered and validated through a real case scenario – Studmap 3.0, a webGIS platform to serve the students of the University of Muenster in their academical life. The portal currently grants availability of geospatial data and web services of regional interest in a smart GIS environment that allows access and comparison of official services with own data. The implementation of Studmap 3.0 aided in the continuous improvement of the proposed architecture model and developed under a design science research cycle that reached its end once the final approval of its users was attained via a usability evaluation. Final strengths and drawbacks of the proposed architecture were ultimately identified together with an expert usability evaluation and a lab-based usability test of the resulting portal interface suitability for academic use. The results fall under the acceptable range with an 83.75 score for the System Usability Scale standardized questions when addressed to experts and a score of 83.87 when addressed to students. For the open-ended questions, the interface received an overall positive critique. A summary of future participants’ opinion on the benefits, drawbacks and proposed improvements was also delivered. Peers interested in similar concepts can use both this model and its final remarks as a reference for their work. KEYWORDS webGIS, geodata, 3D, web portal iv ACRONYMS 2D – Two Dimensional 3D – Three Dimensional API – Application Program Interface db – Database DBMS – Database Management System DEM – Digital Elevation Models ds – Data Store DSM – Digital Surface Models EPSG – European Petroleum Survey Group ESRI – Environmental Systems Research Institute GIS – Geographic Information System GPS – Global Positioning System HTTP – Hypertext Transfer Protocol HTTPS – Hypertext Transfer Protocol Secure ISO – International Standards Organization I3S – Indexed 3D Scene Layers JSON – JavaScript Object Notation LAS – Log-ASCII-Standard LiDAR – Light Detection and Ranging NRW – North Rhine-Westphalia OGC – Open Geospatial Consortium REST – Representational State Transfer RDBMS – Relational Database Management System SQL – Standardized Query Language v SDK – Software Development Kit SLPK – Scene Layer Package SUS – System Usability Scale UAS – Unmanned Aerial Systems UAV – Unmanned Aerial Vehicle URL – Uniform Resource Locators WAB – Web Appbuilder WCS – Web Coverage Service WFS – Web Feature Service WMS – Web Map Service WMTS – Web Map Tiling Service WWU – University of Münster (Westfälische Wilhelms-Universität) XML – Extensible Markup Language vi INDEX OF TEXT ACKNOWLEDGEMENTS .................................................................................................................. ii ACRONYMS ........................................................................................................................................ iv TABLE OF CONTENTS ..................................................................................................................... vi INDEX OF FIGURES ....................................................................................................................... viii INDEX OF TABLES ........................................................................................................................... ix 1. INTRODUCTION ........................................................................................................................ 1 BACKGROUND ........................................................................................................................ 1 MOTIVATION .......................................................................................................................... 1 AIMS AND OBJECTIVES ........................................................................................................... 2 2. THEORETICAL BACKGROUND ............................................................................................ 3 GEOSPATIAL CONTEXT AND WEBGIS..................................................................................... 3 BIG 3D DATA AND GEOSPATIAL SCIENCES .............................................................................. 4 2.2.1 3D geospatial data ............................................................................................................ 4 2.2.2 3D geospatial data management ....................................................................................... 5 GIS SERVERS ......................................................................................................................... 6 2.3.1 Defining concepts .............................................................................................................. 6 2.3.2 Classification..................................................................................................................... 6 WEB MAPPING APIS .............................................................................................................. 8 GEOSPATIAL WEB SERVICES ................................................................................................... 9 WEBGIS APPLICATIONS ....................................................................................................... 10 WEBGIS SERVER ARCHITECTURE ......................................................................................... 11 3. METHODOLOGY ..................................................................................................................... 14 DESIGN-SCIENCE RESEARCH ................................................................................................. 14 STRATEGY IMPLEMENTATION ............................................................................................... 15 4. PROPOSAL ................................................................................................................................ 17 ASSUMPTIONS ...................................................................................................................... 17 4.1.1 Software decision ............................................................................................................ 17 4.1.2 Third party services ......................................................................................................... 17 4.1.3 Architecture decision....................................................................................................... 18 ARCHITECTURE .................................................................................................................... 18 CASE STUDY AND APPLICATIONS .......................................................................................... 20 4.3.1 Studmap 3.0 ..................................................................................................................... 21 4.3.2 Area of interest ................................................................................................................ 21 vii 4.3.3 Data of interest ................................................................................................................ 22 4.3.4 Architecture Implementation ........................................................................................... 26 4.3.5 User roles and privileges ................................................................................................ 36 5. DISCUSSION ............................................................................................................................. 39 DATA.................................................................................................................................... 39 SYSTEM FUNCTIONALITY ...................................................................................................... 40 SYSTEM USABILITY .............................................................................................................. 43 5.3.1 Expert evaluation ............................................................................................................ 43 5.3.2 User testing ..................................................................................................................... 44 EASE OF IMPLEMENTATION .................................................................................................. 47 FUTURE DEVELOPMENTS ...................................................................................................... 48 6. CONCLUSION ........................................................................................................................... 50 BIBLIOGRAPHIC REFERENCES .................................................................................................. 52 APPENDIX .......................................................................................................................................... 60 viii INDEX OF FIGURES Figure 1 ArcGIS Server site architecture adapted from (ESRI, 2018b)................................................ 12 Figure 2 ArcGIS Enterprise multi-machine model adapted from (ESRI, 2018f) .................................. 13 Figure 3 Design research process adapted from (Kanellis & Papadopoulos, 2009) ........................... 14 Figure 4. Design-science research framework implementation ............................................................ 15 Figure 5 Conceptual architecture for the integration of geodata on a web-GIS platform – general diagram ................................................................................................................................................. 19 Figure 6. Area of interest. ..................................................................................................................... 22 Figure 7 LAS File properties and statistics .......................................................................................... 25 Figure 8 General diagram of the architecture implementation for Studmap 3.0 .................................. 26 Figure 9 Feature service error when exceeding feature limit on a web map (left) or web scene (right) .............................................................................................................................................................. 28 Figure 10 Studmap 3.0 Portal interface - Find data section ................................................................ 30 Figure 11 Studmap 3.0 Portal interface - Applications section ............................................................ 30 Figure 12 Studmap 3.0 Portal interface – Data categories section ...................................................... 31 Figure 13 Studmap 3.0 Portal interface - Results by tag filter ............................................................. 31 Figure 14 Accessing the geospatial services from IVVGEO Server using ArcGIS Pro ........................ 32 Figure 15 Creating the studmap_prod enterprise geodatabase ............................................................ 34 Figure 16 Create a connection to the production database using ArcGIS Pro (left) and PostgreSQL (right) .................................................................................................................................................... 34 Figure 17 The ArcGIS Data Store configuration wizard - installed databases .................................... 35 Figure 18 Creation of the studmap_creator user role using the Create Database User tool ............... 37 Figure 19 Boxplot to measure the effectiveness of the user study ......................................................... 46 Figure 20 Boxplot to measure the efficiency of the user study .............................................................. 46 Figure 21 Boxplots of the SUS scores computed for the expert and student evaluations ..................... 47 6 technique in order to avoid the examination of every row and record. However, 3D data imposes a special data structure and relationships between entities, which, in turn, challenges the spatial indexing system whose solution is still under research. In their research, van Oosterom, et al. (2015) discuss the capabilities and limitations of diverse infrastructure solutions available for the manipulation of massive 3D point clouds within a Database Management System (DBMS). Their research reveals that the best implementation strategy depends on the file size, hardware and software solutions available and the required functionality. On the other hand, solutions for converting complex 3D data formats into easily manageable structures are being researched. The concept to compress the entire content of a dataset allows working with great volumes of data while being able to access each individual resource. Indexed 3D Scene layers (I3S) and, respectively, Scene Layer Packages (.SLPK) have been adopted as an OGC community standard for storing and managing 3D geographic data such as 3D objects, integrated meshes and point features (OGC, 2017). This solution focuses on converting the original dataset into a node-based hierarchical spatial index structure to allow the packaging for local storage, in the case of .SLPK package and consumption as a web service, in the case of the I3S layers (ESRI, 2017b). Hence, no DBMS approach is required, nor currently available in this case. GIS SERVERS 2.3.1 Defining concepts GIS data is handled on the web through different applications or programs depending on web browsers, web servers, and map servers to perform certain tasks over the internet. The web browser is the client as it is the one to make a request translated into an URL, while the web server gives the response to that request through the HTTP protocol. The geospatial component is handled by the map or GIS server which provides the core GIS functionality of the application and supports the derived web maps (Agrawal & Gupta, 2017). The link between the web server and map server is done by the application server (Shekhar, Xiong, & Zhou, 2018). 2.3.2 Classification GIS servers are handled with the help of different software options available. In terms of acquisition costs, they can be either open or closed. Idrizi (2014) offers an analysis of the comparison between the two types of software for GIS practices. A. Open source software servers Open source movement in GIScience gathers a valuable community of dedicated peers that contribute with solutions to so-called “fee of costs” platforms. The efforts have gained value among the research area as Gaghegan (2018) argues that GIS platforms can develop in a healthy way if people contribute 7 freely. However, open source software comes with a series of drawbacks and are free only to a certain extent, even though the price cannot compare with the commercial options on the market (Idrizi, 2014). Agrawal & Gupta (2017) offer a review of the open source software options and their impact. Among the open source software options, the ones that are gaining popularity nowadays are GeoServer (2019) and MapServer (MapServer, 2018). An example of an entirely open source geoportal is istSOS (Cannata, Antonovic, Molinari, & Pozzoni, 2015), which combines free software solutions into a web map mashup to serve for flood protection. B. Closed source software servers Commercial software is intended to be user-friendly, requiring little development knowledge and training, while offering good infrastructure support for the client. Usually, closed source software options can be adapted to the users’ specific needs and require fewer maintenance efforts (Idrizi, 2014). What is more, Gaghegan (2018) argues that commercial software is bringing the GIS community one step closer to contributing to a common platform for research in the realm, as producers are nowadays giving the users the opportunity to develop or improve missing functions of the available systems. However, all these benefits come in the exchange of a considerable price and such compromises as adapting to the producer’s specifications and constraints (Idrizi, 2014). A recent review of the options of software available on the market is given by Idrizi (2014). They also offer a comparison between the open source and commercial software options for GIS servers, providing a list of examples of implemented GIS platforms for each case. Among the most popular options of commercial GIS software stands ArcGIS, produced by ESRI. ArcGIS is a GIS dedicated software platform composed of four elements: a geographic information model, storage and management component, disseminating geoinformation, GIS applications, and related web services. The platform allows personalization and development of the built-in features or the creation of new applications or components (Shekhar, Xiong, & Zhou, 2018). ArcGIS Server is a web mapping and geoprocessing software of this platform, which enables the availability of spatial information through the communication between a server computer and other devices established by different web services. This communication is standardized by two protocols: HTTP and HTTPS. The server consists of two servers, a map and a web one, which is connected via APIs (ESRI, 2018b). An example of an ArcGIS driven webGIS platform is given by Walsh, Page, McKnight, Yao, & Morrissey (2015) which have implemented a solution to offer the availability of the LiDAR driven digital terrain model of North Carolina. Chen, He, Zhang, & Nover (2016) have also used the option of ArcGIS Server to combine the 2D and 3D data on the same webGIS platform to ease the management of landslide hazards. More popular examples consist of the National Atlas of the United States (USGS, 2018), the Canadian GeoData Infrastructure (Canada, 2017) and the National Geographic Website (Society, 2017). 8 WEB MAPPING APIS Application Program Interfaces or APIs are software interfaces that specify how pieces of software interact with each other. They can be either consumed or provided as services and they come in two main categories: server-side APIs and client-side APIs. For the communication with the HTTP web server, the web APIs usually use JSON or XML standard formats. For communication with the web browser, the APIs use standard JavaScript formats (Liu, Li, Huang, & Gong, 2015). Web mapping APIs are dedicated to mapping as they are used to establish the communication and integration of web map services through classes of maps and layers. Currently, there are over 20405 web APIs, which include 1112 web mapping APIs, according to ProgrammableWeb (2018). Among the most popular, stands Google Maps, which holds the 8th position among all types of APIs. In their study, Liu, Li, Huang, & Gong (2015) give a review of the popular GIS mapping API choices. Slight changes have been made since which are reflected in the updated version of the selected GIS web mapping APIs in Table 1: Provider URL Authentication Google Maps https://cloud.google.com/maps-platform/ No ESRI ArcGIS https://developers.arcgis.com/ No Microsoft Bing Maps https://www.microsoft.com/en-us/maps/choose-your-bingmaps-api Yes Here Maps https://developer.here.com/ Yes MapQuest https://business.mapquest.com/products/ No OpenLayers http://openlayers.org/ No Yandex https://tech.yandex.com/maps/ Yes OS OpenSpace https://www.ordnancesurvey.co.uk/business-andgovernment/products/os-openspace/api/ Yes GeoServer https://docs.geoserver.org/latest/en/user/rest/api/index.html No GeoTools http://docs.geotools.org/stable/userguide/library/api/ No Table 1 Updated table of selected GIS web mapping APIs (Liu, Li, Huang, & Gong, 2015) (sites last accessed on November 5, 2018) All these web APIs are dedicated to allowing developers to create or customize built-in location-based applications for modern platforms. ESRI’s ArcGIS web APIs are offered for a wide variety of programming languages, including JavaScript, Flex, Silverlight/WPF, .NET and Java (Liu, Li, Huang, & Gong, 2015). The ArcGIS API for JavaScript is now enabling users to integrate 2D and 3D functionality in their webGIS based applications and deploy them on the web browser (ESRI, 2018j). The 3D functionality on the web was given with the release of the 4.0 version of the API through the Indexed 3D scene layers (I3S) capable to support large volumes of 3D geospatial data (ESRI, 2016b). For non-developers, the ArcGIS API for JavaScript is integrated with the Web AppBuilder (WAB) software development kit (SDK), which requires minimal coding and allows easy creation of GIS web applications. This SDK comes with two versions, one for 2D applications and one for 3D, which are based on the two versions of the API, 3.x and 4.x (ESRI, 2018j). 9 The representational state transfer or REST API that uses HTTP requests in a common architecture for web services development (Rouse, 2016). ArcGIS Online and Server use this type of API for the management of web services, system properties and user roles (ESRI, 2017d). GEOSPATIAL WEB SERVICES While APIs represent the interface between two applications, the web services grant the communication between two machines. Web services are software or pieces of software that facilitate the availability of data the Internet through a standardized Extensible Markup Language (XML) which encodes the messages between the users and the services. Web services can be compatible with the characteristics of geospatial data and can hence, allow the access and server-side processing of such data via the Internet. In this way, geospatial technologies can take advantage of the capabilities of web services to offer accessibility of large volumes of multi-dimensional geodata to clients varying from human to application end-users focused on geospatial services (Veenendaal, Brovelli, Li, & Ivánová, 2017). Web geospatial services enable the distribution and real-time processing of big geospatial data to reduce time, costs, duplicated data and required computing capacity. In this context, geospatial data analysts can now take a step forward from the traditional GIS workflows to efficiently explore geospatial information directly on the web without such dependencies as storage space and computing capabilities (Wagemann, Clements, Figuera, Rossi, & Mantovani, 2018). A. OGC-specific web services New directions in the realm are now focused on finding the best techniques to combine data access and analysis throughout web-based services. In this context, the integration of multi-source web services is a great challenge which comes with a series of disadvantages (Chen, He, Zhang, & Nover, 2016) (ESRI, 2018i). Agrawal & Gupta (2017) refer to the need to effectively implement these services as the need for a “revolution”. The Open Geospatial Consortium (OGC) responds to this urgency by being dedicated to initiating the standardization of the geospatial data sharing and interoperability (OGC, 2016b). According to GeoSeer (2018), at 2018-07-18 there were 1224379 OGC layers available. Their standard classifies the web services in web map services (WMS), web map tiling services (WMTS), web feature services (WFS) and web coverage service (WCS). OGC describes the web map service (WMS) as a standard to share maps as georeferenced digital images in different formats (Agrawal & Gupta, 2017). OCS’s definition of WMSs (OGC, 2016a) covers the standards of requesting and retrieving online maps from existing geodatabases to allow the dynamic consuming of multiple maps or layers coming from multiple sources and allowing different styling schemes. The service uses the GetCapabilities and GetMap API calls at a basic level. ESRI (2014c) specifies that WMSs don’t expose actual data while allowing attribute requests throughout other operations. 10 According to the OGC standards, a web map tiling service (WMTS) is a complement for the WMS as it again represents an image covering a map extent for different scales. The difference is that the image is composed of multiple predefined tiles that ease the load of computation needed for large images (OGC, 2010b). An example of WMTS is the basemaps provided by ESRI are tiled map services (ESRI, 2018a). Web feature services (WFS) allows the access of data by its feature and not via an image. Hence, this offers more capabilities together with a rich experience as a client can retrieve or modify features and their properties on the web directly. However, this comes with a considerable cost in terms of response time as the rendering is done at the client-side (OGC, 2010a). A. ESRI-specific web services ArcGIS Server is compatible with many web services, including the OGC compliant ones mentioned earlier. There are 7 services managed by the GIS Server in particular: geocode, geodata, geoprocessing, image, map, scene, and vector tile services. These services can be consumed either by using a web browser or a web mapping application or using a GIS desktop application (ESRI, 2018d). Raster data or mosaics can be published via image services. In an Enterprise setting, these will generate either image or elevation layers depending on the settings of the source. Map services allow the publishing of maps over the Internet and it creates the map image layers. These layers are normally rendered on the server-side generating a low response size. Map services also allow enabling WMS, WMTS or WFs and hence allow the publishing of OGC specific layers (ESRI, 2018e). Feature data is normally published as a feature service which is a feature-enabled map service. Once published, these services will generate feature layers that offer enhanced capabilities and an interactive experience compared to the map image but at a higher response size (ESRI, 2018e). Scene services allow the publishing of complex 3D data structures or scenes. For example, scene layer packages can be consumed via these services (ESRI, Indexed 3D Scene Layers, 2015). These services can only be published as hosted layers as their content is stored in the NoSQL database of ArcGIS Data Store (ESRI, 2018e). WEBGIS APPLICATIONS Altogether with the growth of webGIS, the diversity of web mapping APIs and standardization of geospatial web services approaches to integrate emerging technologies into web mapping applications are setting the base for a new mashup paradigm. Data coming from different sources are being waved into modern, light-weighted web applications to offer their users a more interactive mapping experience (Liu, Li, Huang, & Gong, 2015). As users tend to integrate web programming technologies into their maps, web developers are offering more intuitive ways to build web mapping applications. Configuring applications now becomes easier 11 not only for developers but for non-programming peers as well. Among the plethora of tools, APIs and libraries, Buczkowski (2016) bring forward a classification of the most popular choices for web mapping. A good example of an easy-to-use web application building software is given by ESRI’s WAB for ArcGIS. WAB allows both developers and non-developers to create interactive mobile or desktop web mapping applications. The application can either be deployed using ArcGIS Online or Enterprise platforms or using a Developer Edition dedicated for customization (ESRI, 2014b). What is more, WAB can also integrate 3D data via the newest version of the ArcGIS API for JavaScript (ESRI, 2014a). Stepping forward into the developments of web maps and applications realm also brings important issues to the surface. As tendencies are moving around 3D data, a current concern among web mapping developers is how 3D features can reach the same level of integration as 2D web maps in order to be used together for an enhanced webGIS environment (Chen, He, Zhang, & Nover, 2016). WEBGIS SERVER ARCHITECTURE GIS client-server architectures are network architectures dedicated to spatial data in which a web server works altogether with a map server to exchange geographic and non-geographic information with remote clients. Agrawal & Gupta (2017) discuss the multiple possible options for a webGIS architecture, which can vary between multi-layered, plug-and-play, service-oriented or cloud-based. In general, an internet GIS architecture tends for a three-tiered system consisting of a presentation, an application and a database tier (Shekhar, Xiong, & Zhou, 2018). The presentation layer is the web browser that grants the exchange of information between the client and the web server. The application tier holds the logic of the entire system, ensuring the exchange of information between the client and the database. In the case of webGIS, this is done by the web and the GIS servers via the Internet. The database tier handles the storage and management of the geospatial data in order to return the requested data to the web server (Agrawal & Gupta, 2017). In a three-tiered architecture, each component can physically reside on a different machine. This brings, therefore, both complexity and valuable gains over the traditional approaches, such as increased efficiency, security, scalability and flexibility as the workload is distributed between the different components and hence, between different computers. Such systems can access the database directly, without the need to locally store the data, providing the client with fast responses of complex features and even geospatial analysis tools residing on other systems (Shekhar, Xiong, & Zhou, 2018). Some examples of applied three-tiered architectures are given by Chen, He, Zhang, & Nover (2016), Liu, Li, Huang, & Gong (2015) and Shekhar, Xiong, & Zhou (2018). ESRI’s ArcGIS Server can also be deployed as a multi-layered system, where the web browsers that offer access to the clients and administrators form the presentation tier, the web adaptor, and web and 12 GIS servers represent the application or middleware tier and the database layer is represented by the data server (Figure 1). In this case, the web adaptor is the application which links the GIS server or servers with the web server, by forwarding the requests to the machines where the GIS servers reside (ESRI, 2018b). The data structure for GIS data management that powers web services and allows web editing on the ArcGIS Server is represented by the enterprise geodatabase. It is a relational database management system (RDBMS) with special characteristics destined for a large-scale GIS system: increased efficiency, data integrity, scalability, concurrent access, security and flexibility (ESRI, 2016a). Once registered with the GIS server, this structure allows referencing the source path of the data that is being published and turned into web services (ESRI, 2018c). Figure 1 ArcGIS Server site architecture adapted from (ESRI, 2018b) An advanced deployment is represented by the ArcGIS Enterprise multi-machine system (ESRI, 2018f) which offers a configuration of three main components that can reside on different machines: the ArcGIS Server, the Portal for ArcGIS and the ArcGIS Data Store (Figure 2) to grant the feature service data management. The system requires that all three components work together. The ArcGIS Server Site is federated and integrated with the portal. This system is configured with two data stores which deploy the hosting server: the relational and the non-relational or tile cache data store. While the relational database handles the hosted feature services managed in the portal, the non-relational manages hosted scene services. This configuration offers the highest level of integration of the system 13 components and, hence, the most capabilities availability, at the moment being the only means of publishing complex 3D features via the I3S standard layer (ESRI, 2018g). Figure 2 ArcGIS Enterprise multi-machine model adapted from (ESRI, 2018f) In this architecture model (Figure 2) the communication between the portal and the GIS server is done via the HTTP and HTTPS protocols or can be restricted to HTTPS only. As Figure 2 displays, the portal communicates with the 7080 port for the HTTP protocol and with the 7443 port for the HTTPS (ESRI, 2018l). The GIS server, on the other hand, uses the HTTP port 6080 and the HTTPS port 6443 to communicate with the other machines via the Internet (ESRI, 2016c). The Web Adaptor manages the requests via the 80 and the 443 ports. 14 3. METHODOLOGY The goal of the research is to integrate complex geospatial data and web-based services on an interoperable web platform by proposing and evaluating a GIS client-server architecture. In other words, the proposed approach refers to the creation of an artefact aiming to solve a problem. In this context, a design science research framework (Kanellis & Papadopoulos, 2009) is chosen from the existing research methodologies as the proposed approach intends to build knowledge through the creation of reality. This chapter focuses on describing the concept of design science research and its application to the current research. DESIGN-SCIENCE RESEARCH The paradigm of Design-Science is a research instrument commonly used in Information Sciences to solve problems by creating an artefact. Design-Science Research proposes a framework adapted by the researcher’s aims to shape the artefact according to the results of a series of iterations of five processes as seen in Figure 3: the problem’s awareness, the suggestion, the development, the evaluation and the conclusion (Kanellis & Papadopoulos, 2009). Figure 3 Design research process adapted from (Kanellis & Papadopoulos, 2009) First, the problem (Figure 3) is shaped and addressed by the researcher. Based on empirical knowledge and accumulated experience, solutions to the problem are proposed in the suggestion phase. The suggestions are being pushed forward to the development process when an artefact is created according to the theory. The developed artefact is evaluated according to the suggestions proposed and all findings are cycled back to the problem’s awareness and suggestion, to re-develop and re-evaluate the artefact. The process terminates with the conclusion and the knowledge is built through the two flows: Circumscription and Operation and Goal Knowledge (Kanellis & Papadopoulos, 2009). 15 STRATEGY IMPLEMENTATION According to the design science research paradigm theorized previously, the methodology of the current research adopts the five phases of the framework proposed by Kanellis & Papadopoulos (2009) and adapts them to the main objectives: proposing, implementing and evaluating of a new client-server architecture for complex GIS data integration on a common, student-specific web platform (Figure 4). Figure 4. Design-science research framework implementation As Figure 4 implies, each phase of the design-science research framework reflects one implementation step: 1. The awareness of the problem phase is presented at the beginning of the paper, by stating the two main objectives of the research: the proposal and implementation of a client-server architecture for complex GIS data and web services integration. 2. The suggestion phase reflects in the solutions proposed by the existing GIS integration solutions and the empirical experience gained through the operation and goal knowledge discovery. 3. The development phase is the construction of the artefact based on the theories condensed by the suggestion phase. In the current research, the artefact is represented by the proposal and design of a client-server architecture to serve for the integration of the motivated data and services in a common webGIS platform. 4. The evaluation consists of the practical implementation of the proposed architecture and validation of its functionality throughout the student dedicated platform. Studmap 3.0. A study case emerges at this point to test the theoretical concepts attained and the feasibility of the conceptual architecture proposed previously. The outcome of this phase is feed back to the 22 Figure 6. Area of interest. 4.3.3 Data of interest The data of interest (Table 2) for the implementation of Studmap 3.0 comes from three main sources: the Studmap 1.4 Portal, the newly generated data from the students’ UAS2018 projects and the thirdparty services (OpenNRW, 2017) (IVVgeo, 2018). The data comes in four different coordinate systems: 25832 (ETRS 1989 UTM Zone 32N), 32632 (WGS 1984 UTM Zone 32N), 4326 (GCS WGS 1984) and 3857 (WGS 1984 Web Mercator Auxiliary Sphere). While all items were stored with their original reference system, all projections were handled on-the-fly, while using ArcGIS Pro for the publication of the web services. Moreover, there are six different format types: .shp, .tif, .slpk, WMTS, WFS and WMS. 23 No. Source Reference Name Format EPSG 1 Studmap 14 studmap_prod.sde Uncut areas NRW Studmap 14 .shp 25832 2 UAS2018 studmap_prod.sde Centerline UAS2018 .shp 32632 3 UAS2018 studmap_prod.sde Flight path UAS2018 .shp 32632 4 UAS2018 studmap_prod.sde Flight path points UAS2018 .shp 32632 5 UAS2018 studmap_prod.sde Land cover CORINE dissolve UAS2018 .tif 32632 6 UAS2018 studmap_prod.sde Land cover dissolve UAS2018 .tif 32632 7 UAS2018 studmap_prod.sde Land cover UAS2018 .tif 32632 8 UAS2018 studmap_prod.sde DEM UAS 2018 .tif 32632 9 UAS2018 studmap_prod.sde Elevation UAS2018 .tif 32632 10 UAS2018 studmap_prod.sde LiDAR stations UAS2018 .shp 32632 11 UAS2018 studmap_prod.sde Sensebox Stations UAS2018 .shp 32632 12 UAS2018 studmap_prod.sde Air Quality UAS2018 .shp 4326 13 UAS2018 studmap_prod.sde Study area UAS2018 .shp 32632 14 UAS2018 pc.slpk Point cloud UAS2018 .slpk 32632 15 UAS2018 http://ivv6.maps.arcgis.com NDVI UAS2018 WMTS 32632 16 third-party services http://ivv6.maps.arcgis.com Image composition UAS2018 WMTS 3857 17 third-party services http://ivv6.maps.arcgis.com Orthophoto Multispectral UAS2018 WMTS 3857 18 third-party services http://ivv6.maps.arcgis.com Orthophoto RGB UAS2018 WMTS 3857 19 third-party services http://ivv6.maps.arcgis.com Aspect UAS2018 WMTS 3857 20 third-party services http://ivv6.maps.arcgis.com Slope UAS2018 WMTS 3857 21 third-party services http://ivv6.maps.arcgis.com Hillshade UAS2018 WMTS 3857 22 third-party services https://open.nrw Official German basemap of NRW 1:5000 WMS 3857 23 third-party services https://open.nrw Digital Orthophotos NRW 10cm (Metadata, RGB, CIR, NIR) WMS 3857 24 third-party services https://open.nrw Landscape Information Collection of NRW WMS 3857 25 third-party services https://open.nrw Regional plan of NRW WMS 3857 26 third-party services https://open.nrw Topographical basemap of NRW 1:5000 WMS 3857 27 third-party services https://open.nrw Topographical map of NRW 1:25000 WMS 3857 28 third-party services https://open.nrw DTM Relief of NRW WMS 3857 29 third-party services https://open.nrw DTM Slope of NRW WMS 3857 30 third-party services https://open.nrw DTM Isolines of NRW WMS 3857 31 third-party services https://open.nrw Administrative boundaries of NRW - District municipalities &… WFS 3857 32 third-party services https://open.nrw Administrative boundaries of NRW - Districts & country-level cities WFS 3857 24 33 third-party services https://open.nrw Administrative boundaries of NRW - Administrative districts WFS 3857 34 third-party services https://open.nrw Administrative boundaries of NRW - Border WFS 3857 Table 2 Data of interest 4.3.3.1 Studmap 1.4 data Studmap 1.4 Portal is the former and currently unfunctional GIS platform of the University of Münster. The portal used to encompass data of regional interest to the students, such as web-based map services and geographic data mostly in the common .tif and .shp formats. The structure of the geoportal is described by the Studmap 1.4’s wiki page (IVVGeo-Team, 2018), which offers details related to the functionalities, external WMS services, post-installed packages, server configuration and web client communication. Some of the main features of the Studmap 1.4 portal was to allow the upload of .gpx and .kml file formats and the download of the .tif, .shp and .kml file formats. Besides these, it included a tool to restore the application state and a search button. For the user interface, the map included a measuring tool, the context menu and a viewport layout, altogether with a built-in list of external web map services. Among the classic functions, it included the zoom slider, a graticule tool, a maximum extent button, the layer list, and the select feature tool. Prinz (2014) gives a visual insight into the Studmap 1.4 portal, through a series of screenshots and explanations on the workflow of the GPS (global positioning system) data transfer to a GIS. The data and functionalities of the old Studmap 1.4 platform were investigated in order to offer a selection of relevant data and tools to be included on the newly implemented portal. As any backup files to restore the old database into a new one was lacking, no data migration from one system to another was possible. Hence, all selected data were first organized in a new geodatabase consisting of six feature datasets: biotope cadaster, fauna-flora-habitat, legally protected biotope, nature conservation area, composite surfaces, borders, and activity areas. Each dataset was analyzed for its corresponding online source, using the available list of web servers and their updated versions. One feature that had no online correspondent was found and stored in the initial development geodatabase: the uncut areas of North Rhine-Westphalia, representing the valuable ecological areas uncut by any form of infrastructure. 4.3.3.2 New 3D data of TLS and/or UAV The new data is represented by the recent project deliverables of the students of the WWU university during the 2018 Unmanned Aerial Systems (UAS) class. The class objective is to process and analyze data captured via UAVs (Unmanned Aerial Vehicles) or TLS (Terrestrial Laser Scanning) techniques which always comes in 3D. In this time these data are produced in large amounts every year and lack an efficient way of storage, sharing, and presentation. The data consists of mostly raster and vector datasets, in three main formats .shp, .tif and the newly included file format for the LIDAR data captured 25 via TLS, .las. Moreover, the new data that was already published online by the students of the university comes in the format of tiled map services. The most challenging dataset included in the project consists of a small-size point cloud file in the standard .las format. The file properties and statistics of the file are presented in Figure 7. As observed in the figure, the file consists of approximately 33.5 million points, with a size of 667 MB. For easier processing, the point cloud was converted in a Scene Layer Package (.slpk) using the Create Scene Layer Package Geoprocessing Tool, resulting in a light file of only 61,4 MB. The pc_uas2018.slpk file was locally stored on the same machine that holds the databases as this type of file currently doesn’t allow storage in a database. Figure 7 LAS File properties and statistics 4.3.3.3 Third party services The data coming from the third-party servers is represented by the web services of regional interest to the students of the WWU University of Muenster, to which the portal is dedicated. Hence, the web services selected for this study come from two sources: the official GIS servers of the state of North Rhine-Westphalia (OpenNRW, 2017) and the cloud-based platform of the Institute for Geoinformatics of the University of Münster (IVVgeo, 2018). The services consist of three relevant types, WMS, WFS and WMTS and are selected to satisfy the needs of the students according to the availability offered by the local data providers of North Rhine-Westphalia, to the list of web services available in the wiki page of Studmap 1.4 (IVVGeo-Team, 2018) and to the recently stored projects on the cloud-based platform of the Institute of Geoinformatics. 26 4.3.4 Architecture Implementation 4.3.4.1 General aspects Figure 8 General diagram of the architecture implementation for Studmap 3.0 As the schema in Figure 8 reflects, the data is generated from three different sources: the old portal, Studmap 1.4, the recent projects that generated new 2D and 3D data and the third-party servers. The data coming from the old portal and the recent projects are stored in a common geospatial database which is further cloned for production. One type of data files is, however, stored locally in a converted format, SLPK as this type of data structure doesn’t allow storage in a database. The data coming from the third-party servers is represented by the web map and feature services of the state of North RhineWestphalia and is hosted by the state’s geodatabase servers. The emphasis of this structure stands on the most challenging components to be integrated into the new platform: the different web-based services and the big data represented by the complex 3D point clouds. The creation of the web-based services that were set on the old platform, Studmap 1.4, includes primarily map services (WMS), feature services (WFS) and map tile services (WMTS) and APIs for the use of the third-party applications. What is more, an approach will be developed to migrate the old datasets into the new system and to re-develop the map services and APIs for the current and future third-party applications to integrate with the underlying infrastructure. On the other hand, handling big data in the geodatabase refers to finding the best approach to integrate heavy point cloud data that comes usually in the LAS format. 27 The components of the proposed architecture are distributed across different machines to increase computing power, as per Table 3: System component Machine reference Web server with web adaptor IVVGEO-STUDMAP.uni-muenster.de GIS Sever (ArcGIS Server) IVVGEO-STUDMAP.uni-muenster.de Data Store (SQL and NoSQL) IVVGEO-STUDMAP.uni-muenster.de Web server with web adaptor ivvgeo-portal.uni-muenster.de Portal for ArcGIS ivvgeo-portal.uni-muenster.de Table 3 Components’ distribution The implemented web portal has a hybrid system architecture at all three component levels. The presentation tier consists of desktop GIS applications (ArcGIS Map and ArcGIS Pro) and web applications (Portal for ArcGIS, ArcGIS Server Manager) or REST Admin APIs (ArcGIS Portal Directory, Portal Administrator Directory, ArcGIS Server Administrator Directory) that allow the users and administrators to interact with the IVVGEO portal and server. As for the application layer, it consists of the two web servers holding the logic for both the portal and the server and the IVVGEO Server itself. The database tier consists of two relational data stores (a PostgreSQL enterprise geodatabase and the Managed ArcGIS Data Store) and one non-relational data store (the Tile Cache ArcGIS Data Store). 4.3.4.2 The presentation tier The presentation layer represents all the interfaces that the users can access to interact with the IVVGEO portal and GIS server. The access to the different interfaces can be done according to the users’ roles. Hence, regular roles, such as publishers, viewers or users normally operate with the desktop GIS applications and web applications, which tend to be more user-friendly. The administrators usually interact with the system using the REST Admin APIs. Since this study is intended for the development of a webGIS platform to serve the students for their necessities, the discussion is further focused around the main interface of the system, namely the Studmap 3.0 Portal. The Studmap 3.0 Portal, also referred as IVVGEO, is currently accessible via the REST Admin APIs for administrators’ access and the Portal for ArcGIS web application for all user roles. The web application can be accessed using the web-browser via the following URL: http://ivvgeo-portal.geo.unimuenster.de/portal (site last accessed on February 7, 2019). Up to now, the portal components are organized in different ways: by folders, categories, item types, date modified, date created, shared and status. Items falling into the same category were given a common tag using the name of that category. The content of the Studmap 3.0 Portal is divided between web layers organized in web maps, web scenes and web applications. At this time, there are 46 web layers hosted on the portal divided in seven types of layers and hosted on three main sources as shown in Table 4: 28 The layers are grouped in two web maps, NRW 2Dmap and UAS 2Dmap and two web scenes, NRW 3Dmap and UAS 3Dmap according to their content and relevance. As seen in Figure 9 one layer was not added to any of the enumerated web maps and scenes, namely the “Uncut Areas of NRW Studmap 1.4” as it contains more features than the map extent is able to draw (ESRI, 2018n). Figure 9 Feature service error when exceeding feature limit on a web map (left) or web scene (right) A GIS web application was created for each map using WAB: NRW 2Dapp, NRW 3Dapp, UAS 2Dapp and UAS 3Dapp. All applications’ interfaces are displayed in Appendix A. Each application was configured with personalized widgets (Table 5) according to its purpose and the limitations of the API specific for each type of application, 2D or 3D. Due to the limitations of the application building SDK, capabilities like adding data, editing data, select, query are only available on the 2D applications. Moreover, according to an incompatibility of the developer’s edition of the application builder with the WFS services required, the applications were finally built using the built-in tool of the Portal for ArcGIS, which doesn’t allow customizations for the applications. For a better visualization of the functionality of the main widgets of the applications, they are presented in Appendix B. UAS 2Dapp Studmap 3.0 2Dapp UAS 3Dapp Studmap 3.0 3Dapp Attribute table x x Coordinate x x x x Full screen x x x x Scalebar x x Zoom slider x x x x Search x x x x Home x x x x My location x x x x Basemap gallery x x x x Layer list x x x x Legend x x x x Add Data x x Edit x Query x Type Tile Map image Feature Imagery Scene WMS WFS Nr. 7 12 12 1 1 9 4 Source IVVGEO cloud IVVGEO MapServer IVVGEO FeatureServer IVVGEO ImageServer IVVGEO SceneServer NRW geoserver NRW geoserver Table 4 Web layers types and distribution 29 Select x Measurement x x x x Share x x x x Compass x x Navigate x x Table 5 Widgets configured for each web application of the Studmap 3.0 portal The basemaps used for the web interface of each map and, hence, application consist of the standard tiled map services granted by ESRI. According to Kong, Zhang, & Stonebraker (2015), users tend to favor a familiar, neat and easy to interpret map as a base for their applications. Therefore, the basic Grey Scale basemap offered by ESRI was chosen for the applications created to avoid the user being distracted by an unnecessary load of details. For the users that have the right permissions, one can opt for any of the available basemaps offered by ESRI (2018a) or from the list of dedicated WMS services published to the portal. To grant the ease of manipulation on the IVVGEO Portal for the local students, the Portal items were organized in a more comprehensive way using an ArcGIS Enterprise Site, Studmap 3.0 which can be accessed using: http://ivvgeo-portal.geo.uni-muenster.de/portal/apps/sites/#/studmap30 (site last accessed on February 7, 2019). The site is simple and consists of a short description of the portal, the contact information and three main sections: Find data, Applications and Data categories. The Find Data section offers an enhanced functionality to the portal as users can easily navigate through the data available on the portal either by text or by location. This section is the first one to appear on the Site, together with the description and name of the portal (Figure 10). The Applications section gives access to the 4 main applications of the portal (Figure 11). The Data Categories section is designed to allow students to find the available web layers by sorted categories and is followed by the contact information (Figure 12). This section selects the layers by tag query and hence, each tag that fits to a certain category will be selected for that query (Figure 13). The full extent of Studmap 3.0 Site is presented in Appendix C. 30 Figure 10 Studmap 3.0 Portal interface - Find data section Figure 11 Studmap 3.0 Portal interface - Applications section 31 Figure 12 Studmap 3.0 Portal interface – Data categories section Figure 13 Studmap 3.0 Portal interface - Results by tag filter Students can also benefit from accessing the items available in their portal account by using a desktop GIS application, ArcGIS Pro, by adding the portal connection via the Portal’s URL: https://ivvgeoportal.uni-muenster.de/portal (last accessed on February 7, 2019). Students with publishing rights are also allowed to publish items on the Studmap 3.0 Portal using the ArcGIS Pro application or the portal’s original interface. Using either the ArcGIS Pro or ArcGIS Map applications, users are also allowed to access the services on the IVVGEO Server by creating an ArcGIS Server connection between their local machines and the remote one holding the GIS server application (Figure 14). 38 GRANT SELECT ON ALL TABLES IN SCHEMA studmap_creator, public, sde TO studmap_viewer; GRANT SELECT ON ALL TABLES IN SCHEMA studmap_creator, public, sde TO studmap_editor; GRANT UPDATE ON ALL TABLES IN SCHEMA studmap_creator, sde TO studmap_editor; GRANT INSERT ON ALL TABLES IN SCHEMA studmap_creator, sde TO studmap_editor; 39 5. DISCUSSION This chapter offers the final discussion on the benefits and limitations of the proposed and implemented client-server architecture. The comparison was done in terms of different criteria: data, system functionality, system usability and ease of implementation. The first three criteria were specified according to the trends identified by Veenendaal, Brovelli, & Li (2017) and focus on the technical objectives of this study: the implementation and evaluation of the proposed architecture in terms of complex data integration and functionality according to the students’ needs. The latter focuses more on a personal approach to validate the overall implementation of the entire system. As specified in Chapter 3, the measure for the termination of the design research process proposed for this study is the ultimate usability evaluation of the artefact, namely the implemented portal. This process consisted of two phases: a customized expert usability evaluation followed by a standardized user test. The first phase was an inspection of the interface of the portal followed by a customized questionnaire addressed to specialized people. The first 10 questions were designed according to the standardized SUS system (Sauro & Lewis, 2016) while the last four questions were open-ended and aiming at revealing the experts’ opinion on the strengths, limitations and recommendations for the portal. The second phase consisted of a set of 11 tasks followed by the SUS questionnaire and it was addressed to twenty students. The chapter finally addresses a section dedicated for future developments of the portal based on the conclusions and feedback drawn from this study. DATA The proposed webGIS architecture was evaluated for the data completeness using the implemented model of the Studmap 3.0 webGIS portal according to the three types of sources included in the implemented model: data coming from Studmap 1.4, new data and third-party services. In terms of data migration from the Studmap 1.4 portal to the newly created portal, the lack of backup files from the old system made it impossible to migrate the schema from the old database to a new one. This has led to a new approach of organizing the data and finding the updated online corresponding sources, in the form of web-based services. One feature which had no online correspondent was locally stored as part of the created and registered enterprise geodatabase. In terms of data coming from the new projects, the model has proven to be partly functional for the desired data types. While most formats are easily integrated with the system, the complex 3D point cloud data types were locally stored as the standard .slpk packages. This was due to the incompatibility between the data format allowed being stored in the database, namely the multipoint feature, and the required data format for publishing. Hence, such compromise may cause some difficulties in terms of storage and organization of data as the data of the implemented architecture is now stored in an enterprise geodatabase and a registered folder, both stored on the same machine. 40 In terms of data coming from the third-party services, out of the three web-based services proposed in the initial model, only two were able to satisfy the entire schema: WMS and WFS. The WMTS service type was proven to partially satisfy the system as only the ESRI specific layers were fully compatible with the required parameters for web services publishing as specified by the OGC standards. The data were also evaluated according to its ease of access. At this time, the students can choose to access the portal items from both the Portal’s interface or from the ArcGIS Desktop application. In this case, ArcGIS Pro was used for the publishing of the web services and for enabling the connections to the GIS Server and to the enterprise geodatabase. In this way, the end users benefit from two different options of accessing the data, depending on their familiarity with the products. The access to the tables of the enterprise database can only be done using a DBMS, such as PostgreSQL or using a desktop GIS software, such as ArcGIS Pro. Using a desktop GIS software to access the database tables comes as an asset to the system as the “studmap_prod” enterprise geodatabase is fully compatible with the ArcGIS specific geodatabase and hence, no advanced databases skills are required. For example, the student can interact with the database using basic operations such as drag and drop. SYSTEM FUNCTIONALITY The overall functionality of the implemented system is given by functions imposed by a set of nonfunctional requirements, such as performance, security and cost of equipment. The overall functionality of the implemented portal was ultimately measured using the usability evaluation described in the following section. A. FUNCTIONAL REQUIREMENTS The functional features were defined according to the eight criteria identified by Kong, Zhang, & Stonebraker (2015) and the desired capabilities of the Studmap 3.0 portal in the context of its implementation. The desired range of capabilities was specified at the beginning of this study: to allow students to explore, download, manage and upload data of regional interest in a 3D capable and smart webGIS environment. Therefore, the system was analyzed according to five main aspects, adapted from Kong, Zhang, & Stonebraker (2015) to the web applications created: basemap availability, application elements, map products, information query and location search. The first three refer to the mapping functionality, while the last refer to the database side. Compared to the old portal version, not only does Studmap 3.0 support both 2D and 3D applications, but it also allows a wider range of tools as seen in Table 8: Functional criteria Tools Studmap 1.4 2D Studmap 3.0 2D 3D Basemap availability Basemap gallery x x Application elements Scalebar x x x 41 Zoom slider x x x Pan x x Home x x Full screen x x x Layer list x x x Legend x x Measurement x x x Compass x Navigate x Add data x* x Restore app state x Graticule x Map products Save x x x Share x x x Download x x x Open in Desktop x x Information query Query x Select x x x Edit x Location search Search by location x x x My location x x Table 8 Functionality of Studmap 1.4 and Studmap 3.0 The basemap chosen for the web applications available on the portal is ESRI’s Grey Scale Basemap to meet with the general student’s expectations in terms of level of detail and organization. A student can, however, switch the basemap from the list offered by ESRI using the basemap gallery widget in the web applications or customize it using a published WMS service to set the base interface for new web maps or scenes. For the application elements, the widgets of the built-in edition of ArcGIS’s WAB were added to each application according to both the students’ needs, and the limitations given by the different types of applications. The current developer’s edition of the WAB is incompatible with the WFS services provided by the third-party servers and hence the applications were constructed with a less flexible application builder. The result was the creation of an application for each type of data: 2D and 3D. While the 2D type allows the integration of a wide range of widgets, the 3D applications have less functions and flexibility. All applications were given the traditional functionalities: scalebar, an interactive zoom slider and pan, home, full screen, layer list and legend. What is more, the measurement tool was configured for both 2D and 3D applications, to allow the students to interpret distances and area sizes easier. A compass and a navigation tool were added to the 3D applications for a better location orientation and for a dynamic interaction with the 3D environment. Enhanced functionality was given * WFS, WMTS and .shp not included 42 with the add data widget that was added on the 2D applications. Again, the widget is not available in the 3D type of applications. The widget brings value to the system as one of the main characteristics intended for the creation of the portal was to allow students to interact with the existing services on the portal while temporarily adding their own data. This functionality is also given by the basic publishing options that ArcGIS Portal has students with publishing permissions can publish services on the portal, can add them to web map or scenes, which they can incorporate in new web applications. As the creation of different applications was limited by the widgets available for each type, the information query functionality was only possible for the 2D applications. For students to understand the information better, the 2D applications were designed with such widgets as: attribute table, edit, select and query. Regarding the map products, one can easily save, share or download the web maps and applications available on the portal, with limitation based on the defined user roles hierarchy. Individual web layers can be locally accessed using the ArcGIS desktop application Location search is essential for any webGIS application and hence, has been included in all applications on the portal as well as on the portal’s interface. Using the search by location tool, the students can input the name of a place, its address or its coordinates to locate a point on the map. B. NON-FUNCTIONAL REQUIREMENTS The overall functionality of the system was limited by some the non-functional requirements: system performance and security and cost of equipment. The overall performance of the system is dependent on the resources available to support the installed components as well as on the limitations of the chosen products. Such resources as firewall settings, operating system, hardware (disk space and memory) and web browsers should be considered for a good performance of the system, especially if the current database component is intended to be expanded. Among the product limitations, two major problems were identified. First is the inability of the ArcGIS Portal-specific web maps to draw a large number of features which lead to the decision of excluding the “Uncut Areas of NRW Studmap 1.4” feature service from the web maps and scenes. The second is represented by the products offered by the third-party service providers. As an example, a simple web feature service takes more than one minute to load on an empty map. This is because the services provided by the official institution are lacking a good performance. Geospatial web services coming from external servers should comply not only with the OGC standards, but also with the general interoperability request. A closed source and hence, costly equipment (ESRI Enterprise) were preferred over the open source options on the market. This was done to allow the series of benefits the products are coming with, about the system maintenance and performance, security and available technical support. 43 To eventually ensure the proper communication between the scene cache data store and the GIS server a compromise was made resulting into a less secure configuration of the system. Compared to the conceptual model proposed at the beginning of this study, the components of the implemented system communicate using both HTTP and HTTPS protocols, together with a secure socket level certificate (SSL) allowance. SYSTEM USABILITY To ultimately reach the end of the cycle of the design methodology chosen for this research, the suitability of the platform for the specific academic use was tested using a mixed usability evaluation of the portal’s interface. The usability was counted as the implemented architecture resulted in the creation of a tool – a web-based GIS platform for academic purposes. The evaluation took place in two stages: a customized expert inspection followed by a standard user testing. 5.3.1 Expert evaluation The usability evaluation addressed to the experts was done in the following order: the development of the questions, the selection of the experts and the evaluation of the results. First, the experts were given important details about the product: what is it destined for, which are the end users and what type of items are hosted on the portal. Second, they were asked to inspect the portal’s interface in order to answer the developed questionnaire. The experts were then given the questionnaire to assess their opinion about the usability of the portal. The series of questions was developed according to the key-aspects of the implemented portal’s interface and to the SUS standardized usability questionnaire (Sauro & Lewis, 2016). The questionnaire consists of the ten standard SUS multiple-choice questions with the answer choice from “Strongly Disagree”, “Disagree”, Neutral, “Agree” and “Strongly Agree” and four customized open-ended questions relevant to the interface of the portal. Google Forms was used to collect and organize the questionnaire and its results. All questions are presented in Appendix D. The experts were selected according to their experience with the ESRI Enterprise-specific products, their geographic location, the working status and availability. Table 9 contains a short description of each expert regarding their working status and expertise criteria: Full Name Working status Organization Background Mitzi Araujo Vidal Master student in Geospatial Technologies NOVA IMS, Lisbon Portugal Geographer with Environmental Management Background; GIS Analyst responsible for Spatial Analysis in Public Security; GIS manager with experience with ESRI suite of applications and open source tools; ArcGIS Server and Portal Manager; Currently, finalizing the 44 Masters in Geospatial Technologies with focus on locational privacy protection Tiago H. Moreira de Oliveira PhD student in Information Management & GIS NOVA IMS, Lisbon Portugal Executive Coordinator for R&D national and international Projects at NOVA IMS; Coordinator of the GIS & Science master course; Member of the UNIGIS International Network; Assistant in teaching the courses of Geographic Information Systems and Science; Active participant on scientific events, Author of 45 research papers Table 9 Description of each expert chosen for the usability evaluation Finally, the results of the expert questionnaire were analyzed. The first 10 SUS standard questions were analyzed according to the standard scaling method of the system and resulted in a final score of 83.75%. Considering acceptability measures proposed by Bangor, Kortum, & Miller (2008), the experts have found the portal’s interface as acceptable, in between good and excellent. The last 4 open-ended questions reflected four main aspects of interest for the intended functionality of the portal and are summarized in Table 10. The complete set of results is presented in Appendix E. Criteria of interest Answer summary Benefits The interface is easy to use and offers good access to data and tools of interest to the students. Publishers can create their own apps, using the published layers on the portal. Drawbacks Issues with the 3D rendering were detected. Once authenticated, certain operations require a confirmation/reauthentication. The icon for the share data widget might be misinterpreted as an angle measurement tool. Improvements Improve the 3D data rendering quality, the navigation and visual capacities. Add a tool to help students understand the functionalities and tools on the portal and applications. Overall usability Students shouldn’t face usability problems according to the intended functionality of the portal. The data is easily accessible. The portal can also be considered for the public access. However, the third-party 3D data rendering issues should not be neglected. Table 10 Responses to the open-ended questions of the expert questionnaire 5.3.2 User testing The standardized user testing preceded the expert usability evaluation and took place in four stages: the development of the tasks, the development of the question, the selection of the users and the evaluation of the results. Ten straightforward tasks were designed according to the key functionalities of the portal: ability to access public data and previous projects, ability to compare own data with published services and both 2D and 3D functionality. The intention was to give the participants a heuristic approach to get 45 acquainted with the interface before answering the questionnaire. The estimated time interval of completion for all tasks was 10 to 15 minutes, with a desired 100% completion rate. All tasks were described at the beginning of the form that was sent to the participants. Similar to the expert evaluation, a standard SUS questionnaire (Sauro & Lewis, 2016) was preferred for this phase: 10 multiple choice questions and one open-ended question to ask participants about the overall opinion on the interface. The survey used Google Forms to collect the answers from the participants. All questions together with the description of the tasks are presented in Appendix F. Twenty students were selected, with ages between 23 and 31 years, all being informed about the consequences of the evaluation as volunteers. The users were selected according to the aimed audience of the portal: students that are acquainted with 2D and 3D geospatial data in a webGIS environment. Not all students that have participated in the survey have already studied in WWU. To these participants a brief explanation of the data of interest for the WWU students was given to avoid any ambiguities related to the data referred to during the completion of the tasks. Finally, the results of the expert questionnaire were analysed. The first 10 SUS standard questions were weighted according to the standard analysis. The responses to the open-ended question at the end of the questionnaire were also considered. Although the students were only asked to state general comments about the interface of the portal, the tendency was to divide their answers in different criteria, like the ones used for the expert evaluation. Therefore, the most significant answers have been grouped in four criteria as summarized in Table 11: Criteria of interest Answer summary Benefits The portal is easy to use and learn, intuitive and neat. It is easy to access existing data. The portal is helpful for the students of WWU and the applicability is clear. Both 2D and 3D functionality are covered, together with the direct data import, search data and measurement features. The data is classified in a clear way. Fast responses are given for both 2D and 3D data. Pleasant choice of background. Drawbacks The speed of response is depending on the servers’ capacities. There is a need to meet GPU criteria to run 3D applications. There is no possibility to switch from one application to another. The legend of one layer is missing. The preview of some of the layers is only displayed in a square. There are not many widgets on the applications. Students form other domains might find it difficult to use. Improvements Add the missing legends to the layers for a better interpretation of the maps. Implement a solution to switch from one application to another. Improve the preview of layers. Overall opinion The portal is appreciated by students and applicable for research, well designed and useful for students. It offers a good user experience, but also open for improvements. Table 11 Responses to the open-ended question of the student usability test 46 The tasks were generally well received by the students, all stating that they got used to the interface easily during the evaluation. All results of the test are presented in Appendix G. To measure the efficiency of the test, the time of completion for all tasks was accounted. The time fitted in the estimated interval as the average didn’t surpass 9 minutes (Figure 19). Two students took more than the estimated maximum of 15 minutes to complete the tasks. To measure the effectiveness, the accuracy of completing the tasks was measured. Two out of twenty students couldn’t complete the last three tasks (Figure 20) as they were facing the problem of incompatibility of the graphics requirements of the 3D applications based in the Scene Viewer (ESRI, 2017c). Figure 19 Boxplot to measure the effectiveness of the user study Figure 20 Boxplot to measure the efficiency of the user study 47 The final SUS score for the student evaluation is 83.87%, which, according to the acceptability rate proposed by Bangor, Kortum, & Miller (2008) situate the final product among the third quartile range, which fits in the acceptable range. According to the adjective ratings, students found the final interface of the portal in between good and excellent. As an ultimate step of the evaluation, the SUS scores’ statistics computed for the students were displayed in a boxplot and compared to the ones computed for the experts. As seen in Figure 21, both means are above the score of 80 on the scale from 1 to 100, students tending to appreciate the platform as closer to the excellent rating than the participating experts with a SUS score difference of 0.125. Figure 21 Boxplots of the SUS scores computed for the expert and student evaluations EASE OF IMPLEMENTATION Regarding the overall ease of implementation of the system, the proposed client-server architecture required a well-documented basis with an overall complex setup of the back-end and an overall straightforward configuration of the front-end. The documentation support for the installation and configuration of the systems’ components has proven to be efficient and effective at all main levels: requirements, architecture, technical and end user. However, there were situations where the software documentation regarding the architecture design and technical specifications did not include support for some specific issue. Changes that require unfederation or re-indexing of the services of the portal lead to serious technical issues, that have no feasible solution documented by this date. Another important issue is raised around the ArcGIS Data Stores. Changes that require uninstallation, reinstallation, reconfiguration and back-up of the Data Store can cause failure or incomplete restoring of the databases. Inconsistencies in documentation can also occur. For example, the need to unregister the data store before uninstallation is not mentioned in the main documentation, while it appears as a 54 ESRI. (2017c, September 23). Scene Viewer requirements—ArcGIS Online Help. Retrieved February 5, 20191, from ArcGIS Online: https://doc.arcgis.com/en/arcgisonline/reference/scene-viewer-requirements.htm ESRI. (2017d, June 4). The ArcGIS REST API. Retrieved September 24, 2018, from ArcGIS for Developers: https://developers.arcgis.com/documentation/core-concepts/rest-api/ ESRI. (2018a, April 12). ArcGIS Online Basemaps. Retrieved January 5, 2019, from ArcGIS: https://www.arcgis.com/home/group.html?id=702026e41f6641fb85da88efe79dc166 #overview ESRI. (2018b, June). ArcGIS Server. Retrieved June 6, 2018, from ArcGIS Enterprise: http://enterprise.arcgis.com/en/server/latest/get-started/windows/what-is-arcgis-forserver-.htm ESRI. (2018c, July 10). ArcGIS Server - Register your data with ArcGIS Server using Manager. Retrieved November 22, 2018, from ArcGIS Enterprise: http://enterprise.arcgis.com/en/server/latest/publish-services/windows/registeringyour-data-with-arcgis-server-using-manager.htm ESRI. (2018d, January 11). ArcGIS Server - What types of services can you publish? Retrieved January 20, 2019, from ArcGIS Enterprise: http://enterprise.arcgis.com/en/server/latest/publish-services/windows/what-types-ofservices-can-you-publish.htm ESRI. (2018e, January 12). ArcGIS Server web services—Portal for ArcGIS. Retrieved January 2, 2019, from ArcGIS Enterprise: http://enterprise.arcgis.com/en/portal/latest/use/arcgis-server-services.htm ESRI. (2018f, January 9). Base ArcGIS Enterprise deployment. Retrieved January 9, 2019, from ArcGIS Enterprise: http://enterprise.arcgis.com/en/getstarted/latest/windows/base-arcgis-enterprise-deployment.htm 55 ESRI. (2018g, January 10). Configure a hosting server for portal. Retrieved January 7, 2019, from ArcGIS Enterprise| Portal for ArcGIS: http://enterprise.arcgis.com/en/portal/latest/administer/windows/configure-hostingserver-for-portal.htm ESRI. (2018h, January 10). Levels, roles, and privileges - Portal for ArcGIS. Retrieved January 9, 2019, from ArcGIS Enterprise: http://enterprise.arcgis.com/en/portal/latest/administer/windows/roles.htm ESRI. (2018i, January 12). OGC - Portal for ArcGIS. Retrieved November 11, 2018, from ArcGIS Enterprise: http://enterprise.arcgis.com/en/portal/latest/use/ogc.htm ESRI. (2018j, April 20). Overview | ArcGIS API for JavaScript. Retrieved November 6, 2018, from ArcGIS for Developers: https://developers.arcgis.com/javascript/ ESRI. (2018k, January 12). Portal for ArcGIS - Apps and functionality that require ArcGIS Data Store. Retrieved January 11, 2019, from ArcGIS Enterprise: http://enterprise.arcgis.com/en/portal/latest/administer/windows/what-requiresarcgis-data-store.htm ESRI. (2018l, January 11). Ports used by Portal for ArcGIS. Retrieved January 5, 2019, from ArcGIS Enterprise: http://enterprise.arcgis.com/en/portal/latest/install/windows/ports-used-by-portal-forarcgis.htm ESRI. (2018m, December 21). Scene layer package (.slpk) and Portal for ArcGIS. Retrieved January 24, 2019, from Geonet - The Esri Community: https://community.esri.com/message/821063-scene-layer-package-slpk-and-portalfor-arcgis ESRI. (2018n, December 31). Technical support -Error: Layer did not draw completely because there are too many features to display. Retrieved January 6, 2019, from ESRI: https://support.esri.com/en/technical-article/000014750 56 Gaghegan, M. (2018). Our GIS is too small. The Canadian Geographer(62), 15-26. doi:10.1111/cag.12434 GeoSeer. (2018, July 18). So, how many OGC layers are there? Retrieved from GeoSeer: https://www.geoseer.net/blog/?p=2018-07-18_how_many_layers_are_there GeoServer. (2019, January 19). Retrieved September 16, 2018, from http://geoserver.org/release/stable/ Idrizi, B. Z. (2014, September 1). Defining methodology for selecting most appropriate GIS software. Survey Review, 46(338), 383-389. doi: https://doi.org/10.1179/1752270614Y.0000000116 International Organization for Standardization. (2018, March). Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts (ISO 9241-11). 2. Retrieved January 3, 2019, from https://www.sis.se/api/document/preview/80003410/ IVVgeo. (2018, July 3). Gallery for Geo Münster. Retrieved January 8, 2019, from IVVgeo Geo Münster: http://ivv6.maps.arcgis.com/home/index.html IVVGeo-Team, U. o. (2018, 09 18). StudMap14 Dokumentationen. Retrieved 10 16, 2018, from IVVgeo Wiki: https://ivv6-wiki.unimuenster.de/neu/dokuwiki/doku.php?id=dev_studmap14:old:inhalt:dokumentation Kanellis, P., & Papadopoulos, T. (2009). Conducting Research in Information Systems: An Epistemological Journey. In A. Cater-Steel, & L. Al-Hakim (Eds.), Information Systems Research Methods, Epistemology, and Applications (pp. 1-34). Hershey, PA: IGI Global. doi:10.4018/978-1-60566-040-0.ch001 Kong, N., Zhang, T., & Stonebraker, I. (2015). Evaluation of web GIS functionality in academic libraries. Applied Geography, 60, 288-293. Retrieved May 8, 2018, from https://sciencedirect.com/science/article/pii/s0143622814002720 57 Li, Z., Hodgson, M. E., & Li, W. (2018, January 2). A general-purpose framework for parallel processing of large-scale LiDAR data. International Journal of Digital Earth, 11(1), 26-47. doi:https://doi.org/10.1080/17538947.2016.1269842 Liu, X., Li, S., Huang, W., & Gong, J. (2015, September). Designing sea ice web APIs for ice information services. Earth Sci Informatics, 8(3), 483-497. doi:https://doi.org/10.1007/s12145-015-0207-5 MapServer. (2018, September 23). Welcome to MapServer. Retrieved 9 16, 2018, from http://www.mapserver.org/ OGC. (2010a, November 11). OpenGIS Web Feature Service 2.0 Interface Standard (ISO/DIS 19142). (P. Panagiotis, & A. Vertanos, Eds.) Open Geospatial Consortium Inc. Retrieved December 15, 2018, from https://www.opengeospatial.org/standards/wfs OGC. (2010b, April 06). OpenGIS® Web Map Tile Service Implementation Standard (ISO 9241-11). (J. Masó, K. Pomakis, & N. Julià, Eds.) Open Geospatial Consortium Inc. Retrieved January 15, 2019, from https://www.opengeospatial.org/standards/wmts OGC. (2016a, June 24). Introduction to WMS. Retrieved September 24, 2018, from OGC: http://www.opengeospatial.org/standards/wmshttp://www.opengeospatial.org/standa rds/wms OGC. (2016b, June 24). Welcome to The Open Geospatial Consortium. Retrieved October 24, 2018, from OGC: http://www.opengeospatial.org/ OGC. (2017, September 5). OGC Indexed 3d Scene Layer (I3S) and Scene Layer Package Format Specification. (C. Reed, & T. Belayneh, Eds.) Retrieved January 9, 2019, from OGC: http://docs.opengeospatial.org/cs/17-014r5/17-014r5.html OpenNRW. (2017, January 5). Open Data. Retrieved January 8, 2019, from Open NRW: https://open.nrw/suche 58 Prinz, T. (2014, 05 14). Fünfte Sitzung. Retrieved 10 16, 2018, from GPS in den Geowissenschaften: http://ivvgeo.unimuenster.de/vorlesung/GPS_Script/praktikum5.html ProgrammableWeb. (2018, September 25). API Directory. (D. Berlind, W. Santos, & K. Sundstrom, Editors) Retrieved November 23, 2018, from ProgrammableWeb: https://www.programmableweb.com/category/all/apis Rouse, M. (2016, December 8). RESTful API. Retrieved January 6, 2019, from SearchMicroservices: https://searchmicroservices.techtarget.com/definition/RESTful-API Sauro, J., & Lewis, J. (2016). Standardized Usability Questionnaires. In Quantifying the user experience: Practical statistics for user research (pp. 185-186). Cambridge, MA: Elsevier Inc. Shekhar, S., Xiong, H., & Zhou, X. (2018). Encyclopedia of GIS (2nd ed.). Springer International Publishing. doi:10.1007/978-3-319-17885-1 Society, N. G. (2017, May 16). Maps - National Geographic. Retrieved November 5, 2018, from National Geographic: https://www.nationalgeographic.com/maps USGS. (2018, September 5). The National Map: Small Scale. Retrieved November 5, 2018, from USGS: https://nationalmap.gov/small_scale/ van Oosterom, P., Martinez-Rubi, O., Ivanova, M., Horhammer, M., Geringer, D., Ravada, S., . . . Gonçalves, R. (2015). Massive point cloud data management: Design, implementation and execution of a point cloud benchmark. Computers & Graphics, 49, 92-125. doi:http://dx.doi.org/10.1016/j.cag.2015.01.007 Veenendaal, B., Brovelli, M. A., & Li, S. (2017, October 21). Review of Web Mapping: Eras, Trends and Directions. International Journal of Geo-Information, 6(10), 317. doi:https://doi.org/10.3390/ijgi6100317 Veenendaal, B., Brovelli, M. A., Li, S., & Ivánová, I. (2017, September 12). What is web mapping anyway? ISPRS - International Archives of the Photogrammetry, Remote 59 Sensing and Spatial Information Sciences, XLII-2/W7, 155-160. doi:https://doi.org/10.5194/isprs-archives-XLII-2-W7-155-2017 Wagemann, J., Clements, O., Figuera, R. M., Rossi, A. P., & Mantovani, S. (2018). Geospatial web services pave new ways for server-based on-demand access and processing of Big Earth Data. International Journal of Digital Earth, 11(1), 7-25. doi:https://doi.org/10.1080/17538947.2017.1351583 Walsh, S. J., Page, P. H., McKnight, S., Yao, X., & Morrissey, T. P. (2015). A reservoir siting tool for North Carolina: System design & operations for screening and evaluation. Applied Geography, 60, 139-149. doi:https://doi.org/10.1016/j.apgeog.2015.03.015 Yang, C., Huang, Q., Li, Z., Liu, K., & Hu, F. (2017, January 2). Big Data and cloud computing: innovation opportunities and challenges. International Journal of Digital Earth, 10(1), 13-53. doi:https://doi.org/10.1080/17538947.2016.1239771 60 APPENDIX A. Applications interfaces 1. NRW 3Dapp 2. UAS 3Dapp 61 3. NRW 2Dapp 4. UAS 2Dapp 62 B. Functionality of the main widgets of the Studmap 3.0 applications 1. Basemap gallery 2. Add data 3. Measure 2D 63 4. Measure 3D 5. Select 6. Query 70 6. Diagram of answers to the statement “I thought there was too much inconsistency in this portal.” 7. Diagram of answers to the statement “I would imagine that most people would learn to use this portal very quickly.” 8. Diagram of answers to the statement “I found this portal very cumbersome/awkward to use.” 9. Diagram of answers to the statement “I felt very confident using this portal.” 71 10. Diagram of answers to the statement “I needed to learn a lot of things before I could get going with this portal.” 11. List of answers to the question “Which are the benefits of the interface of the portal addressed to the students' needs?” “The ease of use, since students just need to access one unique application that contains all the required data. Another + is the fact that users can create their own apps, based on the published information.” “Easy access to data and tools.” 12. List of answers to the question “Which are the drawbacks of the interface of the portal addressed to the students' needs?” “The 3D application has some rendering issues.” “The need to confirm an authentication already given, the sharing icon can be understood as a measurement of angles or azimuths.” 13. List of answers to the question “Which are the improvements that you would suggest being done to this portal?” “Some minor navigation or visual improvements. I would suggest the implementation of several tool tips or hints integrated within the application, shortly explaining how to use some features/widgets/functionalities.” “Rendering of the 3D data from other sources.” 14. List of answers to the question “As an overall experience, what comments do you have about the usability of the portal?” “Since this is an application intended for the use of students (experts), I think that they should not face any usability difficulties. 72 I would also consider this application and data for the use of public in general.” “The portal is easy to use, the applications accessible, but there was an issue with rendering the third-party 3D Data.” F. Usability evaluation – user testing questionnaire extracted from Google Forms 73 74 G. Usability evaluation – user testing results extracted from Google Forms 1. Diagram of answers to the statement “I think I would like to use this website frequently.” 75 2. Diagram of answers to the statement “I found this portal unnecessarily complex.” 3. Diagram of answers to the statement “I thought this portal was easy to use.” 4. Diagram of answers to the statement “I think that I would need assistance to be able to use this portal. 5. Diagram of answers to the statement “I found the various functions in this portal were well integrated.” 76 6. Diagram of answers to the statement “I thought there was too much inconsistency in this portal.” 7. Diagram of answers to the statement “I would imagine that most people would learn to use this portal very quickly.” 8. Diagram of answers to the statement “I found this portal very cumbersome/awkward to use.” 9. Diagram of answers to the statement “I felt very confident using this portal.” 77 10. Diagram of answers to the statement “I needed to learn a lot of things before I could get going with this portal.” 11. List of answers to the statement “Please provide any comments about the usability of this portal:” • “Very fast and simple to use, maybe change the application to data or to database” • “This portal looks interesting to access the data to the users, specially it would help the students of WWU. If these kinds of portal existed before, I could have benefited while I (sic!) was working on some projects while I was studying in WWU. This project is very appreciative and can be applied for research purpose for students and researchers. Since the portal works on ArcGIS server, the speed depends on the server capacity. Therefore, (sic!) the capacity of the server would determine the speed. The interface of the portal looks interesting and easy to use even for the new explorer. Good Luck. You did a great job!” • “It looks and works great! Really cool.” • “I think it is very useful geoportal with the access to both maps and meta data information. The different interfaces with 2d and 3d functionalities, import features even from csv (useful for direct field data import), measurement features are really appreciable.” • “Well designed, convenient to use.” • “This portal could be really helpful for the students to explore the existing geospatial data in the common platform.” • “The user interface is very friendly and the portal tools easy to learn. The layers are displayed very quickly, regardless of whether it is 2D or 3D. I only have one suggestion: during the navigation through the layers was missing a legend of the layers. This will allow us to give a better interpretation of the maps. For example, the red and green colors in the aspect and slope rasters should have a meaning.” • “It was well designed, (sic!) and everything was nicely integrated.” • “It's a great one.” • “Helpful tool for the specific are it's located.” • “The portal is simple, fast and looks neat. I could not use the 3D viewer because apparently I have issues with GPU performance.” • “Nice and intuitive interfaces!” 78 • “The portal is very straightforward and user friendly with very clear classification of the data available. I just found little problems switching from one app to other one (from 2D to home page to enter again 3D).” • “The portal looks really nice, it was easy to do all the task and the web performance was good. I also tried "find data" option and was good.” • “It would be nice if the user could start its own app, to explore its own data in combination with the available one. As far as I could see only the predefined apps are available. I found it a bit weird that some layers I try to preview just show as a square, but I think that is more related to the ESRI platform.“ • “Nice and fluent.” • “The user experience was very good. All the tasks were very easy to implement.” • “Portal was quite easy to use from the first time. No need to look for a long time to find a proper bottom or layer, though there were not many of them. Also, (sic!) the dark interface background was really pleasant. “ • “Very user friendly and clean.” • “The applicability of the portal was pretty straightforward. The learning curve is not steep and will only provide minor inconvenience to people who are not even remotely associated with geospatial background.”