Complexity of Structured Text in IEC 61499 Function Blocks: A Survey

Complexity of Structured Text in IEC 61499 Function Blocks: A Survey. Lisa Sonnleithner∗†, Bianca Wiesmayr∗, Antonio Manuel Guti´ errez∗†, Rick Rabiser∗†, Alois Zoitl∗†,Senior Member, IEEE †CDL VaSiCS ∗LIT CPS Lab, Johannes Kepler University Linz Linz, Austria {lisa.sonnleithner, bianca.wiesmayr, antonio.gutierrez, rick.rabiser, alois.zoitl}@jku.at Abstract—The rising complexity of modern production systems raises the need for evaluating the code quality also in control software engineering. Complexity metrics can help identifying code that is difficult to maintain and may benefit from a refactoring. Although complexity metrics have already been adapted to the domain-specific languages (DSLs) used for cyberphysical production systems (CPPSs), literature on whether developers benefit from using these metrics is limited. Also, the complexity perception of code by the domain experts remains to be investigated. The perceived difficulty of comprehending a program can vary among developers and also between different implementation variants of the same functionality. We therefore asked industrial experts, who have experience in the respective DSLs, to rate the complexity of semantically equivalent implementations to find out which metric best represents their complexity perception and whether there are differences between implementation variants. The examples used in the study were inspired, in parts, by an open source library for IEC 61131-3. We found that the Cognitive Complexity metric is a useful representation of the software complexity in the domain, as it best matched the domain experts’ perception of complexity. These metrics should therefore be supported by the analysis tools. Index Terms—IEC 61499, Distributed Control System, CPS, CPPS, Metric, Measure I. INTRODUCTION Domain-specific languages (DSLs) such as those defined in IEC 61131-3 and IEC 61499 are tailored to automation engineers who develop the software of control systems. Such software is found in cyber-physical production systems (CPPSs), where actuators are controlled based on inputs from sensors. Factors that contribute to the complexity of automation software include the need for real-time communication, the close interactions with involved hardware, error handling involving several components, and the need for considering safety and security concerns in certain applications. High code quality can help creating software that is maintainable and understandable. To cope with the rising complexity in the domain, software design patterns, development guidelines, as well as ways of measuring and rating certain attributes of software are needed. As the required software engineering expertise is often not found among domain experts, automated ratings of the software complexity may help raising its quality. Personal preferences and experiences influence the perceived complexity of a piece of software. Available studies on complexity metrics for DSLs for control software (e.g., [1]–[3]) focus on applying complexity metrics from generalpurpose languages (Section II). Available metrics include the number of lines of code (LOC), the Cyclomatic Complexity, or Halstead’s Metric Suite (Section III). The usefulness of such metrics for domain experts developing control software in CPPSs has not been evaluated systematically. We have designed a survey (Section IV) for identifying useful metrics for structured text (ST), which is a textual DSL defined in IEC 61131-3. In IEC 61499, this language is often used for implementing the functionality of a software component, i.e., a Function Block (FB). We collected feedback from 78 respondents (Section V) regarding the perceived complexity of ST code in IEC 61499 FBs, as well as factors that contribute to the complexity in the domain. We used the feedback from quantitative questions (Section VI) to evaluate whether certain metrics represent developers’ perception of complexity (Section VII). The industrial experts also listed various sources of complexity in ST (Section VIII). We discuss threats to validity in Section IX and conclude the paper in Section X. II. RELATED WORK For the past two decades, automatic mechanisms for measuring code complexity, code conventions, or design principles have been widely used in software engineering to enhance code understanding, maintenance, and quality control [4]. In textual programming languages, a catalog of metrics has been established and validated for use by practitioners, with tools such as those provided by Sonar1. Some efforts have been made to adapt these metrics for use in CPPS languages such as those defined in IEC 61131-3 and IEC 61499 as part of the development life-cycle. Wilch and Fischer et al. [3], [5] have explored how to apply certain metrics to evaluate the complexity in IEC 61131-3 to identify potential blocks that require refactoring. They confirmed that measuring the software complexity is important for developers. Neumann et al. [6], [7] use software metrics in IEC 61131-3 to detect runtime performance problems or target conflicts, respectively. Pr¨ ahofer et al. [8] proposed a rule-based language to evaluate certain metrics from categories such as naming conventions, 1https://www.sonarsource.com/ program complexity, and code smells. Zhabelova and Vyatkin [2] proposed an implementation of quality metrics based on IEC 61499 language elements, such as the connections between FBs, and state machines within FBs. Similarly, Sonnleithner et al. [1], [9] reviewed metrics for BFBs in IEC 61499 and redefined the catalog of bad smells proposed by Beck and Fowler [10] for IEC 61499 applications. Also, Stattelmann et al. [11] extended a tool for static code analysis for PLC Sofware, Arcade.PLC [12]. It is used with IEC 61131-3 for systematically calculating metrics during development. Further proposals aim at pre-emptively detecting the propensity to change, based on certain metrics and the use of neural networks [13], to compare the complexity of development between standards IEC 61131-3 and 61499 [14], and to automatically identify the technical debt [15]. However, the existing proposals for analyzing metrics in these languages are based on automatic processes and evaluated with one or two case studies, i.e., not all metrics were validated empirically for their respective fields, and are thus not commonly used in practice. Empirical studies are crucial to evaluate the effectiveness of software engineering techniques, tools, and methods in research and practice [16]. In this work, we focus on an empirical study of the code complexity for Simple Function Blocks in IEC 61499 (those containing ST). Complexity is defined by IEEE as “the degree to which a system or component has a design or implementation that is difficult to understand and verify” [17]. Among the metrics related to understandability (e.g. cyclomatic complexity, Halstead’s Metric Suite), the correlation between the metric cognitive complexity [18] and code understanding have been extensively evaluated by Mu˜ noz Bar´ on et al. [19] using existing understandability reviews in textual programming languages. Jetley et al. [20] propose a visualization to compare differences between systems based on the differences of the source code. This approach, while useful for version management, requires manual evaluation. We do not find, to our knowledge, empirical evaluations of metrics for the understandability of systems implemented with the graphical languages proposed by IEC 61499 and, partly, IEC 61131-3. The aim of our study is to specifically collect experiences from domain experts regarding ST complexity in IEC 61499 FBs. III. METRICS We shortly present various metrics used in software engineering to measure complexity of software. All metrics except for Cognitive Complexity were already applied to ST before. A. Lines of Code Lines of Code (LOC) is a metric to measure the size of a piece of software. There are several variants of calculating the LOC. First, all code lines can be counted as LOC. Then, LOC could be modified by ignoring empty lines. Lastly, also comment lines could be ignored. This version of LOC is oftentimes also referred to as Source Lines of Code (SLOC). In this paper, we use SLOC. B. Cyclomatic Complexity McCabe’s Cyclomatic Complexity [21] is a complexity metric that calculates the number of independent paths of a programs’ control flow graph. It can be calculated by M=E−N+ 2, where Eis the number of edges of the graph and Nthe number of nodes. C. Halstead’s Metric Suite Halstead’s Metric Suite [22] is based on the assumption that a program of any programming language consists of operators and operands. The number of unique and total operators and operands is then used to calculate several metrics, such as the difficulty or the programming effort. Applying this concept to a programming language requires a definition of operators and operands in the context of this language. Halstead’s Metric Suite was applied to ST in [15]. However, the authors did not provide details on their definition of operators and operands in ST. Following the convention used for other textual programming languages, we define the following rules for operators and operands in ST: Operators: AND,OR,XOR,IF,THEN,ELSE,ELSIF, END_IF,FOR,TO,DO,END_FOR,WHILE, END_WHILE,REPEAT,UNTIL,END_REPEAT, CASE,OF,END_CASE,RETURN,EXP,:=,<=,>=, <>,<,>,-,NOT,*,:,MOD,+,= Operands: variables, constants We used the metrics Program Volume, Difficulty and Effort of Halstead’s Metric Suite in our evaluation. D. Cognitive Complexity The metric Cognitive Complexity [18], as the name suggests, aims at formulating values that represent programmers’ intuition about the mental effort that is required to understand code. This metric has not been applied to ST code before. Therefore, we present our concept to calculate the Cognitive Complexity of ST code, based on [18]. In ST, the following statements break the linear flow and, therefore, increase the Cognitive Complexity by 1: •CASE,IF,ELSIF •FOR,WHILE,REPEAT •CONTINUE,EXIT •logical operators AND,OR,NOT,XOR (including a sequence of identical operators) In ST, the following statements lead to nesting: •CASE,IF,ELSIF •FOR,WHILE,REPEAT For each level of nesting, the score of a nesting element is further increased by 1. The following example illustrates how the cognitive complexity can be calculated for ST code: FOR i:= 0 TO 10 DO // +1 IF A // +2 (nesting=1) AND B // +1 (AND) AND C // +0 (still AND) OR D THEN // +1 (OR) ... // now, nesting=2 END_IF; END_FOR; The resulting Cognitive Complexity score for this example is 5. Parts of the code that increase the score are marked bold. IV. RESEARCH METHOD In this section, we describe the research design. First, we present the research questions, followed by a description of how the data was collected and analyzed. The following research questions guided our investigation: RQ 1 How do certain language features and their usage influence the complexity/understandability of ST code? RQ 2 Do the metrics that we discussed in Section III (i.e., Halstead’s Metric Suite, McCabe’s Cyclomatic Complexity, Cognitive Complexity, SLOC) represent developer’s perception of complexity? With the quantitative data that we plan to collect, we want to test the following hypothesis: H0The complexity rating of ST code does not depend on the implementation variant. A. Instrument Development and Data Collection We created an online questionnaire2to collect the necessary data. The questionnaire was developed iteratively. An initial version was created by two of the authors. It was then tested by two other authors who gave detailed feedback based on their experience. Based on the collected feedback, the questionnaire was adjusted and the final version was approved by all researchers. The final questionnaire consists of 5 demographic questions, 3 filtering questions, 9 questions about code complexity, and two open text questions. In the filtering questions, we asked for previous experience in using the languages of IEC 61131-3 and IEC 61499, as well as general programming experience. Survey answers from participants without any programming experience were discarded. All questions regarding code complexity provided a language example in different variants. Participants rated these examples based on their perceived understandability of each example on a 10- point scale from 1 to 10. To ensure that respondents interpret the scale correctly, 1 was marked as “understandable at first glance” and 10 was marked as “impossible to understand”. The data was collected between March 7th 2022 and August 1st 2022. The questionnaire was promoted through the authors’ personal contacts (21) and through social media channels (i.e., LinkedIn and Twitter) including specific interest groups. B. Data Analysis We analyzed the collected survey data in three steps: 1) Data Cleaning: We received 112 survey responses, with 72 respondents completing the survey. As the questions about the complexity rating were presented in a random order, we also included partial responses that completed at least one such question. We excluded one response, because the complexity scale was interpreted wrongly. We came to this conclusion, because of the answers the respondent gave to question Bool, where the respondent gave the highest score to the variant a 2Printable version of the survey: https://doi.org/10.5281/zenodo.7863270 AND b. There was no respondent without any programming experience. Finally, we included 78 responses in our analysis. 2) Quantitative Analysis: We analyzed the quantitative data using descriptive statistics and hypothesis testing [23], [24]. We used the Shapiro-Wilk test [25] to test for normality of each variable. We further analyzed the data set for outliers and excluded extreme outliers (i.e., above Q3+ 3 ∗IQR or below Q1−3∗IQR), if present. We then tested the hypothesis that we introduced above. The underlying data for testing this hypothesis are the ratings for different implementation variants. Therefore, we used a paired difference test. 3) Qualitative Analysis: At the end of the questionnaire, we provided one open text question where we asked participants what they perceive as complex in ST code (51 responses), and to provide general feedback to the survey (32 responses). To analyze these responses, we used open coding [26]. We started with an empty set of codes and developed the codes during the process. This step was performed by one researcher who then discussed open questions with a second researcher, and the final coding was reviewed by a third researcher for consistency. V. RESPONDENT DEMOGRAPHICS The 78 responses that we included in the analysis covered 14 different countries. The majority of respondents are based in Austria (36) or Germany (22), as shown in Figure 1a. Figure 1b illustrates the domain of the respondents, with academia (24) and plant manufacturing industry (23) as the largest groups. Respondents mentioned a large variety of markets that their employer is operating in, see Figure 1c. 24 respondents stated that they had more than 10 years of experience in IEC 61131-3 and/or IEC 61499 programming. Another 19 respondents stated that they had between 5 and 10 years of experience in at least one of these standards. Figure 2 shows a detailed view on respondents’ experience. In general, respondents were very experienced in general programming (59 respondents with more than 5 years of experience). There was no participant who had no experience in either IEC 61131- 3 or IEC 61499. Respondents were generally more experienced in IEC 61131-3 than in IEC 61499. On average, respondents spent 23:55 min on the survey. Four extreme outliers (i.e., above Q3+ 3 ∗IQR or below Q1−3∗IQR) who took over 2 h were excluded when calculating the average. They may have worked on the survey with interruptions. VI. FINDINGS IN QUANTITATIVE QUESTIONS In this section, we present detailed results of the hypothesis testing that we performed on the data. For statistical significance, we used a threshold of p < .05 for all following tests. We also compare the results to metrics. In each question except question Bool, participants rated 2 to 3 implementation variants of a certain functionality. In question Bool, we presented 6 different Boolean expressions. We refer to these variants as V1 to V6. Table I summarizes the results of the descriptive statistics analysis of the quantitative questions. (a) Country (b) Area (c) Market Fig. 1: Summary of the background of respondents. The legend matches the order of the pie chart in clock-wise direction. The first entry of the legend corresponds to the sector at 12 o’clock. Fig. 2: Programming experience of respondents in years (y), which is categorized in experience in IEC 61131-3, IEC 61499, and general programming experience. A. AddDate: Loops/Conditions (2 Variants) The implementation variants add months and years to a given date and output the resulting date. V1 uses WHILE loops. V2 uses IF statements and modulo operations to calculate how many months and years need to be added. There was no statistical significance in the ratings. The Halstead metrics result in higher values for V2, while the SLOC is lower. The Cyclomatic Complexity and the Cognitive Complexity rate the variants equally (Table II). B. FindPairs: Complexity/Performance (2 Variants) The examples search for equal entries in Aand Band store the result in an array. V1 was implemented by using two nested FOR loops with a time complexity of O(n2). The quantitative data are listed in Table III. V2 is more efficient by sorting A and Bfirst, which results in a lower time complexity of O(n). However, respondents rated V1 as easier to understand with statistical significance. All metrics but Cyclomatic Complexity rate V1 easier than V2. Cyclomatic Complexity results in an equal rating for V1 and V2. C. AnalyzeArray: Comments/Methods (3 Variants) The examples perform one out of three available array operations on an input array and supply the result to an output. TABLE I: Summary of descriptive statistics analysis. Question Variant N Mean SD AddDate V1 72 2.99 2.14 V2 72 3.17 2.08 FindPairs V1 70 2.09 1.29 V2 70 5.50 2.35 AnalyzeArray V1 71 3.54 2.13 V2 71 4.15 2.26 V3 71 1.72 1.92 Abs V1 74 1.80 1.48 V2 74 2.45 2.05 Hysteresis V1 75 2.20 1.57 V2 75 4.19 2.30 Round V1 71 4.04 2.24 V2 71 1.76 1.30 Mirror V1 73 2.25 1.54 V2 73 3.99 2.23 V3 73 2.01 1.50 SearchChar V1 73 1.68 1.39 V2 73 3.38 2.15 V3 73 2.97 1.83 Bool V1 74 1.03 0.23 V2 74 1.20 0.52 V3 74 1.43 0.85 V4 74 3.65 2.68 V5 74 3.65 2.69 V6 74 1.59 0.86 TABLE II: Comparison of metrics and respondents’ rating for the two implementation variants of AddDate. V D E CC CogC SLOC Mean SD V1 135.93 7.64 1038.04 3 2 11 2.99 2.14 V2 256.76 16.00 4108.13 3 2 10 3.17 2.08 The code of V1 and V2 is identical, but V1 has additional comments to explain the array operations. Instead of using comments, V3 uses method calls to perform the operations. V2 was voted the most difficult one to understand with statistical significance. V3 was voted the easiest to understand with statistical significance. In this case, V1 and V2 were rated TABLE III: Comparison of metrics and respondents’ rating for the two implementation variants of FindPairs. V D E CC CogC SLOC Mean SD V1 98.99 11.25 1113.65 4 6 10 2.09 1.29 V2 342.35 18.11 6199.01 4 7 20 5.50 2.35 equally by all metrics. The metric results for V3 were also the lowest (Table IV). TABLE IV: Comparison of metrics and respondents’ rating for the three implementation variants of question AnalyzeArray. V D E CC CogC SLOC Mean SD V1 323.33 15.08 4874.86 9 15 22 3.54 2.13 V2 323.33 15.08 4874.86 9 15 22 4.15 2.26 V3 114.45 4.09 468.20 4 3 7 1.72 1.92 D. Abs: Sequential Code/Method Calls (2 Variants) In this question, we presented two variants that calculate the absolute value of a 3-dimensional vector. In V1, the square of each dimension was calculated using a multiplication. A method call was used in V2. The differences in the ratings are not statistically significant. In this case, the Halstead metrics result in higher values for V1. All other metrics rate the variants equally (Table V). TABLE V: Comparison of metrics and respondents’ rating for the two implementation variants of Abs. V D E CC CogC SLOC Mean SD V1 42.00 2.40 100.80 1 0 1 1.80 1.48 V2 33.00 1.33 44.00 1 0 1 2.45 2.05 E. Hysteresis: Branches/Nesting In this question, we presented two implementation variants of a hysteresis function. The first variant solves the problem by using several IF branches. The second variant uses less branches but deeper nesting. The second variant was rated more difficult to understand with statistical significance. The Halstead Difficulty metric rates both variants equally. All other metrics rate the second variant higher (Table VI). TABLE VI: Comparison of metrics and respondents’ rating for the two implementation variants of Hysteresis. V D E CC CogC SLOC Mean SD V1 105.49 8.00 843.89 3 3 9 2.20 1.57 V2 125.02 8.00 1000.16 4 6 10 4.19 2.30 F. Round: If-Condition/Case-Statements (2 Variants) Both implementation variants round N digits of a given input. V1 uses nested IF branches, V2 a CASE statement. V2 was rated as easier to understand with statistical significance. All metrics except Cyclomatic Complexity come to the same result, see Table VII. TABLE VII: Comparison of metrics and respondents’ rating for the two implementation variants of Round. V D E CC CogC SLOC Mean SD V1 155.32 9.00 1397.91 4 7 14 4.04 2.24 V2 81.41 6.42 522.38 5 1 11 1.76 1.30 G. Mirror: Loop Variants (3 Variants) The implementations mirror a given string. V1 solves the problem using a WHILE loop, V2 uses a REPEAT loop, and V3 uses a descending FOR loop. V2 was rated as more difficult to understand than the other variants with statistical significance. The Halstead metrics rate V1 and V2 as equally, and V3 as easier to understand. The Cyclomatic Complexity is the same for all variants. The Cognitive Complexity is equal for V1 and V3, but higher for V2. SLOC rates V3 the lowest, followed by V1 and V2 (Table VIII). TABLE VIII: Comparison of metrics and respondents’ rating for the three implementation variants of Mirror. V D E CC CogC SLOC Mean SD V1 91.38 5.63 513.99 2 1 6 2.25 1.54 V2 91.38 5.63 513.99 2 3 8 3.99 2.23 V3 77.71 4.38 339.98 2 1 5 2.01 1.50 H. SearchChar: Loop Variants (3 Variants) All code samples search the first occurrence of a given character in a string. V1 was implemented using a FOR loop. V2 and V3 were implemented using a WHILE loop. V2 used a more complicated loop condition, while V3 used a RETURN statement within the loop. V1 was rated the easiest to understand with statistical significance. The complexity metrics are listed in Table IX. TABLE IX: Comparison of metrics and respondents’ rating for the three implementation variants of SearchChar. V D E CC CogC SLOC Mean SD V1 100.08 6.50 650.51 3 3 8 1.68 1.39 V2 155.59 12.22 1901.65 3 4 9 3.38 2.15 V3 135.93 10.00 1359.34 3 3 10 2.97 1.83 I. Bool: Combinations of Operators (6 Variants) In this question, we presented six different Boolean expressions. The first expression (V1) was a simple AND expression. V2 and V3 used three and five AND operators, respectively. V4 and V5 each used three mixed AND and OR operators. V6 was identical to V5, but with additional (unnecessary) brackets around the AND operator. The differences in user ratings between V4 and V5, as well as between V3 and V6 are not statistically significant. All other differences between variants are statistically significant. The Cyclomatic Complexity metric and SLOC rate all variants equally. Halstead’s Difficulty and the Cognitive Complexity rate V1-V3 equally. V4-V6 are also rated equally, but higher than V1-V3. Halstead’s Volume and Effort also rate V4-V6 highest. For V1-V3 there is a gradation between ratings (Table X). J. Summary of the Complexity Metrics’ Performance To compare how each metric performed, we used the following rating to calculate an overall score: I: Each time a metric rated two variants differently, although there was no statistically significant difference, we subtracted one point from the overall score. TABLE X: Comparison of metrics and respondents’ rating for the six implementation variants of Bool. V D E CC CogC SLOC Mean SD V1 4.75 0.50 2.38 1 1 1 1.03 0.23 V2 16.25 0.50 8.13 1 1 1 1.2 0.52 V3 30.88 0.50 15.44 1 1 1 1.43 0.85 V4 18.09 1.00 18.09 1 3 1 3.65 2.68 V5 18.09 1.00 18.09 1 3 1 3.65 2.69 V6 18.09 1.00 18.09 1 3 1 1.59 0.86 II: Each time a metric rated two variants differently, where there was a statistically significant difference, we added one point to the overall score. III: Each time a metric rated two variants equally, although there was no statistically significant difference, we added one point to the overall score. IV: Each time a metric rated two variants equally, although there was a statistically significant difference, we subtracted one point from the overall score. V: Each time where there was a statistically significant difference, but the metric rated the variants the opposite way, we subtracted one point from the overall score. Additionally, we also normalized the scores so that the results of each question contribute equally to the overall score. In Table XI and XII, we summarized the scores with and without normalization, respectively. The score without normalization for Halstead’s Volume and Effort and for Cognitive Complexity outperforms the other metrics with an overall score of 11. The Cyclomatic Complexity and SLOC even result in a negative score of -12 and -5, respectively. With the normalized score, the Cognitive Complexity achieves the best result of 5.20. SLOC achieves the second highest score of 2.27 followed by Halstead’s Volume and Effort both scoring 0.93. Halstead’s Difficulty and the Cyclomatic Complexity result in a negative score of −0.47 and −0.07, respectively. We find it interesting, that Halstead’s Volume and Effort perform better in rating how difficult an FB is to understand than Halstead’s Difficulty. While Halstead’s Volume and Effort share the first place for the score without normalization with Cognitive Complexity, the performance drops with normalization. The reason for this is, that these two metrics were able to best reflect developer’s perspective on question Bool. As this question comes with 6 variants and therefore 15 combinations of variants, the performance in this question greatly influences the score without normalization. Because SLOC counts all lines equally, it performs poorly in question Bool, which is the reason for its negative score without normalization. With normalization, the metric performs better, where it ranks second. The Cyclomatic Complexity seems to be an inappropriate metric for reflecting developer’s perception of ST complexity. Overall, Cognitive Complexity performed best, both, with and without normalization. VII. FINDINGS IN QUALITATIVE QUESTIONS At the end of the questionnaire, we asked for respondents’ opinion on what makes ST code complex in the form of an TABLE XI: Summary of the metric ratings. I: Difference in metric rating, but no statistical difference; II: Difference in metric rating, statistical difference; III: No difference in metric rating, no statistical difference; IV: No difference in metric rating, but statistical difference; V: Difference in metric rating opposite of statistical difference. V D E CC CogC SLOC I 5 5 5 0 2 3 II 19 15 19 3 16 9 III 1 1 1 6 4 3 IV 4 8 4 19 7 14 V 0 0 0 1 0 0 Sum 11 3 11 -12 11 -5 TABLE XII: Summary of the metric ratings (normalized). V D E CC CogC SLOC I 2.73 2.73 2.73 0.00 0.40 1.67 II 4.40 3.20 4.40 0.67 4.20 4.00 III 0.07 0.07 0.07 2.80 2.40 1.13 IV 0,80 1.00 0.80 2.53 1.00 1.20 V 0.00 0.00 0.00 1.00 0.00 0.00 Sum 0.93 -0.47 0.93 -0.07 5.20 2.27 open text question. In total, 51 people provided feedback. The result of the coding process is presented in Table XIII. For codes mentioned by at least ten people, we provide detailed insights in the following subsections with examples from the questionnaires. A. Bad Modularization A total of 19 respondents mention bad modularization as a factor that makes ST code hard to understand. Respondents either mentioned suboptimal separation into components or deep nesting of hierarchies. “Too much or too little separation into independent components” [ID 36] “Uses of call FB/Functions to do the tasks rather than writing whole code in one FB” [ID 5] B. Size In total, 19 respondents mention size-related issues as a factor that makes ST code hard to understand. Respondents either directly mention the size of code in terms of lines of TABLE XIII: Summary of the coding result including the number of answers found. Code # Bad Modularization 19 Size 19 Unclear Code 17 Missing Description 14 Unnecessary Complicated Code 12 Complex Operations 11 Bad Format 9 Complicated Language Features 5 Inappropriate Language Features 3 Missing Coding Guidelines 3 Code Smells 1 code or more specifically that code should not contain too many loops or have too many responsibilities. “The longer a program is, the harder it is for me to comprehend it” [ID 56, translated from German] “Too long code (# lines)” [ID 49] “Too many responsibilities” [ID 90] C. Unclear Code The reasons why respondents mentioned unclear code are mainly found in inappropriate naming of variables or functions/methods. However, respondents also mention unclear interfaces, magic numbers, or code that does not speak for itself. “Variables (events, data, etc.) with names hard to relate to their semantic (e.g. ‘Q’)” [ID 85] “Not self explanatory code” [ID 58] D. Missing Description Missing comments or documentation were mentioned by 14 respondents. “Not documenting custom functions” [ID 107] “Code without any comments” [ID 15] E. Unnecessarily Complicated Code In total, 12 respondents mentioned different forms of unnecessarily complicated code. Out of these 12 respondents, six mention smart code (i.e., code that solves a certain problem with a minimum of lines/operations or more efficiently, but at a high cost of understandability). Other respondents also mention unnecessarily mixing loop types or the usage of IFELSEIF, when the problem could also be solved with CASE. “Over optimizing programming style” [ID 4] “Mixture of different loop types” [ID 37] F. Complex Operations Complex operations were mentioned by 11 respondents. “Complex conditional loops” [ID 42] “Highly aggregated code lines.” [ID 43] G. Summary The open text feedback provided important feedback on developer’s perception of complexity of ST code. Particularly interesting was that missing comments were mentioned by 12 respondents, although this contrasts the results of the quantitative question in Section VI-C. While the variant with comments was rated slightly easier to understand, the variant that used methods instead of comments could cut the rating in half. That smart code was mentioned by six respondents as a factor that complicates code confirms the result of the quantitative question in Section VI-B. VIII. DISCUSSION With the collected data of this survey, we got important insights into developers’ perception of ST complexity of IEC 61499 FBs. Smart code (cf. Section VII-E) should be avoided whenever possible. This is the result of question FindPairs and the qualitative feedback. Therefore, complex language constructs should be avoided (RQ 1), even if they provide a performance improvement. However, in time-critical situations this might not always be possible. Question AnalyzeArray leads us to the conclusion that, whenever code gets too complex (a metric threshold could be used to quantify this), it is more beneficial to refactor parts of the code into methods with meaningful names than adding comments to the code (RQ 1). According to question Hysteresis, more IF branches are easier to understand than fewer branches with deeper nesting (RQ 1). Question Round leads to the conclusion that whenever a CASE statement can be used to solve a problem it should be used instead of an IF construct (RQ 1). The most difficult loop to understand seems to be a REPEAT loop (RQ 1). It was ranked more difficult than a WHILE loop and a descending FOR loop in question Mirror. The easiest loop type, according to this survey, is an ascending FOR loop (RQ 1). It was rated easiest when compared to two WHILE loops in question SeachChar. Boolean expressions are perceived as more complex when different operators are used. However, this effect can be reduced by using brackets, even if they are not necessary (RQ 1). These insights can be used as a basis to develop specific coding guidelines for developers. Answering RQ 2, all investigated metrics failed to represent developer’s complexity perception at some questions. Overall, Cognitive Complexity performed best. We therefore suggest to implement this metric in IDEs to support developers. IX. THREATS TO VALIDITY Similarly to other empirical studies, our survey shows a number of threats to validity. Firstly, the survey was designed by only two researchers, based on their experiences with ST and earlier research on metrics. To mitigate this threat, we asked other colleagues for feedback. We refined the survey to incorporate their feedback before publishing it. Nevertheless, other researchers might have asked different questions with different examples of ST code. Another threat to the internal validity of our results is that respondents may have misunderstood questions. We addressed this threat by conducting a test run of the study with experienced colleagues. Furthermore, we provided open questions at the end of the survey for participants to raise concerns. An external threat is that we might not have contacted a representative pool of potential participants. Additionally to choosing contacts from our own network, we also posted the questionnaire to LinkedIn and Twitter, also in special interest groups, to reduce the impact of this threat. Also, we asked respondents for their experience in different relevant areas and the domain they are working in. A further important threat is that we might have wrongly interpreted/coded the answers of the open questions. For the quantitative questions, we performed the suggested statistical tests and removed outliers. For the open question, we used open coding and discussed results among three authors. Also we only present detailed results for codes with more than 10 statements by respondents. X. CONCLUSION AND FUTURE WORK In this paper, we conducted a survey to assess developers’ perception of the complexity of Structure Text Function Blocks in IEC 61499. We received a total of 112 responses, out of which we included 78 in the analysis. We used descriptive statistics and hypothesis testing for the analysis of the quantitative questions. We also compared the results with the ratings of metrics. We concluded that the Cyclomatic Complexity is an inappropriate metric to rate the complexity of ST FBs in IEC 61499. The Cognitive Complexity metric best matched developers’ opinion. Nevertheless, all metrics had shortcomings in reflecting developers’ perception. One advice on good coding practice, that we have probably all heard before, but should still constantly remind ourselves of: “Use always the simplest possible way of programming for the requirement, because you do not program for yourself!” [ID 65] In future work, we will analyze whether there are any statistically significant differences to the answers depending on respondents’ demographics. We also plan to investigate an adapted definition of operators to calculate the Halstead metrics. Furthermore, we plan to evaluate the benefit of using Cognitive Complexity during development. Specifically, we want to evaluate a Cognitive Complexity threshold that can be used to suggest developers to refactor their code. ACKNOWLEDGEMENTS The financial support by the Austrian Federal Ministry for Digital and Economic Affairs, the National Foundation for Research, Technology and Development, and the Christian Doppler Research Association is gratefully acknowledged. We also want to thank all respondents of the survey for their time and valuable input. REFERENCES [1] L. Sonnleithner and A. Zoitl, “A software measure for IEC 61499 Basic Function Blocks,” in 25th IEEE Int. Conf. on Emerging Technologies and Factory Automation (ETFA), 2020. [2] G. Zhabelova and V. Vyatkin, “Towards software metrics for evaluating quality of IEC 61499 automation software,” in 20th IEEE Int. Conf. on Emerging Technologies and Factory Automation (ETFA), 2015. [3] J. Fischer, B. Vogel-Heuser, H. Schneider, N. Langer, M. Felger, and M. Bengel, “Measuring the overall complexity of graphical and textual iec 61131-3 control software,” IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 5784–5791, 2021. [4] S. H. Kan, Metrics and Models in Software Quality Engineering, 2nd ed. USA: Addison-Wesley Longman Publishing Co., Inc., 2002. [5] J. Wilch, J. Fischer, E.-M. Neumann, S. Diehm, M. Schwarz, E. Lah, M. Wander, and B. Vogel-Heuser, “Introduction and evaluation of complexity metrics for network-based, graphical IEC 61131-3 programming languages,” in 45th Annual Conf. of the IEEE Industrial Electronics Society (IECON), vol. 1. IEEE, 2019. [6] E.-M. Neumann, B. Vogel-Heuser, J. Fischer, J. Keller, I. Weiß, S. Diehm, M. Schwarz, T. Englert, M. Stoll, and U. Zell, “Identifying runtime issues in object-oriented IEC 61131-3-compliant control software using metrics,” in 46th Annual Conf. of the IEEE Industrial Electronics Society (IECON), 2020. [7] E. M. Neumann, B. Vogel-Heuser, M. Gnadlinger, J. Fischer, L. Reimoser, S. Diehm, T. Englert, and M. Schwarz, “Metric-based identification of target conflicts in the development of industrial automation software libraries,” in IEEE Int. Conf. on Industrial Engineering and Engineering Management (IEEM), 2022. [8] H. Pr¨ ahofer, F. Angerer, R. Ramler, H. Lacheiner, and F. Grillenberger, “Opportunities and challenges of static code analysis of IEC 61131-3 programs,” in 17th IEEE Int. Conf. on Emerging Technologies Factory Automation (ETFA), 2012. [9] L. Sonnleithner, S. B´ acsi, M. Oberlehner, E. Kutsia, and A. Zoitl, “Do you smell it too? Towards bad smells in IEC 61499 applications,” in 26th IEEE Int. Conf. on Emerging Technologies and Factory Automation (ETFA), 2021. [10] K. Beck, M. Fowler, and G. Beck, “Bad smells in code,” Refactoring: Improving the design of existing code, vol. 1, no. 1999, pp. 75–88, 1999. [11] S. Stattelmann, S. Biallas, B. Schlich, and S. Kowalewski, “Applying static code analysis on industrial controller code,” in 19th IEEE Int. Conf. Emerging Technology and Factory Automation (ETFA), 2014. [12] S. Biallas, J. Brauer, and S. Kowalewski, “Arcade.plc: A verification platform for programmable logic controllers,” in Proc. of the 27th IEEE/ACM Int. Conf. on Automated Software Engineering, ser. ASE. ACM, 2012. [13] L. Kumar and A. Sureka, “Using structured text source code metrics and artificial neural networks to predict change proneness at code tab and program organization level,” in Proc. of the 10th Innovations in Software Engineering Conf., ser. ISEC. ACM, 2017. [14] P. Gsellmann, M. Melik-Merkumians, and G. Schitter, “Comparison of code measures of IEC 61131–3 and 61499 standards for typical automation applications,” in 23rd IEEE Int. Conf. on Emerging Technologies and Factory Automation (ETFA), 2018. [15] L. Capit´ an and B. Vogel-Heuser, “Metrics for software quality in automated production systems as an indicator for technical debt,” in 13th IEEE Conf. on Automation Science and Engineering (CASE), 2017. [16] B. A. Kitchenham, D. Budgen, and P. Brereton, Evidence-based software engineering and systematic reviews, Taylor & Francis Group, Ed. CRC press, 2015, vol. 4. [17] “IEEE standard glossary of software engineering terminology.” IEEE, 1990. [18] G. A. Campbell, “Cognitive complexity: An overview and evaluation,” in Proc. of the Int. Conf. on Technical Debt, ser. TechDebt. ACM, 2018. [19] M. Mu˜ noz Bar´ on, M. Wyrich, and S. Wagner, “An empirical validation of cognitive complexity as a measure of source code understandability,” in Proc.of the 14th ACM / IEEE Int. Symposium on Empirical Software Engineering and Measurement (ESEM). ACM, 2020. [20] R. Jetley, A. Rath, V. Aparajithan, D. Kumar, V. Prasad, and S. Ramaswamy, “An approach for comparison of IEC 61131-3 graphical programs,” in 18th IEEE Int. Conf. on Emerging Technologies and Factory Automation (ETFA), 2013. [21] T. J. McCabe, “A complexity measure,” IEEE Trans. on Software Engineering, vol. SE-2, no. 4, pp. 308–320, 1976. [22] M. H. Halstead, “Elements of software science,” 1977. [23] C. Wohlin, P. Runeson, M. H¨ ost, M. C. Ohlsson, and B. Regnell, “Experimentation in software engineering,” in The Kluwer Int. Series in Software Engineering, 2000. [24] M. Felderer and G. H. Travassos, Contemporary empirical methods in software engineering. Springer, 2020. [25] S. S. Shapiro and M. B. Wilk, “An analysis of variance test for normality (complete samples),” Biometrika, vol. 52, no. 3-4, pp. 591–611, 12 1965. [26] C. Seaman, “Qualitative methods in empirical studies of software engineering,” IEEE Trans. on Software Engineering, vol. 25, no. 4, pp. 557–572, 1999.