Requirements Quality Is Quality in Use [original]

This version is available at https://doi.org/10.14279/depositonce-8475

all other uses, in any current or future media, including reprinting/republishing this material for

advertising or promotional purposes, creating new collective works, for resale or redistribution to

servers or lists, or reuse of any copyrighted component of this work in other works.

Femmer, H.; Vogelsang, A. (2019). Requirements Quality Is Quality in Use. IEEE Software, 36(3),

83–91. https://doi.org/10.1109/ms.2018.110161823

Henning Femmer, Andreas Vogelsang

Requirements Quality Is Quality in Use

Accepted manuscript (Postprint)Journal article |

Requirements Quality is Quality in Use

Henning Femmer and Andreas Vogelsang

Abstract—The quality of requirements engineering artifacts

is widely considered a success factor for software projects.

Currently, the definition of high-quality or good RE artifacts

is often provided through normative references, such as quality

standards, textbooks, or generic guidelines. We see various

problems of such normative references: (1) It is hard to ensure

that the contained rules are complete, (2) the contained rules

are not context-dependent, and (3) the standards lack precise

reasoning why certain criteria are considered bad quality. To

change this understanding, we postulate that creating an RE

artifact is rarely an end in itself, but just a means to understand

and reach the project’s goals. Following this line of thought,

the purpose of an RE artifact is to support the stakeholders

in whatever activities they are performing in the project. This

purpose must define high-quality RE artifacts. To express this

view, we contribute an activity-based RE quality meta model

and show applications of this paradigm. Lastly, we describe the

impacts of this view onto research and practice.

Index Terms—Quality, quality standards, requirements, docu-

mentation, roadmap

Three Actionable Insights:

•Always remember: Requirements engineering artifacts

are a means, not an end.

•Therefore, before writing your requirements, think about

the readers and how they use the artifacts first.

•Use this simple model to define who is using RE artifacts,

what RE artifacts are used for, and how RE artifacts

should therefore look like.

I. CURRENT STANDARDS ARE INCOMPLETE, INADEQUATE

AND IMPRECISE ON REQUIREMENTS QUALITY.

Requirements Engineering (RE) artifacts are central entities

in the software engineering process. Based on these artifacts,

project managers estimate effort, designers create architec-

tures, developers build the system, and test managers set up a

test-strategy. Consequently, quality defects in RE artifacts can

cause expensive consequences in subsequent software devel-

opment activities. Therefore, quality control of RE artifacts is

key for successful software development projects.

The definition of high-quality or good RE artifacts is

often provided through normative references, such as quality

standards or textbooks (e.g., ISO/IEEE/IEC-29148 [1]). We

see various problems of such normative references.

Quality standards are incomplete. Several quality stan-

dards describe quality through a set of abstract criteria. When

analyzing the characteristics in detail, we see that there are

two different types of criteria: Some criteria, such as ambi-

guity, consistency, completeness, and singularity are factors

that describe properties of an RE artifact itself. In contrast,

feasibility, traceability, and verifiability state that activities can

be performed with the artifact. This is a small, yet important

difference: While the former can be assessed by analyzing

just the artifact by itself, the latter describe a relationship

of the artifact in the context of its usage. Yet this usage

context is incompletely represented in the quality standards:

For example, why is it important that requirements can be

implemented (feasible in the terminology of ISO-29148) and

verified, but other activities, such as maintenance, are not

part of the quality model? Therefore, we argue that normative

standards do not consider all activities systematically, and thus,

are missing relevant quality factors.

Quality standards are not context-dependent. One could

go even further and ask about the value of some artifact-based

properties such as singularity or formality. Still widely cited

quality models of the past [2] proclaimed that (all) projects

should strive towards formalized requirements. What is the

purpose and reason behind such a property? A normative

approach does not provide rationales. This is different for

activity-based properties, such as verifiability, since these

properties are defined by their usage: If we need to verify the

requirements, properties of the artifact that increase verifiabil-

ity are important. In particular, we need to understand up-front

how we want to verify the requirements. For a formal veri-

fication, formalized requirements are a reasonable approach.

For manual testing, however, formalized requirements might

actually make them harder to understand and, therefore, harder

to test. This example shows that, in contrast to the normative

definition of quality in RE standards, RE quality usually

depends on the usage context.

Quality standards lack precise reasoning. The standards

remain abstract and vague when defining the criteria. Only

for some criteria, such as ambiguity, the standards provide a

detailed list of factors to avoid (e.g., comparatives). However,

even then, the concrete impact of these factors (such as

comparatives) onto the abstract criteria (such as ambiguity)

remains unclear.

Set of Reqs. /

Reqs. Document

Complete

Aﬀordable

Bounded

Consistent

Unambiguity

Clear Structure

Modifiability and

Extensibility

Traceability

Requirements

Language Criteria

Subjective

Language

Vague Pronouns

Superlatives

Ambiguous Adverbs

and Adjectives

Open-ended, non-

verifiable. Terms

Comparatives

Negatives

Statements

Loopholes

Incomplete

References

Short Sentences

and Paragraphs

One Req. per

Sentence

(Individual)

Requirements

Necessary

Implementation Free

Unambiguous

Consistent

Complete

Singular

Feasible

Traceable

Verifiable

Agreed

Understandable

ISO 29148

Characteristic

ISO 29148 & IREB

Characteristics

IREB

Characteristics

Key:

Fig. 1: This figure compares the quality character-

istics of ISO 29148 and the IREB syllabus. Blue

characteristics are shared characteristics, orange and

green characteristics appear only in one of the

standards.

II. SIDEBAR: COMPARISON OF RE QUALITY

STANDARDS

To get a taste of current RE quality standards,

we compare the ISO/IEC/IEEE-29148 [1] quality

standard with the definition of quality attributes

from the curriculum of the International Require-

ments Engineering Board (IREB), a certification

also widely used in industry.

Both standards define a quality model through

a simple list of characteristics. According to the

standards, good requirements documents are those

in which these characteristics are present. The stan-

dards share nine of the characteristics (see Fig. 1),

mostly those characteristics defined in earlier litera-

ture and standards, such as the IEEE 830. However,

the standards disagree on more characteristics than

they agree on. In particular, the standards disagree

when it comes to concrete language criteria. Un-

fortunately, even when the standards agree on the

characteristics, the definitions of these character-

istics vary in specific details. Take, for example,

consistency. While the IREB definition only con-

siders disagreeing requirements as inconsistent, the

ISO 29148 definition of inconsistency also includes

duplication issues and terminological deficiency. In

addition, the IREB assesses quality characteristics

on a continuous scale, whereas the definitions of the

ISO standard suggest a Boolean interpretation.

At a glance, both standards share the same

approach towards quality, but their details differ

tremendously. This is especially true for the con-

crete, assessable language criteria. We argue that

these differences indicate two problems. First, the

missing agreement on the level of concrete language

criteria indicates that we do not yet know what is

good or bad quality, and that we have little to no

established understanding of the impacts of concrete

language criteria. Second and even more problem-

atic, the missing agreement at the level of abstract

quality characteristics indicates there is neither an

established understanding about nor an established

approach towards quality for RE artifacts as a whole.

Other authors have also noticed these deficien-

cies and came up with quality models that con-

sider the viewpoint of the user. For example, Lind-

land et al. [3] define different facets of quality

(syntactic, semantic, pragmatic), where pragmatic

quality emphasizes the extent to which the re-

quirements model is understood by the user. Deis-

senboeck et al. define an activity-based quality

model [4] for quality-in-use characteristics of source

code, such as maintainability. Our work details the

pragmatic quality facet through the use of activity-

based quality models.

III. GOALS OF REQUIREMENTS ENGINEERING

Let us take a step back. If we want to get to the bottom

of RE artifact quality, we need to reconsider the goals of

requirements engineering itself since RE artifacts should even-

tually support the goals of RE. Following the definitions of the

goals of RE as understood by Glinz [5, p.18], we understand

quality in RE as the degree to which the following goals are

sufficiently fulfilled for system stakeholders and the project

team:

1Understand stakeholders’ needs: High quality in RE

is the degree of correct and complete understanding of

the goals, expectations, and constraints of the system

stakeholders.

2Achieve agreement: High quality in RE is the degree of

agreement on a system that manifests the consensus of

all system stakeholders. To this end, high quality in RE

correctly prioritizes requirements, and ensures that a best-

possible solution is derived for the system stakeholders’

needs (iteration between problem and solution space, see

twin-peaks model [6]).

3Create the same mental model between all system

stakeholders: High quality in RE is the degree to which

these system stakeholders’ needs and the derived consen-

sus are communicated correctly and completely between

all involved system stakeholders in the project.

4Structure & manage requirements-based activities:

Many project activities are structured along the system

stakeholders’ needs, e.g., in the form of requirements.

Some exemplary activities are estimating the costs, plan-

ning the implementation of the system, developing the

system, or testing the system. Consequently, high quality

in RE is the degree to which engineers working with

the requirements (i.e., the information) can efficiently

and effectively use the requirements to execute their

requirements-based activities [7].

RE artifact quality should be defined and assessed with

respect to the achievement of these goals.

IV. A DIFFERENT VIEW ON REQUIREMENTS QUALITY

Gross and Doerr [7] argue that in order to create high-

quality requirements documentations that fit the specific de-

mands of successive document stakeholders, the research

community needs to better understand the particular infor-

mation needs of downstream development roles. We follow

this idea but relate artifact quality to development activities

and not to document stakeholders because information needs

of stakeholders may change depending on the activities they

have to perform in a specific context.

In the following, we describe our approach towards require-

ments artifact quality. We first describe the basic concepts, then

the detailed model, and finally an exemplary quality factor.

A. The Idea

We postulate that creating an RE artifact is rarely an end in

itself, but just a means to reach the project’s goals. In particu-

lar, they are a tool to reach the goals of RE, as described in the

previous section. Following this line of thought, the purpose

of a requirements artifact is to support the stakeholders in

whatever activities they are performing in the project. This

change of view means that it is unreasonable to talk about

good or bad RE artifacts in general. What is good and what is

bad must always be assessed with respect to the given context.

More specifically, good quality depends on the RE artifact

stakeholders and the activities that they conduct with the RE

artifacts. In fact, we argue that common quality criteria, even

completeness and correctness, have to be rethought from a

quality-in-use perspective. This contributes a novel view on

requirements engineering artifact quality, which discusses RE

artifact quality from a quality-in-use viewpoint.

B. The Model

To define RE artifact quality, we designed Activity-Based

RE artifact Quality Models (ABRE-QMs). To describe the

structure of ABRE-QMs, we provide an ABRE-QM meta

model that introduces the concepts needed to describe an

ABRE-QM (see Fig. 2 left):

An artifact is a documented collection of requirements

entities, which is produced during an RE process. An example

for an artifact is a use case document.

An entity is a coherent documented information. An entity

can be an information content item, but can also be further

decomposed, for example into the linguistic components of

such a content item. Examples for entities are a use case, an

alternative flow, or a step within the flow.

A stakeholder role is the role of someone with an interest

in the RE artifact [8], such as a test engineer. Each role

can include other roles that are more generic. For example,

both test engineer and developers are also readers of the

requirements artifact. Therefore, quality factors that affect the

activity read, affect all readers of the artifact, including test

engineers and developers through their included generic role

reader. This allows combining shared activities that multiple

stakeholders must execute.

An activity is an invested effort, which involves one or more

of the aforementioned artifacts, such as creating test cases, and

one or more of the aforementioned stakeholder roles, such

as the test engineer. An activity can be broken down into

subactivities. For example, the testing activity is decomposed

into creating,running, and maintaining test cases.

A quality factor is a property that is or is not present in an

entity. This property must be objectively assessable through a

measure to be used for quality control.

An impact is an explicit relation between a quality factor

and an activity. The impact influences either effectiveness or

efficiency of that activity. This impact is explicitly discussed

through: First, a reason, i.e., an argumentation why the pres-

ence of a specified characteristic (the quality factor) of an

artifact impacts the associated activity; second, consequences

on costs, schedule or quality of the developed system; and

third, a source from which this impact was derived and which

can provide further information, i.e., a requirements quality

standard or corporate guidelines.

A context factor influences the impact of a quality factor.

For example, the problematic impact of a passive voice re-

quirement varies, depending on the background of the reader.

If the reader has no or few domain knowledge, the passive

voice has a stronger impact. In contrast, in cases where the

reader is well aware of the domain and the ideas of the system,

the impact can be less problematic. Context factors can be

human, process or tool factors.

An assessment is a description for evaluating an entity

against a quality factor. The application of an assessment

against an entity results in a (potentially empty) set of quality

defects (cf. [4]). We see three potential categories of assess-

ments: manual, automatic, and semi-automatic assessments.

C. An Exemplary Quality Factor

To foster understanding, this section provides an exemplary

excerpt of an ABRE-QM (see Fig. 2 right). The example shows

the definition of one quality factor, namely the presence of

explicit steps in a use case flow.

1. Artifacts and entities: Ause case document is a common

artifact for specifying functional requirements to software

systems. A use case usually contains a basic flow, which is a

sequence of steps that describes how the user interacts with

the system.

2. Stakeholder roles: For the sake of simplicity, in this

example, we consider only test engineers.

3. Activities: When we analyze how a test engineer pro-

cesses the use case document in a specific project, we discover

that, among other activities, the test engineers goes through the

use case steps and creates test step(s) for the use case’s basic

flow.

4. Quality factors: It is considered good practice in use

cases to explicitly separate each step instead of describing the

whole basic flow in one text block. With the aforementioned

context and activity in mind, we understand why a use case

with this quality factor is considered higher quality: The

test engineer can directly translate the use case steps to test

steps. Therefore, the test engineer’s task of creating a test

sequence can be executed more effectively (and maybe also

more efficiently) when the factor is present in the use case.

Fig. 2 explicates this reasoning through a positive (’+’) impact

in an ABRE-QM. Please note, that for simplicity, we only

discuss one of the impacts of this quality factor here.

5. Context factors: One could consider the applied tool

environment to be a context factors. Depending on the concrete

tools in use, the translation is more or less efficient.

6. Assessment: One could discuss various types of as-

sessments, depending on the tools used. An easy-to-apply

assessment is a manual review, which can spot this quality

defect. In addition, for various requirements management

tools, one could discuss automatic (or at least semi-automatic)

methods through automatic analysis of the use case’s structure.

This example shows the definition of one quality factor. An

ABRE-QM is a composition of a set of such quality factors

Artifact

Entity

Quality

Factor

Stakeholder

Role

Activity

Impact

performs

impacts

present

contains

Context

Factor

Assessment

influenced by

evaluated by

includes generic role

consists of

Artifacts & Entities

Use Case

Document

Basic Flow

Step is explicitly

separated

Test Engineer

(T.E.)

Create Test

Steps

Rationale: T.E. can translate

steps

performs

impacts

present

contains

Tool-Features

Manual Review

inﬂuenced by

evaluated by

Use Case

contains

Stakeholder & Activities

Quality Factors & Impacts

Fig. 2: This figure shows the ABRE-QM meta model (left) and a simple example of a quality factor in an ABRE-QM (right).

The meta model consists of artifacts and their decomposition into entities, quality factors and their impact on activities, which

are performed by certain stakeholder roles. Impacts are influenced by context factors. Lastly, quality factors are evaluated by

assessments. The example discusses why explicitly separated steps in basic flows of use cases are considered good quality. In

this example, we discuss the impact on creating test steps, i.e., explicitly separated steps in basic flows allow more efficient

and effective creation of test steps through reuse.

Req. Engineer Implementer Tester User

Implement

All Stakeholders

Create

Maintain

Find

Understand

Perform

Use Case

Basic Flow

Name

Preconditions

Postconditions

Step UI Design Details

Rationale:

A visual representation of a

UI supports the stakeholder

in understanding, because it

makes the Use Case more

concrete.

Rationale:

Visual elements change

more often than the way

how a user interacts with

the system. Therefore, UC

with UI detail must be

changed more often.

Rationale:

The tester can associate the

UC directly to the UI which

makes running the test case

easier.

Rationale:

A user might have to use a

suboptimal UI, because it

was determined early in

the process.

Run

Test

Derivate

Test Idea

Impacts

Stakeholder & Activities

Artifacts & Entities

Fig. 3: This figure shows a more complex ABRE-QM. It discusses positive as well as negative impacts of UI design details

in use cases: On the one hand, UI details in use case descriptions eases understanding the use case and running tests. On the

other hand, it makes maintaining the use cases less efficient and might lead to suboptimal design, which makes it harder for

the user to perform the use case.

with their respective relations. Fig 3 shows a more extensive

example, where a quality factor has positive and negative

impacts onto activities and needs to be evaluated in context.

This model enables researchers to provide practitioners with

a precise definition of what they consider to be good or bad

quality, why (i.e., due to which consequences) and in which

context (i.e., based on which activities). Practitioners can then

use such a precise quality model and, based on artifacts,

activities and impacts, decide which quality characteristics are

relevant for their context.

V. APPLICATIONS IN RESEARCH AND PRACTICE.

We have applied the proposed meta model for different

purposes. The meta model proved beneficial in several contexts

that are discussed in the following.

Activity-based RE Guidelines. Nowadays, many compa-

nies have generic guidelines to help employees improve their

requirements and to create a baseline for quality. We argue

that guidelines that are defined in an activity-based manner

could help to make these guidelines more complete, precise,

and specific for their context. In a first study [9], practitioners

reported that a translated guideline helps to both discuss

validity of the existing rules and to create guidelines that are

more complete.

Activity-based Tailoring of Requirements Templates. Re-

quirements templates are blueprints that determine the syntac-

tic structure of a single requirement. One reported advantage

of requirements templates is that they facilitate requirements

Loading more pages...