Document [original]

Special Issue Article

Validation of simulation models in the

context of railway vehicle acceptance

Oldrich Polach

, Andreas Bo

¨ttcher

, Dario Vannucci

¨rgen Sima

, Henning Schelle

, Hugues Chollet

Gernoth Go

¨tz

, Mayi Garcia Prada

, Dirk Nicklisch

Laura Mazzola

, Mats Berg

and Martin Osman

Abstract

The evaluation of a reliable validation method, criteria and limit values suitable for model validation in the context of

vehicle acceptance was one of the objectives of the DynoTRAIN project. The presented investigations represent a

unique amount of testing, simulations, comparisons with measurements, and validation evaluations. The on-track meas-

urements performed in four European countries included several different vehicles on a test train equipped to simul-

taneously record track irregularities and rail profiles. The simulations were performed using vehicle models built with the

use of different simulation tools by different partners. The comparisons between simulation and measurement results

were conducted for over 1000 simulations using a set of the same test sections for all vehicle models. The results were

assessed by three different validation approaches: comparing values according to EN 14363; by subjective engineering

judgement by project partners; and using so-called validation metrics, i.e. computable measures developed with the aim

of increasing objectivity while still maintaining the level of agreement with engineering judgement. The proposed valid-

ation method uses the values computed by analogy with EN 14363 and provides validation limits that can be applied to a

set of deviations between simulation and measurement values.

Keywords

Validation, railway vehicle model, simulation, running dynamics, vehicle acceptance, certification

Date received: 22 April 2014; accepted: 20 August 2014

Introduction

Railway vehicle acceptance is one of the signiﬁcant

cost and time drivers during the acquisition of railway

rolling stock. Multi-body simulation tools, which are

widely used in rolling stock design and development

to conduct a wide range of investigations including

the prediction of test results, can contribute to

reduce the time and cost of the testing for the accept-

ance of running characteristics. Meanwhile, the reli-

ability of simulations is becoming widely recognised

and the opportunity to replace some physical tests by

computer simulations has been recently considered in

standards and product speciﬁcations. However, a reli-

able validation of the simulation model is the crucial

condition when considering the application of simu-

lations in the vehicle acceptance context.

The validation of a computer simulation model is a

process of determining the degree to which the model

is an accurate representation of the real world from

the perspective of the intended uses of the model.

contrast with the veriﬁcation, which is primarily dedi-

cated to the checking of the multi-body simulation

code and conducted by the code developers, model

Bombardier Transportation (Switzerland) AG, Switzerland

Alstom Transport Deutschland GmbH, Germany

Ansaldobreda, Italy

Siemens AG, Germany

TU Berlin, Germany

IFSTTAR, France

Bombardier Transportation GmbH, Germany

CAF, Spain

DB Netz AG, Germany

Politecnico di Milano, Italy

KTH, Sweden

RSSB, UK

Corresponding author:

Oldrich Polach, Bombardier Transportation (Switzerland) AG,

Zu¨rcherstrasse 39, Winterthur 8400, Switzerland.

Email: [email protected]t.bombardier.com

Proc IMechE Part F:

J Rail and Rapid Transit

2015, Vol. 229(6) 729–754

!IMechE 2014

Reprints and permissions:

sagepub.co.uk/journalsPermissions.nav

DOI: 10.1177/0954409714554275

pif.sagepub.com

validation has to be carried out by the model devel-

oper and considers the particular model stage and the

particular intended application of the model. The val-

idation consists in comparisons with measurements to

assess the quantitative accuracy of the simulation

model in regard to the intended application, i.e. the

simulations using the validated model. Simply said,

the validation should check if the model is suitable

for the intended simulations, i.e. is it ‘ﬁt for purpose’.

The comparison with measurements used for model

validation should take into account all uncertainties,

errors and scatter of conditions inﬂuencing both meas-

urement as well as simulation: the errors of running

dynamics measurement, the errors in the measurement

of track layout and track irregularities, measure-

ment of rail proﬁles and wheel proﬁles, as well as the

scatter of the test conditions, e.g. friction coeﬃcient

between wheel and rail. The validation assessment

should also take into account the number of repeated

tests used for validation and their reproducibility.

The surveys dedicated to validation of railway

vehicle models by Cooperrider and Law

and by

Gostling and Cooperrider

are both from the advent

of modern computer simulation techniques and dis-

cuss the veriﬁcation of the simulation tools/software,

rather than model validation. Computer simulations

are widely used in the design of railway rolling stock

and in research studies; however, progress in valid-

ation methodologies is rather limited. A number of

publications present particular comparisons between

simulation and measurement and document the valid-

ation of a particular simulation model, e.g. a valid-

ation of tramcar vehicle model,

validation of the

critical speed of a vehicle as it negotiates a large

radius curve

or validation of a tilting train.

However, no systematic investigations have been pre-

sented regarding a validation methodology that con-

siders the simulation of railway vehicles. The state-of-

the-art papers by Evans and Berg

from 2009 as well

Bruni et al.

from 2011 provide some hints regarding

the validation of multi-body railway vehicle models.

Experience with the validation of railway vehicle

models in the context of the vehicle acceptance pro-

cess has been gained over many years in the UK and

resulted in the Railway Group Standard Guidance

Note GM/RC2641.

A vehicle model validated

against stationary tests based on the protocols in

GM/RC2641 can be used in the UK for the assess-

ment of the resistance of railway vehicles to derail-

ment based on the Railway Group Standard GM/

RT2141.

This model validation method has also

been incorporated as recommended practice in the

European standard EN 15273-2 that deals with vehi-

cle gauging.

The validation experience gained by dynamics spe-

cialists in the UK has been used during the prepar-

ation of the model validation process described in

UIC 518.

Furthermore, two model validation trials

were conducted by this committee. The experience

with one of them dealing with the simulations of a

locomotive acceptance tests is published in Jonsson

et al.

The results of the second validation trial con-

cerning a freight wagon with Y25 bogies were pre-

sented and discussed in the framework of the

DynoTRAIN project.

The recent revision of prEN 14363

includes the

possibility to use computer simulations under follow-

ing conditions.

1. Extension of the range of test conditions where the

full test programme has not been completed.

2. Approval of vehicles following modiﬁcation.

3. Approval of new vehicles by comparison with an

already approved reference vehicle.

4. Investigation of dynamic behaviour in the case of

fault modes.

The requirements speciﬁed for the model validation in

prEN 14363 originate from the investigations con-

ducted during the preparation of UIC518 as well as

from the experience gained with the use of simulations

in the UK.

Unfortunately, neither UIC 518 nor prEN 14363

contain a speciﬁcation of the allowable diﬀerences

between simulation and on-track test results. Due to

the lack of quantitative criteria, an assessment by an

independent reviewer is required to ensure that the

model provides a suﬃcient representation of reality

for the intended application. To be able to replace

this requirement was one of the main objectives for

work package 5 (WP5) of the DynoTRAIN project.

Clear, quantitative and measurable criteria and

limit values to assess the diﬀerences between simula-

tion and measurement (also called matching error

limits) in the model validation process represent a cru-

cial requirement when applying simulations to reduce

the amount of physical testing during the vehicle

acceptance process. Such quantitative limits enable

the specialist carrying out simulations to: understand

if a particular model fulﬁls the validation require-

ments or if it needs an improvement; to visualise the

model weaknesses; and to motivate the specialists to

improve their model if needed. Unambiguous quanti-

tative validation criteria and limits ensure that all

vehicle models used in the vehicle acceptance context

have achieved a suﬃcient level of quality.

The objectives of DynoTRAIN WP5 were as

follows.

1. To review the state of the art of building and val-

idation of multi-body railway vehicle models.

2. To test vehicle models by comparisons between

simulations and measurements.

3. To specify the requirements for validation of vehi-

cle models in the context of vehicle acceptance.

The DynoTRAIN WP5 investigations were struc-

tured into ﬁve tasks. The investigations started

730 Proc IMechE Part F: J Rail and Rapid Transit 229(6)

with Task 1 dedicated to the state of the art of vehi-

cle modelling and validation. The review of

suspension and vehicle modelling was summarised

in the state-of-the-art paper presented during the

IAVSD Symposium in Manchester in 2011.

Questionnaires and presentations about model valid-

ation experience showed that the validation is typic-

ally carried out as a synthesis of stationary tests and

on-track measurements, sometimes combined with

validation of component models. Measured track

irregularities and rail proﬁles from along the test

route during the on-track tests are often not avail-

able. This missing data are usually mentioned as the

reason for the observed deviations between simula-

tions and measurements.

Task 2 of DynoTRAIN WP5 was dedicated to

investigations about suspension modelling. It pro-

vided a variety of comparisons and allowed improved

insight in to the modelling of suspension components

(rubber components, suspension with friction, viscous

dampers, and air springs); see the presentations

of some of the results in Mazzola

and Mazzola

and Berg.

The experience gained in Tasks 1 and 2 was

used when modelling the vehicles evaluated in the val-

idation investigations in DynoTRAIN. The prepar-

ation of vehicle models and the identiﬁcation of

uncertain or unknown parameters by comparisons

with stationary tests was the topic of Task 3. Tasks

4 and 5 were dedicated to validation studies and ana-

lyses, which resulted in the proposed new validation

approach.

The presented investigations conducted in

DynoTRAIN WP5 represent a unique body of work

regarding the validation of railway vehicle models in

the context of vehicle acceptance. The measurements

with a test train with several diﬀerent vehicle types

conducted in four European countries and equipped

so as to be able to simultaneously record track irre-

gularities and rail proﬁles

were compared with a

large set of simulations. The validation evaluations

carried out in the framework of the presented inves-

tigations were performed using several vehicle models,

built by seven project partners using three diﬀerent

simulation tools. The proposed process, the criteria

and the validation limits are based on a large investi-

gation using the state of the art in both modelling and

simulation approaches.

The aim of this article is to present the proposed

validation method and to explain the investigations

that lead to this ﬁnal proposal. The rest of this

paper is structured as follows. The next section pre-

sents the tests used for evaluation, simulation models

and model conﬁgurations with diﬀering input param-

eters, selection of simulation input parameters and

test sections selected for comparisons of simulations

and measurements. The section ‘Simulation output

and comparisons with measurements’ describes the

comparisons investigated in regard to deﬁning the

model validation approach. The section ‘Evaluation

of the validation method, criteria and limit values’

presents the evaluations related to the selection of a

suitable validation method and validation limits

(matching errors). The section ‘Proposed validation

method’ presents the proposed method, criteria and

limits for validation of vehicle models used for simu-

lations of on-track tests in the context of vehicle

acceptance. The ‘Discussion’ section is dedicated to

a discussion about the proposed validation method

and about the inﬂuence of model adjustments by com-

parisons with stationary tests. Finally, a summary and

conclusions are provided.

Validation investigations in DynoTRAIN

On-track tests used for validation

The presented model validation investigations used

on-track measurements conducted in the framework

of DynoTRAIN WP1 as well as some measurement

results provided by project partners.

The DynoTRAIN test campaign was conducted in

October 2010 with several diﬀerent vehicles that were

equipped with 10 force measuring wheelsets and sev-

eral acceleration and displacement sensors.

The

train travelled for a total of 20 days of test runs

through Germany, France, Italy and Switzerland at

speeds up to 120 km/h with freight wagons connected

and up to 200 km/h without freight wagons. A mea-

suring vehicle integrated into the test train continu-

ously recorded the track irregularities and rail proﬁle

shapes along all test runs. The test train contained the

following vehicles:

.locomotive DB BR 120;

.DB passenger coach Bim;

.empty freight wagon Sgns with Y25 bogies;

.loaded freight wagon Sgns with Y25 bogies;

.Laas freight vehicle unit consisting of two two-axle

wagons with UIC link suspension; one empty and

one fully loaded; the empty wagon was equipped

with measuring wheelsets.

In addition to the vehicles tested in DynoTRAIN,

another two vehicles were investigated using measure-

ments carried out during the running dynamic accept-

ance tests of these vehicles:

.the High-speed EMU for TCDD (Turkey) manu-

factured by CAF, measurements conducted in

2008;

.DMU IC4 for DSB (Denmark) manufactured by

Ansaldobreda, measurements carried out in 2006.

The uncertainty and error of the measurements used

in the described investigations represent the state of

the art in the vehicle approval process. There were no

investigations in DynoTRAIN WP5 dedicated to

Polach et al. 731

uncertainty of the measured data used for model

validation.

Vehicle models and model configurations

The multi-body vehicle models used for the evaluation

of the validation method were prepared by project

partners using diﬀerent simulation tools, see examples

of models built using the simulation tool Simpack in

Figure 1. Several versions of each vehicle model were

prepared using diﬀerent stages of model parameters,

track irregularities, rail and wheel proﬁles as well as

modelling depth. The diﬀering model versions are

called ‘model conﬁgurations’ in the rest of this

paper. An overview of the vehicle models used in

the presented investigations is shown in Table 1.

The originally proposed set of model conﬁgurations

exceeded the available time and project budget.

Moreover, some model conﬁgurations were not feas-

ible for some vehicles, e.g. if the measurements of

track irregularities or rail proﬁles were not available.

These facts resulted in a large variation in the number

of vehicle model conﬁgurations as can be seen in

Table 1.

The vehicle models used in the investigations rep-

resent fully nonlinear three-dimensional models, as

this is the state of the art in railway engineering and

research. Rigid bodies representing the vehicle body,

bogie frame, wheelset, axle box, etc. are connected by

springs, dampers, friction elements and bump-stops

that model the suspension components. Damper

models consist of a dashpot together with series stiﬀ-

ness. The nonlinear wheel/rail contact models use the

respective contact evaluation method and the respect-

ive version of Kalker’s computer code Fastsim imple-

mented in the utilised simulation tool.

Table 1. Overview of the multi-body simulation models used for the evaluation of the presented validation methodology.

Vehicle Project partner

Simulation

tool

Number of model

configurations

On-track tests used

for validation

Locomotive DB BR 120 Siemens Simpack 24 DynoTRAIN

IFSTTAR VOCO 4 DynoTRAIN

DB passenger coach Bim Bombardier Transportation Simpack 13 DynoTRAIN

IFSTTAR VOCO 4 DynoTRAIN

Freight wagon Sgns, empty Technical University Berlin Simpack 8 DynoTRAIN

IFSTTAR VOCO 6 DynoTRAIN

Freight wagon Sgns, laden Technical University Berlin Simpack 7 DynoTRAIN

Laas freight vehicle, empty Alstom Simpack 5 DynoTRAIN

High-speed EMU (TCDD) CAF SIDIVE 3 Provided by vehicle

manufacturer CAF

DMU IC4, coach T3 (DSB) Ansaldobreda Simpack 2 Provided by vehicle

manufacturer Ansaldobreda

DMU IC4, coach M4C (DSB) Ansaldobreda Simpack 2 Provided by vehicle

manufacturer Ansaldobreda

Figure 1. Examples of multi-body models of the vehicles tested in DynoTRAIN.

732 Proc IMechE Part F: J Rail and Rapid Transit 229(6)

The vehicle models were prepared under the part-

ners’ responsibility. The majority of data regarding

the vehicles tested during the DynoTRAIN test cam-

paign was provided by DB; the remaining information

was estimated or identiﬁed from archive material by

the partner modelling a particular vehicle. The iden-

tiﬁcation of vehicle model parameters of vehicles

tested outside the DynoTRAIN project was fully

the responsibility of the respective partner; vehicle

manufacturers also provided data obtained in their

running tests.

The initial vehicle models were prepared using the

available vehicle data without considering the results

of stationary tests. Project partners were, however,

advised to adjust the mass parameters in their model

before starting any comparisons in order to achieve a

good agreement between the wheel loads obtained

from a static model and the wheel loads measured

during the on-track tests. Then, the initial models

were adjusted with the aim of improving the agree-

ment between the on-track test results and the simu-

lation results, so that several diﬀering conﬁgurations

of the same model could be compared. The vehicle

models adjusted based on the comparisons with the

stationary tests represent other model conﬁgurations.

In order to assess the eﬀect of using actual measured

infrastructure parameters such as track layout, track

irregularities and rail proﬁles, the model conﬁgur-

ations with estimated rail proﬁles (see explanation in

the section ‘Rail proﬁles’) and estimated track irregu-

larities (see explanation in the section ‘Track layout

and track irregularities’) were also prepared and com-

pared with the on-track measurements.

A total of 78 model conﬁgurations were investi-

gated, with diﬀering levels of knowledge on vehicle

data, input parameters regarding the infrastructure,

diﬀerent usage of stationary tests and applying a

diﬀerent depth of modelling detail. Moreover, some

model conﬁgurations of the locomotive BR 120 cre-

ated by Siemens were varied in implementing the

driving torque in test sections where this locomotive

was used as a propelling vehicle. Figure 2 shows the

variety of investigated model conﬁgurations together

with the assessed quantities, which are described in

more detail in the section ‘Simulation output

and comparisons with measurements’. The eﬀect of

using the results of stationary tests for the model val-

idation in regard to the simulation of the on-track

tests, which was investigated by comparing the simu-

lations of the on-track tests using vehicle models

before and after the comparisons with the stationary

tests, is discussed in the section ‘Eﬀect of model

adjustment using stationary tests on the simulation

of on-track tests’. The eﬀects of measured and esti-

mated wheel and rail proﬁles, as well as track irregu-

larities data, on the model validation results are not

presented in this paper for the sake of brevity; readers

interested in those topics are referred to Polach

and Bottcher.

Simulation input parameters

Track layout and track irregularities. The track geometry

data were measured during the DynoTRAIN test

campaign performed by the DB track recording car

‘RAILab I’.

The data were obtained at a sampling

distance of 0.16 m and stored in binary ﬁles.

The manipulation of measured track irregularity

data into a format suitable for simulations was per-

formed by DB Netz AG. As the inertial-platform-

based RAILab I system uses a special ﬁlter algorithm

to separate long wavelengths caused by the track

layout from the track irregularities to be assessed,

the recorded data were de-coloured (transformed

Vehicle

model

Measured

Comparison with

staonary tests

Track

irregularity

Rail

proﬁles

Esmated Esmated Aer

Measured Before Detailed

Simpliﬁed

Assessment

method

Single values

Model

conﬁguraon

Assessment using quanes by analogy with EN 14363

Objecve assessment using validaon metrics

Subjecve assessment by engineering judgement

Plots

Figure 2. Overview of the model configurations and assessment methods evaluated in the framework of the presented

investigations.

Polach et al. 733

backward) using corrective ﬁlters before they were

used in the vehicle dynamics simulations.

For each of the selected track sections, the relevant

RAILab I data were transformed into the format used

in the multi-body simulation package Simpack. Two

input data ﬁles were created for each track section;

one of them containing the track layout (curvature

and cant using high-pass ﬁlters above 70 m) and the

second describing irregularities (lateral and vertical

position of the left and right rails with band-pass ﬁl-

ters between 1 and 70 m).

There were no measurements of track irregularities

available for the on-track measurements conducted

outside of the DynoTRAIN remit. Thus, the simula-

tions with vehicle models DMU IC4 and High-speed

EMU Turkey were carried out using estimated track

irregularities. This estimated track irregularity data

was used not only in the case of missing measured

data but also for comparisons regarding the import-

ance of knowledge about the track irregularities. In

the following discussions the term ‘estimated track

irregularities’ means either generated data based on

the power spectral density as in ORE B176

or mea-

sured track irregularities from other measurements.

The selection of track irregularities to be used instead

of the actual measured data was the responsibility of

the respective partner.

Rail profiles. The rail proﬁles were measured during the

DynoTRAIN test runs by means of an optical mea-

suring device

and recorded at a spacing interval of

0.25 m. For the synchronisation of the measured rail

proﬁles with all the other measuring channels, the

time stamp and counter signal provided by the track

recording car RAILab I was combined with the

odometer signal of the rail proﬁle measuring system

and both were stored together in an additional syn-

chronisation ﬁle.

The implementation of the measured rail proﬁle in

multi-body simulations generates several questions. A

typical recommendation is to use a ‘representative

proﬁle’. However, how do you identify this represen-

tative proﬁle? As the rail proﬁles in curves wear dif-

ferently on the outer and inner rails as well as in a

diﬀerent manner from straight and curved transitions,

the use of one proﬁle for each rail along the whole

investigated section will obviously provide incorrect

results either outside the full curve or in the full

curve, unless there is no wear of rails (new or newly

ground rails).

Continuously varying rail proﬁle along the track

section has been implemented in some of the simula-

tion packages; however, it is still not a state-of-the-art

procedure and thus not applied in this paper. After

several investigations and discussions regarding this

topic, it was ﬁnally agreed to calculate averaged rail

proﬁles from the measured rail proﬁles of the part of

the actual track section with constant track curvature

(i.e. one proﬁle for the left rail and one for the right

rail) and to use these averaged proﬁles for simulations

of this particular track section. Thus, the used proﬁle

may be incorrect in curved transitions and accom-

panying straight track parts. Moreover, if the actual

rail proﬁle changes along the distance, e.g. in some

longer sections, the applied averaged rail proﬁles

may not be fully representative.

The preparation of the averaged rail proﬁles was

performed by DB Netz AG. At ﬁrst the proﬁles were

smoothed and their running surfaces (down to an

appropriate proﬁle gradient) were approximated by

high-order polynomials. Then all proﬁles of the

same rail within the respective track section were ver-

tically aligned to each other at the rail top and lat-

erally at the gauge measuring point (14 mm below the

top of the rail). In order to allow for superposition of

measured track irregularities, the resulting rail proﬁles

were shifted in the lateral direction to meet the

1435 mm nominal track gauge.

For each simulation exercise a mean proﬁle for the

left rail and a mean proﬁle for the right rail were

provided by taking into account all rail proﬁles of

the track section with a constant radius, i.e. section

C–D in Figure 3. These single mean proﬁles for left

and right rails were used in simulations of the com-

plete particular section. The model conﬁgurations

with ‘estimated rail proﬁles’ used the nominal rail pro-

ﬁle and rail inclination of the particular country under

investigation. The simulations of vehicle models

DMU IC4 and High-speed EMU Turkey both

solely used the respective nominal rail proﬁles and

rail inclinations of the particular country as there

were no measurements of rail proﬁles available.

Wheel profiles. The wheel proﬁles of vehicles tested in

DynoTRAIN were measured before and after the

test campaign and the measured data were used in

the simulations. The details regarding the wheel pro-

ﬁle implementation were the individual partner’s

responsibility. The model conﬁgurations with ‘esti-

mated wheel proﬁles’ were carried out using the

designed wheel proﬁle S1002. There were no measure-

ments of wheel proﬁles available for the on-track meas-

urements conducted outside of the DynoTRAIN

project. Hence, the vehicle models DMU IC4 and

High-speed EMU Turkey used the respective nominal

wheel proﬁles: proﬁle S1002 (DMU IC4) or proﬁle GV

1/40 (High-speed EMU Turkey), respectively.

Friction coefficient between wheel and rail. The value of the

friction coeﬃcient between the wheel and rail repre-

sents an uncertain input parameter in the simulation.

The selection of this parameter was the responsibility

of the partner carrying out the simulation. All test

runs selected for validation from the DynoTRAIN

measurements were carried out on a dry rail. In

their simulations each partner used a wheel/rail fric-

tion coeﬃcient of 0.45 or 0.50 to represent those con-

ditions. A few model conﬁgurations used a lower

734 Proc IMechE Part F: J Rail and Rapid Transit 229(6)

friction coeﬃcient than 0.45 or higher than 0.50,

respectively, with the aim of testing for an improve-

ment of the agreement with the measured values. The

majority of simulations used an identical and constant

value of the friction coeﬃcient in the tread and on the

ﬂange; only a few simulations used a lower friction

coeﬃcient on the ﬂange.

The simulations of the test results provided by

vehicle manufacturers used a value of wheel/rail fric-

tion coeﬃcient of 0.45 (simulations of DMU IC4 by

Ansaldobreda) or a friction coeﬃcient of 0.35 (simu-

lations of High-speed EMU Turkey by CAF),

respectively.

Validation exercises

Comparisons between simulation and measurement

results were carried out for all vehicle models and

model conﬁgurations under the same conditions and

in the same manner as for selected representative sec-

tions of test runs, called validation exercises. One val-

idation exercise consisted of one curve passing

scenario including both transitions and parts of

straight track as shown in Figure 3. In this context

the word ‘section’ means a part of the track; it does

not mean section as in the deﬁnition in EN 14363.

A total of 17 validation exercises were selected,

representing all four track zones in EN 14363: straight

track and very-large-radius curves were represented

by four sections, large radius curves (R>600 m) by

two sections; ﬁve sections were used for small radius

curves (400 m 4R4600 m) and six for very-small-

radius curves (250 m 4R<400 m). Table 2 shows

the parameters of the test sections selected for vehicles

tested in DynoTRAIN in terms of the location, track

layout, section length as well as the speed of the test

train in the respective section. It should be noted that

the number of test sections in each test zone based on

EN 14363 reported in this article do not fully comply

with the ﬁnal recommended validation procedure,

because the procedure and the conditions to be used

were not known at the start of the investigations.

Moreover, the test conditions during the

DynoTRAIN running tests did not fully comply

with EN 14363; see Zacher and Kratochwille.

The selection of test sections considered geomet-

rical track quality (irregularities) and wheel/rail con-

tact geometry with the aim of including varying

conditions. The track sections for exercises 2, 3 and

5 were included due to a high vertical disturbance in

the track irregularities. The properties of the wheel/

rail contact geometry were assessed by the calculation

of the equivalent conicity and radial steering index

over the constant curvature sections using the mea-

sured rail proﬁles averaged over a 100 m distance

together with a nominal design wheel proﬁle S1002

and mean track gauge over the respective track sec-

tion. The deﬁnition of a radial steering index was

introduced in UIC 518

to assess the available rolling

radius diﬀerence between left and right wheels. Index

values lower than a value of one represent a contact

geometry that provides a suﬃcient diﬀerence in roll-

ing radius for self-steering wheelsets, whereas values

higher than one represent an insuﬃcient rolling radius

diﬀerence for the considered curve radius. The curve

test sections 4, 5, 7, 9, 14 and 15 show a radial steering

Evaluation based on EN 14363

Subjective assessment, calculation of validation metrics

Guiding force Y11 [kN]

63.2 63.0 63.5 63.7 63.8 63.9

Distance [km]

Simulation

Measurement

63.4 63.6

-20

Track curvature [1/km]

EBAC

Distance

Figure 3. Example of validation exercise with the specification of track sections used for different kind of assessments.

Polach et al. 735

index below one and thus a good contact geometry

regarding curving, whereas sections 1, 2, 3, 6, 12, 13,

16 and 17 give a radial steering index higher than one,

i.e. disadvantageous contact geometry conditions

regarding self-steering of wheelsets. The equivalent

conicities (calculated for a lateral displacement of

the wheelset of 3 mm) in section 8 were medium

values between 0.20 and 0.25 and in section 9 their

values were around 0.1. The sections 10 and 11 were

selected because of the occurrence of very high coni-

cities; the conicity calculated per 100 m distance

varied from medium values up to a few very high

values of around one.

As freight vehicles were included in the test train

only at speeds up to 120 km/h, the Laas wagon and

the Sgns freight wagons were missing in the runs of

the exercises 9, 10 and 11. Each simulation was per-

formed for a part of the test run called ‘part of inter-

est’ (A–F in Figure 3) and some outputs were

evaluated over this part, whereas other outputs were

solely evaluated over the part of the track with con-

stant curve radius (C–D in Figure 3).

Simulation output and comparisons with

measurements

Introduction

The simulations of selected on-track tests were evalu-

ated in the same manner by all partners conducting

simulations. This required an agreement and speciﬁ-

cation of the output data and its format.

As the aim of the validation is the application of

simulation for vehicle acceptance, a comparison of

quantities as they are measured and evaluated accord-

ing to EN 14363

was logically considered as one pos-

sible assessment method. Another typical validation

assessment is a judgement of the comparison between

the time domain signals from simulations and meas-

urements. In contradiction with the quantities based

on EN 14363, which are assessed primarily in track

sections with a constant curvature, the judgement of

time diagrams allows the assessment of the behaviour

in transitions as well as the frequency content of the

signals. A subjective judgement of time or distance dia-

grams thus represents another kind of assessment.

However, an engineering judgement is not measur-

able; the replacement of such an assessment by calcul-

able quantitative criteria is highly preferred. The

evaluation of so-called validation metrics conducted

recently by the Transportation Technology Center

motivated the DynoTRAIN project partners to

include the evaluation of the validation metrics as

the third kind of assessment. These three kinds of

validation assessment were applied to the investigated

vehicle models and model conﬁgurations as shown

schematically in Figure 2. The deﬁnition of these

assessments and agreed simulation outputs are pre-

sented in the following sections.

Assessment using values based on EN 14363

The comparisons between simulations and measure-

ments were performed using an agreed set of output

Table 2. Test runs and parameters of track sections used in the validation exercises performed with vehicles tested in DynoTRAIN.

Exercise

number Line Country

Test zone

according to

EN 14363

Curve radius

(m)

Cant

(mm)

Speed

(km/h)

Section length: whole

section A–F/constant

curvature section

C–D (m)

1 Geislingen– Westerstetten Germany 4 282 120 68 740/400

2 Geislingen– Westerstetten Germany 4 312 100 68 280/140

3 Geislingen– Westerstetten Germany 3 572 155 110 1080/320

4 Uffenheim–Ansbach Germany 3 580 150 110 870/490

5 Uffenheim–Ansbach Germany 3 581 110 110 1130/680

6 Uffenheim–Ansbach Germany 2 864 115 120 750/360

7 Uffenheim-Ansbach Germany 2 694 160 121 690/190

8 Uffenheim–Ansbach Germany 1 10 120 1760/1760

9Wu¨rzburg–Fulda Germany 1 5600/6000 75 200 3300/2644

10 Lichtenfels–Bamberg Germany 1 10 160 3200/3200

11 Lichtenfels–Bamberg Germany 1 10 160 3200/3200

12 Pisa–Firenze Italy 4 295 140 76 504/110

13 Pisa–Firenze Italy 4 292 140 76 968/771

14 Biasca–Go

¨schenen Switzerland 4 278 150 74 424/280

15 Biasca–Go

¨schenen Switzerland 4 294 142 74 384/192

16 St. Giovanni–Firenze Italy 3 442 140 90 510/250

17 St. Giovanni–Firenze Italy 3 406 150 90 651/426

736 Proc IMechE Part F: J Rail and Rapid Transit 229(6)

quantities that are used in testing based on EN 14363.

The simulation and measurement results were ﬁltered

and processed by analogy with the requirements in EN

14363 and compared against each other; this evaluation

considers the part of the track with a constant curva-

ture, i.e. section C–D in Figure 3. Table 3 shows the list

of output quantities, their ﬁltering, processing as well as

the nomenclature and unit. A total of two wheelsets

were used in the validation assessment of each vehicle,

which resulted in a total of 28 parameters related to

wheel/rail forces. The bogie accelerations were mea-

sured on the bogie frame above all the wheelsets in

the lateral direction and above the wheelsets of one

bogie in the vertical direction, resulting in a total of

12 bogie frame acceleration values (not applicable for

the two-axle wagon). The vehicle body accelerations

were measured at the ﬂoor level above both bogie

centre pins in the lateral and vertical directions resulting

in a total of eight car body acceleration values. Thus, a

total of 48 parameters per model conﬁguration and test

section (36 for the two-axle wagon) were applied con-

sisting of quasi-static as well as dynamic wheel/rail

forces and vehicle body and bogie frame accelerations.

Subjective assessments

A subjective engineering judgement is based on a

visual impression of time history plots and power

spectral density (PSD) diagrams. A selected set of

quantities consisting of 20 plots per vehicle model

conﬁguration and test section (for all vehicles apart

from the Laas freight vehicle that had a lower number

of plots) was issued and provided to project partners

for the assessment.

The following quantities were displayed and issued

in the form of distance or time plots:

.lateral wheel/rail forces (Y-forces): four diagrams

per vehicle model conﬁguration and test section;

.vertical wheel/rail forces Q(wheel loads): four

diagrams;

.ratio Y/Q: four diagrams;

.lateral accelerations of the bogie frame above

wheelsets 1 and 2: two diagrams;

.vertical acceleration of the car body above bogie 1:

one diagram;

.lateral acceleration of the car body above bogie 1:

one diagram.

The simulation as well as measurement signals were

ﬁltered using a 20 Hz low-pass ﬁlter, without any

other processing, and displayed for the whole investi-

gated test section (section A–F in Figure 3).

Moreover, PSDs of four acceleration signals were

also provided as diagrams for subjective assessments:

.lateral acceleration of the bogie frame above wheel-

set 1;

.vertical acceleration of the bogie frame above

wheelset 1;

.vertical acceleration of the car body above bogie 1;

.lateral acceleration of the car body above bogie 1.

Table 3. Output quantities used for the assessment by analogy with EN 14363.

Quantity Filtering Processing Notation Unit

Wheel/rail forces,

quasi-static values

Guiding force Low-pass filter 20 Hz 50th percentile (median) Y

qst

Wheel load Low-pass filter 20 Hz 50th percentile (median) Q

qst

Ratio Y/QLow-pass filter 20 Hz 50th percentile (median) Y/Q

qst

–

Sum of guiding forces Low-pass filter 20 Hz 50th percentile (median) Y

qst

Wheel/rail forces,

dynamic values

Guiding force Low-pass filter 20 Hz 0.15 percentile, 99.85 percentile Y

max

Wheel load Low-pass filter 20 Hz 99.85 percentile Q

max

Ratio Y/QLow-pass filter 20 Hz Sliding mean (window 2 m,

step 0.5 m)

0.15 and 99.85 percentile

Y/Q

max

–

Sum of guiding forces Low-pass filter 20 Hz Sliding mean (window 2 m,

step 0.5 m)

0.15 and 99.85 percentile

Y

max

Bogie frame acceleration,

root-mean- square

(RMS) values

Lateral acceleration Band-pass filter 0.4 to 10 Hz RMS value €

yþ

rms m/s

Vertical acceleration Band-pass filter 0.4 to 10 Hz RMS value €

zþ

rms m/s

Bogie frame acceleration,

dynamic values

Lateral acceleration Low-pass filter 10 Hz 0.15 percentile, 99.85 percentile €

yþ

max m/s

Vertical acceleration Low-pass filter 10 Hz 0.15 percentile, 99.85 percentile €

zþ

max m/s

Car body acceleration,

RMS values

Lateral acceleration Band-pass filter 0.4 to 10 Hz RMS-value €

y

rms m/s

Vertical acceleration Band-pass filter 0.4 to 10 Hz RMS-value €

z

rms m/s

Car body acceleration,

dynamic values

Lateral acceleration Band-pass filter 0.4 to 10 Hz 0.15 percentile, 99.85 percentile €

y

max m/s

Vertical acceleration Band-pass filter 0.4 to 10 Hz 0.15 percentile, 99.85 percentile €

z

max m/s

Polach et al. 737

These signals were ﬁltered by a 20 Hz low-pass ﬁlter

for PSDs in the frequency range 0 – 10 Hz.

The project partners were asked to assess the dia-

grams displaying the comparison of the measurement

and simulation signal quantities by a simple binary

assessment ‘Yes/No’. Assessing a diagram with a

Yes means that for the displayed signal quantities of

the particular diagram the assessor considers the

model as validated and vice versa.

As the form of the diagram (size, number of com-

pared curves, scaling of axis, colours, position of

curve in front or background, respectively) can inﬂu-

ence the result of this judgement, it was ﬁrst necessary

to select and agree on a suitable form for the dia-

grams. It was decided to present only two curves in

each diagram, comparing measurement and simula-

tion of a quantity’s distance or time history. The selec-

tion of the scaling of the vertical axis turned out to be

a more diﬃcult question. Figures 4 to 7 show exam-

ples of comparisons between simulation and measure-

ment data for the following four investigated vehicle

models:

.the locomotive DB BR 120 investigated by

Siemens;

.the DB passenger coach Bim investigated by

Bombardier Transportation;

.the loaded freight wagon Sgns investigated by the

Technical University Berlin;

.the Laas freight vehicle investigated by Alstom.

Figure 4 presents the guiding force on the outer wheel

of the leading wheelset obtained for test section 1

(curve radius 282 m) using the same scale for all vehi-

cles to illustrate the diﬀerences in the level of the

investigated values. As can be seen, the position of

the signal is not exactly the same in regard to the

distance. This may lead to slight diﬀerences when cal-

culating the values in the speciﬁed interval with a con-

stant curvature. Other eﬀects can be observed, such as

a signal oﬀset (locomotive model created by Siemens).

For illustration purposes, the same results are dis-

played in Figure 5 in the original form as submitted

for the subjective assessment, together with the per-

centage of positive assessments by project partners.

The diagrams were assessed by 10 partners, i.e. 40%

means that four of the 10 partners considered the pre-

sented results as documenting a validated model. In

addition to a diﬀering scale on the vertical axis, the

Laas freight vehicle results are displayed as a time

diagram over a longer interval compared with the

other vehicles that are presented as distance diagrams.

It can be seen that the assessment of the very light

two-axle wagon Laas is rather strict compared with

the results of the locomotive or loaded freight wagon.

Figure 6 shows the ratio Y/Q at the outer wheel of the

leading wheelset for test section 2 (curve radius 312 m)

and Figure 7 shows the vertical car body acceleration

from test section 8 (straight). Although the Y/Q ratio

has a similar level for all vehicles, the accelerations

signiﬁcantly vary. This opens the question of the selec-

tion of scaling for the presentation of results. When

using an equal scaling, the comparison for light vehi-

cles can barely be assessed as they have low vertical as

well as lateral wheel/rail forces. Also, the assessment

of the acceleration of soft-suspended vehicles is diﬃ-

cult. Alternatively, the use of automatically adjusted

scaling leads to the impression of large diﬀerences,

even if the values are very small. To allow bet-

ter assessments, it was proposed to the project part-

ners to use a ﬁxed scaling with one of three

speciﬁed scale groups; however, the ﬁnal decision

was up to the partner conducting the simulation.

Consequently, the values presented in the evaluated

diagrams are sometimes rather small, whereas in other

cases the peaks are outside of the diagram;

both eﬀects make the subjective assessment more

diﬃcult.

Validation metrics

A validation assessment in terms of a comparison of

time histories between simulated and measured values

generates questions about the subjectivity of this

assessment as stated in the previous section.

Validation metrics represent an approach to quantify-

ing the comparisons of time history curves with the

intent of minimising the subjectivity while still main-

taining a correlation with experts’ opinion.

They are

developed and mainly used for comparisons between

simulation and measurement in the context of model

validation.

A possible metric that could be used to compare

the time domain diagrams is the integral approach

introduced in 1984 by Geers. Integrals of two wave

forms to be compared are computed and used to

evaluate the diﬀerence in the magnitude and phase

of the wave forms expressed in terms of magnitude,

phase and comprehensive error factors, with small

values of the error factor representing good agree-

ment. The magnitude as well as phase form of the

error factors was later adapted by Russell.

The

new phase form by Russell was combined with the

1984 Geers’ metric by Sprague and Geers.

using the same sampling rate and the same length of

time or distance interval for the compared measure-

ment and simulation signals, the deﬁnitions of error

factors proposed in Sprague and Geers

can be

expressed by the following formulas.

Sprague and Geers magnitude error factor

MSG ¼ﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ

i¼1c2

i¼1m2

s1ð1Þ

where the c

are the simulated values and m

are the

measured values.

738 Proc IMechE Part F: J Rail and Rapid Transit 229(6)

Sprague and Geers phase error factor

PSG ¼1

cos1Pn

i¼1cimi

ﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ

i¼1c2

iPn

i¼1m2

ð2Þ

Sprague and Geers comprehensive error factor

CSG ¼ﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ

SG þP2

qð3Þ

The error factors of the validation metrics proposed

by Sprague and Geers and by Russell were calculated

by the project partners to allow comparisons between

simulations and measurements provided in the time

and distance domain plots and used for the subjective

assessment by the partners. The evaluations later

focussed on the validation metric by Sprague and

Geers which appeared to be more promising.

Guiding force Y11 [kN]

63.2 63.3 63.4 63.5 63.6 63.7 63.8 63.9

Distance [km]

Measurement

Simulation

Freight vehicle Laas, empty

Passenger coach Bim

63.2 63.3 63.4 63.5 63.6 63.7 63.8 63.9

Distance [km]

-20

100

-20

100

Guiding force Y11 [kN]

Freight wagon Sgns, laden

Locomotive DB BR120

-20

100

-20

100

63.2 63.3 63.4 63.5 63.6 63.7 63.8 63.9

Distance [km]

63.2 63.3 63.4 63.5 63.6 63.7 63.8 63.9

Distance [km]

Figure 4. Validation examples: guiding force on the outer wheel of leading wheelset, exercise 1, Germany, Geislingen–Westerstetten

line, curve radius 282 m, cant 120 mm, speed 68 km/h.

Polach et al. 739

Evaluation of the validation method,

criteria and limit values

Evaluation of the assessment based on EN 14363

The assessments based on quantities speciﬁed in EN

14363 were carried out using a common preliminary

set of validation limits, which were evaluated from

the proposals provided by the project partners.

These proposals signiﬁcantly deviated not only in

the proposed limit values but also in principle as

shown schematically in Figure 8 that displays the

areas fulﬁlling the proposed validation condition. If

the simulated value S

and measured value M

are

identical, the point is on the diagonal line. A deviation

from this diagonal line represents a deviation between

the simulation and measurement.

Guiding force Y11 [kN]

63.2 63.3 63.4 63.5 63.6 63.7 63.8 63.9

Distance [km]

Measurement

Simulation Freight vehicle Laas, empty

Passenger coach Bim

0102030405060

Time [s]

-60

-20

-40

-30

-10

-20

Guiding force Y11 [kN]

Freight wagon Sgns, laden

Locomotive DB BR120

-100

100

-50

63.2 63.3 63.4 63.5 63.6 63.7 63.8 63.9

Distance [km]

63.2 63.3 63.4 63.5 63.6 63.7 63.8 63.9

Distance [km]

-100

100

-50

80%

50%

40%

50%

Figure 5. Validation examples and subjective assessments. Diagrams from the exercise 1 (as in Figure 4) in the form used for the

subjective assessments by project partners. The values in the circles of each diagram display the percentage of positive assessments.

740 Proc IMechE Part F: J Rail and Rapid Transit 229(6)

The following diﬀering deﬁnitions of the limit con-

dition were proposed.

1. Deviation limit as a percentage of the measured

value (relative deviation limit): see Figure 8(a).

2. Constant deviation limit (absolute deviation

limit): see Figure 8(b).

3. Deviation limit decreasing with the measured

value increasing towards the limit for vehicle

acceptance based on EN 14363, but not falling

below a minimum absolute limit at high measured

values, as shown in Figure 8(e).

Some partners proposed combinations of previous

principles: a relative limit combined with an absolute

deviation limit as shown in Figure 8(c); the addition

of an absolute and a relative deviation limit as dis-

played in Figure 8(d); or an absolute (constant)

Ratio Y/Q11 [-] Ratio Y/Q11 [-]

Freight vehicle Laas, empty

Passenger coach Bim

65.10 65.15 65.20 65.25 65.30 65.35

Distance [km]

-0.2

0.8

1.0

0.2

0.6

0.4

65.10 65.15 65.20 65.25 65.30 65.35

-0.2

0.8

1.0

0.2

0.6

0.4

Ratio Y/Q11 [-] Ratio Y/Q11 [-]

Freight wagon Sgns, laden

Locomotive DB BR120

65.10 65.15 65.20 65.25 65.30 65.35

Distance [km]

0.8

0.2

0.6

0.4

65.10 65.15 65.20 65.25 65.30 65.35

-0.2

0.8

1.0

0.2

0.6

0.4

-0.2

1.0

Measurement

Simulation

Figure 6. Validation examples: the Y/Q ratio for the outer wheel of the leading wheelset, exercise 2, Germany,

Geislingen–Westerstetten line, curve radius 312 m, cant 100 mm, speed 68 km/h.

Polach et al. 741

deviation limit that changes with the measured value

as shown in Figure 8(f).

A reasonable justiﬁcation can be provided for each

of the diﬀerent proposals. Any deviation or error is

usually considered in regard to relative deviation, thus

supporting the approach in Figure 8(a). However, as

the vehicle model is intended to be used for simulation

of vehicle acceptance tests, it is important to achieve

good agreement especially for values that are close to

their limit values for vehicle acceptance, hence sup-

porting the contradicting approach in Figure 8(e).

Finally, it was agreed to use constant validation

limit values (limits for absolute deviation

simulation - measurement), which is quite simple and

at the same time the most appropriate compromise

for the proposals discussed during the investigations.

Freight vehicle Laas, empty

Passenger coach Bim

Car body vert. acceleration [m/s

]

58.4 58.2 58.0 57.8 57.6 57.4 57.2 57.0

Distance [km]

Car body vert. acceleration [m/s

]

-6

-4

-2

56.8

58.4 58.2 58.0 57.8 57.6 57.4 57.2 57.0 56.8

-6

-4

-2

Freight wagon Sgns, laden

Locomotive DB BR120

Car body vert. acceleration [m/s

]

58.4 58.2 58.0 57.8 57.6 57.4 57.2 57.0

Distance [km]

Car body vert. acceleration [m/s

]

56.8

58.4 58.2 58.0 57.8 57.6 57.4 57.2 57.0 56.8

-6

-4

-2

-6

-4

-2

Measurement

Simulation

Figure 7. Validation examples: vertical acceleration of vehicle body over the leading bogie (wheelset), exercise 8, Germany,

Uffenheim–Ansbach line, straight track, speed 120 km/h.

742 Proc IMechE Part F: J Rail and Rapid Transit 229(6)

A set of preliminary validation limits based on the

partners’ proposals was agreed and then applied for

comparisons of model conﬁgurations and for the

investigation of the possible approach for model

validation.

Evaluation of subjective assessments

Comparisons of measurements and simulations of

quantities presented in diagrams were assessed by

the project partners using a simple ‘Yes/No’

method. Due to the large amount of results presented

in the form of diagrams, only a part of the results

could be assessed by project partners. The following

model conﬁgurations of vehicles tested in

DynoTRAIN were selected for this subjective assess-

ment, all representing the initial vehicle models.

1. Conﬁguration F1 using measured data of wheel

and rail proﬁles as well as measured track

irregularities.

2. Conﬁguration D1 using estimated (design) wheel

and rail proﬁles and measured track irregularities.

3. Conﬁguration E1 using measured wheel proﬁles,

estimated (design) rail proﬁles, measured track

irregularities.

4. Conﬁguration C1 using measured wheel and rail

proﬁles, but estimated track irregularities.

These subjective assessments totalled over 6000 dia-

grams, each assessed by seven to 10 project partners,

which resulted in more than 50,000 single assessments.

Moreover, a workshop with invited experts dedicated

to model validation was held on 7th November 2012,

hosted by Siemens AG in Krefeld, Germany. A total

of 26 workshop attendees (academics, experts

from industry, railway companies, testing and

research institutes, members of the standardisation

committee as well as DynoTRAIN project partners)

participated in the subjective engineering judgement

of diagrams. The assessments questionnaire contained

110 selected time or distance plots and 10 PSD

diagrams. The workshop was intended to collect

data about the visual assessment of diagrams contain-

ing comparisons between simulation and measure-

ment data.

An assessment of a vehicle simulation model

requires knowledge about the vehicle itself and

about the boundary conditions of the comparison

(i.e. kind and quality of available measurement data

and parameters of the vehicle model). The informa-

tion collected in the workshop was intended to be

used to investigate the feasibility of replacing the sub-

jective engineering assessment with an objective

metric about the degree of similarity between simula-

tion and measurement data. For this reason the work-

shop procedure stressed the importance of focusing

on each single diagram and the workshop attendees

were asked to assess each diagram separately by a

simple Yes/No method under the following

considerations.

1. Assume that a suﬃcient number of diagrams have

already been assessed, each one containing a com-

parison between simulation and measurement of

the particular vehicle.

2. Assume that until the current, last diagram, all

previous diagrams were considered as satisfying

the validation criteria; some of the previous dia-

grams, however, did not show a good agreement,

so that there are still doubts about whether this

model can be conﬁrmed as validated.

Area fulfilling the

validation condition

Vehicle

acceptance

limit

(a) (b) (c)

(f)

(e)

(d)

Figure 8. The main differences in the definitions of the validation limit conditions proposed by project partners.

Polach et al. 743

3. Answer, if the current diagram conﬁrms that the

actual vehicle model can be considered as vali-

dated or if it conﬁrms your doubts and this vehicle

model thus cannot be validated.

It was intended to ask for an engineering judgement

based on a pure visual impression from the assessed

diagram, so as to not be biased by any consideration

about the actual boundary condition of the simulation

or any consideration about the reasons why the sig-

nals show a particular behaviour. Thus, the requested

judgement could be transformed in to a computable

measure calculated using the data presented in the

diagram without considering any other boundary

condition.

The results of the workshop assessments showed

strong variation. Only six from a total of 120 plots

were assessed unequivocally; an equal assessment by

more than 75% of attendees was provided for 54 plots

(45%) of diagrams. Although it was not possible to

conclude about a replacement of the assessment

results by computable values of investigated valid-

ation metrics, this workshop provided interesting

information. The form of the presentation of dia-

grams comparing the simulation and measurement

(scaling of diagrams, exchange of signals back/front)

signiﬁcantly inﬂuenced the assessment result. From

six pairs of two plots presenting identical data using

a diﬀering scale, only one set received the same assess-

ment for both diagrams. The remaining ﬁve diagram

pairs were assessed diﬀerently, see the example in

Figure 9.

Furthermore, the workshop results showed large

diﬀerences in the ‘level’ of strictness of the individual

assessors. This can be seen in Figure 10 that displays

the percentage of positive assessments in each of the

six groups of plots provided by a particular attendee.

The workshop attendees are ordered from more strict

on the left to less strict on the right. No correlation

could be identiﬁed between the attendee’s strictness

and any of the considered categories based on their

aﬃliation or experience. Although the workshop

assessments were solely related to diagrams, without

any background information about the vehicle type,

test conditions and simulation procedure, and thus

cannot be considered as representative validation

assessments, they illustrate the weakness of subjective

judgements. Therefore, it can be concluded, that a

subjective assessment using engineering judgement

does not ensure the feasibility of an objective model

validation.

Evaluation of validation metrics

The investigations dedicated to validation metrics

were introduced with the aim of replacing a subjective

engineering judgement of time or distance plots by a

computable and thus objective measure. The previous

discussion showed deviations between engineering

judgements provided by diﬀerent assessors, which

will surely make a replacement of this judgement

more diﬃcult. Moreover, the judgement can further

deviate depending on the form and scaling of the dia-

grams in question as discussed in the section

‘Simulation output and comparisons with measure-

ments’. These facts can partly explain the initially

surprising eﬀect of a missing correlation between the

subjective assessments by project partners and the

error factors of the investigated validation metrics.

Nevertheless, the cases resulting in an unexpected

disagreement between the validation metric and sub-

jective assessments (high error factors for diagrams

with high percentages of positive assessments and

vice versa) were further analysed to understand and

possibly modify the validation metrics. These analyses

identiﬁed the following three possible reasons for dis-

agreement between the subjective assessment and val-

idation metrics as demonstrated on examples in

Polach and Bo

¨ttcher.

1. The validation metric error factors are based on a

relative deviation, and thus they do not consider

the magnitude of the evaluated quantity. A rela-

tive deviation between simulation and

85.8 85.6 85.4 85.2 85.0 84.8

Distance [km]

85.8 85.6 85.4 85.2 85.0 84.8

Distance [km]

%48:deifsitaS%82:deifsitaS

Lat. bogie acceleration [m/s

]

Lat. bogie acceleration [m/s

]

Figure 9. Example of workshop results displaying the effect of the form of diagrams on the assessment of plots presenting identical

data. In the right diagram, the scale of the vertical axis is enlarged and the forward and background signal exchanged (red

line ¼measurement, blue line ¼simulation; see online version for colours).

744 Proc IMechE Part F: J Rail and Rapid Transit 229(6)

measurement at a very low magnitude of a mea-

sured quantity is usually neglected in engineering

judgement; however, it can provide large error

factor values suggesting large disagreements.

Although Russell’s deﬁnition of the magnitude

error factor aims to correct this eﬀect, it is not

well suited for the investigated application because

of large diﬀerences in the magnitudes of diﬀerent

quantities.

2. Another drawback of the validation metric is a

strong inﬂuence on the phase error factor by the

level of synchronisation between simulation and

measurement signals, see Polach and Bo

¨ttcher.

A perfect synchronisation is not easy to achieve

and is usually not requested, which can lead to

high values of the phase error as well as the com-

prehensive error factors, suggesting disagreement

between simulation and measurement in spite of

positive visual judgements.

3. The third identiﬁed drawback can occur in the

case of superposition of dynamic oscillations

with a rather high constant quasi-static value. In

this case, the resultant integrals will be given by

the quasi-static value of the investigated quantity.

Thus, if there is agreement in the quasi-static

results between simulation and measurement, the

error factors will be low and likewise for the case

when there is a disagreement in dynamic values.

This results in error factors suggesting very good

agreement in spite of a low subjective acceptance.

Evaluation of final validation method, criteria

and limits

The variations of model input data, model adjust-

ments and modelling depth together with variations

of track input data resulted in more than 1000

simulations of validation exercises. The correlations

between the diﬀerent groups of assessments (EN

14363 quantities, subjective assessments, validation

metrics) as well as the relationship between the assess-

ments and the achieved results were investigated as

shown in Figure 11.

Summarising the correlation analyses and other

results of DynoTRAIN WP5, it is believed that the

comparisons of simulation and measurement data

using quantities based on EN 14363 represent the

best-suited methodology for model validation in the

context of vehicle acceptance. Subjective engineering

judgement can vary depending on the strictness of the

reviewer, and the validation metric, which was con-

sidered as suitable for replacement of the subjective

judgement, does not show any valuable improvement

Figure 10. Workshop results: percentage of positive assessments provided by any particular attendee in each of the six groups

of plots.

Proposal for validaon limits

Validaon

metrics

Results

achieved

Subjecve

assessments

Feedback about

validated models

Final agreement

on validaon limits

Quanes

based on

EN 14363

Data used

for ﬁnal

evaluaon

Figure 11. Schematic representation of the process used to

compare different kinds of validation assessments and to

evaluate the final proposal.

Polach et al. 745

compared with the assessment using quantities based

on EN 14363 and was therefore not considered in the

ﬁnal proposal.

The validation investigations conducted in

DynoTRAIN were carried out under the consider-

ation that the uncertainty of the measurements

used for the evaluation represents the state of the

art in vehicle approval processes. The analyses of

deviations between simulation and measurement

data demonstrated that an assessment of a single

particular quantity and single pairs of the simulated

and measured values do not provide relevant infor-

mation about the model quality. It is in fact more

important to check the overall agreement instead of

concentrating on single maximum diﬀerences

between simulation and measurement data. A

single deviation between simulation and measure-

ment data can be related to a particular eﬀect in

the measurement or a particular deviation between

conditions during the measurement and the input

parameters used in the simulations. It is left to

chance, if such a single deviation between simulation

and measurement will be identiﬁed, when selecting

the test sections used for comparisons, and the

impact of measurement uncertainty on the assess-

ment result increases. Therefore, the model

validation should approve the overall agreement of

the deviations between compared pairs of simulation

and measurement data.

A statistical approach has been selected to assess

this overall agreement; it calculates the mean value

and the standard deviation of the diﬀerences between

the simulation value S

and the measurement value

for each of 12 agreed quantities based on EN

14363 (e.g. for all Y

qst

values) for a speciﬁed minimum

of test sections representing the conditions for vehicle

acceptance as is described in detail in the next section.

The minimum to be used for validation was agreed to

be three sections from each test zone according to EN

14363, thus at least 12 sections, and a minimum of

two diﬀerent measuring signals per quantity. Using

two force measuring wheelsets to fulﬁl the later

requirement for quasi-static and maximum value of

the sum of guiding forces, there are 48 pairs simula-

tion–measurement data points for each of the quan-

tities Q,Yand Y/Q, which results in a total of 432

compared pairs of simulated and measured values, see

Figure 12. The validation evaluations conducted in

DynoTRAIN used even more compared pairs. They

included 14 sections for freight vehicles and 17 sec-

tions for other vehicles, resulting in 504 or 612 com-

pared pairs, respectively.

Nomenclature Unit

Y12qst kN 9.546 10.145 -4.036 7.467 -5.577 -0.159 -12.739 -1.591 4.400 0.725 0.373 -0.487

Y21qst kN -3.031 -1.827 -4.051 -2.889 1.292 -4.020 0.720 -0.898 3.402 1.236 0.190 5.452

Y22qst kN 6.432 1.911 -2.528 -5.332 -0.618 -4.890 0.355 -1.828 -1.155 -0.229 -0.466 5.064

Exercise number 21987654321 11

12 quanes

432 compared pairs S

-M

Quanty Y

qst

48 values

Exercise 1

Exercise 2

Exercise 3

-M

–simulated value

–measured value

Quanty Q

qst

48 values

Quanty Y/Q

qst

48 values

Quanty ΣY

qst

24 values

Quanty Y

max

48 values

Quanty Q

max

48 values

Quanty Y/Q

max

48 values

Quanty ΣY

max

24 values

Quanty ÿ

*rms

24 values

Quanty z

*rms

24 values

Quanty ÿ

*max

24 values

Quanty z

*max

24 values

Nomenclature Unit

Q11qst kN 9.183 9.224 -3.498 3.561 -1.130 1.082 1.125 -2.315 -4.908 -3.811 -3.324 -2.342

Q12qst kN 14.280 13.458 7.007 9.060 -2.725 6.879 -3.629 6.865 3.346 4.935 4.340 9.257

Q21qst kN -10.956 -10.985 -4.517 -8.868 6.466 -5.916 7.469 -0.360 -0.637 -0.943 -0.397 -3.061

Q22qst kN 3.010 3.240 12.726 9.711 5.476 11.764 4.469 8.927 -1.938 -1.447 -1.338 0.556

Exercise number 21987654321 11

Nomenclature Unit

Y/Q11qst - 0.043 0.045 -0.009 0.080 0.034 0.057 -0.032 -0.008 -0.005 -0.009 -0.019 -0.037

Y/Q12qst - 0.044 0.056 -0.085 0.063 -0.037 -0.031 -0.093 -0.017 0.040 0.006 0.003 -0.006

Y/Q21qst - -0.021 -0.012 -0.031 -0.022 0.012 -0.031 0.005 -0.009 0.034 0.011 0.001 0.058

Y/Q22qst - 0.061 0.018 -0.040 -0.065 -0.008 -0.065 -0.001 -0.020 -0.011 -0.003 -0.005 0.061

Exercise number 21987654321 11

Nomenclature Unit

SY1qst kN 0.911 0.037 1.420 3.753 -7.760 7.497 -10.049 -3.269 3.856 -2.313 -1.925 -4.571

SY2qst kN 6.079 5.469 -6.560 -7.960 0.690 -9.015 1.260 -3.088 3.710 -1.177 -1.088 0.284

Exercise number 21987654321 11

Nomenclature Unit

Y11max kN 9.643 26.481 3.361 24.852 7.617 24.694 -0.389 -9.429 -10.569 6.203 20.782 -2.603

Y12max kN 8.196 7.627 -3.963 9.389 -4.638 3.013 -3.130 -22.698 -10.136 3.098 9.479 1.709

Y21max kN -4.301 -4.738 -4.561 -10.280 0.932 1.363 0.942 -1.813 -17.951 -3.297 1.066 16.710

Y22max kN 6.538 1.312 -3.275 -6.104 3.128 -5.762 4.245 -10.227 -20.530 -1.315 -2.811 5.221

Exercise number 21987654321 11

Nomenclature Unit

Q11max kN 10.414 31.348 -3.227 -0.385 10.335 7.152 6.700 -3.816 -11.139 2.481 7.210 -0.130

Q12max kN 13.408 24.260 6.264 9.640 6.708 -0.746 -5.311 2.636 -0.166 6.137 10.513 6.398

Q21max kN -9.344 -9.655 -13.043 -30.597 3.875 -8.173 13.468 -4.015 -4.760 -4.989 6.062 -0.931

Q22max kN 4.960 3.750 10.342 10.359 14.556 6.536 5.555 7.981 -3.112 4.722 2.070 -2.754

Exercise number 21987654321 11

Nomenclature Unit

Y/Q11max - 0.016 -0.040 -0.012 0.045 0.053 0.066 -0.016 -0.113 -0.089 0.048 0.121 -0.042

Y/Q12max - 0.001 0.006 -0.093 0.031 -0.064 -0.034 -0.092 -0.229 -0.090 -0.001 -0.007 0.006

Y/Q21max - -0.032 -0.020 -0.040 -0.097 0.028 -0.029 0.007 -0.030 -0.165 -0.064 -0.008 0.074

Y/Q22max - 0.055 0.019 -0.049 -0.100 0.009 -0.077 0.020 -0.115 -0.174 -0.047 0.000 0.040

Exercise number 21987654321 11

Nomenclature Unit

SY1max kN -1.951 -11.162 3.599 -2.451 -17.992 10.749 -8.682 -8.305 -12.854 5.432 11.062 -3.446

SY2max kN 5.996 6.260 -6.260 -16.183 2.881 -7.197 4.339 -3.538 -3.061 5.301 2.179 2.863

Exercise number 21987654321 11

Nomenclature Unit

sy"*Im m/s

-0.042 0.085 -0.068 -0.138 0.020 -0.065 -0.005 -0.027 -0.490 -0.054 0.037 0.061

sy"*IIm m/s

-0.015 -0.059 -0.036 -0.089 0.033 0.003 0.002 -0.013 -0.645 -0.038 0.041 0.046

Exercise number 21987654321 11

Nomenclature Unit

sz"*Im m/s

0.125 0.128 0.052 0.162 0.190 0.053 0.085 0.082 -0.002 0.295 0.445 0.013

sz"*IIm m/s

0.030 0.023 0.047 0.138 0.189 0.125 0.101 0.105 -0.081 0.117 0.140 0.136

Exercise number 21987654321 11

Nomenclature Unit

y"*Im m/s

-0.177 0.725 0.150 -0.528 -0.182 -0.168 -0.043 0.035 -1.217 -0.160 -0.051 0.204

y"*IIm m/s

-0.129 -0.141 0.014 -0.393 0.278 0.003 -0.021 0.064 -1.113 0.028 0.227 0.019

Exercise number 21987654321 11

Nomenclature Unit

z"*Im m/s

0.300 -0.327 0.087 0.270 0.709 -0.306 0.352 0.274 -0.322 0.677 2.413 -0.137

z"*IIm m/s

0.181 -0.136 0.047 0.379 0.907 -0.187 0.296 0.488 -0.175 0.390 0.546 0.484

Exercise number 21987654321 11

Y11qst kN 10.460 10.404 -2.445 11.146 2.217 7.341 -2.855 -0.772 -0.575 -0.904 -1.831 -4.156

Figure 12. Example of a typical set of comparisons between simulation and measurement values according to the proposed

validation method.

746 Proc IMechE Part F: J Rail and Rapid Transit 229(6)

The preliminary validation limits agreed in an ear-

lier step of the project were used to assess the valid-

ation of all investigated model conﬁgurations. The

feedback about the validated models was then used

for the ﬁnal adjustment of the validation limits as can

be seen in the schematic presentation of this process in

Figure 11.

It turned out that the deviations between simula-

tion and measurement values of wheel loads (both

quasi-static as well as dynamic) are very sensitive to

the static wheel load. Therefore, a validation limit that

was dependent on the static wheel load was intro-

duced instead of constant limit value for both quasi-

static as well as dynamic wheel loads (see Table 4 in

the next section). The constants used in the formulas

deﬁning the validation limits for wheel loads were

adjusted so that the limits for vehicles with high

static wheel loads achieved the range of the originally

proposed validation limits, while the validation limits

for vehicles with low static wheel loads were smaller.

The level of vehicle body accelerations of freight

vehicles and vehicles without bogies or without a sec-

ondary suspension is signiﬁcantly larger than that of

vehicles with a typical soft secondary suspension;

therefore, the validation limits for the accelerations

of the vehicle body of those vehicles were doubled

to account for this eﬀect. The accelerations at the

bogie frame were evaluated, but not proposed as a

mandatory quantity for model validation. The

dynamic behaviour of the bogie or running gear of a

particular vehicle model is suﬃciently approved by

checking the quantities in the wheel/rail contact.

Moreover, the investigations carried out showed

that the application of the bogie frame acceleration

for model validation and the justiﬁcation of a suitable

validation limit are rather diﬃcult and not really

necessary as the bogie dynamics is assessed by

wheel/rail quantities anyway.

The investigations dedicated to PSD diagrams

showed a large variety of results and of deviating

assessments by partners as well as during the work-

shop. Due to limitations of time and resources, a

higher priority was put on the evaluation of other

criteria. The limited investigations of PSD diagrams

did not provide suﬃcient input for an introduction of

criteria and quantitative limits in regard to PSD. This

topic needs further investigation.

Proposed validation method

The proposed validation process is based on a math-

ematical comparison between the results of on-track

tests performed using the normal measuring method

based on EN 14363 and the corresponding simulation

results. The simulation and measurement results of

the speciﬁed quantities have to be compared on at

Table 4. Quantities and limits for model validation in regard to simulation of on-track tests (from Ref. 18, www.tandfonline.com).

Quantity Notation Unit Filtering Processing

Validation limit for

standard deviation

Quasi-static guiding force Y

qst

kN Low-pass filter 20 Hz 50%-value (median) 5

Quasi-static vertical wheel force Q

qst

kN Low-pass filter 20 Hz 50%-value (median) 4 (1 þ0.01 Q

)

- static vertical

wheel force (kN)

Quasi-static ratio Y/Q (Y/Q)

qst

– Low-pass filter 20 Hz 50%-value (median) 0.07

Quasi-static sum of guiding forces Y

qst

kN Low-pass filter 20 Hz 50%-value (median) 6

Guiding force, maximum Y

max

kN Low-pass filter 20 Hz 0.15%/99.85%-value

Vertical wheel force, maximum Q

max

kN Low-pass filter 20 Hz 99.85%-value

6(1þ0.01 Q

)

- static vertical

wheel force (kN)

Ratio Y/Q, maximum (Y/Q)

max

– Sliding mean (2 m window,

step 0.5 m)

0.15%/99.85%-value

0.10

Sum of guiding forces, maximum Y

max

kN Sliding mean (2 m window,

step 0.5 m)

0.15%/99.85%-value

Car body lateral acceleration,

RMS-value

€

y

rms m/s

Band-pass filter 0.4 to 10 Hz RMS-value 0.15

Car body vertical acceleration,

RMS-value

€

z

rms m/s

Band-pass filter 0.4 to 10 Hz RMS-value 0.15

Car body lateral acceleration,

maximum

€

y

max m/s

Band-pass filter 0.4 to 10 Hz 0.15%/99.85%-value

0.40

Car body vertical acceleration,

maximum

€

z

max m/s

Band-pass filter 0.4 to 10 Hz 0.15%/99.85%-value

0.40

Absolute values of simulated value S

as well as measured value M

For freight vehicles and vehicles without bogies or without secondary suspension, these limits have to be doubled.

Polach et al. 747

least 12 track sections, called validation exercises. A

track section can be either a test section as in EN

14363 or a part of a test track longer than the min-

imum length speciﬁed for track sections in the particu-

lar test. Moreover, these sections have to fulﬁl the

other test section requirements in EN 14363 such as

constant curve radius, etc. The selected validation

exercises have to contain sections from all four test

zones, with at least three sections from each test zone.

The track geometric irregularities have to represent

the conditions of the on-track tests.

Each quantity has to be evaluated using at least

two signals, e.g. vertical acceleration above the lead-

ing and trailing bogies, thus, at least 24 simulated

values S

are compared to the corresponding mea-

sured values M

of each quantity. Each compared

simulated as well as measured quantity has to be ﬁl-

tered and processed based on the requirements in

Table 4. The percentiles have to be calculated from

the cumulative curve. For the maximum value calcu-

lated as 0.15% or 99.85%-value, the higher magni-

tude of the 0.15%- and 99.85%-values (absolute

value) is used. The 50%-values (medians) are applied

with their sign to show the agreement of both magni-

tude and direction of those quantities. The diﬀerence

between the simulated value S

and the corres-

ponding measured value M

has to be evaluated for

each value and each quantity; this diﬀerence has to be

transformed so that, if the magnitude of the simula-

tion value is higher than the magnitude of the meas-

urement (simulation overestimating the

measurement), the diﬀerence is positive, and vice

versa

Dv¼ðSvMvÞMv

jjfor Mv6¼ 0

Dv¼Svfor Mv¼0ð4Þ

The following values have to be calculated for the

whole set of diﬀerences D

between the simulation

and measurement for each quantity:

.the mean of the diﬀerences between the simulation

value S

and the measurement value M

;

.the standard deviation of the same set of

diﬀerences.

The standard deviation of the set of diﬀerences

between the simulation value S

and the measurement

value M

for each individual quantity has to be

less than or equal to their validation limit shown in

Table 4. For each quantity the mean of the set of

diﬀerences between the simulation value S

and the

measurement value M

should be less than or equal

to a validation limit equal to two-thirds of the related

validation limit for the standard deviation. The valid-

ation limits for accelerations (standard deviation as

well as mean of diﬀerences) for freight vehicles or

vehicles without a secondary suspension are twice

the relevant limit values for other vehicles.

As an example, Figure 13 explains the calculation

of diﬀerences between the simulation value S

and the

measurement value M

for the quasi-static values of

the sum of guiding forces between wheelset and track,

their transformation, as well as calculation of the

mean value and standard deviation, which are used

for comparison with the validation limits speciﬁed in

Table 4.

Discussion

Advantages of the proposed validation method

The proposed ﬁnal set of validation limits was applied

to assess the validity of all the investigated model con-

ﬁgurations. From a total of 78 model conﬁgurations

evaluated, only 20 fulﬁl the proposed model valid-

ation limits:

.eight from 24 models of the locomotive BR 120

investigated by Siemens;

.10 from 13 models of the Bim coach investigated

by Bombardier Transportation;

.two from four models of the Bim coach investi-

gated by IFSTTAR.

The validated models are the models of vehicles

tested in DynoTRAIN and validated using measured

track irregularities as well as measured wheel and

rail proﬁles. The only successfully validated models

were those of the locomotive BR 120 and the Bim

coach.

The contributions of quantities leading to the fail-

ure of 58 out of the total of 78 model conﬁgurations

are displayed in Figure 14. The failure to validate a

model could be caused by one quantity or more quan-

tities at the same time; an exceedance could be the

result of either the standard deviation of diﬀerences,

or the mean value of diﬀerences or both values at the

same time. The most frequent cause was an excee-

dance of the maximum value of the vertical acceler-

ation of the car body. Other common causes were

Y/Q (quasi-static as well as maximum value), Y

qst

and Y

max

. The wheel loads seldom caused the

limits to be exceeded and there was no exceeding of

the validation limit for the mean value of the diﬀer-

ences of Q

qst

. Thus, it seems that on the basis of the

proposed validation method, the expected model

properties can be easily achieved for the vertical

wheel forces whereas it is rather diﬃcult to achieve

the expected properties for the ratio Y/Q or vertical

acceleration of the vehicle body.

An important advantage of the proposed valid-

ation procedure is that this assessment represents an

overall assessment of a large amount of data that is

impossible to carry out by using engineering judge-

ment of plots, as it is not practically possible to dis-

play, check and document the approval of such a

large number of plots. The calculation of

748 Proc IMechE Part F: J Rail and Rapid Transit 229(6)

-15

-10

-5

ΣY

qst

[kN]

Simulation

Measurement

-5

-4

-3

-2

-1

ΣY

qst

[kN]

-5

-4

-3

-2

-1

ΣY

qst

[kN]

Compared values from

simulation and measurement

Differences between simulation

and measurement

Transformed differences

between simulation and

measurement together with

mean and standard deviation

Difference number

2 4 6 8 10 12 14 16 18 20 242219 23211357 911131517

Difference number

2 4 6 8 10 12 14 16 18 20 242219 23211357 911131517

Comparison number

2 4 6 8 10 12 14 16 18 20 242219 23211357 911131517

Mean value

Standard deviation

Figure 13. Example of data evaluation: simulated and measured values, their differences, transformation of differences in regard to

the sign of the measured value, and calculation of the mean value and standard deviation of the quasi-static sum of guiding forces Y.

20%

40%

60%

80%

Validation failure cause [%]

Mean of differences simulation - measurement

Standard deviation of differences

Mean or/and standard deviation

Figure 14. Contributions of individual quantities to the validation failure observed for 58 out of the 78 investigated model

configurations.

Polach et al. 749

characteristic parameters of the mean and standard

deviation of the diﬀerences between the simulation

values S

and the measurement values M

, and their

comparison with the validation limits, however,

allows a fast identiﬁcation of quantities with a large

deviation. The data of a particular quantity can be

easily checked in detail to identify the validation exer-

cise (section) and the signal (sensor position) that pro-

vides a large deviation between the simulation and

measurement values.

The speciﬁed set of 12 quantities covers the quasi-

static as well as dynamic behaviour of a vehicle in

regard to vehicle acceptance, which is the intended

area of application for a validated model. The signal

processing is carried out by analogy with EN 14363

for both the measurement and simulation, thus allow-

ing direct use of the acceptance tests data.

The weakness of the model in question can be iden-

tiﬁed by a normalisation of the validation criteria,

dividing them by the validation limits, as can be

seen in Figure 15. The model is validated, if the abso-

lute magnitudes of all displayed values are not higher

than one. The vehicle models in Figure 15 were pre-

pared by using the available parameter data, before

any model adjustments by comparisons with either

stationary or on-track tests. This ﬁgure shows nor-

malised values of the mean (left) and standard devi-

ation (right) of the diﬀerences between simulation and

measurement data for two vehicle models of the loco-

motive BR 120 performed by Siemens. The initial

model F1 does not fulﬁl the proposed validation

limits. This model used measured track irregularities

as well as measured wheel and rail proﬁles, but it was

not adjusted based on the measurements. The

improved model T2 after adjustments by comparisons

with on-track tests and with stationary tests meets the

validation limits.

The results conﬁrm the proposed validation criteria

and limits as a suitable methodology for the valid-

ation of railway vehicle models. The proposed valid-

ation method allows not only an objective assessment,

but also a clear identiﬁcation of the model weak-

nesses, see also Polach et al.

Effect of model adjustment using stationary tests

on the simulation of on-track tests

Static and low-speed tests can be used to identify miss-

ing or uncertain vehicle model parameters and to sup-

port vehicle model validation. A comparison of

simulations with available stationary tests is required

as part of the model validation process in prEN

14363.

The simulation and measurements of the sta-

tionary tests are compared and the uncertain model

parameters adjusted if necessary.

However, what is

the eﬀect of a model adjustment based on a compari-

son with stationary tests on the agreement between

simulation and measurement of the on-track test

(ride test)? This is typically neither presented nor inves-

tigated; it is believed that an improved agreement with

stationary tests will implicitly improve the exactness of

the on-track test simulation. In order to investigate

this eﬀect, the validation exercises with on-track tests

were repeated with several versions of the same model,

either before the comparison with stationary tests or

after comparison and adjustment in order to ﬁt the

stationary tests results, respectively.

The stationary tests used during the validation

evaluations of the simulation models diﬀered depend-

ing on the availability of the test results. An overview

of the stationary tests used for comparisons and

model adjustments of vehicles tested in

DynoTRAIN is shown in Table 5. Not all compari-

sons resulted in a proposal of model adjustment.

Particularly the tests performed on a ﬂat curve with

a radius of 150 m representing a part of the test of

safety against derailment on twisted track according

to Method 2 in EN 14363

did not provide any sug-

gestion regarding the parameter adjustment of the

investigated vehicles.

Figure 16 shows comparisons of the validation

results obtained using the proposed validation

0.0

0.5

1.0

1.5

2.0

Value / Validation limit [-]

Standard deviation of differences simulation - measurement

Initial model (configuration F1)

Improved model (configuration T2)

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

Value / Validation limit [-]

Mean of differences simulation - measurement

Figure 15. Example of validation results using the proposed method. Normalised values of the mean (left) and standard deviation

(right) of the differences between simulation and measurement data for two vehicle models of the locomotive BR 120 performed by

Siemens: initial model F1 and improved model T2 after comparisons with on-track as well as stationary tests.

750 Proc IMechE Part F: J Rail and Rapid Transit 229(6)

method for six models of vehicles tested in

DynoTRAIN WP1. The ﬁgure presents comparisons

of the initial model conﬁgurations F1 using measured

track irregularities and measured wheel as well as rail

proﬁles, however, without any model adjustments

based on comparisons with stationary or on-track

tests, and model conﬁgurations T1 after adjustments

based on comparisons with stationary tests stated in

Table 5 (in the case of laden freight wagon Sgns by

TU Berlin the compared conﬁgurations are F2 and T2

with modiﬁed suspension modelling). The parameters

adjusted to improve the models can be illustrated by

considering the example of the locomotive model per-

formed by Siemens, where the modiﬁcations con-

sidered the vertical and lateral stiﬀness of the

secondary suspension, the characteristics of the sec-

ondary lateral bump stop and the height of the

centre of gravity of the vehicle body.

The model adjustments by comparisons with sta-

tionary tests did not lead to expected improvements of

the results regarding the simulations of the on-track

tests. Only the investigations by Siemens and

IFSTTAR regarding the locomotive BR 120 clearly

provided better results for the models after the com-

parison and adjustment due to the stationary tests. In

other cases, the model adjustments introduced using

the stationary tests did not signiﬁcantly aﬀect the

agreement between the simulation and measurement

concerning the on-track tests or provided even worse

results. For example, in the model of the Bim coach

by Bombardier Transportation, an implementation of

friction elements intended to model a rather small

hysteresis in the secondary lateral suspension resulted

in the failure of the model validation due to signiﬁ-

cantly higher lateral car body accelerations compared

with the values measured during the on-track test.

These investigations did not conﬁrm the traditional

opinion regarding the positive eﬀect of model adjust-

ments by comparisons with stationary tests on the

simulation of on-track tests. A possible explanation

is that focussing on the static and low-speed behav-

iour can result in model adjustments that are less

accurate in regard to dynamic behaviour. The station-

ary tests can support the identiﬁcation of model par-

ameters that are unknown or uncertain. A good

agreement between simulation and measurement of

stationary tests, however, does not guarantee a good

agreement between simulation and measurement of

on-track tests. An adequate number of comparisons

between simulations and on-track measurements is

the only suitable and reliable model validation

method in regard to the simulation of on-track tests.

Summary and conclusions

The presented part of the investigations in the frame-

work of the DynoTRAIN project was dedicated to

the evaluation of a validation method suited for simu-

lations in the context of vehicle acceptance. It repre-

sents a unique activity of complex testing,

simulations, comparisons with measurements and

evaluations. The on-track measurements included sev-

eral vehicles, tested using 10 force measuring wheel-

sets in four European countries and a test train

Table 5. Comparisons and adjustments of vehicle models using stationary tests. The tests used for comparisons are marked with a

cross (X); ‘no adjustment’ is stated if the comparison with the respective stationary test did not provide any suggestion for model

adjustment.

Vehicle model

Wheel unloading

test (twist test)

based on

EN 14363

Test in flat

curve R¼

150 m based

on EN 14363

Bogie rotational

resistance test

based on

EN 14363

Bogie lateral

resistance test

to measure the

characteristic

of the lateral

suspension

Sway test -

measurement

of roll

coefficient

Locomotive DB BR 120,

Siemens

(no adjustment)

XXX

Locomotive DB BR 120, IFSTTAR XXX

DB passenger coach Bim,

Bombardier Transportation

(no adjustment)

XX

DB passenger coach Bim,

IFSTTAR

X

(no adjustment)

X

Freight wagon Sgns, empty,

Technical University Berlin

X

(no adjustment)

X

Freight wagon Sgns, empty,

IFSTTAR

XX

Freight wagon Sgns, laden,

Technical University Berlin

X

(no adjustment)

X

Polach et al. 751

0.0

0.5

1.0

1.5

2.0

Value / Validation limit [-]

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

Value / Validation limit [-]

Mean of differences Standard deviation of differences

Differences simulation - measurement

Locomotive

BR 120,

Siemens

Locomotive

BR 120,

IFSTTAR

Passenger

coach Bim,

Bombardier

Transportation

Passenger

coach Bim,

IFSTTAR

Freight wagon

Sgns empty,

IFSTTAR

Freight wagon

Sgns laden,

TU Berlin

0.0

0.5

1.0

1.5

2.0

Value / Validation limit [-]

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

Value / Validation limit [-]

0.0

0.5

1.0

1.5

2.0

Value / Validation limit [-]

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

Value / Validation limit [-]

0.0

0.5

1.0

1.5

2.0

Value / Validation limit [-]

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

Value / Validation limit [-]

0.0

0.5

1.0

1.5

2.0

Value / Validation limit [-]

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

Value / Validation limit [-]

0.0

0.5

1.0

1.5

2.0

Value / Validation limit [-]

Before comparison with stationary tests

After adjustments based on stationary tests

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

Value / Validation limit [-]

Figure 16. Effect of comparisons with stationary tests on the validation results. Normalised values of mean (left) and standard

deviation (right) of the differences between simulation and measurement for the initial vehicle models and models after adjustments

based on comparisons with stationary tests.

752 Proc IMechE Part F: J Rail and Rapid Transit 229(6)

equipped for the simultaneous recording of track irre-

gularities and rail proﬁles. The simulations were per-

formed using several vehicle models, built with the use

of diﬀerent simulation tools by diﬀerent partners. The

comparisons between simulation and measurement

were conducted in a large number of simulations

using a set of the same test sections. The results

were assessed by three diﬀerent validation

approaches: by comparisons based on values accord-

ing to EN 14363; by subjective engineering judgement;

and by using computable measures, so-called valid-

ation metrics.

The proposed model validation criteria and limits

are based on 12 quantities evaluated by analogy with

EN 14363, covering quasi-static and dynamic wheel/

rail force measurements and vertical as well as lateral

vehicle body accelerations. For each quantity, a set of

at least 24 comparisons between simulation and meas-

urement are evaluated using values based on EN

14363 from at least 12 sections that represent all

four test zones as required in EN 14363 from straight

track to curves with a very small radius. The agree-

ment between simulation and measurement is assessed

by comparing the mean value and standard deviation

for a set of diﬀerences between simulated and mea-

sured values of each quantity with the proposed val-

idation limit.

The investigations neither conﬁrm nor deny the

traditional opinion about the positive eﬀect of the

model adjustments by comparisons with stationary

tests on the simulation of on-track tests. This topic

would need further investigation. According to the

presented investigations, comparisons between simu-

lations and on-track measurements represent the only

suitable and reliable model validation method in

regard to the simulation of on-track tests. The pro-

posed method, criteria and limits represent a suitable

methodology for the validation of railway vehicle

models. It represents an overall assessment of a

large number of data, which is impossible to carry

out by using engineering judgement, as it is not prac-

tically possible to display, check and document the

approval of such a large number of plots. This valid-

ation process not only allows an objective assessment,

but also supports an identiﬁcation of the weaknesses

of the model. The presented methodology is proposed

for implementation in a revised standard EN 14363.

Feedback from future applications of this method in

allied projects will help to further improve and

develop the model validation, which is the crucial con-

dition for successful use of simulation to reduce the

amount and cost of physical testing in the railway

vehicle acceptance process.

Funding

This article describes work undertaken in the context of

the DynoTRAIN project, Railway Vehicle Dynamics and

Track Interactions: Total Regulatory Acceptance for the

Interoperable Network (www.triotrain.eu). DynoTRAIN is

a collaborative project – medium-scale focused research

project supported by the European Seventh Framework

Programme, contract number: 234079 and is led by

UNIFE.

References

1. ASME. Guide for verification and validation in computa-

tional solid mechanics. New York, NY: ASME Press,

2006.

2. Cooperrider NK and Law EH. A survey of rail vehicle

testing for validation of theoretical dynamic analyses.

J Dyn Syst, Meas Control 1978; 100: 238–251.

3. Gostling RJ and Cooperrider N. Validation of railway

vehicle lateral dynamics models. Veh Syst Dyn 1983; 12:

179–202.

4. Cheli F, Corradi R, Diana G and Facchinetti A.

Validation of a numerical model for the simulation of

tramcar vehicle dynamics by means of comparison with

experimental data. J Comput Nonlinear Dyn 2007; 2:

299–307.

5. Alfi S, Bruni S and Mazzola L. Numerical methodology

for the evaluation of high speed vehicle stability in

curved track. In: Z Istavan (ed.) The eighth international

conference on railway bogies and running gears,

Budapest, Hungary, 13–16 September 2010, pp.247–256.

Budapest: BME.

6. Kuka N, Ariaudo C and Verardi R. Modelling and

simulation of tilting trains. In: The 22nd international

symposium on dynamics of vehicles on roads and tracks,

Manchester, UK 14–19 August 2011, paper no. 135.

Manchester: Manchester Metropolitan University.

7. Evans J and Berg M. Challenges in simulation of rail

vehicle dynamics. Veh Syst Dyn 2009; 47: 1023–1048.

8. Bruni S, Vinolas J, Berg M, et al. Modelling of suspen-

sion components in rail vehicle dynamics context. Veh

Syst Dyn 2011; 49: 1021–1072.

9. GM/RC2641, Issue 2: 2009. Recommendations for

vehicle static testing.

10. GM/RT2141, Issue 3: 2009. Resistance of railway vehi-

cles to derailment and roll-over.

11. EN 15273-2: 2013. Railway applications - gauges - part

2: rolling stock gauge.

12. UIC Code 518: 2009. Testing and approval of railway

vehicles from the point of view of their dynamic behav-

iour – safety – track fatigue – running behaviour.

13. Jo

¨nsson LO, Nilstam N and Persson I. Using simula-

tions for approval of railway vehicles: a comparison

between measured and simulated track forces. Veh

Syst Dyn 2008; 46: 869–881.

14. prEN 14363: 2013. Railway applications—testing and

simulation for the acceptance of running characteristics

of railway vehicles—running behaviour and stationary

tests.

15. Mazzola L. The influence of modelling of the suspen-

sion components on the virtual homologation of a rail-

way vehicle. In: Pombo J (ed.) The first international

conference on railway technology: research, development

and maintenance, La Palma, Spain, April 2012, paper

no. 75. Stirling, Scotland: Civil-Comp Press.

16. Mazzola L and Berg M. Secondary suspension of rail-

way vehicles - air spring modelling: performance and

critical issues. Proc IMechE, Part F: J Rail Rapid

Transit 2014; 228: 225–241.

Polach et al. 753

17. Zacher M and Kratochwille R. Stationary and on track

tests with different vehicles. Proc IMechE, Part F: J

Rail Rapid Transit 2015; 229(6): 668–690.

18. Polach O and Bo

¨ttcher A. A new approach to define

criteria for rail vehicle model validation. Veh Syst Dyn

2014; 52(Suppl. 1): 125–141.

19. ORE B 176: 1989. Bogies with steered or steering wheel-

sets. Part 1: specifications and preliminary studies. Part

2, specification for a bogie with improved curving

characteristics.

20. EN 14363: 2005. Railway applications—testing for the

acceptance of running characteristics of railway vehi-

cles—testing of running behaviour and stationary tests.

21. Fries R, Walker R and Wilson N. Validation of

dynamic rail vehicle models. In: The 23 rd international

symposium on dynamics of vehicles on roads and tracks,

Qingdao, China, 19–23 August 2013, paper no. 10.1.

22. Schwer LE. Validation metrics for response histories:

perspectives and case studies. Engng Comput 2007; 23:

295–309.

23. Russell DM. Error measures for comparing transient

data: part I: development of a comprehensive error

measure, part II: error measures case study. In: The

68th shock and vibration symposium, Hunt Valley,

MD, 3–6 November, 1997, pp.175–198.

24. Sprague MA and Geers TL. A spectral-element method

for modelling cavitation in transient fluid-structure

interaction. Int J Numer Methods Engng 2004; 60:

2467–2499.

25. Mongiardini M, Ray MH and Anghileri M.

Development of software for the comparison of

curves during the verification and validation of numer-

ical models. In: The seventh European LS-DYNA con-

ference, Salzburg, Austria, 14–15 May, 2009. Stuttgart:

DYNAmore GmbH.

26. Polach O, Bo

¨ttcher A, Vannucci D, et al. Validation of

multi-body models for simulations in authorisation of

rail vehicles. In: Istavan Z (ed.) The ninth international

conference on railway bogies and running gears,

Budapest, Hungary, 9–12 September, 2013,

pp.187–196. Budapest: BME.

27. Evans J. Validation of vehicle dynamic modelling –

some practical experience. In: The 23rd international

symposium on dynamics of vehicles on roads and tracks,

Qingdao, China, 19–23 August 2013, paper no. 2.3.

754 Proc IMechE Part F: J Rail and Rapid Transit 229(6)