1

Towards an Ontology for Scenario Definition for

the Assessment of Automated Vehicles: An
Object-Oriented Framework
Erwin de Gelder1,2,∗ , Jan-Pieter Paardekooper1,3 , Arash Khabbaz Saberi1 , Hala Elrofai1 , Olaf Op den Camp1 ,
Steven Kraines4 , Jeroen Ploeg5,6 , Bart De Schutter2

Abstract—The development of new assessment methods for • scenarios provide the basis and justification for the tests
arXiv:2001.11507v4 [cs.AI] 13 Dec 2021

the performance of automated vehicles is essential to enable used for the scenario-based assessment [4, 12, 13, 14, 6,
the deployment of automated driving technologies, due to the 15],
complex operational domain of automated vehicles. One con-
tributing method is scenario-based assessment in which test • it helps to arrive at an unambiguous description of scenar-
cases are derived from real-world road traffic scenarios obtained ios that is crucial for providing standardized, repeatable,
from driving data. Given the complexity of the reality that is and reproducible tests [12],
being modeled in these scenarios, it is a challenge to define a • standardized descriptions of scenarios can be more easily
structure for capturing these scenarios. An intensional definition compared and classified automatically [16],
that provides a set of characteristics that are deemed to be both
necessary and sufficient to qualify as a scenario assures that the • properly specified scenarios are the basis for evaluating
scenarios constructed are both complete and intercomparable. the coverage of the assessment [6], and
In this article, we develop a comprehensive and operable defini- • properly specified scenarios enable us to translate the
tion of the notion of scenario while considering existing definitions result of a test into an assessment of the AV performance
in the literature. This is achieved by proposing an object-oriented with regards to a particular Operational Design Domain
framework in which scenarios and their building blocks are
defined as classes of objects having attributes, methods, and
(ODD) [17, 18].
relationships with other objects. The object-oriented approach
promotes clarity, modularity, reusability, and encapsulation of Although the notion of scenario is frequently used in the
the objects. We provide definitions and justifications of each of context of automated driving [5, 19, 20, 21, 15, 6, 7, 22, 23],
the terms. Furthermore, the framework is used to translate the only rarely is an explicit definition actually given. Further-
terms in a coding language that is publicly available. more, even those definitions are unclear because of ambiguities
and the use of other undefined terms. From the implementa-
tion perspective, describing scenarios unambiguously becomes
I. I NTRODUCTION more important given the many simulators that are recently
being introduced [24, 25, 26, 27, 28]. To this end, there are
An essential aspect in the development of Automated Ve-
several file formats and methods for defining scenarios for the
hicles (AVs) is the assessment of quality and performance
assessment of AVs, such as OpenSCENARIO [29] and Com-
aspects of the AVs, such as safety, comfort, and efficiency
monRoad [30]. Because the focus of these implementations
[1, 2, 3, 4, 5, 6, 7, 8]. For legal and public acceptance of
is on scenarios that can be simulated, these implementations
AVs, a clear definition of system performance is important, as
describe scenarios at a quantitative level and, consequently,
well as quantitative measures for system quality. According
they do not provide concepts for a qualitative description
to [2], traditional methods for evaluating driver assistance
of a scenario. Furthermore, these implementations and other
systems, such as [9, 10], cannot sufficiently assess quality and
object-oriented approaches used in the field of the assessment
performance aspects of an AV, because they would require too
of AVs [31, 32, 33, 34] mostly lack the definitions and
many resources. A scenario-based approach could be a viable
justifications of each of the terms.
way to perform the AV assessment [6, 11, 8]. For a scenario-
based assessment, proper specification of scenarios is crucial In this work, as a starting point for developing a full
because ontology of scenarios, we propose a novel Object-Oriented
Framework (OOF) that addresses the aforementioned short-
1 TNO, Dept. of Integrated Vehicle Safety, Helmond, The Netherlands comings. To avoid ambiguities in the definitions, we provide
2 Delft University of Technology, Delft Center for Systems and Control, intensional definitions for concepts corresponding to scenarios
Delft, The Netherlands
3 Radboud University, Donders Institute for Brain, Cognition and Be- and all of their essential building blocks (such as activities,
haviour, Nijmegen, The Netherlands actors, and events). These intentional definitions give the
4 Symphony Co., Tokyo, Japan
meaning of the concepts by specifying necessary and sufficient
5 2getthere, Utrecht, The Netherlands
6 Eindhoven University of Technology, Dept. of Mechanical Engineering
conditions for when the concepts should be used. We base
Dynamics and Control group, Eindhoven, The Netherlands the definitions of each of the components on definitions that
∗ Corresponding author: [email protected] are commonly used in the field of the safety assessment of
 2

AVs [14, 13, 35, 36, 37, 29]. While being broadly consistent framework with a real-world scenario. The article is concluded
with existing definitions [14, 13, 38], this framework aims in Section VI.
to be sufficiently explicit to enable the formalization of a
scenario description. More specifically, because we give the II. BACKGROUND
characteristics of the concepts corresponding to scenarios and
specify how those concepts interrelate, we can define the In Section II-A, we explain why we want to present an OOF
scenario components as objects of classes having attributes, for describing scenarios and scenario categories. Section II-B
methods, and relationships with objects that are members provides information on the context for which we want to
of other classes. In addition to the definition of a scenario, define scenarios.
we introduce the concept of a scenario category that is
used to qualitatively describe scenarios, i.e., an abstraction of A. Why an object-oriented framework?
a scenario. Scenario categories enable the categorization of
scenarios in terms of the categories of their typical compo- According to Johnson and Foote [42], an OOF is a “set
nents. The presented OOF provides explicit guidelines for the of classes that embodies an abstract design for solutions to
construction of scenario descriptions that are able to effectively a family of related problems.” The object orientation is used
assess the AV performance. for “a representation, modeling, and abstraction formalism”
The proposed approach brings several benefits. First, we [43], which is why it is considered “not only useful but
provide concepts for a qualitative description of a scenario, also fundamental” [43]. In addition, Patridge [44] notes that
which is useful because it enables to classify scenarios and object-oriented modeling can provide a bridge from traditional
to interpret scenarios. Second, the OOF allows for reusing entity-relation-based data modeling to data modeling that is
and maintaining (the building blocks of) a scenario as well as fully grounded in a formalized ontology. An OOF offers the
performing operations on and interacting with (the building following benefits:
blocks of) a scenario. Third, our framework is supported • Clarity: It provides “a common vocabulary for designers
with the definitions and justifications of each of the concepts. to communicate, document, and explore design alterna-
Fourth, the framework enables the translation of the concepts tives” [45].
and their relationships into object-oriented code. This, in turn, • Modularity: By decomposing a scenario into components,
is used to describe scenarios in a coding language that can the complexity of a scenario itself is reduced. Thus,
be understood by various software agents, such as simulation “modularity makes it easier to understand the effect of
tools, and that can be ported to already available formats like changes” [42].
OpenSCENARIO [29]. • Reusability: An OOF promotes reusability [46, 47, 42].
To illustrate how to use the presented OOF, we have For example, if two classes share certain procedures
implemented the framework in a coding language that and/or properties, these procedures and/or properties
is publicly available at https://github.com/ErwindeGelder/ could be provided by a so-called superclass from which
ScenarioDomainModel1 . This link contains real-life applica- these two classes inherit the procedures and properties,
tions of the presented OOF, such as describing scenarios such that these procedures and properties need to be
extracted from data [39]. The framework is also used as defined only once.
a schema for a database system for storing scenarios and • Encapsulation: Encapsulation assures “that compatible
scenario categories. Such a database can be used to perform changes can be made safely, which facilitates program
scenario-based assessment of AVs2 [40]. To further illustrate evolution and maintenance” [46].
the use of the OOF, this article provides an example with a • Possibility to translate to object-oriented programming
real-world case in which a vehicle approaches a pedestrian languages: As the framework consists of a set of classes,
crossing. The proposed OOF provides a first step towards it can be directly used in an object-oriented coding
an ontology [41] for scenarios for the assessment of AVs. language. The framework then specifies the relationships
In a subsequent study, the formalized concepts presented in between the different classes and provides information on
this article will be used to design an ontology with logical the properties of a class and the possible values.
constraints that enable a computer to perform reasoning on
scenarios.
The outline of the article is as follows. In Section II, B. Context of a scenario
we explain why an OOF is useful and what the context Because the notion of scenario is used in many different
is. We define the notions of scenario, event, activity, and contexts outside of the domain of road traffic, a wide diversity
scenario category in Section III. The OOF that formalizes in definitions of this notion exists (for an overview, see [48,
these definitions is presented in Section IV. In Section V, an 49]). Therefore, it is reasonable to assume that “there is no
application example is provided to illustrate the use of the [generally] ‘correct’ scenario definition” [48]. As a result, to
define the notion of scenario, it is important to consider the
1 As a coding language, Python is used. The code implementation also
context in which it will be used.
contains more methods than presented in this article.
2 An illustration of such an assessment is publicly available at https://github. In this article, the context of a scenario is the assessment
com/ErwindeGelder/ScenarioDomainModel. of AVs, where AVs refer to vehicles equipped with a driving
 3

automation system3 . It is assumed that the assessment method- Table I: Terms and definitions that are used in Section III. For
ology uses scenario-based test cases. The ultimate goal is to more details, see Appendix A.
build a database with all relevant scenarios that an AV has to Term Definition
cope with when driving in the real world [6]. Hence, a scenario
Ego vehicle Vehicle from which the world is perceived
should be a description of a potential use case of an AV. and/or vehicle that must perform a certain task
during a test
III. D EFINITIONS Physical element Object that exists in the three-dimensional space
Actor Physical element that experiences change
One of the main reasons to introduce an OOF is to enable Note: An actor is a physical element, but a
sharing of knowledge between researchers, developers, and physical element is not necessarily an actor.
Static environment Part of the environment that does not change
users. Therefore, it is important that the terms we use are Dynamic environment Part of the environment that does change and
clearly defined. When presenting our OOF in Section IV, we that comprises all actors
will formalize the terms such that they can be used by software Act Combination of an actor and an activity
State variables Description of the present configuration of a
agents. In this section, we define the terms scenario, event, system that can be used to determine the future
activity, and scenario category, thereby providing insight response, given the excitation inputs and the
into the terms used in the next section. We aim to provide equations describing the dynamics
State vector Vector containing all n state variables
intensional definitions that are in accordance with the common Model Differential and algebraic equations that de-
use of these terms in the literature and to provide clarity on scribe the dynamics
what are the necessary and sufficient conditions for when the Mode Period in which a system does not exhibit a
sudden change in an input, a model parameter,
term should be used. or the model
We first define the concept of a scenario in Section III-A.
Next, we define two important components of a scenario:
events and activities, in Sections III-B and III-C, respectively. Ulbrich et al. [13] define a scenario as “the temporal
Lastly, we present the definition of a scenario category in development between several scenes in a sequence of scenes.
Section III-D. Each of the Sections III-A to III-D starts with Every scenario starts with an initial scene. Actions & events
background information. Next, we draw conclusions that lead as well as goals & values may be specified to characterize
to our proposed definition of the corresponding term. After this temporal development in a scenario. Other than a scene,
proposing a definition, each section finishes with remarks and a scenario spans a certain amount of time.” The authors of
implications of the proposed definition. For the definitions [13] state that actions and events link the different scenes. A
provided in Sections III-A to III-D, use is made of the terms further description of actions and events is not given in [13].
listed in Table I. The definitions in Table I are mostly based Another definition of a scenario in the context of automated
on literature; see Appendix A for more details. driving is given by Elrofai et al. [38]. They define a scenario as
“the combination of actions and maneuvers of the host vehicle
A. Scenario in the passive [i.e., static] environment, and the ongoing
Go and Carroll [51] describe a scenario within the field of activities and maneuvers of the immediate surrounding active
system design. They define a scenario as “a description that [i.e., dynamic] environment for a certain period of time.”
contains (1) actors, (2) background information on the actors Saigol et al. [36] define a scenario as “a description of a
and assumptions about their environment, (3) actors’ goals short interaction between an AV and other road users and/or
or objectives, and (4) sequences of actions and events. Some road infrastructure”.
applications may omit one of the elements or they may simply In a concept paper on OpenSCENARIO 2.0 [52], a scenario
or implicitly express it. Although, in general, the elements of is defined as “a ‘description of the temporal development’ of
scenarios are the same in any field, the use of scenarios is road users (actor entities) defined by their actions, where tem-
quite different.” poral activation (defining when) ‘is regulated by’ conditional
Geyer et al. [14] describe a scenario within the context of ‘triggers’. A scenario comprises both scenery and dynamic
automated driving. They use the metaphor of a movie or a elements.”
storybook for describing a scenario and state that “a scenario As a basis for constructing a comprehensive definition for
includes at least one situation within a scene including the the concept of scenario, we list the major characteristics
scenery and dynamic elements. However, [a] scenario further contained in the above definitions as follows:
includes the ongoing activity of one or both actors.” Geyer 1) A scenario corresponds to a time interval. The afore-
et al. [14] define a scene “by a scenery, dynamic elements, and mentioned definitions [51, 14, 13, 38] state that a
optional driving instructions.” In [14], the meaning of activity scenario corresponds to a time interval. van Notten
is not detailed. et al. [48] call such a scenario a chain scenario (“like
3 According to [50], a driving automation system is “the hardware and movies”), as opposed to a snapshot scenario, i.e., a
software that are collectively capable of performing part or all of the dynamic scenario that describes the state at a given time instant
driving task on a sustained basis. This term is used generically to describe (“like photos”).
any system capable of level 1-5 driving automation.” Here, level 1 driving
automation refers to “driver assistance” and level 5 refers to “full driving 2) A scenario consists of two or more events [48, 51, 14,
automation”. For more details, see [50]. 13, 53]. It can be helpful to develop scenarios using
 4

events [49]. Thus, a scenario could be defined as a not necessary because the ego vehicle is just the vehicle
particular sequence of events or, as Kahn [53, p. 143] whose perspective is used to define what is relevant in
writes, “a scenario results from an attempt to describe the scenario.
in more or less detail some hypothetical sequence of 7) A scenario describes the goals or activities of the actors.
events”. Furthermore, Geyer et al. [14] and Ulbrich et al. Either the activities, the goals, or a combination of
[13] use the notion of event for describing a scenario, activities and goals are required to determine how each
although they do not provide a definition of the term actor in a scenario responds to specific events. Note that
event. Because a scenario contains at least a start event this also holds for the ego vehicle since the ego vehicle
and an end event, the minimum number of events is is an actor. When describing a scenario using real-world
two. In Section III-B, we will elaborate on the notion of data, goals do not need to be given; e.g., Elrofai et al.
event. [38] mention the activities of the actors rather than the
3) Real-world traffic scenarios are quantitative scenarios. goals. When describing a scenario that an AV has to cope
Regarding the nature of the data, a scenario can be with, however, the ego vehicle’s goals (i.e., its driving
either qualitative or quantitative [48]. For a real-world mission [14]) could be specified rather than its activities
traffic scenario to be suitable for simulation purposes, it [13]. Note that if the activities of an actor are described
must be described quantitatively. A scenario, however, rather than its goals, an observer might not be able to
can also be described qualitatively, such that it is read- determine whether the actor has successfully responded
able and understandable for human experts. Providing to the scenario.
a qualitative description of a quantitative scenario has Hence, we define a scenario as follows:
become known as a story-and-simulation approach [54].
Note that a qualitative description of a scenario does Definition 1 (Scenario). A scenario is a quantitative descrip-
not uniquely define a quantitative scenario. A qualitative tion of the relevant characteristics and activities and/or goals
description can be regarded as an abstraction of the of the ego vehicle(s), the static environment, the dynamic
quantitative scenario, see also Section III-D. environment, and all events that are relevant to the ego
4) The time interval of a scenario contains all relevant vehicle(s) within the time interval between the first and the
events. According to Geyer et al. [14], “the end of last relevant event.
a scenario is defined by the first irrelevant situation When applying Definition 1 in an OOF, it is possible to
with respect to the scenario”. In a similar manner, we give the “description” of a component of a scenario simply by
require that the time interval of a scenario should contain providing a reference to that component. A reference could
all relevant events. Note that ‘relevant’ is subjective be, e.g., the full name of a file, a pointer pointing to a specific
and, therefore, an event is considered to be relevant part of the computer memory, or an identifier that addresses a
with respect to the perspective of one or more of the specific entry in a database. The advantage of references is that
participating actors, often called the “ego vehicle”. these parts of the scenario can be exchanged across different
5) A scenario includes the description of the environment. scenarios, as these scenarios can use the same references.
A scenario should include the description of the static As an example, an OpenSCENARIO file allows to provide
and dynamic environment. Although the description of a reference to an OpenDRIVE file, which describes a road
the static environment is not a general prerequisite network [55]. As we will see in Section IV, in our proposed
of a scenario, this is often included when speaking framework, a scenario may contain references to physical
about traffic scenarios [14, 13, 38, 19, 30]. The static elements, activities, actors, and events.
environment consists of all relevant4 physical elements
that do not undergo relevant changes with respect to the
ego vehicle (s) within the time interval between the start B. Event
and the end of the scenario. The dynamic environment
consists of all relevant actors that undergo changes that As mentioned in Definition 1, a scenario consists of events.
are relevant to the ego vehicle(s). For example, the road The term event is used in many different fields, e.g.:
may be part of the static environment, but if the change • In computing [56], an event is an action or occurrence
in the road temperature is relevant to the ego vehicle(s), recognized by software. A common source of events are
the road is part of the dynamic environment. inputs by the software users. An event may trigger a state
6) A scenario includes at least one ego vehicle [14, 38]. transition.
Because of the two previously mentioned characteris- • In probability theory, an event is an outcome or a set of
tics, a scenario is required to include at least one ego outcomes of an experiment [57]. For example, a thrown
vehicle. Note that an ego vehicle is often regarded as coin landing on its tail is an event.
the device under test. In this article, however, this is • In the field of hybrid systems theory, “the continuous and
discrete dynamics interact at ‘event’ or ‘trigger’ times
4 The term ‘relevant’ is subjective and depends on the use of the scenario.
when the continuous state [vector] hits certain prescribed
The composer of a scenario typically judges whether something might be sets in the continuous state space” [58]. Moreover, “a
relevant for the scenario. hybrid system can be in one of several modes, [...], and
 5

the system switches from one mode to another due to the Remark 1. Geyer et al. [14], Ulbrich et al. [13] use the term
occurrence of events” [59]. scene to define a scenario. Like an event, we consider a scene
• In the ISO 15926-2 standard, an ontology for long-term to correspond to a temporal snapshot of the entire scenario. A
data integration, access, and exchange is specified in scene can be obtained by taking a temporal cross-section of
which an event is defined as “a possible_individual5 with the entire scenario as described in Definition 1. ♦
zero extent in time, which means that it occurs at an
instant in time” [60]. C. Activity
• In event-based control, a control action is computed when
To describe the dynamic environment of a scenario, activ-
an event is triggered, as opposed to the more traditional
ities are used. A scenario may also describe the activities of
approach where a control action is periodically computed
the ego vehicle.
[61]. In event-based control, the event is triggered at the
Both the terms activity [14, 11, 64, 35, 37] and action [14,
moment at which the system (is about to) reach a certain
13, 65] are used in the context of automated driving. Although,
threshold.
strictly speaking, the terms action and activity have a slightly
Before providing the definition of an event, the following different meaning, they are often used for the same purpose:
is concluded about an event, based on the aforementioned • According to Ulbrich et al. [13], actions may be specified
literature: for characterizing the temporal development in a scenario.
1) An event corresponds to a time instant. For the definition • Elrofai et al. [11] consider an activity as a building block
of event, we consider a hybrid-systems setting with a of the dynamic part of the scenario: “An activity is a time
linear-time model [62]. Therefore, an event happens at evolution of state variables such as speed and heading
some time instant. to describe for instance a lane change, or a braking-to-
2) An event marks a mode transition or the moment a standstill.”
system reaches a threshold. A mode transition may be • In a glossary for scenario catalog development [35], an
induced by either an abrupt change of an input signal, activity is defined as “the state [vector] of an object over
a change of a parameter, a change in the model, or an an interval of time. An activity starts with an event and
external cause. It is also possible that the event marks ends with another event.”
the moment that a system reaches a threshold. • In the ISO 15926-2 standard, an activity is defined as “a

Hence, we define an event as follows: possible_individual that brings about change by causing
the event that marks the beginning, or the event that marks
Definition 2 (Event). An event corresponds to a moment at the ending of a possible_individual” [60].
which a mode transition occurs or a system reaches a specified Before providing the definition of an activity, the following
threshold, where the former can be induced by both internal is concluded about an activity based on the aforementioned
and external causes. literature:
Definition 2 indicates that the moment of an event can 1) An activity corresponds to an inter-event time interval.
be defined in two different ways: (1) by a mode transition As opposed to an event, an activity spans a certain time
or (2) by the system reaching a threshold. The first type interval. Furthermore, the start and the end of an activity
could be a mode transition caused by a sudden driver input. are marked by an event.
An event might also be induced by an external cause, such 2) An activity quantitatively describes the time evolution of
as an environmental change. The second type of event, i.e., one or more state variables. Because activities are build-
related to the system reaching a threshold, is especially useful ing blocks of a scenario and a scenario corresponds to a
when describing test scenarios. For example, consider the ego quantitative description, the activities themselves need to
vehicle approaching a pedestrian that is about to cross the be quantitative as well. Therefore, an activity describes
road [63]. Here, the event marks the moment that the distance the time evolution of one or more state variables, i.e.,
between the vehicle and pedestrian is less than dv,p meters. the trajectory of one or more state variables over an
At the moment of this event, the pedestrian starts to cross the inter-event time interval that corresponds to the activity,
road such that the vehicle would impact with the pedestrian where the term state variable is defined in Table I.
if it would not change its speed or direction [63]. By using 3) An activity is performed by an actor. An activity de-
a variable threshold dv,p , the value is flexible and can be set scribes the time evolution of one or more state variables
differently to define multiple scenarios. and a state variable corresponds to an actor, e.g., the
For the practical implementation of events, a set of condi- acceleration of a vehicle.
tions may be specified. In that case, the event occurs at the Hence, we define an activity as follows:
moment that the conditions are met. In [29], an extensive list
Definition 3 (Activity). An activity is a quantitative descrip-
of possible conditions that can be used to define an event is
tion of the time evolution of one or more state variables of an
given. E.g., a condition could be that the distance between the
actor between two events.
vehicle and the pedestrian is below a certain threshold.
As an example, an activity could describe the longitudinal
5 “An entity that exists in space and time” [60]. acceleration (or, e.g., speed) during an acceleration or decel-
 6

eration. Activities describing the lateral position of a vehicle A given scenario category can comprise multiple scenarios and
with respect to the center of the corresponding lane might, multiple scenario categories can comprise a specific scenario.
e.g., be labeled with “driving straight” or “changing lane”. As a consequence, as opposed to the proposed categorization
of scenarios in [68, 66, 69, 73], scenario categories do not
D. Scenario category need to be mutually exclusive.
The verb “to include” is used to describe the relation
According to Definition 1, a scenario in the context of the between two scenario categories. A scenario category C2 is
performance assessment of an AV needs to be quantitative. said to include a scenario category C1 if C2 comprises all
However, in literature, the term scenario is also used to scenarios that are comprised in C1 . In that case, we can write
refer to a collection of scenarios, where this collection of C2 ⊇ C1 . Thus we have
scenarios is described qualitatively. For example, in [66], a
typology of pre-crash scenarios is proposed. Here, each of C2 ⊇ C1 if C2 3 S ∀ {S : C1 3 S}. (1)
the pre-crash scenarios is an abstraction of many quantitative
We propose to provide scenarios and scenario categories
scenarios. Similar studies have been performed to describe
with additional information in the form of tags. A tag is a
scenarios that lead to highway accidents [67], car-cyclist
keyword or a keyphrase that provides extra information on
accidents [68], and car-pedestrian accidents [69]. In [70], a
a piece of data [74]. For example, items in a database can
taxonomy of scenarios is proposed to qualitatively describe
contain some tags that enable users to quickly retrieve several
challenging scenarios for automated driving. In [23], a distinc-
items that share a certain characteristic described by a tag [75].
tion is made between so-called functional scenarios, abstract
The use of these tags brings several benefits:
scenarios, logical scenarios, and concrete scenarios. These
four types of scenario descriptions represent different levels of • The tags of a scenario can be helpful in determining

abstraction with functional scenarios referring to non-formal which scenario categories do and do not comprise the
human-readable scenarios, abstract scenarios referring to for- scenario.
malized declarative descriptions, logical scenarios referring to • It is easy to select scenarios from a scenario database or

parameterized scenarios with ranges and distributions of the a scenario library by using tags or a combination of tags.
parameters, and concrete scenarios referring to parameterized There is a balance between having generic scenario cat-
scenarios with fixed parameters values. egories — and thus a wide variety among the scenarios
The aforementioned references [66, 67, 68, 69, 70, 23] comprised by the scenario category — and having specific
show that the term scenario is also used to address qualitative scenario categories without much variety among the scenarios
descriptions. Since we define a scenario as a quantitative de- comprised by the scenario category. For some systems, one
scription, we need to introduce a different term to address the may be interested in a very specific set of scenarios, while
qualitative description. We propose to use the term scenario for another system one might be interested in a set of sce-
category to refer to the qualitative description of a scenario. narios with a high variety. To accommodate this, tags can be
A qualitative description can be regarded as an abstraction of structured in hierarchical trees [76]. The different layers of the
a quantitative scenario, whereas a quantitative description can trees can be regarded as different abstraction levels [77].
be regarded as a concretization of a qualitative description. Figure 1 shows two examples of trees of tags taken from
We thus define a scenario category as follows: [16]. These tags describe possible activities of a vehicle,
i.e., the lateral motion control (via steering) and longitudinal
Definition 4 (Scenario category). A scenario category is motion control (via acceleration and deceleration). The tags
a qualitative description of the relevant characteristics and may refer to the objective of an actor in case no activities are
activities and/or goals of the ego vehicle(s), the static envi- defined. For example, a test case in which the ego vehicle’s
ronment, and the dynamic environment. objective is to make a left turn, the tags “Turning” and “Left”
Introducing the concept of scenario categories brings the are applicable. Note that tags may be used not only to classify
following benefits: vehicle behavior, but also traffic and environment situations,
• For a human, it is often easier to interpret a qualitative e.g., “cut-in” or “heavy rain”.
description than a quantitative description.
• Scenarios that have something in common can be grouped IV. O BJECT- ORIENTED FRAMEWORK FOR SCENARIOS
together, which enables characterization of types of sce- We have already explained the use of an OOF in Sec-
narios and facilitates discussion of scenarios. tion II-A. In this section, we present our OOF for scenarios
• The completeness of a set of scenarios can be assessed for the assessment of AVs. The overview of the framework is
by considering the completeness of scenario categories formally represented through class diagrams that are briefly
(see, e.g., [71]) and the completeness of scenarios in each presented in Section IV-A. Next, Section IV-B explains how
category (see, e.g., [72]). a scenario category is formally represented in our framework.
We describe the formal relation between a scenario and a Similarly, in Section IV-C, we describe how a scenario is for-
scenario category with the verb “to comprise”, denoted by 3. mally represented. The OOF can be implemented straightfor-
If a specific scenario category C is an abstraction of a specific wardly in object-oriented languages such as C++ and Python,
scenario S, then we say that C comprises S, or simply C 3 S. since these languages support the definition of classes, the
 7

Vehicle lateral activity

Going straight Changing lane Turning Swerving

Left Left Left

Right Right Right

(a) Lateral activities of a vehicle.

Vehicle longitudinal activity

Reversing Standing still Driving forward

Decelerating

Cruising

Accelerating
(b) Longitudinal activities of a vehicle.
Figure 1: Tags for lateral and longitudinal activities of a vehicle [16]. The lateral activity is relative to the lane in which the
corresponding vehicle is driving.

instantiation of objects from those classes, and concepts such one, or multiple objects of the corresponding class. The arrow
as inheritance and aggregation. An actual implementation of with the text “comprises” and “includes” represent methods
the OOF in a coding language is publicly available at https: that are explained in Section III-D. Here, “comprises” can be
//github.com/ErwindeGelder/ScenarioDomainModel. This link denoted by 3 and “includes” can be denoted by ⊇, see (1).
also contains tutorials for the technical application of the OOF.

B. Scenario category and its attributes

A. Class diagrams
Because all other classes in Figure 2 are subclasses of
In Figures 2 and 3, the gray blocks represent the classes6 Scenario element, these classes inherit the attributes and
that are used to describe a scenario category according to procedures of Scenario element. In our framework, a sce-
Definition 4 and the white solid blocks represent the classes nario element has a human-interpretable name, a unique ID,
that are used to describe a scenario according to Definition 1. and possibly predefined tags that are also interpretable by a
The white dashed blocks represent so-called abstract classes. software agent. So, all other classes in Figure 2 also have
Abstract classes cannot be instantiated. Each class serves as a these attributes. In addition to these attributes, the Qualitative
template for creating objects whereas an object of a particular element class has a human-interpretable description.
class is referred to as the instance of that particular class. The static environment is qualitatively described by one
Figure 2 shows the class-level relationships while Figure 3 or more physical element categories. Because physical ele-
shows the instance-level relationships. In Figure 2, the arrow ment categories qualitatively describe the static environment,
from, e.g., Scenario to Time interval, denotes that Scenario they contain a human-interpretable description of the physical
is a subclass of Time interval. Therefore, all properties of the things they describe.
Time interval are inherited by Scenario. The arrow with the The ego vehicle(s) and the dynamic environment are quali-
diamond in Figure 3 denotes an aggregation. This means that, tatively described by activity categories and actor categories.
e.g., an actor, which is an instance of the Actor class, has In line with Definition 3, Activity category includes the state
an actor category as an attribute. Here, the “1” at the start variable(s). The Model that is used to describe the time evolu-
of the arrow from Actor category to Actor indicates that an tion of the state variable(s) is specified. Note that Model is an
actor has exactly one actor category. Similarly, “2” at the abstract class that serves as a template for different models,
aggregation arrow from Event to Time interval indicates that such as the three examples shown in Figure 2: Sinusoidal,
a time interval contains two events, i.e., the events that define Linear, and Constant. Let z(t) denote the state variable at
the start and the end of the time interval. A “0, 1, . . .” at the time t, then the Sinusoidal model is defined as follows:
start of an aggregation arrow indicates that an object has zero,  
πA π (t − t0 )
6 In the remainder of this paper, when referring to (an instance of) a class,
ż(t) = sin , t ∈ [t0 , t0 + T ], (2)
2T T
italic font is used. Additionally, class names start with capital letters and
instance names with lowercase letters. z(t0 ) = z0 . (3)
 8

Qualitative Quantitative
Scenario element
element element

Physical element Scenario

Activity category Time interval Event Physical element
category category

Actor category Model Scenario Activity Actor

Legend
Abstract class
Sinusoidal Linear Constant Class for qualitative description

Class for quantitative description

Figure 2: Class-level relationships of most classes of our Object-Oriented Framework (OOF).

Legend
Scenario category includes
Abstract class

Class for qualitative description

0, 1, . . . 0, 1, . . . 0, 1, . . .
Class for quantitative description
Physical element
comprises Actor category Activity category
category
1 1 1

Scenario Model Time interval

0, 1, . . . 0, 1, . . . 0, 1, . . . 2, 3, . . . 2

Actor Activity Physical element Event

Figure 3: Instance-level relationships of most classes of our Object-Oriented Framework (OOF).

Here, the amplitude (A), duration (T ), initial time (t0 ), and The Actor category is a subclass of Physical element cat-
initial state (z0 ) are parameters. The Linear and Constant egory so Actor category inherits the properties of Physical
models are described by the following equations, respectively: element category. In addition, Actor category has an attribute
that specifies the type of object. To indicate that an actor is
ż(t) = s, z(t0 ) = z0 , (4) an ego vehicle, the tag “Ego vehicle” is added to the list of
tags of the actor category.
z(t) = z0 . (5)
The Scenario category has physical element categories,
The Linear model contains three parameters, i.e., the slope activity categories, and actor categories as attributes. Another
(s), initial time (t0 ), and initial state (z0 ). The Constant attribute of the Scenario category is the list of acts. These acts
model only has the parameter z0 . Since an activity category describe which actors perform which activities. Note that it is
is a qualitative description, the values of the parameters of possible that one actor performs multiple activities and that
its model are not part of the activity category. Note that one activity is performed by multiple actors.
this article only considers the models Sinusoidal, Linear, and The reader might wonder why we introduce the different
Constant, but more complex models may be necessary to classes for describing a scenario category, i.e., the gray blocks,
describe complex behavior. More complex models are out- instead of only one class for modeling a scenario category.
of-scope of this article, but it is straightforward to extend the The main advantage of the different classes is the reusability
OOF with such models. of the instances of the classes, because these instances can be
 9

exchanged among different scenario categories. For example,

if two scenario categories have the same actor categories, we y
only need to define the actor categories once, whereas if the a
actor categories would not be instances of a class but only O x
properties of the scenario category, we would need to define
the actor categories twice.

Figure 4: Schematic overview of a scenario where both the

C. Scenario and its attributes
ego vehicle and a pedestrian are approaching a non-signalized
To distinguish objects that are directly used to compose pedestrian crossing. The pedestrian has priority.
a scenario, these objects are instantiated from subclasses
of the Quantitative element class. The class Scenario is a
subclass of Time interval and, therefore, it has events that V. E XAMPLE : PEDESTRIAN CROSSING
define the start and the end of the scenario. The Scenario To illustrate the use of the OOF, we describe a scenario
also has physical element, activities, actors, and events as using objects of the classes presented in Section IV. The
attributes. The physical elements, activities, and actors are the scenario is schematically shown in Figure 4. The ego vehicle is
quantitative counterparts of the physical element categories, driving on the right lane of a two-lane road and a pedestrian is
activity categories, and actor categories, just as a scenario is walking on a footway that intersects the road the ego vehicle is
the quantitative counterpart of a scenario category. As with driving on. Both the ego vehicle and the pedestrian are initially
the Scenario category, the Scenario contains a list of acts that approaching the pedestrian crossing. The ego vehicle brakes
describe which actors perform which activities. and comes to a full stop in front of the pedestrian crossing.
A physical element has a physical element category and While the ego vehicle is stationary, the pedestrian crosses the
it may have multiple properties that quantitatively define the road using the pedestrian crossing. When the pedestrian has
object, such as its size, weight, color, radar cross section, etc. passed the ego vehicle, the ego vehicle accelerates. The code
Physical elements can be used to define, e.g., the road layout, of this example is publicly available7 .
static weather and lighting conditions, and infrastructural This particular scenario can be used to formulate a test
elements. scenario for the assessment of an AV. For example, when
assessing a pedestrian automatic emergency braking system
According to Definition 3, an activity quantitatively de-
[63], we are interested in the behavior of the system in case
scribes the evolution of one or more state variables in a
the driver or automation system of the ego vehicle does not
time interval. The state variable(s) are defined by the activity
brake.
category that the activity has as an attribute. Together with
We first describe the scenario qualitatively using our pro-
the Model that is contained by the activity category, the
posed framework. Next, the scenario is described quantita-
time evolution of the state variable is described by a set
tively in Section V-B. In Section V-C, we show which objects
of parameters. The values of the parameters are part of the
are reused and which objects are different if we consider an
activity.
actual test scenario with a crossing pedestrian.
Following Definition 2, an event contains conditions that
describe the threshold or mode transition at the time of the
event. A. Qualitative description of the pedestrian crossing
Similar to a physical element and an activity, an actor has its To describe the scenario according to the presented domain
qualitative counterpart — an actor category — as an attribute. model, objects are instantiated from the classes presented in
Additionally, the Actor contains an initial state vector and a Figures 2 and 3. Figure 5 shows the objects for describing
desired state vector that can be used to specify the intent, the scenario qualitatively. There are two actor categories:
as attributes. Describing the intent is especially useful for one for the ego vehicle and one for the pedestrian. Four
defining a test scenario in terms of the objective of the ego different activity categories are defined: braking, stationary,
vehicle rather than its activities. accelerating, and walking straight. The braking, stationary,
An advantage of having the qualitative counterparts of the and accelerating activity categories contain the state variable
Physical element, Activity, and Actor is that the qualitative vego , i.e., the speed of the ego vehicle, and use the Sinusoidal
description can be reused and exchanged. For example, there model of (2) and (3), the Constant model of (5), and the Linear
can be many different braking activities, but there needs to model of (4), respectively. The activity category walking
be only one activity category for qualitatively defining the straight has the position of the pedestrian (yped ) as its state
braking activity. Here, it is assumed that all braking activities variable and uses the Linear model of (4).
are modeled with the same model and that similar tags apply. The two actor categories, the four activity categories, and
If this is not the case, multiple activity categories need to the physical element category that represents the crosswalk,
be defined, but the number of activity categories will still be 7 See https://github.com/ErwindeGelder/ScenarioDomainModel. The repos-
substantially lower than the number of activities. itory also contains other examples.
 10

Crossing pedestrian::Scenario category Ego qualitative::Actor category Accelerating::Activity category

description: A pedestrian is crossing the road type: Vehicle model: Linear
on a zebra crossing in front of the tags: Ego vehicle, Passenger car state variable: Speed (vego )
ego vehicle tags: Accelerating
physical element: Pedestrian crossing Pedestrian qualitative::Actor category
qualitative type: Pedestrian Walking straight::Activity category
actors: Ego qualitative, Pedestrian qualitative tags: Pedestrian model: Linear
activities: Braking, Stationary, Accelerating, state variable: Position (yped )
Walking straight Braking::Activity category tags: Walking straight
acts: (Ego qualitative, Braking), model: Sinusoidal
(Ego qualitative, Stationary), state variable: Speed (vego ) Pedestrian crossing qualitative::Physical
(Ego qualitative, Accelerating), tags: Braking element category
(Pedestrian qualitative, Walking straight) description: Straight road with two lanes and a
tags: Stationary::Activity category pedestrian crossing
model: Constant tags: Non-signalized zebra crossing
state variable: Speed (vego )
tags: Stationary

Figure 5: The objects that are used to qualitatively describe the scenario that is schematically shown in Figure 4. The first line
of each block shows the name (before the double colon) and the class from which the object is instantiated. The following
lines show the attributes of the object with the name and value of the attribute before and after the colon, respectively. For the
sake of brevity, the unique ID of each object is omitted.

are used by the scenario category. The scenario category has Figure 6. As mentioned in Section III-A, it is possible to refer
four acts. The first three acts assign the first three activity to another source that contains a description of (part of) the
categories to the ego vehicle. The last act assigns the activity static environment, see, e.g., [55].
category walking straight to the pedestrian. The scenario has the previously defined physical element,
actors, and activities as attributes. The acts are used to assign
B. Quantitative description of the pedestrian crossing the first three activities to the ego vehicle and the last activity
to the pedestrian. The scenario also has events marking the
The objects to describe the scenario quantitatively are
start and the end of the scenario. A different scenario can be
shown in Figure 6. The two actors refer to the quantitative
defined by, e.g., changing the parameter values. This illustrates
counterparts of the actor categories in Figure 5. Initial state
that the scenario category in Figure 5 comprises multiple
vectors are listed for each actor using the coordinate frame
scenarios, including the scenarios that only differ from the
that is shown in Figure 4. Since we are describing a real-
scenario in Figure 6 because of different parameter values.
world scenario, there is no need to define goals or intents for
the actors.
There are four events defined. These events mark the time C. Test scenario of the pedestrian crossing
instants that define the start and the end of the activities. For In this example, we consider a test scenario based on the
simplicity, it is assumed that the start of the scenario occurs previously illustrated real-world scenario, see Figure 4. To
at 0 s. describe the test scenario, we reuse the two actor categories
There are four activities defined and each of these activities from Figure 5 (ego qualitative and pedestrian qualitative) and
refers to its qualitative counterpart. The activities contain the the actor describing the pedestrian from Figure 6 (pedestrian
values of the parameters as well as events that mark the start crossing). Figure 7 shows the other objects that are used to
and the end of the activities. As described by the first activity describe this test scenario.
(ego braking), the ego vehicle starts with a speed of 8 m s−1 The scenario category only differs from the scenario cat-
and brakes in 4 s to come to a full stop. By integrating the egory shown in Figure 5 in that it does not contain activity
sinusoidal function of (2) twice, it can be shown that the ego categories that describe the activity of the ego vehicle.
vehicle stops at 4 m from the center of the pedestrian crossing. Two attributes of the quantitative description of the ego
After waiting for 3 s as described by the second activity (ego vehicle are different. First, the initial state vector also includes
stationary), the ego vehicle accelerates with 1.5 m s−2 towards the speed, denoted by vego , at the start of the scenario and the
a speed of 7.5 m s−1 as described by the third activity (ego initial position is further away from the pedestrian crossing,
accelerating). The fourth activity describes the position and such that the ego vehicle’s driver or automation system has
speed of the pedestrian. more time to perceive the pedestrian. Second, because there
The pedestrian crossing describes the entire static environ- are no activities defined for the ego vehicle, the desired state
ment, including the main road the ego vehicle is driving on vector is defined. The goal is to reach the point 80 m in front of
and the footway the pedestrian is walking on. The example in the ego vehicle while driving with a speed of vego = 8 m s−1 .
Figure 6 shows some properties of the road layout to illustrate The event that marks the start of the walking activity of the
how the static environment can be described. Note that, in pedestrian is triggered if the ego vehicle is 2.5 s away from
practice, the quantitative description of the static environment the center of the footway, assuming that the speed of the ego
may contain many more facets than the ones mentioned in vehicle is constant. In case the ego vehicle drives with a speed
 11

Ego brakes for pedestrian::Scenario Start scenario::Event Ego accelerating::Activity

physical element: Pedestrian crossing time: 0 s activity category: Accelerating
actors: Ego, Pedestrian parameters: s = 1.5 m s−2 , z0 = 0 m s−1 ,
activities: Ego braking, Ego stationary, End braking::Event t0 = 7 s
Ego accelerating, Pedestrian walking time: 4 s start event: Start accelerating
acts: (Ego, Ego braking), end event: End scenario
(Ego, Ego stationary), Start accelerating::Event
(Ego, Ego accelerating), time: 7 s Pedestrian walking::Activity
(Pedestrian, Pedestrian walking) activity category: Walking
End scenario::Event parameters: s = 1 m s−1 , z0 = −6 m
start event: Start scenario time: 12 s
end event: End scenario t0 = 0 s
Ego braking::Activity start event: Start scenario
Ego::Actor activity category: Braking end event: End scenario
actor category: Ego qualitative parameters: A = −8 m s−1 , T = 4 s,
properties: {width=1.8 m, length=4.5 m} Pedestrian crossing::Physical element
z0 = 8 m s−1 , t0 = 0 s physical element category: pedestrian crossing
initial state vector: xego = −20 m, start event: Start scenario
yego = −1.5 m, qualitative
end event: End braking properties: {road: {lanes: 2, lanewidth: 3 m,
aego = 90°
desired state vector: xy: [(-60, 0), (60, 0)]},
Ego stationary::Activity footway: {width: 3 m,
Pedestrian::Actor activity category: Stationary xy: [(0, 6), (0, -6)]}}
actor category: Pedestrian qualitative parameters: z0 = 0 m s−1
properties: {width=0.5 m, color=blue} start event: End braking
initial state vector: xped = 0 m, aped = 0° end event: Start accelerating
desired state vector:

Figure 6: The objects that are used to quantitatively describe the scenario that is schematically shown in Figure 4. For the sake
of brevity, the tags and the unique ID of each object are omitted.

Crossing pedestrian::Scenario category Ego::Actor Start walking::Event

description: A pedestrian is crossing the road actor category: Ego qualitative condition: |xego /vego | ≤ 2.5 s
on a zebra crossing in front of the properties: {width=1.8 m, length=4.5 m}
ego vehicle initial state vector: xego = −60 m, End walking::Event
physical element: Pedestrian crossing yego = −1.5 m, contition: yped = 6 m
qualitative aego = 90°,
actors: Ego qualitative, Pedestrian qualitative vego = 8 m s−1 Pedestrian walking::Activity
activities: Walking straight desired state vector: xego = 20 m, activity category: Walking
acts: (Pedestrian qualitative, Walking straight) yego = −1.5 m, parameters: s = 1 m s−1 , z0 = −6 m,
aego = 90°, t0 = At Start walking
Ego must brake for pedestrian::Scenario vego = 8 m s−1 start event: Start walking
physical element: Pedestrian crossing end event: End walking
actors: Ego, Pedestrian
activities: Pedestrian walking End scenario::Event
acts: (Pedestrian, Pedestrian walking) condition: xego ≥ 20 m OR collision OR
events: Start walking, Stop walking yego ≤ −2 m OR yego ≥ −1 m
start event: Start scenario OR t > 100 s.
end event: End scenario

Figure 7: The objects that, together with the objects Ego qualitative, Pedestrian qualitative, Walking straight, and Pedestrian
crossing qualitative from Figure 5 and Start scenario, Pedestrian, and Pedestrian crossing from Figure 6, describe a test
scenario that is schematically shown in Figure 4. For the sake of brevity, the tags and the unique ID of each object are omitted.

of vego = 8 m s−1 , this is at a distance of 20 m, similar to the this activity is at a predefined event with, e.g., the condition
scenario described in Figure 6. |xego /vego | ≤ 1 s AND yped < 0 m.
As with the scenario category, the scenario does not contain
activities of the ego vehicle. Furthermore, the end event of D. Remarks on the example
the scenario is defined differently: now the scenario ends if
the ego vehicle either reaches its destination (xego ≥ 20 m), The example illustrates the benefits of the object-oriented
collides with the pedestrian, deviates too much from its path approach for defining a scenario, which are:
(yego ≤ −2 m or yego ≥ 1 m), or takes too long to reach the • clarity regarding the content of the scenario,
destination (t > 100 s). • modularity, which makes it easy to understand the

Note that this example considers a pedestrian that crosses changes from the real-world scenario in Figure 6 to the
the road at a fixed speed (1 m s−1 ) regardless of the proximity test scenario in Figure 7, and
of the ego vehicle. To model, e.g., the case where the pedes- • reusability, as is illustrated by the objects that are used

trian notices the ego vehicle and accelerates if a collision is more than once.
about to happen, an activity can be added that describes the Furthermore, each object listed in Figures 5 to 7 is directly
increased speed (e.g., 2 m s−1 ) of the pedestrian. The start of translatable to an object in an object-oriented programming
 12

language. As a further illustration that the presented OOF relationships that enable automated reasoning. In this way, an
is practical to use in real life, the framework is used by ontology enables automated classification of scenarios, thereby
TNO’s StreetWise program for storing real-world scenarios helping to overcome problems of data ambiguity [52].
in a database [11]8 .
In the example, two different actors are considered: the A PPENDIX A
ego vehicle and the pedestrian. These are examples of traffic N OMENCLATURE
participants, but an actor is not necessarily a traffic participant. For the definition of scenario, several notions are adopted
For example, road side units that transmit messages in an from the literature. In this section, the concepts of ego vehicle,
infrastructure-to-vehicle communication setting can also be physical element, actor, static environment, dynamic environ-
actors. In this case, the transmission of messages can be ment, act, state variable, state vector, model, and mode, which
considered as an activity. Another example of an actor is the are adopted from literature, are detailed.
road surface in case it is important for the scenario to model
the changing surface temperature.
A. Ego vehicle
The ego vehicle is the main subject of a scenario. In partic-
VI. C ONCLUSIONS
ular, the ego vehicle refers to the vehicle that is perceiving the
The performance assessment of Automated Vehicles (AVs) world through its sensors (see, e.g., [77]). When performing
is essential for the legal and public acceptance of AVs as well tests, the ego vehicle also refers to the vehicle that must
as for the technology development of AVs. Because scenarios perform a specific task (see, e.g., [30, 35]). In this case, the
are crucial for the assessment, a clear definition of a scenario ego vehicle is often referred to as the system under test [4],
is required. In this work, we have proposed a new definition the vehicle under test [5], or the host vehicle [5].
of the concept scenario in the context of the performance
assessment AVs.
B. Physical element
While our definition is consistent with other definitions from
the literature, it is more concrete such that it can directly be A physical element refers to an object that exists in the
implemented using code. We have further defined the notions three-dimensional space.
of event, activity, and scenario category. To formalize the
concepts of scenario, event, activity, and scenario category, C. Actor
an Object-Oriented Framework (OOF) has been proposed. According to Frost [35], “actors are all dynamic components
Using the proposed framework, it is possible to describe a of a scenario, excluding the ego vehicle itself.” Note that, in
scenario in both a qualitative and quantitative manner. The contrast to [35], in the current paper, the ego vehicle’s driver,
framework, represented using class diagrams, can be directly and/or automation system are considered as actors, similar to
translated into a class structure for an object-oriented software [14], because they have the same properties as another driver
implementation. This allows us to translate scenarios into or automation system. While the aforementioned definition of
code, such that both domain experts and software programs, Frost [35] provides a good idea of what an actor could be, we
such as simulation tools, are able to understand the content of use another definition in order to avoid a circular definition: an
the scenarios. To demonstrate this, we have made our imple- actor is a dynamic physical element, i.e., a physical element
mentation in the coding language Python publicly available. that experiences change.
The OOF has been illustrated with an example of an
Remark 2. An actor is also a physical element whereas a
urban scenario with a pedestrian crossing. We have also
physical element is not necessarily an actor. For example, a
demonstrated how this particular scenario can be used to
static road sign is considered a physical element, but because
define a test scenario using the proposed framework. In the
it does not change during the course of a scenario, it is not an
publicly available9 coding implementation of the presented
actor. ♦
OOF, we have shown how to use the proposed OOF from
a real application’s perspective.
The presented framework is applicable for scenario mining D. Static environment
[78, 39] and scenario-based assessment [11, 6] and, there- The static environment refers to the part of the environment
fore, this framework provides a step towards scenario-based that does not change during a scenario. This includes geo-
performance assessment of AVs. The next step is to define spatially stationary elements [13], such as the road network.
scenarios and scenario categories10 that are relevant for an
AV in a specific deployment area. Future work also includes E. Dynamic environment
creating an ontology for scenarios for the assessment of AVs. As opposed to the static environment, the dynamic envi-
The presented OOF could be a good starting point for this ronment refers to the part of the environment that changes
[41]. An ontology allows, among others, to add properties to during the time frame of a scenario. In practice, the dynamic
8 See also https://www.tno.nl/streetwise.
environment mainly consists of the moving actors (other than
9 https://github.com/ErwindeGelder/ScenarioDomainModel the ego vehicle) that are relevant to the ego vehicle. For
10 As a starting point, the 67 scenario categories in [16] can be used. example, the primary use case of OpenSCENARIO [29], a file
 13

format for the description of the dynamic content of driving ACKNOWLEDGMENT

simulations, is to describe “complex, synchronized maneuvers The research leading to this paper has been partially realized
that involve multiple entities like vehicles, pedestrians, and with the Centre of Excellence for Testing and Research of
other traffic participants” [29]; so for OpenSCENARIO, these Autonomous Vehicles at NTU (CETRAN), Singapore.
maneuvers represent the dynamic environment. Roadside units We would like to thank Mark van den Brand and Ludwig
that communicate with vehicles within the communication Friedmann for providing helpful feedback on earlier versions
range [79] are also part of the dynamic environment. Further- of this article.
more, changing (weather) conditions are part of the dynamic
environment.
R EFERENCES
Remark 3. Note that it might not always be obvious whether
an element of the environment belongs to the static or dynamic [1] K. Bengler, K. Dietmayer, B. Färber, M. Maurer,
environment. Most important, however, is that all parts of the C. Stiller, and H. Winner, “Three decades of driver assis-
environment that are relevant to the assessment of an AV are tance systems: Review and future perspectives,” IEEE In-
described in either the static or the dynamic environment. ♦ telligent Transportation Systems Magazine, vol. 6, no. 4,
pp. 6–22, 2014.
[2] W. Wachenfeld and H. Winner, “The release of au-
F. Act tonomous vehicles,” in Autonomous Driving. Springer,
We define an act as a combination of an actor and the 2016, pp. 425–449.
activity that is performed by the actor or the combination [3] T. Helmer, K. Kompaß, L. Wang, T. Kühbeck, and
of actors and the activities they are subjected to. This is in R. Kates, Safety Performance Assessment of Assisted and
accordance with the use of the term act in [29]. Automated Driving in Traffic: Simulation as Knowledge
Synthesis. Springer International Publishing, 2017, pp.
473–494.
G. State variable
[4] J. E. Stellet, M. R. Zofka, J. Schumacher, T. Schamm,
Dorf and Bishop [80, p. 163] write that “the state variables F. Niewels, and J. M. Zöllner, “Testing of advanced driver
describe the present configuration of a system and can be used assistance towards automated driving: A survey and
to determine the future response, given the excitation inputs taxonomy on existing approaches and open questions,”
and the equations describing the dynamics.” In our case, “the in IEEE 18th International Conference on Intelligent
system” could refer to an actor, a component, or a simulation. Transportation Systems, 9 2015, pp. 1455–1462.
For example, a state variable could be the acceleration of an [5] O. Gietelink, J. Ploeg, B. De Schutter, and M. Verhaegen,
actor. “Development of advanced driver assistance systems
with vehicle hardware-in-the-loop simulations,” Vehicle
H. State vector System Dynamics, vol. 44, no. 7, pp. 569–590, 2006.
[6] A. Pütz, A. Zlocki, J. Bock, and L. Eckstein,
A state vector refers to “the vector containing all n state “System validation of highly automated vehicles with
variables” [80, p. 233]. a database of relevant traffic scenarios,” in 12th
ITS European Congress, 2017, pp. 1–8. [Online].
I. Model Available: https://www.pegasusprojekt.de/files/tmpl/pdf/
12th%20ITS%20European%20Congress_Folien.pdf
A dynamical system is often modeled using a differential
[7] C. Roesener, J. Sauerbier, A. Zlocki, F. Fahrenkrog,
equation of the form ż(t) = fθ (z(t), u(t), t) [81], where z(t)
L. Wang, A. Várhelyi, E. de Gelder, J. Dufils, S. Breunig,
represents the state vector at time t, u(t) represents an external
P. Mejuto, F. Tango, and J. Lanati, “A comprehensive
input vector, and the function fθ (·) is parameterized by θ.
evaluation approach for highly automated driving,” in
Note that, technically speaking, z(·), u(·), t, and θ are inputs
25th International Technical Conference on the En-
of the function f , but θ is assumed to be constant for a certain
hanced Safety of Vehicles (ESV), 2017.
time interval. For example, the following first-order model is
[8] S. Riedmaier, T. Ponn, D. Ludwig, B. Schick, and
parameterized by θ = (a, b):
F. Diermeyer, “Survey on scenario-based safety assess-
ż(t) = az(t) + bu(t). (6) ment of automated vehicles,” IEEE Access, vol. 8, pp.
87 456–87 477, 2020.
[9] A. Knapp, M. Neumann, M. Brockmann, R. Walz, and
J. Mode T. Winkle, “Code of practice for the design and evalua-
In some systems, the behavior of the system may suddenly tion of ADAS,” RESPONSE III: a PReVENT Project, 8
change abruptly, e.g., due to a sudden change in an input, a 2009.
model parameter, or the model. Such a transition is called a [10] ISO 26262, “Road Vehicles – Functional Safety,”
mode switch. In each mode, the behavior of the system is International Organization for Standardization, Standard,
described by a model with a fixed function fθ and smooth 2018. [Online]. Available: https://www.iso.org/standard/
input u(·) [59]. 68383.html
 14

[11] H. Elrofai, J.-P. Paardekooper, E. de Gelder, [21] Z. Xiong and J. Olstam, “Orchestration of driving sim-
S. Kalisvaart, and O. Op den Camp, “Scenario- ulator scenarios based on dynamic actor preparation
based safety validation of connected and automated and automated action planning,” Transportation Research
driving,” Netherlands Organization for Applied Part C: Emerging Technologies, vol. 56, pp. 120–131,
Scientific Research, TNO, Tech. Rep., 2018. [On- 2015.
line]. Available: http://publications.tno.nl/publication/ [22] Y. Shao, M. A. Mohd Zulkefli, Z. Sun, and P. Huang,
34626550/AyT8Zc/TNO-2018-streetwise.pdf “Evaluating connected and autonomous vehicles using
[12] A. Aparicio, S. Baurès, J. Bargalló, C. Rodarius, J. Vis- a hardware-in-the-loop testbed and a living lab,” Trans-
sers, O. Bartels, P. Seiniger, P. Lemmen, T. Unselt, portation Research Part C: Emerging Technologies, vol.
M. Ranovona, T. Okawa, and S. Schaub, “Pre-crash per- 102, pp. 121–135, 2019.
formance of collision mitigation and avoidance systems: [23] C. Neurohr, L. Westhofen, M. Butz, M. Bollmann,
Results from the ASSESS project,” in FISITA 2012 World U. Eberle, and R. Galbas, “Criticality analysis for the
Automotive Congress, 2013, pp. 489–505. verification and validation of automated vehicles,” IEEE
[13] S. Ulbrich, T. Menzel, A. Reschka, F. Schuldt, and Access, vol. 9, pp. 18 016–18 041, 2021.
M. Maurer, “Defining and substantiating the terms scene, [24] F. Rosique, P. J. Navarro, C. Fernández, and A. Padilla,
situation, and scenario for automated driving,” in IEEE “A systematic review of perception system and simulators
18th International Conference on Intelligent Transporta- for autonomous vehicles research,” Sensors, vol. 19,
tion Systems, 2015, pp. 982–988. no. 3, p. 648, 2019.
[14] S. Geyer, M. Baltzer, B. Franz, S. Hakuli, M. Kauer, [25] P. Wimmer, M. Düring, H. Chajmowicz, F. Granum,
M. Kienle, S. Meier, T. Weißgerber, K. Bengler, J. King, H. Kolk, O. Op den Camp, P. Scognamiglio,
R. Bruder, F. Flemisch, and H. Winner, “Concept and and M. Wagner, “Toward harmonizing prospective effec-
development of a unified ontology for generating test and tiveness assessment for road safety: Comparing tools in
use-case catalogues for assisted and automated vehicle standard test case simulations,” Traffic Injury Prevention,
guidance,” IET Intelligent Transport Systems, vol. 8, vol. 20, no. sup1, pp. S139–S145, 2019.
no. 3, pp. 183–189, 2014. [26] Q. Chao, H. Bi, W. Li, T. Mao, Z. Wang, M. C. Lin, and
[15] M. R. Zofka, F. Kuhnt, R. Kohlhaas, C. Rist, Z. Deng, “A survey on visual traffic simulation: Models,
T. Schamm, and J. M. Zöllner, “Data-driven simulation evaluations, and applications in autonomous driving,” in
and parametrization of traffic scenarios for the devel- Computer Graphics Forum, vol. 39, no. 1. Wiley Online
opment of advanced driver assistance systems,” in 18th Library, 2020, pp. 287–308.
International Conference on Information Fusion, 7 2015, [27] P. Kaur, S. Taghavi, Z. Tian, and W. Shi, “A survey on
pp. 1422–1428. simulators for testing self-driving cars,” arXiv preprint
[16] E. de Gelder, O. Op den Camp, and N. de Boer, arXiv:2101.05337, 2021.
“Scenario categories for the assessment of automated [28] A. Nehemiah, P. Fryscak, and M. Sasena, “Building
vehicles,” CETRAN, Tech. Rep., 2020, version 1.7. an Autonomous Vehicle (AV) simulation toolchain
[Online]. Available: http://cetran.sg/wp-content/uploads/ with Simulink, RoadRunner and NVIDIA DRIVE
2020/01/REP200121_Scenario_Categories_v1.7.pdf Sim,” Mathworks, Blog post, 2021. [Online]. Available:
[17] H. Weber, J. Bock, J. Klimke, C. Roesener, J. Hiller, https://blogs.mathworks.com/student-lounge/2021/04/26/
R. Krajewski, A. Zlocki, and L. Eckstein, “A framework building-an-autonomous-vehicle-av-simulation-toolchain-with-simul
for definition of logical scenarios for safety assurance of [29] (2019) OpenSCENARIO. Accessed August, 2020.
automated driving,” Traffic Injury Prevention, vol. 20, no. [Online]. Available: https://www.asam.net/standards/
sup1, pp. S65–S70, 2019. detail/openscenario/
[18] M. Gyllenhammar, R. Johansson, F. Warg, D. Chen, H.- [30] M. Althoff, M. Koschi, and S. Manzinger, “Common-
M. Heyn, M. Sanfridson, J. Söderberg, A. Thorsén, and Road: Composable benchmarks for motion planning on
S. Ursing, “Towards an operational design domain that roads,” in IEEE Intelligent Vehicles Symposium (IV), 6
supports the safety argumentation of an automated driv- 2017, pp. 719–726.
ing system,” in 10th European Congress on Embedded [31] W.-t. Tsai, A. Saimi, L. Yu, and R. Paul, “Scenario-
Real Time Systems (ERTS), 2020. based object-oriented testing framework,” in Third In-
[19] A. Ebner, T. Helmer, R. R. Samaha, and P. Scullion, ternational Conference on Quality Software, 2003, pp.
“Identifying and analyzing reference scenarios for the 410–417.
development and evaluation of active safety: Application [32] M. Utting, A. Pretschner, and B. Legeard, “A taxonomy
to preventive pedestrian safety,” International Journal of model-based testing approaches,” Software Testing,
of Intelligent Transportation Systems Research, vol. 9, Verification and Reliability, vol. 22, no. 5, pp. 297–312,
no. 3, pp. 128–138, 2011. 2012.
[20] W. Hulshof, I. Knight, A. Edwards, M. Avery, and [33] M. R. Zofka, S. Klemm, F. Kuhnt, T. Schamm, and J. M.
C. Grover, “Autonomous emergency braking test results,” Zöllner, “Testing and validating high level components
in 23rd International Technical Conference on the En- for automated driving: Simulation framework for traffic
hanced Safety of Vehicles (ESV), 2013, pp. 1–13. scenarios,” in IEEE Intelligent Vehicles Symposium (IV),
 15

2016, pp. 144–150. of scenario development: An overview of techniques,”

[34] D. Wittmann, M. Lienkamp, and C. Wang, “Method for Foresight, vol. 9, no. 1, pp. 5–25, 2007.
comprehensive and adaptive risk analysis for the develop- [50] SAE International, “Taxonomy and definitions for terms
ment of automated driving,” in IEEE 20th International related to driving automation systems for on-road motor
Conference on Intelligent Transportation Systems (ITSC), vehicles,” Tech. Rep. J3016, 6 2018.
2017, pp. 1–7. [51] K. Go and J. M. Carroll, “The blind men and the
[35] A. B. Frost, “MUSICC: Multi User Scenario Catalogue elephant: Views of scenario-based system design,” In-
for CAVs, glossary of terms and definitions,” Catapult teractions, vol. 11, no. 6, pp. 44–53, 2004.
Transport Systems, Tech. Rep., 2018. [52] (2020) OpenSCENARIO 2.0. [Online]. Available:
[36] Z. Saigol, A. Peters, M. Barton, and M. Taylor, “Regu- https://releases.asam.net/OpenSCENARIO/2.0-concepts/
lating and accelerating development of highly automated ASAM_OpenSCENARIO_2-0_Concept_Paper.html
and autonomous vehicles through simulation and mod- [53] H. Kahn, Thinking the Unthinkable. New York: Horizon
elling,” Catapult Transport Systems, Tech. Rep., 2018. Press, 1962.
[37] Special Interest Group (SIG) on SImulation and [54] J. Alcamo, Scenarios as Tools for International Envi-
Modeling (SIM). (2019) Modeling and simulation ronmental Assessment. European Environment Agency,
glossary. Accessed March 2019. [Online]. Available: 2001.
https://www.acm-sigsim-mskr.org/glossary.htm [55] M. Dupuis, M. Strobl, and H. Grezlikowski, “Opendrive
[38] H. Elrofai, D. Worm, and O. Op den Camp, “Sce- 2010 and beyond – status and future of the de facto
nario identification for validation of automated driving standard for the description of road networks,” in Driving
functions,” in Advanced Microsystems for Automotive Simulation Conference Europe, 2010, pp. 231–242.
Applications 2016. Springer, 2016, pp. 153–163. [56] R. Breu, U. Hinkel, C. Hofmann, C. Klein, B. Paech,
[39] E. de Gelder, J. Manders, C. Grappiolo, J.-P. B. Rumpe, and V. Thurner, “Towards a formalization of
Paardekooper, O. Op den Camp, and B. De Schutter, the unified modeling language,” in European Conference
“Real-world scenario mining for the assessment on Object-Oriented Programming, 1997, pp. 344–366.
of automated vehicles,” in IEEE International [57] P. E. Pfeiffer, Concepts of Probability Theory. Courier
Transportation Systems Conference (ITSC), 2020, Corporation, 2013.
pp. 1073–1080. [58] M. S. Branicky, V. S. Borkar, and S. K. Mitter, “A unified
[40] E. de Gelder, H. Elrofai, A. Khabbaz Saberi, O. Op den framework for hybrid control: Model and optimal con-
Camp, J.-P. Paardekooper, and B. De Schutter, “Risk trol theory,” IEEE Transactions on Automatic Control,
quantification for automated vehicles in driving scenar- vol. 43, no. 1, pp. 31–45, 1 1998.
ios,” In preparation, 2021. [59] B. De Schutter, “Optimal control of a class of linear hy-
[41] W. V. Siricharoen, “Ontology modeling and object mod- brid systems with saturation,” SIAM Journal on Control
eling in software engineering,” International Journal of and Optimization, vol. 39, no. 3, pp. 835–851, 2000.
Software Engineering and Its Applications, vol. 3, no. 1, [60] R. Batres, M. West, D. Leal, D. Price, K. Masaki,
pp. 43–59, 2009. Y. Shimada, T. Fuchino, and Y. Naka, “An upper on-
[42] R. E. Johnson and B. Foote, “Designing reusable tology based on ISO 15926,” Computers & Chemical
classes,” Journal of Object-Oriented Programming, Engineering, vol. 31, no. 5-6, pp. 519–534, 2007.
vol. 1, no. 2, pp. 22–35, 1988. [61] W. P. M. H. Heemels, K. H. Johansson, and P. Tabuada,
[43] P. Wegner, “Concepts and paradigms of object-oriented “An introduction to event-triggered and self-triggered
programming,” ACM SIGPLAN OOPS Messenger, vol. 1, control,” in 51st IEEE Conference on Decision and
no. 1, pp. 7–87, 1990. Control, 12 2012, pp. 3270–3285.
[44] C. Patridge, Business Objects: Re-Engineering for Re- [62] R. Alur and D. L. Dill, “A theory of timed automata,”
Use. The BORO Centre, 2005. Theoretical Computer Science, vol. 126, no. 2, pp. 183–
[45] E. Gamma, R. Helm, R. Johnson, and J. Vlissides, “De- 235, 1994.
sign patterns: Abstraction and reuse of object-oriented [63] P. Seiniger, A. Hellmann, O. Bartels, M. Wisch, and
design,” in European Conference on Object-Oriented J. Gail, “Test procedures and results for pedestrian AEB
Programming, 1993, pp. 406–431. systems,” in 24th International Technical Conference on
[46] A. Snyder, “Encapsulation and inheritance in object- the Enhanced Safety of Vehicles (ESV), 2015.
oriented programming languages,” in Object-Oriented [64] S. Childress, B. Nichols, B. Charlton, and S. Coe, “Using
Programming Systems, Languages and Applications, an activity-based model to explore the potential impacts
1986, pp. 38–45. of automated vehicles,” Transportation Research Record,
[47] B. Meyer, “Reusability: The case for object-oriented vol. 2493, no. 1, pp. 99–106, 2015.
design,” IEEE Software, vol. 4, no. 2, p. 50, 1987. [65] G. Bagschik, T. Menzel, and M. Maurer, “Ontology
[48] P. W. F. van Notten, J. Rotmans, M. B. A. Van Asselt, and based scene creation for the development of automated
D. S. Rothman, “An updated scenario typology,” Futures, vehicles,” in IEEE Intelligent Vehicles Symposium (IV),
vol. 35, no. 5, pp. 423–443, 2003. 2018, pp. 1813–1820.
[49] P. Bishop, A. Hines, and T. Collins, “The current state [66] W. G. Najm, J. D. Smith, and M. Yanagisawa, “Pre-crash
 16

scenario typology for crash avoidance research,” U.S. T. Tengeiji, “Harmonized pre-crash scenarios for reach-
Department of Transportation Research and Innovative ing global vision zero,” in 26th International Technical
Technology Administration, Tech. Rep. DOT HS 810 Conference on the Enhanced Safety of Vehicles (ESV),
767, 4 2007. no. 19-0110, 2019.
[67] F. Fahrenkrog, L. Wang, C. Roesener, J. Sauerbier, and [74] G. Smith, Tagging: People-Powered Metadata for the
S. Breunig, “Deliverable D7.3 Impact analysis for super- Social Web. New Riders Publishing, 2007.
vised automated driving applications,” AdaptIVe, Tech. [75] D. H. Craft, P. A. Caro, J. Pasqua, and D. C. Brotsky,
Rep., 2017. “Tagging data assets,” U.S. Patent US 6,704,739 B2, 3
[68] O. Op den Camp, A. Ranjbar, J. Uittenbogaard, E. Rosen, 9, 2004.
R. Fredriksson, and S. de Hair, “Overview of main [76] S. Molloy, T. Ramos, and S. Thakar, “Dynamic hier-
accident scenarios in car-to-cyclist accidents for use archical tagging system and method,” U.S. Patent US
in AEB-system test protocol,” in International Cycling 9,613,099 B2, 4 4, 2017.
Safety Conference, 2014. [77] S. Bonnin, T. H. Weisswange, F. Kummert, and
[69] J. Lenard, A. Badea-Romero, and R. Danton, “Typical J. Schmuedderich, “General behavior prediction by a
pedestrian accident scenarios for the development of combination of scenario-specific models,” IEEE Trans-
autonomous emergency braking test protocols,” Accident actions on Intelligent Transportation Systems, vol. 15,
Analysis & Prevention, vol. 73, pp. 73–80, 2014. no. 4, pp. 1478–1488, 8 2014.
[70] Transport Systems Catapult, “Taxonomy of scenarios for [78] J.-P. Paardekooper, S. Montfort, J. Manders, J. Goos,
automated driving,” Tech. Rep., 2017. E. de Gelder, O. Op den Camp, A. Bracquemond,
[71] F. Hauer, T. Schmidt, B. Holzmüller, and A. Pretschner, and G. Thiolon, “Automatic identification of critical
“Did we test all scenarios for automated and autonomous scenarios in a public dataset of 6000 km of public-road
driving systems?” in IEEE Intelligent Transportation driving,” in 26th International Technical Conference
Systems Conference (ITSC), 2019, pp. 2950–2955. on the Enhanced Safety of Vehicles (ESV), 2019, pp.
[72] E. de Gelder, J.-P. Paardekooper, O. Op den Camp, 1–8. [Online]. Available: https://www-esv.nhtsa.dot.gov/
and B. De Schutter, “Safety assessment of automated Proceedings/26/26ESV-000255.pdf
vehicles: How to determine whether we have collected tions, vol. 37, pp. 380–392, 2014.
enough field data?” Traffic Injury Prevention, vol. 20, [80] R. C. Dorf and R. H. Bishop, Modern Control Systems
no. S1, pp. 162–170, 2019. Twelfth Edition. Prentice Hall, 2011.
[73] A. Lara, J. Skvarce, H. Feifel, M. Wagner, and [81] S. N. Norman, Control Systems Engineering. John Wiley
[79] S. Al-Sultan, M. M. Al-Doori, A. H. Al-Bayatti, and & Sons, 2011.
H. Zedan, “A comprehensive survey on vehicular ad hoc
network,” Journal of Network and Computer Applica-