See discussions, stats, and author profiles for this publication at: [Link]

net/publication/228808836

FlowMonitor - a network monitoring framework for the Network Simulator 3

(NS-3)

Article · January 2009

DOI: 10.4108/ICST.VALUETOOLS2009.7493

CITATIONS READS

100 6,136

3 authors, including:

Pedro Fortuna Manuel Ricardo

Jscrambler University of Porto
8 PUBLICATIONS   136 CITATIONS    155 PUBLICATIONS   1,020 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

BLUECOM+ - Connecting Humans and Systems at Remote Ocean Areas using Cost-effective Broadband Communications View project

AppSeal View project

All content following this page was uploaded by Pedro Fortuna on 26 May 2014.

The user has requested enhancement of the downloaded file.

 FlowMonitor - a network monitoring framework for the
Network Simulator 3 (NS-3)

Gustavo Carneiro Pedro Fortuna Manuel Ricardo

INESC Porto INESC Porto INESC Porto
Faculdade de Engenharia Faculdade de Engenharia Faculdade de Engenharia
Universidade do Porto Universidade do Porto Universidade do Porto
gjc@[Link] mricardo@[Link]
[Link]@[Link]

The authors would like to thank the support received from Network monitoring may pose the following problems: 1) mon-
the Portuguese Fundação para a Ciência e Tecnologia un- itoring strategy—it can follow a passive approach, where the
der the fellowships SFRH/BD/23456/2005 and SFRH/BD/- traffic present in the network is measured, or an active ap-
22514/2005. proach, where test traffic is injected in the network for test-
ing purposes. In both cases the data collected can then be
ABSTRACT sent to a central point where a complete picture of the net-
When networking researchers meet the task of doing simu- work is computed; 2) monitoring points—not every network
lations, there is always a need to evaluate the value of such element can be easily monitored because nodes may lack a
models by measuring a set of well known network perfor- monitoring interface such as the Simple Network Monitoring
mance metrics. However, simulators in general and NS-3 in Protocol (SNMP), or nodes may not support the installa-
particular, require significant programming effort from the tion of additional software. Monitoring points also depend
researcher in order to collect those metrics. on the test objective which can be monitoring network links,
monitoring network queue dynamics, or monitoring the traf-
This paper reports a contribution for NS-3 consisting of a fic generated by specific server applications; 3) monitoring
new flow monitoring module that makes it easier to col- duration—it must be large enough to enable the gathering
lect and save to persistent storage a common set of network of statistically sound results, what implies that a relevant
performance metrics. The module automatically detects all number of events must be captured; this duration may be
flows passing through the network and stores in a file most difficult to define in the passive approach, since the rate of
of the metrics that a researcher might need to analyze about relevant events is, a priori, unknown; 4) synchronization—
the flow, such as bitrates, duration, delays, packet sizes, and we may be interested in monitoring a sequence of events
packet loss ratio. that might be difficult to synchronize in scenarios involving
several nodes; 5) transparency—because network monitor-
The value of this module is demonstrated using an easy to ing often uses the same resources to transmit regular traffic
follow example. It is also validated by comparing the mea- and monitoring control traffic, we may say that monitoring
surements of a simple scenario with the expected values. may affect the results.
Finally, the performance of flow monitoring is characterized
and shown to introduce small overheads. In network simulation environments, network monitoring is
used mostly for characterizing the performance of network
protocols. Monitoring in a simulation environment differs
1. INTRODUCTION from monitoring in real networks in a set of aspects: a)
Network monitoring is accomplished by inspecting the traf-
active monitoring is implicitly employed since the traffic in-
fic in a network; this technique can be employed for purposes
jected in a simulated network is controlled and defined sta-
such as: fault detection—to detect disruptions in the net-
tistically; b) installing probing functions to inspect the traf-
work connectivity, routing or services; performance evalua-
fic and queue dynamics is feasible and easy, and there is no
tion—to measure the network utilization or the performance
need to use monitoring protocols such as SNMP, since the
of network protocols; security monitoring—to detect possi-
results are easily gathered as data resides in the same pro-
ble security problems; and Service Level Agreements (SLA)
cess; c) monitoring is less intrusive because the monitoring
monitoring—to verify if a given network service is perform-
data needs not to traverse and use the network resources; d)
ing as specified in contract.
events are easily synchronized because network simulators
use the same simulation timer and scheduler; e) scenarios of
lots of nodes can be easily addressed.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are Network monitoring in simulated environments do present
not made or distributed for profit or commercial advantage and that copies some problems: the simulation models may not be accu-
bear this notice and the full citation on the first page. To copy otherwise, or rately designed and produce results that may diverge from
republish, to post on servers or to redistribute to lists, requires prior specific the actual protocols. Gathering of results requires the re-
permission and/or a fee.
NSTOOLS 2009, October 19, 2009 — Pisa, Italy. searcher to develop a great deal of code and possibly to
Copyright 2009 ICST 978–963–9799–70–7/00/0004 $5.00.
know and use different scripting languages. This may be ag- helper

gravated by the unwillingness of the researcher to dedicate Routing: Internet-stack Devices:

more effort to programming tasks, rather than focusing on olsr,global-routing (ipv4 impl.) csma | wifi | ...

the research. This may also lead to lower quality simulation node mobility
models. common simulator
core
This paper introduces the FlowMonitor, a network monitor-
ing framework for the Network Simulator 3 (NS-3) which can
be easily used to collect and store network performance data Figure 1: Overview or the main NS-3 modules
from a NS-3 simulation. The main goals behind the Flow-
Monitor development are to automate most of the tasks of
dealing with results gathering in a simple way, to be easily metrics, enabling the use of the framework to measure any
extended, and to be efficient in the consumption of mem- variable needed by the user. It is also possible to select spe-
ory and CPU resources. NS-3 is a state of the art Network cific flows as targets for monitoring instead of considering
Simulation tool that eventually will replace NS-2, a network all the flows. The collected data is used to calculate either
simulator used by many researchers. mean values or probability density functions. The statisti-
cal analysis tool, named analyzer, uses a configuration file
The remainder of this paper is organized as follows: Section which defines the minimum and maximum number of sim-
2 gives an overview of the related work; then, in Section ulation runs required and the specific metrics the user is
3 an introduction to NS-3 is presented; Section 4 details interested in analyzing. The number of simulation runs can
the FlowMonitor NS-3 module, including its requirements be defined by setting a desired confidence interval. The an-
and its architecture; Section 5 presents the validation and alyzer can be configured through command line or using a
results; Section 6 provides the conclusions and future work. GUI developed with C++/GTK+. The generic nature of
this framework provides enhanced flexibility, but at the cost
of simplicity. It requires the programmer to explicitly in-
2. RELATED WORK clude calls to Stat::put method in his code. Although the
There are a number of contributions that use trace files to framework includes code to automate calls for the most com-
gather information to produce network simulation statistics. mon types of metrics, it still can require some integration
Trace graph[8] is a NS-2 trace file analyzer based on Mat- effort in some cases. Its use is limited to NS-2. Porting the
lab that is able to process almost every type of NS2 trace framework to NS-3 is not trivial due to the many differences
file format; it produces a large amount of graphics that pro- between the two versions.
vide various views on the network such as throughput, delay,
jitter and packet losses. In[1], a tracing framework is pro-
posed, called XAV, which was built upon an XML Database 3. NS-3 INTRODUCTION
Management System (DBMS). It was introduced to avoid Network Simulator version 3, NS-3, is a new simulator that
storing redundant information, to speed up trace analysis is intended to eventually replace the aging NS-2 simulator.
and to simplify the access to trace files by post-processing NS-3 officially started around mid 2006, and the first sta-
tools. It requires simulators to produce XAV XML trace ble version was released in June 2008, containing models
format files. These files are imported into the XML DBMS for TCP/IP, WiFi, OLSR, CSMA (Ethernet), and point-to-
which provides an XQuery interface, an SQL-like querying point links, “GOD routing”, among others. Additional stable
mechanism. The authors compared the performance of XAV versions have been subsequently released, including Python
with an equivalent system using flat trace files and AWK for bindings, learning bridge, and real-time scheduler for version
trace parsing, and the results show that XAV is always faster 3.2 (Sep. 2008), emulation, ICMP, and IPv6 addresses in NS-
in situations where the user needs to extract non-multiple 3.3 (Dec. 2008), WiFi improvements, object naming system,
non-consecutive records. and “tap bridge” in NS-3.4 (Apr. 2009). Even though NS-2
still has a greater number of models included in the dis-
The use of trace files can be optimized to a certain extent, tribution, NS-3 has a good development momentum and is
but this approach usually requires a huge amount of disk believed to have a better core architecture, better suited to
space and the simulation performance is degraded by the receive community contributions. Core architecture features
extensive use of I/O operations. Even if sampling tech- such as a COM-like interface aggregation and query model,
niques are used, in large simulations the amount of data automatic memory management, callback objects, and real-
to store is still significant. Moreover, using samples intro- istic packets, make for a healthier environment in which to
duces some precision error. In order to avoid the overhead develop new complex simulation models. In addition, it is
of using trace files, statistics can be calculated during the reportedly[10] one of the better performing simulation tools
simulation runs. In [5], an experiment was conducted which available today.
concluded that using trace files can make simulations take
up to 6 or 7 times longer when compared to an approach NS-3 is organized as a set of modules, as shown in Fig. 1.
where statistics are gathered during the simulation. The The “core” module provides additional C++ syntactic sugar
authors proposed a framework for NS-2 that integrates a to make programming easier, such as smart pointers[6], rich
data collection module with a separated graphical statistic dynamic type system, COM-like[2] interface query system,
analysis tool that reads result files produced by the former. callback objects, tracing, runtime described object attributes,
The data collection module consists of a static C++ class among others.
named Stat. The Stat class provides a put method that can
be called anywhere in the NS-2 code. It can collect abstract One of the unusual characteristics about NS-3 when com-
pared to other network simulators is its tracing architec- do not use pointers (smart or otherwise) and wrap the ex-
ture. Generally, tracing is a facility provided by a simulator isting lower level classes with a more programmer-friendly
by which the user can discover which significant events are interface. In design pattern terminology[7], we might call
happening inside the simulation and under which conditions. this a façade.
Tracing will allow the researcher to derive important met-
rics of a simulation that can be used to quantify the value of 4. THE FLOWMONITOR NS-3 MODULE
a simulated model relative to another module. In NS-2, as 4.1 Requirements
in most simulators, tracing consists in generating a text file
When designing the flow monitoring framework, FlowMon-
describing a series of events, with associated time stamps
itor, a set of goals were taken into consideration, covering
and other properties, one event per line. Common events
aspects such as usability, performance goals, and flexibility.
that are recorded to such files include MAC layer transmis-
sion, reception, and queue packet drop, among others. In
First and foremost the monitoring framework should be as
NS-3, output of events to a text file is also provided, for a
easy to use as possible. Simulation is already very hard
small subset of the events known by NS-3. Another possible
work, and researchers need to focus more on their research
way for storing events, in particular packet transmit/receive
rather than spend time programming the simulator. The
events, is via a PCAP files1 .
flow monitoring framework must be easy to activate with
just a few lines of code. Ideally, the user should not have
However, these are just alternative tracing systems; the main
to configure any scenario-specific parameters, such as list of
tracing system provided by NS-3 is callback based tracing.
flows (e.g. via IP address and port src/dest tuples) that
In NS-3 callback based tracing, trace sources are defined by
will be simulated, since these are likely to change for nu-
NS-3 itself. Each possible trace source in NS-3 is associated
merous reasons, including varying simulation script input
with a specific object class and is identified by a name. The
parameters and random variable seed. The list of flows to
programmer may register a C++ function or method to be
measure should itself be detected by the flow monitor in run-
called when a certain (or a set of) trace source produces a
time, without programmer intervention, much like the exist-
new event. It is then the responsibility of the programmer to
ing “ascii” and “pcap” trace output functions do already in
know what to do in the callback. Common uses for tracing
NS-3.
include 1) writing raw event data to a file, 2) collect statistics
for the occurrence of the event so that only the mean or other
Another important concern is regarding the perfect amount
statistic moment is saved to a file, and 3) react to the event
of data to capture. Clearly, too little data can be risky for a
and change some parameter in the simulation in real time,
researcher. A complex series of simulations, including vary-
for instance to experiment with cross-layer[4] optimizations.
ing input parameters, and multiple runs for generating good
confidence intervals, can take between a few minutes to a
Other modules in NS-3 include “common”, containing data
few days to complete. It can be very frustrating to wait for
types related to the manipulation of packets and headers,
simulation results for days only to discover in the end that
and the “simulator” module, containing time manipulation
there was something that we forgot to measure and which is
primitives and the event scheduler. The “node” module sits
important, causing the researcher to code in the additional
conceptually above the previous modules and provides many
parameter measurement and wait a few more days for new
fundamental features in a network simulator, such as a Node
simulations to be run. Ideally, the simulation framework
class, an abstract base class for a layer-2 interface (NetDe-
should attempt to include a reasonably complete informa-
vice), several address types, including IPv4/6 and MAC-48
tion set, even though most of the information may not be
(EUI-48 in IEEE 802 terminology) address classes, and ab-
needed most of the time, as long as it does not consume too
stract base classes for a TCP/IP stack. The “mobility” mod-
much memory. A large data set is also useful because, in this
ule contains an abstract base class for mobility models. A
way, the researcher is able to run the same simulations once,
MobilityModel object may be aggregated with a Node object
but analyze the results multiple times using multiple views.
to provide the node with the ability to know its own posi-
The reverse is also true. We do not want to save too much
tion. Certain NetDevice implementations, such as WiFiNet-
information regarding flows. Too much information is diffi-
Device, need to know the physical position of a node in or-
cult to store and transmit, or can significantly increase the
der to calculate interference and attenuation values for each
memory footprint of the simulation process. For instance,
transmitted packet, while others, like PointToPoint, do not
it is preferable to use histograms whenever possible rather
need this information at all. Some common mobility mod-
than per-packet data, since per-packet data does not scale
els are included in NS-3, such as static, constant velocity,
well with the simulation time. Histograms, on the other
constant acceleration, random walk, random waypoint, and
hand, do not grow significantly with the number of packets,
random direction[3].
only with the number of flows, while still being very useful
for determining reasonably good approximations of many
NS-3 also contains a couple of routing models, “olsr” and
statistical parameters.
“global-routing”, a module “internet-stack” implementing a
UDP/TCP/IPv4 stack, and few NetDevice implementations,
It is also a goal of this framework that the produced data
including WiFi (infrastructure and adhoc 802.11), CSMA
can be easily processed in order to obtain the final results,
(Ethernet-like), and PointToPoint (very simple PPP-like link).
such as plots and high-level statistics. As mentioned earlier,
Finally, sitting above all these modules is a “helper” module.
researchers usually need to analyze the same data multiple
This module provides a set of very simple C++ classes that
times for the same simulations, which means that this data
1 should end up on a persistent storage medium eventually.
PCAP is a binary format for storing (usually live captured)
packets, used by programs such as wireshark and tcpdump. Prime candidates for storage are 1) binary files (e.g. HDF),
2) ASCII traces, 3) XML files, and 4) SQL database. It is not
completely clear which one of these storage mediums is the
best, since each one has its drawbacks. Binary files, for in-
stance, can store information efficiently and allow fast data
reading, but are difficult to programmatically read/write,
and difficult to extend to accommodate new information
once the format has been defined, jeopardizing future ex-
tensibility. ASCII traces (line-by-line textual representation
of data) are verbose (high formatting overhead), difficult to
extend, and potentially slow to read. XML files have excel-
lent extensibility traits, but are also verbose, slow to read
for large datasets, and requires everything to be read into
memory before any data filtering can be performed. SQL
databases, on the other hand, are very efficient reading and
filtering data without requiring much process memory, but
can also be difficult to manage (except file embedded ones,
like SQLite), difficult to extend with new information, and
more difficult than textual files to find out how to read the
data, since the data format can only be discovered by read- Figure 2: High level view of the FlowMonitor archi-
ing the documentation of the software that produced it or tecture
by using a database access GUI tool.

We have opted to, initially, support only XML for data stor- or IPv6. For instance, a researcher could be simulating an
age output, as well as provide access to in-memory data MPLS switching fabric, whose data plane encapsulates 802.3
structures. Since the FlowMonitor collects reasonably sum- frames, so we need extensibility at the packet acquisition
marized information, it is not expected that XML trees will level to accommodate different data planes. Alternatively,
be very large, and reading such XML trees into memory the concept of “flow” may differ from the usual five-tuple
is not a problem with today’s computing resources. XML (source-ip, dest-ip, protocol, source-port, dest-port). For in-
presents an important advantage over any other format, stance, someone may want to classify flows by a “flow label”
which is the large set of programming libraries for read- or DSCP IPv4 field. Therefore we also need an extensible
ing XML, for almost every programming language, and al- flow classification mechanism.
most any platform. Especially in scripting languages, such
as Python, reading XML is relatively straightforward, and Finally, there is the obvious, but ever so important, require-
requires no additional programming language, such as SQL, ment of low monitoring overhead. While it is true that some
is required knowledge by the researcher. However, this issue memory and computation overhead cannot be avoided when
is highly debatable, and so all the FlowMonitor data struc- monitoring flows, this overhead should be as low as can be
tures are made available for those who wish to serialize data reasonably expected.
into another storage format.

Support for Python based simulations is also one of the

4.2 Architecture Overview
The FlowMonitor module is organized into three groups of
main design goals. At the time of this writing, the NS-
classes, as shown in Fig. 2. The group of “core” classes com-
3 Python bindings lack support for connecting callbacks
prises the FlowMonitor class, FlowProbe, and FlowClassi-
to trace sources (Config::Connect, Object::TraceConnect
fier. The FlowMonitor class is responsible for coordinating
and related APIs). Although supporting trace source call-
efforts regarding probes, and collects end-to-end flow statis-
backs in Python is desired and planned, the main reason
tics. The FlowProbe class is responsible for listening for
for the lack of interest in implementing this feature stems
packet events in a specific point of the simulated space, re-
from the awareness that Python is a very slow scripting lan-
port those events to the global FlowMonitor, and collect
guage and that using Python for per-packet tracing opera-
its own flow statistics regarding only the packets that pass
tions would just massively slow down the simulations to the
through that probe. Finally, the class FlowClassifier pro-
point of not being practical. The reasons for this slowdown
vides a method to translate raw packet data into abstract
include the need to, on a per-call basis, acquire the Python
“flow identifier” and “packet identifier” parameters. These
GIL (Global Interpreter Lock), convert the C++ parame-
identifiers are unsigned 32-bit integers that uniquely iden-
ters into Python format, call the Python code, convert the
tify a flow and a packet within that flow, respectively, for
return values from Python into C++ format, and finally re-
the whole simulation, regardless of the point in which the
lease the GIL. In order to effectively collect data for Python
packet was captured. These abstract identifiers are used
based simulations we need a “configure and forget” approach,
in the communication between FlowProbe and FlowMoni-
wherein a C++ class is made responsible for the actual trac-
tor, and all collected statistics reference only those abstract
ing and reports back to Python just the final results, at the
identifiers in order to keep the core architecture generic and
end of the simulation.
not tied down to any particular flow capture method or clas-
sification system.
The FlowMonitor architecture is also designed with exten-
sibility in mind. One use case of simulation is to research
Another group of classes provides a “default” IPv4 flow moni-
next-generation networks, which may not even use IPv4,
toring implementation. The classes Ipv4FlowProbe and Ipv4-
 tics, consider a simple three-node network, A ↔ B ↔ C.
Consequently, we will have2 three probes, PA , PB , and PC .
When a packet is transmitted from A to C passing through
B, the probe PA will notice the packet and create a Flow-
Probe::FlowStats structure for the flow, storing a delay-
F romF irstP robeSum value of zero. Next, the probe PB
will detect the packet being forwarded, and will increment
the value of delayF romF irstP robeSum in its own Flow-
Stats structure by the transmission delay from A to B. Fi-
nally, the packet arrives at C and will be detected by PC ,
which then adds to its delayF romF irstP robeSum the de-
lay between A and C. In the end, we are able to extract
not only end-to-end mean delay but also partial delays that
the packet experiences along the path. This type of probe-
specific information can be very helpful in ascertaining what
part of the network is responsible for the majority of the de-
lay, for instance. Such level of detail is missing from the
FlowMonitor::FlowStats structure alone.

Figure 3: Data collected by the FlowMonitor What follows is a more detailed description of the individ-
ual attributes in the flow data structures. In FlowMoni-
tor::FlowStats, the following attributes can be found:
Classifier subclass the abstract core base classes FlowProbe
and FlowClassifier, respectively. Ipv4FlowClassifier classi-
timeFirstTxPacket Contains the absolute time when the
fies packets by looking at their IP and TCP/UDP headers.
first packet in the flow was transmitted, i.e. the time
From these packet headers, a tuple (source-ip, destination-
when the flow transmission starts;
ip, protocol, source-port, destination-port) is created, and
a unique flow identifier is assigned for each different tuple timeLastTxPacket Contains the absolute time when the
combination. For each node in the simulation, one instance last packet in the flow was transmitted, i.e. the time
of the class Ipv4FlowProbe is created to monitor that node. when the flow transmission ends;
Ipv4FlowProbe accomplishes this by connecting callbacks
to trace sources in the Ipv4L3Protocol interface of the node. timeFirstRxPacket Contains the absolute time when the
Some improvements were made to these trace sources in or- first packet in the flow was received by an end node,
der to support the flow monitoring framework. i.e. the time when the flow reception starts;
timeLastRxPacket Contains the absolute time when the
Finally, there is also a “helper” group consisting of the single last packet in the flow was received, i.e. the time when
class FlowMonitorHelper, which is modelled in the usual the flow reception ends;
fashion of existing NS-3 helper classes. This helper class
is designed to make the most common case of monitoring delaySum Contains the sum of all end-to-end delays for all
IPv4 flows for a set of nodes extremely simple. It takes care received packets of the flow;
of all the details of creating the single classifier, creating
one Ipv4FlowProbe per node, and creating the FlowMonitor jitterSum Contains the sum of all end-to-end delay jitter
instance. (delay variation) values for all received packets of the
flow. Here we define jitter of a packet as the delay
To summarize this highlevel architecture view, a single sim- variation relatively to the last packet of the stream, i.e.
ulation will typically contain one FlowMonitorHelper in- Jitter {PN } = |Delay {PN } − Delay {PN −1 }|. This
stance, one FlowMonitor, one Ipv4FlowClassifier, and sev- definition is in accordance with the Type-P-One-way-
eral Ipv4FlowProbes, one per Node. Probes capture packets, ipdv as defined in IETF RFC 3393;
then ask the classifier to assign identifiers to each packet, txBytes, txPackets Total number of transmitted bytes and
and report to the global FlowMonitor abstract flow events, packets, respectively, for the flow;
which are finally used for statistical data gathering.
rxBytes, rxPackets Total number of received bytes and
packets, respectively, for the flow;
4.3 Flow Data Structures
The main result of the flow monitoring process is the collec- lostPackets Total number of packets that are assumed to
tion of flow statistics. They are kept in memory data struc- be lost, i.e. those that were transmitted but have not
tures, and can be retrieved via simple “getter” methods. As been reportedly received or forwarded for a long time.
seen Fig. 3, there are two distinct flow statistics data struc- By default, packets missing for a period of over 10
tures, FlowMonitor::FlowStats and FlowProbe::FlowStats. The seconds are assumed to be lost, although this value
former contains complete end-to-end flow statistics, while can be easily configured in runtime;
the latter contains only a small subset of statistics and from 2
Note: in the abstract base architecture it is not implied
the point of view of each probe. that there is one probe per node; however, for the sake of
this example we will assume the IPv4 flow monitoring case,
In order to understand the main utility of FlowProbe statis- which does make such assumption.
timesForwarded Contains the number of times a packet
has been reportedly forwarded, summed for all packets M
X −1
in the flow; N= Hi
i=0
delayHistogram, jitterHistogram, packetSizeHistogram
Histogram versions for the delay, jitter, and packet M −1
sizes, respectively; 1 X
µ= Ci Hi
N i=0
packetsDropped, bytesDropped These attributes also track
the number of lost packets and bytes, but discrimi-
M −1
nates the losses by a reason code. This reason code is 1 X
usually an enumeration defined by the concrete Flow- s2 = (Ci − µ)2 Hi
N − 1 i=0
Probe class, and for each reason code there may be a
vector entry indexed by that code and whose value is
the number of packets or bytes lost due to this reason. In the above equations, M represents the number of bins,
For instance, in the Ipv4FlowProbe case the follow- Hi represents the count of bin i, and Ci the center value of
ing reasons are currently defined: DROP_NO_ROUTE (no bin i.
IPv4 route found for a packet), DROP_TTL_EXPIRE (a
packet was dropped due to an IPv4 TTL field decre- In FlowProbe::FlowStats some additional attributes can be
mented and reaching zero), and DROP_BAD_CHECKSUM found on a per-probe/flow basis:
(a packet had bad IPv4 header checksum and had to
be dropped).
delayFromFirstProbeSum Tracks the sum of all delays
the packet experienced since being reportedly trans-
Some interesting metrics can be derived from the above at-
mitted. The value is always relative to the time the
tributes. For instance:
value was initially detected by the first probe;
delaySum bytes, packets number of bytes and packets, respectively,
mean delay: delay = rxP ackets
that this probe has detected belonging to the flow.
jitterSum
mean jitter: jitter = rxP ackets−1 No distinction is made here between first transmission,
forwarding, and reception events;
txBytes
mean transmitted packet size (byte): Stx = txP ackets
rxBytes
bytesDropped, packetsDropped tracks bytes and pack-
mean received packet size (byte): Srx = rxP ackets ets lost qualified by reason code, similarly to the at-
mean transmitted bitrate (bit/s): tributes with the same name in FlowMonitor::FlowStats.

8 · txBytes
Btx =
timeLastT xP acket − timeF irstT xP acket
4.4 Example
To demonstrate the programming interfaces for using Flow-
mean received bitrate (bit/s): Monitor, we create a simple simulation scenario, illustrated
in Fig. 4, with a grid topology of 3 × 3 WiFi adhoc nodes
8 · rxBytes
Brx = running the OLSR protocol. The nodes transmit CBR UDP
timeLastRxP acket − timeF irstRxP acket flows with transmitted bitrate of 100 kbit/s (application
timesF orwarded data, excluding UPD/IP/MAC overheads), following a sim-
mean hop count: hopcount = 1 + rxP ackets ple node numbering strategy: for i in 0..8, node Ni transmits
packet loss ratio: q = lostP ackets to node N8−i .
rxP ackets+lostP ackets

It is out of scope of this paper to present the full example

Some of the metrics, such as delay, jitter, and packet size, program source code3 . Suffice to say that enabling the flow
are too important to be summarized just by the sum, count monitor is just the matter of replacing the line
and, indirectly, mean values. However, storing all individual [Link](), with something like this (using the
samples for those metrics does not scale well and is too ex- Python language):
pensive in terms of memory/storage. Therefore, histograms
are used instead, as a compromise solution that consumes f l o w m o n h e l p e r = ns3 . F l o w M o n i t o r H e l p e r ( )
monitor = flowmon helper . I n s t a l l A l l ( )
limited memory but is rich enough to allow computation of m o n i t o r . S e t A t t r i b u t e ( ”DelayBinWidth ” ,
reasonable approximations of important statistical proper- ns3 . DoubleValue ( 0 . 0 0 1 ) )
ties. In FlowMonitor, the class Histogram is used to imple- monitor . S e t A t t r i b u t e ( ”JitterBinWidth ” ,
ns3 . DoubleValue ( 0 . 0 0 1 ) )
ment histograms. It offers a single method to count new m o n i t o r . S e t A t t r i b u t e ( ”P a c k e t S i z e B i n W i d t h ” ,
samples, AddValue, a method to configure the bin width, ns3 . DoubleValue ( 2 0 ) )
SetBinWidth, and methods to retrieve the histogram data:
GetNBins, GetBinStart, GetBinEnd, GetBinWidth, Get- ns3 . S i m u l a t o r . Run ( )
BinCount. From this data, estimated values for N (number m o n i t o r . S e r i a l i z e T o X m l F i l e ( ” r e s u l t s . xml ” , True , True )
of samples), µ (mean), and s (standard error) can be easily
computed. From the equations found in [9] (Chapter 2), we 3
The complete source code is available online, at
can derive: [Link]
 Listing 1: Sample script to read and plot the results
e t = ElementTree . p a r s e ( s y s . a r g v [ 1 ] )
bitrates = [ ]
losses = []
delays = [ ]
f o r f l o w in e t . f i n d a l l ( ” F l o w S t a t s / Flow ” ) :
# f i l t e r o u t OLSR
f o r t p l in e t . f i n d a l l ( ” I p v 4 F l o w C l a s s i f i e r / Flow ” ) :
i f t p l . g e t ( ’ f l o w I d ’ ) == f l o w . g e t ( ’ f l o w I d ’ ) :
Figure 4: Example network topology break
i f t p l . g e t ( ” d e s t i n a t i o n P o r t ” ) == ’ 698 ’ :
continue
What the above code does is: l o s s e s . append ( i n t ( f l o w . g e t ( ’ l o s t P a c k e t s ’ ) ) )

rxPackets = i nt ( flow . get ( ’ rxPackets ’ ))

1. Create a new FlowMonitorHelper object; i f r x P a c k e t s == 0 :
b i t r a t e s . append ( 0 )
else :
2. Call the method InstallAll on this object. As a re- t0 = long ( flow . get ( ’ timeFirstRxPacket ’ ) [ : − 2 ] )
sult, the flow monitor will be created and configured t 1 = l o n g ( f l o w . g e t ( ’ t i m e L a s t R x Pa cket ’ ) [ : − 2 ] )
to monitor IPv4 in all the simulation nodes; d u r a t i o n = ( t 1 − t 0 ) ∗ 1 e−9
b i t r a t e s . append ( 8 ∗ l o n g ( f l o w . g e t ( ’ r x B y t e s ’ ) )
/ d u r a t i o n ∗ 1 e −3)
3. Configure some histogram attributes;
d e l a y s . append ( f l o a t ( f l o w . g e t ( ’ delaySum ’ ) [ : − 2 ] )
4. Run the simulation, as before, calling [Link](); ∗ 1 e−9 / r x P a c k e t s )

pylab . subplot (311)

5. Finally, write the flow monitored results to a XML file pylab . h i s t ( b i t r a t e s , b i n s =40)
named “[Link]”. The second parameter of Seri- pylab . x l a b e l ( ”Flow b i t r a t e ( b i t / s ) ” )
alizeToXmlFile indicates if we wish to also save the pylab . y l a b e l ( ”Number o f f l o w s ” )
histogram contents, and the third parameter indicates pylab . subplot (312)
if we wish to save per-probe flow statistics. pylab . h i s t ( l o s s e s , b i n s =40)
pylab . x l a b e l ( ”Number o f l o s t p a c k e t s ” )
pylab . y l a b e l ( ”Number o f f l o w s ” )
To present the results, one would have to write a program pylab . subplot (313)
that reads the data from XML and creates plots. For in- pylab . h i s t ( d e l a y s , b i n s =10)
stance, the Python program in List. 1 reads the XML file pylab . x l a b e l ( ”Delay ( s ) ” )
and plots the histograms of 1) received bitrates, 2) packet pylab . y l a b e l ( ”Number o f f l o w s ” )
losses, and 3) delays, for all flows. The program uses the pylab . s u b p l o t s a d j u s t ( hspace =0.4)
Python XML parsing module ElementTree for reading the pylab . s a v e f i g ( ” r e s u l t s . pdf ”)
XML file, and the matplotlib module for generating the plots.
The results in Fig. 5 are obtained from this and show that
most flows achieved an actual throughput of around 105
kbit/s, except for one flow that only transferred less than
86 kbit/s. Packet losses were generally low except for two 3.0
Number of flows

flows. Finally, mean delays vary between ≈ 20 ms and ≈ 70 2.5

ms. 2.0
1.5
1.0
0.5
5. VALIDATION AND RESULTS 0.085 90 95 100 105 110
For validation and obtaining results, we begin by describing Flow bitrate (bit/s)
7
Number of flows

a simple network scenario, which is then used for validation 6

purposes. Finally performance is evaluated using the same 5
4
scenario and varying the network size. 3
2
1
00 50 100 150 200
5.1 Test scenario Number of lost packets
Fig. 6 shows the network topology and flow configuration for 3.0
Number of flows

the test scenario. It consists of a number of rows of nodes, 2.5

2.0
each row containing a number of nodes, with each node con- 1.5
nected to the next one via a point-to-point link. From left 1.0
0.5
to right, a link is configured with 100 kbit/s, then the next 0.0
0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08
link is configured with 50 kbit/s, the next with 100 kbit/s Delay (s)
again, and so on. The configured delay is zero, and the drop
tail queue maximum size is 100 packets. The rows are not Figure 5: Example program results
vertically connected. The NS-3 GlobalRoutingManager is
used to compute the routing tables at the beginning of the
 be half the transmitted bitrate.

The validation results in Tab. 1 show that, for a scenario

with 2704 nodes (i.e. 1352 flows), the measured results (for
10 simulations) match the theoretical values within a very
small margin of error. The expected values for transmitted
bitrate is slightly less than 100 kbit/s due to the transla-
tion from layer-2 to layer-3, taking into account the fac-
512+20+8
tor 512+20+8+2 due to the PPP header not being present
where packets are monitored by the FlowMonitor. Same
reasoning applies to the received bitrate. The errors found
between estimated and measured values are negligible for
transmitted/received bitrates and delays, and are likely a
result of sampling issues and/or numeric errors. The error
Figure 6: Test scenario
in the packet loss ratio is slightly larger, but this error is
not of great significance. This error can be explained by
the method of measuring losses used by the FlowMonitor,
simulation. Flows are created at the beginning of the sim-
wherein a packet is considered lost if it has not been reported
ulation. Every other node sends one 100 kbit/s UDP flow4
by any probe to have been received or retransmitted for a
to the node that is two hops to the right, and sends another
certain period of time, i.e. only packets that are “missing in
flow to the node that is two hops to the left. Packet size is
action” are considered lost. Naturally, at the end of simula-
the default 512 byte UDP payload.
tion a number of packets are still in transit, some of which
could have been lost, but the potential packet losses are not
5.2 Validation accounted for, hence the error. In Sec. 6.1 we mention a
To validate the flow monitor, we begin by examining the test possible way to overcome this error, as future work.
scenario and deriving some theoretical flow metrics, which
are then compared to values obtained by the measurements 5.3 Performance Results
done with the help of the FlowMonitor module. To evaluate the overhead introduced by flow monitoring, we
ran a series of simulations, increasing the network size be-
The test topology is relatively simple to analyze. In the first tween 16 and 2704 nodes, and measuring the time taken to
link, between the first and second nodes, we have one 100 simulate each scenario, and memory consumption (virtual
kbit/s UDP flow in each direction. For each flow, the first memory size), 1) without collecting any results, 2) with flow
hop of the flow traverses a link whose capacity matches ex- monitoring, and 3) with ascii tracing to a file. We repeat
actly the flow link-layer bitrate. Thus, packet loss rate will each experiment 10 times with different random variable
be zero, and queueing delay will also be null. However, when seeds. The performance results in Fig. 7 show the addi-
the flow reaches the second hop it will have to be transmit- tional overhead, in terms of memory consumption and sim-
ted by a link that is half the capacity of the flow bitrate. ulation wall-clock time, that is incurred by enabling the flow
Consequently, after a few seconds the drop-tail queue will monitor or trace file. The top-left plot shows the total mem-
fill to maximum capacity, causing a queueing delay, and half ory consumed by a simulation while varying the number of
the packets will have to be dropped due to the bottleneck nodes. Three curves are shown: one represents the sim-
link. ulations without FlowMonitor or file tracing enabled, an-
other other one represents the same simulations but with
We can derive the estimated values for delays, losses, and FlowMonitor enabled, and the remaining curve represents
bitrates of each flow. We define S as the number of bits in the simulations with ascii file tracing. The top-right plot
each packet, S = (512 + 20 + 8 + 2) × 8 = 4336 bit, and shows an alternative view of the same information where in-
C1 the bitrate of the link layer of the 100 kbit/s link, C1 = stead the monitoring/tracing overhead is shown. The mon-
100000 bit/s. Then, the delay in the first hop, where there itoring overhead, in percentage, is defined by the formula
is no queueing, is simply the transmission delay, d1 = CS1 = 100 Mmonitor −Mbase
, where Mmonitor is the memory consump-
Mbase
0.04336 s. In steady state, the second hop drop-tail queue
tion with monitoring enabled, and Mbase the memory with-
will be filled, and so packets will experience a delay cor-
out monitoring. Idem for file tracing. The memory overhead
responding to transmitting 99 packets ahead in the queue,
of enabling the FlowMonitor was 23.12% for 2704 nodes, cor-
plus the packet itself, plus the packet that the PPP device is
responding to 45 MB of extra memory consumption, while
currently transmitting. Since the second hop has lower bi-
the overhead with ascii file tracing was 5.82%. The bottom
trate, C2 = 50000 bit/s, and so d2 = 101 × CS2 = 8.75872 s.
two plots show the impact in terms of simulation wallclock
Thus, the total end-to-end delay experienced by the flow will
time of the FlowMonitor and trace file, the left plot show-
be d1 + d2 = 8.80208 s. Regarding packet losses, each flow
ing the three curves separately, the right plot showing the
traverses two hops. As previously explained, packet drop
relative overhead (using a formula similar to the memory
probabilities will be 0 and 0.5 for the first and second hop,
overhead one). For the simulation time, which in case of
respectively. Thus, the end-to-end packet probability will be
FlowMonitor includes the time needed to serialize all data
0.5 for each flow. Consequently, the received bitrate should
to an XML file, the overhead reaches a peak of about 55%
4
Actually, at application layer the flow bitrates are for 500 nodes, but then gradually decreases with the net-
94.465 kbit/s, so that with UDP, IPv4, and MAC header work size until reaching 38.82% (about 90 seconds) for 2704
overhead the bitrate is exactly 100 kbit/s at link layer. nodes, while the overhead of ascii trace file generation is al-
 Metric Measured Value (95% C. I.) Expected Value Mean Error
Tx. bitrate 99646.06 ± 2.68 × 10−5 99631.00 +0.015 %
Rx. bitrate 49832.11 ± 7.83 × 10−5 49815.50 +0.033 %
Delays 8.8005 ± 8.8 × 10−9 8.80208 -0.018 %
Losses 0.4978 ± 1.5 × 10−6 0.5000 -0.44 %

Table 1: Validation results

80 30
75
Base Flow Monitor
Flow Monitor 25 Trace File
70 Trace File

Memory Overhead (%)

65 20
Memory (MiB)

60 15
55
10
50
45 5

400 200 400 600 800 1000 1200 00 200 400 600 800 1000 1200
Number of Nodes Number of Nodes
350 800
Base 700
300 Flow Monitor
250 Trace File 600

Time Overhead (%)

500
200 Flow Monitor
Time (s)

400
150 Trace file
300
100 200
50 100
00 200 400 600 800 1000 1200 00 200 400 600 800 1000 1200
Number of Nodes Number of Nodes

Figure 7: Performance results of the flow monitor

most always above 700%. In the course of all simulations, 6.1 Future Work
over 150 GiB of ascii trace files were generated. Although the FlowMonitor already covers the most impor-
tant functionality needed from such a system, there is room
It should be noted that these values are worst case scenario. for improvement. More data output methods, such as database
Since the simulation scenario is very simple (PointToPoint- and binary file, would of course be welcome. Another use-
NetDevice is just about the simplest and fastest NetDevice ful addition could be to add more options to better control
implementation in NS-3), the additional overhead appears what level of detail is stored in memory. For instance, we
relatively significant. More complex simulations, for ex- might not be interested in per-probe flow statistics, or in
ample WiFi and OLSR, should consume considerably more histograms. FlowMonitor support for multicast/broadcast
memory and CPU time, so that enabling the FlowMonitor flows is another features that could be useful for certain
on such simulations will add an overhead that will be smaller researchers. The FlowMonitor could benefit from a closer
compared to the baseline simulation. integration with NetDevices, e.g. so that it could directly
monitor packet drop events from each NetDevice’s transmis-
sion queue, as well as handle transmission errors from layer
6. CONCLUSIONS 2, instead of relying on the vaguer “packet missing in action”
In this paper we have described a solution that solves a com- measurement method. In some circumstances a researcher
mon problem for all researchers that need to conduct simu- might want to observe how flow metrics evolve over time,
lations in NS-3: how to easily extract flow metrics from ar- instead of just obtaining a summary of the results over the
bitrary simulations? Existing solutions, some of which have entire simulation. This would require saving a periodic snap-
been identified, do not solve this problem effectively, both shot of the flows metrics to a file. It can be done already by
for simulators in general but especially in NS-3. A set of gen- user code, but a convenience API for it would be interesting
eral requirements have been identified, and a new flow mon- to have in FlowMonitor. Finally, we would like to add con-
itoring solution was designed and implemented which meets venient methods to the Histogram class that would compute
those requirements. The simplicity of use of this new frame- the values N , µ, and s that were mentioned in Sec. 4.3.
work has been demonstrated via a very simple example. The
implementation was validated by comparing measured flow The FlowMonitor code for NS version 3.2 is available online
metrics with theoretical results. Performance results show at [Link] In the fu-
that flow monitoring introduces a relatively small overhead, ture, the authors plan to port this code to the latest version
even when used with a base simulation that is already very of NS-3 and propose it for inclusion in the main NS-3 source
efficient to begin with. code repository.
 7. REFERENCES
[1] R. Ben-El-Kezadri, F. Kamoun, and G. Pujolle. XAV:
a fast and flexible tracing framework for network
simulation. In Proceedings of the 11th international
symposium on Modeling, analysis and simulation of
wireless and mobile systems, pages 47–53, Vancouver,
British Columbia, Canada, 2008. ACM.
[2] D. Box. Essential COM. Addison-Wesley, 1998.
[3] T. Camp, J. Boleng, and V. Davies. A survey of
mobility models for ad hoc network research. Wireless
Communications and Mobile Computing,
2(5):483–502, 2002.
[4] G. Carneiro, J. Ruela, and M. Ricardo. Cross-layer
design in 4G wireless terminals. Wireless
Communications, IEEE [see also IEEE Personal
Communications], 11(2):7–13, 2004.
[5] C. Cicconetti, E. Mingozzi, and G. Stea. An
integrated framework for enabling effective data
collection and statistical analysis with ns-2. In
Proceeding from the 2006 workshop on ns-2: the IP
network simulator, page 11, Pisa, Italy, 2006. ACM.
[6] D. Edelson. Smart pointers: They’re smart, but they’re
not pointers. University of California, Santa Cruz,
Computer Research Laboratory, 1992.
[7] E. Gamma, R. Helm, R. Johnson, and J. Vlissides.
Design patterns: elements of reusable object-oriented
software. Addison-Wesley, 1995.
[8] J. Malek and K. Nowak. Trace graph-data
presentation system for network simulator ns. In
Proceedings of the Information Systems - Concepts,
Tools and Applications (ISAT 2003), Poland,
September 2003.
[9] S. W. Smith. The Scientist & Engineer’s Guide to
Digital Signal Processing. California Technical Pub.,
1st edition, 1997.
[10] E. Weingärtner, H. vom Lehn, and K. Wehrle. A
performance comparison of recent network simulators.
In Proceedings of the IEEE International Conference
on Communications 2009 (ICC 2009), Dresden,
Germany, IEEE., June 2009.

View publication stats