NetFlow Services

Benoit Claise
bclaise@[Link]

RIPE 44, Amsterdam

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 2
 Table of Content
• NetFlow Basics
• NetFlow
Version 5 (Router)
Version 7 (Switch)
Version 8 (Router)
Sampled (12000 Series)
• Advanced Concepts
• Troubleshooting
• New Features
• New Features, Version 9 and the IETF
• New Features, MPLS Aware NetFlow
• New Features, BGP Next Hop Aggregation
• Roadmap
• NetFlow FlowCollector
• Deployment Guide
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 3
 NetFlow Basics

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 4

 NetFlow Infrastructure
Ecosystem
Collector

Network Planning

RMON Probe
(Netscout)

Application RMON

NetFlow
FlowCollector:
NetFlow • Data Collection Accounting/Billing
Accounting: • Data Filtering
• Data Export • Data Aggregation
• Data • Data Storage
Aggregation • File System Management Network Data Analyzer

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 5

 NetFlow Possible Applications
NetFlow
Network Monitoring X

Network Planning X

Security Analysis X

Application Monitoring X

User Monitoring X

Traffic Engineering X

Peering Agreement X

Usage-based Billing X

Destination Sensitive Billing X

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 6

 What is a NetFlow Flow?

7 Keys define a flow

• Source Address
• Destination Address
• Source Port
• Destination Port
• Layer 3 Protocol Type
• TOS byte (DSCP)
• Input Logical Exported Data
Interface (ifIndex)

A flow is unidirectional
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 7
 How does it work?

NetFlow Cache

7 identifiers Other data

Flow identifiers Flow data update
Flow identifiers Flow data
Flow identifiers Flow data

Exported Data

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 8

 NetFlow Versions

• Version 5, the most complete version

• Version 7, on the switches
• Version 8, the Router Based Aggregation
• Version 9, the new flexible and extensible
version

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 9

 Data Export

Header
• Sequence number Flow Flow
• Record count Record … Record
• Version number

NetFlow Cache

• Expired flows are grouped together into “Netflow

Export” UDP datagrams for export to a collector
• UDP is used for speed and simplicity

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 10

 NetFlow Principles

• Capture traffic statistics per port, protocol, BGP

AS, network, …
• Support on most of the interface types
• Enable NetFlow on the main interface. But
returns the sub-interface in the flow record
(see new features)
• Supported on fast switching, Cisco Express
Forwarding (CEF) and Distributed CEF

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 11

 NetFlow Principles

• Not a switching path

• 7 flow identifiers
• Unidirectional traffic
• For ingress traffic only (*)
• IP unicast only (*)

(*) See roadmap

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 12

 NetFlow on the Router
Version 5

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 13

 Version 5

• Version 5 adds BGP AS

• Supported on router starting from 11.1 CA and
12.0
• The current version

• Note: No reason to use Netflow version 1 unless

supporting a legacy collection system.

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 14

 Version 5 Flow Format

• Packet Count • Source IP Address From/To

Usage • Byte Count • Destination IP Address

Time • Start sysUpTime • Source TCP/UDP Port

of Day • End sysUpTime • Destination TCP/UDP Port

Port • Input ifIndex Application

• Next Hop Address
Utilization • Output ifIndex • Source AS Number
• Dest. AS Number
Routing Routing
• Type of Service
QoS • Source Prefix Mask and
• TCP Flags • Dest. Prefix Mask Peering
• Protocol

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 15

 Version 5 Export

NetFlow Cache • Flow expired

• Cache full
Flow Entries
• Timer expired
Flow 1
Flow 2 Export V5 UDP
Record
Flow 3 To Collector

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 16

 Version 5 Configuration

router (config-if)#ip route-cache flow

router (config)#ip flow-export destination
[Link] 9996
router (config)#ip flow-export version 5 <peer-as |
origin-as>

Optional configuration
router (config)#ip flow-export source loopback 0
router (config)#ip flow-cache entries <1024-524288>
router (config)#ip flow-cache timeout …

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 17

 Version 5 Show Commands
martel#sh ip cache verbose flow
IP packet size distribution (94452 total packets):
1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480
.000 .199 .342 .300 .094 .028 .012 .005 .013 .000 .001 .000 .000 .000 .000

512 544 576 1024 1536 2048 2560 3072 3584 4096 4608
.000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000
IP Flow Switching Cache, 4456704 bytes
1 active, 65535 inactive, 25322 added
525430 ager polls, 0 flow alloc failures
last clearing of statistics never
Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec)
-------- Flows /Sec /Flow /Pkt /Sec /Flow /Flow
TCP-BGP 7 0.0 2 41 0.0 1.6 7.5
UDP-TFTP 1 0.0 1 67 0.0 0.0 15.1
UDP-other 19884 0.0 3 111 0.1 5.6 15.4
ICMP 5429 0.0 3 41 0.0 0.9 15.5
Total: 25321 0.0 3 97 0.2 4.6 15.4

SrcIf SrcIPaddress DstIf DstIPaddress Pr TOS Flgs Pkts

Port Msk AS Port Msk AS NextHop B/Pk Active
Se0/1 [Link] Se0/0 [Link] 11 00 10 5
B. Claise 00A1 © /24 193
2001, Cisco Systems, Inc. All rights reserved.
C628 /0 0 [Link] 84 39.7 18
 Origin Autonomous System
export
AS 1 AS 2 AS 3 AS 4 AS 5

Packet from AS1 to AS5

• ip flow-export version 5 origin-as

Source AS: AS1
Destination AS: AS5
• Important: the AS fields will stay empty with only
“ip flow-export version 5”

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 19

 Peer Autonomous System
export
AS 1 AS 2 AS 3 AS 4 AS 5

Packet from AS1 to AS5

• ip flow-export version 5 peer-as

Source AS: AS2
Destination AS: AS4
• Important: the AS fields will stay empty with only
“ip flow-export version 5”

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 20

 Asymetric BGP traffic Problem

AS 2 export
AS 1 AS 3 AS 4

AS 5
Packet from AS1 to AS4

Origin-as: AS1 and AS4 CORRECT

Peer-as: AS5 and AS4 WRONG
Because of the source IP address lookup
in the BGP table
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 21
 NetFlow on the Switches
Version 7

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 22

 NetFlow Version 7

• Support for Catalyst switches with a layer 3

board:
Catalyst 5000 with a RSM (Route Switch Module)
Catalyst 6000 with a MSFC (MultiLayer Switching
Feature Card)
• Version 7 uses MultiLayer Switching (MLS) or
CEF with a catalyst 6000 with SUP2
• For IP unicast only, not multicast, not IPX, even if
MLS can do all three
• MLS cache equals to the NetFlow cache.
Confusion in the documentation

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 23

 MLS Example

Candidate Packet
Vlan1

Supervisor
Enable Packet

MSFC
Vlan14

Layer 3 Switched
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 24
 MLS Example

Vlan1

Ping #1
Catalyst Ping #2
Ping #3
Ping #4
MSFC Ping #5
Vlan14

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 25

 MLS Concepts

• MLS is enabled for the whole device, not per

interface like on a router. So no concept of
incoming/outgoing traffic
• MLS is not for layer 2 traffic (see new features)
• MLS export the layer 3 information
• The MLS switching is done in hardware for the
catalyst (5000/6000). Which means that only the
export takes some CPU

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 26

 Version 7 Flow Format

• Packet Count • Source IP Address From/To

Usage
• Byte Count • Destination IP Address

Time • Start sysUpTime • Source TCP/UDP Port

of Day • End sysUpTime • Destination TCP/UDP Port

Port • Input ifIndex Application

• Next Hop Address
Utilization • Output ifIndex • Source AS Number
• Dest. AS Number Routing
• Type of Service • Source Subnet Mask
QoS and
• TCP Flags • Dest. Subnet Mask Peering
• Protocol • RouterSc (router
shortcut)

Added
Note that
from
some
version
of fields
5 are not populated
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 27
 Bad Design

MLS/NDE (not) enabled and export v5 from the MSFC

NFC + NFA
Vlan1

supervisor
export

MSFC
Only export the first Vlan14
packet of the flow
Unless don’t use MLS…
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 28
 Approximate Design

MLS/NDE enabled and export v7 from the SUP

NFC + NFA

Vlan1

export
supervisor

MSFC
Miss the accounting Vlan14
of the first packet of
the flow
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 29
 Better Design
MLS/NDE enabled and export v7 from SUP
export v5 from the MSFC

NFC + NFA
First packet exported from the MSFC
Vlan1

export

supervisor
export

MSFC
Vlan14

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 30

 Best Design
MLS/NDE enabled and export v7 from the SUP
export v5 from the MSFC
First packet exported from the MSFC
NFC + NFA Export in the sc0 vlan (sc0 in vlan1)
Vlan1

export

supervisor
export

Otherwise, MSFC
will account your Vlan14
exported traffic
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 31
 Best Design Problem

• The Collector doesn’t correlate the flows from

the same physical device
• The 2 different directories will be created
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 32
 Best Design Solution

# In case of V7, set

USE_SHORT_CUT_ADDRESS_AS_SOURCE_IP to
"yes" so that FlowCollector will use the
address of the router being short-cut as
the source of the corresponding flow.
Default is set to No
USE_SHORT_CUT_ADDRESS_AS_SOURCE_IP No

• Change the [Link] configuration file

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 33

 The Cat6000

• Hybrid mode (catOS/IOS) or native mode (full IOS)

• MLS is internal (no external MLS RP)
• SUP1 or SUP2, MSFC1 or MSFC2, PFC1 or PFC2
• In PFC1, uses MLS: a cache-based scheme
• In PFC2, uses HW CEF implementation, with a
FIB: PFC2 comes with MSFC2 and SUP2

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 34

 Cat6000 with a SUP2
• The PFC2 (on the SUP2) uses CEF, not MLS
anymore
• We still have the NetFlow for accounting only,
next to the Forwarding Information Base
• Cisco Express Forwarding (CEF) overview
CEF: No route cache, the router maintains a Forwarding
Information Base (FIB) which is a mirror of the routing
table
Uses Forwarding Information Base (FIB) for route
lookup and adjacency for encapsulation
FIB synchronisation between the MSFC and the
supervisor

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 35

 DCEF Example
FIB Synchronisation
No entry in the SUP FIB
Vlan1

Supervisor
Entry created in the
MSFC FIB

MSFC
Vlan14

All entries go through the SUP FIB

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 36
 Cat6000 with a SUP2, CEF mechanism

• Test of 5 inter vlans pings through a cat6000

• The dest. host has no adjacency in the FIB
• The first packet is sent to the MSFC for the ARP
request to be sent in the correct vlan.
This packet is not accounted by the SUP
• If NetFlow is enabled on the MSFC, this packet will
be accounted
• ARP reply arrives and updates MSFC FIB
• The MSFC FIB updates the SUP FIB
• The 4 next pings go through and are accounted by
the SUP version 7 export
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 37
 Cat6000 with a SUP2,
Export or Not on the MSFC?

• (-) Will account ONLY the first packet of a

destination, the one which will complete the
glean adjacency
• (-) The FIB entries remain the time of the
ARP entries. Not updated so often as the
MLS entries!

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 38

 Cat6000 with a SUP2,
Export or Not on the MSFC?

• (+) Will account the first packet of a destination,

the one which will complete the glean adjacency
• (+) Some features still use MLS
• (+) Some features will always go through the MSFC:
NAT, IP access-list with log, etc…
• Conclusion:
The export is needed for accounting accuracy
But less important as for MLS with a SUP1

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 39

 Caches – Cat6000

Export version 7 Export version 5

MLS NetFlow

SUP1 MSFC

Use MLS

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 40

 Caches – Cat6000 with SUP2/PFC2
Export version 7 Export version 5

FIB NetFlow FIB NetFlow

MSFC2
MLS
SUP2

Use CEF

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 41

 Cat6000, Native Mode
mls flow ip full -> flow mask
mls nde src_address [Link] version 7
-> version 7 export source OR
mls nde sender -> NDE enable + NDE from the PFC uses the
source configured from the MSFC!!!!!
interface vlan 1
ip address [Link] [Link]
ip route-cache flow
interface FastEthernet 3/2
ip address [Link] [Link]
ip route-cache flow

ip flow-export source vlan1 -> version 5 export source

ip flow-export version 5
ip flow-export destination [Link] 9996
-> both for version 5 and 7 export
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 42
 Cat6000, Native Mode

Cosmos#sh mls nde

Netflow Data Export enabled
Netflow Data Export configured for port 9996 on Host
[Link]
Source address: [Link], port: 50191
Version: 7
Include Filter not configured
Exclude Filter not configured
Total Netflow Data Export Packets are:
3 packets, 0 no packets, 23 records

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 43

 Cat6000, Native Mode

Cosmos#sh ip flow-export
exportFlow export is enabled
Exporting flows to [Link] (9996)
Exporting using source interface Vlan1
Version 5 flow records
317 flows exported in 218 udp datagrams
0 flows failed due to lack of export packet
60 export packets were sent up to process level
0 export packets were dropped due to no fib
0 export packets were dropped due to adjacency issues

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 44

 Format Comparison
Content V5 V7
Source IP address • zero in case of destination-only
Destination IP address • •
zero in case of destination-only or source-
Source TCP/UDP Port •
destination
Destination TCP/UDP zero in case of destination-only or source-
•
Port destination
Next Hop Router IP
• always zero New
address
Input Physical Interface
• It depends New
Index
Output Physical Interface
• It depends New
Index
Packet Count for this
• •
flow
Start of Flow
• •
Timestamps
End of Flow Timestamps • •

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 45

 Format Comparison
Content V5 V7
IP Protocol (TCP=6, zero in case of destination-only or
•
UDP=17) source-destination
switch sets it to the TOS of first packet
Type Of Service byte •
in flow
TCP flags • always zero

Source AS number • always sero New

Destination AS number • always zero

New
Source Subnet Mask • always zero

Destination Subnet Mask • always zero

Flags (indicate invalid
•
field within the flow)
Shortcut Router IP
•
address
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 46
 New Features

• SUP2/PFC2 (EARL6) supports from 12.1(13)E:

Source and Destination BGP AS
Input and Output ifIndexes
Next Hop

• Note: 12.1(13)E1 if any WAN cards

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 47

 NetFlow on the Router
Version 8

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 48

 Introduction

• Router Based Aggregation, i.e. version 8

• Enables router to summarize NetFlow data
• Reduces NetFlow Export data volume
• Decreases NetFlow Export bandwidth
requirements
• Making collection easier

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 49

 Introduction

• Supported from 12.0(3)T, 12.0(3)S and 12.1 On-

board aggregation, the router maintains extra
NetFlow cache(s), just for accounting.
• Still needs the main cache (version 5)
• When flows expire from the main cache, they are
added to each enabled aggregation cache
• Several aggregations can be enabled at the same
time

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 50

 Aggregations

• Currently 5 aggregations: ProtocolPort, AS,

SourcePrefix, DestinationPrefix, Prefix
• 6 extra aggregations available in IOS 12.0(15)S,
Targeted for 12.2(1)T, containing the TOS
• Requires the new NetFlow Collector 3.5 or above

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 51

 Version 8 - Flow Format
AS Protocol-Port Source-Prefix Destination-Prefix Prefix

Source Prefix • •
Source Prefix Mask • •
Destination Prefix • •
Destination Prefix Mask • •
Source App Port •
Destination App Port •
Input Interface • • •
Output Interface • • •
IP Protocol •
Source AS • • •
Destination AS • • •
First Timestamp • • • • •
Last Timestamp • • • • •
# of Flows • • • • •
# of Packets • • • • •
# of Bytes • • • • •

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 52

 Version 8 - Flow Format

AS- Protocol-Port- Source-Prefix- Destination- Prefix-TOS Prefix-Port

TOS TOS TOS Prefix-TOS

Source Prefix • • •
Source Prefix Mask • • •
Destination Prefix • • •
Destination Prefix Mask • • •
Source App Port • •
Destination App Port • •
Input Interface • • • • •
Output Interface • • • • •
IP Protocol • •
Source AS • • •
Destination AS • • •
TOS • • • • • •
First Timestamp • • • • • •
Last Timestamp • • • • • •
# of Flows • • • • • •
# of Packets • • • • • •
# of Bytes • • • • • •

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 53

 Version 8 Export
NetFlow Main Cache • Flow expired
• Cache full
v 5
Flow Entries • Timer expired
o rt a r y
Flow 1

x p
Export V5
es s
Flow 2
Flow 3
ERecordecTo Collector
UDP

t N
No
Aggreg. Cache

AS-Matrix • Cache full

• Timers expired
Prefix-Matrix
• Flow expired Export V8 UDP
• Cache full ... Record
• Timer expired To Collector
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 54
 Version 8 Configuration

router (config)# ip flow-aggregation cache as

router (config-flow-cache)# export destination
[Link] 9996
router (config-flow-cache)# enabled

router (config)# ip flow-aggregation cache protocol-port

router (config-flow-cache)# export destination
[Link] 9996
router (config-flow-cache)# cache entries 8192
router (config-flow-cache)# enabled

Note the 2 different export ip addresses/ports

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 55
 Version 8 Show Command

router#sh ip cache flow aggregation as

IP Flow Switching Cache, 278528 bytes 2 active, 4094
inactive, 13 added 216 ager polls, 0 flow alloc
failures
SrcIf SrcAS DstIf DstAS Flows Pkts B/Pk Active
Se0/0 0 Se0/2.1 0 1 1 104 0.0
Se0/0 0 Null 0 1 1 59 0.0

Note: you must choose peer-as or origin-as

router (config)# ip flow-export version 5 <peer-as
origin-as>
So that the main cache populates the BGP AS
So that the aggregation cache will contain the
populated BGP AS

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 56

 NetFlow on the 12000 Router
Sampled NetFlow

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 57

 12000 NetFlow Sampling

• Collects and exports NetFlow data for a sample

of the traffic passing through the router, instead
of the entire traffic
• Only for the 12000 router (GSR) so far
• Sampled NetFlow exports the same information
as full NetFlow
• The sampling interval is fixed and not an average
• Sampling advantages: CPU reduced and
possible reduced exported Data
• Sampling disadvantage: no billing possible?

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 58

 12000 NetFlow Sampling

Router(config)#ip flow-sampling-mode packet-interval

<10-16382>
Router(config-if)#ip route-cache flow sampled

Show Command
Router#show ip flow sampling
Flow sampling is enabled
'Packet Interval' sampling mode is configured.
1 out of every 100 packets is being sampled.

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 59

 Status of NetFlow on the 12000 Series

NetFlow Sampled NetFlow

v5 v8 v5 v8

Engine 0 12.0(14)S 12.0(6)S 12.0(14)S 12.0(11)S

Engine 2 PoS N/A N/A 12.0(14)S 12.0(14)S
3xGE N/A N/A 12.0(16)S 12.0(16)S
Engine 3 N/A 12.0(21)S 12.0(21)S 12.0(21)S
Engine 4 N/A N/A N/A N/A
Engine 4+ PoS N/A N/A 12.0(21)S 12.0(21)S

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 60

 Full NetFlow version 8
Engine 3 Line Cards

• No concept of main cache for full NetFlow version

8, the flows are directly created into the
aggregation cache(s)
• Full NetFlow version 8 could be the solution
versus Sampled NetFlow:
No main cache (the flow maintenance is the bottleneck)
Less flow in the aggregations cache
Export less flow
• Same behavior for the future engine 5 Line Cards

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 61

 Advanced Concepts

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 62

 Cache size
Platform Default Netflow Approximate amount
Cache Size of contiguous DRAM
(entries) used by Netflow cache
7x00, uBR7246,
64K 4MB
RSP7000
AS5800, 4x00, 3600,
4K 256KB
2600, 2500, 1600, 1400
VIP with 128MB DRAM 128K 8MB

VIP with 64MB DRAM 64K 4MB

VIP wit h 3 2 M B D R A M 32K 2MB

VIP with 16MB DRAM 2K 128K

Note that the latest IOS images don’t require contiguous DRAM
anymore
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 63
 12000 Line Card Cache size
Platform Default Netflow Approximate amount of
Cache Size (entries) contiguous DRAM used
by Netflow cache
LC with 1024MB DRAM 1M 64MB

LC with 512MB DRAM 512K 32MB

LC with 256MB DRAM 256K 16MB

LC with 128MB DRAM 128K 8MB

LC with 64MB DRAM 64K 4MB

LC with 32MB DRAM 32K 2MB

LC with 16MB DRAM 8K 512kB

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 64

 Version 5 VIP/LC caches

FIB NetFlow FIB NetFlow

RP VIP

FIB NetFlow

VIP2
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 65
 Version 8 VIP/LC Caches

FIB Main Agg. FIB Main Agg.

. .
RP . VIP .

FIB Main Agg.

.
VIP2 .
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 66
 VIP/LC Caches

• Nothing to configure on the VIP/LC (use DCEF)

• VIP: if-con <slot-number>
sh ip cache flow
• LC: attach <slot-number>
sh ip cache flow
Execute-on <slot-number> show …
• Own independent sequence numbering per
VIP/LC
• Note: Don’t export on the engine management
ethernet port on the 12000, even though it’s a
possible configuration
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 67
 Flow Ageing

• When is a flow expired?

Transport is completed (TCP FIN or RST)
After 15 sec of traffic inactivity (the only way for UDP).
The inactive timer
After 30 min of traffic activity. The active timer.
The cache is becoming full

Note that 15sec/30min are the router default timers

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 68

 Active/Inactive Timers
D: Data (UDP)
AT: Active Timer for the flow
IT: Inactive Timer for the flow
DDDDDDDDDD
DDDDDDDDDDDDDDD
DDDDD
AT1 starts

AT1 expires -> export

AT2 starts

AT2 stops
IT1 starts

IT1 expires -> export

Time

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 69

 Various Time in NetFlow

Flow end sysUpTime

Flow start sysUpTime

Router sysUpTime in header
UTC time in header
Time
1970 Flow exported
Flow ends
Router boots Deduced
B. Claise © 2001, Cisco Systems, Inc. All rights reserved.
Flow starts 70
 Various Time in NetFlow

• The UTC depends on the clock

• Synchronization of the VIP clock, the line card
clock (in sync. since 12.0) and the RSM/MSFC
clock
• Attention to the timezone on the collector
• Conclusion: the device clocks must be
synchronized
• NTP is a solution, NTP MIB in 12.1(4)

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 71

 NetFlow Bypasses the Access-list
ACL acceleration
Y N
First packet
in flow?

Lookup entry in Netflow cache

Y N
Pass the
ACL?
Y Output i/f N
Create an Netflow
entry with is null?
Create an output i/f null
Netflow entry
Update the
Discard the Update the Netflow entry stats
Forward the packet Netflow entry stats
packet with CEF Forward the packet
Go through the ACL with CEF
Maybe deny packet
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 72
 NetFlow and DOS attack

Sh ip cache verbose <server ip address> flow

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 73

 Performance (Approximate Number)

• Enabling NetFlow version 5 AND exporting

increases the cpu utilization by around 15 %
(with a max of 20 % depending on the platform)
• Enabling Neflow version 8 increases the cpu
utilization by 2 to 5%, depending on the number
of aggregations enabled
With a multiple of 6% for multiple aggregations
• NetFlow is done in hardware on the cat6000
supervisor and the 12000 Engine 3 Line Cards

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 74

 NetFlow Performance testing:
Results at a Glance

CPU impact:
10,000 active flows: < 4% of additional CPU utilization
45,000 active flows: <12% of additional CPU utilization
65,000 active flows: <16% of additional CPU utilization

NetFlow Data Export (single/dual): no real impact

NetFlow v5 vs. v8: minimal to no impact at all

NetFlow Feature Acceleration: >200 lines of ACLs

Sampled NetFlow on the Cisco 12000:

23 % vs 3 % (65,000 flows, 1:100)
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 75
 Troubleshooting

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 76

 Missing Flows?

2. NetFlow Collector
(NFC) Problem

3. Transfer Problem

Export

1. Router Problem
- Cache
- Export

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 77

 Missing Flows?
- 1. Router Problem

Router#sh ip cache flow (excerpt)

IP Flow Switching Cache, 4456704 bytes
2 active, 65534 inactive, 226352 added
3792086 ager polls, 0 flow alloc failures
Active flows timeout in 40 minutes
Inactive flows timeout in 20 seconds
82038 flows exported in 34439 udp datagrams, 0 failed
last clearing of statistics [Link]

Alloc failures: Number of times the NetFlow code tried to allocate

a flow but could not
Failed: Number of flows that could not be exported by the router
because of output interface limitations

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 78

 Missing Flows?
- 1. Router Problem

Router#sh ip flow export

Flow export is enabled
Exporting flows to [Link] (9996)
Exporting using source interface Loopback0
Version 5 flow records, origin-as
2304658131 flows exported in 219987515 udp datagrams
0 flows failed due to lack of export packet
167 export packets were sent up to process level
0 export packets were punted to the RP
3490 export packets were dropped due to no fib
7012 export packets were dropped due to adjacency issues
0 export packets were dropped enqueuing for the RP d
0 export packets were dropped due to IPC rate limiting
0 export packets were dropped due to output drops

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 79

 Missing Flows?
- 2. NFC Problem

• The Netflow Collector “show tech-

support”

udpPort: 9996, receivedFlows: 80277(0),

receivedFlowrecords: 1771469(0)
discardedFlows: 0, missedFlowrecords:
1115(0), socNum: 13, rcvQSize: 26000

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 80

 Missing Flows?
- 2. NFC Problem

• Netstat -s

udpInDatagrams = 14034 udpInErrors = 0

udpInCksumErrs = 0 udpInOverflows =3218

• In Netflow Collector, the number of missed

records is directly proportional to the number of
rules and the order of rules.

F Filter deny-traffic-x
Deny Srcaddr [Link] [Link]

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 81

 Missing Flows?
- 3. Transfer Problem

• The only remaining explanation

• Don’t forget that the NetFlow exported data are
transported over UDP
• Evaluate the exported traffic

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 82

 Exported Traffic Estimation

• Rule of thumb:
Export 1 % to 1.5% of the total box throughput
• To be more accurate, you need:
packet/sec of throughput (router figures, sh int
switching)
Ex: 150kpps average throughput on a 7500
average number of packets per flow
(sh ip cache flow)
Ex: 20 (a number recently quoted for
Internet backbone traffic)

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 83

 Exported Traffic Estimation

• Example for a 7500:

150kpps / 20 ppflow = 7500 flow / sec
Considering 30 flows per exported packet and
a length of 1500 bytes
7500 /30 *1500 = 375 Kbytes/sec of flow export
traffic from one router

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 84

 Flows/Packet

Number of flow Packet length

in a packet (bytes)

V1 24 Approx. 1200
V5 30 Approx. 1500
V7 28 Approx. 1500
V8 AsMatrix 51 1456
V8 ProtocolPortMatrix 51 1456
V8 SourcePrefixMatrix 44 1436
V8 DestinationPrefixMatrix 44 1436
V8 PrefixMatrix 35 1428

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 85

 New Features

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 86

 ifIndex Persistence

• No guarantee that the ifIndex values for

any “interface” will remain the same after
a reboot.
• The NetFlow exports contain the
input/output interfaces ifIndex
• Introduced in 12.0(11)S, 12.0(11)SC and
12.1(5)T

router(conf) snmp-server ifindex persist

router(conf-if) snmp-server ifindex persist

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 87

 New
NetFlow on Egress for MPLS Traffic
• Introduced in 12.0(10)ST, 12.1(5)T, 12.0(22)S
• For MPLS/VPN traffic only, i.e. the traffic coming
from the core
• Caches traffic on the egress interface, not the
ingress interface.
• Valid for version 5 and version 8

router(config-if)#tag-switching ip flow egress

• Can be enabled on subinterface

• All other NetFlow commands still apply
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 88
 NetFlow on Egress for MPLS Traffic
VPN_A
VPN_A
CE CE
VPN_B
VPN_A
PE P P PE CE
CE
VPN_A
CE P P
VPN_B
PE CE
PE
VPN_B
CE

• Now: enable egress/ingress on one PE

• Can deduce the packets lost in the core
• No accounting if both src and dst VPNs are
part of the same PE

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 89

 Minimum Prefix Mask for
New
Router-Based Aggregation

AS Protocol-Port Source-Prefix Destination-Prefix Prefix

Source Prefix • •
Source Prefix Mask • •
Destination Prefix • •
Destination Prefix Mask • •

• Prefixes come from the routing table

• Introduced in 12.0(11)S, 12.1(2)T
• Only for the Aggregations:
SourcePrefix, DestinationPrefix and Prefix

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 90

 Minimum Prefix Mask for
Router-Based Aggregation

export [Link]/16
R1

[Link]/8 [Link]/16

• Summarization on the router R1

• Lose the granularity unless we specify the
minimum mask of 16

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 91

 Minimum Prefix Mask for
Router-Based Aggregation

• Configuration:

router (config)# ip flow-aggregation cache prefix

router (config-flow-cache)# mask source minimum 24
router (config-flow-cache)# mask destination minimum 16

• SourcePrefix: only source

• DestinationPrefix: only destination

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 92

 New
Dual Flow Export

• Inserted into 12.2(2)T, 12.0(19)S and 12.0(19)ST,

2 redundant export destinations are allowed for
version 5

router(config)#ip flow-export destination [Link] 9996

router(config)#ip flow-export destination [Link] 9997

If try to configure more, you will get:

“Exceeded maximum export destinations”
• Only for the routers, not the catalysts for now

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 93

 New
Cat6000 Aggregations – Version 8

• Add 3 new aggregation schemes:

RouterDestOnly, RouterSrcDst, RouterFullFlow
• Hybrid version since CatOS version 5.5(2)
Not on Native version yet
• Must select the nde version 8 instead of 7
• Require the NetFlow Collector 3.6 or above
• No real aggregations (like version 8 on routers)
Because still IP addresses and no networks
The aggregation is defined by the flow mask

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 94

 Cat6000 Aggregations – Version 8

RouterDstOnly RouterSrcDst RouterFullFlow

Source IP address • •
Destination IP address • • •
Source App Port •
Destination App Port •
IP Protocol •
First Timestamp • • •
Last Timestamp • • •
# of Flows • • •
# of Packets • • •
# of Bytes • • •

No real aggregation like on a router, where we aggregate

IP addresses in prefixes

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 95

 New
Cat6x00 Switched Traffic

• The switched type traffic (intra vlan) is

now accounted with NetFlow
• Since CatOS version 7.(2)
Not on Native version yet

“set mls bridged-flow-statistics enable/disable

<vlan>“

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 96

 New
Cat6x00 New Fields Population

• SUP2/PFC2 (EARL6) supports from 12.1(13)E:

Source and Destination BGP AS
Input and Output ifIndexes
Next Hop

• Note: 12.1(13)E1 if any WAN cards

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 97

 New
Cat6x00 NetFlow Version 5 Support

• SUP2/PFC2 supports NetFlow version 5

from 12.1(13)E
• Some consistency…

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 98

 New
NetFlow on Subinterface

• Introduced in 12.0(21)S
• Under investigation for the 12000

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 99

 New
Egress Sampled NetFlow

• Egress Sampled NetFlow on engine 3

• IP->IP and MPLS->IP cases
• Available 12.0(24)S, for the 12000

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 100

 New

New Features
NetFlow Version 9 and IETF

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 101

 NetFlow Version 9
Why do we need a New Version?

• Fixed formats for export

Easy to implement
Consume little bandwidth
Easy to decipher at the collector
• But
Not flexible and not extensible
• Consequence
Always new aggregations for new combinations of
fields and for new technologies required
New collector versions required each time

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 102

 Version 9 Approach

• Current NetFlow versions are not flexible and not

extensible
• Version 9 based on template and separate flow
record
Template composed of type and length
Flow record composed of template ID
and value
• Whitepaper
[Link]

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 103

 NetFlow Version 9
Packet
1.1.1.
Template 1 Data
Packet 20 Option
FlowSet FlowSet FlowSet
Header

Template Definition (Template FlowSet)

Template
20
ID = 0 Length
Definition

Flow Records (Data FlowSet) Record

Field #1
Record
20 Record Record
Tpl ID Length
Field #2

Field #3
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 104
 NetFlow Version 9
Various Type of Export Packets

1.1.1.
Template 1 Data
Packet 20 Option
Header
FlowSet FlowSet FlowSet
…
1.1.1.
Template 1 Data
Packet 20 Template Data
Header FlowSet FlowSet
… FlowSet FlowSet

1.1.1.
Data 1 Data
Packet 20
Header FlowSet
… FlowSet
…
1.1.1.
Template 1 Template
Packet 20
Header FlowSet
… FlowSet
…
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 105
 Version 9
Example for Template Definition
Template A Template B
Flow Set ID (0 for Template) Flow Set ID (0 for Template)
Length of Template Length of Template
Structure Structure
1001 1002
(Template ID) (Template ID)
3 4
(# of Fields) (# of Fields)
SRC_AS_NUMBER SRC_IP_PREFIX
2 4
DST_AS_NUMBER SRC_AS_NUMBER
2 2
L4_PROTOCOL PACKET_COUNT
2 2
BYTE_COUNT
2
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 106
 Example for Export Packet
As Defined in Same as Template ID
the Previous for Template B; Refer to
Slide Previous Slide
[Link] [Link]
T T
e e
1002 20 64 1001 35
m m
Packet p 2 (# of p 1
Header l records) l
a a
t t 700
365 20
e e 6025
B A

92894 1000
23

Record 1 Record 2

Data for Template B Data for Template A

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 107
 NetFlow version 9 Principles

• Still a push model

• Sent the template regularly (configurable)
• Independent of the underlying protocol,
ready for any reliable protocol (thinking of
SCTP)
• FlowSet Flexibility in the export packet

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 108

 NetFlow version 9 Support

• Out in 12.0(24)S
• Committed for 12.3T

• Cafeteria based aggregation on the router

is not yet available

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 109

 IETF: IP Flow Information Export WG
(IPFIX)

• Internet Protocol Flow Information eXport (IPFIX)

is an effort to standardize flow export
• IPFIX web site for the charter, email archive,
drafts, etc. [Link]
• Cisco’s NetFlow version 9 has been presented a
the first BOF
• Cisco actively participating, authors of the 3
current drafts

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 110

 IPFIX Working Group at IETF

• Requirements draft:
[Link]
[Link]
• Architecture draft:
[Link]
[Link]
• Data Model draft:
[Link]
[Link]

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 111

 Version 9 and IPFIX

• Cisco NetFlow Version 9 draft:

[Link]
[Link]
Next version will become an I-RFC
• “Intellectual Property Rights” Notice on
the IETF web site because there is a patent
for NetFlow

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 112

 IPFIX Next Steps

• The requirement draft will go “last call” pretty

soon
• An evaluation team is created:
– Evaluation existing protocols: NetFlow, CRANE,
LFAP, Diameter, IPDR
– Choose THE base protocol
– Determine which improvements are needed for THE
protocol compared to the requirements
• Hopefully, NetFlow will be chosen

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 113

 NetFlow and the IPFIX Evaluation

• [Link]
• NetFlow compliant to most of the points
• Biggest exceptions:
MUST run on the top of a congestion aware
export protocol
MUST have authenticity, integrity, SHOULD
have confidentiality

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 114

 New

New Features
MPLS aware NetFlow Solution

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 115

 MPLS aware NetFlow
Description

• Provides flow statistics per MPLS and IP packets

MPLS packets:
Labels information
And the V5 fields of the underlying IP packet
IP packets:
Regular IP NetFlow records
• Based on the NetFlow version 9 export
• Configure on ingress interface
• Supported on sampled/non sampled NetFlow

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 116

 NetFlow MPLS Aware
Support

• Supported in 12.0(24)S, then 12.2S and maybe

12.2T
Support on the 12000: Engine 0, 1, 2, 3 and 4+

• Will be supported on 12.0(26)S on the 7200/7500

• The catalyst 6000 will only support the export of
the top label, due to hardware limitations

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 117

 NetFlow MPLS Aware
Flow Keys
• Key Fields (Uniquely • Additional Export Fields
Identifies the flow)
Flows
Source IP address
Packets
Destination IP address
Bytes
IP Protocol
First SysUptime
Input ifIndex
Last SysUptime
Source Application Port
Output interface
Destination Application Port
NetFlow version 5 fields of
DSCP the underlying IP packet
Up to 3 incoming MPLS Type of the top label:
labels of interest with LDP, BGP, VPN, ATOM, TE
experimental bits and end-of- Tunnel MID-PT, unknow
stack bit
The Forwarding Equivalent
Positions of the above labels Class mapping to the top
in the packet label stack label
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 118
 NetFlow MPLS Aware
What is exported?

• Export up to 3 incoming MPLS labels

• Experimental bits and end-of-stack bit
• Positions of the above labels in the label stack
• Type of the top label:
LDP, BGP, VPN, ATOM, TE Tunnel MID-PT,
unknown
• The Forwarding Equivalent Class mapping to the
top label, i. e. the IP address of the IBGP peer in
a MPLS (VPN) environment

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 119

 NetFlow MPLS Aware
What is exported?

• Underlying IP packet: will export the NetFlow V5

fields of the underlying IP packet, when
available:
Src and Dst AS, subnet masks and IGP next hop are
not available! Null will be exported
• Underlying non-IP packet: will export the
NetFlow V5 fields:
Src and Dst IP addresses, protocol, TOS, application
ports and TCP flags will be set to Null!

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 120

 NetFlow MPLS Aware
Configuration

router (config)# ip flow export version 9

router (config)# ip flow-export template options sampling
router (config)# ip flow-export template options export_stats
router (config)# ip flow-export template options timeout 5
router (config)# ip flow-export template refresh-rate 10
router (config)# ip flow-sampling-mode packet-interval 101

router (config)# ip flow-cache mpls label-positions [1] [2] [3]

router (config-if)# ip route-cache flow sampled

Label position is starting from the top label,

1 corresponds to the top of the stack

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 121

 NetFlow MPLS Aware
Show commands

LC-Slot# show ip cache verbose flow

...
SrcIf SrcIPaddress DstIf DstIPaddress Pr TOS Flgs Pkts
Port Msk AS Port Msk AS NextHop B/Pk Active

PO1/0 [Link] PO4/0:1 [Link] 06 00 00 24K

0100 /0 0 0200 /0 0 [Link] 256 34.6
Pos:Lbl-Exp-S 1:12305-6-0 (LDP/[Link]) 2:12312-6-

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 122

 NetFlow MPLS Aware
Typical Example

AS1 AS2 AS3 AS4 AS5

C AR AR C
u CR
CR u
s WR s
t AR AR t
o o
m MPLS m
AR CR CR AR
e e
r r
PoP PoP
s s

Server Farm 1 Server Farm 2

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 123
 NetFlow MPLS Aware
Typical Example

AS1 AS2 AS3 AS4 AS5

C AR AR C
u CR
CR u
s WR s
t AR AR t
o o
m MPLS m
AR CR CR AR
e e
r r
PoP PoP
s s

Another
Server Farm 1 Server Farm 2 solution for CsC
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 124
 New

New Features
BGP Next Hop TOS aggregation

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 125

 NetFlow BGP Next Hop TOS Aggregation

• New NetFlow aggregation on the Router

• Only for the BGP routes
• For IP packets (not MPLS)
• Also available under the VRF interface
• Configure on ingress interface
• Take the BGP Next Hop from the “via” fields in
“sh ip cef <destination_IP_address>”

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 126

 NetFlow BGP Next Hop TOS Aggregation
Support

• Currently on EFT
Currently EFT, since September
• GSR will follow later:
BGP next hop in 12.0(26)S
• Available on a wide range of platforms
Initially 7200 & 7500 then 1720, 2600, 3600, 4500, 4700,
5800, RSP 7000, RSM (Cat5000), 7200, 7500, MGX
Router Processor Module (RPM), 8800, GSR

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 127

 NetFlow BGP Next Hop TOS Aggregation
Flow Keys

• Key Fields (Uniquely

Identifies the flow) • Additional Export Fields

Origin AS Flows

Destination AS Packets

Inbound Interface Bytes

DSCP (*) First SysUptime

Next BGP Hop Last SysUptime

Output Interface

(*) before any recoloring

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 128

 Core Capacity Planning

• The ability to offer SLAs is dependent

upon ensuring that core network
bandwidth is adequately provisioned
• Adequate provisioning (without gross
over provisioning) is dependent upon
accurate core capacity planning

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 129

 Core Capacity Planning
What input?

• Accurate core capacity planning is

dependent upon understanding the core
traffic matrix and flows and mapping these
to the underlying topology

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 130

 We need the Internal Traffic Matrix

AS1 AS2 AS3 AS4 AS5

C AR AR C
u CR
CR u
s s
AR AR
t t
o o
m AR CR CR AR m
e e
r PoP PoP
r
s s
Server Farm 1 Server Farm 2

• “PoP to PoP”, the POP being the AR or CR

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 131
 The External Traffic Matrix is a plus

AS1 AS2 AS3 AS4 AS5

C AR AR C
u CR
CR u
s s
AR AR
t t
o o
m AR CR CR AR m
e e
r PoP PoP
r
s s
Server Farm 1 Server Farm 2
• From “PoP to BGP AS”, the POP being the AR or CR
• The external traffic matrix can influence the internal one
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 132
 NetFlow BGP Next Hop TOS Aggregation
Issue

• Only for IP packets (IP to IP or IP to MPLS)

Example: If a MPLS core starting from the AR, Will
generate flow records from all the AR
Note: if want to/must enable on the CR,
investigate MPLS aware NetFlow

• For non BGP routes, the BGP Next Hop will be

set to [Link]
In other words, no traffic matrix for non BGP routes

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 133

 NetFlow BGP Next Hop TOS Aggregation
Configuration

Router(config)#ip flow-export version 9 [origin-as | peer-as]

[bgp-nexthop]
Router(config)#ip flow-export destination <dest IP> <dest
udp-port>
Router(config)#ip flow-export source <interface>

Router (config)#ip flow-aggregation cache bgp_nexthop_tos

Router (config-flow-cache)#enabled

Router (config-if)#ip route-cache flow

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 134

 NetFlow BGP Next Hop TOS Aggregation
Testing
No Next Hop
Self NetFlow enable

Lo0 [Link]
C75d13--1
C75d13 C75d13--2
C75d13

s0: 1.4.040
AS 112
MPLS or NOT ping [Link]
l0: [Link] ping [Link]
Lo1 [Link] C72d13--1
C72d13

Next Hop Self

c75d13-2#sh ip bgp

Network Next Hop Metric LocPrf Weight Path

*>i1.1.1.1/32 [Link] 0 100 0 1 i
*>i9.9.9.9/32 [Link] 0 100 0 1 7 i

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 135

 NetFlow BGP Next Hop TOS Aggregation
Testing

C75d13--1
C75d13 C75d13--2
C75d13

C72d13--1
C72d13

c75d13-2#sh ip bgp

Network Next Hop Metric LocPrf Weight Path

*>i1.1.1.1/32 [Link] 0 100 0 1 i
*>i9.9.9.9/32 [Link] 0 100 0 1 7 i

sh ip cache verbose flow aggregation bgp-nexthop-tos

Src If Src AS Dst If Dst AS TOS Flows Pkts B/Pk Active

BGP NextHop
Et1/0/1 2 Et1/0/2 1 00 1 5 100 0.0
BGP: [Link] FOR A PING TO [Link]

Src If Src AS Dst If Dst AS TOS Flows Pkts B/Pk Active

BGP NextHop
Et1/0/1 2 Et1/0/2 1 00 1 5 100 0.0
BGP: [Link] FOR A PING TO [Link]
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 136
 Roadmap and Future
Directions

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 137

 External Roadmap for NetFlow

Scalability
Scalability && Optimizing
Optimizing data
data for
for Technology
Technology
Flexibility
Flexibility Flow
Flow processing
processing Coverage
Coverage

Q2 FY2003 Q3 FY2003 Q4+ FY2003

(1) NetFlow v9 (1) Random Sampled (1) NetFlow MIB

(2) BGP Nexthop NetFlow (2) NetFlow IPv6
(3) NetFlow Multicast (2) Flowmask filtering (3) AS Origin & Peer
(4) Enable per Sub- (4) Community ID
interface
(5) NetFlow MPLS (5) NAT
(6) NetFlow ipSec

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 138

 Future Directions

• Cat6000/7600
Version 8 for the native mode
Native mode will support dual export
Add support for version 9
• Cat4000
NetFlow should be supported very soon

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 139

 NetFlow FlowCollector

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 140

 NetFlow FlowCollector

• Flow record reception

• Data volume reduction
Workstation
Filtering, Aggregation
User
• Flexible thread language Filter Interface

• File storage
Flat or binary and Aggregate Config
Files
compression in 3.0
• File cleanup
• Solaris and HP-UX
• No flow de-duplication Flow Consumer
Storage Applications

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 141

 New Feature in NetFlow FlowCollector 4.0

• Support NF V9 data format and templates (inc. new

fields)
• Support user-configurable aggregation schemes
All formats v5 -> v9
• XML message set
• CNS bus support
• Deployment as Linux appliance (Redhat 7.2/IE21xx)
• Performance benchmarking document
(double throughput compared to NFC 3.6)
• Already available

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 142

 Per VPN Usage-based Accounting using
CNS Performance Engine
Accounting/Billing
Internet
InternetOSS
OSS Application
Applications
Applications

Well-defined
VPNSC Data Format
CNS Performance Engine
VPNSC-NetFlow data correlation § Customer
§ Site
CNS § CoS
NetFlow Summarize § Usage
Accounting Data
Per PE I/F
Engines

VPN1 Site1
PE1 VPN1 Site2
PE2
Y Bytes X Bytes

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 143

 NetFlow Partners
Billing Traffic Analysis

Denial of Service Mediation

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 144
 Deployment Guide

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 145

 Where to deploy Netflow?

Billing
Access routers
7xxx
Aggr. routers

12000
core

Full NetFlow

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 146

 Where to deploy Netflow?

Access routers
Accounting/Capacity planning
7xxx
Aggr. routers

12000
core

Full or sampled NetFlow

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 147
 Where to deploy Netflow?

• On the “edges” of the network.

• All routers because NetFlow accounts incoming
traffic only
• For billing, on the aggregation routers because
some GSR line cards only support sampled
NetFlow.
• For accounting, capacity planning, on the
aggregation routers or the GSR. Sampled
NetFlow could be sufficient.

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 148

 Where to deploy Netflow?

• For BGP informations, on the BGP peering

routers
• Can monitor one link, egress and ingress, but
should be on a MPLS PE-CE link.
• Basic principles:
Avoid a flow duplication design. Netflow Collector
doesn’t do flow de-duplication. Done by partner tools
Don’t account your exported data

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 149

 How many NetFlow Collector (NFC)?

• In theory, one collector per POP or Aggregation

Router (7x00 router)
• For VPNSC (MPLS VPN environment), we advice
one Collector per PE
• Basic principles:
Check your Sun capabilities
NFC sizer calculator. Reduce the number of routers per
NFC if needed.
Rule of thumb: 10 routers per NFC

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 150

 Deployment Tricks

• Enable the ifIndex persistence if accounting per

interface
• Look at the router cpu (<60%) and memory before
enabling NetFlow
• Check the export link bandwidth
• Use a dedicated export lan
• If you export too much traffic:
go for the aggregations, don’t export version 5
go for sampled if on a GSR
increase the aggregations timers
• Access-lists still account the traffic
B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 151
 References

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 152

 NetFlow References

• Netflow Services and Applications

[Link]
• Cisco Netflow Technologies Partner
[Link]
[Link]
• Cisco Netflow Collector/Analyzer
[Link]
mt/[Link]

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 153

 NetFlow References

• A complete white paper

[Link]
wk/intsolns/netflsol/[Link]
• An official Cisco Course (2 days)
NetFlow Service Advanced

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 154

 Questions?

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 155

 NetFlow Services
Benoit Claise
bclaise@[Link]

RIPE 44, Amsterdam

B. Claise © 2001, Cisco Systems, Inc. All rights reserved. 156