Monitoring of Ethernet

Messages

IEC 61850 Seminar

Dr. Alexander Apostolov

K02 03 20060309 Page: 1

© OMICRON K02 03 20060309

 Scope

• Discuss Ethernet architectures supporting

IEC 61850 projects
• Not defined in the standard but essential for a
real project !
• Background on Ethernet principles and
technologies
• Practical examples

© OMICRON K02 03 20060309 Page: 2

Transmission Definitions

SIMPLEX
Transmission in one direction only

HALF DUPLEX
Two way means of transmission but data can only
travel in one direction at a time

FULL DUPLEX
Transmission in both directions simultaneously

© OMICRON K02 03 20060309 Page: 3

 Transmission Definitions

PROTOCOL
Rules and procedures that communications networks
use to communicate on the communications medium

CONNECTION
Communications are Connection Oriented or
Connectionless

© OMICRON K02 03 20060309 Page: 4

Transmission Definitions
Point-to-point

Engineering Laptop
Station

Modem IED

Modem

IED

© OMICRON K02 03 20060309 Page: 5

 Transmission Definitions
Shared Access
Substation
Ethernet HMI IED IED IED

Ethernet
Switch

Proxy Substation
Server HMI

RS485
Ethernet
Legacy IEC 61850 IEC 61850
IED IED IED

© OMICRON K02 03 20060309 Page: 6

Transmission Definitions
NETWORK ACCESS

CARRIER SENSE METHODS

CSMA (Carrier Sense Multiple Access)/CD
(Collision Detection)
Medium Access Control (MAC)

TOKEN ACCESS METHOD

The device that has the Token has access to
transmit

RESERVATION METHOD
Each device has a predefined time slot to transmit

© OMICRON K02 03 20060309 Page: 7

 Transmission Definitions

Medium - Ethernet devices attach to a common medium that

provides a path along which the electronic signals will travel:
- historically, this medium has been coaxial copper cable
- more commonly a twisted pair
-fiber optic cabling.

Segment - a single shared medium as an Ethernet segment.

Nodes - devices that attach to that segment are stations or
nodes.

Frame - The nodes communicate in short messages called

frames, which are variably sized chunks of information.

© OMICRON K02 03 20060309 Page: 8

 Transmission Medium

RS 232 - 9/25 pin D connector

EIA 232 - Maximum data rate 20kb/s
- Maximum distance 50 feet

RS 485 - 2/4 conductors

EIA 485 - Maximum data rate 10Mb/s
- Maximum distance 4000 feet

Optical fiber - Immunity to electrical interference

- Advantages in distance and speed

Wireless

© OMICRON K02 03 20060309 Page: 9

 Data Security

Noise corruption of data

• Parity bit check
• Two coordinate parity check
• Checksums
• Cyclic redundancy check

Unauthorized access
• Password protection
• Modem dial back
• Firewalls

© OMICRON K02 03 20060309 Page: 10

Open Systems Interconnection
(OSI) Model
Application Selects appropriate service for application

Presentation Provides code conversion, data reformatting

Coordinates interaction between end

Session
application process
Provides for end to end data integrity and
Transport
quality of service

Network Switches and routes information

Transfers unit of information to other end

Data Link
of physical link

Physical Transmits bit stream to medium

© OMICRON K02 03 20060309 Page: 11

 OSI Stack
PCI =Protocol Control Inform. Commands or data
PDU =Protocol Data Unit Application
H =Header Presentation
PCI PDU
T =Trailer
PCI PDU Session

PCI PDU Transport

PCI PDU Network

PCI PDU Data link

PDU PDU PDU Physical

H fragment T H fragment T H fragment T

Frames (Ethernet, token ring, etc)

© OMICRON K02 03 20060309 Page: 12

 Communications Process

Upper Layer Upper Layer

SAP SAP

Entity with Peer-to-peer dialog Entity with

services to offer services to offer

SAP SAP

Lower Layer Lower Layer

© OMICRON K02 03 20060309 Page: 13

 Ethernet Frame
Pre SFD DA SA Length MAC Data + Pad FCS
Type
7 1 6 6 2 46-1500bytes 4

• Pre: The Preamble is an alternating pattern (7 bytes) of 1 and 0 that

tells receiving stations that a frame is coming
• SFD: Start-of-frame delimiter (1 byte: 10101011) indicating that the
next bit is the left-most bit in the left-most byte of the destination
address.
• DA: Destination address (6 bytes) identifies which station(s) should
receive the frame
• SA: Source addresses (6 bytes) identifies the sending station

© OMICRON K02 03 20060309 Page: 14

 Ethernet Frame
Pre SFD DA SA Length MAC Data + Pad FCS
Type
7 1 6 6 2 46-1500bytes 4
• Length Type: Number of MAC-client data bytes that are contained
in the data field of the frame

• MAC Client Data: A sequence of n bytes (46=< n =<1500) of any

value. (The total frame minimum is 64 bytes). The Pad contains (if
necessary) extra data bytes in order to bring the frame length up to
its minimum size. A minimum Ethernet frame size is 64 bytes from
the Destination MAC Address field through the Frame Check
Sequence.

• FCS: The Frame Check Sequence is a 32-bit cyclic redundancy

check (CRC) value

© OMICRON K02 03 20060309 Page: 15

 Ethernet

• Media Access Control (MAC) Address -

This is the physical address of any device,
such as the NIC in a computer, on the
network.
• The MAC address has two parts, each 3
bytes long.
• The first 3 bytes identify the company that
made the NIC.
• The second 3 bytes are the serial number
of the NIC itself.

© OMICRON K02 03 20060309 Page: 16

 Ethernet

• Unicast - A transmission from one node

addressed specifically to another node.
• Multicast - When a node sends a
packet addressed to a special group
address. Devices that are interested in
this group register to receive packets
addressed to the group.
• Broadcast - When a node sends out a
packet that is intended for transmission
to all other nodes on the network.

© OMICRON K02 03 20060309 Page: 17

 Network Terminology

Network An interconnected group of nodes or

stations linked by communication channels

Node The interface point where one or more

functional units are connected

LAN Local area network (<5km)

WAN Wide area network

Network topology Pattern of nodes and their interconnection

© OMICRON K02 03 20060309 Page: 18

 Ethernet Basic Principle
1. Listen and broadcast if the wire is free
2. If there is a collision then re-transmit in an interval [0, 2N-1]

Ethernet

Collisions are an issue for real time automation

© OMICRON K02 03 20060309 Page: 19

 Switches principles

Internal Switches
Queue

Switches eliminate collisions and are thus

systematically used for industrial applications

© OMICRON K02 03 20060309 Page: 20

 Switches: Performances

Pure delay (store & forward): 5-10μs

Maximum frame size: 1536 bytes
1536 bytes @ 100 Mbps = 123 μs
Goose size << 1536 bytes

Performances bottleneck is no longer the communication

Network but possibly the applications (ms timefame)

© OMICRON K02 03 20060309 Page: 21

Switches: Unicast

© OMICRON K02 03 20060309 Page: 22

Switches: Multicast

2 2

© OMICRON K02 03 20060309 Page: 23

Switches: Broadcast

2 2 2 2

© OMICRON K02 03 20060309 Page: 24

To Process
Binary Input
Wall-to-wall performances
Filtering [2-10]

Logical [1-50]
example (ms)
[5-75]

Communication [2-15]

Switching [0.1]

Communication [2-15]

Logical [1-50] [4-72]

Relaying [1-7]

Process To + 9/147
Binary Output

© OMICRON K02 03 20060309 Page: 25

Switches: other features used in
Substation Automation

• Substation environment !
• Electro-magnetic compatibility
• DC power supply with 20ms voltage dips, possibly
redundant
• Priority management (802.3p):
• Capability to have priority queues in order to further
boost the communication performances
• Virtual LAN – VLAN (802.3q):
• Capability to create logical groups of devices in order
to filter the messages not belonging to one group

© OMICRON K02 03 20060309 Page: 26

Switches: other features used in
Substation Automation

• Management
• Capability to detect a switch failure: SNMP or watch-
dog
• Redundancy management
• No standard today for hard real time redundancy
• Redundancy between switches (not applications)
• See further slides

© OMICRON K02 03 20060309 Page: 27

 Base architectures
STAR RING DOUBLE STAR

Basic scheme since The most common scheme since The most secured
failure of the central good performance/cost ratio scheme since multiple
switch leads to the (save 2 central switches vs. start failures are tolerated
total communication configuration) except the two central
failure switches
simultaneously
Failure of a single fiber or device
does affect the system once
reconfiguration is completed

© OMICRON K02 03 20060309 Page: 28

 Redundancy management
• A way to improve system availability and reliability
• Behavior is not defined in IEC 61850
• Typical example where the architect needs to define what
shall be done… and first check on paper that it can be
supported by the IEDs

• May be applied to:

• Communication infrastructure: tolerate the loss of an optical
fiber or/and the loss of a central switch
• Clients: capability for a client to continue the tasks initiated
by a first client (example: Graphical user interface)
• Servers: capability for a client to switch to a redundant server
if the first one is not operational (example: CT/VT sensor)

• 99.99% availability requests the 3 types of redundancy

© OMICRON K02 03 20060309 Page: 29

 Redundancy management at
communication level
• Defined between Ethernet switches
• If a device has two Ethernet port it must also have an integrated
switch
• Spanning tree mechanisms
• Recalculation of the route between switches using an internal
protocol between switches and avoiding loops
• Base: 802.3d. Typically 30 s reconfiguration time
• Fast: 803w. Typically 100 ms reconfiguration time, some
implementations claims 5 ms per switch (N x 5ms with a ring made of
N switches)
• Other mechanisms
• Detects the failure of the adjacent switch to reconfigure
• For ring topology, less than 1 ms reconfigurations
• Sends two signals at the same time and keep the first one arrived
• No switch-over time, but need to keep a correct buffer size (dual homing)

Substation applications enable to get extremely fast

Switch-over time

© OMICRON K02 03 20060309 Page: 30

 Example (ring)
“Repeaters”
(IEE 802.3 Ethernet switch)
+ Self Healing Manager (SHM)
Ethernet Ports 10/100 Base TX

EEPRO M

SW ITCH

Port MII

SHM N°
Failsafe Self healing ring
output relays m anager

100Mbs Full duplex

opto opto

Prim ary Fiber

Secondary Fiber
Optical Ring

© OMICRON K02 03 20060309 Page: 31

 Example (ring)

X During nominal situation, Ethernet packet goes in

primary fibre always in the same direction, and
only a checking frame (4 bytes) is sent every 5ms
in secondary fibre in the opposite direction.
Primary fibre

1 2 3 4 5 6 7 8 9 10 11

switch switch switch

switch switch
RP EP
C D E
A B

ES RS

Secondary fibre

© OMICRON K02 03 20060309 Page: 32

 Example (ring)
If the connection between 2 switches is broken
X The Ethernet network will continue to run correctly.
X Both SHM start immediately the network self-healing.
Š At one side, the messages received are no more emitted to
the primary fibre but to the secondary fibre.
Š On the other side of the cut off, the messages received to secondary
are emitted to primary and the new topological loop is closed.
Primary fibre

1 2 3 4 5 6 7 8 9 10 11

switch switch switch

switch switch
RP EP
C D E
A B

ES RS

Secondary fibre

© OMICRON K02 03 20060309 Page: 33

Other communication elements
IEC Com. Com.
OSI Model Technology Protocol Services Element

Application MMS/SNTP Report, Control, Gateway/

File Proxy

Presentation

Session
Software

Transport TCP

Network IP Router

Bridge/Switch
Data Link 802.3 GOOSE
/Proxy
Hardware
Physical 802.3

Need to design the communication infrastructure

according to the system constraints and requirements

© OMICRON K02 03 20060309 Page: 34

 Proxy use
GUI
Gateway

Client
T104,
http, etc.
Client Client

Proxy Proxy
IP Routable

Server Server Server

Fast Peer-to-Peer

Bay Bay Bay Bay

Bay Bay

Server (resp. A proxy can

subscriber, clients) replicate the real
performance is time status &
dependant on the measurement of a
number of clients series of servers
(resp. pubslisher, and be possibly
servers) redundant

© OMICRON K02 03 20060309 Page: 35

 IEC 62351
• “Data and Communication Security” series of standard
• Currently at CD stage (votes till August 2005)
• IEC 62351-6: Security for IEC 61850 profiles
• Relies on IEC 62351-4: Profiles including MMS
• Relies on IEC 62351-3: Profiles including TCP
IEC Com. Security
OSI Model Technology Protocol Services Services

Report, Control, Authentification,

Application MMS/SNTP
File Replay

Presentation

Session
Software
Authentification,
Transport TCP Encryption, Replay

Network IP

Authentification,
Data Link 802.3 GOOSE Replay

Hardware
Physical 802.3

© OMICRON K02 03 20060309 Page: 36