MUM Ho Chi Minh, Vietnam
January 2019
Sao Thiên Vương - http://switch-router.com 1
My name: Binh Nguyen Dang Thai
Email:
[email protected] Cellphone: 0909006026
Working for: URANUS INFORMATION
TECHNOLOGY COMPANY
Sao Thiên Vương - http://switch-router.com 2
URANUS has been focusing on wireless
networking market since 2015.
Became MikroTik distributor since 2016.
Website: http://uranus.com.vn
Support website: http://forum.uranus.com.vn
Sao Thiên Vương - http://switch-router.com 3
Using PCC (Per Connection Classifier) to load
balancing multiple Internet links correctly.
Solving common problems beside PCC load
balancing.
Sao Thiên Vương - http://switch-router.com 4
Load Balancing is a technique to distribute the
workload across two or more network links in
order to maximize throughput, minimize
response time, and avoid overload.
Distribution may be symmetrical or asymmetrical
depending on rate of network links.
Useful when the downstream bandwidth
requirement to a single routing device exceeds the
capabilities of a single network link.
Using multiple network links with load balancing,
instead of single network links, may increase
reliability through redundancy
Sao Thiên Vương - http://switch-router.com 5
PCC divides the traffic into streams and then
uses routing rules to sort the traffic evenly (or
not) across multiple network links.
This is done by:
Using a hashing algorithm to first sort the traffic
based on source address, source port, destination
address, destination port or various combinations
thereof.
Using packet marking and marking route and
several routing tables to ensure traffic follows a
specified route out the specified WAN interface.
Sao Thiên Vương - http://switch-router.com 6
MikroTik RouterOS is extremely powerful and
configurable, so this can be a double edged
sword, several possible solutions to the same
problem.
Each has multiple moving pieces
Greatest success with any solution by
understanding the pieces and what they do.
Sao Thiên Vương - http://switch-router.com 7
Terms must to understand
Packet: The container for our data, header and
payload.
Connection: “Conduit” through which host to host
communication occurs, based on Src/Dst addresses
and ports
Mangle Facility: is a kind of 'marker' that marks
packets for future processing with special marks.
They identify a packet based on its mark and
process it accordingly. Mangle marks exist only
within the router, they are not transmitted across the
network.
Sao Thiên Vương - http://switch-router.com 8
Terms must to understand
PCC: PCC matcher will allow to divide traffic
into equal streams with ability to keep packets
with specific set of options in one particular
stream (we can specify this set of options from
src-address, src-port, dst-address, dst-port)
Sao Thiên Vương - http://switch-router.com 9
PCC (cont.) – How does this work?
PCC takes selected fields from IP header, and
with the help of a hashing algorithm converts
selected fields into 32-bit value. This value then is
divided by a specified Denominator and the
remainder then is compared to a specified
Remainder, if equal then packet will be captured.
You can choose from src-address, dst-address,
src-port, dst-port from the header to use in this
operation.
Sao Thiên Vương - http://switch-router.com 10
PCC (cont.) How does this work (cont.)
Sao Thiên Vương - http://switch-router.com 11
PCC (cont.) – Example: 3 WAN Connections
3 PCC rules required
ValuesToHash Denominator Remainder
1st WAN connections
2nd WAN connections
3rd WAN connections
The 1st line means "produce the output of the hash function given the
packet’s both IP addresses, divide it by 3 and if the remainder is 0, perform
the action of marking the connection as WAN1“
The 2nd rule means "produce the output of the hash function given the
packet’s both IP addresses, divide it by 3 and if the remainder is 1, perform
the action of marking the connection as WAN2“
The 3rd rule means "produce the output of the hash function given the
packet’s both IP addresses, divide it by 3 and if the remainder is 2, perform
the action of marking the connection as WAN3“
Sao Thiên Vương - http://switch-router.com 12
PCC (cont.) - How to set PCC, Remember?
2 WAN connections:
2 / 0 First WAN
2 / 1 Second WAN
3 WAN connections:
3 / 0 First WAN
3 / 1 Second WAN
3 / 2 Third WAN and so on...
Sao Thiên Vương - http://switch-router.com 13
PCC (cont.): Where does it found?
Sao Thiên Vương - http://switch-router.com 14
Terms must to understand
Routing Table: Route rules, the rules the
router uses to determine what to do with a
packet. By comparing the destination address
in the packet to the list of routes, the router
decides which interface to send the packet out.
By adding a routing mark with mangle, we can
have multiple routing tables!
Sao Thiên Vương - http://switch-router.com 15
Routing Table (cont.)
Sao Thiên Vương - http://switch-router.com 16
Terms must to understand
Routing-mark:
RouterOS attribute assigned to each packet
Routing-mark can be changed in firewall mangle
facility just before any routing decision:
chain Prerouting – for all incoming traffic
chain Output – for outgoing traffic from router
Every new routing mark has its own routing
table with the same name
By default all packets have the “main” routing
mark
Sao Thiên Vương - http://switch-router.com 17
Terms must to understand
MikroTik packet low: This manual describes
the order in which an IP packet traverses
various internal facilities of the router and
some general information regarding packet
handling, common IP protocols and protocol
options.
Sao Thiên Vương - http://switch-router.com 18
MikroTik packet low (cont.)
Overall Packet flow Diagram (RouterOS v6):
https://wiki.mikrotik.com/wiki/Manual:Packet_Fl
ow
Sao Thiên Vương - http://switch-router.com 19
Scenario: One router MikroTik RB1100AHx2,
many clients, 3 FTTH links
Sao Thiên Vương - http://switch-router.com 20
Set up the basic portion of the network
Private IP address on Bridge interface, Bridge interface is
logical bridge or CPU bridge, included ether4 – ether13.
DHCP Server on Bridge interface
DNS server
Configure PPPoE clients for 3 WAN links, in this case on
ether1, ether2, ether3
Masquerade for 3 PPPoE clients
Defines default route for 3 WAN links
Firewall if required
Create load balancing part of the configuration:
Mangle rules
Routing tables
Sao Thiên Vương - http://switch-router.com 21
Assigns an IP address and ports for Bridged interface
/ip address
add address=192.168.100.1/24 interface=bridge network=192.168.100.0
/interface bridge port
add bridge=bridge hw=no interface=ether6
add bridge=bridge hw=no interface=ether9
add bridge=bridge hw=no interface=ether8
add bridge=bridge hw=no interface=ether10
add bridge=bridge hw=no interface=ether11
add bridge=bridge hw=no interface=ether12
add bridge=bridge hw=no interface=ether13
add bridge=bridge hw=no interface=ether7
add bridge=bridge hw=no interface=ether4
add bridge=bridge hw=no interface=ether5
Sao Thiên Vương - http://switch-router.com 22
Configure PPPoE clients for 3 WAN links
/interface pppoe-client
add disabled=no interface=ether1 keepalive-timeout=60 max-
mru=1480 max-mtu=\
1480 mrru=1600 name=pppoe-out1 password=T9a799
user=t008_ftth_abc
add disabled=no interface=ether2 keepalive-timeout=60 max-
mru=1480 max-mtu=\
1480 mrru=1600 name=pppoe-out2 password=r23hnV
user=t008_ftth_xyz
add disabled=no interface=ether3 keepalive-timeout=60
name=pppoe-out3 \
password=hT2t5g user=t008_ftth_123
Sao Thiên Vương - http://switch-router.com 23
Masquerade for 3 WAN links
/ip firewall nat
add action=masquerade chain=srcnat out-interface=pppoe-out1
add action=masquerade chain=srcnat out-interface=pppoe-out2
add action=masquerade chain=srcnat out-interface=pppoe-out3
Where can we configure on Winbox?
Sao Thiên Vương - http://switch-router.com 24
Define default routes for 3 WAN links
/ip route
add check-gateway=ping distance=1 gateway=pppoe-out3
add check-gateway=ping distance=2 gateway=pppoe-out2
add check-gateway=ping distance=3 gateway=pppoe-out1
What does look like on main routing tables
Sao Thiên Vương - http://switch-router.com 25
Create load balancing part of the configuration:
Define address list for Local Address Network
(LAN)
Mangle rules
Sao Thiên Vương - http://switch-router.com 26
Define LAN address list
/ip firewall address-list
add address=192.168.100.0/24 list=LAN
Sao Thiên Vương - http://switch-router.com 27
Bypass mangle rules for local traffic
We need to ensure that any traffic going to our
local bypasses all the ‘line balancing’ rules.
We do this with an action of ‘accept’. In this case,
internal traffic is on LAN address list.
/ip firewall mangle
add action=accept chain=prerouting
comment="Accept traffic from LAN" \
dst-address-list=LAN src-address-list=LAN
Sao Thiên Vương - http://switch-router.com 28
Topology:
In this topology, there are 4
possible traffic flows
WAN -> Router
Router ->WAN
WAN -> LAN
LAN -> WAN
Sao Thiên Vương - http://switch-router.com 29
Taking care of incoming connections
When a connection is initiated from the
internet through one of the ISPs we need to
ensure that this connections is replied through
the same ISP (from the same public IP)
We need to mark these connections, and then
put them in the proper routing table.
Sao Thiên Vương - http://switch-router.com 30
Router marking – WAN -> Router
Catch the connection from internet to the router, and mark them
add action=mark-connection chain=input comment="Mark all
connections that are initiated from outsite" connection-mark=no-mark
in-interface=pppoe-out1 new-connection-mark=WAN1-to-ROS
passthrough=no
add action=mark-connection chain=input connection-mark=no-mark
in-interface=pppoe-out2 new-connection-mark=WAN2-to-ROS
passthrough=no
add action=mark-connection chain=input connection-mark=no-mark
in-interface=pppoe-out3 new-connection-mark=WAN3-to-ROS
passthrough=no
Sao Thiên Vương - http://switch-router.com 31
Router marking – WAN -> Router (cont.)
Then put these connections into the proper routing tables.
add action=mark-routing chain=output comment="Mark
routing for router's replies" connection-mark=WAN1-to-ROS
new-routing-mark=WAN1 passthrough=no
add action=mark-routing chain=output connection-
mark=WAN2-to-ROS new-routing-mark=WAN2
passthrough=no
add action=mark-routing chain=output connection-
mark=WAN3-to-ROS new-routing-mark=WAN3
passthrough=no
Sao Thiên Vương - http://switch-router.com 32
Router marking – WAN -> Router (cont.)
What does look like on Winbox
Sao Thiên Vương - http://switch-router.com 33
Taking care of the LAN
Same principle applies to the LAN
Connections initiated from the internet through
one ISP, should be replied to through the same
ISP
Sao Thiên Vương - http://switch-router.com 34
LAN marking
add action=mark-connection chain=forward
comment="Mark all connections for NAT inbound"
connection-mark=no-mark in-interface=pppoe-out1 new-
connection-mark=WAN1-to-LAN passthrough=no
add action=mark-connection chain=forward connection-
mark=no-mark in-interface=pppoe-out2 new-connection-
mark=WAN2-to-LAN passthrough=no
add action=mark-connection chain=forward connection-
mark=no-mark in-interface=pppoe-out3 new-connection-
mark=WAN3-to-LAN passthrough=no
Sao Thiên Vương - http://switch-router.com 35
LAN marking (cont.)
add action=mark-routing chain=prerouting connection-
mark=WAN1-to-LAN new-routing-mark=WAN1
passthrough=no src-address-list=LAN
add action=mark-routing chain=prerouting connection-
mark=WAN2-to-LAN new-routing-mark=WAN2
passthrough=no src-address-list=LAN
add action=mark-routing chain=prerouting connection-
mark=WAN3-to-LAN new-routing-mark=WAN3
passthrough=no src-address-list=LAN
Sao Thiên Vương - http://switch-router.com 36
Incoming connections - done
We have ensured that when a connection
from the internet to our router, or services
inside of our network is established, it works.
LAN – partially done
So what about connections outgoing from our
LAN to the internet?
These we actually want to load-balance
Sao Thiên Vương - http://switch-router.com 37
LAN -> WAN mangle
add action=mark-connection chain=prerouting comment="PCC rules“
connection-mark=no-mark dst-address-list=!LAN dst-address-
type=!local new-connection-mark=LAN-to-WAN1
passthrough=yes per-connection-classifier=both-addresses:3/0
src-address-list=LAN
add action=mark-connection chain=prerouting connection-mark=no-
mark dst-address-list=!LAN dst-address-type=!local new-
connection-mark= LAN-to-WAN2 passthrough=yes per-
connection-classifier=both-addresses:3/1 src-address-list=LAN
add action=mark-connection chain=prerouting connection-mark=no-
mark dst-address-list=!LAN dst-address-type=!local new-
connection-mark=LAN-to-WAN3 passthrough=yes per-
connection-classifier=both-addresses:3/2 src-address-list=LAN
Sao Thiên Vương - http://switch-router.com 38
LAN -> WAN mangle (cont.)
add action=mark-routing chain=prerouting comment="Mark
routing for upload packets from marked connections“
connection-mark=LAN-to-WAN1 dst-address-list=!LAN
new-routing-mark=WAN1 passthrough=no src-address-
list=LAN
add action=mark-routing chain=prerouting connection-mark=LAN-
to-WAN2 dst-address-list=!LAN new-routing-mark=WAN2
passthrough=no src-address-list=LAN
add action=mark-routing chain=prerouting connection-mark=LAN-
to-WAN3 dst-address-list=!LAN new-routing-mark=WAN3
passthrough=no src-address-list=LAN
Sao Thiên Vương - http://switch-router.com 39
Mangle in GUI
Sao Thiên Vương - http://switch-router.com 40
Routing tables
Sao Thiên Vương - http://switch-router.com 41
I can access just 1 WAN interface from Internet.
I can not access LAN services from Internet via
inbound NAT.
How to load balance for different rate WAN
links.
Sao Thiên Vương - http://switch-router.com 42
https://wiki.mikrotik.com/wiki/Manual:PCC
https://blog.linitx.com/load-balancing-
multiple-internet-connections/
MikroTik RouterOS Workshop Load Balancing
Best Practice by Warsaw, MUM Europe 2012.
Bandwidth-based load-balancing with failover.
The easy way by Tomas Kirnak.
Load Balancing Using by Steve Discher.
Sao Thiên Vương - http://switch-router.com 43
THANK YOU!
Sao Thiên Vương - http://switch-router.com 44
