Tracking Station Traffic thru VRF Interfaces

Typically the virtual WiFi stations that LANforge creates are encapsulated with a virtual router device which holds its own routing table for the set of devices inside it. We do this to isolate the physical host’s routing table from the virtual mobile device we are emulating. This means there are special ways to operate connections through stations. A standard ping will always send packets through the port having the default gateway (which is your management network). You want to send packets to your test network.


Networks

Management: eth0 192.168.1.101/24 gateway 192.168.1.1
Test LAN:   <AP> 10.0.0.1/24 running dhcpd SSID testssid
Test WAN:   eth1 176.16.0.1/24, running dhcpd
Station:    wlan0 10.0.0.175/24 gateway 10.0.0.1 nameserver 10.0.0.1

Ping from the station

To ping from the station you use a version of the command sudo ip vrf exec wlan0 <command>. From a LANforge, you would shorted that a bit and say:

sudo -s
cd /home/lanforge
./vrf_exec.bash wlan0 ping -I wlan0 www.example.com

Ideally what this does is:

If this DNS server is not reachable or there is no route to that IP from the first response, ping tries harder and starts asking other interfaces if there is a route. This can dodge VRF and start asking on other interfaces. Ping is so persistent that will also check /etc/resolv.conf. This is troublesome because that was probably generated by the host’s dhclient process running on the management network. This can cause DNS queries out to your management network.

Changing resolv.conf

Changing resolv.conf to list the test net nameserver can help. It’s not great for the times when you want to run updates, but its not difficult to spot failed name lookups.

options timeout:1 attempts:1
nameserver 10.0.0.1

Those settings will restrict the amount of time spent by clients doing lookups.

Checking Route Tables

You can check the route tables available with ip route show table all, or grep for your station to see more quickly what route table is assigned to your station. Usually route tables match the number of the station master interface.

root # ip li sh wlan0
22: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master _vrf17 state UP mode DORMANT group default qlen 1000
link/ether 04:f0:21:3f:00:08 brd ff:ff:ff:ff:ff:ff permaddr 04:f0:21:2a:dc:08

root # ip route show table 17
default via 10.40.0.1 dev wlan0
10.40.0.0/20 dev wlan0 scope link src 10.40.0.75
local 10.40.0.75 dev wlan0 proto kernel scope host src 10.40.0.75
broadcast 10.40.15.255 dev wlan0 proto kernel scope link src 10.40.0.75

Prediction the Route

You can match up the route between a station and a destination using ip route get

 root # ./vrf_exec.bash wlan0 ip route get 10.40.0.1 from 10.40.0.75
10.40.0.1 from 10.40.0.75 via 192.168.92.1 dev eth0 uid 0
cache

You might think that looks weird. In fact, I don’t know why eth0 is there at all.

Using Perf to Trace the calls

This is a fascinating way to use perf (which uses eBPF probably) to see what actually is happening when doing a ping:

# ./vrf_exec.bash wlan0 perf trace --no-syscalls --event 'net:*' ping -R -O -c1 -n -B -I wlan0 lanforge.net
PING www.lanforge.net (10.41.0.1) from 10.40.0.75 wlan0: 56(124) bytes of data.
0.000 ping/1880863 net:net_dev_queue(skbaddr: 0xffff88818fffa400, len: 138, name: "wlan0")
0.007 ping/1880863 net:net_dev_start_xmit(name: "wlan0", skbaddr: 0xffff88818fffa400, protocol: 2048, len: 138, network_offset: 14, transport_offset_valid: 1, transport_offset: 74)
0.029 ping/1880863 net:net_dev_xmit(skbaddr: 0xffff88818fffa400, len: 138, name: "wlan0")
From 10.40.0.1 icmp_seq=1 Destination Host Unreachable

--- www.lanforge.net ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

1.133 ping/1880863 net:net_dev_queue(skbaddr: 0xffff888121376ce8, len: 174, name: "eth0")
1.137 ping/1880863 net:net_dev_start_xmit(name: "eth0", skbaddr: 0xffff888121376ce8, protocol: 2048, ip_summed: 3, len: 174, data_len: 108, network_offset: 14, transport_offset_valid: 1, transport_offset: 34, gso_segs: 1, gso_type: 1)
1.140 ping/1880863 net:net_dev_xmit(skbaddr: 0xffff888121376ce8, len: 174, name: "eth0")

Notice how since 10.41.0.1 is not routable (unlike 10.41.0.1) ping digs down to the interface with the default route (eth0) and attempts to use that.

Using getent

Another easy way to see if your configuration is doing what you want is to check how nsswitch is configured by using getent:

# ./vrf_exec.bash sta0000 getent ahosts www.lanforge.net
10.41.0.1 STREAM www.lanforge.net
10.41.0.1 DGRAM
10.41.0.1 RAW

This is much easier to use, if we were paying attention to what was in our /etc/hosts file, we would have spotted this typo sooner!

WiFi Fundamentals Course

The team at Candela Technologies is pleased to offer a free online semester course about the fundamentals of WiFi technology. This course is being taught by industry experts and all the concepts will be taught in a very practical manner keeping the industry applications of technology in mind.

Format

Classes will be delivered live over webinar sessions every Tue and Thu 4pm IST. All sessions will be recorded and the recordings will be made available on the course webpage.

Audience

This course is designed for junior engineers already working in R&D organizations on WiFi technology, engineering students who are interested in the computer networking and Wireless communications domain and also academicians who are interested in developing new skill courses in engineering programs. Prerequisite for this course is basic understanding of computer networking.

Registration

Anyone interested can register for this course by visiting the course webpage. Interested people can also join the Weekly WiFi wisdom Whatsapp group to get whatsapp updates and participate in conversations related to the course. Links to register for the course and group can be found of course webpage.

TR398 issue 3 Mesh Rate vs Range

Today I improved and implemented the TR398 Mesh RvR test in our LANforge software and testbed. This test requires a TR398 + Mesh testbed, and the parts of interest for this particular test are the Root, Node-1, and STA chambers. All three are connected via attenuators. In this test case, the Root and Node-1 chambers vary the attenuation to emulate a backhaul signal level of -60, -70, and -80. At each iteration, a TCP upload and download throughput test is run.

First, I calibrated using two of our CT521a devices acting as AP and/or STA. This requires some initial setup, and physically placing the CT521a in the correct location. After that, the TR398 automation code will do the calibration automatically. Here’s a picture of the main chamber in calibration mode. Note that the antenna being used for the calibration is in the center of the turntable.

The Node-1 system also needs a calibration AP to calibrate its backhaul and STA to Node-1 link. I raised it up on some foam blocks so that the directional antennas were pointing at the calibration antennas similar to how the DUT AP will be positioned:

Once the calibration is complete, the DUT APs are placed back in the position formerly occupied by the calibration devices.

And then run the TR398 test case, which passes all tests as currently written with this DUT:

See the full report here: https://www.candelatech.com/examples/TR-398v2-report-2022-05-05-04-19-30-mesh-rvr.pdf

Happy testing!

–Ben

IPTables and conntrack

When you are using LANforge as a virtual router doing NAT, you might need to see how many NAT table entries you’re handling.  This can be important because NAT entries take memory, and if you want to handle 65,000 simultaneous connections, you might be heading for trouble.

If your LANforge is only generating traffic, you won’t see NAT entries…rather you want to use netstat -ntp to see how many open connections there are.

LANforge uses iptables PREROUTING heavily, forcing each port to have it’s own set of tables. When you type iptables -nvL and see nothing…that’s because nothing is in the tables for your default route, which is probably eth0. You get close with the raw table. Try iptables -S -t raw. You will see PREROUTING entries for every interface:

# iptables -S -t raw
-P PREROUTING ACCEPT
-P OUTPUT ACCEPT
-A PREROUTING -i br2000 -j CT --zone 10001
-A PREROUTING -i eth1 -j CT --zone 10001
-A PREROUTING -i vap13 -j CT --zone 10001
-A PREROUTING -i vap14 -j CT --zone 10001
-A PREROUTING -i eth2 -j CT --zone 10001

This shows we have a CT chain and a zone note for that chain.

When you create a virtual router, add NAT to a port in it, you can view the NAT table entries with conntrack.

* conntrack -L will list them all, but that’s probably not super useful
If you’re running TCP-multicon connections, expect thousands of connections.
* conntrack -C will show how many NAT entries are present, so you can avoid doing a conntrack -L | wc -l

Happy networking!

ath10k-ct firmware released

I am pleased to announce the latest release of the ath10k-ct firmware for QCA WiFi chips.  The 10.1 (wave-1) firmware has improved survey support, changes to how rate-ctrl retries frames, and includes PMF support.  Full release notes here:  http://www.candelatech.com/downloads/ath10k_release_notes.txt

The 10.4 wave-2 firmware includes some significant rate-ctrl improvements, PMF support, and various bug fixes and small features.  Full release notes are here:  http://www.candelatech.com/downloads/ath10k-10-4/ath10k_release_notes_5.txt

In addition, I have uploaded a beta 10.4 firmware that is a rebase of all 900+ Candela patches on top of a more recent upstream QCA firmware.  This is easier to bisect for regressions and has some fixes and other improvements from the QCA firmware itself.  See the beta section if you want to try this out:  http://www.candelatech.com/ath10k-10.4.php

Please report bugs to the ath10k-ct github project or to [email protected]

Pack a Gateworks Full of Radios

Candelatech strives to provide products that are flexible and quiet. The Gateworks Ventana platform provides a great opportunity to provide a high density virtual access point emulation platform. With a bit of shrewd of modification, this GW5400 model can provide 30 or more virtual WiFi access points.

Keeping that many radios cool is what is important. A bit of aluminium plate and heat conductive tape worked wonders for this platform. We’ve seen it run for a month straight without heat related issues. This is effectively a sealed system. It was warm to the touch, but nowhere near hot.

gateworks_n2

gateworks_n1

LANforge 5.3.2 Released

We at Candela Technologies have been busy improving LANforge. In 5.3.2 we have a couple of great new features:

  • WiFi HotSpot 2.0-R2 Support
  • WiFi HS 2.0-R2 Automatic Server Setup
  • Linux 4.0 support
  • Improved 802.11AC support, plus Linux 4.0 features
  • Improved NFS support for mitigating stuck mounts
  • DHCP Vendor-ID extension
  • More WiFi Capacity testing features

We are excited about this release!

Open Engineer Position

Hello!

We are considering hiring another engineer for our business here in Ferndale, WA.

Our best candidate would meet these requirements below, but we will also consider CS or similar students/new-grads with fewer skills and experience but willingness and ability to learn. And if nothing else, we would consider hiring someone just to do some soldering and other relatively straight-forward work for hourly wage on an as-needed basis.

We make network test equipment, including Wifi load generators, RF attenuators, and network impairment emulators. The interesting work is done on Linux, but we also cross-compile a subset of features to Windows and support our GUI on Windows.

We currently have 4 employees, and have just rented a nice new office space in Ferndale. Pay depends on experience and such, but typically we offer very competitive salary (or hourly wages).

* Excellent C++ and/or Java programmer. Perl, html and other such languages are a plus.
* Linux skills (admin, install, configure, trouble-shoot issues)
* Ethernet, Wifi, and other networking skills.
* System Test skills.
* Ability to solder and assemble relatively simple electronics boards.
* And above all, a willingness to learn and do new things with good attitude.

If you are interested, please send resume to greearb x40 candelatech x2E com.

And feel free to forward this to anyone you know who might be interested.

Thanks,
Ben