VPP with sFlow - Part 1

Introduction

In January of 2023, an uncomfortably long time ago at this point, an acquaintance of mine called Ciprian reached out to me after seeing my [DENOG #14 ] presentation. He was interested to learn about IPFIX and was asking if sFlow would be an option. At the time, there was a plugin in VPP called [flowprobe ] which is able to emit IPFIX records. Unfotunately I never really got it to work well in my tests, as either the records were corrupted, sub-interfaces didn’t work, or the plugin would just crash the dataplane entirely. In the meantime, the folks at [Netgate ] submitted quite a few fixes to flowprobe, but it remains an expensive operation computationally. Wouldn’t copying one in a thousand or ten thousand packet headers with flow sampling not be just as good?

In the months that followed, I discussed the feature with the incredible folks at [inMon ], the original designers and maintainers of the sFlow protocol and toolkit. Neil from inMon wrote a prototype and put it on [GitHub ] but for lack of time I didn’t manage to get it to work, which was largely my fault by the way.

However, I have a bit of time on my hands in September and October, and just a few weeks ago, my buddy Pavel from [FastNetMon ] pinged that very dormant thread about sFlow being a potentially useful tool for anti DDoS protection using VPP. And I very much agree!

sFlow: Protocol

Maintenance of the protocol is performed by the [sFlow.org ] consortium, the authoritative source of the sFlow protocol specifications. The current version of sFlow is v5.

sFlow, short for sampled Flow, works at the ethernet layer of the stack, where it inspects one in N datagrams (typically 1:1000 or 1:10000) going through the physical network interfaces of a device. On the device, an sFlow Agent does the sampling. For each sample the Agent takes, the first M bytes (typically 128) are copied into an sFlow Datagram. Sampling metadata is added, such as the ingress (or egress) interface and sampling process parameters. The Agent can then optionally add forwarding information (such as router source- and destination prefix, MPLS LSP information, BGP communties, and what-not). Finally the Agent will periodically read the octet and packet counters of physical network interface(s). Ultimately, the Agent will send the samples and additional information over the network as a UDP datagram, to an sFlow Collector for further processing.

sFlow has been specifically designed to take advantages of the statistical properties of packet sampling and can be modeled using statistical sampling theory. This means that the sFlow traffic monitoring system will always produce statistically quantifiable measurements. You can read more about it in Peter Phaal and Sonia Panchen’s [paper ], I certainly did and my head spun a little bit at the math :)

sFlow: Netlink PSAMPLE

sFlow is meant to be a very lightweight operation for the sampling equipment. It can typically be done in hardware, but there also exist several software implementations. One very clever thing, I think, is decoupling the sampler from the rest of the Agent. The Linux kernel has a packet sampling API called [PSAMPLE ], which allows producers to send samples to a certain group, and then allows consumers to subscribe to samples of a certrain group. The PSAMPLE API uses [NetLink ] under the covers. The cool thing, for me anyway, is that I have a little bit of experience with Netlink due to my work on VPP’s [Linux Control Plane ] plugin.

The idea here is that some sFlow Agent, notably a VPP plugin, will be taking periodic samples from the physical network interfaces, and producing Netlink messages. Then, some other program, notably outside of VPP, can consume these messages and further handle them, creating UDP packets with sFlow samples and counters and other information, and sending them to an sFlow Collector somewhere else on the network.

There’s a handy utility called [psampletest ] which can subscribe to these PSAMPLE netlink groups and retrieve the samples. The first time I used all of this stuff, I wasn’t aware of this utility and I kept on getting errors. It turns out, there’s a kernel module that needs to be loaded: modprobe psample and psampletest helpfully does that for you [ref ], so just make sure the module is loaded and added to /etc/modules before you spend as many hours as I did pulling out hair.

VPP: sFlow Plugin

For the purposes of my initial testing, I’ll simply take a look at Neil’s prototype on [GitHub ] and see what I learn in terms of functionality and performance.

sFlow Plugin: Anatomy

The design is purposefully minimal, to do all of the heavy lifting outside of the VPP dataplane. The plugin will create a new VPP graph node called sflow, which the operator can insert after device-input, in other words, if enabled, the plugin will get a copy of all packets that are read from an input provider, such as dpdk-input or rdma-input. The plugin’s job is to process the packet, and if it’s not selected for sampling, just move it onwards to the next node, typically ethernet-input. Almost all of the interesting action is in node.c

The kicker is, that one in N packets will be selected to sample, after which:

1. the ethernet header (*en) is extracted from the packet
2. the input interface (hw_if_index) is extracted from the VPP buffer. Remember, sFlow works with physical network interfaces!
3. if there are too many samples from this worker thread being worked on, it is discarded and an error counter is incremented. This protects the main thread from being slammed with samples if there are simply too many being fished out of the dataplane.
4. Otherwise: a new sflow_sample_t is created, with all the sampling process metadata filled in the first 128 bytes of the packet are copied into the sample an RPC is dispatched to the main thread, which will send the sample to the PSAMPLE channel

Both a debug CLI command and API call are added:

sflow enable-disable <interface-name> [<sampling_N>]|[disable]        

Some observations:

First off, the sampling_N in Neil’s demo is a global rather than per-interface setting. It would make sense to make this be per-interface, as routers typically have a mixture of 1G/10G and faster 100G network cards available. It was a surprise when I set one interface to 1:1000 and the other to 1:10000 and then saw the first interface change its sampling rate also. It’s a small thing, and will not be an issue to change.

Secondly, sending the RPC to main uses vl_api_rpc_call_main_thread(), which requires a spinlock in src/vlibmemory/memclnt_api.c:649. I’m somewhat worried that when many samples are sent from many threads, there will be lock contention and performance will suffer.

sFlow Plugin: Functional

I boot up the [IPng Lab ] and install a bunch of sFlow tools on it, make sure the psample kernel module is loaded. In this first test I’ll take a look at tablestakes. I compile VPP with the sFlow plugin, and enable that plugin in startup.conf on each of the four VPP routers. For reference, the Lab looks like this:

What I’ll do is start an iperf3 server on vpp0-3 and then hit it from vpp0-0, to generate a few TCP traffic streams back and forth, which will be traversing vpp0-2 and vpp0-1, like so:

pim@vpp0-3:~ $ iperf3 -s -D
pim@vpp0-0:~ $ iperf3 -c vpp0-3.lab.ipng.ch -t 86400 -P 10 -b 10M        

Configuring VPP for sFlow

While this iperf3 is running, I’ll log on to vpp0-2 to take a closer look. The first thing I do, is turn on packet sampling on vpp0-2’s interface that points at vpp0-3, which is Gi10/0/1, and the interface that points at vpp0-0, which is Gi10/0/0. That’s easy enough, and I will use a sampling rate of 1:1000 as these interfaces are GigabitEthernet:

root@vpp0-2:~# vppctl sflow enable-disable GigabitEthernet10/0/0 1000
root@vpp0-2:~# vppctl sflow enable-disable GigabitEthernet10/0/1 1000
root@vpp0-2:~# vppctl show run | egrep '(Name|sflow)'
       Name         State       Calls       Vectors  Suspends      Clocks     Vectors/Call  
sflow              active        5656         24168         0      9.01e2     4.27        

Nice! VPP inserted the sflow node between dpdk-input and ethernet-input where it can do its business. But is it sending data? To answer this question, I can first take a look at the psampletest tool:

root@vpp0-2:~# psampletest 
pstest: modprobe psample returned 0
pstest: netlink socket number = 1637
pstest: getFamily
pstest: generic netlink CMD = 1
pstest: generic family name: psample
pstest: generic family id: 32
pstest: psample attr type: 4 (nested=0) len: 8
pstest: psample attr type: 5 (nested=0) len: 8
pstest: psample attr type: 6 (nested=0) len: 24
pstest: psample multicast group id: 9
pstest: psample multicast group: config
pstest: psample multicast group id: 10
pstest: psample multicast group: packets
pstest: psample found group packets=10
pstest: joinGroup 10
pstest: received Netlink ACK
pstest: joinGroup 10
pstest: set headers...
pstest: serialize...
pstest: print before sending...
pstest: psample netlink (type=32) CMD = 0
pstest: grp=1 in=7 out=9 n=1000 seq=1 pktlen=1514 hdrlen=31 pkt=0x558c08ba4958 q=3 depth=33333333 delay=123456
pstest: send...
pstest: send_psample getuid=0 geteuid=0
pstest: sendmsg returned 140
pstest: free...
pstest: start read loop...
pstest: psample netlink (type=32) CMD = 0
pstest: grp=1 in=1 out=0 n=1000 seq=600320 pktlen=2048 hdrlen=132 pkt=0x7ffe0e4776c8 q=0 depth=0 delay=0
pstest: psample netlink (type=32) CMD = 0
pstest: grp=1 in=1 out=0 n=1000 seq=600321 pktlen=2048 hdrlen=132 pkt=0x7ffe0e4776c8 q=0 depth=0 delay=0
pstest: psample netlink (type=32) CMD = 0
pstest: grp=1 in=1 out=0 n=1000 seq=600322 pktlen=2048 hdrlen=132 pkt=0x7ffe0e4776c8 q=0 depth=0 delay=0
pstest: psample netlink (type=32) CMD = 0
pstest: grp=1 in=2 out=0 n=1000 seq=600423 pktlen=66 hdrlen=70 pkt=0x7ffdb0d5a1e8 q=0 depth=0 delay=0
pstest: psample netlink (type=32) CMD = 0
pstest: grp=1 in=1 out=0 n=1000 seq=600324 pktlen=2048 hdrlen=132 pkt=0x7ffe0e4776c8 q=0 depth=0 delay=0        

I am amazed! The psampletest output shows a few packets, considering I’m asking iperf3 to push 100Mbit using 9000 byte jumboframes (which would be something like 1400 packets/second), I can expect two or three samples per second. I immediately notice a few things:

1. Network Namespace: The Netlink sampling channel belongs to a network namespace. The VPP process is running in the default netns, so its PSAMPLE netlink messages will be in that namespace. Thus, the psampletest and other tools must also run in that namespace. I mention this because in Linux CP, often times the controlplane interfaces are created in a dedicated dataplane network namespace.

2. pktlen and hdrlen: The pktlen is wrong, and this is a bug. In VPP, packets are put into buffers of size 2048, and if there is a jumboframe, that means multiple buffers are concatenated for the same packet. The packet length here ought to be 9000 in one direction. Looking at the in=2 packet with length 66, that looks like a legitimate ACK packet on the way back. But why is the hdrlen set to 70 there? I’m going to want to ask Neil about that.

3. ingress and egress: The in=1 and one packet with in=2 represent the input hw_if_index which is the ifIndex that VPP assigns to its devices. And looking at show interfaces, indeed number 1 corresponds with GigabitEthernet10/0/0 and 2 is GigabitEthernet10/0/1, which checks out:

root@vpp0-2:~# vppctl show int
              Name          Idx    State  MTU (L3/IP4/IP6/MPLS)     Counter          Count     
GigabitEthernet10/0/0         1      up          9000/0/0/0     rx packets             469552764
                                                                rx bytes           4218754400233
                                                                tx packets             133717230
                                                                tx bytes              8887341013
                                                                drops                       6050
                                                                ip4                    469321635
                                                                ip6                       225164
GigabitEthernet10/0/1         2      up          9000/0/0/0     rx packets             133527636
                                                                rx bytes              8816920909
                                                                tx packets             469353481
                                                                tx bytes           4218736200819
                                                                drops                       6060
                                                                ip4                    133489925
                                                                ip6                        29139
        

4. ifIndexes are orthogonal: These in=1 or in=2 ifIndex numbers are constructs of the VPP dataplane. Notably, VPP’s numbering of interface index is strictly orthogonal to Linux, and it’s not guaranteed that there even exists an interface in Linux for the PHY upon which the sampling is happening. Said differently, in=1 here is meant to reference VPP’s GigabitEthernet10/0/0 interface, but in Linux, ifIndex=1 is a completely different interface (lo) in the default network namespace. Similarly in=2 for VPP’s Gi10/0/1 interface corresponds to interface enp1s0 in Linux:

root@vpp0-2:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:f0:01:20 brd ff:ff:ff:ff:ff:ff
        

5. Counters: sFlow periodically polls the interface counters for all interfaces. It will normally use /proc/net/ entries for that, but there are two problems with this:

1. There may not exist a Linux representation of the interface, for example if it’s only doing L2 bridging or cross connects in the VPP dataplane, and it does not have a Linux Control Plane interface, or linux-cp is not used at all.
2. Even if it does exist and it’s the “correct” ifIndex in Linux, for example if the Linux Interface Pair’s tuntap host_vif_index index is used, even then the statistics counters in the Linux representation will only count packets and octets of punted packets, that is to say, the stuff that LinuxCP has decided need to go to the Linux kernel through the TUN/TAP device. Important to note that east-west traffic that goes through the dataplane, is never punted to Linux, and as such, the counters will be undershooting: only counting traffic to the router, not through the router.

VPP sFlow: Performance

Now that I’ve shown that Neil’s proof of concept works, I will take a better look at the performance of the plugin. I’ve made a mental note that the plugin sends RPCs from worker threads to the main thread to marshall the PSAMPLE messages out. I’d like to see how expensive that is, in general. So, I pull boot two Dell R730 machines in IPng’s Lab and put them to work. The first machine will run Cisco’s T-Rex loadtester with 8x 10Gbps ports (4x dual Intel 58299), while the second (identical) machine will run VPP also ith 8x 10Gbps ports (2x Intel i710-DA4).

I will test a bunch of things in parallel. First off, I’ll test L2 (xconnect) and L3 (IPv4 routing), and secondly I’ll test that with and without sFlow turned on. This gives me 8 ports to configure, and I’ll start with the L2 configuration, as follows:

vpp# set int state TenGigabitEthernet3/0/2 up
vpp# set int state TenGigabitEthernet3/0/3 up
vpp# set int state TenGigabitEthernet130/0/2 up
vpp# set int state TenGigabitEthernet130/0/3 up
vpp# set int l2 xconnect TenGigabitEthernet3/0/2 TenGigabitEthernet3/0/3
vpp# set int l2 xconnect TenGigabitEthernet3/0/3 TenGigabitEthernet3/0/2
vpp# set int l2 xconnect TenGigabitEthernet130/0/2 TenGigabitEthernet130/0/3
vpp# set int l2 xconnect TenGigabitEthernet130/0/3 TenGigabitEthernet130/0/2
        

Then, the L3 configuration looks like this:

vpp# lcp create TenGigabitEthernet3/0/0 host-if xe0-0
vpp# lcp create TenGigabitEthernet3/0/1 host-if xe0-1
vpp# lcp create TenGigabitEthernet130/0/0 host-if xe1-0
vpp# lcp create TenGigabitEthernet130/0/1 host-if xe1-1
vpp# set int state TenGigabitEthernet3/0/0 up
vpp# set int state TenGigabitEthernet3/0/1 up
vpp# set int state TenGigabitEthernet130/0/0 up
vpp# set int state TenGigabitEthernet130/0/1 up
vpp# set int ip address TenGigabitEthernet3/0/0 100.64.0.1/31
vpp# set int ip address TenGigabitEthernet3/0/1 100.64.1.1/31
vpp# set int ip address TenGigabitEthernet130/0/0 100.64.4.1/31
vpp# set int ip address TenGigabitEthernet130/0/1 100.64.5.1/31
vpp# ip route add 16.0.0.0/24 via 100.64.0.0
vpp# ip route add 48.0.0.0/24 via 100.64.1.0
vpp# ip route add 16.0.2.0/24 via 100.64.4.0
vpp# ip route add 48.0.2.0/24 via 100.64.5.0
vpp# ip neighbor TenGigabitEthernet3/0/0 100.64.0.0 00:1b:21:06:00:00 static
vpp# ip neighbor TenGigabitEthernet3/0/1 100.64.1.0 00:1b:21:06:00:01 static
vpp# ip neighbor TenGigabitEthernet130/0/0 100.64.4.0 00:1b:21:87:00:00 static
vpp# ip neighbor TenGigabitEthernet130/0/1 100.64.5.0 00:1b:21:87:00:01 static
        

And finally, the Cisco T-Rex configuration looks like this:

- version: 2
  interfaces: [ '06:00.0', '06:00.1', '83:00.0', '83:00.1', '87:00.0', '87:00.1', '85:00.0', '85:00.1' ]
  port_info:
    - src_mac:  00:1b:21:06:00:00
      dest_mac: 9c:69:b4:61:a1:dc
    - src_mac:  00:1b:21:06:00:01
      dest_mac: 9c:69:b4:61:a1:dd

    - src_mac:  00:1b:21:83:00:00 
      dest_mac: 00:1b:21:83:00:01
    - src_mac:  00:1b:21:83:00:01
      dest_mac: 00:1b:21:83:00:00

    - src_mac:  00:1b:21:87:00:00
      dest_mac: 9c:69:b4:61:75:d0
    - src_mac:  00:1b:21:87:00:01
      dest_mac: 9c:69:b4:61:75:d1

    - src_mac:  9c:69:b4:85:00:00
      dest_mac: 9c:69:b4:85:00:01
    - src_mac:  9c:69:b4:85:00:01
      dest_mac: 9c:69:b4:85:00:00        

A little note on the use of ip neighbor in VPP and specific dest_mac in T-Rex. In L2 mode, because the VPP interfaces will be in promiscuous mode and simply pass through any ethernet frame received on interface Te3/0/2 and copy it out on Te3/0/3 and vice-versa, there is no need to tinker with MAC addresses. But in L3 mode, the NIC will only accept ethernet frames addressed to its MAC address, so you can see that for the first port in T-Rex, I am setting dest_mac: 9c:69:b4:61:a1:dc which is the MAC address of Te3/0/0 on VPP. And then on the way out, if VPP wants to send traffic back to T-Rex, I’ll give it a static ARP entry with ip neighbor .. static.

With that said, I can start a baseline loadtest like so:

T-Rex is sending 10Gbps out on all eight interfaces (four of which are L3 routing, and four of which are L2 xconnecting), using packet size of 1514 bytes. This amounts of roughlu 813Kpps per port, or a cool 6.51Mpps in total. And I can see, in this base line configuration, the VPP router is happy to do the work.

I then enable sFlow on the second set of four ports, using a 1:1000 sampling rate:

pim@hvn6-lab:~$ vppctl sflow enable-disable TenGigabitEthernet130/0/0 1000
pim@hvn6-lab:~$ vppctl sflow enable-disable TenGigabitEthernet130/0/1 1000
pim@hvn6-lab:~$ vppctl sflow enable-disable TenGigabitEthernet130/0/2 1000
pim@hvn6-lab:~$ vppctl sflow enable-disable TenGigabitEthernet130/0/3 1000        

This should yield about 3'250 or so samples per second, and things look pretty great:

pim@hvn6-lab:~$ vppctl show err
   Count                  Node                              Reason               Severity 
   5034508               sflow                     sflow packets processed         error  
      4908               sflow                      sflow packets sampled          error  
   5034508               sflow                     sflow packets processed         error  
      5111               sflow                      sflow packets sampled          error  
   5034516             l2-output                      L2 output packets            error  
   5034516              l2-input                       L2 input packets            error  
   5034404               sflow                     sflow packets processed         error  
      4948               sflow                      sflow packets sampled          error  
   5034404             l2-output                      L2 output packets            error  
   5034404              l2-input                       L2 input packets            error  
   5034404               sflow                     sflow packets processed         error  
      4928               sflow                      sflow packets sampled          error  
   5034404             l2-output                      L2 output packets            error  
   5034404              l2-input                       L2 input packets            error  
   5034516             l2-output                      L2 output packets            error  
   5034516              l2-input                       L2 input packets            error          

I can see that the sflow packets sampled is roughly 0.1% of the sflow packets processed which checks out. I can also see in psampletest a flurry of activity, so I’m happy:

pim@hvn6-lab:~$ sudo psampletest 
...
pstest: grp=1 in=9 out=0 n=1000 seq=63388 pktlen=1510 hdrlen=132 pkt=0x7ffd9e786158 q=0 depth=0 delay=0
pstest: psample netlink (type=32) CMD = 0
pstest: grp=1 in=8 out=0 n=1000 seq=63389 pktlen=1510 hdrlen=132 pkt=0x7ffd9e786158 q=0 depth=0 delay=0
pstest: psample netlink (type=32) CMD = 0
pstest: grp=1 in=11 out=0 n=1000 seq=63390 pktlen=1510 hdrlen=132 pkt=0x7ffd9e786158 q=0 depth=0 delay=0
pstest: psample netlink (type=32) CMD = 0
pstest: grp=1 in=10 out=0 n=1000 seq=63391 pktlen=1510 hdrlen=132 pkt=0x7ffd9e786158 q=0 depth=0 delay=0
pstest: psample netlink (type=32) CMD = 0
pstest: grp=1 in=11 out=0 n=1000 seq=63392 pktlen=1510 hdrlen=132 pkt=0x7ffd9e786158 q=0 depth=0 delay=0        

I confirm that all four in interfaced (8, 9, 10 and 11) are sending samples, and those indexes correctly correspond to the VPP dataplane’s sw_if_index for TenGig130/0/0 - 3. Sweet! On this machine, each TenGig network interface has its own dedicated VPP worker thread. Considering I turned on sFlow sampling on four interfaces, I should see the cost I’m paying for the feature:

pim@hvn6-lab:~$ vppctl show run | grep -e '(Name|sflow)'
Name                 State         Calls       Vectors Suspends      Clocks     Vectors/Call  
sflow               active       3908218      14350684        0      9.05e1     3.67
sflow               active       3913266      14350680        0      1.11e2     3.67
sflow               active       3910828      14350687        0      1.08e2     3.67
sflow               active       3909274      14350692        0      5.66e1     3.67        

Alright, so for the 999 packets that went through and the one packet that got sampled, on average VPP is spending between 90 and 111 CPU cycles per packet, and the loadtest looks squeaky clean on T-Rex.

VPP sFlow: Cost of passthru

I decide to take a look at two edge cases. What if there are no samples being taken at all, and the sflow node is merely passing through all packets to ethernet-input? To simulate this, I will set up a bizarrely high sampling rate, say one in ten million. I’ll also make the T-Rex loadtester use only four ports, in other words, a unidirectional loadtest, and I’ll make it go much faster by sending smaller packets, say 128 bytes:

tui>start -f stl/ipng.py -p 0 2 4 6 -m 99% -t size=128

pim@hvn6-lab:~$ vppctl sflow enable-disable TenGigabitEthernet130/0/0 1000 disable
pim@hvn6-lab:~$ vppctl sflow enable-disable TenGigabitEthernet130/0/1 1000 disable
pim@hvn6-lab:~$ vppctl sflow enable-disable TenGigabitEthernet130/0/2 1000 disable
pim@hvn6-lab:~$ vppctl sflow enable-disable TenGigabitEthernet130/0/3 1000 disable

pim@hvn6-lab:~$ vppctl sflow enable-disable TenGigabitEthernet130/0/0 10000000
pim@hvn6-lab:~$ vppctl sflow enable-disable TenGigabitEthernet130/0/2 10000000        

The loadtester is now sending 33.5Mpps or thereabouts (4x 8.37Mpps), and I can confirm that the sFlow plugin is not sampling many packets:

pim@hvn6-lab:~$ vppctl show err
   Count                  Node                              Reason               Severity 
  59777084               sflow                     sflow packets processed         error  
         6               sflow                      sflow packets sampled          error  
  59777152             l2-output                      L2 output packets            error  
  59777152              l2-input                       L2 input packets            error  
  59777104               sflow                     sflow packets processed         error  
         6               sflow                      sflow packets sampled          error  
  59777104             l2-output                      L2 output packets            error  
  59777104              l2-input                       L2 input packets            error  

pim@hvn6-lab:~$ vppctl show run | grep -e '(Name|sflow)'
Name                 State         Calls       Vectors Suspends      Clocks     Vectors/Call  
sflow                active      8186642     369674664        0      1.35e1     45.16
sflow                active     25173660     369674696        0      1.97e1     14.68        

Two observations:

1. One of these is busier than the other. Without looking further, I can already predict that the top one (doing 45.16 vectors/call) is the L3 thread. Reasoning: the L3 code path through the dataplane is a lot more expensive than ‘merely’ L2 XConnect. As such, the packets will spend more time, and therefore the iterations of the dpdk-input loop will be further apart in time. And because of that, it’ll end up consuming more packets on each subsequent iteration, in order to catch up. The L2 path on the other hand, is quicker and therefore will have less packets waiting on subsequent iterations of dpdk-input.
2. The sflow plugin spends between 13.5 and 19.7 CPU cycles shoveling the packets into ethernet-input without doing anything to them. That’s pretty low! And the L3 path is a little bit more efficient per packet, which is very likely because it gets to amort its L1/L2 CPU instruction cache over 45 packets each time it runs, while the L2 path can only amort its instruction cache over 15 or so packets each time it runs.

I let the loadtest run overnight,and the proof is in the pudding: sFlow enabled but not sampling works just fine:

VPP sFlow: Cost of sampling

The other interesting case is to figure out how much CPU it takes to execute the code path with the actual sampling. This one turns out a bit trickier to measure. While leaving the previous loadtest running at 33.5Mpps, I disable sFlow and then re-enable it at an abnormally high ratio of 1:10 packets:

pim@hvn6-lab:~$ vppctl sflow enable-disable TenGigabitEthernet130/0/0 disable
pim@hvn6-lab:~$ vppctl sflow enable-disable TenGigabitEthernet130/0/2 disable
pim@hvn6-lab:~$ vppctl sflow enable-disable TenGigabitEthernet130/0/0 10
pim@hvn6-lab:~$ vppctl sflow enable-disable TenGigabitEthernet130/0/2 10        

The T-Rex view immediately reveals that VPP is not doing very well, as the throughput went from 33.5Mpps all the way down to 7.5Mpps. Ouch! Looking at the dataplane:

pim@hvn6-lab:~$ vppctl show err | grep sflow
 340502528               sflow                     sflow packets processed         error  
  12254462               sflow                      sflow packets dropped          error  
  22611461               sflow                      sflow packets sampled          error  
 422527140               sflow                     sflow packets processed         error  
   8533855               sflow                      sflow packets dropped          error  
  34235952               sflow                      sflow packets sampled          error          

Ha, this new safeguard popped up: remember all the way at the beginning, I explained how there’s a safety net in the sflow plugin that will pre-emptively drop the sample if the RPC channel towards the main thread is seeing too many outstanding RPCs? That’s happening right now, under the moniker sflow packets dropped, and it’s roughly half of the samples.

My first attempt is to back off the loadtester to roughly 1.5Mpps per port (so 6Mpps in total, under the current limit of 7.5Mpps), but I’m disappointed: the VPP instance is now returning 665Kpps per port only, which is horrible, and it’s still dropping samples.

My second attempt is to turn off all ports but last pair (the L2XC port), which returns 930Kpps from the offered 1.5Mpps. VPP is clearly not having a good time here.

Finally, as a validation, I turn off all ports but the first pair (the L3 port, without sFlow), and ramp up the traffic to 8Mpps. Success (unsurprising to me). I also ramp up the second pair (the L2XC port, without sFlow), VPP forwards all 16Mpps and is happy again.

Once I turn on the third pair (the L3 port, with sFlow), even at 1Mpps, the whole situation regresses again: First two ports go down from 8Mpps to 5.2Mpps each; the third (offending) port delivers 740Kpps out of 1Mpps. Clearly, there’s some work to do under high load situations!

Reasoning about the bottle neck

But how expensive is sending samples, really? To try to get at least some pseudo-scientific answer I turn off all ports again, and ramp up the one port pair with (L3 + sFlow at 1:10 ratio) to full line rate: that is 64 byte packets at 14.88Mpps:

tui>stop
tui>start -f stl/ipng.py -m 100% -p 4 -t size=64        

VPP is now on the struggle bus and is returning 3.16Mpps or 21% of that. But, I think it’ll give me some reasonable data to try to feel out where the bottleneck is.

Thread 2 vpp_wk_1 (lcore 3)
Time 6.3, 10 sec internal node vector rate 256.00 loops/sec 27310.73
  vector rates in 3.1607e6, out 3.1607e6, drop 0.0000e0, punt 0.0000e0
             Name                 State     Calls    Vectors   Suspends   Clocks   Vectors/Call  
TenGigabitEthernet130/0/1-outp   active     77906   19943936          0   5.79e0         256.00
TenGigabitEthernet130/0/1-tx     active     77906   19943936          0   6.88e1         256.00
dpdk-input                       polling    77906   19943936          0   4.41e1         256.00
ethernet-input                   active     77906   19943936          0   2.21e1         256.00
ip4-input                        active     77906   19943936          0   2.05e1         256.00
ip4-load-balance                 active     77906   19943936          0   1.07e1         256.00
ip4-lookup                       active     77906   19943936          0   1.98e1         256.00
ip4-rewrite                      active     77906   19943936          0   1.97e1         256.00
sflow                            active     77906   19943936          0   6.14e1         256.00

pim@hvn6-lab:pim# vppctl show err | grep sflow
 551357440               sflow                     sflow packets processed         error  
  19829380               sflow                      sflow packets dropped          error  
  36613544               sflow                      sflow packets sampled          error  
        

OK, the sflow plugin saw 551M packets, selected 36.6M of them for sampling, but ultimately only sent RPCs to the main thread for 16.8M samples after having dropped 19.8M of them. There are three code paths, each one extending the other:

1. Super cheap: pass through. I already learned that it takes about X=13.5 CPU cycles to pass through a packet.
2. Very cheap: select sample and construct the RPC, but toss it, costing Y CPU cycles.
3. Expensive: select sample, and send the RPC. Z CPU cycles in worker, and another amount in main.

Now I don’t know what Y is, but seeing as the selection only copies some data from the VPP buffer into a new sflow_sample_t, and it uses clip_memcpy_fast() for the sample header, I’m going to assume it’s not drastically more expensive than the super cheap case, so for simplicity I’ll guesstimate that it takes Y=20 CPU cyces.

With that guess out of the way, I can see what the sflow plugin is consuming for the third case:

AvgClocks = (Total * X + Sampled * Y + RPCSent * Z) / Total

61.4 = ( 551357440 * 13.5 + 36613544 * 20 + (36613544-19829380) * Z ) / 551357440
61.4 = ( 7443325440 + 732270880 + 16784164 * Z ) / 551357440
33853346816 = 7443325440 + 732270880 + 16784164 * Z
25677750496 = 16784164 * Z 
Z = 1529
        

Good to know! I find spending O(1500) cycles to send the sample pretty reasonable. However, for a dataplane that is trying to do 10Mpps per core, and a core running 2.2GHz, there are really only 220 CPU cycles to spend end-to-end. Spending an order of magnitude more than that once every ten packets feels dangerous to me.

Here’s where I start my conjecture. If I count the CPU cycles spent in the table above, I will see 273 CPU cycles spent on average per packet. The CPU in the VPP router is an E5-2696 v4 @ 2.20GHz, which means it should be able to do 2.2e10/273 = 8.06Mpps per thread, more than double that what I observe (3.16Mpps)! But, for all the vector rates in (3.1607e6), it also managed to emit the packets back out (same number: 3.1607e6).

So why isn’t VPP getting more packets from DPDK? I poke around a bit and find an important clue:

pim@hvn6-lab:~$ vppctl show hard TenGigabitEthernet130/0/0 | grep rx\ missed; \
                sleep 10; vppctl show hard TenGigabitEthernet130/0/0 | grep rx\ missed
    rx missed                                     4065539464
    rx missed                                     4182788310
        

In those ten seconds, VPP missed (4182788310-4065539464)/10 = 11.72Mpps. I already measured that it forwarded 3.16Mpps and you know what? 11.7 + 3.16 is precisely 14.88Mpps. All packets are accounted for! It’s just, DPDK never managed to read them from the hardware: sad-trombone.wav

As a validation, I turned off sFlow while keeping that one port at 14.88Mpps. Now, 10.8Mpps were delivered:

Thread 2 vpp_wk_1 (lcore 3)
Time 14.7, 10 sec internal node vector rate 256.00 loops/sec 40622.64
  vector rates in 1.0794e7, out 1.0794e7, drop 0.0000e0, punt 0.0000e0
             Name                 State     Calls    Vectors   Suspends   Clocks   Vectors/Call  
TenGigabitEthernet130/0/1-outp   active    620012  158723072          0   5.66e0         256.00
TenGigabitEthernet130/0/1-tx     active    620012  158723072          0   7.01e1         256.00
dpdk-input                       polling   620012  158723072          0   4.39e1         256.00
ethernet-input                   active    620012  158723072          0   1.56e1         256.00
ip4-input-no-checksum            active    620012  158723072          0   1.43e1         256.00
ip4-load-balance                 active    620012  158723072          0   1.11e1         256.00
ip4-lookup                       active    620012  158723072          0   2.00e1         256.00
ip4-rewrite                      active    620012  158723072          0   2.02e1         256.00        

Total Clocks: 201 per packet; 2.2GHz/201 = 10.9Mpps, and I am observing 10.8Mpps. As [North of the Border ] would say: “That’s not just good, it’s good enough!”

For completeness, I turned on all eight ports again, at line rate (8x14.88 = 119Mpps ??), and saw that about 29Mpps of that made it through. Interestingly, what was 3.16Mpps in the single-port line rate loadtest, went up slighty to 3.44Mpps now. What puzzles me even more, is that the non-sFlow worker threads are also impacted. I spent some time thinking about this and poking around, but I did not find a good explanation why port pair 0 (L3, no sFlow) and 1 (L2XC, no sFlow) would be impacted. Here’s a screenshot of VPP on the struggle bus:

Hypothesis: Due to the spinlock in vl_api_rpc_call_main_thread(), the worker CPU is pegged for a longer time, during which the dpdk-input PMD can’t run, so it misses out on these sweet sweet packets that the network card had dutifully received for it, resulting in the rx-miss situation. While VPP’s performance measurement shows 273 CPU cycles per packet and 3.16Mpps, this accounts only for 862M cycles, while the thread has 2200M cycles, leaving a whopping 60% of CPU cycles unused in the dataplane. I still don’t understand why other worker threads are impacted, though.

What’s Next

I’ll continue to work with the folks in the sFlow and VPP communities and iterate on the plugin and other sFlow Agent machinery. In an upcoming article, I hope to share more details on how to tie the VPP plugin in to the hsflowd host sflow daemon in a way that the interface indexes, counters and packet lengths are all correct. Of course, the main improvement that we can make is to allow for the system to work better under load, which will take some thinking.

I should do a few more tests with a debug binary and profiling turned on. I quickly ran a perf over the VPP (release / optimized) binary running on the bench, but it merely said 80% of time was spent in libvlib rather than libvnet in the baseline (sFlow turned off).

root@hvn6-lab:/home/pim# perf record -p 1752441 sleep 10
root@hvn6-lab:/home/pim# perf report --stdio --sort=dso 
# Overhead  Shared Object (sFlow)   Overhead  Shared Object (baseline)
# ........  ......................  ........  ........................
#
    79.02%  libvlib.so.24.10          54.27%  libvlib.so.24.10      
    12.82%  libvnet.so.24.10          33.91%  libvnet.so.24.10      
     3.77%  dpdk_plugin.so            10.87%  dpdk_plugin.so        
     3.21%  [kernel.kallsyms]          0.81%  [kernel.kallsyms]     
     0.29%  sflow_plugin.so            0.09%  ld-linux-x86-64.so.2  
     0.28%  libvppinfra.so.24.10       0.03%  libc.so.6 
     0.21%  libc.so.6                  0.01%  libvppinfra.so.24.10  
     0.17%  libvlibapi.so.24.10        0.00%  libvlibmemory.so.24.10
     0.15%  libvlibmemory.so.24.10     
     0.07%  ld-linux-x86-64.so.2  
     0.00%  vpp                   
     0.00%  [vdso]                
     0.00%  libsvm.so.24.10               

Unfortunately, I’m not much of a profiler expert, being merely a network engineer :) so I may have to ask for help. Of course, if you’re reading this, you may also offer help! There’s lots of interesting work to do on this sflow plugin, with matching ifIndex for consumers like hsflowd, reading interface counters from the dataplane (or from the Prometheus Exporter), and most importantly, ensuring it works well, or fails gracefully, under stringent load.

From the cray-cray ideas department, what if we:

1. In worker thread, produced the sample but instead of sending an RPC to main and taking the lock, append it to a producer sample queue and move on. This way, no locks are needed, and each worker thread will have its own producer queue.
2. Create a separate worker (or even pool of workers), running on possibly a different CPU (or in main), that runs a loop iterating on all sflow sample queues consuming the samples and sending them in batches to the PSAMPLE Netlink group, possibly dropping samples if there are too many coming in.

I’m reminded that this pattern exists already – async crypto workers create a crypto-dispatch node that acts as poller for inbound crypto, and it hands off the result back into the worker thread: lockless at the expense of some complexity!

Acknowledgements

The plugin I am testing here is a prototype written by Neil McKee of inMon. I also wanted to say thanks to Pavel Odintsov of FastNetMon and Ciprian Balaceanu for showing an interest in this plugin, and Peter Phaal for facilitating a get-together last year.

Who’s up for making this thing a reality?!