FreeBSD 13 & TCP BBR Congestion Control

Finally TCP BBR is available for FreeBSD new release 13.x. Has been more than 2 years since the announcement that at some point in the future we will be able to use TCP BBR within FreeBSD environment.

However the main question is, What is TCP BBR?

According to Google........

BBR ("Bottleneck?Bandwidth and?Round-trip propagation time") is a new congestion control algorithm developed at Google. Congestion control algorithms?—?running inside every computer, phone or tablet connected to a network?—?that decide how fast to send data.

How does a congestion control algorithm make this decision? The internet has largely used?loss-based congestion control?since the late 1980s, relying only on indications of lost packets as the signal to slow down. This worked well for many years, because internet switches’ and routers’ small buffers were well-matched to the low bandwidth of internet links. As a result, buffers tended to fill up and drop excess packets right at the moment when senders had really begun sending data too fast.

But loss-based congestion control is problematic in today's diverse networks:

• In shallow buffers, packet loss happens before congestion. With today's high-speed, long-haul links that use commodity switches with shallow buffers, loss-based congestion control can result in abysmal throughput because it overreacts, halving the sending rate upon packet loss, even if the packet loss comes from transient traffic bursts (this kind of packet loss can be quite frequent even when the link is mostly idle).
• In deep buffers, congestion happens before packet loss. At the edge of today's internet, loss-based congestion control causes the infamous “bufferbloat” problem, by repeatedly filling the deep buffers in many last-mile links and causing seconds of needless queuing delay.

We need an algorithm that responds to actual congestion, rather than packet loss. BBR tackles this with a ground-up rewrite of congestion control. We started from scratch, using a completely new paradigm: to decide how fast to send data over the network, BBR considers how fast the network is delivering data. For a given network connection, it uses recent measurements of the network's delivery rate and round-trip time to build an explicit model that includes both the maximum recent bandwidth available to that connection, and its minimum recent round-trip delay. BBR then uses this model to control both how fast it sends data and the maximum amount of data it's willing to allow in the network at any time.

TCP BBR it uses recent measurements of the network’s delivery rate and round-trip time to build an explicit model that includes both the maximum recent bandwidth available to that connection and its minimum recent round-trip delay. BBR then uses this model to control both how fast it sends data and the maximum amount of data it’s willing to allow in the network at any time.

Thus utilising the maximum bandwidth of the client,

• More widely adopted all over globally, AWS GCP and CDN’s like Fastly etc.
• CDN’s, Load balancers use TCP BBR to deliver content efficiency, which shows better metrics than earlier.

In this article I will describe how to update an existing FreeBSD to 13 and how to enable TCP BBR (just remember to backup your data in advance).

Upgrade FreeBSD

The FreeBSD update tool supports binary upgrades of i386 and amd64 systems.

Run the following commands to fetch and install the new updates:

# freebsd-update fetch
# freebsd-update install        

The FreeBSD update tool can fetch bits belonging to 13 Release and during the process will may ask to merge configuration files:

# freebsd-update upgrade -r 13.0-RELEAS
# freebsd-update install        

The system must now be rebooted with the newly installed kernel before the non-kernel components are updated:

# shutdown -r now        

After rebooting, the FreeBSD update tool needs to be run again to install the new userland components:

# freebsd-update install        

At this point will require to re-install all third party applications (for example Emacs or other apps which were installed via ports) due to updates in the system libraries

After updating installed third-party applications, run the FreeBSD update tool?again so that it can delete the old (no longer used) system libraries:

# freebsd-update install        

Finally, reboot into 13.0-RELEASE

# shutdown -r now        

Now you should see the following version:

$ freebsd-version

13.0-RELEASE-p11        

Compile New Kernel

Now we are ready to compile the new kernel to activate the TCP BBR.

Create a new file RACK (you can use any name you want) in the folder /usr/src/sys/amd64/conf/RACK. Inside the file will need to add the options for TCP BBR and the file should look like this:

$ cat /usr/src/sys/amd64/conf/RACK

include GENERIC

ident RACK

makeoptions WITH_EXTRA_TCP_STACKS=1

options RATELIMIT

options TCPHPTS        

Next step is to run the following commands (in order) to compile the kernel (this step will take a while)

1) make -j 16 KERNCONF=RACK buildkernel

2) make installkernel KERNCONF=RACK KODIR=/boot/kernel.rack

3) reboot -k kernel.rack        

The old kernel will be available but with the name "kernel.old". After rebooting, will use the new kernel because of the command "reboot -k kernel.rack", however to make it persistent will require to adjust couple of files (will explain later in this article).

Once you have built, installed and rebooted to the new kernel we need to load the RACK kernel module?tcp_bbr.ko:

kldload /boot/kernel.rack/tcp_bbr.ko?        

Now you should see the new module in the functions_available report, by typing the command:

sysctl net.inet.tcp.functions_available        

The output will be:

net.inet.tcp.functions_available

Stack ? ? ? ? ? ? ? ? ? ? ? ? ? D Alias? ? ? ? ? ? ? ? ? ? ? ? ? ? PCB count

freebsd ? ? ? ? ? ? ? ? ? ? ? ? * freebsd? ? ? ? ? ? ? ? ? ? ? ? ? 3

bbr? ? ? ? ? ? ? ? ? ? ? ? ? ? ? bbr ? ? ? ?   ? ? ? ? ? ? ? ? ? ? 0:        

Now will require to change the default to TCP BBR:

sysctl net.inet.tcp.functions_default=bbr        

and the output will be:

net.inet.tcp.functions_default: freebsd -> bbr

root@freebsd # sysctl net.inet.tcp.functions_available

net.inet.tcp.functions_available:

Stack ? ? ? ? ? ? ? ? ? ? ? ? ? D Alias? ? ? ? ? ? ? ? ? ? ? ? ? ? PCB count

freebsd ? ? ? ? ? ? ? ? ? ? ? ? ? freebsd? ? ? ? ? ? ? ? ? ? ? ? ? 3

bbr? ? ? ? ? ? ? ? ? ? ? ? ? ? * bbr   ? ? ? ? ? ? ? ? ? ? ? ? ? ? 0k        

After rebooting, will use the old Kernel, but we can make it persistent.

Modify the Loader

To force FreeBSD to use the new Kernel after rebooting, will require to adjust 3 files:

• /etc/sysctl.conf
• /etc/rc.conf
• /boot/loader.conf

Inside /etc/sysctl.conf we can also add command for optimisation, including the command to enable TCP BBR as a default congestion control function.

The file should looks like this:

$ cat /etc/sysctl.conf

# $FreeBSD$

#

#? This file is read when going to multi-user and its contents piped thru

#? ``sysctl'' to adjust kernel values.? ``man 5 sysctl.conf'' for details.

#




# Uncomment this to prevent users from seeing information about processes that

# are being run under another UID.
        

.bsd.see_other_uids=0

# set to at least 16MB for 10GE hosts

kern.ipc.maxsockbuf=16777216

# set autotuning maximum to at least 16MB too

net.inet.tcp.sendbuf_max=16777216 ?

net.inet.tcp.recvbuf_max=16777216

# enable send/recv autotuning

net.inet.tcp.sendbuf_auto=1

net.inet.tcp.recvbuf_auto=1

# increase autotuning step size?

net.inet.tcp.sendbuf_inc=16384?

net.inet.tcp.recvbuf_inc=524288?

# set this on test/measurement hosts

net.inet.tcp.hostcache.expire=1

# Set congestion control algorithm to Cubic or HTCP

# Make sure the module is loaded at boot time - check loader.conf

# net.inet.tcp.cc.algorithm=cubic ?

net.inet.tcp.cc.algorithm=htcp

net.inet.tcp.functions_default=bbr

net.inet.tcp.functions_inherit_listen_socket_stack=0        

The 2nd change is to add the following line inside /etc/rc.conf:

kld_list="/boot/kernel.rack/tcp_bbr.ko"        

and finally the last change is to modify the /boot/loader.conf file, should look like this:

$ cat /boot/loader.conf

###? Basic configuration options? ############################ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 

kernel="kernel.rack" ? ? ? ? # /boot sub-directory containing kernel and modules ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
bootfile="kernel.rack" ? ? ? # Kernel name (possibly absolute path)

module_path="/boot/kernel.rack"	# Set the module search path

cc_htcp_load="YES"        

After modifying the files, reboot the server and you should see the HTCP algorithm as well as TCP BBR function as the chosen options:

$ sudo ? sysctl net.inet.tcp.cc.available

net.inet.tcp.cc.available:?

CCmod ? ? ? ? ? D PCB count

newreno ? ? ? ? ? 0

htcp? ? ? ? ? ? * 6




$ sudo sysctl net.inet.tcp.functions_available

net.inet.tcp.functions_available:?

Stack ? ? ? ? ? ? ? ? ? ? ? ? ? D Alias? ? ? ? ? ? ? ? ? ? ? ? ? ? PCB count

freebsd ? ? ? ? ? ? ? ? ? ? ? ? ? freebsd? ? ? ? ? ? ? ? ? ? ? ? ? 5

bbr ? ? ? ? ? ? ? ? ? ? ? ? ? ? * bbr? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 1        

Enjoy your new FreeBSD with TCP BBR !!!!!!!!!

Also a big thank should go to the FreeBSD community for their hard work !!!!!!!!

#freebsd #tcp #tcp_bbr