OCI File Storage Service Overview

Introduction

In this article we briefly introduce the OCI File System Storage Service (in short FSS) and talk a bit about some best practices when working with it. We will touch the following topics:

• FSS main components and how they relate each other.
• High Availability and Disaster Recovery best practices.
• Best practices for performance.

The aim of this short article is to provide an introduction to the service and to discuss some basic concepts, it is not meant to be an extensive course or to exhaust the matter.

OCI Basic Concepts

It is out of scope to introduce Oracle Cloud Infrastructure (OCI) concepts and services so we will assume a basic knowledge of them. For those not familiar with OCI, the suggestion is to take a look at:

• ???The OCI documentation.
• ???The Live Labs.

File Storage Service

Oracle Cloud Infrastructure offers a File Storage Service (FSS) which is a durable, scalable, secure, enterprise-grade network file system. OCI FSS supplies the following main capabilities:

• It supports the Network File System version 3.0 (NFSv3) protocol.
• It supports the Network Lock Manager (NLM) protocol for file locking functionality.
• Employs 5-way storage replication for resilience and redundancy.
• Protects data with at-rest and in-transit encryption.
• You can find the complete documentation here.

FSS Components

The File Storage Service is composed of 3 logical entities:

1. File System.
2. Export (and Export Path).
3. Mount Target.

When allocating a FSS resource, the File System is the first entity to be created and exists at the Availability Domain (AD) level. It means that all user data in the FSS is stored inside a single AD, in a highly redundant and reliable fashion.

The Export is a logical container inside the File System and it represents a private storage area inside it. That private area is made available by means of a label called Export Path which is used to identify the private storage area when mounting a file system.

The Mount Target exposes the FSS and is attached to a subnet. We can also describe it as the endpoint to which an Export Path is shared with resources in the subnet. The Mount Target too, is created at the AD level and can only be used by FSSs in the same AD. We can consider the Mount Target as the virtual device that exposes the FS to the OCI network layer. The MT and FSS must be in the same AD but once exposed to an OCI subnet the FSS can be reached from resources in different AD. Since different AD are located in different data centers the number of network round trips will differ based on the AD affinity. It follows that resources colocated with the FSS (in the same AD) will have better performance accessing the FSS than resources located in different ADs.

As we said the Export Path identifies a private storage area or container inside the FSS. For example, if an application need two different file systems to be mounted in a compute instance, we can allocate a single FSS and then create and expose two Export Paths.

Lets consider the simple but meaningful use case described in the diagram below:

In this scenario our application needs to mount two file systems as shown in the picture:

To make the example more realistic lets assume we have three life cycle environments:

• Non PROD
• PROD
• DR

We can create a FS for each environment.

• Non_Prod_FS is used by QLTY and DEV compute instances. Usually the DEV environment doesn't need the best performance so it is OK to place its compute instance in AD2 despite the fact that we will need more network round trips to access the FS.
• Prod_FS is used by PROD compute instance.
• Each FS exports two shares per compute instance: share A and share B but ,since the Non PROD environment contains both the QLTY and DEV compute instances, in that case the FS exports 4 shares.
• For Disaster Recovery purpose, PROD_FS is replicated on DR_FS which resides in AD3 and is used by the DR compute instance which has been created in AD3 as well.
• Since the Mount Target resides at the AD level, to mount shares which resides in DR_FS, we need to allocate a mount target in AD3 (DR_MT).

In this use case we setup a simple Disaster Recovery solution, so in case of a disaster affecting AD1, we will failover the application on the DR compute instance which will use the shares on DR_FS which have been kept in sync with the main site. For details about FSS replication please refer to this blog and to the official documentation.

Best Practices For Performance

Mount Targets

It is a best practice to associate a hostname to each Mount Target. This allows to reference the FSS endpoint using the Fully Qualified Domain Name (FQDN). In turn, this allows to keep the same mount strings when the endpoint IPs change. It is important to note that each mount target requires three internal IP addresses in the subnet to function:

• Two of the IP addresses are used during mount target creation. The third IP address must remain available for the mount target to use for high availability failover.
• The third IP?address is used to create a new VNIC for the mount target during failover. The original primary IP?address is retained.
• The File Storage service doesn't "reserve" the third IP address required for high availability failover.
• Use care to ensure that enough unallocated IP?addresses remain available for your mount targets to use during failover.
• Do not use /30 or smaller subnets for mount target creation because they do not have sufficient available IP addresses for mount target creation.

Mount Options

We assume the File System, the Shares and the Mount Target are properly configured and the OCI VCN and subnets have been setup to allow the NFS protocol to work (please see: security rules for FSS). We also assume that the server side (FSS) exports parameter are initilized with the defaults values. The only parameter we set is the Access Control List (ACL) at share level to allow it be mounted only by compute instances in the specific subnet.

When we use a NFS share, from a performance perspective, there are two main metrics we are interested in:

1. Latency.
2. Throughput.

The File System Latency measures the time it takes for a change done by a client on the FS share to be recognized by the server and to be synchronized on all other clients. For example, lets consider an IOT system where many upstream devices are logging on a FS share and some downstream device are collecting and elaborating that data in near real-time. In this scenario we need to have the smallest possible latency because the collecting devices need to detect the changes as soon as possible.

The File System Throughput measures the amount of data we are able to read and/or write from a FS share in the unit of time. Lets consider a batch application that collects a lot of data from a relational database and writes that data into big sequential files to be loaded into a data lake. In this case what we want is have the best throughput both for writing and reading.

Usually there is a trade-off between the two metrics and it is necessary to find the solution that best suits the specific use case. In the typical FSS scenario, we can only leverage the mount options at client level because the options available at server level (FSS share) do not impact on the performance.

Our investigation was limited to Linux. To find the best compromise between throughput and latency we analyzed the documentation for NFS v3 mount options and realized a basic performance test. The test is done mounting the same share on two compute instances in the same AD with the same mounting parameters. We write to the file system a single 1 GB file from one compute instance using this command:

dd if=/dev/urandom of=bigfile.raw bs=100M count=10        

The command itself reports the throughput while the latency is measured observing the changes in the other compute instance.

Assuming a default mount setup on Oracle Linux 8, with this line in /etc/fstab:

mounttargethost.subnet.vcn.oraclevcn.com:/FS-Staging?? ?/FS-Share?? ?nfs?? ?defaults?? ?0?? ?0        

we get the following default values for the parameters:

rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=<IP address>,mountvers=3,mountport=2048,mountproto=udp,local_lock=none,addr=<IP Address>        

Out-of-the-box, so with the default values, we obtain:

• throughput ~ 230 MB/sec
• latency ~ 5 sec

The throughput is not bad but the latency in some situations could be not acceptable especially since the latency gets bigger for smaller files. In this case the high latency is due to the file attribute caching on the writing client which is controlled by the parameter actimeo. The value is hidden but, by default, is some seconds. Setting actimeo=0 we disable the attribute caching and we get:

• Throughput ~ 90 MB/sec
• Latency ~ 0.5 sec

so the throughput is impacted quite a bit .

The optimal balance would be to keep the attribute caching (actimeo) enabled but with a lower timeout (to get lower latency for smaller files). Another useful strategy is to reduce the amount of attributes changes, in other words, the amount of metadata produced on the file system. Reducing the amount of metadata means that we avoid to register some changes in the files. In this case we decided that we can afford to lose the access (read) timestamp for files and directories (it is a fair assumption since applications usually doesn't care about those). Please note that this strategy is beneficial not only for NFS mounted file systems and, in fact, this is generic mount parameter valid for almost all kind file systems available on Linux. Going back to the best values for the parameters, we verified we got the best latency/throughput compromise with the following:

• actimeo=1

This parameters ensures that, when something changes on a client (host mounting the NFS share), the change is reflected on the NFS server and all other clients within one second.

• noatime

This parameter disables the creation of metadata related to the access (read) timestamp for files.

• nodiratime

This parameter disables the creation of metadata related to the access (read) timestamp for directories.

Using those parameters, with the following mount string:

mounttargethost.subnet.vcn.oraclevcn.com:/FS-Staging?? ?/FS-Share?? ?nfs?? ???actimeo=1,noatime,nodiratime?? ?0?? ?0        

we get:

• throughput ~ 210 MB/sec
• latency ~ 1 sec

As last important point, please consider that the latency and the throughput of the NFS mount depends also on the compute instance network bandwidth which for the OCI compute instances depends, in turn, on the number of the allocated OCPUs. For example, if we mount the share on a standard shape compute instance with just one OCPU we'll get a maximum of 1 Gb/sec network bandwidth, which means a maximum throughput of ~ 100 MB/sec. To get meaningful results for our performance test we had to use two standard shape instances with at least 6 OCPU so with a maximum theoretical network bandwidth of 600 MB/sec. For detail about the relation between compute instance shapes, OCPU and network bandwidth please refer to the specific documentation.

Main Take Aways

I hope this article will prove interesting as both an introduction to the OCI FSS service and as a starting point to understand how to use it effectively. The main points I wanted to highlight were:

• FSS components and how to they relate.
• FSS logical topology.
• The fact that FSS service is bound to an AD and its implications.
• FSS performance metrics and how they are relevant to specific use cases.
• FSS client mount options.