ArahWeave fixes slow browsing on network or shared drive.

Sorry, no pictures, today, just explanation and some numbers.

In Arahne's approach to data, we prefer not using a database, since it creates more problems than it solves. So all your data is just files in directories. Weaves, fabrics, images, all properly in their directories and sub-directories. So in a way, filesystem becomes our database. This gives us great freedom - you can put all your data files in Dropbox, on a NAS, or even on windows disk, which is accessed from shared folder in from a virtual machine. This is all good, until you work a couple of years and accumulate 100.000 files. Then, the operating system begins to show its cracks and we see, what really works well and what not. As you might have guessed or experienced, native Linux is excellent, and if you have all your files locally on SSD disk, it works amazingly fast. But, for many practical reasons, we often prefer to have the files stored on a network drive (NAS), or on Windows partition of the same hard disk. We are obsessive with file size reduction, so we use PNG for images, we compress the weaves, so they take just 100 bytes. So we should have good performance, since our total data size is just a couple of megabytes.

But network access of thousands of small files can still be slow, even if total quantity of data is not big. Same is true for shared folders on Windows - the NTFS filesystem is different from Linux filesystems, permissions are handled differently, and this causes performance penalty. Microsoft is adding Linux capability to Windows 10, and even they are not able to fix the slow disk access of Windows files from Linux. For browsing images and fabrics, we use icon cache, so once the icons are created, the program only reads the directory list and then loads the icons from the cache, which is always stored locally, so it is fast. But for weaves, we do not use any icons, since they are so small - we just load the weave file from disk. But if shared folder or network disk is slow, the browsing becomes slow as well. Our idea was that this slow access in this case is an operating system problem. But the situation did not improve in the last couple of years, even if computers, networks and SSD disks are much faster now. So we have added our own caching of weaves to local disk.

Here are the result of this improvement. We read 40.000 weaves from NFS based NAS disk with SSD over the network. Before caching, it took 32 seconds on first read, 12 seconds on second read, and 9 seconds on third. With the new caching, we read those weaves in 10 seconds on first read, 5 seconds in second read, and 1 second on third. Since loading of weaves is a very common operation, this should speed up the workflow of many users on the network.

But if all your files are local on Linux, there will be no difference.