U2 an In-Memory Database?

A few years ago, there were some speed issues accessing Universe/Unidata (U2) data. Ideas were bounced around as to whether this was a Windows machine issue, the layout of our U2 files, or even poorly resized files. Our team spent a considerable amount of time resizing files to find out that there was little gain from all of the effort. One particular database file that had all of the transactions stuffed in a single record (multi-valued and sub-valued) was changed to have a transaction file instead but did not change any speed issues.

By chance, there was this article that talked about Hana around 2013 that somebody on the team remembered. Hana is an in-memory database from SAP that it developed around 2011 and was thought to be a way to better understand cloud computing. Since the database was in memory, the access would be without disk latency and would not have to be loaded into memory for each read.

Our Java programmer found Terracotta which had an open source version that we could try out and solve our speed issue. Our Java team added the code to our U2WebLink middleware running on UniObjects for Java that we use to read, write, delete, and report on U2 databases.

We had transitioned from telnet to the Web in 2006 using U2WebLink so we would have to add additional APIs to handle which forms could use in-memory database files and which could not. We would have to add at least two API calls: XLr8ReadMem and XLr8WriteMem.

There was the problem if the real database was updated how would the in-memory know and how would the in-memory database know about updating the real database? The latter problem was resolved by issuing a database write/delete on the real database after the in-memory database was updated through the API calls. Unfortunately, the former problem was never figured out.

We began testing a file that had about 100k in records. It took about 15 minutes to read all of the data into the file at start-up. The workaround to get it the read all of that data was to use dataset read which dropped the time to about 10 seconds.

The next problem was how to tell U2WebLink which files it should read. This was resolved by adding parameters to the start-up file.

The last problem we encountered was how to show the user all of the data to pick from which we called lookup. The lookups in our non-memory system rely on issuing a SELECT on the file and displaying a window pop-up that shows around 18 items and using arrows and numbers for navigation.

It sounded simple but the selection and sorting were very painful for this in-memory database. It was eventually resolved to our satisfaction while being able to do this in under 10 milliseconds.

We had a working prototype. It was fast, clean, and worked great. We showed it to one of our clients to see what they thought. Let's put it this way, they saw no value in all of the work we did. They would not pay more for this technology feat we thought we had accomplished.

The project was killed the next day. However, we did learn some great lessons. The client wanted reports to work faster with a cleaner layout. We started on BIRT report writer interface to U2 within a few weeks. By the way, the client did love the BIRT interface more than they ever would have the U2 memory option.