From Tools and APIs to Applications

This journey started out with a simple concept- take a fully working database application built on a LAMP (Linux Apache MySQL PHP/Perl/Python) server and convert it to a serverless, API-driven application. The goal was to learn how to take a very common design pattern for applications (web interface to database with security) and see how it can be adapted to the serverless cloud environment. While it was a simple goal, it took a lot of work and learning. The concepts were fairly straight-forward and there were existing tools to implement all the functions needed for the application. The challenge was more in selecting the tools that would be used from a wealth of possibilities and working out the details of how to make them work together.

The starting point was an application server that had been running for years running several small database applications. It was on an AWS Lightsail virtual Linux server chosen for simplicity in setup to get to the work of building the applications. The application implemented simple password security within the application and otherwise security was based on limiting access to the machine to the web server and ssh using a private key file. These are not sensitive applications so more detailed web server security was not needed.

My first step was to work through in my mind what the goals for this new system would be. It took some iterations as I learned more about the tools available, but these are the goals that fit my needs:

1.??????The applications would be built using javascript to cache data on the local device (minimizing data transfers required) and build the display.

2.??????API servers would be used to access the database and other cloud resources.

3.??????Data would be stored in a relational database when not in use by the application.

4.??????Be entirely serverless

5.??????Provide serverless sign in functionality to control access to the applications and data.

6.??????Provide security at all layers to protect access to data.

The next step was to select which tools to use when building this system. One of the challenges with the cloud world today is not a lack of tools but a myriad of tools to choose from. A lot of information exists on what a product “can” do from the various vendors and other sources, but it is often the details and limitations of how it approaches the problem that need to be sorted out. This takes time and at times I went down a path that seemed quite promising only to hit a brick wall where it might function very well but not integrate with other applications seamlessly or provide enough security for my tastes. The fundamental challenge of creating tools to run in the cloud is the large number of use cases that they might be asked to fulfill. Eventually, I settled on the following architecture that met my needs and best fit my existing codebase and experience (knowing other people with different backgrounds and needs would pick a different set of tools that are better suited to them):

1.??????Web server- AWS Amplify

2.??????API services- AWS API Gateway (interface) and AWS Lambda (functions)

3.??????Database- AWS RDS (MySql option)

4.??????User Authentication- AWS Cognito

5.??????Security- AWS IAM

The third step in the process was figuring out how to install and configure all the pieces so that I could focus on the applications themselves. For me, this was the hard part. Once again, it was not difficult because options were lacking. Products designed to meet requirements for the entire web are built with a plethora of possibilities. The trick is that if you want product A to work with product B and product C, you must choose to use options X, Y and Z (dozens of other options are available, but these are the ones that allow the products to work together).

Figuring out the correct combination of options is what I found challenging. Again, it was not for a lack of documentation or tutorials. A Google search produces a long list of possibilities. My challenge was sorting through the many possibilities to distill them to something that matched my particular (what I thought was simple) use case. Having gone to college and written a couple of books, I come from a world where you explain how something works (theory) and then have problems and exercises putting the theory into use. It has been said that the internet changes everything and there are many different learning styles presented by the people who try to help others out by writing articles to explain what they have done (this is one such article). There are many examples of source code solving a particular problem or showing a basic concept that can be downloaded and run. There is also a wealth of vendor documentation that works to explain all the options available for a product and where you would set them up.

So, while the good news is that there is a wealth of information, that is also the bad news. The challenge which can take a large amount of time is taking all the product documentation, samples and articles that cover pieces of what you are trying to do and put them together into something that works as a whole system to meet your goals. Once again, it involves trying various options and combinations to determine what works best. Many promising paths lead to dead ends where you must rebuild and test again using other options. It can take many iterations until you have the technologies and an installation/configuration process that works together well (my first job out of the Navy was working as a systems engineer for large IT projects so I tend to think of applications as systems).

The End Result

The front end is an evolution of previous versions used against the LAMP server. I tend to favor making the client smart, so I cache data locally (in HTML using session storage) and generate the interfaces programmatically on the client side. For me, it is based on numbers. While the servers provided in web data centers have impressive processing capabilities, they are vastly outnumbered by the many millions of devices that people carry in their pocket or have on their desktops. I will always remember the stories told to me by my first manager after I got out of the Navy. He had worked on the Apollo program’s lunar module and told stories of how they had to program everything in assembler while working on computers with 32 kilobytes RAM running at 2 Megahertz (top technology back then). In my pocket, I have a phone that has 16 Gigabytes RAM and runs at several Gigahertz. There are millions of other people carrying phones and having computers many of which are more powerful than my phone. So, it comes down to a numbers game. With millions of capable devices, it would be a waste of capacity to make the server do everything.

Now I could have just authored every page from scratch, but I have been a big fan of reusable libraries implementing those functions that you use repeatedly with simple function calls for a long time (yes, since FORTRAN). So, I built applications and looked for tasks that I used frequently to see what could possibly replace many lines of code or even single lines of code with tricky syntax with a single function call. For example, if you use HTML tables on your web page and you want to format them nicely, you could write HTML out manually like:

<table style="table-layout:fixed" width="80%" border="1" cellspacing="0" cellpadding="2"><colgroup><col style="width:10%">

I chose to implement a function that takes variables to avoid making sure that I get the quotes , spaces, < and > code all matched up properly with a call such as:

startPageTable(true, 80, 1, 0, 2);

This is an evolving library based on years of trying to build different things (10 years overall with 5 years trying to cache information in browser session memory).

So, if the applications are focused on javascript, what functions are needed from the server environment? Being a database application, the focus on the cloud is to securely store the data and provide the appropriate (controlled) access to it. This leads to the requirement for the following serverless systems:

1.??????Database

2.??????API

3.??????User authentication

4.??????Security

A few of the choices were easy. I chose MySQL as the database because, coming from a LAMP stack, it was an easy migration and more than capable of handling the modest needs of my application. Choosing the API Gateway and Lambda as the serverless API option was also easy in that those are the products designed to fit those functions in the environment where my existing application is hosted (so while there are many good products out there, for me this was easiest). Similar choices of Cognito for user authentication and IAM for security as those are the products built to interface with the rest of the stack that I had chosen.

Next came the hard part- making all the pieces work together and fit my needs. The first challenge was the choice of server-side programming language. The previous version of the application was written on the server side in PHP which does not fit well into the new environment. I tried Java and NodeJS, but it was a fair amount of work to get them to do what I wanted. In the end, I decided to learn and code the application in Python (not to get into a war with other programming language proponents, it was just what was easiest to make work in this environment, for my needs and to match my background).

The next challenge was making all the pieces work together. Most cloud products are like the Swiss army knife- so many options to do so many different tasks. It took many iterations reading examples that did individual pieces of the puzzle to get an installation process that created a stack that worked together. It required a lot of learning on not so much the theory behind it but how the team that made a given product implemented it. As with most things in IT, each of the disciplines has its own dialect so you have to adapt to terms such network subnets, security tokens, and python library imports.

The final challenge in my scenario was implementing security for the data. Many security needs are built into the products chosen. For example, IAM controls access between services, Cognito treats logins securely and the web server allowed me to implement SSL encryption. A challenge for most database applications is dealing with a variety of users, some more privileged than others. This has to tie into database user IDs (not the same as application user IDs) and security rules between resources to ensure that the only way a given user gets data to their web browser is through the correct path that ensures least privilege access to that data. This is where you get into properly connecting resource A to resource B to resource C all while using the proper security token here or access rule there.

In the end, I came up with the following architecture. The items in blue dashed boxes show some of the details that are needed to make the various components fit together. For my applications, I have multiple web servers tied to multiple user authentication pools that access a series of APIs that service different levels of access from public through administrator (with database accounts being stored at the server level based on your application and level of access).

Conclusion

So why am I writing this article? First, it was useful for me to summarize my experiences and process to apply what I learned going forward. I benefit from reflecting on where I have been to help me move forward. Also, I hope that if you have read this article to this point, that it may help others who are looking at the process or searching for a template to wade through the many possibilities provided in the modern web world by multiple providers. In the end, my goal was to produce some simple database applications focusing on a “smart” client to distribute the load to access a database in the cloud using only serverless options. A simple example page is shown below (it is safe, none of those resources exist anymore as they were just prototypes).

For anyone interested in the detailed configuration options, I have included the setup steps that I used to build the serverless environment on my blog:

https://josephbgreene.blogspot.com/2022/05/serverless-installation-overview.html