Accounting in the Serverless Cloud

One of the most fascinating R&D topics in Enterprise Software today is how to utilize the "Serverless Cloud".

This next generation of cloud computing is beginning to have a dramatic impact on IT operations; particularly Financial business applications.

The modern business economy is increasingly moving towards subscription-based service business models. Traditional "back office" functions are moving directly into the "front office" customer experience.

Most applications and services today have a "My Account" feature with access to billing history.

Traditional ERP systems are being stretched to scale and meet the demands of this new era in subscription services. But the capital expenses (CapEx) and costs involved are reaching near unsustainable limits when compared to serverless computing.

"Serverless cloud" is a next generation architecture that solves the challenges of modern businesses using cost-efficient, auto-scaling, usage-based computing infrastructure.

The Challenge - Case Study

Cubic Compass recently had the opportunity to migrate a traditional accounting and billing system to a serverless cloud architecture.

The scale required generating millions of billing, invoice, line items, and ledger entries per month.

The resulting architecture required near real-time reporting and analytics.

Budget and Costs

Our target budget was to spend $250 per month on hosting costs during development, scaling to $1,000 per month for basic operations, and a "not to exceed" budget of $5,000 per month at full scale.

Thanks to the AWS free tier, the $250 per month budget was easily achieved, and initial load tests stayed well below $1,000 per month (at the time of this writing, the project was still in proof of concept review phase and not fully deployed).

As every AWS Developer has likely experienced, there was some initial anxiety around "runaway" costs on AWS. One misconfiguration can put AWS services into a vicious loop, accumulating several thousands of dollars in charges overnight.

Setting limits on auto-scaling and configuring the AWS billing console to send alerts when resource utilization hit predefined high water marks alleviated our concerns about runaway hosting costs.

The Solution - Serverless Cloud

Using Salesforce.com as the system of record for billable Accounts, Amazon Web Services (AWS), and a T-Star Ledger solution from Cubic Compass, a serverless accounting solution was implemented.

The scale of data (billions of records) and retention requirements prevented storing all accounting data in Salesforce. So a hybrid solution was introduced, per Salesforce Large Data Best Practices.

The value of a serverless architecture is best understood when presented in context with traditional ERP and accounting systems.

Traditional ERP systems are a Capital Expense (CapEx). Several servers and large databases are purchased with enough capacity to handle peak processing loads.

As the name implies, there are no servers in a serverless architecture. Storage and compute capacity are automatically scaled like an electrical utility bill. Zero utilization means zero costs. Increased usage equals higher costs. The monthly hosting costs are an Operational Expense (OpEx) of services used.

When servers are not utilized, they still incur a fixed cost. Most servers sit idle until called upon to burst and run batch processes. There is an upper limit to how much processing can be completed in a traditional server-based system.

Serverless Database

Using the Aurora Serverless database from AWS, the database automatically scales both memory and storage to meet computing demands. The database service is configured to go idle after a period of inactivity, reducing billing to $0 per hour when not in use.

Read-replicas are provisioned for near real-time analytics using tools like Tableau. Writes to the master database are typically reflected in the read-replicas within milliseconds.

Function-as-a-Service

Perhaps the most dramatic evolution in serverless architecture is the introduction of Function-as-a-Service (FaaS). FaaS allows developers to upload code to cloud hosting environments where the server resources are automatically provisioned as needed.

FaaS functions are typically priced based on transaction requests and processing duration.

For example, Amazon's FaaS platform, Lambda, is priced at $0.20 per million transactions and fractions of a penny for every second of processing.

Lambda functions may be executed either synchronously or asynchronously. The synchronous functions are performant enough to be used in REST APIs and web services.

Once Lambda functions go idle, the server resources are reclaimed and all billing ceases.

We developed the functions in Java using Martin Fowler's Accounting Patterns. But Lambda supports 6+ languages and provides a Runtime API which allows you to use any additional programming languages to author your functions.

Message Queues

Lambda functions may be assigned to handle messages on a message queue.

In the use case above, a monthly invoice run was configured to put 1M+ messages on a message queue (SQS) which were consumed by up to 1,000 Lambda functions to process the messages in a massively parallel operation.

Message queues also provide high availability and reliability as failed function processes can be retried until the message is successfully processed.

Orchestration and Step Functions

When functions must be conditionally executed, or run in a particular order, then AWS Step Functions are utilized.

AWS step functions allow for complex workflows and orchestrations of Lambda functions.

Extract-Transform-Load (ETL), billing, invoicing, and posting rules are examples of processes orchestrated using step functions.

Note: In 10+ years of developing ETL/ELT processes involving the Salesforce.com API; none have achieved the performance and scale attained using AWS orchestration. Salesforce supports up to 20 active API connections, and out of curiosity we configured 20 parallel ETL tasks using AWS. The results were amazing! (a future post)

Elasticache

Last, but not least, Redis Elasticache was utilized to keep frequently accessed reports and statistics in-memory.

Probably one of the more difficult AWS services to setup; due to VPC networking requirements; Elasticache is also one of the most rewarding resources in any web service architecture.

Frequently requested financial reports that took several seconds to query from an RDS cold-cache, returned in milliseconds from Elasticache.

Applications in Financial Accounting Systems

The ability to dynamically auto-scale many jobs at anytime allowed us to rethink many accounting processes.

End-of-day and end-of-month close batch processes can be initiated in real-time. The immutable nature of accounting ledgers allows for fast append-only operations to execute continuously.

The ability to support manual data entry tasks, adjustments, and bookkeeping entries does not fundamentally go away.

They are always required for reconciliation adjustments. But opportunities to automate these tasks became more evident.

The schemaless quality of Redis cache allows for accumulating statistics on the fly and generating time-series reports without any ETL to a data warehouse.

Billing history and ledger entries can be securely queried from API Gateway; which enables a variety of mobile consumption use cases.

Future of Serverless

Serverless is considered a "disruptive" technology, hence it is predominantly adopted by startups and early stage companies.

But larger, more established organizations do not need to completely throw away existing ERP systems to adopt serverless. Most mobilized accounting applications are fundamentally just an Accounts Receivable (AR) sub-ledger, which can rollup to existing General Ledger (GL) modules.

The adoption of serverless in a hybrid environment largely will become a systems integration exercise. However, the hosting of ERP servers will still be greatly improved with a migration to cloud and hosted within the same regions as serverless resources.

Many "disruptive" technologies that come along are really just fads. However, I strongly advise decision makers to view serverless as a major paradigm shift, similar to the transition from factory generated power to using utility power grids.

The "Serverless Cloud" paradigm shift is well described in Nicholas Carr's book "The Big Switch".

The New Front Office

Based on our background in CRM and front office applications, it's worth noting the increasingly blurry line between traditional front and back office. Finance users increasingly need access to customer relationship, CPQ, and order management systems. Front office Sales and Marketing teams need access to customer spend history for metrics like CLV (Customer Lifetime Value), MRR, and usage history.

As more financial applications move to the cloud, and financial analytics become available to the broader enterprise... we get closer to the ideal "Customer 360°" model, where pulling up an Account or Contact record in Salesforce displays up-to-date related financial information.

Retrospective - The Bad and Ugly

I've mentioned all the good parts so far. But in all honesty, there were certainly some bad and ugly challenges along the way. In the end, though, the benefits outweighed the pains and I can confidently say, now that we know where the land mines are, this is a reference architecture we'll continue to use in future projects.

Cost Conservation

The costs for serverless architecture are exceptionally low. But unlike CapEx fixed expenses, they can be unpredictable.

Most consumer applications actually do want the problem of million users suddenly flooding their site, and the auto-scaling costs of serverless are well aligned with their profit center. More utilization can be correlated with more revenue.

But business applications often are budgeted as a cost center with a fixed number of users... therefore the costs are expected to be far more predictable.

I mentioned the risk prevention steps we took above. There is a general responsibility to make sure applications "turn off the lights" when not in use.

Not "Networkless"

While "Serverless" may be nirvana for Software Development teams, it is definitely not "Networkless".

Our greatest initial challenges were configuring the Virtual Private Cloud (VPC). Some services, by default, are provisioned outside a VPC. This requires fairly detailed knowledge of subnets, security groups, routing tables, and internet gateways to get all these services to work together.... securely.

AWS Cloudformation templates may ultimately ease this pain in the future, but we spent considerable time on Google searches, AWS forums, and StackOverflow.

Thankfully, articles like this one are discoverable to help integrate Lambda functions with RDS.

When staffing a team around AWS infrastructure at scale, the need for a DevOps / SRE / Networking role is critical. It'd be awesome if every Unicorn Software Engineer could monitor and manage their own Serverless environments. But in practice, this is too much to ask.

An objective security audit from an outside firm once per quarter may also be a wise staffing plan. AWS features and best practices evolve constantly. So proceed with confidence, but avoid arrogance. There is always one more thing an outside expert can show you.

Lambda Cold Starts

Because Lambda functions are JIT (Just-In-Time) deployed containers on hardware, they can be incredibly slow on first request. That is basically the trade-off for $0 to $N auto-scaling.

The fix is to implement a polling keep-alive service. We used the lambda-canary blueprint to poll various API endpoints every 1-5 minutes. That ensures at least 1 Lambda container is active at any given time.

Documentation

Some AWS services, like Elasticache, just aren't documented very well. There's a lot of trial and error, process of elimination, and printf debugging involved. If I wasn't already a huge Redis fan, I'm not sure I would have persisted to work through all the configuration challenges. I can only imagine the attrition of those less persistent.

Fortunately, AWS does have outstanding paid support and a customer-centric focus. So escalated issue is bound to get someone's attention.