20 years of Open Source from Grid to Cloud Computing

Given that it's coming to the end of 2024 I was thinking back to what I was up to 20 years ago, in 2004. That feels like a long time ago - Myspace was the social media site, my kids loved playing Runescape, I bought my first (small) mobile phone - previously I had an analogue "brick" phone, I bought the fastest laptop around - the Acer Ferrari - and I looked a lot younger! (The Acer ran red hot, just like the colour!)

But I was also involved in an EPSRC project running out of UCL (with Professor Wolfgang Emmerich) in London to evaluate a new type of open source software for Grid computing. It was called the Open Grid Services Architecture (OGSA), and I (and the family) had to move from Canberra to London for the year (arriving in a dark freezing cold London Xmas was an experience). Grid computing is a type of distributed computing for scientific (and other) workloads, and was designed to enable multiple heterogeneous computing resources (typically provided for free by universities and other research bodies) to cooperate in solving problems that would be too large to run locally.

Some of the problems we had to address included how to set up the distributed OGSA infrastructure at each location securely and enable it to provide the ability for end users to deploy and run arbitrary code (packaged as web services) securely across all the available infrastructure.

What did we learn? That it was harder than it looked - there were lots of complications around security, scale, and the fact that most Grid workloads were batch-oriented and the only way you could get access to computing resources was through complex and heavy-weight resource schedulers. We discovered that traditional users were very good at "gaming" the system to ensure they could "reserve" resources well in advance (often by submitting dummy jobs which they swapped for the real ones at the last minute). On the other hand, we were interested in dynamic, real-time, workflows that needed access to resources on-demand for minutes to hours, not days to weeks.

I was pleasantly surprised to find the UCL OGSA project page is still around! Check it out here.

As well as evaluating distributed open source Grid software, the project was distributed across 4 universities, so it was fun to travel around the UK for work (including my 1st International Conference on Software Engineering, in Edinburgh).

After 20 years what did we learn? Grid was an early type of cloud computing - given the problems with finite and fair resource sharing, an obvious solution was an economic dimension - this was a fundamental aspect of Cloud computing - resources cost real $ and you only have them for a finite period of time. Service discovery was an essential part of the Grid, and that's still the case today - AWS (e.g.) has zillions of services, all readily discoverable. Instaclustr also has a whole bunch of services, but the difference is that they are all open source and available on multiple cloud providers. The heterogeneous nature of the available Grid resources corresponds (loosely) to different instance sizes, and the ability to deploy arbitrary code corresponds to early approaches for component deployment, which involved into serverless computing, Kubernetes and more.

These days I'm the technology evangelist at NetApp Instaclustr, and I write about diverse Big Data and Streaming open source software running on Cloud platforms.

The main resource contribution from this project was this paper (available here)

Brebner, Paul & Emmerich, Wolfgang. (2006). Two Ways to Grid: The Contribution of Open Grid Services Architecture (OGSA) Mechanisms to Service-Centric and Resource-Centric Lifecycles. J. Grid Comput.. 4. 115-131. 10.1007/s10723-005-9008-2. Service Oriented Architectures (SOAs) support service lifecycle tasks, including Development, Deployment, Discovery and Use.

Abstract: We observe that there are two disparate ways to use Grid SOAs such as the Open Grid Services Architecture (OGSA) as exemplified in the Globus Toolkit (GT3/4). One is a traditional enterprise SOA use where end-user services are developed, deployed and resourced behind firewalls, for use by external consumers: a service-centric (or ‘first-order’) approach. The other supports end-user development, deployment, and resourcing of applications across organizations via the use of execution and resource management services: A Resource-centric (or ‘second-order’) approach. We analyze and compare the two approaches using a combination of empirical experiments and an architectural evaluation methodology (scenario, mechanism, and quality attributes) to reveal common and distinct strengths and weaknesses. The impact of potential improvements (which are likely to be manifested by GT4) is estimated, and opportunities for alternative architectures and technologies explored. We conclude by investigating if the two approaches can be converged or combined, and if they are compatible on shared resources.