Builder a Smart(er) Nation: Concluding my Fellowship

After seven quick months sadly it’s time already for me to wrap up my Singapore Smart Nation Fellowship. As a Distinguished Fellow in GovTech (officially Singapore Government Technology Agency) I had a chance to work closely with both the Smart Nation and Digital Government Office (SNDGO) and a variety of teams within GovTech, including our Government Digital Services (GDS) teams, which built maskgowhere.sg and many other useful apps, and our Open Government Products (OGP) group, which brought you cool apps like parking.sg.

The purpose of the fellowship program is to bring an industry perspective into the government, but not in a pure advisory role but fully immersed. Naturally, one of my focus areas was the migration to GCC, our Government on Commercial Cloud initiative. I also spent much of my time mentoring developers, e.g. by discussing project architectures, and shaping our overarching software delivery platform strategy. Lastly, one of the biggest impacts I was hoping to make is building a tighter connection between decision makers and technical staff.

Connecting GovTech Penthouse and Engine Room

It’s not a big surprise that I enjoy riding the Architect Elevator to connect the engine room where software is being built with the organization’s leadership.

GovTech’s engineering teams gave me ample opportunity to visit the engine room. The lowest floors I visited involved debugging a client-side Javascript concurrency problem that used HTML Web Workers to decode incoming data for end-to-end encryption. After some profiling and testing hypotheses, it turned out that the bottleneck wasn’t in decoding at all but in a third party library that we used for splitting the incoming data stream into lines of text. This library contained a complex regular expression that, because it preceded the parallelized section, starved the Web Workers, essentially preventing any form of parallelism. Replacing the expression with simple logic set the concurrency free and sped things up by four times. While the story is nicely told in hindsight, debugging this problem wasn’t easy. The defining moment was likely when we realized that changing the number of parallel threads didn’t make a difference at all. Despite our disappointment, it reminded us of one of the most important rules in debugging:

When the unexpected happens, you have the best opportunity for learning.

At the other end of the elevator, up in the penthouse, the most exciting moment was likely our lunch with PM Lee, our prime minister. We discussed a range of topics from talent acquisition to cyber security and cloud strategy. A prime minister who can combine a clear strategy for growing Singapore into an IT hub with a keen insight into the underlying technology is an equally rare and valuable asset for the nation. 

Much of my discussions on cloud and technology strategy took place with our Permanent Secretary Ng Chee Khern, GovTech’s Chief Executive Kok Ping Soon, and of course Chan Cheow Hoe, our Deputy CE and Chief Digital Technology Officer, who was the one who connected me to GovTech in the first place.

Building a tighter connection across our organizational layers ended up being one of the guiding themes of my fellowship. While the Singapore government has a top notch leadership cadre, most don’t have an IT background with perhaps the notable exception of PM Lee, who posted his Sudoku solver code on-line just for fun. With software playing an ever bigger role both inside the government’s operations but also as an economic driver for the nation, it becomes apparent that managing by objectives and abstractions alone is no longer adequate. 

In today’s environment, managing IT by objectives alone is no longer adequate.

Today’s executives need to be able to evaluate the ramifications of technical decisions.

Singapore is in the fortunate position to have a leadership team who recognizes that being able to evaluate the ramifications of technical decisions is a major asset for a modern technology leader because it increases alignment between strategy and implementation while reducing both cost and risk. So, if you expect your IT to “do more with less”, it’s wise to start with your leadership team.

IT: Grammar and Vocabulary

One of the biggest hurdles that keeps senior executives from getting closer to the IT engine room is the complexity of today’s IT environments, further obscured by the endless stream of buzzwords, jargon, and open source project names. Working with our executives I realized that we can make it much easier for executives to “speak IT” if we split IT’s “language” into two parts:

• The grammar: A language’s grammar defines the structure and rules for composition of words into meaningful sentences. Similarly, the grammar of IT is the set of architecture rules and constraints that define how systems and components can be combined. 
• The vocabulary: This is the long list of (buzz)words and technologies that we use, often wrapped in jargon and fuzzy marketing terms. 

We found that executives are largely put off by IT’s seemingly endless, unnecessarily cryptic, and ever changing vocabulary. In contrast, IT’s grammar, the rules that define how IT operates and makes decisions, is relatively static, and much more approachable to executives. At GovTech we therefore established decision transparency that leaves buzzwords and products aside and instead focuses on the available option space, the inherent trade-offs between the available choices, and the principles that guide our decisions.

Following this approach, we discovered two interesting aspects:

1. Senior executives can much more easily grasp IT concepts and be involved in decision making once we remove the fog of jargon. They are used to making decisions affecting the organizations (or the nation in our case), so the “grammar” of our world is actually quite familiar to them. This often comes as a surprise to both sides that always assumed that executives “aren’t technical”.
2. Presenting IT decisions in a way that emphasizes constraints and trade-offs without falling into buzzwords and jargon is incredibly difficult, but actually very helpful, for IT teams. 

The second item ended up being a useful forcing function to improve our internal communication and documentation. While occasionally painful (“isn’t that part obvious?”, “We already selected the best product!”), it turned out that preparing decision documents for better transparency actually helped the teams better understand the decisions they were making. A few times, I also had to apply one of my hard but important rules:

Things may be complex, but nothing is ever confusing in and of itself.

If something is confusing, that’s because you made it so!

Architecture Discipline

Interestingly, discipline is a word that is being used with caution these days as we’re all looking to become agile and innovative. In reality, though, most creative endeavours are actually quite disciplined and Agile development is no exception: Agile prioritizes work clearly, checks progress in short intervals, and measures customer value frequently. That’s quite disciplined. 

For us, discipline turned out to be an important enabler for architecture, which establishes the base vocabulary for decision making. For example, if you’ve concluded that APIs should be a key part of your strategy (as is often, and rightly so, the case), you need to translate that principle into concrete decisions. For that, you need to know which components are part of the proverbial “big picture”. For example, you’re surely going to want to have service discovery, communication standards, common services such as traffic monitoring, and perhaps a service mesh (as long as you can agree what that is). That’s the list of ingredients - it’s a useful start but no architecture. Architecture tells you how these pieces fit together, whether there are gaps or overlap between the pieces,and whether you have considered the right scope (perhaps your API strategy has a tight connection to your software delivery). 

Architecture also leads you to critical decisions, for example, whether APIs within a (micro-) service-oriented application are handled the same as APIs meant to be accessed by other applications or from outside the organization. Such decisions are often overlooked but particularly meaningful because there’s no simple right-or-wrong answer. Instead you have to make conscious trade-offs and make sure that they’re clear to everyone. 

Distilling critical decisions from teams’ documentation was one of the main contributors to creating this transparency, often applying the models I shared in a previous post.

Thinking like an Architect

One of my more interesting and challenging assignments was to compare two projects that on the surface seemed to pursue similar goals albeit with a very different approach. Naturally, management was interested to know whether we should continue to pursue both projects, whether they should be merged, or whether we should continue one and cancel the other. Needless to say the teams were quite adamant about highlighting the virtues of their respective approaches but somehow appeared to be talking past each other.

The challenge with such a decision lies in the fact that at the surface the projects are indeed similar. We therefore needed to dive into technical details to get to a meaningful answer - without losing the decision makers. Sounds like a classic Architect Elevator exercise!

To arrive at a fair and meaningful answer, we first defined a model that’d contain the dimensions on which we would compare the projects. This model essentially defined our option space, containing items like the basic security architecture, software delivery model, and information flow. With this model, we could rate the projects along a uniform framework, defining the strengths and weaknesses of the respective project along each dimension to come to an objective assessment.

While answers became relatively clear on some dimensions, we realized that others depended on our overall stance regarding how big a role the government should play on managing citizens’ personal data. I enjoyed this project particularly much because it highlighted the immediate connection between technical architecture and policy making.

The Value of Architecture

While increasing decision discipline and building a better connection between penthouse and engine room will sound useful to most, often the question remains how we can quantify the value of architecture. I actually found that to be not that difficult. IT invests significant sums of money into hardware, software, and services. By defining a precise vocabulary and increasing decision transparency and discipline, you’re bound to detect projects that overlap, provision oversized infrastructure, or can be scoped down.

The government being funded by taxpayer’s Dollars makes cost control a particularly important topic. We found that a lot of money can be saved by shortening decision cycles because otherwise teams are incentivized to anticipate all possible scenarios, which will lead to overspend on features that may never be used. We also found that more hardware doesn’t necessarily increase a system’s availability - it also increases complexity, which in turn can be cause for outages. In summary, I can say confidently that architecture paid for itself many times over! 

Talent Development 

Any IT organization is only as strong as its talent pool - Govtech being no exception. Singapore’s high demand for IT talent turns out to be both a blessing and a curse for our government. On one hand it reflects the strength of Singapore’s economic heavyweights, be it financial services, digital unicorns, or start-ups, and the shift towards critical IT capabilities. However, it also means that GovTech competes for talent with the likes of DBS, Grab, and Facebook. 

When having worked with traditional IT organizations looking to boost their talent pool, I routinely reminded them that they likely have more to offer to prospective candidates than they might realize, and I don’t mean in terms of salary. For example, GovTech is one of the nation’s larger employers of software engineers who work on active product development as opposed to localization or support functions. Also, we have the benefit of obtaining rapid feedback on our products as virtually all of them are deployed in the local market to improve our citizens’ lives. For those who feel that working in public sector is somehow less interesting than working for a Silicon Valley-type organization I share one bit of advice:

The company logo is on the outside of the building!

So, don’t choose an employer on the brand name alone. Instead, have a look at the kind of people you’ll be working with, your future managers, and the organization’s overall mission. You’ll find that GovTech doesn’t have to shun the comparison in any of those categories. Over the course of my daily work, but also as being part of a talent scheme evaluation panel I worked with top-notch software developers, inspiring leaders, and caring managers. That’s not a huge surprise when you realize that GovTech has one of the larger software engineering teams in Singapore!

The part that I secretly enjoyed most is the occasional puzzled look on people’s faces when I proudly stated that “I work for the government.” Yup, working for the government can be fun and “cool”!

The GovTech Family 

Despite all the great accomplishments and the many good friends I made at Govtech, it’s time for me to return to industry. However, it’s not really a “good bye” - I’ll always remain part of the extended GovTech family, a connection and a privilege that I will hold in high regard!