Calling All Bold Ideas

DOD software development activities have historically occurred in decentralized ecosystems, using disconnected teams, and are generally viewed as challenging work environments when compared to Silicon Valley. The reasons are largely established and openly cited, ranging from organizational structure, five-year budgeting models, so-called color of money funding requirements, unfavorable (for DOD) contractually binding terms, castle and moat security perimeters, and so on. The result has created an opaque environment where software designs, APIs, re-usable components, and arguably raw data are simply not discoverable even when it meets statutory open architecture requirements. Distribution statements, need to know, and classification are often cited as a justification by proponents arguing to maintain status quo. The same list is often cited as the barrier by proponents seeking to fundamentally shift the way the DOD develops software. In this article, it doesn't matter which of these groups is right or wrong.?

What both sides do agree upon is that the impact of a software defined world is pervasive, and the DoD must figure out how to produce innovative software, in less time, and in greater quantity than it does today. Agile, DevSecOps, workforce training, and a coalition of the willing can only take the department so far. Innovation has always been built around technology, people, and processes. Vannevar Bush demonstrated that innovation acceleration is possible by building a team that, together, can focus equally well in all three areas, not just one or two.

In the last week, a single console command took down Facebook, Instagram, and WhatsApp, and the audit tool intended to prevent such a catastrophe had a bug that failed to stop the command from executing. Yes, even Silicon Valley’s elites struggle to achieve “perfect” software. The ultimate (incorrect) snowflake argument goes like this:

Yes, but that was social media; no one died because they couldn’t read grandma’s post on FB. We’re different in the DOD!

Given the recent events in Afghanistan, I would argue if the outage had occurred roughly a month early, someone would have died. My point is that the lines between even something seemingly as trivial as commercial social media sites and defense software are now that blurry.

package main

import ‘fmt’

func main() {
  var level string
  level = “CRITICAL”
  fmt.PrintLn(“ALL SOFTWARE IS NOW %v”, level)
}        

Today, the DOD service components and organizations rely on a coalition of the willing to share and shape software best practices. Social media posts help people discover technically kindred spirits. PowerPoint engineering is often used to showcase results, because even unclassified data on unclassified dashboards built using unclassified APIs found on commercial tools are deemed “too sensitive” to share with other personnel in the DoD who hold top secret national security clearances.?Community of Practices are used to present documented case studies each month to a few hundred people. The combined civilian and military workforce in DOD numbers well over three million. Cynically stated here to drive this illustration, people applaud the fact that 0.0001% of the workforce listened to three use cases over three hours.?

Infrastructure is Easy, err, Hard

How many of us think about the amount of infrastructure required to generate and distribute electricity? Generally, we pay the bill each month for what we used and the lights stay on at our homes. Mentally, we don’t think about electricity because electricity has faded into the fabric of society. We've all heard this example, and I love it because it is so universally understood. No, electricity isn’t free, but it is easily accessible and relatively inexpensive compared to other monthly budget spends like food, mortgage, car payment, etc. Advances in solar energy have even reached the tipping point where “living off the grid” occurs routinely and regularly, even in austere environments. And like the Facebook example cited above, sometimes the electricity goes off unexpectedly and at the worst time.

How many of us think about the amount of software infrastructure required to produce and operate the applications we depend upon? Software investing decisions in the DOD are gut wrenching decisions. I hear from program managers called upon to make the untenable choice between investing in their area of research specialization, say electronic warfare, or funding the very mundane MINIMAL set of software tools to be productive in custom software development. Yes, you read that correctly. In the DoD determining if money should be spent on research or purchasing, operating, and maintaining a git repository, ticket management tool for building a backlog, a build tool, etc. is a gut wrenching decision for too many programs. It's a slow, bumpy road to travel. Software infrastructure is neither simply accessible, nor is it affordable to procure and operate at the program level.

It is not necessary to be a technophile to realize that the US is already engaged in a software-defined Cold War (hot, by cyberspace definitions?). Generals and flag officers, too numerous to mention here, have publicly lamented the current speed of innovation, the need for a more empowered workforce, and a desire to break down limiting barriers. They seek bold ideas.

Are Bold Ideas in Software Infrastructure Necessary?

Today, every service component and organization is (analogously) left to design, procure, manage, and operate the software infrastructure equivalent of both the electric power plant (generation) and the corresponding electric grid (distribution), in addition to delivering on what they are supposed focus on, like let's arbitrarily pick... naval warfare (because I will always be a sailor). Is this really the best path forward? Are they moving fast enough to meet demand? No.

Today, I know an individual who has five different commercial git server licenses linked to their name – FIVE(!) — as if they were five distinct individuals, not one. Why? Each cluster was purchased and licensed to a specific program, and so to have an account to work and administer each cluster, well, it requires a dedicated license for each. That’s wasteful. That’s five passwords to manage and rotate. That’s five URLs to remember and pay attention to a location when performing administrative functions. It’s also stupid. Is this really the best path forward? No.

Today, the DOD operates what I speculate number in the dozens (hundreds, thousands?) of different ticket management system implementations for teams to create… backlogs. Is this sustainable and scalable? No.

Today, I have heard too many personalized stories of people being told “No.” when asking for routine technology. I will only cite my personal example here. I was told “No” when I asked for Python on my government laptop to slice/dice data sets. No alternatives, didn’t matter that the request was submitted using the Category One (preferred) offering listed on the official supported product list (SPL). It took 48 hours from request to no, ticket closed, good day, sir. I sit in a career ST billet at the SES level, and I'm terribly frustrated here; I cringe to think how aggravating this must be for our GS-12s who shrug and then can't do their job. Is this how it should be? No.

Today, a lack of evidence transparency and statutory obligations placed on authorization officials naturally creates a skepticism and reluctance to accept a different service’s risk posture when operating even mundane systems- a backlog ticketing system, a build server, and “routine” technology that must be present in modern software development. Some use the term reciprocity, but regardless what it is called, the existing friction stifles innovation.

What is a Bold Idea??

If leaders are looking for bold ideas, we should provide them with bold ideas. But what is a bold idea? “We need to be more agile” and “we need to build more software factories” I contend are not bold ideas in DOD FY22. Today, these are merely amplifying statements to rally around activities and efforts already underway. There is value in rallying, but a rallying cry should never be framed as a bold idea.

A bold idea also isn’t a list of wants and needs. We need less friction accessing software infrastructure tooling is a need (it’s no longer a want). Colorless money is a need. More access to Cloud computing is a need. There are many needs in software infrastructure, but none them are bold ideas.

A bold idea is a radical new perspective that makes someone pause and reflect.

The Software is Never Done study was authorized in FY18 and resulted in a bold idea for the DOD. In FY20, Platform One was a bold idea. A bold idea is a radical new perspective that makes someone pause and reflect. Tiffany & Co. started out selling envelopes before a bold idea was presented that led them to pivot and become the high-end jeweler it is recognized for today.

Bold ideas are revolutionary, not evolutionary, and they are never incremental.

For our purposes here, a bold idea is intentionally broad and general. It is intentionally unrefined. It intentionally isn’t a well-articulated strategy, nor is it a tactically executable plan. Here, in this article, I have chosen to define a bold idea as a “What if…” statement intended to be openly discussed, explored, contemplated, carefully weighed and not simply dismissed as absurd.

What if... we stop selling paper envelopes and switch to selling diamonds!?

A bold idea is the beginning of forming a working hypothesis.

Example of a Bold Idea

I would be remiss if I didn't offer up a concrete and pertinent example of a bold idea when it comes to the way the DOD manages software infrastructure. So here it goes - an intentionally broad and general, unrefined, example to consider.

What if… there was an Office of the Undersecretary of Defense (Software)!?

What if… it was empowered to become the electric company equivalent for custom software development infrastructure.?

What if… it provided a singular pooled budget for first order software development tools and technologies that represent the “minimum” for creating custom software.?

and

What if… it operated the mundane infrastructure unique to software development in such a way that it empowered software development innovation in programs so DOD program managers no longer had to make gut wrenching investment choices between generating/distributing electricity and research.

It's now FY22

The DOD has entered FY22 and it needs bold ideas to solve complex software infrastructure problems. It's been four years since Software is Never Done was initiated. If the answer to DOD software infrastructure problems was as simple as calling something an "Enterprise Service" and asking DISA to manage it, I posit that would have already happened. But it hasn't happened. It likely won't happen.

Is the elevation of software infrastructure coordination to the level of undersecretary a bold idea? Yes.

Is it a good idea? That’s for elected and appointed officials to pause and reflect upon; I have no idea.

Regardless of the merits (or lack thereof) in the illustrative example that I just shared, the DOD needs to contemplate and act upon one (or more!) bold ideas to become relevant in the software defined world. I consciously chose those words very carefully, because while the DOD asks how the Silicon Valley elites structure themselves to create amazing software, in all of my time in Silicon Valley I've never heard one of their engineers ask how the DOD would organize infrastructure to solve a software problem.

If I made you pause for even a moment to contemplate the implication of the idea, then I succeeded in what I set out to do here- illustrate the threshold for a bold idea. Being more agile, training the workforce, and spinning up more software factories are legitimate needs, but at the start of FY22, they no longer meet the threshold required to be described as bold ideas.

Never be deterred from sharing bold ideas - something scribbled on the back of an envelope today might lead to diamonds tomorrow!