Scaling - Part 2 of 3

Within the past 3 years the tremendous growth of our platform as come through two avenues – adding more users through both organic expansion of our existing customers and Q2 moving upmarket. 1/3 of the nation’s top 100 banks are now powered by Q2, 1 in 10 people using online banking domestically are logging into a Q2 platform. Not only growth, but rising customer expectation – like everything in our lives, we expect more – more uptime, more performance, at better quality. What this means for the team that must support these services is that not only are they always building out capacity but having to improve those services at the same time - it is truly flying on a plane, making it bigger, safer, and faster all while flying in it. A single minute we support over 14,000 logins, 6 users added to the platform, nearly 7 transactions moving over $4 million – nearly a 300% growth over a minute in 2017.

The team’s focus was planned and predictable - we moved from a start-up building out the services, to then increasing capacity and supporting the growth, then to meeting the challenge of ever increasing customer expectations (especially as we moved upmarket) and now as a $6billion market cap company – achieving true scale. Each next step seemed impossible as we stood at the bottom of the impending summit to climb, but our confidence grew as the lessons we learned were actioned into results. The Hosting team at Q2 is at a point where I now can share some interesting insight and some lessons around our maturing operations, growing from performance-focused to growth-focused – and into this new phase of Scale:

·        Part 1 - Scaling – humans – last week’s article found <here>

·        Part 2 - Scaling – process

·        Part 3 - Scaling - technology

Scaling - process

As our growth continued over the years, so too did the complexity. Adding staff at the rate we were growing, their time in the seat and experience in role were lacking – and this lead to a growing gap between the skills needed to operate the business and the talent in position trusted to do so. The gap – is where chaos lives (as defined by Netflix in a presentation I saw years ago). The natural reaction to stop the pain caused by the chaos is through building a bunch of process and procedures around it – creating a situation where ‘anyone’ can operate in the roles, if we just simplify them enough, write it all down, and then hover over them to make sure all the process are followed. Nobody likes process – but it better than suffering chaos – right?

Wrong. It drives out talent and prevents innovation. What we did instead – is to intentionally reduce the complexity as much as possible, and then hire the best we can – keeping that chaos gap as small as possible. Sure, we had to add some process – but we really try to lead through context, not control.

Truly blessed as a leader – I am very fortunate to enjoy the services of a talented team of dedicated project managers lead by one of the top people managers in Q2 – Beth Brannan-Neely. Beth leads a team through her authenticity, values, and driven focus on outcomes. A certified PMP, Beth’s secret skills of aligning stakeholders and modifying the at-times-seemingly-rigid 13,000 PMP process steps into just the needed high-impact ones. Her team had a tremendous 2020, helping the many teams she supports scale through a record year at Q2 – here’s how….

New problems began showing up in two different ways – but resulting in the same less than desired outcomes:

First, many of the Hosting teams still used the traditional waterfall project management – starting each year with 90% of the year’s plan laid out. Couple of monster projects, many recurring like refreshing compute, adding capacity to storage, …etc, etc – and since these are annual standing projects that take months the team would simply slot them in them into the annual calendar, then try to fill all the new projects around them. The challenge that began to present in the last few years, was on the innovation projects – the new stuff, the stuff that has a high rate of learning as part of the project – these projects were much more difficult to forecast work effort and therefore the timelines. Once one of these new projects started to run into slippage, the sheer number of projects began to stack, and the queues of new work began to overfill. The team was then faced with the age-old decision to either slip everything, pushing back on the next project starts - or - to push the problem project aside and keep going with the others. Words like ‘too busy’ and ‘prioritize’ started to be the go-to solutions – all code words for – please relieve the demand pressure.

Secondly, we began hitting a wall of throughput – the daily operations continued to grow was well, so there was a constant nibble at the resources. Each team owned a technology domain – broadly like ‘cloud’ or ‘datacenter’ or ‘network’ – which meant every team had to work tickets, solve incidents, perform routine care and feeding of maintenance, adding capacity, and be thinking about roadmaps, new innovations and architecture. Being SMEs of their technology, they also had to deal with being tied into several cross-team projects that they owned pieces of critical delivery. So not only were the SMEs dealing with a high level of context switching but their management had to be good at everything as well – operations, metrics, strategy, innovation, coaching – working in and then working on the business. We were simply getting too broad to be any good at anything.

So I did what I always do in these situations – I leaned in on my trusted direct reports, the senior leadership that have worked closely with me for years. The team came through with two big solutions, not just incremental iterations, but step-function changes.

We did a reorganization – moved formally into the tied-and-true ‘Plan, Build, Run’ model – for us it meant ‘Architects, Engineers and Operations’. This aligned like-minded skills together – and then defined more narrowly their areas of responsibility. It then also allowed us to leverage the leadership – one manager focusing on Architecture, working with Development and new technologies; another manager building out the different solutions working with vendors and architects, and then a manager who specializes in operations, metrics and working with customers. This added scale to the operations because we no longer needed to invest into each team for each of these separate disciplines, and are now pooling the resources together, as well as, we are getting more efficiency from the new groupings. Beth’s team now owns the ‘transition’ services across the teams – ensuring alignment, knowledge transfer and requirements are flowing between the teams. Her team is uniquely positioned as partners across the organization to empower the communication to new heights (not typically a strength of the native IT teams).

Next Beth transformed the way we executed projects – tearing up the very fabric that we had nearly perfected over the years. She started by transforming her own team – moving from PMP certifications to Scrum Masters and ITIL v4 credentials. Once the team was reskilled, they leaned into each of the teams and shifted away from waterfall project management into agile methodologies. Agile has been proven and around for a while, however, adapting it to infrastructure teams, changing all the measurements and reporting – was no small task. They created early wins, built on that success and over a short period of time are now managing twenty-one scrum teams. The communication has never been better, blockers identified sooner, resources juggled. The nemesis long-tail projects were chronically always dragging on past due dates, were now turned into short digestible two-weeks sprints. The teams feel more successful, the managers received some much-needed real help managing this ever-growing portfolio of project work. It also helped the innovation – the project that have a high level of learning – hard to forecast time and effort – lends itself much better to agile, where we can recalibrate on process continually.

At the end of 2020, our first year of this transformation – Beth and team have led the charge into scaling the resource utilization, shifting the way we manage work, and helping support the knowledge management of this new organization. It is a lot of change for the teams to adjust to, but her unique gift of aligning the stakeholders and communicating throughout the journey has ensured our success. Not yet perfected, it’s a foundation we will continue to build upon for years to come, growing 30% year-over-year….