Scaling - a 3 part series

When I joined Q2 eight years ago, we had 240 servers supporting about 1 million users. We worked through improving our resiliency, performance, and quality, monitoring those points, and implementing frameworks that drove a performance-driven culture that allowed us to host over 18milion users across 10,000 servers and 45,000 containers. In the past three years alone, we increased to move $4million per minute, adding 7 users a minute and have tripled both the logins and number of transactions flowing across the infrastructure every minute.

The team’s focus was planned and predictable - we moved from a start-up building out the services, to then increasing capacity and supporting the growth, then to meeting the challenge of ever increasing customer expectations (especially as we moved upmarket) and now as a $6billion market cap company – achieving true scale. Each next step seemed impossible as we stood at the bottom of the impending summit to climb, but our confidence grew as the lessons we learned were actioned into results. The Hosting team at Q2 is at a point where I now can share some interesting insight and some lessons around our maturing operations, growing from performance-focused to growth-focused – and into this new phase of Scale:

·        Part 1 - Scaling – humans

·        Part 2 - Scaling – process

·        Part 3 - Scaling - technology

Measuring scale success from the typical technical density, user growth, transaction growth is certainly one way – but the tougher challenge the leadership team faced was much more ambiguous, harder to solve and pushed us to be much more creative – scaling the organization. This is part 1 of my series on increasing scale.

Scaling - humans

I am truly blessed as a leader. Blessed with a strong company mission that aligns us. Blessed with a dominate suite of solutions and services that help customers add real value to the communities they serve. Blessed with success, and surrounded by top management team that cover all my blinds spots, which I have more than the average leader – a gift blended from my impulsive tendency to focus on the goal while overlooking the risk and consequence of a misstep. 

Saying things like ‘optimize’ or ‘reduce redundancy’ sends chills up the spine of ‘builder types’ like me. “If we cannot deliver the service, then what is cost doesn’t really matter – now does it” – spoken in my truest Start-up voice. But we are long past the days of - do you want it fast, reliable or cost effective – pick any two….. the hard job is all three, we must do all three well.

Microsoft’s CEO Satya Nadella stated in an interview with the Dean of the University of Chicago Booth School of Business, the three traits he looks for in leaders were:  

·        Ability to create clarity when none exists

·        A knack for sparking energy

·        An ability to succeed in an over-constrained space

The last one – succeed even with the constrains – the constrains of limited resources, of limited budget, meeting the rising expectations, continual improvement, and competitive pressures that our customers are under.

The team has been with me for a while, level of trust, rawness of how we navigate together is special, and a foundation of our success. The closeness we enjoy also becomes a barrier to growth, expanding the team is hard, joining the team from the outside is even harder. If you are new – you do not get the unsaid context, the dry humor, and the cynical comedy we use to laugh our way through the next challenge. And if you bring a new skill – no one understands it’s value, nor seems comfortable with it, and although badly needed - it is often counter to what got us here. The awful tasting medicine we need.

A case in point – When I started our Network Operations Center (NOC) as a sub-team under Support, staffed by a team of three technicians - 17hrs/day weekdays only. As we began our journey to make Hosting a core competency at Q2, recognizing a need to make the NOC a strategic part of our service delivery, we moved that team into Hosting, and doubled-down on investing in technology, process and people. By the end of 2018, we had expanded the team to 23, millions invested in monitoring tools, and were accountable for not only the two datacenters, adding our large service footprint in both AWS and Azure as well. It was then rebranded to the Integrated Operations Center (IOC) a name more representative of all the mission critical services it oversaw.

Beginning of 2019, a new Senior Director, Carmela Pinzone joined the team, and took the reigns of the IOC. Within her first year, she embarked on an ambitious goal of reorganizing the IOC – creating more scale from an already high performing team, one we could not afford to de-focus with a poorly executed change (which we reminded her of often). She successfully achieved the goal of increased scale through developing two paths – one of increasing capacity and another of increasing capability - reducing the current demand effort while up-leveling the skills of the team to handle higher impact work.

Adding capacity to the operation was through focus on Automation and Hygiene:

·        Working through the 80,000 alerts received each month, finding non-actionable, redundancy and correlation the team reduced the monitoring alerts by 66% and email alerts down by 40%.

·        While the reported fraud cases increased under COVID, newly developed self-service automation allowed customers greater access to data and reporting, reducing cases into the IOC by 58%.

·        New automation targets the stack reporting the issue, alerts the IOC technician as well as running diagnostics on each component, reducing the troubleshooting time by 36%. Now turning this into automated healing to push into a new level of effort reduction.

Adding capability was through an intentional focus on employee Development and Training:

·        Developed a new Training Matrix with ~576 training items - 59% of staff completed all training, 50% completed training to perform new Security Operations Center (SOC) function, leaving 12 technicians now certified on SOC, 9 trained up to Enterprise Application Engineering, and 4 trained on AWS/Azure

·        Working closely with our beloved partners in HR, she help develop and drive an internal emerging leaders program – fostering a focused and intentional effort to shape and empower the next generation of Q2.

This allowed the inbound work to be defined into two tiers - Tier 1 of alert response and remediation, and Tier 2 of cases with increased complexity with more application context.

To take the next step in scale – Carmela then looked to gain financial scale through outsourcing models and staff augmentation for the Tier1 IOC work. She found a perfect partner with Austin Community College (ACC) – where Q2 could leverage talented interns while keeping the investment to create employment locally – aligned with Q2’s mission of investing back into our local communities. The program has been a wonderful, measurable success – with 11 interns going through the program, 3 permanently hired, creating nearly $1/4 million in budget savings and foregoing a capacity expansion of 5 FTE in 2020. We want to acknowledge a special thank you to our key partners at ACC - Hector Aguilar (Dean of Continuing Education), Sam Greer (Employer Outreach and Experiential Learning Programs) and Kathryn Naughton (Internship Coordinator) who have help ensure the success of this program with Q2 – even with Covid turning the education system on it’s head.

With this freed capacity and new capability, we developed and added:

·        a customer-facing Fraud Assist team

·        a Security Operations Center (SOC) to specialize on the Q2’s deep security monitoring and tools

·        developed a dedicated role for a full-time trainer and new hire on-boarding, who has reduced our on-boarding into the IOC by 6 weeks, reducing the time to productivity tremendously.

Looking for even more scale, we recently reorganized Hosting into the proven framework of Plan, Build, Run – or in our speak – Architects, Engineers, Operations – then empowering our Enablement team (PMO) to transition the work between the groups. This allowed us to group similar skills together, and make targeted investments, gaining scale through centralizing and pooling the talent. More importantly, this allowed us to leverage key manager aptitude – each one aligning to the work style of their groups, knowledge experts of that business, and tying their passion into a more focused area. Carmela owns Operations – which is now 63 FTEs across a wide area of execution – from the IOC, to SOC, to internal Corporate HelpDesk, and includes a dedicated technology team. All loosely coupled through a management style of servicing customers and measuring results with data and metrics.

It has been a busy 18months for the IOC and Carmela – enhancing our organizational scale, building out her management lineup, and beginning to establish her team more widely across Q2 both our internal and external customer presence. This new foundation allows us to not only service the customers better but gives us more flexibility how we address the new challenges that are sure to come. Joining Q2, finding her voice, risk changing something good to become something great, has made 2020 not just a year of weird, but a year of achievement building on a foundation of success. 

Working to assemble performant teams, improving our inspirational leadership, and stretching so we become comfortable with being uncomfortable has us focused on building the right organization to operate the sophisticated, complex, value-focused solutions we are constructing and delivering to our partners.

In the next article, we will focus on the Hosting Enablement Office and review their own transformation in 2020, driving agile to scale the typical waterfall project teams, while refocusing on new governance models, data and metrics to help scale the management team as well.