Changing Hiring Practices to Build a More Diverse Technology Organization, A Case Study from Avvo

Introduction

I was the Chief Technology Officer at Avvo from 2016 until 2018, when the company was acquired. One of the proudest achievements of my tenure there was building a more diverse and inclusive technology team.

While I have spoken a bit about our efforts before in talks, I haven’t written about it. Even though we had done well, I hadn’t achieved the ambitious goals I had set when I joined the company. I was always hoping to do better at my current company, and then use the wisdom from building more diverse organizations multiple times to provide higher-value insight.

Given current events, I hear from friends at tech companies that they are re-evaluating their lack of diversity. They want to improve. While companies have tried different tactics for years, we haven’t made much progress as an industry. If you are trying to address diversity for the first time, it is a daunting task. It is easy to try some things, make no progress, and then give up.

This article isn’t a prescription for improving diversity at any technology company. It is a case study for how we grew diversity at Avvo. The different strategies we employed may be useful for your efforts. They may just inspire you to try new things that are appropriate for your company and culture.

Starting Point

Avvo is an online legal marketplace connecting consumers and attorneys in the US. The company overall was fairly diverse from both a gender and a race perspective. When I joined in 2016, the technology team was a different story.

At the start of my tenure, the organization was not very diverse, as displayed in the charts below. Additionally, there was no one from any under-represented groups with a management title in the Technology team.

The team’s status quo was not the result of overt discrimination, but it did demonstrate a lack of prioritization of diversity, inclusion, and reducing unconscious bias.

My vision for the company was to make it an exemplar of diversity and inclusion within the region. I wanted to not only make Avvo’s technology team diverse to prove that it was possible. I also wanted to show that a diverse group is more than capable. I knew that we could use our diversity as part of our employer branding, helping us stand out in a very crowded market for talent.

Making Improving Diversity a Priority for Technology

I set the tone I intended to take as part of my interview process for the role. I didn’t want any potential for disputes between the rest of the senior leadership and me.

Once I had accepted the job, I had a multiple-month notice period with my former company. I used the time to talk to the Director of Engineering and head of recruiting to prioritize diversity in hiring, set expectations, and start building a better pipeline.

Building a Coalition and Setting Expectations

If you are working to build a more diverse team, you have to get recruiting as your partners on the effort. While it may seem natural, it is not. Recruiting (especially external recruiters) are incentivized or measured on their ability to close roles as quickly as possible. Most recruiters I have worked with are delighted to help find diverse candidates, but it is critical to let them know you understand that it means it will take the whole process longer. It is also essential than their managers understand this.

If you have hiring managers reporting to you, you need to get them on board and excited about this effort as well. One measure of a manager is their ability to hire well. Being serious about diversity will likely mean that they will take longer to fill roles for their team.

You may need to assure them that you will take this into account if they miss their KPIs. You cannot set-up or hold conflicting goals for your managers. If you cannot prioritize building diverse teams over hitting KPIs, you will likely fail at both.

One way to handle the increased time to hire is to start sourcing for a role much earlier than scheduled. Spend the time before the position was supposed to open only looking at under-represented candidates. Create an agreement with your manager that you will be able to hire them if you find a suitable candidate early. If you reach the planned opening date for the role and you haven’t located anyone, you can open the pool to your normal channels without losing time.

Your manager will also need to understand the potential impact of a diversity hiring focus on your KPIs.

With your manager’s support, create a reasonable KPI or OKR as your goal to track your efforts. The OKR should give you not only a target but also some cover if there are challenges to some of the tradeoffs you may need to make because you are hiring more slowly. Share your goal with your team. Update them on your progress regularly. Transparency is essential but also helps to enlist the rest of your team doing their part to help achieve the goal.

At Avvo, my diversity goals were part of my OKRs. I shared our progress with a monthly update to the organization. This repeated communication demonstrated the level of importance I placed on this goal and showed our improvement in real-time.

The good news is that as you make progress towards your goal, your time-to-hire will come down while maintaining or increasing your diversity. It does get easier. You will soon stop making tradeoffs as you continue to improve.

Fixing the Interview Process

When I joined Avvo, the hiring process involved multiple pre-interviews, a do-at-home coding challenge, a review meeting, an in-person interview loop, and a follow-up meeting. The process may have taken a single candidate over a month to get from the recruiter’s first e-mail to an offer. Even without taking diversity into account, it was not an efficient process for a company that wanted to grow.

The Avvo process kept inappropriate candidates out by design. It did not find appropriate candidates, however. In machine learning parlance, the false rejection rate was too high to avoid raising the false acceptance rate. If you are trying to screen out, you miss a lot of good people. Often, biases are part of the filters. Those biases reinforced the lack of diversity.

The state of the current process gave me a license to make some drastic changes. I had the support of the recruiting team and many of the managers. They all had stories of losing good candidates because of the length of the process.

There were some employees and managers who defended the status quo hiring process as required to maintain the quality of the development team. Luckily, they were in the minority. After hearing their position and discussing it with them, I decided to move forward, knowing that I had the support of the majority of the group.

Removing the Coding Challenge

Many companies believe in the at-home coding challenge as a critical way to establish a candidates bona-fides before investing too much time in them. Many sources discuss the problems with coding challenges from a diversity perspective. The arguments against the practice that resonated with me were the following:

• Candidates will often take much longer to complete them than companies expect them to. The Avvo guidance was that the candidate should spend no more than two hours on the challenge. One candidate estimated that he spent 12 hours on it. He wanted to make sure that he did well.
• A coding challenge is unpaid labor with no value for the results. The demand sends a message that the company places no worth on the candidates’ time. I do know that some companies pay candidates a nominal fee for completing the challenge. I encourage this, but eliminating the challenge also solves this problem.
• Not all candidates have sufficient time to work on coding challenges. If a candidate is on a job search, they may get challenges from multiple companies, each with an expectation of hours of work. A candidate may already work at a stressful job or multiple jobs. They may be a caretaker to children or other relatives or have a long commute so that they can live in an affordable home. The candidate may have other issues that require their time. The challenge can be an unreasonable burden for those people.
• A coding challenge requires that the candidate have the equipment and connectivity to complete the project. This requirement may not seem like an unreasonable expectation for a software developer, but not every developer may have the economic or personal safety to make this happen.

The combination of all of the above creates a bias towards people who do not have significant constraints on their time and finances. Those biases reinforce and perpetuate the lack of diversity in the industry.

As part of the hiring process, it is still critical to understand a candidate’s technical maturity. We replaced the coding challenge with multiple in-person challenges during the interview loop described later in this article. To not overwhelm our interviewers, we focused on making sure that the recruiter and hiring manager screens established a high culture and technical bar. This expectation on the screening required training the hiring managers on questions to ask for candidates at different levels. As we got comfortable with this process, the new screening process achieved an equivalent level of success identifying candidates while increasing diversity in the candidate pool and significantly decreasing the time-to-hire.

The Interview Loop

There are many articles on reducing bias in the interview loop. We combined the ideas that we felt resonated the best with our culture as well as some of the best ideas I had experienced from prior companies.

Interviewer Training

We required that anyone participating in the interview loop participate in two training sessions:

• An overall interviewer training that myself and the head of recruiting for my team presented. This training included much of the standard content of a traditional interview training effort.
• An implicit bias training led by someone from Ada Developers Academy (a Seattle developer training program focused on women and gender-diverse people). We had hired interns from Ada, and their mentors were required to take this course. It was so valuable that we made a deal with Ada to present that class at Avvo from time to time to help with our interviewer training.

We also required that anyone participating in interviews got qualified on the interview questions.

A group of developers for each of the different disciplines got together to create two or three technical problems to solve reflective of the type of work they did every day. Each question had a set of representative answers in a 3×3 grid. One axis represented the candidate’s experience level. The other axis was unacceptable, acceptable, and excellent solutions.

We asked the same questions to any candidate regardless of the level of the role or their experience level. Over time, we added to the set of representative answers as we heard new answers from candidates.

To train on the question, ideally, the interviewer would attempt to answer the problem themselves. Then they would observe a trained interviewer working with a candidate on the challenge twice. Then a trained interviewer would watch them working with a candidate on the question themselves.

Only once all those steps were completed could the person interview candidates. Any trained interview could interview for any team hiring for a role in that discipline. There was no level required for an interviewer.

The Interviewer Panel

We made sure that there was at least one person from an underrepresented group on any interviewing panel. This requirement was challenging at first. Some of the interviewers ended up having many interviews to do.

We felt it was important for two reasons. If we were interviewing someone from an underrepresented group, they would see someone like themselves and would be able to ask questions (if they wanted) about their experience. If we were interviewing someone, not from an underrepresented group, we would potentially catch any red flags that indicated that they might have challenges in a diverse environment.

This practice turned out to be very valuable.

The Interviews

As the interview training required shadowing, almost every interview would have two people from the interview panel. One person worked with the candidate, and one observed the discussion. This pairing meant that the hiring manager got two interpretations of what happened in the meeting and how the candidate responded to different questions. Multiple perspectives were beneficial in reducing the effect of any interviewers’ biases.

We offered to let candidates bring their tools if they chose so that they could be comfortable rather than presenting the candidates with an abstract problem and having them solve it on a whiteboard or a company laptop. The interviewer and the candidate solved the technical challenges collaboratively. This approach mirrored the way that many teams at Avvo worked in a pair-programming style. The goal was to recreate, as much as possible, the actual working environment of our teams.

Gaining Credibility as an Inclusive Technology Employer

Improving our interview process alone wouldn’t make a significant difference if Avvo was not visible in the market as a company serious about diversity and inclusion.

The team already had multiple people volunteering at the Ada Developers Academy but had never hired an intern. We hired our first two interns from Ada in their next cohort.

We followed that by joining the new Washington Technology Associate Apprenti program (a registered technical apprentice program focusing on underrepresented groups and veterans). We hired two apprentices from the first cohort of that program as well.

We always hired interns and apprentices in pairs because we wanted to make sure that they would have someone else going through a similar experience that they could use for mutual support in addition to their Avvo mentors.

Working with these programs let us partner with people deeply enmeshed in the community. We wanted to learn and listen. Working with them also introduced us to their volunteers, often experienced developers from other companies who felt strongly about increasing representation in technology.

Along with our partnership with training programs, we made sure we attended the local meetups for underrepresented groups. Either one of our recruiters, engineering managers, developers, or I would attend these meetups to listen and understand the challenges that these groups faced in the industry. We networked as well, but only tentatively at first. We wanted to establish credibility and not just come to a single meeting and disappear.

The sustained efforts to become a part of the community taught us a lot and raised awareness of what we were trying to do at Avvo. Candidates would apply and mention the different people that they had already met at the company. Avvo was already hosting meetups for different technologies. Those meetings eventually started to become more diverse as candidates from underrepresented groups visited to see our offices.

Evolving the Culture

The focus on increasing diversity did not have universal support. When I joined, the technology team culture was not very inclusive, and some were resistant to making any changes.

I continued to educate and discuss, but I also continued to push firmly forward.

Our agile coaches and I taught new, more inclusive facilitation techniques for meetings and discussions. One of the senior managers started a mentoring program for everyone in the organization designed to support everyone, but especially those who might feel impostor’s syndrome.

Eventually, those who were not interested in the new culture decided to find other opportunities. In the end, this was a relatively small percentage of the team. Most were interested in being part of the new culture.

The Results

In under two years, we increased the percentage of women in the technology team from 17% to 27%. We increased the percentage of Black, LatinX, and multi-racial people from 2% to 11%. Our age diversity also improved significantly. We did this while increasing the size of the team by nearly 50%. We also significantly improved our employee net promoter scores during this time.

If one thing stands out to me about what we accomplished during this time, it is a question asked by one of our developers during a town hall right after our acquisition. The town hall was hosted by the acquiring company’s executive team to answer questions from their new employees. The Avvo developer said, “we are very proud of the diversity of our technology team at Avvo. What efforts have you put in place to make sure that your team is diverse?”

Acknowledgments

There are several people mentioned in this article by title. I do want to acknowledge them here as this was always a team effort and would not have been successful without their involvement and leadership. Specifically: LaQuita Hester, a fantastic recruiter dedicated to improving diversity in the technology industry; Hunter Davis, Director of Engineering and the creator and guide for our mentoring program; Justin Weiss, Director of Engineering; and Leslie Zavisca, Engineering Manager.

Many, many others were instrumental in ways large and small. Every developer, tester, data scientist, and manager helped.

I also want to acknowledge my executive team peers who had built an excellent company and supported me as I brought my team up to a level of diversity closer to what they had created. My executive peers were Mark Britton, Eric Dahlberg, Monica Williams, Bhavani Murugiah, Sachin Bhatia, Kelly McGill, and Jason Moss.

[This article is a repost from https://blog.kevingoldsmith.com/2020/06/12/changing-hiring-practices-to-build-a-more-diverse-technology-organization-a-case-study-from-avvo/]