Secret Sauce Of A Great Team - 2/4
How We Did It - Trait Surfacing Pre-interview Test
One thing that I never really agreed with in technical interviews was the time boxing. It does not reflect real world situations at all and gives a major edge to luck and memory. If the candidate was lucky enough to have recently worked on or studied a particular topic and that topic comes up in the interview then the stars align and the candidates chances of impressing us shoot way up. If you go on enough technical interviews you will build up a list of typical technologies, questions, and answers, most of which can be researched and remembered for later.
How often on the job are you asked to solve a problem on the spot? Typically what happens is someone comes to you with a request, you have some time to look into it, some time to ask questions, some time to do it, some time to get it back to them. These are typically not 1hr things with people waiting outside of a room asking, ‘Are you done yet? Need more time?’ No, these are projects or tasks that usually take from a few hours to a few days to a few weeks.
So we gave our candidates a project to do and purposefully engineered pressure and circumstances to reflect real world conditions. About three days before the interview I sent an email to the candidate. We didn’t want to give them too much time but enough to feel some pressure. Some of it has changed but in any case we’ll dissect it afterwards:
If you have not done so already please complete the below exercises and send to me before your interview.
The purpose of this exercise is to better understand how you would approach various teams, understand their problems, and work with them to solve problems.
Our definition of DevOps:
DevOps is derived from the words, “Software Development + Operations.” DevOps is a term that was developed around 2010 and has become a popular word used to describe philosophies and methods that bridge together SW Development, QA, and Operations into a single approach, a single workflow. Although there is disagreement across the industry on the exact meaning of DevOps, there is broad consensus on the desired business outcomes - which we will explain in the next section below. Whether we call it DevOps or not the concepts have become integral to maintaining and enhancing competitive advantages.
At CompanyX our definition of DevOps is, “Efficient and sustainable methods of building, deploying, and managing software and supporting infrastructure at scale while addressing operational compliance and cost." In a word DevOps is efficiency. For a business this translates to savings and other competitive advantages.
If you’re still a bit fuzzy in your understanding of DevOps then give this short article a read. It should remove the fuzz. https://www.dhirubhai.net/pulse/understanding-devops-justin-franks
How DevOps measures success:
Speed, efficiency, mitigation, adoption. An important part of measuring success is by collecting feedback from engineers before, during, and after DevOps adoption. In most cases the business should see marked improvements in application time-to-market, reliability, performance, security, technical debt, mean-time-to-recovery, ease of support and administrative overhead. Some of these may be difficult to measure. For example, business risk/exposure due to security breaches, lower customer happiness due to delayed fix/response times, sharing of tools and workflows that save time which can be used to pursue more business opportunities, self-healing and scaling automation which helps meet customer demand and expectations.
DevOps Mission:
Improve, Simplify, Unite. In the context of Engineering and IT, technologies and methods are changing and growing exponentially in both quantity and complexity. Without ways to control and manage this our business will face various competitive disadvantages in cost, time, resources, and innovation. The mission of DevOps is to become a permanent part of the business model so that competitive advantages are kept at the highest levels possible related to the building, deployment, and management of software and supporting infrastructure (cloud, servers, networks, databases, etc.). To bring Operational best practices to Development. To make Operational support easy. To reduce tech debt. To increase developer efficiency. To improve cost ratios. To maintain governance. To increase quality. To unite DevOps across regions and business.
DevOps Strategy:
To bridge Operations (Technical) and Engineering (Software) technologies and methods so we continuously transform the building, deploying, and managing of software and supporting infrastructure (cloud, networks, servers, databases, etc.) into more sustainable, effective, and efficient models at scale. As technologies, methods, and business change so will our strategy. The strategy will follow a lifecycle so it remains robust, flexible, and relevant over time. To work with a designated member of each pod/team who will in turn, over time, help everyone in their pod/team adopt DevOps methods and technologies as part of how they do things so that the benefits of DevOps are realized by everyone in the pod/team.
- Build a core global team of DevOps Leads (DLs). Each region to have one or more DLs.
- Train, empower, support DLs. Maintain alignment between them
- DLs to help Designated Engineers (DEs) in various Pods get up to speed on technologies and methods
- DEs in each pod to help the rest of the team get up to speed as needed
- Measure impact on individuals and team
- Update DLs on current DevOps technologies and methods
- DLs to update DEs
- DEs to update team
- Realign DLs to fit trajectory of business. Repeat from Step 2 as needed.
Exercise 1:
DevOps members will approach and solve ambiguous and complex problems. It will require constant research, learning, and application of new things. It will require one to deeply understand the needs of a group, the technology, and the workflows. With all this in mind, please write up an initial draft of how you may approach various teams and help them solve inefficiencies that are consuming large amounts of their time. Also, please elaborate on how you may change the way a group does things, how you may change their habits in 1,000 words or less.
Exercise 2:
Please write a program in one language of choice that can run on both a Windows and Linux OS. You must not use more than one language. The program will output three things: 1) Total system memory. 2) Total system memory in use. 3) Total system memory available. Output the three things to some file on the local OS.
I really want you to rock the interview. We are looking at a few candidates and I want to do what I can to help you rock it. So I am going to list a few things you should research before coming in to interview. I'm really pulling for you...
1. Understand the concepts of the Non-Make-based tools section
Focus on Octopus and Apache Maven and be able to explain how they work and the differences between the two
2. Understand the concepts of the Continuous integration tools section
Focus on TeamCity and Jenkins and be able to explain how they work and the differences between the two
3. Understand the concepts of the Configuration management tools section
Focus on Chef and Ansible and be able to explain how they work and the differences between the two
4. Research AWS
Understand the EC2 service and how it works. How servers are launched in AWS and how cloud formation templates work
5. Research DevOps tools
Research HashiCorp products and be prepared to explain their purpose at a high level
6. Research PaaS (Platform as a Service)
Compare Heroku vs Elastic Beanstalk
7. Research Docker.
Be able to explain the concepts and how it works
https://www.docker.com
Dissection of intent.
First, did they even take the time to really read through everything and do the exercises? Did they invest the time? If they are genuinely interested in the joining our team then taking a couple of hours out of their life shouldn’t be a problem. Second, the essay told us if they understood DevOps and were comfortable with the role. It also exposed a bit about their critical thinking ability, communication ability, thought process, and people skills. Did they have structured thinking? Third, humility. Did they feel the simple exercises were “below” them? Fourth, work ethic. How much effort did they actually put into it? Was it just a “token effort” like flipping an extra penny from the pocket into a donation bucket, a “spare effort?” Maybe a paragraph or two instead of close to or over the 1,000 word limit instead of really engaging their mind and expressing their thoughts? Fifth, communication. How well do they articulate? Sixth, fluency in at least one programing language. Are they able to express moderately complex logic in at least one language that can be used to build an application -- a hard requirement for the role? Seventh, ability to learn and adapt. Given an almost impossible list of technologies and concepts which they have never heard of or worked with before how much could they absorb and apply to practical situations in a short period of time? Being an engineer means being a professional learner. If we are not constantly learning and applying what we learn in practical ways that benefit the business then we are failing the business. If all of these things checked out then it was simply a matter of giving them all the support they need and making sure we keep pushing ourselves to new levels.
Did everyone complete the pre-interview test? Not at all. You'd be surprised. Even some really Sr Engineers failed to do the essay for whatever reason. Either that or they just put forth a “token” effort, maybe a couple of paragraphs. I figure if they didn't care then I didn't care. What is interesting is that the more Sr Engineers aced the programming exercise but fell down on the essay. The more Jr Engineers and those from underrepresented communities aced the essay but didn’t program as well as the Sr Engineers. Easier for us to learn a skill than to adjust our attitude.
The essay is what told us the most about the person. It told us that Jr Engineers and those from underrepresented communities by far displayed more of the traits we wanted our culture to embody.
“Mental set” or “set effect” as well as “rigidity” were also things I felt were important to suss out. Being unbiased, open-minded, and holistic would be critical for our team. We couldn’t keep approaching and doing things using “that’s the way we’ve always done it” as a reason.
https://en.wikipedia.org/wiki/Set_(psychology)
https://en.wikipedia.org/wiki/Rigidity_(psychology)
“Rigidity can also refer to the tendency to perseverate, which is the inability to change habits and the inability to modify concepts and attitudes once developed. A specific example of rigidity is functional fixedness, which is a difficulty conceiving new uses for familiar objects…”
“Mental sets are subconscious tendencies to approach a problem in a particular way. They are shaped by past experiences and habits. An inappropriate mental set can hamper the solution of straightforward problems.”
“Mental sets represent a form of rigidity in which an individual behaves or believes in a certain way due to prior experience. In the field of psychology, mental sets are typically examined in the process of problem solving, with an emphasis on the process of breaking away from particular mental sets into formulation of insight. Breaking mental sets in order to successfully resolve problems fall under three typical stages: a) tendency to solve a problem in a fixed way, b) unsuccessfully solving a problem using methods suggested by prior experience, and c) realizing that the solution requires different methods. Components of high executive functioning, such as the interplay between working memory and inhibition, are essential to effective switching between mental sets for different situations.”
How We Did It - The Interview
This is where we wanted to line up the Pre-Interview Test with the Person. We needed to know if they matched. Were they really the person behind it? What was their thought process on the essay and the programming test? How well did they carry themselves? How well did they communicate without a computer interface? Dress, appearance, demeanor, poise, coherence, word choice, topic selection. How well would they represent the team and the business in front of others? Since we would be helping a number of people from engineering teams this is where we probed amiable.
I also wanted to exchange in some healthy debate and conflict during the interview. Some direct questions were a must. What would make you want to leave? Why do you consider others when you make decisions? If we invested large amounts of training and resources into you for 2 years would you leave shortly after? What if another company offered you more money after we had spent 2 years investing in you? Why would you want to stay? How do you work with a team? How do you resolve conflict? What if five engineering teams all had great ideas but only one could be used? How would you help everyone reach consensus? How easy is it for you to embrace change? Why is it important? How do you modify the habits of others? Yeah, some of the questions were intended to reinforce things in the essay.
After we probed into the person then we probed into their programming. This test had many sides to it. One side was to emulate a real-world condition where people have a day or two to write some code using resources at their disposal like StackOverflow, Quora, Github, peers, various communities, etc. The programming test may have sounded simple to an experienced engineer, but there were other things I was observing. I wanted to see as much borrowed code and imported modules/libraries as possible -- minimal “wheel reinvention” balanced with practical time consideration. In other words, did they spend 5 hours trying to find things online they could borrow when they could have just wrote it from scratch in 2 hours? One would think they could probably write everything by hand from a cold start in under an hour while needing to reference a bit of syntax and libraries/modules… and that is being generous. But most people seemed to ignore the part that said “that can run on both a Windows and Linux OS” and that opens up a whole new set of inconvenient questions and challenges. That was the part I was actually most interested in knowing their approach to. Some, very Sr engineers included, just didn’t catch the sentence for whatever reason and came back with a solution that only ran on Linux or Windows but not both. There were very few that came back with a solution that ran on both using one language and even then it required some explaining as to why some things didn’t work. Who was over-engineering? Who said, “You know what, this gets us 90% of what we want and it’s easy enough to bridge the remaining 10% with a band aid or two.”
We used some questions to expose the other sides of the test related to thought process and attitude. Why did you do it like this? Why not do this as opposed to that? What if we also wanted the program to do this extra thing, how would you work it in? Why did you choose language X over language Y? How could you make this or that more efficient? What is another way to do this or that? It’s interesting because technical questions can reveal so much about character and attitude. Some may get too defensive when their designs are critiqued, some too open to critique. In short, how do they handle critique? What I am looking for here is some pushback balanced with openness to suggestion and ideas. Are they confident in their design decisions but not so confident that their minds are closed?
After we had drilled into the programming test we asked them to ask us questions and in some cases I reversed the interview process and asked them to interview me for the position. This was probably one of the most telling techniques because the only condition was that they could not ask me a question they did not know the answer to or had an opinion on. In some cases people who were reserved and a bit shy became more aggressive and confident. What was interesting is that the questions they asked highlighted areas and topics they felt most comfortable with. So it was a good way to find their strengths. Most of the time I wasn't able to do justice to their questions but I enjoyed them throwing me to the mat because it was a great learning experience, each and every one.
How and why did they throw me to the mat? Was it to prove their strength and dexterity in a particular area? Or was it because they were passionate and wanted to engage in healthy debate? It was easy to tell. There are those who throw others to the mat to make a point and those who throw others to the mat so each can learn and grow.
It goes without saying that having others on the team interview the candidate for a team fit is also critical. We couldn’t do this in the beginning because we didn’t have anyone on the team. So people from a couple other teams helped us interview candidates. But after we had a few hires we stopped having people from other teams participate in our interviews and relied on our own team members. There is a reason for that. We noticed that having people from other engineering teams participate in the interview created turmoil at times. By turmoil I mean political issues for whatever reason that influenced our ability to hire who we wanted. Perhaps it was because they were viewing the candidate from the eyes of their own team and not ours. In any case, it indeed caused situations where people from other teams voiced concerns about the candidate that were not concerns for anyone on our own team. Situations where our entire team, myself included, really wanted to hire someone but it became difficult due to said issues. So to keep those types of situations from creeping up again and impeding our ability to bring the right types of people into our team we now keep external participants limited to Recruiting and HR. If Recruiting and HR feel it is good to have particular individuals outside of our team included then we rely on them to guide us with that. It has been smooth sailing ever since.
Continue to Part 3 of 4