The Future of Software: Building a Talented Team

The one thing I am perhaps most thankful of all in my career is the talented people I have worked with.  Mentors, colleagues, and people that have worked for me, I am truly blessed to have all those people in my life.  So I thought I would focus 2 posts on my thoughts on talent: what to look for, how to interview, how to retain, common mistakes I see in the industry, and lots of opportunity to get your thoughts as well.  Talent affects everyone, management and engineers alike, so I'm hoping this is a very applicable post.

In my opinion, talent is the most important item by far of the 3 T's required to create great software.  If it is indeed that important, then we need a good way to

1. identify skills required
2. hire talented people with that skillset
3. arrange the talent into healthy teams

Once you have a healthy talented team, in my experience magic happens every time.  You get tremendous quality software on schedule and on budget, it is a beautiful thing indeed!  That is my oversimplified approach to building a healthy talented team.  Once you have such a team, then the next task is growing and retaining them, which is a completely different challenge we will defer to the next post.  So let's focus first on building the team.

In an ideal world, I would just look for the most talented people I can find and hire as many of them as I could, no matter what skills they have and no matter where they are located.  Talented software developers can pick up any technique and any technology required.  I've seen that time and time again in my career.  They can do it remotely, in different time zones, or in the office.  I see limiting the talent pool you draw upon as one of the most common mistakes in the industry.  As long as people are willing to travel every now and then for team meetings, and you have a travel budget, I don't think everyone needs to be in the same physical location or even time zone.  Of course that won't work if you need to be in proximity to physical hardware/devices/labs.  But if it is a pure software job, I would open every position nation-wide if at all possible.  

Unfortunately we don't live in an ideal world, we have schedule, budget, location, and all kinds of other organizational constraints upon how we do our jobs.  Therefore you really can't proceed with step 1 until you understand your organizational constraints.  That is your homework assignment before you even get this process started.

Step 1: Identify Skills Required

Part of defining the skill set is easy.  No matter what software you produce, you need coders and you need quality assurance (QA).  By now I think we can all agree that it isn't a good idea for the people that code to also test the software they code.  Coders of course create automated tests for their code and also perform at least first level testing of their code (often called unit testing), but soon thereafter it is accepted good practice to turn it over to people with a specialized skill set for QA.  I would say without a doubt the skill I have come to respect most of all in software development is quality assurance.  Good QA folks think differently than coders.  And the better the QA person, the more they think differently.  Good coders focus on most efficient use of the technology at hand.  Good QA staff focuses on most effective use of the product.  These two focus points are often at odds with one another.  You know you have put together a great team if QA and coders are constantly debating system design issues!  The end result of that collaboration is something special.

Next up is sizing the team.  The larger the team, the more levels of experience you will need for coders and quality assurance.  A software architect (who can code as well) is always a good idea if you can afford it.  The same goes for a quality assurance lead.  These roles need to be equipped to subdivide the work intelligently so that the most difficult work goes to the most senior people but the bulk of the work is possible and appropriate to be executed by the more junior team members.  If I can afford it, I prefer a 1 quality assurance to 2 engineer ratio.  But I count myself lucky if I can staff 1 QA to 4 engineers. 

Another tip on QA staffing, have your QA team do the build/installation/configuration/upgrade work as well.  Some QA folks get very offended when I suggest this, they say that isn't their job.  The engineers can do that.  In a subsequent post I'll go into detail about operational issues.  Most operational issues come from an oversight where QA is not involved in this critical process.  Engineers tend to think everyone knows all the ins and outs of a technology and can deal with any installation or configuration problem that comes their way.  They don't typically understand the skill set of operations nor the extreme pressures and cost constraints of that staff.  So they will skimp on error handling, user interfaces, eliminating redundancy (I've seen far too many products that have the same setting replicated to hundreds of files or database rows which causes enormous problems when operations needs to update that setting).  QA will fix this problem, they won't let these oversights out the door.  If the product is so difficult that a QA skill set can't install/configure/upgrade the product, then it is too difficult for an operations staff as well and the engineers need to spend more time on that before release.

Now as soon as you have coders and quality assurance attacking a set of requirements, you will need management in some form.  There are lots of different forms of software development project management, traditional waterfall variations using product management, requirements specifications and gantt charts, or the more common agile management variations using business representatives, stories and iteration plans.  I have had great success using both forms of management, and great failure using both forms.  So that is why I deem technique to be less important than talent.  Technique is often imposed upon you by your organization and I think that is just fine, I don't fight technique battles much these days.  Identify any technique required by your organization and make sure your talent accepts that as a course of doing business.  Staff roles that can perform the management tasks required by your organization in addition to their obligations to help the team create great software.

Next up is the project specific set of skills you need.  Most projects require a user interface of some sort (although not every project does, some projects create APIs, libraries, services, or frameworks which are largely without user interfaces).  But don't make the mistake I have made in the past, user interface design is a special skill set and critical to the creation of great software (if that software has a user interface).  So if that is a requirement of your software, look for people with that specific talent. 

Persistence (database) skills are also often neglected as well.  The database marketplace is a whole lot more complicated than it used to be.  Years ago it was much simpler, each organization made an investment in a particular SQL vendor, hired a team of DBAs with experience in that vendor's implementation, and you allocated time from one of those DBAs in your project.  Today we have all sorts of purpose-built NoSQL databases out there.  You will typically need to identify pretty early on in your project what database technologies are required to efficiently and effectively power your system.  If you have a high volume system and follow a micro-services architecture, you will need someone with experience across many different technologies to guide you to the most appropriate database technologies to fulfill the search, processing, and persistence needs of each service.

There are many specializations of each type of skill above, and many other skills as well that I didn't get into.  So I hope I have not offended anyone by not mentioning their skill set.  This is just the basic set, the principle here is to think through what skills you need to be successful under your requirements and constraints.  Keep that list handy every time you plan a budget, talk to HR about staffing, and for certain every time you interview someone.

Step 2: Hire Talent

Now that you have identified the skill set, you will need to staff the team with people talented in those skills and do so in the particular quantities required to stay on schedule.  This is all based on my highest premise for staffing talented teams:

• People have different innate talents
• People in general tend to be happier, more productive, and produce much higher quality deliverables when they are exercising their talents at work on a daily basis

Now this won't apply to every person and every situation.  But this is what I have found in general.  So rule number 1 I have for hiring a team is identifying what people like to do at work.  Per above this tends to be what their talent is and tends to keep them productive and happy as well.  If what they like to do is a match for my list of skills required to meet project demands, then I try to figure out a way to hire them.  One of the best jobs I ever received happened just this way.  I interviewed for a job, it wasn't the job I really wanted, and the interviewer had the sense to ask me what I really wanted to do.  He said that's great, it isn't this particular job, but I think we have another opening that matches this perfectly.  They hired me for a job I was really interested in, not the job in the job description.  I thoroughly enjoyed that job for as long as it lasted.

Unfortunately this is backwards to how most people go about the hiring process and I really think that is a mistake.  The first and sometimes exclusive focus in the industry is to put out a job description and then test each person against that job description.  Companies have elaborate tests for coders, architects, project managers, and so on.  You pass the test, great, you probably get hired.  You fail the test, you don't even go through to the next round of interviews.  I am not a fan of this approach for several reasons:

1. Tests can be studied for.  It is very difficult to devise a test to understand a person's talent.  Tests are devised to measure knowledge, not talent.  A motivated but untalented person can study for a test and pass it.  There are tons of guides to go about studying for interview tests.  I look for talent, not knowledge.  Talented people can very quickly learn whatever knowledge is required, knowledge specifically applicable to your project.
2. Passing a knowledge test does not guarantee the person can deliver what is needed for your project.  Most of what is on a knowledge exam is not required in order to deliver your project.  What is always required is the ability to size up the requirements of a project and translate them into quality deliverables specific to your requirements.  When someone devises a test to measure software development talent I'd love to take a look at it!
3. Knowledge tests do not typically measure motivation.  Is the candidate motivated to pass the test and get the job just for the money, the experience, or the contacts they will acquire?  Or is their motivation to build something special?  I want people on my team that have a passion for building software solutions.

If your organization has knowledge tests available, or even mandates them, I think that is fine as long as you don't use it as an elimination criteria.  I would not eliminate a person because of issues with a knowledge test.  I'm much more concerned about learning what their innate talents are. 

How do you identify a person's set of talents during an interview?  Quite frankly I don't know.  In my experience you can really only see them after working with a person for some time, giving them a wide variety of assignments, and see their resulting quantity and quality of deliverables.  That won't happen in the course of a typical 60 minute interview.  So what can you do in that time?  Here are some ideas for you:

• I don't like to start off an interview with the question of asking people what they like to do.  At the beginning of an interview, people tend to be nervous and clouded by a set of motivations that may not be in your long term best interests.  They may really need this job for financial reasons, they may really hate the job they are in now and want out, or they may want a shorter commute.  If you ask that question first, up front, you tend to get bad answers.  They answer how you think they want you to answer in some form of the job description.  They want this particular job and want to make themselves look perfectly suited for this job.
• A good starting place is to focus on what they have done in the past.  this helps for several reasons.  First of all, if the candidate is nervous, and most are, you don't want to hit them with really tough knowledge questions they may not know.  You want to put them at ease so you get the most out of this limited time you have to spend with them.  And the best way to put someone at ease is to have them talk about themselves.  They know what they have done, they won't fail at answering this question, and most people love to talk about themselves.  Now don't zone out during this process.  You can learn a lot about a person's innate talents looking for non-verbal communication in the midst of their verbal answers.  As a person walks through their work history, it is almost inevitable that they get excited about particular jobs and work assignments and less so on other jobs and work assignments.  That's what you need to know: what people get excited about and what they do not get excited about.  If they gloss over a big area of work or big assignment, that may tell you something, they may not have liked doing that work.  Or they may be nervous and just forgot about it.  Ask them about it and gauge their reaction.
• Once you feel the person you are interviewing is at ease, their answers flow, and you start to get a feel for what they like doing, then you can ask knowledge related questions about the work assignments they got excited about.  You do need an assessment of how much skill they possess, you need to ask those questions during the interview.  But in the beginning portions of the interview, try to tailor them towards their most exciting and successful assignments.  Also be careful not to spend the whole interview on what the candidate has done in the past.  Like I said, people tend to love to talk about themselves and they could consume the whole time slot just doing that.  Limit this whole process of talking about what a candidate did in past jobs to about 1/4 of your time slot.  I have had candidates that go well beyond this time, even as I coerce them away from it.  I take that as a red flag that they will be more interested in themselves than in what the team is doing.  Every member of your team has to be motivated in the team deliverable, not the individual deliverable.  Watch out for how a candidate phrases his or her work history.  Some companies even have an interviewing rule where you only talk about yourself, not your team.  I am not a fan of that approach at all.  I want candidates who talk about their team accomplishment and how they contributed to that accomplishment.  Great software is made by teams, not individuals.
• Next up I like to talk about the job at hand.  You have some idea what they are talented in and excited about as you explore their background per above.  Match that as best you can with the positions you have left open.  It may not be the position they are interviewing for and I'm just fine with that.  Your organization may not though, so be careful and understand how your organization's hiring process works before you make this kind of deviation.  But when you can, and as appropriate, switch to the best role for this person.  Describe the role and see if they are interested and excited about it.  Hopefully they ask a lot of questions.  If they are excited about this role and ask good questions, then you need to switch into knowledge assessment mode.  The candidate shouldn't be too nervous by now, they should be excited and properly motivated for the position, and you can ask them tough questions.
• At this point most people will go to lists of difficult questions published on the internet.  Again, I'm not a fan.  I go about this a different way, admittedly in a bit sneaky fashion.  But I have found it works very well.  I like to go into a pretty detailed and at times deliberately boring lecture on the requirements and high level architecture of the project.  Then during one of the more mundane parts of the lecture, I ask them a question on what I covered.  I put numbers in the question so they need to do an elementary math calculation (for example, a simple multiplication of 2 numbers I stated in the question).  The question can't tell them to multiply the 2 numbers of course, they just need to understand from the lecture and system design it would require them to make that multiplication in order to get the answer.  It could be addition, subtraction, division, whatever fits a system requirement that you think is key for them to have understood from the lecture you gave them.  I get one of 3 answers to this question:
• 1 - they ask me to repeat the question.  That tells me they were not paying attention and really may not be interested in the position.  It goes to motivation.  If people are truly motivated, they will listen closely.  If things get boring or they get lost, they should stop me and ask questions instead of zoning out.  I love it when people interrupt and ask questions, to me it shows they are very interested and have a great desire to absorb information about their potential next job.  And in particular, absorbing the requirements and architecture of a project is a skill that everyone on a talented software team must have.  If they can't do this in a 10 minute portion of an interview session, how are they going to do it as a critical portion of their job?
• 2 - they paid attention to the question but did not absorb the information I just told them.  They can't answer the question.  So they ask me questions about what I just told them so they can come up with the answer.  While this is better than #1 above, it still isn't ideal.  I'd much rather have the candidate interrupt me as I lectured, ask questions, and absorb all the information required so that they can answer this question with ease.  Now for some people that may not be natural, it may even be against their culture.  So be careful to take that into account.  If they ask the right questions now and deliver the correct answer quickly thereafter, I try not to count it too much against them.
• 3 - they state the answer almost immediately.  This is when I know I have encountered someone very special.  They paid attention and understood the primary requirements and design of the system.  They remembered the 2 numbers I told them in the question (most people don't and ask me to repeat the question and the numbers in the question a few times).  They knew to make the rudimentary multiplication calculation of the 2 numbers I gave them in the question and most of the time they think it is a trick question.  Why would I ask something that simple to just multiply 2 numbers?  This is what I'm looking for.  If people stumble on the question, ask them why.  They could be thinking it is a trick question which is fine, most people running interviews ask trick questions!  Just tell them it isn't a trick question and ask for the answer.

By now you should be over 1/2 the way through your interview time slot.  You have a rough idea what their background, experience, and skill set is.  You gave them a description of the overall requirements and architecture of the project at hand.  You have an idea as to their interest in the position.  From here you can go into a lot of different directions.  If the position requires a great deal of experience, you need to ask knowledge questions or questions about exactly what they did and how they did their work in previous work assignments. In particular I love to ask them why they designed/implemented/tested something in their assignment in they way they did. 

"What" types of questions are knowledge questions that can be studied for.  "Why" types of questions are much more difficult.  To answer correctly, you typically have to do the work.  So I tend to let the candidate tell me what they did and I ask them why they did it.  If they have good answers, then they probably did the work themselves.  If they don't, someone else may have done the work and perhaps they just came along for the ride.  You want people on your team that have made decisions, went down an implementation path according to those decisions, and learned from the consequences of those decisions.  In particular I look for people that are honest and say they made the wrong decision in retrospect. That happens in every non-trivial software development project.  You simply won't have time to research every possible path you could take, implement and test all of those paths, and just go forward with the best one.  You will only have time to take one path, you make your best guess on that path, and sometimes you guess wrong.  Understanding you made the wrong choice is an excellent sign of maturity!

Another path at this point is to delve deeper into what they like and do not like to do on a regular basis.  Or the candidate may have a lot of questions for you.  In particular I love it when candidates ask me a lot of questions about the lecture I just gave them in relationship to the work they would be expected to do.  That is excellent sign that they are genuinely interested and motivated for the job.  Look for questions in relationship to what they would do, not just questions about the project overall. When a candidate comes to an interview and doesn't have a lot of questions, that is a red flag for me.  Personally I don't take job changes lightly.  I want to know a lot about my next job before I make a decision.  But that's me, others of course may not think as I do.  But if they don't have many questions in an interview I keep that in mind as an issue of their overall motivation for pursuing this job.

Step 3: Arrange Talent into Healthy Teams

Now for the hard part.  So far we discussed step 1, identifying skills, and step 2, hiring talented people with those skills.  Most development managers think they are done, they just stop there and set out on a death march to deliver the project.  However, the best development managers know better.  They continually repeat steps 1 and 2 until they get a "healthy" team.  My definition of a healthy team is a software development team that consistently produces quality software on time and within budget.  Quite a few very well respected people and organizations (Grady Booch and SalesForce.com for example) also add a 3rd criteria for a healthy team, you must define a regular release cycle that you never miss (not by a month, not even by a day).  I have not had the pleasure of working in an organization that would allow me to adhere to the constraints of a regular release cycle, so I'll defer to those folks to extol the virtues of that approach.  If you don't have a definition for a healthy team in your organization, create one.

If you just stop after step 2, then you have a team of talented people.  And they will produce fantastic software.  But they won't do it on schedule and budget.  They won't be able to predict how long it will take to fulfill a set of requirements or when they will be ready to release.  This will be frustrating for everyone involved, in particular frustrating for your talented software development team.  If you don't address it, the team will disband.  Your organization may not put up with the delays or budget issues caused by the team.  Your team tends to be unequally overworked when they aren't healthy and people will leave under those conditions if there is no resolution in sight.  Your team may get frustrated and decide to move on to a healthy team where their talent is recognized as such because they always deliver on time and within budget.  Organizations don't tend to respect software teams that release late and over budget.  Whatever the case, if the team isn't healthy, they generally don't stick around too long whether by choice or not.

Some people think you can create a team that is healthy right away and take it out on the development manager when they don't.  In my experience that is not possible.  First of all in step 1 you take a guess as to the skills required and the quantity of each skill required.  You won't know for certain until you actually develop the software, finalize the technologies, and get release 1 out the door.  Then you get a flood of feedback from sales, customers, and technical support.  Massive numbers of requests for new features, performance and scalability issues, and defect fixes to prioritize and address.  By the middle of release 2 you have a very good idea what skill set is required to be successful.  Also, by then you have also sized up your talent in comparison to what is required.  You can only tell so much in interviews, sometimes you get it right, sometimes you do not.  Don't expect to be perfect, do the best you can under the circumstances and make staffing adjustments along the way.  For each adjustment, keep in mind everyone's innate talents and move them in a path where they will be increasingly exercising those talents on a daily basis.

Also, most organizations cannot hire employees quickly enough to staff projects.  You may also be dealing with contractors with high turnover rates, displacing contractors with employees, or moving geographies off shore.  All of this needs to be taken into account to transform your talented team into a healthy talented team.

As best you can under your organizational constraints, simply iterate steps 1 and 2 until you achieve your organization's goals for a healthy team.  Talented teams are flexible teams, they are open to change as long as the change moves them in a healthy direction.  The process is simple but the result is rare.  It requires the cooperation of your organization to fund and staff the talent required to be healthy.  It requires the success of your software in the marketplace to achieve that level of funding and management support.  And as we have seen and discussed before, the best software is often not the most successful software.

I have worked in about 40 software development teams in my career and I'd say only 3 have been what I would consider healthy.  I have had several colleagues that have never been part of a healthy team.  It may not be in your control, which is very frustrating.  But if you can pull it off, you will remember it as a highlight of your career.

Summary

Great software talent always produces great software in my experience.  In this post we discuss how to create a talented software team and then iterate upon it to get it healthy.  First you need to identify organizational constraints for the team, then identify the skills required under the constraints identified in order to meet software requirements and deliverables.  Next you staff talented individuals with the skills identified.  During interviews, spend the first 25% of the interview getting to know what a candidate has done in the past to get them at ease.  Spend the next 25% of the interview in lecture mode with a simple but relevant question thrown in to see if the candidate is really paying attention, interested in the job, and talented enough to process the requirements and design of the intended system.  Spend the last half of the interview as required to assess the candidate's level of talent in relation to their interests and the set of skills required to staff your team.  Finally, iterate this whole process over time and releases of your software until you have a healthy team.

In the next post we will discuss how to retain this tremendously talented healthy team.  You may not get a talented team in your career.  You certainly may not get one that is both talented and healthy.  But when you do, you need to work very hard to keep them!

This approach has worked for me in the past, but it certainly isn't the way or the only way to develop great software.  I really look forward to reading your comments, I'm sure everyone's experience is different on this important topic.  Thanks much for reading!

   Todd