A Comprehensive Guide on how to Become a Professional Technical Evaluator.
Being a professional technical evaluator requires a unique set of skills and strategies. During my experience as a technical evaluator, I successfully discovered the appropriate method to evaluate candidates. First of all lets answer the question: How does one become a Professional Technical Evaluator? Well, being able to understand first which are the most important skills a Senior Candidate should possess, how to conduct an interview and how to read and understand a CV. In this article, we'll explore the 10 most important skills a Senior Candidate should posses with examples on how to check those skills as well as some common mistakes which Evaluators should avoid. Without any further ado:
------------------------------------------------------------------------------------------------------------
Skills (in order of importance 1.x having equal importance):
------------------------------------------------------------------------------------------------------------
1.1 Adaptability / Capacity to learn new things. Find out if an candidate doesn't know some concept and check how fast is the candidate able to learn it and to apply it in a practical situation. See "Example problem 3.x".
1.2 Problem Solving. Give the candidate "Example problem 2.x".
1.3. Intelligence level. Give the candidate "Example problem 7.x". A smart candidate should be able to learn easily any new required technical skill as needed.
1.4. Code coverage and quality. Give the candidate "Example problem 6.x".
1.5. Critical Thinking and setting up the right priority. Give the candidate "Example problem 1".
1.6. Creativity. Give the candidate "Example problem 5.x".
2. Check experience / Seniority - Check the candidate's Portfolio and Past Projects: Review the candidate's portfolio and past projects. Look for innovative solutions, unique approaches, and out-of-the-box thinking. Check the candidate's versatility. Was he able to work on different business domains? Was he able to learn multiple different technologies, programming languages? Did he occupied multiple roles? Was he promoted Team Lead by multiple employers? Is he able to self train in order to do his job? Does he has any certifications / skill assessments? Does he understood the importance of each adjacent role in a company: QA, Software Engineer, Business Analyst, Product Owner, etc.?
3. Knowledge. As currently evaluated (definitions without application) it is the least important skill in an senior evaluation and also the only skill considered during the evaluation process by more than 95% of the recruiters. See "Example 8" which explains why it is least important and how to correctly evaluate the knowledge (by practical applications).
A. Communication - Give the candidate "Example problem 1".
B. Team playing - Give the candidate "Example Problem 4".
-----------------------------------------------------------------------------------------------------------
Examples:
-----------------------------------------------------------------------------------------------------------
Example problem 1: How to solve a problem for which he doesn't know the solution, his software engineer colleagues doesn't know how to solve the problem, and on the internet / in the manuals he cannot find any solution.
Does he attempt to break the problem into pieces? If a piece is not solvable, that he attempts to simplify the unsolvable piece by complicating a previous most simple piece? Does he try to generalise and to solve the problem for a particular case? Does he ask the product owner / business analyst if he is enough to provide a solution for a particular case and simplify the requirements instead of a generic case which may prove impossible to solve or which may not be relevant. Is the candidate organised? Which of the piece he will solve first the hardest or the easiest? If is the easiest, when he will reach to the hardest he may not be able to solve it, and then all the time spend on the easiest parts is bad management. The candidate should focus on the hardest piece of the problem first, because in the end if he is unable to solve it, he wouldn't have wasted time with pieces that are not relevant as the entire problem needs to be solved.
Example problem 2.1: Given an string formed with capital letters "AABBCCC", write a function that displays each character and the number of occurrences. A senior candidate should provide an O(n) solution.
Example problem 2.2: Given an int array, find the 2nd highest number. A senior candidate should provide an O(n) solution.
Example problem 3.1: Ask him a definition for example SOLID, if he doesn't know the definition, let him learn it and confront him with a source code that can be improved applying SOLID concepts on it. Is he able to do those improvements?
Example problem 3.2: Ask him to write a code in any programming language that parses file.properties present in all the sub folders of a root folder and to generate an excel report based on a certain properties in the file. Ask him how will he do it? Will he search on the in internet the fastest simple way to do it, and use a more suitable programming language such as Perl or will he use only his current knowledge of JAVA which may prove more time consuming?
Example problem 4: Give a scenario where the candidate should do research on the topic. What does he do after that research? Does he create a confluence page / Propose to have a common database such as Confluence of best practices to be used by all his colleagues? Does he share the knowledge with his team? Does he like helping / coaching junior colleagues?
Example problem 5.1: Ask the candidate to think the flow of an application (for example Service Desk Ticketing flow)
Example problem 5.2: Ask the candidate to write an audit system model for 2 tables such invoices and orders. Would he create an hard coded model in which he adds a new audit table for each given table such as invoice_audit, order_audit or will he create an dynamic model like audit_table, audit_column which he can reuse for all the existing and new to be required audit tables ?
Example problem 6.1: Ask the candidate on a given source code to mention how many unit tests he should write to fully cover the code. Is he able to discover a misuse of the function and to add relevant code such as throw IllegalArgumentException in the cases when the function was called with invalid parameters? This evaluates how efficiently does the candidate write his code, while taking into consideration all the possible scenarios in which the method could be used. A fully covered code will not break and it will not generate bugs / crashes into production. Does he write maintainable code? Does he uses any checkstyle and fix checkstyle warnings such as: Class Complexity, Ciclomatic Complexity etc.
Example problem 6.2: Check if he writes duplicate code or he writes a generic code. For example ask him to write a function which removes duplicate String elements in an ArrayList, and then ask him to remove duplicates in an Integer ArrayList. Does he write a single function using generics or he writes multiple functions?
Example problem 7.1: Give him an timed IQ test and see if he obtains a good score (for IQ lower than 144). Give him an IQ test without time limit for IQ higher than 144. For example for an IQ 165+ problem could be finding the next number (2 distinct solutions) in this series: 37, 8, 98, 1, 311, 821, 541, ? Even after 10 days, if the candidate is able to find those 2 solutions then that person is extremely smart as it is estimated that 99.998% of the population falls below an IQ of 165.
Example problem 7.2: Give him an thinking outside the box problem and see if he is able to solve it. For example uniting 9 equally distant points (3 lines x 3 columns) with 4 lines without lifting the pen off paper. This problem can be extended to 16 points, 25 points etc.
领英推荐
Example problem 7.3: Einstein's riddle: In each house lives a person with a different nationality. These five owners drink a certain type of beverage, smoke a certain brand of cigar, and keep a certain pet. No owners have the same pet, smoke the same brand of cigar, or drink the same beverage. Now to solve, tell me who owns the fish?
Example problem 7.4: Give candidate an practical problem and force him to use his mind in a way not previously engaged.
Example 8: This skill is usually verified by asking various definitions of concepts that can be easily assimilated and which do not have any connection with practical problems encountered in applications, nor does reflect any experience in solving of such practical problem. While this knowledge in itself it is important when hiring junior members, because juniors should know something, right? For seniors, all bets are off, because a senior that has all of the above qualities can easily learn new knowledge and adapt to any situation. The job of software engineering is one of continuous learning hence the irrelevance of the current candidate's knowledge. The assumption of "having knowledge of a definition = knowing how to apply that knowledge" is totally wrong. Having the knowledge of concepts K1, ..., KN doesn't guarantees that the candidate will know concepts KN+1, ..., KN+M required for encountered problems nor that he is able to learn it fast enough to be able to solve those problems. Neil deGrasse Tyson explained very nicely in his discourse on Knowledge vs. Thinking: https://www.youtube.com/watch?v=NwoohzjscHY
Instead of asking: Define what is "SOLID"? - pure theoretical easy to assimilate knowledge.
One should be asking: How do apply SOLID (and showing the candidate SOLID) to optimise this piece of code? - actual knowledge and the correct answer from the candidate proves he is able to correctly apply concept. Extra points if the candidate does the successful job without knowing this concept. This means the candidate was able to use his mind in a way not previously engaged, which is far more important than just knowing the definition of this acronym which doesn't guarantee that the candidate knows how to apply this knowledge.
Instead of asking: What is ACID? - pure definition only relevant when the candidate's job is to constantly create databases that must to adhere and respect ACID acronym.
One should be asking: How would you solve a performance issue on the database ? - actual knowledge of solving encountered problems.
Instead of asking: What is a REST API? - pure definition of which its knowledge doesn't help anyone solve anything.
One should be asking: How would you perform the CRUD operations on a database using REST API(s)? - knowledge that confirms the candidate knows how to correctly match the database operations with REST API(s) methods.
Instead of asking: What methods does class X have? - which can be easily discovered if a professional IDE is used.
One should be asking: Use class X to create me a program that does Y and Z, and through this exercise you can notice if the user knows which methods should use and which ones should he overwrite in order to do his task.
----------------------------------------------------------------------------------------------------------
Common mistakes
----------------------------------------------------------------------------------------------------------
The technical lead should read the CV and deduce the skills of the candidate. If he is not able to deduce certain skills, he should ask for details regarding a certain task which the candidate performed. The candidate should "read" his CV for the evaluator, evaluator should already know what it is written in the CV, or at least should have a copy of the candidate CV in his face while asking the questions if for some reason he cannot retain candidate's activity.
2. If 2 ore more evaluators are present at the interview they both should be "in sync" and do mistakes such as: not paying attention at the question his colleague asked.
3. A Technical Evaluator should be open to a more open to a practical interview which is closer to job for which he hires. This is based on the fact that the candidate will work with all the information required at his disposal including an internet connection. A solely theoretical discussion is only relevant for those ultra secure positions where the candidate doesn't have access to internet and should know everything without a connection to internet.
4. The Evaluation process should be a discussion and not an interrogation. Candidate should be allowed to put questions or to bring arguments counter examples etc.
5. The Evaluator should be able to recognise a valid solution to his own question. Not knowing the correct answer for his own question is something below junior level.
6. The Evaluator should always ask the concise problem, the Candidate should not guess what the problem is, as software engineering is not a "guessing science". It is an exact science. There are logs, there are results, there are certain steps that lead to a problem, and there is source code. So asking answers for a problem while offering incomplete hypothesis for that problem is just wrong.
7. The Evaluator should always have a report file with his set of problems / questions, and after each problem asked, a check on the problem should be made so that he doesn't asks the same question twice.
8. The Evaluator should pay attention to what the candidate says. It is really a bad interview when for example a candidate says he doesn't have experience on a certain topic such as JSF for example and the Evaluators keeps asking JSF questions.
9. The Evaluator should not focus his interview only on definitions, more important skills than the Knowledge skill are described in the Section: Skills, points 1.1 through 1.6.
10. The Evaluator should know that Seniority is not based on the candidate Knowledge (Fact measured by the ability of the candidate to answer certain definitions or not) or his experience on the company's required frameworks. In order to evaluate seniority read Section: Skills, point 2.
11. The Evaluator should offer an detailed feedback to the candidate.
12. The Evaluator should never be late on an interview. In the unforeseen situation that he is late, he should announce the candidate prior to the interview if there are any delays that prevents him to be on time.
13. The Evaluator should not ask inappropriate questions that are in direct violation with the candidate's signed confidentiality policy. For example details of how he integrated certain API(s) with exact method names etc.
14. The Evaluator should respect the candidate and should not allow a junior member of his team to "held the interview", performing such experiments should be between themselves and not with possible future employees.