Measuring the Effectiveness of an Outsourced Team
Ensuring the effectiveness of an outsourced team is critical to the productivity of individual teams and the success of a project.
Any scale-up that decides to optimize costs and decrease time-to-market by hiring an outsourcing IT services vendor needs to know how to measure these suppliers’ effectiveness.?
With more than 160 projects under our belt and a decade of working with the US tech industry, DevPro has plenty of insights into measuring effectiveness.? .
The Difficulties of Measuring IT Services Outsourcing Vendor Effectiveness
There are several challenges to measuring the effectiveness of a vendor, including:
Major Metrics Used in Software Development, Deployment, Operations
Recently, formal metrics like Lines of Code and code complexity are being replaced by multi-dimensional metrics. Agile productivity, operational and quality metrics are? now being used to estimate how well a development team is working.
Developer Productivity:
Line of Code [LOC or source line of code] remains the most popular unit of measuring a developer’s productivity. However, if used as a KPI for productivity, it can be abused by unnecessarily increasing the number of lines of code to the detriment of the project’s quality.
Technical debt is not easy to explain and is difficult to? calculate. Also known as the code debt, this is the sum of technical imperfections that accumulate as the project grows. Refactoring is needed for the optimal functioning of the system under new conditions, but this process is often postponed as a result of budget or human constraints. If left unattended, it grows exponentially and is likely to cause issues.
Code Ownership
Code ownership is a measurement defining percentage of lines authored by a developer in a specific repo in a specific time frame. There are different levels of ownership depending on the level of contribution to the code: those who introduced <50%, >50% or >60% of changes respectively can be named accordingly: minor contributor, major contributor and a file owner.
According to a survey , a strong ownership structure is beneficial for file quality, so weakly owned files and directories could be assigned owners to drive quality.
Code Churn
Code churn is the proportion of code that’s rewritten or reworked not long after it’s written to the entire written code in a unit of time.?
It changes from project to project and from programmer to programmer, but an average of 20-30% should expected. Higher churn rates are expected at the beginning of the project when visibility is low and the architecture is being tested.
Dramatic deviations in the churn rate of the same team within the same project are worthy of attention.
Code coverage
Code coverage allows users to estimate the percentage of code that has been covered by manual and automated tests.
It’s a more objective measure than test coverage and has multiple subcategories, like function coverage, statement coverage, branch and loop coverages.
Agile Process:
Lead Time
Lead time is the time from ticket creation to full release of the feature according to the definition of done.
Lead time = Reaction time [lag] + Cycle time
Cycle Time?
Cycle time is the time it takes to complete a task, from the moment it goes into production until its release.
Both metrics (lead time and cycle time) are great tools for planning and forecasting, especially for similar projects. Control charts are based on these metrics.
Velocity
Velocity is a team's ability to complete a number of story points within a certain timeframe. Average velocity is calculated by dividing the number of completed story points by the number of sprints.
It helps to plan and forecast teams work accurately with known velocity for past periods.
Focus Factor
Focus factor gauges the engagement of the team into delivering of the value in a unit of time, it is derived by dividing velocity by capacity (total engineering time available for a sprint).
A focus factor of less than 60% signals excessive team distraction or other organizational hiccups.
Commitment Reliability or Say / Do Ratio
The commitment reliability is the difference between committed points and the delivered points. Measured in story points, the graph provides an overview of the discrepancies between what was agreed upon and what was produced as deliverable.
Operational and Maintenance [DevOps]:
Mean Time Between Failures [MTBF]
Mean Time Between Failures is a metric that defines the reliability of a product. It is calculated by dividing the time in use without breakages or stoppage (uptime) by the number of breakdowns.
Naturally, the longer the MTBF, the more reliable the product is and the less resources you need to fix errors.
领英推荐
Mean Time to Recover [MTTR]
Mean Time to Recover/Restore defines how much time on average it takes a system or feature to go from downtime to fully operational.?
MTTR is calculated by dividing the total downtime by the number of incidents.
The quicker a DevOps team is able to recover the software product, the less damage is incurred by business.
Mean Time to Resolve [MTTR]
Mean Time to Resolve is the average time needed to fix the bug and take steps to prevent it from recurring. So it’s the Mean Time to Recover from full stop to full operation PLUS the time spent by the team to make sure this bug does not happen again.
Mean Time to Repair
Mean Time to Repair is usually used in the same meaning as Mean Time to Recover, but applies to physical and mechanical fixes of hardware like servers and disks. The testing period is also included in this metric until the full functionality is restored.
Cost of Downtime
According to Veeam Data Protection Report 2021, based on a survey of 3000+ IT professionals, the average hourly cost of downtime is $84,650.
The cost depends on the size and scope of your business - be it ecommerce, FinTech, agriculture or other. Facebook lost $90 million in a 14-hour downtime period in March 2019.
To calculate how much your company could lose per minute or per hour, divide the cost of total damage incurred by the minutes or hours the systems were down.
Deployment Frequency
Deployment frequency is a basic DevOps metric that is used to calculate how often portions of code get deployed.
The best practice is to deploy code often to reduce the quantity and length of duration of the downtime. The Amazon DevOps team deploys code every 11.7 seconds, for example.
Quality & Testing?
Defect Leakage / Escaped Defects / Defects Found in Production
The total number of defects found after the sprint closure and / or on a client side. The later the errors are discovered within a cycle, the more expensive they are to fix, so this metric is vital for the client.
Percentage of Automated Tests Coverage
This metric tells you what ratio of your conducted tests is automated. As time passes, the percentage will grow, signifying the better quality of the output.
Code complexity
Code complexity is a measure that allows us to estimate the percentage of the unnecessarily complex code in the total code. As there are multiple ways to develop the same function, an easier shorter code is always a preference. As code is a multi-user medium, the simpler it is, the easier it is to use, fix and maintain for all.?
Defect Spillover
Defect spillover is the number of defects that don't get fixed within one sprint and are carried as a technical debt to another sprint. Keeping an eye on the metric helps ensure that bug fixing costs less.
Defect Density
Defect density refers to the quality of code and whether it's ready to be released. It’s calculated by dividing the number of defects by the size of the code [which can be either KLOC or functional area].
The Bold Print: Multi-Dimensional Metrics Show You the Bigger Picture?
These are the two key KPIs we monitor during our projects. The metric we use depends on the methodology of the project.
Velocity per FTE = number of completed SP per sprint / Number of FTE
Gross scope per FTE = Business & Tech Debt Completed / Number of FTE
Both of these datasets are then compared to the previous month and the averaged figure for the three months prior. Any deviations in excess of 10% either way will be analyzed to determine if further action is required.
Understanding How Effective Your Outsourcing Team Is: Where to Start
When embarking on a road to outsourcing, you want to be sure that you can measure the output of your team.? A transparent relationship between you and your tech partner will strengthen your relationship and reinforce trust.?
On a grand scale of things, while knowing every single one of development metrics is useful, it’s not necessary to micro-manage and drill down into each of them for the client. Team leads and delivery managers should have the carte blanche to leverage those tools in order to deliver upon the contractually agreed scope of work.
Many of DevPro’s clients create reports for measuring productivity or speed of delivery of the internal team of engineers. When we work with those clients, we agree to extend similar reporting to the outsourced part, as well.
If you don’t want to take such a time-intensive approach, you can just keep an eye on the Velocity per FTE or Gross Scope per FTE. Usually, our Project Managers would send this report with explanation of any deviations exceeding 10% to the compared period.
Are you looking to hire an IT services outsourcing vendor? Talk to our sales team and receive an estimate within 24 hours.?
….
The full article is published at: https://dev.pro/insights/measuring-the-effectiveness-of-an-outsourced-team/