Agile Assessments – How to Assess and Evaluate Agile / Scrum Projects

A common problem many Agile / Scrum adopters have is how to assess and evaluate their projects. This situation arises when auditing organizations transitioning from waterfall approaches, as well as organizations that wish to improve existing practices along the Agile / Scrum value chain.

We have all seen this situation before: You argued for Agile / Scrum, you convinced your supervisors, executives, and peers, you justified the training and transition cost, you retooled and adjusted current processes, you hired consultants and new employees experienced in Agile / Scrum, etc.

But, maybe after the first release of your product or after the tenth project wrapped up, somebody is going to ask the inevitable questions:

• “Are we doing better than before?”
• “How are our projects doing?”
• “Are we doing better than last year?”

These kinds of questions are healthy. As I have pointed out in “How Using Agile / Scrum Solves All Your Problems – NOT!”, simply adopting an Agile / Scrum process will not guarantee that you actually deliver software better or faster.

Because you want to continuously get better at what you are doing, you need to know what areas to focus on. It is important to understand key improvement areas for the next time around or the next budget cycle. Especially considering yearly budgets, you need to know where to spend your money wisely, so you spend it where it counts most.

When moving into an Agile process, there are three times in an agile adoption process you might want to ask these kinds of questions:

1. After the initial Agile / Scrum pilot project was finished – to assess how well that pilot project performed
2. After the rollout of the Agile / Scrum process to multiple projects or across the organization – to assess how projects are doing compared to each other
3. At regular intervals for each project that adopted the Agile / Scrum process – to assess key improvement targets on a regular basis

But even if you already adopted Agile / Scrum years ago, measuring where you stand will help you assess where you are and where you need to go – so regardless, going through an assessment is a great exercise because it summarizes and visualizes where you stand.

How to go about an Agile / Scrum Project Assessment

The thousand mile high view is that you want to get to a radar chart that shows key assessment areas like this:

This chart simply shows 34 assessment items (organized in 7 assessment areas) on an assessment rating scale of 1 (worst) to 5 (best). The assessment rating scale is defined as follows:

• 5 = Consistently followed in the spirit of Agile / Scrum Development; fully implemented role, artifact, process, or best practice the way it is intended in Agile / Scrum
• 4 = More of less consistently followed; mostly implemented role, artifact, process, or best practice
• 3 = Inconsistently followed or done incorrectly; role, artifact, process, or best practice is inconsistently implemented or followed, or used incorrectly (for example, calling regular meeting "Daily Scrums")
• 2 = Rarely followed
• 1 = Never followed / not adhered to / not implemented

The blue line indicates the area that would have been covered if each and every assessment area was a solid 5. The Orange line shows the area representative for the actual sample project.

This kind of radar chart is simple enough to generate in Microsoft Excel. Here is the example data used for the radar chart above.

The assessment areas and items in this table are a good starting point but there is no reason why you could not become more detailed and add additional areas or items as you see fit. Or, if you only want to focus in on Agile / Scrum process issues, you could delete some areas and items. This is simply a baseline for you to take and adapt as needed.

The following section will briefly review each of the assessment areas and items in turn.

1.   Roles & Responsibilities

Are the roles and responsibilities clearly defined and filled with quality personnel, according to Agile / Scrum best practices?

Product Owner

• Single person responsible for maximizing the return on investment (ROI) of the development effort
• Responsible for product vision
• Constantly re-prioritizes the Product Backlog, adjusting any long-term expectations such as release plans
• Final arbiter of requirements questions
• Accepts or rejects each product increment
• Decides whether to ship
• Decides whether to continue development
• Considers stakeholder interests
• May contribute as a team member
• Has a leadership role

Development Team

• Cross-functional (e.g., includes members with testing skills, and often others not traditionally called developers: business analysts, domain experts, etc.)
• Self-organizing / self-managing, without externally assigned roles
• Negotiates commitments with the Product Owner, one Sprint at a time
• Has autonomy regarding how to reach commitments
• Intensely collaborative
• Most successful when located in one team room, particularly for the first few Sprints
• Most successful with long-term, full-time membership
• 7 +/- 2 members

Scrum Master

• Facilitates the Scrum process
• Helps resolve impediments
• Creates an environment conducive to team self-organization
• Captures empirical data to adjust forecasts
• Shields the team from external interference and distractions
• Enforces time boxes
• Keeps Scrum artifacts visible
• Promotes improved engineering practices
• Has no management authority over the team
• Has a leadership role

2.   Planning and Estimation

Are Agile / Scrum best practices used for planning and estimation activities?

User Stories Defined

Are User Stories defined? Are they granular enough? Are the defined according to INVEST principle?

• Independent (to the degree possible), i.e. does not need other user stories to be implemented
• Negotiable
• Valuable, focused on features, not tasks; written in the user’s language
• Estimable
• Small, half a Sprint or less for one Development Team member
• Testable

Estimation Poker or T-Shirt Sizing

Do all team members understand the difference between relative and absolute estimation techniques?

T-Shirt Sizing:

• XS = 1 point
• Small = 2 points
• Medium = 3 points
• Large = 5 points
• XL = 8 points
• EPIC = Product Owner needs to break down

Estimation Poker:

• Based on team votes and consensus
• A relative estimation technique
• Accuracy over precision
• Based on the Fibonacci number sequence (1, 2, 3, 5, 8, 13, 21, 34, 55, …)
• Done by the Development Team members doing the work
• Self corrects for team idiosyncrasies
• Enables more accurate long-term planning

Velocity Tracking

Do you understand your teams’ performance? Is it 36 story points or 56 on average?

How to Calculate:

• Rolling average of sprints
• Max.
• Min.
• Lifetime average

How to Use:

• To determine sprint scope
• To calculate approx. cost of a release
• To track release progress

Remember Best Practices:

• Velocity without a quality metric is not useful
• Velocity of two teams is not comparable
• Velocity changes when the team composition changes
• Velocity increases with teams’ tenure
• Velocity does not equal productivity
• Defects and rejected user stories count against velocity!

3.   Artifacts

Are best practices followed for all relevant Agile / Scrum artifacts?

Product Backlog

• Force-ranked list of desired functionality
• Visible to all stakeholders
• Any stakeholder (including the Development Team) can add items
• Constantly re-prioritized by the Product Owner
• Items at top are more granular than items at bottom
• Maintained during the Backlog Refinement Meeting

Sprint Backlog

• Consists of committed Product Backlog Items negotiated between the team and the Product Owner during the Sprint Planning Meeting
• Scope commitment is fixed during sprint execution
• Initial tasks are identified by the team during Sprint Planning Meeting
• Team will discover additional tasks needed to meet the fixed scope commitment during Sprint execution
• Visible to the team
• Referenced during the Daily Scrum Meeting

Product Increments

• Within the context of a single sprint, an increment is a work product that has been done and is deemed “shippable” or “deployable”
• Developed
• Tested
• Integrated
• Documented
• An increment may, or may not, be released externally depending on the Release Plan

Product Vision

Does a product vision exist? The Product Owner or the Product Management Team identifies the product vision. The product vision is a definition of what your product is, how it will support your company or organizations’ strategy, and who will use the product. On longer projects, the product vision is revisited at least once a year.

Product Roadmap

Does the project / product have a product roadmap? The Product Roadmap is a high-level view of the product requirements, with a loose time frame for when you will develop those requirements. Identifying product requirements and then prioritizing and roughly estimating the effort for those requirements are a large part of creating your product roadmap. On longer projects, the product roadmap is revisited at least twice a year.

Release Plan

Is there a release plan? The Release Plan identifies a high-level timetable for the release of working software. An agile project will have many releases, with the highest-priority features launching first. A typical release includes three-to-five sprints. The release plan is created at the beginning of each release.

4.   Process – Scrum

Scrum is a management framework for incremental product development using one or more cross-functional, self-organizing teams of about seven people each. It provides a structure of roles, meetings, rules, and artifacts. Teams are responsible for creating and adapting their processes within this framework.

Sprint(s)

Does the team follow fixed duration sprints? Scrum uses fixed-length iterations, called Sprints, which are typically two weeks or 30 days long. Scrum teams attempt to build a potentially shippable (properly tested) product increment every iteration.

Sprint Planning

Does the team conduct Sprint Planning Meetings? At the beginning of each Sprint, the Product Owner and team hold a Sprint Planning Meeting to negotiate which Product Backlog Items they will attempt to convert to working product during the Sprint. The Product Owner is responsible for declaring which items are the most important to the business. The team is responsible for selecting the amount of work they feel they can implement without accruing technical debt. The team “pulls” work from the Product Backlog to the Sprint Backlog.

Daily Scrums

Does the team conduct Daily Scrum Meetings? Every day at the same time and place, the Scrum Development Team members spend a total of 15 to 30 minutes reporting to each other. Each team member summarizes what he did the previous day, what he will do today, and what impediments he faces. Standing up at the Daily Scrum will help keep it short. Topics that require additional attention may be discussed by whomever is interested after every team member has reported.

Sprint Review(s)

Does the team conduct Sprints Reviews? After sprint execution, the team holds a Sprint Review Meeting to demonstrate a working product increment to the Product Owner and everyone else who is interested. The meeting should feature a live demonstration, not a report.

After the demonstration, the Product Owner reviews the commitments made at the Sprint Planning Meeting and declares which items he now considers done. For example, a software item that is merely “code complete” is considered not done, because untested software is not shippable.

Incomplete items are returned to the Product Backlog and ranked according to the Product Owners’ revised priorities as candidates for future Sprints.

Sprint Retrospective(s)

Does the team conduct Sprint Retrospectives? Each Sprint ends with a retrospective. At this meeting, the team reflects on its own process. They inspect their behavior and take action to adapt it for future Sprints. Dedicated Scrum Masters will find alternatives to the stale, fearful meetings everyone has come to expect.

An in-depth retrospective requires an environment of psychological safety not found in most organizations. Without safety, the retrospective discussion will either avoid the uncomfortable issues or deteriorate into blaming and hostility.

5.   Process – Engineering Best Practices

Is your project utilizing industry best practice knowledge of what works and what does not work?

Good Design & Good Technical Architecture

Depending on your technology stack, you must pick and choose the appropriate architecture for your application. Component technologies, web services, and commonly used design patterns all provide best practice guidance for your application.

Common Programming Practices

Does your team follow common programming practices such as following a coding standard, peer reviews, code review, implementing unit tests? Do you use source control effectively?

Appropriate Level of Documentation

Are you following the appropriate level of documentation? There are different documentation requirements between a small, internally used, application versus a mission / life critical application made up of millions of lines of code.

Automated Unit Tests

Is your team implemented automated unit tests using a framework like nUnit?

Automated Functional Tests

Has your team automated functional tests? Automated testing tools (such as Selenium) are capable of executing tests, reporting outcomes and comparing results with earlier test runs. Tests carried out with these tools can be run repeatedly, at any time of day. The method or process being used to implement automation is called a test automation framework

Continuous Integration

Does your team utilize continuous integration? A best practice for constructing code includes the daily build and smoke test. Martin Fowler goes further and proposes continuous integration that also integrates the concept of unit tests and self-testing code. Kent Beck supposedly said that “daily builds are for wimps”, meaning only continuous integration would provide the best benefits from a coding perspective.

Ultimately the frequency needs to be decided by the team, but the more frequent, the better. Even though continuous integration and unit tests have gained popularity through XP, you can use these best practices on all types of projects. Standard frameworks to automate builds and testing, such as Jenkins, Hudson, Ant and xUnit, are freely available and new open source tools appear on a regular basis.

Automated Test Status

Is test status automatically generated from all kinds of automated testing (unit, functional) so test status is available and visible to everyone?

Automated Regression Testing

Are automated regression tests executed with every build? Regression testing is one of those often misunderstood terms that everybody throws around, so here is a brief definition:

• Regression testing is any type of software testing that seeks to uncover new defects, or regressions, in existing functionality after changes have been made to a system, such as functional enhancements, patches or configuration changes
• The intent of regression testing is to ensure that a change, such as a software fix, did not introduce new defects. One of the main reasons for regression testing is that it is often extremely difficult for a programmer to figure out how a change in one part of the software will echo in other parts of the software
• Common methods of regression testing include rerunning previously run tests and checking whether program behavior has changed and whether previously fixed faults have re-emerged. Regression testing can be used to test a system efficiently by systematically selecting the appropriate minimum set of tests needed to adequately cover a particular change

Code coverage tools help determine how effective regression tests are. Test automation allows for repeated, cheap, regression test execution cycles.

Without test automation, regression testing quickly leads to tester burnout due to the boring and repetitive nature of rerunning the same tests over and over again.

6.   Delivery Effectiveness

Is your team delivering working software on time according to the Product Owner and end users expectations?

Working Software after each Sprint

After each sprint, is the software “usable”. Can you demo it? Can you interact with it?

Working Software Releases

After a series of sprints, does your team deliver a software release that works?

User Acceptance Testing

If you are delivering to internal customers, do you perform user acceptance testing? If you are delivering to external customers, do you perform external pre-release or Beta testing?

7.   Organizational Support

Does your Agile / Scrum project have organizational support?

Marketing

If delivering to the market, does your Marketing team support your Agile / Scrum project? Are there marketing plans? Advertising? Trade shows? How do you plan to gain market traction?

Sales

If your Agile / Scrum project is supposed to be sold by a professional sales organization, do your sales people know how to sell it? What are the sales targets? If you are selling through software downloads or through app stores (iOS / Android) how are you going to attract “eyeballs”?

IT

Is IT prepared to support your Agile / Scrum project once it is released and goes to market? Are your production servers set up and configure, backups scheduled / verified, in short, is your production system operational?

Stakeholder Involvement

Are your stakeholders aware of your Agile / Scrum project? Did they provide feedback? Are they engaged? Involved stakeholders ensure that you do not run into political headwinds and avoid surprises.

Agile Coach Participation

An Agile Coach can provide valuable insight that help your team be more productive:

• Provide an outside view – coaches bring a new perspective to the organization, team and individuals and they remove intrinsic bias and interpersonal issues
• Identify opportunities for improvement – a good coach provides learning and training for the employees and develops their skills
• Expose challenges – agile coaching creates an environment that allows teams to face the difficulties and work on a proper solution instead of sweeping problem under the table
• Save time and money – coaches bring both the tried and the new practices and processes to a team and organization, reducing the degree of trial and error commonly found in homegrown experimentation

Assessing Multiple Projects

Multiple projects can be assessed in very much the same way. Assessing each individual project and comparing their values shows where one project team is performing better in certain areas than others.

This is important when assessing company-wide capabilities you might want to assess. Or identify key improvement areas across multiple projects.

The following table and radar graph shows a sample comparison of 5 projects:

Mapping Project Improvement Targets

Using the same approach, improvement targets can be identified and visualized, as shown in this example:

Small Assessments versus Enterprise Assessments

Small, single project, assessments can be done fairly quickly and informally. I have assessed projects I was myself part of in a matter of hours. Usually team members know exactly what is going well and what is not going well – and this kind of small assessment can often be tackled as part of the release retrospective. In this case, the assessment is a tool that helps the team assess themselves and potentially measure improvement progress.

Enterprise assessments on the other hand can take months. The largest one I did was for a Fortune 50 manufacturer. The IT project portfolio consisted of more than 400 applications. It took approximately 4 months with a team of 3 assessors, 90+ face to face interviews with various project participants and stakeholders, several workshops and facilitation sessions, two blind surveys, review of hundreds of project artifacts, endless data analysis, and finally a C-suite level presentation of our findings.

The point here is that Agile Assessments can vary widely in scope and purpose.

Some companies simply might want to know how they are doing with their Agile / Scrum transformation as compared to a baseline. Others might try to use an assessment as a tool to identify targets for training budgets. And others might want to identify non-performant projects.

The focus and purpose varies.

Summary

As pointed out earlier, this kind of assessment tool is easily put together. The key to its usefulness is how you conduct the assessment.

When conducting assessments special care has to be taken so project participants do not feel threatened. Especially as a third party coming into a project team or assessing multiple teams, confidentiality is critical so that everybody can be assured that their honest feedback will not be used against them.

When dealing with a large group of project participants and stakeholders across multiple teams, anonymous surveys can be used to validate findings.

Last but not least, transparency is key. Findings should be shared within the team(s) and conclusions and recommendations should be made public. The reason this is critical is so that future assessments will receive the same level of cooperation and openness.