IT Project Portfolio Management

A Program where I worked introduced the Balanced Scorecard to the Information Communications, and Technology (ITC) department of a nuclear weapons clean-up site, Rocky Flats Environmental Technology Site. This effort consisted of several hundred development and deployment projects, ranging from database applications, web applications, network and desktop deployment, Voice over IP, and Cell Site installations, to general IT infrastructure operations.

The management of this diverse set of projects was directed through the Program Management Office. Project managers and their staff provided support, guidance, and project coordination to functional managers and their staff. Project planning, execution, control, and reporting were performed through a collection portfolio management, Balanced Scorecard, Earned Value Analysis, and project selection and measurement processes. This effort focused on the management of software development projects and the issues associated with making “level of effort” development tasks visible to the management team.

Legacy Project Management Environment

Prior to the assumption of the management of the IT department from the previous contractor, there were a variety of forces driving project management activities, shown below, which were the motivation for the introduction of Project Portfolio Management (PPM) and its supporting processes.

These forces created a “gap” between the planning of IT projects and the execution of these project plans. The planning of a collection of projects, although not straightforward, is only half of the solution. Execution of these plans requires a mechanism to define priorities, compare resource requirements across the collection of projects, and understand the consequences of any resource or priority decisions for the collection of projects, not just individual projects. In the absence of this process, the most vocal user usually got the highest priority, at least until a crisis was encountered and the importance of a specific user was altered in light of the current priorities and resource availability.

Stephan Covey calls this “Living in Quadrant I,” where everything is urgent and important ?. This is the quadrant of crises, pressing problems, deadline-driven projects, meetings, and very few preparations. To gain control of a portfolio of projects, the project manager had to “Live in Quadrant II.” This is the quadrant of preparation, prevention, values and priority clarification, planning, and group empowerment. To live in Quadrant II we introduced four systems described in the next paragraphs.

These systems are necessary, but they alone are not sufficient to gain control of the IT project portfolio. A vision of a working IT department is needed. One in which new projects can be absorbed with ease. One in which work is planned on a daily, weekly, monthly, and quarterly basis, with the concurrence of the stakeholders and the suppliers of services and products. One in which a reliable projection of the labor and material costs is provided by the system with the support of the staff.

Components of the Project Lifecycle

The view of the management of portfolios of projects can start from many vantage points. One is to view the process from a project “life cycle” inside a portfolio of projects:

1. Project selection and initiation.
2. Planning and analysis.
3. Implementation.
4. Operation and maintenance.
5. Closeout and removal.

Introduction of PPM in the IT Services Domain

Successful management of a portfolio of IT projects assumes that the best candidates for success can somehow be selected in a methodological manner.

A portfolio management approach was introduced to address the gaps shown in Table 1. This approach resulted in behaviors moving away from Quadrant I’s external drivers and moving toward Quadrant II’s processes. This control-based approach consisted of four (4) components:

1. A Balanced Scorecard strategy to define priorities and establish a connection between every project and a specific strategy and objective.
2. A public registry of projects, resources, and deliverables housed in a Microsoft Project Server.
3. An Earned Value performance reporting and measurement processes to make visible performance metrics for cost and schedule.
4. Processes to select, classify, measure, and guide the implementation of collections of information technology projects.

Balanced Scorecard

Balanced Scorecard (BSC) is a strategy-forming process in which four perspectives of the desired outcome are created; Financial, Stakeholder, Internal Business Processes, and Learnings and Growth. A “strategy map” is constructed which states specific objectives for the deliverables for each perspective. Although BSC may appear to be an esoteric management tool, it is a powerful starting point for the management of portfolios.

BSC provides the metrics to manage projects, trace their specific objectives, and collect projects into initiatives to be included in a portfolio. The question asked by the portfolio manager is “how does this project support a specific strategy?” If there is no clear answer, then either that strategy is not clearly defined or the specific project does not support a strategic goal.

In some organizations, there is a distinction between “public” projects and “private” projects. At the Prime Contractor, we had only public projects. If a project cannot be aligned with a BSC objective, it cannot continue to exist. Clarifying, documenting, communicating, and making all the participants accountable for this goal is a critical success factor.

The Balanced Scorecard consists of four “perspectives:”

• Business results – what are the definable objectives of our efforts? How will we know when we are successful?
• Customer expectations – what would the customer say when asking for our services?
• Internal processes – what work processes must be in place to deliver value to the customer?
• Strategic enablers – what behaviors, attributes, facilities, and resources must be in place before any value can be delivered?

Here's the Balanced Scorecard for managing the portfolio of projects needed to ICT to successfully support the close of the site.

One aspect of a Balanced Scorecard, found difficult to accept at times, is that every project in the portfolio must support an objective in the strategy map. If a project did not support an objective, then the question “why are we doing this,” must be answered. If there was not a strategy–based answer to that question, then the project was dropped.

The organization needed the political will to treat project management in this manner. When this is done, then all work performed by the department will be in support of an identifiable strategic objective.

Centralized Project Repository

The assembly of all the projects in one place was a critical success factor for portfolio management. By “all” it is meant ALL projects, no matter how small or obscure. We installed Microsoft Project Server to house the portfolio of projects. Although it may not be the ideal choice, it provided a central repository for projects and their tasks, resources, and cost information. It also provides Earned Value Analysis of projects once they are underway. We captured labor hours through the Project Server as well as a Defense Contract Audit Agency audited Time and Labor reporting system. These hours along with non–labor costs are applied to the project baseline to compute the cost and schedule variances and project performance indices.

The effort necessary to capture all activities into projects should not be underestimated. Many times political and human roadblocks appeared that said, “… but what you’re trying to measure is not a project.” The solution was to consider all activities to be a “project,” even the level of effort tasks. In this way, all work done by the department was captured in the Project Server. For the level of effort tasks, a project was defined in which those tasks resided on a line, with resources assigned. No matter what the activity it can be found in the project server, with the associated costs and durations.

Earned Value Analysis

Earned Value Analysis can be used to measure and communicate the physical progress of a project based on “work complete,” the effort used to complete the work, and the costs incurred to complete the work. Earned Value Analysis can also be used to evaluate and control project risk by measuring project progress in monetary terms for the actual value delivered to the customers.

Earned Value is a project management technique that provides “leading” performance indicators to allow project managers to identify and control project problems before they become insurmountable. Earned Value provides a significant improvement over the traditional project measures which compare planned expenditures and actual costs, which is equivalent to “driving in the rear view mirror.” Earned Value adds a third measure – the actual work accomplished as a result of the expenditure. Measuring the actual work accomplished provides greater insight into potential project risks. It also provides more accurate estimates for the completion schedule and cost estimates. With these “leading” indicators, project managers can proactively manage their project in ways not found using only cost and schedule measures. There are many references for Earned Value metrics.

Figure 2 describes a “simple” view of these metrics and their relationships.

Using the information captured in the Project Server, cost and schedule performance indices are produced for review by the Program Management Office and senior IT management.

The key to deploying Earned Value in a software development portfolio management system is to define a method of measuring “value” that is consistent with the software development process.

The typical methods of measured value are based either on a binary event or some subjective assessment of the progress that has been made during the reporting period. Both of these approaches fail the integrity test for software project management. This test asks the question – “how do we know that the software will behave as specified?” If it does behave as specified, then the development phase is complete. If not, then rework is needed. In the typical Earned Value Management System the budgets for the task are used to accrue the value rather than the expected business value associated with the task’s completion. [3]

One approach to measuring value is to employ “Technical Performance Measurement.” This approach is used on many engineering and development projects in defense systems. [7,?10] Technical Performance Measurement is the plan for expected technical achievement. The actual progress of the project is compared using periodic measurements or tests. The difference between the planned progress and the actual progress represents a technical variance.

Another approach to measuring software development progress is the use of testable requirements. [13, 14] A requirement can be used to define the condition or capability for a software system to meet its objectives. The IEEE Standard Glossary of Software Engineering Terminology includes definitions of six different types of requirements — functional, design, implementation, interface, performance, and physical requirements.

A testable requirement can be decomposed to a collection of precise, unambiguous, and indivisible set of low–level requirements. These criteria are only met if it is possible to write a test case that would validate whether the requirement has or has not been implemented correctly.?This is the source of the term “testable requirement.”?

Testable requirements provide the following benefits for an EVMS based development method. Testable Requirements:

• Form an overarching technical performance measure for identifying progress to plan.
• Support the goals of a Performance Based Contract
• Are measurable from the software conception phase through system acceptance
• Are “success oriented.” Progress is based on the number of tests passed for both the software and systems level
• Integrate schedule and technical cost objectives in a single performance-based metric.

Applying this testable requirements approach to the portfolio of software projects provides a uniform and consistent means of measuring progress to plan.

The Software Management “Level of Effort” Problem

The traditional method of comparing actual spending to planned spending is inadequate for establishing, assessing, monitoring, and predicting the future performance of a software project. The financial accounting (the normal manner of tracking progress in a software development project) approach of comparing “budget” versus “costs” fails to consider the technical achievements – the “physical progress” – that is accrued for the software development tasks. In a normal software development project, the passage of time is assumed to be equivalent to progress toward the completion of the task. Software projects all too often have no tangible deliverables, which are physically visible. The common result is that large costs are incurred with little useful product being delivered until the end.

The first step in identifying the “value” of software deliverables is to define a set of “verifiable” outcomes that represent this value. These outcomes have two attributes critical to the deployment of earned value:

• They are defined in self–contained units of work.
• Each unit of work is testable to assure the requirements can be verified.

The Earned Value Management principles define the methods to:

• Plan all work scope for the program to completion.
• Break down the work scope into finite pieces that can be assigned to responsible persons or organizations for control of technical, schedule, and cost objectives.
• Integrate program work scope, schedule, and cost objectives into a performance measurement baseline plan against which accomplishments may be measured. Control changes to the baseline plan.
• Use actual costs incurred and recorded in accomplishing the work performed.
• Objectively assess accomplishments at the work performance level.
• Analyze significant variances from the plan, forecast impacts, and prepare an estimate at completion based on performance to date and work to be performed.
• Use EVMS information in the company’s management processes.

Project Selection Process

Deciding which projects will receive funding, which ones have the highest payback, and which ones to focus on is an ill-structured problem. Providing a mechanism for addressing these issues requires three activities:

• Representing the problem by uncovering information and supporting justifications for the project.
• Stating the solution to the problem by gathering options and selecting among them.
• Supporting the justification through measures and tests.

Solutions to ill-structured problems have sociological as well as technical dimensions. Since such problems have multiple answers gaining agreement over these answers is a major effort. In order to succeed here, three items support the project selection are:

• The definitions of the strategies that identify the value of the proposed projects.
• The evaluation principles.
• The process by which projects are evaluated for their contribution to the strategic goals.

There are many selection and evaluation algorithms, some simple some complex. We have chosen two simple approaches:

• Make a list – construct a prioritized list of the projects in order of importance. This seems “too simple” to actually work. If the priorities on the list are the implementation priorities of the objectives in the strategy map, then it is simple.
• Paired Comparison Analysis – for projects whose priorities are difficult to sort out or ones that have other conflicts that a simple linear list cannot deal with, Paired Comparison Analysis is a useful tool. It is a list-making algorithm, but one to sort out the priorities by asking “between these two projects, which one must come first?” Paired Comparison Analysis helps you to work out the importance of a number of options relative to each other. It is particularly useful when you do not have objective data to base this on.

Critical Success Factors for Portfolio Management

For the project portfolio management process to be successful several critical success factors must be in place.

• Process – define a clear and concise process by which projects are categorized, evaluated, and selected.
• Strategy – every project must have a “strategic” purpose and support an objective in the balanced scorecard. If not it should be dropped
• Relationships – collections of projects are difficult to define in the absence of a higher–level mission. The Balanced Scorecard provides the means of organizing projects into initiatives. These initiatives can then be directly traced to Strategy Map objectives.
• Decision framework – deciding which projects are to be executed first can be answered by asking which projects need to be executed to fulfill the strategic objectives in the proper sequence.
• Decision framing – clearly define the problem to be solved, the decision criteria, the realistic tradeoffs, and options.

References

[1]?Ben–Menachem, Mordechai and Roy Gelbard, “Integrated IT Management Toolkit,” Communications of the ACM, April 2002, 45(4), pp. 96–102.

[2] Berkman, Eric, “How to use balanced scorecard,” CIO, May 15, 2002.

[3] Boehm, Barry and Kevin Sullivan, “Software Economics: a Roadmap,” Proceedings on the Future of Software Engineering, Limerick Ireland, 2000.

[4] Clemen, Robert T., Making Hard Decisions: An Introduction to Decision Analysis, 2nd Edition, Duxbury Press, 1995.

[5] Cobbold, I. M. and G. J. G. Lawrie, “The Development of Balanced Scorecard as a Strategic Management Tool,” 2GC Active Management, Maidenhead, England.

[6] Constantine, Larry, “Work Organization: Paradigm for Project Management and Organization,” Communications of the ACM, October 1993, 36(10), pp. 35–43.

[7] Ferraro, Mike, “Technical Performance Measurement – A Program Manager’s Barometer, Program Manager, November/December 2000, pp, 14–20.

[8] Fleming, Quentin and Joel Koppelman, Earned Value Project Management, PMI, 2002.

[9] Mollaghasemi, Mansooreh and Julia Pet–Edwards, Making Multiple–Objective Decisions, IEEE Computer Society Technical Briefing, IEEE Computer Society, 1997.

[10] Pisano, N. D. Commander, USN, “Technical Performance Measurement, Earned Value, and Risk Management: An Integrated Diagnostic Tool for Program Management.”

[11] Scott, Judy E. and Iris Vessey, “Managing Risks In Enterprise Systems Implementations,” Communications of the ACM, April 2002, 45(4), pp. 74–81

[12] Summer, Mary, “Critical Success Factors in Enterprise Wide Information Management Projects,” SIGCPR, 1999.

[13] Wilson, P. B., “Testable Requirements – An Alternate Sizing Measure,” The Journal of the Quality Assurance Institute (October 1995):1

[14] Wilson, P. B., “Sizing Software with Testable Requirements,” Systems Development Management, 34–10–04, Auerbach Publications, 2000.

? The 7 Habits of Highly Effective People, Stephan Covey, Simon & Schuster, 1990