Delivering Quality in Enterprise IT Projects: Four Eyes of Quality Model

(c) Kaali Dass

Introduction

Modern Project Management practice and agile frameworks focus on delivering value faster to customers with expected customer quality. The expected customer quality varies based on industry, organization, stakeholders, and project types. Agile is the new trend in the industry and takes momentum in every business and functional units of an organization and makes inroads to multiple industries. It is changing the way IT projects are managed and delivered. Many professional organizations endorsed the agile Manifesto declared in the year 2001. The agile framework was the best alternative to the structured and sequential predictive method to deliver IT projects. Since then, multiple competing frameworks have emerged to deliver products and services using agile methodologies. In a nutshell, agile helps organizations in three ways:

1)     Continuous Delivery: Instead of waiting for the final product in perfect condition, customers get incremental product features based on their priority, urgency, and need.

2)     Customer Focus: The product owner is part of the project team, collaborate to prioritize the customer needs, and deliver desired outcomes. 

3)     Continuous Improvement: Since the outcomes are delivered incrementally at a regular cadence, any failure in incremental features is not a final product's total failure. It is reversible with less rework, recoverable with minimal effort, and provides multiple opportunities to improve the delivery process and outcomes.

One of the significant disruptions in the agile transformation was in the Quality Assurance (QA) function of the organization. Agile has changed the quality culture in organizations and shifted the dedicated QA team's ownership and responsibility to every member of the agile delivery team. Many IT organizations dissolved the large and dedicated QA teams and embedded the testers within the agile teams. Project Management emphasizes clear roles and responsibility in efficiently performing a specified task and ownership and accountability to complete tasks effectively. The term "Everyone is responsible for Quality" promotes the quality culture in agile organizations. However, "Who from everyone in the team is responsible for quality" and "What are the boundaries of quality," and "How an agile team member can contribute effectively based on the knowledge and skills" is not prescribed in agile frameworks. Many variations and models are being used to implement quality in agile delivery.

Quality starts from the requirement, and everyone performs part of the quality functions to ensure the incremental and final product meets customer needs. Product success depends on meeting customer needs, customer adoption, and customer success. Project success depends on meeting the project's intended value and benefits, and quality is an integral part of product and project success. Project and product success helps organizations to compete in the marketplace, delight customers, and gain and sustain competitive advantage.

This paper analyzes challenges in implementing quality in delivering IT projects in an agile environment and suggests a model to divide overall quality activities among the team based on their skills, knowledge, and understanding of the product quality. This model references large and global enterprise structures and environments and focuses on Enterprise IT projects dealing with cross-functional and distributed teams.

Software Quality

American Society for Quality (ASQ) and the International Organization for Standardization (ISO/IEC 25010:2011) recommend defect management and quality attributes to achieve desired software quality. Many organizations adopted defect management as one of the quality management tools to record and monitor software quality. ISO defines Functional suitability, Reliability, Operability, Performance Efficiency, Security, Compatibility, Maintainability, and Transferability as eight quality attributes. From the project perspective, achieving all quality attributes recommended by ISO/IEC 25010:2011 will dramatically increase the total cost, effort, and time to deliver projects. The project team needs to work with product owners to plan the quality needs and analyze the cost, value, and trade-offs in delivering customer expected product quality to achieve the right quality level. The ultimate project goal is to achieve the customer's implicit and explicit quality needs with a minimal cost.

IT Project Environment and Challenges

Unlike standalone technology products, Information Technology (IT) projects have complex upstream and downstream system and process dependencies. Further, each system and process components have its subsystems and processes aligned with different business groups. Project teams need to engage with multiple stakeholders, business owners, a diverse customer base, competing project delivery goals, objectives, and priorities. Besides, the dependent systems may align with different strategic objectives of the business units within the organization. The dependency factors lead to complexities, challenges in adapting the standard and simplified agile framework. Organizations need to customize the agile frameworks to align with the organization culture, organization structure, and project environment. Even within the organization, where there are multiple business units, it is hard to implement a "one size fits all" approach to deliver all IT projects in a single way.

Large enterprises need to interact with interconnected business units and functional units to deliver products or services. Depending upon the type of product or service, the interaction with operational and business units may vary. The process flow associated with product or service sales starts with Market research, Demand forecasting, Opportunities, Pre-sales, Quotes, Ordering, Fulfillment, Payment, Warranty, and Support. The process flow interacts with functional units like Marketing, Sales, Manufacturing, Vendor Management, Finance, Accounting, Product Support, and Human Resource Management. IT project delivery differs from standalone technology product delivery due to complex interactions with these organizations' processes and functional units. The following diagram (Figure 1.) depicts a typical IT project delivery flow and high-level dependencies in a large global organization.

Figure 1. Enterprise IT Projects

It starts with the Customers / End-Users, Product Owners, Dependent projects, Processes, and Systems, Enterprise Release Management team, and finally, agile teams who are responsible for product delivery. The complexity may increase depending upon involvement from various business units, multiple regions, regulatory, compliance, and project dependency to external vendor products or services.

Agile project management methods serve customers better by dividing big bang delivery into small incremental releases based on customer priority, market demand, and business needs. Centralized decision making is shifted to self-managing teams and empowers them to plan, estimate, and deliver efficiently at a regular cadence. The overall product scope is divided into Epics, Features, and User stories and groomed periodically to add clarity to the requirement and refined to translate into real customer needs. This helps the Product owners decide the best possible outcome and meet the customers' immediate needs.

The incremental delivery approach reduces the risk of complete product failures, and issues can be fixed or reworked in the next sprint cycle. Closer customer collaboration reduces the translation gaps from the requirement to release. The incremental delivery model helps the customers to learn, improve, and add newer product features and capabilities without waiting for months or years. Though agile helps to deliver capabilities faster, the incremental delivery model comes with inherent challenges in providing customer expected quality in IT projects.

IT Quality Assurance

IT projects with multiple integrated systems and process dependencies need dedicated hardening sprints to conduct regression and end-to-end testing to ensure that the incremental product delivery integrates well with dependent business processes, systems, and the environment. In addition to the end-to-end testing, the project team may choose to do specialized testing to comply with regulatory, security and meet additional customer and business service level agreements (SLA).

The following are additional testing conducted to meet software quality and compliance:

1)     Compliance Testing: Primarily used to find any deviations from the regulatory, internal, and external policies and standards. This testing involves complying with the policies, standards, documenting test scenarios, test results, and generating standard documents to comply with regulatory standards. More than a dozen regulatory standards are in place, and standards vary based on the type of product, industry, and end-users. In addition to international and federal standards, local state laws and regulatory measures need to be in the project scope. US Food and Drug Administration (FDA) under CFR Title 21, Payment Card Industry, and General Data Protection Regulation (GDPR), Sarbanes-Oxley (SOX) Act, and California Consumer Privacy Act are some of the regulatory testing standards used in the United States.

2)     Performance Testing: This testing practice is performed to determine the system response and stability to meet the organic growth over a period and seasonal peak loads. The testing team needs to plan for the maximum number of users, concurrent users, peak time, data volume, and test the system for different loads using specialized tools and load testing environments

3)     Availability and Business Resiliency testing: This testing ensures that the applications meet the availability, Service Level Agreement, and compliance needs of the customers. It needs expensive infrastructure and resources to document and test active-active, active-passive scenarios, and test the availability in the on-premise or distributed cloud environments.

4)     Penetration Testing: This testing is performed to evaluate the system security, comply with various organization policies, industry standards security, and protect confidential and sensitive organization data against hacking and exposure to other vulnerabilities.

The product owners should define the product quality goals and objectives and prioritize the testing to plan resources and costs for successful product delivery.

Product Quality: Waterfall

Waterfall development inhibits organization agility and speed. However, extensive planning helps the project team finalize the Scope, Cost, Resources, and Time to complete the project. At the end of the planning process, the organization leadership and stakeholders know the outcome, timeline, cost, and resources to complete the project. Once the planning is complete, the project team spends time to create a Business Requirement Document detailing all the requirements and review with the stakeholders for approval. The third step of the process is to create a functional design document, translate the Business requirement document into Functional requirements by detailing links to processes and systems. The fourth step of the process is to create a Technical Design document to translate functional into technical design. Quality is captured at every step of the process, and the team understands the quality efforts in the final product delivery. This brings the leadership, sponsors, stakeholders, and the project team on the same page and everyone knows their quality responsibilities in making the project successful.

Product Quality: Agile

In an agile delivery process, testers are embedded, and the software delivery process is integrated and included in a sprint cycle. Within a sprint cycle, developers and testers work together and perform collaborative testing and move to the next higher environment to perform additional testing. Parkinson's Law states that "work expands to fill the time available for its completion." This Law is valid for developers working in a short sprint cycle. The developers take the entire time to complete the development and testing, limiting less or no formal QA testing time. Test-Driven Development, Model-Based Testing, and other automated testing DevOps tools and testing practices are recommended in agile development. However, adopting the new way of testing and automation is not widely used due to the organization's culture, leadership commitment, cost, skilled resources, and, most importantly, the team's quality mindset. Agile teams can easily steer away from the initially intended path to deliver the customer expected product quality in the absence of a centralized architecture blueprint, enterprise-level oversight, and governance.

Quality is incrementally built based on how well the customer explains their problems, how well the product owner understand the customer needs, how well the analysts translate customer problems and needs into the product quality. The final product quality rests with the technical team (architects, technical leads, developers, and testers) responsible for building, testing, and releasing the product.

Unlike the waterfall method, agile development scope is in the backlog, and requirements for future sprints are analyzed and groomed, while current sprints are in progress for the next Minimum Viable Product (MVP) delivery. The development team based on their current understanding of the quality and has a narrow view of the feature or user story, focusing only on short-term deliverables to optimize time and efficiency. It is hard to achieve the final product quality unless the quality goals are clearly defined, tracked continuously, and final quality expectations are included during incremental product releases.

Enterprise IT Quality Challenges

Business-critical applications need to follow enterprise release schedules to avoid monthly and quarterly downtimes and business disruptions. Modern organizations eliminated manual and process-intensive Enterprise Release Management Office (ERMO) with an automated process to ensure all production deployments are governed and controlled to minimize business impacts. Business-critical applications follow the automated governance and oversight process defined by the ERMO. This ensures that the changes made to applications comply with regulatory, Security, and quality needs before the final product release. The typical ERMO release environment has Sandbox for Proof of Concepts, Development, Stage, Pre-Production, and Production environments.

The development environment is the primary environment where multiple agile teams develop and test the incremental and final product. It comes with the inherent disadvantage of reproducing the production environment to test all quality needs. It also lacks integration with all the dependent systems and with the latest product versions. The developer can unit test the product for the incremental scope and may have limited capabilities to test the product end to end. Developers may have to API facades, generate test data to validate system dependencies, and use test data to try all the software delivery components. Once the developer is satisfied with the unit and integration testing, the code is migrated to the stage environment. The stage is the next best environment to test the product; however, this may be a few releases behind the production environment. QA starts the formal testing for the release and follows the defect management process to create and categorize the defects based on the severity and impact. Proper defect management starts in stage, and it continues in the next environment, which closer to production. Many large organizations have a Pre-production environment with real data embedded with security profiles to ensure that the release-ready product works fine before the final release. The following Figure 2 shows the Enterprise Release Environment and the Cost of Quality.

Figure 2. Cost of Quality

Quality starts at the requirement level and improve at every stage of the development. In agile development, the Cost of Quality and rework will be less compared to a waterfall project. Still, it will take a considerable amount of time and effort to refactor the code, fix the code, retest, and migrate to multiple environments before released to production. The time and effort spent on rework could be effectively used to deliver additional capabilities for future releases. The next section describes the Four Eyes of Quality Model, implementation details, and benefits.

Four Eyes of Quality Model: Defined

This Four Eyes of the Quality model is designed to achieve desired software quality in a complex IT project environment dealing with multiple dependencies, multiple scrum teams, and multiple business stakeholders. In addition, complex business-critical applications follow enterprise release management schedules to reduce risks and minimal business impact. This model leverages ISO/IEC 25010:2011 defect management and attributes based approach and looks at software quality from multiple perspectives.

This model recommends defining quality goals and objectives during the project initiation phase and achieving the desired quality by leveraging diverse team's skills and strengths. Quality requirements are determined at the product vision, captured at the requirement level, monitored, and tracked during the execution phase, spanning across multiple sprints.

Define Quality Goals and Objectives: Product vision has the needs, capabilities, features, outcome, and value. Besides, the vision should exclusively include product quality goals and objectives. The quality objectives should explicitly call out meeting the Functional suitability, Performance efficiency, Compatibility, Usability, Reliability, Security, Maintainability, and Portability.

Achieve Consistent Product Quality: Product Owners should create product level epics and features, including quality goals and objectives. This should be translated into sprint level user stores for tracking MVP delivery. Aligning features and user stories with the stated product goals and objectives ensure that the user stories are prioritized and committed at a sprint level to maintain consistent incremental quality and meet the final product quality.

Leverage Diverse Team Strength: Quality Assurance efforts should be distributed and diversified across teams to look at the incremental product from their understanding, usage, and different quality eyes.

Figure 3 depicts the Four Eyes of Quality Model implementable in an agile environment. This model embeds quality in the early stages of product development, fosters quality culture at the people level, and integrates with the agile delivery process. The below diagram illustrates how to include additional quality characteristics in agile delivery.

Figure 3: Four Eyes of Quality Model

For example, the product described in Figure 3 focuses high on Functional suitability, Usability, and Security. Everyone is responsible for quality; however, specific team members should exclusively focus on certain aspects of the quality's explicit and implicit needs based on the skills, exposure, and knowledge. QA will be embedded across all the team members and work as one team to deliver customer expected quality.

Product owners can define quality goals and objectives and prioritize them to balance the cost of vs. benefit of achieving higher quality. For example, a SaaS application specialized in delivering certain business services comes with availability, performance, and Security in the product SLA. The project team needs to focus more on business needs and functionality. Conversely, an in-house application dealing with Personally Identifiable Information (PII) needs to emphasize high on product security, regulatory compliance to meet privacy laws,  and adhering to the organization's security policies and standards.

This model helps distribute the quality responsibility and accountability between the team members and customer focus groups, and everyone contributes their share to improve quality incrementally. The next sections describe the team roles, process, and implementation details of this model.

Four Eyes of Quality Model: Agile Team Roles

The Product Owner is the first contact point of the customer, and his or her leadership is a key factor contributing to product success. The Product Owner provides the product vision, adds new capabilities, and changes to existing capabilities to meet the customer's needs and the dynamic demands of the marketplace. The product owner has a short-term and long-term release view. The product roadmap helps the team share the product roadmap, clarify the customer requirements, and explain the incremental value and benefits. The product owner who participates in all the agile ceremonies and contributes to the team needs will immensely benefit the project's success. Finally, the product owner works with the focus groups to get early feedback before releasing and testing the final product to improve end-to-end product quality and achieve project goals and objectives.

Large enterprises have Solution Architects and Business Analysts' roles in translating the business or customer requirements into functional and technical requirements. They understand system dependencies, data sources, integrations to internal and external systems, and focus on leveraging and reusing existing business and technical services. They also do upfront analysis, work with data stewards, dependent stakeholders, legal and compliance teams to ensure that the customer and product needs are flushed out to a granular level in the form of features and user stories. The high-level acceptance criteria mentioned in the epics and features need to be detailed in user stories and coarse enough to be delivered within a sprint. It is specific, measurable, realistic, and testable by everyone in the team. Product Owners and Business Analysts help have a combined strength of clear understanding of the product capabilities, quality needs, functional dependencies, business benefits, impacts, and risks associated with the product release and incremental delivery.

The technical leads, developers, and testers have the technical and system view of the product. They also add additional technical requirements to deliver customer needs and operational product requirements during the grooming process. The operations capabilities include Supportability, Maintainability, Portability, Security, and Reusability. Also, the final product should adhere to the organization's policies, standards, and best practices.

The agile team has varying breadth and depth of product knowledge, business domain experience, organization's capabilities, market needs, infrastructure, system, and technical knowledge and skills. This model leverages individual team members' skills and strengths to build incremental quality to ensure final product quality.

Four Eyes of Quality Model: Implementation

First Eye - The Developer Eye View: The developer view is narrow and focuses on the current sprint scope. The developer's goal is to build and test the code to meet the user story's acceptance criteria. This testing scope involves the current sprint delivery and its dependencies with other scrums deliver items associated with the product release. Test-Driven Development is one of the recommended approaches to complete the first step of the process. During this testing, the developer also writes and automates unit test cases to ensure all technical and functional aspects of the code work independently and integrates with the final product and dependent components. Unit Testing, Integration Testing, and Functionality testing are three types of testing performed by the developer during this testing phase. Developer and QA work together to achieve the first level of foundational quality and ensure the product works to meet the current sprint requirements.

Second Eye - Operations Eye View: Complex IT product releases with multiple scrum teams need additional hardening sprints focusing on real-time integration, regression, and end-to-end testing to test the product in a near-production environment. This ensures all the software releases are up-to-date, and incremental codes are deployed and tested before go-live. The hardening sprints help identify and automate end to end regression and progression test cases for current and future releases.

The testers involved in this testing process need to understand the full product scope, limitations, dependencies, and knowledge of business domains and processes. This is specialized testing of the final product before the release. During this phase of the testing, developers work with specialized testing teams to ensure the product meets non-functional requirements and meet production-ready quality requirements. A high level of collaboration between scrum teams, external teams, and dependent system owners participate in this activity to ensure that the product work as expected. 

Third Eye - Product Owner Eye View: The product owner is close to the customer, and he or she gets a chance to view the product demo and test the incremental product from the customer's perspective. One of the best ways to test the product is to do Scenario-Based Testing. Create different scenarios in which customers can perform in the field and test the features from a delivery perspective and product usage perspective. Further, this can be augmented to perform Persona Based Testing. Modern applications have the flexibility to customize content delivery and access using specific personas and secure access using role bases access control. Performing tests based on each persona ensures that the product changes are done in one persona did not impact other personas, and user experience is optimized for personalized delivery.

The Third Eye testing involves a high collaboration between the Product Owner, Business Analyst, and QA team.

Forth Eye - Customer Eye View: This is where the rubber meets the road. Product or project success depends on how well the customers are adopting new product features. When the customer sees the value, ease of use and benefit of using the product, the product usage, and adaption increases rapidly. This reduces the time and cost of product adoption, reduces change management, and also accelerates time to realize the project's investment value.

Product owners choose a subset of end-users to test the final product. They get an opportunity to see the beta version product and provide early feedback on newly built features before the product release.  Focus groups should represent each segment of a customer base to ensure that the product features are exposed to all types of customers. For example, for a global product launch, the focus groups are selected by choosing one or two end-users representing each global region like Asia Pac, Europe, North America, and South America. Some of the focus groups are selected to represent different age groups, and some are chosen to represent different languages and ethnicity. In this part of the testing, product owners work together with focus groups to test the product from a real customer perspective, gather feedback, and prioritize future releases.

Quality Best Practices

Implementing the new quality model will be successful only if the team continually learn and improve. The following best practices will help to reduce defects and enhance the quality of the software.

1)     Shift-Left Quality: Many agile project management tools provide a template to capture requirements. User stories are written with minimal information may lead to a lot of unambiguity and assumptions. Provide more information about the requirements, additional details, flow-diagrams, wireframes, architecture, and examples to support the needs. This will help product owners and business analysts understand the customer's requirements and avoid translation gaps from release requirements.

2)     Acceptance Criteria: Acceptance criteria provides validation scenarios and acceptance on the user stories. It helps the developer to build and test, and the product owner to accept the product. Detailing acceptance criteria with all functional and non-functional acceptance criteria allows the entire team to validate various aspects of the feature or user stories based on their skills and access.

3)     Emphasize specific quality needs: Grooming calls help the development team understand the requirement and ask clarification questions. In addition to focusing on functional requirements, the grooming calls are an excellent opportunity to highlight quality requirements. This process will reiterate the expected quality for every incremental delivery.

4)     Data-Driven Quality: Leveraging data-driven quality empowers the team to make quick decisions based on real-time quality metrics and associated actions. The metrics also help identify deficiencies in the process, systems, or people and provide an opportunity to improve quality in the future release cycles. Establishing automated Continuous Integration and Continuous Delivery systems and automatic quality scans will show real-time metrics and quality dashboards to identify code issues in the early development phase.

Summary

This article discussed the agile project environment, quality challenges in delivering IT projects. The author presented a Four Eyes of Quality Model, implementation approach, and best practices to improve product quality and IT project success.

About the Author

Kaali Dass is a Senior Project Consultant and has over 25 years of information technology experience spanning multiple industries. He has extensive experience in managing domestic and international projects, including Cisco Systems and LinkedIn. He is also an Adjunct Professor at USC Bovard College and serves as Director of IT – Digital Transformation and IT Security at NC PMI.

Kaali has a Ph.D. in IT Management, an MBA in Finance and holds professional certifications including PMP, CISM, CSQE, ITIL, and CSM.

His focus areas are Portfolio management, Agile project management, IT strategy and planning, Enterprise architecture, Data architecture, IT Security, and Analytics.

For any questions about this article, he can be reached at [email protected] or https://www.dhirubhai.net/in/kaalidass