How can we manage complex designs? A systems engineering perspective.

“How can we design improvement in large systems without understanding the whole system, and if the answer is that we cannot, how is it possible to understand the whole system?” - C. West Churchman

How can we manage the early stages of a complex system design when the requirements are unknown, the design is evolving, and our expertise is limited? Can we just test and iterate toward a safe, effective and compliant design solution? What do we need to build the “right” system? In the following, I am trying to address these questions. Mainly, sharing some of my experience with systems engineering as a management framework for development of complex systems. Here is a list of my topics. You may also jump into the "lessons learned" section to get a gist of what I am discussing here.

1. Where does the ambiguity in complex design come from?
2. What are the adverse outcomes of the ambiguity?
3. How can we navigate the ambiguity?
4. What are the building blocks of a systems engineering process?
5. What are the competing characteristics of a systems engineering process?
6. How can we implement a systems engineering process?
7. How do we measure the performance of a systems engineering process?
8. What are the ownership and execution challenges?

---------------------------------------------------------------------------------------------------------------

1. Where does the ambiguity in complex designs come from?

Maybe we are building a robotic surgery system, a self-driving car, a constellation of satellites, an underwater drone, and maybe a flying carpet (?). These are all complex systems with many requirements and intricate relationships (see my previous article). In early design stages of such complex systems, there is always a risk of stretching thin and losing focus. From a requirements perspective, this is because of an ambiguity arising from:

• Unclear understanding of “what” needs to be built, i.e. not knowing the requirements
• Fuzzy idea about “how” it can be built, i.e. not knowing how to implement the requirements

These two sources of ambiguity are in fact intertwined. It is worthwhile noting that, there are other things that can add to this ambiguity, such as: not having a clear understanding of why the system is needed or who we are building it for. In the beginning, your customer development team plays a key role in answering these questions.

2. What are the adverse outcomes of the ambiguity?

If we are building a complex HW/SW system, the ambiguity in early stages of design can easily lead into a “black box” process with adverse outcomes. Here are four of them:

• Erosion of or drift from the architectural design decisions
• Proliferation of design errors
• Expensive design changes
• An end-product which does not meet safety, efficacy and performance requirements

Think “Therac-25”, for example, a software-controlled radiation therapy machine which overdosed, killed or severely injured several patients in the late 1980s. You might say, we keep testing and iterating toward an effective solution.

But is testing enough?

Feedback is constrained. These complex systems are often times safety critical and cannot be easily tested before meeting some essential risk-related and performance requirements. Plus, we do not necessarily have access to a large base of users. All these challenges aside, bear in mind that testing, specifically for software, never proves the absence of errors in design. In fact, we can test to identify poor implementation, but how about flawed or missing requirements?

3. How can we navigate the ambiguity?

So then, what do we need to navigate this ambiguity? Industries like defense, medical device, aerospace and now even Uber, have realized that this ambiguity can be navigated by implementing an overarching systems engineering process. Indeed, systems engineering is one of the big value drivers for product development companies. It creates economic values, for instance by reducing design errors and changes.

Whether a small startup or a large development team, a tailored systems engineering process would do several good things for us so that we can build the “right” system, i.e. a safe, effective and compliant system that satisfies all the user needs and stakeholder requirements. Mainly:

• Structures the problem and solution domains and intertwine the two
• Increases transparency and alignment in our development activities
• Minimizes the creation of design errors
• Creates an integrated view of the software and hardware disciplines
• Builds a high-level of collaboration and integration
• Organizes the evolving knowledge to maintain the big picture
• Engages customers, users, regulators and stakeholders throughout the design process

How does systems engineering do all these things?! It does it by creating an end-to-end process with the capability of:

• Capturing requirements
• Translating requirements into design
• Making design decisions
• Verifying/validating the design

Going back to our “Therac-25” example, the root cause of its failure was precisely the absence of such a holistic process. These are the topics I am going to further discuss in the following questions.

4. What are the building blocks of a systems engineering process?

Systems engineering is not just about NASA projects, but it also works for small startups. However, it needs to be tailored based on your project needs, stage, complexity and size. Nevertheless, it has some essential domains and related fields that I am listing here. Start from day one. Understand these domains, discover their relationships, connect them together and build interfaces to construct a big picture of your system development process.

Here are the three key domains of your systems engineering process:

• Requirements Management – Requirements are coming from multiple sources: users' needs, regulations, risk analysis, human factor analysis, reliability analysis, etc. How do we elicit, elaborate, analyze and negotiate requirements? How do we create traceability? How do we manage change?
• Design and Integration – How do we design toward integration while requirements are still evolving?
• Verification and Validation – How do we receive feedbacks to ensure requirements are met? How do we engage stakeholders throughout the design process? What is our V&V strategy?

Here are some of the related fields which are essential to your process: 

• Decision Management – How do we make good design decisions? How do we engage stakeholders in decision making?
• Risk Management – How do we build safety into our design? What is our risk assessment process?
• Change Management – How do we review and trace design changes? What level of change management do we need?
• Documentation Management – How and to what level do we capture the evolving knowledge? How do we create traceability?
• Interface Management – How do we create diverse but cohesive teams? How do we create horizontal communication channels between interdisciplinary teams?

 One possibility to construct your systems engineering process is to take these preliminary steps:

1. Blend a top-down design approach with a bottom-up integration approach
2. Add parallel activities including risk management, documentation, etc.
3. Create feedback loops for verification and validation
4. Introduce decision gates at various stages of your process

Build your own process model. Can you visualize it? What are the phases of your process? Where are the decision gates? How do you evaluate each phase?

See this reference for the application of systems engineering methodologies to small satellite development projects. Here is a good example of integration and test management. And this one is an overview of the Kepler systems engineering program.

Consider the following recommendations when constructing your systems engineering process:

• Start building a clear understanding of your systems engineering process from day one. Tailor based on your project needs, size and complexity. Visualize it, break it down into its essential domains, measure it and refine it to a “good enough” point. 
• Create an integrated view of the software and hardware disciplines without over-constraining software engineering or contributing to software risks. The design and development of a complex HW/SW system is more like a marathon than a sprint - more linear and rigid with slower changes and iterations. This is partly because, there is always an interplay between hardware and software development. Understand the similarities and differences between HW and SW development. How do you create such an integrated HW-SW view?
• Create multi-level communication channels. At a high level, implement communications strategies between your customer development, users, clients and product development teams. At a lower level, create interfaces between your interdisciplinary design and engineering teams. What is your communication strategy and how do you make information flow efficiently?
• Identify and take into account all the risk layers (technical–cost–schedule). What is your risk management framework?
• Capture "lessons learned". What can we do better and how can we improve our process?
• Keep in mind that you are going to manufacture your design! Your goal should be building a system that is simple with fewer parts and easy to assemble, but still safe, reliable and compliant. That being said, take into account Design for Manufacturing requirements from day one. On another note, design transfer to manufacturing requires several traceable documents such as the engineering drawings, bill of materials, test reports and so on. Think ahead of time about what you will need for a smooth transition into manufacturing.

5. What are the competing characteristics of our systems engineering process? 

In early stage of system design, we need to find a balance between two process characteristics:

• Process “Rigor” to ensure system safety, reliability and effectiveness
• Process “Flexibility” to allow for evolving requirements, design changes, informal communications and creativity

This in fact presents a unique challenge for adoption of systems engineering to a small startup or a new product development project. Your systems engineering process is further influenced by other constraints such as:

• Limited interactions with customers and users to understand the requirements
• Limited expertise to implement the requirements
• Project constraints such as risks, timelines and budget

6. How can we implement a systems engineering process?

Consider the following steps to implement a systems engineering process:

1. Develop a strategy and roadmap, i.e. understand what systems engineering can do for you, identify what specific problem you need to solve and create a prioritized roadmap for short/long-term objectives
2. Tailor based on your needs. Start from essentials. Measure
3. Establish a systems thinking mindset. This is key to stimulate reflective dialogue, deep insights and informed decision making
4. Build systems engineering capabilities, tools and skills

A good start is to take each of the essential domains and related fields of the systems engineering process (requirements management, risk management, …) and ask “how do we” or “what do we need to”. Examples:

• “How do we or what do we need to” ensure requirements traceability?
• ~ engage users, customers and stakeholders throughout the design process?
• ~ trace design changes?
• ~ capture the evolving knowledge?
• ~ define risk-related requirements and build safety into our system?
• ~ establish consistent assessment of design maturity?
• ~ manage our design reviews?
• ~ manage our communications within interdisciplinary teams?

These types of questions, if the right ones, should guide you toward building tools and capabilities needed to implement your process. Engage your team and your stakeholders to figure out the right questions to ask. Certainly, there are many questions to ask! Start small. Start from the essential questions and build from there. Success in implementation is about testing, failing, learning and improving based on feedbacks. Learn about the best practices, tailor based on your needs and make your process measurable.

Two recommendations:

• Create a focal point and a common language. In the early phases of design, one challenge is to translate system requirements into a preliminary design. At this point you are moving from the problem space into the solution space, or in other words, from a logical design into a physical design. This will be a highly iterative process as we are still trying to figure out our solution, i.e. the “how”. It is essential to create a focal point and a common language to document and share our evolving solution. That focal point could be a preliminary design document and the common language is the systems engineering language. Learn this language to create models of your system and analyze its physical and logical structures. Some of the analysis tools that are useful are: functional flow block diagrams, interface diagrams and timeline diagrams. Specific to software, you could use sequence diagrams and use cases.
• Develop your own best practices and guidelines. These are guidelines that produce the best results for you. For instance, how do you conduct and manage your design reviews? What are the steps you take prior to a design review meeting? How do you manage the review meeting? What are the follow up activities? Document these processes as your best practices. But don’t stop there. Share them with your team, ask for feedbacks and keep improving as you make progress with your project.

7. How do we measure the performance of a systems engineering process?

We need to align measurements based on our needs and design stage. If you cannot figure out the KPIs, your project is likely too complex and large. Find focus. Always start small and from the essentials to prevent project failure. Some questions to ask are: What do you want to measure? How do you want to measure it? How do you translate those measurement results into decision making? Some of these measures could be [INCOSE]: review rate as a process measure, number of deviations from requirements as a product measure and finally number of closed or stabilized requirements as a progress measure.

8. What are the ownership and execution challenges?

We just scratched the surface of the systems engineering concept to manage complex designs. Looking forward, the real challenge surfaces: process ownership and execution. At its core, process ownership and execution are about leadership - creating a shared vision, forming cohesive teams, and finding ways to ignite motivation. We can only start small, stay open, fair and authentic. There are also some questions we need to ask: Who owns the systems engineering processes? Who is accountable for building, maintaining and ensuring the execution of the process? This will be the topic of my next article.

Implementing a systems engineering process fundamentally impacts our culture, the way we work together, and most importantly, the way we think. Because of its impact, you need 100% buy-in from both the development and the leadership teams. And finally, building a system is more than just the system. It is about building collaborations that are mission-driven and fun. To achieve success, we need to build capacity for a systems thinking culture, cultivate openness and build trust. I will talk about this in my next article.

Building a system is more than just the system. It is about building collaborations that are mission-driven and fun. If we get this collaborative culture wrong, we will melt down our resources. We need to figure out how to build a systems thinking culture, cultivate openness and build trust. And That’s the nugget!

Lessons Learned

Ambiguity – From a requirements perspective, the ambiguity in early stages of design mainly originates from unknown and evolving requirements plus our limited understanding about their implementation.

Adverse Outcomes of the Ambiguity – (1) Blackbox process, (2) violation of architectural or system-level design decisions, (3) proliferation of design errors, (4) expensive design changes and (5), and an end-product which does not meet safety, efficacy and performance requirements.

Navigating the Ambiguity – Tailor and implement a systems engineering process as a big value driver to: (1) increase transparency and alignment in your development activities, (2) create a high-level of collaboration with customers, users, regulators and other stakeholders throughout the design process, and (3) reduce the creation of design errors and changes.

Building Blocks of Systems Engineering – Requirements management, design and integration, verification and validation, decision management, change management, risk management, documentation management and interface management. Visualize your process.

Process Competing Characteristics – There are two competing characteristics. (1) Process “Rigor” to ensure system safety, reliability and effectiveness. (2) Process “Flexibility” to allow for evolving requirements, design changes and informal communications. Finding a balance between these two characteristics is a unique challenge for adoption of systems engineering to a small startup.

Implementing the Process – (1) Develop a strategy and roadmap, (2) Tailor based on your needs. Start from essentials. Measure, (3) Establish a systems thinking mindset, (4) Build systems engineering tools and skills.

Measuring Performance – We need to align measurements based on our needs and design stage. What do you want to measure? How do you want to measure it? How do you translate those measurement results into decision making?

Process Ownership and Execution – Who owns the systems engineering processes? Who is accountable for building, maintaining and ensuring the execution of the process? To achieve success, we need to build capacity for systems thinking and get 100% buy-in from both the development team and the leadership.

The Nugget – Building a system is more than just the system. It is about building collaborations that are mission-driven and fun. If we get this collaborative culture wrong, we will melt down our resources. We need to figure out how to build a systems thinking culture, cultivate openness and build trust. And That’s the nugget!