Model Maturity: The Co-Design Web
This is a synopsis -- for a deeper discussion, please proceed to the article on Medium https://richardarthur.medium.com/co-design-web-6f37664ac1e1 or slides (PDF).
I welcome discussion of this topic and my goal is to build some consensus around a tool that can be mutually referenced and applied across businesses and industries to assist in building much-needed support for investments in maturing modeling software.
Goal: A Rubric for Model-based Strategies
The existence of a framework through which we can clearly assess and describe the readiness of a modeling approach enables us to communicate:
- identified risks and gaps (as opportunities for improvement)
- comparative strengths (across alternative approaches, including physical)
- aspirational capabilities (to pursue as a pragmatic target or pilot study)
to guide decision-making relative to strategies, investments, prioritization of resources, and responsible limitations on use.
It is essential the rubric consider multiple factors, not attempt to derive an over-simplified score that misleads without considerable further explanations and caveats. Therefore, the rubric should communicate loosely-coupled factors independently and simultaneously.
Additionally, the rubric should afford a variety of acceptable targets based upon context of use. Therefore, the weighting or pass/fail across different factors should be adjustable.
Form and Function
Regarding the form of the visualization, my personal experience employing a “spiderweb” chart with diverse audiences over the past decade would indicate it sufficiently if not exceptionally intuitive for the task. Additionally, the Excel “Radar” chart can be used to easily create, modify and render assessment instances. The center of the web is a zero score and the outer rim is the best score (typically a 5). The chart below contains two solution options: red performs poorly compared to green.
Red scores a 1 in Realism, Accuracy, Maturity, Productivity and Robustness, a 0 in Confidence and a 2 in Compatibility and Scalability. Green scores a 4 in Realism, Confidence, Maturity, Productivity, and Compatibility and a 5 in Accuracy, Scalability and Robustness.
It is also my experience that this chart is most compelling when it has exactly eight axes. I won’t debate this here — just try examples and I think you’ll agree. As a bonus, this also compels a necessary simplification of any rubic with more than eight factors.
The pragmatic function of the rubric is clarity and completeness in assessing and communicating the risks, gaps, strengths, and aspirational targets as described in the goals above. Specifically, three functional categories for the rubric to characterize are:
- TRUST in the approach: Model Competence
- VALUE in adopting: Cognitive Augmentation
- DURABILITY of that TRUST and VALUE: Architecture
Model Competence
Perhaps the greatest perceived risk in shifting from physical to digital models is the well-justified distrust in model results often cited through the adage “Garbage In — Garbage Out.”
Therefore, before any other considerations, our rubric must consider attributes of the model to test its credibility in the context of the intended use. Specifically, how can we characterize trust in the modeling approach?
Can we assert a “Region of Competence” for a model: where its use is numerically stable (ROBUSTNESS) with minimal simplifying constraints (REALISM) and quantify the bounds of uncertainty (CONFIDENCE) of results with validated, predictive ACCURACY?
Formalisms around “VVUQ” (q.v.) — verification, validation and uncertainty quantification embody an essential component to delivering model competence. Sandia’s PCMM, (q.v.) — predictive capability maturity model is an exemplar rubric for these factors. Further, employing model competence requires expertise in numerical and measurement disciplines to apply correctly, unless wholly embodied into the models.
Cognitive Augmentation
No matter how good the model may be, it is only valuable if it can be used efficiently and effectively. Here we consider the net benefit to the end-user (and ease to expand user community) as well as effort needed for continuity of use by the support team. In effect — can we rely upon stable operational availability and consistently beneficial throughput?
Can we be affirm the labor invested in software management and end-use of the model will: yield PRODUCTIVITY from efficient workflows, reduce waste/rework and improve quality (MATURITY)?
The software Capability Maturity Model (q.v.) and Agile development practices like continual test/integration/deployment contribute to implementation stability and quality, while the UX (user experience) practice can significantly improve human-machine interaction and workflow productivity.
Architecture
The final pitfall to examine considers the modeling software and implementation approach within the organic systems architecture spanning the software ecosystem and ever-advancing disruptions in the underlying hardware infrastructure. If indeeed we trust the approach and finds its use valuable, is the approach agile to change?
Can we assure the implementation will perform capably on current and emerging HPC hardware (SCALABILITY), is interoperable with other software and feature-extensible (FLEXIBILITY)?
In short — cleverly balancing adoption of frameworks and standards for compatibility, portability and optimization vs. added effort to employ design abstractions and hedge implementation against a vast set of speculative architecture features.
The Co-Design Web
Through these eight goals, we then form our spiderweb as a lens into the trustworthiness of the modeling approach, the ease with which it can be deployed, used and supported to yield valuable throughput in the science and engineering — and then the architectural adaptability relative to both hardware and software. The knowledge and expertise to make such assessments spans domain expertise, computational science and systems engineering — components of what the national labs have called Co-Design.
Scoring Factors
Guided by 4-to-5 graded frameworks such as CMM or PCMM, we can abstractly measure a score in a generalized form:
Realism: Can solve problems relevant to collaborative decision-making with acceptable, minimal or no assumptions /abstractions / simplifications.
Accuracy: Can cite results to key decision-making stakeholders without needing additional supporting data.
Confidence: Can credibly bound the error on solution results informing decision-making.
Robustness: Can assert limitations on valid parametric solution space over which the model can be applied and assess input sensitivities.
Productivity: Can (DevOps modify/build/test/install)/(Users set up problems, run & interpret results) w/ minimal instruction /effort — modern tools, processes, scripts.
Maturity: Can confidently add new (DevOps to manage)/(Users to apply) w/ minimal instruction /effort /rework — modern UX & build/test/install/execute.
Scalability: Can reduce wall-clock time and/or increase problem size/complexity w/ additional hardware — and/or efficiently apply over large problem ensemble.
Flexibility: Can adopt/adapt to new architecture ecosystems and use case workflows w/ minimal instruction / effort — modern design for build/ interoperate + leverage community.
(More detail can be found in the original article on Medium: Co-Design Web)
Conclusion and Opportunity
Digital technology can deliver immense value in providing tools that are particularly effective against the “seven deadly wastes” (or Muda) of LEAN. Reusable virtualized assets counter physical inventory, production lag and overproduction. Automation-driven consistency and instant situational visibility increase productivity and reduce waste, defects and rework. Model-based digital thread workflows streamline enterprise “motion” and reduce dependency-based waiting/hand-offs and underuse of talent.
Advances in computing hardware have driven development of sophisticated software systems now blossoming in the connected data and processes of the digital thread — powering the transformational practice of digital engineering. But the reliance upon the models — trust in the models, value from the models, durability of investment in the modeling approaches, these are what urge the assessment of modeling maturity.
The Co-Design Web is proposed as a tool for assessing these factors to advise in selection between alternative strategies, to set targets for desired state of practice and to guide steps when setting a roadmap toward those targets.
Additional References
- Presentation Co-Design Web: Software Maturity Charts (PDF)
- “Advancing Scientific Productivity through Better Scientific Software”, Exascale Computing Project, Jan 2020, Office of Advanced Scientific Computing Research and Office of Advanced Simulation and Computing, U.S. Department of Energy, ECP-U-RPT-2020–0001
- [See also Productivity and Sustainability Improvement Planning (PSIP), Better Scientific Software (BSSw) / Interoperable Design of Extreme-scale Application Software (IDEAS)]
- Verification, Validation and Uncertainty Quantification (VVUQ), American Society of Mechanical Engineers (ASME) Codes and Standards
- “Unified Framework and Survey for Model Verification, Validation and Uncertainty Quantification”, Riedmaier, S., Danquah, B., Schick, B. et al., Archives of Computational Methods in Engineering (2020)
- “Explore”, U.S. Council on Competitiveness, 2019.
- U.S. Department of Defense “Digital Engineering Strategy”, Office of the Deputy Assistant Secretary of Defense for Systems Engineering, prepared for Sec. James Mattis, June 2018
- “Simulation Credibility: Advances in Verification, Validation, and Uncertainty Quantification”, Mehta, U., Eklund, D., Romero, V., Pearce, J., Keim, N., NASA Report JANNAF/GL-2016–0001, Nov. 2016
- “Uncertainty quantification’s role in modeling and simulation planning, and credibility assessment through the predictive capability maturity model”, Rider, W., Witkowski, W., Mousseau, V., Sandia National Lab, Report SAND-2015–20747J, doi:10.1007/978–3–319–11259–6_51–1
- “Simulation Credibility Scale and Credibility Assessment Scale”, Mehta, U., NASA Modeling and Simulation Subcommittee (MSS), Simulation Credibility Workshop, Dec. 2011
- “The Web of System Performance”, Whitworth, B., Fjermestad, J., Mahinda, E., Communications of the ACM, May 2006/Vol. 59, №5.
- Capability Maturity Model Integration (CMMI) via CMMI Institute, formerly Software Capability Maturity Model (CMM), Software Engineering Institute, Carnegie-Mellon University