What are the most common challenges in simulation design and validation, and how can you overcome them?

I like to think of defining the problem and objectives similar to drawing a map before starting a journey. It offers the following benefits: 1. It ensures everyone knows where they're going and follows the same path, preventing confusion or going in circles. 2. It helps save time and money by focusing only on what's important. 3. With clear goals, you can easily check how well you're doing and if you're on the right track. 4. If everyone understands and agrees on the problem, they're more likely to trust and use the results. To sum it up, without a clear plan, the simulation might end up being a lot of work without much real-world use.

A major challenge in simulation design and validation is the lack of a profound understanding of the underlying core domain knowledge (i.e., the physics of the system) and essential mathematical and problem solving skills. This challenge is prevalent across the industries. Solutions to these challenges involve passion to deliver results by continuous education, interdisciplinary collaboration, and the development of user-friendly simulation tools. These strategies bridge knowledge gaps and empower professionals from diverse backgrounds to create accurate simulations, ensuring that these models faithfully replicate real-world systems.

I would say: "Stop playing Chess with yourself". Find your opponent and immerse yourself in a real problem. Even as a talented engineer or scientist and even if you have all the knowledge and equations at your fingertips, remember that true progress happens when you work with stakeholders users and people to tackle real-world problems. Building a high-end model in your university office, is more like playing Chess with yourself, where you know and follow all the rules clearly but do not solve a problem.

A well-defined problem is half-way done to get the expected results. Sometimes, just reading books or articles is not enough to set a well-defined problem - you need to engage with the domain. Specially, in simulation we make lots of assumptions, those need to be aligned with practical scenario.

First take your time understanding the system you are simulating, and determine what is needs to be modeled and what doesn’t. Study how these components interact and what data you need to model them. Simulation outputs can be a lot, and could be conflicting. Define the kpi and outputs that define you objective and find a way to qualify them like scorecards and Pareto points

Once your define your system components clearly, you should identify what goes into each one, and that can help you figure out what data is needed. Dig deep into the data because a lot of times, the statistics and distributions that define a process can vary by the time or status of the system. For instance instead of having one arrival rate for the simulation agents, you can have multiple arrival rates for different days or weeks or even part of the day

I don't see that defining the problem and collecting data are separate steps. The availability of data is part of the game. If you define the problem and the model to solve it wisely, it is more likely that the next steps will go more smoothly.

Collecting appropriate and reliable data sometimes challenging. In lot of cases I see modeler generate data by following distributions. That may tell you approximate or aggregate level stories. Even though the data pattern may not align in individual level stories.

Gathering real-world data for model development and validation can be challenging, for example, when dealing with sensitive, rare, or expensive data. If data is sensitive, one needs to make sure it is compliant with laws and regulations (like GDPR, HIPAA, and FDA). For this purpose, one need to provide data de-identification or anonymization techniques. If data collection is expensive (e.g., data annotation), one way to overcome this challenge is to apply active learning methods. If data is rare, one may employ synthetic data . Validate the simulation against multiple data sources, and employ statistical methods to ensure the accuracy of the model.

In data analysis, appropriate model simulations are very necessary with the aim of reducing biased data. Challenges in data analysis if it is questionnaire data require the following: 1. Objectivity of respondents 2. Increase the inclusion of respondents so that they are right on target 3. The method of distributing questionnaires is traditional without writing down your name to reduce discrimination if the results of filling in are critical

Based on the structure of the problem and objective we need to decide what kind of model is suitable. I have developed an agent-based model for studying people's health seeking behavior. I have started with agent based as different agent play and interact with each rather. Later, I realized that I have some part in the system that has discrete event nature. Finally, came up with hybrid type with both agent-based and discrete event simulation.

Choosing the right methods for operations research is a critical decision. It is not a one-size-fits-all approach; rather, it depends on the specific problem at hand. It is not a random selection; it is a deliberate process involving careful consideration of data availability, problem complexity, and the research objectives. I have learnt that operations research methods vary from mathematical modeling to data analytics, simulation, and optimization techniques. The key is to match the method to the problem, ensuring that it is the most effective tool for uncovering insights, making informed decisions, and achieving tangible results in the real world. In operations research, method selection is a strategic choice, not a random one.

When it comes to model selection, Data Scientists often go for the most complex and accurate model, which is wrong from the business perspective. A real business need is to solve customers' problems. For example, you have to select between two models: based decision trees and, let's say, modern Deep Neural Networks. The first one is not super accurate, but its performance satisfies the customer, and it's simple and interpretable unlike the Neural network, which is highly accurate, but looks like a complete black box for the customer. In this case, it would be better to select the simple model, because if the model cannot provide explanations regarding its predictions, customers will not trust the model.

My advice is: Do not use ABMs for predictions. It can only be used for learning about the boundary conditions and general behaviour of the systems. The choice of modelling approach is certainly an important factor in success of a scientist.

Many models decompose complex interactions down to simple ones. For example, in a complex combat simulation, somewhere in the code, there's a part where the two objects/soldiers interact with each other and there's often a dice roll to decide who wins. Often, when validating the simulation, it's done at the macro level where the results appear correct. But sometimes there's a -1 on the wrong line of code somewhere deep inside the simulation. You'll probably never find it. Also, most simulations worth doing can't be validated. For example, if you're simulating a large war in Asia. There's too much going on so the simulation is interesting or informative but it's unlikely to be correct. You can also use pandemic models as a proof point.

One useful validation tool are what in my company we called "Scenario Benchmarks", essentially the equivalent of a unit test for simulation: In collaboration with domain experts, we collected simple (!) real-world scenarios that could arise, paired with an expectation of the model's behaviour. We'd have a set of them and running those was part of the standard validation pipeline when making changes to the model

Many large real-world systems do not have much data that allow for validation. In this case it is crucial to develop the simulation in conjunction with the necessary subject matter experts. While this is not perfect, in many large and complex systems it can help ensure that you are working in the correct ballpark.

Part of this would be assessing model performance on important subsets of the population as well as overall (using data that wasn't used to build or prune the model), to include protected classes it is illegal to discriminate against.

In my experience - this is very tricky. I saw some literature, they validate their model by benchmarking. What happen if the benchmarked model have some holes. Model benchmarking, in which model outputs are compared with those generated using a different methodology, has been used as a model validation tool for many years, at least since the publication of regulatory guidance on Model Risk Management in SR 11-7 more than a decade ago.

This important to make your model generalize. In my cases - I developed a model and implemented in a study area in USA. Someone in Europe asked me - parameter that you used in you model has no existence in my area - how your model will deal with that? That parameter in my model act as weight - so I said it will be 1 in the model, and model will still produce feasible and sensible results. An exhaustive experiment and analyzing results are keys to show the significance and efficacy of the model.

My graduate advisor is fond of saying "A story has a beginning, middle, and an end." This is obvious, but when presenting results of the simulation, there's a huge tendency to present just the middle. The stakeholders for the simulation need to know enough about how the simulation was constructed, the models used, and the limitations to understand and trust the results. But they don't need your whole life story. They don't need every detail of every algorithm and every calculation. They also don't need a rehash of why it was hard and why it took so long and how you haven't even played Starfield because you were so busy. Don't spend all your time on the middle of the story or you will run out of time to talk about the

Presenting simulation optimization model to stakeholders, especially non-technical audiences, could be challenging. To overcome this, talk from business perspective such as what this model is bringing to the table in terms of business KPIs.

The bottleneck in the whole process of modelling and using a model is the "cognitive load on the modeller and the users". In the world of AI and modelling, it is easy to fall into perfectionism, whether in modelling or in presenting the results. Imagine, many real world problems are complex and multi-criteria. Models have also evolved to the highest stage. However, in the end, it is people who need to understand the results and apply them in practice. Although multiple dimensions of the problem can be analysed, we humans have limited cognitive abilities to understand the results. The highest dimensional representation we can show are 3D scatter diagrams :))

Please see the book A. Kolker "Healthcare Management Engineering: What Does This Fancy Term Really Mean? The Use of Operations Management Methodology for Quantitative Decision-Making in Healthcare Settings", Springer, 2012. The book provides multiple examples of applications of management engineering/operations research methodology. Detailed examples of applications are provided for discrete event simulation, queuing analytic theory, linear and probabilistic optimization, forecasting of a time series, among others. A summary of some fundamental management engineering principles is provided. The 2nd Edition of the book is being prepared and will be submitted to the Springer publisher next month.

Simulations fall on a spectrum from fully-generic to purpose-built one-offs. Many times it can be hard to figure out which was really called for until months after the initial project ends. In my experience, don't go into a new simulation assuming it will be used for the next 10 years, but try to inject flexibility in the internals where ever it wouldn't increase project time.