How can you ensure criteria used in multi-criteria decision analysis are relevant and consistent?

1. Define criteria: Think about what matters most to you in a job. 2. Involve stakeholders: Seek advice from your professors, career services, or alumni who have experience in the industry. They can provide insights based on their experience 3. Assess importance: Assign weights to each criterion based on your personal preferences. 4. Consistency check: Compare each criterion against the others using pairwise comparison. Make sure there are no conflicts or inconsistencies in your ratings. 5. Review and update: Stay open to new information or insights throughout the decision-making process. 6. Validate criteria: Talk to professionals in the field or conduct informational interviews to validate the relevance of your criteria.

In my experience, to make sure criteria in decision analysis make sense and stay consistent, first, clearly state the goals and objectives of the decision. Get everyone involved to share their thoughts. Second, rank the criteria based on how much they help achieve these goals. Make a clear list of what matters most. Third, check that each criterion is easy to measure and connects directly to the goals. Discuss with everyone involved to agree on their importance. Keep reviewing and updating the criteria to match any changes in the situation. This way, your decision-making stays on track and makes sense to everyone involved.

Non-textbook response… a significant portion of primary research is driven by problem solving behavior. Researchers and professionals alike are more likely to succeed with uncovering a viable solution….as long as their focus remains objective and value driven.

Be very clear on the criteria/ variables that you deem important. Let me use a data science example. Let's say you want to cluster your customers into segments. If you provide something irrelevant like height into your clustering algorithm, the algorithm will consider that together with other useful variables like income, purchase frequency etc. Then you have clusters that do not serve your needs. Are you going to do your marketing according to customer height?

One good thing I have learned is that in multi-criteria decision analysis, ensuring the relevance and consistency of criteria involves defining the problem and objectives clearly. This initial step helps in narrowing down the scope and focus of the criteria, aligning them with specific, measurable, achievable, relevant, and time-bound (SMART) objectives. The rest are... peripheral.

In the realm of multi-criteria decision analysis, crafting a meticulous criterion foundation is pivotal. Imagine an urban planning scenario faced with the task of selecting an eco-friendly transportation system. Beyond the obvious factors of efficiency and sustainability, they structured their criteria to include community impact, assessing not just environmental but also social repercussions. This novel approach led to a nuanced criterion structure, ensuring a comprehensive evaluation of alternatives. A carefully structured criterion not only maintains relevance but also eradicates redundancy, fostering astute decisions aligned with overarching objectives.

The criteria should triangulate towards common operationalized variables. Accordingly, if all of your criteria point in different directions, you either have poor measurement, an ill-defined model, or something unexpected happening.

It's necessary to understand the criteria because it'll help you with other things. It's like the base here, the criteria is necessary because it's very closely related to the problem that you're solving.

The Keep it simple rule applies here. It's also a good idea to group criteria in order to simplify the analysis and ensure you keep a manageable number of priorities for your decision.

To ensure criteria relevance and consistency, start by identifying and structuring them based on the problem and objectives. Criteria are the measurable attributes used for performance comparison. For instance, in cost minimization, criteria could include unit price, shipping cost, and product maintenance. Avoid overlapping or conflicting criteria to prevent inconsistencies. Use either a hierarchical or flat structure based on problem complexity. This ensures comprehensive and conflict-free criteria, enhancing the reliability of the decision-making process.

I tell a lot of Data Governance Managers that QA has to be in multiple places within your process. If you only check your work once you run the risk of a multivariate model running awry. There is a balance, I know someone is going to bring up diminishing returns of multiple layers of QA, but each project will dictate how much you need to review. But case in point, don't be afraid to continually check your work for the accuracy and validity of the model.

Clearly define the overarching goal or objective at the start. All criteria should relate directly back to helping achieve this goal. If a criterion doesn't map to the goal, it's likely not relevant.

Involve key stakeholders early in determining criteria. Get their input on what factors are most important for the decision. This helps develop criteria based on diverse needs and values.

In my experience, validation ensures accuracy and meaning through data, expert input, or stakeholder feedback. Prioritization involves assigning weights to reflect importance, using methods like ranking or pairwise comparison. This approach aligns decision criteria with reality, enhancing reliability and coherence in the decision-making process.

Assess Criterion Independence; Check for overlap or redundancy among criteria. If two criteria measure similar aspects, consider combining them or removing the less important one. This prevents double-counting and ensures each criterion provides unique information. Prioritizing Criteria 1. Assign Weights Use methods like pairwise comparison, swing weighting, or point allocation to determine the relative importance of each criterion. This establishes a hierarchy of importance, reflecting their influence on the decision. 2. Conduct Sensitivity Analysis: Explore how changes in criterion weights impact the final decision ranking. This identifies critical criteria that significantly influence outcomes and highlights potential biases.

An example in financial services where this is used amongst many other use case is to assess credit facilities. The process involves defining relevant criteria like interest rates, loan terms, credit limit, fees and charges, collateral requirement. Next step of weighting them to each criteria against goal and outcome and then evaluating alternatives This is followed by normalising data to make it comparable against standard scale and then aggregation. Sensitivity analysis to assess how changes in criteria weights or variations in credit terms might impact the decision.Ensuring consistency in decision-making is pivotal and finally ranking the credit facilities based on their aggregated scores to make an informed decision.

In my field of work thats Supply Chain, while contract management there are multiple criteria’s basis which a service provider is finalised. - Understand the problem statement why you want to switch a provider / define what you want to achieve while starting a new business. - Evaluate their numbers to what you are seeing from existing / look at the number if they are nearing your goals - Prioritise the criteria and weigh them using six sigma methodologies - Once finalised always validate and analyse the performance

MCDA is/can be used in fintechs for example on the loan approval process under the procedures below Trade-off analysis: Analyze the trade-off between approving a high-risk borrower with valuable collateral versus a low-risk borrower with minimal collateral. This helps define acceptable risk/reward thresholds. Documentation: Document the rationale behind each criterion's inclusion, its data source, and its assigned weight. This clarifies the decision-making process and facilitates future audits or adjustments. Review and Update: Monitor loan performance data to assess the effectiveness of the criteria weighting. Update criteria and weights periodically to reflect changes in market conditions or customer preferences.

Ensuring relevance and consistency in multi-criteria decision analysis involves a human touch. Start by involving stakeholders to define crucial criteria, ensuring they reflect real-world needs. Consistency comes from aligning these criteria with the overarching goal and each other, avoiding overlaps or contradictions. Regular validation through feedback loops ensures criteria relevance amid changing contexts. Human insight fine-tunes these criteria, maintaining their effectiveness in decision-making.