How to Use the EASA Management System Assessment Tool (MSAT)

Sofema Online (SOL)??reviews MSAT Guidance and Best Practices

Introduction

The Management System Assessment Tool (MSAT) is designed to assist organizations and assessors in evaluating both the compliance and performance aspects of a Management System (MS).

MSAT is a crucial tool for ensuring that an organization's Management System is not only compliant with regulatory requirements but also effective in managing safety and promoting continuous improvement. By following the structured guidance and best practices outlined in this document, assessors and organizations can effectively use the MSAT to evaluate and enhance the maturity and performance of their Management Systems.

• It emphasizes the necessity for a Management System to not only adhere to regulatory and internal requirements but also to effectively manage safety, establish safety objectives, monitor their achievement, and facilitate continuous improvement.

Key Principles

Dual Focus:?Ensure the MS is both compliance-oriented and performance-oriented.

Comprehensive Understanding:?Assessors and organizations must understand performance-based expectations in addition to prescriptive requirements.

• Beyond Compliance:?While compliance with prescriptive requirements ensures that an organization meets minimum legal and regulatory standards, it doesn't necessarily equate to optimal performance or safety. A comprehensive understanding recognizes the importance of going beyond compliance to achieve excellence in operations and safety management.
• Dynamic Operational Environment:?Modern operational environments are complex and ever-changing. Relying solely on prescriptive requirements may not be sufficient to address new or unforeseen challenges. Performance-based expectations allow organizations to adapt and innovate in response to changing conditions and emerging risks.

Maturity Model Application:?Use the grading system (Present, Suitable, Operating, Effective) as a maturity model to evaluate the MS's performance and safety management capabilities.

Working with a Maturity Modelling Process

The concept of applying a Maturity Model, specifically through the grading system of "Present, Suitable, Operating, Effective," is a strategic approach to evaluate and enhance the Management System's (MS) performance and safety management capabilities within an organization.

This approach allows for a nuanced assessment of the MS's current state, its alignment with organizational goals, particularly in safety management, and its overall effectiveness.

Here's a breakdown of the meaning and deployment of each stage within this maturity model:

Present

• This initial level indicates that the essential elements of the Management System are documented and in place.
• At this stage, the organization has recognized the need for a structured approach to manage safety and compliance but has not yet fully implemented or operationalized these concepts.
• Organizations should ensure that all required MS components are identified, documented, and formally recognized within the organization.
• This includes policies, procedures, and responsibilities.

Suitable

• At this stage, the MS is appropriately designed for the organization's specific activities, complexities, and interfaces.
• It signifies that the system is not just present but is also tailored to meet the unique needs and challenges of the organization.
• Conduct a thorough review to ensure the MS's design effectively addresses the organization's operational context.
• Adjustments should be made to ensure the MS is not only suitable in theory but also in practice, particularly in how it manages safety and interfaces with other operational areas.

Operating

• This level indicates that the MS is active and functioning as intended.
• The focus here is on the operation and scalability of the system, ensuring it can adjust and respond to the organization's changing needs and conditions.
• Monitor and evaluate the MS's performance in real-world operations.

>> This involves assessing whether the system is scalable, flexible, and capable of managing safety effectively under various operational conditions.

>> It also includes identifying areas for improvement to enhance maturity.

Effective

• The highest level of the maturity model signifies that the MS is not only operating as designed but is also achieving its intended outcomes, particularly in managing safety and facilitating continuous improvement.
• The system has a positive impact on the organization's safety culture and overall performance.
• Implement a continuous monitoring and feedback loop to assess the effectiveness of the MS in achieving safety objectives and promoting a culture of continuous improvement.

>> This involves analyzing safety performance data, learning from incidents and near-misses, and making iterative improvements to the system.

Implementation Steps for the Maturity Model:

Assessment:?Begin with a comprehensive assessment of the current MS using the maturity model as a framework. This involves evaluating each element of the MS against the four maturity levels to identify its current position.

Gap Analysis:?Conduct a gap analysis to identify discrepancies between the current state of the MS and the desired maturity level. This analysis should highlight areas requiring development or enhancement.

Action Planning:?Develop an action plan based on the gap analysis. This plan should outline steps to move each element of the MS to the next maturity level, including resources required, timelines, and responsible parties.

Implementation and Monitoring:?Implement the action plan and monitor progress regularly. This involves tracking improvements, assessing the impact of changes, and making adjustments as necessary to ensure continued progress towards the desired maturity level.

Review and Continuous Improvement:?Regularly review the MS's performance and effectiveness. Use feedback and performance data to refine the MS continually and address emerging challenges or opportunities for improvement.

Using the maturity model as outlined allows organizations to systematically improve their MS's performance and safety management capabilities, moving from merely having a system in place to ensuring it effectively supports the organization's safety objectives and operational needs.

Assessment Guidance - Preparation Phase

• Understand Organizational Context:?Assessors should familiarize themselves with the organization’s size, complexity, and the nature of its operations.
• Identify Relevant MS Elements:?Select MS elements relevant to the organization's operational context, avoiding a one-size-fits-all approach.

Assessment Guidance - Evaluation Phase

• Document and Define (Present):?Verify that the MS elements are documented and defined, ensuring that the basic structure of the MS is in place.
• Assess Suitability:?Evaluate whether the MS is appropriately designed for the organization's activities, complexity, and interfaces.
• Evaluate Scalability and Operation:?Determine if the MS is scalable and operating as designed, identifying areas where maturity can be enhanced.
• Measure Effectiveness:?Assess if the MS is effectively managing safety, achieving safety objectives, and fostering continuous improvement.

Best Practices

• Avoid Compliance Checklist Approach:?Use the MSAT as a guide rather than a checklist to ensure a comprehensive evaluation of the MS's maturity and performance.
• Focus on Safety:?Ensure the assessment prioritizes the evaluation of safety operations and the identification of major risks.
• Customize Assessment: Tailor the assessment approach based on the organization’s maturity level, possibly starting with sections more relevant to the current state of the MS.

Conducting the Assessment

• Flexible Sequencing:?There is no fixed order for conducting the assessment; it should be based on the assessor's judgment and the organization's specific context.
• Risk Management Focus:?Consider starting with sections on Safety Risk Management and Safety Assurance to understand the organization's approach to identifying, mitigating, and monitoring risks.

Post-Assessment

• Re-evaluate Suitability and Scalability: After a comprehensive evaluation, reassess the MS and its processes to confirm they are still aligned with the organization's size, complexity, and nature of operations.

The tool Serves both for Compliance and Performance.?It is the responsibility of the organisation and the assessor(s) to ensure that:

• The relevant Management System (MS) elements are present and suitable for the proper functioning of an MS (compliance-orientated);
• The Management System (MS) operates and delivers as expected (performance-orientated).
• The assessment of an MS requires understanding of?performance-based expectations, in addition to the more traditional prescriptive-based requirements.
• These understanding spans both the organisation and the overseeing authority.

Delivering a Performance Assessment

Beyond ‘traditional compliance’ as well as factoring in the size of the organisation together with considerations related to the complexity/nature of its operations:

• The evaluation of an MS should consider the overall capability of the organisation to effectively

>> Manage safety,

>> Set-up safety objectives;

>>? Monitor them and achieving the intended targets.

Note - The grading system ‘Present, Suitable, Operating, or Effective’?(PSOE) should be used as a maturity model, so that the assessment becomes performance orientated, evaluating:

• How the organisation globally performs,
• Manage safety, and
• Ensure safe operations.

The assessment should particularly focus on the levels of maturity, as to whether:

• The MS, globally or for each element, is documented and defined – to a certain extent, it refers to ‘Present’; or
• The design of the MS looks good and appropriate to the activities, nature and complexity, interfaces etc. – in other words, the MS looks ‘‘Suitable”; or
• The MS is “scalable”, “operating” and delivers - there is evidence that the MS operates as designed; however, the desired output is not yet attained and/or more should be achieved as the MS has not yet reached the expected level of maturity; or
• There is evidence that the MS is globally agile and working in an ‘‘effective’ way, having a positive safety impact and is striving for continuous improvement; the desired effect is attained through the achievement of the safety objectives.

Important Note Concerning Useability

• The tool should not be used as a compliance checklist?to verify that all the individual elements of an MS are in place.
• Assessing a MS is not a ticking box exercise where each and every line of the “What to look for” for each section of the tool are complied with.
• It is the responsibility of the assessor to appropriately use the tool ONLY as a guide?to get prepared before the assessment; select, as necessary, what is relevant (based on the size, complexity and the nature of the operations); and provides an overall evaluation of the MS in terms of “maturity” and “performance”.

Important Note Concerning the Meaning of Compliance

• “Being compliant” does not necessarily mean “being safe”.
• Zooming into each element of this tool and notably verifying each process ‘step by step’ or ‘word by word’ runs the risk to be too narrow and may mislead the assessor back to “compliance” and miss out on the ultimate objective: evaluate how safe the operations are and ensure that no major risk has been overlooked.

Evaluation of “Suitability” and “scalability”

• MS “suitability” is specific to each individual organisation and impossible to define for all types of activities and sizes of organisations.
• Some organisations may be small in size but carry out safety critical operations with many interfaces.
• Sometimes the regulatory framework in the domain proposes some guidance criteria per topic, such as ‘complex organisation’ versus ‘non-complex organisation’ to address the human resources; sometimes it does not.

Recommendations:

• Before the assessment to analyse what is appropriate to the organisational set-up;
• At the end of a comprehensive MS evaluation to re-consider whether the overall MS and its associated processes are commensurate with the size of the organisation and the complexity/nature of its aviation products or services.

Note

• Neither “suitability”, nor “Scalability” is about applying specific elements of a MS; or going for a light/complex MS: it is about adapting a MS with all its elements to the specific operational context of the organisation.
• It is the responsibility of the organisation to determine the suitability and/or scalability of its MS and demonstrate to the overseeing Authority that the MS is?appropriately designed and suitable?to effectively deliver as expected.

Conducting and sequencing the MS assessment

• There is no particular order on how to use the different sections of the tool: it is left to the assessor’s discretion to decide on how to conduct an MS assessment.
• Sometimes, starting with section 2 (Safety risk management) then section 3 (Safety assurance) of the tool may be more appropriate, depending on the maturity of the organisation.

Example of Question Relevance:

• The following questions can be relevant:

>>? “What are the main risks of the organisation?

>>? “How does the organisation know it?”

>>? “How does the organisation mitigate these risks?

>>? “How does the organisation know that the mitigation measures are effective?

>>? “Are the safety objectives monitored and ultimately achieved?”

>> “How is it communicated to the senior management and the staff? Etc.

Sometimes, in term of initial deployment or MS implementation, it can be beneficial to understand first-hand the progress made by the organisation, its challenges and then the next steps to complete, before reaching a higher level of maturity.

Next Steps

Please see Sofema Aviation Services ?and Sofema Online or email?[email protected] ?for questions, comments or suggestions.