50 Question Retrospective: A SecOps Questionnaire to Accelerate The New Year

There is a reason why reflection and introspection is such a powerful tool, not only in your personal life but also for improving your security operations team. From the rising frequency of cyber threats to the evolving tactics used by bad actors, the statistics underscore the critical importance of well-defined security operations. In this article, I’ll teach you how to sharpen your focus by using last year’s data and how organizations can leverage a Security Operations Questionnaire to set actionable, strategic goals for the year ahead. By aligning our security measures with these findings, we can build a more resilient and proactive approach to safeguarding our systems and data.

Expected vs. Actual

In cybersecurity planning, the gap between expected outcomes and actual results often highlights critical lessons for improvement. Organizations typically set expectations based on threat modeling, historical data, and resource allocation, aiming to reduce vulnerabilities and respond effectively to incidents. However, actual results can deviate due to unforeseen factors like emerging threats, resource constraints, or gaps in execution. These discrepancies do a disservice to the importance of continuous assessment and adaptability in security strategies. By analyzing the variance between expected and actual outcomes, organizations can identify blind spots, refine their plans, and allocate resources more effectively to meet evolving cybersecurity challenges.

2025 Goal Setting

Goal setting for your security operations team in 2025 is more critical than ever. As the threat landscape continues to evolve, setting clear, measurable objectives provides a strategic roadmap to address emerging risks and strengthen your organization’s defenses. Well-defined goals align the team’s efforts with broader business priorities, ensuring resources are focused on the most pressing security challenges. For 2025, I will be using three areas: Roster Goals, Incident Response, and Vulnerability Management.

When using this questionnaire, keep in your mind the actual vs expected values and 2025 goals.

Roster Goals

1. Number of people (Employees & Consultants) in Security Operations
2. SecOps annual cost of Tier 1 services (Operational Activity)
3. Security Resource annual cost of Tier 2 services (Engineering and Project Work)
4. Security Resource annual cost of a Tier 3 services (Leadership)

SecOps Incident Review

1. On average how many days is the lifecycle (detection to closed) of a Security Incident within your environment?
2. On average how many times does the same (or un-actionable) threat generate an Incident today before it is permanently corrected?
3. What percentage of Security Incidents do you create a Post Incident Report for?
4. Do you have a dedicated Incident Response team inside Security Operations? Y/N
5. Approximate number of Security Events detected per month
6. How many Security Events are closed by first line Security Analysts per month
7. Number of Security Incident Tickets handled by Security Incident Response Process per year
8. Average number of effort hours needed to resolve an Incident
9. Average elapsed time from Incident Ticket creation to Incident Ticket closure (days) by Tier 1 resources
10. Average effort time by Tier 1 resources to close an Incident (hours)
11. Percentage of Incidents resolved by Tier 1 resources
12. Average elapsed time from Incident Ticket creation to Incident Ticket closure (days) by Tier 2 Resources
13. Average effort time by Tier 2 resources to close an Incident (hours)
14. Percentage of Incidents resolved by Tier 2 resources
15. Average elapsed time from Incident Ticket creation to Incident Ticket closure (days) by Tier 3 resources
16. Average effort time by Tier 3 resources to close an Incident (hours)
17. Percentage of Incidents resolved by Tier 3 resources
18. Approximate number of Security Incidents managed per month
19. Total Security Incidents per year handled outside the SIR Solution?
20. Approximate percentage of Incidents that are found to be False Positives after a resource begins their analysis
21. Average effort time expended on a False Positive Incident (hours)
22. Approximate percentage of Incidents that are deemed to be repeats (saw it before, resolved it before, shows up in clusters,…)
23. Approximate percentage of Incidents that inadvertently are missed/dropped/lost

SecOps Vulnerability Review

1. How many days does a HIGH RISK Vulnerability exist within your environment?
2. On average how many times does the same vulnerability get scanned and/or identified before it is remediated?
3. Once Identified how many resources manage a vulnerability until it is remediated?
4. Once remediated is there an automated process or report to support an Audit or Compliance need?
5. Number of times per year a vulnerability scan is run on the same device or application
6. Approximate number of Vulnerabilities managed per month
7. Average number of SecOps effort hours to manage a Vulnerability to completion
8. Average number of non-SecOps effort hours to remediate a Vulnerability
9. Total Vulnerabilities per year handled by the Vulnerability Management Process
10. Total Vulnerabilities per year deemed to warrant remediation action
11. Average effort time by SecOps resources to remediate a vulnerability (hours)
12. Percentage of Vulnerabilities remediated by SecOps resources
13. Average effort time by non-SecOps resources to remediate a vulnerability (hours)
14. Are there agreements or commitments in place for requested remediations to occur within a specific timeframe? (Y/N)
15. Percentage of times when these agreements/commitments are met
16. Approximate percentage of Vulnerabilities that are found to be False Positives after a resource begins their analysis
17. Average effort time expended on a False Positive Vulnerability (hours)
18. Approximate percentage of detected Vulnerabilities that are remediated automatically today
19. Is there an automated patching process in use? (Y/N)
20. Approximate percentage of Vulnerabilities that are deemed to be repeats
21. Average elapsed time between vulnerability scans of the same device/component (days)
22. Average elapsed time from Vulnerability detection to first scan where the same vulnerability is not detected (days) by SecOps resources
23. Is SecOps involved in Application - Server - Database hardening sign-off before promotion to Production? (Y/N)

Conclusion

Leveraging questionnaires and analyzing the gap between expected and actual outcomes are invaluable tools for enhancing your overall cybersecurity enterprise program. Questionnaires provide a structured approach to evaluating current processes, uncovering vulnerabilities, and aligning team efforts with organizational goals. Meanwhile, comparing expectations to reality offers actionable insights into what’s working and where adjustments are needed. Together, these practices foster a culture of continuous improvement, empowering your team to stay ahead of evolving threats and adapt to emerging challenges. By embracing these strategies, organizations can build more resilient, efficient, and proactive cybersecurity programs that safeguard their most critical assets.

Happy New Year to everyone!

-P