Gen-AI & SRE in focus of the C-suite agenda - learnings and recommendations.

The SRE Report 2024 speaks of state of Site Reliability Engineering

The SRE Report 2024 | Catchpoint

• 24% of organizations have breached a contractual service level agreement in the last 12 months. recommends relationship between IT objectives and business outcomes
• 64% agree that monitoring productivity or experience-disruption endpoints is required: showcases paradigm shift in how reliability practitioners now think about visibility
• 81% of organizations have two or more types of telemetry feeding their observability frameworks and 43% have four or more: confronts the fallacy of single tools being able to provide visibility for all stacks

Here are few insights from the report & correlation to my observations .

Working with few corporations ( financial ) over the years I see that C suite focusing more on deriving benefits from Gen AI & apply site reliability engineering practices but have a few fallacies that need to be first corrected.

Recommendation # 1 : ( SRE as a role and practice across SDLC ) The SRE practices go beyond just "optimizing operations" , these focus on optimizing system design. This is a common fallacy - focus on "myopic" objectives of post production operations . Get it right SRE ( as a science ) and its focus must germinate right from requirements . This is part of mindset shift .

Bring SRE embedded to troubleshoot issues inside the product team supported with Product mgmt. and Platform engineering as common .

SRE COE / chapter can be a horizontal but needs dedicated role in the feature teams .

Recommendation # 2 ( Identify system flow and customer journeys ) : System design is also about structures of teams . Teams should embed SRE and set up around flow of value . Hence adopt product centricity with empathy mapping & customer journeys to understand the existing flow , what are the impediments / blockers to customer in the touchpoints and where is efficiency losses on the way due to systemic issues on UI/UX design , performance irritants , etc..

Recommendation # 3 ( focus on business critical journey + loads of instrumentation ) Talking of flow - identify business critical journeys and add more instrumentation on the way behind the scenes . This telemetry of closely connected tools would yield insights to detect system anomalies that can be isolated and hence consume less of error budget. The platform engineering team helps sets up these tools to help SRE localization of issues.

Recommendation # 4 ( mindset ) Increasingly , it is felt the best design is when SREs are involved right from the beginning ( design , architecture ) and NFRs - security , performance , scalability and Infra capacity planning , all requirements & activities needs thinking much before code is written and tested.

Organizations to inculcate the mindset of SRE with below shifts

Recommendation # 5 ( Bring in AI )

SRE focuses on delivery and the stability of the production environment, while DevOps focuses on the end-to-end application lifecycle .If Devops was considered as a class , SRE implements that class. SRE is now going beyond using AIOPS with potential of machine learning and alert-correlation technologies.

With a growing digital footprint , we have volume of data in events , logs and alerts. SRE teams should increase application of use cases in AI to create intelligent IT operations as ML models reliably detect patterns and build insight from past experience. AI and automation applied to operations, AIOps, help teams manage the vast volumes of data and achieve proactive incident resolution.

AI is to relieve the manual toil associated in SRE role and free teams to focus on high-value work.

Recommendation # 6 ( Connected Platforms binding instrumentation data )

Companies are using variety of tools for same purpose . There is variety , velocity and volume ( all 3 Vs ) of data that needs to be ingested and understood as one related view.

Thus , isolated data within dashboards of respective tools may not lead to conclusive evidence on problem , incident management .

What we need ? We need to tie up data from all tools and feed into 1 platform ( that ingests , processes and parses the data - structured , unstructured ) and create views out of this that help identify key KPIs , metrics live. This can act as a single pane of view for various personas - SRE engineering , Cxo , testing lead , development lead , operations head etc..

• Collect the ever-increasing volumes of operations data.
• Diagnose incident causes & Intelligently co relate significant events and patterns based on real-time analysis and tie to problems.
• Apply data feeds from variety of tools , aggregate the flow of data & present onto 1 common platform that has AI and ML insights

Recommendation # 7 ( apply SLO-SLI-Error budget )

Set up define & measures for SLO , SLI and error budget for your organization.

We can only improve if we set up a business objective that is aspirational but at the same time we must have defined SLA that is realistic . In between this zone is tolerance with "how quick we can respond". Error budget ( measured in time for non-availability ) gives that zone of being prepared . Applying learning from incidents & right problem management via live observability platform.

We can thus balance - speed of feature delivery ( business need ) and stability , by accounting risk to still be down in case of errors and confidence to resolve . Thus being good at SRE practice is to have balance - being observable & being reliable ( with high availability , higher scalability , higher performance , flexi capacity , high security )