Getting the most value out of Splunk ITSI and SAP

What exactly do we mean by SAP?

For starters let’s get the terminology out of the way as SAP is a very general term. SAP is the software company that makes the SAP business suite. The SAP business suite is comprised of five modular applications designed to perform essential end-to-end business processes across all industries. SAP ERP (ERP Central Component) is the most popular module out of the five. So, whenever you hear someone say “Yeah, we have SAP at work” that is a very general statement. They are most likely referring to the SAP ERP (Enterprise Resource Planning) module. The other modules are CRM, SRM, SCM and PLM. These modules are all tied together by SAP’s platform called NetWeaver on its latest version (SAP S4/HANA) and we’ll reference that version here.

How do we get SAP data?

Until recently, extracting and using SAP data was a tedious and manual endeavor as most of the data in SAP ERP is plain impenetrable without any sort of custom code development. You’ll find that that a lot of the data lacks timestamps and comes with all sorts of quirks because of its technical legacy dating back to the early 70’s. All of these last-century idiosyncrasies are taken care of when you have the PowerConnect app on your SAP servers sending data over to Splunk. PowerConnect is the technology that’s vetted by both SAP and Splunk that will enable you to get parsed and timestamped SAP data into your Splunk ITSI environment ASAP. It will also give you the flexibility to choose specific data sources and polling periods. Take a look at this video for a more detailed view into the PowerConnect architecture.

PowerConnect Architecture

Decomposing the top infrastructure services in SAP ERP environment into ITSI

There are 2 or 3 high level aspects to model – depending on which ABAP admin you ask. Either you’ll encounter the opinion that says the SAP ERP S4/HANA is comprised of the NetWeaver and HANA or the one that says it’s comprised of Application Servers, Central Services and HANA:

Modeling SAP: First approach vs Second

There is no single way to model this. The model must be useful to the client. With SAP ERP, you’re going to find that a lot of your best practices do not map out cleanly to expectations, so you’ll have to work with the customer to deliver a balanced approach.

Best practice: The customer is always right – but know when to nudge.

Keep in mind you are not here to tell the customer how to conceptualize their architecture. You’re here to conceptualize the customer’s architecture in ITSI while following best practices. Balance each approach on the administrative overhead it will cost and the ease of visualization it provides and then decide with the customer.

Now that you’ve agreed on a model for the SAP ERP infrastructure, tie it to its peers - other SAP products, other platforms like warehouse management systems, HR systems that SAP ERP interacts with. Finally, as with any ITSI implementation make sure you use some time in the decomposition to identify the top-level business that SAP ERP supports. You might not have data on many of these very high-level elements but at the very least you want to call them out in your service tree to present a full end-to-end story.

SAP core infrastructure monitoring

Score some quick wins by getting the basic monitoring out of the way by putting accessible KPI’s into your service model. Here we’re referring to HANA, the central services and App server processes executing on the OS. Monitor process execution along with the mount points that need to be monitored and of course, memory and CPU usage. This will give you something to show the stakeholders to give them an idea of what to expect. You have the option of getting this data from the forwarders or use what SAP is reporting via PowerConnect, but my preference is having a forwarder installed and using it for the low level data.

SAP Platform Metrics

Don’t reinvent the wheel. Follow up with platform metrics available in the data but look at the PowerConnect dashboards. They offer a wealth of categorized metrics that can be quickly be turned into ITSI KPI’s in minutes.

Power Connect: Landscape Overview

Make Friends: You’ll find that basis admins are interested in Transaction performance, Error dumps, STAD metrics and work process utilization. All of these are available in the PowerConnect dashboards for you to convert into ITSI KPI’s.

Get the business transaction data

The transactions you’ll turn into ITSI KPI’s will vary depending on what the business uses SAP for. The data you end up using might be from sales, logistics, B2B, B2C, etc. Start securing access to this data at the start of the project for it to be ready by the time you’re ready to work on it. You’ll need to provide a compelling case on why this data is important as your usual champions (the basis admin team) will have a difficult time understanding why they should be fulling use cases outside of their core responsibilities. Remind them that looking at an SAP system through multiple viewpoints will maximize their detection capabilities. This is the data that you’ll want to model through glass tables, giving the customer a real time view into their their business.

Best practices

I know that was a lot to take in so keep in mind these best practices I have found useful when splunking SAP data into ITSI:

Best Practice: Find the business and technical champions.

You will need two types of champions, a business champion and a technical champion. Identify them ASAP as executing the project without either will be an uphill battle every time you need data outside a champion’s scope. A technical champion will have a hard time seeing why you need business data and vice versa.

Best Practice: Move fast, demo often

Don’t wait until the end of a sprint or a set of tasks to show off your work. Demo it as soon as you have something tangible to show and get feedback. The momentum will help your champions strengthen their case for ITSI and will give them an opportunity to do small course corrections if needed.

Best Practice: Keep the basis admins engaged.

Understanding SAP is difficult. It’s been around since the early 70’s and carries many anachronistic caveats that only start to make sense once you’ve been working on it for years. You will need an engaged basis admin to spend time with you to explain why they need things represented a certain way that might go against your gut instinct.

Best Practice: Don’t lose sight of where your customers perspective is.

As you move forward, make sure you look back to confirm that your champions are in line with the direction you’re taking.

Best Practice: Be consistent

As you branch out to dependencies and sub dependencies be consistent on how you separate and layer services & KPI’s. You’ll want services that are in the same category to show up in the same tier in the service analyzer. You’ll need to make difficult decision that balance how tidy your service analyzer is while providing rapid troubleshooting capabilities with the least amount of clicking. In consulting, the main approach I use to categorizing is the MECE framework. Start by building services that are Mutually Exclusive (they don’t overlap) but Collectively Exhaustive (there cover everything).

Learn from others

3M held a session at .Conf 19 titled “How 3M is Transforming SAP ERP Operations through AIOPS” where you can see best practices in effect. Their approach earned them “The Commander Award” in the Splunk Revolution Awards at .Conf19.

Familiarize yourself with these key concepts:

Time Stamped Data

Native data in SAP ERP is in a binary, non time stamped format. Of course, time stamped data is essential when correlating events from the different SAP ERP transactions. PowerConnect is the SAP addon that will allow you to export time stamped data from SAP ERP into Splunk so that you can search on it and link events together to find root causes much quicker.

SAP Transactions Codes

A Transaction Code (T-Code) is a four-character short cut used to execute a specific transaction inside SAP ERP. For example, if a user needed to create a sales order, they would need to enter the VA01 T-Code within the SAP GUI Client to access the Create Sales Order screen. A T-Code is not the same thing as log file, but it is what you will use as the source type when looking for SAP ERP data in Splunk. So, for example when looking for SAP ERP user login data in Splunk, your query would look something like this: “index=SAP sourcetype=AL08”, where AL08 is the T-Code you would use in SAP ERP to list all users logged on.

Start with these transaction types:

There are about 120,000 T-Codes in the SAP Business Suite, each assigned to a specific task or process in SAP. These T-Codes will be your short cut to some common use cases:

ST06 - Hardware Performance. Allows us to report on all infrastructure beneath any running SAP instance (SAP, Mem, Disk, Network, etc)

SM37 - Batch Schedule: Report on failed jobs, number of executed jobs and runtime of existing jobs.

DB02 - Selected data from DB02: Key database metrics

SM12 - SAP Locks: Show current depth and contents of the lock table

SM51 - SAP Instance data and logs: SAP Work process utilization details.

ST22 – ABAP Short dumps: ABAP dump statistics and kpi’s

WE02 – IDOCS and Messages – Report on the status of IDOCS coming into the system.

Make the SAP Basis team your partner

Walk closely inline with the needs of the SAP Basis team as they will be your main point of contact when unraveling the complexity of an SAP ERP system and modeling it in SAP. They are also a big deal at the end of the day given that the revenue form your organization flows through SAP, Payroll usually happens with SAP and your customers data in locked up in SAP. Bottom line 77% of the worlds transactions happen in SAP and orchestrating out visibility is not only pretty killer to do with ITSI our favorite Splunk premium solution but is well received by your new friends over on the SAP team.

Pro- Move

Build your executive sponsor a glass table with a view of the business they care about. Our favorite? Some iteration of order to cash or supply chain. This is where ITSI shines and the dashboards and visuals for the executive sponsor are fast to build and put you in line for that conversation you've been waiting to have around a raise.

Good luck building out your SAP views in SAP and ping us at [email protected] if you build something amazing you can share.

About the authors: Roman Lopez is a Lead Consultant at RHONDOS and known in the ITSI world as El Fuego - Brant Hubbard is the CEO & Founder of RHONDOS.