DB Connections and MBOSets – Understanding which piece of code may have kept an mboset open for a long period of time (#mbosetMgtPractices)

Most MBOSets are opened and closed within 60 seconds. If a connection is open for longer than it can be an indication that the related code is leaking the MBOSet.

When an MBOSet is opened a request is made to the DB Connection Manager. The DB Connection Manager stores details about the request. The DB connection manager will then provide a DB connection that the MBOSet code will use to perform its operations e.g. retrieve data.

The DB connection manage maintains a pool of database connections and can provide details about the pool usage. The size of the pool is limited by non-Maximo components e.g. Network ports and the number of DB connections that the database will support.

The DBConnection manager watchdog can be configured to log details of DB connections that are open for long periods of time.

These log entries can be configured by setting the dbconnection logger to INFO. I discuss this logger in this article - https://www.dhirubhai.net/pulse/monitoring-number-db-connections-maximo-uses-mark-robbins/

If the a database connection is open for a long period of time then an entry like this will be written to the SystemOut.log file

[01/06/13 23:15:07:154 BST] 00000059 SystemOut????O 01 Jun 2013 23:15:07:154 [INFO] [MXServer] [] DbConnectionWatchDog:Db Connection open:

DbConnectionWatchDog:Db Connection reference id=1136 SPID=132

Create time:1370124408535

Create stack trace:

psdi.server.DbConnectionWatchDog$ConnectionData.<init>(DbConnectionWatchDog.java:59)

psdi.server.DbConnectionWatchDog.addConnection(DbConnectionWatchDog.java:224) psdi.server.ConRef.notifyDBConnectionWatchDog(ConRef.java:146)

psdi.server.DBManager.getConnectionDetail(DBManager.java:1396)

psdi.server.DBManager.getConnection(DBManager.java:1299)

psdi.server.AppService.getDBConnection(AppService.java:557)

psdi.mbo.MboSet.getNextRecordData(MboSet.java:2973)

psdi.mbo.MboSet.fetchMbosActual(MboSet.java:2656) psdi.mbo.MboSet.fetchMbos(MboSet.java:2613)

psdi.mbo.MboSet.getMbo(MboSet.java:1895)

psdi.server.SimpleCronTask.getParameter(SimpleCronTask.java:95)

psdi.server.SimpleCronTask.getParamAsString(SimpleCronTask.java:119)

psdi.server.SimpleCronTask.getParamAsBoolean(SimpleCronTask.java:132)

psdi.iface.jms.JMSQueueCronTask.isOKToRun(JMSQueueCronTask.java:315)

psdi.server.CronTaskManager.updateTimestampForServer(CronTaskManager.java:885)

psdi.server.CronTaskManager$CronIncrementThread.run(CronTaskManager.java:2999)

Life??time:498619 ms

If a piece of code is executed a large number of times then this stack trace will be repeated each time that the MBOSet is left open for a long period of time. Seemingly small insignificant pieces of code can cause significant problems – later articles will demonstrate this

How often are these log messages written?

These entries are normally written every 30 minutes but 7.6 introduced new code to generate these reports when a JVM is shutting down. This shutdown report is very useful when debugging problems in non-production environments.

Experienced support staff can use this log message and the MBO count log message to identify the code that is opening the MBOSet.

Why are these leaks important?

It is important for system administrators to investigate these warnings.

Resolving these kinds of leaks can improve a system's performance in two key areas:

• Reduced memory requirement in the JVM for MBOs and MBOSets
• Reduced requirement for DB connections and cursors

I have worked on numerous cases where leaks have directly contributed to JVM crashes. Resolving the leaks reduced the memory usage and meant that the JVM became stable.

How is this related to garbage collection?

The application server (Websphere / Weblogic) garbage collection policy controls how the JVM frees up memory. The garbage collector will free up memory when there are no references to it. If an MBOSet is not closed then it cannot be processed by the garbage collector – this is important because system administrators could spend large amounts of time trying to tune the garbage collection policy without success. I discuss memory management in these articles https://www.dhirubhai.net/pulse/understanding-how-much-memory-jvm-needs-mini-series-mark-robbins/ / https://www.dhirubhai.net/pulse/heap-management-arguments-explainbestpractices-mark-robbins/

?

Vetasi Advice

Vetasi's tools are able to monitor the DB connections much more frequently than every 30 minutes - the increased frequency means that he Vetasi tools can gather other information about the JVM when the problem occurs e.g. what the JVM was doing at the time and who was logged into it.

Vetasi support staff are trained to identify and investigate these types of leaks. Vetasi have automated processes that analyse the DB connection warnings and generates a report so system administrators can understand how often the problem is occurring.

The report identifies:

• Java class that is most likely to have leaked the DB connection – this will include a reference to the automation script engine if an automation script opened the MBOSet
• Information about how many times this stack trace is written
• Information about how long the connection was open

?Once the code has been identified then a PMR can be raised if it was supplied by IBM.

Vetasi's report has helped identify many leaks and has led to numerous fixes.

Vetasi Support offers consultancy to identify and resolve DB connection leaks. If you believe that your system could be suffering from memory leaks then contact me - I can discuss this with you and I maybe able to give you a sample of our leak analysis.

In a later article I will discuss some of the rules about when the mboset will be closed and why an mboset may not be closed when you expect.

This blog series

This article is one of a series of articles to help system administrators understand the Maximo logs and the underlying architecture.

If you like this article then please share or like it.

Whilst I support the wider Maximo community and encourage the spread of knowledge, when republishing content from this blog please include the originating author along with the article or parts of.

If people do find parts of this blog coming up in blogs/newsletters/communications then please contact me directly. I’m happy to connect on LinkedIn to discuss.

Disclaimer

The postings on this blog are my own and don't necessarily represent Vetasi's positions, strategies or opinions.

The materials on this site are provided "AS IS" and the author will not be liable for any direct, indirect or incidental damages arising out or relating to any use or distribution of them. Readers are advised to test any changes/recommendations thoroughly before use