MBOSet lifecycle and why closing them is important (#mbosetMgtPractices)

MBOSets are a critical part of Maximo and if they are not managed properly then the Maximo JVM can become unstable.

?

When an MBOSet is opened and used then various steps occur including:

• Memory is set aside to store details of the query and associated information e.g. metadata
• Store records that have been retrieved from the database or are being worked on
• Set aside database related resources e.g. database cursor/connection

?These resources are not significant when there is a single MBO in memory but they can become substantial when large numbers of MBOs are loaded into memory.

?The normal life of an MBO can be summarised using the following picture.

?If an MBOSet is not managed properly then consequences can include:

• Higher memory usage with an increased risk of the JVM crashing
• DB Connections left open unnecessarily and using resources on the database, ultimately this can lead to the database running out of resources

?One of the most common leaks occurs when a variable is used to open an MBOSet but the variable is reused without closing the MBOSet.

This diagram shows the problem

Is it still a problem when the MBOSet gets closed eventually e.g. after several minutes?

I have seen cases where an operation processes large numbers of records and processing an individual record causes an MBOSet to be leaked.

In a small test with a single record the record could be processed and the test is finished before the leak is reported in the DB Connection watchdog leak warning log entry

When that operation involves 50,000 records then the process can take much longer and the leak can exhaust the available memory.

Each user may only use a few MBOs but the problem occurs when there are a lot of users using MBOs.

If large numbers of users perform the same operation at the same time then that could cause a sudden build up of MBOs.

?

If JVM1 has high memory then could that cause JVM2 to crash?

Not directly.

The JVMs form a mesh to communicate with each other but the memory is not shared between the JVMs. I discuss the mesh in this article - https://www.dhirubhai.net/pulse/jvms-joining-leaving-websphere-mesh-dcsv1036w-mark-robbins

However if a user performs an operation that causes a JVM to run out of memory then the user is liable to login to another JVM and repeat the operation often with the same result. This often gives the impression that JVM crashes are related.

I have seen this occur when an external tool was being used to load data into Maximo, at the time the tool suffered from memory leaks. The user would use the tool to load the data but the free memory would be low and the JVM would run out of memory and crash when it was used. If the JVM ran out of memory while the dataload was occurring then the user would not receive any error and they would often try the operation again, the second request would be passed to another JVM that would also crash.

Vetasi advice

It is important to monitor the MBO/MBOSet counts to understand when MBOs are being loaded and then they are released.

?Vetasi's software monitors these counts and can warn if they are getting high. The software can also review other indicators and highlight MBOs that could be contributing to the problems.

Vetasi support train support teams on issues like this and share a subset of those materials in this blog. Contact me if you want to improve your debugging skills and reduce the time required to resolve memory leaks.

This blog series

This article is one of a series of articles to help system administrators understand the Maximo logs and the underlying architecture.

If you like this article then please share or like it.

Whilst I support the wider Maximo community and encourage the spread of knowledge, when republishing content from this blog please include the originating author along with the article or parts of.

If people do find parts of this blog coming up in blogs/newsletters/communications then please contact me directly. I’m happy to connect on LinkedIn to discuss.

Disclaimer

The postings on this blog are my own and don't necessarily represent Vetasi's positions, strategies or opinions.

The materials on this site are provided "AS IS" and the author will not be liable for any direct, indirect or incidental damages arising out or relating to any use or distribution of them. Readers are advised to test any changes/recommendations thoroughly before use