Be Careful with OJVM or Bundle Patches

I would like to share my experience with the OJVM patch and how I identified and resolved errors on 19c container databases (CDB/PDB).

RDBMS 19.3 Upgrade to 19.24

Using Bug 36582781 - Database July 2024 Release Update (DB RU) 19.24.0.0.240716 + OJVM Release Update 19.24.0.0.240716 Patch 36414915 (or Combo Patch 36522439).

Patch Application: 36582781 (19.24) ::

Successfully applied on a two-node cluster with one CDB and one PDB.

[oracle@host ~]$ $ORACLE_HOME/OPatch/opatch lspatches

36582781;Database Release Update : 19.24.0.0.240716 (36582781)

36414915;OJVM RELEASE UPDATE: 19.24.0.0.240716 (36414915)

29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)

OPatch succeeded.

Encountered Warning ::

After applying the patch, the PDB showed a warning: "PDB altered with errors."

The alert log indicated a UJS version mismatch and 81,218 invalid objects were found.

SQL> alter pluggable database <pdb_database_name> open;

Warning: PDB altered with errors.

SQL> select count(*) from cdb_objects where status='INVALID';

-- 81218

Alert log snippet ::

2024-08-29T21:41:02.448755+07:00

## jox_ujs_status: op_instance_patched: UJS active in root, ujs state present, its version does not match executable version, returning FALSE in pid 312260

2024-08-29T21:41:07.568764+07:00

[oracle@host ~]$ cat /mount/app/oracle/diag/rdbms/<db_sid>/<db_sid>_1/trace/alert_<db_sid>_1.log | grep -i "jox_ujs_status" | wc -l

1870

Troubleshooting ::

Identified the issue as a Java compatibility problem.

Ran catcon.pl with utlrp.sql, which initially seemed to resolve the issue, but the PDB remained unstable.

Encountered a new error: ORA-00600 internal error.

[oracle@admin]$ $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catcon.pl --n 1 --e --b utlrp --d '''.''' utlrp.sql

catcon::set_log_file_base_path: ALL catcon-related output will be written to [/mount/app/oracle/product/19.0.0.0/dbhome_1/rdbms/admin/utlrp_catcon_119014.lst]

catcon::set_log_file_base_path: catcon: See [/mount/app/oracle/product/19.0.0.0/dbhome_1/rdbms/admin/utlrp*.log] files for output generated by scripts

catcon::set_log_file_base_path: catcon: See [/mount/app/oracle/product/19.0.0.0/dbhome_1/rdbms/admin/utlrp_*.lst] files for spool files, if any

catcon.pl: completed successfully

You have new mail in /var/spool/mail/oracle

[oracle@host admin]$

Despite the successful completion, errors persisted in the alert logs:

ORA-00600: internal error code, arguments: [kqlmlktype:not-KQLM-id], [VR], [], [], [], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [ksipgetctxia:type], [4294967295], [], [], [], [], [], [], [], [], [], []

[oracle@host ~]$ cat /mount/app/oracle/diag/rdbms/<db_sid>/<db_sid>_1/trace/alert_<db_sid>_1.log | grep "ksipgetctxia:type" | wc -l

340

[oracle@host ~]$ cat /mount/app/oracle/diag/rdbms/<db_sid>/<db_sid>_1/trace/alert_<db_sid>_1.log | grep "kqlmlktype:not-KQLM-id" | wc -l

1433

SQL> alter pluggable database <pdb_database_name> open;

Warning: PDB altered with errors.

Rollback ::

Instead of spending more time troubleshooting the ORA-00600 error, I decided to roll back the OJVM patch (36414915) and ran ./datapatch -verbose, which resolved the problems.

[oracle@host2 ~]$ $ORACLE_HOME/OPatch/opatch rollback -id 36414915 -- Need to perform on both nodes.

Oracle Interim Patch Installer version 12.2.0.1.43

Copyright (c) 2024, Oracle Corporation. All rights reserved.

...

OPatch succeeded.

[oracle@host2 ~]$

Datapatch Execution :: -Only run on one node

[oracle@host2 ~]$ $ORACLE_HOME/OPatch/datapatch -verbose

SQL Patching tool version 19.24.0.0.0 Production on Thu Aug 29 22:13:42 2024

...

For the following PDBs: CDB$ROOT

The following interim patches will be rolled back:

36414915 (OJVM RELEASE UPDATE: 19.24.0.0.240716 (36414915))

...

Patch installation complete. Total patches installed: 5

Validating logfiles...done

Patch 36582781 apply (pdb MYPDBNAME): SUCCESS

SQL Patching tool complete on Thu Aug 29 22:19:52 2024

You have new mail in /var/spool/mail/oracle

[oracle@host2 ~]$

Verification ::

SQL> select PATCH_ID, INSTALL_ID, PATCH_TYPE, ACTION, STATUS from dba_registry_sqlpatch;

PATCH_ID INSTALL_ID PATCH_TYPE ACTION STATUS

---------- ---------- ---------- --------- -------------------------

29517242 1 RU APPLY SUCCESS

36582781 2 RU APPLY SUCCESS

36414915 3 INTERIM ROLLBACK SUCCESS

SQL> alter pluggable database MYPDBNAME close immediate;

Pluggable database altered.

SQL> alter pluggable database MYPDBNAME open;

Pluggable database altered.

SQL> select count(*) from cdb_objects where status='INVALID'; -- invalids reduced to 32

---32

Final Note ::

I stopped and started the database and PDBs several times as so many sites suggested and meta link references , but it did not resolve the issues until the rollback was performed and datapatch to sync pdb's.