(Part E) Multinode/Multiuser Installation of Oracle EBS(12.2.0) & Fresh DB(12.1.0.2) using StartCD 51 And Upgrade to 12.2.10 and 19C [Linux x86_64]

1. Apply upgrade 12.2.10 patch (30399999)

1.1 downloaded upgrade patch in /software?

[applmgr@ebsdemo01 software]$ ls *30399999*

p30399999_R12_LINUX_10of11.zip?p30399999_R12_LINUX_1of11.zip?p30399999_R12_LINUX_3of11.zip?p30399999_R12_LINUX_5of11.zip?p30399999_R12_LINUX_7of11.zip?p30399999_R12_LINUX_9of11.zip

p30399999_R12_LINUX_11of11.zip?p30399999_R12_LINUX_2of11.zip?p30399999_R12_LINUX_4of11.zip?p30399999_R12_LINUX_6of11.zip?p30399999_R12_LINUX_8of11.zip

1.2 unzip all 11 patches in Patch Top

unzip -q p30399999_R12_LINUX_1of11.zip -d /u01/apps/CWI/fs_ne/EBSapps/patch/

unzip -q p30399999_R12_LINUX_2of11.zip -d /u01/apps/CWI/fs_ne/EBSapps/patch/

unzip -q p30399999_R12_LINUX_3of11.zip -d /u01/apps/CWI/fs_ne/EBSapps/patch/

unzip -q p30399999_R12_LINUX_4of11.zip -d /u01/apps/CWI/fs_ne/EBSapps/patch/

unzip -q p30399999_R12_LINUX_5of11.zip -d /u01/apps/CWI/fs_ne/EBSapps/patch/

unzip -q p30399999_R12_LINUX_6of11.zip -d /u01/apps/CWI/fs_ne/EBSapps/patch/

unzip -q p30399999_R12_LINUX_7of11.zip -d /u01/apps/CWI/fs_ne/EBSapps/patch/

unzip -q p30399999_R12_LINUX_8of11.zip -d /u01/apps/CWI/fs_ne/EBSapps/patch/

unzip -q p30399999_R12_LINUX_9of11.zip -d /u01/apps/CWI/fs_ne/EBSapps/patch/

unzip -q p30399999_R12_LINUX_10of11.zip -d /u01/apps/CWI/fs_ne/EBSapps/patch/

unzip -q p30399999_R12_LINUX_11of11.zip -d /u01/apps/CWI/fs_ne/EBSapps/patch/

Note: 12.2.10 Patch is applied in downtime mode.?

1.3 Bring Down MT Services (for our case we will bring down admin server and node manager as these are running only)

./adadminsrvctl.sh stop

./adnodemgrctl.sh stop

1.4 IMPORTANT POINTS - Have already tested this patch, hence aware of some known scenarios of failures, hence taking care of these in advance, you may try patch first and then troubleshoot accordingly...?

?a. Temp Tablespace - Extended by 10GB - taken Care in Part C?

b. APPS_TS_TX_DATA Tablespace - Extended by 25GB - taken care in Part C

c. Create required link to Motif library (Oracle Linux 7, RHEL 7, and SLES 12 only), In our case its Linux 7.?

Perform the following command (as root on your system) to create a required link to the Motif library for Oracle Forms and Reports:

[root@ebsdemo01 ~]# cd /usr/lib

[root@ebsdemo01 lib]# ls -ltr libXm.so*

-rwxr-xr-x. 1 root root 2791388 Jun 26?2018 libXm.so.4.0.4

lrwxrwxrwx. 1 root root???14 Oct 12 13:13 libXm.so.4 -> libXm.so.4.0.4

lrwxrwxrwx. 1 root root???14 Oct 12 13:13 libXm.so -> libXm.so.4.0.4

[root@ebsdemo01 lib]# ln -s libXm.so.4.0.4 libXm.so.2

[root@ebsdemo01 lib]# ls -ltr libXm.so*

-rwxr-xr-x. 1 root root 2791388 Jun 26?2018 libXm.so.4.0.4

lrwxrwxrwx. 1 root root???14 Oct 12 13:13 libXm.so.4 -> libXm.so.4.0.4

lrwxrwxrwx. 1 root root???14 Oct 12 13:13 libXm.so -> libXm.so.4.0.4

lrwxrwxrwx. 1 root root???14 Oct 16 13:22 libXm.so.2 -> libXm.so.4.0.4

(Repeat this step for DB Tier also)?

Ref. Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.2) for Linux x86-64 (Doc ID 1330701.1)

d. Using wait_on_failed_job=yes and flags=autoskip in Adop command?

Reason: Patch Fails and adctrl utility Will Not Show Any Workers To Resume Failed Jobs (Doc ID 2756299.1)

Using wait_on_failed_job=yes will allow customers to manipulate failed jobs via the adctrl utility **ONLY** if the failed job is a PL/SQL command. However, Other generate actions such as form or report generation commands will not allow customers to handle jobs through adctrl when the above parameter is set to yes then Use flags=autoskip to skip Error while generating Oracle Form or report Objects. Later review autoskip.log to fix manually the form/report errors.?

Ref. Patch Fails and adctrl utility Will Not Show Any Workers To Resume Failed Jobs (Doc ID 2756299.1)

- Note: When you run patch in downtime mode i.e. option apply_downtime (as done for 12.2.10), patch will directly be applied on RUN edition. You run clean up & clone_fs to clone from RUN Edition to Patch Edition.?

e. Open another Session and run command 'adopmon' to monitor the patch Events

?1.5 Submit below command to apply patch?

adop phase=apply apply_mode=downtime patches=30399999 wait_on_failed_job=yes workers=4 flags=autoskip

1.6 Events from adopmon screen

2021/10/16 13:42:15 ebsdemo01??????EVENT????Application tier services are down.

2021/10/16 13:42:15 ebsdemo01??????EVENT????Continuing with the existing session [Session ID: 2].

2021/10/16 13:42:15 ebsdemo01??????EVENT????ADOP (C.Delta.12)

2021/10/16 13:42:15 ebsdemo01??????EVENT????Session ID: 2

2021/10/16 13:42:15 ebsdemo01??????EVENT????Node: ebsdemo01

2021/10/16 13:42:15 ebsdemo01??????EVENT????Phase: apply

2021/10/16 13:42:15 ebsdemo01??????EVENT????Log: /u01/apps/CWI/fs_ne/EBSapps/log/adop/2/20211016_134149/adop.log

2021/10/16 13:42:16 ebsdemo01??????EVENT????Applying patch 30399999.

2021/10/16 13:42:17 ebsdemo01??????EVENT????Log: /u01/apps/CWI/fs_ne/EBSapps/log/adop/2/20211016_134149/apply/ebsd

???????????????????????????emo01/30399999/log/u30399999.log

..

2021/10/16 16:05:39 ebsdemo01??????EVENT????Output: /u01/apps/CWI/fs_ne/EBSapps/log/adop/2/20211016_134149/apply/e

???????????????????????????bsdemo01/adzdshowlog.out

2021/10/16 16:05:44 ebsdemo01??????EVENT????The apply phase completed successfully.

COMPLETED IN around 4 Hours?

1.7 New Environment File : . EBSapps.env placed under /u01/apps/CWI/

?E-Business Suite Environment Information

?----------------------------------------

?RUN File System??????: /u01/apps/CWI/fs1/EBSapps/appl

?PATCH File System?????: /u01/apps/CWI/fs2/EBSapps/appl

?Non-Editioned File System : /u01/apps/CWI/fs_ne

?DB Host: dbdemo01.<DomainName>?Service/SID: CWI

?E-Business Suite Environment Setting

?------------------------------------

?- Enter [R/r] for sourcing Run File System Environment file, or

?- Enter [P/p] for sourcing Patch File System Environment file, or

?- Enter anything else to exit

?Please choose the environment file you wish to source [R/P]:R

?Sourcing the RUN File System ...

1.8 Start Admin Server ./adadminsrvctl.sh start

1.9 CLEAN UP PHASE

adop phase=cleanup cleanup_mode=full workers=4

============================================================

ADOP (C.Delta.12)

Session ID: 2

Node: ebsdemo01

Phase: cleanup

Log: /u01/apps/CWI/fs_ne/EBSapps/log/adop/2/20211016_173505/adop.log

==============================================================

Processing cleanup actions in parallel.

??Log: /u01/apps/CWI/fs_ne/EBSapps/log/adop/2/20211016_173505/cleanup/ebsdemo01/log/cleanup.log

The cleanup phase completed successfully.

adop exiting with status = 0 (Success)

=> adop -status

Enter the APPS password:

Connected.

====================================================

ADOP (C.Delta.12)

Session Id: 2

Command: status

Output: /u01/apps/CWI/fs_ne/EBSapps/log/adop/2/20211016_195054/adzdshowstatus.out

===============================================================

Node Name????Node Type?Phase??????Status?????Started???????Finished???????Elapsed

--------------- ---------- --------------- --------------- -------------------- -------------------- ------------

ebsdemo01????master???APPLY??????COMPLETED????2021/10/16 12:06:06?2021/10/16 16:05:39?3:59:33

??????????????CLEANUP?????COMPLETED????2021/10/16 17:35:32?2021/10/16 18:36:51?1:01:19

1.10 Bring down Admin server and node manager.?

2. Follow Note 2555355.1: OPMN Issue (FMW Patch requirement) [you can find reference for this in Part D]

In this section there are just commands which we need to run.?

2.1 As applmgr user, source the EBS and FMW environment files

. EBSapps.env

. $FMW_HOME/SetWebtier.env

2.2 Create an alias for orapki utility and create a wallet

a.?pwd

???/u01/apps/CWI

b.?[applmgr@ebsdemo01 CWI]$ alias orapki=$FMW_HOME/oracle_common/bin/orapki

c.?[applmgr@ebsdemo01 CWI]$ mkdir ~/ss

d.?[applmgr@ebsdemo01 CWI]$ cd ~/ss

e.?[applmgr@ebsdemo01 ss]$ orapki wallet create -wallet ./ -auto_login_only

Oracle PKI Tool : Version 11.1.1.9.0

Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

f.?[applmgr@ebsdemo01 ss]$ orapki wallet add -wallet . -dn "CN=FMWSmallCircleOfTrust" -asym_alg RSA -keysize 2048 -sign_alg sha256 -self_signed -validity 3652 -auto_login_only

Oracle PKI Tool : Version 11.1.1.9.0

Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

g.?[applmgr@ebsdemo01 ss]$ orapki wallet display -wallet .

Oracle PKI Tool : Version 11.1.1.9.0

Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:

User Certificates:

Subject:????CN=FMWSmallCircleOfTrust

Trusted Certificates:

Subject:????CN=FMWSmallCircleOfTrust

2.3 The wallet should now be copied to the target location

a.?tr < $CONTEXT_FILE '<>' '?' | awk '/"s_ohs_instance"/ {print $(NF-1)}'

??EBS_web_CWI_OHS1

b.?iName=$(tr < $CONTEXT_FILE '<>' '?' | awk '/"s_ohs_instance"/ {print $(NF-1)}' )

c.?[applmgr@ebsdemo01 ss]$ cd $FMW_HOME/webtier/instances/$iName

??[applmgr@ebsdemo01 EBS_web_CWI_OHS1]$

d.?[applmgr@ebsdemo01 EBS_web_CWI_OHS1]$ find . -name cwallet.sso | fgrep -v /webgate/

???./config/OPMN/opmn/wallet/cwallet.sso

???./config/OHS/EBS_web_CWI/keystores/default/cwallet.sso

???./config/OHS/EBS_web_CWI/proxy-wallet/cwallet.sso

e.?[applmgr@ebsdemo01 EBS_web_CWI_OHS1]$ find . -name cwallet.sso | fgrep -v /webgate/ | while read w ; do echo -e "\n$w"; orapki wallet display -nologo -wallet $w ; done

? ?./config/OPMN/opmn/wallet/cwallet.sso

???Requested Certificates:

???User Certificates:

???Subject:????CN=Self-Signed Certificate for EBS_web_CWI_OHS1\20,OU=OAS,O=ORACLE,L=REDWOODSHORES,ST=CA,C=US

???Trusted Certificates:

???Subject:????CN=Self-Signed Certificate for EBS_web_CWI_OHS1\20,OU=OAS,O=ORACLE,L=REDWOODSHORES,ST=CA,C=US

??./config/OHS/EBS_web_CWI/keystores/default/cwallet.sso

???Requested Certificates:

???User Certificates:

???Subject:????CN=Self-Signed Certificate for EBS_web_CWI\20,OU=OAS,O=ORACLE,L=REDWOODSHORES,ST=CA,C=US

???Trusted Certificates:

???Subject:????CN=Self-Signed Certificate for EBS_web_CWI\20,OU=OAS,O=ORACLE,L=REDWOODSHORES,ST=CA,C=US

??./config/OHS/EBS_web_CWI/proxy-wallet/cwallet.sso

???Requested Certificates:

???User Certificates:

???Subject:????CN=Self-Signed Certificate for EBS_web_CWI\20,OU=OAS,O=ORACLE,L=REDWOODSHORES,ST=CA,C=US

???Trusted Certificates:

???Subject:????CN=Self-Signed Certificate for EBS_web_CWI\20,OU=OAS,O=ORACLE,L=REDWOODSHORES,ST=CA,C=US

f.??[applmgr@ebsdemo01 EBS_web_CWI_OHS1]$ find . -name cwallet.sso | fgrep -v /webgate/ | while read w ; do echo $w; cp -p ~/ss/cwallet.sso $w ; done

??./config/OPMN/opmn/wallet/cwallet.sso

??./config/OHS/EBS_web_CWI/keystores/default/cwallet.sso

??./config/OHS/EBS_web_CWI/proxy-wallet/cwallet.sso

2.4 RE-REGISTER OPMN INSTANCE

Note: Please make sure before re-resgistering the OPMN instance, start the admin server and shut down it just after this step.

a. Obtain the "host, port, and user" variables for WLS Admin from CONTEXT_FILE?

-??[applmgr@ebsdemo01 EBS_web_CWI_OHS1]$ tr < $CONTEXT_FILE '<>' '?' | awk '/"s_wls_admin_host"/ {print $(NF-1)}'

???ebsdemo01

????aHost=$( tr < $CONTEXT_FILE '<>' '?' | awk '/"s_wls_admin_host"/ {print $(NF-1)}' )

-??[applmgr@ebsdemo01 EBS_web_CWI_OHS1]$ tr < $CONTEXT_FILE '<>' '?' | awk '/"s_wls_adminport"/ {print $(NF-1)}'

???7001

????aPort=$( tr < $CONTEXT_FILE '<>' '?' | awk '/"s_wls_adminport"/ {print $(NF-1)}' )

-??[applmgr@ebsdemo01 EBS_web_CWI_OHS1]$ tr < $CONTEXT_FILE '<>' '?' | awk '/"s_wls_admin_user"/ {print $(NF-1)}'

???weblogic

????aUser=$( tr < $CONTEXT_FILE '<>' '?' | awk '/"s_wls_admin_user"/ {print $(NF-1)}' )

b. This re-registration will ensure that the OHS instance is showing properly in Fusion Middleware EM Control.

-?cd $FMW_HOME/webtier/instances/$iName/bin

??[applmgr@ebsdemo01 bin]$ pwd

???/u01/apps/CWI/fs1/FMW_Home/webtier/instances/EBS_web_CWI_OHS1/bin

- ./opmnctl unregisterinstance -adminHost $aHost -adminPort $aPort -adminUsername $aUser -instanceName $iName

???Command requires login to weblogic admin server (ebsdemo01):

?????Username: weblogic

?????Password:

Unregistering instance

Command succeeded.

-?./opmnctl registerinstance -adminHost $aHost -adminPort $aPort -adminUsername $aUser

???Command requires login to weblogic admin server (ebsdemo01):

????Username: weblogic

????Password:

Registering instance

Command succeeded.

- Re - start Admin Server and Node Manager?

3. CLONE FILE SYSTEM?

This action will copy the new run edition code and configuration to the other file system, to ensure that both file systems are in sync before applying patches using the regular adop cycle on the other file system.

3.1 Synchronize the file systems using the command below

adop phase=fs_clone

The fs_clone phase completed successfully.

adop exiting with status = 0 (Success)

3.2 This step is required to ensure ports are mentioned in /etc/services

- Access URL from Windows Host Machine – Open Port from firewall (Linux)

- Add port entries in /etc/services

- Open Port from Firewall?

firewall-cmd --zone=public --add-port=<PortNumber>/tcp --permanent

firewall-cmd --reload

check - iptables-save | grep <port number>

3.3 Start all services?

adstrtal.sh

All enabled services for this node are started.

adstrtal.sh: Exiting with status 0

3.4 Verify EBS version?

SQL> select release_name from apps.fnd_product_groups;

RELEASE_NAME

--------------------------------------------------

12.2.10

This completes the 12.2.10 Upgrade!?

Next Part is Bonus! for JWS (Java Web Start)