Adding a New Compute Node to Bare Metal Exadata

You can add a new database server to an existing?Oracle Real Application Clusters (Oracle RAC)?cluster running on?Oracle Exadata Database Machine.

Assumptions

• New compute node has been reimaged with the same Exadata software version running on existing compute nodes.
• New compute node network has been configured (management, client, private).
• New compute node hardware is X7-2.
• Exadata Image version is 20.1.9.0.0 on all compute nodes.
• Grid Infrastructure version is 19.11.0.0.210420
• Database version is 12.2.0.1.210420
• Elastic Bare Metal Exadata system consists of 4 compute nodes and 3 cell nodes.
• A single software owner (oracle account) is in use for database and grid software.

Preparations

1- Reclaim Space on the new compute node.

This is a Bare Metal Exadata system, so it is recommended to reclaim inactive system logical volume used for DOM0 using script “/opt/oracle.SupportTools/reclaimdisks.sh“

[root@exadb05 ~]# /opt/oracle.SupportTools/reclaimdisks.sh -check
Model is ORACLE SERVER X7-2
Number of LSI controllers: 1
Physical disks found: 4 (252:0 252:1 252:2 252:3)
Logical drives found: 1
Linux logical drive: 0
RAID Level for the Linux logical drive: 5
Physical disks in the Linux logical drive: 4 (252:0 252:1 252:2 252:3)
Dedicated Hot Spares for the Linux logical drive: 0
Global Hot Spares: 0
Valid. Disks configuration: RAID5 from 4 disks with no global and dedicated hot spare disks.
Valid. Booted: Linux. Layout: Linux + DOM0.
[root@exadb05 ~]# /opt/oracle.SupportTools/reclaimdisks.sh -free -reclaim
Model is ORACLE SERVER X7-2
Number of LSI controllers: 1
Physical disks found: 4 (252:0 252:1 252:2 252:3)
Logical drives found: 1
Linux logical drive: 0
RAID Level for the Linux logical drive: 5
Physical disks in the Linux logical drive: 4 (252:0 252:1 252:2 252:3)
Dedicated Hot Spares for the Linux logical drive: 0
Global Hot Spares: 0
[INFO? ? ?] Check for Linux with inactive DOM0 system disk
[INFO? ? ?] Valid Linux with inactive DOM0 system disk is detected
[INFO? ? ?] Number of partitions on the system device /dev/sda: 3
[INFO? ? ?] Higher partition number on the system device /dev/sda: 3
[INFO? ? ?] Last sector on the system device /dev/sda: 3509760000
[INFO? ? ?] End sector of the last partition on the system device /dev/sda: 3509759966
[INFO? ? ?] Remove inactive system logical volume /dev/VGExaDb/LVDbSys3
[INFO? ? ?] Remove xen files from /boot
[INFO? ? ?] Remove ocfs2 logical volume /dev/VGExaDb/LVDbExaVMImages        

2- Copy or merge the contents of the following files using files on one of the existing compute nodes.

You may configure root ssh Passwordless Authentication from first node to new compute node (exadb05).

2.1. Copy the contents of the?/etc/security/limits.conf?file from exadb01 to exadb05

2.2. Edit /etc/hosts on all nodes and add exadb05 private network two IPs (IB network IPs). Make sure that /etc/hosts file on all compute nodes contains private IPs for all compute nodes.

--- exadb05 private IPs (IB IPs)
192.168.10.29
192.168.10.30        

2.3. Copy the?/etc/oracle/cell/network-config/cellinit.ora?file from exadb01 to exadb05 .

2.4. Update the /etc/oracle/cell/network-config/cellinit.ora file on new compute node (exadb05) with the IPs of ib0?and?ib1?interfaces.

[root@exadb05 ~]# vi /etc/oracle/cell/network-config/cellinit.ora
ipaddress1=192.168.10.29/22
ipaddress2=192.168.10.30/22        

2.5. Copy the?/etc/oracle/cell/network-config/cellip.ora?file from exadb01 to exadb05. The content of the?cellip.ora?file should be the same on all database servers. It contains the list of cell nodes private IPs (InfiniBand IPs).

[root@exadb05 ~]# cat /etc/oracle/cell/network-config/cellip.ora
cell="192.168.10.15;192.168.10.16"
cell="192.168.10.17;192.168.10.18"
cell="192.168.10.19;192.168.10.20"        

2.6. Copy the /etc/modprobe.d/exadata.conf file from exadb01 to exadb05. The contents of the configuration file should be the same on all database servers.

2.7. Copy the?/etc/sysctl.conf?file from exadb01 to exadb05. The contents of the file should be the same on all database servers.

Reload sysctl.conf variables on exadb05.

[root@exadb05 ~]# sysctl -p        

2.8. Copy the /etc/oracle/cell/network-config/cellroute.ora?file from exadb01 to exadb05.

The file defines IPs mapping between compute node' private IPs and each cell node' private IPs.

Modify the contents pf the file on the new node (exadb05) to use private IPs (the local?InfiniBand?interfaces) on exadb05.

[root@exadb05 ~]# vi /etc/oracle/cell/network-config/cellroute.ora
# Routes for 192.168.10.15;192.168.10.16
route="192.168.10.16;192.168.10.29"
route="192.168.10.15;192.168.10.30"

# Routes for 192.168.10.17;192.168.10.18
route="192.168.10.18;192.168.10.29"
route="192.168.10.17;192.168.10.30"

# Routes for 192.168.10.19;192.168.10.20
route="192.168.10.20;192.168.10.29"
route="192.168.10.19;192.168.10.30"        

3- Set up the users for the software owners on the new node (exadb05).

If you are using role-separated management, then the users are usually?oracle?and?grid. If you use a single software owner, then the user is usually?oracle.?We are using a single owner in our example.

3.1. Obtain the current group information for oracle account from a exadb01.

[root@exadb01 ~]# id oracle
uid=1001(oracle) gid=1001(oinstall) groups=1001(oinstall),1002(dba)        

3.2. Add groups on the new node (exadb05) as listed on first node.

[root@exadb05 ~]# groupadd -g 1001 oinstall
[root@exadb05 ~]# groupadd -g 1002 dba        

3.3. Add oracle account on the new node (exadb05) as listed on first node.

[root@exadb05 ~]# useradd -u 1001 -g 1001 -G 1001,1002 -m -d /home/oracle -s 
? /bin/bash oracle        

3.4. Create the Oracle Base and Grid home directories on the new node (exadb05).

[root@exadb05 ~]# mkdir -p /u01/app/oracle
[root@exadb05 ~]# mkdir -p /u01/app/19.0.0.0/grid
[root@exadb05 ~]# chown -R oracle:oinstall /u01/app        

3.5. Change the ownership on the?cellip.ora?and?cellinit.ora?files on the new node (exadb05).

[root@exadb05 ~]# chown -R oracle:oinstall /etc/oracle/cell/network-config        

3.6. Set the password for oracle on the new node (exadb05).

4- Set up SSH for the?oracle?account.

4.1. Log in to the?oracle?account on?the new node (exadb05).

4.2. Create the?dcli?group file on?the new node (exadb05). File contains the name of all compute nodes.

[root@exadb05 ~]# su - oracle
[oracle@exadb05 ~]# vi /home/oracle/dbs_group
exadb01
exadb02
exadb03
exadb04
exadb05        

4.3. Create ssh key and copy to all nodes. Run the following command on?the new node (exadb05).

[oracle@exadb05 ~]# ssh-keygen -t rsa
[oracle@exadb05 ~]# dcli -g /home/oracle/dbs_group -l oracle -k -s '-o StrictHostKeyChecking=no'
oracle@exadb01's password:
oracle@exadb02's password:
oracle@exadb03's password:
oracle@exadb04's password:
oracle@exadb05's password:
exadb01: ssh key added
exadb02: ssh key added
exadb03: ssh key added
exadb04: ssh key added
exadb05: ssh key added        

4.4. Verify oracle SSH equivalency.

[oracle@exadb05 ~]$ dcli -g dbs_group -l oracle date
exadb01: Fri Aug 27 10:02:43 EDT 2021
exadb02: Fri Aug 27 10:02:43 EDT 2021
exadb03: Fri Aug 27 10:02:43 EDT 2021
exadb04: Fri Aug 27 10:02:43 EDT 2021
exadb05: Fri Aug 27 10:02:43 EDT 2021        

Clone?Oracle Grid Infrastructure?to?the New Compute Node

Connect to the first compute node (exadb01) as oracle and run all steps below.

1- Verify the hardware and operating system installation using the cluster verification utility (cluvfy).

The phrase?Post-check for hardware and operating system setup was successful?should appear at the end of the report. If the cluster verification utility fails to validate the storage on the replacement server, you can ignore those messages.

--- Verify command
cluvfy stage -post hwos -n <new_server>,<existing_node> -verbose


[oracle@exadb01 ~]$. oraenv
[ORACLE_SID]: +ASM1
[oracle@exadb01 ~]$ cd $ORACLE_HOME/bin
[oracle@exadb01 ~]$ ./cluvfy stage -post hwos -n exadb05,exadb01 -verbose
.
.
.
Post-check for hardware and operating system setup was successful.


CVU operation performed:? ? ? stage -post hwos
Date:? ? ? ? ? ? ? ? ? ? ? ? ?Aug 27, 2021 10:06:59 AM
CVU home:? ? ? ? ? ? ? ? ? ? ?/u01/app/19.0.0.0/grid/
User:? ? ? ? ? ? ? ? ? ? ? ? ?oracle?        

2- Verify peer compatibility.

The phrase?Verification of peer compatibility was successful?should appear at the end of the report.

If the only failed components are related to the physical memory, swap space and disk space, then it is safe to continue.

--- Verify command
cluvfy comp peer -refnode <existing_node> -n <new_server> -orainv oinstall -osdba dba?


[oracle@exadb01 ~]$. oraenv
[ORACLE_SID]: +ASM1
[oracle@exadb01 ~]$ cd $ORACLE_HOME/bin
[oracle@exadb01 ~]$ ./cluvfy comp peer -refnode exadb01 -n exadb05 -orainv oinstall -osdba dba
.
.
.
Verifying Peer Compatibility ...PASSED


Verification of peer compatibility was successful.


CVU operation performed:? ? ? peer compatibility
Date:? ? ? ? ? ? ? ? ? ? ? ? ?Aug 27, 2021 10:13:11 AM
CVU home:? ? ? ? ? ? ? ? ? ? ?/u01/app/19.0.0.0/grid/
User:? ? ? ? ? ? ? ? ? ? ? ? ?oracle?        

3- Verify GI home is ready to be cloned to the new node.

--- Verify command
cluvfy stage -pre crsinst -n <new_server> -verbose


[oracle@exadb01 ~]$. oraenv
[ORACLE_SID]: +ASM1
[oracle@exadb01 ~]$ cd $ORACLE_HOME/bin
[oracle@exadb01 ~]$ ./cluvfy stage -pre crsinst -n exadb05 -verbose
.
.
.
Failures were encountered during execution of CVU verification request "stage -pre crsinst".


Verifying Group Existence: asmadmin ...FAILED
exadb05: PRVG-10461 : Group "asmadmin" selected for privileges "OSASM"
? ? ? ? ? ? ? ? does not exist on node "exadb05".


Verifying Group Existence: asmdba ...FAILED
exadb05: PRVG-10461 : Group "asmdba" selected for privileges "OSDBA"
? ? ? ? ? ? ? ? does not exist on node "exadb05".


Verifying Group Membership: asmadmin ...FAILED
exadb05: PRVG-10460 : User "oracle" does not belong to group "asmadmin"
? ? ? ? ? ? ? ? selected for privileges "OSASM" on node "exadb05".


Verifying Group Membership: asmdba ...FAILED
exadb05: PRVG-10460 : User "oracle" does not belong to group "asmdba"
? ? ? ? ? ? ? ? selected for privileges "OSDBA" on node "exadb05".


Verifying RPM Package Manager database ...INFORMATION
PRVG-11250 : The check "RPM Package Manager database" was not performed because
it needs 'root' user privileges.




CVU operation performed:? ? ? stage -pre crsinst
Date:? ? ? ? ? ? ? ? ? ? ? ? ?Aug 27, 2021 10:15:07 AM
CVU home:? ? ? ? ? ? ? ? ? ? ?/u01/app/19.0.0.0/grid/
User:? ? ? ? ? ? ? ? ? ? ? ? ?oracle


>>> FAILED check - Verifying Group Existence (OSASM, OSDBA, asmadmin, asmdba) is ignorable        

4- Verify node addition pre-requisites.

The phrase?Pre-check for for node addition was successful?should appear at the end of the report.

--- Verify command
cluvfy stage -pre nodeadd -n <new_server> -verbose


[oracle@exadb01 ~]$. oraenv
[ORACLE_SID]: +ASM1
[oracle@exadb01 ~]$ cd $ORACLE_HOME/bin
[oracle@exadb01 ~]$ ./cluvfy stage -pre nodeadd -n exadb05 -verbose
.
.
.
Pre-check for node addition was successful.


CVU operation performed:? ? ? stage -pre nodeadd
Date:? ? ? ? ? ? ? ? ? ? ? ? ?Aug 27, 2021 11:00:11 AM
CVU home:? ? ? ? ? ? ? ? ? ? ?/u01/app/19.0.0.0/grid/
User:? ? ? ? ? ? ? ? ? ? ? ? ?oracle        

5- Add new compute node to the cluster.

--- Add command
addnode.sh -silent "CLUSTER_NEW_NODES={<new_server>}" \
? ? ?"CLUSTER_NEW_VIRTUAL_HOSTNAMES={<new_server>-vip}" \
? ? ?"CLUSTER_NEW_NODE_ROLES={hub}"


[oracle@exadb01 ~]$. oraenv
[ORACLE_SID]: +ASM1
[oracle@exadb01 ~]$ cd $ORACLE_HOME/addnode
[oracle@exadb01 ~]$ ./addnode.sh -silent -ignoreSysPrereqs -ignorePrereq "CLUSTER_NEW_NODES={exadb05}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={exadb05-vip}" "CLUSTER_NEW_NODE_ROLES={hub}"


Copy Files to Remote Nodes in progress.
..................................................? ?6% Done.
..................................................? ?11% Done.
....................
Copy Files to Remote Nodes successful.


Prepare Configuration in progress.


Prepare Configuration successful.
..................................................? ?21% Done.
You can find the log of this install session at:
?/u01/app/oraInventory/logs/addNodeActions2021-08-27_11-14-47AM.log


Instantiate files in progress.


Instantiate files successful.
..................................................? ?49% Done.


Saving cluster inventory in progress.
..................................................? ?83% Done.


Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/19.0.0.0/grid was successful.
Please check '/u01/app/oraInventory/logs/silentInstall2021-08-27_11-14-47AM.log' for more details.


Setup Oracle Base in progress.


Setup Oracle Base successful.
..................................................? ?90% Done.


Update Inventory in progress.
You can find the log of this install session at:
?/u01/app/oraInventory/logs/addNodeActions2021-08-27_11-14-47AM.log


Update Inventory successful.
..................................................? ?97% Done.


As a root user, execute the following script(s):
? ? ? ? 1. /u01/app/oraInventory/orainstRoot.sh
? ? ? ? 2. /u01/app/19.0.0.0/grid/root.sh


Execute /u01/app/oraInventory/orainstRoot.sh on the following nodes:
[exadb05]
Execute /u01/app/19.0.0.0/grid/root.sh on the following nodes:
[exadb05]


The scripts can be executed in parallel on all the nodes.


Successfully Setup Software.
..................................................? ?100% Done.?        

6- Connect to exadb05 as root and execute below two root scripts.

[root@exadb05 ~]$ /u01/app/oraInventory/orainstRoot.sh
[root@exadb05 ~]$ /u01/app/19.0.0.0/grid/root.sh        

7- Check the cluster, once the /u01/app/19.0.0.0/grid/root.sh script successfully completes.

[oracle@exadb05 ~]$ . oraenv
[ORACLE_SID]: +ASM1
[oracle@exadb05 ~]$ $ORACLE_HOME/bin/crsctl check cluster -all
**************************************************************
exadb01:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
exadb02:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
exadb03:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
exadb04:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
exadb05:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online


[oracle@exadb05 ~]$ crsctl status res -t
>>> Confirm all cluster resources are ONLINE on new compute node as they are on existing compute nodes.        

8- Update the GI Inventory.

Add the new node (exadb05) in the GI inventory?on all the other nodes.

--- update command
runInstaller -updatenodelist -ignoreSysPrereqs ORACLE_HOME=<GI_HOME> "CLUSTER_NODES={<list_of_nodes_names>}" LOCAL_NODE=<node_name> CRS=TRUE


[oracle@exadb01 ~]$. oraenv
[ORACLE_SID]: +ASM1
[oracle@exadb01 ~]$ cd $ORACLE_HOME/oui/bin
./runInstaller -updatenodelist -ignoreSysPrereqs ORACLE_HOME=/u01/app/19.0.0.0/grid "CLUSTER_NODES={exadb01,exadb02,exadb03,exadb04,exadb05}" LOCAL_NODE=exadb01 CRS=TRUE
Starting Oracle Universal Installer...


Checking swap space: must be greater than 500 MB.? ?Actual 24318 MB? ? Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.


>>> Note that this has to be executed on every node of your cluster; remember that you need to adapt the LOCAL_NODE parameter on each node.?        

Clone?Oracle Database?Homes to to?the New Compute Node

Connect to the first compute node (exadb01) as oracle and run all steps below.

Oracle database home: /u01/app/oracle/product/12.2.0.1/dbhome_1

1- Add the?Oracle Database?ORACLE_HOME?to?the new node (exadb05).

[oracle@exadb01 ~]$ cd /u01/app/oracle/product/12.2.0.1/dbhome_1/addnode
[oracle@exadb01 ~]$ ./addnode.sh -silent "CLUSTER_NEW_NODES={exadb05}"
.
.
.
WARNING: The following configuration scripts need to be executed as the "root"
user in each cluster node.
/u01/app/oracle/product/12.2.0.1/dbhome_1/root.sh #On nodes exadb05
To execute the configuration scripts:
Open a terminal window.
Log in as root.
Run the scripts on each cluster node.
The Cluster Node Addition of /u01/app/oracle/product/12.2.0.1/dbhome_1 was successful.
Please check '/tmp/silentInstall.log' for more details.        

2- Connect to exadb05 as root and execute below root script.

[root@exadb05 ~]$ /u01/app/oracle/product/12.2.0.1/dbhome_1/root.sh
Check the /u01/app/orcale/product/12.2.0.1/dbhome_1/install/root_exadb05_<date>.log file for the output of the script.        

3- Run the Oracle Database Configuration Assistant (DBCA) in interactive mode to add database instance to the new compute node (exadb05).

[oracle@exadb01 ~]$ cd /u01/app/oracle/product/12.2.0.1/dbhome_1/bin
[oracle@exadb01 ~]$ ./dbca


- On the Database Operation screen, select Instance Management. Click Next.
- On the Instance Operation screen, select Add an instance. Click Next.
- On the Database List screen, select the cluster database to which you want to add an instance.
- The List Instance screen displays the current instances. Click Next to add a new instance.
- The Add Instance screen displays the default name and the newly added node to the cluster. Accept the defaults and click Next.
- On the Summary screen, verify the plan and click Finish.
- On the Progress screen, watch for 100% completion.
- On the Finish screen, acknowledge the confirmation that the new instance was successfully added.?        

4- Verify that the instance has been added.

[oracle@exadb01 ~]$ . oraenv
[ORACLE_SID]: PRODDB1
[oracle@exadb01 ~]$ srvctl status database -d PRODDB
[oracle@exadb01 ~]$ srvctl config database -d PRODDB


>>> Confirm instance PRODDB5 is running on exadb05        

5- Verify the administrative privileges on the new node (exadb05).

[oracle@exadb01 ~]$ . oraenv
[ORACLE_SID]: PRODDB1
[oracle@exadb01 ~]$ cd /u01/app/oracle/product/12.2.0.1/dbhome_1/bin
[oracle@exadb01 ~]$ ./cluvfy comp admprv -o db_config -d /u01/app/oracle/product/12.2.0.1/dbhome_1 -n exadb05        

6- Ensure the instance parameters are set for the new node (exadb05).

- Init parameters
local_listener
cluster_interconnects


[oracle@exadb05 ~]$ . oraenv
[ORACLE_SID]: PRODDB5
[oracle@exadb05 ~]$ sqlplus / as sysdba
SQL> alter system set local_listener = '(ADDRESS=(PROTOCOL=TCP)(HOST=<NEW_NODE_VIP>)(PORT=1521))' scope=spfile sid='PRODDB5';
SQL> alter system set cluster_interconnects = '<NEW_NODE_PRIV-IP1>,<NEW_NODE_PRIV-IP2>' scope=spfile sid='PRODDB5';?


--- Restart PRODDB5 instance to get changes applied.
[oracle@exadb05 ~]$ srvctl stop instance -d PRODDB -i PRODDB5
[oracle@exadb05 ~]$ srvctl start instance -d PRODDB -i PRODDB5
--- Confirm parameters new values?
[oracle@exadb05 ~]$ sqlplus / as sysdba
SQL> show parameter cluster_interconnects
SQL> show parameter local_listener?        

We are done !!!.