Friday, April 25, 2014

Patching Oracle Exadata (Quarterly Full Stack Jan 2014 - 11.2.0.3.0): 17816100

Patching Exadata is divided into 3 categories:

Storage Server Patch
Database Server Patch
Infiniband Switches Patch

I will write all the steps and explain why and how.

Critical NOTE: Before starting [patching, it is highly recommended to restart all Storage Servers and DB Nodes. Why? Because, during patching they will automatically restarted and if the restart is not successful, then you will have big headache! So, in my case everything went fine.


In our case we needed to download patch 17816100: "Quarterly Full Stack Download Patch For Oracle Exadata (Jan 2014 - 11.2.0.3.0)". Download from MOS.

Red Color are my comments, black are commands and commands'outputs.


1. Create folder tmpstage in /u01
[root@exa101 ~]# cd /u01/
[root@exa101 u01]# mkdir tmpstage

2. Copy all patch17816100* and p17938410_112330_Linux-x86-64 to /u01/tmpstage

3. The Quarterly Full Stack Download Patch For Oracle Exadata is a 3.7G tar image file when fully extracted. Due to it's size it has been split
into ~1.2G sized zip files for downloading from ARU. To assemble the image, please follow below procedure :

[root@exa101 tmpstage]# ls -l
total 3792432
-rw-r--r-- 1 root root 1258297983 Apr 14 12:21 p17816100_112030_Linux-x86-64_1of4.zip
-rw-r--r-- 1 root root 1258494785 Apr 14 12:11 p17816100_112030_Linux-x86-64_2of4.zip
-rw-r--r-- 1 root root 1246102503 Apr 14 02:56 p17816100_112030_Linux-x86-64_3of4.zip
-rw-r--r-- 1 root root 116728120 Apr 13 23:16 p17816100_112030_Linux-x86-64_4of4.zip
-rw-r--r-- 1 root root 1774 Apr 17 10:54 p17938410_112330_Linux-x86-64.zip

[root@exa101 tmpstage]# unzip p17816100_112030_Linux-x86-64_1of4.zip
Archive: p17816100_112030_Linux-x86-64_1of4.zip
inflating: 17816100.tar.splitaa
inflating: README.html
inflating: README.txt
inflating: PatchSearch.xml

[root@exa101 tmpstage]# unzip p17816100_112030_Linux-x86-64_2of4.zip
Archive: p17816100_112030_Linux-x86-64_2of4.zip
inflating: 17816100.tar.splitab

[root@exa101 tmpstage]# unzip p17816100_112030_Linux-x86-64_2of4.zip
Archive: p17816100_112030_Linux-x86-64_2of4.zip
replace 17816100.tar.splitab? [y]es, [n]o, [A]ll, [N]one, [r]ename: [root@exa101 tmpstage]# unzip p17816100_112030_Linux-x86-64_2of4.zip

[root@exa101 tmpstage]# unzip p17816100_112030_Linux-x86-64_3of4.zip
Archive: p17816100_112030_Linux-x86-64_3of4.zip
inflating: 17816100.tar.splitac

[root@exa101 tmpstage]# unzip p17816100_112030_Linux-x86-64_4of4.zip
Archive: p17816100_112030_Linux-x86-64_4of4.zip
inflating: 17816100.tar.splitad

[root@exa101 tmpstage]# ls -l
total 7598492
-rw-r--r-- 1 root root 1258291200 Jan 14 10:00 17816100.tar.splitaa
-rw-r--r-- 1 root root 1258291200 Jan 14 10:20 17816100.tar.splitab
-rw-r--r-- 1 root root 1258291200 Jan 14 10:39 17816100.tar.splitac
-rw-r--r-- 1 root root 118661120 Jan 14 10:41 17816100.tar.splitad
-rw-r--r-- 1 root root 1258297983 Apr 14 12:21 p17816100_112030_Linux-x86-64_1of4.zip
-rw-r--r-- 1 root root 1258494785 Apr 14 12:11 p17816100_112030_Linux-x86-64_2of4.zip
-rw-r--r-- 1 root root 1246102503 Apr 14 02:56 p17816100_112030_Linux-x86-64_3of4.zip
-rw-r--r-- 1 root root 116728120 Apr 13 23:16 p17816100_112030_Linux-x86-64_4of4.zip
-rw-r--r-- 1 root root 1774 Apr 17 10:54 p17938410_112330_Linux-x86-64.zip
-rw-rw-r-- 1 root root 3602 Jan 15 07:46 PatchSearch.xml
-rw-r--r-- 1 root root 28716 Jan 14 10:56 README.html
-rw-r--r-- 1 root root 9650 Jan 14 10:56 README.txt

[root@exa101 tmpstage]# cat *.tar.* | tar -xvf -
17816100/
17816100/Infrastructure/
17816100/Infrastructure/ExadataDBNodeUpdate/
17816100/Infrastructure/ExadataDBNodeUpdate/2.39/
17816100/Infrastructure/ExadataDBNodeUpdate/2.39/README.txt
17816100/Infrastructure/ExadataDBNodeUpdate/2.39/p16486998_121110_Linux-x86-64.zip
17816100/Infrastructure/ExadataStorageServer/
17816100/Infrastructure/ExadataStorageServer/11.2.3.3.0/
17816100/Infrastructure/ExadataStorageServer/11.2.3.3.0/README-16278923.html
17816100/Infrastructure/ExadataStorageServer/11.2.3.3.0/README-17809253.html
17816100/Infrastructure/ExadataStorageServer/11.2.3.3.0/p17809253_112330_Linux-x86-64.zip
17816100/Infrastructure/ExadataStorageServer/11.2.3.3.0/p16278923_112330_Linux-x86-64.zip
17816100/Infrastructure/SunRackIIPDUMeteringUnitFirmware/
17816100/Infrastructure/SunRackIIPDUMeteringUnitFirmware/1.06/
17816100/Infrastructure/SunRackIIPDUMeteringUnitFirmware/1.06/README.txt
17816100/Infrastructure/SunRackIIPDUMeteringUnitFirmware/1.06/p16523441_106_Generic.zip
17816100/SystemsManagement/
17816100/SystemsManagement/CloudControl12/
17816100/SystemsManagement/CloudControl12/WLS/
17816100/SystemsManagement/CloudControl12/WLS/14603262/
17816100/SystemsManagement/CloudControl12/WLS/14603262/README.txt
17816100/SystemsManagement/CloudControl12/WLS/14603262/p14603262_111160_Generic.zip
17816100/SystemsManagement/CloudControl12/WLS/16420963/
17816100/SystemsManagement/CloudControl12/WLS/16420963/README.txt
17816100/SystemsManagement/CloudControl12/WLS/16420963/p16420963_1036_Generic.zip
17816100/SystemsManagement/CloudControl12/Repository/
17816100/SystemsManagement/CloudControl12/Repository/11061801/
17816100/SystemsManagement/CloudControl12/Repository/11061801/p11061801_112030_Generic.zip
17816100/SystemsManagement/CloudControl12/Repository/11061801/README.txt
17816100/SystemsManagement/CloudControl12/Repository/16869210/
17816100/SystemsManagement/CloudControl12/Repository/16869210/README.txt
17816100/SystemsManagement/CloudControl12/Repository/16869210/p16869210_112030_Linux-x86-64.zip
17816100/SystemsManagement/CloudControl12/EMFramework/
17816100/SystemsManagement/CloudControl12/EMFramework/16822934/
17816100/SystemsManagement/CloudControl12/EMFramework/16822934/README.txt
17816100/SystemsManagement/CloudControl12/EMFramework/16822934/p16822934_121030_Generic.zip
17816100/SystemsManagement/CloudControl12/EMFramework/13983293/
17816100/SystemsManagement/CloudControl12/EMFramework/13983293/README.txt
17816100/SystemsManagement/CloudControl12/EMFramework/13983293/p13983293_111160_Generic.zip
17816100/README.txt
17816100/README.html
17816100/Database/
17816100/Database/OPatch/
17816100/Database/OPatch/11.2.0.3.6/
17816100/Database/OPatch/11.2.0.3.6/README.txt
17816100/Database/OPatch/11.2.0.3.6/p6880880_112000_Linux-x86-64.zip
17816100/Database/11.2.0.3.22_QDPE_Jan2014/
17816100/Database/11.2.0.3.22_QDPE_Jan2014/README.html
17816100/Database/11.2.0.3.22_QDPE_Jan2014/p17747147_112030_Linux-x86-64.zip
17816100/Database/OPlan/
17816100/Database/OPlan/12.1.0.1.5/
17816100/Database/OPlan/12.1.0.1.5/README.txt
17816100/Database/OPlan/12.1.0.1.5/p11846294_121000_Generic.zip

[root@exa101 tmpstage]# ls -l
total 7598496
drwxr-xr-x 5 3849 42424 4096 Jan 14 07:53 17816100
-rw-r--r-- 1 root root 1258291200 Jan 14 10:00 17816100.tar.splitaa
-rw-r--r-- 1 root root 1258291200 Jan 14 10:20 17816100.tar.splitab
-rw-r--r-- 1 root root 1258291200 Jan 14 10:39 17816100.tar.splitac
-rw-r--r-- 1 root root 118661120 Jan 14 10:41 17816100.tar.splitad
-rw-r--r-- 1 root root 1258297983 Apr 14 12:21 p17816100_112030_Linux-x86-64_1of4.zip
-rw-r--r-- 1 root root 1258494785 Apr 14 12:11 p17816100_112030_Linux-x86-64_2of4.zip
-rw-r--r-- 1 root root 1246102503 Apr 14 02:56 p17816100_112030_Linux-x86-64_3of4.zip
-rw-r--r-- 1 root root 116728120 Apr 13 23:16 p17816100_112030_Linux-x86-64_4of4.zip
-rw-r--r-- 1 root root 1774 Apr 17 10:54 p17938410_112330_Linux-x86-64.zip
-rw-rw-r-- 1 root root 3602 Jan 15 07:46 PatchSearch.xml
-rw-r--r-- 1 root root 28716 Jan 14 10:56 README.html
-rw-r--r-- 1 root root 9650 Jan 14 10:56 README.txt

[root@exa101 tmpstage]#

4. Verify the structure of the contents of the Quarterly Full Stack Download Patch For Oracle Exadata. The top-level directory contains this
README document and some sub-directories in the following layout:

17816100
|
|- README.txt
|
|- README.html
|
|
|- Infrastructure/
| --ExadataStorageServer/
| -11.2.3.3.0/
| -p16278923_112330_Linux-x86-64.zip
| -p17809253_112330_Linux-x86-64.zip
| -README-16278923.html
| -README-17809253.html
|
| --ExadataDBNodeUpdate/
| -2.39/
| -p16486998_121110_Linux-x86-64.zip
| -README.txt
|
| --SunRackIIPDUMeteringUnitFirmware/
| -1.06/
| -p16523441_106_Generic.zip
| -README.txt
|
|
|- Database/
| --11.2.0.3.22_QDPE_Jan2014/
| -p17747147_112030_Linux-x86-64.zip
| -README.html
|
| --OPatch/
| -11.2.0.3.6/
| -p6880880_112000_Linux-x86-64.zip
| -README.txt
|
| --OPlan/
| -12.1.0.1.5/
| -p11846294_121000_Generic.zip
| -README.txt
|
|
|- SystemsManagement/
|
| --CloudControl12/
|
| --EMFramework/
| -13983293/
| -p13983293_111160_Generic.zip
| -README.txt
| -16822934/
| -p16822934_121030_Generic.zip
| -README.txt
|
| --Repository/
| -11061801/
| -p11061801_112030_Generic.zip
| -README.txt
| -16869210/
| -p16869210_112030_Linux-x86-64.zip
| -README.txt
|
| --WLS/
| -14603262/
| -p14603262_111160_Generic.zip
| -README.txt
| -16420963/
| -p16420963_1036_Generic.zip
| -README.txt
|

5. Patching Exadata Cells

5.1 Preparing Exadata Cells for Patch Application

All steps must be done under root user.

5.1.1 Set up the root user SSH equivalence database server to all cell servers:

a. Create a file named cell_group that has one cell host name or IP address per line for each cell to be patched:
[root@exa101 ~]# cat cell_group
exa1cel01
exa1cel02
exa1cel03

b. Check for existing root SSH equivalence. The following command should require no password prompts and no interaction. It should return the
list of host names in the cell_group file.

[root@exa101 ~]# dcli -g cell_group -l root 'hostname -i'
exa1cel01: 10.45.33.4
exa1cel02: 10.45.33.5
exa1cel03: 10.45.33.6


Set up SSH root equivalence if not already done so from the launch server. Do not do this step if you already have root SSH equivalence.

Generate root SSH keys as follows:

ssh-keygen -t rsa

Accept the defaults so the SSH keys are created for the root user.

Push the SSH keys to set up SSH equivalence using the following command. Enter the root password when prompted.

dcli -g cell_group -l root -k


5.1.2 We will perform non-rolling upgrade, so Stop all CRS completely:

[root@exa102 ~]# /u01/app/11.2.0.3/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'exa102'
CRS-2673: Attempting to stop 'ora.crsd' on 'exa102'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'exa102'
CRS-2673: Attempting to stop 'ora.DBFS_DG.dg' on 'exa102'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'exa102'
CRS-2673: Attempting to stop 'ora.rmanst.db' on 'exa102'
CRS-2673: Attempting to stop 'ora.pdmn.dpmn_dg.svc' on 'exa102'
CRS-2673: Attempting to stop 'ora.exa101.vip' on 'exa102'
CRS-2673: Attempting to stop 'ora.cvu' on 'exa102'
CRS-2673: Attempting to stop 'ora.LISTENER_DG.lsnr' on 'exa102'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'exa102'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'exa102'
CRS-2673: Attempting to stop 'ora.exa1db02-dg-vip.vip' on 'exa102'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'exa102'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'exa102'
CRS-2673: Attempting to stop 'ora.oc4j' on 'exa102'
CRS-2677: Stop of 'ora.pdmn.dpmn_dg.svc' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.pdmn.db' on 'exa102'
CRS-2677: Stop of 'ora.exa101.vip' on 'exa102' succeeded
CRS-2677: Stop of 'ora.exa1db02-dg-vip.vip' on 'exa102' succeeded
CRS-2677: Stop of 'ora.cvu' on 'exa102' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.exa102.vip' on 'exa102'
CRS-2677: Stop of 'ora.LISTENER_DG.lsnr' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.exa1db01-dg-vip.vip' on 'exa102'
CRS-2677: Stop of 'ora.exa102.vip' on 'exa102' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'exa102'
CRS-2677: Stop of 'ora.exa1db01-dg-vip.vip' on 'exa102' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'exa102' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'exa102'
CRS-2677: Stop of 'ora.scan2.vip' on 'exa102' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'exa102'
CRS-2677: Stop of 'ora.scan3.vip' on 'exa102' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'exa102' succeeded
CRS-2677: Stop of 'ora.pdmn.db' on 'exa102' succeeded
CRS-2677: Stop of 'ora.rmanst.db' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.DATA_EXA1.dg' on 'exa102'
CRS-2673: Attempting to stop 'ora.RECO_EXA1.dg' on 'exa102'
CRS-2677: Stop of 'ora.RECO_EXA1.dg' on 'exa102' succeeded
CRS-2677: Stop of 'ora.DATA_EXA1.dg' on 'exa102' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'exa102' succeeded
CRS-2677: Stop of 'ora.DBFS_DG.dg' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'exa102'
CRS-2677: Stop of 'ora.asm' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.net2.network' on 'exa102'
CRS-2673: Attempting to stop 'ora.ons' on 'exa102'
CRS-2677: Stop of 'ora.net2.network' on 'exa102' succeeded
CRS-2677: Stop of 'ora.ons' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'exa102'
CRS-2677: Stop of 'ora.net1.network' on 'exa102' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'exa102' has completed
CRS-2677: Stop of 'ora.crsd' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'exa102'
CRS-2673: Attempting to stop 'ora.evmd' on 'exa102'
CRS-2673: Attempting to stop 'ora.asm' on 'exa102'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'exa102'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'exa102'
CRS-2677: Stop of 'ora.evmd' on 'exa102' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'exa102' succeeded
CRS-2677: Stop of 'ora.asm' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'exa102'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'exa102' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'exa102' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'exa102'
CRS-2677: Stop of 'ora.cssd' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'exa102'
CRS-2673: Attempting to stop 'ora.crf' on 'exa102'
CRS-2677: Stop of 'ora.diskmon' on 'exa102' succeeded
CRS-2677: Stop of 'ora.crf' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'exa102'
CRS-2677: Stop of 'ora.gipcd' on 'exa102' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'exa102'
CRS-2677: Stop of 'ora.gpnpd' on 'exa102' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'exa102' has completed
CRS-4133: Oracle High Availability Services has been stopped.


5.1.3 Shut down all cell services on all cells to be patched. This may be done by the root user on each cell by running cellcli -e 'alter cell shutdown services all' or by the following dcli command to do all cells at the same time:

[root@exa101 ~]# dcli -g cell_group -l root "cellcli -e alter cell shutdown services all"
exa1cel01:
exa1cel01: Stopping the RS, CELLSRV, and MS services...
exa1cel01: The SHUTDOWN of services was successful.
exa1cel02:
exa1cel02: Stopping the RS, CELLSRV, and MS services...
exa1cel02: The SHUTDOWN of services was successful.
exa1cel03:
exa1cel03: Stopping the RS, CELLSRV, and MS services...
exa1cel03: The SHUTDOWN of services was successful.
[root@exa101 ~]#


5.2 Applying the Patch to Exadata Cells

5.2.1 Note below cases:

-Do not use the serial console, the ILOM or LO web-based console to start the patchmgr utility. You are using the serial console if the output
from the following command is consoletype=serial.

[root@exa101 ~]# echo $consoletype
pty

For Oracle Exadata Database Machine with Sun hardware cells: Use SSH to the ILOM host name or IP address as the root user. Do the following to
start the serial console,
start /SP/console
To stop it press the Escape key (ESC) followed by ( and then stop /SP/console.

-Use a fresh log in session for each patch or rollback procedure. Do not run the rollback procedure from the same login session that the patch
was applied. Do not re-run a patch application from a login session in which the rollback procedure was run.

-Do not interrupt the patching procedure once it is initiated on a cell.

-Cells automatically reboot, as needed, during patch application. Do not reboot or power cycle cells while applying the patch.

-Do not edit any log file or open log files in writable mode. You may use any of the following to view a log: view, less, more or tail. You may
cause the patch application to be interrupted by editing its logs while applying the patch.

-Monitor patch activity using less -rf patchmgr.stdout from another terminal session or window to see raw log details from the patchmgr utility.


5.2.2 Patching Using the patchmgr Utility


Unzip the patch, it will extract into the patch_11.2.3.3.0.131014.1 directory. Change to the patch_11.2.3.3.0.131014.1 directory.

[root@exa101 ~]# unzip p16278923_112330_Linux-x86-64.zip
Archive: p16278923_112330_Linux-x86-64.zip
creating: patch_11.2.3.3.0.131014.1/
inflating: patch_11.2.3.3.0.131014.1/sundcs_36p_repository_2.1.3_4.pkg
inflating: patch_11.2.3.3.0.131014.1/upgradeIBSwitch.sh
creating: patch_11.2.3.3.0.131014.1/linux.db.rpms/
inflating: patch_11.2.3.3.0.131014.1/linux.db.rpms/perl-XML-Parser-2.34-6.1.2.2.1.x86_64.rpm
creating: patch_11.2.3.3.0.131014.1/plugins/
inflating: patch_11.2.3.3.0.131014.1/plugins/000-check_dummy_bash
inflating: patch_11.2.3.3.0.131014.1/plugins/000-check_dummy_perl
inflating: patch_11.2.3.3.0.131014.1/imageLogger
inflating: patch_11.2.3.3.0.131014.1/ExadataSendNotification.pm
inflating: patch_11.2.3.3.0.131014.1/11.2.3.3.0.131014.1.patch.tar
inflating: patch_11.2.3.3.0.131014.1/dostep.sh
inflating: patch_11.2.3.3.0.131014.1/patchmgr
inflating: patch_11.2.3.3.0.131014.1/ExaXMLNode.pm
inflating: patch_11.2.3.3.0.131014.1/README.txt
inflating: patch_11.2.3.3.0.131014.1/dcli
creating: patch_11.2.3.3.0.131014.1/etc/
creating: patch_11.2.3.3.0.131014.1/etc/config/
inflating: patch_11.2.3.3.0.131014.1/etc/config/inventory.xml
inflating: patch_11.2.3.3.0.131014.1/README.html
inflating: patch_11.2.3.3.0.131014.1/sundcs_36p_repository_1.3.3_2.pkg
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/Torus.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/QuarterRack.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/FatTree.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/HalfRack.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/fetchTopology.pm
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscHalfRack/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscHalfRack/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscHalfRack/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscHalfRack/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscHalfRack/iblinkinfo.log
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrack/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrack/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrack/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrack/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrack/iblinkinfo.log
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/halfrack/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/halfrack/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/halfrack/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/halfrack/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/halfrack/iblinkinfo.log
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/fullX2Rack/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/fullX2Rack/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/fullX2Rack/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/fullX2Rack/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/fullX2Rack/iblinkinfo.log
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrackError/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrackError/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrackError/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrackError/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/quarterrackError/iblinkinfo.log
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyHalf/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyHalf/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyHalf/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyHalf/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyHalf/iblinkinfo.log
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyFull/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyFull/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyFull/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyFull/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/storageOnlyFull/iblinkinfo.log
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscFullRack/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscFullRack/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscFullRack/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscFullRack/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/sscFullRack/iblinkinfo.log
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/twoX2racks/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/twoX2racks/ib_switches.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/twoX2racks/ibnetdiscover.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/twoX2racks/ib_hosts.lst
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/tests/twoX2racks/iblinkinfo.log
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/mapCheck.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/fattree.map
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topologies/nm2ports.map
extracting: patch_11.2.3.3.0.131014.1/ibdiagtools/xmonib.sh
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/infinicheck
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/dcli
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/verify-topology
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/topology-zfs
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/README
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/setup-ssh
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/SampleOutputs.txt
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/ibping_test
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/monitord
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/tar_ibdiagtools
extracting: patch_11.2.3.3.0.131014.1/ibdiagtools/VERSION_FILE
creating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/LinuxAdapter.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/CommonUtils.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/OSAdapter.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/spawnProc.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/remoteLauncher.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/remoteScriptGenerator.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/remoteConfig.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/SolarisAdapter.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/netcheck/runDiagnostics.pm
inflating: patch_11.2.3.3.0.131014.1/ibdiagtools/checkbadlinks.pl
inflating: patch_11.2.3.3.0.131014.1/11.2.3.3.0.131014.1.iso
inflating: patch_11.2.3.3.0.131014.1/ExadataImageNotification.pl
inflating: patch_11.2.3.3.0.131014.1/exadataLogger.pm
[root@exa101 ~]#



Download any patchmgr plug-ins attached to the My Oracle Support note 1487339.1, and install them as documented in the My Oracle Support note.
Oracle recommends reviewing the My Oracle Support note, issues and workarounds listed in the note just before starting to actually apply the
patch.

In our case it is p17938410_112330_Linux-x86-64.zip.

[root@exa101 ~]# cd /u01/tmpstage/

[root@exa101 tmpstage]# ls -l
total 7598496
drwxr-xr-x 5 3849 42424 4096 Jan 14 07:53 17816100
-rw-r--r-- 1 root root 1258291200 Jan 14 10:00 17816100.tar.splitaa
-rw-r--r-- 1 root root 1258291200 Jan 14 10:20 17816100.tar.splitab
-rw-r--r-- 1 root root 1258291200 Jan 14 10:39 17816100.tar.splitac
-rw-r--r-- 1 root root 118661120 Jan 14 10:41 17816100.tar.splitad
-rw-r--r-- 1 root root 1258297983 Apr 14 12:21 p17816100_112030_Linux-x86-64_1of4.zip
-rw-r--r-- 1 root root 1258494785 Apr 14 12:11 p17816100_112030_Linux-x86-64_2of4.zip
-rw-r--r-- 1 root root 1246102503 Apr 14 02:56 p17816100_112030_Linux-x86-64_3of4.zip
-rw-r--r-- 1 root root 116728120 Apr 13 23:16 p17816100_112030_Linux-x86-64_4of4.zip
-rw-r--r-- 1 root root 1774 Apr 17 10:54 p17938410_112330_Linux-x86-64.zip
-rw-rw-r-- 1 root root 3602 Jan 15 07:46 PatchSearch.xml
-rw-r--r-- 1 root root 28716 Jan 14 10:56 README.html
-rw-r--r-- 1 root root 9650 Jan 14 10:56 README.txt

[root@exa101 tmpstage]# unzip -d /root/patch_11.2.3.3.0.131014.1/plugins/ p17938410_112330_Linux-x86-64.zip -x Readme.txt
Archive: p17938410_112330_Linux-x86-64.zip
inflating: /root/patch_11.2.3.3.0.131014.1/plugins/001-post_11_2_3_3_0

[root@exa101 tmpstage]# cd /root/patch_11.2.3.3.0.131014.1/plugins/

[root@exa101 plugins]# ls -l
total 12
-r-xr-xr-x 1 root root 3986 Oct 15 2013 000-check_dummy_bash
-r-xr-xr-x 1 root root 3370 Oct 15 2013 000-check_dummy_perl
-rw-r--r-- 1 root root 3505 Dec 11 15:16 001-post_11_2_3_3_0

[root@exa101 plugins]# pwd
/root/patch_11.2.3.3.0.131014.1/plugins

[root@exa101 plugins]# chmod +x *

[root@exa101 plugins]# ls -l
total 12
-r-xr-xr-x 1 root root 3986 Oct 15 2013 000-check_dummy_bash
-r-xr-xr-x 1 root root 3370 Oct 15 2013 000-check_dummy_perl
-rwxr-xr-x 1 root root 3505 Dec 11 15:16 001-post_11_2_3_3_0


Recommended: Reset the server to a known state using the following command:
This step is only done the first time the cells are patched to this release. It is not necessary to use the command for subsequent cell patching,
even after rolling back the patch.

[root@exa101]# cd /root/patch_11.2.3.3.0.131014.1/

[root@exa101 patch_11.2.3.3.0.131014.1]# ./patchmgr -cells cell_group -reset_force

2014-04-24 20:42:19 +0400 :DONE: reset_force


Clean up any previous patchmgr utility runs using the following command: (As I told in the beginning, use fresh session for each patchmgr
session)

[root@exa101 patch_11.2.3.3.0.131014.1]# ./patchmgr -cells cell_group -cleanup

2014-04-24 20:44:38 +0400 :DONE: Cleanup

Verify that the cells meet prerequisite checks using the following command. Use the -rolling option if you plan to use rolling updates:

[root@exa101 patch_11.2.3.3.0.131014.1]# ./patchmgr -cells cell_group -patch_check_prereq

date: invalid date `%F %T %z'
2014-04-24 20:47:03 +0400 :Working: DO: Check cells have ssh equivalence for root user. Up to 10 seconds per cell ...
2014-04-24 20:47:05 +0400 :SUCCESS: DONE: Check cells have ssh equivalence for root user.
2014-04-24 20:47:05 +0400 :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
2014-04-24 20:48:05 +0400 :SUCCESS: DONE: Initialize files, check space and state of cell services.
2014-04-24 20:48:05 +0400 :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size
correction. Up to 40 minutes ...
2014-04-24 20:48:19 +0400 Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes.


2014-04-24 20:48:20 +0400 :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner
size correction.
2014-04-24 20:48:20 +0400 :Working: DO: Check prerequisites on all cells. Up to 2 minutes ...
2014-04-24 20:49:04 +0400 :SUCCESS: DONE: Check prerequisites on all cells.
2014-04-24 20:49:04 +0400 :Working: DO: Execute plugin check for Patch Check Prereq ...
2014-04-24 20:49:05 +0400 :SUCCESS: DONE: Execute plugin check for Patch Check Prereq.

Output from patchmgr:

[root@exa101 patch_11.2.3.3.0.131014.1]# tail -f patchmgr.stdout
2014-04-24 20:42:19 +0400 :DONE: reset_force
================PatchMgr run ended 2014-04-24 20:42:19 +0400 ===========


================PatchMgr run started 2014-04-24 20:43:37 +0400 ===========
2014-04-24 20:44:38 +0400 :DONE: Cleanup
================PatchMgr run ended 2014-04-24 20:44:44 +0400 ===========


================PatchMgr run started 2014-04-24 20:46:01 +0400 ===========

[INFO]: Install rpm /root/patch_11.2.3.3.0.131014.1/linux.db.rpms/perl-XML-Parser-2.34-6.1.2.2.1.x86_64.rpm
Preparing... ##################################################
perl-XML-Parser ##################################################
2014-04-24 20:47:03 +0400 :Working: DO: Check cells have ssh equivalence for root user. Up to 10 seconds per cell ...
2014-04-24 20:47:04 +0400 ++++++++++++++++++ Logs so far begin ++++++++++
2014-04-24 20:47:04 +0400 ++++++++++++++++++ Logs so far end ++++++++++
2014-04-24 20:47:05 +0400 :SUCCESS: DONE: Check cells have ssh equivalence for root user.
2014-04-24 20:47:05 +0400 :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
2014-04-24 20:48:05 +0400 ++++++++++++++++++ Logs so far begin ++++++++++
exa1cel01:
exa1cel01: 2014-04-24 20:47:10 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel01: 2014-04-24 20:47:10 +0400 [INFO]: dostep called: prechk:no:1406 patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
noforce
exa1cel01: 2014-04-24 20:47:10 +0400 [INFO]: patchmgr launched from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1
exa1cel01: _EXIT_PASS_Cell exa1cel01 10.45.33.4 2014-04-24 20:48:05 +0400:
exa1cel02:
exa1cel02: 2014-04-24 20:47:09 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel02: 2014-04-24 20:47:09 +0400 [INFO]: dostep called: prechk:no:1406 patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
noforce
exa1cel02: 2014-04-24 20:47:09 +0400 [INFO]: patchmgr launched from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1
exa1cel02: _EXIT_PASS_Cell exa1cel02 10.45.33.5 2014-04-24 20:48:04 +0400:
exa1cel03:
exa1cel03: 2014-04-24 20:47:10 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel03: 2014-04-24 20:47:10 +0400 [INFO]: dostep called: prechk:no:1406 patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
noforce
exa1cel03: 2014-04-24 20:47:10 +0400 [INFO]: patchmgr launched from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1
exa1cel03: _EXIT_PASS_Cell exa1cel03 10.45.33.6 2014-04-24 20:48:04 +0400:
exa1cel01:
exa1cel01: _EXIT_PASS_Cell exa1cel01 10.45.33.4
exa1cel02:
exa1cel02: _EXIT_PASS_Cell exa1cel02 10.45.33.5
exa1cel03:
exa1cel03: _EXIT_PASS_Cell exa1cel03 10.45.33.6
exa1cel01:
exa1cel01: [INFO] Free space in /boot (/dev/md4) before clean up is 58Mb
exa1cel01: [INFO] Kernel version on runtime: 2.6.32-400.11.1.el5uek
exa1cel01: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.32-400.11.1.el5uek
exa1cel01: [INFO] Free space in /boot (/dev/md4) after clean up is 58Mb
exa1cel01: [INFO] Size for all files for the kernel 2.6.32-400.11.1.el5uek in /boot is 25Mb
exa1cel01: [INFO] Required free space on /boot is 50Mb
exa1cel01: Cell exa1cel01 10.45.33.4
exa1cel01: _EXIT_PASS_Cell exa1cel01 10.45.33.4 2014-04-24 20:48:05 +0400:
exa1cel02:
exa1cel02: [INFO] Free space in /boot (/dev/md4) before clean up is 58Mb
exa1cel02: [INFO] Kernel version on runtime: 2.6.32-400.11.1.el5uek
exa1cel02: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.32-400.11.1.el5uek
exa1cel02: [INFO] Free space in /boot (/dev/md4) after clean up is 58Mb
exa1cel02: [INFO] Size for all files for the kernel 2.6.32-400.11.1.el5uek in /boot is 25Mb
exa1cel02: [INFO] Required free space on /boot is 50Mb
exa1cel02: Cell exa1cel02 10.45.33.5
exa1cel02: _EXIT_PASS_Cell exa1cel02 10.45.33.5 2014-04-24 20:48:04 +0400:
exa1cel03:
exa1cel03: [INFO] Free space in /boot (/dev/md4) before clean up is 58Mb
exa1cel03: [INFO] Kernel version on runtime: 2.6.32-400.11.1.el5uek
exa1cel03: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.32-400.11.1.el5uek
exa1cel03: [INFO] Free space in /boot (/dev/md4) after clean up is 58Mb
exa1cel03: [INFO] Size for all files for the kernel 2.6.32-400.11.1.el5uek in /boot is 25Mb
exa1cel03: [INFO] Required free space on /boot is 50Mb
exa1cel03: Cell exa1cel03 10.45.33.6
exa1cel03: _EXIT_PASS_Cell exa1cel03 10.45.33.6 2014-04-24 20:48:04 +0400:
2014-04-24 20:48:05 +0400 ++++++++++++++++++ Logs so far end ++++++++++
2014-04-24 20:48:05 +0400 :SUCCESS: DONE: Initialize files, check space and state of cell services.
2014-04-24 20:48:05 +0400 :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size
correction. Up to 40 minutes ...
2014-04-24 20:48:19 +0400 ++++++++++++++++++ Logs so far begin ++++++++++
exa1cel01:
exa1cel01: 2014-04-24 20:48:08 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel01: 2014-04-24 20:48:08 +0400 [INFO]: dostep called: 4p patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel01: _EXIT_PASS_Cell exa1cel01 10.45.33.4 2014-04-24 20:48:09 +0400:
exa1cel01:
exa1cel01: 2014-04-24 20:48:19 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel01: 2014-04-24 20:48:19 +0400 [INFO]: dostep called: 4p_md11_fix_pw patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel01: _EXIT_PASS_Cell exa1cel01 10.45.33.4 2014-04-24 20:48:19 +0400:
exa1cel02:
exa1cel02: 2014-04-24 20:48:08 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel02: 2014-04-24 20:48:08 +0400 [INFO]: dostep called: 4p patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel02: _EXIT_PASS_Cell exa1cel02 10.45.33.5 2014-04-24 20:48:08 +0400:
exa1cel02:
exa1cel02: 2014-04-24 20:48:19 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel02: 2014-04-24 20:48:19 +0400 [INFO]: dostep called: 4p_md11_fix_pw patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel02: _EXIT_PASS_Cell exa1cel02 10.45.33.5 2014-04-24 20:48:19 +0400:
exa1cel03:
exa1cel03: 2014-04-24 20:48:08 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel03: 2014-04-24 20:48:08 +0400 [INFO]: dostep called: 4p patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel03: _EXIT_PASS_Cell exa1cel03 10.45.33.6 2014-04-24 20:48:09 +0400:
exa1cel03:
exa1cel03: 2014-04-24 20:48:19 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel03: 2014-04-24 20:48:19 +0400 [INFO]: dostep called: 4p_md11_fix_pw patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel03: _EXIT_PASS_Cell exa1cel03 10.45.33.6 2014-04-24 20:48:19 +0400:
exa1cel01: Cell exa1cel01 10.45.33.4
exa1cel01: _EXIT_PASS_Cell exa1cel01 10.45.33.4 2014-04-24 20:48:19 +0400:
exa1cel02: Cell exa1cel02 10.45.33.5
exa1cel02: _EXIT_PASS_Cell exa1cel02 10.45.33.5 2014-04-24 20:48:19 +0400:
exa1cel03: Cell exa1cel03 10.45.33.6
exa1cel03: _EXIT_PASS_Cell exa1cel03 10.45.33.6 2014-04-24 20:48:19 +0400:
2014-04-24 20:48:20 +0400 ++++++++++++++++++ Logs so far end ++++++++++
2014-04-24 20:48:20 +0400 :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner
size correction.
2014-04-24 20:48:20 +0400 :Working: DO: Check prerequisites on all cells. Up to 2 minutes ...
2014-04-24 20:49:03 +0400 ++++++++++++++++++ Logs so far begin ++++++++++
exa1cel01:
exa1cel01: 2014-04-24 20:48:20 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel01: 2014-04-24 20:48:20 +0400 [INFO]: dostep called: 5p patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel01: _EXIT_PASS_Cell exa1cel01 10.45.33.4 2014-04-24 20:49:03 +0400: 2014-04-24 20:49:03 +0400 [INFO] Started to patch.
exa1cel02:
exa1cel02: 2014-04-24 20:48:20 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel02: 2014-04-24 20:48:20 +0400 [INFO]: dostep called: 5p patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel02: _EXIT_PASS_Cell exa1cel02 10.45.33.5 2014-04-24 20:49:03 +0400: 2014-04-24 20:49:02 +0400 [INFO] Started to patch.
exa1cel03:
exa1cel03: 2014-04-24 20:48:20 +0400 [INFO]: patchmgr launch attempt from exa101.bog.ge_10.0.0.150_root_patch_11.2.3.3.0.131014.1.
exa1cel03: 2014-04-24 20:48:20 +0400 [INFO]: dostep called: 5p patch_prereq -no-auto-rollback non_rolling 600 3600 900 36000 default
exa1cel03: _EXIT_PASS_Cell exa1cel03 10.45.33.6 2014-04-24 20:49:03 +0400: 2014-04-24 20:49:03 +0400 [INFO] Started to patch.
exa1cel01: 2014-04-24 20:48:20 +0400
exa1cel01: 2014-04-24 20:48:20 +0400 ./install.sh started with command line: -patch
exa1cel01: 2014-04-24 20:48:21 +0400 [INFO] Check md devices have clean state ...
exa1cel01: 2014-04-24 20:48:21 +0400 [INFO] Check disks and partitions ...
exa1cel01: Active image info
exa1cel01: Version : 11.2.3.2.1.130109
exa1cel01: System partition on device : /dev/md6
exa1cel01: Software partition on device : /dev/md8
exa1cel01: Cell boot usb version : 11.2.3.2.1.130109
exa1cel01:
exa1cel01: Patch proposal
exa1cel01: Version : 11.2.3.3.0.131014.1
exa1cel01: System partition on device : /dev/md5
exa1cel01: Software partition on device : /dev/md7
exa1cel01: Upgrade mode : patch
exa1cel01: Zero auto rollback : enabled
exa1cel01: Auto rollback : enabled
exa1cel01: Reset SP : yes
exa1cel01: First boot device : usb
exa1cel01: Use usb on upgrade : no
exa1cel01: Last console in hd grub.conf : web
exa1cel01: Stop on zero boot : disabled
exa1cel01: Redirection log file :
exa1cel01:
exa1cel01: 2014-04-24 20:48:27 +0400 [INFO] Shut down MS before doing ipmitool commands.
exa1cel01:
exa1cel01: Stopping MS services...
exa1cel01: The SHUTDOWN of MS services was successful.
exa1cel01: 2014-04-24 20:48:35 +0400 [INFO] Service Processor (SP) is available
exa1cel01: 2014-04-24 20:48:35 +0400 [INFO] Start MS after doing ipmitool commands.
exa1cel01:
exa1cel01: Starting MS services...
exa1cel01: The STARTUP of MS services was successful.
exa1cel01:
exa1cel01: Stopping MS services...
exa1cel01: The SHUTDOWN of MS services was successful.
exa1cel01: 2014-04-24 20:48:50 +0400 [INFO] No running instance of checkdeveachboot found. Safe to proceed
exa1cel01: 2014-04-24 20:48:50 +0400 [INFO] Check healthy states of Bootable or/and MD device(s)
exa1cel01:
exa1cel01: /opt/oracle.cellos/validations/init.d/checkdeveachboot started at 2014-04-24 20:48:50
exa1cel01: Set default check mode
exa1cel01: Set lock /tmp/cellos/locks/4636.checkdeveachboot
exa1cel01: Check mode is in use
exa1cel01: Check system disks for the cell node
exa1cel01: lsi controller has disks that are mapped to: /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj
/dev/sdk /dev/sdl
exa1cel01: The current system disks with valid partition table: /dev/sda /dev/sdb
exa1cel01: System disk /dev/sda is a bootable device
exa1cel01: Partition /dev/sda1 has a proper boot signature
exa1cel01: System disk /dev/sdb is a bootable device
exa1cel01: Partition /dev/sdb1 has a proper boot signature
exa1cel01: Check for degraded or stopped md devices
exa1cel01: Check CELLBOOT USB
exa1cel01: Check CELLBOOT USB
exa1cel01: USB partition /dev/sdm1 has a valid boot signature for CELLBOOT usb
exa1cel01: USB device /dev/sdm is a bootable device
exa1cel01: USB partition /dev/sdm1 is mountable
exa1cel01: USB has version 11.2.3.2.1.130109
exa1cel01: Reset lock /tmp/cellos/locks/4636.checkdeveachboot
exa1cel01: 0:0:0:0
exa1cel01:
exa1cel01: Starting MS services...
exa1cel01: The STARTUP of MS services was successful.
exa1cel01: /root/_patch_hctap_/_p_/biosbootorder started at 2014-04-24_20:48:59
exa1cel01: Check and set BIOS boot order
exa1cel01: 2014-04-24 20:48:59 +0400 CELLBOOT USB partition: /dev/sdm1
exa1cel01: 2014-04-24 20:48:59 +0400 [INFO] CELLBOOT USB block device: /sys/block/sdm
exa1cel01: 2014-04-24 20:48:59 +0400 CELLBOOT USB vendor: ORACLE
exa1cel01: 2014-04-24 20:48:59 +0400 [INFO] CELLBOOT USB model: UNIGEN-UFD
exa1cel01: 2014-04-24 20:48:59 +0400 [INFO] Looking for CELLBOOT USB pattern "(USB:Port0:ORACLE|USB:.*ORACLE .*UNIGEN-UFD.*)" in BIOS boot order
exa1cel01: 2014-04-24 20:48:59 +0400 [INFO] Looking for RAID pattern "RAID:" in BIOS boot order
exa1cel01:
exa1cel01: Copyright (C) SUN Microsystems 2009.
exa1cel01: BIOSconfig Utility Version 2.2.1
exa1cel01: Build Date: Aug 24 2009
exa1cel01: Build Time: 09:01:30
exa1cel01:
exa1cel01: BIOSconfig Specification Version 2.4
exa1cel01:
exa1cel01: Processing Input BIOS Data....
exa1cel01:
exa1cel01: Success
exa1cel01: 2014-04-24 20:49:03 +0400 [INFO] The current BIOS boot order:
exa1cel01:
exa1cel01:
exa1cel01: USB:02.82;01 ORACLE UNIGEN-UFD
exa1cel01: 00,1a,07
exa1cel01:

exa1cel01:
exa1cel01:
exa1cel01: RAID:Slot0.F0:(Bus 13 Dev 00)PCI RAID Adapter
exa1cel01: 13,00,00
exa1cel01:

exa1cel01:
exa1cel01:
exa1cel01: PXE:IBA GE Slot 0100 v1331
exa1cel01: 01,00,00
exa1cel01:

exa1cel01:
exa1cel01: 2014-04-24 20:49:03 +0400 [INFO] CELLBOOT USB has proper order 1
exa1cel01:
exa1cel01: 2014-04-24 20:49:03 +0400 [INFO] RAID controller has proper order 2
exa1cel01:
exa1cel01: 2014-04-24 20:49:03 +0400 [INFO] Boot device order is correct.
exa1cel01: Cell exa1cel01 10.45.33.4
exa1cel01: _EXIT_PASS_Cell exa1cel01 10.45.33.4 2014-04-24 20:49:03 +0400: 2014-04-24 20:49:03 +0400 [INFO] Started to patch.
exa1cel02: 2014-04-24 20:48:20 +0400
exa1cel02: 2014-04-24 20:48:20 +0400 ./install.sh started with command line: -patch
exa1cel02: 2014-04-24 20:48:20 +0400 [INFO] Check md devices have clean state ...
exa1cel02: 2014-04-24 20:48:21 +0400 [INFO] Check disks and partitions ...
exa1cel02: Active image info
exa1cel02: Version : 11.2.3.2.1.130109
exa1cel02: System partition on device : /dev/md6
exa1cel02: Software partition on device : /dev/md8
exa1cel02: Cell boot usb version : 11.2.3.2.1.130109
exa1cel02:
exa1cel02: Patch proposal
exa1cel02: Version : 11.2.3.3.0.131014.1
exa1cel02: System partition on device : /dev/md5
exa1cel02: Software partition on device : /dev/md7
exa1cel02: Upgrade mode : patch
exa1cel02: Zero auto rollback : enabled
exa1cel02: Auto rollback : enabled
exa1cel02: Reset SP : yes
exa1cel02: First boot device : usb
exa1cel02: Use usb on upgrade : no
exa1cel02: Last console in hd grub.conf : web
exa1cel02: Stop on zero boot : disabled
exa1cel02: Redirection log file :
exa1cel02:
exa1cel02: 2014-04-24 20:48:27 +0400 [INFO] Shut down MS before doing ipmitool commands.
exa1cel02:
exa1cel02: Stopping MS services...
exa1cel02: The SHUTDOWN of MS services was successful.
exa1cel02: 2014-04-24 20:48:35 +0400 [INFO] Service Processor (SP) is available
exa1cel02: 2014-04-24 20:48:35 +0400 [INFO] Start MS after doing ipmitool commands.
exa1cel02:
exa1cel02: Starting MS services...
exa1cel02: The STARTUP of MS services was successful.
exa1cel02:
exa1cel02: Stopping MS services...
exa1cel02: The SHUTDOWN of MS services was successful.
exa1cel02: 2014-04-24 20:48:50 +0400 [INFO] No running instance of checkdeveachboot found. Safe to proceed
exa1cel02: 2014-04-24 20:48:50 +0400 [INFO] Check healthy states of Bootable or/and MD device(s)
exa1cel02:
exa1cel02: /opt/oracle.cellos/validations/init.d/checkdeveachboot started at 2014-04-24 20:48:50
exa1cel02: Set default check mode
exa1cel02: Set lock /tmp/cellos/locks/32222.checkdeveachboot
exa1cel02: Check mode is in use
exa1cel02: Check system disks for the cell node
exa1cel02: lsi controller has disks that are mapped to: /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj
/dev/sdk /dev/sdl
exa1cel02: The current system disks with valid partition table: /dev/sda /dev/sdb
exa1cel02: System disk /dev/sda is a bootable device
exa1cel02: Partition /dev/sda1 has a proper boot signature
exa1cel02: System disk /dev/sdb is a bootable device
exa1cel02: Partition /dev/sdb1 has a proper boot signature
exa1cel02: Check for degraded or stopped md devices
exa1cel02: Check CELLBOOT USB
exa1cel02: Check CELLBOOT USB
exa1cel02: USB partition /dev/sdm1 has a valid boot signature for CELLBOOT usb
exa1cel02: USB device /dev/sdm is a bootable device
exa1cel02: USB partition /dev/sdm1 is mountable
exa1cel02: USB has version 11.2.3.2.1.130109
exa1cel02: Reset lock /tmp/cellos/locks/32222.checkdeveachboot
exa1cel02: 0:0:0:0
exa1cel02:
exa1cel02: Starting MS services...
exa1cel02: The STARTUP of MS services was successful.
exa1cel02: /root/_patch_hctap_/_p_/biosbootorder started at 2014-04-24_20:48:58
exa1cel02: Check and set BIOS boot order
exa1cel02: 2014-04-24 20:48:59 +0400 CELLBOOT USB partition: /dev/sdm1
exa1cel02: 2014-04-24 20:48:59 +0400 [INFO] CELLBOOT USB block device: /sys/block/sdm
exa1cel02: 2014-04-24 20:48:59 +0400 CELLBOOT USB vendor: ORACLE
exa1cel02: 2014-04-24 20:48:59 +0400 [INFO] CELLBOOT USB model: UNIGEN-UFD
exa1cel02: 2014-04-24 20:48:59 +0400 [INFO] Looking for CELLBOOT USB pattern "(USB:Port0:ORACLE|USB:.*ORACLE .*UNIGEN-UFD.*)" in BIOS boot order
exa1cel02: 2014-04-24 20:48:59 +0400 [INFO] Looking for RAID pattern "RAID:" in BIOS boot order
exa1cel02:
exa1cel02: Copyright (C) SUN Microsystems 2009.
exa1cel02: BIOSconfig Utility Version 2.2.1
exa1cel02: Build Date: Aug 24 2009
exa1cel02: Build Time: 09:01:30
exa1cel02:
exa1cel02: BIOSconfig Specification Version 2.4
exa1cel02:
exa1cel02: Processing Input BIOS Data....
exa1cel02:
exa1cel02: Success
exa1cel02: 2014-04-24 20:49:02 +0400 [INFO] The current BIOS boot order:
exa1cel02:
exa1cel02:
exa1cel02: USB:02.82;01 ORACLE UNIGEN-UFD
exa1cel02: 00,1a,07
exa1cel02:

exa1cel02:
exa1cel02:
exa1cel02: RAID:Slot0.F0:(Bus 13 Dev 00)PCI RAID Adapter
exa1cel02: 13,00,00
exa1cel02:

exa1cel02:
exa1cel02:
exa1cel02: PXE:IBA GE Slot 0100 v1331
exa1cel02: 01,00,00
exa1cel02:

exa1cel02:
exa1cel02: 2014-04-24 20:49:02 +0400 [INFO] CELLBOOT USB has proper order 1
exa1cel02:
exa1cel02: 2014-04-24 20:49:02 +0400 [INFO] RAID controller has proper order 2
exa1cel02:
exa1cel02: 2014-04-24 20:49:02 +0400 [INFO] Boot device order is correct.
exa1cel02: Cell exa1cel02 10.45.33.5
exa1cel02: _EXIT_PASS_Cell exa1cel02 10.45.33.5 2014-04-24 20:49:03 +0400: 2014-04-24 20:49:02 +0400 [INFO] Started to patch.
exa1cel03: 2014-04-24 20:48:21 +0400
exa1cel03: 2014-04-24 20:48:21 +0400 ./install.sh started with command line: -patch
exa1cel03: 2014-04-24 20:48:21 +0400 [INFO] Check md devices have clean state ...
exa1cel03: 2014-04-24 20:48:21 +0400 [INFO] Check disks and partitions ...
exa1cel03: Active image info
exa1cel03: Version : 11.2.3.2.1.130109
exa1cel03: System partition on device : /dev/md6
exa1cel03: Software partition on device : /dev/md8
exa1cel03: Cell boot usb version : 11.2.3.2.1.130109
exa1cel03:
exa1cel03: Patch proposal
exa1cel03: Version : 11.2.3.3.0.131014.1
exa1cel03: System partition on device : /dev/md5
exa1cel03: Software partition on device : /dev/md7
exa1cel03: Upgrade mode : patch
exa1cel03: Zero auto rollback : enabled
exa1cel03: Auto rollback : enabled
exa1cel03: Reset SP : yes
exa1cel03: First boot device : usb
exa1cel03: Use usb on upgrade : no
exa1cel03: Last console in hd grub.conf : web
exa1cel03: Stop on zero boot : disabled
exa1cel03: Redirection log file :
exa1cel03:
exa1cel03: 2014-04-24 20:48:27 +0400 [INFO] Shut down MS before doing ipmitool commands.
exa1cel03:
exa1cel03: Stopping MS services...
exa1cel03: The SHUTDOWN of MS services was successful.
exa1cel03: 2014-04-24 20:48:35 +0400 [INFO] Service Processor (SP) is available
exa1cel03: 2014-04-24 20:48:35 +0400 [INFO] Start MS after doing ipmitool commands.
exa1cel03:
exa1cel03: Starting MS services...
exa1cel03: The STARTUP of MS services was successful.
exa1cel03:
exa1cel03: Stopping MS services...
exa1cel03: The SHUTDOWN of MS services was successful.
exa1cel03: 2014-04-24 20:48:50 +0400 [INFO] No running instance of checkdeveachboot found. Safe to proceed
exa1cel03: 2014-04-24 20:48:50 +0400 [INFO] Check healthy states of Bootable or/and MD device(s)
exa1cel03:
exa1cel03: /opt/oracle.cellos/validations/init.d/checkdeveachboot started at 2014-04-24 20:48:50
exa1cel03: Set default check mode
exa1cel03: Set lock /tmp/cellos/locks/28294.checkdeveachboot
exa1cel03: Check mode is in use
exa1cel03: Check system disks for the cell node
exa1cel03: lsi controller has disks that are mapped to: /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj
/dev/sdk /dev/sdl
exa1cel03: The current system disks with valid partition table: /dev/sda /dev/sdb
exa1cel03: System disk /dev/sda is a bootable device
exa1cel03: Partition /dev/sda1 has a proper boot signature
exa1cel03: System disk /dev/sdb is a bootable device
exa1cel03: Partition /dev/sdb1 has a proper boot signature
exa1cel03: Check for degraded or stopped md devices
exa1cel03: Check CELLBOOT USB
exa1cel03: Check CELLBOOT USB
exa1cel03: USB partition /dev/sdm1 has a valid boot signature for CELLBOOT usb
exa1cel03: USB device /dev/sdm is a bootable device
exa1cel03: USB partition /dev/sdm1 is mountable
exa1cel03: USB has version 11.2.3.2.1.130109
exa1cel03: Reset lock /tmp/cellos/locks/28294.checkdeveachboot
exa1cel03: 0:0:0:0
exa1cel03:
exa1cel03: Starting MS services...
exa1cel03: The STARTUP of MS services was successful.
exa1cel03: /root/_patch_hctap_/_p_/biosbootorder started at 2014-04-24_20:48:59
exa1cel03: Check and set BIOS boot order
exa1cel03: 2014-04-24 20:48:59 +0400 CELLBOOT USB partition: /dev/sdm1
exa1cel03: 2014-04-24 20:48:59 +0400 [INFO] CELLBOOT USB block device: /sys/block/sdm
exa1cel03: 2014-04-24 20:48:59 +0400 CELLBOOT USB vendor: ORACLE
exa1cel03: 2014-04-24 20:48:59 +0400 [INFO] CELLBOOT USB model: UNIGEN-UFD
exa1cel03: 2014-04-24 20:48:59 +0400 [INFO] Looking for CELLBOOT USB pattern "(USB:Port0:ORACLE|USB:.*ORACLE .*UNIGEN-UFD.*)" in BIOS boot order
exa1cel03: 2014-04-24 20:48:59 +0400 [INFO] Looking for RAID pattern "RAID:" in BIOS boot order
exa1cel03:
exa1cel03: Copyright (C) SUN Microsystems 2009.
exa1cel03: BIOSconfig Utility Version 2.2.1
exa1cel03: Build Date: Aug 24 2009
exa1cel03: Build Time: 09:01:30
exa1cel03:
exa1cel03: BIOSconfig Specification Version 2.4
exa1cel03:
exa1cel03: Processing Input BIOS Data....
exa1cel03:
exa1cel03: Success
exa1cel03: 2014-04-24 20:49:02 +0400 [INFO] The current BIOS boot order:
exa1cel03:
exa1cel03:
exa1cel03: USB:02.82;01 ORACLE UNIGEN-UFD
exa1cel03: 00,1a,07
exa1cel03:

exa1cel03:
exa1cel03:
exa1cel03: RAID:Slot0.F0:(Bus 13 Dev 00)PCI RAID Adapter
exa1cel03: 13,00,00
exa1cel03:

exa1cel03:
exa1cel03:
exa1cel03: PXE:IBA GE Slot 0100 v1331
exa1cel03: 01,00,00
exa1cel03:

exa1cel03:
exa1cel03: 2014-04-24 20:49:02 +0400 [INFO] CELLBOOT USB has proper order 1
exa1cel03:
exa1cel03: 2014-04-24 20:49:03 +0400 [INFO] RAID controller has proper order 2
exa1cel03:
exa1cel03: 2014-04-24 20:49:03 +0400 [INFO] Boot device order is correct.
exa1cel03: Cell exa1cel03 10.45.33.6
exa1cel03: _EXIT_PASS_Cell exa1cel03 10.45.33.6 2014-04-24 20:49:03 +0400: 2014-04-24 20:49:03 +0400 [INFO] Started to patch.
2014-04-24 20:49:04 +0400 ++++++++++++++++++ Logs so far end ++++++++++
2014-04-24 20:49:04 +0400 :SUCCESS: DONE: Check prerequisites on all cells.
2014-04-24 20:49:04 +0400 :Working: DO: Execute plugin check for Patch Check Prereq ...
2014-04-24 20:49:04 +0400 :WARNING: patchmgr plugin /root/patch_11.2.3.3.0.131014.1/plugins/001-post_11_2_3_3_0.
2014-04-24 20:49:04 +0400 :INFO: patch stage:Patch Check Prereq.
2014-04-24 20:49:04 +0400 :INFO: cells:exa1cel01,exa1cel02,exa1cel03.
2014-04-24 20:49:04 +0400 :INFO: patch action:patch_prereq.
2014-04-24 20:49:04 +0400 :INFO: rolling mode:non-rolling.
2014-04-24 20:49:04 +0400 :INFO: label image version:11.2.3.3.0.131014.1.
2014-04-24 20:49:04 +0400 :INFO: patchmgr log:/root/patch_11.2.3.3.0.131014.1/patchmgr.stdout.
2014-04-24 20:49:04 +0400 ++++++++++++++++++ Logs so far begin ++++++++++
2014-04-24 20:49:05 +0400 ++++++++++++++++++ Logs so far end ++++++++++
2014-04-24 20:49:05 +0400 :SUCCESS: DONE: Execute plugin check for Patch Check Prereq.
================PatchMgr run ended 2014-04-24 20:49:05 +0400 ===========

So, we can start patching:
./patchmgr -cells cell_group -patch

Wait until it finishes!

At the end check the image version:

[root@exa1cel01 ~]# imageinfo

Kernel version: 2.6.39-400.126.1.el5uek #1 SMP Fri Sep 20 10:54:38 PDT 2013 x86_64
Cell version: OSS_11.2.3.3.0_LINUX.X64_131014.1
Cell rpm version: cell-11.2.3.3.0_LINUX.X64_131014.1-1

Active image version: 11.2.3.3.0.131014.1
Active image activated: 2014-04-24 22:09:25 +0400
Active image status: success
Active system partition on device: /dev/md5
Active software partition on device: /dev/md7

In partition rollback: Impossible

Cell boot usb partition: /dev/sdm1
Cell boot usb version: 11.2.3.3.0.131014.1

Inactive image version: 11.2.3.2.1.130109
Inactive image activated: 2013-05-25 10:41:59 +0400
Inactive image status: success
Inactive system partition on device: /dev/md6
Inactive software partition on device: /dev/md8

Boot area has rollback archive for the version: 11.2.3.2.1.130109
Rollback to the inactive partitions: Possible
[root@exa1cel01 ~]# imagehistory
Version : 11.2.3.1.1.120607
Image activation date : 2012-08-30 12:58:48 +0400
Imaging mode : fresh
Imaging status : success

Version : 11.2.3.2.1.130109
Image activation date : 2013-05-25 10:41:59 +0400
Imaging mode : out of partition upgrade
Imaging status : success

Version : 11.2.3.3.0.131014.1
Image activation date : 2014-04-24 22:09:25 +0400
Imaging mode : out of partition upgrade
Imaging status : success

[root@exa1cel01 ~]#


5.3 Updating Database Servers on Oracle Exadata Database Machine

Copy p16486998_121110_Linux-x86-64.zip to /root/ and unzip it:

[root@exa101 ~]# unzip p16486998_121110_Linux-x86-64.zip
Archive: p16486998_121110_Linux-x86-64.zip
inflating: dbnodeupdate.sh
inflating: dbupdate-helpers.zip
inflating: README.txt


[root@exa101 ~]# ./dbnodeupdate.sh -u -l /u01/tmpstage/17816100/Infrastructure/ExadataStorageServer/11.2.3.3.0/p17809253_112330_Linux-x86-64.zip
(*) 2014-04-24 22:39:42: Unzipping helpers (/root/dbupdate-helpers.zip) to /opt/oracle.SupportTools/dbnodeupdate_helpers
(*) 2014-04-24 22:39:42: Initializing logfile /var/log/cellos/dbnodeupdate.log
(*) 2014-04-24 22:39:42: Collecting system configuration details, this may take some time...
(*) 2014-04-24 22:39:51: Validating system details (known issues, best practices). This may take a while
(*) 2014-04-24 22:39:51: Checking free space in /u01
(*) 2014-04-24 22:39:51: Unzipping /u01/tmpstage/17816100/Infrastructure/ExadataStorageServer/11.2.3.3.0/p17809253_112330_Linux-x86-64.zip to
/u01/app/oracle/stage.240414223942, this may take a while
(*) 2014-04-24 22:39:59: Backing up /etc/yum.repos.d/Exadata-computenode.repo in /etc/yum.repos.d.bak/240414223942
(*) 2014-04-24 22:39:59: Backing up /etc/yum.repos.d/Exadata-computenode.repo.sample in /etc/yum.repos.d.bak/240414223942
(*) 2014-04-24 22:39:59: Backing up /etc/yum.repos.d/Exadata-computenode.repo.sample.backup in /etc/yum.repos.d.bak/240414223942

Warning: Network routing configuration requires change before updating database server. See MOS 1306154.1

Continue ? [y/N]
y

Active Image version : 11.2.3.2.1.130302
Active Kernel version : 2.6.18-308.24.1.0.1.el5
Active LVM Name : /dev/mapper/VGExaDb-LVDbSys1
Inactive Image version : 11.2.3.1.1.120607
Inactive LVM Name : /dev/mapper/VGExaDb-LVDbSys2
Current user id : root
Action : upgrade
Upgrading to : 11.2.3.3.0.131014.1
Baseurl : file:///var/www/html/yum/unknown/EXADATA/dbserver/240414223942/x86_64/ (iso)
Iso file : /u01/app/oracle/stage.240414223942/repoimage.iso
Create a backup : Yes
Shutdown stack : No (Currently stack is down)
RPM exclusion list : Not in use (add rpms to /etc/exadata/yum/exclusion.lst and restart dbnodeupdate.sh)
RPM obsolete list : /etc/exadata/yum/obsolete.lst (available post-update, lists rpms removed by the update)
Logfile : /var/log/cellos/dbnodeupdate.log (runid: 240414223942)
Diagfile : /var/log/cellos/dbnodeupdate.240414223942.diag
Server model : SUN FIRE X4170 M2 SERVER
dbnodeupdate.sh rel. : 2.39 (always check MOS 1553103.1 for the latest release of dbnodeupdate)
Automatic checks incl. : Known Issue - Database Server upgrades may hit network routing issues after the upgrade
: Issue - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
: Issue 1.7 - Updating database servers with customized partitions may remove partitions already in use
Note : After upgrading and rebooting run './dbnodeupdate.sh -c' to finish post steps


Continue ? [Y/n]
y
(*) 2014-04-24 22:40:34: Verifying GI and DB's are shutdown
(*) 2014-04-24 22:40:36: Collecting console history for diag purposes
(*) 2014-04-24 22:40:51: Unmount of /boot successful
(*) 2014-04-24 22:40:51: Check for /dev/sda1 successful
(*) 2014-04-24 22:40:51: Mount of /boot successful
(*) 2014-04-24 22:40:51: Disabling stack from starting
(*) 2014-04-24 22:40:51: Performing filesystem backup to /dev/mapper/VGExaDb-LVDbSys2. Avg. 30 minutes (max. 120) depends per
environment.........
(*) 2014-04-24 22:48:52: Backup successful
(*) 2014-04-24 22:48:52: Verifying and updating yum.conf (backup in /etc/yum.conf.240414_223942)
(*) 2014-04-24 22:48:52: Disabling other repositories, generating Exadata repos
(*) 2014-04-24 22:48:52: Generating /etc/yum.repos.d/Exadata-computenode.repo
(*) 2014-04-24 22:48:53: Verifying baseurl
(*) 2014-04-24 22:48:54: OSWatcher stopped successful
(*) 2014-04-24 22:48:54: Cleaning up the yum cache
(*) 2014-04-24 22:48:54: Removing rpm libcxgb3-static.x86_64 (if installed)
(*) 2014-04-24 22:48:54: Removing rpm rpm-build.x86_64 (if installed)
(*) 2014-04-24 22:48:54: Preparing update for releases 11.2.3.3.0 and later
(*) 2014-04-24 22:48:59: Preparation done succesfully
(*) 2014-04-24 22:48:59: Drive /dev/sda has more than two partitions. This is non-standard. If hotspare still exists it will not be reclaimed
(*) 2014-04-24 22:48:59: Performing yum update. Node is expected to reboot when finished
(*) 2014-04-24 22:50:56: Waiting for post rpm script to finish. Sleeping another 60 seconds (60 / 900)

Remote broadcast message (Thu Apr 24 22:51:08 2014):

Exadata post install steps started.
It may take up to 5 minutes.
(*) 2014-04-24 22:51:57: Waiting for post rpm script to finish. Sleeping another 60 seconds (120 / 900)

Remote broadcast message (Thu Apr 24 22:52:24 2014):

Exadata post install steps completed with success
(*) 2014-04-24 22:52:57: Waiting for post rpm script to finish. Sleeping another 60 seconds (180 / 900)
(*) 2014-04-24 22:53:57: All post steps are finished.
(*) 2014-04-24 22:53:57: System will reboot automatically for changes to take effect
(*) 2014-04-24 22:53:57: After reboot run "./dbnodeupdate.sh -c" to complete the upgrade
(*) 2014-04-24 22:53:57: Restoring /etc/yum.repos.d.bak/240414223942/Exadata-computenode.repo to /etc/yum.repos.d
(*) 2014-04-24 22:53:57: Restoring /etc/yum.repos.d.bak/240414223942/Exadata-computenode.repo.sample to /etc/yum.repos.d
(*) 2014-04-24 22:53:57: Restoring /etc/yum.repos.d.bak/240414223942/Exadata-computenode.repo.sample.backup to /etc/yum.repos.d
(*) 2014-04-24 22:54:01: Rebooting now...

Broadcast message from root (pts/2) (Thu Apr 24 22:54:01 2014):

The system is going down for reboot NOW!

Post installation:

[root@exa101 ~]# ./dbnodeupdate.sh -c
(*) 2014-04-24 23:28:48: Unzipping helpers (/root/dbupdate-helpers.zip) to /opt/oracle.SupportTools/dbnodeupdate_helpers
(*) 2014-04-24 23:28:48: Initializing logfile /var/log/cellos/dbnodeupdate.log
(*) 2014-04-24 23:28:49: Collecting system configuration details, this may take some time...

Active Image version : 11.2.3.3.0.131014.1
Active Kernel version : 2.6.39-400.126.1.el5uek
Active LVM Name : /dev/mapper/VGExaDb-LVDbSys1
Inactive Image version : 11.2.3.2.1.130302
Inactive LVM Name : /dev/mapper/VGExaDb-LVDbSys2
Current user id : root
Action : finish-post (perform post steps, relink enable/disable crs)
Shutdown stack : No (Currently stack is down)
Logfile : /var/log/cellos/dbnodeupdate.log (runid: 240414232848)
Diagfile : /var/log/cellos/dbnodeupdate.240414232848.diag
Server model : SUN FIRE X4170 M2 SERVER
dbnodeupdate.sh rel. : 2.39 (always check MOS 1553103.1 for the latest release of dbnodeupdate)
: Issue - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
: Issue 1.7 - Updating database servers with customized partitions may remove partitions already in use

Continue ? [Y/n]
y
(*) 2014-04-24 23:29:19: Verifying GI and DB's are shutdown
(*) 2014-04-24 23:29:21: Verifying firmware updates/validations. Maximum wait time: 30 minutes.
(*) 2014-04-24 23:29:21: If the node reboots during this execution, re-run './dbnodeupdate.sh -c' after the node restarts..
(*) 2014-04-24 23:29:21: Collecting console history for diag purposes
(*) 2014-04-24 23:30:06: ExaWatcher stopped successful
(*) 2014-04-24 23:30:06: Fix for 17885493 applied successful
(*) 2014-04-24 23:30:06: Starting ExaWatcher
(*) 2014-04-24 23:30:06: Relinking all homes
(*) 2014-04-24 23:30:06: Unlocking /u01/app/11.2.0.3/grid
(*) 2014-04-24 23:30:12: Relinking /u01/app/11.2.0.3/grid as grid (with rds option)
(*) 2014-04-24 23:30:24: Skipping /u01/app/oracle/agent11g/agent11g
(*) 2014-04-24 23:30:24: Relinking /u01/app/oracle/product/11.2.0.3/dbhome_1 as oracle (with rds option)
(*) 2014-04-24 23:30:44: Executing /u01/app/11.2.0.3/grid/crs/install/rootcrs.pl -patch
(*) 2014-04-24 23:31:45: Stack started
(*) 2014-04-24 23:31:45: Enabling stack to start at reboot. Disable this when the stack should not be starting on a next boot
(*) 2014-04-24 23:31:45: All post steps are finished.

Finished 1st DB Node upgrading. Do the same in remaining Nodes.


5.4 Infiniband switch upgrading:

Prepare ibswitches.lst file with the name of 1st switch:

[root@exa101 patch_11.2.3.3.0.131014.1]# cat ibswitches.lst
exa1sw-ib3

[root@exa101 patch_11.2.3.3.0.131014.1]# ./patchmgr -ibswitches ibswitches.lst -upgrade -ibswitch_precheck

2014-04-25 00:12:13 +0400 1 of 1 :SUCCESS: DO: Initiate pre-upgrade validation check on InfiniBand switch(es).
----- InfiniBand switch update process started Fri Apr 25 00:12:13 GET 2014 -----
[NOTE ] Log file at /var/log/cellos/upgradeIBSwitch.log

[INFO ] List of InfiniBand switches for upgrade: ( exa1sw-ib3 )
[PROMPT ] Use the default password for all switches? (y/n) [n]: y
[PROMPT ] Mismatch in the list of switches provided ( exa1sw-ib3 ) and the output of the command ibswitches ( exa1sw-ib3 exa1sw-ib2 ). Are you
sure you want to continue? (y/n) [n]: y
[PROMPT ] Updating only 1 switch(es). Are you sure you want to continue? (y/n) [n]: y
[SUCCESS ] Verifying Network connectivity to exa1sw-ib3
[SUCCESS ] Validating verify-topology output
[INFO ] Master Subnet Manager is set to exa1sw-ib3 in all Switches
[PROMPT ] Found 1 InfiniBand switches with subnet manager running. Are you sure you want to continue? (y/n) [n]: y
[INFO ] Found 1 InfiniBand switches with subnet manager running. Continuing as per user request.

[INFO ] ---------- Starting with IBSwitch exa1sw-ib3
[SUCCESS ] exa1sw-ib3 is at 1.3.3-2. Meets minimal patching level 1.3.3-2
[SUCCESS ] Verifying that /tmp has 120M in exa1sw-ib3, found 249M
[SUCCESS ] Verifying that / has 80M in exa1sw-ib3, found 200M
[SUCCESS ] Verifying that exa1sw-ib3 has 120M free memory, found 411M
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 for
exa1sw-ib3
[SUCCESS ] Verifying that exa1sw-ib3 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2014-04-25 Time:(HH:MM:SS) 04:24:36
[SUCCESS ] Pre-update validation on exa1sw-ib3
[SUCCESS ] Overall status

----- InfiniBand switch update process ended Fri Apr 25 00:13:20 GET 2014 -----
2014-04-25 00:13:20 +0400 1 of 1 :SUCCESS: DONE: Initiate pre-upgrade validation check on InfiniBand switch(es).

[root@exa101 patch_11.2.3.3.0.131014.1]# ./patchmgr -ibswitches ibswitches.lst -upgrade

2014-04-25 00:13:38 +0400 1 of 1 :SUCCESS: DO: Initiate upgrade of InfiniBand switches to 2.1.3-4. Expect up to 15 minutes for each switch
----- InfiniBand switch update process started Fri Apr 25 00:13:38 GET 2014 -----
[NOTE ] Log file at /var/log/cellos/upgradeIBSwitch.log

[INFO ] List of InfiniBand switches for upgrade: ( exa1sw-ib3 )
[PROMPT ] Use the default password for all switches? (y/n) [n]: y
[PROMPT ] Mismatch in the list of switches provided ( exa1sw-ib3 ) and the output of the command ibswitches ( exa1sw-ib3 exa1sw-ib2 ). Are you
sure you want to continue? (y/n) [n]: y
[PROMPT ] Updating only 1 switch(es). Are you sure you want to continue? (y/n) [n]: y
[SUCCESS ] Verifying Network connectivity to exa1sw-ib3
[SUCCESS ] Validating verify-topology output
[INFO ] Proceeding with upgrade of InfiniBand switches to version 2.1.3_4
[INFO ] Master Subnet Manager is set to exa1sw-ib3 in all Switches
[PROMPT ] Found 1 InfiniBand switches with subnet manager running. Are you sure you want to continue? (y/n) [n]: y
[INFO ] Found 1 InfiniBand switches with subnet manager running. Continuing as per user request.

[INFO ] ---------- Starting with IBSwitch exa1sw-ib3
[SUCCESS ] Disable Subnet Manager on exa1sw-ib3
[SUCCESS ] Copy firmware packages to exa1sw-ib3
[SUCCESS ] exa1sw-ib3 is at 1.3.3-2. Meets minimal patching level 1.3.3-2
[SUCCESS ] Verifying that /tmp has 120M in exa1sw-ib3, found 138M
[SUCCESS ] Verifying that / has 80M in exa1sw-ib3, found 200M
[SUCCESS ] Verifying that exa1sw-ib3 has 120M free memory, found 301M
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 for
exa1sw-ib3
[SUCCESS ] Verifying that exa1sw-ib3 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2014-04-25 Time:(HH:MM:SS) 04:25:51
[SUCCESS ] Pre-update validation on exa1sw-ib3
[INFO ] Starting upgrade on exa1sw-ib3 to 2.1.3_4. Please give upto 10 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C
during the upgrade
[SUCCESS ] Load firmware 2.1.3_4 onto exa1sw-ib3
[SUCCESS ] Disable Subnet Manager on exa1sw-ib3
[SUCCESS ] Verify that /conf/configvalid is set to 1 on exa1sw-ib3
[SUCCESS ] Set SMPriority to 5 on exa1sw-ib3
[INFO ] Rebooting exa1sw-ib3. Wait for 240 secs before continuing
[SUCCESS ] Reboot exa1sw-ib3
[SUCCESS ] Restart Subnet Manager on exa1sw-ib3
[INFO ] Starting post-update validation on exa1sw-ib3
[SUCCESS ] Inifiniband switch exa1sw-ib3 is at target patching level
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 for
exa1sw-ib3
[SUCCESS ] Verifying that exa1sw-ib3 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2014-04-25 Time:(HH:MM:SS) 04:39:02
[SUCCESS ] Firmware verification on InfiniBand switch exa1sw-ib3
[INFO ] Post-check validation on IBSwitch exa1sw-ib3
[SUCCESS ] Update switch exa1sw-ib3 to 2.1.3_4
[INFO ] InfiniBand Switches ( exa1sw-ib3 ) updated to 2.1.3_4
[SUCCESS ] Overall status

----- InfiniBand switch update process ended Fri Apr 25 00:31:59 GET 2014 -----
2014-04-25 00:31:59 +0400 1 of 1 :SUCCESS: DONE: Upgrade InfiniBand switch(es) to 2.1.3-4.

Edit ibswitches.lst file and replace with 2nd switch's name:

[root@exa101 patch_11.2.3.3.0.131014.1]# cat ibswitches.lst
exa1sw-ib2


[root@exa101 patch_11.2.3.3.0.131014.1]# ./patchmgr -ibswitches ibswitches.lst -upgrade -ibswitch_precheck

2014-04-25 00:33:44 +0400 1 of 1 :SUCCESS: DO: Initiate pre-upgrade validation check on InfiniBand switch(es).
----- InfiniBand switch update process started Fri Apr 25 00:33:45 GET 2014 -----
[NOTE ] Log file at /var/log/cellos/upgradeIBSwitch.log

[INFO ] List of InfiniBand switches for upgrade: ( exa1sw-ib2 )
[PROMPT ] Use the default password for all switches? (y/n) [n]: y
[PROMPT ] Mismatch in the list of switches provided ( exa1sw-ib2 ) and the output of the command ibswitches ( exa1sw-ib3 exa1sw-ib2 ). Are you
sure you want to continue? (y/n) [n]: y
[PROMPT ] Updating only 1 switch(es). Are you sure you want to continue? (y/n) [n]: y
[SUCCESS ] Verifying Network connectivity to exa1sw-ib2
[SUCCESS ] Validating verify-topology output
[INFO ] Master Subnet Manager is set to exa1sw-ib2 in all Switches
[PROMPT ] Found 1 InfiniBand switches with subnet manager running. Are you sure you want to continue? (y/n) [n]: y
[INFO ] Found 1 InfiniBand switches with subnet manager running. Continuing as per user request.

[INFO ] ---------- Starting with IBSwitch exa1sw-ib2
[SUCCESS ] exa1sw-ib2 is at 1.3.3-2. Meets minimal patching level 1.3.3-2
[SUCCESS ] Verifying that /tmp has 120M in exa1sw-ib2, found 249M
[SUCCESS ] Verifying that / has 80M in exa1sw-ib2, found 199M
[SUCCESS ] Verifying that exa1sw-ib2 has 120M free memory, found 411M
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 for
exa1sw-ib2
[SUCCESS ] Verifying that exa1sw-ib2 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2014-04-25 Time:(HH:MM:SS) 00:34:10
[SUCCESS ] Pre-update validation on exa1sw-ib2
[SUCCESS ] Overall status

----- InfiniBand switch update process ended Fri Apr 25 00:34:11 GET 2014 -----
2014-04-25 00:34:11 +0400 1 of 1 :SUCCESS: DONE: Initiate pre-upgrade validation check on InfiniBand switch(es).

[root@exa101 patch_11.2.3.3.0.131014.1]# ./patchmgr -ibswitches ibswitches.lst -upgrade

2014-04-25 00:34:20 +0400 1 of 1 :SUCCESS: DO: Initiate upgrade of InfiniBand switches to 2.1.3-4. Expect up to 15 minutes for each switch
----- InfiniBand switch update process started Fri Apr 25 00:34:21 GET 2014 -----
[NOTE ] Log file at /var/log/cellos/upgradeIBSwitch.log

[INFO ] List of InfiniBand switches for upgrade: ( exa1sw-ib2 )
[PROMPT ] Use the default password for all switches? (y/n) [n]: y
[PROMPT ] Mismatch in the list of switches provided ( exa1sw-ib2 ) and the output of the command ibswitches ( exa1sw-ib3 exa1sw-ib2 ). Are you
sure you want to continue? (y/n) [n]: y
[PROMPT ] Updating only 1 switch(es). Are you sure you want to continue? (y/n) [n]: y
[SUCCESS ] Verifying Network connectivity to exa1sw-ib2
[SUCCESS ] Validating verify-topology output
[INFO ] Proceeding with upgrade of InfiniBand switches to version 2.1.3_4
[INFO ] Master Subnet Manager is set to exa1sw-ib2 in all Switches
[PROMPT ] Found 1 InfiniBand switches with subnet manager running. Are you sure you want to continue? (y/n) [n]: y
[INFO ] Found 1 InfiniBand switches with subnet manager running. Continuing as per user request.

[INFO ] ---------- Starting with IBSwitch exa1sw-ib2
[SUCCESS ] Disable Subnet Manager on exa1sw-ib2
[SUCCESS ] Copy firmware packages to exa1sw-ib2
[SUCCESS ] exa1sw-ib2 is at 1.3.3-2. Meets minimal patching level 1.3.3-2
[SUCCESS ] Verifying that /tmp has 120M in exa1sw-ib2, found 138M
[SUCCESS ] Verifying that / has 80M in exa1sw-ib2, found 199M
[SUCCESS ] Verifying that exa1sw-ib2 has 120M free memory, found 302M
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 for
exa1sw-ib2
[SUCCESS ] Verifying that exa1sw-ib2 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2014-04-25 Time:(HH:MM:SS) 00:35:14
[SUCCESS ] Pre-update validation on exa1sw-ib2
[INFO ] Starting upgrade on exa1sw-ib2 to 2.1.3_4. Please give upto 10 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C
during the upgrade
[SUCCESS ] Load firmware 2.1.3_4 onto exa1sw-ib2
[SUCCESS ] Disable Subnet Manager on exa1sw-ib2
[SUCCESS ] Verify that /conf/configvalid is set to 1 on exa1sw-ib2
[SUCCESS ] Set SMPriority to 5 on exa1sw-ib2
[INFO ] Rebooting exa1sw-ib2. Wait for 240 secs before continuing
[SUCCESS ] Reboot exa1sw-ib2
[SUCCESS ] Restart Subnet Manager on exa1sw-ib2
[INFO ] Starting post-update validation on exa1sw-ib2
[SUCCESS ] Inifiniband switch exa1sw-ib2 is at target patching level
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 for
exa1sw-ib2
[SUCCESS ] Verifying that exa1sw-ib2 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2014-04-25 Time:(HH:MM:SS) 00:48:18
[SUCCESS ] Firmware verification on InfiniBand switch exa1sw-ib2
[INFO ] Post-check validation on IBSwitch exa1sw-ib2
[SUCCESS ] Update switch exa1sw-ib2 to 2.1.3_4
[INFO ] InfiniBand Switches ( exa1sw-ib2 ) updated to 2.1.3_4
[SUCCESS ] Overall status

----- InfiniBand switch update process ended Fri Apr 25 00:52:30 GET 2014 -----
2014-04-25 00:52:30 +0400 1 of 1 :SUCCESS: DONE: Upgrade InfiniBand switch(es) to 2.1.3-4.

at the end, picture of upgraded EXADATAs:



That is All, Good Luck!

Wednesday, April 23, 2014

Never hurry up for telling "It is a BUG in Oracle"!

Today I was upgrading(software of all of components, except DB and GI) Exadata X2 to new version. First I did standby site, so we had to simulate some tests to be sure everything is OK as expected. One of them was to create a new tablespace at primary site (which is in older version) and check if media recovery adds a new datafile at standby site. We are using Oracle Managed Files. Statement was as below:
SQL> CREATE TABLESPACE TEST_DG_1 DATAFILE SIZE 10M AUTOEXTEND ON NEXT 10M MAXSIZE UNLIMITED,
  2  LOGGING ONLINE PERMANENT BLOCKSIZE 8192
  3  EXTENT MANAGEMENT LOCAL DEFAULT  
  4  NOCOMPRESS  SEGMENT SPACE MANAGEMENT MANUAL
  5  /

Tablespace created.
In alert log of standby, we see:
Successfully added datafile 27 to media recovery
Datafile #27: '+DATA_EXA2/xlst/datafile/test_dg_1.624.845671089'
Successfully added datafile 28 to media recovery
Datafile #28: '+DATA_EXA2/xlst/datafile/test_dg_1.633.845671089'
Oops? Why 2 datafiles instead of 1? Customer's 1st opinion: We hit a BUG! :) OK, it is expected reaction :) Lets test: On primary:
SQL> select file_name from dba_data_files where tablespace_name = 'TEST_DG_1';

FILE_NAME
--------------------------------------------------------------------------------
+DATA_EXA1/xldr/datafile/test_dg_1.596.845671087
+DATA_EXA1/xldr/datafile/test_dg_1.602.845671087
Interesting... Lets get Metadata of newly created tablespace:
SQL> set long 1000
SQL> /

DBMS_METADATA.GET_DDL('TABLESPACE','TEST_DG_1')
--------------------------------------------------------------------------------

  CREATE TABLESPACE "TEST_DG_1" DATAFILE
  SIZE 10485760
  AUTOEXTEND ON NEXT 10485760 MAXSIZE 32767M,
  SIZE 104857600
  AUTOEXTEND ON NEXT 104857600 MAXSIZE 32767M
  LOGGING ONLINE PERMANENT BLOCKSIZE 8192
  EXTENT MANAGEMENT LOCAL AUTOALLOCATE DEFAULT
 NOCOMPRESS  SEGMENT SPACE MANAGEMENT MANUAL
2 AUTOEXTEND options... As we know, AUTOEXTEND is for datafile not for tablespace. Suddenly, I realized ',' symbol after 'UNLIMITED' keyword in statement. Removing it and creating again:
SQL> CREATE TABLESPACE TEST_DG_1 DATAFILE SIZE 10M AUTOEXTEND ON NEXT 10M MAXSIZE UNLIMITED
  2  LOGGING ONLINE PERMANENT BLOCKSIZE 8192
  3  EXTENT MANAGEMENT LOCAL DEFAULT  
  4  NOCOMPRESS  SEGMENT SPACE MANAGEMENT MANUAL;

Tablespace created.

SQL> select file_name from dba_data_files where tablespace_name = 'TEST_DG_1';

FILE_NAME
--------------------------------------------------------------------------------
+DATA_EXA1/xldr/datafile/test_dg_1.602.845671459

Yes, now only 1 datafile is created as expected.

For testing, adding many ',' symbols as below:



SQL> CREATE TABLESPACE TEST_DG_2 DATAFILE SIZE 10M , ,;

Tablespace created.

SQL> select file_name from dba_data_files where tablespace_name = 'TEST_DG_2';

FILE_NAME
--------------------------------------------------------------------------------
+DATA_EXA1/xldr/datafile/test_dg_2.596.845671511
+DATA_EXA1/xldr/datafile/test_dg_2.281.845671511
+DATA_EXA1/xldr/datafile/test_dg_2.465.845671511
As you see, What parser understands here: 'DATAFILE' clause is for the 1st datafile, if it is followed by ',' then it means another DATAFILE and so on. Checked Support and met with: Bug 11822439 - DBMS_METADATA.GET_DDL for tablespace shows multiple autoextend clauses (Doc ID 11822439.8) It is very similiar to our case, but in our case it is not BUG of DBMS_METADATA, because DBMS_METADATA told us what really we have (how many datafiles, options etc). Bug of statement parser? Lets execute the statement which is generated by DBMS_METADA:
SQL> CREATE TABLESPACE "TEST_DG_1" DATAFILE
  2  SIZE 10485760
  3  AUTOEXTEND ON NEXT 10485760 MAXSIZE 32767M,
  4  SIZE 104857600
  5  AUTOEXTEND ON NEXT 104857600 MAXSIZE 32767M
  6  LOGGING ONLINE PERMANENT BLOCKSIZE 8192
  7  EXTENT MANAGEMENT LOCAL AUTOALLOCATE DEFAULT
  8  NOCOMPRESS  SEGMENT SPACE MANAGEMENT MANUAL
  9  /

Tablespace created.
So, logically it means it is designed and working as expected :) But for to be 100% sure, I will raise a BUG! Good luck!

Monday, February 10, 2014

Oracle Database 12c: What happens during CDB startup if one the PDB has a media problem?

During Oracle Day 2014, which held in Baku/Azerbaijan, my friend Mahir Quluzade told me that, according his test if there is media problem in one of the PDB and if we want to startup CDB, CDB will not start and it is completely NOT possible to start! Interesting, although I did not tested, I was sure it is by design and there should be workaround. Ok, I got free time today and tested:
SQL> select name, open_mode from v$pdbs;

NAME                           OPEN_MODE
------------------------------ ----------
PDB$SEED                       READ ONLY
PDB1                           READ WRITE
PDB2                           READ WRITE


SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
/u01/app/oracle/oradata/testdb/system01.dbf
/u01/app/oracle/oradata/testdb/sysaux01.dbf
/u01/app/oracle/oradata/testdb/undotbs01.dbf
/u01/app/oracle/oradata/testdb/pdbseed/system01.dbf
/u01/app/oracle/oradata/testdb/users01.dbf
/u01/app/oracle/oradata/testdb/pdbseed/sysaux01.dbf
/u01/app/oracle/oradata/testdb/pdb1/system01.dbf
/u01/app/oracle/oradata/testdb/pdb1/sysaux01.dbf
/u01/app/oracle/oradata/testdb/pdb1/pdb1_users01.dbf
/u01/app/oracle/oradata/testdb/pdb2/system01.dbf
/u01/app/oracle/oradata/testdb/pdb2/sysaux01.dbf
/u01/app/oracle/oradata/testdb/pdb2/pdb2_users01.dbf

12 rows selected.
Now, removing datafile from USERS tablespace in PDB1:
SQL> !rm /u01/app/oracle/oradata/testdb/pdb1/pdb1_users01.dbf

SQL> shutdown immediate
ORA-03113: end-of-file on communication channel
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
[oracle@12cdb ~]$ sqlplus "/ as sysdba"

SQL*Plus: Release 12.1.0.1.0 Production on Sat Feb 8 22:30:53 2014

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 2137886720 bytes
Fixed Size                  2290416 bytes
Variable Size             603983120 bytes
Database Buffers         1526726656 bytes
Redo Buffers                4886528 bytes
Database mounted.
ORA-01157: cannot identify/lock data file 10 - see DBWR trace file
ORA-01110: data file 10: '/u01/app/oracle/oradata/testdb/pdb1/pdb1_users01.dbf'

So, CDB does not start, but it is not permanent situation, just connect into PDB1 and make that datafile offline:
SQL> alter session set container=PDB1;

Session altered.

SQL> alter database datafile '/u01/app/oracle/oradata/testdb/pdb1/pdb1_users01.dbf' offline;

Database altered.


SQL> alter database open;

Database altered.

SQL> select name, open_mode from v$pdbs;

NAME                           OPEN_MODE
------------------------------ ----------
PDB$SEED                       READ ONLY
PDB1                           MOUNTED
PDB2                           MOUNTED


SQL> alter pluggable database all open;

Pluggable database altered.

SQL> select name, open_mode from v$pdbs;

NAME                           OPEN_MODE
------------------------------ ----------
PDB$SEED                       READ ONLY
PDB1                           READ WRITE
PDB2                           READ WRITE
Yes, it is working. It would be not logical, if we could not open other PDBs in the same CDB because of having problem in 1 PDB only. If you apply PSU1 on 12c, you will not meet with such problem, CDB will skip opening problematic PDB and will open other PDBs.

Wednesday, July 24, 2013

Oracle 12 High Availability New Features Series: 1 - Partial Global/Local Indexes

Since its first introduction in Oracle 8.0 in 1997, Oracle has enhanced the functionality of Oracle
Partitioning with every release, by either adding new partitioning techniques, enhancing the scalability,
or extending the manageability and maintenance capabilities. Oracle Database 12c is no different by
offering enhanced composite partitioning strategies and vastly enhanced partition maintenance
operations.

Oracle 12c now provides the ability to index a subset of partitions and to exclude the others.

Local and global indexes can now be created on a subset of the partitions of a table. Partial Global indexes provide more flexibility in index creation for partitioned tables. For example, index segments can be omitted for the most recent partitions to ensure maximum data ingest rates without impacting the overall data model and access for the partitioned object.

Partial Global Indexes save space and improve performance during loads and queries. This feature supports global indexes that include or index a certain subset of table partitions or subpartitions, and exclude the others. This operation is supported using a default table indexing property. When a table is created or altered, a default indexing property can be specified for the table or its partitions.

The CREATE TABLE syntax is extended to support specification of the indexing attributes clause, which is as follows:
[INDEXING { ON | OFF }]

The indexing property determines whether table partitions and subpartitions are included in partial indexes on the table. The default is INDEXING ON.

When you specify the INDEXING clause at the table level, in the table properties clause, you set the default indexing property for the table.

The INDEXING clause may also be specified at the partition and subpartition levels (if ommitted partitions/subpartitions inherit the default indexing property of the table).

The CREATE INDEX syntax is extended to support specification of the FULL or PARTIAL index.

Specify INDEXING FULL to create a full index. In this case the index is decoupled from the table properties. A full index includes all partitions in the underlying table, regardless of their indexing properties.

Specify INDEXING PARTIAL to create a partial index. A partial index includes only partitions in the underlying table with an indexing property of ON.

The default is FULL, if neither FULL or PARTIAL is specified, i.e. Oracle will index all partitions (similar to the behavior of Oracle Database releases prior to Oracle Database 12c.).

CREATE TABLE myt1
(
c1 NUMBER(4),
c2 DATE,
c3 NUMBER(8)
)
INDEXING OFF
PARTITION BY RANGE (c2)
(
PARTITION p1 VALUES LESS THAN (TO_DATE('01-APR-2009','DD-MON-YYYY')) INDEXING ON,
PARTITION p2 VALUES LESS THAN (TO_DATE('01-JUL-2010','DD-MON-YYYY')) INDEXING OFF,
PARTITION p3 VALUES LESS THAN (TO_DATE('01-OCT-2011','DD-MON-YYYY')) INDEXING ON,
PARTITION p4 VALUES LESS THAN (TO_DATE('01-MAR-2012','DD-MON-YYYY')),
PARTITION p5 VALUES LESS THAN (TO_DATE('01-MAR-2013','DD-MON-YYYY'))
);

Partitions which have not INDEXING clause, inherit the default indexing property of the table (INDEXING OFF).

So:
Partitions P1 and P3 are included in all partial global indexes.
Local index partitions (for indexes created PARTIAL) corresponding to the above two table partitions are created usable by default
Other partitions are excluded from all partial global indexes, and created unusable in local indexes (for indexes created PARTIAL)

select partition_name,indexing from user_tab_partitions where table_name='MYT1';
PARTITION_NAME       INDEXING
-------------------- --------------------
P5                           OFF
P4                           OFF
P3                           ON
P2                           OFF
P1                           ON

select index_name, partition_name,STATUS from user_ind_partitions where index_name = 'LP_IDX';

INDEX_NAME                PARTITION_NAME  STATUS
------------------------- --------------------         --------------------
lp_idx                              P2                            UNUSABLE
lp_idx                              P3                            USABLE
lp_idx                              P1                            USABLE
lp_idx                              P4                            UNUSABLE
lp_idx                              P5                            UNUSABLE

Note: Partitions P4 and P5 are not included in all partial global indexes because the default INDEXING property for the table is OFF, and INDEXING ON/OFF has not been specified at the partition level for those partitions; therefore, P4 and P5 partitions inherit the table default indexing property.

This statement will create a Partial Global index on the MYT1 table.

CREATE INDEX gp_idx ON myt1(c3) GLOBAL INDEXING PARTIAL;

select index_name,indexing from dba_indexes where index_name='GP_IDX';

INDEX_NAME                     INDEXING
------------------------------ --------------------
GP_IDX                                PARTIAL

The GP_IDX index in the example will be created to index only those partitions that have INDEXING ON, and exclude the remaining partitions.