We faced with a multiple issue and decided to upgrade EM and install Oracle recommended patches from note:
Enterprise Manager Cloud Control 12c Recommended Plug-Ins and Patches for Database as a Service (DBaaS) (Doc ID 1549855.1)
Specifically:
Patch 20870437: ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.4 PSU (GENERIC)
and
Patch 21415432: enterprise manager for oms plugins 12.1.0.4.14
We had vanilla EM 12.1.0.4 installation without any PSU or patches (except Cloud Framework Plug-In version 12.1.0.2.0).
ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.4 PSU
At the step of installation:
2.9 Run the following command on each OMS instance home to ensure that you do not have any issues with configuration and binary prerequisite checks,
I encountered an error:
I searched through MOS knowledge base and have decided that this error due to the old OPatch version.
I found a note with a same error:
EM12c : OPatchauto fails with : opatchauto finds that weblogic credentials may not be right or admin server is down (Doc ID 1644317.1)
From which I concluded that issue should be resolved after OPatch 11.1.0.11 install.
As I seen from the logs, we have used OPatch version 11.1.0.10.4.
I installed latest OPatch version 11.1.0.x and installation of PSU was ended without any issues.
Enterprise Manager for OMS Plug-Ins 12.1.0.4.14
opatchauto apply -analyze command showed that from 18 sub-patches I could install only 1.
Most of the patches was skipped due to incompatibility checks:
WARNING: Sub-patch "21324890" skipped for incompatibility with components installed in the OMS system.I searched through MOS knowledge base and found a note:
EM 12c: Applying a System Patch to the Enterprise Manager 12.1.0.4 Cloud Control OMS Oracle Home Fails at Analyze with Error: Sub-patch skipped for incompatibility with components installed in the OMS system (Doc ID 1921337.1) which tell us that:
Such incompatibility errors result from two scenarios:
Case 1:The Bundle Patch is comprised of patches for the OMS Oracle Home as well as Plug-in Homes. If one of these Plugins is not present in the environment, then these patches would be skipped automatically as they are not needed.
Case 2: If the plug-in is present but is of a lower version than the plug-in versions mentioned above, then such errors can also result.
Patch 21415432 (Enterprise Manager for OMS Plug-Ins 12.1.0.4.14) contains patches for the below list of plugins:
- Enterprise Manager for Cloud: 12.1.0.8.0
- Enterprise Manager for Fusion Middleware: 12.1.0.6.0
- Enterprise Manager for Oracle Database: 12.1.0.6.0
- Enterprise Manager for Fusion Applications: 12.1.0.6.0
- Enterprise Manager for Storage Management Framework: 12.1.0.4.0
- Enterprise Manager for Siebel: 12.1.0.5.0
- Enterprise Manager for Exadata : 12.1.0.6.0
- Enterprise Manager for Oracle Virtual Infrastructure: 12.1.0.1.0
- Enterprise Manager for Virtualization: 12.1.0.6.0
- Enterprise Manager for My Oracle Support: 12.1.0.6.0
- Enterprise Manager for Chargeback and Capacity Planning: 12.1.0.6.0
- Enterprise Manager for Apache Tomcat: 12.1.0.3.0
- Enterprise Manager for Oracle Database: 12.1.0.7.0
- Enterprise Manager for Storage Management Framework: 12.1.0.5.0
- Enterprise Manager for Cloud: 12.1.0.9.0
- Enterprise Manager for Fusion Middleware: 12.1.0.7.0
EM 12c: Applying a System Patch to the Enterprise Manager 12.1.0.4 Cloud Control OMS Oracle Home Fails at Analyze with Error: Sub-patch skipped for incompatibility with components installed in the OMS system (Doc ID 1921337.1), I have found the emcli commands which can be used to verify versions of plugins installed on the OMS:
./emcli login -username=sysman ./emcli sync ./emcli list_plugins_on_serverI run above commands and found that I have plugin versions which should be patched by the Patch 21415432 (I highlight DB and SMF plugins for simplicity):
OMS name is mgmt.localdomain:4889_Management_Service Plug-in Name Plugin-id Version [revision] Oracle Cloud Framework oracle.sysman.cfw 12.1.0.2.0 Oracle Database oracle.sysman.db 12.1.0.6.0 Oracle Fusion Middleware oracle.sysman.emas 12.1.0.6.0 Oracle MOS (My Oracle Support) oracle.sysman.mos 12.1.0.6.0 Oracle Storage Management Framework oracle.sysman.smf 12.1.0.4.0 Oracle Cloud Application oracle.sysman.ssa 12.1.0.8.0 Oracle Virtual Infrastructure oracle.sysman.vi 12.1.0.1.0 Oracle Virtualization oracle.sysman.vt 12.1.0.6.0 Oracle Exadata oracle.sysman.xa 12.1.0.6.0I continued searching through MOS and found the Bug 21287619 : OPATCHAUTO UNABLE TO IDENTIFY INSTALLED PLUGIN_HOME which was without the resolution but provided some ideas for further investigations.
I started to check them.
At first, I verified that I have correct plugin versions in PLUGIN_HOME:
[oracle@mgmt 21415432]$ ls -la /u01/app/oracle/oms12/plugins/ drwxr-xr-x 14 oracle oinstall 4096 Aug 4 15:05 . drwxr-x--- 16 oracle oinstall 4096 Jul 11 2014 .. drwxr-xr-x 2 oracle oinstall 4096 Aug 4 15:31 backup drwxr-xr-x 12 oracle oinstall 4096 Jul 11 2014 oracle.sysman.cfw.oms.plugin_12.1.0.1.0 drwxr----- 14 oracle oinstall 4096 Aug 10 14:58 oracle.sysman.cfw.oms.plugin_12.1.0.2.0 drwxr-xr-x 18 oracle oinstall 4096 Aug 4 15:03 oracle.sysman.db.oms.plugin_12.1.0.6.0 drwxr-xr-x 18 oracle oinstall 4096 Jul 11 2014 oracle.sysman.emas.oms.plugin_12.1.0.6.0 drwxr-xr-x 13 oracle oinstall 4096 Jul 11 2014 oracle.sysman.mos.oms.plugin_12.1.0.6.0 drwxr-xr-x 12 oracle oinstall 4096 Jul 11 2014 oracle.sysman.smf.oms.plugin_12.1.0.4.0 drwxr-xr-x 14 oracle oinstall 4096 Jul 11 2014 oracle.sysman.ssa.oms.plugin_12.1.0.8.0 drwxr-xr-x 12 oracle oinstall 4096 Jul 11 2014 oracle.sysman.vi.oms.plugin_12.1.0.1.0 drwxr-xr-x 15 oracle oinstall 4096 Jul 11 2014 oracle.sysman.vt.oms.plugin_12.1.0.6.0 drwxr-xr-x 13 oracle oinstall 4096 Jul 11 2014 oracle.sysman.xa.oms.plugin_12.1.0.6.0 drwxr-xr-x 2 oracle oinstall 4096 Jul 11 2014 tmpI reviewed opatch/opatchauto lspatches output:
[oracle@mgmt 21415432]$ opatch lspatches 20870437;ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.4 PSU (GENERIC) [oracle@mgmt 21415432]$ opatchauto lspatches OPatch Automation Tool Copyright (c) 2014, Oracle Corporation. All rights reserved. Oracle Home:/u01/app/oracle/oms12/oms 20870437;ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.4 PSU (GENERIC) The following groups of patch(es) are applied as System Patch bundle(s): 20870437 For more details on installed patch(es) in platform, plugin homes, Please do "/u01/app/oracle/oms12/oms/OPatch/opatch lsinventory -details -oh <desired home path>".opatch lsinv output:
I found an answer in an inventory.xml from Central Inventory:
<HOME NAME="oms12c1" LOC="/u01/app/oracle/oms12/oms" TYPE="O" IDX="3">
<REFHOMELIST>
<REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.db.oms.plugin_12.1.0.4.0"/>
<REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.xa.oms.plugin_12.1.0.4.0"/>
<REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.emas.oms.plugin_12.1.0.4.0"/>
<REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.mos.oms.plugin_12.1.0.5.0"/>
<REFHOME LOC="/u01/app/oracle/oms12/jdk16"/>
<REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.vt.oms.plugin_12.1.0.5.0"/>
<REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.ssa.oms.plugin_12.1.0.6.0"/>
<REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.smf.oms.plugin_12.1.0.2.0"/>
<REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.ssa.oms.plugin_12.1.0.7.0"/>
<REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.db.oms.plugin_12.1.0.5.0"/>
<REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.smf.oms.plugin_12.1.0.3.0"/>
<REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sun.oss7.oms.plugin_12.1.0.4.0"/>
<REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.emas.oms.plugin_12.1.0.5.0"/>
<REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.cfw.oms.plugin_12.1.0.2.0"/>
</REFHOMELIST>
</HOME>
Notice, that inventory.xml contains old plugin versions! It's hard to say what exactly leads to a such behaviour and I didn't have enough time for further investigations.It was enough for me, that inventory.xml dont contain actual plugin versions.
Example 1: oracle.sysman.smf.oms.plugin (Storage Management Framework).
emcli list_plugins_on_server shows version 12.1.0.4:
Oracle Storage Management Framework oracle.sysman.smf 12.1.0.4.0inventory.xml contains information about versions 12.1.0.2/12.1.0.3:
<HOME NAME="OraHome13" LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.smf.oms.plugin_12.1.0.2.0" TYPE="O" IDX="24" PLUGIN="T"> <HOME NAME="OraHome16" LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.smf.oms.plugin_12.1.0.3.0" TYPE="O" IDX="29" PLUGIN="T">Patch 21415432 contains fixes for SMF plugin versions 12.1.0.4/12.1.0.5.
Example 2: oracle.sysman.db.oms.plugin (Database Plugin).
emcli list_plugins_on_server shows version 12.1.0.6:
Oracle Database oracle.sysman.db 12.1.0.6.0inventory.xml contains information about versions 12.1.0.4/12.1.0.5:
<HOME NAME="oracle_sysman_db11" LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.db.oms.plugin_12.1.0.4.0" TYPE="O" IDX="4" PLUGIN="T"> <HOME NAME="OraHome15" LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.db.oms.plugin_12.1.0.5.0" TYPE="O" IDX="28" PLUGIN="T">Patch 21415432 contains fixes for Database Plugin versions 12.1.0.6/12.1.0.7.
Due to such inconsistencies opatch reported a message like below:
WARNING: Sub-patch "19060193" skipped for incompatibility with components installed in the OMS system.I decided to recover lost inventory entities by note:
EM 12c: How to Recover a Lost Inventory Record for the Enterprise Manager 12.1.0.1 Cloud Control OMS and Management Agent (Doc ID 1480237.1)
GOAL
In the case that recovering a lost inventory record may be necessary, this note describes how to accomplish this for the Enterprise Manager 12.1.0.1 Cloud Control OMS and Management Agent.
First of all, I had take backup of Central Inventory, local inventory, ORACLE_HOME of OMS.
8 plugins from 9 total didn't have an actual version in the inventory.xml.
For each of them, I executed steps from the note:
EM 12c: How to Recover a Lost Inventory Record for the Enterprise Manager 12.1.0.1 Cloud Control OMS and Management Agent (Doc ID 1480237.1)
I remind, that before my actions only 1 sub-patch (from 18) could be applied to the OMS.
And after that, opatchauto analyze showed that 9 sub-patches from 18 could be applied to the OMS. Bingo!
OMS Plug-Ins patches contains fixes for 2 versions of each plugin. And only one of them was installed in my environment.
Further installation was ended without any issues.
