opatch and libpicl on Solaris SPARC

This morning while applying recent DBBP (Database Bundle Patch) the datapatch invocation finished with an error:

SQL Patching tool version 12.1.0.2.0 on Thu Jun  2 10:58:41 2016
Copyright (c) 2015, Oracle.  All rights reserved.

Log file for this invocation: /export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_invocation.log

Connecting to database...OK
Bootstrapping registry and package to current versions...done

Queryable inventory could not determine the current opatch status.
Execute 'select dbms_sqlpatch.verify_queryable_inventory from dual'
and/or check the invocation log
/export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_invocation.log
for the complete error.
Prereq check failed, exiting without installing any patches.

Please refer to MOS Note 1609718.1 and/or the invocation log
/export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_invocation.log
for information on how to resolve the above errors.

SQL Patching tool complete on Thu Jun  2 10:58:50 2016

Here are the contents of sqlpatch_invocation.log file:

SQL Patching tool version 12.1.0.2.0 on Thu Jun  2 10:58:41 2016
Copyright (c) 2015, Oracle.  All rights reserved.

Log file for this invocation: /export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_invocation.log

SQL Patching arguments:
  verbose: 1
  force: 0
  prereq: 0
  upgrade_mode_only: 
  oh: 
  bundle_series: 
  ignorable_errors: 
  bootstrap: 
  skip_upgrade_check: 
  userid: 
  pdbs: 

Connecting to database...OK
catcon: ALL catcon-related output will be written to /export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_catcon__catcon_12392.lst
catcon: See /export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_catcon_*.log files for output generated by scripts
catcon: See /export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_catcon__*.lst files for spool files, if any
Bootstrapping registry and package to current versions...done
verify_queryable_inventory returned ORA-20001: Latest xml inventory is not loaded into table

Queryable inventory could not determine the current opatch status.
Execute 'select dbms_sqlpatch.verify_queryable_inventory from dual'
and/or check the invocation log
/export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_invocation.log
for the complete error.
Prereq check failed, exiting without installing any patches.

Please refer to MOS Note 1609718.1 and/or the invocation log
/export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_invocation.log
for information on how to resolve the above errors.

SQL Patching tool complete on Thu Jun  2 10:58:50 2016

The famous "ORA-20001: Latest xml inventory is not loaded into table" error even has a dedicated article on MOS: Queryable Patch Inventory - Issues/Solutions for ORA-20001: Latest xml inventory is not loaded into table (Doc ID 1602089.1).
I tried all of the provided solutions but no one helped me.
As the sqlpatch_invocation.log suggested, I executed the dbms_sqlpatch.verify_queryable_inventory function:

SQL>select dbms_sqlpatch.verify_queryable_inventory from dual;

VERIFY_QUERYABLE_INVENTORY
--------------------------------------------------------------------------------
ORA-20001: Latest xml inventory is not loaded into table

Well, it was nothing new. Then I queried OPATCH_XML_INV external table because it is used internally by the aforementioned function:

SQL>select * from OPATCH_XML_INV ;
ERROR:
ORA-29913: error in executing ODCIEXTTABLEFETCH callout
ORA-29400: data cartridge error
KUP-04004: error while reading file
/export/home/velikikh/oracle/12.1.0.2/home1/QOpatch/qopiprep.bat



no rows selected

Ok, at least I got some steps for further investigation. qopiprep.bat script simply executes opatch command with some additional processing:

$ORACLE_HOME/OPatch/opatch lsinventory -xml  $ORACLE_HOME/QOpatch/xml_file.xml -retry 0 -invPtrLoc $ORACLE_HOME/oraInst.loc >> $ORACLE_HOME/QOpatch/stout.txt

I decided to try executing it manually:

SQL>!opatch lsinv
Java HotSpot(TM) 64-Bit Server VM warning: PICL (libpicl.so.1) is missing. Performance will not be optimal.
Java HotSpot(TM) 64-Bit Server VM warning: PICL (libpicl.so.1) is missing. Performance will not be optimal.
Oracle Interim Patch Installer version 12.1.0.1.12
Copyright (c) 2016, Oracle Corporation.  All rights reserved.
..skipped...

These lines with JVM warning were rather unusual for me and I did not know why they occured.
As I had a feeling that they were related to my problem, I searched through the MOS knowledge base and found an article: VM Warning "PICL (libpicl.so.1) is missing" After Upgrading to Java SE 8u40+ or 7u80+ (Doc ID 2093623.1).
That had a reference to Poonam Bajaj's blog:

 Please note that this warning just indicates that the JVM could not load the PICL library so it won't be able to turn on some of the JIT Compiler optimizations that require the cache line data size information. 
It does not stop the Java process, or impact the process in any other way. 
Also, note that this is not a Java bug. The warning message is just intended to inform the user that the platform it is running on does not have the PICL library.

Well, nothing critical was in those messages but Queryable Opatch Inventory (hereinafter QOpatch) was not working.
To verify my assumption about the negative effect of libpicl message on QOpatch, I made a quick and dirty correction to qopiprep.bat file - I redirected the stderr to /dev/null :):

$ORACLE_HOME/OPatch/opatch lsinventory -xml  $ORACLE_HOME/QOpatch/xml_file.xml -retry 0 -invPtrLoc $ORACLE_HOME/oraInst.loc >> $ORACLE_HOME/QOpatch/stout.txt 2>/dev/null

Having done that, I executed select against OPATCH_XML_INV table:

SQL>select * from opatch_xml_inv;

XML_INVENTORY
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <Inv
entoryInstance> <ora

Allright, at least it started working. I didn't try executing other commands but instead asked the SA (System Administrator) team to install the libpicl.so library.
They installed pkg:/system/libpicl and I finally got datapatch working:

[velikikh@gsmcat OPatch]$ ./datapatch -verbose
SQL Patching tool version 12.1.0.2.0 on Thu Jun  2 13:26:15 2016
Copyright (c) 2015, Oracle.  All rights reserved.

Log file for this invocation: /export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_6653_2016_06_02_13_26_15/sqlpatch_invocation.log

Connecting to database...OK
Bootstrapping registry and package to current versions...done
Determining current state...done

Current state of SQL patches:
Bundle series DBBP:
  ID 160419 in the binary registry and not installed in the SQL registry

Adding patches to installation queue and performing prereq checks...
Installation queue:
  Nothing to roll back
  The following patches will be applied:
    22806133 (DATABASE BUNDLE PATCH: 12.1.0.2.160419 (22806133))

Installing patches...
Patch installation complete.  Total patches installed: 1

Validating logfiles...
Patch 22806133 apply: SUCCESS
  logfile: /export/home/velikikh/oracle/cfgtoollogs/sqlpatch/22806133/20067756/22806133_apply_GSMCAT_2016Jun02_13_28_31.log (no errors)
SQL Patching tool complete on Thu Jun  2 13:32:19 2016

So the conclusion is, while using Oracle Database 12c on Solaris 11 SPARC64, you will definitely need to install the libpicl. This requirement has not mentioned either in Oracle Database documentation or MOS articles (at least I have missed something) but it is present.
From Oracle Database Installation Guide for Oracle Solaris:

The following packages must be installed:
    pkg://solaris/system/dtrace
    pkg://solaris/developer/assembler
    pkg://solaris/developer/build/make
    pkg://solaris/system/xopen/xcu4 (if not already installed as part of standard Oracle Solaris 11 installation)
    pkg://solaris/x11/diagnostic/x11-info-clients
    pkg://solaris/compress/unzip

Although I'd had these packages installed, the datapatch execution completed with the errors till I installed system/picl:

-bash-4.1$ pkg list dtrace assembler make xcu4 x11-info-clients unzip
NAME (PUBLISHER)                                  VERSION                    IFO
compress/unzip                                    6.0-0.175.2.10.0.4.0       i--
developer/assembler                               0.5.11-0.175.2.13.0.2.0    i--
developer/build/make                              0.5.11-0.175.2.0.0.34.0    i--
system/dtrace                                     0.5.11-0.175.2.12.0.4.2    i--
system/xopen/xcu4                                 0.5.11-0.175.2.13.0.1.2    i--
x11/diagnostic/x11-info-clients                   7.7-0.175.2.0.0.42.1406    i--
-bash-4.1$ pkg list picl
NAME (PUBLISHER)                                  VERSION                    IFO
system/picl                                       0.5.11-0.175.2.9.0.3.2     i--