Weblogic /Hyperion Planning -- JPS-01013: The credential store DN cn=CredentialStore,cn=opssSecurityStore,cn=JPSContext,cn=opssRoot is missing

Here is an interesting story that I found myself in recently.. It was an Hyperion environment and customer started to encounter weird JPS-01013 errors while starting the Admin Server of the underlying Weblogic.

Actually, the recent change that was done in the environment; was related with the IP address setting of the repository database (The database that was hosting the schemas created with RCU..).

In brief; customer was going from a single instance to RAC 2 node config, and the jdbc urls needed to be changed ..

Although, such a change will be handled by updating the data sources and maybe some app-specific config;I think the customer have done more than that:) I don't have that part of the story, but somehow they executed RCU again and again in this environment, where we already had all the schemas for the repository in place.. 

The errors were generic, but it was clearly seen that they were all related with the repository schemas .. There was an Weblogic-internal ldap related process failing actually..

First thing we needed to check was the data sources of Weblogic... As the repository hostname and IP addresses have been changed, this might be related with the data source settings..

In order to check the data sources, we needed to use the filesystem directly.. So we opened the jdbc xml files residing in our Weblogic Domain directory/config/jdbc..

The files were there and the jdbc urls defined inside them were correct.. ( so it seemed customer changed it already).. So this was not a jdbc url-related connection problem..

Then we checked the repository database.. The issue was related with the OPSS, so we checked the OPSS records..

Just a little background info for OPSS:

Oracle Platform Security Services (OPSS) provides enterprise product development teams, systems integrators (SIs), and independent software vendors (ISVs) with a standards-based, portable, integrated, enterprise-grade security framework for Java Standard Edition (Java SE) and Java Enterprise Edition (Java EE) applications. OPSS is the underlying security platform that provides security to Oracle Fusion Middleware including products like WebLogic Server, SOA, WebCenter, ADF, OES to name a few. OPSS is designed from the ground up to be portable to third-party application servers.

We first checked the registered OPSS schema owner using the query below;

SELECT OWNER FROM System.SCHEMA_VERSION_REGISTRY$ where COMP_ID = 'OPSS';

We saw that there were 3 Owners/schemas registered for the OPSS component..

Then we checked each schema, by querying the jps_attrs tables.. We checked the value of the attrval column and expected to see the "cn=CredentialStore,cn=opssSecurityStore,cn=JPSContext,cn=opssRoot" there.. 

select * from OPSS_SCHEMA1.jps_attrs where attrval = 'cn=CredentialStore,cn=opssSecurityStore,cn=JPSContext,cn=opssRoot';

We saw that, only one of these registered schemas had that attrval value. At this moment, we concluded that, the data sources were misconfigured... I mean jdbc url settings that we see in those data source files were correct but the information related with the OPSS schema users and passwords were probably wrong.. Probably, this was a result of executing those unnecessary RCUs.. 

Well.. We opened those jdbc data source xml files using our file editor and updated the username to be the name of the OPSS schema that had the needed attrval value..

Then, we updated the password field of those jdbc data source files as well.. (the jdbc files related with OPSS .. for instance -> opss-datasource-jdbc.xml)

Note that, the password fields in the jdbc xml files needed to be updated with encrypted passwords.

So we encyprted the password of the correct OPSS schema and update the files with the encyrpted values..

For encyrpting the password; we sourced the setDomain.env and executed "java weblogic.security.Encrypt <our_password>

Having the jdbc files updated, we also modified the bootstrap credentials of Weblogic using WLST..

This action was also required.. Without this action, Weblogic didn't get the changed values.

In order to modify the bootstrap credential ; we used ->

wls:/offline>

modifyBootStrapCredential(jpsConfigFile='/Oracle/Middleware/user_projects/domains/EPMSystem/config/fmwconfig/jps-config.xml',username='ERM_OPSS',password='testtest')

With that, we could be able start the Admin Server and the problem dissapeared. :)