GRC -- GRC & EBS implementation //errors and solutions

Recently installed GRC 8.6.6, PCG 7.3.3 and PEA 8.6.6 using the Installation guides delivered with the products to implement GRC with an EBS 12.1.3 instance.

The main document to be followed for doing this type of an installation was Enterprise Governance Risk and Compliance (EGRC) Product Information (Doc ID 1084596.1) and basically what we did was the following;

• Installing an 11g Oracle Database for GRC
• Installing a Weblogic 12C for GRC
• Installing ADR 12C on Weblogic
• Installing database for Schemas for GRC using RCU utility
• Creating a Weblogic Domain for GRC
• Deploying GRC application using Weblogic Console
• Upgrading GRC
• Installing PCG on EBS
• Installing critical PCG Patches
• Installing PAE on EBS
• Fixing any errors reported by the functional team

It was not the thing that is frequently done in our customer environments, so maybe that's why we encountered some errors and spent some time to find their solutions, which I find useful to share with you.

We basically encountered 3 errors. 2 of them were caused by the complexity of the documentation and one of them was directly related with a corrupted data.

Let's see what those errors and their solutions are;

ERROR 1 - unable to synchronize access in grc 8.6.6

The connection test that we did for the datasource that we created on GRC, was succesful; but the synchronize acces job (in GRC application) failed with the following error. (error reported in grc.log)

ERROR [ExecutorThread-11] DataSourceService:1343 Error while setting ETL completed
java.lang.RuntimeException: Failed to serialize the object:
Descriptor Exceptions:
---------------------------------------------------------
Exception [EclipseLink-59] (Eclipse Persistence Services - 2.3.1.v20111018-r10243): org.eclipse.persistence.exceptions.DescriptorException
Exception Description: The instance variable [thingSavedStates] is not defined in the domain class [oracle.apps.grc.domain.datasource.SourceSyncState$SourceSyncStateBuilder], or it is not accessible.
Internal Exception: java.lang.NoSuchFieldException: thingSavedStates
Mapping: org.eclipse.persistence.oxm.mappings.XMLCompositeCollectionMapping[thingSavedStates]
Descriptor: XMLDescriptor(oracle.apps.grc.domain.datasource.SourceSyncState$SourceSyncStateBuilder --> [])
Runtime Exceptions:
---------------------------------------------------------
at oracle.apps.grc.domain.datasource.SourceSyncState.toXML(SourceSyncState.java:135)
at oracle.apps.grc.dataservices.dao.impl.spring.datasource.DataSourceDaoSpr.updateSyncState(DataSourceDaoSpr.java:1249)
at oracle.apps.grc.dataservices.dao.impl.spring.datasource.DataSourceDaoSpr.setEtlCompleted(DataSourceDaoSpr.java:1946) 

Solution:

USER_MEM_ARGS should be updated correctly in setDomainEnv.sh.

As documented in:
http://docs.oracle.com/cd/E51797_01/doc.8651/e52268.pdf
page : 2-14

Action plan:

1. Stop application Server

2. Backup setDomainEnv.sh file

3. Update the USER_MEM_ARGS parameter in setDomainEnv.sh file . The modification should be done between the comment and if statement.

# IF USER_MEM_ARGS the environment variable is set, use it to override ALL MEM_ARGS values 

<<<<<Changes should come here>>>>>>>
case "${SERVER_NAME}" in "AdminServer")
USER_MEM_ARGS="-Xms2048M –Xmx2048M" ;;
...
....
USER_MEM_ARGS="${USER_MEM_ARGS} -XX:PermSize=256m -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=128M -Djava.awt.headless=true -Djbo.ampool.maxpoolsize=600000 -Dfile.encoding=UTF-8 -Djavax.xml.bind.context.factory=com.sun.xml.internal.bind.v2.ContextFactory" 

 <<<<<Changes should come here>>>>>>>
if [ "${USER_MEM_ARGS}" != "" ] ; then

4. Ensure the eclipselink-2.3.1.jar file exist in the below path. <MW_HOME>/grc866/grc/WEB-INF/lib/ location

5. Start the application server

6.Retest the issue

ERROR 2 - unable to syncronize access in grc 8.6.6 

The connection test that we did for the datasource that we created on GRC, was succesful; but the syncronize acces job (in GRC application) failed with the following error. (error reported in grc.log)

DEBUG [EtlExtractor-1254779240] GrcLogPrintStream:73 STDOUT (oracle.core.ojdl.logging.ConsoleHandler:118) <Apr 6, 2017 11:10:11 AM EEST> <Error> <Default> <ODI-1217> <Session TCG_SCEN_Users_21 (85154) fails with return code 7000.
ODI-1226: Step TCG_INTR_Users_21 fails after 1 attempt(s).
Caused by: ODI-1240: Flow TCG_INTR_Users_21 fails while performing a Integration operation. This flow loads target table Users.
Caused by: org.apache.bsf.BSFException: exception from Jython:
Traceback (most recent call last):
File "<string>", line 50, in <module>

Solution:

PRE_CLASSPATH was not properly set in the setDomain.env.

As documented in "http://docs.oracle.com/cd/E51797_01/doc.8651/e52268.pdf
page : 2-2" , the PRE_CLASSPATH should be updated properly.

Action Plan:

1. Locate the following lines in the file: 

if [ "${PRE_CLASSPATH}" != "" ] ; then CLASSPATH="${PRE_CLASSPATH}${CLASSPATHSEP}${CLASSPATH}" 

export CLASSPATH 

fi 

2. Add the following before the above lines: 

PRE_CLASSPATH="/grc865/grc/WEB-INF/lib/jython- 2.5.1.jar:${PRE_CLASSPATH}" 

export PRE_CLASSPATH

Note: Replace with the actual path to your middleware home.

ERROR 3 - unable to synchronize access in grc 8.6.6 

The connection test that we did for the datasource that we created on GRC, was succesful; but the synchronize acces job (in GRC application) failed with the following error. (error reported in grc.log)

DEBUG [EtlExtractor-976389891] DataSourceDaoSpr:1234 updateSyncState(oracle.apps.grc.domain.datasource.SourceSyncState@22489656) 

ERROR [EtlExtractor-976389891] AvailableResource:173 myBlocks null 

ERROR [EtlExtractor-976389891] LocalEtlTcgWriter:412 A problem occurred in LocalEtlTcgWriter.writeData: 

java.lang.NullPointerException: myBlocks null 

at oracle.apps.odin.reasonerio.file.page.AvailableResource.askBlock(AvailableResource.java:174) 

at oracle.apps.odin.reasonerio.file.page.PageManager.createNewPage(PageManager.java:92) 

at oracle.apps.odin.reasonerio.file.page.PagingGrccChannelManager.expandChannel(PagingGrccChannelManager.java:498) 

at oracle.apps.odin.reasonerio.file.page.PageChannelController.<init>(PageChannelController.java:79) 

at oracle.apps.odin.reasonerio.file.page.PagingGrccChannelManager.createChannel(PagingGrccChannelManager.java:456) 

at oracle.apps.grc.reasonerio.graph.blockbytype.writer.BBTGraphWriter.getAttributeWriter(BBTGraphWriter.java:148) 

at oracle.apps.grc.appservices.connector.LocalEtlTcgWriter.writeNode(LocalEtlTcgWriter.java:669) 

at oracle.apps.grc.appservices.connector.LocalEtlTcgWriter.persistResults(LocalEtlTcgWriter.java:395) 

at oracle.apps.grc.appservices.connector.LocalEtlExtractor.persistResults(LocalEtlExtractor.java:540) 

at oracle.apps.grc.appservices.connector.LocalEtlExtractor.retrievePersistResultsLocal(LocalEtlExtractor.java:349) 

at oracle.apps.grc.appservices.connector.LocalEtlExtractor.retrievePersistResults(LocalEtlExtractor.java:239) 

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 

at java.lang.reflect.Method.invoke(Method.java:606) 

at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:175) 

at org.python.core.PyObject.__call__(PyObject.java:355) 

at org.python.core.PyMethod.__call__(PyMethod.java:215) 

at org.python.core.PyMethod.instancemethod___call__(PyMethod.java:221) 

at org.python.core.PyMethod.__call__(PyMethod.java:206) 

at org.python.core.PyObject.__call__(PyObject.java:381) 

at org.python.core.PyObject.__call__(PyObject.java:385) 

at org.python.pycode._pyx0.f$0(<string>:51) 

at org.python.pycode._pyx0.call_function(<string>) 

at org.python.core.PyTableCode.call(PyTableCode.java:165) 

at org.python.core.PyCode.call(PyCode.java:18) 

at org.python.core.Py.runCode(Py.java:1204) 

at org.python.core.Py.exec(Py.java:1248) 

at org.python.util.PythonInterpreter.exec(PythonInterpreter.java:172) 

at org.apache.bsf.engines.jython.JythonEngine.exec(JythonEngine.java:144) 

at com.sunopsis.dwg.codeinterpretor.SnpScriptingInterpretor.execInBSFEngine(SnpScriptingInterpretor.java:322) 

at com.sunopsis.dwg.codeinterpretor.SnpScriptingInterpretor.exec(SnpScriptingInterpretor.java:170) 

at com.sunopsis.dwg.dbobj.SnpSessTaskSql.scripting(SnpSessTaskSql.java:2472) 

at oracle.odi.runtime.agent.execution.cmd.ScriptingExecutor.execute(ScriptingExecutor.java:47) 

at oracle.odi.runtime.agent.execution.cmd.ScriptingExecutor.execute(ScriptingExecutor.java:1) 

at oracle.odi.runtime.agent.execution.TaskExecutionHandler.handleTask(TaskExecutionHandler.java:50) 

at com.sunopsis.dwg.dbobj.SnpSessTaskSql.processTask(SnpSessTaskSql.java:2913) 

at com.sunopsis.dwg.dbobj.SnpSessTaskSql.treatTask(SnpSessTaskSql.java:2625) 

at com.sunopsis.dwg.dbobj.SnpSessStep.treatAttachedTasks(SnpSessStep.java:577) 

at com.sunopsis.dwg.dbobj.SnpSessStep.treatSessStep(SnpSessStep.java:468) 

at com.sunopsis.dwg.dbobj.SnpSession.treatSession(SnpSession.java:2128) 

at com.sunopsis.dwg.dbobj.SnpSession.treatSession(SnpSession.java:1930)

Solution:

While working to solve the issues (ERROR 1 and ERROR 2), we tried to run the synchronize access job again and again. After solving those issues,  we realized that; those failed tries corrupted the ETL data.

Action Plan:

1. Take backup of GRC environment database, WLS  filesystem, ETL repository filesystem, GRC Reports filesystem 

Note: your ETL Repository and GRC Reports locations can be found in the GRC application 

(Navigator > Setup and Administration > Manage Application Configuration) 

Example ETL repository: /Oracle/EGRCC/grcc_etl 

Example Report location: /Oracle/EGRCC/grcc_rep 

2. Stop GRC application server 

3. Kill any pending java processes(if they are still active)

4. Clear cache

cd $MW_HOME/user_projects/domains//bin 

rm -rf ..../servers/AdminServer/tmp/* 

rm -rf ..../servers/AdminServer/logs/* 

rm -rf ..../servers/AdminServer/cache/* 

5. Stop and start GRC Database 

6. Drop the SNP_* tables (ODI repo) 

--Drop all the table with name starting with SNP_. 

Check the following query returns 0 rows after dropping the tables;

select count(*) from all_tables where table_name like 'SNP_%'; 

7. Delete "temp.repository" and "raw" directories and their sub directories under home/grc_etl(Example ETL repository: /Oracle/EGRCC/grcc_etl) 

--"DO NOT touch "persistence"

--application should be shut down for deleting these directories.

9. Start the GRC application server 

10. Run Access and Transaction synchronization as the first step after the GRC application is started.