ODA -- odacli command Issue after implementing SSL: A Real SR Process in the Shadow of Missing Steps -- Lessons Learned & Takeaways

Enhancing security in Oracle Database Appliance (ODA) environments through SSL (Secure Socket Layer) configurations can ripple across various system components. Changing certificates, transforming the SSL configuration to a more secure one (with more secure and trusted certificates) can be a little tricky. However, the path to resolving issues encountered during these processes isn't always found in the documentation.

In this post, I will share a real Oracle Service Request (SR) journey around this subject. I will try to share both the technical side of things and those undocumented steps we had to follow.

The Symptom: Silence from odacli

After implementing SSL configuration (renewing the default SSL certificates of DCS agent and DCS controller with the certificates of the customer) on ODA, we hit a wall: the odacli commands simply refused to work. For instance, when tried to run: odacli list-vms, we got the following cryptic message;

DCS-12015: Could not find the user credentials in the DCS agent wallet. Could not find credential for key:xxxx

This clearly pointed to a problem with the DCS Agent wallet lacking the necessary user credentials. Despite following the configuration guides, odacli failed, and the DCS Agent felt completely out of reach.

Initial Moves: Sticking to the Script (Official Oracle Docs)

Oracle's official documentation laid out a seemingly straightforward path:

Configure SSL settings within the dcs yml file(s).
Restart DCS.
Update CLI certificates and dcscli configuration files.

We done all this. Every step was executed properly. Yet, the problem persisted. odacli continued to encounter errors.

The Real Culprit: A Missing Step, An Undocumented Must-Do

Despite the seemingly correct configurations, our back-and-forth with the Oracle support engineer through the SR revealed a critical piece of the puzzle – a step absent from any official documentation:

We get ODACILMTL PASSWORD by the following command;

/u01/app/19.23.0.0/grid/bin/mkstore \ -wrl /opt/oracle/dcs/dcscli/dcscli_wallet \ -viewEntry DCSCLI_CREDENTIAL_MAP@#3#@ODACLIMTLSPASSWORD

We get the password from the output of the command above and we use it to change the password of /opt/oracle/dcs/dcscli/dcs-ca-certs. (--custom keystore. Note that, we get the password related with DCSCLI_CREDENTIAL_MAP.  )

/opt/oracle/dcs/java/1.8.0_411/bin/keytool -storepasswd -keystore /opt/oracle/dcs/dcscli/dcs-ca-certs

We update the conf file with the ODACLIMTLSPASSWORD entries.

These two files : /opt/oracle/dcs/dcscli/dcscli.conf and /opt/oracle/dcs/dcscli/dcscli-adm.conf

The following line: 

TrustStorePasswordKey=ODACLIMTLSPASSWORD

So we do something like a mapping of  wallet and the keystore passwords using the ODACLIMTLPASSWORD.

Skip these, and even with a perfectly configured agent, odacli commands will fail because they can't access the necessary credentials.

Live Intervention and Breakthrough

During a screen-sharing session with the Oracle engineers via Zoom, we went through the following:
Re-verified and, where needed, reconfigured the dcs yml file(s).
Ensured the wallet entry was correctly added.
Executed the crucial mkstore and dcscli commands (above) 
Restarted both the Agent and CLI services.

After these, commands like odacli list-jobs and odacli list-vms started working flawlessly. 

This SR journey left us with some significant takeaways:

"Official documentation may not be always the full story." Some critical steps, like the mkstore credential mapping, might only surface through the SR process itself.

"Configuration details demand absolute precision." File names, paths, and alias definitions in Oracle configurations must be an exact match. Even a minor deviation during the adaptation of Oracle's example configurations to your environment can bring the system down.

"Configuration Files are as Crucial as Logs in Support Requests". Attaching the actual configuration files to your SR significantly accelerates the troubleshooting process for Oracle engineers.

Lessons Learned:

• Documentation Gaps: Document the steps learned from SRs in the internal technical notes.
• The processes behind enhancing security in Oracle environments may extend beyond the confines of official documentation. This experience wasn't just about resolving a technical problem; it was a valuable lesson in enterprise knowledge management. If you find yourself facing a similar situation, remember to explore beyond the documented steps – and make sure those learnings from SRs find their way into your internal knowledge base.