EBS 12.2 -- fs_clone fails on validations: ValidateETCHosts / [ERROR]: Required content is missing from /etc/hosts

Currently we are in the upgrade process of a fresh EBS 12.2 instance. (It is going be the first TEST environment of a new EBS project).  Okay. With that, lets go! ->

As a part of the installation, and as a requirement we are upgrading the newly installed EBS 12.2.0 to 12.2.11 and we failed while running adop's fs_clone as a post action for the upgrade. 

Adop logs were stating the error clearly. (as shared below)

-- ADOPValidations_detailed.log

Started execution : ADOPValidations.java
Validating entries in /etc/hosts
/etc/hosts contains required entries

--txkADOPValidations.log

ValidateETCHosts:
[ERROR]: Required content is missing from /etc/hosts file.

According to the error, there must have been a missing entry in the hosts file of the Apps node(s). However; /etc/hosts file was looking perfectly fine. (shared the contents below).

127.0.0.1 localhost.localdomain localhost

10.10.70.12 erman.ermanexample.com erman

Having the host file in place, with the correct content (apps host, domain and its IP address, along with localhost) in the proper format made me think that this error that adop was encountering might be a misleading one..  That's why we checked everything :).. From FND_NODES to ADOP-related tables &  autoconfig context files... Nothing seemed weird there.. 

Anyways, we were using the "cat" command for displaying the contents of /etc/hosts file at first.. So we decided the check its contents with our favorite text editor (which is "vi") and once we opened it, we saw an empty character at the end of the second line. It might be the cause of the problem. So, we made the customer modify the file and remove that empty char from there.

This didn't help.  So, again we found ourselves checking the clues, searching the Oracle Support and doing stuff like that.. Nevertheless, nothing made us think the otherwise.. There should have been something wrong with the hosts file..

As a last resort, we made the customer to recreate the hosts file from scratch.. Customer did that, and we had that Aha! moment and the problem disappeared :) Most probably, there were some other characters between those hosts entries and this was the cause of the problem.. It was just because adop couldn't read the hosts file as it should..

So, if you see this error, be persistent and insist that there is a problem with the hosts file. This error is not a misleading one, and Oracle's error handling and error messages are becoming point shots, more "to the point" day by day. Just keep that in mind.. See you in the next post.