ODA X9-2 -- SPFs / Fiber not recognized / no lights -- due to Firmware version of Mellanox Dual Port SFP28

This is a short story, but I think it will save time for those who are considering ODA X9 machines. 

These types of issues waste the time of network administrator, and ODA field engineers. 

So lets check what we have here; Recently installed an ODA X9-2 L in a Banking environment. 

The machine had 2 SFP cards. Configured the switches , attached the Fiber cables but couldn't make network up & running. 

ODA config was okay, bond devices were up (but link detected was No), OS interfaces were configured properly and physical side (cabling, switch configuration and everything) was as it's supposed to be. ODA image version was 19.17, which was quite up-to-date, but no luck. 

There was no network and no lights on the SFP cards at all. 

 Also checked the following (but didn't help..)  

Checked the optical fiber transceiver ( it may be inserted reversely) 

Checked the fiber models 

Checked the fiber cables 

Checked the customer's switch side and ports 

 Well.. Long story short; the problem was due to the firmware version of the Mellanox Dual Port SFP28 CX5 25Gb Ethernet Adapter.. (Note that, 16.29.1436 is a problematic version).. 

The version can be checked using "odacli describe-component" and "fwupdate list controller" command. The up-to-date firmware is already there, I mean the files are there, but they are not installed by default in ODA X9-2 with version 19.16 and above. Weird.. 

Anyways; what we need to do, was just updating the firmware of those devices and rebooting. As you may guess; if we have multiple cards (in our case we had 2), we update the firmware of both of the cards and reboot. 

The command used for that operation is "fwupdate update controller -n cX -x metadata.xml" -- we replace that cX with our controller name (we get that from the output of odacli describe-component command) This is already addressed in MOS, but you may not find that particular MOS note easily.. 

Moreover, you may think that you are unaffected (in case you are on ODA image version 19.17), but that is not the case.. 

So I'm sharing that MOS note below as well. 

Network not working on ODA X9-2 systems using Mellanox Dual Port SFP28 CX5 25Gb Ethernet Adapter with firmware 16.29.1436 (Doc ID 2910654.1)