Updating Cisco UCS firmware and Activating firmware on Subordinate and Primary at the same time

Every wonder what happens when you accidentally activate a new firmware on the subordinate AND the primary fabric interconnect for UCS at the same time?

The manual: UpgradingCiscoUCSFrom1.2.1To1.3.1.pdf specifically states that you should activate the subordinate, wait till it completes the update before you proceed with the primary. Well, I did just that for 1.1 to 1.2 but while doing 1.2 to 1.3, I accidentally activated both of them at the same time. If you’ve every updated UCS’ firmware, you would know that even if you follow the instructions, your UCS nodes light up like a christmas tree in UCSM. Status of the IOMs, FIs, and all of the nodes involved go from red, yellow, orange and finally green sometimes taking upwards of 15 minutes. I wished the upgrade guide from Cisco would mention this as my heart stop a few times when I did my first ever update. Now, I just activate the firmware and go to another window to do something else.

Ok, so to answer the topic of this post, I lost connectivity to both fabrics and after 15 minutes of so, the FIs slowly came back up and all status was green! Whew, the lesson I learned is to not try and do multiple things at the same time as I was also trying to fix an OCS problem in another window. Oh, and sleep helps too.