, , , , , , ,

A while back I was able to get an older HP Proliant DL380; which is a G5 unit.  After installing a Linux distribution on it and it turn out to be a quite useful machine; I thought I’d update all the firmwares before adding some CPU’s and RAM.

Things went very badly wrong when an update from HP destroyed the firmware on both of the NICs.

After the update had ran; and filled the screen with errors I found that the machine would not even POST.  It would power on, and then sit there; everything seemingly OK – but not ever making it to a BIOS check screen, much less boot.  I thought I’d bricked it.

I did some reading, and discovered that the fix for this was to pull out both of the power supplies and let the thing sit for about thirty seconds.  When I reseated the power supplies; it sprang back in to life and I breathed a sigh of relief.

At this point it was pretty clear that the NIC’s were very unhappy.  Linux would boot, but no NIC hardware was identified.  Accessing the NIC BIOS was possible; but corruption was evident (see main picture).  The primary BIOS knew that there were two NIC-like devices; but one of them would not ‘enable’; and although the other would enable; it would not function once Linux booted.

My first hope was that an HP Service Pack would address this issue as these bootable ISO images contain firmware updates along with drivers for operating systems long-since retired.

HP – bless them – would like you to give them a sack of cash in the form of a support contract in order to access these service packs.  However if you’re not keen on doing that then this post might help you out.  The last service pack that features firmware updates for the G5 is June 2014.  IE, there is no use downloading newer service pack ISO’s.

Sadly the service pack did not help.  It updated the firmwares of the BIOS and storage controller; but not the NICs, and – it seems, does not contain firmwares to do so anyway.

After hours of digging, two posts assisted me greatly in fixing this issue, one drawing from the other.

This one, and this one.  If for some reason you can’t find the bootable FreeDOS ISO image; then it’s filename was ‘fd11src_live_xdiag.iso’; so you might find that lingering somewhere.  I’ve got a copy of this file along with PDF archives of both posts, so if you need them put somewhere, let me know.

The ISO as cited has the Broadcom firmwares already in place; otherwise you’ll have to download both the firmwares and FreeDOS independently; and then make a single bootable ISO; which is a bit of a pain to do if you’re pressed for time.

The DL380 has two NICs, and as such both will probably need to be reflashed.  The NIC used for iLO is not accessible.  If you access the Proliants main BIOS and look at the list of PCI devices; you should see both NICs listed; along with their bus, device and function identifying numerics.  For me they were (bus/device/function) values of:

3:0:0 & 5:0:0

Record what your values are; you might need them.  Once you’ve got all the pieces in place, the process is fairly straight forward.

Boot off the FreeDOS ISO and using the xdiag tool embedded within the ISO; reflash the NIC firmwares as described in the second blog-post; but a longer description is available in the first post.

If you’re lucky, when you run the pci scan command; both NICs will be identified and listed.  If this is the case then you only need to reinstall the bootcode (Programming Non-volatile Memory… section of the text file).  If – like me – one of the NICs is identified; but one lists as ‘Unknown’, then you’ll also need to follow the instructions in the second post for fixing the NVRAM of the device listed as ‘Unknown’ – this is the ‘Restoring Corrupted NIC….’ section.  If you’re unsure as to whether your NIC’s have been properly identified; then typing ‘device 1’ or ‘device 2’; which would normally select NIC 1 or NIC 2 for modification will result in an error message like ‘Device Not Found’, or ‘Invalid Device’ – something like that.  Basically, the NICs onboard RAM is so messed up, the Proliant has no way of identifying what that memory controls

Fixing the NVRAM is a bit more fiddly and requires the bus/device/function values as discussed.  You have to directly address the NVRAM of the NIC because it’s the corruption of that same NVRAM that is preventing the Proliant as seeing the memory area as a NIC.  Once the NVRAM has been fixed then the Proliant can see it’s a NIC; and then you can proceed to flash the bootcode that tells the NIC how to operate.

So – there you have it.  It’s not for the faint hearted; but after doing a huge amount of reading and trying to assemble all the pieces myself – I got everything back up and running in about fourty minutes.  Both NICs are working just fine.