Mellanox ConnectX 3 / 4

Mellanox ConnectX 3 - 4

Updating firmware:

1. Boot Ubuntu USB stick
2. Enable universe repository (add universe after the security.ubuntu.com line in `/etc/apt/sources.list)`
3. `apt-get update && apt-get install ethtool mstflint`
4. Find PCI ID with `lspci`
5. Find PS ID of mellanox card: `mstflint -d pci-id-here q`
6. Download and unzip firmware for the correct PS ID from https://network.nvidia.com/support/firmware/firmware-downloads/
7. Flash the firmware: `mstflint -d pci-id-here -i name-of-unzip-firmware-file burn`

Quirks:

For some reason my ConnectX3 card started refusing to link at 10Gbit on one of the ports. Possibly related to using a 1Gbit fiber SFP in that port. `ethtool eth3` would only show 1000BaseX and not the 10000Base variants.
This can be fixed with `mstconfig -d pci-id-here reset` and a reboot.
The ConnectX 4 card sometimes loses its ports when reconnecting cables. Requiring a reboot.

Power usage:

ConnectX 3 Lx OCP 2 (MCX341A/MCX342A) seems to use about 3W with nothing connected.
ConnectX 4 Lx OCP 2 (CX4421A) seems to use about 10W with nothing connected.
Docs showing this is probably normal:
https://network.nvidia.com/files/doc-2020/connectx-3-pro-10gbs-ethernet-single-dual-sfp+-port-nic-user-manual-for-ocp.pdf
https://img-en.fs.com/file/user_manual/connectx-4-lx-ethernet-adapter-cards-user-manual.pdf