Outage report: ipv6 broadcast packets and old Ubiquiti radio firmwares

On the week of June 4th we rolled out the Alpha 3 software update to Althea devices. Shortly thereafter the Clatskani network went down.

After fixing other issues in the release it became apparent that radio performance had dramatically dropped. Performance tests on the patched Alpha 3 release showed that the performance drop was not due to the software but due to the underlying radio network that Clatskanie ran on.

Packet dumps showed that radios would receive a ipv6 ping to ff02::1 and respond as the ipv6 specification demands, but in the process they would experience a large latency spike.

Delivering out of order packets as well as adding 300ms of latency to a connection that nominally has 40ms of latency. This caused throughput to drop from ~20mbps to 1–4mbps on the same hardware and physical link.

This issue was resolved by updating the firmware on all of the Nanobeam CPE devices as well as the Rocket M5 sector antenna.

Pings to the ff02::1 address where new in the Alpha3 firmware and used for peer discovery. They could be eliminated by re-factoring the peer discovery protocol, we are investigating if these sorts of issues are common enough to be worth the trouble.