The disk on Ramoth was getting full with logs:

These turned out to be kernel logs, which were frequent repeating warnings of this form, with the majority (75% of a sample of 30,000 traces) passing through one of udpv6_sendmsg, udp_v6_push_pending_frames or udpv6_recv. The only open IPv6 socket was a UDP socket from PostgreSQL to itself, possibly for the stats collector.

This seems similar to #29 and there are other reports at large on the internet. Amusingly, this report refers back to #29! All that I've seen are using version 3.13.0, which may or may not be diagnostic, as it's the stable 14.04 kernel. I've not seen any reports from 16.04, which might indicate it's fixed there... or just that it's a rare enough bug that no one has (publicly) mentioned it for the latest version yet.

Rebooting cleared the state and the warnings stopped, although it created other problems because Ramoth wasn't warm any more and couldn't handle full load, leading to these outages on the 19th.

The issue seems likely to re-occur. Is it worth considering a backport kernel, or disabling IPv6 as suggested by one report?