We searched for multiple days why one of our cluster machines was hanging for a couple of seconds, before it was fenced. We found out that the iptables LOG did put to much output onto the serial.
I wrote a kernel module to create a hugh amount of messages, and the serial line got scrambled. I got this output on the serial:
[ 222.111488] Clocksource tsc unstable (delta = -60129537542 ns). Enable clocksource lover by adding clocksource_failover kernel parameter.
but this in the syslog:
[ 222.111488] Clocksource tsc unstable (delta = -60129537542 ns). Enable clocksource failover by adding clocksource_failover kernel parameter.
The romatic guy in me wants to keep the serial version 🙂