I started logging the packet queue to my server after the 300th segfault (with a one minute cronjob logging netstat -tulpan):
All seems well...
Mon Oct 5 18:42:01 UTC 2009:
udp 1572 0 0.0.0.0:27015 4826/srcds_i686
Mon Oct 5 18:43:01 UTC 2009
udp 1880 0 0.0.0.0:27015 4826/srcds_i686
...until a minute later...
Mon Oct 5 18:44:01 UTC 2009
udp 106372 0 0.0.0.0:27015 4826/srcds_i686
Mon Oct 5 18:45:01 UTC 2009
udp 111548 0 0.0.0.0:27015 4826/srcds_i686
Mon Oct 5 18:46:01 UTC 2009
udp 111340 0 0.0.0.0:27015 4826/srcds_i686
Minutes later the server crashes.
The Recv-Q column jumped from 1880 (prior to that it sits between 2000 and 0) to more than 100k!
Does this seem right? I guess what I'm saying is, could this just be a normal side effect of a crashing server or could this be the cause of a crashing server?
Any ideas of how to look into this further?
Thanks for your time.
More of the report without timestamps:
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111572 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111012 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111464 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 109244 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111016 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 110464 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 110804 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 110984 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111536 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111568 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111504 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 110952 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 110740 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111536 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111504 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111016 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 616 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 988 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 648 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 308 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1296 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1880 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1232 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 616 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 308 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 340 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 3144 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1572 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1880 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 106372 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111548 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111340 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 680 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1296 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1264 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1540 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1668 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 3824 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
* SERVER CRASHED *
All seems well...
Mon Oct 5 18:42:01 UTC 2009:
udp 1572 0 0.0.0.0:27015 4826/srcds_i686
Mon Oct 5 18:43:01 UTC 2009
udp 1880 0 0.0.0.0:27015 4826/srcds_i686
...until a minute later...
Mon Oct 5 18:44:01 UTC 2009
udp 106372 0 0.0.0.0:27015 4826/srcds_i686
Mon Oct 5 18:45:01 UTC 2009
udp 111548 0 0.0.0.0:27015 4826/srcds_i686
Mon Oct 5 18:46:01 UTC 2009
udp 111340 0 0.0.0.0:27015 4826/srcds_i686
Minutes later the server crashes.
The Recv-Q column jumped from 1880 (prior to that it sits between 2000 and 0) to more than 100k!
Does this seem right? I guess what I'm saying is, could this just be a normal side effect of a crashing server or could this be the cause of a crashing server?
Any ideas of how to look into this further?
Thanks for your time.
More of the report without timestamps:
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111572 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111012 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111464 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 109244 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111016 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 110464 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 110804 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 110984 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111536 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111568 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111504 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 110952 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 110740 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111536 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111504 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111016 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 616 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 988 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 648 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 308 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1296 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1880 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1232 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 616 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 308 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 340 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 3144 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1572 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1880 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 106372 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111548 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111340 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 680 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1296 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1264 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1540 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1668 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 3824 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
* SERVER CRASHED *