Server seems to crash after a little over 24 hours

[danooch13]

Hello,

My server seems to run fine for 24 hours plus. But as soon as it hits a little over 24 hours, it seems to segment fault and restart.

Code:

----------------------------------------------
CRASH: Sat Feb 13 18:38:45 EST 2010
Start Line: ./srcds_i686 -console -game cstrike +map de_dust2 +maxplayers 24 -port 27015 -tickrate 100 -debug
[New Thread 2980]
[New Thread 2984]
Core was generated by `./srcds_i686 -console -game cstrike +map de_dust2 +maxplayers 24 -port 27015 -t'.
Program terminated with signal 11, Segmentation fault.
#0  0x00ba506f in ?? ()
#0  0x00ba506f in ?? ()
No symbol table info available.
From        To          Syms Read   Shared Object Library
0x00710830  0x0072669f  Yes (*)     /lib/ld-linux.so.2
(*): Shared library is missing debugging information.
Stack level 0, frame at 0xbfe6d190:
eip = 0xba506f; saved eip End of Source crash report
----------------------------------------------

It always seems to happen creating the same thread number...

I am running alot of addons: eventscripts, mani, sourcemod, metamod, cssdm, strippermod, goremod. It just seems weird that it runs so fine for 24 hours plus then crashes.

Also, look at the attached picture after the server crashes and restarts itself. Look what happens to the server fps!

Any idea why this keeps happening. the server is composed of all new parts: core 2 duo 3 gig wolfdale, new 2 gigs of ram with heatsinks, new gigabyte motherboard....running on ubuntu 9.10 with the realtime kernel.

[danooch13]

It still keeps happening, now alot more....

Code:

----------------------------------------------
CRASH: Mon Feb 15 15:58:10 EST 2010
Start Line: ./srcds_i686 -console -game cstrike +map de_dust2 +maxplayers 24 -port 27015 -tickrate 100 -debug
[New Thread 6952]
[New Thread 6949]
Core was generated by `./srcds_i686 -console -game cstrike +map de_dust2 +maxplayers 24 -port 27015 -t'.
Program terminated with signal 11, Segmentation fault.
#0  0x0018506a in ?? ()
#0  0x0018506a in ?? ()
No symbol table info available.
From        To          Syms Read   Shared Object Library
0x00429830  0x0043f69f  Yes (*)     /lib/ld-linux.so.2
(*): Shared library is missing debugging information.
Stack level 0, frame at 0xbfce0ba0:
eip = 0x18506a; saved eip End of Source crash report
----------------------------------------------

[danooch13]

I have a feeling it could be sourcemod 1.3.1....

It seems to be happening alot more since then.

To try to prevent it, I have my server set to reboot at 4am everyday and i downgraded back down to 1.3.

Anyone have any ideas what else could be causing this?

I have posted my debug.log

[BehaartesEtwas]

srcds tend to crash from time to time, even without plugins... unfortunately we seem to have to live with it...

[danooch13]

Really, well that just is not fun....

Oh well, if everyone is living with it.

Is 2 gigs of ram enough, maybe i should go up to 4 to give it more room for memory relocation etc, and possibly prevent more segmentation faults?

[BehaartesEtwas]

have a look at the swap usage (i.e. run "vmstat 1" and look at "si" and "so" values). if numbers are 0 most of the time (and really close to 0 everytime else :-)) your ram is sufficient and upgrading will not improve the slightest bit. only if you have disabled swap completely your servers will crash if memory is full, else the will usually "only" lag...

my server crashes more often than once per day even though it restartes every night. I am looking for solutions for almost 2 years now and I have found nothing (well ok, in the beginning mani was crashing my server ever 2h or so, but apart from that...).

[danooch13]

Yeah the server never uses swap. Noticed that a while ago, I just figured may be a change from 800mhz ram to 1066 ram would be better.

I thought it was mani, but I have been using it all along. Is mani still a cause for server segfaults? I also have evenscripts, which I heard is a bad one also.

The funny thing is, if the server has no one in it for 24 hours, just bots, it never crashes. But when the server is running "hot" with like 12-20 players for 5-6 hours, thats when it crashes....right when everyone is having a good time.

For now I downgraded sourcemod to 1.3.0 and will see how that works, but I doubt a change.

Code:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
2  0      0 1366100  38956 355792    0    0     8     1  539  113 51  0 49  0
1  0      0 1366092  38956 355792    0    0     0     0 2948 5943 50  0 50  0
1  0      0 1366092  38956 355792    0    0     0     0 2961 5956 49  1 50  0
1  0      0 1366092  38956 355792    0    0     0     0 2973 5995 50  0 50  0
1  0      0 1366092  38956 355792    0    0     0     0 2961 5960 50  0 50  0
1  0      0 1366092  38956 355792    0    0     0     0 2951 5934 50  0 50  0
1  0      0 1366092  38956 355792    0    0     0     0 2958 5945 50  1 50  0
1  0      0 1366092  38956 355792    0    0     0     0 2974 5993 50  0 50  0
1  0      0 1366092  38956 355792    0    0     0     0 2956 5948 50  0 50  0
2  0      0 1366092  38956 355792    0    0     0     0 2964 5968 50  1 50  0
1  0      0 1366092  38956 355792    0    0     0     0 2987 6002 50  0 50  0
1  0      0 1366092  38956 355792    0    0     0     0 2942 5912 50  1 50  0
1  0      0 1366092  38956 355792    0    0     0     0 2944 5901 50  1 50  0
1  0      0 1366092  38956 355792    0    0     0     0 2967 5955 50  0 50  0
1  0      0 1366092  38956 355792    0    0     0     0 2952 5915 52  0 48  0
1  0      0 1366092  38956 355792    0    0     0     0 2960 5955 50  0 50  0
1  0      0 1366092  38956 355792    0    0     0     0 2961 5951 50  1 50  0
1  0      0 1366092  38956 355792    0    0     0     0 2951 5938 50  1 50  0
1  0      0 1366092  38956 355792    0    0     0     0 2971 5981 50  0 50  0
1  0      0 1366092  38956 355792    0    0     0     0 2962 5958 50  0 50  0

Another interesting thing...see the attached image. There are two different pids for srcds....not sure why....not that that has anything to do with anything, just wanted to see if anyone else sees that.

[danooch13]

Ok after a long worn out battle with this thing, I should have listened to my guy feeling in the beginning and what all of the experts said.

Still keeping my fingers crossed, the server is running perfectly. No crashes every few hours when alot of people are in the server.

It all seems to be caused by MANI!!!!

Unbelievable.

Switched over to sourcemod admin and all seems good.

I will report back if something changes.

Once again thanks for all the help!