Strange srcds freeze

[kossolax]

Hello,

I've recently changed my root server for a new one, with new hardware, supposed to be more performant. This said, I've some problems of freeze on my srcds install (css).
I moved every file, expect apache, php, sendmail conf for a fresh install (I was using lampp, now, I'm using apt-get install.) My old server was in 32bits, my new one in 64bits.
I was using Ubuntu 8.04 with a nx-server running with a gnome desktop to lunch game server. I never had lag, or problem with it.
Now, I'm using Ubuntu 10.04 with a x2go-server running with a gnome desktop...

Well, about the freeze, I've no idea where they can come, but I've done some test:
- Disabled apache during a CSS-match, no freeze
- Enabled after 25min of a CSS-match, a freeze of 2-3seconds occurred.
- Disabled. No more freeze.



Here is my apache/mysql/php config: http://www.kossolax.be/conf/

Another side is, my mail server. I don't know if it's linked. But it can send mail to outgoing email. But it cannot distribute it to local user.

This said, I after search, on some map with trigger_push, the server have some freeze with x2go session suspended, but not if it's running.
(DataTable warning: player: Out-of-range value (2000.000000) in SendPropFloat 'm_vecBaseVelocity', clamping.)

since, I really help, and really need to fix this as fast as possible. I'm offering 50$ Paypal, to the one who'll find me how to fix.

Thank you for reading.

[Polleke]

What kind of worker module are you running for apache? The prefork module could be responsible for such kind of lags. It might starts up a set of servers when receiving the internal dummy connection (http://wiki.apache.org/httpd/InternalDummyConnection). You could check your logs for these. Long shot though.

I also wouldn't recommend to run a desktop system. It makes troubleshooting such problems much more problematic.

[kossolax]

(01-25-2011, 04:06 AM)Polleke Wrote:  What kind of worker module are you running for apache? The prefork module could be responsible for such kind of lags. It might starts up a set of servers when receiving the internal dummy connection (http://wiki.apache.org/httpd/InternalDummyConnection). You could check your logs for these. Long shot though.

I also wouldn't recommend to run a desktop system. It makes troubleshooting such problems much more problematic.

I think, I've installed the prefork one. I'll trying to change to the worker.

Thanks, I'll tell you infos.



Edit:
Installing php5 after have installed the apache-worker, enforce me to remove the apache-worker for apache-prefork...

The following packages will be REMOVED:
apache2-mpm-worker
The following NEW packages will be installed:
apache2-mpm-prefork libapache2-mod-php5

What I'm supposed to do?

[Polleke]

Sorry, I added some confusion in my previous post. When I said worker, I meant actually the mpm module. So I actually asked which mpm is active. In your case this turns out to be the prefork one.

But you went a bit too fast and decided to switch from mpm-prefork to mpm-worker. The problem the installer complains about is that it doesn't want to have the PHP module run together with mpm-worker, which is correct, the two don't like each other (thread issues). So, stay with the prefork module for now.

So... instead of taking radical action first take a look if the dummy connection is the cause of your slowdown. I would suggest tail'ing your access log (/var/log/apache2/other_vhosts_access.log in your case) and try if you can match the slowdowns with the dummy connections (showing up as 127.0.0.1 connections).

And again.. I would really suggest running these test outside of X, it's unneeded and adding extra complication.

[kossolax]

You right,

root@ns205148:/var/log/apache2# grep -Ri "127.0.0.1"

Quote:./other_vhosts_access.log | grep -i "23/Jan/2011:22:44"
ns205148.ovh.net:80 127.0.0.1 - - [23/Jan/2011:22:44:03 +0100] "OPTIONS * HTTP/1.0" 200 152 "-" "Apache/2.2.14 (Ubuntu) (internal dummy connection)"
ns205148.ovh.net:80 127.0.0.1 - - [23/Jan/2011:22:44:20 +0100] "OPTIONS * HTTP/1.0" 200 152 "-" "Apache/2.2.14 (Ubuntu) (internal dummy connection)"
ns205148.ovh.net:80 127.0.0.1 - - [23/Jan/2011:22:44:21 +0100] "OPTIONS * HTTP/1.0" 200 152 "-" "Apache/2.2.14 (Ubuntu) (internal dummy connection)"

This said, I've many and many internal connection.
Yesterday, I've stopped apache to test it (22:15)
22:21 started apache, to see if lag occurred
22:24 a freeze of 2-3 seconds, apache stopped.

root@ns205148:/var/log/apache2# grep -Ri "127.0.0.1" ./other_vhosts_access.log | grep -i "23/Jan/2011:22:4"

What should I do now?

[kossolax]

Made some test, Here are result:

22:00 Server started using putty, calling players
22:05 A freeze occurred with 10 players connected, "internal dummy connection" at about the sime time (still many, and many on logs).
22:10 Starting a 7v7 match.
22:40 No lag/freeze. Disabled munin, and some cron. Enabled apache.
22h44 Had a freeze, apache stopped.
23h00 Server stopped, starting apache, enabling cron.


So, looks like apache, but what now?

[Polleke]

I'm not totally convinced yet it is apache, but we'll see. What kind of website are you running at / on your default host?

Munin (and other cron based services) would also have been a big suspect, but are apparently not involved. You mention you also have mail issues, it could be your mail queue (do you have a large queue not being processed perhaps?) kicking in, can you tail your mail log and see if it might be related.

[BehaartesEtwas]

do you have the game server running with realtime scheduling? this can (and will probably) help if any other process on the machine causes the lag by taking away the CPU. see "resched.sh" in my wiki linked in the signature...

[kossolax]

First, thank you for assisting me.

(01-26-2011, 06:00 PM)Polleke Wrote:  Munin (and other cron based services) would also have been a big suspect, but are apparently not involved. You mention you also have mail issues, it could be your mail queue (do you have a large queue not being processed perhaps?) kicking in, can you tail your mail log and see if it might be related.

Mail issue has been fixed with a full reinstall of mail servies the day after I posted.

(01-26-2011, 06:00 PM)Polleke Wrote:  I'm not totally convinced yet it is apache, but we'll see. What kind of website are you running at / on your default host?

On default host, nothing. I'm running many different virtual server, for every domain name link to apache.
There is also many mysql usage while apache is running, due to the amount of connection. That's could be the cause...

(01-26-2011, 08:14 PM)BehaartesEtwas Wrote:  do you have the game server running with realtime scheduling? this can (and will probably) help if any other process on the machine causes the lag by taking away the CPU. see "resched.sh" in my wiki linked in the signature...

I tested for a week setting other server to low process. I've set them to -19 and -1 Except, the match server which is set at 15 for performance.

[Polleke]

Ehhh... that's too many variables. There could be any number of websites running some kind of script doing something weird then.

No way to debug all that.

[kossolax]

Today, I'll install lampp and move database to it. Just to check.
If it doesn't fix. I'll return a 503 error to the most use host. To check. (It has about 1.1Go out of 1.3Go total for database). That'll slow connection, and less MySQL usage.

Once again, thanks for your assist.

[kossolax]

Sorry to up, but I've some news.


I've moved the most viewed site and database to an other root-server. However, game-server still have some freeze... Why?


http://ts-x.eu/munin/localdomain/localhost.localdomain/index.html

[BehaartesEtwas]

because it's a totally different problem... I would say, the problem is more like the game server itself (i.e. probably some plugin -> try removing them all) or the Linux kernel. also I would recommend not running a whole gnome session to launch the game server. try logging in with plain ssh and launch the game server inside a "screen". If you absolutely want to use NX, let it start only a gnome-terminal. but run the game servers still inside a screen and close the gnome-terminal. that's actually what I am doing usually...