srcds down

[cellexpert]

Hey Guys,

I'm having some problems with server crash
Work on Linux platform. One of the machines are running centos and othe arch linux
A common problem among the second machine is a server in each of these two machines fall.

When I realize that a server is offline consult the other on another machine and see that he is offline. Do not know if by coincidence.

But the curious who this fall is unusual because all processes are killed (my main scrip + srcds_run + srcds_linux and even the screen) do not believe that is because the srcds, has already been reinstalled and done tests with various plugins and have at least another 30 servers, counter-strike source in this same configuration (I own a small business lease with a games room).

I'm deducing that it's some sort of invasion. With this already changed passwords for ssh and tried to leave him out of the panel (Work with swiftpanel) running manually and the problem still persists.

Please, if anyone can help me an air, something that I can check.
I'll be grateful!

Sorry my bad english!

[BehaartesEtwas]

well, those high slot-count servers tend sometimes to crash, especially with plugins like deathmatch. does your server restart if you run "quit" via rcon? in that case it's more unlikely that it is a normal crash (as it would just restart the server), but still not impossible depending on the type of crash. how many servers do you have running on each machine? if there are many, maybe you are just out of memory and the process gets OOM killed by the kernel.

If you actually suspect an intrusion you will want to reinstall the whole machines. Once someone had actually root access you cannot be sure of anything any more on that system.

[cellexpert]

Indeed some falls can happen.
But beyond srcds_run srcds_linux and I have a script to keep the servers running even though these two processes die. Even I have cron jobs scheduled to kill the entire srcds 6:00 am and despite that they usually return.

The problem is that all processes actually die from the main script, the screen and the srcds. As if someone had entered the machine and killed all the processes of that User.

About the amount of server I do not put more than a custum server per core area unless they are small servers of 12 slots. On average it comes to a quadcore custum leave 5 to 6 servers on each machine.

Only one person has root access to machines. A trusted person.
Nevertheless I will try to change the passwords and secure them with me just for now.

Thanks

[cellexpert]

The servers crashed again: (


Just those two servers ... One on each machine at the same time.
both servers have been redone this morning and as I mentioned in last post, the passwords had already been changed and even removed the keys of authorized my machine, which I direct access "no password ".

Really do not know what else to do ...
I've been strengthening my request for help: (

Help me Please!

[BehaartesEtwas]

maybe your scripts are just broken...

again: if you actually suspect a break-in, there is no alternative to a complete and fresh re-installation. securing passwords does not lock out an experienced intruder if he got root access once.

[cellexpert]

Hi,

At the moment I am testing both machines with the native kernel distribution itself instead of using my custom kernel.
Soon some new post here

[cellexpert]

I changed the kernels for the natives of each OS and so far no crashes

I submit any news

Thanks

---

False alarm...

The server still crashed ...

[BehaartesEtwas]

did you try running those servers without any plugins?

[cellexpert]

Actually I have not tested without plugins, but does not happen the same problem with another server that is a copy of this and more strange is the fact that all the processes of dying and not just the srcds_linux.

Since my last posting I've reinstalled the server and check out all the mods and plugins until no errors were generated by SourceMod.
And after that the server still fell the same way again but today there was no further falls.

My impression is that the server is being attacked, this is done with some international blockade on the network that makes the server unavailable to the master servers. So looking forward to the processes the kernel just killing them.

After orangebox updates it was possible that the server stay online even without contact with the master server. Now no more ...

Sure, it's just a guess!

This data center has suffered many attacks international

---

I was hasty in saying that not happened yet.
Soon I just post here the inevitable happened

Notice that there is no longer any User Process running css27000

# ps aux | grep css27000
root 25494 0.0 0.0 3980 824 pts/0 S+ 01:42 0:00 grep css27000

I guess I'll shut it down and make a swiftpanel the script as root on loop for the server to not be killed again.
That should work until I find some solution

Thanks!

[BehaartesEtwas]

(02-16-2011, 01:39 PM)cellexpert Wrote:  My impression is that the server is being attacked, this is done with some international blockade on the network that makes the server unavailable to the master servers. So looking forward to the processes the kernel just killing them.

that all does not explain why the start script gets killed too. (plus it sounds kinda unlikely)

as 99% of server problems come from plugins, I still strongly recommend trying without all plugins (simply rename the addons directory), even though you don't believe this being the cause. simply try it out, it cannot hurt.

[cellexpert]

I reached a conclusion that may be flawed in some mod or plugin, which a malicious User should be exploring the server to lock up a point at which processes become zombies and are killed by own kernel.

This is an intermittent problem and make it difficult to leave the server without mods or plugins until something happens or not. Usually happens at night but not always the same schedules.

These are servers that are full all the time, so leave off all the mods and plugins will be tricky ...

Now developed a simple script in pear running as root to check the port 40 in 40 seconds if the process is running, otherwise ... it restarts the process with { su user --command=". / srcds_run ..." }

Thus, even if all User Processes die, it still restarts.

I know that does not solve the problem but at the least, make it less harmful.

Thanks!

[cellexpert]

The crashes continue: (

I do not know what else to do.

My last attempt was to run the main script as root for some probably even killing the attacker User Processes css27000 srcds still would be restarted.

Today was exactly the same thing.
Both srcds on two separate machines crashed while killing all processes in the included User the above script as root. But in another machine I have a firewall installed csf and could not help noticing in the logs as follows:

Quote:Feb 20 01:50:57 wcs01 lfd[9249]: *User Processing* PID:2023 Kill:0 User:rpcuser Time:591679 EXE:/sbin/rpc.statd CMD:rpc.statd
Feb 20 01:50:57 wcs01 lfd[9249]: *User Processing* PID:6688 Kill:0 User:css27060 Time:2731 EXE:/bin/bash CMD:/bin/bash
Feb 20 01:50:57 wcs01 lfd[9249]: *User Processing* PID:6715 Kill:0 User:css27060 Time:2729 EXE:/bin/bash CMD:/bin/sh /home/css27060/orangebox/srcds_run -game cstrike -ip xxx.x.xxx.xxx -nobots -port 27060 -maxplayers 36 -tickrate 66 +fps_max 100 +map de_dust2 +servercfgfile server.cfg +tv_enable 0 +tv_maxclients 15 +tv_port 27160

Note: As yet I do not know the motives of the falls, I decided to censor the IP adress

[BehaartesEtwas]

try running the server inside a screen, i.e.

Code:

system('su css27000 --command="cd /home/css27000/orangebox && screen -mdS css2700 ./srcds_run -game cstrike -ip xxx.x.xxx.xxx -port 27000 -maxplayers 48 -tickrate 66 +fps_max 300 +map de_dust2 +tv_enable 1 +tv_maxclients 15 +tv_port 27100 &>/dev/null &"');

else the server is still a child process of your script and thus might propagate some signal to the script which might lead to its termination.

[cellexpert]

strange that...

Today it happened twice already (and always a server on each machine)

I've really done everything that was possible.
Removed users who were not in use
I changed all the passwords of all machines
Deleted all the keys for ssh
Deactivated unnecessary services

I have left and not really anything else to do except format the drive
being that of a machine is newly installed. about 2 weeks

My biggest concern is whether this is being caused by some malicious hacker or is it just some sort of defense of own kernel.

If there is any flaw in the system, it would be bad for my business

thanks!

[BehaartesEtwas]

(02-21-2011, 11:43 AM)cellexpert Wrote:  Removed users who were not in use
I changed all the passwords of all machines
Deleted all the keys for ssh

again: that all does not help if your machine was really compromised. still by money is on some faulty plugin. there is only one way to find out: run the server without any plugins.