How I "fixed" Server hardlocks

[Thor Erik]

After poking around the web and picking up ideas on this issue this is what I came up with for a specific Ubuntu 9.10 Server system that runs 4 different servers:

4 sets of init.d scripts:
template:

Code:

#!/bin/bash
#replace <newuser> with the user you created above
SRCDS_USER="<newuser>"
#Do not change this path
PATH=/bin:/usr/bin:/sbin:/usr/sbin
#The path to the game you want to host. example = /home/newuser/dod
DIR=/home/<newuser>/HLDS/GG
DAEMON=$DIR/srcds_run
#Change all PARAMS to your needs.
# I have removed the -ip <xxx.xxx.xxx.xxx> portion from this command as the last update to L4D removed the need for it
# If you have issues with your game put the -ip string back into the PARAMS line below
PARAMS="-game cstrike -tickrate 100 -port 27017 +map gg_simpsons_arena +maxplayers 20 -autoupdate" #These options can vary a bit from game to game, google is your friend here.
NAME=SRCDS-GG #needs to be adjusted each time
DESC="Frag Mine Sungame"
case "$1" in
start)
echo "Starting $DESC: $NAME"
if [ -e $DIR ];
then
  cd $DIR
su $SRCDS_USER -l -c "screen -d -m -S $NAME $DAEMON $PARAMS"
  screen -d -m -S $NAME $DAEMON $PARAMS
else echo "No such directory: $DIR!"
fi
;;
stop)
if screen -ls |grep $NAME
then
     echo -n "Stopping $DESC: $NAME"
     kill `screen -ls |grep $NAME |awk -F . '{print $1}'|awk '{print $1}'`
    echo " ... done."
else
    echo "Coulnd't find a running $DESC"
fi
;;
restart)
if screen -ls |grep $NAME
then
     echo -n "Stopping $DESC: $NAME"
     kill `screen -ls |grep $NAME |awk -F . '{print $1}'|awk '{print $1}'`
     echo " ... done."
else
     echo "Couldn't find a running $DESC"
fi
echo -n "Starting $DESC: $NAME"
cd $DIR
screen -d -m -S $NAME $DAEMON $PARAMS
echo " ... done."
;;
status)
# Check whether there's a "srcds" process
# if "checkproc" is installed, you can use this:
# checkproc $DIR/srcds_run && echo "DODS-Server RUNNING" || echo "DODS-Server NOT RUNNING"
# (thx to commander)
ps aux | grep -v grep | grep $NAME > /dev/null
CHECK=$?
[ $CHECK -eq 0 ] && echo "SRCDS is UP" || echo "SRCDS is DOWN"
;;
*)
echo "Usage: $0 {start|stop|status|restart}"
exit 1
;;
esac
exit 0

and for making sure it was responding 4 php scripts:

PHP Code:

<?php
function source_query($ip)
{
$cut = explode(":", $ip);
$HL2_address = $cut[0];
$HL2_port = $cut[1];
$HL2_command = "\377\377\377\377TSource Engine Query\0";
$HL2_socket = fsockopen("udp://".$HL2_address, $HL2_port, $errno, $errstr,3);
fwrite($HL2_socket, $HL2_command);
$JunkHead = fread($HL2_socket,4);
$CheckStatus = socket_get_status($HL2_socket);
return $CheckStatus['unread_bytes'];
}

$ip="127.0.0.1:27017";
$query = source_query($ip); // $ip MUST contain IP:PORT

if($query == 0 )
{
system("service gg restart");
//echo "server down \n";
}
else
{
//echo "server up\n";
}
?>

and corresponding cronjobs running each of the php scripts once a minute.
not ideal, but it works well enough.
Tried to make a watchdog in C(instead of the above php script) but had issues with sockets not working properly(still looking into that though, will post the code if it works)

[BehaartesEtwas]

I did something similar in a shell script (using qstat), but that did not work in some (rare) cases, because the server still responded to the query command but did nothing else. Now I am using an rcon echo command to test if the server is alive. That seems to work for 100%. So if you have the same problem you might want to change it to some rcon query...

[Thor Erik]

(01-28-2010, 06:03 PM)BehaartesEtwas Wrote:  I did something similar in a shell script (using qstat), but that did not work in some (rare) cases, because the server still responded to the query command but did nothing else. Now I am using an rcon echo command to test if the server is alive. That seems to work for 100%. So if you have the same problem you might want to change it to some rcon query...

maybe you could share your solution then? I've spoken to a couple others who have similar issues, I personally haven't come across that issue though

[css]

I've tried to do the same thing with Monit plugin. Also there I noticed that there are some super rare cases when the server is frozen but it responds to normal queries.

Here's the plugin. The most useful part is the /etc/monit.d/srcds script.

http://css.setti.info/development/monit_srcds_plugin/

[lhffan]

@css

That monit script you have done. I just edit the parameters so they match my settings and then i do as below?

./etc/monit.d/srcds