Debian with game servers

[bakak]

Hello everyone, running debian i have a few processes which tend to crash occasionally (game servers), which end up using 100% cpu. I'm looking for a program or script to check the cpu usage of a list of processes by name and if they are at 100% for more than X time, say 30 seconds, kill them. I tried ps-watcher but wasn't able to determine how to accomplish this. Just killing the process at 100% usage wont work as it will hit that for brief periods during normal operation.

Any help is greatly appreciated!

[BehaartesEtwas]

I did something like this once, I can post some script tomorrow. You will need to modify it for your purpose though...

[bakak]

waiting, thanks ..

[BehaartesEtwas]

The script consists of two parts. First part is the script you start, let's call it 'watcher.sh':

Code:

#!/bin/sh

while true; do

rm -rf /tmp/watcher > /dev/null 2>&1
mkdir /tmp/watcher

LIST=`ls -1rt /proc`

for p in $LIST; do
  if [ -e /proc/$p ]; then
    find /proc -maxdepth 1 -name $p -exec iterator.sh \{\} \;
  fi
done

echo "--------------------------------"

sleep 60

done

Second part is 'iterator.sh', which is called every 60 seconds for each process from watcher.sh:

Code:

!/bin/sh

#==============================

CPUPERCENT=95

#==============================

PID=`echo $1 | sed -e "s/\/proc\///"`

INFO=`ps aux | egrep "^[^ ]* *$PID" | sed -e "s/ +/ /"`

USER=`echo $INFO | cut -d " " -f 1`
CPUP=`echo $INFO | cut -d " " -f 3`
VMEM=`echo $INFO | cut -d " " -f 5`
CPUT=`echo $INFO | cut -d " " -f 10`

ICPUP=`echo $CPUP | sed -e "s/\..*//"`

if [ $ICPUP -gt $CPUPERCENT ]; then

  if [ -e /tmp/watcher/$PID ]; then
    kill -9 $PID
    rm -f /tmp/watcher/$PID
  else
    touch /tmp/watcher/$PID
  fi

else

  if [ -e /tmp/watcher/$PID ]; then
    rm -f /tmp/watcher/$PID
  fi
fi

I hope this works, I did not test this actually in this form (my script does something similar but not the same, so I had to rewrite parts). The idea is the following: if a process is found running with a cpu percentage of greater then 95% (can be configured in the second file), a marker file is created in /tmp/watcher. If 60 seconds later the process still has over 95% cpu time (and the marker file exists) the process get killed. Else the marker file gets removed.

So processes will get killed if running with more than 95% cpu time for at least 60 seconds. Note, accidentally a process can also get killed if only reaching 95% very shortly but two times excatly the moment the script is running... But this is very unlikely if your processes don't run often with high cpu%!

[bakak]

Thank you very much, I'll try it


First test result

Code:

/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected
/root/cpucheck/iterator.sh: line 20: [: -gt: unary operator expected

I have this error

[BehaartesEtwas]

uhm, maybe somthing went wrong when "parsing" the output of ps ax. Can you put in line 19 the following lines:

Code:

echo PID: $PID
echo INFO: $INFO
echo CPUP: $CPUP
echo ICPUP: $ICPUP

and post again what you get? (That part wasn't modified and works for me...)

[bakak]

new iterator.sh

Code:

#!/bin/sh

#==============================

CPUPERCENT=100

#==============================

PID=`echo $1 | sed -e "s/\/proc\///"`

INFO=`ps aux | egrep "^[^ ]* *$PID" | sed -e "s/ +/ /"`

USER=`echo $INFO | cut -d " " -f 1`
CPUP=`echo $INFO | cut -d " " -f 3`
VMEM=`echo $INFO | cut -d " " -f 5`
CPUT=`echo $INFO | cut -d " " -f 10`

ICPUP=`echo $CPUP | sed -e "s/\..*//"`

echo PID: $PID
echo INFO: $INFO
echo CPUP: $CPUP
echo ICPUP: $ICPUP

if [ $ICPUP -gt $CPUPERCENT ]; then

  if [ -e /tmp/watcher/$PID ]; then
    kill $PID
    rm -f /tmp/watcher/$PID
  else
    touch /tmp/watcher/$PID
  fi

else

  if [ -e /tmp/watcher/$PID ]; then
    rm -f /tmp/watcher/$PID
  fi
fi

and result

Code:

CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: tty
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: timer_list
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: sysvipc
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: swaps
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: stat
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: softirqs
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: slabinfo
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: scsi
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: partitions
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: pagetypeinfo
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: net
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: mtrr
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: mounts
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: modules
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: misc
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: meminfo
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: locks
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: loadavg
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: kpageflags
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: kpagecount
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: key-users
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: kcore
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: kallsyms
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: irq
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: ioports
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: iomem
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: interrupts
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: ide
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: fs
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: filesystems
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: fb
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: execdomains
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: driver
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: dma
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: diskstats
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: devices
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: crypto
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: cpuinfo
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: cmdline
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: cgroups
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: buddyinfo
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: asound
INFO:
CPUP:
ICPUP:
/root/cpucheck/iterator.sh: line 25: [: -gt: unary operator expected
PID: 21086
INFO: root 21086 0.8 0.0 10156 1296 pts/12 S+ 10:37 0:00 /bin/sh ./watcher.sh
CPUP: 0.8
ICPUP: 0
PID: 21072
INFO: www-data 21072 0.0 0.0 192988 5660 ? S 10:36 0:00 /usr/sbin/apache2 -k start
CPUP: 0.0
ICPUP: 0
--------------------------------

this script check all process, maybe just check srcds_* or srcds_linux only?

[BehaartesEtwas]

um ok, apparently the system I am running the script on is somewhat less strict. replace the new echos with:

Code:

if [ -z "$INFO" ]; then
  exit
fi

[bakak]

yes. now output;

Code:

~/cpucheck# ./watcher.sh
--------------------------------
--------------------------------
--------------------------------
--------------------------------
--------------------------------

I think working

thanks for help

[BehaartesEtwas]

does it also kill the processes? :-)

[bakak]

hmm working but not kill to process

this process stay 2-3 min with 100+ cpu but not killing

[BehaartesEtwas]

any output of the script there? try the following modification (which adds some debug output):

Code:

#!/bin/sh

#==============================

CPUPERCENT=100

#==============================

PID=`echo $1 | sed -e "s/\/proc\///"`

INFO=`ps aux | egrep "^[^ ]* *$PID" | sed -e "s/ +/ /"`

USER=`echo $INFO | cut -d " " -f 1`
CPUP=`echo $INFO | cut -d " " -f 3`
VMEM=`echo $INFO | cut -d " " -f 5`
CPUT=`echo $INFO | cut -d " " -f 10`

ICPUP=`echo $CPUP | sed -e "s/\..*//"`

if [ -z "$INFO" ]; then
  exit
fi

if [ $ICPUP -gt $CPUPERCENT ]; then
  echo $PID is at $ICPUP percent load which is higher than $CPUPERCENT

  if [ -e /tmp/watcher/$PID ]; then
    echo ... trying to kill $PID
    kill $PID
    rm -f /tmp/watcher/$PID
  else
    echo ... first time, adding $PID to watchlist
    touch /tmp/watcher/$PID
  fi

else

  if [ -e /tmp/watcher/$PID ]; then
    echo $PID is at $ICPUP percent load which is NOT higher than $CPUPERCENT, so removing it from watchlist
    rm -f /tmp/watcher/$PID
  fi
fi

[bakak]

no change

CPUPERCENT=90

process running 90+ cpu but not killed

watcher running


cpu information, "ps" instead of "top" command, we could not draw?

because the "ps" the information coming from the same userdir have more than one

example; output of ps aux | egrep "^[^ ]* *$PID" | sed -e "s/ +/ /"

onur 30471 0.0 0.0 24820 1180 ? Ss Dec06 0:00 SCREEN -c /home/onur/.screenrc-a2s-cstrike-18-27015 -mdLS a2s-2-onur-212.175.66.109-27015 bash -c ./srcds_run -console ip 212.175.66.109 -port 27015 -game cstrike -dir /home/onur/212.175.66.109-27015/srcds_ns +map de_dust2 +maxplayers 22 -tickrate 66 +sv_stats 0

onur 30475 0.0 0.0 10336 1504 pts/8 Ss+ Dec06 0:00 /bin/sh ./srcds_run -console ip 212.175.66.109 -port 27015 -game cstrike -dir /home/onur/212.175.66.109-27015/srcds_ns +map de_dust2 +maxplayers 22 -tickrate 66 +sv_stats 0

onur 30489 6.9 1.4 293844 153660 pts/8 Sl+ Dec06 189:15 ./srcds_i686 -console ip 212.175.66.109 -port 27015 -game cstrike -dir /home/onur/212.175.66.109-27015/srcds_ns +map de_dust2 +maxplayers 22 -tickrate 66 +sv_stats 0


here, "cpu" value 6.9 but

top -c output;

30489 onur -99 0 286m 150m 16m S 16 1.5 189:22.36 ./srcds_i686

here, "cpu" value is 16

:/

[BehaartesEtwas]

yeah, cpu% of srcds isn' t a well defined quantity, because srcds is switching between running and sleeping state at a higher frequency (or at least with the same) as the kernel measures cpu%. this way you cannot really rely on that number. But if srcds is really hung I suppose cpu load will be 100% in all programs.

try running an "idler" (see the howto on my wiki) as a test, it should have permanently 100% load and thus get killed...

[bakak]

.. oh thanks