Boosting SRCDS for linux without a RT kernel

[mrzor]

Hey guys !

I've been quite reluctant to operate my srcds server on a RT patched kernel - running GRSEC because the server isn't strictly dedicated to TF2 for one, and it has public facing services (HTTP mirror, SSH).

Features

So I've been playing with what I had under my hands. The following is the script I use, and basically it :

• gives a high priority (nice) to srcds
• gives a RT priority (FIFO) to srcds, with high priority
• uses CFQ IO-scheduler to ensure TF2 has first hand in disk access
• sets task affinity of SRCDS on CPU2, mysql on CPU1 and nginx on CPU3. tweak depending on what you use. I advise leaving CPU0 for other tasks like crons and the like, and for IRQ servicing. Also, don't use irqbalance, or set its CPU mask to keep off your srcds processor.
• sets the clock source to HPET
  

Script

Code:

#!/bin/sh

### srcds_boost.sh, r2

### options
SRCDS_USER="sorciers"
TFDATA_BLOCK_DEVICE="sda"
### options ends here, beware if you customize anything below

# this script uses schedtool where taskset could be used
# at the cost of a three additional loops.

# ensure system clocksource is HPET if available
CLOCKPATH=/sys/devices/system/clocksource/clocksource0/
grep hpet $CLOCKPATH/available_clocksource 2>&1 >/dev/null
if [ $? == 0 ]; then
    if [ `cat $CLOCKPATH/current_clocksource` != "hpet" ]; then
    echo "hpet" > $CLOCKPATH/current_clocksource
    fi
fi

# ensure sda IO scheduler is CFQ if available
IOQPATH=/sys/block/$TFDATA_BLOCK_DEVICE/queue/scheduler
grep cfq $IOQPATH 2>&1 >/dev/null
if [ $? == 0 ]; then
    grep '\[cfq\]' $IOQPATH 2>&1 >/dev/null
    [ $? != 0 ] && echo "cfq" > $IOQPATH
    export CFQ="1"
else
    export CFQ=""
fi

# put SRCDS server on CPU2
# - FIFO scheduling, high prio
# - not nice
# - IO scheduling to realtime, high prio (if CFQ available)
pids=`ps -u $SRCDS_USER -L | grep srcds | grep -Eo '^ ?[0-9]+ +([0-9]+)' | grep -Eo '([0-9]+)$' `
chrt -p -a -f 30 $pids
schedtool -a 2 -n -10 $pids
renice -n -10 -p $pids >/dev/null
[ $CFQ == "1" ] && ionice -c 1 -n 2 -t -p $pids

# put MYSQL on CPU1
pids=`ps ax | grep mysqld | grep -v grep | cut -f 1 -d ' '`
schedtool -a 1 $pids

# put NGINX on CPU3
pids=`ps ax | grep nginx | grep -v grep | cut -f 1 -d ' '`
schedtool -a 3 $pids

# other CPU intensive daemons should be
# using CPU0,1,3

# CPU0 should be busy serving interrupts and most
#      kernel madness

Install

1. This script requires root access and the schedtool program.
2. This script requires manual tweaks. Read it, and make sure you understand what it does. Look up google. Stuff to look for :
   • Nginx/Mysql are just a demo of what you could have running aside of your srcds. Use htop/top on your box to see what is hitting on your resources, and patch script accordingly.
   • Check
   • IOQPATH variable might need to be changed for your system (hda, vda, sdb, dm0, ...). If your game accesses multiple disks, you need to duplicate the block as often as needed. Otherwise you can also change TFDATA_BLOCK_DEVICE
     
3. Copy/paste this in a file such as /root/boost_srcds.sh
4. chmod u+x boost_srcds.sh
   

Now either your server has autorestart or it hasn't. If it has, you want to schedule this script every 10 minutes using cron, so that in the event of a restart, the new server will get boosted soon enough. Here's a cron line for the lazy :

Code:

*/10 * * * * /root/boost_srcds.sh >/dev/null

Otherwise, you can just manually call it after starting the server.

One could extend this script to use a mix of PS and sudo to launch the server using the appropriate user. If you guys really want such a feature I could demo it, but anyone is welcome to show off their shell-fu and share the relevant bits

Last notes

So far so good, so .. just sharing

In any case, CC welcome (or rep ?).

Changelog

• 2011-10-05 : R1
• 2011-10-14 : R2. Improved process finding, and an additional option.
  

[mml]

nice work, thanks

[Mike]

Seems to be pretteh nice. Thanks for sharing mate.

[Anders_91]

How do I get this script working?

[mrzor]

(10-09-2011, 08:56 PM)Anders_91 Wrote:  How do I get this script working?

Edit'd this for you. If you don't get what some parts are doing, do your research. Googling for "man nice", "man ionice", "man schedtool" will provide much insight.

[sarmadsoomro]

This is not working in CentOS.

Can u guide me how to make it run.

Thanks

[mrzor]

(10-10-2011, 03:21 PM)sarmadsoomro Wrote:  This is not working in CentOS.
Can u guide me how to make it run.
Thanks

What is not working ?

[Anders_91]

(10-09-2011, 10:56 PM)mrzor Wrote:  

(10-09-2011, 08:56 PM)Anders_91 Wrote:  How do I get this script working?

Edit'd this for you. If you don't get what some parts are doing, do your research. Googling for "man nice", "man ionice", "man schedtool" will provide much insight.

I don't know how to do it. Would you like to help me?

[sti4n]

Hi. i have a small problem. it says "ERROR: User name does not exist. And Unexpected operator 3 times with 3 different numbers" when I'm trying to launch the sh file..

copy pasted from putty:

Code:

stian@stian-X9SCI-X9SCA:/root$ sudo ./boost_srcds.sh
[: 13: 0: unexpected operator
[: 24: 0: unexpected operator
ERROR: User name does not exist.
********* simple selection *********  ********* selection by list *********
-A all processes                      -C by command name
-N negate selection                   -G by real group ID (supports names)
-a all w/ tty except session leaders  -U by real user ID (supports names)
-d all except session leaders         -g by session OR by effective group name
-e all processes                      -p by process ID
T  all processes on this terminal     -s processes in the sessions given
a  all w/ tty, including other users  -t by tty
g  OBSOLETE -- DO NOT USE             -u by effective user ID (supports names)
r  only running processes             U  processes for specified users
x  processes w/o controlling ttys     t  by tty
*********** output format **********  *********** long options ***********
-o,o user-defined  -f full            --Group --User --pid --cols --ppid
-j,j job control   s  signal          --group --user --sid --rows --info
-O,O preloaded -o  v  virtual memory  --cumulative --format --deselect
-l,l long          u  user-oriented   --sort --tty --forest --version
-F   extra full    X  registers       --heading --no-heading --context
                    ********* misc options *********
-V,V  show version      L  list format codes  f  ASCII art forest
-m,m,-L,-T,H  threads   S  children in sum    -y change -l format
-M,Z  security data     c  true command name  -c scheduling class
-w,w  wide output       n  numeric WCHAN,UID  -H process hierarchy
chrt: invalid option -- 'a'

chrt - manipulate real-time attributes of a process.

Set policy:
  chrt [options] <policy> <priority> {<pid> | <command> [<arg> ...]}

Get policy:
  chrt [options] {<pid> | <command> [<arg> ...]}


Scheduling policies:
  -b | --batch         set policy to SCHED_BATCH
  -f | --fifo          set policy to SCHED_FIFO
  -i | --idle          set policy to SCHED_IDLE
  -o | --other         set policy to SCHED_OTHER
  -r | --rr            set policy to SCHED_RR (default)

Options:
  -h | --help          display this help
  -p | --pid           operate on existing given pid
  -m | --max           show min and max valid priorities
  -v | --verbose       display status information
  -V | --version       output version information

[: 34: ==: unexpected operator

[mrzor]

(10-14-2011, 12:48 PM)sti4n Wrote:  Hi. i have a small problem. it says "ERROR: User name does not exist.

Very true. I forgot to mention that ... Look up the "sorciers" string, and replace that by the user your srcds is running at. Maybe "steam", or "uberuser", whatever.

This should hopefully fix it ...

EDIT: I added an option for it, and also fixed a bug for srcds process detection. Bug might still hit mysql/nginx process matching for now if their PID is below 10000 in some cases. Definitely try latest version

[Mike]

I like this.

[sti4n]

(10-14-2011, 07:49 PM)mrzor Wrote:  

(10-14-2011, 12:48 PM)sti4n Wrote:  Hi. i have a small problem. it says "ERROR: User name does not exist.

Very true. I forgot to mention that ... Look up the "sorciers" string, and replace that by the user your srcds is running at. Maybe "steam", or "uberuser", whatever.

This should hopefully fix it ...

EDIT: I added an option for it, and also fixed a bug for srcds process detection. Bug might still hit mysql/nginx process matching for now if their PID is below 10000 in some cases. Definitely try latest version

Thanks. ok. that got me further.. but now i got a new error...

Code:

stian@stian-X9SCI-X9SCA:/root$ sudo ./boost_srcds.sh
[: 20: 0: unexpected operator
[: 31: 0: unexpected operator
chrt: invalid option -- 'a'

option -a doesn't work. and i can't find a way to fix that...

[mrzor]

(10-16-2011, 12:04 AM)sti4n Wrote:  

Code:

stian@stian-X9SCI-X9SCA:/root$ sudo ./boost_srcds.sh
[: 20: 0: unexpected operator
[: 31: 0: unexpected operator
chrt: invalid option -- 'a'

option -a doesn't work. and i can't find a way to fix that...

Try and remove it. I'm contemplating whether it was a good idea (-a changes priority of all threads as well). I got a nasty plugin induced crash (melee mode im looking at your) that made it impossible to kill the server. Hard reboot was needed.

BE CAREFUL WITH MY SCRIPT, IT'S POTENT !

[sti4n]

(10-16-2011, 12:08 PM)mrzor Wrote:  

(10-16-2011, 12:04 AM)sti4n Wrote:  

Code:

stian@stian-X9SCI-X9SCA:/root$ sudo ./boost_srcds.sh
[: 20: 0: unexpected operator
[: 31: 0: unexpected operator
chrt: invalid option -- 'a'

option -a doesn't work. and i can't find a way to fix that...

Try and remove it. I'm contemplating whether it was a good idea (-a changes priority of all threads as well). I got a nasty plugin induced crash (melee mode im looking at your) that made it impossible to kill the server. Hard reboot was needed.

BE CAREFUL WITH MY SCRIPT, IT'S POTENT !

ok. i will try that. and i will be careful

i have a feeling we live in completely different time zones xD

[Monk]

Adding 'nice' to a process on top of 'chrt' is irrelevant. In linux_sched.c, chrt gives more of a 'benefit' to process scheduling than 'nice' could ever do. Personally in my tests, a process with SCHED_FIFO (unixbyte, dd and others) with a nice of -20 to 20 gives no noticeable changes. I haven't found any cases where nice is even used anymore, personally I think it's ancient cruft from the days of SysV/ancient BSD systems.

You should simply remove it and use say, ionice.