wrench: a Linux command-line srcds control manager.

[jmomo]

Below is a srcds manager script that I've been using for awhile now to take care of my srcds installations. I figure someone else might find it useful.

It is a bash script. Linux only, no Windows.

wrench script
http://pastebin.com/CS81FnF9

README file
http://pastebin.com/mHWJm62R

bash-autocompletion script (makes tab-completion work)
http://pastebin.com/NhEGLWG8

Sample DB
http://pastebin.com/eMKUsADN

Download like so:

wget http://pastebin.com/download.php?i=CS81FnF9 -O- | dos2unix > wrench
wget http://pastebin.com/download.php?i=mHWJm62R -O- | dos2unix > wrench_README.txt
wget http://pastebin.com/download.php?i=NhEGLWG8 -O- | dos2unix > wrench.bash-autocompletion

Note that Pastebin inserts MS-DOS line feeds into their files, so you need to use a tool like dos2unix on the files after downloading.

Do "wrench help" for command examples.

To set up wrench and install a new server, do "wrench setup", and then "wrench install" when all setup requirements are met.

Read the README file for more information about installation. There are a number of dependencies which you will probably have to install to make it work.

--

Notable features:

wrench takes care of your installations from first-time install, startup, updates, stops, reconfiguration, all the way to uninstallation.
wrench was designed for maintaining multiple srcds installations on a single host.
Mandatory symbolically linked installations. All servers are linked servers.
Linked installations use "master" installations as their reference.
For each game type, you only need to download updates once, instead of N* the number of servers you have installed.
New installations take seconds to create once a master has been installed, because nothing needs to be downloaded.
Linked installations require dramatically less filesystem space than standard installations and are easier to backup and move around.
Fully automatic updates for all installed srcds server types.
Automatic update management of each linked installation, including warning users and restarting each installation after an autoupdate.
wrench uses tmux as a terminal multiplexer instead of screen.
wrench can start and stop all of your srcds installations in a single command.
wrench keeps configuration about each game type and server installation in a tiny SQLite database.
wrench has an autocleanup function to clean up old replay files, logs, download cache, and more.
wrench includes a srcds_run replacement called wrench_run mode.
wrench_run features crash notifications by email, automatic crash restarts, and a watchdog process which will restart hung/frozen servers.
wrench_run will automatically stop a server if it crashes too much, to protect your host from runaway crash-restarts.

--

user@host-->wrench help

wrench is a Source Dedicated Server (srcds) management tool for Valve's srcds-based game servers.

For more information, including full documentation, go here:
http://forums.srcds.com/viewpost/118741

Single-argument commands:
wrench install
wrench uninstall
wrench list
wrench listplayers
wrench autocleanup
wrench autoupdate
wrench bootstart
wrench stopall
wrench setup

Server-specific commands:
wrench start MyTarget
wrench stop MyTarget
wrench stopnow MyTarget
wrench restart MyTarget
wrench restartnow MyTarget
wrench console MyTarget
wrench status MyTarget
wrench showconfig MyTarget
wrench reconfig MyTarget
wrench rename MyTarget
wrench relink MyTarget
wrench delink MyTarget
wrench lockdown-master MyTarget
wrench unlock-master MyTarget
wrench update MyTarget
wrench updatenow MyTarget
wrench update-validate MyTarget
wrench update-validatenow MyTarget
wrench gametype MyTarget

[SirTiger]

This is some good stuff and probably the most comprehensive gameserver management batchscript I've seen so far. It features deduplication, console command execution, update monitoring and bulk management features.

Just out of curiousity: Are you sure SQLite isn't overkill in contrast to plain-text delimited files?

Thanks for sharing, very impressive!

[jmomo]

(04-12-2014, 02:50 AM)SirTiger Wrote:  This is some good stuff and probably the most comprehensive gameserver management batchscript I've seen so far. It features deduplication, console command execution, update monitoring and bulk management features.

Just out of curiousity: Are you sure SQLite isn't overkill in contrast to plain-text delimited files?

Thanks for sharing, very impressive!

I wrote this script in BASH on purpose, when it would have been a much better idea to implement it in something like python or perl. I did this mostly for fun and educational reasons.

BASH is very easy to work with and has a low barrier to entry when it comes to the end user being able to work with the existing code. However, it lacks IPC features, libraries, and you can't even source parameters from an external file without writing your own interpreter, unless you are okay with executing arbitrary commands in that file.

For reading in parameters from a file with the "source" command, that's easy. Writing data back out to that file, however, is a challenge. That's why it uses sqlite. This is the same reason that most apps on smartphones also use sqlite to manage their data.

SQL is a really easy language anyway. Even a noob can figure out how to insert, update, or delete a row pretty easily if they had to.

If you wanted to rip out out the sqlite stuff and replace it with just sourcing some config files, that would not be too hard to do.

Wrench writes to it's datbase is when:
New srcds game types are recorded.
New srcds installations are created.
Existing srcds installations are reconfigured.
On failed autoupdates, the AUTOUPDATE field gets set 0 so that it prevents further damage until the admin has figured out why the auto updates are repeatedly failing.

Writing to a config file in bash is kinda hard and there isn't a built-in mechanism for doing it, and bash doesn't really have libraries.

So, sqlite makes sense when you need to write config related data somewhere. It's not really overkill. It's just that there isn't an alternative.



Just a note; that original md5 isn't valid anymore since I updated the pastebin several times. I need to go post it on github or whatever, but I've been busy.

[SirTiger]

Fair point. Extra dependencies pose troubleshooting risks for the developer ('KISS'), but when you put it like that I do believe it is very beneficial not to re-invent the wheel. SQLite itself is lightweight enough to be embedded with MP3 players and whatnot, so performance-wise it doesn't matter either. (Isn't it great when lightweight tools require users to set up a full-fledged MySQL server first?)

[jmomo]

FYI here is the srcinfo for the Fistful of Frags game which was released recently. Just install a new game type by using the "wrench gametype add" command and put in the following:

GAMETYPE = fof
GAMENAME = Fistful of Frags
UPDATER = SteamCMD
HLDSID = fof
SERVER_APPID = 295230
CLIENT_APPID = 265630
BINDIRSUBD =
STEAMINF = fof/steam.inf

Note that there is a bug regarding the "steam_appid.txt" file in the root of the install. Until they fix this, you will need to delete the link and replace it with a real file via "echo 295630 > steam_appid.txt". (this is no longer a problem)

Reference: http://steamcommunity.com/app/265630/discussions/6/540741131821947464/

[jmomo]

If you like tab-completing your commands (and you should), you will want this bash auto-completion script. It will help your commands get auto-completed.

http://pastebin.com/NhEGLWG8

Note that the APPDIR is hard coded into this, along with the SQLDB file, so change as needed.

[jmomo]

I just released wrench version 1.17.

There are very significant changes.



If you are an existing wrench user, you will need to upgrade your DB before you can use the new wrench. I have added some fields to the database.

I don't yet have that upgrade script ready. I'll be doing it shortly and will post it here.



The old wrench was broken in many ways. The setup procedure was guaranteed to fail, things were missing, and there were some serious bugs in there.

The new version of wrench has been tested on different platforms, including Debian stable, unstable, Ubuntu, and CentOS. Unfortunately, CentOS 6.5 users are pretty much screwed due to a bug in libevent that I can't do anything about. You can probably upgrade your tmux and libevent to a newer version so that it works, but that's not something I can help you with. When CentOS 7 comes out, this probably won't be an issue anymore. Read the README file for more information if you really want to get wrench working on CentOS.

wrench was mostly tested on Debian and Ubuntu. Other distributions may have issues. wrench will not work on BSD or OSX, but it could be ported with some efforts.


Here's a basic list of changes:

I've renamed the file from "wrench.sh" to simply "wrench".

wrench now includes a srcds_run replacement called wrench_run mode. wrench_run features crash notifications by email, automatic crash restarts, a watchdog process which will restart hung/frozen servers, will dump console logs on crashes, and more. wrench_run will also stop your server if it crashes too much, to protect your host from runaway crash-restarts.

Setup will now install SteamCMD for you.

You can now add/change/delete game types directly via the "gametype" argument.

The "GAMEARG" field in the database was added. This is the argument passed to "-game" when starting srcds_linux. Previously, we used HLDSID most of the time, but that was just wrong. If you are still using HLDSID in your STARTUPBIN_ARGs, wrench will warn you to fix this on each startup.

Mail messages now include more information.

Fistful of Frags is now included in the default game database.

Serious improvements to input validation of commands.

Added a "help" argument to get some basic info.

Tons of other little internal changes.



Give it a try and let know what you think.

[jmomo]

I just uploaded wrench v1.18. The README file got some improvements. Mostly this was fixes for the cleanup trap, but it also let's users use the qakestat binary that comes with Debian's qstat package if they have that installed instead of qstat.

There was a big mistake in the README file where I was recommending users set autocleanup to run EVERY minute of the 5th hour, instead of once at 0500. This has been fixed. Be sure to update your cron tab if you used that example. The upside is that there isn't a lot of harm in doing so, and I added a lockfile to autocleanup to prevent contention problems.

[jmomo]

I just uploaded wrench v1.21. I fixed a really ugly updater bug that would cause the updater to try forever instead of automatically giving up like it should have.

[Rhapsody]

Hey, can you add Garry's Mod support to wrench? This is by far one of the more comprehensive scripts for Linux servers, since I'm not really looking for a web panel.

[jmomo]

(10-05-2014, 09:06 AM)Rhapsody Wrote:  Hey, can you add Garry's Mod support to wrench? This is by far one of the more comprehensive scripts for Linux servers, since I'm not really looking for a web panel.

You can add it yourself right now.

Use the "wrench gametype add" command to add a new game type.

You will need to look up the information needed, such as the server and client AppIDs.

IMPORTANT: At this time, you won't be able to use "Garry's Mod" as the Game Name because it has a ' character in the name, and that will cause the database insert command to blow up since I'm not doing input validation. I'll have to fix that some day. Just use "Garrys Mod" instead.

I will do a test install later and post a copy of the install config.

[jmomo]

GAMETYPE = gmod
GAMENAME = Garrys Mod
UPDATER = SteamCMD
HLDSID =
SERVER_APPID = 4020
CLIENT_APPID = 4000
GAMEARG = garrysmod
BINDIRSUBD =
STEAMINF = garrysmod/steam.inf

The rest should be defaults or whatever you prefer.

[jmomo]

Note that it looks like Garry's Mod's steam.inf file is broken. The UpToDate URL version says "140419" is current, but the steam.inf file version is "140710".

POSTCHECK_LOCAL_VER=140710, POSTCHECK_REMOTE_VER=140419

This means that automatic updating won't work. You will need to manually update when new versions are released.

Because of how badly the game publishers seem to manage the steam.inf files (TF2, FoF, and CSGO have all broken at least once in the last two years), I will probably have to change the autoupdate system so that it stores the last known version instead of using steam.inf.

This also brings up the issue in wrench that I don't have a switch for enabling or disabling autoupdates on an individual gametype level -- it's either globally enabled or disabled. I need to add this.

What this means is that if you are doing auto updates, and you probably are, you will need to do "wrench reconfig gmod-MASTER" and set ALLOWUPDATE=1 each time you want to update... but then the autoupdater is going to try and autoupdate it and fail and then set it to 0 again.

I'll look into fixing this.

[jmomo]

Oh, also it looks like a lot of people run the beta versions of Garry's mod, so the "-beta dev" argument needs to be added when running SteamCMD, but I don't currently support that. I'll have to add a field for arbitrary arguments to be added for SteamCMD on individual game types.

[Rhapsody]

I tried installing the server over SSH, and when I installed the Garry's Mod server using the parameters you gave me, this is what happened;

http://smashinglim.es/i/OMuvnN.png
http://smashinglim.es/i/2Pv9xs.png
http://smashinglim.es/i/iRdZ4Y.png

These are the settings I used in 'wrench' that would be relevant to this error;

Code:

APPDIR=~/serverstuff/servers # This is the installation directory where wrench adds new installations.
SQLDBFILE=$APPDIR/wrench.db # The location of the SQLite database file.
STEAMCMD_BIN=~/bin/steamcmd/steamcmd.sh
SRCDS_LINUX="srcds_linux"
MAILTO="[REDACTED]" # The default MAILTO address, mostly needed for autoupdate notifications.
REPLAY_ENABLED=0 # Disable replay related features by setting this 0.
REPLAYBASEDIR=/var/www/srcds-replays # The web server replay base directory.