[Monitoring] Server statistics (CPU, FPS, Traffic, Players, Uptime) with Munin

[css]

Server statistics with Munin

Website
http://css.setti.info/development/munin_srcds_plugin/

Download
http://css.setti.info/download/munin_srcds_plugin/munin_srcds_plugin.tar.gz

Overview

These srcds_* plugins graph general server information from Half-Life 2 based
servers. The plugins can be used with any gametype, which work under Valve's
Source engine (ie. srcds).

Some of the supported gametypes are Team Fortress 2, Counter-Strike: Source,
Day of Defeat: Source and all mods based on these games.

Preview

[cookies911]

THis Should Be stickied

[Firefox005]

When I use these plugins it causes my server to be literally flooded with munin-plugin processes, upwards of 500 before they started to choke out top. I am using munin version 1.2.6 and it works fine until I try to use the srcd_* plugins.

Here is my munin-node file. The only thing of note is that my rcon password contains a ! so escaped it with a '\', this worked fine on the CLI as such rcon -Ppassword -a1.2.3.4 -p27015 stats .

[srcds_cpu_69.**.36.4_27015]
env.srcdspass ****************

[srcds_fps_69.**.36.4_27015]
env.srcdspass ****************

[scrds_inout_69.**.36.4_27015]
env.srcdspass ****************

[scrds_players_69.**.36.4_27015]
env.srcdspass ****************

[scrds_uptime_69.**.36.4_27015]
env.srcdspass ****************

[srcds_cpu_69.**.36.7_27015]
env.srcdspass ****************

[srcds_fps_69.**.36.7_27015]
env.srcdspass ****************

[scrds_inout_69.**.36.7_27015]
env.srcdspass ****************

[scrds_players_69.**.36.7_27015]
env.srcdspass****************

[scrds_uptime_69.**.36.7_27015]
env.srcdspass ****************

[srcds_cpu_69.**.36.6_27015]
env.srcdspass ****************

[srcds_fps_69.**.36.6_27015]
env.srcdspass ****************

[scrds_inout_69.**.36.6_27015]
env.srcdspass ****************

[scrds_players_69.**.36.6_27015]
env.srcdspass ****************

[scrds_uptime_69.**.36.6_27015]
env.srcdspass *****************

[srcds_cpu_69.**.36.3_27015]
env.srcdspass ****************

[srcds_fps_69.**.36.3_27015]
env.srcdspass ****************

[scrds_inout_69.**.36.3_27015]
env.srcdspass****************

[scrds_players_69.**.36.3_27015]
env.srcdspass ****************

[scrds_uptime_69.**.36.3_27015]
env.srcdspass ****************



munin-node-configure give me:

Code: Select all
srcds_cpu_ | yes | 69.**.36.3_27015 69.**.36.6_27015 69.**.36.7_27015 69.**.36.4_27015
srcds_fps_ | yes | 69.**.36.3_27015 69.**.36.6_27015 69.**.36.4_27015 69.**.36.7_27015
srcds_inout_ | yes | 69.**.36.4_27015 69.**.36.6_27015 69.**.36.3_27015 69.**.36.7_27015
srcds_players_ | yes | 69.**.36.3_27015 69.**.36.7_27015 69.**.36.6_27015 69.**.36.4_27015
srcds_uptime_ | yes | 69.**.36.4_27015 69.**.36.3_27015 69.**.36.7_27015 69.**.36.6_27015



Running munin-run srcds_fps_69.**.36.4_27015 give me fps.value 192

Any reason why this would create a horrible number of process that tie up my cpu? Any help would be greatly appreciate.

Upon looking through the file munin-node.log I find tons of lines that read

Can't exec "/etc/munin/plugins/srcds_players_69.**.36.4_27015": Permission denied at /usr/sbin/munin-node line 498.
Can't exec "/etc/munin/plugins/srcds_fps_69.**.36.4_27015": Permission denied at /usr/sbin/munin-node line 498.

[css]

Make sure you've added execution rights for the plugin files correctly, ie:

Code:

chmod a+x /usr/share/munin/plugins/srcds_*

I believe the problem is caused by Munin not being able to run the plugin or parts of it. You can further test whether the munin-node works as expected by telnet. Try this (where localhost is the munin-node host):

Code:

telnet localhost 4949
help
list
fetch srcds_players_69.**.36.4_27015

It should print fps.value xxx. If it doesn't, then there's something wrong.

[Firefox005]

Here is the permissions:
-rwxr-xr-x 1 root root 2711 2008-11-01 21:04 srcds_cpu_
-rwxr-xr-x 1 root root 2852 2008-11-01 21:04 srcds_fps_
-rwxr-xr-x 1 root root 3067 2008-11-01 21:04 srcds_inout_
-rwxr-xr-x 1 root root 2340 2008-11-01 21:04 srcds_players_
-rwxr-xr-x 1 root root 2228 2008-11-01 21:04 srcds_uptime_

Here is what happens when I try to telnet:
[root@gs1 plugins]# telnet localhost 4949
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
# munin node at gs1.********.net
list
open_inodes sendmail_mailtraffic entropy irqstats sendmail_mailstats if_err_eth2 processes if_eth3 if_err_eth3 df netstat interrupts swap if_eth2 load sendmail_mailqueue cpu df_inode srcds_fps_69.**.36.4_27015 iostat open_files forks memory vmstat
fetch srcds_fps_69.**.36.4_27015
# timeout pid 14567 - killing...done
.


I only have one plugin loaded because as I said when all of them are loaded it acts like a fork bomb.

When I run munin-run srcds_fps_69.9.36.4_27015 from the CLI it takes a longish time (7-8 secs) before the value is printed:
[root@gs1 plugins]# munin-run srcds_fps_69.9.36.4_27015
fps.value 195

I can send you an strace of when I run munin-run srcds_fps_69.9.36.4_27015. I can tell you that it is littered with this block of lines repeated several time:
stat("/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/site_perl/5.8.7/x86_64-linux-thread-multi/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/site_perl/5.8.7/x86_64-linux-thread-multi/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/site_perl/5.8.6/x86_64-linux-thread-multi/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/site_perl/5.8.6/x86_64-linux-thread-multi/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/site_perl/5.8.8/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/site_perl/5.8.8/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/site_perl/5.8.7/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/site_perl/5.8.7/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/site_perl/5.8.6/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/site_perl/5.8.6/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/site_perl/5.8.5/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/site_perl/5.8.5/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/site_perl/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/site_perl/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/vendor_perl/5.8.7/x86_64-linux-thread-multi/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/vendor_perl/5.8.7/x86_64-linux-thread-multi/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/vendor_perl/5.8.6/x86_64-linux-thread-multi/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/vendor_perl/5.8.6/x86_64-linux-thread-multi/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/vendor_perl/5.8.5/x86_64-linux-thread-multi/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/vendor_perl/5.8.5/x86_64-linux-thread-multi/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/vendor_perl/5.8.8/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/vendor_perl/5.8.8/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/vendor_perl/5.8.7/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/vendor_perl/5.8.7/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/vendor_perl/5.8.6/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/vendor_perl/5.8.6/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/vendor_perl/5.8.5/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/vendor_perl/5.8.5/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/vendor_perl/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/vendor_perl/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/strict.pm", 0x7fff855cef00) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/5.8.8/strict.pmc", 0x7fff855cf040) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/5.8.8/strict.pm", {st_mode=S_IFREG|0644, st_size=3292, ...}) = 0
open("/usr/lib/perl5/5.8.8/strict.pm", O_RDONLY) = 4

however what it can't find is different each time:
/usr/lib64/perl5/site_perl/5.8.7/x86_64-linux-thread-multi/warnings/register.pm
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/warnings.pmc

perl is installed and regular munin works fine.

On a hunch I tried editing munin-node and adding the line user root directly below the [scrds_cpu_69.**.36.4_27015] however after that when I tried to telnet it would say connection refused and /etc/init.d/munin-node status would give me "munin-node dead but subsys locked"

[Firefox005]

Now the rcon tool itself is messing up it just gives me Illegal size -559038737 .


Also never mind about all those cannot stat errors that strace was returning, that is perl trying to find the files, at the end of each block it find the file.

[Firefox005]

After searching around I found out what the problem was: SELinux apparently munin and your plugin was trying to do something that SELinux thought shouldn't happen. I have put SELinux into permissive mode and hopefully can look at the logs and create a profile for munin to allow whatever is that is causing it to break.

Also the rcon tool has problems running in 64bit mode.

Code:

echo 0 >/selinux/enforce

EDIT:

Now that I got it working the only plugins that work are the CPU and FPS plugins. All the other's print NaN , and telenet fetch simply returns a . and running munin-run srcds_players_69.**.36.3_27015 just goes straight to a new prompt without printing anything.

[css]

Telnet is the most reliable way of seeing if the plugin is properly installed and working. "List" command should always print the plugin in the list. Also "fetch srcds_xxxx" should print xxxx.value if everything's OK. This is practically what Munin itself does too.

Sometimes if everything seems to work good, but you don't get graphs, it's good to delete /var/lib/munin/your.domain/*srcds* files. Those are the database files which Munin uses. Quite often they end up being broken if the plugins don't start working easily.

You can try to set "timeout 30" to munin-node.conf to allow Munin to process longer the srcds* plugins. You can also set variable "srcdssamples" to 1 or 2 (default is 5) to take fewer samples. This is practically how many times the "rcon" command is run. Higher values give more precision because the server can report very fluctuating FPS between separate "stats" commands.

Code:

[srcds_*]
env.srcdspass iajisjaja
env.srcdshost 1.2.3.4
env.srcdssamples 1
timeout 30

None of my tips here probably help you with srcds_players or srcds_uptime plugins. It could be that SELinux has very strict rules and it doesn't allow user "munin" to run root owned files or something.

[Firefox005]

Well bugger its seems that the issue lies with the CLI rcon tool rcon. It's that weird "Illegal size -559038737" error that's messing everything up. It's so weird when I run the rcon tool from the CLI it works but when I strace it while running inside munin it fails every time.

[css]

Bugger indeed. It means I have to do some work on the plugins. After creating the Munin plugin I also created Perl version of RCON system. I haven't implemented the Perl version into the Munin plugins, but I knew running external program never was the perfect solution.

I'll post here when I've completed patching the plugins. I expect to get it done until the end of this week. It should be straightforward.

[Firefox005]

css Wrote:Bugger indeed. It means I have to do some work on the plugins. After creating the Munin plugin I also created Perl version of RCON system. I haven't implemented the Perl version into the Munin plugins, but I knew running external program never was the perfect solution.

I'll post here when I've completed patching the plugins. I expect to get it done until the end of this week. It should be straightforward.

Well I also found something called SRCDS.py but that seems to be broken as well, it will only give me the error "Detail Query Error: Unknown response type".


http://sourceforge.net/projects/srcdspy/

Also thank you for all your help.

[css]

Try this:

http://css.setti.info/code/munin-srcds/munin_srcds_plugin/srcds_fps2_
http://css.setti.info/code/munin-srcds/munin_srcds_plugin/Rcon.pm

Copy both files to Munin's plugins directory. Then link srcds_fps2_x.x.x.x_27015 as usual to /etc/munin/plugins/. Restart munin-node and try with the telnet method whether it works.

I added more failure messages in the script, so if something goes wrong it should say more accurately what's wrong.

I'll patch rest of the files later if this works.

[Firefox005]

I get the error "Could not authenticate" and the "14:11:35 L 11/05/2008 - 15:11:28: Addip: "<><><>" was banned by IP "for 10.00 minutes" by "Console" (IP "69.**.36.4")" errors still show up.

[css]

Unban your local IP from the server. Then try again. Do not try too many times if it says "authentication error". Verify that you've got "srcdspass" set correctly and then try again. Note that your current [srcds_...] settings may not cover the srcds_fps2_ plugin.

[Firefox005]

I am indeed an idiot I thought I had changed the setting in plugin.d/munin-node but when I double check it, there was only a duplicate entry for the regular fps. It's working fine now.