Source RCON protocol

[gertjuhh]

First off, my apologies if I'm posting in the wrong section of the forum.
I didn't know where this would be best suited.

Anyway...
I'm working on a PHP script which allows you to monitor and administrate a srcds from either IRC (in combination with a php-irc bot) or within the server itself (by parsing the logs).
I got the traffic both ways working fine and dandy.
I get the logs, can send rcon commands and receive their response, but this is where the problem kicks in.
Some rcon replies (cvarlist for instance) span over multiple packets, now this alone ain't an issue.
However i have not found any way to tell when the last packet of the reply has been received...
I have google-ed my ass off, read this page a million times, but still no luck.
Does anybody have a solution for me?
Right now I'm stuck with using timers and assuming a reply is completed within 0.5 seconds, which does do the trick but frankly just ain't the nicest solution.

To sum things up, is there anything which indicates a rcon reply spanned over multiple packets is complete?

Thanks in advance for any help!

[css]

gertjuhh Wrote:However i have not found any way to tell when the last packet of the reply has been received...

Maybe we can help each others. Here's my solution with Perl:

Code:

#
# Receive data from a socket
#
# The function reads all data from a socket, and parses the actual message
# from the packet data contents.
#
sub sock_receive ($) {
    my $sock = shift;
    # I/O handle to lookup if there's data
    my $can_read = new IO::Select($sock);
    my $reply = "";
    my $buffer;
    my ($size, $request_id, $command_response, $data);
    while ($can_read->can_read(0.5)) {
        $sock->recv($buffer, 4, MSG_PEEK); # Peek size field (32b int)
        $size = unpack('V', $buffer); # Convert bytes to int
        last if (!defined($size));
        $sock->recv($buffer, $size+4, MSG_WAITALL); # Read the whole packet
        ($size, $request_id, $command_response, $data) =
            unpack('VVVZ*x', $buffer); # Convert values from the packet
        $reply .= "$data";
    }

    return ($reply, $request_id, $command_response);
}

This solution works, but I don't like the way I've implemented it. The line which you will probably find useful is:

Code:

$sock->recv($buffer, $size+4, MSG_WAITALL); # Read the whole packet

Pay special attention to the MSG_WAITALL flag. It reads the socket until it has got the amount of data that was specified in the first packet (+4 because the size field itself isn't calculated in the size). You can read more detailed description of flags for recv at php.net comments.

This method is not completely good, because if the first packet contains wrong size field (for some reason) or some of the packets are never send, the script will wait until forever to get the data. There should be timeout around the recv() function (hmm... why haven't I used it, yea, because all the examples for Perl's recv timeout look so ugly.. good suggestions are appreciated).

You can take a look at my whole RCON implementation by downloading Player rate tracking system. There are four functions needed to authenticate, send and receive commands and data from the server. Here's quick copy&paste briefing from the script itself.

Code:

##########################
# Base RCON functions
##########################
# It's possible to use these functions outside of this program. These functions
# work without the rest of the code, so these could be made into a module and
# used wherever needed.
#
# Functions:
# sock_connect
#  + Opens TCP connection to a server.
# sock_receive
#  + Receives RCON reply from a server.
# rcon_auth
#  + Sends RCON password to a server and authorizes the connection.
# rcon_command
#  + Executes RCON command on a server.
# print_data
#  + Prints any string as HEX format. This is for debugging.

PHP and Perl are somewhat similar scripting languages, so it might be even possible to convert the script to PHP with little work.

You can ask or post some feedback about my or your implementation. We could get some good feedback from others too.

[gertjuhh]

Problem is I can't use a timeout as I'm handling an irc connection aswell.
When i started working on the script i made a simple php class which works perfectly for a single request (or more)
eg. a website where you can issue an rcon command from which the output will be printed on screen.
It's not so much the socket stuff that's causing the problem, but the inability to tell when i have received all data belonging to a single rcon request reply (being build up over 1 or more packages).
(Sorry if i can't explain this any better, hard enough to think of a way to explain this, let alone in a foreign language)

p.s. If i understand you script correctly you have a small mistake in there.
You're first reading the first 4 bytes to get the package size, after which you read the length + ANOTHER extra 4 bytes.
Which would come to a total of 4 (length field) + packagesize + 4.
I have never had the problem of where the first 4 bytes would not contain the actual package length

[gertjuhh]

Did some digging around and found my old rcon class.
Made some improvements tonight so you (and others) can have a look.
This class works fine with multi-packet responses.
http://www.madclog.nl/downloads/classes/clsRcon.phps

p.s. this is far from the script I'm using in the php-irc version

[css]

gertjuhh Wrote:Problem is I can't use a timeout as I'm handling an irc connection aswell.

Why not?

gertjuhh Wrote:It's not so much the socket stuff that's causing the problem, but the inability to tell when i have received all data belonging to a single rcon request reply (being build up over 1 or more packages).

Hmm.. it seems you can't. None of the packets include the total length of the reply, so you'll just have to do some sort of timeout in the end. Now it starts to come back into my mind too. I remember using the same kind of "wait 0.5 seconds and assume all packets have arrived" style before doing the implementation I've got now. I guess that's the only way.

gertjuhh Wrote:You're first reading the first 4 bytes to get the package size, after which you read the length + ANOTHER extra 4 bytes. Which would come to a total of 4 (length field) + packagesize + 4. I have never had the problem of where the first 4 bytes would not contain the actual package length

See the flags, I'm just peeking

Code:

$sock->recv($buffer, 4, MSG_PEEK); # Peek size field (32b int)

The MSG_PEEK flag doesn't remove the data from the queue, so the next time I do call to recv, the first four bytes are still the size field. I don't remember exactly why I came up with this solution, but it's probably got something to do with not getting into infinite blocking read if there's no reply from the server - although that's exactly what will happen

This kind of programs should always prepare for unexpected values. If there is bug in the game engines (they tell the size field wrong), then there is bug in your program too. I don't like at all that my script can get into infinite blocking wait because it assumes the size field is correct. I could avoid the problem by adding timeout like you've got - except I have to do it with the dirty looking Perl hack which I don't like at all:

Code:

$rin = '';
vec($rin, fileno(SOCKET), 1) = 1;

# timeout after 10.0 seconds
while (select($rout = $rin, undef, undef, 10.0)) {
    ...
}

The Perl version with all those rins, routs and vec looks stupid compared to the PHP version one-liner:

Code:

while (socket_select($aRead, $aWrite = NULL, $aExcept = NULL, 0, $this->m_iReadTimeout)) {

I hope there's some cool Perl module to help with timeouts that I haven't found yet...

[raydan]

i have a java rcon source code, that can connect multi servers and send many rcon commands to all server at the same time by using thread.

[css]

Good for you.

[gertjuhh]

css Wrote:

gertjuhh Wrote:Problem is I can't use a timeout as I'm handling an irc connection aswell.

Why not?

The php-irc bot i'm building the module for already has some very nice socket functionalities.
It tells when there's any data to read and then allows me to work with it.

css Wrote:

gertjuhh Wrote:It's not so much the socket stuff that's causing the problem, but the inability to tell when i have received all data belonging to a single rcon request reply (being build up over 1 or more packages).

Hmm.. it seems you can't. None of the packets include the total length of the reply, so you'll just have to do some sort of timeout in the end. Now it starts to come back into my mind too. I remember using the same kind of "wait 0.5 seconds and assume all packets have arrived" style before doing the implementation I've got now. I guess that's the only way.

Shame really, but i gues it'll have to do for now...

css Wrote:

gertjuhh Wrote:You're first reading the first 4 bytes to get the package size, after which you read the length + ANOTHER extra 4 bytes. Which would come to a total of 4 (length field) + packagesize + 4. I have never had the problem of where the first 4 bytes would not contain the actual package length

See the flags, I'm just peeking

Code:

$sock->recv($buffer, 4, MSG_PEEK); # Peek size field (32b int)

The MSG_PEEK flag doesn't remove the data from the queue, so the next time I do call to recv, the first four bytes are still the size field. I don't remember exactly why I came up with this solution, but it's probably got something to do with not getting into infinite blocking read if there's no reply from the server - although that's exactly what will happen

Whoops, sorry about that.
I didn't read up on those flags cause I use other functions to work with sockets.
Still I've never had a packet not include the packet size.
Sometimes a packet is send over 2 'tcp replies' (make any sence??), are you taking that into account?
Before i made a fix for this I did have some 'broken' packets

raydan Wrote:i have a java rcon source code, that can connect multi servers and send many rcon commands to all server at the same time by using thread.

How is this relevant to the discussion!?

[css]

gertjuhh Wrote:Still I've never had a packet not include the packet size.
Sometimes a packet is send over 2 'tcp replies' (make any sence??), are you taking that into account?
Before i made a fix for this I did have some 'broken' packets

This is what I noticed also. I was quite surprised to see that it worked like that. I thought that I'd get at least one full packet every time I do recv() on the socket. That's why I first had just loop where I did one recv(), then parsed the contents of the packet assuming that the first four bytes are the size field and the data packet ends to null byte etc. This wasn't the case and I ended up with "broken" packets. This is probably what you meant. I believe I've solved this problem by peeking the expected size of the incoming packet and then doing recv() with MSG_WAIT_ALL, which ensures the full packet is retrieved before continuing - assuming the size field is correct.

[css]

This is how Stormtrooper does it in Psychostats:

Code:

// read a command response from the open RCON stream.
function _rconread2() {
    if (!$this->rconsock) return FALSE;
    $packet = array();
    $this->raw = "";
    $psize = 0;
    $size = 0;
    $total = 0;
    $string1 = "";
    $string2 = "";

    $first = 1;
    $expected = 0;
    do {
        if (!($psize = @fread($this->rconsock, 4))) {    // get the size of the packet (packed)
            break;
        }
        $size = $this->_unpack('V', $psize);        // convert packed size into an integer

        $this->raw = @fread($this->rconsock, $size);

        if ($this->DEBUG) print "DEBUG: Received (size: $size):\n" . $this->hexdump($psize . $this->raw) . "\n";
        $packet = array(
            'requestid'    => $this->_getlong(),
            'responseid'    => $this->_getlong(),
            'string1'    => $this->_getnullstr(),
            'string2'    => $this->_getnullstr(),
        );

        // combine multi-part-packets into single strings
        $string1 .= $packet['string1'];
        $string2 .= $packet['string2'];

        $expected = ($size >= 3096);            // if the size was >= ~3096 we should expect another packet
        $first = 0;                    // first packet has gone through
    } while ($expected);

    if ($packet) {
        $packet['string1'] = $string1;
        $packet['string2'] = $string2;
    }
    return $packet;
}

The key point is this line:

Code:

        $expected = ($size >= 3096);            // if the size was >= ~3096 we should expect another packet

[gertjuhh]

Hmm... That might actually work.
Gonna do some test runs tonight when I get home from work.

To be continued...

[gertjuhh]

Unfortunately this method can't be relied on either
Here's a list with packet lengths from a 'cvarlist sv' response

Code:

int(10)
int(10)
int(4069)
int(4060)
int(4069)
int(3959)

The first 2 belong to the authentication and the rest is from the actual response.
As you can see the last packet has a size >= 3096

[css]

Obviously. There is no reliable way to determine whether you've received the last packet. Stormtrooper makes pretty good assumption that if the last received packet is less than ~3kB in size, then it's quite likely that it's the last packet because there's no more data to send. If it wasn't the last (ie. there's more data to send), then the packet would be filled up to ~4kB.

Let me know when you hit beta phase with your irc bot. I might be interested in testing

[gertjuhh]

css Wrote:Obviously. There is no reliable way to determine whether you've received the last packet. Stormtrooper makes pretty good assumption that if the last received packet is less than ~3kB in size, then it's quite likely that it's the last packet because there's no more data to send. If it wasn't the last (ie. there's more data to send), then the packet would be filled up to ~4kB.

Unfortunately the last packet is received in my example was to close to 4KB for me to rely on this method.
I'm just gonna stick with the 0.5sec wait until I can come up with a better solution.

css Wrote:Let me know when you hit beta phase with your irc bot. I might be interested in testing

If you're interested in testing the module I'm making you can go ahead and setup php-irc in advance.
I'll be sure to drop a message here once the module is ready for (public) download.
Shouldn't be to long as I already had a working module.
Just revising it for more flexibility and better coding style / documentation.

[gertjuhh]

As promised, the module has been released.
Click here for more information.