Free Public Fast Download Server

[gncmatt]

Hi everyone...

I had an idea for a project and I started working on it yesterday.. before I invest too much time into it I just wanted to see if something like this already exists and/or if people would like to use it.

The basic idea is I have an http server that can be used for http download in steam games (sv_downloadurl). I'm working on some programs that would allow someone to upload a map themselves, automatically verified it's a proper map to avoid abuse, compress the map with bz2 and add it to the map folder. Benefits would be that people don't need to all buy their own web site just for fast downloads, and maps would be shared with all servers using it so no one needs to upload the same map twice. And if intrest is high enough some of my other ideas include having the server upload a file directly to your gameserver to avoid needing to download/reupload yourself.

So.. what do you guys think?

[Cookies N' Milk]

It is a great idea but you will be hard pressed to make it a free service. You would need an insane amount of bandwidth for it. I currently have a dedicated webserver with 3 Source servers connecting to it for fastdownload and it takes about 5GBs of transfer a day, (173GB total last month). And that is with only 3 servers using it. Just think about 100 servers, or once it is widely used, 1000+ servers. If each server used only 1GB of transfer a day with 100servers using the service you would need 3TB of bandwidth a month.

I do like the idea though. Not sure if anyone has tried it before. If only a few people use it you should be fine, but if a ton of people use it... it will cost a ton to keep it up. It is worth exploring the idea however

[Drocona]

Maybe if you can get this off the ground (I like the idea) you want to get some "partners" to help you set up a load balancing script, like setting sv_downloadurl to a php file that redirects to a mirror from a list, where other people can set up mirrors too to help with the project. This way you can equally "share" the expenses and bandwidth.

But like Cookies N' Milk said, you need a LOT of bandwidth, maybe you should try and get a 10mbps unmetered dedicated server somewhere in the USA. You will be paying around 130$/month for a P4 3,0GHz with 1GB ram 250gigs HDD and 100mbps unmetered (found it at some website, just google a bit). As you don't need a uber stats dedicated server, most will do (just don't get celerons!)

[Cookies N' Milk]

Drocona Wrote:Maybe if you can get this off the ground (I like the idea) you want to get some "partners" to help you set up a load balancing script, like setting sv_downloadurl to a php file that redirects to a mirror from a list, where other people can set up mirrors too to help with the project. This way you can equally "share" the expenses and bandwidth.

But like Cookies N' Milk said, you need a LOT of bandwidth, maybe you should try and get a 10mbps unmetered dedicated server somewhere in the USA. You will be paying around 130$/month for a P4 3,0GHz with 1GB ram 250gigs HDD and 100mbps unmetered (found it at some website, just google a bit). As you don't need a uber stats dedicated server, most will do (just don't get celerons!)

I like that idea of adding minors to it. Say 10 people set aside some space and bandwidth on their dedicated servers for the project then it would be far more likely that it would work out.

Also I know of a few hosts that do DualCore Procs with 1GB ram and 1000-3000GB bandwidth on a 10Mbit connection for under $150 which isn't too bad.

[Mooga]

Plus remember: you would also need a way to upload sounds and stuff since they need to be in the fast-dl link as well.

[Hazz]

This is a great idea, like the others have said, if you can make it easy to upload files and make sure you have plenty of bandwidth etc then it should be great.

[Cookies N' Milk]

@Mooga: Way to add to the amount of bandwidth needed I completely forgot about sounds and all those extras.

I would be interested to see if the OP can get it to work. It would almost certainly be successful if he can.

[gncmatt]

Wow thanks for all the feedback...

On the server I am currently using i have over 1TB bandwidth and over 1000GB space, I think it should do for a while.

I could so rig sv_downloadurl to do that.. never thought of that... if it becomes nessessary I will look into that.

As far as sounds and stuff I was going to just start by loading on the mani and/or the other popular ones, and I don't think the bandwidth will be too much for those since it's not THAT big plus everyone only needs to download it once.

I think we should try to change our thinking away from the number of servers that will use it and try to think of it on a user level... People need to download maps one time each, if we have 10 servers and 50 people it becomes irrelevant how many servers we have using it because they only need to download the file one time on one server.

---

So I was playing around with this yesterday and I'm trying to figure out the best options to upload maps.. HTTP uploading by itself to my knowledge caps at 2MB unless you change settings in your browser.. So I was thinking of making a java or flash (or finding a prebuilt uploader script).. I was also looking into using anonymous ftp.. my latest idea was to set it up so you can upload files to the ftp server but with the permissions that you can write and not read, so after you upload a file you can't see whats there (to avoid people from deleting things and stuff) but I already found a few problems with that in trying to break it... If I do use the ftp (it uploads to a non-http-accessable area, then I would setup a cronjob of a php script like every 10 min or so that would search in the folder for maps, do the map verification stuff, if it passes to compress it and move it to the correct area, if it fails delete it.

I've setup a pretty weak looking website for it to get things rolling.. http://source.mattsfiles.net (yea I'm not very creative) Also if anyone has any sugguestions on what to call this thing, currently I called it "Matt's Files: Source" But I think it's dumb and too long.. Theres also a forum I put on, feel free to go post a hello or something so I don't look like a complete dumbass talking to myself http://forum.mattsfiles.net

Also someone sugguested to me about putting something like most downloaded maps and stuff.. that could be fun..


Off to class.. back in a few hours to keep working it...

[Cookies N' Milk]

gncmatt Wrote:Wow thanks for all the feedback...

On the server I am currently using i have over 1TB bandwidth and over 1000GB space, I think it should do for a while.

I could so rig sv_downloadurl to do that.. never thought of that... if it becomes nessessary I will look into that.

As far as sounds and stuff I was going to just start by loading on the mani and/or the other popular ones, and I don't think the bandwidth will be too much for those since it's not THAT big plus everyone only needs to download it once.

I think we should try to change our thinking away from the number of servers that will use it and try to think of it on a user level... People need to download maps one time each, if we have 10 servers and 50 people it becomes irrelevant how many servers we have using it because they only need to download the file one time on one server.

---

So I was playing around with this yesterday and I'm trying to figure out the best options to upload maps.. HTTP uploading by itself to my knowledge caps at 2MB unless you change settings in your browser.. So I was thinking of making a java or flash (or finding a prebuilt uploader script).. I was also looking into using anonymous ftp.. my latest idea was to set it up so you can upload files to the ftp server but with the permissions that you can write and not read, so after you upload a file you can't see whats there (to avoid people from deleting things and stuff) but I already found a few problems with that in trying to break it... If I do use the ftp (it uploads to a non-http-accessable area, then I would setup a cronjob of a php script like every 10 min or so that would search in the folder for maps, do the map verification stuff, if it passes to compress it and move it to the correct area, if it fails delete it.

I've setup a pretty weak looking website for it to get things rolling.. http://source.mattsfiles.net (yea I'm not very creative) Also if anyone has any sugguestions on what to call this thing, currently I called it "Matt's Files: Source" But I think it's dumb and too long.. Theres also a forum I put on, feel free to go post a hello or something so I don't look like a complete dumbass talking to myself http://forum.mattsfiles.net

Also someone sugguested to me about putting something like most downloaded maps and stuff.. that could be fun..


Off to class.. back in a few hours to keep working it...

Try looking for swfuploader (I think that is what it is called.) I'm pretty sure it can take 100MB+ uploads. I use it for my YouTube clone script on my IPB forums.

As for multiple servers, the more that are willing to be part of the network the better. It means the servers can share the load rather than have a single server doing all the dirty work. Plus if one goes down it can switch to the other servers.

I might be setting up several servers soon and would be willing to set some space and bandwidth aside for this.

[Spartanfrog]

Sounds like a great idea but also sounds like trouble. I can just imagine people abusing this. One way around that would to have people prove they have a server.

[Cookies N' Milk]

Spartanfrog Wrote:Sounds like a great idea but also sounds like trouble. I can just imagine people abusing this. One way around that would to have people prove they have a server.

It would be simple to keep most abusers away. Like he said, he would have a script to verify all new files, scan them for virus and make sure they are valid map files. It would also be easy enough to setup a registration page where the user must add his server IP so that it could be added to a list of allowed servers.

I got a few ideas. First one would be that registration page. Second one is that people can't just leech off the servers. They also have to upload maps etc and contribute or their account is blocked/locked. That would get rid of 99% of the leechers.

[gncmatt]

I'll look into swfloader, thx.

I've decided how I'm going to route the sv_downloadurl to a php file so I will probably have that done today.. With it going through php I will be able to have more control such as load balancing and disabling hotlinking and some other stuff...

Yea there are many things I have considered about all the possible abuse that could be done and I've countered most of the ones i thought of (conceptually). The main thing I need to get done is the bsp verifier.. It starts out by (obviously) checking to see if it's bsp (or if it's a bz2 it will decompress and check), then it opens it up and verifies it's a map, for security reasons I won't elaborate on what spesificly it checks for but I just wanted to point out it's not just like ".bsp, PASSED!".

I did already consider the verifying server thing with registration.. I think I posted some notes about it on the site somewhere.. for phase 1 i'm going to have it open to everyone to just upload (at least in the first beta).. after that I was thinking about requiring people to give a server address, and it would check to make sure it exists and require some type of verification of ownership (either ftp info or setting a cvar to a spesified random string if they dont' wanna put in their password).

Oooo I didn't think of a virus scanner.. good idea.. what are the possibilities of viruses in bsp files anyway?

I don't know if the allowed servers idea would work because I don't think the server makes any call to the http server, to my knowledge it just directs the user to the link of the files and direct connect from there but I will play with all the options of that when I get there.

The only leeching that bothers me is people hotlinking from other sites to download it via browser.. I don't really mind if people don't upoad maps because that is one of the benifits of the system.. if you use 20 popular custom maps on your server the chances are they would already be on the server.. I dont' want people uploading just for the sake of uploading...

Something else I went over with someone was to maybe have 2 levels of uploading in final version.. one upload page for non-reg people that can only upload files under a certian size and the map will be deleted if it is not downloaded x amount of times within the first 30 days of upload. And registered accounts that are linked with a game server don't have those restrictions.

Oh and I forgot the email on my forum wasn't working.. I'll manually enable whoever signed up and I'll remove email verification for now...

[Cookies N' Milk]

gncmatt Wrote:I'll look into swfloader, thx.

I've decided how I'm going to route the sv_downloadurl to a php file so I will probably have that done today.. With it going through php I will be able to have more control such as load balancing and disabling hotlinking and some other stuff...

Yea there are many things I have considered about all the possible abuse that could be done and I've countered most of the ones i thought of (conceptually). The main thing I need to get done is the bsp verifier.. It starts out by (obviously) checking to see if it's bsp (or if it's a bz2 it will decompress and check), then it opens it up and verifies it's a map, for security reasons I won't elaborate on what spesificly it checks for but I just wanted to point out it's not just like ".bsp, PASSED!".

I did already consider the verifying server thing with registration.. I think I posted some notes about it on the site somewhere.. for phase 1 i'm going to have it open to everyone to just upload (at least in the first beta).. after that I was thinking about requiring people to give a server address, and it would check to make sure it exists and require some type of verification of ownership (either ftp info or setting a cvar to a spesified random string if they dont' wanna put in their password).

Oooo I didn't think of a virus scanner.. good idea.. what are the possibilities of viruses in bsp files anyway?

I don't know if the allowed servers idea would work because I don't think the server makes any call to the http server, to my knowledge it just directs the user to the link of the files and direct connect from there but I will play with all the options of that when I get there.

The only leeching that bothers me is people hotlinking from other sites to download it via browser.. I don't really mind if people don't upoad maps because that is one of the benifits of the system.. if you use 20 popular custom maps on your server the chances are they would already be on the server.. I dont' want people uploading just for the sake of uploading...

Something else I went over with someone was to maybe have 2 levels of uploading in final version.. one upload page for non-reg people that can only upload files under a certian size and the map will be deleted if it is not downloaded x amount of times within the first 30 days of upload. And registered accounts that are linked with a game server don't have those restrictions.

Oh and I forgot the email on my forum wasn't working.. I'll manually enable whoever signed up and I'll remove email verification for now...

It is a really good idea and I'm curious to see how it turns out. As for Viruses... You never know now a days.

[ClanWoof]

How would it work for everyone, as for admin skins and such as well. Since fast downloads is used for custom admin skins and so on.
I think this is a great idea to have the fast downloads.
I'm also looking for something I can use on my website so people can upload pictures and demos and it give them a link to it once it's done.

[gncmatt]

ClanWoof Wrote:How would it work for everyone, as for admin skins and such as well. Since fast downloads is used for custom admin skins and so on.
I think this is a great idea to have the fast downloads.
I'm also looking for something I can use on my website so people can upload pictures and demos and it give them a link to it once it's done.

Other files besides maps and the default mani (or whatever other MAJOR addons need) are very low on my priority list with everything I need to do right now... Eventually I plan on adding support for uploading materials and sounds to allow for custom things...

xyz (the first staff member to join me in this project) was saying that I should setup something for people to upload and share demo files. Doesn't sound too interesting to me as I don't do demos, but if theres enough want for it I guess I wouldn't mind doing that when I'm done with this.. (unless you successfully do your picture/demo thing I wouldn't mind throwing a link to your site on mine).

On that note, I didn't notice any specific tutorials on sv_downloadurl here, if someone writes one up here (and possibly specifically mentioning my thing) I wouldn't mind linking to it here opposed to writing my own for me.. I like the idea of having all the tutorials on one site