Load balance certain files

[pilot830]

in the admin section, have a section or setting where we can define if a file is getting more than 50 downloads, to then queue up that file to be copied to all file servers

so that when people go to access it, they're not all just raping the one file server and then everyones experience becomes poor. so once its on all the file servers, when people visit that particular url, the serverID in the database instead of being a single value, it could be multiple values. so if you got 3 file servers, the serverId value is now 3 values, like so say serverId 8, 9 and 10.. so itd be in the database "8,9,10" and then the php script/code will split the values into 3 and randomly choose one server Id and thats the server Id they land on when they access the shorturl

but then also if that file is ever removed in any way shape or form, that of course, the database remembers, to remove it form all file servers, so its not just left over

 

[enricodias4654]

Take a look here: http://forum.mfscripts.com/viewtopic.php?f=13&t=2258

 

[pilot830]

I don't think its a good idea to have a copy of every file on every file server. I believe my idea is more efficient and makes sense. Only when a file is being downloaded greater than x number of times in the download tracker, would it then sync that file to the other file servers, so that when people go ot download that file or stream it, it wont be laggy or slow

 

[pilot830]

I would write a shell script to talk to the database, find out what files are being downloaded by 30 or more people , from download_tracker, then take that file and copy it to another file server, and then update the serverID of the file to the new file server.. and just run a loop that keeps updating the severId back and forth from a few file servers

but the problem is, there is no way to find out what file server they are pulling the file off.. in download tracker there is no field that says the server

So basically, based on my original post, it would be so great to just have this feature, where the serverId field for the file in the file table, could be like more than 1 value

or the value could be

1|2|3

so it'd start out as just one value, like say server Id 5

but then if the file gets mirrored or copied to another file server, then the serverId value is updated to 2 values

so if it was on server id 3 at first

but now its copied to servers 4 and 5

then the value would be

3|4|5

or however you wanna do it

just the idea that a file can be mirrored onto another file server, if the load is heavy, would be so great

the main reason to implement a system where certain files that are being downloaded by many people, be mirrored to other file servers, is because otherwise, every yetishare based site that gets a lot of downloads, the people will complain "this site buffers too much" and so forth. that is what im getting already for my site.. for videos that are being streamed often. It doesn't make sense to just mirror the contents of an entire file server. It makes more sense to mirror contents for just a specific file

 

[enricodias4654]

Every file on every server? Where did you read that? Read my post again.

My suggestion is to replicate files between servers. It doesn't mean to replicate all files in all servers. The files would be replicated N times in N servers. N would be a number of your choice. If N = 2, each file would be replicated in 2 different servers.

Using the current serverId field is not the way to go. The right way would be creating a different table with a fileId and a serverId field. This way you will have a normalized table n:n.

Ssh is not the best solution for this. Some users may not have shell access and the script already has a way to transfer files between servers using only php. By using php you can also tune the transfer speed to don't use 100% of the server capacity and don't disturb current downloads and uplaods.

 

[pilot830]

Sorry must have misread what you said earlier. In the mean time, I've been faced with situations where a user uploads a file, and it ends up on a file server, say server1.mydomain.com

Now that file is being downloaded by 200 users, and the server possessing the file, server1, is lagging or has reached its max bandwidth already

Since the script has no way to handle this (as of yet), I've had to make a bash shell script that basically

1. detects the current bandwidth speed of the file server, so server1. If the bandwidth speed has reached a certain level, such as 950mbit, then it copies the file to another file server, i.e server2

2. then the database is updated for that file, and the serverId is changed to server2 where the file is now pointed to. so all new traffic goes to server2

then when server1 bandwidth gets below its threshold, say gets back down to 700 or < 950.. then serverId is reupdated to point back to server1 (for the high traffic file)

its a temp solution for now but it works. Any file that gets x number of downloads, it copies that file to another server and prepares to point all new traffic to the new server if need be depending on the system load average and or the bandwidth speed

So far its been working out for me. But it would be amazing if the ability to load balance or mirror files within the script itself was possible. i.e defining in the admin settings, that if a file gets more than 50 downloads it then gets mirrored to other file servers

And then when the file isnt even being downloaded anymore, it removes it from the mirror'd servers and keeps it on one server only