(Server Config) Slow Upload with Direct Server Uploads

[ollie]

Hi All, i think Adam has probably had enough with my tickets so i am going to post here from now on, hopefully someone can help me.

I'm using a dedicated file server with direct uploading but i'm having problems achieving good upload speeds through the website. I am able to achieve over 30MB/s uploading to the file server over FTP. However through the website i can only get 3MB/s.

I've attached my php and nginx config, if anyone has any ideas as to why this is happening please let me know.

I've tried increasing the chunk size but it made no difference

Server specs:

2x Quad Xeon E5620 (16 Cores)
64 GB RAM
120GB SSD 8x 4 TB SATA RAID 10
1Gbps

Thanks in advance,
Ollie

 

[ollie]

Narrowed this down to being php. Still not sure how to fix though

 

[adam]

Hi Ollie,

Is your site on https? If yes ftp is non secure so won't have the overhead of encryption. Might be worth testing upload speed via scp.

Also are you using apache or nginx? The web server will handle the upload itself, then when done its passed to php, so I'd say the web server is the place to look. Does it pause after each chunk or not really?

Thanks,
Adam.

 

[adam]

Sorry I see you're using Nginx. I'll see if there's any info about nginx upload speed issues...

 

[adam]

That nginx config also tells me you're using ssl (sorry its late.. ). Maybe test the scp upload speed as mentioned above.

 

[ollie]

Hi Adam, thank you for the reply, just tested uploading to the file server and got 70MB/s using scp

Downloading/Uploading videos for conversion to the conversion server also goes very slowly (same server as the above test) I also tried a html5 upload test to file server and was able to achieve full speed.

Thanks
Ollie

 

[ollie]

Tested uploading with this php upload script and was able to get full speed also tried a different file server on yetishare and got full speed so i don't know anymore

 

[adam]

@ollie I'll run some tests locally with that script and get back to you with the results.

 

[ollie]

Not sure if this helps or not, but it seems like it uploads 2MB each time. Like it pauses for short amount of time before going onto the next 2MB

https://gyazo.com/d181385dd5627e86b73d160c051bcd29

 

[adam]

That suggests the chunk size is only 2MB. Can you check the upload_max_filesize & post_max_size on the main server? If it's set to 2MB that'll cause the chunk size to be 2MB. You should see it chunk at 100MB.

 

[ollie]

Thanks Adam, didn't realise the main server needed it too. Weird thing is that the speed was fine when i tried a different file server, oh well won't question it thanks for the help!

 

[adam]

Strange, did that resolve the upload speed issue?

 

[ollie]

Yes it has Thanks

 

[vertigo]

Hi Ollie,

Know this post is old but can you confirm the upload speed you achieved and outline how you achieved it.

We have not yet cracked the 2mb/s even with the setting noted.

We are using b2 cloud storage and I know the server only has 100mb connection, but think we should get higher upload speeds.

thx