Hi Ollie,
Is your site on https? If yes ftp is non secure so won't have the overhead of encryption. Might be worth testing upload speed via scp.
Also are you using apache or nginx? The web server will handle the upload itself, then when done its passed to php, so I'd say the web server is the...