When you go into the admin panel under Files, you click on the Download button, it takes you to the Fileserver URL to download when it should be the URL of your primary site. Same thing happens when you get the file download URL's after uploading files.
eg: Download URL should be...