While deleting urls from an abusive user I noticed that majority were 404. Therefore, have built a quick and simple logic that goes through DB url's and checks HTTP response code. It follows 301/302 etc. and in my case removes all urls with 404 response but it could be a page stating that the destination doesn't exist any more. There are cases when server response is extremely slow and these are marked for additional separate drill-down with extended timout. Some are infinite 301/302 loops and for these max hops are set to X number then treated as 404 / removed. These are some ideas. Am thinking about implementing an extra check for page content e.g. see if it contains 'not found' or 'denied' key words etc. to double-check or handle in different ways other codes 5XX.