Re: New redirector

On 24 dec 2008, at 10.24, Stefan Kaltenbrunner  
<stefan@kaltenbrunner.cc> wrote:

> Magnus Hagander wrote:
>> FYI - I've committed a new version of the URL redirector for  
>> downloads.
>> The old version was being used for linkfilter-breakthrough to  
>> distribute
>> viruses :-(
>> Since I was hacking around that code anyway, I didn't just add a  
>> filter
>> to it, but changed around how it works a bit. Apart from it no longer
>> being possible to use it to break through stupid linkblockers, it has
>> also made the URLs easier to read and copy/paste, and we're also  
>> storing
>> the logging information in a way that's much easier to analyze than  
>> before.
>> Do keep your eyes open for bugs, of course :-)
>
> this change broke most of the website replication code and is close  
> to running out some of the website mirrors out of diskspace. It  
> seems that the mirror script is now copying tons of /redir/ 
> <mirrorid> directories to the slaves and some of them contain  
> indvidual copies of the full source tarball for all active releases.
> This causes both disk-usage related issues as well as very long sync- 
> times between wwwmaster and the slaves...
> I don't have time to look into that more closely now so it would ge  
> good if somebody else could

Oh shit...

It shouldn't crawl explicit links to wwwmaster, I thought :( perhaps  
some place is forgetting to make it explicit?
If not, then just making it exclude everything under /redir/  wyen  
mirroring should do the trick.

Unfortunately it'll be a while before I can look at it, so I'd  
appreciate if yet someone else could!

/Magnus