[vtk-developers] Archives are back (was: www.kitware.com/search.html)

[Sebastien BARRE]

At 5/29/2002 10:04 AM, Andy Cedilnik wrote:
>Hello!
>
>Ok, indexing now works properly. We had to hack ht:/dig to ignore robots
>information, but we are all set now.

Just for info (since I will forget all of this in 2 days).
At the moment we indeed have to patch either HtDig or Mailman.

There is an option in HtDig to ignore the robots information, this option 
being just the fact that HtDig will comply with the robots.txt specifications:

A Standard for Robot Exclusion
http://www.robotstxt.org/wc/norobots.html

In robots.txt, you can provide exclusion on a User-agent basis, thus 
allowing HtDig and disallowing everything else.

Sadly, mailman 2.0.x hardcodes robots exclusion in:
         Mailman\Archiver\HyperArch.py:
[...]
    <META NAME="robots" CONTENT="index,nofollow">

This will be solved in mailman 2.2, where the article body will be a 
template that can be customized for each list :
         templates\*\article.html
in that case, we will just have to remove the <META> line in the list 
template, and modify the robots.txt accordingly. No patch :)


--
Sebastien Barre