[vtk-developers] Archives are back (was: www.kitware.com/search.html)
sebastien.barre at kitware.com
Wed May 29 10:46:07 EDT 2002
At 5/29/2002 10:04 AM, Andy Cedilnik wrote:
>Ok, indexing now works properly. We had to hack ht:/dig to ignore robots
>information, but we are all set now.
Just for info (since I will forget all of this in 2 days).
At the moment we indeed have to patch either HtDig or Mailman.
There is an option in HtDig to ignore the robots information, this option
being just the fact that HtDig will comply with the robots.txt specifications:
A Standard for Robot Exclusion
In robots.txt, you can provide exclusion on a User-agent basis, thus
allowing HtDig and disallowing everything else.
Sadly, mailman 2.0.x hardcodes robots exclusion in:
<META NAME="robots" CONTENT="index,nofollow">
This will be solved in mailman 2.2, where the article body will be a
template that can be customized for each list :
in that case, we will just have to remove the <META> line in the list
template, and modify the robots.txt accordingly. No patch :)
More information about the vtk-developers