[Insight-developers] [ITK Community] More info on download step when failed

[Bradley Lowekamp]

Does this improve the log output of why/how it failed? timeout? server disconnect? evil proxy?

Brad

On Jan 15, 2014, at 4:46 PM, Matt McCormick <matt.mccormick at kitware.com> wrote:

> Hi Brad L.,
> 
> Here is a CMake patch that will retry the download if verification fails:
> 
>  http://cmake.org/gitweb?p=stage/cmake.git;a=shortlog;h=refs/heads/robust-ep-download-verify
> 
> Please give it a try.
> 
> Thanks,
> Matt
> 
> On Wed, Jan 15, 2014 at 9:07 AM, Bradley Lowekamp
> <blowekamp at mail.nih.gov> wrote:
>> Cory,
>> 
>> I believe that the night build scripts don't clear out the MD5 test data cache directories which also get downloaded from midas.
>> 
>> Many of the failures occurred on systems with just 2-4 core. It's also not practical to change my build script to be serial. It'd be a waste of resources and things would just not get done.
>> 
>> Additionally the download error was also occurring from source forge too:
>> http://open.cdash.org/viewBuildError.php?buildid=3176009
>> 
>> Brad
>> 
>> On Jan 15, 2014, at 9:00 AM, Cory Quammen <cory.quammen at kitware.com> wrote:
>> 
>>> Hi Brad,
>>> 
>>> I've recently seen a similar problem with downloading files during a
>>> parallel superbuild in another project.
>>> 
>>> My wild guess was that my parallel build issued too many requests on
>>> the server during a short amount of time, perhaps causing some error
>>> in the server or perhaps triggering a denial of service attack
>>> detection algorithm that shut down some of the downloads. Doing a
>>> serial build seemed to solve the problem.
>>> 
>>> Hope that helps,
>>> Cory
>>> 
>>> On Wed, Jan 15, 2014 at 8:39 AM, Bradley Lowekamp
>>> <blowekamp at mail.nih.gov> wrote:
>>>> Hello,
>>>> 
>>>> I have gotten a large number of download failures on my SimpleITK and ITK
>>>> builds from midas3.kitware.com in the base 48 hours. It appears to be just
>>>> me.
>>>> 
>>>> http://open.cdash.org/index.php?project=SimpleITK&date=2014-01-14
>>>> 
>>>> What appears in the build log is as follows:
>>>> 
>>>> CMake Error at PCRE-stamp/download-PCRE.cmake:9 (file):
>>>> 
>>>> file DOWNLOAD HASH mismatch
>>>> 
>>>>   for file:
>>>> [/.../SimpleITK-build/PCRE-prefix/src/method=midas.bitstream.download&checksum=fa69e4c5d8971544acd71d1f10d59193&name=pcre-8.12.tar.gz]
>>>>     expected hash: [fa69e4c5d8971544acd71d1f10d59193]
>>>>       actual hash: [d41d8cd98f00b204e9800998ecf8427e]
>>>> 
>>>> where the actual hash it for that of an empty file. All the build logs I
>>>> have check have had this empty file hash. Last night while the errors were
>>>> occurring, I log onto a fail build and was able to perform to build the
>>>> target with the failed download after deleting the empty file.
>>>> 
>>>> I don't have any information to go on... Was it a time out? Was there an
>>>> error? I need some more information to go on! Any know if there is some
>>>> place to look? or some option I can turn on someplace?
>>>> 
>>>> Thanks,
>>>> Brad
>>>> 
>>>> _______________________________________________
>>>> Powered by www.kitware.com
>>>> 
>>>> Visit other Kitware open-source projects at
>>>> http://www.kitware.com/opensource/opensource.html
>>>> 
>>>> Kitware offers ITK Training Courses, for more information visit:
>>>> http://kitware.com/products/protraining.php
>>>> 
>>>> Please keep messages on-topic and check the ITK FAQ at:
>>>> http://www.itk.org/Wiki/ITK_FAQ
>>>> 
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://www.itk.org/mailman/listinfo/insight-developers
>>>> 
>>>> _______________________________________________
>>>> Community mailing list
>>>> Community at itk.org
>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/community
>>>> 
>> 
>> _______________________________________________
>> Powered by www.kitware.com
>> 
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>> 
>> Kitware offers ITK Training Courses, for more information visit:
>> http://kitware.com/products/protraining.php
>> 
>> Please keep messages on-topic and check the ITK FAQ at:
>> http://www.itk.org/Wiki/ITK_FAQ
>> 
>> Follow this link to subscribe/unsubscribe:
>> http://www.itk.org/mailman/listinfo/insight-developers
>> _______________________________________________
>> Community mailing list
>> Community at itk.org
>> http://public.kitware.com/cgi-bin/mailman/listinfo/community