[Insight-developers] [ITK Community] More info on download step when failed

[Matt McCormick]

The initial patch should improve the robustness by attempting to re-download.

I just pushed another patch on that branch that will also display the
log output of how it failed.  Evil proxy's beware -- your days are
numbered ;-).

Matt



On Wed, Jan 15, 2014 at 4:56 PM, Bradley Lowekamp
<blowekamp at mail.nih.gov> wrote:
> Does this improve the log output of why/how it failed? timeout? server disconnect? evil proxy?
>
> Brad
>
> On Jan 15, 2014, at 4:46 PM, Matt McCormick <matt.mccormick at kitware.com> wrote:
>
>> Hi Brad L.,
>>
>> Here is a CMake patch that will retry the download if verification fails:
>>
>>  http://cmake.org/gitweb?p=stage/cmake.git;a=shortlog;h=refs/heads/robust-ep-download-verify
>>
>> Please give it a try.
>>
>> Thanks,
>> Matt
>>
>> On Wed, Jan 15, 2014 at 9:07 AM, Bradley Lowekamp
>> <blowekamp at mail.nih.gov> wrote:
>>> Cory,
>>>
>>> I believe that the night build scripts don't clear out the MD5 test data cache directories which also get downloaded from midas.
>>>
>>> Many of the failures occurred on systems with just 2-4 core. It's also not practical to change my build script to be serial. It'd be a waste of resources and things would just not get done.
>>>
>>> Additionally the download error was also occurring from source forge too:
>>> http://open.cdash.org/viewBuildError.php?buildid=3176009
>>>
>>> Brad
>>>
>>> On Jan 15, 2014, at 9:00 AM, Cory Quammen <cory.quammen at kitware.com> wrote:
>>>
>>>> Hi Brad,
>>>>
>>>> I've recently seen a similar problem with downloading files during a
>>>> parallel superbuild in another project.
>>>>
>>>> My wild guess was that my parallel build issued too many requests on
>>>> the server during a short amount of time, perhaps causing some error
>>>> in the server or perhaps triggering a denial of service attack
>>>> detection algorithm that shut down some of the downloads. Doing a
>>>> serial build seemed to solve the problem.
>>>>
>>>> Hope that helps,
>>>> Cory
>>>>
>>>> On Wed, Jan 15, 2014 at 8:39 AM, Bradley Lowekamp
>>>> <blowekamp at mail.nih.gov> wrote:
>>>>> Hello,
>>>>>
>>>>> I have gotten a large number of download failures on my SimpleITK and ITK
>>>>> builds from midas3.kitware.com in the base 48 hours. It appears to be just
>>>>> me.
>>>>>
>>>>> http://open.cdash.org/index.php?project=SimpleITK&date=2014-01-14
>>>>>
>>>>> What appears in the build log is as follows:
>>>>>
>>>>> CMake Error at PCRE-stamp/download-PCRE.cmake:9 (file):
>>>>>
>>>>> file DOWNLOAD HASH mismatch
>>>>>
>>>>>   for file:
>>>>> [/.../SimpleITK-build/PCRE-prefix/src/method=midas.bitstream.download&checksum=fa69e4c5d8971544acd71d1f10d59193&name=pcre-8.12.tar.gz]
>>>>>     expected hash: [fa69e4c5d8971544acd71d1f10d59193]
>>>>>       actual hash: [d41d8cd98f00b204e9800998ecf8427e]
>>>>>
>>>>> where the actual hash it for that of an empty file. All the build logs I
>>>>> have check have had this empty file hash. Last night while the errors were
>>>>> occurring, I log onto a fail build and was able to perform to build the
>>>>> target with the failed download after deleting the empty file.
>>>>>
>>>>> I don't have any information to go on... Was it a time out? Was there an
>>>>> error? I need some more information to go on! Any know if there is some
>>>>> place to look? or some option I can turn on someplace?
>>>>>
>>>>> Thanks,
>>>>> Brad
>>>>>
>>>>> _______________________________________________
>>>>> Powered by www.kitware.com
>>>>>
>>>>> Visit other Kitware open-source projects at
>>>>> http://www.kitware.com/opensource/opensource.html
>>>>>
>>>>> Kitware offers ITK Training Courses, for more information visit:
>>>>> http://kitware.com/products/protraining.php
>>>>>
>>>>> Please keep messages on-topic and check the ITK FAQ at:
>>>>> http://www.itk.org/Wiki/ITK_FAQ
>>>>>
>>>>> Follow this link to subscribe/unsubscribe:
>>>>> http://www.itk.org/mailman/listinfo/insight-developers
>>>>>
>>>>> _______________________________________________
>>>>> Community mailing list
>>>>> Community at itk.org
>>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/community
>>>>>
>>>
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Kitware offers ITK Training Courses, for more information visit:
>>> http://kitware.com/products/protraining.php
>>>
>>> Please keep messages on-topic and check the ITK FAQ at:
>>> http://www.itk.org/Wiki/ITK_FAQ
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://www.itk.org/mailman/listinfo/insight-developers
>>> _______________________________________________
>>> Community mailing list
>>> Community at itk.org
>>> http://public.kitware.com/cgi-bin/mailman/listinfo/community
>