[Insight-developers] [ITK Community] More info on download step when failed

[Bradley Lowekamp]

Great! Adding that error code/message is exactly what I was hoping for!

Fortunately, what ever the problem was with my downloads from my site is gone. 

I am going to give that topic a try on my laptop. I don't think I can upgrade all my nightly builds, due to time and questionable stability/compatibility/unknowns.

Thanks!
Brad

On Jan 15, 2014, at 8:13 PM, Matt McCormick <matt.mccormick at kitware.com> wrote:

> The initial patch should improve the robustness by attempting to re-download.
> 
> I just pushed another patch on that branch that will also display the
> log output of how it failed.  Evil proxy's beware -- your days are
> numbered ;-).
> 
> Matt
> 
> 
> 
> On Wed, Jan 15, 2014 at 4:56 PM, Bradley Lowekamp
> <blowekamp at mail.nih.gov> wrote:
>> Does this improve the log output of why/how it failed? timeout? server disconnect? evil proxy?
>> 
>> Brad
>> 
>> On Jan 15, 2014, at 4:46 PM, Matt McCormick <matt.mccormick at kitware.com> wrote:
>> 
>>> Hi Brad L.,
>>> 
>>> Here is a CMake patch that will retry the download if verification fails:
>>> 
>>> http://cmake.org/gitweb?p=stage/cmake.git;a=shortlog;h=refs/heads/robust-ep-download-verify
>>> 
>>> Please give it a try.
>>> 
>>> Thanks,
>>> Matt
>>> 
>>> On Wed, Jan 15, 2014 at 9:07 AM, Bradley Lowekamp
>>> <blowekamp at mail.nih.gov> wrote:
>>>> Cory,
>>>> 
>>>> I believe that the night build scripts don't clear out the MD5 test data cache directories which also get downloaded from midas.
>>>> 
>>>> Many of the failures occurred on systems with just 2-4 core. It's also not practical to change my build script to be serial. It'd be a waste of resources and things would just not get done.
>>>> 
>>>> Additionally the download error was also occurring from source forge too:
>>>> http://open.cdash.org/viewBuildError.php?buildid=3176009
>>>> 
>>>> Brad
>>>> 
>>>> On Jan 15, 2014, at 9:00 AM, Cory Quammen <cory.quammen at kitware.com> wrote:
>>>> 
>>>>> Hi Brad,
>>>>> 
>>>>> I've recently seen a similar problem with downloading files during a
>>>>> parallel superbuild in another project.
>>>>> 
>>>>> My wild guess was that my parallel build issued too many requests on
>>>>> the server during a short amount of time, perhaps causing some error
>>>>> in the server or perhaps triggering a denial of service attack
>>>>> detection algorithm that shut down some of the downloads. Doing a
>>>>> serial build seemed to solve the problem.
>>>>> 
>>>>> Hope that helps,
>>>>> Cory
>>>>> 
>>>>> On Wed, Jan 15, 2014 at 8:39 AM, Bradley Lowekamp
>>>>> <blowekamp at mail.nih.gov> wrote:
>>>>>> Hello,
>>>>>> 
>>>>>> I have gotten a large number of download failures on my SimpleITK and ITK
>>>>>> builds from midas3.kitware.com in the base 48 hours. It appears to be just
>>>>>> me.
>>>>>> 
>>>>>> http://open.cdash.org/index.php?project=SimpleITK&date=2014-01-14
>>>>>> 
>>>>>> What appears in the build log is as follows:
>>>>>> 
>>>>>> CMake Error at PCRE-stamp/download-PCRE.cmake:9 (file):
>>>>>> 
>>>>>> file DOWNLOAD HASH mismatch
>>>>>> 
>>>>>>  for file:
>>>>>> [/.../SimpleITK-build/PCRE-prefix/src/method=midas.bitstream.download&checksum=fa69e4c5d8971544acd71d1f10d59193&name=pcre-8.12.tar.gz]
>>>>>>    expected hash: [fa69e4c5d8971544acd71d1f10d59193]
>>>>>>      actual hash: [d41d8cd98f00b204e9800998ecf8427e]
>>>>>> 
>>>>>> where the actual hash it for that of an empty file. All the build logs I
>>>>>> have check have had this empty file hash. Last night while the errors were
>>>>>> occurring, I log onto a fail build and was able to perform to build the
>>>>>> target with the failed download after deleting the empty file.
>>>>>> 
>>>>>> I don't have any information to go on... Was it a time out? Was there an
>>>>>> error? I need some more information to go on! Any know if there is some
>>>>>> place to look? or some option I can turn on someplace?
>>>>>> 
>>>>>> Thanks,
>>>>>> Brad
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Powered by www.kitware.com
>>>>>> 
>>>>>> Visit other Kitware open-source projects at
>>>>>> http://www.kitware.com/opensource/opensource.html
>>>>>> 
>>>>>> Kitware offers ITK Training Courses, for more information visit:
>>>>>> http://kitware.com/products/protraining.php
>>>>>> 
>>>>>> Please keep messages on-topic and check the ITK FAQ at:
>>>>>> http://www.itk.org/Wiki/ITK_FAQ
>>>>>> 
>>>>>> Follow this link to subscribe/unsubscribe:
>>>>>> http://www.itk.org/mailman/listinfo/insight-developers
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Community mailing list
>>>>>> Community at itk.org
>>>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/community
>>>>>> 
>>>> 
>>>> _______________________________________________
>>>> Powered by www.kitware.com
>>>> 
>>>> Visit other Kitware open-source projects at
>>>> http://www.kitware.com/opensource/opensource.html
>>>> 
>>>> Kitware offers ITK Training Courses, for more information visit:
>>>> http://kitware.com/products/protraining.php
>>>> 
>>>> Please keep messages on-topic and check the ITK FAQ at:
>>>> http://www.itk.org/Wiki/ITK_FAQ
>>>> 
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://www.itk.org/mailman/listinfo/insight-developers
>>>> _______________________________________________
>>>> Community mailing list
>>>> Community at itk.org
>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/community
>>