[Insight-developers] Tarball of external data files

[Bradley Lowekamp]

I am in favor of the environmental variable. I went to the trouble of setting the CMAKE var to point to a common place, but after I blew a away the build tree I neglected to set it back, as it has already downloaded  again I haven't bothered setting it back to the common location.

Brad L.


On Nov 28, 2011, at 12:29 PM, Johnson, Hans J wrote:

> Brad and Cory,
> 
> I've been struggling with this as well.  I've been pondering a "set and
> forget" method to help the 7 developers (each with 3 or more build trees)
> in my lab get this speed up without having to hassle with custom cmake
> configuration each time.
> 
> My idea is to have an "ITK_GLOBAL_DATA_CACHE_PATHS" environmental
> variable, that if set will look there for object files.   Then I can have
> our system administrator set this ENV variable once to our shared NFS
> mount point for our entire system and there would be no end user training
> needed.
> 
> For a minimal amount of end user training of non-connected laptops, each
> person could download once, and set their personal cache in their .bashrc
> file.
> 
> ================================
> My only hesitation to a solution like this is if it has enough global
> appeal to introduce such a variable for ITK.
> 
> Hans
> 
> 
> --
> Hans J. Johnson, Ph.D.
> hans-johnson at uiowa.edu
> Assistant Professor of Psychiatry
> University of Iowa Carver College of Medicine
> W278 GH, 200 Hawkins Drive
> 
> Iowa City, Iowa 52242
> Phone:  319-353-8587
> 
> 
> 
> 
> 
> 
> 
> -----Original Message-----
> From: Brad King <brad.king at kitware.com>
> Date: Mon, 28 Nov 2011 12:07:22 -0500
> To: Cory Quammen <cquammen at cs.unc.edu>
> Cc: ITK <insight-developers at itk.org>
> Subject: Re: [Insight-developers] Tarball of external data files
> 
> On 11/28/2011 11:48 AM, Cory Quammen wrote:
>> While waiting for the completion of 'make ITKData' run for the first
>> time on a new machine, I had time to ponder how to speed up the
>> initial download of ITK's external test data. My solution was to tar
>> up the external data directory that already had the complete data on
>> one machine, transfer it to my new machine, untar it, set the
>> ExternalData_OBJECT_STORES to this directory, and voila, no need to
>> wait the 20-30 minutes that the usual one-by-one download procedure
>> takes when you invoke 'make ITKData' for the first time.
> 
> Nice.  I've been meaning to work on a tool like this but never had time.
> 
> FYI, the ITK release tarballs will come with a .ExternalData directory
> containing all the objects referenced by content links in the tarballs.
> That way release builds won't need to download anything.  The script
> that computes the tarballs is here:
> 
> 
> http://itk.org/gitweb?p=ITK.git;a=blob;f=Utilities/Maintenance/SourceTarbal
> l.bash;hb=v4.0rc03
> 
> Returning to your topic, what we need is a more efficient object
> transport for "make ITKData" to use.  Git uses "packs" to achieve the
> same thing, but it can efficiently compute what objects the client
> needs and produce a custom pack on the server because it knows the
> commit history missing from the client.  All of our objects are
> independent.
> 
> A tarball of objects for getting started is a simple approximation of
> Git's "pack" concept.  Further progress would require client and server
> protocol implementations to download a bunch of objects simultaneously
> from the server and then extract them on the client.
> 
> -Brad
> _______________________________________________
> Powered by www.kitware.com
> 
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
> 
> Kitware offers ITK Training Courses, for more information visit:
> http://kitware.com/products/protraining.html
> 
> Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
> 
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-developers
> 
> 
> 
> ________________________________
> Notice: This UI Health Care e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential and may be legally privileged.  If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited.  Please reply to the sender that you have received the message in error, then delete it.  Thank you.
> ________________________________
> _______________________________________________
> Powered by www.kitware.com
> 
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
> 
> Kitware offers ITK Training Courses, for more information visit:
> http://kitware.com/products/protraining.html
> 
> Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
> 
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-developers