[Insight-developers] Tarball of external data files

[Johnson, Hans J]

Brad and Cory,

I've been struggling with this as well.  I've been pondering a "set and
forget" method to help the 7 developers (each with 3 or more build trees)
in my lab get this speed up without having to hassle with custom cmake
configuration each time.

My idea is to have an "ITK_GLOBAL_DATA_CACHE_PATHS" environmental
variable, that if set will look there for object files.   Then I can have
our system administrator set this ENV variable once to our shared NFS
mount point for our entire system and there would be no end user training
needed.

For a minimal amount of end user training of non-connected laptops, each
person could download once, and set their personal cache in their .bashrc
file.

================================
My only hesitation to a solution like this is if it has enough global
appeal to introduce such a variable for ITK.

Hans


--
Hans J. Johnson, Ph.D.
hans-johnson at uiowa.edu
Assistant Professor of Psychiatry
University of Iowa Carver College of Medicine
W278 GH, 200 Hawkins Drive

Iowa City, Iowa 52242
Phone:  319-353-8587







-----Original Message-----
From: Brad King <brad.king at kitware.com>
Date: Mon, 28 Nov 2011 12:07:22 -0500
To: Cory Quammen <cquammen at cs.unc.edu>
Cc: ITK <insight-developers at itk.org>
Subject: Re: [Insight-developers] Tarball of external data files

On 11/28/2011 11:48 AM, Cory Quammen wrote:
> While waiting for the completion of 'make ITKData' run for the first
> time on a new machine, I had time to ponder how to speed up the
> initial download of ITK's external test data. My solution was to tar
> up the external data directory that already had the complete data on
> one machine, transfer it to my new machine, untar it, set the
> ExternalData_OBJECT_STORES to this directory, and voila, no need to
> wait the 20-30 minutes that the usual one-by-one download procedure
> takes when you invoke 'make ITKData' for the first time.

Nice.  I've been meaning to work on a tool like this but never had time.

FYI, the ITK release tarballs will come with a .ExternalData directory
containing all the objects referenced by content links in the tarballs.
That way release builds won't need to download anything.  The script
that computes the tarballs is here:


http://itk.org/gitweb?p=ITK.git;a=blob;f=Utilities/Maintenance/SourceTarbal
l.bash;hb=v4.0rc03

Returning to your topic, what we need is a more efficient object
transport for "make ITKData" to use.  Git uses "packs" to achieve the
same thing, but it can efficiently compute what objects the client
needs and produce a custom pack on the server because it knows the
commit history missing from the client.  All of our objects are
independent.

A tarball of objects for getting started is a simple approximation of
Git's "pack" concept.  Further progress would require client and server
protocol implementations to download a bunch of objects simultaneously
from the server and then extract them on the client.

-Brad
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html

Kitware offers ITK Training Courses, for more information visit:
http://kitware.com/products/protraining.html

Please keep messages on-topic and check the ITK FAQ at:
http://www.itk.org/Wiki/ITK_FAQ

Follow this link to subscribe/unsubscribe:
http://www.itk.org/mailman/listinfo/insight-developers



________________________________
Notice: This UI Health Care e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential and may be legally privileged.  If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited.  Please reply to the sender that you have received the message in error, then delete it.  Thank you.
________________________________