Proposals:Updating Nrrd library 2009

Nrrd Library

This page describes the procedure for synchronizing the NrrdIO parts of Teem, with the one in Insight/Utilities/NrrdIO

Note that patches may have to be moved in both directions.

Background

• Nrrd started as one of the libraries in Teem <http://teem.sf.net>
• NrrdIO is a subset of Teem containing only the IO functionalities (basically, parts of the air, biff, and nrrd libraries).
• The only part of Teem that's in ITK is in NrrdIO. Also, while Teem has a weakened LGPL license, NrrdIO is licensed identically to ITK.
• Creating NrrdIO from Teem is done by a GNUMakefile (NrrdIO/pre-GNUmakefile) and a perl script (NrrdIO/unteem.pl)
• The back-propagation that should happen is from ITK's NrrdIO back to Teem, because the idea is that NrrdIO should be periodically regenerated from Teem, and ITK's NrrdIO should be an up-to-date copy of the NrrdIO automatically regenerated from Teem
• You should anticipate that the process for updating and synchronizing the libraries will take several days.
• Write access to the Teem project in SourceForge is administered by Gordon Kindlmann (ask the ITK Developers list for contact information).
• The overall process should be coordinated in the ITK Developers list, to make sure that it doesn't interfere with the release schedule

Update Life Cycle

The "SF_" prefix refers to the SVN servers at SourceForge

Generating a Fresh NrrdIO

Synchronizing changes between ITK and Teem

This is a cycle to be run until all worth-while modifications from ITK-NrrdIO have been moved into Teem

When moving changes into ITK, they can simply be copied from the generated NrrdIO into ITK.

When moving changes from ITK, they must be committed as patches to Teem SVN, and then the Generated NrrdIO should be regenerated.

Step by Step Process

Recipe for generating a fresh NrrdIO

General Overview

• Checkout NrrdIO from SourceForge (this will only be used as a source of the files listed in YYY)
• Checkout Teem from SourceForge
• Create a new empty directory( that will contain the new generated NrrdIO)
• Copy files (YYY) from the NrrdIO SourceForge checkout into the newly created directory
• Run the 11 steps script process (detailed in XXX)

Checking out Teem and NrrdIO from sourceforge

You can check-out Teem by using the SVN command:

  svn co http://teem.svn.sourceforge.net/svnroot/teem/teem/trunk teem

Use the following SVN command to checkout NrrdIO:

  svn co http://teem.svn.sourceforge.net/svnroot/teem/NrrdIO/trunk NrrdIO

This is Sourceforge's NrrdIO, which has fallen out of sync with ITK's NrrdIO. After the fixes that have accumulated in ITK's NrrdIO have been back-propagated to Sourceforge's Teem, Gordon (or others) can do whatever is needed to get them back into NrrdIO.

Files that must be taken from SourceForge NrrdIO (and not from Teem)

The official source of these files is the NrrdIO project in SourceForge.

First, get the scripts

• pre-GNUMakefile
• unteem.pl

add to them the list of files can be seen in the pre-GNUMakefile under the variables "NEED_NOT" and "TEEM_HDRS"

• NEED_NOT
  • encodingBzip2.c
  • formatEPS.c
  • formatPNG.c
  • formatPNM.c
  • formatText.c
  • formatVTK.c

• TEEM_HDRS
  • teem32bit.h
  • teemDio.h
  • teemEndian.h
  • teemPng.h
  • teemQnanhibit.h

and add the Licensing Header

• preamble.c

and the following files needed for regenerating the mangling of the library

• sample-GNUmakefile
• sampleIO.c
• mangle.pl

Code Generation

These commands are to be executed in the freshly generated NrrdIO directory.

1. cd to the Freshly generated NrrdIO directory
2. set the TEEM_SRC_ROOT environment variable to the directory where Teem wash checked out.
   • This is the directory that contains the "src" and "include" directories, as well as files like "CMakeLists.txt" and "README.txt"
   • no trailing "/" required
3. rm -f itk_NrrdIO_mangle.h
4. touch itk_NrrdIO_mangle.h
5. make -f pre-GNUmakefile clean
6. setenv ITK_NRRDIO (in csh), or export ITK_NRRDIO="" (in bash)
7. make -f pre-GNUmakefile
8. unsetenv ITK_NRRDIO (in csh), or unset ITK_NRRDIO (in bash)
9. make -f sample-GNUmakefile
10. perl mangle.pl itk >> itk_NrrdIO_mangle.h
11. make -f sample-GNUmakefile clean
12. files are ready to copy to ITK, except privateNrrd.h, where one must replace #include <zlib.h> by #include "itk_zlib.h"

Synchronizing ITK with Teem

What code base to use

In this step we will compare the files in

       Insight/Utilities/NrrdIO

with the file in the Freshly generated NrrdIO directory.

What to do

For all the differences between files in

             Insight/Utilities/NrrdIO

and files in

             Freshly generated NrrdIO directory

Do the following:

1. Understand what the change was
   1. Was this a change made in ITK CVS, that must be moved to Teem, or
   2. Was this a change made in Teem, that must be moved to ITK

In order to figure out the case, you will have to look at the SVN logs of Teem, and at the CVS logs of ITK. To identify the source of the difference.

If it is a change in ITK to be moved to Teem

1. Identify the file in Teem that is the source of the file in question in the freshly generated NrrdIO directory.
2. Apply the same change (or something equivalent) to the corresponding file in Teem (not to the NrrdIO SVN checkout)
3. Verify the Teem Dashboard, to make sure that it is green.
4. Re-run the process for generating a fresh NrrdIO from Teem

If it is a change in Teem to be moved in ITK

• Patch the file in ITK

Progress (for the Update made in 2010)

Table

The table below lists all the files in NrrdIO which are generated from a Teem checkout via NrrdIO/pre-GNUmakefile; these are marked with "*", and the originating directory in Teem is also indicated (the air, biff, and nrrd directories inside teem/src). Changes in these files in ITK's NrrdIO since GLK's last commits in 2005 should be back-propagated to the corresponding files in Teem, so that when NrrdIO is next regenerated from Teem, none of the improvements to ITK's NrrdIO are lost. Handling updates to these files is the hard/interesting part of this project.

The files not marked with a "*" are not generated from Teem via NrrdIO/pre-GNUmakefile; they exist only within NrrdIO. GLK doesn't see why there would have been any changes to these in files in ITK's NrrdIO, because it wouldn't effect how ITK uses Nrrd, but if there are changes they should also be reviewed and possibly back-propagated to Sourceforge's NrrdIO.

File	Check ITK for Bug Fix	Move from NrrdIO to ITK
000-README.txt	Not present in current Sourceforge Teem.	Keep?
* air/754.c	Change of ITK rev 1.8 present in NrrrdIO (l. 132); NrrdIO version updated (l. 358: mant0 >> 19). Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/accessors.c	ITK version 4 years old; NrrdIO version updated (static functions). Keep NrrdIO version.	Moved to ITK. 1/27/10
* air/array.c	ITK version 4 years old; NrrdIO version updated (AIR_CALLOC, check on a->data, ...). Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/arraysNrrd.c	ITK version 4 years old; NrrdIO version updated (AIR_CALLOC, check on a->data, ...). Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/axis.c	Casts from size_t to unsigned int, as in rev 1.13, no longer needed: everything is size_t now. Keep NrrdIO version.	Moved to ITK. 1/27/10
* biff/biffbiff.c	Casts from size_t to int, as in revs 1.8, 1.9, 1.10, no longer needed. Code, e.g biffAddErr() , superceded. Keep NrrdIO version.	Moved to ITK. Committed biff.h as well. 1/27/10
* biff/biffmsg.c	Does not exist in ITK version. Keep NrrdIO version.	Added and committed on 1/27/10.
CMakeLists.txt		Updated to account for new source files and reconcile flags. 1/27/10
* nrrd/comment.c	ITK version 4 years old; not much difference. Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/defaultsNrrd.c	ITK version 4 years old; not much difference. Keep NrrdIO version.	Moved to ITK. 1/27/10
* air/dio.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/encodingAscii.c	Casts from size_t to unsigned int, as in rev 1.6, no longer needed. Code, e.g sprintf()/biffAdd(), superceded. Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/encodingBzip2.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/encoding.c	ITK version 4 years old; NrrdIO version updated(static consts, biffAddf). Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/encodingGzip.c	Casts from size_t to unsigned int, as in rev 1.8, no longer needed. Code, e.g. _nrrdGzRead(),superceded. Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/encodingHex.c	Casts as in rev 1.8 in ITK now handled by AIR_CAST() in NrrdIO, + other NrrdIO version updates. Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/encodingRaw.c	ITK version 4 years old; NrrdIO version updated (temporary bug in gcc 4.2). Keep NrrdIO version (for now).	Moved to ITK. 1/28/10
* air/endianAir.c	ITK version 4 years old; NrrdIO version updated (const correctness). Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/endianNrrd.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 1/28/10
* air/enum.c	ITK version 4 years old; NrrdIO version updated (const correctness). Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/enumsNrrd.c	ITK version 4 years old; NrrdIO version updated (const correctness). Keep NrrdIO version.	Non-NrrdIO taken out. Moved to ITK. 1/27/10
* nrrd/format.c	ITK version 4 years old; NrrdIO version updated (const correctness, biffAddf). Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/formatEPS.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/formatNRRD.c	ITK version 4 years old; NrrdIO version updated (const correctness, biffAddf). Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/formatPNG.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/08/10
* nrrd/formatPNM.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/08/10
* nrrd/formatText.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/08/10
* nrrd/formatVTK.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/19/10 along with randMT.c.
* nrrd/gzio.c	Changes in ITK also found in NrrdIO, + latter version updated (const correctness, biffAddf). Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/keyvalue.c	ITK version 4 years old; NrrdIO version updated (parameters added _nrrdWriteEscaped(), _nrrdKeyValueWrite()). Keep NrrdIO version.	Moved to ITK. 1/28/10
mangle.pl	Not present in current Sourceforge Teem.	Keep?
* nrrd/methodsNrrd.c	Most changes in ITK last 4 years superceded by NrrdIO version (e.g. biffAddf() ). Rollback in rev 1.10 mysterious. Keep NrrdIO version (for now).	Moved to ITK. 1/27/10
* air/miscAir.c	ITK version 4 years old; minor changes (float to double, ...) in NrrdIO version. Keep NrrdIO version.	Non-NrrdIO taken out. Moved to ITK. 1/27/10
* air/mop.c	ITK version 4 years old; minor change (airMopAdd now returns int) in NrrdIO version. Keep NrrdIO version.	Moved to ITK. 1/27/10
* NrrdIO.h	Latest ITK changes, plus recent changes (#ifdef __APPLE__ l 696) seem to appear in NrrdIO version. Keep NrrdIO version.	Use NrrdIO version, and modify it to account for #include "itk_NrrdIO_mangle.h" and TEEM_API as in ITK. Updated in ITK. 1/28/10
NrrdIO_Srcs.txt	Not found in NrrdIO repository.	Not applicable.
* air/parseAir.c	ITK version 4 years old; recent changes (const correctness, AIR_CAST...) in NrrdIO version. Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/parseNrrd.c	ITK casts to unsigned int, rev 1.11, superceded in NrrdIO by dealing with size_t + other changes. Rollback in rev 1.10 mysterious. Keep NrrdIO version (for now).	Moved to ITK. 1/27/10
preamble.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/08/10
pre-GNUmakefile	ITK version 4 years old; minor changes (addprefix l 83, ...) in NrrdIO version. Keep NrrdIO version, but ...	... ITK currently uses cvs not svn.
* air/privateAir.h	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/privateNrrd.h	Rollback in rev 1.12 mysterious. Keep NrrdIO version (for now).	ITK version should use #include "itk_zlib.h", as currently in ITK. Moved to ITK. 1/28/10
qnanhibit.c	Not present in current Sourceforge Teem.	Keep?
* nrrd/read.c	Rollback in rev 1.9 mysterious. Keep NrrdIO version (for now). Add logic for separators in 1.11.	Moved to ITK. 1/27/10
* nrrd/reorder.c	Cast in rev. 1.10 missing (via AIR_CAST), look at pragma (array bound warning) in rev 1.11.	Moved to ITK. 1/27/10
sample-GNUmakefile	Not present in current Sourceforge Teem.	Keep?
sampleIO.c	Not present in current Sourceforge Teem.	Keep?
* air/sane.c	ITK version 4 years old; recent changes in NrrdIO version (airInsane_AIR_NAN). Keep NrrdIO version.	Causes failing tests in cross-compiled Mac.
* nrrd/simple.c	ITK casts to int, rev 1.13, superceded in NrrdIO by dealing with size_t + other changes (const). Keep NrrdIO version.	Moved to ITK. 1/27/10
* air/string.c	ITK casts either superceded in NrrdIO by dealing with size_t or handled by AIR_CAST. Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/subset.c	Rollback in rev 1.7 mysterious. Recent changes in NrrdIO version. Start with NrrdIO version (for now), but look at pragma (array bound warning) in rev 1.8.	Moved to ITK. 1/28/10
* include/teem32bit.h	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/08/10
* include/teemDio.h	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/08/10
* include/teemEndian.h	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/08/10
* include/teemPng.h	ITK version 4 years old; NrrdIO has "vtk_png_mangle.h". Keep NrrdIO version.	Moved to ITK. 2/08/10
* include/teemQnanhibit.h	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/08/10
unteem.pl	Not present in current Sourceforge Teem.	Keep?
* nrrd/write.c	Reconciliation of NrrdIO & ITK versions req'd. !!airStrlen vs (int)airStrlen; rollback in rev 1.10 mysterious; airSinglePrintf(NULL, buff, "%.17lg", val[dd])-type fixes as in ITK rev 1.12.	Moved to ITK. 1/27/10

Reference Material

NrrdIO / Teem Dashboard

• http://my.cdash.org/index.php?project=Teem

For the Curious Minds

What pre-GNUmakefile does

The "pre-GNUmakefile" makefile

• Copies files from a Teem checkout
• Modifies them slightly (through the action of unteem.pl). Specifically, it excises lines of code that are delimited by "/* ---- BEGIN non-NrrdIO */" and "/* ---- END non-NrrdIO */"; this is code that Teem needs but NrrdIO does not.

What sample-GNUmakefile does

Then "sample-GNUmakefile"

• Builds a NrrdIO library
• Builds a small test executable

What mangle.pl does

The script "mangle.pl"

• Looks at all the symbols in the library in order to generate the name-mangling header file required for ITK.

Extra Information (Not part of the process)

If we weren't making NrrdIO for ITK, the name mangling wouldn't be required, and the process would just be:

• make -f pre-GNUmakefile clean
• make -f pre-GNUmakefile
• make -f sample-GNUmakefile
• make -f sample-GNUmakefile clean

where we run make with "sample-GNUmakefile" just to make sure that everything compiles ok.