VTK/Git/Develop/Data: Difference between revisions

From KitwarePublic
< VTK‎ | Git‎ | Develop
Jump to navigationJump to search
(Replaced content with "The instructions previously available on this page have been superseded. See [https://gitlab.kitware.com/vtk/vtk/blob/master/Documentation/dev/git/data.md here].")
 
Line 1: Line 1:
This page documents how to add test data while developing VTK.
The instructions previously available on this page have been supersededSee [https://gitlab.kitware.com/vtk/vtk/blob/master/Documentation/dev/git/data.md here].
See our [[VTK/Git|table of contents]] for more information.
__TOC__
= Setup =
 
The workflow below depends on local hooks to function properly.
Follow the main [[VTK/Git/Develop#Setup|developer setup instructions]] before proceeding.
In particular, run
[http://vtk.org/gitweb?p=VTK.git;a=blob;f=Utilities/SetupForDevelopment.sh;hb=HEAD <code>SetupForDevelopment.sh</code>]:
 
$ ./Utilities/SetupForDevelopment.sh
 
= Workflow =
 
Our workflow for adding data integrates with our standard Git [[VTK/Git/Develop|development process]].
Start by [[VTK/Git/Develop#Create_a_Topic|creating a topic]].
Return here when you reach the "edit files" step.
 
These instructions follow a typical use case of adding a new test with a baseline image.
 
== Add Data ==
 
{| style="width: 100%"
|-
|width=60%|
Copy the data file into your local source tree.
|-
|
:<code>$ mkdir -p ''Some/Module''/Testing/Data/Baseline</code>
:<code>$ cp ~/''MyTest.png'' ''Some/Module''/Testing/Data/Baseline/''MyTest.png''</code>
|align="center"|
[[#Putting it all together|How do you generate baseline images in the first place]]?
|}
 
== Add Test ==
 
{| style="width: 100%"
|-
|width=60%|
Edit the test CMakeLists.txt file and add the test using a <code>vtk_add_test_...</code> call (which references baselines automatically).
For tests not using such a call, reference the data file in an <code>ExternalData_add_test</code> call.
Specify the file inside <code>DATA{...}</code> using a path relative to the test directory:
:<code>$ edit ''Some/Module''/Testing/Cxx/CMakeLists.txt</code>
:{|
|
ExternalData_add_test(VTKData
  NAME ${vtk-module}Cxx-MyTest
  COMMAND ${vtk-module}CxxTests MyTest
          ... -V DATA{../Data/Baseline/MyTest.png,:} ...
  )
|}
* If the data file references other data files, e.g. <code>.mhd -> .raw</code>, follow the link to the ExternalData module on the right and read the documentation on "associated" files.
* Multiple baseline images and other series are handled automatically when the reference ends in the ",:" option; follow the link to the ExternalData module on the right for details.
|align="center"|
[http://vtk.org/gitweb?p=VTK.git;a=blob;f=CMake/ExternalData.cmake;hb=HEAD <code>ExternalData.cmake</code>]
|}
 
== Run CMake ==
 
{| style="width: 100%"
|-
|width=60%|
''CMake will [[#ExternalData|move the original file]].  Keep your own copy if necessary.''
 
Run cmake on the build tree:
:<code>$ cd ../VTK-build</code>
:<code>$ cmake .</code>
:''(Or just run "make" to do a full configuration and build.)''
:<code>$ cd ../VTK</code>
|align="center"|
[[#Recover_Data_File|Need to recover the original file]]?
|-
|
During configuration CMake will display a message such as:
:{|
|
Linked ''Some/Module''/Testing/Data/Baseline/''MyTest.png''.md5 to ExternalData MD5/...
|}
This means that CMake converted the file into a data object referenced by a "content link".
|align="center"|
[http://vtk.org/gitweb?p=VTK.git;a=blob;f=CMake/ExternalData.cmake;hb=HEAD <code>ExternalData.cmake</code>]
|}
 
== Commit ==
 
{| style="width: 100%"
|-
|width=60%|
Continue to [[VTK/Git/Develop#Create_a_Topic|create the topic]] and edit other files as necessary.
Add the content link and commit it along with the other changes:
:<code>$ git add ''Some/Module''/Testing/Data/Baseline/''MyTest.png''.md5</code>
:<code>$ git add ''Some/Module''/Testing/Data/CMakeLists.txt</code>
:<code>$ git commit</code>
|align="center"|
[http://www.kernel.org/pub/software/scm/git/docs/git-add.html <code>git help add</code>]
<br/>
[http://www.kernel.org/pub/software/scm/git/docs/git-commit.html <code>git help commit</code>]
|-
|
The local <code>pre-commit</code> hook will display a message such as:
:{|
|
''Some/Module''/Testing/Data/Baseline/''MyTest.png''.md5: Added content to Git at refs/data/MD5/...
''Some/Module''/Testing/Data/Baseline/''MyTest.png''.md5: Added content to local store at .ExternalData/MD5/...
Content link ''Some/Module''/Testing/Data/Baseline/''MyTest.png''.md5 -> .ExternalData/MD5/...
|}
This means that the pre-commit hook recognized that the content link references a new data object and [[#pre-commit|prepared it for upload]].
|align="center"|
[http://vtk.org/gitweb?p=VTK.git;a=blob;f=Utilities/Scripts/pre-commit;hb=HEAD <code>pre-commit</code>]
|}
 
== Push ==
{| style="width: 100%"
|-
|width=60%|
Follow the instructions to [[VTK/Git/Develop#Share_a_Topic|share the topic]].
When you push it to Gerrit for review using
:<code>$ git gerrit-push</code>
|align="center"|
[http://vtk.org/gitweb?p=VTK.git;a=blob;f=Utilities/Scripts/git-gerrit-push;hb=HEAD <code>git-gerrit-push</code>]
|-
|
part of the output will be of the form
:{|
|
*      ...:refs/data/commits/...      [new branch]
*      HEAD:refs/for/master/''my-topic''  [new branch]
Pushed refs/data and removed local copy:
  MD5/...
|}
This means that the git-gerrit-push script pushed the topic and [[#git-gerrit-push|uploaded the data]] it references.
 
Options for <code>gerrit-push</code>:
* <code>--dry-run</code>: Report push that would occur without actually doing it
* <code>--no-topic</code>: Push the data referenced by the topic but not the topic itself
 
'''Note: One must <code>git gerrit-push</code> from the same work tree as was used to create the commit.  Do not <code>git push</code> to another computer first and try to push to Gerrit from there.  The data will not follow.'''
|}
 
= Building =
 
== Download ==
{| style="width: 100%"
|-
|width=60%|
For the test data to be downloaded and made available to the tests in your build tree the <code>VTKData</code> target must be built.
One may build the target directly, e.g. <code>make VTKData</code>, to obtain the data without a complete build.
The output will be something like
:{|
|
-- Fetching ".../ExternalData/MD5/..."
-- [download 100% complete]
-- Downloaded object: "''VTK-build''/ExternalData/Objects/MD5/..."
|}
|-
|
The downloaded files appear in <code>''VTK-build''/ExternalData</code> by default.
|
|}
 
== Local Store ==
{| style="width: 100%"
|-
|width=60%|
It is possible to configure one or more local ExternalData object stores shared among multiple builds.
Configure for each build the advanced cache entry <code>ExternalData_OBJECT_STORES</code> to a directory on your local disk outside all build trees, e.g. "<code>/home/user/.ExternalData</code>":
:<code>$ cmake -DExternalData_OBJECT_STORES=/home/user/.ExternalData ../VTK</code>
The ExternalData module will store downloaded objects in the local store instead of the build tree.
Once an object has been downloaded by one build it will persist in the local store for re-use by other builds without downloading again.
|align="center"|
[http://vtk.org/gitweb?p=VTK.git;a=blob;f=CMake/ExternalData.cmake;hb=HEAD <code>ExternalData.cmake</code>]
|}
 
= Discussion =
 
A VTK test data file is not stored in the main source tree under version control.
Instead the source tree contains a "content link" that refers to a data object by a hash of its content.
At build time the
[http://vtk.org/gitweb?p=VTK.git;a=blob;f=CMake/ExternalData.cmake;hb=HEAD <code>ExternalData.cmake</code>]
module fetches data needed by enabled tests.
This allows arbitrarily large data to be added and removed without bloating the version control history.
 
The above [[#Workflow|workflow]] allows developers to add a new data file almost as if committing it to the source tree.
The following subsections discuss details of the workflow implementation.
 
== ExternalData ==
 
While [[#Run_CMake|CMake runs]] the
[http://vtk.org/gitweb?p=VTK.git;a=blob;f=CMake/ExternalData.cmake;hb=HEAD ExternalData]
module evaluates [[#Add_Test|DATA{} references]].
VTK [http://vtk.org/gitweb?p=VTK.git;a=blob;f=CMake/vtkExternalData.cmake;hb=HEAD sets]
the <code>ExternalData_LINK_CONTENT</code> option to <code>MD5</code> to enable automatic conversion of raw data files into content links.
When the module detects a real data file in the source tree it performs the following transformation as specified in the module documentation:
* Compute the MD5 hash of the file
* Store the <code>${hash}</code> in a file with the original name plus <code>.md5</code>
* Rename the original file to <code>.ExternalData_MD5_${hash}</code>
The real data now sit in a file that we [http://vtk.org/gitweb?p=VTK.git;a=blob;f=.gitignore;hb=HEAD tell Git to ignore].
For example:
 
  $ '''cat ''Some/Module''/Testing/Data/Baseline/.ExternalData_MD5_477e602800c18624d9bc7a32fa706b97 |md5sum'''
477e602800c18624d9bc7a32fa706b97  -
$ '''cat ''Some/Module''/Testing/Data/Baseline/''MyTest.png''.md5'''
477e602800c18624d9bc7a32fa706b97
 
=== Recover Data File ===
 
To recover the original file after running CMake but before committing, undo the operation:
 
$ '''cd ''Some/Module''/Testing/Data/Baseline'''
$ '''mv .ExternalData_MD5_$(cat MyTest.png.md5) MyTest.png'''
 
== pre-commit ==
 
While [[#Commit|committing]] a new or modified content link the
[http://vtk.org/gitweb?p=VTK.git;a=blob;f=Utilities/Scripts/pre-commit;hb=HEAD <code>pre-commit</code>]
hook moves the real data object from the <code>.ExternalData_MD5_${hash}</code> file left by the ExternalData module
to a local object repository stored in a <code>.ExternalData</code> directory at the top of the source tree.
 
The hook also uses Git plumbing commands to store the data object as a blob in the local Git repository.
The blob is not referenced by the new commit but instead by <code>refs/data/MD5/${hash}</code>.
This keeps the blob alive in the local repository but does not add it to the project history.
For example:
$ '''git for-each-ref --format="%(refname)" refs/data'''
refs/data/MD5/477e602800c18624d9bc7a32fa706b97
$ '''git cat-file blob refs/data/MD5/477e602800c18624d9bc7a32fa706b97 | md5sum'''
477e602800c18624d9bc7a32fa706b97  -
 
== git gerrit-push ==
 
The "<code>git gerrit-push</code>" command is actually an
[http://vtk.org/gitweb?p=VTK.git;a=blob;f=Utilities/DevelopmentSetupScripts/SetupGitAliases.sh;hb=HEAD alias]
for the
[http://vtk.org/gitweb?p=VTK.git;a=blob;f=Utilities/Scripts/git-gerrit-push;hb=HEAD <code>Utilities/Scripts/git-gerrit-push</code>]
script.
In addition to pushing the topic branch to Gerrit the script also detects content links added or modified by the commits in the topic.
It reads the data object hashes from the content links and looks for matching <code>refs/data/</code> entries in the local Git repository.
 
The script pushes the matching data objects to Gerrit inside a temporary commit object disjoint from the rest of history.
For example:
 
$ '''git gerrit-push --dry-run --no-topic'''
*      f59717cfb68a7093010d18b84e8a9a90b6b42c11:refs/data/commits/f59717cfb68a7093010d18b84e8a9a90b6b42c11    [new branch]
Pushed refs/data and removed local copy:
  MD5/477e602800c18624d9bc7a32fa706b97
$ '''git ls-tree -r --name-only f59717cf'''
MD5/477e602800c18624d9bc7a32fa706b97
$ '''git log --oneline f59717cf'''
f59717c data
 
A robot runs every few minutes to fetch the objects from Gerrit and upload them to a
[http://www.vtk.org/files/ExternalData location] that we
[http://vtk.org/gitweb?p=VTK.git;a=blob;f=CMake/vtkExternalData.cmake;hb=HEAD tell ExternalData to search]
at build time.
 
== Putting it all together ==
 
How do you generate baseline images in the first place? If a baseline image is missing, CMake will warn you about it, but you can still configure, build and run the test. When you run the test the first time, it will fail but place a new test image into the build tree. Move that image into the appropriate place in the source tree, then configure and run again and the test should then pass. In summary:
 
Write some new test in:
$ $VTKSRC/some/module /Testing/whateverlang/sometest.langext
 
Tell cmake about it by editing:
$ $VTKSRC/some/module/Testing/whateverlang/CMakeLists.txt
 
Configure and make, cmake will warn at some point that there is no such regression test file.
 
Run the test, it will fail and dump the new image to
$ $VTKBUILD/Testing/Temporary
 
Copy that dat file into the source tree
$ cp $VTKBUILD/Testing/Temporary/sometest.png $VTKSRC/some/module/Testing/Data/Baseline
 
Configure and make, cmake will magically move $VTKSRC/some/module/Testing/Data/Baseline/sometest.png into a hidden place in the guts of your source tree AND create and place a new file containing the md5 of the original into $VTKSRC/some/module/Testing/Data/Baseline/sometest.png.md5
 
run ctest again, it should pass this time
 
commit the new test and data files
$ git add $VTKSRC/some/module/Testing/whateverlang/sometest.langext
$ git add $VTKSRC/some/module/Testing/whateverlang/CMakeLists.txt
$ git add $VTKSRC/some/module/Testing/Data/Baseline/sometest.png.md5
$ git commit
 
push it to gerrit for review (more magic makes sure the original file gets uploaded where it needs to)
$ git gerrit-push
 
== Publishing Data for an External Branch ==
 
The above [[#Workflow|workflow]] works well for developers working on a single machine to contribute changes directly to upstream VTK.
When working in an external branch of VTK, perhaps during a long-term topic development effort, data objects need to be published separately.
 
The workflow for adding data to an external branch of VTK is the same as the above through the [[#Commit|commit]] step, but diverges at the [[#Push|push]] step because one will push to a separate repository.
Our ExternalData infrastructure intentionally hides the real data files from Git so only the content links (<code>.md5</code> files) will be pushed.
The real data objects will still be left in the <code>.ExternalData/MD5</code> directory at the top of the VTK source tree by the [[#pre-commit|pre-commit]] hook.
 
The <code>.ExternalData</code> directory must be published somewhere visible to other machines that want to use the data, such as on a web server.
Once that is done then other machines can be told where to look for the data, e.g.
 
<nowiki>cmake ../VTK "-DExternalData_URL_TEMPLATES=https://username.github.io/VTK/ExternalData/%(algo)/%(hash)"</nowiki>
 
In this example we assume the files are published on a Github Pages [https://help.github.com/articles/creating-project-pages-manually gh-pages] branch in <code>username</code>'s fork of VTK.
Within the <code>gh-pages</code> branch the files are placed at <code>ExternalData/MD5/$md5sum</code> where <code>$md5sum</code> is the MD5 hash of the content
(these are the same names they have in the <code>.ExternalData</code> directory in the original source tree).

Latest revision as of 20:41, 16 March 2015

The instructions previously available on this page have been superseded. See here.