Batchmake tutorial: Difference between revisions

From KitwarePublic
Jump to navigationJump to search
 
(14 intermediate revisions by 3 users not shown)
Line 1: Line 1:
[[BatchMake | < Main BatchMake]]
= Tutorial: Getting Started with BatchMake =
= Tutorial: Getting Started with BatchMake =


==''' Introduction '''==
==''' Introduction '''==
This section explains the purpose of the BatchMake tool, and describes the screen layout. This section is organized as a series of questions and answers about the general capabilities and structure of the BatchMake tool.
This section explains the purpose of the BatchMake tool, and describes the screen layout. This section is organized as a series of questions and answers about the general capabilities and structure of the BatchMake tool.
=== How Do I Get and Run MatchBake Under Microsoft Windows? ===
On the Microsoft Windows platform, BatchMake can be easily installed like any other program, and launched from the Microsoft Windows Start menu.
The BatchMake sources (or the self-installer) are available for download at http://batchmake.org/download.htm
=== What Exactly Does BatchMaker Do? ===
BatchMake is a cross platform tool for batch processing of large amount of data.
For more informations about it, you can visit the BatchMake website at http://batchmake.org/index.htm
=== What Additional Documentation is Available? ===
Other documentation is available at http://batchmake.org/documentation.htm


==''' Running an application with BatchMake '''==
==''' Running an application with BatchMake '''==
For begining, we ar going to do a very simple example. It consists to list all files of a directory.
To begin, we are going to do a very simple example. It consists of listing all the files in a directory.
First, you have to indicate what is the directory that you want to list.
First, you have to indicate the directory that you want to list.
You can do it with the cammand 'Set'. The first argument is the name of the variable,and the second is the path of your directory.
You do this with the command 'Set'. The first argument is the name of the variable, and the second is the path of your directory.
  Set(directory 'C:\Matthieu\BatchMake\Examples')
  Set(directory 'C:\Matthieu\BatchMake\Examples')
After that, BatchMake knows your directory, and so, it can list all files.
After that, BatchMake knows your directory, and so, it can list all files.
  ListFileInDir(Files ${directory})
  ListFileInDir(Files ${directory})
The last thing to do, is to print the names of the files. It is easy with the commad 'echo'
The last thing to do is print the names of the files. It is easy with the commad 'echo'
You can print the list which is "direcory", or you can print each file.
You can print the list which is "directory", or you can print each file.
  foreach(file ${Files})
  foreach(file ${Files})
  echo(${file})
  echo(${file})
  endforeach(file)
  endforeach(file)
  echo(${Files})
  echo(${Files})
If you want to print each file, you need to do a loop with the instruction : 'foreach...endeach'
If you want to print each file, you need to create a loop bracketing the loop commands with 'foreach...endeach'


=== Complete Example : Converter ===  
=== Complete Example : Converter ===  
''' Converter.cxx'''  
''' Converter.cxx'''  


It consists to run an application which convert a tif image into an mha one.
This example runs an application which converts a tif image into an mha one.


First you have to use MetaCommad class in order to receive parameters from the commad line.
First you have to use MetaCommad class in order to receive parameters from the commad line.
Line 128: Line 115:
  endforeach(image)
  endforeach(image)


==''' Sending Data to the central DataBase'''==
= Tutorial: Sending Data to the central DataBase =


=== Sending results: ===
== Sending results: ==


Firstly, we are going to send the name of our inputimage and our ouputimage.
Firstly, we are going to send the name of our inputimage and our ouputimage.
Line 194: Line 181:
[[Image:Dashboard02.JPG]]
[[Image:Dashboard02.JPG]]


==''' MetaCommand Documentation '''==
= Tutorial: Using BatchMake inside a C++ application =
 
BatchMake is a standalone application. It can be run as a command line but also has a front-end GUI. But BatchMake is also a collection of C++ classes that can be used directly inside your C++ applications. This tutorial shows how run a script from your application and how to access the values of your script variable.
=== Introduction: ===
You first have to include the BatchMake parser. Make sure you link the BatchMake libraries with your project.
Thanks to MetaCommand you can easily receive parameters in your software from the command line.The purpose of this class is to provide a unified framework for command line parsing.
''' '''
 
  #include "bmScriptParser.h"
For instance, all programs that use MetaCommand are self-described using the command line –vxml:
Then you should instantiate a ScriptParser object
 
  bm::ScriptParser m_Parser;
$ ./AtlasSummation -vxml
Optionally set the wrapped application directory or/and the application path
<option>
  m_Parser.LoadWrappedApplication( "c:/my_apps/" );
<number>0</number>
  m_Parser.SetBatchMakeBinaryPath( "c:/my_apps/" );  
<name>Outputcount</name>
And finally compile and run the script. Execute() returns true if it successfully compiled and run with no errors, false otherwise.
<tag>oc</tag>
  m_Parser.Execute( "my_script.bms" );
<description>Image of count values for each voxel</description>
After the script is executed, you can retrieve the values of your script variables.
<required>0</required>
  std::vector<BMString> value = m_Parser.GetScriptActionManager()->GetVariable( "my_variable" );
<nvalues>1</nvalues>
<field>
<name>filename</name>
<description></description>
<type>string</type>
<value></value>
<external>0</external>
<required>1</required>
 
Moreover, The description of the parameters can be easily retrieve at runtime via
./your_program –v :
 
 
$ ./AtlasSummation -v
Usage : d:\Work\itkUNCApplications-VC++\bin\debug\AtlasSummation.exe
[-oc <filename>  : Image of count values for each voxel]
[-ao  : Adjust the mean origin to the input images]
[-as  : Adjust the mean size to fit the input images]
[-stdev <bool> (true) : Is the sigma image standard deviation?]
[-LT [number] (4) : lowerThreshold : Minimum number of contributing images for pixel to be counted in output]
[-OS <X> <Y> <Z>  : outputImageSize X Y Z : output size (can't use with adjust size)]
[-OSp <X> <Y> <Z>  : output spacing (default- initial image spacing)]
<infile>  : infile filename
<outputmean>  : output mean filename
<outputvar>  : output variance filename
 
All these advantages are provided for you in the MetaCommand class.
 
=== MetaCommand in use: ===
 
'''Definitions:'''
 
Metacommand differentiates the term ‘Option’ and ‘Field’. Basically an option can be placed anywhere in the command line as long as it is defined by a tag. On the other hand, the fields are placed in order and do not require any tag.
 
'''Programmer’ point of view:'''  
 
To use MetaCommand you have to include the file metaCommand.h in your code.
  #include <metaCommand.h>
 
Then you need to declare a MetaCommand object, for example:
MetaCommand command;
 
To add an option (a tag + a field) you have to use 2 methods:
* To define an option:
  SetOption(std::string name, std::string tag, bool required,
          std::string description)
 
There are two more arguments (the type = flag, and the default value=””) but it is not necessary to put them because they are initialized correctly by default. The first argument is the name of the tag, the second is the string to write in your command line to use this option, the third indicates if the tag is required or not, and the last one is a description of the tag.
 
AddOptionField(std::string optionName, std::string Name,
              MetaCommand::TypeEnumType Type, bool required,
              std::string defValue).
 
The first argument indicates at which tag the field belongs giving the name of the tag (see first argument of SetOption), the second argument is the name of the field, the third one is the type of the field (bool, int, float or string), the fourth argument indicates if the field is required or not, and the last one is the default value (note that you have to put a string even if it is your type is int or float). NOTE: you can use several times the method AddOptionFIeld for a same tag.
 
* To define a field:
You can also just add a field (without the tag) thanks to the following method:
 
  AddField(std::string Name, std::string Description,
          MetaCommand::TypeEnumType Type, bool ExternalData).
The first argument is the name of the field, the second his description, the third one his type and the last one indicates if that data is external to your application or not (e.g : a filename is external but the size of the input image is internal).
 
'''Example'''
 
Let’s define a command line like: ./example input.mha –w 2 output.mha
This will be coded as:
MetaCommand command;
command.SetOption("Write","w",false,"writes the current image to the designated file with a type");
  command.AddOptionField("Write","filename",MetaCommand::STRING,true);
command.AddOptionField("Write","Type",MetaCommand::INT,false,”1”); //by default type=1
  command.AddField("infile","infile filename",MetaCommand::STRING,true);
 
Then the user can parse the command line thanks to the method:
Parse(int argc, char* argv[])
if( !command.Parse(argc,argv) )
{
return 1;
}
 
 
Finally to access the different options, one of these 4 methods can be used:
    * GetValueAsString(Option option, std::string fieldName)
    * GetValueAsFloat(Option option, std::string fieldName)
    * GetValueAsInt(Option option, std::string fieldName)
    * GetValueAsBool(Option option, std::string fieldName)
 
To recover the parameters of an option (tag + field), in the first argument of these methods the name of the tag has to be specified and in the second argument the name of the field.
To recover the parameter of a single field you just need to specify the first argument: the name of the field.
 
 
'''Example:''' to access the parameters of the previous example :
std::string imageIn = command.GetValueAsString("infile");
  std::string imageOut = command.GetValueAsString("Write","filename");
int OutputType = command.GetValueAsInt(“Write","Type");
 
 
 
==  dicomPhilips2meta - Conversion tool from dicom Philips to metaImage ==
 
    * Windows: Media:DicomPhilips2meta.zip (compiled using MSVC6 on Win2K)
    * Linux: Media:DicomPhilips2meta.tgz (compiled using gcc 3.2 on RH9)
 
Usage is:
 
dicomPhilips2meta -t <PhilipsDicom> <outputmeta.mha/mhd>
 
=== MetaCommand Output: ===
To use MetaCommand you have to include the file metaCommand.h in your code.
#include <metaCommand.h>


Then you need to declare a MetaOutput object, for example:
= Tutorial: Adding a test with CTest =
MetaCommand output;
First you have to write a test script (.bms). You then need an executable that takes your .bms file and executes it. This is done by bmActionsTest in Testing/Code/bmActionsTest.cxx.<br>
In your testing directory, open your CMakeLists.txt file and add the lines:


And you should be put this code, before the parsing :
''' '''
   output.SetMetaCommand(&command);
SET(BATCHMAKE_TESTS ${CXX_TEST_PATH}/BatchMakeTests)
  BatchMakeStream bmstream(&command);
SET(TEST_DATA ${WhereYourBMSFilesAre} )
ADD_TEST(MyActionsTest   ${BATCHMAKE_TESTS} ${WhereActionTestIs}/ActionsTest ${TEST_DATA}/MyActionsTest.bms)
ADD_TEST(MyActionsTest2  ${BATCHMAKE_TESTS} ${WhereActionTestIs}/ActionsTest ${TEST_DATA}/MyActionsTest2.bms)
...
You can give a look at the Testing/Code/CMakeLists.txt file; some BatchMake tests are there.


Then you could add a field to see a variable in BatchMake.
= Tutorial: Running BatchMake scripts within Slicer =
In that example , x is an int variable, and we need to see its value in BatchMake:
You can run BatchMake scripts in Slicer. The idea is to generate a script with the inputs of the Slicer module, compile the script and run it (see [[Tutorial: Using BatchMake inside a C++ application]]<br>
  int x = 10;
You can take one of the three existing modules in Slicer as example. They are located in Slicer3/Applications/CLI/BatchMakeApplications.<br>
  output.AddFloatField("Result","Computation Result",x); //Result will be the filed of output
<br>
  output.Write();
If you want to use Condor with BatchMake, you have to have condor installed on your machine. You can either create a condor user on your machine, either be a user with condor privileges. In both cases, the user that run Slicer must have the condor privileges. In the first case, supposing that 'condor' is the name of the condor user, launch Slicer by:  
$ su condor -m -c where/slicer/is/Slicer3


Now, you can access to the variable "x" via BatchMake with the command line :
Once Slicer is started, select the BatchMake module from the module list ( in BatchProcessing ) and make sure you check the "condor" checkbox in the Advanced properties.<br>
Run(output ${example})
It is recommended to set DataDirectory(), GridExecutableDirectory() and GridTransferFile() in your script.
'''Set(result example.Result)'''
Transfering the applications might lead to some path and dependency problems. You probably would like to have the same address for your executables: use GridExecutableDirectory( ) and GridTransferFiles() with one of the following values, NONE, INPUT_FILES or OUTPUT_FILES.  
Echo(x = ${result})
You would also probably like to share the directory where all your input data are with all of your condor machines: use DataDirectory( ) where the path can be a network path accessed by all the condor machines.<br>


But before that, you need to add this command line at the beginning of your script :
Note: slicer modules can be run from Slicer but also from the command line.<br>
  SetAppOption(example.GenerateXMLMetaOutput 1)
Example of EMSegmenter launched from the command line:
  ./EMSegmentBatchMakeModule --mrmlSceneFileName /mnt/midas/batchmake_data/images/EMSegmenterInputs/VolumeData_Input/EMSeg-Brain-MRT1T2.mrml /mnt/midas/batchmake_data/images/EMSegmenterInputs/VolumeData_Input --dirmask Target_NewPatient* --t1m *1*.nhdr --t2m *2*.nhdr /mnt/midas/batchmake_data/images/EMSegmenterOutputs --condor

Latest revision as of 11:05, 19 October 2012

< Main BatchMake

Tutorial: Getting Started with BatchMake

Introduction

This section explains the purpose of the BatchMake tool, and describes the screen layout. This section is organized as a series of questions and answers about the general capabilities and structure of the BatchMake tool.

Running an application with BatchMake

To begin, we are going to do a very simple example. It consists of listing all the files in a directory. First, you have to indicate the directory that you want to list. You do this with the command 'Set'. The first argument is the name of the variable, and the second is the path of your directory.

Set(directory 'C:\Matthieu\BatchMake\Examples')

After that, BatchMake knows your directory, and so, it can list all files.

ListFileInDir(Files ${directory})

The last thing to do is print the names of the files. It is easy with the commad 'echo' You can print the list which is "directory", or you can print each file.

foreach(file ${Files})
echo(${file})
endforeach(file)
echo(${Files})

If you want to print each file, you need to create a loop bracketing the loop commands with 'foreach...endeach'

Complete Example : Converter

Converter.cxx

This example runs an application which converts a tif image into an mha one.

First you have to use MetaCommad class in order to receive parameters from the commad line. If you don't know how to use MetaCommand, you can find some help at the end of this tutorial.

We have this code :

MetaCommand command;
command.SetName("Converter");
command.SetVersion("1.0");
command.SetAuthor("Kitware Inc");
command.SetDescription("Convert tif->mha");
command.AddField("InputImage","InputImage",MetaCommand::STRING);
command.AddField("OutputImage","OutputImage",MetaCommand::STRING);
if(!command.Parse(argc,argv))
{
return 1;
}

std::string InputImage = command.GetValueAsString("InputImage"); 
std::string OutputImage = command.GetValueAsString("OutputImage"); 

reader->SetFileName(InputImage);
writer->SetFileName(OutputImage);

writer->SetInput(reader->GetOutput());
writer->UseCompressionOn();
writer->Update();

return 1;
}

At the beginning of our code, we fill insome fields like the name, the version etc... The most important fields to add, are the input image field and the output image one. The first parameter is the name of the field, the second its "tag", and the last one, its type.

Therefore, our application have one input (InputImage), and one output (OutputImage) which will be the two arguments of the program.

Now, we can run our program with BatchMake.


Converter.bms

Before beginning to write your BatchMake code, we need to specify a new application in the wraper

Wrapper3.JPG

Create a new application with the button "New" :

Wrapper.JPG


Use the button "Browse and Run" to select your application (Converter.exe), and you will see some information about it, like metacommand filelds.

Wrapper1.JPG

At the bottom of the image, you can see the Command line of the application. The two <string> indicate you have to specify two arguments to run Converter => the input image and the ouput one. Now you can click "OK"

Wrapper2.JPG

"Converter" application is now known by BatchMake.


Here is the code of our BatchMake file :

SetApp(converter @Converter)       #"converter" is the variable which defines the application

Wrapper4.JPG

When you type @, BatchMake shows you all the applications it knows.

Set(outputdir 'C:/Example/TifImages')       #Here, we define the output directory where new images will be stored
MakeDirectory(${outputdir})                 #If the output directory doesn't exist, it will be created
Set(inputdir 'C:/Example/MhaImages')      #Here, we define the intput directory where tiff images are
ListFileInDir(files ${inputdir})          #Then we list all files which are in that directory
foreach(i $(files))        #For each file of the directory inputdir

Set(inputimage ${inputdir}/${i})                      #We create a new variable which contains the inputimage file.
SetAppOption(converter.InputImage ${inputimage})      #We sepcify what is the input image for the first field

GetFilename(j ${i} NAME_WITHOUT_EXTENSION)
Set(outputimage ${outputdir}/${j}.mha)                 #We create a new variable which contains the outputimage file.
SetAppOption(converter.OutputImage ${outputimage})     #And the output image for the second field of Metacommand

echo(${converter})          #We can see the line command with echo
Run(output ${converter})    #Running the application...

endforeach(image)

Tutorial: Sending Data to the central DataBase

Sending results:

Firstly, we are going to send the name of our inputimage and our ouputimage.

To send data to the central DataBase of BatchMake, you have to register you on the website :

http://insight-journal.org/batchmake/index.php


When it's done, log you on the DataBase, go to "My Profile", and click on the button "Generate Key" and remember this one.

Now, you can add this code at the begining of your script :

DashboardHost(http://www.insight-journal.org/batchmake)
DashboardUser('Your_FirstName Your_LastName')
DashboardKey('Your_Key')
#The name of your project must be created by a super-admin of BatchMake : Ask him if you want to create a new project
CreateExperiment(exp 'Your project name' 'Title of your new experiment' 'This is a description for this experiment')
# We add a method 
CreateMethod(ConvertMeth exp 'Converter' 'Convert a mha image into a mha compress')


The following cammand should be add after that the variables are seted :

# We add two inputs that will be displayed in the DataBase
AddMethodInput(InputImage ConvertMeth 'InputImage Name') #InputImage is the same variable that we used before.
AddMethodInput(InputImage ConvertMeth 'OutputImage Name') #OutputImage is the same variable that we used before.
# All the path will be displayed, but if you prefre display only the imagename => replace InputImage and OutputImage by i.

The entries of the methods are now created, you can send your new method. with the command :

DashboardSend(ConvertMeth)

Dash01.jpg

The Dashboard has another option that is very interisting : You can display an image in the DataBase.

The only condition is that your image is a png.

But maybe, you would like to display a slice from a 3D image (like a mha) : you can do this too with BatchMake. You just have to write the following command :

 Set(InputLiver 'C:/Matthieu/Tests/Liver.mha')
 #We take the first slice after the middle of the image (2 = axial axe)
 ExtractSlice('${InputLiver}' '${InputLiver}1.png' 2 1 FROM_MIDDLE) 

Liver.bms

The following lines will display diferent slices of the liver image in the database.

Set(InputLiver 'C:/Matthieu/Tests/Liver.mha')
sequence(seq 0 20 5)
foreach(x ${seq})
ExtractSlice('${InputLiver}' '${InputLiver}${x}.png' 2 ${x} FROM_MIDDLE) 
SetIdealOutput(slice '${InputLiver}${x}.png')
AddMethodInput(x ConvertMeth 'Slice #')
AddMethodIdealOutput(slice ConvertMeth 'Slice' png)
DashboardSend(ConvertMeth)
endforeach()

Dashboard02.JPG

Tutorial: Using BatchMake inside a C++ application

BatchMake is a standalone application. It can be run as a command line but also has a front-end GUI. But BatchMake is also a collection of C++ classes that can be used directly inside your C++ applications. This tutorial shows how run a script from your application and how to access the values of your script variable. You first have to include the BatchMake parser. Make sure you link the BatchMake libraries with your project.

#include "bmScriptParser.h"

Then you should instantiate a ScriptParser object

bm::ScriptParser m_Parser;

Optionally set the wrapped application directory or/and the application path

m_Parser.LoadWrappedApplication( "c:/my_apps/" );
m_Parser.SetBatchMakeBinaryPath( "c:/my_apps/" ); 

And finally compile and run the script. Execute() returns true if it successfully compiled and run with no errors, false otherwise.

m_Parser.Execute( "my_script.bms" );

After the script is executed, you can retrieve the values of your script variables.

std::vector<BMString> value = m_Parser.GetScriptActionManager()->GetVariable( "my_variable" );

Tutorial: Adding a test with CTest

First you have to write a test script (.bms). You then need an executable that takes your .bms file and executes it. This is done by bmActionsTest in Testing/Code/bmActionsTest.cxx.
In your testing directory, open your CMakeLists.txt file and add the lines:

SET(BATCHMAKE_TESTS ${CXX_TEST_PATH}/BatchMakeTests)
SET(TEST_DATA ${WhereYourBMSFilesAre} )
ADD_TEST(MyActionsTest   ${BATCHMAKE_TESTS} ${WhereActionTestIs}/ActionsTest ${TEST_DATA}/MyActionsTest.bms)
ADD_TEST(MyActionsTest2  ${BATCHMAKE_TESTS} ${WhereActionTestIs}/ActionsTest ${TEST_DATA}/MyActionsTest2.bms)
...

You can give a look at the Testing/Code/CMakeLists.txt file; some BatchMake tests are there.

Tutorial: Running BatchMake scripts within Slicer

You can run BatchMake scripts in Slicer. The idea is to generate a script with the inputs of the Slicer module, compile the script and run it (see Tutorial: Using BatchMake inside a C++ application
You can take one of the three existing modules in Slicer as example. They are located in Slicer3/Applications/CLI/BatchMakeApplications.

If you want to use Condor with BatchMake, you have to have condor installed on your machine. You can either create a condor user on your machine, either be a user with condor privileges. In both cases, the user that run Slicer must have the condor privileges. In the first case, supposing that 'condor' is the name of the condor user, launch Slicer by:

$ su condor -m -c where/slicer/is/Slicer3

Once Slicer is started, select the BatchMake module from the module list ( in BatchProcessing ) and make sure you check the "condor" checkbox in the Advanced properties.
It is recommended to set DataDirectory(), GridExecutableDirectory() and GridTransferFile() in your script. Transfering the applications might lead to some path and dependency problems. You probably would like to have the same address for your executables: use GridExecutableDirectory( ) and GridTransferFiles() with one of the following values, NONE, INPUT_FILES or OUTPUT_FILES. You would also probably like to share the directory where all your input data are with all of your condor machines: use DataDirectory( ) where the path can be a network path accessed by all the condor machines.

Note: slicer modules can be run from Slicer but also from the command line.
Example of EMSegmenter launched from the command line:

./EMSegmentBatchMakeModule --mrmlSceneFileName /mnt/midas/batchmake_data/images/EMSegmenterInputs/VolumeData_Input/EMSeg-Brain-MRT1T2.mrml /mnt/midas/batchmake_data/images/EMSegmenterInputs/VolumeData_Input --dirmask Target_NewPatient* --t1m *1*.nhdr --t2m *2*.nhdr /mnt/midas/batchmake_data/images/EMSegmenterOutputs --condor