[ITK-dev] TIFFImageIO refactoring, please test

Fri Nov 14 13:41:48 EST 2014

Bill,

There are quite a number of possible compression types which can be used in a tiff file[1]. Some of these are enabled/disable at compilation time based in what external libraries are used. These are defined with preprocessor directives in the tiff configuration file[2]. Ideally CanRead should some how query libtiff about support for this... I am looking for some type of documentation now...

Brad

[1] http://www.awaresystems.be/imaging/tiff/tifftags/compression.html
[2] https://github.com/InsightSoftwareConsortium/ITK/blob/master/Modules/ThirdParty/TIFF/src/itktiff/tif_config.h.in#L218

On Nov 14, 2014, at 12:44 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:

> I added that compression option to the possibilities
> 
> 
> On Fri, Nov 14, 2014 at 12:08 PM, Bradley Lowekamp
> <blowekamp at mail.nih.gov> wrote:
>> Bill,
>> 
>> Your images have an odd compression:
>> 
>> $ tiffinfo
>> /scratch/blowekamp/build/VTK/ExternalData/Testing/Data/libtiff/tiled_64x64_tiff_example.tif
>> TIFF Directory at offset 0x81f6 (33270)
>>  Image Width: 256 Image Length: 256
>>  Tile Width: 64 Tile Length: 64
>>  Bits/Sample: 8
>>  Compression Scheme: AdobeDeflate
>>  Photometric Interpretation: min-is-black
>>  FillOrder: msb-to-lsb
>>  Orientation: row 0 top, col 0 lhs
>>  Samples/Pixel: 1
>>  Rows/Strip: 256
>>  Planar Configuration: single image plane
>>  Page Number: 0-1
>>  DocumentName: foo.tif
>>  White Point: 0.3127-0.329
>>  PrimaryChromaticities:
>> 0.640000,0.330000,0.300000,0.600000,0.150000,0.060000
>>  Predictor: horizontal differencing 2 (0x2)
>> 
>> This "AdobeDeflate" is not supported by the
>> itk::TIFFImageIOInternal::CanRead method. This is what is causing the usage
>> of TIFFRGBARead and not the fact that it's a tiled image.
>> 
>> How is VTK handling this odd compression?
>> 
>> Brad
>> 
>> On Nov 14, 2014, at 11:09 AM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>> 
>> Yes, from the vtk tests. I created them with imagemagik
>> 
>> 
>> On Fri, Nov 14, 2014 at 11:06 AM, Bradley Lowekamp
>> <blowekamp at mail.nih.gov> wrote:
>> 
>> Actually further reading the TIFFReadScanline documentation says that should
>> not work for tiled images. So I guess it must be using the RGBA read method,
>> but I am not sure how that is happening either.
>> 
>> Are you used the tiled tiff images from the VTK tests?
>> 
>> Brad
>> 
>> 
>> On Nov 14, 2014, at 10:46 AM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>> 
>> Meant to say:
>> Wow. The new itk tiff reader read my tiled images.
>> 
>> 
>> On Fri, Nov 14, 2014 at 10:45 AM, Bill Lorensen <bill.lorensen at gmail.com>
>> wrote:
>> 
>> Wow. The new itk tiff reader sad my tiled images.
>> 
>> On Fri, Nov 14, 2014 at 10:25 AM, Bill Lorensen <bill.lorensen at gmail.com>
>> wrote:
>> 
>> There is a TIFFReadTile call in lib tiff. I some tiled images I will
>> try with the updated itk reader. I'll be really syprised if it works
>> since reading tiled images is tricky and took quite a bit of code.
>> 
>> I'll keep you posted.
>> 
>> Bill
>> 
>> On Fri, Nov 14, 2014 at 10:19 AM, Bradley Lowekamp
>> <blowekamp at mail.nih.gov> wrote:
>> 
>> Bill,
>> 
>> There was a tile block of code before, but that was not covered, looked
>> unreachable and was removed.
>> 
>> Currently there are two ways that a tiff page can be read with the
>> TIFFReadRGBAImageOriented, and with the TIFFReadScanline. The RGBA method
>> from libtiff is fairly robust but reads the image into an RGBA buffer so is
>> not memory efficient. Our custom method uses the scanline method [1], which
>> should work with both stripped and tiled pages.
>> 
>> But please test, and add a test? I'm not 100% if there is a test image for
>> tiled images.
>> 
>> Where the difference between stripped and tiled images would really matter
>> is streaming, but that is not there...yet.
>> 
>> Brad
>> 
>> [1] http://www.remotesensing.org/libtiff/libtiff.html#scanlines
>> 
>> 
>> On Nov 14, 2014, at 10:04 AM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>> 
>> Brad,
>> 
>> I recently added tiled image reading to VTK's tiff reader. Do you
>> think that would be useful in IUTK. I don't know how often tiled tiff
>> images are encountered.
>> 
>> BTW I do not mean tiled pyramid tiff images.
>> 
>> Bill
>> 
>> On Fri, Nov 14, 2014 at 9:53 AM, Bradley Lowekamp
>> <blowekamp at mail.nih.gov> wrote:
>> 
>> Greetings,
>> 
>> I have recently refactored ITK's TIFFImageIO to remove a significant amount
>> of dead code, and improve performance up to 3-5X. A number of bug related to
>> reading BigTIFF files have been addressed. Additionally, Tiff tags are now
>> place into the Meta-DataDictionary to allow access to additional TIFF fields
>> such as OME-XML tags ( Originally contributed by Richard Beare
>> http://www.insight-journal.org/browse/publication/728 ).
>> 
>> Significant performance improvements were made to both scalar images and
>> palette images. Handling of certain orientations are now "better" handled,
>> while not changing how the default TOPLEFT orientation is loaded or written.
>> There were a number of cases which appear unreachable and illogical such as
>> support from Zeiss two-componet types, and support for some type of SGI
>> tiled tiff as a 3D image. These were removed.
>> 
>> If you are an active user of TIFF images with ITK, please checkout the
>> latest ITK master and test the refactored TIFFImageIO.
>> 
>> Thanks,
>> Brad
>> 
>> p.s. Here is the git log for the recent changes.
>> 
>> $git log --no-merges 6791b4a3613aade07e3f11f765f1316faf03b113..
>> 
>> commit 46f4125df47aeff9c42c9f2b3699c06e67ffaca9
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Fri Oct 31 14:25:13 2014 -0400
>> 
>> ENH: adding TIFFImageIO test for RGB palette images
>> 
>> Change-Id: I320acc8beac8225616451d11b291b1d7e12a17d0
>> 
>> commit 9adc54c7f53b4da84a41f500d561ae494cb85158
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Fri Oct 24 13:42:02 2014 -0400
>> 
>> BUG: Adding missing parentheses around boolean expression
>> 
>> Change-Id: I669d160ff3ba830f3c33ac87de434db41472eb79
>> 
>> commit 40afd95161727be3274f8094ce19fe45e2c3aa50
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Wed Oct 22 15:05:39 2014 -0400
>> 
>> PERF: Refactor color table lookup
>> 
>> Change to use a index lookup for the color and gray scale
>> palettes. The remove excessive exception checking done on a per pixel
>> basis with the GetColor method. Additionally the InitializeColor
>> method was change do initialize the color palette variables.
>> 
>> An exception no longer occurs when an index exceeds the number of
>> entries in a palette. Instead a module operator is used to prevent
>> out of bounds memory access.
>> 
>> Change-Id: Ia62c65cd8b1be0d66c488540709a8fb961190cc9
>> 
>> commit d98cbeb7206f6007f7d0c25fd4a4becad8973053
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Fri Oct 3 09:01:28 2014 -0400
>> 
>> ENH: Refactor per pixel conversion function to per scan-line method
>> 
>> This patch changes the implementation of conversion from the TIFF
>> scanline buffer from a function call on each pixel to a function call
>> per scanline. The enable efficient copying of similar scanline and
>> better reuse of variables for palette look ups.
>> 
>> This can increase the performance of reading grayscale images by up to
>> 3X for cached files.
>> 
>> Change-Id: I24bb744fda35629f1c1ed7154e26e5708e912059
>> 
>> commit 8e488f40e8cb7defa046700798b77e3944ac33c9
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Tue Sep 30 08:56:58 2014 -0400
>> 
>> ENH: RGBA read images should stay unchanged.
>> 
>> The image that are read with the TIFFReadRGBAImage should not have
>> their pixel types converted. This libtiff method always read images
>> into an ABGR uint8 format. When the tiff file is read by this method
>> it should report that uint8 and RGBA is the component an pixel type
>> for the image. This allows the standard ITK buffer conversion to be
>> performed.
>> 
>> This change remove extra cases and template instantiations.
>> 
>> Additionally the RGBAImageToBuffer method was flipping the image along
>> the y-axis, this has been more explicitly done with the call to
>> TIFFReadRGBAImageOriented.
>> 
>> Change-Id: I022b12921d406edb09730f905213b0db811f1bd4
>> 
>> commit e5834d5b00e4e3fab3255d88e7d8fd917ee722e9
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Fri Sep 26 14:04:09 2014 -0400
>> 
>> ENH: Remove support for TIFF tile as 3D and dead code
>> 
>> This TIFF file format is a file which contains a list of directories,
>> if there are no directories in the file it's not a valid tiff. Code
>> which supported 3D tiles was conditioned on a file with no directories
>> and no data. Additional code was removed related to a private SGI tag
>> TIFFTAG_TILEDEPTH, ImageJ specific tag also pre-conditioned on this
>> invalid case.
>> 
>> Change-Id: Id4ef171c5dbe0c797be206e0059c7488719ef6e6
>> 
>> commit 3ac19a9ffd94e02254900dd0e5e2661c329e3183
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Mon Oct 6 10:01:50 2014 -0400
>> 
>> BUG: Address Coverity warning about null pointer dereferences
>> 
>> The patch addressed the following Coverity warnings:
>> 
>> ** CID 1243373:  Explicit null dereferenced  (FORWARD_NULL)
>> itkTIFFImageIO.cxx:
>> 1192 in itk::TIFFImageIO::ReadTIFFTags()()
>> 
>> ** CID 1243372:  Dereference null return value  (NULL_RETURNS)
>> itkTIFFImageIO.cxx:
>> 1096 in itk::TIFFImageIO::ReadTIFFTags()()
>> 
>> ** CID 1243371:  Dereference before null check  (REVERSE_INULL)
>> itkTIFFImageIO.cxx:
>> 1100 in itk::TIFFImageIO::ReadTIFFTags()()
>> Last login: Fri Oct  3 19:47:29 on ttys004
>> 
>> Change-Id: I02c8435628c860ff03889b9b57e9dd1543bf82da
>> 
>> commit 52ce992f3a57ef9f187f9c4b4ebd7ce20db365bd
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Fri Sep 26 11:04:38 2014 -0400
>> 
>> ENH: Reduce code duplicate in TIFFImageIO::ReadCurrentPage
>> 
>> Refactoring of the pixelOffset to include the number of components
>> remove the need for a separate block for RGB multi-component files.
>> 
>> commit 97b33787ec17b76a61fdb43b5f1e1ea708bbac2c
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Wed Sep 24 15:00:54 2014 -0400
>> 
>> ENH: add arbitrary TIFF TAGs to meta-data dictionary
>> 
>> A more general approach to adding custom tiff tags to the meta-data
>> dictionary has been used based on the implementation of libtiff's
>> PrintDirectory method and the libtiff "Defining New TIFF Tags"
>> documentation.
>> 
>> Additionally a new test image was added generated in Photoshop, which
>> has a novel pixel layout, and also contains additional tag types such
>> as byte, and size specified ascii.
>> 
>> Change-Id: Idb2e669e540cfbe27a932444011f4b9896ca403d
>> 
>> commit 795c71abb8b5c3a09d461ec7b34eefbfdcb460e9
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Mon Sep 22 15:08:36 2014 -0400
>> 
>> BUG: Use array delete operator for array new allocations
>> 
>> Change-Id: Ic9bc3c3792083a1a1b3fd842486a7436a4086150
>> 
>> commit 83e8b6e699b3d9e2da8533dd1ea36dc0692bc9eb
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Tue Sep 23 11:14:15 2014 -0400
>> 
>> STYLE: Fix minor kwstyle defects in test and test results
>> 
>> Change-Id: I4b2641d31acb5c764de4041bb7d21c08b3c99846
>> 
>> commit 869f3275120646829b27d5cd75c23800d9df01cd
>> Author: Richard Beare <Richard.Beare at ieee.org>
>> Date:   Tue Sep 23 11:11:34 2014 -0400
>> 
>> ENH: Include TIFF tags in the MetaDataDictionary
>> 
>> Add contribution from the Insight Journal:
>> http://hdl.handle.net/10380/3170
>> 
>> Change-Id: I20b02394d92c421bad75c5d0af496a6e80089ac0
>> 
>> commit c7a76c9acb319862647a651a2060dd3fb16560d7
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Mon Sep 22 14:46:44 2014 -0400
>> 
>> BUG: Remove dead separated plannar code, add test
>> 
>> The PLANARCONFIG_SEPARATE block in TIFFImageIO::ReadGenericImage, was
>> preceded by a unreachable due to a conditional and exception
>> check. Support for this configuration remains in the TIFFReadRGBAImage
>> method. A test for the configuration has been added.
>> 
>> A new input image is added from the from the libTiff test images:
>> 
>> PlanarConfiguration = 2 (separated samples)
>> -------------------------------------------
>> oxford.tif      601x81 8-bit RGB (lzw) screendump off oxford
>> 
>> Change-Id: Ibb874f80809b70b1aee7a2c9ff1fc8544ed56c04
>> 
>> commit 1b57d75964496dd2cc9e050a4a4425f1163d4b22
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Mon Sep 22 13:32:23 2014 -0400
>> 
>> BUG: Fix right oriented tiff images
>> 
>> The TIFFImageIO::ReadGenericImage method correctly handled top-left
>> oriented images, and bottom-right image, however images oriented with
>> right were non-correctly handled. These cases are now handled
>> correctly by the TIFFRGBAImage generic path.
>> 
>> New test images of the cthead1 image were added which have been
>> flipped the to the different orientation, and the matching orientation
>> tag specified so that they should all be read as equivalent
>> images. These have been added as a test case.
>> 
>> Change-Id: I058382d6d2e18341ab87edd165490be2187c1d15
>> 
>> commit 6263f4389026f1bd770054e86c428b56629d705b
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Thu Sep 18 16:20:41 2014 -0400
>> 
>> ENH: Refactor duplicated code to read a page
>> 
>> Created a method to read a single page or sub-image in a tiff
>> file. Simular code was both in the ReadVolume method for 3-d and the
>> Read method for 2-d. The code block from ReadVolume was used because
>> it contained more typed cases. There for this patch does improve the
>> 2d reading capabilities for tiffio, so that they are consistent with
>> the 3d.
>> 
>> Change-Id: I0fbfb19f04435a703bb3fa09ea4e79179592e443
>> 
>> commit 233571f1e626917bce872e450907bc8e8521e371
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Thu Sep 18 14:59:58 2014 -0400
>> 
>> ENH: Refactor method to convert RGBA image to output buffer
>> 
>> BUG: Address overflow issue when computing page offset with
>> TIFFReadRGBAImage method.
>> 
>> Extract repeated code to convert from ReadRGBAImage to output image
>> type.
>> 
>> Change-Id: Ia90b5e104f22c77493b8a4cd5bf640fd793b8b4b
>> 
>> commit 1cd59825fa65f1123726a32f67c1a443015b40ec
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Thu Sep 18 11:30:41 2014 -0400
>> 
>> BUG: Remove Zeiss 2-channel support code in TIFFImageIO ( and LSMImageIO
>> )
>> 
>> There are no tests for this branch of code. It only works in 3D under
>> certain cases. There are apparent bugs in the code such as not setting
>> all of the input image, and logically dead branches. Some downloaded
>> test images for similar described format do not load. This may effect
>> files with the extensions tif, tiff, or lsm.
>> 
>> Removal of this code will enable expansion of the current code to more
>> generically support multi-sample per pixel images.
>> 
>> For more robust reading of this type of image the SCIFIO remote
>> module should be used.
>> 
>> Change-Id: Idb98d69538db3bc8520881ea4a5e22476238c41e
>> 
>> commit 690cfdc89343fcc8c7d3cc266d652cfb14de9d41
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Tue Sep 16 15:31:52 2014 -0400
>> 
>> ENH: Refactor GenericReadImage into template function
>> 
>> Reduce code duplication by using template method.
>> 
>> Change-Id: I79411ad6595258d54cbfa5a4152cc55994acd8e0
>> 
>> commit 5f73f24bfb668fa9f33804218c0db05cf5cadb3f
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Tue Sep 16 12:09:59 2014 -0400
>> 
>> ENH: Refactor ReadTwoSamplePerPixelImage into template function
>> 
>> Reduce code duplication by using template method.
>> 
>> Change-Id: I45f417486e60a4d3e67ff31fa967dac069b3ce3e
>> 
>> commit 0b573136d636613217df9683ea846db8039f9312
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Tue Sep 16 11:12:45 2014 -0400
>> 
>> ENH: Extract TiffReaderInternal to separate file
>> 
>> Refactored private class to be in separate private header.
>> 
>> Change-Id: Icb4008c1605a3a0b62c563350cf2e3bbe948ec6d
>> 
>> commit 6791b4a3613aade07e3f11f765f1316faf03b113
>> Author: Bradley Lowekamp <blowekamp at mail.nih.gov>
>> Date:   Mon Sep 15 16:46:26 2014 -0400
>> 
>> BUG: Fix overflows computing size of read tiff image
>> 
>> When multiplying the height and width of an tiff image (32-bits),
>> at least one element should be converted to size_t to help prevent
>> numeric overflow.
>> 
>> This change enables reading of larger RGB and large 2D image, that
>> were previously truncated when reading. This does not address
>> potential buffer overflow or resource limits which could occur.
>> 
>> Change-Id: I192370606677c8dc1a903c67c2a0701026256d5a
>> 
>> commit b6fabfeb2a5b8d339b2d459edd3e3c50232b55ff
>> Author: Matt McCormick <matt.mccormick at kitware.com>
>> Date:   Mon Aug 25 00:35:48 2014 -0400
>> 
>> COMP: Do not use _stat64 with MinGW-32.
>> 
>> Unlike MinGW-w64, MinGW-32 aka MinGW, does not define as much of the API
>> including _stat64.  Do not use use _stat64 in TIFFImageIO when building
>> with
>> MinGW-32.
>> 
>> Change-Id: I8f13a4500cdadf4f4adae7b5cf8893fe8b8cad55
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Powered by www.kitware.com
>> 
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>> 
>> Kitware offers ITK Training Courses, for more information visit:
>> http://kitware.com/products/protraining.php
>> 
>> Please keep messages on-topic and check the ITK FAQ at:
>> http://www.itk.org/Wiki/ITK_FAQ
>> 
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/insight-developers
>> 
>> 
>> 
>> 
>> --
>> Unpaid intern in BillsBasement at noware dot com
>> 
>> 
>> 
>> 
>> 
>> --
>> Unpaid intern in BillsBasement at noware dot com
>> 
>> 
>> 
>> 
>> --
>> Unpaid intern in BillsBasement at noware dot com
>> 
>> 
>> 
>> 
>> --
>> Unpaid intern in BillsBasement at noware dot com
>> 
>> 
>> 
>> 
>> 
>> --
>> Unpaid intern in BillsBasement at noware dot com
>> 
>> 
> 
> 
> 
> -- 
> Unpaid intern in BillsBasement at noware dot com