[vtk-developers] CSV file reader/writer improvement plans

[Andras Lasso]

Hi all,

What do people usually use for reading/writing CSV files?

vtkDelimitedTextReader and vtkDelimitedTextWriter work for very specific cases, but have many important limitations:

Reading:

  *   No way to specify column types. There is some heuristics that can sometimes guess numeric column types but it is not usable in general (for example, a numeric column may be empty or a double column may happen to contain only integer values in a specific file).
  *   All rows must contain exactly the same number of columns.
  *   Columns that don't have names cannot be read.
  *   If a field value contains field separator or string separator character then the file is parsed incorrectly.

Writing:

  *   Cannot specify number of digits for writing floating-point numbers (currently something like 6 digits is hardcoded).
  *   Cannot write field values that contain string separator characters (no escaping of " by "" is performed)
  *   If field value may contain field separator character then all values must be enforced to written with string separators, which makes the file very hard to read and edit (normally string separators are only added when needed)

Questions to these answers would help us in planning/deciding if we implement solution for these by improving existing VTK classes, or just in our application:

  *   Is there any plan (or work in progress) to address these limitations?
  *   If we implemented these features, would they be welcome in VTK?
  *   Would storing metadata (column type, format specifier, default value, etc.) in a schema .csv file next to the data file (such as this: https://github.com/Slicer/Slicer/blob/master/Libs/MRML/Core/Testing/TestData/table.schema.csv) would be considered a good solution? Are there any other standards/best practices for storing csv schema?
  *   Would anyone else need these features and could contribute some time for development or writing tests?

Andras

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/vtk-developers/attachments/20170723/b4325287/attachment.html>