<div dir="ltr">We have used the <span style="font-size:12.8px">vtkDelimitedTextReader as the first stage for a CSV data import tool for a government customer, and they are open to moving the tool into an open-source repository. Unfortunately, the project is stalled right now as funding is considered. </span><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Some thoughts inlne:</span></div><div><span style="font-size:12.8px">- The tool is a 'cleaner', so it's designed to take poorly-formed CSV files and let the user make decisions about fixing the data, although there is a non-interactive path that simply drops rows that have problem data. The final result is a VTK table written to an HDF5 file, so this project doesn't use the </span>vtkDelimitedTextWriter. It has pretty good testing written already.</div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jul 23, 2017 at 4:29 PM, Andras Lasso <span dir="ltr"><<a href="mailto:lasso@queensu.ca" target="_blank">lasso@queensu.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-8606126668262057745WordSection1">
<p class="MsoNormal">Hi all,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">What do people usually use for reading/writing CSV files?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">vtkDelimitedTextReader and vtkDelimitedTextWriter work for very specific cases, but have many important limitations:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Reading:<u></u><u></u></p>
<ul style="margin-top:0cm" type="disc">
<li class="gmail-m_-8606126668262057745MsoListParagraph" style="margin-left:0cm">No way to specify column types. There is some heuristics that can sometimes guess numeric column types but it is not usable in general (for example, a numeric column may be empty or
a double column may happen to contain only integer values in a specific file).</li></ul></div></div></blockquote><div><span style="font-size:12.8px">- We read everything as strings and then use vtkVariant to test if it can be converted to int/double. We do use a heuristic to choose column data types.</span></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_-8606126668262057745WordSection1"><ul style="margin-top:0cm" type="disc"><li class="gmail-m_-8606126668262057745MsoListParagraph" style="margin-left:0cm"><u></u><u></u></li><li class="gmail-m_-8606126668262057745MsoListParagraph" style="margin-left:0cm">All rows must contain exactly the same number of columns.</li></ul></div></div></blockquote><div>If a row has too few entries, it is filled with empty values, at least for string input. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_-8606126668262057745WordSection1"><ul style="margin-top:0cm" type="disc"><li class="gmail-m_-8606126668262057745MsoListParagraph" style="margin-left:0cm"><u></u><u></u></li><li class="gmail-m_-8606126668262057745MsoListParagraph" style="margin-left:0cm">Columns that don’t have names cannot be read.</li></ul></div></div></blockquote><div><span style="font-size:12.8px">- I believe </span><span style="font-size:12.8px">vtkDelimitedTextReader</span><span style="font-size:12.8px"> will read columns without a name, but they can't be used in calculations or exported until they have a name - assigning a default name in your code might be enough.</span><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_-8606126668262057745WordSection1"><ul style="margin-top:0cm" type="disc"><li class="gmail-m_-8606126668262057745MsoListParagraph" style="margin-left:0cm"><u></u><u></u></li><li class="gmail-m_-8606126668262057745MsoListParagraph" style="margin-left:0cm">If a field value contains field separator or string separator character then the file is parsed incorrectly.</li></ul></div></div></blockquote><div><span style="font-size:12.8px">- The purpose of the string separator is to allow the field separator to appear inside a field. If you've found this not to work, it's a bug that can be submitted to VTK!</span><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_-8606126668262057745WordSection1"><ul style="margin-top:0cm" type="disc"><li class="gmail-m_-8606126668262057745MsoListParagraph" style="margin-left:0cm"><u></u><u></u></li></ul>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Writing:<u></u><u></u></p>
<ul style="margin-top:0cm" type="disc">
<li class="gmail-m_-8606126668262057745MsoListParagraph" style="margin-left:0cm">Cannot specify number of digits for writing floating-point numbers (currently something like 6 digits is hardcoded).<u></u><u></u></li><li class="gmail-m_-8606126668262057745MsoListParagraph" style="margin-left:0cm">Cannot write field values that contain string separator characters (no escaping of “ by “” is performed)<u></u><u></u></li><li class="gmail-m_-8606126668262057745MsoListParagraph" style="margin-left:0cm">If field value may contain field separator character then all values must be enforced to written with string separators, which makes the file very hard to read and edit (normally string
separators are only added when needed)<u></u><u></u></li></ul>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Questions to these answers would help us in planning/deciding if we implement solution for these by improving existing VTK classes, or just in our application:<u></u><u></u></p>
<ul style="margin-top:0cm" type="disc">
<li class="gmail-m_-8606126668262057745MsoListParagraph" style="margin-left:0cm">Is there any plan (or work in progress) to address these limitations?<u></u><u></u></li><li class="gmail-m_-8606126668262057745MsoListParagraph" style="margin-left:0cm">If we implemented these features, would they be welcome in VTK?</li></ul></div></div></blockquote><div>- Yes, I think so! </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_-8606126668262057745WordSection1"><ul style="margin-top:0cm" type="disc"><li class="gmail-m_-8606126668262057745MsoListParagraph" style="margin-left:0cm"><u></u><u></u></li><li class="gmail-m_-8606126668262057745MsoListParagraph" style="margin-left:0cm">Would storing metadata (column type, format specifier, default value, etc.) in a schema .csv file next to the data file (such as this:
<a href="https://github.com/Slicer/Slicer/blob/master/Libs/MRML/Core/Testing/TestData/table.schema.csv" target="_blank">
https://github.com/Slicer/<wbr>Slicer/blob/master/Libs/MRML/<wbr>Core/Testing/TestData/table.<wbr>schema.csv</a>) would be considered a good solution? Are there any other standards/best practices for storing csv schema?<u></u><u></u></li><li class="gmail-m_-8606126668262057745MsoListParagraph" style="margin-left:0cm">Would anyone else need these features and could contribute some time for development or writing tests?<span class="gmail-HOEnZb"><font color="#888888"><u></u><u></u></font></span></li></ul><span class="gmail-HOEnZb"><font color="#888888">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Andras<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</font></span></div>
</div>
<br>______________________________<wbr>_________________<br>
Powered by <a href="http://www.kitware.com" rel="noreferrer" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" rel="noreferrer" target="_blank">http://www.kitware.com/<wbr>opensource/opensource.html</a><br>
<br>
Search the list archives at: <a href="http://markmail.org/search/?q=vtk-developers" rel="noreferrer" target="_blank">http://markmail.org/search/?q=<wbr>vtk-developers</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/vtk-developers" rel="noreferrer" target="_blank">http://public.kitware.com/<wbr>mailman/listinfo/vtk-<wbr>developers</a><br>
<br>
<br></blockquote></div><br></div></div>