[Paraview-developers] Fast loading of large number of vtk files on disk into paraview

[Hillier, Michael (NRCan/RNCan)]

Hello,

I am trying to improve paraview loading times of many vtk files (on disk). I wrote a plugin that connects to a database (locally - for now) and extracts a potentially long list (100s - 1000s) of files locations which I want to load into paraview fast. Most of the vtk files are only 1 kb, and feel it should be much much faster. Am I doing something wrong in my code? :

void pqDatabaseConnection::setupPipeline( const QStringList& file_list )
{
       pqApplicationCore *core = pqApplicationCore::instance();
       pqObjectBuilder *builder = core->getObjectBuilder();
       pqUndoStack *stack = core->getUndoStack();
       pqDisplayPolicy *displayPolicy = core->getDisplayPolicy();

       pqView *view = pqActiveObjects::instance().activeView();

       int nfiles = (int)file_list.size(); // (e.g. file_list[0] = "C:/file1.vtk", file_list[1] = "C:/file2.vtk", ...)
       for (int j = 0; j < nfiles; j++ ){
              QStringList filej;
              filej.push_back(file_list[j]);
              pqPipelineSource* reader = builder->createReader("sources","LegacyVTKFileReader",filej,this->Server); // <-- this function is culprit
              vtkSMProxy *readerProxy = reader->getProxy();
              readerProxy->UpdateVTKObjects();
              displayPolicy->setRepresentationVisibility(reader->getOutputPort(0),view,false);
       }
}

When looking at the task manager (this is windows) all my cpu cores (16) are barely doing anything; 1 core is doing some work (maybe 20% utilized). Can you parallelize the above? Is the issue with the storage (legacy .vtk asci); is there a more appropriate format or method at reading the large amount of files?

Thanks !

Michael Hillier
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/paraview-developers/attachments/20151112/6e0de262/attachment.html>