[Paraview] In-situ file/image output on Titan with 18k cores

[Hong Yi]

Hi Andy,

I am attaching the python coprocessing script that outputs data (rather than images) so that you can easily have an idea on what type of views I am trying to output. Basically, it is an image with vorticity contours overlaid on subchannel wall contours identified by zero velocity so that researchers can see how flow vorticity changes going through subchannel walls. The grid is unstructured grid. In fact, the simulation software is a variant of PHASTA that allows multi-phase turbulence simulation. Let me know if you need more information.

That explains it - The XML writers won't create a directory if it doesn't exists yet in order to write a file. I did not create the directory beforehand, so that must be the reason. I already suspected that is the reason, many thanks for confirming it.

Thanks for the suggestion on reducing the number of processes that are writing. I will investigate this further when I have time. Let me know if you are aware of any example on using these filters to achieve this purpose.

Thanks again for the helpful information and let me know if you have other suggestions.

Best regards,

Hong


From: Andy Bauer [mailto:andy.bauer at kitware.com]
Sent: Tuesday, November 26, 2013 10:57 AM
To: Hong Yi
Cc: paraview at paraview.org
Subject: Re: [Paraview] In-situ file/image output on Titan with 18k cores

Hi Hong,
Can you describe the type of view you're trying to output? If I remember correctly it was volume rendering of an image data. Also, some details on the grid would be helpful.
As for the parallel XML polydata writer, we were talking about changing it so that it would only write out files for polydata with points and cells in them which may help alleviate this issue some. I'm not sure what's going on with the file -- you may need to share more information to help diagnose the problem. Is it a writing problem during your run or a reading problem during post-processing? The XML writers won't create a directory if it doesn't exists yet in order to write a file.

As for reducing the number of processes that are writing, you could use the vtkAllToNRedistributePolyData filter to reduce the number of processes that contain any points and cells and then create another vtkMPIController that is used by the parallel XML writer to make sure only the proper processes write out any data. This would take quite a bit of custom coding though.

Andy

On Tue, Nov 26, 2013 at 10:31 AM, Hong Yi <hongyi at renci.org<mailto:hongyi at renci.org>> wrote:
I have done several simulation runs linked with ParaView Catalyst for in-situ visualization on Titan with 18k cores and have the following observations/questions hoping to seek input from this list.


1.       It appears IceT-based image compositing for 18k cores takes such a long time that it becomes unpractical to output images in-situ. Specifically, in our case, it takes about 14 minutes for coprocessing for one time point that output a composited image while simulation alone for one time point only takes about 7 seconds. I have also done a simulation run with in-situ visualization on Titan with 64 cores on a much lower resolution mesh (10 million element mesh as opposed on 167 million element mesh for 18k core run), in which case coprocessing with image output for 64 cores takes about 25 seconds. Question: is there any way to improve performance of image compositing for 18k cores for in-situ visualization?

2.       I also tried to avoid image output, but output polydata extracts using XMLPPolyDataWriter instead on 18k cores. In this case, in-situ coprocessing only takes about 20 seconds (compared to 14 minutes with image output). However, too many files are generated to a point that breaks the hard limit on maximal number of files in a directory since the parallel writer writes a vtp file from each of 18k cores. So the output data files have to be broken up into different directories. However, I got "cannot find file" error when I put a directory name as a parameter in coprocessor.CreateWriter() function call in my python script. I tried initially to put "data/vorticity_%t.pvtp" as a parameter, but it fails with "cannot find file" error. Not sure whether this is a bug or I need to put absolute full path in rather than a relative path to the current directory. Another question is whether there are ways to composite these files generated from different cores into one single file while doing coprocessing so only one composite file is generated rather than a huge number of files when running on large number of cores.

Thanks for any input, suggestions, and comments!

Regards,
Hong

_______________________________________________
Powered by www.kitware.com<http://www.kitware.com>

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView

Follow this link to subscribe/unsubscribe:
http://www.paraview.org/mailman/listinfo/paraview

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.paraview.org/pipermail/paraview/attachments/20131126/060d5d8c/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cpscript.py
Type: application/octet-stream
Size: 4025 bytes
Desc: cpscript.py
URL: <http://www.paraview.org/pipermail/paraview/attachments/20131126/060d5d8c/attachment.obj>