VTK/Multicore and Streaming

Multicore and Streaming

Background

This page is to act as a facilitator for current efforts in adding advanced multicore and streaming functionality into the VTK framework. We will be building on some past work by others, and this page is meant to facilitate discussion and collaboration of those contributing via a number of synergistic research efforts.

Discussions

Meeting - 9 June 2009

• Brian would like to have the pipelines on the wiki along with the discussion, design, use cases, etc.
• Berk would like to modify Huy's current changes (in git) to leverage the existing streaming classes in vtk. Huy indicated that the current streamers require extents calculation, which might not always be available. Berk suggested that we look at using pieces instead of extents.
• Need to subclass the composite data pipeline with support for temporal data.
• Huy is still working on Push model. It's implemented but not complete and accessible.
• Berk would like more of the functionality available in the executive. VTK should be changed rather than putting a band-aid on the current implementation.
• Tim will send Huy a simple text pipeline that he would like to have streamed (this includes tables.)
• Need to start moving text analysis filters into VTK proper.
• Huy will remain in the git branch to continue experimentation with the streaming, and we're going to start implementing a text streaming set of examples for testing the new functionality.
• Berk suggested that we need to develop a 'vision' statement (with use cases) for how we want the new multicore/streaming algorithms to work together.

Action items

• Huy will update the wiki to include the papers and a graphvis version of the pipeline on the website that can be used to facilitate discussion
• Tim/Jason will start moving TextAnalysis kit into VTK proper
• Tim/Huy will start developing examples demonstrating text streaming as a means of vetting the streaming/multicore functionality that Huy has already developed.
• Jason/Berk/Huy need to develop a vision statement and place it on the wiki.
• Jason/Berk/Huy need to develop a list of goals and use cases for uniting this work to current efforts and desired efforts in VTK.

Meeting - 8 July 2009

• Updates and Current efforts-
  • Leveraging the existing VTK streaming structures (see http://www.vtk.org/Wiki/VTK/ThreadedStreamingPipeline)
  • Pieces vs. extents
  • subclassing composite data pipeline to access temporal data
  • moving functionality to the executives
• Other efforts-
  • Development of a push model
  • Text analysis pipeline
• Multicore vision statement
  • vision statement
  • new proposals (CSRF10)
  • uniting this work to current and desired efforts
• Wiki updates

Action items

• Huy will work on updating the wiki with the latest technical information and discussions.
• Huy to continue implementation and completion of technical items from last meeting, as well as some incorporation of discussion regarding using extents correctly.
• Berk will start a draft vision statement for the multicore work. Claudio and Jason will follow up with edits.
• Jason will move the text analysis filters from vtkSNL to vtk to facilitate creation of a more comprehensive streaming example. We will meet in two weeks to discuss implementation details of this new example.
• Jason to call Claudio regarding potential new proposal.

Additional action items from July 15 meeting

Jason and Tim added the text analysis kit and tests to vtk last week. We'd like to experiment with two testing strategies:

• Test 1. Using TestTermDictionary, add a TableView to the end of the test and try a streaming pull model for streaming documents to the view by loading a document (or a small collection) and updating the view.
• Test 2. Again using TestTermDictionary as the starting point, add a TableView and a timer to the document loader. Create a push model which updates the view each time a new document is added (via the timer).

Meeting Aug. 12

• Huy is requesting meetings on a more frequent basis to enable improved efficiency.
• Huy will update and commit new scheduler for review by Berk.
• Huy sent out a new update on the text analysis with streaming. Needs to be reviewed by all.
• SNL is struggling with Git. Berk offered several suggestions; Jason will follow up.
• Huy is looking for more complicated pipelines to stress test the streaming capabilities.
• Tim would like to see the text analysis pipeline that Huy developed without the explicit merging tools. Would like to have the merging handled in each of the general filter classes (the filter should accumulate data without the merge in a separate filter.)
• Berk asked about our vision statement. We'd like to make this a generalized capability in vtk, and work to incorporate a vision for all of the multicore efforts at the vtkAlgorithm level.

Action items from Aug. 12

• Jason will set up next meeting for a week from Tuesday.
• Huy will send out discussion material for scheduler.
• Tim to work on github problems at SNL.
• Berk will continue work on vision statement.
• Berk will work to get Jim Ahrens involved in multicore discussions.
• Ken, Tim and Berk to outline some potential use cases for efforts that the multicore effort should incorporate.

Meeting Aug. 25

• Huy sent out information on the new scheduler (http://www.vtk.org/Wiki/VtkExecutionScheduler)
• Berk sent out initial thoughts on the multicore vision and use cases (http://www.vtk.org/Wiki/VTK/Multicore_Vision)
• Berk is going to invite David DeMarle and Utkarsh Ayachit to begin attending the multicore meetings and doing code reviews.
• Tim mentioned similarities between the new scheduler and task management in tbb.

Action items from Aug. 25

• Berk - invite Dave and Utkarsh to join multicore discussions.
• Huy - gather and send Dave and Utkarsh multicore material to review. Also point them to the github repository for code review.
• Berk - continue work on multicore vision
• Jason - send Huy tbb book.
• Jason - arrange next meeting after determining with Berk some review time frames.

Meeting Sept. 23

Agenda:

• code review thoughts: Dave D.
• Recent and current efforts: Huy
  • Text analysis pipeline
• Code commit status
• ARB status: Berk
• Multicore vision development
• Integration with other VTK multicore/streaming efforts
• Upcoming efforts