[vtk-developers] Documentation Parsing

Nikhil Shetty nikhil.shetty at kitware.com
Thu Apr 7 09:19:57 EDT 2011


Hi All,

Attached is a script (also few dependent files)I had written to convert
documentation from vtk format (adapted to slight user modification too) to
doxygen format.

At the outset I should warn you that the script is in written with secret
alien technology (common lisp :) ).
However they have worked well for me I have successfully used it for
converting vtk-style comments to doxygen style for large number of files in
the Titan project.

It uses PPCRE (Portable Perl-Compatible Regular Expression library)
http://weitz.de/cl-ppcre/  which means that the regular expressions are perl
based and at one place it actually uses a one of the perl scripts from VTK
itself.
It has modifications and corrections for some of the problems I faced with
the  VTK perl scripts.

Its meant to be run from the common lisp prompt (cl >) and lets one do
things like.

1. Collect a list of *.cxx or *.h files in a given directory

   Look for "(defparameter H ..."  and  "(defparameter CXX ..." and replace
with whatever directory you please.

2. Iterating through the list of files and deleting different types of of
headers from a list of .cxx or .h files

cl > (iterate-n-apply 'delete-doubleline-header H)
cl > (iterate-n-apply 'delete-doubleline-header CXX)
cl > (iterate-n-apply 'delete-singleline-header H)
cl > (iterate-n-apply 'delete-singleline-header CXX)

3. Inserting headers.

cl >  (iterate-n-apply 'insert-header H)
cl >  (iterate-n-apply 'insert-header CXX)

4. Replacing class documentation to doxygen style

cl > (iterate-n-apply 'loop-class H)

5. Replace method documentation to doxygen style

cl > (iterate-n-apply 'loop-method H)

You will also see that it loads shelisp.lisp (shebang lisp) which is my
unfinished attempt to make it user friendly (SORRY :( ).

-Nix





On Wed, Apr 6, 2011 at 9:31 AM, Robert Maynard
<robert.maynard at kitware.com>wrote:

> Hi All,
>
> Here is the Gerrit branch that allows Description blocks to not have white
> space separation.
>
> http://review.source.kitware.com/#change,1327
>
>
> On Wed, Apr 6, 2011 at 8:39 AM, Robert Maynard <robert.maynard at kitware.com
> > wrote:
>
>> Hi,
>>
>> I am currently testing a small change to the perl script that will
>> allow subsequent blocks that immediately follow tags, qualifiers, or
>> other Description blocks to be translated.
>>
>>
>> On Tue, Apr 5, 2011 at 2:46 PM, Will Schroeder
>> <will.schroeder at kitware.com> wrote:
>> > Fix the script (if possible) IMO.
>> > W
>> >
>> > On Tue, Apr 5, 2011 at 2:33 PM, David Cole <david.cole at kitware.com>
>> wrote:
>> >>
>> >> On Tue, Apr 5, 2011 at 2:26 PM, Brian Helba <brian.helba at kitware.com>
>> >> wrote:
>> >>>
>> >>> There won't be much drudgery, I already have a bash/sed script to do
>> the
>> >>> necessary edits.
>> >>
>> >> But the edits are unnecessary if the script is fixed....
>> >>
>> >>
>> >>>
>> >>> It would be nice to get a quick fix in, so the nightly HTML
>> documentation
>> >>> is more useful.
>> >>
>> >> It has been this way for a long, long time. It would be better to get
>> the
>> >> right fix (rather than quick) in and avoid the unnecessary edits.
>> >>
>> >> But maybe that's just me....
>> >>
>> >> Feel free to resolve this however you think is best.
>> >>
>> >>
>> >>>
>> >>> The issue with the script is that not enough analysis is performed
>> within
>> >>> each loop iteration to determine where a block of grouped comments /
>> >>> function declarations ends. After a "// Description" tag is found at
>> the
>> >>> beginning of an iteration, the script expects an arbitrary number of
>> >>> comment, code or macro lines. A special case could be added inside the
>> loop:
>> >>> while inside a Description block, if another "// Description" tag is
>> >>> encountered, end the outer block and continue to another iteration.
>> >>>
>> >>> On Tue, Apr 5, 2011 at 1:18 PM, David Cole <david.cole at kitware.com>
>> >>> wrote:
>> >>>>
>> >>>> I think it would be time better spent to fix the actual problem.
>> >>>>
>> >>>> I don't think it's intentional that the script requires a newline
>> >>>> preceding the "// Description" blocks... I can't see from the code
>> why that
>> >>>> would be the case. Guess my perl is rusty.
>> >>>>
>> >>>> Is there a brave perl volunteer listening that can save Brian from
>> the
>> >>>> drudgery of adding blank lines here and there...?
>> >>>>
>> >>>>
>> >>>> On Tue, Apr 5, 2011 at 1:02 PM, Brian Helba <brian.helba at kitware.com
>> >
>> >>>> wrote:
>> >>>>>
>> >>>>> In a number of cases, Utilities/Doxygen/doc_header2doxygen.pl is
>> >>>>> failing to parse and translate "// Description:" blocks in VTK
>> header files
>> >>>>> into actual Doxygen-style comment blocks. Specifically, 351 function
>> >>>>> descriptions across 101 classes appear to be affected.
>> >>>>>
>> >>>>> The issue can arise when the line "// Description:" is not preceded
>> by
>> >>>>> a newline. The Perl script does check for other documentation tags
>> (//BTX,
>> >>>>> //ETX) and qualifiers (public, etc.) but only correctly parses the
>> very
>> >>>>> first block; subsequent blocks that immediately follow tags,
>> qualifiers, or
>> >>>>> other Description blocks fail to be translated.
>> >>>>>
>> >>>>> The long-term fix is to make Utilities/Doxygen/
>> doc_header2doxygen.pl
>> >>>>> more robust, as it already attempts to correct for non-conforming
>> code in
>> >>>>> some cases. Also, the standards at
>> >>>>>
>> http://www.vtk.org/Wiki/VTK_Coding_Standards#Documentation_Standards do
>> not
>> >>>>> seem to make it adequately clear that a newline should precede all
>> >>>>> declaration blocks that include a "// Description:". If this is the
>> >>>>> standard, it should be clarified; if not, fixing the Perl script
>> would be a
>> >>>>> much higher priority.
>> >>>>>
>> >>>>> Meanwhile, I'm planning to directly fix the affected documentation
>> by
>> >>>>> simply inserting newlines. Should I modify only the places where
>> parsing
>> >>>>> errors actually do occur, or should I add newlines before all "//
>> >>>>> Description:" blocks that don't already have them (to conform with
>> the
>> >>>>> potential standard), even if they are being parsed correctly?
>> >>>>>
>> >>>>> Brian Helba
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> Powered by www.kitware.com
>> >>>>>
>> >>>>> Visit other Kitware open-source projects at
>> >>>>> http://www.kitware.com/opensource/opensource.html
>> >>>>>
>> >>>>> Follow this link to subscribe/unsubscribe:
>> >>>>> http://www.vtk.org/mailman/listinfo/vtk-developers
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Brian Helba
>> >>> Medical Imaging
>> >>> Kitware, Inc.
>> >>>
>> >>
>> >>
>> >> _______________________________________________
>> >> Powered by www.kitware.com
>> >>
>> >> Visit other Kitware open-source projects at
>> >> http://www.kitware.com/opensource/opensource.html
>> >>
>> >> Follow this link to subscribe/unsubscribe:
>> >> http://www.vtk.org/mailman/listinfo/vtk-developers
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > William J. Schroeder, PhD
>> > Kitware, Inc.
>> > 28 Corporate Drive
>> > Clifton Park, NY 12065
>> > will.schroeder at kitware.com
>> > http://www.kitware.com
>> > (518) 881-4902
>> >
>> > _______________________________________________
>> > Powered by www.kitware.com
>> >
>> > Visit other Kitware open-source projects at
>> > http://www.kitware.com/opensource/opensource.html
>> >
>> > Follow this link to subscribe/unsubscribe:
>> > http://www.vtk.org/mailman/listinfo/vtk-developers
>> >
>> >
>> >
>>
>>
>>
>> --
>> Robert Maynard
>>
>
>
>
> --
> Robert Maynard
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://www.vtk.org/mailman/listinfo/vtk-developers
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/vtk-developers/attachments/20110407/75ce6bff/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: insert-header.lisp
Type: application/octet-stream
Size: 11567 bytes
Desc: not available
URL: <http://public.kitware.com/pipermail/vtk-developers/attachments/20110407/75ce6bff/attachment-0003.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: shelisp.lisp
Type: application/octet-stream
Size: 13886 bytes
Desc: not available
URL: <http://public.kitware.com/pipermail/vtk-developers/attachments/20110407/75ce6bff/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: method-descriptions.pl
Type: application/octet-stream
Size: 1863 bytes
Desc: not available
URL: <http://public.kitware.com/pipermail/vtk-developers/attachments/20110407/75ce6bff/attachment-0005.obj>


More information about the vtk-developers mailing list