[vtk-developers] [LONG] Antialiasing with multisampling method

[Francois Bertel]

Hi everybody,

Folks who are using VTK or ParaView under Unix may have noticed that
antialiasing in on by default. I wrote "may", because it actually depends on
your graphic configuration (graphic card and driver version).
In my case, Linux gentoo x86, NVidia GeForce 6800, driver 1.0-7676.

The consequence of that is some regression tests fail. Even if
VTK/Rendering/Testing/Cxx/vtkRegressionTestImage tries to handle this case,
some test like DistributedData-image fails because it uses compositing and
relies on the values in the color buffer.

In my case, a fast workaround is to run nvidia-settings, go to
"Antialiasing Settings" menu, check "Override Application Setting for
Antialiasing Settings" and set the value to "Off". But it is just a workaround.

So, why antialiasing is on? The short answer is multisampling. The long answer
is following. If you want to skip this explanation, there is a question at
the end of this message, which is actually the real purpose of this e-mail.

<long_answer>

With OpenGL, the usual way to deal with antialiasing for points, lines and
polygons is to call glDisable/Enable() respectively with argument
GL_POINT_SMOOTH, GL_LINE_SMOOTH or GL_POLYGON_SMOOTH (OpenGL Spec 1.5, section
3.3 page 74, section 3.4 page 77, section 3.5 page 84).
http://www.opengl.org/documentation/spec.html

In VTK, such calls appear in VTK/Rendering/vtkOpenGLRenderWindow. They are
controlled by some ivars defined in the superclass
VTK/Rendering/vtkRenderWindow with respectively Get/SetPointSmoothing(),
Get/SetLineSmoothing(), Get/SetPolygonSmoothing().

The other way is to use multisampling. It is part of core OpenGL since
OpenGL 1.3 and used to be ARB extension #5 (GL_ARB_multisample) that used to be
non-ARB extension #25 (GL_SGIS_multisample).
http://oss.sgi.com/projects/ogl-sample/registry/

As it is an expensive operation, it is not supported by all OpenGL contexts.
If you want to use it, you are supposed to obtain a multisample-capable context
using glX or WGL_ARB_multisample extension (OpenGL spec 1.5, F.3 page 281).

In VTK, superclass VTK/Rendering/vtkOpenGLRenderWindow has a MultiSamples
ivar and a static ivar GlobalMaximumNumberOfMultiSamples.
The default value of GlobalMaximumNumberOfMultiSamples is 8. The
default value for MultiSamples is GlobalMaximumNumberOfMultiSamples.

The MultiSamples ivar is actually used in the concrete class
VTK/Rendering/vtkXOpenGLRenderWindow to query a multisample-capable context.
Here are the lines involved in function vtkXOpenGLRenderWindowTryForVisual():

  if (multisamples)
    {
#ifdef GLX_SAMPLE_BUFFERS_SGIS
    attributes[index++] = GLX_SAMPLE_BUFFERS_SGIS;
    attributes[index++] = 1;
    attributes[index++] = GLX_SAMPLES_SGIS;
    attributes[index++] = multisamples;
#endif
    }

It means you will get a multisampling context if ALL the following conditions
exist:
1. You are using Unix (there is a call to glXChooseVisual())
2. You compile VTK on a platform that have a version of glxext.h that
defines macro GLX_SAMPLE_BUFFERS_SGIS
3. You run VTK on a platform that supports some multisampling-capable context.

Once you get a multisampling-capable context, GL_SAMPLE_BUFFERS is equal to 1.
At run time, the way to switch between multisampling or singlesampling
(it means no antialiasing) is to call glEnable/Disable with GL_MULTISAMPLE.

Here is the trap. By default, GL_MULTISAMPLE is on. In addition, if
GL_SAMPLE_BUFFERS is equal to 1 and GL_MULTISAMPLE is on, multisampling is used
on points, lines and polygons *regardless* the value of GL_POINT_SMOOTH,
GL_LINE_SMOOTH and GL_POLYGON_SMOOTH (OpenGL spec 1.5, section 3.3.3 page 77,
section 3.4.4 page 83, section 3.5.6 page 89 ).

So, now you know why.

</long_answer>

Here is my question. There are a couple of ways to make the regression tests
not to fail because of multisampling.

1. Remove multisampling query: we will never get a multisample context, but
users may like to have antialiasing anyway.
2. Once we get a multisample context, call glDisable(GL_MULTISAMPLE) and never
use multisampling. We potentially get a context with a special buffer that
allocate some video memory for nothing, when this memory could be useful to
load a lot of textures.
3. Set default value of GlobalMaximumNumberOfMultiSamples to 0 instead of 8.
4. Set default value of MultiSamples to 0 instead of
   GlobalMaximumNumberOfMultiSamples.
5. On each potentially failing test, call SetMultiSamples(0) at the beginning,
which requires access to the concrete type of RenderWindow.

Solution 1 and 2 are really low level and limit application choice.

Solution 3 and 4 give more control to the application, but by default there
will be no antialiasing (no multisample-capable context) and it has to be set
by the application during the initialization of the render window.

Solution 5 looks easy for VTK tests, less easy for ParaView tests.

Any opinion about thoses solution, or some alternate solution is welcome.

-- 
François Bertel, PhD  | Kitware Inc. Suite 204
1 (518) 371 3971 x113 | 28 Corporate Drive
                      | Clifton Park NY 12065, USA