[vtkusers] Many VTK build tests fail when run from Jenkins/Bamboo or VNC display, even when DISPLAY is set to good opengl-enabled display

[John]

Hi Everybody,

This is more of a question about building/testing VTK than using VTK, so
perhaps i should be posting this to the vtk-deveolopes list, but...

I'm using bamboo & jenkins to build the VTK code from source and run the
testing suite, and I've hit a weird problem:

When i run the build/test scripts from a terminal on my actual Xorg
display, all 1300+ tests succeed.

But when I run the build/test via jenknins/bamboo, or manually from a
VNC display, and use DISPLAY environment variable to *send* the opengl
output windows to my Xorg/nvidia display, I have 347 tests failing (and
967 succeed). If I walk over to my main head and look at the Xorg/nvidia
display, while this is happening, there are opengl windows opening & I
can see most of the tests are working fine. So, there doesn't seem to be
anything *totally* wrong with my setup. But for some reason, 347 of the
tests fail.

Perhaps the reason these tests are failing is obvious to someone with a
more knowledge of the testing process?

I've got a feeling I'm just overlooking something. Or if not, does
anyone have any suggestions re how I could get some more info to debug
the issue?

Ta for any advice!
John

--------------------------
ps if it helps, the full results of a build with all these inexplicably
failing tests can be found at:
https://homunculoid.com/CDash/viewTest.php?onlyfailed&buildid=38

Also, I've isolated a command to run one of the failing tests, and I can
replicate the problem easily, with a script containing just one test:

export DISPLAY=:0
VTK_BUILD_DIR="/data-ssd/data/development/src/vtk/build/jenkins/dashboard/nightly/1/vtk-build"
"${VTK_BUILD_DIR}/bin/vtkChartsCoreCxxTests" "TestMultipleRenderers"
"-E" "25" "-D" "${VTK_BUILD_DIR}/ExternalData/Testing" "-T"
"${VTK_BUILD_DIR}/Testing/Temporary" "-V"
"${VTK_BUILD_DIR}/ExternalData/Charts/Core/Testing/Data/Baseline/TestMultipleRenderers.png"

Running the test from a terminal directly on the Xorg display is all
good, with results showing ImageError=0:
<DartMeasurement name="ImageError" type="numeric/double">0
</DartMeasurement>
<DartMeasurement name="BaselineImage"
type="text/string">Standard</DartMeasurement><DartMeasurement
name="WallTime" type="numeric/double">0.0331759</DartMeasurement>
<DartMeasurement name="CPUTime"
type="numeric/double">0.079182</DartMeasurement>

Running the test from a terminal on a VNC display, but with DISPLAY set
to point to the proper Xorg display, spits out a bunch of opengl info,
which shows it is attempting to run on the right head, and all the GL
info looks good (to me), and then at the bottom are the results which
indicate the test failed:

server glx vendor string:  NVIDIA Corporation
server glx version string:  1.4
server glx extensions:  GLX_EXT_visual_info GLX_EXT_visual_rating
GLX_EXT_import_context GLX_SGIX_fbconfig GLX_SGIX_pbuffer
GLX_SGI_video_sync GLX_SGI_swap_control GLX_EXT_swap_control
GLX_EXT_swap_control_tear GLX_EXT_texture_from_pixmap GLX_EXT_buffer_age
GLX_ARB_create_context GLX_ARB_create_context_profile
GLX_EXT_create_context_es_profile GLX_EXT_create_context_es2_profile
GLX_ARB_create_context_robustness GLX_NV_delay_before_swap
GLX_EXT_stereo_tree GLX_EXT_libglvnd GLX_ARB_context_flush_control
GLX_NV_robustness_video_memory_purge GLX_ARB_multisample
GLX_NV_float_buffer GLX_ARB_fbconfig_float GLX_EXT_framebuffer_sRGB
GLX_NV_copy_image
client glx vendor string:  NVIDIA Corporation
client glx version string:  1.4
glx extensions:  GLX_EXT_visual_info GLX_EXT_visual_rating
GLX_EXT_import_context GLX_SGIX_fbconfig GLX_SGIX_pbuffer
GLX_SGI_video_sync GLX_SGI_swap_control GLX_EXT_swap_control
GLX_EXT_swap_control_tear GLX_EXT_texture_from_pixmap GLX_EXT_buffer_age
GLX_ARB_create_context GLX_ARB_create_context_profile
GLX_EXT_create_context_es_profile GLX_EXT_create_context_es2_profile
GLX_ARB_create_context_robustness GLX_NV_delay_before_swap
GLX_EXT_stereo_tree GLX_ARB_context_flush_control
GLX_NV_robustness_video_memory_purge GLX_ARB_multisample
GLX_NV_float_buffer GLX_ARB_fbconfig_float GLX_EXT_framebuffer_sRGB
GLX_NV_copy_image GLX_ARB_get_proc_address
OpenGL vendor string:  NVIDIA Corporation
OpenGL renderer string:  GeForce GTX 960/PCIe/SSE2
OpenGL version string:  4.5.0 NVIDIA 384.98
OpenGL extensions:
...
lots of extensions
...
X Extensions:  Generic Event Extension, SHAPE, MIT-SHM, XInputExtension,
XTEST, BIG-REQUESTS, SYNC, XKEYBOARD, XC-MISC, XFIXES, RENDER, RANDR,
XINERAMA, Composite, DAMAGE, MIT-SCREEN-SAVER, DOUBLE-BUFFER, RECORD,
DPMS, Present, X-Resource, XVideo, XFree86-VidModeExtension,
XFree86-DGA, DRI2, GLX, NV-GLX, NV-CONTROL, XINERAMA

<DartMeasurement name="ImageError" type="numeric/double">244344
</DartMeasurement>
<DartMeasurement name="BaselineImage"
type="numeric/integer">1</DartMeasurement><DartMeasurementFile
name="TestImage"
type="image/png">/data-ssd/data/development/src/vtk/build/jenkins/dashboard/nightly/1/vtk-build/Testing/Temporary/TestMultipleRenderers.png</DartMeasurementFile>
Failed Image Test ( TestMultipleRenderers.png ) : 244344
<DartMeasurementFile name="DifferenceImage"
type="image/png">/data-ssd/data/development/src/vtk/build/jenkins/dashboard/nightly/1/vtk-build/Testing/Temporary/TestMultipleRenderers.diff.png</DartMeasurementFile><DartMeasurementFile
name="ValidImage"
type="image/png">/data-ssd/data/development/src/vtk/build/jenkins/dashboard/nightly/1/vtk-build/ExternalData/Charts/Core/Testing/Data/Baseline/TestMultipleRenderers_1.png</DartMeasurementFile><DartMeasurement
name="WallTime" type="numeric/double">0.695937</DartMeasurement>
<DartMeasurement name="CPUTime"
type="numeric/double">1.87324</DartMeasurement>

So, I can see two issues here - first, there is a difference image, so
the results were different to the baseline image, and 2nd, the WallTime
& CPUTime are much greater on the failed test, compared to when it
succeeds. I had been hoping the reason for failure might just be
timing-related, and maybe it is, but I had a quick look and I can't see
anything in the TestMultipleRenderers.cxx which stipulates any strict
time limit for the test, so I don't think that's it.

It's puzzling! Pls Halp!