[vtk-developers] "vtkCommonCoreTcl-TestEmptyInput" fails due to use-after-free in vtkDataEncoder

[Sean McBride]

Hi all,

On my Rogue7 dashboard vtkCommonCoreTcl-TestEmptyInput fails intermittently. 

I debugged it a bit and, long story short, the evidence suggested a use-after-free so I enabled MallocScribble (on OS X it writes 0x55 to freed memory) and caught it in lldb:

(lldb) bt
* thread #7: tid = 0x64d84d, 0x00007fff8a7afbc9 libsystem_c.dylib`pthread_mutex_lock + 20, stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00007fff8a7afbc9 libsystem_c.dylib`pthread_mutex_lock + 20
    frame #1: 0x0000000100fd0e3e libvtkCommonCore-6.3.1.dylib`vtkSimpleMutexLock::Lock(this=0x555555555555561d) + 94 at vtkMutexLock.cxx:75
    frame #2: 0x000000013886bd3e libvtkWebCore-6.3.1.dylib`(anonymous namespace)::vtkSharedData::BeginWorker(this=0x5555555555555555) + 158 at vtkDataEncoder.cxx:99
    frame #3: 0x000000013886ad18 libvtkWebCore-6.3.1.dylib`(anonymous namespace)::Worker(calldata=0x000000010f046740) + 248 at vtkDataEncoder.cxx:304
    frame #4: 0x00007fff8a7aa772 libsystem_c.dylib`_pthread_start + 327
    frame #5: 0x00007fff8a7971a1 libsystem_c.dylib`thread_start + 13

Notice the this=0x5555555555555555.  Definitely use-after-free.

Hopefully this is enough for someone that knows this code to squash the bug...!

Cheers,

-- 
____________________________________________________________
Sean McBride, B. Eng                 sean at rogue-research.com
Rogue Research                        www.rogue-research.com 
Mac Software Developer              Montréal, Québec, Canada