[vtkusers] [Insight-developers] wx/vtk weirdness
Francois Bertel
francois.bertel at kitware.com
Mon Nov 30 16:21:57 EST 2009
It is because most of tests are not C++ tests.
Look at the experimental section, my machine is xanth (Ubuntu 9.10
x86_64). There are 3 submissions.
http://www.cdash.org/CDash/index.php?project=VTK#Experimental
1. force locale on C++ tests with French locale:
82 tests failed 1088 passes
2. Fixed event recorder, use French locale:
56 tests failed (-26) 1114 passed
3. Use English locale after event recorder fix:
10 test failed (-46) 1161 passed
On Mon, Nov 30, 2009 at 4:15 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
> Excellent. That's what I have been doing in itk classes that use streams.
>
> I'm surprised that more tests did not fail in vtk.
>
> Bill
>
> On Mon, Nov 30, 2009 at 4:02 PM, Francois Bertel
> <francois.bertel at kitware.com> wrote:
>> VTK/Common/vtkTimePointUtility.cxx
>>
>> I added a imbue(vtkstd::locale::classic()); on the relevant C++ stream.
>>
>> ref: http://public.kitware.com/cgi-bin/viewcvs.cgi/Common/vtkTimePointUtility.cxx?r1=1.2&r2=1.3
>>
>> On Mon, Nov 30, 2009 at 3:53 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>> Can you share what you changed? Which class?
>>>
>>> Bill
>>>
>>> On Mon, Nov 30, 2009 at 3:42 PM, Francois Bertel
>>> <francois.bertel at kitware.com> wrote:
>>>> Not only locale for language like French are different for thousand
>>>> separator but also English is different from C|POSIX !!
>>>>
>>>> An integer of value 2000 will generate the string "2000" in POSIX|C
>>>> locale but will generate the string "2,000" in "en_US.UTF-8" locale.
>>>>
>>>> Example of test that was failing with "en_US.UTF-8":
>>>>
>>>> http://www.cdash.org/CDash/testDetails.php?test=37688960&build=483148
>>>>
>>>> This test is now fixed by forcing C|POSIX in the C++ stream.
>>>>
>>>>
>>>> On Mon, Nov 30, 2009 at 1:04 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>>>> Tom,
>>>>>
>>>>> I don't think we need to be concerned with the per-thread issue. My
>>>>> changes are all local to the methods that need c-locale's. For
>>>>> example, surrounding the ReadImageInformation calls with save/restore
>>>>> locales in many readers fixes the readers.
>>>>>
>>>>> Maybe you can try my simple itkLocale class. I'll send it in a separate mail.
>>>>>
>>>>> Bill
>>>>>
>>>>> On Mon, Nov 30, 2009 at 12:57 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>>>>> Tom,
>>>>>>
>>>>>> What I have done is create an itk class, itkLocale, that in its
>>>>>> constructor saves the current c-locale and creates a new one. The
>>>>>> destructor restores the original locale.
>>>>>>
>>>>>> C++-locales are handled explicitly with imbue.
>>>>>>
>>>>>> I'll try your approach in the itkLocale class.
>>>>>>
>>>>>> BTW, I have not checked in any code yet.
>>>>>>
>>>>>> I think it is time to start an ITK Policy and Procedure page for
>>>>>> internationalization on the itk wiki.
>>>>>>
>>>>>> It can include the work you are doing with unicode.
>>>>>>
>>>>>> I'll give it a stab today. Then we can start a new thread on itk developers.
>>>>>>
>>>>>> Bill
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 30, 2009 at 12:50 PM, Tom Vercauteren
>>>>>> <tom.vercauteren at gmail.com> wrote:
>>>>>>> Hi Bill,
>>>>>>>
>>>>>>> I would just like to mention that in our group we have been facing
>>>>>>> several issues with locale support on mac.
>>>>>>>
>>>>>>> Looking at the documentation of xlocale:
>>>>>>> http://developer.apple.com/Mac/library/documentation/Darwin/Reference/ManPages/man3/xlocale.3.html
>>>>>>> you will see that setlocale will have absolutely no effect (on mac,
>>>>>>> maybe on windows also
>>>>>>> http://msdn.microsoft.com/en-us/library/26c0tb7x.aspx ) if a
>>>>>>> per-thread locale had been specified.
>>>>>>>
>>>>>>> Using newlocale, duplocale, uselocale and freelocale seems a better option:
>>>>>>> http://www.opengroup.org/onlinepubs/9699919799/functions/uselocale.html
>>>>>>> http://www.opengroup.org/onlinepubs/9699919799/functions/newlocale.html
>>>>>>> http://www.opengroup.org/onlinepubs/9699919799/functions/duplocale.html
>>>>>>> http://www.opengroup.org/onlinepubs/9699919799/functions/freelocale.html
>>>>>>>
>>>>>>>
>>>>>>> For example, to put only the LC_NUMERIC parameter to C, here is how we
>>>>>>> chose to proceed:
>>>>>>>
>>>>>>> // Create a new locale based on a copy of the current one (which need
>>>>>>> not be explicitly set)
>>>>>>> // but where LC_NUMERIC has been modified to match the C locale
>>>>>>> locale_t tempLocale = newlocale(LC_NUMERIC_MASK, NULL, NULL);
>>>>>>> // Use this new locale and store the current one to reset it aftwards
>>>>>>> locale_t previousLocale = uselocale(templocale);
>>>>>>>
>>>>>>> [Actual useful code]
>>>>>>>
>>>>>>> // Reset the locale to the previous one
>>>>>>> uselocale(previousLocale)
>>>>>>> // More house keeing
>>>>>>> freelocale(tempLocale)
>>>>>>>
>>>>>>>
>>>>>>> Actually, on mac we also had to do explicitly create thread-specific
>>>>>>> locale to avoid some random crash in heavily threaded code. But that
>>>>>>> is a slightly different topic
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Tom
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Nov 30, 2009 at 17:08, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>>>>>>> I've already made a pass through the itk readers. Several were broken
>>>>>>>> wrt locale.
>>>>>>>>
>>>>>>>> I used a combination of C++ and C locale api's. I encapsulated the C
>>>>>>>> locale in and itk::Locale class.
>>>>>>>>
>>>>>>>> I have not checked in any changes yet, but I went from over 40 failing
>>>>>>>> I/O tests down to 0.
>>>>>>>>
>>>>>>>> Bill
>>>>>>>>
>>>>>>>> On Mon, Nov 30, 2009 at 10:58 AM, Marcus D. Hanwell
>>>>>>>> <marcus.hanwell at kitware.com> wrote:
>>>>>>>>> On Saturday 28 November 2009 13:19:58 Bill Lorensen wrote:
>>>>>>>>>> Luis,
>>>>>>>>>>
>>>>>>>>>> Actually, we can surround certain functions with c-style setlocale and
>>>>>>>>>> it solves the problems. I already fixed StimulateImageIO,
>>>>>>>>>> PolygonGroupSpatialObjectXMLFile and VTKImageIO. I have not checked in
>>>>>>>>>> any changes yet. Still more investigation is needed as to the best
>>>>>>>>>> portable solution.
>>>>>>>>>>
>>>>>>>>>> I do this:
>>>>>>>>>> const char *originalLocale;
>>>>>>>>>> char *currentLocale;
>>>>>>>>>> originalLocale = setlocale(LC_NUMERIC, NULL);
>>>>>>>>>> currentLocale = strdup(originalLocale ? originalLocale : "C");
>>>>>>>>>> setlocale(LC_NUMERIC, "C");
>>>>>>>>>>
>>>>>>>>>> ......
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> setlocale(LC_NUMERIC, currentLocale);
>>>>>>>>>> free(currentLocale);
>>>>>>>>>
>>>>>>>>> A page from the Apache C++ resource site gives some good explanations of why
>>>>>>>>> the C++ locale should be preferred. Mainly because the C locale is a global
>>>>>>>>> resource, whereas in the C++ locale instances of the locale can be created.
>>>>>>>>>
>>>>>>>>> http://stdcxx.apache.org/doc/stdlibug/24-3.html
>>>>>>>>>>
>>>>>>>>>> For example in StimulateImageIO I surrounded the calls in
>>>>>>>>>> StimulateImageIO::InternalReadImageInformation with the above
>>>>>>>>>> snippets.
>>>>>>>>>>
>>>>>>>>>> There are other readers/writers that use C++ streams, and there we
>>>>>>>>>> will have to imbue.
>>>>>>>>>
>>>>>>>>> I think that using the C locale is probably the easiest short term solution,
>>>>>>>>> but to support GUIs using different locales we should be using the C++ locale
>>>>>>>>> class and associated API. Otherwise users of our libraries may suffer from
>>>>>>>>> subtle bugs if reading and writing files in separate threads that are all
>>>>>>>>> using C locales.
>>>>>>>>>>
>>>>>>>>>> Even in some code that uses streams, people still use sscanf, atof,
>>>>>>>>>> etc. In the long run we probably should address these cases with
>>>>>>>>>> stream replacements.
>>>>>>>>>
>>>>>>>>> We should aim to ensure new code uses C++ streams and imbue, and replace
>>>>>>>>> existing code with C++ streams. As you say this should probably be part of a
>>>>>>>>> larger i18n effort which would involve using Unicode strings and other changes
>>>>>>>>> to our libraries.
>>>>>>>>>
>>>>>>>>> Marcus
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Powered by www.kitware.com
>>>>>>>>
>>>>>>>> Visit other Kitware open-source projects at
>>>>>>>> http://www.kitware.com/opensource/opensource.html
>>>>>>>>
>>>>>>>> Kitware offers ITK Training Courses, for more information visit:
>>>>>>>> http://kitware.com/products/protraining.html
>>>>>>>>
>>>>>>>> Please keep messages on-topic and check the ITK FAQ at:
>>>>>>>> http://www.itk.org/Wiki/ITK_FAQ
>>>>>>>>
>>>>>>>> Follow this link to subscribe/unsubscribe:
>>>>>>>> http://www.itk.org/mailman/listinfo/insight-developers
>>>>>>>>
>>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Powered by www.kitware.com
>>>>>
>>>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>>>
>>>>> Please keep messages on-topic and check the VTK FAQ at: http://www.vtk.org/Wiki/VTK_FAQ
>>>>>
>>>>> Follow this link to subscribe/unsubscribe:
>>>>> http://www.vtk.org/mailman/listinfo/vtkusers
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> François Bertel, PhD | Kitware Inc. Suite 204
>>>> 1 (518) 371 3971 x113 | 28 Corporate Drive
>>>> | Clifton Park NY 12065, USA
>>>> _______________________________________________
>>>> Powered by www.kitware.com
>>>>
>>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>>
>>>> Please keep messages on-topic and check the VTK FAQ at: http://www.vtk.org/Wiki/VTK_FAQ
>>>>
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://www.vtk.org/mailman/listinfo/vtkusers
>>>>
>>>
>>
>>
>>
>> --
>> François Bertel, PhD | Kitware Inc. Suite 204
>> 1 (518) 371 3971 x113 | 28 Corporate Drive
>> | Clifton Park NY 12065, USA
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>
>> Please keep messages on-topic and check the VTK FAQ at: http://www.vtk.org/Wiki/VTK_FAQ
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.vtk.org/mailman/listinfo/vtkusers
>>
>
--
François Bertel, PhD | Kitware Inc. Suite 204
1 (518) 371 3971 x113 | 28 Corporate Drive
| Clifton Park NY 12065, USA
More information about the vtkusers
mailing list