[Insight-developers] Why so many tests on the dashboard?
Miller, James V (Research)
millerjv@crd.ge.com
Mon, 24 Mar 2003 14:00:10 -0500
This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.
------_=_NextPart_001_01C2F237.9802F950
Content-Type: text/plain;
charset="iso-8859-1"
A question arose a couple of weeks ago on why the Dashboard's "Test NA"
column has so many entries. Basically, it is reporting that each
architecture has either 379 or 426 tests that were not run on that
architecture.
Dart builds a list of tests that are union of all the tests run on all the
architectures. If on a particular architecture, a test was not in the
Test.xml file, then that architecture gets an NA for the test. The original
intent of this column was to make sure that all the architectures were
running the same number of tests (and have that number be dynamic). It also
allows us to see when people add more tests during the day.
The following build:
<SiteName>hythloth.kitware</SiteName>
<BuildName>zTesting-Linux-gcc322</BuildName>
is configured to build the tests outside of the original source
distribution. I.e. if the Testing directory is copied to outside the
standard source distribution, you can configure it as a separate project
(with its own DSW) and have it find the ITK includes and libraries from your
standard build. This was done because it is a good way to test an official
"installation" of ITK.
Now the kicker, the "test names" for this build are of the form
"./Code/Common/...." whereas normally the "test names" are
"./Testing/Code/Common/....". So Dart thinks this build has a set of tests
that nobody else is testing. Furthermore, this build (as far as Dart is
concerned) is not testing any of the tests that all the other builds are
doing.
We could probably trick the system into doing what we want if the
Testing/CMakeLists.txt (for the configuration where the tests are building
against an installation) can slip in an extra directory called Testing.
So the source tree would look like
Testing
/Code
/Code/Common
....
but the build tree would look like
Testing-bin
/Testing
/Testing/Code
/Testing/Code/Common
....
I am not sure that CMake can do this for us...
<?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" />
Jim Miller
_____________________________________
Visualization & Computer Vision
GE Research
Bldg. KW, Room C218B
P.O. Box 8, Schenectady NY 12301
millerjv@research.ge.com <mailto:millerjv@research.ge.com>
james.miller@research.ge.com
(518) 387-4005, Dial Comm: 8*833-4005,
Cell: (518) 505-7065, Fax: (518) 387-6981
------_=_NextPart_001_01C2F237.9802F950
Content-Type: text/html;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Diso-8859-1">
<META content=3D"MSHTML 6.00.2715.400" name=3DGENERATOR></HEAD>
<BODY>
<DIV><SPAN class=3D777014518-24032003><FONT size=3D2>A question arose a =
couple of=20
weeks ago on why the Dashboard's "Test NA" column has so many =
entries. =20
Basically, it is reporting that each architecture has either 379 =
or=20
426 tests that were not run on that =
architecture.</FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D777014518-24032003><FONT size=3D2>Dart builds a list =
of tests=20
that are union of all the tests run on all the architectures. If =
on a=20
particular architecture, a test was not in the Test.xml file, then that =
architecture gets an NA for the test. The original intent of this =
column=20
was to make sure that all the architectures were running the same =
number of=20
tests (and have that number be dynamic). It also allows us to see =
when=20
people add more tests during the day.</FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D777014518-24032003><FONT size=3D2>The following=20
build:</FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D777014518-24032003><FONT=20
size=3D2><SiteName>hythloth.kitware</SiteName><BR><BuildN=
ame>zTesting-Linux-gcc322</BuildName><BR></FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT size=3D2>is configured to =
build the=20
tests outside of the original source distribution. I.e. if the =
Testing=20
directory is copied to outside the standard source distribution, you =
can=20
configure it as a separate project (with its own DSW) and have it find =
the ITK=20
includes and libraries from your standard build. This was done =
because it=20
is a good way to test an official "installation" of =
ITK.</FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D777014518-24032003><FONT size=3D2>Now the kicker, =
the "test=20
names" for this build are of the form "./Code/Common/...." whereas =
normally the=20
"test names" are "./Testing/Code/Common/....". So Dart thinks =
this build=20
has a set of tests that nobody else is testing. Furthermore, this =
build=20
(as far as Dart is concerned) is not testing any of the tests that all =
the other=20
builds are doing.</FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D777014518-24032003><FONT size=3D2>We could probably =
trick the=20
system into doing what we want if the Testing/CMakeLists.txt (for the=20
configuration where the tests are building against an installation) can =
slip in=20
an extra directory called Testing. </FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D777014518-24032003><FONT size=3D2>So the source tree =
would look=20
like </FONT></SPAN><SPAN class=3D777014518-24032003><FONT=20
size=3D2></FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2>Testing</FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003> <FONT=20
size=3D2>/Code</FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT size=3D2> =
/Code/Common</FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT size=3D2> =
....</FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D777014518-24032003><FONT size=3D2>but the build tree =
would look=20
like</FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2>Testing-bin</FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2> =20
/Testing</FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2> =20
/Testing/Code</FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2> =20
/Testing/Code/Common</FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2> =20
....</FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D777014518-24032003><FONT size=3D2>I am not sure that =
CMake can do=20
this for us...</FONT></SPAN></DIV>
<DIV><SPAN class=3D777014518-24032003></SPAN> </DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D777014518-24032003><FONT =
size=3D2> </DIV></FONT></SPAN>
<DIV class=3DSection1>
<P class=3DMsoNormal><?xml:namespace prefix =3D o ns =3D=20
"urn:schemas-microsoft-com:office:office" /><o:p><FONT=20
size=3D2></FONT></o:p> </P>
<P style=3D"MARGIN: 0in 0in 0pt"><B><SPAN=20
style=3D"COLOR: navy; FONT-FAMILY: 'Comic Sans MS'">Jim =
Miller</SPAN></B>=20
<BR><B><I><SPAN=20
style=3D"FONT-SIZE: 10pt; COLOR: red; FONT-FAMILY: =
Arial">_____________________________________</SPAN></I></B><BR><EM><SPAN=
=20
style=3D"FONT-SIZE: 7.5pt; COLOR: black; FONT-FAMILY: =
Arial">Visualization &=20
Computer Vision</SPAN></EM><I><SPAN=20
style=3D"FONT-SIZE: 7.5pt; COLOR: black; FONT-FAMILY: Arial"><BR><EM>GE =
Research</EM><BR><EM>Bldg. KW, Room C218B</EM><BR><EM>P.O. Box 8, =
Schenectady NY=20
12301</EM><BR><BR></SPAN></I><EM><U><SPAN=20
style=3D"FONT-SIZE: 7.5pt; COLOR: blue"><A=20
href=3D"mailto:millerjv@research.ge.com">millerjv@research.ge.com</A></S=
PAN></U></EM></P>
<P style=3D"MARGIN: 0in 0in 0pt"><EM><U><SPAN=20
style=3D"FONT-SIZE: 7.5pt; COLOR: =
blue">james.miller@research.ge.com</SPAN></U></EM><BR><I><SPAN=20
style=3D"FONT-SIZE: 7.5pt; COLOR: black; FONT-FAMILY: Arial">(518) =
387-4005, Dial=20
Comm: 8*833-4005, </SPAN></I><BR><I><SPAN=20
style=3D"FONT-SIZE: 7.5pt; COLOR: black; FONT-FAMILY: Arial">Cell: =
(518) 505-7065,=20
Fax: (518) 387-6981</SPAN></I> </P>
<P class=3DMsoNormal> <o:p></o:p></P></DIV>
<DIV> </DIV></BODY></HTML>
------_=_NextPart_001_01C2F237.9802F950--