[Dart] RE: Shared source builds

Miller, James V (CRD) millerjv at crd.ge.com
Thu, 31 Jan 2002 10:37:15 -0500


Amitha,

I rolled your changes to my local version of Dart to check them out.
They seem to work. :)

About the only change I have made is that in one of your scripts
you had

set Dart(SharedSourcePrimaryBuildBase) ""

to avoid breaking builds where CMake not been run to set this variable in 
DartConfiguration.tcl. What I did instead was put a line in Defaults.conf

SharedSourcePrimaryBuildBase:

which should cause the variable to exist.  I added the CMake variable to 
the Dart CMake module (but have not checked anything in yet).

I have a couple of concerns...

First, I'd like to get away from having a LastUpdate-Model.xml file. 
My concern is that a shared build may grab an out of date LastUpdate-Model.xml
file.  Instead of writing a LastUpdate-Model.xml file in the primary build
tree, couldn't the secondary builds grab the "actual" Update.xml from the
primary build?

For nighly builds, the secondary builds should be able to trace down the 
primary build tree to grab the appropriate Update.xml since the datestamp
for the primary build is identical to the datestamp for the primary build.
The only trick will be get the Site and BuildName from primary build
(but this information exists in the primary builds DartConfiguration.tcl file).

For continuous and experimental builds, we should be able to do something 
akin to GetLastBuildDirectory() put point it to the primary build tree instead
of the secondary build tree.

Another option for avoiding a LastUpdate-Model.xml file is to just have the secondary
builds tell the Dart server who the primary build was (in terms of Site, BuildName,
and BuildStamp).  Secondary clients would not have an Update.xml file but the 
server will create a link (href) to the primary build's Update.html file.

My goal, however, is to figure out whether or not it make sense for Dart
to "pause" secondary builds until the primary build has finished creating
Update.xml.  Currently, we finesse this problem by setting up our client
scripts that start Dart builds such that the secondary builds do not attempt 
to do anything until the primary build has generated Update.xml. We can continue
along this path but we have an opportunity to simplify client setup.  When 
we have attempted things like this in the past (having one system wait until
another system wrote a file), it proved to be the weak link in the process.
We would constantly have to go to some directory and remove a "lock" file
by hand or touch a file, etc. 

The other motivation is specific for Continuous builds.  Currently, if
a Continous build/test cycle is specified in one command 
"Continuous Start Update Build Test Submit", Dart will exit if Update 
returns no changes. It would be nice to have both primary and secondary
continuous builds take advantage of this (meaning that we can't finesse 
the problem by having the client scripts that drive Dart manage the 
update process).  One option is have Update write out a simple file indicating
whether anything changed. Thus, if continuous builds are broken into 
multiple commands

Continuous Start Update
Continuous Build Test Submit

all command that are not Start and Update will check wether an Update did anything
and do nothing if nothing is changed.





-----Original Message-----
From: Amitha Perera [mailto:perera at cs.rpi.edu]
Sent: Monday, January 28, 2002 11:50 AM
To: Miller, James V (CRD)
Cc: Blezek, Daniel J (CRD)
Subject: Re: Shared source builds


Okay, thanks. :-)

For what it's worth, I've attached the "latest" changes to
DashboardManager.tcl and to Update.tcl. (I've moved much of the work
into Update.tcl, since I think that's where the work should be done.)

Also, note that the very first "if" check in the current sources for
Update is incorrect. (It looks for $BuildStampDir/Update.xml instead
of $BuildStampDir/XML/Update.xml.)

Amitha.



DashboardManager.tcl's Update code:

    Update \
    {
      # If we don't have an existing update, make one
      if { ![file exists [file join $Dart(BuildDirectory) $BuildStampDir XML Update.xml]] } \
      {
        puts "\tBuilding Update.xml"
        set lastUpdateFile [file join $TestingBaseDir
        "LastUpdate-${Model}.xml"]
        if { $Dart(SharedSourcePrimaryBuildBase) == "" } \
        {
          set Status [catch { Update $Model [file join
          $Dart(BuildDirectory) $BuildStampDir XML Update.xml]
          $BuildStampDir Client Primary [file join
          $Dart(BuildDirectory) $lastUpdateFile] } updateStatus]
        } \
        else \
        {
          set Status [catch { Update $Model [file join
          $Dart(BuildDirectory) $BuildStampDir XML Update.xml]
          $BuildStampDir Client Shared [file join
          $Dart(SharedSourcePrimaryBuildBase) $lastUpdateFile ] }
          updateStatus]
        }
        if { $Status } {
          puts $updateStatus
          puts $errorInfo
        }

        # If model is Continuous and no code changes are detected,
        then skip
        # the rest of the commands
        if { $Model == "Continuous" } {
            if {$updateStatus == 1} {
                puts "\tNo files changed, skipping remaining
                commands."
                exit 1
            }
        }
      }
    }



cvs diff of Update.tcl:

Index: Update.tcl
===================================================================
RCS file: /Dart/cvsroot/Dart/Source/Client/Update.tcl,v
retrieving revision 1.36
diff -r1.36 Update.tcl
214c214,221
< proc Update { Model OutFile BuildStampDir ClientServer } {
---
> # args captures the optional arguments used to specify a shared source
> # build. For a shared source build, there will be two additional
> # arguments. The first will be "Primary" or "Shared", indicating if
> # this is the primary build or a shared (or secondary) build. The
> # second will be the path to the primary LastUpdate xml file. (This
> # file will be created by the primary build and read by the shared
> # builds.)
> proc Update { Model OutFile BuildStampDir ClientServer args } {
221a229,234
>   if {[llength $args] < 1} {
>     set sharedSource "No"
>   } else {
>     set sharedSource [lindex $args 0]
>     set lastUpdateFile [lindex $args 1]
>   }
244a258,284
>   # Check if this is a shared source build. If so, copy the update data from
>   # the primary build directory.
>   if { $sharedSource == "Shared" } {
>     puts "\tSharing $lastUpdateFile"
>     if { [file exists $lastUpdateFile] } {
>       set fin [open $lastUpdateFile r]
>       set contents [read $fin]
>       close $fin
>       regsub "<Site>.+?</Site>" $contents "<Site>$Dart(Site)</Site>" contents
>       regsub "<BuildName>.+?</BuildName>" $contents "<BuildName>$Dart(BuildName)</BuildName>"
contents
>       regsub "<BuildStamp>.+?</BuildStamp>" $contents "<BuildStamp>$BuildStamp</BuildStamp>"
contents
>       set fout [open $OutFile w]
>       puts $fout $contents
>       close $fout
>       return 0
>     } else {
>       puts "\t\tIt doesn't exist. Assuming last update had no changes."
>       return 1
>     }
>   }
> 
>   # Otherwise, proceed with the actual update.
>   if { $sharedSource != "No" } {
>     file delete $lastUpdateFile
>   } 
> 
> 
458a499,502
>     # If this is the primary build of a shared source tree, write the last update log
>     if { $sharedSource == "Primary" } {
>       file copy -force $OutFile $lastUpdateFile
>     }