[Paraview-developers] New git repo: reporting changed repo after only pull

Mon May 3 13:20:49 EDT 2010

OK, I finally got a chance to implement this, and it looks alright so far.

This seems like something CTest should simply handle as part of updating a project.  Has anyone submitted a bug (or feature request depending on your POV) to the CMake project?

-Ken

On 4/28/10 8:56 AM, "Dave Partyka" <dave.partyka at kitware.com> wrote:

Hey Ken and Kevin, Here is how to fix ctest_update so that the update step doesn't submit erroneous modified files due to submodule update.

ctest_update only does a pull and then other things to see if there are locally modified files. Because of this, it will always fail in the presence of submodules because after the pull they will report they have been modified until you submodule update. To fix this we have made tiny wrapper scripts that will pull and submodule update in a single step. What you have to do in your dashboard script is configure the appropriate git script (sets the path to git). And then override CTEST_GIT_COMMAND to call that script rather than git directly. Here is what I am doing in paraview_common.

# look for the git mod scripts in the same dir as the script.
get_filename_component(dashboard_self_dir ${CMAKE_CURRENT_LIST_FILE} PATH)

if(UNIX)
  configure_file(${dashboard_self_dir}/gitmod.sh.in <http://gitmod.sh.in>
                 ${CTEST_DASHBOARD_ROOT}/gitmod.sh
                 @ONLY)
  set(CTEST_GIT_COMMAND ${CTEST_DASHBOARD_ROOT}/gitmod.sh)
else()
  configure_file(${dashboard_self_dir}/gitmod.bat.in <http://gitmod.bat.in>
                 ${CTEST_DASHBOARD_ROOT}/gitmod.bat
                 @ONLY)
  set(CTEST_GIT_COMMAND ${CTEST_DASHBOARD_ROOT}/gitmod.bat)
endif()

-------------------------------------------------------------------------------------------------------------------
contents of gitmod.bat.in <http://gitmod.bat.in>

@echo off
set GIT=@CTEST_GIT_COMMAND@
call "%GIT%" %*
if "%1" equ "pull" (
  "%GIT%" submodule update
)

-------------------------------------------------------------------------------------------------------------------
contents of gitmod.sh.in <http://gitmod.sh.in>

#!/bin/sh
GIT="@CTEST_GIT_COMMAND@"
"$GIT" "$@" &&
if test "$1" == "pull"; then
    "$GIT" submodule update
fi

On Thu, Apr 22, 2010 at 5:49 PM, Moreland, Kenneth <kmorel at sandia.gov> wrote:
My script does not do the initial clone.  It updates an existing repository.

The problem is that ctest_update is performing two basic operations: pull from the origin repository (calls git pull) and check that the local repository is consistent (I don't know how, but let's say by calling git status).  The problem is that the local repository won't actually be consistent until the submodules are updated.

Thus, if I call submodule update before ctest_update, I get the following sequence of events.

git submodule update
git pull
check git status

If I call submodule update after calling ctest_update, I get this sequence of events.

git pull
check git status
git submodule update

This is marginally better because now at least the end result is a fully updated repository.  But the problem is that by the time the submodules are updated, the consistency check has already occurred and the erroneous error has already been recorded.

-Ken

On 4/22/10 3:38 PM, "Dave Partyka" <dave.partyka at kitware.com <http://dave.partyka@kitware.com> > wrote:

Sorry you are right, it should be after ctest_update(). What is the reason that ctest_update() is failing for you? Because the submodules don't exist yet?

Does your script do the initial clone?

If so, you're going to want to do something like:

if(NOT EXISTS src_dir)
  clone
  <optionally checkout a branch>
  submodule init
  submodule update
endif()

ctest_update()

submodule update

On Thu, Apr 22, 2010 at 5:18 PM, Moreland, Kenneth <kmorel at sandia.gov <http://kmorel@sandia.gov> > wrote:
How will that work?  If I put it before ctest_update, then it will run before the VTK submodule SHA gets updated.  Thus, it my not grab the commit that you actually need.  Or does the submodule update command implicitly fetch everything from the remote regardless of whether it is needed?

-Ken

On 4/22/10 3:02 PM, "Dave Partyka" <dave.partyka at kitware.com <http://dave.partyka@kitware.com>  <http://dave.partyka@kitware.com> > wrote:

You want to put it before ctest_update is called.

On Thu, Apr 22, 2010 at 3:40 PM, Moreland, Kenneth <kmorel at sandia.gov <http://kmorel@sandia.gov>  <http://kmorel@sandia.gov> > wrote:
I tried adding your execute_process after my call to ctest_update, but I don't think it's quite working.  The problem is that the submodule update command does not get run until after the ctest_update call returns, and by that time CTest has already determined that the update has failed.

Where exactly should I be putting the execute_process?

-Ken

On 4/20/10 7:07 AM, "Dave Partyka" <dave.partyka at kitware.com <http://dave.partyka@kitware.com>  <http://dave.partyka@kitware.com>  <http://dave.partyka@kitware.com> > wrote:

You're going to want to add something like this in your script for initial checkout:

  execute_process(
    COMMAND \"${CTEST_GIT_COMMAND}\" submodule init
    WORKING_DIRECTORY \"${CTEST_SOURCE_DIRECTORY}\"
    )

And for each update you'll want:

  execute_process(
    COMMAND \"${CTEST_GIT_COMMAND}\" submodule update --recursive --
    WORKING_DIRECTORY \"${CTEST_SOURCE_DIRECTORY}\"
    )

Also this will be handy:

# Look for a GIT command-line client.
if(NOT DEFINED CTEST_GIT_COMMAND)
find_program(CTEST_GIT_COMMAND NAMES git git.cmd)
endif()

On Mon, Apr 19, 2010 at 7:16 PM, Dave Partyka <dave.partyka at kitware.com <http://dave.partyka@kitware.com>  <http://dave.partyka@kitware.com>  <http://dave.partyka@kitware.com> > wrote:
I do not know if ctest knows how to update submodules automatically. I have it on my todo list to talk to Brad King about this for the same reason for updating the ParaView dashboards here at Kitware. I'll share what I find out.

On Mon, Apr 19, 2010 at 7:13 PM, Moreland, Kenneth <kmorel at sandia.gov <http://kmorel@sandia.gov>  <http://kmorel@sandia.gov>  <http://kmorel@sandia.gov> > wrote:
Yes.  That seems to fix the problem.  However, is there a way to get ctest to run that after pulling?  Otherwise we'll still get a bunch of errors in the dashboard.

-Ken

On 4/19/10 5:09 PM, "Dave Partyka" <dave.partyka at kitware.com <http://dave.partyka@kitware.com>  <http://dave.partyka@kitware.com>  <http://dave.partyka@kitware.com>  <http://dave.partyka@kitware.com> > wrote:

does 'git submodule update' help?

On Mon, Apr 19, 2010 at 7:02 PM, Moreland, Kenneth <kmorel at sandia.gov <http://kmorel@sandia.gov>  <http://kmorel@sandia.gov>  <http://kmorel@sandia.gov>  <http://kmorel@sandia.gov> > wrote:
I noticed something odd when converting to the new git repositories today.  After only cloning the repository, setting up the modules, and then pulling (using the --rebase flag as described on the Wiki), status reports that I have made a change in my repository even though I have not.  Here is a terminal capture of a newly cloned ParaView repository (after modules and hooks are set up).

kmorel2 0> git status                                /Users/kmorel/src/ParaView
# On branch master
nothing to commit (working directory clean)
kmorel2 1> git pull --rebase                         /Users/kmorel/src/ParaView
remote: Counting objects: 44, done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 26 (delta 20), reused 0 (delta 0)
Unpacking objects: 100% (26/26), done.