|
Notes |
|
|
(0012797)
|
|
Tom Vercauteren
|
|
2008-07-22 08:45
|
|
|
|
|
(0013168)
|
|
Nicolas Savoire
|
|
2008-08-25 07:40
|
|
I have a uploaded a patch (patch_cmake_bzr_support.diff) which adds basic bzr support to ctest.
I am no entirely satisified since it duplicates a lot of code of cvs/svn update handling.
Moreover I had to resort to workarounds such as the update works the same way as cvs/svn. I suppose the source dir is a branch or a checkout of a master bzr branch (I don't think my modifications handle lightweight checkouts). bzr, through the command pull, supports the update of the local repsoitory to a specific date, nevertheless it doesn't support timezone in dates and the given date is first converted to a revision number which is pulled from the parent branch, since the revision number is computed from the local copy, the wrong revision number is compued, moreover the computed revision number corresponds to the first revision commited AFTER the given date.
To work around these issues, first a 'bzr info' command is executed to retrieve the checkout or parent branch (denoted parent_branch), then a 'bzr revno' to get the current revision (denoted cur_rev), then 'bzr log -r cur_rev.. parent_branch' allows to retrieve the log of the parent branch since revison cur_rev and to determine which revision must be pulled accordingly to the given date (let's call this revision today_rev). Then 'bzr pull -r today_rev parent_branch' is called to update the repository to the wanted revision, the output is parsed to build the list of modified files. Finally for each modified file, similarly to svn update handling, 'bzr log -r cur_rev..today_rev --xml file' command is run to retrieve modification infromation for each file. bzr xml-output must be installed (either version 0.4.5 or latest trunk version because version 0.5.0 has a bug which suppresses some needed output). |
|
|
|
(0013171)
|
|
Brad King
|
|
2008-08-25 09:48
|
|
|
I haven't looked at bzr, but I assume it supports global revisions like svn instead of per-file revisions. The main problem with the current CTest update and change-report model is that it was written for CVS's per-file revisions. We need to get some CDash work done to support a new XML spec that has project-wide revisions (atomic commits). Then we plan to overhaul the CTest update stuff to support this more modern update model. |
|
|
|
(0016136)
|
|
Tom Vercauteren
|
|
2009-04-22 11:19
|
|
|
I have uploaded an initial patch for bzr support that uses the new VCS code in the CVS version of cmake. It basically converts the patch by nicos to the current CMake code. |
|
|
|
(0016137)
|
|
Brad King
|
|
2009-04-22 11:53
|
|
Thanks, Tom!
Actually just this morning I committed git support. See issue 0006994. Part of the changes refactored cmCTestSVN to split out the parts of it useful to any globally-versioned tool. There is a new class cmCTestGlobalVC from which both cmCTestSVN and cmCTestGIT derive. It should help reduce the size of the code for cmCTestBZR and avoid duplication. Do you mind updating the patch accordingly? |
|
|
|
(0016138)
|
|
Tom Vercauteren
|
|
2009-04-22 11:58
|
|
Actually I am monitoring issue 0006994 which is why I submitted the patch today :)
My latest patch (cmake-bzr-2009-04-22.patch) should be up to date. It implements cmCTestBZR that derives from cmCTestGlobalVC. |
|
|
|
(0016139)
|
|
Tom Vercauteren
|
|
2009-04-22 12:01
|
|
Argh sorry, I didn't see the Global in "cmCTestGlobalVC"... Wrong copy-paste
So I guess a new version of the patch would be helpful. I'll see if I can get some time (maybe next week) for that. |
|
|
|
(0016141)
|
|
Tom Vercauteren
|
|
2009-04-22 12:48
|
|
Uploaded a new patch (cmake-bzr-2009-04-22-bis.patch) that really makes cmCTestBZR derive from cmCTestGlobalVC and not directly cmCTestVC anymore.
I should also mention that this patch requires the bzr xmloutput plugin (e.g. version 0.8.3):
https://launchpad.net/bzr-xmloutput [^] |
|
|
|
(0016143)
|
|
Brad King
|
|
2009-04-22 13:20
|
|
Thanks. That looks pretty close. We also need a test.
Would you also please copy one of these files:
Tests/CTestUpdateCVS.cmake.in
Tests/CTestUpdateGIT.cmake.in
Tests/CTestUpdateSVN.cmake.in
to Tests/CTestUpdateBZR.cmake.in and modify it accordingly?
These tests create a simple repository and some working trees and check that CTest reports changes correctly. |
|
|
|
(0016151)
|
|
Tom Vercauteren
|
|
2009-04-23 06:27
|
|
|
Today's patch (cmake-bzr-2009-04-23.patch) adds such a unit test and simplifies the implementation of cmCTestBZR. In the previous implementations, we missed the -v option in bzr log and therefore had to find a workaround to get a correct list of modified files. |
|
|
|
(0016153)
|
|
Brad King
|
|
2009-04-23 09:03
|
|
Great, thanks! The test passes for me :)
Your simplifications since the previous patches also look good.
The remaining issue is the nightly start time support (we can work out strptime if necessary, but there is a deeper problem). Please look at the discussion in issue 0006994. We cannot reliably use the most recent revision that occurs before the start time. Someone could publish new revisions that were committed before the nightly start time and cause different dashboard clients to test different versions.
This is a problem for all distributed VCS tools. Some central location must choose and mark a single revision for testing on a given day. Typically this should be the revision published as of the nightly start time. The solution I propose in 0006994 is somewhat git-specific. Do you have any ideas how it might work for bzr? Ideally all information should be passed through the VCS tool protocols, but as a last resort we could use another protocol like http GET. |
|
|
|
(0016157)
|
|
Tom Vercauteren
|
|
2009-04-23 10:25
|
|
I am aware of the limitation you presented in 0006994. The current solution is however sufficiently reliable for our uses which is why we didn't bother adding an additional server-side cron.
I think that the git solution you propose could be extended to bzr by using tags, e.g. the server would start by running
bzr tag --force ctest-nightly-tag
and the clients would simply need to pull this tag
bzr pull -r tag:ctest-nightly-tag ${BZR_SERVER_URL}
By reusing the same tag name every night (and thus the --force option), the repository won't be "polluted" by a multiplicity of tags.
To get a more robust behavior that also works without the server-side cron, the client within cmake could do the following:
- call "bzr tags" to check if the "ctest-nightly-tag" tag actually exists
- If ctest-nightly-tag" exists, call "bzr log -r tag:ctest-nightly-tag ${BZR_SERVER_URL}" to get the date that corresponds to "ctest-nightly-tag"
- If "ctest-nightly-tag" does not exists or is more than 24 hours old cmake would use the behavior proposed in cmake-bzr-2009-04-23.patch. Otherwise, the client would use the above procedure.
Thoughts? |
|
|
|
(0016159)
|
|
Brad King
|
|
2009-04-23 10:54
|
|
Something like that could work, but the tag would have be branch-specific.
Perhaps it is better to have CTest simply update using the commit time heuristic by default, but then define an interface a project can use to override the default update. CTest would still check the work tree revision before and after the update to report changes, but the actual 'pull' operation could be customized with a user script. This would allow each project to do its own thing.
What is probably more important is that CTest submits the revision identifier to CDash, and that CDash reports it. It will take a little work on my end but is not too hard. This is not something that affects your patch though. |
|
|
|
(0016160)
|
|
Brad King
|
|
2009-04-23 10:56
|
|
I was just reading some bzr documentation and thought of another way around the strptime thing. In order to identify the latest revision before the nightly start time, might this work?
bzr log -r $oldrev..before:date:2009-04-22,23:00:00 $url
Then we do not need to parse dates at all. |
|
|
|
(0016161)
|
|
Tom Vercauteren
|
|
2009-04-23 10:59
|
|
That makes sense.
As a side note, bzr tags are branch specific. The actual branch that needs tagging can be specified by the cron job with the -d option (-d ARG, --directory=ARG Branch in which to place the tag). |
|
|
|
(0016162)
|
|
Brad King
|
|
2009-04-23 11:01
|
|
|
If we want CTest to report the exact version tested for a bzr project, how might we get a unique revision number? Does it have to be a combination of the upstream url and its revision number? Since bzr pull does not work for diverged branches, is it guaranteed that the local and remote revisions match? |
|
|
|
(0016163)
|
|
Tom Vercauteren
|
|
2009-04-23 11:10
|
|
Regarding http://www.cmake.org/Bug/view.php?id=6857#c16160 [^]
According to nicos (http://www.cmake.org/Bug/view.php?id=6857#c13168 [^]) "bzr, through the command pull, supports the update of the local repsoitory to a specific date, nevertheless it doesn't support timezone in dates"
Things might have changed since then but I didn't recheck...
Regarding http://www.cmake.org/Bug/view.php?id=6857#c16162 [^]
Yes indeed to be able to get the exact revision that has been used, we need the upstream url and its revision number.
The current heuristic is to get the upstream url by parsing the output of "bzr info". Another option would be to have the upstream url passed has a parameter, or use a combination of both: use the parameter if it is defined and fallback to the "bzr info" parsing if it is not provided. |
|
|
|
(0016166)
|
|
Brad King
|
|
2009-04-23 12:03
|
|
It looks like the date is transformed using the local time zone. For example, I'm in '-0400'. I run
bzr log -r 'date:2009-03-23,11:52:18' http://bazaar-vcs.org/bzr/bzr.dev [^]
and the revision I see is dated 'Mon 2009-03-23 15:52:19 +0000' (note my 11 became 15 in the commit timestamp). If I change the time to '11:52:20' I get a different revision.
We could probably compute the nightly start time and then print to a string in the local time zone to get the proper date. However, the 'before:date:...' trick will not work if no commit after the nightly time exists. I guess this leads to nicos' conclusion.
Regarding the current patch, it seems to use the local OldRevision as the base of the remote log. Might this mismatch with the upstream revisions? |
|
|
|
(0016167)
|
|
Tom Vercauteren
|
|
2009-04-23 13:05
|
|
Right... this approach is far from being bullet-proof. However I don't see a right way to handle such cases.
There are many possible failure scenarios:
- The local sources are neither a branch or a checkout of a master bzr branch
- The local sources are already to a higher revision than the requested nightly revision*
- The local sources have local changes that have been locally committed
- etc.
Anyhow concerning the exact question "Regarding the current patch, it seems to use the local OldRevision as the base of the remote log. Might this mismatch with the upstream revisions?" A mismatch can happen if local changes have been committed locally but then the branches will have diverged and "bzr pull -r ${NewRevision} ${BZR_URL}" will do nothing and exit with an error code. So even if the log might be silently but wrongly interpreted, the update won't happen.
Hop this helps.
* bzr pull won't downgrade a revision without the --overwrite option but using --overwrite is a VERY dangerous operation since if the local sources are bound to the master branch (i.e. it is a checkout) bzr pull --overwrite to an old revision will also downgrade the master branch. This implies that all commits that happened after the old revision are lost on the server! |
|
|
|
(0016168)
|
|
Brad King
|
|
2009-04-23 13:35
|
|
|
I don't care much about use cases where development (with commits) happens in a tree managed a by CTest dashboard script. Nightly builds should be pristine copies of the upstream, so revision numbers should always match (right?). |
|
|
|
(0016169)
|
|
Tom Vercauteren
|
|
2009-04-23 13:46
|
|
As far as I understand it, yes. If the sources for Nightly builds are pristine copies of the upstream as they should, revision numbers should match.
This only requires that the same sources are not used for example for both continuous and nightly builds (due to the --overwrite restriction).
By the way, I realized that the time reported by the log command can be requested in UTC time
bzr log --xml --timezone utc -v -r ${REV_STR} ${BZR_URL}
Note however that this timezone information does not seem to change the way that 'before:date:...' gets interpreted. |
|
|
|
(0016433)
|
|
Brad King
|
|
2009-05-14 10:14
|
|
I've thought quite a bit about the nightly start time issue. The long-term solution is probably to change the model that CTest/CDash use to agree on a version to test. Since CTest clients must contact CDash after doing a build to submit results anyway, there is no reason not to contact it before starting to ask what version to test. This will also facilitate possible future 'test-on-demand' features.
In the short-term I'd like to commit bzr support without addressing the nightly start time problem at all. This will help get it tested and give it some exposure.
Can you please split your patch into two parts? The first part should bring bzr support to where git support is now...just always update to the latest revision from upstream (and a TODO comment for nightly mode). The second part can retain your nightly mode prototype for reference in this issue, but I won't commit it yet. This division also avoids the strptime problem for now. Thanks. |
|
|
|
(0016441)
|
|
Tom Vercauteren
|
|
2009-05-14 13:49
|
|
Sounds great! I have uploaded the updated complete patch as well as the slimmed one (without the nightly part).
Although I am confident that the slimmed patch should work, as a disclaimer, I have to mention that I only tested it through the unit test. The complete patch however is used on a daily basis.
For reference, I'll post another patch that contains only the nightly revision guessing part once you commit BZR support. It's easier that generating a patch for a patch.
Regarding your proposed long-term solution, I makes a lot of sense to me. The only thing is that the transition will be a little more complex since it requires a synchronized move of cmake and cdash. But I guess that this can be sorted out without too much hassle. |
|
|
|
(0016450)
|
|
Brad King
|
|
2009-05-14 16:00
|
|
Thanks!
This new hunk may be a problem though:
+ // bzr 1.14.1 on linux 64 bits crashes when the locale is set to C
+ // So let's make sure it doesn't crash by explicitely putting it to en_EN
+ cmSystemTools::PutEnv("LC_ALL=en_EN");
The 'en_EN' locale is not guaranteed to be available everywhere. Only 'C' is guaranteed to exist. I'm going to commit it without this part. Can you create a more specific test, please? |
|
|
|
(0016452)
|
|
Brad King
|
|
2009-05-14 16:14
|
|
ENH: Teach CTest to handle Bazaar repositories
/cvsroot/CMake/CMake/Modules/CTest.cmake,v <-- Modules/CTest.cmake
new revision: 1.16; previous revision: 1.15
/cvsroot/CMake/CMake/Source/CMakeLists.txt,v <-- Source/CMakeLists.txt
new revision: 1.416; previous revision: 1.415
/cvsroot/CMake/CMake/Source/CTest/cmCTestBZR.cxx,v <-- Source/CTest/cmCTestBZR.cxx
initial revision: 1.1
/cvsroot/CMake/CMake/Source/CTest/cmCTestBZR.h,v <-- Source/CTest/cmCTestBZR.h
initial revision: 1.1
/cvsroot/CMake/CMake/Source/CTest/cmCTestUpdateCommand.cxx,v <-- Source/CTest/cmCTestUpdateCommand.cxx
new revision: 1.17; previous revision: 1.16
/cvsroot/CMake/CMake/Source/CTest/cmCTestUpdateHandler.cxx,v <-- Source/CTest/cmCTestUpdateHandler.cxx
new revision: 1.66; previous revision: 1.65
/cvsroot/CMake/CMake/Source/CTest/cmCTestUpdateHandler.h,v <-- Source/CTest/cmCTestUpdateHandler.h
new revision: 1.13; previous revision: 1.12
/cvsroot/CMake/CMake/Tests/CMakeLists.txt,v <-- Tests/CMakeLists.txt
new revision: 1.86; previous revision: 1.85
/cvsroot/CMake/CMake/Tests/CTestUpdateBZR.cmake.in,v <-- Tests/CTestUpdateBZR.cmake.in
initial revision: 1.1 |
|
|
|
(0016473)
|
|
Tom Vercauteren
|
|
2009-05-15 09:41
|
|
Actually, the comment on bzr 1.14.1 linux 64 bits crashes was just plain wrong. There were indeed some problems when the locale was set to C but they were due to the xml parser of ctest not knowing the encoding used by the bzr xml output plugin.
The attached patch (cmake-bzr-clocale-2009-05-15.patch) fixes the issue with the C locale and adds a unit test that was previously failing when the nightly time revision guessing was also used. |
|
|
|
(0016483)
|
|
Brad King
|
|
2009-05-15 14:58
|
|
I can't quickly reproduce the locale problem, but there may be a better fix.
CTest is using a built-in expat 1.95.2. Perhaps expat 2.0.1 can recognize the "ANSI_X3.4-1968" locale. Try building CMake with the 'CMAKE_USE_SYSTEM_EXPAT' option on, and point it at a system expat 2.0.1. Then see if it can handle the locale without your patch. If so, I'll be happy to update CMake's built-in expat version. Thanks. |
|
|
|
(0016510)
|
|
Tom Vercauteren
|
|
2009-05-18 07:44
|
|
Just tried expat 2.0.1. The "ANSI_X3.4-1968" unknown encoding problem is still there.
Also a patch is required for cmake to compile with expat 2.0.1. On line 231 of cmXMLParser.cxx:
XML_Parser* parser = static_cast<XML_Parser*>(this->Parser);
should be replaced by
XML_Parser parser = static_cast<XML_Parser>(this->Parser);
Except from that all unit tests pass on my machine with expat 2.0.1. |
|
|
|
(0016519)
|
|
Brad King
|
|
2009-05-18 10:35
|
|
Thanks. In that case it is not worth updating the expat version right now.
I've committed cmake-bzr-clocale-2009-05-15.patch:
BUG: Parse more bzr xml output encodings
/cvsroot/CMake/CMake/Source/CTest/cmCTestBZR.cxx,v <-- Source/CTest/cmCTestBZR.cxx
new revision: 1.2; previous revision: 1.1
/cvsroot/CMake/CMake/Tests/CMakeLists.txt,v <-- Tests/CMakeLists.txt
new revision: 1.87; previous revision: 1.86 |
|
|
|
(0018927)
|
|
Brad King
|
|
2009-12-16 14:28
|
|
|
CTest 2.8.0 supports BZR. See issue 0007541 for updates on the nightly start time issue (which affects all DVCS tools). |
|
patch_cmake_bzr_support.diff (37,046) 2008-08-25 07:23