|
Notes |
|
|
(0021011)
|
|
Julien Jomier
|
|
2010-06-15 09:30
|
|
Some questions:
* should be some protection from seconds upgrades?
Yes it should, actually CDash shouldn't upgrade a second time, unless the version is not set correctly in the database, which might be the case.
* why compression for newly submitted data not used after first db upgared (did manual compare of submitted data at field "output" of table "test" after 1st and 2nd upgrades)?
Can you explain this?
Are you using CTest to submit to your data? Could you attach the test XML that is causing trouble? and specify the database you are using? |
|
|
|
(0021015)
|
|
Gleb Varaksin
|
|
2010-06-15 10:12
|
|
I simplified Test.xml to represent here.
Yes, I use CTest to submit data.
Sorry, it is not possible to have access to DB because webserver has no access to external net.
Next I could represent several screen pictures |
|
|
|
(0021016)
|
|
Gleb Varaksin
|
|
2010-06-15 10:25
|
|
Attached:
compressed data.png - content of test represented in mysql command line
cdash (db 1.2.1).png - the table "test" structure before any upgrade
table -test- (db 1.6.4) after first and second update.png - the table "test" structure after first upgrade and repeated command after second upgrade of cdash |
|
|
|
(0021017)
|
|
Gleb Varaksin
|
|
2010-06-15 10:39
|
|
Another one attach:
CDASH_USE_COMPRESSION_0.png - I changed flag to $CDASH_USE_COMPRESSION='0'; at cdash/config.php:75 before commit the same test.
Also, testDetails.php displays this test's info without any errors/warnings.
But when $CDASH_USE_COMPRESSION='1' then loadXML problem:
Warning: DOMDocument::loadXML() [domdocument.loadxml]: PCDATA invalid Char value 20 in Entity, line: 18 in C:\wamp\www\CDash\cdash\common.php on line 39
Is another kind of data needed? |
|
|
|
(0021018)
|
|
Julien Jomier
|
|
2010-06-15 10:41
|
|
|
I'm going to look at it now. |
|
|
|
(0021020)
|
|
Gleb Varaksin
|
|
2010-06-15 10:50
|
|
Sorry, I was added another one file "table -test- (db 1.2.1).png" instead of "cdash (db 1.2.1).png".
So, "table -test- (db 1.2.1).png" picture represents the table "test" in 1.2.1 version (before any upgrade) |
|
|
|
(0021023)
|
|
Julien Jomier
|
|
2010-06-15 16:23
|
|
I tried with your Test.xml and compression ON and it works fine. The 'output' field for the 'test' table should be:
output mediumblob BINARY
Let me know if that's not the case. |
|
|
|
(0021025)
|
|
Gleb Varaksin
|
|
2010-06-16 00:57
|
|
Found:
* after first DB upgrade: in table "test" column "output" the data stored in compressed form. Confirmed after clean update
* this report was submitted based on information after second DB upgrade. It is not correct
My suggestion:
- implement some additional protection to eliminate possibility apply second DB upgrade (from 1.6 to 1.6). I think, it could be possible based on table "version" info:
mysql> select * from version;
+-------+-------+-------+
| major | minor | patch |
+-------+-------+-------+
| 1 | 6 | 4 |
+-------+-------+-------+
1 row in set (0.08 sec)
I believe, some protection exist in cdash. But second upgrade happened on my webserver. I don't know why.
Question: if second upgrade of DB happened by some reason, but how user could do restore the state of "output" field of table "test"? It would be good to have some solution of it.
Thanks,
Gleb |
|
|
|
(0021027)
|
|
Julien Jomier
|
|
2010-06-16 08:16
|
|
Gleb,
Thanks for the report. So apparently there was a bug in CDash 1.6.4 that the full upgrade was not done the first time (when upgrading from 1.2). I have attached a cdashUpgrade.js that you should replace in the javascript directory. I'd start from a fresh 1.2 and start the upgrade again.
Also could you look at the source of the backwardCompatibilityTools page (before click on upgrade) and see if you have these lines:
<script type="text/javascript">
var version='1.2';
</script>
I'm wondering if something is going on there to find the version. This might not solve the problem but will be a first step.
Thanks again for the report. |
|
|
|
(0021030)
|
|
Odile Lemay
|
|
2010-06-16 13:59
|
|
I changed the option in the config.php file and I had to ctest the option --no-compress-output
I have the test output in my cdash.
That solved my problem for the moment (ref bug 10829) :-) |
|
|
|
(0021061)
|
|
Julien Jomier
|
|
2010-06-16 19:18
|
|
I was able to reproduce the issue with the multiple upgrades and I fixed it in the 1.6 branch and the main trunk. Thanks for the report.
Just to be clear, the upgrade doesn't compress the data, it just modifies the field to be a binary blob instead of a text field, but the content doesn't change. I'm guessing that the compression is not working on the machine. This is usually the case when the zlib package for php is not installed on the server and $CDASH_USE_COMPRESSION='1'; in the config.local.php. This causes some failure to compress/uncompress the data. Could you make sure that zlib is supported for your php? on way to know is to create a simple php script with
<?php
phpinfo();
?>
and check if ZLib Support is enabled. Let me know. |
|
|
|
(0021062)
|
|
Gleb Varaksin
|
|
2010-06-17 03:08
|
|
Answering to (0021027): looked to the source of backwardCompatibilityTools page in browsers (FireFox 3.6.3, IE 6.0). No any "var version=" string found!
Also, I decided do not apply cdashUpgrade.js attached here, because found the message about fix of source.
(0021061): Good! I will update cdash from 1.6 brunch repository.
Before incident with upgrading I didn't use zlib. During investigation I switched On/Off using zlib, but the problem didn't resolve (second upgrade was applied already).
Now (before update fix 1.6), I am using DB upgraded once and zlib switched ON (phpinfo(): ZLib Support enabled). So, the test's output displays fine on testDetails page and the data is compressed stored in db (check manually by mysql commande line).
OK, I will update cdash and let you know if something wrong, because I have another issues with displaying information on dashboard (I am going to submit new bug report)
Thanks,
Gleb |
|
|
|
(0021063)
|
|
Julien Jomier
|
|
2010-06-17 06:48
|
|
|
|
|
(0021065)
|
|
Odile Lemay
|
|
2010-06-17 10:51
|
|
This morning I check-out the latest version of CDash from your trunk and I installed it. I applied manually without error to the mysql database the cdash-upgrade-1.6-1.8.sql
I tried in the config.php to set and unset the $CDASH_USE_COMPRESSION='1' and ctest with and without --no-compress-output.
ZLib Support is enabled
Compiled Version is 1.2.3
Linked Version is 1.2.3
I still have the errors in the ssl_error_log and nothing displayed. |
|
|
|
(0021066)
|
|
Julien Jomier
|
|
2010-06-17 10:55
|
|
|
What are the errors in the ssl_error_log? |
|
|
|
(0021067)
|
|
Gleb Varaksin
|
|
2010-06-17 12:00
|
|
Found that issue with displaying compressed data still exists.
Previously I used ctest 2.2.3 (default using). So, test output data send as non-compressed (in Test.xml). Zlib support enabled
After run ctest 2.8.1 the same test data was compressed in Test.xml. But errors (Warning: DOMDocument::loadXML() [domdocument.loadxml]: PCDATA invalid Char value 27 in Entity, line: 230 in C:\wamp\www\CDash\cdash\common.php on line 39) reported on testDetails page. Checked with updated cdash version currently available on site (1-6-4) |
|
|
|
(0021069)
|
|
Julien Jomier
|
|
2010-06-17 13:05
|
|
Odile, Gleb,
in the testDetails.php file (around line 167) there is the code to deal with compression. Could you tell me if the
$uncompressedrow = gzuncompress($testRow["output"]);
passed and returns true or false;
If it's true could you send me the value of $uncrompressedrow by adding:
echo $uncompressedrow;
exit();
If it's false, could you send me the output of $testRow["output"]?
Thanks for the help. |
|
|
|
(0021070)
|
|
Odile Lemay
|
|
2010-06-17 13:42
|
|
falsex?????g??/?O?/1???? [??GHr??3?Wr?????
My code is:
else
echo "false"
echo $testRow["output"];
It`s go in the else and $uncompressedrow seems to be empty.
Odile |
|
|
|
(0021071)
|
|
Odile Lemay
|
|
2010-06-17 13:45
|
|
I run the CDashTest from you web site.
Odile |
|
|
|
(0021072)
|
|
Julien Jomier
|
|
2010-06-17 14:08
|
|
|
Are you running CDash on my.cdash.org? the project CDashTest? |
|
|
|
(0021073)
|
|
Odile Lemay
|
|
2010-06-17 14:13
|
|
your CDashTest configure to write to my CDash.
It should write "Running Test1" and return 0. |
|
|
|
(0021074)
|
|
Julien Jomier
|
|
2010-06-17 14:25
|
|
|
I see, are you running CTest with compression on? |
|
|
|
(0021075)
|
|
Julien Jomier
|
|
2010-06-17 14:30
|
|
Ok something is going on with the gzuncompress could you remove the '@' from this line:
@$uncompressedrow = gzuncompress($testRow["output"]);
-> $uncompressedrow = gzuncompress($testRow["output"]);
and reload the testDetails.php page and see if that works? |
|
|
|
(0021076)
|
|
Odile Lemay
|
|
2010-06-17 14:49
|
|
I removed the @ from the testDetails.php
I run some of my tests.
$CDASH_USE_COMPRESSION 1 false with some caracteres
$CDASH_USE_COMPRESSION 0 nothing (no true)
with option --no-compression-output
$CDASH_USE_COMPRESSION 1 false with some caracteres
$CDASH_USE_COMPRESSION 0 nothing. (no true) |
|
|
|
(0021077)
|
|
Gleb Varaksin
|
|
2010-06-17 14:52
|
|
OK, got it.
This is what I am really wanted to send:
ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-=~!@#$%^&*()_+
Some with control symbols: 's;1m%s' - THIS IS TEMPLATE FOR COLOR
Some with control symbols: GREEN
Some with control symbols: RED
Some with control symbols: BLUE
ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-=~!@#$%^&*()_+
...there are many control symbols around GREEN,RED,BLUE words. Just not displayed here because I used copy from linux screen. My idea is to show the problem around control symbols
This is what really sent in Test.xml (cut):
<Value encoding="base64" compression="gzip">eJxzdHJ2cXVz9/D08vbx9fMPCAwKDgkNC4+IjDIwNDI2MTUzt7DUta1TdFBWUY1T09LQjNfmCs7PTVUozyzJUEjOzyspy
s9RKK7MTcrPKbZSUJeOVi22NsxVLZaONshVV9BVCPHwDFYAohBX3wAfxxBXBTf/IAVnfx//IHzmSEcbGwGN4XQPcnX1AxlFQLEhSHGQqwsRSk1ASp18Ql3Bah3J8D8A67ZaKQ==</Value>
This is displayed on dashboard:
ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-=~!@#$%^&*()_+ Some with control symbols: '[%s;1m%s[0m' - THIS IS TEMPLATE FOR COLOR Some with control symbols: [32;1m GREEN[0m Some with control symbols: [31;1m RED[0m Some with control symbols: [34;1m BLUE[0m ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-=~!@#$%^&*()_+
As you could see, something like '' influence to output. This is cause of corrupting output, I think. The same behaviour was always when I tried to debug output by command "echo $uncompressedrow".
May be it helped |
|
|
|
(0021078)
|
|
Gleb Varaksin
|
|
2010-06-17 14:58
|
|
I forgot to say: if the text not have any control symbols (just the text) then all output compress OK and displays OK.
Also, found that ctest 2.2.3 doesn't compress data by default, probably. This is reason why I did not met the problem with displaying data before. |
|
|
|
(0021079)
|
|
Gleb Varaksin
|
|
2010-06-17 15:03
|
|
To be more correct, this is the python commands which I used for test:
print "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-=~!@#$%^&*()_+"
print "Some with control symbols: '\x1b[%s;1m%s\x1b[0m' - THIS IS TEMPLATE FOR COLOR"
print "Some with control symbols: \x1b[32;1m\tGREEN\x1b[0m"
print "Some with control symbols: \x1b[31;1m\tRED\x1b[0m"
print "Some with control symbols: \x1b[34;1m\tBLUE\x1b[0m"
print "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-=~!@#$%^&*()_+" |
|
|
|
(0021080)
|
|
Julien Jomier
|
|
2010-06-17 15:22
|
|
|
Compression has been added only recently in CTest. Could you make sure that your databases are encoding as UTF_8? |
|
|
|
(0021081)
|
|
Gleb Varaksin
|
|
2010-06-17 16:04
|
|
mysql> SHOW VARIABLES LIKE 'character_set_system';
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| character_set_system | utf8 |
+----------------------+-------+
1 row in set (0.01 sec) |
|
|
|
(0021082)
|
|
Odile Lemay
|
|
2010-06-17 16:16
|
|
Is it possible to make it work without compression, like it was in the old version of cdash.
I was using ctest provided with cmake 2.8.1 for a while, on Linux, Mac and Windows, without problem, before installing the latest version of CDash on Fedore 12.
Are you able to run CDash on Fedora Core 12, with our version of mysql ?
Which version of mysql, php, are you running ?
My cdash is installed on an user account not as root, yours ?
For sure there is differences between our system !! |
|
|
|
(0021084)
|
|
Odile Lemay
|
|
2010-06-17 16:42
|
|
|
yes the database is encoded as UTF_8 |
|
|
|
(0021111)
|
|
Gleb Varaksin
|
|
2010-06-22 02:46
|
|
I did check 1.6 brunch (used many variations of pressing “Upgrade CDash”/refresh page/different initial DB upgrade state).
I. Mainly, the behavior of upgrading cdash left the same with small changes - it takes 2 steps minimum:
Current CDash database schema:1.4
. . . .
Upgrading tables:
Applying 1.6 patches:
CDash Upgrade Successful.
If do not refresh the page Maintenance(backwardCompatibilityTools.php) on browser and press button “Upgrade CDash” again:
Current CDash database schema:1.6
. . . . .
Upgrading tables:
Applying 0.8 patches:
Applying 1.0 patches:
Applying 1.2 patches:
Applying 1.4 patches:
CDash Upgrade Successful.
It really does something during some time. But last action could be repeated many times while not refresh the page. Of course, upgradings (0.8-1.4) goes quickly (probably do nothing).
Only after refresh page (or to enter to this page), after press "Upgrade" do display "Your installation is already up to date".
II. Once found if to refresh page after first upgrade then "Your installation is already up to date" appeared. So no upgrade 0.8-1.4 available any more.
III. 1.6.4 is my work version, so I did upgrade one time. I replaced 1.6.4 by updated 1.6. After, I applied upgrade. It was look like to upgrade 0.8-1.4.
So, questions:
1. Does it mean than upgrade 0.8-1.4 is not important for case I (if it is not avaliable in case II)?
3. Does some corruption in somewhere, if user applies second upgrade? Is it critical?
2. (based on case III) Does it mean that upgrade using updated 1.6 needed in any case (even if user did use 1.6.4 for long time)?
NOTE: "var version=" string appears on backwardCompatibilityTools page after pressing button "Upgrade CDash". Not before.
Thanks,
Gleb |
|
|
|
(0021112)
|
|
Odile Lemay
|
|
2010-06-22 07:25
|
|
I am out of the office for the next 2 weeks.
Gleb are you able to see your output data now ?
Before leaving I had a look to the .sql script.
Julien, is it important to have administrator privilege ?
In my case, I am cdash user but whithout all privileges.
I can not rename table for exemple.
Odile |
|
|
|
(0021113)
|
|
Gleb Varaksin
|
|
2010-06-22 08:12
|
|
I still no able to see the output of those tests which contained special escape characters (like \x1b). But no problem if the text has no any escape characters
Also, when I used ctest 2.2.3 (means no compression used) the problem sequential of symbols was converted to 1-byte symbol. So, "\x1b" was converted to 0x1b (027 dec). Probably, no convertion of escape characters does before compression in ctest 1.8.3. It means, that after decompression output, the escape characters goes to loadXML() which corrupts from it. I suppose, must be some php functions could possible to convert the output after decompression success by gzuncompress(). But I didn't investigate.
Maybe the text must be checked for unicode. I don't know
Gleb |
|
|
|
(0021118)
|
|
Julien Jomier
|
|
2010-06-22 09:15
|
|
|
Odile, Gleb, can we move this discussion via email? julien.jomier [at] kitware .com. Thanks. |
|
|
|
(0021237)
|
|
simonh
|
|
2010-07-05 06:58
|
|
|
Is there any progress on this or 0010829. I've exactly the same problems. Thanks. |
|
|
|
(0021375)
|
|
Odile Lemay
|
|
2010-07-13 08:52
|
|
I had a look to the php files, the sql tables and scripts.
This is what I found and did to solve my cdash problem.
- In order to run the sql scripts for update without errors, the cdash database user should have the privilege to DELETE and ALTER a table.
- In the test table, the output field in my database was "mediumtext" text and should be "mediumblob" binary.
- When I upgrade my database, I pressed the compress button from the cdash maintenance, that corrupt my database, by replacing the ascii text, by encoded, compress binary array.
Because, space is not a problem for us, I decided to re-installed our backup database, I update the database using the functionality provided in the cdash bord. But I didn`t compress.
I modified the .php scripts to be sure that nothing will be encoded and compressed ever. I add to the ctest command line --no-compress-output, and I modified the configuration files.
Have fun :-)
Odile Lemay |
|
Test.xml (1,367) 2010-06-15 10:06
compressed data.png (7,786) 2010-06-15 10:17
cdashUpgrade.js (2,970) 2010-06-16 08:11