From zack.galbreath at kitware.com Tue Feb 21 09:57:31 2017 From: zack.galbreath at kitware.com (Zack Galbreath) Date: Tue, 21 Feb 2017 09:57:31 -0500 Subject: [CDash] open.cdash.org updated Message-ID: Hi folks, I just upgraded open.cdash.org. The most visible change you'll notice is that the Update column now shows the commit that was built rather than the number of files that changed. Going forward, I've set a reminder for myself to upgrade open.cdash.org once a month around this time (the 21st). -------------- next part -------------- An HTML attachment was scrubbed... URL: From nilsgladitz at gmail.com Tue Feb 21 10:03:55 2017 From: nilsgladitz at gmail.com (Nils Gladitz) Date: Tue, 21 Feb 2017 16:03:55 +0100 Subject: [CDash] open.cdash.org updated In-Reply-To: References: Message-ID: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> On 02/21/2017 03:57 PM, Zack Galbreath wrote: > Hi folks, > > I just upgraded open.cdash.org . The most > visible change you'll notice is that the Update column now shows the > commit that was built rather than the number of files that changed. > > Going forward, I've set a reminder for myself to upgrade > open.cdash.org once a month around this time > (the 21st). Could the prebuilt branch be updated along with it? Nils -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.jackson at bluequartz.net Tue Feb 21 10:19:57 2017 From: mike.jackson at bluequartz.net (Michael Jackson) Date: Tue, 21 Feb 2017 10:19:57 -0500 Subject: [CDash] Slack Integration Message-ID: <58AC5A9D.5060700@bluequartz.net> Has anyone thought about allowing users to send build notifications to a slack channel instead of email? -- Michael A. Jackson 400 S. Pioneer Blvd Owner, President Springboro, Ohio 45066 BlueQuartz Software, LLC EMail: mike.jackson at bluequartz.net Voice: 937-790-1601 Web: http://www.bluequartz.net Fax: 937-746-0783 From zack.galbreath at kitware.com Tue Feb 21 12:06:17 2017 From: zack.galbreath at kitware.com (Zack Galbreath) Date: Tue, 21 Feb 2017 12:06:17 -0500 Subject: [CDash] open.cdash.org updated In-Reply-To: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> References: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> Message-ID: On Tue, Feb 21, 2017 at 10:03 AM, Nils Gladitz wrote: > Could the prebuilt branch be updated along with it? > Good idea! I just updated the prebuilt branch too. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nilsgladitz at gmail.com Tue Feb 21 12:07:49 2017 From: nilsgladitz at gmail.com (Nils Gladitz) Date: Tue, 21 Feb 2017 18:07:49 +0100 Subject: [CDash] open.cdash.org updated In-Reply-To: References: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> Message-ID: <63f33efd-261a-9a55-a069-df85fad132d2@gmail.com> On 21.02.2017 18:06, Zack Galbreath wrote: > On Tue, Feb 21, 2017 at 10:03 AM, Nils Gladitz > wrote: > > Could the prebuilt branch be updated along with it? > > > Good idea! I just updated the prebuilt branch too. Awesome. Thank you! Nils -------------- next part -------------- An HTML attachment was scrubbed... URL: From nilsgladitz at gmail.com Thu Feb 23 03:53:41 2017 From: nilsgladitz at gmail.com (Nils Gladitz) Date: Thu, 23 Feb 2017 09:53:41 +0100 Subject: [CDash] open.cdash.org updated In-Reply-To: References: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> Message-ID: <01d5aea5-7cf4-cac5-a76e-92c1bd8099ae@gmail.com> On 02/21/2017 06:06 PM, Zack Galbreath wrote: > On Tue, Feb 21, 2017 at 10:03 AM, Nils Gladitz > wrote: > > Could the prebuilt branch be updated along with it? > > > Good idea! I just updated the prebuilt branch too. I tried upgrading my local install with the prebuilt branch (d127922) but it seems to be missing "public/js/CDash_1487690593983.min.js". I assume that is one of the files that were meant to be pre-generated? Nils -------------- next part -------------- An HTML attachment was scrubbed... URL: From zack.galbreath at kitware.com Thu Feb 23 09:32:54 2017 From: zack.galbreath at kitware.com (Zack Galbreath) Date: Thu, 23 Feb 2017 09:32:54 -0500 Subject: [CDash] open.cdash.org updated In-Reply-To: <01d5aea5-7cf4-cac5-a76e-92c1bd8099ae@gmail.com> References: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> <01d5aea5-7cf4-cac5-a76e-92c1bd8099ae@gmail.com> Message-ID: On Thu, Feb 23, 2017 at 3:53 AM, Nils Gladitz wrote: > I tried upgrading my local install with the prebuilt branch (d127922) but > it seems to be missing "public/js/CDash_1487690593983.min.js". > I assume that is one of the files that were meant to be pre-generated? > Whoops, you're right. I just pushed a new commit to prebuilt that includes our javascript file. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nilsgladitz at gmail.com Thu Feb 23 09:59:19 2017 From: nilsgladitz at gmail.com (Nils Gladitz) Date: Thu, 23 Feb 2017 15:59:19 +0100 Subject: [CDash] open.cdash.org updated In-Reply-To: References: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> <01d5aea5-7cf4-cac5-a76e-92c1bd8099ae@gmail.com> Message-ID: On 02/23/2017 03:32 PM, Zack Galbreath wrote: > On Thu, Feb 23, 2017 at 3:53 AM, Nils Gladitz > wrote: > > I tried upgrading my local install with the prebuilt branch > (d127922) but it seems to be missing > "public/js/CDash_1487690593983.min.js". > > I assume that is one of the files that were meant to be pre-generated? > > > Whoops, you're right. I just pushed a new commit to prebuilt that > includes our javascript file. That fixed it for me. Thanks! Nils -------------- next part -------------- An HTML attachment was scrubbed... URL: From prwolfe at sandia.gov Thu Feb 23 11:35:01 2017 From: prwolfe at sandia.gov (Wolfenbarger, Paul R) Date: Thu, 23 Feb 2017 16:35:01 +0000 Subject: [CDash] slow label query Message-ID: I am hoping for someone with more sql understanding than I have to help with an issue. In api/v1/index.php at line 1004 there is a label query that is causing a bottleneck in my installation $label_query_base = "SELECT b2t.status, b2t.newstatus FROM build2test AS b2t INNER JOIN label2test AS l2t ON (l2t.testid=b2t.testid AND l2t.buildid=b2t.buildid) WHERE b2t.buildid = '$buildid' AND l2t.labelid IN $label_ids"; $label_filter_query = $label_query_base . $limit_sql; $labels_result = pdo_query($label_filter_query); This is returning two non-indexed fields from build2test and joining on indexed fields and since my build2test and label2test tables are each about 43 million records, this takes 20 to 300 seconds each. Multipled by at least 40 builds in the loop and you can see my issue. As a c++ programmer my first instinct was to split the query and make all the operations work on indexes. I did some queries at the command line and got sub-second returns from each query and the php would now look something like $testids = pdo_query( ?select testid from label2test where label2test.buildid=$buildid and label2test.labelid in $label_ids") ; $test_ids = implode("','", $testids) ; $label_filter_query = "select status, newstatus from build2test where buildid = $buildid and testid in $test_ids" ; However, from my reading it seems that multiple queries of this sort are frowned on in the sql world. Before I set up a test instance to check this out do any of you have any insight into how to re-structure the query to get the kind of performance I need? Thanks! ---------------------------------------------------------------------------------------- "If you understand what you're doing, you're not learning anything." -- A. L. Paul R. Wolfenbarger, P.E., M.S. Computational Simulation Infrastructure DevOps Product Owner Sandia National Laboratories P.O. Box 5800 MS 0845 Albuquerque, NM 87185-0845 505-844-5458 phone prwolfe at sandia.gov ---------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mateju.Miroslav at azd.cz Mon Feb 27 02:31:46 2017 From: Mateju.Miroslav at azd.cz (=?utf-8?B?TWF0xJtqxa8gTWlyb3NsYXYsIEluZy4=?=) Date: Mon, 27 Feb 2017 07:31:46 +0000 Subject: [CDash] slow label query In-Reply-To: References: Message-ID: <0863544ce310402a9d7b33b3f4f86cac@azdexchstore3.AZD.LOCAL> Hi Paul, As far as I know (but I am rather a C++ programmer like you than a DB expert), it is better to let MySQL do as much as possible and avoid the overhead of transfers and conversions between MySQL and PHP. I think your issue is related to my problem with build2test table performance that I described in the mail thread Optimizing build2test table. Unfortunately, I haven?t been able to test and push the solution yet but it is quite easy to implement in an existing installation: Just add the testid column as the second column to the buildid index for the build2test table. I have done it using PhpMyAdmin on my production machine. It took about 2 hours (for more than 100 million rows) and the submissions (which were the main issue for me) begun to be processed about 3 times faster after that. Hope this helps, Ing. Miroslav Mat?j? Programmer Analyst A?D Praha s.r.o. Technology Division Research and Development ?irovnick? 2/3146, 106 17 Prague Czech Republic Phone: +420 267 287 476 Web: www.azd.cz From: CDash [mailto:cdash-bounces at public.kitware.com] On Behalf Of Wolfenbarger, Paul R Sent: Thursday, February 23, 2017 5:35 PM To: cdash at public.kitware.com Subject: [CDash] slow label query I am hoping for someone with more sql understanding than I have to help with an issue. In api/v1/index.php at line 1004 there is a label query that is causing a bottleneck in my installation $label_query_base = "SELECT b2t.status, b2t.newstatus FROM build2test AS b2t INNER JOIN label2test AS l2t ON (l2t.testid=b2t.testid AND l2t.buildid=b2t.buildid) WHERE b2t.buildid = '$buildid' AND l2t.labelid IN $label_ids"; $label_filter_query = $label_query_base . $limit_sql; $labels_result = pdo_query($label_filter_query); This is returning two non-indexed fields from build2test and joining on indexed fields and since my build2test and label2test tables are each about 43 million records, this takes 20 to 300 seconds each. Multipled by at least 40 builds in the loop and you can see my issue. As a c++ programmer my first instinct was to split the query and make all the operations work on indexes. I did some queries at the command line and got sub-second returns from each query and the php would now look something like $testids = pdo_query( ?select testid from label2test where label2test.buildid=$buildid and label2test.labelid in $label_ids") ; $test_ids = implode("','", $testids) ; $label_filter_query = "select status, newstatus from build2test where buildid = $buildid and testid in $test_ids" ; However, from my reading it seems that multiple queries of this sort are frowned on in the sql world. Before I set up a test instance to check this out do any of you have any insight into how to re-structure the query to get the kind of performance I need? Thanks! ---------------------------------------------------------------------------------------- "If you understand what you're doing, you're not learning anything." -- A. L. Paul R. Wolfenbarger, P.E., M.S. Computational Simulation Infrastructure DevOps Product Owner Sandia National Laboratories P.O. Box 5800 MS 0845 Albuquerque, NM 87185-0845 505-844-5458 phone prwolfe at sandia.gov ---------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From zack.galbreath at kitware.com Mon Feb 27 16:35:59 2017 From: zack.galbreath at kitware.com (Zack Galbreath) Date: Mon, 27 Feb 2017 16:35:59 -0500 Subject: [CDash] [ITK-dev] open.cdash.org updated In-Reply-To: References: Message-ID: On Thu, Feb 23, 2017 at 2:43 PM, Matt McCormick wrote: > Hi Zack, > > Thanks for the updates! > > Another bug I noticed, that I think we present before the recent > upgrade, too, is that when clicking one "expected" checkbox, it > enables all other "expected" checkboxes. > I see what you mean. These boxes are checked if your local representation of this build has its expected value set to 1. So if you check it for one group they all get turned on. So technically I think it's doing the right thing here, but I agree that it looks funny. This indicates to me that our UI for this form could stand to be improved. I'll whip something up and request feedback. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zack.galbreath at kitware.com Tue Feb 21 14:57:31 2017 From: zack.galbreath at kitware.com (Zack Galbreath) Date: Tue, 21 Feb 2017 09:57:31 -0500 Subject: [CDash] open.cdash.org updated Message-ID: Hi folks, I just upgraded open.cdash.org. The most visible change you'll notice is that the Update column now shows the commit that was built rather than the number of files that changed. Going forward, I've set a reminder for myself to upgrade open.cdash.org once a month around this time (the 21st). -------------- next part -------------- An HTML attachment was scrubbed... URL: From nilsgladitz at gmail.com Tue Feb 21 15:03:55 2017 From: nilsgladitz at gmail.com (Nils Gladitz) Date: Tue, 21 Feb 2017 16:03:55 +0100 Subject: [CDash] open.cdash.org updated In-Reply-To: References: Message-ID: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> On 02/21/2017 03:57 PM, Zack Galbreath wrote: > Hi folks, > > I just upgraded open.cdash.org . The most > visible change you'll notice is that the Update column now shows the > commit that was built rather than the number of files that changed. > > Going forward, I've set a reminder for myself to upgrade > open.cdash.org once a month around this time > (the 21st). Could the prebuilt branch be updated along with it? Nils -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.jackson at bluequartz.net Tue Feb 21 15:19:57 2017 From: mike.jackson at bluequartz.net (Michael Jackson) Date: Tue, 21 Feb 2017 10:19:57 -0500 Subject: [CDash] Slack Integration Message-ID: <58AC5A9D.5060700@bluequartz.net> Has anyone thought about allowing users to send build notifications to a slack channel instead of email? -- Michael A. Jackson 400 S. Pioneer Blvd Owner, President Springboro, Ohio 45066 BlueQuartz Software, LLC EMail: mike.jackson at bluequartz.net Voice: 937-790-1601 Web: http://www.bluequartz.net Fax: 937-746-0783 From zack.galbreath at kitware.com Tue Feb 21 17:06:17 2017 From: zack.galbreath at kitware.com (Zack Galbreath) Date: Tue, 21 Feb 2017 12:06:17 -0500 Subject: [CDash] open.cdash.org updated In-Reply-To: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> References: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> Message-ID: On Tue, Feb 21, 2017 at 10:03 AM, Nils Gladitz wrote: > Could the prebuilt branch be updated along with it? > Good idea! I just updated the prebuilt branch too. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nilsgladitz at gmail.com Tue Feb 21 17:07:49 2017 From: nilsgladitz at gmail.com (Nils Gladitz) Date: Tue, 21 Feb 2017 18:07:49 +0100 Subject: [CDash] open.cdash.org updated In-Reply-To: References: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> Message-ID: <63f33efd-261a-9a55-a069-df85fad132d2@gmail.com> On 21.02.2017 18:06, Zack Galbreath wrote: > On Tue, Feb 21, 2017 at 10:03 AM, Nils Gladitz > wrote: > > Could the prebuilt branch be updated along with it? > > > Good idea! I just updated the prebuilt branch too. Awesome. Thank you! Nils -------------- next part -------------- An HTML attachment was scrubbed... URL: From nilsgladitz at gmail.com Thu Feb 23 08:53:41 2017 From: nilsgladitz at gmail.com (Nils Gladitz) Date: Thu, 23 Feb 2017 09:53:41 +0100 Subject: [CDash] open.cdash.org updated In-Reply-To: References: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> Message-ID: <01d5aea5-7cf4-cac5-a76e-92c1bd8099ae@gmail.com> On 02/21/2017 06:06 PM, Zack Galbreath wrote: > On Tue, Feb 21, 2017 at 10:03 AM, Nils Gladitz > wrote: > > Could the prebuilt branch be updated along with it? > > > Good idea! I just updated the prebuilt branch too. I tried upgrading my local install with the prebuilt branch (d127922) but it seems to be missing "public/js/CDash_1487690593983.min.js". I assume that is one of the files that were meant to be pre-generated? Nils -------------- next part -------------- An HTML attachment was scrubbed... URL: From zack.galbreath at kitware.com Thu Feb 23 14:32:54 2017 From: zack.galbreath at kitware.com (Zack Galbreath) Date: Thu, 23 Feb 2017 09:32:54 -0500 Subject: [CDash] open.cdash.org updated In-Reply-To: <01d5aea5-7cf4-cac5-a76e-92c1bd8099ae@gmail.com> References: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> <01d5aea5-7cf4-cac5-a76e-92c1bd8099ae@gmail.com> Message-ID: On Thu, Feb 23, 2017 at 3:53 AM, Nils Gladitz wrote: > I tried upgrading my local install with the prebuilt branch (d127922) but > it seems to be missing "public/js/CDash_1487690593983.min.js". > I assume that is one of the files that were meant to be pre-generated? > Whoops, you're right. I just pushed a new commit to prebuilt that includes our javascript file. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nilsgladitz at gmail.com Thu Feb 23 14:59:19 2017 From: nilsgladitz at gmail.com (Nils Gladitz) Date: Thu, 23 Feb 2017 15:59:19 +0100 Subject: [CDash] open.cdash.org updated In-Reply-To: References: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> <01d5aea5-7cf4-cac5-a76e-92c1bd8099ae@gmail.com> Message-ID: On 02/23/2017 03:32 PM, Zack Galbreath wrote: > On Thu, Feb 23, 2017 at 3:53 AM, Nils Gladitz > wrote: > > I tried upgrading my local install with the prebuilt branch > (d127922) but it seems to be missing > "public/js/CDash_1487690593983.min.js". > > I assume that is one of the files that were meant to be pre-generated? > > > Whoops, you're right. I just pushed a new commit to prebuilt that > includes our javascript file. That fixed it for me. Thanks! Nils -------------- next part -------------- An HTML attachment was scrubbed... URL: From prwolfe at sandia.gov Thu Feb 23 16:35:01 2017 From: prwolfe at sandia.gov (Wolfenbarger, Paul R) Date: Thu, 23 Feb 2017 16:35:01 +0000 Subject: [CDash] slow label query Message-ID: I am hoping for someone with more sql understanding than I have to help with an issue. In api/v1/index.php at line 1004 there is a label query that is causing a bottleneck in my installation $label_query_base = "SELECT b2t.status, b2t.newstatus FROM build2test AS b2t INNER JOIN label2test AS l2t ON (l2t.testid=b2t.testid AND l2t.buildid=b2t.buildid) WHERE b2t.buildid = '$buildid' AND l2t.labelid IN $label_ids"; $label_filter_query = $label_query_base . $limit_sql; $labels_result = pdo_query($label_filter_query); This is returning two non-indexed fields from build2test and joining on indexed fields and since my build2test and label2test tables are each about 43 million records, this takes 20 to 300 seconds each. Multipled by at least 40 builds in the loop and you can see my issue. As a c++ programmer my first instinct was to split the query and make all the operations work on indexes. I did some queries at the command line and got sub-second returns from each query and the php would now look something like $testids = pdo_query( ?select testid from label2test where label2test.buildid=$buildid and label2test.labelid in $label_ids") ; $test_ids = implode("','", $testids) ; $label_filter_query = "select status, newstatus from build2test where buildid = $buildid and testid in $test_ids" ; However, from my reading it seems that multiple queries of this sort are frowned on in the sql world. Before I set up a test instance to check this out do any of you have any insight into how to re-structure the query to get the kind of performance I need? Thanks! ---------------------------------------------------------------------------------------- "If you understand what you're doing, you're not learning anything." -- A. L. Paul R. Wolfenbarger, P.E., M.S. Computational Simulation Infrastructure DevOps Product Owner Sandia National Laboratories P.O. Box 5800 MS 0845 Albuquerque, NM 87185-0845 505-844-5458 phone prwolfe at sandia.gov ---------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mateju.Miroslav at azd.cz Mon Feb 27 07:31:46 2017 From: Mateju.Miroslav at azd.cz (=?utf-8?B?TWF0xJtqxa8gTWlyb3NsYXYsIEluZy4=?=) Date: Mon, 27 Feb 2017 07:31:46 +0000 Subject: [CDash] slow label query In-Reply-To: References: Message-ID: <0863544ce310402a9d7b33b3f4f86cac@azdexchstore3.AZD.LOCAL> Hi Paul, As far as I know (but I am rather a C++ programmer like you than a DB expert), it is better to let MySQL do as much as possible and avoid the overhead of transfers and conversions between MySQL and PHP. I think your issue is related to my problem with build2test table performance that I described in the mail thread Optimizing build2test table. Unfortunately, I haven?t been able to test and push the solution yet but it is quite easy to implement in an existing installation: Just add the testid column as the second column to the buildid index for the build2test table. I have done it using PhpMyAdmin on my production machine. It took about 2 hours (for more than 100 million rows) and the submissions (which were the main issue for me) begun to be processed about 3 times faster after that. Hope this helps, Ing. Miroslav Mat?j? Programmer Analyst A?D Praha s.r.o. Technology Division Research and Development ?irovnick? 2/3146, 106 17 Prague Czech Republic Phone: +420 267 287 476 Web: www.azd.cz From: CDash [mailto:cdash-bounces at public.kitware.com] On Behalf Of Wolfenbarger, Paul R Sent: Thursday, February 23, 2017 5:35 PM To: cdash at public.kitware.com Subject: [CDash] slow label query I am hoping for someone with more sql understanding than I have to help with an issue. In api/v1/index.php at line 1004 there is a label query that is causing a bottleneck in my installation $label_query_base = "SELECT b2t.status, b2t.newstatus FROM build2test AS b2t INNER JOIN label2test AS l2t ON (l2t.testid=b2t.testid AND l2t.buildid=b2t.buildid) WHERE b2t.buildid = '$buildid' AND l2t.labelid IN $label_ids"; $label_filter_query = $label_query_base . $limit_sql; $labels_result = pdo_query($label_filter_query); This is returning two non-indexed fields from build2test and joining on indexed fields and since my build2test and label2test tables are each about 43 million records, this takes 20 to 300 seconds each. Multipled by at least 40 builds in the loop and you can see my issue. As a c++ programmer my first instinct was to split the query and make all the operations work on indexes. I did some queries at the command line and got sub-second returns from each query and the php would now look something like $testids = pdo_query( ?select testid from label2test where label2test.buildid=$buildid and label2test.labelid in $label_ids") ; $test_ids = implode("','", $testids) ; $label_filter_query = "select status, newstatus from build2test where buildid = $buildid and testid in $test_ids" ; However, from my reading it seems that multiple queries of this sort are frowned on in the sql world. Before I set up a test instance to check this out do any of you have any insight into how to re-structure the query to get the kind of performance I need? Thanks! ---------------------------------------------------------------------------------------- "If you understand what you're doing, you're not learning anything." -- A. L. Paul R. Wolfenbarger, P.E., M.S. Computational Simulation Infrastructure DevOps Product Owner Sandia National Laboratories P.O. Box 5800 MS 0845 Albuquerque, NM 87185-0845 505-844-5458 phone prwolfe at sandia.gov ---------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From zack.galbreath at kitware.com Mon Feb 27 21:35:59 2017 From: zack.galbreath at kitware.com (Zack Galbreath) Date: Mon, 27 Feb 2017 16:35:59 -0500 Subject: [CDash] [ITK-dev] open.cdash.org updated In-Reply-To: References: Message-ID: On Thu, Feb 23, 2017 at 2:43 PM, Matt McCormick wrote: > Hi Zack, > > Thanks for the updates! > > Another bug I noticed, that I think we present before the recent > upgrade, too, is that when clicking one "expected" checkbox, it > enables all other "expected" checkboxes. > I see what you mean. These boxes are checked if your local representation of this build has its expected value set to 1. So if you check it for one group they all get turned on. So technically I think it's doing the right thing here, but I agree that it looks funny. This indicates to me that our UI for this form could stand to be improved. I'll whip something up and request feedback. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zack.galbreath at kitware.com Tue Feb 21 14:57:31 2017 From: zack.galbreath at kitware.com (Zack Galbreath) Date: Tue, 21 Feb 2017 09:57:31 -0500 Subject: [CDash] open.cdash.org updated Message-ID: Hi folks, I just upgraded open.cdash.org. The most visible change you'll notice is that the Update column now shows the commit that was built rather than the number of files that changed. Going forward, I've set a reminder for myself to upgrade open.cdash.org once a month around this time (the 21st). -------------- next part -------------- An HTML attachment was scrubbed... URL: From nilsgladitz at gmail.com Tue Feb 21 15:03:55 2017 From: nilsgladitz at gmail.com (Nils Gladitz) Date: Tue, 21 Feb 2017 16:03:55 +0100 Subject: [CDash] open.cdash.org updated In-Reply-To: References: Message-ID: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> On 02/21/2017 03:57 PM, Zack Galbreath wrote: > Hi folks, > > I just upgraded open.cdash.org . The most > visible change you'll notice is that the Update column now shows the > commit that was built rather than the number of files that changed. > > Going forward, I've set a reminder for myself to upgrade > open.cdash.org once a month around this time > (the 21st). Could the prebuilt branch be updated along with it? Nils -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.jackson at bluequartz.net Tue Feb 21 15:19:57 2017 From: mike.jackson at bluequartz.net (Michael Jackson) Date: Tue, 21 Feb 2017 10:19:57 -0500 Subject: [CDash] Slack Integration Message-ID: <58AC5A9D.5060700@bluequartz.net> Has anyone thought about allowing users to send build notifications to a slack channel instead of email? -- Michael A. Jackson 400 S. Pioneer Blvd Owner, President Springboro, Ohio 45066 BlueQuartz Software, LLC EMail: mike.jackson at bluequartz.net Voice: 937-790-1601 Web: http://www.bluequartz.net Fax: 937-746-0783 From zack.galbreath at kitware.com Tue Feb 21 17:06:17 2017 From: zack.galbreath at kitware.com (Zack Galbreath) Date: Tue, 21 Feb 2017 12:06:17 -0500 Subject: [CDash] open.cdash.org updated In-Reply-To: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> References: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> Message-ID: On Tue, Feb 21, 2017 at 10:03 AM, Nils Gladitz wrote: > Could the prebuilt branch be updated along with it? > Good idea! I just updated the prebuilt branch too. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nilsgladitz at gmail.com Tue Feb 21 17:07:49 2017 From: nilsgladitz at gmail.com (Nils Gladitz) Date: Tue, 21 Feb 2017 18:07:49 +0100 Subject: [CDash] open.cdash.org updated In-Reply-To: References: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> Message-ID: <63f33efd-261a-9a55-a069-df85fad132d2@gmail.com> On 21.02.2017 18:06, Zack Galbreath wrote: > On Tue, Feb 21, 2017 at 10:03 AM, Nils Gladitz > wrote: > > Could the prebuilt branch be updated along with it? > > > Good idea! I just updated the prebuilt branch too. Awesome. Thank you! Nils -------------- next part -------------- An HTML attachment was scrubbed... URL: From nilsgladitz at gmail.com Thu Feb 23 08:53:41 2017 From: nilsgladitz at gmail.com (Nils Gladitz) Date: Thu, 23 Feb 2017 09:53:41 +0100 Subject: [CDash] open.cdash.org updated In-Reply-To: References: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> Message-ID: <01d5aea5-7cf4-cac5-a76e-92c1bd8099ae@gmail.com> On 02/21/2017 06:06 PM, Zack Galbreath wrote: > On Tue, Feb 21, 2017 at 10:03 AM, Nils Gladitz > wrote: > > Could the prebuilt branch be updated along with it? > > > Good idea! I just updated the prebuilt branch too. I tried upgrading my local install with the prebuilt branch (d127922) but it seems to be missing "public/js/CDash_1487690593983.min.js". I assume that is one of the files that were meant to be pre-generated? Nils -------------- next part -------------- An HTML attachment was scrubbed... URL: From zack.galbreath at kitware.com Thu Feb 23 14:32:54 2017 From: zack.galbreath at kitware.com (Zack Galbreath) Date: Thu, 23 Feb 2017 09:32:54 -0500 Subject: [CDash] open.cdash.org updated In-Reply-To: <01d5aea5-7cf4-cac5-a76e-92c1bd8099ae@gmail.com> References: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> <01d5aea5-7cf4-cac5-a76e-92c1bd8099ae@gmail.com> Message-ID: On Thu, Feb 23, 2017 at 3:53 AM, Nils Gladitz wrote: > I tried upgrading my local install with the prebuilt branch (d127922) but > it seems to be missing "public/js/CDash_1487690593983.min.js". > I assume that is one of the files that were meant to be pre-generated? > Whoops, you're right. I just pushed a new commit to prebuilt that includes our javascript file. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nilsgladitz at gmail.com Thu Feb 23 14:59:19 2017 From: nilsgladitz at gmail.com (Nils Gladitz) Date: Thu, 23 Feb 2017 15:59:19 +0100 Subject: [CDash] open.cdash.org updated In-Reply-To: References: <81aaa674-c3ca-b004-2ca1-203b6bb6d7dc@gmail.com> <01d5aea5-7cf4-cac5-a76e-92c1bd8099ae@gmail.com> Message-ID: On 02/23/2017 03:32 PM, Zack Galbreath wrote: > On Thu, Feb 23, 2017 at 3:53 AM, Nils Gladitz > wrote: > > I tried upgrading my local install with the prebuilt branch > (d127922) but it seems to be missing > "public/js/CDash_1487690593983.min.js". > > I assume that is one of the files that were meant to be pre-generated? > > > Whoops, you're right. I just pushed a new commit to prebuilt that > includes our javascript file. That fixed it for me. Thanks! Nils -------------- next part -------------- An HTML attachment was scrubbed... URL: From prwolfe at sandia.gov Thu Feb 23 16:35:01 2017 From: prwolfe at sandia.gov (Wolfenbarger, Paul R) Date: Thu, 23 Feb 2017 16:35:01 +0000 Subject: [CDash] slow label query Message-ID: I am hoping for someone with more sql understanding than I have to help with an issue. In api/v1/index.php at line 1004 there is a label query that is causing a bottleneck in my installation $label_query_base = "SELECT b2t.status, b2t.newstatus FROM build2test AS b2t INNER JOIN label2test AS l2t ON (l2t.testid=b2t.testid AND l2t.buildid=b2t.buildid) WHERE b2t.buildid = '$buildid' AND l2t.labelid IN $label_ids"; $label_filter_query = $label_query_base . $limit_sql; $labels_result = pdo_query($label_filter_query); This is returning two non-indexed fields from build2test and joining on indexed fields and since my build2test and label2test tables are each about 43 million records, this takes 20 to 300 seconds each. Multipled by at least 40 builds in the loop and you can see my issue. As a c++ programmer my first instinct was to split the query and make all the operations work on indexes. I did some queries at the command line and got sub-second returns from each query and the php would now look something like $testids = pdo_query( ?select testid from label2test where label2test.buildid=$buildid and label2test.labelid in $label_ids") ; $test_ids = implode("','", $testids) ; $label_filter_query = "select status, newstatus from build2test where buildid = $buildid and testid in $test_ids" ; However, from my reading it seems that multiple queries of this sort are frowned on in the sql world. Before I set up a test instance to check this out do any of you have any insight into how to re-structure the query to get the kind of performance I need? Thanks! ---------------------------------------------------------------------------------------- "If you understand what you're doing, you're not learning anything." -- A. L. Paul R. Wolfenbarger, P.E., M.S. Computational Simulation Infrastructure DevOps Product Owner Sandia National Laboratories P.O. Box 5800 MS 0845 Albuquerque, NM 87185-0845 505-844-5458 phone prwolfe at sandia.gov ---------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mateju.Miroslav at azd.cz Mon Feb 27 07:31:46 2017 From: Mateju.Miroslav at azd.cz (=?utf-8?B?TWF0xJtqxa8gTWlyb3NsYXYsIEluZy4=?=) Date: Mon, 27 Feb 2017 07:31:46 +0000 Subject: [CDash] slow label query In-Reply-To: References: Message-ID: <0863544ce310402a9d7b33b3f4f86cac@azdexchstore3.AZD.LOCAL> Hi Paul, As far as I know (but I am rather a C++ programmer like you than a DB expert), it is better to let MySQL do as much as possible and avoid the overhead of transfers and conversions between MySQL and PHP. I think your issue is related to my problem with build2test table performance that I described in the mail thread Optimizing build2test table. Unfortunately, I haven?t been able to test and push the solution yet but it is quite easy to implement in an existing installation: Just add the testid column as the second column to the buildid index for the build2test table. I have done it using PhpMyAdmin on my production machine. It took about 2 hours (for more than 100 million rows) and the submissions (which were the main issue for me) begun to be processed about 3 times faster after that. Hope this helps, Ing. Miroslav Mat?j? Programmer Analyst A?D Praha s.r.o. Technology Division Research and Development ?irovnick? 2/3146, 106 17 Prague Czech Republic Phone: +420 267 287 476 Web: www.azd.cz From: CDash [mailto:cdash-bounces at public.kitware.com] On Behalf Of Wolfenbarger, Paul R Sent: Thursday, February 23, 2017 5:35 PM To: cdash at public.kitware.com Subject: [CDash] slow label query I am hoping for someone with more sql understanding than I have to help with an issue. In api/v1/index.php at line 1004 there is a label query that is causing a bottleneck in my installation $label_query_base = "SELECT b2t.status, b2t.newstatus FROM build2test AS b2t INNER JOIN label2test AS l2t ON (l2t.testid=b2t.testid AND l2t.buildid=b2t.buildid) WHERE b2t.buildid = '$buildid' AND l2t.labelid IN $label_ids"; $label_filter_query = $label_query_base . $limit_sql; $labels_result = pdo_query($label_filter_query); This is returning two non-indexed fields from build2test and joining on indexed fields and since my build2test and label2test tables are each about 43 million records, this takes 20 to 300 seconds each. Multipled by at least 40 builds in the loop and you can see my issue. As a c++ programmer my first instinct was to split the query and make all the operations work on indexes. I did some queries at the command line and got sub-second returns from each query and the php would now look something like $testids = pdo_query( ?select testid from label2test where label2test.buildid=$buildid and label2test.labelid in $label_ids") ; $test_ids = implode("','", $testids) ; $label_filter_query = "select status, newstatus from build2test where buildid = $buildid and testid in $test_ids" ; However, from my reading it seems that multiple queries of this sort are frowned on in the sql world. Before I set up a test instance to check this out do any of you have any insight into how to re-structure the query to get the kind of performance I need? Thanks! ---------------------------------------------------------------------------------------- "If you understand what you're doing, you're not learning anything." -- A. L. Paul R. Wolfenbarger, P.E., M.S. Computational Simulation Infrastructure DevOps Product Owner Sandia National Laboratories P.O. Box 5800 MS 0845 Albuquerque, NM 87185-0845 505-844-5458 phone prwolfe at sandia.gov ---------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From zack.galbreath at kitware.com Mon Feb 27 21:35:59 2017 From: zack.galbreath at kitware.com (Zack Galbreath) Date: Mon, 27 Feb 2017 16:35:59 -0500 Subject: [CDash] [ITK-dev] open.cdash.org updated In-Reply-To: References: Message-ID: On Thu, Feb 23, 2017 at 2:43 PM, Matt McCormick wrote: > Hi Zack, > > Thanks for the updates! > > Another bug I noticed, that I think we present before the recent > upgrade, too, is that when clicking one "expected" checkbox, it > enables all other "expected" checkboxes. > I see what you mean. These boxes are checked if your local representation of this build has its expected value set to 1. So if you check it for one group they all get turned on. So technically I think it's doing the right thing here, but I agree that it looks funny. This indicates to me that our UI for this form could stand to be improved. I'll whip something up and request feedback. -------------- next part -------------- An HTML attachment was scrubbed... URL: