[Development] Merge and Integration status report - 2018.11.08

[Liang Qi]

Integrations

* qt5 5.9/5.11 integrations are healthy
* qt5 5.12/dev integrations: skipped qtbase after https://codereview.qt-project.org/#/c/243906/ .
* Issue: https://bugreports.qt.io/browse/QTBUG-71550 , a fix at https://codereview.qt-project.org/#/c/244725/ , but didn't get integrated after a few stages, and I don't think it will be merged soon.

I don't know how to use [Grafana][1] to show the statistic of qtbase 5.12 integrations in coin. There are only 3 successful integrations yesterday:
* https://testresults.qt.io/coin/integration/qt/qtbase/tasks/1543278570
* https://testresults.qt.io/coin/integration/qt/qtbase/tasks/1543265593
* https://testresults.qt.io/coin/integration/qt/qtbase/tasks/1543189785

[1]: https://testresults.qt.io/grafana/

-- Liang

[Oswald Buddenhagen]

On Thu, Nov 08, 2018 at 08:29:47AM +0000, Liang Qi wrote:
> I don't know how to use [Grafana] to show the statistic of qtbase 5.12
> integrations in coin. There are only 3 successful integrations
> yesterday: [...]
>
because this situation has been persisting for weeks and we've been
essentially just burning energy and motivation with doomed integration
attempts, we decided to lock down 5.12 until the situation is under
control. a somewhat optimistic ETA is maybe a week, depending on the CI
team's progress with deploying infrastructure software updates (it's a
bit more involved than it sounds, so bear with them).

if you have critical changes which justify spending extra effort to get
them in soon, talk to liang.

[Liang Qi]

> On 8 Nov 2018, at 13:00, Oswald Buddenhagen <***@qt.io> wrote:
>
> On Thu, Nov 08, 2018 at 08:29:47AM +0000, Liang Qi wrote:
>> I don't know how to use [Grafana] to show the statistic of qtbase 5.12
>> integrations in coin. There are only 3 successful integrations
>> yesterday: [...]
>>
> because this situation has been persisting for weeks and we've been
> essentially just burning energy and motivation with doomed integration
> attempts, we decided to lock down 5.12 until the situation is under
> control. a somewhat optimistic ETA is maybe a week, depending on the CI
> team's progress with deploying infrastructure software updates (it's a
> bit more involved than it sounds, so bear with them).
>
> if you have critical changes which justify spending extra effort to get
> them in soon, talk to liang.

I thought qtbase 5.12 got locked since previous email. Looks like not, or some people(with privilege) are still eager to hit the stage button. I will not try more during tonight.

—Liang

[Oswald Buddenhagen]

On Thu, Nov 08, 2018 at 04:26:04PM +0100, Liang Qi wrote:
> I thought qtbase 5.12 got locked since previous email. Looks like not,
> or some people(with privilege) are still eager to hit the stage
> button. I will not try more during tonight.
>
yes, because someone keeps re-adding thiago and several others to the
release team's gerrit group. that's what you get when you let people
without a system overview try to solve problems.

[Thiago Macieira]

On Thursday, 8 November 2018 08:01:48 PST Oswald Buddenhagen wrote:
> On Thu, Nov 08, 2018 at 04:26:04PM +0100, Liang Qi wrote:
> > I thought qtbase 5.12 got locked since previous email. Looks like not,
> > or some people(with privilege) are still eager to hit the stage
> > button. I will not try more during tonight.
>
> yes, because someone keeps re-adding thiago and several others to the
> release team's gerrit group. that's what you get when you let people
> without a system overview try to solve problems.

My bad.

I staged things at 7 am before reading the 4 am email.

--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center

[Thiago Macieira]

On Thursday, 8 November 2018 04:00:08 PST Oswald Buddenhagen wrote:
> On Thu, Nov 08, 2018 at 08:29:47AM +0000, Liang Qi wrote:
> > I don't know how to use [Grafana] to show the statistic of qtbase 5.12
> > integrations in coin. There are only 3 successful integrations
> > yesterday: [...]
>
> because this situation has been persisting for weeks and we've been
> essentially just burning energy and motivation with doomed integration
> attempts, we decided to lock down 5.12 until the situation is under
> control. a somewhat optimistic ETA is maybe a week, depending on the CI
> team's progress with deploying infrastructure software updates (it's a
> bit more involved than it sounds, so bear with them).
>
> if you have critical changes which justify spending extra effort to get
> them in soon, talk to liang.

Is this done? When is the lock down getting removed?

And who's working on 5.12 during the weekend, as it is still locked down.

--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center

[Liang Qi]

> On 11 Nov 2018, at 20:01, Thiago Macieira <***@intel.com> wrote:
>
> On Thursday, 8 November 2018 04:00:08 PST Oswald Buddenhagen wrote:
>> On Thu, Nov 08, 2018 at 08:29:47AM +0000, Liang Qi wrote:
>>> I don't know how to use [Grafana] to show the statistic of qtbase 5.12
>>> integrations in coin. There are only 3 successful integrations
>>> yesterday: [...]
>>
>> because this situation has been persisting for weeks and we've been
>> essentially just burning energy and motivation with doomed integration
>> attempts, we decided to lock down 5.12 until the situation is under
>> control. a somewhat optimistic ETA is maybe a week, depending on the CI
>> team's progress with deploying infrastructure software updates (it's a
>> bit more involved than it sounds, so bear with them).
>>
>> if you have critical changes which justify spending extra effort to get
>> them in soon, talk to liang.
>
> Is this done? When is the lock down getting removed?
>
> And who's working on 5.12 during the weekend, as it is still locked down.

The fix of the blocker got merged into qtbase 5.12. But the issue of ci/coin is not fixed yet. Talked with Oswald on Friday, we agreed to keep qtbase 5.12 lock a bit longer.

I staged a few times from Friday night to now, got some changes integrated and found some issues in them. Do you have some changes need to stage?

Recently the qtbase integration takes a bit longer, 2.5 hours like https://testresults.qt.io/coin/integration/qt/qtbase/tasks/1543678216 , is a bit lucky. More details see https://bugreports.qt.io/browse/QTQAINFRA-2161 .

—Liang

[Thiago Macieira]

On Sunday, 11 November 2018 11:44:12 PST Liang Qi wrote:
> I staged a few times from Friday night to now, got some changes integrated
> and found some issues in them. Do you have some changes need to stage?

I have a few bug fixes and improvements, but nothing urgent. I was just
wondering about the logic of leaving it locked down during the weekend.

dev is integrating fine.

--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center

[Liang Qi]

> On 11 Nov 2018, at 22:25, Thiago Macieira <***@intel.com> wrote:
>
> On Sunday, 11 November 2018 11:44:12 PST Liang Qi wrote:
>> I staged a few times from Friday night to now, got some changes integrated
>> and found some issues in them. Do you have some changes need to stage?
>
> I have a few bug fixes and improvements, but nothing urgent. I was just
> wondering about the logic of leaving it locked down during the weekend.
>
> dev is integrating fine.

Here is my working log(Nov. 8 to 12) for qtbase 5.12:

=====

Nov. 8

* 1pm, announcement about qtbase 5.12 got locked, but release team and a few people still have privilege to stage
* 4pm, staged https://codereview.qt-project.org/#/c/244725/ the fix for bloker (got a few other changes staged by another person) - http://coin/coin/integration/qt/qtbase/tasks/1543396094

1 stage

Nov. 9

* 7:57am coin sucks(reboot) - http://coin/coin/integration/qt/qtbase/tasks/1543397138
* 8:02am coin reboot again - http://coin/coin/integration/qt/qtbase/tasks/1543404322 (succeed in 2h18m)
* a background build for above http://coin/coin/integration/qt/qtbase/tasks/web_qt_qtbase_1541747658570
* 10:41am staged https://codereview.qt-project.org/#/c/243826/ - http://coin/coin/integration/qt/qtbase/tasks/1543439390 (failed in 2h21m)
* WinRT: tst_QEventDispatcher::registerTimer() QTestLib: This test case check ("(((receivedEventType) == (int(QEvent::Timer))))") failed because the requested timeout (20 ms) was too short, 70 ms would have been sufficient this time.
* 1:12pm restaged - http://coin/coin/integration/qt/qtbase/tasks/1543464639
* 5:14pm coin reboot - http://coin/coin/integration/qt/qtbase/tasks/1543482384
* 5:25pm coin reboot - http://coin/coin/integration/qt/qtbase/tasks/1543507534
* 5:29pm coin reboot - http://coin/coin/integration/qt/qtbase/tasks/1543507580 (succeed in 2h3m)
* 10:45pm staged 5 changes together - http://coin/coin/integration/qt/qtbase/tasks/1543598167 (succeed in 2h28m)

3 stages/1 background build/5 coin reboot
1 stage->succeed

Nov. 10(Saturday)

* 8:42am staged https://codereview.qt-project.org/#/c/244677/ - http://coin/coin/integration/qt/qtbase/tasks/1543619138 (failed in 55m - qmake crash on opensuse)
* 9:54am re-taged - http://coin/coin/integration/qt/qtbase/tasks/1543627241 (failed in 56m - tst_QEventDispatcher::registerTimer() on macOS 10.13)
* 11:37am re-staged - http://coin/coin/integration/qt/qtbase/tasks/1543642827 (failed in 35m - tst_QEventDispatcher::registerTimer() on macOS 10.13)
* 12:21pm re-stage - http://coin/coin/integration/qt/qtbase/tasks/1543642917 (failed in 31m - tst_QEventDispatcher::registerTimer() on macOS 10.13)
* 2:42pm re-stage - http://coin/coin/integration/qt/qtbase/tasks/1543651407 (failed in 1h44m - tst_QEventDispatcher::registerTimer() on WinRT)
* 6:12pm re-stage - http://coin/coin/integration/qt/qtbase/tasks/1543652915 (failed in 2h44m - "Most likely the test has crashed" on WinRT)
* 9:39pm a background build - http://coin/coin/integration/qt/qtbase/tasks/web_qt_qtbase_1541882395456 (succeed in 1h57m)
* 9:42pm staged https://codereview.qt-project.org/#/c/235713/ - http://coin/coin/integration/qt/qtbase/tasks/1543676742 (failed in 1h7m - tst_QTcpServer::ipv6Server(WithSocks5Proxy) on macOS 10.12)

7 stages/1 background build/0 coin reboot
0 stage->succeed

Nov. 11(Sunday)

* 9:28am staged https://codereview.qt-project.org/#/c/244677/ - http://coin/coin/integration/qt/qtbase/tasks/1543677498 (succeed in 35m)
* 10:12am staged 6 changes together - http://coin/coin/integration/qt/qtbase/tasks/1543678216 (succeed in 2h25m)
* 8:06pm staged 8 changes together - http://coin/coin/integration/qt/qtbase/tasks/1543687675 (failed in 7m - real failure)
* 8:27pm staged 7 changes together - http://coin/coin/integration/qt/qtbase/tasks/1543701974 (failed in 35min - tst_QEventDispatcher::registerTimer() on macOS 10.13)
* 9:14pm a background build - http://coin/coin/integration/qt/qtbase/tasks/web_qt_qtbase_1541967249016 (succeed in 1h42m)
* 11:13pm staged 7 changes again - http://coin/coin/integration/qt/qtbase/tasks/1543709881 (succeed in 19s)

5 stages/1 background build/0 coin reboot
1 stage->succeed

Nov. 12

* 8:19am staged 5 changes together - http://coin/coin/integration/qt/qtbase/tasks/1543741463 (succeed in 2h25m)
* 10:53am staged 8 changes togehter - http://coin/coin/integration/qt/qtbase/tasks/1543764990 (failed in 53m - tst_QEventDispatcher::registerTimer() on macOS 10.13)
* 12:00pm a background build - http://coin/coin/integration/qt/qtbase/tasks/web_qt_qtbase_1542020400468 (succeed in 1h50m)
* 2:01pm re-staged 8 changes - https://testresults.qt.io/coin/integration/qt/qtbase/tasks/1543790835 (succeed in 24m)

3 stages/1 background build/0 coin reboot
1 stage->succeed

=====

I think it’s a much better result compared with previous chaos(random staging, and many client lost and etc). I didn't plan to provide an overview of that period yet.

I am not very sure the progress of ci/coin team, see http://lists.qt-project.org/pipermail/development/2018-November/034244.html .

During this period, nobody contacted me to stage their change. So my staging is just based on the history of previous failures.

—Liang

[Thiago Macieira]

On Monday, 12 November 2018 12:38:38 PST Liang Qi wrote:
> * 10:53am staged 8 changes togehter -
> http://coin/coin/integration/qt/qtbase/tasks/1543764990 (failed in 53m -
> tst_QEventDispatcher::registerTimer() on macOS 10.13)

Thanks for all your effort, Liang. It is much appreciated.

The above test seems flaky, but increasing the timeout doesn't help. It
complains that 20 ms wasn't enough, but 20 ms would have been enough. When I
increased to 50 ms, it prints the same thing but saying 50 ms. There are two
more failures in the same test.

So I'm inclined to believe that this is a real failure caused by something
changing in the event dispatcher code relating to macOS. But the weird thing
is that it should be running the QEventDispatcherUNIX class...

--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center

[Allan Sandfeld Jensen]

On Montag, 12. November 2018 22:49:02 CET Thiago Macieira wrote:
> On Monday, 12 November 2018 12:38:38 PST Liang Qi wrote:
> > * 10:53am staged 8 changes togehter -
> > http://coin/coin/integration/qt/qtbase/tasks/1543764990 (failed in 53m -
> > tst_QEventDispatcher::registerTimer() on macOS 10.13)
>
> Thanks for all your effort, Liang. It is much appreciated.
>
> The above test seems flaky, but increasing the timeout doesn't help. It
> complains that 20 ms wasn't enough, but 20 ms would have been enough. When I
> increased to 50 ms, it prints the same thing but saying 50 ms. There are
> two more failures in the same test.
>
> So I'm inclined to believe that this is a real failure caused by something
> changing in the event dispatcher code relating to macOS. But the weird thing
> is that it should be running the QEventDispatcherUNIX class...

I would argue the test should be blacklisted at least on macOS 10.13 as a P0
task. Fixing it can wait.

'Allan

[Oliver Wolff]

On 12/11/2018 23:03, Allan Sandfeld Jensen wrote:
> On Montag, 12. November 2018 22:49:02 CET Thiago Macieira wrote:
>> On Monday, 12 November 2018 12:38:38 PST Liang Qi wrote:
>>> * 10:53am staged 8 changes togehter -
>>> http://coin/coin/integration/qt/qtbase/tasks/1543764990 (failed in 53m -
>>> tst_QEventDispatcher::registerTimer() on macOS 10.13)
>> Thanks for all your effort, Liang. It is much appreciated.
>>
>> The above test seems flaky, but increasing the timeout doesn't help. It
>> complains that 20 ms wasn't enough, but 20 ms would have been enough. When I
>> increased to 50 ms, it prints the same thing but saying 50 ms. There are
>> two more failures in the same test.
>>
>> So I'm inclined to believe that this is a real failure caused by something
>> changing in the event dispatcher code relating to macOS. But the weird thing
>> is that it should be running the QEventDispatcherUNIX class...
> I would argue the test should be blacklisted at least on macOS 10.13 as a P0
> task. Fixing it can wait.
>
> 'Allan
The test also fails on WinRT and thus at least the registerTimer test in
corelib/kernel/qeventdispatcher is blacklisted for this platform. That
one used to be blacklisted on macos as well, but is considered stable
with 5.12. I think blacklisting registerTimer for the gui
eventdispatcher on WinRT would make sense, as both tests use the same
event dispatcher on this platform (I think that's also the case on Windows).

I had a look at the problem when I blacklisted it for WinRT and one
problem was, that QTRY_IMPL used a hard coded step interval of 50 ms,
which of course does not make sense if you try to have a timeout of 25
ms. In addition to that I don't know, how well coin copes with such
short timeout, as VMs tend to be quite slow when run from Coin (in my
experience).

To sum things up, I think tests that are blacklisted for windows/winrt
in tst_qeventdispatcher should at also be blacklisted in
tst_qguieventdispatcher as the same event dispatcher is used
(https://codereview.qt-project.org/245347). I don't know enough about
Mac's approach, but tst_qeventdispatcher::registerTimer was
unblacklisted recently as far as I know.

Olli

>
>
> _______________________________________________
> Development mailing list
> ***@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development

[Ulf Hermann]

> I had a look at the problem when I blacklisted it for WinRT and one
> problem was, that QTRY_IMPL used a hard coded step interval of 50 ms,
> which of course does not make sense if you try to have a timeout of 25
> ms. In addition to that I don't know, how well coin copes with such
> short timeout, as VMs tend to be quite slow when run from Coin (in my
> experience).

Just wait forever on the check in a loop. After 15 minutes the watchdog
kicks in and kills the test. If that happens, you know that it's a real
failure. This works fine for the QML debugger tests.

[Allan Sandfeld Jensen]

On Dienstag, 13. November 2018 09:30:11 CET Ulf Hermann wrote:
> > I had a look at the problem when I blacklisted it for WinRT and one
> > problem was, that QTRY_IMPL used a hard coded step interval of 50 ms,
> > which of course does not make sense if you try to have a timeout of 25
> > ms. In addition to that I don't know, how well coin copes with such
> > short timeout, as VMs tend to be quite slow when run from Coin (in my
> > experience).
>
> Just wait forever on the check in a loop. After 15 minutes the watchdog
> kicks in and kills the test. If that happens, you know that it's a real
> failure. This works fine for the QML debugger tests.

Since hanging randomly is a common issue in our CI, that would be
indistinguishable from a random failure.

'Allan

[Ulf Hermann]

> Since hanging randomly is a common issue in our CI, that would be
> indistinguishable from a random failure.

It rarely hangs for 15 minutes. If that actually happens we usually hear
a lot of other screaming about the CI and then you know that it might be
a false negative. I realize that just waiting in an infinite loop is
somewhat brutal, but I really don't know what else to do here.

[Edward Welbourne]

On Dienstag, 13. November 2018 09:30:11 CET Ulf Hermann wrote:
>> Just wait forever on the check in a loop. After 15 minutes the
>> watchdog kicks in and kills the test. If that happens, you know that
>> it's a real failure. This works fine for the QML debugger tests.

I encourage you to rewrite those tests.
Anything that makes Coin slower is Bad.

Allan Sandfeld Jensen (13 November 2018 09:37)
> Since hanging randomly is a common issue in our CI, that would be
> indistinguishable from a random failure.

In this case it'll actually time out in five minutes (the testlib
watchdog's time-out for a single test). A little more easily mistaken
for Coin flakiness, if anything. (I have seen ample evidence that Coin
sometimes leaves a process no run-time for five minutes; this is what
causes building of configure to fail fairly routinely.)

Note that QTRY_IMPL() first runs its loop for the specified time-out, to
see if it passes; then, if it hasn't passed, runs it for the same
time-out more; the "would have passed if" report is just there to let us
know when extending the timeout might solve the problem. No sense
waiting five minutes when we already know we've failed.

Eddy.

[Ulf Hermann]

On 11/13/18 11:30 AM, Edward Welbourne wrote:
> On Dienstag, 13. November 2018 09:30:11 CET Ulf Hermann wrote:
>>> Just wait forever on the check in a loop. After 15 minutes the
>>> watchdog kicks in and kills the test. If that happens, you know that
>>> it's a real failure. This works fine for the QML debugger tests.
>
> I encourage you to rewrite those tests.
> Anything that makes Coin slower is Bad.

How? The problem was that the OS took minutes to start a process.
However, we definitely need a target process for debugging QML.

Ulf

[Edward Welbourne]

On Dienstag, 13. November 2018 09:30:11 CET Ulf Hermann wrote:
>>>> Just wait forever on the check in a loop. After 15 minutes the
>>>> watchdog kicks in and kills the test. If that happens, you know that
>>>> it's a real failure. This works fine for the QML debugger tests.

On 11/13/18 11:30 AM, Edward Welbourne wrote:
>> I encourage you to rewrite those tests.
>> Anything that makes Coin slower is Bad.

Ulf Hermann (13 November 2018 12:02)
> How? The problem was that the OS took minutes to start a process.

I see. Painful.

> However, we definitely need a target process for debugging QML.

Right. Well, not a lot you can do about that then,

Eddy.

[Edward Welbourne]

Oliver Wolff (13 November 2018 09:05)
> I had a look at the problem when I blacklisted it for WinRT and one
> problem was, that QTRY_IMPL used a hard coded step interval of 50 ms,
> which of course does not make sense if you try to have a timeout of 25
> ms.

Thank you for bringing that to my attention:
https://codereview.qt-project.org/245375

> In addition to that I don't know, how well coin copes with such
> short timeout, as VMs tend to be quite slow when run from Coin (in my
> experience).

A reasonable concern, but it still seems like a good idea to at least use
a time-step that gives the time-out a chance of taking several steps ...

Eddy.

[Thiago Macieira]

On Sunday, 11 November 2018 11:44:12 PST Liang Qi wrote:
> The fix of the blocker got merged into qtbase 5.12. But the issue of ci/coin
> is not fixed yet. Talked with Oswald on Friday, we agreed to keep qtbase
> 5.12 lock a bit longer.

It's been a couple of weeks. Is it unlocked already?

--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center

[Liang Qi]

> On 2 Dec 2018, at 20:23, Thiago Macieira <***@intel.com> wrote:
>
> On Sunday, 11 November 2018 11:44:12 PST Liang Qi wrote:
>> The fix of the blocker got merged into qtbase 5.12. But the issue of ci/coin
>> is not fixed yet. Talked with Oswald on Friday, we agreed to keep qtbase
>> 5.12 lock a bit longer.
>
> It's been a couple of weeks. Is it unlocked already?

Nov 21 13:33:18 EET 2018
https://lists.qt-project.org/pipermail/development/2018-November/034315.html

—Liang

[Thiago Macieira]

On Sunday, 2 December 2018 12:03:32 PST Liang Qi wrote:
> > On 2 Dec 2018, at 20:23, Thiago Macieira <***@intel.com>
> > wrote:

> > On Sunday, 11 November 2018 11:44:12 PST Liang Qi wrote:
> >
> >> The fix of the blocker got merged into qtbase 5.12. But the issue of
> >> ci/coin
is not fixed yet. Talked with Oswald on Friday, we agreed to
> >> keep qtbase 5.12 lock a bit longer.
> >
> >
> > It's been a couple of weeks. Is it unlocked already?
>
>
> Nov 21 13:33:18 EET 2018
> https://lists.qt-project.org/pipermail/development/2018-November/034315.html

Thanks, I never received that email because it was in the middle of the
mailing list being misconfigured.

Someone changed the List-Id headers and reconfigured the mail server, so it
all ended up in the wrong places for me.

The List-Id headers are still wrong, but I've just adjusted my filters. No one
else complained, so it looks like I'm the only one who filters emails...
--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center

[Lorn Potter]

On 3/12/18 7:13 AM, Thiago Macieira wrote:
> The List-Id headers are still wrong, but I've just adjusted my filters. No one
> else complained, so it looks like I'm the only one who filters emails...

I noticed emails in wrong places as well, but have yet to adjust my filters.

[Patrick Stinson]

I’ve also been off about my filters after the changes

> On Dec 2, 2018, at 5:52 PM, Lorn Potter <***@gmail.com> wrote:
>
>
>
>> On 3/12/18 7:13 AM, Thiago Macieira wrote:
>> The List-Id headers are still wrong, but I've just adjusted my filters. No one
>> else complained, so it looks like I'm the only one who filters emails...
>
> I noticed emails in wrong places as well, but have yet to adjust my filters.
> _______________________________________________
> Development mailing list
> ***@lists.qt-project.org
> https://lists.qt-project.org/listinfo/development