[Development] Switch Qt Remote Objects to a Tech Preview for Qt 5.9

[Stottlemyer, Brett (B.S.)]

As the maintainer for the Qt Remote Objects (QtRO) playground project, I would like to officially request moving it from a playground project to a Qt project. For now (Qt 5.9), I’d like to keep it as a Tech Preview, as there are some elements of the API we would still like to extend, and we’d certainly like to protect for changes based on additional feedback.

In a nutshell, Qt Remote Objects takes the mechanisms that Qt already provides for multithreaded processing (Queued connections, marshaling arguments) and extends them to allow QObjects to work across process and/or device boundaries. It does this by allowing a Source object (with a full implementation of the Qt Properties, Signals and Slots) to be shared on the QtRO bus. Replica versions of this object can be acquired, where QtRO handles sending any changes to the Source to all Replicas. I think of the Replicas as “latent copies” of the original object. This seems like a great extension to Qt, and based on discussions at the last Qt World Summit, one that is makes sense to have as an official Qt module.

From a maturity standpoint, it has been running as a playground project for several years now. We have several platforms checked via our own CI, but would like to get coverage on all supported platforms and will certainly address any issues found. Giuseppe D’Angelo was kind enough to add QtRO to the Qt LTS Coverity scans, and except for several issues tied to a copy of moc, it is in good shape. It supports qmake docs, although the docs need some care and feeding (this is ok for a Tech Preview as I understand it).

If this request is well received, I do have one sub-request. According to http://wiki.qt.io/Creating_a_new_module_or_tool_for_Qt, “the playground name needs to be changed to a descriptive Qt module or tool name.” The current name is qtremoteobjects. Are there issues keeping this name?

Thanks for your consideration,
Brett

[Oswald Buddenhagen]

On Mon, Jan 09, 2017 at 02:50:52PM +0000, Stottlemyer, Brett (B.S.) wrote:
> As the maintainer for the Qt Remote Objects (QtRO) playground project,
> I would like to officially request moving it from a playground project
> to a Qt project.
>
i had a quick look at the repo:
- there is still copy of moc's c++ parser in there. not much to do about
it at this point, except that repc is mostly redundant with moc to
start with.
- why not make it an output mode of moc? or at least, it would seem
sensible to make moc output an intermediate json file with the meta
info it extracted, and still leave the code generation to repc -
that would remove the double parsing, and always ensure consistency.
- i see that repc creates qdatastream operators for q_enums & co. -
that *really* seems like generic moc stuff.
- please verify that improvements to moc and the meta type system have
not obsoleted some repc functionality in the first place.
- was that promised discussion vs. qtsystems actually ever had?
- formalities:
- lots of build system hacks for obsolete qt versions
- lagging in coding style practice (replace foreach, other c++11
stuff)

[Stottlemyer, Brett (B.S.)]

On 1/9/17, 10:54 AM, "Development on behalf of Oswald Buddenhagen" <development-bounces+bstottle=***@qt-project.org on behalf of ***@qt.io> wrote:



>i had a quick look at the repo:
>- there is still copy of moc's c++ parser in there. not much to do about
> it at this point, except that repc is mostly redundant with moc to
> start with.

This is a direction I hadn’t really considered. I don’t like duplicating
moc code, but my thought was to put repc into qtbase (if that would be
accepted. Thiago?) and keep the rest of QtRO in a separate repository.

The primary job of repc is to generate QObject code, and the mkspecs add
those automatically for moc processing. I.e, it takes the API definition,
in .rep format, and generates an asymmetric pair of files, one for the
Source and one for the Replica.

The processing of QObject code by repc to generate the .rep input file
format was added afterwards. It allows existing Qt header files to be
used with QtRO with compile time checks. We don’t use that feature much
internally, but others are.

> - why not make it an output mode of moc? or at least, it would seem
> sensible to make moc output an intermediate json file with the meta
> info it extracted, and still leave the code generation to repc -
> that would remove the double parsing, and always ensure consistency.

If you are willing to take the .rep format as a moc input (?) this might
work. If your thought is to take a QObject header file as input, I’m not
sure it does. As a concrete example, Q_PROPERTY on the Source side can be
CONSTANT, but on the Replica side it can’t (it can have a default value
which is then updated when the connection to the Source is established.)
This asymmetry seems like it would be very confusing if it was output by
moc.

> - i see that repc creates qdatastream operators for q_enums & co. -
> that *really* seems like generic moc stuff.

Only in certain cases. It lets moc handle this if possible. I’ll have to
look at this again, the intent was to let moc do as much as possible.

> - please verify that improvements to moc and the meta type system have
> not obsoleted some repc functionality in the first place.

Pretty sure that for the primary use case they don’t, but if you can point
me to specific enhancements to look at, that would be appreciated.

>- was that promised discussion vs. qtsystems actually ever had?

No, but I don’t remember promising to have the discussion. I know there
was a proposal to look for commonality with web sockets. That didn’t
happen either. Are these blockers?

>- formalities:
> - lots of build system hacks for obsolete qt versions

To be honest, Ossi, you see stuff instantly that I need to study in the
build system. We can keep working on it, but patches would be welcomed.

> - lagging in coding style practice (replace foreach, other c++11
> stuff)

Yes, we have been on LTS, so there are necessary changes for 5.7+. Are
these needed before feature freeze? Or can they be addressed as we move
to 5.9?

Overall, I can’t quite tell from your response which items are nice to
have, need to have (by feature freeze) or need to have (by 5.9 release).
I’d appreciate clarification.

Thanks, and thanks for the review,
Brett

[Oswald Buddenhagen]

On Tue, Jan 10, 2017 at 01:42:12AM +0000, Stottlemyer, Brett (B.S.) wrote:
> The processing of QObject code by repc to generate the .rep input file
> format was added afterwards. It allows existing Qt header files to be
> used with QtRO with compile time checks. We don’t use that feature much
> internally, but others are.
>
you mostly lost me here, because i just don't know enough about the data
flows of the qtro build system, so it would be helpful if you outlined
that.

but my naive understanding of rpc implementations is that you actually
want to create some idl (is this what .rep is about?) from the c++, and
in a later step compile that into stubs and skeletons. the former sounds
like a task moc could do as a side effect, while the latter is for repc.

as it happens, we actually have that in qtbase already, in form of the
dbus tools. and as expected, qdbuscpp2xml is yet another tool that pulls
in moc sources ...

and obviously, moc itself could also be split into two tools.
or, the moc frontend could be replaced with clang, for which code
actually already exists, but wasn't deemed quite ready for prime time
yet. maybe we could actually pull that off now that qdoc is switching
to clang as well.

soliciting input primarily from olivier and thiago here ...

> > - please verify that improvements to moc and the meta type system have
> > not obsoleted some repc functionality in the first place.
>
> Pretty sure that for the primary use case they don’t, but if you can point
> me to specific enhancements to look at, that would be appreciated.
>
i'll leave it for others to comment on.

> Overall, I can’t quite tell from your response which items are nice to
> have, need to have (by feature freeze) or need to have (by 5.9
> release). I’d appreciate clarification.
>
it's in the interest of the qt project to unify the offering as much as
possible, minimizing redundant apis and implementations. the former are
by definition before-freeze items, while the latter are as far as they
cannot be done without impacting the api. (api being used in the
broadest sense here, so tools are obviously also covered.)

[Stottlemyer, Brett (B.S.)]

On 1/10/17, 7:11 AM, "Development on behalf of Oswald Buddenhagen" <development-bounces+bstottle=***@qt-project.org on behalf of ***@qt.io> wrote:



>On Tue, Jan 10, 2017 at 01:42:12AM +0000, Stottlemyer, Brett (B.S.) wrote:
>> The processing of QObject code by repc to generate the .rep input file
>> format was added afterwards. It allows existing Qt header files to be
>> used with QtRO with compile time checks. We don’t use that feature much
>> internally, but others are.
>>
>you mostly lost me here, because i just don't know enough about the data
>flows of the qtro build system, so it would be helpful if you outlined
>that.

Sure. I guess to start off, I don’t consider QtRO to be a RPC mechanism.
In my mind, RPC would be akin to exposing a QObject’s slots for calling
individually. With QtRO (which only works Qt to Qt, it isn’t currently
available to other “languages”, like ruby/python/etc.) you talk QObjects.

A Source object is added to the network, and clients (although I’m not
sure that is the right term) can then get a Replica of that object. Any
change to the Source object (Signals emitted, or Q_Properties changed)
then get communicated to every Replica. A Slot call is passed back to
the Source for handling, with any “side-effects” forwarded to all Replicas.

With this picture in mind, the key to getting it to work is to hook into
qt_metacall and pass the invocations between processes. This takes custom
code, so QtRO has repc (REPlica Compiler) for this. It reads a .rep file,
a simple text format for describing the object’s API via Signals, Slot,
Properties and Enums, and generates a Qt header file for the Source and
Replica sides. The mkspecs then add these files for moc to process.

At some level, I see the .rep format as an extreme of a Q_OBJECT definition
in that only the “API” is defined (property, signal, slot, enum), and
everything else is private/pimpl. Specific to QtRO, the Source may differ
from the Replica in that Q_PROPERTYs may be CONSTANT on the Source, but
still have a notify on the Replica. The reason for this is that we want
Replica objects to have default values for properties that are valid
before the connection is established, and then pick up the “real” value
once the connection is made.

>but my naive understanding of rpc implementations is that you actually
>want to create some idl (is this what .rep is about?) from the c++, and
>in a later step compile that into stubs and skeletons. the former sounds
>like a task moc could do as a side effect, while the latter is for repc.

I see it as the opposite for QtRO. The .rep defines the API, repc creates
the Qt headers for the types, then moc generates the code based on the
headers.

>as it happens, we actually have that in qtbase already, in form of the
>dbus tools. and as expected, qdbuscpp2xml is yet another tool that pulls
>in moc sources ...
>
>and obviously, moc itself could also be split into two tools.
>or, the moc frontend could be replaced with clang, for which code
>actually already exists, but wasn't deemed quite ready for prime time
>yet. maybe we could actually pull that off now that qdoc is switching
>to clang as well.

I'm interested to know if you think this is still relevant based on my
reply. I’m not really sure how to answer.

>soliciting input primarily from olivier and thiago here ...

Welcomed.

>it's in the interest of the qt project to unify the offering as much as
>possible, minimizing redundant apis and implementations. the former are
>by definition before-freeze items, while the latter are as far as they
>cannot be done without impacting the api. (api being used in the
>broadest sense here, so tools are obviously also covered.)

Completely agree. At the same time, I don’t want to miss the deadline for
feature freeze, so I’m hoping to get past that hurdle and address some of
your proposals afterwards as a tech preview.

As a playground project I think we haven’t gotten a lot of attention from
The Qt Company and I really do welcome that. The repc tool specifically is
kind of in a weird spot. I’m not sure there are any other tools that are
bootstrapped and aren’t in qtbase. Should repc* not be bootstrapped?
Alternatively, should it be part of qtbase (so moc doesn’t need to be
“3rdParty”)? Until QtRO is promoted to a Qt module, I don’t think these
questions can really be answered.

*QtRO will do type checking at compile time if an API gets generated from
repc. You can also “share” any QObject via QtRO, but the Replica is then
dynamically generated and needs runtime checks. The repc program includes
moc to it can generate a .rep file for any Qobject - thus supporting
compile time checks from existing Qt header files. Oddly enough, the
generated code is then passed through moc again. This is the “second”
use-case I mentioned. I think it is a valid use case, but one I don’t use
much.

Hope that helps,
Brett

[Stottlemyer, Brett (B.S.)]

On 1/10/17, 7:42 PM, "Stottlemyer, Brett (B.S.)" <***@ford.com> wrote:



>On 1/10/17, 7:11 AM, "Development on behalf of Oswald Buddenhagen" <development-bounces+bstottle=***@qt-project.org on behalf of ***@qt.io> wrote:
>
>>you mostly lost me here, because i just don't know enough about the data
>>flows of the qtro build system, so it would be helpful if you outlined
>>that.

...

>*QtRO will do type checking at compile time if an API gets generated from
>repc.

Maybe it is worth expanding on this a little bit. If you are using QtRO from
the beginning of a project, the idea would be for the owner of an interface
to define the API their object will use in a .rep file. In the .pro file,
REPC_SOURCE += someInterface.rep is added, which causes repc to generate the
Source side header file, which provides a QObject class for that API with
pure virtual methods for Slots. The developer then inherits from the base
API class and implements the internal logic to meet the API. Side note:
additional Qt Signals/Slots can be added to the derived class, which aren’t
exposed to Replicas if they aren’t in the .rep file.

Anyone that wants to use this interface would add
REPC_REPLICA += someInterface
to their .pro file, which generates (you guessed it) the Replica side header
file. It is possible to inherit from this class, but usually isn’t necessary
and the class/code can be used directly.

The reason for having moc code as part of repc is to handle the use-case of
someone having an already developed QObject type and wanting to use that
with QtRO. I.e., a class that doesn’t derive from the auto generated
Source class. In this case, moc is used to parse the header, and generate
the .rep for your class. The .rep file can then be used by clients to
get Replicas.

There is another level of detail in QtRO for efficiency and type safety (see
the auto generated someInterfaceSourceAPI template in the Source header).
Feel free to look at that if you want more detail.

Regards,
Brett

[Marc Mutz]

Hi Brett,

On Wednesday 11 January 2017 01:42:38 Stottlemyer, Brett (B.S.) wrote:
> With this picture in mind, the key to getting it to work is to hook into
> qt_metacall and pass the invocations between processes. This takes custom
> code, so QtRO has repc (REPlica Compiler) for this. It reads a .rep file,
> a simple text format for describing the object’s API via Signals, Slot,
> Properties and Enums, and generates a Qt header file for the Source and
> Replica sides. The mkspecs then add these files for moc to process.

Speaking as someone who has almost no clue about QtRO, from an engineering
pov, it feels weird to have an easily-parsable representation of a
QMetaObject, then generate C++ code from it, and then use moc to parse the C++
code to generate a QMetaObject. Wouldn't it be much simpler to generate the
moc files directly from the rep files (maybe even at runtime, with
QMetaObjectBuilder)?

Thanks,
Marc

--
Marc Mutz <***@kdab.com> | Senior Software Engineer
KDAB (Deutschland) GmbH & Co.KG, a KDAB Group Company
Tel: [phone]
KDAB - The Qt, C++ and OpenGL Experts

[Stottlemyer, Brett (B.S.)]

On 1/11/17, 4:43 AM, "***@kdab.com on behalf of Marc Mutz" <***@kdab.com on behalf of ***@kdab.com> wrote:



>Hi Brett,
>
>On Wednesday 11 January 2017 01:42:38 Stottlemyer, Brett (B.S.) wrote:
>> With this picture in mind, the key to getting it to work is to hook into
>> qt_metacall and pass the invocations between processes. This takes custom
>> code, so QtRO has repc (REPlica Compiler) for this. It reads a .rep file,
>> a simple text format for describing the object’s API via Signals, Slot,
>> Properties and Enums, and generates a Qt header file for the Source and
>> Replica sides. The mkspecs then add these files for moc to process.
>
>Speaking as someone who has almost no clue about QtRO, from an engineering
>pov, it feels weird to have an easily-parsable representation of a
>QMetaObject, then generate C++ code from it, and then use moc to parse the C++
>code to generate a QMetaObject. Wouldn't it be much simpler to generate the
>moc files directly from the rep files (maybe even at runtime, with
>QMetaObjectBuilder)?

Hi Marc. I was not trying to give a comprehensive overview (that what docs
are for, right?) just trying to address Ossi’s comments. QtRO does provide
for pushing a standard QObject out on the QtRO network and getting a “dynamic”
Replica of it without needing repc/moc. But while this is possible, it is
a bit painful to work with. The assumption is that the Replica is in a
different process than the QObject it represents. The Replica will match the
API, but it needs extra plumbing and is a specific type (a mouthful,
QRemoteObjectDynamicReplica). It can’t know anything about itself until it
gets information from QtRO about the object is should represent, and then you
need to (in code) look up the methods/properties from the runtime generated
metaobject. There are no compile time checks against anything. Works
reasonably well from QML, but as said, hard from C++. The additional mkspecs
in QtRO hide the additional steps almost completely, and the generated code
provides better performance, the ability to have default values until the
connection is made, and type-safety. Both options are available, though.

Regards,
Brett

[Oswald Buddenhagen]

On Wed, Jan 11, 2017 at 12:42:38AM +0000, Stottlemyer, Brett (B.S.) wrote:
> I guess to start off, I don’t consider QtRO to be a RPC mechanism.
> In my mind, RPC would be akin to exposing a QObject’s slots for calling
> individually. With QtRO (which only works Qt to Qt, it isn’t currently
> available to other “languages”, like ruby/python/etc.) you talk QObjects.
>
technically speaking, it's a distributed object system (which puts it
into the same category as CORBA), but the underlying mechanism is still
rpc.

> At some level, I see the .rep format as an extreme of a Q_OBJECT definition
> in that only the “API” is defined (property, signal, slot, enum), and
> everything else is private/pimpl.
>
yes, that makes .rep an idl format.

> >but my naive understanding of rpc implementations is that you actually
> >want to create some idl (is this what .rep is about?) from the c++, and
> >in a later step compile that into stubs and skeletons. the former sounds
> >like a task moc could do as a side effect, while the latter is for repc.
>
> I see it as the opposite for QtRO. The .rep defines the API, repc creates
> the Qt headers for the types,
>
i'm not sure how it's the opposite, when you effectively confirmed what
i speculated.

> then moc generates the code based on the headers.
>
that's just a rather convoluted way to achieve the same, as marc noted
as well.

> I don’t want to miss the deadline for feature freeze, so I’m hoping to
> get past that hurdle and address some of your proposals afterwards as
> a tech preview.
>
that seems a bit optimistic, given that we're discussing some rather
fundamental aspects of the design.

also, i don't think we've been bothering a lot about deadlines for TP
modules, so who cares.

> As a playground project I think we haven’t gotten a lot of attention
> from The Qt Company and I really do welcome that.
>
from experience, this isn't going to change. it's on you to reach out
(preferably without imposing a deadline).
it's your project, and it will be released in whatever state it is when
you deem it ready.

and if later somebody else with resources suddenly finds it interesting
but finds it to suck a lot, there will be struggling to get it into
shape, or to get rid of it again, to push for a better solution. we've
been there before.

it doesn't help that you apparently haven't even explored the prior art
within qt even though it was pointed out to you two years ago already.

> The repc tool specifically is kind of in a weird spot. I’m not sure
> there are any other tools that are bootstrapped and aren’t in qtbase.
> Should repc* not be bootstrapped? Alternatively, should it be part of
> qtbase (so moc doesn’t need to be “3rdParty”)? Until QtRO is promoted
> to a Qt module, I don’t think these questions can really be answered.
>
there is nothing extraordinary about repc from the buildsystem
perspective.
there are some things to fix, but this is not place to discuss them.
other paragraphs are concerned with the duplication of moc.

> >>*QtRO will do type checking at compile time if an API gets generated from
> >>repc.
>
> Maybe it is worth expanding on this a little bit. [...]
>
afaict, you just described *precisely* what qdbuscpp2xml and
qdbusxml2cpp do, except that you decided to put both tools into one
executable.

the part that is shared with moc is the c++ parser, and as expressed in
the previous mail, the hope is to get rid of that entirely in favor of
clang.

that leaves us with extracting the relevant meta data from the AST, the
intermediate storage of the meta data (the idl), and the code generator.
this is all relatively little code, so sharing *as such* doesn't seem
too useful. however, it's quite ineffcient that the c++ parsing is done
multiple times, so a unified meta data processing pipeline appears
preferable:
- moc parses the c++ to create an idl file that contains all qt-specific
meta data (what it should extract and how it should represent it can
be declared in the headers it includes, so in principle it's possible
to make it quite generic)
- alternatively, the user provides the idl file (both qtro and qtdbus
support that already)
- the idl file is fed to various code generators
- the code generated by repc is not fed back into moc for parsing from
scratch, but instead repc creates a modified idl which is fed directly
into the moc code generator
- come to think of it, qtdbus also has *precisely the same* problem
(see qtbase/mkspecs/features/dbus*). you really managed to replicate
it down to its warts. ;)

a concrete action point would be exploring in how far it's possible (and
actually sensible) to design a shared meta data representation (idl).
qtdbus' xml is horrible (of course it is), but it's easy to process.
repc's "proper" idl is the exact opposite ...

ah, yeah, qtsystems. have a look and report. ;)

just some food for thought. ;)

[Tuukka Turunen]

> -----Original Message-----
> From: Development [mailto:development-
> bounces+tuukka.turunen=***@qt-project.org] On Behalf Of Oswald
> Buddenhagen
> Sent: keskiviikkona 11. tammikuuta 2017 18.50
> To: ***@qt-project.org
> Subject: Re: [Development] Switch Qt Remote Objects to a Tech Preview for
> Qt 5.9
>
> > I don’t want to miss the deadline for feature freeze, so I’m hoping to
> > get past that hurdle and address some of your proposals afterwards as
> > a tech preview.
> >
> that seems a bit optimistic, given that we're discussing some rather
> fundamental aspects of the design.
>
> also, i don't think we've been bothering a lot about deadlines for TP modules,
> so who cares.
>

As this is a module to be added, we need to have it in place latest at the FF. Preferably modules should be in place earlier than FF.

The criteria for the content of the module is of course less strict when it is a TP, but still if it is likely to need full refactoring perhaps waiting for 5.10 is a better option?

Yours,

Tuukka

[Stottlemyer, Brett (B.S.)]

On 1/11/17, 11:50 AM, "Development on behalf of Oswald Buddenhagen" <development-bounces+bstottle=***@qt-project.org on behalf of ***@qt.io> wrote:


>> >but my naive understanding of rpc implementations is that you actually
>> >want to create some idl (is this what .rep is about?) from the c++, and
>> >in a later step compile that into stubs and skeletons. the former sounds
>> >like a task moc could do as a side effect, while the latter is for repc.
>>
>> I see it as the opposite for QtRO. The .rep defines the API, repc creates
>> the Qt headers for the types,
>>
>i'm not sure how it's the opposite, when you effectively confirmed what
>i speculated.

To clarify, I simply meant that you described c++ -> idl, while the normal
process for QtRO is idl -> c++.

>> I don’t want to miss the deadline for feature freeze, so I’m hoping to
>> get past that hurdle and address some of your proposals afterwards as
>> a tech preview.
>>
>that seems a bit optimistic, given that we're discussing some rather
>fundamental aspects of the design.

This is a bit dramatic, isn’t it? Unless I misinterpret, you are not
suggesting the generated code change, just the process for generating it.
As such, it would seem that any change would be a drop-in replacement, so
I don’t see any criticality to the timing, especially for a TP module.

It also seems like your main objection is the use of moc, which is (as
I’ve already said) an edge case. If this is the key sticking point, I
can drop support for converting QObjects to .rep files and remove moc
from the repo.

>also, i don't think we've been bothering a lot about deadlines for TP
>modules, so who cares.

Apparently Tuukka does, since he’s suggesting holding off until 5.10...

>it doesn't help that you apparently haven't even explored the prior art
>within qt even though it was pointed out to you two years ago already.

Looking again now, I believe you are talking about Qt Service Framework. It
does look familiar, so I probably looked at it and discarded it as solving a
separate problem, then forgotten I’d looked.

Top hits on google are

https://blog.qt.io/blog/2009/05/26/qt-service-framework/
https://doc-snapshots.qt.io/qt-mobility/service-frameworks.html

And
https://wiki.qt.io/Service_Framework_API


It provides platform specific implementations of services/daemons with a
common Qt API. “Once the service has been identified the framework starts
the service and returns a pointer to it.” This isn’t the use-case I’m
trying to solve. It doesn’t work between platforms (like QtRO does with
the tcp/ip backend), doesn’t support Q_PROPERTY, doesn’t support default
values, and assumes services can be started from any user code. It is RPC,
not a distributed object system. I don’t consider it “prior art”, except
at a "rectangular device with rounded corners" kind of level.

>afaict, you just described *precisely* what qdbuscpp2xml and
>qdbusxml2cpp do, except that you decided to put both tools into one
>executable.

Right, I DO consider the dbus code to be prior art, in that is solves a
very similar problem, so I leveraged the pieces that I could.

>the part that is shared with moc is the c++ parser, and as expressed in
>the previous mail, the hope is to get rid of that entirely in favor of
>clang.
>
>that leaves us with extracting the relevant meta data from the AST, the
>intermediate storage of the meta data (the idl), and the code generator.
>this is all relatively little code, so sharing *as such* doesn't seem
>too useful. however, it's quite ineffcient that the c++ parsing is done
>multiple times, so a unified meta data processing pipeline appears
>preferable:
>- moc parses the c++ to create an idl file that contains all qt-specific
> meta data (what it should extract and how it should represent it can
> be declared in the headers it includes, so in principle it's possible
> to make it quite generic)
>- alternatively, the user provides the idl file (both qtro and qtdbus
> support that already)
>- the idl file is fed to various code generators
>- the code generated by repc is not fed back into moc for parsing from
> scratch, but instead repc creates a modified idl which is fed directly
> into the moc code generator
> - come to think of it, qtdbus also has *precisely the same* problem
> (see qtbase/mkspecs/features/dbus*). you really managed to replicate
> it down to its warts. ;)
>
>a concrete action point would be exploring in how far it's possible (and
>actually sensible) to design a shared meta data representation (idl).
>qtdbus' xml is horrible (of course it is), but it's easy to process.
>repc's "proper" idl is the exact opposite ...

Is it reasonable to assert that all of the above should either be needed for
both QtRO and QtDbus or neither? If so, I’ll drop the moc support from QtRO
and re-add it once the above is done for qtdbus.

>ah, yeah, qtsystems. have a look and report. ;)

See above.

>just some food for thought. ;)

Appreciated. With removal of moc, can we proceed as a Tech Preview in 5.9?

Regards,
Brett

[Lars Knoll]

I didn't find time to have a more detailed look at the code so far, but hope to get to it over the weekend.

But in general I think we need to put the bars for becoming a TP at the right level. It's not a fully supported module yet, and we put it out especially to collect feedback. So it doesn't have to be 100% at the same level as supported modules. Here are the criteria I think we should have (and that we IMO implicitly used in the past):

* The module solves a problem our users have
- It either implements new and so far non existent functionality
- Or it solves an existing problem in a new and better way (and is intended to replace the old functionality over time)
* The module builds inside the Qt build system on all platforms
- Compilation could be disabled on some platforms, but is not allowed to break any platform we support
* The module is CI controlled
* There is a decent set of automated tests covering the main functionality and most of the API of the module
- The tests pass on all platforms where the module is supported
* APIs have been reviewed, and we are reasonably satisfied with them (we've done larger changes to APIs after TPs before)
* Architecture makes sense
* It follows the Qt coding style and conventions
* Implementation has been checked for sanity
- It's ok to have parts that are flagged as needing further work (those have to be fixed before moving from TP to supported state)

IMO that should cover most of the needs and demands we should have for new modules before offering them as a TP.

From the discussion so far I didn't hear too many things that speak against a TP, the code duplication with moc is one of the issues that fall into the 'flagged and need to be resolved before moving out of TP' category for me. How about the other points in the list above that haven't been discussed yet here?

Cheers,
Lars

On 12 Jan 2017, at 02:19, Stottlemyer, Brett (B.S.) <***@ford.com<mailto:***@ford.com>> wrote:

On 1/11/17, 11:50 AM, "Development on behalf of Oswald Buddenhagen" <development-bounces+bstottle=***@qt-project.org<mailto:development-bounces+bstottle=***@qt-project.org> on behalf of ***@qt.io<mailto:***@qt.io>> wrote:


but my naive understanding of rpc implementations is that you actually
want to create some idl (is this what .rep is about?) from the c++, and
in a later step compile that into stubs and skeletons. the former sounds
like a task moc could do as a side effect, while the latter is for repc.

I see it as the opposite for QtRO. The .rep defines the API, repc creates
the Qt headers for the types,

i'm not sure how it's the opposite, when you effectively confirmed what
i speculated.

To clarify, I simply meant that you described c++ -> idl, while the normal
process for QtRO is idl -> c++.

I don’t want to miss the deadline for feature freeze, so I’m hoping to
get past that hurdle and address some of your proposals afterwards as
a tech preview.

that seems a bit optimistic, given that we're discussing some rather
fundamental aspects of the design.

This is a bit dramatic, isn’t it? Unless I misinterpret, you are not
suggesting the generated code change, just the process for generating it.
As such, it would seem that any change would be a drop-in replacement, so
I don’t see any criticality to the timing, especially for a TP module.

It also seems like your main objection is the use of moc, which is (as
I’ve already said) an edge case. If this is the key sticking point, I
can drop support for converting QObjects to .rep files and remove moc
from the repo.

also, i don't think we've been bothering a lot about deadlines for TP
modules, so who cares.

Apparently Tuukka does, since he’s suggesting holding off until 5.10...

it doesn't help that you apparently haven't even explored the prior art
within qt even though it was pointed out to you two years ago already.

Looking again now, I believe you are talking about Qt Service Framework. It
does look familiar, so I probably looked at it and discarded it as solving a
separate problem, then forgotten I’d looked.

Top hits on google are

https://blog.qt.io/blog/2009/05/26/qt-service-framework/
https://doc-snapshots.qt.io/qt-mobility/service-frameworks.html

And
https://wiki.qt.io/Service_Framework_API


It provides platform specific implementations of services/daemons with a
common Qt API. “Once the service has been identified the framework starts
the service and returns a pointer to it.” This isn’t the use-case I’m
trying to solve. It doesn’t work between platforms (like QtRO does with
the tcp/ip backend), doesn’t support Q_PROPERTY, doesn’t support default
values, and assumes services can be started from any user code. It is RPC,
not a distributed object system. I don’t consider it “prior art”, except
at a "rectangular device with rounded corners" kind of level.

afaict, you just described *precisely* what qdbuscpp2xml and
qdbusxml2cpp do, except that you decided to put both tools into one
executable.

Right, I DO consider the dbus code to be prior art, in that is solves a
very similar problem, so I leveraged the pieces that I could.

the part that is shared with moc is the c++ parser, and as expressed in
the previous mail, the hope is to get rid of that entirely in favor of
clang.

that leaves us with extracting the relevant meta data from the AST, the
intermediate storage of the meta data (the idl), and the code generator.
this is all relatively little code, so sharing *as such* doesn't seem
too useful. however, it's quite ineffcient that the c++ parsing is done
multiple times, so a unified meta data processing pipeline appears
preferable:
- moc parses the c++ to create an idl file that contains all qt-specific
meta data (what it should extract and how it should represent it can
be declared in the headers it includes, so in principle it's possible
to make it quite generic)
- alternatively, the user provides the idl file (both qtro and qtdbus
support that already)
- the idl file is fed to various code generators
- the code generated by repc is not fed back into moc for parsing from
scratch, but instead repc creates a modified idl which is fed directly
into the moc code generator
- come to think of it, qtdbus also has *precisely the same* problem
(see qtbase/mkspecs/features/dbus*). you really managed to replicate
it down to its warts. ;)

a concrete action point would be exploring in how far it's possible (and
actually sensible) to design a shared meta data representation (idl).
qtdbus' xml is horrible (of course it is), but it's easy to process.
repc's "proper" idl is the exact opposite ...

Is it reasonable to assert that all of the above should either be needed for
both QtRO and QtDbus or neither? If so, I’ll drop the moc support from QtRO
and re-add it once the above is done for qtdbus.

ah, yeah, qtsystems. have a look and report. ;)

See above.

just some food for thought. ;)

Appreciated. With removal of moc, can we proceed as a Tech Preview in 5.9?

Regards,
Brett

_______________________________________________
Development mailing list
***@qt-project.org<mailto:***@qt-project.org>
http://lists.qt-project.org/mailman/listinfo/development

[Edward Welbourne]

On 12 January 2017 at 08:39, Lars Knoll wrote:
> Here are the criteria I think we should have (and that we IMO implicitly used in the past):

This smells like something we should be turning into a QUIP.

Eddy.

[Stottlemyer, Brett (B.S.)]

On 12 January 2017 at 08:39, Lars Knoll wrote:

>From the discussion so far I didn't hear too many things that speak against a TP, the code duplication with moc is one of the issues that fall into the 'flagged and need to be resolved before moving out of TP' category for me. How about the other
>points in the list above that haven't been discussed yet here?

Moving the points to below instead of above. My opinion is (obviously) biased.

>* The module solves a problem our users have
> - It either implements new and so far non existent functionality
> - Or it solves an existing problem in a new and better way (and is intended to replace the old functionality over time)

It seems like the two potentially competing current projects are Qt Service Framework and QtDBus. I believe QtRO is
different from both, with “better” being subjective. At the risk of being flamed for why I prefer QtRO, I’d put it
this way:

The Qt Service Framework deals with OS Services/daemons, and only works inter-process, not inter-device. If I can
distinguish between a service and a Service, Service being a different class than a program (I.e., on windows, services
run when no user is logged on). If you need a Service, the Service Framework makes sense. This isn’t the problem
QtRO is trying to solve.

QtDBus is more similar in intent to QtRO. QtRO is peer-to-peer, while Dbus by default goes through a daemon. Bus is
common on Linux platforms, but has additional dependencies on other platforms (Windows, QNX, etc). The key difference
though, is that DBus is intended to translate between languages (java, C, python). This is a disadvantage of QtRO
in many cases, but when Qt is available for both client/server, it provides a much richer level of communication. For
instance, QtRO supports QAIM, including selection, across processes/devices. It supports state (current PROPERTY
values). It supports any type Qt can (or can be made to) pass in Queued connections. Without additional code.

My view is that QtRO complements the other two options, providing strong benefits for the right use-cases.

>* The module builds inside the Qt build system on all platforms
> - Compilation could be disabled on some platforms, but is not allowed to break any platform we support
>* The module is CI controlled

Umm, this feels like a chicken and egg problem to me. The Qt Company controls the CI, and it doesn’t run against play-
ground projects. We have worked with KDAB and implemented CI on 4 platforms, targeting only Qt 5.6.2 currently. Until
Qt’s CI is applied to it, there isn’t a clear way to evaluate build issues. I’ve manually built against 5.8.0rc1 but
not on all platforms. I feel like there needs to be a way to enable Qt’s CI against QtRO before deciding whether to
accept it as a TP project. Can the Qt Company CI team help with this? It is a playground project already hosted on
qt.io. If not, what options are there?

>* There is a decent set of automated tests covering the main functionality and most of the API of the module
> - The tests pass on all platforms where the module is supported
>* APIs have been reviewed, and we are reasonably satisfied with them (we've done larger changes to APIs after TPs before)
>* Architecture makes sense
>* It follows the Qt coding style and conventions
>* Implementation has been checked for sanity
> - It's ok to have parts that are flagged as needing further work (those have to be fixed before moving from TP to supported state)

For all of the above, it seems like you need someone other than the maintainer to assess whether it meets your requirements.
I believe it does. In terms of coding style, it meets the standards for 5.6.2. We still need to apply the C++11 standards
for 5.7+. Also, I’m hoping (if approved) for a recommendation on how keep the Gerrit review history and align with Qt branching.

Regards,
Brett

[Lars Knoll]

Hi Brett,


> On 13 Jan 2017, at 02:58, Stottlemyer, Brett (B.S.) <***@ford.com> wrote:
>
> On 12 January 2017 at 08:39, Lars Knoll wrote:
>
>> From the discussion so far I didn't hear too many things that speak against a TP, the code duplication with moc is one of the issues that fall into the 'flagged and need to be resolved before moving out of TP' category for me. How about the other
>> points in the list above that haven't been discussed yet here?
>
> Moving the points to below instead of above. My opinion is (obviously) biased.
>
>> * The module solves a problem our users have
>> - It either implements new and so far non existent functionality
>> - Or it solves an existing problem in a new and better way (and is intended to replace the old functionality over time)
>
> It seems like the two potentially competing current projects are Qt Service Framework and QtDBus. I believe QtRO is
> different from both, with “better” being subjective. At the risk of being flamed for why I prefer QtRO, I’d put it
> this way:
>
> The Qt Service Framework deals with OS Services/daemons, and only works inter-process, not inter-device. If I can
> distinguish between a service and a Service, Service being a different class than a program (I.e., on windows, services
> run when no user is logged on). If you need a Service, the Service Framework makes sense. This isn’t the problem
> QtRO is trying to solve.
>
> QtDBus is more similar in intent to QtRO. QtRO is peer-to-peer, while Dbus by default goes through a daemon. Bus is
> common on Linux platforms, but has additional dependencies on other platforms (Windows, QNX, etc). The key difference
> though, is that DBus is intended to translate between languages (java, C, python). This is a disadvantage of QtRO
> in many cases, but when Qt is available for both client/server, it provides a much richer level of communication. For
> instance, QtRO supports QAIM, including selection, across processes/devices. It supports state (current PROPERTY
> values). It supports any type Qt can (or can be made to) pass in Queued connections. Without additional code.
>
> My view is that QtRO complements the other two options, providing strong benefits for the right use-cases.

Agreed.
>
>> * The module builds inside the Qt build system on all platforms
>> - Compilation could be disabled on some platforms, but is not allowed to break any platform we support
>> * The module is CI controlled
>
> Umm, this feels like a chicken and egg problem to me. The Qt Company controls the CI, and it doesn’t run against play-
> ground projects. We have worked with KDAB and implemented CI on 4 platforms, targeting only Qt 5.6.2 currently. Until
> Qt’s CI is applied to it, there isn’t a clear way to evaluate build issues. I’ve manually built against 5.8.0rc1 but
> not on all platforms. I feel like there needs to be a way to enable Qt’s CI against QtRO before deciding whether to
> accept it as a TP project. Can the Qt Company CI team help with this? It is a playground project already hosted on
> qt.io. If not, what options are there?

It should be pretty straightforward to enable CI on the module. Pre-conditions are mainly that it’s structured like the other Qt modules. Frederik & Simon what would it take to enable CI on it?
>
>> * There is a decent set of automated tests covering the main functionality and most of the API of the module
>> - The tests pass on all platforms where the module is supported
>> * APIs have been reviewed, and we are reasonably satisfied with them (we've done larger changes to APIs after TPs before)
>> * Architecture makes sense
>> * It follows the Qt coding style and conventions
>> * Implementation has been checked for sanity
>> - It's ok to have parts that are flagged as needing further work (those have to be fixed before moving from TP to supported state)
>
> For all of the above, it seems like you need someone other than the maintainer to assess whether it meets your requirements.
> I believe it does. In terms of coding style, it meets the standards for 5.6.2. We still need to apply the C++11 standards
> for 5.7+. Also, I’m hoping (if approved) for a recommendation on how keep the Gerrit review history and align with Qt branching.

Agree, it needs a bit of review and of course it’s subjective on how strict one wants to be on the criteria. I don’t see it as a problem that it’s still using C++98, although at least the API should be reviewed to be making use of C++11 before it moves out of TP. I’ll try to have a look at the module over the weekend, and will see if I can find someone else to help.

Cheers,
Lars

[Oswald Buddenhagen]

On Fri, Jan 13, 2017 at 01:58:45AM +0000, Stottlemyer, Brett (B.S.) wrote:
> On 12 January 2017 at 08:39, Lars Knoll wrote:
> >* The module solves a problem our users have
> > - It either implements new and so far non existent functionality
> > - Or it solves an existing problem in a new and better way (and is intended to replace the old functionality over time)
>
> It seems like the two potentially competing current projects are Qt Service Framework and QtDBus. I believe QtRO is
> different from both, with “better” being subjective. At the risk of being flamed for why I prefer QtRO, I’d put it
> this way:
>
> The Qt Service Framework deals with OS Services/daemons, and only works inter-process, not inter-device. If I can
> distinguish between a service and a Service, Service being a different class than a program (I.e., on windows, services
> run when no user is logged on). If you need a Service, the Service Framework makes sense. This isn’t the problem
> QtRO is trying to solve.
>
> QtDBus is more similar in intent to QtRO. QtRO is peer-to-peer, while Dbus by default goes through a daemon. Bus is
> common on Linux platforms, but has additional dependencies on other platforms (Windows, QNX, etc). The key difference
> though, is that DBus is intended to translate between languages (java, C, python). This is a disadvantage of QtRO
> in many cases, but when Qt is available for both client/server, it provides a much richer level of communication. For
> instance, QtRO supports QAIM, including selection, across processes/devices. It supports state (current PROPERTY
> values). It supports any type Qt can (or can be made to) pass in Queued connections. Without additional code.
>
> My view is that QtRO complements the other two options, providing strong benefits for the right use-cases.
>
for my taste, there are way too many inconclusive/irrelevant details in
this description. a more layer-oriented approach would be more
convincing imo:

qtdbus is a pure rpc layer.

qtro is a distributed object layer which builds on top of rpc. signals
and slots are exposed rpc. no idea whether you implemented property
propagation on top of generic rpc or via a dedicated protocol part, but
that's just an implementation detail.
it would be perfectly feasible to implement qtro over qtdbus (networked
dbus has been on the table for a decade; dunno if somebody ever pulled
it off).

qtserviceframework is a registration/management layer on top of (local)
rpc.
its intended use appears to be more the creation of platforms rather
than applications, which might explain why it's now bitrotting. ;)
it appears to me that it would be possible to build it on top of a
higher-level representation like qtro, but principally it's indeed rather
orthogonal.


from a purely design perspective, it would seem optimal to have a QtRpc
system of which QtDbus provides a concrete implementation, while QtRo
would be the state extension building on top of it. this would imply
multiple inheritance for an actual instance of qtro.
the principal advantage of this would be a shared idl and surrouding
tooling, which would reduce redundancy on the implementation side.

otoh, we have the existing implementations, and in particular qtdbus
really isn't at a point where it could be just redesigned. we can't drop
the existing qtdbus-xml, and it would be sado-maso to make .rep an
extension of that format.

a more realistic appoach would be having qtdbus as an "invisible"
backend implementation of the generalized qtrpc. note that
qtserviceframework has some of this, which is probably why alex
considered it relevant in the first place.
so a concrete action point would be investigating in how for it's
possible (and actually sensible) to factor out the rpc layer (including
the underlying transport layer, obviously).


it's not really clear to me why you're apparently discarding the idea of
integrating qtro with other object distribution frameworks at the
outset. the systems really aren't fundamentally different, and providing
such integrations may significantly improve qtro adoption where a system
is already established (or is being prescribed).

[Stottlemyer, Brett (B.S.)]

On 1/13/17, 10:09 AM, "Development on behalf of Oswald Buddenhagen" <development-bounces+bstottle=***@qt-project.org on behalf of ***@qt.io> wrote:



>for my taste, there are way too many inconclusive/irrelevant details in
>this description. a more layer-oriented approach would be more
>convincing imo:
>
>qtdbus is a pure rpc layer.
>
>qtro is a distributed object layer which builds on top of rpc. signals
>and slots are exposed rpc. no idea whether you implemented property
>propagation on top of generic rpc or via a dedicated protocol part, but
>that's just an implementation detail.
>it would be perfectly feasible to implement qtro over qtdbus (networked
>dbus has been on the table for a decade; dunno if somebody ever pulled
>it off).
>
>qtserviceframework is a registration/management layer on top of (local)
>rpc.
>its intended use appears to be more the creation of platforms rather
>than applications, which might explain why it's now bitrotting. ;)
>it appears to me that it would be possible to build it on top of a
>higher-level representation like qtro, but principally it's indeed rather
>orthogonal.

My interpretation was that Lars was looking for differences from a Qt user
perspective, and I think my reply was accurate from that perspective. I
would say you are looking at it from the s/w implementation perspective,
which is absolutely valid. And I think your description is well-stated
and accurate as well.

>from a purely design perspective, it would seem optimal to have a QtRpc
>system of which QtDbus provides a concrete implementation, while QtRo
>would be the state extension building on top of it.

[...]

>it's not really clear to me why you're apparently discarding the idea of
>integrating qtro with other object distribution frameworks at the
>outset. the systems really aren't fundamentally different, and providing
>such integrations may significantly improve qtro adoption where a system
>is already established (or is being prescribed).

If you really did mean other object distributions systems, which ones are
you thinking of? Microsoft’s Component Object Model is interesting, but
I’m really not sure how to integrate with it. Did you mean “integrating
QtRO with other RPC mechanisms”, in terms of layering QtRO on top of a
more abstracted RPC? If so, I have a very specific reason: type support.
Many of the types Qt natively support do not work automatically with DBus
due to its intended interplay with other languages. Many useful types
are defined throughout the various Qt modules, and QtRO supports them
all because Qt supports them all. It also will auto generate streaming
operators for needed structures (PODs, in QtRO). QtRO has extensions to
support QAIM types, passing batches of calls with QVariant responses
between processes.

In order to support an agnostic wire-format that works with different
languages, I simply believe support needs to be built into Qt itself,
not a specific module. So QtRO is specific to Qt, while the qtdbus
module only provides a subset of types automatically, but it does provide
such a wire format (and allows you to extend the set of types with
additional code). This has been discussed in the past. I’m not against
the idea, I just don’t think it should be plumbed into QtRO (which it
seems like you agree with).

BTW, Ossi, would which of the build systems fixes you mentioned would

help with the qmake loop on windows CI is seeing? ;)

Regards,
Brett

[Oswald Buddenhagen]

On Sat, Jan 14, 2017 at 05:07:30PM +0000, Stottlemyer, Brett (B.S.) wrote:
> My interpretation was that Lars was looking for differences from a Qt user
> perspective, and I think my reply was accurate from that perspective. I
> would say you are looking at it from the s/w implementation perspective,
> which is absolutely valid. And I think your description is well-stated
> and accurate as well.
>
hmm, not really what i tried to get across. i tried to point out
fundamental, architectural differences, and how the concepts relate to
each other, leaving out all the minor things that represent the current
state of the implementations.

> >from a purely design perspective, it would seem optimal to have a QtRpc
> >system of which QtDbus provides a concrete implementation, while QtRo
> >would be the state extension building on top of it.
>
> [...]
>
> >it's not really clear to me why you're apparently discarding the idea of
> >integrating qtro with other object distribution frameworks at the
> >outset. the systems really aren't fundamentally different, and providing
> >such integrations may significantly improve qtro adoption where a system
> >is already established (or is being prescribed).
>
> If you really did mean other object distributions systems, which ones are
> you thinking of? Microsoft’s Component Object Model is interesting, but
> I’m really not sure how to integrate with it.
>
i was thinking of corba, the example i already mentioned. but i don't
know what is currently relevant in the industries we're targeting.
the task is essentially designing a meta layer (essentially, an idl,
possibly simply adopting an existing one) which can work with different
backends. the essential qt way.
of course, it may be that this task is too complex to get right, in
which case qt bindings for specific systems are a more plausible
approach. i think this is actually what we already do with activeqt.
but then the question forces itself why you had to create *yet another*
distributed object system instead of wrapping an existing one. yes, easy
type handling. see below.

> Did you mean “integrating QtRO with other RPC mechanisms”, in terms of
> layering QtRO on top of a more abstracted RPC?
>
yes, that's the first quoted paragraph above. note that it is a valid
concern independently from the second one.

> If so, I have a very specific reason: type support.
> Many of the types Qt natively support do not work automatically with DBus
> due to its intended interplay with other languages. Many useful types
> are defined throughout the various Qt modules, and QtRO supports them
> all because Qt supports them all. It also will auto generate streaming
> operators for needed structures (PODs, in QtRO). QtRO has extensions to
> support QAIM types, passing batches of calls with QVariant responses
> between processes.
>
well, in the worst case, the data can be simply sent along via blobs. of
course, that kind of defeats the protocol's native type support, but it
will work just fine if qt is on both ends.

> In order to support an agnostic wire-format that works with different
> languages, I simply believe support needs to be built into Qt itself,
> not a specific module.
>
yes

> So QtRO is specific to Qt, while the qtdbus module only provides a
> subset of types automatically, but it does provide such a wire format
> (and allows you to extend the set of types with additional code).
>
in a way, you are using dcop as the transport layer: that was kde's
qdatastream-based predecessor of dbus.

> This has been discussed in the past.
>
indeed, google protocol buffers as a generic data serialization format
have been previously proposed here, but i don't remember the context.

> I’m not against the idea, I just don’t think it should be plumbed into
> QtRO (which it seems like you agree with).
>
yeah. it would require describing all data types in an idl, and
providing primitives for each supported backend protocol (starting with
qdatastream ... in each existing version).

> BTW, Ossi, would which of the build systems fixes you mentioned would
> help with the qmake loop on windows CI is seeing? ;)
>
i don't remember. i'd have to squeeze a full review of your build system
into my todo list.

[Stottlemyer, Brett (B.S.)]

On 1/16/17, 10:14 AM, "Development on behalf of Oswald Buddenhagen" <development-bounces+bstottle=***@qt-project.org on behalf of ***@qt.io> wrote:


>of course, it may be that this task is too complex to get right, in
>which case qt bindings for specific systems are a more plausible
>approach. i think this is actually what we already do with activeqt.
>but then the question forces itself why you had to create *yet another*
>distributed object system instead of wrapping an existing one. yes, easy
>type handling. see below.

The crux of the matter, in my opinion, boils down to a couple of things.

1) Creating a wire-format that works with other languages is a much bigger
undertaking, and one that would need to be integrated into Qt itself, not
QtRO. Based on the (lack of) response from others, it seems this is not
a palatable change at this time.

2) A Qt-to-Qt distributed object system can provide a rich set of features
leveraging Qt’s existing capabilities, without requiring a lot of additional
code by the user. Such a system doesn’t currently exist in Qt, would
provide value, and thus the request to move QtRO out of the playground.
*yet another* isn’t relevant unless you want to delve back into the wire-
format side of this discussion (#1).

For providing Qt new functionality, I feel the existing QtRO design is sound.
If at some point #1 becomes a reality, I would be glad to revisit a generic
idl.

Regards,
Brett

[Tuukka Turunen]

> -----Original Message-----
> From: Development [mailto:development-
> bounces+tuukka.turunen=***@qt-project.org] On Behalf Of Stottlemyer,
> Brett (B.S.)
> Sent: keskiviikkona 18. tammikuuta 2017 4.52
> To: Oswald Buddenhagen <***@qt.io>;
> ***@qt-project.org
> Subject: Re: [Development] Switch Qt Remote Objects to a Tech Preview for
> Qt 5.9
>

...

>
> For providing Qt new functionality, I feel the existing QtRO design is sound.
> If at some point #1 becomes a reality, I would be glad to revisit a generic idl.
>

When QtRO becomes part of Qt, would you continue as the maintainer of the module and have adequate time to polish it so that it can be fully supported in the upcoming Qt releases?

Yours,

Tuukka


> Regards,
> Brett
> _______________________________________________
> Development mailing list
> ***@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development

[Stottlemyer, Brett (B.S.)]

On 1/18/17, 1:52 AM, "Tuukka Turunen" <***@qt.io> wrote:


>>
>
>When QtRO becomes part of Qt, would you continue as the maintainer of the module and have adequate time to polish it so that it can be fully supported in the upcoming Qt releases?
>
>Yours,
>
> Tuukka

Sure, that is the plan. That does however tie into a question I had. Would my Maintainer status for the playground project automatically carry forward? Or will I lose my approver rights once it moves into Qt? I understand Approver rights are not granted lightly, while it is also very difficult to maintain a module without being able to +2 reviews.

What has happened in the past?

Regards,
Brett

[Tuukka Turunen]

> -----Original Message-----
> From: Development [mailto:development-
> bounces+tuukka.turunen=***@qt-project.org] On Behalf Of Stottlemyer,
> Brett (B.S.)
> Sent: keskiviikkona 18. tammikuuta 2017 15.50
> To: ***@qt-project.org
> Subject: Re: [Development] Switch Qt Remote Objects to a Tech Preview for
> Qt 5.9
>
> On 1/18/17, 1:52 AM, "Tuukka Turunen" <***@qt.io> wrote:
>
>
> >>
> >
> >When QtRO becomes part of Qt, would you continue as the maintainer of
> the module and have adequate time to polish it so that it can be fully
> supported in the upcoming Qt releases?
> >
> >Yours,
> >
> > Tuukka
>
> Sure, that is the plan. That does however tie into a question I had. Would
> my Maintainer status for the playground project automatically carry forward?
> Or will I lose my approver rights once it moves into Qt? I understand
> Approver rights are not granted lightly, while it is also very difficult to
> maintain a module without being able to +2 reviews.
>
> What has happened in the past?
>

Hi,

Unless someone objects and proposes a better maintainer (which is quite unlike as you are the main author), you would continue as the maintainer of the module.

Yours,

Tuukka

> Regards,
> Brett
> _______________________________________________
> Development mailing list
> ***@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development

[Shawn Rutledge]

> On 16 Jan 2017, at 16:14, Oswald Buddenhagen <***@qt.io> wrote:
> On Sat, Jan 14, 2017 at 05:07:30PM +0000, Stottlemyer, Brett (B.S.) wrote:
>>
>> If you really did mean other object distributions systems, which ones are
>> you thinking of? Microsoft’s Component Object Model is interesting, but
>> I’m really not sure how to integrate with it.
>>
> i was thinking of corba, the example i already mentioned. but i don't
> know what is currently relevant in the industries we're targeting.

Maybe DDS, which is also from Object Management Group, which uses IDL in approximately the same way, but is extended to support both multicast and point-to-point communication and has the pub-sub metaphor. Unlike many alternatives, it’s peer-to-peer. We have one customer who uses it extensively, so there was a back-burner idea to integrate some sort of support for it in Qt. https://en.wikipedia.org/wiki/Data_Distribution_Service

Speaking of pub-sub, I’ve been hearing about MQTT lately, but haven’t tried to use it. Seems it’s popular with the IOT crowd for now. Has the same disadvantages of D-Bus (as it’s usually used): only TCP and only via a message broker, so it’s not decentralized enough to be truly resilient.

> well, in the worst case, the data can be simply sent along via blobs. of
> course, that kind of defeats the protocol's native type support, but it
> will work just fine if qt is on both ends.

Exactly… it’s going to end up as some sort of byte stream anyway. The correct usage of D-Bus wire protocol can (and should) be refined over time, but at least you have somewhere to start.

I’m not convinced that D-Bus is everything an RPC system should be, but it happens to be popular. Yes it can run across the network - I did that experiment ages ago. That usage is not “supported” mainly because the author didn’t want to bite off more than he could chew; CORBA was already prior art at the time, and it’s clear that distributed object systems are way more complex than IPC on one machine. But I think if you can get away with using the same wire protocol for that too, why not… at least the XML is easy to parse, as Ossi points out. (Although if one was reinventing it today, why use XML… maybe JSON? maybe a binary XML-like self-describing representation. But all of those can be converted anyway.) And you only need it for introspection; you are also free to hard-code your own usage of the wire protocol on both ends without having to write XML first.

What seems to be unique from what you’ve said about qtro so far is automatic serialization without having to write all the stream operators, in cases where they don’t already exist, right? But that’s just a way of getting a byte array, which can indeed be a blob as long as your decoding technique is universal.

So I do like the idea of this being turned into a layer which can run on top of various RPC and object communication mechanisms instead of only one new black box. Ideally it could run on all of them, but that’s many years of work, of course. Later, communicating with non-Qt processes could even be possible. Even if you don’t see it as a goal yet, maybe someone will.

>> This has been discussed in the past.
>>
> indeed, google protocol buffers as a generic data serialization format
> have been previously proposed here, but i don't remember the context.

And from what I read they suck… too much code is generated, which bloats the application and takes too long to compile. Google reinvents the wheel too sometimes, just because they can, and it’s not always elegant, nor does it always pick the tallest giants to stand on their shoulders. The author went on to write CapnCrunch, which I hear good things about. Still didn’t get around to trying it.

[Edward Welbourne]

Hi all,

Back in 12 January 2017, when we were discussing Qt Remotes Object as a
Tech Preview for Qt 5.9, Lars set out [0] some criteria for modules to
enter tech preview. It occurred to me that this was worth turning into
a QUIP, along with criteria for the other transitions of a module's
life-cycle. I've now pushed a first draft of that for review [1].

* [0] https://lists.qt-project.org/pipermail/development/2017-January/028313.html
* [1] https://codereview.qt-project.org/246320

There are some big TODOs that require discussion on this mailing list.
I'll endeavour to formulate whatever consensus I hear in that QUIP.
What are (or should be) the criteria for transitions to the maintained,
obsolete and deprecated states ?

Eddy.

[Tuukka Turunen]

> -----Original Message-----
> From: Development [mailto:development-
> bounces+tuukka.turunen=***@qt-project.org] On Behalf Of Stottlemyer,
> Brett (B.S.)
> Sent: torstaina 12. tammikuuta 2017 3.19
> To: ***@qt-project.org
> Subject: Re: [Development] Switch Qt Remote Objects to a Tech Preview for
> Qt 5.9
>
> On 1/11/17, 11:50 AM, "Development on behalf of Oswald Buddenhagen"
> <development-bounces+bstottle=***@qt-project.org on behalf of
> ***@qt.io> wrote:
>
>
>
> >also, i don't think we've been bothering a lot about deadlines for TP
> >modules, so who cares.
>
> Apparently Tuukka does, since he’s suggesting holding off until 5.10...
>
> >it doesn't help that you apparently haven't even explored the prior art
> >within qt even though it was pointed out to you two years ago already.
>

Yes, when it comes to being part of a "release train" the cargo needs to be in decent shape when the trains leaves the station.

I would like to emphasize that I have nothing against this module being part of Qt 5.9 as a TP - as long as it is in such shape that causes no harm to the Qt 5.9 release schedule.

Many of the delays we have had with previous Qt 5 releases have been partially caused by not being ready at the FF time.

Yours,

Tuukka

[Simon Hausmann]

Hi,

From a CI perspective nothing needs to be done.

From the Gerrit side the UI needs to be changed from submit to merge to staging. Gerrit admins can do that.

The module itself needs to support "make install" and be able to run tests against the installed version of the module.

By default the module will be built against the dev branch of qt5.git. If you use for example a 5.8 branch in qtro, then we'll use qt5 5.8. (there are more options, but this is the gist :)

There are a few modules outside of qt/ that are also CI controlled, it's a rather common thing.

Simon


________________________________
From: Lars Knoll
Sent: Jan 13, 2017 08:25
To: Stottlemyer, Brett (B.S.)
Cc: Qt development mailing list
Subject: Re: [Development] Switch Qt Remote Objects to a Tech Preview for Qt 5.9


Hi Brett,


> On 13 Jan 2017, at 02:58, Stottlemyer, Brett (B.S.) <***@ford.com> wrote:
>
> On 12 January 2017 at 08:39, Lars Knoll wrote:
>
>> From the discussion so far I didn't hear too many things that speak against a TP, the code duplication with moc is one of the issues that fall into the 'flagged and need to be resolved before moving out of TP' category for me. How about the other
>> points in the list above that haven't been discussed yet here?
>
> Moving the points to below instead of above. My opinion is (obviously) biased.
>
>> * The module solves a problem our users have
>> - It either implements new and so far non existent functionality
>> - Or it solves an existing problem in a new and better way (and is intended to replace the old functionality over time)
>
> It seems like the two potentially competing current projects are Qt Service Framework and QtDBus. I believe QtRO is
> different from both, with “better” being subjective. At the risk of being flamed for why I prefer QtRO, I’d put it
> this way:
>
> The Qt Service Framework deals with OS Services/daemons, and only works inter-process, not inter-device. If I can
> distinguish between a service and a Service, Service being a different class than a program (I.e., on windows, services
> run when no user is logged on). If you need a Service, the Service Framework makes sense. This isn’t the problem
> QtRO is trying to solve.
>
> QtDBus is more similar in intent to QtRO. QtRO is peer-to-peer, while Dbus by default goes through a daemon. Bus is
> common on Linux platforms, but has additional dependencies on other platforms (Windows, QNX, etc). The key difference
> though, is that DBus is intended to translate between languages (java, C, python). This is a disadvantage of QtRO
> in many cases, but when Qt is available for both client/server, it provides a much richer level of communication. For
> instance, QtRO supports QAIM, including selection, across processes/devices. It supports state (current PROPERTY
> values). It supports any type Qt can (or can be made to) pass in Queued connections. Without additional code.
>
> My view is that QtRO complements the other two options, providing strong benefits for the right use-cases.

Agreed.
>
>> * The module builds inside the Qt build system on all platforms
>> - Compilation could be disabled on some platforms, but is not allowed to break any platform we support
>> * The module is CI controlled
>
> Umm, this feels like a chicken and egg problem to me. The Qt Company controls the CI, and it doesn’t run against play-
> ground projects. We have worked with KDAB and implemented CI on 4 platforms, targeting only Qt 5.6.2 currently. Until
> Qt’s CI is applied to it, there isn’t a clear way to evaluate build issues. I’ve manually built against 5.8.0rc1 but
> not on all platforms. I feel like there needs to be a way to enable Qt’s CI against QtRO before deciding whether to
> accept it as a TP project. Can the Qt Company CI team help with this? It is a playground project already hosted on
> qt.io. If not, what options are there?

It should be pretty straightforward to enable CI on the module. Pre-conditions are mainly that it’s structured like the other Qt modules. Frederik & Simon what would it take to enable CI on it?
>
>> * There is a decent set of automated tests covering the main functionality and most of the API of the module
>> - The tests pass on all platforms where the module is supported
>> * APIs have been reviewed, and we are reasonably satisfied with them (we've done larger changes to APIs after TPs before)
>> * Architecture makes sense
>> * It follows the Qt coding style and conventions
>> * Implementation has been checked for sanity
>> - It's ok to have parts that are flagged as needing further work (those have to be fixed before moving from TP to supported state)
>
> For all of the above, it seems like you need someone other than the maintainer to assess whether it meets your requirements.
> I believe it does. In terms of coding style, it meets the standards for 5.6.2. We still need to apply the C++11 standards
> for 5.7+. Also, I’m hoping (if approved) for a recommendation on how keep the Gerrit review history and align with Qt branching.

Agree, it needs a bit of review and of course it’s subjective on how strict one wants to be on the criteria. I don’t see it as a problem that it’s still using C++98, although at least the API should be reviewed to be making use of C++11 before it moves out of TP. I’ll try to have a look at the module over the weekend, and will see if I can find someone else to help.

Cheers,
Lars

_______________________________________________
Development mailing list
***@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Simon Hausmann]

Hi,

I scheduled a test build in the CI against 5.8 (as dev continues to be broken). The results are public at


http://testresults.qt.io/coin/integration/playground/qtremoteobjects/tasks/web_playground_qtremoteobjects_1484293648911

There are few issues:

1) namespaced build doesn't work
2) license check fails
3) tests run qmake infinitely on Windows.
4) winrt / msvc 2013 build broken



Simon
________________________________
From: Simon Hausmann
Sent: Jan 13, 2017 08:38
To: Lars Knoll; Stottlemyer, Brett (B.S.)
Cc: Qt development mailing list
Subject: Re: [Development] Switch Qt Remote Objects to a Tech Preview for Qt 5.9

Hi,

From a CI perspective nothing needs to be done.

From the Gerrit side the UI needs to be changed from submit to merge to staging. Gerrit admins can do that.

The module itself needs to support "make install" and be able to run tests against the installed version of the module.

By default the module will be built against the dev branch of qt5.git. If you use for example a 5.8 branch in qtro, then we'll use qt5 5.8. (there are more options, but this is the gist :)

There are a few modules outside of qt/ that are also CI controlled, it's a rather common thing.

Simon


________________________________
From: Lars Knoll
Sent: Jan 13, 2017 08:25
To: Stottlemyer, Brett (B.S.)
Cc: Qt development mailing list
Subject: Re: [Development] Switch Qt Remote Objects to a Tech Preview for Qt 5.9


Hi Brett,


> On 13 Jan 2017, at 02:58, Stottlemyer, Brett (B.S.) <***@ford.com> wrote:
>
> On 12 January 2017 at 08:39, Lars Knoll wrote:
>
>> From the discussion so far I didn't hear too many things that speak against a TP, the code duplication with moc is one of the issues that fall into the 'flagged and need to be resolved before moving out of TP' category for me. How about the other
>> points in the list above that haven't been discussed yet here?
>
> Moving the points to below instead of above. My opinion is (obviously) biased.
>
>> * The module solves a problem our users have
>> - It either implements new and so far non existent functionality
>> - Or it solves an existing problem in a new and better way (and is intended to replace the old functionality over time)
>
> It seems like the two potentially competing current projects are Qt Service Framework and QtDBus. I believe QtRO is
> different from both, with “better” being subjective. At the risk of being flamed for why I prefer QtRO, I’d put it
> this way:
>
> The Qt Service Framework deals with OS Services/daemons, and only works inter-process, not inter-device. If I can
> distinguish between a service and a Service, Service being a different class than a program (I.e., on windows, services
> run when no user is logged on). If you need a Service, the Service Framework makes sense. This isn’t the problem
> QtRO is trying to solve.
>
> QtDBus is more similar in intent to QtRO. QtRO is peer-to-peer, while Dbus by default goes through a daemon. Bus is
> common on Linux platforms, but has additional dependencies on other platforms (Windows, QNX, etc). The key difference
> though, is that DBus is intended to translate between languages (java, C, python). This is a disadvantage of QtRO
> in many cases, but when Qt is available for both client/server, it provides a much richer level of communication. For
> instance, QtRO supports QAIM, including selection, across processes/devices. It supports state (current PROPERTY
> values). It supports any type Qt can (or can be made to) pass in Queued connections. Without additional code.
>
> My view is that QtRO complements the other two options, providing strong benefits for the right use-cases.

Agreed.
>
>> * The module builds inside the Qt build system on all platforms
>> - Compilation could be disabled on some platforms, but is not allowed to break any platform we support
>> * The module is CI controlled
>
> Umm, this feels like a chicken and egg problem to me. The Qt Company controls the CI, and it doesn’t run against play-
> ground projects. We have worked with KDAB and implemented CI on 4 platforms, targeting only Qt 5.6.2 currently. Until
> Qt’s CI is applied to it, there isn’t a clear way to evaluate build issues. I’ve manually built against 5.8.0rc1 but
> not on all platforms. I feel like there needs to be a way to enable Qt’s CI against QtRO before deciding whether to
> accept it as a TP project. Can the Qt Company CI team help with this? It is a playground project already hosted on
> qt.io. If not, what options are there?

It should be pretty straightforward to enable CI on the module. Pre-conditions are mainly that it’s structured like the other Qt modules. Frederik & Simon what would it take to enable CI on it?
>
>> * There is a decent set of automated tests covering the main functionality and most of the API of the module
>> - The tests pass on all platforms where the module is supported
>> * APIs have been reviewed, and we are reasonably satisfied with them (we've done larger changes to APIs after TPs before)
>> * Architecture makes sense
>> * It follows the Qt coding style and conventions
>> * Implementation has been checked for sanity
>> - It's ok to have parts that are flagged as needing further work (those have to be fixed before moving from TP to supported state)
>
> For all of the above, it seems like you need someone other than the maintainer to assess whether it meets your requirements.
> I believe it does. In terms of coding style, it meets the standards for 5.6.2. We still need to apply the C++11 standards
> for 5.7+. Also, I’m hoping (if approved) for a recommendation on how keep the Gerrit review history and align with Qt branching.

Agree, it needs a bit of review and of course it’s subjective on how strict one wants to be on the criteria. I don’t see it as a problem that it’s still using C++98, although at least the API should be reviewed to be making use of C++11 before it moves out of TP. I’ll try to have a look at the module over the weekend, and will see if I can find someone else to help.

Cheers,
Lars

_______________________________________________
Development mailing list
***@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Stottlemyer, Brett (B.S.)]

On 1/13/17, 3:22 AM, "Simon Hausmann" <***@qt.io> wrote:


>I scheduled a test build in the CI against 5.8 (as dev continues to be broken).

What are the implications for feature freeze if dev is still broken?

>There are few issues:
>
>1) namespaced build doesn't work

Fixed

>2) license check fails

Fixed

>3) tests run qmake infinitely on Windows.

Unable to reproduce or get details

>4) winrt / msvc 2013 build broken

Fixed

>Simon

Thanks!

As I understand it, there are several remaining steps to convert to a Qt Tech Preview, most of which I need help with.
1) Move /playground/qtremoteobjects to /qt/qtremoteobjects (need The Qt Company help here)
1a) Be ideal to create to rename the master branch to dev at this point (need The Qt Company help here)
2) Enable CI against the repository (need The Qt Company help here)
3) Submit a change to the qt5 repo to add QtRO (depends on #1 and #2)

I’ve started #3 here: https://codereview.qt-project.org/#/c/183053, but until the repo is moved, I’m unable to add it as a submodule.

Regards,
Brett

[Simon Hausmann]

Hi,

I have scheduled a new build of the current master branch against qt's dev branch:

http://testresults.qt.io/coin/integration/playground/qtremoteobjects/tasks/web_playground_qtremoteobjects_1485097792522


5 out of 21 build configurations are failing.


Simon


________________________________
From: "Stottlemyer, Brett (B.S.)" <***@ford.com>
Sent: Jan 22, 2017 15:13
To: Simon Hausmann; Lars Knoll; Alistair Adams; Buddenhagen Oswald
Cc: Qt development mailing list
Subject: Re: [Development] Switch Qt Remote Objects to a Tech Preview for Qt 5.9


On 1/13/17, 3:22 AM, "Simon Hausmann" <***@qt.io> wrote:


>I scheduled a test build in the CI against 5.8 (as dev continues to be broken).

What are the implications for feature freeze if dev is still broken?

>There are few issues:
>
>1) namespaced build doesn't work

Fixed

>2) license check fails

Fixed

>3) tests run qmake infinitely on Windows.

Unable to reproduce or get details

>4) winrt / msvc 2013 build broken

Fixed

>Simon

Thanks!

As I understand it, there are several remaining steps to convert to a Qt Tech Preview, most of which I need help with.
1) Move /playground/qtremoteobjects to /qt/qtremoteobjects (need The Qt Company help here)
1a) Be ideal to create to rename the master branch to dev at this point (need The Qt Company help here)
2) Enable CI against the repository (need The Qt Company help here)
3) Submit a change to the qt5 repo to add QtRO (depends on #1 and #2)

I’ve started #3 here: https://codereview.qt-project.org/#/c/183053, but until the repo is moved, I’m unable to add it as a submodule.

Regards,
Brett

[Stottlemyer, Brett (B.S.)]

On 1/22/17, 10:51 AM, "Simon Hausmann" <***@qt.io> wrote:



>Hi,
>
>I have scheduled a new build of the current master branch against qt's dev branch:
>
>http://testresults.qt.io/coin/integration/playground/qtremoteobjects/tasks/web_playground_qtremoteobjects_1485097792522
>
>
>5 out of 21 build configurations are failing.

Thanks again, Simon.

All 5 should be addressed by https://codereview.qt-project.org/183062 and https://codereview.qt-project.org/#/c/183059 once merged.

Regards,
Brett