[Development] Summary ABI compabilty

[Marco Bubke]

Hi


I try to gather all arguments so they don't get lost:


Short Term ABI compability(1-2 years):


* better bug fixing

* faster development

* faster adoption of standards

- con: we don't want to adapt them because our solution is better


Long Term ABI compability(5-10 years):


* easier life of Linux packager

- con: you can use Flatpack in many case (disputed)


I think I miss some but feel free to extend the list

[Thiago Macieira]

Post by Marco Bubke
* better bug fixing

By a very small margin. We've got 20 years of experience fixing bugs without
breaking ABI.

Post by Marco Bubke
* faster development

Unproven.

Post by Marco Bubke
* faster adoption of standards
- con: we don't want to adapt them because our solution is better

Unrelated. The standard libraries *do* keep their ABI, mostly. We can choose
to use libstdc++ and libc++ in our ABI and still keep our ABI, though with
some restrictions. We'd be subject to breakages in those libraries if they did
that, accidentally or no, but in those cases the problems would affect almost
all C++ programs anyway.

Post by Marco Bubke
* easier life of Linux packager
- con: you can use Flatpack in many case (disputed)

[Lisandro Damián Nicanor Pérez Meyer]

On martes, 22 de noviembre de 2016 11:33:55 ART Marco Bubke wrote:
[snip]

Post by Marco Bubke
* easier life of Linux packager
- con: you can use Flatpack in many case (disputed)

The amount of flatpacks needed would easily become also a problem (see my mail
with the qtbase.txt link in it a few minutes ago).

[Giuseppe D'Angelo]

Post by Marco Bubke
I think I miss some but feel free to extend the list

Does Qt need to keep BC across incompatible standard C++ library
implementations? (Independently from the long term/short term ABI
breaks)

This does not mean "is Qt affected by a BC break in the standard C++
library that you used to compile Qt", the answer to that is yes (Qt
uses the standard library internally).

* pros:
** You can use Qt compiled against either libc++ or libstdc++, with
"the other one", without recompiling Qt.

* cons:
** End users applications using anything coming from the standard
library still need to be recompiled; what's the problem at recompiling
Qt too? (If the end user application is NOT using the standard library
then this whole discussion is void)
** We can't expose std:: datatypes in our ABI, only in inline
functions, losing significant functionality (and possibly performance)

* subjective cons:
** apart from Qt, noone does that, why should Qt be special in that regard?
** is there anyone really using this feature?

Cheers,

[Thiago Macieira]

Post by Giuseppe D'Angelo
** You can use Qt compiled against either libc++ or libstdc++, with
"the other one", without recompiling Qt.
** apart from Qt, noone does that, why should Qt be special in that regard?
** is there anyone really using this feature?

And that's why we couldn't make the decision: we don't know.

macOS seems to have settled on libc++, but now many Linux distributions are
providing it too. The question is whether long-term people will want this on
Linux or not (Android included).

Right now, since none of the distributions shipping libc++ are doing it right,
you cannot use "the other one" without recompiling Qt anyway. If this way of
building will remain, then we don't have a problem.

[Allan Sandfeld Jensen]

Post by Thiago Macieira

Post by Giuseppe D'Angelo
** You can use Qt compiled against either libc++ or libstdc++, with
"the other one", without recompiling Qt.
** apart from Qt, noone does that, why should Qt be special in that
regard? ** is there anyone really using this feature?

And that's why we couldn't make the decision: we don't know.
macOS seems to have settled on libc++, but now many Linux distributions are
providing it too. The question is whether long-term people will want this
on Linux or not (Android included).
Right now, since none of the distributions shipping libc++ are doing it
right, you cannot use "the other one" without recompiling Qt anyway. If
this way of building will remain, then we don't have a problem.

Can't we statically link to it, so that it doesn't matter which version the
application is using?


`Allan

[Thiago Macieira]

On quarta-feira, 23 de novembro de 2016 00:07:35 PST Allan Sandfeld Jensen

Post by Allan Sandfeld Jensen
Can't we statically link to it, so that it doesn't matter which version the
application is using?

No, same problem, which is exactly what the Linux distros are doing wrong.

C++ requires certain symbols to be shared and obey ODR: the typeinfo for the
base types and the virtual tables for some classes (notably, std::exception).
If we statically link one of the libraries that contains those, then
applications may or may not link to them at runtime. Breaking ODR for those is
problematic.

The solution is simple: we need to dynamically link to a library that provides
those and just those. Both libc++ and libstdc++ have such a library:
respectively, libc++abi and libsupc++. Linux distros need to make sure that
only one of those exists and that both libc++ and libstdc++ link to it.

But even if we do that, it doesn't solve the problem of using the std types in
our ABI, like std::function which we desperately want to use. If our objective
is to do that, then you can't replace.

[Lars Knoll]

I think we are mixing two problems that should be handled separately in this whole thread.

The first one is the question how much of libstdc++ we want to use and expose in our APIs. The second question is about our BC promise between minor and patch level releases. Let's keep these two questions separated and don't mix those in the discussions.

There are a lot of good arguments towards using the STL and libstdc++ more, as it will allow us to interoperate better with the C++ standard, and provides a couple of things that we really want to use. So I can see many good arguments towards going down that route. Doing so will bind the compiled Qt binary to a certain version of that library (ie, it will require a recompile of Qt to switch from libc++ to libstdc++). To a large extent that is no different than the situation we're facing with e.g. different VC++ runtimes. It also doesn't create impossible challenges for the Linux packagers/distributors (or at least the challenges won't be caused by us). So I'm positive towards using more of the standard functionality and APIs (and also exposing them in our APIs).

Breaking BC in Qt minor or patch level releases is a totally different question. So far I have not convincing arguments as to why we would gain a lot here. We've managed for years to fix our bugs and introduce new features without breaking BC, I don't see why this should be different now. We will have a well defined point where we can break BC with Qt 6 and I think it's beneficial to have these breakages only at well defined points. Let's also not forget that breaking BC in minor releases would come at a severe cost for the Linux ecosystem, an ecosystem we want to support as good as we can.

Cheers,
Lars

I think this is something we probably just need to do, as we see more

As we all know there are quite a few things we really would like to use

On 23/11/16 01:05, "Development on behalf of Thiago Macieira" <development-bounces+lars.knoll=***@qt-project.org on behalf of ***@intel.com> wrote:

On quarta-feira, 23 de novembro de 2016 00:07:35 PST Allan Sandfeld Jensen

Post by Allan Sandfeld Jensen
Can't we statically link to it, so that it doesn't matter which version the
application is using?

No, same problem, which is exactly what the Linux distros are doing wrong.

C++ requires certain symbols to be shared and obey ODR: the typeinfo for the
base types and the virtual tables for some classes (notably, std::exception).
If we statically link one of the libraries that contains those, then
applications may or may not link to them at runtime. Breaking ODR for those is
problematic.

The solution is simple: we need to dynamically link to a library that provides
those and just those. Both libc++ and libstdc++ have such a library:
respectively, libc++abi and libsupc++. Linux distros need to make sure that
only one of those exists and that both libc++ and libstdc++ link to it.

But even if we do that, it doesn't solve the problem of using the std types in
our ABI, like std::function which we desperately want to use. If our objective
is to do that, then you can't replace.

--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center

_______________________________________________
Development mailing list
***@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Kevin Kofler]

Post by Lars Knoll
There are a lot of good arguments towards using the STL and libstdc++
more, as it will allow us to interoperate better with the C++ standard,
and provides a couple of things that we really want to use. So I can see
many good arguments towards going down that route. Doing so will bind the
compiled Qt binary to a certain version of that library (ie, it will
require a recompile of Qt to switch from libc++ to libstdc++). To a large
extent that is no different than the situation we're facing with e.g.
different VC++ runtimes. It also doesn't create impossible challenges for
the Linux packagers/distributors (or at least the challenges won't be
caused by us). So I'm positive towards using more of the standard
functionality and APIs (and also exposing them in our APIs).

libstdc++ now has a sub-ABI-version that was already bumped recently for
better C++11 compliance. It can be bumped again, without even changing the
soname, and all libraries using the std:: types in their APIs will have a
broken ABI.

That would leave us distributors only with very unpleasant solutions:
(a) rebuild everything using Qt in reverse dependency order, which has to be
computed first. Alphabetic order, as normally used by the Fedora mass
rebuild scripts, will almost certainly not work, due to symbol lookup
failures.
(b) pin libstc++ to an old version forever.
(c) patch libstdc++ to stick to the old ABI forever.
(d) tweak the build flags of Qt, everything using Qt, all C++ libraries it
uses, everything using those libraries, etc., propagating back and
forth, to stick to the old ABI forever. That is likely all or almost all
C++ stuff in the distribution. So (d) is probably just a very
complicated way to implement (c).
(e) hack things up so that Qt uses the old ABI types for its own APIs
(forever), but the new ABI types for libraries it uses that have already
been rebuilt. That requires patching Qt with libstdc++-specific hacks
and maintaining those patches forever.

Neither of those is really acceptable.

Kevin Kofler

[Konstantin Tokarev]

Post by Kevin Kofler

 There are a lot of good arguments towards using the STL and libstdc++
 more, as it will allow us to interoperate better with the C++ standard,
 and provides a couple of things that we really want to use. So I can see
 many good arguments towards going down that route. Doing so will bind the
 compiled Qt binary to a certain version of that library (ie, it will
 require a recompile of Qt to switch from libc++ to libstdc++). To a large
 extent that is no different than the situation we're facing with e.g.
 different VC++ runtimes. It also doesn't create impossible challenges for
 the Linux packagers/distributors (or at least the challenges won't be
 caused by us). So I'm positive towards using more of the standard
 functionality and APIs (and also exposing them in our APIs).

libstdc++ now has a sub-ABI-version that was already bumped recently for
better C++11 compliance. It can be bumped again, without even changing the
soname, and all libraries using the std:: types in their APIs will have a
broken ABI.
(a) rebuild everything using Qt in reverse dependency order, which has to be
    computed first. Alphabetic order, as normally used by the Fedora mass
    rebuild scripts, will almost certainly not work, due to symbol lookup
    failures.

Gentoo provides revdep-rebuild for ages. Looks like you guys are stuck with crappy tooling.

Post by Kevin Kofler
(b) pin libstc++ to an old version forever.
(c) patch libstdc++ to stick to the old ABI forever.
(d) tweak the build flags of Qt, everything using Qt, all C++ libraries it
    uses, everything using those libraries, etc., propagating back and
    forth, to stick to the old ABI forever. That is likely all or almost all
    C++ stuff in the distribution. So (d) is probably just a very
    complicated way to implement (c).
(e) hack things up so that Qt uses the old ABI types for its own APIs
    (forever), but the new ABI types for libraries it uses that have already
    been rebuilt. That requires patching Qt with libstdc++-specific hacks
    and maintaining those patches forever.
Neither of those is really acceptable.
        Kevin Kofler
_______________________________________________
Development mailing list
http://lists.qt-project.org/mailman/listinfo/development

[Giuseppe D'Angelo]

Post by Kevin Kofler
libstdc++ now has a sub-ABI-version that was already bumped recently for
better C++11 compliance. It can be bumped again, without even changing the
soname, and all libraries using the std:: types in their APIs will have a
broken ABI.

Out of curiosity, why does a ABI break not incur in a soname change?

Cheers,

[Olivier Goffart]

Post by Giuseppe D'Angelo

Post by Kevin Kofler
libstdc++ now has a sub-ABI-version that was already bumped recently for
better C++11 compliance. It can be bumped again, without even changing the
soname, and all libraries using the std:: types in their APIs will have a
broken ABI.

Out of curiosity, why does a ABI break not incur in a soname change?

Because they would use a new abi tag or new inline namespace and so they would
technically not break their ABI. (As the symbols in the old namespace or
without the abi tag would still be present in the binary)

However, if a library (such as Qt) made use of the standard library in their
interface, they would only contain one set of symbols (with the new or the old
tag, but not both). Therefore, the applications have to be build with the same
standard library abi tag than all libraries.

It's a bit unfortunate that this happens like that. But, we should ask:

- How often will it happen in the future? They only change the binary
compatibility if they have a very good reason to do so. Last time it was
required in the GNU's libstdc++ because of a change in C++11 (which specified
things that were left unspecified before). The standard committee is not
easily accepting changes that force implementer to break binary compatibility.
So perhaps not more than once every 6 years. (conservative estimate)

- Why would Qt be different than all the other libraries that uses the
standard library? I mean, when the gcc break happened last year, most C++
applications and libraries had to be rebuilt anyway. Should Qt ABI depends on
the stl, how big is the set of dependencies that would not need to be rebuilt
compared to the amount of package that anyway need to be rebuilt?

So considering these two points, it should not happen often and most packages
would anyway need to be recompiled.

[Thiago Macieira]

Post by Giuseppe D'Angelo
Out of curiosity, why does a ABI break not incur in a soname change?

Because they technically did not break BC. Programs that were compiled and
linked against an older version will continue to run exactly as before.
They've only added new symbols to the library.

It's also not an SC break: programs that used to compile with the old library
will still compile again with the new one. Like with glibc with ELF version,
the newly-linked program will call the new symbols, not the old ones.

This is a domain of breakage that isn't very well explored because it happens
not in the library that made the source code change, but in a second library
that used the first one.

It happens almost exclusively in C++, though if you try hard enough you can do
it in C with some macros too. It happens because C++ has two "features" that
allow it to happen:

1) default parameters (including default template parameters)
2) extensive mangling of symbols

The most likely case of this breakage was that of adding new defaulted
template parameers to template classes. Fortunately, it was one of the cases
caught in the C++ Binary Compatibility guideline on KDE websites.

Inline namespaces actually make it easy to cause this breakage. So we should
not use them in Qt public headers.

GCC ABI tags are like inline namespaces, but looking from another angle.

[Thiago Macieira]

Post by Kevin Kofler
libstdc++ now has a sub-ABI-version that was already bumped recently for
better C++11 compliance. It can be bumped again, without even changing the
soname, and all libraries using the std:: types in their APIs will have a
broken ABI.

[cut]

What option are you choosing for the other packages that do use the libstdc++
API?

[Kevin Kofler]

Post by Thiago Macieira
What option are you choosing for the other packages that do use the
libstdc++ API?

Rebuilding them all, and everything that does not work when building in
random (basically alphabetical, though the order is not guaranteed) order
(with the mass rebuild script) has to be fixed manually.

Kevin Kofler

[Kevin Kofler]

Post by Marco Bubke
* easier life of Linux packager

Where what you call "easier life" is really a question about whether
distributions can continue shipping Qt at all (at least current versions of
Qt, as opposed to sticking to an LTS version forever, even beyond its EOL
and even ignoring future LTS versions) or not.

Kevin Kofler