[Development] Behavior change for QMake install targets since 5.9

[Roland Winklmeier]

Dear Qt developers,

since 5.9 some of my projects install targets are failing. Investigation
has shown that the generated Makefiles reference now a different INSTALL
executable. Before it was 'install' from GNU coreutils and now it is
calling 'qmake -install qinstall'.

Apart from the fact that I could not find anything in the changelog about
this bigger change (only a few references that make install will no longer
modify the timestamp etc), there are several behavior changes affecting
existing install targets.

* Installing a symbol link was deep copying the link target under the link
name. Example:
libfoo.so -> libfoo1.0.0.so
Installing libfoo.so installed the binary libfoo.so without creating a
symbol link.

* Using wildcards to copy several targets is broken. Example:
libfoo.so -> libfoo1.0.0.so
Installing libfoo*.so fails now. Before it was copying the link and the
binary.

Maybe the above use cases are not the perfect solution for a specific
problem, but they were working. Also there might be other features (or even
bugs), users of coreutils install were relying on and are no longer
available.
Is this considered acceptable since the usage of coreutils install was a
private configuration and developers should never rely on its behavior?

Cheers R.

[Simon Hausmann]

Hi,


Yes, it's true that "make install" used to call the GNU install for some files on Unixy platforms, sometimes cp and always copy on Windows. We've changed that in 5.9 to a new built-in command in qmake that should give consistent behavior on all platforms, including the ability to preserve the time stamps. (as you can see, with a mixture of cp, copy and install that was not a given).


You should not see any difference when using "make install", but if you do, please file bug reports.


Since Qt itself has installs libraries alongside with .so symlinks and those are installed as symlinks, I wonder what the difference is between Qt and your application. Can you compare the command lines generated in the Makefiles?


It's true that the internal install command does not support wild-cards. We decided that it is the responsibility of the caller to resolve those (qmake).


Or would you like to use this built-in internal command of qmake yourself by calling it directly?


Simon

________________________________
From: Development <development-bounces+simon.hausmann=***@qt-project.org> on behalf of Roland Winklmeier <***@gmail.com>
Sent: Friday, July 21, 2017 12:15:54 PM
To: ***@qt-project.org
Subject: [Development] Behavior change for QMake install targets since 5.9

Dear Qt developers,

since 5.9 some of my projects install targets are failing. Investigation has shown that the generated Makefiles reference now a different INSTALL executable. Before it was 'install' from GNU coreutils and now it is calling 'qmake -install qinstall'.

Apart from the fact that I could not find anything in the changelog about this bigger change (only a few references that make install will no longer modify the timestamp etc), there are several behavior changes affecting existing install targets.

* Installing a symbol link was deep copying the link target under the link name. Example:
libfoo.so -> libfoo1.0.0.so<http://libfoo1.0.0.so>
Installing libfoo.so installed the binary libfoo.so without creating a symbol link.

* Using wildcards to copy several targets is broken. Example:
libfoo.so -> libfoo1.0.0.so<http://libfoo1.0.0.so>
Installing libfoo*.so fails now. Before it was copying the link and the binary.

Maybe the above use cases are not the perfect solution for a specific problem, but they were working. Also there might be other features (or even bugs), users of coreutils install were relying on and are no longer available.
Is this considered acceptable since the usage of coreutils install was a private configuration and developers should never rely on its behavior?

Cheers R.

[Oswald Buddenhagen]

On Fri, Jul 21, 2017 at 12:17:41PM +0000, Simon Hausmann wrote:
> It's true that the internal install command does not support wild-cards.
> We decided that it is the responsibility of the caller to resolve those
> (qmake).
>
we did? :}
as much as i despise wildcards in source file lists, this behavior is
actually documented for INSTALLS.
compare https://bugreports.qt.io/browse/QTCREATORBUG-17935
(on the upside, qmake is now consistent with qtc :D).

[Simon Hausmann]

I was under the impression that


http://code.qt.io/cgit/qt/qtbase.git/tree/qmake/generators/makefile.cpp#n1304


was the code path hit if the entry in foo.files is not a "//real file or target" (as per the comment a little above).


Simon

________________________________
From: Development <development-bounces+simon.hausmann=***@qt-project.org> on behalf of Oswald Buddenhagen <***@qt.io>
Sent: Friday, July 21, 2017 2:49:03 PM
To: ***@qt-project.org
Subject: Re: [Development] Behavior change for QMake install targets since 5.9

On Fri, Jul 21, 2017 at 12:17:41PM +0000, Simon Hausmann wrote:
> It's true that the internal install command does not support wild-cards.
> We decided that it is the responsibility of the caller to resolve those
> (qmake).
>
we did? :}
as much as i despise wildcards in source file lists, this behavior is
actually documented for INSTALLS.
compare https://bugreports.qt.io/browse/QTCREATORBUG-17935
(on the upside, qmake is now consistent with qtc :D).

[Oswald Buddenhagen]

On Fri, Jul 21, 2017 at 02:55:26PM +0200, Simon Hausmann wrote:
> I was under the impression that
>
>
> [1]http://code.qt.io/cgit/qt/qtbase.git/tree/qmake/generators/makefile.cpp#n1304
>
> was the code path hit if the entry in foo.files is not a "//real file or
> target" (as per the comment a little above).
>
that won't work for make-time generated files, which is precisely the
case for the .so.* files.

of course, the root cause here is that files with known file names are
referred to by wildcard, but we still can't break that "just so".

[Roland Winklmeier]

Hi Simon,

Preamble:
The problem I was trying to solve with wildcards was to install a shared
library target into two different directories. The first directory is
installed with a simple 'INSTALLS += target' line. The other path is
installed manually by a second INSTALLS definition and the files are
selected with a wild card. This was the easiest solution at the time and
the documentation said it was fine to do so.

> Since Qt itself has installs libraries alongside with .so symlinks and
> those are installed as symlinks, I wonder what the difference is between Qt
> and your application. Can you compare the command lines generated in the
> Makefiles?
>
I've checked the Makefiles. The only difference is that -$(INSTALL_FILE) is
replaced by -$(QINSTALL). The problem here is that make install used to be
able to handle wildcards during install time. QMake isn't. The difference
between Qt and my application is explained in the preamble (install to
second directory).

> It's true that the internal install command does not support wild-cards.
> We decided that it is the responsibility of the caller to resolve those
> (qmake).
>
The problem I have is that the build products are not yet available when
qmake is running. Qmake *is* able to resolve wild cards if the files are
available when qmake is running. But the products aren't existing yet. With
Qt 5.8 the install command line was something like

-$(INSTALL) /path/to/libfoo* $(INSTALL_ROOT)/path/to/install/libfoo*

which was perfectly ok on most OS's. I think only MacOS didn't like it.

But now its

-$(QINSTALL) /path/to/libfoo* $(INSTALL_ROOT)/path/to/install/libfoo*

and I get the error "Error: usage: [-exe] source target.

I can completely understand that the installation behavior needed to be
aligned across all operating systems. If we could revise the decision to no
allow wild cards, I would be very happy about it.

Cheers

[Thiago Macieira]

On Friday, 21 July 2017 05:17:41 PDT Simon Hausmann wrote:
> Since Qt itself has installs libraries alongside with .so symlinks and those
> are installed as symlinks, I wonder what the difference is between Qt and
> your application. Can you compare the command lines generated in the
> Makefiles?

It doesn't use $(INSTALL) for those:

install_target: first FORCE
@test -d $(INSTALL_ROOT)/home/tjmaciei/obj/qt/installed/lib64 || mkdir
-p $(INSTALL_ROOT)/home/tjmaciei/obj/qt/installed/lib64
-$(QINSTALL_PROGRAM) ../../lib/$(TARGET) $(INSTALL_ROOT)/home/
tjmaciei/obj/qt/installed/lib64/$(TARGET)
-$(STRIP) --strip-unneeded $(INSTALL_ROOT)/home/tjmaciei/obj/qt/
installed/lib64/$(TARGET)
-$(SYMLINK) $(TARGET) $(INSTALL_ROOT)/home/tjmaciei/obj/qt/installed/
lib64/$(TARGET0)

Noe that this creates a new symlink on the destination. It doesn't try to copy
the existing symlink over to the destination.


Top of the Makefile has:
INSTALL_FILE = install -m 644 -p
INSTALL_PROGRAM = install -m 755 -p
INSTALL_DIR = cp -f -R
QINSTALL = /home/tjmaciei/obj/qt/qt5-release/qtbase/bin/qmake -install
qinstall
QINSTALL_PROGRAM = /home/tjmaciei/obj/qt/qt5-release/qtbase/bin/qmake -install
qinstall -exe
DEL_FILE = rm -f
SYMLINK = ln -f -s

So any rules that have commands using $(INSTALL_FILE) will continue to work.
If the rule is now using QINSTALL, that means its commands were generated by
qmake.

--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center

[Kevin Kofler]

Roland Winklmeier wrote:
> since 5.9 some of my projects install targets are failing. Investigation
> has shown that the generated Makefiles reference now a different INSTALL
> executable. Before it was 'install' from GNU coreutils and now it is
> calling 'qmake -install qinstall'.

Would this:
unix {
QINSTALL=$(INSTALL_FILE)
QINSTALL_PROGRAM=$(INSTALL_PROGRAM)
}
(in your .pro file) work around it?

Kevin Kofler

[Thiago Macieira]

On Friday, 21 July 2017 20:40:44 PDT Kevin Kofler wrote:
> Roland Winklmeier wrote:
> > since 5.9 some of my projects install targets are failing. Investigation
> > has shown that the generated Makefiles reference now a different INSTALL
> > executable. Before it was 'install' from GNU coreutils and now it is
> > calling 'qmake -install qinstall'.
>
> Would this:
> unix {
> QINSTALL=$(INSTALL_FILE)
> QINSTALL_PROGRAM=$(INSTALL_PROGRAM)
> }
> (in your .pro file) work around it?

No, those are Makefile variables, not qmake ones.

qmake writes them unconditionally:
t << "QINSTALL = " << var("QMAKE_QMAKE") << " -install qinstall" <<
endl;
t << "QINSTALL_PROGRAM = " << var("QMAKE_QMAKE") << " -install qinstall -
exe" << endl;

The only way to override them is to trick qmake into adding something to the
Makefile that it doesn't expect to. Needless to say, brittle and unsupported.

--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center