[Development] HDR Support in Qt and Angle

[Boudewijn Rempt]

We've been working with Intel for some time now on figuring out how to support
HDR in Krita. Dmitry has made some initial progress, hacking on Angle and Qt
to make this possible. Right now, only Windows and Direct3D support HDR,
though Intel is also working on support on Linux.

Other Qt applications, like VLC, implement a custom Direct3D widget, but since
all our code is opengl, that wouldn't have worked for Krita.

Here's the first report with some links to code: https://
dimula73.blogspot.com/2018/11/krita-fall-2018-sprint-results-hdr.html

What I'm wondering is, have other people been working on this as well, or
thinking about it? And what would be needed to get our code upstreamed. I
remember that trying to get the modernized code for QOpenGLPainter pushed was
quite hard, so I wanted to reach out as soon as possible.

[Oliver Wolff]

Hi,

On 21/11/2018 10:43, Boudewijn Rempt wrote:

We've been working with Intel for some time now on figuring out how to support
HDR in Krita. Dmitry has made some initial progress, hacking on Angle and Qt
to make this possible. Right now, only Windows and Direct3D support HDR,
though Intel is also working on support on Linux.

Other Qt applications, like VLC, implement a custom Direct3D widget, but since
all our code is opengl, that wouldn't have worked for Krita.

Here's the first report with some links to code: https://
dimula73.blogspot.com/2018/11/krita-fall-2018-sprint-results-hdr.html

What I'm wondering is, have other people been working on this as well, or
thinking about it? And what would be needed to get our code upstreamed. I
remember that trying to get the modernized code for QOpenGLPainter pushed was
quite hard, so I wanted to reach out as soon as possible.

I am not aware of anyone having any plans about that. How hard it is to get things upstreamed into Qt depends on the general approach. If you want to upstream your changes to Angle and use an updated version in Qt it might take a (long) while. The last Angle update we did was not to anything near current HEAD as Angle drops compilers a lot earlier than Qt and we had to find a version that still supported Qt's minimum (MSVC 2015 and mingw 7.3 I think). They are moving much faster than Qt. As soon as we drop MSVC2015 we might be able to update to a newer Sha of Angle, but by then Angle might have risen their requirements as well.

Another approach would be having the Angle patches inside Qt but I image that having HDR support might be a bigger undertaking and thus this might not be a feasible solution.

Olli

[Boudewijn Rempt]

Post by Oliver Wolff
I am not aware of anyone having any plans about that. How hard it is to get
things upstreamed into Qt depends on the general approach. If you want to
upstream your changes to Angle and use an updated version in Qt it might
take a (long) while. The last Angle update we did was not to anything near
current HEAD as Angle drops compilers a lot earlier than Qt and we had to
find a version that still supported Qt's minimum (MSVC 2015 and mingw 7.3 I
think). They are moving much faster than Qt. As soon as we drop MSVC2015 we
might be able to update to a newer Sha of Angle, but by then Angle might
have risen their requirements as well.
Another approach would be having the Angle patches inside Qt but I image
that having HDR support might be a bigger undertaking and thus this might
not be a feasible solution.

Right now, we're actually patching the angle that comes with Qt, though the
patches should apply just fine to a more recent version of Angle.

We've also managed to make not just the demo applet work, but Krita itself,
which means that the whole stack of angle, opengl, qt-opengl and everything
appears to be sane(-ish) :-)

[Oliver Wolff]

Post by Boudewijn Rempt

Post by Oliver Wolff
I am not aware of anyone having any plans about that. How hard it is to get
things upstreamed into Qt depends on the general approach. If you want to
upstream your changes to Angle and use an updated version in Qt it might
take a (long) while. The last Angle update we did was not to anything near
current HEAD as Angle drops compilers a lot earlier than Qt and we had to
find a version that still supported Qt's minimum (MSVC 2015 and mingw 7.3 I
think). They are moving much faster than Qt. As soon as we drop MSVC2015 we
might be able to update to a newer Sha of Angle, but by then Angle might
have risen their requirements as well.
Another approach would be having the Angle patches inside Qt but I image
that having HDR support might be a bigger undertaking and thus this might
not be a feasible solution.

Right now, we're actually patching the angle that comes with Qt, though the
patches should apply just fine to a more recent version of Angle.

Then getting it into Qt should not be too big of an issue as long as it
compiles on every compiler we support and we can find someone who can
test the implementation. I can check the code itself, but won't be able
to give it a test run because I lack a "proper" device.

Post by Boudewijn Rempt
We've also managed to make not just the demo applet work, but Krita itself,
which means that the whole stack of angle, opengl, qt-opengl and everything
appears to be sane(-ish) :-)

Sounds great :)

Post by Boudewijn Rempt
_______________________________________________
Development mailing list
https://lists.qt-project.org/listinfo/development

[Allan Sandfeld Jensen]

Post by Boudewijn Rempt
We've been working with Intel for some time now on figuring out how to
support HDR in Krita. Dmitry has made some initial progress, hacking on
Angle and Qt to make this possible. Right now, only Windows and Direct3D
support HDR, though Intel is also working on support on Linux.
Other Qt applications, like VLC, implement a custom Direct3D widget, but
since all our code is opengl, that wouldn't have worked for Krita.
Here's the first report with some links to code: https://
dimula73.blogspot.com/2018/11/krita-fall-2018-sprint-results-hdr.html
What I'm wondering is, have other people been working on this as well, or
thinking about it? And what would be needed to get our code upstreamed. I
remember that trying to get the modernized code for QOpenGLPainter pushed
was quite hard, so I wanted to reach out as soon as possible.

I have been working on it and have a plan for it. The first steps is a color
space system, see https://codereview.qt-project.org/234095.

Then the very basic of extended colors https://codereview.qt-project.org/
233735

Next I have some work toward an 4xfp16 based rendering backend and image
support, possibly removing the 4xuint16 backend at the same time so there is
only one high-color precision backend.

Then we just need a way to connect that to a backing store, such as using
scRGB in OpenGL which looks like what you have been wokring on :)

Best Regards
'Allan

[Allan Sandfeld Jensen]

Post by Allan Sandfeld Jensen
I have been working on it and have a plan for it. The first steps is a
color space system, see https://codereview.qt-project.org/234095.
Then the very basic of extended colors https://codereview.qt-project.org/
233735
Next I have some work toward an 4xfp16 based rendering backend and image
support, possibly removing the 4xuint16 backend at the same time so there
is only one high-color precision backend.
Then we just need a way to connect that to a backing store, such as using
scRGB in OpenGL which looks like what you have been wokring on :)

Ah, no, that's not what we're doing. OpenGL simply does not support HDR at
all in any meaningful way. Sending f16 textures looks like it works, on
nvidia cards, but it will not work on an intel gpu. And scRGB is something
https://twitter.com/momaku/ status/1059239827244306432

It depends on what you want. Using Display P3 for instance is just wide-gamut
not HDR. You can get that with just the new color-space support and higher
non-extended color precision. That is the primary motivation for doing that
first as wide-gamut monitors are already on the market and in the hands of
many (high-) end-users. Where HDR is still mostly non-standardized on PC
monitors.

The use of extended RGB, as scRGB is also known, is very useful behind the
scenes in any case, as using that you can map to and from any color-space
without clamping, so we can keep something that is internally like sRGB, while
still supporting wide-gamut by later taking the extended-sRGB and turning it
back into a non-extended wide-gamut image. This makes it possibly to put the
clamping and final color-space conversion in the QPA, while keeping the rest
generic. So I would still support at least linear-scRGB (aka extended-linear-
sRGB), even if only as an intermediate format.

It is arguably a strange non-sensical format with many impossible colors, but
it is also very useful.

Also doesn't BT.2020 also require extended color-values above (1.0, 1,0, 1.0)
in most encodings?

'Allan

[Boudewijn Rempt]

Post by Allan Sandfeld Jensen
It depends on what you want. Using Display P3 for instance is just
wide-gamut not HDR. You can get that with just the new color-space support
and higher non-extended color precision. That is the primary motivation for
doing that first as wide-gamut monitors are already on the market and in
the hands of many (high-) end-users. Where HDR is still mostly
non-standardized on PC monitors.

We're specifically working on support for the VESA DisplayHDR standard:
https://displayhdr.org/ . I'm testing with a ASUS ROG Swift PG27UQ monitor.

Post by Allan Sandfeld Jensen
The use of extended RGB, as scRGB is also known, is very useful behind the
scenes in any case, as using that you can map to and from any color-space
without clamping, so we can keep something that is internally like sRGB,
while still supporting wide-gamut by later taking the extended-sRGB and
turning it back into a non-extended wide-gamut image. This makes it
possibly to put the clamping and final color-space conversion in the QPA,
while keeping the rest generic. So I would still support at least
linear-scRGB (aka extended-linear- sRGB), even if only as an intermediate
format.
It is arguably a strange non-sensical format with many impossible colors,
but it is also very useful.
Also doesn't BT.2020 also require extended color-values above (1.0, 1,0,
1.0) in most encodings?

Well... We're not doing image manipulation using Qt classes. We simply have a
buffer, handle that using opencolorio (which has suddenly seen a huge amount
of development) to generate another buffer that gets stuffed into f16 textures
and displayed. The amazing thing is that we've actually massaged Angle and Qt
into making that possible using QOpenGLWidget.

The result is a set of patches that we want to clean up and upstream :-)

[Dmitry Kazakov]

Hi, Allan!

Post by Allan Sandfeld Jensen
The use of extended RGB, as scRGB is also known, is very useful behind the
scenes in any case, as using that you can map to and from any color-space
without clamping, so we can keep something that is internally like sRGB, while
still supporting wide-gamut by later taking the extended-sRGB and turning it
back into a non-extended wide-gamut image. This makes it possibly to put the
clamping and final color-space conversion in the QPA, while keeping the rest
generic. So I would still support at least linear-scRGB (aka extended-linear-
sRGB), even if only as an intermediate format.

As far as I can tell, the most arguable point in scRGB from color
scientists' point of view is that scRGB allows negative values. If we
just don't use negative values, the color space becomes basically a
standard p709-g10 color space, which is widely accepted by everyone.

So the thing we, as developers, should avoid is trying to access wider
color gamut by utilizing negative part of scRGB. Instead we should just
use p2020 primaries directly (DXGI interface supports that perfectly for
the swap chains and my HDR patch supports that as well).

Post by Allan Sandfeld Jensen
Also doesn't BT.2020 also require extended color-values above (1.0, 1,0, 1.0)
in most encodings?

No, BT.2020-PQ doesn't require values higher than 1.0, because the
entire range of 0...10000 nits is compressed into the range of
0.0...1.0. At least by definition by EGL:

https://www.khronos.org/registry/EGL/extensions/EXT/EGL_EXT_gl_colorspace_bt2020_linear.txt

But, anyway, the problem is not in values higher than 1.0.The problem in
negative numbers :)

[Dmitry Kazakov]

Hi, Allan!

It is nice to ear you are working is a adjoint part of color management
in Qt! I think we should coordinate a couple of things to make sure we
can merge our efforts later.

Basically, for implementing correct frame buffer compositing I need to
know the following points:

1) Color space of the framebuffer (swap chain) of the root window. It is
usually set by QSurfaceFormat::setDefaultFormat(). Right now I
implemented that by extending QSurfaceFormat::ColorSpace enum (see [1]).
Do you think it is okay?

2) When rendering legacy Qt widgets, I assume that all of them just have
sRGB color when converting them into the framebuffer's color space. See
[2]. Is such assumption okay?

3) When composing QOpenGLWidgets into the framebuffer I need to know
their color space as well. Should I also reuse enum from
QSurfaceFormat::ColorSpace or better introduce some other enum to avoid
dependency to QSurfaceFormat? See TODO note in [3]

4) Btw, do you have any idea how ICC calibrated and profiled displays
are expected to behave when an application uses p2020-pq or p709-g10 for
the framebuffer color space?


[1] -
https://github.com/dimula73/qtbase/commit/2a525823f04daaf4ef3d0dffce5d2dc52072ec85#diff-2714c4324bddedfa531c6cfd9b7e3ff3L90
[2] -
https://github.com/dimula73/qtbase/commit/765babfc259f9acfa18671db940087d934d574fc#diff-ee6dcc517ae5e38fa93de3a663621648R434
[3] -
https://github.com/dimula73/qtbase/commit/765babfc259f9acfa18671db940087d934d574fc#diff-ee6dcc517ae5e38fa93de3a663621648R301

Post by Allan Sandfeld Jensen

Post by Boudewijn Rempt
We've been working with Intel for some time now on figuring out how to
support HDR in Krita. Dmitry has made some initial progress, hacking on
Angle and Qt to make this possible. Right now, only Windows and Direct3D
support HDR, though Intel is also working on support on Linux.
Other Qt applications, like VLC, implement a custom Direct3D widget, but
since all our code is opengl, that wouldn't have worked for Krita.
Here's the first report with some links to code: https://
dimula73.blogspot.com/2018/11/krita-fall-2018-sprint-results-hdr.html
What I'm wondering is, have other people been working on this as well, or
thinking about it? And what would be needed to get our code upstreamed. I
remember that trying to get the modernized code for QOpenGLPainter pushed
was quite hard, so I wanted to reach out as soon as possible.

I have been working on it and have a plan for it. The first steps is a color
space system, see https://codereview.qt-project.org/234095.
Then the very basic of extended colors https://codereview.qt-project.org/
233735
Next I have some work toward an 4xfp16 based rendering backend and image
support, possibly removing the 4xuint16 backend at the same time so there is
only one high-color precision backend.
Then we just need a way to connect that to a backing store, such as using
scRGB in OpenGL which looks like what you have been wokring on :)
Best Regards
'Allan
_______________________________________________
Development mailing list
https://lists.qt-project.org/listinfo/development