[Development] QtNetwork: using system proxy by default

[Peter Hartmann]

Hello,

I remember this has been discussed before, but yet again: How about
using the system proxy by default, at least on some platforms or by
configure switch? Right now every app developer has to call
QNetworkProxyFactory::setUseSystemConfiguration(true) in his code to use
the system proxies.
On desktop you might want to avoid this depending on your app, but on
mobile (e.g. Blackberry) it might happen that you just have to use the
system configuration and it will not work without it.

IMO there are 4 options:

1. leave as it is, tell everybody to call above method in their app
2. just #ifdef for the platforms where it should be used automatically
3. introduce a configure switch "-use-system-proxy" or so (default no)
4. enable usage of system proxy globally


1. sounds a bit cumbersome for me; I prefer 2. for pragmatic reasons. If
people think 3. is cleaner that is fine with me as well. As Shane told
me, using the system proxy on Windows might lead to several seconds of
delay until the synchronous method that determines the proxy returns, so
for now 4. seems too risky.

Please suggest / comment etc.


Peter

---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

[s***@accenture.com]

> -----Original Message-----
> From: development-bounces+shane.kearns=***@qt-project.org
> [mailto:development-bounces+shane.kearns=***@qt-project.org]
> On Behalf Of Peter Hartmann
> Sent: 11 October 2012 14:12
> To: ***@qt-project.org
> Subject: [Development] QtNetwork: using system proxy by default
>
> Hello,
>
> I remember this has been discussed before, but yet again: How about
> using the system proxy by default, at least on some platforms or by
> configure switch? Right now every app developer has to call
> QNetworkProxyFactory::setUseSystemConfiguration(true) in his code to
> use the system proxies.
> On desktop you might want to avoid this depending on your app, but on
> mobile (e.g. Blackberry) it might happen that you just have to use the
> system configuration and it will not work without it.

Many mobile networks have a mandatory http proxy for the access point.
System configuration is the only reasonable way to get this.

> IMO there are 4 options:
>
> 1. leave as it is, tell everybody to call above method in their app 2.
> just #ifdef for the platforms where it should be used automatically 3.
> introduce a configure switch "-use-system-proxy" or so (default no) 4.
> enable usage of system proxy globally
>
>
> 1. sounds a bit cumbersome for me; I prefer 2. for pragmatic reasons.
> If people think 3. is cleaner that is fine with me as well. As Shane
> told me, using the system proxy on Windows might lead to several
> seconds of delay until the synchronous method that determines the proxy
> returns, so for now 4. seems too risky.

A refinement of option 1 would be to include the setUseSystemConfiguration call in the SDK boilerplate code (along with using showFullScreen() instead of show() on mobiles).

The windows issues are documented in https://bugreports.qt-project.org/browse/QTBUG-10106
- normally it's ok, but the worst case is terrible.
- the worst case is reproduced with some consumer wlan routers, so affects many users.
Nobody complained yet about performance on Mac, but it's a more recent feature and hasn't been tested for perf.
Libproxy performance is unknown, I've only done functional testing.

In all cases, the API is synchronous.
For option 4, I think we need to make an asynchronous wrapper and use it internally anywhere we might block the UI thread with the synchronous version.

Subject to local law, communications with Accenture and its affiliates including telephone calls and emails (including content), may be monitored by our systems for the purposes of security and the assessment of internal compliance with Accenture policy.
______________________________________________________________________________________

www.accenture.com

[Thiago Macieira]

On quinta-feira, 11 de outubro de 2012 14.02.11, ***@accenture.com
wrote:
> Libproxy performance is unknown, I've only done functional testing.

I'd rather not use libproxy until we're certain we can tell it which JS
library to use. It needs to be one of ours, not webkit-gtk or mozjs.

--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center

[Hausmann Simon]

IMHO #4 gives the best out-of-the-box experience.

Is there a way to do the blocking winapi call in a thread on app start-up maybe?

How do other apps (i.e. Chrome) handle this without blocking the user experience?

Simon

--
Sendt fra min Nokia N911.10.12 15:12 skrev Peter Hartmann:
Hello,

I remember this has been discussed before, but yet again: How about
using the system proxy by default, at least on some platforms or by
configure switch? Right now every app developer has to call
QNetworkProxyFactory::setUseSystemConfiguration(true) in his code to use
the system proxies.
On desktop you might want to avoid this depending on your app, but on
mobile (e.g. Blackberry) it might happen that you just have to use the
system configuration and it will not work without it.

IMO there are 4 options:

1. leave as it is, tell everybody to call above method in their app
2. just #ifdef for the platforms where it should be used automatically
3. introduce a configure switch "-use-system-proxy" or so (default no)
4. enable usage of system proxy globally


1. sounds a bit cumbersome for me; I prefer 2. for pragmatic reasons. If
people think 3. is cleaner that is fine with me as well. As Shane told
me, using the system proxy on Windows might lead to several seconds of
delay until the synchronous method that determines the proxy returns, so
for now 4. seems too risky.

Please suggest / comment etc.


Peter

---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.
_______________________________________________
Development mailing list
***@qt-project.org
http://lists.qt-project.org/mailman/listinfo/developmen

[s***@accenture.com]

> IMHO #4 gives the best out-of-the-box experience.
>
> Is there a way to do the blocking winapi call in a thread on app start-
> up maybe?

Doesn't help if the first thing an application wants to do is download something from the network.
It would only help because of the workaround we already have to disable WPAD if it fails (which is not ideal if you start an app at home, then take your laptop to work)

> How do other apps (i.e. Chrome) handle this without blocking the user
> experience?

Chrome doesn't use the windows API, they make a request to "http://wpad" and interpret the pac script themselves.
Firefox is slow to load pages, but UI is not blocked (which points to using the windows API in a thread)

WPAD is inherently spoofable, but the Chrome method is worse than the windows API, as you can't tell if the name came from DNS or multicast name resolution.
In any case, we don't have a javascript interpreter available at the QtNetwork level.

Subject to local law, communications with Accenture and its affiliates including telephone calls and emails (including content), may be monitored by our systems for the purposes of security and the assessment of internal compliance with Accenture policy.
______________________________________________________________________________________

www.accenture.com

[Knoll Lars]

Hi Shane,

(adding the ML back in again…)

wouldn't Windows run the DHCP detection only once at startup (or when network status changes) and cache the result? Why would we need to re-run this inside the Qt app?

Cheers,
Lars

On Oct 15, 2012, at 4:00 PM, ***@accenture.com wrote:

> Hi Lars,
>
> It happens the first time only (because we have a workaround to disable autodetection if it fails)
> When I reproduced it, it was 12 seconds.
>
> The problem is windows sends a DHCP packet with a microsoft extension, and has a long timeout waiting for a reply.
> With good DHCP servers, they'll reply (negatively if they don't support that extension)
> With bad DHCP servers, there is no reply which triggers the problem.
>
> It seems like some wireless access points/home routers have a "bad" or incomplete DHCP implementation
>
> I agree that enabling system proxy by default is what most app developers would want.
> An option would be to always disable DHCP detection, which gets rid of this worst case.
> ________________________________________
> From: Knoll Lars [***@digia.com]
> Sent: 15 October 2012 08:39
> To: Kearns, Shane
> Subject: Re: [Development] QtNetwork: using system proxy by default
>
> I'd agree with Simon that #4 would be the best default option from a user experience point of view. Most likely it's also what most app developers want.
>
> Is the windows problem purely something happening once when initiating the first connection, or does it happen with every connection? And what does this mean in a real use case (ie. how many seconds are we talking about)?
>
> Cheers,
> Lars
>
> On Oct 11, 2012, at 4:34 PM, ***@accenture.com wrote:
>
>>> IMHO #4 gives the best out-of-the-box experience.
>>>
>>> Is there a way to do the blocking winapi call in a thread on app start-
>>> up maybe?
>>
>> Doesn't help if the first thing an application wants to do is download something from the network.
>> It would only help because of the workaround we already have to disable WPAD if it fails (which is not ideal if you start an app at home, then take your laptop to work)
>>
>>> How do other apps (i.e. Chrome) handle this without blocking the user
>>> experience?
>>
>> Chrome doesn't use the windows API, they make a request to "http://wpad" and interpret the pac script themselves.
>> Firefox is slow to load pages, but UI is not blocked (which points to using the windows API in a thread)
>>
>> WPAD is inherently spoofable, but the Chrome method is worse than the windows API, as you can't tell if the name came from DNS or multicast name resolution.
>> In any case, we don't have a javascript interpreter available at the QtNetwork level.
>>
>> Subject to local law, communications with Accenture and its affiliates including telephone calls and emails (including content), may be monitored by our systems for the purposes of security and the assessment of internal compliance with Accenture policy.
>> ______________________________________________________________________________________
>>
>> www.accenture.com
>>
>> _______________________________________________
>> Development mailing list
>> ***@qt-project.org
>> http://lists.qt-project.org/mailman/listinfo/development
>
>
>
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited.
>
> Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
>
> ______________________________________________________________________________________
>
> www.accenture.com
>

[s***@accenture.com]

Apparently not, I don't know why Microsoft implemented it this way.
The result is only cached if a PAC script is successfully downloaded.

The DHCP method doesn't serve the PAC url together with the IP address, it's a separate query sent to the DHCP server by the browser (or us)

If autodetection fails, it returns "no proxy" for this request, but tries autodetection again next time the API is called.

The autodetection procedure looks like this pseudocode (but is all one API call to us):
for(tries=0;tries<2;tries++) {
send(DHCP_INFORM requesting option 252);
if (wait_for_response(6 seconds)) {
pac_url = response;
break;
}
}
if(!pac_url) {
pac_url = "http://wpad/wpad.dat";
}

When using the windows API, we can enable either DNS, DHCP or both.
Using DNS only would be faster, but may not work for some users if their office network used DHCP only deployment.

Also, I checked today and it looks like Chromium have now implemented DHCP autodetection by themselves as well. They are checking it on startup & caching the results.
________________________________________
From: Knoll Lars [***@digia.com]
Sent: 15 October 2012 16:00
To: Kearns, Shane
Cc: <***@qt-project.org>
Subject: Re: [Development] QtNetwork: using system proxy by default

Hi Shane,

(adding the ML back in again…)

wouldn't Windows run the DHCP detection only once at startup (or when network status changes) and cache the result? Why would we need to re-run this inside the Qt app?

Cheers,
Lars

On Oct 15, 2012, at 4:00 PM, ***@accenture.com wrote:

> Hi Lars,
>
> It happens the first time only (because we have a workaround to disable autodetection if it fails)
> When I reproduced it, it was 12 seconds.
>
> The problem is windows sends a DHCP packet with a microsoft extension, and has a long timeout waiting for a reply.
> With good DHCP servers, they'll reply (negatively if they don't support that extension)
> With bad DHCP servers, there is no reply which triggers the problem.
>
> It seems like some wireless access points/home routers have a "bad" or incomplete DHCP implementation
>
> I agree that enabling system proxy by default is what most app developers would want.
> An option would be to always disable DHCP detection, which gets rid of this worst case.
> ________________________________________
> From: Knoll Lars [***@digia.com]
> Sent: 15 October 2012 08:39
> To: Kearns, Shane
> Subject: Re: [Development] QtNetwork: using system proxy by default
>
> I'd agree with Simon that #4 would be the best default option from a user experience point of view. Most likely it's also what most app developers want.
>
> Is the windows problem purely something happening once when initiating the first connection, or does it happen with every connection? And what does this mean in a real use case (ie. how many seconds are we talking about)?
>
> Cheers,
> Lars
>
> On Oct 11, 2012, at 4:34 PM, ***@accenture.com wrote:
>
>>> IMHO #4 gives the best out-of-the-box experience.
>>>
>>> Is there a way to do the blocking winapi call in a thread on app start-
>>> up maybe?
>>
>> Doesn't help if the first thing an application wants to do is download something from the network.
>> It would only help because of the workaround we already have to disable WPAD if it fails (which is not ideal if you start an app at home, then take your laptop to work)
>>
>>> How do other apps (i.e. Chrome) handle this without blocking the user
>>> experience?
>>
>> Chrome doesn't use the windows API, they make a request to "http://wpad" and interpret the pac script themselves.
>> Firefox is slow to load pages, but UI is not blocked (which points to using the windows API in a thread)
>>
>> WPAD is inherently spoofable, but the Chrome method is worse than the windows API, as you can't tell if the name came from DNS or multicast name resolution.
>> In any case, we don't have a javascript interpreter available at the QtNetwork level.
>>
>> Subject to local law, communications with Accenture and its affiliates including telephone calls and emails (including content), may be monitored by our systems for the purposes of security and the assessment of internal compliance with Accenture policy.
>> ______________________________________________________________________________________
>>
>> www.accenture.com
>>
>> _______________________________________________
>> Development mailing list
>> ***@qt-project.org
>> http://lists.qt-project.org/mailman/listinfo/development
>
>
>
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited.
>
> Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
>
> ______________________________________________________________________________________
>
> www.accenture.com
>

[Thiago Macieira]

On segunda-feira, 15 de outubro de 2012 16.33.39, ***@accenture.com
wrote:
> When using the windows API, we can enable either DNS, DHCP or both.
> Using DNS only would be faster, but may not work for some users if their
> office network used DHCP only deployment.
>
> Also, I checked today and it looks like Chromium have now implemented DHCP
> autodetection by themselves as well. They are checking it on startup &
> caching the results.

IIRC, DHCP requires binding to port 67 in order to send the request, which
requires root privileges on Unix systems. That means we can't do it.

I'd create a (wall-clock) timer and enable the DHCP resolution on Windows only
once every 10 minutes. I'd also use it as a fallback if and only if the DNS
resolution failed.

The only drawback is that we'll insist on a bad, cached result if the user
moves from a network with DHCP and no DNS, to a network with no autoproxy. We
could cache the host's IP addresses and discard the cache if they have changed
too. Since we'll do that only once every 10 minutes, the overhead will be
minimal.

--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center

[s***@accenture.com]

> -----Original Message-----
> From: development-bounces+shane.kearns=***@qt-project.org
> [mailto:development-bounces+shane.kearns=***@qt-project.org]
> On Behalf Of Thiago Macieira
> Sent: 15 October 2012 18:41
> To: ***@qt-project.org
> Subject: Re: [Development] QtNetwork: using system proxy by default
>
> On segunda-feira, 15 de outubro de 2012 16.33.39,
> ***@accenture.com
> wrote:
> > When using the windows API, we can enable either DNS, DHCP or both.
> > Using DNS only would be faster, but may not work for some users if
> > their office network used DHCP only deployment.
> >
> > Also, I checked today and it looks like Chromium have now implemented
> > DHCP autodetection by themselves as well. They are checking it on
> > startup & caching the results.
>
> IIRC, DHCP requires binding to port 67 in order to send the request,
> which requires root privileges on Unix systems. That means we can't do
> it.

Yes, that puts this outside our scope.
I'd rather not be implementing this only for windows.

> I'd create a (wall-clock) timer and enable the DHCP resolution on
> Windows only once every 10 minutes. I'd also use it as a fallback if
> and only if the DNS resolution failed.

That doesn't help with the 12 second freeze, since both methods fail if
you are on a network with no autoproxy.
We need to be calling WinHttpGetProxyForUrl or WinHttpDetectAutoProxyConfigUrl
in a thread, like we do for name resolutions.

There is also the non blocking WinHttpGetProxyForUrlEx, but that is new in windows 8.

> The only drawback is that we'll insist on a bad, cached result if the
> user moves from a network with DHCP and no DNS, to a network with no
> autoproxy. We could cache the host's IP addresses and discard the cache
> if they have changed too. Since we'll do that only once every 10
> minutes, the overhead will be minimal.
>

Caching the result and having some ways to invalidate the cache makes sense.
WinHttpDetectAutoProxyConfigUrl is the best thing to use, since it gives a pac file url,
which is the same thing as the static configuration would give us.

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited.

Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.

______________________________________________________________________________________

www.accenture.com

[Thiago Macieira]

On terça-feira, 16 de outubro de 2012 14.55.24, ***@accenture.com
wrote:
> > I'd create a (wall-clock) timer and enable the DHCP resolution on
> > Windows only once every 10 minutes. I'd also use it as a fallback if
> > and only if the DNS resolution failed.
>
> That doesn't help with the 12 second freeze, since both methods fail if
> you are on a network with no autoproxy.
> We need to be calling WinHttpGetProxyForUrl or
> WinHttpDetectAutoProxyConfigUrl in a thread, like we do for name
> resolutions.

Sounds good. I'd recommend reusing the hostinfo thread pool for that too.
--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center

[Peter Hartmann]

What I forgot to say in my first mail:
We would want to use that in 4.8, where we don't want to change the
default behaviour anyhow. How about option 3 below, where the default
can be set to "yes" in Qt5 globally or per platform once we are
confident it works?
The benefit of that option is that it can be easily backported to 4.8
and would not change behaviour, but whoever wants to use it can opt-in.

Peter


On 10/11/2012 03:12 PM, Peter Hartmann wrote:
> (...)
> 1. leave as it is, tell everybody to call above method in their app
> 2. just #ifdef for the platforms where it should be used automatically
> 3. introduce a configure switch "-use-system-proxy" or so (default no)
> 4. enable usage of system proxy globally
>
>
> 1. sounds a bit cumbersome for me; I prefer 2. for pragmatic reasons. If
> people think 3. is cleaner that is fine with me as well. As Shane told
> me, using the system proxy on Windows might lead to several seconds of
> delay until the synchronous method that determines the proxy returns, so
> for now 4. seems too risky.
>
> Please suggest / comment etc.
>
>
> Peter
>
> ---------------------------------------------------------------------
> This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.
> _______________________________________________
> Development mailing list
> ***@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development
>


---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.