[Development] QStringRef conversion functions

[Keith Gardner]

I was wondering if there would be a possibility of getting some of the conversion functions that are in QString implemented in QStringRef (toInt, toFloat, etc...). The reasoning behind this is that I am using the QXMLStreamReader to parse an XML file with over 88,704 floating point values. I am trying to improve the speed at which the file is parsed and I believe one of the bottlenecks is taking an attribute of QStringRef and copying its data into a QString in order to convert to a float. I would prefer that the copy never happen but that would require some more intelligence be added into the QStringRef class.

Would this be possible to add this in time for the Qt 5.0 release? If not, would it be possible to add this for Qt 5.1 release?

Keith Gardner
Software Engineer
Zebra Imaging
9801 Metric Blvd., Suite 200
Austin, TX 78758

[Girish Ramakrishnan]

Hi,

On Mon, Jun 11, 2012 at 11:38 AM, Keith Gardner

Post by Keith Gardner
I was wondering if there would be a possibility of getting some of the
conversion functions that are in QString implemented in QStringRef (toInt,
toFloat, etc...).  The reasoning behind this is that I am using the
QXMLStreamReader to parse an XML file with over 88,704 floating point
values.  I am trying to improve the speed at which the file is parsed and I
believe one of the bottlenecks is taking an attribute of QStringRef and
copying its data into a QString in order to convert to a float.  I would
prefer that the copy never happen but that would require some more
intelligence be added into the QStringRef class.

...

I am tempted to say it's 5.1 material but it would be nice to see how
much gains you are getting out of these new apis and how intrusive the
new code is. Can you push your change to gerrit ?

Thanks,
Girish

[Thiago Macieira]

Post by Girish Ramakrishnan
Hi,
On Mon, Jun 11, 2012 at 11:38 AM, Keith Gardner

Post by Keith Gardner
I was wondering if there would be a possibility of getting some of the
conversion functions that are in QString implemented in QStringRef (toInt,
toFloat, etc...). The reasoning behind this is that I am using the
QXMLStreamReader to parse an XML file with over 88,704 floating point
values. I am trying to improve the speed at which the file is parsed and
I
believe one of the bottlenecks is taking an attribute of QStringRef and
copying its data into a QString in order to convert to a float. I would
prefer that the copy never happen but that would require some more
intelligence be added into the QStringRef class.

I am tempted to say it's 5.1 material but it would be nice to see how
much gains you are getting out of these new apis and how intrusive the
new code is. Can you push your change to gerrit ?

...

We might be able to remove QStringRef before Qt 5.0 and replace it with plain
QString.

[Konstantin Ritt]

Post by Thiago Macieira

Post by Girish Ramakrishnan
Hi,
On Mon, Jun 11, 2012 at 11:38 AM, Keith Gardner

Post by Keith Gardner
I was wondering if there would be a possibility of getting some of the
conversion functions that are in QString implemented in QStringRef (toInt,
toFloat, etc...).  The reasoning behind this is that I am using the
QXMLStreamReader to parse an XML file with over 88,704 floating point
values.  I am trying to improve the speed at which the file is parsed and
I
believe one of the bottlenecks is taking an attribute of QStringRef and
copying its data into a QString in order to convert to a float.  I would
prefer that the copy never happen but that would require some more
intelligence be added into the QStringRef class.

I am tempted to say it's 5.1 material but it would be nice to see how
much gains you are getting out of these new apis and how intrusive the
new code is. Can you push your change to gerrit ?

We might be able to remove QStringRef before Qt 5.0 and replace it with plain
QString.

...

That is what I'd love to see in 5.0.
Making sub-QStrings as cheap as QStringRef currently is would give us
quite significant performance boost in some common use-cases. Then,
QStringRef definitely should go.

Konstantin

[a***@nokia.com]

Hi,

Post by Konstantin Ritt

Post by Thiago Macieira

Post by Girish Ramakrishnan
Hi,
On Mon, Jun 11, 2012 at 11:38 AM, Keith Gardner

Post by Keith Gardner
I was wondering if there would be a possibility of getting some of the
conversion functions that are in QString implemented in QStringRef (toInt,
toFloat, etc...). The reasoning behind this is that I am using the
QXMLStreamReader to parse an XML file with over 88,704 floating point
values. I am trying to improve the speed at which the file is parsed and
I
believe one of the bottlenecks is taking an attribute of QStringRef and
copying its data into a QString in order to convert to a float. I would
prefer that the copy never happen but that would require some more
intelligence be added into the QStringRef class.

I am tempted to say it's 5.1 material but it would be nice to see how
much gains you are getting out of these new apis and how intrusive the
new code is. Can you push your change to gerrit ?

We might be able to remove QStringRef before Qt 5.0 and replace it with plain
QString.

That is what I'd love to see in 5.0.
Making sub-QStrings as cheap as QStringRef currently is would give us
quite significant performance boost in some common use-cases. Then,
QStringRef definitely should go.

...

Won't a QString always require an allocation for the d-ptr? So it'll never be quite as cheap as QStringRef.

Cheers,

Aaron

[Thiago Macieira]

Post by a***@nokia.com
Won't a QString always require an allocation for the d-ptr? So it'll never
be quite as cheap as QStringRef.

Not exactly.

A QStringRef points to an existing QString. Therefore, if we take a QString
that is pointing to a substring of an existing QString, the d-ptr is already
allocated. The new QString would be as cheap as the current QStringRef.

The issue we'll run into is that a substring could be "holding hostage" a huge
chunk of memory.

[a***@nokia.com]

Hi,

Post by Thiago Macieira

Post by a***@nokia.com
Won't a QString always require an allocation for the d-ptr? So it'll never
be quite as cheap as QStringRef.

Not exactly.
A QStringRef points to an existing QString. Therefore, if we take a QString
that is pointing to a substring of an existing QString, the d-ptr is already
allocated. The new QString would be as cheap as the current QStringRef.

How does that work? Where do you store the substring's offset and length data?

Cheers,

Aaron

Post by Thiago Macieira
The issue we'll run into is that a substring could be "holding hostage" a huge
chunk of memory.
--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center
Intel Sweden AB - Registration Number: 556189-6027
Knarrarnäsgatan 15, 164 40 Kista, Stockholm, Sweden
_______________________________________________
Development mailing list
http://lists.qt-project.org/mailman/listinfo/development

[Thiago Macieira]

Post by a***@nokia.com

Post by Thiago Macieira
A QStringRef points to an existing QString. Therefore, if we take a
QString that is pointing to a substring of an existing QString, the
d-ptr is already allocated. The new QString would be as cheap as the
current QStringRef.

How does that work? Where do you store the substring's offset and length data?

In the QString.

You might have missed the other thread where we're discussing moving the data
pointer and length to the main class in QString, QByteArray and QVector.

I've begun doing that work, but turns out that there's a LOT of code to get
ported, starting with moc.

[André Pönitz]

Post by Thiago Macieira

Post by a***@nokia.com
Won't a QString always require an allocation for the d-ptr? So
it'll never be quite as cheap as QStringRef.

Not exactly.
A QStringRef points to an existing QString. Therefore, if we take a
QString that is pointing to a substring of an existing QString, the
d-ptr is already allocated. The new QString would be as cheap as the
current QStringRef.
The issue we'll run into is that a substring could be "holding
hostage" a huge chunk of memory.

This is not much different from current usage. The difference would
be the "outer" string contents being re-allocated as result of
non-const operations on it. This "works" (to a certain degree...)
with QStringRef, but wouldn't with "cheap substrings".

However, I still have to see such a "mutable" case in practice.
The QStringRef uses I am aware of are very limited in scope,
often just saving a single copy of (parts of) the owning strings.
Nice, but not even close to the potential of "cheap substrings".

Andre'

[André Pönitz]

Post by a***@nokia.com

That is what I'd love to see in 5.0. Making sub-QStrings as cheap
as QStringRef currently is would give us quite significant
performance boost in some common use-cases. Then, QStringRef
definitely should go.

Won't a QString always require an allocation for the d-ptr? So it'll
never be quite as cheap as QStringRef.

Depends on the implementation.

One suggestion was to store "enough" data inside the object proper to
not require "external" data fields for those cases. The amount of data
needed depends on the operations one wants to support. As far as I can
see, two pointers + one int (one direct pointer to char data, one
pointer to auxillary data like refcount, flags, alloc, one int for size)
are enough for "everything" including fast direct accesse to the
character array and size. But that's fat (20 byte on 64bit, 12 on 32).
Too fat even. 16/8 would be _much_ better.

If someone came up with a clever idea to _safely_ replace the auxiliary
pointer by a 32 bit offset to the data pointer there is even some hope
for a _really_ good solution, at least for 64 bit....

Andre'

[Keith Gardner]

Sorry to backtrack this thread, but I have some performance information. Just FYI, I haven't made any code changes to Qt, yet.

To give some background information, I am parsing a single xml file with 88,704 floating point values using the QXmlStreamReader. I am running in Windows 7 x64 and am using an SSD to store the file. My compiler is Visual Studio 2010 with a Win32 configuration.

I would take the QStringRef for the argument, convert it to a string using the toString() function and then use toFloat().

This method would parse the entire document in 608 milliseconds in release mode.
stringRef.toString().toFloat()
Changing that code to use wcstod on the QStringRef resulted in parsing the document in 467 millesconds.
wcstod((const wchar_t*)stringref.unicode(),0);

In debug mode, this goes from 2814 milliseconds down to 1850 milliseconds.

I would say that there is a big payout with changing what we currently have.

Thanks,
Keith

-----Original Message-----
From: Girish Ramakrishnan [mailto:***@forwardbias.in]
Sent: Monday, June 11, 2012 4:55 PM
To: Keith Gardner
Cc: ***@qt-project.org
Subject: Re: [Development] QStringRef conversion functions

Hi,

On Mon, Jun 11, 2012 at 11:38 AM, Keith Gardner

Post by Keith Gardner
I was wondering if there would be a possibility of getting some of the
conversion functions that are in QString implemented in QStringRef (toInt,
toFloat, etc...).  The reasoning behind this is that I am using the
QXMLStreamReader to parse an XML file with over 88,704 floating point
values.  I am trying to improve the speed at which the file is parsed and I
believe one of the bottlenecks is taking an attribute of QStringRef and
copying its data into a QString in order to convert to a float.  I would
prefer that the copy never happen but that would require some more
intelligence be added into the QStringRef class.

...

I am tempted to say it's 5.1 material but it would be nice to see how
much gains you are getting out of these new apis and how intrusive the
new code is. Can you push your change to gerrit ?

Thanks,
Girish

[Girish Ramakrishnan]

Hi Keith,
Yes, it makes sense to add QStringRef::toInt/Float/Double. The entire
point of QStringRef was to make things faster and having those apis
will definitely be an improvement.

Girish

On Wed, Jun 13, 2012 at 12:24 PM, Keith Gardner

Sorry to backtrack this thread, but I have some performance information.  Just FYI, I haven't made any code changes to Qt, yet.
To give some background information, I am parsing a single xml file with 88,704 floating point values using the QXmlStreamReader.  I am running in Windows 7 x64 and am using an SSD to store the file.  My compiler is Visual Studio 2010 with a Win32 configuration.
I would take the QStringRef for the argument, convert it to a string using the toString() function and then use toFloat().
       This method would parse the entire document in 608 milliseconds in release mode.
               stringRef.toString().toFloat()
       Changing that code to use wcstod on the QStringRef resulted in parsing the document in 467 millesconds.
               wcstod((const wchar_t*)stringref.unicode(),0);
In debug mode, this goes from 2814 milliseconds down to 1850 milliseconds.
I would say that there is a big payout with changing what we currently have.
Thanks,
Keith
-----Original Message-----
Sent: Monday, June 11, 2012 4:55 PM
To: Keith Gardner
Subject: Re: [Development] QStringRef conversion functions
Hi,
On Mon, Jun 11, 2012 at 11:38 AM, Keith Gardner

Post by Keith Gardner
I was wondering if there would be a possibility of getting some of the
conversion functions that are in QString implemented in QStringRef (toInt,
toFloat, etc...).  The reasoning behind this is that I am using the
QXMLStreamReader to parse an XML file with over 88,704 floating point
values.  I am trying to improve the speed at which the file is parsed and I
believe one of the bottlenecks is taking an attribute of QStringRef and
copying its data into a QString in order to convert to a float.  I would
prefer that the copy never happen but that would require some more
intelligence be added into the QStringRef class.

I am tempted to say it's 5.1 material but it would be nice to see how
much gains you are getting out of these new apis and how intrusive the
new code is. Can you push your change to gerrit ?
Thanks,
Girish

...