From: Helen Craigman Thank you, Mark.

I guess I don't have a choice but to do it this way. (Although it introduces some complications).

Many thanks - Helen
----- Original Message -----
From: Mark Dootson
To: Helen Craigman
Sent: Friday, 24 May, 2013 14:18
Subject: Re: Trying to trigger wxEVT_TREELIST_ITEM_CHECKED programmatically: not implemented?


Hi,

On 24/05/2013 11:19, Helen Craigman wrote:
> Hi Mark
>

> In my code, I do use the CheckItem method, and that's working fine.
> The issue is, after I change the item check state, the event handler
> for EVT_TREELIST_ITEM_CHECKED does not get
> triggered,

Yes, this particular event is only triggered when the change is caused
by the user interface. That is a wxWidgets design decision /feature /
bug - depending on your view. The behaviour is documented though.

> Studying the documentation, I undestood that the way to
> have the event handler invoked, is to programmatically raise the "command
> event" (in this case: EVT_TREELIST_ITEM_CHECKED).

No. (as noted previously). You can created your own custom events but
'mimicking' a built in event is not the way to go. An actual
wxTreeListEvent derives from wxNotifyEvent - you would really need to
create the event properly.

>
> Now an event handler has this in the beginning:
> Quote ----------
> my( $this, $event ) = @_;
> $item = $event->GetItem;
> Unquote -------
>
> So, since it's looking for an event object, I don't know how to call it,
> except for raising EVT_TREELIST_ITEM_CHECKED.
> And in order to raise the event programatically, you need
> wxEVT_TREELIST_ITEM_CHECKED.
>
> By the way, other command event constants do get exported in wxPerl
> (for example wxEVT_COMMAND_CHECKBOX_CLICKED).

true, but this isn't to allow end user usage, rather it it just the way
the event 'macro' sub routines are implemented for this particular event.

>
> How do you propose I should do it?

sub handle_item_change {
my($self, $item) = @_;
# do stuff with $item;
}

sub on_event_item_checked {
my($self, $event) = @_;
my $item = $event->GetItem;
$self->handle_item_change( $item );
}

sub set_item_state_by_code {
my($self, $item, $newstate) = @_;
$self->{treelist}->CheckItem($item, $newstate);
$self->handle_item_change( $item );
}


Regards

Mark

From: Mark Dootson Hi,

On 24/05/2013 11:19, Helen Craigman wrote:
> Hi Mark
>

> In my code, I do use the CheckItem method, and that's working fine.
> The issue is, after I change the item check state, the event handler
> for EVT_TREELIST_ITEM_CHECKED does not get
> triggered,

Yes, this particular event is only triggered when the change is caused
by the user interface. That is a wxWidgets design decision /feature /
bug - depending on your view. The behaviour is documented though.

> Studing the documentation, I undestood that the way to
> have the event handler invoked, is to programatically raise the "command
> event" (in this case: EVT_TREELIST_ITEM_CHECKED).

No. (as noted previously). You can created your own custom events but
'mimicking' a built in event is not the way to go. An actual
wxTreeListEvent derives from wxNotifyEvent - you would really need to
create the event properly.

>
> Now an event handler has this in the beginning:
> Quote ----------
> my( $this, $event ) = @_;
> $item = $event->GetItem;
> Unquote -------
>
> So, since it's looking for an event object, I don't know how to call it,
> except for raising EVT_TREELIST_ITEM_CHECKED.
> And in order to raise the event programatically, you need
> wxEVT_TREELIST_ITEM_CHECKED.
>
> By the way, other command event constants do get exported in wxPerl
> (for example wxEVT_COMMAND_CHECKBOX_CLICKED).

true, but this isn't to allow end user usage, rather it it just the way
the event 'macro' sub routines are implemented for this particular event.

>
> How do you propose I should do it?

sub handle_item_change {
my($self, $item) = @_;
# do stuff with $item;
}

sub on_event_item_checked {
my($self, $event) = @_;
my $item = $event->GetItem;
$self->handle_item_change( $item );
}

sub set_item_state_by_code {
my($self, $item, $newstate) = @_;
$self->{treelist}->CheckItem($item, $newstate);
$self->handle_item_change( $item );
}


Regards

Mark

From: Helen Craigman Hi Mark

In the OP, I have posted just a code snippet, in order not to burden the
forum.

In my code, I do use the CheckItem method, and that's working fine.
The issue is, after I change the item check state, the event handler
for EVT_TREELIST_ITEM_CHECKED does not get
triggered, since the change did not originate from a widget.
I need to have the event handler invoked in order to take care of
the state change implications.

Studing the documentation, I undestood that the way to
have the event handler invoked, is to programatically raise the "command
event" (in this case: EVT_TREELIST_ITEM_CHECKED).

Now an event handler has this in the beginning:
Quote ----------
my( $this, $event ) = @_;
$item = $event->GetItem;
Unquote -------

So, since it's looking for an event object, I don't know how to call it,
except for raising EVT_TREELIST_ITEM_CHECKED.
And in order to raise the event programatically, you need
wxEVT_TREELIST_ITEM_CHECKED.

By the way, other command event constants do get exported in wxPerl
(for example wxEVT_COMMAND_CHECKBOX_CLICKED).

How do you propose I should do it?

Many TIA - Helen.

Johan: you are right, of course. It's 5.16.3 with 2.9.4, and not
like I wrote mistakenly in the OP.

when you
----- Original Message -----
From: Mark Dootson
To: wxperl-users@perl.org
Sent: Friday, 24 May, 2013 12:35
Subject: Re: Trying to trigger wxEVT_TREELIST_ITEM_CHECKED programmatically:
not implemented?


Hi Helen,

On 24/05/2013 04:43, Helen Craigman wrote:
> Hi wxPerl-users
> I am using wxPerl with wxWidgets 2.9.5. I am using TreeListCtrl, and need
> to implement triggering EVT_COMMAND_TREELIST_ITEM_CHECKED
> programmatically.

In Wx we don't change state of anything by raising events. To change
state in our code, we call the object methods. To change the checked
state of a wxTreeListItem

use Wx qw( :checkbox );
....

$wxtreelistctrl->CheckItem(
$wxtreelistitem,
wxCHK_CHECKED
);

$wxtreelistctrl->CheckItem(
$wxtreelistitem,
wxCHK_UNCHECKED
);

> ---------------------------
> For either wxEVT_TREELIST_ITEM_CHECKED or
> wxEVT_COMMAND_TREELIST_ITEM_CHECKED, compilation fails with the error:
>
> ------------
> "wxEVT_TREELIST_ITEM_CHECKED" is not exported by the Wx module
> ------------

As noted above, the wxEVT_TREELIST_ITEM_CHECKED constant is not exported
at it has no practical use in user code.

Hope it helps

Mark

From: Mark Dootson Hi Helen,

On 24/05/2013 04:43, Helen Craigman wrote:
> Hi wxPerl-users
> I am using wxPerl with wxWidgets 2.9.5. I am using TreeListCtrl, and need
> to implement triggering EVT_COMMAND_TREELIST_ITEM_CHECKED
> programmatically.

In Wx we don't change state of anything by raising events. To change
state in our code, we call the object methods. To change the checked
state of a wxTreeListItem

use Wx qw( :checkbox );
....

$wxtreelistctrl->CheckItem(
$wxtreelistitem,
wxCHK_CHECKED
);

$wxtreelistctrl->CheckItem(
$wxtreelistitem,
wxCHK_UNCHECKED
);

> ---------------------------
> For either wxEVT_TREELIST_ITEM_CHECKED or
> wxEVT_COMMAND_TREELIST_ITEM_CHECKED, compilation fails with the error:
>
> ------------
> "wxEVT_TREELIST_ITEM_CHECKED" is not exported by the Wx module
> ------------

As noted above, the wxEVT_TREELIST_ITEM_CHECKED constant is not exported
at it has no practical use in user code.

Hope it helps

Mark

From: Johan Vromans "Helen Craigman" <ga0982@gmail.com> writes:

> System: Windows XP SP3, Citrus Perl 5.16.4, wxWidgets 2.9.5

Interesting... The latest version I can find on the downloads site is
Citrus Perl 5.16.3, with wxWidgets 2.9.4.

> "wxEVT_TREELIST_ITEM_CHECKED" is not exported by the Wx module

It seems that the TreeList controls are not yet wrapped in this version
of wxPerl. I'm sure Mark will fix that soon.

-- Johan

From: Helen Craigman Hi wxPerl-users

I am using wxPerl with wxWidgets 2.9.5. I am using TreeListCtrl, and need
to implement triggering EVT_COMMAND_TREELIST_ITEM_CHECKED
programmatically.

Here is a code excerpt:
------------------------

# ...
use Wx qw( :treectrl :listctrl wxDefaultPosition wxDefaultSize
# ...
wxEVT_TREELIST_ITEM_CHECKED);
# ...
my $tree = Wx::TreeListCtrl->new( $panel_tree, -1, wxDefaultPosition,
[400,200], wxTL_CHECKBOX|wxTL_MULTIPLE);
# ...

# Create an EVT_COMMAND_TREELIST_ITEM_CHECKED artificially, and
# trigger it
my $evt = Wx::CommandEvent->new(wxEVT_COMMAND_TREELIST_ITEM_CHECKED,
$tree->GetId);
$evt->SetEventObject($tree);
$evt->SetId($tree->GetId);
$tree->GetEventHandler->ProcessEvent($evt);
# ...
---------------------------

For either wxEVT_TREELIST_ITEM_CHECKED or
wxEVT_COMMAND_TREELIST_ITEM_CHECKED, compilation fails with the error:

------------
"wxEVT_TREELIST_ITEM_CHECKED" is not exported by the Wx module
------------
Your advice will be appreciated.
Many TIA - Helen

System: Windows XP SP3, Citrus Perl 5.16.4, wxWidgets 2.9.5

From: Helen Craigman Dear Mark and other experts:

I trust you have successfully installed 2.9.4 on Strawberry Perl.

I would be thankful to you, though, if you could look at the error messages
here: www.perlmonks.org/?node_id=1033560 and comment on what was done (or
gone) wrong.

Please note (see there) that another user [bulk88] got a test failure at:

perl.exe "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib\lib',
'blib\arch')" t/*.t

(same place as me) (his error messages are a bit diffrent though, since, I
think, he used MS VC++ and I use gcc coming with the SB Perl installation).

Many TIA

Helen

>From: Mark Dootson
>Date: May 12, 2013 11:30
>Subject: Re: Strawberry Perl 5.16.3 with wxWidgets 2.9.4 and wxPerl latest
>Message ID: 518F7D30.3030903@znix.com
>Hi,

>Building Wx for Strawberry is V simple. I've no idea why HelenCr is
>having problems. Anyhow, I have been through the process of building
>with Strawberry, found that it works perfectly ok - and posted to Perl
>Monks.

>Cheers

>Mark

>On 11/05/2013 22:42, James Lynes wrote:
>> Thanks Johan.
>>
>> I have also suggested installing Citrus Perl on the PM thread and an AM
>> has suggested ppm several times. No idea why HelenCr continues to try
>> and build it herself. For my part I'm just trying to be helpful and
>> maybe learn something in the process.
>>
>> Thanks again for the response.
>>
>> James
>>
>>
>> On Sat, May 11, 2013 at 5:03 PM, Johan Vromans <jvromans@squirrel.nl
>> <mailto:jvromans@squirrel.nl>> wrote:
>>
>> James Lynes <jmlynesjr@gmail.com <mailto:jmlynesjr@gmail.com>>
>> writes:
>>
>> > Has anyone successfully built Strawberry Perl 5.16.3 with
>> wxWidgets 2.9.4
>> > and wxPerl .9921?
>>
>> My *personal* advice is to not bother with Strawberry, but install
>> Citrus Perl which gives you 5.16.3 and Wx 2.9.4 cpmplete out of the
>> box.
>>
>> -- Johan

From: Helen Craigman Dear Mark and other experts:

I trust you have successfully installed 2.9.4 on Strawberry Perl.

I would be thankful to you, though, if you could look at the error messages here: www.perlmonks.org/?node_id=1033560 and comment on what was done (or gone) wrong.

Please note (see there) that another user [bulk88] got a test failure at:

perl.exe "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib\lib', 'blib\arch')" t/*.t

(same place as me) (his error messages are a bit diffrent though, since, I think, he used MS VC++ and I use gcc coming with the SB Perl installation).

Many TIA

Helen

>From: Mark Dootson
>Date: May 12, 2013 11:30
>Subject: Re: Strawberry Perl 5.16.3 with wxWidgets 2.9.4 and wxPerl latest
>Message ID: 518F7D30.3030903@znix.com
>Hi,

>Building Wx for Strawberry is V simple. I've no idea why HelenCr is
>having problems. Anyhow, I have been through the process of building
>with Strawberry, found that it works perfectly ok - and posted to Perl Monks.

>Cheers

>Mark

>On 11/05/2013 22:42, James Lynes wrote:
>> Thanks Johan.
>>
>> I have also suggested installing Citrus Perl on the PM thread and an AM
>> has suggested ppm several times. No idea why HelenCr continues to try
>> and build it herself. For my part I'm just trying to be helpful and
>> maybe learn something in the process.
>>
>> Thanks again for the response.
>>
>> James
>>
>>
>> On Sat, May 11, 2013 at 5:03 PM, Johan Vromans <jvromans@squirrel.nl
>> <mailto:jvromans@squirrel.nl>> wrote:
>>
>> James Lynes <jmlynesjr@gmail.com <mailto:jmlynesjr@gmail.com>> writes:
>>
>> > Has anyone successfully built Strawberry Perl 5.16.3 with
>> wxWidgets 2.9.4
>> > and wxPerl .9921?
>>
>> My *personal* advice is to not bother with Strawberry, but install
>> Citrus Perl which gives you 5.16.3 and Wx 2.9.4 cpmplete out of the box.
>>
>> -- Johan

From: James Lynes Mark:

Thanks once again for you assistance.

James



On Sun, May 12, 2013 at 7:29 AM, Mark Dootson <mark.dootson@znix.com> wrote:

> Hi,
>
> Building Wx for Strawberry is V simple. I've no idea why HelenCr is having
> problems. Anyhow, I have been through the process of building with
> Strawberry, found that it works perfectly ok - and posted to Perl Monks.
>
> Cheers
>
> Mark
>
>
> On 11/05/2013 22:42, James Lynes wrote:
>
>> Thanks Johan.
>>
>> I have also suggested installing Citrus Perl on the PM thread and an AM
>> has suggested ppm several times. No idea why HelenCr continues to try
>> and build it herself. For my part I'm just trying to be helpful and
>> maybe learn something in the process.
>>
>> Thanks again for the response.
>>
>> James
>>
>>
>> On Sat, May 11, 2013 at 5:03 PM, Johan Vromans <jvromans@squirrel.nl
>> <mailto:jvromans@squirrel.nl>> wrote:
>>
>>
>> James Lynes <jmlynesjr@gmail.com <mailto:jmlynesjr@gmail.com>>
>> writes:
>>
>> > Has anyone successfully built Strawberry Perl 5.16.3 with
>> wxWidgets 2.9.4
>> > and wxPerl .9921?
>>
>> My *personal* advice is to not bother with Strawberry, but install
>> Citrus Perl which gives you 5.16.3 and Wx 2.9.4 cpmplete out of the
>> box.
>>
>> -- Johan
>>
>>
>>
>

From: Mark Dootson Hi,

Building Wx for Strawberry is V simple. I've no idea why HelenCr is
having problems. Anyhow, I have been through the process of building
with Strawberry, found that it works perfectly ok - and posted to Perl
Monks.

Cheers

Mark

On 11/05/2013 22:42, James Lynes wrote:
> Thanks Johan.
>
> I have also suggested installing Citrus Perl on the PM thread and an AM
> has suggested ppm several times. No idea why HelenCr continues to try
> and build it herself. For my part I'm just trying to be helpful and
> maybe learn something in the process.
>
> Thanks again for the response.
>
> James
>
>
> On Sat, May 11, 2013 at 5:03 PM, Johan Vromans <jvromans@squirrel.nl
> <mailto:jvromans@squirrel.nl>> wrote:
>
> James Lynes <jmlynesjr@gmail.com <mailto:jmlynesjr@gmail.com>> writes:
>
> > Has anyone successfully built Strawberry Perl 5.16.3 with
> wxWidgets 2.9.4
> > and wxPerl .9921?
>
> My *personal* advice is to not bother with Strawberry, but install
> Citrus Perl which gives you 5.16.3 and Wx 2.9.4 cpmplete out of the box.
>
> -- Johan
>
>

From: James Lynes Thanks Johan.

I have also suggested installing Citrus Perl on the PM thread and an AM has
suggested ppm several times. No idea why HelenCr continues to try and build
it herself. For my part I'm just trying to be helpful and maybe learn
something in the process.

Thanks again for the response.

James


On Sat, May 11, 2013 at 5:03 PM, Johan Vromans <jvromans@squirrel.nl> wrote:

> James Lynes <jmlynesjr@gmail.com> writes:
>
> > Has anyone successfully built Strawberry Perl 5.16.3 with wxWidgets 2.9.4
> > and wxPerl .9921?
>
> My *personal* advice is to not bother with Strawberry, but install
> Citrus Perl which gives you 5.16.3 and Wx 2.9.4 cpmplete out of the box.
>
> -- Johan
>

From: Johan Vromans James Lynes <jmlynesjr@gmail.com> writes:

> Has anyone successfully built Strawberry Perl 5.16.3 with wxWidgets 2.9.4
> and wxPerl .9921?

My *personal* advice is to not bother with Strawberry, but install
Citrus Perl which gives you 5.16.3 and Wx 2.9.4 cpmplete out of the box.

-- Johan

From: James Lynes Good day:

Has anyone successfully built Strawberry Perl 5.16.3 with wxWidgets 2.9.4
and wxPerl .9921?

If so, and you have a few moments could you go over to Perl Monks and look
at the thread "Build of Alien::wxWidgets failing: help". Poster HelenCr has
been fighting with this for almost a week now.

Thanks for any knowledge that can be passed along.

James

Not a Windows user.....

From: Steve Cookson So ideally, you want an option wxSTAY_AT_BACK :)

However, on a different point, on Linux, if I use

$pid = Wx::ExecuteArgs( [ ... ], wxEXEC_SYNC );

I could probably use the $pid to show the application (at least, I can if I
right-click, and so probably I could from a command, too). I don't know
about windows, but maybe there is a similar command.

Maybe I'd have to use wxEXEC_ASYNC, so I could loop around checking until it
started and then put it at the front.

Good luck.

Regards

Steve




-----Original Message-----
From: Johan Vromans [mailto:jvromans@squirrel.nl]
Sent: 03 May 2013 08:53
To: Steve Cookson
Subject: Re: Problem with fg/bg windows

"Steve Cookson" <steve.cookson@sca-uk.com> writes:

> There is a frame-level option wxSTAY_ON_TOP

Yes, but this would need to be applied to application B. Unfortunately,
Application B is an external application I do not have control over.

-- Johan

From: Steve Cookson Hi Johan,

Forgot to copy list.

There is a frame-level option wxSTAY_ON_TOP

http://forums.wxwidgets.org/viewtopic.php?f=1
<http://forums.wxwidgets.org/viewtopic.php?f=1&t=37048> &t=37048

Maybe that would help.

Regards

Steve

On 3 May 2013 03:46, "Johan Vromans" <jvromans@squirrel.nl> wrote:


Hi,

I have a fullscreen application, A, that runs on Windows.
This application starts another fullscreen application, B.

Sometimes, especially when starting B takes several seconds, application
A is marked as "non responsive", which apparently raises its window. If
in the mean time application B showed its main window, the window of A
now covers the window of B, effectively dead-locking the system.

A crude solution is hiding A's main window:

$topwindow->Show(0);
system( B );
$topwindow->Show(1);

This is suboptimal since the user may get a view of, and opportunity to
interact with, the desktop, which is not desired.

Using

Wx::ExecuteArgs( [ ... ], wxEXEC_SYNC );

instead of 'system' does not seem to make a difference.

Any suggestions on how to improve this?

-- Johan

From: Johan Vromans Hi,

I have a fullscreen application, A, that runs on Windows.
This application starts another fullscreen application, B.

Sometimes, especially when starting B takes several seconds, application
A is marked as "non responsive", which apparently raises its window. If
in the mean time application B showed its main window, the window of A
now covers the window of B, effectively dead-locking the system.

A crude solution is hiding A's main window:

$topwindow->Show(0);
system( B );
$topwindow->Show(1);

This is suboptimal since the user may get a view of, and opportunity to
interact with, the desktop, which is not desired.

Using

Wx::ExecuteArgs( [ ... ], wxEXEC_SYNC );

instead of 'system' does not seem to make a difference.

Any suggestions on how to improve this?

-- Johan

From: Mark Dootson Hi,

On 02/05/2013 10:34, Johan Vromans wrote:

> The question is: do we consider wxWidgets to be 'external world'. That
> answer is most likely 'yes'. But more important: do we consider wxPerl
> to be 'external world'? I'd say 'no'. Therefore, what I'd expect to pass
> to a wxPerl routine is a string in Perl's internal encoding. The wxPerl
> wrapper should take care of encoding the string into whatever encoding
> wxWidgets requires.
>
> Compare this to {some, several, many, all} DBD drivers that handle the
> internal/UTF-8 conversions transparently.
>
> Alternatively, it's an (almost) equally good decision to require all
> strings passed to a wxPerl routine to be encoded in UTF-8. For a program
> that is correctly equipped to handle multibyte encoded strings it will
> not make a difference.
>
> -- Johan

I'd agree with that. For Wx 0.9922 I've changed the conversion to
wxString so that it always uses a UTF-8 conversion.

At least I'm assuming that's what SvPVutf8_nolen does. It seems to, and
the docs say it does. Contrary advice from perlapi utf8 gurus most
welcome. Essentially I want

char * buffer = SomePerlApi( SV );
where &buffer is the address at the start of a stream of UTF-8 octets.
I think SvPVutf8_nolen( SV ) does the job.

In testing this change though, I had to go back to Perl 5.8.9 to
contrive a case where it made any difference. Even there, it might be an
issue with older module versions - but given it was Perl 5.8.9 I was not
inclined to investigate further.

So I'd say from testing that wxPerl already handled things transparently
and continues to do so.

In short, I think that

my $string = decode( $someencoding, $externaldata );

or

my $string = $some_other_Perl_string;

# ... optionally some string operations on $string

$wxobject->SetValue( $string );

Should always work and if it doesn't it is a probably a bug in wxPerl.
This is regardless of utf8 flags etc etc. It should just work from an
end user perspective.

If someone demonstrates an instance where the above doesn't work, I'll
endeavour to get it fixed.


Cheers

Mark

From: Johan Vromans Mark Dootson <mark.dootson@znix.com> writes:

> None of my machines can be ASCII or EBCDIC by whatever definition this
> doc entry uses [...] What exactly is an ASCII machine?

ASCII just means: non-EBCDIC.

> Anyhow, I find that after
>
> $string = decode("utf8", $octets)
>
> $string always has the utf8 flag set, even if $octets is entirely
> ASCII data.

Yes, that is my experience as well. And this does not conform to the
documentation. I've been informed that the documentation is wrong, and
that *all* documentation concerning the utf8 flag seems to be wrong as
well -- or misleading at best.

> my $string = decode("utf8", $octets);
>
> ...do whatever string operations in Perl
>
> $wxobject->SetValue($string);
>
> will always work OK just providing something you did in
> ...do whatever string operations in Perl
> didn't strip the utf8 flag off.

I seem to recall that Perl sometimes automatically will upgrade a
string, but I never heard about downgrading. So unless you downgrade
explicitly this is not supposed happen.

> Clear as mud?
> For me too.

Me too :(

> For myself, if I were writing code that handled multibyte char sets in
> existing Wx releases I would do
>
> my $string = decode("UTF-8", $octets);
>
> ...do whatever string operations in Perl
>
> utf8::upgrade($string);
> $wxobject->SetValue($string);
>
> If you believe that utf8::upgrade($string) is not necessary, then
> don't use it. If you're belief is correct, all will work fine.

As long as the string operations did not downgrade the string,
utf8::upgrade($string) is a harmless no-op.

> The more I think about this and actually test what happens, the more I
> lean towards just always expecting string values that get passed to
> wxPerl wrappers to be UTF-8.

When passing a Perl string to the external world it must always be
encoded. This already holds for writing data to files.

The question is: do we consider wxWidgets to be 'external world'. That
answer is most likely 'yes'. But more important: do we consider wxPerl
to be 'external world'? I'd say 'no'. Therefore, what I'd expect to pass
to a wxPerl routine is a string in Perl's internal encoding. The wxPerl
wrapper should take care of encoding the string into whatever encoding
wxWidgets requires.

Compare this to {some, several, many, all} DBD drivers that handle the
internal/UTF-8 conversions transparently.

Alternatively, it's an (almost) equally good decision to require all
strings passed to a wxPerl routine to be encoded in UTF-8. For a program
that is correctly equipped to handle multibyte encoded strings it will
not make a difference.

-- Johan

From: Octavian Rasnita From: "Mark Dootson" <mark.dootson@znix.com>


> Hi,
>
> On 02/05/2013 00:17, Steve Cookson wrote:
>
>> Or just
>>
>> sub libDecode ($$){
>> return decode(@_);
>> }
>>
>> At least all the things that might go wrong will all be here.
>
> You're unduly worried ( probably my fault ).
>
> my $string = decode($encoding, $binary);
>


So no need of utf8::upgrade()?

Octavian

From: Mark Dootson Hi,

On 01/05/2013 20:14, Octavian Rasnita wrote:
> Yep, good to know.
>
> It would be nice if WxPerl would announce somehow that a font doesn't
> have the necessary glifs (maybe with a warning).

Nice to have, but there is no reasonable and practical implementation I
can think of. I am aware of how to check if a given font has a
particular glyph technically - but it is non trivial. This also won't
tell me if the operating system font handler will correctly substitute a
glyph from elsewhere. And, it isn't part of wxWidgets.

It's only a problem in decade old Windows XP anyway. Elsewhere if you
just accept the default GUI font for text type controls all will be the
best it can be.

> And btw, if the constant wxVSCROLL is not necessary, (do I understood
> correctly that it is not necessary under Linux either?) and if it
> creates problems under Windows, it would be helpful if it would give an
> error when it is used, or it would be helpful if it would be skipped at
> least under platforms that have a problem with it.

Not quite that simple. It is really just a number that gets added to a
flags mask. It isn't unique. The number might actually mean something.
It just doesn't mean wxVSCROLL in this context.

I suppose it might be possible to manually go through the wxWidgets code
and check what the acceptable flags are for every class. But we'd have
to re-do it for every wxWidgets release. For me, that is too much
maintenance burden.

Sorry I couldn't add these but hope you understand why.

Regards

Mark

From: Mark Dootson Hi,

On 02/05/2013 00:17, Steve Cookson wrote:

> Or just
>
> sub libDecode ($$){
> return decode(@_);
> }
>
> At least all the things that might go wrong will all be here.

You're unduly worried ( probably my fault ).

my $string = decode($encoding, $binary);

Is fine.

Cheers

Mark

From: Steve Cookson Hi Guys,

I don't have anything broken in this release in the 2 languages that I
currently support (English and Portuguese). But even so, the whole utf8
process has been a bit time consuming. As we become more multilingual, I'm
thinking that a global change to rename decode to libDecode (or something)
might put all the contentious stuff in one place rather than spread out all
over my code.

So, at the risk of suggesting that my Grandmother suck eggs:

sub libDecode ($$){
#
##### Standard decode subroutine to put all UTF bits in one place.
#
my $encoding = shift;
my $string = shift;

# I could even put:
#
# $encoding = "UTF-8" if $encoding eq "utf8";
# just changed in one place in case it doesn't work.

$string = decode(encoding, string );
utf8::upgrade($string);
return $string;
}

Or just

sub libDecode ($$){
return decode(@_);
}

At least all the things that might go wrong will all be here.

Regards

Steve.

-----Original Message-----
From: Mark Dootson [mailto:mark.dootson@znix.com]
Sent: 01 May 2013 18:11
To: wxperl-users@perl.org
Subject: wxString and UTF-8, utf8 etc etc etc again

Hi,

perldoc for the module Encode says:

---------------------------------------------------------------------
CAVEAT: When you run $string = decode("utf8", $octets) , then $string
might not be equal to $octets. Though both contain the same data, the
UTF8 flag for $string is on unless $octets consists entirely of ASCII
data on ASCII machines or EBCDIC on EBCDIC machines.
---------------------------------------------------------------------

None of my machines can be ASCII or EBCDIC by whatever definition this
doc entry uses as my testing on a variety of platforms shows that on
Perl 5.8.8 through Perl 5.16.2 the above is most certainly not true.
I shouldn't be surprised really. What exactly is an ASCII machine?

Anyhow, I find that after

$string = decode("utf8", $octets)

$string always has the utf8 flag set, even if $octets is entirely ASCII
data.

So .......

my $string = decode("utf8", $octets);

...do whatever string operations in Perl

$wxobject->SetValue($string);

will always work OK just providing something you did in
...do whatever string operations in Perl
didn't strip the utf8 flag off. Not that you'd know if it did.

Clear as mud?

For me too.

For myself, if I were writing code that handled multibyte char sets in
existing Wx releases I would do

my $string = decode("UTF-8", $octets);

...do whatever string operations in Perl

utf8::upgrade($string);
$wxobject->SetValue($string);

If you believe that utf8::upgrade($string) is not necessary, then don't
use it. If you're belief is correct, all will work fine.

The more I think about this and actually test what happens, the more I
lean towards just always expecting string values that get passed to
wxPerl wrappers to be UTF-8. By that I mean don't bother to test if Perl
thinks it is UTF-8 or not, just attempt to convert the data buffer
assuming that it is. I think I'll do it for the next release of Wx.

Then stuff is much simpler and easy for user to understand and fits with
current dogma on how stuff should work.

The bottom line for anyone thinking 'is there something I'll have to
change in my code?' - the answer is no. Unless it breaks. In which case
- complain here.

Cheers

Mark

From: Mark Dootson Hi,

perldoc for the module Encode says:

---------------------------------------------------------------------
CAVEAT: When you run $string = decode("utf8", $octets) , then $string
might not be equal to $octets. Though both contain the same data, the
UTF8 flag for $string is on unless $octets consists entirely of ASCII
data on ASCII machines or EBCDIC on EBCDIC machines.
---------------------------------------------------------------------

None of my machines can be ASCII or EBCDIC by whatever definition this
doc entry uses as my testing on a variety of platforms shows that on
Perl 5.8.8 through Perl 5.16.2 the above is most certainly not true.
I shouldn't be surprised really. What exactly is an ASCII machine?

Anyhow, I find that after

$string = decode("utf8", $octets)

$string always has the utf8 flag set, even if $octets is entirely ASCII
data.

So .......

my $string = decode("utf8", $octets);

...do whatever string operations in Perl

$wxobject->SetValue($string);

will always work OK just providing something you did in
...do whatever string operations in Perl
didn't strip the utf8 flag off. Not that you'd know if it did.

Clear as mud?

For me too.

For myself, if I were writing code that handled multibyte char sets in
existing Wx releases I would do

my $string = decode("UTF-8", $octets);

...do whatever string operations in Perl

utf8::upgrade($string);
$wxobject->SetValue($string);

If you believe that utf8::upgrade($string) is not necessary, then don't
use it. If you're belief is correct, all will work fine.

The more I think about this and actually test what happens, the more I
lean towards just always expecting string values that get passed to
wxPerl wrappers to be UTF-8. By that I mean don't bother to test if Perl
thinks it is UTF-8 or not, just attempt to convert the data buffer
assuming that it is. I think I'll do it for the next release of Wx.

Then stuff is much simpler and easy for user to understand and fits with
current dogma on how stuff should work.

The bottom line for anyone thinking 'is there something I'll have to
change in my code?' - the answer is no. Unless it breaks. In which case
- complain here.

Cheers

Mark

From: Octavian Rasnita Yep, good to know.

It would be nice if WxPerl would announce somehow that a font doesn't have
the necessary glifs (maybe with a warning).

And btw, if the constant wxVSCROLL is not necessary, (do I understood
correctly that it is not necessary under Linux either?) and if it creates
problems under Windows, it would be helpful if it would give an error when
it is used, or it would be helpful if it would be skipped at least under
platforms that have a problem with it.

--Octavian

----- Original Message -----
From: "Mark Dootson" <mark.dootson@znix.com>
To: <wxperl-users@perl.org>
Sent: Wednesday, May 01, 2013 8:18 PM
Subject: Re: Can we print UTF-8 chars in Wx::TextCtrl fields?


> Hi,
>
> Just a clarification.
>
> Setting the font so that you get glyphs displayed properly is only an
> issue on Windows XP.
>
> More recent versions of Windows have default GUI fonts that have a much
> wider range of glyphs available so this isn't an issue.
>
> Unless, strangely, you force 'Verdana' which seems to be the font
> Microsoft forgot.
>
> Regards
>
> Mark
>
>
> On 01/05/2013 17:35, Mark Dootson wrote:
>> Hi,
>>
>> On 01/05/2013 15:21, Octavian Rasnita wrote:
>>> Thank you Mark! Now the chars are displayed fine.
>>>
>>> Too bad that I need to use Windows...
>>>
>>> BTW, what happends if this program runs under Linux or Mac?
>>> Will WxPerl use another sans-serif font available under these platforms?
>>
>> Yes - a similar font chosen on the basis of wxFONTFAMILY_SWISS,
>> wxFONTSTYLE_NORMAL, wxFONTWEIGHT_NORMAL.
>>
>> For both Linux and MacOSX (modern editions I've just tested at least) if
>> the font does not contain a glyph to represent a character, the font
>> engine will check other fonts until it finds one that does.
>>
>> Regards
>>
>> Mark
>>
>

From: Mark Dootson Hi,

Just a clarification.

Setting the font so that you get glyphs displayed properly is only an
issue on Windows XP.

More recent versions of Windows have default GUI fonts that have a much
wider range of glyphs available so this isn't an issue.

Unless, strangely, you force 'Verdana' which seems to be the font
Microsoft forgot.

Regards

Mark


On 01/05/2013 17:35, Mark Dootson wrote:
> Hi,
>
> On 01/05/2013 15:21, Octavian Rasnita wrote:
>> Thank you Mark! Now the chars are displayed fine.
>>
>> Too bad that I need to use Windows...
>>
>> BTW, what happends if this program runs under Linux or Mac?
>> Will WxPerl use another sans-serif font available under these platforms?
>
> Yes - a similar font chosen on the basis of wxFONTFAMILY_SWISS,
> wxFONTSTYLE_NORMAL, wxFONTWEIGHT_NORMAL.
>
> For both Linux and MacOSX (modern editions I've just tested at least) if
> the font does not contain a glyph to represent a character, the font
> engine will check other fonts until it finds one that does.
>
> Regards
>
> Mark
>

From: Mark Dootson Hi,

On 01/05/2013 15:21, Octavian Rasnita wrote:
> Thank you Mark! Now the chars are displayed fine.
>
> Too bad that I need to use Windows...
>
> BTW, what happends if this program runs under Linux or Mac?
> Will WxPerl use another sans-serif font available under these platforms?

Yes - a similar font chosen on the basis of wxFONTFAMILY_SWISS,
wxFONTSTYLE_NORMAL, wxFONTWEIGHT_NORMAL.

For both Linux and MacOSX (modern editions I've just tested at least)
if the font does not contain a glyph to represent a character, the font
engine will check other fonts until it finds one that does.

Regards

Mark

From: Mark Dootson Hi,

On 01/05/2013 16:49, steveco.1959@gmail.com wrote:

> Well all this just serves to deepen my confusion.
>
> 1) What is the difference between:
>
> $line = decode( 'UTF-8', $orig );
>
> and
>
> $line = decode( 'utf8', $orig );

Always use
decode( 'UTF-8', $orig );

'UTF-8' means what it says.

In my opinion, 'utf8' means "something really quite like utf8 in all but
a few respects but which isn't UTF-8 and is a left over from the dog's
breakfast of Perl Unicode string handling, encoding and source code
handling that took a decade to fix."

Perl's documents currently refer to UTF-8 as 'strict UTF-8'. There's no
sanity to it. Why the docs don't just say "'utf8' is really a left over
from an era of big mistakes", I don't know.

> Why did the latter not work for Octavian?

It isn't the difference between 'utf8' and 'UTF-8' that caused
Octavian's code to fail.

>
> 2) Mark, your earlier logic seemed clear and unassailable, yet now you seem
> to change your mind.

I got worn down. It is, after all a community project. The logic seemed
clear and unassailable to me too. When faced with an argument that
simply ignores everything you say you are left with the option of
repeating yourself for ever, ignoring the opposite argument, or giving
up and agreeing. Life is short so I gave up and agreed. I always try to
take the approach that even if the other fellow is wrong in principle,
what exactly would be the downside to agreeing. It leaves you with the
time and energy available to go on repeating yourself forever on the
important stuff.

It won't break much I don't think.

> I use: $line = decode( 'utf8', $orig );
>
> and I never have a problem, but according to this logic that is luck.
>
> I accept this and I am happy to use utf8::upgrade($string);
>
> I think we should assume that in the general case there will always be some
> Perl processing before wxWidgets sees the string.
>
> The general case is:
>
> 1 - Retrieve data from file or database (this maybe automatically decoded or
> not, depending on the database and the driver);
> 2 - Do something to it (thus may be a null operation);
> 3 - Pass to wxWidgets to display to user.
>
> To conserve string lengths and string processing (eg a simple alphabetical
> sort in utf8). If there is to be decoding, it must take place at between 1
> and 2 above.
>
> When you say:
>
>> So, my thinking is that I'll change it for builds against wxWidgets
>> 2.9.x and above
>
> What does "it" mean? That you will include utf8::upgrade($string) in the
> interface?

No, the code will just assume that the string passed is valid UTF-8 and
attempt to convert it to a wxString accordingly. It will never call the
libc option.

>
> I can't see any harm in this. Just setting a character bit to 1 before an
> operation and again later at worst just seems redundant.
>
> But if we have the position where decode is called twice, this will create
> problems for me. A doubly decoded value gets corrupted and becomes a
> diamond with a question mark in it, or some such value.

Hope above assures you this won't happen. (We won't be double decoding.)

Cheers

Mark

From: steveco.1959 Hi Guys,

> Well, this morning I'm inclined to agree that this ought to be the case.
> At least for:

> $orig = readline($datafile);
> $line = decode( 'UTF-8', $orig );
> $w = Wx::StaticText->new( ... );
> $w->SetLabel($line);

> On the other hand I'm reluctant to introduce something that I'm certain
> will break someone's code somewhere ( which is the entire basis for my
> objection to making a change. )

> So, my thinking is that I'll change it for builds against wxWidgets
> 2.9.x and above and announce on this list and in docs that strings
> passed for wxString must be valid UTF-8.

Well all this just serves to deepen my confusion.

1) What is the difference between:

$line = decode( 'UTF-8', $orig );

and

$line = decode( 'utf8', $orig );

I use the latter and Octavian has used the former and they both seem to
work.

Why did the latter not work for Octavian?

Which is correct?

2) Mark, your earlier logic seemed clear and unassailable, yet now you seem
to change your mind.

You said:

> So basically, if the scalar is marked as 'utf8' then it gets converted
> into a wxString as such. If not, you're at the mercy of libc and local
> system settings. It may work. It may not.

> Solution - your conversion of external data should be

> my $string = decode($encoding, $binary);
> utf8::upgrade($string);

> This should be platform independent and work - always. Perl's string
> functions should all work OK on $string.

I use: $line = decode( 'utf8', $orig );

and I never have a problem, but according to this logic that is luck.

I accept this and I am happy to use utf8::upgrade($string);

I think we should assume that in the general case there will always be some
Perl processing before wxWidgets sees the string.

The general case is:

1 - Retrieve data from file or database (this maybe automatically decoded or
not, depending on the database and the driver);
2 - Do something to it (thus may be a null operation);
3 - Pass to wxWidgets to display to user.

To conserve string lengths and string processing (eg a simple alphabetical
sort in utf8). If there is to be decoding, it must take place at between 1
and 2 above.

When you say:

> So, my thinking is that I'll change it for builds against wxWidgets
> 2.9.x and above

What does "it" mean? That you will include utf8::upgrade($string) in the
interface?

I can't see any harm in this. Just setting a character bit to 1 before an
operation and again later at worst just seems redundant.

But if we have the position where decode is called twice, this will create
problems for me. A doubly decoded value gets corrupted and becomes a
diamond with a question mark in it, or some such value.

Regards

Steve.

From: Mark Dootson Octavian,

I loaded up a Windows XP system, installed ActivePerl 5.14.5, and found
the issue.

The 'Arial' font on your system is not the same as the 'Arial' font on
my Windows Vista machine. Whether this is because I have a version of MS
Office installed which comes with enhanced fonts or it is a Vista vs XP
issue I am not sure.

Anyway, a little experimentation found me a font installed by default on
Windows XP systems that does seem to have a more extensive set of glyphs.

If you use the font 'Microsoft Sans Serif' everything should then work.
( assuming you have XP Service Pack 2 installed )

Microsoft Sans Serif - from wikipedia

----------------------------------------------------------------------

Windows XP - SP1 = Version 1.10 of the font includes 1119 glyphs (1209
characters, 26 blocks), supporting Unicode ranges Alphabetic
Presentation forms, Arabic, Arabic Presentation forms A-B, Cyrillic,
General Punctuation, Greek and Coptic, Hebrew, Latin Extended-A, Latin
Extended-B, Latin Extended Additional, Mathematical Operators, Thai.
Supported code pages include 1250-1258, Macintosh US Roman, 874, 864,
862, 708. Font is smoothed at 0-6 points, hinted at 7-14 points, hinted
and smoothed at 15 and above points. OpenType features includes init,
isol, medi, fina, liga for default Arabic script.

Windows XP - SP2 = Version 1.41 of the font includes 2257 glyphs (2301
characters, 28 blocks), which extended Unicode ranges to include
Combining Diacritical Marks, Currency Symbols, Cyrillic Supplement,
Geometric Shapes, Greek Extended, IPA Extensions, Number Forms, Spacing
Modifier Letters. New OpenType scripts include Arabic MAR script.
Additional OpenType features includes rlig for Arabic scripts.

Windows Vista - Version 5.00 - includes 3053 glyphs (2788 characters, 36
blocks), which extended Unicode ranges to include Arabic Supplement,
Combining Diacritical Marks Supplement, Combining Half Marks, Latin
Extended-C, Latin Extended-D, Phonetic Extensions, Phonetic Extensions
Supplement, Specials, Superscripts and Subscripts. New OpenType scripts
include Arabic URD (Urdu), Cyrillic (default), Hebrew (default), Latin
(default, Romanian), Thai (default). Additional OpenType features
includes ccmp, mark, mkmk for Arabic scripts; locl for Arabic URD (Urdu)
script; mark, mkmk for default Cyrillic; dlig, ccmp, mark for default
Hebrew; ccmp, mark, mkmk for Latin scripts; locl for Romanian Latin;
ccmp, mark, mkmk for Thai.

----------------------------------------------------------------------

I have attached an amended test.pl that works for me on a basic Windows
XP install with ActivePerl 5.14.5. The only change absolutely required
from your last test zip is to change the name of the font to
'Microsoft Sans Serif'.

Regards

Mark

On 01/05/2013 06:34, Octavian Rasnita wrote:
> I have tried this but it still displays squares instead of UTF-8 chars.
> (I deleted the line with decode() and I set the font to "Arial".)
>
> The new test file is at:
>
> http://maestrodex.ro/static/test3.zip
>
> --Octavian
>
> ----- Original Message ----- From: "Mark Dootson" <mark.dootson@znix.com>
> To: "Octavian Rasnita" <orasnita@gmail.com>
> Cc: <steveco.1959@gmail.com>; <wxperl-users@perl.org>
> Sent: Tuesday, April 30, 2013 10:43 PM
> Subject: Re: Can we print UTF-8 chars in Wx::TextCtrl fields?
>
>
>> Hi,
>>
>> Comment out the line
>>
>> $text = decode('utf8', $text );
>>
>> you do not need it.
>>
>> Change the font name requested to 'Arial'.
>>
>> Everything should work.
>>
>> I'll try to figure out if there's a way to query a font to check if it
>> has glyphs for particular code points. The old Font Encoding setting
>> seems useless here.
>>
>> Regards
>>
>> Mark
>>
>>
>>
>> On 30/04/2013 20:29, Octavian Rasnita wrote:
>>> Hi Mark,
>>>
>>> I tried your suggestion and I removed the constant wxVSCROLL from the
>>> attributes of Wx::TextCtrl constructor, but the UTF-8 encoded chars
>>> still appear as squares.
>>> I use Windows XP Pro and ActivePerl 5.14.2.
>>> (I need to use ActivePerl and not another distribution because I need to
>>> create a COM server with this application.)
>>>
>>> Then I tried adding:
>>> utf8::upgrade( $text );
>>> then
>>> $textfield->AppendText( $text );
>>>
>>> But no difference. Those chars still appear as squares.
>>>
>>> Then I also added:
>>>
>>> use Encode;
>>> $text = decode('utf8', $text );
>>> utf8::upgrade( $text );
>>>
>>> But this time it gave the following error:
>>>
>>> Cannot decode string with wide characters at D:/usr/lib/Encode.pm
>>> line 176.
>>>
>>> The scalar $text is obtain from an SQLite database and I connect to it
>>> using:
>>>
>>> my $dbh = DBI->connect("dbi:SQLite:test.db");
>>> $dbh->do("PRAGMA cache_size = 80000");
>>> $dbh->do("PRAGMA synchronous = OFF");
>>> $dbh->{sqlite_unicode} = 1;
>>>
>>> The new cod with the SQLite db is at:
>>>
>>> http://maestrodex.ro/static/test2.zip
>>>
>>> I selected the record from this DB in command line and I've seen that
>>> the special char "ț" appears as 2 chars, so I think the char is added
>>> well in DB.
>>>
>>> --Octavian
>>>
>>> ----- Original Message ----- From: "Mark Dootson"
>>> <mark.dootson@znix.com>
>>> To: <steveco.1959@gmail.com>; <wxperl-users@perl.org>
>>> Sent: Monday, April 29, 2013 4:32 PM
>>> Subject: Re: Can we print UTF-8 chars in Wx::TextCtrl fields?
>>>
>>>
>>>> Hi,
>>>>
>>>> A Perl scalar has a character buffer to store character or byte data.
>>>> This data can be interpreted and stored by Perl in one of two formats:
>>>>
>>>> 1. Perl's internal data format
>>>> 2. A number octets (bytes) representing a UTF-8 encoded string.
>>>>
>>>> Internally it is just a memory buffer. Each scalar has a utf8 flag.
>>>> This tells Perl internally how to interpret its data buffer. Either as
>>>> Perl's internal data format or as UTF-8 encoded text. If the utf8 flag
>>>> is on, Perl regards the buffer as UTF-8 encoded text. If the utf8 flag
>>>> is off, Perl regards the buffer as containing data in Perl's internal
>>>> format.
>>>>
>>>> So, say I load some binary data that I know is text encoded using
>>>> 'ISO-8859-1'.
>>>>
>>>> Then I would do:
>>>>
>>>> my $string = decode('ISO-8859-1', $binary);
>>>>
>>>> This gets $string which contains data in Perl's internal format. The
>>>> utf8 flag for the scalar '$string' is off As you have noted below, I
>>>> can't pass '$binary' to any of Perl's string functions. The results
>>>> will be unpredictable and mostly bad.
>>>>
>>>> The evil starts due to some special features when we use decode to
>>>> convert a UTF-8 encoded string.
>>>>
>>>> my $string = decode('utf8', $binary);
>>>>
>>>> If $binary can be converted to $string using single byte characters,
>>>> then $string will be in Perl's internal data format and marked as
>>>> such. (utf8 flag off). If $binary contains multiple byte characters
>>>> the $string will contain a series of bytes representing a UTF-8
>>>> encoded string and the scalar '$string' will have the utf8 flag on.
>>>>
>>>> Within Perl it should not matter whether the scalar is marked UTF-8 or
>>>> not - so long as the utf8 flag correctly reflects what's in the
>>>> scalar's data buffer.
>>>>
>>>> The problem comes when we come to pass the data to the wxWidgets
>>>> library.
>>>>
>>>> The source macro that does this is:
>>>>
>>>> #define WXSTRING_INPUT( var, type, arg ) \
>>>> var = ( SvUTF8( arg ) ) ? \
>>>> wxString( SvPVutf8_nolen( arg ), wxConvUTF8 ) \
>>>> : wxString( SvPV_nolen( arg ), wxConvLibc );
>>>>
>>>>
>>>> So basically, if the scalar is marked as 'utf8' then it gets converted
>>>> into a wxString as such. If not, you're at the mercy of libc and local
>>>> system settings. It may work. It may not.
>>>>
>>>>
>>>> Solution - your conversion of external data should be
>>>>
>>>> my $string = decode($encoding, $binary);
>>>> utf8::upgrade($string);
>>>>
>>>> This should be platform independent and work - always. Perl's string
>>>> functions should all work OK on $string.
>>>>
>>>> The key points
>>>>
>>>> my $string = decode('utf8', $binary);
>>>>
>>>> It depends on the content of $binary whether $string has the utf8 flag
>>>> set.
>>>>
>>>> my $string = decode('utf8', $binary);
>>>> utf8::upgrade( $string );
>>>>
>>>> $string always has utf8 flag set. You could just do
>>>> utf8::upgrade($binary) but that would be a special case for when
>>>> $binary actually contains UTF-8 bytes. The two step method applies to
>>>> any encoding.
>>>>
>>>> Perl can't know that a scalar contains UTF-8 encoded text unless you
>>>> tell it.
>>>>
>>>> The statement:
>>>>
>>>> 'use utf8;' Is not needed anywhere here of course as it indicates that
>>>> the source code is encoded in UTF-8. Nothing more. Functions
>>>> utf8::upgrade etc. are always available.
>>>>
>>>> If you have a list of scalars containing strings as in
>>>>
>>>> @combo_options
>>>>
>>>> then the same applies - to each individual scalar / string in the list.
>>>>
>>>>
>>>> Hope it helps.
>>>>
>>>>
>>>> Mark
>>>>
>>>>
>>>> On 29/04/2013 12:29, steveco.1959@gmail.com wrote:
>>>>
>>>>> Hi Mark, I'm a relative new comer to utf8 so please take everything I
>>>>> say
>>>>> with a pinch of salt but your answer looks a bit qualified: if
>>>>> scalar, if
>>>>> marked. That implies if I want a Perl list (say @combo_options) for a
>>>>> Wx::ComboBox, then that won't work? Is that how it is?
>>>>>
>>>>> And I don't know what marked means.
>>>>>
>>>>> The real problem for me is that this feels like the wrong place to
>>>>> decode.
>>>>> There are lots of things I might want to do with a string before I
>>>>> display
>>>>> it. I might want to sort it, or trim white space, or substitute a
>>>>> place-marker with a value. And for these I need it to be decoded
>>>>> before I
>>>>> process it. If I have a very simple app with no string processing,
>>>>> then
>>>>> this approach would be great but not otherwise.
>>>>>
>>>>> I did have a lot of issues with utf8 at the beginning sometimes I had
>>>>> display issues with a utf8 string and sometimes not. There seemed to
>>>>> be no
>>>>> particular rhyme or reason to it. And as you say, it works
>>>>> differently on
>>>>> Windows and Linux. Finally, everything is very sensitive to small
>>>>> errors,
>>>>> like having a non-existent style code.
>>>>>
>>>>> So I use a policy which is that when I read a value into my program
>>>>> from a
>>>>> database or a file, I always decode immediately. That way I know
>>>>> that all
>>>>> my variables are decoded and processable.
>>>>>
>>>>> Then I encode before I write back to the file or db.
>>>>>
>>>>> If I have an issue now, it is always where I have not done a:
>>>>>
>>>>> $var = decode("utf8",$row->{ATT_BOOKING_COMMENT_TXT}) ;
>>>>>
>>>>> Anyhow let me know what you think.
>>>>>
>>>>> Regards
>>>>>
>>>>> Steve
>>>>>
>>>>
>>>
>>
>

From: Mark Dootson Hi,

On 01/05/2013 07:34, Johan Vromans wrote:
> Mark Dootson <mark.dootson@znix.com> writes:
>
>> On 30/04/2013 19:19, Johan Vromans wrote:
>>
>>> We may assume that the Perl string is in Perl's internal encoding.
>>
>> No we may not.
>
> In that case you'll run into all kinds of encoding problems anyway.

If you attempt any string operations, indeed you will.

>
> See e.g. perlunitut.
>
>> I kind of like the existing solution which doesn't break existing code
>> all over the place and simply requires the coder to be specific about
>> the format of the data they are sending.
>
> My main concern is: If I have correctly decoded string data, will it
> work when passed to wxWdigets. For example:
>
> $orig = readline($datafile);
> $line = decode( 'utf8', $orig );
> $w = Wx::StaticText->new( ... );
> $w->SetLabel($line);
>
> If an explicit utf8::upgrade were required in this case, my feelings
> tell me something is wrong.

Well, this morning I'm inclined to agree that this ought to be the case.
At least for:

$orig = readline($datafile);
$line = decode( 'UTF-8', $orig );
$w = Wx::StaticText->new( ... );
$w->SetLabel($line);

On the other hand I'm reluctant to introduce something that I'm certain
will break someone's code somewhere ( which is the entire basis for my
objection to making a change. )

So, my thinking is that I'll change it for builds against wxWidgets
2.9.x and above and announce on this list and in docs that strings
passed for wxString must be valid UTF-8.

I'll probably just use SvPVutf8_nolen on everything if this tests ok.
(For info of casual reader - the force part in SvPVutf8_force refers to
changing the SV to have a pv ( string ) representation only - nothing to
do with utf8. You would use it if you expected the C / C++ code might
change the value directly so it would force Perl to re-evaluate the next
time you used the SV in a number context. In our code the pv value will
never be changed directly.)

For anyone interested, the relevant code is in cpp/helpers.h wrapped in
a three part if/else

#if defined(wxUSE_UNICODE_UTF8) && wxUSE_UNICODE_UTF8

// Mac OSX and Linux

#elif wxUSE_UNICODE

// Windows

#else

// 2.8 ANSI build ( ignore it )

#endif

Macros

WXCHAR_INPUT, WXCHAR_OUTPUT, WXSTRING_INPUT, WXSTRING_OUTPUT

are used via the typemap.

Functions wxPli_wxChar_2_sv and wxPli_wxString_2_sv are also used
throughout the Wx code.

You will note that the return value from a wxString or multibyte char is
always flagged as utf8.


Regards

Mark

From: Johan Vromans Mark Dootson <mark.dootson@znix.com> writes:

> On 30/04/2013 19:19, Johan Vromans wrote:
>
>> We may assume that the Perl string is in Perl's internal encoding.
>
> No we may not.

In that case you'll run into all kinds of encoding problems anyway.

See e.g. perlunitut.

> I kind of like the existing solution which doesn't break existing code
> all over the place and simply requires the coder to be specific about
> the format of the data they are sending.

My main concern is: If I have correctly decoded string data, will it
work when passed to wxWdigets. For example:

$orig = readline($datafile);
$line = decode( 'utf8', $orig );
$w = Wx::StaticText->new( ... );
$w->SetLabel($line);

If an explicit utf8::upgrade were required in this case, my feelings
tell me something is wrong.

-- Johan

From: Mark Dootson Hi,

Comment out the line

$text = decode('utf8', $text );

you do not need it.

Change the font name requested to 'Arial'.

Everything should work.

I'll try to figure out if there's a way to query a font to check if it
has glyphs for particular code points. The old Font Encoding setting
seems useless here.

Regards

Mark



On 30/04/2013 20:29, Octavian Rasnita wrote:
> Hi Mark,
>
> I tried your suggestion and I removed the constant wxVSCROLL from the
> attributes of Wx::TextCtrl constructor, but the UTF-8 encoded chars
> still appear as squares.
> I use Windows XP Pro and ActivePerl 5.14.2.
> (I need to use ActivePerl and not another distribution because I need to
> create a COM server with this application.)
>
> Then I tried adding:
> utf8::upgrade( $text );
> then
> $textfield->AppendText( $text );
>
> But no difference. Those chars still appear as squares.
>
> Then I also added:
>
> use Encode;
> $text = decode('utf8', $text );
> utf8::upgrade( $text );
>
> But this time it gave the following error:
>
> Cannot decode string with wide characters at D:/usr/lib/Encode.pm line 176.
>
> The scalar $text is obtain from an SQLite database and I connect to it
> using:
>
> my $dbh = DBI->connect("dbi:SQLite:test.db");
> $dbh->do("PRAGMA cache_size = 80000");
> $dbh->do("PRAGMA synchronous = OFF");
> $dbh->{sqlite_unicode} = 1;
>
> The new cod with the SQLite db is at:
>
> http://maestrodex.ro/static/test2.zip
>
> I selected the record from this DB in command line and I've seen that
> the special char "ț" appears as 2 chars, so I think the char is added
> well in DB.
>
> --Octavian
>
> ----- Original Message ----- From: "Mark Dootson" <mark.dootson@znix.com>
> To: <steveco.1959@gmail.com>; <wxperl-users@perl.org>
> Sent: Monday, April 29, 2013 4:32 PM
> Subject: Re: Can we print UTF-8 chars in Wx::TextCtrl fields?
>
>
>> Hi,
>>
>> A Perl scalar has a character buffer to store character or byte data.
>> This data can be interpreted and stored by Perl in one of two formats:
>>
>> 1. Perl's internal data format
>> 2. A number octets (bytes) representing a UTF-8 encoded string.
>>
>> Internally it is just a memory buffer. Each scalar has a utf8 flag.
>> This tells Perl internally how to interpret its data buffer. Either as
>> Perl's internal data format or as UTF-8 encoded text. If the utf8 flag
>> is on, Perl regards the buffer as UTF-8 encoded text. If the utf8 flag
>> is off, Perl regards the buffer as containing data in Perl's internal
>> format.
>>
>> So, say I load some binary data that I know is text encoded using
>> 'ISO-8859-1'.
>>
>> Then I would do:
>>
>> my $string = decode('ISO-8859-1', $binary);
>>
>> This gets $string which contains data in Perl's internal format. The
>> utf8 flag for the scalar '$string' is off As you have noted below, I
>> can't pass '$binary' to any of Perl's string functions. The results
>> will be unpredictable and mostly bad.
>>
>> The evil starts due to some special features when we use decode to
>> convert a UTF-8 encoded string.
>>
>> my $string = decode('utf8', $binary);
>>
>> If $binary can be converted to $string using single byte characters,
>> then $string will be in Perl's internal data format and marked as
>> such. (utf8 flag off). If $binary contains multiple byte characters
>> the $string will contain a series of bytes representing a UTF-8
>> encoded string and the scalar '$string' will have the utf8 flag on.
>>
>> Within Perl it should not matter whether the scalar is marked UTF-8 or
>> not - so long as the utf8 flag correctly reflects what's in the
>> scalar's data buffer.
>>
>> The problem comes when we come to pass the data to the wxWidgets library.
>>
>> The source macro that does this is:
>>
>> #define WXSTRING_INPUT( var, type, arg ) \
>> var = ( SvUTF8( arg ) ) ? \
>> wxString( SvPVutf8_nolen( arg ), wxConvUTF8 ) \
>> : wxString( SvPV_nolen( arg ), wxConvLibc );
>>
>>
>> So basically, if the scalar is marked as 'utf8' then it gets converted
>> into a wxString as such. If not, you're at the mercy of libc and local
>> system settings. It may work. It may not.
>>
>>
>> Solution - your conversion of external data should be
>>
>> my $string = decode($encoding, $binary);
>> utf8::upgrade($string);
>>
>> This should be platform independent and work - always. Perl's string
>> functions should all work OK on $string.
>>
>> The key points
>>
>> my $string = decode('utf8', $binary);
>>
>> It depends on the content of $binary whether $string has the utf8 flag
>> set.
>>
>> my $string = decode('utf8', $binary);
>> utf8::upgrade( $string );
>>
>> $string always has utf8 flag set. You could just do
>> utf8::upgrade($binary) but that would be a special case for when
>> $binary actually contains UTF-8 bytes. The two step method applies to
>> any encoding.
>>
>> Perl can't know that a scalar contains UTF-8 encoded text unless you
>> tell it.
>>
>> The statement:
>>
>> 'use utf8;' Is not needed anywhere here of course as it indicates that
>> the source code is encoded in UTF-8. Nothing more. Functions
>> utf8::upgrade etc. are always available.
>>
>> If you have a list of scalars containing strings as in
>>
>> @combo_options
>>
>> then the same applies - to each individual scalar / string in the list.
>>
>>
>> Hope it helps.
>>
>>
>> Mark
>>
>>
>> On 29/04/2013 12:29, steveco.1959@gmail.com wrote:
>>
>>> Hi Mark, I'm a relative new comer to utf8 so please take everything I
>>> say
>>> with a pinch of salt but your answer looks a bit qualified: if
>>> scalar, if
>>> marked. That implies if I want a Perl list (say @combo_options) for a
>>> Wx::ComboBox, then that won't work? Is that how it is?
>>>
>>> And I don't know what marked means.
>>>
>>> The real problem for me is that this feels like the wrong place to
>>> decode.
>>> There are lots of things I might want to do with a string before I
>>> display
>>> it. I might want to sort it, or trim white space, or substitute a
>>> place-marker with a value. And for these I need it to be decoded
>>> before I
>>> process it. If I have a very simple app with no string processing, then
>>> this approach would be great but not otherwise.
>>>
>>> I did have a lot of issues with utf8 at the beginning sometimes I had
>>> display issues with a utf8 string and sometimes not. There seemed to
>>> be no
>>> particular rhyme or reason to it. And as you say, it works
>>> differently on
>>> Windows and Linux. Finally, everything is very sensitive to small
>>> errors,
>>> like having a non-existent style code.
>>>
>>> So I use a policy which is that when I read a value into my program
>>> from a
>>> database or a file, I always decode immediately. That way I know
>>> that all
>>> my variables are decoded and processable.
>>>
>>> Then I encode before I write back to the file or db.
>>>
>>> If I have an issue now, it is always where I have not done a:
>>>
>>> $var = decode("utf8",$row->{ATT_BOOKING_COMMENT_TXT}) ;
>>>
>>> Anyhow let me know what you think.
>>>
>>> Regards
>>>
>>> Steve
>>>
>>
>

From: Mark Dootson

On 30/04/2013 19:19, Johan Vromans wrote:

> We may assume that the Perl string is in Perl's internal encoding.

No we may not.

> AFAIK, when a buffer contains valid UTF-8 (e.g., as result of an earlier
> decode), utf8::upgrade is a no-op.

Not necessarily. See perldoc for decode and UTF-8 strings containing
only single byte characters. Perl has taken this approach for whatever
reason it has taken it. Doesn't concern us. We are only concerned with
marking scalars containing UTF-8 strings with utf8 flag when passing to
WXSTRING_INPUT macro.

>
> As of Perl 5.14, use feature 'unicode_strings' will make sure that all
> strings are, indeed, UTF-8. This takes the burden off the programmer to
> call utf8::upgrade (and knowing when to call it).
>

'strings' perhaps - but not necessarily the content of valid scalars you
may wish to pass to the wxWidgets library wxString functions. In any
case, this is a Perl internals thing which should not be taxing us here
it is irrelevant to the original question.

>> If I read your proposal correctly, you want to demand that all data
>> buffers that may get passed to the wxString conversion function are
>> valid UTF-8?
>
> Not really. They should be in Perl's internal coding, and thus can be
> safely and transparently upgraded to UTF-8.

They will only be valid strings in Perl's internal coding if the user
has previously coded it to be so.

>
> But in the end I think feature 'unicode_strings' will be the best and
> most elegant solution.

I kind of like the existing solution which doesn't break existing code
all over the place and simply requires the coder to be specific about
the format of the data they are sending.

Cheers

Mark