develooper Front page | perl.perl5.porters | Postings from July 2011

normalization filters?

Thread Next
From:
Tom Christiansen
Date:
July 20, 2011 11:02
Subject:
normalization filters?
Message ID:
10447.1311184906@chthon
Is there any reason one couldn't, or even shouldn't, have 
normalization I/O layers available?

    open($fh_in,  "< :encoding(UTF-8) :normalize(NFD)", $input_file)
    open($fh_out, "> :encoding(UTF-8) :normalize(NFC)", $output_file)

Put another way, is there anything wrong with the general advice to NFD 
incoming data and to NFC outgoing data?  

Are there any particular provisos called for?

I know of various "services" that already seem to do this, 
and not too many people get *too* upset by it.

On another matter, even though I sometimes think of diddling $/ for it, 
I still wonder whether a \R-sensitive I/O layer is the best way to handle
input with unknown linebreaks.  But unless we convert those all to \n,
chomp() is going to be messy.

--tom

Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About