Using Unicode scripts

Re: Using Unicode scripts

"yzzzzz" <yzzzzz@netcour rier.com> writes:
[color=blue]
> Hi,
>
> I am writing my python programs using a Unicode text editor. The files are
> encoded in UTF-8. Python's default encoding seems to be Latin 1 (ISO-8859-1)
> or maybe Windows-1252 (CP1252) which aren't compatible with UTF-8.
>
> For example, if I type print "é", it prints é. If I use a unicode string:
> a=u"é" and if I choose to encode it in UTF-8, I get 4 Latin 1 characters,
> which makes sense if the interpreter thinks I typed in u"é".
>
> How can I solve this problem?
>
> Thank you
>
> PS. I have no problem using Unicode strings in Python, I know how to
> manipulate and convert them, I'm just looking for how to specify the default
> encoding for the scripts I write.[/color]

Use Python 2.3, and read PEP 263.

Thomas

Re: Using Unicode scripts

yzzzzz wrote:[color=blue]
> Hi,[/color]

Hi "yzzzzz",
[color=blue]
> I am writing my python programs using a Unicode text editor. The files are
> encoded in UTF-8. Python's default encoding seems to be Latin 1 (ISO-8859-1)
> or maybe Windows-1252 (CP1252) which aren't compatible with UTF-8.
>
> For example, if I type print "é", it prints é. If I use a unicode string:
> a=u"é" and if I choose to encode it in UTF-8, I get 4 Latin 1 characters,
> which makes sense if the interpreter thinks I typed in u"é".
>
> How can I solve this problem?[/color]

You might want to read the thread on this list/newsgroup I started
yesterday called "Unicode problem"

Is it feasible for you to upgrade to Python 2.3? If so I'd recommend you
do it already. 2.3 is pretty close to release now and it has support for
source files in Unicode format. If your Unicode editor saves the text
file with a BOM (it should) then under Python 2.3 your scripts will work
as expected.
[color=blue]
> Thank you
>
> PS. I have no problem using Unicode strings in Python, I know how to
> manipulate and convert them, I'm just looking for how to specify the default
> encoding for the scripts I write.[/color]

See http://www.python.org/peps/pep-0263.html This is how it is
implemented in Python 2.3.

-- Gerhard

Re: Using Unicode scripts

OK, problem solved!
I got the new Python, it all works. I just had to add the UTF-8 BOM myself
(UltraEdit doesn't do it by default) but that wasn't too difficult to do
(copy and paste a ZWNBSP).

One last question: I'm using windows, so the console's encoding is CP437. If
I try to print a unicode string, the string is converted to CP437 and
printed and that works fine. However if I try to print a normal
(non-unicode) string from a UTF-8 encoded file with BOM, for example print
"é", it sends out the two UTF-8 bytes é which appear as lines in the CP437
charset. But if I print the exact same character in a Latin 1 encoded file,
it comes out as the Latin 1 byte for "é" which shows up as a theta in CP437.
This means that Python doesn't take into account the specified encoding
(Latin 1 or UTF-8) and prints out the raw bytes as they appear in the source
file, regardless of the encoding used. Is this normal? (this isn't really a
problem for me as I am only going to use unicode strings now)