I have written a program in Python, using version 3.3.3. This program is just a simple weight loss tracking program that uses a pictograph for visual representation of whatever number the user has input. The characters used in the pictograph are all unicode (□▪▫∙). I have used the open() function as follows:  open('Pictograph.txt', mode = 'a', encoding = 'utf8')   Prior to using open(), I was using tkinter's askopenfile() which would allow me to let the user choose which file they'd want to save to, but would not allow me to set the encoding to UTF-8. I finally stumbled upon the encoding option of open(), which works well, but only in the Python shell. (I was having problems writing those characters to a standard .txt file before this, encoding = 'utf8' made the difference for that problem, but did not help anything in the command line.)  Now, I have a 'help' option that enables the user to see a legend of the pictograph, and in so display cp1252 text, as well as the four Unicode characters listed above.    I just captured this image, but prior to that I felt it had to do with text encoding in the command line. Is there any way to allow my program to set the command line to where it can read those characters and not crash?  This is what is supposed to print (as seen in the shell):    Any insight would be greatly appreciated.    Okay, so this is a duplicate. That's it? I tried looking for an answer for over an hour. I didn't find anything that would solve my problem. I would like not to have to spend another hour trying to find the page I am supposedly duplicating.
4534
When I try to print a Unicode string in a Windows console, I get an error . UnicodeEncodeError: 'charmap' codec can't encode character .... I assume this is because the Windows console does not accept Unicode-only characters. What's the best way around this? Is there any way I can make Python automatically print a ? instead of failing in this situation? Edit:  I'm using Python 2.5.  Note: @LasseV.Karlsen answer with the checkmark is sort of outdated (from 2008). Please use the solutions/answers/suggestions below with care!!  is more relevant as of today (6 Jan 2016).
28362
I want to have my sections in independent .txt files in the folder of my assignment. So I use \input{text.txt}. The problem is, I write in Danish and we have the 'æ' 'ø' 'å' characters that aren't normally recognized by LaTeX. I fixed it within the master .tex file, so when I write the letters, it shows up nicely in my document. It does not however show up nicely, if I put one of the characters in the file name or within the .txt files. Now, I don't mind that I have to avoid them in file names, but I really need them to be recognized within the file.  \documentclass[10pt,a4paper]{article} \usepackage[danish]{babel} \renewcommand{\danishhyphenmins}{22} \usepackage{lmodern} \usepackage[T1]{fontenc} \usepackage[utf8x]{inputenc} \usepackage[danish=quotes]{csquotes}  \begin{document} æ ø å \input{text.txt} \end{document}   I can't upload the .txt file, but if you copy the three letters, put it in a document called text.txt file and put it in the folder, you'll see what I mean.  If I delete \input{text.txt} from the document, or avoid using those three letters in the input file, everything works perfectly and it writes 'æ ø å' in the output file.  I asked this question on another forum, but they weren't much help. They talked about it being the wrong encoding and told me, that if I saved it as .tex files there would be a dialogue that would enable me to change the encoding of the input document so it would be compatible, however there is no such thing in the 'Save as' dialogues I get from TeXworks. I also have it set to utf8 in my preferences in TeXworks, however that doesn't seem to influence much of anything.
80408
What are the differences between UTF-8, UTF-16, and UTF-32?  I understand that they will all store Unicode, and that each uses a different number of bytes to represent a character. Is there an advantage to choosing one over the other?
14911
I am only using the 128 character set defined in the original ANSI standard.  But as a whole how are the files implmeneted differently.    I am not concerned with the display, i.e. if a tab is displayed with 6 or 8 characters but the actual internal representation in memory  One difference I've heard is the use of \r\n (Windows) vs. \n for line termination (Linux).
56580
On Unix, I can either use \r (carriage return) or \b (backspace) to overwrite the current line (print over text already visible) in the shell.  Can I achieve the same effect in a Windows command line from a Python script?  I tried the curses module but it doesn't seem to be available on Windows.
25058
What is the fastest, easiest tool or method to convert text files between character sets?  Specifically, I need to convert from UTF-8 to ISO-8859-15 and vice versa.  Everything goes: one-liners in your favorite scripting language, command-line tools or other utilities for OS, web sites, etc.  Best solutions so far:  On Linux/UNIX/OS X/cygwin:   Gnu  suggested by  is best used as a filter. It seems to be universally available. Example:  $ iconv -f UTF-8 -t ISO-8859-15 in.txt &gt; out.txt   As pointed out by , there is an . Gnu  () suggested by  will convert one or several files in-place. Example:  $ recode UTF8..ISO-8859-15 in.txt   This one uses shorter aliases:  $ recode utf8..l9 in.txt   Recode also supports surfaces which can be used to convert between different line ending types and encodings:  Convert newlines from LF (Unix) to CR-LF (DOS):  $ recode ../CR-LF in.txt   Base64 encode file:  $ recode ../Base64 in.txt   You can also combine them.  Convert a Base64 encoded UTF8 file with Unix line endings to Base64 encoded Latin 1 file with Dos line endings:  $ recode utf8/Base64..l1/CR-LF/Base64 file.txt    On Windows with  ():    PS C:\&gt; gc -en utf8 in.txt | Out-File -en ascii out.txt  (No ISO-8859-15 support though; it says that supported charsets are unicode, utf7, utf8, utf32, ascii, bigendianunicode, default, and oem.)   Edit  Do you mean iso-8859-1 support? Using "String" does this e.g. for vice versa  gc -en string in.txt | Out-File -en utf8 out.txt   Note: The possible enumeration values are "Unknown, String, Unicode, Byte, BigEndianUnicode, UTF8, UTF7, Ascii".   CsCvt -  is another great command line based conversion tool for Windows.
20826
In a text file, there is a string "I don't like this".  However, when I read it into a string, it becomes "I don\xe2\x80\x98t like this". I understand that \u2018 is the unicode representation of "'". I use   f1 = open (file1, "r") text = f1.read()   command to do the reading.  Now, is it possible to read the string in such a way that when it is read into the string, it is "I don't like this", instead of "I don\xe2\x80\x98t like this like this"?  Second edit: I have seen some people use mapping to solve this problem, but really, is there no built-in conversion that does this kind of ANSI to unicode ( and vice versa) conversion?
60896
If I execute the following simple script:  #!/bin/bash printf "%-20s %s\n" "Früchte und Gemüse"   "foo" printf "%-20s %s\n" "Milchprodukte"        "bar" printf "%-20s %s\n" "12345678901234567890" "baz"   It prints:  Früchte und Gemüse foo Milchprodukte        bar 12345678901234567890 baz   that is, text with umlauts (such as ü) is "shrunk" by one character per umlaut.  Certainly, I have some wrong setting somewhere, but I am not able to figure out which one that could be.  This occurs if the file's encoding is UTF-8.  If I change its encoding to latin-1, the alignment is correct, but the umlauts are rendered wrong:  Fr�chte und Gem�se   foo Milchprodukte        bar 12345678901234567890 baz
91508
I am completely new to Linux and Ubuntu.   I have written python code in Windows and want to run it in Ubuntu. It uses a text file for input. The 'path-to-file' in windows looks as follows   c:user\documents\python\file.txt   How is the location written for Ubuntu?
78059
Example: In a Terminal command xdpyinfo | grep resolution (that I have taken from an answer), does the | character mean that the xdpyinfo output shall be sent to the grep program input for printing a line containing the resolution string? Where should I seek for such an information to avoid filling this forum by too simple questions? Thanks.
34476
With reference to the following thread:   What is the best way to programatically determine the correct charset encoding of an inputstream/file ?  I have tried using the following:  File in =  new File(args[0]); InputStreamReader r = new InputStreamReader(new FileInputStream(in)); System.out.println(r.getEncoding());   But on a file which I know to be encoded with ISO8859_1 the above code yields ASCII, which is not correct, and does not allow me to correctly render the content of the file back to the console.
50662
I understand that e.g. catfish and gnome-search-utils both can search inside file contents that are UTF-8 encoded. To be able to search for words or numbers within text files one would have to convert them via iconv into UTF-8 first.   If the file is known, text editors like gedit or mousepad have no trouble with UTF-16.   Why is there no search tool (GUI or command-line) with any of the Linux distributions that can handle UTF-16 encoded txt files?   I'm on Xubuntu.
103740
I'm trying to visualize some data with TikZ datavisualization library. Given foo.tex  \documentclass{article} \usepackage{tikz} \usetikzlibrary{datavisualization}  \begin{document} \begin{tikzpicture}   \datavisualization [ scientific axes, visualize as scatter ] data [ read from file=foo.dat ]; \end{tikzpicture} \end{document}   and foo.dat:  x, y 1, 2 3, 4 5, 6   It fails with the following error (repeated multiple times):  ! Package PGF Math Error: Sorry, an internal routine of the floating point unit got an ill-formatted floating point number `'. The unreadable part was near ''..  See the PGF Math package documentation for explanation. Type  H &lt;return&gt;  for immediate help.  ...                                                l.7 ... scatter ] data [ read from file=foo.dat ];   If I'd rather include data from foo.dat inline the document compiles normally.  What am I missing and how can I fix the problem?  I'm running TeXLive 2014 on Windows 7. All files use Unix line ending.  EDIT: I've removed all commas in my data file and explicitly set separator to \space, now it works.  One more quertion: TikZ documentation says:     /pgf/data/separator=&lt;character&gt; (no default, initially ,)   What does this mean? What is the difference between initial value and default value?
37484
I am trying to print a large quantity (several megabytes) of UTF-8 encoded text which consists of Chinese and Latin characters (and maybe a sprinkling of others). I would like to print it in several columns per page, in a very small, condensed font, preferably with control over line spacing. I'd quite like inter-column lines, but I can live without them. The aim is to print to PDF for transfer, as well as to paper.  I have tried enscript and a2ps, but neither of these support Unicode.  I have also tried paps, but this produces bitmapped outputs which cannot be PDF'd effectively, and also look terrible.  Is there a modern way to print UTF-8 text like this without resorting to something like constructing it manually in Python?
77611
Good day. Using only Linux OS tools, how would you convert the following text from this: cat latest.csv apsga,alp1acscf0001vm001oam001p1n004v001,192.7.129.7,UVP apsga,alp1acscf0001vm001oam001p1n004v002,192.7.129.8,UVP apsga,alp1acscf0001vm002oam001p1n004v001,192.7.129.9,UVP apsga,alp1acscf0001vm003oam001p1n004v001,192.7.129.10,UVP apsga,alp1acscf0001vm004cif001p1n004v001,192.7.129.11,UVP apsga,alp1acscf0001vm004cif001p1n004v002,192.7.129.12,UVP apsga,alp1acscf0001vm005cif001p1n004v001,192.7.129.13,UVP  I managed to get the following result: cat latest.csv`|awk -F',' '{print $2, $3}'|awk '{ $1 = substr($1,1,21); print $1&quot;,&quot;$2}' alp1acscf0001vm001oam,192.7.129.7 alp1acscf0001vm001oam,192.7.129.8 alp1acscf0001vm002oam,192.7.129.9 alp1acscf0001vm003oam,192.7.129.10 alp1acscf0001vm004cif,192.7.129.11 alp1acscf0001vm004cif,192.7.129.12 alp1acscf0001vm005cif,192.7.129.13  I would like the end result to look like this: ALP1A/alp1acscf0001v,UUID,192.7.129.7,SSH2,alp1acscf0001vm001oam ALP1A/alp1acscf0001v,UUID,192.7.129.8,SSH2,alp1acscf0001vm001oam ALP1A/alp1acscf0001v,UUID,192.7.129.9,SSH2,alp1acscf0001vm002oam ALP1A/alp1acscf0001v,UUID,192.7.129.10,SSH2,alp1acscf0001vm003oam ALP1A/alp1acscf0001v,UUID,192.7.129.11,SSH2,alp1acscf0001vm004cif ALP1A/alp1acscf0001v,UUID,192.7.129.12,SSH2,alp1acscf0001vm004cif ALP1A/alp1acscf0001v,UUID,192.7.129.13,SSH2,alp1acscf0001vm005cif  Thanks in advance, Bjoern
7145
Is it possible/legal to somehow encode CR/LF characters into a CSV file?  (as part of a CSV standard?)  If so how should i encode CR/LF?
26110
I have a problem with the endfloat package and unicode characters.  When I compile the following, everything works as expected:  \documentclass{article} \usepackage[utf8]{inputenc} \usepackage[T1]{fontenc} \usepackage{graphicx}  \begin{document}  \begin{figure} \caption{Ö} \end{figure}  \end{document}   However, when I also load the endfloat package (\usepackage{endfloat}), I get the following error message:  ! Package inputenc Error: Unicode char \u8:ÖÖ not set up for use with LaTeX.   This could be a bug, but my suspicion is that I'm making a mistake somewhere.  I'm hardly the first person to use special characters with the endfloat package.  However, the above example is so simple, I don't see where I could have made a mistake.  Any advice is appreciated.  Edit: The file is of course encoded using utf-8.