This is one of those things that seems like it should be laughably easy but I'm stuck...  I need to open a CSV file that is stored in SHIFT_JIS encoding and decode it to Unicode and also encode in UTF-8. It sounds like this is pretty straightforward in ruby 1.9 but I'm not feeling particularly adventurous on my production Rails app, so I need to stick with 1.8.7.   Any help would be appreciated.
123364
I have a Ruby script that generates a UTF8 CSV file remotely in a Linux machine and then transfers the file to a Windows machine thru SFTP.   I then need to open this file with Excel, but Excel doesn't get UTF8, so I always need to open the file in a text editor that has the capability to convert UTF8 to ANSI.  I would love to do this programmatically using Ruby and avoid the manual conversion step. What's the easiest way to do it?  PS: I tried using iconv but had no success.
26314
Please, do you know if there's any possibility how to use the listings package together with XeLaTeX? I have to use XelaTeX and therefore I have UTF-8 encoded files. This is not compatible with listings which allows one-byte encoding. (I know there is listingsutf8 package, but it converts files from UTF-8 which I don't need.)  The listings package works somehow, bu I only can use English letters. I've fond some solution like a list of special characters (  ) but it seems it works only for some letters. It there anything that would support UTF-8 in listing 'naturally'?
10511
I have two issues with a csv file that occur when I try and edit it in Excel.   I have an ISBN column that gets reformatted to scientific format every time I save the file The encoding for the file is changed from UTF-8 to ANSI when I open the file.   It is really, really frustrating and a complete waste of time.  Is there any way that I can force Excel to leave the format of the file alone?
50662
I understand that e.g. catfish and gnome-search-utils both can search inside file contents that are UTF-8 encoded. To be able to search for words or numbers within text files one would have to convert them via iconv into UTF-8 first.   If the file is known, text editors like gedit or mousepad have no trouble with UTF-16.   Why is there no search tool (GUI or command-line) with any of the Linux distributions that can handle UTF-16 encoded txt files?   I'm on Xubuntu.
78486
For example, if I have a unicode string, I can encode it as an ASCII string like so:  &gt;&gt;&gt; u'\u003cfoo/\u003e'.encode('ascii') '&lt;foo/&gt;'   However, I have e.g. this ASCII string:  '\u003foo\u003e'   ... that I want to turn into the same ASCII string as in my first example above:  '&lt;foo/&gt;'
68712
I am working on a text written in Italian and Chinese and I need to extract only Chinese characters using AWK. How can I do this?  I tried:  [The range of Chinese Unicode chars is 4E00 thru 9FFF (344 270 200 thru 351 277 277) so the test should be &gt;"\343" and &lt;"\352" (to avoid picking up any 4 char UTF-8 codes)]:  { f=0; for ( i=1; i&lt;=length; i++) if(substr($0, i, 1)&gt;"\343" &amp;&amp;substr($0, i, 1)&lt;"\352") f = 1   print $f }   But there is an error or more errors. I can't find it / them
95793
Important note: I'm using Cygwin    I retrieve JSON from a file that I alter before sending it to a server using curl. By now everything is ok except one thing, when the server receive the JSON, every special characters (with accents, etc) aren't well encoded. I suppose this was due to the fact that my JSON wasn't encoded in UTF-8 before being sent but I don't manage to convert it.  Here is the code:  sed -i 's/\r//g' $file fileContent=`cat $file`  result=$(jq -c -j ".docs[$docIndex] + { \"_rev\": \"$rev\" }"&lt;&lt;&lt;"$fileContent")            result="{\"docs\":[$result]}"  result=$result | sed 's/\r//g' result=$result | iconv -t "UTF-8" s=$(curl -H "Content-Type: application/json; charset=UTF-8" -H "Cache-Control: no-cache" -d "$result" $2/$3/_bulk_docs --silent)   My bash script and my JSON files are both encoded in UTF-8. My LANG variable seems to be UTF-8. I checked with this: [[ $LANG =~ UTF-8$ ]] &amp;&amp; echo "Uses UTF-8 encoding.."  Any idea?    Update  Here is the full script:  #!/bin/bash  # $1 = directory containing JSON files # $2 = server url (+ port) # $3 = database name  #Loop on every file of the given directory for file in "$1"/*; do      #Try to insert every document of a specific JSON file and retrieve their status (200 = OK; 409 = conflict)     allStatus=$(curl -H "Content-Type: application/json" -H "Cache-Control: no-cache" --data-binary "@$file" $2/$3/_bulk_docs --silent | jq '.[] |.status' | tr -d '\r')     docIndex=0      #Loop on every status (one status = one document)     while IFS=' ' read -ra statusArray; do       for status in "${statusArray[@]}"; do          #Remove unwanted windows characters of the file         sed -i 's/\r//g' $file         fileContent=`cat $file`          #Retrieve the id of the current document         id=`jq -r -j ".docs[$docIndex]._id"&lt;&lt;&lt;"$fileContent" | tr -d '\r'`          if [ "$status" = "409" ]         then              #Retrieve the revision of the current document and add it to the document              rev=$(curl -X GET --header 'Accept: application/json' $2/$3/$id?revs=true --silent | jq -r -j '._rev' | tr -d '\r')             result=$(jq -c -j ".docs[$docIndex] + { \"_rev\": \"$rev\" }"&lt;&lt;&lt;"$fileContent")              #Wrap the document inside {"docs":[]}             result="{\"docs\":[$result]}"              #Remove unwanted windows characters before sending the document (again)             result=$result | sed 's/\r//g'             result=$result | iconv -t "UTF-8"             s=$(curl -H "Content-Type: application/json; charset=UTF-8" -H "Cache-Control: no-cache" -d "$result" $2/$3/_bulk_docs --silent)              #Echo result             echo $s         else           #Status should be 200.           echo 'status: '$status ' - ' $id         fi          docIndex=$((docIndex+1))        done     done &lt;&lt;&lt; "$allStatus" done   This script aims to upsert documents inside a NoSQL database. If first try to insert and if if fails it retrieves a property (the revision) of the document, append to it and then retry once.  I know this script could be improved but this is my first bash script and it won't be used in production (only for testing, etc.).
105734
In my web app, I submit some form fields with jQuery's $.getJSON() method. I am having some problems with the encoding. The character-set of my app is charset=ISO-8859-1, but I think these fields are submitted with UTF-8.   How I can set encoding used in $.getJSON calls?
54650
Currently I had built a file upload functionality in Lightning component which accepts the CSV file, read it as text(Each row in CSV is split using \n and each column in the row is split using ,). Then each columns in each row are validated in Javascript and if everything is fine,the data is saved in the backend. While uploading the data, if the file contains , in its data then it is taken as another column. Eg: X-36936,SS-341487,11-WT,414081,"LENDERS, THE HACK",2-Manage,,,,,,,,,,,,12/1/2013,3150,,0,,Dealing,2018,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  Here "LENDERS, THE HACK" is a single column but it is taken as 2 columns (LENDERS --> 1st column and THE HACK --> 2nd column).Could anyone please help me how to resolve this issue?
4578
I'm writing a book using Latex in what I want to write code in Python. My operating system is Linux and my editor is Kile with:  \usepackage[T1]{fontenc} \usepackage[utf8]{inputenc} \usepackage{verbments}   But I can not see the characters in Spanish  \begin{pyglist}[language = python, encoding = utf8]     print ('Tomorrow is another day') \end{pyglist}   I perform the compilation with the command  pdflatex -shell-escape -interaction = nonstopmode   but I get the following error.  *** Error while highlighting: UnicodeDecodeError: 'utf-8' codec can not decode byte 0xf1 in position 9: invalid continuation byte     (file "/usr/lib/python3.3/site-packages/pygments/lexer.py", line 164, in get_tokens)   I also have observed that verbments generates a file called name.pyg.sty that curiously is codified in ISO-8859-15 instead of UTF-8 as all my other files.
57454
I'm currently working on a Java project that is emitting the following warning when I compile:  /src/com/myco/apps/AppDBCore.java:439: warning: unmappable character for encoding UTF8     [javac]         String copyright = "� 2003-2008 My Company. All rights reserved.";   I'm not sure how SO will render the character before the date, but it should be a copyright symbol, and is displayed in the warning as a question mark in a diamond.  It's worth noting that the character appears in the output artifact correctly, but the warnings are a nuisance and the file containing this class may one day be touched by a text editor that saves the encoding incorrectly...  How can I inject this character into the "copyright" string so that the compiler is happy, and the symbol is preserved in the file without potential re-encoding issues?
37769
I don't know much of Ubuntu, but is Ubuntu able to decode these sort of things? Or do I need to install some program? If I don't, how do I use Ubuntu to read the file?
48210
I've heard people talking about "base 64 encoding" here and there.  What is it used for?
76414
Background: This question relates to versions of Delphi below 2009 (ie without Unicode support built in). I have a specification that requires me to transmit a Unicode encoded string over a TCP connection but I do not have Delphi 2009.   Question Is there a single function or very small library (I don't need too much bulk) that I can use to encode a single string into UTF-8 immediately prior sending over the wire? As a second part of my question: if there are UTF-8 encoded strings being sent back as a response, I guess I would then need another function to get it back into a Delphi string format. I understand the limitations of such Unicode support in this way.
70981
I'm having problems trying to export a data extension from Salesforce Marketing Cloud to FTP.  I created an Automation to extract this file, but I want to extract in UTF-8, my database only import files in this format.  So, how can I convert my data extract to UTF-8, instead of UTF-16?  I've tried to add a new Data Extract Activity, using ExtractType: Custom UTF-16 to UTF-8 Encoding. But that it isn't working, occur an error in this activity. I've fund that some guy did a , but I can not find that option in my Salesforce Marketing Cloud Business Unit.
33183
I sometimes need to open text files that are encoded in EUC-KR.    man gedit gives:      --encoding           Set the character encoding to be  used  for  opening  the  files           listed on the command line.    This can be used to open specific files with the desired encoding:  gedit --encoding euc-kr readme.txt   However, each file to be opened requires a similar command to be typed.  I can run iconv on the file(s), but then I have to deal with different file versions.    Do I have better options for dealing with non-utf8 text files?
87456
How do I prepare CSV files for use in ArcGIS Desktop.   I ask because I have some troubles using CSV files because ArcGIS attributes wrong field types to my columns and also misinterprets special characters such as á or ê.   I have read in the Esri forum that there is a so-called schema.ini file that defines somehow the field types e.g "Col22=V002 Text" see here   That's kind of funny because I have often seen these .ini files on my disc but never actually wondered what they are good for. It is kind of weird that Excel stores such metadata in an extra file since other programs like R don't do so.   I already tried to manipulate this .ini file with little success since I didn't find out how to apply for example "string" type. There are some information on MS sites, see here:  but I couldn't find a solution.  Also I didn't really like the idea to work with this .ini file because it is quite a bunch of work to define and type all the fieldnames when I have let's say 50 columns. And these .ini files might get lost, etc.