I'm scraping news articles from various sites, using GAE and Python.  The code where I scrape one article url at a time leads to the following error:  UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 8858: ordinal not in range(128)   Here's my code in its simplest form:  from google.appengine.api import urlfetch  def fetch(url):     headers = {'User-Agent' : "Chrome/11.0.696.16"}     result = urlfetch.fetch(url,headers)     if result.status_code == 200:         return result.content   Here is another variant I have tried, with the same result:  def fetch(url):     headers = {'User-Agent' : "Chrome/11.0.696.16"}     result = urlfetch.fetch(url,headers)     if result.status_code == 200:         s = result.content         s = s.decode('utf-8')         s = s.encode('utf-8')         s = unicode(s,'utf-8')         return s   Here's the ugly, brittle one, which also doesn't work:  def fetch(url):     headers = {'User-Agent' : "Chrome/11.0.696.16"}     result = urlfetch.fetch(url,headers)     if result.status_code == 200:         s = result.content          try:             s = s.decode('iso-8859-1')         except:             pass         try:             s = s.decode('ascii')         except:              pass         try:             s = s.decode('GB2312')         except:             pass         try:             s = s.decode('Windows-1251')         except:             pass         try:             s = s.decode('Windows-1252')         except:             s = "did not work"          s = s.encode('utf-8')         s = unicode(s,'utf-8')         return s   The last variant returns s as the string "did not work" from the last except.  So, am I going to have to expand my clumsy try/except construction to encompass all possible encodings (will that even work?), or is there an easier way?  Why have I decided to scrape the entire html, not just the BeautifulSoup? Because I want to do the soupifying later, to avoid DeadlineExceedError in GAE.  Have I read all the excellent articles about Unicode, and how it should be done? Yes. However, I have failed to find a solution that does not assume I know the incoming encoding, which I don't, since I'm scraping different sites every day.
21512
I received some text that is encoded, but I don't know what charset was used. Is there a way to determine the encoding of a text file using Python?  deals with C#.
45470
I'm trying to make a URL with a tilde character (~) in it:  \usepackage{url} ... \url{http://lyle.smu.edu/~tspell/jaws}   It looks right in the resulting PDF, but when clicking on it the browser goes to a wrong URL: http://lyle.smu.edu/%18tspell/jaws  If I type in the browser %7e (escape code for '~') instead of %18 the browser then goes to the correct URL.  Any idea how to get a correct link in the PDF?
65003
I'd like to know do I normalize a URL in python.   For example, If I have a url string like : " goo/bar.html"  I need a library in python that will transform the extra space (or any other non normalized character) to a proper URL.
57381
I'm having some brain failure in understanding reading and writing text to a file (Python 2.4).  # The string, which has an a-acute in it. ss = u'Capit\xe1n' ss8 = ss.encode('utf8') repr(ss), repr(ss8)      ("u'Capit\xe1n'", "'Capit\xc3\xa1n'")   print ss, ss8 print &gt;&gt; open('f1','w'), ss8  &gt;&gt;&gt; file('f1').read() 'Capit\xc3\xa1n\n'   So I type in Capit\xc3\xa1n into my favorite editor, in file f2.  Then:  &gt;&gt;&gt; open('f1').read() 'Capit\xc3\xa1n\n' &gt;&gt;&gt; open('f2').read() 'Capit\\xc3\\xa1n\n' &gt;&gt;&gt; open('f1').read().decode('utf8') u'Capit\xe1n\n' &gt;&gt;&gt; open('f2').read().decode('utf8') u'Capit\\xc3\\xa1n\n'   What am I not understanding here? Clearly there is some vital bit of magic (or good sense) that I'm missing. What does one type into text files to get proper conversions?  What I'm truly failing to grok here, is what the point of the UTF-8 representation is, if you can't actually get Python to recognize it, when it comes from outside. Maybe I should just JSON dump the string, and use that instead, since that has an asciiable representation! More to the point, is there an ASCII representation of this Unicode object that Python will recognize and decode, when coming in from a file?  If so, how do I get it?  &gt;&gt;&gt; print simplejson.dumps(ss) '"Capit\u00e1n"' &gt;&gt;&gt; print &gt;&gt; file('f3','w'), simplejson.dumps(ss) &gt;&gt;&gt; simplejson.load(open('f3')) u'Capit\xe1n'
104638
I'm playing around, trying to write some code to use the  APIs to shorten a URL.  After reading , I tried:     TRIM_API_URL = 'http://api.tr.im/api'    auth_handler = urllib2.HTTPBasicAuthHandler()    auth_handler.add_password(realm='tr.im',                              uri=TRIM_API_URL,                              user=USERNAME,                              passwd=PASSWORD)    opener = urllib2.build_opener(auth_handler)    urllib2.install_opener(opener)    response = urllib2.urlopen('%s/trim_simple?url=%s'                               % (TRIM_API_URL, url_to_trim))    url = response.read().strip()   response.code is 200 (I think it should be 202).  url is valid, but the basic HTTP authentication doesn't seem to have worked, because the shortened URL isn't in my list of URLs (at ).  After reading  I also tried:     TRIM_API_URL = 'api.tr.im/api'    password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()    password_mgr.add_password(None, TRIM_API_URL, USERNAME, PASSWORD)    auth_handler = urllib2.HTTPBasicAuthHandler(password_mgr)    opener = urllib2.build_opener(auth_handler)    urllib2.install_opener(opener)    response = urllib2.urlopen('http://%s/trim_simple?url=%s'                               % (TRIM_API_URL, url_to_trim))    url = response.read().strip()   But I get the same results.  (response.code is 200 and url is valid, but not recorded in my account at .)  If I use query string parameters instead of basic HTTP authentication, like this:     TRIM_API_URL = 'http://api.tr.im/api'    response = urllib2.urlopen('%s/trim_simple?url=%s&amp;username=%s&amp;password=%s'                               % (TRIM_API_URL,                                  url_to_trim,                                  USERNAME,                                  PASSWORD))    url = response.read().strip()   ...then not only is url valid but it's recorded in my tr.im account. (Though response.code is still 200.)  There must be something wrong with my code though (and not tr.im's API), because  $ curl -u yacitus:xxxx http://api.tr.im/api/trim_url.json?url=http://www.google.co.uk   ...returns:  {"trimpath":"hfhb","reference":"nH45bftZDWOX0QpVojeDbOvPDnaRaJ","trimmed":"11\/03\/2009","destination":"http:\/\/www.google.co.uk\/","trim_path":"hfhb","domain":"google.co.uk","url":"http:\/\/tr.im\/hfhb","visits":0,"status":{"result":"OK","code":"200","message":"tr.im URL Added."},"date_time":"2009-03-11T10:15:35-04:00"}   ...and the URL does appear in my list of URLs on .  And if I run:  $ curl -u yacitus:xxxx http://api.tr.im/api/trim_url.json?url=http://www.google.co.uk   ...again, I get:  {"trimpath":"hfhb","reference":"nH45bftZDWOX0QpVojeDbOvPDnaRaJ","trimmed":"11\/03\/2009","destination":"http:\/\/www.google.co.uk\/","trim_path":"hfhb","domain":"google.co.uk","url":"http:\/\/tr.im\/hfhb","visits":0,"status":{"result":"OK","code":"201","message":"tr.im URL Already Created [yacitus]."},"date_time":"2009-03-11T10:15:35-04:00"}   Note code is 201, and message is "tr.im URL Already Created [yacitus]."  I must not be doing the basic HTTP authentication correctly (in either attempt).  Can you spot my problem?  Perhaps I should look and see what's being sent over the wire?  I've never done that before.  Are there Python APIs I can use (perhaps in pdb)?  Or is there another tool (preferably for Mac OS X) I can use?
61580
What is the quickest way to HTTP GET in Python if I know the content will be a string? I am searching the documentation for a quick one-liner like:  contents = url.get("http://example.com/foo/bar")   But all I can find using Google are httplib and urllib - and I am unable to find a shortcut in those libraries.  Does standard Python 2.5 have a shortcut in some form as above, or should I write a function url_get?   I would prefer not to capture the output of shelling out to wget or curl.
98690
Let's say I open a webpage with some Unicode characters, say, Cyrillic, in the address like this:        When I try to copy it from the address bar somewhere else, it becomes unreadable rubbish:        I guess this is for compatibility. However for readability I want to copy it straight away with proper Unicode characters.  What and how should I tweak to make that possible?  Edit: The answers are good, but they are hacky. Are there any settings in about:config?
90369
Here is the code:  import bpy import csv import sys csv_path = sys.argv[1]  file = open(csv_path) reader = csv.reader(file)  header = next(reader) # 1st line is header  data = [ ] for row in reader:     name0 = row[0]     name1 = row[0]     pathout = 'test-videos/' + row[5] + '.m4v'     s = bpy.context.scene     f = s.objects['FirstName']     f.data.body = name0     f1 = s.objects['FirstName01']     f1.data.body = name1+'?'     bpy.context.scene.render.filepath = pathout     bpy.ops.render.render(animation=True)  bpy.ops.wm.quit_blender()   And here is the error message:  Read blend: /PI-SAT/proj001/proj001-master.blend Traceback (most recent call last): File "/PI-SAT/proj001/proj001master.py", line 9, in &lt;module&gt; header = next(reader) # 1st line is header File"/Applications/Blender/blender.app/Contents/Resources/2.79/python/ lib/python3.5/codecs.py", line 321, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xef in position 23: invalid continuation byte Error: File format is not supported in file '/PI-SAT/proj001/test.csv' ... opened default scene instead; saving will write to /PISAT/proj001/test.csv   I have read the file with a different script using the exact same code as in lines 2 - 15 and then printing the names and pathout. Running the script from the command line, not from within Blender, it works perfectly.  I have also run the part of the script within the for loop outside of the loop as a script from the command line and it works perfectly.  What am I missing about the CSV file? I created it in Excel on my Mac; is that a problem?
3118
In     ... the number "5000" does not paste as such but instead as  5 0 U+200C U+200B 0 0   The HTML source looks like this:  perhaps the easiest would be &lt;code&gt;\edef\tmp{\everypar{\widowpenalty\the\widowpenalty\relax}}\tmp\widowpenalty=-50&amp;zwnj;&amp;#8203;00&lt;/code&gt; that may reset   Or (smaller snippet):  \widowpenalty=-50&amp;zwnj;&amp;#8203;00&lt;/code&gt; that   So in the HTML, we have extraneous &amp;zwnj; and &amp;#8203; in between the two middle digits of 5000.  Upon pasting and trying out this code, this first led to a mysterious compiler error. The same issue happened within a (now deleted) comment thread below ; there, these or similar two characters were inserted after the letter "r" within the string \ref ("\ref").  Note that not all browsers wrap the same way. The above screenshot was for Firefox 19.0.2, in which it breaks between \relax}} and \tmp, with 5000 insidiously left intact on the second line but with the two special characters remaining in the middle. So that is why this took me completely by surprise. Other browsers might indeed only break on the invisible Unicode characters, such as Chrome 25:    Or on a smaller screen with the mobile site, multiple line breaks might show, possibly still leaving the Unicode characters unused:
122327
I want to I check whether a string is in ASCII or not.  I am aware of ord(), however when I try ord('é'), I have TypeError: ord() expected a character, but string of length 2 found. I understood it is caused by the way I built Python (as explained in ).   Is there another way to check?
7145
Is it possible/legal to somehow encode CR/LF characters into a CSV file?  (as part of a CSV standard?)  If so how should i encode CR/LF?
81778
When you visit the global search page:        and want to search for a tag like we used to do on the site search like so:     [haskell] pattern   you're directed to the error page with the Panda.  If tag search is supposed to work, can this be fixed? If tag search is not supposed to work there can you at a minimum not throw an error?
77800
I am trying to process telugu text with Python. Text is rendering correctly in chrome browser like this.     Unfortunately it is not rendering correctly in default terminal or python shell in terminal.      Can you help in finding what is causing the problem?  What I have tried so far:  Installed all available telugu fonts  sudo apt-get install fonts-telu-extra fonts-telu fonts-lohit-telu   Setup unicode support in terminal   locale              LANG=en_US.UTF-8 LANGUAGE=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=en_US.UTF-8   Enable te_IN in /etc/locale.gen  Setup telugu language from Language Support.  Download custom telugu fonts and load them.  None of these have solved the problem. Without proper rendering, it is very hard to understand text. Any help in troubleshooting this is highly appreciated.
2897
Today, I was caught redhanded while attempting to code golf a password generator for Twitter.  import string as s,random;print ''.join(random.sample(s.letters+s.digits+s.punctuation,9))   90 chars. Since that is a lot of spare space, I decided to raise the bar and make it executable too.  echo -e "#!/usr/bin/python\nimport string as s,random;print ''.join(random.sample(s.letters+s.digits+s.punctuation,9))"&gt;pg;chmod +x pg;./pg   139 chars. Nice, except obviously bash chokes on the exclamation point.  badp@delta:~$ echo -e "#!/usr/bin/python\nimport string as s,random;print ''.join(random.sample(s.letters+s.digits+s.punctuation,9))"&gt;pg;chmod +x pg;./pg bash: !/usr/bin/python\nimport: event not found   Pesky exclamation point. "Let's escape it," I thought! I do have one spare character after all.  echo -e "#\!/usr/bin/python\nimport string as s,random;print ''.join(random.sample(s.letters+s.digits+s.punctuation,9))"&gt;pg;chmod +x pg;./pg   Obviously...  badp@delta:~$ echo -e "#\!/usr/bin/python\nimport string as s,random;print ''.join(random.sample(s.letters+s.digits+s.punctuation,9))"&gt;pg;chmod +x pg;./pg ./pg: line 2: syntax error near unexpected token `(' ./pg: line 2: `import string as s,random;print ''.join(random.sample(s.letters+s.digits+s.punctuation,9))' badp@delta:~$ cat pg #\!/usr/bin/python import string as s,random;print ''.join(random.sample(s.letters+s.digits+s.punctuation,9))   Leaving my asinine code golfing aside -- I can't explain this.  With \!, the exclamation point was escaped, except it really wasn't, because the \! was left as-is for echo to pick up.  One solution could have been using \x21 instead, but I'm not convinced that's the proper way of escaping an exclamation point in a bash command.  tl;dr: How do you properly escape an exclamation point in a bash command?
4578
I'm writing a book using Latex in what I want to write code in Python. My operating system is Linux and my editor is Kile with:  \usepackage[T1]{fontenc} \usepackage[utf8]{inputenc} \usepackage{verbments}   But I can not see the characters in Spanish  \begin{pyglist}[language = python, encoding = utf8]     print ('Tomorrow is another day') \end{pyglist}   I perform the compilation with the command  pdflatex -shell-escape -interaction = nonstopmode   but I get the following error.  *** Error while highlighting: UnicodeDecodeError: 'utf-8' codec can not decode byte 0xf1 in position 9: invalid continuation byte     (file "/usr/lib/python3.3/site-packages/pygments/lexer.py", line 164, in get_tokens)   I also have observed that verbments generates a file called name.pyg.sty that curiously is codified in ISO-8859-15 instead of UTF-8 as all my other files.
123507
I have done it many times (clipping a raster with a polygon layer or by extension) but today I got an error for the first time and I don't know how to solve it. I am always using the same type of raster layers. When running the clipping tool I got the following message:  gdal_translate -of GTiff -ot Byte -projwin 410282.185971 4694950.41162 410783.882475 4694524.96815 -co COMPRESS=DEFLATE -co PREDICTOR=1 -co ZLEVEL=6 -co TFW=YES "C:/Users/FERRAN ALA/Desktop/Justificació Fontanals/QGIS/Topo 1_5000/bt5mv20sd0f283077st1r031.sid" "C:/Users/FERRAN ALA/Desktop/Justificació Fontanals/QGIS/background/image.tif"   GDAL command output:     'ascii' codec can't encode character u'\xf3' in position 201: ordinal not in range(128) See log for more details   Does anybody know what that is?
123690
When looking into it in WebmasterTools, I saw that there is a "Fetch as Google - Temporarily Unreachable Issue". It is for an image, and font.  How do I resolve this issue?