I am trying to lowercase text in a text file. The text file is about 7 GB and I am trying to go through it line by line and lowercase all words. But I get a memory error and I don't understand why because I don't load the 7GB file at once.  My code:  import os directory = '.'  for filename in os.listdir(directory):  # iterate through text files     if filename.endswith(".txt"):         with open(filename, "r", encoding='utf8') as r: # open file in read mode              with open("./data/finalData.txt", "w", encoding='utf8') as w: # open the destination file where I want to write the lowercase text data                 for line in r:                     line = line.lower()                           w.writelines(line) print("Lowercasing done!")    What I get is:  MemoryError                               Traceback (most recent call last) &lt;ipython-input-5-282ad07ebddc&gt; in &lt;module&gt;       5         with open(filename, "r", encoding='utf8') as r:       6              with open("./data/finalData.txt", "w", encoding='utf8') as w: ----&gt; 7                 for line in r:       8                     line = line.lower()       9                     w.writelines(line)  MemoryError:
54110
I have a very big file 4GB and when I try to read it my computer hangs. So I want to read it piece by piece and after processing each piece store the processed piece into another file and read next piece.  Is there any method to yield these pieces ?  I would love to have a lazy method.
86471
I have an empty file (only zeroes are in it) of size 9,0KB and I need to write another file (with size 1,1KB) to it, but the first file must not lose its size or the rest of its contents. So if the whole file is 00000000000000... now, I need to write second file in it and leave the zeroes as they are. I have tried to use dd, but I haven't succeed - file resizes.  dd if=out/one.img of=out/go.img   Does anybody know how can I do it?
110525
In a letter-class document that had no previous errors I am suddenly getting the error message:  "There's no line here to end" \opening{text}   I've used this file many times and today it starts giving this error. How do I troubleshoot this?
34971
I would like to cat file, but with limit to max length to each line. For example I a have file with 10 lines each have 10000 characters and I would like to print first 100 characters from each line. Is something like that possible with cat or some alternative? thx.
32406
Technically, any odd number of backslashes, as described in .  &gt;&gt;&gt; r'\'   File "&lt;stdin&gt;", line 1     r'\'        ^ SyntaxError: EOL while scanning string literal &gt;&gt;&gt; r'\\' '\\\\' &gt;&gt;&gt; r'\\\'   File "&lt;stdin&gt;", line 1     r'\\\'          ^ SyntaxError: EOL while scanning string literal   It seems like the parser could just treat backslashes in raw strings as regular characters (isn't that what raw strings are all about?), but I'm probably missing something obvious.
11923
I'm trying to split a string up into words and punctuation, adding the punctuation to the list produced by the split.  For instance:  &gt;&gt;&gt; c = "help, me" &gt;&gt;&gt; print c.split() ['help,', 'me']   What I really want the list to look like is:  ['help', ',', 'me']   So, I want the string split at whitespace with the punctuation split from the words.  I've tried to parse the string first and then run the split:  &gt;&gt;&gt; for character in c: ...     if character in ".,;!?": ...             outputCharacter = " %s" % character ...     else: ...             outputCharacter = character ...     separatedPunctuation += outputCharacter &gt;&gt;&gt; print separatedPunctuation help , me &gt;&gt;&gt; print separatedPunctuation.split() ['help', ',', 'me']   This produces the result I want, but is painfully slow on large files.  Is there a way to do this more efficiently?
82700
Is it possible to read and write Word (2003 and 2007) files in Python without using a COM object? I know that I can:  f = open('c:\file.doc', "w") f.write(text) f.close()   but Word will read it as an HTML file not a native .doc file.
37773
I wrote a bash script that processes a list of filenames (using glob expansion, as in for f in *) and then outputs a subset of this list to a file. Subsequently I read the content of this file into an array and perform a plain binary search for specific filenames using the obvious &lt; and &gt; operators for comparing strings.  Considering that I would like the script to work on as many different environments like Linux, MacOS, MinGW, ... (even though it uses things like [[ and stat which are less portable), my questions are:   Do I need to sort the file content (either with sort or additional bash code) or is the glob expansion always sorted - in every environment? Do the conditional operators use the same "sorting" as the expansion (or after sort)?  Would expansion or sort return file10.txt after file2.txt (in what cases?) but using conditional operators file10.txt would be before file2.txt ? What sort option would I  use to fix this? Are there any caveats if some of my filenames are in Unicode?   Are there any issues using specific versions of bash?   Does LC_COLLATE affect any of the above?     I obviously need the file content to match the sorting "method" of the operators in order for the binary search to work as expected...
2957
Which is correct? Why?   Not enough memory No enough memory    seems to be significantly more popular on the Internet than the .  I want to use such construction as an error message in the program. I think it is a short version of the sentence "There is not enough memory".
93075
If I want to tail a 25 GB textfile, does the tail command read the whole file?  Since a file might be scattered on a disk I imagine it has to, but I do not understand such internals well.
11666
A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table).  What I need is to remove all the repetitions while (preferably, but this can be sacrificed for a significant performance boost) maintaining the original sequence order. In the result each line is to be unique. If there were 100 equal lines (usually the duplicates are spread across the file and won't be neighbours) there is to be only one of the kind left.  I have written a program in Scala (consider it Java if you don't know about Scala) to implement this. But maybe there are faster C-written native tools able to do this faster?  UPDATE: the awk '!seen[$0]++' filename solution seemed working just fine for me as long as the files were near 2 GiB or smaller but now as I am to clean-up a 8 GiB file it doesn't work any more. It seems taking infinity on a Mac with 4 GiB RAM and a 64-bit Windows 7 PC with 4 GiB RAM and 6 GiB swap just runs out of memory. And I don't feel enthusiastic about trying it on Linux with 4 GiB RAM given this experience.
36590
I want to find, say, 10 most common word in a text file. Firstly, solution should be optimized for keystrokes (in other words - my time). Secondly, for the performance. Here is what I have so far to get top 10:  cat test.txt | tr -c '[:alnum:]' '[\n*]' | uniq -c | sort -nr | head  -10   6 k   2 g   2 e   2 a   1 r   1 k22   1 k   1 f   1 eeeeeeeeeeeeeeeeeeeee   1 d   I could make a java, python etc. program where I store (word, numberOfOccurences) in a dictionary and sort the value or I could use MapReduce, but I optimize for keystrokes.   Are there any false positives? Is there a better way?
67258
I have a text file that I can change using other applications (for example openoffice).  But when I try to change and save it using gedit, I am getting error from gedit:  Could not save the file /media/sf_Ubuntu/BuildNotes.txt. Unexpected error: Error renaming temporary file: Text file busy   the permission of BuildNotes.txt is as follow:  -rwxrwx--- 1 root vboxsf  839 2012-10-26 12:08 BuildNotes.txt   and user id is:  m@m-Linux:/media/sf_Ubuntu$ id uid=1000(m) gid=1000(m) groups=4(adm),20(dialout),24(cdrom),46(plugdev),105(lpadmin),119(admin),122(sambashare),1000(m),1001(vboxsf)   What is the problem and how I can fix it?
77902
I have a small text file that I'd like to read into a scalar variable exactly as it is in the file (preserving line separators and other whitespace).  The equivalent in Python would be something like  buffer = ""  try:     file = open("fileName", 'rU')     try:         buffer += file.read()     finally:         file.close() except IOError:     buffer += "The file could not be opened."   This is for simply redisplaying the contents of the file on a web page, which is why my error message is going into my file buffer.
92353
Assuming that the grep tool should be used, I'd like to search for the text string "800x600" throughout the entire file system.  I tried:  grep -r 800x600 /   but it doesn't work.  What I believe my command should do is grep recursively through all files/folders under root for the text "800x600" and list the search results.  What am I doing wrong?
26354
I was doing a very simple search:  grep -R Milledgeville ~/Documents   And after some time this error appeared:  grep: memory exhausted   How can I avoid this?  I have 10GB of RAM on my system and few applications running, so I am really surprised a simple grep runs out of memory. ~/Documents is about 100GB and contains all kinds of files.  grep -RI might not have this problem, but I want to search in binary files too.
17335
file_A (~500MB, 1.6M lines) consists of all equal length search terms, 1 per line, not sorted.  file_B consists of all equal length text lines, 1 per line, not sorted  I've been able to run "grep -F -f file_A file_B >> output.txt" with any size file_B without problem on a box with 52GB ram. Problem is I'm now limited to 4GB ram and thus the size of file_A is now too large for this to run without exhausting available memory.  Short of manually chopping up file_A into smaller bites, is there any easy way to script this to grep for first 1000 lines of file_A, then when thats finished to automatically grep for lines 1001-2000, ect. until I've gone through all of file_A?