I have a list of dictionaries each of them describing a file (file format, filename, filesize, ... and a full path to the file [always unique]). The goal is to exclude all but one dictionaries describing copies of the same file (I just want a single dict (entry) per file, no matter how many copies there are.   In other words: if 2 (or more) dicts differ only in a single key (i.e. path) - leave only one of them).  For example, here is the source list:  src_list = [{'filename': 'abc', 'filetype': '.txt', ... 'path': 'C:/'},             {'filename': 'abc', 'filetype': '.txt', ... 'path': 'C:/mydir'},             {'filename': 'def', 'filetype': '.zip', ... 'path': 'C:/'},             {'filename': 'def', 'filetype': '.zip', ... 'path': 'C:/mydir2'}]   The result should look like this:  dst_list = [{'filename': 'abc', 'filetype': '.txt', ... 'path': 'C:/'},             {'filename': 'def', 'filetype': '.zip', ... 'path': 'C:/mydir2'}]
8585
Is there a built-in that removes duplicates from list in Python, whilst preserving order? I know that I can use a set to remove duplicates, but that destroys the original order. I also know that I can roll my own like this:  def uniq(input):   output = []   for x in input:     if x not in output:       output.append(x)   return output   (Thanks to  for that .)  But I'd like to avail myself of a built-in or a more Pythonic idiom if possible.  Related question:
48958
I have been trying to create a file called ~/dictwords.txt, which contains the number of words found in the file /usr/share/dict/words.  I already created the file then I removed it because I did it wrong I originally did the echo command. Then I tried this command and was so excited because I thought I was correct.  wc -w /usr/share/dict/words &gt; ~/dictwords.txt   It's still wrong, I am supposed to have just the number, without a file name.
95955
I have a file that I would like to break up into multiple files with uniq values for the first column. For example, here is a file:  fileA.txt   1    Cat 1    Dog 1    Frog 2    Boy 2    Girl 3    Tree 3    Leaf 3    Branch 3    Trunk   I would like my output to look something like this:  file1.txt   1    Cat 2    Boy 3    Tree   file2.txt  1    Dog 2    Girl 3    Leaf   file3.txt  1    Frog 3    Branch   file4.txt  3    Trunk   If a value does not exist, I want it to be skipped. I have tried to search for similar situations to mine, but I've come up short. Does anyone have idea of how to do this?  Edit: My awk version is: awk version 20070501
52830
I need to build a list of files present in a directory but also pass in a file containing a list of directories to exclude.  I've seen the below suggestion but I've had no luck with it when it comes to the type flag:  find $dir_name $(printf "! -name %s " $(cat $exclude_file))   Is there a way I can do it or will I need to create the entire list and then work through it removing ones that partially match ones in the exclude directory.
82674
by the following command we can print the duplicate lines from file  uniq -d string file.txt   but how we can do it on csv file ?  we need to print the duplicate lines only on fields 1,2 from csv file - not include field 3  FS - ","  for example:   spark2-thrift-sparkconf,spark.history.fs.logDirectory,{{spark_history_dir}}  spark2-thrift-sparkconf,spark.history.fs.logDirectory,true  spark2-thrift-sparkconf,spark.history.Log.logDirectory,true  spark2-thrift-sparkconf,spark.history.DF.logDirectory,true   expected results:   spark2-thrift-sparkconf,spark.history.fs.logDirectory,{{spark_history_dir}}  spark2-thrift-sparkconf,spark.history.fs.logDirectory,true   second:  how exclude the duplicate lines from the csv file ( I mean to delete only the duplicate lines on fields 1,2   expected output:   spark2-thrift-sparkconf,spark.history.Log.logDirectory,true  spark2-thrift-sparkconf,spark.history.DF.logDirectory,true
111191
I'm trying to find all duplicate files (based on MD5 hash) and ordered by file size. So far I have this:  find . -type f -print0 | xargs -0 -I "{}" sh -c 'md5sum "{}" |  cut -f1 -d " " | tr "\n" " "; du -h "{}"' | sort -h -k2 -r | uniq -w32 --all-repeated=separate   The output of this is:  1832348bb0c3b0b8a637a3eaf13d9f22 4.0K   ./picture.sh 1832348bb0c3b0b8a637a3eaf13d9f22 4.0K   ./picture2.sh 1832348bb0c3b0b8a637a3eaf13d9f22 4.0K   ./picture2.s  d41d8cd98f00b204e9800998ecf8427e 0      ./test(1).log   Is this the most efficient way?
118370
I was using the following command   curl -silent http://api.openstreetmap.org/api/0.6/relation/2919627 http://api.openstreetmap.org/api/0.6/relation/2919628 | grep node | awk '{print $3}' | uniq   when I wondered why uniq wouldn't remove the duplicates. Any idea why ?
27723
I have a single directory that contains dozens of directories inside of it.  I'm new to command line and I'm struggling to come up with a command that will zip each sub-directory into a unique sub-directory.zip file.  So in the end my primary directory will be filled with all of my original sub-directories, plus the corresponding .zip files that contain the zipped up content of each sub-directory.  Is something like this possible? If so, please show me how it's done.
8318
I have a list like so.  blue-image1.jpg, blue-image2.jpg, blue-image3.jpg red-image1.jpg, red-image2.jpg, red-image3.jpg   It's was originally a csv format but I added newlines because they needed a prefix at first. Now I need to put them together again so it becomes like this.  blue-image1.jpg,blue-image2.jpg,blue-image3.jpg red-image1.jpg,red-image2.jpg,red-image3.jpg   I tried cat list.txt | tr ',\n' ',' and cat list.txt | sed 's/,\n/,/g'  but neither seem to work properly.
77381
I have a folder structure with a bunch of *.csv files scattered across the folders. Now I want to copy all *.csv files to another destination keeping the folder structure.  It works by doing:  cp --parents *.csv /target cp --parents */*.csv" /target cp --parents */*/*.csv /target cp --parents */*/*/*.csv /target ...   and so on, but I would like to do it using one command.
101990
I have a file that contains a list of names. i.e.:  Long Name One (001) Long Name Two (201) Long Name Three (123) ...   with spaces and some special characters. I wanted to make directories out of these names, i.e.:  cat file | xargs -l1 mkdir   It makes individual directories separated by spaces, i.e. Long, Name, One, Two, Three, instead of  Long Name One (001), Long Name Two (201), Long Name Three (123).  How can I do that?
32302
dir(re.compile(pattern))    does not return pattern as one of the lists's elements. Namely it returns:  ['__copy__', '__deepcopy__', 'findall', 'finditer', 'match', 'scanner', 'search', 'split', 'sub', 'subn']   According to the manual, it is supposed to contain      the object's attributes' names, the   names of its class's attributes, and   recursively of the attributes of its   class's base classes.   It says also that     The list is not necessarily complete.   Is there a way to get the complete list? I always assumed that dir returns a complete list but apparently it does not...  Also: is there a way to list only attributes? Or only methods?  Edit: this is actually a bug in python -> supposedly it is fixed in the 3.0 branch (and perhaps also in 2.6)
85741
I have a large csv file with these format file 1:  id,ICD,CT 16214497,008.8,1 16227244,401.1,45 27154870,780.79,1   I have another txt file with one column of list of id (header id). file 2:  id 16214497 27154870   I want another file that has only data limited to the list of id in file 2  output file:  id,ICD,CT 16214497,008.8,1 27154870,780.79,1
82404
I want to rsync only certain file types (e.g. .py) and I want to exclude files in some directories (e.g. venv).  This is what I have tried:  rsync -avz --include='*/' --exclude='venv/' --include='*.py' --exclude='*' /tmp/src/ /tmp/dest/   But it doesn't work.  What am I missing?  I also followed the answer to  but it didn't help.
8370
I'm trying to make a zip function that'll do all the steps I typically do when zipping a file or folder. Say my current directory is ~, and I want to zip a folder named Folder1, whose file path is ~/Folder2/Folder1.  Issuing  cd ~/Folder2 &amp;&amp; zip -r Folder1 Folder1 &amp;&amp; cd -   does exactly what I want. Namely, it zips so that the output Folder1.zip does not contain folders that lead up to the folder I'm zipping (namely, /home/user/Folder2), it zips recursively, and it takes in the first argument of zip with the exact name as the file or folder I'm zipping (Folder1.zip).  So I want to make a function in my .zshrc something akin to  coolerzip() { cd /path/to/file &amp;&amp; zip -r file file &amp;&amp; cd - }   where it need only take in one argument, which in this case, it will be ~/Folder2/Folder1. Then Folder1 will be the name for both files in coolerzip, and I will cd right into ~/Folder2 so Folder1 lies in my current directory before zipping. Is there any way I can achieve this so that coolerzip only requires one argument rather than three?
94174
When I am writing code in Python, I often need to remove items from a list or other sequence type based on some criteria. I haven't found a solution that is elegant and efficient, as removing items from a list you are currently iterating through is bad.  For example, you can't do this:  for name in names:     if name[-5:] == 'Smith':         names.remove(name)   I usually end up doing something like this:  toremove = [] for name in names:     if name[-5:] == 'Smith':         toremove.append(name) for name in toremove:     names.remove(name) del toremove   This is innefficient, fairly ugly and possibly buggy (how does it handle multiple 'John Smith' entries?). Does anyone have a more elegant solution, or at least a more efficient one?   How about one that works with dictionaries?
40934
I have a couple of big text files and in the file UNIQS.txt I have a list of strings to grep from another file. The code I use is  grep -f UNIQS.txt EEP_VSL.uniqs.sam &gt; UNIQ_templates.sam   which does nothing - the file generated is empty. But when I do  grep -F -f UNIQS.txt EEP_VSL.uniqs.sam &gt; UNIQ_templates.sam   it works correctly. This confuses me because I didn't think grep would interpret the entries in UNIQS.txt as regexp patterns without quotes and slashes and so on being in the file (which there aren't). Is it the case in general that if you are getting the patterns from a file then it will automatically think that they are regexp patterns?  Edit: In the UNIQS.txt file, there are newline separated strings of the form  HWI-ST365:215:D0GH0ACXX:2:1101:10034:186783   (called template names) and the file EEP_VSL... tab separated columns, with about 14 columns and the first column is the template name, so basically I want to extract the line corresponding to each template in the file.