Input: file with sorted lines  Output: file with 'unique' lines that match adjacent lines if we are to remove all digits  Example  Input  abbylove2007 abbylove2008 abbylove2012 AbbyLove2014 abby1994lover abby2007lover abbylovesaal2018 abbylovesbsb2003   Output  abbylove2007 abby1994lover   Here abbylove2007 and abby1994lover are left since there are more than one adjacent line that matches abbylove\d+ and abby\d+lover
122664
Input: file with sorted lines  Output: file with (first or any) unique line ids from the input file  Example  Input  abbylove abbylove abbylove abbylover abbylover abbylovesyou abbylovesyou abbylsmith abbylsmith abbylyn abbylynn   Output  1 4 6 8 10 11   0-indexed also OK.  How to create such an output? I don't find such an option in shell uniq and sort commands.  Updated.  I'm trying to find unique adjacent lines from a file that differs only in years, that they contain. For example, I'd like to filter the file  abbylove2016 abbylove2017 abbylove2018 abb1999ylover abb2005ylover abbyloves2001you abbyloves2006you abbylsm1980ith abbylsm2010ith abbylyn2002 abbylynn1999   and get only  abbylove2016 abb1999ylover abbyloves2001you abbylsm1980ith abbylyn2002 abbylynn1999   I thought removing years could help...
94270
The command that I have prints out two of the lines whenever FW_6.0.0 is found, below is the code:  grep -oP 'FW_6.0.0, (.*)$' file   Below is the output, both has the same value of FW_6.0.0  FW_6.0.0, SUCCESS FW_6.0.0, OK   I would like to match two words, that is FW_6.0.0 and SUCCESS that can be found on the same line so that it prints this out:  FW_6.0.0, SUCCESS and eliminates FW_6.0.0, OK
119344
File1.txt      id                            No     gi|371443199|gb|JH556661.1| 7907290     gi|371443198|gb|JH556662.1| 7573913     gi|371443197|gb|JH556663.1| 7384412     gi|371440577|gb|JH559283.1| 6931777   File2.txt   id                              P       R       S  gi|367088741|gb|AGAJ01056324.1| 5       5       0  gi|371443198|gb|JH556662.1|     2       2       0  gi|367090281|gb|AGAJ01054784.1| 4       4       0  gi|371440577|gb|JH559283.1|     21      19      2   output.txt   id                              P       R       S  NO  gi|371443198|gb|JH556662.1|     2       2       0  7573913  gi|371440577|gb|JH559283.1|     21      19      2  6931777   File1.txt has two columns &amp; File2.txt has four columns. I want to join both files which has unique id (array[1] should match in both files (file1.txt &amp; file2.txt) and give ouput only matched id (see output.txt).  I have tried join -v &lt;(sort file1.txt) &lt;(sort file2.txt). Any help with awk or join commands requested.
70561
I have a text file:  deiauk 1611516 afsdf 765 minkra 18415151 asdsf 4152 linkra sfsfdsfs sdfss 4555 deiauk1 sdfsfdsfs 1561 51 deiauk2 115151 5454 4 deiauk 1611516 afsdf ddfgfgd luktol1 4545 4 9 luktol 1   and I want to match exactly deiauk.  When I do this:  grep "deiauk" file.txt   I get this result:  deiauk 1611516 afsdf 765 deiauk1 sdfsfdsfs 1561 51 deiauk2 115151 5454 4   but I only need this:  deiauk 1611516 afsdf 765 deiauk 1611516 afsdf ddfgfgd   I know there's a -w option, but then my string has to mach whole line.
18132
Consider a text file with the following entries:  aaa bbb ccc ddd eee fff ggg hhh iii   Given a pattern (e.g. fff), I would like to grep the file above to get in the output:  all_lines except (pattern_matching_lines  U (B lines_before) U (A lines_after))   For example, if B = 2 and A = 1, the output with pattern = fff should be:  aaa bbb ccc hhh iii   How can I do this with grep or other command line tools?    Note, when I try:  grep -v 'fff'  -A1 -B2 file.txt   I don't get what I want. I instead get:  aaa bbb ccc ddd eee fff -- -- fff ggg hhh iii
97947
I’m taking two input files, one with certain ID numbers, and another with a large list of ID numbers and additional columns. The latter file contains multiple lines for each ID number and I need to extract all lines that match an ID from the first file. Those lines then must be printed in a new file.  Edit 1: Replaced sample files with excerpts from actual  Edit 2: Removed extra spaces that were in excerpt, but not actual file. Files likely need to be sanitized in some way, but how is unclear.  file1:  AT1G56430 AT3G55190 AT3G22880   file2:  AT1G01010|GO:0043090|RCA AT1G56430|GO:0010233|IGI  AT1G56430|GO:0009555|IGI  AT1G56430|GO:0030418|IGI   expected output  AT1G56430|GO:0010233|IGI  AT1G56430|GO:0009555|IGI  AT1G56430|GO:0030418|IGI   [[  I have tried:  awk -F'|' 'NR==FNR{c[$1$2]++;next};c[$1$2] &gt; 0' file1 file2 &gt; output.txt   and:  grep -Ff file2 file1 &gt; output.txt   I’m aware that there are many somewhat similar questions posted in these forums and others. However, these don’t mention how the output is handled… nor do they mention duplicates. I’ve tried solutions from 4 of them, have been messing with this for many hours and keep getting the same problem: a blank output file.   I’m new to awk and I greatly appreciate the help. Sorry if this is a simple problem with syntax etc; please let me know. Thanks for the help.
4618
I have a file containing two columns and 10 million rows. The first column contains many repeated values, but there is a distinct value in column 2. I want to remove the repeated rows and want to keep only one using awk. Note: the file is sorted with values in column 1. For example:    1.123 -4.0 2.234 -3.5 2.234 -3.1 2.234 -2.0 4.432 0.0 5.123 +0.2 8.654 +0.5 8.654 +0.8 8.654 +0.9 . . . .   Expected output  1.123 -4.0 2.234 -3.5 4.432 0.0 5.123 +0.2 8.654 +0.5 . . . .
24166
It took me a while to word this best.  Problem 1:  Is there a way to find lines in a text file that have the same string at the beginning and the same strings at the end?   Beginning and end strings not necessarily equal to each other.  Text between the two strings is not consistant in number of characters or number of hyphens.  A hyphen is in between notable strings (which I added to my large text file).  For example:  File:    AAAAA-tjhut-ghyut-FILE1.txt   BBB-fhuj-dnbhg-frt76-FILE2.txt   AAAAA-ghtujukr-hgute7gr-xhyutt-FILE1.txt   AAAAA-VGYT-4d7tt-xhyutt-FILE1.txt   CCCC-fjuk-dryy-fhuj-FILE3.txt   Lines to detect:    AAAAA-tjhut-ghyut-FILE1.txt   AAAAA-ghtujukr-hgute7gr-xhyutt-FILE1.txt   AAAAA-VGYT-4d7tt-xhyutt-FILE1.txt   Problem 2:  Is there a way to remove all but one of the lines containing the duplicate strings at the beginning and end from above?  Before:    AAAAA-tjhut-ghyut-FILE1.txt   AAAAA-ghtujukr-hgute7gr-xhyutt-FILE1.txt   AAAAA-VGYT-4d7tt-xhyutt-FILE1.txt   After:    AAAAA-tjhut-ghyut-FILE1.txt   Big thanks for any advice.  Note: I think using the occurances of the first hyphen and last hyphen can be used to locate the duplicate strings.
86826
Objective: Merge the contents of two files using common key present in the files   file1.txt  =========  key1   11  key2   12  key3   13    file2.txt  =========  key2   22  key3   23  key4   24  key5   25    Expected Output :  ==================  key1   11  key2   12    22  key3   13    23   key4   24  key5   25   Approaches tried:   join command:  join -a 1 -a 2 file1.txt file2.txt ## full outer join  awk:  awk 'FNR==NR{a[$1]=$2;next;}{ print $0, a[$1]}' 2.txt 1.txt    Approach 2 is resulting in a right outer join and NOT a full outer join:     key1  11    key2  12    22    key3  13    23    What needs to be modified in approach 2 to result in a full outer join?
91941
$ cat data.txt  aaaaaa aaaaaa cccccc aaaaaa aaaaaa bbbbbb $ cat data.txt | uniq aaaaaa cccccc aaaaaa bbbbbb $ cat data.txt | sort | uniq aaaaaa bbbbbb cccccc $   The result that I need is to display all the lines from the original file removing all the duplicates (not just the consecutive ones), while maintaining the original order of statements in the file.  Here, in this example, the result that I actually was looking for was  aaaaaa cccccc bbbbbb   How can I perform this generalized uniq operation in general?
31301
I have some files and each files have distinct name but all of them have the same "-statement" at the end like this:  flower-statement.mp3 leave-statement.mp3  ball-statement.mp3 goal-statement.mp3   So I want to remove this: -statement  Is that possible?
9931
This one-liner removes duplicate lines from text input without pre-sorting.  For example:  $ cat &gt;f q w e w r $ awk '!a[$0]++' &lt;f q w e r $    The original code I have found on the internets read:  awk '!_[$0]++'  This was even more perplexing to me as I took _ to have a special meaning in awk, like in Perl, but it turned out to be just a name of an array.  Now, I understand the logic behind the one-liner:  each input line is used as a key in a hash array, thus, upon completion, the hash contains unique lines in the order of arrival.  What I would like to learn is how exactly this notation is interpreted by awk. E.g. what the bang sign (!) means and the other elements of this code snippet.  How does it work?
68515
How can I separate specific numbers from a column and add it to a new file? For example, take all numbers that are "134" from column 3, I tried using   awk '$3 == 134" file.txt   but it didn't work, also tried using   awk -F, '{print $3 == 134 }' file.txt    didn't work.  Pink          132           134 Blue          189           167 Yellow        2000          134 Green         678           145 Black         789           145 White         780           135   So the new file will contain just the following   Pink          132           134 Yellow        2000          134
85741
I have a large csv file with these format file 1:  id,ICD,CT 16214497,008.8,1 16227244,401.1,45 27154870,780.79,1   I have another txt file with one column of list of id (header id). file 2:  id 16214497 27154870   I want another file that has only data limited to the list of id in file 2  output file:  id,ICD,CT 16214497,008.8,1 27154870,780.79,1
43335
Given a file L with one non-negative integer per line and text file F, what would be a fast way to keep only those lines in F, whose line number appears in file L?  Example:  $ cat L.txt 1 3  $ cat F.txt Hello World Hallo Welt Hola mundo  $ command-in-question -x L.txt F.txt Hello World Hola mundo   I'm looking for a command that can handle a file L with 500 million or more entries; file L is sorted numerically.  Note: I'm halfway through an implementation for a command-in-question but I just wondered, whether one might be able to use some Unix tools here as well.    Update: Thank for all the answers, I learned a lot today! I would like to accept more one answer, but that's not possible.
94844
I have a file1.txt  USA Joe 123.123.123 Russia Marry 458.786.892 Canada Greg 151.844.165 Latvia Grace 125.895.688   and file2.txt  1 123.123.123 2 151.844.165 3 465.879.515   and I want to create a new file result.txt where I print my only those lines that adresses (xxx.xxx.xxx) are both  in file1 and file2 so my result should be  USA Joe 123.123.123 Canada Greg 151.844.165   I need to use awk, but how I need to use it for both files?
65056
Is there some better solution for printing unique lines other than a combination of sort and uniq?