A regular file contains data organized in columns, separated by space. Based on the value of the second field, I want to split that initial data file into several other files.  The data file has 54,242 rows and 399 columns. Data structure is so:   SNP Chr Pos ID_01 ID_02 ... rs01 1 200 AA BB ... rs02 25 10 BA AA ... rs03 1 30 AA BB ... rs04 X 90 BB AA ...   My output should consist of each record (or row) in as many file"${chr}" as there are different $chr values in column 2. So for example:  file1 will contain all the records whose 2nd field is 1:  SNP Chr Pos ID_01 ID_02 ... rs01 1 200 AA BB ... rs03 1 30 AA BB ...   file25 will contain all the records whose 2nd field is 25:   SNP Chr Pos ID_01 ID_02 ... rs02 25 10 BA AA ...   And so on...
62543
I have a .txt that can be exemplified like this:  NAME | CODE name1 | 001 name2 | 001 name3 | 002 name4 | 003 name5 | 003 name6 | 003   I need to write a script to split this file according to the CODE column, so in this case I'd get this:  file 1: NAME | CODE name1 | 001 name2 | 001  file 2: NAME | CODE name3 | 002  file 3: NAME | CODE name4 | 003 name5 | 003 name6 | 003   According to some research, using awk would work:  $ awk -F, '{print &gt; $2".txt"}' inputfile   The thing is, I also need to include the header to the first line and I need the file names to be different. Instead of 001.txt, for example, I need the file name to be something like FILE_$FILENAME_IDK.txt.
95955
I have a file that I would like to break up into multiple files with uniq values for the first column. For example, here is a file:  fileA.txt   1    Cat 1    Dog 1    Frog 2    Boy 2    Girl 3    Tree 3    Leaf 3    Branch 3    Trunk   I would like my output to look something like this:  file1.txt   1    Cat 2    Boy 3    Tree   file2.txt  1    Dog 2    Girl 3    Leaf   file3.txt  1    Frog 3    Branch   file4.txt  3    Trunk   If a value does not exist, I want it to be skipped. I have tried to search for similar situations to mine, but I've come up short. Does anyone have idea of how to do this?  Edit: My awk version is: awk version 20070501
90489
Data file is fixed length file, and I want to read the file record by record and do transformations to the subsequent records based on the prior records (and write the results into another file).  Example:  CTD1234abcdTRN0001 AA 5678defg BB 8910erty CTD5678qwerTRN5678 AA 9876bvcn BB 8765zxcv   Now I want TRN0001 from the CTD record to be written to the subsequent AA and BB records, similarly for the following CTD records. My output should look like:  CTD1234abcdTRN0001 AA 5678defgTRN0001 BB 8910ertyTRN0001 CTD5678qwerTRN5678 AA 9876bvcnTRN5678 BB 8765zxcvTRN5678   Example 2:  AAABBBB11115678xxxx BBBCCCC22221234YYYY MTD0001abcdTRN12345 abcdedfg AA 0002bcde BB 0003defg CCCDDDD33331234zzzz MTD0003qwerTRN56789 defghigk AA 9876bvcn BB 8765zxcv XXXBBBB11115678aaaa YYYCCCC22221234bbbb    should become  AAABBBB11115678xxxx BBBCCCC22221234YYYY MTD0001abcdTRN12345 abcdedfg AA 0002bcdeTRN12345 BB 0003defgTRN12345 CCCDDDD33331234zzzz MTD0003qwerTRN56789 defghigk AA 9876bvcnTRN56789 BB 8765zxcvTRN56789 XXXBBBB11115678aaaa YYYCCCC22221234bbbb    Can you please help me on this, how can you achieve this using UNIX shell scripting?
45653
I'm compiling a database of business address info, including key staff for each business. The Google spreadsheet I've inherited for this project includes a column for 'key staff' (in column B), with multiple key staff names listed in the same cell, separated by line breaks (ie CHAR(10) ). There is one business per row. The number of lines in the 'key staff' cell varies row-by-row.  My initial sheet looks like this:    I need to do the following to optimize this sheet:   split each multi-line ‘key staff’ cell so each key staff name appears on its own row. This requires that new row/s be inserted beneath the original row. duplicate the data from all other cells on the original row (i.e. from columns A &amp; C:E), so that each new row contains the full data for each business I need an automated process - I'll have about 1000 businesses to process so can't do this with any manual steps   The sheet should then look like this:    Using =TRANSPOSE(SPLIT(B1,CHAR(10))) obviously only goes part of the way - it doesn’t insert new rows and doesn’t duplicate the surrounding column entries. All help gratefully accepted!
101762
I have two files in these formats:   file1 :  air smell hand dude road cat   file 2:   air,4,21,01,13,3,2 smell,21,4,2,5,6 dude,1,31,42,1 road,1,4,2,1,4 cat,1,5,6,3,1 hand,1,4,2,1,6 mouse,1,3,5,6,2   what I want to do is print the entire row of file 2, if the first string in column 1 of file 2 is found in file 1, and I want to keep the order of file 1.   expected output:   air,4,21,01,13,3,2 smell,21,4,2,5,6 hand,1,4,2,1,6 dude,1,31,42,1 road,1,4,2,1,4 cat,1,5,6,3,1
36486
if I have a csv file in the following format:  column1,column2,column3,column4,column5,column6,column7,column8   and I want awk to only print columns 2 till 7 I would use:  awk -F',' '{print $2 "," $3 "," $4 "," $5 "," $6 "," $7}' file.csv   and get:  column2,column3,column4,column5,column6,column7   is there a way to concatenate the columns 2-7 to simplify the command. As I'm thinking of a file with quite a bit more columns, my awk command would get horribly long.
94149
I have a gene list file. Some thing like this      SWT21     SSA1     NRP1     EFB1     TFC3     MDM10   I have another file which also contains the names of these genes in my list along with other essential information about them. The second file looks like this:  chrI    147593  151166  YAL001C -   TFC3 chrI    143706  147531  YAL002W +   VPS8 chrI    142173  143160  YAL003W +   EFB1 chrI    140759  141407  YAL004W +   YAL004W chrI    139502  141431  YAL005C -   SSA1 chrI    137697  138345  YAL007C -   ERP2 chrI    136913  137510  YAL008W +   FUN14 chrI    135853  136633  YAL009W +   SPO7 chrI    134183  135665  YAL010C -   MDM10   I want to extract those lines in the 2nd file which have gene names as are present in first file.
28527
Consider the below file:   foo,boo,900 foo,boo,900 foo,boo,850   I need to compare the a field ($3) with the next record, if the difference is equal or more than 50, then print the record.   i.e from the sample above, $3 from second record - $3 from the third record = 50, then the output would be:   foo,boo,850
91459
I have file1 likes:  0   AFFX-SNP-000541  NA 0   AFFX-SNP-002255  NA 1   rs12103          0.6401 1   rs12103_1247494  0.696 1   rs12142199       0.7672   And a file2:  0   AFFX-SNP-000541   1 0   AFFX-SNP-002255   1 1   rs12103           0.5596 1   rs12103_1247494   0.5581 1   rs12142199        0.4931   And would like a file3 such that:  0   AFFX-SNP-000541     NA       1 0   AFFX-SNP-002255     NA       1 1   rs12103             0.6401   0.5596 1   rs12103_1247494     0.696    0.5581 1   rs12142199          0.7672   0.4931   Which means to put the 4th column of file2 to file1 by the name of the 2nd column.
76106
I have 3 csv files like this.  csv 1:  1,aaaa,bbb,2014-04-01 2,qwe,rty,2014-04-03 3,zxc,cvb,2014-04-05   csv 2:   2,j,k,2014-04-01 3,a,s,2014-04-04 5,g,h,2014-04-08   csv 3:  2,a,s,d,f,g,2014-04-01 3,d,f,g,h,j,2014-04-06 4,c,v,b,n,m,2014-04-09   How can I merge all by the first column?  SELECT * FROM csv1 JOIN csv2 where csv1[0]= csv2[0] --[0] is the position of the first column   The output should be:   csv1 fields     | csv2 fields |  csv4 fields   2,qwe,rty,2014-04-03,a,s,2014-04-04,a,s,d,f,g,2014-04-01  3,zxc,cvb,2014-04-05,g,h,2014-04-08,d,f,g,h,j,2014-04-06
48641
I need to split a .txt file into smaller ones containing 100 lines each, including the header. I don't know if this is relevant, but the original file is delimited like this:  COLUMN1 | COLUMN2 | COLUMN3 1 | 2 | 3 4 | 5 | 6 7 | 8 | 9   I need every file generated from this split to have the header line. Also, they need to be generated in/moved to another directory and follow a name pattern, like file_01.txt, file_02.txt, etc
50273
Is there a utility that split file by newline symbol? e.g if a file contains the following lines,  aa bbb cccc   If I want to split it to 3 files, the desired output would be:  aa, bbb And cccc (in 3 different files)  I already checked the split command, it only cut file by file sizes, not what I want.  If I don't wrote a utility myself, is there any standard tool to use?
90066
How to create a new file merging selective columns from two separate files using awk? Without messing up the elements orders of BOTH files.  Exemple: File 3 may contain column 1,2,3 from File 1 and column 4 from File 2.   File 1 A   23  8   T A   63  9   9 B   45  3   J  File 2 A   0 A   6    B   5  File 3 A   23  8   0 A   63  9   6 B   45  3   5
38558
I have a file with the lines like below.  title1:A1 title2:A2 title3:A3 title4:A4 title5:A5  title1:B1 title2:B2 title3:B3 title4:B4 title5:B5  title1:C1 title2:C2 title3:C3 title4:C4 title5:C5  title1:D1 title2:D2 title3:D3 title4:D4 title5:D5  How can I achieve this?   title1    title2     title3    title4 A1         A2         A3         A4 B1         B2         B3         B4 C1         C2         C3         C4 D1         D2         D3         D4
89240
How can I add another 2 values after matching the values from 2 columns in another file just like VLOOKUP?  Sample below. Value in column 6 &amp; 7 from file1, when matched with column 1 &amp; 2 from file2, will add column 9 &amp; 10 in file1 with the value of column 3 &amp; 4 from file2.  file1.txt  1 1 1 1 1 5 9 1  2 2 2 2 2 7 8 2  3 3 3 3 3 7 7 3  4 4 4 4 4 8 6 4   file2.txt  5 9 A B  8 6 E F  7 7 G H  7 8 C D   output.txt  1 1 1 1 1 5 9 1 A B  2 2 2 2 2 7 8 2 C D  3 3 3 3 3 7 7 3 G H  4 4 4 4 4 8 6 4 E F   Thanks,
12169
I have 100 files with 57,816 rows each.I want to do an outer join of these files by merging on basis of the common column.  I am using R programming for this but this is very slow.  fileList &lt;- list.files(, pattern=".txt") pm_list=lapply(fileList, read.table) merged_pm=merge_all(pm_list, by = "gene_short_name")   Is there any quick way to do this in bash? The other method that I can use is SQL but I have to create 100 tables first and then load them up before join,which is not very efficient way to do.  The number of rows are equal in each file, thats why I want to merge  based on common column and can't use cbind from R as the constants in the common column are a bit up and down in different files and not present at same location in each file.Below are two sample files.I want to join based on 'gene_short_name'  gene_short_name FPKM56  MT-TF   0.90 MT-TV   0 MT-RNR1 310.015 MT-TL1  0 MT-TM   0   File 2 is below:  gene_short_name FPKM53  MT-TF   0 MT-TV   0.344 MT-TM   0.10 MT-TL1  0 MT-RNR1 0 MT-ND2  158.332
100842
Is it possible to copy the whole rows of File1 in a new File3 following the instruction given by File2 by using a simple bash script using sed or awk?  File1: /*two or more columns*/ AC    456324 DC    689712 GH    123677 KL    236587  File2: /*one column*/ AC DC  File3:  AC    456324 DC    689712   I'm actually performing this using Python dictionaries and I wondered if you knew a simple way out.