I want to write a data parser script. The example data is: name: John Doe description: AM email: john@doe.cc lastLogon: 999999999999999 status: active name: Jane Doe description: HR email: jane@doe.cc lastLogon: 8888888888 status: active ... name: Foo Bar description: XX email: foo@bar.cc status: inactive  The key-value pairs are always in the same order (name, description, email, lastLogon, status), but some of the fields may be missing. It is also not guaranteed that the first record is complete. The expected output is delimiter-separated (e.g. CSV) values: John Doe,AM,john@doe.cc,999999999999999,active Jane Doe,HR,jane@doe.cc,8888888888,active ... Foo Bar,XX,foo@bar.cc,n/a,inactive  My solution is by using a while read loop. The main part of my script: while read line; do     grep -q '^name:' &lt;&lt;&lt; &quot;$line&quot; &amp;&amp; status=''     case &quot;${line,,}&quot; in         name*) # capture value ;;         desc*) # capture value ;;         email*) # capture value ;;         last*) # capture value ;;         status*) # capture value ;;     esac      if test -n &quot;$status&quot;; then         printf '%s,%s,%s,%s,%s\n' &quot;${name:-n\a}&quot; ... etc ...         unset name ... etc ...     fi done &lt; input.txt  This works. But obviously, very slow. The execution time with 703 lines of data: real    0m37.195s user    0m2.844s sys     0m22.984s  I'm thinking about the awk approach but I'm not experienced enough using it.
121388
How can I get the details and transpose it to horizontal form? Every record ends after Couse. Couse will never be blank or null. Note: These four headers will be there for the below data: Name, City, Age, Couse If you see the second record, there isn't any &quot;Name&quot;: &quot;&quot; -&gt; missing so it should be null in place of that and the remaining will be appended after that with a pipe separated like this:     null                | Ors  | 11 | MB I have data like below in the demo.txt file &quot;Name&quot;:&quot;asxadadad  ,aaf dsf&quot; &quot;City&quot;:&quot;Mum&quot; &quot;Age&quot;:&quot;23&quot; &quot;Couse&quot;:&quot;BBS&quot; &quot;City&quot;:&quot;Ors&quot; &quot;Age&quot;:&quot;11&quot; &quot;Couse&quot;:&quot;MB&quot; &quot;Name&quot;:&quot;adad sf&quot; &quot;City&quot;:&quot;Kol&quot; &quot;Age&quot;:&quot;21&quot; &quot;Couse&quot;:&quot;BB&quot; &quot;Name&quot;:&quot;pqr&quot; &quot;Age&quot;:&quot;21&quot; &quot;Couse&quot;:&quot;NN&quot;  Expected Output: asxadadad  ,aaf dsf | Mum  | 23 | BBS null                | Ors  | 11 | MB adad sf             | Kol  | 21 | BB pqr                 | null | 21 | NN  I tried the below code: but not working my logic counter=0 var_0='Couse'   while read -r line     echo &quot;$line&quot;     counter=$(( counter + 1 ))     var_1=`echo &quot;$line&quot; | grep -oh &quot;Couse&quot;`     if [ $var_0 == $var_1 ]    then         head -$counter demo.txt &gt; temp.txt         sed -i '1,$counter' demo.txt         counter = 0    else         echo &quot;No thing to do&quot;    fi   done &lt; demo.txt
52191
If there's a "First World Problems" for scripting, this would be it.   I have the following code in a script I'm updating:  if [ $diffLines -eq 1 ]; then         dateLastChanged=$(stat --format '%y' /.bbdata | awk '{print $1" "$2}' | sed 's/\.[0-9]*//g')          mailx -r "Systems and Operations &lt;sysadmin@[redacted].edu&gt;" -s "Warning Stale BB Data" jadavis6@[redacted].edu &lt;&lt;EOI         Last Change: $dateLastChanged          This is an automated warning of stale data for the UNC-G Blackboard Snapshot process. EOI  else         echo "$diffLines have changed" fi   The script sends email without issues, but the mailx command is nested within an if statement so I appear to be left with two choices:    Put EOI on a new line and break indentation patterns or Keep with indentation but use something like an echo statement to get mailx to suck up my email.   I'm open to alternatives to heredoc, but if there's a way to get around this it's my preferred syntax.
26448
I'm having trouble parsing through a CSV file to add users.  I wrote a small script to add a list of 1000 users:  export IFS=","  cat users.csv | while read a b c d e f g; do useradd -d -u $a -U $d ; done   The CSV file is formatted as:  ps2946,male,Phil Harris,Recording,6508 Hollywood Blvd,Nov 16 1950,760 901 3537 mc3225,male,Mel Blanc,Radio,6385 Hollywood Blvd.,Oct 1 1983,626 647 5560 fn1029,male,Franklin Pangborn,Motion pictures,1500 Vine Street,Mar 10 1911,858 497 6108 we9602,undefined,Webb Pierce,Recording,1600 Vine Street,Sep 3 1988,559 595 6063   However upon running the script it iterates through the entire file repeating:  useradd: invalid home directory '-u'   I used export IFS to separate the words by comma and use them as the parameters for the useradd command.   I read the manual and used the value for the home directory, but I feel like my entire script is off.  Can someone please help me in properly formatting a script to add users from a CSV file?  For reference I used this thread as a base:
19174
I need to remove every line that has a value of 2 or less in the 8th field (column).   My data looks like this:  12-31   Airport 189 379 41  49.70946503 -124.91377258   2   2880    30.8 01-01   AlberniElementary   165 331 16  49.26100922 -124.80662537   4   5760    26.1 01-09   BamfieldMarine  161 323 23  48.83490372 -125.13572693   2   2875    27.4 01-10   BamfieldMarine  161 323 23  48.83490372 -125.13572693   3   3068    38.6   I understand that using awk I can strip off the values desired and print them to another file, and I understand that sed would edit the current file. In either case, I need to retain the original file.   Note: Please provide thorough explanations with your solutions. It is not suffice to just write the command, please annotate suggestions.   Further note: The data has a header line, so most likely solution will need to      awk 'FNR >1'   I suppose?
124264
I have below inputs with huge number of rows  11|ABCD|19900101123123445455|555|AAA|50505050|0000009030 11|ABCD|19900101123123445455|555|AAA|50505050|0000000199 13|ABCD|201803010YYY66666666|600|ETC|20180300|0000084099 11|ABCD|19900101123123445455|555|AAA|50505050|0008995001   And I need to get below output  11|ABCD|19900101123123445455|555|AAA|50505050|9004230 13|ABCD|201803010YYY66666666|600|ETC|20180300|84099   I have been trying with below awk but having too limited knowledge with arrays.  cat test|awk -F"|" '{ a[$1]++;b[$2]++;c[$3]++;d[$4]++;e[$5]++;f[$6]+=$6 }; END { for (i in a); print i, f[i]}'   I need to sum last column of column number 6 and print all first 5 columns, which are separated by pipe and last 6th column as sum of 6th column.
79103
I'm trying to extract all the IP address from a file.  FILE  *&gt;i85.95.2.160/29   172.16.24.4              0    200      0 64515 4445 15412 i *&gt;i85.112.80.0/28   172.16.24.4              0    200      0 6774 64521 i *&gt;i85.112.80.96/27  172.16.24.4              0    200      0 6774 64521 i *&gt;i85.115.64.96/27  172.16.24.4              0    200      0 2300 64961 i   OUTPUT  85.95.2.160 85.112.80.0 85.112.80.96 85.115.64.96   This is what I have so far:  $ awk &lt; ips '{ print $1 }' grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' ips | sort | uniq   The problem is that from the list of IP addresses it goes from the beginning up to:  84.241.254.8/29   It does not extract any of the following ip addresses:  84.241.254.32 85.115.240.0 91.232.100.72 103.248.251.48 121.58.160.79 160.218.26.16 200.6.224.96 223.255.231.240   Any ideas why these aren't getting extracted?
18388
I am trying to extract a value from a long string that may change over time. So for example the string could look something like this  ....../filename-1.9.0.3.tar.gz"&lt;....   And what I want to extract is the value between filename- and .tar.gz, essentially the file version (1.9.0.3 in this case). The reason I need to do it this way is because I may later run the command and the value will be 1.9.0.6 or 2.0.0.2 or something entirely different.  How can I do this? I'm currently only using grep, but I wouldn't mind using other utilities such as sed or awk or cut or whatever. To be perfectly clear, I need to extract only the file version part of the string, since it is very long (on both sides) everything else needs to be cut out somehow.
88107
I have a CSV file users.csv with a list of userNames, userIDs, and other data:  username, userid, sidebar_side, sidebar_colour "John Lennon", 90123412, "left", "blue" "Paul McCartny", 30923833, "left", "black" "Ringo Starr", 77392318, "right", "blue" "George Harrison", 72349482, "left", "green"   In another file toremove.txt I have a list of userIDs:  30923833 77392318   Is there a clever, efficient way to remove all the rows from the users.csv file which contain the IDs in toremove.txt? I have written a simple Python app to parse the two files and write to a new file only those lines that are not found in toremove.txt, but it is extraordinarily slow. Perhaps some sed or awk magic can help here?  This is the desired result, considering the examples above:  username, userid, sidebar_side, sidebar_colour "John Lennon", 90123412, "left", "blue" "George Harrison", 72349482, "left", "green"
38760
I have a comma-separated list of data points. They should be plotted as a scatter plot, and should be labeled using data from the .csv file. Having specified the first column of my file as string type, I still keep running into errors saying Could not parse input [...] as a floating point number, sorry.  Given a CSV file of the following format, how can I read Name and Z as labels?  Name,X,Y,Z ODIN,250,45,2001   Here is my MWE:  \documentclass{article}  \usepackage{tikz,pgfplots,pgfplotstable}     \pgfplotsset{compat=1.5.1}  \begin{document}  \pgfplotstabletypeset[columns/Name/.style={string type}]{table.csv};  \begin{tikzpicture}     \begin{axis}[     visualization depends on=\thisrow{Name} \as \labela,     visualization depends on=\thisrow{Z} \as \labelb,     every node near coord/.append style={font={\tiny}},     nodes near coords=\labela/\labelb,     nodes near coords align={horizontal},]      \addplot[only marks] table [x=X,y=Y,col sep=comma] {table.csv};      \end{axis} \end{tikzpicture}  \end{document}   Note that if the data in column Name is just floats it should work fine.
121409
My script (should) acts differently, depending on the presence of the data in the input stream. So I can invoke it like this:  $ my-script.sh   or:  $ my-script.sh &lt;&lt;-MARK     Data comes...     ...data goes. MARK   or:  $ some-command | my-script.sh   where two last cases should read the data, while first case should notice the data is missing and act accordingly.  The crucial part (excerpt) of the script is:  #!/bin/bash local myData; read -d '' -t 0 myData;  [ -z "${myData}" ] &amp;&amp; {     # Notice the lack of the data. } || {     # Process the data. }   I use read to read input data, then option -d '' to read multiple lines, as this is expected, and the -t 0 to set timeout to zero. Why the timeout? According to  (typing left unchanged; bold is mine):     -t timeout time out and return failure if a complete line of input is               not read withint TIMEOUT seconds.  The value of the TMOUT               variable is the default timeout.  TIMEOUT may be a               fractional number.  If TIMEOUT is 0, read returns success only               if input is available on the specified file descriptor.  The               exit status is greater than 128 if the timeout is exceeded   So I in case 2 and 3 it should read the data immediately, as I understand it. Unfortunately it doesn't. As -t can take fractional values (according to above man page), changing read line to:  read -d '' -t 0.01 myData;   actually reads the data when data is present and skips it (after 10ms timeout) if it is not. But it should also work when TIMEOUT is set to real 0.  Why it actually doesn't? How can this be fixed? And is there, perhaps, alternative solution to the problem of "act differently depending on the presence of the data"?  UPDATE  Thanks to @Isaac I found a misleading discrepancy between quoted on-line version and my local one (normally I do not have locale set to en_US, so help read gave me translation which I couldn't paste here, and looking up for on-line translation was faster than setting new env---but that caused the whole problem).  So for 4.4.12 version of Bash it says:     If TIMEOUT is 0, read returns               immediately, without trying to read any data, returning               success only if input is available on the specified               file descriptor.   This gives a little bit different impression than "If TIMEOUT is 0, read returns success only if input is available on the specified file descriptor"---for me it implied actually an attempt to read the data.  So finally I tested this and it worked perfectly:  read -t 0 &amp;&amp; read -d '' myData;   The meaning: see if there's anything to read and if it succeed, just read it.  So as to base question, the correct answer was provided by Isaac. And as to alternative solution I prefer the above "read &amp;&amp; read" method.
114183
I have file with content like below and want to convert my output like below   Input  1,a,b,c 2,b,c 3,e,f 4,l  Required output  1,a 1,b 1,c 2,b 2,c 3,e 3,f 4,l    Values on first field is unique and no duplicate lines for 1st field in the input.  I am new to scripting and not sure how can we do this.
25290
I have a script as below:  #!/bin/bash df -k | tr -s " " "," | awk 'BEGIN {FS=","} {print $1,$5}'|sed 1d &gt; file1.txt while read partition percentUsed do if [ $percentUsed &gt; 75 ] then echo Partition: ${partition} space is ${percentUsed} else echo Pration: $partition: OK!! fi done &lt; file1.txt   The script is executing properly, however, it is creating a zero byte file by the name 75. Any ideas why this is happening?  $ sh diskUsed.sh Partition: C:/Users/Public/Documents/CYGWIN/bin space is 75% Partition: C:/Users/Public/Documents/CYGWIN/lib space is 75% Partition: C:/Users/Public/Documents/CYGWIN space is 75% Partition: C: space is 75% Partition: H: space is 91%   $ ls -lrt total 2 -rwxrwxrwx 1 diwvibh Domain Users 284 Nov 21 04:17 diskUsed.sh -rw-r--r-- 1 diwvibh Domain Users 133 Nov 21 04:29 file1.txt -rw-r--r-- 1 diwvibh Domain Users   0 Nov 21 04:29 75
110344
I am working on mac with sed, perl, awk, bash..  I have a large-ish (10GB) text file which has 13 fields (columns) of TAB delimited data. Unfortunately some of these lines have extraneous TABs , so I want to delete the entire line where we have extra TABs , and thus unequal fields.  (I don't mind discarding the lines in their entirety)   What I currently have writes the number of fields into another file.    awk -F'\t' '{print NF}' infile  &gt; fieldCount  head fieldCount 13 13 10 13 13 13 14 13 13 13   I would like to construct a short script that removes any line with more (or less) than 13 proper fields (from the original file).    speed is helpful as I have to do this on multiple files doing it in one sweep would be cool I currently am porting the fieldCount file into Python, trying to load with line by line.   EDIT:  vaild (13 columns)  a       b       c       d       e       f       g       h       i       j       k       l       m   invalid (14 columns)  a       b       c       d       e       f       g       h       i       j       k       l       m       n
60487
I have this input:  sdkxyosl 1 safkls 2 asdf--asdfasxy_asd 5 dkd8k  jasd 29 sdi44sw 43 asasd afsdfs 10 rklyasd 4   I need this output:  sdi44sw 43 dkd8k  jasd 29 asasd afsdfs 10 asdf--asdfasxy_asd 5 rklyasd 4 safkls 2 sdkxyosl 1   So i need to sort the lines by the last column.  I don't know how many columns are in one line.  I just can't figure it out, how to do it. I don't have "perl powers". I just have ~average scripting powers with sed, awk, cut, etc..   Does somebody know how to do it?
16555
I'm wondering if there is any sed or awk command that can sum all columns for the rows with an identical identifier. For example, my file data.txt is set out as below, except that it contains ~1800 columns of data and ~1400 rows.  ABCD:1234  1.23  0.23  0.83   0 ABCD:1234    0    1.10  0.21   0  EFGH:5678    0    1.90  0.12  8.21 IJKL:9999    1.22  0    1.84  9.21 IJKL:9999    1.44  0   12.94   0 IJKL:9999    1.32  0   24.12   2.43   And how I would like it to be after the command would be like this:  ABCD:1234  1.23  1.33  1.04  0 EFGH:5678    0    1.90  0.12  8.21 IJKL:9999   3.98   0   38.9  11.64   I'm not sure if this is even possible with awk or sed (I'm a biologist and still learning the basics of Unix). Any help would be much appreciated.
94435
I have some Java executable (jar) that is run my some shell script from a cron job once every night. That executable does not print log statements &quot;as usual&quot; just by printing them out in a sequential manner like line after line (print after print), but while it processes its data its printing a single line with status data and then &quot;overwrite&quot; or &quot;update&quot; just that single line over and over again, until its done with this part of processing. Examples: A common program would output something line this: Program XYZ started..&lt;cr+lf&gt; processing 1&lt;cr+lf&gt; processing 2&lt;cr+lf&gt; processing 3&lt;cr+lf&gt; im done&lt;cr+lf&gt;  Simple! I can easily redirect its output to a file and I'm done. But the program I will have to deal with does it more like this: Program XYZ started..&lt;cr+lf&gt; processing 1&lt;cr&gt; processing 1 - part a&lt;cr&gt; processing 1 - part b&lt;cr&gt; processing 1 - part c&lt;cr&gt; processing 1 - part d&lt;cr&gt; &lt;cr+lf&gt; processing 2&lt;cr&gt; processing 2 - part a&lt;cr&gt; processing 2 - part b&lt;cr&gt; processing 2 - part c&lt;cr&gt; processing 2 - part d&lt;cr&gt; &lt;cr+lf&gt; processing 3&lt;cr+lf&gt; im done&lt;cr+lf&gt;  but much more intensive than I showed here. So it just not always overwrite status lines, but does it a lot while processing. So just redirecting or appending its output to a log file does not seem to make a lot of sense here, since it will just clutter the log file with millions of screen-updates that can not be viewed and understood that easily by humans. So my question is: is there a way or tool for programs that may behave like that to &quot;log&quot; or maybe &quot;screenshot every 30 seconds&quot; their output to a log file?
65345
Suppose I have a file (call it sample.txt) that looks like this:  Row1,10 Row2,20 Row3,30 Row4,40   I want to be able to work on a stream from this file that is essentially the pairwise combination of all four rows (so we should end up with 16 in total). For instance, I'm looking for a streaming (ie efficient) command where the output is:  Row1,10 Row1,10 Row1,10 Row2,20 Row1,10 Row3,30 Row1,10 Row4,40 Row2,20 Row1,10 Row1,20 Row2,20 ... Row4,40 Row4,40   My use case is that I want to stream this output into another command (like awk) to compute some metric about this pairwise combination.  I have a way to do this in awk but my concern is that my use of the END{} block means that I'm basically storing the entire file in memory before I output. Example code:  awk '{arr[$1]=$1} END{for (a in arr){ for (a2 in arr) { print arr[a] " " arr[a2]}}}' samples/rows.txt  Row3,30 Row3,30 Row3,30 Row4,40 Row3,30 Row1,10 Row3,30 Row2,20 Row4,40 Row3,30 Row4,40 Row4,40 Row4,40 Row1,10 Row4,40 Row2,20 Row1,10 Row3,30 Row1,10 Row4,40 Row1,10 Row1,10 Row1,10 Row2,20 Row2,20 Row3,30 Row2,20 Row4,40 Row2,20 Row1,10 Row2,20 Row2,20   Is there an efficient streaming way to do this without having to essentially store the file in memory and then output in the END block?