I have a huge csv file with 10 fields separated by commas. Unfortunately, some lines are malformed and do not contain exactly 10 commas (what causes some problems when I want to read the file into R). How can I filter out only the lines that contain exactly 10 commas?
110344
I am working on mac with sed, perl, awk, bash..  I have a large-ish (10GB) text file which has 13 fields (columns) of TAB delimited data. Unfortunately some of these lines have extraneous TABs , so I want to delete the entire line where we have extra TABs , and thus unequal fields.  (I don't mind discarding the lines in their entirety)   What I currently have writes the number of fields into another file.    awk -F'\t' '{print NF}' infile  &gt; fieldCount  head fieldCount 13 13 10 13 13 13 14 13 13 13   I would like to construct a short script that removes any line with more (or less) than 13 proper fields (from the original file).    speed is helpful as I have to do this on multiple files doing it in one sweep would be cool I currently am porting the fieldCount file into Python, trying to load with line by line.   EDIT:  vaild (13 columns)  a       b       c       d       e       f       g       h       i       j       k       l       m   invalid (14 columns)  a       b       c       d       e       f       g       h       i       j       k       l       m       n
65603
Though I try to avoid it, I occasionally have to open a CSV file in Excel. When I do, it formats columns containing numbers, which makes them useless for my purposes. As far as I can tell, the only way to prevent this from happening on import is to rename the file so the extension isn't .csv and use the import wizard to specify the format of each column individually. For files with 50-60 columns, this is impractical.  Since every answer for this oft-asked question on the internet suggests either some means of converting the formatted numbers back once the file is open (which won't work for me - I want to solve the general problem, not a few specific cases) or manually selecting the format type of each column (which I don't want to do), I'm looking for a way to set a global preference or style such that all columns of all CSV files opened are always formatted as text.  I know about "armoring" the numbers with quotes, too, but the files I get don't come like that and I was hoping to avoid having to pre-process the files so Excel doesn't screw them up.  Is there a way to do specifically this: Always format all columns in opened CSV files as text, without manually selecting each column every time during import?  I'm using Excel 2003, but I'll take answers for 2007 if that's what you know.
5365
Related to  I have a CSV file which contains a mixture of numeric and text fields: ID,Txt,Int,Dbl 12345,abc,45,56.78 12346,9,65,23.12 12347,10,66,23.13  According to this  if the top 8 rows contain a text value, the field will be classified as text, yet I am finding that the second field is classified as Long. This means that the value &quot;abc&quot; is omitted:  How can I force ArcMap to recognise this as a text field, while maintaining the data in CSV format? A workaround will be to create a geodatabase table, define the field as text, and Load the data, but I'm hoping to keep the table in CSV (it's the output of another process).
90307
I have a text file as follows:  test1,test2,test3 test4,test5 test6,test7,test8,test9,test10 test11,test12,test13,test14   How can I replace commas with semicolons, starting with the second one (and continuing to the last one)?  I want to get output as follows:  test1,test2;test3 test4,test5 test6,test7;test8;test9;test10 test11,test12;test13;test14
2320
We will generate a csv file with below values  yp1234,577,1,3 yp5678,577,3,5 yp9012,132,8,9   I need to extract data and create files based on second column. If it's 577 then the whole line has to be extracted and placed in a separate file. I mean I need a file having lines with second column as 577 alone and another file with second column as 132 alone  I tried using IF but didn't work
59495
I have a issue since I'm not familiar with awk. I have csv file generated from sar -d output converted to csv style:  12:33:41        unix,restarts 12:35:00,lofi4096,0,0.0,0,0,0.0,0.0 ,iscsi0,0,0.0,0,0,0.0,0.0 ,scsi_vhc,0,0.0,0,0,0.0,0.0 ,nfs1,0,0.0,0,0,0.0,0.0  12:45:00,lofi4096,0,0.0,0,0,0.0,0.0 ,iscsi0,0,0.0,0,0,0.0,0.0 ,scsi_vhc,0,0.0,0,0,0.0,0.0 ,nfs1,0,0.0,0,0,0.0,0.0   and I want to convert to this  12:33:41        unix,restarts 12:35:00,lofi4096,0,0.0,0,0,0.0,0.0 12:35:00,iscsi0,0,0.0,0,0,0.0,0.0 12:35:00,scsi_vhc,0,0.0,0,0,0.0,0.0 12:35:00,nfs1,0,0.0,0,0,0.0,0.0  12:45:00,lofi4096,0,0.0,0,0,0.0,0.0 12:45:00,iscsi0,0,0.0,0,0,0.0,0.0 12:45:00,scsi_vhc,0,0.0,0,0,0.0,0.0 12:45:00,nfs1,0,0.0,0,0,0.0,0.0   My attempt but since awk reads line by line I don't know out to keep the previous line value. I would expect that what I have should work. Any idea how I can obtain my desired result. I tried with awk, but I guess this should be possible with sed or the hard way with a custom shell script ( I'm trying to avoid doing this part ).  #!/usr/bin/awk -f BEGIN {         FS=","; } {         print $1         if ($1 != "") {                 mydate=$1;                 print $0;         }         else {                 print $mydate","$0;         } }   Running the system with Solaris 11.1.
114553
I have a file that looks like this toy example.  My actual file has 4 million lines, about 10 of which I need to delete.  ID  Data1  Data2 1    100    100 2    100    200 3    200    100 ID  Data1  Data2 4    100    100 ID  Data1  Data2 5    200    200   I want to delete the lines that look like the header, except for the first line.  Final file:  ID  Data1  Data2 1    100    100 2    100    200 3    200    100 4    100    100 5    200    200   How can I do this?
3494
Can you recommend a Java library for reading, parsing, validating and mapping rows in a comma separated value (CSV) file to Java value objects (JavaBeans)?
31151
I have a CSV file (db.txt)like this:  30.0,   0.0,    0.0 60.0,   1.9098, 5.8779 90.0,   6.9098, 9.5106 120.0,  13.09,  9.5106 150.0,  18.09,  5.8779 180.0,  20.0,   0.0   I need to typeset it. My problem is related with the wrong alignment of decimal point and the number of decimals (must be a constant for each column):  \renewcommand*{\dtlrealalign}{r}  \DTLloaddb[     noheader,     keys={x,y,theta},     headers={         \shortstack{$\boldsymbol{\theta_{2,i}}$},         \shortstack{X},         \shortstack{Y}              }   ]{db}{db.txt} \begin{table}[t]   \caption{Database file}   \centering   \DTLdisplaydb{db} \end{table}     Any advice, please?
53214
I am attempting to write a filter using something like sed or awk to do the following:   If a given pattern does not exist in the input, copy the entire input to the output If the pattern exists in the input, copy only the lines after the first occurrence to the output   This happens to be for a "git clean" filter, but that's probably not important. The important aspect is this needs to be implemented as a filter, because the input is provided on stdin.  I know how to use sed to delete lines up to a pattern, eg. 1,/pattern/d but that deletes the entire input if /pattern/ is not matched anywhere.  I can imagine writing a whole shell script that creates a temporary file, does a grep -q or something, and then decides how to process the input. I'd prefer to do this without messing around creating a temporary file, if possible. This needs to be efficient because git might call it frequently.
4618
I have a file containing two columns and 10 million rows. The first column contains many repeated values, but there is a distinct value in column 2. I want to remove the repeated rows and want to keep only one using awk. Note: the file is sorted with values in column 1. For example:    1.123 -4.0 2.234 -3.5 2.234 -3.1 2.234 -2.0 4.432 0.0 5.123 +0.2 8.654 +0.5 8.654 +0.8 8.654 +0.9 . . . .   Expected output  1.123 -4.0 2.234 -3.5 4.432 0.0 5.123 +0.2 8.654 +0.5 . . . .
55217
I have a file that has a list of paths like so:  "1"       "/user/bin/share" "2"       "/home/user/.local" "3"       "/root/"   Is there a way to extract just the paths? I dont want the numbers or quotation marks. How can I sed or grep the paths out of the file? What regex would be required for such a task?
35181
I have a big file and need to split into two files. Suppose in the first file the 1000 lines should be selected and put into another file and delete those lines in the first file.  I tried using split but it is creating multiple chunks.
116194
Possible Duplicate:          Compiling my document with many PGF Plots, each containing thousands of data points from a CSV file takes too long (minutes). I realize that it does not make sense to have this many data points -- were there, say, a tenth, it would be enough to recreate the same plot on paper.  One of my files, for example, consists of 216000 lines of such a format: 0.000000000000000000e+00 0.000000000000000000e+00  1.388888888888888888e-04 -2.182787284255027771e-11 Of course, that is far too much. I then read the file as:  \documentclass{scrartcl}  \usepackage{pgfplots} \pgfplotsset{compat=1.5.1}  \begin{document}  \begin{tikzpicture}     \begin{loglogaxis}[         \addplot[mark=*, color=red] file {Data/plotXYZ.dat};     \end{loglogaxis} \end{tikzpicture}  \end{document}   Is it possible for the package to take only every x-th line to speed up the compilation process? Or are there other ways to make it (significantly) faster?  I am running Ubuntu and a friend of mine recommended me to write an awk script that would go through a .csv file and retain only every x-th line. However, I have no clue how I would do that, as I have never used awk before. If this would turn out to be the option of choice, could someone help me setting up such a script?
6452
I have a CSV file, in.csv, which has an integer array column:  1,2,4,"{100,200,300}",,1 1,2,4,"{100,200,300,400,500}",,2 1,2,4,"{100,200,300,600.900,1200,1500}",1,3   I want to have an output file, out.csv, like:  1,2,4,,1 1,2,4,,2 1,2,4,1,3   I tried:  cut -d , -f4 -- complement in.csv &gt; out.csv   But it did not work out.  (I also have the fifth column which has value for third row but not for others).
79546
I have a .csv file (on a mac) that has a bunch of empty lines, e.g.:  "1", "2", "lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum   lorem ipsum ","2","3","4" "1", "2", "lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum   lorem ipsum ","2","3","4"   Which I want to convert to:  "1", "2", "lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum ","2","3","4" "1", "2", "lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum  lorem ipsum ","2","3","4"   I know there must be a one liner but I don't know awk or sed. Any tips greatly appreciated!
110365
I have a file contains around million number of lines. In the lines I have a field called transactionid, which has repetitive values. What I need to do is to count them distinctly.  No matter of how many times a value is repeated, it should be counted only once.