I hava 10 files, each file is 500M.    I use Long randomValue = Math.abs(random.nextLong()); to generate those files. Every line in the file is a randomValue. And a file has about 26374000 lines.    When I read those files one by one in another program and count cost time, I find the input become more faster. Why?   read 0 file 26373542 lines time : 27046ms read 1 file 26373627 lines time : 24155ms read 2 file 26373676 lines time : 19227ms read 3 file 26373768 lines time : 22875ms read 4 file 26373681 lines time : 20813ms read 5 file 26373774 lines time : 18297ms read 6 file 26373787 lines time : 10556ms read 7 file 26373557 lines time : 11614ms read 8 file 26373566 lines time : 9751ms read 9 file 26373653 lines time : 13372ms   This is my program:    Long start = new Date().getTime(); FileReader fr = new FileReader(inFile); BufferedReader br = new BufferedReader(fr); String num; while((num = br.readLine()) != null) {     sorted[j++] = Long.parseLong(num);     count++; } Long end = new Date().getTime(); System.out.println("read " + i + " file " + j + "lines time : " + (mid - start) + "ms"); br.close(); fr.close();
915
How do you write (and run) a correct micro-benchmark in Java?  I'm looking for some code samples and comments illustrating various things to think about.  Example: Should the benchmark measure time/iteration or iterations/time, and why?  Related:
106050
Hi I'm pretty new to Stack Overflow so I hope that I'm doing this correctly and that someone out there has the answer I need.  I'm currently coding a program in Java with Eclipse IDE an my question is this:  I need a snippet of code that does the following  It's supposed to get a .TXT file containing text and from that .TXT file  count the number of rows and print it,  count the number of words and print it,  count the number of characters and print it. And finally make a list of the top 10 words used and print that.  Allt the printing is done to system outprintln  I'm pretty new to Java and am having some difficulties.  Anyone out there who can provide me with these lines of code or that knows where i can find them? I want to study the code provided that's how I learn best=)  Thanks to all   Didnt find the edit button sorry...  I Added this to my question:  Hehe it´s an assignment but not a homework assignment ok i see well i could provide what i've done so far, i think im pretty close but it´s not working for me. Is there anything i have missed?  // Class    Tip   import java.io.*; import java.util.*;  class Tip {     public static void main(String [] args) throws Exception     {          String root = System.getProperty("user.dir");            InputStream is = new FileInputStream( root + "\\tip.txt" );         Scanner scan = new Scanner( is );          String tempString = "";         int lines = 0;         int words = 0;         Vector&lt;Integer&gt; wordLength = new Vector&lt;Integer&gt;();         int avarageWordLength = 0;          while(scan.hasNextLine() == true)         {                 tempString = scan.nextLine();                 lines++;         }          is.close();          is = new FileInputStream( root );         scan = new Scanner( is );          while(scan.hasNext() == true)         {                 tempString = scan.next();                 wordLength.add(tempString.length());                 words++;         }          for(Integer i : wordLength)         {                 avarageWordLength += i;         }         avarageWordLength /= wordLength.size();           System.out.println("Lines : " + lines);         System.out.println("Words : " + words);         System.out.println("Words Avarage Length : " + avarageWordLength);          is.close();          } }
39750
I've copied a large file to a USB disk mounted on a Linux system with async. This returns to a command prompt relatively quickly, but when I type sync, of course, it all has to go to disk, and that takes a long time.  I understand that it's going to be slow, but is there somewhere where I can watch a counter go down to zero? Watching buffers in top doesn't help.
17935
I have a file formatted like this:  train/t/temple/east_asia/00000025.jpg 94 train/t/temple/east_asia/00000865.jpg 94 ... train/s/swamp/00000560.jpg 92 train/s/swamp/00000935.jpg 92 .... train/m/mountain/00000428.jpg 68 train/m/mountain/00000126.jpg 68   The last number is the class number. I have 50 different classes, and each class has 1,000 lines. I would like to take a random sample of size N from each class, and store the result in another text file.
67491
System.IO.Directory.GetFiles() returns a string[].  What is the default sort order for the returned values?  I'm assuming by name, but if so how much does the current culture effect it?  Can you change it to something like creation date?   Update: MSDN points out that the sort order is not guaranteed for .Net 3.5, but the 2.0 version of the page doesn't say anything at all and neither page will help you sort by things like creation or modification time.  That information is lost once you have the array (it contains only strings).  I could build a comparer that would check for each file it gets, but that means accessing the file system repeatedly when presumably the .GetFiles() method already does this.  Seems very inefficient.
9898
Code like this often happens: l = [] while foo:     # baz     l.append(bar)     # qux  This is really slow if you're about to append thousands of elements to your list, as the list will have to be constantly resized to fit the new elements. In Java, you can create an ArrayList with an initial capacity. If you have some idea how big your list will be, this will be a lot more efficient. I understand that code like this can often be refactored into a list comprehension. If the for/while loop is very complicated, though, this is unfeasible. Is there an equivalent for us Python programmers?
88347
I'm trying to update the timestamp to the current time on all of the xml files in my directory (recursively).  I'm using Mac OSX 10.8.5.  On about 300,000 files, the following echo command takes 10 seconds:  for file in `find . -name "*.xml"`; do echo &gt;&gt; $file; done   However, the following touch command takes 10 minutes! :   for file in `find . -name "*.xml"`; do touch $file; done   Why is echo so much faster than touch here?
26737
I'm wondering why Nautilus is very slow when opening a directory containing lots of files. My /usr/lib dir for example has 1900 files and it takes approximately 5+ seconds to show everything. It has been like this since I installed Ubuntu few months ago and it's really quite annoying sometimes. I don't have powerful hardware but I know that Windows Explorer is so much faster than this.   Is there anything that can be done to speed it up?  Ubuntu 10.04
92456
Alright, I have been doing the following (variable names have been changed):   FileInputStream fis = null; try {     fis = new FileInputStream(file);      ... process ...  } catch (IOException e) {     ... handle error ... } finally {     if (fis != null)         fis.close(); }    Recently, I started using FindBugs, which suggests that I am not properly closing streams.  I decide to see if there's anything that can be done with a finally{} block, and then I see, oh yeah, close() can throw IOException.  What are people supposed to do here?  The Java libraries throw too many checked exceptions.
88829
I want to write a program for this: In a folder I have n number of files; first read one file and perform some operation then store result in a separate file. Then read 2nd file, perform operation again and save result in new 2nd file. Do the same procedure for n number of files. The program reads all files one by one and stores results of each file separately. Please give examples how I can do it.
77625
So I have a program written in C++.  It can tell me how long it took to do all the calculations and it does a lot of quite heavy, multi-threaded calculation.   I've just noticed that if I run the program on the exact same machine, it takes around 20-21 seconds to do all the calculations if started from the TTY, and only about 0.2 seconds if I start it from GNOME terminal.  What is causing that? It's literally the exact same file on the same machine.
11442
I have an application that writes information to file. This information is used post-execution to determine pass/failure/correctness of the application. I'd like to be able to read the file as it is being written so that I can do these pass/failure/correctness checks in real time.  I assume it is possible to do this, but what are the gotcha's involved when using Java? If the reading catches up to the writing, will it just wait for more writes up until the file is closed, or will the read throw an exception at this point? If the latter, what do I do then?   My intuition is currently pushing me towards BufferedStreams. Is this the way to go?
121409
My script (should) acts differently, depending on the presence of the data in the input stream. So I can invoke it like this:  $ my-script.sh   or:  $ my-script.sh &lt;&lt;-MARK     Data comes...     ...data goes. MARK   or:  $ some-command | my-script.sh   where two last cases should read the data, while first case should notice the data is missing and act accordingly.  The crucial part (excerpt) of the script is:  #!/bin/bash local myData; read -d '' -t 0 myData;  [ -z "${myData}" ] &amp;&amp; {     # Notice the lack of the data. } || {     # Process the data. }   I use read to read input data, then option -d '' to read multiple lines, as this is expected, and the -t 0 to set timeout to zero. Why the timeout? According to  (typing left unchanged; bold is mine):     -t timeout time out and return failure if a complete line of input is               not read withint TIMEOUT seconds.  The value of the TMOUT               variable is the default timeout.  TIMEOUT may be a               fractional number.  If TIMEOUT is 0, read returns success only               if input is available on the specified file descriptor.  The               exit status is greater than 128 if the timeout is exceeded   So I in case 2 and 3 it should read the data immediately, as I understand it. Unfortunately it doesn't. As -t can take fractional values (according to above man page), changing read line to:  read -d '' -t 0.01 myData;   actually reads the data when data is present and skips it (after 10ms timeout) if it is not. But it should also work when TIMEOUT is set to real 0.  Why it actually doesn't? How can this be fixed? And is there, perhaps, alternative solution to the problem of "act differently depending on the presence of the data"?  UPDATE  Thanks to @Isaac I found a misleading discrepancy between quoted on-line version and my local one (normally I do not have locale set to en_US, so help read gave me translation which I couldn't paste here, and looking up for on-line translation was faster than setting new env---but that caused the whole problem).  So for 4.4.12 version of Bash it says:     If TIMEOUT is 0, read returns               immediately, without trying to read any data, returning               success only if input is available on the specified               file descriptor.   This gives a little bit different impression than "If TIMEOUT is 0, read returns success only if input is available on the specified file descriptor"---for me it implied actually an attempt to read the data.  So finally I tested this and it worked perfectly:  read -t 0 &amp;&amp; read -d '' myData;   The meaning: see if there's anything to read and if it succeed, just read it.  So as to base question, the correct answer was provided by Isaac. And as to alternative solution I prefer the above "read &amp;&amp; read" method.
55005
What would be the best way to list 100 first files in a directory sorted by the created timestamp (oldest first). The directory is quite large (around 100000 files). ls piped to head takes an awful long time to complete.   Edit:  Filesystem is ext3. limiting the number of files in the folder is not worth the effort, since this is going to be a rare "clean-up" operation and files are generated by a 3rd party software.   Using file modified time, instead of the creation time provides and acceptable solution.
78992
I have several files that I ftp each hour.  The receiving system needs to have some sort of identifier that they came from the same batch, so I would like to append a timestamp to the filename.  That alone is fairly easy, but since I want each file to have the same timestamp (so it can serve as the batch identifier), I cannot figure out how to do this.  So, I have:  file1.txt file2.txt file3.txt   And I want to have:  file1_20141110184303.txt file2_20141110184303.txt file3_20141110184303.txt   Any use of the date +%Y%m%d%H%M%S after the first use will obviously result in a different values in seconds, so I would like to have the first timestamp appended to the remaining files.
116194
Possible Duplicate:          Compiling my document with many PGF Plots, each containing thousands of data points from a CSV file takes too long (minutes). I realize that it does not make sense to have this many data points -- were there, say, a tenth, it would be enough to recreate the same plot on paper.  One of my files, for example, consists of 216000 lines of such a format: 0.000000000000000000e+00 0.000000000000000000e+00  1.388888888888888888e-04 -2.182787284255027771e-11 Of course, that is far too much. I then read the file as:  \documentclass{scrartcl}  \usepackage{pgfplots} \pgfplotsset{compat=1.5.1}  \begin{document}  \begin{tikzpicture}     \begin{loglogaxis}[         \addplot[mark=*, color=red] file {Data/plotXYZ.dat};     \end{loglogaxis} \end{tikzpicture}  \end{document}   Is it possible for the package to take only every x-th line to speed up the compilation process? Or are there other ways to make it (significantly) faster?  I am running Ubuntu and a friend of mine recommended me to write an awk script that would go through a .csv file and retain only every x-th line. However, I have no clue how I would do that, as I have never used awk before. If this would turn out to be the option of choice, could someone help me setting up such a script?
107537
Given the table:      Column    |            Type               id           | integer                       latitude     | numeric(9,6)                  longitude    | numeric(9,6)                  speed        | integer                       equipment_id | integer                       created_at   | timestamp without time zone Indexes:     "geoposition_records_pkey" PRIMARY KEY, btree (id)   The table has 20 million records that is not, relatively speaking, a large number. But it makes sequential scans slow.  How can I get the last record (max(created_at))  of each equipment_id?   I've tried both the following queries, with several variants that I've read through many answers of this topic:  select max(created_at),equipment_id from geoposition_records group by equipment_id;  select distinct on (equipment_id) equipment_id,created_at    from geoposition_records order by equipment_id, created_at desc;   I have also tried creating btree indexes for equipment_id,created_at but Postgres finds that using a seqscan is faster. Forcing enable_seqscan = off is of no use either since reading the index is as slow as the seq scan, probably worse.  The query must run periodically returning always the last.  Using Postgres 9.3.  Explain/analyze (with 1.7 million records):  set enable_seqscan=true; explain analyze select max(created_at),equipment_id from geoposition_records group by equipment_id; "HashAggregate  (cost=47803.77..47804.34 rows=57 width=12) (actual time=1935.536..1935.556 rows=58 loops=1)" "  -&gt;  Seq Scan on geoposition_records  (cost=0.00..39544.51 rows=1651851 width=12) (actual time=0.029..494.296 rows=1651851 loops=1)" "Total runtime: 1935.632 ms"  set enable_seqscan=false; explain analyze select max(created_at),equipment_id from geoposition_records group by equipment_id; "GroupAggregate  (cost=0.00..2995933.57 rows=57 width=12) (actual time=222.034..11305.073 rows=58 loops=1)" "  -&gt;  Index Scan using geoposition_records_equipment_id_created_at_idx on geoposition_records  (cost=0.00..2987673.75 rows=1651851 width=12) (actual time=0.062..10248.703 rows=1651851 loops=1)" "Total runtime: 11305.161 ms"