I have this scenario: There is a huge text file around 500MB ,by whose contents I have to make reports.Each line have some values separated by a space.out of several values, two of which are URL and RESPONSETIME. If a url's response time is more that 8000ms I have to make a report of how many times overall that url was hit, and out of them how many times response time was greater than 8000ms, so final report will look something like this  URL total hits delayed response url1     100     5  url2     1000    18   I have done my part of search on google so don't suggest me that. Using a list will not be a solution because you can't modify the objects of a list while iterating through it. Anyone please suggest ideas.
78
We all know you can't do the following because of ConcurrentModificationException:  for (Object i : l) {     if (condition(i)) {         l.remove(i);     } }   But this apparently works sometimes, but not always. Here's some specific code:  public static void main(String[] args) {     Collection&lt;Integer&gt; l = new ArrayList&lt;&gt;();      for (int i = 0; i &lt; 10; ++i) {         l.add(4);         l.add(5);         l.add(6);     }      for (int i : l) {         if (i == 5) {             l.remove(i);         }     }      System.out.println(l); }   This, of course, results in:  Exception in thread "main" java.util.ConcurrentModificationException   Even though multiple threads aren't doing it. Anyway.  What's the best solution to this problem? How can I remove an item from the collection in a loop without throwing this exception?  I'm also using an arbitrary Collection here, not necessarily an ArrayList, so you can't rely on get.
11487
i have 8000 lines similar to this   &lt;Flow flowId="1" timeFirstTxPacket="+0.0ns" timeFirstRxPacket="+924100.0ns" timeLastTxPacket="+199984927558.0ns" timeLastRxPacket="+199675473275.0ns" delaySum="+2287566662167.0ns" jitterSum="+65280162191.0ns" lastDelay="+3511349483.0ns" txBytes="161956" rxBytes="116536" txPackets="3125" rxPackets="2251" lostPackets="640" timesForwarded="0"&gt;   and what i need to do is summing rxPackets in each line together and txPackets too   i mean total sum of them in 8000 lines  thanks in advance  &lt;Flow flowId="1" timeFirstTxPacket="+0.0ns" timeFirstRxPacket="+924100.0ns" timeLastTxPacket="+199984927558.0ns" timeLastRxPacket="+199675473275.0ns" delaySum="+2287566662167.0ns" jitterSum="+65280162191.0ns" lastDelay="+3511349483.0ns" txBytes="161956" rxBytes="116536" txPackets="3125" rxPackets="2251" lostPackets="640" timesForwarded="0"&gt;  &lt;Flow flowId="2" timeFirstTxPacket="+3000000.0ns" timeFirstRxPacket="+3254190.0ns" timeLastTxPacket="+199988489445.0ns" timeLastRxPacket="+199631899307.0ns" delaySum="+12081144328484.0ns" jitterSum="+105606015634.0ns" lastDelay="+9407845452.0ns" txBytes="251896" rxBytes="124840" txPackets="4867" rxPackets="2413" lostPackets="1757" timesForwarded="0"&gt;  &lt;Flow flowId="3" timeFirstTxPacket="+4000000.0ns" timeFirstRxPacket="+4254098.0ns" timeLastTxPacket="+199971756470.0ns" timeLastRxPacket="+198959480315.0ns" delaySum="+5150205600987.0ns" jitterSum="+74009967633.0ns" lastDelay="+9862217799.0ns" txBytes="163044" rxBytes="90532" txPackets="3150" rxPackets="1751" lostPackets="1013" timesForwarded="0"&gt;
16837
I've written a quick-and-dirty script to time some reports from a web service:  BASE_URL='http://example.com/json/webservice/' FIRST=1 FINAL=10000  for report_code in $(seq 1 $FINAL); do   (time -p response=$(curl --write-out %{http_code} --silent -O ${BASE_URL}/${report_code}) ) 2&gt; ${report_code}.time    echo $response  # &lt;------- this is out of scope!  How do I fix that?   if [[ $response = '404' ]]; then     echo "Deleting report # ${report_code}!"     rm ${report_code}   else     echo "${report_code} seems to be good!"   fi done   I need to wrap the time command in a subshell so I can redirect its output, but that makes the value of $response unavailable to the parent shell.  How do I get around this problem?
80374
If I have this string:     2+24*48/32   what is the most efficient approach for creating this list:     ['2', '+', '24', '*', '48', '/', '32']
13333
I need to display whole file before tracking it for a new changes, not only the last 10 lines (yep, I know it's not tail conceptually). In other words, something like cat -f would do, if it would ever existed. Tail's man doesn't give me any ideas. The only option I see right now is to somehow combine cat all but last 10 lines and tail -f output.  Any hints, please?
84017
I've looked around, and can't find any detailed explanation of the output at all. Most is indeed easy to understand, but there is one part that flummoxes me:  Time per request:       109537.505 [ms] (mean) Time per request:       109.538 [ms] (mean, across all concurrent requests)   This means - to me - that if you measure across all concurrent requests (whatever that means) then the requests are suddenly returned 100x faster. Why would the time per request change that much? This makes no sense.  Is there any detailed explanation of this and other parts of the ab output?
63486
Say you need to have a list/array of integers which you need iterate frequently, and I mean extremely often. The reasons may vary, but say it's in the heart of the inner most loop of a high volume processing.  In general, one would opt for using Lists (List) due to their flexibility in size. On top of that, msdn documentation claims Lists use an array internally and should perform just as fast (a quick look with Reflector confirms this). Neverless, there is some overhead involved.  Did anyone actually measure this? would iterating 6M times through a list take the same time as an array would?
11666
A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table).  What I need is to remove all the repetitions while (preferably, but this can be sacrificed for a significant performance boost) maintaining the original sequence order. In the result each line is to be unique. If there were 100 equal lines (usually the duplicates are spread across the file and won't be neighbours) there is to be only one of the kind left.  I have written a program in Scala (consider it Java if you don't know about Scala) to implement this. But maybe there are faster C-written native tools able to do this faster?  UPDATE: the awk '!seen[$0]++' filename solution seemed working just fine for me as long as the files were near 2 GiB or smaller but now as I am to clean-up a 8 GiB file it doesn't work any more. It seems taking infinity on a Mac with 4 GiB RAM and a 64-bit Windows 7 PC with 4 GiB RAM and 6 GiB swap just runs out of memory. And I don't feel enthusiastic about trying it on Linux with 4 GiB RAM given this experience.
17935
I have a file formatted like this:  train/t/temple/east_asia/00000025.jpg 94 train/t/temple/east_asia/00000865.jpg 94 ... train/s/swamp/00000560.jpg 92 train/s/swamp/00000935.jpg 92 .... train/m/mountain/00000428.jpg 68 train/m/mountain/00000126.jpg 68   The last number is the class number. I have 50 different classes, and each class has 1,000 lines. I would like to take a random sample of size N from each class, and store the result in another text file.
17551
I have a large file (8GB for Example). How can I split it into multiple parts, let's say 3 equal parts, and after that how do I integrate them later?
8318
I have a list like so.  blue-image1.jpg, blue-image2.jpg, blue-image3.jpg red-image1.jpg, red-image2.jpg, red-image3.jpg   It's was originally a csv format but I added newlines because they needed a prefix at first. Now I need to put them together again so it becomes like this.  blue-image1.jpg,blue-image2.jpg,blue-image3.jpg red-image1.jpg,red-image2.jpg,red-image3.jpg   I tried cat list.txt | tr ',\n' ',' and cat list.txt | sed 's/,\n/,/g'  but neither seem to work properly.
16713
shows how to prevent a long listing from being split across several pages when the code is written directly into the tex file. I want to achieve the same effect when including code in separate files using \lstinputlisting like so:  \lstinputlisting[label=lst:mylist,caption=A Caption]{code/parametrised.xml}   How do I modify the listing environment definition given in the above question to do so?  \lstnewenvironment{code}[1][]%   {\minipage{\linewidth}     \lstset{basicstyle=\ttfamily\footnotesize,frame=single,#1}}   {\endminipage}   Any help would be greatly appreciated!  Edit: The question has been rephrased for clarity - The answers here relate to preventing a listing from being split across pages by ensuring it's treated like a unbreakable float. Howvever, this also means that listings longer than a page will not be paginated and will run off the bottom of the page.
120559
I'm storing an ArrayList of Ids in a processing script that I want to spit out as a comma delimited list for output to the debug log. Is there a way I can get this easily without looping through things?  EDIT: Thanks to Joel for pointing out the List(Of T) that is available in .net 2.0 and above. That makes things TONS easier if you have it available.
48641
I need to split a .txt file into smaller ones containing 100 lines each, including the header. I don't know if this is relevant, but the original file is delimited like this:  COLUMN1 | COLUMN2 | COLUMN3 1 | 2 | 3 4 | 5 | 6 7 | 8 | 9   I need every file generated from this split to have the header line. Also, they need to be generated in/moved to another directory and follow a name pattern, like file_01.txt, file_02.txt, etc
20332
i just made a test, to see how much data i can put in a string before i reach the the heapsize limit and i get an unexpected behaviour   string  csvFile = '"XXXXXXX","XXPX XXRXXXX","XXPX BP","XLXXXXR","XXXXVXXX PRXXXDXXXX","XXPX XXXXVXXX","XDRXXXDXX LXGXLX","XXXXRX XXLX","LXXXX XX","DXXX XXXZXX XXXXVXXX","DXXX XXXX XXXXVXXX","XXDXXX BP","XXDXXX XXXXXLX","PXRXXXX XVX","DXXXXXXXZXXXX","XXXX XXRXXXXRX","XXGXXXX XXRXXXXRX","DXXX DX XXXXXXX","LXXGX DX XXXXXXX","PXD","XXXXX XXXXRXXXXXLX","XXXX RXXXPXXX","XXDXRXZZX RXXXPXXX","PRXVXXXXX RXXXPXXX","XXXXXX RXXXPXXX","XXP RXXXPXXX","XXXXRX XXLXXXXX 1","XXXXRX XXLXXXXX 2","XXXXRX XXX","XXDXRXZZX XXXXL","XXDXRXZZX XXRXXXXRX","XXXXXX XXRXXXXRX","PRXVXXXXX XXRXXXXRX","XXP XXRXXXXRX","XXXXX XLXXXXX","XXDXXX PRXXXXX","DXXX XXXXXZXXXX","XXPX DXXXXXXXX","XHXXVX XXXXXRX","DXXX XXXXXXXXX XXXXXRX","DXXX XXXDXXZX XXXXXRX","XXPXRXX XXXXLX","XXPXRXX RXXXDXX XXXXDXXX","PRXXXXXXXXXXXXXXZXXXX PRXXXXXXXXXX","DXXX PRXXXXXXXXXXXXXXZXXXX PRXXXXXXXXXX"';  string  csvFile2='',csvFile3; system.debug(limits.getLimitHeapSize()); system.debug(limits.getHeapSize()); for(integer i=0;i&lt;3618;i++){ csvFile2+='\n'+csvFile; }   system.debug(1.0*limits.getHeapSize()); system.debug((1.0*limits.getHeapSize())/limits.getLimitHeapSize()*100);   with 3618 iteraction i get    Heap size 3001245.0 Which correspond to ~50% of the heap size limit in the Synchronous context   but if i go to 3619 iteractions in the for, i get the error "Apex heap size too large: 6001368".... which surprise me a lot...   I'm not getting out why this behaviour is occurring. Do you have any similar experience? There is a reason for this?  Seems more about limits on strings, in effect the string csvFile2 overcome 3M characters... but i dont know
9019
What's a good way of extracting say, lines 20 -45 out of a huge text file. Non-interactively of course!
740
I have a list of arbitrary length, and I need to split it up into equal size chunks and operate on it. There are some obvious ways to do this, like keeping a counter and two lists, and when the second list fills up, add it to the first list and empty the second list for the next round of data, but this is potentially extremely expensive.  I was wondering if anyone had a good solution to this for lists of any length, e.g. using generators.  I was looking for something useful in itertools but I couldn't find anything obviously useful. Might've missed it, though.  Related question: