I'm looking to implement a similar program as tail in my python program.  I can listen to a named pipe via Unix:  tail -f mypipe  This works great (I can echo strings to said pipe), but I'd like to do it in python. I'd like to set the data coming through the pipe to a variable (it's preferable for a var to get updated with the last datum sent over the pipe).  So far I have:  cwd = os.getcwd() testpath = os.path.join(cwd, 'mypipe')  if os.path.exists(testpath):     os.remove(testpath)     mkfifo = os.mkfifo(testpath, 0644)     rp = open(testpath, 'r')     response = rp.read()     print 'response is ', response   There are two issues:   response reads once, but I don't know how I would read the stream upon data being received (something like a listener) I seem to be running into a blocking issue whereby the pipe is claimed and will not allow other programs to access it.  I don't totally understand accessing fifo pipes without blocks and am having a hard time w/ the docs for this.   I've attempted something like what is found  (the highest upvoted answer), but it doesn't seem to work for me.  I get an error:  IOError: [Errno 29] Illegal seek
46341
I'm writing a log file viewer for a web application and for that I want to paginate through the lines of the log file.  The items in the file are line based with the newest item at the bottom. So I need a tail() method that can read n lines from the bottom and support an offset.  This is hat I came up with: def tail(f, n, offset=0):     &quot;&quot;&quot;Reads a n lines from f with an offset of offset lines.&quot;&quot;&quot;     avg_line_length = 74     to_read = n + offset     while 1:         try:             f.seek(-(avg_line_length * to_read), 2)         except IOError:             # woops.  apparently file is smaller than what we want             # to step back, go to the beginning instead             f.seek(0)         pos = f.tell()         lines = f.read().splitlines()         if len(lines) &gt;= to_read or pos == 0:             return lines[-to_read:offset and -offset or None]         avg_line_length *= 1.3  Is this a reasonable approach?  What is the recommended way to tail log files with offsets?
54085
I have explored almost    , to no avail.  Let me describe the problem in detail:  I run some unattended scripts and these can produce standard output and standard error lines, I want to capture them in their precise order as displayed by a terminal emulator and then add a prefix like "STDERR: " and "STDOUT: " to them.  I have tried using pipes and even epoll-based approach on them, to no avail. I think solution is in pty usage, although I am no master at that. I have also peeked into the source code of , but that has not been much productive.   Ideally I would use  instead of Bash to accomplish this, but I have not been able to. Seems like pipes automatically forbid keeping a correct lines order because of buffering.  Has somebody been able to do something similar? Or it is just impossible? I think that if a terminal emulator can do it, then it's not - maybe by creating a small C program handling the PTY(s) differently?  Ideally I would use asynchronous input to read these 2 streams (STDOUT and STDERR) and then re-print them second my needs, but order of input is crucial!  NOTE: I am aware of  but it does not work for me with Bash scripts and cannot be easily edited to add a prefix (since it basically wraps plenty of syscalls).  Update: added below two gists       (sub-second random delays can be added in the sample script I provided for a proof of consistent results)  Update: solution to this question would also solve , as @Gilles pointed out. However I have come to the conclusion that it's not possible to do what asked here and there. When using 2&gt;&amp;1 both streams are correctly merged at the pty/pipe level, but to use the streams separately and in correct order one should indeed use the approach of  that involves syscall hooking and can be seen as dirty in many ways.  I will be eager to update this question if somebody can disprove the above.
56892
I want to reopen the stdin and stdout (and perhaps stderr while I'm at it) filehandles, so that future calls to printf() or putchar() or puts() will go to a file, and future calls to getc() and such will come from a file.  1) I don't want to permanently lose standard input/output/error. I may want to reuse them later in the program.  2) I don't want to open new filehandles because these filehandles would have to be either passed around a lot or global (shudder).  3) I don't want to use any open() or fork() or other system-dependent functions if I can't help it.  So basically, does it work to do this:  stdin = fopen("newin", "r");   And, if it does, how can I get the original value of stdin back? Do I have to store it in a FILE * and just get it back later?
53614
There are some commands which filter or act on input, and then pass it along as output, I think usually to stdout - but some commands will just take the stdin and do whatever they do with it, and output nothing.  I'm most familiar with OS X and so there are two that come to mind immediately are pbcopy and pbpaste- which are means of accessing the system clipboard.  Anyhow, I know that if I want to take stdout and spit the output to go to both stdout and a file then I can use the tee command. And I know a little about xargs, but I don't think that's what I'm looking for.  I want to know how I can split stdout to go between two (or more) commands. For example:  cat file.txt | stdout-split -c1 pbcopy -c2 grep -i errors   There is probably a better example than that one, but I really am interested in knowing how I can send stdout to a command that does not relay it and while keeping stdout from being "muted" - I'm not asking about how to cat a file and grep part of it and copy it to the clipboard - the specific commands are not that important.  Also - I'm not asking how to send this to a file and stdout - this may be a "duplicate" question (sorry) but I did some looking and could only find similar ones that were asking about how to split between stdout and a file - and the answers to those questions seemed to be tee, which I don't think will work for me.  Finally, you may ask "why not just make pbcopy the last thing in the pipe chain?" and my response is 1) what if I want to use it and still see the output in the console? 2) what if I want to use two commands which do not output stdout after they process the input?  Oh, and one more thing - I realize I could use tee and a named pipe (mkfifo) but I was hoping for a way this could be done inline, concisely, without a prior setup :)
711
Does anything symbolic happen in chaining bash commands via a pipe or is it all compute-pass-compute-pass?   For example in head t.txt -n 5 | tail -n 2, is head t.txt -n 5 getting computed and then tail -n 2 executes over it. Or first there is some abstraction to tell the shell that lines 3 to 5 are to be read? It might not make a difference in this example, but I guess can in other scenarios.
111299
I have the following code in a batch script:  mpirun -np 6 ./laplace &lt;&lt;END 100 100 100 0.01 100 3 2 1 END | tail -n 1 &gt; output   But it isn't working. What I want it to do is to use the HERE file as input to the mpirun command, and then pipe the output to the tail command. However, I think the HERE file and tail output things are getting confused.  How should I write this so that it does what I want?
118188
I tried the following command after watch  on pipe shenanigans.  man -k . | dmenu -l 20 | awk '{print $1}' | xargs -r man -Tpdf | zathura -   It basically prints a list of manpages to dmenu for the user to select one of them, then it uses xargs to run man -Tpdf % (print to stdout a pdf of the manpage git from the xargs' input) and pass the pdf to a pdf reader (zathura).  The problem is that (as you can see in the video) the pdf reader starts even before I select one manpage in dmenu.  And if I click Esc and select none, the pdf reader is still open showing no document at all.  How can I make the pdf reader (and any other command in a pipe chain) to only run when its input reach a end-of-file or when it receives an input at all? Or, alternatively, how can I make a pipe chain to stop after one of the chained commands returns a non-zero exit status (so that if dmenu returns an error for not selecting an option, the following commands are not run)?
18131
Test environment:  $ mkdir testdir $ cd testdir $ echo | tee "file  name"   Now, ls will print file?name (under Ubuntu GNU bash, at least), while ls | cat will print file and name in separate lines.  What I'd like to achieve is, print file?name with the  piped version, so something like  $ something ls | cat` or `ls | something | cat` or `ls | something cat   In other words, how to fool a command like ls here into thinking, it has a TTY, when it does not, in the shell?  Note: ls is just an example program here, I'm looking for generic solution, not program specific like ls -q | cat would be.
288
When I just used pipe in bash, I didn't think more about this. But when I read some C code example using system call pipe() together with fork(), I wonder how to understand pipes, including both anonymous pipes and named pipes.  It is often heard that "everything in Linux/Unix is a file". I wonder if a pipe is actually a file so that one part it connects writes to the pipe file, and the other part reads from the pipe file? If yes, where is the pipe file  for an anonymous pipe created? In /tmp, /dev, or ...?   However, from examples of named pipes, I also learned that using pipes has space and time performance advantage over explicitly using temporary files, probably because there are no files involved in implementation of pipes. Also pipes seem not store data as files do. So I doubt a pipe is actually a file.
37002
I'm trying to watch for any new output of a log file. Another script (not under my control) is deleting the file then creating a new one with the same name. Using tail -f doesn't work because the file is being deleted.
7780
I have a script which calls two commands:  long_running_command | print_progress   The long_running_command prints progress but I'm unhappy with it. I'm using print_progress to make it nicer (namely, I print the progress in a single line).  The problem: Connection a pipe to stdout also activates a 4K buffer, so the nice print program gets nothing ... nothing ... nothing ... a whole lot ... :)  How can I disable the 4K buffer for the long_running_command (no, I do not have the source)?
25914
So I like to harden my bash scripts wherever I can (and when not able to delegate to a language like Python/Ruby) to ensure errors do not go uncaught.  In that vein I have a strict.sh, which contains things like:  set -e set -u set -o pipefail   And source it in other scripts. However, while pipefail would pick up:  false | echo it kept going | true  It will not pick up:  echo The output is '`false; echo something else`'    The output would be   The output is ''  False returns non-zero status, and no-stdout. In a pipe it would have failed, but here the error isn't caught. When this is actually a calculation stored in a variable for later, and the value is set to blank, this may then cause later problems.  So - is there a way to get bash to treat a non-zero returncode inside a backtick as reason enough to exit?
96825
Say I have the following script:  #!/bin/bash for i in $(seq 1000) do     cp /etc/passwd tmp     cat tmp | head -1 | head -1 | head -1 &gt; tmp  #this is the key line     cat tmp done   On the key line, I read and write the same file tmp which sometimes fails.  (I read it is because of race conditions because the processes in the pipeline are executed in parallel, which I do not understand why - each head needs to take the data from the previous one, doesn't it? This is NOT my main question, but you can answer it as well.)  When I run the script, it outputs about 200 lines. Is there any way I can force this script to output always 0 lines (so the I/O redirection to tmp is always prepared first and so the data is always destroyed)? To be clear, I mean changing the system settings, not this script.  Thanks for your ideas.
100688
If this is my subprocess:  import time, sys for i in range(200):     sys.stdout.write( 'reading %i\n'%i )     time.sleep(.02)   And this is the script controlling and modifying the output of the subprocess:  import subprocess, time, sys  print 'starting'  proc = subprocess.Popen(     'c:/test_apps/testcr.py',     shell=True,     stdin=subprocess.PIPE,     stdout=subprocess.PIPE  )  print 'process created'  while True:     #next_line = proc.communicate()[0]     next_line = proc.stdout.readline()     if next_line == '' and proc.poll() != None:         break     sys.stdout.write(next_line)     sys.stdout.flush()  print 'done'   Why is readline and communicate waiting until the process is done running?  Is there a simple way to pass (and modify) the subprocess' stdout real-time?  BTW, I've seen , but I don't need the logging features (and havn't bothered understand much of it).  I'm on Windows XP.
91345
The  tells me that /proc/$pid/mem “can be used to access the pages of a process's memory”. But a straightforward attempt to use it only gives me  $ cat /proc/$$/mem /proc/self/mem cat: /proc/3065/mem: No such process cat: /proc/self/mem: Input/output error   Why isn't cat able to print its own memory (/proc/self/mem)? And what is this strange “no such process” error when I try to print the shell's memory (/proc/$$/mem, obviously the process exists)? How can I read from /proc/$pid/mem, then?
122931
I have a file named /tmp/urlFile where each line represents a url. I am trying to read from the file as follows:  cat "/tmp/urlFile" | while read url do     echo $url done   If the last line doesn't end with a newline character, that line won't be read. I was wondering why?   Is it possible to read all the lines, regardless if they are ended with a new line or not?
13333
I need to display whole file before tracking it for a new changes, not only the last 10 lines (yep, I know it's not tail conceptually). In other words, something like cat -f would do, if it would ever existed. Tail's man doesn't give me any ideas. The only option I see right now is to somehow combine cat all but last 10 lines and tail -f output.  Any hints, please?