Using pipes

When it comes to using pipes within your Python code, we have two options, the first of which is to use the os module in order to create your pipes, and the second is to leverage multiprocessing.Pipe().

The difference between the two is that the multiprocessing.Pipe implementation is, typically, far more high level than the os.pipe implementation. A key thing to note is that you can only perform bidirectional communication using the multiprocessing.Pipe implementation, as os.pipe is unidirectional.

In this example, we look at how we can pass information from thrown exceptions from child processes back up to parent processes:

import multiprocessing
import os, sys
import traceback

class MyProcess(multiprocessing.Process):
 def __init__(self, pipein):
  super(MyProcess, self).__init__()
  self.pipein = pipein

def run(self):
  try:
    raise Exception("This broke stuff")
  except:
    except_type, except_class, tb = sys.exc_info()

    self.pipein = os.fdopen(self.pipein, 'w')
    self.pipein.write(str(except_type))
    self.pipein.close()

def main():
pipeout, pipein = os.pipe()

childProcess = MyProcess(pipein)
childProcess.start()
childProcess.join()

os.close(pipein)
pipeout = os.fdopen(pipeout)

pipeContent = pipeout.read()
print("Exception: {}".format(pipeContent))

if __name__ == '__main__':
main()