os.system() vs. os.popen() when using bash process substitution: ls: cannot access ‘/dev/fd/63’: No such file or directory

I’m trying to write a Python program that takes a file argument and sends it to a subprocess, using os.popen(). It works great for normal files. But it doesn’t work when using bash Process Substitution (short article).

Why does os.system() work but os.popen() fail for the same command (on Ubuntu)? Here is run.py:

#!/usr/bin/env python3
import os
import sys

cmd = "ls -l %s" % sys.argv[1]
os.system(cmd)
pipe = os.popen(cmd)
print(pipe.read(), end="")

Output:

# Both os.system("ls -l file") and os.popen("ls -l file") work great
$ run.py file
-rw-r--r-- 1 peter peter 313080 Apr 15 12:27 file
-rw-r--r-- 1 peter peter 313080 Apr 15 12:27 file

# os.system("ls -l /proc/self/fd/11") works great but os.popen("ls -l /proc/self/fd/11") fails
$ run.py <(cat file)
lr-x------ 1 peter peter 64 Apr 15 12:27 /proc/self/fd/11 -> 'pipe:[171197]'
ls: cannot access '/proc/self/fd/11': No such file or directory

Does anybody know why os.popen("ls -l /proc/self/fd/11") fails like this?

I’ve created a Perl version, perl-run.pl, to be a similar as I could muster to the Python run.pl version above, and both Perl’s system() and open() work as I expect:

#!/usr/bin/perl -w
use strict;

my $cmd = "ls -l $ARGV[0]";
system($cmd);
open(my $i, $cmd . " |");
print <$i>;

Output:

$ perl-run.pl <(cat file) 
lr-x------ 1 peter peter 64 Apr 15 12:27 /proc/self/fd/11 -> 'pipe:[165704]'
lr-x------ 1 peter peter 64 Apr 15 12:27 /proc/self/fd/11 -> pipe:[165704]

Answer

The os.popen method is actually just a wrapper for subprocess.Popen. If we look at the documentation for subprocess.Popen, we find:

If close_fds is true, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. Otherwise when close_fds is false, file descriptors obey their inheritable flag as described in Inheritance of File Descriptors.

This means that when using a shell construct like <(some command), the file descriptor associated with that redirection gets closed before executing the subprocess, so the corresponding /dev/fd/nn file disappears.

There’s no way to control this flag when calling os.popen, but you can instead use the subprocess module directly and write something like:

#!/usr/bin/env python3
import os
import subprocess
import sys

cmd = "ls -l %s" % sys.argv[1]
os.system(cmd)
pipe = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, close_fds=False)
stdout, stderr = pipe.communicate()
print(stdout, end="")

See the subprocess documentation for details on interacting with Popen objects.