check if large number of directories exist in Python

Is there a trick to quickly check whether 10000 Windows directories exist in Python? I currently stored them in a list, I wonder how to do the check quickly.

Answer

Use with multi-process

os.access("/file/path/foo.txt", os.F_OK)

# check file is exists
os.F_OK
# check file is readable
os.R_OK
# check file is wirteable
os.W_OK
# check file is execute
os.X_OK

It is a simple test

In [1]: import os, pathlib

In [2]: p = "/home/lpc/gitlab/config/test"

In [3]: %timeit pathlib.Path(p).exists()
5.85 µs ± 32.9 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [4]: %timeit os.path.exists(p)
1.03 µs ± 4.69 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [5]: %timeit os.access(p, os.F_OK)
526 ns ± 2.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [6]: def check(p):
    ...:     try:
    ...:         f = open(p)
    ...:         f.close
    ...:         return True
    ...:     except:
    ...:         pass

In [7]: %timeit check(p)
1.52 µs ± 4.41 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [8]: %timeit os.path.isdir(p)
1.05 µs ± 4.87 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Leave a Reply

Your email address will not be published. Required fields are marked *