_ctypes.cpython-39-x86_64-linux-gnu.so: undefined symbol: PyFloat_Type in embedded Python loaded with dlopen

I am using embedded Python (3.9) in ubuntu 20.04 and trying to import ctypes which produces the error _ctypes.cpython-39-x86_64-linux-gnu.so: undefined symbol: PyFloat_Type.

I am compiling a shared object, which is loaded dynamically using dlopen().

CMake is used to build the shared object. I am stating Python3 dependency like so:
find_package(Python3 REQUIRED COMPONENTS Development Development.Embed) and link using target_link_libraries(${target_name} Boost::filesystem Python3::Python)

If I understand correctly, this tells CMake to link directly with libpython3.9.so (I also tried to explicitly state linking to libpython3.9.so, but that did not solve the issue).
I do see that libpython3.9.so exports PyFloat_Type and that _ctypes.cpython-39-x86_64-linux-gnu.so does not.

The import is simply done by the PyRun_SimpleString() function: PyRun_SimpleString("import ctypes").

I should state that I have seen on the web some solutions, but none of them worked (like exporting LD_FLAGS="-rdynamic", but does also did not help).

I should also point out that importing using the interpreter (python3.9) works well.

Here is the build command generated by CMake:
/usr/bin/c++ -fPIC -g -Xlinker -export-dynamic -shared -Wl,-soname,mytest.python3.so -o mytest.python3.so CMakeFiles/mytest.python3.dir/[mydir]/[myobjects].o /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.71.0 /usr/lib/x86_64-linux-gnu/libpython3.9.so /usr/lib/x86_64-linux-gnu/libpython3.9.so

Thanks for any help in advance!

Answer

When a C-extension is imported in CPython on Linux, dlopen is used under the hood (and per default with RTLD_LOCAL-flag).

A C-extension usually needs functionality from the Python-library (libpythonX.Y.so), like for example PyFloat_Type. However, on Linux the C-extension isn’t linked against the libpythonX.Y.so (the situation is different on Windows, see this or this for more details) – the missing function-definition/functionality will be provided by the python-executable.

In order to be able to do so, the executable must be linked with -Xlinker -export-dynamic, otherwise loader will not be able to use the symbols from the execuable for shared objects loaded with dlopen.

Now, if the embeded python is not the executable, but a shared object, which loaded with ldopen itself, we need to ensure that its symbols are added to the dynamic table. Building this shared object with -Xlinker -export-dynamic doesn’t make much sense (it is not an executable after all) but doesn’t break anything – the important part, how dlopen is used.

In order to make symbols from text.python.so visible for shared objects loaded later with ldopen, it should be opened with flag RTLD_GLOBAL:

RTLD_GLOBAL
The symbols defined by this shared object will be made
available for symbol resolution of subsequently loaded
shared objects.


As long as the embedded python is loaded with dlopen, the main executable doesn’t need to be build/linked with -Xlinker -export-dynamic.

However, if the main executable is linked against the embedded-python-shared-object, -Xlinker -export-dynamic is neccessary, otherwise the python-symbols want be visible when dlopen is used during the import of c-extension.


One might ask, why aren’t C-extension linked against libpython in the first place?

Due to used RTLD_LOCAL, every C-extension would have its own (uninitialized) version of Python-interpreter (as the symbols from the libpython would not be interposed) and crash as soon as used.

To make it work, dlopen should be open with RTLD_GLOBAL-flag – but this is not a sane default option.