Python: Running code from source that includes extension modules

Question

I have started looking into distutils because I need to write an extension module in C (for performance reasons) and distutils seems to be the most straight-forward way.

I have had success building a C file into a Python extension module using "python setup.py build" but I am wondering what the recommended way for using that module during development is. While writing Python code I am used to just run the code from the source directory. But the built extension modules .so of course does not just end up on sys.path magically.

So how do I run my code so it will find the built extension module? Do I pass the output directory on the command line manually or is there some other solution? I would like to still be able to run the code from the source directory as I am using PyCharm to edit and debug the code.

Comments

I'm running OS X 10.8 and Python 3.2, sorry I didn't mention it. But I assume the differences to Linux are minimal.

The current directory is included in sys.path, otherwise I wouldn't be able to import modules in the same directory. But the problem is that the built extension module is in a subdirectory of the "build" directory:

$ find -name '*.so'
./build/lib.macosx-10.8-x86_64-3.2/_foo.so

And so I can't import it without manually adding that directory to sys.path. I'm convinced, someone on this list can shout at me, telling me that I got it completely backwards and that there's a straightforward and intuitive way to develop extension modules!

Answer 1

You can run

 python setup.py build_ext -i

That will build your extension module and install it right into your package structure.

BTW, if you use Cython instead of plain C, you can use pyximport to get on-the-fly extension module builds during development.

Comments

This is really very much what I was looking for! I've set up PyCharm to run this command (by configuring it as an "external tool", maybe there's a simpler way), before running the actual application. o/

> BTW, if you use Cython instead of plain C, you can use pyximport to get on-the-fly extension module builds during development.

I will look into that too, that sounds very convenient. But am I right, that to use Cython the non-Python code needs to be written in the Cython language, which means I can't just copy&past C code into it? For my current project, this is exactly what I do, because the C code I use already existed.

---

It's better than that. Don't copy/paste your code. Just declare it in Cython and you can call straight into the existing C functions cutting out most of the boilerplate involved in making C code accessible to
Python: http://docs.cython.org/src/userguide/external_C_code.html

You'll sometimes need a short Cython wrapper function to convert from Python types to corresponding C types. But this is about 5 lines of easy to read Cython code vs maybe 30 lines of hard to follow C code.

Having written CPython extension modules both by hand and using Cython I strongly recommend to use Cython.

Answer 2

Doesn't Python on Linux (I assume that since you mentioned the module's .so) support having current-dir '.' in $PYTHONPATH? Works fine on Windows.

Check with "python -v script.py | grep ".