I have struggled with project directory structure for Python projects in the past. Some things (like relative imports in projects containing multiple modules in different folders) are not immediately clear to most people, which is why I'm posting a concise little write-up of my knowledge on the topic so far. I have created a Github repository with an example project to assist me in explaining some things; I'll be referring to that regularly throughout this post.
Structuring your project
Let's start off with the full directory structure:
pydirstructure/ ├── LICENSE.md ├── pydirstructure │ ├── __init__.py │ ├── module_a.py │ ├── module_b.py ├── README.md ├── requirements.txt ├── setup.py └── tests └── test_a.py
The name of the root directory is not relevant, but I almost always name it
identically to the package name. The directory containing the actual code for
the package does have to have a relevant name (since it'll be used in
<package_name>), hence it's called
pydirstructure, the name of our package.
Most of the files located in the root folder are not going to be very important
to your project itself, since they're often configuration files for CI hooks,
examples and other stuff. However,
requirements.txt in a
sense) are very important to your Python project.
requirements.txt contains the dependencies for your project and can be
pip freeze > requirements.txt.
setup.py actually contains information regarding installing, testing and
deploying your package. Most of it should speak for itself, but I'll go into
three little handy things.
1. Parsing requirements.txt
You can see (on line 9 in
setup.py) that I'm parsing the requirements file
instead of adding all requirements manually to the
setup functions keyword
parse_requirements gives a list of installation requirement objects,
which can be converted to strings using
2. Versioning in toplevel __init__.py
I prefer to keep track of the package version in
refer to that version from all other places. This allows me to keep the version
string in only one place (I'd forget to update all of them, guaranteed, if I
had more than one).
3. Entry points
I often write projects that offer commandline interfaces as part of their
functionality. You can specify those in
setup using the
console_scripts. The syntax is pretty simple:
So if we make the main function in pydirstructure/module_a.py a script:
That's a simple example of a
setup.py that you can use to start out. Of
course you can do way more complicated things, but I'd still be typing in 5
years. There's plenty of documentation online!
This is the file that glues your
package module together. It makes things
like this possible:
import pydirstructure as pds s1 = pds.SomeClass().do_something('waddup readers :D') s2 = pds.SomeOtherClass().do_something_else('waddup yalllll') print(s1) print(s2)
SomeClass is in
SomeOtherClass is in
A package can have multiple
__init__.py files; there should be one in every
module directory (so if we had more directories below the package directory,
each of those should have an
__init__.py to be recognized as a module).
Editable installation with pip
Due to relative imports, you can't simply make each Python file in a structured
package runnable and expect to be able to run all the different files on their
own. However, once you get your
setup.py up and running, you can use the
pip command in the project directory to install it editable:
pip install -e .
(protip: work in a virtual environment if you're going to do this, which you should already be doing anyway).