Mojo allows you to access the entire Python ecosystem, but environments can vary depending on how Python was installed. It's worth taking some time to understand exactly how modules and packages work in Python, as there are a few complications to be aware of. If you've had trouble calling into Python code before, this will help you get started. Next week we'll build a GUI app using Python libraries to call into performant Mojo functions.
Let's start with Python, if we have two files in the same directory:
If mod.py has a single function and a variable:
You can call it from any file in the same directory:
mod.py is treated as a module named mod. You can also import any module that is on sys.path, let's take a look:
Because I'm running the python interpreter from /home/j/blog, that's the last thing on my python path, and the reason mod.py is accessible.
Just like your system $PATH environment variable, each path is checked in descending order until it finds your module.
If we look inside /usr/lib/python3.11, we'll find the standard library for Python 3.11:
Everything there can be used as a module because it's on sys.path, for example we can import a function from tempfile.py:
We can also add to sys.path with an env var to give us access to modules elsewhere:
If you check the first entry on sys.path it'll now be there:
You can also edit sys.path directly:
To convince ourselves this works create a file at /tmp/mod2.py:
You can now access mod2 as a module:
Anything inside that module is accessible:
It's considered bad practice to directly modify sys.path, you should use the module system as intended, but it's a good way to see how things work under the hood.
There are a lot more details on how Python finds and loads modules and packages, you can read the module search path docs and the site module docs. And if you're wondering about how sys.path is generated, it's described at a high level in the source code and the sys path init docs.
The most important part for our purposes is sys.prefix which is the base where the lib and bin folders are located for the python binary we're using:
Now notice the prefix is two folders above the python executable, this is the root where sys.path will start searching from:
Depending on the system you're running on this will look very different, but you'll always see lib and bin at sys.prefix, and this is where the first few paths like /usr/lib/python3.11 and /usr/lib/python3.11/site-pacakges are added. There are ways to hack this as per the docs above, but that's the general rule.
Calling Python from Mojo🔥
You can install the Mojo SDK here.
In Mojo we can use any Python module in sys.path in the same way. To work out exactly which environment is being used, let's start by checking the prefix:
Calling Python could result in an error, so we either need to use def main(): as the entrypoint, fn main() raises: to mark it as a strict Mojo function that can throw an error, or handle the errors in a try except block:
In the next release of Mojo, this will print the error from Python if something goes wrong, currently a generic error is thrown for Python failures:
You can also print everything that's on sys.path from Mojo:
Notice anything different here? We don't have our current directory on path! You can add it with:
Now any modules in the current directly are available, let's access the mod.py we created earlier from Mojo:
What is a Package?
Mojo behaves the same as Python, a package can be a subfolder containing an __init__.🔥 or __init__.mojo with initialization logic. It also allows you to access other modules in the same directory:
A question that often comes up in both Python and Mojo is relative imports, this is possible with the above structure. car, plane and common can access each other because they are sub-packages of the package vehicles:
fuel and refuel are now accessible from ./vehicles/car/drive.🔥:
This is different to other languages where file structure doesn't matter as much, keep this in mind as you build out your code base.
Note that as of Python 3.3 the __init__.py file is not required if you don't have any initialization logic due to PEP 420, however it's still currently a requirement in Mojo to mark a folder as a package.
Creating a virtual environment with venv
venv comes with Python and can be used to generate a virtual environment from the python binary we have on path, for more details you can read a primer here.
First check which python binary you're using to make sure it's the one you want:
Check to make sure Mojo is linking to the same libpython as the executable we're going to use to create the venv:
We can find the associated libpython:
Make sure that's the same as ~/.modular/modular.cfg, you want libpython[major].[minor].so, if you're on macOS it'll end in .dylib:
Now that we're certain Mojo is linking to the correct Python on our system, we can create a virtual environment and install dependencies into it:
Activating a venv is simple, it just adds a few env vars and modifies your $PATH so that ~/venv/bin is at the top of your system $PATH variable:
If we look inside that path, you can see these commands will now take precedence:
Install a library for pretty printing named rich into the venv and take a look at the path:
The two site-packages folders that were controlled by my system package manager have been removed, and the venvs /home/j/venv/lib/python3.11/site-packages is now in our sys.path. Let's have a look inside:
rich and it's dependencies were installed into the venv so we can now access the module.
Let's check if sys.prefix has changed:
It has, but how are we still getting all the standard library modules in our sys.path? The base_prefix is what adds the python modules from the base installation:
We can access modules from Mojo in the same way so long as the venv is activated, and print where rich is located from Mojo:
This works, but it's finicky and breakable. python must use the same sys.prefix when we create the venv as the one we're linking to from Mojo. And the venv must be activated before running Mojo, so that ~/venv/bin is at the top of our system $PATH env var.
Installing libpython with Conda
To use a specific version of Python with you can install it with conda and link to the libpython that's installed. It not only isolates python dependencies, but also C/C++ system libraries like openssl and cuda that are notorious for causing cross-platform and linux distribution problems.
If you don't have conda, you can install miniconda here
Now we have an isolated environment with its own system libraries, python packages, and importantly a fresh libpython which Mojo uses directly for python interop.
For Mojo to use this libpython, and by extension the python environment at runtime, you can set the $MOJO_PYTHON_LIBRARY env var:
Or you can edit it directly in ~/.modular/modular.cfg with an absolute path under the python_lib key like shown previously:
Now when you run mojo again you'll see that it's getting modules from the directories in our isolated python3.10 environment:
Great, now it's not a mystery where our Python modules are coming from!
Let's install a python package into our conda instance now:
numpy is complicated underneath and requires many system libraries for fortran routines, linear algebra, and hardware acceleration. All the system libraries above are all installed to the isolated environment. System libs are checked for compatibility against the version of python and all the other packages you're installing. Having an older or newer distribution won't break it, and we don't have to mess around with our system package manager to install system dependencies. Installing numpy on Apple Silicon uses entirely different libraries to take advantage of different hardware:
Now we can access numpy from inside Python and print the path to the module:
If we can't find something in conda simply install it with `pip`:
And it'll now be available:
Now to create a reproducible environment you can run:
This works as a lockfile for the specific arch and os which you're running on, we're including system libraries that are specific to Linux. If you're building something cross-platform, let conda resolve all the dependencies, and just specify what you need:
Just be careful, as pip dependencies have to be added back in manually when using this technique. It's best to edit it manually, remove the prefix, and set minimum versions as required so that it ends up looking like this:
As a user you can install this environment by running:
Make sure to always put -n <env-name>, so it installs to your <conda-install>/base/envs/ folder and ignores any hard coded prefix.
The aim of this post was to make it clear about how you can access python modules from Mojo, so that you can troubleshoot anything that goes wrong yourself. And to demonstrate the two most common methods of creating virtual environments. There are other solutions like poetry and pdm that have nice features, but conda is the most foolproof way as it installs any version of python with all the required system libraries to an isolated environment. This mitigates the huge amount of system configuration problems and library conflicts that can arise when distributing Python applications.
In my next post we'll be creating a GUI app using Python libraries that call performant Mojo functions, stay tuned!