The future of Python is written in Rust

If you follow some of the more popular packages in the Python ecosystem you might have seen some announcements that say that they have rewritten either the whole package or just some parts of it in Rust.

Python has been forever dubbed as the "easy" language to learn, always the first to be recommended for beginners, while Rust has some notoriety for being hard to grok, mainly because of its explicit memory model.

Some notable examples of this trend are:

• pydantic - A popular data validation library, the core has been rewritten in Rust.
• pola.rs -  A dataframe library that is built on the Apache Arrow format.
• ruff - A linter that claims impressive performance increases over the usual suspects; pylint and flake8.
• tokenizers - Hugging Face's implementation of the most common tokenizers

Why are Python developers moving over to Rust?

Who cares about speed anyway?

Python developers often argue that runtime speed is never the bottleneck in applications, which can be true but of course, there are exceptions to this. What would you do if you hit a roadblock like this? Chalk it up to "Python being slow" or look for alternative solutions.

Rust is an amazing systems programming language that enables you to write blazing-fast applications, although the learning curve can be pretty steep. During compilation, Rust can usually detect memory management, parallelization, and a bunch of other issues that the Python interpreter/VM would never be able to catch, even at runtime.

Of course, other languages offer similar guarantees, but Rust being a language without a runtime, is a perfect candidate for writing extensions for other languages as these can be called from any other language, like Python!

How?

PyO3 is the most common tool that enables seamless interoperation between the two languages nowadays. You can get started with PyO3 to write a native Python module in Rust in a matter of minutes.

Let's implement a small library in Rust that exposes one function called hello_python.

The easiest way to get started is with maturin, a zero-config tool for creating the scaffolding required.

mkdir hello_python && cd hello_python
 
# Create and activate our Python virtual environment
python3 -m venv venv
source venv/bin/activate

# Install maturin
pip install maturin
 
# Initialize the Rust project and related scaffolding
maturin init
✔ 🤷 Which kind of bindings to use? · pyo3

The output should be a folder structure like this:

tree -L 2                                                                                                                                       
.
├── Cargo.toml
├── pyproject.toml
├── src
│   └── lib.rs
└── venv
    ├── bin
    ├── include
    ├── lib
    └── pyvenv.cfg

5 directories, 4 files

Let's edit the Rust code under src/lib.rs.

As mentioned above, all we want our Rust function to do is print Hello from Rust! on the terminal.

This looks something like this:

use pyo3::prelude::*;

#[pyfunction]
fn hi() -> PyResult<String> {
    Ok("Hello from Rust!".to_string())
}

#[pymodule]
fn hello_python(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(hi, m)?)?;
    Ok(())
}

Nothing complicated, we register a Python module called hello_python that is implemented in Rust and adds one function to it, called hi.

To compile our amazing library, all you have to do is run this command:

maturin develop

Finished dev [unoptimized + debuginfo] target(s) in 0.61s

📦 Built wheel for CPython 3.11 to /var/folders/nk//T/.x/hello_python-0.1.0-cp311-cp311-x.whl

🛠  Installed hello_python-0.1.0

Nice! As we have already activated our virtual environment which the package got installed in by Maturin, all we have to do is give it a spin!

import hello_python

print(hello_python.hi())

If we run this, lo and behold!

python hello.py

Hello from Rust!

Python++

With easy-to-use tooling, comes great power. This level of frictionlessness enables such developer productivity that I won't be surprised if we will see more and more Python maintainers making the switch to Rust as their core language.

Of course, Python is here to stay and it's making progress in performance in its regard lately (the improvements in 3.11 are nothing short of amazing!) so there is no need to worry for newcomers, Python will still be the recommended language for years to come - but the libraries you use will become more and more powerful.