So, you've got some super-secret Python package, and you're itching to share it with the world. But wait, you don't want everyone and their cat to see your precious source code? Well, you're in luck! In this guide, we're going to show you how to keep your code as hidden as a ninja in the shadows while still sharing it on the not-so-secret Python software repository, PyPI.

Sharing Is Caring...But Not That Much

You know, there are times when you want to play nice and share your Python package with others but still keep your source code under lock and key. It's like sharing your ice cream but not letting anyone see the flavor. To pull off this ninja-level secrecy, we're going to employ Cython – the magical cloak for your Python code.

Keeping Secrets with Cython

Cython – it's not just for making Python faster; it's also your partner in crime for keeping your code hidden. It's like a magician's hat that turns your Python code into mysterious C code and then conjures it into binary form. Your code is now hidden away, safe from prying eyes.

A Symphony of Code Management with Poetry

Enter Poetry – the maestro of Python dependency management. It simplifies the orchestra of dependencies in your Python project. But, it's not always a one-size-fits-all scenario. Sometimes, you need to get a bit creative, like composing a jazz solo in the middle of a classical symphony.

Marching to Your Own Beat

Now, if you're a rebel who doesn't want to follow the Poetry bandwagon or if you've got another trick up your sleeve, don't fret. You can apply the techniques we're about to drop on you to other tools and methods. Traditional setup.py and twine, anyone? It's like using a classic recipe with a modern twist – a pinch of nostalgia and a dash of innovation.

Crafting Your Cython Cloak

Alright, let's get down to business and start weaving our Cython cloak. We need to customize the wheel building process in Poetry. It's like tailoring a bespoke suit for your package. To do this, you'll need to perform some dark incantations in your pyproject.toml file:

[tool.poetry.build]
script = "build.py"
generate-setup-file = false

Poetry usually relies on your pyproject.toml file to create a setup.py file behind the scenes. But, in our cloak-and-dagger scenario, we don't want that. Instead, we want our very own custom build script, lovingly named build.py. Poetry will summon this script when we're ready to build our release wheel. But don't worry, it won't use this script when mere mortals install your package. In the next section, we'll reveal the secrets of this script and how it works its magic.

Oh, and by the way, we need to make sure Cython is on our side. We'll add it to our Poetry development dependencies so that only the chosen ones (package developers) have access to its power. You can do this by chanting poetry add --dev Cython or by manually inscribing the desired version in the tool.poetry.dev-dependencies section and then invoking poetry update:

[tool.poetry.dev-dependencies]
Cython = "^3.0.0"  # Latest version at the time of publishing

Lastly, to keep our secret source code truly secret, we need to make sure Poetry doesn't spill the beans. By default, Poetry tosses all files in your project's source directory into the package, including the Python files we want to keep under wraps. To fix this, we have to lay down the law by adding an exclude key in our tool.poetry section. Just remember, this exclusion is like a matching game, so consult the documentation if you need more clues:

[tool.poetry]
# ...
exclude = ["SRC/**/*.py"]  # Replace SRC with the root of your source
include = ["SRC/**/*.so"]  # And/or Windows/Mac equivalents

Unveiling the Magic: The build.py Script

Now, let's unveil the mysteries of the build.py script – the heart and soul of our project-building process. But fear not, it's not as daunting as it sounds. In fact, it's more of a friendly wizard that helps you with a few simple tasks:

1. Gathering Python Files: It goes on a scavenger hunt to collect all the Python files from your project.
2. Cython Conversion: With a flick of its wand, it transforms these Python files into enigmatic binary blobs using Cython.
3. Integration with Poetry: It reunites these mystical binaries with your source tree, ready to be summoned by Poetry.

First things first, let's set the stage. You'll need to adjust the SOURCE_DIR variable to match your project's hideout:

import multiprocessing
from pathlib import Path
from typing import List

from setuptools import Extension, Distribution

from Cython.Build import cythonize
from Cython.Distutils.build_ext import new_build_ext as cython_build_ext

SOURCE_DIR = Path("SRC")  # Replace "SRC" with your project's root
BUILD_DIR = Path("BUILD")

Next, we'll conjure a function that finds all the Python files and transforms them into Distutils/Setuptools Extension objects. These are like the secret scrolls that Python build scripts, including Cython, use:

def get_extension_modules() -> List[Extension]:
    extension_modules: List[Extension] = []

    for py_file in SOURCE_DIR.rglob("*.py"):
        module_path = py_file.with_suffix("")
        module_path = str(module_path).replace("/", ".")
        extension_module = Extension(
            name=module_path,
            sources=[str(py_file)]
        )
        extension_modules.append(extension_module)

    return extension_modules

Now, let's call upon a function that takes these Extension objects and performs the Cython incantation. It uses Cython's cythonize function, complete with some mystical configurations:

def cythonize_helper(extension_modules: List[Extension]) -> List[

Extension]:
    return cythonize(
        module_list=extension_modules,
        build_dir=BUILD_DIR,  # Avoid building in the source tree
        annotate=False,  # No need to generate an .html output file
        nthreads=multiprocessing.cpu_count() * 2,  # Parallelize the build
        compiler_directives={"language_level": "3", "annotation_typing": False},  # Indicate Python 3 usage to Cython without using type hints annotations
        force=True,  # (Optional) Always rebuild, even if files are untouched
    )

Finally, let's weave the threads of magic together. We'll enlist the Setuptools Distribution object to orchestrate the build. It's like conducting a symphony of code:

extension_modules = cythonize_helper(get_extension_modules())

# Use Setuptools to collect files
distribution = Distribution({
    "ext_modules": extension_modules,
    "cmdclass": {
        "build_ext": cython_build_ext,
    },
})

# Copy all files back to the source directory
# This ensures that Poetry can include them in its build
build_ext_cmd = distribution.get_command_obj("build_ext")
build_ext_cmd.ensure_finalized()
build_ext_cmd.inplace = 1
build_ext_cmd.run()

Unveiling the Wheel Package

Hooray! You've made it this far. Now, it's time to unveil the final act – creating the .whl package for your project. It's like packaging your secrets into a mystical scroll. Just chant poetry build --format wheel, and the magic will happen.

Once the spell is complete, you can open the resulting .whl file using a .zip extractor. Inside, you'll find binary versions of all your Python files, as elusive as shadows in the night.

Oh, one word of caution: make sure not to distribute an sdist package. It contains uncompiled source code. By specifying --format wheel, you ensure that only the wheel package is created, eliminating any chance of accidental PyPI exposure. If you want to see the magic happen in detail, you can use the -vvv flag, which provides a sneak peek into the files being built and added to the .whl package.

Share Your Secrets on PyPI

Now, with your wheel file at the ready, it's time to share your secrets with the world on PyPI. Thankfully, this is a well-trodden path when using Poetry. Many guides are available to hold your hand through the process, like this one. They all rely on the trusty poetry publish command.

Final Thoughts from the Shadows

One last thing to remember about your wheel files: you'll need one for each Python version and platform (OS + architecture) you want to support. These files contain pre-compiled code, so they're not as flexible as Python source code, which can sneak into different systems like a ninja. It's a common practice among projects on PyPI, like TensorFlow.

To make your life easier in generating these wheels for your target platforms, you can enlist the help of CI/CD build matrices.

And if you ever find yourself lost in the dark arts of this tutorial, fear not! We've set up a sample repository on our GitHub here. Explore it, seek guidance, or share your tales of woe. I'm here to assist, lurking in the shadows, waiting to help. While you're at it, why not check out some of my other intriguing public repositories?

— prasad89 (GitHub)