Three Dollar Quill

All right, here's the deal. I want to work on pulling code from the motrfile into MOTR proper, and part of that is going to be developing two interfaces that the current behavior doesn't need, so this is going to be, a little interesting. I've talked about these before, but I figure I'll try to just run through them quickly from first-ish principles.

Let's call these interfaces Package and Platform. A given repository provides one or more packages, which can run on one or more platforms. A package is a pip-installable library. A platform is a particular version of Python running on a particular host system. This can mean different versions of CPython, or PyPy, Brython, Pyodide, maybe Nuitka, etc.

Different steps in the Action graph can target different combinations of platforms, but the overall graph should generally work on a consistent set of packages, so we essentially want some way to combine "two functions that convert a set of packages to a sequence of graph operations" into "one function that converts a set of packages to a sequence of graph operations".

I'd been considering special-casing the "one-package" case, but it's probably less effort to start with to do the following:

• Generate a list of package "roots" (at graph creation time). These "roots" are the directory that contains the pyproject.toml file.
• From this, pass each project root into pyproject-build, and from that, extract a path and a package name.
• We now have a list of package names, and if I avoid special-casing, then I can just always use that name for the "suffix" purpose.
• From the paths, I generate a constraint file. This is an output, and it's passed as an input to pip install commands, via the constraints environment variable. It should be accompanied by one of the names from the list of names. Which means that list needs to be constructed at graph creation time.

This is... a little vexing. I can avoid relying on the actual name by constructing a requirements.txt file associated with the path to the root for each package. So, fundamentally, at graph time we're always working with a path-to-the-root, and only when the graph is executed do we construct the mapping from those paths to the actual packages.

So, if the Package class is only relevant at graph time, then the only information it can use is path-to-the-root. That simplifies things in a complicated way.

Taking it from the top:

• Generate a list of roots
• Have a custom action that takes in the list of roots; its helper should produce a constraints file and a mapping from roots to requirements files
• We want to combine these with actions that take multiple roots to create output; to avoid duplicated code, actions should take a single root and requirements file, and there should be a library function to map over these.

(It also occurs to me that I'll probably not be able to rely on any of this for RPython projects, that is, projects using PyPy for Python 2.7.)

So, the two ways to take multiple packages are, either to dispatch to a function that takes single packages, or to discard the package information entirely, and dispatch a single call to a function that takes no packages.

Let's see how tasks relate...

I should switch to requirements + constraints ASAP, and hardcode the root list for now.

I should replace the Flit action with a PyprojectBuild action, but maintain the current behavior on the filesystem for the moment.

At some point I need to rewrite the Pip class, but it needs to take an optional root argument, potentially with an associated suffix.

It may make sense to divide the Pip class into a "package" version and a "no-package" version. That's an open question right now.

(Because every stage of the pip setup needs to at least know the suffix, then the package information is effectively part of the class, and doesn't need to be passed along with the requirements needed for the associated action.)

I'm going to need to take some time to work through all of the implications of this, but not right now. Right now, I need to get ready for bed, maybe do some writing.

Good night.