Repository navigation
Monorepo / Monobuild support? #936
Description
- I have searched the issues of this repo and believe that this is not a duplicate.
- I have searched the documentation and believe that my question is not covered.
Feature Request
The goal is to allow a developer to make changes to multiple python packages in their repo without having to submit and update lock files along the way.
Cargo, which poetry seems to be modeled after, supports this with two features
- Path dependencies. You can specify where to find a dependency on disk for local development while still supporting version constraints for public consumption
- Workspaces let you define a group of packages you can perform operations on, like
cargo test --allwill run tests on all packages in a workspace.
It looks like some monobuild tools exist for python (buck, pants, bazel) but
- Some don't support all major platforms
- They assume you are not using pypi and do not publish to pypi and expect all dependencies to be submitted to SCM.
Reacted by Rakha Kanz Kautsar, Brian M. Carr, David Roeca, Ricardo Madriz, Eric Pfahl, Kostis Anagnostopoulos, Jason Kuhrt, Samuel Bishop, Martin Liu, Oleh Stolyar and 135 more
Reacted by Santiago Fraire Willemoes, Simon Biggs, joe-narvaez, Aryan Iyappan, Vineet Joshi, Tstepro, Ricardo Gaspar, Mehdi Nassim KHODJA, Huon Wilson, Michael Salaverry and 11 more
Reacted by finete, Vincent Lecrubier, Lyle Scott III, Santiago Fraire Willemoes, joe-narvaez, Aryan Iyappan, Tstepro, marklidenberg, Nyakku Shigure, Kory Taborn and 3 more
Reacted by finete, Vincent Lecrubier, joe-narvaez, Aryan Iyappan, Tstepro, marklidenberg, Nyakku Shigure, Elliot Rampono, Jay Kuroyanagi and Alexandros Solanos
Activity
floer32 commentedon Mar 14, 2019
[I'm just somebody driving by]
I've had to deal with the monorepo (anti)pattern1️⃣ and I agree it'd be very helpful to have help in dealing with it and packaging it; but also have learned it's really hard to do good things for monorepo without making compromises to normal patterns.
To me it sounds like this could be hard to get into Poetry, at least right from the start. It's easier to imagine another library where someone chooses an option like buck/pants/bazel and then creates a helper to make that work well with Poetry and vice/versa
1️⃣: it's not always an antipattern, I know. too often, it is, and many best practices are abandoned. So it can make it hard to develop monorepo-related features without specific good examples that are targetted for support. TLDR could be good to link to an OSS example (or contrive one and link to that)
epage commentedon Mar 14, 2019
I understand. I hate the religious view taken towards monorepos. I have found what I call mini-monorepos to be useful, small repos that serve a single purpose.
For example, in the Rust world, I have a package for generic testing of conditions, called
predicates. I've split it into 3 different packages,predicates-coreto define the interfaces,predicates, andpredicates-treefor rendering predicate failures in a way similar to pytest. I did these splits so that (1) people aren't forced into dependencies they don't need and (2) Rust is thankfully strict on semver and so the split also represents a splitting of compatibility guarantees. It is more important to provide a stable API for vocab terms (predicates-core) than for implementations that aren't generally passed around.I specially suggested continuing to follow after Cargo's model, like poetry has done in other ways, for monobuild support rather than getting into the more complex requirements related to tools like buck/pants/bazel (fast as possible, vendor-all-the-deps, etc). If someone needs the requirements of those tools, they should probably just use those tools instead. From at least my brief look, it seemed like they don't do a good job of interoperating with other python build / dependency management systems.
davidroeca commentedon Apr 1, 2019
Also would love support here! Specifically for developing a library that has different components each with their own set of potentially bulky dependencies (e.g. core, server type 1, server type 2, client, etc.). Also helpful when trying to expose the same interface that supports different backends (similarly, you don't want to install every backend, just the one you want).
The only OSS library I could find that emulates this approach is toga -- it would be great if poetry could handle dependency resolution for these sorts of libraries.
The toga quickstart explains how the dependencies are managed with a bunch of
setup.pyfiles.NGaffney commentedon Apr 3, 2019
I'm interested in this also. How far does the current implementation of editable installs get you towards your use case?
epage commentedon Apr 3, 2019
Use cases
Rust can solve this at two levels
So regarding the first, editable installs might cover this if you can mix path dependencies with version dependencies which is the key feature needed to partially handle my specified use cases.
NGaffney commentedon Apr 4, 2019
I tried some of this out today and it looks like the first feature you describe is nearly, but not quite supported.
You can declare a dependency both as a dev and non-dev dependency, when you do this the dev version should take precedence (at least that's how I interpret this), allowing a package to be installed as editable during local dev. Then for final build the
--no-devflag would remove the dev version from consideration.I've made it work with a toy example which contains a path dependency in dev and non-dev mode but not for a local vs pypi dependency.
guillaumep commentedon May 20, 2019
I am currently trying to find a way to build a Python monorepo without using heavy-duty tools like Bazel.
I have seen a repository which uses
yarnandlernafrom the JavaScript world to build Python packages:https://github.com/pymedphys/pymedphys
Using yarn you can declare workspaces, and using lerna you can execute commands in all workspaces. You can then use the
scriptsection ofpackage.jsonin order to run Python build commands.I've yet to try this technique with Poetry (and the pymedphys repository does not use it), however I feel it might be worth exploring.
stale commentedon Nov 13, 2019
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
davidroeca commentedon Nov 13, 2019
I still think this could be useful
daveisfera commentedon Dec 19, 2019
yarn workspacesmakes it possible to have a set of shared libs and that are used across multiple apps in a monorepo, so I think it's a good model to follow or at least borrow ideas fromremram44 commentedon Jan 17, 2020
Something I am running into right now is that using Poetry with path dependencies is very unfriendly to the Docker cache.
To correctly leverage the cache (read: don't install all dependencies from scratch on every build) I would want to install the dependencies first, and then copy my own code over and install it. However Poetry refuses to do anything (even
poetry export -f requirements.txt) if my code has not already been copied in.I am considering writing my own parser for the
poetry.lockfile, topip installmy dependencies from there before I copy my code over.One of two things could help me:
path = ...dependencies were part of the workspace, some command could install the other dependencies while ignoring those)poetry exportonly needpyproject.tomlandpoetry.lock, and work even if the code is not there (I'm not sure why it needs it right now)kbakk commentedon Mar 7, 2020
It will be interesting to see how #1993 will be solved - it does imply the Poetry repo becoming a kind of monorepo, right?Edit: Err, sorry, I just assumed it would be done in the current repo, but it seems to not follow a monorepo structure (python-poetry/core)38 remaining items
johnbendi commentedon Jun 20, 2022
I think when considering a monorepo strategy there are quite several issues to consider including dependency management and the delineation between components/libraries and projects. This blog post gets really close to my understanding of how a monorepo should be structured and maybe the discussion can build upon the work done there:
Opendoor Labs Monorepo strategy
michaeloliverx commentedon Sep 1, 2022
I think
pnpmfrom the JavaScript ecosystem really got this right and many popular open source projects adopted it. It might be worth a look for some inspiration.gerbenoostra commentedon Sep 1, 2022
DavidVujic commentedon Sep 1, 2022
The code in this PR would enable Workspace support in Poetry.
We would then be able to have Python code in a monorepo, containing projects - each one defined by a
pyproject.toml- and reuse shared Python packages. The shared packages are allowed to be in a folder structure relative to the project (and not needed to be a subfolder to thepyproject.tomlfile referring to the package).The architecture I would recommend for monorepos (if this feature would be enabled) is called
Polylith. I have linked to a post describing it further up in this thread.neersighted commentedon Oct 4, 2022
Going to fold this into #2270 as we have two parallel issues -- feel free to continue the discussion here, but I'd like to formally track this category of feature in one place.
gerbenoostra commentedon Jan 9, 2023
Above I mentioned that I tried to work around poetry's limitations, and still use poetry in a mono repo. I've finally took some time to improve the repo, and also have written a blogpost to explain the approach.
Perhaps these utility scripts are beneficial.
Blogpost: https://gerben-oostra.medium.com/python-poetry-mono-repo-without-limitations-dd63b47dc6b8
Example repo: https://gitlab.com/gerbenoostra/poetry-monorepo/
kapilt commentedon Mar 6, 2023
@gerbenoostra fwiw I went ahead with a plugin to do the alternate version mentioned in your blog post (modify the wheels from lock files as a post build / pre deploy step). its got some additional primary goals (freezing versions in wheels) https://github.com/cloud-custodian/poetry-plugin-freeze
DavidVujic commentedon Mar 7, 2023
As another alternative, there's a plugin that handles the wheel & sdist packaging by doing work just before the
poetry buildcommand kicks in. It's a plugin called Multiproject, that is the foundation for making architectures like Polylith possible in Python. I'm an advocate for that type of architecture 😄If you want to know more, here's the docs (Multiproject is of course referenced in there): https://davidvujic.github.io/python-polylith-docs/
gerbenoostra commentedon Mar 7, 2023
A really nice to the point solution without other assumptions.
As you describe, it solves the application building side (pinning all versions), and not the library approach (allowing version ranges).
But as mono repos are typically (though perhaps not always) about creating applications, instead of creating externally shared libraries, this probably works most of the time.
gerbenoostra commentedon Mar 7, 2023
Nice to see multiple initiatives here. I like the single command here, just build it and one get's a full library.
Note that
github-actions commentedon Feb 29, 2024
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.