Repository navigation
Conversation
A new ConcurrencyManager implements limits (threading) for concurrency classes within rustic_core, with configuration through ConcurrencyOptions. Limits may be a set number of threads or a percentage of available CPUs. Additional mechanisms are provided to help integrate with iterators, pariter parallel maps and Rayon parallel iterators and pools. A new ConcurrencyBackend (and ConcurrentBackend trait) provide access to the manager from within existing code paths. Updates to the Repository class support this, as well as exporting the options as part of its own RepositoryOptions. The only changes to existing code are the minimum necessary to support the new manager and backend. The only concurrency class enforced in this commit is the Backend class.
Enforce concurrent limits within the internal processing of rustic_core. This is done by acquiring concurrency guards around relevant code according to its concurrency class and using appropriately sized concurrent maps or pools during processing.
Use concurrency limits for threads, pools or parallel maps created as part of command execution, to ensure sufficient threads exist for the limits. Add a handful of new concurrency guards to cover some CPU intensive processing in the check, prune and copy commands.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds concurrency management to Rustic. It arose from my own desires to:
Options (at the repository level) configure concurrency for four classes of activity:
Each class limits how many threads may be active simultaneously (actual parallelism depends on the hardware) and are set as a fixed number or a percentage of available CPUs. A single "availability" setting adjusts defaults together, or can disable limits entirely. Defaults aim to maintain current performance; my own testing even showed improvments in some cases, perhaps due to less contention. When disabled, code reverts exactly to earlier concurrency levels.
After a few failed attempts to rationalize the actual threading top down, I ended up with this implementation: a barrier approach where threads request the concurrency class they need, sharing ownership of a resource with a quota for active threads. As a cooperative system, it's not without its flaws, but it has several key benefits such as playing nicely with existing code regardless of how the threads are created, working with larger pools than are required by the limits, and supporting threads active in more than one concurrency class. Furthermore, the impact to existing code is modest, existing logic/control flow remains intact, and it can revert to exactly the previous concurrency behavior if disabled.
The sole API change is adding a ConcurrencyOptions field to RepositoryOptions, supported via [repository.concurrency] in a config file or --concurrency-* options (and RUSTIC_CONCURRENCY_* environment variables) for the CLI.
I've left the PR as three commits:
Happy to address any questions.