Skip to content

Concurrency management - #584

Open
db3l wants to merge 3 commits into
rustic-rs:mainfrom
db3l:concurrency
Open

db3l wants to merge 3 commits into
rustic-rs:mainfrom
db3l:concurrency

Conversation

@db3l

@db3l db3l commented Oct 5, 2026

Copy link
Copy Markdown

This PR adds concurrency management to Rustic. It arose from my own desires to:

  • Limit the CPU usage of Rustic when I didn't want to cede an entire machine to its execution. Not all of my systems have a readily available way to impose such a limit.
  • Improve support for a low network latency backend with an underpowered CPU and slow storage (an older NAS with hard drives). The request rate of Rustic was overwhelming the server and thrashing the disks, yielding poor performance and some failed operations.
  • Increase writing concurrency for higher latency backends to boost performance, when the trade off of slightly increased backup size due to less deduplication is acceptable.

Options (at the repository level) configure concurrency for four classes of activity:

  • CPU intensive operations.
  • Read activities for data from a backend.
  • Write activities for data to a backend.
  • Overall backend request rate limiting.

Each class limits how many threads may be active simultaneously (actual parallelism depends on the hardware) and are set as a fixed number or a percentage of available CPUs. A single "availability" setting adjusts defaults together, or can disable limits entirely. Defaults aim to maintain current performance; my own testing even showed improvments in some cases, perhaps due to less contention. When disabled, code reverts exactly to earlier concurrency levels.

After a few failed attempts to rationalize the actual threading top down, I ended up with this implementation: a barrier approach where threads request the concurrency class they need, sharing ownership of a resource with a quota for active threads. As a cooperative system, it's not without its flaws, but it has several key benefits such as playing nicely with existing code regardless of how the threads are created, working with larger pools than are required by the limits, and supporting threads active in more than one concurrency class. Furthermore, the impact to existing code is modest, existing logic/control flow remains intact, and it can revert to exactly the previous concurrency behavior if disabled.

The sole API change is adding a ConcurrencyOptions field to RepositoryOptions, supported via [repository.concurrency] in a config file or --concurrency-* options (and RUSTIC_CONCURRENCY_* environment variables) for the CLI.

I've left the PR as three commits:

  1. The core changes. This contains ConcurrencyManager and ConcurrencyBackend together with minimal changes to integrate them into the existing code. The backend class is the only limit enforced as is entirely internal to ConcurrencyBackend.
  2. Integrate concurrency into the internals (backends, archiver and blob processing). This covers the bulk of the changes to support concurrency limits. Guards are acquired by code according to its class, and appropriately sized concurrent maps or pools are used for any internal concurrent operations performed during archiving and packing operations.
  3. Integrate concurrency into the top level command handlers. This uses concurrency limits to size any top level threads, pools or parallel maps created during command execution, ensuring they are sufficient to meet limits. Limits primarily continue to be enforced by the lower level code, but a few additional CPU class guards in check, prune and copy cover notable local processing.

Happy to address any questions.

db3l added 3 commits October 5, 2026 12:28
A new ConcurrencyManager implements limits (threading) for concurrency
classes within rustic_core, with configuration through ConcurrencyOptions.
Limits may be a set number of threads or a percentage of available CPUs.
Additional mechanisms are provided to help integrate with iterators,
pariter parallel maps and Rayon parallel iterators and pools.

A new ConcurrencyBackend (and ConcurrentBackend trait) provide access to
the manager from within existing code paths.  Updates to the Repository
class support this, as well as exporting the options as part of its own
RepositoryOptions.

The only changes to existing code are the minimum necessary to support the
new manager and backend.  The only concurrency class enforced in this
commit is the Backend class.
Enforce concurrent limits within the internal processing of rustic_core.
This is done by acquiring concurrency guards around relevant code according
to its concurrency class and using appropriately sized concurrent maps or
pools during processing.
Use concurrency limits for threads, pools or parallel maps created as part
of command execution, to ensure sufficient threads exist for the limits.
Add a handful of new concurrency guards to cover some CPU intensive
processing in the check, prune and copy commands.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant