Skip to content

DOC-313: Kinesis: add option for a rust based mock #731

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jbeemster wants to merge 1 commit into
base: main
Choose a base branch
from
Open

DOC-313: Kinesis: add option for a rust based mock #731

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/content/docs/aws/capabilities/config/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,7 @@ Also see [OpenSearch configuration variables](#opensearch) which are used to man
| `KINESIS_SHARD_LIMIT` | `100` (default), `Infinity` (to disable) | Integer value , causing the Kinesis API to start throwing exceptions to mimic the default shard limit. |
| `KINESIS_ON_DEMAND_STREAM_COUNT_LIMIT` | `10` (default), `Infinity` (to disable) | Integer value , causing the Kinesis API to start throwing exceptions to mimic the default on demand stream count limit. |
| `KINESIS_LATENCY` | `500` (default), `0` (to disable)| Integer value of milliseconds, causing the Kinesis API to delay returning a response in order to mimic latency from a live AWS call. |
| `KINESIS_MOCK_PROVIDER_ENGINE` | `node` (default) \| `scala` | String value of `node` (default) or `scala` that determines the underlying build of Kinesis Mock. |
| `KINESIS_MOCK_PROVIDER_ENGINE` | `node` (default) \| `scala` \| `rust` | String value of `node` (default), `scala`, or `rust` that determines the underlying build of Kinesis Mock. See [Performance Tuning](/aws/services/kinesis#performance-tuning) for guidance on choosing an engine. |
| `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` | `512m` (default) | JVM memory format string that sets the maximum memory size for the Kinesis Mock Scala server, corresponds to the JVM `-Xmx` flag. |
| `KINESIS_MOCK_INITIAL_HEAP_SIZE` | `256m` (default) | JVM memory format string that sets the initial memory size for the Kinesis Mock Scala server, corresponds to the JVM `-Xms` flag. |

Expand Down
41 changes: 33 additions & 8 deletions src/content/docs/aws/services/kinesis.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,21 +170,44 @@ You can fetch the CloudWatch logs for your Lambda function reading records from

### Performance Tuning

For high-volume workloads or large payloads, we recommend switching to the Scala engine via the `KINESIS_MOCK_PROVIDER_ENGINE=scala` flag, delivering up to 10x better performance compared to the default Node.js engine.
By default, Kinesis Mock runs on the Node.js engine (`KINESIS_MOCK_PROVIDER_ENGINE=node`).
For high-volume workloads, large payloads, or long-running setups, you can switch to a different engine using the `KINESIS_MOCK_PROVIDER_ENGINE` flag.

Additionally, the following parameters can be tuned:
#### Scala engine

The Scala engine (`KINESIS_MOCK_PROVIDER_ENGINE=scala`) delivers up to 10x better performance compared to the default Node.js engine.
The following parameters can be tuned for the Scala engine:

- Increase `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` beyond the default `512m` to reduce JVM memory pressure.
- Increase `KINESIS_MOCK_INITIAL_HEAP_SIZE` beyond the default `256m` to pre-allocate more JVM heap memory.
- Reduce `KINESIS_LATENCY` artificial response delays from the default `500` milliseconds (or disable entirely with `0`).

Refer to our [Kinesis configuration documentation](https://docs.localstack.cloud/references/configuration/#kinesis) for more details on these parameters.

:::note
`KINESIS_MOCK_MAXIMUM_HEAP_SIZE` and `KINESIS_MOCK_INITIAL_HEAP_SIZE` are only applicable when using the Scala engine.
Future versions of LocalStack will likely default to using the `scala` engine over the less-performant `node` version currently in use.
:::

#### Rust engine

@jbeemster jbeemster Jun 26, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its hard to benchmark Scala vs Rust accurately as Scala does not adhere to API data plane limits like 1MiB/s ingress rates per shard but all of my tests show lower CPU + Memory usage and equivalent or better throughput once shards are scaled up high enough for the Rust side to really get going.


The Rust engine (`KINESIS_MOCK_PROVIDER_ENGINE=rust`), introduced in LocalStack 2026.07, is a LocalStack-native rebuild of Kinesis Mock and is the recommended choice for any demanding or long-lived workload.
It provides:

- **Native multi-account and multi-region support** from a single binary, regardless of how many accounts you use — this removes the per-account instance overhead described under [Limitations](#limitations).
- **Faithful enforcement of all AWS API limits**, including per-shard rate ingress/egress limits.
- **An improved persistence model** that replaces the legacy unbounded in-memory queue, allowing the mock to run under sustained load for long periods without the risk of running out of memory.

:::caution
This is the first release of the Rust engine and it should be considered experimental for anyone opting in.

Persistence is **not compatible** between the Rust engine and the Node.js or Scala engines, in either direction.
Any existing Kinesis persistence (including Cloud Pods and state created with the `node` or `scala` engines) will not carry over when you switch to `rust` — and vice versa.
Switching engines effectively starts from a fresh state.
:::

#### All engines

Regardless of the selected engine, you can reduce `KINESIS_LATENCY` artificial response delays from the default `500` milliseconds (or disable them entirely with `0`).

Refer to our [Kinesis configuration documentation](https://docs.localstack.cloud/references/configuration/#kinesis) for more details on these parameters.

@quetzalliwrites quetzalliwrites Jun 28, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey @jbeemster, we don't have a /references directory in docs, curious about where you found this link?

(this looks like an old link from the v1 docs site from over a year ago)

It would be helpful to know where this old url came from, such as did you find in in search results? Want to double check if I need to update the 301s for this repo.

@jbeemster jbeemster Jun 29, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was a link that got moved from the old section - see line 181 in this diff!


## Resource Browser

The LocalStack Web Application provides a Resource Browser for managing Kinesis Streams & Kafka Clusters.
Expand All @@ -209,8 +232,10 @@ The following code snippets and sample applications provide practical examples o

## Limitations

In multi-account setups, each AWS account launches a separate instance of Kinesis Mock, which is very resource intensive when a large number of AWS accounts are used.
[This Kinesis Mock issue](https://github.com/etspaceman/kinesis-mock/issues/377) is being used to keep track of this feature.
When using the default Node.js or the Scala engine, each AWS account launches a separate instance of Kinesis Mock in multi-account setups, which is very resource intensive when a large number of AWS accounts are used.
[This Kinesis Mock issue](https://github.com/etspaceman/kinesis-mock/issues/377) is being used to keep track of this feature for the upstream engines.

The [Rust engine](#performance-tuning) (`KINESIS_MOCK_PROVIDER_ENGINE=rust`) is not affected by this limitation, as it natively supports any number of accounts and regions from a single binary.

## API Coverage

Expand Down