deleting logical clusters does not free up memory

On large scale installations, larger chunks of memory are being retained after logicalclusters are deleted. The memory also is not fully re-used on the creation of new logicalclusters. This can even lead to OOMs.

The effect can also be observed over multiple days where memory is still in use

Deleting the pods frees up the memory again.

We should investigate where the memory is being held and how to release it properly or re-use it on the second creation run.

@SimonTheLeg

@SimonTheLeg I have been thinking about this since yesterday.

Upon deleting the pods as shown above, the memory is being freed.

Now I have two assumptions :-

Go GC needs hints to aggressively return the memory. iiuc u can tune this with GOMEMLIMIT , GOGC.
- iirc Go Runtime keep the memory pages for a duration before returning it to OS buffer, in order to minimise sys-calls in case it needs to allocate memory later. I recall it is 5 mins by default.
Otherwise, I have a list of potential memory leaks. We would need to try each fix separately till we reach the ultimate issue.

Before we invest into this, is it a high priority ?

cc @mirzakopic @ntnn

@Elbehery

@Elbehery its quite big priority, as we seen OOM in the tests

@mjudeikis

@Elbehery its quite big priority, as we seen OOM in the tests

Thanks @mjudeikis

It is also quiet involving one :)

@SimonTheLeg

@SimonTheLeg any conclusions here ?

Navigation Menu

Repository navigation

deleting logical clusters does not free up memory #4071

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

deleting logical clusters does not free up memory #4071

Description

Activity

Elbehery commented on May 13, 2026

mjudeikis commented on May 20, 2026

Elbehery commented on May 20, 2026

Elbehery commented on May 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions