[WPE 2.38] Possible memory leak or GC not working

[gnulux]

Hi [Recipient Name],

I am currently running cog 0.12.3 with Webkit 2.38.6 on an IMX8plus ​ARM64 Yocto embedded system, displaying a JavaScript page generated from ReactJS.

A few months ago, I updated my Yocto meta-webkit to use WebPlatformForEmbedded/WPEWebKit wpe-2.38. I configured the following memory limitation variables:

WPE_POLL_MAX_MEMORY="databaseprocess:10m,networkprocess:5m,webprocess:140m,rpcprocess:10m"
WPE_RAM_SIZE="182m"

After restarting cog, the WPEWebProcess initially operates as expected, keeping memory usage below 180MB, and I can observe garbage collection (GC) reclaiming memory in the logs. However, if I leave my application open on COG for approximately seven days without any screen operations, I notice a continuous increase in WPEWebProcess memory consumption, as shown in the graph below. It appears that the GC stops functioning during this period.

Could you please advise on how I can effectively monitor what is causing this progressive memory growth?

[justinmichaud]

Does this just happen on your app? Or do you experience it on other sites too? Please also try disabling JIT, to see if you observe the same trend. You can do that by setting the environment variable JSC_useJIT=0. Confirm that your settings are working with JSC_validateOptions=1 JSC_invalidAndShouldCrash=1

[gnulux]

Hi ,
The issue seams to be related to my app but i didn't validate on another website or app .
Contrary to what I mentioned in my last update, the application is actively being used via the touchscreen when the leak occurs.
I have configured the following settings: JSC_useJIT=0, JSC_validateOptions=1, and JSC_invalidAndShouldCrash=1 and my application still works . I will provide an update on the trend in a few days.

[justinmichaud]

my application still works

This means the settings aren't being applied unfortunately, it should crash

[gnulux]

Hi,

You mean the process should crash at start ?
This is the command i pass on command line :
JSC_useJIT=0 JSC_validateOptions=1 JSC_invalidAndShouldCrash=1 COG_PLATFORM_WL_VIEW_FULLSCREEN=1 COG_URL=http://127.0.0.1:8080 /usr/bin/cog -P wl

The same command with JSC_logGC=1 , show GC output in journalctl -xf -u cog
And the application start normally

[justinmichaud]

I guess if logGC works, it is probably fine. Let's see if the memory usage changes

[gnulux]

Hi ,

Following the activation of JSC_useJIT=0 last Friday, we've observed a complete reversal in memory consumption. The current trend now shows a significant reduction in memory usage, as illustrated in the attached graph

[justinmichaud]

Fun! This is great, I know what your problem is: we just keep tiering up new jit code.

We have discussed adding API to turn off jitting after a while. There are tons of things that can be done about this, but it is a bit of ways off and requires some deep thought.

So for now: maybe try setting JSC_jitMemoryReservationSize to something small, and measure the memory/performance of your application?

It is also worth noting that since this is JIT code and you need the memory, you can always ask nicely for the vm to release it. If you compile with developer mode enabled and run with JSC_useDollarVM=1, you can run:

$vm.triggerMemoryPressure()

in your application code. This should reduce your memory usage, and if it doesn't, it is totally a bug that we need to fix.

*** Developer mode / dollarVM is very dangerous for production, it is just for testing ***

If that works, perhaps we need to make some memory pressure API.

[gnulux]

Understood , but i have several devices in production and i'm wondering if it is safe to only disable JIT ? or do i have to set JSC_jitMemoryReservationSize. Regarding my last tests it seems to work fine without JIT and consume less memory than before.

[justinmichaud]

If you are fine with the decreased performance, it is totally safe to disable JIT

[gnulux]

Hi,

Coming back on my issue not fully resolved yet

Almost all my devices works pretty well except two of them when the memory still growing in a day.
JIT is still disabled.
We've added a routine in order to restart cog when memory goes up to 250M (see bellow).

Nothing weird appears from the inspector
I think i have to analyse JavascriptCore behaviour but how can i dump Webkit heap into a file ?

[justinmichaud]

Is there anything different about the two devices? Are they all running the same page? Are you certain that they aren't jitting? You can run with JSC_dumpDisassembly=1 to be sure (try with and without JIT to confirm your options are being applied).

The first thing I would try is to manually gc. If manually gcing fixes your issue, we can look into options for lowering the gc threshold. Running $vm.triggerMemoryPressure() and $vm.gc in your JS should do this.

Then, it would be nice to confirm there is no JS memory leak (ex: appending some object to Window)

You can use remote WebInspector to analyze the JS heap, and you can build with system malloc (see the cmake option) to use something like HeapTrack to look at the WebKit native heap.

Alternatively, if you have a case that easily reproduces this issue in Epiphany on the desktop, I can take a look.

For WebInspector, I have some notes that may or may not be helpful: https://github.com/justinmichaud/justinmichaud.github.io/wiki/Building-WPEWebKit-for-32%E2%80%90bit-Raspberry-Pi-3-(Buildroot-DRM-config)#pi-webinspector

[gnulux]

​All the devices have the same image with same application and following cog settings
WPE_POLL_MAX_MEMORY="databaseprocess:10m,networkprocess:5m,webprocess:140m,rpcprocess:10m"
WPE_RAM_SIZE="182m"
JSC_useJIT=0

The two devices are in production so I need customer validation first before to change the settings but on my integration device (same apps, version, and settings) when I enable JSC_dumpDisassembly=1 and JSC_useIJIT is commented I can see outputs in the console.

We will see if we can add triggerMemoryPressure() in our code.

For the WebInspector , it does not show the Memory tab on my version of Epiphany 48 Weird!!
Unfortunately I can not reproduce this behaviour on my device or on a desktop with Epiphany. The issue occurs only on the production environment.
For now , I should only be able to connect to WebIn​spector on these devices.
In the meanwhile , I'm preparing a new image with heaptrack and Webkit build with cmalloc.
Also found this link https://blogs.igalia.com/plampe/tracking-webkit-s-memory-allocations-with-malloc-heap-breakdown/ is it avalaible in 2.38 already ?

Your previous link is very usefull by the way , thks

[justinmichaud]

cc @Scony re: heaptrack

This does seem like a head scratcher if it doesn't reproduce on desktop epiphany. Your integration device doesn't reproduce this trend either?

What commit of 2.38 are you on?

[gnulux]

I'm on the latest commit of wpe-2.38 branch (Building from Yocto recipe).
git://github.com/WebPlatformForEmbedded/WPEWebKit.git;protocol=https;branch=wpe-2.38

I do not reproduce the trends on integration device. To be more precise the device is connected to extenal industrial equipments and receive trafic that I cannot fully simulate on integration. Furthermore FS in production are readOnly but COG env variables can be modified.

Does your need is that Epiphany shows the same content than COG ? I think i can do this with port-redirect but not sure .

[justinmichaud]

My guess is: if this can't be reproduced with epiphany or with the integration device, and if your production device has the commits from October, then there might be something going on with your app.

The content is very important. It is also important to rule out any JS memory leaks (such as appending data to an array, but never clearing it).

Please confirm that the production device has the commits from October. "Exclude non-user portions of the main thread stack from stack scanning on 32 bits." fixes a similar issue.