Fire it up in a docker container and limit the RAM usage that way.
This is a trick I learned years ago with other mmap-based systems.
Are you sure it really limits the RAM? You’re still using the same kernel, and if a process is using more memory than another one I would expect the kernel to keep more of its memory page in RAM than the other.
What was your strategy to measure that?