Memory consumption in Revolution R is left to the operating system and cannot be controlled in R itself.
R will be default try to grab as much memory as it needs to complete a job from the OS. On Linux you can use a system utility like 'ulimit' to set per-user process limitations, on Windows you can use the 'Windows System Resource Manager' tool.