The 512K result size limit being hit is a throttle limit set for Rserve on each grid node. See where this property is set in the admin console in the attached screen shot. This is set to keep over-sized object data requests from swamping the server.
The limit can be increased to accommodate large results but the change is a not a global setting. It must be increased on each node in the grid.
Article ID: 3104178 - Last Review: Oct 31, 2015 - Revision: 1