There isn’t a Solar2D API for getting the device’s total or current memory usage, but you can do so via iOS/Android native calls.
The documentation’s points about reducing (texture) memory usage were written several years ago, and while that is still a good practice, it isn’t as important as before. Some 5-7 years ago, for instance, lower end devices might have had so little RAM that unoptimised games wouldn’t have been able to run. Now, with devices packing more RAM in general, you can get by with less optimised games (see the entire AAA games industry for example).
If your app exceeds the maximum available RAM for the device, then the OS may start to try releasing some memory by closing other background apps (which the users won’t like). If the device runs out of RAM, then the app will likely terminate.
As for what @vb66r55 said, if the individual textures are too large, then they won’t get rendered.