vm: fix a null dereference on out-of-memory

vm: fix a null dereference on out-of-memory

	. also make other out-of-memory conditions less fatal
	. add a test case for a user program using all the memory
	  it can
	. remove some diagnostic prints for situations that are normal
	  when running out of memory so running the test isn't noisy