libc: fix needless malloc failures

The NetBSD libc malloc implementation performs its own out-of-memory
check, presumably for performance reasons.  The check makes a strong
assumption about the address space layout, which is that memory-
mapped pages are always located above the heap.  However, this
assumption does not necessarily hold on MINIX3, thus resulting in
malloc reporting an out-of-memory condition without the system
actually being out of memory at all.  Evidence suggests that in
particular dynamically linked (i.e., pkgsrc) binaries were affected
by this issue - most notably git.

Change-Id: If542fbace0a1cce12aa9e075d51992cbbbf26e94

diff --git a/lib/libc/stdlib/malloc.c b/lib/libc/stdlib/malloc.c
index 5d35e4b41fb8dfabc7e758a640cf383325e3bc96..0570244eda5cb1f809118ceb2b8c2120dcb5db00 100644 (file)
--- a/lib/libc/stdlib/malloc.c
+++ b/lib/libc/stdlib/malloc.c
@@ -818,8 +818,10 @@ imalloc(size_t size)
 
     if ((size + malloc_pagesize) < size)       /* Check for overflow */
        result = NULL;
+#ifndef __minix
     else if ((size + malloc_pagesize) >= (uintptr_t)page_dir)
        result = NULL;
+#endif /* !__minix */
     else if (size <= malloc_maxsize)
        result = malloc_bytes(size);
     else