VM: fix race condition communicating with VM

After processing certain asynchronous requests from VFS, VM would send
an asynchronous reply without supplying the AMF_NOREPLY flag.  As a
result, this asynchronous reply could be taken as the result of an
ipc_sendrec() call, causing the entire VM/VFS communication to become
desynchronized.  The end result was a deadlock-induced panic during a
later request.

This bug was exposed because of the higher-than-usual concurrency
level in the NetBSD rc scripts.  The fix consists of properly setting
the AMF_NOREPLY flag for asynchronous replies.

Change-Id: Iafafe2fdd67f212ecbf27a53862cefba2e4cf7e8

diff --git a/minix/servers/vm/pagefaults.c b/minix/servers/vm/pagefaults.c
index e851683d7cb735cb09d92e784d460b6dfd1f6a36..ff396415f47bca1b25be62a407e930fb4631eca0 100644 (file)
--- a/minix/servers/vm/pagefaults.c
+++ b/minix/servers/vm/pagefaults.c
@@ -197,7 +197,7 @@ static void handle_memory_continue(struct vmproc *vmp, message *m,
 
 static void handle_memory_final(struct hm_state *state, int result)
 {
-       int r;
+       int r, flag;
 
        assert(state);
        assert(state->valid == VALID);
@@ -215,9 +215,15 @@ static void handle_memory_final(struct hm_state *state, int result)
                        assert(state->caller == VFS_PROC_NR);
                        /* If a transaction ID was set, reset it */
                        msg.m_type = TRNS_ADD_ID(msg.m_type, state->transid);
-               }
+                       flag = AMF_NOREPLY;
+               } else
+                       flag = 0;
 
-               if(asynsend3(state->caller, &msg, 0) != OK) {
+               /*
+                * Use AMF_NOREPLY only if there was a transaction ID, which
+                * signifies that VFS issued the request asynchronously.
+                */
+               if(asynsend3(state->caller, &msg, flag) != OK) {
                        panic("handle_memory_final: asynsend3 failed");
                }