Thomas Cort [Tue, 22 Dec 2015 03:07:01 +0000 (03:07 +0000)]
mined: fix buffer overflow in input()
input() is used to accept filenames when saving, regular
expressions when searching, and other input. It writes
the characters into buffers such as file and exp_buf and
others which are of length LINE_LEN.
To prevent writing beyond the end of the intended buffer,
truncate the input at LINE_LEN - 1 and ensure that the
string is NULL terminated.
In order to resolve page faults on file-mapped pages, VM may need to
communicate (through VFS) with a file system. The file system must
therefore not be the one to cause, and thus end up being blocked on,
such page faults. To resolve this potential deadlock, the safecopy
system was previously extended with the CPF_TRY flag, which causes the
kernel to return EFAULT to the caller of a safecopy function upon
getting a pagefault, bypassing VM and thus avoiding the loop. VFS was
extended to repeat relevant file system calls that returned EFAULT,
after resolving the page fault, to keep these soft faults from being
exposed to applications.
However, general UNIX I/O semantics dictate that if an I/O transfer
partially succeeded before running into a failure, the partial result
is to be returned. Proper file system implementations may therefore
end up returning partial success rather than the EFAULT code resulting
from a soft fault. Since VFS does not get the EFAULT code in this
case, it does not know that a soft fault occurred, and thus does not
repeat the call either. The end result is that an application may get
partial I/O results (e.g., a short read(2)) even on regular files.
Applications cannot reasonably be expected to deal with this.
Due to the fact that most of the current file system implementations
do not implement proper partial-failure semantics, this problem is not
yet widespread. In fact, it has only occurred on direct block device
I/O so far. However, the next generation of file system services will
be implementing proper I/O semantics, thus exacerbating the problem.
To remedy this situation, this patch changes the CPF_TRY semantics:
whenever the kernel experiences a soft fault during a safecopy call,
in addition to returning FAULT, the kernel also stores a mark in the
grant created with CPF_TRY. Instead of testing on EFAULT, VFS checks
whether the grant was marked, as part of revoking the grant. If the
grant was indeed marked by the kernel, VFS repeats the file system
operation, regardless of its initial return value. Thus, the EFAULT
code now only serves to make the file system fail the call faster.
The approach is currently supported for both direct and magic grants,
but is used only with magic grants - arguably the only case where it
makes sense. Indirect grants should not have CPF_TRY set; in a chain
of indirect grants, the original grant is marked, as it should be.
In order to avoid potential SMP issues, the mark stored in the grant
is its grant identifier, so as to discard outdated kernel writes.
Whether this is necessary or effective remains to be evaluated.
This patch also cleans up the grant structure a bit, removing reserved
space and thus making the structure slightly smaller. The structure
is used internally between system services only, so there is no need
for binary compatibility.
With this change, obtaining an existing free grant is no longer an
operation of O(n) complexity. As a result, the now-deprecated
getgrant/setgrant part of the grants API also no longer has a
performance advantage.
The memory grant identifier for safecopies now includes a sequence
number in its upper bits, to prevent accidental reuse of a grant ID
after revocation and subsequent reallocation. This should increase
overall system robustness by a tiny amount, and possibly help catch
bugs in system services early on. For now, the lower 20 bits of the
grant ID are used as grant table slot index (thus allowing for up to
a million grants per process), and the next 11 bits of the (signed
32-bit) grant ID are used to store the per-slot sequence number. As
grant IDs are never exposed to userland, the split can be changed
later on without breaking the userland ABI.
Apply the x86 overflow check from git-d09f72c to ARM code as well.
Not just stack traces, but also system services can trigger this
case, possibly as a result of being handed bad pointers by userland,
ending in a kernel panic.
Changed all K&R style functions to ANSI-style declarations within the
kernel directory. The code compiles and aparently works for i386. For
arm my toolchain does not work, but I have changed the code with great
care. Also, the make command fails for the test suite. Therefore, I
strongly recommand to review the code with care.
Edited by David van Moolenbroek to convert really all K&R functions.
MIB: add support for System V IPC information node
The kernel.ipc.sysvipc_info node is the gateway from NetBSD ipcs(1)
and ipcrm(1) to the IPC server, and thus necessary for a clean
import of these two utilities. The MIB service implementation uses
the preexisting (Linux-specific) information calls on the IPC server
to obtain the information.
As mentioned in previous patches, services may not subscribe to
process events from specific processes only, since this results in
race conditions. However, the IPC server can safely turn on and off
its entire subscription based on whether any System V IPC semaphores
(and, in the future, message queues) are allocated at all. Since
the System V IPC facilities are not so commonly used, this removes
the extra round trip from PM to the IPC server and back for caught
signals and process exits in the common case.
- rewrite the semop(2) implementation so that it now conforms to the
specification, including atomicity, support for blocking more than
once, range checks, but also basic fairness support;
- fix permissions checking;
- fix missing time adjustments;
- fix off-by-one errors and other bugs;
- do not allocate dynamic memory for GETALL/SETALL;
- add test88, which properly tests the semaphore functionality.
PM: generic process event publish/subscribe system
Now that there are services other than PM and VFS that implement
userland system calls directly, these services may need to know about
events related to user processes. In particular, signal delivery may
have to interrupt blocking system calls, and certain cleanup tasks may
have to be performed after a user process exits.
This patch aims to implement a generic, lasting solution for this
problem, by allowing services to subscribe to "signal delivered"
and/or "process exit" events from PM. PM publishes such events by
sending messages to its subscribed services, which must then reply an
acknowledgment message.
For now, only the two aforementioned events are implemented, and only
the IPC service makes use of the process event facility.
The new process event publish/subscribe system replaces the previous
VM notify-sig/watch-exit/query-exit system, which was unsound: 1) it
allowed subscription to events from individual processes, and suffered
from fundamental race conditions as a result; 2) it relied on "not too
many" processes making use of the IPC server functionality in order to
avoid loss of notifications. In addition, it had the "ipc" process
name hardcoded, did not distinguish between signal delivery and exits,
and added a roundtrip to VM for all events from all processes.
Closer to KNF, better coding practices, more similar to other
services, no more global variables, a few more comments, that
kind of stuff. No major functional changes.
- switch to the NetBSD identifier system; it is not only better, but
also required for porting NetBSD ipcs(1) and ipcrm(1); however, it
requires that slots not be moved, and that results in some changes;
- synchronize some other things with NetBSD: where keys are kept, as
well as various non-permission mode flags;
- fix semctl(2) vararg retrieval and message field type;
- use SUSPEND instead of weird reply exceptions in the call table;
- fix several memory leaks and at least one missing permission check;
- improve the atomicity of semop(2) by a small amount, even though
its atomicity is still broken at a fundamental level;
- use the new cheaper way to retrieve the current time;
- resolve all level-5 LLVM warnings.
Specifically, add support for the IPC_INFO, SEM_INFO, and SEM_STAT
semctl(2) operations, similar to how information about shared memory
is already exposed as well. The MINIX3 ipcs(1) utility already had
support for these operations, and can now actually use them, too.
- About 80% of PM's process table consisted of per-signal sigaction
structures. This is information not used by the MIB service, and
can safely be stored outside the main process table.
- The MIB service does not need most of the VFS process table, so VFS
now generates a "light" version of its table upon request, with just
the fields used by the MIB service.
The result is a size reduction of the MIB service of about 700KB.
Instead of pulling in process tables itself, ProcFS now queries the
MIB service for process information. This reduces ProcFS's memory
usage by about 1MB. The change does have two negative consequences.
First, getting all the original /proc/<pid>/psinfo fields filled in
would take a lot of extra effort. Since the only program that uses
those files at all is mtop(1), we reformat psinfo to expose only the
information used by mtop(1). This means that with this patch, older
copies of MINIX3 ps and top will cease to work.
Second, since both MIB and ProcFS update their own view of the
process list only once per clock tick, ProcFS' view may now be
outdated by up to two clock ticks. This is unlikely to pose a
problem in practice.
Due to differences in (mainly) measuring and accumulating CPU times,
the two top programs end up serving different purposes: the NetBSD
top is a system administration tool, while the MINIX3 top (now mtop)
is a performance debugging tool. Therefore, we keep both.
The newly imported BSD top has a few MINIX3-specific changes. CPU
statistics separate system time from kernel time, rather than kernel
time from time spent on handling interrupts. Memory statistics show
numbers that are currently relevant for MINIX3. Swap statistics are
disabled entirely. All of these changes effectively bring it closer
to how mtop already worked as well.
No changes except for one cosmetic adjustment: NetBSD has chosen to
rename the standard TT column to TTY and not shorten tty names; we
undo those changes, making ps(1) behave more in accordance with the
specification and its manual page, and, most importantly for us, not
use an incredibly wide TTY column to print "console".
Adapt libc devname(3) to make use of it, so that such device name
queries are now several orders of magnitude faster. The database
is created and updated at system bootup time.
Imported with no changes, but not all parts are expected to be
functional. The libc nlist functionality is enabled for the
purpose of successful linking, although the nlist functionaly has
not been tested on MINIX3 nor is it needed for how we use libkvm.
In terms of function calls: kvm_getproc2, kvm_getargv2,
kvm_getenvv2, and kvm_getlwps are expected to work, whereas
kvm_getproc, kvm_getargv, kvm_getenvv, and kvm_getfiles are not.
Now that uname(3) uses sysctl(2), we no longer need sysuname(2).
Backward compatibility is retained for old statically linked
binaries for a short while.
Also remove the now-obsolete MINIX3-specific "arch" field from the
utsname structure. While this is an ABI break at the libc level,
it should pose no problems in practice, because:
- statically linked programs (i.e., all of the base system) are not
affected, as they will use headers synchronized with libc;
- the structure is getting smaller, thus, older dynamically linked
programs (typically in pkgsrc) using the new libc will end up with
garbage in the "arch" field, but it is unlikely they will use this
field anyway, since it was specific to MINIX3;
- new dynamically linked programs using an old libc could end up with
memory corruption, but this is not a scenario that is expected to
occur in the first place - certainly not with programs from pkgsrc.
PM uses its own process table entry as source for kernel signals,
and temporarily changes its own process group to make the signals
arrive at the right processes. However, the value is never reset,
with as result that the temporary value shows up in ps(1) output.
So far, VM reported only the number of bytes actually allocated to
each process. This patch adds two additional fields: the sum of the
byte sizes of all the virtual address ranges in the process, and that
number minus the part of the process stack that is not actually
mapped in. Unfortunately, we have to guess where the process stack
is, so the second field is not necessarily accurate.
This functionality is required for BSD top(1), as exposed through
the CTL_KERN KERN_CP_TIME sysctl(2) call. The idea is that the
overall time spent in the system is divided into five categories.
While NetBSD uses a separate category for the kernel ("system") and
interrupts, we redefine "system" to mean userspace system services
and "interrupts" to mean time spent in the kernel, thereby providing
the same categories as MINIX3's own top(1), while adding the "nice"
category which, like on NetBSD, is used for time spent by processes
with a priority lowered by the system administrator.
The new MIB service implements the sysctl(2) system call which, as
we adopt more NetBSD code, is an increasingly important part of the
operating system API. The system call is implemented in the new
service rather than as part of an existing service, because it will
eventually call into many other services in order to gather data,
similar to ProcFS. Since the sysctl(2) functionality is used even
by init(8), the MIB service is added to the boot image.
MIB stands for Management Information Base, and the MIB service
should be seen as a knowledge base of management information.
The MIB service implementation of the sysctl(2) interface is fairly
complete; it incorporates support for both static and dynamic nodes
and imitates many NetBSD-specific quirks expected by userland. The
patch also adds trace(1) support for the new system call, and adds
a new test, test87, which tests the fundamental operation of the
MIB service rather thoroughly.
The user of the script may now override the default name of the
host platform's GNU make utility by passing in a MAKE variable.
Along with the previous commits and upcoming documentation changes,
this fixes #93.
These utilities were already largely broken and are now also obsolete.
In addition, they have too many issues (for example, dependencies on
Linux specifics) to keep around. The few usable features left in the
clientctl script are not LLVM specific and, if anything, should be
recreated somewhere else.
ASR instrumentation is now performed on all applicable system services
if the system is built with MKASR=yes. This setting automatically
enables MKMAGIC=yes, which in turn enables MKBITCODE=yes.
The number of extra rerandomized service binaries to be generated can
be set by passing ASRCOUNT=n to the build system, where n is a number
between 1 and 65536. The default ASRCOUNT is 3, meaning that each
service will have one randomized base binary and three additional
rerandomized binaries. As before, update_asr(8) can be used for
runtime rerandomization.
The magic runtime library is now built as part of the regular build, if
the MKMAGIC=yes flag is passed to the build system. The library has
been renamed from "magic" to "magicrt" to resolve a name clash with BSD
file(1)'s libmagic. All its level-5 LLVM warnings have been resolved.
The final library, "libmagicrt.bcc", is now stored in the destination
library directory rather than in the source tree.
Until now, the program name of a service was always the file name
(without directory) of the service binary. The program name is used
to, among other things, find the corresponding system.conf entry.
With ASR moving to a situation where all rerandomized service binaries
are stored in a single directory, this can no longer be maintained.
Instead, the service(8) command can now be instructed to override the
service program name, using its new -progname option.
It was not used or tested on x86 in practice, and the automated arm
tests should obviate the need for a dummy-only x86 implementation.
It should be noted that this change is merely the simplest way to
deal with conflicts with live update (for the second time now).
Lionel Sambuc [Thu, 15 Oct 2015 15:01:16 +0000 (17:01 +0200)]
NetBSD re-synchronization of the source tree
This brings our tree to NetBSD 7.0, as found on -current on the
10-10-2015.
This updates:
- LLVM to 3.6.1
- GCC to GCC 5.1
- Replace minix/commands/zdump with usr.bin/zdump
- external/bsd/libelf has moved to /external/bsd/elftoolchain/
- Import ctwm
- Drop sprintf from libminc
The NetBSD libc malloc implementation performs its own out-of-memory
check, presumably for performance reasons. The check makes a strong
assumption about the address space layout, which is that memory-
mapped pages are always located above the heap. However, this
assumption does not necessarily hold on MINIX3, thus resulting in
malloc reporting an out-of-memory condition without the system
actually being out of memory at all. Evidence suggests that in
particular dynamically linked (i.e., pkgsrc) binaries were affected
by this issue - most notably git.
The minix set is now divided into minix-base, minix-comp, minix-games,
minix-kernel, minix-man and minix-tests.
This allows massive space savings on the installlation CD because only
the base system used for installation is stored uncompressed. Also, it
makes the system more modular.
Lionel Sambuc [Fri, 9 Oct 2015 12:36:26 +0000 (14:36 +0200)]
Upgrade releasetools/sort_set
- retire the old shell script
- import perl script
The perl scripts has the following advantages:
- The sorting should be more stable, even accross different OSes.
- The sorted output is automatically formatted into columns
- It is much faster, even on large inputs.
The CD now boots directly from the ISO 9660 filesystem instead of using
MBR partitioning with Minix file systems. This saves some space on the
CD and reduces memory requirements by some unknown amount as the root
ramdisk is completely eliminated.
The x86 hard drive image creation is also rewritten in the same
fashion.
The setup is modified to be more NetBSD-like (unpacking sets
tarballs instead of blindly copying the CD contents). Splitting MINIX
into sets is done in another commit due to it being a nightmare to
rebase.
Since MINIX lacks union mounts for now, a bunch of ramdisks are
generated at run-time to make parts of the filesystem writeable for the
CD. This solution isn't ideal, but it's enough for an installation CD.
isofs now uses an in-memory directory listing built on-the-fly instead
of parsing the ISO 9660 data structures over and over for almost every
request. This yields huge performance improvements.
The directory listing is allocated dynamically, but Minix servers aren't
normally supposed to do that because critical servers would crash if the
system runs out of memory. isofs is quite frugal, won't allocate memory
after having the whole directory tree cached and is not that critical
(its most important job is to serve as a root file system during
installation).
The benefits and elegance of this scheme far outweights this small
problem in practice.
It's a fix for correcting cd9660 device node creation in makefs. This
fix was commited in NetBSD on May 30, 2014, after the current NetBSD
source code import, hence the cherrypicking.
This patch adds support for the wait4 system call, and with that the
wait3 call as well. The implementation is absolutely minimal: only
user and system times of the exited child are returned (with all other
rusage fields left zero), and there is no support for tracers. Still,
this should cover the main use cases of wait4.
- the userland call is now made to PM only, and PM relays the call to
other servers as appropriate; this is an ABI change that will
ultimately allow us to add proper support for wait3() and the like;
for the moment there is backward compatibility;
- the getrusage-specific kernel subcall has been removed, as it
provided only redundant functionality, and did not provide the means
to be extended correctly in the future - namely, allowing the kernel
to return different values depending on whether resource usage of
the caller (self) or its children was requested;
- VM is now told whether resource usage of the caller (self) or its
children is requested, and it refrains from filling in wrong values
for information it does not have;
- VM now uses the correct unit for the ru_maxrss values;
- VFS is cut out of the loop entirely, since it does not provide any
values at the moment; a comment explains how it should be readded.
The current value was both wrong (counting spawned kernel signals
rather than delivered user signals) and returned for the calling
process even if the request was for the process's children.
For now we are better off not populating this field at all.
The current values were both inaccurate (especially for dynamically
linked executables) and using the wrong unit (bytes, instead of
kilobytes times ticks-of-execution). For now we are better off not
populating these fields at all.
POSIX states that times() and getrusage() should only return child
user and system times of terminated children for which wait*() has
returned their PIDs.
- move MINIX3-specific files into minix/lib/libpuffs;
- resynchronize the remaining files with NetBSD code;
- remove a few unnecessary changes;
- put remaining MINIX3-specific changes in __minix blocks;
- sort out the source files being linked at all.
The result is that libpuffs now successfully links against FUSE
file system programs again. It can successfully mount, perform
some of the most basic operations, and unmount the file system.
- no longer inject fewer faults than instructed;
- no longer apply a limit on the number of injected faults;
- refactory to allow for random faults (type 99);
- also allow for stop faults (type 50);
- massive dead code cleanup;
- move outdated test cruft into tests/ subdirectory; it is kept only
as an example of how to use swifi.
- move from minix/commands to minix/usr.sbin;
- install into /usr/sbin instead of /usr/bin;
- move manual page into source directory;
- resolve compilation warning;
- convert to KNF.