POSIX states that when interrupted, partially successful pipe
operations should return the partial result rather than EINTR. VFS
previously wouldn't look at the partial result, and not clear it
either, which would result in a panic upon the next pipe operation.
The main purpose of this patch is to fix handling of unpause calls
from PM while another call is ongoing. The solution to this problem
sparked a full revision of the threading model, consisting of a large
number of related changes:
- all active worker threads are now always associated with a process,
and every process has at most one active thread working for it;
- the process lock is always held by a process's worker thread;
- a process can now have both normal work and postponed PM work
associated to it;
- timer expiry and non-postponed PM work is done from the main thread;
- filp garbage collection is done from a thread associated with VFS;
- reboot calls from PM are now done from a thread associated with PM;
- the DS events handler is protected from starting multiple threads;
- support for a system worker thread has been removed;
- the deadlock recovery thread has been replaced by a parameter to the
worker_start() function; the number of worker threads has
consequently been increased by one;
- saving and restoring of global but per-thread variables is now
centralized in worker_suspend() and worker_resume(); err_code is now
saved and restored in all cases;
- the concept of jobs has been removed, and job_m_in now points to a
message stored in the worker thread structure instead;
- the PM lock has been removed;
- the separate exec lock has been replaced by a lock on the VM
process, which was already being locked for exec calls anyway;
- PM_UNPAUSE is now processed as a postponed PM request, from a thread
associated with the target process;
- the FP_DROP_WORK flag has been removed, since it is no longer more
than just an optimization and only applied to processes operating on
a pipe when getting killed;
- assignment to "fp" now takes place only when obtaining new work in
the main thread or a worker thread, when resuming execution of a
thread, and in the special case of exiting processes during reboot;
- there are no longer special cases where the yield() call is used to
force a thread to run.
The T_DUMPCORE implementation was not only broken - it would currently
produce a coredump of the tracer process rather than the traced
process - but also deeply flawed, and fixing it would require serious
alteration of PM's internal state machine. It should be possible to
implement the same functionality in userland, and that is now the
suggested way forward. For now, also remove the (identical) utilities
using T_DUMPCORE: dumpcore(1) and gcore(1).
Previously, processing of some replies coming from character drivers
could block on locks, and therefore, such processing was done from
threads that were associated to the character driver process. The
hidden consequence of this was that if all threads were in use, VFS
could drop replies coming from the driver. This patch returns VFS to
a situation where the replies from character drivers are processed
instantly from the main thread, by removing the situations that may
cause VFS to block while handling those replies.
- change the locking model for select, so that it will never block
on any processing that happens after the select call has been set
up, in particular processing of character driver select replies;
- clearly mark all select routines that may never block;
- protect against race conditions in do_select as result of the
locking that still does happen there (as is required for pipes);
- also handle select timers from the main thread;
- move processing of character driver replies into device.c.
These days, DEV_OPEN calls to character drivers block the calling
thread until completion or failure, and thus never return SUSPEND to
the caller. The same already applied to BDEV_OPEN calls to block
drivers. It has thus become impossible for a process to enter a state
of being blocked on a device open call.
There is currently no support for restarting device open calls to
restarted character drivers. This support was present in the _DOPEN
logic, but was already no longer triggering. In the future, this case
should be handled by the thread performing the open request.
Not doing so caused PFS to commit protocol violations by relying on
stale information when sending replies. This stale information always
happened to be correct, which is why the problem went unnoticed.
Change-Id: Ia42ca670718d6e731193cd2c34a3ff455f8a94d3
- change all sync char drivers into async drivers;
- retire support for the sync protocol in libchardev;
- remove async dev style, as this is now the default;
- remove dev_status from VFS;
- clean up now-unused protocol messages.
The async char protocol already has this, so this patch closes the
gap between the two protocols a bit. Support for this flag has been
added to all sync char drivers that support CANCEL at all.
The LOG driver was already using the asynchronous protocol, but it
did not support the nonblocking transfer flag. This has been fixed
as well.
As with w_task, this ensures that the field remains cleared if it is
not used. Without this, worker_stop could mistakenly identify a thread
as talking to a device driver rather than a (crashed) file server.
Thomas Veerman [Sat, 24 Aug 2013 10:23:41 +0000 (12:23 +0200)]
VFS: set w_task only when needed
It was always set, but not always cleared, when talking to asynchronous
drivers. This could cause erratic behavior upon a driver crash.
Normally, a worker thread's w_task field is set when it's about to
communicate with a driver or FS. Then upon receiving a reply we can
do sanity checks (that the thread we want to wake up was actually
waiting for a reply). Also, when a driver/FS crashes, we can identify
which worker threads were talking to the crashed endpoint and handle
the error gracefully.
Asynchronous drivers are a bit special, though. In most cases, the
sender of the request is not interested in the reply (the sender was
suspended and only wants to know whether the request was successfully
caried out or not). However, the open request is special, as the reply
carries information needed by the sender. This is the only request
where a worker thread actually yields and waits for the result. This is
also the only case where we're interested in setting w_task for
asynchronous drivers.
* Removed startup code patches in lib/csu regarding kernel to userland
ABI.
* Aligned stack layout on NetBSD stack layout.
* Generate valid stack pointers instead of offsets by taking into account
_minix_kerninfo->kinfo->user_sp.
* Refactored stack generation, by moving part of execve in two
functions {minix_stack_params(), minix_stack_fill()} and using them
in execve(), rs and vm.
* Changed load offset of rtld (ld.so) to:
execi.args.stack_high - execi.args.stack_size - 0xa00000
which is 10MB below the main executable stack.
The main motivation for this change is that only Loris supports
multithreading, and Loris supports dynamic thread allocation, so the
number of supported threads can be implemented as a bit flag (i.e.,
either 1 or "at least as many as VFS has"). The ABI break obviates the
need to support file system versioning at this time, and several
other aspects are better implemented as flags as well. Other changes:
- replace peek/bpeek test upon mount with FS flag as well;
- mark libsffs as 64-bit file size capable;
- remove old (3.2.1) getdents support.
- pass in file system type through mount(2), and return this type in
statvfs structures as generated by [f]statvfs(2);
- align mount flags field with NetBSD's, splitting out service flags
which are not to be passed to VFS;
- remove limitation of mount ABI to 16-byte labels, so that labels
can be made larger in the future;
- introduce new m11 message union type for mount(2) as side effect.
The following types are modified (old -> new):
* _BSD_USECONDS_T_ int -> unsigned int
* __socklen_t __int32_t -> __uint32_t
* blksize_t uint32_t -> int32_t
* rlim_t uint32_t -> uint64_t
On ARM:
* _BSD_CLOCK_T_ int -> unsigned int
On Intel:
* _BSD_CLOCK_T_ int -> unsigned long
Lionel Sambuc [Fri, 23 Aug 2013 18:27:27 +0000 (20:27 +0200)]
Adapt the type used for adjtime_delta
clock_t is currently a signed type, but in NetBSD this is not the
case. As we plan on aligning our types we have to change this as this
prevents negative delta from being correctly used.
Ben Gras [Wed, 8 Jan 2014 16:43:19 +0000 (17:43 +0100)]
VM bugfix & regression test
The bug in the offset correction code for the 'shrink region from
below' case can easily case an assert(foundregion->offset == offset)
to trigger (if the blocks are touched afterwards, e.g. on fork())
as the offsets become wrong. This commit is a fix & regression test.
We currently have a few of the POSIX tests failing because filemmap
is enabled by default. The working assumption is that these program
pass a pointer to the file server that points to a not yet loaded
data segment. When the file server tries to access that data it
therefore generates a pagefault and a call to itself it can not
handle because it is unable to first return the current call.
arm:make the kernel server and driver binaries identical.
Make the kernel server and driver binaries indentical for the different
ARM platforms. We no longer need to define the AM335X or DM37XX flags
during compilation. The remaining differences are all located in u-boot.
arm:refactor replace cmdline.txt by bootargs passed to the kernel.
Put the boot arguments in uEnv.txt and not in cmdline.txt to allow
a more dynamic configuration of the system. We now also pass the
u-boot board_name parameter to the kernel.
arm:manage versioning of u-boot and upgrade u-boot
Replaced the wget download of u-boot by a versioned git checkout
this allows us to better manage the u-boot and MLO version we ship
while still allowing us to build ofline.
This changes replaces the BASE_URL setting by U_BOOT_BIN_DIR and
also updates to a newer build of u-boot.
arm:add board_id to machine to enable runtime configuration.
Modified the machine struct in include/minix/type.h to have an
additional field called board_id. This fields can be read out
by userland and drivers at runtime to enable automatic
configuration. The board_id field contains information about
the hardware architecture / board and such.
Ben Gras [Tue, 19 Nov 2013 16:01:23 +0000 (16:01 +0000)]
testmfs, testisofs: MFS format, ISO functionality
testmfs: catch MFS format changes
This test tests mkfs.mfs will generate the same FS image given the same
input files. mkproto creates a proto file (normalizing directory entry
order). The assumption is that a change in the output flags a tacit
change in FS format, and that a FS format change will cause the image
to change.
. Changes to mkfs.mfs that innocently change the format can
change the sha1 output in the script along with it.
. The assumption is that corresponding versions of mkfs.mfs and
MFS will always work together; otherwise a lot breaks (ramdisk etc.)
. Therefore, as long as a generated FS image stays the same with the
same input now, incompatible MFS changes will still be flagged,
even if they work together with the current mkfs.mfs.
testisofs: test ISO filesystem
. to test isofs: prepare an ISO FS image using writeisofs, copy it
to a RAM device, mount it using the iso9660fs server, compare the
SHA1 contents of the files on the ISO with the inputs.
. use su to run certain commands in the script as root
run script: run shell script tests
. they are installed without .sh so should be
searched for as such
. add diagnostic when tests are skipped
Ben Gras [Tue, 19 Nov 2013 15:59:52 +0000 (15:59 +0000)]
libminixfs: allow non-pagesize-multiple FSes
The memory-mapped files implementation (mmap() etc.) is implemented with
the help of the filesystems using the in-VM FS cache. Filesystems tell it
about all cached blocks and their metadata. Metadata is: device offset and,
if any (and known), inode number and in-inode offset. VM can then map in
requested memory-mapped file blocks, and request them if necessary.
A limitation of this system is that filesystem block sizes that are not
a multiple of the VM system (and VM hardware) page size are not possible;
we can't map blocks in partially. (We can copy, but then the benefits of
mapping and sharing the physical pages is gone.) So until before this
commit various pieces of caching code assumed page size multiple
blocksizes. This isn't strictly necessary as long as mmap() needn't be
supported on that FS.
This change allows the in-FS cache code (libminixfs) to allocate any-sized
blocks, and will not interact with the VM cache for non-pagesize-multiple
blocks. In that case it will also signal requestors, by failing 'peek'
requests, that mmap() should not be supported on this FS. VM and VFS
will then gracefully fail all file-mapping mmap() calls, and exec() will
fall back to copying executable blocks instead of mmap()ping executables.
As a result, 3 diagnostics that signal file-mapped mmap()s failing
(hitherto an unusual occurence) are disabled, as ld.so does file-mapped
mmap()s to map in objects it needs. On FSes not supporting it this situation
is legitimate and shouldn't cause so much noise. ld.so will revert to its own
minix-specific allocate+copy style of starting executables if mmap()s fail.
Ben Gras [Tue, 19 Nov 2013 15:58:05 +0000 (15:58 +0000)]
mkfs.mfs, mkproto: minor features
. mkfs.mfs: -T option to set timestamp of files on FS
. mkproto: normalize (sort) order of directory entries
. mkproto bugfix: always print mode in 3 digits (%03o)
Thomas Cort [Mon, 18 Nov 2013 23:00:57 +0000 (18:00 -0500)]
usr.bin/calendar: re-sync with upstream.
Jenkins was failing on usr.bin/calendar/calendar.c:386:6: with
"error: 'fd' may be used uninitialized in this function"
when the level of error checking was turned up.
The error was recently fixed upstream. This commit updates
calendar.c to upstream rev 1.50, updates calendar files too.
Thomas Cort [Sun, 10 Nov 2013 17:06:27 +0000 (12:06 -0500)]
eepromread: support for reading from /dev/eeprom
eepromread could only read EEPROMs through the /dev/i2c interface.
Once the cat24c256 driver is started and claims/reserves the
device, it can no longer be read through the /dev/i2c interface.
This patch adds support for reading from EEPROMs through the
/dev/eeprom interface. For example, to read the on-board eeprom
on the BBB, one would do `eepromread -f /dev/eepromb1s50 -i`.
Ben Gras [Tue, 5 Nov 2013 14:43:44 +0000 (15:43 +0100)]
x86_hdimage.sh -i: crossbuild x86 release CD
. build writeisofs as a native tool too for it
. also mkfs.mfs: make missing file in proto nonlethal
. make setup script a little more self-sufficient
. hdboot: use INSTALL_FILE instead of INSTALL so that the
results get added to the METALOG