Threads need to pick a blocking mechanism and somehow stick to it reliably, so there can be a neat and tidy Thread::kill function.  A signal should knock a thread out of anything, but that can still have races.  It would be nice if Thread::kill didn't have to be virtual, and if it could be worked into a generalized thread-to-thread messaging paradigm, but that's problematic when one of the major purposes of having threads in the first place is to be able to use other polling mechanisms in separate threads, i.e. blocking for libpcap or X11 or whatever.

Maybe it would be best to remove virtual void main() and instead have a worker-thread model, where threads can perform work, wait, or be signalled.  The work, wait and signal functions would be virtual, which would hide a lot of the complexity about whether it's okay to block, or whether we've been signalled.  And then threads could determine their own needs for block and signal; most threads might use EventPoll to do that work, but some might use something like pcap or X11 instead.  No inter-thread messaging at a low-level, but maybe some higher-level interfaces that could be implemented on top of them.

If moving to a worker model, why not provide a work unit, too, and the possibility of some messaging primitives?  That also has the benefit of mirroring some of the hardware models it'd be nice for the event system and threading to be compatible with, like that of Octeon.  And it allows for lossless messaging between threads, whereas signal/wait can have problems with races and spurious wakeups, right?  Especially if there's any chance of the signal being consumed by something like a system call, rather than an instance of wait.  But with some slightly-heavyweight locking, there's no chance of those kinds of races, because e.g. the thread state lock would be held during work processing.  That nearly serializes processing, though.  *Unless* we have a lockless work-queueing model in addition to a wakeup mechanism.  Right?  Then we can guarantee that a thread will acquire a lock, check for work, and only then block.  There's still races, though, such as with pcap, or X11, which won't let us specify a lock to relinquish at wait time.  Argh!

Maybe too much generalization is a curse, and it would be better to find a limited and reliable model to support, and make sure all the applications we care about in the immediate future are plausible.

Why not just register threads that need to generate callbacks as an "EventSource" with the EventThread/EventSystem?  This is non-ideal for a world in which we would have multiple EventThreads but it's a start for now.  But with this, the EventSource could include a pipe to read/write on, which would be very neat and tidy.

I tried moving towards making EventThread the normative model for threads, and parameterizing anything using callbacks on EventThread (as a CallbackScheduler mostly, but also for timeouts, although there's no reason to not have timeouts in a different thread, and merely schedule them back to EventThread.)  This is OK, but mostly devolves into single-threading anyway.

If polling code is simplified a little further and moved to common/poll/..., it makes sense indeed to move polling into *every* thread, and to use pipes to communicate between threads (or, where possible, we could use user events like with kqueue.)  That can all mostly be done under the hood after the fact for real if we begin with the EventSource model, in which new sources are registered with the EventSystem.  Then we have one-direction communication, which is probably most of what we need anyway.  Right?

Actually, to avoid races, the method of inter-thread communication must be a condvar so that we can savely use mutexes.  Otherwise we can go to sleep with work waiting, right?  Unless we get really fussy about the data on the inter-thread pipe, but why not just use condvars instead?  Then we just need to have a WorkerThread which has block/signal/work abstractions, and make EventPoll one of the few that uses something other than the default.  And have a single poll thread, of course.  Yes, this doesn't solve the pcap race problem, but nothing will; pcap is thread hell.