The QNX kernel doesn't do anything with strings. No parsing, no formatting, no messages.

Linux suffers from being too bloated for real time. Millions of lines of kernel, all of which have to be made preemptable. It's the wrong architecture for real time. So it took two decades to try to fix this.

L4's contribution was speed. It was assumed that microkernels, and especially capability-based microkernels were fundamentally slower than monolithic kernels. This is why Linux (1991) is monolithic. Yet L4 (1994) was the fastest operating system in existence at the time, despite being a microkernel and capability based. It's too bad those dates aren't reversed, or we might have had a fast, capability-based, microkernel Linux :(

Oh, and anytime as needed the OS could serialize out your process, pipe it across the network to another machine, and resume. Useful for load balancing, or relocating a program to be near the data it is accessing. Unless your program pays special attention to the clock, it wouldn't notice.

I still think about Amoeba from time to time, and imagine what could have been if we had gone down that route instead.

E.g. You spawn a process on another computer and then the connection drops.

" We took a substantially different approach with seL4; its model for managing kernel memory is seL4’s main contribution to OS design. Motivated by the desire to reason about resource usage and isolation, we subject all kernel memory to authority conveyed by capabili- ties (except for the fixed amount used by the kernel to boot up, including its strictly bounded stack). "

I guess I should've said seL4 took capabilities to the extreme.

I'm not bothered by the kernel bloat. There's a lot of dev time being invested in Linux and while the desktop is not as much of a priority as say the server space a performant kernel on handhelds and other such devices and the dev work to get it there will benefit the desktop users like myself.

In the end, all I could practically use it for was as a bluetooth audio connector.

A bunch of similar embedded systems. And blackberry, if anyone's still using them.

So much like a well structured main method in a C program or other C like language where main just orchestrates the calling of other functions and such. In this case main might initialize different things where the QNX kernel doesn't but the idea or general concept remains.

I'm no kernel dev but this sounds good to me. Keeps things simple.

The process of serving an HTTP request is broken into a large number of fine grained stages and plugin modules may hook into any or all of these to modify the input and output to each stage.

The same basic idea makes it easy to turn any application concept into a modules-and-core architecture. From the day I read (skimmed) that book a decade or two ago this pattern has been burned into my brain

Pretty much everywhere there is some undercurrent of "use this ultra-small generic interface for everything and life will be easier". With games and ECS, microkernels and IPC-for-everything, with frontend frameworks and components that only communicate between themselves via props and events, with event sourcing and CQRS backends, Actors in Erlang, with microservices only communicating via the network to enforce encapsulation... Perhaps even Haskell's functional-core-imperative-shell could count as that?

I feel like OOP _tried_ to get to this point, with dependency injection and interface segregation, but didn't quite get there due to bad ergonomics, verbosity and because it was still too easy to break the rules. But it was definitely an attempt at improving things.

https://netflixtechblog.com/kubernetes-and-kernel-panics-ed6...

I understand it's less elegant, efficient, etc. But sometimes commodity wins over correctness.

For automotive and aerospace, you really want the control systems to be able to run no matter what.

I'm unclear on why you put "many cycles to spare for misses" in quotes, as if it's unimportant. If a linux/arm (or x86) solution is displacing a much lower speed "real real time" solution, that's the situation...the extra cycles mean you can tolerate some misses while still being as granular as what you're replacing. Not for every use case, but for many.

As for the quotes, it was a direct citation, not a way to dismiss what you said.

Actual realtime systems are about workload scheduling that takes into account processing deadlines. Hard realtime systems can make guarantees about processing latencies, and can preemptively kill or skip tasks if the result would arrive too late. But this is not something that the Linux kernel can provide, because it is a system property rather than about just the kernel: you can't provide any hard guarantees if you have no time bounds for your data processing workload. So any discussion about -rt in the context of the Linux kernel will always be about soft realtime only.

In the case of the kernel, it is indeed too large to be considered hard realtime. Best case we can make it into a firmer realtime than it currently is. But I would place it nowhere near avionics flight calculators (like fly-by-wire systems).

For example, if I have a video game where there's collision against the walls, I can understand this as potentially colliding against "multiple things simultaneously", since I'm likely to describe the scene as a composite of bounding boxes, polygons, etc.

But to get an answer for what to do in response when I contact a wall, I have to come up with an algorithm that tests all the relevant shapes or volumes.

The concurrency bug that appears when doing this in a naive way is that I test one, give an answer to that, then modify the answer when testing the others. That can lead to losing information and "popping through" a wall. And the direction in which I pop through depends on which one is tested first.

The conventional gamedev solution to that is to define down the solution set so that it no longer matters which order I test the walls in: with axis aligned boxes, I can say "move only the X axis first, then move only the Y axis". Now there is a fixed order, and a built-in bias to favor one or the other axis. But this is enough for the gameplay of your average platforming game.

The generalization on that is to describe it as a constraint optimization problem: there are some number of potential solutions, and they can be ranked relative to the "unimpeded movement" heuristic, which is usually desirable when clipping around walls. That solution set is then filtered down through the collision tests, and the top ranked one becomes the answer for that timestep.

Problems of this nature come up with resource allocation, scheduling, etc. Some kind of coordinating mechanism is needed, and OS kernels tend to shoulder a lot of the burden for this.

It's different from real-time in that real-time is a specification of what kind of performance constraint you are solving for, vs allowing any kind of performance outcome that returns acceptable concurrent answers.

I bet there are hard-realtime (commercial) OSes running on ARM, and the ability to use a lower-specced (cheaper, simpler, consuming less power) core may be seen as an advantage enough to pay for the OS license.

The power issue is real, but it might well be the same price or cheaper - a standard ARM that gets stamped out by the million can cost less than a "simpler" microcontroller with a smaller production run.

Maybe they used the quotes because they were quoting you, haha.

Perhaps even that's not an example of such a system, 0.3 microseconds is close to the allowable real-time budget, and QC would probably not scrap a $20k part if you were off by that much once.

But in practice, every time I've heard "soft real time" suggested, the failure mode is not a sub-microsecond miss but a 100 millisecond plus deadlock, where a hardware watchdog would be needed to drop the whole system offline and probably crash the tool (hopefully fusing at the tool instead of destroying spindle bearings, axis ball screws, or motors and gearboxes) and scrap the part.

I’m trying to understand where the roadblock on a rPi + small FPGA hybrid board for $50 fails at the task… and it sounds like the OS/firmware doesn’t suffice. (Or a SoC, like a Zynq.)

Eg, if we could guarantee that the 1.5GHz core won’t “be off” by more than 1us on responding and the FPGA can manage IO directly to buffer out (some of) the jitter, then the cost of many hobby systems with “(still not quite) hard” real time systems would come down to reasonable.

But it's not really so much about being fast, it's about being able to guarantee that your system can respond to an event within a given amount of time every time. (At least that is how a friend who works in embedded/real time explained it to me.)

Any system that deadlocks.

If you really need realtime, then you really need it and "close enough" doesn't really exist.

This is just my perception as an outsider though.

1) There is always a possibility that something fails to run by its due date. Planes crash sometimes. Cars won't start some times. Factory machinery makes scrap parts sometimes. In a great many applications, missing a real time deadline results in degraded quality, not end of life, or regional catastrophy. The care that must be taken to lower the probability of failure needs to be in proportion to the consequence of the failure. Airplanes have redundant systems to reduce (but not eliminate) possibility of failure, while cars and trucks generally don't.

2) Even in properly working real time systems, there is a tolerance window on execution time. As machines change modes of operation, the amount of calculation effort to complete a cycle changes. If the machine is in a warm up phase, it may be doing minimal calculations, and the scan cycle is fast. Later it may be doing a quality control function that needs to do calculations on inputs from numerous sensors, and the scan cycle slows down. So long as the scan cycle doesn't exceed the limit for the process, the variation doesn't cause problems.

Much easier to just use an RTOS and save the expensive testing.

Most software cares about the average case, or, in the case of the Windows 10/11 start menu animation, the average across all supported machines apparently going 20 years into the future.

Examples of the latter include audio and video playback and video games. Nobody wants pauses or glitches, but if you get one once in a while, nobody dies. So people deliver these on non-RT operating systems for cost reasons.

No. This is a common misconception. The distinction between a hard realtime system and a soft realtime system is simply whether missing a timing deadline leads to a) failure of the system or b) degradation of the system (but the system continues to operate). Safety is not part of it.

Interacting with the real physical world often imposes “hard realtime” constraints (think signal processing). Whether this has safety implications simply depends on the application.

On the one hand they aren't safety critical.

On the other, I can imagine someone getting chewed out or even fired for a pause or a glitch in a professional performance.

Probably the same with live commercial video compositing.

But there are plenty of performers who apparently rely on Linux boxes and gumption.

Do you really stand by the statement of this rhetorical question? Because if yes: this attitude is a big reason for why web apps are so unpleasant to work with compared to locally running applications. Depending on the application, even 16ms vs 32ms can make a big difference.

Even hard-realtime systems have a failure rate, in practice if not in theory - even a formally verified system might encounter a hardware bug. So it's always a case of tradeoffs between failure rate and other factors (like cost). If commodity operating systems can push their failure rate down a few orders of magnitude, that moves the needle, at least for some applications.

FWIW by-device/by-transistor-count, the bulk of "hard realtime systems" with millisecond-scale latency requirements are just audio.

The sexy stuff are all real applications too. But mostly we need this just so we don't hear pops and echos in our video calls.

[1] Actually even that's wrong: for sure there are teams of people within MS (and Apple, and anyone else in this space) measuring latency behavior at the top-level app layer and doing tuning all the way through the stack. App latency excursions can impact streams too, though ideally you have some layer of insulation there.

(I didn't know the term, trying to be helpful if others don't)

https://www.audioadvice.com/audioquest-nrg-dragon-high-curre...

linux RTOS is at microseconds granularity but it still can not 100% guarantee it, anything in cache nature (L2 cache, TLB miss) are hard for hard real time.

a dual kernel with xenomai could improve it, but it is not widely used somehow, only used in industrial controls I think.

linux RT is great for audio, multimedia etc as well, where real-time is crucial, but not a MUST.

yup that's why you'd pin the memory and the core for the critical task. which, alas, will affect performance of the other cores and all other tasks. and whoosh there goes the BOM...

which again as we both probably are familiar with leads to the SoC designs with a real time core microcontroller and a HPC microprocessor on the same package. which leads to the question how to architect the combined system of real-time microcontroller and compute power but soft real time microprocessor such that the overall system remains sufficiently reliable...

oh joy and fun!

Even if a system can run (RT-)Linux, it doesn’t mean it’s suitable for real-time. Hardware for real-time projects needs much lower interrupt latency than a lot of hardware provides. Preemption isn’t the only thing necessary to support real-time requirements.

But it all depends on what your application is. There are a lot of applications that are "almost real-time" in need. For those, this might be useful.

But you can do all these proposed "soft" use cases with embedded Linux today. It's not like low-latency software video or audio playback is not possible, or wasn't possible twenty years ago. You only run into problems on busy systems where non-preemptible I/O could regularly get in the way. That's seldom a concern in embedded environments.

I think there are compelling reasons for making the kernel fully-preemptible, giving people more control over scheduling, and so forth. But these reasons have relatively little to do with wanting Linux to supersede minimalistic realtime OSes or bare-metal code. It's just good hygiene that will result in an OS that, even in non-RT applications, behaves better under load.

Big complex hardware is a no-no for true real-time.

That's why AbsInt and WCET tools mainly has simple CPU architectures. 8051 will truly live forever.

btw, Zephyr RTOS.

1. Suppliers have not given you sufficient information for you to be able to prove an upper bound on the time taken. (That must happen a lot.)

2. The system is so complicated that you are not totally confident of the correctness of your proof of the upper bound.

3. The only upper bound that can prove with reasonable confidence is so amazingly bad that you'd be better off with cheaper, simpler hardware.

4. There really isn't a worst case. There might, for example, be a situation equivalent to "roll the dice until you don't get snake eyes". In networking, for example, sometimes after a collision both parties try again after a random delay so the situation is resolved eventually with probability one but there's no actual upper bound. A complex CPU and memory system might have something like that? Perhaps you'd be happy with "the probability of this operation taking more than 2000 clock cycles is less than 10^-13" but perhaps not.

The bounds are usually in CPU cycles, so a faster CPU can sometimes be used even if it takes more cycles. CPUs capable of running Linux usually have higher latency (in cycles) than microcontrollers, but as long as that can be kept under the (wall clock) duration limits with bounded-time it's fine. There will still be cases where the worst-case latency to fetch from DRAM in an RT-Linux system will be higher than a slower MCU fetching from internal SRAM, so RT-Linux won't take over all these systems.

https://www.etas.com/en/applications/etas-middleware-solutio...

And already on many bigger MCUs, GPIOs are just separate blocks on a shared internal bus like AHB/APB that connects together all the chip IP, causing unpredictable latencies.

There are advanced real time cores like the Arm Coretex-R82. In fact many real time systems are becoming quite powerful due to the need to process and aggregate ever increasing amounts of sensor data.

SpaceX uses x86 processors for their rockets. That small drone copter NASA put on Mars uses "big-ish" ARM cores that can probably run older versions of Android.

https://news.ycombinator.com/item?id=26907669

"We have Best-Effort and Guaranteed-Delivery APIs"

"I want Guaranteed Delivery!!!"

"If the GD logging interface is offline or slow, you'll take downtime; is that okay?"

"NO NO Must not take downtime!"

"If you need it logged, and can't log it, what do you do?"

These days I just point to the CAP theorem and suggest that logging is the same as any other distributed system. Because there's a wikipedia article with a triangle and the word "theorem" people seem to accept that.

[edit: added "GD" to clarify that I was referring to the guaranteed delivery logging api, not the best effort logging API]

What people probably mean is "logging is important, let's avoid losing log messages if possible", which is what "best" in "best effort" stands for. For example it's often a good idea to have a local log queue, to avoid data loss in case of a temporary log server downtime.

Some systems the logs are journaled records for the business or are discoverable artifacts for compliance. In highly secure environments logs are not only durable but measures are taken to fingerprint them and their ordering (like ratchet hashing) to ensure integrity is invariant.

I would note that using disk based logging is generally harmful in these situations IMO. Network based logging is less likely to cause blocking at some OS level or other sorts of jitter that’s harder to mask. Typically I develop logging as an in memory thing that offloads to a remote service over the network. The durability of the memory store can be an issue in highly sensitive workloads, and you’ll want to do synchronous disk IO for that case to ensure durability and consistent time budgets, but for almost all application disk less logging is preferable.

Depends on your failure model. I'd consider e.g. "received in memory by at least 3/5 remote servers in separate datacenters" to be safer than "committed to local disk".

In a network partition, you are either offline or your data is not consistent.

If you're writing local to your system, you're losing data if there's a single device failure.

https://en.wikipedia.org/wiki/CAP_theorem

People often don't think precisely about what they say or want, and also often don't think through corner cases such as "what if XYZ breaks or gets slow?"

And don't get me started on "log" messages that are 300mb events. Per log. Sigh.

For companies that sell API access logs in one form or another are how bills are reconciled and usage metered.

Having a local queue doesn't mean logging to the service is delayed, it can be sent immediately. All the local queue does is give you some resiliency, by being able to retry if the first logging attempt fails.

If an OOM happens, all bets are off anyway, since it has as much likelihood of taking out your application as it does your buffering code. The local buffering code might very well be part of the application in the first place, so the fate of the buffering code is the same as the application anyway.

It seems you're trying very hard to contrive a situation where doing nothing is better than taking reasonable steps to counter occasional network hiccups.

I think you've completely misunderstood me then. I haven't taken a stance at all on what should be done. I'm only trying to agree with the grandparent poster about logging ultimately reflecting CAP Theorem.

To me, those different requirements imply that they should be treated differently by the code, probably even under distinct flows: synchronously, and ideally to somewhere that I can later compress like hell and store in some very cheap long term storage.

Whereas the debug logs that I use for debugging? Rotate out after 30 to 90d, … and yeah, best effort is fine.

(The audit logs might also end up in one's normal logs too, for convenience.)

> [edit: added "GD" to clarify that I was referring to the guaranteed delivery logging api, not the best effort logging API]

i read GD as god-damned :-)

You can have both guaranteed delivery and no downtime if your whole system is so deterministic that anything that normally would result in blocking just will not, cannot happen. In other words it should be a hard real-time system that is formally verified top to bottom, down to the last transistor. Does anyone actually do that? Verify the program and the hardware to prove that it will never run out of memory for logs and such?

Continuing this thought, logs are probably generated endlessly, so either whoever wants them has to also guarantee that that they are processedand disposed of right after being logged... or there is a finite ammount of log messages that can be stored (arbitrary number like 10 000) but the user (of logs) has to guarantee that they will take the "mail" out of the box sooner than it overfills (at some predictable, deterministic rate). So really that means even if OUR system is mathematically perfect, we're just making the downtime someone elses problem - namely, the consumer of the infinite logs.

That, or we guarantee that the final resources of our self-contained, verified system will last longer than the finite shelf life of the system as a whole (like maybe 5 years for another arbitrary number)

Promtail[1] can do this if you're using Loki for logging.

[1] https://grafana.com/docs/loki/latest/send-data/promtail/

a) If your service crashes and it's in-memory, you lose logs

b) If your service can't push logs off (upstream service is down or slow) you either drop logs, run out of memory, or block

In other words it's good enough.

E.g., a service I wrote wrote logs to an ELK setup; we logged over TCP. But the logging was async: we didn't wait for logs to make it to ELK, and if the logging services went down, we just queued up logs locally. (To a point; at some point, the buffer fills up, and logs were discarded. The process would make a note of this if it happened, locally.)

I agree, when stuff is properly written. I don't remember the exact details, but at least with UDP the asyncness is built-in: there is no backpressure whatsoever. So poorly written software can just send udp to heart's end.

Reminds me a litte of the oldtimers trick of adding a sleep(1000) somewhere so they could later come back and have some resources later, or if they needed a quick win with the client.

Now cloud companies are using malloc(300000000) it to fake resource usage. /s

I haven’t really thought about it since then, but I can imagine lots of used cases for something like an embedded application with raspberry pi where you don’t quite want to make the leap into a microcontroller running an RTOS.

I don't recall if this was just an idea or was actually implemented. I also have seen only the one mention of RpiOS on an RTOS so I'm curious about that.

I've worked with systems that ran Linux as a task of uITRON as well as threadX, both on somewhat obscure ARM hardware. Linux managed the MMU but had a large carveout for the RTOS code. They had some strange interrupt management so that Linux could 'disable interrupts' but while Linux IRQs were disabled, an RTOS IRQ could still fire and context switch back to an RTOS task. I haven't seen anything like this on RPi though, but it's totally doable.

I think I'm right in asserting that a fully-upstreamed RT kernel won't actually do anything different from a normal one unless you're actually running RT processes on it. The reason it's taken so long to upstream has been the trade-offs that have been needed to enable RT, and (per the article) there aren't many of those left.

The sort of thing it could help with is servicing hardware that does run hard realtime. For example, you have an RTOS doing direct control of a robot or medical device or whatever, and you have a UI pendant or the like that a user is interacting with. If linux on that pendant can make some realtime latency guarantees, you may be able to simplify communication between the two without risking dropping bits on the floor.

Conversely, for the common user it could improve things like audio/video streaming, in theory but I haven't looked into details or how much trouble there is currently.

Is there any known upper bound on, say, how long a memory access instruction takes on x86?

But for things that really matter, I've tested by configuring the MMU to disable caching for the memory that the realtime code lives in and uses to emulate 0% hitrate. And there's usually still a fair amount of variance on top of that depending on if the memory controller has a small cache, and where the memory controller is in its refresh cycle.

For soft real time, I don't see a problem. But for hard real time, it seems a bit scary.

I've had to compile&install real-time kernel as a requirement for a robot arm (franka) control computer. It would be nice to know a bit more than just how to install the kernel.

Generally, having experience with Greenhills in a previous job, for personal projects like robotics or control systems I would recommend programming a microcontroller directly rather than dealing with SoC with RTOS. Modern STM32s with Cortex chips have enough processing power to run pretty much anything.

A common logging strategy is to use a ring buffer with a separate isolated process reading from the ring. The vast majority of the time the ring buffer handles temporary disruptions (eg slow disk I/O to write messages to disk) but in the rare failure scenarios you simply overwrite events in the buffer and increment an atomic overwritten event counter. Events do not get silently dropped but you prioritize forward progress at the cost of data loss in rare scenarios.

Microkernels and pushing everything to userspace just moves the tradeoffs around. If your driver is in userspace and blocks writing a log message because the log daemon is blocked or the I/O device it is writing the log to is overloaded it does the same thing. Your realtime thread won't get what it needs from the driver within your time limit.

It all comes down to CAP theorem stuff. If you always want the kernel (or any other software) to be able to make forward progress within specific time limits then you must be willing to tolerate some data loss in failure scenarios. How much and how often it happens depends on specific design factors, memory usage, etc.

Unfortunately, within a kernel that’s as big as Linux, that would leave you with many, many, many counters. All of which need to be exported and monitored somehow.

That is, I would expect moving to counters to be less information, period. That not the case?

In Zephyr, we have a synchronous printk() too, as for low-level debugging and platform bringup that's usually desirable (i.e. I'd like to see the dump from just before the panic please!).

For production logging use, though, there is a fancier log system[1] designed around latency boundaries that essentially logs a minimally processed stream to a buffer than then gets flushed from a low priority thread. And this works, and avoids the kinds of problems detailed in the linked article. But it's fiddly to configure, expensive in an RTOS environment (you need RAM for that thread stack and the buffer), depends on having a I/O backend that is itself async/low-latency, and has the mentioned misfeature where when things blow up, it's usually failed to flush the information you need out of its buffer.

[1] Somewhat but not completely orthogonal with printk. Both can be implemented in terms of each others, mostly. Sometimes.

I know there was some level of argument about whether it's best to overwrite older content (ring-buffer-style, probably keeps the most important stuff) or drop messages at input time (faster, probably fewer messages dropped overall). But logging isn't my area of expertise and I forget the details.

But again, the general point being that this is a complicated problem with tradeoffs, where most developers up the stack tend to think of it as a fixed facility that shouldn't ever fail or require developer bandwidth. And it's not, it's hard.

On a BeagleBone Black, it typically gives jitter on the order of hundreds of nanoseconds. I would consider it "hard" real-time (as do they). I'm able to schedule tasks periodically on the scale of tens of microseconds, and they never get missed.

It differs from this in that Real-Time Linux attempts to make Linux itself preemptive, whereas Xenomai is essentially its own kernel, running Linux as a task on top. It provides an ABI which allows you to run your own tasks alongside or at higher prio than Linux. This sidesteps the `printk()` issue, for instance, since Xenomai doesn't care. It will gladly context switch out of printk in order to run your tasks.

The downside is that you can't make normal syscalls while inside of the Xenomai context. Well... you can, but obviously this invalidates the realtime model. For example, calling `printf()` or `malloc()` inside of a xenomai task is not preemptable. The Xenomai ABI does its best to replicate everything you may need as far as syscalls, which works great as long as you're happy doing your own heap allocations.

[1]: https://xenomai.org/

I hope it will be ready soon. I'm working in a project that has strict serial communication requirements and it has caused us a lot of headaches.

Is it other interrupts on the CPU that break your process out of the "real time" requirement, I find this all so interesting.

Boot time is not a bottleneck for my application (however long it takes, the client will take longer…), and I’m sure there’s some more optimal libc to use, but I’m not sure the juice is worth the squeeze.

I’m also interested in what others are doing.

But who using a RTOS now would take the systems-integration cost/risk of switching? Would this put Android closer to Metal performance?