This is the bit I mentioned a moment ago. A big part of what makes our racks unique is
the embedded RoT and SP
in every sled. We’ve long had an SpSim construct that served as a stub for the service
processor and would return the right info at the right times. But that’s simulation, not
emulation. It’s always going to be happy path, and that’s just not enough fidelity for
real testing cases.
As such, we built actual emulated Cortex-M7 and Cortex-M33 devices for illumos and Linux (x86 and ARM). The repo for this tool (sp-emu) is open source. Each of the two devices runs in a single thread, and (mostly) faithfully emulates the three main requirements: instructions, time, and interrupts.
From there, you can flash our Hubris firmware directly to these emulated devices, and then communicate by way of an ordinary socket. The Voxel lab can speak with these virtual devices via the Management Gateway Service (MGS) and Inter-Processor Communication Channel (IPCC). You can learn everything you want and more about these concepts in the Management Gateway Service repo and Hubris docs.
There really is no "run real microcontrollers as virtual devices with this one neat trick!"
I can think of. But the real challenge of sp-emu’s job is time. Remember all of the code
we’re running there is meant for silicon. Hubris schedules off a 1ms SysTick and the emulator doesn’t
use wall time; as such, we had to invent a clock. That said, emulated hardware is also
slower than the real thing. So there’s… well, less time per time. So instead of trying
to get all temporal about it, time became instructions. For example, I give you maybe_tick:
if self.systick == 0 {
self.systick = self.syst_rvr;
}
self.systick -= 1;
if self.systick == 0 {
self.systick = self.syst_rvr;
self.tick_events = self.tick_events.wrapping_add(1);
if self.syst_csr & 2 != 0
&& self.mode == Mode::Thread
&& !self.primask
{
self.exception_entry(15, bus); // SysTick exception
}
}In this case, we count instructions against the kernel’s own reload value and fire exceptions anywhere the single-priority design makes it safe to take it — in thread mode, interrupts unmasked. By the same token, peripheral access costs one tick of our clock (an instruction), whereas on silicon it might stall for dozens of cycles on the bus.
You may have noticed self.exception_entry which is defined
elsewhere in the code:
fn exception_entry(&mut self, vecnum: u32, bus: &mut Bus)
In short, this function models everything the hardware does on a Cortex-M core taking
an exception, all the way between "exception becomes active" to the first handler
instruction. vecnum here is an ARMv7-M exception number (also the vector table index
for that exception’s handler). bus is the memory system which is used to actually
push the frame and read the table.
Hubris has a non-preemptible kernel, though it schedules tasks preemptively. SVCall, PendSV, SysTick and all IRQs run at the same priority level, while faults keep a higher priority. As such, the emulator doesn’t have to nest handlers. It processes pending IRQs based on the PRIMASK, FAULTMASK, and BASEPRI settings. If one of them pended PendSV, that will perform the deferred context switch for the kernel. For more details on how it works internally, you can check this lovely bit of documentation.
if let Some(irq) = bus.next_irq() {
if self.basepri == 0 || (bus.irq_prio(irq) as u32) < self.basepri {
bus.clear_pending(irq);
self.exception_entry(16 + irq as u32, bus);
return;
}
}
if bus.take_pendsv() {
self.exception_entry(14, bus);
}There’s a lot more we can talk about here, but to sum it all up you can look at this pseudo-ish loop:
loop {
cpu.step(&mut bus, host.as_mut())?; // one Thumb-2 instruction
cpu.maybe_tick(&mut bus); // SysTick: time is instructions
cpu.maybe_interrupt(&mut bus); // NVIC and PendSV delivery
if cpu.cycles % 4096 == 0 {
bus.pump_eth(host.as_mut()); // MGS packets in and out
bus.pump_uart(host.as_mut());
}
}Note: yes I’m aware I said there’s no cycles then have a field named cycles. That’s the "fake it till you make it" part of emulation.
That’s the overall heartbeat of sp-emu, and it runs ad infinitum. Everything else exists to make those calls seem real.