Dear Backers,

What a busy summer it has been! We’re overdue for a comprehensive update. So, without further ado, here’s the latest news on Precursor.

OS Update (Xous 0.9)

The most exciting news is that Xous is now in a “beta” release, at version 0.9. As Xobs explained in his previous backer update, this means your Precursor device will now come with an OS that has a libstd target that works with the latest stable Rust.

Prior to this development, we were a no-std target, which means we had no heap-allocated structures: no Vec, no HashMap, and threads and closures had significant limitations. In short, we were unable to use anything that required allocations with stable Rust. Now, we’re able to unlock some of the most powerful Rust features. This means developers like you will also be able to pull from a much larger range of crates, with caveats; because Xous is a work in progress, some std features will be unimplemented and, of course, there is also the tiny memory footprint of the target device to keep in mind.

In addition to adding std to Xous, we have also completed all the hardware drivers, including but not limited to our final suite of hardware-accelerated cryptographic primitives (AES, Curve25519, SHA512), TRNG enhancements, and other goodies like the audio Codec, real time clock, and power management.

On top of that, we’ve started to layer in a more fully-functional gam, or Graphical Abstraction Manager. The gam can now conjure dialog boxes with text input, progress bars, radio buttons, and check boxes, as well as notifications and menus. Our core UX framework is rolled from scratch because we incorporate the concept of “trusted canvases”. The goal of the trusted canvas framework is to make it more difficult for a malicious program to exploit human weaknesses by presenting fake-but-convincing looking prompts for password input. With the gam, the “look and feel” of content is controlled by the OS, which allows the OS to enforce “style guides” that visually differentiate trusted and untrusted content.

Left: Main menu. Center: Password dialog. L & R D-pad keys toggle password visibility. Right: Gateware update underway. The random lines in the background show how the `gam` defaces less-trusted content when more-trusted content is in the foreground.

Finally, Xous is starting to grow a network stack, thanks to Samblenny’s hard efforts (for those who don’t recall, Sam is the developer who contributed Xous’ Unicode-ready font rendering engine). Sam has been going through the EC implementation with a fine-toothed comb and refactoring the WF200 interface, improving the stability and extensibility of the COM bus, all while doing a very detailed job of mapping out power consumption and thinking through code attack surfaces. In addition to being able to scan for available SSIDs, Xous devices can now join a network. I for one am extremely pleased with the excellent progress being made on the network stack, one of the toughest aspects of Precursor.

In order to better support developers who are excited to get started with Xous, Xobs has been hard at work extending both our Renode and our “Hosted Mode” emulation resources:

• Renode is an emulator which can execute RISC-V instructions to boot a virtual Precursor, complete with virtual screen. It strives to emulate hardware peripherals in a register-accurate fashion, so you can use it to probe kernel and driver bugs.
• Hosted Mode is a simulator, which runs native CPU instructions (e.g., x86_64) and uses your OS’s native thread libraries in order to boot a virtual Precursor. It makes no attempt to simulate the hardware at all; most hardware functions are just replaced with stubs that return fake values. It’s most useful for debugging UX code.

If you’re excited to join our efforts even before you receive your hardware, check out Renode or Hosted Mode!

Secured Boot Flow

We’ve also added our first cut at a root of trust plus secured boot flow to Precursor. With this flow, you can now generate root keys (one Ed25519 and one AES-256) entirely within the device and secure them with a password of your choice. These keys are injected directly into the gateware image. In other words, Precursor is an FPGA that can re-write its own bitstream for the purpose of burying secret keys within a particular region of the SoC’s internal logic. We do this so these root keys can be further encrypted by the FPGA’s native AES-256 bitstream encryption, whose key can either be burned into a set of eFuses or battery backed RAM (BBRAM).

We currently support a BBRAM key programming flow, where a Precursor device can generate its own AES-256 FPGA key. The key is then uploaded into the FPGA with the help of a script running on a helper Raspberry Pi device. A helper device is necessary because the FPGA must be held in reset while the BBRAM key is programmed; unlike an eFuse key, the FPGA cannot self-provision a BBRAM key. However, the helper script (written entirely in pure Python so you can easily and thoroughly audit it) runs entirely out of RAM on the helper Raspberry Pi device. By unplugging the helper device from the network prior to burning the BBRAM key, and then immediately turning the device off after the procedure, you have a good chance at protecting the secrecy of your BBRAM key. If you’re very paranoid, you can also destroy the helper Raspberry Pi device and/or its boot microSD card to ensure no copies of the BBRAM key were “accidentally” made.

The BBRAM key is best suited for users who have ephemeral secrets they would rather be lost forever than be discovered in case an attack is mounted on their device. It’s not suited for securing long-term secrets (such as the storage of cryptocurrency). Keeping keys in the volatile BBRAM ups the ante for attackers attempting to mount a WBSTAR exploit on the device, especially once the JTAG access ports have been glued shut; an attacker would have to mill away the glue seal without removing power to the device (or accidentally shorting power to ground). This is not an insurmountable barrier, but at a minimum requires an attacker with specialized skills and equipment. It’s also very well suited for developers (like us) who want to develop and test encrypted boot flows but can’t afford to burn eFuse keys and later throw devices away when those keys are lost.

In addition to establishing a trust root in the FPGA hardware itself using BBRAM keys, we extend the trust root up to the kernel using Ed25519 signature checks on the loader and the kernel, with a final self-circular Ed25519 signature check on the gateware itself. Thanks to our Curve25519 and SHA512 hardware accelerators, we were able to fit the entire signature check + UX in under 32kiB of code, allowing a secure boot routine to reside entirely within the SoC gateware as a ROM. The purpose of these signature checks is to protect code-at-rest (that is, detect if someone has attempted to tamper with the code in your ROM). We deliberately steer around the problem of distributing signed updates by providing an empty “third party” public key slot, but no real solution on how to populate it. This enables user groups who want to distribute signed binaries for updates to do so of their own accord.

I feel key distribution is an incredibly hard problem that’s outside the scope of Precursor. After taking all this effort to audit my own hardware, it would be a downgrade of trustworthiness to rely on, for example, an X.509 Certificate Authority I can’t directly audit to a similar standard. I also know we’re not up to the task of operating a root CA of our own, so it would be a disservice to pretend to act as one for the Precursor ecosystem. Therefore, the “third party” key is kept blank to force the user community to at least have a conversation about the problem, instead of shrink-wrapping the issue in a rote implementation of X.509.

More details on the secure boot flow can be found at our wiki.

EMC Testing and Supply Chain Update

I finally have some good news to share on EMC testing: we’ve passed and with a decent amount of margin at that. As suspected, the biggest offender for EMC is our class-D amplifier, which is almost marginal on our conducted emissions tests because of the long, unshielded wire that connects the speaker to the mainboard. However, the extensive RF shielding built into Precursor, intended to keep the bad guys out, also serves to keep the noise in. Fewer emissions mean fewer side channels (note that I did not say no side channels!)

We’re not out of purgatory, however – we’ve just ascended from one level of Hell to another. Apparently, despite having passing results, the testing units are being withheld to go through an “auditor”. I’ve never had such an “audit” done before on an EMC test – usually by this point I’d have a certificate in hand – but I suspect I’m receiving extra scrutiny because my device looks like a mobile phone. Even though I don’t operate on licensed spectrum, most phones do. I could imagine testing houses get called out on marginal test results more often for mobile phones because licensed spectrum holders are more likely to verify the testing shop’s results. That, and this device is "weird" compared to typical mobile phones, thus attracting the attention of auditors.

There’s also an extra-special level of Hell coming, which is now that we have an EMC certificate, we are locked into a very specific arrangement of BOM and board layout in order for the EMC certificates to be valid. This means I have almost no latitude to, for example, re-design Precursor to use a substitute chip that is more available. While there is a process and standard for allowed changes to a design, it’s not nearly broad enough to deal with the scope of supply chain issues we are facing today. I wouldn’t be surprised if stories came out a few years from now about how hawkish corporations managed to snipe start-ups out of the market by reading the pre-release EMC certification reports of up-and-coming competitors, and then buying out the global stock of the cheapest component on their BOM with no allowable substitute, thus effectively shutting down or at least significantly delaying their production.

Consequently, it’s quite possible the current run of Precursors could be the only run of Precursors for the next year or so. The chance I can re-purchase the entire BOM of parts and have every component arrive anytime in 2022 is pretty slim – I’ve spot-checked, and several key components are currently listing 52+ week lead times. This puts me in a really hard spot, because now I’m having to forecast demand for a product that hasn’t even shipped its first unit; plus I have no way to pay for the inventory: it’s not practical to crowdfund a second run of Precursors, when the first run hasn’t even started rolling off the production line!

I’m envious of folks who sell open hardware as kits that don’t have to secure FCC and CE certifications – at least they can redesign their boards to use a more available crystal or power switch, without having to worry about then spending thousands of dollars and months on re-testing and certification to work around component availability issues.

So long as we are facing refractory supply chain conditions, I’m considering a pivot once we’ve shipped the current round of fully-assembled devices to backers. At least temporarily, we might sell kit-only solutions instead of completed units, so we can make reasonable component substitutions without having to re-certify for every small change. You’ll have to take my word that any substitutions we make would not, in practice, degrade its compliance level, as we would lack the thousands of dollars to pay off the certification cartels so we can produce updated papers to please customs officers. But, by selling kits of parts, we’ll have more flexibility to work around component shortages through careful substitutions and then at least we’d be able to make DIY parts available to territories which have DIY certification exemptions.

Production Ramp

Perhaps the most significant outcome of Xous 0.9 is we’re now at a state where we could burn it onto the production devices we plan to ship later this year. We’re still missing a couple of key pieces – such as the Plausibly Deniable Database (PDDB) – our answer to “what is Xous’ filesystem” - and a fleshed-out networking stack. However, both of these items can be rolled out in parallel with the production effort, and can be applied as firmware updates.

Thus, I’ll be turning more attention toward production efforts. I just received a kit status audit and we have received almost all the components in the warehouse, except for the FPGA. We still have a few hair-raising close calls to deal with, as some components were shipped with shortages and we had some last-minute cancellations that forced us to do spot-buys at multiples over our originally quoted price. Assuming the FPGAs come as scheduled – and that is still a big if – we’re probably going to be lucky if we break even once all the cost overruns resulting from the COVID-era supply chain troubles are factored in.

In addition to marshalling components while walking the tight rope between EMC compliance and allowable components substitutions, we still have a few significant details to close out:

• The PVD steel process is still not yet finalized. We’ve had a hell of a time getting the finish just perfect and this has been hard to do without a visit on-site to the shops doing the CNC and plating. I’m optimistic that by the next backer update, we’ll have good news on that front: I just saw a video of a promising-looking sample that passes the basic qualification tests arrive in my inbox a couple days ago, so now we’re doing the DHL-and-customs tango to get a final review sample here in Singapore.
• We still need to design the “gift box” – you know, that wasteful retail box consumers feel entitled to receive their products in. Spoiler alert: it’s not going to be super fancy. We don’t have the budget or time to create a delightful, social-medial-ready unboxing experience – but it’s still a thing that has to be designed, as we have to protect a phone-like object through shipping and handling.
• Some regulatory material such as our user manuals (which is basically just a declaration of conformity to EMC standards) and screen-printing with our certification information is still being reviewed and finalized (see the previous comment about post-certification audit).
• The battery is still getting its certification for air shipment. This should be more-or-less rubber-stamp, but we had to run production first, as they need dozens of production samples, since the certification process is destructive (they literally punch holes in the batteries to induce failures).
• Our keyboard vendor is running late on delivering material to us. Hopefully we can keep them on track, as we still have some significant process development to do in terms of the assembly flow for the keyboards to make sure they have a consistent feel and a reliable action.
• The final test infrastructure needs to absorb all the latest Xous changes, so we can automate more of the production test.

These “details” will be more than enough to keep us busy until November, at which point we’re hoping to see our SMT line fire up for the first time in South Korea. Of course, a huge wild card in all of this is how the COVID-19 delta variant is going to play out. I have never launched a product without being on-site for the initial production run; and for a product as complicated as Precursor, I think it would be an ill idea to not be physically present to review and supervise the quality of the work being done. It may just work out that I’ll have to suffer a pair of two-week quarantines getting to and from South Korea to ensure production stability and quality. But, if there’s anything the COVID era has taught me, is there is no such thing as “a new normal” – just varying degrees of reacting to an unpredictable pandemic and even more unpredictable policymakers and populaces.

Happy Hacking!

-b.