We missed our estimated shipping date by a couple of weeks, but Precursor is finally shipping to backers - more than half of the crowdfunding orders have already shipped and all outstanding orders should be shipped very soon. The global electronics supply chain is still a mess, but we remain undaunted. On the plus side, logistical delays freed up enough time to complete the first pass of a Braille version of Precursor!

The Bad

About a month ago, I sent a note indicating that "the shippening is nigh." This was true, but for larger values of nigh than I was hoping.

The Precursor units are finished with production, and the very last box arrived at the Mouser warehouse in Texas on Feb 28th, as the tracking information above shows (as a side note, Crowd Supply was acquired by Mouser a few years ago, so their fulfillment is handled by Mouser’s warehouse). Normally, at this point, the usually very efficient Mouser warehouse would kick in and dispatch these packages to your address, and the original campaign promised ship date would have been fulfilled.

These days, nothing goes smoothly. As of the time of writing, most but not all of the crowdfunding orders had tracking numbers issued; hopefully, by the end of this week, the rest of the post-campaign pre-orders will have been shipped as well. A local Covid wave, labor shortages, and unseasonably high demand for electronic parts have conspired to make the last logistical hop from warehouse to backers just as treacherous as the previous hops. If it’s frustrating for you to not receive your units on time, it’s twice as frustrating for us to have battled through an entire pandemic and epic supply chain shortage to have us miss our promise to you because of a hiccup at the last mile. It’s like pushing yourself throughout an entire marathon to be in first place, and then twisting your ankle a block before the finish line. If you have any questions or need to update your shipping address again, please contact Crowd Supply.

The Ugly

While it’s annoying that the backer units and current pre-orders are a couple weeks late, unfortunately, things are not on course to get better. Once all existing orders are filled, Crowd Supply will have about 70 Precursor units remaining in the stock they bought as part of the campaign, plus we have about 100 units of factory stock that will free up once we’re confident the initial wave of returns and exchanges are serviced. Once those are depleted, anyone who wants a Precursor will either have to turn to the secondhand market or wait until the next production run. I’ve been tracking the supply chain situation and my original sanguine predictions about a recovery in early 2022 were very wrong. At the moment, many key parts and sub-assemblies are still showing lead times of over one year. This means the realistic date for our next production lot is sometime in early 2023, but we’re trying our best to pull that in to late 2022 — there are just two components delaying the works, both of them highly efficient TI voltage regulators that would be used in your typical mobile phone.

This is definitely going to hurt our original plan of building a fecund developer community in 2022. If we can’t get hardware to interested developers, how can we then build applications that bootstrap the rest of the community? On the upside, because Precursor is such a bespoke, unique product, I’m optimistic that even if we’re delayed a few months, we occupy a unique enough niche that the people who desire a trustable, verifiable security device today will still need it even a few months later (and perhaps with the current geopolitics, the market for such a device may even grow). If some big industry player comes along in the meantime and meets or exceeds our level of openness and transparency, that would wipe us out. However, it would be a win overall for the world: I’ll gladly take the hit if it means a victory for the open source community as a whole.

The good news is that the backbone of Precursor is largely just me and Xobs, and both of us can adjust to this new timeline while continuing to serve the backers and users who are able to receive hardware. In a way, this could be a mixed blessing: modern products are mostly software effort, and Precursor is no exception. However, we’re different from your typical startup in that instead of hiring more developers to release more code faster, we simply take more time with a smaller team to get things done. Taking a page from the slow-food movement, you could say we’re taking a "slow code" approach: smaller teams, taking more time to make fewer features, but at a higher quality and release standard. We are agile, but deliberate. So, with our slower approach, a longer ramp-up time enables us to build a higher quality product, albeit with less revenue and thus some finagling to make ends meet. Fortunately, we don’t have any VC’s to haggle with, no promises of "up and right" growth curves, and we both live in Singapore, a country with a functional social safety net, public health care, and an abundance of thrifty options for everyday essentials such as food and transportation.

The outlook for the rest of the world as a whole isn’t as rosy, unfortunately. I was recently introduced to some work by Donella Meadows originally published back in the 1970’s (pre-dating desktop computers and China’s one-child policy) modeling the overshoot and collapse of systems. If you have the time, the video is worth a watch, and there’s also some online course material as well.

Her work showed that the "carrying capacity" of a system — that is, the capacity of a system to achieve some metric of productivity or population — can go through a decline, after which it can never return to its original level. Below is one example graph from the on-line course material, showing a hypothetical situation of a worker who starts out doing 16 hours a day, but asymptotically reduces their carrying capacity for work to 0 hours per day due to an irreversible decline in health.

This is admittedly a contrived example with extreme parameters, but there are numerous real-world systems that have been measured and show a similar effect. The key take-away is that during a systemic shock, where an underlying resource is depleted, the surviving population will expend more effort to exploit the remaining resources at ever greater costs.

For example, a plot of land may start with the vegetation to support up to 10,000 deer, but after a population explosion, the carrying capacity of the same plot of land does not recover to 10,000 — it reduces to an order of 100 or so deer. This is because the remaining deer are feeding on the slower-growing roots and rhizomes of the grasses, instead of the fast-growing leaves or blades, thus preventing a recovery of the productivity of the land and leading to a new stable state, with a much lower population of deer.

In other words, the surviving population cannibalizes the means of resource production, lowering the overall availability of resources and thus preventing a full recovery.

In trying to explore the roots of the extended supply chain crisis, I’m finding that the global supply chain is looking a bit closer to this over-grazed plateau. Although facially we can look at metrics such as raw capacity of wafers per month per fab and things might look on track to recover, there are some fragile roots of the supply chain that are being aggressively exploited.

For example, there is a shortage of wirebonding equipment. This is a highly specialized and very necessary step to turn a large variety of chips into packaged devices ready for circuit board assembly. There are only a couple of vendors of wirebonding equipment in the world, and the production capacity of this equipment might just be a couple dozen units per month. In normal times, this would be fine: the production capacity is sufficient to match the rate of decommissioning plus organic foundry growth of a few percent a year. However, it is far too small to be able to keep up with the recent double-digit surge in catch-up demand, especially with reduced productivity of the existing facilities due to post-Covid staffing and geopolitical factors.

So, like the starving deer digging into the earth to eat the roots of the grass, large companies whose production lines of cars and phones were held up by a lack of chips, dug around to find the root causes, and they identified one of the many bottlenecks to be a lack of wirebonding capacity. These companies, flush with cash, did exactly what they were supposed to do to protect their shareholders — they took their excess cash and bought up all the new production of wirebonders, and consigned them to factories with the condition that this new equipment may only be used exclusively for their production runs.

The wirebonding companies are happy because in a low-margin business, it’s difficult to pay for new capital equipment; and the large companies are happy because relative to a downed production line, a couple dozen wirebonders is chump change.

However, the carrying capacity of the ecosystem is smaller than before — because the wirebonders purchased for a specific company’s production can’t be used for the ecosystem as a whole. Large companies now have guaranteed capacity, at the expense of lower overall global efficiency. This means one can no longer simply multiply the number of bonders installed in factories by the throughput of chips per bonder to derive the global carrying capacity for wirebonding. Before the pandemic, if customer A’s production was done, you’d stick customer B’s product on the bonder and you’d have near-100% utilization of the equipment, because the wirebonding company owned its wirebonders and were free to deploy them efficiently. Now, with customer A owning the wire bonding equipment outright, once customer A’s production is done, that equipment simply sits idle: the post-pandemic carrying capacity of the global wirebonding market is fundamentally less than before.

On top of that, general capacity expansion is delayed by many months, since all of the new wirebonding capacity is being dedicated exclusively to the large companies that can pay for it, thus potentially explaining, at least in part, why we’re seeing some components with 100+ week lead times. The big, strong companies are eating at the trough first, leaving the smaller companies to starve.

This is just one small, specific example of what’s going on, but the same story is being repeated all over the supply chain — from the polymer films used to construct chip packages, to the carriers used to transport wafers, to the equipment used to dice wafers and test finished chips. The chip industry is full of highly efficient, extremely niche but absolutely critical service providers that are short on cash relative to the value they bring. Meanwhile, there is a glut of cash in large companies and private equity firms, and shifts in geopolitical trade policy are creating the conditions favorable for vertical consolidation and monopolization of the supply chain.

If it is the case that the post-pandemic production capacity is more geographically distributed but more vertically integrated and less efficient, this would ultimately lead to the die-off of many of the smaller and less healthy hardware companies, leading to less competition in the market, and in the end, higher prices to consumers, less choice, and overall fewer electronic products being produced, even though more dollars than ever are being invested into expanding and installing new production capacity.

This isn’t a very rosy-sounding message for small hardware makers, but, for every cloud there is a silver lining.

The Good

Near the end of Precursor’s crowdfunding campaign, I ran a piece showing a variant of Precursor sporting a Braille keyboard to accommodate blind users.

Above: Photo of the device as shown during the original campaign post.

That project got put on the back burner while we were struggling to get Xous up to release readiness, while simultaneously chasing down hardware production and logistics issues.

While unplanned delays are frustrating, they also create an opportunity for me to allocate my time to other related projects that I care about. So, in the past couple of weeks, I managed to pick up the Braille edition again and finish an initial draft of a text-to-speech implementation, as well as tape out a revision of the PCB that uses some brand-new, low-profile mechanical scissor switches that came on the market sometime in the last year. Fortunately, I got the keyboard switches before Shenzhen went on lockdown, and my prototype PCBs got stuck in a customs jam in Hong Kong for a week, but luckily they made it to Singapore just today.

The actual text-to-speech engine (espeak-ng) is written in C, so it required writing a set of FFI bindings for the server, as well as a re-implementation of many of the C standard library functions (such as malloc, free, sscanf, etc.) since they don’t exist on Xous. A number of configurable options such as the voice model and language are also statically selected and compiled into the binary to reduce its overall footprint. The whole engine is then isolated into a separate build system and distributed as an ELF executable that is copied into the final disk image, in order to keep the codebases of Xous and espeak-ng clean and separated.

Getting the text-to-speech engine to work was a real exercise of Xous’ concurrency and performance, since it’s all about real-time audio rendering without glitches, while also being able to cleanly abort a rendering-in-progress in the case a user selects a new piece of text to translate mid-translation. Fortunately, it all came together and I’m fairly pleased with the first draft of it. I’m glad that I will soon be able to fulfill my overdue promise to Matt Campbell, the original blind developer who reached out to me and inspired this work, and send him a prototype with a Braille keyboard and a basic text-to-speech engine so he can offer further guidance and help on how to improve the user experience.

Here’s a quick demo video of how it looks (but more importantly sounds) on the "regular" Precursor that I’m using for development. It’s helpful for me to debug it with the screen in place, but the final Braille prototype will lose the screen and just have a Braille keyboard that takes up the entire bezel:

I learned a lot putting this demo together, and I also got to file off some rough edges in Xous while I was at it, particularly in the audio codec. So overall, this was a good use of some quiet time while the last-mile supply chain sorts itself out.

Happy Hacking!

-bunnie