In stock
View Purchasing OptionsProject update 9 of 9
Hello supporters and backers.
After too long of a silence following the first production batch fulfillment, we are back to public firmware/software issue-solving, feature additions, and the next production batch.
Here we will update you on the production and fulfillment issues and statistics. In addition, we will also talk about the software, drivers, portability, and firmware issues now known about thanks to the community and GitHub contributors. Lastly, we will elaborate on the added features - Tx side, portability - and the bug fixes pushed over the past month.
During the initial production phase, we divided the process into two steps: supplying the boards to the backers first, and supplying the remaining boards to customer service (CS). Each board underwent a testing phase, using a dedicated application and tester hardware. The overall yield of that production was approximately 95%, with around five out of every 100 boards identified as failing. In the case of the "full" board version, the yield was slightly lower at around 93% (more components and complexity). Both we and the contract manufacturer (CM) have gained experience, and we are enhancing the manufacturing process and the post-production handling process to achieve higher yields.
Unfortunately, during the fulfillment phase, some backers who purchased the "full" version received the wrong boards. While we do not have a full understanding of the logistical factors that contributed to the issue, we would like to acknowledge the response and resolution from the Crowd Supply and Mouser teams. As a result, the fulfillment process was temporarily paused to ensure proper cataloging of the boards at Mouser’s warehouse and consequently, some existing orders had delayed shipping.
Fortunately a few lessons have been learned:
The current production run is expected to finish in 6-8 weeks (not including fulfillment). We are implementing the lessons learned from the first round. Components for this batch of production are secured and the PCB manufacturing has begun.
During the last few months, we have been working on an architectural redesign of the firmware and software of CaribouLite. This process slowly started to emerge just before the first boards shipped to the backers. We removed the dependence on the older native kernel module and started to write our own module with async I/O API (poll) features that works closely with a more efficient DMA engine. Then, we started to code the arbitrary waveform Tx part (firmware, driver, and software). More on that next.
Software and firmware were redesigned to provide a more consistent interface with the radio on the board. The fact that CaribouLite is a mix of many independent components (FPGA, modem, mixer, BCM internals) requires each of these components to be tested and developed independently. Nevertheless, receiving signals on a certain frequency requires methodically communicating with a mixer, FPGA, modem, SMI, and clock source altogether.
These distributed tasks have been integrated into two main parts of the software:
SoapySDR is still there to provide even higher levels of control and I/O, but it should reflect exactly the "c" interface as described above. This software and firmware redesign have been documented in the code and span the software, dependencies, submodules, and portability issues that frustrated our GitHub contributors.
In the first version, we started with the basic functionality of a dual channel RX and the transmission of CW signal on the TX Side. This part was essential for hardware development and testing to be possible. During the last few months, we have been developing the transmit (Tx) part that allows streaming samples at 4 MSPS to the modem through the radio interface. This part is almost over, but is still having a few synchronization problems. The transmission is to either of the two channels. It may be possible to transmit concurrently the same waveform to both channels (not tested yet).
Known issues:
sudo IssueContributors had an issue with the need to "sudo" every functionality on CaribouLite. Even third-party software that used CaribouLite (such as SDR++, SoapyUtils, SoapyRemote, Python, etc…) needed to be run as root to have access to CaribouLite.
There were two reasons for this constraint:
/dev/mem device to access SOC's flattened physical device memory and control their registers. Regardless of that, accessing this device requires root, and this requirement cannot overridden (for a good reason) by groups and udev rules. To remove this issue, the low-level io_utils API was redesigned with spidev and gpiomem APIs. Both APIs allow a udev based user-level full access to the relevant hardware resources. In addition, they are both well defined and documented along different mainline kernel distributions.So, the sudo‘ing issue is soon to be removed.
CaribouLite was developed to support RPI Zero and was tested on it at its first stages (with the legacy software). Along with the development of the new kernel driver and other features (including FPGA synthesis and placing with IceStorm over RPI), we continued developing over the more powerful RPI4 board. As a result, some parametric features of the software were neglected (such as memory limitations). Once those parameters are tweaked - mainly in the kernel module - the support for RPI Zero should be just fine. This task is in our backlog, and we would love to partner with a contributor who can take it on as a project.
We fully understand that both cross-distro support (DragonOS <=> Raspbian) and the RPI-Zero compatibility issues are critical to resolve for the community.
CaribouLite is being developed on Raspbian 32/64-bit distributions. Following the issues reported by the users, we have created a DragonOS bench on which software is being deployed and tested. From our analysis, the majority of the compatibility issues span on some possible reasons, including:
Directory structure - or "Where SoapySDR’s .so files can be found?" and CaribouLite’s installed locations. We tried to solve this issue through the install.sh script, and contributors have created a few pull requests to correct this script according to their boards. As we would like to have a consolidated version that fits different distributions of RPI, we encourage users to come forward with their suggestions and recommendations.
More on that in the following sub-issue.
In a nutshell, samples need to be streamed in and out of the FPGA by the Broadcom microprocessor. This steaming physical interface consists of 8-bit I/O, R/W control signals, and data-ready clocks (one for Rx and the other for Tx). This interface is called SMI and it is like a "parallel" SPI.
On the higher (software) level, the streaming is based on DMA (direct memory access) transactions - whenever the user requests to "read" the "SDR file", this request is translated into buffer preparations, DMA request, and execution. To perform this streaming functionality a kernel module was written that mediates the "file-read / write" requests by the userspace application into hardware buffer transactions through the DMA.
The current kernel module loading technique is based on the "hot-replacement" of the smi-dev which is loaded by default in Linux with an alternative .ko object blob. This module is compiled against a given kernel version and then "attached" to it as an extension. But whenever the target system updates the OS (periodic kernel updates), the kernel module is not anymore valid and it needs to be re-compiled against the "new kernel version". One solution for this issue is to integrate the kernel module code into the mainline Linux kernel repository. Then, every time the system updates its kernel version, the pre-compiled binary of the SMI module is also updated automatically.
We need to push a fully debugged version of the SMI driver into the kernel code. We also want to have the kernel module to be generalized to support not only CaribouLite but many other streaming hardware applications (think - ultrasound probes, thermal cameras, audio devices, A/D, or DAC). That’s why, currently, we need to keep in mind that RPI automatically updates its OS every once in a while. The same for DragonOS, and after such an update, the. ko file needs to be recompiled and re-linked.
We are currently working on removing this "hard-linking" of the .ko binary blob from the userspace software and integrating it into the OS in a more native way. We are reaching out to users who have experience in this process, inviting them to contribute their ideas and insights (i.e. using systemd /etc/modules-load.d).
Kernel module loss of samples issue: On both Rx and Tx, we receive and transmit over constant sizes buffers (MTUs - 128 k samples long). Within the transaction of the buffer by the DMA, it is rare to witness samples lost as the transaction does not involve any software (it does involve DMA and HW priorities though). The problem arises once we finish a transaction and want to set up another one after it. This setup time is time-consuming and may cause packet loss when OS is occupied by many tasks and context switches. To solve this problem, we need to integrate the "cyclic DMA" transactions feature into the current module. This should allow also using a shorter MTU (which is currently large to compensate for system loads). To have this feature integrated, we are calling out to the community to contribute your advice and experience on this matter
This project is still learning its baby steps, as a new Open-Source property of the SDR community. We accepted questions from a few worried contributors, on whether the project is "abandoned". The straight answer is - no. We are working on solving problems and improving the software side, and we are grateful for the help we are getting from the community. We encourage the community to check, correct and rewrite parts of the code as needed (C / C++ / Verilog). We are fully accepting our limitations on the software side and would very much appreciate your features, addition, and corrections, in a coordinated manner.
CaribouLite RPi HAT is part of Qorvo RF Accelerator