Why I sold my Raspberry Pi 4 for a Rock Pi 4

I sold my Raspberry Pi 4B 4GB recently and replaced it with a Rock Pi 4A. I had owned it for about 6 months before finally letting it go at a significant loss and honestly, that decision was not as hard as I thought. On this piece, I document my anticipations, disappointments, and epiphanies over the course of 6 months owning the Raspberry Pi 4B.

Disclaimer: I'm not affiliated with nor sponsored by Radxa/Allnet, the manufacturers of Rock Pi. I'm writing this piece purely to share my experiences with both the Raspberry Pi 4B 4GB and the Rock Pi 4A 4GB.

I. Need. More. Memory.

It was February 2020, I was running my self-hosted apps all on the Kraken cluster then. At that point in time I wanted to run a metric-monitoring stack (Prometheus + Grafana) for the cluster but after reviewing the memory requirements, I quickly realized that not even all the memory on a single Raspberry Pi 3B node was enough (although I'm cognizant that 1GB isn't much in the grander scheme of things).

Browsing around, the most obvious choice was the next model in the Raspberry Pi line that was just released a few weeks earlier then, the Raspberry Pi 4B 4GB.

The Shiny Raspberry Pi 4B

There were quite a few aspects about the Raspberry Pi 4B 4GB that got me excited: 4GB Memory, Gigabit Ethernet, yada yada. But what I anticipated the most was the long-overdue elimination of the painful USB 2.0 bus bottleneck that plagued all Raspberry Pi models from Zero to 3B+.

In the Raspberry Pi 4, the ethernet is built directly into the SoC and no longer shares a bottlenecked bus with USB devices. This means we can unleash the full potential of the gigabit ethernet port without worrying about performance degradation of the micro-SD card or external drives, and vice-versa.

On top of all those changes, the Raspberry Pi 4B doubled the micro-SD card slot bandwidth from 20MB/s to 40MB/s. While this is still not cutting-edge performance, it's still a significant and welcome improvement, especially for tasks that have high demands for sequential I/O.

Delay no more!

With those specs, I made a trip down to Amicus @ Sim Lim Tower to snag one from the shelves.

What Belies the Shine

At the point of purchase and over 6 months of usage, I've uncovered many issues that really made me second-guess my purchase.

These are the pain points I've identified, each of which I will cover in detail in dedicated sections:

1. Power concerns
2. Heat dissipation issues
3. No official cooling solutions
4. 32-bit Operating System
5. Operating Systems Available and Compatibility with k8s
6. Storage performance
7. Storage longevity

Power Concerns

Interestingly, a red flag had already surfaced prior to purchase, but I conveniently ignored it in favor of my excitement; the power supply. There were many complaints online that electronically-marked USB-C cables did not work due to a flaw with the Raspberry Pi 4 hardware design that caused it to detect the charging cable as an audio accessory.

For context, standards-compliant USB-C cables that support more than 3.0A (Thanks ferrybig from /r/selfhosted for correcting me with the USB C spec) current have a tiny microchip embedded within it that enables smart features such as voltage and USB protocol negotiation which provides information on supported voltages and data transfer rates to the other end. The microchip also doubles as protection for your device from over-voltage and other electrical risks. We have seen from the sacrifices of Benson Leung, full-time Google engineer and part-time USB-C vigilante, going around on Amazon testing USB-C cables and frying his Chromebook Pixel in the process, what kind of damage non-compliant cables can potentially do to your devices.

Google engineer fries Pixel testing USB Type-C cable | Engadget

You might not remember Benson Leung, the Google engineer that tasked himself with examining USB Type-C cables. He’s been diligently doing so for months, but he’s calling his tests to a halt after one went horribly wrong. Leung bought a USB 3.1 Type-C SuperSpeed cable (it’s since been removed) from S…

Engadget

I find it rather ridiculous that in order to power the Raspberry Pi 4B, one had to source for cables that did not have the embedded microchip and hence potentially unsafe.

Other than that issue, I also had concerns that my existing power supply, the Anker PowerPort 10, would not be sufficient for the Raspberry Pi 4, given that it only supports up to 5V 2.0A. The official page states that the Raspberry Pi 4B needs at least 5V 3.0A to work, which means I'd need to get something like a Qualcomm QuickCharge 3.0 brick specifically for it.

In the end, out of an abundance of caution I bought the extortionately priced official Raspberry Pi 4B power supply at S$18 (US$13.40). To put things into perspective, that power brick costed 20.93% of a Raspberry Pi 4B 4GB priced at S$84.99 (US$64.03)!

Heat Dissipation Issues

Heat was a real problem with the Raspberry Pi 4B. With an ambient temperature in Singapore at around 31°C, the Raspberry Pi  4B idles at around 52°C, and quickly hits a toasty 80°C on medium load, after which it thermal-throttles itself to oblivion.

$ sudo vcgencmd measure_temp
temp=81.2'C

I realized this when I tried running hardware-accelerated transcode of videos from HEVC to H.264 in Jellyfin. The first few seconds would render perfect in real-time, beyond that the CPU/GPU throttles and the video renders at 0.25x speed. At this speed, I'd have to wait 4 seconds just to watch 1 second.

To demonstrate what 0.25x means, here's an example: to render a full 40-minute episode, I'd have to wait for a whole 2h 40m for it to render. This pretty much renders the video unwatchable on the Raspberry Pi 4 (pun intended). Shockingly, this performance actually comparable to that of software transcode on the Raspberry Pi 3B without any cooling.

Besides video transcoding, I've tried applications that do not generate as much load, such as running PostgreSQL and MariaDB. I thought running databases on the Raspberry Pi 4 was the obvious choice since it has double the I/O bandwidth of a Raspberry Pi 3. However, even that pushed the Raspberry Pi 4 to its thermal throttling limits, and though I did not run precise database performance benchmarks, I did measure an average of 1.2s longer load times on my Nextcloud home page, over 10 refreshes with browser caching disabled, during which the CPU utilization on the Raspberry Pi 4 would peak.

If this trend holds true, it means I've spent more money to purchase a device with poorer performance than my existing Raspberry Pi 3Bs (at least without investment on cooling solutions) 🤦‍♂️.

No Official Cooling Solutions

I searched far and wide for a solution to the heat problem. I've came across solutions on both ends of the price spectrum.

The cheapest ones are those generic, colored heatsinks that cost around S$6 (US$4.61) for a set.

Unsurprisingly, the pricier cases are from reputable Raspberry Pi accessory manufacturers. The most pricey one was the ArgonOne from ArgonForty at S$32.50 (US$25), an intricately designed heatsink-case combo with a software-controlled PWM-fan for active-cooling.

Personally, I'm very sensitive to background noise and am easily distracted by it so I was looking for passive-cooling solutions. However, upon doing a cursory search, there were quite a number of users out there facing issues with passive cooling.

The Great Raspberry Pi Cooling Bake-Off: Comparing Passive Heatsinks and Active Cooling for the Raspberry Pi 4|downey.io

Why is my Raspberry Pi 4 running so hot? You may know you need something to cool it down, but what? In this post we compare the performance of various Raspberry Pi coolers. All the way from the humble heatsink to a massive cooling tower complete with RGB fans.

Tim Downey

Tim Downey has written a fantastic piece on different cooling solutions he tested for the Raspberry Pi 4. In particular, in one of his tests, he had the ArgonNEO which was essentially an ArgonOne without a fan. On running CPU-intensive tasks, his case reached temperatures above 80°C! The ArgonNEO is not a low-quality case by any standards, it's a pretty chunky aluminum case that can hold and dissipate quite a lot of heat! These temperatures on the surfaces of the case are not just potentially damaging for the furniture but also dangerous for kids and pets. It seems like cooling has become a matter of safety as well, and not just performance.

I went for an active cooling solution in the end after having doubts on the adequacy of passive cooling and bought a generic cheap dual-fan heatsink off Shopee at S$6.09 (US$4.68).

At that point, I was rather disappointed that the Raspberry Pi 4 is practically unusable out-of-the-box for anything more than lightweight applications and simple shell scripts even though it has so much performance headroom. With that kind of performance impact, I had hoped that the Raspberry Pi foundation at least provided some semblance of an official add-on cooling solution instead of forcing the user to go out of his/her way to get it to work as it was designed to.

32-bit Operating System

The official/recommended operating system for the Raspberry Pi 4B is Raspbian (now known as Raspberry Pi OS). It's a 32-bit Debian-based operating system, and the problem lies in this number of bits.

The Raspberry Pi 3B and the Raspberry Pi 4B are both running 64-bit CPUs (arm64 architecture) but are constrained to running 32-bit applications due to its operating system. There has been an ongoing debate in the Raspbian community for many years, about the benefits of going 64-bit and the general yet somewhat infuriating consensus was that it's not worth the effort.

Why you should run a 64 bit OS on your Raspberry Pi4

A comparision of two Raspberry Pi4 running a 32 bit and a 64 bit OS

MediumMatteo Croce

Based on a user, Matteo Croce's findings, 64-bit operating systems confer a non-trivial performance improvement over its 32-bit counterpart. Not to mention, arm64 is becoming more widely-supported, due in no small part to the release of AWS Graviton ARM virtual machines and the advent of Apple's M1 Arm Macbooks. With the rise of arm64, its 32-bit counterpart is slowly but surely on the path to obsolescence.

There are not many viable alternatives with 64-bit support other than Ubuntu Server which is rather heavy and bloated, so much that SSH logins are obscenely slow and takes up to a good 3 seconds to get to the bash prompt.

Operating Systems Available and Compatibility with k8s

I've been running HypriotOS for a long time now on my Kraken cluster for it's Docker-native environment but at that point, it's no longer an advantage as I run Kubernetes (k3s) on the cluster which uses CRI-O as its container runtime instead of Docker. Regardless of HypriotOS or Raspbian, I've always faced issues with CPU limiting on Kubernetes, where CPU limits on Kubernetes pods are not enforced and as a result, apps may use more CPU than they're assigned.

I've done some research and found that this is due to CONFIG_CFS_BANDWIDTH not being set in the kernel configs in both Raspbian and the latest HypriotOS versions. To date, I've not found a single operating system that's built for the Raspberry Pi platform, and is both lightweight and also fully Kubernetes-compliant.

The solution to all operating system issues here is Armbian. It's by far the best OS out there for a headless single-board computer in my humble opinion. It has the necessary kernel configurations for CPU limiting and even has zram swapping and logging to zram pre-installed and configured.

Best of all? It's very lightweight.

Support of Raspberry Pi

Hello everyone, I would like to build an image for the Rasperry Pi 1, 2 and Zero. I could not found a download for a RPi on armbian.com. Then I saw that `rpi` is disabled in `/lib/configuration.sh`. Even when enabling it, I get an error during build, saying [ error ] Building failed, check config...

Armbian forumMathiasRenner

Unfortunately, the Armbian developer community are unable to support the Raspberry Pi platform due to issues with it running a proprietary firmware that has full control on its own Broadcom proprietary hardware (I know, surprisingly right? For all the open-source support Raspberry Pi gets).

Their concerns are justifiable and understandable, but the consequence is that I can no longer see the day when such a fantastic OS will come to the Raspberry Pi 4.

Storage Performance

The Raspberry Pi 4 runs on the same micro-SD card storage as its predecessors, as do many other affordable SBCs on the market. Normally, this isn't a real issue for me until I got my hands on a Rock Pi 4A which has an eMMC storage module as an option.

The performance gain is nothing short of astounding. I'm getting around 150MB/s reads and 100MB/s writes on the 32GB eMMC storage module (fio benchmarks to come in a full review!).

As much as I agree with the motivation behind the use of micro-SD cards in Raspberry Pis, eMMC modules have become much cheaper over the last few years due to economies of scale, plus the performance that eMMC confers is significant enough that I can no longer turn a blind eye to this feature. There are many boards at a similar price point (Banana Pi, Rock64, Rock Pi) that have eMMC support, and on hindsight, the Raspberry Pi 4 is priced a little too steep for the specs it offers.

Storage Longevity

A month before I decided to sell the Raspberry Pi 4B, I experienced an SD-card failure on one of my Raspberry Pi 3Bs in the Kraken cluster. What's worse is that the failed node happens to be my Kubernetes master. That essentially resulted in the loss of the entire cluster's state and I had to reinitialize the cluster from scratch, restore application state from backups and contend with some data loss, all of which did not make for a particularly fun experience. This made me a little paranoid of SD-cards since and I started subconsciously keeping time in my mind, counting down to the day that the SD-cards would fail.

eMMC storage would mitigate this issue as it has a longer lifespan as compared to an average micro-SD card due to higher quality NANDs being used. I've explained this NAND quality issue and SD-card wear in a previous piece on log2ram, do give it a read if you'd like to know more.

Extend The Lifespan of Your Raspberry Pi’s SD Card with log2ram

Log2ram is a software that redirects logs to the memory instead of the SD Card. Here’s how I used it to extend my SD Card’s lifespan in my Raspberry Pi.

Ikarus.sgWill Ho

Hello Rock Pi 4A!

In March 2020, a month into using the Raspberry Pi 4B, I chanced upon the Rock Pi 4. There are 3 variants of Rock Pi 4 at the time of writing, 4A, 4B and 4C, with the main difference being the absence of WiFi and Bluetooth on the 4A and Dual-display support on the 4C. I chose the 4A as I run a cluster of home servers and will probably never see the day I'll use any of those wireless features or even a single display, let alone dual-display.

Here are the core specifications of interest for a headless setup:

Of course, an SBC setup is not just about the board itself. Zooming into the full costs involved in getting a working setup:

From this view, we see that a Rock Pi 4A setup costs S$7.52 (US$5.69) more than that of a Raspberry Pi 4B 4GB. Assuming that you'd like to go for an equivalently-sized eMMC storage (which you always should given the option!), the price difference widens slightly to S$18.65 (US$14.10).

Interpreting the Numbers

What all of these means is that at S$7.52 (US$5.69) more, you get a device that has 2 more cores, option for both eMMC and NVMe SSD storage and better power compatibility, supporting both USB-PD and Qualcomm QuickCharge power sources.

Topping up another S$11.13 (US$8.41) will net you a 32GB eMMC storage module which has 3-5x the performance of your 32 GB SD card, depending on the speed class of the SD card we're comparing the eMMC against.

eMMC or Not?

Opting for eMMC simplifies your setup by reducing the number variables given that there is only 1 eMMC type compared to the gazillion SD Card speed classes and their respective compatibilities (or lack thereof) with each SBC. Goodbye UHS-1, UHS-3, Class 10, V6, V10, V30, V60, V90!

Fun fact: There's even an entire page on the SD Association site explaining the various speed classes.

Besides simplicity, using eMMC storage increases the overall reliability of your setup as well due to the fact that eMMC uses higher quality NANDs than SD cards despite sharing the same technology.

I've had SD cards fail on me as I explained briefly earlier and it wasn't something I'd like to relive any time soon. Paying a couple more dollars to get all these benefits and a peace of mind is a no-brainer in my opinion.

What the Numbers Don't Say

All the accessories listed (eMMC, heatsink, USB-PD power brick, USB Type-C cable, are official, i.e. they can be bought from the same online store. To put into perspective how important this is, let's compare the accessories experience with the Raspberry Pi 4.

For the Raspberry Pi 4 I had to:

1. Source the heatsink + fan separately
2. Test the cooling capacity vs workload of the cooling solution purchased, repeat 1 & 2 until sufficient cooling is attained
3. Pay an extortionate price for a power brick with a built-in USB Type-C cable with no way to extend the reach of and no other use than to power the Raspberry Pi 4
4. Finally after all that, still having to source an SD card with a compatible speed class and hopefully at a reasonable price,

There were no such frustrations for the Rock Pi 4:

1. The official cooling solution was passive (thank goodness!) and well-tested by the community and the manufacturer over several iterations.
2. The power brick is universal and you may use any USB Type-C cable of any length to carry power to the SBC.
3. The power bricks are USB-PD and very affordable, meaning that should you decide to decommission your device one day, the power brick may be repurposed somewhere else.
4. Usage of eMMC would eliminate SD-card compatibility and reliability concerns.

Having a one-stop shop for all the required accessories to get a reliable, running setup without breaking the bank is a huge deal for me. I can now just make the purchase and focus on planning my infrastructure and compute allocations.

What the Rock Pi 4 Cannot Do (Yet)

Before I sold the Raspberry Pi 4, I was using it for transcoding my videos on the fly as I watched them on Jellyfin. The Raspberry Pi, having possibly the largest community among all SBCs out there, unsurprisingly, has the best support for hardware-accelerated transcoding on open-source media streaming platforms.

The Rock Pi 4, on the other hand, is not the most popular SBC, though the popularity of its processor, the RK3399, is without doubt, rising in the SBC community. Therefore, unfortunately, I wasn't able to run hardware-accelerated transcodes on the Rock Pi 4A, so it's not really a drop-in replacement for the Raspberry Pi 4.

That wasn't the end of the world for me however, as I pivoted to another transcoding solution instead. I started looking into boards that have the amd64 CPU architecture (i.e. Intel/AMD x86 64-bit processors), relying on VA-API hardware-accelerated transcoding instead.

For the uninitiated, here's the definition of VA-API:

Video Acceleration API (VA-API) is an open source API that allows applications such as VLC media player or GStreamer to use hardware video acceleration capabilities, usually provided by the graphics processing unit (GPU).
...
The VA-API specification was originally designed by Intel for its GMA (Graphics Media Accelerator) series of GPU hardware... (Wikipedia)

I did some quick digging on amd64 architecture boards and it turns out that at that point in time, Radxa had just released probably the cheapest amd64 SBCs around in the market, the Rock Pi X (more on that in another post). With the same level of official accessories support, I bought the Rock Pi X, and have been using it ever since for transcoding. Turns out I could get equivalent transcoding performance to the Raspberry Pi 4B on active cooling, without the inconsistencies associated with thermal-throttling. I'd call that a win!

My hope is that the RK3399 processor would be better supported by the growing community and eventually obtain some semblance of hardware-accelerated transcoding in the near future.

Wrapping Up

If you're still reading at this point, you're most likely sitting on the fence as to what SBC to buy to set up your home servers.

In the past, I used to think that Raspberry Pi alternatives were not worth considering even if they were significantly cheaper as I had the feeling that they were not well supported by the community and were often buggy.

It may have held true in the years before 2019, but in 2020 alone, I have seen the alternatives improve leaps and bounds, and at this point, they are not just on par, some are perhaps better in terms of hardware specs and software reliability than the Raspberry Pi.

At this point, I can no longer in good conscience, recommend a Raspberry Pi to anyone anymore.

My personal advice to you is:

Skip the Raspberry Pi 4B, save yourself some headaches, get a Rock Pi 4 or one of those alternatives.

I went through these headaches and shared them here with you so that you don't have to.

Don't try this at home 😉