blog.md

Improving embedded software development¹

Most firmware sucks. To be frank, most firmware developers suck. But if you googled for this article or it came up in your feed, you're likely a decent engineer trying to improve². This article is for you.

But maybe this article isn't for you? I mean, you look pretty smart with your videogame t-shirt and ripped jeans and all. Before you overcommit the three minutes to read this, check the

I'm Too Powerful For This Article Checklist

1. my source is under version control and hell yeah
2. my source is on github/bitbucket/a server. Of course
3. my source is testable and
4. my tests run every time I push to remote. This means
5. my source is machine-independent. I don't need hardware to run, seriously. And absolutely
6. I have an automated "build, test, archive" pipeline like Jenkins. And I always use the Oxford comma.

Still here? Great. Let's begin.

Why firmware development sucks

Firmware developers often come from a software-adjacent field like mechanical engineering, computer engineering, or electrical engineering. They are technical enough to be effective and willing to do work software engineers don't want to touch because eww yuck hardware amirite? But because of this, many firmware developers lack formal software training and best practices. It's not all their fault, though. Consider the runtime and development environments:

1. Logging is hard (no filesystem, no persistent storage, no network, and no console).
2. C is still heavily favored over C++ by SDK and library developers, so put away your ultra-modern 30 year old language and use this 50 year old language instead.³
3. printf debugging can bog down the cpu and bus, distorting runtime behavior.
4. Try getting peripheral devices to pause while debugging. Go on, I'll wait. They won't though! Bwahahaha. A little hardware joke for you.
5. Multithreading is nonexistent. Or rather, you have main and interrupt threads, which is not the threading model you learned in school, which you didn't learn in school because you were studying fluid dynamics.
6. The tools are old, expensive, proprietary, quirky, and hard to script.

Bad news first: The firmware environment will not improve. The tools will remain shitty. The languages will remain old. Embrace it. Because [firmware] is war baby. And war is hell. ⁴ But firmware developers can improve. If you can learn any technical discipine, you can learn this too. It is in fact quite easy: Write software instead.

Applying software craft to firmware

Software, like all engineering, is a craft. It is not art⁵. No one needs the muse at their shoulder to write solid code. While the three practices described here aren't comprehensive⁶, when applied to firmware, they will improve that firmware.

First, the basics.

Version control and versioning

It's 2020 and I shouldn't have to start here. But I do.

Embedded codebases are often small. Firmware teams start out small. Poor project hygiene is often excused or unnoticed. But this poor hygiene doesn't scale for hardware development timelines, manufacturing, production, revision, testing, verification, automation, or team growth. Use version control from the first line of code. Take your craft seriously⁵.

1. Use git and gitflow⁷ to organize your work into different feature, fix, and release branches. yolo push to master is not a branch management strategy.
2. Use a remote hosted repo. Accountability improves code quality.
3. Use embroidered semantic versioning to fully define the build product. An example from a recent codebase: major.minor.patch.bootloader_version.hardware_target.info.build_type.

Second, the code.

Testability

For our high-impact-low-word-count article we will focus on a single practice that will most directly improve firmware: Testing.

Testing is the way and the truth and the life.

And I don't mean Big T Big D Big D ⁸. I mean testable firmware. You already test your code to some extent. Here's what I mean:

Testability Criteria

1. Data independence: All data, and context (state) are supplied to functions as parameters, with no direct access to system data or special registers.
2. Platform indepedence: All hardware-specific registers and includes are isolated to hardware-specific inline files. It can targeted to custom hardware or development PC at buildtime without edits.
3. Automated: All testing runs automatically at buildtime without edits.

Firmware is often hard to test. The codespace can be small, printf debugging has serious drawbacks, logging is unavailable, and realtime processes (BLE, peripherals, etc) stall when testing. So let's just...not deal with that and write machine-independent firmware instead. Target it at a PC for testing firmware and target it at custom hardware for everything else (including testing hardware).

There is another, more significant, benefit: Better code. Code written to be machine-independent is less configuration (state, context, runtime) dependent. Functional code⁹ is usually easier to debug, to revise, to verify, and to reuse.

As bluntly as possible: Untestable code should not be written. If you are writing untestable code, stop.

Data independence

One way we write untestable code is making it depend on data and state that aren't easily replicated in the test environment.

Take a look at your driver code (SPI or flash or accelerometer are good places to start). Look for any functions that use statically defined context. Look for any direct access to machine registers (which are essentially global variables). We can achieve data independence by rewriting our code to transform only data, not context.

Data Independence: Example

https://gist.github.com/9dafa0f47c4460d2d721fd8a0df367d1

Hardware Independence

Another way we write untestable code is to make it impossible to compile without the intrinsics and runtime libraries available only for that target. Hardware independence means we isolate any hardware-specific details at writetime and then we target it at buildtime.

Take another look through driver files. Look for any platform-specific includes or libraries. These are effectively hardware too, unless your SDK and library providers also provide x86/linux/whatever builds. Look for any machine intrinsics or platform-specific types. There's probably quite a lot! That's all hardware too.

The solution is well-known and has a catchy acronym: Write a Hardware Abstraction Layer ("HAL").

Nota bene¹⁰: Chip makers sometimes provide HALs but they are there to re-target the code to hardware variants. Your HAL should provide a complete separation between platform (code, registers, intrinsics) and application.

This approach to writing HALs is not as thorough nor comprehensive as some but it is very low overhead.

Hardware Independence: Example

In firmware you occasionally need a busy-wait. To prevent multiple files from containing #include "nrf_delay.h" (a platforms-specific file), you write a wrapper function in time.c and tuck the dependency there: https://gist.github.com/a75320111ca00f7d2f1340917d440cdf

This works the other way as well, for example by writing print or log statements in .c files, then providing a minimal (possibly empty) implementation for your target and redirecting to stdout on PC.

Nota bene One serious mistake is to write essentially different codebases in different .inl folders while letting the .c files atrophy to mere shims. Avoid this at all costs. The more code you put in to .inl files the greater risk of having meaningfully different behavior on target hardware and developmnent platform. For example, if a function is ten lines of application code with one hardware-specific line, leave that function in the .c file. Create another function containing the single hardware-specific line. Move that function to the .inl and call it from your .c function instead.

Third, the automation.

Automate the Pipeline

An automated pipeline ("continuous integration") can feel like overkill, especially for firmware. Why bother? There is only one engineer on the team! Firmware doesn't change very fast! We only have a single seat of that weird compiler! We only built one hardware prototype! Etc.

But if we go to the effort of writing testable code, an automated pipeline is how we reap that most important benefit: Development time. It lets you go faster by making it safe to go fast. It tests all the things you didn't think about or forgot. It tells you if something broke immediately when you break it. It lets you codify the requirements, edge cases, and bugfixes as tests. Every test you write improves the current codebase. With an automated pipeline that test benefits you every subsequent build as well.

This section is purposefully slim because firmware often uses weird toolchains so it's hard to get very specific. But it can be made to work. For example, my company's Nordic development setup served 5 developers across 3 timezones but boiled down to one license on a stick pc using flexlm over vpn for development. This same stick pc became our build machine via bitbucket webhook with Jenkins. Lots of words! But minimal infrastructure. If you need some help, hit me in the comments.

Conclusion

That's all there is to it, just some simple ways to adapt software craft to the underserved and opaque firmware domain. Last, if you are building embedded devices that talk to mobile devices, consider how to improve iOS development by writing it more like firmware[link to ios article].

Footnotes

1. 0.1.0, 2/12/20 ↩

2. Bad engineers, on the other hand, tend to stay bad. ↩

3. C and C++ are awesome. But complaining about languages is fun! ↩

4. Pressfield, The War of Art. ↩

5. Creation and Destruction ↩ ↩²

6. There are many books on improving software development craft. My recent favorites are: Clean Code and Clean Architecture. ↩

7. There are many good articles but this is a good place to start. ↩

8. Test Driven Development. If you're interested,Test Driven Development For Embedded Systems book is a good start. ↩

9. functional programming improves firmware, too. ↩

10. Latin for "Hey I can translate latin on google". That will come in handy with the dominarum! ↩