This expansion card prototype was funded by Framework Computer through their Expansion Card Developer Program.

Closed-Case Debugging (CCD)

A day will dawn in every person’s life where they do something completely stupid. For some of us, that something will take the form of bricking (or very nearly bricking) our daily carry laptop.

A few of those somethings will be caused by hacking our laptop’s embedded controller1, and a few of those will be caused by a compiler upgrade2 (more on this later). If you find yourself facing either of these two things you might be able to recover – or at least diagnose! – your machine the old fashioned way.

Closed-Case Debugging (CCD) is pretty straightforward: you get access to a debug port without cracking open your machine. That debug port on the “device under test” (DUT) might be accessible through a headphone jack, four weird magnetic pins on the side, a USB A port with some questionable standards compliance, etc.

Chromebooks all3 have one! Some phones do, too.

In short, it’s so that you don’t have to do this:

I must stress that this is *not* closed-case debugging.

Framework Laptop 13

On the Framework Laptop 134 the rear right USB C port doubles as a CCD port, offering UART access to the embedded controller. It also just so happens that the embedded controller is very chatty and provides a full debug console.

This debug port works any time the embedded controller is on, which means that you can use it to debug the power sequence from another computer while your DUT is “turned off.”

Connect CC1 and CC2 through one 5.1kΩ resistor each to GND.

SBU1 and SBU2 will switch over to EC TX and EC RX respectively.

Debug port parameters:

  • 3.3V TTL
  • 115200 8N1
  • No flow control

This configuration is only usable in one orientation.

A 1 2 3 4 5 6 7 8 9 10 11 12
signal GND nc nc nc CC1
nc nc SBU1
nc nc nc GND
signal GND nc nc nc SBU2
nc nc CC2
nc nc nc GND
B 12 11 10 9 8 7 6 5 4 3 2 1

receptacle pinout

Unlike the aforementioned “USB A port with questionable standards compliance,” this is entirely compliant: the SBU (or Side-Band Use) pins are allocated for exactly this purpose! Okay, they’re actually allocated for an implementation-defined purpose. That’s the point. For this implementation, they’re allocated for this purpose. Don’t at me.

Existing solutions

The existence of dedicated CCD in phones, laptops, and other devices begat a number of devices to fill the need. Among those is Google’s Servo (now on v4). New Servo boards are almost unobtainable by the general public. They are, I’m told, highly flexible.

I do not own a Servo, so I cannot comment on its effectiveness.

You could also use a simple USB C breakout (make sure to get one that breaks out every pin and doesn’t include any CC resistors out-of-the-box).

EC Card 2

Not satisfied with the unavailability of a Servo and annoyed that I’d bought a couple USB C breakouts that, while they broke out SBU1/2 and CC1/2, couldn’t be coaxed into working properly5, I set out to design my own CCD breakout.

Because it’s for the Framework Laptop, I wanted to make it an expansion card. Furthermore, I wanted it to be totally self-contained and not require an external USB<->UART interface.

I settled on an FT230X and started with Framework’s Expansion Card reference design.

And wow was it ever ugly! My first pass at laying out a board is not at all worth sharing, but suffice it to say that I trusted the auto-router too much and hell, I didn’t even lay a ground plane. That was EC Card 1. It never reached production thanks to the fine folks over at Framework telling me that I had done a bad job and should feel bad6.

EC Card 2 turned out better.

I chose PCBway for prototype production and assembly7. They would not assemble the USB C connectors, however, so I did that on my own.

Combined with some expansion card shells 3D-printed at the library, it looks quite nice!

The final product
Including a cute 'how do I use this, anyway?' label. Sorry for the poor photo quality.

The design is open source at DHowett/framework-ec-debug-card (HowettNET GitLab) .

When you’re using it, you get serial output directly from the embedded controller.

Critically, it also works if the laptop is turned off.

The staggered pin header you see in the original layout was inspired by SparkFun’s locking headers. I intended for it to retain a 2.54mm pin header in place as a debug probe, but it doesn’t hold. It is still useful in that it lets you repurpose the board as a simply FTDI carrier or access SBU1/SBU2 without USB serial.

It turns out that 100 of these expansion card screws fit in a teaspoon!

Known issues

There are some!

Well first, the USB port is off center. Whoops.

In addition to that:

  • The microUSB port hangs over the edge of the card, and it is sharp!
  • There’s a minor power leakage from the pull-up on RX by the FT230X into the EC, which can power it when it is not supposed to be powered. The MEC1521 seems to be guarded against this causing any damage, but it does mean that on occasion the EC starts up and sends over a bunch of garbage before you click the power button.
  • FTDI chips are great, but maybe a bit over-featured and expensive.

A recovery story

I authored a guide on hacking your laptop’s embedded controller, and so I led at least two people down a path that resulted in them rendering their laptops unbootable. That sucked. As far as I can tell, it was entirely due to some undefined behavior2 that lay at the bottom of a well waiting for a well-meaning person to fall in.

I stumbled into this issue in exactly the same way anybody stumbles into anything: blindly. I reflashed my EC, rebooted, pressed power and pop. Nothing. A whirr of fans, and then all the lights went out.

The debug console told me that it was crashing only after I pressed the power button. The EC was running fine until then. And then, it was panicing with a watchdog failure.

Now I lucked out, because I had been working on bringing the RW firmware8 online. I could just reboot into a working copy… but those folks on the forum weren’t quite so lucky.

With access to the EC debug console in hand, and them with some access to UART as well, I built a payload that would overwrite the miscompiled code in memory as well as a build system to automatically produce that payload9.

It worked. It worked! Both of them recovered their machines.

However: I sent one of them an EC Card 2, and that didn’t work. Drat!

v3 plans?

I don’t know if I’m going to make more of these, but if I do I’d like to fix a couple annoyances.

  • nrp suggested using a USB C port instead of microUSB, so that it could be placed in a USB C expansion card shell.
  • I should set back the microUSB port to address the 1mm overhang where the sharp edges stick off the edge of the laptop and can cut your hand to ribbons.
  • Look at the voltage leak from the technician machine to the DUT, if possible10.

Makers: e-mail me!

I still have some of these, as yet unassembled. If you’re into EC hacking and feel like you could benefit from one, reach out to me with a description of your project!

  1. Hacking your Framework Laptop’s EC for fun and profit ↩︎

  2. An explanation on Framework’s EC repo, plus the fix submitted to the ChromeOS project ↩︎ ↩︎

  3. I think. I am not sure. ↩︎

  4. I know this to be true for the 11th generation, I believe it to be true for the 12th and nobody has any idea what the 13th or AMD boards hold. This is doubly true for the recently-announced Framework Laptop 16. ↩︎

  5. The breakouts were perfectly fine, I was just using a receptacle breakout and I did not have any cables that passed CC and SBU unmanipulated. ↩︎

  6. They are the nicest people in the world, and Kieran was an amazing help in getting the layout and the design just right. Nobody actually told me that I should feel bad. ↩︎

  7. The prototype run of 10 cards cost about $150 (2022, USD) assembled and shipped. ↩︎

  8. The EC has two copies of the firmware, but the second one is unused as of April 2023. I have an open pull request that makes it work again! ↩︎

  9. DHowett/framework-espi-recovery (HowettNET GitLab)  ↩︎

  10. It may not be possible, given the design of the sideband UART. VBUS is not provided by the laptop before poweron. ↩︎