Blogs

I used to think “being good on-call” meant being fast at debugging. But the older I get (in engineer years, which is like dog years), the more I think on-call is mostly about whether the system can tell you what’s happening.

At MedCost, I started leaning hard on traces and correlation IDs. Not in a fancy way—just consistent. Same header names, same log fields, same story across services. When something broke, I could actually follow a request like it was a breadcrumb trail.

Also: I finally wrote down the “first three things I check” during an incident and put it in a runbook. That tiny act saved me from re-learning the same lesson at 2:00 AM.

Switching into healthcare work was a reminder that code is the easy part. The hard part is making sure the code matches reality. Claims workflows have a lot of “it depends,” and “it depends” is where bugs go to hide.

I started keeping a little personal glossary. Not a formal doc—just my own notes like “this term sounds like it means X, but actually it means Y.” It felt silly, but it paid off every time I had to explain something back to someone else.

Also, I swear half of engineering is learning the right nouns.

First month on a new platform always feels like joining a TV show in season five. Everyone else knows the lore. I’m over here asking what a thing is called and why it exists.

I leaned into my favorite coping mechanism: diagrams. I drew the request flow. I drew the message flow. I drew the “who owns which database” map (which is always spicy). Every time I got confused, I updated the picture.

Outside of work I kept it simple—walks, a good playlist, and trying not to think about Azure resource naming at dinner.

The worst bugs are the ones that make you question your own memory. “Did I already fix that?” “Why is it only happening for this one user?” “Is the universe mad at me?”

I eventually realized the system was doing two similar operations that looked identical in logs. Once we started stamping the same correlation ID across the HTTP call and the follow-up message (and the downstream service), everything snapped into focus. It wasn’t haunted. It was just a normal race condition wearing a Halloween costume.

That week also reminded me to go outside. Debugging is faster when you’ve seen sunlight recently.

We had a change that was logically simple but operationally terrifying. You know the kind: touches a core payment workflow, has a bunch of “should never happen” edge cases, and would definitely happen.

We wrapped it behind a flag, deployed it early, and enabled it in small doses. When a weird corner case showed up, we turned it off for a subset of traffic, fixed it calmly, and tried again.

After work I celebrated the best way I know how: a snack, a dumb show, and the quiet joy of not being paged.

At GreenDot we started doing tiny reliability drills—kill a dependency, force a retry storm, and see what breaks. Nothing dramatic, just enough to keep our mental map honest.

We learned our dashboards were decent, but our alerts were a little too loud and a little too late. After tuning them, on-call felt less like guessing and more like reading a story.

The best part was the vibe: it felt like a team puzzle. We ordered lunch, fixed a few runbooks, and called it a win.

Building transaction workflows with messaging is awesome, because your services stop calling each other like anxious coworkers. But you pay for that freedom in one currency: reality.

Reality is duplicates. Reality is timeouts. Reality is a message that gets re-delivered when you least expect it. Once I accepted that, I stopped writing consumers that assumed the universe was polite. I started writing consumers that were basically like: “Yep, I’ve seen this before. We’re good.”

I also started getting oddly into retry policies. I’m not proud. But I am at peace.

The first few times I used Docker, I did the classic thing: I broke something, then I cursed Docker, then I went back to running everything locally “the old way.”

Later, we containerized services to keep environments consistent across dev and CI. The payoff was immediate: fewer mysterious “only fails in staging” issues, fewer setup arguments, and a way easier time collaborating across teams.

Outside of work I was also tinkering with small projects at home, so the “package it once” mindset started bleeding into everything. Honestly? Not mad about it.

We had an endpoint that was fine… until it wasn’t. Under load it slowed down and made everyone assume the API was the problem. Spoiler: it was the query.

I made a fresh coffee, pulled an execution plan, and did my favorite detective routine: “Where is the expensive operator and what did we do to deserve it?” Turns out we were pulling way more rows than we thought, then sorting in a way that hurt. Nothing magical—just a mismatch between our mental model and what the database was actually doing.

The best part of tuning is the moment after the fix when the graphs relax. It’s like the system lets out a sigh.

Working with frontend teams taught me to love boring consistency. Not “boring” as in uncreative—boring as in predictable. Same paging style. Same error shape. Same naming. The little stuff.

One day we got ahead of a looming deadline by just sitting down and agreeing on what the API would return, how we’d version it if we needed to change it, and what we’d consider a breaking change.

Afterward I felt the same satisfaction as reorganizing a toolbox. Everything is where you expect it.

At some point I realized I actually enjoy setting up little services for fun. Not because I’m trying to “grind,” but because it scratches the same itch as solving a puzzle.

I played with containerized apps, logs, dashboards, and the occasional “why is my network doing this?” moment. It made me calmer at work, because I got used to thinking in terms of: observe → hypothesize → test → repeat.

Plus it gave me an excuse to buy a label maker. If you know, you know.

Remote work has perks, but it also quietly turns you into a desk goblin. One week I realized I’d spent an entire day inside and my only travel was from chair to fridge.

I started taking a couple calls on a walk. Not every meeting works like that (please don’t walk during “incident review”), but for quick syncs it was perfect. I came back more focused, and honestly more patient.

It also made the Florida weather feel less like “the thing outside” and more like “part of my day.”

I had a low‑key plan: drive a bit, see family, read a book that wasn’t about software. Then a minor release needed a quick patch before the holidays. It wasn’t a fire, just a “hey, can you take a look?” situation.

We fixed it fast, wrote a note for the next person, and I actually got to unplug. It was a small win, but I remember thinking: this is what good process buys you—less chaos, more real life.

Also, I discovered I genuinely enjoy cooking when no one is paging me. Who knew.

I used to think runbooks were overkill. Then I got paged, groggy, and had to remember “that one weird workaround” from six months ago. My memory was not a reliable storage medium.

So I started writing tiny runbooks: where logs live, how to replay a message safely, what metrics matter, what “normal” looks like. Nothing fancy. Just enough that I could act confidently.

Bonus: it made onboarding easier, and it made me look way more organized than I actually am.

When I build an integration, I start with hope. Then I meet the edge cases: rate limits, timeouts, occasional 500s that recover if you just… wait. Eventually you accept that your job is to build a system that stays calm when somebody else’s system is having a day.

I learned to be very intentional about timeouts and retries. Not “retry forever,” but “retry like you respect the other service” and “don’t turn one outage into a bigger outage.”

When it went well, it felt like good plumbing. When it went badly, it felt like I was negotiating with the internet.

Early in my career, reviews felt like a debate club. Now I treat reviews more like a safety check: will this break prod, will this be hard to maintain, will someone else understand it at 2 AM?

The best trick I learned was simple: ask what problem the code is solving before commenting on the implementation. Half the time, the best review comment is “can you add a sentence explaining the why?”

And yes, I still leave nitpicks sometimes. I’m human.

In my early months working in fintech, I learned that the “real work” often happens after the happy path. You don’t just process a transaction—you reconcile it. You don’t just store a number—you explain where it came from.

That changed how I wrote services. I started thinking about audit trails, idempotency, and “can we tell a clear story about this event later?” because support and operations will absolutely need that story.

It’s nerdy, but I genuinely loved the feeling of a clean, balanced ledger at the end of a run.

2020 was weird for everyone, and my brain definitely felt it. Still, I was deep in building services, learning new patterns, and trying to keep my code clean while everything else felt… not clean.

I started doing a small thing that helped a lot: at the end of the day, I wrote down what I learned. One paragraph. Nothing fancy. It kept the weeks from blurring together and made me feel like I was actually progressing.

Also: I discovered that making coffee became a tiny ritual that separated “home” from “work,” even when both were the same room.

First week at a new job is always the same: lots of intros, lots of tools, and a bunch of tabs I’m afraid to close because I’ll never find them again.

I was working in .NET Core services, learning the transaction flows, and getting comfortable with messaging. I remember thinking, “okay, I get it,” and then immediately discovering a new queue or a new edge case. Humbling. Great. Love it.

Outside of work, I tried to keep my routine stable. Sleep, decent food, and the occasional “touch grass” reminder. Starting strong matters.

I used to treat communication like “extra.” Like, after I finished coding, I’d explain it. But projects got bigger, and I realized misalignment costs way more than a few minutes writing a clear note.

I started doing small things: leading with a summary, writing down decisions, and making action items painfully explicit. Not because I’m naturally organized, but because it kept work moving.

It also made my life outside work calmer. Fewer late surprises, fewer “wait, what are we doing?” moments. More time to enjoy regular human stuff.