New Site Name!

Running with scissors will set your computer on fire. Maybe.

I’ve renamed this site from “Nightmare Before Devops” to “Running Production With Scissors.” I chose the original name years ago from a joke that came out of a conversation at DevopsDay, but it doesn’t feel like what my posts are actually usually about at this point.

Also, the nightmares continue.

And the new .run TLD makes the domain name its own little programming joke.

Son of Coyote

Back in January, I recreated a write-up I had done long ago of the Coyote/Road Runner cartoons to determine the root cause patterns of Coyote’s inevitable failure to catch the Road Runner.

Lo and behold, Warner Bros. has put much of the Looney Toons oeuvre on HBO Max. While it does not currently offer every existing episode featuring Wile E. Coyote, it does host a large number of them, including the very first installment from 1949.

They also have a couple of the handful of Wile E Coyote vs. Bugs Bunny episodes. These entries tend to stand out from the Road Runner cartoons for two main reasons: Coyote speaks in the BB cartoons, which he never does in the RR cartoons, and his plots and devices tend to function as planned. His failures to catch Bugs instead come about because Bugs always remains at least one step ahead. I’ve omitted them from this analysis.

A couple other production notes:

  • All cartoons until the mid-60s were directed or co-directed by the legendary Chuck Jones. (I’ve noted in the table which were not directed by Jones.)
  • The batch from the mid-60s look much cheaper than the movie-theater quality cartoons from the 50s because they are much cheaper. They were produced for weekly Saturday morning cartoons, so watching them sequentially really highlights the frequent reuse of some scenes (for example, Coyote poring through the ACME Mail Order Catalog and then dropping his order in the mailbox), backgrounds, and props.

Notes about methodology:

  • I did not include cartoons from the first episode, even if they are available on HBO Max.
  • I’ve added a new outcome called “Outsmarted,” for plots and devices that might have succeeded had not the Road Runner tricked or outwitted the Coyote.
  • I was tempted to distinguish between “Engineering Failures” and “Engineer/Operator Failures” but I did not actually end up breaking them out.

I did a trap-by-trap write-up for the first cartoon, Fast and Furry-ous. Subsequent cartoons are summarized in the table.

  • Fast and Furry-ous (1949 C. Jones)
    • Traps
      • Trap 1: Hides in crevice at side of road, holds a… shield? Pan lid? In road as R. Runner approaches
        • Result: R. Runner stops short just in front of shield? Pan lid? Turns, runs in the other direction. Coyote drops lid on the road in frustration and prepares to run after R Runner. R Runner returns, picks up shield? pan lid? and Coyote runs straight into it.
        • RCA: Eng/Operator error.
      • Trap 2: “One Genuine Boomerang” (no brand listed on box)
        • Result: When Coyote throws the boomerang, it immediately returns from the other direction and smacks him in the back of the head. Except it wasn’t actually the boomerang he threw. R Runner had his own identical boomerang and was standing right behind him.

          When Coyote turns to run after R Runner, his boomerang returns… and hits him in the back of the head.
        • RCA: Outsmarted by R Runner, followed by operator error due to distraction
      • Trap 3: Coyote paints a crosswalk with School Crossing sign on the road, then dresses as a schoolgirl skipping across.
        • Result: R Runner speeds through the crosswalk, sending Coyote spinning. R Runner then returns holding a sign that reads “Road Runners Can’t Read.”
        • RCA: Eng failure. Coyote should know that road runners can’t read.
      • Trap 4: Coyote straps himself to a rocket, aims for R Runner on a facing mountain grade, then lights the rocket’s fuse.
        • Result: The rocket inexplicably launches straight up into the air, sending Coyote’s head through a rock outcropping immediately above.
        • RCA: cartoon physics. Although, really, strapping yourself to a large explosive does not seem like a good idea.
      • Trap 5: Coyote situates a boulder on a cliff above the road, with a plumb line hanging from the boulder above the road below, planning to remove the boulder’s keystone just as R Runner passes below, thus crushing (HA-HA!) R Runner.
        • Result: once the keystone is removed, the boulder initially begins leaning toward the road but then changes direction and crushes Coyote, who is standing behind it.
        • RCA: cartoon physics
      • Trap 6: Coyote paints a fork in the road which leads into a trompe l’oeil he paints into a sheer rock face.
        • Result: R Runner comes racing along and runs into the trompe l’oeil as if it were a real tunnel. Coyote, surprised, decides to run after him, smacking into the rock
        • RCA: cartoon physics + knee-jerk reaction
      • Trap 7: Coyote plants a stick of dynamite in the middle of the road, covering it with a pile of dirt.
        • Result: When Coyote pushes the dynamite’s plunger down, it explodes in his face.
        • RCA: defective product, probably.
      • Trap 8: Coyote dons an ACME brand “Super Outfit” and attempts to fly off a cliff.
        • Result: Coyote falls off the cliff, crashing into the ground below.
        • RCA: Engineer stupidity
      • Trap 9: Coyote straps a snow generator to his back which he build with a refrigerator, an ACE (not ACME) brand electric motor, and a meat grinder, dons a pair of skis, and attempts to ski down an rock mesa straight into R Runner.
        • Result: He builds up too much speed, flies across the road and off a cliff on the other side.
        • RCA: You have to ask?
      • Trap 10: Coyote puts on “Fleet-Foot Jet-Propelled Tennis Shoes.”
        • Result: R Runner taunts him, leading him into a series of highway overpasses until the shoes apparently run out of fuel.
        • RCA: A draw
      • Trap 11: The shoes run out under a sign reading “Short Cut,” which Coyote dutifully follows, then hides behind a billboard for “Jones Motel” while holding an axe.
        • Result: When Coyote hears R Runner coming, he jumps into the road, draws back the axe… and gets run over by a bus. In which R Runner is a passenger, making the “BEEP BEEP” call.
        • RCA: I’m not sure how to call this one.
NameEngineering / Engineer FailuresACME FailuresCartoon PhysicsOutsmartedEng SuccessesUnknown / OtherNotes
Fast and Furry-ous (1949)5031.5(.5 x draw)
(1 x unknown)
= 1.5
Hook, Line and Stinker (1958)5
(including 2 x .5 partials)
(including 2 x .5 partials)
Unbranded product failures: sledgehammer with poorly attached head; dynamite detonating wire which re-rolls itself back up; sticky pulley
Beep Prepared (1961)5010.5.5Dir: C Jones & Maurice Noble
To Beep or Not to Beep (1963)6.503.5001Dir: C Jones & Maurice Noble; most failures involve a catapult which, it turns out, was built by Road Runner Manufacturing Co. That Coyote continued to use it after the first 2-3 failures is its own failure
Boulder Wham (1965)3.5002 x .5.50Dir: Rudy Larriva
Hairied and Hurried (1965)3
(2 + 2 X .5)
.51.501Dir: Rudy Larriva;
Partial ACME failure: pop-up barrier pops up randomly
Highway Runnery (1965)301100Dir: Rudy Larriva
Chaser on the Rocks (1965)2.5
(1 + 3 X .5) Rudy Larriva
Shot and Bothered (1966)1.502101.5Dir: Rudy Larriva
Out and Out Rout (1966)3.50.5200Dir: Rudy Larriva
The Solid Tin Coyote (1966)700000Dir: Rudy Larriva;
Coyote builds a giant remote-controlled replica of himself; hijinks ensue
Clippety Clobbered (1966)2 x .502.53 x .51Dir: Rudy Larriva
Sugar and Spies (1966)401300
Dir: Robert McKimson;
Coyote gets hit in head with a spy kit and uses contents throughout
Chariots of Fur (1994)3.502
(1 + 2 x .5)
01.51Dir: Chuck Jones
The Whizzard of Ow (2003)1002 x .53.5.5Dir: Chris Kelly;
2 battling wizards in the sky; ACME Book of Magic + a black cat fall into Coyote’s den

Note about “Hairied and Hurried:” includes Coyote performing a test of a kite-delivery system using “practice” bombs. However, the production bomb is much larger and slides off the kite line back to Coyote. Yeah, I’ve never seen that happen in production environments.

Eng FailuresACME FailuresCartoon PhysicsOutsmartedEng SuccessesOther / UnknownTotal Traps

Even with the introduction of the “Outsmarted” cause, the percentages are reasonably close to those in the original post, all of which date to the 1950s by the original directing/writing pair of Chuck Jones and Michael Maltese.

Riddle Me This Job Listing

Jump to the Site Reliability job description best practices

In Ancient Greek mythology, the Sphinx guarded the entrance to the city of Thebes, challenging would-be entrants with a riddle and eating anyone who failed to answer correctly. Tourism apparently was not a major industry in Thebes, because no one could solve the Sphinx’s riddle, and it apparently did not occur to the Thebans that they should maybe build a second entrance to the city.

One day, Oedipus, fresh off murdering his father, the (now former) king of Thebes, met up with the Sphinx. She sprang her riddle on him and… he answered it correctly. The Sphinx, apparently unable to come up with a new riddle or any other skill to feed herself, then threw herself off a cliff. Oedipus entered Thebes where, as a reward for single-handedly rescuing the city’s tourism industry, he was made king and married off to the queen, AKA his mother.

Most ancient sources did not specify the riddle, although the one now commonly associated with the story goes like so: What walks on four feet in the morning, two in the afternoon and three at night?

Answer: An SRE crawling out of bed to get their laptop after getting paged at 3AM, somehow managing to walk upright most of the day, then crawling back into bed at night using only one hand because the other is carrying their laptop.

I’ve been on both sides of SRE job postings and I’m actively in the middle of a job search now, so I’ve got a lot of opinions about what SRE job descriptions should and should not look like. The list comes from efforts trying to encourage diversity of applicants, to optimize for both my time and the potential candidate/employer, and not to be a total asshole to job seekers.

Karen’s List of Best Practices for SRE Job Postings

1. State the Opening’s Level

I see a lot of listings for “Site Reliability Engineer” with no indication of level. Is it entry-level? No, they seem to want some experience, although they often don’t give concrete ranges. While in some rare cases, generally at startups, none of the engineering listings may have specified levels, I will usually see this on a site where the product dev levels are spelled out in the listing title. I suspect, in many of these cases, the eng manager comes from a product dev background and either has no idea how to level SREs or possibly even thinks any amount of experience after 2-3 years is irrelevant.

Whatever the reason, please do one of the following:

  • Explicitly state a level in the listing so the candidate doesn’t waste their time clicking to find out you want someone junior.
  • If you’re open to multiple levels, say that: e.g. “Sr or Staff SRE opening”
  • If you have multiple openings at different levels, say that
  • If you frankly have no idea what level SRE you need, say that, unless you think someone may hoodwink you. (I never would, though.)

Also consider that SREs can come from a range of different engineering roles. If you’re ok with hiring a very experienced developer who wants to become an SRE but may need some training, state as much in the description (and be honest beforehand whether you have the people to teach them and that you really will give them the support they need).

Of course, putting a potential salary range serves as another way to hint to people about whether or not the role is a level-match, and it could also provide secondary signals if you do not get any resumes from people who are actually qualified.

2. Keep the “Must-Have” Specific Technology List Short

And by short, I mean pretty much non-existent.

Look, a smart, motivated engineer can learn whatever thingie they need. Seriously, they weren’t born knowing how to PXE boot a goddamn bare-metal Linux server. They can learn. You want to make sure they are willing and able to learn your shit, not that they come in knowing it, because, guess what? Your shit will change over time anyways. That’s what technology does.

You probably do want to make sure they have had to deal with “a” database, “a” monitoring system, “a” build system, but it really shouldn’t need to be “my specific DB/dashboard/pipeline.” Now, if you really, really need someone who knows, for example, Kafka inside and out because [stuff happened] and you need someone to hit the ground running NOW, say that. That’s ok; that’s good, even. But if you need someone who is already an expert with running a dozen+ specific pieces of software, especially if they’re complex, you probably need to get real, be ready to shell out a lot of money, or take a hard look at how you’ve chosen your stack’s tech. Also maybe try to figure out how to retain the engineers who knew this stuff in the first place, because apparently losing that knowledge creates… issues.

And who cares if a candidate has “worked with” Docker for 3 years? What if all they’d done that entire time was run docker pull and docker run? I guess you could list specific tasks they have performed around Docker, but now we’re back to that thing about not making a laundry list of required experiences.

Here’s another important reason to keep this list short: you’re selecting against diversity, particularly against female candidates, who tend to be much less likely to Dunning-Kruger themselves into thinking they have a depth of knowledge which they do not actually possess.

So just stop with these goddamn everything-but-the-kitchen-sink “requirements” lists, people. I’m going to start shaming you on Twitter, for real. (Yeah, I’m sure you’re quaking at the thought of my 3 followers seeing that.)

3. Talk About Your Tech Stack

While you don’t want to list every (or any) component of your entire tech stack and platform infrastructure as “must-haves,” you should still let the job seeker know what you use. They may think running SQL Server on Linux is the most fun ever, or they may see that you still monitor everything with Nagios and you aren’t moving off, which is a major deal-killer to them, as it would be to any sane person. And adding specific tech will also make searches by keyword easier, so candidates with or interested in matching platform skills may even be able to find you more easily.

Again, just make it clear that these specific services are not requirements or even nice-to-haves. “Hey, this is some of the stuff we use, in case you’re interested! If you don’t know these, though, that’s ok, too! We’ll figure out if you can learn them when we interview you because that’s something we should be trying to figure out in the interview anyway, right? Right?”

4. Explain What Your SREs Actually Do

I should maybe put this one at the top. Do you think an SRE sits downstream of product engineers and just has to somehow make their pile of code work? Or do your SREs (or your vision of SRE, if this is your first) have real standing, make decisions for the wider engineering team about how to measure and improve service stability and performance, and work directly with the other teams at all stages of product lifecycle? Is site relability (uptime, happy customers, etc.) a first-class citizen or just necessary overhead? And if your answer is the latter, maybe, um, spend some time thinking about that.

5. Be Real

Maybe forgo the standard lists of what you’re looking for and talk instead about what the org actually needs. “We want to improve the productivity of product engineers by reducing the number/duration of production incidents that eat their time.” “We are building out our next-gen platform and want a collaborative SRE to help build resilience into the design and implementation.” Etc.

6. Be Honest

This one should go without saying, but I know I’m not the only person who took a job that bore no resemblance other than job title to what was published and then discussed with the recruiter, management, and interviewers. People hired under deliberately or accidentally false pretenses will probably leave fast and now you’re back at step 0 in filling the role, and how much fun was that for anyone the last time?

7. Write for Diverse Candidates

I’m not an expert on exactly how to do this, although not having a white male engineer write the listing would be a good starting place. There’s plenty of information on the Internet with advice on how to approach inclusive listings, though. And while it should go without saying, if the hiring manager and the recruiter and everyone else touching the listing are all part of the majority tech demographic, find someone who does not look and sound like them to check the description for tone and other red flags. And if all your candidates still look like your hiring manager and existing team, fix it. A number of consultants and services exist who specialize in helping recruit for diversity.

Just make sure you actively address inclusion and equity so you can keep those employees once you actually find and hire them. That’s another story, though.

Obviously, all these points are optional. And they only partially address a worse problem, namely that most organizations seem to have job listings that could be interchangeable, give or take a few weird combinations of specific required hands-on experience, when the reality of the role in each org varies greatly. That lack of awareness or honestly is also something that seriously needs to change because I can’t imagine whom it actually benefits in the long run.

But that’s also a topic for another post.

Blog at

Up ↑

%d bloggers like this: