Can I Break This? (RailsConf '21)

This year I had the opportunity to present a talk at RailsConf 2021!

The conference was 100% remote, so I recorded everything in advance, and then had a live Q&A session over discord. As of May 2021, the conference-branded version of the talk is now available on YouTube:

Click here to download my slides (with code errors corrected):

Errata:

  • At 20:00 I demonstrate the use of lock! on two account records. A couple of conference-goers rightly pointed out that such a strategy could be subject to a deadlock, a topic that I cut for time. In practice, deadlocks are one of many low-probability reasons a method might fail, so thankfully the resiliency strategies I cover should help mitigate them.

    I'd also add that it's not actually the explicit use of lock! so much as the transaction that is the source of the deadlock. Most SQL databases will implicitly lock rows on UPDATE or DELETE, and by eagerly locking both records up-front, we may actually be reducing the overall chance of a deadlock occurring.

  • At 24:30, the slide should read deliver_later instead of deliver_now.

  • At 27:39, the slide should read deliver_later instead of deliver_now.

  • At 35:01 I state that most job backends guarantee "at-least-once execution," but on second thought, I'm not actually sure this is true. My understanding is that Redis-backed queues tend to use a delete-on-pickup strategy, which means that if the server process crashes or is halted, these jobs may never have a chance to complete.

    The pro version of sidekiq claims to optionally support at-least-once durability, by relying on Redis' atomic RPOPLPUSH to move jobs to an in-progress queue, deleting them only after they complete. (Similar to the way that most DB-backed queues work by default.)

    Of course, if you can't guarantee message delivery at the time of enqueue, then you effectively don't have at-least-once execution guarantees, regardless of your queue backend.

  • At 36:57, the upper right save method should read something like if !order.in_progress? ... instead of if !order.completed? ... because the subsequent update sets in_progress: true, and we must check that state in order to avoid race conditions. (I had been planning on adding a segment covering state machines, but it got cut for time. If you're interested in how I track state across parts of persistence operations, check out the steady_state gem!)