Managing Critical State and Distributed Periodic Scheduling with Cron

<p>Sign up for the Thursday night companion discussions at <a href= "https://bookclub.dev/thursdays">bookclub.dev/thursdays</a>. Discussions go from 7-9 pm Eastern.</p> <p>This week we look at chapter 23 <a href= "https://sre.google/sre-book/managing-critical-state/">Managing Critical State: Distributed Consensus for Reliability</a> and chapter 24 <a href= "https://sre.google/sre-book/distributed-periodic-scheduling/">Distributed Periodic Scheduling with Cron</a> from Site Reliability Engineering: How Google Runs Production Systems.</p> <p><strong>Things to monitor with a distributed consensus system</strong></p> <ul> <li>The number of members running in each consensus group, and the status of each process (healthy or not healthy)</li> <li>Persistently lagging replicas</li> <li>Whether or not a leader exists</li> <li>Number of leader changes</li> <li>Consensus transaction number - in a healthy system, this number will be incremented over time</li> <li>Number of proposals seen and number of proposals agreed upon</li> <li>Throughput and latency</li> <li>Latency distributions for proposal acceptance</li> <li>Distributions of network latencies observed between parts of the system in different locations</li> <li>The amount of time acceptors spend on durable logging</li> <li>Overall bytes accepted per second in the system</li> </ul> <p><strong>Projects mentioned</strong></p> <ul> <li><a href="https://zookeeper.apache.org/">Apache ZooKeeper™</a></li> <li><a href="https://www.consul.io/">Consul</a></li> <li><a href="https://etcd.io/">etcd</a></li> </ul> <p> </p> <p><strong>Articles from this week's chapters</strong></p> <ul> <li><a href= "http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.24.3690&rep=rep1&type=pdf"> Harvest, Yield, and Scalable Tolerant Systems</a></li> <li><a href= "https://awoc.wolski.fi/dlib/big-data/GiLy02-CAP.pdf">Brewer’s Conjecture and the Feasibility of</a><br /> <a href= "https://awoc.wolski.fi/dlib/big-data/GiLy02-CAP.pdf">Consistent, Available, Partition-Tolerant Web</a><br /> <a href= "