DevOps'ish 327: AWS writes off Bahrain, forcing the registries to pay for open source, Kubernetes 1.37 hardening, and more

Your CI/CD pipeline is automated. Why are database changes still waiting on tickets?

Developers shouldn’t have to choose between moving fast and changing databases safely. Liquibase Secure puts policy checks, drift detection, targeted rollback, and audit evidence directly into existing delivery workflows across 65+ database platforms. See Liquibase Secure in action →

Kubernetes v1.37: Native Histograms Graduates to Beta (6 minute read)

Richa Banker of Google covers the NativeHistograms feature gate graduating to beta and on by default in v1.37, after landing as alpha in v1.36 under KEP-5808. The problem with classic Prometheus histograms is well known: you guess le bucket boundaries up front, every boundary spawns a _bucket series, and quantile math interpolates badly when the buckets are coarse. Native histograms use exponential buckets in a single time series instead. The implementation lives in k8s.io/component-base/metrics, so kube-apiserver, kube-scheduler, kubelet, kube-controller-manager, and kube-proxy all pick it up, with defaults of BucketFactor: 1.1 and MaxBucketNumber: 160. Components dual-expose classic buckets and native spans in the same Protobuf payload, so the upgrade itself breaks nothing. The Prometheus side is where you can hurt yourself. Text scraping only carries classic buckets, so you need Prometheus 3.0 or newer with scrape_native_histograms: true per job, and you must also set always_scrape_classic_histograms: true during migration or Prometheus stops ingesting _bucket, _count, and _sum and every histogram_quantile dashboard you own goes dark. Prometheus 2.40 through 2.x only offers a global feature flag, all or nothing, and that flag is deprecated in 3.9 and later. Rollback is a scrape config change. The order of operations: enable both formats, migrate queries to histogram_count() and histogram_sum(), verify alerts in staging, then drop classic scraping for the roughly 90 percent series reduction.

Kubernetes v1.37: Hardening Container Storage with Bind Mount Options and EmptyDir Permissions (7 minute read)

Nispriha Jagan and Neeraj Krishna Gopalakrishna of Red Hat walk through two alpha storage hardening features driven by SIG Node and SIG Storage. VolumeBindMountOptions adds a bindMountOptions field on volumeMounts so you can set noexec, nosuid, and nodev on the bind mount the runtime creates inside the container, and it works with emptyDir, PersistentVolumes, CSI, projected volumes, ConfigMaps, and Secrets, though not image volumes. EmptyDirVolumeMode adds a mode field to emptyDir so you can ask for 01777 sticky bit semantics or something tighter like 0750 instead of the hardcoded 0777. Both close a gap flagged in issue #48912 and the 1.24 security audit (finding NCC-E003660-7HM): readOnlyRootFilesystem does not buy much when any writable volume is a place to drop a binary and run it. Both gates are off by default and need enabling on the API server and kubelet. bindMountOptions requires a container runtime that supports the CRI mount_options field and advertises it via runtimeFeatures; the scheduler steers pods away from nodes that lack it, and the kubelet rejects the pod if one lands there anyway, so no silent degradation. mode needs no runtime support, but fsGroup overrides it, and a kubelet without the gate silently falls back to 0777, which is exactly the version skew case to watch. Linux only. Not to be confused with PV mountOptions, which the CSI driver applies at the storage layer. Track KEP-5855 and KEP-5502.

AWS says it can’t restore service to Bahrain, UAE facilities 6 months after Iran strikes (2 minute read)

Annie Palmer reports that AWS has posted its first public update since April on the Gulf facilities hit by drone strikes in March, and some of it is not coming back. On its health dashboard, AWS said damage at multiple Bahrain sites exceeded what its services are made to withstand and that “after a thorough assessment, we have determined that we are unable to restore access to the resources and data hosted exclusively in this region.” A separate post says one UAE availability zone is likewise unrecoverable while two others are still being restored. Back in March, Amazon said two UAE data centers took direct drone hits and a Bahrain facility was damaged by a nearby strike, told customers to move workloads elsewhere, and called Middle East operations “unpredictable.” Iran’s Islamic Revolutionary Guard Corps claimed responsibility, but the article does not have a customer count, a data loss figure, compensation details, or region identifiers by name. AWS promises more on the UAE “in the coming months” and on Bahrain in early 2027. The operational lesson is old and now has a case study: a region is a blast radius, not a guarantee. Data hosted exclusively in one region can be gone for good, and the durability numbers on the pricing page never covered ordnance. If cross-region replication and tested failover are still on your roadmap, this is the week to move them up.

Confessions of an Unrepentant Slop Snob (11 minute read)

Charity Majors spent the year telling everyone at Honeycomb that outcomes are all that matter and how the work got made is irrelevant, while developing what she calls a violent disgust reflex for AI-generated text. She admits she found Claude-isms catchy as recently as December 2025; by spring, phrases such as “I’d value your take on this” showing up in her inbox were triggering rage, and by summer it had spread to newsletters, blogs, and social. Inside the company, half the staff was furious about twenty-page slop docs and replies that open with “Claude says,” and the other half were furious that colleagues would not adapt. Her way out of the contradiction is a spectrum: language does functional jobs (diffs, structured data, formal proofs) where AI is just another tool, and relational jobs (opinions, intros, performance reviews) where the whole value is that a specific person thought it. The further toward the personal end, the more AI text reads as a trust violation, which she illustrates with a coworker’s line: “I wanted to know what THEY THOUGHT.” She hedges that the scale is not a universal truth, and the full framework lives in Honeycomb’s AI Norms and Values docs rather than here. The practical move is boring and correct: write down your team’s communication norms now, because “a human wrote this” is no longer the default assumption.

Why I Stopped Arguing With People (7 minute read)

Cong Wang, Linux kernel maintainer for the networking traffic control subsystem with over 1,000 patches since 2009 and now a startup founder, explains why he quit arguing for technical correctness in code reviews, design meetings, and mailing list threads. His case: “Sometimes I won on points and lost the person.” Most arguments are ego defense rather than idea exchange, and “we are emotional animals who occasionally think,” so bringing an airtight proof to a feeling accomplishes nothing because the feeling does not read. He leans on chapter 2 of the Tao Te Ching to argue that being visibly correct manufactures someone visibly wrong. The rules he lands on: discuss pros and cons with smart people, skip right-and-wrong fights with ego-driven ones, give advice only when someone explicitly asks, treat a competitor’s disagreement as your moat rather than a debate to win, and spend the saved energy eliciting feedback on yourself. The thin spot will be obvious to anyone who has maintained a subsystem: review exists precisely because letting people touch the stove is not an option once the stove is in production. Take the ego framing and the ask-first rule. Leave the consequences-are-the-only-teacher part at the door of any system where the consequences land on users.

Nobody pays for open source. We can force them to. (23 minute read)

Laurie Voss, npm co-founder who ran the registry for five years, has been sitting on this post since 2013 and finally has a thesis. Open source is an evolutionarily stable strategy: the most permissive license wins the code game every time (React in 2017, Elastic, HashiCorp, and Redis all relearned this), and the supply game goes to whoever is the default. That is how JFrog pulls $532 million a year, Docker $207 million, and Snyk roughly $326 million selling dependable delivery of free code while 60 percent of maintainers earn nothing, per Tidelift’s 2024 survey, and 136 developers wrote over 80 percent of the code in the fifty most-used packages. The voluntary approaches have been tried: GitHub Sponsors crossed $100 million in total payouts this July, Sentry’s Open Source Pledge, Sovereign Tech Fund, Flossbank, Ruby Together, and Feross’s terminal ads that got banned and turned into npm fund. His fix: registries meter corporate use the way Docker Hub already does, carve a fixed royalty off that revenue, and pay it pro rata to every package in paying customers’ dependency trees, monthly, with no application form. The thin spots are real, and he mostly concedes them. Gaming is inevitable. His answer to why GitHub, JFrog, or Sonatype would volunteer a revenue cut is developer pressure, and coordinated refusal aimed at a dozen vendors is easier to write than to organize. Still, if you pay for a private mirror, pull up that invoice and ask where the money lands.

what if my git host were a static site generator? (5 minute read)

Charlotte Som has run her own git forges since 2015, cycling from Gogs to Gitea to Forgejo, and she is done with Forgejo OOMing and filling disks on tiny personal boxes under ambient scraper traffic. Her answer is sorcery, a read-only git repo viewer shaped like a static site generator: every push rebuilds an overview page, the tree for each branch tip, and syntax-highlighted source files to disk, so serving is basically sendfile. History browsing runs through a small read-only git client in JavaScript that reads the served .git directory, backed by a QUERY route that returns requested objects as a binary bundle so browsers never chew through packfiles; she notes a naive packfile index fetch on linux.git to view one commit would burn over 400MiB. Writes happen only over SSH via sorcery-ssh as a ForceCommand, so the public surface is your sshd. No accounts, no key management, no issues, no PRs, and the frontend ships about 9 KB of gzipped JS on her own microframework. She also argues that Anubis-style proof-of-work gates are an admission that the app cannot survive real traffic, though her supporting claim that GitHub Pages ran on one machine for years is unsourced. One person’s project with no collaboration features, and history views need JavaScript. The lesson for anyone running read-heavy, write-sparse services is older than git: precompute the output instead of paying per request.

Guest Post: Rust Is Tier-1 Language at Microsoft (4 minute read)

Victor Ciura, a Principal Engineer on Microsoft’s Rust tooling team, uses this Rust Foundation guest post to make the case that Rust now sits alongside C++, C#, and TypeScript as a Tier-1 language inside Microsoft, which he defines as secure toolchain builds, developer tooling, quality workflows, deep platform integration, and SDL compliance. The centerpiece is rustc_codegen_utc, a rustc backend in the same family as rustc_codegen_llvm, rustc_codegen_gcc, and rustc_codegen_cranelift, except it targets the MSVC backend. The pitch is practical: Windows already has decades of binary hardening, Hotpatch servicing, SPGO, crash dump analysis, and coverage tooling built around MSVC, and plugging rustc into that beats reimplementing all of it. Ciura says the backend has been production-ready since early 2026, self-hosted since Rust 1.90, and builds more than 100 internal Microsoft repos with rollout continuing weekly. The thin part: the post never says whether or when rustc_codegen_utc ships outside Microsoft, there are no performance numbers, and Ciura admits the interop story beyond codegen (FFI contracts, bindings, build systems) is still unsolved, pointing at the Rust Foundation’s Interoperability Initiative for that. If you ship native code on Windows, the takeaway is that Rust and C++ are converging on one toolchain there, and hybrid projects are getting a real support story.

The lifecycle of a sharded Postgres query (24 minute read)

PlanetScale’s engineering team (no individual byline) traces one SELECT with a JOIN across a two-table, four-shard Neki database, and it is the most useful vendor post on sharded Postgres in a while. The router speaks the full Postgres wire protocol, including a SCRAM-SHA-256 auth exchange written entirely in Go and both the simple and extended query protocols, and parses SQL with roughly 18,000 lines of Go built to accept and reject syntax exactly the way Postgres does. Planning pulls schema from an authoritative shard and topology from etcd, then picks between nested loop and hash joins based on row counts; with 100,000 customers and 1,000,000 orders it builds a hash table from customers, eight shard queries instead of thousands of scatter-gather round trips, and spills to disk if memory runs out. Queries reach Postgres through a per-instance sidecar over long-lived gRPC streams, which pools connections and resets session state and role to prevent poisoning. The AVG rewrite into per-shard SUM and COUNT is a nice reminder why averaging averages gives you the wrong number ($46.25 instead of $40 in their example). The pitch is Neki, obviously, and it ends with a signup link. The honest part is the closing lesson: reshard orders by customer_id so the join happens inside Postgres, and the router never runs it. Pick shard keys with your joins in mind, or pay for it on every query forever.

Homebrew/BrewUI - AGPL-3.0 - Homebrew’s official macOS GUI. A native SwiftUI app from the Homebrew org itself, not a third-party wrapper, for discovering, installing, updating, and managing packages with the stated design goal of never hiding what brew is doing underneath. Install it as the homebrew-app cask. Requirements are aggressive: macOS 26 or newer, and the app launches Homebrew in a clean zsh with no rc files loaded, so shell aliases, exported variables, and a custom PATH are ignored, and any Homebrew configuration has to live in a brew.env file. Also worth clocking the AGPL-3.0 license, a departure from Homebrew’s BSD-2-Clause core.

localsend/localsend - Apache-2.0 - An open-source cross-platform alternative to AirDrop. Moves files and messages between nearby devices over the LAN with no internet and no third-party server, using a REST API over HTTPS with a TLS cert generated on the fly per device. Runs on Windows, macOS, Linux, Android, iOS, and Fire OS, is in basically every package manager that matters (Winget, Scoop, Chocolatey, Homebrew, Flathub, Nixpkgs, Snap, AUR, F-Droid, and the app stores), and ships a localsend-cli for terminal work. Two operational notes: it does not auto-update, so install from a store or package manager, and it needs TCP and UDP 53317 open on the receiver, so check the firewall and turn off AP isolation before blaming the software.

mulgadc/spinifex - AGPL-3.0 - Open source AWS-compatible platform for secure edge deployments. Run EC2, VPC, S3, and EBS-like services on bare metal with full control. Mulga Defense Corporation’s Go reimplementation of the AWS service surface (EC2, EBS, S3, VPC, IAM, ALB/NLB, EKS on k3s, ECR, ECS, RDS) for bare metal, on-prem, and disconnected edge sites, with unmodified AWS CLI, SDKs, and Terraform pointed at it. The gateway does SigV4 auth and publishes to a NATS bus for stateless daemons; no etcd and no Kubernetes in the control plane; compute is QEMU-driven over QMP. Wants Ubuntu 26.04 or Debian 13 and a Linux bridge. Release cadence is roughly weekly; v1.20.0 landed September 16, and the README says up front that Claude Code writes boilerplate, docs, and tests while architecture and security stay human. A commercial license sits alongside the AGPL.

cirrostratuscloud/kube53 - No license - Kubernetes on Route53: a hosted zone as the datastore, kubectl-compatible Lambda apiserver, and an ECS/ALB reconciler. A joke that works. Every object gets serialized to JSON, base64 encoded, split into 255-character chunks, and stored as a multi-string TXT record, with a 413 for anything over roughly 4,000 characters. API Gateway plus one Lambda speaks just enough of the Kubernetes REST API for real kubectl to create, get, and delete things. An EventBridge tick fires a Step Function that reconciles the records into ECS/Fargate tasks, ALBs, and EventBridge Scheduler jobs. Pods are synthetic and read-only, built from ecs:ListTasks. The README opens with “This is a joke,” which is accurate, but it provisions a real VPC, NAT gateway, ALB, and Fargate tasks, so run tofu destroy when the novelty wears off. Origin is a LinkedIn post by Yannick van Rooyen at Cirrostratus.cloud, and it has no license file, so all rights are reserved by default.

Want to reach 6,800+ developers, SREs, & DevOps leaders every week?

DevOps'ish delivers a 46% open rate and 10% click rate — well above industry averages — to a global audience of practitioners, VCs, and C-suite decision-makers. Your brand gets exclusive placement (one sponsor per issue, zero competition) in a text-based newsletter that bypasses ad blockers entirely. Get in front of the people who build and run the internet!