Last week, I enabled a workflow to automate more of the DevOps’ish process. One thing I did not test as a part of this workflow was link tracking. To be honest, I did not think it would matter. That assumption bit me hard. Within minutes, reports from readers came in saying Gmail was flagging the links in the e-mail as suspicious.

As it turns out, disabling the link tracking also bypassed a bug in Mailchimp’s platform. When the automation kicked off with link tracking enabled it triggered the bug in the Mailchimp platform. My apologies for ever making you question the validity of the e-mails for this newsletter. The lesson I’ve re-learned here is never to trust your assumptions. Trust but verify all features.

Mailchimp is working on the bug and gave me a very viable workaround. E-mail is hard, y’all. I’ve managed email delivery at a few companies now. If you think it’s easy, try doing it en masse, at scale. It’s a constant game of give and take that you win and lose at often. Mailchimp is a real delight to work with and I appreciate their support.

I’m hopeful this week will be different. If you see any issues, please let me know. Thanks for reading!

“Care about what other people think and you will always be their prisoner.” —Lao Tzu

Use Lead Time Metric to Improve Your CI/CD Process
Check out GoCD’s latest blog in CD Metrics series. It talks about what lead time means in CD context and guides you on how to identify bottlenecks and improve your CI/CD process. SPONSORED

Triangle DevOps presents DevOps is Not War with Chris Short of Red Hat
Over the past 500 years, there have been 16 cases of a rising power threatening to displace a ruling power. 75% of those cases resulted in war. Although your organizational transformation probably won’t lead to war, it could be contentious. History can help prevent conflict when driving change. This talk will analyze human tendencies, historical data, and provide real-world examples of how to avoid friction during your DevOps journey. SPONSORED

DevOps’ish Top Five from Last Week

  1. 7 Habits Of Highly Effective CI/CD Pipelines
  2. 12 ways to get smarter about Kubernetes
  3. 20 VS Code Extensions You’ll Actually Use
  4. We Don’t Do That Here
  5. High prevalence of poor mental health among tech (specifically Ops) employees?


Former engineer Liz Fong-Jones has ‘grave concerns’ about Google — I have a helluva lot of respect for Liz Fong-Jones. Standing up for what you believe in and walking the talk at great personal risk is heroic. We truly don’t deserve her.

The Secret History of Women in Coding — Computer programming once had much better gender balance than it does today. What went wrong?

You Guys

U-M Flint professor lodges Title IX complaint against WSU — “[Mark Perry, an economics professor at U-M Flint, submitted a Title IX complaint against WSU because the university hosted a summer workshop for Black Girls CODE… Perry says WSU was acting as a venue sponsor for Black Girls CODE from July 30, 2018 to Aug. 10, 2018. Because the Black Girls CODE program only allowed girls ages 13 to 17 to participate, it was discriminatory to boys and therefore a breach of Title IX.” Mark Perry’s pettiness knows no bounds.

Joe Beda on Kubernetes & the CNCF — “Kubernetes was always viewed by the creators as something to be build on. It was never really viewed as the end goal.”

Venture capital is still very much a boys’ club — “Only 9.65% of decision-makers at U.S. venture capital firms are women. The breakdown was 105 female decision-makers out of an industry total of 1,088.”

A record 7 million Americans are 3 months behind on their car payments, a red flag for the economy — A horrifying economic indicator.

Salary Negotiation Tips from White Men in Tech: Part 1 – Career Advice for Women in Security


CVE-2019-5736: runc container breakout (all versions) — Another nasty vulnerability impacting virtually the entire container ecosystem. This vulnerability was interesting in the sense that it was clearly embargoed but, not every Kubernetes vendor got their systems patched by the deadline. As the day wore on, it became apparent IBM and Microsoft had missed the boat on patching this vulnerability. Open source security is a difficult world. Be open and collaborative but, be secretive and limit communications regarding vulnerabilities. Things break, communications systems break, and all we can do is try to improve things for next time. Other runc vulnerability articles from this week:

Detecting exploits of CVE-2019-5736: runc container breakout — Sysdig has put together a guide for detecting the runc exploit.

Stop Disabling SELinux — “Every time you run setenforce 0, you make Dan Walsh weep.”

Diversity Is Vital to Advance Security — “Diversity isn’t just different appearances or labels. It’s beyond that. It’s about diversity of thought, the differences in our problem-solving processes and perspectives — and it is a critical component of true innovation.” I absolutely could not agree more.

Commit to your lock-in — Embracing that little bit of lock-in can enable a lot more engineering time to other things. Data gravity is a thing; embrace it and move forward as it suits your needs.

Automation - Just do it! — Listed as a benefit, “Personal development, over time you will gain confidence and be able to contribute some valuable automation”

GNU Health Federation Information System moves from MongoDB to PostgreSQL — MongoDB’s demise is in progress. Their stance against open source is not something the industry can embrace.

Red Hat Satellite to standardize on PostgreSQL backend

USN-3887-1: snapd vulnerability — Eeek! “Chris Moberly discovered that snapd versions 2.28 through 2.37 incorrectly validated and parsed the remote socket address when performing access controls on its UNIX socket. A local attacker could use this to access privileged socket APIs and obtain administrator privileges.”

The CNCF 2018 annual report

Fun fact: GPS uses 10 bits to store the week. That means it runs out… oh heck – April 6, 2019 — If you’re running older GPS gear it might be time to replace it.

What’s the future of Linux distributions? — I have so many thoughts here. SO MANY…

Ansible Community Update — February 2019 — Lots of good stuff in the works in the Ansible community.


Running your own DBaaS based on your preferred DBs, Kubernetes operators and containerized storage — “This blog was intended to outline some of the things to think about when running your own DBaaS. Overall I recommend doing so — however I also suggest you think long and hard about operations automation and, of course, underlying data resilience.” —Evan Powell, CEO of MayaData (Editor’s Note: Such a good piece)

chris-short/ — The DevOps needs a refresh. It’s a work in progress for me but pull requests are VERY welcomed.

The Rise of Bare-Metal Kubernetes Servers — If you recall, I said in my annual update that, “Kubernetes Will Start to Replace The Hypervisor”. It’s already happening.

Kubernetes Policies – PodCTL — Brian and new co-host John Osborne (@OpenShiftFed) discuss policies in and around Kubernetes.

Running Static Pods in Kubernetes — Static pods are configured to be start at Kubelet daemon or whenever Kubelet daemon reloads itself. All Master components run as Static pod and configured to be run at start/reload of Kubelet daemon.

Kubernetes Podcast from Google — Interview with author and maintainer of minikube, Dan Lorenc

What’s the right amount of swap space for a modern Linux system? — Complete the survey and voice your opinion on how much swap space to allocate.

Building Small Containers (Kubernetes Best Practices) — “The first step in deploying to Kubernetes is putting your app inside a container. let’s explore how you can create a small and secure container images”

The 10 Kubernetes Commandments — “Bryan Liles and Carlos Amedee explore topics from booting Kubernetes clusters to running complex workloads as a list of 10 items.”

How to explain Kubernetes Operators in plain English — What are Kubernetes Operators, and why are they so valuable to organizations working with containers? Here’s a primer for IT leaders – and anyone needing to demystify the concept.

Setting up a Kubernetes cluster with Kubespray — Kubespray has been mentioned in DevOps’ish before (088, 094). It’s a composable way to install Kubernetes on damn near anything x86 based using Ansible. It’s good stuff.

Inspector Murphy, The Kubernetes quality enforcer

Single Sign-On for Kubernetes: An Introduction — “One of the great things about Kubernetes is that it completely separates authentication and authorization. Authentication (Authn) meaning the act of identifying who the user is and authorization (Authz) meaning the act of working out if they’re allowed to perform some action.”

Kubernetes as an API standard — Justin Cormack is on to something here.

Pimp my Kubernetes Shell — Making a more usable Kubernetes shell is within reach.

Cluster-level Logging in Kubernetes with Fluentd

How to build a Serverless Single Page App — OpenFaaS is good stuff.

How does rootless Podman work?

Fathom: An Open Source Google Analytics Alternative — As of this issue, I’m starting a long process of moving away from embedding Google products into this web property. Google AdSense has been removed. Google Analytics is next and Fathom will likely be my replacement. Google Fonts is going to be the hard one.

Networking tool comics! — Admit it; you might not be great at networking. This should help.

Managing EC2 servers at scale: Ansible and RunCommand — Ansible vs SSM RunCommand: which one to use to manage a fleet of servers

bincyber/vigilant — A security controller for Kubernetes

shawnxlw/infra-dev-env — A docker image that contains the necessary tools for doing Infrastructure Development

A handy flowchart for when to use blockchain

DevOps’ish Tweet of the Week

Notes from this week’s issue can be found on GitHub.