DevOps'ish

DevOps, Cloud Native, Open Source, and the ‘ish in between.

126: Ransom Attacks Hit Git, Managing Many Clusters, Kubernetes Failure Stories, Ansible-based Kubernetes Operators, Deadlines are Horrible, & More

Friday afternoon (US east coast time), some oddities around git repos being taken for ransom started to pop-up. Safe to say, I’m paranoid af when it comes to the software delivery pipeline (GitHub and GitLab are both used to manage DevOps’ish). I immediately changed passwords everywhere I have code stored (and you should too if you haven’t already). GitLab chimed in on the issue to confirm it did not appear to be a total loss of data on affected repos and provided potential fixes. GitLab also provided extensive incident details I recommend checking out. The cause? The age-old problem of exposing version control dot directory when deploying a web site. Don’t publicly expose .git or how we downloaded your website’s source code.

Find your next tech job
No more inbox spam from recruiters! Indeed Prime matches you with top tech companies and only sends tech job opportunities that match your career goals, technical skill set, location, and salary preferences. Join for free today! SPONSORED

Bay Area DevOps meetup, May 9 in Mountain View
Cloud Foundry, IBM, LogDNA share how open ecosystems, interoperability & multi-cloud are here to stay.
RSVP now to save your seat.
Try LogDNA - Start free trial SPONSORED

This newsletter won't run without sponsors. Sponsor DevOps'ish and connect with thousands of motivated DevOps, cloud native, and open source professionals from across the globe.

DevOps’ish Last Week’s Top Five

  1. Tweet: Hertz has sued Accenture for failing to deliver on a $32 million “digital transformation” project.
  2. Python Project Tooling explained
  3. The Difference Between Goals, Strategies, Metrics, OKRs, KPIs, and KRIs
  4. Istio the Easy Way (cough nomedium.dev cough)
  5. How to run systemd in a container

People

Deadlines are horrible — I have stopped apologizing for being “behind” on Kubernetes community work. I wish I could do more and commit to more things but, it’s not my full-time job. Open source work is largely volunteer-based and we shouldn’t be this stressed helping folks out. Right?

Product Marketing Manager — Ansible Automation — There are two or three open reqs on my team right now. If you’re interested let me know (if I don’t know you personally, send your LinkedIn profile too). I’ll send you a unique URL to apply if I think you might be a good fit. Not a good fit but still need a job? Indeed Prime is a sponsor this week.

Brand loyalty: Red Hat employees permanently inked with new company logo — Red Hat revealed its new logo this week. Personally, I really like it. Red Hatters have gotten the company logo tattooed on themselves for years. It’s not a cult; it’s called a great place to work. I’m sorry if folks haven’t ever had that feeling.

Call for AWS Birds-of-Feather at KubeCon Barcelona — Going to KubeCon Barcelona and use AWS? That’s probably a hard, yes. There’s a BoF forming and if you’re interested, speak up.

What do companies expect from Python devs in 2019?

Sorting algorithm reference, for coding interviews and computer science classes — I’m approaching forty years old and learn new things every day. I’ll be referencing this more and more as we all move towards a world full of data and code.

Upskill Your Team To Address The Cloud, Kubernetes Skills Gap

You should have a personal web site — Yes, you must have a personal web site. chrisshort.net is massive and has served me very well over the years. I modeled it a little bit after Bridget Kromhout’s site which is also great.

Gail Duval Talks Mandrake Linux and /e/ Telephone — I loved Mandrake back in the day.

DevOps’ish Telegram — Join the over 230+ DevOps, Kubernetes, SRE, and other technology professionals discussing real-world problems, breaking technology events and outages, and the occasional Spotify playlist.

Process

Accelerate: State of DevOps 2019 Survey: Nicole Forsgren, PhD is conducting the State of DevOps 2019 Survey. Your input is incredibly important. On several occasions, I have referenced the 2018 report since its release for real-world work that impacts real numbers. Nicole’s group also wrote, Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations, which I cannot recommend enough either.

A Week Later, Docker Offers Scant Details on Hub Attack — Docker did conduct a very long maintenance this week. But, it hasn’t offered much in terms of additional details regarding the Docker Hub Breach last week. There will likely be more to this story in the future.

Compilation of public failure/horror stories related to Kubernetes

80% of developers are not addressing Docker security — Super awesome to read after the Docker Hub Breach.

Canonical Fires Shots at Red Hat with Ubuntu Advantage Launch — ”Direct contrast to the complexity and cost of offerings from Red Hat and VMware” There is a lot to be said for a simplified billing model.

How companies adopt and apply cloud native infrastructure — Survey results reveal the path organizations face as they integrate cloud native infrastructure and harness the full power of the cloud.

Kubernetes Deployment Strategies

What a pain in the Azzz-ure: Microsoft Azure, SharePoint, etc knocked offline by DNS blunder — I appreciate the difficult nature of DNS. But, outages like this are going to become less and less tolerated as more reliability is baked into infrastructure.

Brand New: New Logo for Red Hat — ”The new logo is quite literal: it’s a red hat, but it’s red AF and hat AF.”

Tools

Many Kubernetes Clusters — ”Zalando runs 100+ Kubernetes clusters on AWS. Each cluster runs in its own AWS account. We always create a pair of prod/non-prod clusters per ‘product community’, i.e. only half of our clusters (50+) are marked as “production” and have full 24x7 on-call support.” The reasoning behind Zalando’s Kubernetes deployment methods as detailed in the article are great. A must read.

Sherlock changelog — Like all those awesome colors in ls? They come at a price thanks to LS_COLORS. I’d wish I’d known this tweak years ago.

Memory Limit of POD and OOM Killer — ”Kubernetes manages the Pod memory limit with cgroup and OOM killer. We need to be careful to separate the OS OOM and the pods OOM.”

I forgot how to manage a server — ”My config management does this for me. Whether it’s Puppet, Ansible, Chef, … all of the boring parts of being a sysadmin have been hidden behind management tools. Yet here I am, trying to quickly configure a personal server, without my company-managed config management to aid me.” In a world of automation, the manual becomes foreign.

Building Operators with Ansible — I have been working hard with my co-workers and OpenShift team to make sure Ansible-based Operators for Kubernetes are a known solution. Want to run Day 2 operations inside your Kubernetes cluster? You might already have what you need in the form of an Ansible Playbook or Role. Operators are pretty dope technology. Making them with Ansible is something I’ll be talking about in two mini-theater sessions at Red Hat Summit this week.

Introducing Red Hat Quay 3 - A Registry for your Linux and Windows Containers — Normally, I would not feature a product announcement from Red Hat unless it was something I worked on. But, as I mentioned last week, I really like Quay. It is now capable of multiarch builds which is very handy as well as a few other nice features from a container registry.

Grafana Dashboards for Kubernetes Administrators — Minimum Viable Dashboards for Kubernetes via Grafana

Why Script Based Deployments to Kubernetes Don’t Scale — ”Why use Spinnaker when I can just keep doing the same thing I’ve done before?”

Synthetic Kubernetes cluster monitoring with Kuberhealthy — ”By enabling some simple synthetic checking, we stand a much better chance of catching these kinds of ephemeral and limited-scope disturbances in our infrastructure before customers or developers notice.”

A Guided Kubernetes Meditation — Take a deep breath and CrashLoopBackOff, y’all. Namaste, bitches.

Kubernetes Ingress Tutorial: Beginners Series — ”You will learn the concept of ingress resource and ingress controllers used for routing external traffic to Kubernetes deployments.”

Cilium 1.5: Scaling to 5k nodes and 100k pods, BPF-based SNAT, and Rolling Key Updates for Transparent Encryption — Cilium 1.5 now officially supports an eye-watering, “5k nodes, 100k pods and 20k services.”

gopls — gopls (pronounced: “go please”) is an implementation of the Language Server Protocol (LSP) server for Go. The LSP allows any text editor to be extended with IDE-like features.

derailed/k9s: 🐶 Kubernetes CLI To Manage Your Clusters In Style!

infracloudio/botkube: App that helps you monitor your Kubernetes cluster, debug critical deployments & gives recommendations for standard practices

se7entyse7en/pydockenv: Python virtual environment, but backed by Docker!

bloomreach/s4cmd: Super S3 command line tool

docker/buildx: Docker CLI plugin for extended build capabilities with BuildKit

micronaut-projects/micronaut-core: Micronaut Application Framework

DevOps’ish Tweet of the Week

This thread from Justin Garrison is everything.


DevOps’ish is a weekly newsletter assembled by open source contributor, DevOps veteran, and CNCF Ambassador Chris Short. Join the 3,002 member community and subscribe now!


whoami

I'm Chris Short, 20+ veteran of the IT industry and 11 year veteran of the US Air Force. I help people and companies embrace DevOps practices and tools through writing and public speaking. I am a staunch advocate for transparency and open source solutions to problems.

Join the DevOps'ish group on Telegram for insight and in-depth discussions about real technical challenges facing real people. Follow me on Twitter and LinkedIn.


Share