NOTE: Please read my Disclaimer before breaking out the tar and feathers.
What a series of unfortunate events for Docker in 2019. In what appeared to be a massive talent flush due to what looks like a potential earnings miss, the Great Docker Culling of 2019 happened. Docker appears to have laid off the vast majority of its well-known talent. Andrea Luzzardi, Sam Alba, and Gareth Rushgrove are among a slew of recent Docker layoffs discussed in this newsletter earlier this year. According to one source teams were, “killed,” and Docker, “missed their number, and by a lot.”
Fast forward to Friday night on the US east coast (like we weren’t going to notice?!?). Many people (myself included) received an e-mail from Docker about a Docker Hub breach impacting at least 190,000 accounts. According to the e-mail, “Data includes usernames and hashed passwords… as well as Github and Bitbucket tokens for Docker autobuilds.” Audit any Docker Hub tokens right now. Docker also, “revoked GitHub tokens and access keys. This means your autobuilds will fail.” Nothing like a page on a weekend because Docker broke your builds. Check your Docker Hub Linked Accounts and re-link them. You’ll then likely have to do a weird do-si-do in the Build config of one of your image pages to get everything working as is.
This Docker Hub breach is a significant breach. If any of the tokens of any of the common base images had been compromised — packages like Alpine, busybox, Node.JS, or any of the major databases — these could have easily permeated into the wild with little or no knowledge. From my point of view, the only way to be sure you’re not affected is to somehow verify with the image provider that their account has been cleaned up and redeploy all your containers. Cleaned up as directed by Docker (note the URL, “success”). Why? Because it’s likely some upstream used Docker Hub even if you didn’t. In other words, “Nuke the entire site from orbit. It’s the only way to be sure.” Yes, it’s that bad until it’s confirmed otherwise.
What a Docker freaking mess we’re in. At the risk of being extra petty, I can’t help but mention I started using Quay when I joined Red Hat and I’m pretty happy with it. It’s a container registry, not an Alexander Wang piece. Quay is not perfect but, I’m not expecting a whole lot here. It looks like I’ll be moving more images off Docker Hub in the future.
Hit send on your last tech job application
One application, many tech opportunities. Indeed Prime makes job hunting quick and easy. Save time and let us do the heavy lifting for you by matching you with top tech companies. Join for free today! SPONSORED
Log Management Modernized
With LogDNA’s fast, multi-cloud logging platform, DevOps and Engineering teams can easily and quickly aggregate all system and application logs into one efficient platform.
Whether on-premise, in the cloud, or a hybrid solution, we have you covered. Don’t take our word for it. Try it yourself.
DevOps’ish Top Five from Last Week
- How much does a DevOps engineer make?
- What Happened When The DEA Demanded Passwords From LastPass
- The Problem with SSH Agent Forwarding
- The 5 communication problems that destroy company morale
- Alikhll/golang-developer-roadmap: Roadmap to becoming a Go developer in 2019
Google Walkout Organizers Say They’re Facing Retaliation — ”Two employee activists at Google say they have been retaliated against for helping to organize a walkout among thousands of Google workers in November.” I’m pretty sure this is illegal, no? The dispute intensified at a town hall on Friday. #NotOkGoogle
Hire People or Optimize Processes: A cost-benefit analysis for engineering leaders — Help to figure out if you should hire more people or optimize your widget making.
Kubernetes jobs hunt: How to land that role — Trying to get a job working with Kubernetes? Consider these five tips.
The Difference Between Goals, Strategies, Metrics, OKRs, KPIs, and KRIs — The differences and similarities between the most common types of business measurement systems
DNS over HTTPS is coming whether ISPs and governments like it or not — Encrypting your DNS queries in the payload of an HTTPS packet means that countries and companies can’t as easily hijack DNS to control internet access or to monitor employee activity. Conversely, blocking known bad DNS entries (sinkhole) and using DNS query logs to hunt for indicators of compromise are common security measures. This becomes a MUCH harder problem.
DevOps’ish Telegram — Join the over 230+ DevOps, Kubernetes, SRE, and other technology professionals discussing real-world problems and solutions to modern-day issues.
Accelerate: State of DevOps 2019 Survey: Nicole Forsgren, PhD is conducting the State of DevOps 2019 Survey. Your input is incredibly important. On several occasions, I have referenced the 2018 report since its release for real-world work that impacts real numbers. Nicole’s group also wrote, Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations, which I cannot recommend enough either.
Note: DevOps’ish may earn compensation for sales from links on this post through affiliate programs.
How You Can Help Localize Kubernetes Docs — If you’re interested in helping an existing Kubernetes localization please help us out!
7 advantages of open source for agile teams — “Adopting DevOps and agile practices is how companies are managing the velocity of change. I imagine it’s possible to do agile or DevOps in a closed source world. The question then becomes, why put that hurdle there?”
Open Source Software: The Complete Wired Guide — Some knowledge about open source software that’s easily consumable by the unknowing.
Forge Your Future with Open Source: Build Your Skills. Build Your Network. Build the Future of Technology. by VM (Vicky) Brasseur — And if you want to get going in Open Source, I highly recommend Vicky’s book.
Slack files to go public, reports $138.9M in losses on revenue of $400.6M — And the tech IPOs just keep on coming.
Some internet outages predicted for the coming month as ‘768k Day’ approaches — Some routers will crash soon due to ternary content-addressable memory needs of over 768K. BGP and DNS are the bubble gum and duct tape of the internet.
Apple spends more than $30 million a month on Amazon Web Services — People freaked out when they heard this number. Apple uses a lot of different cloud providers for various reasons (I used to work for one). But, I think the bigger story is, Apple Slashed Amazon Cloud Spending 50 Percent in Bid for Self-Sufficiency
A Roadmap to Convergence – OpenTracing — “We are creating a new, unified set of libraries and specifications for observability telemetry. It will merge the OpenTracing and OpenCensus projects, and provide a supported migration path.”
The DevOps Institute Has Been Brandjacked This is some pretty gross behavior by QuickStart but, trademark and copyright law exist for a reason. I certainly hope QuickStart Learning Inc. finds out about the law very soon.
Accenture sued over website redesign so bad it Hertz: Car hire biz demands $32m+ for ‘defective’ cyber-revamp — This is so bad y’all. This Twitter thread is also gold.
DevOps’ish with Chris Short – Newsletterers – The Tim Show – S02E01 — Ever wonder how the DevOps’ish newsletter started, how I build it, or what it’s like to write a newsletter? Check out fellow Red Hatter Tim Hildred’s podcast on that very topic!
Running Drupal in Kubernetes with Docker in production — ”You really have to be on your game in the world of containerized-Drupal-in-production!”
Packets-per-second limits in EC2 — ”[W]e determined that each EC2 instance type has a packet-per-second budget. Surprisingly, this budget goes toward the total of incoming and outgoing packets. Even more surprisingly, the same budget gets split between multiple network interfaces, with some additional performance penalty.”
How to run systemd in a container — Trying to run systemd in a container is hard af. I’ve had to do it to test deployments before and I have to say, Dan’s method is probably better than any I’ve ever seen. I want to change the Podman name so bad though, Man.
Kubernetes Tutorial - Step by Step Guide to Basic Kubernetes Concepts — This is a nice introduction to k8s from Auth0. Good for them for getting a little out of their comfort zone to help lower the barrier to entry a little.
Kubernetes Network Policy APIs — “This post explores multiple ways network policy can be expressed in Kubernetes.” In other words, way more than I want to know about Kubernetes networking. But, good to have as a reference.
Improving the security of Kubernetes clusters using Istio — Istio does SO MUCH.
Istio the Easy Way — Oh good there’s an easy way! Thank you, Christian Posta and solo.io for this.
Python Project Tooling explained — Instant bookmark. Please share with others that are new or even a little old to Python.
All That You Need to Know About Microsoft’s New Programming Language: Bosque — ”The Bosque programming language is a Microsoft Research project that is investigating language designs for writing code that is simple, obvious, and easy to reason about for both humans and machines.”
Open Sourcing Jingo, a Faster JSON Encoder for Go — This package provides the ability to encode golang structs to a buffer as JSON very quickly.
Building platform stacks with in-house scripts vs. Kubernetes Operators — STOP HAND ROLLING SCRIPTS! YOUR ARTISANAL SCRIPTS DON’T BELONG! GET OFF MY LAWN!
ricardbejarano/haproxy — 🏎 Built-from-source container image of the HAProxy load balancer
cdr/sshcode — Run VS Code on any server over SSH
bxcodec/gotcha — gotcha: inmemory-cache in Go (Golang) with customizable algorithm
mhausenblas/kboom — The Kubernetes scale & soak load tester (do you want to shred some clusters with me?!?)
GoogleCloudPlatform/berglas — A tool for managing secrets on Google Cloud (I hope Google does End of Life this before the newsletter goes live)
DevOps’ish Tweet of the Week
On the Docker breach: Even if your company doesn't rely on Docker Hub for production, if a developer in your org enabled auto builds and linked to GitHub via oauth for a personal project, when that oauth token is compromised, _all_ repos on GH they had access to are vulnerable.— Kenn @ Enigma2020 (@kennwhite) April 27, 2019
Notes from this week’s issue can be found here.
Sponsor DevOps'ish and put your brand in front of thousands of highly skilled operators, maintainers, developers, and leaders from Amazon, Apple, Google, IBM, Intel, Microsoft, Red Hat, many of the Fortune 100, and beyond. Download the DevOps'ish Sponsorship Prospectus now!
Join the Conversation
Join the DevOps'ish group on Telegram for insight and in-depth discussions about real technical challenges facing real people. Also, join //devopsish for a stream of news and content throughout the week.