Nothing worthwhile in life is ever easy. Take the tale of Docker, Kubernetes, and the rest of the container ecosystem that is making life easier for DevOps and admins working in hybrid cloud environments. In less than half a decade, this technology has gone from being relatively unknown to being an essential ingredient to a successful commercial software stack.
To someone looking at this rise and success from a distance, it looks as if it's been nothing but smooth sailing, filled with open source kumbaya moments as all of the large corporations with skin-in-the-game worked in full cooperation.
Phil Estes painted a different picture of container history at Open Source 101 in Raleigh last weekend, speaking from the perspective of someone who had a front row seat. To hear him tell it, this rise and success is a story filled with intrigue, and enough drama to keep a daytime soap opera going for a season or two.
[Container World delivers real-world case studies from the cloud-native ecosystem, hands-on technical education, the best speakers and cutting-edge startups under one roof. Get your ticket.]
Estes has been at IBM since 1998, currently as a senior technical staff member working in the office of the CTO. He's also joined at the hip with Docker, Kubernetes, the Cloud Native Computing Foundation and all things containers. He got there through his job at Big Blue.
"In my role at IBM, about four years ago I moved from our Linux technology center to our cloud division, so I have been working with Linux for about a decade," he said. "I joined the cloud division and wanted to continue working open source. We have people in Cloud Foundry and OpenStack, but that summer, in 2014, my manager said, 'This Docker thing sounds cool. A lot of people are excited about it. We should go figure this out. We should send some people to understand this community and get involved.'"
Estes connected with the Docker community by plugging around on IRC, Slack, and GitHub. His first pull request was merged a month or so later and he's had code in every Docker release since. The next year he became a maintainer, which meant he could commit code to the repository.
"IBM was very interested in helping Docker make sure that the project itself was under open governance," he said, "that many different vendors could get involved and it wasn't fully driven by Docker. A lot of our story this morning will be about the pitfalls and the ups and downs of that part of the container story."
By the time Estes was formally introduced to the Docker community in 2014, the container rush had already been going full-speed-ahead for over a year, so when he signed onto the the Docker IRC channel, people from the likes of Red Hat, Google, and CoreOS were already on board. Even Microsoft was there, having joined about the same time as Estes, to start porting the Docker engine to Windows.
It wasn't just big vendors that were involved, either.
"Independents, students, lots of people were all involved in the Docker project," Estes remembered. "What that generated was basically an open source fire hose. Docker was such a popular project that we were getting 130-150 pull requests submitted a week."
At one point the project had over 25 maintainers, with about two-thirds working directly for Docker. About that time, the number of contributors on Docker's GitHub repository passed the 1,000 mark. Not all of them were active; the list included anyone who had ever submitted a pull request.
"One of the problems was that it became very difficult to process nontrivial pull requests," he said. "If you think about it, Red Hat's involved, Microsoft is involved, Google, IBM, and many others. Every one of them had an idea about how Docker should work."
According to Estes, a large part of the problem with pull requests revolved around Docker and its CTO Solomon Hykes, whom he compared to Steve Jobs, who "doesn't take your input on how that Mac operates, how it looks."
"Solomon had similar ideas that Docker should feel and operate in a certain way," he said. "This led to a lot of difficult discussions and very long periods of back and forth on some pull requests."
Often pull requests sat on the shelf for six months to a year waiting to be processed. Sometimes the wait was as long as two years. This obviously wasn't good for morale.
"One of the hardest things for a contributor is to have your pull request rejected," Estes explained. "In an open source community, one of the difficult things is to determine when have we, the maintainers, decided that this isn't acceptable for our repository, and to do that in a way that's gracious to the contributor and the time they've spent.
"One of the things many of us felt we learned is that we actually were waiting way too long to close pull requests. There were a lot of disgruntled submitters over time, especially if they spent six months kind of hoping that this gets in and then you say, 'No, actually it can't get in.'"
Docker's actions were also causing other frictions among developers. Although nearly a third of the maintainers weren't associated with Docker, many high level decisions regarding issues such as design and UX were being made from inside the company. This led to complaints such as, "Docker controls this project. It's not really open source." Or: "It's open source for fixing bugs, but everything else is behind the curtain."
"Open source software plus a product, that's hard," Estes said. "That's hard for anybody; it was extremely hard for Docker. People were, at this point, openly asking, 'Is Docker really open? Is it really open source?' Obviously Docker started to have some detractors as well."
It wasn't just Docker, with it's $1 billion valuation, that was feeding the discontent. Red Hat, Google, CoreOS and others all had a hand in stirring the stew, and each had its own idea of how that stew should taste when done. Add to that the fact that at about that time, Kubernetes was arriving on the scene.
"Obviously, a lot of people wanted to have a hand in the control of this main developer tool within what seemed to be a multi-billion dollar industry. You can't ignore that that added to the tensions."
It was in this environment -- perhaps taking advantage of it -- that CoreOS made a grab for its own slice of the container market pie that was almost exclusively owned by Docker. In December 2014, as the first Docker conference to feature a large number of vendors was underway in Amsterdam, CoreOS announced it's own competing container engine, rkt, a move with the potential to fragment the container market.
"Everybody perceived the timing was a little bit of a jab during the middle of their big conference," Estes observed.
Red Hat was also stirring up a bit of discontent by adding it's own spices -- and potential for fragmentation -- to the stew.
"Basically Red Hat said, 'Here's a bunch of pull requests we've worked on for a long time upstream in the Docker repository that haven't been accepted. We think they're important, so when you install Red Hat's Docker engine, you get these patches as well.'"
In other words, Red Hat was basically pushing out some of its own code under the Docker brand. The company went so far as to publish a list with descriptions of the patches on its Project Atomic website, which are still available today. Docker pushed back and accused Red Hat of forking the Docker engine with its actions.
"That kind of talk heated up to where at a Red Hat Summit that year, Docker had a booth and they were giving out T-shirts. 'Accept no imitations' was printed on the shirt."
This eventually led to the great Docker fork that never happened, when in August 2016 The New Stack published an article, "A Docker Fork: Talk of a Split Is Now on the Table"
"In the article it talks about maybe there was a secret meeting in New York between a bunch of vendors and they're going to use the Cloud Native Computing Foundation to create this variant of Docker," Estes explained. "Obviously, it generated tons of activity on Twitter, tons of blog posts."
A secret meeting in a smoke filled room is sort of a "say it ain't so" situation in open circles.
"Some good things came out of that," Estes said. "The important thing is that people got attracted to this word 'boring.' 'Hey, we just need a boring container runtime. Docker is running too fast.' They created this swarm mode that's kind of like an alternate to Kubernetes. They did that without a lot of involvement from the community."
Even Craig McLuckie, one of Kubernetes' original developers, got in on the "boring" discussion, tweeting, "Docker must be allowed to innovate and profit (they have legitimately done amazing work), but we need boring core infrastructure today."
One of first lights signaling calmer seas for container traffic came after CoreOS's introduction of rkt during DockerCon in Amsterdam. The following June, at the next DockerCon in San Francisco, a consortium consisting of the Linux Foundation, Google, Docker, CoreOS, Red Hat, IBM and others announced the formation of the Open Container Initiative to relieve tensions and establish standards so that different companies' products work and play well with others.
As for forking Docker, cooler heads prevailed. According to Estes, this led to "several levelheaded people very well known in the industry saying, 'This is not time to fork Docker. This is not time to talk about taking control. Let's think through how we can use the open source foundations and the players that are already in place to see how we can solve some of these tensions.'"
In other words, the Cloud Native Computing Foundation ended up not being used as a device to fragment containers, but as a unifying force.
These days, the waters around the docks are relatively calm. All of the various players in the container game have learned that they basically all share the same goals. Hopefully everyone paid attention and container history won't repeat itself.
Phil Estes will be on a keynote panel at this year's Container World at the Santa Clara Convention Center in Santa Clara, California. The discussion, "Future View--What Will the Container Landscape Look Like in 5 Years," will take place on Wednesday, February 28 at 9 a.m.