“We want to modernize our infrastructure in such a way that we can perform daily code commits into our production environments”. If this is your strategy for this year or the next, then this is the blog for you. This is the next blog in the series around VMware Cloud Foundation, where we will discuss what vSphere with Kubernetes is and how you can get started with it. Everyone likes to sit in the conference presentations delivered by organizations like Netflix, Uber, etc. where they talk about how agile their development teams are and how they can perform tens to a thousand commits a day to their production environment. Although that sounds interesting, you really need to revamp your development, operations, and security teams to enable that, which might not be the right choice given your IT organization structure. But, we can certainly look at modernizing our infrastructure such that we deliver on two key things, allow developers the flexibility to write their apps in the language of their choice (using containers) without having to worry about environment parity across Test/Dev, UAT, Pre-Prod and Prod, and second, eliminate shadow IT by providing the development teams the infrastructure they deserve inside your datacenter or at least something that is controlled by the central IT department rather than having a cloud sprawl which leads to not only increased cost, but also many other security and management challenges.
This is where vSphere with Kubernetes comes in. VMware has always allowed organizations to run containers starting from vSphere Integrated Containers to VMware PKS (Essentials, Enterprise, or Cloud). But, with vSphere 7, VMware completely modernized the vSphere hypervisor (Project Pacific) allowing customers to not only run containers alongside their VMs directly on the ESXi host but also allowed customers to deploy managed Kubernetes (K8s) clusters on top of their vSphere clusters. And you know VMware is serious around K8s when you see the integrations that they deliver not only at the compute layer but also storage integrations using Cloud Native Storage, network integrations using NSX-T and NSX-T Container Plugin, Automated Deployment, and Integrations with Harbor Image Registry, Monitoring and logging integrations using the vRealize Suite of products. It is truly an enterprise-grade offering that allows you to run containers and K8s inside your datacenter environments. But, being enterprise-grade also comes with its own set of challenges. It means that someone has to design and deploy this architecture and then manage it moving forward. In this blog, we will look at three approaches for getting started with vSphere with Kubernetes:
- Virtually Ghetto’s Nested Setup: This is by far the easiest way to get started and get your feet wet with vSphere with Kubernetes. All you need is an existing vSphere environment (vSAN or VMFS), a VMware account to download the required binaries (vSphere, vCenter, NSX-T Manager, and NSX-T Edge VMs), licenses for NSX-T and vSphere with Kubernetes Add-On. Once you have all the prereqs, all you have to do is download the script from William’s GitHub page. Update the variables in the first 150 lines and that’s it. Wait for about an hour, and you will have a nested 3-node ESXi cluster running vSAN, a vCenter 7 instance to manage the cluster and an NSX-T environment with an Edge Cluster that you can use to enable the Workload Management Feature (Another 30 mins I guess), and you can start deploying vSphere pods on your nested ESXi setup.
- Manual Deployment with vSphere: This was not on my list when I was looking to deploy vSphere with Kubernetes. In a couple of webinars that I attended around vSphere 7, it looked like vSphere with Kubernetes is only supported with VMware Cloud Foundation. But, looking at William’s powercli script, it looked like VMware Cloud Foundation was not required after all, so I tried to find some documentation and got my hands on this vSphere with Kubernetes Configuration and Management Guide. Based on this guide, you can manually configure your vSphere, vSAN, and NSX-T environment and get vSphere with Kubernetes on top of your existing cluster. So that was what I did last weekend, but unfortunately, even after following all the steps in the guide, I couldn’t enable the Workload Management feature in my cluster because it couldn’t find any compatible clusters.
Here is a summary of all the steps you need to enable this feature:
- Compute Configuration:
- Create a vSphere Cluster
- Enable vSphere HA and DRS (Fully Automated)
- Storage Configuration:
- Configure and enable vSAN
- Create Storage Policies for placement of Kubernetes control plane VMs, container image cache, and container ephemeral disks (I choose to use a single policy for all three requirements)
- NSX-T Configuration:
- Install and configure NSX-T Manager
- Create Overlay, VLAN and Edge Transport Zones
- Create Uplink Profiles for host and edge logical networking
- Create IP Pools for tunnel endpoints for hosts
- Create host transport nodes
- Configure and Deploy NSX Edge Node VMs
- Create an NSX Edge Cluster
- Create an NSX-T Tier 0 Uplink Segment
- Create an NSX-T Tier 0 Gateway
- Compute Configuration:
I think the reason that I couldn’t get this cluster to a “compatible” cluster for workload management has something to do with vLCM functionality.
- Automated Deployment using VMware Cloud Foundation: This is the “supported” way for deploying and managing vSphere with Kubernetes inside your datacenter. All the manual tasks that we listed above are completely automated using the “Workload Domain Deployment” and “Adding an Edge Cluster” to the workload domain Post Deployment steps. It was during this automated deployment process that I realized that I had to choose between using vLCM to manage my workload domain(Not supported with Workload Management aka vSphere with Kubernetes) or using VUM to manage my workload domain(Supported option for Workload Management), which was weird. This made me think that having vLCM enabled in my vSphere cluster blocked me from configuring my existing vSphere/vSAN cluster for Workload Management. But, anyways if you are thinking about running vSphere with Kubernetes in Production environments, VMware Cloud Foundation 4 is the only supported option, so its better to use the automated deployment workflow, so you know that you have correctly configured your environment for vSphere with Kubernetes.
I was able to get vSphere with Kubernetes running on my VMware Cloud Foundation environment running on Lenovo ThinkAgile VX, so in my next blog posts, we will definitely talk about the different integrations and operational level details around vSphere with Kubernetes. Till then, if you have tried the manual approach and can help me troubleshoot my environment, please reach out. 🙂