關於Kubernetes的工程設計

Recently, I have been assigned to implement a task manage system based on etcd storage backend. Some of its design and implmentation are inspired by Kubernetes.

As a Go developer, dealing with Kubernetes may be an everyday routine, and you may konw a few of it, such as the basic concpets , architecure overview and processing flow. But when you dig into its source code, you will find some of its designs may be interesting, and it provides a lot insights for Go development, especially for NoSQL-based system development as well as micro-service development.

Table of Contents

• API Server
  • The runtime.Object
  • The useful client
• Controller
  • Dealing with Object Relationship
• Reference
  
  

API Server

Kubernetes API server is metadata service that only deals with CRUD of Object. It uses etcd as its persistent storage. It is worth noticing that other componets in Kubernetes architecture interact with API server to persist meta data.

The runtime.Object

This runtime is not refer to the golang runtime， but for the Kubernete runtime defined in Kubernete apimachinery, which declares Scheme, typing, encoding, decoding, and conversion packages for Kubernetes and Kubernetes-like API objects.

It is a good convention for a software project to define its own runtime as a package, then other components in the project imports and follows the interfaces of it.

• Every Object follows the same schema in outer layer, which is :

{ 
   "kind": ...
   "apiVersion": ...
   "spce" : ...
   "status": ...
}

• Every Object has embed TypeMeta, and TypeMeta has implemented runtime Object. So that every Object like Deployment, Pod and so on has implement runtime Object Interface
• In storage package that defines interface to interact with etcd storage, its parameters used to serialize or deserialize etcd value are runtime Object.
• In restful API handler, runtime Object is also being used. The business logic to create an Object is easy to understand here. Such that no need to implement different handler for different Object

To summarize the above interesting points, letting all Object to imlepment runtime.Object interface make API server much simpler:

• CRUD of all Object follows the same business logic, and for API server , all Object are basically the same.

• Some drawbacks can not be avoided, such as using reflect, which may bring to performance issues. API server has a map with key of kind string and value of Type to register differents Object, for the reason that API server must know which kind of Object it is going to create/update/delete( refer to addKonwnTypes source code), and which schema is used for serialization/deserialization. Introducing a universal interface runtime Object will definitely lead to this kind o issues.

• Also note that the endpoint of restful API are dynamic that it consists of Object kind, and the router will be registered, which depends on the Object map.

• The advantange is also obvious that, you can introduce CRD to API server that define your own schema of Object and customize behaviour of it.

By designing API server in such a concise style, it cut down the development overhead and bring the most facinating features to Kubernetes.

The useful client

Kubernetes/go-client is a individual componet from API server, but its main purpose is to wrap interaction with API server, also comes with cache , auth and sync features.

Components like Controller , Scheduler and kubectl interact with API server through this client, saving a lot development overhead and make the process of collaborative development more consistent.

Addtionally, it provides facinating features under the hood, like Reflector, Informer and Indexer, which are sophisticated design. It make the implement of other componet in Kubernetes much easier, especially for Controller

It is a good convension to provide a client/sdk for an API server .

Controller

Kubernetes Controller deal with different kind of Object and perform different logic to make Object to be in desirable state. By decoupling the Controller logic from API server make development much sophisticated .

Dealing with Object Relationship

There is always an issue that may bothers developers, which is how to design a system that only relies on a NoSQL database as the backend storage, as there are some object relationship issues, e.g. Pod is subresource of Deployment

• How to get all Pods of a Deployment ?
• How to get all Pending Pod ?
• How if I delete Pod of a Deployment?

Since NoSQL database is not good at handling Object Relationship as Relational Database, introducing Controller helps taking good care of it.

// todo.

References