本文個人博客地址:https://www.huweihuang.com/kubernetes-notes/develop/csi-provisioner.html
本文主要分析
csi-provisioner
的源碼,關於開發一個Dynamic Provisioner
,具體可參考nfs-client-provisioner的源碼分析
1. Dynamic Provisioner
1.1. Provisioner Interface
開發Dynamic Provisioner
需要實現Provisioner接口,該接口有兩個方法,分別是:
- Provision:創建存儲資源,並且返回一個PV對象。
- Delete:移除對應的存儲資源,但並沒有刪除PV對象。
1.2. 開發provisioner的步驟
- 寫一個
provisioner
實現Provisioner
接口(包含Provision
和Delete
的方法)。 - 通過該
provisioner
構建ProvisionController
。 - 執行
ProvisionController
的Run
方法。
2. CSI Provisioner
CSI Provisioner的源碼可參考:https://github.com/kubernetes-csi/external-provisioner。
2.1. Main 函數
2.1.1. 讀取環境變量
源碼如下:
var (
provisioner = flag.String("provisioner", "", "Name of the provisioner. The provisioner will only provision volumes for claims that request a StorageClass with a provisioner field set equal to this name.")
master = flag.String("master", "", "Master URL to build a client config from. Either this or kubeconfig needs to be set if the provisioner is being run out of cluster.")
kubeconfig = flag.String("kubeconfig", "", "Absolute path to the kubeconfig file. Either this or master needs to be set if the provisioner is being run out of cluster.")
csiEndpoint = flag.String("csi-address", "/run/csi/socket", "The gRPC endpoint for Target CSI Volume")
connectionTimeout = flag.Duration("connection-timeout", 10*time.Second, "Timeout for waiting for CSI driver socket.")
volumeNamePrefix = flag.String("volume-name-prefix", "pvc", "Prefix to apply to the name of a created volume")
volumeNameUUIDLength = flag.Int("volume-name-uuid-length", -1, "Truncates generated UUID of a created volume to this length. Defaults behavior is to NOT truncate.")
showVersion = flag.Bool("version", false, "Show version.")
provisionController *controller.ProvisionController
version = "unknown"
)
func init() {
var config *rest.Config
var err error
flag.Parse()
flag.Set("logtostderr", "true")
if *showVersion {
fmt.Println(os.Args[0], version)
os.Exit(0)
}
glog.Infof("Version: %s", version)
...
}
通過init函數
解析相關參數,其實provisioner
指明爲PVC提供PV的provisioner的名字,需要和StorageClass
對象中的provisioner
字段一致。
2.1.2. 獲取clientset對象
源碼如下:
// get the KUBECONFIG from env if specified (useful for local/debug cluster)
kubeconfigEnv := os.Getenv("KUBECONFIG")
if kubeconfigEnv != "" {
glog.Infof("Found KUBECONFIG environment variable set, using that..")
kubeconfig = &kubeconfigEnv
}
if *master != "" || *kubeconfig != "" {
glog.Infof("Either master or kubeconfig specified. building kube config from that..")
config, err = clientcmd.BuildConfigFromFlags(*master, *kubeconfig)
} else {
glog.Infof("Building kube configs for running in cluster...")
config, err = rest.InClusterConfig()
}
if err != nil {
glog.Fatalf("Failed to create config: %v", err)
}
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
glog.Fatalf("Failed to create client: %v", err)
}
// snapclientset.NewForConfig creates a new Clientset for VolumesnapshotV1alpha1Client
snapClient, err := snapclientset.NewForConfig(config)
if err != nil {
glog.Fatalf("Failed to create snapshot client: %v", err)
}
csiAPIClient, err := csiclientset.NewForConfig(config)
if err != nil {
glog.Fatalf("Failed to create CSI API client: %v", err)
}
通過讀取對應的k8s的配置,創建clientset
對象,用來執行k8s對應的API,其中主要包括對PV和PVC等對象的創建刪除等操作。
2.1.3. k8s版本校驗
// The controller needs to know what the server version is because out-of-tree
// provisioners aren't officially supported until 1.5
serverVersion, err := clientset.Discovery().ServerVersion()
if err != nil {
glog.Fatalf("Error getting server version: %v", err)
}
獲取了k8s的版本信息,因爲provisioners的功能在k8s 1.5及以上版本才支持。
2.1.4. 連接 csi socket
// Generate a unique ID for this provisioner
timeStamp := time.Now().UnixNano() / int64(time.Millisecond)
identity := strconv.FormatInt(timeStamp, 10) + "-" + strconv.Itoa(rand.Intn(10000)) + "-" + *provisioner
// Provisioner will stay in Init until driver opens csi socket, once it's done
// controller will exit this loop and proceed normally.
socketDown := true
grpcClient := &grpc.ClientConn{}
for socketDown {
grpcClient, err = ctrl.Connect(*csiEndpoint, *connectionTimeout)
if err == nil {
socketDown = false
continue
}
time.Sleep(10 * time.Second)
}
在Provisioner
會停留在初始化狀態,直到csi socket
連接成功才正常運行。如果連接失敗,會暫停10秒
後重試,其中涉及以下2個參數:
- csiEndpoint:CSI Volume的gRPC地址,默認通過爲
/run/csi/socket
。 - connectionTimeout:連接CSI driver socket的超時時間,默認爲10秒。
2.1.5. 構造csi-Provisioner對象
// Create the provisioner: it implements the Provisioner interface expected by
// the controller
csiProvisioner := ctrl.NewCSIProvisioner(clientset, csiAPIClient, *csiEndpoint, *connectionTimeout, identity, *volumeNamePrefix, *volumeNameUUIDLength, grpcClient, snapClient)
provisionController = controller.NewProvisionController(
clientset,
*provisioner,
csiProvisioner,
serverVersion.GitVersion,
)
通過參數clientset
,csiAPIClient
, csiEndpoint
, connectionTimeout
, identity
, volumeNamePrefix
, volumeNameUUIDLength
,grpcClient
, snapClient
構造csi-Provisioner對象。
通過csiProvisioner
構造ProvisionController
對象。
2.1.6. 運行ProvisionController
func main() {
provisionController.Run(wait.NeverStop)
}
ProvisionController
實現了具體的PV和PVC的相關邏輯,Run
方法以常駐進程的方式運行。
2.2. Provision和Delete方法
2.2.1. Provision方法
csiProvisioner
的Provision
方法具體源碼參考:https://github.com/kubernetes-csi/external-provisioner/blob/master/pkg/controller/controller.go#L336
Provision
方法用來創建存儲資源,並且返回一個PV
對象。其中入參是VolumeOptions
,用來指定PV
對象的相關屬性。
1、構造PV相關屬性
pvName, err := makeVolumeName(p.volumeNamePrefix, fmt.Sprintf("%s", options.PVC.ObjectMeta.UID), p.volumeNameUUIDLength)
if err != nil {
return nil, err
}
2、構造CSIPersistentVolumeSource相關屬性
driverState, err := checkDriverState(p.grpcClient, p.timeout, needSnapshotSupport)
if err != nil {
return nil, err
}
...
// Resolve controller publish, node stage, node publish secret references
controllerPublishSecretRef, err := getSecretReference(controllerPublishSecretNameKey, controllerPublishSecretNamespaceKey, options.Parameters, pvName, options.PVC)
if err != nil {
return nil, err
}
nodeStageSecretRef, err := getSecretReference(nodeStageSecretNameKey, nodeStageSecretNamespaceKey, options.Parameters, pvName, options.PVC)
if err != nil {
return nil, err
}
nodePublishSecretRef, err := getSecretReference(nodePublishSecretNameKey, nodePublishSecretNamespaceKey, options.Parameters, pvName, options.PVC)
if err != nil {
return nil, err
}
...
volumeAttributes := map[string]string{provisionerIDKey: p.identity}
for k, v := range rep.Volume.Attributes {
volumeAttributes[k] = v
}
...
fsType := ""
for k, v := range options.Parameters {
switch strings.ToLower(k) {
case "fstype":
fsType = v
}
}
if len(fsType) == 0 {
fsType = defaultFSType
}
3、創建CSI CreateVolumeRequest
// Create a CSI CreateVolumeRequest and Response
req := csi.CreateVolumeRequest{
Name: pvName,
Parameters: options.Parameters,
VolumeCapabilities: volumeCaps,
CapacityRange: &csi.CapacityRange{
RequiredBytes: int64(volSizeBytes),
},
}
...
glog.V(5).Infof("CreateVolumeRequest %+v", req)
rep := &csi.CreateVolumeResponse{}
...
opts := wait.Backoff{Duration: backoffDuration, Factor: backoffFactor, Steps: backoffSteps}
err = wait.ExponentialBackoff(opts, func() (bool, error) {
ctx, cancel := context.WithTimeout(context.Background(), p.timeout)
defer cancel()
rep, err = p.csiClient.CreateVolume(ctx, &req)
if err == nil {
// CreateVolume has finished successfully
return true, nil
}
if status, ok := status.FromError(err); ok {
if status.Code() == codes.DeadlineExceeded {
// CreateVolume timed out, give it another chance to complete
glog.Warningf("CreateVolume timeout: %s has expired, operation will be retried", p.timeout.String())
return false, nil
}
}
// CreateVolume failed , no reason to retry, bailing from ExponentialBackoff
return false, err
})
if err != nil {
return nil, err
}
if rep.Volume != nil {
glog.V(3).Infof("create volume rep: %+v", *rep.Volume)
}
respCap := rep.GetVolume().GetCapacityBytes()
if respCap < volSizeBytes {
capErr := fmt.Errorf("created volume capacity %v less than requested capacity %v", respCap, volSizeBytes)
delReq := &csi.DeleteVolumeRequest{
VolumeId: rep.GetVolume().GetId(),
}
delReq.ControllerDeleteSecrets = provisionerCredentials
ctx, cancel := context.WithTimeout(context.Background(), p.timeout)
defer cancel()
_, err := p.csiClient.DeleteVolume(ctx, delReq)
if err != nil {
capErr = fmt.Errorf("%v. Cleanup of volume %s failed, volume is orphaned: %v", capErr, pvName, err)
}
return nil, capErr
}
Provison
方法核心功能是調用p.csiClient.CreateVolume(ctx, &req)
。
4、構造PV對象
pv := &v1.PersistentVolume{
ObjectMeta: metav1.ObjectMeta{
Name: pvName,
},
Spec: v1.PersistentVolumeSpec{
PersistentVolumeReclaimPolicy: options.PersistentVolumeReclaimPolicy,
AccessModes: options.PVC.Spec.AccessModes,
Capacity: v1.ResourceList{
v1.ResourceName(v1.ResourceStorage): bytesToGiQuantity(respCap),
},
// TODO wait for CSI VolumeSource API
PersistentVolumeSource: v1.PersistentVolumeSource{
CSI: &v1.CSIPersistentVolumeSource{
Driver: driverState.driverName,
VolumeHandle: p.volumeIdToHandle(rep.Volume.Id),
FSType: fsType,
VolumeAttributes: volumeAttributes,
ControllerPublishSecretRef: controllerPublishSecretRef,
NodeStageSecretRef: nodeStageSecretRef,
NodePublishSecretRef: nodePublishSecretRef,
},
},
},
}
if driverState.capabilities.Has(PluginCapability_ACCESSIBILITY_CONSTRAINTS) {
pv.Spec.NodeAffinity = GenerateVolumeNodeAffinity(rep.Volume.AccessibleTopology)
}
glog.Infof("successfully created PV %+v", pv.Spec.PersistentVolumeSource)
return pv, nil
Provision
方法只是通過VolumeOptions
參數來構建PV
對象,並沒有執行具體PV
的創建或刪除的操作。
不同類型的Provisioner
的,一般是PersistentVolumeSource
類型和參數不同,例如csi-provisioner
對應的PersistentVolumeSource
爲CSI
,並且需要傳入CSI
相關的參數:
Driver
VolumeHandle
FSType
VolumeAttributes
ControllerPublishSecretRef
NodeStageSecretRef
NodePublishSecretRef
2.2.2. Delete方法
csiProvisioner
的delete
方法具體源碼參考:https://github.com/kubernetes-csi/external-provisioner/blob/master/pkg/controller/controller.go#L606
func (p *csiProvisioner) Delete(volume *v1.PersistentVolume) error {
if volume == nil || volume.Spec.CSI == nil {
return fmt.Errorf("invalid CSI PV")
}
volumeId := p.volumeHandleToId(volume.Spec.CSI.VolumeHandle)
_, err := checkDriverState(p.grpcClient, p.timeout, false)
if err != nil {
return err
}
req := csi.DeleteVolumeRequest{
VolumeId: volumeId,
}
// get secrets if StorageClass specifies it
storageClassName := volume.Spec.StorageClassName
if len(storageClassName) != 0 {
if storageClass, err := p.client.StorageV1().StorageClasses().Get(storageClassName, metav1.GetOptions{}); err == nil {
// Resolve provision secret credentials.
// No PVC is provided when resolving provision/delete secret names, since the PVC may or may not exist at delete time.
provisionerSecretRef, err := getSecretReference(provisionerSecretNameKey, provisionerSecretNamespaceKey, storageClass.Parameters, volume.Name, nil)
if err != nil {
return err
}
credentials, err := getCredentials(p.client, provisionerSecretRef)
if err != nil {
return err
}
req.ControllerDeleteSecrets = credentials
}
}
ctx, cancel := context.WithTimeout(context.Background(), p.timeout)
defer cancel()
_, err = p.csiClient.DeleteVolume(ctx, &req)
return err
}
Delete
方法主要是調用了p.csiClient.DeleteVolume(ctx, &req)
方法。
2.3. 總結
csi provisioner
實現了Provisioner
接口,其中包含Provison
和Delete
兩個方法:
Provision
:調用csiClient.CreateVolume
方法,同時構造並返回PV對象。Delete
:調用csiClient.DeleteVolume
方法。
csi provisioner
的核心方法都調用了csi-client
相關方法。
3. csi-client
csi client
的相關代碼參考:https://github.com/container-storage-interface/spec/blob/master/lib/go/csi/v0/csi.pb.go
3.1. 構造csi-client
3.1.1. 構造grpcClient
// Provisioner will stay in Init until driver opens csi socket, once it's done
// controller will exit this loop and proceed normally.
socketDown := true
grpcClient := &grpc.ClientConn{}
for socketDown {
grpcClient, err = ctrl.Connect(*csiEndpoint, *connectionTimeout)
if err == nil {
socketDown = false
continue
}
time.Sleep(10 * time.Second)
}
通過連接csi socket
,連接成功才構造可用的grpcClient
。
3.1.2. 構造csi-client
通過grpcClient
構造csi-client
。
// Create the provisioner: it implements the Provisioner interface expected by
// the controller
csiProvisioner := ctrl.NewCSIProvisioner(clientset, csiAPIClient, *csiEndpoint, *connectionTimeout, identity, *volumeNamePrefix, *volumeNameUUIDLength, grpcClient, snapClient)
NewCSIProvisioner
// NewCSIProvisioner creates new CSI provisioner
func NewCSIProvisioner(client kubernetes.Interface,
csiAPIClient csiclientset.Interface,
csiEndpoint string,
connectionTimeout time.Duration,
identity string,
volumeNamePrefix string,
volumeNameUUIDLength int,
grpcClient *grpc.ClientConn,
snapshotClient snapclientset.Interface) controller.Provisioner {
csiClient := csi.NewControllerClient(grpcClient)
provisioner := &csiProvisioner{
client: client,
grpcClient: grpcClient,
csiClient: csiClient,
csiAPIClient: csiAPIClient,
snapshotClient: snapshotClient,
timeout: connectionTimeout,
identity: identity,
volumeNamePrefix: volumeNamePrefix,
volumeNameUUIDLength: volumeNameUUIDLength,
}
return provisioner
}
csiClient := csi.NewControllerClient(grpcClient)
...
type controllerClient struct {
cc *grpc.ClientConn
}
func NewControllerClient(cc *grpc.ClientConn) ControllerClient {
return &controllerClient{cc}
}
3.2. csiClient.CreateVolume
csi provisoner
中調用csiClient.CreateVolume
代碼如下:
opts := wait.Backoff{Duration: backoffDuration, Factor: backoffFactor, Steps: backoffSteps}
err = wait.ExponentialBackoff(opts, func() (bool, error) {
ctx, cancel := context.WithTimeout(context.Background(), p.timeout)
defer cancel()
rep, err = p.csiClient.CreateVolume(ctx, &req)
if err == nil {
// CreateVolume has finished successfully
return true, nil
}
if status, ok := status.FromError(err); ok {
if status.Code() == codes.DeadlineExceeded {
// CreateVolume timed out, give it another chance to complete
glog.Warningf("CreateVolume timeout: %s has expired, operation will be retried", p.timeout.String())
return false, nil
}
}
// CreateVolume failed , no reason to retry, bailing from ExponentialBackoff
return false, err
})
CreateVolumeRequest的構造:
// Create a CSI CreateVolumeRequest and Response
req := csi.CreateVolumeRequest{
Name: pvName,
Parameters: options.Parameters,
VolumeCapabilities: volumeCaps,
CapacityRange: &csi.CapacityRange{
RequiredBytes: int64(volSizeBytes),
},
}
...
req.VolumeContentSource = volumeContentSource
...
req.AccessibilityRequirements = requirements
...
req.ControllerCreateSecrets = provisionerCredentials
具體的Create
實現方法如下:
其中
csiClient
是個接口類型
具體代碼參考controllerClient.CreateVolume
func (c *controllerClient) CreateVolume(ctx context.Context, in *CreateVolumeRequest, opts ...grpc.CallOption) (*CreateVolumeResponse, error) {
out := new(CreateVolumeResponse)
err := grpc.Invoke(ctx, "/csi.v0.Controller/CreateVolume", in, out, c.cc, opts...)
if err != nil {
return nil, err
}
return out, nil
}
3.3. csiClient.DeleteVolume
csi provisoner
中調用csiClient.DeleteVolume
代碼如下:
func (p *csiProvisioner) Delete(volume *v1.PersistentVolume) error {
...
req := csi.DeleteVolumeRequest{
VolumeId: volumeId,
}
// get secrets if StorageClass specifies it
...
ctx, cancel := context.WithTimeout(context.Background(), p.timeout)
defer cancel()
_, err = p.csiClient.DeleteVolume(ctx, &req)
return err
}
DeleteVolumeRequest的構造:
req := csi.DeleteVolumeRequest{
VolumeId: volumeId,
}
...
req.ControllerDeleteSecrets = credentials
將構造的DeleteVolumeRequest
傳給DeleteVolume
方法。
具體的Delete
實現方法如下:
具體代碼參考:controllerClient.DeleteVolume
func (c *controllerClient) DeleteVolume(ctx context.Context, in *DeleteVolumeRequest, opts ...grpc.CallOption) (*DeleteVolumeResponse, error) {
out := new(DeleteVolumeResponse)
err := grpc.Invoke(ctx, "/csi.v0.Controller/DeleteVolume", in, out, c.cc, opts...)
if err != nil {
return nil, err
}
return out, nil
}
4. ProvisionController.Run
自定義的provisioner
實現了Provisoner接口
的Provision
和Delete
方法,這兩個方法主要對後端存儲做創建和刪除操作,並沒有對PV對象進行創建和刪除操作。
PV對象的相關操作具體由ProvisionController
中的provisionClaimOperation
和deleteVolumeOperation
具體執行,同時調用了具體provisioner
的Provision
和Delete
兩個方法來對存儲數據做處理。
func main() {
provisionController.Run(wait.NeverStop)
}
這塊代碼邏輯可參考:nfs-client-provisioner 源碼分析
參考文章: