user-guide.md

• User Guide

• User Guide

  • Set Storage Class
    • An Example of Storage Class
    • Parameters in Storage Class
      • type, maxSize, minSize, stepSize
      • fsType
      • replica
      • tags
    • Other Parameters
      • Set Default Storage Class
      • Expand Volume
      • Topology Awareness
    • Disk Type Matrix
    • Instance Type Matrix
    • Disk Compatibility Matrix
  • Set Snapshot Class
    • An Example of Snapshot Class
    • Parameters in Storage Class
      • tags
  • Volume Management
    • Prerequisite
      • Create Storage Class
    • Create Volume
    • Mount Volume
    • Unmount Volume
    • Delete Volume
  • Expand Volume
    • Prerequisite
    • Expand Volume Online
    • Expand Volume Offline
    • Check
  • Clone Volume
    • Prerequisite
    • Cloning
  • Snapshot Management
    • Prerequisite
      • Create a volume pre-populated data
    • Create Snapshot
      • Create Volume Snapshot Class
      • Craete Volume Snapshot
    • Restore Volume from Snapshot
      • Restore
      • Check Data
    • Delete Snapshot
  • Topology Awareness
    • Prerequisite
      • Example Kubernetes Cluster
      • Create Storage Class
    • Create Volume
    • Create Workload
  • Static Volume Provisioning
    • Process
      • Prerequisite
      • Create Storage Class
      • Create PV
      • Create PVC

User Guide

Set Storage Class

An Example of Storage Class

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
  name: csi-qingcloud
provisioner: disk.csi.qingcloud.com
parameters:
  type: "0"
  fsType: "ext4"
  replica: "2"
  tags: "tag-y7uu1q2a"
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate

Parameters in Storage Class

type, maxSize, minSize, stepSize

See details in QingCloud docs。

Disk	type	maxSize(GB)	minSize(GB)	stepSize(GB)
High Performance	0	2000	20	10
High Capacity	2	5000	20	10
Super High Performance	3	2000	20	10
NeonSAN	5	24000	20	10
NeonSAN HDD	6	32000	20	10
Standard	100	2000	10	10
SSD Enterprise	200	2000	20	10

fsType

Support ext3, ext4, xfs. Default is ext4.

replica

1 represents single duplication disk，2 represents multiple duplication disk. Default is 2.

tags

The ID of QingCloud Tag resource, split by a single comma. Disks created by this plugin will be attached with the specified tags.

Other Parameters

Set Default Storage Class

In annotation, please set the value of .metadata.annotations.storageclass.beta.kubernetes.io/is-default-class as true. See details in Kubernetes docs

Expand Volume

Set the value of .allowVolumeExpansion as true. See details in Kubernetes docs

Topology Awareness

We can set Immediate or WaitForFirstConsumer as the value of .volumeBindingMode. See details in Kubernetes docs

Disk Type Matrix

Disk	type
High Performance	0
High Capacity	2
Super High Performance	3
NeonSAN	5
NeonSAN HDD	6
NeonSAN RDMA	7
Third Party Storage	20
Standard	100
SSD Enterprise	200

Instance Type Matrix

Instance	type
High Performance	0
Super High Performance	1
Super High Performance SAN	6
High Performance SAN	7
Standard	101
Enterprise1	201
Enterprise2	202
Enterprise3	203
Premium	301
NvidiaAmpereG3	1003

Disk Compatibility Matrix

	[0]High Performance Disk	[2]High Capacity Disk	[3]Super High Performance Disk	[5]NeonSAN Disk	[6]NeonSAN HDD Disk	[7]NeonSAN RDMA Disk	[20]Third Party Storage Disk	[100]Standard Disk	[200]SSD Enterprise Disk
[0]High Performance Instance	✓	✓	-	✓	✓	✓	-	✓	-
[1]Super High Performance Instance	-	✓	✓	✓	✓	✓	-	-	✓
[6]Super High Performance SAN Instance	-	-	-	✓	✓	✓	-	-	-
[7]High Performance SAN Instance	-	-	-	✓	✓	✓	-	-	-
[101]Standard Instance	✓	✓	-	✓	✓	✓	✓	✓	-
[201]Enterprise1 Instance	-	✓	✓	✓	✓	✓	-	-	✓
[202]Enterprise2 Instance	-	✓	✓	✓	✓	✓	-	-	✓
[203]Enterprise3 Instance	-	-	-	✓	✓	✓	-	-	✓
[301]Premium Instance	-	✓	✓	✓	✓	✓	-	-	✓
[1003]NvidiaAmpereG3 Instance	-	✓	-	✓	✓	✓	-	-	-

Set Snapshot Class

An Example of Snapshot Class

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotClass
metadata:
  name: csi-qingcloud
driver: disk.csi.qingcloud.com
parameters:
  tags: "tag-y7uu1q2a"
deletionPolicy: Delete

Parameters in Storage Class

tags

The ID of QingCloud Tag resource, split by a single comma. Snapshots created by this plugin will be attached with the specified tags.

Volume Management

Volume management including dynamical provisioning/deleting volume, attaching/detaching volume. Please reference Example YAML Files。

Prerequisite

• Kubernetes 1.16+ Cluster
• Installed QingCloud CSI plugin
• Created QingCloud CSI storage class

Create Storage Class

• Create

$ kubectl create -f sc.yaml

• Check

$ kubectl get sc
NAME            PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
csi-qingcloud   disk.csi.qingcloud.com   Delete          Immediate           true                   40h

Create Volume

• Create

$ kubectl create -f pvc.yaml 
persistentvolumeclaim/pvc-example created

• Check

$ kubectl get pvc pvc-example
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-example   Bound    pvc-76429525-a930-11e9-9a6a-5254ef68c8c1   20Gi       RWO            csi-qingcloud   25m

Mount Volume

• Create Deployment

$ kubectl create -f deploy-nginx.yaml 
deployment.apps/deploy-nginx created

• Check

$ kubectl exec -ti deploy-nginx-84474cf674-zfhbs /bin/bash
# cd /mnt
# ls
lost+found

Unmount Volume

• Delete Deployment

$ kubectl delete deploy deploy-nginx
deployment.extensions "deploy-nginx" deleted

Delete Volume

• Delete

$ kubectl delete pvc pvc-example
persistentvolumeclaim "pvc-example" deleted

• Check

$ kubectl get pvc pvc-example
Error from server (NotFound): persistentvolumeclaims "pvc-example" not found

Expand Volume

This feature could expand the capacity of volume.

Prerequisite

• Kubernetes 1.17+ cluster
• Enable (Default enabled) ExpandCSIVolumes=true in feature-gates
• Set allowVolumeExpansion as true in storage class
• Create a Pod mounting a volume

Expand Volume Online

csi-qingcloud support online volume expansion from v1.3.6.

• Change volume capacity

$ kubectl patch pvc pvc-example -p '{"spec":{"resources":{"requests":{"storage": "40Gi"}}}}'
persistentvolumeclaim/pvc-example patched

• Check volume capacity

$ kubectl get pvc pvc-example
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-example   Bound    pvc-906f5760-a935-11e9-9a6a-5254ef68c8c1   40Gi       RWO            csi-qingcloud   6m7s
$ kubectl get po
NAME                            READY   STATUS    RESTARTS   AGE
deploy-nginx-6c444c9b7f-d6n29   1/1     Running   0          3m38s

• check

$ kubectl exec -ti deploy-nginx-84474cf674-zfhbs /bin/bash
# df -hl | grep /mnt
/dev/vdg         40G   48M   40G   1% /mnt

Expand Volume Offline

For previous versions, csi-qingcloud only supports offline volume expansion. The procedure of offline volume expansion is shown as follows.

1. Ensure volume in unmounted status
2. Edit the capacity of PVC
3. Mount volume on workload. Reference Example YAML files

Unmount Volume

$ kubectl scale deploy deploy-nginx --replicas=0

Expand Volume

• Change volume capacity

$ kubectl patch pvc pvc-example -p '{"spec":{"resources":{"requests":{"storage": "40Gi"}}}}'
persistentvolumeclaim/pvc-example patched

• Mount volume

$ kubectl scale deploy deploy-nginx --replicas=1

• Check volume capacity

$ kubectl get pvc pvc-example
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-example   Bound    pvc-906f5760-a935-11e9-9a6a-5254ef68c8c1   40Gi       RWO            csi-qingcloud   6m7s
$ kubectl get po
NAME                            READY   STATUS    RESTARTS   AGE
deploy-nginx-6c444c9b7f-d6n29   1/1     Running   0          3m38s

Check

$ kubectl exec -ti deploy-nginx-6c444c9b7f-d6n29 /bin/bash
root@deploy-nginx-6c444c9b7f-d6n29:/# s
bash: s: command not found
root@deploy-nginx-6c444c9b7f-d6n29:/# df -ah
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/vdc         40G   49M   40G   1% /mnt
...

Clone Volume

A Clone is defined as a duplicate of an existing Kubernetes Volume. Please reference Example YAML files.

Prerequisite

• Kubernetes 1.16+ cluster
• Enable (Default enabled) VolumePVCDataSource=true feature gate
• Install QingCloud CSI plugin
• Create QingCloud CSI storage class
• Create a volume

Cloning

• Find volume

$ kubectl get pvc
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-example   Bound    pvc-d1fb263e-b368-4339-8f8b-448446f4b840   20Gi       RWO            csi-qingcloud   32s

• Clone volume

$ kubectl create -f pvc-clone.yaml
persistentvolumeclaim/pvc-clone created

• Check

$ kubectl get pvc pvc-clone
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-clone   Bound    pvc-529d2502-02bd-442b-a69f-d3eff28316a8   20Gi       RWO            csi-qingcloud   31s

Snapshot Management

Snapshot management contains creating/deleting snapshot and restoring volume from snapshpot. Please reference Example YAML files.

Prerequisite

• Kubernetes 1.17+ cluster
• Enable (Default enabled) VolumeSnapshotDataSource=true feature gate at kube-apiserver and kube-controller-manager
• Install QingCloud CSI plugin
• Create QingCloud CSI storage class
• Create a volume

Create a volume pre-populated data

• Create volume

$ kubectl create -f original-pvc.yaml
persistentvolumeclaim/pvc-snap-1 created

• Check

$ kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-snap-1   Bound    pvc-28090960-9eeb-11e9-9a6a-5254ef68c8c1   20Gi       RWO            csi-qingcloud   37s

• Write data

$ kubectl create -f deploy-writer.yaml 
deployment.apps/fio created

$ kubectl get po
NAME                   READY   STATUS    RESTARTS   AGE
fio-645b5d6499-8tc7p   1/1     Running   0          23s

$ kubectl exec -ti fio-645b5d6499-8tc7p /bin/bash
root@fio-645b5d6499-8tc7p:/# cd root
root@fio-645b5d6499-8tc7p:/# ./start-test.sh
crtl+c (5 秒后执行此命令，停止写数据)
root@fio-645b5d6499-8tc7p:/# ls -lh /mnt
total 20G
drwx------ 2 root root  16K Jul  5 06:09 lost+found
-rw-r--r-- 1 root root    0 Jul  5 06:10 rand-write.0.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 rand-write.1.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 rand-write.2.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 rand-write.3.0
-rw-r--r-- 1 root root  10G Jul  5 06:10 seq-write.0.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 seq-write.1.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 seq-write.2.0
-rw-r--r-- 1 root root 9.6G Jul  5 06:10 seq-write.3.0

Create Snapshot

Create Volume Snapshot Class

$ kubectl create -f snapshot-class.yaml 
volumesnapshotclass.snapshot.storage.k8s.io/csi-qingcloud created

$ kubectl get volumesnapshotclass
NAME            AGE
csi-qingcloud   16s

Craete Volume Snapshot

$ kubectl create -f volume-snapshot.yaml 
volumesnapshot.snapshot.storage.k8s.io/snap-1 created

$ kubectl get volumesnapshot
NAME     AGE
snap-1   91s

Restore Volume from Snapshot

Restore

$ kubectl create -f restore-pvc.yaml 
persistentvolumeclaim/pvc-snap-2 created

$ kubectl get pvc pvc-snap-2
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-snap-2   Bound    pvc-b8a05427-9eef-11e9-9a6a-5254ef68c8c1   20Gi       RWO            csi-qingcloud   52s

Check Data

Compare the difference between restored volume with original volume.

$ kubectl create -f deploy-viewer.yaml 
deployment.apps/nginx created

$ kubectl get po |grep snap-example 
snap-example-85dd9b646c-56g85   1/1     Running   0          3m6s

$ kubectl exec -ti snap-example-85dd9b646c-56g85 /bin/bash
root@snap-example-85dd9b646c-56g85:/# ls /mnt -lh
total 20G
drwx------ 2 root root  16K Jul  5 06:09 lost+found
-rw-r--r-- 1 root root    0 Jul  5 06:10 rand-write.0.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 rand-write.1.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 rand-write.2.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 rand-write.3.0
-rw-r--r-- 1 root root  10G Jul  5 06:10 seq-write.0.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 seq-write.1.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 seq-write.2.0
-rw-r--r-- 1 root root 9.6G Jul  5 06:10 seq-write.3.0

Delete Snapshot

$ kubectl delete volumesnapshot snap-1
volumesnapshot.snapshot.storage.k8s.io "snap-1" deleted

Topology Awareness

Topology awareness is used at Kubernetes clusters whose nodes across different available zones or having different types of instance. Please reference Example YAML files.

Prerequisite

• Kubernetes 1.16+ cluster
• Enable CSINodeInfo=true feature gate at Kubernetes control plane and Kubelet
• Install QingCloud CSI plugin and enable Topology=true feature gate at external-provisioner sidecar container
• Set QingCloud CSI storage class

Example Kubernetes Cluster

In QingCloud Pek3 zone, a Kubernetes v1.15 cluster with same types of instance is created and node1 and node2 running in Pek3c, node3 and node4 running in Pek3b, node5 and node6 running in Pek3d.

Create Storage Class

• Volume binding mode can be set as WaitForFirstConsumer or Immediate. Please reference Kubernetes docs.

$ kubectl create -f sc.yaml

$ kubectl get sc csi-qingcloud -oyaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-qingcloud
parameters:
  fsType: ext4
  replica: "2"
  type: "100"
provisioner: disk.csi.qingcloud.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Create Volume

• Create

$ kubectl create -f pvc.yaml
persistentvolumeclaim/pvc-topology created

• If VolumeBindingMode set as WaitForFirstConsumer, the status of PVC is shown as Pending. After Pod mounted PVC is sheduled, the PVC status will change to Bound.

$ kubectl get pvc pvc-topology
NAME           STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-topology   Pending                                      csi-qingcloud   31s

Create Workload

• Create Deployment

$ kubectl create -f deploy.yaml
deployment.apps/nginx-topology created

• Check if Pods scheduled on node 3

$ kubectl get po -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP               NODE    NOMINATED NODE   READINESS GATES
nginx-topology-79d8d5d86d-4lvcl    1/1     Running   0          52s   10.233.92.27     node3   <none>           <none>

• Check volume bound

$ kubectl get pvc pvc-topology
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-topology   Bound    pvc-5b34120c-6119-4c86-b9de-e152304683e6   20Gi       RWO            csi-qingcloud   2m48s

• The volume named pvc-topology contains topology information and can be mounted on special nodes. In this example, the volume can be mounted on node3 or node4.

Static Volume Provisioning

Static volume provisioning is also called pre-provisioning volume. The process is shown below.

1. Create QingCloud disk
2. Create PV
3. Create PVC

Process

Prerequisite

• Create Kubernetes cluster on QingCloud IaaS platform
• Install QingCloud CSI plugin
• Create QingCloud disk

Create Storage Class

$ kubectl get sc csi-qingcloud -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"csi-qingcloud","namespace":""},"parameters":{"fsType":"ext4","maxSize":"500","minSize":"10","stepSize":"10","type":"0"},"provisioner":"csi-qingcloud","reclaimPolicy":"Delete"}
  creationTimestamp: 2018-08-06T02:20:19Z
  name: csi-qingcloud
  resourceVersion: "1355065"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/csi-qingcloud
  uid: 43f25337-991f-11e8-b5aa-525445c0b555
parameters:
  fsType: ext4
  type: "0"
provisioner: csi-qingcloud
reclaimPolicy: Delete
volumeBindingMode: Immediate

Create PV

• Edit

$ vi pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: csi-qingcloud
  name: pv-static
spec:
  capacity:
    storage: 20Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: csi-qingcloud
  csi:
    driver: csi-qingcloud
    fsType: ext4
    volumeAttributes:
      fsType: ext4
      type: "0"
    volumeHandle: vol-jjtedp2i

• Create

$ kubectl create -f pv.yaml

• Check

$ kubectl get pv pv-static
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM     STORAGECLASS    REASON    AGE
pv-static   20Gi       RWO            Delete           Available             csi-qingcloud             8m

Create PVC

• Edit

$ vi pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: csi-qingcloud
  name: pvc-static
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: csi-qingcloud

• Create

$ kubectl create -f pvc.yaml

• Check

$  kubectl get pvc pvc-static
NAME         STATUS    VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-static   Bound     pv-static   20Gi       RWO            csi-qingcloud   11s

$ kubectl get pv pv-static 
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                STORAGECLASS    REASON    AGE
pv-static   20Gi       RWO            Delete           Bound     default/pvc-static   csi-qingcloud             12m