189
Cloud Container Engine FAQs Issue 01 Date 2021-01-22 HUAWEI TECHNOLOGIES CO., LTD.

FAQs - HUAWEI CLOUD · 2020. 12. 25. · Cloud Container Engine FAQs Issue 01 Date 2020-12-25 HUAWEI TECHNOLOGIES CO., LTD

  • Upload
    others

  • View
    10

  • Download
    0

Embed Size (px)

Citation preview

  • Cloud Container Engine

    FAQs

    Issue 01

    Date 2021-01-22

    HUAWEI TECHNOLOGIES CO., LTD.

  • Copyright © Huawei Technologies Co., Ltd. 2021. All rights reserved.

    No part of this document may be reproduced or transmitted in any form or by any means without priorwritten consent of Huawei Technologies Co., Ltd. Trademarks and Permissions

    and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.All other trademarks and trade names mentioned in this document are the property of their respectiveholders. NoticeThe purchased products, services and features are stipulated by the contract made between Huawei andthe customer. All or part of the products, services and features described in this document may not bewithin the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,information, and recommendations in this document are provided "AS IS" without warranties, guaranteesor representations of any kind, either express or implied.

    The information in this document is subject to change without notice. Every effort has been made in thepreparation of this document to ensure accuracy of the contents, but all statements, information, andrecommendations in this document do not constitute a warranty of any kind, express or implied.

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. i

  • Contents

    1 Common Questions.................................................................................................................1

    2 Billing FAQs...............................................................................................................................32.1 How Is CCE Billed/Charged?................................................................................................................................................ 32.2 How Do I Change the Billing Mode of a CCE Cluster from Pay-per-Use to Yearly/Monthly?......................52.3 Can I Change the Billing Mode of CCE Nodes from Pay-per-Use to Yearly/Monthly?....................................62.4 Which Resource Quotas Should I Pay Attention To When Using CCE?................................................................72.5 Which Invoice Modes Are Supported by HUAWEI CLOUD?..................................................................................... 92.6 Will I Be Notified When My Balance Is Insufficient?.................................................................................................. 92.7 Will I Be Notified When My Account Balance Changes?...........................................................................................92.8 Can I Delete a Yearly/Monthly-Billed CCE Cluster Directly When It Expires?.................................................... 92.9 How Do I Unsubscribe From CCE?.................................................................................................................................... 92.10 Is Refund for CCE Supported?........................................................................................................................................ 10

    3 Cluster FAQs........................................................................................................................... 113.1 Cluster Creation..................................................................................................................................................................... 113.1.1 How Do I Restore a CCE Cluster That Fails to Be Created? What Are the Precautions for Creating aNode?............................................................................................................................................................................................... 113.1.2 Is Management Scale of a Cluster Related to the Number of Master Nodes?........................................... 123.1.3 Does CCE Support Windows Clusters?....................................................................................................................... 123.1.4 How Do I Update the Root Certificate When Creating a CCE Cluster?.......................................................... 123.2 Cluster Running..................................................................................................................................................................... 123.2.1 How Do I Rectify the Fault When the Cluster Status Is Unavailable?............................................................ 123.2.2 How Do I Reset or Reinstall a CCE Cluster?............................................................................................................. 153.2.3 How Do I Check Whether a Cluster Is in Multi-Master Mode?.........................................................................153.2.4 Where Do I Find the CPU and Memory Specifications of a Master Node?.................................................. 153.2.5 Can I Directly Connect to the Master Node of a Cluster?...................................................................................163.2.6 How Do I Retrieve Data After a Cluster Is Deleted?............................................................................................. 163.2.7 How Do I Update a Namespace in Terminating State in Kubernetes?.......................................................... 173.2.8 Changing the Mode of the Docker Device Mapper...............................................................................................193.2.9 Adding a Second Data Disk to a Node in a CCE Cluster..................................................................................... 24

    4 Node FAQs...............................................................................................................................274.1 Node Creation........................................................................................................................................................................ 274.1.1 What Is the Default OS of CCE Nodes?..................................................................................................................... 27

    Cloud Container EngineFAQs Contents

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. ii

  • 4.1.2 How Do I Troubleshoot Insufficient EIPs When a Node Is Added?.................................................................. 274.1.3 What Should I Do If a Node Fails to Be Accepted Because It Fails to Be Installed?..................................284.1.4 Using a Private Image to Build a Worker Node Image (OBT).......................................................................... 294.2 Node Running........................................................................................................................................................................ 334.2.1 What Should I Do If a Cluster Is Available But Some Nodes Are Unavailable?.......................................... 334.2.2 How Do I Troubleshoot the Failure to Remotely Log In to a Node in a CCE Cluster?.............................. 414.2.3 How Do I Log In to a Node Using a Password and Reset the Password?..................................................... 424.2.4 How Do I Collect Logs of Nodes in a CCE Cluster?............................................................................................... 424.2.5 What Can I Do If the Container Network Becomes Unavailable After yum update Is Used toUpgrade the OS?.......................................................................................................................................................................... 434.2.6 What Should I Do If the vdb Disk of a Node Is Damaged and the Node Cannot Be Recovered AfterReset?............................................................................................................................................................................................... 444.2.7 Which Ports Are Used to Install kubelet on CCE Cluster Nodes?..................................................................... 454.2.8 How Do I Configure a Pod to Use the Acceleration Capability of a GPU Node?........................................464.2.9 What Should I Do If I/O Suspension Occasionally Occurs When SCSI EVS Disks Are Used?..................464.2.10 What Should I Do If Excessive Docker Audit Logs Affect the Disk I/O?...................................................... 494.3 Specification Change........................................................................................................................................................... 504.3.1 How Do I Change the Node Specifications in a CCE Cluster?........................................................................... 504.3.2 What Should I Do If I Fail to Restart or Create Workloads on a Node After Modifying the NodeSpecifications?............................................................................................................................................................................... 524.3.3 How Do I Expand the Docker Disk Space of a CCE Node?................................................................................. 534.3.4 Can I Change the IP Address of a Node in a CCE Cluster?................................................................................. 544.3.5 How Do I Configure a Node Scaling Policy?............................................................................................................ 54

    5 Node Pool FAQs..................................................................................................................... 565.1 What Should I Do If No Node Creation Record Is Displayed When the Node Pool Is Being Expanding?............................................................................................................................................................................................................ 56

    6 Workload FAQs...................................................................................................................... 576.1 Workload Abnormalities.....................................................................................................................................................576.1.1 Fault Locating and Troubleshooting for Abnormal Workloads......................................................................... 576.1.2 What Should I Do If Pod Scheduling Fails?.............................................................................................................. 606.1.3 What Should I Do If Image Re-pull Fails?.................................................................................................................656.1.4 What Should I Do If Container Restart Fails?..........................................................................................................686.1.5 What Should I Do If a Pod Fails to Be Evicted?...................................................................................................... 746.1.6 What Should I Do If a Storage Volume Cannot Be Mounted or the Mounting Times Out?.................. 776.1.7 What Should I Do If a Workload Remains in the Creating State?................................................................... 786.1.8 What Should I Do If Pods in the Terminating State Cannot Be Deleted?..................................................... 796.1.9 What Should I Do If a Workload Is Stopped Caused by Pod Deletion?......................................................... 806.1.10 What Should I Do If an Error Occurs When Deploying a Service on the GPU Node?............................ 816.1.11 What Should I Do If Sandbox-Related Errors Are Reported When the Pod Remains in the CreatingState?................................................................................................................................................................................................826.2 Container Configuration..................................................................................................................................................... 836.2.1 When Is Pre-stop Processing Used?............................................................................................................................ 836.2.2 How Do I Set the Number of Pods for a Workload?............................................................................................ 83

    Cloud Container EngineFAQs Contents

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. iii

  • 6.2.3 How Do I Pause and Resume a Workload on the CCE Console? How Do I Restart a Container?........846.2.4 How Do I Set an FQDN for Accessing a Specified Container in the Same Namespace?......................... 846.2.5 What Should I Do If Health Check Probes Occasionally Fail?........................................................................... 856.2.6 How Do I Change the Permission of the Secret Mounted to a Container from 644 to 444?.................856.2.7 How Do I Set Java Probes for a Workload?............................................................................................................. 886.2.8 How Do I Set the umask Value for a Container?................................................................................................... 926.2.9 What Can I Do If an Error Is Reported When a Deployed Container Is Started After the JVM StartupHeap Memory Parameter Is Specified for ENTRYPOINT in Dockerfile?....................................................................936.3 Alarm Monitoring................................................................................................................................................................. 936.3.1 How Long Are the Events of a Workload Stored?................................................................................................. 936.4 Scheduling Policies............................................................................................................................................................... 946.4.1 How Do I Evenly Distribute Multiple Workloads to Each Node?..................................................................... 946.4.2 How Do I Prevent a Container on a Node from Being Evicted?....................................................................... 946.5 Others....................................................................................................................................................................................... 956.5.1 What Should I Do If a Scheduled Task Cannot Be Restarted After Being Stopped for a Period ofTime?................................................................................................................................................................................................ 966.5.2 Why Does the Client Fail to Receive Subscription Information After an Application Is Deployed onCCE?.................................................................................................................................................................................................. 966.5.3 What Is a Headless Service When I Create a StatefulSet?..................................................................................986.5.4 How Do I Set the Upper and Lower Limits of CPU and Memory Resources for a Container?...............986.5.5 What Should I Do If Error Message "Auth is empty" Is Displayed When a Private Image Is Pulled?............................................................................................................................................................................................................ 996.5.6 Why Cannot a Pod Be Scheduled to a Node?.......................................................................................................1006.5.7 What Is the Image Pull Policy for Containers in a CCE Cluster?.................................................................... 100

    7 Network Management FAQs............................................................................................1027.1 Network Planning.............................................................................................................................................................. 1027.1.1 What Is the Relationship Between Clusters, VPCs, and Subnets?.................................................................. 1027.1.2 How Do I View the VPC CIDR Block?....................................................................................................................... 1037.1.3 How Do I Set the VPC CIDR Block and Subnet CIDR Block for a CCE Cluster?.........................................1037.1.4 How Do I Set a Container CIDR Block for a CCE Cluster?................................................................................ 1047.1.5 What Is Yangtse and What Scenarios Does It Apply to?................................................................................... 1057.1.6 What Is an ENI?............................................................................................................................................................... 1067.1.7 Selecting a Network Model When Creating a Cluster on CCE........................................................................1077.1.8 Planning CIDR Blocks for a CCE Cluster.................................................................................................................. 1127.2 Network Fault...................................................................................................................................................................... 1187.2.1 What Should I Do If a Service Released in a Workload Cannot Be Accessed from Public Networks?......................................................................................................................................................................................................... 1187.2.2 Why Does the Browser Return Error Code 404 When I Access a Deployed Application?..................... 1217.2.3 What Should I Do If a Container Fails to Connect to the Internet?..............................................................1217.2.4 What Can I Do If a VPC Subnet Cannot Be Deleted?.........................................................................................1227.2.5 How Do I Restore a Faulty Container NIC?............................................................................................................1227.2.6 What Should I Do If a Node Fails to Connect to the Internet (Public Network)?................................... 1237.2.7 How Do I Resolve a Conflict Between the VPC CIDR Block and the Container CIDR Block?............... 124

    Cloud Container EngineFAQs Contents

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. iv

  • 7.2.8 What Should I Do If the Java Error "Connection reset by peer" Is Reported During Layer-4 ELBHealth Check............................................................................................................................................................................... 1247.2.9 How Do I Locate the Service Event Indicating That No Node Is Available for Binding?....................... 1257.3 Security Hardening............................................................................................................................................................ 1267.3.1 How Do I Harden the VPC Security Group Rules for CCE Cluster Nodes?..................................................1267.3.2 How Do I Prevent Cluster Nodes from Being Exposed to Public Networks?............................................. 1277.4 Network Configuration.....................................................................................................................................................1277.4.1 How Does CCE Communicate with Other HUAWEI CLOUD Services over an Intranet?....................... 1277.4.2 How Do I Set the Port When Configuring the Workload Access Mode on CCE?..................................... 1287.4.3 How Can I Achieve Compatibility Between Ingress's property and Kubernetes client-go?.................. 1307.4.4 How Can I Achieve Compatibility Between ExtendPathMode and Kubernetes client-go?................... 1327.5 Others..................................................................................................................................................................................... 1347.5.1 How Do I Obtain an Ingress TLS Certificate?........................................................................................................1357.5.2 Can Multiple NICs Be Bound to a Node in a CCE Cluster?...............................................................................1367.5.3 Why Is the Backend Server Group of an ELB Automatically Deleted After a Service Is Published tothe ELB?........................................................................................................................................................................................ 1377.5.4 Can the Container CIDR Block Be Modified After a Cluster Is Created?......................................................1377.5.5 Why Cannot an Ingress Be Created After the Namespace Is Changed?......................................................137

    8 Storage Management FAQs..............................................................................................1398.1 What Are the Differences Among CCE Storage Classes in Terms of Persistent Storage and Multi-nodeMounting?.................................................................................................................................................................................... 1398.2 Can I Add a Node Without a 100 GB Data Disk?................................................................................................... 1418.3 Can I Restore an EVS Disk Used as a Persistent Volume in a CCE Cluster After the Disk Is Deleted orExpires?..........................................................................................................................................................................................1418.4 What Should I Do If the Host Cannot Be Found When Files Need to Be Uploaded to OBS During theAccess to the CCE Service from a Public Network?....................................................................................................... 1418.5 How Many Nodes (ECSs) Can an SFS File System Be Mounted to?.................................................................142

    9 Chart FAQs............................................................................................................................ 1439.1 Can the Memory Be Expanded for a Workload Deployed Using a Sample Chart?.....................................1439.2 How Do I View the Image Versions of a Sample Chart and Workload?.........................................................1449.3 What Should I Do If the nginx-ingress Add-on Fails to Be Installed on a Cluster and Remains in theCreating State?........................................................................................................................................................................... 1459.4 How Do I Rectify Failures When the NVIDIA Driver Is Used to Start Containers on GPU Nodes?........1479.5 What Should I Do If Residual Process Resources Exist Due to an Earlier npd Add-on Version?............ 1489.6 What Should I Do If a Chart Release Cannot Be Deleted Because the Chart Format Is Incorrect?...... 149

    10 API and kubectl FAQs...................................................................................................... 15110.1 How Can I Access a CCE Cluster?............................................................................................................................... 15110.2 Can the Resources Created Using APIs or kubectl Be Displayed on the CCE Console?...........................15110.3 How Do I Download the Configuration File When I Connect to a Cluster Through Kubectl?..............152

    11 DNS FAQs............................................................................................................................ 15511.1 What Should I Do If Domain Name Resolution Fails?........................................................................................ 15511.2 Why Does a Container in a CCE Cluster Fail to Perform DNS Resolution?................................................. 157

    Cloud Container EngineFAQs Contents

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. v

  • 11.3 Why Cannot the Domain Name of the Tenant Zone Be Resolved After the Subnet DNS ConfigurationIs Modified?................................................................................................................................................................................. 16011.4 How Do I Optimize the Configuration If the External Domain Name Resolution Is Slow or TimesOut?................................................................................................................................................................................................ 16011.5 How Do I Configure DNS Policies in a CCE Cluster?........................................................................................... 161

    12 FAQs About Related Services......................................................................................... 16312.1 How Do I Create a Docker Image in CCE?.............................................................................................................. 16312.2 How Do I Upload Files to Docker Images?............................................................................................................. 16312.3 How Do I Upload My Images to CCE?......................................................................................................................16512.4 How Do I Build a Private Docker Image Repository?.......................................................................................... 16512.5 Why Does CCE Display Node Disk Usage Inconsistently with Cloud Eye?...................................................16612.6 What Are the Differences Between CCE and ServiceStage?............................................................................. 167

    13 Reference............................................................................................................................ 16813.1 How Do I Expand the Capacity of a Docker Container?.................................................................................... 16813.2 How Can Container IP Addresses Survive a Container Restart?...................................................................... 16913.3 How Do I Add a Cloud Volume for a Workload?..................................................................................................17113.4 How Do I Expand the Capacity of an EVS Volume?............................................................................................ 17213.5 How Do I Back Up Data in CCE Container Storage?........................................................................................... 17413.6 Can On-premises Kubernetes Clusters Interconnect with HUAWEI CLOUD Storage Services?............17413.7 Does CCE Support nginx-ingress?.............................................................................................................................. 17513.8 How Do I Install and Configure kubectl to Remotely Perform Operations on Kubernetes Clusters?......................................................................................................................................................................................................... 17513.9 How Do I Use kubectl to Set the Workload Access Type to LoadBalancer (ELB)?................................... 17613.10 How Do I Connect CCE to a Private DNS?............................................................................................................18113.11 What Should I Do If I Do Not Have the Permissions to Access the CCE Console?.................................181

    Cloud Container EngineFAQs Contents

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. vi

  • 1 Common QuestionsCluster Management

    ● How Do I Restore a CCE Cluster That Fails to Be Created? What Are thePrecautions for Creating a Node?

    ● Is Management Scale of a Cluster Related to the Number of MasterNodes?

    ● How Do I Reset or Reinstall a CCE Cluster?● How Do I Update the Root Certificate When Creating a CCE Cluster?

    Node/Node Pool Management● What Should I Do If a Cluster Is Available But Some Nodes Are

    Unavailable?● What Should I Do If a Node Fails to Be Accepted Because It Fails to Be

    Installed?● What Should I Do If I/O Suspension Occasionally Occurs When SCSI EVS

    Disks Are Used?● What Should I Do If Excessive Docker Audit Logs Affect the Disk I/O?

    Workload Management● What Should I Do If Pod Scheduling Fails?● What Should I Do If Image Re-pull Fails?● What Should I Do If Container Restart Fails?● What Should I Do If Pods in the Terminating State Cannot Be Deleted?● What Is the Image Pull Policy for Containers in a CCE Cluster?● How Do I Upload Files to Docker Images?● How Do I Update a Namespace in Terminating State in Kubernetes?

    NetworkingWhy Does the Browser Return Error Code 404 When I Access a DeployedApplication?

    How Do I Obtain an Ingress TLS Certificate?

    Cloud Container EngineFAQs 1 Common Questions

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 1

  • How Do I Set the VPC CIDR Block and Subnet CIDR Block for a CCE Cluster?

    What Should I Do If a Node Fails to Connect to the Internet (PublicNetwork)?

    How Do I Optimize the Configuration If the External Domain NameResolution Is Slow or Times Out?

    What Should I Do If Domain Name Resolution Fails?

    What Is Yangtse and What Scenarios Does It Apply to?

    Cloud Container EngineFAQs 1 Common Questions

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 2

  • 2 Billing FAQs2.1 How Is CCE Billed/Charged?

    Billing ItemsCloud Container Engine (CCE) is free of charge. You only pay for the resources(such as nodes) created when you are using CCE. There are two types of billingitems:

    1. Clusters: The cluster fee is the cost of resources used by master nodes. Thefee varies with the cluster type and cluster size. Cluster types include VMcluster and BMS cluster (the number of master nodes determines whether acluster is highly available). Cluster size (also called management scale)indicates the maximum number of nodes allowed in a cluster.

    NO TE

    The management scale indicates the number of ECSs or BMSs in a cluster.

    For more details, see CCE Pricing Details.2. IaaS resources: The cost of IaaS resources created to run worker nodes in

    your cluster is billed. IaaS resources, which are created either manually orautomatically, include ECSs, EVS disks, EIPs, bandwidth, and load balancers.For more pricing details, see Product Pricing Details.

    Billing ModesCCE is billed on a pay-per-use or yearly/monthly basis.

    ● Pay-per-use: It is a pay-after-use mode. Billing starts when a resource isprovisioned and stops when the resource is deleted. You can use cloudresources as required and stop paying for them when you no longer needthem. There is no upfront payment for excess capacity.

    Cloud Container EngineFAQs 2 Billing FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 3

    https://www.huaweicloud.com/intl/en-us/pricing/index.html?tab=detail#/ccehttps://www.huaweicloud.com/intl/en-us/pricing/index.html?tab=detail#/ecs

  • NO TE

    The following are pricing principles in the case of CCE cluster hibernation or nodeshutdown. Note that there are many types of cluster nodes and ECS is used as anexample.● Cluster hibernation: After a cluster is hibernated, the billing of resources used by

    master nodes will stop.● Node shutdown: Worker node billing stops when the node is stopped. Note that

    hibernating a cluster will not stop worker nodes in the cluster. To stop an ECS, login to the ECS console. For details, see Stopping a Node.The operation of stopping an ECS is free of charge. After a pay-per-use ECSwithout local disks or FPGAs is stopped, the ECS, its vCPUs, memory, and imagesare not billed. However, other resources used by the ECS, such as EVS disks, EIPs,and bandwidth, are still billed. The vCPU and memory resources of the stoppedECS are reclaimed. When the ECS is restarted, the vCPU and memory resourcesmust be requested again. However, if the resources are insufficient, the restart mayfail. To avoid a restart failure, wait for several minutes before attempting anotherrestart or modify the ECS specifications. After an ECS with local hard disks (suchas enhanced disks and GPUs) and FPGAs is stopped, ECS billing continues andresources such as vCPUs and memory are retained. For details, see ECS Billing.

    ● Yearly/monthly: It is a pay-before-use mode. Yearly/monthly billing provides amore significant discount than pay-per-use and is recommended for long-term use of cloud services. When you purchase a yearly/monthly package, thesystem will deduct the package cost from your cloud account based on thechosen specifications.

    ● Billing mode change: The billing mode cannot be changed within the billingcycle.

    NO TICE

    Clusters follow a tiered pricing plan. Pricing for each tier varies with cluster sizeand type.

    Configuration ChangesFrom pay-per-use to yearly/monthly billing: You can change the cluster billingmode from pay-per-use to yearly/monthly billing. After the change, master nodes,worker nodes, and cloud resources (such as EVS disks and EIPs) used by yourcluster will all be billed on a yearly/monthly basis and a new order will begenerated. The nodes and cloud resources will be ready for use immediately afteryou pay for the new order.

    From yearly/monthly billing to pay-per-use: Clusters billed on a yearly/monthlybasis cannot change to pay-per-use within the billing cycle. Note that pay-per-useclusters can be directly deleted, but clusters billed on a yearly/monthly basiscannot be deleted. To stop using the clusters billed on a yearly/monthly basis, goto the Billing Center and unsubscribe from them.

    Notes

    ● Cash coupons will not be returned after you downgrade specifications of thecloud servers that are purchased using cash coupons.

    ● You will need to pay the price difference between the original and newspecifications after upgrading cloud server specifications.

    Cloud Container EngineFAQs 2 Billing FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 4

    https://support.huaweicloud.com/intl/en-us/usermanual-cce/cce_01_0036.htmlhttps://support.huaweicloud.com/intl/en-us/productdesc-ecs/ecs_01_0065.htmlhttps://account-intl.huaweicloud.com/usercenter/?locale=en-us#/userindex/retreatManagement

  • ● Downgrading cloud server specifications (the amount of CPU or memoryresources) will impair cloud server performance.

    ● If you downgrade cloud server specifications and then upgrade it to theoriginal specifications, you will still need to pay the price difference incurredby the upgrade.

    2.2 How Do I Change the Billing Mode of a CCE Clusterfrom Pay-per-Use to Yearly/Monthly?

    Currently, clusters support pay-per-use and yearly/monthly billing modes.

    For details on how to buy clusters, see Buying a Hybrid Cluster.

    Changing to Yearly/Monthly Billing ModeTo change the billing mode of the clusters you have purchased from pay-per-useto yearly/monthly, perform the following steps:

    Step 1 Log in to the CCE console. In the navigation pane, choose ResourceManagement > Clusters. In the card view of the clusters for which you willchange the billing mode, click Change Billing Mode.

    Figure 2-1 Changing to the yearly/monthly billing mode

    Step 2 On the Change Billing Mode page, choose the master and worker nodes that willchange to the yearly/monthly billing mode.

    Cloud Container EngineFAQs 2 Billing FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 5

    https://support.huaweicloud.com/intl/en-us/usermanual-cce/cce_01_0028.htmlhttps://console-intl.huaweicloud.com/cce2.0/?locale=en-us

  • Figure 2-2 Changing billing mode for master and worker nodes

    Step 3 Click OK. Wait until the order is processed and the payment is complete.

    ----End

    2.3 Can I Change the Billing Mode of CCE Nodes fromPay-per-Use to Yearly/Monthly?

    Currently, nodes support pay-per-use and yearly/monthly billing modes.

    For details, see Buying a Node.

    ProcedureTo change the billing mode of the nodes you have purchased from pay-per-use toyearly/monthly, perform the following steps:

    Step 1 Log in to the CCE console. In the navigation pane, choose ResourceManagement > Clusters. In the card view of the clusters for which you willchange the billing mode, click Change Billing Mode.

    Figure 2-3 Changing to the yearly/monthly billing mode

    Step 2 On the Change Billing Mode page, choose the nodes that will be change to theyearly/monthly billing mode.

    Cloud Container EngineFAQs 2 Billing FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 6

    https://support.huaweicloud.com/intl/en-us/usermanual-cce/cce_01_0033.htmlhttps://console-intl.huaweicloud.com/cce2.0/?locale=en-us

  • NO TE

    By default, the Change entire cluster to yearly/monthly billing mode is selected. If youwant to change certain nodes of the cluster to the yearly/monthly billing mode, deselectthis option.

    Figure 2-4 Changing nodes to the yearly/monthly billing mode

    If you want to change the whole cluster to the yearly/monthly billing mode, selectthis option and the nodes that need to be changed to the yearly/monthly billingmode.

    Figure 2-5 Changing the cluster and nodes of the cluster to the yearly/monthlybilling mode

    Step 3 Click OK. Wait until the order is processed and the payment is complete.

    ----End

    2.4 Which Resource Quotas Should I Pay Attention ToWhen Using CCE?

    CCE restricts only the number of clusters. However, when using the CCE, youmay also be using other HUAWEI CLOUD services, such as Elastic Cloud Server(ECS), Elastic Volume Service (EVS), Virtual Private Cloud (VPC), Elastic LoadBalance (ELB), and Software Repository for Containers (SWR).

    Cloud Container EngineFAQs 2 Billing FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 7

  • What Is Quota?Quotas are enforced for service resources on the platform to prevent unforeseenspikes in resource usage. Quotas can limit the number or amount of resourcesavailable to users, such as the maximum number of ECSs or EVS disks that can becreated.

    If the existing resource quota cannot meet your service requirements, you canapply for a higher quota.

    How Do I View My Quota?1. Log in to the HUAWEI CLOUD console.

    2. Click in the upper left corner to select a region and a project.3. In the upper right corner of the page, choose Resources > My Quotas.

    The Service Quota page is displayed.

    Figure 2-6 My Quotas

    4. On this page, you can view the total quota and used quota of resources.If a quota cannot meet your business requirements, click Increase Quota.

    How Do I Increase My Quota?1. Log in to the HUAWEI CLOUD console.2. In the upper right corner of the page, choose Resources > My Quotas.

    The Service Quota page is displayed.

    Figure 2-7 My Quotas

    3. Click Increase Quota.4. On the Submit Service Ticket page, set parameters as required and submit a

    service ticket.In the Problem Description area, enter the required quota and reason for theadjustment.

    Cloud Container EngineFAQs 2 Billing FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 8

    https://console-intl.huaweicloud.com/ticket/?locale=en-us#/ticketindex/createIndexhttps://console-intl.huaweicloud.com/ticket/?locale=en-us#/ticketindex/createIndex

  • 5. Select I have read and agree to the Tenant Authorization Letter and clickSubmit.

    2.5 Which Invoice Modes Are Supported by HUAWEICLOUD?

    HUAWEI CLOUD supports two issuing invoices by billing cycle and by order.

    You can choose Contacts and Invoices > Invoices on the Billing Center to issue aninvoice.

    2.6 Will I Be Notified When My Balance Is Insufficient?You can set a balance threshold on the recharge page. The system will check yourbalance when you purchase a product and send a notification if your balance islower than or equal to the threshold.

    2.7 Will I Be Notified When My Account BalanceChanges?

    The system will notify you via email or SMS message of your account balancechanges, including whether your online topping up is successful.

    2.8 Can I Delete a Yearly/Monthly-Billed CCE ClusterDirectly When It Expires?

    After a yearly/monthly-billed cluster expires, you can delete the cluster after alldata is backed up.

    If you do not renew or delete an expired cluster, the system provides differentretention periods based on your customer level. After the retention period elapses,your CCE cluster will be deleted. Pay attention to the resource expiration time,back up data, and renew your resource subscription if needed.

    2.9 How Do I Unsubscribe From CCE?Yearly/monthly-billed CCE resources can be unsubscribed from, including therenewed part and currently used part. You cannot use these resources afterunsubscription. A handling fee will be charged for unsubscribing from a resource.

    Note● Unsubscribing from CCE involves the renewed resources and the resources

    that are being used. After the unsubscription, these resources becomeunavailable.

    ● Solution product portfolios can only be unsubscribed from as a whole.

    Cloud Container EngineFAQs 2 Billing FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 9

  • ● If an order contains resources in a primary-secondary relationship, you needto unsubscribe from the resources separately.

    ● For details about unsubscribing from resources, see Unsubscription Rules.

    Procedure

    CA UTION

    ● Before requesting an unsubscription, ensure that you have migrated or backedup any data saved on CCE that will be unsubscribed from. After theunsubscription is complete, CCE and any data it contains will be permanentlydeleted.

    ● The middle of the unsubscription page displays a message showing the numberof unsubscriptions you have performed and the remaining allowed number.

    1. Go to the Unsubscription page on the Billing Center.2. Click the Active Resources tab.3. Unsubscribe from a single resource or from resources in a batch.

    – To unsubscribe from a single resource, click Unsubscribe from Resourceat the row of the target resource name.

    – To unsubscribe from resources in a batch, select the target resources fromthe resource list and click Unsubscribe from Resources in the upper leftcorner of the resource list.

    4. On the Unsubscribe page, confirm the unsubscription information, select areason for the unsubscription, and click Confirm.

    2.10 Is Refund for CCE Supported?Currently, automatic refund is not supported.

    Cloud Container EngineFAQs 2 Billing FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 10

    https://support.huaweicloud.com/intl/en-us/usermanual-billing/en-us_topic_0083138805.htmlhttps://account-intl.huaweicloud.com/usercenter/?locale=en-us#/userindex/retreatManagement

  • 3 Cluster FAQs3.1 Cluster Creation

    3.1.1 How Do I Restore a CCE Cluster That Fails to Be Created?What Are the Precautions for Creating a Node?

    The possible causes are as follows: The Network Time Protocol daemon (ntpd) isnot installed or fails to be installed, Kubernetes components fail to pass the pre-verification, and the disk partition is incorrect. The current solution is to create acluster again.

    Precautions for creating a node:

    ● The node images in the same cluster must be the same. Pay attention to thiswhen creating, adding, or accepting nodes in a cluster.

    ● When creating a node, if you need to allocate user space from the data disk,do not set the data storage path to key directories. For example, if the dataneeds to be stored in the /home directory, you are advised to set the directoryto /home/test instead of /home.

    NO TE

    The mount path of the user space data cannot be set to the root directory /.Otherwise, the mounting fails. Mount paths can be as follows:● /opt/xxxx (excluding /opt/cloud)● /mnt/xxxx (excluding /mnt/paas)● /tmp/xxx● /xxxx (It cannot conflict with the system directory, such as bin, lib, home, root,

    boot, dev, etc, lost+found, mnt, proc, sbin, srv, tmp, var, media, opt, selinux, sys andusr.)

    If the mount path is set to /home/paas, /var/paas, /mnt/paas, or /opt/cloud, thesystem or node installation will fail.

    Cloud Container EngineFAQs 3 Cluster FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 11

  • 3.1.2 Is Management Scale of a Cluster Related to theNumber of Master Nodes?

    Management scale indicates the maximum number of nodes that can bemanaged by a cluster. If you select 50 nodes, the cluster can manage a maximumof 50 nodes. The cluster management scale cannot be modified after a cluster iscreated. Exercise caution when creating a cluster.

    The number of master nodes varies according to the cluster specification, but isnot affected by the management scale.

    After the multi-master node mode is enabled, three master nodes will be created.If a master node is faulty, the cluster can still be available without affectingservice functions.

    3.1.3 Does CCE Support Windows Clusters?The feature of creating Windows clusters has been removed from CCE.

    You can use CCE to create hybrid clusters.

    Description

    Hybrid cluster: VM nodes and BMS nodes can be deployed and managed at thesame time. The following scenarios are supported: independent VM scenarios(original VM clusters and VM nodes), independent physical machine scenarios(BMS nodes), and hybrid deployment of VMs and physical machines.

    3.1.4 How Do I Update the Root Certificate When Creating aCCE Cluster?

    The root certificate of CCE clusters is the basic certificate for Kubernetesauthentication. Both the Kubernetes cluster control plane and the certificate arehosted on HUAWEI CLOUD CCE. CCE will periodically update the certificate. Thiscertificate is not open to users but will not expire.

    The X.509 certificate is enabled on Kubernetes clusters by default. CCE willautomatically maintain and update the X.509 certificate.

    Obtaining a Cluster Certificate

    You can obtain a cluster certificate on the CCE console to access Kubernetes. Fordetails, see Obtaining a Cluster Certificate.

    3.2 Cluster Running

    3.2.1 How Do I Rectify the Fault When the Cluster Status IsUnavailable?

    If the cluster is Unavailable, perform the following operations to rectify the fault:

    Cloud Container EngineFAQs 3 Cluster FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 12

    https://support.huaweicloud.com/intl/en-us/usermanual-cce/cce_01_0028.htmlhttps://support.huaweicloud.com/intl/en-us/usermanual-cce/cce_01_0175.html

  • Fault LocatingTroubleshooting methods are sorted based on the occurrence probability of thepossible causes. You are advised to check the possible causes from high probabilityto low probability to quickly locate the cause of the problem.

    If the fault persists after a possible cause is rectified, check other possible causes.

    ● Check Item 1: Whether the Security Group Is Modified● Check Item 2: Whether There Are Residual Listeners and Backend Server

    Groups on the Load Balancer

    Figure 3-1 Fault locating

    Check Item 1: Whether the Security Group Is Modified

    Step 1 Log in to the public cloud management console, choose Service List > Network >Virtual Private Cloud. In the navigation pane, choose Access Control > SecurityGroups to find the security group of the master node in the cluster.

    The name of this security group is in the format of Cluster name-cce-control-ID,as shown in Figure 3-2.

    Figure 3-2 Master node in the cluster

    Step 2 Click the security group. On the details page that is displayed, ensure that thesecurity group rules of the master node are the same as those marked by the redframes in the following figure.

    Cloud Container EngineFAQs 3 Cluster FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 13

    https://console-intl.huaweicloud.com/vpc/?locale=en-us#/vpcs

  • Figure 3-3 Viewing inbound rules of the security group

    Inbound rule parameter description:

    ● 4789: used for network access between containers.

    ● 5443-5444: used by kubelet of the node to listen to kube-api of the masternode.

    ● 9443: used by canal of the node to listen to canal-api of the master node.

    ● 8445: used by storage_driver of the node to access csms-storagemgr of themaster node.

    Figure 3-4 Viewing outbound rules of the security group

    ----End

    Check Item 2: Whether There Are Residual Listeners and Backend ServerGroups on the Load Balancer

    Reproducing the Problem

    A cluster exception occurs when a LoadBalancer Service is being created ordeleted. After the fault is rectified, the Service is deleted successfully, but there areresidual listeners and backend server group.

    Step 1 Pre-create a CCE cluster. In the cluster, use the official Nginx image to createworkloads, preset load balancers, Services, and ingresses.

    Step 2 Ensure that the cluster is running properly and the Nginx workload is stable.

    Step 3 Create and delete 10 LoadBalancer Services every 20 seconds.

    Cloud Container EngineFAQs 3 Cluster FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 14

  • Step 4 An injection exception occurs in the cluster. For example, the etcd pod isunavailable or the cluster is hibernated.

    ----End

    Possible Cause

    There are residual listeners and backend server groups on the load balancer.

    Solution

    Manually clear residual listeners and backend server groups.

    Step 1 Log in to the management console and choose Network > Elastic Load Balancefrom the service list.

    Step 2 In the load balancer list, click the name of the target load balancer to go to thedetails page. On the Listeners tab page, locate the target listener and delete it.

    Step 3 On the Backend Server Groups tab page, locate the target backend server groupand delete it.

    ----End

    3.2.2 How Do I Reset or Reinstall a CCE Cluster?CCE clusters cannot be reset or reinstalled. If a cluster becomes unavailable,submit a service ticket or delete the cluster and purchase a new one.

    CCE supports resetting nodes. For details, see Resetting a Node.

    3.2.3 How Do I Check Whether a Cluster Is in Multi-MasterMode?

    Log in to the CCE console. In the navigation pane, choose ResourceManagement > Clusters. Click the name of the cluster to be viewed. In the upperright corner of the cluster details page, view the number of master nodes.

    ● 3: The cluster is in multi-master mode.● 1: The cluster is in single-master mode.

    NO TICE

    The number of master nodes cannot be changed after the cluster is created. If youwant to adjust the number, you need to create a new cluster.

    3.2.4 Where Do I Find the CPU and Memory Specifications of aMaster Node?

    CCE creates a master node during cluster creation. The specifications of themaster node vary based on the cluster management scale (50 nodes, 200 nodes,or 1,000 nodes) you have selected. Master node fee is included in the clustercreation fee. To view the specifications of the master node, perform the followingsteps:

    Cloud Container EngineFAQs 3 Cluster FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 15

    https://console-intl.huaweicloud.com/ticket/?locale=en-us#/ticketindex/createIndexhttps://support.huaweicloud.com/intl/en-us/usermanual-cce/cce_01_0003.htmlhttps://console-intl.huaweicloud.com/cce2.0/?locale=en-us

  • Step 1 Log in to the CCE console. In the navigation pane, choose ResourceManagement > Clusters. In the cluster list, click the name of the cluster whosespecifications you want to view.

    Figure 3-5 Clicking a cluster name

    Step 2 In the Master Node area in the upper right corner of the cluster details page,view the specifications of the master node.

    Figure 3-6 Master node specifications

    ----End

    3.2.5 Can I Directly Connect to the Master Node of a Cluster?CCE allows you to use kubectl to connect a cluster. For details, see Connecting toa CCE Cluster Using kubectl or web-terminal.

    However, you are not allowed to log in to the master node to perform relatedoperations.

    3.2.6 How Do I Retrieve Data After a Cluster Is Deleted?After a cluster is deleted, the workload on the cluster will also be deleted andcannot be restored. Therefore, exercise caution when deleting a cluster.

    Cloud Container EngineFAQs 3 Cluster FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 16

    https://console-intl.huaweicloud.com/cce2.0/?locale=en-ushttps://support.huaweicloud.com/intl/en-us/usermanual-cce/cce_01_0107.htmlhttps://support.huaweicloud.com/intl/en-us/usermanual-cce/cce_01_0107.html

  • 3.2.7 How Do I Update a Namespace in Terminating State inKubernetes?

    In Kubernetes, a namespace has two common states: Active and Terminating. TheTerminating state is rare. When a namespace has running resources but thenamespace is deleted, the namespace becomes Terminating. In this case, thenamespace will be automatically deleted by the system after the Kubernetesreclaims the resources in the namespace.

    However, in some cases, even if no resource is running in the namespace, thenamespace in the Terminating state still cannot be deleted.

    To solve this problem, perform the following steps:

    Step 1 View the namespace details.$ kubectl get ns | grep rdbrdbms Terminating 6d21h

    $ kubectl get ns rdbms -o yamlapiVersion: v1kind: Namespacemetadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"name":"rdbms"}} creationTimestamp: "2020-05-07T15:19:43Z" deletionTimestamp: "2020-05-07T15:33:23Z" name: rdbms resourceVersion: "84553454" selfLink: /api/v1/namespaces/rdbms uid: 457788ddf-53d7-4hde-afa3-1fertg21ewe1spec: finalizers: - kubernetesstatus: phase: Terminating

    Step 2 View resources in the namespace.# View resources that can be isolated using namespaces in the Kubernetes cluster.$ kubectl api-resources -o name --verbs=list --namespaced | xargs -n 1 kubectl get --show-kind --ignore-not-found -n rdbms

    After running the command above, no resource is occupied in the namespacerdbms.

    Step 3 Delete the namespace.

    Directly delete the namespace rdbms.$ kubectl delete ns rdbmsError from server (Conflict): Operation cannot be fulfilled on namespaces "rdbms": The system is ensuring all content is removed from this namespace. Upon completion, this namespace will automatically be purged by the system.

    The system displays a message indicating that the deletion operation will becomplete until the system deletes all useless resources.

    Step 4 Forcibly delete the namespace.$ kubectl delete ns rdbms --force --grace-period=0warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.Error from server (Conflict): Operation cannot be fulfilled on namespaces "rdbms": The system is ensuring all content is removed from this namespace. Upon completion, this namespace will automatically be purged by the system.

    Cloud Container EngineFAQs 3 Cluster FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 17

  • After running the command above, the namespace still cannot be deleted.

    Step 5 Use the native API to delete these resources.

    Obtain the namespace details.$ kubectl get ns rdbms -o json > rdbms.json

    Check the JSON configuration defined by the namespace, edit the JSON file, anddelete the spec part.

    $ cat rdbms.json{ "apiVersion": "v1", "kind": "Namespace", "metadata": { "annotations": { "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"rdbms\"}}\n" }, "creationTimestamp": "2019-10-14T12:17:44Z", "deletionTimestamp": "2019-10-14T12:30:27Z", "name": "rdbms", "resourceVersion": "8844754", "selfLink": "/api/v1/namespaces/rdbms", "uid": "29067ddf-56d7-4cce-afa3-1fbdbb221ab1" }, "spec": { "finalizers": [ "kubernetes" ] }, "status": { "phase": "Terminating" }}

    After the PUT request is executed, the namespace is automatically deleted.

    $ curl --cacert /root/ca.crt --cert /root/client.crt --key /root/client.key -k -H "Content-Type:application/json" -X PUT --data-binary @rdbms.json https://x.x.x.x:5443/api/v1/namespaces/rdbms/finalize { "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "rdbms", "selfLink": "/api/v1/namespaces/rdbms/finalize", "uid": "29067ddf-56d7-4cce-afa3-1fbdbb221ab1", "resourceVersion": "8844754", "creationTimestamp": "2019-10-14T12:17:44Z", "deletionTimestamp": "2019-10-14T12:30:27Z", "annotations": { "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"rdbms\"}}\n" } }, "spec": {

    }, "status": { "phase": "Terminating" }

    Description

    ● For details about how to obtain the cluster certificate, see Obtaining aCluster Certificate.

    ● https://x.x.x.x:5443 indicates the address for connecting to the cluster. Toobtain the address, perform the following steps:

    Cloud Container EngineFAQs 3 Cluster FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 18

    https://support.huaweicloud.com/intl/en-us/usermanual-cce/cce_01_0175.htmlhttps://support.huaweicloud.com/intl/en-us/usermanual-cce/cce_01_0175.html

  • Log in to the CCE console. In the navigation pane, choose ResourceManagement > Clusters. Click the name of the cluster to be connected andobtain the IP address and port number next to Internal API Server Addressin the Basic Information pane.

    Figure 3-7 Obtaining the access address

    Check whether the namespace has been deleted.

    $ kubectl get ns | grep rdb

    ----End

    3.2.8 Changing the Mode of the Docker Device MapperCurrently, private CCE clusters use Device Mapper as the Docker storage driver.

    Device Mapper is developed based on the kernel framework and supports manyadvanced volume management technologies on Linux.

    Docker Device Mapper storage driver leverages the thin provisioning and snapshotcapabilities of this framework to manage images and containers.

    For CCE clusters of v1.7.3-r6 or earlier, the Docker Device Mapper is set to theloop-lvm mode by default. By default, Docker generates data and metadata filesin the /var/lib/docker/devicemapper/devicemapper directory. The two files areattached to loop devices and used as block devices. After multiple containers areattached to the files, the performance deteriorates dramatically.

    The loop-lvm mode enables you to use Docker out of the box, without additionalconfiguration. This mode is not recommended in the production environment. TheDocker Device Mapper also supports the direct-lvm mode. This mode enables youto use raw partitions (no file systems). In the medium-load and high-densityenvironments, this mode provides better performance.

    To ensure system stability, you need to set the Docker Device Mapper to thedirect-lvm mode.

    CCE allows you to change the mode of the Device Mapper on VM nodes runningon EulerOS, CentOS, and SUSE.

    Cloud Container EngineFAQs 3 Cluster FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 19

    https://console-intl.huaweicloud.com/cce2.0/?locale=en-us

  • NO TICE

    ● Changing the Docker Device Mapper mode on a node requires a data disk.Therefore, in the change process, the system automatically creates a 100 GBSATA disk and binds it to the node. This data disk requires extra fees. For detailson the fee calculation method, see EVS Pricing Details.

    ● When the Docker Device Mapper mode on a node is changed to direct-lvm,the container and image data on the node will be deleted. Therefore, you mustback up the container and image data of the node to a private imagerepository or open source image repository before changing the mode.

    Procedure

    Step 1 Check whether the Docker Device Mapper mode on a node is direct-lvm.

    Method 1:

    1. Log in to a node on which you want to view the Docker Device Mapper mode.2. Enter the following command to view the configuration information under

    Storage Driver.docker info– If the values of the Data file and Metadata file parameters under

    Storage Driver are /dev/loopx, the Docker Device Mapper mode of thecurrent node is loop-lvm. Change the mode by following Step 2.Example:

    – If the values of the Data file and Metadata file parameters underStorage Driver are left blank and the value of Pool Name is vgpaas-thinpool, the Docker Device Mapper mode of the current node is direct-lvm. You do not need to change the mode.

    Cloud Container EngineFAQs 3 Cluster FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 20

    https://www.huaweicloud.com/intl/en-us/pricing/index.html?tab=detail#/evs

  • Example:

    Method 2:

    1. Log in to a node on which you want to view the Docker Device Mapper mode.

    2. Enter the following command and check whether the command outputcontains the information listed below:

    cat /etc/docker/daemon.json"dm.thinpooldev=/dev/mapper/vgpaas-thinpool"

    – If the command output contains the preceding information, the DockerDevice Mapper mode of the current node is direct-lvm. You do not needto change the mode.

    – If the command output does not contain the preceding information or amessage indicating that a file such as daemon.json is unavailable isdisplayed, the Docker Device Mapper mode of the current node is notdirect-lvm. Change the mode by following Step 2.

    Step 2 (Optional) If no elastic IP address is bound to the node for which the DockerDevice Mapper mode needs to be changed, bind an elastic IP address.

    Cloud Container EngineFAQs 3 Cluster FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 21

  • Step 3 Log in to the node with an elastic IP address as the root user.

    Step 4 Create a configuration file.

    touch config.yaml

    Step 5 Copy the following content to the configuration file:user: domainName: username: password: projectName: apiGatewayIp: iamHostname: ecsHostname: evsHostname: swrAddr: defaultPassword: defaultPrivateKey: hosts: - host: user: root password: privateKey: serverId: - host: user: root password: privateKey: serverId:

    Table 3-1 Parameter description

    Parameter Description Example

    domainName Tenant name -

    username User name -

    password User password, which isenclosed in quotation marks('' '')

    -

    projectName Name of the project towhich the to-be-configurednode belongs

    ap-southeast-1

    apiGatewayIp IP address of an APIgateway

    -

    iamHostname Endpoint of the IAM serviceQuery the endpoint throughRegions and Endpoints.

    iam.ap-southeast-1.myhuaweicloud.com

    ecsHostname Endpoint of the ECS serviceQuery the endpoint throughRegions and Endpoints.

    ecs.ap-southeast-1.myhuaweicloud.com

    Cloud Container EngineFAQs 3 Cluster FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 22

    https://developer.huaweicloud.com/intl/en-us/endpointhttps://developer.huaweicloud.com/intl/en-us/endpoint

  • Parameter Description Example

    evsHostname Endpoint of the EVS serviceQuery the endpoint throughRegions and Endpoints.

    evs.ap-southeast-1.myhuaweicloud.com

    swrAddr Address of a softwarerepository

    -

    defaultPassword (Optional) Default loginpassword of a node. Thevalue must be enclosed inquotation marks ('' '').

    -

    defaultPrivateKey (Optional) Absolute path tothe default key file forlogging in to a node. Thevalue must be enclosed inquotation marks ('' '').

    -

    hosts Host array structure [1].You can set multiple nodesfor which you want tochange the Device Mappermode. The followingparameters must beincluded: user, password/privateKey, and serverId.For details about the hostarray structure, see Table3-2.

    -

    Table 3-2 Parameter description about the host array structure

    Parameter Description Example

    host IP address of the nodefor which you want tochange the DeviceMapper mode. This nodemust be in the samesubnet as the currentlogged-in node.

    -

    user User name. Set thisparameter to root.

    -

    Cloud Container EngineFAQs 3 Cluster FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 23

    https://developer.huaweicloud.com/intl/en-us/endpoint

  • Parameter Description Example

    password Password for the rootuser on the node forwhich you want tochange the DeviceMapper mode. The valuemust be enclosed inquotation marks ('' '').NOTE

    Set either password orprivateKey.

    -

    privateKey Absolute path to the keyfile of the root user onthe node for which youwant to change theDevice Mapper mode.The value must beenclosed in quotationmarks ('' '').NOTE

    Set either password orprivateKey.

    -

    serverId ID of the ECScorresponding to thenode for which you wantto change the DeviceMapper mode

    076311b7-4c05-48f6-ba27-f0cfe29d424f

    Step 6 Modify the configuration of the nodes in the cluster.

    It takes about 3 to 5 minutes to configure a node.

    curl -k https://:20202/swr/v2/domains/op_svc_servicestage/namespaces/op_svc_servicestage/repositories/default/packages/cluster-versions/versions/base/file_paths/cceadm -1 -O;chmod u+x cceadm; ./cceadmbatch-config-docker --conf=./config.yaml

    Replace with the address of a software repository, which is thesame as the value of swrAddr in Table 3-1.

    ----End

    3.2.9 Adding a Second Data Disk to a Node in a CCE ClusterYou can use the pre-installation script feature to configure CCE cluster nodes(ECSs). For details, see Buying a Hybrid Cluster - Advanced KubernetesSettings.

    Cloud Container EngineFAQs 3 Cluster FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 24

    https://support.huaweicloud.com/intl/en-us/usermanual-cce/cce_01_0028.html#cce_01_0028__li1824844253210https://support.huaweicloud.com/intl/en-us/usermanual-cce/cce_01_0028.html#cce_01_0028__li1824844253210

  • NO TE

    ● When creating a node in a cluster of v1.13.10 or later, if a data disk is not managed byLVM, follow the instructions in this section to format the data disk before adding thedisk. Otherwise, the data disk will still be managed by LVM.

    ● When creating a node in a cluster of v1.13.10 or earlier, if a data disk is not managed byLVM, format the data disk. Otherwise, either this data disk or the first data disk will bemanaged by LVM, which is not as expected.

    Before using this feature, write a script that can format data disks and save it toyour OBS bucket. This script must be executed by user root.

    Input Parameters

    1. Set the script name to formatdisk.sh, save the script to your OBS bucket, andobtain the address of the script in OBS. For details, see Accessing an ObjectUsing Its URL.

    2. You need to specify the size of the Docker data disk (the data disk managedby LVM is called the Docker data disk). The size of the Docker disk must bedifferent from that of the second disk. For example, the Docker data disk is100 GB and the new disk is 110 GB.

    3. Set the mount path of the second data disk, for example, /data/code.

    Run the following command in the pre-installation script to format the disk:

    cd /tmp;curl -k -X GET OBS bucket address /formatdisk.sh -1 -O;fdisk -l;sleep 30;bash -x formatdisk.sh 100 /data/code;fdisk -l

    Example script (formatdisk.sh):

    dockerdisksize=$1mountdir=$2systemdisksize=40i=0while [ 20 -gt $i ]; do echo $i; if [ $(lsblk -o KNAME,TYPE | grep disk | grep -v nvme | awk '{print $1}' | awk '{ print "/dev/"$1}' |wc -l) -ge 3 ]; then break else sleep 5 fi; i=$[i+1] done all_devices=$(lsblk -o KNAME,TYPE | grep disk | grep -v nvme | awk '{print $1}' | awk '{ print "/dev/"$1}')for device in ${all_devices[@]}; do isRawDisk=$(lsblk -n $device 2>/dev/null | grep disk | wc -l) if [[ ${isRawDisk} > 0 ]]; then # is it partitioned ? match=$(lsblk -n $device 2>/dev/null | grep -v disk | wc -l) if [[ ${match} > 0 ]]; then # already partited [[ -n "${DOCKER_BLOCK_DEVICES}" ]] && echo "Raw disk ${device} has been partition, will skip this device" continue fi else isPart=$(lsblk -n $device 2>/dev/null | grep part | wc -l) if [[ ${isPart} -ne 1 ]]; then # not parted [[ -n "${DOCKER_BLOCK_DEVICES}" ]] && echo "Disk ${device} has not been partition, will skip this device" continue fi

    Cloud Container EngineFAQs 3 Cluster FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 25

    https://support.huaweicloud.com/intl/en-us/usermanual-obs/obs_03_0319.htmlhttps://support.huaweicloud.com/intl/en-us/usermanual-obs/obs_03_0319.html

  • # is used ? match=$(lsblk -n $device 2>/dev/null | grep -v part | wc -l) if [[ ${match} > 0 ]]; then # already used [[ -n "${DOCKER_BLOCK_DEVICES}" ]] && echo "Disk ${device} has been used, will skip this device" continue fi isMount=$(lsblk -n -o MOUNTPOINT $device 2>/dev/null) if [[ -n ${isMount} ]]; then # already used [[ -n "${DOCKER_BLOCK_DEVICES}" ]] && echo "Disk ${device} has been used, will skip this device" continue fi isLvm=$(sfdisk -lqL 2>>/dev/null | grep $device | grep "8e.*Linux LVM") if [[ ! -n ${isLvm} ]]; then # part system type is not Linux LVM [[ -n "${DOCKER_BLOCK_DEVICES}" ]] && echo "Disk ${device} system type is not Linux LVM, will skip this device" continue fi fi block_devices_size=$(lsblk -n -o SIZE $device 2>/dev/null | awk '{ print $1}') if [[ ${block_devices_size}"x" != "${dockerdisksize}Gx" ]] && [[ ${block_devices_size}"x" != "${systemdisksize}Gx" ]]; thenecho "np1

    w" | fdisk $device mkfs -t ext4 ${device}1 mkdir -p $mountdir echo "${device}1 $mountdir ext4 noatime 0 0" | tee -a /etc/fstab >/dev/null mount $mountdir fidone

    NO TE

    If the preceding example cannot be executed, use the dos2unix tool to convert the format.

    Cloud Container EngineFAQs 3 Cluster FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 26

  • 4 Node FAQs4.1 Node Creation

    4.1.1 What Is the Default OS of CCE Nodes?Currently, CCE nodes support EulerOS 2.2, EulerOS 2.5, CentOS 7.4, andCentOS7.6.

    4.1.2 How Do I Troubleshoot Insufficient EIPs When a Node IsAdded?

    Symptom

    When a node is added, EIP is set to Automatically assign. The node cannot becreated, and a message indicating that EIPs are insufficient is displayed.

    Figure 4-1 Purchasing an EIP

    Solution

    Two methods are available to solve the problem.

    Cloud Container EngineFAQs 4 Node FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 27

  • ● Method 1: Unbind the VMs bound with EIPs and add a node again.

    a. Log in to the management console.b. Choose Computing > Elastic Cloud Server.c. In the ECS list, locate the target ECS and click its name.d. On the ECS details page, click the EIPs tab. In the EIP list, click Unbind at

    the row of the target ECS and click Yes.

    Figure 4-2 Unbinding an EIP

    e. Return to the Buy Node page on the CCE console and click Use existingto add an EIP.

    Figure 4-3 Using an unbound EIP

    ● Method 2: Submit a service ticket.Public cloud limits the quotas of user resources, that is, the number andcapacity of resources. If the existing resource quota cannot meet your servicerequirements, you can submit a service ticket to increase your quota. Onceyour application is approved, your quota will be updated and a notificationwill be sent to you.

    4.1.3 What Should I Do If a Node Fails to Be Accepted BecauseIt Fails to Be Installed?

    SymptomA node fails to be accepted into a cluster, and an error message is displayed,indicating that the node fails to be installed.

    Possible CauseLog in to the node and check the /var/paas/sys/log/baseagent/baseagent.loginstallation log. The following error information is displayed:

    Cloud Container EngineFAQs 4 Node FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 28

    https://console-intl.huaweicloud.com/ticket/?locale=en-us#/ticketindex/createIndexhttps://console-intl.huaweicloud.com/ticket/?locale=en-us#/ticketindex/createIndex

  • Check the LVM settings of the node. It is found that the LVM logical volume is notcreated in /dev/vdb.

    Solution

    Run the following command to manually create a logical volume:

    pvcreate /dev/vdb vgcreate vgpaas /dev/vdb

    After the node is reset on the GUI, the node becomes normal.

    4.1.4 Using a Private Image to Build a Worker Node Image(OBT)

    Constraints● This function is in the OBT only in specific regions, for example, AP-Singapore.● This function is available only for clusters of v1.15 or later.

    Image OS and Kernel Version Requirements

    You have added a dedicated label to the image. Both the label key and value arecce. The image OS version must be EulerOS 2.5 or CentOS 7.6.

    Table 4-1 Mappings between clusters, OSs, and kernels

    OS Cluster Version Kernel

    CentOS Linux release7.6

    v1.17.9-r0 3.10.0-1062.12.1.el7.x86_64

    v1.15.11-r1 3.10.0-1062.12.1.el7.x86_64

    v1.15.6-r1 3.10.0-1062.1.1.el7.x86_64

    EulerOS release 2.5 v1.17.9-r0 3.10.0-862.14.1.5.h428.eulerosv2r7.x86_64

    v1.15.11-r1 3.10.0-862.14.1.5.h428.eulerosv2r7.x86_64

    v1.15.6-r1 3.10.0-862.14.1.5.h328.eulerosv2r7.x86_64

    ● When creating an image, follow the instructions in this section to preventunexpected problems.

    ● To log in to VMs created from base images, users are required to have thesudo root or root permissions.

    Preparation

    Notes

    Cloud Container EngineFAQs 4 Node FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 29

  • ● Components lvm2, conntrack, sudo, NetworkManager, and ntpd are requiredfor creating a private image. Ensure that these components have beeninstalled.

    ● Before creating an image, you need to create two ECSs and bind an EIP toeach ECS. One ECS functions as the executor, and the other functions as thehost for creating an image. It takes about 10 minutes to create an image,which generates traffic and consumes resources. An EIP is bound to remotelytransfer the installation package and send installation dependencycommands. Recommended ECS specifications: 4 vCPUs and 8 GB memory

    ● During image build, an agent will be injected. You need to check whether thecreated image is available only in the current region.

    ● After the image is created, the ECSs will not be deleted. You need to deletethem manually.

    ● The private image installation package contains the script and dependentcomponents required for installing the node. The package version variesdepending on the cluster version.

    ● Ensure that TCP port 22 is enabled in the new inbound rule of the securitygroup.

    Procedure

    Step 1 Uploading the init_envs.conf File

    The init_envs.conf file stores the configurations of the VM created from the baseimage. Apply for a server on the ECS console or use an existing server, log in theserver, and upload the init_envs.conf file to the /root directory on the server.

    The following is an example of the init_envs.conf file. Set the parameters basedon the description in Table 4-2.

    DOMAIN_NAME=''USER_NAME=''PROJECT_NAME=''PROJECT_ID=''IMS_ENDPOINT=''KEY_PAIR_NAME=''IMAGE_NAME=''

    Table 4-2 Description of the init_envs.conf file

    Parameter Description

    DOMAIN_NAME Account that creates an image.

    USER_NAME User that creates an image.

    PROJECT_NAME Region to which the project belongs.View the region and project ID on the My Credentialspage.

    PROJECT_ID Project ID.View the region and project ID on the My Credentialspage.

    Cloud Container EngineFAQs 4 Node FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 30

    https://console-intl.huaweicloud.com/iam/?locale=en-us#/myCredentialhttps://console-intl.huaweicloud.com/iam/?locale=en-us#/myCredential

  • Parameter Description

    IMS_ENDPOINT ims.region.myhuaweicloud.comFor details about regions, see Regions and Endpoints.Example value: ims.cn-north-4.myhuaweicloud.com

    KEY_PAIR_NAME (Optional) Name of the key pair, which is the same asthe name of the key pair file in the /root directory.

    IMAGE_NAME Optional. The default value is the BASIC-NODE-IMG-timestamp.

    Step 2 Obtain the key file. (Skip this step if you log in to the server using a password.)

    A key file is the authentication file required for creating an ECS. You can useexisting keys or create new keys. For example, log in to the server and upload thekey file named Keypair.pem to the /root directory to create an ECS.

    1. Log in to the HUAWEI CLOUD management console.2. Choose Service List > Computing > Elastic Cloud Server.3. In the navigation pane, choose Key Pair. On the page displayed, click Create

    Key Pair.4. Enter a key pair name and click OK.5. In the dialog box displayed, click OK.

    View and save the key pair. To ensure security, a key pair can be downloadedonly once. Keep the key pair secure for login.For details about how to create a key pair, see Creating a Key Pair.

    ----End

    Creating a Node Image

    Step 1 (Optional) Obtain a base image ID from IMS.

    NO TE

    For details about how to use an image file to generate an image ID, see Appendix.

    Step 2 Log in to the server, upload the init_envs.conf and optionally Keypair.pem files tothe /root directory, and set parameters in the init_envs.conf file.

    Step 3 Run the image creation script.

    Cloud Container EngineFAQs 4 Node FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 31

    https://developer.huaweicloud.com/intl/en-us/endpoint?IMShttps://support.huaweicloud.com/intl/en-us/usermanual-ecs/en-us_topic_0014250631.html

  • Table 4-3 Commands to be run

    Site Command

    HUAWEICLOUD

    Click here to obtain the installation package (for the Singaporeregion only).Decompress the installation package. When executing create.sh inthe node-image/conf directory, add the following five parameters.The following is an example:bash create.sh ${NODE_EIP} ${PASSWORD} ${ECS_PASSWORD} ${ECS_INSTANCE_ID} ${LINUX_ROLE}

    The parameters are described as follows:NODE_EIP=${1:-""} #EIP address of the server that creates the imagePASSWORD=${2:-""} #Password for logging in to HUAWEI CLOUD.This password is used to obtain the token and create an IMS image.ECS_PASSWORD=${3:-"} #Password for logging in to the node thatcreates the image. If the key pair mode is used, this parameter is leftblank.ECS_INSTANCE_ID=${4:-""} #Instance ID of the ECS used to createthe imageLINUX_ROLE=${5:-"root"} #The default user is root. If a non-rootuser is used, set the permission as follows:/etc/sudoersUsername ALL=(ALL) NOPASSWD: ALL

    Step 4 After the image is created, use the image ID for verification.

    ----End

    AppendixThis operation is required only when an image file is used to generate an imageID. Perform the following operations:

    Step 1 Obtain a base image file from a trusted HUAWEI CLOUD image repository.

    For details about how to obtain the image ID, see Quickly Importing an ImageFile (Windows).

    Step 2 Import the obtained image file to an OBS bucket of your account.

    Figure 4-4 Importing the image file to an OBS bucket

    Step 3 On IMS, click Create Image on the Private Images tab page. Select Image Filefor Source, which is the image file in the OBS bucket. Set the system disk to 40GB, configure other parameters as required, and click Create Now.

    Cloud Container EngineFAQs 4 Node FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 32

    https://cce-statics.ap-southeast-3.obs.ap-southeast-3.myhuaweicloud.com/package/node-image/node-image-v1.17.9-r0.tgzhttps://support.huaweicloud.com/intl/en-us/usermanual-ims/ims_01_0341.htmlhttps://support.huaweicloud.com/intl/en-us/usermanual-ims/ims_01_0341.html

  • Figure 4-5 Creating an image

    ----End

    4.2 Node Running

    4.2.1 What Should I Do If a Cluster Is Available But SomeNodes Are Unavailable?

    If the cluster status is available but some nodes in the cluster are unavailable,perform the following operations to rectify the fault:

    Troubleshooting

    Troubleshooting methods are sorted based on the occurrence probability of thepossible causes. You are advised to check the possible causes from high probabilityto low probability to quickly locate the cause of the problem.

    If the fault persists after a possible cause is rectified, check other possible causes.

    ● Check Item 1: Whether the Node Is Overloaded● Check Item 2: Whether the ECS Is Deleted or Faulty● Check Item 3: Whether You Can Log In to the ECS● Check Item 4: Whether the Security Group Is Modified● Check Item 5: Whether the Security Group Rules Contain the Security

    Group Policy for the Communication Between the Master Node and theWorker Node

    Cloud Container EngineFAQs 4 Node FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 33

  • ● Check Item 6: Whether the Disk Is Abnormal● Check Item 7: Whether Internal Components Are Normal● Check Item 8: Whether the DNS Address Is Correct● Check Item 9: Whether the vdb Disk on the Node Is Deleted

    Figure 4-6 Troubleshooting process

    Check Item 1: Whether the Node Is OverloadedSymptom

    The node connection in the cluster is abnormal. Multiple nodes report write errors,but services are not affected.

    Fault locating

    Step 1 Log in to the CCE console. In the navigation pane, choose ResourceManagement > Nodes.

    Step 2 Click the name of an unavailable node to go to the node details page.

    Step 3 On the Monitoring tab page, click View Monitoring Details to go to the AOMconsole and view historical monitoring records.

    Cloud Container EngineFAQs 4 Node FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 34

    https://console-intl.huaweicloud.com/cce2.0/?locale=en-us

  • Figure 4-7 Host Monitoring - View Monitor Graphs

    A too high CPU or memory usage of the node will result in a high network latencyor trigger system OOM. Therefore, the node is displayed as unavailable.

    ----End

    Solution

    1. You are advised to migrate services to reduce the workloads on the node andset the resource upper limit for the workloads.

    2. Clear data on the CCE nodes in the cluster.3. Add more nodes to the cluster.4. You can also restart the node on the ECS console.5. Reset the node. For details, see Resetting a Node.

    After the node becomes available, the workload is restored.

    Check Item 2: Whether the ECS Is Deleted or Faulty

    Step 1 Check whether the cluster is available.

    Log in to the CCE console, and choose Resource Management > Clusters in thenavigation pane. On the page displayed, check whether the cluster is available.

    ● If the cluster is unavailable, contact technical support by submitting a serviceticket to rectify the fault.

    ● If the cluster is available but some nodes in the cluster are unavailable, go toStep 2.

    Step 2 Log in to the ECS console. In the navigation pane, choose Elastic Cloud Server toview the ECS status.● If the ECS status is Deleted, go back to the CCE console, choose Resource

    Management > Nodes in the navigation pane, delete the correspondingnode, and then create another one.

    ● If the ECS status is Stopped or Frozen, restore the ECS first. It takes about 3minutes to restore the ECS.

    ● If the ECS status is Faulty, restart the ECS. If the ECS is still faulty, contacttechnical support by submitting a service ticket to rectify the fault.

    Cloud Container EngineFAQs 4 Node FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 35

    https://support.huaweicloud.com/intl/en-us/usermanual-cce/cce_01_0003.htmlhttps://console-intl.huaweicloud.com/cce2.0/?locale=en-ushttps://console-intl.huaweicloud.com/ticket/?locale=en-us#/ticketindex/createIndexhttps://console-intl.huaweicloud.com/ticket/?locale=en-us#/ticketindex/createIndexhttps://console-intl.huaweicloud.com/ecm/?locale=en-us#/ecs/manager/vmListhttps://console-intl.huaweicloud.com/ticket/?locale=en-us#/ticketindex/createIndex

  • ● If the ECS status is Running, log in to the ECS to locate the fault according toCheck Item 7: Whether Internal Components Are Normal.

    ----End

    Check Item 3: Whether You Can Log In to the ECS

    Step 1 Log in to the HUAWEI CLOUD management console. Choose Service List >Computing > Elastic Cloud Server.

    Step 2 In the ECS list, locate the newly created node (generally named in the format ofcluster name-random number) in the cluster and click Remote Login in theOperation column.

    Step 3 Check whether the node name displayed on the page is the same as that on theVM and whether the password or key can be used to log in to the node.

    Figure 4-8 Checking the node name displayed on the page

    Figure 4-9 Checking the node name on the VM and whether the node can belogged in to

    If the node names are inconsistent and the password and key cannot be used tolog in to the node, Cloud-Init problems occurred when an ECS was created. In thiscase, restart the node and submit a service ticket to the ECS personnel to locatethe root cause.

    ----End

    Cloud Container EngineFAQs 4 Node FAQs

    Issue 01 (2021-01-22) Copyright © Huawei Technologies Co., Ltd. 36

    https://console-intl.huaweicloud.com/ecm/?locale=en-us#/ecs/manager/vmList

  • Check Item 4: Whether the Security Group Is Modified

    Step 1 Log in to the management console, and choose Service List > Network > VirtualPrivate Cloud. In the navigation pane, choose Access Control > Security Groups,and locate the security group of the master node.

    The name of this security group is in the format of Cluster name-cce-control-ID,as shown in the following figure.

    You can search for the security group by cluster name.

    Figure 4-10 Master node in the cluster

    Step 2 Click the security group. On the details page displayed, ensure that the securitygroup rules of the master node are the same as those in the following figure.

    Figure 4-11 Viewing inbound rules of the security group

    Inbound rule parameter description:

    ● 4789: used