kOps로 AWS에 K8S 배포하기

가시다님과 함께하는 PKOS 1주차 스터디.

Kops 설치 방법(EC2 Instance)

OS
- Amazon Linux 2

 curl -LO <https://github.com/kubernetes/kops/releases/download/$>(curl -s <https://api.github.com/repos/kubernetes/kops/releases/latest> | grep tag_name | cut -d '"' -f 4)/kops-linux-amd64
chmod +x kops-linux-amd64
mv kops-linux-amd64 /usr/local/bin/kops

Route53 도메인 설정

Route53에서 서브도메인을 지원하기에 서브도메인을 이용하여 생성한다.
생성 후 나오는 NameServers의 값을 저장한다.

 aws route53 create-hosted-zone --name kops.domain.co.kr --caller-reference 1
---
{
    "Location": "<https://route53.amazonaws.com/2013-04-01/hostedzone/Z0801636DLEL2UTQFMB6>",
    "HostedZone": {
        "Id": "/hostedzone/Z0801636DLEL2UTQFMB6",
        "Name": "kops.domain.co.kr.",
        "CallerReference": "1",
        "Config": {
            "PrivateZone": false
        },
        "ResourceRecordSetCount": 2
    },
    "ChangeInfo": {
        "Id": "/change/C00464822MVNRLF9UM031",
        "Status": "PENDING",
        "SubmittedAt": "2023-03-06T03:01:44.432000+00:00"
    },
    "DelegationSet": {
        "NameServers": [
            "ns-451.awsdns-56.com",
            "ns-1671.awsdns-16.co.uk",
            "ns-1239.awsdns-26.org",
            "ns-536.awsdns-03.net"
        ]
    }
}

상위 도메인인 domain.co.kr에서 서브도메인으로 라우팅을 할 수 있도록 Record를 서브도메인 명으로 생성한다.
aws cli로 간단하게 처리를 하기 위해 아래 JSON 파일을 생성한다.

 {
  "Comment": "CREATE/DELETE/UPSERT a record ",
  "Changes": [{
    "Action": "CREATE",
    "ResourceRecordSet": {
      "Name": "kops.domain.co.kr",
      "Type": "NS",
      "TTL": 172800,
      "ResourceRecords": [
        { "Value": "ns-451.awsdns-56.com"},
        { "Value": "ns-1671.awsdns-16.co.uk"},
        { "Value": "ns-1239.awsdns-26.org"},
        { "Value": "ns-536.awsdns-03.net"}
      ]
    }}]
}

위와 같은 json 파일 생성 후 아래 명령어로 서브도메인을 생성한다.

 aws route53 change-resource-record-sets --hosted-zone-id Z14NHEMS72ESY4 --change-batch file://Record.json
---
{
    "ChangeInfo": {
        "Id": "/change/C00498933LQH5QIAAL9R",
        "Status": "PENDING",
        "SubmittedAt": "2023-03-06T03:18:17.965000+00:00",
        "Comment": "CREATE/DELETE/UPSERT a record "
    }
}

JSON 파일에 오류가 없는 한 Pending Status가 고유 ID와 함께 반환되어야 한다.
고유 ID를 포함한 아래 명령어로 Route53의 전파 상태를 확인한다. INSYNC는 변경 내용이 Route53 DNS 서버에 전파 되었음을 나타낸다.

 aws route53 get-change --id /change/C00498933LQH5QIAAL9R
---
{
    "ChangeInfo": {
        "Id": "/change/C00498933LQH5QIAAL9R",
        "Status": "INSYNC",
        "SubmittedAt": "2023-03-06T03:18:17.965000+00:00",
        "Comment": "CREATE/DELETE/UPSERT a record "
    }
}

INSYNC 상태가 보인다면 아래 명령어로 정상적으로 NameServer가 질의되는 것을 확인하면 된다.

 dig ns kops.domain.co.kr
---
; <<>> DiG 9.16.1-Ubuntu <<>> ns kops.domain.co.kr
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3571
;; flags: qr rd ad; QUERY: 1, ANSWER: 12, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
 
;; QUESTION SECTION:
;kops.domain.co.kr.             IN      NS
 
;; ANSWER SECTION:
kops.domain.co.kr.      0       IN      NS      ns-1239.awsdns-26.org.
kops.domain.co.kr.      0       IN      NS      ns-1671.awsdns-16.co.uk.
kops.domain.co.kr.      0       IN      NS      ns-451.awsdns-56.com.
kops.domain.co.kr.      0       IN      NS      ns-536.awsdns-03.net.
ns-1239.awsdns-26.org.  0       IN      A       205.251.196.215
ns-1239.awsdns-26.org.  0       IN      AAAA    2600:9000:5304:d700::1
ns-1671.awsdns-16.co.uk. 0      IN      A       205.251.198.135
ns-1671.awsdns-16.co.uk. 0      IN      AAAA    2600:9000:5306:8700::1
ns-451.awsdns-56.com.   0       IN      A       205.251.193.195
ns-451.awsdns-56.com.   0       IN      AAAA    2600:9000:5301:c300::1
ns-536.awsdns-03.net.   0       IN      A       205.251.194.24
ns-536.awsdns-03.net.   0       IN      AAAA    2600:9000:5302:1800::1
 
;; Query time: 0 msec
;; SERVER: 172.29.192.1#53(172.29.192.1)
;; WHEN: Mon Mar 06 12:33:49 KST 2023
;; MSG SIZE  rcvd: 452

클러스터 저장용 S3 버킷 생성

Kops는 클러스터 설치 이후에도 클러스터를 관리할 수 있는데, 이를 위해 클러스터의 상태나 사용하는 키 정보들을 지속적으로 추적해야 하는데 이 정보를 S3에 저장할 수 있다.
다수의 클러스터에서 동일한 S3 버킷을 사용할 수 있고, 사용자는 이 S3 버킷을 같은 클러스터를 운영하는 운영자에게 공유할 수 있다.
하지만 이 S3 버킷에 접근 가능한 사람은 모든 클러스터에 관리자 권한으로 접근이 가능하게되니, 운영팀 이외에 공유되지 않도록 해야한다.
저는 Kops 운영 인스턴스를 AWS에 생성을 해두었기에 Instance Profile로 설정하여 사용하기로 했습니다.

S3 Bucket 생성

 aws s3api create-bucket --bucket cluster.kops.domain.co.kr \\
--region ap-northeast-2 \\
--object-ownership BucketOwnerEnforced \\
--create-bucket-configuration LocationConstraint=ap-northeast-2 \\
--versioning-configuration Status=Enable

S3 Bucket Public Access Block 설정

 aws s3api put-public-access-block --bucket cluster.kops.domain.co.kr \\
--public-access-block-configuration \\
BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

S3 Bucket Policy 설정

KopsS3-Bucket-Policy.json

 {
  "Id": "Policy1678159299540",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1678159297382",
      "Action": [
        "s3:*",
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::cluster.kops.domain.co.kr/*",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123456789012:role/domain-SKR-KOPS-role"
        ]
      }
    }
  ]
}

Bucket Policy 설정

aws s3api put-bucket-policy --bucket cluster.kops.domain.co.kr --policy file://KopsS3-Bucket-Policy.json

EC2 Instance Profile 설정

IAM Instance Profile Trust 파일 생성

KopsS3-EC2-Trust.json

 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

IAM Instance Profile Permission 파일 생성

KopsS3-EC2-Permission.json

 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "EventBridgeActions",
            "Effect": "Allow",
            "Action": [
                "events:*",
                "schemas:*",
                "scheduler:*",
                "pipes:*"
            ],
            "Resource": "*"
        },
        {
            "Sid": "IAMCreateServiceLinkedRoleForApiDestinations",
            "Effect": "Allow",
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "arn:aws:iam::*:role/aws-service-role/AmazonEventBridgeApiDestinationsServiceRolePolicy",
            "Condition": {
                "StringEquals": {
                    "iam:AWSServiceName": "apidestinations.events.amazonaws.com"
                }
            }
        },
        {
            "Sid": "IAMCreateServiceLinkedRoleForAmazonEventBridgeSchemas",
            "Effect": "Allow",
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "arn:aws:iam::*:role/aws-service-role/schemas.amazonaws.com/AWSServiceRoleForSchemas",
            "Condition": {
                "StringEquals": {
                    "iam:AWSServiceName": "schemas.amazonaws.com"
                }
            }
        },
        {
            "Sid": "SecretsManagerAccessForApiDestinations",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:CreateSecret",
                "secretsmanager:UpdateSecret",
                "secretsmanager:DeleteSecret",
                "secretsmanager:GetSecretValue",
                "secretsmanager:PutSecretValue"
            ],
            "Resource": "arn:aws:secretsmanager:*:*:secret:events!*"
        },
        {
            "Sid": "IAMPassRoleAccessForEventBridge",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::*:role/*",
            "Condition": {
                "StringLike": {
                    "iam:PassedToService": "events.amazonaws.com"
                }
            }
        },
        {
            "Sid": "IAMPassRoleAccessForScheduler",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::*:role/*",
            "Condition": {
                "StringLike": {
                    "iam:PassedToService": "scheduler.amazonaws.com"
                }
            }
        },
        {
            "Sid": "IAMPassRoleAccessForPipes",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::*:role/*",
            "Condition": {
                "StringLike": {
                    "iam:PassedToService": "pipes.amazonaws.com"
                }
            }
        },
        {
            "Action": [
                "sqs:*"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AcceptVpcPeeringConnection",
                "ec2:AcceptVpcEndpointConnections",
                "ec2:AllocateAddress",
                "ec2:AssignIpv6Addresses",
                "ec2:AssignPrivateIpAddresses",
                "ec2:AssociateAddress",
                "ec2:AssociateDhcpOptions",
                "ec2:AssociateRouteTable",
                "ec2:AssociateSubnetCidrBlock",
                "ec2:AssociateVpcCidrBlock",
                "ec2:AttachClassicLinkVpc",
                "ec2:AttachInternetGateway",
                "ec2:AttachNetworkInterface",
                "ec2:AttachVpnGateway",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CreateCarrierGateway",
                "ec2:CreateCustomerGateway",
                "ec2:CreateDefaultSubnet",
                "ec2:CreateDefaultVpc",
                "ec2:CreateDhcpOptions",
                "ec2:CreateEgressOnlyInternetGateway",
                "ec2:CreateFlowLogs",
                "ec2:CreateInternetGateway",
                "ec2:CreateLocalGatewayRouteTableVpcAssociation",
                "ec2:CreateNatGateway",
                "ec2:CreateNetworkAcl",
                "ec2:CreateNetworkAclEntry",
                "ec2:CreateNetworkInterface",
                "ec2:CreateNetworkInterfacePermission",
                "ec2:CreateRoute",
                "ec2:CreateRouteTable",
                "ec2:CreateSecurityGroup",
                "ec2:CreateSubnet",
                "ec2:CreateTags",
                "ec2:CreateVpc",
                "ec2:CreateVpcEndpoint",
                "ec2:CreateVpcEndpointConnectionNotification",
                "ec2:CreateVpcEndpointServiceConfiguration",
                "ec2:CreateVpcPeeringConnection",
                "ec2:CreateVpnConnection",
                "ec2:CreateVpnConnectionRoute",
                "ec2:CreateVpnGateway",
                "ec2:DeleteCarrierGateway",
                "ec2:DeleteCustomerGateway",
                "ec2:DeleteDhcpOptions",
                "ec2:DeleteEgressOnlyInternetGateway",
                "ec2:DeleteFlowLogs",
                "ec2:DeleteInternetGateway",
                "ec2:DeleteLocalGatewayRouteTableVpcAssociation",
                "ec2:DeleteNatGateway",
                "ec2:DeleteNetworkAcl",
                "ec2:DeleteNetworkAclEntry",
                "ec2:DeleteNetworkInterface",
                "ec2:DeleteNetworkInterfacePermission",
                "ec2:DeleteRoute",
                "ec2:DeleteRouteTable",
                "ec2:DeleteSecurityGroup",
                "ec2:DeleteSubnet",
                "ec2:DeleteTags",
                "ec2:DeleteVpc",
                "ec2:DeleteVpcEndpoints",
                "ec2:DeleteVpcEndpointConnectionNotifications",
                "ec2:DeleteVpcEndpointServiceConfigurations",
                "ec2:DeleteVpcPeeringConnection",
                "ec2:DeleteVpnConnection",
                "ec2:DeleteVpnConnectionRoute",
                "ec2:DeleteVpnGateway",
                "ec2:DescribeAccountAttributes",
                "ec2:DescribeAddresses",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeCarrierGateways",
                "ec2:DescribeClassicLinkInstances",
                "ec2:DescribeCustomerGateways",
                "ec2:DescribeDhcpOptions",
                "ec2:DescribeEgressOnlyInternetGateways",
                "ec2:DescribeFlowLogs",
                "ec2:DescribeInstances",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeIpv6Pools",
                "ec2:DescribeLocalGatewayRouteTables",
                "ec2:DescribeLocalGatewayRouteTableVpcAssociations",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeMovingAddresses",
                "ec2:DescribeNatGateways",
                "ec2:DescribeNetworkAcls",
                "ec2:DescribeNetworkInterfaceAttribute",
                "ec2:DescribeNetworkInterfacePermissions",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribePrefixLists",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSecurityGroupReferences",
                "ec2:DescribeSecurityGroupRules",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeStaleSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeTags",
                "ec2:DescribeVpcAttribute",
                "ec2:DescribeVpcClassicLink",
                "ec2:DescribeVpcClassicLinkDnsSupport",
                "ec2:DescribeVpcEndpointConnectionNotifications",
                "ec2:DescribeVpcEndpointConnections",
                "ec2:DescribeVpcEndpoints",
                "ec2:DescribeVpcEndpointServiceConfigurations",
                "ec2:DescribeVpcEndpointServicePermissions",
                "ec2:DescribeVpcEndpointServices",
                "ec2:DescribeVpcPeeringConnections",
                "ec2:DescribeVpcs",
                "ec2:DescribeVpnConnections",
                "ec2:DescribeVpnGateways",
                "ec2:DetachClassicLinkVpc",
                "ec2:DetachInternetGateway",
                "ec2:DetachNetworkInterface",
                "ec2:DetachVpnGateway",
                "ec2:DisableVgwRoutePropagation",
                "ec2:DisableVpcClassicLink",
                "ec2:DisableVpcClassicLinkDnsSupport",
                "ec2:DisassociateAddress",
                "ec2:DisassociateRouteTable",
                "ec2:DisassociateSubnetCidrBlock",
                "ec2:DisassociateVpcCidrBlock",
                "ec2:EnableVgwRoutePropagation",
                "ec2:EnableVpcClassicLink",
                "ec2:EnableVpcClassicLinkDnsSupport",
                "ec2:ModifyNetworkInterfaceAttribute",
                "ec2:ModifySecurityGroupRules",
                "ec2:ModifySubnetAttribute",
                "ec2:ModifyVpcAttribute",
                "ec2:ModifyVpcEndpoint",
                "ec2:ModifyVpcEndpointConnectionNotification",
                "ec2:ModifyVpcEndpointServiceConfiguration",
                "ec2:ModifyVpcEndpointServicePermissions",
                "ec2:ModifyVpcPeeringConnectionOptions",
                "ec2:ModifyVpcTenancy",
                "ec2:MoveAddressToVpc",
                "ec2:RejectVpcEndpointConnections",
                "ec2:RejectVpcPeeringConnection",
                "ec2:ReleaseAddress",
                "ec2:ReplaceNetworkAclAssociation",
                "ec2:ReplaceNetworkAclEntry",
                "ec2:ReplaceRoute",
                "ec2:ReplaceRouteTableAssociation",
                "ec2:ResetNetworkInterfaceAttribute",
                "ec2:RestoreAddressToClassic",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:UnassignIpv6Addresses",
                "ec2:UnassignPrivateIpAddresses",
                "ec2:UpdateSecurityGroupRuleDescriptionsEgress",
                "ec2:UpdateSecurityGroupRuleDescriptionsIngress"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:*",
                "organizations:DescribeAccount",
                "organizations:DescribeOrganization",
                "organizations:DescribeOrganizationalUnit",
                "organizations:DescribePolicy",
                "organizations:ListChildren",
                "organizations:ListParents",
                "organizations:ListPoliciesForTarget",
                "organizations:ListRoots",
                "organizations:ListPolicies",
                "organizations:ListTargetsForPolicy"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*",
                "s3-object-lambda:*"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:*",
                "route53domains:*",
                "cloudfront:ListDistributions",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticbeanstalk:DescribeEnvironments",
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:GetBucketWebsite",
                "ec2:DescribeVpcs",
                "ec2:DescribeVpcEndpoints",
                "ec2:DescribeRegions",
                "sns:ListTopics",
                "sns:ListSubscriptionsByTopic",
                "cloudwatch:DescribeAlarms",
                "cloudwatch:GetMetricStatistics"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "apigateway:GET",
            "Resource": "arn:aws:apigateway:*::/domainnames"
        },
        {
            "Action": "ec2:*",
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "elasticloadbalancing:*",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "cloudwatch:*",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "autoscaling:*",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:AWSServiceName": [
                        "autoscaling.amazonaws.com",
                        "ec2scheduled.amazonaws.com",
                        "elasticloadbalancing.amazonaws.com",
                        "spot.amazonaws.com",
                        "spotfleet.amazonaws.com",
                        "transitgateway.amazonaws.com"
                    ]
                }
            }
        }
    ]
}

IAM Role 및 Instnace Profile 생성

 aws iam create-role --role-name domain-SKR-KOPS-role --assume-role-policy-document file://KopsS3-EC2-Trust.json
aws iam put-role-policy --role-name domain-SKR-KOPS-role --policy-name domain-SKR-KOPS-policy --policy-document file://KopsS3-EC2-Permission.json
aws iam create-instance-profile --instance-profile-name domain-SKR-KOPS-instance-profile
aws iam add-role-to-instance-profile --instance-profile-name domain-SKR-KOPS-instance-profile --role-name domain-SKR-KOPS-role

EC2 Instance에 Instance Profile 연결

 # Instance ID를 meta-data로 확인
curl http://169.254.169.254/latest/meta-data/instance-id
# InstanceID를 갖고 Role 설정
aws ec2 associate-iam-instance-profile --instance-id i-0f5816da773b2efeb --iam-instance-profile Name=domain-SKR-KOPS-instance-profile

K8S Cluster 생성

kubectl 설치(EC2 Instance)

 curl -LO "<https://dl.k8s.io/release/$>(curl -L -s <https://dl.k8s.io/release/stable.txt>)/bin/linux/amd64/kubectl"
curl -LO "<https://dl.k8s.io/$>(curl -L -s <https://dl.k8s.io/release/stable.txt>)/bin/linux/amd64/kubectl.sha256"
echo "$(cat kubectl.sha256)  kubectl" | sha256sum --check
install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client --output=yaml
rm -rf ./kubectl*

ssh key 생성

 ssh-keygen
---
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:0yGvi5rMwH77upu8l6tk3r66iD/ImnqcnOS72hlxwCo root@SKRI-Kops-01
The key's randomart image is:
+---[RSA 2048]----+
|                 |
| .               |
|  o     . .      |
| . .     + .     |
|E . .   S o      |
|...o     o       |
|.*+oo  ..        |
|.*O%.+o. .       |
|O+O=^#Bo.        |
+----[SHA256]-----+

환경 변수 설정

cluster 이름(이전에 생성한 kops.domain.co.kr 도메인)과 s3명을 입력한다.

 echo export KOPS_CLUSTER_NAME=kops.domain.co.kr >> ~/.bashrc
echo export KOPS_STATE_STORE=s3://cluster.kops.domain.co.kr >> ~/.bashrc
source ~/.bashrc

클러스터 생성

 kops create cluster \\
    --name=${KOPS_CLUSTER_NAME} \\
    --cloud=aws \\
    --networking amazonvpc \\
    --zones=ap-northeast-2a,ap-northeast-2c \\
    --network-cidr 172.30.0.0/16 \\
    --master-size t3.medium \\
    --node-size t3.medium \\
    --node-count=2 \\
    --kubernetes-version "1.24.10" \\
    --ssh-public-key ~/.ssh/id_rsa.pub -y

생성된 클러스터 확인

 source <(kubectl completion bash)
echo 'source <(kubectl completion bash)' >> ~/.bashrc
echo 'alias k=kubectl' >> ~/.bashrc
echo 'complete -F __start_kubectl k' >> ~/.bashrc
source ~/.bashrc
k get no
---
NAME                  STATUS   ROLES           AGE   VERSION
i-025ff97f4a34dd7fa   Ready    node            19m   v1.24.10
i-0c69a38808d05a0d7   Ready    control-plane   20m   v1.24.10
i-0e37ceecc596e40da   Ready    node            19m   v1.24.10

블로그의 정보

나의 삽질저장소

softPine

내 블로그 - 관리자 홈 전환	`Q` `Q`
새 글 쓰기	`W` `W`

글 수정 (권한 있는 경우)	`E` `E`
댓글 영역으로 이동	`C` `C`

이 페이지의 URL 복사	`S` `S`
맨 위로 이동	`T` `T`
티스토리 홈 이동	`H` `H`
단축키 안내	`Shift` + `/` `⇧` + `/`

티스토리툴바

개인정보

단축키

내 블로그

블로그 게시글

모든 영역

	curl -LO <https://github.com/kubernetes/kops/releases/download/$>(curl -s <https://api.github.com/repos/kubernetes/kops/releases/latest> \| grep tag_name \| cut -d '"' -f 4)/kops-linux-amd64
	chmod +x kops-linux-amd64
	mv kops-linux-amd64 /usr/local/bin/kops

	aws route53 create-hosted-zone --name kops.domain.co.kr --caller-reference 1
	---
	{
	"Location": "<https://route53.amazonaws.com/2013-04-01/hostedzone/Z0801636DLEL2UTQFMB6>",
	"HostedZone": {
	"Id": "/hostedzone/Z0801636DLEL2UTQFMB6",
	"Name": "kops.domain.co.kr.",
	"CallerReference": "1",
	"Config": {
	"PrivateZone": false
	},
	"ResourceRecordSetCount": 2
	},
	"ChangeInfo": {
	"Id": "/change/C00464822MVNRLF9UM031",
	"Status": "PENDING",
	"SubmittedAt": "2023-03-06T03:01:44.432000+00:00"
	},
	"DelegationSet": {
	"NameServers": [
	"ns-451.awsdns-56.com",
	"ns-1671.awsdns-16.co.uk",
	"ns-1239.awsdns-26.org",
	"ns-536.awsdns-03.net"
	]
	}
	}

	{
	"Comment": "CREATE/DELETE/UPSERT a record ",
	"Changes": [{
	"Action": "CREATE",
	"ResourceRecordSet": {
	"Name": "kops.domain.co.kr",
	"Type": "NS",
	"TTL": 172800,
	"ResourceRecords": [
	{ "Value": "ns-451.awsdns-56.com"},
	{ "Value": "ns-1671.awsdns-16.co.uk"},
	{ "Value": "ns-1239.awsdns-26.org"},
	{ "Value": "ns-536.awsdns-03.net"}
	]
	}}]
	}

	aws route53 change-resource-record-sets --hosted-zone-id Z14NHEMS72ESY4 --change-batch file://Record.json
	---
	{
	"ChangeInfo": {
	"Id": "/change/C00498933LQH5QIAAL9R",
	"Status": "PENDING",
	"SubmittedAt": "2023-03-06T03:18:17.965000+00:00",
	"Comment": "CREATE/DELETE/UPSERT a record "
	}
	}

	aws route53 get-change --id /change/C00498933LQH5QIAAL9R
	---
	{
	"ChangeInfo": {
	"Id": "/change/C00498933LQH5QIAAL9R",
	"Status": "INSYNC",
	"SubmittedAt": "2023-03-06T03:18:17.965000+00:00",
	"Comment": "CREATE/DELETE/UPSERT a record "
	}
	}

	dig ns kops.domain.co.kr
	---
	; <<>> DiG 9.16.1-Ubuntu <<>> ns kops.domain.co.kr
	;; global options: +cmd
	;; Got answer:
	;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3571
	;; flags: qr rd ad; QUERY: 1, ANSWER: 12, AUTHORITY: 0, ADDITIONAL: 0
	;; WARNING: recursion requested but not available

	;; QUESTION SECTION:
	;kops.domain.co.kr. IN NS

	;; ANSWER SECTION:
	kops.domain.co.kr. 0 IN NS ns-1239.awsdns-26.org.
	kops.domain.co.kr. 0 IN NS ns-1671.awsdns-16.co.uk.
	kops.domain.co.kr. 0 IN NS ns-451.awsdns-56.com.
	kops.domain.co.kr. 0 IN NS ns-536.awsdns-03.net.
	ns-1239.awsdns-26.org. 0 IN A 205.251.196.215
	ns-1239.awsdns-26.org. 0 IN AAAA 2600:9000:5304:d700::1
	ns-1671.awsdns-16.co.uk. 0 IN A 205.251.198.135
	ns-1671.awsdns-16.co.uk. 0 IN AAAA 2600:9000:5306:8700::1
	ns-451.awsdns-56.com. 0 IN A 205.251.193.195
	ns-451.awsdns-56.com. 0 IN AAAA 2600:9000:5301:c300::1
	ns-536.awsdns-03.net. 0 IN A 205.251.194.24
	ns-536.awsdns-03.net. 0 IN AAAA 2600:9000:5302:1800::1

	;; Query time: 0 msec
	;; SERVER: 172.29.192.1#53(172.29.192.1)
	;; WHEN: Mon Mar 06 12:33:49 KST 2023
	;; MSG SIZE rcvd: 452

	aws s3api create-bucket --bucket cluster.kops.domain.co.kr \\
	--region ap-northeast-2 \\
	--object-ownership BucketOwnerEnforced \\
	--create-bucket-configuration LocationConstraint=ap-northeast-2 \\
	--versioning-configuration Status=Enable

	aws s3api put-public-access-block --bucket cluster.kops.domain.co.kr \\
	--public-access-block-configuration \\
	BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

	{
	"Id": "Policy1678159299540",
	"Version": "2012-10-17",
	"Statement": [
	{
	"Sid": "Stmt1678159297382",
	"Action": [
	"s3:*",
	],
	"Effect": "Allow",
	"Resource": "arn:aws:s3:::cluster.kops.domain.co.kr/*",
	"Principal": {
	"AWS": [
	"arn:aws:iam::123456789012:role/domain-SKR-KOPS-role"
	]
	}
	}
	]
	}

	{
	"Version": "2012-10-17",
	"Statement": [
	{
	"Sid": "",
	"Effect": "Allow",
	"Principal": {
	"Service": "ec2.amazonaws.com"
	},
	"Action": "sts:AssumeRole"
	}
	]
	}

	{
	"Version": "2012-10-17",
	"Statement": [
	{
	"Sid": "EventBridgeActions",
	"Effect": "Allow",
	"Action": [
	"events:*",
	"schemas:*",
	"scheduler:*",
	"pipes:*"
	],
	"Resource": "*"
	},
	{
	"Sid": "IAMCreateServiceLinkedRoleForApiDestinations",
	"Effect": "Allow",
	"Action": "iam:CreateServiceLinkedRole",
	"Resource": "arn:aws:iam::*:role/aws-service-role/AmazonEventBridgeApiDestinationsServiceRolePolicy",
	"Condition": {
	"StringEquals": {
	"iam:AWSServiceName": "apidestinations.events.amazonaws.com"
	}
	}
	},
	{
	"Sid": "IAMCreateServiceLinkedRoleForAmazonEventBridgeSchemas",
	"Effect": "Allow",
	"Action": "iam:CreateServiceLinkedRole",
	"Resource": "arn:aws:iam::*:role/aws-service-role/schemas.amazonaws.com/AWSServiceRoleForSchemas",
	"Condition": {
	"StringEquals": {
	"iam:AWSServiceName": "schemas.amazonaws.com"
	}
	}
	},
	{
	"Sid": "SecretsManagerAccessForApiDestinations",
	"Effect": "Allow",
	"Action": [
	"secretsmanager:CreateSecret",
	"secretsmanager:UpdateSecret",
	"secretsmanager:DeleteSecret",
	"secretsmanager:GetSecretValue",
	"secretsmanager:PutSecretValue"
	],
	"Resource": "arn:aws:secretsmanager:::secret:events!*"
	},
	{
	"Sid": "IAMPassRoleAccessForEventBridge",
	"Effect": "Allow",
	"Action": "iam:PassRole",
	"Resource": "arn:aws:iam:::role/",
	"Condition": {
	"StringLike": {
	"iam:PassedToService": "events.amazonaws.com"
	}
	}
	},
	{
	"Sid": "IAMPassRoleAccessForScheduler",
	"Effect": "Allow",
	"Action": "iam:PassRole",
	"Resource": "arn:aws:iam:::role/",
	"Condition": {
	"StringLike": {
	"iam:PassedToService": "scheduler.amazonaws.com"
	}
	}
	},
	{
	"Sid": "IAMPassRoleAccessForPipes",
	"Effect": "Allow",
	"Action": "iam:PassRole",
	"Resource": "arn:aws:iam:::role/",
	"Condition": {
	"StringLike": {
	"iam:PassedToService": "pipes.amazonaws.com"
	}
	}
	},
	{
	"Action": [
	"sqs:*"
	],
	"Effect": "Allow",
	"Resource": "*"
	},
	{
	"Effect": "Allow",
	"Action": [
	"ec2:AcceptVpcPeeringConnection",
	"ec2:AcceptVpcEndpointConnections",
	"ec2:AllocateAddress",
	"ec2:AssignIpv6Addresses",
	"ec2:AssignPrivateIpAddresses",
	"ec2:AssociateAddress",
	"ec2:AssociateDhcpOptions",
	"ec2:AssociateRouteTable",
	"ec2:AssociateSubnetCidrBlock",
	"ec2:AssociateVpcCidrBlock",
	"ec2:AttachClassicLinkVpc",
	"ec2:AttachInternetGateway",
	"ec2:AttachNetworkInterface",
	"ec2:AttachVpnGateway",
	"ec2:AuthorizeSecurityGroupEgress",
	"ec2:AuthorizeSecurityGroupIngress",
	"ec2:CreateCarrierGateway",
	"ec2:CreateCustomerGateway",
	"ec2:CreateDefaultSubnet",
	"ec2:CreateDefaultVpc",
	"ec2:CreateDhcpOptions",
	"ec2:CreateEgressOnlyInternetGateway",
	"ec2:CreateFlowLogs",
	"ec2:CreateInternetGateway",
	"ec2:CreateLocalGatewayRouteTableVpcAssociation",
	"ec2:CreateNatGateway",
	"ec2:CreateNetworkAcl",
	"ec2:CreateNetworkAclEntry",
	"ec2:CreateNetworkInterface",
	"ec2:CreateNetworkInterfacePermission",
	"ec2:CreateRoute",
	"ec2:CreateRouteTable",
	"ec2:CreateSecurityGroup",
	"ec2:CreateSubnet",
	"ec2:CreateTags",
	"ec2:CreateVpc",
	"ec2:CreateVpcEndpoint",
	"ec2:CreateVpcEndpointConnectionNotification",
	"ec2:CreateVpcEndpointServiceConfiguration",
	"ec2:CreateVpcPeeringConnection",
	"ec2:CreateVpnConnection",
	"ec2:CreateVpnConnectionRoute",
	"ec2:CreateVpnGateway",
	"ec2:DeleteCarrierGateway",
	"ec2:DeleteCustomerGateway",
	"ec2:DeleteDhcpOptions",
	"ec2:DeleteEgressOnlyInternetGateway",
	"ec2:DeleteFlowLogs",
	"ec2:DeleteInternetGateway",
	"ec2:DeleteLocalGatewayRouteTableVpcAssociation",
	"ec2:DeleteNatGateway",
	"ec2:DeleteNetworkAcl",
	"ec2:DeleteNetworkAclEntry",
	"ec2:DeleteNetworkInterface",
	"ec2:DeleteNetworkInterfacePermission",
	"ec2:DeleteRoute",
	"ec2:DeleteRouteTable",
	"ec2:DeleteSecurityGroup",
	"ec2:DeleteSubnet",
	"ec2:DeleteTags",
	"ec2:DeleteVpc",
	"ec2:DeleteVpcEndpoints",
	"ec2:DeleteVpcEndpointConnectionNotifications",
	"ec2:DeleteVpcEndpointServiceConfigurations",
	"ec2:DeleteVpcPeeringConnection",
	"ec2:DeleteVpnConnection",
	"ec2:DeleteVpnConnectionRoute",
	"ec2:DeleteVpnGateway",
	"ec2:DescribeAccountAttributes",
	"ec2:DescribeAddresses",
	"ec2:DescribeAvailabilityZones",
	"ec2:DescribeCarrierGateways",
	"ec2:DescribeClassicLinkInstances",
	"ec2:DescribeCustomerGateways",
	"ec2:DescribeDhcpOptions",
	"ec2:DescribeEgressOnlyInternetGateways",
	"ec2:DescribeFlowLogs",
	"ec2:DescribeInstances",
	"ec2:DescribeInternetGateways",
	"ec2:DescribeIpv6Pools",
	"ec2:DescribeLocalGatewayRouteTables",
	"ec2:DescribeLocalGatewayRouteTableVpcAssociations",
	"ec2:DescribeKeyPairs",
	"ec2:DescribeMovingAddresses",
	"ec2:DescribeNatGateways",
	"ec2:DescribeNetworkAcls",
	"ec2:DescribeNetworkInterfaceAttribute",
	"ec2:DescribeNetworkInterfacePermissions",
	"ec2:DescribeNetworkInterfaces",
	"ec2:DescribePrefixLists",
	"ec2:DescribeRouteTables",
	"ec2:DescribeSecurityGroupReferences",
	"ec2:DescribeSecurityGroupRules",
	"ec2:DescribeSecurityGroups",
	"ec2:DescribeStaleSecurityGroups",
	"ec2:DescribeSubnets",
	"ec2:DescribeTags",
	"ec2:DescribeVpcAttribute",
	"ec2:DescribeVpcClassicLink",
	"ec2:DescribeVpcClassicLinkDnsSupport",
	"ec2:DescribeVpcEndpointConnectionNotifications",
	"ec2:DescribeVpcEndpointConnections",
	"ec2:DescribeVpcEndpoints",
	"ec2:DescribeVpcEndpointServiceConfigurations",
	"ec2:DescribeVpcEndpointServicePermissions",
	"ec2:DescribeVpcEndpointServices",
	"ec2:DescribeVpcPeeringConnections",
	"ec2:DescribeVpcs",
	"ec2:DescribeVpnConnections",
	"ec2:DescribeVpnGateways",
	"ec2:DetachClassicLinkVpc",
	"ec2:DetachInternetGateway",
	"ec2:DetachNetworkInterface",
	"ec2:DetachVpnGateway",
	"ec2:DisableVgwRoutePropagation",
	"ec2:DisableVpcClassicLink",
	"ec2:DisableVpcClassicLinkDnsSupport",
	"ec2:DisassociateAddress",
	"ec2:DisassociateRouteTable",
	"ec2:DisassociateSubnetCidrBlock",
	"ec2:DisassociateVpcCidrBlock",
	"ec2:EnableVgwRoutePropagation",
	"ec2:EnableVpcClassicLink",
	"ec2:EnableVpcClassicLinkDnsSupport",
	"ec2:ModifyNetworkInterfaceAttribute",
	"ec2:ModifySecurityGroupRules",
	"ec2:ModifySubnetAttribute",
	"ec2:ModifyVpcAttribute",
	"ec2:ModifyVpcEndpoint",
	"ec2:ModifyVpcEndpointConnectionNotification",
	"ec2:ModifyVpcEndpointServiceConfiguration",
	"ec2:ModifyVpcEndpointServicePermissions",
	"ec2:ModifyVpcPeeringConnectionOptions",
	"ec2:ModifyVpcTenancy",
	"ec2:MoveAddressToVpc",
	"ec2:RejectVpcEndpointConnections",
	"ec2:RejectVpcPeeringConnection",
	"ec2:ReleaseAddress",
	"ec2:ReplaceNetworkAclAssociation",
	"ec2:ReplaceNetworkAclEntry",
	"ec2:ReplaceRoute",
	"ec2:ReplaceRouteTableAssociation",
	"ec2:ResetNetworkInterfaceAttribute",
	"ec2:RestoreAddressToClassic",
	"ec2:RevokeSecurityGroupEgress",
	"ec2:RevokeSecurityGroupIngress",
	"ec2:UnassignIpv6Addresses",
	"ec2:UnassignPrivateIpAddresses",
	"ec2:UpdateSecurityGroupRuleDescriptionsEgress",
	"ec2:UpdateSecurityGroupRuleDescriptionsIngress"
	],
	"Resource": "*"
	},
	{
	"Effect": "Allow",
	"Action": [
	"iam:*",
	"organizations:DescribeAccount",
	"organizations:DescribeOrganization",
	"organizations:DescribeOrganizationalUnit",
	"organizations:DescribePolicy",
	"organizations:ListChildren",
	"organizations:ListParents",
	"organizations:ListPoliciesForTarget",
	"organizations:ListRoots",
	"organizations:ListPolicies",
	"organizations:ListTargetsForPolicy"
	],
	"Resource": "*"
	},
	{
	"Effect": "Allow",
	"Action": [
	"s3:*",
	"s3-object-lambda:*"
	],
	"Resource": "*"
	},
	{
	"Effect": "Allow",
	"Action": [
	"route53:*",
	"route53domains:*",
	"cloudfront:ListDistributions",
	"elasticloadbalancing:DescribeLoadBalancers",
	"elasticbeanstalk:DescribeEnvironments",
	"s3:ListBucket",
	"s3:GetBucketLocation",
	"s3:GetBucketWebsite",
	"ec2:DescribeVpcs",
	"ec2:DescribeVpcEndpoints",
	"ec2:DescribeRegions",
	"sns:ListTopics",
	"sns:ListSubscriptionsByTopic",
	"cloudwatch:DescribeAlarms",
	"cloudwatch:GetMetricStatistics"
	],
	"Resource": "*"
	},
	{
	"Effect": "Allow",
	"Action": "apigateway:GET",
	"Resource": "arn:aws:apigateway:*::/domainnames"
	},
	{
	"Action": "ec2:*",
	"Effect": "Allow",
	"Resource": "*"
	},
	{
	"Effect": "Allow",
	"Action": "elasticloadbalancing:*",
	"Resource": "*"
	},
	{
	"Effect": "Allow",
	"Action": "cloudwatch:*",
	"Resource": "*"
	},
	{
	"Effect": "Allow",
	"Action": "autoscaling:*",
	"Resource": "*"
	},
	{
	"Effect": "Allow",
	"Action": "iam:CreateServiceLinkedRole",
	"Resource": "*",
	"Condition": {
	"StringEquals": {
	"iam:AWSServiceName": [
	"autoscaling.amazonaws.com",
	"ec2scheduled.amazonaws.com",
	"elasticloadbalancing.amazonaws.com",
	"spot.amazonaws.com",
	"spotfleet.amazonaws.com",
	"transitgateway.amazonaws.com"
	]
	}
	}
	}
	]
	}

	aws iam create-role --role-name domain-SKR-KOPS-role --assume-role-policy-document file://KopsS3-EC2-Trust.json
	aws iam put-role-policy --role-name domain-SKR-KOPS-role --policy-name domain-SKR-KOPS-policy --policy-document file://KopsS3-EC2-Permission.json
	aws iam create-instance-profile --instance-profile-name domain-SKR-KOPS-instance-profile
	aws iam add-role-to-instance-profile --instance-profile-name domain-SKR-KOPS-instance-profile --role-name domain-SKR-KOPS-role

	# Instance ID를 meta-data로 확인
	curl http://169.254.169.254/latest/meta-data/instance-id
	# InstanceID를 갖고 Role 설정
	aws ec2 associate-iam-instance-profile --instance-id i-0f5816da773b2efeb --iam-instance-profile Name=domain-SKR-KOPS-instance-profile

	curl -LO "<https://dl.k8s.io/release/$>(curl -L -s <https://dl.k8s.io/release/stable.txt>)/bin/linux/amd64/kubectl"
	curl -LO "<https://dl.k8s.io/$>(curl -L -s <https://dl.k8s.io/release/stable.txt>)/bin/linux/amd64/kubectl.sha256"
	echo "$(cat kubectl.sha256) kubectl" \| sha256sum --check
	install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
	kubectl version --client --output=yaml
	rm -rf ./kubectl*

	ssh-keygen
	---
	Generating public/private rsa key pair.
	Enter file in which to save the key (/root/.ssh/id_rsa):
	Enter passphrase (empty for no passphrase):
	Enter same passphrase again:
	Your identification has been saved in /root/.ssh/id_rsa.
	Your public key has been saved in /root/.ssh/id_rsa.pub.
	The key fingerprint is:
	SHA256:0yGvi5rMwH77upu8l6tk3r66iD/ImnqcnOS72hlxwCo root@SKRI-Kops-01
	The key's randomart image is:
	+---[RSA 2048]----+
	\| \|
	\| . \|
	\| o . . \|
	\| . . + . \|
	\|E . . S o \|
	\|...o o \|
	\|.*+oo .. \|
	\|.*O%.+o. . \|
	\|O+O=^#Bo. \|
	+----[SHA256]-----+

	echo export KOPS_CLUSTER_NAME=kops.domain.co.kr >> ~/.bashrc
	echo export KOPS_STATE_STORE=s3://cluster.kops.domain.co.kr >> ~/.bashrc
	source ~/.bashrc

	kops create cluster \\
	--name=${KOPS_CLUSTER_NAME} \\
	--cloud=aws \\
	--networking amazonvpc \\
	--zones=ap-northeast-2a,ap-northeast-2c \\
	--network-cidr 172.30.0.0/16 \\
	--master-size t3.medium \\
	--node-size t3.medium \\
	--node-count=2 \\
	--kubernetes-version "1.24.10" \\
	--ssh-public-key ~/.ssh/id_rsa.pub -y

	source <(kubectl completion bash)
	echo 'source <(kubectl completion bash)' >> ~/.bashrc
	echo 'alias k=kubectl' >> ~/.bashrc
	echo 'complete -F __start_kubectl k' >> ~/.bashrc
	source ~/.bashrc
	k get no
	---
	NAME STATUS ROLES AGE VERSION
	i-025ff97f4a34dd7fa Ready node 19m v1.24.10
	i-0c69a38808d05a0d7 Ready control-plane 20m v1.24.10
	i-0e37ceecc596e40da Ready node 19m v1.24.10

Kops 설치 방법(EC2 Instance)

Route53 도메인 설정

클러스터 저장용 S3 버킷 생성

S3 Bucket 생성

S3 Bucket 생성

S3 Bucket Public Access Block 설정

S3 Bucket Policy 설정

EC2 Instance Profile 설정

IAM Instance Profile Trust 파일 생성

IAM Instance Profile Permission 파일 생성

IAM Role 및 Instnace Profile 생성

EC2 Instance에 Instance Profile 연결

K8S Cluster 생성

kubectl 설치(EC2 Instance)

ssh key 생성

환경 변수 설정

클러스터 생성

생성된 클러스터 확인

블로그의 정보

활동하기

공유하기

다른 글

티스토리툴바

개인정보

단축키

내 블로그

블로그 게시글

모든 영역