728x90
반응형
스테이징이라는 새로운 인프라 환경을 구축하고 컨테이너 전달 프로세스를 통해 코드 저장소를 확장하고, Argo CD 깃옵스 도구를 사용해 컨테이너를 배포할 수 있도록 구현
참고: 스테이징 환경과 추가 모듈(인그레스, 데이터베이스) 소스 코드는 교재에서 제공하는 것을 사용
스테이징 인프라 환경 구축
로컬에서 구축하고 테스트한 서비스를 AWS 기반 클라우드 인프라에 배포하는 스테이징 인프라를 구축
샌드박스 환경에 항공편 마이크로서비스와 예약 마이크로서비스의 요구사항을 반영하도록 인프라 코드를 업데이트 ⇒ 인그레스 모듈과 데이터베이스 모듈을 추가
인그레스 모듈
https://github.com/implementing-microservices/module-aws-traefik.git
개발 단계에서 사용한 Traefik을 AWS 환경에서도 활용해 인그레스 컨트롤러를 구현
Traefik을 사용해 로드밸런서의 메시지를 쿠버네티스에 배포된 마이크로서비스로 라우팅
데이터베이스 모듈
https://github.com/implementing-microservices/module-aws-db
각 마이크로서비스가 서로 다른 데이터베이스를 사용하므로 각 마이크로서비스에 적합한 AWS 서비스 인스턴스와 데이터베이스 서비스가 작동하는데 필요한 네트워크 구성 및 접근 정책을 제공
AWS 엘라스틱캐시 서비스를 사용해 레디스 데이터 스토어를, AWS RDS를 사용해 MySQL 인스턴스를 프로비저닝
깃허브 저장소를 포크해서 스테이징 환경 저장소를 생성
https://github.com/implementing-microservices/infra-staging-env
레포지터리 시크릿 생성
CI/CD 워크플로에서 AWS 서비스에 접근할 수 있도록 AWS 계정의 자격증명 정보와 MySQL 패스워드를 깃허브 저장소의 시크릿에 등록
주의: MySQL 패스워드에 특수문자가 포함된 경우 오류가 발생
깃허브 액션을 활성화
스테이징 인프라 환경 코드를 로컬 개발 환경으로 복사
c:\msur> git clone https://github.com/naanjini/infra-staging-env.git
Cloning into 'infra-staging-env'...
remote: Enumerating objects: 83, done.
remote: Total 83 (delta 0), reused 0 (delta 0), pack-reused 83
Receiving objects: 100% (83/83), 14.32 KiB | 666.00 KiB/s, done.
Resolving deltas: 100% (35/35), done.스테이징 인프라 환경을 정의하는 main.tf 파일을 편집
사용자 환경 정보를 설정하고, 테라폼 버전에서 발생할 수 있는 문제점을 수정한 코드를 반영
c:\msur\infra-staging-env\main.tf
terraform {
backend "s3" {
bucket = "sk403-003-bucket" #
key = "terraform/backend-staging" #
region = "us-west-2" #
}
}
locals {
env_name = "sk403-003-staging" #
aws_region = "us-west-2" #
k8s_cluster_name = "sk403-003-ms-cluster" #
}
variable "mysql_password" {
type = string
description = "Expected to be retrieved from environment variable TF_VAR_mysql_password"
}
provider "aws" {
region = local.aws_region
}
data "aws_eks_cluster" "msur" {
name = module.aws-kubernetes-cluster.eks_cluster_id
}
module "aws-network" {
source = "github.com/implementing-microservices/module-aws-network"
env_name = local.env_name
vpc_name = "sk403-003-msur-VPC" #
cluster_name = local.k8s_cluster_name
aws_region = local.aws_region
main_vpc_cidr = "10.10.0.0/16"
public_subnet_a_cidr = "10.10.0.0/18"
public_subnet_b_cidr = "10.10.64.0/18"
private_subnet_a_cidr = "10.10.128.0/18"
private_subnet_b_cidr = "10.10.192.0/18"
}
module "aws-kubernetes-cluster" {
source = "github.com/implementing-microservices/module-aws-kubernetes"
ms_namespace = "sk403-003-microservices" #
env_name = local.env_name
aws_region = local.aws_region
cluster_name = local.k8s_cluster_name
vpc_id = module.aws-network.vpc_id
cluster_subnet_ids = module.aws-network.subnet_ids
nodegroup_subnet_ids = module.aws-network.private_subnet_ids
nodegroup_disk_size = "20"
nodegroup_instance_types = ["t3.medium"]
nodegroup_desired_size = 1
nodegroup_min_size = 1
nodegroup_max_size = 5
}
# Create namespace
# Use kubernetes provider to work with the kubernetes cluster API
provider "kubernetes" {
# load_config_file = false #
cluster_ca_certificate = base64decode(data.aws_eks_cluster.msur.certificate_authority.0.data)
host = data.aws_eks_cluster.msur.endpoint
exec {
api_version = "client.authentication.k8s.io/v1alpha1"
command = "aws-iam-authenticator"
args = ["token", "-i", "${data.aws_eks_cluster.msur.name}"]
}
}
# Create a namespace for microservice pods
resource "kubernetes_namespace" "ms-namespace" {
metadata {
name = "sk403-003-microservices" #
}
}
module "argo-cd-server" {
source = "github.com/november11th/module-argo-cd" #
aws_region = local.aws_region
kubernetes_cluster_id = data.aws_eks_cluster.msur.id
kubernetes_cluster_name = module.aws-kubernetes-cluster.eks_cluster_name
kubernetes_cluster_cert_data = module.aws-kubernetes-cluster.eks_cluster_certificate_data
kubernetes_cluster_endpoint = module.aws-kubernetes-cluster.eks_cluster_endpoint
eks_nodegroup_id = module.aws-kubernetes-cluster.eks_cluster_nodegroup_id
}
module "aws-databases" {
source = "github.com/implementing-microservices/module-aws-db"
aws_region = local.aws_region
mysql_password = var.mysql_password
vpc_id = module.aws-network.vpc_id
eks_id = data.aws_eks_cluster.msur.id
eks_sg_id = module.aws-kubernetes-cluster.eks_cluster_security_group_id
subnet_a_id = module.aws-network.private_subnet_ids[0]
subnet_b_id = module.aws-network.private_subnet_ids[1]
env_name = local.env_name
route53_id = module.aws-network.route53_id
}
module "traefik" {
source = "github.com/november11th/module-aws-traefik/" #
aws_region = local.aws_region
kubernetes_cluster_id = data.aws_eks_cluster.msur.id
kubernetes_cluster_name = module.aws-kubernetes-cluster.eks_cluster_name
kubernetes_cluster_cert_data = module.aws-kubernetes-cluster.eks_cluster_certificate_data
kubernetes_cluster_endpoint = module.aws-kubernetes-cluster.eks_cluster_endpoint
eks_nodegroup_id = module.aws-kubernetes-cluster.eks_cluster_nodegroup_id
}
데이터베이스 작업 권한을 부여할 사용자 그룹을 생성하고, 운영 계정을 추가
c:\msur> aws iam create-group --group-name sk403-003-DB-Ops
{
"Group": {
"Path": "/",
"GroupName": "sk403-003-DB-Ops",
"GroupId": "AGPAUFTZFRUTQR5GT3PRF",
"Arn": "arn:aws:iam::286943186215:group/sk403-003-DB-Ops",
"CreateDate": "2022-03-18T00:02:07+00:00"
}
}
c:\msur> aws iam add-user-to-group --user-name sk403-003-ops-account --group-name sk403-003-DB-Ops
생성한 그룹에 RDS와 엘라스틱캐시 접근 권한 정책을 추가
c:\msur> aws iam attach-group-policy --group-name sk403-003-DB-Ops --policy-arn arn:aws:iam::aws:policy/AmazonRDSFullAccess
c:\msur> aws iam attach-group-policy --group-name sk403-003-DB-Ops --policy-arn arn:aws:iam::aws:policy/AmazonElastiCacheFullAccess
테라폼 명령을 실행해 업데이트된 코드의 형식을 지정하고 유효성을 검사
c:\msur\infra-staging-env> terraform fmt
c:\msur\infra-staging-env> terraform init
c:\msur\infra-staging-env> terraform validate
c:\msur\infra-staging-env> terraform plan
스테이징 인프라 코드를 커밋하고 CI/CD 파이프라인을 실행
c:\msur\infra-staging-env> git add .
c:\msur\infra-staging-env> git commit -m "Staging environment with database"
c:\msur\infra-staging-env> git push origin
c:\msur\infra-staging-env> git tag -a v1.0 -m "Initial staging environment build"
c:\msur\infra-staging-env> git push origin v1.0
쿠버네티스 클러스터 접속 테스트
kubeconfig 파일을 다운로드해서 KUBECONFIG 환경변수로 설정하거나 ~/.kube/config 파일로 복사 후 노드 및 서비스 확인 ⇒ 스테이징 인프라 환경이 정상적으로 구성된 것을 확인
c:\msur\infra-staging-env> kubectl get node
NAME STATUS ROLES AGE VERSION
ip-10-10-229-65.us-west-2.compute.internal Ready <none> 11m v1.21.5-eks-9017834
c:\msur\infra-staging-env> kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
argocd msur-argocd-application-controller ClusterIP 172.20.87.205 <none> 8082/TCP 9m49s
argocd msur-argocd-applicationset-controller ClusterIP 172.20.141.149 <none> 7000/TCP 9m49s
argocd msur-argocd-dex-server ClusterIP 172.20.139.54 <none> 5556/TCP,5557/TCP 9m49s
argocd msur-argocd-redis ClusterIP 172.20.5.109 <none> 6379/TCP 9m49s
argocd msur-argocd-repo-server ClusterIP 172.20.105.244 <none> 8081/TCP 9m49s
argocd msur-argocd-server ClusterIP 172.20.103.120 <none> 80/TCP,443/TCP 9m49s
default kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 17m
default ms-traefik-ingress LoadBalancer 172.20.88.201 adba2f785f92748eb9186a62a98f2b47-e7017c9924295219.elb.us-west-2.amazonaws.com 80:31039/TCP,443:31625/TCP 9m55s
kube-system kube-dns ClusterIP 172.20.0.10 <none> 53/UDP,53/TCP 17m
c:\msur\infra-staging-env> kubectl get ns
NAME STATUS AGE
argocd Active 12m
default Active 20m
kube-node-lease Active 20m
kube-public Active 20m
kube-system Active 20m
sk403-003-microservices Active 12m
쿠버네티스 시크릿 생성
항공편 정보 마이크로서비스에서 MySQL 서버 연결에 사용할 비밀번호를 저장할 시크릿을 생성
c:\msur> kubectl create secret generic mysql --from-literal password=PASSWORD -n sk403-003-microservices
secret/mysql created
항공편 정보 컨테이너 게시
항공 마이크로서비스를 빌드하고 컨테이너 레지스트리(여기에서는 도커 허브를 사용)에 게시하기 위한 CI/CD 파이프라인을 구축
도커 허브 저장소 생성
마이크로서비스 컨테이너 빌드 파이프라인 구성
인프라 환경 구성과 동일하게 깃허브 액션을 이용해서 마이크로서비스의 컨테이너 빌드 파이프라인을 구성
도커 허브 계정 정보를 깃허브 저장소의 시크릿에 등록
워크플로가 컨테이너 게시를 위해 도커 허브와 통신할 수 있도록 도커 허브 계정 정보를 항공편 깃허브 저장소의 시크릿에 등록
Dockerfile 수정
윈도우 환경에서는 sh 파일에 실행 권한이 설정되지 않는 문제가 있으므로, 명시적으로 실행 권한을 부여하는 코드를 추가
C:\msur\ms-flights\Dockerfile
# Alpine Linux-based, tiny Node container:
FROM node:12-alpine3.9 as base
COPY ./ /opt/app
WORKDIR /opt/app
USER root
RUN rm -rf node_modules \
&& chown -R node /opt/app
USER node
FROM base as release
USER root
RUN npm install --only=production \
#&& apk add --no-cache tini \
&& chown -R node /opt/app \
# sh 파일에 실행 권한을 부여
&& chmod +x -R ./shell/
USER node
ENV HOME_DIR=/opt/app \
NODE_ENV=production \
PORT=5501
ENTRYPOINT ./shell/run-db-migraton.sh && node server.js
FROM base as build
USER root
RUN npm install -g nodemon \
&& npm install \
&& chown -R node /opt/app
USER node
깃허브 액션 생성 및 워크플로 코드 추가
https://github.com/implementing-microservices/ms-flights/blob/master/.github/workflows/main.yml 코드를 붙여 넣은 후 단위 테스트 부분을 주석 처리
v1.0 태그를 릴리스에 푸시해서 CI/CD 워크플로를 트리거
워크플로가 정상적으로 완료되면 도커 허브 레지스트리에 이미지가 푸시된 것을 확인
마이크로서비스 컨테이너 배포
인프라에 설치한 깃옵스 배포 도구인 Argo CD를 사용해 항공편 마이크로서비스를 배포
반복 가능한 배포를 위해 헬름 패키지를 포함한 새로운 배포 저장소를 생성 후 헬름 패키지(마이크로서비스 배포 방법을 설명)를 배포 저장소로 푸시하면 Argo CD가 스테이징 환경으로 컨테이너를 배포
마이크로서비스 배포 저장소 생성
헬름 차트를 저장하고 관리할 수 있는 마이크로서비스 배포를 위한 깃허브 저장소를 생성
로컬 개발 환경에 저장소를 복제
c:\msur> git clone https://github.com/naanjini/ms-deploy.git
Cloning into 'ms-deploy'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (3/3), done.
로컬 개발 환경에 헬름 CLI 설치
https://github.com/helm/helm/releases ⇒ 다운로드 받은 파일을 압축 해제한 후 helm.exe 파일을 C:\msur 폴더로 복사
헬름 차트 생성
C:\kubernetes\ms-deploy> helm create ms-flights
Creating my-flights
C:\kubernetes\ms-deploy> cd ms-flights
C:\kubernetes\ms-deploy\my-flights> dir
C 드라이브의 볼륨: OS
볼륨 일련 번호: 00E8-93DB
C:\kubernetes\ms-deploy\ms-flights 디렉터리
2022-03-06 오후 05:21 <DIR> .
2022-03-06 오후 05:21 <DIR> ..
2022-03-06 오후 05:21 349 .helmignore
2022-03-06 오후 05:21 1,146 Chart.yaml ⇐ 차트 (쿠버네티스 리소스와 배포를 설명하는 파일)
2022-03-06 오후 05:21 <DIR> charts
2022-03-06 오후 05:21 <DIR> templates ⇐ 기본적인 배포를 위한 쿠버네티스 YAML 템플릿을 포함
2022-03-06 오후 05:21 1,877 values.yaml ⇐ 차트 값 (템플릿을 채우는데 사용해야 하는 값을 정의)
3개 파일 3,372 바이트
4개 디렉터리 267,208,876,032 바이트 남음
배포 패키지 생성을 위해 마이크로서비스에 맞춰서 템플릿을 수정
C:\kubernetes\ms-deploy\my-flights\templates\deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "my-flights.fullname" . }}
labels:
{{- include "my-flights.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "my-flights.selectorLabels" . | nindent 6 }}
template:
metadata:
{{- with .Values.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "my-flights.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "my-flights.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
env: # MySQL 데이터베이스에 연결하기 위한 템플릿 환경변수
- name: MYSQL_HOST
value: {{ .Values.MYSQL_HOST | quote }}
- name: MYSQL_USER
value: {{ .Values.MYSQL_USER | quote }}
- name : MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: {{ .Values.MYSQLSecretName }}
key: {{ .Values.MYSQLSecretKey }}
- name: MYSQL_DATABASE
value: {{ .Values.MYSQL_DATABASE | quote }}
ports:
- name: http
containerPort: 5501 # 항공편 마이크로서비스에 바인딩하고 컨테이너가 노출하는 TCP 포트
protocol: TCP
livenessProbe:
httpGet:
path: /ping # 활성 프로브
port: http
readinessProbe:
httpGet:
path: /health # 준비성 프로브
port: http
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
values.yaml 파일에 패키지에서 사용하는 값을 설정
C:\kubernetes\ms-deploy\my-flights\values.yaml
replicaCount: 1
# 이미지 수정 (도커 허브에 등록된 저장소와 태그로 변경)
image:
repository: myanjini/flights
pullPolicy: IfNotPresent
tag: 'v1.0'
# MySQL 구성 값
MYSQL_HOST: rds.sk403-003-staging.sk403-003-msur-vpc.com
MYSQL_USER: microservices
MYSQL_DATABASE: microservices_db
MYSQLSecretName: mysql
MYSQLSecretKey: password
… (생략) …
# 인그레스 속성 업데이트
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: traefik
hosts:
- host: flightsvc.com
paths:
- pathType: Prefix
path: '/flights'
tls: []
# 리소스 속성 주석 처리
# resources:
# {}
# # We usually recommend not to specify default resources and to leave this as a conscious
# # choice for the user. This also increases chances charts run on environments with little
# # resources, such as Minikube. If you do want to specify resources, uncomment the following
# # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# # limits:
# # cpu: 100m
# # memory: 128Mi
# # requests:
# # cpu: 100m
# # memory: 128Mi
… (생략) …
패키지 테스트와 커밋
c:\msur\ms-deploy\ms-flights> helm install --debug --dry-run flight-info .
install.go:178: [debug] Original chart version: ""
install.go:195: [debug] CHART PATH: c:\msur\ms-deploy\ms-flights
NAME: flight-info
LAST DEPLOYED: Mon Mar 21 09:41:07 2022
NAMESPACE: default
STATUS: pending-install
REVISION: 1
USER-SUPPLIED VALUES:
{}
COMPUTED VALUES:
MYSQL_DATABASE: microservices_db
MYSQL_HOST: rds.sk403-003-staging.sk403-003-msur-vpc.com
MYSQL_USER: microservices
MYSQLSecretKey: password
MYSQLSecretName: mysql
affinity: {}
autoscaling:
enabled: false
maxReplicas: 100
minReplicas: 1
targetCPUUtilizationPercentage: 80
fullnameOverride: ""
image:
pullPolicy: IfNotPresent
repository: myanjini/flights
tag: v1.0
imagePullSecrets: []
ingress:
annotations:
kubernetes.io/ingress.class: traefik
enabled: true
hosts:
- host: flightsvc.com
paths:
- path: /flights
tls: []
nameOverride: ""
nodeSelector: {}
podAnnotations: {}
podSecurityContext: {}
replicaCount: 1
securityContext: {}
service:
port: 80
type: ClusterIP
serviceAccount:
annotations: {}
create: true
name: ""
tolerations: []
HOOKS:
---
# Source: ms-fligths/templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
name: "flight-info-ms-fligths-test-connection"
labels:
helm.sh/chart: ms-fligths-0.1.0
app.kubernetes.io/name: ms-fligths
app.kubernetes.io/instance: flight-info
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
annotations:
"helm.sh/hook": test
spec:
containers:
- name: wget
image: busybox
command: ['wget']
args: ['flight-info-ms-fligths:80']
restartPolicy: Never
MANIFEST:
---
# Source: ms-fligths/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: flight-info-ms-fligths
labels:
helm.sh/chart: ms-fligths-0.1.0
app.kubernetes.io/name: ms-fligths
app.kubernetes.io/instance: flight-info
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
---
# Source: ms-fligths/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: flight-info-ms-fligths
labels:
helm.sh/chart: ms-fligths-0.1.0
app.kubernetes.io/name: ms-fligths
app.kubernetes.io/instance: flight-info
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
spec:
type: ClusterIP
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app.kubernetes.io/name: ms-fligths
app.kubernetes.io/instance: flight-info
---
# Source: ms-fligths/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: flight-info-ms-fligths
labels:
helm.sh/chart: ms-fligths-0.1.0
app.kubernetes.io/name: ms-fligths
app.kubernetes.io/instance: flight-info
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ms-fligths
app.kubernetes.io/instance: flight-info
template:
metadata:
labels:
app.kubernetes.io/name: ms-fligths
app.kubernetes.io/instance: flight-info
spec:
serviceAccountName: flight-info-ms-fligths
securityContext:
{}
containers:
- name: ms-fligths
securityContext:
{}
image: "myanjini/flights:v1.0"
imagePullPolicy: IfNotPresent
env: # MySQL 데이터베이스에 연결하기 위한 템플릿 환경변수
- name: MYSQL_HOST
value: "rds.sk403-003-staging.sk403-003-msur-vpc.com"
- name: MYSQL_USER
value: "microservices"
- name : MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: mysql
key: password
- name: MYSQL_DATABASE
value: "microservices_db"
ports:
- name: http
containerPort: 5501
protocol: TCP
livenessProbe:
httpGet:
path: /ping
port: http
readinessProbe:
httpGet:
path: /health
port: http
resources:
null
---
# Source: ms-fligths/templates/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: flight-info-ms-fligths
labels:
helm.sh/chart: ms-fligths-0.1.0
app.kubernetes.io/name: ms-fligths
app.kubernetes.io/instance: flight-info
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: "flightsvc.com"
http:
paths:
- path: /flights
backend:
service:
name: flight-info-ms-fligths
port:
number: 80
NOTES:
1. Get the application URL by running these commands:
http://flightsvc.com/flights
c:\msur\ms-deploy\ms-flights> git add .
warning: LF will be replaced by CRLF in ms-flights/.helmignore.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in ms-flights/Chart.yaml.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in ms-flights/templates/NOTES.txt.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in ms-flights/templates/_helpers.tpl.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in ms-flights/templates/deployment.yaml.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in ms-flights/templates/hpa.yaml.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in ms-flights/templates/ingress.yaml.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in ms-flights/templates/service.yaml.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in ms-flights/templates/serviceaccount.yaml.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in ms-flights/templates/tests/test-connection.yaml.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in ms-flights/values.yaml.
The file will have its original line endings in your working directory
c:\msur\ms-deploy\ms-flights> git commit -m "init commit"
[main ba78b72] init commit
11 files changed, 435 insertions(+)
create mode 100644 ms-flights/.helmignore
create mode 100644 ms-flights/Chart.yaml
create mode 100644 ms-flights/templates/NOTES.txt
create mode 100644 ms-flights/templates/_helpers.tpl
create mode 100644 ms-flights/templates/deployment.yaml
create mode 100644 ms-flights/templates/hpa.yaml
create mode 100644 ms-flights/templates/ingress.yaml
create mode 100644 ms-flights/templates/service.yaml
create mode 100644 ms-flights/templates/serviceaccount.yaml
create mode 100644 ms-flights/templates/tests/test-connection.yaml
create mode 100644 ms-flights/values.yaml
c:\msur\ms-deploy\ms-flights> git push origin
Enumerating objects: 17, done.
Counting objects: 100% (17/17), done.
Delta compression using up to 8 threads
Compressing objects: 100% (15/15), done.
Writing objects: 100% (16/16), 5.96 KiB | 677.00 KiB/s, done.
Total 16 (delta 0), reused 0 (delta 0), pack-reused 0
To https://github.com/naanjini/ms-deploy.git
220c4ac..ba78b72 main -> main
스테이징 환경에 설치한 Argo CD 인스턴스에 로그인하고 ms-deploy 저장소를 지정한 다음 동기화된 배포를 설정
c:\msur\ms-deploy\ms-flights> kubectl get pods -n argocd ⇐ Argo CD 파드 확인
NAME READY STATUS RESTARTS AGE
msur-argocd-application-controller-0 1/1 Running 0 121m
msur-argocd-applicationset-controller-b5d78f5b8-85cj4 1/1 Running 0 121m
msur-argocd-dex-server-6469f6ddb7-w2tsr 1/1 Running 0 121m
msur-argocd-notifications-controller-655cc7b448-dz4vj 1/1 Running 0 121m
msur-argocd-redis-7bd95bdc89-fjw7h 1/1 Running 0 121m
msur-argocd-repo-server-574758b969-nnm6x 1/1 Running 0 121m
msur-argocd-server-6f796b848-7d8cm 1/1 Running 0 121m
c:\msur\ms-deploy\ms-flights> kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" ⇐ Argo CD 로그인에 사용할 비밀번호 확인 (BASE64로 인코딩되어 있음)
WG02SGctS1lmQVJuSU1CUg== ⇒ BASE64로 인코딩 한 값(Xm6Hg-KYfARnIMBR)을 로그인에 사용
c:\msur\ms-deploy\ms-flights> kubectl port-forward svc/msur-argocd-server 8443:443 -n argocd
Forwarding from 127.0.0.1:8443 -> 8080 ⇐ 로컬 개발 환경에서 클러스터로 요청을 전달할 수 있도록 설정
Forwarding from [::1]:8443 -> 8080
Argo CD 접속 (localhost:8443) 후 애플리케이션 생성
헬름 패키지에 작성한 내용이 쿠버네티스 클러스터에 적용되도록 동기화(SYNC)
항공편 서비스 테스트
ms-traefik-ingress 서비스의 외부 IP로 접근 (이때, Host 요청 헤더의 값으로 flightsvc.com을 설정)
C:\msur\ms-deploy\ms-flights> kubectl get svc ms-traefik-ingress
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ms-traefik-ingress LoadBalancer 172.20.57.169 ad3afe49242d94c22b21bfd3c237bdc8-1c88b65e605eefc4.elb.us-west-2.amazonaws.com 80:31912/TCP,443:32506/TCP 123m
C:\msur\ms-deploy\ms-flights> curl --header "Host: flightsvc.com" "ad3afe49242d94c22b21bfd3c237bdc8-1c88b65e605eefc4.elb.us-west-2.amazonaws.com/flights?flight_no=AA2532&departure_date_time=2020-05-17T13:20"
{"flight_id":"edcc03a4-7f4e-40d1-898d-bf84a266f1b9","origin_code":"LAX","destination_code":"DCA"}
리소스 정리
모듈 업데이트
C:\msur\infra-staging-env> terraform get --update
Downloading git::https://github.com/november11th/module-argo-cd.git for argo-cd-server...
- argo-cd-server in .terraform\modules\argo-cd-server
Downloading git::https://github.com/implementing-microservices/module-aws-db.git for aws-databases...
- aws-databases in .terraform\modules\aws-databases
Downloading git::https://github.com/implementing-microservices/module-aws-kubernetes.git for aws-kubernetes-cluster...
- aws-kubernetes-cluster in .terraform\modules\aws-kubernetes-cluster
Downloading git::https://github.com/implementing-microservices/module-aws-network.git for aws-network...
- aws-network in .terraform\modules\aws-network
Downloading git::https://github.com/november11th/module-aws-traefik.git for traefik...
- traefik in .terraform\modules\traefik
작업 디렉토리 초기화
C:\msur\infra-staging-env> terraform init
Initializing modules...
Initializing the backend...
╷
│ Error: Invalid legacy provider address
│
│ This configuration or its associated state refers to the unqualified provider "aws".
│
│ You must complete the Terraform 0.13 upgrade process before upgrading to later versions.
╵
╷
│ Error: Invalid legacy provider address
│
│ This configuration or its associated state refers to the unqualified provider "helm".
│
│ You must complete the Terraform 0.13 upgrade process before upgrading to later versions.
╵
╷
│ Error: Invalid legacy provider address
│
│ This configuration or its associated state refers to the unqualified provider "kubernetes".
│
│ You must complete the Terraform 0.13 upgrade process before upgrading to later versions.
╵
╷
│ Error: Invalid legacy provider address
│
│ This configuration or its associated state refers to the unqualified provider "local".
│
│ You must complete the Terraform 0.13 upgrade process before upgrading to later versions.
╵
Terraform 0.14로 테라폼 코드 업그레이드
C:\msur\infra-staging-env> terraform state replace-provider "registry.terraform.io/-/aws" "hashicorp/aws"
C:\msur\infra-staging-env> terraform state replace-provider "registry.terraform.io/-/helm" "hashicorp/helm"
C:\msur\infra-staging-env> terraform state replace-provider "registry.terraform.io/-/kubernetes" "hashicorp/kubernetes"
C:\msur\infra-staging-env> terraform state replace-provider "registry.terraform.io/-/local" "hashicorp/local"
다시 작업 디렉터리 초기화 및 리소스 삭제
C:\msur\infra-staging-env> terraform init
C:\msur\infra-staging-env> terraform destroy
var.mysql_password
Expected to be retrieved from environment variable TF_VAR_mysql_password
Enter a value: PASSWORD
:
Plan: 0 to add, 0 to change, 37 to destroy.
╷
│ Warning: Argument is deprecated
│
│ with module.aws-databases.aws_db_instance.mysql-db,
│ on .terraform\modules\aws-databases\main.tf line 45, in resource "aws_db_instance" "mysql-db":
│ 45: name = var.mysql_database
│
│ Use db_name instead
╵
Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
:
module.aws-kubernetes-cluster.aws_iam_role.ms-node: Destroying... [id=sk403-003-ms-cluster-sk403-003-staging.node]
module.aws-kubernetes-cluster.aws_iam_role.ms-node: Destruction complete after 1s
╷
│ Error: context deadline exceeded
│
│
╵
╷
│ Error: uninstallation completed with 1 error(s): uninstall: Failed to purge the release: release: not found
│
│
╵
╷
│ Error: context deadline exceeded
│
│
╵
C:\msur\infra-staging-env> terraform destroy
var.mysql_password
Expected to be retrieved from environment variable TF_VAR_mysql_password
Enter a value: PASSWORD
:
Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
:
Destroy complete! Resources: 9 destroyed.
S3 버킷 삭제
IAM 사용자, 사용자 그룹, 역할, 정책 삭제
AWS Management Console을 통해 리소스 삭제 확인
Amazon EKS > 클러스터
EC2 > 대시보드 > 인스턴스 (종료 확인), 로드 밸런서, 탄력적 IP
VPC > VPC, 서브넷, 라우팅 테이블, 인터넷 게이트웨이, 탄력적 IP, NAT 게이트웨이 (삭제 확인), …
728x90
반응형
댓글