airflow k8s (helm) 설치

마땅히 할만한 qa서버를 못 구해서 다른 분들처럼 걍 로컬에서 VM에 올리기로 함.

 

VM은 최소 CPU 2개, RAM 2기가, 하드 20기가 필요. (아니면 실행이 안 됨.)

네트워크 설정은 아래 내용 참고

https://sooonworld.tistory.com/17

• 위 내용대로 했으면, ssh 접속은 192.168.11.3으로 하면 됨.

1. K8s 설치

https://kubernetes.io/ko/docs/tasks/tools/install-kubectl-linux/

리눅스에 kubectl 설치 및 설정

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

설치 확인

[root@localhost ~]# kubectl version --cdlient
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:21:19Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1

 

2. minikube 설치

https://minikube.sigs.k8s.io/docs/start/

minikube start

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
install minikube-linux-amd64 /usr/local/bin/minikube

설치 확인 및 시작

[root@localhost tmp]# minikube version
minikube version: v1.30.1
commit: 08896fd1dc362c097c925146c4a0d0dac715ace0
 
[root@localhost tmp]# minikube start --vm-driver=none
* Centos 7.5.1804 (vbox/amd64) 의 minikube v1.30.1
* 유저 환경 설정 정보에 기반하여 none 드라이버를 사용하는 중
 
X Exiting due to GUEST_MISSING_CONNTRACK: Sorry, Kubernetes 1.26.3 requires conntrack to be installed in root's path

 

3. helm 설치

https://helm.sh/docs/intro/install/

Installing Helm

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

 

4. kube 실행 (로컬 VM을 클러스터에 편입 시켜서 쓸 거라 이렇게 함.)

/etc/hosts/에 QA서버 추가

{IP주소} kubernetes.default.svc.cluster.local

kubeconfig는 따로 전달 받아서 처리. (아예 /etc/rc.local에 넣음)

export KUBECONFIG={kubeconfig파일경로}

 

5. airflow 설치

https://airflow.apache.org/docs/helm-chart/1.9.0/

Helm Chart for Apache Airflow — helm-chart Documentation

helm repo add apache-airflow https://airflow.apache.org
helm upgrade --install airflow apache-airflow/airflow --namespace airflow --create-namespace

확인

[root@localhost tmp]# kubectl get po -n airflow
NAME                                 READY   STATUS    RESTARTS   AGE
airflow-postgresql-0                 1/1     Running   0          5m21s
airflow-redis-0                      1/1     Running   0          5m21s
airflow-scheduler-547445f855-rcm5l   2/2     Running   0          5m21s
airflow-statsd-586dbdcc6b-smc9j      1/1     Running   0          5m21s
airflow-triggerer-6ff55547d9-4b4ns   2/2     Running   0          5m21s
airflow-webserver-5664788bf7-sr6dl   1/1     Running   1          5m21s
airflow-worker-0                     2/2     Running   0          5m21s

 

6. nginx ingress에 추가

https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/ingress.md

GitHub - airflow-helm/charts: The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes w

일단 확인

kubectl get all -n ingress-nginx

airflow_qa_ingress.yaml 생성해서 아래 내용 넣음.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: airflow-ingress
  namespace: airflow
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: {접근할주소(airflow.foobar.com이라든가)}
    http:
      paths:
        - path: /
          backend:
            serviceName: airflow-webserver
            servicePort: 8080

적용 및 확인

# kubectl apply -f airflow_qa_ingress.yaml
ingress.extensions/airflow-ingress configured
 
[root@localhost yaml]# kubectl get ing -n airflow
NAME              CLASS    HOSTS                       ADDRESS                                       PORTS   AGE
airflow-ingress   <none>   {설정한도메인}               IP주소쭉~                                      80      7m

 

7. 설정 변경
기존 설정 저장

helm show values apache-airflow/airflow > airflow_qa_values.yaml
cp airflow_qa_values.yaml airflow_qa_values.yaml.bak

KubernetesExecutor로 변경

# Airflow executor
# One of: LocalExecutor, LocalKubernetesExecutor, CeleryExecutor, KubernetesExecutor, CeleryKubernetesExecutor
executor: "KubernetesExecutor"

현재 리비전 확인

[root@localhost yaml]# helm ls -n airflow
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/apps/kubernetes/admin_qa.conf
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/apps/kubernetes/admin_qa.conf
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
airflow airflow         1               2023-05-15 18:39:05.861306557 +0900 KST deployed        airflow-1.9.0   2.5.3

적용

helm upgrade --install airflow apache-airflow/airflow -n airflow -f airflow_qa_values.yaml

변경된 리비전 확인

[root@localhost yaml]# helm ls -n airflow
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/apps/kubernetes/admin_qa.conf
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/apps/kubernetes/admin_qa.conf
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
airflow airflow         2               2023-05-18 13:34:19.137905013 +0900 KST deployed        airflow-1.9.0   2.5.3