k8s环境搭建

k8s环境搭建

文档介绍在Docker for mac中的k8s如何把环境搭建好。

清单:

  • docker for mac中启动k8s
  • k8s网络环境介绍
  • ingress-nginx安装(网关)
  • k8s-dashboard安装(官方提供的UI管理)
  • weavescope安装(集群监控)
  • mysql安装(事例)

通过以上清单,可以完成大部分日常开发工作。

docker for mac中启动k8s

要启动docker for mac中的k8s,需要下载很多k8s相关镜像,这部分镜像国内网络无法下载,我会列出来所有的镜像可以自己从其他途径下载,也可以直接通过我给的脚本下载。

镜像下载:

#!/bin/bash
kube_version=v1.15.5
kube_proxy_version=$kube_version
kube_controller_manager_version=$kube_version
kube_scheduler_version=$kube_version
kube_apiserver_version=$kube_version

coredns_version=1.3.1
pause_version=3.1
etcd_version=3.3.10
# 镜像列表的前面的是需要下载的镜像,等号后面的是能够正常下载下来的镜像
image_list=(
  "k8s.gcr.io/kube-proxy:${kube_proxy_version}=gotok8s/kube-proxy:${kube_proxy_version}"
  "k8s.gcr.io/kube-controller-manager:${kube_controller_manager_version}=gotok8s/kube-controller-manager:${kube_controller_manager_version}"
  "k8s.gcr.io/kube-scheduler:${kube_scheduler_version}=gotok8s/kube-scheduler:${kube_scheduler_version}"
  "k8s.gcr.io/kube-apiserver:${kube_apiserver_version}=gotok8s/kube-apiserver:${kube_apiserver_version}"
  "k8s.gcr.io/coredns:${coredns_version}=gotok8s/coredns:${coredns_version}"
  "k8s.gcr.io/pause:${pause_version}=gotok8s/pause:${pause_version}"
  "k8s.gcr.io/etcd:${etcd_version}=gotok8s/etcd:${etcd_version}"
)
OLD_IFS="$IFS"
for value in "${image_list[@]}"; do
  IFS="="
  read -r -a image <<<"$value"
  image_name=${image[0]}
  repeat_image_name=${image[1]}
  echo "pull $image_name from $repeat_image_name"
  docker pull "$repeat_image_name"
  docker tag "$repeat_image_name" "$image_name"
  docker rmi "$repeat_image_name"
done
IFS="$OLD_IFS"

下载好了后直接通过docker for mac打开就好了, 大概等5分钟左右:
image.png

k8s单节点集群就安装完成了。

k8s网络环境介绍

k8s中的网络不通比较难找问题,下面先澄清一下:

  • docker for mac中宿主机是无法直接访问容器的。具体详见:https://docs.docker.com/docker-for-mac/networking/
  • 容器是可以直接访问宿主机的,容器里面有一个docker0的桥接网卡。

ingress-nginx安装

有了k8s集群,集群里面部署的应用我们没办法直接访问,除非使用nodePort的服务形式暴露出来,这样暴露出来有个问题就是:我们直接通过pod所在的机器ip和端口来访问,每次重启应用可能会在不同的机器上,这样就导致访问的ip一直处于变化中。为了解决这个问题,我们私有自建集群就可以通过以下几种方式来解决:

  • MetaLB。和提供给云厂商的负载均衡一样的。

  • ingress。Ingress 公开了从集群外部到集群内service的HTTP和HTTPS路由。 流量路由由 Ingress 资源上定义的规则控制。

       internet
          |
     [ Ingress ]
     --|-----|--
     [ Services ]
    

以上两种是比较推荐来解决ip漂移问题的方案。

安装ingress

  1. 安装ingress-controller。下载文件:https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/mandatory.yaml保存成ingress-nginx.yaml
    kubectl apply -f ingress-nginx.yaml
    

这里可以直接把这个文件下载到本机,里面有一个镜像:quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0这个镜像在国内同样可能无法下载,需要自己去下载下来。对于这个版本,提供一个替代下载地址:registry.cn-shenzhen.aliyuncs.com/yangqiang-pub/kubernetes-ingress-controller:0.28.0可以把文件里面的镜像替换成这个地址。

  1. 安装service。
    kubectl apply -f service.yaml
    

    我们主要通过NodePort的方式暴露出去,还有其他选择参考:https://kubernetes.github.io/ingress-nginx/deploy/baremetal/

    service.yaml

    kind: Service
    apiVersion: v1
    metadata:
     name: ingress-nginx
     namespace: ingress-nginx
     labels:
       app.kubernetes.io/name: ingress-nginx
       app.kubernetes.io/part-of: ingress-nginx
    spec:
     externalTrafficPolicy: Local
     type: NodePort
     selector:
       app.kubernetes.io/name: ingress-nginx
       app.kubernetes.io/part-of: ingress-nginx
     ports:
       - name: http
         port: 80
         protocol: TCP
         targetPort: http
         nodePort: 30000
       - name: https
         port: 443
         protocol: TCP
         targetPort: https
         nodePort: 30001
    

    这里的端口号必须是30000-32767。所以我们这里直接把30000和30001当成80以及443来使用。尽管使用--service-node-port-rangeAPI服务器标志重新配置NodePort范围以包含非特权端口并能够公开端口80和443。这样做可能会导致意外问题。

到此ingress-nginx就安装完成了。

k8s-dashboard安装(官方提供的UI管理)

  1. 安装dashboard。如果镜像下载不下来,也要替换。
    kubectl apply -f dashboard.yaml
    

    dashboard.yaml

    # Copyright 2017 The Kubernetes Authors.
    #
    # Licensed under the Apache License, Version 2.0 (the "License");
    # you may not use this file except in compliance with the License.
    # You may obtain a copy of the License at
    #
    #     http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.
    # 修改说明:修改两个镜像的拉起策略都为imagePullPolicy: IfNotPresent 防止每次从远程拉不到镜像的问题
    apiVersion: v1
    kind: Namespace
    metadata:
     name: kubernetes-dashboard
    
    ---
    
    apiVersion: v1
    kind: ServiceAccount
    metadata:
     labels:
       k8s-app: kubernetes-dashboard
     name: kubernetes-dashboard
     namespace: kubernetes-dashboard
    
    ---
    
    kind: Service
    apiVersion: v1
    metadata:
     labels:
       k8s-app: kubernetes-dashboard
     name: kubernetes-dashboard
     namespace: kubernetes-dashboard
    spec:
     ports:
       - port: 443
         targetPort: 8443
     selector:
       k8s-app: kubernetes-dashboard
    
    ---
    
    apiVersion: v1
    kind: Secret
    metadata:
     labels:
       k8s-app: kubernetes-dashboard
     name: kubernetes-dashboard-certs
     namespace: kubernetes-dashboard
    type: Opaque
    
    ---
    
    apiVersion: v1
    kind: Secret
    metadata:
     labels:
       k8s-app: kubernetes-dashboard
     name: kubernetes-dashboard-csrf
     namespace: kubernetes-dashboard
    type: Opaque
    data:
     csrf: ""
    
    ---
    
    apiVersion: v1
    kind: Secret
    metadata:
     labels:
       k8s-app: kubernetes-dashboard
     name: kubernetes-dashboard-key-holder
     namespace: kubernetes-dashboard
    type: Opaque
    
    ---
    
    kind: ConfigMap
    apiVersion: v1
    metadata:
     labels:
       k8s-app: kubernetes-dashboard
     name: kubernetes-dashboard-settings
     namespace: kubernetes-dashboard
    
    ---
    
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
     labels:
       k8s-app: kubernetes-dashboard
     name: kubernetes-dashboard
     namespace: kubernetes-dashboard
    rules:
     # Allow Dashboard to get, update and delete Dashboard exclusive secrets.
     - apiGroups: [""]
       resources: ["secrets"]
       resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
       verbs: ["get", "update", "delete"]
       # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
     - apiGroups: [""]
       resources: ["configmaps"]
       resourceNames: ["kubernetes-dashboard-settings"]
       verbs: ["get", "update"]
       # Allow Dashboard to get metrics.
     - apiGroups: [""]
       resources: ["services"]
       resourceNames: ["heapster", "dashboard-metrics-scraper"]
       verbs: ["proxy"]
     - apiGroups: [""]
       resources: ["services/proxy"]
       resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
       verbs: ["get"]
    
    ---
    
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
     labels:
       k8s-app: kubernetes-dashboard
     name: kubernetes-dashboard
    rules:
     # Allow Metrics Scraper to get metrics from the Metrics server
     - apiGroups: ["metrics.k8s.io"]
       resources: ["pods", "nodes"]
       verbs: ["get", "list", "watch"]
    
    ---
    
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
     labels:
       k8s-app: kubernetes-dashboard
     name: kubernetes-dashboard
     namespace: kubernetes-dashboard
    roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: kubernetes-dashboard
    subjects:
     - kind: ServiceAccount
       name: kubernetes-dashboard
       namespace: kubernetes-dashboard
    
    ---
    
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
     name: kubernetes-dashboard
    roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: kubernetes-dashboard
    subjects:
     - kind: ServiceAccount
       name: kubernetes-dashboard
       namespace: kubernetes-dashboard
    
    ---
    
    kind: Deployment
    apiVersion: apps/v1
    metadata:
     labels:
       k8s-app: kubernetes-dashboard
     name: kubernetes-dashboard
     namespace: kubernetes-dashboard
    spec:
     replicas: 1
     revisionHistoryLimit: 10
     selector:
       matchLabels:
         k8s-app: kubernetes-dashboard
     template:
       metadata:
         labels:
           k8s-app: kubernetes-dashboard
       spec:
         containers:
           - name: kubernetes-dashboard
             image: kubernetesui/dashboard:v2.0.0-rc5
             imagePullPolicy: IfNotPresent
             ports:
               - containerPort: 8443
                 protocol: TCP
             args:
               - --auto-generate-certificates
               - --namespace=kubernetes-dashboard
               # Uncomment the following line to manually specify Kubernetes API server Host
               # If not specified, Dashboard will attempt to auto discover the API server and connect
               # to it. Uncomment only if the default does not work.
               # - --apiserver-host=http://my-address:port
             volumeMounts:
               - name: kubernetes-dashboard-certs
                 mountPath: /certs
                 # Create on-disk volume to store exec logs
               - mountPath: /tmp
                 name: tmp-volume
             livenessProbe:
               httpGet:
                 scheme: HTTPS
                 path: /
                 port: 8443
               initialDelaySeconds: 30
               timeoutSeconds: 30
             securityContext:
               allowPrivilegeEscalation: false
               readOnlyRootFilesystem: true
               runAsUser: 1001
               runAsGroup: 2001
         volumes:
           - name: kubernetes-dashboard-certs
             secret:
               secretName: kubernetes-dashboard-certs
           - name: tmp-volume
             emptyDir: {}
         serviceAccountName: kubernetes-dashboard
         nodeSelector:
           "beta.kubernetes.io/os": linux
         # Comment the following tolerations if Dashboard must not be deployed on master
         tolerations:
           - key: node-role.kubernetes.io/master
             effect: NoSchedule
    
    ---
    
    kind: Service
    apiVersion: v1
    metadata:
     labels:
       k8s-app: dashboard-metrics-scraper
     name: dashboard-metrics-scraper
     namespace: kubernetes-dashboard
    spec:
     ports:
       - port: 8000
         targetPort: 8000
     selector:
       k8s-app: dashboard-metrics-scraper
    
    ---
    
    kind: Deployment
    apiVersion: apps/v1
    metadata:
     labels:
       k8s-app: dashboard-metrics-scraper
     name: dashboard-metrics-scraper
     namespace: kubernetes-dashboard
    spec:
     replicas: 1
     revisionHistoryLimit: 10
     selector:
       matchLabels:
         k8s-app: dashboard-metrics-scraper
     template:
       metadata:
         labels:
           k8s-app: dashboard-metrics-scraper
         annotations:
           seccomp.security.alpha.kubernetes.io/pod: 'runtime/default'
       spec:
         containers:
           - name: dashboard-metrics-scraper
             image: kubernetesui/metrics-scraper:v1.0.3
             imagePullPolicy: IfNotPresent
             ports:
               - containerPort: 8000
                 protocol: TCP
             livenessProbe:
               httpGet:
                 scheme: HTTP
                 path: /
                 port: 8000
               initialDelaySeconds: 30
               timeoutSeconds: 30
             volumeMounts:
             - mountPath: /tmp
               name: tmp-volume
             securityContext:
               allowPrivilegeEscalation: false
               readOnlyRootFilesystem: true
               runAsUser: 1001
               runAsGroup: 2001
         serviceAccountName: kubernetes-dashboard
         nodeSelector:
           "beta.kubernetes.io/os": linux
         # Comment the following tolerations if Dashboard must not be deployed on master
         tolerations:
           - key: node-role.kubernetes.io/master
             effect: NoSchedule
         volumes:
           - name: tmp-volume
             emptyDir: {}
    
  2. 安装好后还无法访问,需要创建账号和绑定角色。
    kubectl apply -f admin-role.yaml
    

    admin-role.yaml

    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
     name: admin
     annotations:
       rbac.authorization.kubernetes.io/autoupdate: "true"
    roleRef:
     kind: ClusterRole
     name: cluster-admin
     apiGroup: rbac.authorization.k8s.io
    subjects:
    - kind: ServiceAccount
     name: admin
     namespace: kube-system
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
     name: admin
     namespace: kube-system
     labels:
       kubernetes.io/cluster-service: "true"
       addonmanager.kubernetes.io/mode: Reconcile
    
  3. 启动。执行以下命令,然后访问:http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
    kubectl proxy
    

    这样可以直接访问,稍后会使用ingress配置来访问。

  4. 配置通过ingress启动。

    kubectl apply -f ingress.yaml
    

    ingress.yaml

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
     name: kubernetes-dashboard
     namespace: kubernetes-dashboard
     annotations:
       kubernetes.io/ingress.class: "nginx"
       nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    
    spec:
     tls:
     - hosts:
       # 访问的域名
       - dashboard.yangqiang.im
       # 证书名称
       secretName: yangqiang.im
     rules:
       - host: dashboard.yangqiang.im
         http:
           paths:
           - path: /
             backend:
               serviceName: kubernetes-dashboard
               servicePort: 443
    
    1. 创建证书查看教程:https://yangqiang.im/?p=745

    2. 在k8s集群中创建证书。

      kubectl create secret tls yangqiang.im --cert=/Users/carlton/Documents/Ssl/yangqiang.im/fullchain.cer --key=/Users/carlton/Documents/Ssl/yangqiang.im/yangqiang.im.key -n kubernetes-dashboard
      
    3. 因为你的域名没办法直接指向到你自己的机器。所以这里在hosts中新增:127.0.0.1 dashboard.yangqiang.im。然后通过地址:https://dashboard.yangqiang.im就能直接访问了。

    注意:nginx.ingress.kubernetes.io/backend-protocol: &quot;HTTPS&quot;这个配置是对应到pod提供的服务是否是https还是http的。当前应用就是提供的443所以用HTTS,一般我们自己的应用都是80或者其他端口这里需要修改成nginx.ingress.kubernetes.io/backend-protocol: &quot;HTTP&quot;

weavescope安装(集群监控)

weavescope是一个k8s集群监控软件,贴个图:

image.png

这个安装很简单,就不贴细节了。官网安装参考:https://www.weave.works/docs/scope/latest/installing/#k8s

ingress配置:

kubectl apply -f ingress.yaml

ingress.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: weave-scope 
  namespace: weave
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  rules:
    - host: weave-scope.yangqiang.im
      http:
        paths:
        - path: /
          backend:
            serviceName: weave-scope-app
            servicePort: app

可以通过:http://weave-scope.yangqiang.im来访问,同样hosts新增:127.0.0.1 weave-scope.yangqiang.im

mysql安装

因为mysql是走的tcp端口,ingress-nginx配置tcp不是标准协议,所以拿来示范如何配置tcp。

  1. 安装mysql。
    kubectl apply -f mysql.yaml
    

    mysql.yaml

    kind: Deployment
    apiVersion: apps/v1beta2
    metadata:
     labels:
       app: mysql
     name: mysql
    spec:
     replicas: 1
     revisionHistoryLimit: 1
     selector:
       matchLabels:
         app: mysql
     template:
       metadata:
         labels:
           app: mysql
       spec:
         volumes:
         - name: pvc
           hostPath: 
             # 这里是mysql数据存储的地方,是宿主机上的地址
             path: /Users/carlton/Documents/K8s/data
         containers:
         - name: mysql
           image: mysql:5.7.13 
           volumeMounts:
           - name: pvc
             mountPath: "/var/lib/mysql"
             subPath: "mysql/data"
           - name: pvc
             mountPath: "/etc/mysql/conf.d"
             subPath: "mysql/conf.d"
           ports:
           - containerPort: 3306
             protocol: TCP
           env:
           - name: TZ
             value: 'Asia/Shanghai'
           - name: MYSQL_ROOT_PASSWORD
             value: "123456"
    
  2. 安装service。
    kubectl apply -f service.yaml
    

    service.yaml

    kind: Service
    apiVersion: v1
    metadata:
     name: mysql
     labels:
       app: mysql   
    spec:
     type: ClusterIP
     ports:
       - name: mysql-3306
         port: 3306
         targetPort: 3306
         protocol: TCP
     selector:
       app: mysql
    
  3. 配置ingress。tcp或者udp服务不需要安装ingress,因为不是标准的协议。如果要通过ingress访问,则需要在ingress-controller应用里面配置。
    1. 配置服务转发。找到ingress部署文件ingress-nginx.yaml,然后更新:
      kubectl apply -f ingress-nginx.yaml
      

      ingress-nginx.yaml

      ......
      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: tcp-services
        namespace: ingress-nginx
        labels:
        app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
      data:
       30100: "default/mysql:3306"
      ......
      

    注意:30100: &quot;default/mysql:3306&quot;这个的意思是,default命名空间下面的mysql应用提供的服务端口是3306。然后ingress通过30100端口往外暴露。

    1. 配置ingress的服务暴露给外网,修改ingress的service.yaml文件
      kubectl apply -f service.yaml
      

      service.yaml

      kind: Service
      apiVersion: v1
      metadata:
      name: ingress-nginx
      namespace: ingress-nginx
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      spec:
      externalTrafficPolicy: Local
      type: NodePort
      selector:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      ports:
        - name: http
          port: 80
          protocol: TCP
          targetPort: http
          nodePort: 30000
        - name: https
          port: 443
          protocol: TCP
          targetPort: https
          nodePort: 30001
        # 在ingress-controller的config-map中配置了30100 -> 3306。这里是暴露30100
        # 整个过程:mysql容器c暴露3306,然后mysql服务s也是暴露的3306,ingress配置"30100": "dev/mysql:3306(服务暴露的端口)",ingerss服务拿着30100暴露30100给宿主。
        # c3306->s3306->ingress_30100->外网宿主机30100
        - name: mysql-3306
          # 随便一个只要不冲突就可以,这个是ingress-controller的服务向集群暴露端口
          port: 30100
          protocol: TCP
          # 这个必须是config-map中配置的对应端口
       targetPort: 30100
          # 这个是主机暴露给外网的端口
      nodePort: 30100
      

      到此为止,我们可以通过127.0.0.1:30100来直接连接到mysql数据库。这是pod的tcp端口暴露的方式。

发表评论

电子邮件地址不会被公开。 必填项已用*标注

Captcha Code