Envoy Proxy使用介绍教程（一）: 新型L3~L7层访问代理软件Envoy的使用

小咪咪 2023-09-24 22:03 65阅读 0赞

### 目录 ###

*  [目录][Link 1]
 *  [说明][Link 2]
 *  [主要特点][Link 3]
 *  [工作过程][Link 4]
 *  [部署启动][Link 5]
    
     *  [用docker启动][docker]
     *  [用yum直接安装envoy][yum_envoy]
     *  [用Docker镜像编译envoy][Docker_envoy]
     *  [自己在容器外准备编译环境][Link 6]
     *  [在CentOS上构建编译环境][CentOS]
 *  [配置文件][Link 7]
 *  [代理性能测试][Link 8]
 *  [代理性能继续测试][Link 9]
 *  [问题记录][Link 10]
    
     *  [/lib64/libc.so.6: version \`GLIBC\_2.18’ not found][lib64_libc.so.6_ version _GLIBC_2.18_ not found]
 *  [参考][Link 11]

### 说明 ###

这篇笔记比较老，不再更新维护，请移步最新的手册：[envoy相关笔记][envoy]。

[Envoy][]一个较新的7层代理软件，专门为现在的面向服务架构设计的，据说已经在Lyft、apple、google等公司得到应用。Envoy文档的文档地址是：[https://www.envoyproxy.io/docs/envoy/latest/about\_docs ][https_www.envoyproxy.io_docs_envoy_latest_about_docs]。

### 主要特点 ###

Envoy的设计思路是“再增加一层”，对应用屏蔽了依赖服务的地址，使应用看到的是一个单一、纯粹的网络环境（只需要与localhost交互）。使用envoy后，在每个应用度看来，它只需要与localhost交互，不需要知道依赖的其它服务的地址，通过访问loaclhost就能实际访问到依赖的远端服务。

Envoy是伴随应用部署的，每个应用Server都有一个对应的Envoy，Envoy接管了这个应用所有的流量，负责将它们转发到目的地址。

“加层”的设计思路，使Envoy没有“侵入性”，从而不受应用使用的编程语言的限制，适用范围更广。“加层”还使应用看到的网络环境纯粹单一，从而使应用部署更加方便。

Envoy使用C++11开发，兼顾了性能和开发效率。

Envoy首先是一个L3/L4代理，然后在此基础上通过插件（filter）的方式实现了L7代理，以及对HTTP/2、gRPC、MongoDB L7、DynamoDB L7等协议的支持。负载均衡、健康检查、统计、限速等常见需求都支持。

### 工作过程 ###

用一句话就可以交代清楚：

`Host`上的`Downstream`通过本地envoy提供的多个`Listener`中的一个，连接到由多个`Upstream`组成的`Cluster`。

Host:  就是部署了应用和Envoy的机器
    Downstream：访问envoy的是downstream（下游）
    Listener：envoy的监听器，也就是envoy暴露出来的服务地址
    Upstream：downstream真正想要访问的服务地址，envoy将请求转发给upstream
    Cluster：提供同样服务的一组upstream

多个这样的Host组合在一起就是一个`Mesh`。

Envoy是单进程多线程，master线程负责管理、接收连接，连接被接受后交给worker进程，100%无阻塞。

Listener可以创建多个，建议在每台机器上部署一个envoy，创建多个不同的listner，这样便于管理和统计，Listener现在只支持TCP。

每个Listener的L3/L4层filter插件都是单独配置的。

filter分为`READ`、`WRITE`、`READ/WRITE`三类，READ在收到downstream的发送的数据时调用，WRITE在向downstream发送数据时调用，`READ/WRITE`接收、发送数据时都会被调用。

### 部署启动 ###

部署方式有多种，根据自己情况选择。

#### 用docker启动 ####

Envoy不提供已经编译好的二进制的文件，只提供docker镜像。

可以在Envoy提供的镜像基础上，制作一个新的镜像，用自己的配置覆盖原先的配置，Dockerfile如下：

FROM envoyproxy/envoy:v1.8.0
    COPY envoy.yaml /etc/envoy/envoy.yaml

可以用下面的脚本运行：

IMAGE=envoylocal:v1.8.0
    docker build -t $IMAGE  ./
    docker run -d --rm --network=host --cpuset-cpus=1,3 $IMAGE

#### 用yum直接安装envoy ####

[Envoy的编译][Envoy 1]过程很繁琐，放在后面介绍。

[vbatts/envoy ][vbatts_envoy]提供了envoy的rpm包，[spec][]，可以用yum直接安装envoy：

wget https://copr.fedorainfracloud.org/coprs/vbatts/envoy/repo/epel-7/vbatts-envoy-epel-7.repo
    mv vbatts-envoy-epel-7.repo /etc/yum.repos.d/
    yum install envoy

#### 用Docker镜像编译envoy ####

[Building Envoy with Bazel][]中介绍了编译方法，要安装[Bazel][]，一个项目构建工具，下面是CentOS上的安装方法：

wget https://copr.fedorainfracloud.org/coprs/vbatts/bazel/repo/epel-7/vbatts-bazel-epel-7.repo  
    mv vbatts-bazel-epel-7.repo   /etc/yum.repos.d/
    yum install bazel -y

Envoy用到了最新的C++语法，需要用5以及以上版本的gcc编译。这一点需要特别注意，CentOS默认安装的还是gcc 4.8。[gcc的编译][gcc]比较复杂，建议还是用envoy提供的docker镜像编译，编译环境还可以保持一致，这里就使用这种方式。

先下载源代码：

git clone https://github.com/envoyproxy/envoy.git
    cd envoy
    git checkout v1.8.0

然后进入envoy目录进行编译：

cd envoy
    ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.release.server_only'

编译得到的envoy位于build\_release目录中：

$ ls -lh build_release
    -r-xr-xr-x 1 root root 367M Dec 14 15:43 build_release/envoy

`./ci/do_ci.sh`脚本中有多个构建目标，例如bazel.debug、bazel.release、bazel.release.server\_only等，根据自己需要选择。

默认的编译镜像是ubuntu，用它编译得到的envoy依赖glibc-2.8，在使用glibc-2.7的centos7上运行时会遇到动态链接库不匹配的问题：

$/tmp/envoy-docker-build/envoy/source/exe/envoy-fastbuild -h
    /tmp/envoy-docker-build/envoy/source/exe/envoy-fastbuild: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /tmp/envoy-docker-build/envoy/source/exe/envoy-fastbuild)

这个问题没有找到好的解决方法，后面章节有记录当时的调查过程，后来发现envoy还有一个基于centos的编译镜像：

$ docker search envoyproxy |grep envoyproxy |grep build
    envoyproxy/envoy-build                                                          1
    envoyproxy/envoy-build-centos                                                   1
    gonitro/envoyproxy              A build of the Envoy proxy container that is…   1
    envoyproxy/envoy-build-ubuntu                                                   0

用这个镜像编译得到的envoy，在centos上运行时没有依赖库不匹配的问题。通过环境变量设置要使用的编译镜像：

export ENVOY_DOCKER_BUILD_DIR=/data/envoy/envoy-docker-build
    export IMAGE_NAME=envoyproxy/envoy-build-centos
    export IMAGE_ID=latest
    ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.dev'

变量`ENVOY_DOCKER_BUILD_DIR`是编译生成的文件的存放目录。

#### 自己在容器外准备编译环境 ####

如果用envory提供的镜像编译，只需要一行命令：

./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.dev'

需要对这个命令背后发生的事情有所了解，搞清楚编译过程，才能知道从哪里开始阅读代码，以及怎样修改、添加代码。

脚本`./ci/run_envoy_docker.sh`的用途是启动编译容器，这个脚本的参数是要在容器内执行的命令：

# ...  省略部分内容 ...
    docker run --rm -t -i -e HTTP_PROXY=${http_proxy} -e HTTPS_PROXY=${https_proxy} \
      -u "${USER}":"${USER_GROUP}" -v "${ENVOY_DOCKER_BUILD_DIR}":/build \
      -v "$PWD":/source -e NUM_CPUS --cap-add SYS_PTRACE "${IMAGE_NAME}":"${IMAGE_ID}" \
      /bin/bash -lc "groupadd --gid $(id -g) -f envoygroup && useradd -o --uid $(id -u) --gid $(id -g) \
      --no-create-home --home-dir /source envoybuild && su envoybuild -c \"cd source && $*\""

可以看到，它就是将当前目录挂载到了容器的source目录：`-v "$PWD":/source`，然后在容器内的source目录中执行传入的参数：`cd source && $*`。

在容器内执行的`./ci/do_cli.sh`脚本才是重点，以`bazel.dev`目标为例：

# ./ci/do_cli.sh
    elif [[ "$1" == "bazel.dev" ]]; then
      setup_clang_toolchain
      # This doesn't go into CI but is available for developer convenience.
      echo "bazel fastbuild build with tests..."
      cd "${ENVOY_CI_DIR}"
      echo "Building..."
      bazel build ${BAZEL_BUILD_OPTIONS} -c fastbuild //source/exe:envoy-static
      # Copy the envoy-static binary somewhere that we can access outside of the
      # container for developers.
      cp -f \
        "${ENVOY_CI_DIR}"/bazel-bin/source/exe/envoy-static \
        "${ENVOY_DELIVERY_DIR}"/envoy-fastbuild
      echo "Building and testing..."
      bazel test ${BAZEL_TEST_OPTIONS} -c fastbuild //test/...
      exit 0

函数`setup_clang_toolchain`和变量`ENVOY_CI_DIR`是在`./ci/build_setup.sh`中定义的：

# ./ci/build_setup.sh
    function setup_clang_toolchain() {
      export PATH=/usr/lib/llvm-7/bin:$PATH
      export CC=clang
      export CXX=clang++
      export ASAN_SYMBOLIZER_PATH=/usr/lib/llvm-7/bin/llvm-symbolizer
      echo "$CC/$CXX toolchain configured"
    }
    # This is where we build for bazel.release* and bazel.dev.
    export ENVOY_CI_DIR="${ENVOY_SRCDIR}"/ci

因此最终是在`./ci`目录中执行下面两行bazel命令的：

bazel build ${BAZEL_BUILD_OPTIONS} -c fastbuild //source/exe:envoy-static
      bazel test ${BAZEL_TEST_OPTIONS} -c fastbuild //test/...

目标`//source/exe:envoy-static`位于`source/exe/BUILD`文件中：

envoy_cc_binary(
        name = "envoy-static",
        stamped = True,
        deps = ["envoy_main_entry_lib"],
    )
    ```conf
    
    在`source/exe/BUILD`中继续找deps中的envoy_main_entry_lib：
    
    ```conf
    envoy_cc_library(
        name = "envoy_main_entry_lib",
        srcs = ["main.cc"],
        external_deps = [
            "abseil_symbolize",
        ],
        deps = [
            ":envoy_main_common_lib",
        ],
    )

到这里，我们知道了`source/exe/main.cc`是程序的入口。

编译得到的文件在哪里呢？`do_cli.sh`中的命令将其拷贝到了`${ENVOY_DELIVERY_DIR}`目录：

cp -f \
        "${ENVOY_CI_DIR}"/bazel-bin/source/exe/envoy-static \
        "${ENVOY_DELIVERY_DIR}"/envoy-fastbuild

`ENVOY_DELIVERY_DIR`在`ci/build_setup.sh`中定义：

export BUILD_DIR=/build
    if [[ ! -d "${BUILD_DIR}" ]]
    then
      echo "${BUILD_DIR} mount missing - did you forget -v <something>:${BUILD_DIR}? Creating."
      mkdir -p "${BUILD_DIR}"
    fi
    
    export ENVOY_BUILD_DIR="${BUILD_DIR}"/envoy
    mkdir -p "${ENVOY_BUILD_DIR}"
    cp -f "${ENVOY_SRCDIR}"/ci/WORKSPACE "${ENVOY_BUILD_DIR}"
    
    export ENVOY_DELIVERY_DIR="${ENVOY_BUILD_DIR}"/source/exe
    mkdir -p "${ENVOY_DELIVERY_DIR}"

这里的`/build`目录是容器中目录，在启动容器的时候，将容器外的目录挂载到了/build上：

[[ -z "${ENVOY_DOCKER_BUILD_DIR}" ]] && ENVOY_DOCKER_BUILD_DIR=/tmp/envoy-docker-build
    
    mkdir -p "${ENVOY_DOCKER_BUILD_DIR}"
    # Since we specify an explicit hash, docker-run will pull from the remote repo if missing.
    docker run --rm -t -i -e HTTP_PROXY=${http_proxy} -e HTTPS_PROXY=${https_proxy} \
      -u "${USER}":"${USER_GROUP}" -v "${ENVOY_DOCKER_BUILD_DIR}":/build \
      -v "$PWD":/source -e NUM_CPUS --cap-add SYS_PTRACE "${IMAGE_NAME}":"${IMAGE_ID}" \
      /bin/bash -lc "groupadd --gid $(id -g) -f envoygroup && useradd -o --uid $(id -u) --gid $(id -g) \
      --no-create-home --home-dir /source envoybuild && su envoybuild -c \"cd source && $*\""

`${ENVOY_DOCKER_BUILD_DIR}`就是编译得到的文件的存放路径，默认是`/tmp/envoy-docker-build`，在里面可以找到编译得到的文件。

#### 在CentOS上构建编译环境 ####

在容器中编译时，最终执行的构建命令是：

NUM_CPUS=`grep -c ^processor /proc/cpuinfo`
    ENVOY_SRCDIR=envoy项目根路径
    bazel build --strategy=Genrule=standalone --spawn_strategy=standalone --verbose_failures \
    --package_path %workspace%:${ENVOY_SRCDIR} --action_env=HOME --action_env=PYTHONUSERBASE \
    --jobs=${NUM_CPUS} --show_task_finish  -c fastbuild //source/exe:envoy-static

如果在centos上编译，需要安装[依赖][Link 12]：

yum install cmake libtool libstdc++ ninja-build lld patch
    go get github.com/bazelbuild/buildtools/buildifier

git需要使用支持`-C`参数的版本，CentOS 7.2默认安装的是git 1.8没有-C参数，会有下面的错误：

+ git -C /root/.cache/bazel/_bazel_root/644de998b20ecf2f1d2ac88b14ed30b9/external/com_github_golang_protobuf fetch origin aa810b61a9c79d51363740d207bb46cf8e620ed5:aa810b61a9c79d51363740d207bb46cf8e620ed5
    Unknown option: -C
    usage: git [--version] [--help] [-c name=value]
               [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
               [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
               [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
               <command> [<args>]

将原先的git删除，然后安装[ius.io][]中的`git2u`：

yum erase git 
    yum install https://centos7.iuscommunity.org/ius-release.rpm
    yum --disablerepo=* --disableexcludes=ius --enablerepo=ius search git2u
    yum install git2u

确保git有`-C`参数：

$ git -h
    Unknown option: -h
    usage: git [--version] [--help] [-C <path>] [-c name=value]
               [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
               [-p | --paginate | --no-pager] [--no-replace-objects] [--bare]
               [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
               <command> [<args>]

还需要安装buildifier：

go get github.com/bazelbuild/buildtools/buildifier

如果上面获取buildifier的命令出错，进入buildifier的代码目录中，用bazel编译：

cd $GOPATH/src/github.com/bazelbuild/buildtools/buildifier
    bazel build //buildifier
    cp bazel-bin/buildifier/linux_amd64_stripped/buildifier /usr/bin/

如果遇到下面的错误：

/data/envoy/envoy/ci/build_container/build_recipes/libevent.sh: line 33: ninja: command not found
    
    real	0m21.102s
    user	0m9.201s
    sys	0m6.128s
    make: *** [/data/bazel_cache/_bazel_root/cb5046fbf6899613ea85d6e4bf221570/external/envoy_deps_cache_2c744dffd279d7e9e0910ce594eb4f4f/libevent.dep] Error 1
    Successful build of /data/bazel_cache/_bazel_root/cb5046fbf6899613ea85d6e4bf221570/external/envoy_deps_cache_2c744dffd279d7e9e0910ce594eb4f4f/gperftools.dep
    make: Leaving directory `/data/bazel_cache/_bazel_root/cb5046fbf6899613ea85d6e4bf221570/external/envoy_deps'
    DEBUG: /data/envoy/envoy/bazel/repositories.bzl:121:5: External dep build exited with return code: 1
    DEBUG: /data/envoy/envoy/bazel/repositories.bzl:123:9:  External dependency build failed, check above log for errors and ensure all prerequisites at https://github.com/envoyproxy/envoy/blob/master/bazel/README.md#quick-start-bazel-build-for-developers are met.
    ERROR: /data/envoy/envoy/bazel/repositories.bzl:272:13: no such package '@envoy_deps//': External dep build failed and referenced by '//external:tcmalloc_and_profiler'
    ERROR: Analysis of target '//source/exe:envoy-static' failed; build aborted: Analysis failed
    INFO: Elapsed time: 53.766s
    INFO: 0 processes.
    FAILED: Build did NOT complete successfully (1 packages loaded, 2 targets configured)

需要安装ninja-build：

yum install -y  ninja-build

还需要做一个符号链接，默认只有`nginja-build`命令，没有`ninja`命令，：

$ which ninja-build
    /usr/bin/ninja-build
    $ ln -s /usr/bin/ninja-build /usr/bin/ninja

如果遇到下面的问题，说明gcc版本太低，需要升级到5以上版本：

gcc: error: unrecognized command line option '-std=c++14'
    Target //source/exe:envoy-static failed to build
    INFO: Elapsed time: 74.728s, Critical Path: 0.27s, Remote (0.00% of the time): [queue: 0.00%, setup: 0.00%, process: 0.00%]
    INFO: 1 process: 1 local.
    FAILED: Build did NOT complete successfully

### 配置文件 ###

配置文件是yaml格式的，envoy提供了一个配置文件样例[google\_com\_proxy.v2.yaml][google_com_proxy.v2.yaml]：

admin:
      access_log_path: /tmp/admin_access.log
      address:
        socket_address:
          protocol: TCP
          address: 127.0.0.1
          port_value: 9901
    static_resources:`
      listeners:
      - name: listener_0
        address:
          socket_address:
            protocol: TCP
            address: 0.0.0.0
            port_value: 10000
        filter_chains:
        - filters:
          - name: envoy.http_connection_manager
            config:
              stat_prefix: ingress_http
              route_config:
                name: local_route
                virtual_hosts:
                - name: local_service
                  domains: ["*"]
                  routes:
                  - match:
                      prefix: "/"
                    route:
                      host_rewrite: www.google.com
                      cluster: service_google
              http_filters:
              - name: envoy.router
      clusters:
      - name: service_google
        connect_timeout: 0.25s
        type: LOGICAL_DNS
        # Comment out the following line to test on v6 networks
        dns_lookup_family: V4_ONLY
        lb_policy: ROUND_ROBIN
        hosts:
          - socket_address:
              address: google.com
              port_value: 443
    tls_context: { sni: www.google.com }

配置文件由`admin`、`static_resources`和`clusters`三部分组成。

admin中配置的是envoy的管理地址，static\_resources中配置的是listener，clusters中配置的是listener中用到的cluster。[Configuration reference][]是envoy的配置参考手册。

### 代理性能测试 ###

测试环境与[API网关Kong学习笔记（十九）：Kong的性能测试][API_Kong_Kong]中的环境一致。在部署kong的机器上部署了envoy，配置文件envoy.yaml内容如下：

admin:
      access_log_path: /tmp/admin_access.log
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 9901
    static_resources:
      listeners:
      - name: listener_0
        address:
          socket_address:
            protocol: TCP
            address: 0.0.0.0
            port_value: 9000
        filter_chains:
        - filters:
          - name: envoy.http_connection_manager
            config:
              stat_prefix: ingress_http
              route_config:
                name: local_route
                virtual_hosts:
                - name: local_service
                  domains: ["webshell.com"]
                  routes:
                  - match:
                      prefix: "/"
                    route:
                      host_rewrite: webshell.com
                      cluster: service_webshell
              http_filters:
              - name: envoy.router
      clusters:
      - name: service_webshell
        connect_timeout: 0.25s
        type: LOGICAL_DNS
        # Comment out the following line to test on v6 networks
        dns_lookup_family: V4_ONLY
        lb_policy: ROUND_ROBIN
        hosts:
          - socket_address:
              address: 172.16.129.18
              port_value: 80

测试方法也与[API网关Kong学习笔记（十九）：Kong的性能测试][API_Kong_Kong]中的方法一致，upstream也相同。

注意：这里还是用siege压测的，后来发现siege的自身的效率特别低，压满对方一个核，自己也要耗用一个核。后来改用[wrk][]进行压测。最新的压测数据以后抽时间整理到博客上。

用[siege][]压测：

siege -c 16 -b -t 1M -H "Host: webshell.com"  10.10.64.58:9000/ping

并发（个）         1      2      4      8       10      12     14     16      18     32      50
    直连Pod（每秒)     2.3k   4.4k   7.8k   11.6k   11.8k   12k    12.3k  12.5k   12.4k  12.4k   12.7
    通过Nginx（每秒）  1.6k   3k     5.3k   8.4k    9.4k    10.5k  10.9k  11.9k   12k    12.8k   13.5k
    通过Kong（每秒)    1.2k   2.4k   4.2k   6.3k    6.5k    6.6k   6.7k   6.6k    6.6k   6.9k    7.1k
    
    Kong优化（每秒)    1.3k   0.0k   0.0k   0.0k    0.0k    0.0k   0.0k   7.9k    0.0k   8.2k    0.0k
    通过Envoy（每秒)   1.3k   0.0k   0.0k   0.0k    0.0k    0.0k   0.0k   7.3k    0.0k   8.4k    0.0k

单纯转发方面，Envoy和优化后的Kong相差不多，低于预期，Envoy的转发性能应当与Nginx不相上下才对，需要查询下资料，看看是否存在配置不当或理解有误的地方。

### 代理性能继续测试 ###

在Google Group中找到Matt Klein对Envoy性能测试的[解释][Link 13]： 默认情况下，envoy会开启一些比较影响性能的特性，比如request ID generation和dynamic stat generation，将开启了这些特性的envoy和nginx进行对比是不合理的。

参照Matt Klein给出的[配置文件][Link 14]，修改了envoy.yaml，将`generate_request_id`和`dynamic_stats`设置为false。

admin:
      access_log_path: /tmp/admin_access.log
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 9901
    static_resources:
      listeners:
      - name: listener_0
        address:
          socket_address:
            protocol: TCP
            address: 0.0.0.0
            port_value: 9000
        filter_chains:
        - filters:
          - name: envoy.http_connection_manager
            config:
              stat_prefix: ingress_http
              generate_request_id: false
              route_config:
                name: local_route
                virtual_hosts:
                - name: local_service
                  domains: ["webshell.com"]
                  routes:
                  - match:
                      prefix: "/"
                    route:
                      host_rewrite: webshell.com
                      cluster: service_webshell
              http_filters:
              - name: envoy.router
                type: decoder
                config:
                  dynamic_stats: false
      clusters:
      - name: service_webshell
        connect_timeout: 0.25s
        type: static
        lb_policy: ROUND_ROBIN
        hosts:
          - socket_address:
              address: 172.16.129.24
              port_value: 80

同样条件下nginx的处理能力是12k/s，envoy是8.2k/s，默认配置的envoy是7.3k/s。

[nginx、kong、enovy代理转发功能的性能测试结果对比][nginx_kong_enovy]中给出了更详细的测试结果。

### 问题记录 ###

#### /lib64/libc.so.6: version \`GLIBC\_2.18’ not found ####

文件默认是动态链接的：

$ file /tmp/envoy-docker-build/envoy/source/exe/envoy-fastbuild
    /tmp/envoy-docker-build/envoy/source/exe/envoy-fastbuild: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=5d25f466c3410c0dfa735d7d4358beb76b2da507, not stripped

在其它机器上运行自己编译的envoy可能遇到动态链接库不匹配的问题，例如：

[//source/exe:envoy-static is not statically linked][source_exe_envoy-static is not statically linked]中回答static链接的问题，envoy的static链接不包含系统库，如果要连接系统库，需要自己设置。

Matt Klein在[Statically linked envoy binary?][Statically linked envoy binary]中解释说，如果完全静态连接，会有问题：

> I strongly recommend not fully statically linking. I’ve run into problems in the past trying to do this. Basically, you really need to know what you are doing to have this work and not run into any production issues.
> 
> The only type of static linking we officially support is what is done in our official CI builds. If you want to go beyond this you are on your own. \`\`\`

说是可以用[glibc dependency creeped up to 2.18 ][glibc dependency creeped up to 2.18]中的解决方法，看了一圈是直接删除libc.so中的符号，[Relax glibc requirement to 2.17][]

# Remove the symbol __cxa_thread_atexit_impl from libc6 to remove
      # the dependency on glibc 2.18.
      sed -i 's/__cxa_thread_atexit_impl/__dead_beef_dead_beef___/g' \
          "${INSTALL_ROOT}/lib/x86_64-linux-gnu/libc.so.6"

但是这个修改方案后来又被移除了，改成了使用glibc2.17…[Relax glibc dependency to 2.17][]。

最后发现envoy还有一个centos的编译镜像：

换用CentOS镜像编译：

export ENVOY_DOCKER_BUILD_DIR=/data/envoy/envoy-docker-build
    export IMAGE_NAME=envoyproxy/envoy-build-centos
    export IMAGE_ID=latest
    ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.dev'

然后又遇到了下面的问题：

In file included from source/exe/main_common.cc:6:
    bazel-out/k8-fastbuild/bin/source/common/common/_virtual_includes/compiler_requirements_lib/common/common/compiler_requirements.h:14:2: error: "Your toolchain has set _GLIBCXX_USE_CXX11_ABI to a value that uses a std::string "           "implementation that is not thread-safe. This may cause rare and difficult-to-debug errors "       "if std::string is passed between threads in any way. If you accept this risk, you may define "    "ENVOY_IGNORE_GLIBCXX_USE_CXX11_ABI_ERROR=1 in your build."
    #error "Your toolchain has set _GLIBCXX_USE_CXX11_ABI to a value that uses a std::string "         \
     ^
    1 error generated.
    Target //source/exe:envoy-static failed to build
    INFO: Elapsed time: 409.560s, Critical Path: 22.00s
    INFO: 399 processes: 399 local.

直接修改./source/common/common/compiler\_requirements.h文件绕过，去掉下面第二行defined前面的`!`

#if defined(_GLIBCXX_USE_CXX11_ABI) && _GLIBCXX_USE_CXX11_ABI != 1 &&                              \
        !defined(ENVOY_IGNORE_GLIBCXX_USE_CXX11_ABI_ERROR)

### 来源：[Envoy Proxy使用介绍教程（一）: 新型L3~L7层访问代理软件Envoy的使用-李佶澳][Envoy Proxy_ _L3_L7_Envoy_-] ###

[Link 1]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html#%E7%9B%AE%E5%BD%95
[Link 2]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html#%E8%AF%B4%E6%98%8E
[Link 3]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html#%E4%B8%BB%E8%A6%81%E7%89%B9%E7%82%B9
[Link 4]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html#%E5%B7%A5%E4%BD%9C%E8%BF%87%E7%A8%8B
[Link 5]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html#%E9%83%A8%E7%BD%B2%E5%90%AF%E5%8A%A8
[docker]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html#%E7%94%A8docker%E5%90%AF%E5%8A%A8
[yum_envoy]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html#%E7%94%A8yum%E7%9B%B4%E6%8E%A5%E5%AE%89%E8%A3%85envoy
[Docker_envoy]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html#%E7%94%A8docker%E9%95%9C%E5%83%8F%E7%BC%96%E8%AF%91envoy
[Link 6]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html#%E8%87%AA%E5%B7%B1%E5%9C%A8%E5%AE%B9%E5%99%A8%E5%A4%96%E5%87%86%E5%A4%87%E7%BC%96%E8%AF%91%E7%8E%AF%E5%A2%83
[CentOS]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html#%E5%9C%A8centos%E4%B8%8A%E6%9E%84%E5%BB%BA%E7%BC%96%E8%AF%91%E7%8E%AF%E5%A2%83
[Link 7]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html#%E9%85%8D%E7%BD%AE%E6%96%87%E4%BB%B6
[Link 8]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html#%E4%BB%A3%E7%90%86%E6%80%A7%E8%83%BD%E6%B5%8B%E8%AF%95
[Link 9]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html#%E4%BB%A3%E7%90%86%E6%80%A7%E8%83%BD%E7%BB%A7%E7%BB%AD%E6%B5%8B%E8%AF%95
[Link 10]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html#%E9%97%AE%E9%A2%98%E8%AE%B0%E5%BD%95
[lib64_libc.so.6_ version _GLIBC_2.18_ not found]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html#lib64libcso6-version-glibc_218-not-found
[Link 11]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html#%E5%8F%82%E8%80%83
[envoy]: https://www.lijiaocn.com/soft/envoy/
[Envoy]: https://www.envoyproxy.io/
[https_www.envoyproxy.io_docs_envoy_latest_about_docs]: https://www.envoyproxy.io/docs/envoy/latest/about_docs
[Envoy 1]: https://www.envoyproxy.io/docs/envoy/latest/install/building#building
[vbatts_envoy]: https://copr.fedorainfracloud.org/coprs/vbatts/envoy/
[spec]: https://github.com/vbatts/copr-build-envoy
[Building Envoy with Bazel]: https://github.com/envoyproxy/envoy/blob/master/bazel/README.md
[Bazel]: https://docs.bazel.build/versions/master/install.html
[gcc]: https://gcc.gnu.org/releases.html
[Link 12]: https://github.com/envoyproxy/envoy/blob/master/bazel/README.md#quick-start-bazel-build-for-developers
[ius.io]: https://ius.io/GettingStarted/
[google_com_proxy.v2.yaml]: https://github.com/envoyproxy/envoy/blob/master/configs/google_com_proxy.v2.yaml
[Configuration reference]: https://www.envoyproxy.io/docs/envoy/latest/configuration/configuration
[API_Kong_Kong]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/03/kong-features-19-kong-performance.html#%E7%8E%AF%E5%A2%83
[wrk]: https://www.lijiaocn.com/%E6%96%B9%E6%B3%95/2018/11/02/webserver-benchmark-method.html#wrk
[siege]: https://www.lijiaocn.com/%E6%96%B9%E6%B3%95/2018/11/02/webserver-benchmark-method.html#siege
[Link 13]: https://groups.google.com/forum/#!topic/envoy-users/Tgd3HZMfIiU
[Link 14]: https://groups.google.com/d/msg/envoy-users/Tgd3HZMfIiU/rsUFyyf2EQAJ
[nginx_kong_enovy]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/18/nginx-kong-enovy-perfomances.html
[source_exe_envoy-static is not statically linked]: https://github.com/envoyproxy/envoy/issues/1250
[Statically linked envoy binary]: https://groups.google.com/d/msg/envoy-users/XK9M-RhWrxg/bgvUqtG2BwAJ
[glibc dependency creeped up to 2.18]: https://bugs.chromium.org/p/chromium/issues/detail?id=749077
[Relax glibc requirement to 2.17]: https://chromium.googlesource.com/chromium/src.git/+/c2e32f4639a6c56eef7fc4f65160ac2896eca0f1%5E%21/#F5
[Relax glibc dependency to 2.17]: https://chromium.googlesource.com/chromium/src.git/+/e158b0ebe8ba891c971dd2f3fbddfccd696af119%5E%21/#F1
[Envoy Proxy_ _L3_L7_Envoy_-]: https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2018/12/12/envoy-01-usage.html