SkyWalking 调研记录

曾经终败给现在 2022-10-29 03:30 61阅读 0赞

## 外部摘要 ##

**Nginx+SpringBoot反向代理，链路无法串联原因：因为agent中设置了namespace，导致获取http header头 sw8 时，发生异常。**

*  **原获取方式：request.getHeader("sw8") ，增加namespace后，应该变成 request.getHeader("xxxx-sw8")，故链路无法链接起来。**
 *  **sw8：包含了加密后的 traceId，span等信息**
 *  **解决方案：更改lib/skywalking目录下的 span.lua 和 tracer.lua 中的  sw8，替换为 xxxx-sw8**

**集群搭建教程（skywalking集群、nacos集群、es集群、nginx 负载均衡）：**[https://my.oschina.net/shadowolf/blog/4550365][https_my.oschina.net_shadowolf_blog_4550365]

**官方集群搭建博客：**[https://skywalking.apache.org/zh/2020-04-19-skywalking-quick-start/\#][https_skywalking.apache.org_zh_2020-04-19-skywalking-quick-start]

**SpringCloud+SkyWalking链路追踪集合：**[https://www.iocoder.cn/Spring-Cloud/SkyWalking/][https_www.iocoder.cn_Spring-Cloud_SkyWalking]

**全链路监控：**[https://blog.csdn.net/weixin\_39944375/article/details/111581268][https_blog.csdn.net_weixin_39944375_article_details_111581268]

**Skywalking系列博客：**[http://www.itmuch.com/tags/Skywalking/][http_www.itmuch.com_tags_Skywalking]

skywalking（集群）：Tag-8.3.0，es：7（集群），nacos（注册配置中心+mysql持久化），下载地址：[https://github.com/apache/skywalking/tags][https_github.com_apache_skywalking_tags]

模块：RocketBotUI，OAPServer，JavaAgent（oracle plugin单独获取）

example：[https://www.cnblogs.com/MikeYao/p/14374264.html][https_www.cnblogs.com_MikeYao_p_14374264.html]

1、javaAgent介绍及使用，包括plugins以及配置属性介绍

[https://github.com/apache/skywalking/blob/master/docs/en/setup/service-agent/java-agent/README.md][https_github.com_apache_skywalking_blob_master_docs_en_setup_service-agent_java-agent_README.md]

2、Oracle agent-plugins、

[https://github.com/SkyAPM/java-plugin-extensions][https_github.com_SkyAPM_java-plugin-extensions]

2.1、Nginx Plugin、Openresty安装

[https://github.com/apache/skywalking-nginx-lua][https_github.com_apache_skywalking-nginx-lua]

[https://openresty.org/cn/installation.html][https_openresty.org_cn_installation.html]

[https://zhuanlan.zhihu.com/p/111523894][https_zhuanlan.zhihu.com_p_111523894]

3、插件模块介绍及外部插件获取方式

[https://www.cnblogs.com/kebibuluan/p/13151144.html][https_www.cnblogs.com_kebibuluan_p_13151144.html]

4、B站视频

[https://www.bilibili.com/video/BV1ZJ411s7Mn?p=5][https_www.bilibili.com_video_BV1ZJ411s7Mn_p_5]

**5、告警配置相关**

[https://blog.csdn.net/z69183787/article/details/113769229][https_blog.csdn.net_z69183787_article_details_113769229]

6、前端告警上报功能

[SkyWalking 前端监控的应用 | 云原生社区 (cloudnative.to)][SkyWalking _ _ _ _cloudnative.to]

[https://skywalking.apache.org/zh/2020-10-29-skywalking8-2-release/][https_skywalking.apache.org_zh_2020-10-29-skywalking8-2-release]

7、ES7集群部署+SkyWalking部署

[https://blog.csdn.net/alanzy123/article/details/109100936][https_blog.csdn.net_alanzy123_article_details_109100936]

7.1、关于集群部署

[https://blog.csdn.net/yy756127197/article/details/107463560][https_blog.csdn.net_yy756127197_article_details_107463560]

7.2、集群相关配置属性

[https://skywalking-handbook.netlify.app/installation/configuration/][https_skywalking-handbook.netlify.app_installation_configuration]

8、配合日志框架打印 traceId、span

[https://lux-sun.blog.csdn.net/article/details/109220200][https_lux-sun.blog.csdn.net_article_details_109220200]

[https://blog.csdn.net/xixingzhe2/article/details/114640591][https_blog.csdn.net_xixingzhe2_article_details_114640591]

8.1、上报请求参数

[https://blog.csdn.net/qq\_37362891/article/details/107191901][https_blog.csdn.net_qq_37362891_article_details_107191901]

9、调用链忽略跟踪（心跳等）

[https://blog.csdn.net/z69183787/article/details/114698808][https_blog.csdn.net_z69183787_article_details_114698808]

10、SkyWalking 数据清理机制(TTL) elasticsearch

[https://my.oschina.net/u/2344188/blog/4678400][https_my.oschina.net_u_2344188_blog_4678400]

**后端（Oap）又分为三个角色**

*  混合Mixed（默认）：默认角色，OAP应承担以下责任，1.接收代理跟踪或指标，2.进行L1聚合，3.内部通讯（发送/接收），4.进行L2聚合，5.持久化，6.报警
 *  接收者Receiver：1.接收代理跟踪或指标，2.进行L1聚合，3.内部通讯（发送/接收）
 *  聚合器Aggregator：4.进行L2聚合，5.持久化，6.报警
 *      Mixed: Receive agent data, Level 1 aggregate, Level 2 aggregate
            Receiver: Receive agent data, Level 1 aggregate
            Aggregator: Level 2 aggregate
            后端集群角色配置，默认 Mixed 代表 Receiver 和 Aggregator 角色共存。
            当我们的微服务规模较大的时候，可以通过调整 Receiver 和 Aggregator 集群来指责分离，提高 OAP 集群接收数据的效率。

`SW_CORE_ROLE` 在集群模式下可以还可以设置成Receiver和Aggregator来提高系统性能。  
如果集群只有Receiver节点，在收集数据时会报错，因为内部通讯（发送/接收）找不到接收节点而失败。

11、SkyWalking 存储介绍 及相关配置调优说明

[https://blog.csdn.net/smooth00/article/details/96479544][https_blog.csdn.net_smooth00_article_details_96479544]

12、ES7-Head插件安装：自行百度，需安装npm+grunt

13、实操，agent上报+数据应用

[https://www.yuque.com/liuzhiqing/note/voksxx][https_www.yuque.com_liuzhiqing_note_voksxx]

14、SkyWalking 不使用 mq 进行agent与oap通信的原因？

[https://github.com/apache/skywalking/blob/master/docs/en/FAQ/why\_mq\_not\_involved.md][https_github.com_apache_skywalking_blob_master_docs_en_FAQ_why_mq_not_involved.md]

15、SkyWalking Query协议及 相关接口

graphql：[https://github.com/apache/skywalking-query-protocol][https_github.com_apache_skywalking-query-protocol]

说明文档：[https://github.com/apache/skywalking/blob/master/docs/en/protocols/query-protocol.md][https_github.com_apache_skywalking_blob_master_docs_en_protocols_query-protocol.md]

99、异常与问题

1.  Skywalking ContextManager.activeSpan抛NullPointerException的bug：[https://www.jianshu.com/p/52bbc503be19][https_www.jianshu.com_p_52bbc503be19]

## 相关配置文件变更 ##

### 1、boot-demo ###

controller：

package com.example.demo.controller;
    
    import org.springframework.beans.factory.annotation.Autowired;
    import org.springframework.web.bind.annotation.*;
    
    import java.util.HashMap;
    import java.util.Map;
    import java.lang.Thread;
    
    import java.util.HashMap;
    import java.util.Iterator;
    import java.util.Map;
    import java.util.Map.Entry;
    import java.util.Set;
    import java.util.List;
    
    @RestController
    public class DemoController {
    
    /**
         * demo
        * 用于调试agent是否上报成功
         */
        @GetMapping("/demo/{id}")
        public Map<String,String> demo(@PathVariable("id") Integer id) throws Exception {
    System.out.println("Controller-demo receive id = "+id);
    Map<String,String> map = new HashMap();
    map.put("demo-id",id+"");
            return map;
        }
    
    /**
         * sleep
    * 用于调试告警信息及alarm
         */
        @GetMapping("/sleep/{id}")
        public Map<String,String> sleep(@PathVariable("id") Integer id) throws Exception {
    Thread.sleep(1500);
    System.out.println("Controller-sleep receive id = "+id);
    Map<String,String> map = new HashMap();
    map.put("sleep-id",id+"");
            return map;
        }
    
    
    
    /**
         * alarm
    * 接收来自skywalking的webhook
         */
        @PostMapping("/alarm")
        public List<Map<String,Object>> alarm(@RequestBody List<Map<String,Object>> list) throws Exception {
    //通过EntrySet取出map数据[for-each循环]
    		System.out.println("-------[for-each循环遍历]通过EntrySet取出map数据-------");
    		Set<Entry<String, Object>> entrys = list.get(0).entrySet();  //此行可省略，直接将map.entrySet()写在for-each循环的条件中
    		for(Entry<String, Object> entry:entrys){
    			System.out.println("key值："+entry.getKey()+" value值："+entry.getValue().toString());
    		}
            return list;
        }
    }

**启动命令**

java -javaagent:C:\Users\gy25\Desktop\skywalking-apm-bin-es7\agent\skywalking-agent.jar -jar demo-0.0.1-SNAPSHOT.jar

**启动参数说明/配置覆盖**

**配置覆盖**  
我们每次部署应用都需要复制一份agent，修改其中的服务名称，这样很麻烦。可以使用skywalking提供的配置覆盖功能通过启动命令动态指定服务名，这样agent只需要部署一份即可。skywalking支持的几种配置方式：

**1、系统配置(System properties)：**

使用skywalking.+配置文件中的配置名作为系统配置项来进行覆盖。

为什么需要添加前缀?

agent的系统配置和环境与目标应用共享，所以加上前缀可以有效避免冲突。

`java -Dxxxx=xxxx -Dxxxx=xxxx，`就是系统属性. 有关Skywalking的参数全部都要加 skywalking.+key作为属性的key.

案例，通过如下进行agent.service\_name的覆盖

-Dskywalking.agent.service_name=application_name

**2、探针配置（Agent options）:**  
add the properties after the agent path in JVMarguments

\-javaagent:/path/to/skywalking-agent.jar=\[option1\]=\[value1\],\[option2\]=\[value2\]

案例，通过如下进行agent.service\_name的覆盖

-javaagent:/path/to/skywalking-agent.jar=agent.service_name=application_name

特殊字符，如果配置中包含分割父（，或者=），就必须使用引号包裹起来

-javaagent:/path/to/skywalking-agent.jar=agent.ignore_suffix='.jpg,.jpeg'

**3、系统环境变量（system environment variables）**

案例，由于agent.service\_name配置项如下所示：

#The service name in UI
    agent.service_name=${SW_AGNET_NAME:Your_ApplicationName}

可以在环境变量中设置SW\_AGENT\_NAME的值来指定服务名。

覆盖优先级  
探针配置>系统配置>系统环境变量>配置文件中的值

### 2、SkyWalking-Agent ###

目录：skywalking-apm-bin-es7\\agent

配置文件：\\config\\agent.config

jar：skywalking-agent.jar

系统参数说明：[https://blog.csdn.net/lt326030434/article/details/107121511/][https_blog.csdn.net_lt326030434_article_details_107121511]

# 只列出了一些重要的
    # 以下属性都可以在javaagent 启动中，增加 -Dskywalking.agent.service_name=demo
    
    # The agent namespace，UI展示可分类，上报命名空间
    # agent.namespace=${SW_AGENT_NAMESPACE:default-namespace}
    
    # The service name in UI ，上报服务名
    agent.service_name=${SW_AGENT_NAME:DEMO}
    
    # Backend service addresses. 上报skywalking-oap地址
    collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:127.0.0.1:11800}
    
    # Logging file_name agent 日志名，默认logs目录下
    logging.file_name=${SW_LOGGING_FILE_NAME:skywalking-api.log}
    
    # Logging level
    logging.level=${SW_LOGGING_LEVEL:INFO}
    
    # Logging dir
    # logging.dir=${SW_LOGGING_DIR:""}

### 3、SkyWalking-Main（OAP+UI） ###

启动脚本：\\bin\\startup.bat

oap-server目录：skywalking-apm-bin-es7

oap-server配置文件：\\config\\application.yml

oap-server启动脚本：\\bin\\oapService.bat

oap-server可选plugin：skywalking-apm-bin-es7\\agent\\optional-plugins，移至 plugins 目录即可启用

核心类：oap-libs/\*，org.apache.skywalking.oap.server.starter.OAPServerStartUp

# 默认standalone单机模式，集群模式可选择注册中心（zk、nacos、etcd、consul等）
    
    #存储选用elasticsearch7，可以更改es7相关属性，集群或单机，namespace=cluster-name
    storage:
      selector: ${SW_STORAGE:elasticsearch7}
      elasticsearch7:
        nameSpace: ${SW_NAMESPACE:""}
        clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:localhost:9200}
    
    #配置中心，可选择（zk、nacos等，与注册中心雷同）
    configuration:
      selector: ${SW_CONFIGURATION:none}
    # 定义定期清理ES中数据的时间，单位为天，可根据需求自行延长
    core
        recordDataTTL: ${SW_CORE_RECORD_DATA_TTL:3} # Unit is day
        metricsDataTTL: ${SW_CORE_METRICS_DATA_TTL:7} # Unit is day

webapp-目录：skywalking-apm-bin-es7

webapp配置文件：\\webapp\\webapp.yml

jar：\\webapp\\skywalking-webapp.jar

webapp启动脚本：\\bin\\webappService.bat

# 配置主站入口
    server:
      port: 8088
    
    collector:
      path: /graphql
      ribbon:
        ReadTimeout: 10000
        # Point to all backend's restHost:restPort, split by ,
        # sw core-rest api地址
        listOfServers: 127.0.0.1:12800

### 4、SkyWalking-alarm ###

配置目录文件：skywalking-apm-bin-es7\\config\\alarm-settings.yml

# 下面的默认注释有些标注不是太清楚，service_resp_time 应该是最近5分钟内 所有上报应用
    # 的响应时间超过阈值 1000的次数大于 2次，即调用最后 webhooks中的 http请求。
    # 可选参数可以自行查阅官方文档
    rules:
      # Rule unique name, must be ended with `_rule`.
      service_resp_time_rule:
        metrics-name: service_resp_time
        op: ">"
        threshold: 1000
        period: 5
        count: 2
        silence-period: 2
        message: Response time of service {name} is more than 1000ms in 3 minutes of last 10 minutes.
      service_sla_rule:
        ...
        message: Successful rate of service {name} is lower than 80% in 2 minutes of last 10 minutes
      service_resp_time_percentile_rule:
        ...
        message: Percentile response time of service {name} alarm in 3 minutes of last 10 minutes, due to more than one condition of p50 > 1000, p75 > 1000, p90 > 1000, p95 > 1000, p99 > 1000
      service_instance_resp_time_rule:
        ...
        message: Response time of database access {name} is more than 1000ms in 2 minutes of last 10 minutes
      endpoint_relation_resp_time_rule:
        ...
        message: Response time of endpoint relation {name} is more than 1000ms in 2 minutes of last 10 minutes
    
    webhooks:
      - http://127.0.0.1:8080/alarm
    #  - http://127.0.0.1/notify/
    #  - http://127.0.0.1/go-wechat/

### 5、ES7 ###

1、单机版正常安装即可，此处先忽略，默认端口 9200

2、head插件需要优先安装 npm+grunt+head插件，默认端口 9100

[https_my.oschina.net_shadowolf_blog_4550365]: https://my.oschina.net/shadowolf/blog/4550365
[https_skywalking.apache.org_zh_2020-04-19-skywalking-quick-start]: https://skywalking.apache.org/zh/2020-04-19-skywalking-quick-start/#
[https_www.iocoder.cn_Spring-Cloud_SkyWalking]: https://www.iocoder.cn/Spring-Cloud/SkyWalking/
[https_blog.csdn.net_weixin_39944375_article_details_111581268]: https://blog.csdn.net/weixin_39944375/article/details/111581268
[http_www.itmuch.com_tags_Skywalking]: http://www.itmuch.com/tags/Skywalking/
[https_github.com_apache_skywalking_tags]: https://github.com/apache/skywalking/tags
[https_www.cnblogs.com_MikeYao_p_14374264.html]: https://www.cnblogs.com/MikeYao/p/14374264.html
[https_github.com_apache_skywalking_blob_master_docs_en_setup_service-agent_java-agent_README.md]: https://github.com/apache/skywalking/blob/master/docs/en/setup/service-agent/java-agent/README.md
[https_github.com_SkyAPM_java-plugin-extensions]: https://github.com/SkyAPM/java-plugin-extensions
[https_github.com_apache_skywalking-nginx-lua]: https://github.com/apache/skywalking-nginx-lua
[https_openresty.org_cn_installation.html]: https://openresty.org/cn/installation.html
[https_zhuanlan.zhihu.com_p_111523894]: https://zhuanlan.zhihu.com/p/111523894
[https_www.cnblogs.com_kebibuluan_p_13151144.html]: https://www.cnblogs.com/kebibuluan/p/13151144.html
[https_www.bilibili.com_video_BV1ZJ411s7Mn_p_5]: https://www.bilibili.com/video/BV1ZJ411s7Mn?p=5
[https_blog.csdn.net_z69183787_article_details_113769229]: https://blog.csdn.net/z69183787/article/details/113769229
[SkyWalking _ _ _ _cloudnative.to]: https://cloudnative.to/blog/end-user-tracing-in-a-skywalking-observed-browser/
[https_skywalking.apache.org_zh_2020-10-29-skywalking8-2-release]: https://skywalking.apache.org/zh/2020-10-29-skywalking8-2-release/
[https_blog.csdn.net_alanzy123_article_details_109100936]: https://blog.csdn.net/alanzy123/article/details/109100936
[https_blog.csdn.net_yy756127197_article_details_107463560]: https://blog.csdn.net/yy756127197/article/details/107463560
[https_skywalking-handbook.netlify.app_installation_configuration]: https://skywalking-handbook.netlify.app/installation/configuration/
[https_lux-sun.blog.csdn.net_article_details_109220200]: https://lux-sun.blog.csdn.net/article/details/109220200
[https_blog.csdn.net_xixingzhe2_article_details_114640591]: https://blog.csdn.net/xixingzhe2/article/details/114640591
[https_blog.csdn.net_qq_37362891_article_details_107191901]: https://blog.csdn.net/qq_37362891/article/details/107191901
[https_blog.csdn.net_z69183787_article_details_114698808]: https://blog.csdn.net/z69183787/article/details/114698808
[https_my.oschina.net_u_2344188_blog_4678400]: https://my.oschina.net/u/2344188/blog/4678400
[https_blog.csdn.net_smooth00_article_details_96479544]: https://blog.csdn.net/smooth00/article/details/96479544
[https_www.yuque.com_liuzhiqing_note_voksxx]: https://www.yuque.com/liuzhiqing/note/voksxx
[https_github.com_apache_skywalking_blob_master_docs_en_FAQ_why_mq_not_involved.md]: https://github.com/apache/skywalking/blob/master/docs/en/FAQ/why_mq_not_involved.md
[https_github.com_apache_skywalking-query-protocol]: https://github.com/apache/skywalking-query-protocol
[https_github.com_apache_skywalking_blob_master_docs_en_protocols_query-protocol.md]: https://github.com/apache/skywalking/blob/master/docs/en/protocols/query-protocol.md
[https_www.jianshu.com_p_52bbc503be19]: https://www.jianshu.com/p/52bbc503be19
[https_blog.csdn.net_lt326030434_article_details_107121511]: https://blog.csdn.net/lt326030434/article/details/107121511/