一个Spring Boot Admin 监控多个Nacos集群

今天药忘吃喽~ 2024-04-20 11:47 73阅读 0赞

## 背景 ##

我们有多个系统，每个系统一个集群，每个集群都部署了自己的Spring Boot Admin（以下简称Admin），用起来不仅不方便，私有化部署的时候还得多部署几个服务，为了解决这个问题，我想到了是否可以用一个Admin同时监控多个集群，这里集群指监控Nacos集群。

## 实现 ##

通过查看Nacos的服务注册源码、Admin监控的服务发现源码，最终得出结论：重写NacosServiceManager、NamingService类，即可实现。

*  为了监控多个Namespace，nacos的服务发现配置通过**分号分割**即可
 *  为了区别与原来只能订阅单个Namespace，将所有的重写类定义为**Multixxx**
 *  将自定义的**MultiNacosServiceManager**类定义为主要Bean

**MultiNacosServiceManager**  
这个类用来管理NamingService，包括创建NamingService，NamingMaintainService。

import com.alibaba.cloud.nacos.NacosServiceManager;
    import com.alibaba.nacos.api.NacosFactory;
    import com.alibaba.nacos.api.exception.NacosException;
    import com.alibaba.nacos.api.naming.NamingMaintainService;
    import com.alibaba.nacos.api.naming.NamingService;
    import com.alibaba.nacos.client.naming.NacosNamingService;
    import org.apache.commons.lang3.SerializationUtils;
    
    import java.util.*;
    
    import static com.alibaba.nacos.api.PropertyKeyConst.NAMESPACE;
    
    public class MultiNacosServiceManager extends NacosServiceManager {
        
    
        //namespace分隔符
        public static final String SEMICOLON = ";";
    
        private MultiNacosNamingService multiNacosNamingService;
    
        @Override
        public NamingService getNamingService(Properties properties) {
        
            if (Objects.isNull(this.multiNacosNamingService)) {
        
                multiNacosNamingService = buildNamingService(properties);
            }
            return multiNacosNamingService;
        }
    
        //这个服务就只取第一个了，简单点
        @Override
        public NamingMaintainService getNamingMaintainService(Properties properties) {
        
            String namespace = properties.getProperty(NAMESPACE);
            if (namespace.contains(SEMICOLON)) {
        
                String[] namespaces = namespace.split(";");
                properties.setProperty(NAMESPACE, namespaces[0]);
            }
            return super.getNamingMaintainService(properties);
        }
    
        private MultiNacosNamingService buildNamingService(Properties properties) {
        
            if (Objects.isNull(multiNacosNamingService)) {
        
                synchronized (MultiNacosServiceManager.class) {
        
                    if (Objects.isNull(multiNacosNamingService)) {
        
                        try {
        
                            String namespace = properties.getProperty(NAMESPACE);
                            if (namespace.contains(SEMICOLON)) {
        
                                List<NacosNamingService> multiNacosNamingService = new ArrayList<>();
                                //每个namespace创建一个namingService
                                for (String ns : namespace.split(SEMICOLON)) {
        
                                    Properties newProperties = SerializationUtils.clone(properties);
                                    newProperties.setProperty(NAMESPACE, ns);
                                    NacosNamingService namingService = (NacosNamingService) NacosFactory.createNamingService(newProperties);
                                    multiNacosNamingService.add(namingService);
                                }
                                return new MultiNacosNamingService(multiNacosNamingService);
                            } else {
        
                                NacosNamingService namingService = (NacosNamingService) NacosFactory.createNamingService(properties);
                                return new MultiNacosNamingService(Collections.singletonList(namingService));
                            }
                        } catch (NacosException e) {
        
                            throw new RuntimeException(e);
                        }
                    }
                }
            }
    
            return multiNacosNamingService;
        }
    }

**MultiNacosNamingService**  
将多个 nacosNamingService 组合为一个对外提供服务，原有的NamingService 只支持单个namespace，将原来有NamingService方法都重写为支持多个namespace。

import com.alibaba.nacos.api.exception.NacosException;
    import com.alibaba.nacos.api.naming.NamingService;
    import com.alibaba.nacos.api.naming.listener.EventListener;
    import com.alibaba.nacos.api.naming.pojo.Instance;
    import com.alibaba.nacos.api.naming.pojo.ListView;
    import com.alibaba.nacos.api.naming.pojo.ServiceInfo;
    import com.alibaba.nacos.api.selector.AbstractSelector;
    import com.alibaba.nacos.client.naming.NacosNamingService;
    
    import java.util.ArrayList;
    import java.util.List;
    
    import static de.codecentric.boot.admin.server.domain.values.StatusInfo.STATUS_DOWN;
    import static de.codecentric.boot.admin.server.domain.values.StatusInfo.STATUS_UP;
    
    
    /**
     * 将多个 nacosNamingService 组合为一个对外提供服务
     */
    public class MultiNacosNamingService implements NamingService {
        
    
        private List<NacosNamingService> nacosNamingServices;
    
        public MultiNacosNamingService(List<NacosNamingService> nacosNamingServices) {
        
            this.nacosNamingServices = nacosNamingServices;
        }
    
        @Override
        public void registerInstance(String serviceName, String ip, int port) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.registerInstance(serviceName, ip, port);
            }
        }
    
        @Override
        public void registerInstance(String serviceName, String groupName, String ip, int port) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.registerInstance(serviceName, groupName, ip, port);
            }
        }
    
        @Override
        public void registerInstance(String serviceName, String ip, int port, String clusterName) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.registerInstance(serviceName, ip, port, clusterName);
            }
        }
    
        @Override
        public void registerInstance(String serviceName, String groupName, String ip, int port, String clusterName) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.registerInstance(serviceName, groupName, ip, port, clusterName);
            }
        }
    
        @Override
        public void registerInstance(String serviceName, Instance instance) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.registerInstance(serviceName, instance);
            }
        }
    
        @Override
        public void registerInstance(String serviceName, String groupName, Instance instance) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.registerInstance(serviceName, groupName, instance);
            }
        }
    
        @Override
        public void deregisterInstance(String serviceName, String ip, int port) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.deregisterInstance(serviceName, ip, port);
            }
        }
    
        @Override
        public void deregisterInstance(String serviceName, String groupName, String ip, int port) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.deregisterInstance(serviceName, groupName, ip, port);
            }
        }
    
        @Override
        public void deregisterInstance(String serviceName, String ip, int port, String clusterName) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.deregisterInstance(serviceName, ip, port, clusterName);
            }
        }
    
        @Override
        public void deregisterInstance(String serviceName, String groupName, String ip, int port, String clusterName) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.deregisterInstance(serviceName, groupName, ip, port, clusterName);
            }
        }
    
        @Override
        public void deregisterInstance(String serviceName, Instance instance) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.deregisterInstance(serviceName, instance);
            }
        }
    
        @Override
        public void deregisterInstance(String serviceName, String groupName, Instance instance) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.deregisterInstance(serviceName, groupName, instance);
            }
        }
    
        @Override
        public List<Instance> getAllInstances(String serviceName) throws NacosException {
        
            List<Instance> instances = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                instances.addAll(nacosNamingService.getAllInstances(serviceName));
            }
            return instances;
        }
    
        @Override
        public List<Instance> getAllInstances(String serviceName, String groupName) throws NacosException {
        
            List<Instance> instances = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                instances.addAll(nacosNamingService.getAllInstances(serviceName, groupName));
            }
            return instances;
        }
    
        @Override
        public List<Instance> getAllInstances(String serviceName, boolean subscribe) throws NacosException {
        
            List<Instance> instances = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                instances.addAll(nacosNamingService.getAllInstances(serviceName, subscribe));
            }
            return instances;
        }
    
        @Override
        public List<Instance> getAllInstances(String serviceName, String groupName, boolean subscribe) throws NacosException {
        
            List<Instance> instances = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                instances.addAll(nacosNamingService.getAllInstances(serviceName, groupName, subscribe));
            }
            return instances;
        }
    
        @Override
        public List<Instance> getAllInstances(String serviceName, List<String> clusters) throws NacosException {
        
            List<Instance> instances = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                instances.addAll(nacosNamingService.getAllInstances(serviceName, clusters));
            }
            return instances;
        }
    
        @Override
        public List<Instance> getAllInstances(String serviceName, String groupName, List<String> clusters) throws NacosException {
        
            List<Instance> instances = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                instances.addAll(nacosNamingService.getAllInstances(serviceName, groupName, clusters));
            }
            return instances;
        }
    
        @Override
        public List<Instance> getAllInstances(String serviceName, List<String> clusters, boolean subscribe) throws NacosException {
        
            List<Instance> instances = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                instances.addAll(nacosNamingService.getAllInstances(serviceName, clusters, subscribe));
            }
            return instances;
        }
    
        @Override
        public List<Instance> getAllInstances(String serviceName, String groupName, List<String> clusters, boolean subscribe) throws NacosException {
        
            List<Instance> instances = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                instances.addAll(nacosNamingService.getAllInstances(serviceName, groupName, clusters, subscribe));
            }
            return instances;
        }
    
        @Override
        public List<Instance> selectInstances(String serviceName, boolean healthy) throws NacosException {
        
            List<Instance> instances = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                instances.addAll(nacosNamingService.selectInstances(serviceName, healthy));
            }
            return instances;
        }
    
        @Override
        public List<Instance> selectInstances(String serviceName, String groupName, boolean healthy) throws NacosException {
        
            List<Instance> instances = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                instances.addAll(nacosNamingService.selectInstances(serviceName, groupName, healthy));
            }
            return instances;
        }
    
        @Override
        public List<Instance> selectInstances(String serviceName, boolean healthy, boolean subscribe) throws NacosException {
        
            List<Instance> instances = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                instances.addAll(nacosNamingService.selectInstances(serviceName, healthy, subscribe));
            }
            return instances;
        }
    
        @Override
        public List<Instance> selectInstances(String serviceName, String groupName, boolean healthy, boolean subscribe) throws NacosException {
        
            List<Instance> instances = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                instances.addAll(nacosNamingService.selectInstances(serviceName, groupName, healthy, subscribe));
            }
            return instances;
        }
    
        @Override
        public List<Instance> selectInstances(String serviceName, List<String> clusters, boolean healthy) throws NacosException {
        
            List<Instance> instances = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                instances.addAll(nacosNamingService.selectInstances(serviceName, clusters, healthy));
            }
            return instances;
        }
    
        @Override
        public List<Instance> selectInstances(String serviceName, String groupName, List<String> clusters, boolean healthy) throws NacosException {
        
            List<Instance> instances = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                instances.addAll(nacosNamingService.selectInstances(serviceName, groupName, clusters, healthy));
            }
            return instances;
        }
    
        @Override
        public List<Instance> selectInstances(String serviceName, List<String> clusters, boolean healthy, boolean subscribe) throws NacosException {
        
            List<Instance> instances = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                instances.addAll(nacosNamingService.selectInstances(serviceName, clusters, healthy, subscribe));
            }
            return instances;
        }
    
        @Override
        public List<Instance> selectInstances(String serviceName, String groupName, List<String> clusters, boolean healthy, boolean subscribe) throws NacosException {
        
            List<Instance> instances = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                instances.addAll(nacosNamingService.selectInstances(serviceName, groupName, clusters, healthy, subscribe));
            }
            return instances;
        }
    
        @Override
        public Instance selectOneHealthyInstance(String serviceName) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                Instance instance = nacosNamingService.selectOneHealthyInstance(serviceName);
                if (instance != null) {
        
                    return instance;
                }
            }
            return null;
        }
    
        @Override
        public Instance selectOneHealthyInstance(String serviceName, String groupName) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                Instance instance = nacosNamingService.selectOneHealthyInstance(serviceName, groupName);
                if (instance != null) {
        
                    return instance;
                }
            }
            return null;
        }
    
        @Override
        public Instance selectOneHealthyInstance(String serviceName, boolean subscribe) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                Instance instance = nacosNamingService.selectOneHealthyInstance(serviceName, subscribe);
                if (instance != null) {
        
                    return instance;
                }
            }
            return null;
        }
    
        @Override
        public Instance selectOneHealthyInstance(String serviceName, String groupName, boolean subscribe) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                Instance instance = nacosNamingService.selectOneHealthyInstance(serviceName, groupName, subscribe);
                if (instance != null) {
        
                    return instance;
                }
            }
            return null;
        }
    
        @Override
        public Instance selectOneHealthyInstance(String serviceName, List<String> clusters) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                Instance instance = nacosNamingService.selectOneHealthyInstance(serviceName, clusters);
                if (instance != null) {
        
                    return instance;
                }
            }
            return null;
        }
    
        @Override
        public Instance selectOneHealthyInstance(String serviceName, String groupName, List<String> clusters) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                Instance instance = nacosNamingService.selectOneHealthyInstance(serviceName, groupName, clusters);
                if (instance != null) {
        
                    return instance;
                }
            }
            return null;
        }
    
        @Override
        public Instance selectOneHealthyInstance(String serviceName, List<String> clusters, boolean subscribe) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                Instance instance = nacosNamingService.selectOneHealthyInstance(serviceName, clusters, subscribe);
                if (instance != null) {
        
                    return instance;
                }
            }
            return null;
        }
    
        @Override
        public Instance selectOneHealthyInstance(String serviceName, String groupName, List<String> clusters, boolean subscribe) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                Instance instance = nacosNamingService.selectOneHealthyInstance(serviceName, groupName, clusters, subscribe);
                if (instance != null) {
        
                    return instance;
                }
            }
            return null;
        }
    
        @Override
        public void subscribe(String serviceName, EventListener listener) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.subscribe(serviceName, listener);
            }
        }
    
        @Override
        public void subscribe(String serviceName, String groupName, EventListener listener) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.subscribe(serviceName, groupName, listener);
            }
        }
    
        @Override
        public void subscribe(String serviceName, List<String> clusters, EventListener listener) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.subscribe(serviceName, clusters, listener);
            }
        }
    
        @Override
        public void subscribe(String serviceName, String groupName, List<String> clusters, EventListener listener) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.subscribe(serviceName, groupName, clusters, listener);
            }
        }
    
        @Override
        public void unsubscribe(String serviceName, EventListener listener) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.unsubscribe(serviceName, listener);
            }
        }
    
        @Override
        public void unsubscribe(String serviceName, String groupName, EventListener listener) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.unsubscribe(serviceName, groupName, listener);
            }
        }
    
        @Override
        public void unsubscribe(String serviceName, List<String> clusters, EventListener listener) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.unsubscribe(serviceName, clusters, listener);
            }
        }
    
        @Override
        public void unsubscribe(String serviceName, String groupName, List<String> clusters, EventListener listener) throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.unsubscribe(serviceName, groupName, clusters, listener);
            }
        }
    
        @Override
        public ListView<String> getServicesOfServer(int pageNo, int pageSize) throws NacosException {
        
            ListView<String> listView = new ListView<>();
            List<String> data = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                data.addAll(nacosNamingService.getServicesOfServer(pageNo, pageSize).getData());
            }
            listView.setData(data);
            return listView;
        }
    
        @Override
        public ListView<String> getServicesOfServer(int pageNo, int pageSize, String groupName) throws NacosException {
        
            ListView<String> listView = new ListView<>();
            List<String> data = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                data.addAll(nacosNamingService.getServicesOfServer(pageNo, pageSize, groupName).getData());
            }
            listView.setData(data);
            return listView;
        }
    
        @Override
        public ListView<String> getServicesOfServer(int pageNo, int pageSize, AbstractSelector selector) throws NacosException {
        
            ListView<String> listView = new ListView<>();
            List<String> data = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                data.addAll(nacosNamingService.getServicesOfServer(pageNo, pageSize, selector).getData());
            }
            listView.setData(data);
            return listView;
        }
    
        @Override
        public ListView<String> getServicesOfServer(int pageNo, int pageSize, String groupName, AbstractSelector selector) throws NacosException {
        
            ListView<String> listView = new ListView<>();
            List<String> data = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                data.addAll(nacosNamingService.getServicesOfServer(pageNo, pageSize, groupName, selector).getData());
            }
            listView.setData(data);
            return listView;
        }
    
        @Override
        public List<ServiceInfo> getSubscribeServices() {
        
            List<ServiceInfo> data = new ArrayList<>();
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                data.addAll(nacosNamingService.getSubscribeServices());
            }
            return data;
        }
    
        @Override
        public String getServerStatus() {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                String serverStatus = nacosNamingService.getServerStatus();
                if (STATUS_DOWN.equals(serverStatus)) {
        
                    return STATUS_DOWN;
                }
            }
            return STATUS_UP;
        }
    
    
        @Override
        public void shutDown() throws NacosException {
        
            for (NacosNamingService nacosNamingService : nacosNamingServices) {
        
                nacosNamingService.shutDown();
            }
        }
    }

**MultiNacosServiceAutoConfiguration**  
将MultiNacosServiceManager 设置为自动加载Bean，激活为主要的Bean。

import com.alibaba.cloud.nacos.ConditionalOnNacosDiscoveryEnabled;
    import com.alibaba.cloud.nacos.NacosServiceManager;
    import org.springframework.cloud.client.ConditionalOnDiscoveryEnabled;
    import org.springframework.context.annotation.Bean;
    import org.springframework.context.annotation.Configuration;
    import org.springframework.context.annotation.Primary;
    
    @Configuration(proxyBeanMethods = false)
    @ConditionalOnDiscoveryEnabled
    @ConditionalOnNacosDiscoveryEnabled
    public class MultiNacosServiceAutoConfiguration {
        
    
        @Bean
        @Primary
        public NacosServiceManager multiNacosServiceManager() {
        
            return new MultiNacosServiceManager();
        }
    }

## 总结 ##

最终你能发现admin监控会同时注册到多个集群中，admin服务列表能看到多个集群的服务。另外要注意的是，要适当调整admin监控服务的内存，毕竟监控的服务变多了。  
通过一个月的运行，目前admin监控运行稳定，相关功能一切正常。

--------------------

> 其他企业级监控：  
> Prometheus 系列文章
> 
> 1.  [Prometheus 的介绍和安装][Prometheus]
> 2.  [直观感受PromQL及其数据类型][PromQL]
> 3.  [PromQL之选择器和运算符][PromQL 1]
> 4.  [PromQL之函数][PromQL 2]
> 5.  [Prometheus 告警机制介绍及命令解读][Prometheus 1]
> 6.  [Prometheus 告警模块配置深度解析][Prometheus 2]
> 7.  [Prometheus 配置身份认证][Prometheus 3]
> 8.  [Prometheus 动态拉取监控服务][Prometheus 4]
> 9.  [Prometheus 监控云Mysql和自建Mysql][Prometheus _Mysql_Mysql]
> 
> --------------------
> 
> Grafana 系列文章，版本：OOS v9.3.1
> 
> 1.  [Grafana 的介绍和安装][Grafana]
> 2.  [Grafana监控大屏配置参数介绍（一）][Grafana 1]
> 3.  [Grafana监控大屏配置参数介绍（二）][Grafana 2]
> 4.  [Grafana监控大屏可视化图表][Grafana 3]
> 5.  [Grafana 查询数据和转换数据][Grafana 4]
> 6.  [Grafana 告警模块介绍][Grafana 5]
> 7.  [Grafana 告警接入飞书通知][Grafana 6]
> 
> --------------------

[Prometheus]: https://itsaysay.blog.csdn.net/article/details/128993876
[PromQL]: https://itsaysay.blog.csdn.net/article/details/128697766
[PromQL 1]: https://itsaysay.blog.csdn.net/article/details/128725788
[PromQL 2]: https://itsaysay.blog.csdn.net/article/details/128734102
[Prometheus 1]: https://itsaysay.blog.csdn.net/article/details/129035181
[Prometheus 2]: https://itsaysay.blog.csdn.net/article/details/129115894
[Prometheus 3]: https://itsaysay.blog.csdn.net/article/details/128741317
[Prometheus 4]: https://itsaysay.blog.csdn.net/article/details/128748383
[Prometheus _Mysql_Mysql]: https://itsaysay.blog.csdn.net/article/details/129248268
[Grafana]: https://itsaysay.blog.csdn.net/article/details/128260639
[Grafana 1]: https://itsaysay.blog.csdn.net/article/details/128280232
[Grafana 2]: https://blog.csdn.net/weixin_40972073/article/details/128292597
[Grafana 3]: https://itsaysay.blog.csdn.net/article/details/128309734
[Grafana 4]: https://itsaysay.blog.csdn.net/article/details/128379173
[Grafana 5]: https://itsaysay.blog.csdn.net/article/details/128586826
[Grafana 6]: https://itsaysay.blog.csdn.net/article/details/128665497