【SpringCloud】【Hoxton】Hystrix全面解析

叁歲伎倆 2023-07-15 02:47 65阅读 0赞

[01 基础环境准备][01]  
[02 一文读懂Eureka][02 _Eureka]  
[03 Zookeeper注册中心][03 Zookeeper]  
[04 Consule注册中心][04 Consule]  
[05 Ribbon][]  
[06 OpenFegin][]  
[07 Hystrix全面解析][07 Hystrix]  
[08 Gateway全面解析][08 Gateway]  
[09 Config配置中心][09 Config]  
[10 Bus消息总线][10 Bus]

# 1 分布式系统面临的问题 #

复杂分布式体系结构中的应用程序有数十个依赖关系,每个依赖关系在某些时候将不可避免地失败。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70]  
左图中的请求需要调用A,P,H,I 四个服务,如果一切顺利则没有什么问题,关键是如果I服务超时会出现什么情况呢?  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 1]  
(2) 雪崩  
多个微服务之间调用的时候,假设微服务A调用微服务B和微服务C,微服务B和微服务C又调用其它的微服务,这就是所谓的“扇出"。如果扇出的链路上某个微服务的调用响应时间过长或者不可用,对微服务A的调用就会占用越来越多的系统资源,进而引起系统崩溃,所谓的“雪崩效应”。所以通常当你发现一个模块下的某个实例失败后,这时候这个模块依然还会接收流量,然后这个有问题的模块还调用了其他的模块,这样就会发生级联故障,或者叫雪崩。

# 2 Hystrix是什么 #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 2]  
Hystrix是个用于处理分布式系统的延迟和容错的开源库,在分布式奈统里,许多依赖不可避免的会调用失败,比如超时、异常等。Hystrix能够保证在一个依赖出问题的情況下,不会导致整体服务失败,避免级联故障,以提高分布式系统的弹性。“断路器”本身是一种开关装置,当某个服务单元发生故障之后,通过断路器的故障监控(类似熔断保险丝),向调用方返回个符合预期的、可处理的备选响应( Fallback)，而不是长时间的等待或者抛出调用方无法处理的异常,这样就保证了服务调用方的线程不会被长时间、不必要地占用,从而避免了故障在分布式系统中的蔓延,乃至雪崩。

(1) 服务降级  
当服务器压力剧增的情况下，根据实际业务情况及流量，对一些服务和页面有策略的不处理或换种简单的方式处理，从而释放服务器资源以保证核心交易正常运作或高效运作。

使用场景：服务异常、超时、服务熔断触发服务降级、线程池/信号量打满也会导致服务降级

(2) 服务熔断  
高压电路中，如果某个地方的电压过高，熔断器就会熔断，对电路进行保护。股票交易中，如果股票指数过高，也会采用熔断机制，暂停股票的交易。同样，在微服务架构中，熔断机制也是起着类似的作用。当扇出链路的某个微服务不可用或者响应时间太长时，会进行服务的降级，进而熔断该节点微服务的调用，快速返回错误的响应信息。当检测到该节点微服务调用响应正常后，恢复调用链路。

(3) 服务限流  
限流的目的是通过对并发访问/请求进行限速或者一个时间窗口内的的请求进行限速来保护系统，一旦达到限制速率则可以拒绝服务（定向到错误页或告知资源没有了）、排队或等待（比如秒杀、评论、下单）、降级（返回兜底数据或默认数据，如商品详情页库存默认有货）。

# 3 新建工程 provider-hystrix #

![在这里插入图片描述][20200315014721530.png]

(1) pom

<dependencies>
        
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>runtime</scope>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
        </dependency>
        <dependency>
            <groupId>com.zrs.springcloud</groupId>
            <artifactId>commons</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>
    </dependencies>

(2) application.yml

server:
      port: 7004
    spring:
      application:
        name: provider-hystrix
    eureka:
      client:
        #是否将自己注册到EurekaServer
        register-with-eureka: true
        #是否从EurekaServer获取已有的注册服务
        fetch-registry: true
        #注册地址
        service-url:
          defaultZone: http://localhost:7000/eureka/
      instance:
        instance-id: provider-hystrix
        prefer-ip-address: true

(3) 主启动类

@SpringBootApplication
    @EnableEurekaClient
    public class HystrixApplication { 
        public static void main(String[] args) { 
            SpringApplication.run(HystrixApplication.class);
        }
    }

(4) Controller

@RestController
    @RequestMapping("/hystrix")
    public class HystrixController { 
        @GetMapping("/hello")
        public String hello(){ 
            return "Hello Thread："+Thread.currentThread().getName();
        }
        @GetMapping("/threeseconds")
        public String threeseconds(){ 
            try { 
                Thread.sleep(3000);
            } catch (InterruptedException e) { 
                e.printStackTrace();
            }
            return "hello threeseconds";
        }
    }

(5) 启动 测试  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 3]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 4]

# 4 新建工程customer-hystrix #

![在这里插入图片描述][20200315014835150.png]

## 4.1 pom ##

<dependencies>
        
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
        </dependency>
        
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
        </dependency>
        
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-openfeign</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>runtime</scope>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
        </dependency>
        <dependency>
            <groupId>com.zrs.springcloud</groupId>
            <artifactId>commons</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>
    </dependencies>

## 4.2 application.yml ##

server:
      port: 8004
    spring:
      application:
        name: customer-hystrix
    eureka:
      client:
        #是否将自己注册到EurekaServer
        register-with-eureka: true
        #是否从EurekaServer获取已有的注册服务
        fetch-registry: true
        #注册地址
        service-url:
          defaultZone: http://localhost:7000/eureka/
      instance:
        instance-id: customer-hystrix
        prefer-ip-address: true

## 4.3 主程序启动 ##

@SpringBootApplication
    @EnableFeignClients
    public class HystrixCustomerApplication { 
        public static void main(String[] args) { 
            SpringApplication.run(HystrixCustomerApplication.class);
                }
    }

## 4.4 FeginService ##

@Component
    @FeignClient("PROVIDER-HYSTRIX")
    @RequestMapping("/hystrix")
    public interface FeginService { 
     
        @GetMapping("/hello")
        String hello();
       
        @GetMapping("/threeseconds")
        String threeseconds();
    }

## 4.5 controller ##

@RequestMapping("/customer")
    @RestController
    public class HystrixController { 
    
        @Autowired
        private FeginService feginService;
    
    
        @GetMapping("/hello")
        public String hello(){ 
            return  feginService.hello();
        }
    
    
        @GetMapping("/threeseconds")
        public String threeseconds(){ 
            return feginService.threeseconds();
        }
    
    }

## 4.6 启动 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 5]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 6]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 7]

# 5 降级配置 #

## 5.1 provider-hystrix配置 ##

(1) 修改HystrixController

(2) 修改启动类

@SpringBootApplication
    @EnableEurekaClient
    @EnableCircuitBreaker
    public class HystrixApplication { 
        public static void main(String[] args) { 
            SpringApplication.run(HystrixApplication.class);
        }
    }

(3) 测试  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 8]

## 5.2 customer-hystrix降级配置 ##

(1) application.yml添加配置文件

#开启降级
    feign:
      hystrix:
        enabled: true

(2) 修改主启动类

@SpringBootApplication
    @EnableFeignClients
    @EnableHystrix
    public class HystrixCustomerApplication { 
        public static void main(String[] args) { 
            SpringApplication.run(HystrixCustomerApplication.class);
        }
    }

(3) 修改HystrixController

@RequestMapping("/customer")
    @RestController
    public class HystrixController { 
    
        @Autowired
        private FeginService feginService;
    
        @GetMapping("/hello")
        public String hello(){ 
            return  feginService.hello();
        }
    
        @HystrixCommand(fallbackMethod = "fallbackMethod",commandProperties = { 
                @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds",value = "2000")
        })
    
        @GetMapping("/threeseconds")
        public String threeseconds(){ 
            return feginService.threeseconds();
        }
    
        public String fallbackMethod(){ 
            return "服务器正忙,请稍后访问。Thread："+Thread.currentThread().getName();
        }
    }

(4) 重启运行  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 9]

# 6 hystrix降级解耦 #

## 6.1 @DefaultProperties ##

(1) DefaultProperties作用：全局降级方法。修改customer-hystrix的HystrixController

/** * @Description: @DefaultProperties全局降级方法 * @Auther: zhurongsheng * @Date: 2020/3/14 22:32 */
    @RequestMapping("/customer")
    @RestController
    @DefaultProperties(defaultFallback = "commonfallbackMethod")
    public class HystrixController { 
        @Autowired
        private FeginService feginService;
        @GetMapping("/hello")
        public String hello(){ 
            return  feginService.hello();
        }
        @HystrixCommand(commandProperties = { 
                @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds",value = "2000")
        })
        @GetMapping("/threeseconds")
        public String threeseconds(){ 
            return feginService.threeseconds();
        }
        public String commonfallbackMethod(){ 
            return "服务器正忙,请稍后访问。Thread："+Thread.currentThread().getName();
        }
    }

(2) 重启，测试  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 10]

## 6.2 customer-hystrix配置文件分离 ##

(1) FeginService

@FeignClient(value = "PROVIDER-HYSTRIX",fallback = FeginFallbackService.class,path = "/hystrix")
    @Service
    public interface FeginService { 
     
        @GetMapping("/hello")
        String hello();
    
        @GetMapping("/threeseconds")
        String threeseconds();
    }

(2) 接口实现类，FeginFallbackService

@Service
    public class FeginFallbackService implements FeginService{ 
        @Override
        public String hello() { 
            return "FeginFallbackService invoke hello";
        }
        @Override
        public String threeseconds() { 
            return "FeginFallbackService invoke threeseconds";
        }
    }

(3) 修改Controller，HystrixController

@RequestMapping("/customer")
    @RestController
    public class HystrixController { 
       
        @Autowired
        private FeginService feginService;
       
        @GetMapping("/hello")
        public String hello(){ 
            return  feginService.hello();
        }
      
        @HystrixCommand(commandProperties = { 
                @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds",value = "2000")
        })
        @GetMapping("/threeseconds")
        public String threeseconds(){ 
            return feginService.threeseconds();
        }
    }

(4) 重启测试  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 11]

# 7 熔断 #

服务熔断：在一定周期内，服务异常次数达到设定的阈值或百分比，则触发熔断，熔断后，后面的请求将都走默认处理方法defaultFallback

## 7.1 修改provider-hystrix的 HystrixController ##

@RestController
    @RequestMapping("/hystrix")
    public class HystrixController { 
        @GetMapping("/hello")
        public String hello() { 
            return "Hello Thread：" + Thread.currentThread().getName();
        }
        /** * fallbackMethod 降级方法 * timeoutInMilliseconds: 2秒为正常响应，超过两秒调用降级方法 */
        @HystrixCommand(fallbackMethod = "fallbackMethod", commandProperties = { 
                @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "2000")
        })
        @GetMapping("/threeseconds")
        public String threeseconds() { 
            try { 
                Thread.sleep(3000);
            } catch (InterruptedException e) { 
                e.printStackTrace();
            }
            return "hello threeseconds";
        }
        public String fallbackMethod() { 
            return "服务器正忙,请稍后访问。Thread：" + Thread.currentThread().getName();
        }
        /** * =====================熔断配置======================= * circuitBreaker.enabled: 启动熔断 * circuitBreaker.requestVolumeThreshold：请求次数 * circuitBreaker.errorThresholdPercentage： 失败率达到多少启动熔断 */
        @HystrixCommand(fallbackMethod = "circuitBreakerFallback",commandProperties = { 
                @HystrixProperty(name="circuitBreaker.enabled",value = "true"),
                @HystrixProperty(name="circuitBreaker.requestVolumeThreshold",value = "10"),
                @HystrixProperty(name="circuitBreaker.errorThresholdPercentage",value = "50")
        })
        @GetMapping("/{id}")
        public String circuitBreaker(@PathVariable("id") Integer id) { 
            if (id<0){ 
                throw  new RuntimeException("id ="+id+"，不能为负数");
            }
            return "调用成功，O(∩_∩)O~ id=" + id;
        }
        public String circuitBreakerFallback(@PathVariable("id") Integer id) { 
            return "id =" + id + ", 不能为负数，o(╥﹏╥)o~ ";
        }
    }

## 7.2 启动观察 ##

(1) 正常访问  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 12]  
(2) 错误访问  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 13]  
(3) 多次错误访问后  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 14]

# 8 服务监控hystrixDashboard #

除了隔离依赖服务的週用以外, Hystrix还提供了往实时的调用监控( Hystrix Dashboard), Hystrix:会持续地记录所有通过 Hystrix发起的请求的执行信息,并以统计报表和图形的形式展示给用户,包括每秒执行多少请求多少成功,多少失败等。 Netflix 通过hystrix- metrics- event- stream项目实现了对以上指标的监控。 Spring Cloud也提供了 Hystrix Dashboard的整合,对监控内容转化成可视化界面。

## 8.1 pom ##

<dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-hystrix-dashboard</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>runtime</scope>
            <optional>true</optional>
        </dependency>
    </dependencies>

## 8.2 application.yml ##

server:
      port: 9001

## 8.3 启动类，并启动 ##

@SpringBootApplication
    @EnableHystrixDashboard
    public class HystrixDashbordApplication { 
        public static void main(String[] args) { 
            SpringApplication.run(HystrixDashbordApplication.class);
        }
    }

## 8.4 访问 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 15]

## 8.5 在provider-hystrix里添加HystrixConfig配置文件，重启服务 ##

@Configuration
    public class HystrixConfig { 
        @Bean
        public ServletRegistrationBean myServlet(){ 
            HystrixMetricsStreamServlet streamServlet = new HystrixMetricsStreamServlet();
            ServletRegistrationBean registrationBean = new ServletRegistrationBean(streamServlet);
            registrationBean.setLoadOnStartup(1);
            registrationBean.addUrlMappings("/hystrix.stream");
            registrationBean.setName("HystrixMetricsStreamServlet");
            return registrationBean;
        }
    }

## 8.6 dashboard添加provider-hystrix ##

http://localhost:7004/hystrix.stream

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 16]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 17]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 18]

# GITHUB #

#分支 Hystrix-enviroment-release-v1.0
    https://github.com/zhurongsheng666/spring-cloud-hoxton

[01]: https://blog.csdn.net/qq_34125999/article/details/104732409
[02 _Eureka]: https://blog.csdn.net/qq_34125999/article/details/104744751
[03 Zookeeper]: https://blog.csdn.net/qq_34125999/article/details/104788216
[04 Consule]: https://blog.csdn.net/qq_34125999/article/details/104809738
[05 Ribbon]: https://blog.csdn.net/qq_34125999/article/details/104859515
[06 OpenFegin]: https://blog.csdn.net/qq_34125999/article/details/104865809
[07 Hystrix]: https://blog.csdn.net/qq_34125999/article/details/104872389
[08 Gateway]: https://blog.csdn.net/qq_34125999/article/details/104878977
[09 Config]: https://blog.csdn.net/qq_34125999/article/details/104887587
[10 Bus]: https://blog.csdn.net/qq_34125999/article/details/104890652
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70]: /images/20230529/630b58c8181645a9a09d053efad020a2.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 1]: /images/20230529/dd90121efda84072a79d07fcf3e501b3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 2]: /images/20230529/1d9930d86f2d49eb832913b6dcd2d513.png
[20200315014721530.png]: /images/20230529/24ad18c8495b4f3fa4f2ec6dfb0d4881.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 3]: /images/20230529/b369cd4f84df470591dee327be8c9057.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 4]: /images/20230529/f58310158d3d4c64b6921a1292bb7bd9.png
[20200315014835150.png]: /images/20230529/2182e3bb48d34f98b6ec7b0fc24f2402.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 5]: /images/20230529/4ab66ab426ff493e969feb11d8aa525b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 6]: /images/20230529/c83d58a8ff1f4f039c1e199b34359dd0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 7]: /images/20230529/10afb81ef6314858adc3d74082cc87a0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 8]: /images/20230529/8e35ffb0e56c48f29e0571a68efc570f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 9]: /images/20230529/1ba9734d5a4e4fc1992269e57bd643c0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 10]: /images/20230529/4bfbcb48c0724137b7b3e7a667e73f34.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 11]: /images/20230529/09e24307c72a491198198f413c4eee74.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 12]: /images/20230529/414ea50a3e9748249ae73263ae727b18.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 13]: /images/20230529/926ae4d83f754cbfa664ba4cf1ffad4a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 14]: /images/20230529/48a3e0cc1ad942b088ee712ece50e5fe.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 15]: /images/20230529/998daf9598ec49e9b9191ca9fb2228c4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 16]: /images/20230529/86516d3617b94dc689295a43f85466dd.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 17]: /images/20230529/fc8788d745f74b59bb1bcf20b3643f13.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTI1OTk5_size_16_color_FFFFFF_t_70 18]: /images/20230529/896a45cb679a404f9dde792c2d7a0a68.png