Spring Cloud Hystrix 源码系列：HystrixCommandAspect 入口解析

小灰灰 2022-03-01 04:35 291阅读 0赞

在[前面的文章][Link 1]已经基本上讲过Hystrix的由来，功能介绍及工作原理。本篇开始正式进入Hystrix 源码分析阶段，将带你深入简出的理解HystrixCommandAspect、HystrixCommand、HystrixObservableCommand的内部构建。本文基于[hystrix-core][] 1.5.18（近年来几乎很少更新，建议升级）。

**目录**

1. HystrixCommandAspect

2. HystrixInvokable

3. HystrixCommandKey和HystrixCommandGroupKey和HystrixThreadPoolKey

4. AbstractCommand.toObservable

5. Observable的defer/just/lift

--------------------

该如何下手呢？之前介绍过Hystrix的注解用法需要两步，重点关注[第一步][Link 2]，它是突破口。

引入 **spring-cloud-starter-hystrix** 依赖，**spring-cloud-netflix-core** 的 jar 包中包含 *META-INF/spring.factories* 文件（Spring 中也的 [Java SPI][] 的加载机制）。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2l0c29mdGNoZW5mZWk_size_16_color_FFFFFF_t_70][]

### 1. HystrixCommandAspect ###

HystrixCommandAspect就是Aop对标记了Hystrix的方法的入口类。

@Configuration
    public class HystrixCircuitBreakerConfiguration {
        public HystrixCircuitBreakerConfiguration() {
        }
        //重要的入口类
        @Bean
        public HystrixCommandAspect hystrixCommandAspect() {
            return new HystrixCommandAspect();
        }
    }
    @Aspect
    public class HystrixCommandAspect {
        @Pointcut("@annotation(com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand)")
        public void hystrixCommandAnnotationPointcut() {
        }
        // 请求的合并
        @Pointcut("@annotation(com.netflix.hystrix.contrib.javanica.annotation.HystrixCollapser)")
        public void hystrixCollapserAnnotationPointcut() {
        }
    
        @Around("hystrixCommandAnnotationPointcut() || hystrixCollapserAnnotationPointcut()")
        public Object methodsAnnotatedWithHystrixCommand(ProceedingJoinPoint joinPoint) throws Throwable {
            Method method = AopUtils.getMethodFromTarget(joinPoint);
            Validate.notNull(method, "failed to get method from joinPoint: %s", new Object[]{joinPoint});
            if (method.isAnnotationPresent(HystrixCommand.class) && method.isAnnotationPresent(HystrixCollapser.class)) {
                throw new IllegalStateException("method cannot be annotated with HystrixCommand and HystrixCollapser annotations at the same time");
            } else {
                HystrixCommandAspect.MetaHolderFactory metaHolderFactory = (HystrixCommandAspect.MetaHolderFactory)META_HOLDER_FACTORY_MAP.get(HystrixCommandAspect.HystrixPointcutType.of(method));
                //如果是@HystrixCommand，最终调用CommandMetaHolderFactory.create创建metaHolder
                MetaHolder metaHolder = metaHolderFactory.create(joinPoint);
                // 创建HystrixInvokable,只是一个空接口，没有任何方法，只是用来标记具备可执行的能力
                HystrixInvokable invokable = HystrixCommandFactory.getInstance().create(metaHolder);
                ExecutionType executionType = metaHolder.isCollapserAnnotationPresent() ? metaHolder.getCollapserExecutionType() : metaHolder.getExecutionType();
    
                try {
                    Object result;
                    if (!metaHolder.isObservable()) {
                        // 利用工具CommandExecutor来执行
                        result = CommandExecutor.execute(invokable, executionType, metaHolder);
                    } else {
                        result = this.executeObservable(invokable, executionType, metaHolder);
                    }
    
                    return result;
                } catch (HystrixBadRequestException var9) {
                    throw var9.getCause();
                } catch (HystrixRuntimeException var10) {
                    throw this.getCauseOrDefault(var10, var10);
                }
            }
        }
        
    }
    //创建Command
    public class HystrixCommandFactory {
        private static final HystrixCommandFactory INSTANCE = new HystrixCommandFactory();
        private HystrixCommandFactory() {
        }
        public static HystrixCommandFactory getInstance() {
            return INSTANCE;
        }
        public HystrixInvokable create(MetaHolder metaHolder) {
            Object executable;
            if (metaHolder.isCollapserAnnotationPresent()) {
                executable = new CommandCollapser(metaHolder);
            } else if (metaHolder.isObservable()) {//如果切入的方法是Observable类型
                executable = new GenericObservableCommand(HystrixCommandBuilderFactory.getInstance().create(metaHolder));
            } else {//比如：public String hello()方法
                executable = new GenericCommand(HystrixCommandBuilderFactory.getInstance().create(metaHolder));
            }
    
            return (HystrixInvokable)executable;
        }
    }
    //新建GenericCommand：有点熟悉了吧
    public class GenericCommand extends AbstractHystrixCommand<Object> {
        private static final Logger LOGGER = LoggerFactory.getLogger(GenericCommand.class);
    
        public GenericCommand(HystrixCommandBuilder builder) {
            super(builder);
        }
        // 执行具体的方法，如：serviceName的hello
        protected Object run() throws Exception {
            LOGGER.debug("execute command: {}", this.getCommandKey().name());
            return this.process(new AbstractHystrixCommand<Object>.Action() {
                Object execute() {
                    return GenericCommand.this.getCommandAction().execute(GenericCommand.this.getExecutionType());
                }
            });
        }
        // 执行fallback方法，如：serviceName的error()
        protected Object getFallback() {
            final CommandAction commandAction = this.getFallbackAction();
            if (commandAction != null) {
                try {
                    return this.process(new AbstractHystrixCommand<Object>.Action() {
                        Object execute() {
                            MetaHolder metaHolder = commandAction.getMetaHolder();
                            Object[] args = CommonUtils.createArgsForFallback(metaHolder, GenericCommand.this.getExecutionException());
                            return commandAction.executeWithArgs(metaHolder.getFallbackExecutionType(), args);
                        }
                    });
                } catch (Throwable var3) {
                    LOGGER.error(FallbackErrorMessageBuilder.create().append(commandAction, var3).build());
                    throw new FallbackInvocationException(var3.getCause());
                }
            } else {
                return super.getFallback();
            }
        }
        //实现了HystrixExecutable的方法
        public R execute() {
            try {
                return this.queue().get();
            } catch (Exception var2) {
                throw Exceptions.sneakyThrow(this.decomposeException(var2));
            }
        }
    }
    //HystrixInvokable（GenericCommand）委托CommandExecutor 来执行
    public class CommandExecutor {
        // 全文的关键方法
        public static Object execute(HystrixInvokable invokable, ExecutionType executionType, MetaHolder metaHolder) throws RuntimeException {
            Validate.notNull(invokable);
            Validate.notNull(metaHolder);
            switch(executionType) {
            case SYNCHRONOUS:
                // 首先将 invokable 转换为 HystrixExecutable，再执行 HystrixCommand的execute() 方法
                return castToExecutable(invokable, executionType).execute();
            case ASYNCHRONOUS://如果切入的目标方法是Future返回类型时
                HystrixExecutable executable = castToExecutable(invokable, executionType);
                if (metaHolder.hasFallbackMethodCommand() && ExecutionType.ASYNCHRONOUS == metaHolder.getFallbackExecutionType()) {
                    return new FutureDecorator(executable.queue());
                }
    
                return executable.queue();
            case OBSERVABLE:
                HystrixObservable observable = castToObservable(invokable);
                return ObservableExecutionMode.EAGER == metaHolder.getObservableExecutionMode() ? observable.observe() : observable.toObservable();
            default:
                throw new RuntimeException("unsupported execution type: " + executionType);
            }
        }
        // HystrixExecutable 的 execute() 方法由 HystrixCommand.execute() 实现
        private static HystrixExecutable castToExecutable(HystrixInvokable invokable, ExecutionType executionType) {
            if (invokable instanceof HystrixExecutable) {
                return (HystrixExecutable)invokable;
            } else {
                throw new RuntimeException("Command should implement " + HystrixExecutable.class.getCanonicalName() + " interface to execute in: " + executionType + " mode");
            }
        }
    }
    public interface HystrixExecutable<R> extends HystrixInvokable<R> {
        R execute();
        Future<R> queue();
        Observable<R> observe();
    }

### 2. HystrixInvokable ###

HystrixInvokable 很关键，它的实现类是 GenericCommand 。接下来看看类图，三个抽象父类 AbstractHystrixCommand**、**HystrixCommand**、**AbstractCommand 帮助 GenericCommand 做了不少公共的事情，而 GenericCommand 负责执行具体的方法和fallback时的方法。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2l0c29mdGNoZW5mZWk_size_16_color_FFFFFF_t_70 1][]

### 3. HystrixCommandKey和HystrixCommandGroupKey和HystrixThreadPoolKey ###

比如有：@HystrixCommand(fallbackMethod = "helloFallback", groupKey = "Group1", threadPoolKey = "HystrixFooServiceGaGa") 这几个是什么关系呢？

commandKey：每个command都有对应的commandKey可以认为是command的名字，默认情况下（注解未指定），命令名称来源于切入方法的方法名（`method.getName()）。`自定义使用HystrixCommandKey.Factory.asKey("HelloWorld")，说实话感觉没啥用，不需要自定义。

groupKey：Hystrix使用命令分组将一起的命令进行管理，比如报告、警报、仪表盘或组/库。默认情况下（注解未指定），名称来源于切入方法所属类的类名名（obj.getClass().getSimpleName()），Hystrix使用 [`HystrixCommandGroupKey`][HystrixCommandGroupKey] 来定义命令线程池，除非单独定义线程池。

验证：commandKey和groupKey可以看源码CommandMetaHolderFactory.create以及HystrixCommandBuilderFactory中的createGenericSetterBuilder方法。

threadPoolKey：线程池主要体现是用于监测、指标发布、缓存和其他此类用途的HystrixThreadPool。当隔离方式为线程时，有隔离的作用。AbstractCommand.threadPool通过HystrixThreadPool.getInstance获取。

> 具有相同的threadPoolKey的使用同一个线程池，如果注解未指定threadPoolKey，则使用groupKey作为HystrixThreadPoolKey

《Spring Cloud 微服务实战》推荐：通常情况下，尽量通过 HystrixThreadPoolKey 的方式来指定线程池的划分， **而不是通过组名的默认方式实现划分**， 因为多个不同的命令可能 从业务逻辑上来看属于同 一个组， 但是往往从实现本身上需要跟其他命令进行隔离。

//验证逻辑
    abstract class AbstractCommand<R> implements HystrixInvokableInfo<R>, HystrixObservable<R> {
        private static HystrixThreadPoolKey initThreadPoolKey(HystrixThreadPoolKey threadPoolKey, HystrixCommandGroupKey groupKey, String threadPoolKeyOverride) {
            if (threadPoolKeyOverride == null) {
                // we don't have a property overriding the value so use either HystrixThreadPoolKey or HystrixCommandGroup
                if (threadPoolKey == null) {
                    /* use HystrixCommandGroup if HystrixThreadPoolKey is null */
                    return HystrixThreadPoolKey.Factory.asKey(groupKey.name());
                } else {
                    return threadPoolKey;
                }
            } else {
                // we have a property defining the thread-pool so use it instead
                return HystrixThreadPoolKey.Factory.asKey(threadPoolKeyOverride);
            }
        }
        //线程池启作用
        private Observable<R> executeCommandWithSpecifiedIsolation(final AbstractCommand<R> _cmd) {
            ...
            if (threadState.compareAndSet(ThreadState.NOT_USING_THREAD, ThreadState.STARTED)) {
                HystrixCounters.incrementGlobalConcurrentThreads();
                threadPool.markThreadExecution();
            ...
        }
    }

### 4. AbstractCommand.**toObservable** ###

由于最终入口都是 toObservable()，就从 AbstractCommand的 Observable<R> toObservable() 方法开始。Hystrix 使用观察者模式，**Observable** 即被观察者，被观察者些状态变更时，观察者可以做出各项响应。因为 Hystrix 基于RxJava，RxJava 不熟悉的童鞋可以参考我[之前的分享][Link 3]。

源码实现比较复杂，可以用一张图解释，这种方式是“军师型”，排兵布阵，先创造了各个处理者，然后创造被观察者，再设置Observable发生各种情况时由谁来处理，完全掌控全局。

解释下Action0、Func1这种对象，他们与Action、Func和Runnable、Callable类似，是一个可以被执行的实体。Action没有返回值，Action0…ActionN表示有0..N个参数，Action0就表示没有参数；Func有返值，0..N一样表示参数。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2l0c29mdGNoZW5mZWk_size_16_color_FFFFFF_t_70 2][]

简单的的来看就是：创建一个Observable，然后绑定各种事件对应的处理者，各类doOnXXXX，表示发生XXX事件时做什么事情。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2l0c29mdGNoZW5mZWk_size_16_color_FFFFFF_t_70 3][]

abstract class AbstractCommand<R> implements HystrixInvokableInfo<R>, HystrixObservable<R> {
    	public Observable<R> toObservable() {
    		final AbstractCommand<R> _cmd = this;
    		// 命令执行结束后的清理者
    		final Action0 terminateCommandCleanup = new Action0() {...};
    		// 取消订阅时处理者
    		final Action0 unsubscribeCommandCleanup = new Action0() {...};
    		// 重点：Hystrix 核心逻辑: 断路器、隔离
    		final Func0<Observable<R>> applyHystrixSemantics = new Func0<Observable<R>>() {...};
    		// 发射数据(OnNext表示发射数据)时的Hook
    		final Func1<R, R> wrapWithAllOnNextHooks = new Func1<R, R>() {...};
    		// 命令执行完成的Hook
    		final Action0 fireOnCompletedHook = new Action0() {...};
    		// 通过Observable.defer()创建一个Observable
    		return Observable.defer(new Func0<Observable<R>>() {
    			@Override
    			public Observable<R> call() {
    				final boolean requestCacheEnabled = isRequestCachingEnabled();
    				final String cacheKey = getCacheKey();
    				// 首先尝试从请求缓存中获取结果
    				if (requestCacheEnabled) {
    					HystrixCommandResponseFromCache<R> fromCache = (HystrixCommandResponseFromCache<R>) requestCache.get(cacheKey);
    					if (fromCache != null) {
    						isResponseFromCache = true;
    						return handleRequestCacheHitAndEmitValues(fromCache, _cmd);
    					}
    				}
    				// 使用上面的Func0：applyHystrixSemantics 来创建Observable
    				Observable<R> hystrixObservable =
    						Observable.defer(applyHystrixSemantics)
    								.map(wrapWithAllOnNextHooks);
    				Observable<R> afterCache;
    				// 如果启用请求缓存，将Observable包装成HystrixCachedObservable并进行相关处理
    				if (requestCacheEnabled && cacheKey != null) {
    					HystrixCachedObservable<R> toCache = HystrixCachedObservable.from(hystrixObservable, _cmd);
    					...
    				} else {
    					afterCache = hystrixObservable;
    				}
    				// 返回Observable
    				return afterCache
    						.doOnTerminate(terminateCommandCleanup)   
    						.doOnUnsubscribe(unsubscribeCommandCleanup)
    						.doOnCompleted(fireOnCompletedHook);
    			}
    		});
    	}
    }

**applyHystrixSemantics变量：Hystrix 核心逻辑: 断路器、隔离**

// applyHystrixSemantics 是一个Func0(理解为执行实体或处理者),表示没有参数,返回值是Observable。
    final Func0<Observable<R>> applyHystrixSemantics = new Func0<Observable<R>>() {
        // Func0 做的事情如下
        @Override
        public Observable<R> call() {
            // 如果未订阅,返回一个"哑炮" Observable, 即一个不会发射任何数据的Observable
            if (commandState.get().equals(CommandState.UNSUBSCRIBED)) {
                return Observable.never();
            }
            // 调用applyHystrixSemantics()来创建Observable
            return applyHystrixSemantics(_cmd);
        }
    };

**applyHystrixSemantics方法**

// Semantics 译为语义, 应用Hystrix语义很拗口，其实就是应用Hystrix的断路器、隔离特性
    private Observable<R> applyHystrixSemantics(final AbstractCommand<R> _cmd) {
        // 源码中有很多executionHook、eventNotifier的操作，这是Hystrix拓展性的一种体现。这里面啥事也没做，留了个口子，开发人员可以拓展
        executionHook.onStart(_cmd);
        // 判断断路器是否开启
        if (circuitBreaker.attemptExecution()) {
            // 获取执行信号
            final TryableSemaphore executionSemaphore = getExecutionSemaphore();
            final AtomicBoolean semaphoreHasBeenReleased = new AtomicBoolean(false);
            final Action0 singleSemaphoreRelease = new Action0() {...};
            final Action1<Throwable> markExceptionThrown = new Action1<Throwable>() {...};
            // 判断是否信号量拒绝
            if (executionSemaphore.tryAcquire()) {
                try {
                    // 重点：处理隔离策略和Fallback策略
                    return executeCommandAndObserve(_cmd)
                            .doOnError(markExceptionThrown)
                            .doOnTerminate(singleSemaphoreRelease)
                            .doOnUnsubscribe(singleSemaphoreRelease);
                } catch (RuntimeException e) {
                    return Observable.error(e);
                }
            } else {
                return handleSemaphoreRejectionViaFallback();
            }
        } 
        // 开启了断路器,执行Fallback
        else {
            return handleShortCircuitViaFallback();
        }
    }

**executeCommandAndObserve方法：处理隔离策略和Fallback策略**

// 处理隔离策略和各种Fallback
    private Observable<R> executeCommandAndObserve(final AbstractCommand<R> _cmd) {
        final HystrixRequestContext currentRequestContext = HystrixRequestContext.getContextForCurrentThread();
        final Action1<R> markEmits = new Action1<R>() {...};
        final Action0 markOnCompleted = new Action0() {...};
        // 利用Func1获取处理Fallback的 Observable
        final Func1<Throwable, Observable<R>> handleFallback = new Func1<Throwable, Observable<R>>() {
            @Override
            public Observable<R> call(Throwable t) {
                circuitBreaker.markNonSuccess();
                Exception e = getExceptionFromThrowable(t);
                executionResult = executionResult.setExecutionException(e);
                // 拒绝处理
                if (e instanceof RejectedExecutionException) {
                    return handleThreadPoolRejectionViaFallback(e);
                // 超时处理    
                } else if (t instanceof HystrixTimeoutException) {
                    return handleTimeoutViaFallback();
                } else if (t instanceof HystrixBadRequestException) {
                    return handleBadRequestByEmittingError(e);
                } else {
                    ...
                    return handleFailureViaFallback(e);
                }
            }
        };
        final Action1<Notification<? super R>> setRequestContext ...
        Observable<R> execution;
        // 利用特定的隔离策略来处理
        if (properties.executionTimeoutEnabled().get()) {
            execution = executeCommandWithSpecifiedIsolation(_cmd)
                    .lift(new HystrixObservableTimeoutOperator<R>(_cmd));
        } else {
            execution = executeCommandWithSpecifiedIsolation(_cmd);
        }
        return execution.doOnNext(markEmits)
                .doOnCompleted(markOnCompleted)
                // 绑定Fallback的处理者
                .onErrorResumeNext(handleFallback)
                .doOnEach(setRequestContext);
    }

**executeCommandWithSpecifiedIsolation方法：隔离处理**

private Observable<R> executeCommandWithSpecifiedIsolation(final AbstractCommand<R> _cmd) {
        // 线程池隔离
        if (properties.executionIsolationStrategy().get() == ExecutionIsolationStrategy.THREAD) {
            // 再次使用 Observable.defer(), 通过执行Func0来得到Observable
            return Observable.defer(new Func0<Observable<R>>() {
                @Override
                public Observable<R> call() {
                    // 收集metric信息
                    metrics.markCommandStart(commandKey, threadPoolKey, ExecutionIsolationStrategy.THREAD);
                    ...
                    try {
                          ... // 获取真正的用户Task
                        return getUserExecutionObservable(_cmd);
                    } catch (Throwable ex) {
                        return Observable.error(ex);
                    }
                    ...
                }
                // 绑定各种处理者
            }).doOnTerminate(new Action0() {...})
                .doOnUnsubscribe(new Action0() {...})
                // 绑定超时处理者
                .subscribeOn(threadPool.getScheduler(new Func0<Boolean>() {
                @Override
                public Boolean call() {
                    return properties.executionIsolationThreadInterruptOnTimeout().get() && _cmd.isCommandTimedOut.get() == TimedOutStatus.TIMED_OUT;
                }
            }));
        } 
        // 信号量隔离，和线程池大同小异，全部省略了
        else {
            return Observable.defer(new Func0<Observable<R>>() {...}
        }
    }

**getUserExecutionObservable()方法：执行用户任务**

private Observable<R> getUserExecutionObservable(final AbstractCommand<R> _cmd) {
            Observable<R> userObservable;
            try {
                userObservable = getExecutionObservable();
            } catch (Throwable ex) {
                // the run() method is a user provided implementation so can throw instead of using Observable.onError
                // so we catch it here and turn it into Observable.error
                userObservable = Observable.error(ex);
            }
    
            return userObservable
                    .lift(new ExecutionHookApplication(_cmd))
                    .lift(new DeprecatedOnRunHookApplication(_cmd));
        }
        @Override
        final protected Observable<R> getExecutionObservable() {
            return Observable.defer(new Func0<Observable<R>>() {
                @Override
                public Observable<R> call() {
                    try {
                        //run()是GenericCommand.run，执行切入的目标方法，前面讲过的
                        return Observable.just(run());
                    } catch (Throwable ex) {
                        return Observable.error(ex);
                    }
                }
            }).doOnSubscribe(new Action0() {
                @Override
                public void call() {
                    // Save thread on which we get subscribed so that we can interrupt it later if needed
                    executionThread.set(Thread.currentThread());
                }
            });
        }

### 5. **Observable的defer/**just/lift ###

defer译为延迟，就是说：必须有观察者订阅时，Observable 才开始发射数据。而defer()的参数是个Func0，是一个会返回Observable的执行实体。

just是将一个对象映射为Observable类。

lift(Operator)是返回一个新的Subscriber ，对传递进来的Subscriber形成代理 ，对onNext()/onCompleted()/onError()方法进行拦截。

return Observable.defer(new Func0<Observable<R>>() {
        @Override
        public Observable<R> call() {
            // 再一次使用Observable.defer()技能，这次用的是applyHystrixSemantics这个Func0
            Observable<R> hystrixObservable =
                    Observable.defer(applyHystrixSemantics)
                            .map(wrapWithAllOnNextHooks);
            ... // 此处忽略了请求缓存处理,上面已有提及
            Observable<R> afterCache;
            ...
            // 为Observable绑定几个特定事件的处理者,这都是上门创建的Action0
            return afterCache
                    .doOnTerminate(terminateCommandCleanup) 
                    .doOnUnsubscribe(unsubscribeCommandCleanup) 
                    .doOnCompleted(fireOnCompletedHook);
        }
    });

如果你熟悉rxjava的话，Hystrix 的源码其实基本上都可以读懂的。

[Link 1]: https://blog.csdn.net/alex_xfboy/article/details/85069742
[hystrix-core]: https://mvnrepository.com/artifact/com.netflix.hystrix/hystrix-core
[Link 2]: https://blog.csdn.net/alex_xfboy/article/details/85069742#2.2%C2%A0%E5%9F%BA%E6%9C%AC%E7%94%A8%E6%B3%95
[Java SPI]: https://blog.csdn.net/alex_xfboy/article/details/77581605
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2l0c29mdGNoZW5mZWk_size_16_color_FFFFFF_t_70]: /images/20220301/f7de58e1861e4b789d7cddac7b39311d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2l0c29mdGNoZW5mZWk_size_16_color_FFFFFF_t_70 1]: /images/20220301/c7e9c39d69b54181b99c01f0cc0c023d.png
[HystrixCommandGroupKey]: http://netflix.github.io/Hystrix/javadoc/index.html?com/netflix/hystrix/HystrixCommandGroupKey.html
[Link 3]: https://blog.csdn.net/alex_xfboy/article/details/83089734
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2l0c29mdGNoZW5mZWk_size_16_color_FFFFFF_t_70 2]: /images/20220301/fac1751688e841fda847bae0759fb471.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2l0c29mdGNoZW5mZWk_size_16_color_FFFFFF_t_70 3]: /images/20220301/5c80cea2a0894afabd200359fb35701f.png