分布式RPC框架xxl-rpc的源码分析

￡神魔★判官ぃ 2024-04-18 10:06 98阅读 0赞

目前版本：1.4.2-SNAPSHOT

官网概述：XXL-RPC 是一个分布式服务框架，提供稳定高性能的RPC远程服务调用功能。拥有"高性能、分布式、注册中心、负载均衡、服务治理"等特性。现已开放源代码，开箱即用。

一、rpc框架一般会提供的功能

1、服务提供端(Provider)

2、服务消费端(Invoker)：通过生成代理对象，封装底层通信细节 完成类似“本地方法”的调用。

3、注册中心：xxl-rpc提供 local本地注册、xxl-registry注册、zk注册等方式

4、负载均衡策略：xxl-rpc提供 随机、轮询、一致性hash、最不经常使用(LFU)、最近最久未使用(LRU)供invoker选择

5、序列化方式：xxl-rpc提供 Hessian、Jackson、Kryo、Protostuff等序列化方式

6、通信方式：xxl-rpc提供 NETTY、NETTY\_HTTP、MINA、JETTY等几种通信方式

二、xxl-rpc实现思路：

Provider向注册中心注册服务信息(提供了三种注册方式)，Invoker的代理对象会从注册中心发现可用的服务信息、根据选择的负载均衡策略调用服务 并根据选择的底层通信方式和序列化方式进行网络间通信。当然这些由xxl-rpc封装实现，对用户透明，使用时只需要通过配置+注解即可工作。

三、系统架构图(虽然v1.3版本, 但和1.4版本思想不变)：

![输入图片说明][format_png]

四、实现

1、系统核心结构图：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xkdzIwMTUxMDgwMzAwNg_size_16_color_FFFFFF_t_70][]

2、服务提供者(Provider):

a、核心类：XxlRpcProviderFactory

部分参数结构：

public class XxlRpcProviderFactory {
    	private static final Logger logger = LoggerFactory.getLogger(XxlRpcProviderFactory.class);
    
    	// ---------------------- config ----------------------
    
    	private NetEnum netType; // Netty/Jetty等网络通信类型
    	private Serializer serializer; // Protostuff/Jackson/Hessian等序列化方式
    
    	private String ip;					// for registry
    	private int port;					// default port
    	private String accessToken;
    
    	private Class<? extends ServiceRegistry> serviceRegistryClass; // local/xxl-rpc/zk等注册中心
    	private Map<String, String> serviceRegistryParam; // 服务注册参数

start方法，注册实例的初始化，服务注册/下线后的一系列抽象方法进行善后工作：

public void start() throws Exception {
    		// start server
    		serviceAddress = IpUtil.getIpPort(this.ip, port);
            // 实例化net通信服务类
    		server = netType.serverClass.newInstance();
            // 通过钩子方法在通信服务启动关闭时，对服务注册中心进行一系列操作
    		server.setStartedCallback(new BaseCallback() {		// serviceRegistry started
    			@Override
    			public void run() throws Exception {
    				// start registry
    				if (serviceRegistryClass != null) {
                        // 服务启动时注册中心实例化
    					serviceRegistry = serviceRegistryClass.newInstance();
    					serviceRegistry.start(serviceRegistryParam);
    					if (serviceData.size() > 0) {
    						serviceRegistry.registry(serviceData.keySet(), serviceAddress);
    					}
    				}
    			}
    		});
    		server.setStopedCallback(new BaseCallback() {		// serviceRegistry stoped
    			@Override
    			public void run() {
    				// stop registry
    				if (serviceRegistry != null) {
    					if (serviceData.size() > 0) {
                            // 服务下线时, 服务从注册中心摘除
    						serviceRegistry.remove(serviceData.keySet(), serviceAddress);
    					}
    					serviceRegistry.stop();
    					serviceRegistry = null;
    				}
    			}
    		});
    		server.start(this);
    	}

真正的服务提供实现方法invokeService：

/**
         * 由于与服务消费者的交互都是通过net封装的通信层实现，因此该方法在net通信实现中调用
    	 * invoke service
    	 * 
    	 * @param xxlRpcRequest
    	 * @return
    	 */
    	public XxlRpcResponse invokeService(XxlRpcRequest xxlRpcRequest) {
    
    		//  make response
    		XxlRpcResponse xxlRpcResponse = new XxlRpcResponse();
    		xxlRpcResponse.setRequestId(xxlRpcRequest.getRequestId());
    
    		// match service bean
    		String serviceKey = makeServiceKey(xxlRpcRequest.getClassName(), xxlRpcRequest.getVersion());
    		Object serviceBean = serviceData.get(serviceKey);
    
    		// valid
    		if (serviceBean == null) {
    			xxlRpcResponse.setErrorMsg("The serviceKey["+ serviceKey +"] not found.");
    			return xxlRpcResponse;
    		}
    
    		if (System.currentTimeMillis() - xxlRpcRequest.getCreateMillisTime() > 3*60*1000) {
    			xxlRpcResponse.setErrorMsg("The timestamp difference between admin and executor exceeds the limit.");
    			return xxlRpcResponse;
    		}
    		if (accessToken!=null && accessToken.trim().length()>0 && !accessToken.trim().equals(xxlRpcRequest.getAccessToken())) {
    			xxlRpcResponse.setErrorMsg("The access token[" + xxlRpcRequest.getAccessToken() + "] is wrong.");
    			return xxlRpcResponse;
    		}
    
            // 通过反射拿到对应的实现类，并调用，将处理结果返回给服务消费者
    		try {
    			// invoke
    			Class<?> serviceClass = serviceBean.getClass();
    			String methodName = xxlRpcRequest.getMethodName();
    			Class<?>[] parameterTypes = xxlRpcRequest.getParameterTypes();
    			Object[] parameters = xxlRpcRequest.getParameters();
    
                Method method = serviceClass.getMethod(methodName, parameterTypes);
                method.setAccessible(true);
    			Object result = method.invoke(serviceBean, parameters);
    
    			/*FastClass serviceFastClass = FastClass.create(serviceClass);
    			FastMethod serviceFastMethod = serviceFastClass.getMethod(methodName, parameterTypes);
    			Object result = serviceFastMethod.invoke(serviceBean, parameters);*/
    
    			xxlRpcResponse.setResult(result);
    		} catch (Throwable t) {
    			// catch error
    			logger.error("xxl-rpc provider invokeService error.", t);
    			xxlRpcResponse.setErrorMsg(ThrowableUtil.toString(t));
    		}
    
    		return xxlRpcResponse;
    	}

以上实现已可以正常提供服务，实现服务注册、摘除并可被消费端调用；但强大的Spring提供了一系列扩张工具类，下面看看：

b、XxlRpcProviderFactory的实现类XxlRpcSpringProviderFactory实现Spring的一些拓展类，方便处理初始化和“善后”工作：

/**
     * xxl-rpc provider (for spring)
     *
     * @author xuxueli 2018-10-18 18:09:20
     */
    public class XxlRpcSpringProviderFactory extends XxlRpcProviderFactory
            implements ApplicationContextAware, InitializingBean,DisposableBean {
     
        ………
        ………
    
    
        // ---------------------- util ----------------------
    
        // 从应用上下文中拿到XxlRpcService注解的bean，实例化后并向注册中心添加服务
        @Override
        public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
    
            Map<String, Object> serviceBeanMap = applicationContext.getBeansWithAnnotation(XxlRpcService.class);
            if (serviceBeanMap!=null && serviceBeanMap.size()>0) {
                for (Object serviceBean : serviceBeanMap.values()) {
                    // valid
                    if (serviceBean.getClass().getInterfaces().length ==0) {
                        throw new XxlRpcException("xxl-rpc, service(XxlRpcService) must inherit interface.");
                    }
                    // add service
                    XxlRpcService xxlRpcService = serviceBean.getClass().getAnnotation(XxlRpcService.class);
    
                    String iface = serviceBean.getClass().getInterfaces()[0].getName();
                    String version = xxlRpcService.version();
    
                    super.addService(iface, version, serviceBean);
                }
            }
    
            // TODO，addServices by api + prop
    
        }
    
        // 初始化和“善后”操作
        @Override
        public void afterPropertiesSet() throws Exception {
            this.prepareConfig();
            super.start();
        }
    
        @Override
        public void destroy() throws Exception {
            super.stop();
        }

这样结合Spring的强大注解，即可轻松实现配置(这里是针对使用spring的服务可以轻松简洁的接入)-相比之下就不用在手动的注册服务了，如下：

@Configuration
    public class XxlRpcProviderConfig {
        private Logger logger = LoggerFactory.getLogger(XxlRpcProviderConfig.class);
    
        @Value("${xxl-rpc.remoting.port}")
        private int port;
    
        @Value("${xxl-rpc.registry.xxlregistry.address}")
        private String address;
    
        @Value("${xxl-rpc.registry.xxlregistry.env}")
        private String env;
    
        @Bean
        public XxlRpcSpringProviderFactory xxlRpcSpringProviderFactory() {
    
            XxlRpcSpringProviderFactory providerFactory = new XxlRpcSpringProviderFactory();
            providerFactory.setPort(port);
            providerFactory.setServiceRegistryClass(XxlRegistryServiceRegistry.class);
            providerFactory.setServiceRegistryParam(new HashMap<String, String>(){
       {
                put(XxlRegistryServiceRegistry.XXL_REGISTRY_ADDRESS, address);
                put(XxlRegistryServiceRegistry.ENV, env);
            }});
            
            logger.info(">>>>>>>>>>> xxl-rpc provider config init finish.");
            return providerFactory;
        }
    
    }

\------------------------OK以上是Provider基本思路----------------------

2、服务消费者(Invoker):

a、核心实现类XxlRpcReferenceBean，该类会提供代理对象，封装底层通信细节等

public class XxlRpcReferenceBean {
    	private static final Logger logger = LoggerFactory.getLogger(XxlRpcReferenceBean.class);
    
    	// ---------------------- config ----------------------
    
    	private NetEnum netType;  // net 通信方式
    	private Serializer serializer; // 序列化方式
    	private CallType callType;  // 回调方式(sync/future等)
    	private LoadBalance loadBalance;  // 负载均衡方式
    
    	private Class<?> iface;
    	private String version;
    
    	private long timeout = 1000;
    
    	private String address;
    	private String accessToken;
    
    	private XxlRpcInvokeCallback invokeCallback; // 回调方式
    
    	private XxlRpcInvokerFactory invokerFactory; // 调用工厂，注册中心及服务选择，下一步详细介绍

主要实现方法->创建代理对象，如下

// ---------------------- util ----------------------
    
    	public Object getObject() {
    		return Proxy.newProxyInstance(Thread.currentThread()
    				.getContextClassLoader(), new Class[] { iface },
    				new InvocationHandler() {
    					@Override
    					public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
    
                           // 1、target方法名以及传递参数等获取
    						// method param
    						String className = method.getDeclaringClass().getName();	// iface.getName()
    						String varsion_ = version;
    						String methodName = method.getName();
    						Class<?>[] parameterTypes = method.getParameterTypes();
    						Object[] parameters = args;
    
    						// filter for generic
    						if (className.equals(XxlRpcGenericService.class.getName()) && methodName.equals("invoke")) {
    
    							Class<?>[] paramTypes = null;
    							if (args[3]!=null) {
    								String[] paramTypes_str = (String[]) args[3];
    								if (paramTypes_str.length > 0) {
    									paramTypes = new Class[paramTypes_str.length];
    									for (int i = 0; i < paramTypes_str.length; i++) {
    										paramTypes[i] = ClassUtil.resolveClass(paramTypes_str[i]);
    									}
    								}
    							}
    
    							className = (String) args[0];
    							varsion_ = (String) args[1];
    							methodName = (String) args[2];
    							parameterTypes = paramTypes;
    							parameters = (Object[]) args[4];
    						}
    
    						// filter method like "Object.toString()"
    						if (className.equals(Object.class.getName())) {
    							logger.info(">>>>>>>>>>> xxl-rpc proxy class-method not support [{}#{}]", className, methodName);
    							throw new XxlRpcException("xxl-rpc proxy class-method not support");
    						}
    
                            // 2、注册中心地址集选择，并选择合适的负载均衡策略 得到服务的地址
    						// address
    						String finalAddress = address;
    						if (finalAddress==null || finalAddress.trim().length()==0) {
    							if (invokerFactory!=null && invokerFactory.getServiceRegistry()!=null) {
    								// discovery
    								String serviceKey = XxlRpcProviderFactory.makeServiceKey(className, varsion_);
    								TreeSet<String> addressSet = invokerFactory.getServiceRegistry().discovery(serviceKey);
    								// load balance
    								if (addressSet==null || addressSet.size()==0) {
    									// pass
    								} else if (addressSet.size()==1) {
    									finalAddress = addressSet.first();
    								} else {
    									finalAddress = loadBalance.xxlRpcInvokerRouter.route(serviceKey, addressSet);
    								}
    
    							}
    						}
    						if (finalAddress==null || finalAddress.trim().length()==0) {
    							throw new XxlRpcException("xxl-rpc reference bean["+ className +"] address empty");
    						}
                            
                            // 3、以下是请求方式的选择sync/future/callback等，选择之后调用服务即可
    						// request
    						XxlRpcRequest xxlRpcRequest = new XxlRpcRequest();
    	                    xxlRpcRequest.setRequestId(UUID.randomUUID().toString());
    	                    xxlRpcRequest.setCreateMillisTime(System.currentTimeMillis());
    	                    xxlRpcRequest.setAccessToken(accessToken);
    	                    xxlRpcRequest.setClassName(className);
    	                    xxlRpcRequest.setMethodName(methodName);
    	                    xxlRpcRequest.setParameterTypes(parameterTypes);
    	                    xxlRpcRequest.setParameters(parameters);
    	                    
    	                    // send
    						if (CallType.SYNC == callType) {
    							// future-response set
    							XxlRpcFutureResponse futureResponse = new XxlRpcFutureResponse(invokerFactory, xxlRpcRequest, null);
    							try {
    								// do invoke
    								client.asyncSend(finalAddress, xxlRpcRequest);
    
    								// future get
    								XxlRpcResponse xxlRpcResponse = futureResponse.get(timeout, TimeUnit.MILLISECONDS);
    								if (xxlRpcResponse.getErrorMsg() != null) {
    									throw new XxlRpcException(xxlRpcResponse.getErrorMsg());
    								}
    								return xxlRpcResponse.getResult();
    							} catch (Exception e) {
    								logger.info(">>>>>>>>>>> xxl-rpc, invoke error, address:{}, XxlRpcRequest{}", finalAddress, xxlRpcRequest);
    
    								throw (e instanceof XxlRpcException)?e:new XxlRpcException(e);
    							} finally{
    								// future-response remove
    								futureResponse.removeInvokerFuture();
    							}
    						} else if (CallType.FUTURE == callType) {
    							// future-response set
    							XxlRpcFutureResponse futureResponse = new XxlRpcFutureResponse(invokerFactory, xxlRpcRequest, null);
                                try {
    								// invoke future set
    								XxlRpcInvokeFuture invokeFuture = new XxlRpcInvokeFuture(futureResponse);
    								XxlRpcInvokeFuture.setFuture(invokeFuture);
    
                                    // do invoke
                                    client.asyncSend(finalAddress, xxlRpcRequest);
    
                                    return null;
                                } catch (Exception e) {
    								logger.info(">>>>>>>>>>> xxl-rpc, invoke error, address:{}, XxlRpcRequest{}", finalAddress, xxlRpcRequest);
    
    								// future-response remove
    								futureResponse.removeInvokerFuture();
    
    								throw (e instanceof XxlRpcException)?e:new XxlRpcException(e);
                                }
    
    						} else if (CallType.CALLBACK == callType) {
    
    							// get callback
    							XxlRpcInvokeCallback finalInvokeCallback = invokeCallback;
    							XxlRpcInvokeCallback threadInvokeCallback = XxlRpcInvokeCallback.getCallback();
    							if (threadInvokeCallback != null) {
    								finalInvokeCallback = threadInvokeCallback;
    							}
    							if (finalInvokeCallback == null) {
    								throw new XxlRpcException("xxl-rpc XxlRpcInvokeCallback（CallType="+ CallType.CALLBACK.name() +"） cannot be null.");
    							}
    
    							// future-response set
    							XxlRpcFutureResponse futureResponse = new XxlRpcFutureResponse(invokerFactory, xxlRpcRequest, finalInvokeCallback);
    							try {
    								client.asyncSend(finalAddress, xxlRpcRequest);
    							} catch (Exception e) {
    								logger.info(">>>>>>>>>>> xxl-rpc, invoke error, address:{}, XxlRpcRequest{}", finalAddress, xxlRpcRequest);
    
    								// future-response remove
    								futureResponse.removeInvokerFuture();
    
    								throw (e instanceof XxlRpcException)?e:new XxlRpcException(e);
    							}
    
    							return null;
    						} else if (CallType.ONEWAY == callType) {
                                client.asyncSend(finalAddress, xxlRpcRequest);
                                return null;
                            } else {
    							throw new XxlRpcException("xxl-rpc callType["+ callType +"] invalid");
    						}
    
    					}
    				});
    	}

2、Invoker调用工厂XxlRpcInvokerFactory：

/**
     * xxl-rpc invoker factory, init service-registry
     *
     * @author xuxueli 2018-10-19
     */
    public class XxlRpcInvokerFactory {
        private static Logger logger = LoggerFactory.getLogger(XxlRpcInvokerFactory.class);
    
        // ---------------------- default instance ----------------------
    
        private static volatile XxlRpcInvokerFactory instance = new XxlRpcInvokerFactory(LocalServiceRegistry.class, null);
        public static XxlRpcInvokerFactory getInstance() {
            return instance;
        }
    
    
        // ---------------------- config ----------------------
    
        private Class<? extends ServiceRegistry> serviceRegistryClass;          // class.forname
        private Map<String, String> serviceRegistryParam;

和服务端类似，负载与注册中心交互的核心类，不在赘述

\--------- 以上服务消费端已经基本可以完成通信，但强大的Spring仍提供的一些很好的扩张，如下 --------

3、基于Spring的实现类XxlRpcSpringInvokerFactory：

/**
     * xxl-rpc invoker factory, init service-registry and spring-bean by annotation (for spring)
     *
     * @author xuxueli 2018-10-19
     */
    public class XxlRpcSpringInvokerFactory extends InstantiationAwareBeanPostProcessorAdapter 
            implements InitializingBean,DisposableBean, BeanFactoryAware {
        private Logger logger = LoggerFactory.getLogger(XxlRpcSpringInvokerFactory.class);
    
        // ---------------------- config ----------------------
    
        private Class<? extends ServiceRegistry> serviceRegistryClass;          // class.forname
        private Map<String, String> serviceRegistryParam;
    
    
        public void setServiceRegistryClass(Class<? extends ServiceRegistry> serviceRegistryClass) {
            this.serviceRegistryClass = serviceRegistryClass;
        }
    
        public void setServiceRegistryParam(Map<String, String> serviceRegistryParam) {
            this.serviceRegistryParam = serviceRegistryParam;
        }
    
    
        // ---------------------- util ----------------------
    
        private XxlRpcInvokerFactory xxlRpcInvokerFactory;

实现Spring的一些拓展类 以方便更好的支持注解、服务init、destory等，如下：

// 1、该类参数初始化完成后进行处理
        @Override
        public void afterPropertiesSet() throws Exception {
            // start invoker factory
            xxlRpcInvokerFactory = new XxlRpcInvokerFactory(serviceRegistryClass, serviceRegistryParam);
            xxlRpcInvokerFactory.start();
        }
    
        // 2、实例化之后进行处理
        @Override
        public boolean postProcessAfterInstantiation(final Object bean, final String beanName) throws BeansException {
    
            // 3、这里其实通过遍历所有的bean找到通过有XxlRpcReference注解过的bean的字段，并将该字段实例化，通过XxlRpcReferenceBean(开始介绍的核心类)拿到代理对象，然后就和之前一样了
            // collection
            final Set<String> serviceKeyList = new HashSet<>();
    
            // parse XxlRpcReferenceBean
            ReflectionUtils.doWithFields(bean.getClass(), new ReflectionUtils.FieldCallback() {
                @Override
                public void doWith(Field field) throws IllegalArgumentException, IllegalAccessException {
                    if (field.isAnnotationPresent(XxlRpcReference.class)) {
                        // valid
                        Class iface = field.getType();
                        if (!iface.isInterface()) {
                            throw new XxlRpcException("xxl-rpc, reference(XxlRpcReference) must be interface.");
                        }
    
                        XxlRpcReference rpcReference = field.getAnnotation(XxlRpcReference.class);
    
                        // init reference bean
                        XxlRpcReferenceBean referenceBean = new XxlRpcReferenceBean(
                                rpcReference.netType(),
                                rpcReference.serializer().getSerializer(),
                                rpcReference.callType(),
                                rpcReference.loadBalance(),
                                iface,
                                rpcReference.version(),
                                rpcReference.timeout(),
                                rpcReference.address(),
                                rpcReference.accessToken(),
                                null,
                                xxlRpcInvokerFactory
                        );
    
                        Object serviceProxy = referenceBean.getObject();
    
                        // set bean
                        field.setAccessible(true);
                        field.set(bean, serviceProxy);
    
                        logger.info(">>>>>>>>>>> xxl-rpc, invoker factory init reference bean success. serviceKey = {}, bean.field = {}.{}",
                                XxlRpcProviderFactory.makeServiceKey(iface.getName(), rpcReference.version()), beanName, field.getName());
    
                        // collection
                        String serviceKey = XxlRpcProviderFactory.makeServiceKey(iface.getName(), rpcReference.version());
                        serviceKeyList.add(serviceKey);
    
                    }
                }
            });
    
            // mult discovery
            if (xxlRpcInvokerFactory.getServiceRegistry() != null) {
                try {
                    xxlRpcInvokerFactory.getServiceRegistry().discovery(serviceKeyList);
                } catch (Exception e) {
                    logger.error(e.getMessage(), e);
                }
            }
    
            return super.postProcessAfterInstantiation(bean, beanName);
        }
    
    
        // 4、服务销毁, “善后”与注册中心的联系
        @Override
        public void destroy() throws Exception {
    
            // stop invoker factory
            xxlRpcInvokerFactory.stop();
        }
    
        private BeanFactory beanFactory;
    
        @Override
        public void setBeanFactory(BeanFactory beanFactory) throws BeansException {
            this.beanFactory = beanFactory;
        }

这样之后Spring接入方可以通过简洁的配置即可 如下：

@Configuration
    public class XxlRpcInvokerConfig {
        private Logger logger = LoggerFactory.getLogger(XxlRpcInvokerConfig.class);
    
    
        @Value("${xxl-rpc.registry.xxlregistry.address}")
        private String address;
    
        @Value("${xxl-rpc.registry.xxlregistry.env}")
        private String env;
    
    
        @Bean
        public XxlRpcSpringInvokerFactory xxlJobExecutor() {
    
            XxlRpcSpringInvokerFactory invokerFactory = new XxlRpcSpringInvokerFactory();
            invokerFactory.setServiceRegistryClass(XxlRegistryServiceRegistry.class);
            invokerFactory.setServiceRegistryParam(new HashMap<String, String>(){
       {
                put(XxlRegistryServiceRegistry.XXL_REGISTRY_ADDRESS, address);
                put(XxlRegistryServiceRegistry.ENV, env);
            }});
    
            logger.info(">>>>>>>>>>> xxl-rpc invoker config init finish.");
            return invokerFactory;
        }
    
    }

\--------------------以上基本的服务provider/invoker介绍完毕-----------------------

还有一些通信细节没有介绍，如序列化方式、负载均衡策略、Net通信方式、注册方式以及回调方式，这些不是本文重点，可以看作工具 以供xxl-rpc的用户选择使用, 接入方可以通过配置即可轻松选择；其中：

1、Net通信方式如Netty、Mina等都是不小的话题，可以拓展学习

2、注册方式：这里通过local、xxl-registry、zk等方式，在生产环境也很常用，可以拓展学习

以上是个人理解，如有问题请指出，谢谢！

[format_png]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL3h1eHVlbGkveHhsLXJwYy9tYXN0ZXIvZG9jL2ltYWdlcy9pbWdfRE5xNi5wbmc?x-oss-process=image/format,png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xkdzIwMTUxMDgwMzAwNg_size_16_color_FFFFFF_t_70]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/d0d344b9a9424c53963e2955808e1e68.png