Redis之反序列化失败问题追踪解决

我就是我 2022-12-26 10:25 432阅读 0赞

### 事故起因 ###

由于业务需要数据库需要多加两个字段，然后就一顿操作猛如虎。重启项目当get redis存放的的对象时报错了。报错示例如下：

com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 1388917379
    Serialization trace:
    	at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)
    	at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
    	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99)
    	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
    	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:764)
    	at org.redisson.codec.KryoCodec$1.decode(KryoCodec.java:120)
    	at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:316)
    	at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:110)
    	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
    	at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
    	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
    	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
    	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
    	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
    	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
    	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
    	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    	at java.lang.Thread.run(Thread.java:775)

com.esotericsoftware.kryo.KryoException: Unable to find class: 人名
    Serialization trace:
    	at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
    	at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
    	at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
    	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99)
    	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
    	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:764)
    	at org.redisson.codec.KryoCodec$1.decode(KryoCodec.java:120)
    	at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:316)
    	at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:110)
    	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
    	at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
    	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
    	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
    	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
    	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
    	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
    	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
    	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    	at java.lang.Thread.run(Thread.java:775)
    Caused by: java.lang.ClassNotFoundException: 人名
    	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
    	at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:93)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    	at java.lang.Class.forName0(Native Method)
    	at java.lang.Class.forName(Class.java:348)
    	at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136)
    	... 38 common frames omitted

原因描述大概原因是redis的反序列化异常，通过报错信息定位源码报错地方，在做源码分析之前我们需要首先了解下kryo这个工具：kryo是一个高性能的序列化/反序列化工具，由于其变长存储特性并使用了字节码生成机制，拥有较高的运行速度和较小的体积。而redis可显示的配置序列化工具我们这里使用就是Kryo速度比较快。但是默认的反序列化方式如果序列化时的对象需要和反序列化对象属性保持一致。具体问题就是如下：

// FieldSerializer
    public T read (Kryo kryo, Input input, Class<T> type) {
    	......	
    	T object = create(kryo, input, type);
    	kryo.reference(object);
    
    	CachedField[] fields = this.fields;
    	for (int i = 0, n = fields.length; i < n; i++)
    		fields[i].read(input, object);
    
    	}
        .......	
    }
    
    
    // ObjectField
    public void read (Input input, Object object) {
        .......
    	if (TRACE) trace("kryo", "Read field: " + this + " (" + type.getName() + ")" + " pos=" + input.position());
    	    Object value;
    
    		Class concreteType = valueClass;
    		Serializer serializer = this.serializer;
    		if (concreteType == null) {
    			Registration registration = kryo.readClass(input);
    			if (registration == null)
    				value = null;
    			else {
    				if (serializer == null) serializer = registration.getSerializer();
    				// if (generics != null)
    					serializer.setGenerics(kryo, generics);
    					value = kryo.readObject(input, registration.getType(), serializer);
    				}
    			}
          ........
        }
    }
    
    
    // DefaultClassResolver （ readClass (Input input) -> readName (Input input) ）
    protected Registration readName (Input input) {
    		int nameId = input.readVarInt(true);
    		if (nameIdToClass == null) nameIdToClass = new IntMap();
    		Class type = nameIdToClass.get(nameId);
    		if (type == null) {
    			// Only read the class name the first time encountered in object graph.
    			String className = input.readString();
    			type = getTypeByName(className);
    			if (type == null) {
    				try {
    					type = Class.forName(className, false, kryo.getClassLoader());
    				} catch (ClassNotFoundException ex) {
    					throw new KryoException("Unable to find class: " + className, ex);
    				}
    				if (nameToClass == null) nameToClass = new ObjectMap();
    				nameToClass.put(className, type);
    			}
    			nameIdToClass.put(nameId, type);
    			if (TRACE) trace("kryo", "Read class name: " + className);
    		} else {
    			if (TRACE) trace("kryo", "Read class name reference " + nameId + ": " + className(type));
    		}
    		return kryo.getRegistration(type);
    }

### 事故分析及解决方案 ###

这个过程就是反序列化对象给对应的属性赋值的过程而这个过程就是在遍历属性的过程中，如果新的对象新增字段时，对应的序列化的二进制码是按照属性对应位置进行反序列化赋值的，当两个属性的属性不匹配时就会报错 或者说反序列化出来的对象属性值不对应，这样就给我们造成了困扰。当然如果你的缓存可以清除的话可以选择暂停服务清除缓存既可以避免这类事故发生，当时我们时不能停服的只能一台一台服务启动线上无感发布的，而且缓存的数据量大约在4千万左右，所以清除缓存不太现实。通过源码分析及GitHub上redis的问题反馈如果给存放redis的实体添加@DefaultSerializer(CompatibleFieldSerializer.class)注解而这个CompatibleFieldSerializer 的反序列化方式是通过对比属性才进行赋值的，这样就避免了对实体新增删除字段造成反序列化失败的事故发生。

public T read (Kryo kryo, Input input, Class<T> type) {
    		T object = create(kryo, input, type);
    		kryo.reference(object);
    		ObjectMap context = kryo.getGraphContext();
    		CachedField[] fields = (CachedField[])context.get(this);
    		if (fields == null) {
    			int length = input.readVarInt(true);
    			if (TRACE) trace("kryo", "Read " + length + " field names.");
    			String[] names = new String[length];
    			for (int i = 0; i < length; i++)
    				names[i] = input.readString();
    
    			fields = new CachedField[length];
    			CachedField[] allFields = getFields();
    			outer:
    			for (int i = 0, n = names.length; i < n; i++) {
    				String schemaName = names[i];
    				for (int ii = 0, nn = allFields.length; ii < nn; ii++) {
    					if (allFields[ii].field.getName().equals(schemaName)) {
    						fields[i] = allFields[ii];
    						continue outer;
    					}
    				}
    				if (TRACE) trace("kryo", "Ignore obsolete field: " + schemaName);
    			}
    			context.put(this, fields);
    		}
    		......
    }

最后分享下我们的解决方案通过添加两层缓存首先判定新的redis key是否能get到，如果为空就用旧key获取赋值给新的对象和新key ，并删除旧的缓存这样就会在一段时间内慢慢过渡沉淀最终统一新的key和对象。

**如有披露或问题欢迎留言或者入群探讨**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5NDcwNzMz_size_16_color_FFFFFF_t_70][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5NDcwNzMz_size_16_color_FFFFFF_t_70]: /images/20221120/917cad2f2cfd4ae0887578f40e8eba30.png