应用部署引起上游服务抖动问题分析及优化实践方案

作者：京东物流 朱永昌

背景介绍

本文主要围绕应用部署引起上游服务抖动问题展开，结合百川分流系统实例，提供分析、解决思路，并提供一套切实可行的实践方案。

百川分流系统作为交易订单中心的专用网关，为交易订单中心提供统一的对外标准服务（包括接单、修改、取消、回传等），对内则基于配置规则将流量分发到不同业务线的应用上。随着越来越多的流量切入百川系统，因系统部署引起服务抖动导致上游系统调用超时的问题也逐渐凸显出来。为提供稳定的交易服务系统，提升系统可用率，需要对该问题进行优化。

经调研，集团内部现有两种预热方案：

（1）JSF官方提供的预热方案；

（2）行云编排部署结合录制回放的预热方案。两种方法均无法达到预期效果。

关于方案

（1）首先，使用的前提条件是JSF消费端必需升级JSF版本到1.7.6，百川分流系统上游调用方有几十个，推动所有调用方升级版本比较困难；其次，JSF平台预热规则以接口纬度进行配置，百川分流系统对外提供46个接口，配置复杂；最关键的是该方案的预热规则配置的是在一个固定预热周期（比如1分钟）内某个接口的预热权重（接收调用量比例），简单理解就是小流量试跑，这就决定了该方案无法对系统资源进行充分预热，预热周期过后全部流量进入依然会因需要创建或初始化资源引起服务抖动，对于交易接单服务来说，抖动就会导致接单失败，有卡单风险。

关于方案

（2）通过录制线上流量进行压测回放来实现预热，适合读接口，但对于写接口如果不做特殊处理会影响线上数据；针对这个问题，目前的解决方案是通过压测标识来识别压测预热流量，但交易业务逻辑复杂，下游依赖繁多，相关系统目前并不支持。单独改造的话，接口多、风险高。

基于以上情况，我们通过百川分流系统部署引起上游服务抖动这个实例，追踪其表象线索，深入研读JSF源码，最终找到导致服务抖动的关键因素，开发了一套更加有效的预热方案，验证结果表明该方案预热效果明显，服务调用方方法性能MAX值降低90%，降到了超时时间范围内，消除了因机器部署引起上游调用超时的问题。

问题现象

系统上线部署期间，纯配接单服务上游调用方反馈接单服务抖动，出现调用超时现象。

查看此服务UMP打点，发现此服务的方法性能监控MAX值最大3073ms，未超过调用方设置的超时时间10000ms（如图1所示）

图1 服务内部监控打点

查看此服务PFinder性能监控，发现上游调用方应用调用此服务的方法性能监控MAX值多次超过10000ms（可以直接查看调用方的UMP打点，若调用方无法提供UMP打点时，也可借助PFinder的应用拓扑功能进行查看，如图2所示）

图2 服务外部监控打点

分析思路

从上述问题现象可以看出，在系统上线部署期间服务提供方接口性能MAX值并无明显抖动，但服务调用方接口性能MAX值抖动明显。由此，可以确定耗时不在服务提供方内部处理逻辑上，而是在进入服务提供方内部处理逻辑之前（或者之后），那么在之前或者之后具体都经历了什么呢？我们不着急回答这个问题，先基于现有的一些线索逐步进行追踪探索。

线索一：部署过程中机器CPU会有短暂飙升（如图3所示）

如果此时有请求调用到当前机器，接口性能势必会受到影响。因此，考虑机器部署完成且待机器CPU平稳后再上线JSF服务，这可以通过调整JSF延迟发布参数来实现。具体配置如下：

然而，实践证明JSF服务确实延迟了2分钟才上线（如图4所示），且此时CPU已经处于平稳状态，但是JSF上线瞬间又引起了CPU的二次飙升，同时调用方仍然会出现服务调用超时的现象。

图3 机器部署过程CPU短暂飙升

图4 部署和JSF上线瞬间均导致CPU飙升

线索二：JSF上线瞬间JVM线程数飙升（如图5所示）

图5 JSF上线瞬间线程数飙升

使用jstack命令工具查看线程堆栈，可以发现数量增长最多的线程是JSF-BZ线程，且都处于阻塞等待状态：

"JSF-BZ-22000-137-T-350" #1038 daemon prio=5 os_prio=0 tid=0x00007f02bcde9000 nid=0x6fff waiting on condition [0x00007efa10284000]   java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for   (a java.util.concurrent.SynchronousQueue$TransferStack)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)   Locked ownable synchronizers:- None"JSF-BZ-22000-137-T-349" #1037 daemon prio=5 os_prio=0 tid=0x00007f02bcde7000 nid=0x6ffe waiting on condition [0x00007efa10305000]   java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for   (a java.util.concurrent.SynchronousQueue$TransferStack)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)   Locked ownable synchronizers:- None"JSF-BZ-22000-137-T-348" #1036 daemon prio=5 os_prio=0 tid=0x00007f02bcdd8000 nid=0x6ffd waiting on condition [0x00007efa10386000]   java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for   (a java.util.concurrent.SynchronousQueue$TransferStack)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)   Locked ownable synchronizers:- None...

通过关键字“JSF-BZ”可以在JSF源码中检索，可以找到关于“JSF-BZ”线程池初始化源码如下：

private static synchronized ThreadPoolExecutor initPool(ServerTransportConfig transportConfig) {    final int minPoolSize, aliveTime, port = transportConfig.getPort();    int maxPoolSize = transportConfig.getServerBusinessPoolSize();    String poolType = transportConfig.getServerBusinessPoolType();    if ("fixed".equals(poolType)) { minPoolSize = maxPoolSize;    aliveTime = 0;    } else if ("cached".equals(poolType)) { minPoolSize = 20;    maxPoolSize = Math.max(minPoolSize, maxPoolSize);    aliveTime = 60000;    } else { throw new IllegalConfigureException(21401, "server.threadpool", poolType);    }    String queueType = transportConfig.getPoolQueueType();    int queueSize = transportConfig.getPoolQueueSize();    boolean isPriority = "priority".equals(queueType);    BlockingQueue configQueue = ThreadPoolUtils.buildQueue(queueSize, isPriority);    NamedThreadFactory threadFactory = new NamedThreadFactory("JSF-BZ-" + port, true);    RejectedExecutionHandler handler = new RejectedExecutionHandler() {        private int i = 1;        public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) { if (this.i++ % 7 == 0) {            this.i = 1;            BusinessPool.LOGGER.warn("[JSF-23002]Task:{} has been reject for ThreadPool exhausted! pool:{}, active:{}, queue:{}, taskcnt: {}", new Object[] { r, Integer.valueOf(executor.getPoolSize()), Integer.valueOf(executor.getActiveCount()), Integer.valueOf(executor.getQueue().size()), Long.valueOf(executor.getTaskCount()) });        }        RejectedExecutionException err = new RejectedExecutionException("[JSF-23003]Biz thread pool of provider has bean exhausted, the server port is " + port);        ProviderErrorHook.getErrorHookInstance().onProcess(new ProviderErrorEvent(err));        throw err;        }    };    LOGGER.debug("Build " + poolType + " business pool for port " + port + " [min: " + minPoolSize + " max:" + maxPoolSize + " queueType:" + queueType + " queueSize:" + queueSize + " aliveTime:" + aliveTime + "]");    return new ThreadPoolExecutor(minPoolSize, maxPoolSize, aliveTime, TimeUnit.MILLISECONDS, configQueue, (ThreadFactory)threadFactory, handler);}public static BlockingQueue buildQueue(int size, boolean isPriority) {    BlockingQueue queue;    if (size == 0) {      queue = new SynchronousQueue();    }    else if (isPriority) {      queue = (size < 0) ? new PriorityBlockingQueue() : new PriorityBlockingQueue(size);    } else {      queue = (size < 0) ? new LinkedBlockingQueue() : new LinkedBlockingQueue(size);    }         return queue;  }

另外，JSF官方文档关于线程池的说明如下：

结合JSF源码以及JSF官方文档说明，可以知道JSF-BZ线程池的阻塞队列用的是SynchronousQueue，这是一个同步阻塞队列，其中每个put必须等待一个take，反之亦然。JSF-BZ线程池默认使用的是伸缩无队列线程池，初始线程数为20个，那么在JSF上线的瞬间，大批量并发请求进入，初始化线程远不够用，因此新建了大量线程。

既然知道了是由于JSF线程池初始化线程数量不足导致的，那么我们可以考虑在应用启动时对JSF线程池进行预热，也就是说在应用启动时创建足够数量的线程备用。通过查阅JSF源码，我们找到了如下方式实现JSF线程池的预热：

// 从Spring上下文获取JSF ServerBean，可能有多个Map serverBeanMap = applicationContext.getBeansOfType(ServerBean.class);if (CollectionUtils.isEmpty(serverBeanMap)) {    log.error("application preheat, jsf thread pool preheat failed, serverBeanMap is empty.");    return;}// 遍历所有serverBean，分别做预热处理serverBeanMap.forEach((serverBeanName, serverBean) -> {    if (Objects.isNull(serverBean)) {        log.error("application preheat, jsf thread pool preheat failed, serverBean is null, serverBeanName:{}", serverBeanName);        return;    }    // 启动ServerBean，启动后才可以获取到Server    serverBean.start();    Server server = serverBean.getServer();    if (Objects.isNull(server)) {        log.error("application preheat, jsf thread pool preheat failed, JSF Server is null, serverBeanName:{}", serverBeanName);        return;    }    ServerTransportConfig serverTransportConfig = server.getTransportConfig();    if (Objects.isNull(serverTransportConfig)) {        log.error("application preheat, jsf thread pool preheat failed, serverTransportConfig is null, serverBeanName:{}", serverBeanName);        return;    }    // 获取JSF业务线程池    ThreadPoolExecutor businessPool = BusinessPool.getBusinessPool(serverTransportConfig);    if (Objects.isNull(businessPool)) {        log.error("application preheat, jsf biz pool preheat failed, businessPool is null, serverBeanName:{}", serverBeanName);        return;    }    int corePoolSize = businessPool.getCorePoolSize();    int maxCorePoolSize = Math.max(corePoolSize, 500);    if (maxCorePoolSize > corePoolSize) {        // 设置JSF server核心线程数        businessPool.setCorePoolSize(maxCorePoolSize);    }    // 初始化JSF业务线程池所有核心线程    if (businessPool.getPoolSize() < maxCorePoolSize) {        businessPool.prestartAllCoreThreads();    }}

线索三：JSF-BZ线程池预热完成后，JSF上线瞬间JVM线程数仍有升高

继续使用jstack命令工具查看线程堆栈，对比后可以发现数量有增长的线程是JSF-SEV-WORKER线程：

"JSF-SEV-WORKER-139-T-129" #1295 daemon prio=5 os_prio=0 tid=0x00007ef66000b800 nid=0x7289 runnable [0x00007ef627cf8000]   java.lang.Thread.State: RUNNABLEat sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)- locked  (a io.netty.channel.nio.SelectedSelectionKeySet)- locked  (a java.util.Collections$UnmodifiableSet)- locked  (a sun.nio.ch.EPollSelectorImpl)at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:805)at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)at java.lang.Thread.run(Thread.java:745)   Locked ownable synchronizers:- None"JSF-SEV-WORKER-139-T-128" #1293 daemon prio=5 os_prio=0 tid=0x00007ef60c002800 nid=0x7288 runnable [0x00007ef627b74000]   java.lang.Thread.State: RUNNABLEat sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)- locked  (a io.netty.channel.nio.SelectedSelectionKeySet)- locked  (a java.util.Collections$UnmodifiableSet)- locked  (a sun.nio.ch.EPollSelectorImpl)at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:805)at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)at java.lang.Thread.run(Thread.java:745)   Locked ownable synchronizers:- None"JSF-SEV-WORKER-139-T-127" #1291 daemon prio=5 os_prio=0 tid=0x00007ef608001000 nid=0x7286 runnable [0x00007ef627df9000]   java.lang.Thread.State: RUNNABLEat sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)- locked  (a io.netty.channel.nio.SelectedSelectionKeySet)- locked  (a java.util.Collections$UnmodifiableSet)- locked  (a sun.nio.ch.EPollSelectorImpl)at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:805)at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)at java.lang.Thread.run(Thread.java:745)   Locked ownable synchronizers:- None

那么JSF-SEV-WORKER线程是做什么的？我们是不是也可以对它做预热操作？带着这些疑问，再次查阅JSF源码：

private synchronized EventLoopGroup initChildEventLoopGroup() {     NioEventLoopGroup nioEventLoopGroup = null;     int threads = (this.childNioEventThreads > 0) ? this.childNioEventThreads : Math.max(8, Constants.DEFAULT_IO_THREADS);      NamedThreadFactory threadName = new NamedThreadFactory("JSF-SEV-WORKER", isDaemon());     EventLoopGroup eventLoopGroup = null;     if (isUseEpoll()) {       EpollEventLoopGroup epollEventLoopGroup = new EpollEventLoopGroup(threads, (ThreadFactory)threadName);     } else {       nioEventLoopGroup = new NioEventLoopGroup(threads, (ThreadFactory)threadName);     }      return (EventLoopGroup)nioEventLoopGroup;}

从JSF源码中可以看出JSF-SEV-WORKER线程是JSF内部使用Netty处理网络通信创建的线程，仔细研读JSF源码同样可以找到预热JSF-SEV-WORKER线程的方法，代码如下：

// 通过serverTransportConfig获取NioEventLoopGroup// 其中，serverTransportConfig的获取方式可参考JSF-BZ线程预热代码NioEventLoopGroup eventLoopGroup = (NioEventLoopGroup) serverTransportConfig.getChildEventLoopGroup();int threadSize = this.jsfSevWorkerThreads;while (threadSize-- > 0) {    new Thread(() -> {        // 通过手工提交任务的方式创建JSF-SEV-WORKER线程达到预热效果        eventLoopGroup.submit(() -> log.info("submit thread to netty by hand, threadName:{}", Thread.currentThread().getName()));    }).start();}

JSF-BZ线程、JSF-SEV-WORKER线程预热效果如下图所示：

图6 JSF-BZ/JSF-SEV-WORKER线程预热效果

挖掘源码线索

至此，经过JSF延迟发布、JSF内部线程池预热后，系统部署引起服务调用方抖动超时的现象有一定缓解（从原来的10000ms-20000ms降低到5000ms-10000ms），虽然说是有效果，但还有些不尽如人意。应该还是有优化空间的，现在是时候考虑我们最开始留下的那个疑问了：“服务调用方在进入服务提供方内部处理逻辑之前（或者之后），具体都经历了什么？”。最容易想到的肯定是中间经过了网络，但是网络因素基本可以排除，因为在部署过程中机器网络性能正常，那么还有哪些影响因素呢？此时我们还是要回归到JSF源码中去寻找线索。

图7 JSF源码中Provider内部处理过程

经过仔细研读JSF源码，我们可以发现JSF内部对于接口出入参有一系列编码、解码、序列化、反序列化的操作，而且在这些操作中我们有了惊喜的发现：本地缓存，部分源码如下：

DESC_CLASS_CACHE

private static final ConcurrentMap<String, Class> DESC_CLASS_CACHE = new ConcurrentHashMap<String, Class>();private static Class desc2class(ClassLoader cl, String desc) throws ClassNotFoundException {  switch (desc.charAt(0)) {    case 'V':      return void.class;    case 'Z': return boolean.class;    case 'B': return byte.class;    case 'C': return char.class;    case 'D': return double.class;    case 'F': return float.class;    case 'I': return int.class;    case 'J': return long.class;    case 'S': return short.class;    case 'L':      desc = desc.substring(1, desc.length() - 1).replace('/', '.');      break;    case '[':      desc = desc.replace('/', '.');      break;    default:      throw new ClassNotFoundException("Class not found: " + desc);  }     if (cl == null)    cl = ClassLoaderUtils.getCurrentClassLoader();   Class clazz = DESC_CLASS_CACHE.get(desc);  if (clazz == null) {    clazz = Class.forName(desc, true, cl);    DESC_CLASS_CACHE.put(desc, clazz);  }   return clazz;}

NAME_CLASS_CACHE

private static final ConcurrentMap<String, Class> NAME_CLASS_CACHE = new ConcurrentHashMap<String, Class>();private static Class name2class(ClassLoader cl, String name) throws ClassNotFoundException {  int c = 0, index = name.indexOf('[');  if (index > 0) {        c = (name.length() - index) / 2;    name = name.substring(0, index);  }   if (c > 0) {        StringBuilder sb = new StringBuilder();    while (c-- > 0) {      sb.append("[");    }    if ("void".equals(name)) { sb.append('V'); }    else if ("boolean".equals(name)) { sb.append('Z'); }    else if ("byte".equals(name)) { sb.append('B'); }    else if ("char".equals(name)) { sb.append('C'); }    else if ("double".equals(name)) { sb.append('D'); }    else if ("float".equals(name)) { sb.append('F'); }    else if ("int".equals(name)) { sb.append('I'); }    else if ("long".equals(name)) { sb.append('J'); }    else if ("short".equals(name)) { sb.append('S'); }    else { sb.append('L').append(name).append(';'); }     name = sb.toString();  }  else {        if ("void".equals(name)) return void.class;     if ("boolean".equals(name)) return boolean.class;     if ("byte".equals(name)) return byte.class;     if ("char".equals(name)) return char.class;     if ("double".equals(name)) return double.class;     if ("float".equals(name)) return float.class;     if ("int".equals(name)) return int.class;     if ("long".equals(name)) return long.class;     if ("short".equals(name)) return short.class;    }   if (cl == null)    cl = ClassLoaderUtils.getCurrentClassLoader();   Class clazz = NAME_CLASS_CACHE.get(name);  if (clazz == null) {    clazz = Class.forName(name, true, cl);    NAME_CLASS_CACHE.put(name, clazz);  }   return clazz;}

SerializerCache

private ConcurrentHashMap _cachedSerializerMap;public Serializer getSerializer(Class cl) throws HessianProtocolException {  Serializer serializer = (Serializer)_staticSerializerMap.get(cl);  if (serializer != null) {    return serializer;  }    if (this._cachedSerializerMap != null) {    serializer = (Serializer)this._cachedSerializerMap.get(cl);    if (serializer != null) {      return serializer;    }  }     int i = 0;  for (; serializer == null && this._factories != null && i < this._factories.size();     i++) {        AbstractSerializerFactory factory = this._factories.get(i);        serializer = factory.getSerializer(cl);  }     if (serializer == null)  {    if (isZoneId(cl)) {      ZoneIdSerializer zoneIdSerializer = ZoneIdSerializer.getInstance();    } else if (isEnumSet(cl)) {      serializer = EnumSetSerializer.getInstance();    } else if (JavaSerializer.getWriteReplace(cl) != null) {      serializer = new JavaSerializer(cl, this._loader);    }    else if (HessianRemoteObject.class.isAssignableFrom(cl)) {      serializer = new RemoteSerializer();        }    else if (Map.class.isAssignableFrom(cl)) {      if (this._mapSerializer == null) {        this._mapSerializer = new MapSerializer();      }      serializer = this._mapSerializer;    } else if (Collection.class.isAssignableFrom(cl)) {      if (this._collectionSerializer == null) {        this._collectionSerializer = new CollectionSerializer();      }            serializer = this._collectionSerializer;    } else if (cl.isArray()) {      serializer = new ArraySerializer();    } else if (Throwable.class.isAssignableFrom(cl)) {      serializer = new ThrowableSerializer(cl, getClassLoader());    } else if (InputStream.class.isAssignableFrom(cl)) {      serializer = new InputStreamSerializer();    } else if (Iterator.class.isAssignableFrom(cl)) {      serializer = IteratorSerializer.create();    } else if (Enumeration.class.isAssignableFrom(cl)) {      serializer = EnumerationSerializer.create();    } else if (Calendar.class.isAssignableFrom(cl)) {      serializer = CalendarSerializer.create();    } else if (Locale.class.isAssignableFrom(cl)) {      serializer = LocaleSerializer.create();    } else if (Enum.class.isAssignableFrom(cl)) {      serializer = new EnumSerializer(cl);    }   }  if (serializer == null) {    serializer = getDefaultSerializer(cl);  }    if (this._cachedSerializerMap == null) {    this._cachedSerializerMap = new ConcurrentHashMap