线上JAVA应用平稳运行一段时间后出现JVM崩溃问题

一、问题是怎么发现的

系统是一个定时任务系统，需要定时执行业务代码，业务代码主要是访问MYSQL数据库和缓存进行操作，该开始启动，系统日志一切正常，但是运行一段时间到凌晨后，系统就自动崩溃了，java进程没有了，只留下了程序崩溃日志如下：

cat: /proc/1/environ: Permission denied[admin@host-11-40-38-52 ~]$ more hs_err_pid231.log## A fatal error has been detected by the Java Runtime Environment:##  SIGSEGV (0xb) at pc=0x00007f21a8c21325, pid=231, tid=139779725313792## JRE version: Java(TM) SE Runtime Environment (8.0_20-b26) (build 1.8.0_20-b26)# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.20-b23 mixed mode linux-amd64 compressed oops)# Problematic frame:# V  [libjvm.so+0x858325]  LoadKlassNode::make(PhaseGVN&, Node*, Node*, TypePtrconst*, TypeKlassPtr const*)+0x45## Core dump written. Default location: /home/admin/core or core.231## If you would like to submit a bug report, please visit:#   http://bugreport.sun.com/bugreport/crash.jsp#---------------  T H R E A D  ---------------Current thread (0x00007f21a4b80800):  JavaThread "C2 CompilerThread5" daemon [_thread_in_native, id=398, stack(0x00007f2100cfc000,0x00007f2100dfd000)]siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000010Registers:RAX=0x00007f21a48fbab0, RBX=0x00007f21a48fe4b0, RCX=0x0000000000000000, RDX=0x00000000fffffff4RSP=0x00007f2100df9c40, RBP=0x00007f2100df9c80, RSI=0x00007f20b00e9050, RDI=0x00007f1fe84e3650R8 =0x00007f21a48fe4b0, R9 =0x00007f21a48fbab0, R10=0x00007f1ffc059d90, R11=0x00007f21a8cf3560R12=0x00007f2100dfad00, R13=0x00007f1fe84e3650, R14=0x00007f2100dfa100, R15=0x00007f2100df9ee0RIP=0x00007f21a8c21325, EFLAGS=0x0000000000010296, CSGSFS=0x0000000000000033, ERR=0x0000000000000004  TRAPNO=0x000000000000000eTop of Stack: (sp=0x00007f2100df9c40)0x00007f2100df9c40:   00007f20b00e9050 00007f21a48fbab00x00007f2100df9c50:   00007f2100df9c80 00007f21a48fe4b00x00007f2100df9c60:   00007f21a48fd440 00007f1fe84e36500x00007f2100df9c70:   00007f21a8cf3560 00007f2100df9ee00x00007f2100df9c80:   00007f2100df9da0 00007f21a891acb60x00007f2100df9c90:   00007f1ffc5122a0 00007f20b00301d00x00007f2100df9ca0:   00007f2100df9ce0 00007f1ffc933fc00x00007f2100df9cb0:   00007f2100df9ee0 00007f1ffc933fc00x00007f2100df9cc0:   00007f2100df9cf0 00007f21a89ac7980x00007f2100df9cd0:   00007f2100df9cf0 00007f2100dfad000x00007f2100df9ce0:   0000000000000000 00000000000000020x00007f2100df9cf0:   00007f1ffc933fc0 00007f1ffc93ba680x00007f2100df9d00:   0000000000000000 00007f1ffc93ba680x00007f2100df9d10:   00007f2100df9d30 00007f21000000020x00007f2100df9d20:   00007f1ffc93b828 00000000000000000x00007f2100df9d30:   00007f20b0029970 00000005000000040x00007f2100df9d40:   00007f20b00301d0 00007f00000001b00x00007f2100df9d50:   000000c700000e83 00007f1ffc93b8280x00007f2100df9d60:   00007f2100dfa100 00007f1ffc5126a00x00007f2100df9d70:   0000000000001019 00007f1ffc933fc00x00007f2100df9d80:   00007f2100df9ee0 00007f2100df9db00x00007f2100df9d90:   0000000000000000 00007f2100df9fd80x00007f2100df9da0:   00007f2100df9df0 00007f21a8cd1b0a0x00007f2100df9db0:   00007f2100df9ee0 00000000000000000x00007f2100df9dc0:   00007f1f00000001 00007f2100df9ee00x00007f2100df9dd0:   0000000000000000 00007f2100df9ee00x00007f2100df9de0:   0000000000000000 00000000000001b00x00007f2100df9df0:   00007f2100df9e30 00007f21a8cd46f00x00007f2100df9e00:   00007f2100df9e30 00007f2100dfabd00x00007f2100df9e10:   00007f20b00fd458 00007f2100df9ee00x00007f2100df9e20:   000000000000001f 00007f1ffc9365080x00007f2100df9e30:   00007f2100df9e80 00007f21a8cd4967Instructions: (pc=0x00007f21a8c21325)0x00007f21a8c21305:   c0 48 8b 02 4c 8b 67 10 49 89 fe 48 89 d7 ff 500x00007f21a8c21315:   28 8b 50 10 31 c9 83 ea 0e 83 fa 07 48 0f 42 c80x00007f21a8c21325:   8b 41 10 83 e8 10 83 f8 02 0f 87 9c 01 00 00 800x00007f21a8c21335:   79 32 00 0f 84 92 01 00 00 49 8b 06 49 89 df 48Register to memory mapping:RAX=0x00007f21a48fbab0 is an unknown valueRBX=0x00007f21a48fe4b0 is an unknown valueRCX=0x0000000000000000 is an unknown valueRDX=0x00000000fffffff4 is an unknown valueRSP=0x00007f2100df9c40 is pointing into the stack for thread: 0x00007f21a4b80800RBP=0x00007f2100df9c80 is pointing into the stack for thread: 0x00007f21a4b80800RSI=0x00007f20b00e9050 is an unknown valueRDI=0x00007f1fe84e3650 is an unknown valueR8 =0x00007f21a48fe4b0 is an unknown valueR9 =0x00007f21a48fbab0 is an unknown valueR10=0x00007f1ffc059d90 is an unknown valueR11=0x00007f21a8cf3560:  in /home/export/servers/jdk1.8.0_20/jre/lib/amd64/server/libjvm.so at 0x00007f21a83c9000R12=0x00007f2100dfad00 is pointing into the stack for thread: 0x00007f21a4b80800R13=0x00007f1fe84e3650 is an unknown valueR14=0x00007f2100dfa100 is pointing into the stack for thread: 0x00007f21a4b80800R15=0x00007f2100df9ee0 is pointing into the stack for thread: 0x00007f21a4b80800Stack: [0x00007f2100cfc000,0x00007f2100dfd000],  sp=0x00007f2100df9c40,  free space=1015kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)V  [libjvm.so+0x858325]  LoadKlassNode::make(PhaseGVN&, Node*, Node*, TypePtr const*, TypeKlassPtr const*)+0x45V  [libjvm.so+0x551cb6]  Parse::catch_inline_exceptions(SafePointNode*)+0x946V  [libjvm.so+0x908b0a]  Parse::do_exceptions()+0xbaV  [libjvm.so+0x90b6f0]  Parse::do_one_block()+0x180V  [libjvm.so+0x90b967]  Parse::do_all_blocks()+0x127V  [libjvm.so+0x90fd58]  Parse::Parse(JVMState*, ciMethod*, float, Parse*)+0x7b8V  [libjvm.so+0x3e7d09]  ParseGenerator::generate(JVMState*, Parse*)+0x99V  [libjvm.so+0x4931cc]  Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool)+0x126cV  [libjvm.so+0x3e67e8]  C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0x198V  [libjvm.so+0x49ba4a]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0xc8aV  [libjvm.so+0x49e420]  CompileBroker::compiler_thread_loop()+0x620V  [libjvm.so+0xa2990f]  JavaThread::thread_main_inner()+0xdfV  [libjvm.so+0xa29a3c]  JavaThread::run()+0x11cV  [libjvm.so+0x8e75f8]  java_start(Thread*)+0x108C  [libpthread.so.0+0x7aa1]  start_thread+0xd1Current CompileTask:C2:43781461 13331   !   4       com.mysql.cj.NativeSession::execSQL (579 bytes)---------------  P R O C E S S  ---------------Java Threads: ( => current thread )  0x00007f2010003800 JavaThread "qtp1589683045-1271" [_thread_blocked, id=142149, stack(0x00007f20fa995000,0x00007f20faa96000)]  0x00007f1f58001800 JavaThread "Keep-Alive-Timer" daemon [_thread_blocked, id=140557, stack(0x00007f20f830e000,0x00007f20f840f000)]  0x00007f1fb005e800 JavaThread "logback-1" daemon [_thread_blocked, id=42936, stack(0x00007f20fad99000,0x00007f20fae9a000)]  0x00007f1f7c0c3000 JavaThread "logback-8" daemon [_thread_blocked, id=565, stack(0x00007f20ea6f6000,0x00007f20ea7f7000)]  0x00007f1f50003000 JavaThread "logback-7" daemon [_thread_blocked, id=521, stack(0x00007f20fab97000,0x00007f20fac98000)]  0x00007f1f10001000 JavaThread "logback-6" daemon [_thread_blocked, id=520, stack(0x00007f20f002b000,0x00007f20f012c000)]  0x00007f1f74004800 JavaThread "logback-5" daemon [_thread_blocked, id=519, stack(0x00007f20f1830000,0x00007f20f1931000)]  0x00007f1f5441d000 JavaThread "UMP-CheckFileRemovedThread" daemon [_thread_blocked, id=518, stack(0x00007f20ea7f7000,0x00007f20ea8f8000)]  0x00007f1f5441b800 JavaThread "UMP-ProfilerFileUpdateThread" daemon [_thread_blocked, id=517, stack(0x00007f20ea8f8000,0x00007f20ea9f9000)]  0x00007f1f54419800 JavaThread "UMP-WriteLog2FileThread-jvmLogger" daemon [_thread_blocked, id=516, stack(0x00007f20ea9f9000,0x00007f20eaafa000)]  0x00007f1f54418000 JavaThread "UMP-WriteLog2FileThread-commonLogger" daemon [_thread_blocked, id=515, stack(0x00007f20eaafa000,0x00007f20eabfb000)]  0x00007f1f54416000 JavaThread "UMP-WriteLog2FileThread-tpLogger" daemon [_thread_blocked, id=514, stack(0x00007f20eabfb000,0x00007f20eacfc000)]  0x00007f1f5440f800 JavaThread "UMP-WriteLog2FileThread-bizLogger" daemon [_thread_blocked, id=513, stack(0x00007f20eacfc000,0x00007f20eadfd000)]  0x00007f1f54410800 JavaThread "UMP-WriteLog2FileThread-businessLogger" daemon[_thread_blocked, id=512, stack(0x00007f20eadfd000,0x00007f20eaefe000)]  0x00007f1f54408800 JavaThread "UMP-WriteLog2FileThread-aliveLogger" daemon [_thread_blocked, id=511, stack(0x00007f20eaefe000,0x00007f20eafff000)]  0x00007f1f543da000 JavaThread "DatebookHikariCP2 housekeeper" daemon [_thread_blocked, id=509, stack(0x00007f20f012c000,0x00007f20f022d000)]  0x00007f1f543cf800 JavaThread "Jim-NioEventLoop - 1" daemon [_thread_in_native, id=508, stack(0x00007f20f042d000,0x00007f20f052e000)]  0x00007f1f54259800 JavaThread "SystemClock" daemon [_thread_blocked, id=507, stack(0x00007f20f172f000,0x00007f20f1830000)]  0x00007f1f54216000 JavaThread "DatebookHikariCP4 housekeeper" daemon [_thread_blocked, id=505, stack(0x00007f20f289f000,0x00007f20f29a0000)]  0x00007f1f540b5000 JavaThread "UMP-WriteTPLogThread" daemon [_thread_blocked,id=504, stack(0x00007f20fa490000,0x00007f20fa591000)]  0x00007f1f50002000 JavaThread "logback-4" daemon [_thread_blocked, id=503, stack(0x00007f20fa38f000,0x00007f20fa490000)]  0x00007f1f74003000 JavaThread "logback-3" daemon [_thread_blocked, id=502, stack(0x00007f20f29a0000,0x00007f20f2aa1000)]  0x00007f21a4bc4000 JavaThread "http-nio-8001-Acceptor-0" daemon [_thread_in_native, id=501, stack(0x00007f20f1d31000,0x00007f20f1e32000)]  0x00007f21a4bc2000 JavaThread "http-nio-8001-ClientPoller-1" daemon [_thread_in_native, id=500, stack(0x00007f20f1e32000,0x00007f20f1f33000)]  0x00007f21a4bc0000 JavaThread "http-nio-8001-ClientPoller-0" daemon [_thread_in_native, id=499, stack(0x00007f20f1f33000,0x00007f20f2034000)]  0x00007f21a4bbe800 JavaThread "http-nio-8001-exec-10" daemon [_thread_blocked, id=498, stack(0x00007f20f2034000,0x00007f20f2135000)]  0x00007f21a4958800 JavaThread "http-nio-8001-exec-9" daemon [_thread_blocked,id=497, stack(0x00007f20f2135000,0x00007f20f2236000)]  0x00007f21a4956800 JavaThread "http-nio-8001-exec-8" daemon [_thread_blocked,id=496, stack(0x00007f20f2236000,0x00007f20f2337000)]  0x00007f21a4955000 JavaThread "http-nio-8001-exec-7" daemon [_thread_blocked,id=495, stack(0x00007f20f2337000,0x00007f20f2438000)]  0x00007f21a4953800 JavaThread "http-nio-8001-exec-6" daemon [_thread_blocked,id=494, stack(0x00007f20f2438000,0x00007f20f2539000)]  0x00007f21a4952000 JavaThread "http-nio-8001-exec-5" daemon [_thread_blocked,id=493, stack(0x00007f20f2539000,0x00007f20f263a000)]  0x00007f21a4b7e800 JavaThread "http-nio-8001-exec-4" daemon [_thread_blocked,id=492, stack(0x00007f20f263a000,0x00007f20f273b000)]  0x00007f21a4b7d000 JavaThread "http-nio-8001-exec-3" daemon [_thread_blocked,id=491, stack(0x00007f20f273b000,0x00007f20f283c000)]  0x00007f21a4b7b800 JavaThread "http-nio-8001-exec-2" daemon [_thread_blocked,id=490, stack(0x00007f20f2cfd000,0x00007f20f2dfe000)]  0x00007f21a4b7a000 JavaThread "http-nio-8001-exec-1" daemon [_thread_blocked,id=489, stack(0x00007f20f2dfe000,0x00007f20f2eff000)]  0x00007f21a4b78800 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=488, stack(0x00007f20f2aa1000,0x00007f20f2ba2000)]  0x00007f1f7cb62800 JavaThread "scheduling-1" [_thread_in_native, id=487, stack(0x00007f20f2ba2000,0x00007f20f2ca3000)]  0x00007f1f74001000 JavaThread "logback-2" daemon [_thread_blocked, id=486, stack(0x00007f20f2eff000,0x00007f20f3000000)]  0x00007f1f7ca2b800 JavaThread "Jim-ConfigPoolingHeartbeat-1571143858733" daemon [_thread_in_native, id=442, stack(0x00007f20f800b000,0x00007f20f810c000)]  0x00007f1f7c913800 JavaThread "mysql-cj-abandoned-connection-cleanup" daemon [_thread_blocked, id=441, stack(0x00007f20f8833000,0x00007f20f8934000)]  0x00007f1f7c7b8800 JavaThread "configurator-supervise" daemon [_thread_blocked, id=440, stack(0x00007f20f810c000,0x00007f20f820d000)]  0x00007f1f7c7b7000 JavaThread "Thread-51" daemon [_thread_blocked, id=439, stack(0x00007f20f820d000,0x00007f20f830e000)]  0x00007f1f7c71e000 JavaThread "JSF-Future-Checker-CB-1-T-1" daemon [_thread_blocked, id=436, stack(0x00007f20f8b34000,0x00007f20f8c35000)]  0x00007f1f7c71d800 JavaThread "JSF-Future-Checker-0-T-1" daemon [_thread_blocked, id=435, stack(0x00007f20f9bc2000,0x00007f20f9cc3000)]  0x00007f1f7c3ba000 JavaThread "logback-1" daemon [_thread_blocked, id=433, stack(0x00007f20f9635000,0x00007f20f9736000)]  0x00007f21a4e27800 JavaThread "NioBlockingSelector.BlockPoller-1" daemon [_thread_in_native, id=429, stack(0x00007f20fa591000,0x00007f20fa692000)]  0x00007f2030001000 JavaThread "qtp1589683045-59 Selector15" [_thread_in_native, id=414, stack(0x00007f20fb39f000,0x00007f20fb4a0000)]  0x00007f21a4da0800 JavaThread "GC Daemon" daemon [_thread_blocked, id=413, stack(0x00007f20fbafd000,0x00007f20fbbfe000)]  0x00007f21a4baa000 JavaThread "Service Thread" daemon [_thread_blocked, id=411, stack(0x00007f20fbeff000,0x00007f20fc000000)]  0x00007f21a4b9a800 JavaThread "C1 CompilerThread17" daemon [_thread_blocked, id=410, stack(0x00007f21000f0000,0x00007f21001f1000)]  0x00007f21a4b98800 JavaThread "C1 CompilerThread16" daemon [_thread_blocked, id=409, stack(0x00007f21001f1000,0x00007f21002f2000)]  0x00007f21a4b96800 JavaThread "C1 CompilerThread15" daemon [_thread_blocked, id=408, stack(0x00007f21002f2000,0x00007f21003f3000)]  0x00007f21a4b94800 JavaThread "C1 CompilerThread14" daemon [_thread_blocked, id=407, stack(0x00007f21003f3000,0x00007f21004f4000)]  0x00007f21a4b92000 JavaThread "C1 CompilerThread13" daemon [_thread_blocked, id=406, stack(0x00007f21004f4000,0x00007f21005f5000)]  0x00007f21a4b90000 JavaThread "C1 CompilerThread12" daemon [_thread_blocked, id=405, stack(0x00007f21005f5000,0x00007f21006f6000)]  0x00007f21a4b8e000 JavaThread "C2 CompilerThread11" daemon [_thread_blocked, id=404, stack(0x00007f21006f6000,0x00007f21007f7000)]  0x00007f21a4b8b800 JavaThread "C2 CompilerThread10" daemon [_thread_blocked, id=403, stack(0x00007f21007f7000,0x00007f21008f8000)]  0x00007f21a4b89800 JavaThread "C2 CompilerThread9" daemon [_thread_blocked, id=402, stack(0x00007f21008f8000,0x00007f21009f9000)]  0x00007f21a4b87000 JavaThread "C2 CompilerThread8" daemon [_thread_blocked, id=401, stack(0x00007f21009f9000,0x00007f2100afa000)]  0x00007f21a4b85000 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=400, stack(0x00007f2100afa000,0x00007f2100bfb000)]  0x00007f21a4b83000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=399, stack(0x00007f2100bfb000,0x00007f2100cfc000)]=>0x00007f21a4b80800 JavaThread "C2 CompilerThread5" daemon [_thread_in_native,id=398, stack(0x00007f2100cfc000,0x00007f2100dfd000)]  0x00007f21a4b76800 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=397, stack(0x00007f2100dfd000,0x00007f2100efe000)]  0x00007f21a4b74000 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=396, stack(0x00007f2100efe000,0x00007f2100fff000)]  0x00007f21a4b72000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=395, stack(0x00007f2108098000,0x00007f2108199000)]  0x00007f21a4b6f800 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=394, stack(0x00007f2108199000,0x00007f210829a000)]  0x00007f21a4b6e000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=393, stack(0x00007f210829a000,0x00007f210839b000)]  0x00007f21a4b6b000 JavaThread "qtp1589683045-38 Selector14" [_thread_in_native, id=392, stack(0x00007f210839b000,0x00007f210849c000)]  0x00007f21a4b69000 JavaThread "qtp1589683045-37 Selector13" [_thread_in_native, id=391, stack(0x00007f210849c000,0x00007f210859d000)]  0x00007f21a4b67000 JavaThread "qtp1589683045-36 Selector12" [_thread_in_native, id=390, stack(0x00007f210859d000,0x00007f210869e000)]  0x00007f21a4b65000 JavaThread "qtp1589683045-35 Selector11" [_thread_in_native, id=389, stack(0x00007f210869e000,0x00007f210879f000)]  0x00007f21a4b62800 JavaThread "qtp1589683045-34 Selector10" [_thread_in_native, id=388, stack(0x00007f210879f000,0x00007f21088a0000)]  0x00007f21a4b60800 JavaThread "qtp1589683045-33 Selector9" [_thread_in_native, id=387, stack(0x00007f21088a0000,0x00007f21089a1000)]  0x00007f21a4b5e800 JavaThread "qtp1589683045-32 Selector8" [_thread_in_native, id=386, stack(0x00007f21089a1000,0x00007f2108aa2000)]  0x00007f21a4b5c800 JavaThread "qtp1589683045-31 Selector7" [_thread_in_native, id=385, stack(0x00007f2108aa2000,0x00007f2108ba3000)]  0x00007f21a4b5a800 JavaThread "qtp1589683045-30 Selector6" [_thread_in_native, id=384, stack(0x00007f2108ba3000,0x00007f2108ca4000)]  0x00007f21a4b58800 JavaThread "qtp1589683045-29 Selector5" [_thread_in_native, id=383, stack(0x00007f2108ca4000,0x00007f2108da5000)]  0x00007f21a4b56800 JavaThread "qtp1589683045-28 Selector4" [_thread_in_native, id=382, stack(0x00007f2108da5000,0x00007f2108ea6000)]  0x00007f21a4b54800 JavaThread "qtp1589683045-27 Selector3" [_thread_in_native, id=381, stack(0x00007f2108ea6000,0x00007f2108fa7000)]  0x00007f21a4b52800 JavaThread "qtp1589683045-26 Selector2" [_thread_in_native, id=380, stack(0x00007f2108fa7000,0x00007f21090a8000)]  0x00007f21a4b50800 JavaThread "qtp1589683045-25 Selector1" [_thread_in_native, id=379, stack(0x00007f21090a8000,0x00007f21091a9000)]  0x00007f21a49ad800 JavaThread "qtp1589683045-24 Selector0" [_thread_in_native, id=378, stack(0x00007f21091a9000,0x00007f21092aa000)]  0x00007f21a49ac000 JavaThread "qtp1589683045-23 Acceptor15 SelectChannelConnector@0.0.0.0:7001 STARTING" [_thread_blocked, id=377, stack(0x00007f21092aa000,0x00007f21093ab000)]  0x00007f21a49a8800 JavaThread "qtp1589683045-22 Acceptor14 SelectChannelConnector@0.0.0.0:7001 STARTING" [_thread_blocked, id=376, stack(0x00007f21093ab000,0x00007f21094ac000)]  0x00007f21a49a6800 JavaThread "qtp1589683045-21 Acceptor13 SelectChannelConnector@0.0.0.0:7001 STARTING" [_thread_blocked, id=375, stack(0x00007f21094ac000,0x00007f21095ad000)]  0x00007f21a49a4800 JavaThread "qtp1589683045-20 Acceptor12 SelectChannelConnector@0.0.0.0:7001 STARTING" [_thread_blocked, id=374, stack(0x00007f21095ad000,0x00007f21096ae000)]  0x00007f21a49a2800 JavaThread "qtp1589683045-19 Acceptor11 SelectChannelConnector@0.0.0.0:7001 STARTING" [_thread_blocked, id=373, stack(0x00007f21096ae000,0x00007f21097af000)]  0x00007f21a49a0800 JavaThread "qtp1589683045-18 Acceptor10 SelectChannelConnector@0.0.0.0:7001 STARTING" [_thread_blocked, id=372, stack(0x00007f21097af000,0x00007f21098b0000)]  0x00007f21a499e800 JavaThread "qtp1589683045-17 Acceptor9 SelectChannelConnector@0.0.0.0:7001 STARTING" [_thread_blocked, id=371, stack(0x00007f21098b0000,0x00007f21099b1000)]  0x00007f21a499d800 JavaThread "qtp1589683045-16 Acceptor7 SelectChannelConnector@0.0.0.0:7001 STARTING" [_thread_in_native, id=370, stack(0x00007f21099b1000,0x00007f2109ab2000)]  0x00007f21a4994800 JavaThread "qtp1589683045-15 Acceptor8 SelectChannelConnector@0.0.0.0:7001 STARTING" [_thread_blocked, id=369, stack(0x00007f2109ab2000,0x00007f2109bb3000)]  0x00007f21a4992800 JavaThread "qtp1589683045-14 Acceptor6 SelectChannelConnector@0.0.0.0:7001 STARTING" [_thread_blocked, id=368, stack(0x00007f2109bb3000,0x00007f2109cb4000)]  0x00007f21a4990800 JavaThread "qtp1589683045-13 Acceptor3 SelectChannelConnector@0.0.0.0:7001 STARTING" [_thread_blocked, id=367, stack(0x00007f2109cb4000,0x00007f2109db5000)]  0x00007f21a498e800 JavaThread "qtp1589683045-12 Acceptor2 SelectChannelConnector@0.0.0.0:7001 STARTING" [_thread_blocked, id=366, stack(0x00007f2109db5000,0x00007f2109eb6000)]  0x00007f21a498c800 JavaThread "qtp1589683045-11 Acceptor5 SelectChannelConnector@0.0.0.0:7001 STARTING" [_thread_blocked, id=365, stack(0x00007f2109eb6000,0x00007f2109fb7000)]  0x00007f21a498a800 JavaThread "qtp1589683045-10 Acceptor4 SelectChannelConnector@0.0.0.0:7001 STARTING" [_thread_blocked, id=364, stack(0x00007f2109fb7000,0x00007f210a0b8000)]  0x00007f21a4988800 JavaThread "qtp1589683045-9 Acceptor1 SelectChannelConnector@0.0.0.0:7001 STARTING" [_thread_blocked, id=363, stack(0x00007f210a0b8000,0x00007f210a1b9000)]  0x00007f21a4987000 JavaThread "qtp1589683045-8 Acceptor0 SelectChannelConnector@0.0.0.0:7001 STARTING" [_thread_blocked, id=362, stack(0x00007f210a1b9000,0x00007f210a2ba000)]  0x00007f21a497c000 JavaThread "HashSessionScavenger-0" daemon [_thread_blocked, id=361, stack(0x00007f210a2ba000,0x00007f210a3bb000)]  0x00007f21a46a5800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=360, stack(0x00007f210abec000,0x00007f210aced000)]  0x00007f21a4676800 JavaThread "Finalizer" daemon [_thread_blocked, id=359, stack(0x00007f210aced000,0x00007f210adee000)]  0x00007f21a4674800 JavaThread "Reference Handler" daemon [_thread_blocked, id=358, stack(0x00007f210adee000,0x00007f210aeef000)]  0x00007f21a400b000 JavaThread "main" [_thread_in_native, id=243, stack(0x00007f21a9e2b000,0x00007f21a9f2c000)]Other Threads:  0x00007f21a466d800 VMThread [stack: 0x00007f210aeef000,0x00007f210aff0000] [id=355]  0x00007f21a4bad000 WatcherThread [stack: 0x00007f20fbdfe000,0x00007f20fbeff000] [id=412]VM state:not at safepoint (normal execution)VM Mutex/Monitor currently owned by a thread: NoneHeap: PSYoungGen      total 1305088K, used 1009509K [0x000000076ab00000, 0x00000007c0000000, 0x00000007c0000000)  eden space 1209856K, 82% used [0x000000076ab00000,0x00000007a7b58fa8,0x00000007b4880000)  from space 95232K, 10% used [0x00000007ba300000,0x00000007bac804c0,0x00000007c0000000)  to   space 92672K, 0% used [0x00000007b4880000,0x00000007b4880000,0x00000007ba300000) ParOldGen       total 2796544K, used 50182K [0x00000006c0000000, 0x000000076ab00000, 0x000000076ab00000)  object space 2796544K, 1% used [0x00000006c0000000,0x00000006c31019c8,0x000000076ab00000) Metaspace       used 69176K, capacity 71536K, committed 71808K, reserved 1112064K  class space    used 7973K, capacity 8461K, committed 8576K, reserved 1048576KCard table byte_map: [0x00007f21937c9000,0x00007f2193fca000] byte_map_base: 0x00007f21901c9000Marking Bits: (ParMarkBitMap*) 0x00007f21a9320ac0 Begin Bits: [0x00007f20a4000000, 0x00007f20a8000000) End Bits:   [0x00007f20a8000000, 0x00007f20ac000000)Polling page: 0x00007f21a9f33000CodeCache: size=245760Kb used=45854Kb max_used=45878Kb free=199905Kb bounds [0x00007f219438a000, 0x00007f219709a000, 0x00007f21a338a000] total_blobs=11396 nmethods=10825 adapters=479 compilation: enabledCompilation events (10 events):Event: 42853.758 Thread 0x00007f21a4b98800 nmethod 13322 0x00007f21967d2650 code [0x00007f21967d2840, 0x00007f21967d30f8]Event: 42941.467 Thread 0x00007f21a4b89800 13327       4       java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue::add (9 bytes)Event: 42941.467 Thread 0x00007f21a4b87000 13328   !   4       java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue::offer (124 bytes)Event: 42941.467 Thread 0x00007f21a4b80800 13329       4       java.util.concurrent.ThreadPoolExecutor::ensurePrestart (41 bytes)Event: 42941.469 Thread 0x00007f21a4b80800 nmethod 13329 0x00007f21967d0dd0 code [0x00007f21967d0f20, 0x00007f21967d0ff8]Event: 42941.493 Thread 0x00007f21a4b87000 nmethod 13328 0x00007f219708a2d0 code [0x00007f219708a540, 0x00007f219708b680]Event: 42941.493 Thread 0x00007f21a4b89800 nmethod 13327 0x00007f2197070390 code [0x00007f2197070600, 0x00007f2197071700]Event: 43421.467 Thread 0x00007f21a4b74000 13330       4       com.mysql.cj.result.DefaultColumnDefinition:: (41 bytes)Event: 43421.469 Thread 0x00007f21a4b74000 nmethod 13330 0x00007f2196854910 code [0x00007f2196854a60, 0x00007f2196854b78]Event: 43781.449 Thread 0x00007f21a4b80800 13331   !   4       com.mysql.cj.NativeSession::execSQL (579 bytes)GC Heap History (10 events):Event: 2901.101 GC heap before{Heap before GC invocations=9 (full 3): PSYoungGen      total 1223168K, used 1048576K [0x000000076ab00000, 0x00000007c0000000, 0x00000007c0000000)  eden space 1048576K, 100% used [0x000000076ab00000,0x00000007aab00000,0x00000007aab00000)  from space 174592K, 0% used [0x00000007aab00000,0x00000007aab00000,0x00000007b5580000)  to   space 101376K, 0% used [0x00000007b9d00000,0x00000007b9d00000,0x00000007c0000000) ParOldGen       total 2796544K, used 50126K [0x00000006c0000000, 0x000000076ab00000, 0x000000076ab00000)  object space 2796544K, 1% used [0x00000006c0000000,0x00000006c30f39b8,0x000000076ab00000) Metaspace       used 67592K, capacity 69946K, committed 70144K, reserved 1112064K  class space    used 7943K, capacity 8414K, committed 8448K, reserved 1048576KEvent: 2901.121 GC heap afterHeap after GC invocations=9 (full 3): PSYoungGen      total 1296896K, used 14251K [0x000000076ab00000, 0x00000007c0000000, 0x00000007c0000000)  eden space 1195520K, 0% used [0x000000076ab00000,0x000000076ab00000,0x00000007b3a80000)  from space 101376K, 14% used [0x00000007b9d00000,0x00000007baaeaf98,0x00000007c0000000)  to   space 100864K, 0% used [0x00000007b3a80000,0x00000007b3a80000,0x00000007b9d00000) ParOldGen       total 2796544K, used 50150K [0x00000006c0000000, 0x000000076ab00000, 0x000000076ab00000)  object space 2796544K, 1% used [0x00000006c0000000,0x00000006c30f99c8,0x000000076ab00000) Metaspace       used 67592K, capacity 69946K, committed 70144K, reserved 1112064K  class space    used 7943K, capacity 8414K, committed 8448K, reserved 1048576K}Event: 10747.104 GC heap before{Heap before GC invocations=10 (full 3): PSYoungGen      total 1296896K, used 1209771K [0x000000076ab00000, 0x00000007c0000000, 0x00000007c0000000)  eden space 1195520K, 100% used [0x000000076ab00000,0x00000007b3a80000,0x00000007b3a80000)  from space 101376K, 14% used [0x00000007b9d00000,0x00000007baaeaf98,0x00000007c0000000)  to   space 100864K, 0% used [0x00000007b3a80000,0x00000007b3a80000,0x00000007b9d00000) ParOldGen       total 2796544K, used 50150K [0x00000006c0000000, 0x000000076ab00000, 0x000000076ab00000)  object space 2796544K, 1% used [0x00000006c0000000,0x00000006c30f99c8,0x000000076ab00000) Metaspace       used 68322K, capacity 70672K, committed 70912K, reserved 1112064K  class space    used 7963K, capacity 8423K, committed 8448K, reserved 1048576KEvent: 10747.119 GC heap afterHeap after GC invocations=10 (full 3): PSYoungGen      total 1296384K, used 10324K [0x000000076ab00000, 0x00000007c0000000, 0x00000007c0000000)  eden space 1195520K, 0% used [0x000000076ab00000,0x000000076ab00000,0x00000007b3a80000)  from space 100864K, 10% used [0x00000007b3a80000,0x00000007b44951c8,0x00000007b9d00000)  to   space 99840K, 0% used [0x00000007b9e80000,0x00000007b9e80000,0x00000007c0000000) ParOldGen       total 2796544K, used 50158K [0x00000006c0000000, 0x000000076ab00000, 0x000000076ab00000)  object space 2796544K, 1% used [0x00000006c0000000,0x00000006c30fb9c8,0x000000076ab00000) Metaspace       used 68322K, capacity 70672K, committed 70912K, reserved 1112064K  class space    used 7963K, capacity 8423K, committed 8448K, reserved 1048576K}Event: 18892.422 GC heap before{Heap before GC invocations=11 (full 3): PSYoungGen      total 1296384K, used 1205844K [0x000000076ab00000, 0x00000007c0000000, 0x00000007c0000000)  eden space 1195520K, 100% used [0x000000076ab00000,0x00000007b3a80000,0x00000007b3a80000)  from space 100864K, 10% used [0x00000007b3a80000,0x00000007b44951c8,0x00000007b9d00000)  to   space 99840K, 0% used [0x00000007b9e80000,0x00000007b9e80000,0x00000007c0000000) ParOldGen       total 2796544K, used 50158K [0x00000006c0000000, 0x000000076ab00000, 0x000000076ab00000)  object space 2796544K, 1% used [0x00000006c0000000,0x00000006c30fb9c8,0x000000076ab00000) Metaspace       used 68759K, capacity 71020K, committed 71168K, reserved 1112064K  class space    used 7970K, capacity 8429K, committed 8448K, reserved 1048576KEvent: 18892.439 GC heap afterHeap after GC invocations=11 (full 3): PSYoungGen      total 1299968K, used 9668K [0x000000076ab00000, 0x00000007c0000000, 0x00000007c0000000)  eden space 1200128K, 0% used [0x000000076ab00000,0x000000076ab00000,0x00000007b3f00000)  from space 99840K, 9% used [0x00000007b9e80000,0x00000007ba7f11f8,0x00000007c0000000)  to   space 97792K, 0% used [0x00000007b3f00000,0x00000007b3f00000,0x00000007b9e80000) ParOldGen       total 2796544K, used 50166K [0x00000006c0000000, 0x000000076ab00000, 0x000000076ab00000)  object space 2796544K, 1% used [0x00000006c0000000,0x00000006c30fd9c8,0x000000076ab00000) Metaspace       used 68759K, capacity 71020K, committed 71168K, reserved 1112064K  class space    used 7970K, capacity 8429K, committed 8448K, reserved 1048576K}Event: 27681.725 GC heap before{Heap before GC invocations=12 (full 3): PSYoungGen      total 1299968K, used 1209796K [0x000000076ab00000, 0x00000007c0000000, 0x00000007c0000000)  eden space 1200128K, 100% used [0x000000076ab00000,0x00000007b3f00000,0x00000007b3f00000)  from space 99840K, 9% used [0x00000007b9e80000,0x00000007ba7f11f8,0x00000007c0000000)  to   space 97792K, 0% used [0x00000007b3f00000,0x00000007b3f00000,0x00000007b9e80000) ParOldGen       total 2796544K, used 50166K [0x00000006c0000000, 0x000000076ab00000, 0x000000076ab00000)  object space 2796544K, 1% used [0x00000006c0000000,0x00000006c30fd9c8,0x000000076ab00000) Metaspace       used 68898K, capacity 71216K, committed 71424K, reserved 1112064K  class space    used 7971K, capacity 8429K, committed 8448K, reserved 1048576KEvent: 27681.740 GC heap afterHeap after GC invocations=12 (full 3): PSYoungGen      total 1297920K, used 9809K [0x000000076ab00000, 0x00000007c0000000, 0x00000007c0000000)  eden space 1200128K, 0% used [0x000000076ab00000,0x000000076ab00000,0x00000007b3f00000)  from space 97792K, 10% used [0x00000007b3f00000,0x00000007b48944d0,0x00000007b9e80000)  to   space 95232K, 0% used [0x00000007ba300000,0x00000007ba300000,0x00000007c0000000) ParOldGen       total 2796544K, used 50174K [0x00000006c0000000, 0x000000076ab00000, 0x000000076ab00000)  object space 2796544K, 1% used [0x00000006c0000000,0x00000006c30ff9c8,0x000000076ab00000) Metaspace       used 68898K, capacity 71216K, committed 71424K, reserved 1112064K  class space    used 7971K, capacity 8429K, committed 8448K, reserved 1048576K}Event: 36688.570 GC heap before{Heap before GC invocations=13 (full 3): PSYoungGen      total 1297920K, used 1209937K [0x000000076ab00000, 0x00000007c0000000, 0x00000007c0000000)  eden space 1200128K, 100% used [0x000000076ab00000,0x00000007b3f00000,0x00000007b3f00000)  from space 97792K, 10% used [0x00000007b3f00000,0x00000007b48944d0,0x00000007b9e80000)  to   space 95232K, 0% used [0x00000007ba300000,0x00000007ba300000,0x00000007c0000000) ParOldGen       total 2796544K, used 50174K [0x00000006c0000000, 0x000000076ab00000, 0x000000076ab00000)  object space 2796544K, 1% used [0x00000006c0000000,0x00000006c30ff9c8,0x000000076ab00000) Metaspace       used 69049K, capacity 71376K, committed 71552K, reserved 1112064K  class space    used 7972K, capacity 8461K, committed 8576K, reserved 1048576KEvent: 36688.584 GC heap afterHeap after GC invocations=13 (full 3): PSYoungGen      total 1305088K, used 9729K [0x000000076ab00000, 0x00000007c0000000, 0x00000007c0000000)  eden space 1209856K, 0% used [0x000000076ab00000,0x000000076ab00000,0x00000007b4880000)  from space 95232K, 10% used [0x00000007ba300000,0x00000007bac804c0,0x00000007c0000000)  to   space 92672K, 0% used [0x00000007b4880000,0x00000007b4880000,0x00000007ba300000) ParOldGen       total 2796544K, used 50182K [0x00000006c0000000, 0x000000076ab00000, 0x000000076ab00000)  object space 2796544K, 1% used [0x00000006c0000000,0x00000006c31019c8,0x000000076ab00000) Metaspace       used 69049K, capacity 71376K, committed 71552K, reserved 1112064K  class space    used 7972K, capacity 8461K, committed 8576K, reserved 1048576K}Deoptimization events (10 events):Event: 12281.446 Thread 0x00007f1f7cb62800 Uncommon trap: reason=unreached action=reinterpret pc=0x00007f2195f964f0 method=com.mysql.cj.result.DefaultColumnDefinition.findColumn(Ljava/lang/String;ZI)I @ 28Event: 13726.994 Thread 0x00007f1f74001000 Uncommon trap: reason=unreached action=reinterpret pc=0x00007f2196c65958 method=java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()Ljava/util/concurrent/RunnableScheduledFuture; @ 59Event: 14951.443 Thread 0x00007f1f7cb62800 Uncommon trap: reason=range_check action=make_not_entrant pc=0x00007f21966609f0 method=java.util.DualPivotQuicksort.sort([IIIZ)V @ 56Event: 14951.507 Thread 0x00007f1f543cf800 Uncommon trap: reason=unreached action=reinterpret pc=0x00007f2196a0fad0 method=io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue.poll()Ljava/lang/Object; @ 28Event: 14951.507 Thread 0x00007f1f543cf800 Uncommon trap: reason=unreached action=reinterpret pc=0x00007f2196196cf8 method=io.netty.buffer.PooledByteBuf.internalNioBuffer()Ljava/nio/ByteBuffer; @ 6Event: 16841.448 Thread 0x00007f1f543cf800 Uncommon trap: reason=unreached action=reinterpret pc=0x00007f21965db2ec method=java.util.concurrent.locks.AbstractQueuedSynchronizer.doReleaseShared()V @ 69Event: 43243.574 Thread 0x00007f1f54408800 Uncommon trap: reason=unreached action=reinterpret pc=0x00007f21959945c4 method=com.jd.ump.profiler.util.MessagesStore$WriteLog2File.run()V @ 22Event: 43243.574 Thread 0x00007f1f54410800 Uncommon trap: reason=unreached action=reinterpret pc=0x00007f21959945c4 method=com.jd.ump.profiler.util.MessagesStore$WriteLog2File.run()V @ 22Event: 43243.575 Thread 0x00007f1f5440f800 Uncommon trap: reason=unreached action=reinterpret pc=0x00007f21959945c4 method=com.jd.ump.profiler.util.MessagesStore$WriteLog2File.run()V @ 22Event: 43243.575 Thread 0x00007f1f54418000 Uncommon trap: reason=unreached action=reinterpret pc=0x00007f21959945c4 method=com.jd.ump.profiler.util.MessagesStore$WriteLog2File.run()V @ 22Internal exceptions (10 events):Event: 42812.465 Thread 0x00007f1f04001000 Exception  (0x000000079fe54f30) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u20/1074/hotspot/src/share/vm/prims/jni.cpp, line 716]Event: 42812.466 Thread 0x00007f1f04001000 Exception  (0x000000079fe57398) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u20/1074/hotspot/src/share/vm/prims/jni.cpp, line 742]Event: 42813.148 Thread 0x00007f1f04001000 Exception  (0x000000079fe578c0) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u20/1074/hotspot/src/share/vm/prims/jni.cpp, line 716]Event: 42813.148 Thread 0x00007f1f04001000 Exception  (0x000000079fe59d28) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u20/1074/hotspot/src/share/vm/prims/jni.cpp, line 742]Event: 42821.451 Thread 0x00007f1f7cb62800 Exception  (0x000000079f1a5d90) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u20/1074/hotspot/src/share/vm/prims/jni.cpp, line 716]

二、排查问题的详细过程

当JVM 的初始化完成后，类在调用执行过程中，执行引擎会把字节码转为机器码，然后在操作系统中才能执行。在字节码转换为机器码的过程中，虚拟机中还存在着一道编译，那就是即时编译。

最初，JVM 中的字节码是由解释器（ Interpreter ）完成编译的，当虚拟机发现某个方法或代码块的运行特别频繁的时候，就会把这些代码认定为热点代码。

为了提高热点代码的执行效率，在运行时，即时编译器（JIT，Just In Time）会把这些代码编译成与本地平台相关的机器码，并进行各层次的优化，然后保存到内存中。

C1 编译器

C1 编译器是一个简单快速的编译器，主要的关注点在于局部性的优化，适用于执行时间较短或对启动性能有要求的程序，也称为Client Compiler，例如，GUI 应用对界面启动速度就有一定要求。

C2 编译器

C2 编译器是为长期运行的服务器端应用程序做性能调优的编译器，适用于执行时间较长或对峰值性能有要求的程序，也称为Server Compiler，例如，服务器上长期运行的 Java 应用对稳定运行就有一定的要求。

触发标准

在 HotSpot 虚拟机中，热点探测是 JIT 的触发标准。

热点探测是基于计数器的热点探测，采用这种方法的虚拟机会为每个方法建立计数器统计方法的执行次数，如果执行次数超过一定的阈值就认为它是“热点方法” 。虚拟机为每个方法准备了两类计数器：方法调用计数器（Invocation Counter）和回边计数器（Back Edge Counter）。在确定虚拟机运行参数的前提下，这两个计数器都有一个确定的阈值，当计数器超过阈值溢出了，就会触发 JIT 编译。

方法调用计数器