Tencent Kona JDK11无暂停内存管理-ZGC生产实践

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"腾讯大数据JVM团队基于OpenJDK11自研的Tencent Kona JDK11，目前已将ZGC特性孵化成熟，性能优于OpenJDK所提供的版本，使Java能够轻松构建响应时间在ms级别的强实时性在线服务，极大提高研发和运维效率，目前在腾讯内部多业务场景生产落地，实现业务延迟SLA 提升2-3个数量级。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"随着2021年4月30日Tencent Kona JDK 11.0.10-GA 正式对外发布，生产可用的ZGC也正式对外开源。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"经过二十多年的发展，Java语言的生态已经庞大无比，应用范围覆盖了从嵌入式设备到大型数据中心等场景，形成了各色各样的业务形态。不同的业务关注点不尽相同，如部分离线应用关注整个系统的吞吐率，而不太关注单个进程的停顿时间；另外一些应用则对于GC停顿的时间有严格的要求，比如以下业务形态："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"在线服务交互。和用户交互的UI线程，需要按照特定的频率进行屏幕的刷新，比方说普通60HZ刷新率的屏幕，在播放动画时，需要在1s内刷新60次，才能保持屏幕画面的连续性，即需要在15ms内完成一次刷新，如果此时由于GC停顿导致UI线程挂起，则会导致画面出现撕裂感，最终导致用户体验的下降。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"竞价广告。在竞价广告应用场景下，如Real Time Bidding中的广告竞价投放，一个广告栏请求播放广告时，不同的广告主则要根据当前的用户价值进行交易竞价，通常来说需要在规定的时间内（一般为100ms到200ms）达成交易，否则就会错失一次广告曝光机会，此时GC停顿的控制就显得非常重要。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"量化交易。在交易机会出现时，交易机构需要以最快的速度达成交易，对于实时性的要求就更为严苛。如果出现由于GC停顿造成的延迟，轻则错失交易机会，重则导致亏损。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为了满足不同的业务需求，Java的GC算法也在不停迭代，对于特定的应用，选择其最适合的GC算法，才能更高效的帮助业务实现其业务目标。对于这些延迟敏感的应用来说，GC停顿已经成为阻碍Java广泛应用的一大顽疾，需要更适合的GC算法以满足这些业务的需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"近些年来，服务器的性能越来越强劲，各种应用可使用的堆内存也越来越大，常见的堆大小从10G到百G级别，部分机型甚至可以到达TB级别，在这类大堆应用上，传统的GC，如CMS、G1的停顿时间也跟随着堆大小的增长而同步增加，即堆大小指数级增长时，停顿时间也会指数级增长。特别是当触发Full GC时，停顿可达分钟级别。当业务应用需要提供高服务级别协议（Service Level Agreement，SLA），例如99.99%的响应时间不能超过100ms，此时CMS、G1等就无法满足业务的需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为满足当前应用对于超低停顿、高SLA的需求，并应对大堆和超大堆带来的挑战，伴随着2018年发布的JDK 11，A Scalable Low-Latency Garbage Collector - ZGC应运而生。腾讯大数据JVM团队的Tencent Kona JDK作为OpenJDK Hotspot VM下游分支，也致力于在LTS的JDK11版本上提供Production Ready的ZGC功能，满足公司内部客户的需求。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1.GC停顿"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.1 停顿之由来"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Hotspot虚拟机上，GC算法均是基于Mark Sweep或者Mark Compact实现的，也可以称为Tracing GC。对于这类标记扫描的GC算法来说，需要通过Mark找到所有活着的对象，然后将死对象清除，或者把所有的活对象拷贝到另外一块区域，以达到清理内存的目的。因此，所有的Tracing GC均需要以下三个步骤，其整体过程如下面动图所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/3e\/a8\/3e4e01b36aaaa696fa141330106489a8.gif","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"找出所有的GC Roots集合：这是Tracing GC算法的起点，GC Roots主要为运行时的关键数据结构中存放的指向堆对象的指针，如线程栈上的堆对象指针等。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"标记过程：从GC Roots开始遍历整个对象图，找出所有存货的对象。而剩余未被标记的对象则为死对象。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"清理过程：将死掉的对象清理掉，释放其占用的内存。当然清理时可以直接释放对象内存，也可以将所有的活对象移动到一块连续的区域里，并将原来的内存空间释放。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到，上面三个步骤均需要一致的信息，如GC Roots需要是完整的，不能在扫描时被随意修改；标记过程，需要扫描到所有活着的对象，其他线程不能随意修改对象图；清理过程如果需要搬移对象，则需要更新所有指向该对象的地方，如对象A指向B，B被搬动之后，A中的指针需要同步更新。因此要求在这三步中，采取同步措施，而最简单的同步措施就是暂停所有的Java线程，即Stop-The-World（STW），在STW期间，GC线程就可以安全的访问各种运行时数据、对象图、更新对象指针等。如果需要降低STW的时间，则需要将GC的不同阶段的任务移出STW，和Java线程进行并发执行，这个时候就需要算法和数据结构方面的更改，以满足GC线程和Java线程对当前GC的一致性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不同的GC算法实现，其STW阶段需要完成的任务大相径庭，造成不同GC算法STW时长的不同。所有的任务均在STW阶段完成时，这类GC就不需要和应用线程抢占CPU，从应用整体来看，最终的吞吐率是比较高的，如Parallel GC；而当STW阶段的任务减少时，则需要在并发阶段增加相应的任务——即部分GC任务需要和业务线程一起运行，相互抢占CPU，这类GC根据不同的任务划分，最终在吞吐率和停顿之间达到一个平衡，如CMS和G1致力于以较小的吞吐率损失换取较小的停顿和较高的响应；ZGC和ShenandoahGC则关注极致停顿，尽一切可能减少STW的工作量，从而实现ms级别的停顿时间。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.2 业界解决方案"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为了与Apple的iOS系统进行竞争，Google主导的Android系统需要解决的一大问题就是显示卡顿问题，通过对GC算法的不断演进，实现基于Baker Barrier的Concurrent Copy GC算法，停顿时间控制在几个ms级别，小于15ms的刷新约束，补全了Java在嵌入式设备中的短板，使得松散的Java生态能够实现和严格控制的Apple生态一样流畅的系统。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在ZGC和ShenandoahGC出现之前，Hotspot JVM的GC停顿很难平稳的控制在百毫秒以下，极大地阻碍了OpenJDK在金融行业等延迟敏感的场景的应用。但是作为OpenJDK的下游分支，Azul的Zing虚拟机凭借其闭源的C4 GC，实现了近乎“无停顿”的低延迟，在前十几年中大放异彩，频繁出现在各类交易系统中。云上的各类系统和普通的桌面应用，则面临着无低延迟GC可用的窘境。为满足业务需求，一般会采用C++等Native语言重写一些重要模块，如腾讯的广告系统采用C++实现，或者购置Zing等实现低延迟GC的VM，抑或在CMS和G1上八仙过海、各显神通，利用经验调整参数来满足基本的业务需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hotspot上的停顿，导致备选方案面临C++的高开发门槛、经费成本和调参经验等各种问题，以及大堆管理问题，使得停顿成为Java开发者心中的梦魇，阻碍了Java在低延迟需求业务的应用，此乃当前Hotspot JVM上的停顿之殇、开发者之痛。为了解决这一问题，ZGC采用了和Azul的Zing VM相似的GC算法，从JDK11开始开源孵化，直到JDK15补全各类功能，成为真正可以商用的正式版本，保证了Java停顿时间不会随着堆大小和业务规模的增加而增长。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"JDK11在2018年下半年发布，是最新的Long-Term Support版本，而后续LTS版本为JDK17，将于2021年下半年发布，JDK12到JDK16属于中间过渡开发版本，不会像JDK11和JDK17一样提供持续的更新和修复。ZGC在OpenJDK11上属于Experimental实验特性，无法满足业务的商用需求，腾讯JVM团队为了提前满足业务的需求，在Tencent Kona JDK11持续的更新和修复的同时，将ZGC的各项功能补全，并进行了长期的验证落地，使得Tencent Kona JDK11上的ZGC能够达到商用水平，让停顿敏感的业务应用在JDK11这个LTS版本上实现超低GC延迟。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2. ZGC简介"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ZGC是由提议JEP 333（https:\/\/openjdk.java.net\/jeps\/333）引入Hotspot Runtime，其目标是为了彻底解决GC停顿带来的延迟问题，总的设计目标为："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每次GC总的停顿时间控制在10ms以下"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相对于G1，应用的吞吐率降低不超过15%"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持大堆和特大堆（8MB~16TB），并且停顿时间不随堆大小的增长而增长"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/9e\/b1\/9e28c60d3bcc77f92439839fcba018b1.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由设计目标可知，ZGC主要是为现在及未来大堆的管理问题服务，致力于以最小的性能损失换取最大的停顿优势。从Oracle发布的测试数据来看(参见[1])，上图中SPECjbb2015上ZGC的吞吐率（max-JOPS）和Parallel GC、G1GC相差无几，而体现停顿影响的指标critical-JOPS则提升了20%+；在暂停时间上，ZGC则不会超过10ms，而Parallel GC和G1GC则高达100ms+，如下图所示。因此，ZGC尤其适合对延迟比较敏感的大堆任务。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/7b\/be\/7b4d4bd405c1b5e5eb4f0e8b1f7d15be.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.1 ZGC算法实现"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为减少停顿，需要减少STW中执行的任务，ZGC主要在以下三个方面进行推进："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"GC Roots的扫描。将能够移除到STW以外的Roots扫描外移到并发阶段，Roots扫描的并发外移需要对Roots的数据结构进行改造，以支持GC线程和Java线程同时操作。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"Runtime数据结构的处理。在Runtime中维护了很多张表来记录Meta（class、method、jit code等），并且Java存在一类特殊的弱引用，即java.lang.ref.Reference及其子类，需要额外处理。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"对象移动的并发化改造。为了能够让移动对象和Java线程同时运行，需要增加Read barrier来保证每次对象field读取的正确性。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ZGC在GC算法的处理逻辑上有很大的变更，但是在整体逻辑上，与其前辈GC算法一样，都是Mark&Compact形式。具体实现上，ZGC下面六个阶段通过来实现低延迟的GC算法，如下图所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/0e\/95\/0e47dc93713f2d08061a8a3300b38e95.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一个阶段是Pause Mark Start：主要做一些全局状态的设置和全局数据结构的初始化这类轻量化的任务，标明后续并发阶段需要做GC的Concurrent Mark。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二个阶段是Concurrent Mark & Remap：将耗时占比最大的GC Roots进行并发化改造，支持并发Roots标记。从GC Roots进行对象图的并发标记。上一轮GC的指针更新（Remap）通过Piggyback，放到当前阶段执行，从而减少对对象图的遍历。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三个阶段是Pause Mark End：这一阶段做Concurrent Mark的同步，结束并发标记阶段，同时设置部分全局变量。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第四个阶段是Concurrent Prepare：这一阶段主要做java.lang.ref.Reference等弱引用的处理，并选择出需要Compact的ZGC Region。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第五个阶段是Pause Relocate Start：这一阶段和第三阶段比较类似，主要是全局同步，设置全局变量，并指示Relocate阶段的开始。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第六个阶段是Concurrent Relocate：并发的搬移对象。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相对于其他GC，ZGC需要三个STW阶段来做全局的同步，但每个STW中的任务都很明确，需要完成的任务的时间和CPU的处理速度正相关，因此可以做到ms级别的停顿。相对于G1GC，ZGC的难点在于如何进行GC Roots的并发化改造和对象搬移的并发化改造。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/14\/13\/14e23cf8ee1b1557fd929d953e8c7213.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对于对象搬迁的并发化改造，ZGC则采用Colored Pointer来实现轻量级的Read Barrier，如上图所示。对于64bit的系统，高位bit中拿出4个bit来指示不同的处理状态，两个Mark位表明该对象指针是否已经被标记，采用两个Mark bit可以在前后不同的GC时使用不同的Mark bit；Remapped位表示当前对象指针是否已经调整为搬移之后的对象指针；Finalizable位主要是为Finalizable对象服务，用来表示该对象指针是否仅经Finalize对象标记，主要供Mark阶段和弱引用处理阶段使用。通过Colored指针，不同的GC阶段，当前Runtime的正确的指针颜色仅为一种颜色(Marked或者Remapped)，就可以通过下图所示，测试对象指针是否为bad color即可，在x86上最终实现为一条test指令和一条jne跳转指令。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/27\/b4\/270d1f29503f3b6f548a9e9354e755b4.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Colored Pointer导致不同的时期，对象的指针的高位是不同，如下图中的对象指针0x0000000012345678，在程序运行过程中，可能以下面三种状态被Java线程感知到：Remapped状态、Mark1状态、Mark0状态。为了使得这几种不同的状态（不同值的指针），指向同一份对象，ZGC完全利用了操作系统的虚拟地址和物理地址转换，使得这三种状态的虚拟地址指针指向同一份物理地址，因此ZGC的Java堆需要在虚拟地址中占用三份地址。ZGC通过内存文件来占用实际的物理内存，然后将这个内存文件映射到Remapped、Mark0和Mark1指向的虚拟地址。可以看出，虽然表面上ZGC的Java Heap占用了三份虚拟地址，但是实际的物理地址只有一份。这也是linux的命令top或者ps看到启用ZGC的Java进程RSS内存膨胀三倍的原因，但开启ZGC之后观察到的RSS消耗并非实际物理内存消耗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/6b\/05\/6bdb1c8b75aa57045168367116c89905.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.2 ZGC算法的开销"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ZGC对于业务线程的影响主要集中在以下五个方面："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"Read barrier的开销。在Java程序中，对象指针的读取次数要远超于对象指针的写入次数，Read Barrier的插入点要远多于Write Barrier的插入点，因此ZGC的Read Barrier会对程序的性能产生较大的负面影响。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"JIT方法的entry barrier开销。如果JIT之后的代码包含了已经死掉的java对象，那么该方法就应该丢弃掉，因此JIT的代码需要在进入时利用一个entry barrier来保证自身和其包含的meta信息的有效性。ZGC对每个JIT代码都生成nmethod entry barrier，会对JIT方法产生轻微的性能损失。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"Frame barrier开销。为并发进行Java栈帧的扫描，降低Stack Roots扫描对STW时间的影响，当前Hotspot采用StackWaterMark来进行并发扫栈。同时为了降低业务线程扫描栈帧的工作量，Hotspot中采用单个栈帧扫描的方式，即在回栈时如果超过当前stack water mark，就会陷入stack mark barrier，修复caller的java对象指针。参见https:\/\/openjdk.java.net\/jeps\/376"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"其他Runtime改造产生的锁结构带来的开销。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"ZGC中大部分的GC工作放在并发阶段，因此并发阶段GC线程和Java业务线程抢占CPU，导致的对业务线程的抢占开销。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看出ZGC为了降低STW造成的停顿影响，采取的措施是极致的并发化改造，也就是以轻微的性能损失换取最低的停顿影响。当前最新的ZGC实现停顿已经达到ms级别，低于Linux内核的背景噪声，即调度开销和系统调用开销，也有可能造成10ms级别的影响，可以说ZGC使得Java不能服务实时业务的古板印象得到彻底的颠覆。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3. ZGC使用与调参"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.1 ZGC典型应用场景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此之蜜糖、彼之砒霜，不同的GC算法都有其长短处，ZGC出现的最大优势是能够在保证停顿时间控制10ms以下，但为了实现这种高SLA的停顿时间，其代价是性能的损失和内存消耗。从前面介绍可以看出，为了降低STW中的工作，很多GC任务做了并发化改造，而并发化改造的代价则散乱在各种运行细节中，通过整个OpenJDK社区的持续投入，当前ZGC在性能损失场景中的性能下降已经控制在很小的范围内。对于性能来说，不同的配置对性能的影响是不同的，如充足的内存下即大堆场景，ZGC在各类Benchmark中能够超过G1大约5%到20%，而在小堆情况下，则要低于G1大约10%；不同的配置对于应用的影响不尽相同，开发者需要根据使用场景来合理判断。当前ZGC不支持压缩指针和分代GC，其内存占用相对于G1来说要稍大，在小堆情况下较为明显，而在大堆情况下，这些多占用的内存则显得不那么突出。因此，以下两类应用强烈建议使用ZGC来提升业务体验："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"超大堆应用。超大堆（百G以上）下，CMS或者G1如果发生Full GC，停顿会在分钟级别，可能会造成业务的终端，强烈推荐使用ZGC。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"高SLA需求的应用。如对响应时间有P999时限要求的实时和软实时应用，此类应用无论堆大小，均推荐采用低停顿的ZGC。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.2 ZGC参数设置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ZGC之美不仅在于其超低的STW停顿，也在于其参数的简单，绝大部分生产场景都可以自适应。当然，极端情况下，还是有可能需要对ZGC个别参数做个调整，大致可以分为三类："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"堆大小：Xmx。ZGC能够通过极致的低延迟满足业务高标准SLA的服务准入条件，但是与所有编程语言的concurrent GC类似，延迟是以内存空间作为trade-off的。当分配速率过高，超过回收速率，造成堆内存不够时，会触发Allocation Stall，这类Stall会减缓当前的用户线程。因此，当我们在GC日志中看到Allocation Stall，通常可以认为堆空间偏小或者concurrent gc threads数偏小。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"GC触发时机：ZAllocationSpikeTolerance, ZCollectionInterval。ZAllocationSpikeTolerance用来估算当前的堆内存分配速率，在当前剩余的堆内存下，ZAllocationSpikeTolerance越大，估算的达到OOM的时间越快，ZGC就会更早地进行触发GC。ZCollectionInterval用来指定GC发生的间隔，以秒为单位触发GC。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"GC线程：ParallelGCThreads， ConcGCThreads。ParallelGCThreads是设置STW任务的GC线程数目，默认为CPU个数的60%；ConcGCThreads是并发阶段GC线程的数目，默认为CPU个数的12.5%。增加GC线程数目，可以加快GC完成任务，减少各个阶段的时间，但也会增加CPU的抢占开销，可根据生产情况调整。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由上可以看出ZGC需要调整的参数十分简单，通常设置Xmx即可满足业务的需求，大大减轻Java开发者的负担。当前Tencent Kona JDK11上开启ZGC的参数为：“-XX:+UnlockExperimentalVMOptions -XX:+UseZGC”。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4. ZGC生产注意事项"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.1 RSS内存异常现象"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由前面ZGC原理可知，ZGC采用多映射multi-mapping的方法实现了三份虚拟内存指向同一份物理内存。而Linux统计进程RSS内存占用的算法是比较脆弱的，这种多映射的方式并没有考虑完整，因此根据当前Linux采用大页和小页时，其统计的开启ZGC的Java进程的内存表现是不同的。在内核使用小页的Linux版本上，这种三映射的同一块物理内存会被linux的RSS占用算法统计3次，因此通常可以看到使用ZGC的Java进程的RSS内存膨胀了三倍左右，但是实际占用只有统计数据的三分之一，会对运维或者其他业务造成一定的困扰。而在内核使用大页的Linux版本上，这部分三映射的物理内存则会统计到hugetlbfs inode上，而不是当前Java进程上。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.2 共享内存调整"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ZGC需要在share memory中建立一个内存文件来作为实际物理内存占用，因此当要使用的Java的堆大小大于\/dev\/shm的大小时，需要对\/dev\/shm的大小进行调整。通常来说，命令如下（下面是将\/dev\/shm调整为64G）："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"vi \/etc\/fstab"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"tmpfs \/dev\/shm tmpfs defaults,size=65536M 0 0"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先修改fstab中shm配置的大小，size的值根据需求进行修改，然后再进行shm的mount和umount。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"umount \/dev\/shm"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"mount \/dev\/shm"}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.3 mmap节点上限调整"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ZGC的堆申请和传统的GC有所不同，需要占用的memory mapping数目更多，即每个ZPage需要mmap映射三次，这样系统中仅Java Heap所占用的mmap个数为(Xmx \/ zpage_size) * 3，默认情况下zpage_size的大小为2M。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为了给JNI等native模块中的mmap映射数目留出空间，内存映射的数目应该调整为(Xmx \/ zpage_size) * 3 * 1.2。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"默认的系统memory mapping数目由文件 \/proc\/sys\/vm\/max_map_count 指定，通常数目为65536，当给JVM配置一个很大的堆时，需要调整该文件的配置，使得其大于(Xmx \/ zpage_size) * 3 * 1.2。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"5. ZGC在腾讯的大规模生产实践"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前Tencent Kona JDK11的ZGC已经在腾讯广告大数据场景，腾讯云VPC、WAF等业务场景上长期稳定运行，并协助业务取得了优异的性能表现。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.1 支持广告海量数据查询"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hermes是腾讯自研的大数据实时分析系统，具有海量数据实时接入和存储、低延迟查询分析的特性，支持千级维度的多维分析，以及日增量万亿的海量日志接入和查询分析。在广告业务实时OLAP分析业务中，要求Hermes系统上99%的SQL查询端到端延迟不超过3s，而采用默认配置的G1 GC时仅98.1%的SQL查询端到端延迟不超过3s。通过切换Kona 11 ZGC，SQL端到端延迟满足率上升为99.5%，同时单个SQL查询中GC造成的延迟不超过20ms。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.2 超大堆支持"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"腾讯VPC团队为腾讯云提供网络控制服务，该服务主要存储用于云上资源通信的网络配置等信息，并提供配置信息的查询、修改和下发等服务。业务要求在512G内存的机器上尽可能多的存储配置信息（最大支持800M的监听数目），并且保证压力场景下读写延迟不超过1s。采用G1GC则会高频率出现大量的延迟超过10s，通过腾讯大数据JVM团队配合切换ZGC，并解决ZGC在业务遇到的Mark Stack Overflow、进入Safepoint缓慢、ZGC Mark假死等问题，最终使得业务能够在压力场景下，将预期的业务存储容量提升12.5%，同时读写延迟不超过50ms。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.3 助力提升SLA"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"腾讯WAF团队采用Java来快速实现产品功能迭代及上线，其中，旁路安全服务是一个基于Netty框架的Http服务，此服务对时延要求很严格，需要达到99.99%端到端请求时延小于80ms的SLA目标。因此GC的STW对此服务有一定负面影响，需要进一步降低“世界暂停”时间。在使用ZGC之前，WAF团队使用的是G1GC，前期花费了大量时间对G1 GC进行选项调试，并进行了代码层面的修改。但由于G1GC本身的不足，仍然存在请求抖动延迟，无法达到既定的SLA目标。在腾讯大数据JVM团队的配合下，切换ZGC之后，该业务的P9999请求延迟稳定小于80ms，为用户提供了更快速、稳定的服务。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"6.社区回馈"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"腾讯大数据JVM团队在支持业务切换ZGC的同时，将遇到的相关问题和修复积极向社区报告和回馈，争做OpenJDK社区好公民。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"6.1 ZGC与VectorAPI联合使用问题"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在广告某业务中，上线VectorAPI以提升机器学习效率，同时打开ZGC以满足服务SLA，在业务运行过程中出现结果非预期现象，并且社区存在类似的错误报告。通过对JIT生产的汇编代码进行分析，发现存在load barrier缺失现象。经分析，在C2的Vector优化阶段需要对Vector节点进行Unbox操作，该优化阶段会新生成一个load节点，并且又未考虑GC Barrier对load操作的影响，即ZGC需要对load操作生成load barrier，从而导致这个新生成的load节点缺少load barrier信息，最终未能生成相关barrier指令。通过对Vector优化阶段新生产的load操作增加GC barrier处理流程，使得该阶段能够生成带gc barrier信息的load节点，从而在不同GC选项下均能生成对应正确的barrier代码。该修复贡献给社区后以P2优先级合入JDK16。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/0a\/52\/0a076b65e0a12800663e779e9bbyyb52.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"6.2 ZGC Mark Stack Overflow问题"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"腾讯云某业务对ZGC进行灰度时，出现JVM进程崩溃现象，相关日志显示是由于ZGC标记阶段使用的Mark Stack超过预先设定的8G内存导致，而通常情况下Mark Stack的使用不会超过32M。经过业务全力配合，拿到一个可复现的场景进行深入分析，发现在这种场景下ZGC对于Mark Stack的使用存在两个缺陷：第一，大量的Mark Stack未使用满就塞入全局队列，造成单个Stack内存碎片问题；第二，大量的对象被多次压入Mark Stack中，造成Stack中的Entry重复率很高，浪费Stack空间。腾讯大数据JVM团队作出快速修复，验证Mark Stack Overflow问题可解后，将该问题和修复报告OpenJDK社区，社区基于提交的patch给出了更为优雅的修复方案，并将腾讯大数据JVM团队作为co-author联合提交代码入库，目前两个问题的修复均已入库JDK17。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/d8\/09\/d8efc6d05e7a03aaf5f272d888493b09.gif","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"6.3 ZGC Mark假死问题"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在分析ZGC在业务上的表现时，需要打开gc debug log选项，在启动后不久，出现进程卡死现象。分析发现绝大部分gc worker线程处在“Concurrent Mark Try Terminate”阶段，并且在等待“Concurrent Mark Idle”阶段的log文件读写锁，另外一个gc worker线程处于写log过程中，由此可以分析出由于gc worker线程均在抢log文件锁，导致gc worker线程最终形成一种动态死锁状态，即所有的gc worker线程均处于“等锁->拿锁->释放锁”这种无限循环中。这种假死现象是由于ZGC的Concurrent Mark退出机制导致的，在退出机制中所有的gc worker线程会等待1ms来进行状态同步，而等待结束后会进行相关log打印，这个打印需要前述log文件锁，从而导致动态假死现象出现。腾讯大数据JVM团队快速修复该问题，并提交社区，目前该贡献已合入JDK17中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/71\/fc\/71dd191e5a5c5f25bde34c9e10e75ffc.gif","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"7. Tencent Kona JDK 开源"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"腾讯大数据JVM团队的Tencent Kona JDK最新版本已经正式对外发布，大家可以使用Tencent Kona JDK 11.0.10-GA享受到ZGC带来的好处。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Tencent Kona JDK 8.0.5-GA 同步更新 OpenJDK 8u282ga"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/github.com\/Tencent\/TencentKona-8","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/github.com\/Tencent\/TencentKona-8"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Tencent Kona JDK 11.0.10-GA 同步更新 OpenJDK 11.0.10-ga"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/github.com\/Tencent\/TencentKona-11","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/github.com\/Tencent\/TencentKona-11"}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"8. 参考文献"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"[1] The Design of ZGC. "},{"type":"link","attrs":{"href":"http:\/\/cr.openjdk.java.net\/~pliden\/slides\/ZGC-PLMeetup-2019.pdf","title":null,"type":null},"content":[{"type":"text","text":"http:\/\/cr.openjdk.java.net\/~pliden\/slides\/ZGC-PLMeetup-2019.pdf"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"[2] How JavaScript works: memory management + how to handle 4 common memory leaks. "},{"type":"link","attrs":{"href":"https:\/\/blog.sessionstack.com\/how-javascript-works-memory-management-how-to-handle-4-common-memory-leaks-3f28b94cfbec","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/blog.sessionstack.com\/how-javascript-works-memory-management-how-to-handle-4-common-memory-leaks-3f28b94cfbec"}]}]}]}