{"id":228931,"date":"2022-05-09T15:15:04","date_gmt":"2022-05-09T22:15:04","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/java\/?p=228931"},"modified":"2022-09-19T13:25:28","modified_gmt":"2022-09-19T20:25:28","slug":"debugging-a-jvm-crash-for-linkedin-part-1","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/java\/debugging-a-jvm-crash-for-linkedin-part-1\/","title":{"rendered":"Debugging a JVM Crash for LinkedIn &#8211; Part 1"},"content":{"rendered":"<h2><a name=\"_Toc99970364\"><\/a>Introduction<\/h2>\n<p>As part of the work we do in the Java Engineering Group we provide support to many first- and third-party customers of the <a href=\"https:\/\/www.microsoft.com\/openjdk\">Microsoft Build of OpenJDK<\/a>, particularly those who run <a href=\"https:\/\/azure.microsoft.com\/en-ca\/develop\/java\/\">Java workloads on Azure<\/a>. One of our largest customers is LinkedIn.<\/p>\n<p>In the past couple of years our group has worked with LinkedIn to assist with their migration to Java 11 and their adoption of the Microsoft Build of OpenJDK.<\/p>\n<p>As is typical of support, we occasionally need to investigate crashes in the Java Virtual Machine (JVM). Recently, LinkedIn reported a JVM crash in one of their services powering Espresso, LinkedIn\u2019s in-house NoSQL database. \u00a0At the time, this service was running JDK 11.0.8. Even with the large number of services and JVM instances running Java, the crashes LinkedIn has seen are few and far between. This one is a very rare crash itself, happening only once per day across ~80 Linux hosts that run the service.<\/p>\n<p>In this series of blog posts, we will walk through the process we used to investigate the root cause of this JVM crash, armed only with the HotSpot error log file and the crash dump provided by LinkedIn. This series is broken down as follows:<\/p>\n<ul>\n<li><a href=\"#part1-analyzing-log\">Part 1: Analyzing the Log<\/a><\/li>\n<li><a href=\"https:\/\/devblogs.microsoft.com\/java\/debugging-a-jvm-crash-for-linkedin-part-2\/\">Part 2: Analyzing the Core Dump<\/a><\/li>\n<li><a href=\"https:\/\/devblogs.microsoft.com\/java\/debugging-a-jvm-crash-for-linkedin-part-3\/\">Part 3: Looking for a Fix<\/a><\/li>\n<\/ul>\n<h2><a name=\"part1-analyzing-log\"><\/a>Analyzing the Log<\/h2>\n<p>The first thing we like to do when we receive a crash report is to look at the HotSpot error log file. In this case, the header at the top of the file includes the following:<\/p>\n<pre class=\"prettyprint\"># A fatal error has been detected by the Java Runtime Environment:\r\n#\r\n#\u00a0 SIGSEGV (0xb) at pc=0x00007ffb860d705f, pid=19538, tid=7764\r\n#\r\n# JRE version: OpenJDK Runtime Environment Microsoft (11.0.8+10) (build 11.0.8+10-20200812)\r\n# Java VM: OpenJDK 64-Bit Server VM Microsoft (11.0.8+10-20200812, mixed mode, tiered, g1 gc, linux-amd64)\r\n# Problematic frame:\r\n# J 52708 c2 sun.security.jgss.krb5.Krb5Context.initSecContext(Ljava\/io\/InputStream;I)[B java.security.jgss@11.0.8 (698 bytes) @ 0x00007ffb860d705f [0x00007ffb860d47c0+0x000000000000289f]<\/pre>\n<p>This tells us that we&#8217;re dealing with a <span style=\"font-family: terminal, monaco, monospace;\">SIGSEGV<\/span>, a Linux Segmentation Fault (<a href=\"https:\/\/komodor.com\/learn\/sigsegv-segmentation-faults-signal-11-exit-code-139\/\">https:\/\/komodor.com\/learn\/sigsegv-segmentation-faults-signal-11-exit-code-139\/<\/a>), which is signal 11 (<span style=\"font-family: terminal, monaco, monospace;\">0xb<\/span>). This type of fault occurs when the OS detects that a process is trying to access memory that it is not allowed to. Often this means that there&#8217;s a bad pointer floating around that is either being used for reading or writing a memory address that is not mapped to the process. Alternatively, it could mean that the pointer is valid, but the process is attempting to read more memory starting at that address than it is allowed to.<\/p>\n<p>The crash is happening in the following method frame:<\/p>\n<pre class=\"prettyprint\"># J 52708 c2 sun.security.jgss.krb5.Krb5Context.initSecContext(Ljava\/io\/InputStream;I)[B java.security.jgss@11.0.8 (698 bytes) @ 0x00007ffb860d705f [0x00007ffb860d47c0+0x000000000000289f]<\/pre>\n<p>The thread that crashed is shown in the \u201c<span style=\"font-family: terminal, monaco, monospace;\">T H R E A D<\/span>\u201d section. Here&#8217;s an excerpt from the top of that section:<\/p>\n<pre class=\"prettyprint\">Current thread (0x00007ff7fbaa0000):\u00a0 JavaThread \"DataStreamer for file &lt;file omitted&gt;\" daemon [_thread_in_Java, id=7764, stack(0x00007ff75185a000,0x00007ff75195b000)]\r\n\r\nStack: [0x00007ff75185a000,0x00007ff75195b000],\u00a0 sp=0x00007ff751956420,\u00a0 free space=1009k\r\n\r\nNative frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)\r\n\r\nJ 52708 c2 sun.security.jgss.krb5.Krb5Context.initSecContext(Ljava\/io\/InputStream;I)[B java.security.jgss@11.0.8 (698 bytes) @ 0x00007ffb860d705f [0x00007ffb860d47c0+0x000000000000289f]\r\n\r\nJ 51577 c2 sun.security.jgss.GSSContextImpl.initSecContext(Ljava\/io\/InputStream;Ljava\/io\/OutputStream;)I java.security.jgss@11.0.8 (515 bytes) @ 0x00007ffb85f89a80 [0x00007ffb85f861e0+0x00000000000038a0]\r\n\r\nJ 68479 c2 com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge([B)[B jdk.security.jgss@11.0.8 (117 bytes) @ 0x00007ffb84b7c890 [0x00007ffb84b7c5a0+0x00000000000002f0]\r\n\r\n\u2026<\/pre>\n<p>We can see the problematic method at the top of that call stack. Note that it is prefixed with &#8220;<span style=\"font-family: terminal, monaco, monospace;\">J<\/span>&#8220;, so it appears that this method consists of compiled Java code, not interpreted code.<\/p>\n<p>One thing we sometimes check when a crash like this happens is to see if there was a recent compilation of the method, as compilation is triggered by profiling counters which can sometimes be buggy. To do this, we look at the &#8220;<span style=\"font-family: terminal, monaco, monospace;\">Compilation events<\/span>&#8221; section:<\/p>\n<pre class=\"prettyprint\">Compilation events (20 events):\r\n\r\nEvent: 966063.635 Thread 0x00007ffb948bf800 92423\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 4\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 com.mysql.jdbc.ConnectionPropertiesImpl::getUseTimezone (8 bytes)\r\n\r\nEvent: 966063.636 Thread 0x00007ffb948bf800 nmethod 92423 0x00007ffb83fe6490 code [0x00007ffb83fe6620, 0x00007ffb83fe66d8]\r\n\r\nEvent: 966102.461 Thread 0x00007ffb948bf800 92424\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 4\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 org.apache.kafka.common.Node::&lt;init&gt; (34 bytes)\r\n\r\n\u2026\r\n\r\nEvent: 966301.993 Thread 0x00007ffb948bf800 nmethod 92432 0x00007ffb848b8a10 code [0x00007ffb848b8be0, 0x00007ffb848b9418]<\/pre>\n<p>Nothing in that list of compilation events seems relevant here. Next, we check to see if there were any recent deoptimization events affecting this method, as it may indicate some interesting and relevant activity. For example, another crash we investigated recently was due to the fact that profiling counters were not being reset after a deoptimization event. We look in the \u201c<span style=\"font-family: terminal, monaco, monospace;\">Deoptimization events<\/span>&#8221; section:<\/p>\n<pre class=\"prettyprint\">Deoptimization events (20 events):\r\n\r\nEvent: 965907.465 Thread 0x00007ff8abe6d800 Uncommon trap: trap_request=0xffffff14 fr.pc=0x00007ffb84512b48 relative=0x00000000000006c8\r\n\r\nEvent: 965907.465 Thread 0x00007ff8abe6d800 Uncommon trap: reason=tenured action=make_not_entrant pc=0x00007ffb84512b48 method=sun.util.locale.provider.CalendarDataUtility$CalendarWeekParameterGetter.getObject(Ljava\/util\/spi\/CalendarDataProvider;Ljava\/util\/Locale;Ljava\/lang\/String;[Ljava\/lang\/Objec\r\n\r\nEvent: 965907.465 Thread 0x00007ff8abe6d800 DEOPT PACKING pc=0x00007ffb84512b48 sp=0x00007ff8899f2cc0\r\n\r\n\u2026\r\n\r\nEvent: 966291.973 Thread 0x00007ff8b492f000 DEOPT UNPACKING pc=0x00007ffb7c28f425 sp=0x00007ff86102c878 mode 2<\/pre>\n<p>Again, nothing there seems relevant.<\/p>\n<p>Our next step is to investigate the memory address that the application is trying to access. In the <span style=\"font-family: terminal, monaco, monospace;\">T H R E A D<\/span> section we&#8217;re provided with <span style=\"font-family: terminal, monaco, monospace;\">siginfo<\/span>, which gives us more information on the signal that was triggered:<\/p>\n<pre class=\"prettyprint\">siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007ffb7a000000<\/pre>\n<p>As we&#8217;ve seen, the type of signal was <span style=\"font-family: terminal, monaco, monospace;\">SIGSEGV<\/span>. The actual type of error was a <span style=\"font-family: terminal, monaco, monospace;\">SEGV_MAPERR<\/span>, which occurs when a process tries to access an address that&#8217;s not mapped into the address space of the process (see <a href=\"https:\/\/stackoverflow.com\/questions\/1000002\/what-is-segv-maperr\">https:\/\/stackoverflow.com\/questions\/1000002\/what-is-segv-maperr<\/a>). Which address are we talking about? The one indicated by <span style=\"font-family: terminal, monaco, monospace;\">si_addr<\/span>, which is <span style=\"font-family: terminal, monaco, monospace;\">0x00007ffb7a000000<\/span>.<\/p>\n<p>Let&#8217;s see if we can find out any more information about that address from the log file. We do a search and find it in several places! Our first hit, after the <span style=\"font-family: terminal, monaco, monospace;\">siginfo<\/span>, is here:<\/p>\n<pre class=\"prettyprint\">Heap:\r\n\r\n\u00a0garbage-first heap\u00a0\u00a0 total 10485760K, used 5056421K [0x00007ff8fa000000, 0x00007ffb7a000000)\r\n\r\n\u00a0 region size 8192K, 60 young (491520K), 21 survivors (172032K)\r\n\r\n\u00a0Metaspace\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 used 161880K, capacity 163125K, committed 164004K, reserved 165888K<\/pre>\n<p>OK, this seems interesting! It looks like we&#8217;re accessing the byte *right after the heap ends* &#8211; note that the range is <span style=\"font-family: terminal, monaco, monospace;\">[0x00007ff8fa000000, 0x00007ffb7a000000)<\/span>, i.e., inclusive on the lower end and exclusive on the higher end. In the log file, you can inspect the various Heap Regions this application is using:<\/p>\n<pre class=\"prettyprint\">Heap Regions: E=young(eden), S=young(survivor), O=old, HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free, A=archive, TAMS=top-at-mark-start (previous, next)\r\n\r\n|\u00a0\u00a0 0|0x00007ff8fa000000, 0x00007ff8fa800000, 0x00007ff8fa800000|100%|HS|\u00a0 |TAMS 0x00007ff8fa800000, 0x00007ff8fa000000| Complete\r\n\r\n|\u00a0\u00a0 1|0x00007ff8fa800000, 0x00007ff8fb000000, 0x00007ff8fb000000|100%|HC|\u00a0 |TAMS 0x00007ff8fb000000, 0x00007ff8fa800000| Complete\r\n\r\n\u2026\r\n\r\n|1278|0x00007ffb79000000, 0x00007ffb79800000, 0x00007ffb79800000|100%| E|CS|TAMS 0x00007ffb79000000, 0x00007ffb79000000| Complete\r\n\r\n|1279|0x00007ffb79800000, 0x00007ffb7a000000, 0x00007ffb7a000000|100%| E|CS|TAMS 0x00007ffb79800000, 0x00007ffb79800000| Complete<\/pre>\n<p>You can see here that the bad address is just off the end of the last Heap Region, and thus off the end of the heap altogether.<\/p>\n<p>We also see this address referenced several times in various GC Heap History events, e.g.:<\/p>\n<pre class=\"prettyprint\">GC Heap History (20 events):\r\n\r\nEvent: 966300.329 GC heap before\r\n\r\n{Heap before GC invocations=894510 (full 0):\r\n\r\n\u00a0garbage-first heap\u00a0\u00a0 total 10485760K, used 8059819K [0x00007ff8fa000000, 0x00007ffb7a000000)\r\n\r\n\u00a0 region size 8192K, 426 young (3489792K), 14 survivors (114688K)\r\n\r\n\u00a0Metaspace\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 used 161880K, capacity 163125K, committed 164004K, reserved 165888K\r\n\r\n}<\/pre>\n<p>All right, so it seems like we have a handle on where the bad memory access is. But why is this happening? This is starting to smell like a compiler bug, because it&#8217;s the compiler&#8217;s responsibility to make sure all memory accesses are legal. The Java programmers who wrote this application could not have made this mistake &#8211; they&#8217;re protected from having to worry about native memory management.<\/p>\n<p>Our next step is to inspect the JIT-compiled code and see if we can figure out what&#8217;s going on. The log file tells us that the crash happens at the instruction with a program counter of <span style=\"font-family: terminal, monaco, monospace;\">0x00007ffb860d705f<\/span>. Here are the instructions around that memory location:<\/p>\n<pre class=\"prettyprint\">Instructions: (pc=0x00007ffb860d705f)\r\n\r\n0x00007ffb860d6f5f:\u00a0\u00a0 24 a0 00 00 00 49 8b 4a 18 48 85 c9 75 21 be f6\r\n0x00007ffb860d6f6f:\u00a0\u00a0 ff ff ff 66 90 c5 f8 77 e8 04 84 1b f6 48 8b 8c\r\n0x00007ffb860d6f7f:\u00a0\u00a0 24 b8 00 00 00 eb 08 48 8b 8c 24 b8 00 00 00 8b\r\n0x00007ffb860d6f8f:\u00a0\u00a0 59 10 85 db 0f 86 32 01 00 00 8b d3 ff ca 3b d3\r\n0x00007ffb860d6f9f:\u00a0\u00a0 0f 83 43 2f 00 00 48 8b 7d 18 44 8b 57 10 45 85\r\n0x00007ffb860d6faf:\u00a0\u00a0 d2 0f 86 32 2f 00 00 41 3b d2 0f 83 29 2f 00 00\r\n0x00007ffb860d6fbf:\u00a0\u00a0 41 ba 08 00 00 00 41 3b da 44 8b c3 45 0f 4f c2\r\n0x00007ffb860d6fcf:\u00a0\u00a0 4c 63 8c 24 c8 00 00 00 46 0f be 5c 0f 18 46 0f\r\n0x00007ffb860d6fdf:\u00a0\u00a0 be 54 09 18 45 33 da 46 88 5c 09 18 44 8b 94 24\r\n0x00007ffb860d6fef:\u00a0\u00a0 c8 00 00 00 41 ff c2 44 89 94 24 c8 00 00 00 45\r\n0x00007ffb860d6fff:\u00a0\u00a0 3b d0 7c cc 44 8b c3 41 83 c0 f9 41 3b d0 41 bb\r\n0x00007ffb860d700f:\u00a0\u00a0 00 00 00 80 45 0f 4c c3 45 3b d0 7d 79 45 8b da\r\n0x00007ffb860d701f:\u00a0\u00a0 44 89 9c 24 c8 00 00 00 41 ba 40 1f 00 00 45 8b\r\n0x00007ffb860d702f:\u00a0\u00a0 d8 44 2b 9c 24 c8 00 00 00 45 3b da 45 0f 4f da\r\n0x00007ffb860d703f:\u00a0\u00a0 44 03 9c 24 c8 00 00 00 66 0f 1f 84 00 00 00 00\r\n0x00007ffb860d704f:\u00a0\u00a0 00 4c 63 8c 24 c8 00 00 00 c4 a1 7a 7e 44 09 18\r\n0x00007ffb860d705f:\u00a0\u00a0 c4 a1 79 ef 44 0f 18 c4 a1 79 d6 44 09 18 44 8b\r\n0x00007ffb860d706f:\u00a0\u00a0 8c 24 c8 00 00 00 41 83 c1 08 44 89 8c 24 c8 00\r\n0x00007ffb860d707f:\u00a0\u00a0 00 00 45 3b cb 7c ca 4d 8b 9f 08 01 00 00 41 85\r\n0x00007ffb860d708f:\u00a0\u00a0 03 45 3b c8 7c 98 44 8b 9c 24 c8 00 00 00 44 3b\r\n0x00007ffb860d709f:\u00a0\u00a0 db 7d 29 44 8b 8c 24 c8 00 00 00 66 90 4d 63 c1\r\n0x00007ffb860d70af:\u00a0\u00a0 46 0f be 54 07 18 46 0f be 5c 01 18 45 33 d3 46\r\n0x00007ffb860d70bf:\u00a0\u00a0 88 54 01 18 41 ff c1 44 3b cb 7c e1 44 8b 55 10\r\n0x00007ffb860d70cf:\u00a0\u00a0 4c 8b 84 24 a0 00 00 00 45 89 50 10 41 83 fa 03\r\n0x00007ffb860d70df:\u00a0\u00a0 0f 84 43 3c 00 00 41 83 fa 01 0f 84 8d 3c 00 00\r\n0x00007ffb860d70ef:\u00a0\u00a0 41 83 fa 10 0f 84 d7 3c 00 00 49 8b 87 18 01 00\r\n0x00007ffb860d70ff:\u00a0\u00a0 00 4c 8b d0 49 83 c2 18 48 bd 60 79 70 72 f8 7f\r\n0x00007ffb860d710f:\u00a0\u00a0 00 00 4d 3b 97 28 01 00 00 0f 83 96 1c 00 00 4d\r\n0x00007ffb860d711f:\u00a0\u00a0 89 97 18 01 00 00 41 0f 0d 8a c0 00 00 00 4c 8b\r\n0x00007ffb860d712f:\u00a0\u00a0 95 b8 00 00 00 4c 89 10 48 89 68 08 48 c7 40 10\r\n0x00007ffb860d713f:\u00a0\u00a0 00 00 00 00 48 89 84 24 a8 00 00 00 48 8b 00 4c\r\n0x00007ffb860d714f:\u00a0\u00a0 8b d0 49 83 e2 07 49 83 fa 05 0f 85 a2 28 00 00<\/pre>\n<p>Great! So, what do we do next? Well, if you can read hex assembly directly, you can probably take it from here yourself. If not, you\u2019ll need to stick around for the next step &#8211; disassembly!<\/p>\n<p>To make this hex dump of instructions more readable, we turned to the Online Disassembler, or ODA (<a href=\"https:\/\/onlinedisassembler.com\/odaweb\/\">https:\/\/onlinedisassembler.com\/odaweb\/<\/a>). (Unfortunately, since the initial preparation of this blog post, ODA seems to have been removed. Hopefully it shows up in a different location soon. The instructions that follow refer to the original version of ODA.).<\/p>\n<p>ODA is a very handy tool that allows you to paste in some hex instructions and see the disassembly. You can change the architecture, syntax style, endianness, etc.<\/p>\n<p>Setting the architecture is very important \u2013 if that\u2019s not set correctly you will very likely see incorrect disassembly. In this case, for a 64-bit JVM, we set the Arch to i386:x86-64:intel and leave the Syntax Style as DEFAULT (which is intel-mnemonic).<\/p>\n<p>In the text box at the bottom left, we insert the instructions from <span style=\"font-family: terminal, monaco, monospace;\">0x00007ffb860d705f<\/span>:<\/p>\n<pre class=\"prettyprint\">c4 a1 79 ef 44 0f 18 c4 a1 79 d6 44 09 18 44 8b<\/pre>\n<p>In the disassembly pane, we see that this corresponds to the following:<\/p>\n<pre class=\"prettyprint\">c4a179ef440f18\u00a0\u00a0\u00a0\u00a0\u00a0 vpxor xmm0, xmm0, XMMWORD PTR [rdi+r9*1+0x18]<\/pre>\n<p>All right, so it looks like we have identified the offending instruction!<\/p>\n<h2><a name=\"_Toc99970364\"><\/a>What&#8217;s Next?<\/h2>\n<p>For fun, we\u2019ll see if we can locate this instruction in the core dump in the next post! Stay tuned for Part 2 of this series.<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction As part of the work we do in the Java Engineering Group we provide support to many first- and third-party customers of the Microsoft Build of OpenJDK, particularly those who run Java workloads on Azure. One of our largest customers is LinkedIn. In the past couple of years our group has worked with LinkedIn [&hellip;]<\/p>\n","protected":false},"author":83088,"featured_media":227205,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1,249],"tags":[25,248,741,319,28],"class_list":["post-228931","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-java","category-openjdk","tag-debugging","tag-java","tag-java-11","tag-openjdk","tag-testing"],"acf":[],"blog_post_summary":"<p>Introduction As part of the work we do in the Java Engineering Group we provide support to many first- and third-party customers of the Microsoft Build of OpenJDK, particularly those who run Java workloads on Azure. One of our largest customers is LinkedIn. In the past couple of years our group has worked with LinkedIn [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/java\/wp-json\/wp\/v2\/posts\/228931","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/java\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/java\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/java\/wp-json\/wp\/v2\/users\/83088"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/java\/wp-json\/wp\/v2\/comments?post=228931"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/java\/wp-json\/wp\/v2\/posts\/228931\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/java\/wp-json\/wp\/v2\/media\/227205"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/java\/wp-json\/wp\/v2\/media?parent=228931"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/java\/wp-json\/wp\/v2\/categories?post=228931"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/java\/wp-json\/wp\/v2\/tags?post=228931"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}