{"title":"DevLog of zzoon","link":[{"@attributes":{"href":"https:\/\/blogs.igalia.com\/zzoon\/atom.xml","rel":"self"}},{"@attributes":{"href":"https:\/\/blogs.igalia.com\/zzoon\/"}}],"updated":"2022-03-03T10:10:19+09:00","id":{},"author":{"name":"Hyunjun Ko","email":{}},"entry":[{"title":"A Complement story in the Fosdem 2022 Talk","link":{"@attributes":{"href":"\/graphics\/mesa\/2022\/02\/21\/complement-story\/"}},"updated":"2022-02-21T00:00:00+09:00","id":"https:\/\/blogs.igalia.com\/zzoon\/graphics\/mesa\/2022\/02\/21\/complement-story","content":"<p>At 6th February, fortunately I had a chance to have a lightening talk at <a href=\"https:\/\/fosdem.org\/2022\/schedule\/room\/dgraphics\/\">the graphics devroom<\/a> of <a href=\"https:\/\/fosdem.org\/2022\/\">fosdem 2022<\/a>. At the time I presented a summary of the status of turnip driver development in mesa and I\u2019d like to write some more things that I couldn\u2019t say at the talk as a complement to the talk.<\/p>\n\n<h1 id=\"history-of-the-development\">History of the development.<\/h1>\n\n<p>I write here again, Turnip is the code name of Qualcomm Adreno GPU\u2019s open-source vulkan driver. I dived into this GPU at 2018 and participated in developing the Freedreno OpenGL driver, that is founded and leaded by <a href=\"https:\/\/gitlab.freedesktop.org\/robclark\">Rob Clark<\/a>. It was new for me to develop a user-space graphics driver at that time and it was a good chance to learn and experience of how GPUs work, how to develop a graphics driver and how to communicate with <a href=\"https:\/\/gitlab.freedesktop.org\/mesa\/mesa\">Mesa<\/a> community. Due to the contributions to the freedreno driver at that time, I could be a committer at the Mesa community. And at the beginning of 2020, <a href=\"www.igalia.com\">Igalia<\/a> including me started working on the opensource vulkan driver, that is \u201cTurnip\u201d driver. Actually the turnip development started already at 2018 by Google people and it\u2019s definately lucky for us to get a chance to participate in the turnip development.<\/p>\n\n<p>As I said in the talk it was immature though it works fine at the beginning thus lots of extensions and features are implemented since 2020.\nJoanathan Marek and Connor Abbott played an important role at this time(and still) and they implemented very basic and essential features from the scratch (and copied from freedreno if necessary). So Igalia could get involved in the progress of the implementation of extensions and features on top of what they have done. At 2021, we could speed-up the progress based on accumulated experiences and knowledges, especially Danylo\u2019s join was a great addition to the project and he\u2019s been always doing a great job since then. I won\u2019t write details about this in this post but you can refer to <a href=\"https:\/\/blogs.igalia.com\/siglesias\/\">Samuel<\/a> and <a href=\"https:\/\/blogs.igalia.com\/dpiliaiev\/\">Danylo<\/a>\u2019s blog, who are my great colleagues at Igalia. It\u2019s a pity that there\u2019s no posts about my works though. Hopefully I could write some more posts about it in the near future. (I got a list to write already! XD)<\/p>\n\n<h1 id=\"one-of-main-goals-playing-games\">One of main goals: Playing games!<\/h1>\n\n<p>Especially I\u2019d like to highlight this in this post, probably this is why I write this post.<\/p>\n\n<p>I presented things like below at the \u201cWhat happend at 2021\u201d<\/p>\n\n<ul>\n  <li>Make it run for windows games with dxvk\/vkd3d on linux\/arm\n    <ul>\n      <li>with x86 emulators (Fex, Box86)<\/li>\n      <li>Some window games started running!<\/li>\n    <\/ul>\n  <\/li>\n<\/ul>\n\n<p>And what about this at the \u201cWhat\u2019ll happen in 2022\u201d?<\/p>\n\n<ul>\n  <li>Focusing on real world use cases.\n    <ul>\n      <li>Still not enough games running on arm.<\/li>\n      <li>Trying to run more window games via wine(proton)<\/li>\n    <\/ul>\n  <\/li>\n<\/ul>\n\n<p>Cool! Isn\u2019t it?<\/p>\n\n<p>Actually I realized many people out there are very interested in this, that is to \u201cmake games run on linux\/arm\u201d, of course, including windows\/x86 games. And I know people want to test by themselves with their devices if possible but there are some issues to do it.<\/p>\n\n<ul>\n  <li>First, qualcomm devices are very expensive comparing to RasberryPi or other arm devices.<\/li>\n  <li>Second, the setup is so tricky to make and there\u2019s no good document for the setup.<\/li>\n  <li>Third, the setup is unstable to run every game so far even if you\u2019ve completed the setup.<\/li>\n<\/ul>\n\n<p>Honestly I\u2019m not going to write details like how to do in this post. Instead, I\u2019d like to show some efforts to do it as an example of what we\u2019ve tried. I said in the talk that we\u2019ve been trying this complicated setup because we want to test turnip with real cases but there are not enough native games running on linux\/arm yet. Here are cases that we\u2019ve been trying.<\/p>\n\n<h2 id=\"the-first-case-is-here--httpsgitlabfreedesktoporgmesamesa-issues4738\">The first case is here : <a href=\"https:\/\/gitlab.freedesktop.org\/mesa\/mesa\/-\/issues\/4738\">https:\/\/gitlab.freedesktop.org\/mesa\/mesa\/-\/issues\/4738<\/a>.<\/h2>\n\n<p>One day at 2021, one issue has been raised in the community. As you see at the system information, someone was trying to run window games with turnip on a qualcomm device.<\/p>\n\n<p>There were 2 kind of setups on Android:\nOne, most complicated setup, is using virgl-vtest to get OpenGL calls from windows games as you see below. It seems, at that time, the ExaGear couldn\u2019t access to GPU directly then they used virgl-vtest-server to access to GPU with turnip, which causes performance degradation.<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>-------------------------------------------------------\nWindow Games\n-------------------------------------------------------\nExaGear:    (virgl) | &lt;----&gt; |  virgl-vtest\n                    |        | ------------------------\n Wine on            |        |  turnip\/zink\n  x86 Ubuntu 18.04  |        | ------------------------\n                    |        |  Ubuntu 20.04 on chroot\n-------------------------------------------------------\n                  Android 10\n-------------------------------------------------------\n\n<\/code><\/pre><\/div><\/div>\n<p>So there was another try to not use virgl-vtest-server and seems it was successful for a few games with another setup like this:<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>---------------------------------\nWindow Games\n---------------------------------\nWINE(proton using dxvk)\n---------------------------------\nX86 emulator(Box86)\n---------------------------------\nturnip\/zink\n---------------------------------\nUbuntu 20.04 on Termux proot\n---------------------------------\nAndroid 10\n---------------------------------\n<\/code><\/pre><\/div><\/div>\n\n<p>Which looks simpler but still complicated. As you can see in the issue, Danylo managed to set up the system and found root causes and fixed all of it.<\/p>\n\n<p>Now it looks someone has succeeded in accessing to GPU directly in this emulaotr so we wouldn\u2019t need to use virgl-vtest any more since we got new issue for the setup including ExaGear. :)\nSee <a href=\"https:\/\/gitlab.freedesktop.org\/mesa\/mesa\/-\/issues\/6024\">https:\/\/gitlab.freedesktop.org\/mesa\/mesa\/-\/issues\/6024<\/a> in detail.<\/p>\n\n<h2 id=\"the-second-case-is-here-httpsgitlabfreedesktoporgmesamesa-issues5723\">The second case is here: <a href=\"https:\/\/gitlab.freedesktop.org\/mesa\/mesa\/-\/issues\/5723\">https:\/\/gitlab.freedesktop.org\/mesa\/mesa\/-\/issues\/5723<\/a><\/h2>\n\n<p>That is similar to the second setup of the first case, which is like<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>---------------------------------\nWindow Games\n---------------------------------\nWINE\n---------------------------------\nX86 emulator(Box86)\n---------------------------------\nturnip\/zink\n---------------------------------\nLinux\n---------------------------------\n\n<\/code><\/pre><\/div><\/div>\n\n<p>Yeah, starts using linux directly instead of Termux for android, thus it got simple a bit more.\nYou can see the instructions and some trouble shootings here for this setup: <a href=\"https:\/\/github.com\/Heasterian\/Box86-64-on-SD845-mobian\">https:\/\/github.com\/Heasterian\/Box86-64-on-SD845-mobian<\/a><\/p>\n\n<h2 id=\"the-third-case-is-to-run-android-games-with-just-replacing-the-proprietary-driver-with-turnip\">The third case is to run android games with just replacing the proprietary driver with turnip<\/h2>\n<p>Which is most simple. You can see instructions how to build the turnip driver for Android <a href=\"https:\/\/docs.mesa3d.org\/android.html\">here<\/a>\nBut note that there seems to be not enough vulkan games on Android yet, which means that we need to try with x86 emulators to test real games.\nAlso You can see some efforts on this setup at Danylo\u2019s blog as the following:<\/p>\n\n<p><a href=\"https:\/\/blogs.igalia.com\/dpiliaiev\/turnips-in-the-wild-part-1\">https:\/\/blogs.igalia.com\/dpiliaiev\/turnips-in-the-wild-part-1\/<\/a>\n<a href=\"https:\/\/blogs.igalia.com\/dpiliaiev\/turnips-in-the-wild-part-2\">https:\/\/blogs.igalia.com\/dpiliaiev\/turnips-in-the-wild-part-2<\/a>\n<a href=\"https:\/\/blogs.igalia.com\/dpiliaiev\/gfxreconstruct-test-mobile-gpus\/\">https:\/\/blogs.igalia.com\/dpiliaiev\/gfxreconstruct-test-mobile-gpus\/<\/a><\/p>\n\n<p>Additionally there was also a talk about emulation on arm for playing games: <a href=\"https:\/\/fosdem.org\/2022\/schedule\/event\/fex\">FEX-Emu: Fast(-er) x86 emulation for AArch64<\/a>\nI think you can get more informations to run x86 games on arm\/linux and I believe you can have fun too with this talk.<\/p>\n\n<p>Ok, until here, I did tell a bit more that I missed at the talk. Hopefully it could be useful to someone interested.\nAlso I\u2019ll give it a try to write far more details, how to setup and run games step by step for example, in the near future.<\/p>\n\n<p>Thanks for reading!<\/p>\n\n"},{"title":"Improvements for GStreamer Intel-MSDK plugins","link":{"@attributes":{"href":"\/gstreamer\/2018\/04\/04\/improvements-of-gst\/"}},"updated":"2018-04-04T00:00:00+09:00","id":"https:\/\/blogs.igalia.com\/zzoon\/gstreamer\/2018\/04\/04\/improvements-of-gst","content":"<p>Last November I had a chance to dive into Intel Media SDK plugins in gst-plugins-bad. It was very good chance for me to learn how gstreamer elements are handling special memory with its own infrastructures like GstBufferPool and GstMemory. In this post I\u2019m going to talk about the improvements of GStreamer Intel MSDK plugins in release 1.14, which is what I\u2019ve been through since last November.<\/p>\n\n<p>First of all, for those not familiar with Media SDK I\u2019m going to explain what Media SDK is briefly.\nMedia SDK(Aka. MSDK) is the cross-platform API to access Intel\u2019s hardware accelerated video encoder and decoder functions on Windows and Linux. You can get more information about MSDK <a href=\"https:\/\/software.intel.com\/en-us\/media-sdk\">here<\/a> and <a href=\"https:\/\/software.intel.com\/en-us\/blogs\/2017\/03\/31\/media-sdk-open-source-launch\">here<\/a>.<\/p>\n\n<p>But on Linux so far, it\u2019s hard to set up environment to make MSDK driver working.\nIf you want to set up development environment for MSDK and see what\u2019s working on linux, you should follow the steps described in <a href=\"https:\/\/software.intel.com\/en-us\/articles\/how-to-setup-media-server-studio-on-secondary-os-of-linux\">this page<\/a>. But I would recommend you refer to the <a href=\"https:\/\/blogs.igalia.com\/vjaquez\/2017\/11\/24\/intel-mediasdk-on-debian-testing\">Victor\u2019s post<\/a>, which is very-well explained for this a little bit annoying stuff.<\/p>\n\n<p>Additionally the driver in linux supports only Skylake as far as I know, which is very disappointing for users of other chipsets. I(and you probably) hope we can work on it without any specific patch\/driver (since it is open source!) and any dependency of chipset. As far as I know, Intel has been considering this issue to be solved, so we\u2019ll see what\u2019s going to happen in the future.<\/p>\n\n<p>Back to gstreamer, gstreamer plugins using MSDK landed in 2016.\nAt that time they were working fine with basic features for playback and encoding but there were several things to be improved, especially for performance.<\/p>\n\n<p>Eventually, in the middle of last March, GStreamer 1.14 has been released including improvements of MSDK plugins, which is what I want to talk in this post.<\/p>\n\n<p>So let me begin now.<\/p>\n\n<h2 id=\"suuports-bufferpool-and-starts-using-video-memory\">Suuports bufferpool and starts using video memory.<\/h2>\n\n<p>This is a key feature that improves the preformance.<\/p>\n\n<p>In MSDK, there are two types of memory supported in the driver.\nOne is \u201cSystem Memory\u201d and another is \u201cVideo Memory\u201d. (There is one more type of memory, which is called \u201cOpaque Memory\u201d but won\u2019t talk about it in this post)<\/p>\n\n<p>System memory is a memory allocated on user space, which is normal.\nSystem memory is being used in the plugins already, which is quite simple to use but not recommended since the performance is not good enough.<\/p>\n\n<p>Video memory is a memory used by hardware acceleration device, also known as GPU, to hold frame and other types of video data.<\/p>\n\n<p>For applications to use video memory, something specific on its platform should be implemented. On linux for example, we can use VA-API to handle video memory through MSDK.\nAnd that\u2019s included to the 1.14 release, which means we still need to implement something specific on Windows like this way.<\/p>\n\n<p>To implement using video memory, I needed to implement GstMSDK(System\/Video)Memory to generalize how to access and map the memory in the way of GStreamer. And GstMSDKBufferPool can allocate this type of memory and can be proposed to upstream and can be used in each MSDK element itself. There were lots of problems and argues during this work since the design of MSDK APIs and GStreamer infrastructure don\u2019t match perfectly.<\/p>\n\n<p>You can see discussion and patches here in <a href=\"https:\/\/bugzilla.gnome.org\/show_bug.cgi?id=790752\">this issue<\/a>.<\/p>\n\n<p>In addition, if using video memory on linux, we can use DMABuf by acquiring fd handle via VA-API at allocation time. Recently this has been done only for DMABuf export in <a href=\"https:\/\/bugzilla.gnome.org\/show_bug.cgi?id=793707\">this issue<\/a> though it\u2019s not included in 1.14 release.<\/p>\n\n<h2 id=\"sharing-contextsession\">Sharing context\/session<\/h2>\n<p>For resource utilization, there needs to share MSDK session with each MSDK plugin in a pipeline.\nA MSDK session maintains context for the use of any of decode,encode and convert(VPP) functions. Yes it\u2019s just like a handle of the driver. One session can run exactly one of decode, encode and convert(VPP).<\/p>\n\n<p>So let\u2019s think about an usual transcoding pipeline. It should be like this, src - decoder - converter - encoder - sink. In this case we should use same MSDK session to decode, convert and encode. Otherwise we should copy the data from upstream to work with different session because sessions cannot share data, which should get much worse.<\/p>\n\n<p>Also there\u2019s one more thing. MSDK supports joining session. If application wants(or has) to use multiple sessions, it can join sessions to share data and we need to support it in the level of GStreamer as well.<\/p>\n\n<p>All of these can be achieved by GstContext which provides a way of sharing not only between elements. You can see the patch in <a href=\"https:\/\/bugzilla.gnome.org\/show_bug.cgi?id=790752\">this issue<\/a>, same as MSDK Bufferpool issue.<\/p>\n\n<h2 id=\"adds-vp8-and-mpeg2-decoder\">Adds vp8 and mpeg2 decoder.<\/h2>\n<p><a href=\"https:\/\/github.com\/sreerenjb\">Sree<\/a> has added mpeg2\/vc1 decoder and I have added vp8 decoder.<\/p>\n\n<h2 id=\"supports-a-number-of-algorithms-and-tuning-options-in-encoder\">Supports a number of algorithms and tuning options in encoder<\/h2>\n<p>Encoders are exposing a number of rate control algorithms now and more encoder tuning options like trellis-quantiztion (h264), slice size control (h264), B-pyramid prediction(h264), MB-level bitrate control, frame partitioning and adaptive I\/B frame insertion were added. The encoder now also handles force-key-unit events and can insert frame-packing SEIs for side-by-side and top-bottom stereoscopic 3D video.<\/p>\n\n<p>All of this has been done by <a href=\"https:\/\/github.com\/sreerenjb\">Sree<\/a> and you can see the details in <a href=\"https:\/\/bugzilla.gnome.org\/show_bug.cgi?id=791637\">this issue<\/a><\/p>\n\n<h2 id=\"capability-of-encoders-sinkpad-is-improved-by-using-vpp\">Capability of encoder\u2019s sinkpad is improved by using VPP.<\/h2>\n<p>MSDK encoders had accepted only NV12 raw data since MSDK encoder supports only NV12 format. But other formats can be handled too if we convert them to NV12 by using VPP, which is also supported in the MSDK driver.\nThis has been done by <a href=\"https:\/\/coaxion.net\/\">slomo<\/a> and I fixed a bug related to it. See <a href=\"https:\/\/bugzilla.gnome.org\/show_bug.cgi?id=789847\">this bug<\/a> for more details.<\/p>\n\n<p>You can find all of patches for MSDK plugins <a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gst-plugins-bad\/log\/sys\/msdk\">here<\/a>.<\/p>\n\n<p>As I said in the top of this post, all MSDK stuffs should be opened first and should support some of major Intel chipsets at least even if not all.\nBut now, the one thing that I can say is GStreamer MSDK plugins are being improved continuously.\nWe can see what\u2019s happening in the near future.<\/p>\n\n<p>Finally I want to say that <a href=\"https:\/\/github.com\/sreerenjb\">Sree<\/a> and <a href=\"https:\/\/blogs.igalia.com\/vjaquez\/\">Victor<\/a> helped me a lot as a reviewer and an adviser with this work. I really appreciate it.<\/p>\n\n<p>Thanks for reading!<\/p>\n\n"},{"title":"Support GstContext for VA-API elements","link":{"@attributes":{"href":"\/gstreamer\/gstreamer-vaapi\/2017\/08\/21\/support-gstcontext\/"}},"updated":"2017-08-21T00:00:00+09:00","id":"https:\/\/blogs.igalia.com\/zzoon\/gstreamer\/gstreamer-vaapi\/2017\/08\/21\/support-gstcontext","content":"<p>Since I started working on gstreamer-vaapi, one of what\u2019s disappointing me is that vaapisink is not so popular even though it should be the best choice on vaapi installed machine.\nThere are some reasonable causes and one of the reasons is probably it doesn\u2019t provide a convinient way to be integrated for application developers.<\/p>\n\n<p>Until now, we provided a way to set X11 window handle by gst_video_overlay_set_window_handle, which is to tell the overlay to display video output to a specific window.\nBut this is not enough since VA and X11 Display handle is managed internally inside gstreamer-vaapi elements, which means that users can\u2019t handle them by themselves.<\/p>\n\n<p>In short, there was no way to share display handle created by application.  Also we have some additional problems due to this issue as the following.<\/p>\n<ul>\n  <li>If users want to handle multiple display seperatedly, it can\u2019t be possible. <a href=\"https:\/\/bugzilla.gnome.org\/show_bug.cgi?id=754820\">bug 754820<\/a><\/li>\n  <li>If users run multiple decoding pipelines with vaapisink, performance is down critically since there\u2019s some locks in each vaapisink with same VADisplay. <a href=\"https:\/\/bugzilla.gnome.org\/show_bug.cgi?id=747946\">bug 747946<\/a><\/li>\n<\/ul>\n\n<p>Recently we have merged a series of patches to provide a way to set external VA Display and X11 Display from application via GstContext. GstContext provides a way of sharing not only between elements but also with the application using queries and messages. (For more details, see <a href=\"https:\/\/developer.gnome.org\/gstreamer\/stable\/gstreamer-GstContext.html\">https:\/\/developer.gnome.org\/gstreamer\/stable\/gstreamer-GstContext.html<\/a>)<\/p>\n\n<p>By these patches, application can set its own VA Display and X11 Display to VA-API elements as the following:<\/p>\n<ul>\n  <li>Create VADisplay instance by vaGetDisplay, it doesn\u2019t need to be initialized at startup and terminated at endup.<\/li>\n  <li>Call gst_element_set_context with the context to which each display instance is set.<\/li>\n<\/ul>\n\n<p>Example: sharing an VADisplay and X11 display with the bus callback, this is almost same as other examples using GstContext.<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-c\" data-lang=\"c\"><span class=\"k\">static<\/span> <span class=\"n\">GstBusSyncReply<\/span>\n<span class=\"nf\">bus_sync_handler<\/span> <span class=\"p\">(<\/span><span class=\"n\">GstBus<\/span> <span class=\"o\">*<\/span> <span class=\"n\">bus<\/span><span class=\"p\">,<\/span> <span class=\"n\">GstMessage<\/span> <span class=\"o\">*<\/span> <span class=\"n\">msg<\/span><span class=\"p\">,<\/span> <span class=\"n\">gpointer<\/span> <span class=\"n\">data<\/span><span class=\"p\">)<\/span>\n<span class=\"p\">{<\/span>\n  <span class=\"k\">switch<\/span> <span class=\"p\">(<\/span><span class=\"n\">GST_MESSAGE_TYPE<\/span> <span class=\"p\">(<\/span><span class=\"n\">msg<\/span><span class=\"p\">))<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">case<\/span> <span class=\"n\">GST_MESSAGE_NEED_CONTEXT<\/span><span class=\"p\">:{<\/span>\n      <span class=\"k\">const<\/span> <span class=\"n\">gchar<\/span> <span class=\"o\">*<\/span><span class=\"n\">context_type<\/span><span class=\"p\">;<\/span>\n      <span class=\"n\">GstContext<\/span> <span class=\"o\">*<\/span><span class=\"n\">context<\/span><span class=\"p\">;<\/span>\n      <span class=\"n\">GstStructure<\/span> <span class=\"o\">*<\/span><span class=\"n\">s<\/span><span class=\"p\">;<\/span>\n      <span class=\"n\">VADisplay<\/span> <span class=\"n\">va_display<\/span><span class=\"p\">;<\/span>\n      <span class=\"n\">Display<\/span> <span class=\"o\">*<\/span><span class=\"n\">x11_display<\/span><span class=\"p\">;<\/span>\n\n      <span class=\"n\">gst_message_parse_context_type<\/span> <span class=\"p\">(<\/span><span class=\"n\">msg<\/span><span class=\"p\">,<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">context_type<\/span><span class=\"p\">);<\/span>\n      <span class=\"n\">gst_println<\/span> <span class=\"p\">(<\/span><span class=\"s\">\"Got need context %s from %s\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">context_type<\/span><span class=\"p\">,<\/span>\n          <span class=\"n\">GST_MESSAGE_SRC_NAME<\/span> <span class=\"p\">(<\/span><span class=\"n\">msg<\/span><span class=\"p\">));<\/span>\n\n      <span class=\"k\">if<\/span> <span class=\"p\">(<\/span><span class=\"n\">g_strcmp0<\/span> <span class=\"p\">(<\/span><span class=\"n\">context_type<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"gst.vaapi.app.Display\"<\/span><span class=\"p\">)<\/span> <span class=\"o\">!=<\/span> <span class=\"mi\">0<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">break<\/span><span class=\"p\">;<\/span>\n\n      <span class=\"n\">x11_display<\/span> <span class=\"o\">=<\/span> <span class=\"cm\">\/* Get X11 Display somehow *\/<\/span>\n      <span class=\"n\">va_display<\/span> <span class=\"o\">=<\/span> <span class=\"n\">vaGetDisplay<\/span> <span class=\"p\">(<\/span><span class=\"n\">x11_display<\/span><span class=\"p\">);<\/span>\n\n      <span class=\"n\">context<\/span> <span class=\"o\">=<\/span> <span class=\"n\">gst_context_new<\/span> <span class=\"p\">(<\/span><span class=\"s\">\"gst.vaapi.app.Display\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">TRUE<\/span><span class=\"p\">);<\/span>\n      <span class=\"n\">s<\/span> <span class=\"o\">=<\/span> <span class=\"n\">gst_context_writable_structure<\/span> <span class=\"p\">(<\/span><span class=\"n\">context<\/span><span class=\"p\">);<\/span>\n      <span class=\"n\">gst_structure_set<\/span> <span class=\"p\">(<\/span><span class=\"n\">s<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"va-display\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">G_TYPE_POINTER<\/span><span class=\"p\">,<\/span> <span class=\"n\">va_display<\/span><span class=\"p\">,<\/span> <span class=\"nb\">NULL<\/span><span class=\"p\">);<\/span>\n      <span class=\"n\">gst_structure_set<\/span> <span class=\"p\">(<\/span><span class=\"n\">s<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"x11-display\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">G_TYPE_POINTER<\/span><span class=\"p\">,<\/span> <span class=\"n\">x11_display<\/span><span class=\"p\">,<\/span> <span class=\"nb\">NULL<\/span><span class=\"p\">);<\/span>\n\n      <span class=\"n\">gst_element_set_context<\/span> <span class=\"p\">(<\/span><span class=\"n\">GST_ELEMENT<\/span> <span class=\"p\">(<\/span><span class=\"n\">GST_MESSAGE_SRC<\/span> <span class=\"p\">(<\/span><span class=\"n\">msg<\/span><span class=\"p\">)),<\/span> <span class=\"n\">context<\/span><span class=\"p\">);<\/span>\n      <span class=\"n\">gst_context_unref<\/span> <span class=\"p\">(<\/span><span class=\"n\">context<\/span><span class=\"p\">);<\/span>\n      <span class=\"k\">break<\/span><span class=\"p\">;<\/span>\n    <span class=\"p\">}<\/span>   \n    <span class=\"nl\">default:<\/span>\n      <span class=\"k\">break<\/span><span class=\"p\">;<\/span>\n  <span class=\"p\">}<\/span>\n\n  <span class=\"k\">return<\/span> <span class=\"n\">GST_BUS_PASS<\/span><span class=\"p\">;<\/span>\n<span class=\"p\">}<\/span><\/code><\/pre><\/figure>\n\n<p>Also you can find the entire example code <a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gstreamer-vaapi\/tree\/tests\/elements\/test-vaapicontext.c\">here<\/a>.<\/p>\n\n<p>Furthermore, we know we need to support Wayland for this feature. See <a href=\"https:\/\/bugzilla.gnome.org\/show_bug.cgi?id=705821\">bug 705821<\/a>.\nThere\u2019s already some pending patches but they need to be rebased and modified based on the current way. I\u2019ll be working on this in the near future.<\/p>\n\n<p>We really want to test this feature more especially in practical cases until next release.\nI\u2019d appreciate if someone reports any bug or issue and I promise I\u2019d focus on it precisely.<\/p>\n\n<p>Thanks for reading!<\/p>\n"},{"title":"100 commits in GStreamer","link":{"@attributes":{"href":"\/gstreamer\/2017\/07\/06\/100-commits\/"}},"updated":"2017-07-06T15:05:06+09:00","id":"https:\/\/blogs.igalia.com\/zzoon\/gstreamer\/2017\/07\/06\/100-commits","content":"<p>It\u2019s been 3 years since I\u2019ve started working on GStreamer, meanwhile I contributed over 100 commits fortunately!<\/p>\n\n<p>Let\u2019s look at my commits in each project in GStreamer.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gstreamer\/log\/?qt=author&amp;q=zzoon\">gstreamer: 7 commits<\/a><\/li>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gst-plugins-base\/log\/?qt=author&amp;q=zzoon\">gst-plugins-base: 5 commits<\/a><\/li>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gst-plugins-good\/log\/?qt=author&amp;q=zzoon\">gst-plugins-good: 13 commits<\/a><\/li>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gst-plugins-bad\/log\/?qt=author&amp;q=zzoon\">gst-plugins-bad: 18 commits<\/a><\/li>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gst-rtsp-server\/log\/?qt=author&amp;q=zzoon\">gst-rtsp-server: 10 commits<\/a><\/li>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gstreamer-vaapi\/log\/?qt=author&amp;q=zzoon\">gstreamer-vaapi: 76 commits<\/a><\/li>\n<\/ul>\n\n<p>Now that I write this article, I have made 128 commits.<\/p>\n\n<p>In Samsung Electronics, which was my previous company that I had been working for, I had a chance to work on gstreamer, which is main multimedia framework on Tizen. Since then, I realized that there are lots of opportunity in open source world and I started enjoying contribution to this project.<\/p>\n\n<p>This is my first commit:<\/p>\n<ul>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gst-rtsp-server\/commit\/?id=a8e604355cc18445f1b9154abc9efb6d990c32bf\">media: correct misspelled words in description<\/a><\/li>\n<\/ul>\n\n<p>Yes. It\u2019s just a fix typo. This landed in just five minutes after I proposed and I realized that maintainers are looking at all issues in bugzilla. To be honest, I doubted it a bit. :P<\/p>\n\n<p>Looking at other commits that I was really happy with.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gst-rtsp-server\/commit\/?id=de590b4b2a8aa1c9de9bf263cd63e23aca4ca0ea\">rtsp-stream: fix to get valid each stream data for request-aux-sender signal<\/a><\/li>\n<\/ul>\n\n<p>While I was working on gst-rtsp-server at that time, I found it\u2019s not working properly for RTP retransmission on the server. I reported the issue and discussion went very positive, then my proposed patches landed finally thanks to Sebastian.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gst-plugins-good\/commit\/?id=2b1f52755df222d56f1e9c4a52826539e8f9c062\">rtpmanager: add new on-new-sender-ssrc, on-sender-ssrc-active signals<\/a><\/li>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gst-rtsp-server\/commit\/?id=a51337974cdb1aa56db5e4ef87a1bb52a7423979\">stream: listen to sender ssrc signals<\/a><\/li>\n<\/ul>\n\n<p>This was enhancement of infrastructure for RTSP\/RTP in GStreamer, which is providing a way to report stats for sender\/receiver.<\/p>\n\n<p>Then I contributed huge patches of creating new APIs for transformation between SDP and GstCaps including removing duplicated codes. Thanks, Sebastian again.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gst-plugins-base\/commit\/?id=682b52365254d28215c2f071d083a4b6e0b21ae5\">sdp: add helper fuctions from\/to sdp from\/to caps<\/a><\/li>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gst-plugins-good\/commit\/?id=330003951392356b934625b6fb9498398edbd713\">rtspsrc: replace duplicated codes to call new base sdp apis<\/a><\/li>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gst-plugins-bad\/commit\/?id=1e5eb72532613efb15bb687df68b6fb5048896c0\">sdpdemux: replace duplicated codes to call new base sdp apis<\/a><\/li>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gst-rtsp-server\/commit\/?id=924f914172795fa714725b890f773d8b2ecffda2\">sdp: replace duplicated codes to call new base sdp apis<\/a><\/li>\n<\/ul>\n\n<p>Until this time I focused on RTSP\/RTP streaming on server side since I was working on Miracast on Tizen which uses gst-rtsp-server.\nAt this time I started looking for company so that I could work on open source more closely. Eventually I found <a href=\"https:\/\/igalia.com\">Igalia<\/a>, which is doing great work in open source world including Webkit, Chromium and GStreamer.<\/p>\n\n<p>Since I joined Igalia I have been focusing on gstreamer-vaapi with my great colleague <a href=\"https:\/\/www.igalia.com\/nc\/igalia-247\/igalian\/item\/vjaquez\">Victor<\/a>, who is one of maintainers of GStreamer project. I got to have much more chances to contribute than before.\nAs I said, I worked on RTSP server side before, which means that I should focus on encoder, muxer and networking stuff. But since this move, I got started focusing on playback including decoder and sink to be playable on various platforms.<\/p>\n\n<p>These are my best patches I think.<\/p>\n<ul>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gst-plugins-bad\/commit\/?id=2bb3baa8c9677685215f8ff3d7204f43d750a472\">glupload: Use bufferpool to allocate new buffer in GLTextureUploadMeta<\/a><\/li>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gstreamer-vaapi\/commit\/?id=f974c91d73b669e90cfaeed095c8e6f7dbee9103\">libs: vaapitexturemap: implement GstVaapiTextureMap<\/a><\/li>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gstreamer-vaapi\/commit\/?id=a80e10ac5c85107592da3ce327d7739415cc9cb7\">libs: display{egl,glx}: cache GstVaapiTextures<\/a><\/li>\n  <li><a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gstreamer-vaapi\/commit\/?id=5a4540e6540e78be2ca4d491543c92b47dc91d06\">plugins: reset textures at negotiation\/shutdown<\/a><\/li>\n<\/ul>\n\n<p>By this set of patches, performance of playback on GL\/VAAPI has been improved dramatically.<\/p>\n\n<p>Besides, I have contributed some patches that improve vaapi decoder and encoder, most of them is for h264, which also makes me happy.<\/p>\n\n<p>During the last three years I worked for GStreamer, I grew up with more capability of SW development, the idea of open source and more deep insight for the world of software.\nI give a deep appreciation for Igalia that gave me this opportunity, and also I thank you, Victor, for giving me a lot of motivation.<\/p>\n\n<p>Even at this moment, I\u2019m still working, enjoying and sometimes struggling with GStreamer.\nI really want to keep continuing this work and find a chance to contribute something new which could be applied on GStreamer.<\/p>\n\n<p>Thanks for reading!<\/p>\n"},{"title":"Libva-rust(libva binding to rust) in development stage","link":{"@attributes":{"href":"\/libva,\/rust\/2017\/03\/20\/libva-release\/"}},"updated":"2017-03-20T12:15:06+09:00","id":"https:\/\/blogs.igalia.com\/zzoon\/libva,\/rust\/2017\/03\/20\/libva-release","content":"<p>Since Rust language appeared in the world, I felt strongly this is the language I should learn.<\/p>\n\n<p>This is because:<\/p>\n\n<ul>\n  <li>Rust guarantees to prevent from common bugs in C\/C++ such as memory corruption and race condition, which are very painful to fix whenever you encounter in large project.<\/li>\n  <li>Rust guarantees it doesn\u2019t lose performance even supporting these features!<\/li>\n<\/ul>\n\n<p>I don\u2019t think that Rust aims at replacing C\/C++, but it\u2019s worth learning for C\/C++ developers like me at least.  So I\u2019ve been searching and thinking of what I can do with this new language.  In the end of last year, I decided to implement libva bindings to Rust.<\/p>\n\n<p>Here are advantages of doing this project.<\/p>\n\n<ul>\n  <li>I\u2019m working on <a href=\"https:\/\/cgit.freedesktop.org\/gstreamer\/gstreamer-vaapi\/\">gstreamer-vaapi<\/a> project, which means that I\u2019m familiar with VA-API and middleware using this.<\/li>\n  <li>This kind of binding the existing project to another language makes me understanding the project much more than the moment.<\/li>\n  <li>Simultaneously, I could also learn new language in the level of practical development.<\/li>\n  <li>H\/W acceleration is a critical feature, especially for laptop or other embedded systems. So this project could be a good option for those trying to use H\/W acceleration for playback on linux.<\/li>\n<\/ul>\n\n<p>Finally, I did open this internal project on github, named as <a href=\"https:\/\/github.com\/zzoon\/libva-rust\">libva-rust<\/a>.<br \/>\nThere is one example, creating an VASurface and putting raw data to the surface and displaying it only on X11 window.<\/p>\n\n<p>Let\u2019s see the example code briefly.<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-rust\" data-lang=\"rust\"><span class=\"k\">let<\/span> <span class=\"n\">va_disp<\/span> <span class=\"o\">=<\/span> <span class=\"nn\">VADisplay<\/span><span class=\"p\">::<\/span><span class=\"nf\">initialize<\/span><span class=\"p\">(<\/span><span class=\"n\">native_display<\/span> <span class=\"k\">as<\/span> <span class=\"o\">*<\/span><span class=\"k\">mut<\/span> <span class=\"n\">VANativeDisplay<\/span><span class=\"p\">)<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n\n<span class=\"k\">let<\/span> <span class=\"n\">va_surface<\/span> <span class=\"o\">=<\/span> <span class=\"nn\">VASurface<\/span><span class=\"p\">::<\/span><span class=\"nf\">new<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">va_disp<\/span><span class=\"p\">,<\/span> <span class=\"n\">WIDTH<\/span><span class=\"p\">,<\/span> <span class=\"n\">HEIGHT<\/span><span class=\"p\">,<\/span> <span class=\"nn\">ffi<\/span><span class=\"p\">::<\/span><span class=\"n\">VA_RT_FORMAT_YUV420<\/span><span class=\"p\">,<\/span> <span class=\"mi\">1<\/span><span class=\"p\">)<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n\n<span class=\"k\">let<\/span> <span class=\"n\">va_config<\/span> <span class=\"o\">=<\/span> <span class=\"nn\">VAConfig<\/span><span class=\"p\">::<\/span><span class=\"nf\">new<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">va_disp<\/span><span class=\"p\">,<\/span> <span class=\"nn\">ffi<\/span><span class=\"p\">::<\/span><span class=\"n\">VAProfileMPEG2Main<\/span><span class=\"p\">,<\/span> <span class=\"nn\">ffi<\/span><span class=\"p\">::<\/span><span class=\"n\">VAEntrypointVLD<\/span><span class=\"p\">,<\/span> <span class=\"mi\">1<\/span><span class=\"p\">)<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n\n<span class=\"k\">let<\/span> <span class=\"n\">va_context<\/span> <span class=\"o\">=<\/span> <span class=\"nn\">VAContext<\/span><span class=\"p\">::<\/span><span class=\"nf\">new<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">va_disp<\/span><span class=\"p\">,<\/span>\n                                <span class=\"o\">&amp;<\/span><span class=\"n\">va_config<\/span><span class=\"p\">,<\/span>\n                                <span class=\"o\">&amp;<\/span><span class=\"n\">va_surface<\/span><span class=\"p\">,<\/span>\n                                <span class=\"n\">WIDTH<\/span> <span class=\"k\">as<\/span> <span class=\"nb\">i32<\/span><span class=\"p\">,<\/span>\n                                <span class=\"n\">HEIGHT<\/span> <span class=\"k\">as<\/span> <span class=\"nb\">i32<\/span><span class=\"p\">,<\/span>\n                                <span class=\"mi\">0<\/span><span class=\"p\">)<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span><\/code><\/pre><\/figure>\n\n<p>Initalization for VA-API.<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-rust\" data-lang=\"rust\"><span class=\"nn\">test_draw<\/span><span class=\"p\">::<\/span><span class=\"nf\">image_generate<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">va_disp<\/span><span class=\"p\">,<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">va_image<\/span><span class=\"p\">,<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">va_image_buf<\/span><span class=\"p\">);<\/span>\n\n<span class=\"n\">va_image<\/span><span class=\"nf\">.put_image<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">va_disp<\/span><span class=\"p\">,<\/span>\n                   <span class=\"o\">&amp;<\/span><span class=\"n\">va_surface<\/span><span class=\"p\">,<\/span>\n                   <span class=\"mi\">0<\/span><span class=\"p\">,<\/span>\n                   <span class=\"mi\">0<\/span><span class=\"p\">,<\/span>\n                   <span class=\"n\">WIDTH<\/span><span class=\"p\">,<\/span>\n                   <span class=\"n\">HEIGHT<\/span><span class=\"p\">,<\/span>\n                   <span class=\"mi\">0<\/span><span class=\"p\">,<\/span>\n                   <span class=\"mi\">0<\/span><span class=\"p\">,<\/span>\n                   <span class=\"n\">WIDTH<\/span><span class=\"p\">,<\/span>\n                   <span class=\"n\">HEIGHT<\/span><span class=\"p\">);<\/span><\/code><\/pre><\/figure>\n\n<p>Draw raw data to VaapiImage in test_draw.rs and put it to the created surface.<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-rust\" data-lang=\"rust\"><span class=\"n\">va_surface<\/span><span class=\"nf\">.put_surface<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">va_disp<\/span><span class=\"p\">,<\/span> <span class=\"n\">win<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"n\">WIDTH<\/span><span class=\"p\">,<\/span> <span class=\"n\">HEIGHT<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"n\">WIDTH<\/span><span class=\"p\">,<\/span> <span class=\"n\">HEIGHT<\/span><span class=\"p\">);<\/span><\/code><\/pre><\/figure>\n\n<p>Finally, display it by putting the surface to created X11 window.<br \/>\nIt\u2019s simple as you see, but the important first step.<\/p>\n\n<p>My first goal is providing general and easy a set of \u201crusty\u201d APIs so that this could be integrated into other rust-multimedia project like <a href=\"https:\/\/github.com\/pcwalton\/rust-media\">rust-media<\/a>.<\/p>\n\n<p>Another potential goal is implementation of vaapi plugins in gstreamer, written in Rust. Recently, Sebastian has been working on this(<a href=\"https:\/\/github.com\/sdroege\/rsplugin\">https:\/\/github.com\/sdroege\/rsplugin<\/a>), I would really like to get involved in this project.<\/p>\n\n<p>There are tons of things to do for the moment.<br \/>\nHere\u2019s to-do list for now.<\/p>\n\n<ul>\n  <li>Implement vp8 decoder first: Simply, it looks easier than h26x decoder. Is there any useful rust h26x parser out there, by the way?<\/li>\n  <li>Manipulate raw data using Rust apis like Bit\/Byte Reader\/Writer.<\/li>\n  <li>Implement general try-catch statement in Rust.<\/li>\n  <li>Make test cases.<\/li>\n  <li>Support wayland.<\/li>\n<\/ul>\n\n<p>Yes. It has a long way to go still and I don\u2019t have enough time to focus on this project.  But I\u2019ll be managing to keep working on this project.<\/p>\n\n<p>So feel free to use and absolutely welcome contributions including issue report, bug fix, providing a patch, etc.<\/p>\n\n<p>Thanks!<\/p>\n\n"}]}