{"@attributes":{"version":"2.0"},"channel":{"title":"Posts on Openinx Blog","link":"https:\/\/openinx.github.io\/posts\/","description":"Recent content in Posts on Openinx Blog","image":{"title":"Openinx Blog","url":"https:\/\/openinx.github.io\/%3Clink%20or%20path%20of%20image%20for%20opengraph,%20twitter-cards%3E","link":"https:\/\/openinx.github.io\/%3Clink%20or%20path%20of%20image%20for%20opengraph,%20twitter-cards%3E"},"generator":"Hugo -- 0.125.7","language":"en","lastBuildDate":"Sun, 11 May 2025 20:22:24 -0700","item":[{"title":"Thoughts of SaaS services","link":"https:\/\/openinx.github.io\/posts\/2025-05-11-new-us-saas\/","pubDate":"Sun, 11 May 2025 20:22:24 -0700","guid":"https:\/\/openinx.github.io\/posts\/2025-05-11-new-us-saas\/","description":"I have been working at a US company for almost a month recently. An interesting fact is that I have counted the SaaS services I have used for specific scenarios since joining, and surprisingly, there are as many as 23 of them. All of these SaaS services are provided by third-party companies. In other words, Databricks is a customer of all these companies. This aroused my curiosity and made me want to learn about the situation of these companies."},{"title":"What I learned from ByteDance","link":"https:\/\/openinx.github.io\/posts\/2025-05-04-review-bytedance-journey\/","pubDate":"Sun, 04 May 2025 23:16:23 -0700","guid":"https:\/\/openinx.github.io\/posts\/2025-05-04-review-bytedance-journey\/","description":"I recently had the pleasure of joining Databricks, but I often reflect on my profoundly rewarding experience at ByteDance. Before joining ByteDance, I was a software engineer who had written code for a decade. My core responsibility was to tackle system design and code development for complex software while ensuring robust operations in production environments. At ByteDance, however, I led a 16-person technical team (mostly Senior+ engineers) responsible for building the competitiveness of ByteDance\u2019s public cloud EMR open-source engines\u2014a role that was immensely challenging yet transformative for me."},{"title":"Iceberg Summit 2025 - Part 2","link":"https:\/\/openinx.github.io\/posts\/2025-04-11-iceberg-summit-2025-2\/","pubDate":"Fri, 11 Apr 2025 21:56:43 -0700","guid":"https:\/\/openinx.github.io\/posts\/2025-04-11-iceberg-summit-2025-2\/","description":"This is the second article of my thoughts for Iceberg Summit 2025 (Here is Part-1), which is not limited to Iceberg but focuses on Data Lake. I am trying to share what these creative teams are doing in this field, why it is a problem, what the solution is, and how it will develop in the future. These insights come from various sharing, discussions, and debates, but are limited by my personal understanding."},{"title":"Iceberg Summit 2025 - Part 1","link":"https:\/\/openinx.github.io\/posts\/2025-04-10-iceberg-summit-2025-1\/","pubDate":"Thu, 10 Apr 2025 17:12:24 -0700","guid":"https:\/\/openinx.github.io\/posts\/2025-04-10-iceberg-summit-2025-1\/","description":"I attended the Apache Iceberg Summit 2025 in the US in the past two days. The conference was held at the Hyatt Regency Hotel next to Union Square in downtown San Francisco. This time, two floors of the hotel were rented as the venue. Mainstream companies in the Data field are basically sponsors of the conference, including AWS, Databricks, Snowflake, Cloudera, Dremio, Microsoft, Airbyte, CelerData, Confluent, eltloop, Google, IMB, Minio, Qlik, Redpanda, Starburst, StreamNative, e6data, daft, PuppyGraph, RisingWave, Telmai, wherobots."},{"title":"SQL Server's Column Store Update Design","link":"https:\/\/openinx.github.io\/posts\/2022-07-01-realtime-update-in-columar-table\/","pubDate":"Fri, 01 Jul 2022 12:00:24 +0800","guid":"https:\/\/openinx.github.io\/posts\/2022-07-01-realtime-update-in-columar-table\/","description":"I recently read Microsoft’s 2015 VLDB paper, Real-Time Analytical Processing with SQL Server [1]. SQL Server was among the earlier products to ship and productionize an HTAP row\/column update design. Row-store (row-wise index) tradeoffs for OLTP are well understood; efficient millisecond-scale column store updates have several designs in the wild\u2014Kudu [2], Positional Delta Tree [3], and others\u2014but they receive far less discussion than row stores. After a careful review of SQL Server’s approach, I find it worth sharing."},{"title":"PingCAP Hackathon 2019","link":"https:\/\/openinx.github.io\/posts\/2019-10-26-pingcap-hackthon2019\/","pubDate":"Sat, 26 Oct 2019 13:16:43 +0800","guid":"https:\/\/openinx.github.io\/posts\/2019-10-26-pingcap-hackthon2019\/","description":"I joined TiDB’s Hackathon 2019 and finally have time to write this recap.\nA 48-hour hackathon around TiDB: build a demo, present in six minutes. Scoring weights practicality, ease of use, and performance for TiDB (40%), completeness (30%), innovation (20%), and presentation (10%).\nOur team of three: captain Yi Wu from PingCAP’s US office (ex-Facebook RocksDB, now RocksDB at PingCAP), and Bokang Zhang in Beijing (TiKV). All three work on storage, so we aimed at the bottom of the stack."},{"title":"A Journey to Optimize HBase 2.x Write Performance","link":"https:\/\/openinx.github.io\/posts\/2019-09-10-tuning-the-hbase-2-write-performance\/","pubDate":"Tue, 10 Sep 2019 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2019-09-10-tuning-the-hbase-2-write-performance\/","description":"How good is HBase 2.x write performance? Let’s benchmark it.\nTest environment: five-node cluster; each node has twelve 800GB SSDs, 24 CPU cores, 128GB RAM. HBase and HDFS are co-located\u2014RegionServer and DataNode on the same host for better write locality (at least one replica local). Software: HBase 2.1.2, HDFS 2.6.0, OpenJDK 1.8.0_202.\nPer RegionServer we use 50GB heap and 50GB off-heap (100GB total)\u2014heap mainly for memstore (~36GB), off-heap mainly for BucketCache (~36GB)."},{"title":"Future Work Worth Pursuing for HBase 2.x in the Community","link":"https:\/\/openinx.github.io\/posts\/2019-07-15-future-work-for-hbase2.x\/","pubDate":"Fri, 06 Sep 2019 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2019-07-15-future-work-for-hbase2.x\/","description":"HBase 2.0.0 shipped on April 30, 2018\u2014about 15 months ago as of this writing. HBase 2.0.x is EOL with no further releases. HBase 2.1 is at 2.1.6 and likely will not be maintained much longer. Stable 2.x lines will be 2.2.x and 2.3.x, especially 2.2.x as the version battle-tested at large shops.\nWork I think deserves focus on HBase 2.x:\nProcedure V2 and Assignment V2 give distributed workflows atomicity through a framework\u2014a pain point in 1."},{"title":"Recommending Our Book: HBase Principles and Practice","link":"https:\/\/openinx.github.io\/posts\/2019-09-06-hbase-princple-and-practice\/","pubDate":"Fri, 06 Sep 2019 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2019-09-06-hbase-princple-and-practice\/","description":"After working in the Apache HBase community for a while, I noticed a few relentlessly productive leaders: Michael Stack, contributing day in and day out for a decade, and Duo Zhang, recently promoted to HBase project chair. Stack\u2014a senior engineer I could call “uncle” by age\u2014works across my entire day: morning mail shows a JIRA reply, afternoon another patch, late night comments on someone else’s patch. Duo Zhang\u2014“Duo God” in Lei Jun’s blog\u2014sits to my right."},{"title":"HBaseConAsia 2019 Is Coming","link":"https:\/\/openinx.github.io\/posts\/2019-07-08-hbaseconasia2019-is-coming\/","pubDate":"Mon, 08 Jul 2019 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2019-07-08-hbaseconasia2019-is-coming\/","description":"The third Apache HBaseConAsia summit takes place in Beijing on July 20. As the Apache HBase community’s premier user conference in Asia, HBaseCon has run since 2012. More than 20 experts and community leaders from leading internet and big-data companies will share the latest on HBase and the surrounding ecosystem.\nApache HBase is a highly available, high-performance, multi-version distributed NoSQL database on Apache Hadoop\u2014Google Bigtable’s open-source counterpart. It delivers fast random read\/write at scale on commodity servers."},{"title":"A Primer on HBase Filters","link":"https:\/\/openinx.github.io\/posts\/2019-07-02-blog-for-filter-list\/","pubDate":"Tue, 02 Jul 2019 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2019-07-02-blog-for-filter-list\/","description":"Motivation\nDatabases need flexible query patterns. A KV store with only Get, Put, and Scan would frustrate users\u2014real workloads are richer. For orders: “this user’s orders in the last three months” needs at least (1) filter by user and (2) filter by time range, combined with AND. Scanning the whole table on the client and filtering locally would crush the cluster. Server-side filters are required.\nYou also see OR, e.g. orders for Alice or Bob in the last three months\u2014AND vs OR, often mixed."},{"title":"Nebula Beijing Meetup Summary","link":"https:\/\/openinx.github.io\/posts\/2019-06-29-nebula-meetup\/","pubDate":"Sat, 29 Jun 2019 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2019-06-29-nebula-meetup\/","description":"Nebula held its second Beijing meetup in Jingdong Beichen. CEO Xiaomeng Ye led Geobase, Ant Financial’s graph database. Two technical directors\u2014Heng Chen and Fenglin Hou (dutor)\u2014drive storage (similar to TiKV) and the query engine (similar to TiDB), respectively.\nThe architecture separates storage and compute. The compute layer is SQL-like, with Go syntax for edge hops and pipelines to simplify nested subqueries. Storage is key-value with static hash partitioning by key (no dynamic partitioning yet)."},{"title":"Further GC optimization for HBase3.x: Reading HFileBlock into offheap directly","link":"https:\/\/openinx.github.io\/posts\/2019-06-23-offheap-block-reading\/","pubDate":"Sun, 23 Jun 2019 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2019-06-23-offheap-block-reading\/","description":"In HBASE-21879, we redesigned the offheap read path: read the HFileBlock from HDFS to pooled offheap ByteBuffers directly, while before HBASE-21879 we just read the HFileBlock to heap which would still lead to high GC pressure.\nAfter few months of development and testing, all subtasks have been resovled now except the HBASE-21946 (It depends on HDFS-14483 and our HDFS teams are working on this, we expect the HDFS-14483 to be included in hadoop 2."},{"title":"From HBase Off-Heap to Netty Memory Management","link":"https:\/\/openinx.github.io\/posts\/2019-02-23-netty-memory-management\/","pubDate":"Sat, 23 Feb 2019 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2019-02-23-netty-memory-management\/","description":"HBase Off-Heap Today\nHBase is a widely used distributed NoSQL database. Many workloads\u2014feeds, ads, and similar\u2014demand high throughput and low latency. HBase 2.0 off-heaped the core read and write paths: allocations go to JVM off-heap memory, which is not GC-managed and must be freed explicitly. On the write path, request buffers are allocated off-heap until data is written to the WAL and memstore. The memstore’s ConcurrentSkipListMap holds references to cells, not cell bodies; actual data lives in MSLAB chunks for easier off-heap management."},{"title":"HBaseCon West 2018 Talk - HBase Practice at Xiaomi","link":"https:\/\/openinx.github.io\/posts\/2018-06-18-hbaseconwest2018\/","pubDate":"Mon, 18 Jun 2018 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2018-06-18-hbaseconwest2018\/","description":"HBaseConWest2018 was held on June 18 in San Jose, California, hosted by Hortonworks. Attending HBaseCon West in Silicon Valley each year has become routine for the Xiaomi HBase team\u2014our community presence is well known (seven HBase Committers, two PMC members), and the company is willing to share a year-in-review of internal practice and community contributions.\nIn 2018 we submitted the talk “HBase Practice at Xiaomi,” spent considerable effort preparing it, and rehearsed in English three times internally."},{"title":"Becoming an HBase Committer","link":"https:\/\/openinx.github.io\/posts\/2017-10-22-become-hbase-committer\/","pubDate":"Sun, 22 Oct 2017 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2017-10-22-become-hbase-committer\/","description":"On October 20, I accepted an invitation from the Apache HBase community and became an HBase Committer. At Xiaomi I maintain our internal HBase branch and production clusters, and the company already had six HBase Committers\u2014including one PMC member\u2014so becoming a Committer was a natural next step rather than something extraordinary. Compared with someone doing HBase at a company with no Committers, it takes considerably more time and effort.\nSome observations about the community:"},{"title":"HBase Region Balance in Practice","link":"https:\/\/openinx.github.io\/posts\/2016-06-21-hbase-balance\/","pubDate":"Wed, 28 Jun 2017 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2016-06-21-hbase-balance\/","description":"HBase is a distributed key-value database that supports automatic load balancing. With the balance switch (balance_switch) enabled, the HMaster process automatically selects regions according to a specified policy and assigns them to RegionServers with lower load. The official distribution currently supports two region-selection policies: DefaultLoadBalancer and StochasticLoadBalancer, both described in detail below. Because all HBase data (including HLog, meta, HStoreFile, and so on) is written to HDFS, region moves are very lightweight."},{"title":"HBaseCon West 2017 Session Notes","link":"https:\/\/openinx.github.io\/posts\/2017-06-28-hbaseconwest2017\/","pubDate":"Wed, 28 Jun 2017 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2017-06-28-hbaseconwest2017\/","description":"Notes on HBaseCon West 2017 presentations:\n1. HBase at Xiaomi\nPresented jointly by Zhe Yang and Guanghao Zhang\u2014both became HBase Committers in 2016 (Xiaomi has produced eight HBase Committers in total, including two PMC members, and has resolved hundreds of issues). Highlights included:\nLessons from upgrading clusters from 0.94 to 0.98. Experience using G1GC for internal HBase deployments. Community contributions and improvements in 2016, including ordered replication log push, Scan optimizations, async client development, and related benchmark results."},{"title":"HBase HLog Replay Ordering Inconsistency","link":"https:\/\/openinx.github.io\/posts\/2016-06-13-hbase-replicate-hlog\/","pubDate":"Mon, 13 Jun 2016 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2016-06-13-hbase-replicate-hlog\/","description":"In an HBase master-slave replication cluster, as shown on the left in the figure below, Region-Server-X and Region-Server-Y are two RegionServers in the master cluster. Under normal conditions, writes to Region-A append logs to Hlog-X on Region-Server-X, and Region-Server-X asynchronously applies those HLog entries in batches to the slave cluster. If Region-Server-X then crashes, Region-A is taken over by Region-Server-Y. Subsequent writes to Region-A append logs to Hlog-Y, while Region-Server-Y starts a new thread to replay Hlog-X."},{"title":"TokuDB's Multi-Version Concurrency Control (MVCC)","link":"https:\/\/openinx.github.io\/posts\/2015-12-13-ft-mvcc\/","pubDate":"Sun, 13 Dec 2015 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2015-12-13-ft-mvcc\/","description":"This article covers transaction isolation in TokuDB. The source implementation is complex; for clarity, we focus on the most essential parts and omit minor details.\nBackground In traditional relational databases (Oracle, MySQL, SQL Server, and others), transactions are central to both engineering and discussion. The core properties of a transaction are ACID. A (atomicity) means a transaction’s sub-operations have only two outcomes: all succeed on commit, or all are undone on rollback."},{"title":"TokuDB's Index Structure: Fractal Tree Implementation","link":"https:\/\/openinx.github.io\/posts\/2015-11-25-ft-index-implement\/","pubDate":"Wed, 25 Nov 2015 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2015-11-25-ft-index-implement\/","description":"This article analyzes TokuDB’s index structure\u2014the fractal tree\u2014from an engineering implementation perspective. It describes the on-disk storage layout of ft-index in detail, how ft-index implements point queries, range queries, and insert\/delete\/update operations, and throughout the discussion attempts a detailed comparison with InnoDB’s B+ tree from multiple angles.\nIntroduction to the Fractal Tree The fractal tree is a write-optimized on-disk index data structure. In general, fractal trees offer good write performance (insert\/update\/delete) while still providing read performance close to that of a B+ tree."},{"title":"Vitess Explained","link":"https:\/\/openinx.github.io\/posts\/2015-02-02-vitess-analysis\/","pubDate":"Mon, 02 Feb 2015 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2015-02-02-vitess-analysis\/","description":"Online version: PPT\nThis is a technical talk on Vitess I gave internally at my company. The main topics are:\nVitess feature modules, capabilities, and system architecture Vitess sharding approach SQL syntax supported by Vitess How Vitess resharding works Vitess data backup internals Pros and cons of Vitess compared with traditional relational databases and NoSQL A side note\nTo try building an online PPT with Markdown, I experimented with a few options."},{"title":"Google 2015 Campus Hiring Written Test \u2014 Round B","link":"https:\/\/openinx.github.io\/posts\/2014-09-16-google-apac-2015-test-round-b\/","pubDate":"Tue, 16 Sep 2014 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2014-09-16-google-apac-2015-test-round-b\/","description":"I took Google’s online campus hiring written test yesterday. Skip algorithms for a day and your skills slide right back down.\nProblem A. Password Attacker Description How many passwords of length M can be formed from N distinct characters, such that every password uses all N characters at least once?\nSolution 1: Brute Force For each solution to the equation below, add the number of distinct permutations. If one solution is X1, …, Xn, that group contributes M!"},{"title":"How LevelDB Compaction Works","link":"https:\/\/openinx.github.io\/posts\/2014-08-17-leveldb-compaction\/","pubDate":"Tue, 16 Sep 2014 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2014-08-17-leveldb-compaction\/","description":"Basic LevelDB constraints With the default options, LevelDB follows these basic constraints:\nLevelDB has 7 levels: 0, 1, 2, 3, 4, 5, and 6. SSTables on level 0 are about 4 MB each. On level i (i > 0), each SSTable is at most 2 MB. Level 0 ideally has 4 SSTables, should stay within 8, and must not exceed 12. The total storage used by all SSTables on level i (i > 0) should stay around 10^i MB."},{"title":"LeetCode 151 Summary","link":"https:\/\/openinx.github.io\/posts\/2014-07-20-leetcode-151-finished\/","pubDate":"Sun, 20 Jul 2014 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2014-07-20-leetcode-151-finished\/","description":"After grinding LeetCode for several days, I finally finished. Code is here.\nReverse Words in a String Simulation on strings\nEvaluate Reverse Polish Notation Simulation \u2014 evaluate postfix expressions\nMax Points on a Line Given N points in the plane, find a line that passes through the most points. Enumerate each point as the origin, compute relative coordinates, then compute y\/x and use a hash table to count duplicates and find the maximum."},{"title":"Kazoo, the ZooKeeper Python Client","link":"https:\/\/openinx.github.io\/posts\/2014-06-07-learning-from-kazoo\/","pubDate":"Sat, 07 Jun 2014 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2014-06-07-learning-from-kazoo\/","description":"Zookeeper hardly needs an introduction \u2014 it is a distributed coordination service. A few questions I was curious about:\nHow does ZooKeeper implement asynchronous watcher callbacks? (code-level details) How does ZooKeeper implement distributed locks? How do Queue, barrier, and similar recipes work? I read the Python client kazoo and got the general picture.\nA simple client example #!\/usr\/bin\/python import logging from time import sleep from kazoo.client import KazooClient # print log to console logging."},{"title":"The K-th Element in Algorithms","link":"https:\/\/openinx.github.io\/posts\/2014-03-02-the-k-the-number-in-algorithm\/","pubDate":"Sun, 02 Mar 2014 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2014-03-02-the-k-the-number-in-algorithm\/","description":"Source code and related articles are available here.\nThis article discusses the K-th largest or K-th smallest element in a sequence. Since the K-th largest can be reduced to finding the (N\u2212K+1)-th smallest (where N is the sequence length), we focus on the K-th smallest element.\nProblems covered in this article:\nGiven an integer sequence, find the K-th smallest element. Given an integer sequence that supports dynamic updates, answer queries for the K-th smallest element."},{"title":"How Redis Implements Its Dictionary","link":"https:\/\/openinx.github.io\/posts\/2014-02-13-the-implement-of-redis-dict\/","pubDate":"Thu, 13 Feb 2014 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2014-02-13-the-implement-of-redis-dict\/","description":"Hash Table\nA hash table is an array of size buckets, table[0...size-1]. Hashing an object yields an index; we store the object in table[value]. When a bucket holds multiple objects, they are chained into a linked list. This collision strategy is called separate chaining.\nLoad Factor\nIf a hash table has size buckets and used stored elements, used \/ size is the load factor. When loadFactor <= 1, expected lookup cost is O(1)."},{"title":"Plane Sweep Techniques in ACM Programming Contests","link":"https:\/\/openinx.github.io\/posts\/2013-01-01-plane-sweep-thinking\/","pubDate":"Tue, 01 Jan 2013 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2013-01-01-plane-sweep-thinking\/","description":"Abstract: Plane sweep is widely used in computational geometry, computer graphics, grid computing, and related areas. Many classic algorithms use plane sweep to greatly reduce time complexity \u2014 e.g. segment intersection, union of axis-aligned rectangle perimeters, rectangle intersection, spatial collision detection, Voronoi diagram construction, closest pair of points, and more.\nThis article introduces several plane sweep algorithms commonly used in ACM programming contests. By purpose, they fall into:\nData aggregation; Detecting geometric positional relationships; Closest pair of points."},{"title":"My Public Talks","link":"https:\/\/openinx.github.io\/posts\/2012-01-01-my-share\/","pubDate":"Sun, 01 Jan 2012 13:27:08 +0800","guid":"https:\/\/openinx.github.io\/posts\/2012-01-01-my-share\/","description":"2021.01.08 Flink Forward Asia 2021 Beijing: The Best Practice of Integrating Apache Flink with Apache Iceberg 2021.04.25 Flink+Iceberg Shanghai Meetup: How Flink and Iceberg Solve the Challenges of Data Lake Ingestion 2020.12.15 Flink Forward Asia 2020 Beijing: How to Analyze CDC Data in Iceberg Data Lake Using Flink, Blog 2019.07.20 HBaseConAsia 2019 Beijing: Further GC optimization for HBase 2.x: Reading HFileBlock into offheap directly. 2018.10.18 Internal Xiaomi talk on the HBase read path PDF 2018."}]}}