{"@attributes":{"version":"2.0"},"channel":{"title":"Samir P.","link":"https:\/\/samir.pages.dev\/","description":"A place to share learnings, ideas, short notes, guides, and reflections from projects and everyday life, focused on clarity and usefulness. Lessons learned and experiences are distilled into actionable takeaways for curious readers.","generator":"Hugo 0.169.0 & FixIt v1.0.0-beta.5","language":"en","managingEditor":"netproxy@duck.com (Samir)","webMaster":"netproxy@duck.com (Samir)","lastBuildDate":"Fri, 03 Jul 2026 00:00:00 +0000","item":[{"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","link":"https:\/\/samir.pages.dev\/posts\/database-schema-migrations\/","pubDate":"Fri, 03 Jul 2026 00:00:00 +0000","author":"netproxy@duck.com (Samir)","guid":"https:\/\/samir.pages.dev\/posts\/database-schema-migrations\/","category":"Software-Engineering","description":"<img src=\"https:\/\/samir.pages.dev\/files\/db-schema-migration.webp\" alt=\"featured image\" referrerpolicy=\"no-referrer\"><div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Disclaimer<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">I&rsquo;m not endorsing or discouraging any specific tool in this post. This is purely from my personal experience working with microservices and different databases across various projects over the years. What worked best for me might not be the right fit for your team or stack. Evaluate based on your own constraints \u2014 team size, database engine, deployment model, and risk tolerance. Every tool mentioned here has trade-offs, and the &ldquo;best&rdquo; choice depends entirely on your context.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 class=\"heading-element\" id=\"why-schema-migrations-matter\"><span>Why Schema Migrations Matter<\/span>\n <a href=\"#why-schema-migrations-matter\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Every application that talks to a database eventually needs to change its schema. New columns, new tables, renamed fields, dropped constraints. In a monolith with one database, this is manageable. In a microservices architecture with multiple databases (maybe even different engines \u2014 <a href=\"https:\/\/www.postgresql.org\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">PostgreSQL<\/a> for transactional data, <a href=\"https:\/\/clickhouse.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ClickHouse<\/a> for analytics), it gets messy fast.<\/p>\n<p>The problems I kept running into:<\/p>\n<ul>\n<li><strong>Who applied what, and when?<\/strong> No audit trail of schema changes.<\/li>\n<li><strong>Works on my machine.<\/strong> Dev database has the column, staging doesn&rsquo;t.<\/li>\n<li><strong>Two pipelines running at once.<\/strong> Race condition corrupts the migration state.<\/li>\n<li><strong>Failed migration halfway through.<\/strong> Database stuck in a dirty state, all future migrations blocked.<\/li>\n<li><strong>Shared schema, multiple services.<\/strong> Service A needs a new column, Service B still reads the old one. How do you not break either?<\/li>\n<li><strong>Sharded clusters.<\/strong> DDL needs to propagate to 20+ nodes consistently.<\/li>\n<\/ul>\n<p>After trying a few approaches (hand-written SQL scripts, ORM-based migrations, custom tooling), I settled on <a href=\"https:\/\/github.com\/golang-migrate\/migrate\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">golang-migrate<\/a> \u2014 a database-agnostic migration tool written in Go that works as both a CLI and a library.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"why-not-flyway-or-liquibase\"><span>Why Not Flyway or Liquibase?<\/span>\n <a href=\"#why-not-flyway-or-liquibase\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Before landing on golang-migrate, I tried two of the most popular migration tools: <a href=\"https:\/\/www.red-gate.com\/products\/flyway\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Flyway<\/a> and <a href=\"https:\/\/www.liquibase.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Liquibase<\/a>.<\/p>\n<p>Both worked fine for PostgreSQL. But my stack also included a <a href=\"https:\/\/clickhouse.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ClickHouse<\/a> cluster with <a href=\"https:\/\/zookeeper.apache.org\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ZooKeeper<\/a> coordination \u2014 and that&rsquo;s where things fell apart.<\/p>\n<div class=\"details admonition danger open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-bolt\" aria-hidden=\"true\"><\/i>The ClickHouse + ZooKeeper Problem<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p>ClickHouse in clustered mode requires <code>ON CLUSTER<\/code> DDL statements that propagate schema changes across all shards via ZooKeeper. Neither Flyway nor Liquibase had proper support for this at the time I evaluated them:<\/p>\n<ul>\n<li><strong><a href=\"https:\/\/www.red-gate.com\/products\/flyway\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Flyway<\/a><\/strong> \u2014 The community ClickHouse driver was incomplete. <code>ON CLUSTER<\/code> DDL propagation wasn&rsquo;t handled correctly, and the tool would report success while only one node received the schema change. Multi-statement migrations (needed for creating both <code>_local<\/code> and <code>Distributed<\/code> tables) required hacks.<\/li>\n<li><strong><a href=\"https:\/\/www.liquibase.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Liquibase<\/a><\/strong> \u2014 The ClickHouse extension existed but was buggy with ZooKeeper-coordinated DDL. The changelog tracking didn&rsquo;t play well with ClickHouse&rsquo;s eventual consistency model across shards.<\/li>\n<\/ul>\n<p>golang-migrate&rsquo;s <a href=\"https:\/\/github.com\/golang-migrate\/migrate\/tree\/master\/database\/clickhouse\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ClickHouse driver<\/a> supports <code>x-multi-statement=true<\/code> in the connection URL, handles <code>ON CLUSTER<\/code> DDL natively, and works with both the native TCP protocol (port 9440 with TLS) and the standard ClickHouse cluster topology.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<p>If your stack is PostgreSQL-only or MySQL-only, Flyway and Liquibase are solid choices with larger ecosystems and excellent Java integration. But if you&rsquo;re dealing with ClickHouse clusters, sharded analytical databases, or exotic engines \u2014 golang-migrate&rsquo;s lightweight driver model handled it significantly better in my experience.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"the-architecture--high-level-overview\"><span>The Architecture \u2014 High-Level Overview<\/span>\n <a href=\"#the-architecture--high-level-overview\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Here&rsquo;s the architecture I ended up with across projects:<\/p>\n<pre><code>%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: {&#39;primaryColor&#39;: &#39;#4fc3f7&#39;, &#39;primaryTextColor&#39;: &#39;#000&#39;, &#39;primaryBorderColor&#39;: &#39;#0288d1&#39;, &#39;lineColor&#39;: &#39;#455a64&#39;, &#39;secondaryColor&#39;: &#39;#a5d6a7&#39;, &#39;tertiaryColor&#39;: &#39;#fff9c4&#39;, &#39;background&#39;: &#39;#fafafa&#39;}}}%%\ngraph TD\n DEV[&#34;Developer&#34;]\n GIT[&#34;Git Repository&#34;]\n WEBHOOK[&#34;Push Event&#34;]\n PIPELINE[&#34;GitHub Actions&#34;]\n PG[&#34;PostgreSQL&#34;]\n CH[&#34;ClickHouse Cluster&#34;]\n\n DEV --&gt;|&#34;git push&#34;| GIT\n GIT --&gt;|&#34;branch event&#34;| WEBHOOK\n WEBHOOK --&gt;|&#34;triggers workflow&#34;| PIPELINE\n PIPELINE --&gt;|&#34;migrate up&#34;| PG\n PIPELINE --&gt;|&#34;migrate up&#34;| CH\n\n style DEV fill:#e3f2fd,stroke:#1565c0,stroke-width:2px,color:#000\n style GIT fill:#fce4ec,stroke:#c62828,stroke-width:2px,color:#000\n style WEBHOOK fill:#fff3e0,stroke:#e65100,stroke-width:2px,color:#000\n style PIPELINE fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px,color:#000\n style PG fill:#e8eaf6,stroke:#283593,stroke-width:2px,color:#000\n style CH fill:#fff8e1,stroke:#f57f17,stroke-width:2px,color:#000<\/code><\/pre><p>The key insight: <strong>migrations are just versioned SQL files in a git repository.<\/strong> The CI\/CD pipeline is the only thing that applies them to databases. No developer runs migrations manually against staging or production.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"repository-structure\"><span>Repository Structure<\/span>\n <a href=\"#repository-structure\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Here&rsquo;s what a mature migration project looks like:<\/p>\n<div class=\"file-tree-wrapper\"><ul class=\"file-tree\" data-level=\"0\"><li class=\"file-tree-item file-tree-folder\">\n <span class=\"file-tree-label file-tree-toggle\"><i class=\"file-tree-icon fa-regular fa-folder-open\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">db-migrations\/<\/span>\n <\/span><ul class=\"file-tree\" data-level=\"1\"><li class=\"file-tree-item file-tree-folder is-collapsed\">\n <span class=\"file-tree-label file-tree-toggle\"><i class=\"file-tree-icon fa-regular fa-folder-open\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">clickhouse\/<\/span>\n <\/span><ul class=\"file-tree\" data-level=\"2\"><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">001_create_core_tables.up.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">001_create_core_tables.down.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">002_seed_reference_data.up.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">002_seed_reference_data.down.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">003_create_merge_tables.up.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">003_create_merge_tables.down.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">004_create_parameterized_views.up.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">004_create_parameterized_views.down.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">005_add_analytics_tables.up.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">005_add_analytics_tables.down.sql<\/span>\n <\/span><\/li><\/ul><\/li><li class=\"file-tree-item file-tree-folder is-collapsed\">\n <span class=\"file-tree-label file-tree-toggle\"><i class=\"file-tree-icon fa-regular fa-folder-open\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">postgresql\/<\/span>\n <\/span><ul class=\"file-tree\" data-level=\"2\"><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">001_create_users_table.up.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">001_create_users_table.down.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">002_create_api_keys_table.up.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">002_create_api_keys_table.down.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">003_create_feedback_table.up.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">003_create_feedback_table.down.sql<\/span>\n <\/span><\/li><\/ul><\/li><li class=\"file-tree-item file-tree-folder is-collapsed\">\n <span class=\"file-tree-label file-tree-toggle\"><i class=\"file-tree-icon fa-regular fa-folder-open\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">.github\/<\/span>\n <\/span><ul class=\"file-tree\" data-level=\"2\"><li class=\"file-tree-item file-tree-folder is-collapsed\">\n <span class=\"file-tree-label file-tree-toggle\"><i class=\"file-tree-icon fa-regular fa-folder-open\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">workflows\/<\/span>\n <\/span><ul class=\"file-tree\" data-level=\"3\"><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">migrate.yml<\/span>\n <\/span><\/li><\/ul><\/li><\/ul><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">Makefile<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">.gitignore<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">README.md<\/span>\n <\/span><\/li><\/ul><\/li><\/ul><\/div>\n<p>Key decisions in this structure:<\/p>\n<ul>\n<li><strong>Separate directories per database engine<\/strong> \u2014 Each has independent versioning. ClickHouse might be at version 005 while PostgreSQL is at version 003.<\/li>\n<li><strong><code>.github\/workflows\/migrate.yml<\/code><\/strong> \u2014 GitHub Actions workflow that installs golang-migrate and runs migrations automatically on push.<\/li>\n<li><strong>No secrets in the repo<\/strong> \u2014 Database credentials are stored as <a href=\"https:\/\/docs.github.com\/en\/actions\/security-for-github-actions\/security-guides\/using-secrets-in-github-actions\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">GitHub Actions secrets<\/a> and injected as environment variables at runtime.<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"how-golang-migrate-works\"><span>How golang-migrate Works<\/span>\n <a href=\"#how-golang-migrate-works\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"migration-files\"><span>Migration Files<\/span>\n <a href=\"#migration-files\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Each migration is a pair of files \u2014 one to apply, one to reverse:<\/p>\n<ul>\n<li><strong><code>.up.sql<\/code><\/strong> \u2014 applies the change (moving forward)<\/li>\n<li><strong><code>.down.sql<\/code><\/strong> \u2014 reverses the change (rollback)<\/li>\n<\/ul>\n<p>The numeric prefix determines execution order. golang-migrate compares these numbers against what&rsquo;s stored in the database and applies only what&rsquo;s new.<\/p>\n<h3 class=\"heading-element\" id=\"the-schema_migrations-table\"><span>The schema_migrations Table<\/span>\n <a href=\"#the-schema_migrations-table\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>golang-migrate creates a single tracking table in each database:<\/p>\n<pre><code>CREATE TABLE schema_migrations (\n version BIGINT NOT NULL,\n dirty BOOLEAN NOT NULL\n);<\/code><\/pre><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Column<\/th>\n <th>Purpose<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><code>version<\/code><\/td>\n <td>Highest applied migration number<\/td>\n <\/tr>\n <tr>\n <td><code>dirty<\/code><\/td>\n <td><code>true<\/code> if a migration failed midway<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p>On each run, the tool checks: <em>&ldquo;What&rsquo;s the current version? Are there SQL files with a higher number? If yes, apply them in order.&rdquo;<\/em><\/p>\n<pre><code>%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: {&#39;primaryColor&#39;: &#39;#bbdefb&#39;, &#39;primaryTextColor&#39;: &#39;#000&#39;, &#39;lineColor&#39;: &#39;#1565c0&#39;}}}%%\ngraph LR\n FILES[&#34;SQL Files: 001 to 005&#34;]\n TABLE[&#34;schema_migrations: version=3&#34;]\n DIFF[&#34;Diff: 004, 005 pending&#34;]\n APPLY[&#34;Apply 004 and 005&#34;]\n DONE[&#34;version=5, dirty=false&#34;]\n\n FILES --&gt; DIFF\n TABLE --&gt; DIFF\n DIFF --&gt; APPLY --&gt; DONE\n\n style FILES fill:#e3f2fd,stroke:#1565c0,stroke-width:2px\n style TABLE fill:#fff3e0,stroke:#e65100,stroke-width:2px\n style DIFF fill:#fce4ec,stroke:#c62828,stroke-width:2px\n style APPLY fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px\n style DONE fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px<\/code><\/pre><h3 class=\"heading-element\" id=\"core-commands\"><span>Core Commands<\/span>\n <a href=\"#core-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Command<\/th>\n <th>What it does<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><code>migrate up<\/code><\/td>\n <td>Apply all pending <code>.up.sql<\/code> files in order<\/td>\n <\/tr>\n <tr>\n <td><code>migrate up N<\/code><\/td>\n <td>Apply only the next N migrations<\/td>\n <\/tr>\n <tr>\n <td><code>migrate down<\/code><\/td>\n <td>Apply <code>.down.sql<\/code> to rollback (I rarely use this in production)<\/td>\n <\/tr>\n <tr>\n <td><code>migrate version<\/code><\/td>\n <td>Show current applied version number<\/td>\n <\/tr>\n <tr>\n <td><code>migrate force V<\/code><\/td>\n <td>Reset version to V, clears dirty flag<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p>Basic CLI usage:<\/p>\n<pre><code># Apply all pending migrations\nmigrate -source file:\/\/.\/clickhouse \\\n -database &#34;clickhouse:\/\/host:9440?database=mydb&amp;username=user&amp;password=pass&amp;x-multi-statement=true&amp;secure=true&#34; \\\n up\n\n# Check current version\nmigrate -source file:\/\/.\/postgresql \\\n -database &#34;postgres:\/\/user:pass@host:5432\/mydb?sslmode=require&#34; \\\n version<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"branch-based-environment-routing\"><span>Branch-Based Environment Routing<\/span>\n <a href=\"#branch-based-environment-routing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>One pattern that worked well: <strong>the branch you push to determines which environment gets migrated.<\/strong><\/p>\n<pre><code>%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: {&#39;primaryColor&#39;: &#39;#e3f2fd&#39;, &#39;lineColor&#39;: &#39;#37474f&#39;}}}%%\ngraph TD\n subgraph &#34;Git Branches&#34;\n DEV_BRANCH[&#34;dev branch&#34;]\n MAIN_BRANCH[&#34;main branch&#34;]\n end\n\n subgraph &#34;Pipeline Decision&#34;\n DEV_ENV[&#34;ENV=dev, dev secrets&#34;]\n PROD_ENV[&#34;ENV=prod, prod secrets&#34;]\n end\n\n subgraph &#34;Target Databases&#34;\n DEV_CH[&#34;ClickHouse Dev&#34;]\n DEV_PG[&#34;PostgreSQL Dev&#34;]\n PROD_CH[&#34;ClickHouse Prod&#34;]\n PROD_PG[&#34;PostgreSQL Prod&#34;]\n end\n\n DEV_BRANCH --&gt;|&#34;push&#34;| DEV_ENV\n MAIN_BRANCH --&gt;|&#34;merge&#34;| PROD_ENV\n DEV_ENV --&gt; DEV_CH\n DEV_ENV --&gt; DEV_PG\n PROD_ENV --&gt; PROD_CH\n PROD_ENV --&gt; PROD_PG\n\n style DEV_BRANCH fill:#e3f2fd,stroke:#1565c0,stroke-width:2px\n style MAIN_BRANCH fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px\n style DEV_ENV fill:#e3f2fd,stroke:#1565c0,stroke-width:2px\n style PROD_ENV fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px\n style DEV_CH fill:#fff3e0,stroke:#e65100,stroke-width:2px\n style DEV_PG fill:#e8eaf6,stroke:#283593,stroke-width:2px\n style PROD_CH fill:#fff3e0,stroke:#e65100,stroke-width:2px\n style PROD_PG fill:#e8eaf6,stroke:#283593,stroke-width:2px<\/code><\/pre><p>With GitHub Actions, this is implemented using branch-conditional secrets:<\/p>\n<pre><code># In your workflow:\nenv:\n POSTGRES_URL: ${{ github.ref == &#39;refs\/heads\/main&#39; &amp;&amp; secrets.PG_PROD_URL || secrets.PG_DEV_URL }}\n CLICKHOUSE_URL: ${{ github.ref == &#39;refs\/heads\/main&#39; &amp;&amp; secrets.CH_PROD_URL || secrets.CH_DEV_URL }}<\/code><\/pre><p>Same migration SQL files, same workflow \u2014 only the connection URL changes based on which branch triggered the run.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"secrets-management--keep-it-simple\"><span>Secrets Management \u2014 Keep It Simple<\/span>\n <a href=\"#secrets-management--keep-it-simple\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Database credentials should <strong>never<\/strong> be in your repository. With modern CI\/CD platforms like <a href=\"https:\/\/docs.github.com\/en\/actions\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">GitHub Actions<\/a>, <a href=\"https:\/\/docs.gitlab.com\/ee\/ci\/variables\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">GitLab CI<\/a>, or <a href=\"https:\/\/circleci.com\/docs\/env-vars\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">CircleCI<\/a>, this is straightforward \u2014 store the full database URL as a secret and inject it at runtime.<\/p>\n<pre><code>%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: {&#39;primaryColor&#39;: &#39;#e8f5e9&#39;, &#39;lineColor&#39;: &#39;#2e7d32&#39;}}}%%\ngraph LR\n subgraph &#34;Your Repository&#34;\n CODE[&#34;SQL migration files&#34;]\n end\n\n subgraph &#34;CI\/CD Platform Secrets&#34;\n SEC[&#34;POSTGRES_URL, CLICKHOUSE_URL&#34;]\n end\n\n subgraph &#34;Runtime&#34;\n RUN[&#34;migrate -database $URL up&#34;]\n end\n\n CODE --&gt;|&#34;push&#34;| RUN\n SEC --&gt;|&#34;injected as env vars&#34;| RUN\n\n style CODE fill:#e3f2fd,stroke:#1565c0,stroke-width:2px\n style SEC fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px\n style RUN fill:#fff3e0,stroke:#e65100,stroke-width:2px<\/code><\/pre><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Where to store secrets<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><ul>\n<li><strong><a href=\"https:\/\/docs.github.com\/en\/actions\/security-for-github-actions\/security-guides\/using-secrets-in-github-actions\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">GitHub Actions Secrets<\/a><\/strong> \u2014 Encrypted, only exposed to workflows, never shown in logs<\/li>\n<li><strong><a href=\"https:\/\/www.vaultproject.io\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">HashiCorp Vault<\/a><\/strong> \u2014 For teams that need dynamic credentials, rotation, and audit trails<\/li>\n<li><strong><a href=\"https:\/\/aws.amazon.com\/secrets-manager\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">AWS Secrets Manager<\/a><\/strong> \/ <a href=\"https:\/\/cloud.google.com\/security\/products\/secret-manager\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">GCP Secret Manager<\/a> \u2014 Cloud-native secret storage<\/li>\n<li><strong><a href=\"https:\/\/www.doppler.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Doppler<\/a><\/strong> \u2014 Universal secrets manager that syncs across platforms<\/li>\n<\/ul>\n<p>The key rule: <strong>the database URL is assembled by the CI\/CD platform, not by your code.<\/strong> Your migration script just reads <code>$DATABASE_URL<\/code> from the environment.<\/p><\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 class=\"heading-element\" id=\"the-cicd-pipeline--github-actions\"><span>The CI\/CD Pipeline \u2014 GitHub Actions<\/span>\n <a href=\"#the-cicd-pipeline--github-actions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Here&rsquo;s a complete GitHub Actions workflow that runs migrations automatically on push:<\/p>\n<pre><code># .github\/workflows\/migrate.yml\nname: Database Migrations\n\non:\n push:\n branches: [main, dev]\n paths:\n - &#39;clickhouse\/**&#39;\n - &#39;postgresql\/**&#39;\n\nenv:\n MIGRATE_VERSION: &#34;v4.17.0&#34;\n\njobs:\n migrate:\n runs-on: ubuntu-latest\n steps:\n - uses: actions\/checkout@v4\n\n - name: Install golang-migrate\n run: |\n curl -L https:\/\/github.com\/golang-migrate\/migrate\/releases\/download\/$MIGRATE_VERSION\/migrate.linux-amd64.tar.gz | tar xz\n sudo mv migrate \/usr\/local\/bin\/\n\n - name: Run ClickHouse migrations\n run: migrate -source file:\/\/.\/clickhouse -database &#34;$CLICKHOUSE_URL&#34; up\n env:\n CLICKHOUSE_URL: ${{ github.ref == &#39;refs\/heads\/main&#39; &amp;&amp; secrets.CH_PROD_URL || secrets.CH_DEV_URL }}\n\n - name: Run PostgreSQL migrations\n run: migrate -source file:\/\/.\/postgresql -database &#34;$POSTGRES_URL&#34; up\n env:\n POSTGRES_URL: ${{ github.ref == &#39;refs\/heads\/main&#39; &amp;&amp; secrets.PG_PROD_URL || secrets.PG_DEV_URL }}<\/code><\/pre><pre><code>%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: {&#39;primaryColor&#39;: &#39;#f3e5f5&#39;, &#39;lineColor&#39;: &#39;#4a148c&#39;}}}%%\ngraph TD\n subgraph &#34;Pipeline Trigger&#34;\n PUSH[&#34;Git push \/ merge&#34;]\n end\n\n subgraph &#34;Setup&#34;\n CHECKOUT[&#34;Checkout repository&#34;]\n INSTALL[&#34;Install golang-migrate&#34;]\n end\n\n subgraph &#34;Run Migrations&#34;\n CH_RUN[&#34;migrate -database $CLICKHOUSE_URL up&#34;]\n PG_RUN[&#34;migrate -database $POSTGRES_URL up&#34;]\n end\n\n subgraph &#34;Result&#34;\n SUCCESS[&#34;Workflow passes \u2713&#34;]\n FAILURE[&#34;Workflow fails \u2717&#34;]\n end\n\n PUSH --&gt; CHECKOUT --&gt; INSTALL\n INSTALL --&gt; CH_RUN --&gt; PG_RUN\n PG_RUN --&gt;|&#34;exit 0&#34;| SUCCESS\n PG_RUN --&gt;|&#34;exit != 0&#34;| FAILURE\n CH_RUN --&gt;|&#34;exit != 0&#34;| FAILURE\n\n style PUSH fill:#e3f2fd,stroke:#1565c0,stroke-width:2px\n style CHECKOUT fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px\n style INSTALL fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px\n style CH_RUN fill:#fff3e0,stroke:#e65100,stroke-width:2px\n style PG_RUN fill:#e8eaf6,stroke:#283593,stroke-width:2px\n style SUCCESS fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px\n style FAILURE fill:#ffcdd2,stroke:#b71c1c,stroke-width:2px<\/code><\/pre><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Key details<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><ul>\n<li><strong>golang-migrate is downloaded from <a href=\"https:\/\/github.com\/golang-migrate\/migrate\/releases\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">GitHub Releases<\/a><\/strong> \u2014 fast, reliable, and you pin the exact version.<\/li>\n<li><strong>Branch-based routing<\/strong> via GitHub expression: <code>main<\/code> \u2192 production secrets, everything else \u2192 dev secrets.<\/li>\n<li><strong>Path filter<\/strong> (<code>paths: ['clickhouse\/**', 'postgresql\/**']<\/code>) \u2014 the workflow only runs when migration files actually change, not on every push.<\/li>\n<li><strong>Database URLs stored as <a href=\"https:\/\/docs.github.com\/en\/actions\/security-for-github-actions\/security-guides\/using-secrets-in-github-actions\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">GitHub Secrets<\/a><\/strong> \u2014 never exposed in logs, encrypted at rest by GitHub.<\/li>\n<\/ul>\n<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 class=\"heading-element\" id=\"clickhouse-migration-patterns\"><span>ClickHouse Migration Patterns<\/span>\n <a href=\"#clickhouse-migration-patterns\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>This section covers patterns specific to <a href=\"https:\/\/clickhouse.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ClickHouse<\/a> clustered deployments. These are the patterns that Flyway and Liquibase couldn&rsquo;t handle properly.<\/p>\n<h3 class=\"heading-element\" id=\"pattern-1-local--distributed-tables\"><span>Pattern 1: Local + Distributed Tables<\/span>\n <a href=\"#pattern-1-local--distributed-tables\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>In a sharded ClickHouse cluster, you always create tables in pairs \u2014 a <code>_local<\/code> table on each shard, and a <code>Distributed<\/code> table that routes queries across shards:<\/p>\n<pre><code>-- 001_create_events.up.sql\n\n-- Local table: exists on every shard, stores actual data\nCREATE TABLE IF NOT EXISTS analytics.events_local ON CLUSTER &#39;{cluster}&#39;\n(\n event_id UUID DEFAULT generateUUIDv4(),\n event_time DateTime64(3),\n user_id UInt64,\n event_type LowCardinality(String),\n properties String,\n site_key LowCardinality(String)\n)\nENGINE = ReplicatedMergeTree(&#39;\/clickhouse\/tables\/{shard}\/events_local&#39;, &#39;{replica}&#39;)\nPARTITION BY toYYYYMM(event_time)\nORDER BY (site_key, event_time, user_id)\nTTL event_time &#43; INTERVAL 2 YEAR;\n\n-- Distributed table: routes queries to all shards\nCREATE TABLE IF NOT EXISTS analytics.events ON CLUSTER &#39;{cluster}&#39;\nAS analytics.events_local\nENGINE = Distributed(&#39;{cluster}&#39;, &#39;analytics&#39;, &#39;events_local&#39;, cityHash64(site_key));<\/code><\/pre><pre><code>%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: {&#39;primaryColor&#39;: &#39;#fff3e0&#39;, &#39;lineColor&#39;: &#39;#e65100&#39;}}}%%\ngraph TD\n subgraph &#34;Distributed Table (Router)&#34;\n DIST[&#34;analytics.events - Distributed&#34;]\n end\n\n subgraph &#34;Shard 1&#34;\n L1[&#34;events_local&#34;]\n end\n subgraph &#34;Shard 2&#34;\n L2[&#34;events_local&#34;]\n end\n subgraph &#34;Shard 3&#34;\n L3[&#34;events_local&#34;]\n end\n subgraph &#34;Shard N...&#34;\n LN[&#34;events_local&#34;]\n end\n\n DIST --&gt;|&#34;site_key hash \u2192 shard 1&#34;| L1\n DIST --&gt;|&#34;site_key hash \u2192 shard 2&#34;| L2\n DIST --&gt;|&#34;site_key hash \u2192 shard 3&#34;| L3\n DIST --&gt;|&#34;site_key hash \u2192 shard N&#34;| LN\n\n style DIST fill:#fff3e0,stroke:#e65100,stroke-width:2px\n style L1 fill:#e3f2fd,stroke:#1565c0,stroke-width:2px\n style L2 fill:#e3f2fd,stroke:#1565c0,stroke-width:2px\n style L3 fill:#e3f2fd,stroke:#1565c0,stroke-width:2px\n style LN fill:#e3f2fd,stroke:#1565c0,stroke-width:2px<\/code><\/pre><h3 class=\"heading-element\" id=\"pattern-2-merge-tables-for-cross-database-reads\"><span>Pattern 2: Merge Tables for Cross-Database Reads<\/span>\n <a href=\"#pattern-2-merge-tables-for-cross-database-reads\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>When you need to read from multiple source databases without copying data, ClickHouse&rsquo;s <a href=\"https:\/\/clickhouse.com\/docs\/en\/engines\/table-engines\/special\/merge\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Merge engine<\/a> is incredibly powerful:<\/p>\n<pre><code>-- 003_create_merge_tables.up.sql\n\n-- Read from ALL databases matching the pattern\nDROP TABLE IF EXISTS analytics.all_run_data ON CLUSTER &#39;{cluster}&#39;;\nCREATE TABLE analytics.all_run_data ON CLUSTER &#39;{cluster}&#39;\n(\n run_id UUID,\n site_key LowCardinality(String),\n lot_id String,\n wafer_id String,\n program String,\n start_time DateTime64(3),\n end_time DateTime64(3),\n status LowCardinality(String)\n)\nENGINE = Merge(REGEXP(&#39;.*_source&#39;), &#39;^run_data$&#39;);<\/code><\/pre><p>This creates a <strong>virtual table<\/strong> that reads from every database matching <code>*_source<\/code> and every table named <code>run_data<\/code>. No data is copied \u2014 it&rsquo;s a live federated view across 30+ source databases.<\/p>\n<pre><code>%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: {&#39;primaryColor&#39;: &#39;#e8f5e9&#39;, &#39;lineColor&#39;: &#39;#2e7d32&#39;}}}%%\ngraph LR\n subgraph &#34;Merge Table (Virtual)&#34;\n MERGE[&#34;analytics.all_run_data - Merge Engine&#34;]\n end\n\n subgraph &#34;Source Databases (30&#43;)&#34;\n DB1[&#34;site_a_source.run_data&#34;]\n DB2[&#34;site_b_source.run_data&#34;]\n DB3[&#34;site_c_source.run_data&#34;]\n DBN[&#34;site_n_source.run_data&#34;]\n end\n\n MERGE --&gt; DB1\n MERGE --&gt; DB2\n MERGE --&gt; DB3\n MERGE --&gt; DBN\n\n style MERGE fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px\n style DB1 fill:#e3f2fd,stroke:#1565c0,stroke-width:1px\n style DB2 fill:#e3f2fd,stroke:#1565c0,stroke-width:1px\n style DB3 fill:#e3f2fd,stroke:#1565c0,stroke-width:1px\n style DBN fill:#e3f2fd,stroke:#1565c0,stroke-width:1px<\/code><\/pre><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Merge engine: no data duplication<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">The Merge engine stores <strong>zero data<\/strong>. It&rsquo;s purely metadata \u2014 column definitions and a regex pattern. When queried, it fans out reads to all matching tables in real-time. If an upstream database adds a new table matching the pattern, it&rsquo;s automatically included. This is perfect for multi-tenant or multi-site architectures where new data sources appear regularly.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"pattern-3-parameterized-views-clickhouse-231\"><span>Pattern 3: Parameterized Views (ClickHouse 23.1+)<\/span>\n <a href=\"#pattern-3-parameterized-views-clickhouse-231\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><a href=\"https:\/\/clickhouse.com\/docs\/en\/sql-reference\/statements\/create\/view#parameterized-view\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Parameterized views<\/a> let you create reusable query interfaces with runtime parameters:<\/p>\n<pre><code>-- 004_create_parameterized_views.up.sql\n\nCREATE OR REPLACE VIEW analytics.run_summary_view ON CLUSTER &#39;{cluster}&#39;\nAS SELECT\n _database AS source_db,\n run_id,\n ifNull(lot_id, &#39;&#39;) AS lot_id,\n ifNull(wafer_id, &#39;&#39;) AS wafer_id,\n ifNull(program, &#39;&#39;) AS program,\n start_time,\n end_time,\n dateDiff(&#39;second&#39;, start_time, end_time) AS duration_seconds,\n count(*) OVER (PARTITION BY lot_id) AS wafers_in_lot,\n row_number() OVER (PARTITION BY lot_id ORDER BY start_time) - 1 AS sequence_num\nFROM analytics.all_run_data\nWHERE _database = {site_db:String}\nORDER BY start_time DESC;<\/code><\/pre><p>The application calls: <code>SELECT * FROM analytics.run_summary_view(site_db='site_a_source')<\/code> \u2014 clean API contract, all complexity hidden in the view.<\/p>\n<h3 class=\"heading-element\" id=\"pattern-4-seed-data-migrations\"><span>Pattern 4: Seed Data Migrations<\/span>\n <a href=\"#pattern-4-seed-data-migrations\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Some migrations insert reference data rather than changing schema:<\/p>\n<pre><code>-- 002_seed_reference_data.up.sql\n\nINSERT INTO analytics.site_registry\n (site_key, display_name, region, country, enabled)\nVALUES\n (&#39;site_a&#39;, &#39;Austin Facility&#39;, &#39;americas&#39;, &#39;US&#39;, true),\n (&#39;site_b&#39;, &#39;Munich Facility&#39;, &#39;emea&#39;, &#39;DE&#39;, true),\n (&#39;site_c&#39;, &#39;Tokyo Facility&#39;, &#39;apac&#39;, &#39;JP&#39;, true),\n (&#39;site_d&#39;, &#39;Bangalore Facility&#39;, &#39;apac&#39;, &#39;IN&#39;, true);\n\nINSERT INTO analytics.bin_config (bin_number, bin_name, color, quality)\nVALUES\n (0, &#39;PASS&#39;, &#39;#4caf50&#39;, &#39;good&#39;),\n (1, &#39;FAIL_ELECTRICAL&#39;, &#39;#f44336&#39;, &#39;bad&#39;),\n (2, &#39;FAIL_PARAMETRIC&#39;, &#39;#ff9800&#39;, &#39;bad&#39;),\n (3, &#39;RETEST&#39;, &#39;#2196f3&#39;, &#39;retest&#39;);\n-- ... 256 entries<\/code><\/pre><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Seed data via Distributed tables<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">When inserting seed data in a sharded cluster, always INSERT via the <strong>Distributed<\/strong> table (not <code>_local<\/code>). The Distributed engine automatically routes each row to the correct shard based on the sharding key. If you insert directly into <code>_local<\/code>, the data only exists on one shard.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"multi-shard-version-broadcast\"><span>Multi-Shard Version Broadcast<\/span>\n <a href=\"#multi-shard-version-broadcast\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>One challenge with ClickHouse clusters: the <code>schema_migrations<\/code> table uses a plain <code>MergeTree<\/code> engine (not replicated), so after a migration runs, only one shard knows the current version. Load-balanced queries hitting other shards might report the wrong version.<\/p>\n<p>The solution: after each successful migration, broadcast the version to all shards via parallel HTTP requests:<\/p>\n<pre><code>%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: {&#39;primaryColor&#39;: &#39;#f3e5f5&#39;, &#39;lineColor&#39;: &#39;#6a1b9a&#39;}}}%%\ngraph TD\n MIGRATE[&#34;migrate up&#34;]\n VERSION[&#34;Read new version&#34;]\n BROADCAST[&#34;Broadcast to all shards&#34;]\n S1[&#34;Shard 1 \u2713&#34;]\n S2[&#34;Shard 2 \u2713&#34;]\n S3[&#34;Shard 3 \u2713&#34;]\n SN[&#34;Shard 20 \u2713&#34;]\n\n MIGRATE --&gt; VERSION --&gt; BROADCAST\n BROADCAST --&gt; S1\n BROADCAST --&gt; S2\n BROADCAST --&gt; S3\n BROADCAST --&gt; SN\n\n style MIGRATE fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px\n style VERSION fill:#e3f2fd,stroke:#1565c0,stroke-width:2px\n style BROADCAST fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px\n style S1 fill:#c8e6c9,stroke:#1b5e20,stroke-width:1px\n style S2 fill:#c8e6c9,stroke:#1b5e20,stroke-width:1px\n style S3 fill:#c8e6c9,stroke:#1b5e20,stroke-width:1px\n style SN fill:#c8e6c9,stroke:#1b5e20,stroke-width:1px<\/code><\/pre><div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Enterprise ClickHouse note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">If you&rsquo;re using <a href=\"https:\/\/clickhouse.com\/cloud\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ClickHouse Cloud<\/a> or enterprise ClickHouse with <a href=\"https:\/\/clickhouse.com\/docs\/en\/guides\/sre\/keeper\/clickhouse-keeper\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ClickHouse Keeper<\/a>, you can change the migrations table engine to <code>ReplicatedMergeTree<\/code> and delete the broadcast step entirely \u2014 replication handles it automatically.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 class=\"heading-element\" id=\"postgresql-migration-patterns\"><span>PostgreSQL Migration Patterns<\/span>\n <a href=\"#postgresql-migration-patterns\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p><a href=\"https:\/\/www.postgresql.org\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">PostgreSQL<\/a> is where migrations are most pleasant to work with, thanks to its unique support for <strong>transactional DDL<\/strong>.<\/p>\n<h3 class=\"heading-element\" id=\"transactional-ddl--postgresqls-superpower\"><span>Transactional DDL \u2014 PostgreSQL&rsquo;s Superpower<\/span>\n <a href=\"#transactional-ddl--postgresqls-superpower\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>PostgreSQL Transactional DDL<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p>PostgreSQL is one of the few databases that supports <strong>transactional DDL<\/strong> \u2014 meaning <code>CREATE TABLE<\/code>, <code>ALTER TABLE<\/code>, <code>DROP INDEX<\/code>, and most other DDL statements can all run inside a transaction. If any statement fails, the <strong>entire migration<\/strong> rolls back atomically. You never end up in a dirty state.<\/p>\n<p>This is a major advantage over ClickHouse (no DDL transactions) and MySQL (implicit commits on DDL).<\/p>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"example-users-table-with-indexes\"><span>Example: Users Table with Indexes<\/span>\n <a href=\"#example-users-table-with-indexes\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>-- 001_create_users_table.up.sql\nCREATE TABLE IF NOT EXISTS users (\n user_id VARCHAR(50) PRIMARY KEY,\n display_name VARCHAR(255),\n first_name VARCHAR(100),\n last_name VARCHAR(100),\n email VARCHAR(255),\n country VARCHAR(5), -- ISO 3166-1 alpha-2\n first_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),\n last_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW()\n);\n\nCREATE INDEX IF NOT EXISTS idx_users_email ON users(email);\nCREATE INDEX IF NOT EXISTS idx_users_last_seen ON users(last_seen_at);\n\nCOMMENT ON TABLE users IS &#39;Auto-populated on first JWT authentication&#39;;\nCOMMENT ON COLUMN users.user_id IS &#39;Unique identifier from JWT sub claim&#39;;<\/code><\/pre><h3 class=\"heading-element\" id=\"example-api-keys-with-partial-indexes\"><span>Example: API Keys with Partial Indexes<\/span>\n <a href=\"#example-api-keys-with-partial-indexes\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>-- 002_create_api_keys_table.up.sql\nCREATE TABLE IF NOT EXISTS api_keys (\n id UUID PRIMARY KEY DEFAULT gen_random_uuid(),\n team_name VARCHAR(100) NOT NULL,\n key_encrypted TEXT NOT NULL, -- AES-256-GCM encrypted\n key_prefix VARCHAR(8) NOT NULL, -- First 8 chars for fast lookup\n description TEXT,\n primary_owner VARCHAR(50) NOT NULL,\n secondary_owner VARCHAR(50),\n created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),\n expires_at TIMESTAMPTZ, -- NULL = never expires\n active BOOLEAN NOT NULL DEFAULT TRUE,\n CONSTRAINT uq_api_keys_prefix UNIQUE (key_prefix)\n);\n\n-- Partial index: only index active keys (hot path for auth lookups)\nCREATE INDEX IF NOT EXISTS idx_api_keys_active_prefix\n ON api_keys(key_prefix) WHERE active = TRUE;\n\nCREATE INDEX IF NOT EXISTS idx_api_keys_team ON api_keys(team_name);\nCREATE INDEX IF NOT EXISTS idx_api_keys_owner ON api_keys(primary_owner);<\/code><\/pre><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Partial indexes<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><a href=\"https:\/\/www.postgresql.org\/docs\/current\/indexes-partial.html\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Partial indexes<\/a> (<code>WHERE active = TRUE<\/code>) are a PostgreSQL feature that keeps the index small and fast. Only rows matching the condition are indexed \u2014 ideal for status columns where you almost always query one value.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"create-index-concurrently\"><span>CREATE INDEX CONCURRENTLY<\/span>\n <a href=\"#create-index-concurrently\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>For large tables in production, standard index creation locks the table. Use <code>CREATE INDEX CONCURRENTLY<\/code> to avoid this \u2014 but note it <strong>cannot run inside a transaction<\/strong>, so it needs its own migration file:<\/p>\n<pre><code>-- 004_add_events_index_concurrently.up.sql\n-- NOTE: This migration cannot be wrapped in a transaction\nCREATE INDEX CONCURRENTLY IF NOT EXISTS idx_events_timestamp\n ON events(created_at DESC);<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"advisory-locks--preventing-race-conditions\"><span>Advisory Locks \u2014 Preventing Race Conditions<\/span>\n <a href=\"#advisory-locks--preventing-race-conditions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>In a CI\/CD environment, it&rsquo;s possible for two pipelines to trigger simultaneously. If both try to run migrations at the same time, you get corrupted state.<\/p>\n<pre><code>%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: {&#39;primaryColor&#39;: &#39;#e8eaf6&#39;, &#39;lineColor&#39;: &#39;#283593&#39;, &#39;actorTextColor&#39;: &#39;#000&#39;, &#39;actorBkg&#39;: &#39;#e3f2fd&#39;, &#39;signalColor&#39;: &#39;#1565c0&#39;}}}%%\nsequenceDiagram\n participant P1 as Pipeline 1\n participant DB as PostgreSQL\n participant P2 as Pipeline 2\n\n rect rgb(232, 245, 233)\n P1-&gt;&gt;DB: pg_advisory_lock(migrate_hash)\n DB--&gt;&gt;P1: Lock GRANTED \u2713\n end\n\n rect rgb(255, 235, 238)\n P2-&gt;&gt;DB: pg_advisory_lock(migrate_hash)\n DB--&gt;&gt;P2: BLOCKED (waiting...)\n end\n\n rect rgb(232, 245, 233)\n P1-&gt;&gt;DB: Apply migration 005\n P1-&gt;&gt;DB: Apply migration 006\n P1-&gt;&gt;DB: pg_advisory_unlock()\n end\n\n rect rgb(227, 242, 253)\n DB--&gt;&gt;P2: Lock GRANTED (finally)\n P2-&gt;&gt;DB: Check version \u2192 already 006\n P2-&gt;&gt;DB: Nothing to apply \u2713\n P2-&gt;&gt;DB: pg_advisory_unlock()\n end<\/code><\/pre><p>golang-migrate handles this natively for PostgreSQL using <a href=\"https:\/\/www.postgresql.org\/docs\/current\/explicit-locking.html#ADVISORY-LOCKS\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">advisory locks<\/a>. Only one migration instance can run at a time per database. The second one waits (or fails fast, depending on configuration).<\/p>\n<p><strong>Key point:<\/strong> This is automatic for PostgreSQL. For ClickHouse and MySQL, you need to handle concurrency at the pipeline level (e.g., disallow concurrent builds in your CI\/CD config, or use a distributed lock via <a href=\"https:\/\/redis.io\/docs\/latest\/develop\/use\/patterns\/distributed-locks\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Redis<\/a> \/ <a href=\"https:\/\/developer.hashicorp.com\/consul\/docs\/dynamic-app-config\/sessions\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Consul<\/a>).<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"dirty-state-recovery\"><span>Dirty State Recovery<\/span>\n <a href=\"#dirty-state-recovery\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>This is the scenario that causes the most panic the first time you encounter it:<\/p>\n<pre><code>%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: {&#39;primaryColor&#39;: &#39;#e8f5e9&#39;, &#39;lineColor&#39;: &#39;#2e7d32&#39;}}}%%\nstateDiagram-v2\n [*] --&gt; Clean: Normal operation\n Clean --&gt; Migrating: migrate up triggered\n\n state Migrating {\n [*] --&gt; Applying\n Applying --&gt; Applied: SQL succeeds\n Applying --&gt; Failed: SQL error\n }\n\n Migrating --&gt; Clean: All migrations succeed\n Migrating --&gt; Dirty: Migration fails midway\n\n state Dirty {\n [*] --&gt; Blocked\n Blocked --&gt; Blocked: All migrate up attempts FAIL\n }\n\n Dirty --&gt; Recovery: Developer intervention\n Recovery --&gt; Clean: migrate force N &#43; fix SQL\n Clean --&gt; Migrating: Re-run migrate up<\/code><\/pre><p><strong>What happens:<\/strong><\/p>\n<ol>\n<li>Migration 005 starts executing<\/li>\n<li>SQL statement fails (syntax error, constraint violation, missing column, etc.)<\/li>\n<li>The database is now in an <strong>indeterminate state<\/strong> \u2014 some changes applied, some didn&rsquo;t<\/li>\n<li>golang-migrate sets <code>dirty = true<\/code> in <code>schema_migrations<\/code><\/li>\n<li>All future <code>migrate up<\/code> commands <strong>refuse to run<\/strong> until you resolve this<\/li>\n<\/ol>\n<p><strong>How to fix it:<\/strong><\/p>\n<pre><code>%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: {&#39;primaryColor&#39;: &#39;#fff3e0&#39;, &#39;lineColor&#39;: &#39;#e65100&#39;}}}%%\ngraph LR\n A[&#34;1. Check state&#34;]\n B[&#34;2. Inspect DB&#34;]\n C[&#34;3. Fix SQL file&#34;]\n D[&#34;4. Force version&#34;]\n E[&#34;5. Re-run migrate up&#34;]\n\n A --&gt; B --&gt; C --&gt; D --&gt; E\n\n style A fill:#ffcdd2,stroke:#c62828,stroke-width:2px\n style B fill:#fff3e0,stroke:#e65100,stroke-width:2px\n style C fill:#fff9c4,stroke:#f57f17,stroke-width:2px\n style D fill:#e3f2fd,stroke:#1565c0,stroke-width:2px\n style E fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px<\/code><\/pre><pre><code># 1. Check current state\nmigrate -source file:\/\/.\/migrations -database &#34;$URL&#34; version\n# Output: 5 (dirty)\n\n# 2-3. Fix the SQL file or manually clean up partial changes in the DB\n\n# 4. Force back to last good version\nmigrate -source file:\/\/.\/migrations -database &#34;$URL&#34; force 4\n\n# 5. Re-run (applies the fixed migration 005)\nmigrate -source file:\/\/.\/migrations -database &#34;$URL&#34; up<\/code><\/pre><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Prevention strategies<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><ul>\n<li><strong>PostgreSQL:<\/strong> Transactional DDL means a failed migration auto-rolls back. You rarely hit dirty state.<\/li>\n<li><strong>ClickHouse:<\/strong> No DDL transactions. Keep migrations atomic \u2014 <strong>one DDL statement per file<\/strong>. If it fails, nothing partial applied.<\/li>\n<li><strong>General:<\/strong> Always test migrations against a local Docker database before pushing.<\/li>\n<\/ul>\n<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 class=\"heading-element\" id=\"zero-downtime-migrations--the-expand-contract-pattern\"><span>Zero-Downtime Migrations \u2014 The Expand-Contract Pattern<\/span>\n <a href=\"#zero-downtime-migrations--the-expand-contract-pattern\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>This is the pattern I use when multiple services share a database and you need to change a column without downtime. It&rsquo;s sometimes called &ldquo;<a href=\"https:\/\/martinfowler.com\/bliki\/ParallelChange.html\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">expand and contract<\/a>&rdquo; or &ldquo;parallel change.&rdquo;<\/p>\n<p><strong>Scenario:<\/strong> Two services read\/write a <code>full_name<\/code> column. Service A now needs separate <code>first_name<\/code> and <code>last_name<\/code> columns.<\/p>\n<pre><code>%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: {&#39;primaryColor&#39;: &#39;#e3f2fd&#39;, &#39;lineColor&#39;: &#39;#1565c0&#39;}}}%%\ntimeline\n title Zero-Downtime Column Migration (Weeks, Not Minutes)\n section Phase 1 - Expand (Week 1)\n Migration 001 : ADD first_name, last_name (NULLABLE)\n Migration 002 : Backfill existing rows from full_name\n section Phase 2 - Transition (Week 2-3)\n Deploy Service A : Dual-write both old &#43; new columns\n : Reads from new columns\n Service B unchanged : Still writes only full_name\n section Phase 3 - Contract (Week 4)\n Deploy Service B : Now uses first_name &#43; last_name\n Migration 003 : Backfill NULLs from transition window\n Migration 004 : DROP full_name (only after ALL services migrated)<\/code><\/pre><pre><code>%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: {&#39;primaryColor&#39;: &#39;#e8f5e9&#39;, &#39;lineColor&#39;: &#39;#2e7d32&#39;}}}%%\ngraph TD\n subgraph &#34;Phase 1: EXPAND Schema&#34;\n M1[&#34;Migration 1: ADD new columns&#34;]\n M2[&#34;Migration 2: Backfill data&#34;]\n end\n\n subgraph &#34;Phase 2: TRANSITION (dual-write)&#34;\n S1[&#34;Deploy Service A: dual-write&#34;]\n S2[&#34;Service B unchanged: writes old column&#34;]\n end\n\n subgraph &#34;Phase 3: CONTRACT Schema&#34;\n S2B[&#34;Deploy Service B: uses new columns&#34;]\n M3[&#34;Migration 3: Backfill NULLs&#34;]\n M4[&#34;Migration 4: DROP old column&#34;]\n end\n\n M1 --&gt; M2 --&gt; S1\n S1 --&gt; S2\n S2 --&gt; S2B --&gt; M3 --&gt; M4\n\n style M1 fill:#bbdefb,stroke:#0d47a1,stroke-width:2px\n style M2 fill:#bbdefb,stroke:#0d47a1,stroke-width:2px\n style S1 fill:#fff3e0,stroke:#e65100,stroke-width:2px\n style S2 fill:#fff9c4,stroke:#f57f17,stroke-width:2px\n style S2B fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px\n style M3 fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px\n style M4 fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px<\/code><\/pre><div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>The mistake I made early on<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">I once dropped a column in the same deployment that switched the service to new columns. During the rolling deployment window, old pods were still reading the dropped column. Instant 500s across the fleet. The expand-contract pattern exists specifically to avoid this \u2014 you only contract (drop old columns) after <strong>ALL<\/strong> consumers have migrated and been stable for days.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 class=\"heading-element\" id=\"forward-only-migration-policy\"><span>Forward-Only Migration Policy<\/span>\n <a href=\"#forward-only-migration-policy\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>After a few production incidents, I adopted a strict <strong>forward-only<\/strong> policy:<\/p>\n<pre><code>%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: {&#39;primaryColor&#39;: &#39;#ffebee&#39;, &#39;lineColor&#39;: &#39;#c62828&#39;}}}%%\ngraph LR\n subgraph &#34;What I do&#34;\n FORWARD[&#34;Always move forward&#34;]\n end\n\n subgraph &#34;What I avoid&#34;\n ROLLBACK[&#34;Never roll back in prod&#34;]\n end\n\n FORWARD -.-|&#34;instead of&#34;| ROLLBACK\n\n style FORWARD fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px\n style ROLLBACK fill:#ffcdd2,stroke:#b71c1c,stroke-width:2px<\/code><\/pre><p><strong>Rules I follow:<\/strong><\/p>\n<ul>\n<li>I write <code>.down.sql<\/code> files for local dev\/testing only<\/li>\n<li>In production, I <strong>never<\/strong> run <code>migrate down<\/code><\/li>\n<li>If a migration was wrong, I write a <strong>new forward migration<\/strong> that corrects it<\/li>\n<li>All DDL uses <code>IF NOT EXISTS<\/code> \/ <code>IF EXISTS<\/code> \/ <code>CREATE OR REPLACE<\/code> for idempotency<\/li>\n<li>Migrations can be re-run safely (important for ClickHouse shard broadcast scenarios)<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"practical-tips-from-experience\"><span>Practical Tips from Experience<\/span>\n <a href=\"#practical-tips-from-experience\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"1-one-change-per-migration-file\"><span>1. One Change Per Migration File<\/span>\n <a href=\"#1-one-change-per-migration-file\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Don&rsquo;t pack multiple unrelated changes into one migration:<\/p>\n<div class=\"file-tree-wrapper\"><ul class=\"file-tree\" data-level=\"0\"><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">005_add_email_column.up.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">005_add_email_column.down.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">006_create_orders_table.up.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">006_create_orders_table.down.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">007_add_orders_user_index.up.sql<\/span>\n <\/span><\/li><li class=\"file-tree-item\">\n <span class=\"file-tree-label\"><i class=\"file-tree-icon fa-regular fa-file-lines\" aria-hidden=\"true\"><\/i><span class=\"file-tree-label__inner\">007_add_orders_user_index.down.sql<\/span>\n <\/span><\/li><\/ul><\/div>\n<p>If migration 005 has &ldquo;add email column&rdquo; AND &ldquo;create orders table&rdquo; and the orders creation fails, you can&rsquo;t easily recover the email column part.<\/p>\n<h3 class=\"heading-element\" id=\"2-make-migrations-idempotent\"><span>2. Make Migrations Idempotent<\/span>\n <a href=\"#2-make-migrations-idempotent\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>-- GOOD: Idempotent (safe to re-run)\nCREATE TABLE IF NOT EXISTS users (...);\nCREATE INDEX IF NOT EXISTS idx_users_email ON users(email);\nDROP TABLE IF EXISTS old_events ON CLUSTER &#39;{cluster}&#39;;\nCREATE OR REPLACE VIEW analytics.summary_view AS ...;\n\n-- BAD: Non-idempotent (fails if already applied)\nCREATE TABLE users (...);\nCREATE INDEX idx_users_email ON users(email);<\/code><\/pre><h3 class=\"heading-element\" id=\"3-test-against-real-databases-locally\"><span>3. Test Against Real Databases Locally<\/span>\n <a href=\"#3-test-against-real-databases-locally\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Don&rsquo;t just lint your SQL. Run your migrations against real instances via <a href=\"https:\/\/www.docker.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Docker<\/a>:<\/p>\n<pre><code># PostgreSQL\ndocker run -d --name pg-test -e POSTGRES_PASSWORD=test -p 5433:5432 postgres:16\nmigrate -source file:\/\/.\/postgresql \\\n -database &#34;postgres:\/\/postgres:test@localhost:5433\/postgres?sslmode=disable&#34; up\n\n# ClickHouse (single node, sufficient for DDL testing)\ndocker run -d --name ch-test -p 9000:9000 -p 8123:8123 clickhouse\/clickhouse-server:24.3\nmigrate -source file:\/\/.\/clickhouse \\\n -database &#34;clickhouse:\/\/localhost:9000\/default?x-multi-statement=true&#34; up<\/code><\/pre><h3 class=\"heading-element\" id=\"4-version-numbers-sequential-integers-vs-timestamps\"><span>4. Version Numbers: Sequential Integers vs. Timestamps<\/span>\n <a href=\"#4-version-numbers-sequential-integers-vs-timestamps\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>golang-migrate supports both. I prefer <strong>sequential integers<\/strong> (<code>001<\/code>, <code>002<\/code>, <code>003<\/code>) for smaller teams. Timestamps (<code>20260703120000<\/code>) are better when multiple developers create migrations on different branches \u2014 fewer merge conflicts.<\/p>\n<h3 class=\"heading-element\" id=\"5-document-the-why-in-sql-comments\"><span>5. Document the &ldquo;Why&rdquo; in SQL Comments<\/span>\n <a href=\"#5-document-the-why-in-sql-comments\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>-- 003_add_partition_key.up.sql\n-- WHY: Queries by month are 50x faster with monthly partitions.\n-- Old data (&gt;2 years) auto-drops via TTL without manual cleanup.\nALTER TABLE analytics.events_local ON CLUSTER &#39;{cluster}&#39;\n MODIFY TTL event_time &#43; INTERVAL 2 YEAR;<\/code><\/pre><h3 class=\"heading-element\" id=\"6-use-on-cluster-for-every-ddl-in-clickhouse\"><span>6. Use <code>ON CLUSTER<\/code> for EVERY DDL in ClickHouse<\/span>\n <a href=\"#6-use-on-cluster-for-every-ddl-in-clickhouse\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>If you forget <code>ON CLUSTER<\/code> on even one statement, you&rsquo;ll have schema drift between shards that&rsquo;s incredibly painful to debug.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"comparison-with-other-tools\"><span>Comparison with Other Tools<\/span>\n <a href=\"#comparison-with-other-tools\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Feature<\/th>\n <th><a href=\"https:\/\/github.com\/golang-migrate\/migrate\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">golang-migrate<\/a><\/th>\n <th><a href=\"https:\/\/www.red-gate.com\/products\/flyway\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Flyway<\/a><\/th>\n <th><a href=\"https:\/\/www.liquibase.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Liquibase<\/a><\/th>\n <th><a href=\"https:\/\/atlasgo.io\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Atlas<\/a><\/th>\n <th><a href=\"https:\/\/github.com\/pressly\/goose\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">goose<\/a><\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>Language<\/td>\n <td>Go (CLI + lib)<\/td>\n <td>Java<\/td>\n <td>Java\/XML<\/td>\n <td>Go<\/td>\n <td>Go<\/td>\n <\/tr>\n <tr>\n <td>Config overhead<\/td>\n <td>Zero (just SQL files)<\/td>\n <td>Medium<\/td>\n <td>High (XML\/YAML)<\/td>\n <td>Low (HCL)<\/td>\n <td>Low<\/td>\n <\/tr>\n <tr>\n <td>Database support<\/td>\n <td><a href=\"https:\/\/github.com\/golang-migrate\/migrate#databases\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">15+<\/a><\/td>\n <td>20+<\/td>\n <td>25+<\/td>\n <td>10+<\/td>\n <td>12+<\/td>\n <\/tr>\n <tr>\n <td>ClickHouse clusters<\/td>\n <td>Works natively<\/td>\n <td>Incomplete<\/td>\n <td>Buggy<\/td>\n <td>Limited<\/td>\n <td>Works<\/td>\n <\/tr>\n <tr>\n <td>PostgreSQL support<\/td>\n <td>Excellent<\/td>\n <td>Excellent<\/td>\n <td>Excellent<\/td>\n <td>Excellent<\/td>\n <td>Excellent<\/td>\n <\/tr>\n <tr>\n <td>Dirty state handling<\/td>\n <td><code>force<\/code> command<\/td>\n <td><code>repair<\/code> command<\/td>\n <td>Manual<\/td>\n <td>Built-in<\/td>\n <td>Built-in<\/td>\n <\/tr>\n <tr>\n <td>Advisory locks<\/td>\n <td>Yes (PostgreSQL)<\/td>\n <td>Yes<\/td>\n <td>Yes<\/td>\n <td>Yes<\/td>\n <td>Yes<\/td>\n <\/tr>\n <tr>\n <td>Embedded in Go apps<\/td>\n <td>Native<\/td>\n <td>No<\/td>\n <td>No<\/td>\n <td>Yes<\/td>\n <td>Yes<\/td>\n <\/tr>\n <tr>\n <td>Docker image<\/td>\n <td>Yes (17MB)<\/td>\n <td>Yes (300MB+)<\/td>\n <td>Yes (400MB+)<\/td>\n <td>Yes<\/td>\n <td>Yes<\/td>\n <\/tr>\n <tr>\n <td>JVM required<\/td>\n <td>No<\/td>\n <td>Yes<\/td>\n <td>Yes<\/td>\n <td>No<\/td>\n <td>No<\/td>\n <\/tr>\n <tr>\n <td>Learning curve<\/td>\n <td>Low<\/td>\n <td>Medium<\/td>\n <td>High<\/td>\n <td>Medium<\/td>\n <td>Low<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><hr>\n<h2 class=\"heading-element\" id=\"using-golang-migrate-as-a-go-library\"><span>Using golang-migrate as a Go Library<\/span>\n <a href=\"#using-golang-migrate-as-a-go-library\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Beyond the CLI, you can embed migrations directly in your Go application using Go&rsquo;s <a href=\"https:\/\/pkg.go.dev\/embed\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><code>embed<\/code><\/a> package:<\/p>\n<pre><code>package main\n\nimport (\n &#34;database\/sql&#34;\n &#34;embed&#34;\n &#34;log&#34;\n &#34;os&#34;\n\n &#34;github.com\/golang-migrate\/migrate\/v4&#34;\n &#34;github.com\/golang-migrate\/migrate\/v4\/database\/postgres&#34;\n &#34;github.com\/golang-migrate\/migrate\/v4\/source\/iofs&#34;\n _ &#34;github.com\/lib\/pq&#34;\n)\n\n\/\/go:embed migrations\/postgresql\/*.sql\nvar migrationFS embed.FS\n\nfunc main() {\n db, err := sql.Open(&#34;postgres&#34;, os.Getenv(&#34;DATABASE_URL&#34;))\n if err != nil {\n log.Fatal(err)\n }\n\n driver, err := postgres.WithInstance(db, &amp;postgres.Config{})\n if err != nil {\n log.Fatal(err)\n }\n\n source, err := iofs.New(migrationFS, &#34;migrations\/postgresql&#34;)\n if err != nil {\n log.Fatal(err)\n }\n\n m, err := migrate.NewWithInstance(&#34;iofs&#34;, source, &#34;postgres&#34;, driver)\n if err != nil {\n log.Fatal(err)\n }\n\n if err := m.Up(); err != nil &amp;&amp; err != migrate.ErrNoChange {\n log.Fatal(&#34;migration failed: &#34;, err)\n }\n\n log.Println(&#34;Migrations applied successfully&#34;)\n}<\/code><\/pre><p>This is useful for self-migrating applications \u2014 the binary carries its own migrations embedded at compile time. Deploy a new version and it auto-migrates on startup. No separate migration step needed.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"the-abstraction-layer-pattern\"><span>The Abstraction Layer Pattern<\/span>\n <a href=\"#the-abstraction-layer-pattern\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>One architectural pattern I&rsquo;ve found invaluable for analytics workloads: <strong>never let applications query source databases directly.<\/strong> Instead, build an abstraction layer via migrations:<\/p>\n<pre><code>%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: {&#39;primaryColor&#39;: &#39;#e8f5e9&#39;, &#39;lineColor&#39;: &#39;#2e7d32&#39;}}}%%\ngraph TD\n subgraph &#34;Applications (API Layer)&#34;\n APP[&#34;Your Service&#34;]\n end\n\n subgraph &#34;Abstraction Layer (Your Migrations)&#34;\n VIEWS[&#34;Parameterized Views&#34;]\n MERGE[&#34;Merge Tables&#34;]\n LOCAL[&#34;Owned Tables&#34;]\n end\n\n subgraph &#34;Source Systems (Not Yours)&#34;\n SRC1[&#34;Source DB 1&#34;]\n SRC2[&#34;Source DB 2&#34;]\n SRC3[&#34;Source DB N...&#34;]\n end\n\n APP --&gt; VIEWS\n APP --&gt; LOCAL\n VIEWS --&gt; MERGE\n MERGE --&gt; SRC1\n MERGE --&gt; SRC2\n MERGE --&gt; SRC3\n\n style APP fill:#e3f2fd,stroke:#1565c0,stroke-width:2px\n style VIEWS fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px\n style MERGE fill:#fff3e0,stroke:#e65100,stroke-width:2px\n style LOCAL fill:#e8eaf6,stroke:#283593,stroke-width:2px\n style SRC1 fill:#f5f5f5,stroke:#757575,stroke-width:1px\n style SRC2 fill:#f5f5f5,stroke:#757575,stroke-width:1px\n style SRC3 fill:#f5f5f5,stroke:#757575,stroke-width:1px<\/code><\/pre><p><strong>Why this matters:<\/strong><\/p>\n<ul>\n<li>If an upstream team renames a column, you update ONE migration (the Merge table definition) \u2014 your views and app code stay unchanged.<\/li>\n<li>The parameterized views add NULL safety (<code>ifNull(col, default)<\/code>), computed fields, and deduplication logic that every consumer needs.<\/li>\n<li>New source databases are automatically included via the regex pattern in Merge tables.<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"full-pipeline--end-to-end\"><span>Full Pipeline \u2014 End to End<\/span>\n <a href=\"#full-pipeline--end-to-end\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><pre><code>%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: {&#39;primaryColor&#39;: &#39;#f5f5f5&#39;, &#39;lineColor&#39;: &#39;#37474f&#39;}}}%%\ngraph TD\n subgraph &#34;Developer Workflow&#34;\n WRITE[&#34;Write SQL migration file&#34;]\n TEST[&#34;Test against Docker DB&#34;]\n PUSH[&#34;git push to dev&#34;]\n PR[&#34;Create PR to main&#34;]\n REVIEW[&#34;Code review&#34;]\n MERGE[&#34;Merge to main&#34;]\n end\n\n subgraph &#34;GitHub Actions Workflow&#34;\n TRIGGER[&#34;Workflow triggered&#34;]\n INSTALL[&#34;Install golang-migrate&#34;]\n CH_MIG[&#34;migrate -database $CH_URL up&#34;]\n PG_MIG[&#34;migrate -database $PG_URL up&#34;]\n RESULT[&#34;Pass \/ Fail notification&#34;]\n end\n\n subgraph &#34;Databases&#34;\n CH_CLUSTER[&#34;ClickHouse Cluster&#34;]\n PG_DB[&#34;PostgreSQL&#34;]\n end\n\n WRITE --&gt; TEST --&gt; PUSH\n PUSH --&gt;|&#34;dev env&#34;| TRIGGER\n PUSH -.-&gt;|&#34;later&#34;| PR --&gt; REVIEW --&gt; MERGE\n MERGE --&gt;|&#34;prod env&#34;| TRIGGER\n TRIGGER --&gt; INSTALL\n INSTALL --&gt; CH_MIG --&gt; PG_MIG --&gt; RESULT\n CH_MIG --&gt; CH_CLUSTER\n PG_MIG --&gt; PG_DB\n\n style WRITE fill:#e3f2fd,stroke:#1565c0,stroke-width:2px\n style TEST fill:#e3f2fd,stroke:#1565c0,stroke-width:2px\n style PUSH fill:#e3f2fd,stroke:#1565c0,stroke-width:2px\n style PR fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px\n style REVIEW fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px\n style MERGE fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px\n style TRIGGER fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px\n style INSTALL fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px\n style CH_MIG fill:#fff3e0,stroke:#e65100,stroke-width:2px\n style PG_MIG fill:#e8eaf6,stroke:#283593,stroke-width:2px\n style RESULT fill:#e0f2f1,stroke:#004d40,stroke-width:2px\n style CH_CLUSTER fill:#fff3e0,stroke:#e65100,stroke-width:2px\n style PG_DB fill:#e8eaf6,stroke:#283593,stroke-width:2px<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"key-takeaways\"><span>Key Takeaways<\/span>\n <a href=\"#key-takeaways\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><ol>\n<li><strong>Migrations are code.<\/strong> Treat them like application code \u2014 version controlled, reviewed, tested in CI.<\/li>\n<li><strong>The pipeline is the only actor.<\/strong> No human runs migrations manually against shared environments.<\/li>\n<li><strong>Advisory locks prevent races.<\/strong> Without them, concurrent pipelines will corrupt your migration state.<\/li>\n<li><strong>Dirty state is recoverable.<\/strong> Don&rsquo;t panic. Check the state, fix the SQL, force back, re-run.<\/li>\n<li><strong>Expand-contract for shared schemas.<\/strong> Never drop a column that another service still reads.<\/li>\n<li><strong>Never store credentials in the repo.<\/strong> Use your CI\/CD platform&rsquo;s secrets (GitHub Secrets, Vault, etc.) to inject database URLs at runtime.<\/li>\n<li><strong>One change per file.<\/strong> Granular migrations are easier to debug, easier to force-recover, easier to reason about.<\/li>\n<li><strong>Forward-only in production.<\/strong> Never roll back \u2014 always write a new migration to correct mistakes.<\/li>\n<li><strong>Test your migration tool against your actual database engines.<\/strong> What works for PostgreSQL might not work for ClickHouse clusters \u2014 I learned this the hard way with Flyway and Liquibase.<\/li>\n<li><strong>Build an abstraction layer.<\/strong> Merge tables + parameterized views insulate your app from upstream schema changes.<\/li>\n<\/ol>\n<hr>\n<h2 class=\"heading-element\" id=\"resources\"><span>Resources<\/span>\n <a href=\"#resources\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><ul>\n<li><a href=\"https:\/\/github.com\/golang-migrate\/migrate\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">golang-migrate<\/a> \u2014 18.7k stars, supports 15+ databases<\/li>\n<li><a href=\"https:\/\/github.com\/golang-migrate\/migrate\/blob\/master\/MIGRATIONS.md\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Migration Best Practices (golang-migrate docs)<\/a><\/li>\n<li><a href=\"https:\/\/www.postgresql.org\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">PostgreSQL<\/a> \u2014 Transactional DDL makes migrations significantly safer<\/li>\n<li><a href=\"https:\/\/www.postgresql.org\/docs\/current\/explicit-locking.html#ADVISORY-LOCKS\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">PostgreSQL Advisory Locks<\/a><\/li>\n<li><a href=\"https:\/\/www.postgresql.org\/docs\/current\/sql-createindex.html#SQL-CREATEINDEX-CONCURRENTLY\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">PostgreSQL CREATE INDEX CONCURRENTLY<\/a> \u2014 Non-blocking index creation for large tables<\/li>\n<li><a href=\"https:\/\/www.postgresql.org\/docs\/current\/indexes-partial.html\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">PostgreSQL Partial Indexes<\/a><\/li>\n<li><a href=\"https:\/\/martinfowler.com\/bliki\/ParallelChange.html\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Expand and Contract Pattern \u2014 Martin Fowler<\/a><\/li>\n<li><a href=\"https:\/\/clickhouse.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ClickHouse<\/a> \u2014 Columnar analytics database<\/li>\n<li><a href=\"https:\/\/clickhouse.com\/docs\/knowledgebase\/schema_migration_tools\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ClickHouse Schema Migration Tools<\/a> \u2014 Official comparison of migration tools for ClickHouse<\/li>\n<li><a href=\"https:\/\/clickhouse.com\/docs\/en\/engines\/table-engines\/special\/merge\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ClickHouse Merge Engine<\/a> \u2014 Virtual tables for cross-database reads<\/li>\n<li><a href=\"https:\/\/clickhouse.com\/docs\/en\/engines\/table-engines\/special\/distributed\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ClickHouse Distributed Engine<\/a> \u2014 Sharded query routing<\/li>\n<li><a href=\"https:\/\/clickhouse.com\/docs\/en\/sql-reference\/statements\/create\/view#parameterized-view\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ClickHouse Parameterized Views<\/a><\/li>\n<li><a href=\"https:\/\/clickhouse.com\/docs\/en\/guides\/sre\/keeper\/clickhouse-keeper\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ClickHouse Keeper<\/a> \u2014 ZooKeeper alternative for coordination<\/li>\n<li><a href=\"https:\/\/www.red-gate.com\/products\/flyway\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Flyway<\/a> \u2014 JVM-based migration tool (strong PostgreSQL\/MySQL support)<\/li>\n<li><a href=\"https:\/\/www.liquibase.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Liquibase<\/a> \u2014 XML\/YAML\/JSON changelog-based migrations<\/li>\n<li><a href=\"https:\/\/atlasgo.io\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Atlas<\/a> \u2014 Declarative schema management for Go<\/li>\n<li><a href=\"https:\/\/github.com\/pressly\/goose\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">goose<\/a> \u2014 Another Go migration tool (similar philosophy)<\/li>\n<li><a href=\"https:\/\/docs.github.com\/en\/actions\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">GitHub Actions<\/a> \u2014 CI\/CD platform for automating migration workflows<\/li>\n<li><a href=\"https:\/\/docs.github.com\/en\/actions\/security-for-github-actions\/security-guides\/using-secrets-in-github-actions\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">GitHub Actions Secrets<\/a> \u2014 Encrypted secret storage for database credentials<\/li>\n<li><a href=\"https:\/\/www.vaultproject.io\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">HashiCorp Vault<\/a> \u2014 Dynamic secrets management for enterprise teams<\/li>\n<li><a href=\"https:\/\/www.docker.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Docker<\/a> \u2014 Container runtime for testing migrations locally<\/li>\n<\/ul>"},{"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","link":"https:\/\/samir.pages.dev\/posts\/grafana-stack-vs-elastic-observability\/","pubDate":"Wed, 01 Apr 2026 00:00:00 +0000","author":"netproxy@duck.com (Samir)","guid":"https:\/\/samir.pages.dev\/posts\/grafana-stack-vs-elastic-observability\/","category":"Software-Engineering","description":"<img src=\"https:\/\/samir.pages.dev\/files\/grafana-vs-elastic.webp\" alt=\"featured image\" referrerpolicy=\"no-referrer\"><p>I needed to pick an observability platform for a self-hosted environment. Mix of modern and legacy hardware, on-prem CI\/CD, and a future requirement for self-hosted LLM integration. The two real contenders were the <strong>Grafana stack<\/strong> (Grafana + <a href=\"https:\/\/grafana.com\/oss\/loki\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Loki<\/a> + <a href=\"https:\/\/grafana.com\/oss\/mimir\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Mimir<\/a> + <a href=\"https:\/\/grafana.com\/oss\/tempo\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Tempo<\/a> + <a href=\"https:\/\/grafana.com\/oss\/alloy\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Alloy<\/a> + <a href=\"https:\/\/grafana.com\/oss\/pyroscope\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Pyroscope<\/a>) and the <strong>Elastic stack<\/strong> (<a href=\"https:\/\/www.elastic.co\/elasticsearch\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Elasticsearch<\/a> + <a href=\"https:\/\/www.elastic.co\/kibana\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Kibana<\/a> + Elastic Agent).<\/p>\n<p>I ran both side by side for a few months. These are my notes.<\/p>\n<div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Scope<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">Everything here is about <strong>self-managed, on-premises deployments<\/strong>. If you&rsquo;re using Elastic Cloud, the cost and ops picture changes a lot \u2014 managed services take care of scaling, patching, and node management for you.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 class=\"heading-element\" id=\"dashboards\"><span>Dashboards<\/span>\n <a href=\"#dashboards\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Grafana wins this one easily. The killer feature is multi-datasource dashboards \u2014 one panel pulling from Prometheus, another from Loki, another from Elasticsearch or a random REST API, all on the same dashboard. Kibana only talks to Elasticsearch. If you want a single pane of glass across multiple backends, Grafana is it.<\/p>\n<p>The chart library is also wider. Heatmaps, flame graphs (huge for profiling), geo maps, canvas panels (drag-and-drop layout), node graphs for topology views. Kibana has improved over the years, but it still feels more limited for custom visualization.<\/p>\n<p>The other big deal: <strong><a href=\"https:\/\/grafana.com\/docs\/grafana\/latest\/as-code\/observability-as-code\/git-sync\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Git Sync<\/a><\/strong> landed as GA in Grafana v13.0. The workflow is: edit a dashboard in the browser, hit save, it commits to a Git branch. Open a PR, get it reviewed, merge \u2014 Grafana auto-applies. Your entire dashboard history lives in Git with full diff capability. Roll back a bad change with <code>git revert<\/code>. Kibana has no equivalent \u2014 dashboards live in an internal Elasticsearch index with no version history you can meaningfully browse.<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th><\/th>\n <th>Grafana<\/th>\n <th>Kibana<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>Multi-datasource<\/td>\n <td>Yes \u2014 any backend<\/td>\n <td>No \u2014 Elasticsearch only<\/td>\n <\/tr>\n <tr>\n <td>Git Sync (bidirectional)<\/td>\n <td>GA in v13.0<\/td>\n <td>None<\/td>\n <\/tr>\n <tr>\n <td>Community templates<\/td>\n <td>Thousands on grafana.com<\/td>\n <td>Limited<\/td>\n <\/tr>\n <tr>\n <td>Chart variety<\/td>\n <td>Flame graphs, geo maps, heatmaps, canvas, node graphs<\/td>\n <td>Standard charts<\/td>\n <\/tr>\n <tr>\n <td>Embeddable<\/td>\n <td>Yes, via iframes or public dashboards<\/td>\n <td>Limited<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><hr>\n<h2 class=\"heading-element\" id=\"telemetry-collection-and-real-time-monitoring\"><span>Telemetry Collection and Real-Time Monitoring<\/span>\n <a href=\"#telemetry-collection-and-real-time-monitoring\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Tie on both fronts. Both support <a href=\"https:\/\/opentelemetry.io\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">OTLP<\/a> natively, so you can use a single standard for logs, metrics, and traces regardless of which backend stores them.<\/p>\n<p><a href=\"https:\/\/prometheus.io\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Prometheus<\/a> + Mimir handles metrics on the Grafana side \u2014 Prometheus for scraping and short-term, Mimir for horizontal scaling and long-term retention. Elastic has TSDB mode since v8.7 for efficient time-series storage.<\/p>\n<p>For distributed tracing, Grafana has Tempo (stores traces cheaply in object storage), Elastic has its APM (mature, integrated with the rest of the Elastic stack). Both are solid \u2014 pick whichever query language your team already knows. If your devs think in PromQL, go Grafana. If they&rsquo;re deep in Elasticsearch query DSL already, Elastic APM will feel natural.<\/p>\n<p>The real difference is in agent resource footprint:<\/p>\n<ul>\n<li><strong>Alloy<\/strong> (Grafana): Single Go binary. Scrapes Prometheus endpoints, tails log files, receives OTLP, forwards everything. In my testing, idle memory sat around 50-100 MB.<\/li>\n<li><strong>Elastic Agent<\/strong>: Java-based. Typically 512 MB to 1 GB+ idle. Same story for Logstash if you&rsquo;re using it in your pipeline.<\/li>\n<\/ul>\n<p>On a server with 4 GB total RAM, one Logstash instance eats 25% of your memory before it processes a single log line. Alloy barely registers. On modern hardware with 32+ GB, the gap doesn&rsquo;t matter. But if you&rsquo;re deploying collectors on older machines (and in an on-prem environment, you almost certainly are), this is worth thinking about.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"log-management--the-tricky-one\"><span>Log Management \u2014 The Tricky One<\/span>\n <a href=\"#log-management--the-tricky-one\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>This is where the two stacks make fundamentally different architectural bets. Neither is wrong \u2014 they just optimize for different things.<\/p>\n<h3 class=\"heading-element\" id=\"lokis-approach\"><span>Loki&rsquo;s Approach<\/span>\n <a href=\"#lokis-approach\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Loki indexes by <strong>labels only<\/strong> \u2014 things like <code>{host=&quot;web-01&quot;, app=&quot;nginx&quot;, env=&quot;prod&quot;}<\/code>. The actual log content is stored as compressed chunks but never indexed. That means:<\/p>\n<ul>\n<li>Storage overhead in my experience is roughly 0.5-1x raw volume (often smaller than raw because of compression)<\/li>\n<li>Queries on labels are lightning fast<\/li>\n<li>But searching <em>inside<\/em> log content is basically grep \u2014 <code>|= &quot;ERROR&quot;<\/code> works, but complex full-text queries across millions of lines get slow<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"elasticsearchs-approach\"><span>Elasticsearch&rsquo;s Approach<\/span>\n <a href=\"#elasticsearchs-approach\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Elasticsearch builds a <strong>full inverted index<\/strong> on every field of every log line. Every token is searchable. That means:<\/p>\n<ul>\n<li>Best-in-class full-text search \u2014 complex regex, aggregations, field-level queries all instant<\/li>\n<li>But roughly 3-6x storage overhead from the indexing (varies with mapping and replica config)<\/li>\n<li>8-16 GB JVM heap minimum for a production cluster<\/li>\n<li>Expensive to scale if you have high log volume<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"what-i-actually-did\"><span>What I Actually Did<\/span>\n <a href=\"#what-i-actually-did\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>I ended up running <strong>both<\/strong>. Loki handles day-to-day ops \u2014 dashboards showing recent logs by service, alerting on error patterns, tailing live logs. Elasticsearch sits behind Grafana as a datasource for when I actually need to do forensic deep-dive searches \u2014 &ldquo;find every request with this specific trace ID across all services from last Tuesday.&rdquo;<\/p>\n<p>Grafana queries Elasticsearch natively, so you don&rsquo;t need Kibana at all. One UI, two log backends, each playing to its strength.<\/p>\n<pre><code>graph TD\n G[Grafana UI - Single Pane] --&gt; L[Loki]\n G --&gt; E[Elasticsearch]\n L -.- D[Daily Ops: dashboards, alerts, log tails]\n E -.- F[Forensic Search: full-text, deep-dive queries]<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"alerting\"><span>Alerting<\/span>\n <a href=\"#alerting\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Grafana wins. The whole <a href=\"https:\/\/grafana.com\/docs\/grafana\/latest\/alerting\/set-up\/provision-alerting-resources\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">alerting system is provisionable<\/a> as YAML or Terraform \u2014 alert rules, contact points, notification policies, silence rules. All of it. Drop a YAML file, hit the API, changes apply immediately. No restart.<\/p>\n<p>What makes this different from Elastic:<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th><\/th>\n <th>Grafana<\/th>\n <th>Elastic<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>Config as YAML\/Terraform<\/td>\n <td>Yes<\/td>\n <td>No<\/td>\n <\/tr>\n <tr>\n <td>Hot-reload (no restart)<\/td>\n <td>Yes<\/td>\n <td>No \u2014 restart needed<\/td>\n <\/tr>\n <tr>\n <td>Multi-datasource rules<\/td>\n <td>Yes<\/td>\n <td>No \u2014 Elasticsearch data only<\/td>\n <\/tr>\n <tr>\n <td>Alertmanager-compatible routing<\/td>\n <td>Yes<\/td>\n <td>No<\/td>\n <\/tr>\n <tr>\n <td>Native Webex webhook<\/td>\n <td>Yes<\/td>\n <td>No<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p>The multi-datasource alerting is the one that really changed things for me. You can write a single alert rule that says: &ldquo;fire if CPU from Prometheus is above 90% for 5 minutes AND the word OOM appears in Loki logs from the same host.&rdquo; Cross-signal correlation in one rule. Kibana can only alert on Elasticsearch data.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"gitops\"><span>GitOps<\/span>\n <a href=\"#gitops\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>This is where Grafana&rsquo;s lead is biggest. Here&rsquo;s what you can store as code and manage through Git:<\/p>\n<ul>\n<li><strong>Dashboards<\/strong> \u2014 JSON files, provisionable via Git Sync, ConfigMap, or Terraform<\/li>\n<li><strong>Datasources<\/strong> \u2014 YAML files dropped into a provisioning directory<\/li>\n<li><strong>Alert rules<\/strong> \u2014 YAML, same directory<\/li>\n<li><strong>Contact points and routing<\/strong> \u2014 YAML<\/li>\n<li><strong>Folders, teams, permissions<\/strong> \u2014 via the <a href=\"https:\/\/registry.terraform.io\/providers\/grafana\/grafana\/latest\/docs\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Terraform provider<\/a><\/li>\n<\/ul>\n<p>Everything is text. You can <code>grep<\/code> it, <code>diff<\/code> it, put it in a PR, run linting in CI. Standard software engineering workflow applied to your monitoring config.<\/p>\n<p>Kibana? Saved objects export exists, but it&rsquo;s a manual JSON dump. The format is proprietary \u2014 good luck reading a Kibana dashboard export and figuring out what changed between two versions. There&rsquo;s no community tooling ecosystem around it. No Terraform provider for Kibana dashboards. No Git sync. If you treat infrastructure-as-code as non-negotiable, the choice is clear.<\/p>\n<div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Lock-in Signal<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">If you can&rsquo;t <code>git diff<\/code> your monitoring config, you can&rsquo;t review it, test it in staging, or roll it back safely. That&rsquo;s a red flag for any production system.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 class=\"heading-element\" id=\"ml-and-anomaly-detection\"><span>ML and Anomaly Detection<\/span>\n <a href=\"#ml-and-anomaly-detection\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Elastic has <a href=\"https:\/\/www.elastic.co\/guide\/en\/machine-learning\/current\/ml-overview.html\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">built-in ML<\/a> \u2014 anomaly detection on time-series data, log categorization (auto-grouping similar log messages), forecasting, and root cause analysis. It works out of the box, no external dependencies. The downside: ML jobs run on dedicated ML nodes that need Platinum license, plus separate servers with 8-16 GB RAM each. Those ML nodes don&rsquo;t share resources with your data nodes.<\/p>\n<p>Grafana has no built-in ML at all. But if you already run <a href=\"https:\/\/mlflow.org\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">MLflow<\/a> (pretty common in teams doing any model tracking or experiment management), you can point Grafana&rsquo;s HTTP datasource at MLflow&rsquo;s REST API. Query experiment metrics, model accuracy over time, prediction outputs \u2014 all visualized in Grafana dashboards alongside your infrastructure metrics. No extra subscription, no dedicated nodes.<\/p>\n<p>Bottom line: if you already have MLflow or another ML platform in your stack, Grafana gives you unified visibility without additional cost. If you have nothing and want anomaly detection with zero setup, Elastic&rsquo;s built-in stuff works well \u2014 just factor in the node cost and licensing.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"on-prem-llm-integration\"><span>On-Prem LLM Integration<\/span>\n <a href=\"#on-prem-llm-integration\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Grafana wins here for a simple reason: no license gating. The HTTP datasource queries any endpoint that speaks HTTP. Point it at Ollama, vLLM, a local OpenAI-compatible API, or a routing proxy like LiteLLM. Build whatever workflow you want \u2014 summarize alerts with an LLM, auto-generate runbooks from incident patterns, query knowledge bases. All in the OSS tier.<\/p>\n<p>Elastic&rsquo;s AI features have been in flux. Their managed AI capabilities require Cloud Connect (an Elastic Cloud account), which defeats the purpose of an air-gapped on-prem setup. If you can&rsquo;t phone home to Elastic&rsquo;s cloud, those features aren&rsquo;t available to you. Check their current docs for the latest on what&rsquo;s available without cloud connectivity \u2014 this space is moving fast.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"automated-remediation\"><span>Automated Remediation<\/span>\n <a href=\"#automated-remediation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>The pattern that works well with Grafana for closing the &ldquo;detect \u2192 act&rdquo; loop:<\/p>\n<pre><code>graph LR\n A[Alert Fires] --&gt; B[Grafana Webhook]\n B --&gt; C[Agent\/Script Diagnoses]\n C --&gt; D[POST to Annotations API]\n D --&gt; E[Annotation on Dashboard]<\/code><\/pre><p>Per-alert-policy webhooks fire to any HTTP endpoint. The agent can be a Python script, an n8n workflow, a custom FastAPI service \u2014 whatever you want. Results post back as annotations so your team can see what happened and when, right on the dashboard that triggered the alert.<\/p>\n<p>For the SSH execution layer (actually running remediation commands on servers), I looked at four tools:<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Tool<\/th>\n <th>Runtime<\/th>\n <th>Grafana integration<\/th>\n <th>Best for<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong><a href=\"https:\/\/www.rundeck.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Rundeck<\/a><\/strong><\/td>\n <td>Java (~512 MB)<\/td>\n <td>Native webhook receiver<\/td>\n <td>Ops teams, ad hoc tasks, simple runbooks<\/td>\n <\/tr>\n <tr>\n <td><strong>Ansible + <a href=\"https:\/\/github.com\/semaphoreui\/semaphore\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Semaphore<\/a><\/strong><\/td>\n <td>Go (~100-200 MB)<\/td>\n <td>API trigger via webhook<\/td>\n <td>Complex multi-step playbooks<\/td>\n <\/tr>\n <tr>\n <td><strong>StackStorm<\/strong><\/td>\n <td>Multiple services (~4-8 GB)<\/td>\n <td>Native webhook sensor<\/td>\n <td>Event-driven automation at scale<\/td>\n <\/tr>\n <tr>\n <td><strong>SaltStack<\/strong><\/td>\n <td>Agent-based<\/td>\n <td>Manual integration<\/td>\n <td>Very large fleets (1000+ nodes)<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p>Rundeck is the easiest to get running. Single process, Apache 2.0 license, built-in web UI for building runbooks. For most teams starting out, this is the right answer. If your automation outgrows it (complex multi-host playbooks, conditional logic, rolling updates), Ansible + Semaphore UI gives you the richest module ecosystem with a lightweight Go-based web frontend.<\/p>\n<p>StackStorm is overkill unless you specifically need its event-rule engine. SaltStack&rsquo;s community has been declining since the VMware acquisition \u2014 I wouldn&rsquo;t start a new project with it today.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"resource-footprint\"><span>Resource Footprint<\/span>\n <a href=\"#resource-footprint\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>The numbers that convinced me. For 100 GB\/day of log ingestion:<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Metric<\/th>\n <th>Loki<\/th>\n <th>Elasticsearch<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>Storage per day<\/td>\n <td>50-100 GB<\/td>\n <td>300-600 GB<\/td>\n <\/tr>\n <tr>\n <td>Storage per year<\/td>\n <td>18-37 TB<\/td>\n <td>109-219 TB<\/td>\n <\/tr>\n <tr>\n <td>Indexing approach<\/td>\n <td>Label-only (compressed chunks)<\/td>\n <td>Full inverted index<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p>At one year retention, you&rsquo;re looking at roughly <strong>3-6x more disk<\/strong> on the Elastic side depending on your config. On commodity NVMe storage at ~$100\/TB, that&rsquo;s a real budget line item.<\/p>\n<p>For the full stack RAM comparison:<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Component<\/th>\n <th>Grafana Stack<\/th>\n <th>Elastic Stack<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>Metrics (Mimir)<\/td>\n <td>2-4 GB<\/td>\n <td>\u2014<\/td>\n <\/tr>\n <tr>\n <td>Logs (Loki)<\/td>\n <td>2-4 GB<\/td>\n <td>\u2014<\/td>\n <\/tr>\n <tr>\n <td>Traces (Tempo)<\/td>\n <td>1-2 GB<\/td>\n <td>\u2014<\/td>\n <\/tr>\n <tr>\n <td>Dashboard (Grafana)<\/td>\n <td>512 MB-1 GB<\/td>\n <td>\u2014<\/td>\n <\/tr>\n <tr>\n <td>Search + Storage (Elasticsearch)<\/td>\n <td>\u2014<\/td>\n <td>8-16 GB JVM heap<\/td>\n <\/tr>\n <tr>\n <td>Dashboard (Kibana)<\/td>\n <td>\u2014<\/td>\n <td>1-2 GB<\/td>\n <\/tr>\n <tr>\n <td><strong>Total<\/strong><\/td>\n <td><strong>~6-11 GB<\/strong><\/td>\n <td><strong>~9-18 GB<\/strong><\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p>The RAM difference is smaller than the storage difference, but Elasticsearch&rsquo;s JVM heap is fixed allocation \u2014 you&rsquo;re paying that 8-16 GB whether you&rsquo;re querying or idle. Loki and Mimir scale more linearly with actual load.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"pricing\"><span>Pricing<\/span>\n <a href=\"#pricing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>The core Grafana OSS stack (Grafana, Loki, Mimir, Tempo) is AGPLv3 \u2014 <strong>free<\/strong>. Alloy is Apache 2.0. All of it. Git Sync, multi-datasource alerting, Terraform provisioning, OTLP collection, distributed tracing, profiling. Enterprise only gates two things: SAML\/OIDC SSO and fine-grained folder-level RBAC.<\/p>\n<p>Elastic&rsquo;s licensing is more layered:<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Feature<\/th>\n <th>Required License<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>Core search and analytics<\/td>\n <td>Free (Basic)<\/td>\n <\/tr>\n <tr>\n <td>ML \/ Anomaly detection<\/td>\n <td>Platinum<\/td>\n <\/tr>\n <tr>\n <td>AIOps \/ AI Assistant<\/td>\n <td>Enterprise<\/td>\n <\/tr>\n <tr>\n <td>SAML SSO<\/td>\n <td>Platinum<\/td>\n <\/tr>\n <tr>\n <td>Field-level security<\/td>\n <td>Platinum<\/td>\n <\/tr>\n <tr>\n <td>Dedicated ML nodes<\/td>\n <td>Platinum (plus the server hardware)<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p>So if you need anomaly detection on Elastic, the real cost is: Platinum subscription + new servers for ML nodes (8-16 GB RAM each, separate from data nodes). On Grafana&rsquo;s side, the same outcome costs: MLflow (free) + HTTP datasource (free) + your existing compute.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"vendor-lock-in\"><span>Vendor Lock-in<\/span>\n <a href=\"#vendor-lock-in\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>This is the one people underestimate until they need to migrate.<\/p>\n<p><strong>Grafana exit path:<\/strong><\/p>\n<ul>\n<li>Run <code>git clone<\/code> on your dashboard\/alerting repos. Done.<\/li>\n<li>All dashboards are JSON, all config is YAML<\/li>\n<li>Downgrading from Enterprise to OSS only loses SAML SSO<\/li>\n<li>Your data stays in Prometheus\/Mimir\/Loki \u2014 none of which are Grafana-proprietary<\/li>\n<\/ul>\n<p><strong>Elastic exit path:<\/strong><\/p>\n<ul>\n<li>Kibana dashboards are stored in an internal <code>.kibana<\/code> index. The format is proprietary and deeply nested.<\/li>\n<li>No Git sync means no history, no diff, no audit trail<\/li>\n<li>Dropping from Platinum to Basic loses ML, AIOps, SAML SSO, and field-level security<\/li>\n<li>Rebuilding your Kibana dashboards in another tool means starting from scratch \u2014 there&rsquo;s no export format that other tools understand<\/li>\n<\/ul>\n<p>That asymmetry is real. Grafana&rsquo;s entire configuration surface is text files in version control. Elastic&rsquo;s is locked in a database you can&rsquo;t meaningfully read or port.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"scorecard\"><span>Scorecard<\/span>\n <a href=\"#scorecard\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Dimension<\/th>\n <th>Winner<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>Dashboards<\/td>\n <td><strong>Grafana<\/strong><\/td>\n <\/tr>\n <tr>\n <td>Telemetry Collection<\/td>\n <td>Tie<\/td>\n <\/tr>\n <tr>\n <td>Log Search (full-text)<\/td>\n <td><strong>Elastic<\/strong><\/td>\n <\/tr>\n <tr>\n <td>Log Storage Efficiency<\/td>\n <td><strong>Grafana<\/strong><\/td>\n <\/tr>\n <tr>\n <td>Real-Time Monitoring<\/td>\n <td>Tie<\/td>\n <\/tr>\n <tr>\n <td>Alerting<\/td>\n <td><strong>Grafana<\/strong><\/td>\n <\/tr>\n <tr>\n <td>GitOps<\/td>\n <td><strong>Grafana<\/strong><\/td>\n <\/tr>\n <tr>\n <td>Built-in ML<\/td>\n <td><strong>Elastic<\/strong><\/td>\n <\/tr>\n <tr>\n <td>MLflow Integration<\/td>\n <td><strong>Grafana<\/strong><\/td>\n <\/tr>\n <tr>\n <td>On-Prem LLM<\/td>\n <td><strong>Grafana<\/strong><\/td>\n <\/tr>\n <tr>\n <td>Automation<\/td>\n <td><strong>Grafana<\/strong><\/td>\n <\/tr>\n <tr>\n <td>Resource Efficiency<\/td>\n <td><strong>Grafana<\/strong><\/td>\n <\/tr>\n <tr>\n <td>Pricing<\/td>\n <td><strong>Grafana<\/strong><\/td>\n <\/tr>\n <tr>\n <td>Vendor Lock-in<\/td>\n <td><strong>Grafana<\/strong><\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Grafana 10 \u2014 Elastic 2 \u2014 Tie 2<\/strong><\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"what-i-went-with\"><span>What I Went With<\/span>\n <a href=\"#what-i-went-with\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p><strong>Primary<\/strong>: Grafana + Prometheus + Mimir (metrics), Loki (logs), Tempo (traces), Alloy (collection), Pyroscope (profiling).<\/p>\n<p><strong>Hybrid<\/strong>: Elasticsearch as a Grafana datasource for full-text log search. No Kibana.<\/p>\n<p><strong>Automation<\/strong>: Rundeck for runbook automation, with a path to Ansible + Semaphore if the playbooks get complex enough.<\/p>\n<p>Elastic&rsquo;s full-text search is still the best in the industry. I just don&rsquo;t think it should be your primary observability UI anymore. Use it as a specialized backend that Grafana queries \u2014 you get the best of both worlds without the lock-in, the JVM overhead as your primary stack, or the Kibana maintenance burden.<\/p>\n<hr>\n<div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Disclaimer<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">For educational purposes. Based on my own experience and publicly available docs. I have no affiliation with Grafana Labs, Elastic, or any vendor mentioned here. Your mileage may vary.<\/div>\n <\/div>\n<\/div>"},{"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","link":"https:\/\/samir.pages.dev\/posts\/vector-databases-rag-llm\/","pubDate":"Wed, 10 Dec 2025 00:00:00 +0000","author":"netproxy@duck.com (Samir)","guid":"https:\/\/samir.pages.dev\/posts\/vector-databases-rag-llm\/","category":"AI-ML","description":"<img src=\"https:\/\/samir.pages.dev\/files\/rag-flow-animation.gif\" alt=\"featured image\" referrerpolicy=\"no-referrer\"><p><strong>A Practical Guide to Building Knowledge-Enhanced AI Applications<\/strong><\/p>\n<hr>\n<h2 id=\"introduction\">1. Introduction to Vector Databases<\/h2>\r\n<h3 class=\"heading-element\" id=\"what-is-a-vector-database\"><span>What is a Vector Database?<\/span>\n <a href=\"#what-is-a-vector-database\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>A <strong>vector database<\/strong> is a specialized database designed to store and query high-dimensional vector embeddings efficiently. Unlike traditional databases that excel at exact matches (&ldquo;find users named John&rdquo;), vector databases excel at <strong>similarity searches<\/strong> (&ldquo;find documents semantically similar to this query&rdquo;).<\/p>\n<div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Why This Matters<\/strong>: Vector databases are the backbone of modern AI applications\u2014enabling semantic search, recommendation systems, and knowledge-augmented LLMs.<\/div>\n <\/div>\n<\/div>\n<p><strong>Key Characteristics:<\/strong><\/p>\n<ul>\n<li><strong>Similarity Search<\/strong>: Find the most similar items based on vector distance<\/li>\n<li><strong>Scalability<\/strong>: Handle billions of vectors with sub-second query times<\/li>\n<li><strong>AI-Native<\/strong>: Optimized for machine learning pipelines<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"why-llms-need-vector-databases\"><span>Why LLMs Need Vector Databases<\/span>\n <a href=\"#why-llms-need-vector-databases\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Large Language Models (GPT-4, Claude, Llama) have critical limitations that vector databases solve:<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>LLM Limitation<\/th>\n <th>Vector DB Solution<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Knowledge Cutoff<\/strong><\/td>\n <td>Store and retrieve current information<\/td>\n <\/tr>\n <tr>\n <td><strong>Hallucinations<\/strong><\/td>\n <td>Ground responses in factual data<\/td>\n <\/tr>\n <tr>\n <td><strong>No Long-Term Memory<\/strong><\/td>\n <td>Persist context across sessions<\/td>\n <\/tr>\n <tr>\n <td><strong>Token Limits<\/strong><\/td>\n <td>Retrieve only relevant information<\/td>\n <\/tr>\n <tr>\n <td><strong>Lacks Domain Knowledge<\/strong><\/td>\n <td>Inject specialized expertise<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"real-world-applications\"><span>Real-World Applications<\/span>\n <a href=\"#real-world-applications\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Vector databases power many AI applications you use daily:<\/p>\n<ul>\n<li><strong>Customer Support Bots<\/strong>: Retrieve relevant help articles to answer user questions<\/li>\n<li><strong>Enterprise Search<\/strong>: Find documents by meaning, not just keywords<\/li>\n<li><strong>Code Assistants<\/strong>: Search codebases for similar implementations<\/li>\n<li><strong>Research Tools<\/strong>: Find related papers and citations<\/li>\n<li><strong>Recommendation Systems<\/strong>: &ldquo;Users who liked X also liked Y&rdquo;<\/li>\n<li><strong>Chatbots with Memory<\/strong>: Remember past conversations<\/li>\n<\/ul>\n<hr>\n<h2 id=\"embeddings\">2. Understanding Embeddings<\/h2>\r\n<h3 class=\"heading-element\" id=\"what-are-embeddings\"><span>What are Embeddings?<\/span>\n <a href=\"#what-are-embeddings\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Embeddings<\/strong> are numerical representations of data (text, images, etc.) as dense vectors of floating-point numbers. These vectors capture semantic meaning in a way machines can process and compare.<\/p>\n<p><strong>Example<\/strong>: The sentence &ldquo;The quick brown fox&rdquo; becomes a vector like:<\/p>\n<pre><code>[0.0123, -0.0456, 0.0789, -0.0234, 0.0567, ..., 0.0891] (1536 dimensions)<\/code><\/pre><h3 class=\"heading-element\" id=\"how-embeddings-are-created\"><span>How Embeddings Are Created<\/span>\n <a href=\"#how-embeddings-are-created\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Neural networks (specifically <strong>transformers<\/strong>) create embeddings through a training process:<\/p>\n<ol>\n<li><strong>Input Processing<\/strong>: Text is broken into tokens (words or subwords)<\/li>\n<li><strong>Contextual Understanding<\/strong>: The transformer processes all tokens together, understanding relationships<\/li>\n<li><strong>Vector Output<\/strong>: The final layer produces a fixed-size vector representing the entire input<\/li>\n<\/ol>\n<p><strong>Training Method - Contrastive Learning<\/strong>:<\/p>\n<ul>\n<li>The model sees pairs of similar texts (e.g., question and its answer)<\/li>\n<li>It learns to place similar texts close together in vector space<\/li>\n<li>Dissimilar texts are pushed apart<\/li>\n<li>After training on millions of pairs, the model understands semantic relationships<\/li>\n<\/ul>\n<div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">You don&rsquo;t need to train your own embedding model. Pre-trained models like OpenAI&rsquo;s <code>text-embedding-3-small<\/code> already understand language well. You just use them via API.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"why-similar-meanings-cluster-together\"><span>Why Similar Meanings Cluster Together<\/span>\n <a href=\"#why-similar-meanings-cluster-together\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The training process creates a <strong>semantic space<\/strong> where:<\/p>\n<ul>\n<li>&ldquo;Dog&rdquo; and &ldquo;Puppy&rdquo; \u2192 Close together (similar meaning)<\/li>\n<li>&ldquo;Dog&rdquo; and &ldquo;Airplane&rdquo; \u2192 Far apart (unrelated)<\/li>\n<li>&ldquo;King - Man + Woman \u2248 Queen&rdquo; \u2192 Vector arithmetic captures relationships<\/li>\n<\/ul>\n<p>This is why searching for &ldquo;automobile&rdquo; can find documents about &ldquo;cars&rdquo;\u2014they occupy nearby regions in the embedding space.<\/p>\n<h3 class=\"heading-element\" id=\"semantic-space-visualization\"><span>Semantic Space Visualization<\/span>\n <a href=\"#semantic-space-visualization\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>When text is converted to embeddings, semantically similar items cluster together in vector space:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/vector-embedding-semantic-space.webp\" title=\"Semantic embedding visualization showing word clustering based on meaning\" data-thumbnail=\"\/files\/vector-embedding-semantic-space.webp\" data-sub-html=\"<h2>Vector Embedding Semantic Space - 3D visualization showing how semantically similar words cluster together. Animals (Wolf, Dog, Cat, Kitten) cluster on one side while Fruits (Banana, Apple) cluster on another.<\/h2><p>Semantic embedding visualization showing word clustering based on meaning<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/vector-embedding-semantic-space.webp' alt=\"Vector Embedding Semantic Space - 3D visualization showing how semantically similar words cluster together. Animals (Wolf, Dog, Cat, Kitten) cluster on one side while Fruits (Banana, Apple) cluster on another.\"><\/a><figcaption class=\"image-caption\">Semantic embedding visualization showing word clustering based on meaning<\/figcaption>\n <\/figure><\/p>\n<p><strong>Key Insight<\/strong>: A query for &ldquo;Kitten&rdquo; naturally finds related animal terms, not fruits\u2014because they&rsquo;re close in vector space.<\/p>\n<h3 class=\"heading-element\" id=\"dense-vs-sparse-embeddings\"><span>Dense vs Sparse Embeddings<\/span>\n <a href=\"#dense-vs-sparse-embeddings\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Type<\/th>\n <th>Description<\/th>\n <th>Example<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Dense<\/strong><\/td>\n <td>Every dimension has a value; captures semantic meaning<\/td>\n <td><code>[0.01, 0.74, 0.52, ...]<\/code><\/td>\n <\/tr>\n <tr>\n <td><strong>Sparse<\/strong><\/td>\n <td>Most dimensions are zero; captures keyword presence<\/td>\n <td><code>{&quot;cat&quot;: 4, &quot;dog&quot;: 1}<\/code><\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">Modern systems combine both in <strong>hybrid search<\/strong> for semantic understanding AND keyword precision.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"dimension-trade-offs\"><span>Dimension Trade-offs<\/span>\n <a href=\"#dimension-trade-offs\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>More dimensions capture more nuance, but at a cost:<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Dimensions<\/th>\n <th>Speed<\/th>\n <th>Accuracy<\/th>\n <th>Memory<\/th>\n <th>Best For<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>384<\/strong><\/td>\n <td>Fastest<\/td>\n <td>Good<\/td>\n <td>~1.5 KB\/vector<\/td>\n <td>Real-time apps, chatbots<\/td>\n <\/tr>\n <tr>\n <td><strong>768<\/strong><\/td>\n <td>Fast<\/td>\n <td>Better<\/td>\n <td>~3 KB\/vector<\/td>\n <td>Balanced performance<\/td>\n <\/tr>\n <tr>\n <td><strong>1536<\/strong><\/td>\n <td>Medium<\/td>\n <td>Excellent<\/td>\n <td>~6 KB\/vector<\/td>\n <td>Most production use cases<\/td>\n <\/tr>\n <tr>\n <td><strong>3072<\/strong><\/td>\n <td>Slower<\/td>\n <td>Best<\/td>\n <td>~12 KB\/vector<\/td>\n <td>Maximum accuracy needed<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Rule of Thumb<\/strong>: Start with 1536 dimensions (OpenAI&rsquo;s default). Only go larger if accuracy is critical and you have the infrastructure.<\/p>\n<h3 class=\"heading-element\" id=\"popular-embedding-models\"><span>Popular Embedding Models<\/span>\n <a href=\"#popular-embedding-models\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Model<\/th>\n <th>Dimensions<\/th>\n <th>Provider<\/th>\n <th>Best For<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><code>text-embedding-3-small<\/code><\/td>\n <td>1536<\/td>\n <td>OpenAI<\/td>\n <td>Cost-effective general use<\/td>\n <\/tr>\n <tr>\n <td><code>text-embedding-3-large<\/code><\/td>\n <td>3072<\/td>\n <td>OpenAI<\/td>\n <td>Maximum accuracy<\/td>\n <\/tr>\n <tr>\n <td><code>all-MiniLM-L6-v2<\/code><\/td>\n <td>384<\/td>\n <td>HuggingFace<\/td>\n <td>Fast, open-source<\/td>\n <\/tr>\n <tr>\n <td><code>BGE-large-en-v1.5<\/code><\/td>\n <td>1024<\/td>\n <td>BAAI<\/td>\n <td>Top open-source performance<\/td>\n <\/tr>\n <tr>\n <td><code>embed-multilingual-v3.0<\/code><\/td>\n <td>1024<\/td>\n <td>Cohere<\/td>\n <td>100+ languages<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><hr>\n<h2 id=\"vector-search\">3. Vector Search and Similarity<\/h2>\r\n<h3 class=\"heading-element\" id=\"distance-metrics\"><span>Distance Metrics<\/span>\n <a href=\"#distance-metrics\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>To find similar vectors, we measure &ldquo;distance&rdquo; between them:<\/p>\n<p><strong>Cosine Similarity<\/strong> (most common for text):<\/p>\n<blockquote>\n<p>\ud835\udc2c\ud835\udc22\ud835\udc26\ud835\udc22\ud835\udc25\ud835\udc1a\ud835\udc2b\ud835\udc22\ud835\udc2d\ud835\udc32 = (\ud835\udc00 \u22c5 \ud835\udc01) \u00f7 (\u2016\ud835\udc00\u2016 \u2a2f \u2016\ud835\udc01\u2016)<\/p>\n<\/blockquote>\n<ul>\n<li><strong>Score of 1.0<\/strong> = Identical meaning<\/li>\n<li><strong>Score of 0.0<\/strong> = Completely unrelated<\/li>\n<li><strong>Score of -1.0<\/strong> = Opposite meaning<\/li>\n<\/ul>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Metric<\/th>\n <th>Best For<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Cosine<\/strong><\/td>\n <td>Text embeddings<\/td>\n <\/tr>\n <tr>\n <td><strong>Euclidean (L2)<\/strong><\/td>\n <td>Image embeddings<\/td>\n <\/tr>\n <tr>\n <td><strong>Dot Product<\/strong><\/td>\n <td>Normalized vectors<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"similarity-scoring-example\"><span>Similarity Scoring Example<\/span>\n <a href=\"#similarity-scoring-example\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Query: &ldquo;Is Windows 8 any good?&rdquo;<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/semantic-similarity-scoring.webp\" title=\"Semantic similarity scoring demonstration\" data-thumbnail=\"\/files\/semantic-similarity-scoring.webp\" data-sub-html=\"<h2>Semantic Similarity Scoring - Query matched against documents showing scores: 0.88 for &ldquo;Windows 10 is good&rdquo;, 0.82 for &ldquo;I love Windows 11!&rdquo;, 0.88 for &ldquo;Not enjoying Windows 10&rdquo;, 0.87 for &ldquo;Always found Windows 8 kinda weird&rdquo;<\/h2><p>Semantic similarity scoring demonstration<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/semantic-similarity-scoring.webp' alt=\"Semantic Similarity Scoring - Query matched against documents showing scores: 0.88 for \u201cWindows 10 is good\u201d, 0.82 for \u201cI love Windows 11!\u201d, 0.88 for \u201cNot enjoying Windows 10\u201d, 0.87 for \u201cAlways found Windows 8 kinda weird\u201d\"><\/a><figcaption class=\"image-caption\">Semantic similarity scoring demonstration<\/figcaption>\n <\/figure><\/p>\n<div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Pitfall<\/strong>: &ldquo;Windows 10 is good&rdquo; scores 0.88\u2014as high as actual Windows 8 content! Embeddings treat version numbers as semantically similar. This is why <strong>hybrid search<\/strong> matters.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"vector-indexing-algorithms\"><span>Vector Indexing Algorithms<\/span>\n <a href=\"#vector-indexing-algorithms\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Searching billions of vectors requires <strong>Approximate Nearest Neighbor (ANN)<\/strong> algorithms. Here&rsquo;s how the main ones work:<\/p>\n<h4 class=\"heading-element\" id=\"hnsw-hierarchical-navigable-small-world\"><span>HNSW (Hierarchical Navigable Small World)<\/span>\n <a href=\"#hnsw-hierarchical-navigable-small-world\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>The most popular algorithm for production systems. Think of it as a <strong>multi-level express train system<\/strong>:<\/p>\n<p><strong>How it works:<\/strong><\/p>\n<ol>\n<li>Creates multiple layers of connected nodes (vectors)<\/li>\n<li>Top layer: Few nodes, long-distance connections (express trains)<\/li>\n<li>Bottom layer: All nodes, short-distance connections (local stops)<\/li>\n<li>Search: Start at top, quickly narrow down, then refine at bottom<\/li>\n<\/ol>\n<p><strong>Key Parameters:<\/strong><\/p>\n<ul>\n<li><code>M<\/code>: Connections per node (higher = better recall, more memory)<\/li>\n<li><code>ef<\/code>: Search width (higher = more accurate, slower)<\/li>\n<\/ul>\n<p><strong>Trade-off<\/strong>: Excellent speed and accuracy, but uses more memory than other methods.<\/p>\n<h4 class=\"heading-element\" id=\"ivf-inverted-file-index\"><span>IVF (Inverted File Index)<\/span>\n <a href=\"#ivf-inverted-file-index\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Uses <strong>clustering<\/strong> to organize vectors into buckets:<\/p>\n<p><strong>How it works:<\/strong><\/p>\n<ol>\n<li><strong>Training<\/strong>: Use k-means to create cluster centers (centroids)<\/li>\n<li><strong>Indexing<\/strong>: Assign each vector to its nearest centroid<\/li>\n<li><strong>Search<\/strong>: Find nearest centroids to query, then search only those clusters<\/li>\n<\/ol>\n<p><strong>Key Parameter:<\/strong><\/p>\n<ul>\n<li><code>nprobe<\/code>: How many clusters to search (higher = more accurate, slower)<\/li>\n<\/ul>\n<p><strong>Trade-off<\/strong>: Fast for huge datasets, but requires training data and careful tuning.<\/p>\n<h4 class=\"heading-element\" id=\"pq-product-quantization\"><span>PQ (Product Quantization)<\/span>\n <a href=\"#pq-product-quantization\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><strong>Compression<\/strong> technique that shrinks vectors dramatically:<\/p>\n<p><strong>How it works:<\/strong><\/p>\n<ol>\n<li>Split each vector into sub-vectors (e.g., 1536D \u2192 8 chunks of 192D)<\/li>\n<li>Replace each sub-vector with a code pointing to a codebook entry<\/li>\n<li>Store only the codes (8 bytes instead of 6 KB!)<\/li>\n<\/ol>\n<p><strong>Trade-off<\/strong>: Massive memory savings (32-64x), but loses some accuracy.<\/p>\n<h4 class=\"heading-element\" id=\"algorithm-comparison\"><span>Algorithm Comparison<\/span>\n <a href=\"#algorithm-comparison\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Algorithm<\/th>\n <th>How It Works<\/th>\n <th>Search Speed<\/th>\n <th>Memory<\/th>\n <th>Accuracy<\/th>\n <th>Best For<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>HNSW<\/strong><\/td>\n <td>Graph navigation<\/td>\n <td>O(log N)<\/td>\n <td>High<\/td>\n <td>Excellent<\/td>\n <td>Production systems<\/td>\n <\/tr>\n <tr>\n <td><strong>IVF<\/strong><\/td>\n <td>Cluster search<\/td>\n <td>O(\u221aN)<\/td>\n <td>Medium<\/td>\n <td>Good<\/td>\n <td>Large-scale search<\/td>\n <\/tr>\n <tr>\n <td><strong>PQ<\/strong><\/td>\n <td>Compressed codes<\/td>\n <td>O(N)<\/td>\n <td>Very Low<\/td>\n <td>Moderate<\/td>\n <td>Billions of vectors<\/td>\n <\/tr>\n <tr>\n <td><strong>IVF-PQ<\/strong><\/td>\n <td>Clusters + compression<\/td>\n <td>O(\u221aN)<\/td>\n <td>Low<\/td>\n <td>Good<\/td>\n <td>Balance of all factors<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Most production systems use HNSW<\/strong> (Pinecone, Weaviate, Qdrant). It offers the best balance of speed and accuracy. Use IVF-PQ only when you have billions of vectors and limited memory.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"rag-fundamentals\">4. RAG (Retrieval-Augmented Generation)<\/h2>\r\n<h3 class=\"heading-element\" id=\"what-is-rag\"><span>What is RAG?<\/span>\n <a href=\"#what-is-rag\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>RAG<\/strong> enhances LLM responses by retrieving relevant information from external sources before generating an answer.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/rag-architecture-overview.webp\" title=\"Basic RAG pipeline architecture\" data-thumbnail=\"\/files\/rag-architecture-overview.webp\" data-sub-html=\"<h2>RAG Architecture Overview - Flow: User query \u2192 Embedding model \u2192 Vector database \u2192 Retrieved documents \u2192 LLM \u2192 Answer<\/h2><p>Basic RAG pipeline architecture<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/rag-architecture-overview.webp' alt=\"RAG Architecture Overview - Flow: User query \u2192 Embedding model \u2192 Vector database \u2192 Retrieved documents \u2192 LLM \u2192 Answer\"><\/a><figcaption class=\"image-caption\">Basic RAG pipeline architecture<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"rag-pipeline-steps\"><span>RAG Pipeline Steps<\/span>\n <a href=\"#rag-pipeline-steps\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li><strong>User Query<\/strong> \u2192 User asks a question<\/li>\n<li><strong>Query Embedding<\/strong> \u2192 Convert query to vector<\/li>\n<li><strong>Vector Search<\/strong> \u2192 Find similar documents<\/li>\n<li><strong>Context Retrieval<\/strong> \u2192 Fetch document content<\/li>\n<li><strong>Prompt Construction<\/strong> \u2192 Combine query + context<\/li>\n<li><strong>LLM Generation<\/strong> \u2192 Generate grounded answer<\/li>\n<\/ol>\n<div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Why RAG Works<\/strong>: LLMs can&rsquo;t read thousands of documents and remember them. Fine-tuning influences style, not knowledge. RAG retrieves relevant knowledge dynamically at runtime.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"rag-benefits\"><span>RAG Benefits<\/span>\n <a href=\"#rag-benefits\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Benefit<\/th>\n <th>Description<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>No Retraining<\/strong><\/td>\n <td>Update knowledge by updating the database<\/td>\n <\/tr>\n <tr>\n <td><strong>Verifiable<\/strong><\/td>\n <td>Can cite sources for answers<\/td>\n <\/tr>\n <tr>\n <td><strong>Cost-Effective<\/strong><\/td>\n <td>Cheaper than fine-tuning<\/td>\n <\/tr>\n <tr>\n <td><strong>Up-to-Date<\/strong><\/td>\n <td>Add new information instantly<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"hallucination-prevention\"><span>Hallucination Prevention<\/span>\n <a href=\"#hallucination-prevention\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>RAG combats hallucinations by:<\/p>\n<ol>\n<li>Providing factual context in the prompt<\/li>\n<li>Instructing the LLM to answer <strong>only<\/strong> from provided context<\/li>\n<li>Allowing &ldquo;I don&rsquo;t know&rdquo; when context is insufficient<\/li>\n<\/ol>\n<hr>\n<h2 id=\"document-processing\">5. Document Processing Pipeline<\/h2>\r\n<h3 class=\"heading-element\" id=\"basic-chunking-pipeline\"><span>Basic Chunking Pipeline<\/span>\n <a href=\"#basic-chunking-pipeline\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/document-processing-pipeline.webp\" title=\"Basic document chunking pipeline\" data-thumbnail=\"\/files\/document-processing-pipeline.webp\" data-sub-html=\"<h2>Document Processing Pipeline - Flow: PDF \u2192 Raw text \u2192 Text chunks \u2192 Embeddings \u2192 Vector database<\/h2><p>Basic document chunking pipeline<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/document-processing-pipeline.webp' alt=\"Document Processing Pipeline - Flow: PDF \u2192 Raw text \u2192 Text chunks \u2192 Embeddings \u2192 Vector database\"><\/a><figcaption class=\"image-caption\">Basic document chunking pipeline<\/figcaption>\n <\/figure><\/p>\n<p><strong>Steps:<\/strong><\/p>\n<ol>\n<li><strong>Extract<\/strong> raw text from documents (PDF, DOCX, HTML)<\/li>\n<li><strong>Chunk<\/strong> into fixed-size pieces (500-1000 characters)<\/li>\n<li><strong>Embed<\/strong> each chunk to a vector<\/li>\n<li><strong>Store<\/strong> (vector, chunk) pairs in database<\/li>\n<\/ol>\n<h3 class=\"heading-element\" id=\"chunking-strategies-compared\"><span>Chunking Strategies Compared<\/span>\n <a href=\"#chunking-strategies-compared\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>There are multiple ways to split documents. Each has trade-offs:<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Strategy<\/th>\n <th>How It Works<\/th>\n <th>Pros<\/th>\n <th>Cons<\/th>\n <th>Best For<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Fixed-size<\/strong><\/td>\n <td>Split every N characters<\/td>\n <td>Simple, predictable<\/td>\n <td>May break mid-sentence<\/td>\n <td>Quick prototyping<\/td>\n <\/tr>\n <tr>\n <td><strong>Sentence-based<\/strong><\/td>\n <td>Split at sentence boundaries<\/td>\n <td>Grammatically correct<\/td>\n <td>Variable chunk sizes<\/td>\n <td>Articles, blogs<\/td>\n <\/tr>\n <tr>\n <td><strong>Recursive<\/strong><\/td>\n <td>Try paragraphs \u2192 sentences \u2192 words<\/td>\n <td>Balances size and meaning<\/td>\n <td>More complex<\/td>\n <td>Mixed documents<\/td>\n <\/tr>\n <tr>\n <td><strong>Semantic<\/strong><\/td>\n <td>Use embeddings to find topic shifts<\/td>\n <td>Best coherence<\/td>\n <td>Computationally expensive<\/td>\n <td>Technical docs<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Recursive Chunking<\/strong> (most recommended):<\/p>\n<ol>\n<li>First, try splitting by paragraph (<code>\\n\\n<\/code>)<\/li>\n<li>If chunks are still too large, split by sentence (<code>. <\/code>)<\/li>\n<li>If still too large, split by words<\/li>\n<li>This preserves natural document structure<\/li>\n<\/ol>\n<h3 class=\"heading-element\" id=\"chunk-size-trade-offs\"><span>Chunk Size Trade-offs<\/span>\n <a href=\"#chunk-size-trade-offs\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Size<\/th>\n <th>Tokens<\/th>\n <th>Behavior<\/th>\n <th>Best For<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Small<\/strong> (~256)<\/td>\n <td>~64<\/td>\n <td>Precise retrieval, less context<\/td>\n <td>Specific fact lookup<\/td>\n <\/tr>\n <tr>\n <td><strong>Medium<\/strong> (~512)<\/td>\n <td>~128<\/td>\n <td>Balanced approach<\/td>\n <td>General Q&amp;A<\/td>\n <\/tr>\n <tr>\n <td><strong>Large<\/strong> (~1024)<\/td>\n <td>~256<\/td>\n <td>More context, fewer chunks<\/td>\n <td>Complex explanations<\/td>\n <\/tr>\n <tr>\n <td><strong>Very Large<\/strong> (~2048)<\/td>\n <td>~512<\/td>\n <td>Full paragraphs<\/td>\n <td>Broad topic summaries<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Rule of Thumb<\/strong>: Match chunk size to expected query length. Short questions \u2192 smaller chunks. Complex questions \u2192 larger chunks.<\/p>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Always use chunk overlap<\/strong> (10-20%). This prevents losing information that spans chunk boundaries. If a key fact is at the edge of two chunks, overlap ensures it appears in at least one complete chunk.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"advanced-summarization-before-embedding\"><span>Advanced: Summarization Before Embedding<\/span>\n <a href=\"#advanced-summarization-before-embedding\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/document-summarization-pipeline.webp\" title=\"Enhanced pipeline with LLM-based summarization\" data-thumbnail=\"\/files\/document-summarization-pipeline.webp\" data-sub-html=\"<h2>Document Summarization Pipeline - Flow: PDF \u2192 Raw text \u2192 Text chunks \u2192 Small LLM \u2192 Summary \u2192 Embeddings \u2192 Vector database<\/h2><p>Enhanced pipeline with LLM-based summarization<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/document-summarization-pipeline.webp' alt=\"Document Summarization Pipeline - Flow: PDF \u2192 Raw text \u2192 Text chunks \u2192 Small LLM \u2192 Summary \u2192 Embeddings \u2192 Vector database\"><\/a><figcaption class=\"image-caption\">Enhanced pipeline with LLM-based summarization<\/figcaption>\n <\/figure><\/p>\n<p><strong>Benefits:<\/strong><\/p>\n<ul>\n<li>Removes filler text that confuses embeddings<\/li>\n<li>Creates denser, more meaningful representations<\/li>\n<li>Normalizes formatting across document types<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"parent-document-retrieval\"><span>Parent Document Retrieval<\/span>\n <a href=\"#parent-document-retrieval\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>For hierarchical documents (books, papers):<\/p>\n<ol>\n<li>Index smaller chunks (paragraphs) for precise retrieval<\/li>\n<li>When retrieved, also fetch the parent section for context<\/li>\n<li>Or use &ldquo;windowed&rdquo; approach\u2014retrieve neighboring chunks<\/li>\n<\/ol>\n<hr>\n<h2 id=\"prompt-engineering\">6. Prompt Engineering with RAG<\/h2>\r\n<h3 class=\"heading-element\" id=\"dynamic-few-shot-prompting\"><span>Dynamic Few-Shot Prompting<\/span>\n <a href=\"#dynamic-few-shot-prompting\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Instead of hardcoding examples, retrieve relevant examples from a vector database:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/few-shot-prompting-vector-db.webp\" title=\"Dynamic few-shot prompt construction\" data-thumbnail=\"\/files\/few-shot-prompting-vector-db.webp\" data-sub-html=\"<h2>Few-Shot Prompting with Vector Database - User query flows to vector database for examples, combined into prompt sent to LLM<\/h2><p>Dynamic few-shot prompt construction<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/few-shot-prompting-vector-db.webp' alt=\"Few-Shot Prompting with Vector Database - User query flows to vector database for examples, combined into prompt sent to LLM\"><\/a><figcaption class=\"image-caption\">Dynamic few-shot prompt construction<\/figcaption>\n <\/figure><\/p>\n<p><strong>Flow:<\/strong><\/p>\n<ol>\n<li>User asks a question<\/li>\n<li>Vector DB retrieves similar Q&amp;A examples<\/li>\n<li>Examples are injected into the prompt<\/li>\n<li>LLM generates answer using those examples as guidance<\/li>\n<\/ol>\n<h3 class=\"heading-element\" id=\"temperature-settings\"><span>Temperature Settings<\/span>\n <a href=\"#temperature-settings\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Temperature<\/th>\n <th>Behavior<\/th>\n <th>Best For<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>0.0 - 0.3<\/strong><\/td>\n <td>Deterministic, factual<\/td>\n <td>RAG, Q&amp;A, summarization<\/td>\n <\/tr>\n <tr>\n <td><strong>0.4 - 0.7<\/strong><\/td>\n <td>Balanced<\/td>\n <td>General conversation<\/td>\n <\/tr>\n <tr>\n <td><strong>0.8 - 1.0+<\/strong><\/td>\n <td>Creative, diverse<\/td>\n <td>Brainstorming, fiction<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>For RAG<\/strong>: Use <strong>low temperature (0.0-0.3)<\/strong>. Higher temperatures increase hallucination risk\u2014defeating the purpose of retrieval-augmented generation.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"hybrid-search\">7. Hybrid Search Strategies<\/h2>\r\n<h3 class=\"heading-element\" id=\"limitations-of-pure-semantic-search\"><span>Limitations of Pure Semantic Search<\/span>\n <a href=\"#limitations-of-pure-semantic-search\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Semantic search can fail for:<\/p>\n<ul>\n<li><strong>Entity-specific queries<\/strong>: &ldquo;Windows 8&rdquo; retrieves Windows 10\/11 content<\/li>\n<li><strong>Exact terminology<\/strong>: Medical terms, legal citations, product SKUs<\/li>\n<li><strong>Negations<\/strong>: &ldquo;not Python&rdquo; still retrieves Python content<\/li>\n<li><strong>Rare terms<\/strong>: Words not well-represented in training data<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"understanding-keyword-search-sparse\"><span>Understanding Keyword Search (Sparse)<\/span>\n <a href=\"#understanding-keyword-search-sparse\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Before combining approaches, understand how keyword search works:<\/p>\n<h4 class=\"heading-element\" id=\"tf-idf-term-frequency---inverse-document-frequency\"><span>TF-IDF (Term Frequency - Inverse Document Frequency)<\/span>\n <a href=\"#tf-idf-term-frequency---inverse-document-frequency\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>A classic algorithm that scores documents based on keyword importance:<\/p>\n<ul>\n<li><strong>TF (Term Frequency)<\/strong>: How often does the word appear in this document?<\/li>\n<li><strong>IDF (Inverse Document Frequency)<\/strong>: How rare is this word across ALL documents?<\/li>\n<li><strong>Score = TF \u00d7 IDF<\/strong><\/li>\n<\/ul>\n<p><strong>Intuition<\/strong>: If &ldquo;quantum&rdquo; appears 5 times in a document and is rare across your corpus, that document scores high for &ldquo;quantum&rdquo; queries.<\/p>\n<h4 class=\"heading-element\" id=\"bm25-best-match-25\"><span>BM25 (Best Match 25)<\/span>\n <a href=\"#bm25-best-match-25\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>An improved version of TF-IDF used by most search engines:<\/p>\n<ul>\n<li>Adds <strong>document length normalization<\/strong> (short docs don&rsquo;t unfairly win)<\/li>\n<li>Diminishing returns for repeated terms (10 mentions isn&rsquo;t 10x better than 1)<\/li>\n<li>Tunable parameters (k1, b) for different use cases<\/li>\n<\/ul>\n<p><strong>When BM25 Shines<\/strong>: Exact terminology, product codes, legal citations, medical terms.<\/p>\n<h3 class=\"heading-element\" id=\"combining-dense-and-sparse-vectors\"><span>Combining Dense and Sparse Vectors<\/span>\n <a href=\"#combining-dense-and-sparse-vectors\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/hybrid-search-architecture.webp\" title=\"Hybrid search combining dense and sparse vectors\" data-thumbnail=\"\/files\/hybrid-search-architecture.webp\" data-sub-html=\"<h2>Hybrid Search Architecture - Input flows to Dense embedding model AND Sparse embedding model, combined via Pinecone\/hybrid into Hybrid index<\/h2><p>Hybrid search combining dense and sparse vectors<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/hybrid-search-architecture.webp' alt=\"Hybrid Search Architecture - Input flows to Dense embedding model AND Sparse embedding model, combined via Pinecone\/hybrid into Hybrid index\"><\/a><figcaption class=\"image-caption\">Hybrid search combining dense and sparse vectors<\/figcaption>\n <\/figure><\/p>\n<p><strong>How It Works:<\/strong><\/p>\n<ol>\n<li><strong>Dense Vector<\/strong> \u2192 Captures semantic meaning (&ldquo;car&rdquo; finds &ldquo;automobile&rdquo;)<\/li>\n<li><strong>Sparse Vector<\/strong> \u2192 Captures exact keywords (BM25\/TF-IDF)<\/li>\n<li><strong>Combined Query<\/strong> \u2192 Search both indexes simultaneously<\/li>\n<li><strong>Score Fusion<\/strong> \u2192 Merge results using Reciprocal Rank Fusion (RRF)<\/li>\n<\/ol>\n<h4 class=\"heading-element\" id=\"reciprocal-rank-fusion-rrf\"><span>Reciprocal Rank Fusion (RRF)<\/span>\n <a href=\"#reciprocal-rank-fusion-rrf\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>The standard method for combining search results:<\/p>\n<blockquote>\n<p>\ud835\udc11\ud835\udc11\ud835\udc05(\ud835\udc85) = \u2211 \ud835\udfcf \/ (\ud835\udc8c + \ud835\udc2b\ud835\udc1a\ud835\udc27\ud835\udc24\u1d63(\ud835\udc85)) for each \ud835\udc93 \u2208 \ud835\udc79<\/p>\n<\/blockquote>\n<p>Where:<\/p>\n<ul>\n<li><code>d<\/code> = document<\/li>\n<li><code>R<\/code> = set of ranking methods (dense, sparse)<\/li>\n<li><code>k<\/code> = constant (typically 60)<\/li>\n<li><code>rank_r(d)<\/code> = position of document d in ranking r<\/li>\n<\/ul>\n<p><strong>Why It Works<\/strong>: Documents appearing in BOTH dense and sparse results get boosted. A document ranked #3 in dense and #5 in sparse will outrank one that&rsquo;s #1 in only one method.<\/p>\n<h3 class=\"heading-element\" id=\"when-to-use-hybrid-search\"><span>When to Use Hybrid Search<\/span>\n <a href=\"#when-to-use-hybrid-search\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Scenario<\/th>\n <th>Approach<\/th>\n <th>Why<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>General Q&amp;A<\/td>\n <td>Dense (semantic)<\/td>\n <td>Users phrase questions differently than docs<\/td>\n <\/tr>\n <tr>\n <td>Technical docs<\/td>\n <td>Hybrid<\/td>\n <td>Need both concepts AND specific terms<\/td>\n <\/tr>\n <tr>\n <td>Legal\/Medical<\/td>\n <td>Hybrid (favor sparse)<\/td>\n <td>Exact terminology is critical<\/td>\n <\/tr>\n <tr>\n <td>Product search with SKUs<\/td>\n <td>Hybrid<\/td>\n <td>Must match exact product codes<\/td>\n <\/tr>\n <tr>\n <td>Conversational AI<\/td>\n <td>Dense<\/td>\n <td>Natural language varies widely<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"search-type-effectiveness\"><span>Search Type Effectiveness<\/span>\n <a href=\"#search-type-effectiveness\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Search Type<\/th>\n <th>Finds<\/th>\n <th>Misses<\/th>\n <th>Example<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Dense only<\/strong><\/td>\n <td>Synonyms, paraphrases<\/td>\n <td>Exact codes, rare terms<\/td>\n <td>&ldquo;Tell me about vehicles&rdquo; \u2192 finds &ldquo;car&rdquo;, &ldquo;automobile&rdquo;<\/td>\n <\/tr>\n <tr>\n <td><strong>Sparse only<\/strong><\/td>\n <td>Exact matches<\/td>\n <td>Semantic variations<\/td>\n <td>&ldquo;SKU-12345&rdquo; \u2192 finds exact match<\/td>\n <\/tr>\n <tr>\n <td><strong>Hybrid<\/strong><\/td>\n <td>Both semantic AND exact<\/td>\n <td>Rarely misses anything<\/td>\n <td>&ldquo;Windows 8 issues&rdquo; \u2192 finds Windows 8 specifically<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Rule of Thumb<\/strong>: If exact keywords matter, use hybrid search. Most modern vector databases (Pinecone, Qdrant, Weaviate) support it natively. Start with a 50\/50 weight, then tune based on your data.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"reranking\">8. Reranking and Context Compression<\/h2>\r\n<h3 class=\"heading-element\" id=\"why-rerank\"><span>Why Rerank?<\/span>\n <a href=\"#why-rerank\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Initial vector search isn&rsquo;t perfect:<\/p>\n<ul>\n<li>Relevant documents may rank low<\/li>\n<li>Too much context overwhelms the LLM<\/li>\n<li>Documents buried in the middle get ignored (&ldquo;lost in the middle&rdquo; problem)<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"how-reranking-works\"><span>How Reranking Works<\/span>\n <a href=\"#how-reranking-works\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/reranking-search-results.webp\" title=\"Search result reranking process\" data-thumbnail=\"\/files\/reranking-search-results.webp\" data-sub-html=\"<h2>Reranking Search Results - Initial results pass through Rerank model, scores adjust (20%, 15%, 80%), final ranking reorders results<\/h2><p>Search result reranking process<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/reranking-search-results.webp' alt=\"Reranking Search Results - Initial results pass through Rerank model, scores adjust (20%, 15%, 80%), final ranking reorders results\"><\/a><figcaption class=\"image-caption\">Search result reranking process<\/figcaption>\n <\/figure><\/p>\n<p><strong>Two-Stage Retrieval:<\/strong><\/p>\n<ol>\n<li><strong>Stage 1<\/strong>: Vector search retrieves broad candidates (fast, ~100 docs)<\/li>\n<li><strong>Stage 2<\/strong>: Reranker re-scores top candidates by relevance (precise, ~20 docs)<\/li>\n<\/ol>\n<h3 class=\"heading-element\" id=\"bi-encoder-vs-cross-encoder\"><span>Bi-Encoder vs Cross-Encoder<\/span>\n <a href=\"#bi-encoder-vs-cross-encoder\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The key to understanding reranking is understanding these two architectures:<\/p>\n<h4 class=\"heading-element\" id=\"bi-encoder-used-for-initial-retrieval\"><span>Bi-Encoder (Used for Initial Retrieval)<\/span>\n <a href=\"#bi-encoder-used-for-initial-retrieval\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><ul>\n<li>Embeds query and document <strong>separately<\/strong><\/li>\n<li>Compares pre-computed embeddings using cosine similarity<\/li>\n<li><strong>Speed<\/strong>: Can search millions of docs in milliseconds<\/li>\n<li><strong>Accuracy<\/strong>: Good, but misses nuanced query-document relationships<\/li>\n<\/ul>\n<h4 class=\"heading-element\" id=\"cross-encoder-used-for-reranking\"><span>Cross-Encoder (Used for Reranking)<\/span>\n <a href=\"#cross-encoder-used-for-reranking\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><ul>\n<li>Processes query AND document <strong>together<\/strong> as one input<\/li>\n<li>Considers every word interaction between query and document<\/li>\n<li><strong>Speed<\/strong>: Slow (must process each doc individually)<\/li>\n<li><strong>Accuracy<\/strong>: Excellent (understands exact relevance)<\/li>\n<\/ul>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Type<\/th>\n <th>How It Works<\/th>\n <th>Speed<\/th>\n <th>Accuracy<\/th>\n <th>Stage<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Bi-Encoder<\/strong><\/td>\n <td>Embed separately, compare<\/td>\n <td>Milliseconds for millions<\/td>\n <td>Good<\/td>\n <td>Initial retrieval<\/td>\n <\/tr>\n <tr>\n <td><strong>Cross-Encoder<\/strong><\/td>\n <td>Process together<\/td>\n <td>Seconds for dozens<\/td>\n <td>Excellent<\/td>\n <td>Reranking<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Analogy<\/strong>: Bi-encoders are like speed dating (quick impressions). Cross-encoders are like in-depth interviews (thorough evaluation).<\/p>\n<h3 class=\"heading-element\" id=\"why-two-stages\"><span>Why Two Stages?<\/span>\n <a href=\"#why-two-stages\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>You can&rsquo;t use cross-encoders for initial search\u2014scoring 1 million documents would take hours. Instead:<\/p>\n<ol>\n<li><strong>Stage 1 (Bi-Encoder)<\/strong>: Cast a wide net, retrieve ~100 candidates fast<\/li>\n<li><strong>Stage 2 (Cross-Encoder)<\/strong>: Carefully evaluate top ~20 candidates<\/li>\n<li>Return the best ~5-10 to the LLM<\/li>\n<\/ol>\n<p>This gives you the best of both worlds: speed AND accuracy.<\/p>\n<h3 class=\"heading-element\" id=\"popular-reranking-models\"><span>Popular Reranking Models<\/span>\n <a href=\"#popular-reranking-models\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Model<\/th>\n <th>Type<\/th>\n <th>Speed<\/th>\n <th>Accuracy<\/th>\n <th>Notes<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><code>rerank-english-v3.0<\/code><\/td>\n <td>Cohere API<\/td>\n <td>Fast<\/td>\n <td>Excellent<\/td>\n <td>Production-ready, paid<\/td>\n <\/tr>\n <tr>\n <td><code>bge-reranker-large<\/code><\/td>\n <td>Open Source<\/td>\n <td>Medium<\/td>\n <td>Excellent<\/td>\n <td>Best open-source option<\/td>\n <\/tr>\n <tr>\n <td><code>ms-marco-MiniLM<\/code><\/td>\n <td>Open Source<\/td>\n <td>Fast<\/td>\n <td>Good<\/td>\n <td>Lightweight, fast<\/td>\n <\/tr>\n <tr>\n <td><code>cross-encoder\/ms-marco-MiniLM-L-6-v2<\/code><\/td>\n <td>HuggingFace<\/td>\n <td>Fast<\/td>\n <td>Good<\/td>\n <td>Easy to deploy<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"contextual-compression\"><span>Contextual Compression<\/span>\n <a href=\"#contextual-compression\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Use a smaller LLM to:<\/p>\n<ul>\n<li>Extract only relevant portions from each document<\/li>\n<li>Discard irrelevant context<\/li>\n<li>Reduce token usage and cost<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"the-lost-in-the-middle-problem\"><span>The &ldquo;Lost in the Middle&rdquo; Problem<\/span>\n <a href=\"#the-lost-in-the-middle-problem\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Research shows LLMs have a <strong>U-shaped attention pattern<\/strong>:<\/p>\n<ul>\n<li>Pay most attention to the <strong>beginning<\/strong> of context<\/li>\n<li>Pay good attention to the <strong>end<\/strong> of context<\/li>\n<li><strong>Ignore or forget<\/strong> information in the middle<\/li>\n<\/ul>\n<p><strong>Implications for RAG:<\/strong><\/p>\n<ul>\n<li>Don&rsquo;t just append retrieved docs in order of similarity score<\/li>\n<li>Put the MOST relevant document first<\/li>\n<li>Put the SECOND most relevant document last<\/li>\n<li>Less critical docs go in the middle<\/li>\n<\/ul>\n<div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Lost in the Middle<\/strong>: If you retrieve 10 documents and the answer is in document #5, the LLM might miss it entirely. Reorder your context strategically!<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"query-transformation\">9. Query Transformation Techniques<\/h2>\r\n<p>Sometimes the user&rsquo;s query isn&rsquo;t optimal for retrieval. Query transformation techniques improve retrieval by reformulating the query before searching.<\/p>\n<h3 class=\"heading-element\" id=\"hyde-hypothetical-document-embeddings\"><span>HyDE (Hypothetical Document Embeddings)<\/span>\n <a href=\"#hyde-hypothetical-document-embeddings\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Problem<\/strong>: User queries are short and may not match document vocabulary.<\/p>\n<p><strong>Solution<\/strong>: Generate a hypothetical answer, then search for documents similar to that answer.<\/p>\n<p><strong>How it works:<\/strong><\/p>\n<ol>\n<li>User asks: &ldquo;How do I fix memory leaks in Python?&rdquo;<\/li>\n<li>LLM generates a hypothetical answer (even if imperfect)<\/li>\n<li>Embed the hypothetical answer (not the question)<\/li>\n<li>Search for real documents similar to this hypothetical<\/li>\n<li>Retrieved docs are often more relevant than direct query search<\/li>\n<\/ol>\n<p><strong>When to use<\/strong>: Technical queries, specialized domains where users and documents use different vocabulary.<\/p>\n<h3 class=\"heading-element\" id=\"multi-query-expansion\"><span>Multi-Query Expansion<\/span>\n <a href=\"#multi-query-expansion\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Problem<\/strong>: A single query may miss relevant documents phrased differently.<\/p>\n<p><strong>Solution<\/strong>: Generate multiple variations of the query and search with all of them.<\/p>\n<p><strong>How it works:<\/strong><\/p>\n<ol>\n<li>Original query: &ldquo;Python web frameworks&rdquo;<\/li>\n<li>Generate variations:\n<ul>\n<li>&ldquo;Django vs Flask comparison&rdquo;<\/li>\n<li>&ldquo;Best backend frameworks for Python&rdquo;<\/li>\n<li>&ldquo;Building web apps with Python&rdquo;<\/li>\n<\/ul>\n<\/li>\n<li>Search with each query<\/li>\n<li>Combine results (using RRF or union)<\/li>\n<\/ol>\n<p><strong>When to use<\/strong>: Ambiguous queries, broad topics, when recall is more important than precision.<\/p>\n<h3 class=\"heading-element\" id=\"step-back-prompting\"><span>Step-Back Prompting<\/span>\n <a href=\"#step-back-prompting\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Problem<\/strong>: Specific questions may miss broader context needed to answer well.<\/p>\n<p><strong>Solution<\/strong>: Generate a more general version of the query first.<\/p>\n<p><strong>How it works:<\/strong><\/p>\n<ol>\n<li>Specific query: &ldquo;What&rsquo;s the boiling point of water at 2000m altitude?&rdquo;<\/li>\n<li>Step-back query: &ldquo;How does altitude affect boiling point?&rdquo;<\/li>\n<li>Retrieve documents for BOTH queries<\/li>\n<li>Combine context (general + specific)<\/li>\n<\/ol>\n<p><strong>When to use<\/strong>: Complex questions that require background knowledge.<\/p>\n<h3 class=\"heading-element\" id=\"query-transformation-comparison\"><span>Query Transformation Comparison<\/span>\n <a href=\"#query-transformation-comparison\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Technique<\/th>\n <th>When to Use<\/th>\n <th>Latency Impact<\/th>\n <th>Recall Improvement<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>HyDE<\/strong><\/td>\n <td>Technical\/specialized queries<\/td>\n <td>+1 LLM call<\/td>\n <td>+15-25%<\/td>\n <\/tr>\n <tr>\n <td><strong>Multi-Query<\/strong><\/td>\n <td>Ambiguous queries<\/td>\n <td>+3-5 searches<\/td>\n <td>+10-20%<\/td>\n <\/tr>\n <tr>\n <td><strong>Step-Back<\/strong><\/td>\n <td>Complex questions<\/td>\n <td>+1 LLM call, +1 search<\/td>\n <td>Better context<\/td>\n <\/tr>\n <tr>\n <td><strong>Query Rewriting<\/strong><\/td>\n <td>Poor user queries<\/td>\n <td>+1 LLM call<\/td>\n <td>Variable<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Start simple<\/strong>: Most RAG systems work fine without query transformation. Add these techniques only if you see retrieval quality issues.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"advanced-rag\">10. Advanced RAG Patterns<\/h2>\r\n<p>Basic RAG (chunk \u2192 embed \u2192 retrieve \u2192 generate) works well for simple use cases. For more complex scenarios, consider these advanced patterns:<\/p>\n<h3 class=\"heading-element\" id=\"crag-corrective-rag\"><span>CRAG (Corrective RAG)<\/span>\n <a href=\"#crag-corrective-rag\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Problem<\/strong>: Retrieved documents might be irrelevant or outdated.<\/p>\n<p><strong>Solution<\/strong>: Evaluate retrieval quality and self-correct if needed.<\/p>\n<p><strong>How it works:<\/strong><\/p>\n<ol>\n<li>Retrieve documents normally<\/li>\n<li>Use an LLM to evaluate: &ldquo;Are these documents relevant to the query?&rdquo;<\/li>\n<li>If confident \u2192 proceed to generation<\/li>\n<li>If uncertain \u2192 try alternative retrieval (e.g., web search)<\/li>\n<li>If irrelevant \u2192 fall back to web search or &ldquo;I don&rsquo;t know&rdquo;<\/li>\n<\/ol>\n<p><strong>When to use<\/strong>: Knowledge bases that may be incomplete, time-sensitive information.<\/p>\n<h3 class=\"heading-element\" id=\"self-rag-self-reflective-rag\"><span>Self-RAG (Self-Reflective RAG)<\/span>\n <a href=\"#self-rag-self-reflective-rag\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Problem<\/strong>: Not every query needs retrieval. Basic RAG always retrieves, wasting resources.<\/p>\n<p><strong>Solution<\/strong>: Let the LLM decide WHEN to retrieve and verify its own outputs.<\/p>\n<p><strong>How it works:<\/strong><\/p>\n<ol>\n<li>LLM evaluates: &ldquo;Do I need external information for this query?&rdquo;<\/li>\n<li>If yes \u2192 retrieve and generate with context<\/li>\n<li>If no \u2192 generate directly from knowledge<\/li>\n<li>After generation \u2192 LLM verifies: &ldquo;Is this answer supported by the context?&rdquo;<\/li>\n<\/ol>\n<p><strong>When to use<\/strong>: Mixed query types (some factual, some conversational), cost-sensitive applications.<\/p>\n<h3 class=\"heading-element\" id=\"agentic-rag\"><span>Agentic RAG<\/span>\n <a href=\"#agentic-rag\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Problem<\/strong>: Complex questions require multiple retrieval steps and reasoning.<\/p>\n<p><strong>Solution<\/strong>: LLM acts as an agent that can retrieve, reason, and retrieve again.<\/p>\n<p><strong>How it works:<\/strong><\/p>\n<ol>\n<li>LLM analyzes the question<\/li>\n<li>Breaks it into sub-questions if needed<\/li>\n<li>Retrieves information for each sub-question<\/li>\n<li>Reasons over retrieved information<\/li>\n<li>May retrieve again if gaps are found<\/li>\n<li>Synthesizes final answer<\/li>\n<\/ol>\n<p><strong>Example<\/strong>: &ldquo;Compare the economic policies of the last 3 US presidents&rdquo;<\/p>\n<ul>\n<li>Agent retrieves info on President 1<\/li>\n<li>Agent retrieves info on President 2<\/li>\n<li>Agent retrieves info on President 3<\/li>\n<li>Agent synthesizes comparison<\/li>\n<\/ul>\n<p><strong>When to use<\/strong>: Research questions, multi-hop reasoning, complex analysis.<\/p>\n<h3 class=\"heading-element\" id=\"rag-pattern-selection\"><span>RAG Pattern Selection<\/span>\n <a href=\"#rag-pattern-selection\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Pattern<\/th>\n <th>Complexity<\/th>\n <th>Best For<\/th>\n <th>Key Benefit<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Basic RAG<\/strong><\/td>\n <td>Simple<\/td>\n <td>Straightforward Q&amp;A<\/td>\n <td>Easy to implement<\/td>\n <\/tr>\n <tr>\n <td><strong>CRAG<\/strong><\/td>\n <td>Medium<\/td>\n <td>Incomplete knowledge bases<\/td>\n <td>Handles retrieval failures<\/td>\n <\/tr>\n <tr>\n <td><strong>Self-RAG<\/strong><\/td>\n <td>Medium<\/td>\n <td>Mixed query types<\/td>\n <td>Efficient (skips unnecessary retrieval)<\/td>\n <\/tr>\n <tr>\n <td><strong>Agentic RAG<\/strong><\/td>\n <td>High<\/td>\n <td>Complex research<\/td>\n <td>Multi-step reasoning<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Start with Basic RAG<\/strong>. Only add complexity when you have evidence that basic RAG isn&rsquo;t working. Each additional pattern adds latency and cost.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"evaluation\">11. RAG Evaluation Framework<\/h2>\r\n<p>You can&rsquo;t improve what you don&rsquo;t measure. RAG systems require evaluation at two stages: retrieval and generation.<\/p>\n<h3 class=\"heading-element\" id=\"retrieval-metrics\"><span>Retrieval Metrics<\/span>\n <a href=\"#retrieval-metrics\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>These measure how well your system finds relevant documents:<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Metric<\/th>\n <th>What It Measures<\/th>\n <th>Formula<\/th>\n <th>Good Score<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Recall@k<\/strong><\/td>\n <td>% of relevant docs in top k results<\/td>\n <td>relevant_in_k \/ total_relevant<\/td>\n <td>&gt; 0.8<\/td>\n <\/tr>\n <tr>\n <td><strong>Precision@k<\/strong><\/td>\n <td>% of top k results that are relevant<\/td>\n <td>relevant_in_k \/ k<\/td>\n <td>&gt; 0.6<\/td>\n <\/tr>\n <tr>\n <td><strong>MRR<\/strong><\/td>\n <td>Position of first relevant result<\/td>\n <td>1 \/ rank_of_first_relevant<\/td>\n <td>&gt; 0.7<\/td>\n <\/tr>\n <tr>\n <td><strong>NDCG<\/strong><\/td>\n <td>Ranking quality (position matters)<\/td>\n <td>DCG \/ ideal_DCG<\/td>\n <td>&gt; 0.7<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Example<\/strong>: For query &ldquo;What is RAG?&rdquo;, if you retrieve 10 docs and 3 are relevant:<\/p>\n<ul>\n<li>If relevant docs are at positions 1, 2, 5 \u2192 Good (high MRR, good NDCG)<\/li>\n<li>If relevant docs are at positions 6, 8, 10 \u2192 Bad (low MRR, poor NDCG)<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"generation-metrics-ragas-framework\"><span>Generation Metrics (RAGAS Framework)<\/span>\n <a href=\"#generation-metrics-ragas-framework\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>RAGAS is the standard framework for evaluating RAG generation quality:<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Metric<\/th>\n <th>What It Measures<\/th>\n <th>How It&rsquo;s Calculated<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Faithfulness<\/strong><\/td>\n <td>Is answer grounded in context?<\/td>\n <td>LLM checks if claims are supported<\/td>\n <\/tr>\n <tr>\n <td><strong>Answer Relevancy<\/strong><\/td>\n <td>Does answer address the question?<\/td>\n <td>LLM scores relevance<\/td>\n <\/tr>\n <tr>\n <td><strong>Context Precision<\/strong><\/td>\n <td>Are retrieved docs actually useful?<\/td>\n <td>% of context that contributed to answer<\/td>\n <\/tr>\n <tr>\n <td><strong>Context Recall<\/strong><\/td>\n <td>Did we retrieve all needed info?<\/td>\n <td>Can answer be derived from context alone?<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"building-an-evaluation-dataset\"><span>Building an Evaluation Dataset<\/span>\n <a href=\"#building-an-evaluation-dataset\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>To evaluate your RAG system, you need:<\/p>\n<ol>\n<li><strong>Test Questions<\/strong>: 50-100 representative queries<\/li>\n<li><strong>Ground Truth Answers<\/strong>: What the correct answer should be<\/li>\n<li><strong>Relevant Documents<\/strong>: Which docs should be retrieved<\/li>\n<\/ol>\n<p><strong>Quick Start<\/strong>:<\/p>\n<ul>\n<li>Extract real user questions from logs<\/li>\n<li>Have domain experts write answers<\/li>\n<li>Run evaluation weekly to catch regressions<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"evaluation-best-practices\"><span>Evaluation Best Practices<\/span>\n <a href=\"#evaluation-best-practices\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li><strong>Separate retrieval and generation evaluation<\/strong> - A bad answer might be due to poor retrieval OR poor generation<\/li>\n<li><strong>Test edge cases<\/strong> - Queries with no answer, ambiguous queries, multi-hop questions<\/li>\n<li><strong>Track metrics over time<\/strong> - Catch regressions early<\/li>\n<li><strong>Use human evaluation<\/strong> for final quality assessment - Automated metrics don&rsquo;t catch everything<\/li>\n<\/ol>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Minimum viable evaluation<\/strong>: Start with 50 test questions and track Recall@10 for retrieval and Faithfulness for generation. Expand from there.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"debugging\">12. Debugging RAG Failures<\/h2>\r\n<p>When your RAG system gives wrong answers, use this systematic approach to find and fix the problem.<\/p>\n<h3 class=\"heading-element\" id=\"common-failure-modes\"><span>Common Failure Modes<\/span>\n <a href=\"#common-failure-modes\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Symptom<\/th>\n <th>Likely Cause<\/th>\n <th>Solution<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>Returns irrelevant documents<\/td>\n <td>Embedding model mismatch<\/td>\n <td>Try domain-specific embeddings<\/td>\n <\/tr>\n <tr>\n <td>Misses obvious answers<\/td>\n <td>Chunks too small<\/td>\n <td>Increase chunk size + overlap<\/td>\n <\/tr>\n <tr>\n <td>&ldquo;I don&rsquo;t know&rdquo; for known facts<\/td>\n <td>Document not indexed<\/td>\n <td>Check ingestion pipeline<\/td>\n <\/tr>\n <tr>\n <td>Contradictory answers<\/td>\n <td>Multiple conflicting sources<\/td>\n <td>Add source reliability scoring<\/td>\n <\/tr>\n <tr>\n <td>Slow responses<\/td>\n <td>Too many docs retrieved<\/td>\n <td>Reduce k, add reranking<\/td>\n <\/tr>\n <tr>\n <td>Hallucinations despite context<\/td>\n <td>LLM ignoring context<\/td>\n <td>Lower temperature, stronger instructions<\/td>\n <\/tr>\n <tr>\n <td>Wrong version info<\/td>\n <td>Semantic similarity ignores numbers<\/td>\n <td>Use hybrid search<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"debugging-checklist\"><span>Debugging Checklist<\/span>\n <a href=\"#debugging-checklist\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>When a query fails, check these in order:<\/p>\n<p><strong>1. Is the document even in the index?<\/strong><\/p>\n<ul>\n<li>Search for an exact phrase from the expected document<\/li>\n<li>If not found \u2192 ingestion problem<\/li>\n<\/ul>\n<p><strong>2. Is the document retrieved?<\/strong><\/p>\n<ul>\n<li>Look at similarity scores of retrieved docs<\/li>\n<li>If expected doc has low score \u2192 embedding or chunking problem<\/li>\n<\/ul>\n<p><strong>3. Is the right chunk retrieved?<\/strong><\/p>\n<ul>\n<li>Check if the answer spans multiple chunks<\/li>\n<li>If answer is split \u2192 adjust chunk size\/overlap<\/li>\n<\/ul>\n<p><strong>4. Is the LLM using the context?<\/strong><\/p>\n<ul>\n<li>Check if answer matches retrieved context<\/li>\n<li>If LLM ignores context \u2192 adjust prompt, lower temperature<\/li>\n<\/ul>\n<p><strong>5. Is there conflicting information?<\/strong><\/p>\n<ul>\n<li>Check for contradictory docs in results<\/li>\n<li>If present \u2192 add source filtering or recency scoring<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"retrieval-quality-debugging\"><span>Retrieval Quality Debugging<\/span>\n <a href=\"#retrieval-quality-debugging\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Test #1: Direct phrase search<\/strong><\/p>\n<ul>\n<li>Search for exact text from a document you know exists<\/li>\n<li>If it doesn&rsquo;t appear in top 10 \u2192 indexing problem<\/li>\n<\/ul>\n<p><strong>Test #2: Synonym search<\/strong><\/p>\n<ul>\n<li>Search using synonyms of known document content<\/li>\n<li>If it works \u2192 your embedding model is fine<\/li>\n<li>If it fails \u2192 consider different embedding model<\/li>\n<\/ul>\n<p><strong>Test #3: Compare dense vs sparse<\/strong><\/p>\n<ul>\n<li>Run query through dense search only<\/li>\n<li>Run query through sparse (keyword) search only<\/li>\n<li>Compare results \u2192 decide if hybrid search would help<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"when-to-suspect-each-component\"><span>When to Suspect Each Component<\/span>\n <a href=\"#when-to-suspect-each-component\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Component<\/th>\n <th>Suspect If&hellip;<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Chunking<\/strong><\/td>\n <td>Answers are partially correct, missing context<\/td>\n <\/tr>\n <tr>\n <td><strong>Embedding Model<\/strong><\/td>\n <td>Synonyms don&rsquo;t match, domain terms fail<\/td>\n <\/tr>\n <tr>\n <td><strong>Index Config<\/strong><\/td>\n <td>High recall but slow, or fast but missing results<\/td>\n <\/tr>\n <tr>\n <td><strong>Retrieval k<\/strong><\/td>\n <td>Good docs exist but not in top k<\/td>\n <\/tr>\n <tr>\n <td><strong>Reranking<\/strong><\/td>\n <td>Good docs retrieved but ranked low<\/td>\n <\/tr>\n <tr>\n <td><strong>Prompt<\/strong><\/td>\n <td>Correct context but wrong answer<\/td>\n <\/tr>\n <tr>\n <td><strong>LLM Temperature<\/strong><\/td>\n <td>Answers vary wildly, or include made-up facts<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Don&rsquo;t guess<\/strong> - measure! Log every query, retrieval result, and answer. Use these logs to identify patterns in failures.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"cost\">13. Cost Optimization<\/h2>\r\n<p>RAG systems can get expensive at scale. Here&rsquo;s how to optimize costs without sacrificing quality.<\/p>\n<h3 class=\"heading-element\" id=\"cost-components\"><span>Cost Components<\/span>\n <a href=\"#cost-components\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Component<\/th>\n <th>Cost Driver<\/th>\n <th>Typical Cost<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Embedding API<\/strong><\/td>\n <td>Per token<\/td>\n <td>~$0.02 per 1M tokens<\/td>\n <\/tr>\n <tr>\n <td><strong>Vector Database<\/strong><\/td>\n <td>Storage + queries<\/td>\n <td>$0.01-0.10 per 1K queries<\/td>\n <\/tr>\n <tr>\n <td><strong>LLM API<\/strong><\/td>\n <td>Input + output tokens<\/td>\n <td>$0.50-5.00 per 1K queries<\/td>\n <\/tr>\n <tr>\n <td><strong>Reranking<\/strong><\/td>\n <td>Per document scored<\/td>\n <td>~$0.001 per document<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Key Insight<\/strong>: LLM calls dominate costs (often 80-90% of total).<\/p>\n<h3 class=\"heading-element\" id=\"cost-reduction-strategies\"><span>Cost Reduction Strategies<\/span>\n <a href=\"#cost-reduction-strategies\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"1-query-routing-biggest-impact\"><span>1. Query Routing (Biggest Impact)<\/span>\n <a href=\"#1-query-routing-biggest-impact\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Not every query needs RAG. Route simple queries directly to the LLM.<\/p>\n<ul>\n<li>&ldquo;What&rsquo;s 2+2?&rdquo; \u2192 Direct LLM (no retrieval needed)<\/li>\n<li>&ldquo;What&rsquo;s our refund policy?&rdquo; \u2192 RAG (needs company docs)<\/li>\n<\/ul>\n<p><strong>Savings<\/strong>: 30-50% reduction in RAG operations.<\/p>\n<h4 class=\"heading-element\" id=\"2-aggressive-caching\"><span>2. Aggressive Caching<\/span>\n <a href=\"#2-aggressive-caching\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Cache at multiple levels:<\/p>\n<ul>\n<li><strong>Query cache<\/strong>: Same query \u2192 same results<\/li>\n<li><strong>Embedding cache<\/strong>: Same text \u2192 same embedding<\/li>\n<li><strong>Answer cache<\/strong>: Frequent questions \u2192 cached answers<\/li>\n<\/ul>\n<p><strong>Savings<\/strong>: 20-40% reduction in API calls.<\/p>\n<h4 class=\"heading-element\" id=\"3-tiered-retrieval\"><span>3. Tiered Retrieval<\/span>\n <a href=\"#3-tiered-retrieval\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Only use expensive operations when needed:<\/p>\n<ol>\n<li>Fast vector search (always)<\/li>\n<li>Reranking (only if initial results are uncertain)<\/li>\n<li>Query transformation (only if initial retrieval fails)<\/li>\n<\/ol>\n<p><strong>Savings<\/strong>: 15-25% reduction in compute.<\/p>\n<h4 class=\"heading-element\" id=\"4-context-optimization\"><span>4. Context Optimization<\/span>\n <a href=\"#4-context-optimization\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Reduce tokens sent to LLM:<\/p>\n<ul>\n<li>Retrieve fewer documents (5 instead of 10)<\/li>\n<li>Use smaller chunks<\/li>\n<li>Compress context before sending<\/li>\n<\/ul>\n<p><strong>Savings<\/strong>: 20-40% reduction in LLM costs.<\/p>\n<h4 class=\"heading-element\" id=\"5-model-selection\"><span>5. Model Selection<\/span>\n <a href=\"#5-model-selection\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Use cheaper models for appropriate tasks:<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Task<\/th>\n <th>Expensive Option<\/th>\n <th>Cheaper Alternative<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>Embedding<\/td>\n <td><code>text-embedding-3-large<\/code><\/td>\n <td><code>text-embedding-3-small<\/code><\/td>\n <\/tr>\n <tr>\n <td>Query routing<\/td>\n <td>GPT-4<\/td>\n <td>GPT-3.5 or classifier<\/td>\n <\/tr>\n <tr>\n <td>Simple Q&amp;A<\/td>\n <td>GPT-4<\/td>\n <td>GPT-3.5 or Claude Haiku<\/td>\n <\/tr>\n <tr>\n <td>Complex reasoning<\/td>\n <td>GPT-4<\/td>\n <td>Keep GPT-4 (worth the cost)<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"cost-estimation-example\"><span>Cost Estimation Example<\/span>\n <a href=\"#cost-estimation-example\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Scenario<\/strong>: 10,000 queries\/month, 5 docs retrieved per query<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Approach<\/th>\n <th>Embedding<\/th>\n <th>Vector DB<\/th>\n <th>LLM<\/th>\n <th>Total\/Month<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Basic RAG<\/strong><\/td>\n <td>$0.20<\/td>\n <td>$1.00<\/td>\n <td>$50.00<\/td>\n <td>~$51<\/td>\n <\/tr>\n <tr>\n <td><strong>With reranking<\/strong><\/td>\n <td>$0.20<\/td>\n <td>$1.00<\/td>\n <td>$70.00<\/td>\n <td>~$71<\/td>\n <\/tr>\n <tr>\n <td><strong>Optimized<\/strong><\/td>\n <td>$0.10<\/td>\n <td>$0.50<\/td>\n <td>$25.00<\/td>\n <td>~$26<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Optimizations applied<\/strong>: Query routing (50%), caching (20%), context compression (30%).<\/p>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Measure before optimizing<\/strong>. Log costs per query type. You&rsquo;ll often find 80% of costs come from 20% of query types\u2014focus there first.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"implementation\">14. Popular Vector Databases<\/h2>\r\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Database<\/th>\n <th>Type<\/th>\n <th>Key Features<\/th>\n <th>Best For<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Pinecone<\/strong><\/td>\n <td>Managed SaaS<\/td>\n <td>Serverless, hybrid search, easy setup<\/td>\n <td>Production without DevOps<\/td>\n <\/tr>\n <tr>\n <td><strong>Weaviate<\/strong><\/td>\n <td>Open Source\/Cloud<\/td>\n <td>GraphQL API, built-in ML modules<\/td>\n <td>Flexible deployments<\/td>\n <\/tr>\n <tr>\n <td><strong>Qdrant<\/strong><\/td>\n <td>Open Source\/Cloud<\/td>\n <td>Rust-based, very fast, good filtering<\/td>\n <td>High performance needs<\/td>\n <\/tr>\n <tr>\n <td><strong>Milvus<\/strong><\/td>\n <td>Open Source<\/td>\n <td>Distributed, GPU support, massive scale<\/td>\n <td>Billions of vectors<\/td>\n <\/tr>\n <tr>\n <td><strong>Chroma<\/strong><\/td>\n <td>Open Source<\/td>\n <td>Simple API, in-memory option<\/td>\n <td>Prototyping, local dev<\/td>\n <\/tr>\n <tr>\n <td><strong>pgvector<\/strong><\/td>\n <td>Postgres Extension<\/td>\n <td>SQL integration, familiar tooling<\/td>\n <td>Teams using Postgres<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"deployment-options\"><span>Deployment Options<\/span>\n <a href=\"#deployment-options\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Option<\/th>\n <th>Pros<\/th>\n <th>Cons<\/th>\n <th>Best For<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Managed SaaS<\/strong> (Pinecone)<\/td>\n <td>Zero DevOps, high reliability<\/td>\n <td>Higher cost, vendor lock-in<\/td>\n <td>Small teams, quick start<\/td>\n <\/tr>\n <tr>\n <td><strong>Managed Cloud<\/strong> (Weaviate Cloud, Qdrant Cloud)<\/td>\n <td>Balance of control and convenience<\/td>\n <td>Medium cost<\/td>\n <td>Growing teams<\/td>\n <\/tr>\n <tr>\n <td><strong>Self-Hosted<\/strong> (Milvus, Qdrant, Weaviate)<\/td>\n <td>Full control, lowest cost at scale<\/td>\n <td>DevOps required<\/td>\n <td>Large teams, enterprises<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"choosing-a-vector-database\"><span>Choosing a Vector Database<\/span>\n <a href=\"#choosing-a-vector-database\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Choose Pinecone if<\/strong>: You want zero infrastructure management and fastest time-to-production.<\/p>\n<p><strong>Choose Weaviate if<\/strong>: You need flexibility and like GraphQL APIs.<\/p>\n<p><strong>Choose Qdrant if<\/strong>: Performance is critical and you want both cloud and self-hosted options.<\/p>\n<p><strong>Choose Milvus if<\/strong>: You have billions of vectors and need distributed architecture.<\/p>\n<p><strong>Choose Chroma if<\/strong>: You&rsquo;re prototyping or building local-first applications.<\/p>\n<p><strong>Choose pgvector if<\/strong>: You already use PostgreSQL and want to add vector search without new infrastructure.<\/p>\n<h3 class=\"heading-element\" id=\"production-considerations\"><span>Production Considerations<\/span>\n <a href=\"#production-considerations\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Security<\/strong><\/p>\n<ul>\n<li>Never expose API keys in client-side code<\/li>\n<li>Implement rate limiting<\/li>\n<li>Sanitize queries to prevent prompt injection<\/li>\n<li>Use row-level security for multi-tenant applications<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<p><strong>Performance:<\/strong><\/p>\n<ul>\n<li>Batch embedding requests when indexing<\/li>\n<li>Use async operations for concurrent retrieval<\/li>\n<li>Cache frequently-asked queries<\/li>\n<li>Monitor query latency and adjust index parameters<\/li>\n<\/ul>\n<p><strong>Reliability:<\/strong><\/p>\n<ul>\n<li>Set up automated backups<\/li>\n<li>Use replicas for high availability<\/li>\n<li>Implement graceful degradation when DB is slow<\/li>\n<\/ul>\n<hr>\n<h2 id=\"best-practices\">15. Best Practices Summary<\/h2>\r\n<h3 class=\"heading-element\" id=\"chunking-checklist\"><span>Chunking Checklist<\/span>\n <a href=\"#chunking-checklist\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><ol>\n<li><strong>Preserve context<\/strong>: Don&rsquo;t split mid-sentence<\/li>\n<li><strong>Use overlap<\/strong>: 10-20% prevents information loss<\/li>\n<li><strong>Respect structure<\/strong>: Use headers as natural break points<\/li>\n<li><strong>Include metadata<\/strong>: Store source, page number, section title<\/li>\n<li><strong>Test empirically<\/strong>: Evaluate with different chunk sizes<\/li>\n<\/ol>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"embedding-model-selection\"><span>Embedding Model Selection<\/span>\n <a href=\"#embedding-model-selection\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Use Case<\/th>\n <th>Model<\/th>\n <th>Rationale<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>General English<\/td>\n <td><code>text-embedding-3-small<\/code><\/td>\n <td>Cost-effective<\/td>\n <\/tr>\n <tr>\n <td>Maximum accuracy<\/td>\n <td><code>text-embedding-3-large<\/code><\/td>\n <td>Best quality<\/td>\n <\/tr>\n <tr>\n <td>Multi-language<\/td>\n <td><code>embed-multilingual-v3.0<\/code><\/td>\n <td>100+ languages<\/td>\n <\/tr>\n <tr>\n <td>Open-source<\/td>\n <td><code>BGE-large-en-v1.5<\/code><\/td>\n <td>Top OSS performance<\/td>\n <\/tr>\n <tr>\n <td>Low latency<\/td>\n <td><code>all-MiniLM-L6-v2<\/code><\/td>\n <td>Fast, 384 dimensions<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"key-takeaways\"><span>Key Takeaways<\/span>\n <a href=\"#key-takeaways\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li><strong>Embeddings<\/strong> transform text into searchable vectors capturing semantic meaning<\/li>\n<li><strong>Vector databases<\/strong> enable fast similarity search at scale (HNSW is the standard)<\/li>\n<li><strong>RAG<\/strong> grounds LLM responses in factual, retrievable knowledge<\/li>\n<li><strong>Chunking<\/strong> significantly impacts quality\u2014use recursive chunking with overlap<\/li>\n<li><strong>Hybrid search<\/strong> combines semantic + keyword for best coverage<\/li>\n<li><strong>Reranking<\/strong> improves precision using cross-encoders<\/li>\n<li><strong>Evaluation<\/strong> is essential\u2014track Recall@k and Faithfulness at minimum<\/li>\n<li><strong>Cost optimization<\/strong> starts with query routing and caching<\/li>\n<\/ol>\n<h3 class=\"heading-element\" id=\"implementation-roadmap\"><span>Implementation Roadmap<\/span>\n <a href=\"#implementation-roadmap\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Phase 1: Basic RAG<\/strong> (Start here)<\/p>\n<ol>\n<li>Choose embedding model (start with <code>text-embedding-3-small<\/code>)<\/li>\n<li>Set up vector database (Pinecone for managed, Chroma for local)<\/li>\n<li>Implement basic chunking (512 tokens, 10% overlap)<\/li>\n<li>Build retrieve \u2192 generate pipeline<\/li>\n<li>Test with 20-30 sample questions<\/li>\n<\/ol>\n<p><strong>Phase 2: Quality Improvements<\/strong> (When basic RAG isn&rsquo;t enough)<\/p>\n<ul>\n<li>Add hybrid search if keyword matching matters<\/li>\n<li>Add reranking if relevant docs are retrieved but ranked low<\/li>\n<li>Adjust chunk sizes based on retrieval quality<\/li>\n<li>Build evaluation dataset, track metrics<\/li>\n<\/ul>\n<p><strong>Phase 3: Advanced Features<\/strong> (For complex use cases)<\/p>\n<ul>\n<li>Query transformation (HyDE, multi-query) for better retrieval<\/li>\n<li>CRAG\/Self-RAG for handling failures<\/li>\n<li>Agentic RAG for multi-hop reasoning<\/li>\n<li>Cost optimization for scale<\/li>\n<\/ul>\n<div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Start Simple, Iterate<\/strong><\/p>\n<p>Don&rsquo;t add complexity until you have evidence it&rsquo;s needed. Each additional component adds latency, cost, and maintenance burden.<\/p>\n<p><strong>The best RAG system is the simplest one that meets your quality requirements.<\/strong><\/p>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"quick-decision-guide\"><span>Quick Decision Guide<\/span>\n <a href=\"#quick-decision-guide\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Question<\/th>\n <th>If Yes \u2192<\/th>\n <th>If No \u2192<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>Do exact keywords matter?<\/td>\n <td>Use hybrid search<\/td>\n <td>Dense search is fine<\/td>\n <\/tr>\n <tr>\n <td>Are relevant docs ranked low?<\/td>\n <td>Add reranking<\/td>\n <td>Skip reranking<\/td>\n <\/tr>\n <tr>\n <td>Do queries use different vocabulary than docs?<\/td>\n <td>Try HyDE<\/td>\n <td>Basic retrieval is fine<\/td>\n <\/tr>\n <tr>\n <td>Are some queries too complex?<\/td>\n <td>Consider Agentic RAG<\/td>\n <td>Basic RAG is fine<\/td>\n <\/tr>\n <tr>\n <td>Is cost a concern?<\/td>\n <td>Implement query routing + caching<\/td>\n <td>Optimize later<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><hr>\n<p><strong>Further Reading:<\/strong><\/p>\n<ul>\n<li><a href=\"https:\/\/weaviate.io\/blog\/what-is-a-vector-database\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Weaviate: What is a Vector Database?<\/a><\/li>\n<li><a href=\"https:\/\/python.langchain.com\/docs\/use_cases\/question_answering\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">LangChain RAG Documentation<\/a><\/li>\n<li><a href=\"https:\/\/www.pinecone.io\/learn\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Pinecone Learning Center<\/a><\/li>\n<li><a href=\"https:\/\/arxiv.org\/abs\/2005.11401\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">RAG Paper (arXiv:2005.11401)<\/a><\/li>\n<li><a href=\"https:\/\/docs.ragas.io\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">RAGAS Evaluation Framework<\/a><\/li>\n<li><a href=\"https:\/\/arxiv.org\/abs\/2307.03172\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Lost in the Middle Paper<\/a><\/li>\n<\/ul>"},{"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","link":"https:\/\/samir.pages.dev\/posts\/serverless-newsletter-and-contact-system\/","pubDate":"Sat, 08 Nov 2025 10:00:00 +0000","author":"netproxy@duck.com (Samir)","guid":"https:\/\/samir.pages.dev\/posts\/serverless-newsletter-and-contact-system\/","category":"Projects","description":"<img src=\"https:\/\/samir.pages.dev\/files\/serverless-newsletter-and-contact-system-og.webp\" alt=\"featured image\" referrerpolicy=\"no-referrer\"><div class=\"details admonition success open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-check\" aria-hidden=\"true\"><\/i>\ud83d\ude80 Live Demo &amp; Source Code<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><ul>\n<li><strong>GitHub Repository<\/strong>: <a href=\"https:\/\/github.com\/SamirPaulb\/newsletter-and-contact-system\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">github.com\/SamirPaulb\/newsletter-and-contact-system<\/a> \u2b50<\/li>\n<li><strong>Production Ready<\/strong>: Running on Cloudflare&rsquo;s global edge network with 275+ locations worldwide<\/li>\n<li><strong>Tech Stack<\/strong>: JavaScript, Cloudflare Workers, KV Storage, GitHub Actions, Multiple Email Providers<\/li>\n<\/ul>\n<\/div>\n <\/div>\n<\/div>\n<h2 class=\"heading-element\" id=\"introduction-the-power-of-edge-computing\"><span>Introduction: The Power of Edge Computing<\/span>\n <a href=\"#introduction-the-power-of-edge-computing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Imagine building a newsletter system that can handle <strong>1 subscriber or 1 million subscribers<\/strong> without changing a single line of infrastructure code. No servers to provision, no databases to scale, no load balancers to configure. This is the reality of edge computing with Cloudflare Workers.<\/p>\n<p>In this comprehensive guide, I&rsquo;ll walk you through building a production-ready newsletter and contact management system that:<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Feature<\/th>\n <th>Description<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>\ud83c\udf0d <strong>Global Distribution<\/strong><\/td>\n <td>Runs on 275+ edge locations with &lt;50ms latency worldwide<\/td>\n <\/tr>\n <tr>\n <td>\ud83d\udce7 <strong>Multi-Provider Support<\/strong><\/td>\n <td>Gmail SMTP, MailerLite API, Cloudflare Email Routing<\/td>\n <\/tr>\n <tr>\n <td>\ud83d\udee1\ufe0f <strong>Intelligent Protection<\/strong><\/td>\n <td>Progressive CAPTCHA that adapts to threat levels<\/td>\n <\/tr>\n <tr>\n <td>\ud83d\udcca <strong>Infinite Scalability<\/strong><\/td>\n <td>Same code handles 1 to 1M+ subscribers<\/td>\n <\/tr>\n <tr>\n <td>\ud83d\udcb0 <strong>Cost Effective<\/strong><\/td>\n <td>~$5\/month for 1000 subscribers<\/td>\n <\/tr>\n <tr>\n <td>\ud83d\udd10 <strong>Enterprise Security<\/strong><\/td>\n <td>Zero Trust, rate limiting, DDoS protection built-in<\/td>\n <\/tr>\n <tr>\n <td>\ud83e\udd16 <strong>Full Automation<\/strong><\/td>\n <td>RSS monitoring, batch processing, auto-backups<\/td>\n <\/tr>\n <tr>\n <td>\ud83d\udcf0 <strong>Universal Feed Parser<\/strong><\/td>\n <td>Supports RSS 2.0, RSS 1.0, Atom, JSON Feed<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>\ud83d\udca1 Why Cloudflare Workers?<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">Traditional server-based solutions require complex scaling strategies, database replication, and significant DevOps overhead. Cloudflare Workers eliminate all of this by running your code at the edge, automatically scaling to handle any load.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"-system-homepage\"><span>\ud83c\udfe0 System Homepage<\/span>\n <a href=\"#-system-homepage\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/screenshot-newsletter-contact-management-home-page.webp\" title=\"Clean and modern homepage interface showing the newsletter &amp; contact management system with quick access to Subscribe, Unsubscribe, Contact, and Status pages. The interface displays system information including email provider configuration, RSS feed status, batch size settings, and bot protection status\" data-thumbnail=\"\/files\/screenshot-newsletter-contact-management-home-page.webp\" data-sub-html=\"<h2>Newsletter and Contact Management System Homepage<\/h2><p>Clean and modern homepage interface showing the newsletter &amp; contact management system with quick access to Subscribe, Unsubscribe, Contact, and Status pages. The interface displays system information including email provider configuration, RSS feed status, batch size settings, and bot protection status<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/screenshot-newsletter-contact-management-home-page.webp' alt=\"Newsletter and Contact Management System Homepage\"><\/a><figcaption class=\"image-caption\">Clean and modern homepage interface showing the newsletter & contact management system with quick access to Subscribe, Unsubscribe, Contact, and Status pages. The interface displays system information including email provider configuration, RSS feed status, batch size settings, and bot protection status<\/figcaption>\n <\/figure><\/p>\n<h2 class=\"heading-element\" id=\"how-serverless-works-under-the-hood\"><span>How Serverless Works Under the Hood<\/span>\n <a href=\"#how-serverless-works-under-the-hood\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Before diving into our newsletter system, let&rsquo;s understand the revolutionary technology powering it. Cloudflare Workers isn&rsquo;t just &ldquo;serverless&rdquo; - it&rsquo;s a fundamentally different approach to computing that enables true infinite scalability.<\/p>\n<h3 class=\"heading-element\" id=\"the-evolution-from-servers-to-serverless\"><span>The Evolution from Servers to Serverless<\/span>\n <a href=\"#the-evolution-from-servers-to-serverless\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/serverless-evolution-traditional-to-edge.webp\" title=\"The evolution of hosting infrastructure from physical servers (100% resource usage) through VMs, containers, FaaS, to edge computing with Cloudflare Workers (0.1% resource usage)\" data-thumbnail=\"\/files\/serverless-evolution-traditional-to-edge.webp\" data-sub-html=\"<h2>Evolution from Traditional Servers to Edge Computing with Cloudflare Workers<\/h2><p>The evolution of hosting infrastructure from physical servers (100% resource usage) through VMs, containers, FaaS, to edge computing with Cloudflare Workers (0.1% resource usage)<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/serverless-evolution-traditional-to-edge.webp' alt=\"Evolution from Traditional Servers to Edge Computing with Cloudflare Workers\"><\/a><figcaption class=\"image-caption\">The evolution of hosting infrastructure from physical servers (100% resource usage) through VMs, containers, FaaS, to edge computing with Cloudflare Workers (0.1% resource usage)<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"v8-isolates-the-secret-sauce\"><span>V8 Isolates: The Secret Sauce<\/span>\n <a href=\"#v8-isolates-the-secret-sauce\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Unlike traditional serverless platforms that use containers or VMs, Cloudflare Workers uses <strong>V8 Isolates<\/strong> - the same technology that powers Chrome&rsquo;s JavaScript engine. This is the key to achieving microsecond cold starts and infinite scalability.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/v8-isolates-vs-lambda-cold-start.webp\" title=\"Comparing cold starts: AWS Lambda with 500-3000ms container spin-up vs Cloudflare Workers with 5 microsecond isolate creation\" data-thumbnail=\"\/files\/v8-isolates-vs-lambda-cold-start.webp\" data-sub-html=\"<h2>V8 Isolates vs AWS Lambda Cold Start Comparison<\/h2><p>Comparing cold starts: AWS Lambda with 500-3000ms container spin-up vs Cloudflare Workers with 5 microsecond isolate creation<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/v8-isolates-vs-lambda-cold-start.webp' alt=\"V8 Isolates vs AWS Lambda Cold Start Comparison\"><\/a><figcaption class=\"image-caption\">Comparing cold starts: AWS Lambda with 500-3000ms container spin-up vs Cloudflare Workers with 5 microsecond isolate creation<\/figcaption>\n <\/figure><\/p>\n<h4 class=\"heading-element\" id=\"what-are-v8-isolates\"><span>What are V8 Isolates?<\/span>\n <a href=\"#what-are-v8-isolates\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>V8 Isolates are lightweight JavaScript execution contexts that provide:<\/p>\n<ol>\n<li><strong>Memory Isolation<\/strong>: Each worker runs in its own memory space<\/li>\n<li><strong>CPU Isolation<\/strong>: Prevents one worker from affecting others<\/li>\n<li><strong>Zero Cold Starts<\/strong>: New isolates spin up in microseconds<\/li>\n<li><strong>Shared Runtime<\/strong>: Multiple isolates run in the same process<\/li>\n<\/ol>\n<pre><code>\/\/ This is what happens under the hood when your Worker receives a request\n\n\/\/ 1. Request arrives at edge location\n\/\/ 2. V8 engine (already running) creates new isolate\nclass WorkerIsolate {\n constructor(code) {\n this.context = new V8.Context(); \/\/ Microseconds\n this.globals = this.setupGlobals();\n this.code = code;\n }\n\n async handleRequest(request) {\n \/\/ Your worker code executes here\n \/\/ Completely isolated from other workers\n return await this.code.fetch(request);\n }\n}\n\n\/\/ 3. Isolate destroyed after request completes\n\/\/ Total overhead: ~0.005ms vs 500-3000ms for containers<\/code><\/pre><h3 class=\"heading-element\" id=\"the-edge-network-architecture\"><span>The Edge Network Architecture<\/span>\n <a href=\"#the-edge-network-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Cloudflare Workers run on every one of Cloudflare&rsquo;s 275+ edge locations worldwide. Here&rsquo;s how a request flows through the system:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/cloudflare-edge-request-flow-sequence.webp\" title=\"Complete request flow through Cloudflare&#39;s edge network showing DNS resolution, V8 engine processing, isolate creation, KV storage access, and response generation in under 50ms\" data-thumbnail=\"\/files\/cloudflare-edge-request-flow-sequence.webp\" data-sub-html=\"<h2>Cloudflare Edge Request Flow Sequence Diagram<\/h2><p>Complete request flow through Cloudflare&#39;s edge network showing DNS resolution, V8 engine processing, isolate creation, KV storage access, and response generation in under 50ms<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/cloudflare-edge-request-flow-sequence.webp' alt=\"Cloudflare Edge Request Flow Sequence Diagram\"><\/a><figcaption class=\"image-caption\">Complete request flow through Cloudflare's edge network showing DNS resolution, V8 engine processing, isolate creation, KV storage access, and response generation in under 50ms<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"request-handling-deep-dive\"><span>Request Handling Deep Dive<\/span>\n <a href=\"#request-handling-deep-dive\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Let&rsquo;s examine exactly what happens when a request hits your Worker:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/worker-request-lifecycle-execution.webp\" title=\"Detailed request lifecycle showing TLS termination, WAF, rate limiting, caching, isolate creation, and resource limits (10-50ms CPU, 128MB memory, 50 subrequests)\" data-thumbnail=\"\/files\/worker-request-lifecycle-execution.webp\" data-sub-html=\"<h2>Worker Request Lifecycle and Execution Flow<\/h2><p>Detailed request lifecycle showing TLS termination, WAF, rate limiting, caching, isolate creation, and resource limits (10-50ms CPU, 128MB memory, 50 subrequests)<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/worker-request-lifecycle-execution.webp' alt=\"Worker Request Lifecycle and Execution Flow\"><\/a><figcaption class=\"image-caption\">Detailed request lifecycle showing TLS termination, WAF, rate limiting, caching, isolate creation, and resource limits (10-50ms CPU, 128MB memory, 50 subrequests)<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"memory-management-in-isolates\"><span>Memory Management in Isolates<\/span>\n <a href=\"#memory-management-in-isolates\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>One of the most impressive aspects of V8 Isolates is their memory efficiency:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/v8-isolates-memory-efficiency-comparison.webp\" title=\"Memory comparison: Traditional containers require 1.5GB for 3 concurrent requests vs V8 Isolates needing only 130MB for the same load\" data-thumbnail=\"\/files\/v8-isolates-memory-efficiency-comparison.webp\" data-sub-html=\"<h2>V8 Isolates Memory Efficiency vs Traditional Containers<\/h2><p>Memory comparison: Traditional containers require 1.5GB for 3 concurrent requests vs V8 Isolates needing only 130MB for the same load<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/v8-isolates-memory-efficiency-comparison.webp' alt=\"V8 Isolates Memory Efficiency vs Traditional Containers\"><\/a><figcaption class=\"image-caption\">Memory comparison: Traditional containers require 1.5GB for 3 concurrent requests vs V8 Isolates needing only 130MB for the same load<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"global-distribution--anycast-routing\"><span>Global Distribution &amp; Anycast Routing<\/span>\n <a href=\"#global-distribution--anycast-routing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Cloudflare Workers leverage Anycast IP routing to automatically route users to the nearest edge location:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/cloudflare-anycast-global-distribution.webp\" title=\"Anycast routing automatically directs users to the nearest edge location with the same worker code running everywhere and KV storage providing eventual consistency\" data-thumbnail=\"\/files\/cloudflare-anycast-global-distribution.webp\" data-sub-html=\"<h2>Cloudflare Anycast Global Distribution Network<\/h2><p>Anycast routing automatically directs users to the nearest edge location with the same worker code running everywhere and KV storage providing eventual consistency<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/cloudflare-anycast-global-distribution.webp' alt=\"Cloudflare Anycast Global Distribution Network\"><\/a><figcaption class=\"image-caption\">Anycast routing automatically directs users to the nearest edge location with the same worker code running everywhere and KV storage providing eventual consistency<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"cold-starts-comparison\"><span>Cold Starts Comparison<\/span>\n <a href=\"#cold-starts-comparison\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The elimination of cold starts is what makes Workers truly revolutionary:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/cold-start-comparison-lambda-vs-workers.webp\" title=\"Gantt chart showing cold start times: AWS Lambda (850ms total), Azure Functions (880ms total), vs Cloudflare Workers (50ms total with 1ms isolate creation)\" data-thumbnail=\"\/files\/cold-start-comparison-lambda-vs-workers.webp\" data-sub-html=\"<h2>Cold Start Comparison: Lambda vs Azure vs Workers<\/h2><p>Gantt chart showing cold start times: AWS Lambda (850ms total), Azure Functions (880ms total), vs Cloudflare Workers (50ms total with 1ms isolate creation)<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/cold-start-comparison-lambda-vs-workers.webp' alt=\"Cold Start Comparison: Lambda vs Azure vs Workers\"><\/a><figcaption class=\"image-caption\">Gantt chart showing cold start times: AWS Lambda (850ms total), Azure Functions (880ms total), vs Cloudflare Workers (50ms total with 1ms isolate creation)<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"the-magic-of-zero-infrastructure\"><span>The Magic of Zero Infrastructure<\/span>\n <a href=\"#the-magic-of-zero-infrastructure\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Here&rsquo;s what you DON&rsquo;T need to manage with Cloudflare Workers:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/cloudflare-workers-zero-infrastructure-mindmap.webp\" title=\"Comprehensive mindmap showing everything you don&#39;t need to manage with Cloudflare Workers: no servers, no operations, no complexity, with built-in features like auto-scaling and DDoS protection\" data-thumbnail=\"\/files\/cloudflare-workers-zero-infrastructure-mindmap.webp\" data-sub-html=\"<h2>Cloudflare Workers Zero Infrastructure Mindmap<\/h2><p>Comprehensive mindmap showing everything you don&#39;t need to manage with Cloudflare Workers: no servers, no operations, no complexity, with built-in features like auto-scaling and DDoS protection<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/cloudflare-workers-zero-infrastructure-mindmap.webp' alt=\"Cloudflare Workers Zero Infrastructure Mindmap\"><\/a><figcaption class=\"image-caption\">Comprehensive mindmap showing everything you don't need to manage with Cloudflare Workers: no servers, no operations, no complexity, with built-in features like auto-scaling and DDoS protection<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"resource-limits-and-pricing-model\"><span>Resource Limits and Pricing Model<\/span>\n <a href=\"#resource-limits-and-pricing-model\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Understanding the limits helps you design efficient Workers:<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Resource<\/th>\n <th>Free Tier<\/th>\n <th>Paid Plan<\/th>\n <th>Why It Matters<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Requests<\/strong><\/td>\n <td>100,000\/day<\/td>\n <td>10M\/month + $0.50\/million<\/td>\n <td>Generous for most applications<\/td>\n <\/tr>\n <tr>\n <td><strong>CPU Time<\/strong><\/td>\n <td>10ms\/request<\/td>\n <td>50ms\/request<\/td>\n <td>Forces efficient code<\/td>\n <\/tr>\n <tr>\n <td><strong>Memory<\/strong><\/td>\n <td>128MB<\/td>\n <td>128MB<\/td>\n <td>Sufficient for most use cases<\/td>\n <\/tr>\n <tr>\n <td><strong>Script Size<\/strong><\/td>\n <td>1MB<\/td>\n <td>10MB<\/td>\n <td>Keeps workers fast<\/td>\n <\/tr>\n <tr>\n <td><strong>Subrequests<\/strong><\/td>\n <td>50<\/td>\n <td>50<\/td>\n <td>Enough for complex workflows<\/td>\n <\/tr>\n <tr>\n <td><strong>Environment Variables<\/strong><\/td>\n <td>64<\/td>\n <td>64<\/td>\n <td>Use KV for more config<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>\u26a1 Performance Optimization Tips<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><ol>\n<li><strong>Minimize subrequests<\/strong>: Batch API calls when possible<\/li>\n<li><strong>Use KV caching<\/strong>: Cache expensive computations<\/li>\n<li><strong>Optimize script size<\/strong>: Use tree-shaking and minification<\/li>\n<li><strong>Leverage cache API<\/strong>: Cache responses at the edge<\/li>\n<li><strong>Avoid CPU-intensive operations<\/strong>: Offload to external services if needed<\/li>\n<\/ol>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"why-this-architecture-enables-infinite-scale\"><span>Why This Architecture Enables Infinite Scale<\/span>\n <a href=\"#why-this-architecture-enables-infinite-scale\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The combination of V8 Isolates and edge distribution creates a system that can handle any load:<\/p>\n<ol>\n<li><strong>No Shared State<\/strong>: Each request is independent<\/li>\n<li><strong>Instant Scaling<\/strong>: New isolates created in microseconds<\/li>\n<li><strong>Global Distribution<\/strong>: Load spread across 275+ locations<\/li>\n<li><strong>Resource Isolation<\/strong>: One worker can&rsquo;t affect others<\/li>\n<li><strong>Automatic Failover<\/strong>: If one location fails, traffic routes elsewhere<\/li>\n<\/ol>\n<pre><code>\/\/ This same code handles 1 request or 1 billion requests\nexport default {\n async fetch(request, env) {\n \/\/ No difference in code for any scale\n \/\/ Cloudflare handles all the complexity\n return new Response(&#34;Hello World&#34;);\n }\n}<\/code><\/pre><div class=\"details admonition success open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-check\" aria-hidden=\"true\"><\/i>\ud83c\udfaf The Bottom Line<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">Cloudflare Workers represent a paradigm shift in computing. By leveraging V8 Isolates instead of containers, eliminating cold starts, and distributing execution globally, they achieve what was previously impossible: <strong>true infinite scalability with zero infrastructure management<\/strong>.<\/div>\n <\/div>\n<\/div>\n<h2 class=\"heading-element\" id=\"system-architecture-overview\"><span>System Architecture Overview<\/span>\n <a href=\"#system-architecture-overview\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Our newsletter system leverages Cloudflare&rsquo;s global edge network to achieve unprecedented scalability and performance. Here&rsquo;s the complete architecture:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/newsletter-system-architecture-overview.webp\" title=\"Complete architecture showing global edge network, worker platform with KV storage, multiple email providers, scheduled jobs, GitHub backups, and security layers\" data-thumbnail=\"\/files\/newsletter-system-architecture-overview.webp\" data-sub-html=\"<h2>Newsletter System Architecture Overview on Cloudflare Workers<\/h2><p>Complete architecture showing global edge network, worker platform with KV storage, multiple email providers, scheduled jobs, GitHub backups, and security layers<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/newsletter-system-architecture-overview.webp' alt=\"Newsletter System Architecture Overview on Cloudflare Workers\"><\/a><figcaption class=\"image-caption\">Complete architecture showing global edge network, worker platform with KV storage, multiple email providers, scheduled jobs, GitHub backups, and security layers<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"key-architectural-decisions\"><span>Key Architectural Decisions<\/span>\n <a href=\"#key-architectural-decisions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li><strong>Serverless at the Edge<\/strong>: No servers, containers, or VMs to manage<\/li>\n<li><strong>Distributed KV Storage<\/strong>: Automatic replication across regions<\/li>\n<li><strong>Multi-Provider Email<\/strong>: Failover capability and provider flexibility<\/li>\n<li><strong>Progressive Security<\/strong>: Intelligent bot protection that adapts to threats<\/li>\n<li><strong>Automated Operations<\/strong>: Self-maintaining with cleanup and backups<\/li>\n<\/ol>\n<h2 class=\"heading-element\" id=\"core-features-deep-dive\"><span>Core Features Deep Dive<\/span>\n <a href=\"#core-features-deep-dive\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"1-multi-provider-email-architecture\"><span>1. Multi-Provider Email Architecture<\/span>\n <a href=\"#1-multi-provider-email-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The system implements a <strong>factory pattern<\/strong> for seamless provider switching:<\/p>\n<pre><code>\/\/ Email Provider Factory Pattern with Validation\nexport class EmailFactory {\n static createProvider(config, env) {\n const provider = config.EMAIL_PROVIDER?.toLowerCase();\n\n switch (provider) {\n case &#39;gmail&#39;:\n \/\/ Validate Gmail configuration\n const gmailValidation = GmailProvider.validateConfig(config);\n if (!gmailValidation.valid) {\n throw new Error(`Gmail config errors: ${gmailValidation.errors.join(&#39;, &#39;)}`);\n }\n return new GmailProvider(config);\n\n case &#39;mailerlite&#39;:\n \/\/ Validate MailerLite configuration\n if (!config.MAILERLITE_API_TOKEN) {\n throw new Error(&#39;MailerLite API token required&#39;);\n }\n return new MailerLiteProvider(config, env);\n\n case &#39;worker-email&#39;:\n \/\/ Validate Worker Email configuration\n const workerValidation = WorkerEmailProvider.validateConfig(config);\n if (!workerValidation.valid) {\n throw new Error(`Worker Email errors: ${workerValidation.errors.join(&#39;, &#39;)}`);\n }\n return new WorkerEmailProvider(config, env);\n\n default:\n throw new Error(`Unknown provider: ${provider}`);\n }\n }\n\n \/\/ Send newsletter with automatic retry and fallback\n static async sendNewsletter(config, env, { recipients, post }) {\n const provider = this.createProvider(config, env);\n\n \/\/ Create responsive HTML email\n const html = this.createNewsletterHtml(post, config);\n const text = this.createNewsletterText(post, config);\n\n \/\/ Send with batch processing\n return await provider.sendBatchEmail({\n recipients: recipients,\n subject: `New Post: ${post.title}`,\n html: html,\n text: text\n });\n }\n}<\/code><\/pre><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>\ud83d\udce7 Email Provider Comparison<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Provider<\/th>\n <th>Best For<\/th>\n <th>Limits<\/th>\n <th>Cost<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Gmail SMTP<\/strong><\/td>\n <td>Small lists<\/td>\n <td>500\/day (free)<\/td>\n <td>Free<\/td>\n <\/tr>\n <tr>\n <td><strong>MailerLite<\/strong><\/td>\n <td>Growing lists<\/td>\n <td>1000\/month free<\/td>\n <td>$10+<\/td>\n <\/tr>\n <tr>\n <td><strong>Worker Email<\/strong><\/td>\n <td>Custom domains<\/td>\n <td>Unlimited*<\/td>\n <td>Pay-per-use<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"2-intelligent-bot-protection\"><span>2. Intelligent Bot Protection<\/span>\n <a href=\"#2-intelligent-bot-protection\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Our system implements a <strong>progressive CAPTCHA system<\/strong> that only challenges suspicious users:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/progressive-captcha-bot-protection-flow.webp\" title=\"State diagram showing intelligent bot protection that progressively challenges suspicious users with Turnstile CAPTCHA based on rate limiting and abuse detection\" data-thumbnail=\"\/files\/progressive-captcha-bot-protection-flow.webp\" data-sub-html=\"<h2>Progressive CAPTCHA Bot Protection Flow<\/h2><p>State diagram showing intelligent bot protection that progressively challenges suspicious users with Turnstile CAPTCHA based on rate limiting and abuse detection<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/progressive-captcha-bot-protection-flow.webp' alt=\"Progressive CAPTCHA Bot Protection Flow\"><\/a><figcaption class=\"image-caption\">State diagram showing intelligent bot protection that progressively challenges suspicious users with Turnstile CAPTCHA based on rate limiting and abuse detection<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"3-universal-feed-parser---rss-atom-json-feed\"><span>3. Universal Feed Parser - RSS, Atom, JSON Feed<\/span>\n <a href=\"#3-universal-feed-parser---rss-atom-json-feed\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The system includes a <strong>powerful universal feed parser<\/strong> that automatically detects and parses any feed format:<\/p>\n<pre><code>\/\/ Universal Feed Parser - Supports multiple formats\nexport function parseFeed(content, contentType = &#39;&#39;) {\n \/\/ Auto-detect format\n const trimmed = content.trim();\n\n \/\/ Check if it&#39;s JSON Feed\n if (trimmed.startsWith(&#39;{&#39;) || contentType.includes(&#39;json&#39;)) {\n return parseJsonFeed(trimmed);\n }\n\n \/\/ Otherwise, parse as XML-based feed\n return parseXmlFeed(trimmed);\n}\n\n\/\/ Intelligent format detection\nexport function detectFeedType(content) {\n const trimmed = content.trim();\n\n \/\/ JSON Feed detection\n if (trimmed.startsWith(&#39;{&#39;)) {\n const json = JSON.parse(trimmed);\n if (json.version?.includes(&#39;jsonfeed.org&#39;)) {\n return &#39;json-feed&#39;;\n }\n }\n\n \/\/ Atom detection\n if (trimmed.includes(&#39;xmlns=&#34;http:\/\/www.w3.org\/2005\/Atom&#34;&#39;)) {\n return &#39;atom&#39;;\n }\n\n \/\/ RSS 1.0\/RDF detection\n if (trimmed.includes(&#39;&lt;rdf:RDF&#39;) || trimmed.includes(&#39;purl.org\/rss\/1.0&#39;)) {\n return &#39;rss-1.0&#39;;\n }\n\n \/\/ RSS 2.0 detection\n if (trimmed.includes(&#39;&lt;rss&#39;) &amp;&amp; trimmed.includes(&#39;version=&#34;2.0&#34;&#39;)) {\n return &#39;rss-2.0&#39;;\n }\n\n return &#39;unknown&#39;;\n}<\/code><\/pre><div class=\"details admonition success open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-check\" aria-hidden=\"true\"><\/i>\ud83d\udcf0 Supported Feed Formats<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Format<\/th>\n <th>Example<\/th>\n <th>Auto-Detected<\/th>\n <th>Features<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>RSS 2.0<\/strong><\/td>\n <td>WordPress, Blogger<\/td>\n <td>\u2705<\/td>\n <td>Most common, widely supported<\/td>\n <\/tr>\n <tr>\n <td><strong>RSS 1.0\/RDF<\/strong><\/td>\n <td>Older CMS systems<\/td>\n <td>\u2705<\/td>\n <td>Namespace support<\/td>\n <\/tr>\n <tr>\n <td><strong>Atom<\/strong><\/td>\n <td>GitHub, modern blogs<\/td>\n <td>\u2705<\/td>\n <td>Modern XML format<\/td>\n <\/tr>\n <tr>\n <td><strong>JSON Feed<\/strong><\/td>\n <td>Micro.blog, modern APIs<\/td>\n <td>\u2705<\/td>\n <td>JSON-based, easy to parse<\/td>\n <\/tr>\n <tr>\n <td><strong>Custom Namespaces<\/strong><\/td>\n <td>Dublin Core, Content<\/td>\n <td>\u2705<\/td>\n <td>dc:creator, content:encoded<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"4-automated-rss-processing-workflow\"><span>4. Automated RSS Processing Workflow<\/span>\n <a href=\"#4-automated-rss-processing-workflow\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/\/ Complete RSS Processing Pipeline\nasync function discoverFromRssAndQueue(env, config) {\n \/\/ 1. Validate feed URL\n if (!isValidFeedUrl(config.RSS_FEED_URL)) {\n console.error(&#39;Invalid feed URL&#39;);\n return;\n }\n\n \/\/ 2. Fetch with proper headers\n const response = await fetch(config.RSS_FEED_URL, {\n headers: {\n &#39;User-Agent&#39;: config.USER_AGENT,\n &#39;Accept&#39;: &#39;application\/rss&#43;xml, application\/atom&#43;xml, application\/xml, text\/xml, application\/json, *\/*&#39;\n }\n });\n\n \/\/ 3. Parse with universal parser\n const feedContent = await response.text();\n const feedType = detectFeedType(feedContent);\n console.log(`Detected feed type: ${feedType}`);\n\n const items = parseFeed(feedContent, response.headers.get(&#39;content-type&#39;));\n\n \/\/ 4. Process new items\n for (const item of items) {\n \/\/ Check if already sent\n const already = await alreadySent(env, config, item.guid);\n if (already) continue;\n\n \/\/ Queue for sending\n await createEmailQueue({\n post: {\n url: item.url,\n title: item.title,\n description: item.description,\n author: item.author,\n categories: item.categories,\n pubDate: item.pubDate\n },\n subscribers: await getAllSubscribers(env, config),\n feedType: feedType\n });\n }\n}<\/code><\/pre><h2 class=\"heading-element\" id=\"data-flow-diagrams\"><span>Data Flow Diagrams<\/span>\n <a href=\"#data-flow-diagrams\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"newsletter-subscription-flow\"><span>Newsletter Subscription Flow<\/span>\n <a href=\"#newsletter-subscription-flow\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/newsletter-subscription-flow-sequence.webp\" title=\"Complete subscription flow showing form submission, Turnstile verification, KV storage check, welcome email sending, and GitHub backup queuing\" data-thumbnail=\"\/files\/newsletter-subscription-flow-sequence.webp\" data-sub-html=\"<h2>Newsletter Subscription Flow Sequence Diagram<\/h2><p>Complete subscription flow showing form submission, Turnstile verification, KV storage check, welcome email sending, and GitHub backup queuing<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/newsletter-subscription-flow-sequence.webp' alt=\"Newsletter Subscription Flow Sequence Diagram\"><\/a><figcaption class=\"image-caption\">Complete subscription flow showing form submission, Turnstile verification, KV storage check, welcome email sending, and GitHub backup queuing<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"batch-email-processing\"><span>Batch Email Processing<\/span>\n <a href=\"#batch-email-processing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/batch-email-processing-pipeline.webp\" title=\"Email queue processing showing batch creation (100 emails each), sequential sending with 5-minute delays, retry logic, and dead letter queue for failed emails\" data-thumbnail=\"\/files\/batch-email-processing-pipeline.webp\" data-sub-html=\"<h2>Batch Email Processing Pipeline with Error Handling<\/h2><p>Email queue processing showing batch creation (100 emails each), sequential sending with 5-minute delays, retry logic, and dead letter queue for failed emails<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/batch-email-processing-pipeline.webp' alt=\"Batch Email Processing Pipeline with Error Handling\"><\/a><figcaption class=\"image-caption\">Email queue processing showing batch creation (100 emails each), sequential sending with 5-minute delays, retry logic, and dead letter queue for failed emails<\/figcaption>\n <\/figure><\/p>\n<h2 class=\"heading-element\" id=\"quick-start-guide\"><span>Quick Start Guide<\/span>\n <a href=\"#quick-start-guide\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"-deploy-in-5-minutes\"><span>\ud83d\ude80 Deploy in 5 Minutes<\/span>\n <a href=\"#-deploy-in-5-minutes\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>\u26a1 One-Click Deploy<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p>The fastest way to get started is to fork the repository and deploy with Wrangler CLI:<\/p>\n<pre><code># Quick deploy\ngit clone https:\/\/github.com\/SamirPaulb\/newsletter-and-contact-system.git\ncd cloudflare-workers\nnpm install\nwrangler deploy<\/code><\/pre><\/div>\n <\/div>\n<\/div>\n<h2 class=\"heading-element\" id=\"implementation-guide\"><span>Implementation Guide<\/span>\n <a href=\"#implementation-guide\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"step-1-project-setup\"><span>Step 1: Project Setup<\/span>\n <a href=\"#step-1-project-setup\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Clone the repository\ngit clone https:\/\/github.com\/SamirPaulb\/newsletter-and-contact-system.git\ncd cloudflare-workers\n\n# Install dependencies\nnpm install\n\n# Configure wrangler.toml\ncp wrangler.example.toml wrangler.toml<\/code><\/pre><h3 class=\"heading-element\" id=\"step-2-configuration\"><span>Step 2: Configuration<\/span>\n <a href=\"#step-2-configuration\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Edit <code>wrangler.toml<\/code> with your settings:<\/p>\n<pre><code>name = &#34;newsletter-system&#34;\nmain = &#34;src\/index.js&#34;\ncompatibility_date = &#34;2024-12-31&#34;\n\n[triggers]\ncrons = [\n &#34;0 0 * * *&#34;, # Daily at midnight UTC\n &#34;0 0 * * sat&#34; # Weekly on Saturday\n]\n\n[[kv_namespaces]]\nbinding = &#34;KV&#34;\nid = &#34;your-kv-namespace-id&#34;\n\n[vars]\nEMAIL_PROVIDER = &#34;gmail&#34;\nRSS_FEED_URL = &#34;https:\/\/your-site.com\/feed.xml&#34;\nBATCH_SIZE = 100\nBATCH_WAIT_MINUTES = 5<\/code><\/pre><h3 class=\"heading-element\" id=\"step-3-set-secrets\"><span>Step 3: Set Secrets<\/span>\n <a href=\"#step-3-set-secrets\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Required secrets\nwrangler secret put GMAIL_USER\nwrangler secret put GMAIL_PASSWORD\nwrangler secret put GITHUB_TOKEN\nwrangler secret put TURNSTILE_SITE_KEY\nwrangler secret put TURNSTILE_SECRET_KEY<\/code><\/pre><h3 class=\"heading-element\" id=\"step-4-deploy\"><span>Step 4: Deploy<\/span>\n <a href=\"#step-4-deploy\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Deploy to Cloudflare Workers\nwrangler deploy\n\n# Verify deployment\ncurl https:\/\/your-worker.workers.dev\/health<\/code><\/pre><h2 class=\"heading-element\" id=\"security-deep-dive\"><span>Security Deep Dive<\/span>\n <a href=\"#security-deep-dive\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"multi-layer-defense-strategy\"><span>Multi-Layer Defense Strategy<\/span>\n <a href=\"#multi-layer-defense-strategy\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/multi-layer-security-defense-strategy.webp\" title=\"Four-layer security architecture: Edge protection (DDoS, WAF), Application security (CORS, Turnstile), Data security (encryption, sanitization), and Access control (Zero Trust, RBAC)\" data-thumbnail=\"\/files\/multi-layer-security-defense-strategy.webp\" data-sub-html=\"<h2>Multi-Layer Security Defense Strategy<\/h2><p>Four-layer security architecture: Edge protection (DDoS, WAF), Application security (CORS, Turnstile), Data security (encryption, sanitization), and Access control (Zero Trust, RBAC)<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/multi-layer-security-defense-strategy.webp' alt=\"Multi-Layer Security Defense Strategy\"><\/a><figcaption class=\"image-caption\">Four-layer security architecture: Edge protection (DDoS, WAF), Application security (CORS, Turnstile), Data security (encryption, sanitization), and Access control (Zero Trust, RBAC)<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"key-security-features\"><span>Key Security Features<\/span>\n <a href=\"#key-security-features\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li><strong>Rate Limiting<\/strong>: Two-tier system (per-IP and global)<\/li>\n<li><strong>CORS Protection<\/strong>: Restricted to your domain only<\/li>\n<li><strong>Input Validation<\/strong>: Email validation with disposable domain blocking<\/li>\n<li><strong>XSS Prevention<\/strong>: All user inputs sanitized<\/li>\n<li><strong>Bot Protection<\/strong>: Progressive CAPTCHA with Turnstile<\/li>\n<li><strong>Zero Trust Integration<\/strong>: Status page requires authentication<\/li>\n<\/ol>\n<div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>\ud83d\udd12 Security Best Practices<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><ul>\n<li>Never commit secrets to Git<\/li>\n<li>Use Cloudflare&rsquo;s secret management<\/li>\n<li>Enable 2FA on all accounts<\/li>\n<li>Regularly rotate API keys<\/li>\n<li>Monitor rate limit logs<\/li>\n<\/ul>\n<\/div>\n <\/div>\n<\/div>\n<h2 class=\"heading-element\" id=\"live-demo--user-interface\"><span>Live Demo &amp; User Interface<\/span>\n <a href=\"#live-demo--user-interface\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"-interactive-forms\"><span>\ud83c\udfa8 Interactive Forms<\/span>\n <a href=\"#-interactive-forms\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Experience the system in action with these live, embedded forms:<\/p>\n<h4 class=\"heading-element\" id=\"-contact-form-with-auto-subscribe\"><span>\ud83d\udcac Contact Form with Auto-Subscribe<\/span>\n <a href=\"#-contact-form-with-auto-subscribe\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><div style=\"width: 100%; max-width: 600px; margin: 20px auto; padding: 0; box-sizing: border-box;\">\n <iframe src=\"https:\/\/web.spq.workers.dev\/contact\" title=\"Contact Form\"\n scrolling=\"no\"\n style=\"width: 100%; height: 890px; border: none; display: block; background: transparent; overflow: hidden;\"\n loading=\"lazy\" allowtransparency=\"true\">\n <\/iframe>\n<\/div>\n<div class=\"details admonition example\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-list-ul\" aria-hidden=\"true\"><\/i>\ud83d\udcf8 Screenshot: Contact Form<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/screenshot-contact-page.webp\" title=\"Comprehensive contact form interface with fields for name, email, phone, subject, and message. Features an auto-subscribe checkbox option and Cloudflare Turnstile CAPTCHA for security. The form provides instant validation feedback and confirmation emails upon submission\" data-thumbnail=\"\/files\/screenshot-contact-page.webp\" data-sub-html=\"<h2>Contact Form with Auto-Subscribe Option<\/h2><p>Comprehensive contact form interface with fields for name, email, phone, subject, and message. Features an auto-subscribe checkbox option and Cloudflare Turnstile CAPTCHA for security. The form provides instant validation feedback and confirmation emails upon submission<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/screenshot-contact-page.webp' alt=\"Contact Form with Auto-Subscribe Option\"><\/a><figcaption class=\"image-caption\">Comprehensive contact form interface with fields for name, email, phone, subject, and message. Features an auto-subscribe checkbox option and Cloudflare Turnstile CAPTCHA for security. The form provides instant validation feedback and confirmation emails upon submission<\/figcaption>\n <\/figure><\/p>\n<\/div>\n <\/div>\n<\/div>\n<h4 class=\"heading-element\" id=\"-newsletter-subscription-form\"><span>\ud83d\udcec Newsletter Subscription Form<\/span>\n <a href=\"#-newsletter-subscription-form\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><div style=\"width: 100%; max-width: 600px; margin: 20px auto; padding: 0; box-sizing: border-box;\">\n <iframe src=\"https:\/\/web.spq.workers.dev\/subscribe\" title=\"Subscribe to Newsletter\"\n scrolling=\"no\"\n style=\"width: 100%; height: 540px; border: none; display: block; background: transparent; overflow: hidden;\"\n loading=\"lazy\" allowtransparency=\"true\">\n <\/iframe>\n<\/div>\n<div class=\"details admonition example\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-list-ul\" aria-hidden=\"true\"><\/i>\ud83d\udcf8 Screenshot: Subscription Form<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/screenshot-subscribe-page.webp\" title=\"Clean newsletter subscription interface featuring an email input field and Cloudflare Turnstile CAPTCHA verification. The form ensures bot protection while maintaining user-friendly design with instant validation feedback\" data-thumbnail=\"\/files\/screenshot-subscribe-page.webp\" data-sub-html=\"<h2>Newsletter Subscription Form with Turnstile CAPTCHA<\/h2><p>Clean newsletter subscription interface featuring an email input field and Cloudflare Turnstile CAPTCHA verification. The form ensures bot protection while maintaining user-friendly design with instant validation feedback<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/screenshot-subscribe-page.webp' alt=\"Newsletter Subscription Form with Turnstile CAPTCHA\"><\/a><figcaption class=\"image-caption\">Clean newsletter subscription interface featuring an email input field and Cloudflare Turnstile CAPTCHA verification. The form ensures bot protection while maintaining user-friendly design with instant validation feedback<\/figcaption>\n <\/figure><\/p>\n<\/div>\n <\/div>\n<\/div>\n<h4 class=\"heading-element\" id=\"-unsubscribe-page\"><span>\ud83d\udc4b Unsubscribe Page<\/span>\n <a href=\"#-unsubscribe-page\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><div style=\"width: 100%; max-width: 600px; margin: 20px auto; padding: 0; box-sizing: border-box;\">\n <iframe src=\"https:\/\/web.spq.workers.dev\/unsubscribe\" title=\"Unsubscribe from Newsletter\"\n scrolling=\"no\"\n style=\"width: 100%; height: 540px; border: none; display: block; background: transparent; overflow: hidden;\"\n loading=\"lazy\" allowtransparency=\"true\">\n <\/iframe>\n<\/div>\n<div class=\"details admonition example\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-list-ul\" aria-hidden=\"true\"><\/i>\ud83d\udcf8 Screenshot: Unsubscribe Page<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/screenshot-unsubscribe-page.webp\" title=\"Simple unsubscribe interface with email input field and Turnstile CAPTCHA. Users can quickly opt-out from newsletters with a single click while the system maintains security through bot protection\" data-thumbnail=\"\/files\/screenshot-unsubscribe-page.webp\" data-sub-html=\"<h2>Newsletter Unsubscribe Page<\/h2><p>Simple unsubscribe interface with email input field and Turnstile CAPTCHA. Users can quickly opt-out from newsletters with a single click while the system maintains security through bot protection<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/screenshot-unsubscribe-page.webp' alt=\"Newsletter Unsubscribe Page\"><\/a><figcaption class=\"image-caption\">Simple unsubscribe interface with email input field and Turnstile CAPTCHA. Users can quickly opt-out from newsletters with a single click while the system maintains security through bot protection<\/figcaption>\n <\/figure><\/p>\n<\/div>\n <\/div>\n<\/div>\n<h2 class=\"heading-element\" id=\"performance--scalability\"><span>Performance &amp; Scalability<\/span>\n <a href=\"#performance--scalability\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"benchmark-results\"><span>Benchmark Results<\/span>\n <a href=\"#benchmark-results\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/load-test-scalability-benchmark.webp\" title=\"Load test results showing zero overhead scaling from 1 request\/second to 100,000 requests\/second with consistent performance\" data-thumbnail=\"\/files\/load-test-scalability-benchmark.webp\" data-sub-html=\"<h2>Load Test Scalability Benchmark Results<\/h2><p>Load test results showing zero overhead scaling from 1 request\/second to 100,000 requests\/second with consistent performance<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/load-test-scalability-benchmark.webp' alt=\"Load Test Scalability Benchmark Results\"><\/a><figcaption class=\"image-caption\">Load test results showing zero overhead scaling from 1 request\/second to 100,000 requests\/second with consistent performance<\/figcaption>\n <\/figure><\/p>\n<div class=\"details admonition success open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-check\" aria-hidden=\"true\"><\/i>\ud83d\udcca Performance Metrics<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><ul>\n<li><strong>Response Time<\/strong>: &lt;50ms globally (P95)<\/li>\n<li><strong>Throughput<\/strong>: 100,000+ requests\/second<\/li>\n<li><strong>Availability<\/strong>: 99.99% SLA<\/li>\n<li><strong>Storage<\/strong>: Unlimited KV pairs<\/li>\n<li><strong>Email Sending<\/strong>: 500-2000\/day (provider dependent)<\/li>\n<\/ul>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"cost-analysis\"><span>Cost Analysis<\/span>\n <a href=\"#cost-analysis\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Subscribers<\/th>\n <th>Monthly Cost<\/th>\n <th>Cost per Subscriber<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>1,000<\/td>\n <td>~$5<\/td>\n <td>$0.005<\/td>\n <\/tr>\n <tr>\n <td>10,000<\/td>\n <td>~$20<\/td>\n <td>$0.002<\/td>\n <\/tr>\n <tr>\n <td>100,000<\/td>\n <td>~$75<\/td>\n <td>$0.00075<\/td>\n <\/tr>\n <tr>\n <td>1,000,000<\/td>\n <td>~$500<\/td>\n <td>$0.0005<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><em>Based on Cloudflare Workers pricing with daily newsletter sends<\/em><\/p>\n<h2 class=\"heading-element\" id=\"advanced-features\"><span>Advanced Features<\/span>\n <a href=\"#advanced-features\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"1-webhook-integration\"><span>1. Webhook Integration<\/span>\n <a href=\"#1-webhook-integration\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/\/ Webhook notification on new subscriber\nasync function notifyWebhook(subscriber, config) {\n await fetch(config.WEBHOOK_URL, {\n method: &#39;POST&#39;,\n headers: {\n &#39;Content-Type&#39;: &#39;application\/json&#39;,\n &#39;X-Webhook-Secret&#39;: config.WEBHOOK_SECRET\n },\n body: JSON.stringify({\n event: &#39;subscriber.created&#39;,\n data: subscriber,\n timestamp: new Date().toISOString()\n })\n });\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"2-ab-testing-support\"><span>2. A\/B Testing Support<\/span>\n <a href=\"#2-ab-testing-support\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/\/ Split testing for email campaigns\nfunction selectEmailVariant(subscriber, variants) {\n const hash = cyrb53(subscriber.email);\n const bucketIndex = hash % variants.length;\n return variants[bucketIndex];\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"3-analytics-integration\"><span>3. Analytics Integration<\/span>\n <a href=\"#3-analytics-integration\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/\/ Track email opens with pixel tracking\nfunction generateTrackingPixel(emailId, subscriberId) {\n const trackingId = generateUUID();\n return `&lt;img src=&#34;https:\/\/your-worker.dev\/track\/${trackingId}&#34;\n width=&#34;1&#34; height=&#34;1&#34; style=&#34;display:none&#34;&gt;`;\n}<\/code><\/pre><h2 class=\"heading-element\" id=\"monitoring--maintenance\"><span>Monitoring &amp; Maintenance<\/span>\n <a href=\"#monitoring--maintenance\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"automated-maintenance-tasks\"><span>Automated Maintenance Tasks<\/span>\n <a href=\"#automated-maintenance-tasks\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/weekly-maintenance-schedule-gantt.webp\" title=\"Saturday maintenance schedule showing 30 minutes for data cleanup, 20 minutes for GitHub backups, 10 minutes for report generation, and 10 minutes for integrity verification\" data-thumbnail=\"\/files\/weekly-maintenance-schedule-gantt.webp\" data-sub-html=\"<h2>Weekly Maintenance Schedule Gantt Chart<\/h2><p>Saturday maintenance schedule showing 30 minutes for data cleanup, 20 minutes for GitHub backups, 10 minutes for report generation, and 10 minutes for integrity verification<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/weekly-maintenance-schedule-gantt.webp' alt=\"Weekly Maintenance Schedule Gantt Chart\"><\/a><figcaption class=\"image-caption\">Saturday maintenance schedule showing 30 minutes for data cleanup, 20 minutes for GitHub backups, 10 minutes for report generation, and 10 minutes for integrity verification<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"health-monitoring-endpoints\"><span>Health Monitoring Endpoints<\/span>\n <a href=\"#health-monitoring-endpoints\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Endpoint<\/th>\n <th>Purpose<\/th>\n <th>Access<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><code>\/health<\/code><\/td>\n <td>Basic health check<\/td>\n <td>Public<\/td>\n <\/tr>\n <tr>\n <td><code>\/status<\/code><\/td>\n <td>Detailed system status<\/td>\n <td>Zero Trust<\/td>\n <\/tr>\n <tr>\n <td><code>\/debug<\/code><\/td>\n <td>Configuration debug<\/td>\n <td>Zero Trust<\/td>\n <\/tr>\n <tr>\n <td><code>\/metrics<\/code><\/td>\n <td>Performance metrics<\/td>\n <td>Zero Trust<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>\ud83d\udd0d Cloudflare Zero Trust Integration<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">The status page is protected with <strong>Cloudflare Zero Trust<\/strong> for enterprise-grade security. Only authorized administrators can access system metrics and debug information. This prevents information disclosure while maintaining operational visibility.<\/div>\n <\/div>\n<\/div>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/screenshot-cloudflare-zero-trust-login-page.webp\" title=\"Cloudflare Zero Trust authentication interface protecting the admin status page. Shows the secure login prompt that prevents unauthorized access to sensitive system information while maintaining enterprise-grade security compliance\" data-thumbnail=\"\/files\/screenshot-cloudflare-zero-trust-login-page.webp\" data-sub-html=\"<h2>Zero Trust Protected Status Page Access<\/h2><p>Cloudflare Zero Trust authentication interface protecting the admin status page. Shows the secure login prompt that prevents unauthorized access to sensitive system information while maintaining enterprise-grade security compliance<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/screenshot-cloudflare-zero-trust-login-page.webp' alt=\"Zero Trust Protected Status Page Access\"><\/a><figcaption class=\"image-caption\">Cloudflare Zero Trust authentication interface protecting the admin status page. Shows the secure login prompt that prevents unauthorized access to sensitive system information while maintaining enterprise-grade security compliance<\/figcaption>\n <\/figure><\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/screenshot-status-page-access-page.webp\" title=\"Cloudflare Turnstile CAPTCHA to block unwanted bots\" data-thumbnail=\"\/files\/screenshot-status-page-access-page.webp\" data-sub-html=\"<h2>Automatic Cloudflare Turnstile CAPTCHA<\/h2><p>Cloudflare Turnstile CAPTCHA to block unwanted bots<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/screenshot-status-page-access-page.webp' alt=\"Automatic Cloudflare Turnstile CAPTCHA\"><\/a><figcaption class=\"image-caption\">Cloudflare Turnstile CAPTCHA to block unwanted bots<\/figcaption>\n <\/figure><\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/screenshot-system-status-page.webp\" title=\"Real-time system status dashboard showing KV storage statistics, subscriber count, email queue metrics, and system health indicators. Provides administrators with comprehensive monitoring of all newsletter system components and performance metrics\" data-thumbnail=\"\/files\/screenshot-system-status-page.webp\" data-sub-html=\"<h2>System Status Dashboard<\/h2><p>Real-time system status dashboard showing KV storage statistics, subscriber count, email queue metrics, and system health indicators. Provides administrators with comprehensive monitoring of all newsletter system components and performance metrics<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/screenshot-system-status-page.webp' alt=\"System Status Dashboard\"><\/a><figcaption class=\"image-caption\">Real-time system status dashboard showing KV storage statistics, subscriber count, email queue metrics, and system health indicators. Provides administrators with comprehensive monitoring of all newsletter system components and performance metrics<\/figcaption>\n <\/figure><\/p>\n<h2 class=\"heading-element\" id=\"troubleshooting-guide\"><span>Troubleshooting Guide<\/span>\n <a href=\"#troubleshooting-guide\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"common-issues-and-solutions\"><span>Common Issues and Solutions<\/span>\n <a href=\"#common-issues-and-solutions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Issue<\/th>\n <th>Cause<\/th>\n <th>Solution<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>Emails not sending<\/td>\n <td>Invalid credentials<\/td>\n <td>Verify Gmail App Password<\/td>\n <\/tr>\n <tr>\n <td>Rate limit hit<\/td>\n <td>Too many requests<\/td>\n <td>Wait 24 hours or adjust limits<\/td>\n <\/tr>\n <tr>\n <td>Turnstile failing<\/td>\n <td>Invalid keys<\/td>\n <td>Check site\/secret key match<\/td>\n <\/tr>\n <tr>\n <td>KV write errors<\/td>\n <td>Quota exceeded<\/td>\n <td>Upgrade KV plan<\/td>\n <\/tr>\n <tr>\n <td>Backup failing<\/td>\n <td>GitHub token expired<\/td>\n <td>Regenerate PAT token<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"debug-commands\"><span>Debug Commands<\/span>\n <a href=\"#debug-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Check worker logs\nwrangler tail\n\n# Test email sending\ncurl -X POST https:\/\/your-worker.dev\/check-now\n\n# Verify configuration\ncurl https:\/\/your-worker.dev\/debug\n\n# Run maintenance manually\ncurl -X POST https:\/\/your-worker.dev\/maintenance<\/code><\/pre><h2 class=\"heading-element\" id=\"future-enhancements\"><span>Future Enhancements<\/span>\n <a href=\"#future-enhancements\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"roadmap\"><span>Roadmap<\/span>\n <a href=\"#roadmap\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ul>\n<li data-task=\" \"><i class=\"fa-regular fa-square\" aria-hidden=\"true\"><\/i><span title=\"Unchecked\"><strong>GraphQL API<\/strong> for advanced integrations<\/span><\/li>\n<li data-task=\" \"><i class=\"fa-regular fa-square\" aria-hidden=\"true\"><\/i><span title=\"Unchecked\"><strong>Multi-language support<\/strong> with i18n<\/span><\/li>\n<li data-task=\" \"><i class=\"fa-regular fa-square\" aria-hidden=\"true\"><\/i><span title=\"Unchecked\"><strong>Template engine<\/strong> for custom email designs<\/span><\/li>\n<li data-task=\" \"><i class=\"fa-regular fa-square\" aria-hidden=\"true\"><\/i><span title=\"Unchecked\"><strong>Segmentation<\/strong> for targeted campaigns<\/span><\/li>\n<li data-task=\" \"><i class=\"fa-regular fa-square\" aria-hidden=\"true\"><\/i><span title=\"Unchecked\"><strong>Click tracking<\/strong> and engagement analytics<\/span><\/li>\n<li data-task=\" \"><i class=\"fa-regular fa-square\" aria-hidden=\"true\"><\/i><span title=\"Unchecked\"><strong>Drip campaigns<\/strong> with automation workflows<\/span><\/li>\n<li data-task=\" \"><i class=\"fa-regular fa-square\" aria-hidden=\"true\"><\/i><span title=\"Unchecked\"><strong>SMS notifications<\/strong> via Twilio integration<\/span><\/li>\n<li data-task=\" \"><i class=\"fa-regular fa-square\" aria-hidden=\"true\"><\/i><span title=\"Unchecked\"><strong>Dashboard UI<\/strong> with Cloudflare Pages<\/span><\/li>\n<\/ul>\n<h2 class=\"heading-element\" id=\"conclusion\"><span>Conclusion<\/span>\n <a href=\"#conclusion\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Building a newsletter system with Cloudflare Workers demonstrates the power of edge computing. With <strong>zero infrastructure to manage<\/strong>, <strong>infinite scalability<\/strong>, and <strong>enterprise-grade security<\/strong>, this architecture represents the future of web applications.<\/p>\n<p>The combination of:<\/p>\n<ul>\n<li>\ud83c\udf0d Global edge deployment<\/li>\n<li>\ud83d\udce7 Multi-provider email support<\/li>\n<li>\ud83d\udee1\ufe0f Intelligent bot protection<\/li>\n<li>\ud83d\udcca Automated operations<\/li>\n<li>\ud83d\udcb0 Cost-effective scaling<\/li>\n<\/ul>\n<p>Makes this solution perfect for projects ranging from personal blogs to enterprise newsletters serving millions of subscribers.<\/p>\n<div class=\"details admonition success open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-check\" aria-hidden=\"true\"><\/i>\ud83c\udfaf Key Takeaways<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><ol>\n<li><strong>Serverless is the future<\/strong>: No servers, no worries<\/li>\n<li><strong>Edge computing wins<\/strong>: &lt;50ms latency globally<\/li>\n<li><strong>Security by default<\/strong>: Multiple layers of protection<\/li>\n<li><strong>Infinitely scalable<\/strong>: Same code for 1 or 1M users<\/li>\n<li><strong>Cost-effective<\/strong>: Pay only for what you use<\/li>\n<\/ol>\n<\/div>\n <\/div>\n<\/div>\n<h2 class=\"heading-element\" id=\"get-started-today\"><span>Get Started Today<\/span>\n <a href=\"#get-started-today\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Ready to build your own infinitely scalable newsletter system?<\/p>\n<ol>\n<li><strong>Fork the repository<\/strong>: <a href=\"https:\/\/github.com\/SamirPaulb\/newsletter-and-contact-system\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">github.com\/SamirPaulb\/newsletter-and-contact-system<\/a><\/li>\n<li><strong>Follow the setup guide<\/strong> above<\/li>\n<li><strong>Deploy in minutes<\/strong> with Wrangler<\/li>\n<li><strong>Scale to millions<\/strong> without changing anything<\/li>\n<\/ol>\n<h2 class=\"heading-element\" id=\"share-your-feedback\"><span>Share Your Feedback<\/span>\n <a href=\"#share-your-feedback\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>I&rsquo;d love to hear about your experience building with Cloudflare Workers! Have you implemented similar systems? What challenges did you face? What features would you add?<\/p>\n<p><strong>Please share your thoughts, questions, and suggestions in the comments below!<\/strong> Your feedback helps improve this project and guides future enhancements.<\/p>\n<hr>\n<p><em>If you found this guide helpful, please consider:<\/em><\/p>\n<ul>\n<li>\u2b50 <strong>Starring the GitHub repository<\/strong><\/li>\n<li>\ud83d\udd04 <strong>Sharing with your network<\/strong><\/li>\n<li>\ud83d\udcac <strong>Leaving a comment<\/strong> with your thoughts<\/li>\n<li>\ud83d\udc1b <strong>Reporting issues<\/strong> or suggesting features<\/li>\n<\/ul>\n<p><em>Happy building! \ud83d\ude80<\/em><\/p>\n"},{"title":"System Design Notes","link":"https:\/\/samir.pages.dev\/posts\/system-design\/","pubDate":"Mon, 28 Jul 2025 10:00:00 +0800","author":"netproxy@duck.com (Samir)","guid":"https:\/\/samir.pages.dev\/posts\/system-design\/","category":"Software-Engineering","description":"<img src=\"https:\/\/samir.pages.dev\/files\/system-design-og.webp\" alt=\"featured image\" referrerpolicy=\"no-referrer\"><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>About This Guide<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">This comprehensive guide covers everything you need to know about system design, from basic concepts to advanced distributed systems. Perfect for software engineers preparing for system design interviews or architecting real-world applications.<\/div>\n <\/div>\n<\/div>\n<h2 class=\"heading-element\" id=\"1-introduction-to-system-design\"><span>1. Introduction to System Design<\/span>\n <a href=\"#1-introduction-to-system-design\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>System design is the process of defining the architecture, components, modules, interfaces, and data for a system to satisfy specified requirements. In the context of software engineering, it involves making high-level decisions about how to build scalable, reliable, and maintainable systems.<\/p>\n<h3 class=\"heading-element\" id=\"why-system-design-matters\"><span>Why System Design Matters<\/span>\n <a href=\"#why-system-design-matters\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Modern Application Requirements<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p>Modern applications serve millions or even billions of users simultaneously. These systems must be:<\/p>\n<ul>\n<li><strong>Scalable<\/strong>: Handle growing amounts of work<\/li>\n<li><strong>Reliable<\/strong>: Continue functioning under failures<\/li>\n<li><strong>Available<\/strong>: Accessible when users need them<\/li>\n<li><strong>Maintainable<\/strong>: Easy to update and modify<\/li>\n<li><strong>Performant<\/strong>: Respond quickly to user requests<\/li>\n<li><strong>Cost-effective<\/strong>: Operate within reasonable budgets<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"the-journey-ahead\"><span>The Journey Ahead<\/span>\n <a href=\"#the-journey-ahead\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>This guide takes you through a comprehensive exploration of system design, from fundamental concepts to complex real-world implementations. Each section builds upon previous knowledge, creating a complete understanding of how modern distributed systems work.<\/p>\n<h3 class=\"heading-element\" id=\"learning-approach\"><span>Learning Approach<\/span>\n <a href=\"#learning-approach\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Throughout this guide, we&rsquo;ll use:<\/p>\n<ul>\n<li><strong>Visual diagrams<\/strong> (in Mermaid format) to illustrate architectures<\/li>\n<li><strong>Real-world examples<\/strong> from companies like Netflix, Uber, and Twitter<\/li>\n<li><strong>Code examples<\/strong> in Python to demonstrate concepts<\/li>\n<li><strong>Trade-off analyses<\/strong> to understand decision-making<\/li>\n<li><strong>Practical exercises<\/strong> to reinforce learning<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"2-system-design-fundamentals\"><span>2. System Design Fundamentals<\/span>\n <a href=\"#2-system-design-fundamentals\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"21-scalability\"><span>2.1 Scalability<\/span>\n <a href=\"#21-scalability\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Scalability is the ability of a system to handle increased load by adding resources. There are two primary approaches:<\/p>\n<h4 class=\"heading-element\" id=\"vertical-scaling-scale-up\"><span>Vertical Scaling (Scale Up)<\/span>\n <a href=\"#vertical-scaling-scale-up\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Vertical scaling means upgrading the hardware of your existing server \u2014 adding more CPU cores, RAM, or faster disks. It is the simplest way to handle more load because your application code doesn&rsquo;t change at all; you just run the same code on a more powerful machine.<\/p>\n<p><strong>Why it works well initially:<\/strong> A single powerful machine can handle millions of operations per second. For most startups and early-stage products, a well-tuned single server (say, 64-core machine with 512 GB RAM) is far cheaper and simpler than distributing work across many machines. Airbnb and Twitter both started on a single database server.<\/p>\n<p><strong>Why it eventually fails:<\/strong> Hardware has hard physical limits \u2014 you cannot buy a server with 1 million CPU cores. Beyond a certain point, the most powerful server available gets dramatically more expensive per unit of performance (the curve is non-linear). And critically, vertical scaling gives you a <strong>single point of failure<\/strong> \u2014 if that one machine dies, everything goes down. You also need downtime to upgrade hardware.<\/p>\n<p><strong>Advantages:<\/strong><\/p>\n<ul>\n<li>No application code changes \u2014 the simplest possible scale-up path<\/li>\n<li>Data consistency is trivial \u2014 everything is on one machine, no distributed coordination<\/li>\n<li>Lower operational complexity \u2014 one server to monitor, backup, and maintain<\/li>\n<\/ul>\n<p><strong>Disadvantages:<\/strong><\/p>\n<ul>\n<li>Hard upper limit \u2014 the most powerful server you can buy has finite capacity<\/li>\n<li>Single point of failure \u2014 one hardware failure takes down your entire system<\/li>\n<li>Expensive at the high end \u2014 doubling capacity can cost 4\u00d7 or 10\u00d7 the price<\/li>\n<li>Requires planned downtime to perform hardware upgrades<\/li>\n<\/ul>\n<h4 class=\"heading-element\" id=\"horizontal-scaling-scale-out\"><span>Horizontal Scaling (Scale Out)<\/span>\n <a href=\"#horizontal-scaling-scale-out\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Horizontal scaling means adding more machines and distributing the load across them. Instead of one big server, you have 10 (or 1000) commodity servers working together. This is the approach used by every major internet company at scale.<\/p>\n<p><strong>Why it works at internet scale:<\/strong> You can add servers at any time without downtime. If one server fails, the others continue serving traffic. Commodity servers (standard AWS EC2 instances, etc.) follow normal economies of scale \u2014 doubling your server count roughly doubles your capacity at a predictable cost.<\/p>\n<p><strong>The core challenge:<\/strong> Your application must be designed to run correctly across multiple machines. State (like user sessions) cannot live in one server&rsquo;s memory \u2014 it must be stored externally (Redis, database). Requests from the same user might go to different servers. Keeping data consistent across nodes requires careful design (see \u00a713, \u00a720).<\/p>\n<p><strong>Advantages:<\/strong><\/p>\n<ul>\n<li>No theoretical upper limit \u2014 just keep adding machines<\/li>\n<li>Better fault tolerance \u2014 losing one of ten servers has minimal impact<\/li>\n<li>No downtime to scale \u2014 add servers without restarting anything<\/li>\n<li>Cost scales linearly \u2014 predictable cost as you grow<\/li>\n<\/ul>\n<p><strong>Disadvantages:<\/strong><\/p>\n<ul>\n<li>Application must be stateless \u2014 session data and state needs external storage<\/li>\n<li>Data consistency is harder \u2014 reads from different nodes may see different data<\/li>\n<li>Network overhead \u2014 inter-service calls add latency vs. in-process function calls<\/li>\n<li>Requires a load balancer \u2014 clients need a single endpoint that routes to any server<\/li>\n<\/ul>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/vertical-vs-horizontal-scaling.webp\" title=\"Vertical vs Horizontal Scaling \u2014 Pros, cons, and use-cases for scaling strategies; essential for system design interviews\" data-thumbnail=\"\/files\/vertical-vs-horizontal-scaling.webp\" data-sub-html=\"<h2>Comparison diagram showing vertical scaling (scale-up) vs horizontal scaling (scale-out) \u2014 explains when to use each approach for system scalability and performance<\/h2><p>Vertical vs Horizontal Scaling \u2014 Pros, cons, and use-cases for scaling strategies; essential for system design interviews<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/vertical-vs-horizontal-scaling.webp' alt=\"Comparison diagram showing vertical scaling (scale-up) vs horizontal scaling (scale-out) \u2014 explains when to use each approach for system scalability and performance\"><\/a><figcaption class=\"image-caption\">Vertical vs Horizontal Scaling \u2014 Pros, cons, and use-cases for scaling strategies; essential for system design interviews<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"22-performance-vs-scalability\"><span>2.2 Performance vs Scalability<\/span>\n <a href=\"#22-performance-vs-scalability\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>These concepts are often confused but have fundamentally different causes and solutions. Mixing them up leads to wasted engineering effort.<\/p>\n<p><strong>Performance problem:<\/strong> Your system is slow even with a single user. The bottleneck is in the code itself \u2014 an inefficient algorithm, a missing database index, an N+1 query that fires 1000 SQL calls to build one response, or a blocking synchronous call that holds up the entire request.<\/p>\n<p><em>How to diagnose:<\/em> Profile a single request in isolation. If it&rsquo;s slow with zero concurrent users, you have a performance problem. Tools: <code>EXPLAIN<\/code> on SQL queries, profilers (cProfile, py-spy), APM tools (Datadog, New Relic).<\/p>\n<p><em>How to fix:<\/em> Optimize the slow code \u2014 add database indexes, fix inefficient queries, cache computed results, upgrade to a faster algorithm. These fixes are often 10\u2013100\u00d7 improvements with no infrastructure changes.<\/p>\n<p><strong>Scalability problem:<\/strong> Your system works fine for one user but degrades (or crashes) as concurrent users increase. A single request might take 50ms, but at 10,000 concurrent requests the average climbs to 5 seconds. The bottleneck is resource contention \u2014 too many requests competing for the same CPU, database connections, or memory.<\/p>\n<p><em>How to diagnose:<\/em> Run a load test (k6, Locust, JMeter). If p50 latency is fine but p99 is terrible, or latency is fine at 100 RPS but terrible at 1,000 RPS, you have a scalability problem.<\/p>\n<p><em>How to fix:<\/em> Add capacity \u2014 more servers, more database replicas, a caching layer to absorb reads, connection pooling to share database connections, or async processing to decouple slow operations from the critical path.<\/p>\n<div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Don&rsquo;t Confuse the Two<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">Adding servers to a system with a performance problem won&rsquo;t help much \u2014 if each request takes 5 seconds because of a bad query, having 10 servers just means 10 servers each taking 5 seconds. Fix the performance problem first, then scale.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"23-latency-vs-throughput\"><span>2.3 Latency vs Throughput<\/span>\n <a href=\"#23-latency-vs-throughput\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Latency<\/strong> is how long a single operation takes from start to finish \u2014 the time one request waits before it gets a response. Measured in milliseconds (ms) or microseconds (\u00b5s). Example: a database query completes in 5ms.<\/p>\n<p><strong>Throughput<\/strong> is how many operations a system can complete per unit of time \u2014 measured in requests per second (RPS) or queries per second (QPS). Example: a system handles 50,000 database reads per second.<\/p>\n<p><strong>The key insight \u2014 they are often in tension with each other:<\/strong><\/p>\n<p>To maximize throughput, you often want to batch operations: instead of writing each row to disk immediately, buffer 1,000 writes and flush them together. This dramatically increases throughput (fewer disk seeks, better compression), but it increases the latency of each individual write because it now has to wait in the buffer.<\/p>\n<p>Conversely, minimizing latency \u2014 responding to each request immediately \u2014 often means lower throughput, because you can&rsquo;t take advantage of batching.<\/p>\n<p><strong>Real examples:<\/strong><\/p>\n<ul>\n<li><strong>Kafka<\/strong> trades higher individual-message latency (messages are batched before sending) for enormous throughput (millions of messages\/second per broker)<\/li>\n<li><strong>Databases<\/strong> trade latency for throughput via write-ahead logs and group commits<\/li>\n<li><strong>CDN caching<\/strong> improves both \u2014 cache hits have lower latency AND the origin server gets less load (higher effective throughput)<\/li>\n<\/ul>\n<p><strong>Design implication:<\/strong> You need to know your application&rsquo;s priority. An online transaction (credit card charge) must have low latency \u2014 waiting 2 seconds is unacceptable. A data analytics pipeline processes terabytes of events \u2014 throughput is what matters, and 30-second batch latency is fine. Design accordingly.<\/p>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Amdahl&rsquo;s Law<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">Even if you parallelize 90% of a task, the serial 10% becomes the bottleneck. A system processing requests in parallel still has serial sections (database writes, locking) that cap throughput. Identifying and optimizing the serial bottleneck gives the biggest gains.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"24-key-latency-numbers-every-engineer-should-know\"><span>2.4 Key Latency Numbers Every Engineer Should Know<\/span>\n <a href=\"#24-key-latency-numbers-every-engineer-should-know\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Latency Reference Numbers<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p>Understanding these numbers helps make informed design decisions:<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Operation<\/th>\n <th>Latency<\/th>\n <th>Relative Scale<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>L1 cache reference<\/td>\n <td>0.5 ns<\/td>\n <td>1x<\/td>\n <\/tr>\n <tr>\n <td>Branch mispredict<\/td>\n <td>5 ns<\/td>\n <td>10x<\/td>\n <\/tr>\n <tr>\n <td>L2 cache reference<\/td>\n <td>7 ns<\/td>\n <td>14x<\/td>\n <\/tr>\n <tr>\n <td>Mutex lock\/unlock<\/td>\n <td>100 ns<\/td>\n <td>200x<\/td>\n <\/tr>\n <tr>\n <td>Main memory reference<\/td>\n <td>100 ns<\/td>\n <td>200x<\/td>\n <\/tr>\n <tr>\n <td>Compress 1KB with Snappy<\/td>\n <td>10,000 ns (10 \u00b5s)<\/td>\n <td>20,000x<\/td>\n <\/tr>\n <tr>\n <td>Send 1KB over 1 Gbps network<\/td>\n <td>10,000 ns (10 \u00b5s)<\/td>\n <td>20,000x<\/td>\n <\/tr>\n <tr>\n <td>Read 1 MB sequentially from memory<\/td>\n <td>250,000 ns (250 \u00b5s)<\/td>\n <td>500,000x<\/td>\n <\/tr>\n <tr>\n <td>Round trip within same datacenter<\/td>\n <td>500,000 ns (500 \u00b5s)<\/td>\n <td>1,000,000x<\/td>\n <\/tr>\n <tr>\n <td>Read 1 MB sequentially from SSD<\/td>\n <td>1,000,000 ns (1 ms)<\/td>\n <td>2,000,000x<\/td>\n <\/tr>\n <tr>\n <td>Disk seek<\/td>\n <td>10,000,000 ns (10 ms)<\/td>\n <td>20,000,000x<\/td>\n <\/tr>\n <tr>\n <td>Read 1 MB sequentially from disk<\/td>\n <td>30,000,000 ns (30 ms)<\/td>\n <td>60,000,000x<\/td>\n <\/tr>\n <tr>\n <td>Send packet CA\u2192Netherlands\u2192CA<\/td>\n <td>150,000,000 ns (150 ms)<\/td>\n <td>300,000,000x<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><\/div>\n <\/div>\n<\/div>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Key Takeaways<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><ul>\n<li><strong>Memory is fast, disk is slow<\/strong><\/li>\n<li><strong>SSD is 20x faster<\/strong> than traditional disk<\/li>\n<li><strong>Network calls within datacenter<\/strong> are relatively fast<\/li>\n<li><strong>Cross-continent network calls<\/strong> are expensive<\/li>\n<li>Always prefer <strong>sequential reads over random seeks<\/strong><\/li>\n<\/ul>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"25-availability\"><span>2.5 Availability<\/span>\n <a href=\"#25-availability\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Availability is the percentage of time a system is operational and accessible.<\/p>\n<p><strong>Availability Calculation:<\/strong><\/p>\n<pre><code>Availability = (Total Time - Downtime) \/ Total Time \u00d7 100%<\/code><\/pre><p><strong>Standard Availability Tiers (The Nines):<\/strong><\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Availability<\/th>\n <th>Downtime per Year<\/th>\n <th>Downtime per Month<\/th>\n <th>Downtime per Week<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>99% (two nines)<\/td>\n <td>3.65 days<\/td>\n <td>7.31 hours<\/td>\n <td>1.68 hours<\/td>\n <\/tr>\n <tr>\n <td>99.9% (three nines)<\/td>\n <td>8.77 hours<\/td>\n <td>43.83 minutes<\/td>\n <td>10.08 minutes<\/td>\n <\/tr>\n <tr>\n <td>99.99% (four nines)<\/td>\n <td>52.60 minutes<\/td>\n <td>4.38 minutes<\/td>\n <td>1.01 minutes<\/td>\n <\/tr>\n <tr>\n <td>99.999% (five nines)<\/td>\n <td>5.26 minutes<\/td>\n <td>26.30 seconds<\/td>\n <td>6.05 seconds<\/td>\n <\/tr>\n <tr>\n <td>99.9999% (six nines)<\/td>\n <td>31.56 seconds<\/td>\n <td>2.63 seconds<\/td>\n <td>0.61 seconds<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Achieving High Availability:<\/strong><\/p>\n<p>The core principle is: <strong>eliminate every single point of failure<\/strong>. A single point of failure (SPOF) is any component whose failure brings the entire system down. Every SPOF you identify and add redundancy to improves your availability.<\/p>\n<ol>\n<li>\n<p><strong>Eliminate Single Points of Failure (SPOF)<\/strong> \u2014 Run at least two instances of every critical component. If you have one database server and it crashes, everything breaks. If you have a primary + replica with automatic failover, the replica takes over in seconds. Apply this thinking everywhere: two load balancers, multiple web servers, replicated databases, redundant network links. For critical systems, deploy across multiple availability zones (AZs) or regions so a datacenter failure doesn&rsquo;t take you down.<\/p>\n<\/li>\n<li>\n<p><strong>Implement Health Checks<\/strong> \u2014 Load balancers and orchestration systems (Kubernetes) constantly probe your services with health check requests (e.g., <code>GET \/health<\/code>). When a service stops responding or returns errors, traffic is automatically stopped to that instance and rerouted to healthy ones. A service can fail silently (the process keeps running but stops serving correct responses) \u2014 health checks detect this. The response must come back in under a timeout (typically 2\u20135 seconds) or the check fails.<\/p>\n<\/li>\n<li>\n<p><strong>Use Load Balancers<\/strong> \u2014 A load balancer sits in front of your application servers and distributes incoming requests across all healthy instances. If one server fails, the load balancer detects it via health checks and stops sending traffic there \u2014 users experience no disruption. Load balancers themselves can be a SPOF, so production systems run two load balancers (active-passive or active-active). Cloud providers (AWS ELB, GCP Load Balancer) handle this for you automatically.<\/p>\n<\/li>\n<li>\n<p><strong>Design for Graceful Degradation<\/strong> \u2014 When a non-critical dependency is down, return a degraded but functional response rather than failing completely. Amazon&rsquo;s product pages still show the product even if the &ldquo;Customers who bought this also bought&rdquo; recommendation service is down \u2014 they just hide that section. Circuit breakers (see \u00a715.2) implement this pattern automatically.<\/p>\n<\/li>\n<li>\n<p><strong>Plan for Recovery, Not Just Prevention<\/strong> \u2014 No matter how redundant your system is, failures will occur. Define your RTO (Recovery Time Objective \u2014 how quickly must service be restored?) and RPO (Recovery Point Objective \u2014 how much data loss is acceptable?). Test your failover procedures regularly; a failover plan that has never been tested will fail when you need it.<\/p>\n<\/li>\n<\/ol>\n<hr>\n<h2 class=\"heading-element\" id=\"3-back-of-the-envelope-estimation\"><span>3. Back-of-the-Envelope Estimation<\/span>\n <a href=\"#3-back-of-the-envelope-estimation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Being able to quickly estimate system requirements is crucial for system design. This section covers the estimation techniques used in interviews and real-world planning.<\/p>\n<h3 class=\"heading-element\" id=\"31-power-of-two\"><span>3.1 Power of Two<\/span>\n <a href=\"#31-power-of-two\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Understand data volume units:<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Power<\/th>\n <th>Exact Value<\/th>\n <th>Approximate Value<\/th>\n <th>Short Name<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>10<\/td>\n <td>1,024<\/td>\n <td>1 thousand<\/td>\n <td>1 KB<\/td>\n <\/tr>\n <tr>\n <td>20<\/td>\n <td>1,048,576<\/td>\n <td>1 million<\/td>\n <td>1 MB<\/td>\n <\/tr>\n <tr>\n <td>30<\/td>\n <td>1,073,741,824<\/td>\n <td>1 billion<\/td>\n <td>1 GB<\/td>\n <\/tr>\n <tr>\n <td>40<\/td>\n <td>1,099,511,627,776<\/td>\n <td>1 trillion<\/td>\n <td>1 TB<\/td>\n <\/tr>\n <tr>\n <td>50<\/td>\n <td>1,125,899,906,842,624<\/td>\n <td>1 quadrillion<\/td>\n <td>1 PB<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"32-common-performance-numbers\"><span>3.2 Common Performance Numbers<\/span>\n <a href=\"#32-common-performance-numbers\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>These are rough but useful benchmarks to anchor your estimates. You don&rsquo;t need to memorize exact numbers \u2014 the order of magnitude is what matters in an interview or capacity planning session.<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Resource<\/th>\n <th>Typical Capacity<\/th>\n <th>Why It Matters<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>Single web server (8-core)<\/td>\n <td>1,000\u20135,000 RPS<\/td>\n <td>Baseline for estimating how many servers you need<\/td>\n <\/tr>\n <tr>\n <td>Database connection pool<\/td>\n <td>50\u2013100 connections per instance<\/td>\n <td>Connections are expensive; exceeding this causes queuing<\/td>\n <\/tr>\n <tr>\n <td>Single MySQL read (indexed)<\/td>\n <td>~10,000 QPS<\/td>\n <td>How many servers you need for your read traffic<\/td>\n <\/tr>\n <tr>\n <td>Single MySQL write<\/td>\n <td>~2,000\u20135,000 QPS<\/td>\n <td>Writes are slower; often the bottleneck<\/td>\n <\/tr>\n <tr>\n <td>Redis throughput<\/td>\n <td>100,000\u2013200,000 ops\/sec<\/td>\n <td>Why Redis works as a cache in front of databases<\/td>\n <\/tr>\n <tr>\n <td>Average web page size<\/td>\n <td>1\u20132 MB<\/td>\n <td>Used to estimate egress bandwidth cost<\/td>\n <\/tr>\n <tr>\n <td>Video streaming (1080p)<\/td>\n <td>3\u20138 Mbps<\/td>\n <td>Used to size CDN and origin bandwidth<\/td>\n <\/tr>\n <tr>\n <td>Mobile API calls per session<\/td>\n <td>5\u201310 requests<\/td>\n <td>Used to estimate total API traffic from DAU<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>How to use these numbers:<\/strong><\/p>\n<p>Say you&rsquo;re designing a system with 10 million daily active users (DAU), each making 10 API calls per day. That&rsquo;s 100 million API calls\/day \u00f7 86,400 seconds \u2248 <strong>1,160 RPS average<\/strong>. With a 3\u00d7 peak factor, you need to handle ~3,500 RPS peak. One well-tuned web server handles ~2,000\u20133,000 RPS, so you need <strong>2\u20133 web servers<\/strong> as a baseline \u2014 before caching, before CDN.<\/p>\n<p>If 30% of those calls hit the database directly (the rest served from cache), that&rsquo;s ~350 QPS to the database at peak. A single MySQL replica handles 5,000\u201310,000 reads\/sec, so <strong>one read replica with a cache layer is sufficient<\/strong> at this scale.<\/p>\n<p>This is the thinking interviewers look for \u2014 not exact numbers, but using rough benchmarks to reach reasonable architectural decisions.<\/p>\n<h3 class=\"heading-element\" id=\"33-example-estimation-twitter-like-system\"><span>3.3 Example Estimation: Twitter-Like System<\/span>\n <a href=\"#33-example-estimation-twitter-like-system\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition example open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-list-ul\" aria-hidden=\"true\"><\/i>Twitter System Requirements<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Given Requirements:<\/strong><\/p>\n<ul>\n<li>300 million monthly active users (MAU)<\/li>\n<li>50% use Twitter daily<\/li>\n<li>Average user posts 2 tweets per day<\/li>\n<li>10% of tweets contain media<\/li>\n<li>Data retained for 5 years<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<p><strong>Calculations:<\/strong><\/p>\n<div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Step-by-Step Calculations<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Daily Active Users (DAU):<\/strong><\/p>\n<pre><code>DAU = 300M \u00d7 50% = 150M<\/code><\/pre><p><strong>Tweet Traffic:<\/strong><\/p>\n<pre><code>Tweets per day = 150M \u00d7 2 = 300M\nTweets per second (average) = 300M \/ 86400 \u2248 3,500 tweets\/second\nPeak QPS = 3,500 \u00d7 2 = 7,000 tweets\/second (assuming 2x peak factor)<\/code><\/pre><p><strong>Storage Estimation:<\/strong><\/p>\n<pre><code>Average tweet size:\n- Tweet text: 280 chars \u00d7 2 bytes = 560 bytes\n- Metadata (ID, timestamp, user_id, etc.): 200 bytes\n- Total per tweet: ~800 bytes \u2248 1 KB\n\nMedia tweets (10%):\n- Image average: 500 KB\n- Total per media tweet: 500 KB &#43; 1 KB \u2248 500 KB\n\nDaily storage:\n- Text tweets: 270M \u00d7 1 KB = 270 GB\n- Media tweets: 30M \u00d7 500 KB = 15 TB\n- Total per day = 15.27 TB\n\n5-year storage:\n15.27 TB\/day \u00d7 365 days \u00d7 5 years \u2248 27.9 PB<\/code><\/pre><p><strong>Bandwidth Estimation:<\/strong><\/p>\n<pre><code>Ingress (upload):\n15.27 TB\/day \u00f7 86,400 seconds \u2248 177 MB\/second\n\nEgress (assume each tweet viewed 10 times):\n177 MB\/s \u00d7 10 = 1.77 GB\/second<\/code><\/pre><p><strong>Memory for Cache (80-20 rule):<\/strong><\/p>\n<pre><code>20% of tweets generate 80% of traffic\nDaily cache: 15.27 TB \u00d7 0.2 = 3 TB<\/code><\/pre><\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 class=\"heading-element\" id=\"4-the-system-design-interview-framework\"><span>4. The System Design Interview Framework<\/span>\n <a href=\"#4-the-system-design-interview-framework\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>A systematic approach to solving any system design problem.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/interview-framework-steps.webp\" title=\"System Design Interview Framework \u2014 4-step approach: requirements, high-level design, deep dive, and wrap-up for FAANG interviews\" data-thumbnail=\"\/files\/interview-framework-steps.webp\" data-sub-html=\"<h2>System design interview 4-step framework \u2014 clarify requirements, high-level design, deep dive, and wrap up<\/h2><p>System Design Interview Framework \u2014 4-step approach: requirements, high-level design, deep dive, and wrap-up for FAANG interviews<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/interview-framework-steps.webp' alt=\"System design interview 4-step framework \u2014 clarify requirements, high-level design, deep dive, and wrap up\"><\/a><figcaption class=\"image-caption\">System Design Interview Framework \u2014 4-step approach: requirements, high-level design, deep dive, and wrap-up for FAANG interviews<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"41-the-4-step-framework\"><span>4.1 The 4-Step Framework<\/span>\n <a href=\"#41-the-4-step-framework\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>System Design Interview Steps<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><h4 class=\"heading-element\" id=\"step-1-understand-the-problem-and-establish-design-scope-3-10-minutes\"><span>Step 1: Understand the Problem and Establish Design Scope (3-10 minutes)<\/span>\n <a href=\"#step-1-understand-the-problem-and-establish-design-scope-3-10-minutes\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><strong>Ask clarifying questions:<\/strong><\/p>\n<ul>\n<li>Who are the users?<\/li>\n<li>How many users?<\/li>\n<li>What features are essential?<\/li>\n<li>What is the scale we&rsquo;re designing for?<\/li>\n<li>What is the expected traffic pattern?<\/li>\n<li>What technologies does the company use?<\/li>\n<\/ul>\n<h4 class=\"heading-element\" id=\"step-2-propose-high-level-design-and-get-buy-in-10-15-minutes\"><span>Step 2: Propose High-Level Design and Get Buy-In (10-15 minutes)<\/span>\n <a href=\"#step-2-propose-high-level-design-and-get-buy-in-10-15-minutes\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><strong>Key actions:<\/strong><\/p>\n<ul>\n<li>Draw initial architecture diagram<\/li>\n<li>Identify major components<\/li>\n<li>Explain data flow<\/li>\n<li>Get feedback from interviewer<\/li>\n<\/ul>\n<h4 class=\"heading-element\" id=\"step-3-design-deep-dive-10-25-minutes\"><span>Step 3: Design Deep Dive (10-25 minutes)<\/span>\n <a href=\"#step-3-design-deep-dive-10-25-minutes\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><strong>Focus areas:<\/strong><\/p>\n<ul>\n<li>Dig into 2-3 components based on interviewer interest<\/li>\n<li>Discuss trade-offs<\/li>\n<li>Address bottlenecks<\/li>\n<li>Consider edge cases<\/li>\n<\/ul>\n<h4 class=\"heading-element\" id=\"step-4-wrap-up-3-5-minutes\"><span>Step 4: Wrap Up (3-5 minutes)<\/span>\n <a href=\"#step-4-wrap-up-3-5-minutes\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><strong>Final touches:<\/strong><\/p>\n<ul>\n<li>Identify system bottlenecks<\/li>\n<li>Discuss potential improvements<\/li>\n<li>Recap design decisions<\/li>\n<li>Mention monitoring and operations<\/li>\n<\/ul>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"42-key-principles\"><span>4.2 Key Principles<\/span>\n <a href=\"#42-key-principles\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Start Simple, Then Iterate<\/strong><\/p>\n<ul>\n<li>Begin with a basic design<\/li>\n<li>Add complexity incrementally<\/li>\n<li>Justify each addition<\/li>\n<\/ul>\n<p><strong>Focus on What Matters<\/strong><\/p>\n<ul>\n<li>Prioritize based on requirements<\/li>\n<li>Don&rsquo;t over-engineer<\/li>\n<li>Keep the user experience in mind<\/li>\n<\/ul>\n<p><strong>Think About Trade-offs<\/strong><\/p>\n<ul>\n<li>No perfect solution exists<\/li>\n<li>Discuss pros and cons<\/li>\n<li>Choose based on requirements<\/li>\n<\/ul>\n<p><strong>Use Real Numbers<\/strong><\/p>\n<ul>\n<li>Apply back-of-the-envelope calculations<\/li>\n<li>Validate your assumptions<\/li>\n<li>Show quantitative reasoning<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"5-networking-and-communication\"><span>5. Networking and Communication<\/span>\n <a href=\"#5-networking-and-communication\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Network Fundamentals<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">Understanding how data moves through networks is fundamental to system design. This section covers essential networking concepts that impact system performance, reliability, and scalability.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"51-the-osi-model-and-tcpip\"><span>5.1 The OSI Model and TCP\/IP<\/span>\n <a href=\"#51-the-osi-model-and-tcpip\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Model Overview<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">The OSI (Open Systems Interconnection) model has 7 layers, but for practical system design, focus on the TCP\/IP model, which is more pragmatic and widely used in modern applications.<\/div>\n <\/div>\n<\/div>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/osi-tcp-ip-model.webp\" title=\"OSI and TCP\/IP Network Models \u2014 Understanding network layers for architecture design and optimization\" data-thumbnail=\"\/files\/osi-tcp-ip-model.webp\" data-sub-html=\"<h2>Diagram comparing OSI model (7 layers) with TCP\/IP model (4 layers) \u2014 network communication protocols for system design<\/h2><p>OSI and TCP\/IP Network Models \u2014 Understanding network layers for architecture design and optimization<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/osi-tcp-ip-model.webp' alt=\"Diagram comparing OSI model (7 layers) with TCP\/IP model (4 layers) \u2014 network communication protocols for system design\"><\/a><figcaption class=\"image-caption\">OSI and TCP\/IP Network Models \u2014 Understanding network layers for architecture design and optimization<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"52-http-and-https\"><span>5.2 HTTP and HTTPS<\/span>\n <a href=\"#52-http-and-https\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>HTTP (Hypertext Transfer Protocol)<\/strong> is the foundation of data communication on the web.<\/p>\n<p><strong>Common HTTP Methods:<\/strong><\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Method<\/th>\n <th>Purpose<\/th>\n <th>Idempotent<\/th>\n <th>Safe<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>GET<\/td>\n <td>Retrieve data<\/td>\n <td>Yes<\/td>\n <td>Yes<\/td>\n <\/tr>\n <tr>\n <td>POST<\/td>\n <td>Create resource<\/td>\n <td>No<\/td>\n <td>No<\/td>\n <\/tr>\n <tr>\n <td>PUT<\/td>\n <td>Update\/Replace resource<\/td>\n <td>Yes<\/td>\n <td>No<\/td>\n <\/tr>\n <tr>\n <td>PATCH<\/td>\n <td>Partial update<\/td>\n <td>No<\/td>\n <td>No<\/td>\n <\/tr>\n <tr>\n <td>DELETE<\/td>\n <td>Remove resource<\/td>\n <td>Yes<\/td>\n <td>No<\/td>\n <\/tr>\n <tr>\n <td>HEAD<\/td>\n <td>Get headers only<\/td>\n <td>Yes<\/td>\n <td>Yes<\/td>\n <\/tr>\n <tr>\n <td>OPTIONS<\/td>\n <td>Query methods<\/td>\n <td>Yes<\/td>\n <td>Yes<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>HTTP Status Codes:<\/strong><\/p>\n<div class=\"details admonition note\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Common HTTP Status Codes<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><pre><code>1xx: Informational\n 100 Continue\n 101 Switching Protocols\n\n2xx: Success\n 200 OK\n 201 Created\n 202 Accepted\n 204 No Content\n\n3xx: Redirection\n 301 Moved Permanently\n 302 Found (Temporary Redirect)\n 304 Not Modified\n 307 Temporary Redirect\n 308 Permanent Redirect\n\n4xx: Client Errors\n 400 Bad Request\n 401 Unauthorized\n 403 Forbidden\n 404 Not Found\n 409 Conflict\n 429 Too Many Requests\n\n5xx: Server Errors\n 500 Internal Server Error\n 502 Bad Gateway\n 503 Service Unavailable\n 504 Gateway Timeout<\/code><\/pre><\/div>\n <\/div>\n<\/div>\n<p><strong>HTTPS = HTTP + SSL\/TLS<\/strong><\/p>\n<p>HTTPS adds encryption and authentication through SSL\/TLS:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/https-ssl-tls-handshake.webp\" title=\"SSL\/TLS Handshake Process \u2014 How HTTPS secures data; critical for system security and authentication\" data-thumbnail=\"\/files\/https-ssl-tls-handshake.webp\" data-sub-html=\"<h2>Sequence diagram showing SSL\/TLS handshake process \u2014 encryption and secure communication establishment between client and server<\/h2><p>SSL\/TLS Handshake Process \u2014 How HTTPS secures data; critical for system security and authentication<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/https-ssl-tls-handshake.webp' alt=\"Sequence diagram showing SSL\/TLS handshake process \u2014 encryption and secure communication establishment between client and server\"><\/a><figcaption class=\"image-caption\">SSL\/TLS Handshake Process \u2014 How HTTPS secures data; critical for system security and authentication<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"53-rest-vs-graphql-vs-grpc-vs-websockets\"><span>5.3 REST vs GraphQL vs gRPC vs WebSockets<\/span>\n <a href=\"#53-rest-vs-graphql-vs-grpc-vs-websockets\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>REST (Representational State Transfer)<\/strong><\/p>\n<p>Characteristics:<\/p>\n<ul>\n<li>Resource-based (nouns in URLs)<\/li>\n<li>Stateless<\/li>\n<li>Uses standard HTTP methods<\/li>\n<li>Returns JSON or XML<\/li>\n<\/ul>\n<p>Example:<\/p>\n<pre><code>GET \/api\/users\/123\nGET \/api\/users\/123\/posts\nPOST \/api\/users\/123\/posts\nPUT \/api\/users\/123\nDELETE \/api\/users\/123<\/code><\/pre><p>Pros:<\/p>\n<ul>\n<li>Simple and widely understood<\/li>\n<li>Cacheable<\/li>\n<li>Good tooling support<\/li>\n<\/ul>\n<p>Cons:<\/p>\n<ul>\n<li>Over-fetching or under-fetching data<\/li>\n<li>Multiple round trips for related data<\/li>\n<li>Versioning can be challenging<\/li>\n<\/ul>\n<p><strong>GraphQL<\/strong><\/p>\n<p>Characteristics:<\/p>\n<ul>\n<li>Client specifies exactly what data it needs<\/li>\n<li>Single endpoint<\/li>\n<li>Strongly typed schema<\/li>\n<li>Real-time updates via subscriptions<\/li>\n<\/ul>\n<p>Example:<\/p>\n<pre><code>query {\n user(id: &#34;123&#34;) {\n name\n email\n posts(limit: 10) {\n title\n createdAt\n }\n }\n}<\/code><\/pre><p>Pros:<\/p>\n<ul>\n<li>No over-fetching\/under-fetching<\/li>\n<li>Single request for complex data<\/li>\n<li>Self-documenting API<\/li>\n<\/ul>\n<p>Cons:<\/p>\n<ul>\n<li>More complex backend<\/li>\n<li>Caching is harder<\/li>\n<li>Potential for expensive queries<\/li>\n<\/ul>\n<p><strong>gRPC (Google Remote Procedure Call)<\/strong><\/p>\n<p>Characteristics:<\/p>\n<ul>\n<li>Uses Protocol Buffers (binary format)<\/li>\n<li>HTTP\/2 based<\/li>\n<li>Strongly typed<\/li>\n<li>Bi-directional streaming<\/li>\n<\/ul>\n<p>Example (protobuf):<\/p>\n<pre><code>service UserService {\n rpc GetUser (UserRequest) returns (UserResponse);\n rpc ListUsers (ListUsersRequest) returns (stream UserResponse);\n}\n\nmessage UserRequest {\n string user_id = 1;\n}\n\nmessage UserResponse {\n string user_id = 1;\n string name = 2;\n string email = 3;\n}<\/code><\/pre><p>Pros:<\/p>\n<ul>\n<li>Very fast (binary protocol)<\/li>\n<li>Streaming support<\/li>\n<li>Strong typing<\/li>\n<\/ul>\n<p>Cons:<\/p>\n<ul>\n<li>Not browser-friendly<\/li>\n<li>Less human-readable<\/li>\n<li>Steeper learning curve<\/li>\n<\/ul>\n<p><strong>WebSockets<\/strong><\/p>\n<p>Characteristics:<\/p>\n<ul>\n<li>Full-duplex communication<\/li>\n<li>Persistent connection<\/li>\n<li>Real-time bidirectional data flow<\/li>\n<\/ul>\n<p>Use cases:<\/p>\n<ul>\n<li>Chat applications<\/li>\n<li>Live feeds<\/li>\n<li>Real-time gaming<\/li>\n<li>Collaborative editing<\/li>\n<\/ul>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/websocket-connection-flow.webp\" title=\"WebSocket Connection Flow \u2014 Real-time communication protocol for live applications and low-latency messaging\" data-thumbnail=\"\/files\/websocket-connection-flow.webp\" data-sub-html=\"<h2>Diagram showing WebSocket connection establishment and bidirectional communication flow \u2014 real-time data exchange for chat and live updates<\/h2><p>WebSocket Connection Flow \u2014 Real-time communication protocol for live applications and low-latency messaging<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/websocket-connection-flow.webp' alt=\"Diagram showing WebSocket connection establishment and bidirectional communication flow \u2014 real-time data exchange for chat and live updates\"><\/a><figcaption class=\"image-caption\">WebSocket Connection Flow \u2014 Real-time communication protocol for live applications and low-latency messaging<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"54-api-authentication\"><span>5.4 API Authentication<\/span>\n <a href=\"#54-api-authentication\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>1. Basic Authentication<\/strong><\/p>\n<pre><code>Authorization: Basic base64(username:password)<\/code><\/pre><ul>\n<li>Simple but insecure (credentials in every request)<\/li>\n<li>Use only over HTTPS<\/li>\n<\/ul>\n<p><strong>2. API Keys<\/strong><\/p>\n<pre><code>X-API-Key: your-api-key-here<\/code><\/pre><ul>\n<li>Simple to implement<\/li>\n<li>Good for server-to-server<\/li>\n<li>Hard to revoke individual keys<\/li>\n<\/ul>\n<p><strong>3. OAuth 2.0<\/strong><\/p>\n<p>Most common flows:<\/p>\n<p><strong>Authorization Code Flow (for web apps):<\/strong>\n<figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/oauth2-authorization-code-flow.webp\" title=\"OAuth 2.0 Authorization Flow \u2014 Secure user authentication and third-party app authorization pattern\" data-thumbnail=\"\/files\/oauth2-authorization-code-flow.webp\" data-sub-html=\"<h2>Sequence diagram of OAuth 2.0 authorization code flow \u2014 secure authentication and authorization for third-party integrations<\/h2><p>OAuth 2.0 Authorization Flow \u2014 Secure user authentication and third-party app authorization pattern<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/oauth2-authorization-code-flow.webp' alt=\"Sequence diagram of OAuth 2.0 authorization code flow \u2014 secure authentication and authorization for third-party integrations\"><\/a><figcaption class=\"image-caption\">OAuth 2.0 Authorization Flow \u2014 Secure user authentication and third-party app authorization pattern<\/figcaption>\n <\/figure><\/p>\n<p><strong>4. JWT (JSON Web Tokens)<\/strong><\/p>\n<p>Structure: <code>header.payload.signature<\/code><\/p>\n<pre><code>\/\/ Header\n{\n &#34;alg&#34;: &#34;HS256&#34;,\n &#34;typ&#34;: &#34;JWT&#34;\n}\n\n\/\/ Payload\n{\n &#34;sub&#34;: &#34;user123&#34;,\n &#34;name&#34;: &#34;John Doe&#34;,\n &#34;iat&#34;: 1516239022,\n &#34;exp&#34;: 1516242622\n}\n\n\/\/ Signature\nHMACSHA256(\n base64UrlEncode(header) &#43; &#34;.&#34; &#43;\n base64UrlEncode(payload),\n secret\n)<\/code><\/pre><p>Pros:<\/p>\n<ul>\n<li>Stateless (no server-side session storage)<\/li>\n<li>Portable across domains<\/li>\n<li>Contains user information<\/li>\n<\/ul>\n<p>Cons:<\/p>\n<ul>\n<li>Can&rsquo;t revoke before expiration<\/li>\n<li>Token size can be large<\/li>\n<li>Vulnerable if secret is exposed<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"6-load-balancing\"><span>6. Load Balancing<\/span>\n <a href=\"#6-load-balancing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Load balancers distribute incoming traffic across multiple servers, improving availability and preventing any single server from becoming a bottleneck.<\/p>\n<h3 class=\"heading-element\" id=\"61-why-load-balancers\"><span>6.1 Why Load Balancers?<\/span>\n <a href=\"#61-why-load-balancers\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Benefits:<\/strong><\/p>\n<ul>\n<li><strong>High Availability<\/strong>: If one server fails, traffic routes to healthy servers<\/li>\n<li><strong>Scalability<\/strong>: Add more servers to handle increased load<\/li>\n<li><strong>Performance<\/strong>: Distribute load evenly<\/li>\n<li><strong>Flexibility<\/strong>: Perform maintenance without downtime<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"62-load-balancer-architecture\"><span>6.2 Load Balancer Architecture<\/span>\n <a href=\"#62-load-balancer-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/load-balancer-architecture.webp\" title=\"Load Balancer Architecture \u2014 How to distribute traffic and ensure high availability in production systems\" data-thumbnail=\"\/files\/load-balancer-architecture.webp\" data-sub-html=\"<h2>Architecture diagram showing load balancer distributing traffic across multiple web servers \u2014 high availability and fault tolerance<\/h2><p>Load Balancer Architecture \u2014 How to distribute traffic and ensure high availability in production systems<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/load-balancer-architecture.webp' alt=\"Architecture diagram showing load balancer distributing traffic across multiple web servers \u2014 high availability and fault tolerance\"><\/a><figcaption class=\"image-caption\">Load Balancer Architecture \u2014 How to distribute traffic and ensure high availability in production systems<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"63-load-balancing-algorithms\"><span>6.3 Load Balancing Algorithms<\/span>\n <a href=\"#63-load-balancing-algorithms\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Choosing the Right Algorithm<\/strong>: The best load balancing algorithm depends on your specific use case. For stateless applications with similar request complexity, Round Robin works well. For long-lived connections (like WebSockets), use Least Connections. For session affinity, use IP Hash or Consistent Hashing.<\/div>\n <\/div>\n<\/div>\n<p><strong>1. Round Robin<\/strong><\/p>\n<ul>\n<li>Distributes requests sequentially<\/li>\n<li>Simple and fair distribution<\/li>\n<li>Doesn&rsquo;t account for server load or capacity<\/li>\n<\/ul>\n<pre><code>class RoundRobinLoadBalancer:\n def __init__(self, servers):\n self.servers = servers\n self.current = 0\n\n def get_server(self):\n server = self.servers[self.current]\n self.current = (self.current &#43; 1) % len(self.servers)\n return server<\/code><\/pre><p><strong>2. Weighted Round Robin<\/strong><\/p>\n<ul>\n<li>Assigns weight to each server based on capacity<\/li>\n<li>Servers with higher capacity get more requests<\/li>\n<\/ul>\n<pre><code>class WeightedRoundRobinLoadBalancer:\n def __init__(self, servers_with_weights):\n # servers_with_weights = [(&#39;server1&#39;, 5), (&#39;server2&#39;, 3), (&#39;server3&#39;, 2)]\n self.servers = []\n for server, weight in servers_with_weights:\n self.servers.extend([server] * weight)\n self.current = 0\n\n def get_server(self):\n server = self.servers[self.current]\n self.current = (self.current &#43; 1) % len(self.servers)\n return server<\/code><\/pre><p><strong>3. Least Connections<\/strong><\/p>\n<ul>\n<li>Routes to server with fewest active connections<\/li>\n<li>Good for long-lived connections<\/li>\n<\/ul>\n<pre><code>class LeastConnectionsLoadBalancer:\n def __init__(self, servers):\n self.connections = {server: 0 for server in servers}\n\n def get_server(self):\n return min(self.connections, key=self.connections.get)\n\n def on_connection_established(self, server):\n self.connections[server] &#43;= 1\n\n def on_connection_closed(self, server):\n self.connections[server] -= 1<\/code><\/pre><p><strong>4. Least Response Time<\/strong><\/p>\n<ul>\n<li>Routes to server with lowest average response time<\/li>\n<li>Dynamically adapts to server performance<\/li>\n<\/ul>\n<p><strong>5. IP Hash<\/strong><\/p>\n<ul>\n<li>Uses client IP to determine server<\/li>\n<li>Ensures same client always routes to same server<\/li>\n<li>Good for session persistence<\/li>\n<\/ul>\n<pre><code>import hashlib\n\nclass IPHashLoadBalancer:\n def __init__(self, servers):\n self.servers = servers\n\n def get_server(self, client_ip):\n hash_value = int(hashlib.md5(client_ip.encode()).hexdigest(), 16)\n server_index = hash_value % len(self.servers)\n return self.servers[server_index]<\/code><\/pre><p><strong>6. Random<\/strong><\/p>\n<ul>\n<li>Randomly selects server<\/li>\n<li>Simple and works well with many servers<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"64-load-balancer-layers\"><span>6.4 Load Balancer Layers<\/span>\n <a href=\"#64-load-balancer-layers\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Layer 4 vs Layer 7 Trade-off<\/strong>: Layer 4 load balancing is faster (10-100 microseconds latency) but less intelligent. Layer 7 adds 1-10 milliseconds latency but enables content-based routing, URL rewriting, and application-aware decisions. Use Layer 4 for raw throughput, Layer 7 for microservices architectures.<\/div>\n <\/div>\n<\/div>\n<p><strong>Layer 4 (Transport Layer) Load Balancing<\/strong><\/p>\n<ul>\n<li>Operates at TCP\/UDP level<\/li>\n<li>Routes based on IP and port<\/li>\n<li>Fast (no packet inspection)<\/li>\n<li>Cannot make routing decisions based on content<\/li>\n<\/ul>\n<p><strong>Layer 7 (Application Layer) Load Balancing<\/strong><\/p>\n<ul>\n<li>Operates at HTTP level<\/li>\n<li>Can route based on URL, headers, cookies<\/li>\n<li>More intelligent routing<\/li>\n<li>Slower (needs to inspect packets)<\/li>\n<li>Can do SSL termination<\/li>\n<\/ul>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/layer4-vs-layer7-load-balancing.webp\" title=\"Layer 4 vs Layer 7 Load Balancing \u2014 Understanding transport vs application layer routing and performance trade-offs\" data-thumbnail=\"\/files\/layer4-vs-layer7-load-balancing.webp\" data-sub-html=\"<h2>Comparison of Layer 4 (transport) and Layer 7 (application) load balancing \u2014 routing strategies at different network levels<\/h2><p>Layer 4 vs Layer 7 Load Balancing \u2014 Understanding transport vs application layer routing and performance trade-offs<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/layer4-vs-layer7-load-balancing.webp' alt=\"Comparison of Layer 4 (transport) and Layer 7 (application) load balancing \u2014 routing strategies at different network levels\"><\/a><figcaption class=\"image-caption\">Layer 4 vs Layer 7 Load Balancing \u2014 Understanding transport vs application layer routing and performance trade-offs<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"65-health-checks\"><span>6.5 Health Checks<\/span>\n <a href=\"#65-health-checks\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Load balancers need to know which servers are healthy:<\/p>\n<pre><code>import time\nimport requests\n\nclass HealthChecker:\n def __init__(self, servers, check_interval=10):\n self.servers = servers\n self.healthy_servers = set(servers)\n self.check_interval = check_interval\n\n def check_health(self, server):\n try:\n response = requests.get(f&#34;{server}\/health&#34;, timeout=2)\n return response.status_code == 200\n except:\n return False\n\n def monitor(self):\n while True:\n for server in self.servers:\n if self.check_health(server):\n self.healthy_servers.add(server)\n else:\n self.healthy_servers.discard(server)\n time.sleep(self.check_interval)\n\n def get_healthy_servers(self):\n return list(self.healthy_servers)<\/code><\/pre><h3 class=\"heading-element\" id=\"66-session-persistence-sticky-sessions\"><span>6.6 Session Persistence (Sticky Sessions)<\/span>\n <a href=\"#66-session-persistence-sticky-sessions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Challenge: With load balancing, subsequent requests from the same user might go to different servers.<\/p>\n<p><strong>Solutions:<\/strong><\/p>\n<p><strong>1. Sticky Sessions at Load Balancer<\/strong><\/p>\n<ul>\n<li>Load balancer remembers which server served a client<\/li>\n<li>Routes all requests from that client to same server<\/li>\n<li>Problem: If server dies, session is lost<\/li>\n<\/ul>\n<p><strong>2. Shared Session Storage<\/strong><\/p>\n<ul>\n<li>Store sessions in Redis\/Memcached<\/li>\n<li>Any server can serve any request<\/li>\n<li>More scalable and fault-tolerant<\/li>\n<\/ul>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/session-persistence-shared-storage.webp\" title=\"Shared Session Storage \u2014 How to scale stateless servers using centralized session management with Redis\" data-thumbnail=\"\/files\/session-persistence-shared-storage.webp\" data-sub-html=\"<h2>Architecture showing session persistence using Redis as shared session storage \u2014 stateless server scaling for distributed systems<\/h2><p>Shared Session Storage \u2014 How to scale stateless servers using centralized session management with Redis<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/session-persistence-shared-storage.webp' alt=\"Architecture showing session persistence using Redis as shared session storage \u2014 stateless server scaling for distributed systems\"><\/a><figcaption class=\"image-caption\">Shared Session Storage \u2014 How to scale stateless servers using centralized session management with Redis<\/figcaption>\n <\/figure><\/p>\n<p><strong>3. Client-Side Sessions (JWT)<\/strong><\/p>\n<ul>\n<li>Session data stored in token on client<\/li>\n<li>Stateless servers<\/li>\n<li>Token passed with each request<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"7-caching-strategies\"><span>7. Caching Strategies<\/span>\n <a href=\"#7-caching-strategies\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Performance Optimization<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">Caching is one of the most effective ways to improve system performance by storing frequently accessed data in fast storage. Well-designed caching can reduce latency by 10-100x and significantly decrease database load.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"71-what-to-cache\"><span>7.1 What to Cache?<\/span>\n <a href=\"#71-what-to-cache\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Caching Best Practices<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p>Good candidates for caching:<\/p>\n<ul>\n<li><strong>Read-heavy data<\/strong>: Data that is read frequently but updated infrequently<\/li>\n<li><strong>Expensive computations<\/strong>: Results that take significant CPU time to calculate<\/li>\n<li><strong>Database query results<\/strong>: Frequently-run queries with stable results<\/li>\n<li><strong>API responses<\/strong>: Third-party API calls with consistent data<\/li>\n<li><strong>Rendered HTML pages<\/strong>: Static or semi-static page content<\/li>\n<li><strong>Session data<\/strong>: User authentication and session information<\/li>\n<li><strong>Static assets<\/strong>: Images, CSS, JavaScript files<\/li>\n<\/ul>\n<div class=\"details admonition warning\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>What NOT to Cache<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p>Avoid caching:<\/p>\n<ul>\n<li>Highly dynamic data that changes frequently<\/li>\n<li>User-specific sensitive data without proper isolation<\/li>\n<li>Data requiring strong consistency guarantees<\/li>\n<li>Data with low hit rates (rarely accessed)<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"72-cache-levels\"><span>7.2 Cache Levels<\/span>\n <a href=\"#72-cache-levels\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/cache-levels-architecture.webp\" title=\"Cache Levels in System \u2014 Comprehensive caching strategy: browser, CDN, application, and database caching\" data-thumbnail=\"\/files\/cache-levels-architecture.webp\" data-sub-html=\"<h2>Multi-level caching architecture from browser to database \u2014 optimization strategy showing cache hierarchy and performance gains<\/h2><p>Cache Levels in System \u2014 Comprehensive caching strategy: browser, CDN, application, and database caching<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/cache-levels-architecture.webp' alt=\"Multi-level caching architecture from browser to database \u2014 optimization strategy showing cache hierarchy and performance gains\"><\/a><figcaption class=\"image-caption\">Cache Levels in System \u2014 Comprehensive caching strategy: browser, CDN, application, and database caching<\/figcaption>\n <\/figure><\/p>\n<p><strong>1. Client-Side Caching<\/strong><\/p>\n<ul>\n<li>Browser cache<\/li>\n<li>LocalStorage \/ SessionStorage<\/li>\n<li>Service Workers<\/li>\n<\/ul>\n<p><strong>2. CDN Caching<\/strong><\/p>\n<ul>\n<li>Caches static assets close to users<\/li>\n<li>Reduces origin server load<\/li>\n<\/ul>\n<p><strong>3. Application-Level Caching<\/strong><\/p>\n<ul>\n<li>In-memory caches (Redis, Memcached)<\/li>\n<li>Application server cache<\/li>\n<\/ul>\n<p><strong>4. Database Caching<\/strong><\/p>\n<ul>\n<li>Query result cache<\/li>\n<li>Buffer pool cache<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"73-caching-patterns\"><span>7.3 Caching Patterns<\/span>\n <a href=\"#73-caching-patterns\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Caching Pattern Selection<\/strong>: Choose your caching pattern based on read\/write ratios and consistency requirements:<\/p>\n<ul>\n<li><strong>Cache-Aside<\/strong>: Best for read-heavy workloads with eventual consistency tolerance (90% of use cases)<\/li>\n<li><strong>Write-Through<\/strong>: When strong consistency is required between cache and database<\/li>\n<li><strong>Write-Behind<\/strong>: For write-heavy workloads where eventual consistency is acceptable<\/li>\n<li><strong>Refresh-Ahead<\/strong>: For predictable access patterns with hot data<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<p><strong>1. Cache-Aside (Lazy Loading)<\/strong><\/p>\n<p>Most common pattern:<\/p>\n<pre><code>def get_user(user_id):\n # Try cache first\n user = cache.get(f&#34;user:{user_id}&#34;)\n\n if user is None:\n # Cache miss - load from database\n user = database.query(f&#34;SELECT * FROM users WHERE id = {user_id}&#34;)\n\n # Store in cache for next time\n cache.set(f&#34;user:{user_id}&#34;, user, ttl=3600)\n\n return user<\/code><\/pre><p>Flow:<\/p>\n<ol>\n<li>Application checks cache<\/li>\n<li>If data exists (cache hit), return it<\/li>\n<li>If data doesn&rsquo;t exist (cache miss):\n<ul>\n<li>Load from database<\/li>\n<li>Write to cache<\/li>\n<li>Return data<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<p>Pros:<\/p>\n<ul>\n<li>Only requested data is cached<\/li>\n<li>Cache failure doesn&rsquo;t break application<\/li>\n<\/ul>\n<p>Cons:<\/p>\n<ul>\n<li>Initial request is slow (cache miss)<\/li>\n<li>Stale data possible if cache doesn&rsquo;t expire<\/li>\n<\/ul>\n<p><strong>2. Read-Through Cache<\/strong><\/p>\n<p>Cache sits between application and database:<\/p>\n<pre><code>class ReadThroughCache:\n def __init__(self, cache, database):\n self.cache = cache\n self.database = database\n\n def get(self, key):\n # Cache handles database lookup automatically\n return self.cache.get(key, loader=lambda: self.database.get(key))<\/code><\/pre><p>Flow:<\/p>\n<ol>\n<li>Application requests data from cache<\/li>\n<li>Cache checks if data exists<\/li>\n<li>If not, cache loads from database automatically<\/li>\n<li>Cache returns data<\/li>\n<\/ol>\n<p>Pros:<\/p>\n<ul>\n<li>Cleaner application code<\/li>\n<li>Consistent caching logic<\/li>\n<\/ul>\n<p>Cons:<\/p>\n<ul>\n<li>Cache becomes critical dependency<\/li>\n<li>Potential bottleneck<\/li>\n<\/ul>\n<p><strong>3. Write-Through Cache<\/strong><\/p>\n<p>Data written to cache and database simultaneously:<\/p>\n<pre><code>def update_user(user_id, data):\n # Write to both cache and database\n cache.set(f&#34;user:{user_id}&#34;, data)\n database.update(f&#34;UPDATE users SET ... WHERE id = {user_id}&#34;)<\/code><\/pre><p>Pros:<\/p>\n<ul>\n<li>Cache always consistent with database<\/li>\n<li>No stale data<\/li>\n<\/ul>\n<p>Cons:<\/p>\n<ul>\n<li>Slower writes (two operations)<\/li>\n<li>Cache may contain rarely-read data<\/li>\n<\/ul>\n<p><strong>4. Write-Behind (Write-Back) Cache<\/strong><\/p>\n<p>Data written to cache immediately, database updated asynchronously:<\/p>\n<pre><code>def update_user(user_id, data):\n # Write to cache immediately\n cache.set(f&#34;user:{user_id}&#34;, data)\n\n # Queue database write for later\n write_queue.push({\n &#39;operation&#39;: &#39;update&#39;,\n &#39;table&#39;: &#39;users&#39;,\n &#39;id&#39;: user_id,\n &#39;data&#39;: data\n })<\/code><\/pre><p>Pros:<\/p>\n<ul>\n<li>Very fast writes<\/li>\n<li>Can batch database writes<\/li>\n<\/ul>\n<p>Cons:<\/p>\n<ul>\n<li>Risk of data loss if cache fails<\/li>\n<li>More complex to implement<\/li>\n<li>Eventual consistency<\/li>\n<\/ul>\n<p><strong>5. Refresh-Ahead<\/strong><\/p>\n<p>Proactively refresh cache before expiration:<\/p>\n<pre><code>import time\nimport threading\n\ndef refresh_ahead_cache(key, ttl, refresh_threshold=0.8):\n data = cache.get(key)\n\n if data is None:\n # Cache miss - load and cache\n data = load_from_database(key)\n cache.set(key, data, ttl=ttl)\n else:\n # Check if approaching expiration\n remaining_ttl = cache.ttl(key)\n if remaining_ttl &lt; (ttl * refresh_threshold):\n # Asynchronously refresh\n threading.Thread(\n target=lambda: cache.set(key, load_from_database(key), ttl=ttl)\n ).start()\n\n return data<\/code><\/pre><h3 class=\"heading-element\" id=\"74-cache-eviction-policies\"><span>7.4 Cache Eviction Policies<\/span>\n <a href=\"#74-cache-eviction-policies\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>When cache is full, which data should be removed?<\/p>\n<p><strong>1. LRU (Least Recently Used)<\/strong><\/p>\n<ul>\n<li>Evicts least recently accessed items<\/li>\n<li>Most commonly used<\/li>\n<li>Good for general-purpose caching<\/li>\n<\/ul>\n<p><strong>2. LFU (Least Frequently Used)<\/strong><\/p>\n<ul>\n<li>Evicts items accessed least often<\/li>\n<li>Good for data with varying access patterns<\/li>\n<\/ul>\n<p><strong>3. FIFO (First In First Out)<\/strong><\/p>\n<ul>\n<li>Evicts oldest items first<\/li>\n<li>Simple but not always optimal<\/li>\n<\/ul>\n<p><strong>4. TTL (Time To Live)<\/strong><\/p>\n<ul>\n<li>Items expire after fixed time<\/li>\n<li>Good for time-sensitive data<\/li>\n<\/ul>\n<p><strong>Implementation of LRU Cache:<\/strong><\/p>\n<pre><code>from collections import OrderedDict\n\nclass LRUCache:\n def __init__(self, capacity):\n self.cache = OrderedDict()\n self.capacity = capacity\n\n def get(self, key):\n if key not in self.cache:\n return None\n # Move to end (most recently used)\n self.cache.move_to_end(key)\n return self.cache[key]\n\n def put(self, key, value):\n if key in self.cache:\n # Update and move to end\n self.cache.move_to_end(key)\n self.cache[key] = value\n\n if len(self.cache) &gt; self.capacity:\n # Remove least recently used (first item)\n self.cache.popitem(last=False)\n\n# Usage\ncache = LRUCache(capacity=3)\ncache.put(&#34;user:1&#34;, {&#34;name&#34;: &#34;Alice&#34;})\ncache.put(&#34;user:2&#34;, {&#34;name&#34;: &#34;Bob&#34;})\ncache.put(&#34;user:3&#34;, {&#34;name&#34;: &#34;Charlie&#34;})\ncache.put(&#34;user:4&#34;, {&#34;name&#34;: &#34;David&#34;}) # Evicts user:1<\/code><\/pre><h3 class=\"heading-element\" id=\"75-cache-considerations\"><span>7.5 Cache Considerations<\/span>\n <a href=\"#75-cache-considerations\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>1. Cache Expiration (TTL)<\/strong><\/p>\n<div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>TTL Strategy Pitfalls<\/strong>: Setting TTL too short causes cache misses and unnecessary database hits. Setting it too long risks serving stale data. Monitor cache hit ratios and adjust TTL dynamically based on actual data update frequency. For critical data, implement cache invalidation events rather than relying solely on TTL.<\/div>\n <\/div>\n<\/div>\n<p>Set appropriate TTL based on data characteristics:<\/p>\n<pre><code># Fast-changing data\ncache.set(&#34;stock_price:AAPL&#34;, price, ttl=60) # 1 minute\n\n# Slow-changing data\ncache.set(&#34;user_profile:123&#34;, profile, ttl=3600) # 1 hour\n\n# Rarely changing data\ncache.set(&#34;product_category&#34;, categories, ttl=86400) # 24 hours<\/code><\/pre><p><strong>2. Cache Stampede Prevention<\/strong><\/p>\n<div class=\"details admonition danger open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-bolt\" aria-hidden=\"true\"><\/i>Danger<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Cache Thundering Herd<\/strong>: When a popular cached item expires, hundreds or thousands of requests can simultaneously hit the database, causing a spike in load. This can bring down the database. Implement one of: (1) Locking (only first request loads), (2) Probabilistic early refresh (reload before expiration), or (3) Refresh-Ahead pattern.<\/div>\n <\/div>\n<\/div>\n<p>Problem: When cached item expires, multiple requests simultaneously hit database.<\/p>\n<p>Solution: Use locks or probabilistic early expiration:<\/p>\n<pre><code>import threading\nimport random\n\nlock_dict = {}\n\ndef get_with_stampede_protection(key, ttl, load_function):\n data = cache.get(key)\n\n if data is None:\n # Get or create lock for this key\n if key not in lock_dict:\n lock_dict[key] = threading.Lock()\n\n lock = lock_dict[key]\n\n # Only one thread loads data\n with lock:\n # Double-check cache (another thread may have loaded it)\n data = cache.get(key)\n if data is None:\n data = load_function()\n cache.set(key, data, ttl=ttl)\n\n # Probabilistic early refresh\n remaining_ttl = cache.ttl(key)\n if remaining_ttl &lt; ttl * 0.2: # Last 20% of TTL\n if random.random() &lt; 0.1: # 10% chance\n threading.Thread(\n target=lambda: cache.set(key, load_function(), ttl=ttl)\n ).start()\n\n return data<\/code><\/pre><p><strong>3. Cache Warming<\/strong><\/p>\n<p>Pre-populate cache with frequently accessed data:<\/p>\n<pre><code>def warm_cache():\n &#34;&#34;&#34;Run during application startup&#34;&#34;&#34;\n # Load popular users\n popular_users = database.query(&#34;SELECT * FROM users ORDER BY popularity DESC LIMIT 1000&#34;)\n for user in popular_users:\n cache.set(f&#34;user:{user.id}&#34;, user, ttl=3600)\n\n # Load popular products\n popular_products = database.query(&#34;SELECT * FROM products ORDER BY sales DESC LIMIT 5000&#34;)\n for product in popular_products:\n cache.set(f&#34;product:{product.id}&#34;, product, ttl=7200)<\/code><\/pre><p><strong>4. Distributed Caching<\/strong><\/p>\n<p>For high availability and scalability:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/distributed-redis-cluster.webp\" title=\"Redis Cluster Architecture \u2014 Distributed Redis deployment for scalability and fault tolerance in caching systems\" data-thumbnail=\"\/files\/distributed-redis-cluster.webp\" data-sub-html=\"<h2>Distributed Redis cluster with master-replica architecture \u2014 scalable caching and high availability for in-memory data store<\/h2><p>Redis Cluster Architecture \u2014 Distributed Redis deployment for scalability and fault tolerance in caching systems<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/distributed-redis-cluster.webp' alt=\"Distributed Redis cluster with master-replica architecture \u2014 scalable caching and high availability for in-memory data store\"><\/a><figcaption class=\"image-caption\">Redis Cluster Architecture \u2014 Distributed Redis deployment for scalability and fault tolerance in caching systems<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"76-redis-vs-memcached\"><span>7.6 Redis vs Memcached<\/span>\n <a href=\"#76-redis-vs-memcached\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Feature<\/th>\n <th>Redis<\/th>\n <th>Memcached<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>Data Structures<\/td>\n <td>Strings, Lists, Sets, Sorted Sets, Hashes, Bitmaps<\/td>\n <td>Only strings<\/td>\n <\/tr>\n <tr>\n <td>Persistence<\/td>\n <td>Yes (RDB, AOF)<\/td>\n <td>No<\/td>\n <\/tr>\n <tr>\n <td>Replication<\/td>\n <td>Yes<\/td>\n <td>No (requires external tools)<\/td>\n <\/tr>\n <tr>\n <td>Transactions<\/td>\n <td>Yes<\/td>\n <td>No<\/td>\n <\/tr>\n <tr>\n <td>Pub\/Sub<\/td>\n <td>Yes<\/td>\n <td>No<\/td>\n <\/tr>\n <tr>\n <td>Lua Scripting<\/td>\n <td>Yes<\/td>\n <td>No<\/td>\n <\/tr>\n <tr>\n <td>Multi-threading<\/td>\n <td>Single-threaded (6.0+ has I\/O threads)<\/td>\n <td>Multi-threaded<\/td>\n <\/tr>\n <tr>\n <td>Memory Efficiency<\/td>\n <td>Good<\/td>\n <td>Slightly better<\/td>\n <\/tr>\n <tr>\n <td>Use Case<\/td>\n <td>Complex caching, session store, queues<\/td>\n <td>Simple key-value caching<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><hr>\n<h2 class=\"heading-element\" id=\"8-content-delivery-networks-cdn\"><span>8. Content Delivery Networks (CDN)<\/span>\n <a href=\"#8-content-delivery-networks-cdn\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>CDNs are geographically distributed networks of servers that deliver content to users based on their location.<\/p>\n<h3 class=\"heading-element\" id=\"81-how-cdn-works\"><span>8.1 How CDN Works<\/span>\n <a href=\"#81-how-cdn-works\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/cdn-architecture.webp\" title=\"CDN Global Distribution \u2014 How content delivery networks reduce latency and improve performance worldwide\" data-thumbnail=\"\/files\/cdn-architecture.webp\" data-sub-html=\"<h2>CDN architecture with edge servers distributed globally \u2014 content delivery network for fast, low-latency content distribution<\/h2><p>CDN Global Distribution \u2014 How content delivery networks reduce latency and improve performance worldwide<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/cdn-architecture.webp' alt=\"CDN architecture with edge servers distributed globally \u2014 content delivery network for fast, low-latency content distribution\"><\/a><figcaption class=\"image-caption\">CDN Global Distribution \u2014 How content delivery networks reduce latency and improve performance worldwide<\/figcaption>\n <\/figure><\/p>\n<p><strong>CDN Request Flow:<\/strong><\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/cdn-request-flow.webp\" title=\"CDN Request Flow \u2014 Understanding cache hits, misses, and origin server interactions in CDN architecture\" data-thumbnail=\"\/files\/cdn-request-flow.webp\" data-sub-html=\"<h2>Sequence diagram showing CDN request flow and cache hit\/miss scenarios \u2014 how content delivery networks serve cached content<\/h2><p>CDN Request Flow \u2014 Understanding cache hits, misses, and origin server interactions in CDN architecture<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/cdn-request-flow.webp' alt=\"Sequence diagram showing CDN request flow and cache hit\/miss scenarios \u2014 how content delivery networks serve cached content\"><\/a><figcaption class=\"image-caption\">CDN Request Flow \u2014 Understanding cache hits, misses, and origin server interactions in CDN architecture<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"82-benefits-of-cdn\"><span>8.2 Benefits of CDN<\/span>\n <a href=\"#82-benefits-of-cdn\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li>\n<p><strong>Reduced Latency<\/strong><\/p>\n<ul>\n<li>Content served from nearest location<\/li>\n<li>Fewer network hops<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Reduced Origin Load<\/strong><\/p>\n<ul>\n<li>CDN handles most traffic<\/li>\n<li>Origin serves only cache misses<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Better Availability<\/strong><\/p>\n<ul>\n<li>Multiple edge servers provide redundancy<\/li>\n<li>Can serve stale content if origin is down<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>DDoS Protection<\/strong><\/p>\n<ul>\n<li>Distributed architecture absorbs attacks<\/li>\n<li>Traffic filtered at edge<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Cost Savings<\/strong><\/p>\n<ul>\n<li>Reduced bandwidth from origin<\/li>\n<li>Lower infrastructure costs<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<h3 class=\"heading-element\" id=\"83-what-to-put-on-cdn\"><span>8.3 What to Put on CDN?<\/span>\n <a href=\"#83-what-to-put-on-cdn\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Static Assets (Best for CDN):<\/strong><\/p>\n<ul>\n<li>Images<\/li>\n<li>CSS files<\/li>\n<li>JavaScript files<\/li>\n<li>Fonts<\/li>\n<li>Videos<\/li>\n<li>PDFs and downloadable files<\/li>\n<\/ul>\n<p><strong>Dynamic Content (Possible with advanced CDNs):<\/strong><\/p>\n<ul>\n<li>API responses with caching headers<\/li>\n<li>Personalized content (with edge computing)<\/li>\n<li>HTML pages<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"84-cdn-considerations\"><span>8.4 CDN Considerations<\/span>\n <a href=\"#84-cdn-considerations\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>1. Cache Invalidation<\/strong><\/p>\n<p>Challenge: How to update cached content?<\/p>\n<p><strong>Solutions:<\/strong><\/p>\n<p><strong>a. Time-Based (TTL)<\/strong><\/p>\n<pre><code>Cache-Control: max-age=86400 # Cache for 24 hours<\/code><\/pre><p><strong>b. Versioned URLs<\/strong><\/p>\n<pre><code>\/static\/app.js?v=1.2.3\n\/static\/app.1.2.3.js\n\/static\/app.abc123hash.js<\/code><\/pre><p><strong>c. Cache Purge API<\/strong><\/p>\n<pre><code>import requests\n\ndef purge_cdn_cache(url):\n # Cloudflare example\n response = requests.post(\n &#39;https:\/\/api.cloudflare.com\/client\/v4\/zones\/{zone_id}\/purge_cache&#39;,\n headers={&#39;Authorization&#39;: &#39;Bearer {api_token}&#39;},\n json={&#39;files&#39;: [url]}\n )\n return response.json()<\/code><\/pre><p><strong>2. Cache-Control Headers<\/strong><\/p>\n<pre><code># Don&#39;t cache\nCache-Control: no-store\n\n# Cache but revalidate\nCache-Control: no-cache\n\n# Cache for 1 hour\nCache-Control: max-age=3600\n\n# Cache for 1 hour, revalidate after expiry\nCache-Control: max-age=3600, must-revalidate\n\n# Private (browser only, not CDN)\nCache-Control: private, max-age=3600\n\n# Public (can be cached by CDN)\nCache-Control: public, max-age=86400\n\n# Immutable (never changes)\nCache-Control: public, max-age=31536000, immutable<\/code><\/pre><p><strong>3. CDN Push vs Pull<\/strong><\/p>\n<p><strong>Pull CDN (Most Common):<\/strong><\/p>\n<ul>\n<li>CDN fetches content from origin on first request<\/li>\n<li>Origin remains source of truth<\/li>\n<li>Lazy loading<\/li>\n<\/ul>\n<p><strong>Push CDN:<\/strong><\/p>\n<ul>\n<li>You upload content directly to CDN<\/li>\n<li>No origin server needed<\/li>\n<li>Manual or automated uploads<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"9-database-systems\"><span>9. Database Systems<\/span>\n <a href=\"#9-database-systems\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Database Selection<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">Choosing the right database is one of the most important system design decisions. The choice between SQL and NoSQL, along with specific database technologies, can significantly impact your system&rsquo;s performance, scalability, and maintainability.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"91-sql-vs-nosql\"><span>9.1 SQL vs NoSQL<\/span>\n <a href=\"#91-sql-vs-nosql\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Decision Framework<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">The choice between SQL and NoSQL is not binary. Many modern systems use polyglot persistence, combining different database types for different use cases within the same application.<\/div>\n <\/div>\n<\/div>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/sql-vs-nosql-database-types.webp\" title=\"Database Types Classification \u2014 SQL vs NoSQL: relational, document, key-value, graph, and column-family databases\" data-thumbnail=\"\/files\/sql-vs-nosql-database-types.webp\" data-sub-html=\"<h2>Classification diagram of SQL relational and NoSQL database types \u2014 choosing the right database technology<\/h2><p>Database Types Classification \u2014 SQL vs NoSQL: relational, document, key-value, graph, and column-family databases<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/sql-vs-nosql-database-types.webp' alt=\"Classification diagram of SQL relational and NoSQL database types \u2014 choosing the right database technology\"><\/a><figcaption class=\"image-caption\">Database Types Classification \u2014 SQL vs NoSQL: relational, document, key-value, graph, and column-family databases<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"92-when-to-use-sql\"><span>9.2 When to Use SQL?<\/span>\n <a href=\"#92-when-to-use-sql\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>ACID Guarantees<\/strong>: SQL databases provide ACID properties (Atomicity, Consistency, Isolation, Durability) which are critical for financial transactions, inventory management, and any scenario where data integrity cannot be compromised. If a transaction fails midway, all changes are rolled back - ensuring your data never ends up in an inconsistent state.<\/div>\n <\/div>\n<\/div>\n<p><strong>Use SQL (Relational Databases) When:<\/strong><\/p>\n<ol>\n<li>\n<p><strong>Data is structured and relationships are important<\/strong><\/p>\n<ul>\n<li>E-commerce (orders, customers, products)<\/li>\n<li>Financial systems (accounts, transactions)<\/li>\n<li>CRM systems<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>ACID compliance is required<\/strong><\/p>\n<ul>\n<li>Banking and financial applications<\/li>\n<li>Inventory management<\/li>\n<li>Any system where data consistency is critical<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Complex queries needed<\/strong><\/p>\n<ul>\n<li>Reporting and analytics<\/li>\n<li>Joins across multiple tables<\/li>\n<li>Aggregations and complex filtering<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Data integrity constraints<\/strong><\/p>\n<ul>\n<li>Foreign keys<\/li>\n<li>Unique constraints<\/li>\n<li>Check constraints<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<p><strong>Example Schema (E-commerce):<\/strong><\/p>\n<pre><code>CREATE TABLE users (\n id BIGINT PRIMARY KEY AUTO_INCREMENT,\n email VARCHAR(255) UNIQUE NOT NULL,\n name VARCHAR(255) NOT NULL,\n created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP\n);\n\nCREATE TABLE products (\n id BIGINT PRIMARY KEY AUTO_INCREMENT,\n name VARCHAR(255) NOT NULL,\n price DECIMAL(10, 2) NOT NULL,\n stock INT NOT NULL,\n category_id BIGINT,\n created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n FOREIGN KEY (category_id) REFERENCES categories(id)\n);\n\nCREATE TABLE orders (\n id BIGINT PRIMARY KEY AUTO_INCREMENT,\n user_id BIGINT NOT NULL,\n total_amount DECIMAL(10, 2) NOT NULL,\n status ENUM(&#39;pending&#39;, &#39;paid&#39;, &#39;shipped&#39;, &#39;delivered&#39;) DEFAULT &#39;pending&#39;,\n created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n FOREIGN KEY (user_id) REFERENCES users(id)\n);\n\nCREATE TABLE order_items (\n id BIGINT PRIMARY KEY AUTO_INCREMENT,\n order_id BIGINT NOT NULL,\n product_id BIGINT NOT NULL,\n quantity INT NOT NULL,\n price DECIMAL(10, 2) NOT NULL,\n FOREIGN KEY (order_id) REFERENCES orders(id),\n FOREIGN KEY (product_id) REFERENCES products(id)\n);<\/code><\/pre><h3 class=\"heading-element\" id=\"93-when-to-use-nosql\"><span>9.3 When to Use NoSQL?<\/span>\n <a href=\"#93-when-to-use-nosql\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Use NoSQL When:<\/strong><\/p>\n<ol>\n<li>\n<p><strong>Massive scale required<\/strong><\/p>\n<ul>\n<li>Billions of rows<\/li>\n<li>Petabytes of data<\/li>\n<li>Millions of operations per second<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Schema flexibility needed<\/strong><\/p>\n<ul>\n<li>Rapidly evolving data models<\/li>\n<li>Each record has different fields<\/li>\n<li>Semi-structured or unstructured data<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>High availability over consistency<\/strong><\/p>\n<ul>\n<li>Social media feeds<\/li>\n<li>Real-time analytics<\/li>\n<li>IoT data<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Denormalized data is acceptable<\/strong><\/p>\n<ul>\n<li>Data duplication is okay<\/li>\n<li>Joins are rare<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<p><strong>NoSQL Types:<\/strong><\/p>\n<p><strong>1. Key-Value Stores (Redis, DynamoDB)<\/strong><\/p>\n<p>Simplest NoSQL model:<\/p>\n<pre><code># Redis example\ncache.set(&#34;user:123&#34;, json.dumps({\n &#34;name&#34;: &#34;John Doe&#34;,\n &#34;email&#34;: &#34;john@example.com&#34;\n}))\n\nuser = json.loads(cache.get(&#34;user:123&#34;))<\/code><\/pre><p>Use cases:<\/p>\n<ul>\n<li>Session storage<\/li>\n<li>Caching<\/li>\n<li>Real-time analytics<\/li>\n<li>Shopping carts<\/li>\n<\/ul>\n<p><strong>2. Document Databases (MongoDB, CouchDB)<\/strong><\/p>\n<p>Store JSON-like documents:<\/p>\n<pre><code>\/\/ MongoDB example\n{\n &#34;_id&#34;: ObjectId(&#34;507f1f77bcf86cd799439011&#34;),\n &#34;name&#34;: &#34;John Doe&#34;,\n &#34;email&#34;: &#34;john@example.com&#34;,\n &#34;addresses&#34;: [\n {\n &#34;type&#34;: &#34;home&#34;,\n &#34;street&#34;: &#34;123 Main St&#34;,\n &#34;city&#34;: &#34;San Francisco&#34;,\n &#34;zip&#34;: &#34;94102&#34;\n },\n {\n &#34;type&#34;: &#34;work&#34;,\n &#34;street&#34;: &#34;456 Market St&#34;,\n &#34;city&#34;: &#34;San Francisco&#34;,\n &#34;zip&#34;: &#34;94103&#34;\n }\n ],\n &#34;orders&#34;: [\n {&#34;order_id&#34;: &#34;ORD001&#34;, &#34;total&#34;: 99.99},\n {&#34;order_id&#34;: &#34;ORD002&#34;, &#34;total&#34;: 149.99}\n ]\n}<\/code><\/pre><p>Use cases:<\/p>\n<ul>\n<li>Content management<\/li>\n<li>User profiles<\/li>\n<li>Product catalogs<\/li>\n<li>Real-time analytics<\/li>\n<\/ul>\n<p><strong>3. Column-Family Stores (Cassandra, HBase)<\/strong><\/p>\n<p>Optimized for write-heavy workloads:<\/p>\n<pre><code>Row Key: user_123\n Column Family: profile\n name: &#34;John Doe&#34;\n email: &#34;john@example.com&#34;\n\n Column Family: activity\n 2024-01-15:10:30:00: &#34;login&#34;\n 2024-01-15:10:31:22: &#34;viewed_product_456&#34;\n 2024-01-15:10:35:45: &#34;added_to_cart_456&#34;<\/code><\/pre><p>Use cases:<\/p>\n<ul>\n<li>Time-series data<\/li>\n<li>Event logging<\/li>\n<li>IoT sensor data<\/li>\n<li>Message systems<\/li>\n<\/ul>\n<p><strong>4. Graph Databases (Neo4j, Amazon Neptune)<\/strong><\/p>\n<p>Optimized for relationships:<\/p>\n<pre><code>\/\/ Neo4j example\n(John:Person {name: &#34;John Doe&#34;})\n-[:FRIENDS_WITH]-&gt;(Jane:Person {name: &#34;Jane Smith&#34;})\n-[:WORKS_AT]-&gt;(Company:Company {name: &#34;Acme Corp&#34;})\n&lt;-[:WORKS_AT]-(Bob:Person {name: &#34;Bob Johnson&#34;})\n-[:LIKES]-&gt;(Product:Product {name: &#34;Widget&#34;})<\/code><\/pre><p>Use cases:<\/p>\n<ul>\n<li>Social networks<\/li>\n<li>Recommendation engines<\/li>\n<li>Fraud detection<\/li>\n<li>Knowledge graphs<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"94-database-performance-optimization\"><span>9.4 Database Performance Optimization<\/span>\n <a href=\"#94-database-performance-optimization\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Performance Optimization Hierarchy<\/strong>: Follow this order for maximum impact:<\/p>\n<ol>\n<li><strong>Add indexes<\/strong> on frequently queried columns (10-100x speedup)<\/li>\n<li><strong>Optimize slow queries<\/strong> using EXPLAIN (5-50x speedup)<\/li>\n<li><strong>Add caching layer<\/strong> for hot data (10-100x speedup)<\/li>\n<li><strong>Scale reads with replicas<\/strong> when CPU-bound (linear scaling)<\/li>\n<li><strong>Shard database<\/strong> only when vertical scaling exhausted (adds complexity)<\/li>\n<\/ol><\/div>\n <\/div>\n<\/div>\n<p><strong>1. Indexing<\/strong><\/p>\n<div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Index Trade-offs<\/strong>: Every index speeds up reads but slows down writes. An insert\/update\/delete must update all indexes on that table. A table with 10 indexes will have 10x slower writes. Only index columns that are frequently used in WHERE, JOIN, or ORDER BY clauses. Monitor index usage with <code>EXPLAIN<\/code> and remove unused indexes.<\/div>\n <\/div>\n<\/div>\n<p>Indexes speed up queries but slow down writes:<\/p>\n<pre><code>-- Create index\nCREATE INDEX idx_users_email ON users(email);\nCREATE INDEX idx_products_category ON products(category_id);\n\n-- Composite index\nCREATE INDEX idx_orders_user_status ON orders(user_id, status);\n\n-- Full-text index\nCREATE FULLTEXT INDEX idx_products_search ON products(name, description);<\/code><\/pre><p><strong>Types of Indexes:<\/strong><\/p>\n<ul>\n<li><strong>B-Tree Index<\/strong> (default): Good for equality and range queries<\/li>\n<li><strong>Hash Index<\/strong>: Only for equality comparisons<\/li>\n<li><strong>Full-Text Index<\/strong>: For text search<\/li>\n<li><strong>Spatial Index<\/strong>: For geographic data<\/li>\n<\/ul>\n<p><strong>Index Best Practices:<\/strong><\/p>\n<ul>\n<li>Index columns used in WHERE, JOIN, ORDER BY<\/li>\n<li>Don&rsquo;t over-index (slows writes)<\/li>\n<li>Use composite indexes for multi-column queries<\/li>\n<li>Monitor index usage and remove unused indexes<\/li>\n<\/ul>\n<p><strong>2. Query Optimization<\/strong><\/p>\n<pre><code>-- Bad: SELECT *\nSELECT * FROM users WHERE email = &#39;john@example.com&#39;;\n\n-- Good: Select only needed columns\nSELECT id, name, email FROM users WHERE email = &#39;john@example.com&#39;;\n\n-- Use EXPLAIN to analyze queries\nEXPLAIN SELECT * FROM orders\nWHERE user_id = 123 AND status = &#39;pending&#39;;\n\n-- Avoid N&#43;1 queries\n-- Bad: Query in loop\nfor user_id in user_ids:\n orders = db.query(f&#34;SELECT * FROM orders WHERE user_id = {user_id}&#34;)\n\n-- Good: Single query\norders = db.query(f&#34;SELECT * FROM orders WHERE user_id IN ({&#39;,&#39;.join(user_ids)})&#34;)<\/code><\/pre><p><strong>3. Connection Pooling<\/strong><\/p>\n<pre><code>import psycopg2\nfrom psycopg2 import pool\n\n# Create connection pool\nconnection_pool = pool.SimpleConnectionPool(\n minconn=5,\n maxconn=20,\n host=&#34;localhost&#34;,\n database=&#34;mydb&#34;,\n user=&#34;user&#34;,\n password=&#34;password&#34;\n)\n\ndef execute_query(query):\n conn = connection_pool.getconn()\n try:\n cursor = conn.cursor()\n cursor.execute(query)\n result = cursor.fetchall()\n return result\n finally:\n connection_pool.putconn(conn)<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"10-database-sharding-and-partitioning\"><span>10. Database Sharding and Partitioning<\/span>\n <a href=\"#10-database-sharding-and-partitioning\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Sharding: Last Resort Scaling<\/strong>: Sharding adds significant complexity - cross-shard joins become application-level operations, distributed transactions are complex, and rebalancing data is challenging. Exhaust these options first: (1) Optimize queries and add indexes, (2) Add read replicas, (3) Vertical scaling (bigger machine), (4) Caching layer. Only shard when you absolutely must handle massive scale (billions of rows, millions of QPS).<\/div>\n <\/div>\n<\/div>\n<p>When a single database can&rsquo;t handle the load, we need to split data across multiple databases.<\/p>\n<h3 class=\"heading-element\" id=\"101-vertical-vs-horizontal-partitioning\"><span>10.1 Vertical vs Horizontal Partitioning<\/span>\n <a href=\"#101-vertical-vs-horizontal-partitioning\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Vertical Partitioning (Splitting by columns)<\/strong><\/p>\n<pre><code>Original Table:\nusers: id, name, email, address, bio, profile_pic\n\nSplit into:\nusers_basic: id, name, email\nusers_extended: id, address, bio, profile_pic<\/code><\/pre><p>Use when:<\/p>\n<ul>\n<li>Different columns accessed with different frequencies<\/li>\n<li>Some columns are rarely used<\/li>\n<li>Reduce I\/O for common queries<\/li>\n<\/ul>\n<p><strong>Horizontal Partitioning\/Sharding (Splitting by rows)<\/strong><\/p>\n<pre><code>users table split by region:\nusers_us: All US users\nusers_eu: All EU users\nusers_asia: All ASIA users<\/code><\/pre><h3 class=\"heading-element\" id=\"102-sharding-strategies\"><span>10.2 Sharding Strategies<\/span>\n <a href=\"#102-sharding-strategies\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Choosing a Sharding Strategy<\/strong>:<\/p>\n<ul>\n<li><strong>Range-Based<\/strong>: Use for time-series data or when range queries are common (e.g., &ldquo;get all orders from last month&rdquo;)<\/li>\n<li><strong>Hash-Based<\/strong>: Best for even distribution and single-key lookups (e.g., user profiles, sessions)<\/li>\n<li><strong>Geographic<\/strong>: Essential for data residency compliance (GDPR, data sovereignty laws)<\/li>\n<li><strong>Directory-Based<\/strong>: Most flexible but adds latency and single point of failure (shard directory)<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<p><strong>1. Range-Based Sharding<\/strong><\/p>\n<p>Split data based on a continuous range of the shard key \u2014 typically ID ranges, alphabetical ranges, or time ranges.<\/p>\n<pre><code>def get_shard_by_range(user_id):\n if user_id &lt; 1000000:\n return &#34;shard_1&#34;\n elif user_id &lt; 2000000:\n return &#34;shard_2&#34;\n elif user_id &lt; 3000000:\n return &#34;shard_3&#34;\n else:\n return &#34;shard_4&#34;<\/code><\/pre><p><strong>Pros:<\/strong> Range queries are fast \u2014 &ldquo;get all orders from last month&rdquo; maps to a single shard (or a small set of consecutive shards) with no cross-shard fan-out. Easy to understand and operate.<\/p>\n<p><strong>Cons:<\/strong> If your data grows non-uniformly, some shards get much more data than others \u2014 called a <strong>hot shard<\/strong> problem. For example, if most users were created recently, <code>shard_4<\/code> (user_id &gt; 3M) could be 10\u00d7 larger than <code>shard_1<\/code>. Adding new shards means splitting existing ranges, which requires migrating data.<\/p>\n<p><strong>Best for:<\/strong> Time-series data (IoT, logs, financial ticks) where you always query recent ranges, and old shards naturally become read-only.<\/p>\n<p><strong>2. Hash-Based Sharding<\/strong><\/p>\n<p>Apply a hash function to the shard key, then take the modulo to distribute rows evenly.<\/p>\n<pre><code>import hashlib\n\ndef get_shard_by_hash(user_id, num_shards):\n hash_value = int(hashlib.md5(str(user_id).encode()).hexdigest(), 16)\n shard_id = hash_value % num_shards\n return f&#34;shard_{shard_id}&#34;<\/code><\/pre><p><strong>Pros:<\/strong> Hash functions distribute data very evenly \u2014 with a good hash function and many keys, each shard gets roughly equal load. No hot shard problem. Single-key lookups (get user by ID) are O(1) \u2014 just hash the key and query one shard.<\/p>\n<p><strong>Cons:<\/strong> The biggest operational headache: when you add a shard, <code>num_shards<\/code> changes, so <code>hash(key) % num_shards<\/code> changes for most keys \u2014 meaning most data needs to be migrated to a different shard. This is why <strong>consistent hashing<\/strong> (\u00a718) was invented \u2014 it minimizes remapping when nodes are added\/removed, only ~1\/N keys need to move. Range queries are also impossible \u2014 &ldquo;get all users created in January&rdquo; would require querying all shards and merging.<\/p>\n<p><strong>Best for:<\/strong> Any workload where you primarily access data by a single known key (user profiles, session data, product catalog). The go-to choice when you just need even distribution.<\/p>\n<p><strong>3. Geographic Sharding<\/strong><\/p>\n<p>Route data to the shard closest to where it originates or is most often accessed.<\/p>\n<pre><code>def get_shard_by_geo(user_location):\n if user_location in [&#39;US&#39;, &#39;CA&#39;, &#39;MX&#39;]:\n return &#34;shard_americas&#34;\n elif user_location in [&#39;GB&#39;, &#39;FR&#39;, &#39;DE&#39;]:\n return &#34;shard_europe&#34;\n elif user_location in [&#39;CN&#39;, &#39;JP&#39;, &#39;IN&#39;]:\n return &#34;shard_asia&#34;\n else:\n return &#34;shard_global&#34;<\/code><\/pre><p><strong>Pros:<\/strong> Data latency is minimized \u2014 a European user&rsquo;s data lives in the European shard, physically co-located with the servers serving that user. Critical for regulatory compliance: GDPR requires EU user data to remain within the EU; geographic sharding is the standard way to achieve data residency requirements.<\/p>\n<p><strong>Cons:<\/strong> User population is not evenly spread globally \u2014 the Americas shard will be far larger than, say, an Africa shard. If a user travels internationally, their requests may cross regions, adding latency. Cross-shard queries (e.g., global analytics) are expensive.<\/p>\n<p><strong>Best for:<\/strong> Applications with legal data residency requirements, or where you need to minimize latency for geographically dispersed users (social networks, cloud storage, global SaaS).<\/p>\n<p><strong>4. Directory-Based Sharding<\/strong><\/p>\n<p>Maintain a lookup table:<\/p>\n<pre><code># Shard directory\nshard_directory = {\n &#34;user:1&#34;: &#34;shard_1&#34;,\n &#34;user:2&#34;: &#34;shard_1&#34;,\n &#34;user:3&#34;: &#34;shard_2&#34;,\n &#34;user:4&#34;: &#34;shard_3&#34;,\n}\n\ndef get_shard_by_directory(user_id):\n return shard_directory.get(f&#34;user:{user_id}&#34;, &#34;shard_default&#34;)<\/code><\/pre><p><strong>Pros:<\/strong> You have complete control over which data lives where \u2014 useful when you need to co-locate related data (e.g., put all data for a large enterprise customer on its own shard for isolation). Moving data between shards is just a metadata update (change the directory entry), not a data migration. You can also use this to gradually rebalance an overloaded shard without disrupting the application.<\/p>\n<p><strong>Cons:<\/strong> The directory itself becomes a <strong>critical dependency<\/strong> \u2014 every single database query must first hit the directory to find its shard. This adds latency (an extra network hop) and means the directory must be highly available. If the directory is down, the entire application is down. You can cache directory lookups to mitigate the latency, but cache invalidation adds complexity.<\/p>\n<p><strong>Best for:<\/strong> Systems with irregular data distribution requirements, large multi-tenant applications where different customers need different shard placements, or systems that need fine-grained control over data placement for compliance.<\/p>\n<h3 class=\"heading-element\" id=\"103-challenges-with-sharding\"><span>10.3 Challenges with Sharding<\/span>\n <a href=\"#103-challenges-with-sharding\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition danger open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-bolt\" aria-hidden=\"true\"><\/i>Danger<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Sharding Operational Complexity<\/strong>: Sharding introduces these hard problems:<\/p>\n<ol>\n<li><strong>Cross-shard joins<\/strong>: Must fetch from multiple shards and join in application (slow)<\/li>\n<li><strong>Distributed transactions<\/strong>: 2PC is slow and error-prone (avoid if possible)<\/li>\n<li><strong>Data rebalancing<\/strong>: Adding shards requires massive data migration (hours\/days downtime)<\/li>\n<li><strong>Hotspots<\/strong>: Uneven sharding key distribution causes some shards to be overloaded<\/li>\n<li><strong>Query routing<\/strong>: Routing layer becomes complex and critical<\/li>\n<\/ol><\/div>\n <\/div>\n<\/div>\n<p><strong>1. Cross-Shard Joins<\/strong><\/p>\n<div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Denormalization is Necessary<\/strong>: With sharding, you cannot do efficient cross-shard joins. Denormalize data by storing redundant copies across shards. Accept the write overhead and potential staleness. Example: Store user_name and user_email in orders table even though they&rsquo;re in users table.<\/div>\n <\/div>\n<\/div>\n<p>Problem: Can&rsquo;t join tables across different databases.<\/p>\n<p>Solution: Denormalize data or handle joins at application level:<\/p>\n<pre><code>def get_user_with_orders(user_id):\n # Get shard for user\n user_shard = get_shard(user_id)\n user = user_shard.query(f&#34;SELECT * FROM users WHERE id = {user_id}&#34;)\n\n # Orders might be on different shard\n order_shard = get_shard_for_orders(user_id)\n orders = order_shard.query(f&#34;SELECT * FROM orders WHERE user_id = {user_id}&#34;)\n\n # Combine in application\n user[&#39;orders&#39;] = orders\n return user<\/code><\/pre><p><strong>2. Distributed Transactions<\/strong><\/p>\n<div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>2PC Performance Penalty<\/strong>: Two-Phase Commit (2PC) requires locking all shards and coordinating across them. In MySQL, 2PC adds 20-50% latency overhead and can deadlock if timing is wrong. Better approach: Use Saga pattern - break transaction into compensating transactions, or ensure shard key is included in all transactions so data stays on one shard.<\/div>\n <\/div>\n<\/div>\n<p>Problem: Transaction spanning multiple shards.<\/p>\n<p>Solution: Two-Phase Commit (2PC) or avoid distributed transactions:<\/p>\n<pre><code>def transfer_money(from_user_id, to_user_id, amount):\n from_shard = get_shard(from_user_id)\n to_shard = get_shard(to_user_id)\n\n if from_shard == to_shard:\n # Same shard - normal transaction\n with from_shard.transaction():\n from_shard.execute(f&#34;UPDATE accounts SET balance = balance - {amount} WHERE user_id = {from_user_id}&#34;)\n to_shard.execute(f&#34;UPDATE accounts SET balance = balance &#43; {amount} WHERE user_id = {to_user_id}&#34;)\n else:\n # Different shards - use 2PC or saga pattern\n # (More complex - see distributed transactions section)\n pass<\/code><\/pre><p><strong>3. Rebalancing<\/strong><\/p>\n<p>Problem: Adding\/removing shards requires data migration.<\/p>\n<p>Solution: Consistent hashing or careful planning:<\/p>\n<pre><code>def rebalance_shards(old_shards, new_shards):\n # Calculate which data needs to move\n for key in get_all_keys():\n old_shard = hash(key) % old_shards\n new_shard = hash(key) % new_shards\n\n if old_shard != new_shard:\n # Migrate data\n data = get_data_from_shard(old_shard, key)\n write_data_to_shard(new_shard, key, data)\n delete_data_from_shard(old_shard, key)<\/code><\/pre><h3 class=\"heading-element\" id=\"104-sharding-architecture\"><span>10.4 Sharding Architecture<\/span>\n <a href=\"#104-sharding-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/database-sharding-architecture.webp\" title=\"Database Sharding \u2014 Horizontal partitioning strategies for scaling large databases; range, hash, geo, and directory-based sharding\" data-thumbnail=\"\/files\/database-sharding-architecture.webp\" data-sub-html=\"<h2>Database sharding and horizontal partitioning architecture \u2014 distributing data across multiple databases for massive scale<\/h2><p>Database Sharding \u2014 Horizontal partitioning strategies for scaling large databases; range, hash, geo, and directory-based sharding<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/database-sharding-architecture.webp' alt=\"Database sharding and horizontal partitioning architecture \u2014 distributing data across multiple databases for massive scale\"><\/a><figcaption class=\"image-caption\">Database Sharding \u2014 Horizontal partitioning strategies for scaling large databases; range, hash, geo, and directory-based sharding<\/figcaption>\n <\/figure><\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"11-database-replication\"><span>11. Database Replication<\/span>\n <a href=\"#11-database-replication\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Replication Benefits<\/strong>: Database replication is the easiest way to scale read traffic. With 3 read replicas, you can handle 3x the read load. Replication also provides data redundancy (if master fails, promote a replica) and enables running analytics\/backups on replicas without impacting production.<\/div>\n <\/div>\n<\/div>\n<p>Database replication is the process of copying data from one database to another to ensure redundancy, improve availability, and enhance read performance.<\/p>\n<h3 class=\"heading-element\" id=\"111-master-slave-replication\"><span>11.1 Master-Slave Replication<\/span>\n <a href=\"#111-master-slave-replication\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The most common replication pattern:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/database-replication-master-slave.webp\" title=\"Database Replication \u2014 Master-slave and multi-master replication strategies for availability and performance\" data-thumbnail=\"\/files\/database-replication-master-slave.webp\" data-sub-html=\"<h2>Master-slave database replication architecture \u2014 how to replicate data for high availability and read scalability<\/h2><p>Database Replication \u2014 Master-slave and multi-master replication strategies for availability and performance<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/database-replication-master-slave.webp' alt=\"Master-slave database replication architecture \u2014 how to replicate data for high availability and read scalability\"><\/a><figcaption class=\"image-caption\">Database Replication \u2014 Master-slave and multi-master replication strategies for availability and performance<\/figcaption>\n <\/figure><\/p>\n<p><strong>How it works:<\/strong><\/p>\n<ol>\n<li>All write operations go to the master database<\/li>\n<li>Master propagates changes to slave databases<\/li>\n<li>Read operations are distributed across slave databases<\/li>\n<li>Improves read performance through load distribution<\/li>\n<\/ol>\n<p><strong>Advantages:<\/strong><\/p>\n<ul>\n<li>Better read performance (distribute reads across multiple servers)<\/li>\n<li>Data backup (slaves serve as backups)<\/li>\n<li>Analytics without impacting production (run reports on slaves)<\/li>\n<li>High availability (promote slave if master fails)<\/li>\n<\/ul>\n<p><strong>Disadvantages:<\/strong><\/p>\n<ul>\n<li>Replication lag (slaves might be slightly behind master)<\/li>\n<li>Single point of failure for writes (only one master)<\/li>\n<li>Complexity in failover scenarios<\/li>\n<\/ul>\n<div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Replication Lag Problem<\/strong>: Replication is usually asynchronous - slaves can be seconds or even minutes behind the master. If a user writes data and immediately reads from a replica, they might not see their own write! Solutions: (1) Read-after-write consistency (read from master for short time after write), (2) Sticky sessions (always read from same replica), or (3) Use timestamps\/version numbers to detect stale reads.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"112-master-master-replication\"><span>11.2 Master-Master Replication<\/span>\n <a href=\"#112-master-master-replication\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Both databases accept writes and sync with each other:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/high-availability-architecture.webp\" title=\"High Availability Design \u2014 Redundancy, failover strategies, and fault tolerance for mission-critical systems\" data-thumbnail=\"\/files\/high-availability-architecture.webp\" data-sub-html=\"<h2>High availability architecture with redundancy and automatic failover \u2014 ensures system continues despite component failures<\/h2><p>High Availability Design \u2014 Redundancy, failover strategies, and fault tolerance for mission-critical systems<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/high-availability-architecture.webp' alt=\"High availability architecture with redundancy and automatic failover \u2014 ensures system continues despite component failures\"><\/a><figcaption class=\"image-caption\">High Availability Design \u2014 Redundancy, failover strategies, and fault tolerance for mission-critical systems<\/figcaption>\n <\/figure><\/p>\n<p><strong>Advantages:<\/strong><\/p>\n<ul>\n<li>No single point of failure for writes<\/li>\n<li>Better write performance (distribute writes)<\/li>\n<li>Lower latency for geographically distributed systems<\/li>\n<\/ul>\n<p><strong>Disadvantages:<\/strong><\/p>\n<ul>\n<li>Complex conflict resolution<\/li>\n<li>Potential for inconsistency<\/li>\n<li>More difficult to manage<\/li>\n<\/ul>\n<div class=\"details admonition danger open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-bolt\" aria-hidden=\"true\"><\/i>Danger<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Multi-Master Conflicts<\/strong>: With master-master replication, two masters can accept conflicting writes. Example: User A updates email to &ldquo;<a href=\"mailto:a@example.com\">a@example.com<\/a>&rdquo; on Master 1, User B updates same user&rsquo;s email to &ldquo;<a href=\"mailto:b@example.com\">b@example.com<\/a>&rdquo; on Master 2. Which wins? Last-Write-Wins (LWW) is simple but can lose data. Version vectors detect conflicts but require application-level resolution. Avoid multi-master if possible.<\/div>\n <\/div>\n<\/div>\n<p><strong>Conflict Resolution Strategies:<\/strong><\/p>\n<pre><code># Last Write Wins (LWW)\ndef resolve_conflict_lww(value1, value2):\n &#34;&#34;&#34;\n Choose the value with the latest timestamp\n &#34;&#34;&#34;\n if value1.timestamp &gt; value2.timestamp:\n return value1\n return value2\n\n# Version Vector\nclass VersionVector:\n def __init__(self):\n self.vector = {}\n\n def increment(self, node_id):\n self.vector[node_id] = self.vector.get(node_id, 0) &#43; 1\n\n def merge(self, other):\n &#34;&#34;&#34;Merge two version vectors&#34;&#34;&#34;\n for node_id, version in other.vector.items():\n self.vector[node_id] = max(\n self.vector.get(node_id, 0),\n version\n )\n\n def compare(self, other):\n &#34;&#34;&#34;\n Returns:\n - &#39;before&#39; if self &lt; other\n - &#39;after&#39; if self &gt; other\n - &#39;concurrent&#39; if conflicting\n &#34;&#34;&#34;\n self_greater = False\n other_greater = False\n\n all_nodes = set(self.vector.keys()) | set(other.vector.keys())\n\n for node in all_nodes:\n self_ver = self.vector.get(node, 0)\n other_ver = other.vector.get(node, 0)\n\n if self_ver &gt; other_ver:\n self_greater = True\n elif other_ver &gt; self_ver:\n other_greater = True\n\n if self_greater and not other_greater:\n return &#39;after&#39;\n elif other_greater and not self_greater:\n return &#39;before&#39;\n else:\n return &#39;concurrent&#39;<\/code><\/pre><h3 class=\"heading-element\" id=\"113-replication-methods\"><span>11.3 Replication Methods<\/span>\n <a href=\"#113-replication-methods\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Synchronous Replication:<\/strong><\/p>\n<ul>\n<li>Master waits for slave acknowledgment before confirming write<\/li>\n<li>Strong consistency<\/li>\n<li>Higher latency<\/li>\n<li>Used when data consistency is critical<\/li>\n<\/ul>\n<pre><code>def synchronous_write(data):\n # Write to master\n master.write(data)\n\n # Wait for all slaves to acknowledge\n for slave in slaves:\n slave.write(data)\n slave.acknowledge()\n\n return &#34;Write successful&#34;<\/code><\/pre><p><strong>Asynchronous Replication:<\/strong><\/p>\n<ul>\n<li>Master confirms write immediately<\/li>\n<li>Slaves updated eventually<\/li>\n<li>Lower latency<\/li>\n<li>Risk of data loss if master fails<\/li>\n<\/ul>\n<pre><code>def asynchronous_write(data):\n # Write to master\n master.write(data)\n\n # Queue replication (don&#39;t wait)\n for slave in slaves:\n replication_queue.enqueue({\n &#39;slave&#39;: slave,\n &#39;data&#39;: data\n })\n\n return &#34;Write successful&#34;<\/code><\/pre><p><strong>Semi-Synchronous Replication:<\/strong><\/p>\n<p>The master confirms a write only after at least one slave has acknowledged receiving the data \u2014 but doesn&rsquo;t wait for all slaves. This is the default in MySQL 5.5+ and is a pragmatic middle ground used by most production systems.<\/p>\n<p><strong>Why it&rsquo;s better than purely asynchronous:<\/strong> If the master crashes immediately after confirming a write, at least one replica already has that data. You haven&rsquo;t lost the write \u2014 you just need to promote the right replica. Pure async replication has a window where a confirmed write exists only in the (now-crashed) master&rsquo;s memory.<\/p>\n<p><strong>Why it&rsquo;s better than fully synchronous:<\/strong> You only wait for one acknowledgment, not all. If you have 3 replicas and one is slow or in a different AZ, you don&rsquo;t block on it. Typical added latency is just one extra network round-trip to the nearest replica (~0.5\u20131ms within the same datacenter).<\/p>\n<p><strong>Practical usage:<\/strong> MySQL&rsquo;s <code>rpl_semi_sync_master_enabled<\/code> implements this. Google Cloud SQL and AWS RDS use a form of semi-synchronous replication for their managed database services. For most applications, semi-synchronous replication gives you the durability guarantee you actually need without the full latency cost of synchronous replication.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"12-distributed-systems-concepts\"><span>12. Distributed Systems Concepts<\/span>\n <a href=\"#12-distributed-systems-concepts\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Distributed Systems Foundation<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">Understanding distributed systems is crucial for designing scalable applications. Modern applications rarely run on a single machine - they are distributed across multiple servers, data centers, and geographical regions.<\/div>\n <\/div>\n<\/div>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/distributed-systems-overview.webp\" title=\"Distributed Systems Overview \u2014 Nodes, consensus algorithms (Raft\/Paxos), leader election, and replicated data stores in a fault-tolerant cluster\" data-thumbnail=\"\/files\/distributed-systems-overview.webp\" data-sub-html=\"<h2>Distributed system architecture showing leader election, consensus, and data replication across nodes<\/h2><p>Distributed Systems Overview \u2014 Nodes, consensus algorithms (Raft\/Paxos), leader election, and replicated data stores in a fault-tolerant cluster<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/distributed-systems-overview.webp' alt=\"Distributed system architecture showing leader election, consensus, and data replication across nodes\"><\/a><figcaption class=\"image-caption\">Distributed Systems Overview \u2014 Nodes, consensus algorithms (Raft\/Paxos), leader election, and replicated data stores in a fault-tolerant cluster<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"121-what-is-a-distributed-system\"><span>12.1 What is a Distributed System?<\/span>\n <a href=\"#121-what-is-a-distributed-system\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Definition<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">A distributed system is a collection of independent computers that appears to users as a single coherent system.<\/div>\n <\/div>\n<\/div>\n<p><strong>Characteristics:<\/strong><\/p>\n<ul>\n<li><strong>Multiple autonomous computers<\/strong>: Independent nodes with their own CPU, memory, and storage<\/li>\n<li><strong>Connected through a network<\/strong>: Communication via TCP\/IP, message queues, or RPC<\/li>\n<li><strong>Coordinate actions through message passing<\/strong>: No shared memory between nodes<\/li>\n<li><strong>No shared memory<\/strong>: Each node has isolated memory space<\/li>\n<li><strong>Failures are partial<\/strong>: Some components can fail while others continue working<\/li>\n<\/ul>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Why Distributed Systems?<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Benefits:<\/strong><\/p>\n<ol>\n<li><strong>Scalability<\/strong>: Handle more load by adding machines (horizontal scaling)<\/li>\n<li><strong>Reliability<\/strong>: System continues working even if components fail<\/li>\n<li><strong>Performance<\/strong>: Process data closer to users (reduced latency)<\/li>\n<li><strong>Availability<\/strong>: No single point of failure with proper design<\/li>\n<\/ol>\n<p><strong>Challenges:<\/strong><\/p>\n<ol>\n<li><strong>Network is unreliable<\/strong>: Messages can be lost or delayed<\/li>\n<li><strong>Partial failures<\/strong>: Some components fail while others work<\/li>\n<li><strong>Synchronization<\/strong>: Coordinating actions is difficult without shared memory<\/li>\n<li><strong>Consistency<\/strong>: Keeping data consistent across nodes is complex<\/li>\n<\/ol>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"122-fallacies-of-distributed-computing\"><span>12.2 Fallacies of Distributed Computing<\/span>\n <a href=\"#122-fallacies-of-distributed-computing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>First identified by Peter Deutsch and colleagues at Sun Microsystems in the 1990s, these eight assumptions are mistakes that developers who come from single-machine programming routinely make. Violating them causes outages, data corruption, and subtle bugs that only appear in production.<\/p>\n<div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>The Eight Fallacies \u2014 and What to Do Instead<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>1. The network is reliable.<\/strong>\nReality: Networks fail constantly. Packets are lost, connections time out, switches reboot, NICs malfunction. In a large distributed system, network events happen every day. <em>What to do:<\/em> Implement retries with exponential backoff and jitter. Use circuit breakers to stop hammering failing services. Design your protocol to be idempotent so retries don&rsquo;t cause double-processing.<\/p>\n<p><strong>2. Latency is zero.<\/strong>\nReality: Even within the same datacenter, a round-trip network call takes ~0.5ms. Cross-region calls take 100\u2013200ms. This sounds small, but if you make 100 sequential remote calls to render one page, that&rsquo;s 50ms just in network overhead. <em>What to do:<\/em> Batch operations to reduce round trips. Cache frequently-needed remote data locally. Use async patterns so network calls don&rsquo;t block the critical path.<\/p>\n<p><strong>3. Bandwidth is infinite.<\/strong>\nReality: Network interfaces have finite capacity (typically 1\u201325 Gbps in cloud environments), and bandwidth costs money. Sending large payloads frequently saturates links and increases costs. <em>What to do:<\/em> Compress data in transit (gzip, snappy, protobuf instead of JSON). Paginate large responses. Use CDNs to offload static data delivery from your origin servers.<\/p>\n<p><strong>4. The network is secure.<\/strong>\nReality: Traffic on internal networks can be intercepted by a compromised host within the same datacenter. Supply-chain attacks and insider threats are real. <em>What to do:<\/em> Encrypt all traffic, even on internal networks (mTLS via service mesh). Authenticate service-to-service calls, not just client-to-server. Adopt zero-trust networking \u2014 assume the internal network is hostile.<\/p>\n<p><strong>5. Topology doesn&rsquo;t change.<\/strong>\nReality: Servers are added and removed constantly. DNS entries change. Load balancers update their backends. Cloud auto-scaling adds\/removes instances in seconds. A hardcoded IP address will break. <em>What to do:<\/em> Use service discovery (Consul, Kubernetes DNS). Never hardcode IPs. Use connection pooling with health checks that detect topology changes.<\/p>\n<p><strong>6. There is one administrator.<\/strong>\nReality: In a microservices system, each service is owned by a different team with different deployment schedules and on-call rotations. A change by one team can break another team&rsquo;s service. <em>What to do:<\/em> Publish stable API contracts (OpenAPI specs). Use consumer-driven contract testing (Pact) so API changes that break consumers are caught before deployment. Communicate across teams before making breaking changes.<\/p>\n<p><strong>7. Transport cost is zero.<\/strong>\nReality: Serializing data (JSON encoding, Protobuf marshalling), compressing it, and routing it through network stacks consumes CPU cycles on both ends. High-frequency, small-payload RPC calls can consume significant CPU even if individual calls are fast. <em>What to do:<\/em> Choose efficient serialization formats (Protocol Buffers instead of XML). Batch small messages into larger ones. Profile your serialization overhead \u2014 it&rsquo;s often surprising how much CPU JSON encoding consumes at scale.<\/p>\n<p><strong>8. The network is homogeneous.<\/strong>\nReality: Your system likely spans multiple cloud providers, on-premises datacenters, different operating systems, different programming languages, and different network hardware with varying capabilities. A packet from your Go service must be correctly interpreted by your Java service running on a different OS. <em>What to do:<\/em> Use language-neutral serialization (Protobuf, Avro, JSON). Version your APIs explicitly. Test across the actual environments you deploy to, not just your development machine.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 class=\"heading-element\" id=\"13-cap-theorem-and-consistency-models\"><span>13. CAP Theorem and Consistency Models<\/span>\n <a href=\"#13-cap-theorem-and-consistency-models\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Fundamental Trade-off<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">The CAP theorem, proposed by Eric Brewer, is one of the most important concepts in distributed systems. It defines the trade-offs between consistency, availability, and partition tolerance.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"131-cap-theorem\"><span>13.1 CAP Theorem<\/span>\n <a href=\"#131-cap-theorem\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>CAP Theorem Statement<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p>The CAP theorem states that a distributed system can provide at most <strong>TWO<\/strong> of the following three guarantees simultaneously:<\/p>\n<ul>\n<li><strong>C (Consistency)<\/strong>: All nodes see the same data at the same time<\/li>\n<li><strong>A (Availability)<\/strong>: Every request receives a response (success or failure)<\/li>\n<li><strong>P (Partition Tolerance)<\/strong>: System continues operating despite network partitions<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/cap-theorem-venn-diagram.webp\" title=\"CAP Theorem \u2014 Consistency vs Availability vs Partition Tolerance; critical concept for distributed system design decisions\" data-thumbnail=\"\/files\/cap-theorem-venn-diagram.webp\" data-sub-html=\"<h2>CAP theorem Venn diagram showing consistency, availability, and partition tolerance trade-offs \u2014 fundamental distributed systems principle<\/h2><p>CAP Theorem \u2014 Consistency vs Availability vs Partition Tolerance; critical concept for distributed system design decisions<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/cap-theorem-venn-diagram.webp' alt=\"CAP theorem Venn diagram showing consistency, availability, and partition tolerance trade-offs \u2014 fundamental distributed systems principle\"><\/a><figcaption class=\"image-caption\">CAP Theorem \u2014 Consistency vs Availability vs Partition Tolerance; critical concept for distributed system design decisions<\/figcaption>\n <\/figure><\/p>\n<p><strong>In Reality:<\/strong><\/p>\n<ul>\n<li>Network partitions will happen (P is mandatory)<\/li>\n<li>Must choose between C and A during partition<\/li>\n<li>When partition heals, system needs reconciliation strategy<\/li>\n<\/ul>\n<div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Partition Tolerance is Mandatory<\/strong>: In the real world, network partitions WILL happen - switches fail, network cables get cut, data centers lose connectivity. You cannot choose &ldquo;CA&rdquo; because you cannot avoid network partitions. The real choice is: During a partition, do you want Consistency (CP) or Availability (AP)?<\/div>\n <\/div>\n<\/div>\n<p><strong>CP Systems (Consistency + Partition Tolerance):<\/strong><\/p>\n<ul>\n<li>Wait for partition to heal<\/li>\n<li>Return errors during partition<\/li>\n<li>Example: Banking systems, MongoDB (with majority write concern)<\/li>\n<\/ul>\n<div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>CP System Trade-off<\/strong>: CP systems (like banking systems) choose to be unavailable during network partitions rather than show inconsistent data. If you can&rsquo;t reach a quorum of nodes, writes fail. This prevents showing incorrect account balances but means some operations fail during network issues.<\/div>\n <\/div>\n<\/div>\n<p><strong>AP Systems (Availability + Partition Tolerance):<\/strong><\/p>\n<ul>\n<li>Always accept reads\/writes<\/li>\n<li>Resolve conflicts later<\/li>\n<li>Example: Social media feeds, DNS, Cassandra<\/li>\n<\/ul>\n<div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>AP System Trade-off<\/strong>: AP systems (like social media feeds) stay available during network partitions but accept eventual consistency. You might temporarily see old data or conflicting updates, but the system never returns errors. Facebook chooses AP - it&rsquo;s better to show a slightly stale news feed than to be completely unavailable.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"132-consistency-models\"><span>13.2 Consistency Models<\/span>\n <a href=\"#132-consistency-models\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Consistency Model Selection<\/strong>: Choose based on your use case:<\/p>\n<ul>\n<li><strong>Strong Consistency<\/strong>: Financial transactions, inventory systems (can&rsquo;t sell same item twice)<\/li>\n<li><strong>Eventual Consistency<\/strong>: Social media feeds, analytics, caching (temporary staleness is acceptable)<\/li>\n<li><strong>Read-Your-Writes<\/strong>: User profiles, settings (users must see their own changes immediately)<\/li>\n<li><strong>Causal Consistency<\/strong>: Comment threads, chat (replies must appear after original message)<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<p><strong>Strong Consistency:<\/strong><\/p>\n<ul>\n<li>After a write, all reads see that value<\/li>\n<li>Highest consistency guarantee<\/li>\n<li>Higher latency<\/li>\n<\/ul>\n<pre><code>def strong_consistency_read():\n # Read from master or wait for replication\n return master.read()<\/code><\/pre><p><strong>Eventual Consistency:<\/strong><\/p>\n<ul>\n<li>If no new updates, eventually all reads return same value<\/li>\n<li>Lower latency<\/li>\n<li>Used in AP systems<\/li>\n<\/ul>\n<pre><code>def eventual_consistency_read():\n # Read from any replica\n # Might return stale data\n return random.choice(replicas).read()<\/code><\/pre><p><strong>Read-Your-Writes Consistency:<\/strong><\/p>\n<ul>\n<li>Users see their own writes immediately<\/li>\n<li>Others might see stale data<\/li>\n<\/ul>\n<pre><code>session_writes = {}\n\ndef write(user_id, data):\n master.write(data)\n session_writes[user_id] = time.now()\n\ndef read(user_id):\n if user_id in session_writes:\n # Read from master for this user\n return master.read()\n else:\n # Can read from replica for others\n return replica.read()<\/code><\/pre><p><strong>Monotonic Reads:<\/strong><\/p>\n<ul>\n<li>If user reads value A, subsequent reads won&rsquo;t return older values<\/li>\n<\/ul>\n<pre><code>user_last_read_time = {}\n\ndef monotonic_read(user_id):\n last_time = user_last_read_time.get(user_id, 0)\n\n # Only read from replicas caught up to last_time\n valid_replicas = [\n r for r in replicas\n if r.last_sync_time &gt;= last_time\n ]\n\n result = random.choice(valid_replicas).read()\n user_last_read_time[user_id] = time.now()\n return result<\/code><\/pre><p><strong>Causal Consistency:<\/strong><\/p>\n<ul>\n<li>Related operations are seen in correct order<\/li>\n<li>Independent operations can be seen in any order<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"133-base-properties\"><span>13.3 BASE Properties<\/span>\n <a href=\"#133-base-properties\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Alternative to ACID for distributed systems:<\/p>\n<ul>\n<li><strong>B<\/strong>asically <strong>A<\/strong>vailable: System appears to work most of the time<\/li>\n<li><strong>S<\/strong>oft state: State may change without input (due to eventual consistency)<\/li>\n<li><strong>E<\/strong>ventual consistency: System becomes consistent over time<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"14-message-queues-and-event-driven-architecture\"><span>14. Message Queues and Event-Driven Architecture<\/span>\n <a href=\"#14-message-queues-and-event-driven-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Asynchronous Communication<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">Message queues are essential for building scalable, decoupled distributed systems. They enable asynchronous communication between services, improving system reliability and flexibility.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"141-why-message-queues\"><span>14.1 Why Message Queues?<\/span>\n <a href=\"#141-why-message-queues\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Key Benefits<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p>Message queues enable asynchronous communication between services with these benefits:<\/p>\n<p><strong>1. Decoupling<\/strong>: Services don&rsquo;t need to know about each other directly<\/p>\n<ul>\n<li>Producers and consumers are independent<\/li>\n<li>Can evolve separately without coordination<\/li>\n<\/ul>\n<p><strong>2. Reliability<\/strong>: Messages persisted until successfully processed<\/p>\n<ul>\n<li>Messages won&rsquo;t be lost if consumer is down<\/li>\n<li>Retry mechanisms for failed processing<\/li>\n<\/ul>\n<p><strong>3. Scalability<\/strong>: Add more consumers to handle increased load<\/p>\n<ul>\n<li>Horizontal scaling of message processing<\/li>\n<li>Load distribution across multiple consumers<\/li>\n<\/ul>\n<p><strong>4. Buffering<\/strong>: Handle traffic spikes gracefully<\/p>\n<ul>\n<li>Queue absorbs burst traffic<\/li>\n<li>Consumers process at sustainable rate<\/li>\n<\/ul>\n<\/div>\n <\/div>\n<\/div>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/message-queue-architecture.webp\" title=\"Message Queue System \u2014 Async communication pattern using Kafka, RabbitMQ for system scalability and resilience\" data-thumbnail=\"\/files\/message-queue-architecture.webp\" data-sub-html=\"<h2>Message queue architecture with producers and consumers \u2014 asynchronous processing for decoupled system components<\/h2><p>Message Queue System \u2014 Async communication pattern using Kafka, RabbitMQ for system scalability and resilience<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/message-queue-architecture.webp' alt=\"Message queue architecture with producers and consumers \u2014 asynchronous processing for decoupled system components\"><\/a><figcaption class=\"image-caption\">Message Queue System \u2014 Async communication pattern using Kafka, RabbitMQ for system scalability and resilience<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"142-message-queue-patterns\"><span>14.2 Message Queue Patterns<\/span>\n <a href=\"#142-message-queue-patterns\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>1. Point-to-Point (Queue)<\/strong><\/p>\n<p>Each message consumed by exactly one consumer:<\/p>\n<pre><code>class MessageQueue:\n def __init__(self):\n self.queue = []\n self.lock = threading.Lock()\n\n def send(self, message):\n with self.lock:\n self.queue.append(message)\n\n def receive(self):\n with self.lock:\n if self.queue:\n return self.queue.pop(0)\n return None\n\n# Usage\nqueue = MessageQueue()\n\n# Producer\nqueue.send({&#34;order_id&#34;: 123, &#34;action&#34;: &#34;process&#34;})\n\n# Consumer\nmessage = queue.receive()\nif message:\n process_order(message)<\/code><\/pre><p><strong>2. Publish-Subscribe (Topic)<\/strong><\/p>\n<p>Each message consumed by all subscribers:<\/p>\n<pre><code>class PubSubTopic:\n def __init__(self):\n self.subscribers = []\n self.messages = []\n\n def subscribe(self, subscriber):\n self.subscribers.append(subscriber)\n\n def publish(self, message):\n for subscriber in self.subscribers:\n subscriber.receive(message)\n\n# Usage\ntopic = PubSubTopic()\n\n# Subscribers\nemail_service = EmailService()\nsms_service = SMSService()\npush_service = PushService()\n\ntopic.subscribe(email_service)\ntopic.subscribe(sms_service)\ntopic.subscribe(push_service)\n\n# Publisher\ntopic.publish({&#34;event&#34;: &#34;order_placed&#34;, &#34;user_id&#34;: 123})\n# All three services receive the message<\/code><\/pre><h3 class=\"heading-element\" id=\"143-popular-message-queue-systems\"><span>14.3 Popular Message Queue Systems<\/span>\n <a href=\"#143-popular-message-queue-systems\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Choosing the right message queue significantly impacts your system&rsquo;s behaviour. Here is what each tool is actually good at and when to reach for it:<\/p>\n<p><strong>RabbitMQ<\/strong> is a traditional message broker built around the <a href=\"https:\/\/www.amqp.org\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">AMQP protocol<\/a>. It excels when you need flexible routing logic \u2014 messages can be sent to exchanges that route them to queues based on routing keys, patterns, or headers. It supports per-message acknowledgements, dead-letter queues (where failed messages land for inspection), message priorities, and request\/reply patterns. RabbitMQ stores messages only as long as consumers need them; once consumed, they are deleted. <strong>Use it when<\/strong> you need complex routing, task queues where each job should be processed exactly once, or request-reply workflows.<\/p>\n<p><strong>Apache Kafka<\/strong> is fundamentally different from RabbitMQ \u2014 it is a distributed, persistent, append-only log, not a traditional queue. Messages are written to topics divided into partitions, and they stay there for a configurable retention period (days or weeks), not just until consumed. Multiple consumer groups can independently read the same messages at different offsets, enabling multiple services to process the same event stream. Kafka handles millions of events per second per broker with sequential disk I\/O (see \u00a744). <strong>Use it when<\/strong> you need event streaming, real-time data pipelines, audit logs, or when multiple independent consumers need to process the same events (e.g., analytics pipeline + notification service both consuming order events).<\/p>\n<p><strong>Amazon SQS<\/strong> is a fully managed, serverless queue \u2014 there are no servers to provision, patch, or scale. It handles queues of virtually any size and integrates natively with Lambda, EC2, and other AWS services. It offers two flavors: Standard (at-least-once delivery, best-effort ordering) and FIFO (exactly-once processing, strict ordering). <strong>Use it when<\/strong> you&rsquo;re already on AWS, want zero operational overhead, and need a simple work queue for decoupling services.<\/p>\n<p><strong>Redis Pub\/Sub<\/strong> broadcasts messages to all current subscribers in real time \u2014 with sub-millisecond latency because everything is in memory. However, if no subscriber is listening when a message is published, the message is <strong>permanently lost<\/strong>. Redis does not persist pub\/sub messages. <strong>Use it when<\/strong> you need real-time notifications or live updates where some message loss is acceptable (e.g., live chat &ldquo;user is typing&rdquo; indicator, live score updates, real-time dashboard refreshes).<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th><\/th>\n <th>RabbitMQ<\/th>\n <th>Kafka<\/th>\n <th>Amazon SQS<\/th>\n <th>Redis Pub\/Sub<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Throughput<\/strong><\/td>\n <td>High<\/td>\n <td>Very high<\/td>\n <td>High<\/td>\n <td>Very high<\/td>\n <\/tr>\n <tr>\n <td><strong>Message retention<\/strong><\/td>\n <td>Until consumed<\/td>\n <td>Days\/weeks<\/td>\n <td>14 days max<\/td>\n <td>None<\/td>\n <\/tr>\n <tr>\n <td><strong>Message ordering<\/strong><\/td>\n <td>Per-queue<\/td>\n <td>Per-partition<\/td>\n <td>FIFO mode only<\/td>\n <td>Broadcast order<\/td>\n <\/tr>\n <tr>\n <td><strong>Multiple consumers<\/strong><\/td>\n <td>Competing consumers<\/td>\n <td>Independent offsets<\/td>\n <td>Competing consumers<\/td>\n <td>All subscribers<\/td>\n <\/tr>\n <tr>\n <td><strong>Persistence<\/strong><\/td>\n <td>Yes (disk-backed)<\/td>\n <td>Yes (log)<\/td>\n <td>Yes (managed)<\/td>\n <td>No<\/td>\n <\/tr>\n <tr>\n <td><strong>Complexity<\/strong><\/td>\n <td>Medium<\/td>\n <td>High<\/td>\n <td>Low<\/td>\n <td>Very low<\/td>\n <\/tr>\n <tr>\n <td><strong>Best for<\/strong><\/td>\n <td>Task queues, routing<\/td>\n <td>Event streaming<\/td>\n <td>Simple AWS queues<\/td>\n <td>Real-time push<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"144-event-driven-architecture-example\"><span>14.4 Event-Driven Architecture Example<\/span>\n <a href=\"#144-event-driven-architecture-example\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Event Bus\nclass EventBus:\n def __init__(self):\n self.handlers = {}\n\n def subscribe(self, event_type, handler):\n if event_type not in self.handlers:\n self.handlers[event_type] = []\n self.handlers[event_type].append(handler)\n\n def publish(self, event):\n event_type = event.get(&#39;type&#39;)\n if event_type in self.handlers:\n for handler in self.handlers[event_type]:\n try:\n handler(event)\n except Exception as e:\n print(f&#34;Handler error: {e}&#34;)\n\n# Services\ndef send_confirmation_email(event):\n user_id = event[&#39;user_id&#39;]\n order_id = event[&#39;order_id&#39;]\n print(f&#34;Sending email to user {user_id} for order {order_id}&#34;)\n\ndef update_inventory(event):\n items = event[&#39;items&#39;]\n print(f&#34;Updating inventory for {items}&#34;)\n\ndef send_notification(event):\n user_id = event[&#39;user_id&#39;]\n print(f&#34;Sending push notification to user {user_id}&#34;)\n\n# Setup\nevent_bus = EventBus()\nevent_bus.subscribe(&#39;order_placed&#39;, send_confirmation_email)\nevent_bus.subscribe(&#39;order_placed&#39;, update_inventory)\nevent_bus.subscribe(&#39;order_placed&#39;, send_notification)\n\n# Trigger event\nevent_bus.publish({\n &#39;type&#39;: &#39;order_placed&#39;,\n &#39;order_id&#39;: 12345,\n &#39;user_id&#39;: 789,\n &#39;items&#39;: [&#39;item1&#39;, &#39;item2&#39;]\n})<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"15-microservices-architecture\"><span>15. Microservices Architecture<\/span>\n <a href=\"#15-microservices-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"151-monolith-vs-microservices\"><span>15.1 Monolith vs Microservices<\/span>\n <a href=\"#151-monolith-vs-microservices\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/microservices-architecture.webp\" title=\"Microservices Architecture \u2014 Service decomposition, API gateway pattern, and inter-service communication for distributed systems\" data-thumbnail=\"\/files\/microservices-architecture.webp\" data-sub-html=\"<h2>Microservices architecture diagram with API gateway and independent services \u2014 service-oriented design for scalability and maintainability<\/h2><p>Microservices Architecture \u2014 Service decomposition, API gateway pattern, and inter-service communication for distributed systems<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/microservices-architecture.webp' alt=\"Microservices architecture diagram with API gateway and independent services \u2014 service-oriented design for scalability and maintainability\"><\/a><figcaption class=\"image-caption\">Microservices Architecture \u2014 Service decomposition, API gateway pattern, and inter-service communication for distributed systems<\/figcaption>\n <\/figure><\/p>\n<p><strong>Monolith Advantages:<\/strong><\/p>\n<ul>\n<li>Simple to develop and test<\/li>\n<li>Easy deployment<\/li>\n<li>Good for small teams<\/li>\n<li>Lower latency (no network calls)<\/li>\n<\/ul>\n<p><strong>Monolith Disadvantages:<\/strong><\/p>\n<ul>\n<li>Difficult to scale<\/li>\n<li>Long deployment times<\/li>\n<li>One bug can bring down entire system<\/li>\n<li>Hard to adopt new technologies<\/li>\n<\/ul>\n<p><strong>Microservices Advantages:<\/strong><\/p>\n<ul>\n<li>Independent scaling<\/li>\n<li>Technology diversity<\/li>\n<li>Fault isolation<\/li>\n<li>Faster deployment<\/li>\n<li>Team autonomy<\/li>\n<\/ul>\n<p><strong>Microservices Disadvantages:<\/strong><\/p>\n<ul>\n<li>Increased complexity<\/li>\n<li>Network latency<\/li>\n<li>Data consistency challenges<\/li>\n<li>Testing is harder<\/li>\n<li>Operational overhead<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"152-microservices-best-practices\"><span>15.2 Microservices Best Practices<\/span>\n <a href=\"#152-microservices-best-practices\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>1. Single Responsibility<\/strong><\/p>\n<p>Each service should do one thing well:<\/p>\n<pre><code>Good:\n- UserService: Manages users\n- OrderService: Manages orders\n- EmailService: Sends emails\n\nBad:\n- UserOrderEmailService: Does everything<\/code><\/pre><p><strong>2. Database per Service<\/strong><\/p>\n<p>Each service has its own database:<\/p>\n<pre><code># User Service\nclass UserService:\n def __init__(self):\n self.db = UserDatabase()\n\n def create_user(self, data):\n return self.db.insert(data)\n\n def get_user(self, user_id):\n return self.db.query(user_id)\n\n# Order Service\nclass OrderService:\n def __init__(self):\n self.db = OrderDatabase() # Different database\n self.user_service = UserServiceClient()\n\n def create_order(self, user_id, items):\n # Get user info via API (not direct DB access)\n user = self.user_service.get_user(user_id)\n\n order = {\n &#39;user_id&#39;: user_id,\n &#39;items&#39;: items,\n &#39;total&#39;: self.calculate_total(items)\n }\n return self.db.insert(order)<\/code><\/pre><p><strong>3. API Gateway Pattern<\/strong><\/p>\n<p>Single entry point for all clients:<\/p>\n<pre><code>class APIGateway:\n def __init__(self):\n self.user_service = UserServiceClient()\n self.order_service = OrderServiceClient()\n self.product_service = ProductServiceClient()\n\n def get_user_dashboard(self, user_id):\n # Aggregate data from multiple services\n user = self.user_service.get_user(user_id)\n orders = self.order_service.get_user_orders(user_id)\n recommendations = self.product_service.get_recommendations(user_id)\n\n return {\n &#39;user&#39;: user,\n &#39;recent_orders&#39;: orders[:5],\n &#39;recommended_products&#39;: recommendations\n }<\/code><\/pre><p><strong>4. Circuit Breaker Pattern<\/strong><\/p>\n<p>Prevent cascading failures:<\/p>\n<pre><code>from enum import Enum\nimport time\n\nclass CircuitState(Enum):\n CLOSED = &#34;closed&#34; # Normal operation\n OPEN = &#34;open&#34; # Failing, reject requests\n HALF_OPEN = &#34;half_open&#34; # Testing if recovered\n\nclass CircuitBreaker:\n def __init__(self, failure_threshold=5, timeout=60):\n self.failure_count = 0\n self.failure_threshold = failure_threshold\n self.timeout = timeout\n self.state = CircuitState.CLOSED\n self.last_failure_time = None\n\n def call(self, func, *args, **kwargs):\n if self.state == CircuitState.OPEN:\n if time.time() - self.last_failure_time &gt; self.timeout:\n self.state = CircuitState.HALF_OPEN\n else:\n raise Exception(&#34;Circuit breaker is OPEN&#34;)\n\n try:\n result = func(*args, **kwargs)\n self.on_success()\n return result\n except Exception as e:\n self.on_failure()\n raise e\n\n def on_success(self):\n self.failure_count = 0\n self.state = CircuitState.CLOSED\n\n def on_failure(self):\n self.failure_count &#43;= 1\n self.last_failure_time = time.time()\n\n if self.failure_count &gt;= self.failure_threshold:\n self.state = CircuitState.OPEN\n\n# Usage\npayment_circuit = CircuitBreaker(failure_threshold=5, timeout=60)\n\ndef process_payment(amount):\n return payment_circuit.call(payment_service.charge, amount)<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"16-service-mesh-and-api-gateway\"><span>16. Service Mesh and API Gateway<\/span>\n <a href=\"#16-service-mesh-and-api-gateway\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"161-api-gateway\"><span>16.1 API Gateway<\/span>\n <a href=\"#161-api-gateway\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>An <strong>API Gateway<\/strong> is the single entry point for all client requests to a microservices system. It handles cross-cutting concerns so individual services don&rsquo;t have to.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/api-gateway-pattern.webp\" title=\"API Gateway Pattern \u2014 Routing, rate limiting, authentication, and protocol translation for scalable microservices\" data-thumbnail=\"\/files\/api-gateway-pattern.webp\" data-sub-html=\"<h2>API Gateway pattern for routing, authentication, and request management \u2014 centralized entry point for microservices<\/h2><p>API Gateway Pattern \u2014 Routing, rate limiting, authentication, and protocol translation for scalable microservices<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/api-gateway-pattern.webp' alt=\"API Gateway pattern for routing, authentication, and request management \u2014 centralized entry point for microservices\"><\/a><figcaption class=\"image-caption\">API Gateway Pattern \u2014 Routing, rate limiting, authentication, and protocol translation for scalable microservices<\/figcaption>\n <\/figure><\/p>\n<p><strong>What an API Gateway does:<\/strong><\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Concern<\/th>\n <th>Description<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Authentication \/ Authorization<\/strong><\/td>\n <td>Verify JWT tokens or API keys before routing<\/td>\n <\/tr>\n <tr>\n <td><strong>Rate Limiting<\/strong><\/td>\n <td>Enforce per-client request quotas<\/td>\n <\/tr>\n <tr>\n <td><strong>Request Routing<\/strong><\/td>\n <td>Map <code>\/api\/users<\/code> \u2192 UserService, <code>\/api\/orders<\/code> \u2192 OrderService<\/td>\n <\/tr>\n <tr>\n <td><strong>SSL Termination<\/strong><\/td>\n <td>Decrypt HTTPS at the gateway, use plain HTTP internally<\/td>\n <\/tr>\n <tr>\n <td><strong>Load Balancing<\/strong><\/td>\n <td>Distribute traffic across service instances<\/td>\n <\/tr>\n <tr>\n <td><strong>Request\/Response Transformation<\/strong><\/td>\n <td>Protocol translation (REST \u2192 gRPC), header injection<\/td>\n <\/tr>\n <tr>\n <td><strong>Circuit Breaking<\/strong><\/td>\n <td>Fail fast when downstream services are unhealthy<\/td>\n <\/tr>\n <tr>\n <td><strong>Observability<\/strong><\/td>\n <td>Centralized access logs, request tracing<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Popular API Gateways:<\/strong><\/p>\n<ul>\n<li><strong><a href=\"https:\/\/konghq.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Kong<\/a><\/strong> \u2014 Open source, plugin-based, high performance (built on NGINX\/OpenResty)<\/li>\n<li><strong><a href=\"https:\/\/aws.amazon.com\/api-gateway\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">AWS API Gateway<\/a><\/strong> \u2014 Fully managed, native Lambda integration<\/li>\n<li><strong><a href=\"https:\/\/nginx.org\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">NGINX \/ Caddy<\/a><\/strong> \u2014 Lightweight, often used as a reverse proxy with gateway functionality<\/li>\n<li><strong><a href=\"https:\/\/traefik.io\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Traefik<\/a><\/strong> \u2014 Cloud-native, auto-discovers services in Kubernetes<\/li>\n<\/ul>\n<pre><code>class APIGateway:\n def __init__(self):\n self.routes = {}\n self.rate_limiter = DistributedRateLimiter(redis_client)\n self.auth = JWTAuthenticator()\n\n def register_route(self, path_prefix, service_url):\n self.routes[path_prefix] = service_url\n\n def handle_request(self, request):\n # 1. Authentication\n token = request.headers.get(&#39;Authorization&#39;, &#39;&#39;).replace(&#39;Bearer &#39;, &#39;&#39;)\n if not self.auth.verify(token):\n return Response(status=401, body=&#39;Unauthorized&#39;)\n\n # 2. Rate limiting\n client_id = self.auth.get_subject(token)\n if not self.rate_limiter.allow_request(client_id, limit=1000, window=3600):\n return Response(status=429, body=&#39;Too Many Requests&#39;)\n\n # 3. Route to downstream service\n for prefix, service_url in self.routes.items():\n if request.path.startswith(prefix):\n downstream_path = request.path[len(prefix):]\n return self.forward(request, service_url, downstream_path)\n\n return Response(status=404, body=&#39;Not Found&#39;)\n\n def forward(self, request, service_url, path):\n &#34;&#34;&#34;Forward request to downstream service.&#34;&#34;&#34;\n import requests as http\n response = http.request(\n method=request.method,\n url=f&#34;{service_url}{path}&#34;,\n headers=request.headers,\n data=request.body,\n timeout=5\n )\n return Response(status=response.status_code, body=response.text)<\/code><\/pre><h3 class=\"heading-element\" id=\"162-service-mesh\"><span>16.2 Service Mesh<\/span>\n <a href=\"#162-service-mesh\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>A <strong>service mesh<\/strong> manages service-to-service communication within a microservices cluster. Unlike an API Gateway (which handles north-south traffic from clients), a service mesh handles <strong>east-west traffic<\/strong> between services.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/service-mesh-architecture.webp\" title=\"Service Mesh \u2014 Istio and Envoy sidecar proxy architecture for microservices communication, observability, and security\" data-thumbnail=\"\/files\/service-mesh-architecture.webp\" data-sub-html=\"<h2>Service mesh architecture with Istio pattern \u2014 service-to-service communication, observability, and reliability<\/h2><p>Service Mesh \u2014 Istio and Envoy sidecar proxy architecture for microservices communication, observability, and security<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/service-mesh-architecture.webp' alt=\"Service mesh architecture with Istio pattern \u2014 service-to-service communication, observability, and reliability\"><\/a><figcaption class=\"image-caption\">Service Mesh \u2014 Istio and Envoy sidecar proxy architecture for microservices communication, observability, and security<\/figcaption>\n <\/figure><\/p>\n<p><strong>How it works \u2014 Sidecar Proxy Pattern:<\/strong><\/p>\n<ul>\n<li>Each service pod gets a sidecar proxy (typically <a href=\"https:\/\/www.envoyproxy.io\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Envoy<\/a>)<\/li>\n<li>All inbound\/outbound traffic passes through the proxy<\/li>\n<li>The control plane configures proxies centrally (Istio&rsquo;s <code>istiod<\/code>)<\/li>\n<\/ul>\n<p><strong>What a Service Mesh provides:<\/strong><\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Feature<\/th>\n <th>Description<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Mutual TLS (mTLS)<\/strong><\/td>\n <td>Automatic encryption between services, certificate rotation<\/td>\n <\/tr>\n <tr>\n <td><strong>Load Balancing<\/strong><\/td>\n <td>Intelligent traffic splitting, canary deployments<\/td>\n <\/tr>\n <tr>\n <td><strong>Circuit Breaking<\/strong><\/td>\n <td>Automatic failure detection and request short-circuiting<\/td>\n <\/tr>\n <tr>\n <td><strong>Observability<\/strong><\/td>\n <td>Distributed tracing, metrics per service, traffic visualization<\/td>\n <\/tr>\n <tr>\n <td><strong>Retry \/ Timeout<\/strong><\/td>\n <td>Configurable retry policies without application code changes<\/td>\n <\/tr>\n <tr>\n <td><strong>Traffic Management<\/strong><\/td>\n <td>A\/B testing, blue-green deployments, fault injection<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Popular Service Meshes:<\/strong><\/p>\n<ul>\n<li><strong><a href=\"https:\/\/istio.io\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Istio<\/a><\/strong> \u2014 Most feature-rich, uses Envoy sidecars + istiod control plane<\/li>\n<li><strong><a href=\"https:\/\/linkerd.io\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Linkerd<\/a><\/strong> \u2014 Lightweight, Rust-based data plane<\/li>\n<li><strong><a href=\"https:\/\/aws.amazon.com\/app-mesh\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">AWS App Mesh<\/a><\/strong> \u2014 AWS-native, integrates with ECS and EKS<\/li>\n<li><strong><a href=\"https:\/\/www.consul.io\/docs\/connect\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Consul Connect<\/a><\/strong> \u2014 HashiCorp, multi-platform<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"163-api-gateway-vs-service-mesh\"><span>16.3 API Gateway vs Service Mesh<\/span>\n <a href=\"#163-api-gateway-vs-service-mesh\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Concern<\/th>\n <th>API Gateway<\/th>\n <th>Service Mesh<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>Traffic direction<\/td>\n <td>North-South (client \u2192 system)<\/td>\n <td>East-West (service \u2192 service)<\/td>\n <\/tr>\n <tr>\n <td>Authentication<\/td>\n <td>External clients (JWT, API keys)<\/td>\n <td>Internal mTLS between services<\/td>\n <\/tr>\n <tr>\n <td>Typical location<\/td>\n <td>Cluster edge \/ perimeter<\/td>\n <td>Inside the cluster (sidecar)<\/td>\n <\/tr>\n <tr>\n <td>Scope<\/td>\n <td>Single entry point<\/td>\n <td>Every service-to-service call<\/td>\n <\/tr>\n <tr>\n <td>Examples<\/td>\n <td>Kong, AWS API Gateway, NGINX<\/td>\n <td>Istio, Linkerd, Consul<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Use Both Together<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">Production systems often use both: an <strong>API Gateway<\/strong> at the perimeter handles external clients and auth, while a <strong>Service Mesh<\/strong> manages internal service communication, mTLS, and observability. They are complementary, not alternatives.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 class=\"heading-element\" id=\"25-scaling-from-zero-to-million-users\"><span>25. Scaling from Zero to Million Users<\/span>\n <a href=\"#25-scaling-from-zero-to-million-users\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Progressive Scaling Journey<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">This section walks through building a system that scales from a single server to handling millions of concurrent users. Understanding this progression helps you make informed architecture decisions based on your current scale and future needs.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"251-stage-1-single-server\"><span>25.1 Stage 1: Single Server<\/span>\n <a href=\"#251-stage-1-single-server\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Start Simple<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">At the beginning, everything runs on one server. This is perfectly fine for early-stage startups and MVPs. Don&rsquo;t over-engineer for scale you don&rsquo;t have yet!<\/div>\n <\/div>\n<\/div>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/zero-to-million-users-evolution.webp\" title=\"Scaling from Zero to Million Users \u2014 Architecture evolution stages and strategies for rapid growth\" data-thumbnail=\"\/files\/zero-to-million-users-evolution.webp\" data-sub-html=\"<h2>System evolution and scaling journey from zero to million users \u2014 architecture growth stages and scaling strategies<\/h2><p>Scaling from Zero to Million Users \u2014 Architecture evolution stages and strategies for rapid growth<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/zero-to-million-users-evolution.webp' alt=\"System evolution and scaling journey from zero to million users \u2014 architecture growth stages and scaling strategies\"><\/a><figcaption class=\"image-caption\">Scaling from Zero to Million Users \u2014 Architecture evolution stages and strategies for rapid growth<\/figcaption>\n <\/figure><\/p>\n<p><strong>Architecture:<\/strong><\/p>\n<ul>\n<li>All code in one place<\/li>\n<li>Single database<\/li>\n<li>Simple deployment<\/li>\n<li>No external services<\/li>\n<\/ul>\n<p><strong>Configuration:<\/strong><\/p>\n<ul>\n<li>1 web server<\/li>\n<li>1 database<\/li>\n<li>~1,000-5,000 concurrent users<\/li>\n<\/ul>\n<p><strong>Issues emerge when:<\/strong><\/p>\n<ul>\n<li>Single point of failure<\/li>\n<li>Can&rsquo;t scale beyond server capacity<\/li>\n<li>Database bottleneck<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"252-stage-2-separate-web-and-database-tiers\"><span>25.2 Stage 2: Separate Web and Database Tiers<\/span>\n <a href=\"#252-stage-2-separate-web-and-database-tiers\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/zero-to-million-users-evolution.webp\" title=\"Scaling from Zero to Million Users \u2014 Architecture evolution stages and strategies for rapid growth\" data-thumbnail=\"\/files\/zero-to-million-users-evolution.webp\" data-sub-html=\"<h2>System evolution and scaling journey from zero to million users \u2014 architecture growth stages and scaling strategies<\/h2><p>Scaling from Zero to Million Users \u2014 Architecture evolution stages and strategies for rapid growth<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/zero-to-million-users-evolution.webp' alt=\"System evolution and scaling journey from zero to million users \u2014 architecture growth stages and scaling strategies\"><\/a><figcaption class=\"image-caption\">Scaling from Zero to Million Users \u2014 Architecture evolution stages and strategies for rapid growth<\/figcaption>\n <\/figure><\/p>\n<p><strong>Changes:<\/strong><\/p>\n<ul>\n<li>Add load balancer (distribute traffic)<\/li>\n<li>Multiple web servers (can scale horizontally)<\/li>\n<li>Separate database (can upgrade independently)<\/li>\n<\/ul>\n<p><strong>Capacity:<\/strong><\/p>\n<ul>\n<li>~100,000 concurrent users<\/li>\n<li>Load balancer: 1-2 Gbps throughput<\/li>\n<li>Web servers: 5-10 servers<\/li>\n<li>Database: Single instance (might be beefy)<\/li>\n<\/ul>\n<p><strong>Implementation Example:<\/strong><\/p>\n<pre><code># Load Balancer (Round Robin)\nclass LoadBalancer:\n def __init__(self, servers):\n self.servers = servers\n self.current = 0\n\n def route_request(self, request):\n server = self.servers[self.current]\n self.current = (self.current &#43; 1) % len(self.servers)\n return server.handle(request)\n\n# Web Servers (Stateless)\nclass WebServer:\n def __init__(self, db_connection):\n self.db = db_connection\n\n def handle(self, request):\n if request.path == &#39;\/users\/123&#39;:\n user = self.db.query(f&#34;SELECT * FROM users WHERE id = 123&#34;)\n return json.dumps(user)\n\n# Setup\nservers = [WebServer(db_connection), WebServer(db_connection)]\nlb = LoadBalancer(servers)<\/code><\/pre><h3 class=\"heading-element\" id=\"253-stage-3-add-caching-layer\"><span>25.3 Stage 3: Add Caching Layer<\/span>\n <a href=\"#253-stage-3-add-caching-layer\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/cache-levels-architecture.webp\" title=\"Cache Levels in System \u2014 Comprehensive caching strategy: browser, CDN, application, and database caching\" data-thumbnail=\"\/files\/cache-levels-architecture.webp\" data-sub-html=\"<h2>Multi-level caching architecture from browser to database \u2014 optimization strategy showing cache hierarchy and performance gains<\/h2><p>Cache Levels in System \u2014 Comprehensive caching strategy: browser, CDN, application, and database caching<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/cache-levels-architecture.webp' alt=\"Multi-level caching architecture from browser to database \u2014 optimization strategy showing cache hierarchy and performance gains\"><\/a><figcaption class=\"image-caption\">Cache Levels in System \u2014 Comprehensive caching strategy: browser, CDN, application, and database caching<\/figcaption>\n <\/figure><\/p>\n<p><strong>Benefits:<\/strong><\/p>\n<ul>\n<li>Reduce database load<\/li>\n<li>Faster response times<\/li>\n<li>Better user experience<\/li>\n<\/ul>\n<p><strong>Implementation:<\/strong><\/p>\n<pre><code>from functools import wraps\nimport time\n\nclass CachingWebServer:\n def __init__(self, db, cache):\n self.db = db\n self.cache = cache\n\n def get_user(self, user_id):\n # Check cache first\n cache_key = f&#34;user:{user_id}&#34;\n cached_user = self.cache.get(cache_key)\n\n if cached_user:\n return cached_user\n\n # Cache miss - query database\n user = self.db.query(f&#34;SELECT * FROM users WHERE id = {user_id}&#34;)\n\n # Store in cache for 1 hour\n self.cache.set(cache_key, user, ttl=3600)\n\n return user<\/code><\/pre><h3 class=\"heading-element\" id=\"254-stage-4-add-cdn-for-static-content\"><span>25.4 Stage 4: Add CDN for Static Content<\/span>\n <a href=\"#254-stage-4-add-cdn-for-static-content\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/cdn-architecture.webp\" title=\"CDN Global Distribution \u2014 How content delivery networks reduce latency and improve performance worldwide\" data-thumbnail=\"\/files\/cdn-architecture.webp\" data-sub-html=\"<h2>CDN architecture with edge servers distributed globally \u2014 content delivery network for fast, low-latency content distribution<\/h2><p>CDN Global Distribution \u2014 How content delivery networks reduce latency and improve performance worldwide<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/cdn-architecture.webp' alt=\"CDN architecture with edge servers distributed globally \u2014 content delivery network for fast, low-latency content distribution\"><\/a><figcaption class=\"image-caption\">CDN Global Distribution \u2014 How content delivery networks reduce latency and improve performance worldwide<\/figcaption>\n <\/figure><\/p>\n<p><strong>CDN Benefits:<\/strong><\/p>\n<ul>\n<li>Serve content from location closest to user<\/li>\n<li>Reduce origin server load<\/li>\n<li>Faster page loads globally<\/li>\n<li>Lower bandwidth costs<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"255-stage-5-database-replication\"><span>25.5 Stage 5: Database Replication<\/span>\n <a href=\"#255-stage-5-database-replication\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/database-replication-master-slave.webp\" title=\"Database Replication \u2014 Master-slave and multi-master replication strategies for availability and performance\" data-thumbnail=\"\/files\/database-replication-master-slave.webp\" data-sub-html=\"<h2>Master-slave database replication architecture \u2014 how to replicate data for high availability and read scalability<\/h2><p>Database Replication \u2014 Master-slave and multi-master replication strategies for availability and performance<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/database-replication-master-slave.webp' alt=\"Master-slave database replication architecture \u2014 how to replicate data for high availability and read scalability\"><\/a><figcaption class=\"image-caption\">Database Replication \u2014 Master-slave and multi-master replication strategies for availability and performance<\/figcaption>\n <\/figure><\/p>\n<p><strong>Benefits:<\/strong><\/p>\n<ul>\n<li>Master for writes<\/li>\n<li>Slaves for reads<\/li>\n<li>Better read performance<\/li>\n<li>High availability<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"256-stage-6-database-sharding\"><span>25.6 Stage 6: Database Sharding<\/span>\n <a href=\"#256-stage-6-database-sharding\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/database-sharding-architecture.webp\" title=\"Database Sharding \u2014 Horizontal partitioning strategies for scaling large databases; range, hash, geo, and directory-based sharding\" data-thumbnail=\"\/files\/database-sharding-architecture.webp\" data-sub-html=\"<h2>Database sharding and horizontal partitioning architecture \u2014 distributing data across multiple databases for massive scale<\/h2><p>Database Sharding \u2014 Horizontal partitioning strategies for scaling large databases; range, hash, geo, and directory-based sharding<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/database-sharding-architecture.webp' alt=\"Database sharding and horizontal partitioning architecture \u2014 distributing data across multiple databases for massive scale\"><\/a><figcaption class=\"image-caption\">Database Sharding \u2014 Horizontal partitioning strategies for scaling large databases; range, hash, geo, and directory-based sharding<\/figcaption>\n <\/figure><\/p>\n<p><strong>When to shard:<\/strong><\/p>\n<ul>\n<li>Database size &gt; 1TB<\/li>\n<li>Single master at capacity<\/li>\n<li>Need to scale beyond vertical limits<\/li>\n<\/ul>\n<p><strong>Sharding key considerations:<\/strong><\/p>\n<ul>\n<li>Even distribution<\/li>\n<li>Immutable after shard creation<\/li>\n<li>Avoid hot keys<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"257-stage-7-message-queue-for-async-processing\"><span>25.7 Stage 7: Message Queue for Async Processing<\/span>\n <a href=\"#257-stage-7-message-queue-for-async-processing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/message-queue-architecture.webp\" title=\"Message Queue System \u2014 Async communication pattern using Kafka, RabbitMQ for system scalability and resilience\" data-thumbnail=\"\/files\/message-queue-architecture.webp\" data-sub-html=\"<h2>Message queue architecture with producers and consumers \u2014 asynchronous processing for decoupled system components<\/h2><p>Message Queue System \u2014 Async communication pattern using Kafka, RabbitMQ for system scalability and resilience<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/message-queue-architecture.webp' alt=\"Message queue architecture with producers and consumers \u2014 asynchronous processing for decoupled system components\"><\/a><figcaption class=\"image-caption\">Message Queue System \u2014 Async communication pattern using Kafka, RabbitMQ for system scalability and resilience<\/figcaption>\n <\/figure><\/p>\n<p><strong>Benefits:<\/strong><\/p>\n<ul>\n<li>Non-blocking operations<\/li>\n<li>Decouple producers and consumers<\/li>\n<li>Scale workers independently<\/li>\n<li>Handle traffic spikes<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"19-unique-id-generation-in-distributed-systems\"><span>19. Unique ID Generation in Distributed Systems<\/span>\n <a href=\"#19-unique-id-generation-in-distributed-systems\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Generating globally unique IDs is critical for distributed systems. IDs are used for order numbers, message IDs, user IDs, and any entity that must be uniquely addressable across all nodes without coordination.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/unique-id-generation-distributed.webp\" title=\"Distributed ID Generation \u2014 Snowflake ID, UUID, and timestamp-based strategies for generating unique identifiers at scale\" data-thumbnail=\"\/files\/unique-id-generation-distributed.webp\" data-sub-html=\"<h2>Unique ID generation in distributed systems \u2014 Snowflake algorithm, UUID, and distributed sequence strategies<\/h2><p>Distributed ID Generation \u2014 Snowflake ID, UUID, and timestamp-based strategies for generating unique identifiers at scale<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/unique-id-generation-distributed.webp' alt=\"Unique ID generation in distributed systems \u2014 Snowflake algorithm, UUID, and distributed sequence strategies\"><\/a><figcaption class=\"image-caption\">Distributed ID Generation \u2014 Snowflake ID, UUID, and timestamp-based strategies for generating unique identifiers at scale<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"191-requirements\"><span>19.1 Requirements<\/span>\n <a href=\"#191-requirements\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Unique IDs must be:<\/p>\n<ul>\n<li>Unique across all servers<\/li>\n<li>Sortable (roughly ordered by time)<\/li>\n<li>Decentralized (no single point of failure)<\/li>\n<li>Fast generation<\/li>\n<li>Fit in 64 bits (for efficiency)<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"192-snowflake-id-algorithm\"><span>19.2 Snowflake ID Algorithm<\/span>\n <a href=\"#192-snowflake-id-algorithm\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Twitter&rsquo;s Snowflake algorithm is industry standard:<\/p>\n<pre><code>| 1 | 41 | 10 | 12 |\n|Unused |Timestamp |Machine ID |Sequence|\n|1 bit |ms since |Data center|Counter |\n| |epoch |&#43; Worker | |<\/code><\/pre><p><strong>Structure (64 bits):<\/strong><\/p>\n<ul>\n<li>Bit 0: Unused (ensures positive numbers)<\/li>\n<li>Bits 1-41: Timestamp (42 bits, milliseconds since epoch)<\/li>\n<li>Bits 42-51: Machine ID (10 bits, 1024 machines)<\/li>\n<li>Bits 52-63: Sequence number (12 bits, 4096 IDs per ms per machine)<\/li>\n<\/ul>\n<p><strong>Capacity:<\/strong><\/p>\n<ul>\n<li>2^41 timestamps \u2248 69 years<\/li>\n<li>2^10 machines = 1,024 machines<\/li>\n<li>2^12 sequences = 4,096 IDs per millisecond per machine<\/li>\n<li>Total: ~4 million IDs per second per machine<\/li>\n<\/ul>\n<p><strong>Implementation:<\/strong><\/p>\n<pre><code>import time\nimport threading\n\nclass SnowflakeIDGenerator:\n EPOCH = 1288834974657 # Twitter Epoch\n WORKER_ID_BITS = 10\n DATACENTER_ID_BITS = 5\n SEQUENCE_BITS = 12\n\n MAX_WORKER_ID = (1 &lt;&lt; WORKER_ID_BITS) - 1\n MAX_DATACENTER_ID = (1 &lt;&lt; DATACENTER_ID_BITS) - 1\n MAX_SEQUENCE = (1 &lt;&lt; SEQUENCE_BITS) - 1\n\n WORKER_ID_SHIFT = SEQUENCE_BITS\n DATACENTER_ID_SHIFT = SEQUENCE_BITS &#43; WORKER_ID_BITS\n TIMESTAMP_SHIFT = SEQUENCE_BITS &#43; WORKER_ID_BITS &#43; DATACENTER_ID_BITS\n\n def __init__(self, worker_id, datacenter_id):\n if worker_id &gt; self.MAX_WORKER_ID or worker_id &lt; 0:\n raise ValueError(f&#34;Worker ID must be &lt;= {self.MAX_WORKER_ID}&#34;)\n if datacenter_id &gt; self.MAX_DATACENTER_ID or datacenter_id &lt; 0:\n raise ValueError(f&#34;Datacenter ID must be &lt;= {self.MAX_DATACENTER_ID}&#34;)\n\n self.worker_id = worker_id\n self.datacenter_id = datacenter_id\n self.sequence = 0\n self.last_timestamp = -1\n self.lock = threading.Lock()\n\n def _current_timestamp(self):\n return int(time.time() * 1000)\n\n def generate_id(self):\n with self.lock:\n timestamp = self._current_timestamp()\n\n if timestamp &lt; self.last_timestamp:\n raise Exception(&#34;Clock moved backwards&#34;)\n\n if timestamp == self.last_timestamp:\n self.sequence &#43;= 1\n if self.sequence &gt; self.MAX_SEQUENCE:\n # Wait for next millisecond\n timestamp = self._current_timestamp()\n self.sequence = 0\n else:\n self.sequence = 0\n\n self.last_timestamp = timestamp\n\n id = (\n (timestamp - self.EPOCH) &lt;&lt; self.TIMESTAMP_SHIFT |\n self.datacenter_id &lt;&lt; self.DATACENTER_ID_SHIFT |\n self.worker_id &lt;&lt; self.WORKER_ID_SHIFT |\n self.sequence\n )\n\n return id\n\n# Usage\nid_gen = SnowflakeIDGenerator(worker_id=1, datacenter_id=1)\nfor _ in range(5):\n print(id_gen.generate_id())<\/code><\/pre><h3 class=\"heading-element\" id=\"193-other-id-generation-methods\"><span>19.3 Other ID Generation Methods<\/span>\n <a href=\"#193-other-id-generation-methods\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>UUID:<\/strong><\/p>\n<ul>\n<li>128 bits<\/li>\n<li>Guaranteed unique<\/li>\n<li>Not sortable<\/li>\n<li>Larger ID size<\/li>\n<\/ul>\n<p><strong>Database Auto-Increment:<\/strong><\/p>\n<ul>\n<li>Simple<\/li>\n<li>Requires network call<\/li>\n<li>Single point of failure potential<\/li>\n<li>Exposes data volume<\/li>\n<\/ul>\n<p><strong>Timestamp + Random:<\/strong><\/p>\n<ul>\n<li>Simple<\/li>\n<li>Sortable<\/li>\n<li>Risk of collision with multiple generators<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"20-distributed-transactions-saga-pattern-and-cqrs\"><span>20. Distributed Transactions, Saga Pattern, and CQRS<\/span>\n <a href=\"#20-distributed-transactions-saga-pattern-and-cqrs\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>When Things Span Multiple Services<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">In microservices, a single business operation (like placing an order) often touches multiple services and databases. Since each service owns its own database, you cannot use a standard database transaction across them. This section covers patterns for maintaining data consistency across service boundaries.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"201-the-problem-distributed-transactions\"><span>20.1 The Problem: Distributed Transactions<\/span>\n <a href=\"#201-the-problem-distributed-transactions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Classic ACID transactions work within a single database. Once you split data across services, you lose that guarantee:<\/p>\n<pre><code>Monolith:\nBEGIN TRANSACTION\n UPDATE inventory SET stock = stock - 1 WHERE product_id = 123\n INSERT INTO orders VALUES (...)\n UPDATE accounts SET balance = balance - 99.99\nCOMMIT \u2190 all or nothing\n\nMicroservices:\nOrderService.createOrder() \u2192 writes to Order DB\nInventoryService.reserve() \u2192 writes to Inventory DB\nPaymentService.charge() \u2192 writes to Payment DB\n\u2190 NO global COMMIT, partial failures are real<\/code><\/pre><h3 class=\"heading-element\" id=\"202-two-phase-commit-2pc\"><span>20.2 Two-Phase Commit (2PC)<\/span>\n <a href=\"#202-two-phase-commit-2pc\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The traditional distributed transaction protocol. Rarely used in modern microservices due to performance and operational complexity.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/distributed-transaction-2pc.webp\" title=\"Two-Phase Commit (2PC) \u2014 Distributed transaction coordination across multiple services with prepare and commit phases\" data-thumbnail=\"\/files\/distributed-transaction-2pc.webp\" data-sub-html=\"<h2>Distributed transaction 2PC coordination protocol \u2014 two-phase commit for ACID guarantees across multiple databases<\/h2><p>Two-Phase Commit (2PC) \u2014 Distributed transaction coordination across multiple services with prepare and commit phases<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/distributed-transaction-2pc.webp' alt=\"Distributed transaction 2PC coordination protocol \u2014 two-phase commit for ACID guarantees across multiple databases\"><\/a><figcaption class=\"image-caption\">Two-Phase Commit (2PC) \u2014 Distributed transaction coordination across multiple services with prepare and commit phases<\/figcaption>\n <\/figure><\/p>\n<p><strong>How 2PC works:<\/strong><\/p>\n<p><strong>Phase 1 \u2014 Prepare:<\/strong><\/p>\n<ul>\n<li>Coordinator sends <code>PREPARE<\/code> to all participants<\/li>\n<li>Each participant locks resources and writes to WAL, responds <code>VOTE_COMMIT<\/code> or <code>VOTE_ABORT<\/code><\/li>\n<\/ul>\n<p><strong>Phase 2 \u2014 Commit\/Abort:<\/strong><\/p>\n<ul>\n<li>If all voted <code>COMMIT<\/code> \u2192 coordinator sends <code>COMMIT<\/code> to all<\/li>\n<li>If any voted <code>ABORT<\/code> \u2192 coordinator sends <code>ROLLBACK<\/code> to all<\/li>\n<\/ul>\n<pre><code>class TwoPhaseCommitCoordinator:\n def __init__(self, participants):\n self.participants = participants\n\n def execute_transaction(self, operations):\n &#34;&#34;&#34;Execute distributed transaction across all participants.&#34;&#34;&#34;\n # Phase 1: Prepare\n votes = []\n for participant, operation in zip(self.participants, operations):\n try:\n vote = participant.prepare(operation)\n votes.append(vote)\n except Exception:\n votes.append(&#39;ABORT&#39;)\n\n # Phase 2: Commit or Rollback\n if all(v == &#39;COMMIT&#39; for v in votes):\n for participant in self.participants:\n participant.commit()\n return True\n else:\n for participant in self.participants:\n participant.rollback()\n return False<\/code><\/pre><p><strong>Problems with 2PC:<\/strong><\/p>\n<ul>\n<li><strong>Blocking<\/strong>: If coordinator crashes after Phase 1, participants hold locks indefinitely<\/li>\n<li><strong>Latency<\/strong>: Synchronous round-trips across services add 20\u201350% overhead<\/li>\n<li><strong>Availability<\/strong>: System is unavailable during coordinator failure (CP, not AP)<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"203-saga-pattern\"><span>20.3 Saga Pattern<\/span>\n <a href=\"#203-saga-pattern\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The preferred approach for distributed transactions in microservices. Instead of one atomic transaction, a Saga is a <strong>sequence of local transactions<\/strong> where each step publishes an event that triggers the next. If any step fails, <strong>compensating transactions<\/strong> undo previous steps.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/saga-pattern-distributed.webp\" title=\"Saga Pattern \u2014 Distributed transaction management through compensating transactions in microservices architectures\" data-thumbnail=\"\/files\/saga-pattern-distributed.webp\" data-sub-html=\"<h2>Saga pattern for distributed transactions \u2014 choreography and orchestration approaches for microservices consistency<\/h2><p>Saga Pattern \u2014 Distributed transaction management through compensating transactions in microservices architectures<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/saga-pattern-distributed.webp' alt=\"Saga pattern for distributed transactions \u2014 choreography and orchestration approaches for microservices consistency\"><\/a><figcaption class=\"image-caption\">Saga Pattern \u2014 Distributed transaction management through compensating transactions in microservices architectures<\/figcaption>\n <\/figure><\/p>\n<p><strong>Two Saga Implementation Styles:<\/strong><\/p>\n<p><strong>1. Choreography (Event-driven)<\/strong>\nEach service reacts to events and triggers the next service. No central coordinator.<\/p>\n<pre><code># Order Service - publishes event\nclass OrderService:\n def create_order(self, user_id, items, total):\n order = self.db.create_order(user_id, items, total, status=&#39;PENDING&#39;)\n # Publish event - Inventory Service will react\n self.kafka.publish(&#39;order.created&#39;, {\n &#39;order_id&#39;: order.id, &#39;items&#39;: items, &#39;total&#39;: total\n })\n return order\n\n# Inventory Service - reacts to order.created\nclass InventoryService:\n def on_order_created(self, event):\n try:\n self.reserve_items(event[&#39;items&#39;])\n self.kafka.publish(&#39;inventory.reserved&#39;, {&#39;order_id&#39;: event[&#39;order_id&#39;]})\n except InsufficientStockError:\n # Trigger compensation\n self.kafka.publish(&#39;inventory.failed&#39;, {&#39;order_id&#39;: event[&#39;order_id&#39;]})\n\n# Order Service - compensates on failure\nclass OrderService:\n def on_inventory_failed(self, event):\n self.db.update_order(event[&#39;order_id&#39;], status=&#39;CANCELLED&#39;)\n self.kafka.publish(&#39;order.cancelled&#39;, {&#39;order_id&#39;: event[&#39;order_id&#39;]})<\/code><\/pre><p><strong>2. Orchestration (Central coordinator)<\/strong>\nA Saga orchestrator explicitly calls each service and handles failures.<\/p>\n<pre><code>class OrderSagaOrchestrator:\n def execute(self, order_id, user_id, items, total):\n steps_completed = []\n\n try:\n # Step 1: Reserve inventory\n self.inventory_service.reserve(items)\n steps_completed.append(&#39;inventory&#39;)\n\n # Step 2: Process payment\n self.payment_service.charge(user_id, total)\n steps_completed.append(&#39;payment&#39;)\n\n # Step 3: Confirm order\n self.order_service.confirm(order_id)\n steps_completed.append(&#39;order&#39;)\n\n return {&#39;status&#39;: &#39;success&#39;, &#39;order_id&#39;: order_id}\n\n except Exception as e:\n # Compensate in reverse order\n self._compensate(steps_completed, order_id, items, user_id, total)\n return {&#39;status&#39;: &#39;failed&#39;, &#39;reason&#39;: str(e)}\n\n def _compensate(self, steps, order_id, items, user_id, total):\n if &#39;payment&#39; in steps:\n self.payment_service.refund(user_id, total)\n if &#39;inventory&#39; in steps:\n self.inventory_service.release(items)\n if &#39;order&#39; in steps:\n self.order_service.cancel(order_id)<\/code><\/pre><p><strong>Choreography vs Orchestration:<\/strong><\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th><\/th>\n <th>Choreography<\/th>\n <th>Orchestration<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>Coordination<\/td>\n <td>Decentralized (events)<\/td>\n <td>Centralized (orchestrator)<\/td>\n <\/tr>\n <tr>\n <td>Coupling<\/td>\n <td>Loose<\/td>\n <td>Tighter (orchestrator knows all services)<\/td>\n <\/tr>\n <tr>\n <td>Visibility<\/td>\n <td>Hard to trace flow<\/td>\n <td>Easy to see whole saga in one place<\/td>\n <\/tr>\n <tr>\n <td>Failure handling<\/td>\n <td>Each service handles its own<\/td>\n <td>Orchestrator manages compensation<\/td>\n <\/tr>\n <tr>\n <td>Best for<\/td>\n <td>Simple flows, few steps<\/td>\n <td>Complex flows, many steps<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"204-cqrs-command-query-responsibility-segregation\"><span>20.4 CQRS (Command Query Responsibility Segregation)<\/span>\n <a href=\"#204-cqrs-command-query-responsibility-segregation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Separate the <strong>write model<\/strong> (Commands) from the <strong>read model<\/strong> (Queries). This enables independent scaling, optimized read\/write schemas, and eventual consistency.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/event-sourcing-cqrs.webp\" title=\"CQRS and Event Sourcing \u2014 Command\/query separation and event-driven state reconstruction for scalable distributed architectures\" data-thumbnail=\"\/files\/event-sourcing-cqrs.webp\" data-sub-html=\"<h2>Event sourcing and CQRS architecture \u2014 separating read and write models for scalable distributed systems<\/h2><p>CQRS and Event Sourcing \u2014 Command\/query separation and event-driven state reconstruction for scalable distributed architectures<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/event-sourcing-cqrs.webp' alt=\"Event sourcing and CQRS architecture \u2014 separating read and write models for scalable distributed systems\"><\/a><figcaption class=\"image-caption\">CQRS and Event Sourcing \u2014 Command\/query separation and event-driven state reconstruction for scalable distributed architectures<\/figcaption>\n <\/figure><\/p>\n<p><strong>The Core Idea:<\/strong><\/p>\n<pre><code>Traditional: CQRS:\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\nSingle Model Write Model (Commands) Read Model (Queries)\n \u2502 \u2502 \u2502\n \u251c\u2500\u2500 GET users \u2500\u2500\u25ba Commands DB \u2500\u2500\u25ba Read DB\n \u2514\u2500\u2500 POST users \u2502 (normalized) \u2502 (denormalized,\n \u2514\u2500\u2500 publish events \u2500\u2500\u25ba optimized views)<\/code><\/pre><pre><code># Command side - handles writes\nclass CreateOrderCommand:\n def __init__(self, user_id, items, total):\n self.user_id = user_id\n self.items = items\n self.total = total\n\nclass OrderCommandHandler:\n def handle_create(self, command: CreateOrderCommand):\n order = self.orders_db.insert({\n &#39;user_id&#39;: command.user_id,\n &#39;items&#39;: command.items,\n &#39;total&#39;: command.total,\n &#39;status&#39;: &#39;pending&#39;\n })\n # Publish event to update read models\n self.event_bus.publish(&#39;OrderCreated&#39;, order)\n return order.id\n\n# Query side - handles reads (separate optimized DB\/view)\nclass OrderQueryHandler:\n def get_user_orders(self, user_id, limit=10):\n # Read from denormalized read model (pre-joined, pre-aggregated)\n return self.read_db.query(\n &#34;SELECT * FROM user_orders_view WHERE user_id = ? ORDER BY created_at DESC LIMIT ?&#34;,\n [user_id, limit]\n )<\/code><\/pre><h3 class=\"heading-element\" id=\"205-event-sourcing\"><span>20.5 Event Sourcing<\/span>\n <a href=\"#205-event-sourcing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Instead of storing the <strong>current state<\/strong>, store the <strong>sequence of events<\/strong> that led to it. State is reconstructed by replaying events.<\/p>\n<pre><code># Event store: append-only log of all events\nclass EventStore:\n def append(self, aggregate_id, event):\n &#34;&#34;&#34;Append event to the store.&#34;&#34;&#34;\n self.store.append({\n &#39;aggregate_id&#39;: aggregate_id,\n &#39;event_type&#39;: event[&#39;type&#39;],\n &#39;data&#39;: event[&#39;data&#39;],\n &#39;timestamp&#39;: time.time(),\n &#39;version&#39;: self.get_next_version(aggregate_id)\n })\n\n def get_events(self, aggregate_id):\n &#34;&#34;&#34;Retrieve all events for an aggregate.&#34;&#34;&#34;\n return [e for e in self.store if e[&#39;aggregate_id&#39;] == aggregate_id]\n\n# Reconstruct current state by replaying events\nclass BankAccount:\n def __init__(self, account_id, event_store):\n self.account_id = account_id\n self.balance = 0\n self._replay(event_store.get_events(account_id))\n\n def _replay(self, events):\n for event in events:\n if event[&#39;event_type&#39;] == &#39;MoneyDeposited&#39;:\n self.balance &#43;= event[&#39;data&#39;][&#39;amount&#39;]\n elif event[&#39;event_type&#39;] == &#39;MoneyWithdrawn&#39;:\n self.balance -= event[&#39;data&#39;][&#39;amount&#39;]\n\n def deposit(self, amount, event_store):\n event_store.append(self.account_id, {\n &#39;type&#39;: &#39;MoneyDeposited&#39;,\n &#39;data&#39;: {&#39;amount&#39;: amount}\n })\n self.balance &#43;= amount\n\n# Benefits:\n# - Full audit trail (every change recorded)\n# - Time travel: reconstruct state at any point in time\n# - Event replay for debugging and analytics\n# - Natural fit with CQRS (events update read models)<\/code><\/pre><p><strong>When to use Event Sourcing:<\/strong><\/p>\n<ul>\n<li>Financial systems (audit trail required)<\/li>\n<li>Systems needing time-travel queries<\/li>\n<li>Complex domain models with many state transitions<\/li>\n<li>Systems that need to rebuild read models from scratch<\/li>\n<\/ul>\n<div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Event Sourcing Complexity<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">Event sourcing adds significant complexity. Only use it when the benefits (audit trail, time travel, CQRS compatibility) justify the cost. Most CRUD applications do not need event sourcing.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 class=\"heading-element\" id=\"26-design-a-key-value-store\"><span>26. Design a Key-Value Store<\/span>\n <a href=\"#26-design-a-key-value-store\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"261-requirements\"><span>26.1 Requirements<\/span>\n <a href=\"#261-requirements\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Functional:<\/strong><\/p>\n<ul>\n<li>GET(key): retrieve value<\/li>\n<li>PUT(key, value): store value<\/li>\n<li>DELETE(key): remove value<\/li>\n<\/ul>\n<p><strong>Non-Functional:<\/strong><\/p>\n<ul>\n<li>Single node (vertical scalability only)<\/li>\n<li>O(1) time complexity for all operations<\/li>\n<li>Persistent storage<\/li>\n<li>Efficient memory usage<\/li>\n<li>Hardware-aware optimization<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"262-architecture\"><span>26.2 Architecture<\/span>\n <a href=\"#262-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/key-value-store-architecture.webp\" title=\"Key-Value Store Architecture \u2014 Design patterns for distributed key-value systems and NoSQL storage optimization\" data-thumbnail=\"\/files\/key-value-store-architecture.webp\" data-sub-html=\"<h2>Key-value store distributed architecture design \u2014 fast data access pattern for caching, sessions, and real-time features<\/h2><p>Key-Value Store Architecture \u2014 Design patterns for distributed key-value systems and NoSQL storage optimization<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/key-value-store-architecture.webp' alt=\"Key-value store distributed architecture design \u2014 fast data access pattern for caching, sessions, and real-time features\"><\/a><figcaption class=\"image-caption\">Key-Value Store Architecture \u2014 Design patterns for distributed key-value systems and NoSQL storage optimization<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"263-implementation\"><span>26.3 Implementation<\/span>\n <a href=\"#263-implementation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>import os\nimport json\nimport time\nfrom threading import Lock\n\nclass SimpleKVStore:\n def __init__(self, data_dir=&#34;.\/data&#34;):\n self.data_dir = data_dir\n self.memory_index = {} # In-memory hash map\n self.wal_file = os.path.join(data_dir, &#34;wal.log&#34;) # Write-ahead log\n self.data_file = os.path.join(data_dir, &#34;data.db&#34;) # Actual data\n self.lock = Lock()\n \n os.makedirs(data_dir, exist_ok=True)\n self._load_from_disk()\n \n def _load_from_disk(self):\n &#34;&#34;&#34;Load index from disk on startup&#34;&#34;&#34;\n if os.path.exists(self.wal_file):\n with open(self.wal_file, &#39;r&#39;) as f:\n for line in f:\n entry = json.loads(line.strip())\n if entry[&#39;op&#39;] == &#39;PUT&#39;:\n self.memory_index[entry[&#39;key&#39;]] = entry[&#39;value&#39;]\n elif entry[&#39;op&#39;] == &#39;DELETE&#39;:\n self.memory_index.pop(entry[&#39;key&#39;], None)\n \n def get(self, key):\n with self.lock:\n if key not in self.memory_index:\n return None\n return self.memory_index[key]\n \n def put(self, key, value):\n with self.lock:\n # Write-ahead log (WAL)\n entry = {&#39;op&#39;: &#39;PUT&#39;, &#39;key&#39;: key, &#39;value&#39;: value, &#39;timestamp&#39;: time.time()}\n with open(self.wal_file, &#39;a&#39;) as f:\n f.write(json.dumps(entry) &#43; &#39;\\n&#39;)\n \n # Update in-memory index\n self.memory_index[key] = value\n \n # Periodic flush to disk (simplified)\n if len(self.memory_index) % 1000 == 0:\n self._flush_to_disk()\n \n def delete(self, key):\n with self.lock:\n if key not in self.memory_index:\n return False\n \n # Write deletion to WAL\n entry = {&#39;op&#39;: &#39;DELETE&#39;, &#39;key&#39;: key, &#39;timestamp&#39;: time.time()}\n with open(self.wal_file, &#39;a&#39;) as f:\n f.write(json.dumps(entry) &#43; &#39;\\n&#39;)\n \n # Remove from memory\n del self.memory_index[key]\n return True\n \n def _flush_to_disk(self):\n &#34;&#34;&#34;Periodically flush in-memory index to disk&#34;&#34;&#34;\n with open(self.data_file, &#39;w&#39;) as f:\n for key, value in self.memory_index.items():\n f.write(f&#34;{key}={json.dumps(value)}\\n&#34;)\n\n# Usage\nstore = SimpleKVStore()\nstore.put(&#34;user:1&#34;, {&#34;name&#34;: &#34;Alice&#34;, &#34;age&#34;: 30})\nstore.put(&#34;user:2&#34;, {&#34;name&#34;: &#34;Bob&#34;, &#34;age&#34;: 25})\n\nprint(store.get(&#34;user:1&#34;)) # {&#34;name&#34;: &#34;Alice&#34;, &#34;age&#34;: 30}\n\nstore.delete(&#34;user:2&#34;)\nprint(store.get(&#34;user:2&#34;)) # None<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"28-design-twitter-with-microservices\"><span>28. Design Twitter with Microservices<\/span>\n <a href=\"#28-design-twitter-with-microservices\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"281-high-level-architecture\"><span>28.1 High-Level Architecture<\/span>\n <a href=\"#281-high-level-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/twitter-microservices-architecture.webp\" title=\"Twitter Microservices Design \u2014 Real-time social media system architecture for billions of users and interactions\" data-thumbnail=\"\/files\/twitter-microservices-architecture.webp\" data-sub-html=\"<h2>Twitter-like social media system microservices architecture \u2014 distributed service design for massive-scale social platform<\/h2><p>Twitter Microservices Design \u2014 Real-time social media system architecture for billions of users and interactions<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/twitter-microservices-architecture.webp' alt=\"Twitter-like social media system microservices architecture \u2014 distributed service design for massive-scale social platform\"><\/a><figcaption class=\"image-caption\">Twitter Microservices Design \u2014 Real-time social media system architecture for billions of users and interactions<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"282-core-services-explained\"><span>28.2 Core Services Explained<\/span>\n <a href=\"#282-core-services-explained\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Tweet Service:<\/strong><\/p>\n<ul>\n<li>Create, read, update, delete tweets<\/li>\n<li>Validate tweet content<\/li>\n<li>Handle media uploads<\/li>\n<li>Publish tweet events<\/li>\n<\/ul>\n<p><strong>Feed Service:<\/strong><\/p>\n<ul>\n<li>Generate personalized feed<\/li>\n<li>Combine tweets from followed accounts<\/li>\n<li>Sort chronologically<\/li>\n<li>Cache frequently accessed feeds<\/li>\n<\/ul>\n<p><strong>Search Service:<\/strong><\/p>\n<ul>\n<li>Full-text search on tweets<\/li>\n<li>Handle hashtags<\/li>\n<li>Trending topics<\/li>\n<li>Use Elasticsearch or similar<\/li>\n<\/ul>\n<p><strong>Follow Service:<\/strong><\/p>\n<ul>\n<li>Manage follower\/following relationships<\/li>\n<li>Social graph<\/li>\n<li>Notification for new followers<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"283-handling-the-thundering-herd\"><span>28.3 Handling the &ldquo;Thundering Herd&rdquo;<\/span>\n <a href=\"#283-handling-the-thundering-herd\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Challenge: When celebrity tweets, millions access simultaneously.<\/p>\n<p><strong>Solution: Fan-Out Cache<\/strong><\/p>\n<pre><code># When a celebrity tweets\ndef create_celebrity_tweet(user_id, content):\n tweet = TweetService.create(user_id, content)\n \n # Get all followers\n followers = FollowService.get_followers(user_id)\n \n # Pre-populate feeds for active followers\n for follower_id in followers[:100000]: # Top 100K followers\n feed_key = f&#34;feed:{follower_id}&#34;\n cache.prepend(feed_key, tweet) # Add to front of feed\n \n # Enqueue notification\n notification_queue.enqueue({\n &#39;user_id&#39;: follower_id,\n &#39;event&#39;: &#39;new_tweet&#39;,\n &#39;from_user&#39;: user_id\n })<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"29-design-google-maps--location-based-services\"><span>29. Design Google Maps \/ Location-Based Services<\/span>\n <a href=\"#29-design-google-maps--location-based-services\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"291-architecture-overview\"><span>29.1 Architecture Overview<\/span>\n <a href=\"#291-architecture-overview\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/location-based-services-architecture.webp\" title=\"Location-Based Services \u2014 Geospatial database design and efficient proximity search implementation\" data-thumbnail=\"\/files\/location-based-services-architecture.webp\" data-sub-html=\"<h2>Location-based services and Google Maps system architecture \u2014 geospatial queries and proximity search at scale<\/h2><p>Location-Based Services \u2014 Geospatial database design and efficient proximity search implementation<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/location-based-services-architecture.webp' alt=\"Location-based services and Google Maps system architecture \u2014 geospatial queries and proximity search at scale\"><\/a><figcaption class=\"image-caption\">Location-Based Services \u2014 Geospatial database design and efficient proximity search implementation<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"292-geospatial-indexing\"><span>29.2 Geospatial Indexing<\/span>\n <a href=\"#292-geospatial-indexing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Storing location data efficiently:<\/p>\n<pre><code># Using Geohash for spatial indexing\ndef geohash_encode(latitude, longitude, precision=5):\n &#34;&#34;&#34;\n Geohash encodes lat\/long into a short string\n Nearby locations have similar geohashes\n &#34;&#34;&#34;\n # Simplified geohash (actual implementation is more complex)\n lat_bits = bin(int((latitude &#43; 90) \/ 180 * (2 ** 20)))[2:].zfill(20)\n lon_bits = bin(int((longitude &#43; 180) \/ 360 * (2 ** 20)))[2:].zfill(20)\n \n # Interleave bits\n interleaved = &#39;&#39;.join(lon_bits[i] &#43; lat_bits[i] for i in range(20))\n \n # Convert to base32\n base32 = &#34;0123456789bcdefghjkmnpqrstuvwxyz&#34;\n geohash = &#39;&#39;\n for i in range(0, len(interleaved), 5):\n chunk = interleaved[i:i&#43;5]\n geohash &#43;= base32[int(chunk, 2)]\n \n return geohash[:precision]\n\n# Nearby search using geohash\ndef find_nearby_places(latitude, longitude, radius_km=1):\n &#34;&#34;&#34;Find all places within radius&#34;&#34;&#34;\n center_geohash = geohash_encode(latitude, longitude, precision=7)\n \n # Get geohashes for surrounding cells\n nearby_geohashes = get_neighbors(center_geohash)\n \n places = []\n for geohash in nearby_geohashes:\n # Query database for places with this geohash prefix\n db_places = PlaceDB.query(f&#34;geohash LIKE &#39;{geohash}%&#39;&#34;)\n \n # Filter by distance\n for place in db_places:\n dist = haversine_distance(latitude, longitude, place.lat, place.lon)\n if dist &lt;= radius_km:\n places.append(place)\n \n return places\n\ndef haversine_distance(lat1, lon1, lat2, lon2):\n &#34;&#34;&#34;Calculate distance between two points in km&#34;&#34;&#34;\n import math\n R = 6371 # Earth radius in km\n \n dlat = math.radians(lat2 - lat1)\n dlon = math.radians(lon2 - lon1)\n \n a = math.sin(dlat\/2)**2 &#43; math.cos(math.radians(lat1)) * math.cos(math.radians(lat2)) * math.sin(dlon\/2)**2\n c = 2 * math.atan2(math.sqrt(a), math.sqrt(1-a))\n \n return R * c<\/code><\/pre><h3 class=\"heading-element\" id=\"293-direction-finding-shortest-path\"><span>29.3 Direction Finding (Shortest Path)<\/span>\n <a href=\"#293-direction-finding-shortest-path\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Simplified Dijkstra&#39;s algorithm for shortest path\nimport heapq\n\nclass DirectionFinder:\n def __init__(self, road_network):\n self.graph = road_network # Graph of intersections and roads\n \n def find_shortest_path(self, start_id, end_id):\n &#34;&#34;&#34;Find shortest path between two locations&#34;&#34;&#34;\n distances = {start_id: 0}\n previous = {}\n pq = [(0, start_id)] # (distance, node)\n visited = set()\n \n while pq:\n current_dist, current_node = heapq.heappop(pq)\n \n if current_node in visited:\n continue\n \n visited.add(current_node)\n \n # Check neighbors\n for neighbor, weight in self.graph[current_node]:\n if neighbor in visited:\n continue\n \n new_dist = current_dist &#43; weight\n \n if neighbor not in distances or new_dist &lt; distances[neighbor]:\n distances[neighbor] = new_dist\n previous[neighbor] = current_node\n heapq.heappush(pq, (new_dist, neighbor))\n \n # Reconstruct path\n path = []\n current = end_id\n while current in previous:\n path.append(current)\n current = previous[current]\n path.append(start_id)\n path.reverse()\n \n return path, distances.get(end_id, float(&#39;inf&#39;))<\/code><\/pre><h2 class=\"heading-element\" id=\"17-rate-limiting\"><span>17. Rate Limiting<\/span>\n <a href=\"#17-rate-limiting\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Rate limiting controls how many requests a client can make within a time window, protecting services from abuse, DDoS attacks, and runaway consumers.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/rate-limiting-algorithms.webp\" title=\"Rate Limiting Algorithms \u2014 Token bucket, leaky bucket, sliding window counter implementations for API protection\" data-thumbnail=\"\/files\/rate-limiting-algorithms.webp\" data-sub-html=\"<h2>Rate limiting algorithms comparison \u2014 token bucket, leaky bucket, fixed window, and sliding window counters<\/h2><p>Rate Limiting Algorithms \u2014 Token bucket, leaky bucket, sliding window counter implementations for API protection<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/rate-limiting-algorithms.webp' alt=\"Rate limiting algorithms comparison \u2014 token bucket, leaky bucket, fixed window, and sliding window counters\"><\/a><figcaption class=\"image-caption\">Rate Limiting Algorithms \u2014 Token bucket, leaky bucket, sliding window counter implementations for API protection<\/figcaption>\n <\/figure><\/p>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Where to Apply Rate Limiting<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><ul>\n<li><strong>API Gateway level<\/strong>: Protect all services uniformly before requests reach your application<\/li>\n<li><strong>Per user \/ IP<\/strong>: Prevent single bad actor from degrading service for everyone<\/li>\n<li><strong>Per endpoint<\/strong>: More aggressive limits on expensive operations (e.g. <code>\/search<\/code> vs <code>\/ping<\/code>)<\/li>\n<li><strong>Distributed rate limiting<\/strong>: Use Redis so limits apply across all API server instances<\/li>\n<\/ul>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"171-rate-limiting-algorithms\"><span>17.1 Rate Limiting Algorithms<\/span>\n <a href=\"#171-rate-limiting-algorithms\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>1. Token Bucket Algorithm<\/strong><\/p>\n<p>Pre-calculated tokens in a bucket, remove on each request:<\/p>\n<pre><code>import time\nfrom threading import Lock\n\nclass TokenBucket:\n def __init__(self, capacity, refill_rate):\n &#34;&#34;&#34;\n capacity: max tokens in bucket\n refill_rate: tokens added per second\n &#34;&#34;&#34;\n self.capacity = capacity\n self.refill_rate = refill_rate\n self.tokens = capacity\n self.last_refill = time.time()\n self.lock = Lock()\n \n def allow_request(self, tokens=1):\n with self.lock:\n self._refill()\n \n if self.tokens &gt;= tokens:\n self.tokens -= tokens\n return True\n return False\n \n def _refill(self):\n now = time.time()\n elapsed = now - self.last_refill\n self.tokens = min(\n self.capacity,\n self.tokens &#43; elapsed * self.refill_rate\n )\n self.last_refill = now\n\n# Usage\nbucket = TokenBucket(capacity=10, refill_rate=1) # 10 tokens, 1 per second\n\nfor i in range(15):\n if bucket.allow_request():\n print(f&#34;Request {i&#43;1}: Allowed&#34;)\n else:\n print(f&#34;Request {i&#43;1}: Rate limited&#34;)<\/code><\/pre><p><strong>2. Leaky Bucket Algorithm<\/strong><\/p>\n<p>Drain bucket at fixed rate:<\/p>\n<pre><code>from collections import deque\nimport time\nfrom threading import Lock\n\nclass LeakyBucket:\n def __init__(self, capacity, outflow_rate):\n &#34;&#34;&#34;\n capacity: max requests in queue\n outflow_rate: requests processed per second\n &#34;&#34;&#34;\n self.capacity = capacity\n self.outflow_rate = outflow_rate\n self.queue = deque()\n self.last_leak = time.time()\n self.lock = Lock()\n \n def allow_request(self):\n with self.lock:\n self._leak()\n \n if len(self.queue) &lt; self.capacity:\n self.queue.append(time.time())\n return True\n return False\n \n def _leak(self):\n now = time.time()\n elapsed = now - self.last_leak\n leak_count = int(elapsed * self.outflow_rate)\n \n for _ in range(leak_count):\n if self.queue:\n self.queue.popleft()\n \n self.last_leak = now<\/code><\/pre><p><strong>3. Sliding Window Counter<\/strong><\/p>\n<p>Hybrid approach combining fixed and sliding window:<\/p>\n<pre><code>import time\nfrom threading import Lock\n\nclass SlidingWindowCounter:\n def __init__(self, window_size, max_requests):\n &#34;&#34;&#34;\n window_size: time window in seconds\n max_requests: max requests allowed in window\n &#34;&#34;&#34;\n self.window_size = window_size\n self.max_requests = max_requests\n self.counters = {} # {window_start: count}\n self.lock = Lock()\n \n def allow_request(self, identifier):\n with self.lock:\n now = time.time()\n window_start = int(now \/ self.window_size) * self.window_size\n \n # Remove old windows\n for key in list(self.counters.keys()):\n if key &lt; window_start - self.window_size:\n del self.counters[key]\n \n # Check current window\n key = f&#34;{identifier}:{window_start}&#34;\n if key not in self.counters:\n self.counters[key] = 0\n \n if self.counters[key] &lt; self.max_requests:\n self.counters[key] &#43;= 1\n return True\n return False<\/code><\/pre><h3 class=\"heading-element\" id=\"172-distributed-rate-limiting\"><span>17.2 Distributed Rate Limiting<\/span>\n <a href=\"#172-distributed-rate-limiting\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>For multi-server deployments, use Redis:<\/p>\n<pre><code>import redis\nimport time\n\nclass DistributedRateLimiter:\n def __init__(self, redis_client, key_prefix=&#34;rate_limit&#34;):\n self.redis = redis_client\n self.key_prefix = key_prefix\n \n def allow_request(self, identifier, limit, window):\n &#34;&#34;&#34;\n identifier: user_id or IP\n limit: max requests\n window: time window in seconds\n &#34;&#34;&#34;\n key = f&#34;{self.key_prefix}:{identifier}&#34;\n now = time.time()\n window_start = now - window\n \n # Remove old entries\n self.redis.zremrangebyscore(key, 0, window_start)\n \n # Count requests in current window\n count = self.redis.zcard(key)\n \n if count &lt; limit:\n # Add current request\n self.redis.zadd(key, {str(now): now})\n # Set expiration\n self.redis.expire(key, window &#43; 1)\n return True\n \n return False\n\n# Usage\nredis_client = redis.Redis(host=&#39;localhost&#39;, port=6379)\nlimiter = DistributedRateLimiter(redis_client)\n\nfor i in range(5):\n if limiter.allow_request(&#39;user:123&#39;, limit=3, window=60):\n print(f&#34;Request {i&#43;1}: Allowed&#34;)\n else:\n print(f&#34;Request {i&#43;1}: Rate limited&#34;)<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"18-consistent-hashing\"><span>18. Consistent Hashing<\/span>\n <a href=\"#18-consistent-hashing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Consistent hashing solves the problem of routing requests to nodes in a dynamic cluster (cache servers, storage nodes) where nodes can be added or removed without remapping most keys.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/consistent-hashing-ring.webp\" title=\"Consistent Hashing Ring \u2014 Virtual nodes and hash ring design for distributed cache and storage systems\" data-thumbnail=\"\/files\/consistent-hashing-ring.webp\" data-sub-html=\"<h2>Consistent hashing ring with virtual nodes \u2014 minimal key remapping when cluster topology changes<\/h2><p>Consistent Hashing Ring \u2014 Virtual nodes and hash ring design for distributed cache and storage systems<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/consistent-hashing-ring.webp' alt=\"Consistent hashing ring with virtual nodes \u2014 minimal key remapping when cluster topology changes\"><\/a><figcaption class=\"image-caption\">Consistent Hashing Ring \u2014 Virtual nodes and hash ring design for distributed cache and storage systems<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"181-problem-statement\"><span>18.1 Problem Statement<\/span>\n <a href=\"#181-problem-statement\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Standard hashing breaks when servers are added\/removed:<\/p>\n<pre><code># Standard hashing - doesn&#39;t work well with dynamic servers\ndef standard_hash(key, num_servers):\n return hash(key) % num_servers<\/code><\/pre><p>When you add\/remove a server, most keys rehash to different locations!<\/p>\n<h3 class=\"heading-element\" id=\"182-consistent-hashing-solution\"><span>18.2 Consistent Hashing Solution<\/span>\n <a href=\"#182-consistent-hashing-solution\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Maps both keys and servers onto a circle:<\/p>\n<pre><code>import hashlib\nfrom bisect import bisect_right\n\nclass ConsistentHashing:\n def __init__(self, num_virtual_nodes=150):\n self.num_virtual_nodes = num_virtual_nodes\n self.ring = {} # hash -&gt; server_id\n self.sorted_hashes = []\n self.servers = set()\n \n def _hash(self, key):\n &#34;&#34;&#34;Hash function&#34;&#34;&#34;\n return int(hashlib.md5(str(key).encode()).hexdigest(), 16)\n \n def add_server(self, server_id):\n &#34;&#34;&#34;Add server to ring&#34;&#34;&#34;\n self.servers.add(server_id)\n \n for i in range(self.num_virtual_nodes):\n virtual_key = f&#34;{server_id}:{i}&#34;\n hash_value = self._hash(virtual_key)\n self.ring[hash_value] = server_id\n self.sorted_hashes.append(hash_value)\n \n self.sorted_hashes.sort()\n \n def remove_server(self, server_id):\n &#34;&#34;&#34;Remove server from ring&#34;&#34;&#34;\n self.servers.discard(server_id)\n \n hashes_to_remove = []\n for i in range(self.num_virtual_nodes):\n virtual_key = f&#34;{server_id}:{i}&#34;\n hash_value = self._hash(virtual_key)\n hashes_to_remove.append(hash_value)\n del self.ring[hash_value]\n \n for h in hashes_to_remove:\n self.sorted_hashes.remove(h)\n \n def get_server(self, key):\n &#34;&#34;&#34;Get server for key&#34;&#34;&#34;\n if not self.servers:\n return None\n \n hash_value = self._hash(key)\n \n # Find the first server hash &gt;= key hash\n idx = bisect_right(self.sorted_hashes, hash_value)\n \n if idx == len(self.sorted_hashes):\n idx = 0\n \n server_hash = self.sorted_hashes[idx]\n return self.ring[server_hash]\n\n# Usage\nch = ConsistentHashing()\nch.add_server(&#34;server:1&#34;)\nch.add_server(&#34;server:2&#34;)\nch.add_server(&#34;server:3&#34;)\n\n# Map keys to servers\nfor key in [&#34;user:1&#34;, &#34;user:2&#34;, &#34;user:3&#34;, &#34;user:4&#34;, &#34;user:5&#34;]:\n server = ch.get_server(key)\n print(f&#34;{key} -&gt; {server}&#34;)\n\n# Add new server - only some keys rehash\nprint(&#34;\\nAdding server:4...&#34;)\nch.add_server(&#34;server:4&#34;)\n\nfor key in [&#34;user:1&#34;, &#34;user:2&#34;, &#34;user:3&#34;, &#34;user:4&#34;, &#34;user:5&#34;]:\n server = ch.get_server(key)\n print(f&#34;{key} -&gt; {server}&#34;)<\/code><\/pre><p><strong>Benefits:<\/strong><\/p>\n<ul>\n<li>Adding\/removing server affects only 1\/N of keys<\/li>\n<li>Virtual nodes ensure better distribution<\/li>\n<li>Works well for cache distribution<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"27-design-a-url-shortener\"><span>27. Design a URL Shortener<\/span>\n <a href=\"#27-design-a-url-shortener\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"271-requirements\"><span>27.1 Requirements<\/span>\n <a href=\"#271-requirements\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Functional:<\/strong><\/p>\n<ul>\n<li>Shorten long URLs<\/li>\n<li>Redirect short URL to long URL<\/li>\n<li>Custom short URLs (optional)<\/li>\n<\/ul>\n<p><strong>Non-Functional:<\/strong><\/p>\n<ul>\n<li>High availability<\/li>\n<li>Low latency<\/li>\n<li>Shortened URL unique and not guessable<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"272-architecture\"><span>27.2 Architecture<\/span>\n <a href=\"#272-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/url-shortener-architecture.webp\" title=\"URL Shortener System \u2014 System design for high-traffic URL shortening services with caching and analytics\" data-thumbnail=\"\/files\/url-shortener-architecture.webp\" data-sub-html=\"<h2>URL shortener service system design architecture \u2014 encoding, caching, and redirect optimization for link shortening<\/h2><p>URL Shortener System \u2014 System design for high-traffic URL shortening services with caching and analytics<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/url-shortener-architecture.webp' alt=\"URL shortener service system design architecture \u2014 encoding, caching, and redirect optimization for link shortening\"><\/a><figcaption class=\"image-caption\">URL Shortener System \u2014 System design for high-traffic URL shortening services with caching and analytics<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"273-implementation\"><span>27.3 Implementation<\/span>\n <a href=\"#273-implementation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>import string\nimport random\nfrom datetime import datetime\n\nclass URLShortener:\n def __init__(self, base_url=&#34;http:\/\/short.url&#34;, cache=None, db=None):\n self.base_url = base_url\n self.cache = cache or {}\n self.db = db or {}\n self.characters = string.ascii_letters &#43; string.digits\n \n def _generate_short_code(self, length=6):\n &#34;&#34;&#34;Generate random short code&#34;&#34;&#34;\n return &#39;&#39;.join(random.choice(self.characters) for _ in range(length))\n \n def shorten(self, long_url, custom_code=None):\n &#34;&#34;&#34;Create shortened URL&#34;&#34;&#34;\n short_code = custom_code or self._generate_short_code()\n \n # Check if custom code exists\n if short_code in self.db:\n return None # Already exists\n \n url_record = {\n &#39;long_url&#39;: long_url,\n &#39;short_code&#39;: short_code,\n &#39;created_at&#39;: datetime.now(),\n &#39;expires_at&#39;: None,\n &#39;click_count&#39;: 0\n }\n \n # Store in database\n self.db[short_code] = url_record\n \n # Cache it\n self.cache[short_code] = long_url\n \n return f&#34;{self.base_url}\/{short_code}&#34;\n \n def expand(self, short_code):\n &#34;&#34;&#34;Get original URL&#34;&#34;&#34;\n # Check cache first\n if short_code in self.cache:\n return self.cache[short_code]\n \n # Check database\n if short_code in self.db:\n long_url = self.db[short_code][&#39;long_url&#39;]\n # Update cache\n self.cache[short_code] = long_url\n # Increment click count\n self.db[short_code][&#39;click_count&#39;] &#43;= 1\n return long_url\n \n return None\n \n def get_stats(self, short_code):\n &#34;&#34;&#34;Get URL statistics&#34;&#34;&#34;\n if short_code not in self.db:\n return None\n \n record = self.db[short_code]\n return {\n &#39;short_code&#39;: short_code,\n &#39;long_url&#39;: record[&#39;long_url&#39;],\n &#39;created_at&#39;: record[&#39;created_at&#39;],\n &#39;click_count&#39;: record[&#39;click_count&#39;]\n }\n\n# Usage\nshortener = URLShortener()\n\n# Create shortened URL\nshort = shortener.shorten(&#34;https:\/\/www.example.com\/very\/long\/url\/path&#34;)\nprint(f&#34;Shortened: {short}&#34;)\n\n# Expand shortened URL\nlong = shortener.expand(short.split(&#39;\/&#39;)[-1])\nprint(f&#34;Expanded: {long}&#34;)\n\n# Get statistics\ncode = short.split(&#39;\/&#39;)[-1]\nstats = shortener.get_stats(code)\nprint(f&#34;Stats: {stats}&#34;)<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"30-design-an-object-storage-system-s3\"><span>30. Design an Object Storage System (S3)<\/span>\n <a href=\"#30-design-an-object-storage-system-s3\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"301-high-level-design\"><span>30.1 High-Level Design<\/span>\n <a href=\"#301-high-level-design\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/object-storage-s3-architecture.webp\" title=\"Object Storage Architecture \u2014 S3-compatible distributed storage system design for petabyte-scale data\" data-thumbnail=\"\/files\/object-storage-s3-architecture.webp\" data-sub-html=\"<h2>Object storage system S3-like architecture \u2014 scalable distributed storage for massive unstructured data<\/h2><p>Object Storage Architecture \u2014 S3-compatible distributed storage system design for petabyte-scale data<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/object-storage-s3-architecture.webp' alt=\"Object storage system S3-like architecture \u2014 scalable distributed storage for massive unstructured data\"><\/a><figcaption class=\"image-caption\">Object Storage Architecture \u2014 S3-compatible distributed storage system design for petabyte-scale data<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"302-implementation-concepts\"><span>30.2 Implementation Concepts<\/span>\n <a href=\"#302-implementation-concepts\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>from dataclasses import dataclass\nfrom datetime import datetime\nimport hashlib\nimport os\n\n@dataclass\nclass ObjectMetadata:\n bucket: str\n key: str\n size: int\n content_type: str\n etag: str # hash for integrity\n created_at: datetime\n replicas: list # list of storage nodes with copies\n\nclass S3LikeStorage:\n def __init__(self, storage_nodes=3):\n self.metadata = {} # bucket\/key -&gt; metadata\n self.storage_nodes = [f&#34;node_{i}&#34; for i in range(storage_nodes)]\n self.object_store = {} # actual data storage\n \n def put_object(self, bucket, key, data, content_type=&#34;binary&#34;):\n &#34;&#34;&#34;Upload object&#34;&#34;&#34;\n # Calculate hash\n etag = hashlib.md5(data).hexdigest()\n \n # Store metadata\n metadata = ObjectMetadata(\n bucket=bucket,\n key=key,\n size=len(data),\n content_type=content_type,\n etag=etag,\n created_at=datetime.now(),\n replicas=self.storage_nodes.copy()\n )\n \n full_key = f&#34;{bucket}\/{key}&#34;\n self.metadata[full_key] = metadata\n self.object_store[full_key] = data\n \n # Replicate to other nodes\n self._replicate(full_key, data)\n \n return etag\n \n def get_object(self, bucket, key):\n &#34;&#34;&#34;Download object&#34;&#34;&#34;\n full_key = f&#34;{bucket}\/{key}&#34;\n \n if full_key not in self.object_store:\n return None\n \n return self.object_store[full_key]\n \n def delete_object(self, bucket, key):\n &#34;&#34;&#34;Delete object&#34;&#34;&#34;\n full_key = f&#34;{bucket}\/{key}&#34;\n \n if full_key in self.metadata:\n del self.metadata[full_key]\n if full_key in self.object_store:\n del self.object_store[full_key]\n \n return True\n \n def list_objects(self, bucket, prefix=&#34;&#34;):\n &#34;&#34;&#34;List objects in bucket&#34;&#34;&#34;\n objects = []\n for key in self.object_store:\n if key.startswith(f&#34;{bucket}\/&#34;):\n obj_key = key.replace(f&#34;{bucket}\/&#34;, &#34;&#34;)\n if obj_key.startswith(prefix):\n objects.append(obj_key)\n return objects\n \n def _replicate(self, key, data):\n &#34;&#34;&#34;Replicate data to storage nodes&#34;&#34;&#34;\n for node in self.storage_nodes:\n # In real system: write to remote storage node\n pass\n\n# Usage\ns3 = S3LikeStorage()\n\n# Upload\netag = s3.put_object(&#34;my-bucket&#34;, &#34;file.txt&#34;, b&#34;Hello World&#34;)\nprint(f&#34;Uploaded with ETag: {etag}&#34;)\n\n# Download\ndata = s3.get_object(&#34;my-bucket&#34;, &#34;file.txt&#34;)\nprint(f&#34;Downloaded: {data}&#34;)\n\n# List\nobjects = s3.list_objects(&#34;my-bucket&#34;)\nprint(f&#34;Objects: {objects}&#34;)<\/code><\/pre><h2 class=\"heading-element\" id=\"31-design-youtube--video-streaming-platform\"><span>31. Design YouTube \/ Video Streaming Platform<\/span>\n <a href=\"#31-design-youtube--video-streaming-platform\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"311-architecture-overview\"><span>31.1 Architecture Overview<\/span>\n <a href=\"#311-architecture-overview\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/video-streaming-architecture.webp\" title=\"Video Streaming Platform \u2014 End-to-end video system design: upload, processing, storage, and adaptive delivery\" data-thumbnail=\"\/files\/video-streaming-architecture.webp\" data-sub-html=\"<h2>YouTube-like video streaming platform architecture \u2014 video processing, transcoding, and CDN delivery at scale<\/h2><p>Video Streaming Platform \u2014 End-to-end video system design: upload, processing, storage, and adaptive delivery<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/video-streaming-architecture.webp' alt=\"YouTube-like video streaming platform architecture \u2014 video processing, transcoding, and CDN delivery at scale\"><\/a><figcaption class=\"image-caption\">Video Streaming Platform \u2014 End-to-end video system design: upload, processing, storage, and adaptive delivery<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"312-video-processing-pipeline\"><span>31.2 Video Processing Pipeline<\/span>\n <a href=\"#312-video-processing-pipeline\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>from enum import Enum\n\nclass VideoQuality(Enum):\n LOW = &#34;360p&#34;\n MEDIUM = &#34;480p&#34;\n HIGH = &#34;720p&#34;\n HD = &#34;1080p&#34;\n FULL_HD = &#34;4K&#34;\n\nclass VideoProcessingService:\n def __init__(self):\n self.processing_queue = []\n self.storage = {}\n \n def upload_video(self, user_id, video_file):\n &#34;&#34;&#34;Handle video upload&#34;&#34;&#34;\n video_id = self._generate_id()\n \n # Store raw video\n raw_path = f&#34;raw\/{video_id}\/{video_file.name}&#34;\n self.storage[raw_path] = video_file\n \n # Queue for processing\n job = {\n &#39;video_id&#39;: video_id,\n &#39;user_id&#39;: user_id,\n &#39;raw_path&#39;: raw_path,\n &#39;status&#39;: &#39;queued&#39;\n }\n self.processing_queue.append(job)\n \n return video_id\n \n def process_video(self, job):\n &#34;&#34;&#34;Process video: transcode to multiple qualities&#34;&#34;&#34;\n video_id = job[&#39;video_id&#39;]\n raw_path = job[&#39;raw_path&#39;]\n \n # Transcode to different qualities\n qualities = [VideoQuality.LOW, VideoQuality.MEDIUM, \n VideoQuality.HIGH, VideoQuality.HD]\n \n processed_paths = []\n for quality in qualities:\n output_path = f&#34;processed\/{video_id}\/{quality.value}.mp4&#34;\n self._transcode(raw_path, output_path, quality)\n processed_paths.append(output_path)\n \n # Generate thumbnail\n thumbnail_path = f&#34;thumbnails\/{video_id}\/thumb.jpg&#34;\n self._generate_thumbnail(raw_path, thumbnail_path)\n \n # Update metadata\n self._update_metadata(video_id, {\n &#39;processed_paths&#39;: processed_paths,\n &#39;thumbnail&#39;: thumbnail_path,\n &#39;status&#39;: &#39;ready&#39;\n })\n \n return processed_paths\n \n def _transcode(self, input_path, output_path, quality):\n &#34;&#34;&#34;Transcode video to specified quality&#34;&#34;&#34;\n # Simplified - real implementation uses FFmpeg\n print(f&#34;Transcoding {input_path} to {quality.value}&#34;)\n self.storage[output_path] = f&#34;transcoded_{quality.value}&#34;\n \n def _generate_thumbnail(self, video_path, output_path):\n &#34;&#34;&#34;Extract thumbnail from video&#34;&#34;&#34;\n print(f&#34;Generating thumbnail for {video_path}&#34;)\n self.storage[output_path] = &#34;thumbnail_data&#34;\n \n def _generate_id(self):\n import uuid\n return str(uuid.uuid4())\n \n def _update_metadata(self, video_id, metadata):\n &#34;&#34;&#34;Update video metadata&#34;&#34;&#34;\n print(f&#34;Updating metadata for {video_id}: {metadata}&#34;)<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"32-design-a-chat-system\"><span>32. Design a Chat System<\/span>\n <a href=\"#32-design-a-chat-system\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"321-architecture\"><span>32.1 Architecture<\/span>\n <a href=\"#321-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/chat-system-architecture.webp\" title=\"Chat System Architecture \u2014 Real-time messaging design with persistence, notifications, and scalability\" data-thumbnail=\"\/files\/chat-system-architecture.webp\" data-sub-html=\"<h2>Real-time chat system architecture with WebSockets and message persistence \u2014 low-latency messaging for communication apps<\/h2><p>Chat System Architecture \u2014 Real-time messaging design with persistence, notifications, and scalability<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/chat-system-architecture.webp' alt=\"Real-time chat system architecture with WebSockets and message persistence \u2014 low-latency messaging for communication apps\"><\/a><figcaption class=\"image-caption\">Chat System Architecture \u2014 Real-time messaging design with persistence, notifications, and scalability<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"322-implementation\"><span>32.2 Implementation<\/span>\n <a href=\"#322-implementation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>import json\nfrom datetime import datetime\nfrom collections import defaultdict\n\nclass ChatSystem:\n def __init__(self):\n self.connections = {} # user_id -&gt; websocket\n self.messages = [] # message history\n self.online_users = set()\n self.typing_status = defaultdict(set) # chat_id -&gt; set of typing users\n \n def connect_user(self, user_id, websocket):\n &#34;&#34;&#34;User connects to chat&#34;&#34;&#34;\n self.connections[user_id] = websocket\n self.online_users.add(user_id)\n \n # Notify others user is online\n self._broadcast_presence(user_id, &#39;online&#39;)\n \n return True\n \n def disconnect_user(self, user_id):\n &#34;&#34;&#34;User disconnects&#34;&#34;&#34;\n if user_id in self.connections:\n del self.connections[user_id]\n self.online_users.discard(user_id)\n \n # Notify others user is offline\n self._broadcast_presence(user_id, &#39;offline&#39;)\n \n def send_message(self, from_user, to_user, content):\n &#34;&#34;&#34;Send message from one user to another&#34;&#34;&#34;\n message = {\n &#39;id&#39;: len(self.messages) &#43; 1,\n &#39;from&#39;: from_user,\n &#39;to&#39;: to_user,\n &#39;content&#39;: content,\n &#39;timestamp&#39;: datetime.now().isoformat(),\n &#39;read&#39;: False\n }\n \n # Store message\n self.messages.append(message)\n \n # Deliver if recipient online\n if to_user in self.connections:\n self._deliver_message(to_user, message)\n \n return message[&#39;id&#39;]\n \n def send_group_message(self, from_user, group_id, content):\n &#34;&#34;&#34;Send message to group&#34;&#34;&#34;\n message = {\n &#39;id&#39;: len(self.messages) &#43; 1,\n &#39;from&#39;: from_user,\n &#39;group_id&#39;: group_id,\n &#39;content&#39;: content,\n &#39;timestamp&#39;: datetime.now().isoformat()\n }\n \n self.messages.append(message)\n \n # Deliver to all group members\n group_members = self._get_group_members(group_id)\n for member in group_members:\n if member != from_user and member in self.connections:\n self._deliver_message(member, message)\n \n return message[&#39;id&#39;]\n \n def mark_typing(self, user_id, chat_id, is_typing):\n &#34;&#34;&#34;Update typing status&#34;&#34;&#34;\n if is_typing:\n self.typing_status[chat_id].add(user_id)\n else:\n self.typing_status[chat_id].discard(user_id)\n \n # Notify other participants\n self._broadcast_typing(chat_id, user_id, is_typing)\n \n def get_message_history(self, user_id, other_user_id, limit=50):\n &#34;&#34;&#34;Get chat history between two users&#34;&#34;&#34;\n history = []\n for msg in reversed(self.messages):\n if ((msg.get(&#39;from&#39;) == user_id and msg.get(&#39;to&#39;) == other_user_id) or\n (msg.get(&#39;from&#39;) == other_user_id and msg.get(&#39;to&#39;) == user_id)):\n history.append(msg)\n if len(history) &gt;= limit:\n break\n return list(reversed(history))\n \n def _deliver_message(self, user_id, message):\n &#34;&#34;&#34;Deliver message via WebSocket&#34;&#34;&#34;\n if user_id in self.connections:\n websocket = self.connections[user_id]\n # In real system: websocket.send(json.dumps(message))\n print(f&#34;Delivered to {user_id}: {message}&#34;)\n \n def _broadcast_presence(self, user_id, status):\n &#34;&#34;&#34;Notify all users about presence change&#34;&#34;&#34;\n for uid, ws in self.connections.items():\n if uid != user_id:\n # Send presence notification\n print(f&#34;User {user_id} is {status}&#34;)\n \n def _broadcast_typing(self, chat_id, user_id, is_typing):\n &#34;&#34;&#34;Notify typing status&#34;&#34;&#34;\n pass\n \n def _get_group_members(self, group_id):\n &#34;&#34;&#34;Get members of a group&#34;&#34;&#34;\n # Simplified - would query database\n return []\n\n# Usage\nchat = ChatSystem()\n\n# Users connect\nchat.connect_user(&#39;alice&#39;, &#39;websocket_alice&#39;)\nchat.connect_user(&#39;bob&#39;, &#39;websocket_bob&#39;)\n\n# Send message\nmsg_id = chat.send_message(&#39;alice&#39;, &#39;bob&#39;, &#39;Hello Bob!&#39;)\n\n# Get history\nhistory = chat.get_message_history(&#39;alice&#39;, &#39;bob&#39;)\nprint(f&#34;Message history: {history}&#34;)<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"33-design-a-news-feed-system\"><span>33. Design a News Feed System<\/span>\n <a href=\"#33-design-a-news-feed-system\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"331-architecture\"><span>33.1 Architecture<\/span>\n <a href=\"#331-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/news-feed-architecture.webp\" title=\"News Feed System \u2014 Designing scalable social media feeds with ranking, caching, and real-time updates\" data-thumbnail=\"\/files\/news-feed-architecture.webp\" data-sub-html=\"<h2>Social media news feed system architecture \u2014 real-time feed generation and delivery for billions of interactions<\/h2><p>News Feed System \u2014 Designing scalable social media feeds with ranking, caching, and real-time updates<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/news-feed-architecture.webp' alt=\"Social media news feed system architecture \u2014 real-time feed generation and delivery for billions of interactions\"><\/a><figcaption class=\"image-caption\">News Feed System \u2014 Designing scalable social media feeds with ranking, caching, and real-time updates<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"332-feed-generation-strategies\"><span>33.2 Feed Generation Strategies<\/span>\n <a href=\"#332-feed-generation-strategies\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Fan-out on Write (Push Model):<\/strong><\/p>\n<pre><code>class FanoutOnWrite:\n def __init__(self):\n self.feeds = {} # user_id -&gt; list of post_ids\n self.posts = {} # post_id -&gt; post data\n self.followers = {} # user_id -&gt; list of follower_ids\n \n def create_post(self, user_id, content):\n &#34;&#34;&#34;User creates a post&#34;&#34;&#34;\n post_id = len(self.posts) &#43; 1\n post = {\n &#39;id&#39;: post_id,\n &#39;user_id&#39;: user_id,\n &#39;content&#39;: content,\n &#39;created_at&#39;: datetime.now()\n }\n \n self.posts[post_id] = post\n \n # Fan-out: push to all followers&#39; feeds\n followers = self.followers.get(user_id, [])\n for follower_id in followers:\n if follower_id not in self.feeds:\n self.feeds[follower_id] = []\n self.feeds[follower_id].insert(0, post_id) # Add to front\n \n return post_id\n \n def get_feed(self, user_id, limit=10):\n &#34;&#34;&#34;Get user&#39;s news feed&#34;&#34;&#34;\n post_ids = self.feeds.get(user_id, [])[:limit]\n return [self.posts[pid] for pid in post_ids]<\/code><\/pre><p><strong>Fan-out on Read (Pull Model):<\/strong><\/p>\n<pre><code>class FanoutOnRead:\n def __init__(self):\n self.posts = {} # user_id -&gt; list of posts\n self.following = {} # user_id -&gt; list of followed user_ids\n \n def create_post(self, user_id, content):\n &#34;&#34;&#34;User creates a post&#34;&#34;&#34;\n post_id = len(sum(self.posts.values(), [])) &#43; 1\n post = {\n &#39;id&#39;: post_id,\n &#39;user_id&#39;: user_id,\n &#39;content&#39;: content,\n &#39;created_at&#39;: datetime.now()\n }\n \n if user_id not in self.posts:\n self.posts[user_id] = []\n self.posts[user_id].insert(0, post)\n \n return post_id\n \n def get_feed(self, user_id, limit=10):\n &#34;&#34;&#34;Aggregate feed on-the-fly&#34;&#34;&#34;\n followed_users = self.following.get(user_id, [])\n \n # Collect posts from all followed users\n all_posts = []\n for followed_id in followed_users:\n all_posts.extend(self.posts.get(followed_id, []))\n \n # Sort by timestamp and return top posts\n all_posts.sort(key=lambda p: p[&#39;created_at&#39;], reverse=True)\n return all_posts[:limit]<\/code><\/pre><p><strong>Hybrid Approach:<\/strong><\/p>\n<pre><code>class HybridFeedService:\n def __init__(self):\n self.fanout_write = FanoutOnWrite()\n self.fanout_read = FanoutOnRead()\n self.celebrity_threshold = 10000 # Users with &gt;10K followers\n \n def create_post(self, user_id, content):\n &#34;&#34;&#34;Smart fan-out based on user popularity&#34;&#34;&#34;\n follower_count = len(self.fanout_write.followers.get(user_id, []))\n \n if follower_count &gt; self.celebrity_threshold:\n # Celebrity: use pull model\n return self.fanout_read.create_post(user_id, content)\n else:\n # Regular user: use push model\n return self.fanout_write.create_post(user_id, content)\n \n def get_feed(self, user_id, limit=10):\n &#34;&#34;&#34;Get personalized feed&#34;&#34;&#34;\n # Mix pre-computed and real-time feeds\n precomputed = self.fanout_write.get_feed(user_id, limit \/\/ 2)\n realtime = self.fanout_read.get_feed(user_id, limit \/\/ 2)\n \n # Merge and sort\n combined = precomputed &#43; realtime\n combined.sort(key=lambda p: p[&#39;created_at&#39;], reverse=True)\n \n return combined[:limit]<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"34-design-a-search-autocomplete-system\"><span>34. Design a Search Autocomplete System<\/span>\n <a href=\"#34-design-a-search-autocomplete-system\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"341-trie-based-implementation\"><span>34.1 Trie-Based Implementation<\/span>\n <a href=\"#341-trie-based-implementation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/search-autocomplete-architecture.webp\" title=\"Autocomplete System \u2014 Efficient search suggestion implementation using trie and ranking algorithms\" data-thumbnail=\"\/files\/search-autocomplete-architecture.webp\" data-sub-html=\"<h2>Search autocomplete system with trie data structure \u2014 real-time search suggestions and query optimization<\/h2><p>Autocomplete System \u2014 Efficient search suggestion implementation using trie and ranking algorithms<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/search-autocomplete-architecture.webp' alt=\"Search autocomplete system with trie data structure \u2014 real-time search suggestions and query optimization\"><\/a><figcaption class=\"image-caption\">Autocomplete System \u2014 Efficient search suggestion implementation using trie and ranking algorithms<\/figcaption>\n <\/figure><\/p>\n<pre><code>class TrieNode:\n def __init__(self):\n self.children = {}\n self.is_end = False\n self.frequency = 0\n self.suggestions = [] # Top suggestions from this node\n\nclass AutocompleteSystem:\n def __init__(self):\n self.root = TrieNode()\n \n def add_query(self, query, frequency=1):\n &#34;&#34;&#34;Add search query to trie&#34;&#34;&#34;\n node = self.root\n \n for char in query.lower():\n if char not in node.children:\n node.children[char] = TrieNode()\n node = node.children[char]\n \n node.is_end = True\n node.frequency &#43;= frequency\n \n def search(self, prefix, limit=5):\n &#34;&#34;&#34;Get autocomplete suggestions&#34;&#34;&#34;\n node = self.root\n \n # Navigate to prefix\n for char in prefix.lower():\n if char not in node.children:\n return []\n node = node.children[char]\n \n # Collect all completions\n suggestions = []\n self._collect_suggestions(node, prefix, suggestions)\n \n # Sort by frequency and return top results\n suggestions.sort(key=lambda x: x[1], reverse=True)\n return [s[0] for s in suggestions[:limit]]\n \n def _collect_suggestions(self, node, prefix, suggestions):\n &#34;&#34;&#34;DFS to collect all completions&#34;&#34;&#34;\n if node.is_end:\n suggestions.append((prefix, node.frequency))\n \n for char, child_node in node.children.items():\n self._collect_suggestions(child_node, prefix &#43; char, suggestions)\n\n# Usage\nautocomplete = AutocompleteSystem()\n\n# Add popular queries\nqueries = [\n (&#34;system design&#34;, 1000),\n (&#34;system design interview&#34;, 800),\n (&#34;system design patterns&#34;, 600),\n (&#34;system architecture&#34;, 500),\n (&#34;distributed systems&#34;, 900)\n]\n\nfor query, freq in queries:\n autocomplete.add_query(query, freq)\n\n# Search\nresults = autocomplete.search(&#34;sys&#34;)\nprint(f&#34;Suggestions for &#39;sys&#39;: {results}&#34;)<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"35-design-a-notification-system\"><span>35. Design a Notification System<\/span>\n <a href=\"#35-design-a-notification-system\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"351-multi-channel-notification-system\"><span>35.1 Multi-Channel Notification System<\/span>\n <a href=\"#351-multi-channel-notification-system\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/notification-system-architecture.webp\" title=\"Notification System \u2014 Multi-channel messaging system design for reliable user notifications\" data-thumbnail=\"\/files\/notification-system-architecture.webp\" data-sub-html=\"<h2>Multi-channel notification system architecture \u2014 sending notifications via push, email, SMS across multiple channels<\/h2><p>Notification System \u2014 Multi-channel messaging system design for reliable user notifications<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/notification-system-architecture.webp' alt=\"Multi-channel notification system architecture \u2014 sending notifications via push, email, SMS across multiple channels\"><\/a><figcaption class=\"image-caption\">Notification System \u2014 Multi-channel messaging system design for reliable user notifications<\/figcaption>\n <\/figure><\/p>\n<pre><code>from abc import ABC, abstractmethod\nfrom enum import Enum\n\nclass NotificationType(Enum):\n EMAIL = &#34;email&#34;\n SMS = &#34;sms&#34;\n PUSH = &#34;push&#34;\n IN_APP = &#34;in_app&#34;\n\nclass NotificationChannel(ABC):\n @abstractmethod\n def send(self, recipient, message):\n pass\n\nclass EmailChannel(NotificationChannel):\n def send(self, recipient, message):\n print(f&#34;Sending email to {recipient}: {message}&#34;)\n # Implementation: SMTP, SendGrid, etc.\n\nclass SMSChannel(NotificationChannel):\n def send(self, recipient, message):\n print(f&#34;Sending SMS to {recipient}: {message}&#34;)\n # Implementation: Twilio, AWS SNS, etc.\n\nclass PushChannel(NotificationChannel):\n def send(self, recipient, message):\n print(f&#34;Sending push to {recipient}: {message}&#34;)\n # Implementation: FCM, APNS, etc.\n\nclass InAppChannel(NotificationChannel):\n def send(self, recipient, message):\n print(f&#34;Sending in-app notification to {recipient}: {message}&#34;)\n # Implementation: WebSocket, polling, etc.\n\nclass NotificationService:\n def __init__(self):\n self.channels = {\n NotificationType.EMAIL: EmailChannel(),\n NotificationType.SMS: SMSChannel(),\n NotificationType.PUSH: PushChannel(),\n NotificationType.IN_APP: InAppChannel()\n }\n self.user_preferences = {} # user_id -&gt; list of enabled channels\n self.notification_queue = []\n \n def send_notification(self, user_id, message, channels=None):\n &#34;&#34;&#34;Send notification via specified channels&#34;&#34;&#34;\n if channels is None:\n channels = self.user_preferences.get(user_id, \n [NotificationType.PUSH, NotificationType.IN_APP])\n \n for channel_type in channels:\n if channel_type in self.channels:\n channel = self.channels[channel_type]\n \n # Add to queue for async processing\n self.notification_queue.append({\n &#39;user_id&#39;: user_id,\n &#39;channel&#39;: channel,\n &#39;message&#39;: message\n })\n \n def process_queue(self):\n &#34;&#34;&#34;Process notification queue&#34;&#34;&#34;\n while self.notification_queue:\n notif = self.notification_queue.pop(0)\n try:\n notif[&#39;channel&#39;].send(notif[&#39;user_id&#39;], notif[&#39;message&#39;])\n except Exception as e:\n print(f&#34;Failed to send notification: {e}&#34;)\n # Retry logic here\n\n# Usage\nnotification_service = NotificationService()\n\n# Send multi-channel notification\nnotification_service.send_notification(\n user_id=&#39;user123&#39;,\n message=&#39;Your order has shipped!&#39;,\n channels=[NotificationType.EMAIL, NotificationType.PUSH]\n)<\/code><\/pre><h2 class=\"heading-element\" id=\"21-common-design-patterns-in-system-design\"><span>21. Common Design Patterns in System Design<\/span>\n <a href=\"#21-common-design-patterns-in-system-design\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"211-creational-patterns\"><span>21.1 Creational Patterns<\/span>\n <a href=\"#211-creational-patterns\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Singleton Pattern<\/strong><\/p>\n<p>Ensures only one instance of a class:<\/p>\n<pre><code>class Database:\n _instance = None\n _lock = __import__(&#39;threading&#39;).Lock()\n \n def __new__(cls):\n if cls._instance is None:\n with cls._lock:\n if cls._instance is None:\n cls._instance = super().__new__(cls)\n cls._instance._initialized = False\n return cls._instance\n \n def __init__(self):\n if self._initialized:\n return\n self.connection = self._create_connection()\n self._initialized = True\n \n def _create_connection(self):\n print(&#34;Creating database connection&#34;)\n return &#34;connection_object&#34;\n\n# Usage\ndb1 = Database()\ndb2 = Database()\nprint(db1 is db2) # True - same instance<\/code><\/pre><p><strong>Factory Pattern<\/strong><\/p>\n<p>Create objects without specifying exact classes:<\/p>\n<pre><code>from abc import ABC, abstractmethod\n\nclass Cache(ABC):\n @abstractmethod\n def get(self, key): pass\n @abstractmethod\n def set(self, key, value): pass\n\nclass RedisCache(Cache):\n def get(self, key):\n return f&#34;Redis get {key}&#34;\n def set(self, key, value):\n print(f&#34;Redis set {key}={value}&#34;)\n\nclass MemcachedCache(Cache):\n def get(self, key):\n return f&#34;Memcached get {key}&#34;\n def set(self, key, value):\n print(f&#34;Memcached set {key}={value}&#34;)\n\nclass CacheFactory:\n @staticmethod\n def create_cache(cache_type):\n if cache_type == &#34;redis&#34;:\n return RedisCache()\n elif cache_type == &#34;memcached&#34;:\n return MemcachedCache()\n raise ValueError(f&#34;Unknown cache type: {cache_type}&#34;)\n\n# Usage\ncache = CacheFactory.create_cache(&#34;redis&#34;)\ncache.set(&#34;key&#34;, &#34;value&#34;)<\/code><\/pre><p><strong>Builder Pattern<\/strong><\/p>\n<p>Construct complex objects step by step:<\/p>\n<pre><code>class QueryBuilder:\n def __init__(self):\n self.select_fields = []\n self.from_table = None\n self.where_clause = None\n self.order_by = None\n self.limit_count = None\n \n def select(self, *fields):\n self.select_fields = list(fields)\n return self\n \n def from_table(self, table):\n self.from_table = table\n return self\n \n def where(self, condition):\n self.where_clause = condition\n return self\n \n def order_by(self, field, direction=&#34;ASC&#34;):\n self.order_by = f&#34;{field} {direction}&#34;\n return self\n \n def limit(self, count):\n self.limit_count = count\n return self\n \n def build(self):\n query = f&#34;SELECT {&#39;,&#39;.join(self.select_fields)} FROM {self.from_table}&#34;\n \n if self.where_clause:\n query &#43;= f&#34; WHERE {self.where_clause}&#34;\n if self.order_by:\n query &#43;= f&#34; ORDER BY {self.order_by}&#34;\n if self.limit_count:\n query &#43;= f&#34; LIMIT {self.limit_count}&#34;\n \n return query\n\n# Usage\nquery = (QueryBuilder()\n .select(&#34;id&#34;, &#34;name&#34;, &#34;email&#34;)\n .from_table(&#34;users&#34;)\n .where(&#34;age &gt; 18&#34;)\n .order_by(&#34;name&#34;)\n .limit(10)\n .build())\n\nprint(query)<\/code><\/pre><h3 class=\"heading-element\" id=\"212-structural-patterns\"><span>21.2 Structural Patterns<\/span>\n <a href=\"#212-structural-patterns\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Adapter Pattern<\/strong><\/p>\n<p>Convert one interface to another:<\/p>\n<pre><code>class LegacyPaymentSystem:\n def make_payment(self, amount):\n return f&#34;Legacy system processed ${amount}&#34;\n\nclass ModernPaymentInterface:\n def charge(self, amount):\n pass\n\nclass PaymentAdapter(ModernPaymentInterface):\n def __init__(self, legacy_system):\n self.legacy = legacy_system\n \n def charge(self, amount):\n return self.legacy.make_payment(amount)\n\n# Usage\nlegacy = LegacyPaymentSystem()\nadapter = PaymentAdapter(legacy)\nresult = adapter.charge(100)\nprint(result)<\/code><\/pre><p><strong>Decorator Pattern<\/strong><\/p>\n<p>Add functionality to objects dynamically:<\/p>\n<pre><code>class Coffee:\n def cost(self):\n return 5.0\n \n def description(self):\n return &#34;Coffee&#34;\n\nclass CoffeeDecorator(Coffee):\n def __init__(self, coffee):\n self.coffee = coffee\n\nclass Milk(CoffeeDecorator):\n def cost(self):\n return self.coffee.cost() &#43; 1.0\n \n def description(self):\n return self.coffee.description() &#43; &#34;, Milk&#34;\n\nclass Espresso(CoffeeDecorator):\n def cost(self):\n return self.coffee.cost() &#43; 2.0\n \n def description(self):\n return self.coffee.description() &#43; &#34;, Espresso&#34;\n\n# Usage\ncoffee = Coffee()\ncoffee = Milk(coffee)\ncoffee = Espresso(coffee)\n\nprint(f&#34;{coffee.description()}: ${coffee.cost()}&#34;) \n# Coffee, Milk, Espresso: $8.0<\/code><\/pre><p><strong>Facade Pattern<\/strong><\/p>\n<p>Provide simplified interface to complex subsystem:<\/p>\n<pre><code>class VideoEncoder:\n def encode(self, video):\n return f&#34;Encoded {video}&#34;\n\nclass AudioProcessor:\n def process(self, audio):\n return f&#34;Processed {audio}&#34;\n\nclass Subtitles:\n def generate(self, video):\n return f&#34;Generated subtitles for {video}&#34;\n\nclass VideoConversionFacade:\n def __init__(self):\n self.encoder = VideoEncoder()\n self.audio = AudioProcessor()\n self.subtitles = Subtitles()\n \n def convert_video(self, video_file):\n &#34;&#34;&#34;Single method to convert video with all options&#34;&#34;&#34;\n encoded = self.encoder.encode(video_file)\n processed_audio = self.audio.process(video_file)\n subs = self.subtitles.generate(video_file)\n \n return f&#34;Converted: {encoded}, {processed_audio}, {subs}&#34;\n\n# Usage\nconverter = VideoConversionFacade()\nresult = converter.convert_video(&#34;movie.mp4&#34;)\nprint(result)<\/code><\/pre><h3 class=\"heading-element\" id=\"213-behavioral-patterns\"><span>21.3 Behavioral Patterns<\/span>\n <a href=\"#213-behavioral-patterns\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Observer Pattern<\/strong><\/p>\n<p>Objects notify observers of state changes:<\/p>\n<pre><code>from abc import ABC, abstractmethod\n\nclass Observer(ABC):\n @abstractmethod\n def update(self, event):\n pass\n\nclass EventEmitter:\n def __init__(self):\n self.observers = []\n \n def subscribe(self, observer):\n self.observers.append(observer)\n \n def unsubscribe(self, observer):\n self.observers.remove(observer)\n \n def emit(self, event):\n for observer in self.observers:\n observer.update(event)\n\nclass EmailNotifier(Observer):\n def update(self, event):\n print(f&#34;Sending email: {event}&#34;)\n\nclass LogNotifier(Observer):\n def update(self, event):\n print(f&#34;Logging: {event}&#34;)\n\n# Usage\nemitter = EventEmitter()\nemitter.subscribe(EmailNotifier())\nemitter.subscribe(LogNotifier())\n\nemitter.emit(&#34;User registered&#34;)<\/code><\/pre><p><strong>Strategy Pattern<\/strong><\/p>\n<p>Select algorithm at runtime:<\/p>\n<pre><code>from abc import ABC, abstractmethod\n\nclass PaymentStrategy(ABC):\n @abstractmethod\n def pay(self, amount):\n pass\n\nclass CreditCardPayment(PaymentStrategy):\n def pay(self, amount):\n return f&#34;Paid ${amount} with credit card&#34;\n\nclass PayPalPayment(PaymentStrategy):\n def pay(self, amount):\n return f&#34;Paid ${amount} with PayPal&#34;\n\nclass CryptoCurrency(PaymentStrategy):\n def pay(self, amount):\n return f&#34;Paid ${amount} with cryptocurrency&#34;\n\nclass ShoppingCart:\n def __init__(self, payment_strategy):\n self.strategy = payment_strategy\n \n def checkout(self, amount):\n return self.strategy.pay(amount)\n\n# Usage\ncart = ShoppingCart(CreditCardPayment())\nprint(cart.checkout(100))\n\ncart.strategy = PayPalPayment()\nprint(cart.checkout(50))<\/code><\/pre><p><strong>State Pattern<\/strong><\/p>\n<p>Alter behavior when state changes:<\/p>\n<pre><code>from abc import ABC, abstractmethod\n\nclass State(ABC):\n @abstractmethod\n def process(self, order):\n pass\n\nclass PendingState(State):\n def process(self, order):\n print(&#34;Order is pending&#34;)\n order.state = ProcessingState()\n\nclass ProcessingState(State):\n def process(self, order):\n print(&#34;Order is being processed&#34;)\n order.state = ShippedState()\n\nclass ShippedState(State):\n def process(self, order):\n print(&#34;Order has shipped&#34;)\n order.state = DeliveredState()\n\nclass DeliveredState(State):\n def process(self, order):\n print(&#34;Order delivered&#34;)\n\nclass Order:\n def __init__(self):\n self.state = PendingState()\n \n def process(self):\n self.state.process(self)\n\n# Usage\norder = Order()\norder.process() # Pending\norder.process() # Processing\norder.process() # Shipped\norder.process() # Delivered<\/code><\/pre><h2 class=\"heading-element\" id=\"41-netflix-architecture\"><span>41. Netflix Architecture<\/span>\n <a href=\"#41-netflix-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"411-high-level-overview\"><span>41.1 High-Level Overview<\/span>\n <a href=\"#411-high-level-overview\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Netflix serves 280+ million paid subscribers in 190 countries, streaming 100+ million hours of content daily. Their engineering choices are documented extensively on the <a href=\"https:\/\/netflixtechblog.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Netflix Tech Blog<\/a>.<\/p>\n<p><strong>Key Components:<\/strong><\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/netflix-architecture.webp\" title=\"Netflix Architecture \u2014 How Netflix built a massive-scale streaming platform using microservices and cloud computing\" data-thumbnail=\"\/files\/netflix-architecture.webp\" data-sub-html=\"<h2>Netflix streaming service architecture case study \u2014 microservices, AWS, and CDN for reliable video streaming at scale<\/h2><p>Netflix Architecture \u2014 How Netflix built a massive-scale streaming platform using microservices and cloud computing<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/netflix-architecture.webp' alt=\"Netflix streaming service architecture case study \u2014 microservices, AWS, and CDN for reliable video streaming at scale\"><\/a><figcaption class=\"image-caption\">Netflix Architecture \u2014 How Netflix built a massive-scale streaming platform using microservices and cloud computing<\/figcaption>\n <\/figure><\/p>\n<p><strong>Architecture Principles:<\/strong><\/p>\n<ul>\n<li><strong>500+ microservices<\/strong> \u2014 each owned by a small team (<a href=\"https:\/\/netflixtechblog.com\/full-cycle-developers-at-netflix-a08c31f83249\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">&ldquo;you build it, you run it&rdquo;<\/a>)<\/li>\n<li><strong><a href=\"https:\/\/netflixtechblog.com\/the-netflix-simian-army-16e57fbab116\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Chaos Monkey<\/a><\/strong> \u2014 randomly terminates production instances to test resilience<\/li>\n<li><strong><a href=\"https:\/\/openconnect.netflix.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Open Connect CDN<\/a><\/strong> \u2014 Netflix&rsquo;s own CDN deployed at ISPs worldwide, serves 99%+ of Netflix traffic<\/li>\n<li><strong><a href=\"https:\/\/netflixtechblog.com\/per-title-encode-optimization-7e99442b62a2\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Adaptive Bitrate Streaming<\/a><\/strong> \u2014 per-title encode optimization for 1000s of quality levels<\/li>\n<li><strong><a href=\"https:\/\/netflixtechblog.com\/active-active-for-multi-regional-resiliency-c47719f6685b\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Multi-region active-active<\/a><\/strong> \u2014 traffic can shift regions automatically on failure<\/li>\n<li><strong><a href=\"https:\/\/netflixtechblog.com\/zuul-2-the-netflix-journey-to-asynchronous-non-blocking-systems-45947377acc7\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Zuul API Gateway<\/a><\/strong> \u2014 handles 100+ billion edge requests per day<\/li>\n<\/ul>\n<p><strong>Database &amp; Storage:<\/strong><\/p>\n<ul>\n<li><strong><a href=\"https:\/\/netflixtechblog.com\/caching-for-a-global-netflix-7bcc457012f1\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">EVCache<\/a><\/strong> (Memcached + replication) \u2014 distributed caching layer for user\/content data<\/li>\n<li><strong><a href=\"https:\/\/netflixtechblog.com\/scaling-time-series-data-storage-part-i-ec2b6d44ba39\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Cassandra<\/a><\/strong> \u2014 distributed storage for viewing history and user state<\/li>\n<li><strong>AWS S3<\/strong> \u2014 masters and encoded video assets<\/li>\n<li><strong><a href=\"https:\/\/github.com\/Netflix\/hollow\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Hollow<\/a><\/strong> \u2014 in-memory read-only dataset for fast catalog data access<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"412-video-streaming-pipeline\"><span>41.2 Video Streaming Pipeline<\/span>\n <a href=\"#412-video-streaming-pipeline\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>class NetflixStreamingArchitecture:\n &#34;&#34;&#34;Simplified Netflix streaming architecture&#34;&#34;&#34;\n \n def __init__(self):\n self.video_qualities = [&#39;240p&#39;, &#39;480p&#39;, &#39;720p&#39;, &#39;1080p&#39;, &#39;4K&#39;]\n self.cdn_locations = []\n self.user_bandwidth_data = {}\n \n def select_video_quality(self, user_id):\n &#34;&#34;&#34;Adaptive bitrate streaming&#34;&#34;&#34;\n bandwidth = self.user_bandwidth_data.get(user_id, 5.0) # Mbps\n \n if bandwidth &gt;= 25:\n return &#39;4K&#39;\n elif bandwidth &gt;= 5:\n return &#39;1080p&#39;\n elif bandwidth &gt;= 3:\n return &#39;720p&#39;\n elif bandwidth &gt;= 1.5:\n return &#39;480p&#39;\n else:\n return &#39;240p&#39;\n \n def get_nearest_cdn(self, user_location):\n &#34;&#34;&#34;Find closest CDN server&#34;&#34;&#34;\n # Geographic routing\n min_distance = float(&#39;inf&#39;)\n nearest_cdn = None\n \n for cdn in self.cdn_locations:\n distance = self.calculate_distance(user_location, cdn.location)\n if distance &lt; min_distance:\n min_distance = distance\n nearest_cdn = cdn\n \n return nearest_cdn<\/code><\/pre><p><strong>Key Learnings:<\/strong><\/p>\n<ul>\n<li>Use CDN for global content delivery<\/li>\n<li>Adaptive streaming based on network conditions<\/li>\n<li>Pre-positioning content close to users<\/li>\n<li>Chaos engineering for resilience testing<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"42-uber-tech-stack\"><span>42. Uber Tech Stack<\/span>\n <a href=\"#42-uber-tech-stack\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Uber&rsquo;s platform handles millions of real-time ride requests globally, requiring sub-second driver matching, live GPS tracking, dynamic pricing, and payment processing at planetary scale.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/uber-architecture.webp\" title=\"Uber Architecture \u2014 Real-time dispatch, geospatial driver matching, and trip lifecycle management for on-demand transportation\" data-thumbnail=\"\/files\/uber-architecture.webp\" data-sub-html=\"<h2>Uber real-time matching and routing system architecture \u2014 geospatial matching for ride-sharing at planet scale<\/h2><p>Uber Architecture \u2014 Real-time dispatch, geospatial driver matching, and trip lifecycle management for on-demand transportation<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/uber-architecture.webp' alt=\"Uber real-time matching and routing system architecture \u2014 geospatial matching for ride-sharing at planet scale\"><\/a><figcaption class=\"image-caption\">Uber Architecture \u2014 Real-time dispatch, geospatial driver matching, and trip lifecycle management for on-demand transportation<\/figcaption>\n <\/figure><\/p>\n<p><strong>Key Architectural Highlights:<\/strong><\/p>\n<ul>\n<li><strong>Dispatch service<\/strong> uses geospatial indexing (Google S2 library) to find nearby drivers in O(log n)<\/li>\n<li><strong>Supply\/Demand service<\/strong> dynamically adjusts surge pricing based on driver\/rider ratios per hex cell<\/li>\n<li><strong>Schemaless<\/strong> (NoSQL layer on MySQL) stores trip data with flexible schemas<\/li>\n<li><strong>Apache Kafka<\/strong> powers the real-time event pipeline \u2014 location updates, trip events, payments<\/li>\n<li><strong>Redis<\/strong> stores driver locations with 5-second TTL; lookup by geohash prefix<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"421-dispatch-system\"><span>42.1 Dispatch System<\/span>\n <a href=\"#421-dispatch-system\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Core of Uber&rsquo;s platform matching riders with drivers:<\/p>\n<pre><code>import heapq\nfrom dataclasses import dataclass\nfrom typing import Tuple\n\n@dataclass\nclass Driver:\n id: str\n location: Tuple[float, float] # (lat, lon)\n available: bool\n\n@dataclass\nclass RideRequest:\n rider_id: str\n pickup_location: Tuple[float, float]\n dropoff_location: Tuple[float, float]\n\nclass UberDispatchSystem:\n def __init__(self):\n self.available_drivers = [] # Available drivers\n self.pending_requests = [] # Ride requests queue\n \n def add_driver(self, driver):\n &#34;&#34;&#34;Add driver to available pool&#34;&#34;&#34;\n if driver.available:\n self.available_drivers.append(driver)\n \n def request_ride(self, request):\n &#34;&#34;&#34;Find nearest available driver&#34;&#34;&#34;\n if not self.available_drivers:\n return None\n \n # Find k nearest drivers\n nearest_drivers = self._find_nearest_drivers(\n request.pickup_location,\n k=5\n )\n \n if not nearest_drivers:\n return None\n \n # Assign to nearest available driver\n driver = nearest_drivers[0]\n driver.available = False\n self.available_drivers.remove(driver)\n \n return {\n &#39;driver_id&#39;: driver.id,\n &#39;driver_location&#39;: driver.location,\n &#39;eta&#39;: self._calculate_eta(driver.location, request.pickup_location)\n }\n \n def _find_nearest_drivers(self, location, k=5):\n &#34;&#34;&#34;Find k nearest available drivers&#34;&#34;&#34;\n distances = []\n \n for driver in self.available_drivers:\n dist = self._haversine_distance(location, driver.location)\n distances.append((dist, driver))\n \n # Sort by distance and return top k\n distances.sort(key=lambda x: x[0])\n return [driver for _, driver in distances[:k]]\n \n def _haversine_distance(self, loc1, loc2):\n &#34;&#34;&#34;Calculate distance between two points&#34;&#34;&#34;\n import math\n lat1, lon1 = loc1\n lat2, lon2 = loc2\n \n R = 6371 # Earth radius in km\n \n dlat = math.radians(lat2 - lat1)\n dlon = math.radians(lon2 - lon1)\n \n a = (math.sin(dlat\/2)**2 &#43; \n math.cos(math.radians(lat1)) * math.cos(math.radians(lat2)) * \n math.sin(dlon\/2)**2)\n \n c = 2 * math.atan2(math.sqrt(a), math.sqrt(1-a))\n \n return R * c\n \n def _calculate_eta(self, from_loc, to_loc):\n &#34;&#34;&#34;Estimate time to arrival&#34;&#34;&#34;\n distance = self._haversine_distance(from_loc, to_loc)\n avg_speed = 40 # km\/h in city\n return distance \/ avg_speed * 60 # minutes<\/code><\/pre><p><strong>Key Technologies:<\/strong><\/p>\n<ul>\n<li>Microservices architecture<\/li>\n<li>Cassandra for trip data<\/li>\n<li>Redis for real-time driver locations<\/li>\n<li>Apache Kafka for event streaming<\/li>\n<li>Geospatial indexing for driver matching<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"43-how-discord-stores-trillions-of-messages\"><span>43. How Discord Stores Trillions of Messages<\/span>\n <a href=\"#43-how-discord-stores-trillions-of-messages\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Discord serves 19+ million active servers with hundreds of billions of messages. Their storage architecture evolved multiple times as scale grew \u2014 documented in their engineering blog posts: <a href=\"https:\/\/discord.com\/blog\/how-discord-stores-billions-of-messages\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">How Discord Stores Billions of Messages<\/a> and <a href=\"https:\/\/discord.com\/blog\/how-discord-stores-trillions-of-messages\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">How Discord Stores Trillions of Messages<\/a>.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/discord-message-storage.webp\" title=\"Discord Message Storage \u2014 How Discord stores trillions of messages using Cassandra\/ScyllaDB with channel-based partitioning\" data-thumbnail=\"\/files\/discord-message-storage.webp\" data-sub-html=\"<h2>Discord message storage architecture \u2014 Cassandra-based partitioned storage with ScyllaDB migration for billions of messages<\/h2><p>Discord Message Storage \u2014 How Discord stores trillions of messages using Cassandra\/ScyllaDB with channel-based partitioning<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/discord-message-storage.webp' alt=\"Discord message storage architecture \u2014 Cassandra-based partitioned storage with ScyllaDB migration for billions of messages\"><\/a><figcaption class=\"image-caption\">Discord Message Storage \u2014 How Discord stores trillions of messages using Cassandra\/ScyllaDB with channel-based partitioning<\/figcaption>\n <\/figure><\/p>\n<p><strong>Evolution Timeline:<\/strong><\/p>\n<ul>\n<li><strong>2015<\/strong> \u2014 MongoDB (single replica set). Hit 100 million stored messages, queries slowing.<\/li>\n<li><strong>2017<\/strong> \u2014 Migrated to <a href=\"https:\/\/discord.com\/blog\/how-discord-stores-billions-of-messages\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Cassandra<\/a>. Better write throughput, partition by <code>channel_id<\/code>, cluster by <code>message_id<\/code> (Snowflake ID \u2014 time-sorted).<\/li>\n<li><strong>2022<\/strong> \u2014 Migrated from Cassandra to <a href=\"https:\/\/discord.com\/blog\/how-discord-stores-trillions-of-messages\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ScyllaDB<\/a>. ScyllaDB is a C++ reimplementation of Cassandra with better tail latency and memory management.<\/li>\n<\/ul>\n<p><strong>Key Design Insight:<\/strong> Snowflake IDs as the clustering key means messages are automatically time-ordered within each partition. Pagination (load older messages) is a simple <code>WHERE message_id &lt; $cursor<\/code> range scan \u2014 extremely efficient.<\/p>\n<p><strong>Hot Partition Problem:<\/strong> Popular channels generate far more writes than others. Discord solved this with <a href=\"https:\/\/discord.com\/blog\/how-discord-stores-trillions-of-messages\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">data service<\/a> workers that batch and coalesce writes.<\/p>\n<h3 class=\"heading-element\" id=\"431-data-model-evolution\"><span>43.1 Data Model Evolution<\/span>\n <a href=\"#431-data-model-evolution\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Discord evolved from MongoDB to Cassandra to handle scale:<\/p>\n<pre><code># Cassandra schema for messages\nclass DiscordMessageStorage:\n &#34;&#34;&#34;\n Cassandra table design:\n \n CREATE TABLE messages (\n channel_id bigint,\n message_id bigint,\n author_id bigint,\n content text,\n timestamp timestamp,\n PRIMARY KEY ((channel_id), message_id)\n ) WITH CLUSTERING ORDER BY (message_id DESC);\n &#34;&#34;&#34;\n \n def __init__(self):\n self.message_buckets = {} # Simplified storage\n \n def store_message(self, channel_id, message_id, author_id, content):\n &#34;&#34;&#34;Store message&#34;&#34;&#34;\n if channel_id not in self.message_buckets:\n self.message_buckets[channel_id] = []\n \n message = {\n &#39;message_id&#39;: message_id,\n &#39;author_id&#39;: author_id,\n &#39;content&#39;: content,\n &#39;timestamp&#39;: __import__(&#39;time&#39;).time()\n }\n \n # Insert at beginning (newest first)\n self.message_buckets[channel_id].insert(0, message)\n \n def get_recent_messages(self, channel_id, limit=50):\n &#34;&#34;&#34;Get most recent messages&#34;&#34;&#34;\n if channel_id not in self.message_buckets:\n return []\n \n return self.message_buckets[channel_id][:limit]\n \n def get_messages_before(self, channel_id, before_id, limit=50):\n &#34;&#34;&#34;Get messages before a message_id (pagination)&#34;&#34;&#34;\n if channel_id not in self.message_buckets:\n return []\n \n messages = self.message_buckets[channel_id]\n \n # Find messages before the given message_id\n result = []\n for msg in messages:\n if msg[&#39;message_id&#39;] &lt; before_id:\n result.append(msg)\n if len(result) &gt;= limit:\n break\n \n return result<\/code><\/pre><p><strong>Key Learnings:<\/strong><\/p>\n<ul>\n<li>Use partition key = channel_id for efficient queries<\/li>\n<li>Clustering key = message_id for sorting<\/li>\n<li>Hot partition problem solved with proper sharding<\/li>\n<li>Cassandra chosen for write-heavy workload<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"44-how-kafka-achieves-high-throughput\"><span>44. How Kafka Achieves High Throughput<\/span>\n <a href=\"#44-how-kafka-achieves-high-throughput\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Apache Kafka was <a href=\"https:\/\/www.linkedin.com\/blog\/engineering\/open-source\/kafka-linkedin-data-systems-and-software-engineering\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">originally built at LinkedIn<\/a> to handle their activity stream data, then open-sourced. The <a href=\"https:\/\/kafka.apache.org\/documentation\/#design\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">official Kafka design docs<\/a> explain the architectural decisions that enable 2M+ writes\/second per broker.<\/p>\n<h3 class=\"heading-element\" id=\"441-architecture-principles\"><span>44.1 Architecture Principles<\/span>\n <a href=\"#441-architecture-principles\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/kafka-high-throughput.webp\" title=\"Kafka Architecture \u2014 High-throughput distributed message streaming for real-time data processing\" data-thumbnail=\"\/files\/kafka-high-throughput.webp\" data-sub-html=\"<h2>Apache Kafka high throughput streaming architecture \u2014 distributed event streaming for real-time data pipelines<\/h2><p>Kafka Architecture \u2014 High-throughput distributed message streaming for real-time data processing<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/kafka-high-throughput.webp' alt=\"Apache Kafka high throughput streaming architecture \u2014 distributed event streaming for real-time data pipelines\"><\/a><figcaption class=\"image-caption\">Kafka Architecture \u2014 High-throughput distributed message streaming for real-time data processing<\/figcaption>\n <\/figure><\/p>\n<pre><code>class KafkaOptimizations:\n &#34;&#34;&#34;\n Key optimizations that make Kafka fast:\n &#34;&#34;&#34;\n \n def sequential_io(self):\n &#34;&#34;&#34;\n 1. Sequential I\/O\n - Kafka writes to disk sequentially\n - Much faster than random I\/O\n - Modern disks: sequential writes = memory writes\n &#34;&#34;&#34;\n pass\n \n def zero_copy(self):\n &#34;&#34;&#34;\n 2. Zero-Copy Technology\n - Data transferred directly from file to socket\n - No copying to application memory\n - Uses sendfile() system call\n &#34;&#34;&#34;\n # Traditional: Disk -&gt; Kernel -&gt; User Space -&gt; Kernel -&gt; Socket\n # Zero-copy: Disk -&gt; Kernel -&gt; Socket\n pass\n \n def batch_compression(self):\n &#34;&#34;&#34;\n 3. Batch Compression\n - Compress multiple messages together\n - Better compression ratio\n - Reduces network bandwidth\n &#34;&#34;&#34;\n messages = [&#34;msg1&#34;, &#34;msg2&#34;, &#34;msg3&#34;]\n compressed_batch = self.compress(messages)\n return compressed_batch\n \n def page_cache(self):\n &#34;&#34;&#34;\n 4. Page Cache\n - Relies on OS page cache\n - Recently written\/read data cached in memory\n - No need for application-level cache\n &#34;&#34;&#34;\n pass<\/code><\/pre><p><strong>Performance Numbers:<\/strong><\/p>\n<ul>\n<li>2 million writes\/second per server<\/li>\n<li>Hundreds of MB\/s throughput<\/li>\n<li>Sub-millisecond latency<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"45-instagrams-scaling-journey\"><span>45. Instagram&rsquo;s Scaling Journey<\/span>\n <a href=\"#45-instagrams-scaling-journey\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Instagram launched in 2010, hit 1 million users in 3 months, and was acquired by Facebook in 2012 with just 13 engineers. Its architecture is a textbook case in pragmatic scaling. Key posts: <a href=\"https:\/\/instagram-engineering.com\/sharding-ids-at-instagram-1cf5a71e5a5c\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Sharding &amp; IDs at Instagram<\/a>, <a href=\"https:\/\/instagram-engineering.com\/what-powers-instagram-hundreds-of-instances-dozens-of-technologies-adf2e22da2ad\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">What Powers Instagram<\/a>.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/instagram-scaling-journey.webp\" title=\"Instagram Scaling \u2014 Architecture evolution from startup to 2 billion users: Django, PostgreSQL, Redis, Cassandra, and global CDN\" data-thumbnail=\"\/files\/instagram-scaling-journey.webp\" data-sub-html=\"<h2>Instagram scaling journey and architecture evolution \u2014 PostgreSQL sharding, Cassandra, Redis, and CDN at massive scale<\/h2><p>Instagram Scaling \u2014 Architecture evolution from startup to 2 billion users: Django, PostgreSQL, Redis, Cassandra, and global CDN<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/instagram-scaling-journey.webp' alt=\"Instagram scaling journey and architecture evolution \u2014 PostgreSQL sharding, Cassandra, Redis, and CDN at massive scale\"><\/a><figcaption class=\"image-caption\">Instagram Scaling \u2014 Architecture evolution from startup to 2 billion users: Django, PostgreSQL, Redis, Cassandra, and global CDN<\/figcaption>\n <\/figure><\/p>\n<p><strong>Key Architecture Decisions:<\/strong><\/p>\n<ul>\n<li><strong>Python\/Django<\/strong> throughout \u2014 optimized existing code rather than rewriting in a &ldquo;faster&rdquo; language (<a href=\"https:\/\/instagram-engineering.com\/django-instagram-9024a38eb1af\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">engineering post<\/a>)<\/li>\n<li><strong><a href=\"https:\/\/instagram-engineering.com\/sharding-ids-at-instagram-1cf5a71e5a5c\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">PostgreSQL with custom sharding<\/a><\/strong> \u2014 12 logical shards, Snowflake-style IDs embedding shard number<\/li>\n<li><strong>Redis<\/strong> for follower\/following graph, activity feeds, and session data \u2014 serving ~100M items from memory<\/li>\n<li><strong><a href=\"https:\/\/instagram-engineering.com\/building-for-a-billion-users-7770fb609c2\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Cassandra<\/a><\/strong> for user relationship data at scale<\/li>\n<li><strong>CloudFront CDN<\/strong> for photos \u2014 media delivery accounts for 99%+ of egress bandwidth<\/li>\n<li><strong>Celery + RabbitMQ<\/strong> for async tasks (notifications, feed updates)<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"451-feed-ranking-algorithm\"><span>45.1 Feed Ranking Algorithm<\/span>\n <a href=\"#451-feed-ranking-algorithm\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>class InstagramFeedRanking:\n &#34;&#34;&#34;\n Instagram&#39;s feed ranking considers multiple signals\n &#34;&#34;&#34;\n \n def __init__(self):\n self.weights = {\n &#39;recency&#39;: 0.3,\n &#39;relationship&#39;: 0.25,\n &#39;engagement&#39;: 0.25,\n &#39;content_type&#39;: 0.2\n }\n \n def calculate_score(self, post, user):\n &#34;&#34;&#34;Calculate post relevance score&#34;&#34;&#34;\n # Recency score\n time_decay = self._time_decay_score(post.created_at)\n \n # Relationship score\n relationship_score = self._relationship_strength(user, post.author)\n \n # Engagement score\n engagement_score = self._engagement_probability(post, user)\n \n # Content type preference\n content_score = self._content_type_score(post, user)\n \n # Weighted sum\n total_score = (\n self.weights[&#39;recency&#39;] * time_decay &#43;\n self.weights[&#39;relationship&#39;] * relationship_score &#43;\n self.weights[&#39;engagement&#39;] * engagement_score &#43;\n self.weights[&#39;content_type&#39;] * content_score\n )\n \n return total_score\n \n def _time_decay_score(self, created_at):\n &#34;&#34;&#34;Recent posts score higher&#34;&#34;&#34;\n import time\n age_hours = (time.time() - created_at) \/ 3600\n return 1.0 \/ (1.0 &#43; age_hours)\n \n def _relationship_strength(self, user, author):\n &#34;&#34;&#34;How close is the user to the author&#34;&#34;&#34;\n # Factors: follows, DMs, likes, comments\n return 0.8 # Placeholder\n \n def _engagement_probability(self, post, user):\n &#34;&#34;&#34;Likelihood user will engage&#34;&#34;&#34;\n # ML model predicts like\/comment probability\n return 0.7 # Placeholder\n \n def _content_type_score(self, post, user):\n &#34;&#34;&#34;User preference for content type&#34;&#34;&#34;\n # Photo vs video vs carousel preference\n return 0.6 # Placeholder<\/code><\/pre><p><strong>Scaling Techniques:<\/strong><\/p>\n<ul>\n<li>PostgreSQL with custom sharding<\/li>\n<li>Redis for feed caching<\/li>\n<li>Cassandra for user relationships<\/li>\n<li>memcached for general caching<\/li>\n<li>Horizontal scaling of web servers<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"22-solid-principles\"><span>22. SOLID Principles<\/span>\n <a href=\"#22-solid-principles\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>SOLID principles are five design principles that help make software more maintainable, flexible, and scalable.<\/p>\n<h3 class=\"heading-element\" id=\"221-single-responsibility-principle-srp\"><span>22.1 Single Responsibility Principle (SRP)<\/span>\n <a href=\"#221-single-responsibility-principle-srp\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Definition:<\/strong> A class should have only one reason to change.<\/p>\n<p><strong>Bad Example:<\/strong><\/p>\n<pre><code># Violates SRP - User class handles too much\nclass User:\n def __init__(self, name, email):\n self.name = name\n self.email = email\n \n def save_to_database(self):\n # Database logic here\n print(f&#34;Saving {self.name} to database&#34;)\n \n def send_email(self):\n # Email logic here\n print(f&#34;Sending email to {self.email}&#34;)<\/code><\/pre><p><strong>Good Example:<\/strong><\/p>\n<pre><code># Each class has ONE responsibility\nclass User:\n def __init__(self, name, email):\n self.name = name\n self.email = email\n\nclass UserRepository:\n def save(self, user):\n print(f&#34;Saving {user.name} to database&#34;)\n\nclass EmailService:\n def send(self, user):\n print(f&#34;Sending email to {user.email}&#34;)\n\n# Usage\nuser = User(&#34;Alice&#34;, &#34;alice@example.com&#34;)\nUserRepository().save(user)\nEmailService().send(user)<\/code><\/pre><p><strong>Why it matters:<\/strong><\/p>\n<ul>\n<li>Easy to test each class<\/li>\n<li>Changes in database don&rsquo;t affect email logic<\/li>\n<li>Clear separation of concerns<\/li>\n<\/ul>\n<hr>\n<h3 class=\"heading-element\" id=\"222-openclosed-principle-ocp\"><span>22.2 Open\/Closed Principle (OCP)<\/span>\n <a href=\"#222-openclosed-principle-ocp\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Definition:<\/strong> Classes should be open for extension, closed for modification.<\/p>\n<p><strong>Bad Example:<\/strong><\/p>\n<pre><code># Violates OCP - must modify for new discounts\nclass Discount:\n def calculate(self, customer_type, price):\n if customer_type == &#34;regular&#34;:\n return price * 0.1\n elif customer_type == &#34;vip&#34;:\n return price * 0.2\n elif customer_type == &#34;premium&#34;: # Need to modify!\n return price * 0.3<\/code><\/pre><p><strong>Good Example:<\/strong><\/p>\n<pre><code># Open for extension, closed for modification\nclass Discount:\n def calculate(self, price):\n return price * 0.1\n\nclass VIPDiscount(Discount):\n def calculate(self, price):\n return price * 0.2\n\nclass PremiumDiscount(Discount):\n def calculate(self, price):\n return price * 0.3\n\n# Usage - no modification needed\ndef apply_discount(discount: Discount, price: float):\n return price - discount.calculate(price)\n\nprint(apply_discount(VIPDiscount(), 100)) # 80\nprint(apply_discount(PremiumDiscount(), 100)) # 70<\/code><\/pre><p><strong>Why it matters:<\/strong><\/p>\n<ul>\n<li>Add new features without changing existing code<\/li>\n<li>Reduces risk of breaking existing functionality<\/li>\n<li>Easy to extend with new discount types<\/li>\n<\/ul>\n<hr>\n<h3 class=\"heading-element\" id=\"223-liskov-substitution-principle-lsp\"><span>22.3 Liskov Substitution Principle (LSP)<\/span>\n <a href=\"#223-liskov-substitution-principle-lsp\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Definition:<\/strong> Subclasses should be substitutable for their parent classes.<\/p>\n<p><strong>Bad Example:<\/strong><\/p>\n<pre><code># Violates LSP - Square changes Rectangle behavior\nclass Rectangle:\n def __init__(self, width, height):\n self.width = width\n self.height = height\n \n def set_width(self, width):\n self.width = width\n \n def set_height(self, height):\n self.height = height\n \n def area(self):\n return self.width * self.height\n\nclass Square(Rectangle):\n def set_width(self, width):\n self.width = width\n self.height = width # Changes both!\n \n def set_height(self, height):\n self.width = height # Changes both!\n\n# Problem:\nrect = Rectangle(5, 10)\nrect.set_width(3)\nprint(rect.area()) # 30 \u2713\n\nsquare = Square(5, 5)\nsquare.set_width(3)\nprint(square.area()) # 9 (unexpected if treated as Rectangle)<\/code><\/pre><p><strong>Good Example:<\/strong><\/p>\n<pre><code># Correct - use composition instead\nclass Shape:\n def area(self):\n raise NotImplementedError\n\nclass Rectangle(Shape):\n def __init__(self, width, height):\n self.width = width\n self.height = height\n \n def area(self):\n return self.width * self.height\n\nclass Square(Shape):\n def __init__(self, side):\n self.side = side\n \n def area(self):\n return self.side * self.side\n\n# Works correctly\nshapes = [Rectangle(5, 10), Square(5)]\nfor shape in shapes:\n print(shape.area()) # 50, 25<\/code><\/pre><p><strong>Why it matters:<\/strong><\/p>\n<ul>\n<li>Prevents unexpected behavior<\/li>\n<li>Code works correctly with any subclass<\/li>\n<li>Makes inheritance more reliable<\/li>\n<\/ul>\n<hr>\n<h3 class=\"heading-element\" id=\"224-interface-segregation-principle-isp\"><span>22.4 Interface Segregation Principle (ISP)<\/span>\n <a href=\"#224-interface-segregation-principle-isp\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Definition:<\/strong> Clients shouldn&rsquo;t depend on methods they don&rsquo;t use.<\/p>\n<p><strong>Bad Example:<\/strong><\/p>\n<pre><code># Violates ISP - forces unnecessary methods\nclass Worker:\n def work(self):\n raise NotImplementedError\n \n def eat(self):\n raise NotImplementedError\n\nclass Human(Worker):\n def work(self):\n print(&#34;Working&#34;)\n \n def eat(self):\n print(&#34;Eating lunch&#34;)\n\nclass Robot(Worker):\n def work(self):\n print(&#34;Working&#34;)\n \n def eat(self):\n pass # Robots don&#39;t eat! Forced to implement<\/code><\/pre><p><strong>Good Example:<\/strong><\/p>\n<pre><code># Split into specific interfaces\nclass Workable:\n def work(self):\n raise NotImplementedError\n\nclass Eatable:\n def eat(self):\n raise NotImplementedError\n\nclass Human(Workable, Eatable):\n def work(self):\n print(&#34;Working&#34;)\n \n def eat(self):\n print(&#34;Eating lunch&#34;)\n\nclass Robot(Workable):\n def work(self):\n print(&#34;Working&#34;)\n # No eat() method needed!<\/code><\/pre><p><strong>Why it matters:<\/strong><\/p>\n<ul>\n<li>Classes only implement what they need<\/li>\n<li>More flexible and maintainable<\/li>\n<li>Clear interface contracts<\/li>\n<\/ul>\n<hr>\n<h3 class=\"heading-element\" id=\"225-dependency-inversion-principle-dip\"><span>22.5 Dependency Inversion Principle (DIP)<\/span>\n <a href=\"#225-dependency-inversion-principle-dip\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Definition:<\/strong> Depend on abstractions, not concrete implementations.<\/p>\n<p><strong>Bad Example:<\/strong><\/p>\n<pre><code># Violates DIP - depends on concrete class\nclass MySQLDatabase:\n def save(self, data):\n print(f&#34;Saving to MySQL: {data}&#34;)\n\nclass UserService:\n def __init__(self):\n self.db = MySQLDatabase() # Tightly coupled!\n \n def create_user(self, user):\n self.db.save(user)\n\n# Problem: Can&#39;t easily switch to PostgreSQL<\/code><\/pre><p><strong>Good Example:<\/strong><\/p>\n<pre><code># Depend on abstraction\nfrom abc import ABC, abstractmethod\n\nclass Database(ABC):\n @abstractmethod\n def save(self, data):\n pass\n\nclass MySQLDatabase(Database):\n def save(self, data):\n print(f&#34;Saving to MySQL: {data}&#34;)\n\nclass PostgreSQLDatabase(Database):\n def save(self, data):\n print(f&#34;Saving to PostgreSQL: {data}&#34;)\n\nclass UserService:\n def __init__(self, database: Database):\n self.db = database # Depends on abstraction\n \n def create_user(self, user):\n self.db.save(user)\n\n# Easy to switch implementations\nservice1 = UserService(MySQLDatabase())\nservice2 = UserService(PostgreSQLDatabase())<\/code><\/pre><p><strong>Why it matters:<\/strong><\/p>\n<ul>\n<li>Easy to swap implementations<\/li>\n<li>Better for testing (can use mock database)<\/li>\n<li>Loose coupling between components<\/li>\n<\/ul>\n<hr>\n<h3 class=\"heading-element\" id=\"226-solid-principles-summary\"><span>22.6 SOLID Principles Summary<\/span>\n <a href=\"#226-solid-principles-summary\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Principle<\/th>\n <th>Key Idea<\/th>\n <th>Benefit<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>S<\/strong>RP<\/td>\n <td>One class, one job<\/td>\n <td>Easy to maintain<\/td>\n <\/tr>\n <tr>\n <td><strong>O<\/strong>CP<\/td>\n <td>Extend, don&rsquo;t modify<\/td>\n <td>Safe to add features<\/td>\n <\/tr>\n <tr>\n <td><strong>L<\/strong>SP<\/td>\n <td>Subclass = Parent<\/td>\n <td>Reliable inheritance<\/td>\n <\/tr>\n <tr>\n <td><strong>I<\/strong>SP<\/td>\n <td>Small, focused interfaces<\/td>\n <td>No unused methods<\/td>\n <\/tr>\n <tr>\n <td><strong>D<\/strong>IP<\/td>\n <td>Depend on abstractions<\/td>\n <td>Flexible architecture<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"real-world-application\"><span>Real-World Application<\/span>\n <a href=\"#real-world-application\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Complete example using all SOLID principles\n\nfrom abc import ABC, abstractmethod\n\n# DIP - Abstract payment interface\nclass PaymentProcessor(ABC):\n @abstractmethod\n def process(self, amount):\n pass\n\n# ISP - Separate interfaces\nclass Refundable(ABC):\n @abstractmethod\n def refund(self, amount):\n pass\n\n# OCP - Extensible payment types\nclass StripePayment(PaymentProcessor, Refundable):\n def process(self, amount):\n return f&#34;Stripe processed ${amount}&#34;\n \n def refund(self, amount):\n return f&#34;Stripe refunded ${amount}&#34;\n\nclass PayPalPayment(PaymentProcessor):\n def process(self, amount):\n return f&#34;PayPal processed ${amount}&#34;\n # No refund - some payment methods don&#39;t support it\n\n# SRP - Each class has one responsibility\nclass Order:\n def __init__(self, items, total):\n self.items = items\n self.total = total\n\nclass OrderProcessor:\n def __init__(self, payment: PaymentProcessor):\n self.payment = payment\n \n def process_order(self, order: Order):\n result = self.payment.process(order.total)\n print(result)\n return True\n\n# LSP - Can substitute any PaymentProcessor\nprocessors = [\n OrderProcessor(StripePayment()),\n OrderProcessor(PayPalPayment())\n]\n\norder = Order([&#34;item1&#34;, &#34;item2&#34;], 100)\nfor processor in processors:\n processor.process_order(order) # Works with any processor<\/code><\/pre><p><strong>When to Use SOLID:<\/strong><\/p>\n<ul>\n<li>\u2705 Building large applications<\/li>\n<li>\u2705 Code that will be maintained long-term<\/li>\n<li>\u2705 Teams working on same codebase<\/li>\n<li>\u2705 When you need flexibility<\/li>\n<\/ul>\n<p><strong>When to Skip:<\/strong><\/p>\n<ul>\n<li>\u26a0\ufe0f Small scripts or prototypes<\/li>\n<li>\u26a0\ufe0f One-time use code<\/li>\n<li>\u26a0\ufe0f Very simple applications<\/li>\n<\/ul>\n<p>Remember: SOLID principles guide good design but aren&rsquo;t absolute rules. Use judgment based on your project&rsquo;s needs.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"additional-real-world-system-designs\"><span>Additional Real-World System Designs<\/span>\n <a href=\"#additional-real-world-system-designs\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h2 class=\"heading-element\" id=\"36-design-pastebin-text-sharing-service\"><span>36. Design Pastebin (Text Sharing Service)<\/span>\n <a href=\"#36-design-pastebin-text-sharing-service\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Pastebin allows users to share text snippets via short URLs.<\/p>\n<h3 class=\"heading-element\" id=\"361-requirements\"><span>36.1 Requirements<\/span>\n <a href=\"#361-requirements\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Functional:<\/strong><\/p>\n<ul>\n<li>Users paste text and get a unique short URL<\/li>\n<li>Users access URL to view content<\/li>\n<li>Optional expiration time<\/li>\n<li>Track view statistics<\/li>\n<\/ul>\n<p><strong>Non-Functional:<\/strong><\/p>\n<ul>\n<li>High availability<\/li>\n<li>Low latency for URL generation<\/li>\n<li>Handle 10M users, 10M writes\/month, 100M reads\/month<\/li>\n<li>10:1 read-to-write ratio<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"362-capacity-estimation\"><span>36.2 Capacity Estimation<\/span>\n <a href=\"#362-capacity-estimation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Storage:<\/strong><\/p>\n<pre><code>Average paste size: 1 KB\nMetadata: ~270 bytes (URL, expiration, timestamps)\nTotal per paste: ~1.27 KB\n\nMonthly storage: 1.27 KB \u00d7 10M = 12.7 GB\/month\n3-year storage: 12.7 GB \u00d7 36 = 457 GB<\/code><\/pre><p><strong>Traffic:<\/strong><\/p>\n<pre><code>Write QPS: 10M\/month \u00f7 2.5M seconds = 4 writes\/second\nRead QPS: 100M\/month \u00f7 2.5M seconds = 40 reads\/second\nPeak: 2\u00d7 average = 8 writes\/sec, 80 reads\/sec<\/code><\/pre><h3 class=\"heading-element\" id=\"363-high-level-design\"><span>36.3 High-Level Design<\/span>\n <a href=\"#363-high-level-design\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/url-shortener-architecture.webp\" title=\"Pastebin \/ URL Shortener Architecture \u2014 System design for high-traffic text sharing and link shortening with caching and analytics\" data-thumbnail=\"\/files\/url-shortener-architecture.webp\" data-sub-html=\"<h2>URL shortener service system design architecture \u2014 encoding, caching, and redirect optimization for link shortening<\/h2><p>Pastebin \/ URL Shortener Architecture \u2014 System design for high-traffic text sharing and link shortening with caching and analytics<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/url-shortener-architecture.webp' alt=\"URL shortener service system design architecture \u2014 encoding, caching, and redirect optimization for link shortening\"><\/a><figcaption class=\"image-caption\">Pastebin \/ URL Shortener Architecture \u2014 System design for high-traffic text sharing and link shortening with caching and analytics<\/figcaption>\n <\/figure><\/p>\n<p><strong>Core Components:<\/strong><\/p>\n<ul>\n<li><strong>Web Servers<\/strong>: Handle paste creation and retrieval requests. Stateless, horizontally scalable.<\/li>\n<li><strong>Short ID Generator<\/strong>: Produces unique 7-character Base62 codes via MD5 hash of content+timestamp, avoiding collisions.<\/li>\n<li><strong>Object Store (S3-compatible)<\/strong>: Stores raw paste content (text blobs). Separate from metadata for cost efficiency at scale.<\/li>\n<li><strong>Metadata Database (SQL)<\/strong>: Stores <code>short_id<\/code>, <code>object_key<\/code>, <code>expires_at<\/code>, <code>view_count<\/code>, <code>created_at<\/code>. PostgreSQL with index on <code>short_id<\/code> and <code>expires_at<\/code>.<\/li>\n<li><strong>Cache (Redis)<\/strong>: LRU cache keyed by <code>paste:&lt;short_id&gt;<\/code> storing the content. Dramatically reduces S3 fetches for hot pastes.<\/li>\n<li><strong>Expiration Worker<\/strong>: Background job periodically deletes expired paste metadata and S3 objects (avoid orphaned storage).<\/li>\n<\/ul>\n<p><strong>Data Flow:<\/strong><\/p>\n<ol>\n<li><strong>Write path<\/strong>: Client \u2192 Web Server \u2192 generate <code>short_id<\/code> \u2192 store content in S3 \u2192 write metadata to DB \u2192 cache the content \u2192 return short URL.<\/li>\n<li><strong>Read path<\/strong>: Client \u2192 Web Server \u2192 check Redis cache \u2192 on miss: query DB for <code>object_key<\/code> \u2192 fetch from S3 \u2192 update cache \u2192 return content.<\/li>\n<\/ol>\n<p><strong>Scaling Considerations:<\/strong><\/p>\n<ul>\n<li>10 million users, 4 writes\/sec, 40 reads\/sec \u2014 a single PostgreSQL instance handles this comfortably.<\/li>\n<li>At 10\u00d7 scale, move to read replicas for the metadata DB.<\/li>\n<li>S3 provides virtually unlimited object storage at low cost.<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"364-implementation\"><span>36.4 Implementation<\/span>\n <a href=\"#364-implementation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>import hashlib\nimport base64\nfrom datetime import datetime, timedelta\n\nclass PastebinService:\n def __init__(self, db, object_store, cache):\n self.db = db\n self.object_store = object_store\n self.cache = cache\n \n def create_paste(self, content, expiration_minutes=None):\n &#34;&#34;&#34;Create a new paste and return short URL&#34;&#34;&#34;\n # Generate unique ID using MD5 &#43; Base62\n unique_str = content &#43; str(datetime.now().timestamp())\n hash_value = hashlib.md5(unique_str.encode()).hexdigest()\n short_id = self._base62_encode(int(hash_value[:16], 16))[:7]\n \n # Store content in object store\n object_key = f&#34;pastes\/{short_id}&#34;\n self.object_store.put(object_key, content)\n \n # Store metadata in database\n paste_metadata = {\n &#39;short_id&#39;: short_id,\n &#39;object_key&#39;: object_key,\n &#39;created_at&#39;: datetime.now(),\n &#39;expires_at&#39;: datetime.now() &#43; timedelta(minutes=expiration_minutes) \n if expiration_minutes else None,\n &#39;size_bytes&#39;: len(content)\n }\n self.db.insert(&#39;pastes&#39;, paste_metadata)\n \n return short_id\n \n def get_paste(self, short_id):\n &#34;&#34;&#34;Retrieve paste content&#34;&#34;&#34;\n # Check cache first\n cache_key = f&#34;paste:{short_id}&#34;\n cached = self.cache.get(cache_key)\n if cached:\n return cached\n \n # Get metadata from database\n metadata = self.db.query(f&#34;SELECT * FROM pastes WHERE short_id = &#39;{short_id}&#39;&#34;)\n if not metadata:\n return None\n \n # Check if expired\n if metadata[&#39;expires_at&#39;] and datetime.now() &gt; metadata[&#39;expires_at&#39;]:\n return None\n \n # Fetch content from object store\n content = self.object_store.get(metadata[&#39;object_key&#39;])\n \n # Cache for future requests\n self.cache.set(cache_key, content, ttl=3600)\n \n # Update view count asynchronously\n self.db.execute(f&#34;UPDATE pastes SET views = views &#43; 1 WHERE short_id = &#39;{short_id}&#39;&#34;)\n \n return content\n \n def _base62_encode(self, num):\n &#34;&#34;&#34;Encode number to Base62&#34;&#34;&#34;\n chars = &#34;0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ&#34;\n if num == 0:\n return chars[0]\n \n result = []\n while num &gt; 0:\n result.append(chars[num % 62])\n num \/\/= 62\n \n return &#39;&#39;.join(reversed(result))\n\n# Usage\nservice = PastebinService(db, s3, redis)\nurl = service.create_paste(&#34;Hello World!&#34;, expiration_minutes=60)\ncontent = service.get_paste(url)<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"37-design-a-web-crawler\"><span>37. Design a Web Crawler<\/span>\n <a href=\"#37-design-a-web-crawler\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>A web crawler systematically browses the web to index pages for search engines.<\/p>\n<h3 class=\"heading-element\" id=\"371-requirements\"><span>37.1 Requirements<\/span>\n <a href=\"#371-requirements\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Functional:<\/strong><\/p>\n<ul>\n<li>Crawl 1 billion links<\/li>\n<li>Generate reverse index (words \u2192 pages)<\/li>\n<li>Extract titles and snippets<\/li>\n<li>Respect robots.txt<\/li>\n<li>Handle duplicate content<\/li>\n<\/ul>\n<p><strong>Non-Functional:<\/strong><\/p>\n<ul>\n<li>Scalable to billions of pages<\/li>\n<li>Fresh content (recrawl weekly)<\/li>\n<li>Avoid infinite loops<\/li>\n<li>Politeness (rate limiting per domain)<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"372-capacity-estimation\"><span>37.2 Capacity Estimation<\/span>\n <a href=\"#372-capacity-estimation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>1 billion links to crawl\nAverage page size: 500 KB\nCrawl frequency: Weekly\nMonthly crawls: 4 billion pages\nMonthly storage: 500 KB \u00d7 4B = 2 PB\/month\n\nWrite QPS: 4B \/ 2.5M seconds = 1,600 writes\/second\nSearch QPS: 100B \/ 2.5M seconds = 40,000 searches\/second<\/code><\/pre><h3 class=\"heading-element\" id=\"373-architecture\"><span>37.3 Architecture<\/span>\n <a href=\"#373-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/data-pipeline-architecture.webp\" title=\"Web Crawler Architecture \u2014 Distributed crawl pipeline: URL frontier, fetchers, parser, dedup filter, and reverse index store\" data-thumbnail=\"\/files\/data-pipeline-architecture.webp\" data-sub-html=\"<h2>Data pipeline and stream processing architecture \u2014 ETL processes for handling large-scale data workflows<\/h2><p>Web Crawler Architecture \u2014 Distributed crawl pipeline: URL frontier, fetchers, parser, dedup filter, and reverse index store<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/data-pipeline-architecture.webp' alt=\"Data pipeline and stream processing architecture \u2014 ETL processes for handling large-scale data workflows\"><\/a><figcaption class=\"image-caption\">Web Crawler Architecture \u2014 Distributed crawl pipeline: URL frontier, fetchers, parser, dedup filter, and reverse index store<\/figcaption>\n <\/figure><\/p>\n<p><strong>Core Components:<\/strong><\/p>\n<ul>\n<li><strong>URL Frontier<\/strong>: Priority queue of URLs to crawl. Implemented as a distributed queue (Kafka or Redis ZSET with priority score). Separates &ldquo;to crawl&rdquo; from &ldquo;crawled&rdquo; sets.<\/li>\n<li><strong>Fetchers (Workers)<\/strong>: Stateless worker processes that pull URLs from the frontier, issue HTTP GET, and pass raw HTML to the parser. Scale horizontally. Respect <code>robots.txt<\/code> and rate-limit per domain.<\/li>\n<li><strong>HTML Parser \/ Link Extractor<\/strong>: Extracts <code>&lt;a href&gt;<\/code> links from fetched pages. Normalizes URLs (strip query params, resolve relative paths).<\/li>\n<li><strong>Deduplication Filter<\/strong>: Uses <a href=\"https:\/\/en.wikipedia.org\/wiki\/SimHash\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">SimHash<\/a> or MD5 of content to detect near-duplicate pages. Bloom filter for fast URL deduplication (seen-URL check before fetching).<\/li>\n<li><strong>Content Store (Object Storage)<\/strong>: Raw HTML stored in S3-compatible store for reprocessing. Partition by domain hash for cost-efficient retrieval.<\/li>\n<li><strong>Reverse Index Builder<\/strong>: Extracts words, computes TF-IDF scores, and writes to an inverted index (Elasticsearch or custom sharded index).<\/li>\n<li><strong>Scheduler<\/strong>: Re-crawls popular pages more frequently (every day), less-popular pages weekly. Uses crawl frequency heuristics based on historical change rate.<\/li>\n<\/ul>\n<p><strong>Design Decisions:<\/strong><\/p>\n<ul>\n<li>Use BFS from seed URLs, with priority weighting by PageRank or domain authority.<\/li>\n<li>Respect <a href=\"https:\/\/en.wikipedia.org\/wiki\/Robots_exclusion_standard\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Crawl-Delay<\/a> from <code>robots.txt<\/code> \u2014 critical for politeness.<\/li>\n<li>Avoid crawl traps (infinite URL generation) by canonicalizing URLs and limiting depth.<\/li>\n<li>For 1 billion links at 1,600 writes\/second: distribute across 10\u201320 fetcher nodes, each handling ~100 QPS.<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"374-implementation\"><span>37.4 Implementation<\/span>\n <a href=\"#374-implementation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>import hashlib\nfrom collections import deque\nfrom urllib.parse import urljoin, urlparse\nimport requests\nfrom bs4 import BeautifulSoup\n\nclass WebCrawler:\n def __init__(self, seed_urls, max_pages=1000):\n self.to_crawl = deque(seed_urls)\n self.crawled = set()\n self.page_signatures = {}\n self.max_pages = max_pages\n self.crawled_count = 0\n \n def crawl(self):\n &#34;&#34;&#34;Main crawling loop&#34;&#34;&#34;\n while self.to_crawl and self.crawled_count &lt; self.max_pages:\n url = self.to_crawl.popleft()\n \n if url in self.crawled:\n continue\n \n try:\n page_data = self.fetch_page(url)\n if not page_data:\n continue\n \n # Check for duplicates\n signature = self.create_signature(page_data[&#39;content&#39;])\n if signature in self.page_signatures:\n print(f&#34;Duplicate content: {url}&#34;)\n continue\n \n self.page_signatures[signature] = url\n self.crawled.add(url)\n self.crawled_count &#43;= 1\n \n # Extract and queue new URLs\n new_urls = self.extract_urls(url, page_data[&#39;content&#39;])\n for new_url in new_urls:\n if new_url not in self.crawled:\n self.to_crawl.append(new_url)\n \n # Process page (index, store, etc.)\n self.process_page(url, page_data)\n \n print(f&#34;Crawled ({self.crawled_count}): {url}&#34;)\n \n except Exception as e:\n print(f&#34;Error crawling {url}: {e}&#34;)\n \n def fetch_page(self, url):\n &#34;&#34;&#34;Fetch page content&#34;&#34;&#34;\n try:\n response = requests.get(url, timeout=5)\n if response.status_code == 200:\n return {\n &#39;url&#39;: url,\n &#39;content&#39;: response.text,\n &#39;headers&#39;: dict(response.headers)\n }\n except:\n return None\n \n def create_signature(self, content):\n &#34;&#34;&#34;Create page signature for duplicate detection&#34;&#34;&#34;\n # Normalize and hash content\n normalized = content.lower().strip()\n return hashlib.md5(normalized.encode()).hexdigest()\n \n def extract_urls(self, base_url, html_content):\n &#34;&#34;&#34;Extract all URLs from page&#34;&#34;&#34;\n soup = BeautifulSoup(html_content, &#39;html.parser&#39;)\n urls = []\n \n for link in soup.find_all(&#39;a&#39;, href=True):\n url = urljoin(base_url, link[&#39;href&#39;])\n \n # Filter only HTTP(S) URLs\n if urlparse(url).scheme in [&#39;http&#39;, &#39;https&#39;]:\n urls.append(url)\n \n return urls\n \n def process_page(self, url, page_data):\n &#34;&#34;&#34;Process crawled page - index, store, etc.&#34;&#34;&#34;\n # Extract title and snippet\n soup = BeautifulSoup(page_data[&#39;content&#39;], &#39;html.parser&#39;)\n title = soup.find(&#39;title&#39;).text if soup.find(&#39;title&#39;) else url\n \n # Create reverse index (word -&gt; URLs)\n words = self.tokenize(page_data[&#39;content&#39;])\n # Store in reverse index...\n \n # Store document\n # Store in document service...\n \n print(f&#34; Title: {title}&#34;)\n print(f&#34; Words: {len(words)}&#34;)\n \n def tokenize(self, text):\n &#34;&#34;&#34;Simple tokenization&#34;&#34;&#34;\n soup = BeautifulSoup(text, &#39;html.parser&#39;)\n text = soup.get_text()\n words = text.lower().split()\n return [w.strip(&#39;.,!?;:()[]{}&#39;) for w in words if len(w) &gt; 2]\n\n# Usage\ncrawler = WebCrawler([&#39;https:\/\/example.com&#39;], max_pages=100)\ncrawler.crawl()<\/code><\/pre><p><strong>Advanced Features:<\/strong><\/p>\n<pre><code>class PoliteCrawler(WebCrawler):\n &#34;&#34;&#34;Crawler with politeness - rate limiting per domain&#34;&#34;&#34;\n \n def __init__(self, seed_urls, max_pages=1000, delay_seconds=1):\n super().__init__(seed_urls, max_pages)\n self.domain_last_crawl = {}\n self.delay_seconds = delay_seconds\n \n def can_crawl_domain(self, url):\n &#34;&#34;&#34;Check if enough time has passed since last crawl&#34;&#34;&#34;\n import time\n domain = urlparse(url).netloc\n last_crawl = self.domain_last_crawl.get(domain, 0)\n \n if time.time() - last_crawl &lt; self.delay_seconds:\n return False\n \n self.domain_last_crawl[domain] = time.time()\n return True\n \n def fetch_page(self, url):\n &#34;&#34;&#34;Fetch with politeness check&#34;&#34;&#34;\n if not self.can_crawl_domain(url):\n return None\n \n return super().fetch_page(url)<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"38-design-mintcom-personal-finance-tracker\"><span>38. Design Mint.com (Personal Finance Tracker)<\/span>\n <a href=\"#38-design-mintcom-personal-finance-tracker\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Mint.com aggregates financial data from multiple sources and provides insights.<\/p>\n<h3 class=\"heading-element\" id=\"381-requirements\"><span>38.1 Requirements<\/span>\n <a href=\"#381-requirements\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Functional:<\/strong><\/p>\n<ul>\n<li>Connect to multiple bank accounts<\/li>\n<li>Automatically categorize transactions<\/li>\n<li>Budget tracking and alerts<\/li>\n<li>Financial insights and trends<\/li>\n<li>Bill reminders<\/li>\n<\/ul>\n<p><strong>Non-Functional:<\/strong><\/p>\n<ul>\n<li>Highly secure (financial data)<\/li>\n<li>Near real-time sync<\/li>\n<li>Support millions of users<\/li>\n<li>99.99% uptime<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"382-architecture\"><span>38.2 Architecture<\/span>\n <a href=\"#382-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/data-pipeline-architecture.webp\" title=\"Mint.com Architecture \u2014 Financial data aggregation pipeline: bank connectors, transaction sync, ML categorization, and budget alerting\" data-thumbnail=\"\/files\/data-pipeline-architecture.webp\" data-sub-html=\"<h2>Data pipeline and stream processing architecture \u2014 ETL processes for handling large-scale data workflows<\/h2><p>Mint.com Architecture \u2014 Financial data aggregation pipeline: bank connectors, transaction sync, ML categorization, and budget alerting<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/data-pipeline-architecture.webp' alt=\"Data pipeline and stream processing architecture \u2014 ETL processes for handling large-scale data workflows\"><\/a><figcaption class=\"image-caption\">Mint.com Architecture \u2014 Financial data aggregation pipeline: bank connectors, transaction sync, ML categorization, and budget alerting<\/figcaption>\n <\/figure><\/p>\n<p><strong>Core Components:<\/strong><\/p>\n<ul>\n<li><strong>Bank Aggregator Integration<\/strong>: Third-party services like <a href=\"https:\/\/plaid.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Plaid<\/a> or Yodlee act as intermediaries that connect to thousands of banks via OAuth \/ credential-based scraping. Mint never stores bank credentials directly.<\/li>\n<li><strong>Transaction Sync Service<\/strong>: Scheduled pull (every 4\u20136 hours) from aggregator API. Stores new transactions in PostgreSQL partitioned by <code>user_id<\/code> and <code>account_id<\/code>.<\/li>\n<li><strong>Transaction Categorization Engine<\/strong>: ML model (Naive Bayes \/ gradient boosting) classifies transactions based on merchant name and description. Falls back to user-defined rules.<\/li>\n<li><strong>Budget Alerting Service<\/strong>: Pub\/Sub consumer that aggregates monthly spend per category and compares against user-defined budgets. Emits alert events when thresholds exceeded.<\/li>\n<li><strong>Notification Service<\/strong>: Delivers alerts via push notification, email, or SMS (similar architecture to \u00a735).<\/li>\n<li><strong>Read API<\/strong>: Serves transaction history, spending trends, and net worth calculations to the frontend. Uses Redis for caching aggregated summaries.<\/li>\n<li><strong>Encryption Layer<\/strong>: All bank credentials (if stored at all) are encrypted at rest using AES-256. PCI DSS compliance required.<\/li>\n<\/ul>\n<p><strong>Data Flow:<\/strong><\/p>\n<ol>\n<li>User links bank account \u2192 OAuth flow with Plaid \u2192 Mint receives access token.<\/li>\n<li>Sync worker polls Plaid every 6 hours \u2192 new transactions land in DB.<\/li>\n<li>Categorization engine processes uncategorized transactions \u2192 writes category back.<\/li>\n<li>Budget checker runs after each sync \u2192 publishes alert events if budget exceeded.<\/li>\n<li>Notification service delivers alerts to user devices.<\/li>\n<\/ol>\n<h3 class=\"heading-element\" id=\"383-key-components\"><span>38.3 Key Components<\/span>\n <a href=\"#383-key-components\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Transaction Categorization (ML-based):<\/strong><\/p>\n<pre><code>from sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.naive_bayes import MultinomialNB\n\nclass TransactionCategorizer:\n def __init__(self):\n self.vectorizer = TfidfVectorizer(max_features=1000)\n self.classifier = MultinomialNB()\n self.categories = [\n &#39;Groceries&#39;, &#39;Restaurants&#39;, &#39;Transportation&#39;,\n &#39;Entertainment&#39;, &#39;Bills&#39;, &#39;Shopping&#39;, &#39;Healthcare&#39;,\n &#39;Travel&#39;, &#39;Income&#39;, &#39;Other&#39;\n ]\n \n def train(self, transactions, labels):\n &#34;&#34;&#34;Train categorizer on labeled data&#34;&#34;&#34;\n descriptions = [t[&#39;description&#39;] for t in transactions]\n X = self.vectorizer.fit_transform(descriptions)\n self.classifier.fit(X, labels)\n \n def categorize(self, transaction):\n &#34;&#34;&#34;Predict category for new transaction&#34;&#34;&#34;\n description = transaction[&#39;description&#39;]\n X = self.vectorizer.transform([description])\n category_idx = self.classifier.predict(X)[0]\n confidence = max(self.classifier.predict_proba(X)[0])\n \n return {\n &#39;category&#39;: self.categories[category_idx],\n &#39;confidence&#39;: confidence\n }\n\nclass MintService:\n def __init__(self):\n self.categorizer = TransactionCategorizer()\n self.accounts = {}\n self.transactions = []\n \n def sync_accounts(self, user_id):\n &#34;&#34;&#34;Sync transactions from all connected accounts&#34;&#34;&#34;\n accounts = self.get_user_accounts(user_id)\n \n new_transactions = []\n for account in accounts:\n # Fetch from bank API (via Plaid\/Yodlee)\n recent_transactions = self.fetch_transactions(account)\n \n for txn in recent_transactions:\n # Categorize automatically\n category_info = self.categorizer.categorize(txn)\n txn[&#39;category&#39;] = category_info[&#39;category&#39;]\n txn[&#39;auto_categorized&#39;] = True\n \n new_transactions.append(txn)\n \n # Store transactions\n self.store_transactions(user_id, new_transactions)\n \n # Check budget alerts\n self.check_budget_alerts(user_id)\n \n return new_transactions\n \n def get_user_accounts(self, user_id):\n &#34;&#34;&#34;Get all linked accounts for user&#34;&#34;&#34;\n return [] # Query from database\n \n def fetch_transactions(self, account):\n &#34;&#34;&#34;Fetch transactions from bank via aggregator&#34;&#34;&#34;\n return [] # Call Plaid\/Yodlee API\n \n def store_transactions(self, user_id, transactions):\n &#34;&#34;&#34;Store transactions in database&#34;&#34;&#34;\n pass\n \n def check_budget_alerts(self, user_id):\n &#34;&#34;&#34;Check if user exceeded budget categories&#34;&#34;&#34;\n budgets = self.get_user_budgets(user_id)\n spending = self.calculate_spending(user_id)\n \n alerts = []\n for category, limit in budgets.items():\n if spending.get(category, 0) &gt; limit:\n alerts.append({\n &#39;type&#39;: &#39;budget_exceeded&#39;,\n &#39;category&#39;: category,\n &#39;limit&#39;: limit,\n &#39;actual&#39;: spending[category]\n })\n \n if alerts:\n self.send_alerts(user_id, alerts)\n \n return alerts\n \n def get_user_budgets(self, user_id):\n &#34;&#34;&#34;Get budget limits for user&#34;&#34;&#34;\n return {\n &#39;Groceries&#39;: 500,\n &#39;Restaurants&#39;: 300,\n &#39;Entertainment&#39;: 200\n }\n \n def calculate_spending(self, user_id):\n &#34;&#34;&#34;Calculate spending by category this month&#34;&#34;&#34;\n return {\n &#39;Groceries&#39;: 450,\n &#39;Restaurants&#39;: 350, # Exceeded!\n &#39;Entertainment&#39;: 150\n }\n \n def send_alerts(self, user_id, alerts):\n &#34;&#34;&#34;Send push notification\/email&#34;&#34;&#34;\n print(f&#34;Alerts for user {user_id}: {alerts}&#34;)<\/code><\/pre><p><strong>Security Considerations:<\/strong><\/p>\n<ul>\n<li>Encrypt bank credentials at rest<\/li>\n<li>Use OAuth2 for authentication<\/li>\n<li>PCI DSS compliance<\/li>\n<li>End-to-end encryption<\/li>\n<li>Regular security audits<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"39-design-a-social-network-graph\"><span>39. Design a Social Network Graph<\/span>\n <a href=\"#39-design-a-social-network-graph\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Social graphs (friend networks, follower graphs) power features like friend suggestions, mutual connections, six degrees of separation, and social feed generation. Efficient graph traversal at scale (billions of users, trillions of edges) requires specialized storage and query strategies.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/social-network-graph-architecture.webp\" title=\"Social Network Graph Architecture \u2014 Graph DB, adjacency list cache, BFS engine, and friend recommendation service at billion-user scale\" data-thumbnail=\"\/files\/social-network-graph-architecture.webp\" data-sub-html=\"<h2>Social network graph database architecture \u2014 adjacency list, BFS traversal, and friend recommendation at scale<\/h2><p>Social Network Graph Architecture \u2014 Graph DB, adjacency list cache, BFS engine, and friend recommendation service at billion-user scale<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/social-network-graph-architecture.webp' alt=\"Social network graph database architecture \u2014 adjacency list, BFS traversal, and friend recommendation at scale\"><\/a><figcaption class=\"image-caption\">Social Network Graph Architecture \u2014 Graph DB, adjacency list cache, BFS engine, and friend recommendation service at billion-user scale<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"391-requirements\"><span>39.1 Requirements<\/span>\n <a href=\"#391-requirements\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Functional:<\/strong><\/p>\n<ul>\n<li>Add\/remove friendships<\/li>\n<li>Find friends of user<\/li>\n<li>Find mutual friends<\/li>\n<li>Friend recommendations<\/li>\n<li>Degree of separation<\/li>\n<\/ul>\n<p><strong>Non-Functional:<\/strong><\/p>\n<ul>\n<li>Billions of users<\/li>\n<li>Millions of connections per user<\/li>\n<li>Fast friend queries (&lt; 100ms)<\/li>\n<li>Scale horizontally<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"392-data-structures\"><span>39.2 Data Structures<\/span>\n <a href=\"#392-data-structures\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Adjacency List (Most Common):<\/strong><\/p>\n<pre><code>class SocialGraph:\n def __init__(self):\n self.adjacency_list = {} # user_id -&gt; set of friend_ids\n \n def add_friendship(self, user1, user2):\n &#34;&#34;&#34;Add bidirectional friendship&#34;&#34;&#34;\n if user1 not in self.adjacency_list:\n self.adjacency_list[user1] = set()\n if user2 not in self.adjacency_list:\n self.adjacency_list[user2] = set()\n \n self.adjacency_list[user1].add(user2)\n self.adjacency_list[user2].add(user1)\n \n def remove_friendship(self, user1, user2):\n &#34;&#34;&#34;Remove friendship&#34;&#34;&#34;\n if user1 in self.adjacency_list:\n self.adjacency_list[user1].discard(user2)\n if user2 in self.adjacency_list:\n self.adjacency_list[user2].discard(user1)\n \n def get_friends(self, user_id):\n &#34;&#34;&#34;Get all friends of user&#34;&#34;&#34;\n return self.adjacency_list.get(user_id, set())\n \n def get_mutual_friends(self, user1, user2):\n &#34;&#34;&#34;Find mutual friends between two users&#34;&#34;&#34;\n friends1 = self.get_friends(user1)\n friends2 = self.get_friends(user2)\n return friends1 &amp; friends2 # Set intersection\n \n def are_friends(self, user1, user2):\n &#34;&#34;&#34;Check if two users are friends&#34;&#34;&#34;\n return user2 in self.adjacency_list.get(user1, set())\n \n def get_friend_count(self, user_id):\n &#34;&#34;&#34;Get number of friends&#34;&#34;&#34;\n return len(self.adjacency_list.get(user_id, set()))\n \n def friend_recommendations(self, user_id, limit=10):\n &#34;&#34;&#34;Recommend friends (friends of friends)&#34;&#34;&#34;\n friends = self.get_friends(user_id)\n recommendations = {}\n \n # Find friends of friends\n for friend in friends:\n for friend_of_friend in self.get_friends(friend):\n if friend_of_friend == user_id:\n continue\n if friend_of_friend in friends:\n continue\n \n # Count mutual friends\n recommendations[friend_of_friend] = \\\n recommendations.get(friend_of_friend, 0) &#43; 1\n \n # Sort by number of mutual friends\n sorted_recs = sorted(\n recommendations.items(),\n key=lambda x: x[1],\n reverse=True\n )\n \n return [user for user, _ in sorted_recs[:limit]]\n \n def degrees_of_separation(self, user1, user2):\n &#34;&#34;&#34;Find shortest path between two users (BFS)&#34;&#34;&#34;\n if user1 == user2:\n return 0\n \n from collections import deque\n \n queue = deque([(user1, 0)])\n visited = {user1}\n \n while queue:\n current_user, distance = queue.popleft()\n \n for friend in self.get_friends(current_user):\n if friend == user2:\n return distance &#43; 1\n \n if friend not in visited:\n visited.add(friend)\n queue.append((friend, distance &#43; 1))\n \n return -1 # Not connected\n\n# Usage\ngraph = SocialGraph()\ngraph.add_friendship(&#34;alice&#34;, &#34;bob&#34;)\ngraph.add_friendship(&#34;bob&#34;, &#34;charlie&#34;)\ngraph.add_friendship(&#34;charlie&#34;, &#34;david&#34;)\n\nprint(graph.degrees_of_separation(&#34;alice&#34;, &#34;david&#34;)) # 3\nprint(graph.friend_recommendations(&#34;alice&#34;)) # [&#39;charlie&#39;]<\/code><\/pre><p><strong>Database Schema for Scale:<\/strong><\/p>\n<pre><code>-- Users table\nCREATE TABLE users (\n user_id BIGINT PRIMARY KEY,\n username VARCHAR(50) UNIQUE,\n created_at TIMESTAMP\n);\n\n-- Friendships table (bidirectional)\nCREATE TABLE friendships (\n user_id1 BIGINT,\n user_id2 BIGINT,\n created_at TIMESTAMP,\n PRIMARY KEY (user_id1, user_id2),\n INDEX idx_user1 (user_id1),\n INDEX idx_user2 (user_id2)\n);\n\n-- Denormalized for faster queries\nCREATE TABLE user_friends_count (\n user_id BIGINT PRIMARY KEY,\n friend_count INT,\n updated_at TIMESTAMP\n);<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"40-design-amazon-sales-rank-by-category\"><span>40. Design Amazon Sales Rank by Category<\/span>\n <a href=\"#40-design-amazon-sales-rank-by-category\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Track best-selling products per category in real-time.<\/p>\n<h3 class=\"heading-element\" id=\"401-requirements\"><span>40.1 Requirements<\/span>\n <a href=\"#401-requirements\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Functional:<\/strong><\/p>\n<ul>\n<li>Track sales for millions of products<\/li>\n<li>Calculate rank per category<\/li>\n<li>Update ranks in near real-time<\/li>\n<li>Support historical rankings<\/li>\n<\/ul>\n<p><strong>Non-Functional:<\/strong><\/p>\n<ul>\n<li>Handle millions of sales\/hour<\/li>\n<li>Rankings updated within 1 hour<\/li>\n<li>Support 1000s of categories<\/li>\n<li>Scalable to billions of products<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"402-architecture\"><span>40.2 Architecture<\/span>\n <a href=\"#402-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/data-pipeline-architecture.webp\" title=\"Data Pipeline \u2014 ETL and stream processing architecture for big data analytics and real-time processing\" data-thumbnail=\"\/files\/data-pipeline-architecture.webp\" data-sub-html=\"<h2>Big data pipeline and stream processing architecture \u2014 ETL processes for handling large-scale data workflows<\/h2><p>Data Pipeline \u2014 ETL and stream processing architecture for big data analytics and real-time processing<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/data-pipeline-architecture.webp' alt=\"Big data pipeline and stream processing architecture \u2014 ETL processes for handling large-scale data workflows\"><\/a><figcaption class=\"image-caption\">Data Pipeline \u2014 ETL and stream processing architecture for big data analytics and real-time processing<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"403-implementation\"><span>40.3 Implementation<\/span>\n <a href=\"#403-implementation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>from collections import defaultdict\nimport heapq\n\nclass SalesRankingService:\n def __init__(self):\n self.sales_by_category = defaultdict(lambda: defaultdict(int))\n # category -&gt; {product_id -&gt; sales_count}\n \n self.ranks_by_category = defaultdict(list)\n # category -&gt; [(sales_count, product_id), ...]\n \n def record_sale(self, product_id, category_id, timestamp):\n &#34;&#34;&#34;Record a product sale&#34;&#34;&#34;\n self.sales_by_category[category_id][product_id] &#43;= 1\n \n def calculate_ranks(self, category_id):\n &#34;&#34;&#34;Calculate rankings for category&#34;&#34;&#34;\n sales = self.sales_by_category[category_id]\n \n # Sort by sales count (descending)\n sorted_products = sorted(\n sales.items(),\n key=lambda x: x[1],\n reverse=True\n )\n \n # Store ranks\n ranks = []\n for rank, (product_id, sales_count) in enumerate(sorted_products, 1):\n ranks.append({\n &#39;rank&#39;: rank,\n &#39;product_id&#39;: product_id,\n &#39;sales_count&#39;: sales_count\n })\n \n return ranks\n \n def get_top_n(self, category_id, n=100):\n &#34;&#34;&#34;Get top N products in category&#34;&#34;&#34;\n ranks = self.calculate_ranks(category_id)\n return ranks[:n]\n \n def get_product_rank(self, product_id, category_id):\n &#34;&#34;&#34;Get rank of specific product&#34;&#34;&#34;\n ranks = self.calculate_ranks(category_id)\n \n for item in ranks:\n if item[&#39;product_id&#39;] == product_id:\n return item[&#39;rank&#39;]\n \n return None\n\n# For scale: Use sliding window aggregation\nclass StreamingSalesRank:\n &#34;&#34;&#34;Real-time ranking using sliding window&#34;&#34;&#34;\n \n def __init__(self, window_hours=24):\n self.window_hours = window_hours\n self.hourly_buckets = defaultdict(lambda: defaultdict(lambda: defaultdict(int)))\n # timestamp_hour -&gt; category -&gt; product -&gt; count\n \n def record_sale(self, product_id, category_id, timestamp):\n &#34;&#34;&#34;Record sale in time bucket&#34;&#34;&#34;\n import datetime\n hour_bucket = timestamp.replace(minute=0, second=0, microsecond=0)\n self.hourly_buckets[hour_bucket][category_id][product_id] &#43;= 1\n \n def get_rankings(self, category_id):\n &#34;&#34;&#34;Get rankings for category using sliding window&#34;&#34;&#34;\n import datetime\n now = datetime.datetime.now()\n start_window = now - datetime.timedelta(hours=self.window_hours)\n \n # Aggregate sales in window\n total_sales = defaultdict(int)\n for hour_bucket, categories in self.hourly_buckets.items():\n if hour_bucket &gt;= start_window:\n for product_id, count in categories[category_id].items():\n total_sales[product_id] &#43;= count\n \n # Calculate ranks\n sorted_products = sorted(\n total_sales.items(),\n key=lambda x: x[1],\n reverse=True\n )\n \n return [(rank, pid, count) \n for rank, (pid, count) in enumerate(sorted_products, 1)]<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"46-design-a-system-that-scales-to-millions-of-users-on-aws\"><span>46. Design a System that Scales to Millions of Users on AWS<\/span>\n <a href=\"#46-design-a-system-that-scales-to-millions-of-users-on-aws\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"overview\"><span>Overview<\/span>\n <a href=\"#overview\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Scaling a web application from a single user to millions requires an iterative approach: <strong>Benchmark \u2192 Profile \u2192 Address Bottlenecks \u2192 Repeat<\/strong>. This section demonstrates progressive scaling strategies using AWS services, applicable to any cloud platform.<\/p>\n<h3 class=\"heading-element\" id=\"problem-requirements\"><span>Problem Requirements<\/span>\n <a href=\"#problem-requirements\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Use Cases:<\/strong><\/p>\n<ul>\n<li>User makes read\/write requests<\/li>\n<li>Service processes requests and stores data<\/li>\n<li>Evolution from 1 user to 10 million users<\/li>\n<li>Maintain high availability throughout<\/li>\n<\/ul>\n<p><strong>Traffic Assumptions:<\/strong><\/p>\n<ul>\n<li>10 million users<\/li>\n<li>1 billion writes per month (~400 writes\/sec)<\/li>\n<li>100 billion reads per month (~40,000 reads\/sec)<\/li>\n<li>100:1 read-to-write ratio<\/li>\n<li>1 KB per write<\/li>\n<li>1 TB new content per month<\/li>\n<li>36 TB in 3 years<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"evolution-stage-1-single-server-1-100-users\"><span>Evolution Stage 1: Single Server (1-100 Users)<\/span>\n <a href=\"#evolution-stage-1-single-server-1-100-users\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Architecture:<\/strong><\/p>\n<ul>\n<li>Single EC2 instance running web server<\/li>\n<li>MySQL database on same instance<\/li>\n<li>Public static IP (Elastic IP)<\/li>\n<li>Route 53 for DNS mapping<\/li>\n<\/ul>\n<p><strong>Characteristics:<\/strong><\/p>\n<ul>\n<li>Vertical scaling only<\/li>\n<li>No redundancy<\/li>\n<li>Simple monitoring (CloudWatch, CPU, memory)<\/li>\n<li>Open ports: 80 (HTTP), 443 (HTTPS), 22 (SSH - whitelisted IPs only)<\/li>\n<\/ul>\n<pre><code># Simple single-server setup\nclass SingleServerApp:\n def __init__(self):\n self.db = MySQLDatabase()\n \n def handle_request(self, user_id, data):\n # Process request\n result = self.process(data)\n # Store in local database\n self.db.save(user_id, result)\n return result<\/code><\/pre><h3 class=\"heading-element\" id=\"evolution-stage-2-separate-components-100-1000-users\"><span>Evolution Stage 2: Separate Components (100-1000 Users)<\/span>\n <a href=\"#evolution-stage-2-separate-components-100-1000-users\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Key Changes:<\/strong><\/p>\n<ul>\n<li>Move MySQL to RDS (managed database service)<\/li>\n<li>Store static content on S3 (object storage)<\/li>\n<li>Implement VPC with public\/private subnets<\/li>\n<\/ul>\n<p><strong>Benefits:<\/strong><\/p>\n<ul>\n<li>Independent scaling of components<\/li>\n<li>Reduced load on web server<\/li>\n<li>Automatic backups and multi-AZ replication (RDS)<\/li>\n<li>Server-side encryption at rest<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"evolution-stage-3-horizontal-scaling-1000-10000-users\"><span>Evolution Stage 3: Horizontal Scaling (1,000-10,000 Users)<\/span>\n <a href=\"#evolution-stage-3-horizontal-scaling-1000-10000-users\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Key Changes:<\/strong><\/p>\n<ul>\n<li>Add Elastic Load Balancer (ELB)<\/li>\n<li>Multiple web servers across availability zones<\/li>\n<li>Master-Slave MySQL replication<\/li>\n<li>Separate web and application tiers<\/li>\n<li>CloudFront CDN for static content<\/li>\n<\/ul>\n<p><strong>Load Balancer Configuration:<\/strong><\/p>\n<pre><code>class LoadBalancedArchitecture:\n def __init__(self):\n self.load_balancer = ELB()\n self.web_servers = [\n WebServer(az=&#39;us-east-1a&#39;),\n WebServer(az=&#39;us-east-1b&#39;),\n WebServer(az=&#39;us-east-1c&#39;)\n ]\n self.app_servers = [\n AppServer(type=&#39;read_api&#39;),\n AppServer(type=&#39;write_api&#39;)\n ]\n self.db_master = RDS_Master()\n self.db_slaves = [RDS_Slave(), RDS_Slave()]\n \n def route_request(self, request):\n # ELB distributes to web servers\n web_server = self.load_balancer.select_server()\n \n # Web server routes to appropriate app server\n if request.method == &#39;GET&#39;:\n app_server = self.select_read_server()\n else:\n app_server = self.select_write_server()\n \n return app_server.handle(request)\n \n def select_read_server(self):\n # Route reads to slave replicas\n return random.choice(self.app_servers)<\/code><\/pre><p><strong>Architecture:<\/strong><\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/database-replication-master-slave.webp\" title=\"Database Replication \u2014 Master-slave and multi-master replication strategies for availability and performance\" data-thumbnail=\"\/files\/database-replication-master-slave.webp\" data-sub-html=\"<h2>Master-slave database replication architecture \u2014 how to replicate data for high availability and read scalability<\/h2><p>Database Replication \u2014 Master-slave and multi-master replication strategies for availability and performance<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/database-replication-master-slave.webp' alt=\"Master-slave database replication architecture \u2014 how to replicate data for high availability and read scalability\"><\/a><figcaption class=\"image-caption\">Database Replication \u2014 Master-slave and multi-master replication strategies for availability and performance<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"evolution-stage-4-caching-layer-10000-100000-users\"><span>Evolution Stage 4: Caching Layer (10,000-100,000 Users)<\/span>\n <a href=\"#evolution-stage-4-caching-layer-10000-100000-users\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Key Changes:<\/strong><\/p>\n<ul>\n<li>Add ElastiCache (Redis\/Memcached)<\/li>\n<li>Cache frequently accessed data from MySQL<\/li>\n<li>Store session data in cache (enables stateless web servers)<\/li>\n<li>Add more read replicas<\/li>\n<\/ul>\n<p><strong>Why Caching?<\/strong><\/p>\n<ul>\n<li>Memory access: 250 microseconds<\/li>\n<li>SSD access: 1,000 microseconds (4x slower)<\/li>\n<li>Disk access: 20,000 microseconds (80x slower)<\/li>\n<\/ul>\n<pre><code>class CachedArchitecture:\n def __init__(self):\n self.cache = Redis() # ElastiCache\n self.db_master = RDS_Master()\n self.db_slaves = [RDS_Slave() for _ in range(3)]\n \n def read_data(self, key):\n # Try cache first\n cached = self.cache.get(key)\n if cached:\n return cached\n \n # Cache miss - read from slave\n slave = random.choice(self.db_slaves)\n data = slave.query(key)\n \n # Update cache\n self.cache.set(key, data, ttl=3600)\n return data\n \n def write_data(self, key, value):\n # Write to master\n self.db_master.write(key, value)\n \n # Invalidate cache\n self.cache.delete(key)\n \n def store_session(self, session_id, session_data):\n # Store session in Redis for stateless servers\n self.cache.set(f&#34;session:{session_id}&#34;, session_data, ttl=1800)<\/code><\/pre><h3 class=\"heading-element\" id=\"evolution-stage-5-autoscaling-100000-1m-users\"><span>Evolution Stage 5: Autoscaling (100,000-1M Users)<\/span>\n <a href=\"#evolution-stage-5-autoscaling-100000-1m-users\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Key Changes:<\/strong><\/p>\n<ul>\n<li>Auto Scaling Groups for web and app servers<\/li>\n<li>CloudWatch metrics trigger scaling<\/li>\n<li>Chef\/Puppet\/Ansible for configuration management<\/li>\n<li>Enhanced monitoring and alerting<\/li>\n<\/ul>\n<p><strong>Autoscaling Configuration:<\/strong><\/p>\n<pre><code>class AutoScalingConfig:\n def __init__(self):\n self.web_server_group = AutoScalingGroup(\n min_instances=2,\n max_instances=20,\n desired_capacity=5,\n availability_zones=[&#39;us-east-1a&#39;, &#39;us-east-1b&#39;, &#39;us-east-1c&#39;]\n )\n \n self.app_server_group = AutoScalingGroup(\n min_instances=3,\n max_instances=30,\n desired_capacity=10\n )\n \n def configure_scaling_policies(self):\n # Scale up when CPU &gt; 70% for 5 minutes\n scale_up_policy = ScalingPolicy(\n metric=&#39;CPUUtilization&#39;,\n threshold=70,\n comparison=&#39;GreaterThanThreshold&#39;,\n evaluation_periods=2,\n period=300,\n action=&#39;add&#39;,\n adjustment=2 # Add 2 instances\n )\n \n # Scale down when CPU &lt; 30% for 10 minutes\n scale_down_policy = ScalingPolicy(\n metric=&#39;CPUUtilization&#39;,\n threshold=30,\n comparison=&#39;LessThanThreshold&#39;,\n evaluation_periods=2,\n period=600,\n action=&#39;remove&#39;,\n adjustment=1 # Remove 1 instance\n )\n \n return [scale_up_policy, scale_down_policy]<\/code><\/pre><p><strong>Monitoring Stack:<\/strong><\/p>\n<ul>\n<li>CloudWatch for metrics<\/li>\n<li>CloudTrail for API logging<\/li>\n<li>PagerDuty for incident management<\/li>\n<li>Sentry for error tracking<\/li>\n<li>New Relic for application performance<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"evolution-stage-6-advanced-scaling-1m-10m-users\"><span>Evolution Stage 6: Advanced Scaling (1M-10M Users)<\/span>\n <a href=\"#evolution-stage-6-advanced-scaling-1m-10m-users\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Key Changes:<\/strong><\/p>\n<ul>\n<li>Database sharding or federation<\/li>\n<li>NoSQL databases (DynamoDB) for specific use cases<\/li>\n<li>Data warehouse (Redshift) for analytics<\/li>\n<li>Asynchronous processing with SQS and Lambda<\/li>\n<\/ul>\n<p><strong>When to Use NoSQL:<\/strong><\/p>\n<ul>\n<li>High write throughput (&gt; 400 writes\/sec)<\/li>\n<li>Simple key-value access patterns<\/li>\n<li>Denormalized data structures<\/li>\n<li>Geographic distribution needs<\/li>\n<\/ul>\n<p><strong>Asynchronous Processing:<\/strong><\/p>\n<pre><code>class AsyncArchitecture:\n def __init__(self):\n self.sqs = SQS_Queue(&#39;image-processing-queue&#39;)\n self.lambda_function = Lambda(&#39;thumbnail-creator&#39;)\n self.s3 = S3_Bucket(&#39;user-images&#39;)\n self.dynamodb = DynamoDB_Table(&#39;image-metadata&#39;)\n \n def upload_image(self, user_id, image_data):\n # 1. Upload original to S3\n image_key = f&#34;images\/{user_id}\/{uuid.uuid4()}.jpg&#34;\n self.s3.put_object(image_key, image_data)\n \n # 2. Send message to queue for processing\n message = {\n &#39;user_id&#39;: user_id,\n &#39;image_key&#39;: image_key,\n &#39;created_at&#39;: datetime.now().isoformat()\n }\n self.sqs.send_message(message)\n \n # 3. Return immediately (non-blocking)\n return {&#39;image_key&#39;: image_key, &#39;status&#39;: &#39;processing&#39;}\n \n def process_image_async(self, message):\n # Lambda function triggered by SQS\n image_key = message[&#39;image_key&#39;]\n user_id = message[&#39;user_id&#39;]\n \n # Download original\n original = self.s3.get_object(image_key)\n \n # Create thumbnail\n thumbnail = self.create_thumbnail(original)\n thumbnail_key = image_key.replace(&#39;.jpg&#39;, &#39;_thumb.jpg&#39;)\n self.s3.put_object(thumbnail_key, thumbnail)\n \n # Update metadata\n self.dynamodb.put_item({\n &#39;user_id&#39;: user_id,\n &#39;image_key&#39;: image_key,\n &#39;thumbnail_key&#39;: thumbnail_key,\n &#39;status&#39;: &#39;completed&#39;,\n &#39;processed_at&#39;: datetime.now().isoformat()\n })<\/code><\/pre><p><strong>Database Sharding Strategy:<\/strong><\/p>\n<pre><code>class ShardedDatabase:\n def __init__(self, num_shards=8):\n self.shards = [RDS_Instance(f&#39;shard-{i}&#39;) for i in range(num_shards)]\n self.num_shards = num_shards\n \n def get_shard(self, user_id):\n # Hash-based sharding\n shard_id = hash(user_id) % self.num_shards\n return self.shards[shard_id]\n \n def write_user_data(self, user_id, data):\n shard = self.get_shard(user_id)\n shard.write(user_id, data)\n \n def read_user_data(self, user_id):\n shard = self.get_shard(user_id)\n return shard.read(user_id)<\/code><\/pre><h3 class=\"heading-element\" id=\"cost-optimization-strategies\"><span>Cost Optimization Strategies<\/span>\n <a href=\"#cost-optimization-strategies\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Right-sizing Instances:<\/strong><\/p>\n<ul>\n<li>Use T3 instances for variable workloads (burstable CPU)<\/li>\n<li>Use C5 for compute-intensive tasks<\/li>\n<li>Use R5 for memory-intensive workloads<\/li>\n<\/ul>\n<p><strong>Reserved Instances:<\/strong><\/p>\n<ul>\n<li>1-year or 3-year commitments for baseline capacity<\/li>\n<li>Save up to 75% vs on-demand pricing<\/li>\n<\/ul>\n<p><strong>Spot Instances:<\/strong><\/p>\n<ul>\n<li>Use for fault-tolerant, flexible workloads<\/li>\n<li>Save up to 90% vs on-demand<\/li>\n<\/ul>\n<pre><code>class CostOptimizedScaling:\n def __init__(self):\n self.baseline_servers = [\n EC2_Reserved(&#39;c5.large&#39;) for _ in range(5) # Reserved instances\n ]\n self.burst_servers = [] # Spot instances\n self.peak_servers = [] # On-demand instances\n \n def handle_traffic_spike(self, current_load):\n if current_load &gt; self.baseline_capacity():\n # Try to use spot instances first\n spot_instance = self.request_spot_instance(&#39;c5.large&#39;, max_price=0.05)\n if spot_instance:\n self.burst_servers.append(spot_instance)\n else:\n # Fall back to on-demand\n on_demand = self.launch_on_demand(&#39;c5.large&#39;)\n self.peak_servers.append(on_demand)<\/code><\/pre><h3 class=\"heading-element\" id=\"disaster-recovery-and-multi-region\"><span>Disaster Recovery and Multi-Region<\/span>\n <a href=\"#disaster-recovery-and-multi-region\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Strategies:<\/strong><\/p>\n<ol>\n<li><strong>Backup and Restore<\/strong> (RPO: hours, RTO: hours, cheapest)<\/li>\n<li><strong>Pilot Light<\/strong> (RPO: minutes, RTO: 10s of minutes)<\/li>\n<li><strong>Warm Standby<\/strong> (RPO: seconds, RTO: minutes)<\/li>\n<li><strong>Multi-Region Active-Active<\/strong> (RPO: none, RTO: none, most expensive)<\/li>\n<\/ol>\n<pre><code>class MultiRegionArchitecture:\n def __init__(self):\n self.primary_region = Region(&#39;us-east-1&#39;)\n self.secondary_region = Region(&#39;us-west-2&#39;)\n self.route53 = Route53_HealthCheck()\n \n def configure_failover(self):\n # Health check on primary\n self.route53.add_health_check(\n endpoint=self.primary_region.load_balancer,\n interval=30,\n failure_threshold=3\n )\n \n # Failover routing policy\n self.route53.add_record(\n name=&#39;api.example.com&#39;,\n type=&#39;A&#39;,\n primary=self.primary_region.load_balancer,\n secondary=self.secondary_region.load_balancer,\n failover=True\n )\n \n def sync_data(self):\n # Cross-region replication for S3\n self.primary_region.s3.enable_cross_region_replication(\n destination=self.secondary_region.s3\n )\n \n # Database replication\n self.primary_region.rds.create_read_replica(\n target_region=&#39;us-west-2&#39;\n )<\/code><\/pre><h3 class=\"heading-element\" id=\"key-takeaways\"><span>Key Takeaways<\/span>\n <a href=\"#key-takeaways\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li><strong>Start Simple<\/strong>: Begin with single server, scale incrementally<\/li>\n<li><strong>Monitor Everything<\/strong>: Use metrics to identify real bottlenecks<\/li>\n<li><strong>Horizontal Scaling<\/strong>: Easier to scale than vertical<\/li>\n<li><strong>Caching is Critical<\/strong>: Dramatically reduces database load<\/li>\n<li><strong>Decouple Components<\/strong>: Use queues for asynchronous processing<\/li>\n<li><strong>Plan for Failure<\/strong>: Multi-AZ, auto-healing, backups<\/li>\n<li><strong>Cost vs Performance<\/strong>: Balance based on business needs<\/li>\n<\/ol>\n<p><strong>Progressive Evolution:<\/strong><\/p>\n<ul>\n<li>1-100 users: Single server<\/li>\n<li>100-1K users: Separate database, object storage<\/li>\n<li>1K-10K users: Load balancing, horizontal scaling<\/li>\n<li>10K-100K users: Caching layer, more replicas<\/li>\n<li>100K-1M users: Autoscaling, configuration management<\/li>\n<li>1M-10M users: Sharding, NoSQL, async processing, multi-region<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"47-design-a-query-cache-for-search-engine\"><span>47. Design a Query Cache for Search Engine<\/span>\n <a href=\"#47-design-a-query-cache-for-search-engine\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"overview-1\"><span>Overview<\/span>\n <a href=\"#overview-1\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Design a distributed key-value cache to save results of recent web search queries, reducing latency and load on backend search services. This pattern applies to any high-traffic query system (Google, Bing, e-commerce search).<\/p>\n<h3 class=\"heading-element\" id=\"problem-requirements-1\"><span>Problem Requirements<\/span>\n <a href=\"#problem-requirements-1\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Use Cases:<\/strong><\/p>\n<ul>\n<li>User sends search request \u2192 cache hit (fast response)<\/li>\n<li>User sends search request \u2192 cache miss (query backend, cache result)<\/li>\n<li>High availability with low latency<\/li>\n<\/ul>\n<p><strong>Traffic Assumptions:<\/strong><\/p>\n<ul>\n<li>10 million users<\/li>\n<li>10 billion queries per month (~4,000 queries\/sec)<\/li>\n<li>Traffic not evenly distributed (popular queries dominate)<\/li>\n<li>Limited cache memory (cannot store everything)<\/li>\n<\/ul>\n<p><strong>Capacity Estimation:<\/strong><\/p>\n<p>Each cache entry:<\/p>\n<ul>\n<li>Query: 50 bytes<\/li>\n<li>Title: 20 bytes<\/li>\n<li>Snippet: 200 bytes<\/li>\n<li><strong>Total: 270 bytes per entry<\/strong><\/li>\n<\/ul>\n<p>If all 10 billion queries were unique:<\/p>\n<ul>\n<li>270 bytes \u00d7 10 billion = <strong>2.7 TB\/month<\/strong><\/li>\n<li>Reality: Popular queries repeat, need LRU eviction<\/li>\n<\/ul>\n<p><strong>QPS Calculation:<\/strong><\/p>\n<ul>\n<li>2.5 million seconds per month<\/li>\n<li>10 billion queries \/ 2.5M seconds = <strong>4,000 QPS<\/strong><\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"high-level-architecture\"><span>High-Level Architecture<\/span>\n <a href=\"#high-level-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/api-gateway-pattern.webp\" title=\"API Gateway Pattern \u2014 Routing, rate limiting, authentication, and protocol translation for scalable APIs\" data-thumbnail=\"\/files\/api-gateway-pattern.webp\" data-sub-html=\"<h2>API Gateway pattern for routing, authentication, and request management \u2014 centralized entry point for microservices<\/h2><p>API Gateway Pattern \u2014 Routing, rate limiting, authentication, and protocol translation for scalable APIs<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/api-gateway-pattern.webp' alt=\"API Gateway pattern for routing, authentication, and request management \u2014 centralized entry point for microservices\"><\/a><figcaption class=\"image-caption\">API Gateway Pattern \u2014 Routing, rate limiting, authentication, and protocol translation for scalable APIs<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"component-design\"><span>Component Design<\/span>\n <a href=\"#component-design\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"1-query-api-server\"><span>1. Query API Server<\/span>\n <a href=\"#1-query-api-server\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Responsibilities:<\/p>\n<ul>\n<li>Parse and normalize queries<\/li>\n<li>Check cache first<\/li>\n<li>Route to search backend on miss<\/li>\n<li>Update cache with results<\/li>\n<\/ul>\n<pre><code>class QueryAPIServer:\n def __init__(self, cache, reverse_index_service, document_service):\n self.cache = cache\n self.reverse_index = reverse_index_service\n self.doc_service = document_service\n \n def parse_query(self, raw_query):\n &#34;&#34;&#34;Normalize query for consistent cache keys.&#34;&#34;&#34;\n # Remove HTML\/markup\n query = self.strip_html(raw_query)\n \n # Fix typos (simple example)\n query = self.spell_check(query)\n \n # Normalize: lowercase, trim whitespace\n query = query.lower().strip()\n \n # Remove stop words for better matching\n query = self.remove_stop_words(query)\n \n # Convert to boolean operations\n terms = query.split()\n boolean_query = &#39; AND &#39;.join(terms)\n \n return boolean_query\n \n def process_query(self, raw_query):\n # 1. Parse and normalize\n query = self.parse_query(raw_query)\n \n # 2. Check cache\n results = self.cache.get(query)\n if results is not None:\n print(f&#34;Cache HIT for query: {query}&#34;)\n return results\n \n print(f&#34;Cache MISS for query: {query}&#34;)\n \n # 3. Query backend services\n doc_ids = self.reverse_index.search(query)\n results = self.doc_service.fetch_documents(doc_ids)\n \n # 4. Update cache\n self.cache.set(query, results)\n \n return results\n \n def strip_html(self, text):\n # Remove HTML tags\n import re\n return re.sub(r&#39;&lt;[^&gt;]&#43;&gt;&#39;, &#39;&#39;, text)\n \n def spell_check(self, query):\n # Simplified: use edit distance or ML model\n corrections = {\n &#39;gogle&#39;: &#39;google&#39;,\n &#39;amazn&#39;: &#39;amazon&#39;,\n &#39;facebok&#39;: &#39;facebook&#39;\n }\n return corrections.get(query, query)\n \n def remove_stop_words(self, query):\n stop_words = {&#39;the&#39;, &#39;a&#39;, &#39;an&#39;, &#39;and&#39;, &#39;or&#39;, &#39;but&#39;, &#39;in&#39;, &#39;on&#39;, &#39;at&#39;}\n words = query.split()\n filtered = [w for w in words if w not in stop_words]\n return &#39; &#39;.join(filtered)<\/code><\/pre><h4 class=\"heading-element\" id=\"2-lru-cache-implementation\"><span>2. LRU Cache Implementation<\/span>\n <a href=\"#2-lru-cache-implementation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Use doubly-linked list + hash map for O(1) operations:<\/p>\n<pre><code>class Node:\n def __init__(self, query, results):\n self.query = query\n self.results = results\n self.prev = None\n self.next = None\n\n\nclass DoublyLinkedList:\n def __init__(self):\n self.head = None\n self.tail = None\n \n def move_to_front(self, node):\n &#34;&#34;&#34;Move existing node to front (most recently used).&#34;&#34;&#34;\n if node == self.head:\n return # Already at front\n \n # Remove from current position\n if node.prev:\n node.prev.next = node.next\n if node.next:\n node.next.prev = node.prev\n if node == self.tail:\n self.tail = node.prev\n \n # Move to front\n node.prev = None\n node.next = self.head\n if self.head:\n self.head.prev = node\n self.head = node\n if self.tail is None:\n self.tail = node\n \n def append_to_front(self, node):\n &#34;&#34;&#34;Add new node to front.&#34;&#34;&#34;\n node.next = self.head\n node.prev = None\n if self.head:\n self.head.prev = node\n self.head = node\n if self.tail is None:\n self.tail = node\n \n def remove_from_tail(self):\n &#34;&#34;&#34;Remove least recently used node from tail.&#34;&#34;&#34;\n if self.tail is None:\n return None\n \n removed = self.tail\n if self.tail.prev:\n self.tail.prev.next = None\n self.tail = self.tail.prev\n else:\n # Only one node\n self.head = None\n self.tail = None\n \n return removed\n\n\nclass LRUCache:\n def __init__(self, max_size):\n self.max_size = max_size\n self.size = 0\n self.lookup = {} # query -&gt; node\n self.linked_list = DoublyLinkedList()\n \n def get(self, query):\n &#34;&#34;&#34;Get cached results. O(1) time complexity.&#34;&#34;&#34;\n node = self.lookup.get(query)\n if node is None:\n return None\n \n # Move to front (mark as recently used)\n self.linked_list.move_to_front(node)\n return node.results\n \n def set(self, query, results):\n &#34;&#34;&#34;Cache query results. O(1) time complexity.&#34;&#34;&#34;\n node = self.lookup.get(query)\n \n if node is not None:\n # Update existing entry\n node.results = results\n self.linked_list.move_to_front(node)\n else:\n # New entry\n if self.size &gt;= self.max_size:\n # Evict least recently used\n evicted = self.linked_list.remove_from_tail()\n if evicted:\n del self.lookup[evicted.query]\n self.size -= 1\n \n # Add new node\n new_node = Node(query, results)\n self.linked_list.append_to_front(new_node)\n self.lookup[query] = new_node\n self.size &#43;= 1\n \n def clear(self):\n &#34;&#34;&#34;Clear entire cache.&#34;&#34;&#34;\n self.lookup.clear()\n self.linked_list = DoublyLinkedList()\n self.size = 0\n\n\n# Example usage\ncache = LRUCache(max_size=1000)\n\n# Cache miss\nresults1 = cache.get(&#34;python tutorials&#34;) # None\n\n# Store in cache\ncache.set(&#34;python tutorials&#34;, [&#34;Tutorial 1&#34;, &#34;Tutorial 2&#34;, &#34;Tutorial 3&#34;])\n\n# Cache hit\nresults2 = cache.get(&#34;python tutorials&#34;) # [&#34;Tutorial 1&#34;, &#34;Tutorial 2&#34;, &#34;Tutorial 3&#34;]<\/code><\/pre><h4 class=\"heading-element\" id=\"3-cache-update-strategies\"><span>3. Cache Update Strategies<\/span>\n <a href=\"#3-cache-update-strategies\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><strong>When to update cache:<\/strong><\/p>\n<ol>\n<li>Page content changes<\/li>\n<li>Page is added\/removed<\/li>\n<li>Page rank changes (search ranking algorithm updates)<\/li>\n<\/ol>\n<p><strong>TTL (Time To Live) Approach:<\/strong><\/p>\n<pre><code>import time\n\nclass CacheEntryWithTTL:\n def __init__(self, query, results, ttl_seconds=3600):\n self.query = query\n self.results = results\n self.created_at = time.time()\n self.ttl = ttl_seconds\n \n def is_expired(self):\n return (time.time() - self.created_at) &gt; self.ttl\n\n\nclass LRUCacheWithTTL(LRUCache):\n def get(self, query):\n node = self.lookup.get(query)\n if node is None:\n return None\n \n # Check if expired\n if isinstance(node, CacheEntryWithTTL) and node.is_expired():\n # Remove expired entry\n self.linked_list.remove_from_tail()\n del self.lookup[query]\n self.size -= 1\n return None\n \n self.linked_list.move_to_front(node)\n return node.results<\/code><\/pre><p><strong>Cache Invalidation Strategies:<\/strong><\/p>\n<pre><code>class CacheInvalidationService:\n def __init__(self, cache):\n self.cache = cache\n \n def invalidate_by_pattern(self, pattern):\n &#34;&#34;&#34;Invalidate all queries matching a pattern.&#34;&#34;&#34;\n # Example: Invalidate all queries containing &#34;python&#34;\n keys_to_delete = [\n key for key in self.cache.lookup.keys() \n if pattern in key\n ]\n for key in keys_to_delete:\n node = self.cache.lookup[key]\n # Remove from linked list and lookup\n del self.cache.lookup[key]\n self.cache.size -= 1\n \n def invalidate_url(self, url):\n &#34;&#34;&#34;When a URL is updated, invalidate queries that returned it.&#34;&#34;&#34;\n # In production, maintain reverse index: URL -&gt; queries\n pass<\/code><\/pre><h3 class=\"heading-element\" id=\"distributed-caching-at-scale\"><span>Distributed Caching at Scale<\/span>\n <a href=\"#distributed-caching-at-scale\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>For 4,000 QPS and millions of entries, single-server cache won&rsquo;t work.<\/p>\n<p><strong>Option 1: Replicate Cache on Every Server<\/strong><\/p>\n<ul>\n<li>Simple but inefficient memory usage<\/li>\n<li>Low cache hit rate (each server has different subset)<\/li>\n<\/ul>\n<p><strong>Option 2: Dedicated Cache Servers (Recommended)<\/strong><\/p>\n<ul>\n<li>Use consistent hashing to shard cache<\/li>\n<li>Multiple cache servers with replication<\/li>\n<\/ul>\n<pre><code>import hashlib\nimport bisect\n\nclass ConsistentHashRing:\n def __init__(self, num_virtual_nodes=150):\n self.num_virtual_nodes = num_virtual_nodes\n self.ring = {} # hash_value -&gt; server_id\n self.sorted_hashes = []\n \n def add_server(self, server_id):\n for i in range(self.num_virtual_nodes):\n virtual_key = f&#34;{server_id}:{i}&#34;\n hash_value = int(hashlib.md5(virtual_key.encode()).hexdigest(), 16)\n self.ring[hash_value] = server_id\n bisect.insort(self.sorted_hashes, hash_value)\n \n def remove_server(self, server_id):\n for i in range(self.num_virtual_nodes):\n virtual_key = f&#34;{server_id}:{i}&#34;\n hash_value = int(hashlib.md5(virtual_key.encode()).hexdigest(), 16)\n del self.ring[hash_value]\n self.sorted_hashes.remove(hash_value)\n \n def get_server(self, query):\n query_hash = int(hashlib.md5(query.encode()).hexdigest(), 16)\n idx = bisect.bisect_right(self.sorted_hashes, query_hash)\n if idx == len(self.sorted_hashes):\n idx = 0\n server_hash = self.sorted_hashes[idx]\n return self.ring[server_hash]\n\n\nclass DistributedCache:\n def __init__(self, cache_servers):\n self.hash_ring = ConsistentHashRing()\n self.cache_servers = {} # server_id -&gt; LRUCache instance\n \n for server_id in cache_servers:\n self.add_cache_server(server_id)\n \n def add_cache_server(self, server_id):\n self.hash_ring.add_server(server_id)\n self.cache_servers[server_id] = LRUCache(max_size=100000)\n \n def get(self, query):\n server_id = self.hash_ring.get_server(query)\n cache = self.cache_servers[server_id]\n return cache.get(query)\n \n def set(self, query, results):\n server_id = self.hash_ring.get_server(query)\n cache = self.cache_servers[server_id]\n cache.set(query, results)\n\n\n# Example: 5 cache servers\ndistributed_cache = DistributedCache([&#39;cache-1&#39;, &#39;cache-2&#39;, &#39;cache-3&#39;, &#39;cache-4&#39;, &#39;cache-5&#39;])\n\n# Queries are automatically sharded\ndistributed_cache.set(&#34;python tutorials&#34;, [&#34;Result 1&#34;, &#34;Result 2&#34;])\ndistributed_cache.set(&#34;java tutorials&#34;, [&#34;Result 3&#34;, &#34;Result 4&#34;])\n\nresults = distributed_cache.get(&#34;python tutorials&#34;)<\/code><\/pre><h3 class=\"heading-element\" id=\"monitoring-and-metrics\"><span>Monitoring and Metrics<\/span>\n <a href=\"#monitoring-and-metrics\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/monitoring-observability-stack.webp\" title=\"Monitoring Stack \u2014 Complete observability solution: logs, metrics, traces, and alerting for production systems\" data-thumbnail=\"\/files\/monitoring-observability-stack.webp\" data-sub-html=\"<h2>Monitoring and observability stack architecture \u2014 logging, metrics, and tracing for system health and performance monitoring<\/h2><p>Monitoring Stack \u2014 Complete observability solution: logs, metrics, traces, and alerting for production systems<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/monitoring-observability-stack.webp' alt=\"Monitoring and observability stack architecture \u2014 logging, metrics, and tracing for system health and performance monitoring\"><\/a><figcaption class=\"image-caption\">Monitoring Stack \u2014 Complete observability solution: logs, metrics, traces, and alerting for production systems<\/figcaption>\n <\/figure><\/p>\n<p><strong>Key Metrics:<\/strong><\/p>\n<ol>\n<li><strong>Cache Hit Rate<\/strong>: (Hits \/ Total Requests) \u00d7 100%<\/li>\n<li><strong>Cache Miss Latency<\/strong>: Time to fetch from backend<\/li>\n<li><strong>Memory Usage<\/strong>: Prevent OOM<\/li>\n<li><strong>Eviction Rate<\/strong>: How often entries are removed<\/li>\n<\/ol>\n<pre><code>class CacheMetrics:\n def __init__(self):\n self.hits = 0\n self.misses = 0\n self.total_latency = 0.0\n self.evictions = 0\n \n def record_hit(self):\n self.hits &#43;= 1\n \n def record_miss(self, latency):\n self.misses &#43;= 1\n self.total_latency &#43;= latency\n \n def record_eviction(self):\n self.evictions &#43;= 1\n \n def get_stats(self):\n total_requests = self.hits &#43; self.misses\n hit_rate = (self.hits \/ total_requests * 100) if total_requests &gt; 0 else 0\n avg_miss_latency = (self.total_latency \/ self.misses) if self.misses &gt; 0 else 0\n \n return {\n &#39;hit_rate&#39;: hit_rate,\n &#39;total_requests&#39;: total_requests,\n &#39;hits&#39;: self.hits,\n &#39;misses&#39;: self.misses,\n &#39;avg_miss_latency_ms&#39;: avg_miss_latency * 1000,\n &#39;evictions&#39;: self.evictions\n }<\/code><\/pre><h3 class=\"heading-element\" id=\"complete-system-integration\"><span>Complete System Integration<\/span>\n <a href=\"#complete-system-integration\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>class SearchEngineQueryCache:\n def __init__(self):\n self.distributed_cache = DistributedCache(\n [&#39;cache-1&#39;, &#39;cache-2&#39;, &#39;cache-3&#39;, &#39;cache-4&#39;, &#39;cache-5&#39;]\n )\n self.metrics = CacheMetrics()\n self.reverse_index = ReverseIndexService()\n self.doc_service = DocumentService()\n \n def search(self, raw_query):\n # Parse query\n query = self.parse_query(raw_query)\n \n # Check distributed cache\n start_time = time.time()\n results = self.distributed_cache.get(query)\n \n if results is not None:\n # Cache hit\n self.metrics.record_hit()\n return results\n \n # Cache miss - query backend\n doc_ids = self.reverse_index.search(query)\n results = self.doc_service.fetch_documents(doc_ids)\n \n # Record metrics\n latency = time.time() - start_time\n self.metrics.record_miss(latency)\n \n # Update cache\n self.distributed_cache.set(query, results)\n \n return results\n \n def parse_query(self, raw_query):\n # Same as QueryAPIServer implementation\n return raw_query.lower().strip()<\/code><\/pre><h3 class=\"heading-element\" id=\"trade-offs-and-considerations\"><span>Trade-offs and Considerations<\/span>\n <a href=\"#trade-offs-and-considerations\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Memory vs Latency:<\/strong><\/p>\n<ul>\n<li>Larger cache = higher hit rate but more memory cost<\/li>\n<li>Optimal size depends on query distribution (Zipf&rsquo;s law)<\/li>\n<\/ul>\n<p><strong>TTL Selection:<\/strong><\/p>\n<ul>\n<li>Short TTL (minutes): Fresh data, but more backend load<\/li>\n<li>Long TTL (hours): Less backend load, but stale data risk<\/li>\n<li>Adaptive TTL based on query popularity<\/li>\n<\/ul>\n<p><strong>Consistency:<\/strong><\/p>\n<ul>\n<li>Cache-aside pattern: Simple but can serve stale data<\/li>\n<li>Write-through: Always consistent but slower writes<\/li>\n<li>Eventual consistency acceptable for search (not financial data)<\/li>\n<\/ul>\n<p><strong>Replication:<\/strong><\/p>\n<ul>\n<li>Single cache node: Fast but single point of failure<\/li>\n<li>Replicated cache: High availability but consistency challenges<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"key-takeaways-1\"><span>Key Takeaways<\/span>\n <a href=\"#key-takeaways-1\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li><strong>LRU eviction<\/strong> balances memory usage and hit rate<\/li>\n<li><strong>Consistent hashing<\/strong> enables horizontal scaling<\/li>\n<li><strong>Cache-aside pattern<\/strong> works well for read-heavy workloads<\/li>\n<li><strong>TTL prevents stale data<\/strong> without manual invalidation<\/li>\n<li><strong>Monitoring hit rate<\/strong> guides cache size tuning<\/li>\n<li><strong>Distributed caching<\/strong> required for high QPS (4,000+)<\/li>\n<\/ol>\n<p><strong>Performance Impact:<\/strong><\/p>\n<ul>\n<li>Memory access: 0.25 ms<\/li>\n<li>Cache miss + SSD: 1 ms<\/li>\n<li>Cache miss + HDD: 20 ms<\/li>\n<li><strong>80x speedup<\/strong> vs disk for popular queries!<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"48-design-an-e-commerce-website-amazon-like\"><span>48. Design an E-Commerce Website (Amazon-like)<\/span>\n <a href=\"#48-design-an-e-commerce-website-amazon-like\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"overview-2\"><span>Overview<\/span>\n <a href=\"#overview-2\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Design a scalable e-commerce platform handling millions of products, orders, and users with features like product catalog, cart, checkout, inventory management, and order processing.<\/p>\n<h3 class=\"heading-element\" id=\"problem-requirements-2\"><span>Problem Requirements<\/span>\n <a href=\"#problem-requirements-2\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Functional Requirements:<\/strong><\/p>\n<ul>\n<li>User registration\/authentication<\/li>\n<li>Product browsing and search<\/li>\n<li>Shopping cart management<\/li>\n<li>Checkout and payment processing<\/li>\n<li>Order tracking<\/li>\n<li>Inventory management<\/li>\n<li>Seller portal (for marketplace model)<\/li>\n<li>Reviews and ratings<\/li>\n<\/ul>\n<p><strong>Non-Functional Requirements:<\/strong><\/p>\n<ul>\n<li>100 million active users<\/li>\n<li>50 million products<\/li>\n<li>1 million orders per day (~12 orders\/sec)<\/li>\n<li>99.99% availability<\/li>\n<li>Sub-second search latency<\/li>\n<li>ACID transactions for payments<\/li>\n<li>PCI DSS compliance<\/li>\n<\/ul>\n<p><strong>Capacity Estimation:<\/strong><\/p>\n<ul>\n<li>Storage: 50M products \u00d7 10KB metadata = 500 GB<\/li>\n<li>Images: 50M products \u00d7 5 images \u00d7 500KB = 125 TB<\/li>\n<li>Orders: 1M\/day \u00d7 365 \u00d7 5 years \u00d7 5KB = 9 TB<\/li>\n<li>Peak traffic: 12 orders\/sec \u00d7 10 (peak multiplier) = 120 orders\/sec<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"high-level-architecture-1\"><span>High-Level Architecture<\/span>\n <a href=\"#high-level-architecture-1\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/microservices-architecture.webp\" title=\"E-Commerce Architecture \u2014 Amazon-style microservices: product, cart, order, inventory, payment, and notification services behind an API gateway\" data-thumbnail=\"\/files\/microservices-architecture.webp\" data-sub-html=\"<h2>Microservices architecture diagram with API gateway and independent services \u2014 service-oriented design for scalability and maintainability<\/h2><p>E-Commerce Architecture \u2014 Amazon-style microservices: product, cart, order, inventory, payment, and notification services behind an API gateway<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/microservices-architecture.webp' alt=\"Microservices architecture diagram with API gateway and independent services \u2014 service-oriented design for scalability and maintainability\"><\/a><figcaption class=\"image-caption\">E-Commerce Architecture \u2014 Amazon-style microservices: product, cart, order, inventory, payment, and notification services behind an API gateway<\/figcaption>\n <\/figure><\/p>\n<p><strong>Core Services:<\/strong><\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Service<\/th>\n <th>Responsibility<\/th>\n <th>Database<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Product Service<\/strong><\/td>\n <td>Catalog, search, recommendations<\/td>\n <td>PostgreSQL + Elasticsearch<\/td>\n <\/tr>\n <tr>\n <td><strong>Inventory Service<\/strong><\/td>\n <td>Stock levels, reservations<\/td>\n <td>PostgreSQL (ACID)<\/td>\n <\/tr>\n <tr>\n <td><strong>Cart Service<\/strong><\/td>\n <td>Shopping cart state<\/td>\n <td>Redis (fast, ephemeral)<\/td>\n <\/tr>\n <tr>\n <td><strong>Order Service<\/strong><\/td>\n <td>Order lifecycle, checkout<\/td>\n <td>PostgreSQL (ACID transactions)<\/td>\n <\/tr>\n <tr>\n <td><strong>Payment Service<\/strong><\/td>\n <td>Payment processing, refunds<\/td>\n <td>PostgreSQL + Payment gateway (Stripe\/Braintree)<\/td>\n <\/tr>\n <tr>\n <td><strong>User Service<\/strong><\/td>\n <td>Auth, profiles, addresses<\/td>\n <td>PostgreSQL<\/td>\n <\/tr>\n <tr>\n <td><strong>Notification Service<\/strong><\/td>\n <td>Email\/SMS\/Push confirmations<\/td>\n <td>Kafka + SendGrid\/Twilio<\/td>\n <\/tr>\n <tr>\n <td><strong>Search Service<\/strong><\/td>\n <td>Full-text product search, filtering<\/td>\n <td>Elasticsearch<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Traffic Flow:<\/strong><\/p>\n<ol>\n<li>User requests hit the <strong>API Gateway<\/strong> (authentication, rate limiting, routing).<\/li>\n<li><strong>Product browsing<\/strong> \u2192 Product Service \u2192 PostgreSQL (cached in Redis).<\/li>\n<li><strong>Add to cart<\/strong> \u2192 Cart Service \u2192 Redis (7-day TTL).<\/li>\n<li><strong>Checkout<\/strong> \u2192 Order Service \u2192 reserves inventory (Inventory Service) \u2192 processes payment \u2192 publishes <code>order_confirmed<\/code> to Kafka.<\/li>\n<li><strong>Downstream consumers<\/strong> listen on Kafka: Inventory Service (deduct stock), Notification Service (send confirmation), Analytics Service.<\/li>\n<\/ol>\n<p><strong>Key Design Decisions:<\/strong><\/p>\n<ul>\n<li>Cart in Redis: ephemeral, fast, lost on cache flush but acceptable (users can rebuild cart).<\/li>\n<li>Inventory reservation uses DB-level row locking (<code>SELECT ... FOR UPDATE<\/code>) to prevent overselling.<\/li>\n<li>Payment service integrates with Stripe or Braintree \u2014 never store raw card data (PCI compliance).<\/li>\n<li>Order service uses Saga pattern for distributed transaction: if payment fails, compensation transactions release inventory.<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"database-schema\"><span>Database Schema<\/span>\n <a href=\"#database-schema\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Product Schema\nclass Product:\n id: UUID (PK)\n seller_id: UUID (FK)\n name: str\n description: text\n category_id: UUID (FK)\n price: decimal(10, 2)\n currency: str\n brand: str\n created_at: timestamp\n updated_at: timestamp\n is_active: boolean\n\nclass ProductImage:\n id: UUID (PK)\n product_id: UUID (FK)\n image_url: str\n is_primary: boolean\n display_order: int\n\nclass Inventory:\n id: UUID (PK)\n product_id: UUID (FK)\n warehouse_id: UUID (FK)\n quantity: int\n reserved_quantity: int # For pending orders\n updated_at: timestamp\n\n# Order Schema\nclass Order:\n id: UUID (PK)\n user_id: UUID (FK)\n status: enum(&#39;pending&#39;, &#39;confirmed&#39;, &#39;shipped&#39;, &#39;delivered&#39;, &#39;cancelled&#39;)\n subtotal: decimal(10, 2)\n tax: decimal(10, 2)\n shipping: decimal(10, 2)\n total: decimal(10, 2)\n shipping_address_id: UUID (FK)\n billing_address_id: UUID (FK)\n created_at: timestamp\n updated_at: timestamp\n\nclass OrderItem:\n id: UUID (PK)\n order_id: UUID (FK)\n product_id: UUID (FK)\n quantity: int\n unit_price: decimal(10, 2)\n total_price: decimal(10, 2)\n\nclass Payment:\n id: UUID (PK)\n order_id: UUID (FK)\n payment_method: enum(&#39;credit_card&#39;, &#39;debit_card&#39;, &#39;paypal&#39;, &#39;crypto&#39;)\n amount: decimal(10, 2)\n status: enum(&#39;pending&#39;, &#39;completed&#39;, &#39;failed&#39;, &#39;refunded&#39;)\n transaction_id: str # From payment gateway\n created_at: timestamp<\/code><\/pre><h3 class=\"heading-element\" id=\"core-service-implementations\"><span>Core Service Implementations<\/span>\n <a href=\"#core-service-implementations\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"1-product-service\"><span>1. Product Service<\/span>\n <a href=\"#1-product-service\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>from flask import Flask, jsonify, request\nimport redis\nimport psycopg2\n\nclass ProductService:\n def __init__(self):\n self.db = psycopg2.connect(&#34;dbname=products user=admin&#34;)\n self.cache = redis.Redis(host=&#39;localhost&#39;, port=6379)\n \n def get_product(self, product_id):\n # Try cache first\n cache_key = f&#34;product:{product_id}&#34;\n cached = self.cache.get(cache_key)\n if cached:\n return json.loads(cached)\n \n # Cache miss - query database\n cursor = self.db.cursor()\n cursor.execute(&#34;&#34;&#34;\n SELECT p.*, array_agg(pi.image_url) as images\n FROM products p\n LEFT JOIN product_images pi ON p.id = pi.product_id\n WHERE p.id = %s AND p.is_active = true\n GROUP BY p.id\n &#34;&#34;&#34;, (product_id,))\n \n product = cursor.fetchone()\n if product:\n product_data = self.serialize_product(product)\n # Cache for 1 hour\n self.cache.setex(cache_key, 3600, json.dumps(product_data))\n return product_data\n \n return None\n \n def search_products(self, query, filters, page=1, page_size=20):\n # Delegate to Elasticsearch for full-text search\n es_query = {\n &#34;query&#34;: {\n &#34;bool&#34;: {\n &#34;must&#34;: {\n &#34;multi_match&#34;: {\n &#34;query&#34;: query,\n &#34;fields&#34;: [&#34;name^3&#34;, &#34;description&#34;, &#34;brand^2&#34;]\n }\n },\n &#34;filter&#34;: self.build_filters(filters)\n }\n },\n &#34;from&#34;: (page - 1) * page_size,\n &#34;size&#34;: page_size,\n &#34;sort&#34;: [{&#34;_score&#34;: &#34;desc&#34;}, {&#34;created_at&#34;: &#34;desc&#34;}]\n }\n \n results = self.elasticsearch.search(index=&#34;products&#34;, body=es_query)\n return results[&#39;hits&#39;][&#39;hits&#39;]\n \n def build_filters(self, filters):\n filter_clauses = []\n if filters.get(&#39;category&#39;):\n filter_clauses.append({&#34;term&#34;: {&#34;category_id&#34;: filters[&#39;category&#39;]}})\n if filters.get(&#39;price_range&#39;):\n filter_clauses.append({\n &#34;range&#34;: {\n &#34;price&#34;: {\n &#34;gte&#34;: filters[&#39;price_range&#39;][&#39;min&#39;],\n &#34;lte&#34;: filters[&#39;price_range&#39;][&#39;max&#39;]\n }\n }\n })\n if filters.get(&#39;brand&#39;):\n filter_clauses.append({&#34;terms&#34;: {&#34;brand&#34;: filters[&#39;brand&#39;]}})\n \n return filter_clauses<\/code><\/pre><h4 class=\"heading-element\" id=\"2-cart-service-redis-based\"><span>2. Cart Service (Redis-based)<\/span>\n <a href=\"#2-cart-service-redis-based\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>import json\nfrom datetime import timedelta\n\nclass CartService:\n def __init__(self):\n self.redis = redis.Redis(host=&#39;localhost&#39;, port=6379)\n \n def add_to_cart(self, user_id, product_id, quantity):\n cart_key = f&#34;cart:{user_id}&#34;\n \n # Get current cart\n cart = self.get_cart(user_id)\n \n # Add\/update item\n if product_id in cart:\n cart[product_id][&#39;quantity&#39;] &#43;= quantity\n else:\n # Fetch product details\n product = self.product_service.get_product(product_id)\n cart[product_id] = {\n &#39;product_id&#39;: product_id,\n &#39;name&#39;: product[&#39;name&#39;],\n &#39;price&#39;: product[&#39;price&#39;],\n &#39;quantity&#39;: quantity,\n &#39;image&#39;: product[&#39;images&#39;][0] if product[&#39;images&#39;] else None\n }\n \n # Save cart (expires in 7 days)\n self.redis.setex(cart_key, timedelta(days=7), json.dumps(cart))\n return cart\n \n def get_cart(self, user_id):\n cart_key = f&#34;cart:{user_id}&#34;\n cart_data = self.redis.get(cart_key)\n return json.loads(cart_data) if cart_data else {}\n \n def remove_from_cart(self, user_id, product_id):\n cart_key = f&#34;cart:{user_id}&#34;\n cart = self.get_cart(user_id)\n \n if product_id in cart:\n del cart[product_id]\n self.redis.setex(cart_key, timedelta(days=7), json.dumps(cart))\n \n return cart\n \n def clear_cart(self, user_id):\n cart_key = f&#34;cart:{user_id}&#34;\n self.redis.delete(cart_key)\n \n def calculate_totals(self, user_id):\n cart = self.get_cart(user_id)\n subtotal = sum(\n item[&#39;price&#39;] * item[&#39;quantity&#39;] \n for item in cart.values()\n )\n tax = subtotal * 0.08 # 8% tax\n shipping = 10.00 if subtotal &lt; 50 else 0.00 # Free shipping over $50\n total = subtotal &#43; tax &#43; shipping\n \n return {\n &#39;subtotal&#39;: subtotal,\n &#39;tax&#39;: tax,\n &#39;shipping&#39;: shipping,\n &#39;total&#39;: total\n }<\/code><\/pre><h4 class=\"heading-element\" id=\"3-order-service-with-transaction-management\"><span>3. Order Service with Transaction Management<\/span>\n <a href=\"#3-order-service-with-transaction-management\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>from contextlib import contextmanager\nimport uuid\nfrom datetime import datetime\n\nclass OrderService:\n def __init__(self):\n self.db = psycopg2.connect(&#34;dbname=orders user=admin&#34;)\n self.kafka_producer = KafkaProducer(bootstrap_servers=[&#39;localhost:9092&#39;])\n \n @contextmanager\n def transaction(self):\n &#34;&#34;&#34;Context manager for database transactions.&#34;&#34;&#34;\n conn = self.db\n try:\n yield conn\n conn.commit()\n except Exception as e:\n conn.rollback()\n raise e\n \n def create_order(self, user_id, cart_items, shipping_address, payment_method):\n &#34;&#34;&#34;Create order with ACID guarantees.&#34;&#34;&#34;\n \n with self.transaction() as conn:\n cursor = conn.cursor()\n \n # 1. Create order\n order_id = str(uuid.uuid4())\n totals = self.cart_service.calculate_totals(user_id)\n \n cursor.execute(&#34;&#34;&#34;\n INSERT INTO orders (id, user_id, status, subtotal, tax, shipping, total, shipping_address_id, created_at)\n VALUES (%s, %s, &#39;pending&#39;, %s, %s, %s, %s, %s, %s)\n RETURNING id\n &#34;&#34;&#34;, (\n order_id, user_id, totals[&#39;subtotal&#39;], totals[&#39;tax&#39;],\n totals[&#39;shipping&#39;], totals[&#39;total&#39;], shipping_address[&#39;id&#39;], datetime.now()\n ))\n \n # 2. Create order items\n for product_id, item in cart_items.items():\n cursor.execute(&#34;&#34;&#34;\n INSERT INTO order_items (id, order_id, product_id, quantity, unit_price, total_price)\n VALUES (%s, %s, %s, %s, %s, %s)\n &#34;&#34;&#34;, (\n str(uuid.uuid4()), order_id, product_id, item[&#39;quantity&#39;],\n item[&#39;price&#39;], item[&#39;price&#39;] * item[&#39;quantity&#39;]\n ))\n \n # 3. Reserve inventory\n for product_id, item in cart_items.items():\n cursor.execute(&#34;&#34;&#34;\n UPDATE inventory\n SET reserved_quantity = reserved_quantity &#43; %s\n WHERE product_id = %s AND quantity - reserved_quantity &gt;= %s\n RETURNING id\n &#34;&#34;&#34;, (item[&#39;quantity&#39;], product_id, item[&#39;quantity&#39;]))\n \n if cursor.rowcount == 0:\n raise InsufficientInventoryError(f&#34;Product {product_id} out of stock&#34;)\n \n # 4. Process payment\n payment_result = self.payment_service.process_payment(\n order_id, payment_method, totals[&#39;total&#39;]\n )\n \n if not payment_result[&#39;success&#39;]:\n raise PaymentFailedError(&#34;Payment processing failed&#34;)\n \n # 5. Update order status\n cursor.execute(&#34;&#34;&#34;\n UPDATE orders SET status = &#39;confirmed&#39; WHERE id = %s\n &#34;&#34;&#34;, (order_id,))\n \n # 6. Publish event to Kafka\n self.kafka_producer.send(&#39;order-events&#39;, {\n &#39;event_type&#39;: &#39;order_created&#39;,\n &#39;order_id&#39;: order_id,\n &#39;user_id&#39;: user_id,\n &#39;total&#39;: totals[&#39;total&#39;],\n &#39;timestamp&#39;: datetime.now().isoformat()\n })\n \n # 7. Clear cart\n self.cart_service.clear_cart(user_id)\n \n return {\n &#39;order_id&#39;: order_id,\n &#39;status&#39;: &#39;confirmed&#39;,\n &#39;total&#39;: totals[&#39;total&#39;]\n }<\/code><\/pre><h4 class=\"heading-element\" id=\"4-inventory-management\"><span>4. Inventory Management<\/span>\n <a href=\"#4-inventory-management\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>class InventoryService:\n def __init__(self):\n self.db = psycopg2.connect(&#34;dbname=inventory user=admin&#34;)\n self.kafka_consumer = KafkaConsumer(\n &#39;order-events&#39;,\n bootstrap_servers=[&#39;localhost:9092&#39;],\n value_deserializer=lambda m: json.loads(m.decode(&#39;utf-8&#39;))\n )\n \n def listen_for_orders(self):\n &#34;&#34;&#34;Background worker to process order events.&#34;&#34;&#34;\n for message in self.kafka_consumer:\n event = message.value\n \n if event[&#39;event_type&#39;] == &#39;order_confirmed&#39;:\n self.deduct_inventory(event[&#39;order_id&#39;])\n elif event[&#39;event_type&#39;] == &#39;order_cancelled&#39;:\n self.release_inventory(event[&#39;order_id&#39;])\n \n def deduct_inventory(self, order_id):\n &#34;&#34;&#34;Move reserved quantity to actual deduction.&#34;&#34;&#34;\n with self.transaction() as conn:\n cursor = conn.cursor()\n \n # Get order items\n cursor.execute(&#34;&#34;&#34;\n SELECT product_id, quantity FROM order_items WHERE order_id = %s\n &#34;&#34;&#34;, (order_id,))\n \n order_items = cursor.fetchall()\n \n for product_id, quantity in order_items:\n cursor.execute(&#34;&#34;&#34;\n UPDATE inventory\n SET quantity = quantity - %s,\n reserved_quantity = reserved_quantity - %s,\n updated_at = NOW()\n WHERE product_id = %s\n &#34;&#34;&#34;, (quantity, quantity, product_id))\n \n def check_availability(self, product_id, quantity):\n &#34;&#34;&#34;Check if product is available.&#34;&#34;&#34;\n cursor = self.db.cursor()\n cursor.execute(&#34;&#34;&#34;\n SELECT (quantity - reserved_quantity) as available\n FROM inventory\n WHERE product_id = %s\n &#34;&#34;&#34;, (product_id,))\n \n result = cursor.fetchone()\n return result[0] &gt;= quantity if result else False<\/code><\/pre><h3 class=\"heading-element\" id=\"search-implementation-with-elasticsearch\"><span>Search Implementation with Elasticsearch<\/span>\n <a href=\"#search-implementation-with-elasticsearch\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>from elasticsearch import Elasticsearch\n\nclass SearchService:\n def __init__(self):\n self.es = Elasticsearch([&#39;localhost:9200&#39;])\n \n def index_product(self, product):\n &#34;&#34;&#34;Index product for search.&#34;&#34;&#34;\n self.es.index(index=&#39;products&#39;, id=product[&#39;id&#39;], body={\n &#39;name&#39;: product[&#39;name&#39;],\n &#39;description&#39;: product[&#39;description&#39;],\n &#39;category&#39;: product[&#39;category&#39;],\n &#39;brand&#39;: product[&#39;brand&#39;],\n &#39;price&#39;: product[&#39;price&#39;],\n &#39;rating&#39;: product.get(&#39;rating&#39;, 0),\n &#39;num_reviews&#39;: product.get(&#39;num_reviews&#39;, 0),\n &#39;in_stock&#39;: product.get(&#39;in_stock&#39;, True),\n &#39;created_at&#39;: product[&#39;created_at&#39;]\n })\n \n def autocomplete(self, query):\n &#34;&#34;&#34;Suggest completions for search query.&#34;&#34;&#34;\n result = self.es.search(index=&#39;products&#39;, body={\n &#39;suggest&#39;: {\n &#39;product-suggest&#39;: {\n &#39;prefix&#39;: query,\n &#39;completion&#39;: {\n &#39;field&#39;: &#39;name.completion&#39;,\n &#39;size&#39;: 10,\n &#39;skip_duplicates&#39;: True\n }\n }\n }\n })\n \n suggestions = result[&#39;suggest&#39;][&#39;product-suggest&#39;][0][&#39;options&#39;]\n return [s[&#39;text&#39;] for s in suggestions]<\/code><\/pre><h3 class=\"heading-element\" id=\"scalability-optimizations\"><span>Scalability Optimizations<\/span>\n <a href=\"#scalability-optimizations\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>1. Database Sharding:<\/strong><\/p>\n<ul>\n<li>Shard orders by user_id (user affinity)<\/li>\n<li>Shard products by category (hot products in separate shards)<\/li>\n<\/ul>\n<p><strong>2. Caching Strategy:<\/strong><\/p>\n<ul>\n<li>Product details: Redis (1-hour TTL)<\/li>\n<li>Cart: Redis (7-day TTL)<\/li>\n<li>Search results: Redis (15-minute TTL)<\/li>\n<\/ul>\n<p><strong>3. CDN for Static Assets:<\/strong><\/p>\n<ul>\n<li>Product images<\/li>\n<li>CSS\/JS bundles<\/li>\n<li>Category thumbnails<\/li>\n<\/ul>\n<p><strong>4. Async Processing:<\/strong><\/p>\n<ul>\n<li>Order confirmation emails<\/li>\n<li>Inventory updates<\/li>\n<li>Analytics events<\/li>\n<li>Review indexing<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"key-takeaways-2\"><span>Key Takeaways<\/span>\n <a href=\"#key-takeaways-2\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li><strong>Use Redis for cart<\/strong> - Fast, supports TTL, scales horizontally<\/li>\n<li><strong>ACID for payments<\/strong> - PostgreSQL transactions ensure consistency<\/li>\n<li><strong>Elasticsearch for search<\/strong> - Fast full-text search with facets<\/li>\n<li><strong>Kafka for events<\/strong> - Decouple order processing<\/li>\n<li><strong>Inventory reservation<\/strong> - Prevent overselling during checkout<\/li>\n<li><strong>Idempotency keys<\/strong> - Prevent duplicate orders on retry<\/li>\n<\/ol>\n<hr>\n<h2 class=\"heading-element\" id=\"49-design-an-online-bookstore-amazon-books--goodreads\"><span>49. Design an Online Bookstore (Amazon Books \/ Goodreads)<\/span>\n <a href=\"#49-design-an-online-bookstore-amazon-books--goodreads\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"overview-3\"><span>Overview<\/span>\n <a href=\"#overview-3\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Design a comprehensive online bookstore with features for browsing, purchasing, reading reviews, creating reading lists, and personalized book recommendations.<\/p>\n<h3 class=\"heading-element\" id=\"problem-requirements-3\"><span>Problem Requirements<\/span>\n <a href=\"#problem-requirements-3\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Functional Requirements:<\/strong><\/p>\n<ul>\n<li>Book catalog browsing and search<\/li>\n<li>Book details, reviews, and ratings<\/li>\n<li>Purchase and download (for e-books)<\/li>\n<li>Reading lists and bookshelves<\/li>\n<li>Book recommendations<\/li>\n<li>Author pages and following<\/li>\n<li>Book clubs and discussions<\/li>\n<\/ul>\n<p><strong>Non-Functional Requirements:<\/strong><\/p>\n<ul>\n<li>50 million users<\/li>\n<li>10 million books in catalog<\/li>\n<li>1 billion reviews<\/li>\n<li>100K concurrent users<\/li>\n<li>Sub-second search response<\/li>\n<li>99.9% availability<\/li>\n<li>Personalized recommendations<\/li>\n<\/ul>\n<p><strong>Capacity Estimation:<\/strong><\/p>\n<ul>\n<li>Book metadata: 10M books \u00d7 5KB = 50 GB<\/li>\n<li>Book covers: 10M \u00d7 200KB = 2 TB<\/li>\n<li>Reviews: 1B \u00d7 500 bytes = 500 GB<\/li>\n<li>E-books: 1M \u00d7 5MB = 5 TB<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"high-level-architecture-2\"><span>High-Level Architecture<\/span>\n <a href=\"#high-level-architecture-2\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/microservices-architecture.webp\" title=\"Online Bookstore Architecture \u2014 Microservices for book catalog, inventory, orders, recommendations, and reviews behind an API gateway\" data-thumbnail=\"\/files\/microservices-architecture.webp\" data-sub-html=\"<h2>Microservices architecture diagram with API gateway and independent services \u2014 service-oriented design for scalability and maintainability<\/h2><p>Online Bookstore Architecture \u2014 Microservices for book catalog, inventory, orders, recommendations, and reviews behind an API gateway<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/microservices-architecture.webp' alt=\"Microservices architecture diagram with API gateway and independent services \u2014 service-oriented design for scalability and maintainability\"><\/a><figcaption class=\"image-caption\">Online Bookstore Architecture \u2014 Microservices for book catalog, inventory, orders, recommendations, and reviews behind an API gateway<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"database-schema-1\"><span>Database Schema<\/span>\n <a href=\"#database-schema-1\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Book Schema\nclass Book:\n isbn: str (PK)\n title: str\n subtitle: str\n author_ids: UUID[] (FK)\n publisher: str\n publication_date: date\n language: str\n pages: int\n format: enum(&#39;hardcover&#39;, &#39;paperback&#39;, &#39;ebook&#39;, &#39;audiobook&#39;)\n description: text\n genre_ids: UUID[]\n price: decimal(10, 2)\n avg_rating: float\n num_ratings: int\n cover_image_url: str\n ebook_url: str # S3 link\n created_at: timestamp\n\nclass Author:\n id: UUID (PK)\n name: str\n bio: text\n birth_date: date\n nationality: str\n photo_url: str\n num_followers: int\n\nclass Review:\n id: UUID (PK)\n user_id: UUID (FK)\n book_isbn: str (FK)\n rating: int (1-5)\n review_text: text\n num_likes: int\n num_comments: int\n spoiler: boolean\n created_at: timestamp\n updated_at: timestamp\n\nclass Bookshelf:\n id: UUID (PK)\n user_id: UUID (FK)\n name: str # &#34;Want to Read&#34;, &#34;Currently Reading&#34;, &#34;Read&#34;\n is_public: boolean\n created_at: timestamp\n\nclass BookshelfItem:\n id: UUID (PK)\n bookshelf_id: UUID (FK)\n book_isbn: str (FK)\n added_at: timestamp\n reading_progress: int # Percentage read\n started_at: timestamp\n finished_at: timestamp<\/code><\/pre><h3 class=\"heading-element\" id=\"core-service-implementations-1\"><span>Core Service Implementations<\/span>\n <a href=\"#core-service-implementations-1\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"1-book-service-with-multi-format-support\"><span>1. Book Service with Multi-Format Support<\/span>\n <a href=\"#1-book-service-with-multi-format-support\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>class BookService:\n def __init__(self):\n self.db = PostgreSQL()\n self.cache = Redis()\n self.s3 = S3Client()\n \n def get_book(self, isbn):\n &#34;&#34;&#34;Get book details with caching.&#34;&#34;&#34;\n cache_key = f&#34;book:{isbn}&#34;\n cached = self.cache.get(cache_key)\n if cached:\n return json.loads(cached)\n \n book = self.db.query(&#34;&#34;&#34;\n SELECT b.*, \n array_agg(DISTINCT a.name) as authors,\n array_agg(DISTINCT g.name) as genres\n FROM books b\n LEFT JOIN book_authors ba ON b.isbn = ba.book_isbn\n LEFT JOIN authors a ON ba.author_id = a.id\n LEFT JOIN book_genres bg ON b.isbn = bg.book_isbn\n LEFT JOIN genres g ON bg.genre_id = g.id\n WHERE b.isbn = %s\n GROUP BY b.isbn\n &#34;&#34;&#34;, (isbn,))\n \n if book:\n book_data = self.serialize(book)\n self.cache.setex(cache_key, 3600, json.dumps(book_data))\n return book_data\n \n return None\n \n def get_similar_books(self, isbn, limit=10):\n &#34;&#34;&#34;Find similar books using content-based filtering.&#34;&#34;&#34;\n book = self.get_book(isbn)\n if not book:\n return []\n \n # Find books with similar genres and authors\n similar = self.db.query(&#34;&#34;&#34;\n WITH book_features AS (\n SELECT \n b.isbn,\n array_agg(DISTINCT bg.genre_id) as genres,\n array_agg(DISTINCT ba.author_id) as authors\n FROM books b\n LEFT JOIN book_genres bg ON b.isbn = bg.book_isbn\n LEFT JOIN book_authors ba ON b.isbn = ba.book_isbn\n WHERE b.isbn = %s\n GROUP BY b.isbn\n )\n SELECT b.isbn, b.title, b.avg_rating,\n -- Calculate similarity score\n (\n -- Genre overlap\n cardinality(array_intersect(\n (SELECT genres FROM book_features),\n array_agg(DISTINCT bg2.genre_id)\n )) * 2 &#43;\n -- Author overlap\n cardinality(array_intersect(\n (SELECT authors FROM book_features),\n array_agg(DISTINCT ba2.author_id)\n )) * 3\n ) as similarity_score\n FROM books b\n LEFT JOIN book_genres bg2 ON b.isbn = bg2.book_isbn\n LEFT JOIN book_authors ba2 ON b.isbn = ba2.book_isbn\n WHERE b.isbn != %s\n GROUP BY b.isbn\n HAVING similarity_score &gt; 0\n ORDER BY similarity_score DESC, b.avg_rating DESC\n LIMIT %s\n &#34;&#34;&#34;, (isbn, isbn, limit))\n \n return similar\n \n def download_ebook(self, user_id, isbn):\n &#34;&#34;&#34;Generate pre-signed URL for e-book download.&#34;&#34;&#34;\n # Verify user owns the book\n if not self.verify_purchase(user_id, isbn):\n raise UnauthorizedError(&#34;User has not purchased this e-book&#34;)\n \n book = self.get_book(isbn)\n if not book[&#39;ebook_url&#39;]:\n raise NotFoundError(&#34;E-book not available for this title&#34;)\n \n # Generate pre-signed URL (valid for 1 hour)\n download_url = self.s3.generate_presigned_url(\n &#39;get_object&#39;,\n Params={&#39;Bucket&#39;: &#39;ebooks&#39;, &#39;Key&#39;: book[&#39;ebook_url&#39;]},\n ExpiresIn=3600\n )\n \n # Log download for analytics\n self.analytics.track(&#39;ebook_downloaded&#39;, {\n &#39;user_id&#39;: user_id,\n &#39;isbn&#39;: isbn,\n &#39;timestamp&#39;: datetime.now()\n })\n \n return download_url<\/code><\/pre><h4 class=\"heading-element\" id=\"2-review-service-with-cassandra-for-scale\"><span>2. Review Service with Cassandra for Scale<\/span>\n <a href=\"#2-review-service-with-cassandra-for-scale\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>from cassandra.cluster import Cluster\n\nclass ReviewService:\n def __init__(self):\n self.cluster = Cluster([&#39;cassandra-node-1&#39;, &#39;cassandra-node-2&#39;])\n self.session = self.cluster.connect(&#39;bookstore&#39;)\n self.cache = Redis()\n \n def create_review(self, user_id, book_isbn, rating, review_text, spoiler=False):\n &#34;&#34;&#34;Create a new book review.&#34;&#34;&#34;\n review_id = uuid.uuid4()\n \n # Insert into Cassandra (denormalized for fast reads)\n self.session.execute(&#34;&#34;&#34;\n INSERT INTO reviews_by_book (book_isbn, review_id, user_id, rating, review_text, spoiler, num_likes, created_at)\n VALUES (%s, %s, %s, %s, %s, %s, 0, %s)\n &#34;&#34;&#34;, (book_isbn, review_id, user_id, rating, review_text, spoiler, datetime.now()))\n \n self.session.execute(&#34;&#34;&#34;\n INSERT INTO reviews_by_user (user_id, review_id, book_isbn, rating, review_text, created_at)\n VALUES (%s, %s, %s, %s, %s, %s)\n &#34;&#34;&#34;, (user_id, review_id, book_isbn, rating, review_text, datetime.now()))\n \n # Update book&#39;s average rating (async via Kafka)\n self.kafka_producer.send(&#39;review-events&#39;, {\n &#39;event_type&#39;: &#39;review_created&#39;,\n &#39;book_isbn&#39;: book_isbn,\n &#39;rating&#39;: rating\n })\n \n # Invalidate cache\n self.cache.delete(f&#34;reviews:book:{book_isbn}&#34;)\n self.cache.delete(f&#34;book:{book_isbn}&#34;)\n \n return review_id\n \n def get_reviews_for_book(self, book_isbn, sort_by=&#39;helpful&#39;, page=1, page_size=20):\n &#34;&#34;&#34;Get paginated reviews for a book.&#34;&#34;&#34;\n cache_key = f&#34;reviews:book:{book_isbn}:sort:{sort_by}:page:{page}&#34;\n cached = self.cache.get(cache_key)\n if cached:\n return json.loads(cached)\n \n # Cassandra query (already sorted by timestamp)\n rows = self.session.execute(&#34;&#34;&#34;\n SELECT review_id, user_id, rating, review_text, num_likes, created_at\n FROM reviews_by_book\n WHERE book_isbn = %s\n LIMIT %s\n &#34;&#34;&#34;, (book_isbn, page_size))\n \n reviews = [self.serialize_review(row) for row in rows]\n \n # Sort in application layer if needed\n if sort_by == &#39;helpful&#39;:\n reviews.sort(key=lambda r: r[&#39;num_likes&#39;], reverse=True)\n elif sort_by == &#39;recent&#39;:\n reviews.sort(key=lambda r: r[&#39;created_at&#39;], reverse=True)\n \n # Cache for 5 minutes\n self.cache.setex(cache_key, 300, json.dumps(reviews))\n \n return reviews\n \n def like_review(self, review_id, user_id):\n &#34;&#34;&#34;Like a review (increment counter).&#34;&#34;&#34;\n # Use lightweight transaction to prevent double-liking\n result = self.session.execute(&#34;&#34;&#34;\n UPDATE reviews_by_book\n SET num_likes = num_likes &#43; 1\n WHERE book_isbn = %s AND review_id = %s\n IF NOT EXISTS IN user_likes(%s)\n &#34;&#34;&#34;, (book_isbn, review_id, user_id))\n \n if result.was_applied:\n # Record user liked this review\n self.session.execute(&#34;&#34;&#34;\n INSERT INTO review_likes (review_id, user_id, created_at)\n VALUES (%s, %s, %s)\n &#34;&#34;&#34;, (review_id, user_id, datetime.now()))\n \n return result.was_applied<\/code><\/pre><h4 class=\"heading-element\" id=\"3-recommendation-engine\"><span>3. Recommendation Engine<\/span>\n <a href=\"#3-recommendation-engine\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>import numpy as np\nfrom sklearn.metrics.pairwise import cosine_similarity\n\nclass RecommendationService:\n def __init__(self):\n self.db = PostgreSQL()\n self.feature_store = Redis()\n self.ml_model = self.load_model()\n \n def get_recommendations(self, user_id, num_recommendations=10):\n &#34;&#34;&#34;Generate personalized book recommendations.&#34;&#34;&#34;\n # 1. Get user&#39;s reading history\n user_books = self.get_user_books(user_id)\n if not user_books:\n return self.get_popular_books()\n \n # 2. Hybrid approach: Collaborative &#43; Content-based\n collaborative_recs = self.collaborative_filtering(user_id)\n content_recs = self.content_based_filtering(user_books)\n \n # 3. Combine with weighted average\n combined = self.combine_recommendations(\n collaborative_recs, \n content_recs, \n weights=[0.6, 0.4]\n )\n \n # 4. Re-rank with business logic\n final_recs = self.rerank(combined, user_id)\n \n return final_recs[:num_recommendations]\n \n def collaborative_filtering(self, user_id):\n &#34;&#34;&#34;Find similar users and recommend their liked books.&#34;&#34;&#34;\n # Get user embedding from feature store\n user_embedding_key = f&#34;user_embedding:{user_id}&#34;\n user_embedding = self.feature_store.get(user_embedding_key)\n \n if not user_embedding:\n # Generate embedding from user&#39;s ratings\n user_embedding = self.generate_user_embedding(user_id)\n self.feature_store.setex(user_embedding_key, 86400, user_embedding)\n \n user_embedding = np.frombuffer(user_embedding, dtype=np.float32)\n \n # Find similar users (using pre-computed embeddings)\n similar_users = self.find_similar_users(user_embedding, top_k=50)\n \n # Get books liked by similar users\n recommended_isbns = self.db.query(&#34;&#34;&#34;\n SELECT DISTINCT b.book_isbn, COUNT(*) as score\n FROM bookshelf_items b\n WHERE b.user_id IN %s\n AND b.book_isbn NOT IN (\n SELECT book_isbn FROM bookshelf_items WHERE user_id = %s\n )\n GROUP BY b.book_isbn\n ORDER BY score DESC\n LIMIT 100\n &#34;&#34;&#34;, (tuple(similar_users), user_id))\n \n return recommended_isbns\n \n def content_based_filtering(self, user_books):\n &#34;&#34;&#34;Recommend books similar to what user has read.&#34;&#34;&#34;\n # Extract genres and authors from user&#39;s books\n user_genres = Counter()\n user_authors = set()\n \n for book in user_books:\n user_genres.update(book[&#39;genres&#39;])\n user_authors.update(book[&#39;authors&#39;])\n \n # Find books matching preferred genres\/authors\n top_genres = [g for g, _ in user_genres.most_common(5)]\n \n similar_books = self.db.query(&#34;&#34;&#34;\n SELECT DISTINCT b.isbn, b.title,\n (\n -- Genre match score\n cardinality(array_intersect(\n ARRAY[%s]::uuid[],\n array_agg(bg.genre_id)\n )) * 2 &#43;\n -- Author match score\n CASE WHEN ba.author_id = ANY(%s::uuid[]) THEN 5 ELSE 0 END\n ) as score\n FROM books b\n LEFT JOIN book_genres bg ON b.isbn = bg.book_isbn\n LEFT JOIN book_authors ba ON b.isbn = ba.book_isbn\n WHERE b.isbn NOT IN %s\n GROUP BY b.isbn\n HAVING score &gt; 0\n ORDER BY score DESC, b.avg_rating DESC\n LIMIT 100\n &#34;&#34;&#34;, (top_genres, list(user_authors), tuple([b[&#39;isbn&#39;] for b in user_books])))\n \n return similar_books\n \n def generate_user_embedding(self, user_id):\n &#34;&#34;&#34;Generate 128-dim embedding for user based on preferences.&#34;&#34;&#34;\n # Get user&#39;s ratings\n ratings = self.db.query(&#34;&#34;&#34;\n SELECT book_isbn, rating FROM reviews_by_user WHERE user_id = %s\n &#34;&#34;&#34;, (user_id,))\n \n # Use matrix factorization (SVD) or neural network\n # Simplified: weighted average of book embeddings\n user_vector = np.zeros(128)\n total_weight = 0\n \n for isbn, rating in ratings:\n book_embedding = self.get_book_embedding(isbn)\n weight = rating \/ 5.0 # Normalize rating to 0-1\n user_vector &#43;= book_embedding * weight\n total_weight &#43;= weight\n \n if total_weight &gt; 0:\n user_vector \/= total_weight\n \n return user_vector.tobytes()<\/code><\/pre><h4 class=\"heading-element\" id=\"4-reading-list--bookshelf-management\"><span>4. Reading List \/ Bookshelf Management<\/span>\n <a href=\"#4-reading-list--bookshelf-management\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>class BookshelfService:\n def __init__(self):\n self.db = PostgreSQL()\n self.cache = Redis()\n \n def add_to_shelf(self, user_id, book_isbn, shelf_name=&#39;want_to_read&#39;):\n &#34;&#34;&#34;Add book to user&#39;s bookshelf.&#34;&#34;&#34;\n # Get or create shelf\n shelf = self.db.query(&#34;&#34;&#34;\n INSERT INTO bookshelves (user_id, name)\n VALUES (%s, %s)\n ON CONFLICT (user_id, name) DO UPDATE SET user_id = user_id\n RETURNING id\n &#34;&#34;&#34;, (user_id, shelf_name))\n \n shelf_id = shelf[&#39;id&#39;]\n \n # Add book to shelf\n self.db.execute(&#34;&#34;&#34;\n INSERT INTO bookshelf_items (bookshelf_id, book_isbn, added_at)\n VALUES (%s, %s, NOW())\n ON CONFLICT (bookshelf_id, book_isbn) DO NOTHING\n &#34;&#34;&#34;, (shelf_id, book_isbn))\n \n # Invalidate cache\n self.cache.delete(f&#34;bookshelf:{user_id}:{shelf_name}&#34;)\n \n return True\n \n def update_reading_progress(self, user_id, book_isbn, progress_percentage):\n &#34;&#34;&#34;Update reading progress for a book.&#34;&#34;&#34;\n self.db.execute(&#34;&#34;&#34;\n UPDATE bookshelf_items\n SET reading_progress = %s,\n started_at = COALESCE(started_at, NOW()),\n finished_at = CASE WHEN %s &gt;= 100 THEN NOW() ELSE NULL END\n WHERE bookshelf_id IN (\n SELECT id FROM bookshelves WHERE user_id = %s\n ) AND book_isbn = %s\n &#34;&#34;&#34;, (progress_percentage, progress_percentage, user_id, book_isbn))\n \n # Move to &#34;Read&#34; shelf if finished\n if progress_percentage &gt;= 100:\n self.add_to_shelf(user_id, book_isbn, &#39;read&#39;)\n \n def get_reading_stats(self, user_id):\n &#34;&#34;&#34;Get user&#39;s reading statistics.&#34;&#34;&#34;\n stats = self.db.query(&#34;&#34;&#34;\n SELECT \n COUNT(CASE WHEN finished_at IS NOT NULL THEN 1 END) as books_read,\n COUNT(CASE WHEN reading_progress &gt; 0 AND reading_progress &lt; 100 THEN 1 END) as currently_reading,\n COUNT(CASE WHEN reading_progress = 0 THEN 1 END) as want_to_read,\n AVG(CASE WHEN finished_at IS NOT NULL \n THEN EXTRACT(EPOCH FROM (finished_at - started_at)) \/ 86400 \n END) as avg_days_to_finish\n FROM bookshelf_items bi\n JOIN bookshelves bs ON bi.bookshelf_id = bs.id\n WHERE bs.user_id = %s\n &#34;&#34;&#34;, (user_id,))\n \n return stats<\/code><\/pre><h3 class=\"heading-element\" id=\"scalability-optimizations-1\"><span>Scalability Optimizations<\/span>\n <a href=\"#scalability-optimizations-1\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>1. Caching Strategy:<\/strong><\/p>\n<ul>\n<li>Book details: 1-hour TTL (rarely changes)<\/li>\n<li>Reviews: 5-minute TTL (frequently updated)<\/li>\n<li>User embeddings: 24-hour TTL<\/li>\n<li>Popular books: 1-hour TTL<\/li>\n<\/ul>\n<p><strong>2. Database Sharding:<\/strong><\/p>\n<ul>\n<li>Books: Shard by ISBN prefix<\/li>\n<li>Reviews: Shard by book_isbn (Cassandra partition key)<\/li>\n<li>Users: Shard by user_id<\/li>\n<\/ul>\n<p><strong>3. Search Optimization:<\/strong><\/p>\n<ul>\n<li>Elasticsearch with custom analyzers for book titles<\/li>\n<li>Autocomplete with edge n-grams<\/li>\n<li>Faceted search (genre, author, rating, price)<\/li>\n<\/ul>\n<p><strong>4. ML Model Serving:<\/strong><\/p>\n<ul>\n<li>TensorFlow Serving for real-time recommendations<\/li>\n<li>Batch processing for embedding generation (nightly)<\/li>\n<li>A\/B testing framework for recommendation algorithms<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"key-takeaways-3\"><span>Key Takeaways<\/span>\n <a href=\"#key-takeaways-3\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li><strong>Cassandra for reviews<\/strong> - Write-heavy, needs horizontal scaling<\/li>\n<li><strong>Hybrid recommendations<\/strong> - Combine collaborative + content-based<\/li>\n<li><strong>User embeddings<\/strong> - Cache in Redis for fast lookups<\/li>\n<li><strong>Pre-signed URLs<\/strong> - Secure e-book downloads without exposing S3<\/li>\n<li><strong>Reading progress tracking<\/strong> - Engages users, improves recommendations<\/li>\n<li><strong>Denormalization<\/strong> - Store avg_rating on book to avoid aggregation queries<\/li>\n<\/ol>\n<hr>\n<h2 class=\"heading-element\" id=\"50-design-an-online-multiplayer-game-backend-real-time\"><span>50. Design an Online Multiplayer Game Backend (Real-Time)<\/span>\n <a href=\"#50-design-an-online-multiplayer-game-backend-real-time\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"overview-4\"><span>Overview<\/span>\n <a href=\"#overview-4\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Design a scalable backend for a real-time multiplayer game (like Fortnite, Among Us, or League of Legends) supporting matchmaking, game state synchronization, chat, leaderboards, and anti-cheat.<\/p>\n<h3 class=\"heading-element\" id=\"problem-requirements-4\"><span>Problem Requirements<\/span>\n <a href=\"#problem-requirements-4\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Functional Requirements:<\/strong><\/p>\n<ul>\n<li>Player authentication and profiles<\/li>\n<li>Matchmaking system<\/li>\n<li>Real-time game state synchronization<\/li>\n<li>In-game chat<\/li>\n<li>Leaderboards and rankings<\/li>\n<li>Match history<\/li>\n<li>Anti-cheat detection<\/li>\n<li>Game replay system<\/li>\n<\/ul>\n<p><strong>Non-Functional Requirements:<\/strong><\/p>\n<ul>\n<li>10 million daily active users<\/li>\n<li>500K concurrent players<\/li>\n<li>Sub-50ms latency for game actions<\/li>\n<li>99.99% uptime during matches<\/li>\n<li>Support 100-player battle royale matches<\/li>\n<li>Global distribution (multiple regions)<\/li>\n<\/ul>\n<p><strong>Capacity Estimation:<\/strong><\/p>\n<ul>\n<li>500K concurrent \/ 100 players per match = 5,000 concurrent matches<\/li>\n<li>Game state updates: 20 updates\/sec \u00d7 100 players \u00d7 5,000 matches = 10M updates\/sec<\/li>\n<li>Bandwidth: 10M updates \u00d7 200 bytes = 2 GB\/sec<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"high-level-architecture-3\"><span>High-Level Architecture<\/span>\n <a href=\"#high-level-architecture-3\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/multiplayer-game-architecture.webp\" title=\"Multiplayer Game Architecture \u2014 High-level design for real-time online multiplayer games with matchmaking and regional servers\" data-thumbnail=\"\/files\/multiplayer-game-architecture.webp\" data-sub-html=\"<h2>Online multiplayer game high-level system architecture \u2014 matchmaking, game server fleet, anti-cheat, and multi-region deployment<\/h2><p>Multiplayer Game Architecture \u2014 High-level design for real-time online multiplayer games with matchmaking and regional servers<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/multiplayer-game-architecture.webp' alt=\"Online multiplayer game high-level system architecture \u2014 matchmaking, game server fleet, anti-cheat, and multi-region deployment\"><\/a><figcaption class=\"image-caption\">Multiplayer Game Architecture \u2014 High-level design for real-time online multiplayer games with matchmaking and regional servers<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"detailed-game-server-architecture\"><span>Detailed Game Server Architecture<\/span>\n <a href=\"#detailed-game-server-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/game-server-detailed-architecture.webp\" title=\"Game Server Detail \u2014 Internal game server design: game loop, physics, state sync, and Kafka event streaming\" data-thumbnail=\"\/files\/game-server-detailed-architecture.webp\" data-sub-html=\"<h2>Game server instance architecture \u2014 WebSocket handler, game loop, physics engine, collision detection, and state synchronization<\/h2><p>Game Server Detail \u2014 Internal game server design: game loop, physics, state sync, and Kafka event streaming<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/game-server-detailed-architecture.webp' alt=\"Game server instance architecture \u2014 WebSocket handler, game loop, physics engine, collision detection, and state synchronization\"><\/a><figcaption class=\"image-caption\">Game Server Detail \u2014 Internal game server design: game loop, physics, state sync, and Kafka event streaming<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"core-service-implementations-2\"><span>Core Service Implementations<\/span>\n <a href=\"#core-service-implementations-2\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"1-matchmaking-service\"><span>1. Matchmaking Service<\/span>\n <a href=\"#1-matchmaking-service\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>import asyncio\nimport redis\nfrom dataclasses import dataclass\nfrom typing import List\nimport heapq\n\n@dataclass\nclass Player:\n player_id: str\n skill_rating: int # ELO or MMR\n region: str\n queue_time: float\n preferred_roles: List[str]\n\nclass MatchmakingService:\n def __init__(self):\n self.redis = redis.Redis(host=&#39;localhost&#39;, port=6379)\n self.queue_key_prefix = &#34;matchmaking:queue:&#34;\n self.min_players = 10 # For 10-player matches\n self.max_skill_diff = 200 # Maximum MMR difference\n \n async def add_to_queue(self, player: Player):\n &#34;&#34;&#34;Add player to matchmaking queue.&#34;&#34;&#34;\n queue_key = f&#34;{self.queue_key_prefix}{player.region}&#34;\n \n # Add to Redis sorted set (score = skill_rating)\n self.redis.zadd(queue_key, {\n json.dumps({\n &#39;player_id&#39;: player.player_id,\n &#39;skill_rating&#39;: player.skill_rating,\n &#39;queue_time&#39;: time.time(),\n &#39;preferred_roles&#39;: player.preferred_roles\n }): player.skill_rating\n })\n \n # Start matchmaking check\n asyncio.create_task(self.find_matches(player.region))\n \n async def find_matches(self, region):\n &#34;&#34;&#34;Find suitable matches from queue.&#34;&#34;&#34;\n queue_key = f&#34;{self.queue_key_prefix}{region}&#34;\n \n # Get all players in queue\n players_data = self.redis.zrange(queue_key, 0, -1, withscores=True)\n players = [json.loads(p[0]) for p in players_data]\n \n if len(players) &lt; self.min_players:\n return # Not enough players\n \n # Sort by skill rating\n players.sort(key=lambda p: p[&#39;skill_rating&#39;])\n \n # Try to create balanced matches\n matches = []\n current_match = []\n \n for player in players:\n if not current_match:\n current_match.append(player)\n continue\n \n # Check skill difference\n avg_skill = sum(p[&#39;skill_rating&#39;] for p in current_match) \/ len(current_match)\n skill_diff = abs(player[&#39;skill_rating&#39;] - avg_skill)\n \n # Check queue time (expand search range over time)\n queue_time = time.time() - player[&#39;queue_time&#39;]\n max_diff = self.max_skill_diff &#43; (queue_time * 10) # &#43;10 MMR per second\n \n if skill_diff &lt;= max_diff:\n current_match.append(player)\n \n if len(current_match) == self.min_players:\n matches.append(current_match)\n current_match = []\n \n # Create game sessions for matches\n for match in matches:\n await self.create_game_session(match, region)\n \n # Remove matched players from queue\n for player in match:\n self.redis.zrem(queue_key, json.dumps(player))\n \n async def create_game_session(self, players: List[dict], region: str):\n &#34;&#34;&#34;Allocate game server and start match.&#34;&#34;&#34;\n # Find available game server in region\n game_server = await self.allocate_game_server(region)\n \n # Create game session\n session_id = str(uuid.uuid4())\n \n # Store session info in Redis\n self.redis.hmset(f&#34;game_session:{session_id}&#34;, {\n &#39;session_id&#39;: session_id,\n &#39;server_ip&#39;: game_server[&#39;ip&#39;],\n &#39;server_port&#39;: game_server[&#39;port&#39;],\n &#39;region&#39;: region,\n &#39;players&#39;: json.dumps([p[&#39;player_id&#39;] for p in players]),\n &#39;status&#39;: &#39;waiting&#39;,\n &#39;created_at&#39;: time.time()\n })\n \n # Notify players (via WebSocket or push notification)\n for player in players:\n await self.notify_player(player[&#39;player_id&#39;], {\n &#39;event&#39;: &#39;match_found&#39;,\n &#39;session_id&#39;: session_id,\n &#39;server_ip&#39;: game_server[&#39;ip&#39;],\n &#39;server_port&#39;: game_server[&#39;port&#39;]\n })\n \n return session_id\n \n async def allocate_game_server(self, region):\n &#34;&#34;&#34;Find or spin up game server instance.&#34;&#34;&#34;\n # Check for idle servers\n idle_servers_key = f&#34;game_servers:idle:{region}&#34;\n server_data = self.redis.lpop(idle_servers_key)\n \n if server_data:\n return json.loads(server_data)\n \n # No idle servers - request new instance from orchestrator\n new_server = await self.orchestrator.provision_server(region)\n return new_server<\/code><\/pre><h4 class=\"heading-element\" id=\"2-real-time-game-server\"><span>2. Real-Time Game Server<\/span>\n <a href=\"#2-real-time-game-server\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>import asyncio\nimport websockets\nfrom dataclasses import dataclass\nfrom typing import Dict, Set\nimport numpy as np\n\n@dataclass\nclass GameState:\n players: Dict[str, &#39;PlayerState&#39;]\n projectiles: List[&#39;Projectile&#39;]\n game_time: float\n tick_number: int\n\n@dataclass\nclass PlayerState:\n player_id: str\n position: np.ndarray # [x, y, z]\n velocity: np.ndarray\n rotation: float\n health: int\n score: int\n is_alive: bool\n\nclass GameServer:\n def __init__(self, session_id, max_players=100):\n self.session_id = session_id\n self.max_players = max_players\n self.tick_rate = 20 # 20 updates per second (50ms tick)\n self.tick_duration = 1.0 \/ self.tick_rate\n \n self.game_state = GameState(\n players={},\n projectiles=[],\n game_time=0.0,\n tick_number=0\n )\n \n self.connections: Dict[str, websockets.WebSocketServerProtocol] = {}\n self.input_buffer: Dict[str, List] = {} # Player inputs per tick\n \n async def start_server(self, host=&#39;0.0.0.0&#39;, port=8765):\n &#34;&#34;&#34;Start WebSocket server and game loop.&#34;&#34;&#34;\n # Start WebSocket server\n server = await websockets.serve(self.handle_connection, host, port)\n \n # Start game loop\n asyncio.create_task(self.game_loop())\n \n await server.wait_closed()\n \n async def handle_connection(self, websocket, path):\n &#34;&#34;&#34;Handle player WebSocket connection.&#34;&#34;&#34;\n try:\n # Authenticate player\n auth_msg = await websocket.recv()\n auth_data = json.loads(auth_msg)\n player_id = auth_data[&#39;player_id&#39;]\n session_token = auth_data[&#39;session_token&#39;]\n \n if not await self.verify_session(player_id, session_token):\n await websocket.close(1008, &#34;Authentication failed&#34;)\n return\n \n # Add player to game\n self.connections[player_id] = websocket\n self.spawn_player(player_id)\n \n # Listen for player inputs\n async for message in websocket:\n await self.handle_player_input(player_id, message)\n \n except websockets.exceptions.ConnectionClosed:\n # Player disconnected\n await self.handle_disconnect(player_id)\n \n async def game_loop(self):\n &#34;&#34;&#34;Main game loop running at fixed tick rate.&#34;&#34;&#34;\n last_tick = time.time()\n \n while True:\n current_time = time.time()\n delta_time = current_time - last_tick\n \n if delta_time &gt;= self.tick_duration:\n # Process game tick\n await self.process_tick(delta_time)\n last_tick = current_time\n else:\n # Sleep until next tick\n await asyncio.sleep(self.tick_duration - delta_time)\n \n async def process_tick(self, delta_time):\n &#34;&#34;&#34;Process one game tick.&#34;&#34;&#34;\n self.game_state.tick_number &#43;= 1\n self.game_state.game_time &#43;= delta_time\n \n # 1. Process player inputs\n for player_id, inputs in self.input_buffer.items():\n for input_data in inputs:\n self.apply_player_input(player_id, input_data)\n self.input_buffer.clear()\n \n # 2. Update physics\n self.update_physics(delta_time)\n \n # 3. Check collisions\n self.check_collisions()\n \n # 4. Update game logic (scoring, win conditions, etc.)\n self.update_game_logic()\n \n # 5. Broadcast state to all players\n await self.broadcast_state()\n \n # 6. Check for match end\n if self.check_match_end():\n await self.end_match()\n \n async def handle_player_input(self, player_id, message):\n &#34;&#34;&#34;Handle player input message.&#34;&#34;&#34;\n input_data = json.loads(message)\n \n # Validate input timestamp (prevent cheating)\n server_time = self.game_state.game_time\n client_time = input_data.get(&#39;timestamp&#39;, 0)\n \n if abs(server_time - client_time) &gt; 0.5: # 500ms tolerance\n # Potential lag or time manipulation\n await self.flag_suspicious_activity(player_id, &#34;timestamp_mismatch&#34;)\n return\n \n # Buffer input for next tick\n if player_id not in self.input_buffer:\n self.input_buffer[player_id] = []\n self.input_buffer[player_id].append(input_data)\n \n def apply_player_input(self, player_id, input_data):\n &#34;&#34;&#34;Apply player input to game state.&#34;&#34;&#34;\n if player_id not in self.game_state.players:\n return\n \n player = self.game_state.players[player_id]\n \n # Movement\n if &#39;move&#39; in input_data:\n direction = np.array(input_data[&#39;move&#39;]) # [x, y]\n speed = 5.0 # units per second\n player.velocity = direction * speed\n \n # Rotation\n if &#39;rotation&#39; in input_data:\n player.rotation = input_data[&#39;rotation&#39;]\n \n # Actions (shoot, use ability, etc.)\n if &#39;action&#39; in input_data:\n action = input_data[&#39;action&#39;]\n if action == &#39;shoot&#39;:\n self.create_projectile(player)\n elif action == &#39;use_ability&#39;:\n self.use_ability(player, input_data.get(&#39;ability_id&#39;))\n \n def update_physics(self, delta_time):\n &#34;&#34;&#34;Update positions based on velocities.&#34;&#34;&#34;\n for player in self.game_state.players.values():\n if player.is_alive:\n player.position &#43;= player.velocity * delta_time\n \n # Apply friction\n player.velocity *= 0.9\n \n # Clamp to map bounds\n player.position = np.clip(player.position, 0, 1000)\n \n # Update projectiles\n for projectile in self.game_state.projectiles:\n projectile.position &#43;= projectile.velocity * delta_time\n \n def check_collisions(self):\n &#34;&#34;&#34;Check for collisions between entities.&#34;&#34;&#34;\n # Player vs Projectile collisions\n for projectile in self.game_state.projectiles[:]:\n for player in self.game_state.players.values():\n if not player.is_alive:\n continue\n \n # Skip if projectile owner is the player\n if projectile.owner_id == player.player_id:\n continue\n \n # Simple distance-based collision\n distance = np.linalg.norm(player.position - projectile.position)\n if distance &lt; 2.0: # Collision radius\n # Apply damage\n player.health -= projectile.damage\n if player.health &lt;= 0:\n player.is_alive = False\n self.handle_player_death(player, projectile.owner_id)\n \n # Remove projectile\n self.game_state.projectiles.remove(projectile)\n break\n \n async def broadcast_state(self):\n &#34;&#34;&#34;Send game state to all connected players.&#34;&#34;&#34;\n # Use delta compression - only send changes\n state_update = {\n &#39;tick&#39;: self.game_state.tick_number,\n &#39;time&#39;: self.game_state.game_time,\n &#39;players&#39;: {}\n }\n \n # Only include alive players and their essential data\n for player_id, player in self.game_state.players.items():\n if player.is_alive:\n state_update[&#39;players&#39;][player_id] = {\n &#39;pos&#39;: player.position.tolist(),\n &#39;rot&#39;: player.rotation,\n &#39;hp&#39;: player.health\n }\n \n # Serialize and send\n message = json.dumps(state_update)\n \n # Broadcast to all players\n disconnected = []\n for player_id, websocket in self.connections.items():\n try:\n await websocket.send(message)\n except websockets.exceptions.ConnectionClosed:\n disconnected.append(player_id)\n \n # Clean up disconnected players\n for player_id in disconnected:\n await self.handle_disconnect(player_id)\n \n def handle_player_death(self, player, killer_id):\n &#34;&#34;&#34;Handle player elimination.&#34;&#34;&#34;\n # Update scores\n if killer_id in self.game_state.players:\n self.game_state.players[killer_id].score &#43;= 100\n \n # Broadcast death event\n asyncio.create_task(self.broadcast_event({\n &#39;event&#39;: &#39;player_eliminated&#39;,\n &#39;player_id&#39;: player.player_id,\n &#39;killer_id&#39;: killer_id\n }))<\/code><\/pre><h4 class=\"heading-element\" id=\"3-leaderboard-service\"><span>3. Leaderboard Service<\/span>\n <a href=\"#3-leaderboard-service\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>class LeaderboardService:\n def __init__(self):\n self.redis = redis.Redis(host=&#39;localhost&#39;, port=6379)\n self.cassandra = CassandraCluster([&#39;cassandra-node&#39;])\n \n def update_player_score(self, player_id, score, match_id):\n &#34;&#34;&#34;Update player&#39;s score after match.&#34;&#34;&#34;\n # Update global leaderboard (Redis Sorted Set)\n self.redis.zincrby(&#39;leaderboard:global&#39;, score, player_id)\n \n # Update regional leaderboard\n player_region = self.get_player_region(player_id)\n self.redis.zincrby(f&#39;leaderboard:region:{player_region}&#39;, score, player_id)\n \n # Store in Cassandra for history\n self.cassandra.execute(&#34;&#34;&#34;\n INSERT INTO player_scores (player_id, match_id, score, timestamp)\n VALUES (%s, %s, %s, %s)\n &#34;&#34;&#34;, (player_id, match_id, score, datetime.now()))\n \n def get_top_players(self, limit=100, region=None):\n &#34;&#34;&#34;Get top players on leaderboard.&#34;&#34;&#34;\n key = &#39;leaderboard:global&#39; if not region else f&#39;leaderboard:region:{region}&#39;\n \n # Get top players (descending order)\n top_players = self.redis.zrevrange(key, 0, limit-1, withscores=True)\n \n # Format response\n leaderboard = []\n for rank, (player_id, score) in enumerate(top_players, 1):\n player_data = self.get_player_profile(player_id)\n leaderboard.append({\n &#39;rank&#39;: rank,\n &#39;player_id&#39;: player_id,\n &#39;username&#39;: player_data[&#39;username&#39;],\n &#39;score&#39;: int(score),\n &#39;avatar&#39;: player_data[&#39;avatar_url&#39;]\n })\n \n return leaderboard\n \n def get_player_rank(self, player_id):\n &#34;&#34;&#34;Get player&#39;s current rank.&#34;&#34;&#34;\n rank = self.redis.zrevrank(&#39;leaderboard:global&#39;, player_id)\n return rank &#43; 1 if rank is not None else None<\/code><\/pre><h4 class=\"heading-element\" id=\"4-anti-cheat-service\"><span>4. Anti-Cheat Service<\/span>\n <a href=\"#4-anti-cheat-service\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>import numpy as np\nfrom sklearn.ensemble import IsolationForest\n\nclass AntiCheatService:\n def __init__(self):\n self.redis = redis.Redis()\n self.ml_model = IsolationForest(contamination=0.01)\n self.flags_threshold = 3 # Flags before ban\n \n async def analyze_player_behavior(self, player_id, match_data):\n &#34;&#34;&#34;Analyze player behavior for cheating patterns.&#34;&#34;&#34;\n features = self.extract_features(match_data)\n \n # Check for anomalies\n is_anomaly = self.ml_model.predict([features])[0] == -1\n \n if is_anomaly:\n await self.flag_player(player_id, &#34;behavioral_anomaly&#34;, features)\n \n # Rule-based checks\n if self.check_impossible_movement(match_data):\n await self.flag_player(player_id, &#34;impossible_movement&#34;)\n \n if self.check_perfect_accuracy(match_data):\n await self.flag_player(player_id, &#34;suspicious_accuracy&#34;)\n \n if self.check_wallhacks(match_data):\n await self.flag_player(player_id, &#34;wall_hacks&#34;)\n \n def extract_features(self, match_data):\n &#34;&#34;&#34;Extract features for ML model.&#34;&#34;&#34;\n return np.array([\n match_data[&#39;kills_per_minute&#39;],\n match_data[&#39;accuracy&#39;],\n match_data[&#39;headshot_percentage&#39;],\n match_data[&#39;reaction_time_avg&#39;],\n match_data[&#39;movement_speed_variance&#39;]\n ])\n \n async def flag_player(self, player_id, reason, details=None):\n &#34;&#34;&#34;Flag player for suspicious activity.&#34;&#34;&#34;\n flag_count = self.redis.incr(f&#34;anticheat:flags:{player_id}&#34;)\n \n # Log flag\n await self.log_flag(player_id, reason, details)\n \n # Auto-ban if threshold reached\n if flag_count &gt;= self.flags_threshold:\n await self.ban_player(player_id, reason=&#34;multiple_flags&#34;)<\/code><\/pre><h3 class=\"heading-element\" id=\"scalability-optimizations-2\"><span>Scalability Optimizations<\/span>\n <a href=\"#scalability-optimizations-2\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>1. Regional Game Servers:<\/strong><\/p>\n<ul>\n<li>Deploy servers in multiple regions (US, EU, Asia)<\/li>\n<li>Route players to nearest region for low latency<\/li>\n<\/ul>\n<p><strong>2. Game Server Fleet Management:<\/strong><\/p>\n<ul>\n<li>Kubernetes for orchestration<\/li>\n<li>Auto-scaling based on queue size<\/li>\n<li>Warm pool of idle servers<\/li>\n<\/ul>\n<p><strong>3. State Synchronization:<\/strong><\/p>\n<ul>\n<li>UDP for low-latency game state (tolerate packet loss)<\/li>\n<li>TCP for critical actions (weapon purchases, match results)<\/li>\n<\/ul>\n<p><strong>4. Database Strategy:<\/strong><\/p>\n<ul>\n<li>Cassandra for match history (write-heavy)<\/li>\n<li>Redis for real-time leaderboards<\/li>\n<li>PostgreSQL for player profiles<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"key-takeaways-4\"><span>Key Takeaways<\/span>\n <a href=\"#key-takeaways-4\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li><strong>Fixed tick rate<\/strong> - Ensures consistent gameplay across clients<\/li>\n<li><strong>Client-side prediction<\/strong> - Reduces perceived latency<\/li>\n<li><strong>Server reconciliation<\/strong> - Prevents cheating with authoritative server<\/li>\n<li><strong>UDP for game state<\/strong> - Lower latency than TCP<\/li>\n<li><strong>Regional distribution<\/strong> - Critical for low-latency global gameplay<\/li>\n<li><strong>Anti-cheat ML<\/strong> - Detect anomalous behavior patterns<\/li>\n<li><strong>Redis sorted sets<\/strong> - Perfect for leaderboards<\/li>\n<\/ol>\n<hr>\n<h2 class=\"heading-element\" id=\"51-design-a-collaborative-document-editor-google-docs\"><span>51. Design a Collaborative Document Editor (Google Docs)<\/span>\n <a href=\"#51-design-a-collaborative-document-editor-google-docs\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"overview-5\"><span>Overview<\/span>\n <a href=\"#overview-5\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Design a real-time collaborative document editor where multiple users can simultaneously edit the same document with conflict-free synchronization, version history, comments, and rich text formatting.<\/p>\n<h3 class=\"heading-element\" id=\"problem-requirements-5\"><span>Problem Requirements<\/span>\n <a href=\"#problem-requirements-5\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Functional Requirements:<\/strong><\/p>\n<ul>\n<li>Real-time collaborative editing<\/li>\n<li>Rich text formatting (bold, italic, headings, lists, etc.)<\/li>\n<li>Document sharing and permissions<\/li>\n<li>Comments and suggestions<\/li>\n<li>Version history<\/li>\n<li>Auto-save<\/li>\n<li>Offline editing with sync<\/li>\n<\/ul>\n<p><strong>Non-Functional Requirements:<\/strong><\/p>\n<ul>\n<li>100 million users<\/li>\n<li>50 million documents<\/li>\n<li>Support 100 concurrent editors per document<\/li>\n<li>Sub-100ms latency for edits<\/li>\n<li>99.99% availability<\/li>\n<li>Conflict-free synchronization<\/li>\n<\/ul>\n<p><strong>Capacity Estimation:<\/strong><\/p>\n<ul>\n<li>Storage: 50M documents \u00d7 100KB average = 5 TB<\/li>\n<li>Versions: 50M documents \u00d7 10 versions \u00d7 100KB = 50 TB<\/li>\n<li>Concurrent editors: 100K active documents \u00d7 10 editors = 1M WebSocket connections<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"high-level-architecture-4\"><span>High-Level Architecture<\/span>\n <a href=\"#high-level-architecture-4\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/websocket-connection-flow.webp\" title=\"Collaborative Editor Architecture \u2014 WebSocket-based real-time synchronization for concurrent document editing with CRDT conflict resolution\" data-thumbnail=\"\/files\/websocket-connection-flow.webp\" data-sub-html=\"<h2>Diagram showing WebSocket connection establishment and bidirectional communication flow \u2014 real-time data exchange for chat and live updates<\/h2><p>Collaborative Editor Architecture \u2014 WebSocket-based real-time synchronization for concurrent document editing with CRDT conflict resolution<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/websocket-connection-flow.webp' alt=\"Diagram showing WebSocket connection establishment and bidirectional communication flow \u2014 real-time data exchange for chat and live updates\"><\/a><figcaption class=\"image-caption\">Collaborative Editor Architecture \u2014 WebSocket-based real-time synchronization for concurrent document editing with CRDT conflict resolution<\/figcaption>\n <\/figure><\/p>\n<p><strong>Core Services:<\/strong><\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Service<\/th>\n <th>Responsibility<\/th>\n <th>Key Technology<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Auth Service<\/strong><\/td>\n <td>JWT token issuance and validation<\/td>\n <td>PostgreSQL + Redis sessions<\/td>\n <\/tr>\n <tr>\n <td><strong>Document Service<\/strong><\/td>\n <td>CRUD for documents, versions, metadata<\/td>\n <td>MongoDB (flexible schema for document content)<\/td>\n <\/tr>\n <tr>\n <td><strong>Collaboration Service<\/strong><\/td>\n <td>WebSocket server, CRDT operation broadcast<\/td>\n <td>Node.js + WebSockets + Redis pub\/sub<\/td>\n <\/tr>\n <tr>\n <td><strong>Presence Service<\/strong><\/td>\n <td>Track online users, cursor positions<\/td>\n <td>Redis (TTL-based)<\/td>\n <\/tr>\n <tr>\n <td><strong>Comment Service<\/strong><\/td>\n <td>Inline comments and suggestions<\/td>\n <td>PostgreSQL<\/td>\n <\/tr>\n <tr>\n <td><strong>Notification Service<\/strong><\/td>\n <td>Share\/mention notifications<\/td>\n <td>Kafka + FCM<\/td>\n <\/tr>\n <tr>\n <td><strong>Export Service<\/strong><\/td>\n <td>PDF\/DOCX export<\/td>\n <td>Background job workers<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Traffic Flow:<\/strong><\/p>\n<ol>\n<li>User opens document \u2192 Auth Service validates JWT.<\/li>\n<li>Client establishes <strong>WebSocket<\/strong> to Collaboration Service (sticky session via consistent hash on doc_id).<\/li>\n<li>User types \u2192 local CRDT operation applied immediately \u2192 operation sent over WebSocket.<\/li>\n<li>Collaboration Service broadcasts to all other editors of same document.<\/li>\n<li>Document Service <strong>auto-saves<\/strong> every 30 seconds to MongoDB.<\/li>\n<li>Version snapshots taken on explicit save or every 100 operations.<\/li>\n<\/ol>\n<p><strong>Scaling WebSockets:<\/strong><\/p>\n<ul>\n<li>Each Collaboration Service node handles ~10K concurrent WebSocket connections.<\/li>\n<li>Redis Pub\/Sub used to relay operations between nodes (editor A on Node 1, editor B on Node 2 \u2014 both receive updates via Redis channel <code>doc:&lt;doc_id&gt;<\/code>).<\/li>\n<li>At 1M connections \u2192 ~100 Collaboration Service nodes.<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"crdt-conflict-free-replicated-data-type-architecture\"><span>CRDT (Conflict-free Replicated Data Type) Architecture<\/span>\n <a href=\"#crdt-conflict-free-replicated-data-type-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/eventual-consistency-pattern.webp\" title=\"Eventual Consistency \u2014 Distributed system pattern balancing availability and consistency in CAP theorem\" data-thumbnail=\"\/files\/eventual-consistency-pattern.webp\" data-sub-html=\"<h2>Eventual consistency pattern in distributed systems \u2014 accepting temporary inconsistency for availability<\/h2><p>Eventual Consistency \u2014 Distributed system pattern balancing availability and consistency in CAP theorem<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/eventual-consistency-pattern.webp' alt=\"Eventual consistency pattern in distributed systems \u2014 accepting temporary inconsistency for availability\"><\/a><figcaption class=\"image-caption\">Eventual Consistency \u2014 Distributed system pattern balancing availability and consistency in CAP theorem<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"core-service-implementations-3\"><span>Core Service Implementations<\/span>\n <a href=\"#core-service-implementations-3\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"1-operational-transformation-ot--crdt-for-sync\"><span>1. Operational Transformation (OT) \/ CRDT for Sync<\/span>\n <a href=\"#1-operational-transformation-ot--crdt-for-sync\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>from dataclasses import dataclass\nfrom typing import List, Tuple\nimport uuid\n\n@dataclass\nclass Character:\n char: str\n char_id: str # Unique ID for each character\n site_id: str # Client\/user who created it\n position: List[int] # Fractional indexing position\n\nclass CRDTDocument:\n &#34;&#34;&#34;\n CRDT-based document using fractional indexing.\n Enables conflict-free concurrent editing.\n &#34;&#34;&#34;\n def __init__(self, doc_id):\n self.doc_id = doc_id\n self.characters: List[Character] = []\n self.tombstones = set() # Deleted character IDs\n self.version = 0\n \n def local_insert(self, char, position, site_id):\n &#34;&#34;&#34;Insert character at position (local operation).&#34;&#34;&#34;\n # Generate unique fractional position\n frac_pos = self.generate_position(position)\n \n char_id = f&#34;{site_id}:{uuid.uuid4()}&#34;\n character = Character(\n char=char,\n char_id=char_id,\n site_id=site_id,\n position=frac_pos\n )\n \n # Insert into sorted array\n self.characters.insert(position, character)\n self.version &#43;= 1\n \n # Return operation to broadcast\n return {\n &#39;type&#39;: &#39;insert&#39;,\n &#39;char_id&#39;: char_id,\n &#39;char&#39;: char,\n &#39;position&#39;: frac_pos,\n &#39;site_id&#39;: site_id,\n &#39;version&#39;: self.version\n }\n \n def local_delete(self, position, site_id):\n &#34;&#34;&#34;Delete character at position (local operation).&#34;&#34;&#34;\n if position &gt;= len(self.characters):\n return None\n \n character = self.characters[position]\n char_id = character.char_id\n \n # Mark as tombstone (don&#39;t actually delete for consistency)\n self.tombstones.add(char_id)\n self.characters.pop(position)\n self.version &#43;= 1\n \n return {\n &#39;type&#39;: &#39;delete&#39;,\n &#39;char_id&#39;: char_id,\n &#39;site_id&#39;: site_id,\n &#39;version&#39;: self.version\n }\n \n def remote_insert(self, operation):\n &#34;&#34;&#34;Apply remote insert operation.&#34;&#34;&#34;\n char_id = operation[&#39;char_id&#39;]\n \n # Check if already applied (idempotency)\n if any(c.char_id == char_id for c in self.characters):\n return False\n \n # Check if previously deleted\n if char_id in self.tombstones:\n return False\n \n character = Character(\n char=operation[&#39;char&#39;],\n char_id=char_id,\n site_id=operation[&#39;site_id&#39;],\n position=operation[&#39;position&#39;]\n )\n \n # Find insertion point based on fractional position\n insert_idx = self.find_insert_position(character.position)\n self.characters.insert(insert_idx, character)\n \n return True\n \n def remote_delete(self, operation):\n &#34;&#34;&#34;Apply remote delete operation.&#34;&#34;&#34;\n char_id = operation[&#39;char_id&#39;]\n \n # Find and remove character\n for i, char in enumerate(self.characters):\n if char.char_id == char_id:\n self.characters.pop(i)\n self.tombstones.add(char_id)\n return True\n \n # Already deleted\n self.tombstones.add(char_id)\n return False\n \n def generate_position(self, index):\n &#34;&#34;&#34;Generate fractional position for insertion.&#34;&#34;&#34;\n if index == 0:\n if not self.characters:\n return [0.5]\n else:\n prev_pos = [0.0]\n next_pos = self.characters[0].position\n elif index &gt;= len(self.characters):\n prev_pos = self.characters[-1].position\n next_pos = [1.0]\n else:\n prev_pos = self.characters[index - 1].position\n next_pos = self.characters[index].position\n \n # Generate position between prev and next\n return self.between(prev_pos, next_pos)\n \n def between(self, prev, next):\n &#34;&#34;&#34;Generate position between two fractional positions.&#34;&#34;&#34;\n # Simplified fractional indexing\n if len(prev) &lt; len(next):\n prev = prev &#43; [0] * (len(next) - len(prev))\n elif len(next) &lt; len(prev):\n next = next &#43; [0] * (len(prev) - len(next))\n \n result = []\n for i in range(len(prev)):\n if prev[i] &lt; next[i]:\n result.append((prev[i] &#43; next[i]) \/ 2)\n break\n result.append(prev[i])\n \n return result\n \n def to_string(self):\n &#34;&#34;&#34;Convert document to string.&#34;&#34;&#34;\n return &#39;&#39;.join(char.char for char in self.characters)\n \n def find_insert_position(self, position):\n &#34;&#34;&#34;Binary search for insertion position.&#34;&#34;&#34;\n left, right = 0, len(self.characters)\n while left &lt; right:\n mid = (left &#43; right) \/\/ 2\n if self.characters[mid].position &lt; position:\n left = mid &#43; 1\n else:\n right = mid\n return left<\/code><\/pre><h4 class=\"heading-element\" id=\"2-collaboration-service\"><span>2. Collaboration Service<\/span>\n <a href=\"#2-collaboration-service\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>import asyncio\nimport websockets\nfrom collections import defaultdict\n\nclass CollaborationService:\n def __init__(self):\n self.documents: Dict[str, CRDTDocument] = {}\n self.connections: Dict[str, Set[websockets.WebSocketServerProtocol]] = defaultdict(set)\n self.user_cursors: Dict[str, Dict] = defaultdict(dict) # doc_id -&gt; {user_id: cursor_pos}\n self.redis = redis.Redis()\n \n async def handle_connection(self, websocket, path):\n &#34;&#34;&#34;Handle WebSocket connection for collaborative editing.&#34;&#34;&#34;\n doc_id = None\n user_id = None\n \n try:\n # Authenticate and get document ID\n auth_msg = await websocket.recv()\n auth_data = json.loads(auth_msg)\n doc_id = auth_data[&#39;doc_id&#39;]\n user_id = auth_data[&#39;user_id&#39;]\n token = auth_data[&#39;token&#39;]\n \n # Verify permissions\n if not await self.verify_permission(user_id, doc_id, token):\n await websocket.close(1008, &#34;Unauthorized&#34;)\n return\n \n # Add to document&#39;s connection pool\n self.connections[doc_id].add(websocket)\n \n # Load document if not in memory\n if doc_id not in self.documents:\n await self.load_document(doc_id)\n \n # Send current document state\n await websocket.send(json.dumps({\n &#39;type&#39;: &#39;init&#39;,\n &#39;content&#39;: self.documents[doc_id].to_string(),\n &#39;version&#39;: self.documents[doc_id].version,\n &#39;active_users&#39;: await self.get_active_users(doc_id)\n }))\n \n # Broadcast user joined\n await self.broadcast_to_document(doc_id, {\n &#39;type&#39;: &#39;user_joined&#39;,\n &#39;user_id&#39;: user_id,\n &#39;username&#39;: await self.get_username(user_id)\n }, exclude=websocket)\n \n # Handle incoming operations\n async for message in websocket:\n await self.handle_operation(doc_id, user_id, message, websocket)\n \n except websockets.exceptions.ConnectionClosed:\n pass\n finally:\n # Clean up on disconnect\n if doc_id and user_id:\n self.connections[doc_id].discard(websocket)\n if doc_id in self.user_cursors:\n self.user_cursors[doc_id].pop(user_id, None)\n \n await self.broadcast_to_document(doc_id, {\n &#39;type&#39;: &#39;user_left&#39;,\n &#39;user_id&#39;: user_id\n })\n \n async def handle_operation(self, doc_id, user_id, message, sender_ws):\n &#34;&#34;&#34;Handle edit operation from client.&#34;&#34;&#34;\n operation = json.loads(message)\n op_type = operation[&#39;type&#39;]\n \n doc = self.documents[doc_id]\n \n if op_type == &#39;insert&#39;:\n # Apply operation to CRDT\n op = doc.remote_insert(operation)\n if op:\n # Broadcast to other users\n await self.broadcast_to_document(doc_id, operation, exclude=sender_ws)\n \n # Save to Redis for persistence\n await self.save_operation(doc_id, operation)\n \n elif op_type == &#39;delete&#39;:\n op = doc.remote_delete(operation)\n if op:\n await self.broadcast_to_document(doc_id, operation, exclude=sender_ws)\n await self.save_operation(doc_id, operation)\n \n elif op_type == &#39;cursor&#39;:\n # Update cursor position\n self.user_cursors[doc_id][user_id] = {\n &#39;position&#39;: operation[&#39;position&#39;],\n &#39;user_id&#39;: user_id\n }\n # Broadcast cursor update\n await self.broadcast_to_document(doc_id, {\n &#39;type&#39;: &#39;cursor_update&#39;,\n &#39;user_id&#39;: user_id,\n &#39;position&#39;: operation[&#39;position&#39;]\n }, exclude=sender_ws)\n \n elif op_type == &#39;format&#39;:\n # Handle formatting (bold, italic, etc.)\n await self.broadcast_to_document(doc_id, operation, exclude=sender_ws)\n await self.save_operation(doc_id, operation)\n \n async def broadcast_to_document(self, doc_id, message, exclude=None):\n &#34;&#34;&#34;Broadcast message to all users editing the document.&#34;&#34;&#34;\n if doc_id not in self.connections:\n return\n \n message_json = json.dumps(message)\n disconnected = []\n \n for ws in self.connections[doc_id]:\n if ws == exclude:\n continue\n \n try:\n await ws.send(message_json)\n except websockets.exceptions.ConnectionClosed:\n disconnected.append(ws)\n \n # Clean up disconnected clients\n for ws in disconnected:\n self.connections[doc_id].discard(ws)\n \n async def save_operation(self, doc_id, operation):\n &#34;&#34;&#34;Save operation to Redis for crash recovery.&#34;&#34;&#34;\n key = f&#34;doc_ops:{doc_id}&#34;\n self.redis.rpush(key, json.dumps(operation))\n self.redis.expire(key, 3600) # Keep for 1 hour\n \n # Trigger async save to database\n asyncio.create_task(self.persist_to_db(doc_id))<\/code><\/pre><h4 class=\"heading-element\" id=\"3-document-service\"><span>3. Document Service<\/span>\n <a href=\"#3-document-service\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>class DocumentService:\n def __init__(self):\n self.mongo = MongoClient(&#39;mongodb:\/\/localhost:27017\/&#39;)\n self.db = self.mongo[&#39;docs_db&#39;]\n self.cache = redis.Redis()\n self.s3 = S3Client()\n \n def create_document(self, user_id, title, content=&#34;&#34;):\n &#34;&#34;&#34;Create new document.&#34;&#34;&#34;\n doc_id = str(uuid.uuid4())\n \n doc = {\n &#39;_id&#39;: doc_id,\n &#39;title&#39;: title,\n &#39;content&#39;: content,\n &#39;owner_id&#39;: user_id,\n &#39;created_at&#39;: datetime.now(),\n &#39;updated_at&#39;: datetime.now(),\n &#39;version&#39;: 1,\n &#39;collaborators&#39;: [user_id]\n }\n \n self.db.documents.insert_one(doc)\n \n # Set default permissions\n self.permission_service.set_permission(doc_id, user_id, &#39;owner&#39;)\n \n return doc_id\n \n def get_document(self, doc_id):\n &#34;&#34;&#34;Retrieve document with caching.&#34;&#34;&#34;\n # Try cache first\n cache_key = f&#34;doc:{doc_id}&#34;\n cached = self.cache.get(cache_key)\n if cached:\n return json.loads(cached)\n \n # Fetch from MongoDB\n doc = self.db.documents.find_one({&#39;_id&#39;: doc_id})\n if doc:\n doc[&#39;_id&#39;] = str(doc[&#39;_id&#39;])\n self.cache.setex(cache_key, 300, json.dumps(doc))\n return doc\n \n return None\n \n def save_document(self, doc_id, content):\n &#34;&#34;&#34;Save document content (debounced auto-save).&#34;&#34;&#34;\n self.db.documents.update_one(\n {&#39;_id&#39;: doc_id},\n {\n &#39;$set&#39;: {\n &#39;content&#39;: content,\n &#39;updated_at&#39;: datetime.now()\n },\n &#39;$inc&#39;: {&#39;version&#39;: 1}\n }\n )\n \n # Invalidate cache\n self.cache.delete(f&#34;doc:{doc_id}&#34;)\n \n # Create version snapshot\n asyncio.create_task(self.create_version_snapshot(doc_id, content))\n \n async def create_version_snapshot(self, doc_id, content):\n &#34;&#34;&#34;Create version snapshot for history.&#34;&#34;&#34;\n doc = self.get_document(doc_id)\n \n version = {\n &#39;doc_id&#39;: doc_id,\n &#39;version&#39;: doc[&#39;version&#39;],\n &#39;content&#39;: content,\n &#39;created_at&#39;: datetime.now(),\n &#39;created_by&#39;: doc[&#39;owner_id&#39;]\n }\n \n # Store in S3 for cost efficiency\n s3_key = f&#34;versions\/{doc_id}\/v{doc[&#39;version&#39;]}.json&#34;\n self.s3.put_object(\n Bucket=&#39;document-versions&#39;,\n Key=s3_key,\n Body=json.dumps(version)\n )\n \n # Store metadata in PostgreSQL\n self.version_db.execute(&#34;&#34;&#34;\n INSERT INTO document_versions (doc_id, version, s3_key, created_at)\n VALUES (%s, %s, %s, %s)\n &#34;&#34;&#34;, (doc_id, doc[&#39;version&#39;], s3_key, datetime.now()))\n \n def get_version_history(self, doc_id, limit=50):\n &#34;&#34;&#34;Get version history for document.&#34;&#34;&#34;\n versions = self.version_db.query(&#34;&#34;&#34;\n SELECT version, created_at, created_by\n FROM document_versions\n WHERE doc_id = %s\n ORDER BY version DESC\n LIMIT %s\n &#34;&#34;&#34;, (doc_id, limit))\n \n return versions\n \n def restore_version(self, doc_id, version):\n &#34;&#34;&#34;Restore document to specific version.&#34;&#34;&#34;\n # Get version from S3\n s3_key = f&#34;versions\/{doc_id}\/v{version}.json&#34;\n version_data = self.s3.get_object(Bucket=&#39;document-versions&#39;, Key=s3_key)\n version_content = json.loads(version_data[&#39;Body&#39;].read())\n \n # Update current document\n self.save_document(doc_id, version_content[&#39;content&#39;])\n \n return version_content<\/code><\/pre><h4 class=\"heading-element\" id=\"4-permission-service\"><span>4. Permission Service<\/span>\n <a href=\"#4-permission-service\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>class PermissionService:\n def __init__(self):\n self.db = PostgreSQL()\n \n def set_permission(self, doc_id, user_id, role):\n &#34;&#34;&#34;Set user permission for document.&#34;&#34;&#34;\n self.db.execute(&#34;&#34;&#34;\n INSERT INTO document_permissions (doc_id, user_id, role, granted_at)\n VALUES (%s, %s, %s, NOW())\n ON CONFLICT (doc_id, user_id) DO UPDATE SET role = %s\n &#34;&#34;&#34;, (doc_id, user_id, role, role))\n \n def check_permission(self, user_id, doc_id, required_permission):\n &#34;&#34;&#34;Check if user has permission.&#34;&#34;&#34;\n result = self.db.query(&#34;&#34;&#34;\n SELECT role FROM document_permissions\n WHERE doc_id = %s AND user_id = %s\n &#34;&#34;&#34;, (doc_id, user_id))\n \n if not result:\n return False\n \n role = result[&#39;role&#39;]\n \n # Permission hierarchy: owner &gt; editor &gt; commenter &gt; viewer\n permissions = {\n &#39;owner&#39;: [&#39;read&#39;, &#39;write&#39;, &#39;share&#39;, &#39;delete&#39;],\n &#39;editor&#39;: [&#39;read&#39;, &#39;write&#39;, &#39;comment&#39;],\n &#39;commenter&#39;: [&#39;read&#39;, &#39;comment&#39;],\n &#39;viewer&#39;: [&#39;read&#39;]\n }\n \n return required_permission in permissions.get(role, [])<\/code><\/pre><h3 class=\"heading-element\" id=\"client-side-implementation-javascript\"><span>Client-Side Implementation (JavaScript)<\/span>\n <a href=\"#client-side-implementation-javascript\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>class CollaborativeEditor {\n constructor(docId, userId) {\n this.docId = docId;\n this.userId = userId;\n this.ws = null;\n this.crdt = new CRDTDocument(docId);\n this.editor = document.getElementById(&#39;editor&#39;);\n \n this.connect();\n }\n \n connect() {\n this.ws = new WebSocket(`wss:\/\/api.docs.com\/ws`);\n \n this.ws.onopen = () =&gt; {\n \/\/ Authenticate\n this.ws.send(JSON.stringify({\n doc_id: this.docId,\n user_id: this.userId,\n token: localStorage.getItem(&#39;auth_token&#39;)\n }));\n };\n \n this.ws.onmessage = (event) =&gt; {\n const message = JSON.parse(event.data);\n this.handleMessage(message);\n };\n \n \/\/ Listen for local edits\n this.editor.addEventListener(&#39;input&#39;, (e) =&gt; {\n this.handleLocalEdit(e);\n });\n }\n \n handleLocalEdit(event) {\n const operation = this.crdt.local_insert(\n event.data,\n this.editor.selectionStart,\n this.userId\n );\n \n \/\/ Send to server\n this.ws.send(JSON.stringify(operation));\n \n \/\/ Update UI optimistically\n \/\/ (already done by browser)\n }\n \n handleMessage(message) {\n if (message.type === &#39;insert&#39;) {\n \/\/ Apply remote insert\n if (this.crdt.remote_insert(message)) {\n \/\/ Update editor content\n this.updateEditor();\n }\n } else if (message.type === &#39;delete&#39;) {\n if (this.crdt.remote_delete(message)) {\n this.updateEditor();\n }\n } else if (message.type === &#39;cursor_update&#39;) {\n \/\/ Show other user&#39;s cursor\n this.showRemoteCursor(message.user_id, message.position);\n }\n }\n \n updateEditor() {\n const currentCursor = this.editor.selectionStart;\n this.editor.value = this.crdt.to_string();\n \/\/ Restore cursor position\n this.editor.setSelectionRange(currentCursor, currentCursor);\n }\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"scalability-optimizations-3\"><span>Scalability Optimizations<\/span>\n <a href=\"#scalability-optimizations-3\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>1. Connection Pooling:<\/strong><\/p>\n<ul>\n<li>WebSocket gateway with connection pooling<\/li>\n<li>Multiple servers behind load balancer with sticky sessions<\/li>\n<\/ul>\n<p><strong>2. Document Sharding:<\/strong><\/p>\n<ul>\n<li>Shard documents by doc_id<\/li>\n<li>Co-locate related data (document + operations)<\/li>\n<\/ul>\n<p><strong>3. Caching:<\/strong><\/p>\n<ul>\n<li>Recent documents in Redis<\/li>\n<li>Version snapshots in S3 (cold storage)<\/li>\n<\/ul>\n<p><strong>4. Compression:<\/strong><\/p>\n<ul>\n<li>Compress operations in transit<\/li>\n<li>Store compressed versions<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"key-takeaways-5\"><span>Key Takeaways<\/span>\n <a href=\"#key-takeaways-5\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li><strong>CRDT for conflict resolution<\/strong> - Guarantees eventual consistency<\/li>\n<li><strong>Fractional indexing<\/strong> - Enables efficient position tracking<\/li>\n<li><strong>WebSockets for real-time<\/strong> - Bi-directional communication<\/li>\n<li><strong>Debounced auto-save<\/strong> - Reduce database writes<\/li>\n<li><strong>Version snapshots<\/strong> - S3 for cost-efficient storage<\/li>\n<li><strong>Cursor broadcasting<\/strong> - Enhance collaboration awareness<\/li>\n<li><strong>Permission hierarchy<\/strong> - Fine-grained access control<\/li>\n<\/ol>\n<hr>\n<h2 class=\"heading-element\" id=\"52-design-an-online-food-delivery-system-uber-eats--doordash\"><span>52. Design an Online Food Delivery System (Uber Eats \/ DoorDash)<\/span>\n <a href=\"#52-design-an-online-food-delivery-system-uber-eats--doordash\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"overview-6\"><span>Overview<\/span>\n <a href=\"#overview-6\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Design a scalable food delivery platform connecting restaurants, delivery partners, and customers with real-time order tracking, optimized delivery routing, and dynamic pricing.<\/p>\n<h3 class=\"heading-element\" id=\"problem-requirements-6\"><span>Problem Requirements<\/span>\n <a href=\"#problem-requirements-6\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Functional Requirements:<\/strong><\/p>\n<ul>\n<li>Restaurant browsing and menu search<\/li>\n<li>Order placement and tracking<\/li>\n<li>Real-time delivery tracking<\/li>\n<li>Payment processing<\/li>\n<li>Ratings and reviews<\/li>\n<li>Push notifications<\/li>\n<li>Delivery optimization<\/li>\n<\/ul>\n<p><strong>Non-Functional Requirements:<\/strong><\/p>\n<ul>\n<li>50 million users<\/li>\n<li>500K restaurants<\/li>\n<li>100K concurrent orders<\/li>\n<li>Sub-second search latency<\/li>\n<li>Real-time location tracking with &lt; 5-second updates<\/li>\n<li>99.9% availability<\/li>\n<li>Support 50 geographic regions<\/li>\n<\/ul>\n<p><strong>Capacity Estimation:<\/strong><\/p>\n<ul>\n<li>100K concurrent orders \u00d7 50 states = 2M orders\/day<\/li>\n<li>Each order: Order details (1KB) + Location data (500B) = 1.5KB<\/li>\n<li>2M orders \u00d7 1.5KB \u00d7 30 days = 90 GB\/month (metadata)<\/li>\n<li>Location updates: 100K active deliveries \u00d7 1 update\/5sec = 20K updates\/sec<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"high-level-architecture-5\"><span>High-Level Architecture<\/span>\n <a href=\"#high-level-architecture-5\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/microservices-architecture.webp\" title=\"Food Delivery Architecture \u2014 Uber Eats\/DoorDash-style microservices: restaurant catalog, order management, real-time driver dispatch, and ETA service\" data-thumbnail=\"\/files\/microservices-architecture.webp\" data-sub-html=\"<h2>Microservices architecture diagram with API gateway and independent services \u2014 service-oriented design for scalability and maintainability<\/h2><p>Food Delivery Architecture \u2014 Uber Eats\/DoorDash-style microservices: restaurant catalog, order management, real-time driver dispatch, and ETA service<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/microservices-architecture.webp' alt=\"Microservices architecture diagram with API gateway and independent services \u2014 service-oriented design for scalability and maintainability\"><\/a><figcaption class=\"image-caption\">Food Delivery Architecture \u2014 Uber Eats\/DoorDash-style microservices: restaurant catalog, order management, real-time driver dispatch, and ETA service<\/figcaption>\n <\/figure><\/p>\n<p><strong>Core Services:<\/strong><\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Service<\/th>\n <th>Responsibility<\/th>\n <th>Key Technology<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Restaurant Service<\/strong><\/td>\n <td>Menu catalog, availability, ratings<\/td>\n <td>PostgreSQL + Elasticsearch<\/td>\n <\/tr>\n <tr>\n <td><strong>Order Service<\/strong><\/td>\n <td>Order lifecycle (placed \u2192 picked up \u2192 delivered)<\/td>\n <td>PostgreSQL (ACID) + Kafka<\/td>\n <\/tr>\n <tr>\n <td><strong>User Service<\/strong><\/td>\n <td>Customer profiles, addresses, payment methods<\/td>\n <td>PostgreSQL<\/td>\n <\/tr>\n <tr>\n <td><strong>Driver Service<\/strong><\/td>\n <td>Driver onboarding, availability status, ratings<\/td>\n <td>PostgreSQL<\/td>\n <\/tr>\n <tr>\n <td><strong>Matching Service<\/strong><\/td>\n <td>Assign nearest available driver to order<\/td>\n <td>Redis Geo + Kafka<\/td>\n <\/tr>\n <tr>\n <td><strong>Location Service<\/strong><\/td>\n <td>Real-time GPS tracking (drivers), ETA<\/td>\n <td>Redis (TTL 30s) + WebSockets<\/td>\n <\/tr>\n <tr>\n <td><strong>Payment Service<\/strong><\/td>\n <td>Charge customer, pay restaurant + driver<\/td>\n <td>Stripe + scheduled payouts<\/td>\n <\/tr>\n <tr>\n <td><strong>Notification Service<\/strong><\/td>\n <td>Order status push notifications<\/td>\n <td>FCM\/APNs + WebSockets<\/td>\n <\/tr>\n <tr>\n <td><strong>ETA Service<\/strong><\/td>\n <td>Estimate delivery time based on distance + traffic<\/td>\n <td>Road network graph (Dijkstra \/ A*)<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Traffic Flow:<\/strong><\/p>\n<ol>\n<li>Customer places order \u2192 <strong>Order Service<\/strong> creates order, charges payment, publishes <code>order_placed<\/code> to Kafka.<\/li>\n<li><strong>Matching Service<\/strong> finds nearest available driver using geospatial query (Redis GEO or PostGIS).<\/li>\n<li>Driver accepts \u2192 location updates stream every 5 seconds via WebSocket \u2192 <strong>Location Service<\/strong> stores in Redis.<\/li>\n<li>Customer app subscribes to real-time order updates (WebSocket or SSE).<\/li>\n<li>On delivery \u2192 Order Service marks <code>delivered<\/code>, triggers payment disbursement to restaurant and driver.<\/li>\n<\/ol>\n<p><strong>Key Design Challenges:<\/strong><\/p>\n<ul>\n<li><strong>Real-time location<\/strong>: 20K GPS updates\/sec \u2014 Redis GEOADD with 30-second TTL per driver.<\/li>\n<li><strong>Driver matching<\/strong>: Must be fast (&lt;200ms). Use geohash to narrow candidates, then rank by proximity + rating.<\/li>\n<li><strong>Surge pricing<\/strong>: Adjust prices dynamically based on demand\/supply ratio per geo zone.<\/li>\n<li><strong>Reliability<\/strong>: If driver goes offline mid-delivery, re-dispatch to another driver within SLA.<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"detailed-location--delivery-service-architecture\"><span>Detailed Location &amp; Delivery Service Architecture<\/span>\n <a href=\"#detailed-location--delivery-service-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/uber-architecture.webp\" title=\"Uber Architecture \u2014 Real-time matching, routing, and geospatial system design for on-demand transportation\" data-thumbnail=\"\/files\/uber-architecture.webp\" data-sub-html=\"<h2>Uber real-time matching and routing system architecture \u2014 geospatial matching for ride-sharing at planet scale<\/h2><p>Uber Architecture \u2014 Real-time matching, routing, and geospatial system design for on-demand transportation<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/uber-architecture.webp' alt=\"Uber real-time matching and routing system architecture \u2014 geospatial matching for ride-sharing at planet scale\"><\/a><figcaption class=\"image-caption\">Uber Architecture \u2014 Real-time matching, routing, and geospatial system design for on-demand transportation<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"core-service-implementations-4\"><span>Core Service Implementations<\/span>\n <a href=\"#core-service-implementations-4\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"1-order-service\"><span>1. Order Service<\/span>\n <a href=\"#1-order-service\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>from enum import Enum\nfrom datetime import datetime, timedelta\n\nclass OrderStatus(Enum):\n PENDING = &#34;pending&#34;\n CONFIRMED = &#34;confirmed&#34;\n PREPARING = &#34;preparing&#34;\n READY = &#34;ready&#34;\n PICKED_UP = &#34;picked_up&#34;\n IN_TRANSIT = &#34;in_transit&#34;\n DELIVERED = &#34;delivered&#34;\n CANCELLED = &#34;cancelled&#34;\n\nclass OrderService:\n def __init__(self):\n self.db = PostgreSQL()\n self.cache = redis.Redis()\n self.kafka_producer = KafkaProducer()\n self.payment_service = PaymentService()\n \n def create_order(self, customer_id, restaurant_id, items, delivery_address):\n &#34;&#34;&#34;Create new order.&#34;&#34;&#34;\n order_id = str(uuid.uuid4())\n \n # Calculate order total\n total = self.calculate_total(restaurant_id, items)\n \n # Create order in database\n order = {\n &#39;id&#39;: order_id,\n &#39;customer_id&#39;: customer_id,\n &#39;restaurant_id&#39;: restaurant_id,\n &#39;items&#39;: items,\n &#39;status&#39;: OrderStatus.PENDING.value,\n &#39;subtotal&#39;: total[&#39;subtotal&#39;],\n &#39;tax&#39;: total[&#39;tax&#39;],\n &#39;delivery_fee&#39;: total[&#39;delivery_fee&#39;],\n &#39;total&#39;: total[&#39;total&#39;],\n &#39;delivery_address&#39;: delivery_address,\n &#39;created_at&#39;: datetime.now(),\n &#39;estimated_delivery&#39;: datetime.now() &#43; timedelta(minutes=45)\n }\n \n self.db.execute(&#34;&#34;&#34;\n INSERT INTO orders (id, customer_id, restaurant_id, items, total, status, created_at)\n VALUES (%s, %s, %s, %s, %s, %s, %s)\n &#34;&#34;&#34;, (order_id, customer_id, restaurant_id, json.dumps(items), \n total[&#39;total&#39;], OrderStatus.PENDING.value, datetime.now()))\n \n # Process payment\n payment_result = self.payment_service.charge(\n customer_id, \n total[&#39;total&#39;],\n order_id\n )\n \n if not payment_result[&#39;success&#39;]:\n self.db.execute(&#34;UPDATE orders SET status = %s WHERE id = %s&#34;,\n (OrderStatus.CANCELLED.value, order_id))\n raise PaymentFailedError(&#34;Payment failed&#34;)\n \n # Update status\n self.update_order_status(order_id, OrderStatus.CONFIRMED)\n \n # Publish event to Kafka\n self.kafka_producer.send(&#39;order-events&#39;, {\n &#39;event_type&#39;: &#39;order_created&#39;,\n &#39;order_id&#39;: order_id,\n &#39;restaurant_id&#39;: restaurant_id,\n &#39;total&#39;: total[&#39;total&#39;],\n &#39;timestamp&#39;: datetime.now().isoformat()\n })\n \n return order_id\n \n def calculate_total(self, restaurant_id, items):\n &#34;&#34;&#34;Calculate order total with taxes and fees.&#34;&#34;&#34;\n # Get menu items and prices\n prices = self.get_menu_prices(restaurant_id, [item[&#39;menu_item_id&#39;] for item in items])\n \n subtotal = sum(\n prices[item[&#39;menu_item_id&#39;]] * item[&#39;quantity&#39;]\n for item in items\n )\n \n # Tax\n tax_rate = 0.08 # 8%\n tax = subtotal * tax_rate\n \n # Delivery fee (base &#43; distance-based)\n delivery_distance = self.get_delivery_distance(restaurant_id, None) # Will add address\n delivery_fee = 2.99 &#43; (delivery_distance * 0.5)\n \n # Promotions\/discounts\n discount = self.apply_promotions(restaurant_id, customer_id, subtotal)\n \n total = subtotal &#43; tax &#43; delivery_fee - discount\n \n return {\n &#39;subtotal&#39;: subtotal,\n &#39;tax&#39;: tax,\n &#39;delivery_fee&#39;: delivery_fee,\n &#39;discount&#39;: discount,\n &#39;total&#39;: total\n }\n \n def update_order_status(self, order_id, new_status):\n &#34;&#34;&#34;Update order status and notify parties.&#34;&#34;&#34;\n old_status = self.get_order_status(order_id)\n \n if new_status == old_status:\n return\n \n self.db.execute(&#34;&#34;&#34;\n UPDATE orders SET status = %s, updated_at = NOW() WHERE id = %s\n &#34;&#34;&#34;, (new_status.value, order_id))\n \n # Cache invalidation\n self.cache.delete(f&#34;order:{order_id}&#34;)\n \n # Publish status change event\n order = self.get_order(order_id)\n self.kafka_producer.send(&#39;order-status-updates&#39;, {\n &#39;order_id&#39;: order_id,\n &#39;status&#39;: new_status.value,\n &#39;customer_id&#39;: order[&#39;customer_id&#39;],\n &#39;restaurant_id&#39;: order[&#39;restaurant_id&#39;],\n &#39;timestamp&#39;: datetime.now().isoformat()\n })\n \n # Send notifications\n asyncio.create_task(self.notify_status_change(order, new_status))<\/code><\/pre><h4 class=\"heading-element\" id=\"2-delivery-service-with-real-time-tracking\"><span>2. Delivery Service with Real-Time Tracking<\/span>\n <a href=\"#2-delivery-service-with-real-time-tracking\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>import math\n\nclass DeliveryService:\n def __init__(self):\n self.redis = redis.Redis()\n self.db = PostgreSQL()\n self.route_optimizer = RouteOptimizer()\n \n def assign_delivery(self, order_id, restaurant_location, delivery_address):\n &#34;&#34;&#34;Assign order to nearest available delivery partner.&#34;&#34;&#34;\n # Find nearby available delivery partners (within 5 km)\n partners = self.find_nearby_partners(\n restaurant_location,\n radius_km=5\n )\n \n if not partners:\n raise NoAvailableDeliveryError(&#34;No delivery partners available&#34;)\n \n # Score each partner based on:\n # 1. Distance to restaurant\n # 2. Current load (orders being delivered)\n # 3. Acceptance rate\n # 4. Delivery speed rating\n \n scores = []\n for partner in partners:\n distance = self.calculate_distance(\n restaurant_location,\n partner[&#39;location&#39;]\n )\n \n load = partner[&#39;current_orders&#39;] # Higher load = lower priority\n acceptance_rate = partner[&#39;acceptance_rate&#39;]\n speed_rating = partner[&#39;avg_delivery_time&#39;]\n \n # Weighted score (lower is better)\n score = (\n distance * 0.4 &#43;\n load * 0.3 &#43;\n (1 - acceptance_rate) * 20 &#43;\n speed_rating * 0.1\n )\n \n scores.append((partner[&#39;id&#39;], score))\n \n # Select partner with best score\n best_partner_id = min(scores, key=lambda x: x[1])[0]\n \n # Create delivery record\n delivery_id = str(uuid.uuid4())\n self.db.execute(&#34;&#34;&#34;\n INSERT INTO deliveries (id, order_id, delivery_partner_id, status, created_at)\n VALUES (%s, %s, %s, &#39;assigned&#39;, NOW())\n &#34;&#34;&#34;, (delivery_id, order_id, best_partner_id))\n \n # Notify delivery partner\n asyncio.create_task(self.notify_delivery_partner(\n best_partner_id,\n order_id,\n restaurant_location,\n delivery_address\n ))\n \n return delivery_id\n \n def find_nearby_partners(self, location, radius_km=5):\n &#34;&#34;&#34;Find available delivery partners using geospatial index.&#34;&#34;&#34;\n # Use Redis geospatial commands\n nearby = self.redis.georadius(\n &#39;delivery_partners:locations&#39;,\n location[&#39;longitude&#39;],\n location[&#39;latitude&#39;],\n radius_km,\n unit=&#39;km&#39;,\n count=50,\n sort=&#39;ASC&#39;\n )\n \n partners = []\n for partner_id in nearby:\n partner_data = self.redis.hgetall(f&#34;partner:{partner_id}&#34;)\n if partner_data.get(b&#39;status&#39;) == b&#39;available&#39;:\n partners.append({\n &#39;id&#39;: partner_id.decode(),\n &#39;location&#39;: self.get_partner_location(partner_id),\n &#39;current_orders&#39;: int(partner_data.get(b&#39;current_orders&#39;, 0)),\n &#39;acceptance_rate&#39;: float(partner_data.get(b&#39;acceptance_rate&#39;, 0.9)),\n &#39;avg_delivery_time&#39;: float(partner_data.get(b&#39;avg_delivery_time&#39;, 30))\n })\n \n return partners\n \n def update_delivery_location(self, delivery_partner_id, latitude, longitude, timestamp=None):\n &#34;&#34;&#34;Update delivery partner&#39;s real-time location.&#34;&#34;&#34;\n location_key = f&#34;partner_location:{delivery_partner_id}&#34;\n \n # Store in Redis geospatial index\n self.redis.geoadd(\n &#39;delivery_partners:locations&#39;,\n longitude, latitude,\n delivery_partner_id\n )\n \n # Store detailed location with timestamp\n self.redis.hset(\n f&#34;partner:{delivery_partner_id}&#34;,\n mapping={\n &#39;latitude&#39;: latitude,\n &#39;longitude&#39;: longitude,\n &#39;last_update&#39;: timestamp or time.time()\n }\n )\n \n # Broadcast location update to customers tracking this delivery\n deliveries = self.get_active_deliveries_for_partner(delivery_partner_id)\n \n for delivery in deliveries:\n # Notify customer via WebSocket\n asyncio.create_task(self.broadcast_location_update(\n delivery[&#39;order_id&#39;],\n delivery_partner_id,\n {\n &#39;latitude&#39;: latitude,\n &#39;longitude&#39;: longitude,\n &#39;timestamp&#39;: timestamp or time.time()\n }\n ))\n \n def calculate_distance(self, location1, location2):\n &#34;&#34;&#34;Calculate distance between two coordinates (Haversine formula).&#34;&#34;&#34;\n R = 6371 # Earth&#39;s radius in km\n \n lat1, lon1 = math.radians(location1[&#39;latitude&#39;]), math.radians(location1[&#39;longitude&#39;])\n lat2, lon2 = math.radians(location2[&#39;latitude&#39;]), math.radians(location2[&#39;longitude&#39;])\n \n dlat = lat2 - lat1\n dlon = lon2 - lon1\n \n a = math.sin(dlat\/2)**2 &#43; math.cos(lat1) * math.cos(lat2) * math.sin(dlon\/2)**2\n c = 2 * math.asin(math.sqrt(a))\n \n return R * c\n \n def estimate_delivery_time(self, restaurant_location, delivery_location, current_time):\n &#34;&#34;&#34;Estimate delivery time based on distance, traffic, and partner metrics.&#34;&#34;&#34;\n distance = self.calculate_distance(restaurant_location, delivery_location)\n \n # Get traffic data (via Google Maps API or internal model)\n traffic_factor = self.get_traffic_factor(\n restaurant_location,\n delivery_location,\n current_time\n )\n \n # Base time: distance \/ average speed\n avg_speed = 30 # km\/h\n base_time = (distance \/ avg_speed) * 60 # minutes\n \n # Adjust for traffic\n estimated_time = base_time * traffic_factor\n \n # Add buffer for restaurant prep time (already prepared by this point)\n # Add buffer for minor delays\n estimated_time &#43;= 5 # 5 minute buffer\n \n return int(estimated_time)<\/code><\/pre><h4 class=\"heading-element\" id=\"3-search-service-with-filtering\"><span>3. Search Service with Filtering<\/span>\n <a href=\"#3-search-service-with-filtering\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>class RestaurantSearchService:\n def __init__(self):\n self.es = Elasticsearch([&#39;localhost:9200&#39;])\n \n def search_restaurants(self, location, query, filters):\n &#34;&#34;&#34;Search for restaurants with filters.&#34;&#34;&#34;\n \n es_query = {\n &#34;query&#34;: {\n &#34;bool&#34;: {\n &#34;must&#34;: [\n {\n &#34;multi_match&#34;: {\n &#34;query&#34;: query,\n &#34;fields&#34;: [&#34;name^3&#34;, &#34;cuisine_types&#34;, &#34;description&#34;]\n }\n },\n {\n &#34;geo_distance&#34;: {\n &#34;distance&#34;: &#34;5km&#34;,\n &#34;location&#34;: {\n &#34;lat&#34;: location[&#39;latitude&#39;],\n &#34;lon&#34;: location[&#39;longitude&#39;]\n }\n }\n }\n ],\n &#34;filter&#34;: []\n }\n },\n &#34;size&#34;: 20,\n &#34;sort&#34;: [\n {\n &#34;_geo_distance&#34;: {\n &#34;location&#34;: {\n &#34;lat&#34;: location[&#39;latitude&#39;],\n &#34;lon&#34;: location[&#39;longitude&#39;]\n },\n &#34;order&#34;: &#34;asc&#34;,\n &#34;unit&#34;: &#34;km&#34;\n }\n }\n ]\n }\n \n # Add filters\n if filters.get(&#39;cuisine&#39;):\n es_query[&#39;query&#39;][&#39;bool&#39;][&#39;filter&#39;].append({\n &#34;terms&#34;: {&#34;cuisine_types&#34;: filters[&#39;cuisine&#39;]}\n })\n \n if filters.get(&#39;min_rating&#39;):\n es_query[&#39;query&#39;][&#39;bool&#39;][&#39;filter&#39;].append({\n &#34;range&#34;: {&#34;rating&#34;: {&#34;gte&#34;: filters[&#39;min_rating&#39;]}}\n })\n \n if filters.get(&#39;delivery_time_max&#39;):\n es_query[&#39;query&#39;][&#39;bool&#39;][&#39;filter&#39;].append({\n &#34;range&#34;: {&#34;avg_delivery_time&#34;: {&#34;lte&#34;: filters[&#39;delivery_time_max&#39;]}}\n })\n \n if filters.get(&#39;price_level&#39;):\n es_query[&#39;query&#39;][&#39;bool&#39;][&#39;filter&#39;].append({\n &#34;terms&#34;: {&#34;price_level&#34;: filters[&#39;price_level&#39;]}\n })\n \n results = self.es.search(index=&#34;restaurants&#34;, body=es_query)\n \n # Format results\n restaurants = []\n for hit in results[&#39;hits&#39;][&#39;hits&#39;]:\n restaurant = hit[&#39;_source&#39;]\n restaurant[&#39;distance_km&#39;] = hit[&#39;sort&#39;][0]\n restaurants.append(restaurant)\n \n return restaurants<\/code><\/pre><h4 class=\"heading-element\" id=\"4-dynamic-pricing-service\"><span>4. Dynamic Pricing Service<\/span>\n <a href=\"#4-dynamic-pricing-service\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>class PricingService:\n def __init__(self):\n self.redis = redis.Redis()\n self.ml_model = self.load_demand_model()\n \n def calculate_delivery_fee(self, order_id, restaurant_location, delivery_location, \n order_time=None):\n &#34;&#34;&#34;Calculate dynamic delivery fee based on demand.&#34;&#34;&#34;\n \n if order_time is None:\n order_time = datetime.now()\n \n # Base distance\n distance = self.calculate_distance(restaurant_location, delivery_location)\n base_fee = 2.99 &#43; (distance * 0.5)\n \n # Demand multiplier (surge pricing)\n demand_level = self.estimate_demand(order_time)\n \n # demand_level values: 1.0 (normal) to 2.0&#43; (surge)\n # Capped at 2.5x to prevent excessive surge\n demand_multiplier = min(demand_level, 2.5)\n \n # Check for promotions\/discounts\n discount = self.get_applicable_promotions(\n restaurant_location,\n delivery_location,\n order_time\n )\n \n final_fee = base_fee * demand_multiplier\n final_fee = max(0, final_fee - discount) # Can&#39;t go negative\n \n return round(final_fee, 2)\n \n def estimate_demand(self, order_time):\n &#34;&#34;&#34;Estimate current demand using ML model.&#34;&#34;&#34;\n # Features: hour of day, day of week, recent order count, weather, etc.\n \n hour = order_time.hour\n day_of_week = order_time.weekday()\n \n # Get recent order count (last 30 minutes)\n recent_orders_key = f&#34;orders:recent:{int(order_time.timestamp() \/ 1800)}&#34;\n recent_count = int(self.redis.get(recent_orders_key) or 0)\n \n # Predict demand\n features = [hour, day_of_week, recent_count]\n demand_level = self.ml_model.predict([features])[0]\n \n return demand_level\n \n def get_applicable_promotions(self, restaurant_location, delivery_location, order_time):\n &#34;&#34;&#34;Find applicable promotions for order.&#34;&#34;&#34;\n promotions = []\n \n # Time-based promotions (lunch rush, late night, etc.)\n hour = order_time.hour\n if 23 &lt;= hour or hour &lt; 6:\n promotions.append((&#39;late_night_discount&#39;, 0.50))\n \n # Location-based promotions\n # ...\n \n # First-time user promotion\n # ...\n \n # Maximum discount\n return min(promo[1] for promo in promotions) if promotions else 0<\/code><\/pre><h3 class=\"heading-element\" id=\"scalability-optimizations-4\"><span>Scalability Optimizations<\/span>\n <a href=\"#scalability-optimizations-4\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>1. Geographic Sharding:<\/strong><\/p>\n<ul>\n<li>Shard data by region\/city<\/li>\n<li>Deploy services in each region for low latency<\/li>\n<\/ul>\n<p><strong>2. Redis Geospatial Index:<\/strong><\/p>\n<ul>\n<li>Track all active delivery partners and their locations<\/li>\n<li>Fast nearest-neighbor queries<\/li>\n<\/ul>\n<p><strong>3. Event-Driven Architecture:<\/strong><\/p>\n<ul>\n<li>Kafka for order events<\/li>\n<li>Asynchronous updates to analytics, ratings, etc.<\/li>\n<\/ul>\n<p><strong>4. Caching Strategy:<\/strong><\/p>\n<ul>\n<li>Restaurant data: 1-hour TTL<\/li>\n<li>Menu data: 2-hour TTL<\/li>\n<li>Delivery locations: Real-time (Redis)<\/li>\n<\/ul>\n<p><strong>5. Database Optimization:<\/strong><\/p>\n<ul>\n<li>Orders: PostgreSQL (strong consistency needed)<\/li>\n<li>Delivery history: Cassandra (time-series data)<\/li>\n<li>Locations: Redis (real-time)<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"key-takeaways-6\"><span>Key Takeaways<\/span>\n <a href=\"#key-takeaways-6\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li><strong>Geospatial indexing<\/strong> - Critical for location-based matching<\/li>\n<li><strong>Real-time tracking<\/strong> - WebSocket for live delivery updates<\/li>\n<li><strong>Dynamic pricing<\/strong> - ML-based demand estimation<\/li>\n<li><strong>Intelligent assignment<\/strong> - Score-based delivery partner matching<\/li>\n<li><strong>Regional deployment<\/strong> - Minimize latency for critical operations<\/li>\n<li><strong>Event sourcing<\/strong> - Track all state changes for auditing<\/li>\n<li><strong>Eventual consistency<\/strong> - Between regions is acceptable<\/li>\n<\/ol>\n<hr>\n<h2 class=\"heading-element\" id=\"53-advanced-system-design-topics\"><span>53. Advanced System Design Topics<\/span>\n <a href=\"#53-advanced-system-design-topics\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"testing-distributed-systems\"><span>Testing Distributed Systems<\/span>\n <a href=\"#testing-distributed-systems\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Challenges in Testing Distributed Systems:<\/strong><\/p>\n<ul>\n<li>Non-deterministic behavior<\/li>\n<li>Network partitions<\/li>\n<li>Race conditions<\/li>\n<li>Timing issues<\/li>\n<\/ul>\n<p><strong>Testing Strategies:<\/strong><\/p>\n<pre><code># Example: Chaos Engineering Test\nclass ChaosTest:\n def __init__(self, system):\n self.system = system\n \n def test_network_partition(self):\n &#34;&#34;&#34;Simulate network partition between nodes.&#34;&#34;&#34;\n # Isolate node from cluster\n self.system.isolate_node(&#39;node-2&#39;)\n \n # Verify system continues operating\n assert self.system.is_available()\n \n # Restore network\n self.system.restore_node(&#39;node-2&#39;)\n \n # Verify data consistency\n assert self.system.verify_consistency()\n \n def test_node_failure(self):\n &#34;&#34;&#34;Test system behavior when node crashes.&#34;&#34;&#34;\n self.system.kill_node(&#39;node-1&#39;)\n \n # Verify failover\n assert self.system.leader_elected()\n assert self.system.is_available()\n \n def test_latency_injection(self):\n &#34;&#34;&#34;Inject random latencies to find race conditions.&#34;&#34;&#34;\n self.system.inject_latency(min_ms=100, max_ms=500)\n \n # Run concurrent operations\n results = self.system.concurrent_writes(count=1000)\n \n # Verify eventual consistency\n self.system.wait_for_convergence()\n assert self.system.verify_consistency()<\/code><\/pre><p><strong>Tools:<\/strong><\/p>\n<ul>\n<li><strong>Jepsen<\/strong> - Distributed systems testing framework<\/li>\n<li><strong>TLA+<\/strong> - Formal specification language (by Leslie Lamport)<\/li>\n<li><strong>Chaos Monkey<\/strong> (Netflix) - Random failure injection<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"distributed-consensus-algorithms\"><span>Distributed Consensus Algorithms<\/span>\n <a href=\"#distributed-consensus-algorithms\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Paxos Algorithm:<\/strong><\/p>\n<pre><code>class PaxosNode:\n def __init__(self, node_id):\n self.node_id = node_id\n self.promised_id = None\n self.accepted_id = None\n self.accepted_value = None\n \n # Phase 1a: Proposer sends Prepare(n)\n def prepare(self, proposal_id):\n if proposal_id &gt; (self.promised_id or 0):\n self.promised_id = proposal_id\n return {\n &#39;promised&#39;: True,\n &#39;accepted_id&#39;: self.accepted_id,\n &#39;accepted_value&#39;: self.accepted_value\n }\n return {&#39;promised&#39;: False}\n \n # Phase 2a: Proposer sends Accept(n, value)\n def accept(self, proposal_id, value):\n if proposal_id &gt;= (self.promised_id or 0):\n self.promised_id = proposal_id\n self.accepted_id = proposal_id\n self.accepted_value = value\n return {&#39;accepted&#39;: True}\n return {&#39;accepted&#39;: False}<\/code><\/pre><p><strong>Raft Consensus (Simplified):<\/strong><\/p>\n<ul>\n<li>Leader election<\/li>\n<li>Log replication<\/li>\n<li>Safety guarantees<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"time-series-databases\"><span>Time Series Databases<\/span>\n <a href=\"#time-series-databases\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Use Cases:<\/strong><\/p>\n<ul>\n<li>Metrics monitoring<\/li>\n<li>IoT sensor data<\/li>\n<li>Financial market data<\/li>\n<li>Application performance monitoring (APM)<\/li>\n<\/ul>\n<p><strong>Design Considerations:<\/strong><\/p>\n<pre><code>class TimeSeriesDB:\n def __init__(self):\n self.segments = {} # time_bucket -&gt; data\n self.compression_enabled = True\n \n def write(self, metric_name, timestamp, value, tags=None):\n &#34;&#34;&#34;Write time series data point.&#34;&#34;&#34;\n # Determine time bucket (e.g., hourly)\n time_bucket = self.get_time_bucket(timestamp)\n \n # Create segment if not exists\n if time_bucket not in self.segments:\n self.segments[time_bucket] = TimeSeriesSegment(time_bucket)\n \n # Append to segment\n self.segments[time_bucket].append({\n &#39;metric&#39;: metric_name,\n &#39;timestamp&#39;: timestamp,\n &#39;value&#39;: value,\n &#39;tags&#39;: tags or {}\n })\n \n # Compress old segments\n if len(self.segments) &gt; 24: # Keep 24 hours hot\n self.compress_old_segments()\n \n def query(self, metric_name, start_time, end_time, aggregation=&#39;avg&#39;):\n &#34;&#34;&#34;Query time series data with aggregation.&#34;&#34;&#34;\n results = []\n \n # Find relevant segments\n segments = self.get_segments_in_range(start_time, end_time)\n \n for segment in segments:\n data_points = segment.get_metric(metric_name)\n results.extend(data_points)\n \n # Aggregate\n if aggregation == &#39;avg&#39;:\n return sum(r[&#39;value&#39;] for r in results) \/ len(results)\n elif aggregation == &#39;max&#39;:\n return max(r[&#39;value&#39;] for r in results)\n elif aggregation == &#39;min&#39;:\n return min(r[&#39;value&#39;] for r in results)\n \n return results\n \n def compress_old_segments(self):\n &#34;&#34;&#34;Compress segments older than 24 hours.&#34;&#34;&#34;\n cutoff = datetime.now() - timedelta(hours=24)\n \n for time_bucket, segment in list(self.segments.items()):\n if segment.timestamp &lt; cutoff:\n # Downsample and compress\n compressed = segment.downsample(factor=10)\n self.segments[time_bucket] = compressed<\/code><\/pre><p><strong>Popular Time Series DBs:<\/strong><\/p>\n<ul>\n<li>InfluxDB<\/li>\n<li>Prometheus<\/li>\n<li>TimescaleDB (PostgreSQL extension)<\/li>\n<li>Facebook Gorilla<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"authorization-systems\"><span>Authorization Systems<\/span>\n <a href=\"#authorization-systems\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Role-Based Access Control (RBAC):<\/strong><\/p>\n<pre><code>class RBACSystem:\n def __init__(self):\n self.roles = {} # role_name -&gt; permissions\n self.user_roles = {} # user_id -&gt; [role_names]\n \n def create_role(self, role_name, permissions):\n &#34;&#34;&#34;Create role with permissions.&#34;&#34;&#34;\n self.roles[role_name] = set(permissions)\n \n def assign_role(self, user_id, role_name):\n &#34;&#34;&#34;Assign role to user.&#34;&#34;&#34;\n if user_id not in self.user_roles:\n self.user_roles[user_id] = []\n self.user_roles[user_id].append(role_name)\n \n def check_permission(self, user_id, resource, action):\n &#34;&#34;&#34;Check if user has permission.&#34;&#34;&#34;\n user_roles = self.user_roles.get(user_id, [])\n \n for role in user_roles:\n permissions = self.roles.get(role, set())\n if f&#34;{resource}:{action}&#34; in permissions:\n return True\n \n return False\n\n# Usage\nrbac = RBACSystem()\nrbac.create_role(&#39;admin&#39;, [&#39;user:create&#39;, &#39;user:read&#39;, &#39;user:update&#39;, &#39;user:delete&#39;])\nrbac.create_role(&#39;viewer&#39;, [&#39;user:read&#39;])\nrbac.assign_role(&#39;user123&#39;, &#39;admin&#39;)\nrbac.check_permission(&#39;user123&#39;, &#39;user&#39;, &#39;delete&#39;) # True<\/code><\/pre><p><strong>Attribute-Based Access Control (ABAC):<\/strong><\/p>\n<ul>\n<li>More flexible than RBAC<\/li>\n<li>Uses attributes (user, resource, environment)<\/li>\n<li>Example: &ldquo;Allow if user.department == resource.department&rdquo;<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"batch-vs-stream-processing\"><span>Batch vs Stream Processing<\/span>\n <a href=\"#batch-vs-stream-processing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Batch Processing (MapReduce):<\/strong><\/p>\n<pre><code># Map function\ndef map_word_count(document):\n words = document.split()\n return [(word, 1) for word in words]\n\n# Reduce function\ndef reduce_word_count(word, counts):\n return (word, sum(counts))\n\n# MapReduce framework\nclass MapReduce:\n def __init__(self, num_workers=4):\n self.num_workers = num_workers\n \n def run(self, documents, map_fn, reduce_fn):\n # Map phase (parallel)\n mapped = []\n with ThreadPoolExecutor(max_workers=self.num_workers) as executor:\n futures = [executor.submit(map_fn, doc) for doc in documents]\n for future in futures:\n mapped.extend(future.result())\n \n # Shuffle phase (group by key)\n shuffled = {}\n for key, value in mapped:\n if key not in shuffled:\n shuffled[key] = []\n shuffled[key].append(value)\n \n # Reduce phase (parallel)\n results = []\n with ThreadPoolExecutor(max_workers=self.num_workers) as executor:\n futures = [\n executor.submit(reduce_fn, key, values)\n for key, values in shuffled.items()\n ]\n results = [future.result() for future in futures]\n \n return results<\/code><\/pre><p><strong>Stream Processing (Kafka Streams):<\/strong><\/p>\n<pre><code>from kafka import KafkaConsumer, KafkaProducer\nimport json\n\nclass StreamProcessor:\n def __init__(self, input_topic, output_topic):\n self.consumer = KafkaConsumer(\n input_topic,\n bootstrap_servers=[&#39;localhost:9092&#39;],\n value_deserializer=lambda m: json.loads(m.decode(&#39;utf-8&#39;))\n )\n self.producer = KafkaProducer(\n bootstrap_servers=[&#39;localhost:9092&#39;],\n value_serializer=lambda m: json.dumps(m).encode(&#39;utf-8&#39;)\n )\n self.output_topic = output_topic\n \n # Windowed aggregation state\n self.windows = {} # window_key -&gt; aggregate\n \n def process(self):\n &#34;&#34;&#34;Process stream with windowed aggregation.&#34;&#34;&#34;\n for message in self.consumer:\n event = message.value\n \n # Extract timestamp and create window key\n timestamp = event[&#39;timestamp&#39;]\n window_key = self.get_window_key(timestamp, window_size=60)\n \n # Update window aggregate\n if window_key not in self.windows:\n self.windows[window_key] = {&#39;count&#39;: 0, &#39;sum&#39;: 0}\n \n self.windows[window_key][&#39;count&#39;] &#43;= 1\n self.windows[window_key][&#39;sum&#39;] &#43;= event[&#39;value&#39;]\n \n # Emit windowed aggregate\n self.producer.send(self.output_topic, {\n &#39;window&#39;: window_key,\n &#39;avg&#39;: self.windows[window_key][&#39;sum&#39;] \/ self.windows[window_key][&#39;count&#39;],\n &#39;count&#39;: self.windows[window_key][&#39;count&#39;]\n })\n \n def get_window_key(self, timestamp, window_size):\n &#34;&#34;&#34;Get window key for tumbling window.&#34;&#34;&#34;\n return int(timestamp \/ window_size) * window_size<\/code><\/pre><h3 class=\"heading-element\" id=\"service-mesh\"><span>Service Mesh<\/span>\n <a href=\"#service-mesh\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>What is Service Mesh?<\/strong><\/p>\n<ul>\n<li>Infrastructure layer for service-to-service communication<\/li>\n<li>Handles service discovery, load balancing, encryption, authentication<\/li>\n<li>Popular: Istio, Linkerd, Consul<\/li>\n<\/ul>\n<p><strong>Key Components:<\/strong><\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/service-mesh-architecture.webp\" title=\"Service Mesh \u2014 Istio and service mesh architecture for microservices communication and observability\" data-thumbnail=\"\/files\/service-mesh-architecture.webp\" data-sub-html=\"<h2>Service mesh architecture with Istio pattern \u2014 service-to-service communication, observability, and reliability<\/h2><p>Service Mesh \u2014 Istio and service mesh architecture for microservices communication and observability<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/service-mesh-architecture.webp' alt=\"Service mesh architecture with Istio pattern \u2014 service-to-service communication, observability, and reliability\"><\/a><figcaption class=\"image-caption\">Service Mesh \u2014 Istio and service mesh architecture for microservices communication and observability<\/figcaption>\n <\/figure><\/p>\n<p><strong>Benefits:<\/strong><\/p>\n<ul>\n<li>Automatic retries and circuit breaking<\/li>\n<li>Mutual TLS encryption<\/li>\n<li>Distributed tracing<\/li>\n<li>Traffic shaping and routing<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"distributed-logging-and-tracing\"><span>Distributed Logging and Tracing<\/span>\n <a href=\"#distributed-logging-and-tracing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Distributed Tracing Architecture:<\/strong><\/p>\n<pre><code>import uuid\nfrom datetime import datetime\n\nclass Span:\n def __init__(self, trace_id, span_id, parent_span_id, operation_name):\n self.trace_id = trace_id\n self.span_id = span_id\n self.parent_span_id = parent_span_id\n self.operation_name = operation_name\n self.start_time = datetime.now()\n self.end_time = None\n self.tags = {}\n self.logs = []\n \n def set_tag(self, key, value):\n self.tags[key] = value\n \n def log(self, message):\n self.logs.append({\n &#39;timestamp&#39;: datetime.now(),\n &#39;message&#39;: message\n })\n \n def finish(self):\n self.end_time = datetime.now()\n \n def duration_ms(self):\n if self.end_time:\n return (self.end_time - self.start_time).total_seconds() * 1000\n return None\n\nclass Tracer:\n def __init__(self, service_name):\n self.service_name = service_name\n self.spans = []\n \n def start_span(self, operation_name, parent_span=None):\n &#34;&#34;&#34;Start a new span.&#34;&#34;&#34;\n trace_id = parent_span.trace_id if parent_span else str(uuid.uuid4())\n span_id = str(uuid.uuid4())\n parent_span_id = parent_span.span_id if parent_span else None\n \n span = Span(trace_id, span_id, parent_span_id, operation_name)\n span.set_tag(&#39;service.name&#39;, self.service_name)\n \n self.spans.append(span)\n return span\n \n def inject(self, span, carrier):\n &#34;&#34;&#34;Inject span context into carrier (e.g., HTTP headers).&#34;&#34;&#34;\n carrier[&#39;X-Trace-Id&#39;] = span.trace_id\n carrier[&#39;X-Span-Id&#39;] = span.span_id\n \n def extract(self, carrier):\n &#34;&#34;&#34;Extract span context from carrier.&#34;&#34;&#34;\n return {\n &#39;trace_id&#39;: carrier.get(&#39;X-Trace-Id&#39;),\n &#39;parent_span_id&#39;: carrier.get(&#39;X-Span-Id&#39;)\n }\n\n# Usage example\ntracer = Tracer(&#39;api-service&#39;)\n\n# In API handler\ndef handle_request(request):\n # Extract parent span context\n parent_context = tracer.extract(request.headers)\n \n # Start span\n span = tracer.start_span(&#39;handle_api_request&#39;)\n if parent_context[&#39;parent_span_id&#39;]:\n span.parent_span_id = parent_context[&#39;parent_span_id&#39;]\n span.trace_id = parent_context[&#39;trace_id&#39;]\n \n span.set_tag(&#39;http.method&#39;, request.method)\n span.set_tag(&#39;http.url&#39;, request.url)\n \n try:\n # Process request\n result = process_request(request, span)\n span.set_tag(&#39;http.status_code&#39;, 200)\n return result\n except Exception as e:\n span.set_tag(&#39;error&#39;, True)\n span.log(f&#34;Error: {str(e)}&#34;)\n raise\n finally:\n span.finish()<\/code><\/pre><p><strong>Popular Tracing Systems:<\/strong><\/p>\n<ul>\n<li>Jaeger (Uber)<\/li>\n<li>Zipkin (Twitter)<\/li>\n<li>AWS X-Ray<\/li>\n<li>Google Cloud Trace<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"anomaly-detection-systems\"><span>Anomaly Detection Systems<\/span>\n <a href=\"#anomaly-detection-systems\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Statistical Anomaly Detection:<\/strong><\/p>\n<pre><code>import numpy as np\nfrom sklearn.ensemble import IsolationForest\n\nclass AnomalyDetector:\n def __init__(self):\n self.model = IsolationForest(contamination=0.01) # 1% anomalies\n self.trained = False\n \n def train(self, historical_data):\n &#34;&#34;&#34;Train on historical metrics.&#34;&#34;&#34;\n # historical_data: array of [cpu, memory, latency, error_rate]\n self.model.fit(historical_data)\n self.trained = True\n \n def detect(self, current_metrics):\n &#34;&#34;&#34;Detect if current metrics are anomalous.&#34;&#34;&#34;\n if not self.trained:\n raise ValueError(&#34;Model not trained&#34;)\n \n prediction = self.model.predict([current_metrics])\n # -1 = anomaly, 1 = normal\n return prediction[0] == -1\n \n def get_anomaly_score(self, metrics):\n &#34;&#34;&#34;Get anomaly score (lower = more anomalous).&#34;&#34;&#34;\n return self.model.score_samples([metrics])[0]\n\n# Usage\ndetector = AnomalyDetector()\n\n# Train on normal data\nhistorical_data = np.array([\n [50, 60, 100, 0.1], # [CPU%, Memory%, Latency ms, Error%]\n [55, 62, 105, 0.12],\n [52, 58, 98, 0.09],\n # ... more normal samples\n])\ndetector.train(historical_data)\n\n# Check current metrics\ncurrent = [95, 90, 500, 5.0] # High CPU, memory, latency, errors\nif detector.detect(current):\n print(&#34;Anomaly detected! Alert the team.&#34;)<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"54-references-and-further-reading\"><span>54. References and Further Reading<\/span>\n <a href=\"#54-references-and-further-reading\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"-essential-books\"><span>\ud83d\udcda Essential Books<\/span>\n <a href=\"#-essential-books\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li>\n<p><strong>&ldquo;Designing Data-Intensive Applications&rdquo; by Martin Kleppmann<\/strong><\/p>\n<ul>\n<li>The definitive guide to distributed systems<\/li>\n<li>Covers replication, partitioning, transactions, consensus<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>&ldquo;System Design Interview&rdquo; by Alex Xu (Vol 1 &amp; 2)<\/strong><\/p>\n<ul>\n<li>Practical interview preparation<\/li>\n<li>Real-world system designs<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>&ldquo;Database Internals&rdquo; by Alex Petrov<\/strong><\/p>\n<ul>\n<li>Deep dive into database storage engines<\/li>\n<li>LSM trees, B-trees, replication<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>&ldquo;Designing Distributed Systems&rdquo; by Brendan Burns<\/strong><\/p>\n<ul>\n<li>Patterns for Kubernetes and containers<\/li>\n<li>Microservices architectures<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>&ldquo;Site Reliability Engineering&rdquo; by Google<\/strong><\/p>\n<ul>\n<li>SRE practices from Google<\/li>\n<li>Monitoring, incident response, automation<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<h3 class=\"heading-element\" id=\"-engineering-blogs\"><span>\ud83c\udf10 Engineering Blogs<\/span>\n <a href=\"#-engineering-blogs\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Recommended Reading:<\/strong><\/p>\n<ul>\n<li><strong><a href=\"https:\/\/netflixtechblog.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Netflix Tech Blog<\/a><\/strong> \u2014 Microservices, chaos engineering, CDN, Open Connect<\/li>\n<li><strong><a href=\"https:\/\/eng.uber.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Uber Engineering<\/a><\/strong> \u2014 Geospatial systems, real-time data processing, Kafka<\/li>\n<li><strong><a href=\"https:\/\/engineering.fb.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Meta Engineering<\/a><\/strong> \u2014 Scale challenges, distributed systems, React<\/li>\n<li><strong><a href=\"https:\/\/blog.twitter.com\/engineering\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Twitter Engineering<\/a><\/strong> \u2014 Real-time systems, graph databases, timeline service<\/li>\n<li><strong><a href=\"https:\/\/engineering.linkedin.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">LinkedIn Engineering<\/a><\/strong> \u2014 Data infrastructure, Kafka (invented here), Voldemort<\/li>\n<li><strong><a href=\"https:\/\/medium.com\/airbnb-engineering\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Airbnb Engineering<\/a><\/strong> \u2014 Microservices, ML infrastructure, data quality<\/li>\n<li><strong><a href=\"https:\/\/medium.com\/@Pinterest_Engineering\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Pinterest Engineering<\/a><\/strong> \u2014 Visual search, recommendation systems, Kraken<\/li>\n<li><strong><a href=\"https:\/\/dropbox.tech\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Dropbox Engineering<\/a><\/strong> \u2014 Distributed file systems, sync, Magic Pocket<\/li>\n<li><strong><a href=\"https:\/\/engineering.atspotify.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Spotify Engineering<\/a><\/strong> \u2014 Event-driven architecture, ML, Backstage<\/li>\n<li><strong><a href=\"https:\/\/instagram-engineering.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Instagram Engineering<\/a><\/strong> \u2014 Photo storage, feed ranking, Django at scale<\/li>\n<li><strong><a href=\"https:\/\/discord.com\/blog\/engineering\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Discord Engineering<\/a><\/strong> \u2014 Message storage, ScyllaDB migration, Rust<\/li>\n<li><strong><a href=\"https:\/\/blog.cloudflare.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Cloudflare Blog<\/a><\/strong> \u2014 Edge computing, DDoS mitigation, Workers<\/li>\n<li><strong><a href=\"https:\/\/blog.bytebytego.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ByteByteGo<\/a><\/strong> \u2014 System design visual explainers (Alex Xu)<\/li>\n<li><strong><a href=\"http:\/\/highscalability.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">High Scalability<\/a><\/strong> \u2014 Architecture case studies from real companies<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"-important-papers\"><span>\ud83d\udcdd Important Papers<\/span>\n <a href=\"#-important-papers\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li>\n<p><strong><a href=\"https:\/\/static.googleusercontent.com\/media\/research.google.com\/en\/\/archive\/gfs-sosp2003.pdf\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">&ldquo;The Google File System&rdquo; (GFS)<\/a><\/strong><\/p>\n<ul>\n<li>Distributed file system fundamentals, chunk servers, master node<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong><a href=\"https:\/\/static.googleusercontent.com\/media\/research.google.com\/en\/\/archive\/mapreduce-osdi04.pdf\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">&ldquo;MapReduce: Simplified Data Processing on Large Clusters&rdquo;<\/a><\/strong><\/p>\n<ul>\n<li>Batch processing framework, parallel computation model<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong><a href=\"https:\/\/static.googleusercontent.com\/media\/research.google.com\/en\/\/archive\/bigtable-osdi06.pdf\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">&ldquo;Bigtable: A Distributed Storage System&rdquo;<\/a><\/strong><\/p>\n<ul>\n<li>Wide-column store design, basis for HBase and Cassandra<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong><a href=\"https:\/\/www.allthingsdistributed.com\/files\/amazon-dynamo-sosp2007.pdf\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">&ldquo;Dynamo: Amazon&rsquo;s Highly Available Key-value Store&rdquo;<\/a><\/strong><\/p>\n<ul>\n<li>Eventual consistency, consistent hashing, vector clocks<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong><a href=\"https:\/\/static.googleusercontent.com\/media\/research.google.com\/en\/\/archive\/chubby-osdi06.pdf\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">&ldquo;The Chubby Lock Service for Loosely-Coupled Distributed Systems&rdquo;<\/a><\/strong><\/p>\n<ul>\n<li>Distributed locking and coordination, leader election<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong><a href=\"https:\/\/static.googleusercontent.com\/media\/research.google.com\/en\/\/archive\/spanner-osdi2012.pdf\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">&ldquo;Spanner: Google&rsquo;s Globally-Distributed Database&rdquo;<\/a><\/strong><\/p>\n<ul>\n<li>Globally consistent transactions using TrueTime API<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong><a href=\"https:\/\/notes.stephenholiday.com\/Kafka.pdf\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">&ldquo;Kafka: a Distributed Messaging System for Log Processing&rdquo;<\/a><\/strong><\/p>\n<ul>\n<li>Log-based message queue, sequential I\/O, consumer groups<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>&ldquo;Raft Consensus Algorithm&rdquo;<\/strong><\/p>\n<ul>\n<li>Understandable distributed consensus<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<h3 class=\"heading-element\" id=\"-online-courses\"><span>\ud83c\udf93 Online Courses<\/span>\n <a href=\"#-online-courses\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ul>\n<li><strong>MIT 6.824 Distributed Systems<\/strong> (Free on YouTube)<\/li>\n<li><strong>Grokking the System Design Interview<\/strong> (educative.io)<\/li>\n<li><strong>System Design for Tech Interviews<\/strong> (Udemy)<\/li>\n<li><strong>Distributed Systems<\/strong> by Chris Colohan (Coursera)<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"-tools-and-frameworks\"><span>\ud83d\udee0\ufe0f Tools and Frameworks<\/span>\n <a href=\"#-tools-and-frameworks\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Databases:<\/strong><\/p>\n<ul>\n<li>PostgreSQL, MySQL (SQL)<\/li>\n<li>MongoDB, Cassandra, DynamoDB (NoSQL)<\/li>\n<li>Redis, Memcached (Cache)<\/li>\n<li>Elasticsearch (Search)<\/li>\n<\/ul>\n<p><strong>Message Queues:<\/strong><\/p>\n<ul>\n<li>Apache Kafka<\/li>\n<li>RabbitMQ<\/li>\n<li>AWS SQS\/SNS<\/li>\n<li>Google Pub\/Sub<\/li>\n<\/ul>\n<p><strong>Infrastructure:<\/strong><\/p>\n<ul>\n<li>Kubernetes (Container orchestration)<\/li>\n<li>Docker (Containerization)<\/li>\n<li>Terraform (Infrastructure as Code)<\/li>\n<li>Prometheus + Grafana (Monitoring)<\/li>\n<\/ul>\n<p><strong>Load Balancers:<\/strong><\/p>\n<ul>\n<li>NGINX<\/li>\n<li>HAProxy<\/li>\n<li>AWS ELB\/ALB<\/li>\n<li>Google Cloud Load Balancing<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"-system-design-patterns-summary\"><span>\ud83d\udca1 System Design Patterns Summary<\/span>\n <a href=\"#-system-design-patterns-summary\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Pattern<\/th>\n <th>Use Case<\/th>\n <th>Example<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Load Balancing<\/strong><\/td>\n <td>Distribute traffic<\/td>\n <td>NGINX, HAProxy<\/td>\n <\/tr>\n <tr>\n <td><strong>Caching<\/strong><\/td>\n <td>Reduce latency<\/td>\n <td>Redis, CDN<\/td>\n <\/tr>\n <tr>\n <td><strong>Sharding<\/strong><\/td>\n <td>Scale database<\/td>\n <td>Hash-based, Range-based<\/td>\n <\/tr>\n <tr>\n <td><strong>Replication<\/strong><\/td>\n <td>High availability<\/td>\n <td>Master-Slave, Multi-Master<\/td>\n <\/tr>\n <tr>\n <td><strong>Rate Limiting<\/strong><\/td>\n <td>Prevent abuse<\/td>\n <td>Token bucket, Sliding window<\/td>\n <\/tr>\n <tr>\n <td><strong>Circuit Breaker<\/strong><\/td>\n <td>Fault tolerance<\/td>\n <td>Netflix Hystrix<\/td>\n <\/tr>\n <tr>\n <td><strong>CQRS<\/strong><\/td>\n <td>Separate read\/write<\/td>\n <td>Event sourcing<\/td>\n <\/tr>\n <tr>\n <td><strong>Event Sourcing<\/strong><\/td>\n <td>Audit trail<\/td>\n <td>Kafka, EventStore<\/td>\n <\/tr>\n <tr>\n <td><strong>Saga<\/strong><\/td>\n <td>Distributed transactions<\/td>\n <td>Choreography, Orchestration<\/td>\n <\/tr>\n <tr>\n <td><strong>Bulkhead<\/strong><\/td>\n <td>Isolation<\/td>\n <td>Thread pools, Process isolation<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"-interview-preparation-checklist\"><span>\ud83c\udfaf Interview Preparation Checklist<\/span>\n <a href=\"#-interview-preparation-checklist\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Fundamentals:<\/strong><\/p>\n<ul>\n<li>\u2705 CAP theorem<\/li>\n<li>\u2705 Consistency models<\/li>\n<li>\u2705 Load balancing algorithms<\/li>\n<li>\u2705 Caching strategies<\/li>\n<li>\u2705 Database indexing<\/li>\n<li>\u2705 Sharding and partitioning<\/li>\n<\/ul>\n<p><strong>Advanced Topics:<\/strong><\/p>\n<ul>\n<li>\u2705 Distributed transactions<\/li>\n<li>\u2705 Consensus algorithms (Paxos, Raft)<\/li>\n<li>\u2705 Consistent hashing<\/li>\n<li>\u2705 Rate limiting<\/li>\n<li>\u2705 Monitoring and alerting<\/li>\n<li>\u2705 Security best practices<\/li>\n<\/ul>\n<p><strong>System Designs:<\/strong><\/p>\n<ul>\n<li>\u2705 URL shortener<\/li>\n<li>\u2705 Twitter\/social media<\/li>\n<li>\u2705 Chat system<\/li>\n<li>\u2705 Video streaming (YouTube)<\/li>\n<li>\u2705 E-commerce platform<\/li>\n<li>\u2705 Ride-sharing service<\/li>\n<li>\u2705 Food delivery system<\/li>\n<li>\u2705 Search engine<\/li>\n<li>\u2705 Notification system<\/li>\n<li>\u2705 Collaborative editor<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"-online-resources\"><span>\ud83d\udd17 Online Resources<\/span>\n <a href=\"#-online-resources\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ul>\n<li><strong><a href=\"https:\/\/github.com\/donnemartin\/system-design-primer\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">System Design Primer<\/a><\/strong> \u2014 270K+ star GitHub repo with diagrams, flashcards, and mock interviews<\/li>\n<li><strong><a href=\"http:\/\/highscalability.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">High Scalability Blog<\/a><\/strong> \u2014 Real architecture case studies from companies<\/li>\n<li><strong><a href=\"https:\/\/blog.bytebytego.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ByteByteGo<\/a><\/strong> \u2014 System design visual explainers by Alex Xu<\/li>\n<li><strong><a href=\"https:\/\/interviewready.io\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">InterviewReady<\/a><\/strong> \u2014 Structured system design interview prep<\/li>\n<li><strong><a href=\"https:\/\/pdos.csail.mit.edu\/6.824\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">MIT 6.824 Distributed Systems<\/a><\/strong> \u2014 Free lecture notes and videos from MIT<\/li>\n<li><strong><a href=\"https:\/\/martin.kleppmann.com\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Martin Kleppmann&rsquo;s Blog<\/a><\/strong> \u2014 Deep dives on distributed systems concepts<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"-key-metrics-to-know\"><span>\ud83d\udcca Key Metrics to Know<\/span>\n <a href=\"#-key-metrics-to-know\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Metric<\/th>\n <th>Value<\/th>\n <th>Context<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>L1 cache reference<\/td>\n <td>0.5 ns<\/td>\n <td>CPU cache<\/td>\n <\/tr>\n <tr>\n <td>L2 cache reference<\/td>\n <td>7 ns<\/td>\n <td>CPU cache<\/td>\n <\/tr>\n <tr>\n <td>RAM reference<\/td>\n <td>100 ns<\/td>\n <td>Main memory<\/td>\n <\/tr>\n <tr>\n <td>SSD random read<\/td>\n <td>150 \u03bcs<\/td>\n <td>Storage<\/td>\n <\/tr>\n <tr>\n <td>HDD seek<\/td>\n <td>10 ms<\/td>\n <td>Storage<\/td>\n <\/tr>\n <tr>\n <td>Network within datacenter<\/td>\n <td>0.5 ms<\/td>\n <td>Internal<\/td>\n <\/tr>\n <tr>\n <td>Network between continents<\/td>\n <td>150 ms<\/td>\n <td>Global<\/td>\n <\/tr>\n <tr>\n <td>Read 1MB from RAM<\/td>\n <td>250 \u03bcs<\/td>\n <td>Sequential<\/td>\n <\/tr>\n <tr>\n <td>Read 1MB from SSD<\/td>\n <td>1,000 \u03bcs<\/td>\n <td>Sequential<\/td>\n <\/tr>\n <tr>\n <td>Read 1MB from HDD<\/td>\n <td>20,000 \u03bcs<\/td>\n <td>Sequential<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><hr>\n<p><strong>End of System Design Complete Guide<\/strong><\/p>\n<p><em>Remember: System design is about trade-offs. There&rsquo;s no single &ldquo;right&rdquo; answer\u2014only solutions that better fit specific requirements and constraints.<\/em><\/p>\n<p><em>Good luck with your interviews and building amazing systems! \ud83d\ude80<\/em><\/p>"},{"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","link":"https:\/\/samir.pages.dev\/posts\/operating-system\/","pubDate":"Fri, 29 Nov 2024 00:00:00 +0000","author":"netproxy@duck.com (Samir)","guid":"https:\/\/samir.pages.dev\/posts\/operating-system\/","category":"Computer-Science","description":"<img src=\"https:\/\/samir.pages.dev\/files\/operating-systems-og.webp\" alt=\"featured image\" referrerpolicy=\"no-referrer\"><p><strong>A Comprehensive Journey Through Operating System Concepts, Architecture, and Implementation<\/strong><\/p>\n<h2 id=\"introduction\">1. Introduction to Operating Systems<\/h2>\n<h3 class=\"heading-element\" id=\"what-is-an-operating-system\"><span>What is an Operating System?<\/span>\n <a href=\"#what-is-an-operating-system\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>An <strong>Operating System (OS)<\/strong> is sophisticated system software that manages computer hardware resources and provides an environment for application programs to execute efficiently. It serves as the critical intermediary layer between the bare metal hardware and user applications, abstracting complex hardware details while providing essential services.<\/p>\n<p><strong>Core Responsibilities:<\/strong><\/p>\n<ul>\n<li><strong>Resource Management<\/strong>: Orchestrates CPU time, memory allocation, storage access, and I\/O device operations<\/li>\n<li><strong>Hardware Abstraction<\/strong>: Shields applications from hardware complexity through standardized interfaces<\/li>\n<li><strong>Process Isolation<\/strong>: Ensures programs run independently without interfering with each other<\/li>\n<li><strong>Security Enforcement<\/strong>: Implements access controls and protection mechanisms<\/li>\n<li><strong>User Interface<\/strong>: Provides means for users to interact with the system<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"why-study-operating-systems\"><span>Why Study Operating Systems?<\/span>\n <a href=\"#why-study-operating-systems\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Understanding operating systems is fundamental for several critical reasons:<\/p>\n<div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Foundation of All Computing<\/strong><\/p>\n<ul>\n<li>Nearly every application executes atop an operating system<\/li>\n<li>OS behavior directly impacts application performance, security, and reliability<\/li>\n<li>Modern cloud infrastructure, mobile devices, and embedded systems all rely on OS principles<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Performance Optimization<\/strong><\/p>\n<ul>\n<li>Deep OS knowledge enables writing high-performance code<\/li>\n<li>Understanding scheduling, caching, and memory management leads to better algorithmic choices<\/li>\n<li>Ability to diagnose and resolve performance bottlenecks<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Security and Reliability<\/strong><\/p>\n<ul>\n<li>OS provides the trust boundary for system security<\/li>\n<li>Knowledge of OS internals helps identify and prevent vulnerabilities<\/li>\n<li>Understanding process isolation and memory protection prevents common exploits<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Career Advancement<\/strong><\/p>\n<ul>\n<li>Essential knowledge for system programming, embedded development, and infrastructure roles<\/li>\n<li>Required for roles in kernel development, device drivers, and low-level software<\/li>\n<li>Valuable for cloud architecture, distributed systems, and high-performance computing<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"historical-evolution\"><span>Historical Evolution<\/span>\n <a href=\"#historical-evolution\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"first-generation-1945-1955-vacuum-tubes-and-plugboards\"><span>First Generation (1945-1955): Vacuum Tubes and Plugboards<\/span>\n <a href=\"#first-generation-1945-1955-vacuum-tubes-and-plugboards\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><ul>\n<li><strong>Characteristics<\/strong>: No operating system concept; direct hardware programming<\/li>\n<li><strong>Programming Method<\/strong>: Machine language with plugboards for connections<\/li>\n<li><strong>Key Limitation<\/strong>: One program at a time; no multiprogramming<\/li>\n<li><strong>Example Systems<\/strong>: ENIAC, UNIVAC I<\/li>\n<\/ul>\n<h4 class=\"heading-element\" id=\"second-generation-1955-1965-transistors-and-batch-systems\"><span>Second Generation (1955-1965): Transistors and Batch Systems<\/span>\n <a href=\"#second-generation-1955-1965-transistors-and-batch-systems\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><ul>\n<li><strong>Key Innovation<\/strong>: Introduction of batch processing<\/li>\n<li><strong>Batch System Concept<\/strong>: Group similar jobs together; process sequentially<\/li>\n<li><strong>Advantages<\/strong>: Reduced setup time; improved resource utilization<\/li>\n<li><strong>Programming<\/strong>: Assembly language and early compilers (FORTRAN, COBOL)<\/li>\n<li><strong>Notable Systems<\/strong>: IBM 7094, Atlas Computer<\/li>\n<\/ul>\n<h4 class=\"heading-element\" id=\"third-generation-1965-1980-integrated-circuits-and-multiprogramming\"><span>Third Generation (1965-1980): Integrated Circuits and Multiprogramming<\/span>\n <a href=\"#third-generation-1965-1980-integrated-circuits-and-multiprogramming\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><ul>\n<li><strong>Breakthrough<\/strong>: Integrated circuits enabled smaller, faster, and cheaper computers<\/li>\n<li><strong>Multiprogramming<\/strong>: Multiple jobs in memory simultaneously; CPU switches between them<\/li>\n<li><strong>Time-Sharing<\/strong>: Interactive computing with multiple users<\/li>\n<li><strong>Key Concepts<\/strong>: Spooling, virtual memory, file systems<\/li>\n<li><strong>Landmark Systems<\/strong>: IBM System\/360, UNIX (1969), MULTICS<\/li>\n<\/ul>\n<h4 class=\"heading-element\" id=\"fourth-generation-1980-present-personal-computers\"><span>Fourth Generation (1980-Present): Personal Computers<\/span>\n <a href=\"#fourth-generation-1980-present-personal-computers\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><ul>\n<li><strong>Revolution<\/strong>: Microprocessors enabled personal computers<\/li>\n<li><strong>GUI Introduction<\/strong>: Graphical User Interfaces (Windows, Mac OS, X Window System)<\/li>\n<li><strong>Network Operating Systems<\/strong>: Client-server architectures<\/li>\n<li><strong>Notable Systems<\/strong>: MS-DOS, Windows, Mac OS, Linux (1991)<\/li>\n<\/ul>\n<h4 class=\"heading-element\" id=\"fifth-generation-1990-present-mobile-and-distributed-computing\"><span>Fifth Generation (1990-Present): Mobile and Distributed Computing<\/span>\n <a href=\"#fifth-generation-1990-present-mobile-and-distributed-computing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><ul>\n<li><strong>Mobile OS<\/strong>: iOS (2007), Android (2008)<\/li>\n<li><strong>Cloud Computing<\/strong>: Virtualization, containerization<\/li>\n<li><strong>Distributed Systems<\/strong>: Microservices architecture<\/li>\n<li><strong>IoT<\/strong>: Embedded and real-time operating systems<\/li>\n<li><strong>Modern Trends<\/strong>: Rust-based OS development, RISC-V architecture<\/li>\n<\/ul>\n<hr>\n<h2 id=\"hardware-fundamentals\">2. Computer Hardware Fundamentals<\/h2>\n<p>Understanding operating systems requires solid knowledge of underlying hardware components.<\/p>\n<h3 class=\"heading-element\" id=\"the-transistor-building-block-of-computing\"><span>The Transistor: Building Block of Computing<\/span>\n <a href=\"#the-transistor-building-block-of-computing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>A <strong>transistor<\/strong> is a semiconductor device functioning as an electronic switch with two states: conducting (ON) or non-conducting (OFF). These binary states represent the fundamental 0s and 1s of digital computing.<\/p>\n<p><strong>Key Facts:<\/strong><\/p>\n<ul>\n<li>Modern CPUs contain billions of transistors (Apple M2: ~20 billion; AMD EPYC: ~40+ billion)<\/li>\n<li>Transistor density follows Moore&rsquo;s Law: doubling approximately every two years<\/li>\n<li>Advanced manufacturing processes now reach 3nm and smaller<\/li>\n<\/ul>\n<p><strong>Transistor Evolution:<\/strong><\/p>\n<pre><code>1947: First transistor (Bell Labs) - Point-contact type\n1959: Planar silicon transistor - Foundation for ICs\n1971: Intel 4004 - 2,300 transistors\n2023: AMD EPYC Genoa - 57 billion transistors<\/code><\/pre><h3 class=\"heading-element\" id=\"integrated-circuits-ics\"><span>Integrated Circuits (ICs)<\/span>\n <a href=\"#integrated-circuits-ics\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>An <strong>Integrated Circuit<\/strong> is a complete electronic circuit fabricated on a single semiconductor wafer (typically silicon). It contains millions or billions of interconnected transistors, resistors, and capacitors forming complex circuitry.<\/p>\n<p><strong>Types:<\/strong><\/p>\n<ul>\n<li><strong>SSI (Small-Scale Integration)<\/strong>: &lt; 100 components<\/li>\n<li><strong>MSI (Medium-Scale Integration)<\/strong>: 100-1000 components<\/li>\n<li><strong>LSI (Large-Scale Integration)<\/strong>: 1000-100,000 components<\/li>\n<li><strong>VLSI (Very Large-Scale Integration)<\/strong>: &gt; 100,000 components<\/li>\n<li><strong>ULSI (Ultra Large-Scale Integration)<\/strong>: &gt; 1 million components<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"central-processing-unit-cpu\"><span>Central Processing Unit (CPU)<\/span>\n <a href=\"#central-processing-unit-cpu\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The <strong>CPU<\/strong> is the primary computational engine, often called the &ldquo;brain&rdquo; of the computer. It fetches instructions from memory, decodes them, executes operations, and stores results.<\/p>\n<h4 class=\"heading-element\" id=\"cpu-architecture-components\"><span>CPU Architecture Components<\/span>\n <a href=\"#cpu-architecture-components\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><strong>1. Arithmetic Logic Unit (ALU)<\/strong><\/p>\n<pre><code>Functions:\n\u251c\u2500\u2500 Arithmetic Operations\n\u2502 \u251c\u2500\u2500 Addition, Subtraction\n\u2502 \u251c\u2500\u2500 Multiplication, Division\n\u2502 \u2514\u2500\u2500 Modulo operations\n\u2514\u2500\u2500 Logical Operations\n \u251c\u2500\u2500 AND, OR, XOR, NOT\n \u251c\u2500\u2500 Bit shifting (Left\/Right)\n \u2514\u2500\u2500 Comparison operations<\/code><\/pre><p><strong>Implementation Concept:<\/strong><\/p>\n<pre><code>\/\/ Simplified ALU operation representation\nuint32_t alu_execute(uint8_t operation, uint32_t operand_a, uint32_t operand_b) {\n switch(operation) {\n case ALU_ADD: return operand_a &#43; operand_b;\n case ALU_SUB: return operand_a - operand_b;\n case ALU_AND: return operand_a &amp; operand_b;\n case ALU_OR: return operand_a | operand_b;\n case ALU_XOR: return operand_a ^ operand_b;\n case ALU_SHL: return operand_a &lt;&lt; operand_b;\n case ALU_SHR: return operand_a &gt;&gt; operand_b;\n default: return 0; \/\/ Invalid operation\n }\n}<\/code><\/pre><p><strong>2. Control Unit (CU)<\/strong><\/p>\n<p>The Control Unit orchestrates CPU operations by:<\/p>\n<ul>\n<li>Fetching instructions from memory<\/li>\n<li>Decoding instruction opcodes<\/li>\n<li>Generating control signals for other components<\/li>\n<li>Managing instruction sequencing<\/li>\n<\/ul>\n<p><strong>Instruction Cycle (Fetch-Decode-Execute):<\/strong><\/p>\n<pre><code>1. FETCH: Retrieve instruction from memory[PC]\n2. DECODE: Interpret opcode and extract operands\n3. EXECUTE: Perform operation via ALU or other units\n4. STORE: Write results back to registers\/memory\n5. INCREMENT: Update Program Counter (PC)<\/code><\/pre><p><strong>3. Registers<\/strong><\/p>\n<p>Registers are ultra-fast storage locations within the CPU for immediate data access.<\/p>\n<p><strong>Essential Register Types:<\/strong><\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Register<\/th>\n <th>Purpose<\/th>\n <th>Description<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Program Counter (PC)<\/strong><\/td>\n <td>Next Instruction Address<\/td>\n <td>Points to memory address of next instruction to execute<\/td>\n <\/tr>\n <tr>\n <td><strong>Instruction Register (IR)<\/strong><\/td>\n <td>Current Instruction<\/td>\n <td>Holds instruction currently being executed<\/td>\n <\/tr>\n <tr>\n <td><strong>Accumulator (ACC)<\/strong><\/td>\n <td>ALU Results<\/td>\n <td>Stores intermediate arithmetic\/logic results<\/td>\n <\/tr>\n <tr>\n <td><strong>Memory Address Register (MAR)<\/strong><\/td>\n <td>Memory Address<\/td>\n <td>Contains address for memory read\/write operations<\/td>\n <\/tr>\n <tr>\n <td><strong>Memory Data Register (MDR)<\/strong><\/td>\n <td>Memory Data<\/td>\n <td>Buffer for data read from or written to memory<\/td>\n <\/tr>\n <tr>\n <td><strong>Stack Pointer (SP)<\/strong><\/td>\n <td>Stack Top<\/td>\n <td>Points to top of the runtime stack<\/td>\n <\/tr>\n <tr>\n <td><strong>Status Register (FLAGS)<\/strong><\/td>\n <td>Processor State<\/td>\n <td>Contains condition codes (Zero, Carry, Negative, Overflow)<\/td>\n <\/tr>\n <tr>\n <td><strong>General Purpose Registers<\/strong><\/td>\n <td>Data Storage<\/td>\n <td>R0-R15 (ARM), RAX-R15 (x86-64) for various operations<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Register Example (x86-64 Architecture):<\/strong><\/p>\n<pre><code>; Example: Adding two numbers using registers\nmov rax, 10 ; Load 10 into RAX register\nmov rbx, 20 ; Load 20 into RBX register\nadd rax, rbx ; Add RBX to RAX (result: RAX = 30)\nmov [result], rax ; Store result in memory<\/code><\/pre><p><strong>4. Cache Memory Hierarchy<\/strong><\/p>\n<p>Modern CPUs employ multi-level caching to bridge the speed gap between processor and main memory.<\/p>\n<p><strong>Cache Levels:<\/strong><\/p>\n<pre><code>CPU Registers (&lt; 1 KB)\n \u2193 Access: 0 cycles\nL1 Cache (32-64 KB per core)\n \u2193 Access: 1-4 cycles\nL2 Cache (256-512 KB per core)\n \u2193 Access: 10-20 cycles\nL3 Cache (8-64 MB shared)\n \u2193 Access: 40-70 cycles\nMain Memory (4-256 GB)\n \u2193 Access: 100-300 cycles\nSSD Storage (128 GB - 4 TB)\n \u2193 Access: ~50,000 cycles\nHDD Storage (500 GB - 20 TB)\n \u2193 Access: ~10,000,000 cycles<\/code><\/pre><p><strong>Cache Properties:<\/strong><\/p>\n<ul>\n<li><strong>Cache Line<\/strong>: Typically 64 bytes; unit of transfer between cache levels<\/li>\n<li><strong>Cache Hit<\/strong>: Data found in cache (fast access)<\/li>\n<li><strong>Cache Miss<\/strong>: Data not in cache, must fetch from lower level (slow)<\/li>\n<li><strong>Cache Coherence<\/strong>: Maintaining consistency across multiple caches in multicore systems<\/li>\n<\/ul>\n<p><strong>Cache Coherence Protocols:<\/strong><\/p>\n<ul>\n<li><strong>MESI Protocol<\/strong> (Modified, Exclusive, Shared, Invalid)<\/li>\n<li><strong>MOESI Protocol<\/strong> (adds Owned state)<\/li>\n<li>Ensures all CPUs see consistent memory values<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"graphics-processing-unit-gpu\"><span>Graphics Processing Unit (GPU)<\/span>\n <a href=\"#graphics-processing-unit-gpu\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>GPUs<\/strong> are specialized processors optimized for parallel computation, originally designed for graphics rendering but now widely used for general-purpose computing.<\/p>\n<p><strong>Key Differences from CPUs:<\/strong><\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Aspect<\/th>\n <th>CPU<\/th>\n <th>GPU<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Core Count<\/strong><\/td>\n <td>4-64 high-performance cores<\/td>\n <td>1000s of simpler cores<\/td>\n <\/tr>\n <tr>\n <td><strong>Design Philosophy<\/strong><\/td>\n <td>Optimized for latency<\/td>\n <td>Optimized for throughput<\/td>\n <\/tr>\n <tr>\n <td><strong>Task Suitability<\/strong><\/td>\n <td>Complex, sequential tasks<\/td>\n <td>Highly parallel, repetitive tasks<\/td>\n <\/tr>\n <tr>\n <td><strong>Memory<\/strong><\/td>\n <td>Large cache hierarchy<\/td>\n <td>High-bandwidth memory<\/td>\n <\/tr>\n <tr>\n <td><strong>Control Flow<\/strong><\/td>\n <td>Complex branching<\/td>\n <td>SIMT (Single Instruction, Multiple Threads)<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>GPU Applications:<\/strong><\/p>\n<ul>\n<li>Graphics rendering and ray tracing<\/li>\n<li>Machine learning and deep neural networks<\/li>\n<li>Scientific simulations (computational fluid dynamics, molecular modeling)<\/li>\n<li>Cryptocurrency mining<\/li>\n<li>Video encoding\/decoding<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"memory-systems\"><span>Memory Systems<\/span>\n <a href=\"#memory-systems\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Memory Hierarchy Goals:<\/strong><\/p>\n<ol>\n<li><strong>Speed<\/strong>: Fast access for frequently used data<\/li>\n<li><strong>Capacity<\/strong>: Large storage for all program data<\/li>\n<li><strong>Cost<\/strong>: Balance between performance and affordability<\/li>\n<\/ol>\n<p><strong>Primary Memory (Volatile)<\/strong><\/p>\n<p><strong>Random Access Memory (RAM)<\/strong><\/p>\n<ul>\n<li><strong>DRAM (Dynamic RAM)<\/strong>: Main memory; needs periodic refresh; slower but denser\n<ul>\n<li>DDR4: 2133-3200 MT\/s, 16-64 GB typical<\/li>\n<li>DDR5: 4800-8400 MT\/s, 16-128 GB typical<\/li>\n<\/ul>\n<\/li>\n<li><strong>SRAM (Static RAM)<\/strong>: Cache memory; no refresh needed; faster but more expensive\n<ul>\n<li>Used in L1\/L2\/L3 caches<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><strong>Memory Access Patterns:<\/strong><\/p>\n<pre><code>\/\/ Demonstrating cache-friendly vs cache-unfriendly access\n\n\/\/ Cache-friendly: Sequential access (spatial locality)\nvoid sum_array_row_major(int arr[N][N]) {\n for (int i = 0; i &lt; N; i&#43;&#43;) {\n for (int j = 0; j &lt; N; j&#43;&#43;) {\n sum &#43;= arr[i][j]; \/\/ Sequential memory access\n }\n }\n}\n\n\/\/ Cache-unfriendly: Column-major in row-major storage\nvoid sum_array_column_major(int arr[N][N]) {\n for (int j = 0; j &lt; N; j&#43;&#43;) {\n for (int i = 0; i &lt; N; i&#43;&#43;) {\n sum &#43;= arr[i][j]; \/\/ Jumps across cache lines\n }\n }\n}<\/code><\/pre><p><strong>Secondary Storage (Non-Volatile)<\/strong><\/p>\n<p><strong>Solid State Drives (SSDs)<\/strong><\/p>\n<ul>\n<li><strong>Technology<\/strong>: NAND flash memory<\/li>\n<li><strong>Speed<\/strong>: 500-7000 MB\/s (SATA to NVMe)<\/li>\n<li><strong>Latency<\/strong>: 50-100 microseconds<\/li>\n<li><strong>Wear Leveling<\/strong>: Distributes writes to prevent cell degradation<\/li>\n<\/ul>\n<p><strong>Hard Disk Drives (HDDs)<\/strong><\/p>\n<ul>\n<li><strong>Technology<\/strong>: Magnetic storage on rotating platters<\/li>\n<li><strong>Speed<\/strong>: 100-200 MB\/s<\/li>\n<li><strong>Latency<\/strong>: 5-15 milliseconds (seek time + rotational delay)<\/li>\n<li><strong>Capacity<\/strong>: Up to 20 TB per drive<\/li>\n<\/ul>\n<p><strong>Storage Technologies Comparison:<\/strong><\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Technology<\/th>\n <th>Speed<\/th>\n <th>Latency<\/th>\n <th>Cost per GB<\/th>\n <th>Durability<\/th>\n <th>Use Case<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>RAM<\/strong><\/td>\n <td>20+ GB\/s<\/td>\n <td>50-100 ns<\/td>\n <td>High<\/td>\n <td>Volatile<\/td>\n <td>Active data<\/td>\n <\/tr>\n <tr>\n <td><strong>NVMe SSD<\/strong><\/td>\n <td>3-7 GB\/s<\/td>\n <td>50-100 \u03bcs<\/td>\n <td>Medium<\/td>\n <td>Limited writes<\/td>\n <td>OS, applications<\/td>\n <\/tr>\n <tr>\n <td><strong>SATA SSD<\/strong><\/td>\n <td>500 MB\/s<\/td>\n <td>50-100 \u03bcs<\/td>\n <td>Medium-Low<\/td>\n <td>Limited writes<\/td>\n <td>General storage<\/td>\n <\/tr>\n <tr>\n <td><strong>HDD<\/strong><\/td>\n <td>100-200 MB\/s<\/td>\n <td>5-15 ms<\/td>\n <td>Low<\/td>\n <td>High<\/td>\n <td>Archival, bulk storage<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"system-bus-architecture\"><span>System Bus Architecture<\/span>\n <a href=\"#system-bus-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The <strong>System Bus<\/strong> is a collection of parallel electrical conductors connecting CPU, memory, and I\/O devices.<\/p>\n<p><strong>Bus Types:<\/strong><\/p>\n<p><strong>1. Data Bus<\/strong><\/p>\n<ul>\n<li>Carries actual data between components<\/li>\n<li>Width determines data transfer rate (32-bit, 64-bit)<\/li>\n<li>Bidirectional<\/li>\n<\/ul>\n<p><strong>2. Address Bus<\/strong><\/p>\n<ul>\n<li>Carries memory addresses<\/li>\n<li>Width determines addressable memory space\n<ul>\n<li>32-bit: 4 GB (2\u00b3\u00b2 bytes)<\/li>\n<li>64-bit: 16 Exabytes (2\u2076\u2074 bytes)<\/li>\n<\/ul>\n<\/li>\n<li>Unidirectional (CPU to memory\/devices)<\/li>\n<\/ul>\n<p><strong>3. Control Bus<\/strong><\/p>\n<ul>\n<li>Carries control signals (read, write, interrupt, clock)<\/li>\n<li>Coordinates bus operations<\/li>\n<li>Bidirectional<\/li>\n<\/ul>\n<p><strong>Bus Operation Example:<\/strong><\/p>\n<pre><code>CPU wants to read memory location 0x1000:\n\n1. CPU places 0x1000 on Address Bus\n2. CPU asserts READ control signal on Control Bus\n3. Memory responds by placing data at 0x1000 onto Data Bus\n4. CPU reads data from Data Bus\n5. CPU de-asserts READ signal<\/code><\/pre><hr>\n<h2 id=\"os-architecture\">3. Operating System Architecture<\/h2>\n<h3 class=\"heading-element\" id=\"os-design-principles\"><span>OS Design Principles<\/span>\n <a href=\"#os-design-principles\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Separation of Mechanism and Policy<\/strong><\/p>\n<ul>\n<li><strong>Mechanism<\/strong>: <em>How<\/em> to do something (implementation)<\/li>\n<li><strong>Policy<\/strong>: <em>What<\/em> should be done (decision-making)<\/li>\n<li><strong>Benefit<\/strong>: Flexibility; policies can change without altering mechanisms<\/li>\n<\/ul>\n<p><strong>Example:<\/strong><\/p>\n<pre><code>Mechanism: Context switching between processes\nPolicy: Which process to run next (scheduling algorithm)<\/code><\/pre><p><strong>Optimization for Common Case<\/strong><\/p>\n<ul>\n<li>Design decisions should favor frequently occurring scenarios<\/li>\n<li>Identify workload characteristics and usage patterns<\/li>\n<li>Balance generality with specialized optimization<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"kernel-architectures\"><span>Kernel Architectures<\/span>\n <a href=\"#kernel-architectures\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"1-monolithic-kernel\"><span>1. Monolithic Kernel<\/span>\n <a href=\"#1-monolithic-kernel\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><strong>Concept<\/strong>: All OS services run in kernel space with full hardware access.<\/p>\n<p><strong>Structure:<\/strong><\/p>\n<pre><code>User Space\n\u2502\n\u251c\u2500 Application 1\n\u251c\u2500 Application 2\n\u2514\u2500 Application N\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 System Call Interface \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\nKernel Space (Single Address Space)\n\u2502\n\u251c\u2500 Process Management\n\u251c\u2500 Memory Management\n\u251c\u2500 File System\n\u251c\u2500 Device Drivers\n\u251c\u2500 Network Stack\n\u2514\u2500 Scheduling\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500<\/code><\/pre><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Advantages of Monolithic Kernel<\/strong>:<\/p>\n<ul>\n<li><strong>Performance<\/strong>: Direct function calls within kernel; no context switching overhead<\/li>\n<li><strong>Simplicity<\/strong>: Straightforward design and implementation<\/li>\n<li><strong>Efficiency<\/strong>: Minimal message passing overhead<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Disadvantages of Monolithic Kernel<\/strong>:<\/p>\n<ul>\n<li><strong>Reliability<\/strong>: Bug in any component can crash entire system<\/li>\n<li><strong>Maintainability<\/strong>: Large, complex codebase difficult to maintain<\/li>\n<li><strong>Security<\/strong>: Large attack surface; any vulnerability affects whole kernel<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<p><strong>Examples<\/strong>: Linux, traditional UNIX, Windows 9x<\/p>\n<p><strong>Linux Kernel Structure:<\/strong><\/p>\n<pre><code>\/\/ Simplified representation of kernel subsystems\nstruct linux_kernel {\n struct process_manager {\n struct task_struct *process_list;\n struct scheduler *sched;\n } pm;\n\n struct memory_manager {\n struct page_table *pgd;\n struct vm_area *vma_list;\n } mm;\n\n struct file_system {\n struct super_block *sb;\n struct inode_cache *icache;\n } fs;\n\n struct network_stack {\n struct socket *socket_list;\n struct protocol_handler *protocols;\n } net;\n};<\/code><\/pre><h4 class=\"heading-element\" id=\"2-microkernel-architecture\"><span>2. Microkernel Architecture<\/span>\n <a href=\"#2-microkernel-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><strong>Concept<\/strong>: Minimal kernel with essential services only; other services run in user space.<\/p>\n<p><strong>Structure:<\/strong><\/p>\n<pre><code>User Space\n\u2502\n\u251c\u2500 File Server\n\u251c\u2500 Device Drivers\n\u251c\u2500 Network Server\n\u251c\u2500 Applications\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 IPC (Message Passing) \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\nKernel Space (Minimal)\n\u2502\n\u251c\u2500 IPC Mechanism\n\u251c\u2500 Basic Scheduling\n\u251c\u2500 Low-level Memory Management\n\u2514\u2500 Hardware Abstraction\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500<\/code><\/pre><p><strong>Essential Kernel Services:<\/strong><\/p>\n<ul>\n<li>Inter-Process Communication (IPC)<\/li>\n<li>Basic thread scheduling<\/li>\n<li>Low-level address space management<\/li>\n<li>Interrupt handling<\/li>\n<\/ul>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Advantages of Microkernel<\/strong>:<\/p>\n<ul>\n<li><strong>Fault Isolation<\/strong>: Service crash doesn&rsquo;t affect kernel<\/li>\n<li><strong>Modularity<\/strong>: Easy to add\/remove services<\/li>\n<li><strong>Security<\/strong>: Smaller trusted computing base (TCB)<\/li>\n<li><strong>Portability<\/strong>: Hardware-specific code isolated<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<div class=\"details admonition danger open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-bolt\" aria-hidden=\"true\"><\/i>Danger<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Disadvantages of Microkernel<\/strong>:<\/p>\n<ul>\n<li><strong>Performance<\/strong>: IPC overhead for service communication<\/li>\n<li><strong>Complexity<\/strong>: More complex system structure<\/li>\n<li><strong>Development Effort<\/strong>: Requires careful IPC design<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<p><strong>Examples<\/strong>: MINIX 3, L4 microkernel family, QNX, seL4<\/p>\n<p><strong>Microkernel IPC Example:<\/strong><\/p>\n<pre><code>\/\/ Simplified IPC message passing\nstruct message {\n int src_pid;\n int dst_pid;\n int msg_type;\n char data[1024];\n};\n\n\/\/ Send message to file server\nint read_file(const char *path, char *buffer, size_t size) {\n struct message msg;\n msg.src_pid = getpid();\n msg.dst_pid = FILE_SERVER_PID;\n msg.msg_type = MSG_READ_FILE;\n strcpy(msg.data, path);\n\n \/\/ Send to microkernel IPC\n send_message(&amp;msg);\n\n \/\/ Wait for reply\n receive_message(&amp;msg);\n\n \/\/ Copy data from reply\n memcpy(buffer, msg.data, size);\n return msg.msg_type; \/\/ Return value\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"3-hybrid-kernel\"><span>3. Hybrid Kernel<\/span>\n <a href=\"#3-hybrid-kernel\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><strong>Concept<\/strong>: Combines monolithic and microkernel approaches; some services in kernel for performance, others in user space.<\/p>\n<p><strong>Structure:<\/strong><\/p>\n<pre><code>User Space\n\u2502\n\u251c\u2500 User-mode Services\n\u251c\u2500 Device Drivers (some)\n\u2514\u2500 Applications\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Mixed Interface \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\nKernel Space\n\u2502\n\u251c\u2500 Core Kernel (Microkernel-like)\n\u2502 \u251c\u2500 IPC\n\u2502 \u251c\u2500 Scheduling\n\u2502 \u2514\u2500 Low-level MM\n\u2502\n\u2514\u2500 Integrated Services (Monolithic-like)\n \u251c\u2500 File System\n \u251c\u2500 Device Drivers (critical)\n \u2514\u2500 Network Stack\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500<\/code><\/pre><p><strong>Advantages:<\/strong><\/p>\n<ul>\n<li>Balances performance and modularity<\/li>\n<li>Critical services in kernel for speed<\/li>\n<li>Non-critical services in user space for safety<\/li>\n<\/ul>\n<p><strong>Examples<\/strong>:<\/p>\n<ul>\n<li><strong>Windows NT\/modern Windows<\/strong>: NT kernel (hybrid)<\/li>\n<li><strong>macOS XNU<\/strong>: Mach microkernel + BSD monolithic components<\/li>\n<\/ul>\n<p><strong>XNU (macOS\/iOS Kernel) Architecture:<\/strong><\/p>\n<pre><code>\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 User Space Applications \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 System Libraries \u2502\n\u2502 (libSystem, Foundation, etc.) \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 XNU Kernel \u2502\n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 \u2502 BSD Layer (Monolithic) \u2502 \u2502\n\u2502 \u2502 - POSIX compatibility \u2502 \u2502\n\u2502 \u2502 - Networking (TCP\/IP) \u2502 \u2502\n\u2502 \u2502 - VFS (Virtual File System)\u2502 \u2502\n\u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502\n\u2502 \u2502 Mach Layer (Microkernel) \u2502 \u2502\n\u2502 \u2502 - IPC (Mach ports) \u2502 \u2502\n\u2502 \u2502 - Virtual Memory \u2502 \u2502\n\u2502 \u2502 - Task\/Thread management \u2502 \u2502\n\u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502\n\u2502 \u2502 I\/O Kit (C&#43;&#43; drivers) \u2502 \u2502\n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 Hardware \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/code><\/pre><h4 class=\"heading-element\" id=\"4-exokernel\"><span>4. Exokernel<\/span>\n <a href=\"#4-exokernel\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><strong>Concept<\/strong>: Expose hardware resources directly to applications; OS provides minimal abstraction.<\/p>\n<p><strong>Philosophy<\/strong>: Applications know best how to manage resources for their specific needs.<\/p>\n<p><strong>Advantages:<\/strong><\/p>\n<ul>\n<li>Maximum performance and flexibility<\/li>\n<li>Application-level resource management<\/li>\n<li>Eliminates unnecessary abstraction layers<\/li>\n<\/ul>\n<p><strong>Disadvantages:<\/strong><\/p>\n<ul>\n<li>Complex application development<\/li>\n<li>Limited protection between applications<\/li>\n<li>Difficult to program<\/li>\n<\/ul>\n<p><strong>Examples<\/strong>: MIT Exokernel, Aegis<\/p>\n<h4 class=\"heading-element\" id=\"5-unikernel\"><span>5. Unikernel<\/span>\n <a href=\"#5-unikernel\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><strong>Concept<\/strong>: Single address space machine image; application and OS services compiled into single binary.<\/p>\n<p><strong>Structure:<\/strong><\/p>\n<pre><code>\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Application Code \u2502\n\u2502 &#43; OS Services (LibOS) \u2502\n\u2502 (Single Binary) \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 Hypervisor \/ Hardware \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/code><\/pre><p><strong>Advantages:<\/strong><\/p>\n<ul>\n<li>Minimal attack surface<\/li>\n<li>Fast boot times (milliseconds)<\/li>\n<li>Small memory footprint<\/li>\n<li>Optimized for specific applications<\/li>\n<\/ul>\n<p><strong>Disadvantages:<\/strong><\/p>\n<ul>\n<li>Lack of dynamic process management<\/li>\n<li>Limited debugging tools<\/li>\n<li>Application must be recompiled for changes<\/li>\n<\/ul>\n<p><strong>Examples<\/strong>: MirageOS (OCaml), IncludeOS (C++), HalVM (Haskell)<\/p>\n<p><strong>Use Cases<\/strong>:<\/p>\n<ul>\n<li>Cloud microservices<\/li>\n<li>IoT devices<\/li>\n<li>Network appliances<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"system-layers\"><span>System Layers<\/span>\n <a href=\"#system-layers\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>User Mode vs Kernel Mode<\/strong><\/p>\n<p>Modern processors provide privilege levels (protection rings) to enforce security and stability.<\/p>\n<p><strong>x86 Protection Rings:<\/strong><\/p>\n<pre><code>Ring 3 (Lowest Privilege) \u2500\u2500\u2500 User Applications\nRing 2 (Device Drivers - rarely used)\nRing 1 (Device Drivers - rarely used)\nRing 0 (Highest Privilege) \u2500\u2500\u2500 Operating System Kernel<\/code><\/pre><p><strong>ARM Exception Levels (ARMv8):<\/strong><\/p>\n<pre><code>EL0 \u2500\u2500\u2500 User Applications\nEL1 \u2500\u2500\u2500 Operating System Kernel\nEL2 \u2500\u2500\u2500 Hypervisor\nEL3 \u2500\u2500\u2500 Secure Monitor (TrustZone)<\/code><\/pre><p><strong>Mode Transition:<\/strong>\nUser programs cannot directly access hardware or privileged instructions. They must request OS services via <strong>system calls<\/strong>, which trigger a mode switch.<\/p>\n<p><strong>Mode Switch Process:<\/strong><\/p>\n<pre><code>1. Application executes syscall instruction (INT 0x80, SYSCALL, SVC)\n2. CPU switches from User Mode to Kernel Mode\n3. CPU saves user context (registers, program counter)\n4. CPU jumps to kernel&#39;s system call handler\n5. Kernel validates request and executes service\n6. Kernel prepares return value\n7. CPU restores user context\n8. CPU switches back to User Mode\n9. Application resumes with result<\/code><\/pre><p><strong>Protection Benefits:<\/strong><\/p>\n<ul>\n<li><strong>Isolation<\/strong>: Faulty applications can&rsquo;t crash the system<\/li>\n<li><strong>Security<\/strong>: Malicious code can&rsquo;t directly manipulate hardware<\/li>\n<li><strong>Stability<\/strong>: Critical OS data structures protected from corruption<\/li>\n<\/ul>\n<hr>\n<h2 id=\"boot-process\">4. Boot Process and System Initialization<\/h2>\n<h3 class=\"heading-element\" id=\"x86-boot-sequence\"><span>x86 Boot Sequence<\/span>\n <a href=\"#x86-boot-sequence\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>1. Power-On Self-Test (POST)<\/strong><\/p>\n<pre><code>Power On \u2192 CPU Reset \u2192 POST\n\u2502\n\u251c\u2500 Test CPU registers\n\u251c\u2500 Check memory\n\u251c\u2500 Initialize chipset\n\u251c\u2500 Detect hardware\n\u2514\u2500 Display boot screen<\/code><\/pre><p><strong>2. BIOS\/UEFI Initialization<\/strong><\/p>\n<p><strong>Legacy BIOS (Basic Input\/Output System):<\/strong><\/p>\n<ul>\n<li>Stored in ROM\/flash on motherboard<\/li>\n<li>Executes in 16-bit real mode<\/li>\n<li>Loads Master Boot Record (MBR) from boot device<\/li>\n<\/ul>\n<p><strong>MBR Structure (512 bytes):<\/strong><\/p>\n<pre><code>Offset Size Description\n0x000 446 Boot code\n0x1BE 16 Partition entry 1\n0x1CE 16 Partition entry 2\n0x1DE 16 Partition entry 3\n0x1EE 16 Partition entry 4\n0x1FE 2 Boot signature (0x55AA)<\/code><\/pre><p><strong>Simple MBR Boot Code (x86 Assembly):<\/strong><\/p>\n<pre><code>; boot.asm - Minimal bootloader\nBITS 16 ; 16-bit real mode\nORG 0x7C00 ; BIOS loads us at 0x7C00\n\nstart:\n ; Setup segments\n xor ax, ax\n mov ds, ax\n mov es, ax\n mov ss, ax\n mov sp, 0x7C00\n\n ; Display message\n mov si, message\n call print_string\n\n ; Hang\n jmp $\n\nprint_string:\n lodsb ; Load byte from [SI] into AL\n or al, al ; Check if zero (end of string)\n jz done\n mov ah, 0x0E ; BIOS teletype function\n int 0x10 ; Call BIOS video interrupt\n jmp print_string\ndone:\n ret\n\nmessage db &#39;Booting...&#39;, 0x0D, 0x0A, 0\n\ntimes 510-($-$$) db 0 ; Pad to 510 bytes\ndw 0xAA55 ; Boot signature<\/code><\/pre><p><strong>UEFI (Unified Extensible Firmware Interface):<\/strong><\/p>\n<ul>\n<li>Modern replacement for BIOS<\/li>\n<li>Supports 32-bit and 64-bit modes<\/li>\n<li>GPT (GUID Partition Table) instead of MBR<\/li>\n<li>More secure (Secure Boot)<\/li>\n<li>Includes a boot manager<\/li>\n<\/ul>\n<p><strong>UEFI Boot Process:<\/strong><\/p>\n<pre><code>Power On \u2192 UEFI Firmware \u2192 Boot Manager \u2192 Boot Loader \u2192 OS Kernel<\/code><\/pre><p><strong>3. Bootloader Execution<\/strong><\/p>\n<p><strong>GRUB (Grand Unified Bootloader):<\/strong><\/p>\n<pre><code>Stage 1: Bootstrap code (in MBR or GPT)\n \u2193\nStage 1.5: Filesystem drivers (if needed)\n \u2193\nStage 2: Full GRUB with menu\n \u2193\nKernel Loading: Load kernel image into memory\n \u2193\nKernel Execution: Jump to kernel entry point<\/code><\/pre><p><strong>Multiboot Specification:<\/strong><\/p>\n<pre><code>\/\/ Multiboot header (must be in first 8KB of kernel)\n#define MULTIBOOT_HEADER_MAGIC 0x1BADB002\n#define MULTIBOOT_HEADER_FLAGS 0x00000003\n\n.section .multiboot\n.align 4\nmultiboot_header:\n .long MULTIBOOT_HEADER_MAGIC\n .long MULTIBOOT_HEADER_FLAGS\n .long -(MULTIBOOT_HEADER_MAGIC &#43; MULTIBOOT_HEADER_FLAGS)<\/code><\/pre><p><strong>4. Protected Mode Transition<\/strong><\/p>\n<p>Real mode has limitations:<\/p>\n<ul>\n<li>Only 1 MB addressable memory<\/li>\n<li>No memory protection<\/li>\n<li>16-bit operations<\/li>\n<\/ul>\n<p><strong>Entering Protected Mode:<\/strong><\/p>\n<pre><code>; Disable interrupts\ncli\n\n; Load Global Descriptor Table (GDT)\nlgdt [gdt_descriptor]\n\n; Set PE (Protection Enable) bit in CR0\nmov eax, cr0\nor eax, 0x1\nmov cr0, eax\n\n; Far jump to load CS with code segment selector\njmp 0x08:protected_mode\n\n[BITS 32]\nprotected_mode:\n ; Setup data segments\n mov ax, 0x10 ; Data segment selector\n mov ds, ax\n mov es, ax\n mov fs, ax\n mov gs, ax\n mov ss, ax\n\n ; Setup stack\n mov esp, 0x90000\n\n ; Jump to kernel\n call kernel_main<\/code><\/pre><p><strong>Global Descriptor Table (GDT):<\/strong><\/p>\n<pre><code>struct gdt_entry {\n uint16_t limit_low;\n uint16_t base_low;\n uint8_t base_middle;\n uint8_t access;\n uint8_t granularity;\n uint8_t base_high;\n} __attribute__((packed));\n\nstruct gdt_ptr {\n uint16_t limit;\n uint32_t base;\n} __attribute__((packed));\n\nstruct gdt_entry gdt[5];\n\nvoid gdt_set_gate(int num, uint32_t base, uint32_t limit, uint8_t access, uint8_t gran) {\n gdt[num].base_low = (base &amp; 0xFFFF);\n gdt[num].base_middle = (base &gt;&gt; 16) &amp; 0xFF;\n gdt[num].base_high = (base &gt;&gt; 24) &amp; 0xFF;\n gdt[num].limit_low = (limit &amp; 0xFFFF);\n gdt[num].granularity = ((limit &gt;&gt; 16) &amp; 0x0F) | (gran &amp; 0xF0);\n gdt[num].access = access;\n}\n\nvoid gdt_install() {\n struct gdt_ptr gp;\n gp.limit = (sizeof(struct gdt_entry) * 5) - 1;\n gp.base = (uint32_t)&amp;gdt;\n\n gdt_set_gate(0, 0, 0, 0, 0); \/\/ Null segment\n gdt_set_gate(1, 0, 0xFFFFFFFF, 0x9A, 0xCF); \/\/ Code segment\n gdt_set_gate(2, 0, 0xFFFFFFFF, 0x92, 0xCF); \/\/ Data segment\n gdt_set_gate(3, 0, 0xFFFFFFFF, 0xFA, 0xCF); \/\/ User code\n gdt_set_gate(4, 0, 0xFFFFFFFF, 0xF2, 0xCF); \/\/ User data\n\n gdt_flush(&amp;gp); \/\/ Assembly function to load GDT\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"arm-boot-process\"><span>ARM Boot Process<\/span>\n <a href=\"#arm-boot-process\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>1. ROM Bootloader (Primary Bootloader)<\/strong><\/p>\n<ul>\n<li>Hardcoded in SoC ROM<\/li>\n<li>Minimal functionality<\/li>\n<li>Loads secondary bootloader from boot media (SD, eMMC, SPI)<\/li>\n<\/ul>\n<p><strong>2. SPL (Secondary Program Loader) \/ U-Boot<\/strong><\/p>\n<ul>\n<li>Initializes DRAM controller<\/li>\n<li>Sets up clocks and power management<\/li>\n<li>Loads main bootloader or kernel<\/li>\n<\/ul>\n<p><strong>U-Boot Commands:<\/strong><\/p>\n<pre><code># Load kernel from SD card\nfatload mmc 0:1 ${kernel_addr_r} zImage\n\n# Load device tree\nfatload mmc 0:1 ${fdt_addr_r} bcm2710-rpi-3-b.dtb\n\n# Boot kernel\nbootz ${kernel_addr_r} - ${fdt_addr_r}<\/code><\/pre><p><strong>3. Device Tree (DTB)<\/strong><\/p>\n<ul>\n<li>Hardware description format<\/li>\n<li>Describes hardware topology, addresses, interrupts<\/li>\n<li>Passed to kernel during boot<\/li>\n<\/ul>\n<p><strong>Device Tree Example (Simplified):<\/strong><\/p>\n<pre><code>\/dts-v1\/;\n\n\/ {\n compatible = &#34;raspberrypi,3-model-b&#34;, &#34;brcm,bcm2837&#34;;\n model = &#34;Raspberry Pi 3 Model B&#34;;\n\n memory@0 {\n device_type = &#34;memory&#34;;\n reg = &lt;0x00000000 0x40000000&gt;; \/\/ 1GB\n };\n\n cpus {\n #address-cells = &lt;1&gt;;\n #size-cells = &lt;0&gt;;\n\n cpu@0 {\n device_type = &#34;cpu&#34;;\n compatible = &#34;arm,cortex-a53&#34;;\n reg = &lt;0&gt;;\n };\n \/\/ ... more CPUs\n };\n\n soc {\n uart0: serial@7e201000 {\n compatible = &#34;brcm,bcm2835-uart&#34;;\n reg = &lt;0x7e201000 0x1000&gt;;\n interrupts = &lt;2 25&gt;;\n };\n };\n};<\/code><\/pre><p><strong>4. Kernel Initialization<\/strong><\/p>\n<p><strong>Linux Kernel Boot (ARM64):<\/strong><\/p>\n<pre><code>1. Decompress kernel (if compressed)\n2. Setup initial page tables\n3. Enable MMU and caches\n4. Parse device tree\n5. Initialize memory subsystem\n6. Mount initial ramdisk (initramfs)\n7. Initialize drivers\n8. Start init process (PID 1)<\/code><\/pre><h3 class=\"heading-element\" id=\"kernel-initialization-deep-dive\"><span>Kernel Initialization Deep Dive<\/span>\n <a href=\"#kernel-initialization-deep-dive\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Early Boot Process:<\/strong><\/p>\n<pre><code>\/\/ Simplified Linux kernel initialization (kernel\/init\/main.c)\n\nasmlinkage __visible void __init start_kernel(void) {\n char *command_line;\n\n \/\/ 1. Setup architecture-specific components\n setup_arch(&amp;command_line);\n\n \/\/ 2. Initialize memory management\n mm_init();\n\n \/\/ 3. Setup scheduler\n sched_init();\n\n \/\/ 4. Initialize IRQs and timers\n init_IRQ();\n init_timers();\n\n \/\/ 5. Initialize console for early output\n console_init();\n\n \/\/ 6. Initialize rest of kernel subsystems\n vfs_caches_init();\n signals_init();\n\n \/\/ 7. Rest of initialization (drivers, etc.)\n rest_init();\n}\n\nstatic noinline void __init rest_init(void) {\n \/\/ Create kernel threads\n kernel_thread(kernel_init, NULL, CLONE_FS);\n kernel_thread(kthreadd, NULL, CLONE_FS | CLONE_FILES);\n\n \/\/ CPU idle loop\n cpu_startup_entry(CPUHP_ONLINE);\n}\n\nstatic int __init kernel_init(void *unused) {\n \/\/ Try to run init program\n if (!try_to_run_init_process(&#34;\/sbin\/init&#34;) ||\n !try_to_run_init_process(&#34;\/etc\/init&#34;) ||\n !try_to_run_init_process(&#34;\/bin\/init&#34;) ||\n !try_to_run_init_process(&#34;\/bin\/sh&#34;))\n return 0;\n\n panic(&#34;No working init found.&#34;);\n}<\/code><\/pre><hr>\n<h2 id=\"process-management\">5. Process Management<\/h2>\n<h3 class=\"heading-element\" id=\"process-concept\"><span>Process Concept<\/span>\n <a href=\"#process-concept\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>A <strong>process<\/strong> is an instance of a program in execution. It represents the dynamic, active entity, as opposed to a program which is a passive, static collection of instructions.<\/p>\n<p><strong>Process Components:<\/strong><\/p>\n<ol>\n<li><strong>Program Code (Text Section)<\/strong>: Executable instructions<\/li>\n<li><strong>Program Counter<\/strong>: Current instruction position<\/li>\n<li><strong>Processor Registers<\/strong>: Register contents<\/li>\n<li><strong>Stack<\/strong>: Temporary data (function parameters, return addresses, local variables)<\/li>\n<li><strong>Data Section<\/strong>: Global variables<\/li>\n<li><strong>Heap<\/strong>: Dynamically allocated memory<\/li>\n<\/ol>\n<p><strong>Process vs Program:<\/strong><\/p>\n<pre><code>Program: Passive entity (file on disk containing instructions)\nProcess: Active entity (program loaded in memory and executing)\n\nExample:\n- &#34;notepad.exe&#34; is a program\n- When you run it, each window is a separate process<\/code><\/pre><h3 class=\"heading-element\" id=\"process-memory-layout\"><span>Process Memory Layout<\/span>\n <a href=\"#process-memory-layout\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>High Memory (0xFFFFFFFF)\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Kernel Space \u2502 \u2190 OS kernel (1GB on 32-bit Linux)\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 Stack \u2502 \u2190 Grows downward\n\u2502 \u2193 \u2502 Function calls, local variables\n\u2502 \u2502\n\u2502 (Unused) \u2502\n\u2502 \u2502\n\u2502 \u2191 \u2502\n\u2502 Heap \u2502 \u2190 Grows upward\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 Dynamic allocation (malloc\/new)\n\u2502 BSS Segment \u2502 \u2190 Uninitialized data\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 Data Segment \u2502 \u2190 Initialized data\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 Text Segment \u2502 \u2190 Program code (read-only)\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\nLow Memory (0x00000000)<\/code><\/pre><p><strong>Example:<\/strong><\/p>\n<pre><code>#include &lt;stdlib.h&gt;\n\nint global_initialized = 42; \/\/ Data segment\nint global_uninitialized; \/\/ BSS segment\nconst char *text_data = &#34;Hello&#34;; \/\/ Text\/Read-only segment\n\nvoid function() {\n int stack_var = 10; \/\/ Stack\n int *heap_var = malloc(sizeof(int)); \/\/ Heap\n *heap_var = 20;\n free(heap_var);\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"process-control-block-pcb\"><span>Process Control Block (PCB)<\/span>\n <a href=\"#process-control-block-pcb\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The <strong>PCB<\/strong> (also called Task Control Block or TCB) is the kernel data structure representing a process.<\/p>\n<p><strong>PCB Contents:<\/strong><\/p>\n<pre><code>struct task_struct { \/\/ Linux representation (simplified)\n \/\/ 1. Process State\n volatile long state; \/\/ Running, Ready, Waiting, etc.\n\n \/\/ 2. Process ID\n pid_t pid; \/\/ Process ID\n pid_t tgid; \/\/ Thread group ID\n struct task_struct *parent; \/\/ Parent process\n\n \/\/ 3. CPU Scheduling Information\n int prio; \/\/ Priority\n int static_prio; \/\/ Static priority\n int normal_prio; \/\/ Normal priority\n unsigned int policy; \/\/ Scheduling policy\n struct sched_entity se; \/\/ Scheduling entity\n\n \/\/ 4. Memory Management\n struct mm_struct *mm; \/\/ Memory descriptor\n\n \/\/ 5. File Descriptors\n struct files_struct *files; \/\/ Open file information\n\n \/\/ 6. CPU Context\n struct thread_struct thread; \/\/ CPU registers, PC, SP\n\n \/\/ 7. Signal Handling\n struct signal_struct *signal;\n\n \/\/ 8. Process Times\n u64 utime; \/\/ User mode time\n u64 stime; \/\/ Kernel mode time\n\n \/\/ 9. Process Credentials\n uid_t uid, euid; \/\/ User ID, Effective UID\n gid_t gid, egid; \/\/ Group ID, Effective GID\n\n \/\/ 10. List Management\n struct list_head tasks; \/\/ Process list\n struct list_head sibling; \/\/ Sibling list\n struct list_head children; \/\/ Children list\n};<\/code><\/pre><h3 class=\"heading-element\" id=\"process-states\"><span>Process States<\/span>\n <a href=\"#process-states\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>A process transitions through various states during its lifetime.<\/p>\n<p><strong>Five-State Model:<\/strong><\/p>\n<pre><code> \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 New \u2502 \u2190 Process being created\n \u2514\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u2193\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 Scheduler \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u250c\u2500\u2500\u2500\u2192\u2502 Ready \u2502 \u2190 \u2500 \u2500 \u2500 \u2500 \u2500 \u2500 \u2500 \u2500 \u2524 Running \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2518 Dispatch \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 \u2502\n \u2502 \u2502 I\/O or\n \u2502 I\/O Complete \u2502 Event Wait\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 \u2502\n \u2502 \u2193\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2514\u2500\u2500\u2500\u2500\u2524 Waiting \u2502 \u2502 Terminated \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/code><\/pre><p><strong>State Descriptions:<\/strong><\/p>\n<ol>\n<li><strong>New<\/strong>: Process is being created; PCB allocated but not ready<\/li>\n<li><strong>Ready<\/strong>: Process is waiting to be assigned to CPU<\/li>\n<li><strong>Running<\/strong>: Instructions are being executed<\/li>\n<li><strong>Waiting (Blocked)<\/strong>: Process waiting for event (I\/O completion, signal)<\/li>\n<li><strong>Terminated (Zombie)<\/strong>: Process finished execution; waiting for parent to read exit status<\/li>\n<\/ol>\n<p><strong>State Transitions Example (Linux):<\/strong><\/p>\n<pre><code>\/\/ Process states in Linux (linux\/sched.h)\n#define TASK_RUNNING 0\n#define TASK_INTERRUPTIBLE 1 \/\/ Can be woken by signals\n#define TASK_UNINTERRUPTIBLE 2 \/\/ Cannot be interrupted\n#define __TASK_STOPPED 4\n#define __TASK_TRACED 8\n#define EXIT_ZOMBIE 16\n#define EXIT_DEAD 32\n\n\/\/ State transition example\nvoid state_transition() {\n set_current_state(TASK_INTERRUPTIBLE); \/\/ Ready \u2192 Waiting\n schedule(); \/\/ Yield CPU\n\n \/\/ After wake-up\n \/\/ Waiting \u2192 Ready \u2192 Running (when scheduled)\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"process-creation\"><span>Process Creation<\/span>\n <a href=\"#process-creation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>UNIX: fork() System Call<\/strong><\/p>\n<p><code>fork()<\/code> creates a new process by duplicating the calling process.<\/p>\n<p><strong>Characteristics:<\/strong><\/p>\n<ul>\n<li>Child process gets a copy of parent&rsquo;s address space (Copy-on-Write)<\/li>\n<li>Child has unique PID<\/li>\n<li>fork() returns twice: 0 in child, child&rsquo;s PID in parent<\/li>\n<\/ul>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Copy-on-Write Optimization<\/strong>: fork() doesn&rsquo;t immediately copy all memory! Instead, parent and child share the same pages until one tries to write. This makes fork() extremely fast and memory-efficient, even for large processes.<\/div>\n <\/div>\n<\/div>\n<p><strong>Fork Example:<\/strong><\/p>\n<pre><code>#include &lt;stdio.h&gt;\n#include &lt;unistd.h&gt;\n#include &lt;sys\/types.h&gt;\n\nint main() {\n pid_t pid;\n int x = 10;\n\n printf(&#34;Before fork: x = %d\\n&#34;, x);\n\n pid = fork();\n\n if (pid &lt; 0) {\n \/\/ Fork failed\n fprintf(stderr, &#34;Fork failed\\n&#34;);\n return 1;\n } else if (pid == 0) {\n \/\/ Child process\n x = 20;\n printf(&#34;Child: x = %d, PID = %d\\n&#34;, x, getpid());\n } else {\n \/\/ Parent process\n x = 30;\n printf(&#34;Parent: x = %d, Child PID = %d\\n&#34;, x, pid);\n }\n\n return 0;\n}\n\n\/* Output:\nBefore fork: x = 10\nChild: x = 20, PID = 1235\nParent: x = 30, Child PID = 1235\n*\/<\/code><\/pre><p><strong>exec() Family<\/strong><\/p>\n<p>exec() replaces the current process image with a new program.<\/p>\n<p><strong>Common exec variants:<\/strong><\/p>\n<ul>\n<li><code>execl()<\/code>: List of arguments<\/li>\n<li><code>execv()<\/code>: Vector (array) of arguments<\/li>\n<li><code>execle()<\/code>: List + environment<\/li>\n<li><code>execve()<\/code>: Vector + environment<\/li>\n<\/ul>\n<p><strong>Fork + Exec Pattern:<\/strong><\/p>\n<pre><code>#include &lt;stdio.h&gt;\n#include &lt;unistd.h&gt;\n#include &lt;sys\/wait.h&gt;\n\nint main() {\n pid_t pid = fork();\n\n if (pid == 0) {\n \/\/ Child: Replace with &#34;\/bin\/ls&#34; program\n char *args[] = {&#34;\/bin\/ls&#34;, &#34;-l&#34;, &#34;\/tmp&#34;, NULL};\n execv(&#34;\/bin\/ls&#34;, args);\n\n \/\/ If execv returns, it failed\n perror(&#34;execv failed&#34;);\n return 1;\n } else {\n \/\/ Parent: Wait for child to finish\n int status;\n waitpid(pid, &amp;status, 0);\n printf(&#34;Child exited with status %d\\n&#34;, WEXITSTATUS(status));\n }\n\n return 0;\n}<\/code><\/pre><p><strong>Windows: CreateProcess()<\/strong><\/p>\n<p>Windows doesn&rsquo;t use fork(); it creates processes differently.<\/p>\n<pre><code>#include &lt;windows.h&gt;\n#include &lt;stdio.h&gt;\n\nint main() {\n STARTUPINFO si;\n PROCESS_INFORMATION pi;\n\n ZeroMemory(&amp;si, sizeof(si));\n si.cb = sizeof(si);\n ZeroMemory(&amp;pi, sizeof(pi));\n\n \/\/ Create child process\n if (!CreateProcess(\n NULL, \/\/ Application name\n &#34;notepad.exe&#34;, \/\/ Command line\n NULL, \/\/ Process attributes\n NULL, \/\/ Thread attributes\n FALSE, \/\/ Inherit handles\n 0, \/\/ Creation flags\n NULL, \/\/ Environment\n NULL, \/\/ Current directory\n &amp;si, \/\/ Startup info\n &amp;pi \/\/ Process info\n )) {\n printf(&#34;CreateProcess failed (%d)\\n&#34;, GetLastError());\n return 1;\n }\n\n \/\/ Wait for child\n WaitForSingleObject(pi.hProcess, INFINITE);\n\n \/\/ Close handles\n CloseHandle(pi.hProcess);\n CloseHandle(pi.hThread);\n\n return 0;\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"process-termination\"><span>Process Termination<\/span>\n <a href=\"#process-termination\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Normal Termination:<\/strong><\/p>\n<ul>\n<li>Program executes last statement<\/li>\n<li>Calls <code>exit()<\/code> system call<\/li>\n<li>Returns from <code>main()<\/code><\/li>\n<\/ul>\n<p><strong>Abnormal Termination:<\/strong><\/p>\n<ul>\n<li>Fatal error (division by zero, segmentation fault)<\/li>\n<li>Killed by another process (kill command)<\/li>\n<li>Parent terminates (orphan processes)<\/li>\n<\/ul>\n<p><strong>Exit Status:<\/strong><\/p>\n<pre><code>#include &lt;stdlib.h&gt;\n\nint main() {\n \/\/ exit(0) - Success\n \/\/ exit(1) - General error\n \/\/ exit(127) - Command not found\n\n if (error_occurred) {\n exit(EXIT_FAILURE);\n }\n\n return EXIT_SUCCESS;\n}<\/code><\/pre><p><strong>Zombie and Orphan Processes:<\/strong><\/p>\n<p><strong>Zombie Process:<\/strong><\/p>\n<ul>\n<li>Process terminated but PCB still exists<\/li>\n<li>Waiting for parent to read exit status via <code>wait()<\/code><\/li>\n<li>Shows as <code>&lt;defunct&gt;<\/code> in <code>ps<\/code> output<\/li>\n<\/ul>\n<div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Zombie Processes<\/strong>: If a parent never calls wait(), the child&rsquo;s PCB remains allocated even after the child terminates. This wastes kernel resources. Long-running parent processes should always wait() for their children or ignore SIGCHLD to automatically reap zombies.<\/div>\n <\/div>\n<\/div>\n<pre><code>\/\/ Creating a zombie\n#include &lt;unistd.h&gt;\n\nint main() {\n if (fork() == 0) {\n \/\/ Child exits immediately\n exit(0);\n }\n\n \/\/ Parent doesn&#39;t call wait()\n sleep(60); \/\/ Child becomes zombie for 60 seconds\n return 0;\n}<\/code><\/pre><p><strong>Orphan Process:<\/strong><\/p>\n<ul>\n<li>Parent terminates before child<\/li>\n<li>Child is adopted by init (PID 1) or systemd<\/li>\n<li>Init\/systemd periodically calls <code>wait()<\/code> to reap orphans<\/li>\n<\/ul>\n<p><strong>Process Wait Example:<\/strong><\/p>\n<pre><code>#include &lt;sys\/wait.h&gt;\n#include &lt;stdio.h&gt;\n#include &lt;unistd.h&gt;\n\nint main() {\n pid_t pid = fork();\n\n if (pid == 0) {\n printf(&#34;Child process\\n&#34;);\n sleep(2);\n exit(42); \/\/ Exit with status 42\n } else {\n int status;\n pid_t child = wait(&amp;status); \/\/ Wait for any child\n\n if (WIFEXITED(status)) {\n printf(&#34;Child %d exited with status %d\\n&#34;,\n child, WEXITSTATUS(status));\n }\n\n \/\/ Alternative: waitpid() for specific child\n \/\/ waitpid(pid, &amp;status, 0);\n }\n\n return 0;\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"context-switching\"><span>Context Switching<\/span>\n <a href=\"#context-switching\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Context Switch<\/strong>: Saving the state of current process and loading the state of the next process.<\/p>\n<p><strong>Context Switch Steps:<\/strong><\/p>\n<pre><code>1. Save current process state (registers, PC, SP) to PCB\n2. Update PCB (state, timing information)\n3. Move PCB to appropriate queue (ready, waiting)\n4. Select next process (scheduling algorithm)\n5. Update new process PCB (state = RUNNING)\n6. Restore new process context from PCB\n7. Switch memory management (page tables)\n8. Resume execution of new process<\/code><\/pre><div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Context Switch Overhead - A Hidden Cost<\/strong>:<\/p>\n<ul>\n<li><strong>Direct Cost<\/strong>: Time to save\/restore registers (~microseconds)<\/li>\n<li><strong>Indirect Cost<\/strong>: Cache flush \u2192 huge performance hit! Modern CPUs with large caches can lose 1-10% performance per context switch<\/li>\n<li><strong>TLB Flush<\/strong>: Virtual memory translation caches invalidated<\/li>\n<li><strong>Pipeline Flush<\/strong>: CPU instruction pipeline emptied<\/li>\n<\/ul>\n<p><strong>Practical Impact<\/strong>: High context switch rates (&gt;1000\/sec) cause significant throughput degradation even though each individual switch is &ldquo;fast&rdquo;<\/p>\n<\/div>\n <\/div>\n<\/div>\n<p><strong>Minimizing Context Switches:<\/strong><\/p>\n<ul>\n<li>Use threads instead of processes (shared address space)<\/li>\n<li>Increase time slices (but hurts responsiveness)<\/li>\n<li>CPU affinity (keep process on same CPU)<\/li>\n<li>Reduce I\/O operations (batch I\/O requests)<\/li>\n<\/ul>\n<p><strong>Linux Context Switch Implementation (Simplified):<\/strong><\/p>\n<pre><code>\/\/ arch\/x86\/include\/asm\/switch_to.h\n\n#define switch_to(prev, next, last) do { \\\n \/* Save previous task&#39;s context *\/ \\\n asm volatile( \\\n &#34;pushfl\\n\\t&#34; \/* Save flags *\/ \\\n &#34;pushl %%ebp\\n\\t&#34; \/* Save base pointer *\/\\\n &#34;movl %%esp,%[prev_sp]\\n\\t&#34; \/* Save stack ptr *\/ \\\n &#34;movl %[next_sp],%%esp\\n\\t&#34; \/* Load next stack*\/\\\n &#34;movl $1f,%[prev_ip]\\n\\t&#34; \/* Save return IP *\/ \\\n &#34;pushl %[next_ip]\\n\\t&#34; \/* Push next IP *\/ \\\n &#34;jmp __switch_to\\n\\t&#34; \/* Context switch *\/ \\\n &#34;1:\\t&#34; \\\n &#34;popl %%ebp\\n\\t&#34; \/* Restore base ptr *\/ \\\n &#34;popfl\\n&#34; \/* Restore flags *\/ \\\n : [prev_sp] &#34;=m&#34; (prev-&gt;thread.sp), \\\n [prev_ip] &#34;=m&#34; (prev-&gt;thread.ip) \\\n : [next_sp] &#34;m&#34; (next-&gt;thread.sp), \\\n [next_ip] &#34;m&#34; (next-&gt;thread.ip), \\\n [prev] &#34;a&#34; (prev), \\\n [next] &#34;d&#34; (next) \\\n : &#34;memory&#34;, &#34;cc&#34; \\\n ); \\\n} while (0)<\/code><\/pre><h3 class=\"heading-element\" id=\"process-hierarchies\"><span>Process Hierarchies<\/span>\n <a href=\"#process-hierarchies\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>UNIX Process Tree:<\/strong><\/p>\n<pre><code>init (PID 1) or systemd\n\u251c\u2500 sshd\n\u2502 \u2514\u2500 bash (login shell)\n\u2502 \u2514\u2500 vim\n\u251c\u2500 cron\n\u2502 \u2514\u2500 backup_script\n\u251c\u2500 Apache httpd\n\u2502 \u251c\u2500 httpd (worker 1)\n\u2502 \u251c\u2500 httpd (worker 2)\n\u2502 \u2514\u2500 httpd (worker N)\n\u2514\u2500 X Window System\n \u2514\u2500 window_manager\n \u251c\u2500 terminal_emulator\n \u2514\u2500 browser<\/code><\/pre><p><strong>Viewing Process Tree (Linux):<\/strong><\/p>\n<pre><code>$ pstree\ninit\u2500\u252c\u2500acpid\n \u251c\u2500cron\n \u251c\u2500dbus-daemon\n \u251c\u2500dhclient\n \u251c\u2500login\u2500\u2500\u2500bash\u2500\u2500\u2500pstree\n \u251c\u2500rsyslogd\u2500\u2500\u25003*[{rsyslogd}]\n \u251c\u2500sshd\u2500\u2500\u2500sshd\u2500\u2500\u2500bash\n \u2514\u2500udevd\u2500\u2500\u25002*[udevd]\n\n$ ps -ef # Full process list\n$ ps aux # BSD-style format<\/code><\/pre><hr>\n<h2 id=\"thread-management\">6. Thread Management and Concurrency<\/h2>\n<h3 class=\"heading-element\" id=\"thread-concept\"><span>Thread Concept<\/span>\n <a href=\"#thread-concept\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>A <strong>thread<\/strong> is the smallest unit of execution within a process. Unlike processes which have separate address spaces, threads share the same memory space and resources, making them lightweight and efficient.<\/p>\n<p><strong>Key Differences: Process vs Thread<\/strong><\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Aspect<\/th>\n <th>Process<\/th>\n <th>Thread<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Memory Space<\/strong><\/td>\n <td>Separate address space<\/td>\n <td>Shared address space<\/td>\n <\/tr>\n <tr>\n <td><strong>Resource Overhead<\/strong><\/td>\n <td>High (memory, file descriptors)<\/td>\n <td>Low (shares with process)<\/td>\n <\/tr>\n <tr>\n <td><strong>Creation Time<\/strong><\/td>\n <td>Slow (milliseconds)<\/td>\n <td>Fast (microseconds)<\/td>\n <\/tr>\n <tr>\n <td><strong>Context Switch<\/strong><\/td>\n <td>Expensive (page tables, cache misses)<\/td>\n <td>Cheaper (same page tables)<\/td>\n <\/tr>\n <tr>\n <td><strong>Communication<\/strong><\/td>\n <td>IPC mechanisms needed<\/td>\n <td>Direct memory access<\/td>\n <\/tr>\n <tr>\n <td><strong>Isolation<\/strong><\/td>\n <td>High (protected)<\/td>\n <td>Low (shared memory)<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Thread Components:<\/strong><\/p>\n<pre><code>Each thread has its own:\n\u251c\u2500\u2500 Program Counter (PC)\n\u251c\u2500\u2500 Stack (local variables, function calls)\n\u251c\u2500\u2500 CPU Registers\n\u251c\u2500\u2500 Thread-local storage\n\u2514\u2500\u2500 Stack pointer\n\nShared by all threads in process:\n\u251c\u2500\u2500 Process memory (heap, code, globals)\n\u251c\u2500\u2500 File descriptors\n\u251c\u2500\u2500 Signal handlers\n\u2514\u2500\u2500 Current working directory<\/code><\/pre><h3 class=\"heading-element\" id=\"user-level-vs-kernel-level-threads\"><span>User-Level vs Kernel-Level Threads<\/span>\n <a href=\"#user-level-vs-kernel-level-threads\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>User-Level Threads<\/strong><\/p>\n<ul>\n<li>Managed by thread library (not OS)<\/li>\n<li>No kernel involvement<\/li>\n<li>Fast context switching<\/li>\n<li>Cannot utilize multiprocessors<\/li>\n<li>If one thread blocks, entire process blocked<\/li>\n<\/ul>\n<pre><code>\/\/ User-level thread example (POSIX Threads)\n#include &lt;pthread.h&gt;\n#include &lt;stdio.h&gt;\n\nvoid* thread_function(void *arg) {\n printf(&#34;Thread ID: %ld\\n&#34;, (long)pthread_self());\n return NULL;\n}\n\nint main() {\n pthread_t thread;\n\n \/\/ Create user-level thread\n pthread_create(&amp;thread, NULL, thread_function, NULL);\n\n \/\/ Wait for thread completion\n pthread_join(thread, NULL);\n\n return 0;\n}<\/code><\/pre><p><strong>Kernel-Level Threads<\/strong><\/p>\n<ul>\n<li>Managed by OS kernel<\/li>\n<li>Full kernel support<\/li>\n<li>Can run on multiple processors<\/li>\n<li>Slower context switching (syscall overhead)<\/li>\n<li>Better for I\/O-bound operations<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"thread-models\"><span>Thread Models<\/span>\n <a href=\"#thread-models\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>1. One-to-One Model<\/strong><\/p>\n<pre><code>User Thread 1 \u2194 Kernel Thread 1\nUser Thread 2 \u2194 Kernel Thread 2\nUser Thread 3 \u2194 Kernel Thread 3\n\nAdvantages:\n- Full parallelism\n- Blocking doesn&#39;t stop other threads\n- OS sees all threads\n\nDisadvantages:\n- High kernel overhead\n- Limited number of threads\n- Resource consumption<\/code><\/pre><p><strong>2. Many-to-One Model<\/strong><\/p>\n<pre><code>User Thread 1 \u2510\nUser Thread 2 \u251c\u2192 Kernel Thread 1\nUser Thread 3 \u2518\n\nAdvantages:\n- Low overhead\n- Fast context switching\n- No syscalls needed\n\nDisadvantages:\n- No true parallelism\n- One blocking thread blocks all\n- Cannot use multiprocessors<\/code><\/pre><p><strong>3. Many-to-Many Model<\/strong><\/p>\n<pre><code>User Thread 1 \u2510 \u250c\u2192 Kernel Thread 1\nUser Thread 2 \u251c\u2500\u2192 \u251c\u2192 Kernel Thread 2\nUser Thread 3 \u2524 \u2514\u2192 Kernel Thread 3\nUser Thread 4 \u2518\n\nBest of both worlds:\n- Multiple kernel threads\n- Efficient thread management\n- True parallelism with low overhead<\/code><\/pre><h3 class=\"heading-element\" id=\"thread-synchronization\"><span>Thread Synchronization<\/span>\n <a href=\"#thread-synchronization\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Critical Section<\/strong>: Code segment that accesses shared resources<\/p>\n<p><strong>Mutual Exclusion Problem<\/strong>: Ensuring only one thread enters critical section at a time<\/p>\n<div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Race Condition Danger<\/strong>: Without synchronization, the statement <code>shared_counter++<\/code> (which consists of 3 operations: READ \u2192 INCREMENT \u2192 WRITE) can interleave between threads, causing lost updates. For example:<\/p>\n<pre><code>Thread 1: READ (10) \u2192 INCREMENT (11)\nThread 2: READ (10) \u2192 INCREMENT (11) \u2190 Oops! Both read 10\nThread 1: WRITE (11)\nThread 2: WRITE (11) \u2190 Expected 12, got 11!<\/code><\/pre><\/div>\n <\/div>\n<\/div>\n<p><strong>Solution 1: Mutex (Mutual Exclusion Lock)<\/strong><\/p>\n<pre><code>#include &lt;pthread.h&gt;\n#include &lt;stdio.h&gt;\n\npthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;\nint shared_counter = 0;\n\nvoid* increment_counter(void *arg) {\n for (int i = 0; i &lt; 1000000; i&#43;&#43;) {\n pthread_mutex_lock(&amp;mutex);\n shared_counter&#43;&#43;; \/\/ Critical section\n pthread_mutex_unlock(&amp;mutex);\n }\n return NULL;\n}\n\nint main() {\n pthread_t threads[2];\n\n pthread_create(&amp;threads[0], NULL, increment_counter, NULL);\n pthread_create(&amp;threads[1], NULL, increment_counter, NULL);\n\n pthread_join(threads[0], NULL);\n pthread_join(threads[1], NULL);\n\n printf(&#34;Counter value: %d (expected: 2000000)\\n&#34;, shared_counter);\n\n pthread_mutex_destroy(&amp;mutex);\n return 0;\n}<\/code><\/pre><p><strong>Solution 2: Semaphore<\/strong><\/p>\n<pre><code>#include &lt;semaphore.h&gt;\n#include &lt;pthread.h&gt;\n#include &lt;stdio.h&gt;\n\nsem_t semaphore;\nint shared_resource = 0;\n\nvoid* worker(void *arg) {\n for (int i = 0; i &lt; 5; i&#43;&#43;) {\n sem_wait(&amp;semaphore); \/\/ Acquire (P operation)\n\n printf(&#34;Thread %ld accessing resource\\n&#34;,\n (long)pthread_self());\n shared_resource&#43;&#43;;\n\n sem_post(&amp;semaphore); \/\/ Release (V operation)\n }\n return NULL;\n}\n\nint main() {\n sem_init(&amp;semaphore, 0, 1); \/\/ Binary semaphore\n\n pthread_t threads[3];\n for (int i = 0; i &lt; 3; i&#43;&#43;) {\n pthread_create(&amp;threads[i], NULL, worker, NULL);\n }\n\n for (int i = 0; i &lt; 3; i&#43;&#43;) {\n pthread_join(threads[i], NULL);\n }\n\n sem_destroy(&amp;semaphore);\n return 0;\n}<\/code><\/pre><p><strong>Solution 3: Condition Variables<\/strong><\/p>\n<pre><code>#include &lt;pthread.h&gt;\n#include &lt;stdio.h&gt;\n\npthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;\npthread_cond_t condition = PTHREAD_COND_INITIALIZER;\nint ready = 0;\n\nvoid* waiter_thread(void *arg) {\n pthread_mutex_lock(&amp;mutex);\n\n while (!ready) { \/\/ Always use while, not if\n printf(&#34;Waiting...\\n&#34;);\n pthread_cond_wait(&amp;condition, &amp;mutex);\n }\n\n printf(&#34;Proceeding!\\n&#34;);\n pthread_mutex_unlock(&amp;mutex);\n return NULL;\n}\n\nvoid* signaler_thread(void *arg) {\n sleep(2);\n\n pthread_mutex_lock(&amp;mutex);\n ready = 1;\n pthread_cond_signal(&amp;condition); \/\/ Wake one waiter\n pthread_mutex_unlock(&amp;mutex);\n\n return NULL;\n}\n\nint main() {\n pthread_t t1, t2;\n\n pthread_create(&amp;t1, NULL, waiter_thread, NULL);\n pthread_create(&amp;t2, NULL, signaler_thread, NULL);\n\n pthread_join(t1, NULL);\n pthread_join(t2, NULL);\n\n return 0;\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"producer-consumer-problem\"><span>Producer-Consumer Problem<\/span>\n <a href=\"#producer-consumer-problem\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Scenario<\/strong>: Producer generates data, Consumer uses data, bounded buffer between them<\/p>\n<div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Classic Synchronization Problem<\/strong>: The Producer-Consumer problem demonstrates critical synchronization concepts:<\/p>\n<ul>\n<li><strong>Mutual Exclusion<\/strong>: Only one thread accesses buffer at a time<\/li>\n<li><strong>Resource Counting<\/strong>: Track empty\/full slots<\/li>\n<li><strong>Deadlock Avoidance<\/strong>: Proper ordering of semaphore operations<\/li>\n<li><strong>Real-world Applications<\/strong>: Message queues, thread pools, I\/O buffering<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<pre><code>#include &lt;pthread.h&gt;\n#include &lt;stdio.h&gt;\n#include &lt;stdlib.h&gt;\n#include &lt;semaphore.h&gt;\n\n#define BUFFER_SIZE 5\n\nint buffer[BUFFER_SIZE];\nint in = 0, out = 0;\n\nsem_t empty; \/\/ Counts empty slots\nsem_t full; \/\/ Counts full slots\npthread_mutex_t mutex;\n\nvoid* producer(void *arg) {\n for (int i = 0; i &lt; 10; i&#43;&#43;) {\n sem_wait(&amp;empty); \/\/ Wait for empty slot\n pthread_mutex_lock(&amp;mutex);\n\n buffer[in] = i * 10;\n printf(&#34;Producer: Added %d at %d\\n&#34;, buffer[in], in);\n in = (in &#43; 1) % BUFFER_SIZE;\n\n pthread_mutex_unlock(&amp;mutex);\n sem_post(&amp;full); \/\/ Signal full slot\n }\n return NULL;\n}\n\nvoid* consumer(void *arg) {\n for (int i = 0; i &lt; 10; i&#43;&#43;) {\n sem_wait(&amp;full); \/\/ Wait for full slot\n pthread_mutex_lock(&amp;mutex);\n\n int value = buffer[out];\n printf(&#34;Consumer: Consumed %d from %d\\n&#34;, value, out);\n out = (out &#43; 1) % BUFFER_SIZE;\n\n pthread_mutex_unlock(&amp;mutex);\n sem_post(&amp;empty); \/\/ Signal empty slot\n }\n return NULL;\n}\n\nint main() {\n pthread_t prod, cons;\n\n sem_init(&amp;empty, 0, BUFFER_SIZE);\n sem_init(&amp;full, 0, 0);\n pthread_mutex_init(&amp;mutex, NULL);\n\n pthread_create(&amp;prod, NULL, producer, NULL);\n pthread_create(&amp;cons, NULL, consumer, NULL);\n\n pthread_join(prod, NULL);\n pthread_join(cons, NULL);\n\n sem_destroy(&amp;empty);\n sem_destroy(&amp;full);\n pthread_mutex_destroy(&amp;mutex);\n\n return 0;\n}<\/code><\/pre><hr>\n<h2 id=\"cpu-scheduling\">7. CPU Scheduling Algorithms<\/h2>\n<h3 class=\"heading-element\" id=\"scheduling-criteria\"><span>Scheduling Criteria<\/span>\n <a href=\"#scheduling-criteria\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Maximize:<\/strong><\/p>\n<ul>\n<li><strong>CPU Utilization<\/strong>: Keep CPU as busy as possible (40-90% is reasonable)<\/li>\n<li><strong>Throughput<\/strong>: Number of processes completed per unit time<\/li>\n<\/ul>\n<p><strong>Minimize:<\/strong><\/p>\n<ul>\n<li><strong>Turnaround Time<\/strong>: Time from submission to completion<\/li>\n<li><strong>Waiting Time<\/strong>: Time spent in ready queue<\/li>\n<li><strong>Response Time<\/strong>: Time from submission to first response<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"scheduling-algorithms-complete-analysis\"><span>Scheduling Algorithms (Complete Analysis)<\/span>\n <a href=\"#scheduling-algorithms-complete-analysis\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"1-first-come-first-served-fcfs\"><span>1. First-Come-First-Served (FCFS)<\/span>\n <a href=\"#1-first-come-first-served-fcfs\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><strong>Algorithm<\/strong>: Process executes for entire duration, no preemption<\/p>\n<pre><code>\/\/ FCFS Scheduler Implementation\nstruct process {\n int pid;\n int arrival_time;\n int burst_time;\n int start_time;\n int end_time;\n};\n\nvoid fcfs_schedule(struct process procs[], int n) {\n int current_time = 0;\n double total_turnaround = 0, total_wait = 0;\n\n for (int i = 0; i &lt; n; i&#43;&#43;) {\n procs[i].start_time = current_time;\n procs[i].end_time = current_time &#43; procs[i].burst_time;\n current_time = procs[i].end_time;\n\n int turnaround = procs[i].end_time - procs[i].arrival_time;\n int wait = procs[i].start_time - procs[i].arrival_time;\n\n total_turnaround &#43;= turnaround;\n total_wait &#43;= wait;\n\n printf(&#34;P%d: Wait=%d, Turnaround=%d\\n&#34;,\n procs[i].pid, wait, turnaround);\n }\n\n printf(&#34;Average Wait Time: %.2f\\n&#34;, total_wait \/ n);\n printf(&#34;Average Turnaround: %.2f\\n&#34;, total_turnaround \/ n);\n}\n\n\/*\nExample: Processes with burst times [24, 3, 3]\nFCFS: P1_____________________ P2__ P3__\n 0 24 27 30\n\nWaiting times: P1=0, P2=24, P3=27\nAverage wait: 17ms\n*\/<\/code><\/pre><div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Convoy Effect Problem<\/strong>: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation.<\/div>\n <\/div>\n<\/div>\n<h4 class=\"heading-element\" id=\"2-shortest-job-first-sjf\"><span>2. Shortest Job First (SJF)<\/span>\n <a href=\"#2-shortest-job-first-sjf\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><strong>Non-preemptive<\/strong>: Each process runs completely<\/p>\n<pre><code>void sjf_schedule(struct process procs[], int n) {\n \/\/ Sort by burst time (ascending)\n for (int i = 0; i &lt; n - 1; i&#43;&#43;) {\n for (int j = 0; j &lt; n - i - 1; j&#43;&#43;) {\n if (procs[j].burst_time &gt; procs[j&#43;1].burst_time) {\n struct process temp = procs[j];\n procs[j] = procs[j&#43;1];\n procs[j&#43;1] = temp;\n }\n }\n }\n\n fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list\n}\n\n\/*\nExample: Processes [24, 3, 3]\nSorted: [3, 3, 24]\n\nGantt Chart:\nP2_ P3_ P1_____________________\n0 3 6 30\n\nAverage wait: (0 &#43; 3 &#43; 6) \/ 3 = 3ms (vs 17ms for FCFS!)\n*\/<\/code><\/pre><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>SJF Optimization<\/strong>: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an <strong>82% reduction<\/strong>! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously.<\/div>\n <\/div>\n<\/div>\n<div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Starvation Risk<\/strong>: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms.<\/div>\n <\/div>\n<\/div>\n<p><strong>Preemptive SJF (Shortest Remaining Time First - SRTF)<\/strong><\/p>\n<pre><code>void srtf_schedule(struct process procs[], int n) {\n int remaining[n];\n for (int i = 0; i &lt; n; i&#43;&#43;) {\n remaining[i] = procs[i].burst_time;\n }\n\n int time = 0;\n int total_processes = n;\n\n while (total_processes &gt; 0) {\n \/\/ Find process with shortest remaining time\n int min_index = -1;\n int min_remaining = INT_MAX;\n\n for (int i = 0; i &lt; n; i&#43;&#43;) {\n if (remaining[i] &gt; 0 &amp;&amp; remaining[i] &lt; min_remaining &amp;&amp;\n procs[i].arrival_time &lt;= time) {\n min_remaining = remaining[i];\n min_index = i;\n }\n }\n\n if (min_index == -1) {\n time&#43;&#43;;\n continue;\n }\n\n \/\/ Execute for 1 time unit\n remaining[min_index]--;\n time&#43;&#43;;\n\n if (remaining[min_index] == 0) {\n total_processes--;\n }\n }\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"3-round-robin-rr-scheduling\"><span>3. Round Robin (RR) Scheduling<\/span>\n <a href=\"#3-round-robin-rr-scheduling\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><strong>Overview<\/strong>: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It&rsquo;s designed specifically for time-sharing systems where fairness and responsiveness are critical.<\/p>\n<p><strong>Key Concept<\/strong>: Time Quantum (Time Slice) - Fixed time period allocated to each process<\/p>\n<div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Time Quantum Selection is Critical<\/strong>:<\/p>\n<ul>\n<li><strong>Too small (1-10ms)<\/strong>: Excessive context switching overhead \u2192 CPU thrashing<\/li>\n<li><strong>Too large (100ms+)<\/strong>: Behaves like FCFS \u2192 poor response time<\/li>\n<li><strong>Sweet spot<\/strong>: 80% of CPU bursts should be shorter than the quantum<\/li>\n<li><strong>Linux default<\/strong>: ~100ms (configurable via <code>sched_latency_ns<\/code>)<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<p><strong>Characteristics<\/strong>:<\/p>\n<ul>\n<li><strong>Preemptive<\/strong>: CPU is preempted after time quantum expires<\/li>\n<li><strong>Fair<\/strong>: All processes get equal CPU time in rotation<\/li>\n<li><strong>FIFO Queue<\/strong>: Processes are arranged in circular queue<\/li>\n<li><strong>No Starvation<\/strong>: Every process eventually gets CPU time<\/li>\n<\/ul>\n<hr>\n<h5 class=\"heading-element\" id=\"round-robin-algorithm-flow\"><span>Round Robin Algorithm Flow<\/span>\n <a href=\"#round-robin-algorithm-flow\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h5><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/round-robin-algorithm-flow.webp\" title=\"Round Robin Algorithm Flow Diagram - CPU Scheduling\" data-thumbnail=\"\/files\/round-robin-algorithm-flow.webp\" data-sub-html=\"<h2>Round Robin Scheduling Algorithm Flowchart showing the complete execution flow with decision points for quantum expiration and process completion<\/h2><p>Round Robin Algorithm Flow Diagram - CPU Scheduling<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/round-robin-algorithm-flow.webp' alt=\"Round Robin Scheduling Algorithm Flowchart showing the complete execution flow with decision points for quantum expiration and process completion\"><\/a><figcaption class=\"image-caption\">Round Robin Algorithm Flow Diagram - CPU Scheduling<\/figcaption>\n <\/figure><\/p>\n<p><em>Figure 1: Round Robin scheduling algorithm flowchart illustrating the process dequeue, execution, and re-queue mechanism with time quantum management.<\/em><\/p>\n<hr>\n<h5 class=\"heading-element\" id=\"process-state-transitions-in-round-robin\"><span>Process State Transitions in Round Robin<\/span>\n <a href=\"#process-state-transitions-in-round-robin\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h5><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/round-robin-process-state-transitions.webp\" title=\"Round Robin Process State Machine\" data-thumbnail=\"\/files\/round-robin-process-state-transitions.webp\" data-sub-html=\"<h2>Process State Diagram for Round Robin Scheduling showing transitions between New, Ready, Running, Waiting, and Terminated states with quantum expiration handling<\/h2><p>Round Robin Process State Machine<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/round-robin-process-state-transitions.webp' alt=\"Process State Diagram for Round Robin Scheduling showing transitions between New, Ready, Running, Waiting, and Terminated states with quantum expiration handling\"><\/a><figcaption class=\"image-caption\">Round Robin Process State Machine<\/figcaption>\n <\/figure><\/p>\n<p><em>Figure 2: Process state transitions in Round Robin scheduling with time quantum expiration triggering preemption and re-queueing of processes.<\/em><\/p>\n<hr>\n<h5 class=\"heading-element\" id=\"circular-queue-mechanism\"><span>Circular Queue Mechanism<\/span>\n <a href=\"#circular-queue-mechanism\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h5><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/round-robin-circular-queue.webp\" title=\"Circular Queue Data Structure in Round Robin\" data-thumbnail=\"\/files\/round-robin-circular-queue.webp\" data-sub-html=\"<h2>Circular Ready Queue in Round Robin Scheduling demonstrating FIFO process management with CPU dispatch and re-enqueue mechanism<\/h2><p>Circular Queue Data Structure in Round Robin<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/round-robin-circular-queue.webp' alt=\"Circular Ready Queue in Round Robin Scheduling demonstrating FIFO process management with CPU dispatch and re-enqueue mechanism\"><\/a><figcaption class=\"image-caption\">Circular Queue Data Structure in Round Robin<\/figcaption>\n <\/figure><\/p>\n<p><em>Figure 3: Circular ready queue mechanism showing how processes are dispatched to CPU and re-enqueued when time quantum expires before completion.<\/em><\/p>\n<hr>\n<h5 class=\"heading-element\" id=\"detailed-implementation-with-comments\"><span>Detailed Implementation with Comments<\/span>\n <a href=\"#detailed-implementation-with-comments\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h5><pre><code>#include &lt;stdio.h&gt;\n#include &lt;stdlib.h&gt;\n\n\/\/ Process structure with all necessary attributes\ntypedef struct {\n int pid; \/\/ Process ID\n int arrival_time; \/\/ When process arrives in ready queue\n int burst_time; \/\/ Total CPU time needed\n int remaining_time; \/\/ Remaining CPU time\n int completion_time; \/\/ When process completes\n int waiting_time; \/\/ Total time spent waiting\n int turnaround_time; \/\/ completion_time - arrival_time\n int response_time; \/\/ First time process gets CPU - arrival_time\n int first_response; \/\/ Flag to track first CPU allocation\n} Process;\n\n\/\/ Circular queue for ready processes\ntypedef struct {\n Process *processes[100];\n int front;\n int rear;\n int count;\n} ReadyQueue;\n\n\/\/ Initialize queue\nvoid init_queue(ReadyQueue *q) {\n q-&gt;front = 0;\n q-&gt;rear = -1;\n q-&gt;count = 0;\n}\n\n\/\/ Check if queue is empty\nint is_empty(ReadyQueue *q) {\n return q-&gt;count == 0;\n}\n\n\/\/ Enqueue process at rear\nvoid enqueue(ReadyQueue *q, Process *p) {\n q-&gt;rear = (q-&gt;rear &#43; 1) % 100; \/\/ Circular increment\n q-&gt;processes[q-&gt;rear] = p;\n q-&gt;count&#43;&#43;;\n}\n\n\/\/ Dequeue process from front\nProcess* dequeue(ReadyQueue *q) {\n if (is_empty(q)) return NULL;\n\n Process *p = q-&gt;processes[q-&gt;front];\n q-&gt;front = (q-&gt;front &#43; 1) % 100; \/\/ Circular increment\n q-&gt;count--;\n return p;\n}\n\n\/**\n * Round Robin Scheduling Algorithm\n *\n * Parameters:\n * processes: Array of processes to schedule\n * n: Number of processes\n * time_quantum: Fixed time slice for each process\n *\n * Algorithm Steps:\n * 1. Create ready queue and add all processes\n * 2. While queue is not empty:\n * a. Dequeue process from front\n * b. If remaining_time &lt;= quantum: Execute completely\n * c. Else: Execute for quantum, then re-enqueue\n * 3. Calculate metrics (waiting time, turnaround time)\n *\/\nvoid round_robin_schedule(Process processes[], int n, int time_quantum) {\n ReadyQueue ready_queue;\n init_queue(&amp;ready_queue);\n\n \/\/ Add all processes to ready queue (assume all arrive at time 0)\n for (int i = 0; i &lt; n; i&#43;&#43;) {\n processes[i].remaining_time = processes[i].burst_time;\n processes[i].first_response = 0; \/\/ Not yet responded\n enqueue(&amp;ready_queue, &amp;processes[i]);\n }\n\n int current_time = 0;\n int completed = 0;\n\n printf(&#34;\\n=== Round Robin Scheduling (Quantum = %d) ===\\n\\n&#34;, time_quantum);\n printf(&#34;Time\\tProcess\\tAction\\n&#34;);\n printf(&#34;----\\t-------\\t------\\n&#34;);\n\n \/\/ Main scheduling loop\n while (!is_empty(&amp;ready_queue)) {\n \/\/ Dequeue next process from front of ready queue\n Process *current = dequeue(&amp;ready_queue);\n\n \/\/ Record response time (first time process gets CPU)\n if (current-&gt;first_response == 0) {\n current-&gt;response_time = current_time - current-&gt;arrival_time;\n current-&gt;first_response = 1;\n }\n\n \/\/ Determine execution time for this time slice\n int execution_time;\n if (current-&gt;remaining_time &lt;= time_quantum) {\n \/\/ Process will complete within this quantum\n execution_time = current-&gt;remaining_time;\n current-&gt;remaining_time = 0;\n\n printf(&#34;%d\\tP%d\\tExecute for %dms (COMPLETE)\\n&#34;,\n current_time, current-&gt;pid, execution_time);\n\n current_time &#43;= execution_time;\n current-&gt;completion_time = current_time;\n current-&gt;turnaround_time = current-&gt;completion_time - current-&gt;arrival_time;\n current-&gt;waiting_time = current-&gt;turnaround_time - current-&gt;burst_time;\n\n completed&#43;&#43;;\n } else {\n \/\/ Process needs more time - will be preempted\n execution_time = time_quantum;\n current-&gt;remaining_time -= time_quantum;\n\n printf(&#34;%d\\tP%d\\tExecute for %dms (PREEMPT, %dms remaining)\\n&#34;,\n current_time, current-&gt;pid, execution_time, current-&gt;remaining_time);\n\n current_time &#43;= execution_time;\n\n \/\/ Context switch: Re-enqueue at rear of ready queue\n enqueue(&amp;ready_queue, current);\n }\n }\n\n printf(&#34;\\n=== Process Completion Summary ===\\n&#34;);\n printf(&#34;PID\\tBurst\\tCompletion\\tTurnaround\\tWaiting\\tResponse\\n&#34;);\n printf(&#34;---\\t-----\\t----------\\t----------\\t-------\\t--------\\n&#34;);\n\n float avg_turnaround = 0, avg_waiting = 0, avg_response = 0;\n for (int i = 0; i &lt; n; i&#43;&#43;) {\n printf(&#34;P%d\\t%d\\t%d\\t\\t%d\\t\\t%d\\t%d\\n&#34;,\n processes[i].pid,\n processes[i].burst_time,\n processes[i].completion_time,\n processes[i].turnaround_time,\n processes[i].waiting_time,\n processes[i].response_time);\n\n avg_turnaround &#43;= processes[i].turnaround_time;\n avg_waiting &#43;= processes[i].waiting_time;\n avg_response &#43;= processes[i].response_time;\n }\n\n printf(&#34;\\n=== Performance Metrics ===\\n&#34;);\n printf(&#34;Average Turnaround Time: %.2f ms\\n&#34;, avg_turnaround \/ n);\n printf(&#34;Average Waiting Time: %.2f ms\\n&#34;, avg_waiting \/ n);\n printf(&#34;Average Response Time: %.2f ms\\n&#34;, avg_response \/ n);\n printf(&#34;Number of Context Switches: %d\\n&#34;, current_time \/ time_quantum);\n}<\/code><\/pre><hr>\n<h5 class=\"heading-element\" id=\"gantt-chart-example\"><span>Gantt Chart Example<\/span>\n <a href=\"#gantt-chart-example\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h5><p><strong>Example Problem<\/strong>: Schedule 3 processes with Round Robin (Quantum = 4ms)<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Process<\/th>\n <th>Burst Time<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td>P1<\/td>\n <td>24 ms<\/td>\n <\/tr>\n <tr>\n <td>P2<\/td>\n <td>3 ms<\/td>\n <\/tr>\n <tr>\n <td>P3<\/td>\n <td>3 ms<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><p><strong>Execution Timeline<\/strong>:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/round-robin-gantt-chart.webp\" title=\"Round Robin Gantt Chart - Process Execution Timeline\" data-thumbnail=\"\/files\/round-robin-gantt-chart.webp\" data-sub-html=\"<h2>Round Robin Gantt Chart showing process execution timeline with 4ms time quantum for three processes P1, P2, P3<\/h2><p>Round Robin Gantt Chart - Process Execution Timeline<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/round-robin-gantt-chart.webp' alt=\"Round Robin Gantt Chart showing process execution timeline with 4ms time quantum for three processes P1, P2, P3\"><\/a><figcaption class=\"image-caption\">Round Robin Gantt Chart - Process Execution Timeline<\/figcaption>\n <\/figure><\/p>\n<p><em>Figure 4: Gantt chart visualization of Round Robin scheduling with quantum=4ms, demonstrating process interleaving and completion times.<\/em><\/p>\n<p><strong>Step-by-Step Execution<\/strong>:<\/p>\n<pre><code>Time 0-4: P1 executes (24-4=20ms remaining) \u2192 Re-queue\nTime 4-7: P2 executes (3-3=0ms) \u2192 Complete \u2713\nTime 7-10: P3 executes (3-3=0ms) \u2192 Complete \u2713\nTime 10-14: P1 executes (20-4=16ms remaining) \u2192 Re-queue\nTime 14-18: P1 executes (16-4=12ms remaining) \u2192 Re-queue\nTime 18-22: P1 executes (12-4=8ms remaining) \u2192 Re-queue\nTime 22-26: P1 executes (8-4=4ms remaining) \u2192 Re-queue\nTime 26-30: P1 executes (4-4=0ms) \u2192 Complete \u2713<\/code><\/pre><p><strong>Metrics Calculation<\/strong>:<\/p>\n<ul>\n<li><strong>P1<\/strong>: Turnaround = 30-0 = 30ms, Waiting = 30-24 = 6ms, Response = 0ms<\/li>\n<li><strong>P2<\/strong>: Turnaround = 7-0 = 7ms, Waiting = 7-3 = 4ms, Response = 4ms<\/li>\n<li><strong>P3<\/strong>: Turnaround = 10-0 = 10ms, Waiting = 10-3 = 7ms, Response = 7ms<\/li>\n<\/ul>\n<hr>\n<h5 class=\"heading-element\" id=\"impact-of-time-quantum\"><span>Impact of Time Quantum<\/span>\n <a href=\"#impact-of-time-quantum\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h5><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/round-robin-time-quantum-impact.webp\" title=\"Time Quantum Selection Impact on Performance\" data-thumbnail=\"\/files\/round-robin-time-quantum-impact.webp\" data-sub-html=\"<h2>Time Quantum Impact Comparison showing performance tradeoffs between small, optimal, and large quantum values in Round Robin scheduling<\/h2><p>Time Quantum Selection Impact on Performance<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/round-robin-time-quantum-impact.webp' alt=\"Time Quantum Impact Comparison showing performance tradeoffs between small, optimal, and large quantum values in Round Robin scheduling\"><\/a><figcaption class=\"image-caption\">Time Quantum Selection Impact on Performance<\/figcaption>\n <\/figure><\/p>\n<p><em>Figure 5: Comparative analysis of time quantum values demonstrating the balance between response time, context switch overhead, and throughput in Round Robin scheduling.<\/em><\/p>\n<p><strong>Quantum Selection Guidelines<\/strong>:<\/p>\n<ul>\n<li><strong>Rule of Thumb<\/strong>: 80% of CPU bursts should be shorter than quantum<\/li>\n<li><strong>Typical Values<\/strong>: 10-100 milliseconds<\/li>\n<li><strong>Linux Default<\/strong>: ~100ms (adjustable via <code>sched_latency_ns<\/code>)<\/li>\n<li><strong>Context Switch Cost<\/strong>: Usually 1-10 microseconds<\/li>\n<\/ul>\n<hr>\n<h5 class=\"heading-element\" id=\"quantum-comparison-example\"><span>Quantum Comparison Example<\/span>\n <a href=\"#quantum-comparison-example\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h5><pre><code>\/\/ Demonstrate effect of different quantum values\nvoid compare_quantum_values() {\n Process procs[] = {\n {1, 0, 53, 0, 0, 0, 0, 0, 0},\n {2, 0, 17, 0, 0, 0, 0, 0, 0},\n {3, 0, 68, 0, 0, 0, 0, 0, 0},\n {4, 0, 24, 0, 0, 0, 0, 0, 0}\n };\n\n int quantum_values[] = {1, 5, 10, 20, 100};\n\n printf(&#34;Quantum\\tAvg Wait\\tAvg Response\\tContext Switches\\n&#34;);\n printf(&#34;-------\\t--------\\t------------\\t----------------\\n&#34;);\n\n for (int i = 0; i &lt; 5; i&#43;&#43;) {\n Process test_procs[4];\n memcpy(test_procs, procs, sizeof(procs));\n\n int quantum = quantum_values[i];\n round_robin_schedule(test_procs, 4, quantum);\n\n \/\/ Results would show:\n \/\/ Q=1: High wait time, excellent response, ~162 switches\n \/\/ Q=5: Good wait time, good response, ~32 switches\n \/\/ Q=10: Good wait time, good response, ~16 switches\n \/\/ Q=20: Moderate wait, moderate response, ~8 switches\n \/\/ Q=100: High wait time, poor response, ~2 switches (like FCFS)\n }\n}<\/code><\/pre><hr>\n<h5 class=\"heading-element\" id=\"advantages-and-disadvantages\"><span>Advantages and Disadvantages<\/span>\n <a href=\"#advantages-and-disadvantages\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h5><p><strong>Advantages<\/strong>:\n\u2713 <strong>Fair CPU allocation<\/strong>: Every process gets equal time\n\u2713 <strong>No starvation<\/strong>: All processes eventually execute\n\u2713 <strong>Good response time<\/strong>: Especially for interactive systems\n\u2713 <strong>Simple to implement<\/strong>: Easy to understand and code\n\u2713 <strong>Preemptive<\/strong>: Can handle time-sharing systems<\/p>\n<p><strong>Disadvantages<\/strong>:\n\u2717 <strong>Context switch overhead<\/strong>: More switches = more overhead\n\u2717 <strong>Higher average turnaround<\/strong>: Compared to SJF\n\u2717 <strong>Performance depends on quantum<\/strong>: Hard to choose optimal value\n\u2717 <strong>Not ideal for batch systems<\/strong>: Throughput may suffer\n\u2717 <strong>Penalizes CPU-bound processes<\/strong>: Long processes wait repeatedly<\/p>\n<hr>\n<h5 class=\"heading-element\" id=\"real-world-usage\"><span>Real-World Usage<\/span>\n <a href=\"#real-world-usage\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h5><p><strong>Linux Time-Sharing Scheduler (Pre-CFS)<\/strong>:<\/p>\n<pre><code>\/\/ Simplified Linux Round Robin concept (older versions)\n#define DEF_TIMESLICE (100 * HZ \/ 1000) \/\/ 100ms default\n\nvoid schedule() {\n struct task_struct *next;\n\n \/\/ Select next task from run queue\n next = pick_next_task_rr();\n\n if (next-&gt;time_slice == 0) {\n \/\/ Quantum expired - move to expired array\n next-&gt;time_slice = DEF_TIMESLICE;\n enqueue_task_expired(next);\n } else {\n \/\/ Continue with current process\n context_switch(prev, next);\n }\n}<\/code><\/pre><p><strong>Modern Applications<\/strong>:<\/p>\n<ul>\n<li><strong>Interactive Systems<\/strong>: Desktop environments, mobile OS<\/li>\n<li><strong>Time-Sharing Systems<\/strong>: Multi-user Unix\/Linux systems<\/li>\n<li><strong>Real-Time Systems<\/strong>: Combined with priority scheduling<\/li>\n<li><strong>Virtual Machines<\/strong>: CPU time sharing among VMs<\/li>\n<\/ul>\n<hr>\n<h5 class=\"heading-element\" id=\"practice-problems\"><span>Practice Problems<\/span>\n <a href=\"#practice-problems\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h5><p><strong>Problem 1<\/strong>: Four processes arrive at time 0 with burst times P1=10, P2=1, P3=2, P4=1. Calculate average waiting time for quantum=1.<\/p>\n<p><strong>Solution<\/strong>:<\/p>\n<pre><code>Time 0-1: P1 \u2192 Re-queue (9 remaining)\nTime 1-2: P2 \u2192 Complete\nTime 2-4: P3 \u2192 Complete\nTime 4-5: P4 \u2192 Complete\nTime 5-6: P1 \u2192 Re-queue (8 remaining)\n... continues ...\nTime 12-13: P1 \u2192 Complete\n\nWaiting Times: P1=(13-10)=3, P2=(2-1)=1, P3=(4-2)=2, P4=(5-1)=4\nAverage = (3&#43;1&#43;2&#43;4)\/4 = 2.5ms<\/code><\/pre><p><strong>Problem 2<\/strong>: Why is quantum=1 inefficient even though it provides best response time?<\/p>\n<p><strong>Answer<\/strong>: Context switch overhead dominates. If context switch takes 0.1ms and quantum is 1ms, system spends ~9% of time just switching contexts rather than executing processes.<\/p>\n<h4 class=\"heading-element\" id=\"4-priority-scheduling\"><span>4. Priority Scheduling<\/span>\n <a href=\"#4-priority-scheduling\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>void priority_schedule(struct process procs[], int n) {\n \/\/ Sort by priority (lower number = higher priority)\n for (int i = 0; i &lt; n - 1; i&#43;&#43;) {\n for (int j = 0; j &lt; n - i - 1; j&#43;&#43;) {\n if (procs[j].priority &gt; procs[j&#43;1].priority) {\n struct process temp = procs[j];\n procs[j] = procs[j&#43;1];\n procs[j&#43;1] = temp;\n }\n }\n }\n\n \/\/ Apply FCFS on priority-sorted list\n fcfs_schedule(procs, n);\n}\n\n\/*\nPriority Inversion Problem:\n- Low-priority task holds resource needed by high-priority task\n- High-priority task waits for low-priority task\n- Solution: Priority inheritance (boost lock holder&#39;s priority)\n*\/<\/code><\/pre><h4 class=\"heading-element\" id=\"5-linux-cfs-completely-fair-scheduler\"><span>5. Linux CFS (Completely Fair Scheduler)<\/span>\n <a href=\"#5-linux-cfs-completely-fair-scheduler\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><strong>Concept<\/strong>: Each process gets fair share of CPU based on nice value<\/p>\n<pre><code>\/\/ Simplified CFS concept\ntypedef struct {\n long vruntime; \/\/ Virtual runtime\n long weight; \/\/ Priority weight\n} cfs_task;\n\nvoid cfs_update_runtime(cfs_task *task, long delta_exec) {\n \/\/ Nice value determines weight:\n \/\/ weight = 1024 \/ (1.25 ^ nice_value)\n task-&gt;vruntime &#43;= (delta_exec * 1024) \/ task-&gt;weight;\n}\n\n\/*\nRed-Black Tree maintains tasks by vruntime:\n\n P2 (vruntime=10)\n \/ \\\n P1 P3\n (v=5) (v=15)\n\nSelection: Always pick leftmost (smallest vruntime)\nThis ensures fairness while minimizing context switches\n*\/<\/code><\/pre><h3 class=\"heading-element\" id=\"context-switching-overhead\"><span>Context Switching Overhead<\/span>\n <a href=\"#context-switching-overhead\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Costs of Context Switching:<\/strong><\/p>\n<pre><code>Direct Cost: ~1-10 microseconds\n- Save\/restore registers\n- Switch address space (if different process)\n\nIndirect Cost: ~milliseconds\n- Cache misses (cold cache)\n- TLB misses\n- Increased memory bandwidth\n\nExample:\n- Context switch: 10 \u03bcs\n- CPU stall for cache miss: 100 ns\n- 100&#43; misses per switch = 10 \u03bcs &#43; 10 \u03bcs = 20 \u03bcs visible cost<\/code><\/pre><hr>\n<h2 id=\"synchronization\">8. Synchronization Primitives<\/h2>\n<h3 class=\"heading-element\" id=\"race-conditions-and-critical-sections\"><span>Race Conditions and Critical Sections<\/span>\n <a href=\"#race-conditions-and-critical-sections\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Race Condition<\/strong>: Multiple threads\/processes access shared data, result depends on timing<\/p>\n<pre><code>\/\/ Classic race condition example\nint counter = 0;\n\n\/\/ Without synchronization (WRONG!)\nvoid increment_unsafe() {\n counter&#43;&#43;; \/\/ Not atomic! Translates to:\n \/\/ 1. Load counter into register\n \/\/ 2. Increment register\n \/\/ 3. Store register back to counter\n}\n\n\/\/ With synchronization (CORRECT)\npthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;\n\nvoid increment_safe() {\n pthread_mutex_lock(&amp;lock);\n counter&#43;&#43;; \/\/ Now atomic\n pthread_mutex_unlock(&amp;lock);\n}\n\n\/*\nWithout locks, with 1000 increments per thread (2 threads):\nExpected: 2000\nActual: Random (often 1200, 1500, etc.)\n\nReason: Interleaved execution:\nThread 1: Load (0) Increment (1) ...\nThread 2: Load (0) Increment (1) Store (1) ...\nThread 1: ... Store (1)\nResult: Both stored 1, lost an increment\n*\/<\/code><\/pre><h3 class=\"heading-element\" id=\"semaphores-detailed\"><span>Semaphores Detailed<\/span>\n <a href=\"#semaphores-detailed\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Binary Semaphore<\/strong> (value: 0 or 1)<\/p>\n<pre><code>#include &lt;semaphore.h&gt;\n\nsem_t mutex;\nsem_init(&amp;mutex, 0, 1); \/\/ Initial value = 1\n\n\/\/ Usage\nsem_wait(&amp;mutex); \/\/ P() operation - decrement\n\/\/ Critical section\nsem_post(&amp;mutex); \/\/ V() operation - increment<\/code><\/pre><p><strong>Counting Semaphore<\/strong> (value: 0 to N)<\/p>\n<pre><code>sem_t available_resources;\nsem_init(&amp;available_resources, 0, 5); \/\/ 5 resources available\n\nvoid* worker(void *arg) {\n sem_wait(&amp;available_resources); \/\/ Acquire resource\n\n \/\/ Use resource\n printf(&#34;Using resource\\n&#34;);\n sleep(1);\n\n sem_post(&amp;available_resources); \/\/ Release resource\n return NULL;\n}\n\n\/*\nExample with 3 threads and 2 resources:\nThread 1: sem_wait() \u2192 success (available=1)\nThread 2: sem_wait() \u2192 success (available=0)\nThread 3: sem_wait() \u2192 blocks (available=0)\nThread 1: sem_post() \u2192 (available=1), wakes Thread 3\n*\/<\/code><\/pre><h3 class=\"heading-element\" id=\"readers-writers-problem\"><span>Readers-Writers Problem<\/span>\n <a href=\"#readers-writers-problem\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Requirement<\/strong>: Multiple readers can access resource simultaneously, but writers need exclusive access<\/p>\n<pre><code>#include &lt;pthread.h&gt;\n#include &lt;semaphore.h&gt;\n#include &lt;stdio.h&gt;\n\nint read_count = 0;\nint shared_data = 0;\n\npthread_mutex_t read_count_lock = PTHREAD_MUTEX_INITIALIZER;\nsem_t write_lock;\n\nvoid* reader(void *arg) {\n while (1) {\n \/\/ Entry section\n pthread_mutex_lock(&amp;read_count_lock);\n read_count&#43;&#43;;\n if (read_count == 1) {\n sem_wait(&amp;write_lock); \/\/ First reader blocks writers\n }\n pthread_mutex_unlock(&amp;read_count_lock);\n\n \/\/ Critical section (read)\n printf(&#34;Reader %ld: %d\\n&#34;, (long)pthread_self(), shared_data);\n\n \/\/ Exit section\n pthread_mutex_lock(&amp;read_count_lock);\n read_count--;\n if (read_count == 0) {\n sem_post(&amp;write_lock); \/\/ Last reader allows writers\n }\n pthread_mutex_unlock(&amp;read_count_lock);\n }\n return NULL;\n}\n\nvoid* writer(void *arg) {\n while (1) {\n sem_wait(&amp;write_lock); \/\/ Exclusive access\n\n \/\/ Critical section (write)\n shared_data&#43;&#43;;\n printf(&#34;Writer %ld: Updated to %d\\n&#34;,\n (long)pthread_self(), shared_data);\n\n sem_post(&amp;write_lock);\n }\n return NULL;\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"atomic-operations-and-memory-barriers\"><span>Atomic Operations and Memory Barriers<\/span>\n <a href=\"#atomic-operations-and-memory-barriers\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>#include &lt;stdatomic.h&gt;\n#include &lt;stdint.h&gt;\n\n\/\/ Atomic variable\natomic_int counter = ATOMIC_VAR_INIT(0);\n\nvoid increment_atomic() {\n atomic_fetch_add(&amp;counter, 1); \/\/ Guaranteed atomic\n}\n\n\/\/ Memory barriers\nvoid process_data(int *data) {\n *data = 42;\n atomic_thread_fence(memory_order_release); \/\/ Release barrier\n \/\/ All writes before this point visible to other threads\n}\n\nvoid read_data(int *data) {\n atomic_thread_fence(memory_order_acquire); \/\/ Acquire barrier\n \/\/ All reads after this point see changes made before\n int value = *data;\n}\n\n\/*\nMemory Ordering Options:\n- memory_order_relaxed: No synchronization\n- memory_order_release: Release semantics\n- memory_order_acquire: Acquire semantics\n- memory_order_seq_cst: Sequential consistency (default, most expensive)\n*\/<\/code><\/pre><h3 class=\"heading-element\" id=\"lock-free-data-structures\"><span>Lock-Free Data Structures<\/span>\n <a href=\"#lock-free-data-structures\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>#include &lt;stdatomic.h&gt;\n\n\/\/ Lock-free counter\ntypedef struct {\n atomic_long value;\n} LockFreeCounter;\n\nvoid increment(LockFreeCounter *counter) {\n atomic_fetch_add(&amp;counter-&gt;value, 1);\n}\n\n\/\/ Lock-free stack using Compare-And-Swap (CAS)\ntypedef struct Node {\n int data;\n struct Node *next;\n} Node;\n\ntypedef struct {\n atomic_ptr_t top;\n} LockFreeStack;\n\nvoid push(LockFreeStack *stack, int value) {\n Node *new_node = malloc(sizeof(Node));\n new_node-&gt;data = value;\n\n Node *old_top;\n do {\n old_top = atomic_load(&amp;stack-&gt;top);\n new_node-&gt;next = old_top;\n } while (!atomic_compare_exchange_strong(&amp;stack-&gt;top,\n &amp;old_top,\n new_node));\n}\n\n\/*\nCAS (Compare-And-Swap):\n- Atomically compare value at address with expected\n- If equal, store new value\n- Return whether operation succeeded\n- No explicit locks needed\n- Wait-free for many scenarios\n*\/<\/code><\/pre><hr>\n<h2 id=\"deadlock-handling\">9. Deadlock Handling<\/h2>\n<h3 class=\"heading-element\" id=\"91-understanding-deadlocks\"><span>9.1 Understanding Deadlocks<\/span>\n <a href=\"#91-understanding-deadlocks\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>A <strong>deadlock<\/strong> is a situation where a set of processes are blocked because each process is holding a resource and waiting for another resource held by another process in the set.<\/p>\n<h4 class=\"heading-element\" id=\"coffman-conditions-necessary-conditions-for-deadlock\"><span>Coffman Conditions (Necessary Conditions for Deadlock)<\/span>\n <a href=\"#coffman-conditions-necessary-conditions-for-deadlock\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>For a deadlock to occur, <strong>all four<\/strong> conditions must hold simultaneously:<\/p>\n<ol>\n<li><strong>Mutual Exclusion<\/strong>: At least one resource must be held in a non-shareable mode<\/li>\n<li><strong>Hold and Wait<\/strong>: A process must be holding at least one resource and waiting for additional resources<\/li>\n<li><strong>No Preemption<\/strong>: Resources cannot be forcibly removed from processes<\/li>\n<li><strong>Circular Wait<\/strong>: A circular chain of processes exists where each process holds resources needed by the next<\/li>\n<\/ol>\n<pre><code>\/*\n * Example: Deadlock Scenario\n * Process P1 holds Resource R1, needs R2\n * Process P2 holds Resource R2, needs R1\n *\/\n\n#include &lt;pthread.h&gt;\n#include &lt;stdio.h&gt;\n#include &lt;unistd.h&gt;\n\npthread_mutex_t resource1 = PTHREAD_MUTEX_INITIALIZER;\npthread_mutex_t resource2 = PTHREAD_MUTEX_INITIALIZER;\n\nvoid* process1(void *arg) {\n printf(&#34;P1: Acquiring Resource 1...\\n&#34;);\n pthread_mutex_lock(&amp;resource1);\n printf(&#34;P1: Got Resource 1\\n&#34;);\n\n sleep(1); \/\/ Simulate some work\n\n printf(&#34;P1: Acquiring Resource 2...\\n&#34;);\n pthread_mutex_lock(&amp;resource2); \/\/ DEADLOCK! P2 holds R2\n printf(&#34;P1: Got Resource 2\\n&#34;);\n\n \/\/ Critical section\n printf(&#34;P1: Working with both resources\\n&#34;);\n\n pthread_mutex_unlock(&amp;resource2);\n pthread_mutex_unlock(&amp;resource1);\n return NULL;\n}\n\nvoid* process2(void *arg) {\n printf(&#34;P2: Acquiring Resource 2...\\n&#34;);\n pthread_mutex_lock(&amp;resource2);\n printf(&#34;P2: Got Resource 2\\n&#34;);\n\n sleep(1); \/\/ Simulate some work\n\n printf(&#34;P2: Acquiring Resource 1...\\n&#34;);\n pthread_mutex_lock(&amp;resource1); \/\/ DEADLOCK! P1 holds R1\n printf(&#34;P2: Got Resource 1\\n&#34;);\n\n \/\/ Critical section\n printf(&#34;P2: Working with both resources\\n&#34;);\n\n pthread_mutex_unlock(&amp;resource1);\n pthread_mutex_unlock(&amp;resource2);\n return NULL;\n}\n\nint main() {\n pthread_t t1, t2;\n\n pthread_create(&amp;t1, NULL, process1, NULL);\n pthread_create(&amp;t2, NULL, process2, NULL);\n\n pthread_join(t1, NULL);\n pthread_join(t2, NULL);\n\n return 0;\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"92-deadlock-prevention\"><span>9.2 Deadlock Prevention<\/span>\n <a href=\"#92-deadlock-prevention\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Strategy<\/strong>: Ensure at least one Coffman condition cannot hold.<\/p>\n<h4 class=\"heading-element\" id=\"921-breaking-mutual-exclusion\"><span>9.2.1 Breaking Mutual Exclusion<\/span>\n <a href=\"#921-breaking-mutual-exclusion\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><ul>\n<li>Make resources shareable (not always possible)<\/li>\n<li>Use read-write locks for read-heavy workloads<\/li>\n<\/ul>\n<pre><code>#include &lt;pthread.h&gt;\n\npthread_rwlock_t shared_resource = PTHREAD_RWLOCK_INITIALIZER;\n\nvoid* reader(void *arg) {\n pthread_rwlock_rdlock(&amp;shared_resource); \/\/ Multiple readers OK\n \/\/ Read data\n printf(&#34;Reader %ld reading\\n&#34;, (long)pthread_self());\n pthread_rwlock_unlock(&amp;shared_resource);\n return NULL;\n}\n\nvoid* writer(void *arg) {\n pthread_rwlock_wrlock(&amp;shared_resource); \/\/ Exclusive write\n \/\/ Write data\n printf(&#34;Writer %ld writing\\n&#34;, (long)pthread_self());\n pthread_rwlock_unlock(&amp;shared_resource);\n return NULL;\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"922-breaking-hold-and-wait\"><span>9.2.2 Breaking Hold and Wait<\/span>\n <a href=\"#922-breaking-hold-and-wait\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><ul>\n<li>Require processes to request all resources at once<\/li>\n<\/ul>\n<pre><code>#include &lt;pthread.h&gt;\n#include &lt;stdbool.h&gt;\n\ntypedef struct {\n pthread_mutex_t *locks;\n int num_locks;\n pthread_mutex_t allocation_lock;\n} ResourceManager;\n\n\/\/ Acquire all resources atomically\nbool acquire_all_resources(ResourceManager *rm, int *resource_ids, int count) {\n pthread_mutex_lock(&amp;rm-&gt;allocation_lock);\n\n \/\/ Try to acquire all locks\n for (int i = 0; i &lt; count; i&#43;&#43;) {\n if (pthread_mutex_trylock(&amp;rm-&gt;locks[resource_ids[i]]) != 0) {\n \/\/ Failed to acquire, release all acquired locks\n for (int j = 0; j &lt; i; j&#43;&#43;) {\n pthread_mutex_unlock(&amp;rm-&gt;locks[resource_ids[j]]);\n }\n pthread_mutex_unlock(&amp;rm-&gt;allocation_lock);\n return false;\n }\n }\n\n pthread_mutex_unlock(&amp;rm-&gt;allocation_lock);\n return true;\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"923-breaking-no-preemption\"><span>9.2.3 Breaking No Preemption<\/span>\n <a href=\"#923-breaking-no-preemption\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><ul>\n<li>Allow resources to be preempted (forcibly removed)<\/li>\n<li>Save process state and restore later<\/li>\n<\/ul>\n<pre><code>\/\/ Conceptual example - actual implementation is OS-specific\nvoid preemptable_resource_request(Process *p, Resource *r) {\n if (!try_acquire(r)) {\n \/\/ Release all currently held resources\n release_all_resources(p);\n\n \/\/ Save process state\n save_state(p);\n\n \/\/ Add to waiting queue\n add_to_queue(p, r);\n\n \/\/ Context switch to another process\n schedule_next_process();\n }\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"924-breaking-circular-wait\"><span>9.2.4 Breaking Circular Wait<\/span>\n <a href=\"#924-breaking-circular-wait\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><ul>\n<li><strong>Resource Ordering<\/strong>: Impose total ordering on all resources<\/li>\n<\/ul>\n<pre><code>\/*\n * Deadlock Prevention: Resource Ordering\n * Always acquire resources in increasing order of their IDs\n *\/\n\n#include &lt;pthread.h&gt;\n#include &lt;stdio.h&gt;\n\npthread_mutex_t resource1 = PTHREAD_MUTEX_INITIALIZER; \/\/ ID: 1\npthread_mutex_t resource2 = PTHREAD_MUTEX_INITIALIZER; \/\/ ID: 2\n\nvoid* safe_process1(void *arg) {\n \/\/ Always acquire in order: R1 before R2\n pthread_mutex_lock(&amp;resource1);\n printf(&#34;P1: Got Resource 1\\n&#34;);\n\n pthread_mutex_lock(&amp;resource2);\n printf(&#34;P1: Got Resource 2\\n&#34;);\n\n \/\/ Critical section - NO DEADLOCK!\n printf(&#34;P1: Working safely\\n&#34;);\n\n pthread_mutex_unlock(&amp;resource2);\n pthread_mutex_unlock(&amp;resource1);\n return NULL;\n}\n\nvoid* safe_process2(void *arg) {\n \/\/ Also acquire in order: R1 before R2\n pthread_mutex_lock(&amp;resource1);\n printf(&#34;P2: Got Resource 1\\n&#34;);\n\n pthread_mutex_lock(&amp;resource2);\n printf(&#34;P2: Got Resource 2\\n&#34;);\n\n \/\/ Critical section\n printf(&#34;P2: Working safely\\n&#34;);\n\n pthread_mutex_unlock(&amp;resource2);\n pthread_mutex_unlock(&amp;resource1);\n return NULL;\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"93-deadlock-avoidance\"><span>9.3 Deadlock Avoidance<\/span>\n <a href=\"#93-deadlock-avoidance\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Strategy<\/strong>: Make informed decisions about resource allocation to ensure the system never enters an unsafe state.<\/p>\n<h4 class=\"heading-element\" id=\"931-safe-state\"><span>9.3.1 Safe State<\/span>\n <a href=\"#931-safe-state\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>A state is <strong>safe<\/strong> if there exists a sequence of all processes &lt;P1, P2, &hellip;, Pn&gt; such that for each Pi, the resources that Pi can still request can be satisfied by:<\/p>\n<ul>\n<li>Currently available resources<\/li>\n<li>Resources held by all Pj where j &lt; i<\/li>\n<\/ul>\n<h4 class=\"heading-element\" id=\"932-bankers-algorithm\"><span>9.3.2 Banker&rsquo;s Algorithm<\/span>\n <a href=\"#932-bankers-algorithm\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Developed by Edsger Dijkstra, simulates allocation to test for safe state.<\/p>\n<pre><code>#include &lt;stdio.h&gt;\n#include &lt;stdbool.h&gt;\n#include &lt;string.h&gt;\n\n#define NUM_PROCESSES 5\n#define NUM_RESOURCES 3\n\ntypedef struct {\n int available[NUM_RESOURCES]; \/\/ Available resources\n int maximum[NUM_PROCESSES][NUM_RESOURCES]; \/\/ Max demand\n int allocation[NUM_PROCESSES][NUM_RESOURCES]; \/\/ Currently allocated\n int need[NUM_PROCESSES][NUM_RESOURCES]; \/\/ Remaining need\n} BankersState;\n\n\/\/ Calculate need matrix: Need = Maximum - Allocation\nvoid calculate_need(BankersState *state) {\n for (int i = 0; i &lt; NUM_PROCESSES; i&#43;&#43;) {\n for (int j = 0; j &lt; NUM_RESOURCES; j&#43;&#43;) {\n state-&gt;need[i][j] = state-&gt;maximum[i][j] - state-&gt;allocation[i][j];\n }\n }\n}\n\n\/\/ Check if system is in safe state\nbool is_safe_state(BankersState *state) {\n int work[NUM_RESOURCES];\n bool finish[NUM_PROCESSES] = {false};\n int safe_sequence[NUM_PROCESSES];\n int count = 0;\n\n \/\/ Initialize work = available\n memcpy(work, state-&gt;available, sizeof(work));\n\n \/\/ Find a process that can finish\n while (count &lt; NUM_PROCESSES) {\n bool found = false;\n\n for (int i = 0; i &lt; NUM_PROCESSES; i&#43;&#43;) {\n if (!finish[i]) {\n \/\/ Check if process i can finish\n bool can_finish = true;\n for (int j = 0; j &lt; NUM_RESOURCES; j&#43;&#43;) {\n if (state-&gt;need[i][j] &gt; work[j]) {\n can_finish = false;\n break;\n }\n }\n\n if (can_finish) {\n \/\/ Process i can finish\n \/\/ Add its allocated resources back to work\n for (int j = 0; j &lt; NUM_RESOURCES; j&#43;&#43;) {\n work[j] &#43;= state-&gt;allocation[i][j];\n }\n\n safe_sequence[count&#43;&#43;] = i;\n finish[i] = true;\n found = true;\n }\n }\n }\n\n if (!found) {\n \/\/ No process could finish - not in safe state\n printf(&#34;System is NOT in safe state!\\n&#34;);\n return false;\n }\n }\n\n \/\/ Found safe sequence\n printf(&#34;System is in SAFE state.\\nSafe sequence: &#34;);\n for (int i = 0; i &lt; NUM_PROCESSES; i&#43;&#43;) {\n printf(&#34;P%d &#34;, safe_sequence[i]);\n }\n printf(&#34;\\n&#34;);\n return true;\n}\n\n\/\/ Request resources using Banker&#39;s Algorithm\nbool request_resources(BankersState *state, int process_id, int request[]) {\n printf(&#34;\\nProcess P%d requesting: &#34;, process_id);\n for (int i = 0; i &lt; NUM_RESOURCES; i&#43;&#43;) {\n printf(&#34;%d &#34;, request[i]);\n }\n printf(&#34;\\n&#34;);\n\n \/\/ Check if request &lt;= need\n for (int i = 0; i &lt; NUM_RESOURCES; i&#43;&#43;) {\n if (request[i] &gt; state-&gt;need[process_id][i]) {\n printf(&#34;Error: Process exceeded its maximum claim!\\n&#34;);\n return false;\n }\n }\n\n \/\/ Check if request &lt;= available\n for (int i = 0; i &lt; NUM_RESOURCES; i&#43;&#43;) {\n if (request[i] &gt; state-&gt;available[i]) {\n printf(&#34;Process must wait - resources not available\\n&#34;);\n return false;\n }\n }\n\n \/\/ Pretend to allocate resources\n BankersState test_state = *state;\n for (int i = 0; i &lt; NUM_RESOURCES; i&#43;&#43;) {\n test_state.available[i] -= request[i];\n test_state.allocation[process_id][i] &#43;= request[i];\n test_state.need[process_id][i] -= request[i];\n }\n\n \/\/ Check if resulting state is safe\n if (is_safe_state(&amp;test_state)) {\n \/\/ Safe - commit the allocation\n *state = test_state;\n printf(&#34;Request GRANTED\\n&#34;);\n return true;\n } else {\n \/\/ Unsafe - rollback\n printf(&#34;Request DENIED (would lead to unsafe state)\\n&#34;);\n return false;\n }\n}\n\n\/\/ Example usage\nint main() {\n BankersState state = {\n .available = {3, 3, 2},\n .maximum = {\n {7, 5, 3}, \/\/ P0\n {3, 2, 2}, \/\/ P1\n {9, 0, 2}, \/\/ P2\n {2, 2, 2}, \/\/ P3\n {4, 3, 3} \/\/ P4\n },\n .allocation = {\n {0, 1, 0}, \/\/ P0\n {2, 0, 0}, \/\/ P1\n {3, 0, 2}, \/\/ P2\n {2, 1, 1}, \/\/ P3\n {0, 0, 2} \/\/ P4\n }\n };\n\n calculate_need(&amp;state);\n\n printf(&#34;Initial state:\\n&#34;);\n printf(&#34;Available: &#34;);\n for (int i = 0; i &lt; NUM_RESOURCES; i&#43;&#43;) {\n printf(&#34;%d &#34;, state.available[i]);\n }\n printf(&#34;\\n\\n&#34;);\n\n is_safe_state(&amp;state);\n\n \/\/ P1 requests (1, 0, 2)\n int request1[] = {1, 0, 2};\n request_resources(&amp;state, 1, request1);\n\n return 0;\n}\n\n\/*\nOutput Analysis:\n- Algorithm checks if granting request leads to safe state\n- If safe: Grant request\n- If unsafe: Process must wait\n- Prevents deadlock by conservative allocation\n*\/<\/code><\/pre><h3 class=\"heading-element\" id=\"94-deadlock-detection\"><span>9.4 Deadlock Detection<\/span>\n <a href=\"#94-deadlock-detection\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Strategy<\/strong>: Allow deadlocks to occur, detect them, and recover.<\/p>\n<h4 class=\"heading-element\" id=\"941-resource-allocation-graph-rag\"><span>9.4.1 Resource Allocation Graph (RAG)<\/span>\n <a href=\"#941-resource-allocation-graph-rag\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>#include &lt;stdio.h&gt;\n#include &lt;stdbool.h&gt;\n\n#define MAX_PROCESSES 10\n#define MAX_RESOURCES 10\n\ntypedef struct {\n int num_processes;\n int num_resources;\n \/\/ Adjacency matrix: graph[i][j] = 1 if edge from i to j\n int graph[MAX_PROCESSES &#43; MAX_RESOURCES][MAX_PROCESSES &#43; MAX_RESOURCES];\n} RAG;\n\n\/\/ Detect cycle using DFS\nbool has_cycle_util(RAG *rag, int v, bool visited[], bool rec_stack[]) {\n if (!visited[v]) {\n visited[v] = true;\n rec_stack[v] = true;\n\n int total_nodes = rag-&gt;num_processes &#43; rag-&gt;num_resources;\n\n for (int i = 0; i &lt; total_nodes; i&#43;&#43;) {\n if (rag-&gt;graph[v][i]) {\n if (!visited[i] &amp;&amp; has_cycle_util(rag, i, visited, rec_stack)) {\n return true;\n } else if (rec_stack[i]) {\n return true;\n }\n }\n }\n }\n\n rec_stack[v] = false;\n return false;\n}\n\nbool detect_deadlock(RAG *rag) {\n int total_nodes = rag-&gt;num_processes &#43; rag-&gt;num_resources;\n bool visited[MAX_PROCESSES &#43; MAX_RESOURCES] = {false};\n bool rec_stack[MAX_PROCESSES &#43; MAX_RESOURCES] = {false};\n\n for (int i = 0; i &lt; total_nodes; i&#43;&#43;) {\n if (has_cycle_util(rag, i, visited, rec_stack)) {\n printf(&#34;DEADLOCK DETECTED! Cycle found in RAG\\n&#34;);\n return true;\n }\n }\n\n printf(&#34;No deadlock detected\\n&#34;);\n return false;\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"942-wait-for-graph-for-single-instance-resources\"><span>9.4.2 Wait-For Graph (for single-instance resources)<\/span>\n <a href=\"#942-wait-for-graph-for-single-instance-resources\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * Wait-For Graph: Variant of RAG for single-instance resources\n * Process Pi waits for Pj if Pi is waiting for a resource held by Pj\n * Deadlock exists iff WFG contains a cycle\n *\/\n\ntypedef struct {\n int num_processes;\n bool wait_for[MAX_PROCESSES][MAX_PROCESSES]; \/\/ wait_for[i][j] = true if Pi waits for Pj\n} WaitForGraph;\n\n\/\/ Detect cycle in wait-for graph\nbool detect_deadlock_wfg(WaitForGraph *wfg) {\n for (int start = 0; start &lt; wfg-&gt;num_processes; start&#43;&#43;) {\n bool visited[MAX_PROCESSES] = {false};\n int current = start;\n\n while (true) {\n if (visited[current]) {\n if (current == start) {\n printf(&#34;Deadlock: Cycle detected starting at P%d\\n&#34;, start);\n return true;\n }\n break;\n }\n\n visited[current] = true;\n\n \/\/ Find next process in potential cycle\n bool found_next = false;\n for (int next = 0; next &lt; wfg-&gt;num_processes; next&#43;&#43;) {\n if (wfg-&gt;wait_for[current][next]) {\n current = next;\n found_next = true;\n break;\n }\n }\n\n if (!found_next) break;\n }\n }\n\n return false;\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"95-deadlock-recovery\"><span>9.5 Deadlock Recovery<\/span>\n <a href=\"#95-deadlock-recovery\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Once deadlock is detected, recovery strategies include:<\/p>\n<h4 class=\"heading-element\" id=\"951-process-termination\"><span>9.5.1 Process Termination<\/span>\n <a href=\"#951-process-termination\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * Recovery Strategy 1: Abort processes to break cycle\n *\/\n\n\/\/ Abort all deadlocked processes\nvoid abort_all_deadlocked() {\n \/\/ Terminate all processes in deadlock\n \/\/ Restart them from beginning\n \/\/ Simple but expensive\n}\n\n\/\/ Abort processes one at a time\nvoid abort_one_at_a_time(int *deadlocked_processes, int count) {\n \/\/ Selection criteria (choose minimum cost):\n \/\/ 1. Process priority\n \/\/ 2. How long process has computed\n \/\/ 3. Resources used\n \/\/ 4. Resources needed to complete\n \/\/ 5. Number of processes that need to be terminated\n\n int min_cost_process = select_min_cost_process(deadlocked_processes, count);\n terminate_process(min_cost_process);\n\n \/\/ Check if deadlock still exists\n if (detect_deadlock()) {\n abort_one_at_a_time(deadlocked_processes, count - 1);\n }\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"952-resource-preemption\"><span>9.5.2 Resource Preemption<\/span>\n <a href=\"#952-resource-preemption\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * Recovery Strategy 2: Preempt resources from processes\n *\/\n\ntypedef struct {\n int process_id;\n int resources_held[MAX_RESOURCES];\n int execution_time;\n int priority;\n} ProcessInfo;\n\n\/\/ Select victim process for preemption\nint select_victim(ProcessInfo *processes, int count) {\n int victim = 0;\n int min_cost = INT_MAX;\n\n for (int i = 0; i &lt; count; i&#43;&#43;) {\n \/\/ Cost factors:\n \/\/ - Number of resources held\n \/\/ - Execution time (prefer younger processes)\n \/\/ - Priority (prefer low priority)\n\n int resource_count = 0;\n for (int j = 0; j &lt; MAX_RESOURCES; j&#43;&#43;) {\n resource_count &#43;= processes[i].resources_held[j];\n }\n\n int cost = resource_count * 10 &#43;\n processes[i].execution_time -\n processes[i].priority * 5;\n\n if (cost &lt; min_cost) {\n min_cost = cost;\n victim = i;\n }\n }\n\n return victim;\n}\n\n\/\/ Rollback victim to safe state\nvoid rollback_and_preempt(int victim_id) {\n \/\/ 1. Save state (checkpoint)\n save_checkpoint(victim_id);\n\n \/\/ 2. Preempt resources\n release_all_resources(victim_id);\n\n \/\/ 3. Rollback to previous checkpoint\n restore_checkpoint(victim_id);\n\n \/\/ To avoid starvation: use aging or limits on preemption count\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"96-practical-deadlock-handling-in-modern-systems\"><span>9.6 Practical Deadlock Handling in Modern Systems<\/span>\n <a href=\"#96-practical-deadlock-handling-in-modern-systems\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"linux-approach\"><span>Linux Approach<\/span>\n <a href=\"#linux-approach\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * Linux uses multiple strategies:\n * 1. Timeouts on lock acquisition (trylock with timeout)\n * 2. Lock ordering conventions\n * 3. Lockdep: Runtime deadlock detector (detects potential deadlocks)\n *\/\n\n#include &lt;pthread.h&gt;\n#include &lt;errno.h&gt;\n#include &lt;time.h&gt;\n\n\/\/ Trylock with timeout\nbool acquire_with_timeout(pthread_mutex_t *lock, int timeout_ms) {\n struct timespec timeout;\n clock_gettime(CLOCK_REALTIME, &amp;timeout);\n timeout.tv_sec &#43;= timeout_ms \/ 1000;\n timeout.tv_nsec &#43;= (timeout_ms % 1000) * 1000000;\n\n if (timeout.tv_nsec &gt;= 1000000000) {\n timeout.tv_sec&#43;&#43;;\n timeout.tv_nsec -= 1000000000;\n }\n\n int result = pthread_mutex_timedlock(lock, &amp;timeout);\n if (result == ETIMEDOUT) {\n printf(&#34;Lock acquisition timeout - potential deadlock\\n&#34;);\n return false;\n }\n\n return (result == 0);\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"database-systems\"><span>Database Systems<\/span>\n <a href=\"#database-systems\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * Database Deadlock Handling:\n * - Wait-Die: Older transaction waits, younger dies\n * - Wound-Wait: Older transaction wounds (preempts) younger\n *\/\n\ntypedef struct {\n int transaction_id;\n long timestamp; \/\/ Transaction start time\n} Transaction;\n\n\/\/ Wait-Die Scheme\nvoid wait_die(Transaction *requester, Transaction *holder) {\n if (requester-&gt;timestamp &lt; holder-&gt;timestamp) {\n \/\/ Older transaction waits\n wait_for_resource(requester);\n } else {\n \/\/ Younger transaction dies (aborts)\n abort_transaction(requester);\n restart_transaction(requester);\n }\n}\n\n\/\/ Wound-Wait Scheme\nvoid wound_wait(Transaction *requester, Transaction *holder) {\n if (requester-&gt;timestamp &lt; holder-&gt;timestamp) {\n \/\/ Older wounds younger (preempts)\n abort_transaction(holder);\n grant_resource(requester);\n } else {\n \/\/ Younger waits\n wait_for_resource(requester);\n }\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"97-comparison-of-deadlock-handling-methods\"><span>9.7 Comparison of Deadlock Handling Methods<\/span>\n <a href=\"#97-comparison-of-deadlock-handling-methods\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Method<\/th>\n <th>Prevention<\/th>\n <th>Avoidance<\/th>\n <th>Detection + Recovery<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Approach<\/strong><\/td>\n <td>Prevent one Coffman condition<\/td>\n <td>Don&rsquo;t enter unsafe state<\/td>\n <td>Allow deadlock, then fix<\/td>\n <\/tr>\n <tr>\n <td><strong>Resource Utilization<\/strong><\/td>\n <td>Low (conservative)<\/td>\n <td>Medium<\/td>\n <td>High (optimistic)<\/td>\n <\/tr>\n <tr>\n <td><strong>Runtime Overhead<\/strong><\/td>\n <td>Low<\/td>\n <td>Medium (safety check)<\/td>\n <td>Low (periodic detection)<\/td>\n <\/tr>\n <tr>\n <td><strong>Implementation<\/strong><\/td>\n <td>Easiest<\/td>\n <td>Complex (Banker&rsquo;s)<\/td>\n <td>Medium<\/td>\n <\/tr>\n <tr>\n <td><strong>Used In<\/strong><\/td>\n <td>Simple systems<\/td>\n <td>Real-time systems<\/td>\n <td>Databases, OSes<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"key-takeaways\"><span>Key Takeaways<\/span>\n <a href=\"#key-takeaways\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li><strong>Deadlock requires all 4 Coffman conditions simultaneously<\/strong><\/li>\n<li><strong>Prevention<\/strong>: Break at least one condition (most common: resource ordering)<\/li>\n<li><strong>Avoidance<\/strong>: Use Banker&rsquo;s algorithm to stay in safe state<\/li>\n<li><strong>Detection<\/strong>: Use RAG or WFG to find cycles<\/li>\n<li><strong>Recovery<\/strong>: Terminate processes or preempt resources<\/li>\n<li><strong>Real systems use hybrid approaches<\/strong>: Prevention + detection with timeouts<\/li>\n<\/ol>\n<hr>\n<h2 id=\"memory-management\">10. Memory Management<\/h2>\n<h3 class=\"heading-element\" id=\"101-memory-management-basics\"><span>10.1 Memory Management Basics<\/span>\n <a href=\"#101-memory-management-basics\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Memory management<\/strong> is responsible for:<\/p>\n<ul>\n<li>Keeping track of which parts of memory are in use<\/li>\n<li>Allocating and deallocating memory for processes<\/li>\n<li>Managing the swap between main memory and disk<\/li>\n<li>Protecting memory space between processes<\/li>\n<\/ul>\n<h4 class=\"heading-element\" id=\"memory-hierarchy\"><span>Memory Hierarchy<\/span>\n <a href=\"#memory-hierarchy\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>Registers (CPU) &lt;-- Fastest, smallest, most expensive\n \u2193\nCache (L1, L2, L3)\n \u2193\nMain Memory (RAM)\n \u2193\nSecondary Storage (SSD)\n \u2193\nTertiary Storage (HDD) &lt;-- Slowest, largest, cheapest<\/code><\/pre><h3 class=\"heading-element\" id=\"102-address-binding\"><span>10.2 Address Binding<\/span>\n <a href=\"#102-address-binding\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Address binding<\/strong> is the process of mapping logical addresses to physical addresses.<\/p>\n<pre><code>\/*\n * Types of Addresses:\n * 1. Symbolic address: Variable names in source code (e.g., &#34;x&#34;)\n * 2. Relocatable address: Relative address from program start (e.g., &#34;14 bytes from start&#34;)\n * 3. Absolute address: Physical memory address (e.g., &#34;0x400014&#34;)\n *\/\n\n\/\/ Example: Compile-time binding (rarely used)\n\/\/ Address is fixed at compile time\nint x = 10; \/\/ Compiler assigns absolute address 0x1000\n\n\/\/ Run-time binding (most common in modern systems)\n\/\/ Address determined when process loads\nint *ptr = malloc(sizeof(int)); \/\/ Address assigned at runtime<\/code><\/pre><h4 class=\"heading-element\" id=\"binding-times\"><span>Binding Times<\/span>\n <a href=\"#binding-times\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><ol>\n<li><strong>Compile time<\/strong>: If memory location known a priori (obsolete)<\/li>\n<li><strong>Load time<\/strong>: Addresses generated when program loaded<\/li>\n<li><strong>Execution time<\/strong>: Addresses can change during execution (requires hardware support - MMU)<\/li>\n<\/ol>\n<h3 class=\"heading-element\" id=\"103-logical-vs-physical-address-space\"><span>10.3 Logical vs Physical Address Space<\/span>\n <a href=\"#103-logical-vs-physical-address-space\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * Logical Address (Virtual Address):\n * - Generated by CPU\n * - Used by user programs\n * - Address space from 0 to MAX\n *\n * Physical Address:\n * - Actual location in memory\n * - Seen by memory unit\n * - Mapped by MMU (Memory Management Unit)\n *\/\n\n\/\/ Example: MMU Translation\ntypedef struct {\n uint32_t logical_address;\n uint32_t physical_address;\n uint32_t base_register; \/\/ Relocation register\n uint32_t limit_register; \/\/ Size of process memory\n} AddressTranslation;\n\nuint32_t mmu_translate(AddressTranslation *at, uint32_t logical_addr) {\n \/\/ Check if address is within bounds\n if (logical_addr &gt;= at-&gt;limit_register) {\n printf(&#34;ERROR: Segmentation fault - address out of bounds\\n&#34;);\n return 0xFFFFFFFF; \/\/ Error indicator\n }\n\n \/\/ Simple relocation: Physical = Logical &#43; Base\n at-&gt;physical_address = logical_addr &#43; at-&gt;base_register;\n\n printf(&#34;Logical: 0x%X \u2192 Physical: 0x%X\\n&#34;,\n logical_addr, at-&gt;physical_address);\n\n return at-&gt;physical_address;\n}\n\n\/\/ Example usage\nint main() {\n AddressTranslation at = {\n .base_register = 0x10000, \/\/ Process starts at 0x10000\n .limit_register = 0x5000 \/\/ Process size is 0x5000 bytes\n };\n\n \/\/ Translate logical address 0x1234\n mmu_translate(&amp;at, 0x1234); \/\/ \u2192 Physical: 0x11234\n\n \/\/ Attempt to access beyond limit\n mmu_translate(&amp;at, 0x6000); \/\/ \u2192 ERROR!\n\n return 0;\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"104-contiguous-memory-allocation\"><span>10.4 Contiguous Memory Allocation<\/span>\n <a href=\"#104-contiguous-memory-allocation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"fixed-partitioning-static\"><span>Fixed Partitioning (Static)<\/span>\n <a href=\"#fixed-partitioning-static\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>#define NUM_PARTITIONS 4\n\ntypedef struct {\n uint32_t start_address;\n uint32_t size;\n bool is_allocated;\n int process_id;\n} Partition;\n\nPartition partitions[NUM_PARTITIONS] = {\n {0x00000, 0x10000, false, -1}, \/\/ 64KB\n {0x10000, 0x20000, false, -1}, \/\/ 128KB\n {0x30000, 0x30000, false, -1}, \/\/ 192KB\n {0x60000, 0x40000, false, -1} \/\/ 256KB\n};\n\n\/*\n * Problems with fixed partitioning:\n * 1. Internal fragmentation: Unused space within partition\n * 2. Limited number of processes\n * 3. Inflexible partition sizes\n *\/<\/code><\/pre><h4 class=\"heading-element\" id=\"dynamic-partitioning-variable\"><span>Dynamic Partitioning (Variable)<\/span>\n <a href=\"#dynamic-partitioning-variable\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>typedef struct MemoryBlock {\n uint32_t start_address;\n uint32_t size;\n bool is_free;\n int process_id;\n struct MemoryBlock *next;\n} MemoryBlock;\n\n\/\/ Global memory list\nMemoryBlock *memory_list = NULL;\n\nvoid init_memory(uint32_t total_size) {\n memory_list = malloc(sizeof(MemoryBlock));\n memory_list-&gt;start_address = 0;\n memory_list-&gt;size = total_size;\n memory_list-&gt;is_free = true;\n memory_list-&gt;process_id = -1;\n memory_list-&gt;next = NULL;\n}\n\n\/\/ Display current memory state\nvoid display_memory() {\n MemoryBlock *current = memory_list;\n printf(&#34;\\n=== Memory Map ===\\n&#34;);\n while (current != NULL) {\n printf(&#34;[0x%06X - 0x%06X] Size: %6d bytes | &#34;,\n current-&gt;start_address,\n current-&gt;start_address &#43; current-&gt;size - 1,\n current-&gt;size);\n\n if (current-&gt;is_free) {\n printf(&#34;FREE\\n&#34;);\n } else {\n printf(&#34;Process %d\\n&#34;, current-&gt;process_id);\n }\n current = current-&gt;next;\n }\n printf(&#34;==================\\n\\n&#34;);\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"105-memory-allocation-algorithms\"><span>10.5 Memory Allocation Algorithms<\/span>\n <a href=\"#105-memory-allocation-algorithms\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"1051-first-fit\"><span>10.5.1 First Fit<\/span>\n <a href=\"#1051-first-fit\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Allocate the <strong>first<\/strong> hole that is large enough.<\/p>\n<pre><code>\/*\n * First Fit: Fast, but may leave small unusable holes at beginning\n *\/\n\nbool first_fit(int process_id, uint32_t required_size) {\n MemoryBlock *current = memory_list;\n\n while (current != NULL) {\n if (current-&gt;is_free &amp;&amp; current-&gt;size &gt;= required_size) {\n \/\/ Found suitable block\n if (current-&gt;size &gt; required_size) {\n \/\/ Split the block\n MemoryBlock *new_block = malloc(sizeof(MemoryBlock));\n new_block-&gt;start_address = current-&gt;start_address &#43; required_size;\n new_block-&gt;size = current-&gt;size - required_size;\n new_block-&gt;is_free = true;\n new_block-&gt;process_id = -1;\n new_block-&gt;next = current-&gt;next;\n\n current-&gt;next = new_block;\n current-&gt;size = required_size;\n }\n\n current-&gt;is_free = false;\n current-&gt;process_id = process_id;\n\n printf(&#34;First Fit: Allocated %d bytes for Process %d at 0x%X\\n&#34;,\n required_size, process_id, current-&gt;start_address);\n\n return true;\n }\n current = current-&gt;next;\n }\n\n printf(&#34;First Fit: Allocation failed for Process %d\\n&#34;, process_id);\n return false;\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"1052-best-fit\"><span>10.5.2 Best Fit<\/span>\n <a href=\"#1052-best-fit\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Allocate the <strong>smallest<\/strong> hole that is large enough.<\/p>\n<pre><code>\/*\n * Best Fit: Minimizes wasted space but slower (searches entire list)\n * May create many tiny unusable holes\n *\/\n\nbool best_fit(int process_id, uint32_t required_size) {\n MemoryBlock *current = memory_list;\n MemoryBlock *best_block = NULL;\n uint32_t min_leftover = UINT32_MAX;\n\n \/\/ Search for best fitting block\n while (current != NULL) {\n if (current-&gt;is_free &amp;&amp; current-&gt;size &gt;= required_size) {\n uint32_t leftover = current-&gt;size - required_size;\n if (leftover &lt; min_leftover) {\n min_leftover = leftover;\n best_block = current;\n }\n }\n current = current-&gt;next;\n }\n\n if (best_block == NULL) {\n printf(&#34;Best Fit: Allocation failed for Process %d\\n&#34;, process_id);\n return false;\n }\n\n \/\/ Allocate the best block\n if (best_block-&gt;size &gt; required_size) {\n MemoryBlock *new_block = malloc(sizeof(MemoryBlock));\n new_block-&gt;start_address = best_block-&gt;start_address &#43; required_size;\n new_block-&gt;size = best_block-&gt;size - required_size;\n new_block-&gt;is_free = true;\n new_block-&gt;process_id = -1;\n new_block-&gt;next = best_block-&gt;next;\n\n best_block-&gt;next = new_block;\n best_block-&gt;size = required_size;\n }\n\n best_block-&gt;is_free = false;\n best_block-&gt;process_id = process_id;\n\n printf(&#34;Best Fit: Allocated %d bytes for Process %d at 0x%X (leftover: %d)\\n&#34;,\n required_size, process_id, best_block-&gt;start_address, min_leftover);\n\n return true;\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"1053-worst-fit\"><span>10.5.3 Worst Fit<\/span>\n <a href=\"#1053-worst-fit\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Allocate the <strong>largest<\/strong> hole.<\/p>\n<pre><code>\/*\n * Worst Fit: Leaves largest leftover (may be more usable)\n * Slowest (searches entire list)\n *\/\n\nbool worst_fit(int process_id, uint32_t required_size) {\n MemoryBlock *current = memory_list;\n MemoryBlock *worst_block = NULL;\n uint32_t max_size = 0;\n\n \/\/ Search for largest block\n while (current != NULL) {\n if (current-&gt;is_free &amp;&amp; current-&gt;size &gt;= required_size) {\n if (current-&gt;size &gt; max_size) {\n max_size = current-&gt;size;\n worst_block = current;\n }\n }\n current = current-&gt;next;\n }\n\n if (worst_block == NULL) {\n printf(&#34;Worst Fit: Allocation failed for Process %d\\n&#34;, process_id);\n return false;\n }\n\n \/\/ Allocate the worst (largest) block\n if (worst_block-&gt;size &gt; required_size) {\n MemoryBlock *new_block = malloc(sizeof(MemoryBlock));\n new_block-&gt;start_address = worst_block-&gt;start_address &#43; required_size;\n new_block-&gt;size = worst_block-&gt;size - required_size;\n new_block-&gt;is_free = true;\n new_block-&gt;process_id = -1;\n new_block-&gt;next = worst_block-&gt;next;\n\n worst_block-&gt;next = new_block;\n worst_block-&gt;size = required_size;\n }\n\n worst_block-&gt;is_free = false;\n worst_block-&gt;process_id = process_id;\n\n printf(&#34;Worst Fit: Allocated %d bytes for Process %d at 0x%X\\n&#34;,\n required_size, process_id, worst_block-&gt;start_address);\n\n return true;\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"1054-next-fit\"><span>10.5.4 Next Fit<\/span>\n <a href=\"#1054-next-fit\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Similar to First Fit, but starts search from last allocation point.<\/p>\n<pre><code>\/*\n * Next Fit: Variant of First Fit that continues from last allocation\n * Distributes holes more evenly across memory\n *\/\n\nMemoryBlock *last_allocated = NULL;\n\nbool next_fit(int process_id, uint32_t required_size) {\n if (last_allocated == NULL) {\n last_allocated = memory_list;\n }\n\n MemoryBlock *current = last_allocated;\n MemoryBlock *start = current;\n\n \/\/ Search from last allocation point\n do {\n if (current-&gt;is_free &amp;&amp; current-&gt;size &gt;= required_size) {\n \/\/ Found suitable block\n if (current-&gt;size &gt; required_size) {\n MemoryBlock *new_block = malloc(sizeof(MemoryBlock));\n new_block-&gt;start_address = current-&gt;start_address &#43; required_size;\n new_block-&gt;size = current-&gt;size - required_size;\n new_block-&gt;is_free = true;\n new_block-&gt;process_id = -1;\n new_block-&gt;next = current-&gt;next;\n\n current-&gt;next = new_block;\n current-&gt;size = required_size;\n }\n\n current-&gt;is_free = false;\n current-&gt;process_id = process_id;\n last_allocated = current-&gt;next ? current-&gt;next : memory_list;\n\n printf(&#34;Next Fit: Allocated %d bytes for Process %d at 0x%X\\n&#34;,\n required_size, process_id, current-&gt;start_address);\n\n return true;\n }\n\n current = current-&gt;next ? current-&gt;next : memory_list;\n } while (current != start);\n\n printf(&#34;Next Fit: Allocation failed for Process %d\\n&#34;, process_id);\n return false;\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"106-fragmentation\"><span>10.6 Fragmentation<\/span>\n <a href=\"#106-fragmentation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"external-fragmentation\"><span>External Fragmentation<\/span>\n <a href=\"#external-fragmentation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Total memory space exists to satisfy request, but it&rsquo;s not contiguous.<\/p>\n<pre><code>\/*\n * Example of External Fragmentation:\n *\n * Before:\n * [P1: 10KB] [FREE: 5KB] [P2: 10KB] [FREE: 5KB] [P3: 10KB]\n *\n * Request: 8KB process\n * Problem: Total free = 10KB, but largest contiguous = 5KB\n *\n * Solution: Compaction\n *\/\n\nvoid compact_memory() {\n MemoryBlock *current = memory_list;\n uint32_t compact_address = 0;\n\n printf(&#34;Starting memory compaction...\\n&#34;);\n\n \/\/ Move all allocated blocks to beginning\n while (current != NULL) {\n if (!current-&gt;is_free) {\n if (current-&gt;start_address != compact_address) {\n printf(&#34;Moving Process %d from 0x%X to 0x%X\\n&#34;,\n current-&gt;process_id,\n current-&gt;start_address,\n compact_address);\n\n \/\/ In real system, copy memory content here\n current-&gt;start_address = compact_address;\n }\n compact_address &#43;= current-&gt;size;\n }\n current = current-&gt;next;\n }\n\n \/\/ Merge all free blocks at the end\n current = memory_list;\n MemoryBlock *prev = NULL;\n uint32_t total_free = 0;\n\n while (current != NULL) {\n if (current-&gt;is_free) {\n total_free &#43;= current-&gt;size;\n MemoryBlock *temp = current;\n current = current-&gt;next;\n\n if (prev != NULL) {\n prev-&gt;next = current;\n }\n free(temp);\n } else {\n prev = current;\n current = current-&gt;next;\n }\n }\n\n \/\/ Create single large free block\n if (total_free &gt; 0) {\n MemoryBlock *free_block = malloc(sizeof(MemoryBlock));\n free_block-&gt;start_address = compact_address;\n free_block-&gt;size = total_free;\n free_block-&gt;is_free = true;\n free_block-&gt;process_id = -1;\n free_block-&gt;next = NULL;\n\n if (prev != NULL) {\n prev-&gt;next = free_block;\n } else {\n memory_list = free_block;\n }\n }\n\n printf(&#34;Compaction complete. Free space: %d bytes at 0x%X\\n&#34;,\n total_free, compact_address);\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"internal-fragmentation\"><span>Internal Fragmentation<\/span>\n <a href=\"#internal-fragmentation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Allocated memory is larger than requested, wasted space within allocated block.<\/p>\n<pre><code>\/*\n * Example: Internal Fragmentation\n * Process requests 1 byte, system allocates minimum block of 64 bytes\n * Wasted: 63 bytes (internal fragmentation)\n *\/\n\n#define MIN_ALLOCATION_SIZE 64\n\nvoid* allocate_with_internal_frag(size_t requested_size) {\n size_t actual_size = requested_size;\n\n if (actual_size &lt; MIN_ALLOCATION_SIZE) {\n actual_size = MIN_ALLOCATION_SIZE;\n printf(&#34;Internal Fragmentation: Requested %zu, allocated %zu, wasted %zu bytes\\n&#34;,\n requested_size, actual_size, actual_size - requested_size);\n }\n\n return malloc(actual_size);\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"107-buddy-system-allocator\"><span>10.7 Buddy System Allocator<\/span>\n <a href=\"#107-buddy-system-allocator\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Efficient compromise between fixed and variable partitioning.<\/p>\n<pre><code>#include &lt;math.h&gt;\n\n#define MAX_ORDER 10 \/\/ 2^10 = 1024 units\n#define MEMORY_SIZE (1 &lt;&lt; MAX_ORDER)\n\ntypedef struct BuddyBlock {\n uint32_t address;\n int order; \/\/ Block size = 2^order\n bool is_free;\n struct BuddyBlock *next;\n} BuddyBlock;\n\nBuddyBlock *free_lists[MAX_ORDER &#43; 1];\n\nvoid init_buddy_system() {\n for (int i = 0; i &lt;= MAX_ORDER; i&#43;&#43;) {\n free_lists[i] = NULL;\n }\n\n \/\/ Initially one large free block\n free_lists[MAX_ORDER] = malloc(sizeof(BuddyBlock));\n free_lists[MAX_ORDER]-&gt;address = 0;\n free_lists[MAX_ORDER]-&gt;order = MAX_ORDER;\n free_lists[MAX_ORDER]-&gt;is_free = true;\n free_lists[MAX_ORDER]-&gt;next = NULL;\n}\n\n\/\/ Calculate buddy address\nuint32_t get_buddy_address(uint32_t address, int order) {\n return address ^ (1 &lt;&lt; order);\n}\n\nvoid* buddy_alloc(size_t size) {\n \/\/ Find minimum order that fits request\n int order = 0;\n while ((1 &lt;&lt; order) &lt; size) {\n order&#43;&#43;;\n }\n\n if (order &gt; MAX_ORDER) {\n printf(&#34;Buddy: Request too large\\n&#34;);\n return NULL;\n }\n\n \/\/ Find available block\n int current_order = order;\n while (current_order &lt;= MAX_ORDER &amp;&amp; free_lists[current_order] == NULL) {\n current_order&#43;&#43;;\n }\n\n if (current_order &gt; MAX_ORDER) {\n printf(&#34;Buddy: No memory available\\n&#34;);\n return NULL;\n }\n\n \/\/ Remove block from free list\n BuddyBlock *block = free_lists[current_order];\n free_lists[current_order] = block-&gt;next;\n\n \/\/ Split block until we reach desired order\n while (current_order &gt; order) {\n current_order--;\n\n \/\/ Create buddy block\n BuddyBlock *buddy = malloc(sizeof(BuddyBlock));\n buddy-&gt;address = block-&gt;address &#43; (1 &lt;&lt; current_order);\n buddy-&gt;order = current_order;\n buddy-&gt;is_free = true;\n buddy-&gt;next = free_lists[current_order];\n free_lists[current_order] = buddy;\n\n printf(&#34;Buddy: Split block at 0x%X (order %d) into 0x%X and 0x%X\\n&#34;,\n block-&gt;address, current_order &#43; 1,\n block-&gt;address, buddy-&gt;address);\n }\n\n block-&gt;order = order;\n block-&gt;is_free = false;\n\n printf(&#34;Buddy: Allocated %d bytes (2^%d) at 0x%X\\n&#34;,\n 1 &lt;&lt; order, order, block-&gt;address);\n\n return (void *)(uintptr_t)block-&gt;address;\n}\n\nvoid buddy_free(void *ptr, int order) {\n uint32_t address = (uint32_t)(uintptr_t)ptr;\n\n printf(&#34;Buddy: Freeing block at 0x%X (order %d)\\n&#34;, address, order);\n\n \/\/ Try to coalesce with buddy\n while (order &lt; MAX_ORDER) {\n uint32_t buddy_addr = get_buddy_address(address, order);\n\n \/\/ Search for free buddy\n BuddyBlock *prev = NULL;\n BuddyBlock *current = free_lists[order];\n bool found_buddy = false;\n\n while (current != NULL) {\n if (current-&gt;address == buddy_addr &amp;&amp; current-&gt;is_free) {\n \/\/ Found free buddy - coalesce\n if (prev != NULL) {\n prev-&gt;next = current-&gt;next;\n } else {\n free_lists[order] = current-&gt;next;\n }\n\n free(current);\n found_buddy = true;\n\n printf(&#34;Buddy: Coalescing 0x%X and 0x%X into order %d\\n&#34;,\n address, buddy_addr, order &#43; 1);\n\n \/\/ Move to parent block\n address = address &lt; buddy_addr ? address : buddy_addr;\n order&#43;&#43;;\n break;\n }\n prev = current;\n current = current-&gt;next;\n }\n\n if (!found_buddy) {\n break;\n }\n }\n\n \/\/ Add to free list\n BuddyBlock *new_block = malloc(sizeof(BuddyBlock));\n new_block-&gt;address = address;\n new_block-&gt;order = order;\n new_block-&gt;is_free = true;\n new_block-&gt;next = free_lists[order];\n free_lists[order] = new_block;\n}\n\n\/*\n * Buddy System Advantages:\n * - Fast allocation\/deallocation (O(log n))\n * - Automatic coalescing\n * - Reduces external fragmentation\n *\n * Disadvantages:\n * - Internal fragmentation (power of 2 sizes)\n * - Complex implementation\n *\/<\/code><\/pre><h3 class=\"heading-element\" id=\"108-slab-allocator-linux-kernel\"><span>10.8 Slab Allocator (Linux Kernel)<\/span>\n <a href=\"#108-slab-allocator-linux-kernel\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Efficient allocator for kernel objects.<\/p>\n<pre><code>\/*\n * Slab Allocator: Cache objects for quick reuse\n * Used in Linux kernel for allocating kernel structures\n *\/\n\n#define SLAB_SIZE 4096 \/\/ One page\n#define OBJECTS_PER_SLAB 64\n\ntypedef struct SlabObject {\n bool is_free;\n void *data;\n struct SlabObject *next;\n} SlabObject;\n\ntypedef struct Slab {\n void *memory; \/\/ Actual slab memory\n SlabObject *free_list; \/\/ Free objects in this slab\n int free_count;\n struct Slab *next;\n} Slab;\n\ntypedef struct SlabCache {\n size_t object_size;\n Slab *partial_slabs; \/\/ Slabs with some free objects\n Slab *full_slabs; \/\/ Slabs with no free objects\n Slab *empty_slabs; \/\/ Slabs with all objects free\n void (*constructor)(void *); \/\/ Object constructor\n void (*destructor)(void *); \/\/ Object destructor\n} SlabCache;\n\nSlabCache* slab_cache_create(size_t object_size,\n void (*ctor)(void *),\n void (*dtor)(void *)) {\n SlabCache *cache = malloc(sizeof(SlabCache));\n cache-&gt;object_size = object_size;\n cache-&gt;partial_slabs = NULL;\n cache-&gt;full_slabs = NULL;\n cache-&gt;empty_slabs = NULL;\n cache-&gt;constructor = ctor;\n cache-&gt;destructor = dtor;\n\n printf(&#34;Slab: Created cache for %zu-byte objects\\n&#34;, object_size);\n\n return cache;\n}\n\nSlab* slab_create(size_t object_size) {\n Slab *slab = malloc(sizeof(Slab));\n slab-&gt;memory = malloc(SLAB_SIZE);\n slab-&gt;free_count = OBJECTS_PER_SLAB;\n slab-&gt;next = NULL;\n\n \/\/ Initialize free list\n slab-&gt;free_list = NULL;\n for (int i = 0; i &lt; OBJECTS_PER_SLAB; i&#43;&#43;) {\n SlabObject *obj = malloc(sizeof(SlabObject));\n obj-&gt;is_free = true;\n obj-&gt;data = (char *)slab-&gt;memory &#43; (i * object_size);\n obj-&gt;next = slab-&gt;free_list;\n slab-&gt;free_list = obj;\n }\n\n printf(&#34;Slab: Created new slab with %d objects\\n&#34;, OBJECTS_PER_SLAB);\n\n return slab;\n}\n\nvoid* slab_alloc(SlabCache *cache) {\n Slab *slab = cache-&gt;partial_slabs;\n\n \/\/ No partial slabs, try empty or create new\n if (slab == NULL) {\n slab = cache-&gt;empty_slabs;\n if (slab != NULL) {\n \/\/ Move from empty to partial\n cache-&gt;empty_slabs = slab-&gt;next;\n slab-&gt;next = cache-&gt;partial_slabs;\n cache-&gt;partial_slabs = slab;\n } else {\n \/\/ Create new slab\n slab = slab_create(cache-&gt;object_size);\n slab-&gt;next = cache-&gt;partial_slabs;\n cache-&gt;partial_slabs = slab;\n }\n }\n\n \/\/ Allocate object from slab\n SlabObject *obj = slab-&gt;free_list;\n slab-&gt;free_list = obj-&gt;next;\n obj-&gt;is_free = false;\n slab-&gt;free_count--;\n\n \/\/ Call constructor if present\n if (cache-&gt;constructor) {\n cache-&gt;constructor(obj-&gt;data);\n }\n\n \/\/ Move to full list if no more free objects\n if (slab-&gt;free_count == 0) {\n cache-&gt;partial_slabs = slab-&gt;next;\n slab-&gt;next = cache-&gt;full_slabs;\n cache-&gt;full_slabs = slab;\n }\n\n printf(&#34;Slab: Allocated object at %p (free in slab: %d)\\n&#34;,\n obj-&gt;data, slab-&gt;free_count);\n\n return obj-&gt;data;\n}\n\nvoid slab_free(SlabCache *cache, void *ptr) {\n \/\/ Find which slab contains this object\n \/\/ (In real implementation, use more efficient lookup)\n\n \/\/ Simplified: assume we know the slab\n \/\/ In reality, calculate from ptr and SLAB_SIZE\n\n printf(&#34;Slab: Freed object at %p\\n&#34;, ptr);\n\n \/\/ Call destructor if present\n if (cache-&gt;destructor) {\n cache-&gt;destructor(ptr);\n }\n\n \/\/ Return object to slab&#39;s free list\n \/\/ Move slab between lists as needed\n}\n\n\/*\n * Slab Allocator Benefits:\n * 1. No repeated initialization overhead\n * 2. Cache-friendly (objects close in memory)\n * 3. Minimal fragmentation\n * 4. Fast allocation (pre-allocated slabs)\n *\n * Used in Linux for:\n * - task_struct (process descriptors)\n * - file descriptors\n * - inode caching\n * - dentry caching\n *\/<\/code><\/pre><h3 class=\"heading-element\" id=\"109-memory-allocation-comparison\"><span>10.9 Memory Allocation Comparison<\/span>\n <a href=\"#109-memory-allocation-comparison\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Algorithm<\/th>\n <th>Speed<\/th>\n <th>Fragmentation<\/th>\n <th>Use Case<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>First Fit<\/strong><\/td>\n <td>Fast<\/td>\n <td>Medium external<\/td>\n <td>General purpose<\/td>\n <\/tr>\n <tr>\n <td><strong>Best Fit<\/strong><\/td>\n <td>Slow<\/td>\n <td>High external (tiny holes)<\/td>\n <td>Memory constrained<\/td>\n <\/tr>\n <tr>\n <td><strong>Worst Fit<\/strong><\/td>\n <td>Slow<\/td>\n <td>Low external<\/td>\n <td>Large requests<\/td>\n <\/tr>\n <tr>\n <td><strong>Next Fit<\/strong><\/td>\n <td>Fast<\/td>\n <td>Medium external<\/td>\n <td>Balanced distribution<\/td>\n <\/tr>\n <tr>\n <td><strong>Buddy System<\/strong><\/td>\n <td>Medium<\/td>\n <td>Medium internal<\/td>\n <td>Kernel memory<\/td>\n <\/tr>\n <tr>\n <td><strong>Slab Allocator<\/strong><\/td>\n <td>Very Fast<\/td>\n <td>Low<\/td>\n <td>Kernel objects<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"1010-practical-example-complete-memory-manager\"><span>10.10 Practical Example: Complete Memory Manager<\/span>\n <a href=\"#1010-practical-example-complete-memory-manager\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/\/ Complete example demonstrating multiple allocation strategies\nint main() {\n printf(&#34;=== Memory Management Demo ===\\n\\n&#34;);\n\n \/\/ Initialize memory: 1MB\n init_memory(0x100000);\n\n printf(&#34;--- First Fit Allocation ---\\n&#34;);\n first_fit(1, 0x10000); \/\/ P1: 64KB\n first_fit(2, 0x20000); \/\/ P2: 128KB\n first_fit(3, 0x15000); \/\/ P3: 84KB\n display_memory();\n\n printf(&#34;--- Deallocation ---\\n&#34;);\n \/\/ Free P2 (creates hole in middle)\n MemoryBlock *current = memory_list;\n while (current != NULL) {\n if (current-&gt;process_id == 2) {\n current-&gt;is_free = true;\n current-&gt;process_id = -1;\n printf(&#34;Freed Process 2\\n&#34;);\n break;\n }\n current = current-&gt;next;\n }\n display_memory();\n\n printf(&#34;--- Compaction ---\\n&#34;);\n compact_memory();\n display_memory();\n\n printf(&#34;--- Buddy System Demo ---\\n&#34;);\n init_buddy_system();\n void *p1 = buddy_alloc(100); \/\/ Will allocate 128 (2^7)\n void *p2 = buddy_alloc(200); \/\/ Will allocate 256 (2^8)\n void *p3 = buddy_alloc(50); \/\/ Will allocate 64 (2^6)\n buddy_free(p2, 8); \/\/ Free and coalesce\n buddy_free(p1, 7);\n buddy_free(p3, 6);\n\n return 0;\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"key-takeaways-1\"><span>Key Takeaways<\/span>\n <a href=\"#key-takeaways-1\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li><strong>Memory management handles allocation, deallocation, and protection<\/strong><\/li>\n<li><strong>First Fit is fastest, Best Fit minimizes waste, Worst Fit leaves larger holes<\/strong><\/li>\n<li><strong>External fragmentation<\/strong>: Free memory is scattered (solution: compaction)<\/li>\n<li><strong>Internal fragmentation<\/strong>: Allocated block larger than needed<\/li>\n<li><strong>Buddy System<\/strong>: Power-of-2 allocation with fast coalescing<\/li>\n<li><strong>Slab Allocator<\/strong>: Pre-allocates objects for fast reuse (Linux kernel)<\/li>\n<li><strong>Modern systems use hybrid approaches<\/strong> combining multiple strategies<\/li>\n<\/ol>\n<hr>\n<h2 id=\"virtual-memory\">11. Virtual Memory Systems<\/h2>\n<h3 class=\"heading-element\" id=\"111-virtual-memory-overview\"><span>11.1 Virtual Memory Overview<\/span>\n <a href=\"#111-virtual-memory-overview\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Virtual Memory<\/strong> separates logical memory from physical memory, allowing:<\/p>\n<ul>\n<li>Larger address space than physical RAM<\/li>\n<li>Efficient use of physical memory<\/li>\n<li>Process isolation and protection<\/li>\n<li>Memory sharing between processes<\/li>\n<li>Simplified programming model<\/li>\n<\/ul>\n<pre><code>\/*\n * Key Benefits:\n * 1. Programs larger than physical RAM can run\n * 2. Each process has own address space (0 to MAX)\n * 3. Protection: process cannot access others&#39; memory\n * 4. Shared libraries occupy physical memory once\n *\/\n\n\/\/ Example: 32-bit system\n#define VIRTUAL_ADDRESS_SPACE 0xFFFFFFFF \/\/ 4GB per process\n\/\/ Even if physical RAM is only 1GB!<\/code><\/pre><h3 class=\"heading-element\" id=\"112-demand-paging\"><span>11.2 Demand Paging<\/span>\n <a href=\"#112-demand-paging\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Demand paging<\/strong> loads pages into memory only when needed.<\/p>\n<pre><code>\/*\n * Demand Paging Process:\n * 1. CPU generates virtual address\n * 2. MMU looks up page table\n * 3. If page present: translate to physical address\n * 4. If page not present: PAGE FAULT\n * a. OS handles page fault\n * b. Load page from disk\n * c. Update page table\n * d. Restart instruction\n *\/\n\n#include &lt;stdint.h&gt;\n#include &lt;stdbool.h&gt;\n\n#define PAGE_SIZE 4096 \/\/ 4KB pages\n#define NUM_PAGES 1024 \/\/ 1024 pages = 4MB address space\n#define NUM_FRAMES 256 \/\/ 256 frames = 1MB physical RAM\n\ntypedef struct {\n bool present; \/\/ Is page in memory?\n uint32_t frame; \/\/ Physical frame number\n bool dirty; \/\/ Modified since loaded?\n bool accessed; \/\/ Recently accessed? (for LRU)\n uint32_t protection; \/\/ Read\/Write\/Execute bits\n} PageTableEntry;\n\ntypedef struct {\n PageTableEntry entries[NUM_PAGES];\n} PageTable;\n\n\/\/ Simulate page fault handler\nvoid handle_page_fault(PageTable *pt, uint32_t page_number) {\n printf(&#34;Page Fault! Loading page %d from disk...\\n&#34;, page_number);\n\n \/\/ Find free frame (or evict a page)\n uint32_t free_frame = find_free_frame();\n\n if (free_frame == UINT32_MAX) {\n \/\/ No free frames - need to evict\n free_frame = page_replacement_algorithm(pt);\n }\n\n \/\/ Load page from disk (simulated)\n load_page_from_disk(page_number, free_frame);\n\n \/\/ Update page table\n pt-&gt;entries[page_number].present = true;\n pt-&gt;entries[page_number].frame = free_frame;\n pt-&gt;entries[page_number].dirty = false;\n pt-&gt;entries[page_number].accessed = true;\n\n printf(&#34;Page %d loaded into frame %d\\n&#34;, page_number, free_frame);\n}\n\n\/\/ Virtual to physical address translation\nuint32_t translate_address(PageTable *pt, uint32_t virtual_address) {\n uint32_t page_number = virtual_address \/ PAGE_SIZE;\n uint32_t offset = virtual_address % PAGE_SIZE;\n\n if (page_number &gt;= NUM_PAGES) {\n printf(&#34;ERROR: Invalid virtual address 0x%X\\n&#34;, virtual_address);\n return UINT32_MAX;\n }\n\n PageTableEntry *pte = &amp;pt-&gt;entries[page_number];\n\n if (!pte-&gt;present) {\n \/\/ Page fault!\n handle_page_fault(pt, page_number);\n }\n\n \/\/ Mark as accessed (for page replacement)\n pte-&gt;accessed = true;\n\n \/\/ Calculate physical address\n uint32_t physical_address = (pte-&gt;frame * PAGE_SIZE) &#43; offset;\n\n printf(&#34;Virtual 0x%X (page %d, offset %d) \u2192 Physical 0x%X (frame %d)\\n&#34;,\n virtual_address, page_number, offset, physical_address, pte-&gt;frame);\n\n return physical_address;\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"113-page-replacement-algorithms\"><span>11.3 Page Replacement Algorithms<\/span>\n <a href=\"#113-page-replacement-algorithms\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>When all frames are occupied, must choose a victim page to evict.<\/p>\n<h4 class=\"heading-element\" id=\"1131-fifo-first-in-first-out\"><span>11.3.1 FIFO (First-In-First-Out)<\/span>\n <a href=\"#1131-fifo-first-in-first-out\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * FIFO: Replace oldest page in memory\n * Simple but suffers from Belady&#39;s Anomaly\n *\/\n\n#include &lt;stdint.h&gt;\n\n#define MAX_FRAMES 3\n\ntypedef struct {\n uint32_t frames[MAX_FRAMES];\n int front; \/\/ Oldest page index\n int count;\n} FIFOQueue;\n\nvoid init_fifo(FIFOQueue *q) {\n q-&gt;front = 0;\n q-&gt;count = 0;\n for (int i = 0; i &lt; MAX_FRAMES; i&#43;&#43;) {\n q-&gt;frames[i] = UINT32_MAX;\n }\n}\n\nbool is_page_in_memory(FIFOQueue *q, uint32_t page) {\n for (int i = 0; i &lt; MAX_FRAMES; i&#43;&#43;) {\n if (q-&gt;frames[i] == page) {\n return true;\n }\n }\n return false;\n}\n\nint fifo_replace(FIFOQueue *q, uint32_t page) {\n if (is_page_in_memory(q, page)) {\n printf(&#34;Page %d already in memory\\n&#34;, page);\n return 0; \/\/ No fault\n }\n\n if (q-&gt;count &lt; MAX_FRAMES) {\n \/\/ Still have free frames\n q-&gt;frames[q-&gt;count] = page;\n q-&gt;count&#43;&#43;;\n printf(&#34;FIFO: Loaded page %d into frame %d\\n&#34;, page, q-&gt;count - 1);\n } else {\n \/\/ Replace oldest page\n uint32_t victim = q-&gt;frames[q-&gt;front];\n q-&gt;frames[q-&gt;front] = page;\n q-&gt;front = (q-&gt;front &#43; 1) % MAX_FRAMES;\n printf(&#34;FIFO: Replaced page %d with page %d\\n&#34;, victim, page);\n }\n\n return 1; \/\/ Page fault occurred\n}\n\n\/\/ Example: Belady&#39;s Anomaly\nvoid demonstrate_belady_anomaly() {\n uint32_t reference_string[] = {1, 2, 3, 4, 1, 2, 5, 1, 2, 3, 4, 5};\n int length = sizeof(reference_string) \/ sizeof(reference_string[0]);\n\n FIFOQueue queue;\n init_fifo(&amp;queue);\n\n int page_faults = 0;\n printf(&#34;=== FIFO Page Replacement ===\\n&#34;);\n for (int i = 0; i &lt; length; i&#43;&#43;) {\n printf(&#34;Accessing page %d: &#34;, reference_string[i]);\n page_faults &#43;= fifo_replace(&amp;queue, reference_string[i]);\n }\n printf(&#34;Total page faults: %d\\n\\n&#34;, page_faults);\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"1132-lru-least-recently-used\"><span>11.3.2 LRU (Least Recently Used)<\/span>\n <a href=\"#1132-lru-least-recently-used\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * LRU: Replace page that hasn&#39;t been used for longest time\n * Optimal but expensive to implement exactly\n *\/\n\ntypedef struct {\n uint32_t page;\n uint64_t last_access_time;\n} LRUFrame;\n\ntypedef struct {\n LRUFrame frames[MAX_FRAMES];\n int count;\n uint64_t current_time;\n} LRUCache;\n\nvoid init_lru(LRUCache *cache) {\n cache-&gt;count = 0;\n cache-&gt;current_time = 0;\n for (int i = 0; i &lt; MAX_FRAMES; i&#43;&#43;) {\n cache-&gt;frames[i].page = UINT32_MAX;\n cache-&gt;frames[i].last_access_time = 0;\n }\n}\n\nint find_lru_victim(LRUCache *cache) {\n int victim_idx = 0;\n uint64_t oldest_time = cache-&gt;frames[0].last_access_time;\n\n for (int i = 1; i &lt; MAX_FRAMES; i&#43;&#43;) {\n if (cache-&gt;frames[i].last_access_time &lt; oldest_time) {\n oldest_time = cache-&gt;frames[i].last_access_time;\n victim_idx = i;\n }\n }\n\n return victim_idx;\n}\n\nint lru_access(LRUCache *cache, uint32_t page) {\n cache-&gt;current_time&#43;&#43;;\n\n \/\/ Check if page already in memory\n for (int i = 0; i &lt; cache-&gt;count; i&#43;&#43;) {\n if (cache-&gt;frames[i].page == page) {\n cache-&gt;frames[i].last_access_time = cache-&gt;current_time;\n printf(&#34;LRU: Page %d hit (updated access time)\\n&#34;, page);\n return 0; \/\/ No fault\n }\n }\n\n \/\/ Page fault\n if (cache-&gt;count &lt; MAX_FRAMES) {\n \/\/ Free frame available\n cache-&gt;frames[cache-&gt;count].page = page;\n cache-&gt;frames[cache-&gt;count].last_access_time = cache-&gt;current_time;\n cache-&gt;count&#43;&#43;;\n printf(&#34;LRU: Loaded page %d into frame %d\\n&#34;, page, cache-&gt;count - 1);\n } else {\n \/\/ Replace LRU page\n int victim_idx = find_lru_victim(cache);\n uint32_t victim_page = cache-&gt;frames[victim_idx].page;\n\n cache-&gt;frames[victim_idx].page = page;\n cache-&gt;frames[victim_idx].last_access_time = cache-&gt;current_time;\n\n printf(&#34;LRU: Replaced page %d (LRU) with page %d\\n&#34;, victim_page, page);\n }\n\n return 1; \/\/ Page fault\n}\n\n\/\/ Approximate LRU using Clock algorithm (Second Chance)\ntypedef struct {\n uint32_t page;\n bool reference_bit;\n} ClockFrame;\n\ntypedef struct {\n ClockFrame frames[MAX_FRAMES];\n int hand; \/\/ Clock hand position\n int count;\n} ClockAlgorithm;\n\nvoid init_clock(ClockAlgorithm *clock) {\n clock-&gt;hand = 0;\n clock-&gt;count = 0;\n for (int i = 0; i &lt; MAX_FRAMES; i&#43;&#43;) {\n clock-&gt;frames[i].page = UINT32_MAX;\n clock-&gt;frames[i].reference_bit = false;\n }\n}\n\nint clock_replace(ClockAlgorithm *clock, uint32_t page) {\n \/\/ Check if already in memory\n for (int i = 0; i &lt; clock-&gt;count; i&#43;&#43;) {\n if (clock-&gt;frames[i].page == page) {\n clock-&gt;frames[i].reference_bit = true; \/\/ Set reference bit\n return 0; \/\/ No fault\n }\n }\n\n \/\/ Page fault\n if (clock-&gt;count &lt; MAX_FRAMES) {\n \/\/ Free frame available\n clock-&gt;frames[clock-&gt;count].page = page;\n clock-&gt;frames[clock-&gt;count].reference_bit = true;\n clock-&gt;count&#43;&#43;;\n printf(&#34;Clock: Loaded page %d into frame %d\\n&#34;, page, clock-&gt;count - 1);\n } else {\n \/\/ Search for victim using clock algorithm\n while (true) {\n if (!clock-&gt;frames[clock-&gt;hand].reference_bit) {\n \/\/ Found victim (reference bit = 0)\n uint32_t victim = clock-&gt;frames[clock-&gt;hand].page;\n clock-&gt;frames[clock-&gt;hand].page = page;\n clock-&gt;frames[clock-&gt;hand].reference_bit = true;\n\n printf(&#34;Clock: Replaced page %d with page %d\\n&#34;, victim, page);\n\n clock-&gt;hand = (clock-&gt;hand &#43; 1) % MAX_FRAMES;\n break;\n } else {\n \/\/ Give second chance\n clock-&gt;frames[clock-&gt;hand].reference_bit = false;\n clock-&gt;hand = (clock-&gt;hand &#43; 1) % MAX_FRAMES;\n }\n }\n }\n\n return 1; \/\/ Page fault\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"1133-optimal-min\"><span>11.3.3 Optimal (MIN)<\/span>\n <a href=\"#1133-optimal-min\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * Optimal: Replace page that will not be used for longest time\n * Theoretical best, not implementable (requires future knowledge)\n * Used as benchmark to compare other algorithms\n *\/\n\nint optimal_replace(uint32_t current_pages[], int num_frames,\n uint32_t reference_string[], int length,\n int current_position) {\n \/\/ Find page not used for longest time in future\n int victim_idx = -1;\n int farthest = current_position;\n\n for (int i = 0; i &lt; num_frames; i&#43;&#43;) {\n int j;\n for (j = current_position; j &lt; length; j&#43;&#43;) {\n if (current_pages[i] == reference_string[j]) {\n if (j &gt; farthest) {\n farthest = j;\n victim_idx = i;\n }\n break;\n }\n }\n\n \/\/ Page never used again - perfect victim!\n if (j == length) {\n return i;\n }\n }\n\n return victim_idx != -1 ? victim_idx : 0;\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"114-translation-lookaside-buffer-tlb\"><span>11.4 Translation Lookaside Buffer (TLB)<\/span>\n <a href=\"#114-translation-lookaside-buffer-tlb\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>TLB<\/strong> is a hardware cache for page table entries.<\/p>\n<pre><code>\/*\n * TLB (Translation Lookaside Buffer):\n * Fast cache for page table entries\n * Typically 64-128 entries, fully associative\n * 95%&#43; hit rate in practice\n *\/\n\n#define TLB_SIZE 16\n\ntypedef struct {\n uint32_t page_number;\n uint32_t frame_number;\n bool valid;\n} TLBEntry;\n\ntypedef struct {\n TLBEntry entries[TLB_SIZE];\n int next_replacement; \/\/ For FIFO replacement\n uint64_t hits;\n uint64_t misses;\n} TLB;\n\nvoid init_tlb(TLB *tlb) {\n for (int i = 0; i &lt; TLB_SIZE; i&#43;&#43;) {\n tlb-&gt;entries[i].valid = false;\n }\n tlb-&gt;next_replacement = 0;\n tlb-&gt;hits = 0;\n tlb-&gt;misses = 0;\n}\n\nbool tlb_lookup(TLB *tlb, uint32_t page_number, uint32_t *frame_number) {\n for (int i = 0; i &lt; TLB_SIZE; i&#43;&#43;) {\n if (tlb-&gt;entries[i].valid &amp;&amp;\n tlb-&gt;entries[i].page_number == page_number) {\n *frame_number = tlb-&gt;entries[i].frame_number;\n tlb-&gt;hits&#43;&#43;;\n printf(&#34;TLB HIT: Page %d \u2192 Frame %d\\n&#34;, page_number, *frame_number);\n return true;\n }\n }\n\n tlb-&gt;misses&#43;&#43;;\n printf(&#34;TLB MISS: Page %d\\n&#34;, page_number);\n return false;\n}\n\nvoid tlb_insert(TLB *tlb, uint32_t page_number, uint32_t frame_number) {\n \/\/ FIFO replacement in TLB\n tlb-&gt;entries[tlb-&gt;next_replacement].page_number = page_number;\n tlb-&gt;entries[tlb-&gt;next_replacement].frame_number = frame_number;\n tlb-&gt;entries[tlb-&gt;next_replacement].valid = true;\n\n tlb-&gt;next_replacement = (tlb-&gt;next_replacement &#43; 1) % TLB_SIZE;\n\n printf(&#34;TLB INSERT: Page %d \u2192 Frame %d\\n&#34;, page_number, frame_number);\n}\n\nvoid tlb_flush(TLB *tlb) {\n \/\/ Called on context switch\n for (int i = 0; i &lt; TLB_SIZE; i&#43;&#43;) {\n tlb-&gt;entries[i].valid = false;\n }\n printf(&#34;TLB FLUSHED (context switch)\\n&#34;);\n}\n\n\/\/ Complete address translation with TLB\nuint32_t translate_with_tlb(TLB *tlb, PageTable *pt, uint32_t virtual_addr) {\n uint32_t page_number = virtual_addr \/ PAGE_SIZE;\n uint32_t offset = virtual_addr % PAGE_SIZE;\n uint32_t frame_number;\n\n \/\/ Step 1: Check TLB\n if (tlb_lookup(tlb, page_number, &amp;frame_number)) {\n \/\/ TLB hit - fast path!\n return (frame_number * PAGE_SIZE) &#43; offset;\n }\n\n \/\/ Step 2: TLB miss - check page table\n if (!pt-&gt;entries[page_number].present) {\n \/\/ Page fault\n handle_page_fault(pt, page_number);\n }\n\n frame_number = pt-&gt;entries[page_number].frame;\n\n \/\/ Step 3: Update TLB\n tlb_insert(tlb, page_number, frame_number);\n\n return (frame_number * PAGE_SIZE) &#43; offset;\n}\n\nvoid print_tlb_stats(TLB *tlb) {\n uint64_t total = tlb-&gt;hits &#43; tlb-&gt;misses;\n double hit_rate = (double)tlb-&gt;hits \/ total * 100.0;\n\n printf(&#34;\\n=== TLB Statistics ===\\n&#34;);\n printf(&#34;Hits: %lu\\n&#34;, tlb-&gt;hits);\n printf(&#34;Misses: %lu\\n&#34;, tlb-&gt;misses);\n printf(&#34;Hit Rate: %.2f%%\\n&#34;, hit_rate);\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"115-multi-level-page-tables\"><span>11.5 Multi-Level Page Tables<\/span>\n <a href=\"#115-multi-level-page-tables\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Reduce memory overhead of large page tables.<\/p>\n<pre><code>\/*\n * Problem: Single-level page table for 32-bit address space\n * 4KB pages: 2^32 \/ 2^12 = 2^20 (1 million) entries\n * If each entry = 4 bytes: 4MB per process!\n *\n * Solution: Multi-level (hierarchical) page tables\n *\/\n\n\/\/ Two-level page table for 32-bit address space\n#define PAGE_DIR_SIZE 1024 \/\/ 10 bits\n#define PAGE_TABLE_SIZE 1024 \/\/ 10 bits\n\/\/ Offset: 12 bits\n\/\/ Total: 10 &#43; 10 &#43; 12 = 32 bits\n\ntypedef struct {\n PageTableEntry entries[PAGE_TABLE_SIZE];\n} SecondLevelPageTable;\n\ntypedef struct {\n SecondLevelPageTable *tables[PAGE_DIR_SIZE];\n bool present[PAGE_DIR_SIZE];\n} PageDirectory;\n\nvoid init_page_directory(PageDirectory *pd) {\n for (int i = 0; i &lt; PAGE_DIR_SIZE; i&#43;&#43;) {\n pd-&gt;tables[i] = NULL;\n pd-&gt;present[i] = false;\n }\n}\n\nuint32_t translate_two_level(PageDirectory *pd, uint32_t virtual_addr) {\n \/\/ Extract indices\n uint32_t dir_index = (virtual_addr &gt;&gt; 22) &amp; 0x3FF; \/\/ Bits 31-22\n uint32_t table_index = (virtual_addr &gt;&gt; 12) &amp; 0x3FF; \/\/ Bits 21-12\n uint32_t offset = virtual_addr &amp; 0xFFF; \/\/ Bits 11-0\n\n printf(&#34;Virtual 0x%08X: dir=%d, table=%d, offset=%d\\n&#34;,\n virtual_addr, dir_index, table_index, offset);\n\n \/\/ Check if page table exists\n if (!pd-&gt;present[dir_index]) {\n printf(&#34;Page directory entry not present - allocate page table\\n&#34;);\n pd-&gt;tables[dir_index] = malloc(sizeof(SecondLevelPageTable));\n pd-&gt;present[dir_index] = true;\n }\n\n SecondLevelPageTable *page_table = pd-&gt;tables[dir_index];\n PageTableEntry *pte = &amp;page_table-&gt;entries[table_index];\n\n if (!pte-&gt;present) {\n \/\/ Page fault\n printf(&#34;Page fault at dir=%d, table=%d\\n&#34;, dir_index, table_index);\n return UINT32_MAX;\n }\n\n uint32_t physical_addr = (pte-&gt;frame * PAGE_SIZE) &#43; offset;\n return physical_addr;\n}\n\n\/*\n * Advantages of Multi-Level Page Tables:\n * 1. Only allocate page tables for used address space\n * 2. Typical process uses small fraction of address space\n * 3. Example: Process using 8MB of 4GB space\n * - Single level: 4MB page table\n * - Two level: ~8KB page tables\n * 4. Can extend to 3, 4, or more levels (x86-64 uses 4 levels)\n *\/<\/code><\/pre><h3 class=\"heading-element\" id=\"116-page-table-entry-structure\"><span>11.6 Page Table Entry Structure<\/span>\n <a href=\"#116-page-table-entry-structure\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * Typical Page Table Entry (x86)\n *\/\n\ntypedef struct {\n unsigned int present : 1; \/\/ 1 = page in memory\n unsigned int rw : 1; \/\/ 1 = read\/write, 0 = read-only\n unsigned int user : 1; \/\/ 1 = user mode can access\n unsigned int pwt : 1; \/\/ Page write-through\n unsigned int pcd : 1; \/\/ Page cache disabled\n unsigned int accessed : 1; \/\/ Has page been accessed?\n unsigned int dirty : 1; \/\/ Has page been written to?\n unsigned int pat : 1; \/\/ Page attribute table\n unsigned int global : 1; \/\/ Don&#39;t flush on context switch\n unsigned int available : 3; \/\/ Available for OS use\n unsigned int frame : 20; \/\/ Physical frame number (4KB aligned)\n} x86_PTE;\n\nvoid print_pte(x86_PTE *pte) {\n printf(&#34;PTE: Frame=%d, P=%d, R\/W=%d, U\/S=%d, A=%d, D=%d\\n&#34;,\n pte-&gt;frame, pte-&gt;present, pte-&gt;rw, pte-&gt;user,\n pte-&gt;accessed, pte-&gt;dirty);\n}\n\n\/\/ Check page permissions\nbool check_access(x86_PTE *pte, bool is_write, bool is_user_mode) {\n if (!pte-&gt;present) {\n printf(&#34;Page fault: Page not present\\n&#34;);\n return false;\n }\n\n if (is_write &amp;&amp; !pte-&gt;rw) {\n printf(&#34;Protection fault: Write to read-only page\\n&#34;);\n return false;\n }\n\n if (is_user_mode &amp;&amp; !pte-&gt;user) {\n printf(&#34;Protection fault: User mode accessing supervisor page\\n&#34;);\n return false;\n }\n\n return true;\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"117-inverted-page-tables\"><span>11.7 Inverted Page Tables<\/span>\n <a href=\"#117-inverted-page-tables\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Alternative approach for large address spaces.<\/p>\n<pre><code>\/*\n * Inverted Page Table:\n * One entry per physical frame (not per virtual page)\n * Saves space for 64-bit systems\n *\/\n\n#define NUM_PHYSICAL_FRAMES 256\n\ntypedef struct {\n uint32_t pid; \/\/ Process ID\n uint32_t virtual_page; \/\/ Virtual page number\n bool valid;\n} InvertedPageTableEntry;\n\ntypedef struct {\n InvertedPageTableEntry entries[NUM_PHYSICAL_FRAMES];\n} InvertedPageTable;\n\nuint32_t inverted_translate(InvertedPageTable *ipt, uint32_t pid,\n uint32_t virtual_addr) {\n uint32_t virtual_page = virtual_addr \/ PAGE_SIZE;\n uint32_t offset = virtual_addr % PAGE_SIZE;\n\n \/\/ Search for matching entry (linear search or use hash table)\n for (int frame = 0; frame &lt; NUM_PHYSICAL_FRAMES; frame&#43;&#43;) {\n if (ipt-&gt;entries[frame].valid &amp;&amp;\n ipt-&gt;entries[frame].pid == pid &amp;&amp;\n ipt-&gt;entries[frame].virtual_page == virtual_page) {\n\n uint32_t physical_addr = (frame * PAGE_SIZE) &#43; offset;\n printf(&#34;Inverted PT: Found frame %d for PID=%d, VPage=%d\\n&#34;,\n frame, pid, virtual_page);\n return physical_addr;\n }\n }\n\n printf(&#34;Page fault: No mapping found\\n&#34;);\n return UINT32_MAX;\n}\n\n\/*\n * Pros:\n * - Fixed size regardless of address space size\n * - One table for entire system\n *\n * Cons:\n * - Slower lookup (need to search)\n * - Difficult to implement shared memory\n *\n * Used in: IBM PowerPC, HP PA-RISC\n *\/<\/code><\/pre><h3 class=\"heading-element\" id=\"118-memory-mapped-files\"><span>11.8 Memory-Mapped Files<\/span>\n <a href=\"#118-memory-mapped-files\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Map files directly into virtual address space.<\/p>\n<pre><code>\/*\n * Memory-Mapped Files:\n * Access file contents as if they were in memory\n * OS handles loading pages from file on demand\n *\/\n\n#include &lt;sys\/mman.h&gt;\n#include &lt;sys\/stat.h&gt;\n#include &lt;fcntl.h&gt;\n#include &lt;unistd.h&gt;\n\nvoid example_mmap() {\n \/\/ Open file\n int fd = open(&#34;example.dat&#34;, O_RDWR);\n if (fd == -1) {\n perror(&#34;open&#34;);\n return;\n }\n\n \/\/ Get file size\n struct stat sb;\n if (fstat(fd, &amp;sb) == -1) {\n perror(&#34;fstat&#34;);\n close(fd);\n return;\n }\n\n \/\/ Map file into memory\n char *mapped = mmap(NULL, \/\/ Let OS choose address\n sb.st_size, \/\/ Map entire file\n PROT_READ | PROT_WRITE, \/\/ Read\/write access\n MAP_SHARED, \/\/ Changes written to file\n fd,\n 0); \/\/ Offset 0\n\n if (mapped == MAP_FAILED) {\n perror(&#34;mmap&#34;);\n close(fd);\n return;\n }\n\n printf(&#34;File mapped at %p, size %ld bytes\\n&#34;, mapped, sb.st_size);\n\n \/\/ Access file like normal memory!\n mapped[0] = &#39;X&#39;; \/\/ Modifies file\n printf(&#34;First byte: %c\\n&#34;, mapped[0]);\n\n \/\/ Unmap when done\n munmap(mapped, sb.st_size);\n close(fd);\n}\n\n\/*\n * Benefits of mmap:\n * 1. Simplified file I\/O (no read\/write calls)\n * 2. Automatic caching by OS\n * 3. Shared memory between processes\n * 4. Efficient for large files (load on demand)\n *\/<\/code><\/pre><h3 class=\"heading-element\" id=\"119-copy-on-write-cow\"><span>11.9 Copy-on-Write (COW)<\/span>\n <a href=\"#119-copy-on-write-cow\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Optimize fork() by deferring copying.<\/p>\n<pre><code>\/*\n * Copy-on-Write:\n * When fork() called, don&#39;t copy pages immediately\n * Parent and child share pages (marked read-only)\n * On write attempt, copy page at that moment\n *\/\n\nvoid demonstrate_cow() {\n int *data = malloc(1000000 * sizeof(int));\n \/\/ Fill with data...\n\n pid_t pid = fork();\n\n if (pid == 0) {\n \/\/ Child process\n printf(&#34;Child: data pointer = %p\\n&#34;, data);\n\n \/\/ Reading is fine (pages shared)\n printf(&#34;Child reading: data[0] = %d\\n&#34;, data[0]);\n\n \/\/ Writing triggers copy\n printf(&#34;Child writing to data[0]...\\n&#34;);\n data[0] = 999; \/\/ COW: Page copied now!\n\n printf(&#34;Child: data[0] = %d\\n&#34;, data[0]);\n } else {\n \/\/ Parent process\n sleep(1);\n printf(&#34;Parent: data[0] = %d\\n&#34;, data[0]); \/\/ Still original value\n }\n}\n\n\/*\n * COW Implementation:\n * 1. Mark all pages as read-only in child&#39;s page table\n * 2. Set COW bit in page table entries\n * 3. On write, page fault handler:\n * a. Check if COW page\n * b. Allocate new frame\n * c. Copy data\n * d. Update page table (mark writable)\n * e. Resume execution\n *\/<\/code><\/pre><h3 class=\"heading-element\" id=\"1110-thrashing\"><span>11.10 Thrashing<\/span>\n <a href=\"#1110-thrashing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>When system spends more time paging than executing.<\/p>\n<pre><code>\/*\n * Thrashing occurs when:\n * - Too many processes competing for limited frames\n * - Each process doesn&#39;t have enough frames\n * - Constant page faults\n * - CPU utilization drops dramatically\n *\/\n\n\/\/ Working Set Model\ntypedef struct {\n uint32_t *pages; \/\/ Pages in working set\n int size; \/\/ Current size\n int max_size; \/\/ Window size\n uint64_t time_window; \/\/ Time window (e.g., last 1000 refs)\n} WorkingSet;\n\n\/\/ Calculate working set size\nint calculate_working_set(uint32_t reference_string[], int length, int window) {\n bool in_working_set[NUM_PAGES] = {false};\n int working_set_size = 0;\n\n \/\/ Consider last &#39;window&#39; references\n int start = (length - window &gt; 0) ? length - window : 0;\n\n for (int i = start; i &lt; length; i&#43;&#43;) {\n uint32_t page = reference_string[i];\n if (!in_working_set[page]) {\n in_working_set[page] = true;\n working_set_size&#43;&#43;;\n }\n }\n\n return working_set_size;\n}\n\n\/*\n * Solutions to Thrashing:\n * 1. Working Set Model: Allocate frames based on working set size\n * 2. Page Fault Frequency: Adjust allocation based on fault rate\n * 3. Suspend processes if insufficient memory\n * 4. Increase physical memory\n *\/<\/code><\/pre><h3 class=\"heading-element\" id=\"1111-page-replacement-algorithm-comparison\"><span>11.11 Page Replacement Algorithm Comparison<\/span>\n <a href=\"#1111-page-replacement-algorithm-comparison\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Algorithm<\/th>\n <th>Advantages<\/th>\n <th>Disadvantages<\/th>\n <th>Use Case<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>FIFO<\/strong><\/td>\n <td>Simple, low overhead<\/td>\n <td>Belady&rsquo;s anomaly<\/td>\n <td>Rarely used alone<\/td>\n <\/tr>\n <tr>\n <td><strong>Optimal<\/strong><\/td>\n <td>Best possible (fewest faults)<\/td>\n <td>Impossible to implement<\/td>\n <td>Benchmark only<\/td>\n <\/tr>\n <tr>\n <td><strong>LRU<\/strong><\/td>\n <td>Good performance<\/td>\n <td>Expensive to implement exactly<\/td>\n <td>Common approximation target<\/td>\n <\/tr>\n <tr>\n <td><strong>Clock (Second Chance)<\/strong><\/td>\n <td>Good approximation of LRU<\/td>\n <td>Slightly more complex than FIFO<\/td>\n <td>Linux, Windows<\/td>\n <\/tr>\n <tr>\n <td><strong>LFU<\/strong><\/td>\n <td>Protects frequently used pages<\/td>\n <td>Difficult to implement<\/td>\n <td>Specialized cases<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"key-takeaways-2\"><span>Key Takeaways<\/span>\n <a href=\"#key-takeaways-2\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Virtual Memory Fundamentals - Master These Concepts<\/strong>:<\/p>\n<ol>\n<li><strong>Virtual memory separates logical from physical address space<\/strong> \u2192 Programs can use more memory than physically available<\/li>\n<li><strong>Demand paging loads pages only when accessed (lazy loading)<\/strong> \u2192 Saves memory, improves startup time<\/li>\n<li><strong>TLB caches page table entries for fast translation (95%+ hit rate)<\/strong> \u2192 Critical for performance!<\/li>\n<li><strong>Multi-level page tables save space for sparse address spaces<\/strong> \u2192 4-level on x86-64<\/li>\n<li><strong>Page replacement algorithms<\/strong>: FIFO (simple), LRU (optimal), Clock (practical) \u2192 Choose based on workload<\/li>\n<li><strong>Copy-on-Write optimizes fork() by deferring page copying<\/strong> \u2192 Makes process creation fast<\/li>\n<li><strong>Thrashing occurs when system spends more time paging than executing<\/strong> \u2192 Monitor page fault rate!<\/li>\n<li><strong>Memory-mapped files provide efficient file I\/O<\/strong> \u2192 Zero-copy I\/O<\/li>\n<\/ol><\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"file-systems\">12. File Systems<\/h2>\n<h3 class=\"heading-element\" id=\"121-file-system-basics\"><span>12.1 File System Basics<\/span>\n <a href=\"#121-file-system-basics\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>A <strong>file system<\/strong> organizes and stores data on storage devices, providing:<\/p>\n<ul>\n<li>Persistent storage<\/li>\n<li>Hierarchical namespace<\/li>\n<li>Access control and permissions<\/li>\n<li>Efficient space management<\/li>\n<\/ul>\n<pre><code>\/*\n * File: Named collection of related information\n * Directory: Container for files and other directories\n * Path: Location of file in hierarchy\n *\/\n\n\/\/ File attributes (metadata)\ntypedef struct {\n char name[256];\n uint64_t size; \/\/ Size in bytes\n uint32_t inode_number; \/\/ Unique identifier\n uint16_t mode; \/\/ Permissions\n uint32_t uid; \/\/ Owner user ID\n uint32_t gid; \/\/ Owner group ID\n time_t created; \/\/ Creation time\n time_t modified; \/\/ Last modification\n time_t accessed; \/\/ Last access\n uint32_t link_count; \/\/ Number of hard links\n} FileMetadata;<\/code><\/pre><h3 class=\"heading-element\" id=\"122-file-operations\"><span>12.2 File Operations<\/span>\n <a href=\"#122-file-operations\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * Core File Operations (System Calls)\n *\/\n\n#include &lt;fcntl.h&gt;\n#include &lt;unistd.h&gt;\n#include &lt;sys\/types.h&gt;\n#include &lt;sys\/stat.h&gt;\n\n\/\/ Open file: Get file descriptor\nint fd = open(&#34;file.txt&#34;, O_RDWR | O_CREAT, 0644);\nif (fd == -1) {\n perror(&#34;open&#34;);\n return -1;\n}\n\n\/\/ Read from file\nchar buffer[1024];\nssize_t bytes_read = read(fd, buffer, sizeof(buffer));\nprintf(&#34;Read %zd bytes\\n&#34;, bytes_read);\n\n\/\/ Write to file\nconst char *data = &#34;Hello, World!&#34;;\nssize_t bytes_written = write(fd, data, strlen(data));\n\n\/\/ Move file pointer (seek)\noff_t new_pos = lseek(fd, 0, SEEK_SET); \/\/ Seek to beginning\n\/\/ SEEK_CUR: relative to current position\n\/\/ SEEK_END: relative to end of file\n\n\/\/ Get file information\nstruct stat file_info;\nif (fstat(fd, &amp;file_info) == 0) {\n printf(&#34;File size: %ld bytes\\n&#34;, file_info.st_size);\n printf(&#34;Inode: %lu\\n&#34;, file_info.st_ino);\n printf(&#34;Permissions: %o\\n&#34;, file_info.st_mode &amp; 0777);\n}\n\n\/\/ Close file\nclose(fd);\n\n\/*\n * File Descriptor Table:\n * Process maintains table mapping fd to open file structures\n * fd 0: stdin, fd 1: stdout, fd 2: stderr\n * New files get lowest available fd\n *\/<\/code><\/pre><h3 class=\"heading-element\" id=\"123-directory-structure\"><span>12.3 Directory Structure<\/span>\n <a href=\"#123-directory-structure\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"unixlinux-file-system-hierarchy\"><span>Unix\/Linux File System Hierarchy<\/span>\n <a href=\"#unixlinux-file-system-hierarchy\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * Unix\/Linux Filesystem Hierarchy Standard (FHS)\n *\n * \/ Root directory\n * \u251c\u2500\u2500 bin\/ Essential user binaries (ls, cp, mv)\n * \u251c\u2500\u2500 boot\/ Boot loader files (kernel, initrd)\n * \u251c\u2500\u2500 dev\/ Device files (disks, terminals)\n * \u251c\u2500\u2500 etc\/ System configuration files\n * \u251c\u2500\u2500 home\/ User home directories\n * \u251c\u2500\u2500 lib\/ Shared libraries\n * \u251c\u2500\u2500 mnt\/ Temporary mount points\n * \u251c\u2500\u2500 opt\/ Optional application packages\n * \u251c\u2500\u2500 proc\/ Process information (virtual filesystem)\n * \u251c\u2500\u2500 root\/ Root user&#39;s home directory\n * \u251c\u2500\u2500 sbin\/ System binaries (fsck, init)\n * \u251c\u2500\u2500 sys\/ Kernel and system information (virtual)\n * \u251c\u2500\u2500 tmp\/ Temporary files\n * \u251c\u2500\u2500 usr\/ User programs and data\n * \u2502 \u251c\u2500\u2500 bin\/ User binaries\n * \u2502 \u251c\u2500\u2500 lib\/ Libraries\n * \u2502 \u2514\u2500\u2500 share\/ Shared data\n * \u2514\u2500\u2500 var\/ Variable data (logs, mail, temp files)\n * \u251c\u2500\u2500 log\/ System logs\n * \u2514\u2500\u2500 tmp\/ Temporary files\n *\/\n\n\/\/ Directory operations\n#include &lt;dirent.h&gt;\n\nvoid list_directory(const char *path) {\n DIR *dir = opendir(path);\n if (!dir) {\n perror(&#34;opendir&#34;);\n return;\n }\n\n struct dirent *entry;\n while ((entry = readdir(dir)) != NULL) {\n printf(&#34;%s&#34;, entry-&gt;d_name);\n\n \/\/ Determine file type\n switch (entry-&gt;d_type) {\n case DT_REG: printf(&#34; (file)\\n&#34;); break;\n case DT_DIR: printf(&#34; (directory)\\n&#34;); break;\n case DT_LNK: printf(&#34; (symlink)\\n&#34;); break;\n case DT_CHR: printf(&#34; (char device)\\n&#34;); break;\n case DT_BLK: printf(&#34; (block device)\\n&#34;); break;\n case DT_FIFO: printf(&#34; (FIFO)\\n&#34;); break;\n case DT_SOCK: printf(&#34; (socket)\\n&#34;); break;\n default: printf(&#34; (unknown)\\n&#34;); break;\n }\n }\n\n closedir(dir);\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"124-file-allocation-methods\"><span>12.4 File Allocation Methods<\/span>\n <a href=\"#124-file-allocation-methods\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"1241-contiguous-allocation\"><span>12.4.1 Contiguous Allocation<\/span>\n <a href=\"#1241-contiguous-allocation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * Contiguous Allocation:\n * File occupies consecutive blocks on disk\n *\n * Pros:\n * - Simple\n * - Fast sequential access\n * - Easy random access\n *\n * Cons:\n * - External fragmentation\n * - Difficult to grow files\n * - Need to know file size in advance\n *\/\n\ntypedef struct {\n char filename[256];\n uint32_t start_block; \/\/ Starting block number\n uint32_t length; \/\/ Number of blocks\n} ContiguousFile;\n\nvoid read_contiguous_file(ContiguousFile *file, uint32_t offset) {\n uint32_t block_size = 4096;\n uint32_t block_number = file-&gt;start_block &#43; (offset \/ block_size);\n uint32_t block_offset = offset % block_size;\n\n printf(&#34;Reading from block %d, offset %d\\n&#34;, block_number, block_offset);\n \/\/ Read from disk block...\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"1242-linked-allocation\"><span>12.4.2 Linked Allocation<\/span>\n <a href=\"#1242-linked-allocation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * Linked Allocation:\n * Each block contains pointer to next block\n *\n * Pros:\n * - No external fragmentation\n * - File can grow easily\n * - No need to know size in advance\n *\n * Cons:\n * - Slow random access\n * - Space overhead (pointers)\n * - Poor reliability (broken link loses rest of file)\n *\/\n\n#define BLOCK_SIZE 4096\n#define DATA_SIZE (BLOCK_SIZE - sizeof(uint32_t))\n\ntypedef struct FileBlock {\n uint8_t data[DATA_SIZE];\n uint32_t next_block; \/\/ Next block number (or -1 for end)\n} FileBlock;\n\ntypedef struct {\n char filename[256];\n uint32_t first_block;\n} LinkedFile;\n\nvoid read_linked_file(LinkedFile *file, uint32_t offset) {\n uint32_t current_block = file-&gt;first_block;\n uint32_t bytes_to_skip = offset;\n\n \/\/ Follow chain to find correct block\n while (bytes_to_skip &gt; DATA_SIZE) {\n FileBlock block;\n \/\/ Read block from disk...\n\n current_block = block.next_block;\n if (current_block == (uint32_t)-1) {\n printf(&#34;Offset beyond end of file\\n&#34;);\n return;\n }\n\n bytes_to_skip -= DATA_SIZE;\n }\n\n printf(&#34;Reading from block %d, offset %d\\n&#34;, current_block, bytes_to_skip);\n}\n\n\/\/ FAT (File Allocation Table) Variant\n#define NUM_BLOCKS 10000\n\ntypedef struct {\n uint32_t fat[NUM_BLOCKS]; \/\/ fat[i] = next block after block i\n} FAT;\n\n\/*\n * FAT Advantages:\n * - Random access improved (FAT can be cached in memory)\n * - Only need to traverse FAT, not actual disk blocks\n * Used in: MS-DOS, FAT32\n *\/<\/code><\/pre><h4 class=\"heading-element\" id=\"1243-indexed-allocation\"><span>12.4.3 Indexed Allocation<\/span>\n <a href=\"#1243-indexed-allocation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * Indexed Allocation:\n * Index block contains pointers to all data blocks\n *\n * Pros:\n * - Fast random access\n * - No external fragmentation\n * - Supports direct access\n *\n * Cons:\n * - Index block overhead\n * - Maximum file size limited by index block size\n *\/\n\n#define POINTERS_PER_BLOCK (BLOCK_SIZE \/ sizeof(uint32_t)) \/\/ 1024 pointers\n\ntypedef struct {\n uint32_t block_pointers[POINTERS_PER_BLOCK];\n} IndexBlock;\n\ntypedef struct {\n char filename[256];\n uint32_t index_block_number;\n uint32_t file_size;\n} IndexedFile;\n\nvoid read_indexed_file(IndexedFile *file, uint32_t offset) {\n uint32_t block_num = offset \/ BLOCK_SIZE;\n uint32_t block_offset = offset % BLOCK_SIZE;\n\n if (block_num &gt;= POINTERS_PER_BLOCK) {\n printf(&#34;File too large for single index block\\n&#34;);\n return;\n }\n\n \/\/ Read index block\n IndexBlock index;\n \/\/ read_disk_block(file-&gt;index_block_number, &amp;index);\n\n uint32_t data_block = index.block_pointers[block_num];\n printf(&#34;Reading from block %d, offset %d\\n&#34;, data_block, block_offset);\n}\n\n\/\/ Multi-level Indexed (used in Unix inode)\ntypedef struct {\n uint32_t direct[12]; \/\/ Direct block pointers\n uint32_t single_indirect; \/\/ Points to block of pointers\n uint32_t double_indirect; \/\/ Points to block of indirect blocks\n uint32_t triple_indirect; \/\/ Three levels of indirection\n} MultiLevelIndex;\n\n\/*\n * Unix inode addressing:\n * - Direct: 12 * 4KB = 48KB\n * - Single indirect: 1024 * 4KB = 4MB\n * - Double indirect: 1024 * 1024 * 4KB = 4GB\n * - Triple indirect: 1024 * 1024 * 1024 * 4KB = 4TB\n * Total: ~4TB max file size\n *\/<\/code><\/pre><h3 class=\"heading-element\" id=\"125-unixlinux-inode-structure\"><span>12.5 Unix\/Linux Inode Structure<\/span>\n <a href=\"#125-unixlinux-inode-structure\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * Inode: Index Node\n * Contains all file metadata except name\n * Name stored in directory entry\n *\/\n\ntypedef struct {\n uint16_t mode; \/\/ File type and permissions\n uint16_t uid; \/\/ User ID\n uint32_t size; \/\/ File size in bytes\n uint32_t atime; \/\/ Access time\n uint32_t ctime; \/\/ Change time (metadata)\n uint32_t mtime; \/\/ Modification time (content)\n uint32_t dtime; \/\/ Deletion time\n uint16_t gid; \/\/ Group ID\n uint16_t links_count; \/\/ Hard link count\n uint32_t blocks; \/\/ Number of 512-byte blocks\n uint32_t flags; \/\/ File flags\n uint32_t block[15]; \/\/ Block pointers (12 direct &#43; 3 indirect)\n} Inode;\n\n\/\/ Directory entry\ntypedef struct {\n uint32_t inode_number;\n uint16_t rec_len; \/\/ Length of this directory entry\n uint8_t name_len; \/\/ Length of filename\n uint8_t file_type; \/\/ File type\n char name[255]; \/\/ Filename (variable length)\n} DirEntry;\n\n\/*\n * Hard Link: Multiple directory entries point to same inode\n * Symbolic Link: File containing path to another file\n *\/\n\n\/\/ Create hard link\nlink(&#34;original.txt&#34;, &#34;hardlink.txt&#34;); \/\/ Both point to same inode\n\n\/\/ Create symbolic link\nsymlink(&#34;original.txt&#34;, &#34;symlink.txt&#34;); \/\/ Contains path string\n\n\/\/ Differences:\n\/\/ - Hard link: Same inode, survives if original deleted\n\/\/ - Sym link: Different inode, breaks if original deleted<\/code><\/pre><h3 class=\"heading-element\" id=\"126-virtual-file-system-vfs\"><span>12.6 Virtual File System (VFS)<\/span>\n <a href=\"#126-virtual-file-system-vfs\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * VFS: Abstract layer above file systems\n * Provides uniform interface for different FS types\n * Allows ext4, FAT, NFS, etc. to coexist\n *\/\n\n\/\/ VFS abstraction\ntypedef struct FileSystemOperations {\n int (*open)(const char *path, int flags);\n ssize_t (*read)(int fd, void *buf, size_t count);\n ssize_t (*write)(int fd, const void *buf, size_t count);\n int (*close)(int fd);\n int (*mkdir)(const char *path, mode_t mode);\n \/\/ ... more operations\n} FSOperations;\n\n\/\/ Different file systems implement interface\nFSOperations ext4_ops = {\n .open = ext4_open,\n .read = ext4_read,\n .write = ext4_write,\n \/\/ ...\n};\n\nFSOperations fat32_ops = {\n .open = fat32_open,\n .read = fat32_read,\n .write = fat32_write,\n \/\/ ...\n};\n\n\/*\n * VFS Components:\n * 1. Superblock: Filesystem metadata\n * 2. Inode: File metadata\n * 3. Dentry: Directory entry cache\n * 4. File: Open file representation\n *\/\n\ntypedef struct VFS_Superblock {\n uint32_t magic; \/\/ Filesystem type\n uint32_t block_size;\n uint64_t total_blocks;\n uint64_t free_blocks;\n FSOperations *ops; \/\/ Filesystem-specific operations\n} VFS_Superblock;<\/code><\/pre><h3 class=\"heading-element\" id=\"127-free-space-management\"><span>12.7 Free Space Management<\/span>\n <a href=\"#127-free-space-management\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"1271-bit-vector-bitmap\"><span>12.7.1 Bit Vector (Bitmap)<\/span>\n <a href=\"#1271-bit-vector-bitmap\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * Bitmap: One bit per block\n * 0 = free, 1 = allocated\n *\n * Pros: Simple, efficient to find free blocks\n * Cons: Extra space overhead\n *\/\n\n#define TOTAL_BLOCKS 100000\n#define BITMAP_SIZE (TOTAL_BLOCKS \/ 8) \/\/ Bytes needed\n\nuint8_t bitmap[BITMAP_SIZE];\n\n\/\/ Set bit (mark block as allocated)\nvoid mark_allocated(uint32_t block_num) {\n uint32_t byte_index = block_num \/ 8;\n uint8_t bit_offset = block_num % 8;\n bitmap[byte_index] |= (1 &lt;&lt; bit_offset);\n}\n\n\/\/ Clear bit (mark block as free)\nvoid mark_free(uint32_t block_num) {\n uint32_t byte_index = block_num \/ 8;\n uint8_t bit_offset = block_num % 8;\n bitmap[byte_index] &amp;= ~(1 &lt;&lt; bit_offset);\n}\n\n\/\/ Check if block is free\nbool is_free(uint32_t block_num) {\n uint32_t byte_index = block_num \/ 8;\n uint8_t bit_offset = block_num % 8;\n return (bitmap[byte_index] &amp; (1 &lt;&lt; bit_offset)) == 0;\n}\n\n\/\/ Find first free block\nuint32_t find_free_block() {\n for (uint32_t i = 0; i &lt; BITMAP_SIZE; i&#43;&#43;) {\n if (bitmap[i] != 0xFF) { \/\/ Not all bits set\n \/\/ Find first zero bit\n for (uint8_t bit = 0; bit &lt; 8; bit&#43;&#43;) {\n if ((bitmap[i] &amp; (1 &lt;&lt; bit)) == 0) {\n return (i * 8) &#43; bit;\n }\n }\n }\n }\n return UINT32_MAX; \/\/ No free blocks\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"1272-linked-list\"><span>12.7.2 Linked List<\/span>\n <a href=\"#1272-linked-list\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * Linked List of Free Blocks\n *\n * Pros: No extra space needed (use free blocks themselves)\n * Cons: Cannot efficiently find contiguous space\n *\/\n\ntypedef struct FreeBlock {\n uint32_t next_free; \/\/ Stored in free block itself\n} FreeBlock;\n\ntypedef struct {\n uint32_t head; \/\/ First free block\n} FreeList;\n\nFreeList free_list = {.head = 0};\n\nuint32_t allocate_block(FreeList *list) {\n if (list-&gt;head == UINT32_MAX) {\n return UINT32_MAX; \/\/ No free blocks\n }\n\n uint32_t allocated = list-&gt;head;\n\n \/\/ Read free block to get next pointer\n FreeBlock block;\n \/\/ read_disk_block(allocated, &amp;block);\n\n list-&gt;head = block.next_free;\n\n printf(&#34;Allocated block %d\\n&#34;, allocated);\n return allocated;\n}\n\nvoid free_block(FreeList *list, uint32_t block_num) {\n FreeBlock block;\n block.next_free = list-&gt;head;\n\n \/\/ Write block back with updated pointer\n \/\/ write_disk_block(block_num, &amp;block);\n\n list-&gt;head = block_num;\n printf(&#34;Freed block %d\\n&#34;, block_num);\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"1273-grouping\"><span>12.7.3 Grouping<\/span>\n <a href=\"#1273-grouping\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * Grouping: Store addresses of n free blocks in first free block\n * Last address points to another block with n free addresses\n *\/\n\n#define ADDRESSES_PER_BLOCK 512\n\ntypedef struct FreeBlockGroup {\n uint32_t count;\n uint32_t blocks[ADDRESSES_PER_BLOCK];\n} FreeBlockGroup;<\/code><\/pre><h3 class=\"heading-element\" id=\"128-disk-layout-examples\"><span>12.8 Disk Layout Examples<\/span>\n <a href=\"#128-disk-layout-examples\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"ext4-filesystem-layout\"><span>ext4 Filesystem Layout<\/span>\n <a href=\"#ext4-filesystem-layout\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * ext4 (Fourth Extended Filesystem) - Linux default\n *\n * Structure:\n * [Boot Block][Block Group 0][Block Group 1]...[Block Group N]\n *\n * Each Block Group:\n * - Superblock (backup in some groups)\n * - Group Descriptors\n * - Block Bitmap\n * - Inode Bitmap\n * - Inode Table\n * - Data Blocks\n *\/\n\ntypedef struct ext4_superblock {\n uint32_t s_inodes_count; \/\/ Total inodes\n uint32_t s_blocks_count; \/\/ Total blocks\n uint32_t s_r_blocks_count; \/\/ Reserved blocks\n uint32_t s_free_blocks_count; \/\/ Free blocks\n uint32_t s_free_inodes_count; \/\/ Free inodes\n uint32_t s_first_data_block; \/\/ First data block\n uint32_t s_log_block_size; \/\/ Block size = 1024 &lt;&lt; s_log_block_size\n uint32_t s_blocks_per_group; \/\/ Blocks per group\n uint32_t s_inodes_per_group; \/\/ Inodes per group\n uint32_t s_magic; \/\/ Magic signature (0xEF53)\n \/\/ ... many more fields\n} ext4_superblock_t;\n\n#define EXT4_SUPER_MAGIC 0xEF53<\/code><\/pre><h3 class=\"heading-element\" id=\"129-file-system-operations-implementation\"><span>12.9 File System Operations Implementation<\/span>\n <a href=\"#129-file-system-operations-implementation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * Create a new file\n *\/\n\nint create_file(const char *path, mode_t mode) {\n \/\/ 1. Parse path to find parent directory\n char parent_path[PATH_MAX];\n char filename[256];\n \/\/ split_path(path, parent_path, filename);\n\n \/\/ 2. Allocate new inode\n uint32_t new_inode_num = allocate_inode();\n if (new_inode_num == UINT32_MAX) {\n return -ENOSPC; \/\/ No space\n }\n\n \/\/ 3. Initialize inode\n Inode *inode = get_inode(new_inode_num);\n inode-&gt;mode = mode | S_IFREG; \/\/ Regular file\n inode-&gt;uid = getuid();\n inode-&gt;gid = getgid();\n inode-&gt;size = 0;\n inode-&gt;atime = inode-&gt;mtime = inode-&gt;ctime = time(NULL);\n inode-&gt;links_count = 1;\n inode-&gt;blocks = 0;\n\n \/\/ 4. Add directory entry in parent\n DirEntry entry = {\n .inode_number = new_inode_num,\n .name_len = strlen(filename),\n .file_type = DT_REG,\n };\n strcpy(entry.name, filename);\n\n \/\/ add_dir_entry(parent_path, &amp;entry);\n\n printf(&#34;Created file: %s (inode %d)\\n&#34;, path, new_inode_num);\n return 0;\n}\n\n\/*\n * Delete a file\n *\/\n\nint delete_file(const char *path) {\n \/\/ 1. Find inode\n uint32_t inode_num = path_to_inode(path);\n if (inode_num == UINT32_MAX) {\n return -ENOENT; \/\/ File not found\n }\n\n Inode *inode = get_inode(inode_num);\n\n \/\/ 2. Decrement link count\n inode-&gt;links_count--;\n\n if (inode-&gt;links_count == 0) {\n \/\/ 3. Free data blocks\n for (int i = 0; i &lt; 12 &amp;&amp; inode-&gt;block[i] != 0; i&#43;&#43;) {\n free_block(&amp;free_list, inode-&gt;block[i]);\n }\n\n \/\/ Handle indirect blocks...\n\n \/\/ 4. Free inode\n free_inode(inode_num);\n\n printf(&#34;Deleted file: %s\\n&#34;, path);\n } else {\n printf(&#34;Decremented link count for: %s (links remaining: %d)\\n&#34;,\n path, inode-&gt;links_count);\n }\n\n \/\/ 5. Remove directory entry\n \/\/ remove_dir_entry(path);\n\n return 0;\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"1210-file-permissions-unixlinux\"><span>12.10 File Permissions (Unix\/Linux)<\/span>\n <a href=\"#1210-file-permissions-unixlinux\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * Unix File Permissions:\n * 9 bits: rwx rwx rwx (owner, group, others)\n * 3 special bits: setuid, setgid, sticky\n *\/\n\n\/\/ Permission bits\n#define S_IRUSR 0400 \/\/ User read\n#define S_IWUSR 0200 \/\/ User write\n#define S_IXUSR 0100 \/\/ User execute\n\n#define S_IRGRP 0040 \/\/ Group read\n#define S_IWGRP 0020 \/\/ Group write\n#define S_IXGRP 0010 \/\/ Group execute\n\n#define S_IROTH 0004 \/\/ Others read\n#define S_IWOTH 0002 \/\/ Others write\n#define S_IXOTH 0001 \/\/ Others execute\n\n\/\/ Check permissions\nbool can_access(Inode *inode, uid_t uid, gid_t gid, int required_perms) {\n mode_t mode = inode-&gt;mode;\n\n \/\/ Root can do anything\n if (uid == 0) {\n return true;\n }\n\n \/\/ Check owner permissions\n if (uid == inode-&gt;uid) {\n mode &gt;&gt;= 6; \/\/ Shift to owner bits\n }\n \/\/ Check group permissions\n else if (gid == inode-&gt;gid) {\n mode &gt;&gt;= 3; \/\/ Shift to group bits\n }\n \/\/ Others permissions (no shift needed)\n\n return (mode &amp; required_perms) == required_perms;\n}\n\n\/\/ Change permissions\nint chmod_file(const char *path, mode_t new_mode) {\n uint32_t inode_num = path_to_inode(path);\n Inode *inode = get_inode(inode_num);\n\n \/\/ Only owner or root can change permissions\n if (getuid() != 0 &amp;&amp; getuid() != inode-&gt;uid) {\n return -EPERM;\n }\n\n inode-&gt;mode = (inode-&gt;mode &amp; ~0777) | (new_mode &amp; 0777);\n printf(&#34;Changed permissions of %s to %o\\n&#34;, path, new_mode);\n\n return 0;\n}\n\n\/\/ Example usage\nchmod(&#34;file.txt&#34;, 0644); \/\/ rw-r--r--\nchmod(&#34;script.sh&#34;, 0755); \/\/ rwxr-xr-x<\/code><\/pre><h3 class=\"heading-element\" id=\"1211-journaling-file-systems\"><span>12.11 Journaling File Systems<\/span>\n <a href=\"#1211-journaling-file-systems\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * Journaling: Log changes before applying them\n * Ensures consistency after crash\n *\n * Types:\n * 1. Data journaling: Log both metadata and data\n * 2. Metadata journaling: Log only metadata (most common)\n * 3. Ordered: Write data before metadata\n *\/\n\ntypedef struct JournalEntry {\n uint64_t transaction_id;\n uint32_t operation; \/\/ CREATE, DELETE, MODIFY\n uint32_t inode_num;\n time_t timestamp;\n \/\/ ... operation-specific data\n} JournalEntry;\n\n\/\/ Write operation with journaling\nvoid journaled_write(const char *path, const void *data, size_t size) {\n \/\/ 1. Begin transaction - write to journal\n JournalEntry entry = {\n .transaction_id = get_next_txn_id(),\n .operation = OP_WRITE,\n .timestamp = time(NULL),\n };\n write_to_journal(&amp;entry);\n\n \/\/ 2. Write data to journal\n write_data_to_journal(data, size);\n\n \/\/ 3. Commit transaction in journal\n commit_journal_transaction(entry.transaction_id);\n\n \/\/ 4. Apply changes to filesystem (checkpoint)\n apply_to_filesystem(path, data, size);\n\n \/\/ 5. Mark transaction as complete in journal\n complete_journal_transaction(entry.transaction_id);\n\n printf(&#34;Journaled write completed\\n&#34;);\n}\n\n\/*\n * Recovery after crash:\n * 1. Scan journal for incomplete transactions\n * 2. Replay committed transactions\n * 3. Discard uncommitted transactions\n *\/\n\n\/\/ Used in: ext4, XFS, NTFS, HFS&#43;<\/code><\/pre><h3 class=\"heading-element\" id=\"1212-file-system-performance\"><span>12.12 File System Performance<\/span>\n <a href=\"#1212-file-system-performance\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * Performance Optimizations:\n *\n * 1. Block Caching (Buffer Cache)\n * 2. Read-ahead (Prefetching)\n * 3. Write-behind (Delayed writes)\n * 4. File System Check (fsck) minimization via journaling\n *\/\n\n\/\/ Simple cache implementation\n#define CACHE_SIZE 1024\n\ntypedef struct CacheEntry {\n uint32_t block_num;\n uint8_t data[BLOCK_SIZE];\n bool dirty;\n time_t last_access;\n} CacheEntry;\n\nCacheEntry cache[CACHE_SIZE];\n\nuint8_t* read_block_cached(uint32_t block_num) {\n \/\/ Check cache first\n for (int i = 0; i &lt; CACHE_SIZE; i&#43;&#43;) {\n if (cache[i].block_num == block_num) {\n cache[i].last_access = time(NULL);\n printf(&#34;Cache HIT: block %d\\n&#34;, block_num);\n return cache[i].data;\n }\n }\n\n \/\/ Cache miss - read from disk\n printf(&#34;Cache MISS: block %d\\n&#34;, block_num);\n\n \/\/ Find LRU entry to evict\n int lru_idx = find_lru_cache_entry();\n\n \/\/ Write back if dirty\n if (cache[lru_idx].dirty) {\n write_block_to_disk(cache[lru_idx].block_num, cache[lru_idx].data);\n }\n\n \/\/ Load new block\n read_block_from_disk(block_num, cache[lru_idx].data);\n cache[lru_idx].block_num = block_num;\n cache[lru_idx].dirty = false;\n cache[lru_idx].last_access = time(NULL);\n\n return cache[lru_idx].data;\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"key-takeaways-3\"><span>Key Takeaways<\/span>\n <a href=\"#key-takeaways-3\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>File System Essentials<\/strong>:<\/p>\n<ol>\n<li><strong>File systems organize data on persistent storage with hierarchical structure<\/strong> \u2192 Essential for data management<\/li>\n<li><strong>Unix inode separates metadata from filename (allows hard links)<\/strong> \u2192 Enables flexible file organization<\/li>\n<li><strong>Allocation methods<\/strong>: Contiguous (fast but fragmented), Linked (flexible but slow), Indexed (balanced) \u2192 Choose based on access patterns<\/li>\n<li><strong>VFS provides uniform interface for different filesystem types<\/strong> \u2192 Mount ext4, FAT32, NFS seamlessly<\/li>\n<li><strong>Free space management<\/strong>: Bitmap (fast), Linked list (space-efficient), Grouping (hybrid) \u2192 ext4 uses bitmaps<\/li>\n<li><strong>Unix permissions<\/strong>: 9 bits (rwx for owner\/group\/others)** \u2192 Foundational security model<\/li>\n<li><strong>Journaling ensures filesystem consistency after crashes<\/strong> \u2192 Metadata or full journaling<\/li>\n<li><strong>Caching critical for performance (buffer cache, read-ahead)<\/strong> \u2192 Reduces disk I\/O dramatically<\/li>\n<\/ol><\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"io-management\">13. I\/O Management<\/h2>\n<h3 class=\"heading-element\" id=\"131-io-subsystem-overview\"><span>13.1 I\/O Subsystem Overview<\/span>\n <a href=\"#131-io-subsystem-overview\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The I\/O subsystem manages communication between CPU and peripheral devices.<\/p>\n<pre><code>\/*\n * I\/O Components:\n * 1. CPU: Initiates I\/O operations\n * 2. Controller: Device-specific interface\n * 3. Device: Physical peripheral\n * 4. Bus\/Channel: Communication pathway\n *\/\n\ntypedef struct {\n uint8_t status; \/\/ Device status\n uint16_t control; \/\/ Control signals\n uint32_t data_register; \/\/ Data buffer\n} IODevice;\n\n\/*\n * Device Controller:\n * - Handles protocol specifics\n * - Manages data transfers\n * - Signals completion to CPU\n *\/\n\ntypedef struct {\n uint16_t status_register; \/\/ Device status\n uint16_t control_register; \/\/ Commands\n uint32_t data_port; \/\/ Data transfers\n uint32_t address_register; \/\/ Memory address for DMA\n} DeviceController;<\/code><\/pre><h3 class=\"heading-element\" id=\"132-io-techniques\"><span>13.2 I\/O Techniques<\/span>\n <a href=\"#132-io-techniques\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"1321-programmed-io-polling\"><span>13.2.1 Programmed I\/O (Polling)<\/span>\n <a href=\"#1321-programmed-io-polling\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * Programmed I\/O (Polling):\n * CPU continuously checks device status\n *\n * Pros: Simple implementation\n * Cons: CPU wastes cycles waiting (busy-waiting)\n *\/\n\n#define DEVICE_STATUS 0x1000\n#define DEVICE_DATA 0x1004\n#define DEVICE_BUSY 0x01\n#define DEVICE_READY 0x02\n\n\/\/ Polling approach\nvoid read_data_polling() {\n \/\/ Wait for device to be ready\n while ((inb(DEVICE_STATUS) &amp; DEVICE_READY) == 0) {\n \/\/ Busy wait - CPU does nothing useful\n }\n\n \/\/ Read data\n uint8_t data = inb(DEVICE_DATA);\n printf(&#34;Read: 0x%02X\\n&#34;, data);\n}\n\nvoid write_data_polling(uint8_t data) {\n \/\/ Wait for device to not be busy\n while ((inb(DEVICE_STATUS) &amp; DEVICE_BUSY) != 0) {\n \/\/ Busy wait\n }\n\n \/\/ Write data\n outb(DEVICE_DATA, data);\n printf(&#34;Wrote: 0x%02X\\n&#34;, data);\n}\n\n\/*\n * Polling pseudocode:\n * while (device_busy) ; \/\/ Wait (wastes CPU cycles)\n * read_data() \/\/ Get data from port\n *\/<\/code><\/pre><h4 class=\"heading-element\" id=\"1322-interrupt-driven-io\"><span>13.2.2 Interrupt-Driven I\/O<\/span>\n <a href=\"#1322-interrupt-driven-io\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * Interrupt-Driven I\/O:\n * Device signals CPU when ready (via interrupt)\n * CPU context switches to handler\n *\n * Pros: CPU not wasting cycles\n * Cons: Higher overhead per interrupt\n *\/\n\nvolatile uint8_t data_received = 0;\n\n\/\/ Interrupt handler (Interrupt Service Routine - ISR)\nvoid device_interrupt_handler() {\n printf(&#34;Interrupt received!\\n&#34;);\n\n \/\/ Read data from device\n data_received = inb(DEVICE_DATA);\n\n \/\/ Clear interrupt flag\n outb(DEVICE_CONTROL, 0x00);\n\n printf(&#34;Data: 0x%02X\\n&#34;, data_received);\n\n \/\/ Return to interrupted code\n}\n\nvoid read_data_interrupt() {\n \/\/ Enable interrupts for this device\n outb(DEVICE_CONTROL, 0x01);\n\n \/\/ Wait for data (using sleep or event)\n sleep(5); \/\/ Timeout after 5 seconds\n\n \/\/ Data now available in data_received\n printf(&#34;Data available: 0x%02X\\n&#34;, data_received);\n}\n\n\/*\n * Interrupt Processing:\n * 1. Device signals interrupt\n * 2. CPU saves current state\n * 3. CPU jumps to interrupt handler\n * 4. Handler processes device\n * 5. Handler clears interrupt\n * 6. CPU restores state and continues\n *\/<\/code><\/pre><h4 class=\"heading-element\" id=\"1323-direct-memory-access-dma\"><span>13.2.3 Direct Memory Access (DMA)<\/span>\n <a href=\"#1323-direct-memory-access-dma\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * DMA: Device transfers data directly to memory\n * CPU not involved in each byte transfer\n *\n * Pros: Minimal CPU overhead, fast transfers\n * Cons: Complex implementation, requires DMA controller\n *\/\n\ntypedef struct {\n uint32_t source_address; \/\/ Device address\n uint32_t dest_address; \/\/ Memory address\n uint32_t transfer_count; \/\/ Number of bytes\n uint8_t direction; \/\/ Read\/Write\n uint8_t control; \/\/ Enable, interrupt on complete\n} DMARequest;\n\n\/\/ DMA Channel descriptor\ntypedef struct {\n uint32_t memory_address;\n uint32_t transfer_count;\n uint8_t control; \/\/ Enable, interrupt flag\n uint8_t status; \/\/ Complete flag\n} DMAChannel;\n\n\/\/ Initialize DMA transfer\nvoid setup_dma_read(uint32_t device_addr, uint32_t mem_addr, uint32_t bytes) {\n \/\/ Set up DMA channel\n DMAChannel *dma = (DMAChannel *)0x8000; \/\/ DMA controller base address\n\n dma-&gt;memory_address = mem_addr;\n dma-&gt;transfer_count = bytes;\n dma-&gt;control = 0x01; \/\/ Enable DMA\n\n \/\/ Device starts transfer, DMA controller handles it\n printf(&#34;DMA transfer initiated: %d bytes from 0x%X to 0x%X\\n&#34;,\n bytes, device_addr, mem_addr);\n}\n\n\/\/ DMA completion interrupt\nvoid dma_completion_handler() {\n DMAChannel *dma = (DMAChannel *)0x8000;\n\n if (dma-&gt;status &amp; 0x01) {\n printf(&#34;DMA transfer complete!\\n&#34;);\n \/\/ Process transferred data...\n }\n}\n\n\/*\n * DMA Advantages:\n * - CPU free to do other work\n * - Fast bulk transfers\n * - Multiple DMA channels for concurrent transfers\n *\/<\/code><\/pre><h3 class=\"heading-element\" id=\"133-disk-scheduling-algorithms\"><span>13.3 Disk Scheduling Algorithms<\/span>\n <a href=\"#133-disk-scheduling-algorithms\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>When multiple I\/O requests queued, choose order to minimize seek time.<\/p>\n<h4 class=\"heading-element\" id=\"1331-fcfs-first-come-first-served\"><span>13.3.1 FCFS (First-Come-First-Served)<\/span>\n <a href=\"#1331-fcfs-first-come-first-served\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * FCFS: Service requests in order received\n * Pros: Fair, simple\n * Cons: High average seek time (disk may jump around)\n *\/\n\n#define DISK_TRACKS 200\n#define QUEUE_SIZE 32\n\ntypedef struct {\n int track;\n int id;\n} DiskRequest;\n\nvoid disk_schedule_fcfs(DiskRequest queue[], int queue_size) {\n int current_track = 0;\n int total_seek = 0;\n\n printf(&#34;=== FCFS Disk Scheduling ===\\n&#34;);\n for (int i = 0; i &lt; queue_size; i&#43;&#43;) {\n int seek = abs(queue[i].track - current_track);\n total_seek &#43;= seek;\n\n printf(&#34;Request %d: Track %d (seek: %d)\\n&#34;,\n queue[i].id, queue[i].track, seek);\n\n current_track = queue[i].track;\n }\n\n printf(&#34;Total head movement: %d tracks\\n&#34;, total_seek);\n}\n\n\/\/ Example: Queue = [98, 183, 37, 122, 14, 165, 30, 85]\n\/\/ FCFS movement: 0-&gt;98 (98) -&gt;183 (85) -&gt;37 (146) -&gt;122 (85) -&gt;14 (108) -&gt;165 (151) -&gt;30 (135) -&gt;85 (55) = 863<\/code><\/pre><h4 class=\"heading-element\" id=\"1332-sstf-shortest-seek-time-first\"><span>13.3.2 SSTF (Shortest Seek Time First)<\/span>\n <a href=\"#1332-sstf-shortest-seek-time-first\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * SSTF: Serve closest request next\n * Pros: Lower seek time than FCFS\n * Cons: Starvation possible (outer tracks if inner always busy)\n *\/\n\nvoid disk_schedule_sstf(DiskRequest queue[], int queue_size) {\n bool served[QUEUE_SIZE] = {false};\n int current_track = 0;\n int total_seek = 0;\n\n printf(&#34;=== SSTF Disk Scheduling ===\\n&#34;);\n\n for (int i = 0; i &lt; queue_size; i&#43;&#43;) {\n \/\/ Find closest unserved request\n int next_idx = -1;\n int min_seek = INT_MAX;\n\n for (int j = 0; j &lt; queue_size; j&#43;&#43;) {\n if (!served[j]) {\n int seek = abs(queue[j].track - current_track);\n if (seek &lt; min_seek) {\n min_seek = seek;\n next_idx = j;\n }\n }\n }\n\n served[next_idx] = true;\n total_seek &#43;= min_seek;\n\n printf(&#34;Request %d: Track %d (seek: %d)\\n&#34;,\n queue[next_idx].id, queue[next_idx].track, min_seek);\n\n current_track = queue[next_idx].track;\n }\n\n printf(&#34;Total head movement: %d tracks\\n&#34;, total_seek);\n}\n\n\/\/ Same queue: [98, 183, 37, 122, 14, 165, 30, 85]\n\/\/ SSTF movement: 0-&gt;30 (30) -&gt;37 (7) -&gt;85 (48) -&gt;98 (13) -&gt;122 (24) -&gt;165 (43) -&gt;183 (18) -&gt;14 (169) = 352<\/code><\/pre><h4 class=\"heading-element\" id=\"1333-scan-elevator-algorithm\"><span>13.3.3 SCAN (Elevator Algorithm)<\/span>\n <a href=\"#1333-scan-elevator-algorithm\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * SCAN: Head moves in one direction, serves requests, then reverses\n * Pros: Fair, prevents starvation, good for heavy load\n * Cons: Slightly higher seek time than SSTF\n *\/\n\nvoid disk_schedule_scan(DiskRequest queue[], int queue_size) {\n bool served[QUEUE_SIZE] = {false};\n int current_track = 0;\n int total_seek = 0;\n int direction = 1; \/\/ 1 = towards higher tracks, -1 = towards lower\n\n printf(&#34;=== SCAN (Elevator) Disk Scheduling ===\\n&#34;);\n\n for (int i = 0; i &lt; queue_size; i&#43;&#43;) {\n \/\/ Find closest request in current direction\n int next_idx = -1;\n int best_track = -1;\n\n for (int j = 0; j &lt; queue_size; j&#43;&#43;) {\n if (!served[j]) {\n int track = queue[j].track;\n\n \/\/ Check if in current direction\n if ((direction &gt; 0 &amp;&amp; track &gt;= current_track) ||\n (direction &lt; 0 &amp;&amp; track &lt;= current_track)) {\n \/\/ Find closest in direction\n if (best_track == -1 ||\n (direction &gt; 0 &amp;&amp; track &lt; best_track) ||\n (direction &lt; 0 &amp;&amp; track &gt; best_track)) {\n best_track = track;\n next_idx = j;\n }\n }\n }\n }\n\n \/\/ If no request in current direction, reverse\n if (next_idx == -1) {\n direction *= -1; \/\/ Reverse direction\n \/\/ Find closest in new direction\n for (int j = 0; j &lt; queue_size; j&#43;&#43;) {\n if (!served[j]) {\n best_track = queue[j].track;\n next_idx = j;\n break;\n }\n }\n }\n\n served[next_idx] = true;\n int seek = abs(queue[next_idx].track - current_track);\n total_seek &#43;= seek;\n\n printf(&#34;Request %d: Track %d (seek: %d, direction: %s)\\n&#34;,\n queue[next_idx].id, queue[next_idx].track, seek,\n direction &gt; 0 ? &#34;up&#34; : &#34;down&#34;);\n\n current_track = queue[next_idx].track;\n }\n\n printf(&#34;Total head movement: %d tracks\\n&#34;, total_seek);\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"1334-c-scan-circular-scan\"><span>13.3.4 C-SCAN (Circular SCAN)<\/span>\n <a href=\"#1334-c-scan-circular-scan\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>\/*\n * C-SCAN: After reaching end, jump to start and scan again\n * Pros: More uniform seek time distribution\n * Cons: Higher seek time for first few tracks after reversal\n *\/\n\nvoid disk_schedule_c_scan(DiskRequest queue[], int queue_size) {\n bool served[QUEUE_SIZE] = {false};\n int current_track = 0;\n int total_seek = 0;\n\n printf(&#34;=== C-SCAN Disk Scheduling ===\\n&#34;);\n\n for (int i = 0; i &lt; queue_size; i&#43;&#43;) {\n \/\/ Find closest request &gt;= current_track\n int next_idx = -1;\n int best_track = INT_MAX;\n\n for (int j = 0; j &lt; queue_size; j&#43;&#43;) {\n if (!served[j] &amp;&amp; queue[j].track &gt;= current_track) {\n if (queue[j].track &lt; best_track) {\n best_track = queue[j].track;\n next_idx = j;\n }\n }\n }\n\n \/\/ If none found, wrap to start\n if (next_idx == -1) {\n best_track = INT_MAX;\n for (int j = 0; j &lt; queue_size; j&#43;&#43;) {\n if (!served[j]) {\n if (queue[j].track &lt; best_track) {\n best_track = queue[j].track;\n next_idx = j;\n }\n }\n }\n }\n\n served[next_idx] = true;\n int seek = abs(queue[next_idx].track - current_track);\n total_seek &#43;= seek;\n\n printf(&#34;Request %d: Track %d (seek: %d)\\n&#34;,\n queue[next_idx].id, queue[next_idx].track, seek);\n\n current_track = queue[next_idx].track;\n }\n\n printf(&#34;Total head movement: %d tracks\\n&#34;, total_seek);\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"134-disk-scheduling-comparison\"><span>13.4 Disk Scheduling Comparison<\/span>\n <a href=\"#134-disk-scheduling-comparison\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Disk Scheduling Trade-offs<\/strong>:<\/p>\n<ul>\n<li><strong>FCFS<\/strong>: Fair but causes excessive head movement (thrashing) \u2192 Poor for workstations<\/li>\n<li><strong>SSTF<\/strong>: Minimizes seeks but starves outer tracks \u2192 Risky for critical data<\/li>\n<li><strong>SCAN\/C-SCAN<\/strong>: Balanced fairness and performance \u2192 Best for most systems (recommended)<\/li>\n<li><strong>LOOK<\/strong>: Practical SCAN variant \u2192 Similar performance to SCAN<\/li>\n<\/ul><\/div>\n <\/div>\n<\/div>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Algorithm<\/th>\n <th>Avg Seek Time<\/th>\n <th>Starvation<\/th>\n <th>Implementation<\/th>\n <th>Notes<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>FCFS<\/strong><\/td>\n <td>High<\/td>\n <td>No<\/td>\n <td>Simple<\/td>\n <td>Fair but inefficient<\/td>\n <\/tr>\n <tr>\n <td><strong>SSTF<\/strong><\/td>\n <td>Low-Medium<\/td>\n <td>Yes<\/td>\n <td>Medium<\/td>\n <td>Good but can starve outer tracks<\/td>\n <\/tr>\n <tr>\n <td><strong>SCAN<\/strong><\/td>\n <td>Medium<\/td>\n <td>No<\/td>\n <td>Complex<\/td>\n <td>Fair and good under heavy load<\/td>\n <\/tr>\n <tr>\n <td><strong>C-SCAN<\/strong><\/td>\n <td>Medium<\/td>\n <td>No<\/td>\n <td>Complex<\/td>\n <td>More uniform service<\/td>\n <\/tr>\n <tr>\n <td><strong>LOOK<\/strong><\/td>\n <td>Low-Medium<\/td>\n <td>No<\/td>\n <td>Complex<\/td>\n <td>Like SCAN but doesn&rsquo;t go to end<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"135-interrupt-handling\"><span>13.5 Interrupt Handling<\/span>\n <a href=\"#135-interrupt-handling\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * Interrupt Levels (Interrupt Request - IRQ)\n * x86 PC: 16 IRQs (IRQ0-15)\n * - IRQ0: System timer\n * - IRQ1: Keyboard\n * - IRQ8: Real-time clock\n * - IRQ14-15: IDE disk controllers\n *\/\n\n\/\/ Interrupt Descriptor Table (IDT) entry\ntypedef struct {\n uint16_t offset_low; \/\/ Lower 16 bits of handler address\n uint16_t selector; \/\/ Code segment selector\n uint8_t reserved;\n uint8_t type; \/\/ Gate type (trap, interrupt)\n uint16_t offset_high; \/\/ Upper 16 bits of handler address\n} IDTEntry;\n\n#define IDT_SIZE 256\nIDTEntry idt[IDT_SIZE];\n\n\/\/ Register interrupt handler\nvoid register_interrupt_handler(int irq, void (*handler)(void)) {\n uint32_t handler_addr = (uint32_t)handler;\n\n idt[irq].offset_low = handler_addr &amp; 0xFFFF;\n idt[irq].offset_high = (handler_addr &gt;&gt; 16) &amp; 0xFFFF;\n idt[irq].selector = 0x08; \/\/ Kernel code segment\n idt[irq].type = 0x8E; \/\/ 32-bit interrupt gate\n\n printf(&#34;Registered handler for IRQ%d at 0x%X\\n&#34;, irq, handler_addr);\n}\n\n\/\/ Interrupt context (saved CPU state)\ntypedef struct {\n uint32_t edi, esi, ebp, esp;\n uint32_t ebx, edx, ecx, eax;\n uint32_t ds, es, fs, gs;\n uint32_t error_code;\n uint32_t eip;\n uint32_t cs;\n uint32_t eflags;\n} InterruptContext;\n\n\/\/ Generic interrupt handler wrapper\nvoid generic_interrupt_handler(InterruptContext *ctx) {\n printf(&#34;Interrupt at 0x%X, Error: 0x%X\\n&#34;, ctx-&gt;eip, ctx-&gt;error_code);\n\n \/\/ Acknowledge interrupt to PIC (Programmable Interrupt Controller)\n outb(0x20, 0x20); \/\/ Send EOI to master PIC\n\n \/\/ Call appropriate handler based on interrupt number...\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"key-takeaways-4\"><span>Key Takeaways<\/span>\n <a href=\"#key-takeaways-4\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>I\/O Management Principles<\/strong>:<\/p>\n<ol>\n<li><strong>I\/O Techniques<\/strong>: Polling (simple, wasteful), Interrupts (efficient), DMA (best for bulk transfers)** \u2192 Use DMA for high throughput<\/li>\n<li><strong>Disk Scheduling<\/strong>: FCFS (fair, high seek), SSTF (low seek, starvation), SCAN (fair, medium seek)** \u2192 SCAN is production standard<\/li>\n<li><strong>Interrupt Handling<\/strong>: Device signals CPU via interrupt, handler processes request** \u2192 Minimize handler time<\/li>\n<li><strong>Performance depends on workload and device characteristics<\/strong> \u2192 Profile before optimizing<\/li>\n<\/ol><\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"system-calls\">14. System Calls and Interrupts<\/h2>\n<h3 class=\"heading-element\" id=\"141-system-calls-basics\"><span>14.1 System Calls Basics<\/span>\n <a href=\"#141-system-calls-basics\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>System calls<\/strong> are the interface between user programs and the OS kernel, providing controlled access to hardware and kernel services.<\/p>\n<pre><code>\/*\n * System Call Execution Flow:\n * 1. User program calls system call wrapper\n * 2. Arguments loaded into CPU registers\/stack\n * 3. Trap instruction (int 0x80 on x86)\n * 4. CPU switches to kernel mode\n * 5. Kernel handles request\n * 6. Return to user mode with result\n *\n * Context Switch Cost: Very expensive!\n *\/\n\n#include &lt;unistd.h&gt;\n#include &lt;fcntl.h&gt;\n#include &lt;sys\/stat.h&gt;\n\n\/\/ System calls frequently used in Unix\/Linux\nint fd = open(&#34;file.txt&#34;, O_RDONLY); \/\/ open() syscall\nssize_t n = read(fd, buf, 100); \/\/ read() syscall\nssize_t m = write(1, &#34;hello&#34;, 5); \/\/ write() syscall\nclose(fd); \/\/ close() syscall\n\npid_t pid = fork(); \/\/ fork() syscall\nexecve(&#34;\/bin\/ls&#34;, argv, env); \/\/ execve() syscall\n\nint status;\nwait(&amp;status); \/\/ wait() syscall<\/code><\/pre><h3 class=\"heading-element\" id=\"142-mode-switching-user-to-kernel\"><span>14.2 Mode Switching: User to Kernel<\/span>\n <a href=\"#142-mode-switching-user-to-kernel\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * User Mode vs Kernel Mode\n *\n * User Mode:\n * - Limited privileges\n * - Can&#39;t access hardware directly\n * - Can&#39;t modify memory protection\n * - Must use syscalls for privileged operations\n *\n * Kernel Mode:\n * - Full privileges\n * - Direct hardware access\n * - Can modify CPU registers\n * - Can access all memory\n *\/\n\n\/\/ x86 Privilege Levels (Rings):\n\/\/ Ring 0: Kernel (highest privilege)\n\/\/ Ring 1, 2: Rarely used\n\/\/ Ring 3: User (lowest privilege)\n\ntypedef struct {\n uint32_t eax, ebx, ecx, edx;\n uint32_t esi, edi, ebp, esp;\n uint32_t eip; \/\/ Instruction pointer\n uint32_t cs; \/\/ Code segment (determines privilege)\n uint32_t eflags; \/\/ Flags register\n uint32_t ss; \/\/ Stack segment\n} CPUState;\n\n\/*\n * Switching from User to Kernel:\n * 1. User executes &#34;int 0x80&#34; (software interrupt on x86)\n * 2. CPU changes to kernel mode\n * 3. Kernel trap handler takes over\n * 4. Kernel executes requested syscall\n * 5. &#34;iret&#34; instruction returns to user mode\n *\/<\/code><\/pre><h3 class=\"heading-element\" id=\"143-exception-and-trap-handling\"><span>14.3 Exception and Trap Handling<\/span>\n <a href=\"#143-exception-and-trap-handling\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * Exceptions: Internal CPU events (errors, traps)\n * Examples: Division by zero, page fault, invalid opcode\n *\n * Traps: Intentional exceptions used for syscalls\n *\/\n\n\/\/ x86 Exception Types\n#define EXC_DIVIDE_BY_ZERO 0 \/\/ Division by zero\n#define EXC_DEBUG 1 \/\/ Debugger breakpoint\n#define EXC_NMI 2 \/\/ Non-maskable interrupt\n#define EXC_BREAKPOINT 3 \/\/ Breakpoint trap\n#define EXC_OVERFLOW 4 \/\/ Signed overflow\n#define EXC_BOUNDS 5 \/\/ Bounds check\n#define EXC_INVALID_OP 6 \/\/ Invalid opcode\n#define EXC_NO_MATH_COPROC 7 \/\/ Math coprocessor not available\n#define EXC_DOUBLE_FAULT 8 \/\/ Exception during exception\n#define EXC_SEGMENT_OVERRUN 9 \/\/ Segment overrun\n#define EXC_BAD_TSS 10 \/\/ Invalid Task State Segment\n#define EXC_SEGMENT_NOT_PRES 11 \/\/ Segment not present\n#define EXC_STACK_FAULT 12 \/\/ Stack segment fault\n#define EXC_GEN_PROTECT_FAULT 13 \/\/ General protection fault\n#define EXC_PAGE_FAULT 14 \/\/ Page fault\n#define EXC_RESERVED 15 \/\/ Reserved\n#define EXC_MATH_FAULT 16 \/\/ Floating point error\n#define EXC_ALIGNMENT_CHECK 17 \/\/ Alignment check\n\n\/\/ Exception handler\nvoid exception_handler(int exc_number, InterruptContext *ctx) {\n switch (exc_number) {\n case EXC_DIVIDE_BY_ZERO:\n printf(&#34;Division by zero at 0x%X\\n&#34;, ctx-&gt;eip);\n signal_process(ctx-&gt;process_id, SIGFPE); \/\/ Floating point exception\n break;\n\n case EXC_PAGE_FAULT:\n printf(&#34;Page fault at 0x%X\\n&#34;, ctx-&gt;eip);\n \/\/ Try to handle page fault\n if (handle_page_fault(ctx-&gt;cr2)) { \/\/ CR2 contains faulting address\n return; \/\/ Handled successfully\n }\n signal_process(ctx-&gt;process_id, SIGSEGV); \/\/ Segmentation violation\n break;\n\n case EXC_GEN_PROTECT_FAULT:\n printf(&#34;General protection fault at 0x%X\\n&#34;, ctx-&gt;eip);\n signal_process(ctx-&gt;process_id, SIGSEGV);\n break;\n\n default:\n printf(&#34;Unhandled exception %d at 0x%X\\n&#34;, exc_number, ctx-&gt;eip);\n signal_process(ctx-&gt;process_id, SIGABRT); \/\/ Abort\n }\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"144-system-call-implementation\"><span>14.4 System Call Implementation<\/span>\n <a href=\"#144-system-call-implementation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * System Call Dispatch\n * Kernel maintains table of system call handlers\n *\/\n\n#define MAX_SYSCALLS 512\n\ntypedef long (*syscall_handler_t)(long arg1, long arg2, long arg3,\n long arg4, long arg5, long arg6);\n\nsyscall_handler_t syscall_table[MAX_SYSCALLS] = {\n [0] = sys_read,\n [1] = sys_write,\n [2] = sys_open,\n [3] = sys_close,\n [4] = sys_stat,\n \/\/ ... many more syscalls\n [11] = sys_execve,\n [20] = sys_getpid,\n [39] = sys_mkdir,\n \/\/ ... up to 512 syscalls on modern Linux\n};\n\n\/\/ Syscall entry point (assembly calls this)\nlong syscall_dispatcher(long syscall_number, long arg1, long arg2,\n long arg3, long arg4, long arg5, long arg6) {\n if (syscall_number &gt;= MAX_SYSCALLS) {\n return -ENOSYS; \/\/ Function not implemented\n }\n\n syscall_handler_t handler = syscall_table[syscall_number];\n if (!handler) {\n return -ENOSYS;\n }\n\n \/\/ Call the actual syscall handler\n return handler(arg1, arg2, arg3, arg4, arg5, arg6);\n}\n\n\/\/ Example: sys_write implementation\nlong sys_write(long fd, long buffer, long count, long arg4, long arg5, long arg6) {\n \/\/ Validate file descriptor\n if (fd &lt; 0 || fd &gt;= MAX_FDS) {\n return -EBADF; \/\/ Bad file descriptor\n }\n\n \/\/ Validate buffer is in user space\n if (!is_user_memory((void *)buffer, count)) {\n return -EFAULT; \/\/ Bad address\n }\n\n OpenFile *file = current_process-&gt;file_descriptors[fd];\n if (!file) {\n return -EBADF;\n }\n\n \/\/ Check write permissions\n if (!file-&gt;writable) {\n return -EACCES; \/\/ Permission denied\n }\n\n \/\/ Perform the write\n ssize_t written = file-&gt;write(file, (void *)buffer, count);\n\n return written;\n}\n\n\/\/ Example: sys_read implementation\nlong sys_read(long fd, long buffer, long count, long arg4, long arg5, long arg6) {\n if (fd &lt; 0 || fd &gt;= MAX_FDS) {\n return -EBADF;\n }\n\n if (!is_user_memory((void *)buffer, count)) {\n return -EFAULT;\n }\n\n OpenFile *file = current_process-&gt;file_descriptors[fd];\n if (!file) {\n return -EBADF;\n }\n\n if (!file-&gt;readable) {\n return -EACCES;\n }\n\n ssize_t bytes_read = file-&gt;read(file, (void *)buffer, count);\n return bytes_read;\n}\n\n\/\/ Example: sys_fork implementation\nlong sys_fork(long arg1, long arg2, long arg3, long arg4, long arg5, long arg6) {\n Process *current = get_current_process();\n\n \/\/ Allocate new process structure\n Process *child = allocate_process();\n if (!child) {\n return -ENOMEM; \/\/ No memory\n }\n\n \/\/ Copy process image (or use COW)\n copy_process_image(current, child);\n\n \/\/ Set parent-child relationship\n child-&gt;parent_pid = current-&gt;pid;\n child-&gt;ppid = current-&gt;pid;\n\n \/\/ Add to process list\n add_to_process_list(child);\n\n \/\/ In parent: return child PID\n \/\/ In child: this is handled by returning 0 to child\n return child-&gt;pid;\n}\n\n\/\/ Example: sys_exit implementation\nlong sys_exit(long exit_code, long arg2, long arg3, long arg4, long arg5, long arg6) {\n Process *current = get_current_process();\n\n \/\/ Store exit code\n current-&gt;exit_code = exit_code;\n current-&gt;state = PROCESS_ZOMBIE;\n\n \/\/ Close all file descriptors\n for (int i = 0; i &lt; MAX_FDS; i&#43;&#43;) {\n if (current-&gt;file_descriptors[i]) {\n sys_close(i, 0, 0, 0, 0, 0);\n }\n }\n\n \/\/ Free memory pages\n free_process_memory(current);\n\n \/\/ Wake up parent if waiting\n if (current-&gt;parent_pid != 0) {\n Process *parent = get_process_by_pid(current-&gt;parent_pid);\n if (parent &amp;&amp; parent-&gt;state == PROCESS_WAITING) {\n parent-&gt;state = PROCESS_READY;\n }\n }\n\n \/\/ Schedule next process\n schedule_next();\n\n return 0; \/\/ Never returns\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"145-virtual-system-call-vsyscall\"><span>14.5 Virtual System Call (vsyscall)<\/span>\n <a href=\"#145-virtual-system-call-vsyscall\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * vsyscall: Virtual system call (fast path for common syscalls)\n * Avoids context switch overhead for frequently-used syscalls\n * Mapped to user space (read-only)\n * Used for: gettimeofday, clock_gettime, getcpu\n *\/\n\n\/\/ vsyscall page mapped at fixed address (typically 0xFFFFE000)\n#define VSYSCALL_ADDR 0xFFFFE000\n\ntypedef struct {\n int (*gettimeofday)(struct timeval *tv, struct timezone *tz);\n long (*clock_gettime)(clockid_t clock_id, struct timespec *tp);\n int (*getcpu)(unsigned *cpu, unsigned *node);\n} vsyscall_functions_t;\n\n\/\/ User program can call without context switch\ntime_t fast_gettimeofday() {\n vsyscall_functions_t *vsyscall = (vsyscall_functions_t *)VSYSCALL_ADDR;\n struct timeval tv;\n\n vsyscall-&gt;gettimeofday(&amp;tv, NULL);\n return tv.tv_sec;\n}\n\n\/*\n * Replaced by vDSO (Virtual Dynamic Shared Object) in modern Linux\n * vDSO: Kernel-provided shared library\n * Functions: clock_gettime, gettimeofday, clock_nanosleep\n * Advantages:\n * - Faster than vsyscall\n * - More secure (randomized)\n * - Easy to upgrade\n *\/<\/code><\/pre><h3 class=\"heading-element\" id=\"146-comparison-user-vs-kernel-mode-operations\"><span>14.6 Comparison: User vs Kernel Mode Operations<\/span>\n <a href=\"#146-comparison-user-vs-kernel-mode-operations\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Operation<\/th>\n <th>User Mode<\/th>\n <th>Kernel Mode<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Direct memory access<\/strong><\/td>\n <td>Limited (own pages only)<\/td>\n <td>Unrestricted (all pages)<\/td>\n <\/tr>\n <tr>\n <td><strong>I\/O operations<\/strong><\/td>\n <td>Via syscalls<\/td>\n <td>Direct<\/td>\n <\/tr>\n <tr>\n <td><strong>CPU register access<\/strong><\/td>\n <td>Limited<\/td>\n <td>Full<\/td>\n <\/tr>\n <tr>\n <td><strong>Mode switching<\/strong><\/td>\n <td>No (need syscall)<\/td>\n <td>Can switch to user<\/td>\n <\/tr>\n <tr>\n <td><strong>Performance<\/strong><\/td>\n <td>Slower (syscall overhead)<\/td>\n <td>Faster (direct)<\/td>\n <\/tr>\n <tr>\n <td><strong>Safety<\/strong><\/td>\n <td>Protected (isolated)<\/td>\n <td>Privileged (risky)<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"key-takeaways---system-calls\"><span>Key Takeaways - System Calls<\/span>\n <a href=\"#key-takeaways---system-calls\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>System Calls Bridge User and Kernel Worlds<\/strong>:<\/p>\n<ol>\n<li><strong>System calls provide controlled interface to kernel services<\/strong> \u2192 Sandboxed access to hardware<\/li>\n<li><strong>Mode switch (user to kernel) involves context save\/restore<\/strong> \u2192 Expensive! (100s of cycles)<\/li>\n<li><strong>Syscall dispatch uses lookup table indexed by syscall number<\/strong> \u2192 Fast lookup via syscall table<\/li>\n<li><strong>vDSO\/vsyscall optimize frequent low-overhead syscalls<\/strong> \u2192 gettimeofday(), clock_gettime() avoid mode switch<\/li>\n<li><strong>Exceptions handle errors and traps within kernel<\/strong> \u2192 Division by zero, page faults, etc.<\/li>\n<\/ol><\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"ipc\">15. Inter-Process Communication (IPC)<\/h2>\n<h3 class=\"heading-element\" id=\"151-ipc-overview\"><span>15.1 IPC Overview<\/span>\n <a href=\"#151-ipc-overview\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Processes need to communicate and synchronize. IPC mechanisms provide this.<\/p>\n<pre><code>\/*\n * IPC Types:\n * 1. Pipes &amp; FIFOs\n * 2. Message Queues\n * 3. Shared Memory\n * 4. Sockets\n * 5. Signals\n * 6. Semaphores\n *\/<\/code><\/pre><h3 class=\"heading-element\" id=\"152-pipes-and-named-pipes-fifos\"><span>15.2 Pipes and Named Pipes (FIFOs)<\/span>\n <a href=\"#152-pipes-and-named-pipes-fifos\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * Pipe: Unidirectional communication channel\n * - Created by pipe() syscall\n * - Buffered (4096 bytes typically)\n * - Used for parent-child communication\n *\/\n\n#include &lt;unistd.h&gt;\n\nvoid pipe_example() {\n int pipefd[2]; \/\/ pipefd[0] = read end, pipefd[1] = write end\n\n if (pipe(pipefd) == -1) {\n perror(&#34;pipe&#34;);\n return;\n }\n\n pid_t pid = fork();\n\n if (pid == 0) {\n \/\/ Child process\n close(pipefd[1]); \/\/ Close write end\n\n char buffer[100];\n ssize_t n = read(pipefd[0], buffer, sizeof(buffer));\n\n printf(&#34;Child received: %.*s\\n&#34;, (int)n, buffer);\n\n close(pipefd[0]);\n exit(0);\n } else {\n \/\/ Parent process\n close(pipefd[0]); \/\/ Close read end\n\n const char *msg = &#34;Hello from parent&#34;;\n write(pipefd[1], msg, strlen(msg));\n\n close(pipefd[1]);\n wait(NULL);\n }\n}\n\n\/*\n * Named Pipe (FIFO):\n * - Persistent named pipe\n * - Created with mkfifo()\n * - Can be opened by unrelated processes\n *\/\n\n#include &lt;sys\/stat.h&gt;\n#include &lt;fcntl.h&gt;\n\nvoid fifo_example() {\n \/\/ Create FIFO\n mkfifo(&#34;\/tmp\/myfifo&#34;, 0666);\n\n \/\/ Writer process\n if (fork() == 0) {\n int fd = open(&#34;\/tmp\/myfifo&#34;, O_WRONLY);\n write(fd, &#34;Data&#34;, 4);\n close(fd);\n exit(0);\n }\n\n \/\/ Reader process\n int fd = open(&#34;\/tmp\/myfifo&#34;, O_RDONLY);\n char buffer[10];\n read(fd, buffer, 4);\n printf(&#34;Received: %.4s\\n&#34;, buffer);\n close(fd);\n\n unlink(&#34;\/tmp\/myfifo&#34;);\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"153-message-queues\"><span>15.3 Message Queues<\/span>\n <a href=\"#153-message-queues\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * Message Queue: POSIX IPC mechanism\n * - Persistent (survives process termination)\n * - Multiple senders\/receivers\n * - Messages have types\n *\/\n\n#include &lt;mqueue.h&gt;\n#include &lt;string.h&gt;\n\ntypedef struct {\n long mtype;\n char mtext[256];\n} Message;\n\nvoid message_queue_example() {\n struct mq_attr attr = {\n .mq_flags = 0,\n .mq_maxmsg = 10,\n .mq_msgsize = sizeof(Message),\n };\n\n \/\/ Create message queue\n mqd_t mq = mq_open(&#34;\/myqueue&#34;, O_CREAT | O_RDWR, 0666, &amp;attr);\n if (mq == (mqd_t)-1) {\n perror(&#34;mq_open&#34;);\n return;\n }\n\n \/\/ Producer: Send message\n if (fork() == 0) {\n Message msg = {\n .mtype = 1,\n };\n strcpy(msg.mtext, &#34;Hello from producer&#34;);\n\n if (mq_send(mq, (const char *)&amp;msg, sizeof(msg), 0) == -1) {\n perror(&#34;mq_send&#34;);\n }\n\n mq_close(mq);\n exit(0);\n }\n\n \/\/ Consumer: Receive message\n Message msg;\n if (mq_receive(mq, (char *)&amp;msg, sizeof(msg), NULL) == -1) {\n perror(&#34;mq_receive&#34;);\n }\n\n printf(&#34;Received: type=%ld, text=%s\\n&#34;, msg.mtype, msg.mtext);\n\n mq_close(mq);\n mq_unlink(&#34;\/myqueue&#34;);\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"154-shared-memory\"><span>15.4 Shared Memory<\/span>\n <a href=\"#154-shared-memory\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * Shared Memory: Fast IPC (no copying)\n * - Multiple processes access same memory region\n * - Requires synchronization (mutex\/semaphore)\n * - Linux: System V IPC or POSIX shm\n *\/\n\n#include &lt;sys\/ipc.h&gt;\n#include &lt;sys\/shm.h&gt;\n#include &lt;string.h&gt;\n\ntypedef struct {\n int counter;\n char data[256];\n pthread_mutex_t lock;\n} SharedData;\n\nvoid shared_memory_example() {\n key_t key = ftok(&#34;\/tmp&#34;, &#39;S&#39;); \/\/ Create unique key\n\n \/\/ Create or get shared memory segment\n int shmid = shmget(key, sizeof(SharedData), IPC_CREAT | 0666);\n if (shmid == -1) {\n perror(&#34;shmget&#34;);\n return;\n }\n\n \/\/ Attach to process address space\n SharedData *shared = (SharedData *)shmat(shmid, NULL, 0);\n if (shared == (void *)-1) {\n perror(&#34;shmat&#34;);\n return;\n }\n\n \/\/ Initialize on first use\n static int initialized = 0;\n if (!initialized) {\n memset(shared, 0, sizeof(SharedData));\n pthread_mutex_init(&amp;shared-&gt;lock, NULL);\n initialized = 1;\n }\n\n \/\/ Multiple processes can access\n pthread_mutex_lock(&amp;shared-&gt;lock);\n shared-&gt;counter&#43;&#43;;\n strcpy(shared-&gt;data, &#34;Shared data&#34;);\n pthread_mutex_unlock(&amp;shared-&gt;lock);\n\n printf(&#34;Counter: %d, Data: %s\\n&#34;, shared-&gt;counter, shared-&gt;data);\n\n \/\/ Detach from memory\n shmdt(shared);\n\n \/\/ Only last process to detach should delete\n \/\/ shmctl(shmid, IPC_RMID, NULL);\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"155-sockets\"><span>15.5 Sockets<\/span>\n <a href=\"#155-sockets\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * Sockets: Network IPC (local and remote)\n * - Unix domain sockets: Local machine only\n * - Network sockets: TCP\/IP over network\n *\/\n\n#include &lt;sys\/socket.h&gt;\n#include &lt;sys\/un.h&gt;\n#include &lt;netinet\/in.h&gt;\n#include &lt;string.h&gt;\n\n\/\/ Unix domain socket server\nvoid unix_socket_server() {\n int sockfd = socket(AF_UNIX, SOCK_STREAM, 0);\n if (sockfd == -1) {\n perror(&#34;socket&#34;);\n return;\n }\n\n struct sockaddr_un addr = {\n .sun_family = AF_UNIX,\n };\n strcpy(addr.sun_path, &#34;\/tmp\/server.sock&#34;);\n\n \/\/ Remove old socket file if exists\n unlink(addr.sun_path);\n\n if (bind(sockfd, (struct sockaddr *)&amp;addr, sizeof(addr)) == -1) {\n perror(&#34;bind&#34;);\n return;\n }\n\n if (listen(sockfd, 1) == -1) {\n perror(&#34;listen&#34;);\n return;\n }\n\n printf(&#34;Server listening on \/tmp\/server.sock\\n&#34;);\n\n struct sockaddr_un peer_addr;\n socklen_t peer_addr_len = sizeof(peer_addr);\n\n int connfd = accept(sockfd, (struct sockaddr *)&amp;peer_addr, &amp;peer_addr_len);\n if (connfd == -1) {\n perror(&#34;accept&#34;);\n return;\n }\n\n char buffer[256];\n ssize_t n = recv(connfd, buffer, sizeof(buffer), 0);\n printf(&#34;Received: %.*s\\n&#34;, (int)n, buffer);\n\n close(connfd);\n close(sockfd);\n}\n\n\/\/ Unix domain socket client\nvoid unix_socket_client() {\n int sockfd = socket(AF_UNIX, SOCK_STREAM, 0);\n if (sockfd == -1) {\n perror(&#34;socket&#34;);\n return;\n }\n\n struct sockaddr_un addr = {\n .sun_family = AF_UNIX,\n };\n strcpy(addr.sun_path, &#34;\/tmp\/server.sock&#34;);\n\n if (connect(sockfd, (struct sockaddr *)&amp;addr, sizeof(addr)) == -1) {\n perror(&#34;connect&#34;);\n return;\n }\n\n const char *msg = &#34;Hello from client&#34;;\n send(sockfd, msg, strlen(msg), 0);\n\n close(sockfd);\n}\n\n\/\/ TCP socket (network)\nvoid tcp_socket_server(int port) {\n int sockfd = socket(AF_INET, SOCK_STREAM, 0);\n\n struct sockaddr_in addr = {\n .sin_family = AF_INET,\n .sin_port = htons(port),\n .sin_addr.s_addr = htonl(INADDR_ANY),\n };\n\n int opt = 1;\n setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &amp;opt, sizeof(opt));\n\n bind(sockfd, (struct sockaddr *)&amp;addr, sizeof(addr));\n listen(sockfd, 5);\n\n printf(&#34;Server listening on port %d\\n&#34;, port);\n\n struct sockaddr_in client_addr;\n socklen_t client_addr_len = sizeof(client_addr);\n\n int connfd = accept(sockfd, (struct sockaddr *)&amp;client_addr, &amp;client_addr_len);\n\n char buffer[256];\n recv(connfd, buffer, sizeof(buffer), 0);\n printf(&#34;Received: %s\\n&#34;, buffer);\n\n close(connfd);\n close(sockfd);\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"156-signals\"><span>15.6 Signals<\/span>\n <a href=\"#156-signals\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * Signals: Asynchronous notifications\n * - Limited information (only signal number)\n * - Can interrupt process execution\n * - Common signals: SIGTERM, SIGKILL, SIGCHLD\n *\/\n\n#include &lt;signal.h&gt;\n\nvoid signal_handler(int sig) {\n switch (sig) {\n case SIGTERM:\n printf(&#34;Received SIGTERM (terminate)\\n&#34;);\n exit(0);\n case SIGINT:\n printf(&#34;Received SIGINT (interrupt, Ctrl&#43;C)\\n&#34;);\n exit(0);\n case SIGCHLD:\n printf(&#34;Child process terminated\\n&#34;);\n break;\n case SIGUSR1:\n printf(&#34;Received SIGUSR1 (user-defined)\\n&#34;);\n break;\n }\n}\n\nvoid signal_example() {\n \/\/ Register signal handler\n signal(SIGTERM, signal_handler);\n signal(SIGINT, signal_handler);\n signal(SIGCHLD, signal_handler);\n signal(SIGUSR1, signal_handler);\n\n printf(&#34;Process %d waiting for signals...\\n&#34;, getpid());\n\n \/\/ Infinite loop, waiting for signals\n while (1) {\n pause(); \/\/ Wait for signal\n }\n}\n\n\/\/ Send signal to process\nvoid send_signal_example() {\n pid_t target_pid = 1234;\n\n \/\/ Send SIGUSR1\n if (kill(target_pid, SIGUSR1) == -1) {\n perror(&#34;kill&#34;);\n }\n}\n\n\/\/ Signal mask (block\/unblock signals)\nvoid signal_mask_example() {\n sigset_t set;\n\n \/\/ Create set with SIGTERM\n sigemptyset(&amp;set);\n sigaddset(&amp;set, SIGTERM);\n\n \/\/ Block SIGTERM\n if (sigprocmask(SIG_BLOCK, &amp;set, NULL) == -1) {\n perror(&#34;sigprocmask&#34;);\n }\n\n printf(&#34;SIGTERM is now blocked\\n&#34;);\n\n \/\/ Unblock SIGTERM\n if (sigprocmask(SIG_UNBLOCK, &amp;set, NULL) == -1) {\n perror(&#34;sigprocmask&#34;);\n }\n\n printf(&#34;SIGTERM is now unblocked\\n&#34;);\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"157-ipc-comparison\"><span>15.7 IPC Comparison<\/span>\n <a href=\"#157-ipc-comparison\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Mechanism<\/th>\n <th>Speed<\/th>\n <th>Persistence<\/th>\n <th>Relationship<\/th>\n <th>Use Case<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Pipe<\/strong><\/td>\n <td>Very Fast<\/td>\n <td>No<\/td>\n <td>Parent-child<\/td>\n <td>Simple stream<\/td>\n <\/tr>\n <tr>\n <td><strong>FIFO<\/strong><\/td>\n <td>Fast<\/td>\n <td>Yes<\/td>\n <td>Unrelated<\/td>\n <td>Named pipe<\/td>\n <\/tr>\n <tr>\n <td><strong>Message Queue<\/strong><\/td>\n <td>Medium<\/td>\n <td>Yes<\/td>\n <td>Any<\/td>\n <td>Asynchronous messages<\/td>\n <\/tr>\n <tr>\n <td><strong>Shared Memory<\/strong><\/td>\n <td>Fastest<\/td>\n <td>Yes<\/td>\n <td>Any<\/td>\n <td>High-speed data sharing<\/td>\n <\/tr>\n <tr>\n <td><strong>Socket<\/strong><\/td>\n <td>Medium<\/td>\n <td>No<\/td>\n <td>Any (network)<\/td>\n <td>Network communication<\/td>\n <\/tr>\n <tr>\n <td><strong>Signal<\/strong><\/td>\n <td>Very Fast<\/td>\n <td>No<\/td>\n <td>Any<\/td>\n <td>Event notification<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"key-takeaways---ipc\"><span>Key Takeaways - IPC<\/span>\n <a href=\"#key-takeaways---ipc\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Choosing the Right IPC Mechanism<\/strong>:<\/p>\n<ol>\n<li><strong>Pipes: Simple, unidirectional, parent-child only<\/strong> \u2192 Use for process pipelines (ls | grep)<\/li>\n<li><strong>Message Queues: Asynchronous, persistent, typed messages<\/strong> \u2192 Use for decoupled communication<\/li>\n<li><strong>Shared Memory: Fastest but needs synchronization<\/strong> \u2192 Use for high-throughput data sharing<\/li>\n<li><strong>Sockets: Flexible, supports local and network communication<\/strong> \u2192 Use for network services<\/li>\n<li><strong>Signals: Lightweight but limited information<\/strong> \u2192 Use for event notifications<\/li>\n<li><strong>Choose based on performance, persistence, and relationship needs<\/strong> \u2192 Profile your use case!<\/li>\n<\/ol><\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"security\">16. Security and Protection<\/h2>\n<h3 class=\"heading-element\" id=\"161-security-fundamentals\"><span>16.1 Security Fundamentals<\/span>\n <a href=\"#161-security-fundamentals\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/security-cia-triad.webp\" title=\"CIA Triad - Information Security Fundamentals\" data-thumbnail=\"\/files\/security-cia-triad.webp\" data-sub-html=\"<h2>CIA Triad Security Model showing Confidentiality, Integrity, and Availability goals with encryption, access control, and redundancy mechanisms<\/h2><p>CIA Triad - Information Security Fundamentals<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/security-cia-triad.webp' alt=\"CIA Triad Security Model showing Confidentiality, Integrity, and Availability goals with encryption, access control, and redundancy mechanisms\"><\/a><figcaption class=\"image-caption\">CIA Triad - Information Security Fundamentals<\/figcaption>\n <\/figure><\/p>\n<p><em>Figure 6: CIA Triad security model illustrating the three pillars of information security: Confidentiality (encryption, access control), Integrity (checksums, digital signatures), and Availability (redundancy, load balancing).<\/em><\/p>\n<pre><code>\/*\n * CIA Triad:\n * - Confidentiality: Prevent unauthorized access\n * - Integrity: Prevent unauthorized modification\n * - Availability: Ensure authorized access\n *\/\n\n\/\/ Security levels in operating systems\ntypedef enum {\n SECURITY_NONE = 0,\n SECURITY_LOW = 1,\n SECURITY_MEDIUM = 2,\n SECURITY_HIGH = 3,\n SECURITY_TOP_SECRET = 4\n} SecurityLevel;<\/code><\/pre><h3 class=\"heading-element\" id=\"162-authentication-and-authorization\"><span>16.2 Authentication and Authorization<\/span>\n <a href=\"#162-authentication-and-authorization\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/authentication-authorization-flow.webp\" title=\"Authentication vs Authorization Process Flow\" data-thumbnail=\"\/files\/authentication-authorization-flow.webp\" data-sub-html=\"<h2>Authentication and Authorization Flow Sequence Diagram showing login, credential verification, and access control process<\/h2><p>Authentication vs Authorization Process Flow<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/authentication-authorization-flow.webp' alt=\"Authentication and Authorization Flow Sequence Diagram showing login, credential verification, and access control process\"><\/a><figcaption class=\"image-caption\">Authentication vs Authorization Process Flow<\/figcaption>\n <\/figure><\/p>\n<p><em>Figure 7: Authentication and authorization sequence diagram demonstrating user login, credential verification, session token generation, and resource access control.<\/em><\/p>\n<pre><code>\/*\n * Authentication: Verify identity (who are you?)\n * Authorization: Check permissions (what can you do?)\n *\/\n\n#include &lt;string.h&gt;\n#include &lt;crypt.h&gt;\n\ntypedef struct {\n char username[32];\n char password_hash[128];\n uid_t uid;\n gid_t gid;\n time_t last_login;\n} UserAccount;\n\n\/\/ Password authentication\nbool authenticate_user(const char *username, const char *password) {\n \/\/ Read user from \/etc\/shadow (simplified)\n UserAccount user;\n if (!get_user_account(username, &amp;user)) {\n return false;\n }\n\n \/\/ Hash the provided password\n char *hash = crypt(password, user.password_hash);\n\n \/\/ Compare with stored hash\n if (strcmp(hash, user.password_hash) == 0) {\n user.last_login = time(NULL);\n update_user_account(&amp;user);\n return true;\n }\n\n return false;\n}\n\n\/\/ Multi-factor authentication\nbool mfa_authenticate(const char *username, const char *password, const char *token) {\n \/\/ First factor: password\n if (!authenticate_user(username, password)) {\n return false;\n }\n\n \/\/ Second factor: OTP token\n if (!verify_otp_token(username, token)) {\n return false;\n }\n\n return true;\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"163-access-control-models\"><span>16.3 Access Control Models<\/span>\n <a href=\"#163-access-control-models\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"1631-discretionary-access-control-dac\"><span>16.3.1 Discretionary Access Control (DAC)<\/span>\n <a href=\"#1631-discretionary-access-control-dac\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/discretionary-access-control-dac.webp\" title=\"DAC - Owner-Controlled Access\" data-thumbnail=\"\/files\/discretionary-access-control-dac.webp\" data-sub-html=\"<h2>Discretionary Access Control DAC model showing owner-based permission management for files and resources<\/h2><p>DAC - Owner-Controlled Access<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/discretionary-access-control-dac.webp' alt=\"Discretionary Access Control DAC model showing owner-based permission management for files and resources\"><\/a><figcaption class=\"image-caption\">DAC - Owner-Controlled Access<\/figcaption>\n <\/figure><\/p>\n<p><em>Figure 8: Discretionary Access Control (DAC) model where resource owners set permissions (rwx) for users, groups, and others, as implemented in Unix\/Linux file systems.<\/em><\/p>\n<pre><code>\/*\n * DAC: Owner controls access to their resources\n * Used in Unix\/Linux file permissions\n *\/\n\n\/\/ Unix permission structure\ntypedef struct {\n uid_t owner_uid;\n gid_t owner_gid;\n mode_t permissions; \/\/ rwxrwxrwx\n} DACPermissions;\n\nbool check_dac_access(DACPermissions *perm, uid_t uid, gid_t gid, int access_mode) {\n mode_t mode = perm-&gt;permissions;\n\n \/\/ Root has all permissions\n if (uid == 0) {\n return true;\n }\n\n \/\/ Owner permissions\n if (uid == perm-&gt;owner_uid) {\n mode &gt;&gt;= 6; \/\/ Get owner bits\n return (mode &amp; access_mode) == access_mode;\n }\n\n \/\/ Group permissions\n if (gid == perm-&gt;owner_gid) {\n mode &gt;&gt;= 3; \/\/ Get group bits\n mode &amp;= 0x7;\n return (mode &amp; access_mode) == access_mode;\n }\n\n \/\/ Others permissions\n mode &amp;= 0x7;\n return (mode &amp; access_mode) == access_mode;\n}\n\n\/\/ Example: chmod command implementation\nvoid change_permissions(const char *path, mode_t new_mode) {\n \/\/ Verify caller is owner or root\n struct stat st;\n stat(path, &amp;st);\n\n if (getuid() != 0 &amp;&amp; getuid() != st.st_uid) {\n printf(&#34;Permission denied: not owner\\n&#34;);\n return;\n }\n\n \/\/ Change permissions\n chmod(path, new_mode);\n printf(&#34;Changed permissions to %o\\n&#34;, new_mode);\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"1632-mandatory-access-control-mac\"><span>16.3.2 Mandatory Access Control (MAC)<\/span>\n <a href=\"#1632-mandatory-access-control-mac\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/mandatory-access-control-mac.webp\" title=\"MAC - System-Enforced Security Policy\" data-thumbnail=\"\/files\/mandatory-access-control-mac.webp\" data-sub-html=\"<h2>Mandatory Access Control MAC with security labels and Bell-LaPadula Model enforcing information flow control<\/h2><p>MAC - System-Enforced Security Policy<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/mandatory-access-control-mac.webp' alt=\"Mandatory Access Control MAC with security labels and Bell-LaPadula Model enforcing information flow control\"><\/a><figcaption class=\"image-caption\">MAC - System-Enforced Security Policy<\/figcaption>\n <\/figure><\/p>\n<p><em>Figure 9: Mandatory Access Control (MAC) system with hierarchical security labels (Top Secret, Secret, Confidential, Unclassified) and Bell-LaPadula model enforcing &ldquo;No Read Up, No Write Down&rdquo; rules.<\/em><\/p>\n<pre><code>\/*\n * MAC: System enforces security policy\n * User cannot override (unlike DAC)\n * Used in: SELinux, AppArmor\n *\/\n\ntypedef struct {\n SecurityLevel classification;\n char category[32];\n bool need_to_know;\n} MACLabel;\n\ntypedef struct {\n SecurityLevel clearance;\n char categories[10][32];\n int num_categories;\n} UserClearance;\n\n\/\/ Bell-LaPadula Model: No read up, no write down\nbool mac_can_read(UserClearance *user, MACLabel *object) {\n \/\/ Can read if clearance &gt;= classification\n return user-&gt;clearance &gt;= object-&gt;classification;\n}\n\nbool mac_can_write(UserClearance *user, MACLabel *object) {\n \/\/ Can write if clearance &lt;= classification\n return user-&gt;clearance &lt;= object-&gt;classification;\n}\n\n\/\/ SELinux-style context\ntypedef struct {\n char user[32]; \/\/ SELinux user\n char role[32]; \/\/ Role\n char type[32]; \/\/ Type\/Domain\n char level[32]; \/\/ MLS level\n} SELinuxContext;\n\nbool selinux_check_access(SELinuxContext *subject,\n SELinuxContext *object,\n const char *permission) {\n \/\/ Check against SELinux policy\n \/\/ Simplified: check type enforcement\n return selinux_policy_allows(subject-&gt;type, object-&gt;type, permission);\n}<\/code><\/pre><h4 class=\"heading-element\" id=\"1633-role-based-access-control-rbac\"><span>16.3.3 Role-Based Access Control (RBAC)<\/span>\n <a href=\"#1633-role-based-access-control-rbac\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/role-based-access-control-rbac.webp\" title=\"RBAC - Role-Based Permissions\" data-thumbnail=\"\/files\/role-based-access-control-rbac.webp\" data-sub-html=\"<h2>Role-Based Access Control RBAC showing users assigned to roles with permissions for enterprise resource management<\/h2><p>RBAC - Role-Based Permissions<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/role-based-access-control-rbac.webp' alt=\"Role-Based Access Control RBAC showing users assigned to roles with permissions for enterprise resource management\"><\/a><figcaption class=\"image-caption\">RBAC - Role-Based Permissions<\/figcaption>\n <\/figure><\/p>\n<p><em>Figure 10: Role-Based Access Control (RBAC) architecture where users are assigned roles (Admin, Developer, Viewer) that determine their permissions to access organizational resources.<\/em><\/p>\n<pre><code>\/*\n * RBAC: Access based on user&#39;s role\n * Simplifies management for large organizations\n *\/\n\ntypedef enum {\n PERM_READ = 0x01,\n PERM_WRITE = 0x02,\n PERM_EXECUTE = 0x04,\n PERM_DELETE = 0x08,\n PERM_ADMIN = 0x10\n} Permission;\n\ntypedef struct {\n char role_name[32];\n Permission permissions;\n char resources[10][64];\n int num_resources;\n} Role;\n\ntypedef struct {\n char username[32];\n Role *roles[5];\n int num_roles;\n} User;\n\nbool rbac_check_access(User *user, const char *resource, Permission required) {\n \/\/ Check if any of user&#39;s roles grant the permission\n for (int i = 0; i &lt; user-&gt;num_roles; i&#43;&#43;) {\n Role *role = user-&gt;roles[i];\n\n \/\/ Check if role has permission\n if ((role-&gt;permissions &amp; required) != required) {\n continue;\n }\n\n \/\/ Check if role applies to this resource\n for (int j = 0; j &lt; role-&gt;num_resources; j&#43;&#43;) {\n if (strcmp(role-&gt;resources[j], resource) == 0) {\n return true;\n }\n }\n }\n\n return false;\n}\n\n\/\/ Example: Database RBAC\nvoid create_role_example() {\n Role admin_role = {\n .role_name = &#34;Administrator&#34;,\n .permissions = PERM_READ | PERM_WRITE | PERM_EXECUTE | PERM_DELETE | PERM_ADMIN,\n .num_resources = 1\n };\n strcpy(admin_role.resources[0], &#34;*&#34;); \/\/ All resources\n\n Role dev_role = {\n .role_name = &#34;Developer&#34;,\n .permissions = PERM_READ | PERM_WRITE | PERM_EXECUTE,\n .num_resources = 2\n };\n strcpy(dev_role.resources[0], &#34;\/code\/*&#34;);\n strcpy(dev_role.resources[1], &#34;\/docs\/*&#34;);\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"164-encryption-basics\"><span>16.4 Encryption Basics<\/span>\n <a href=\"#164-encryption-basics\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/encryption-symmetric-asymmetric.webp\" title=\"Symmetric vs Asymmetric Encryption\" data-thumbnail=\"\/files\/encryption-symmetric-asymmetric.webp\" data-sub-html=\"<h2>Encryption Symmetric and Asymmetric cryptography comparison showing key usage for data protection<\/h2><p>Symmetric vs Asymmetric Encryption<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/encryption-symmetric-asymmetric.webp' alt=\"Encryption Symmetric and Asymmetric cryptography comparison showing key usage for data protection\"><\/a><figcaption class=\"image-caption\">Symmetric vs Asymmetric Encryption<\/figcaption>\n <\/figure><\/p>\n<p><em>Figure 11: Encryption fundamentals comparing symmetric encryption (same key for encrypt\/decrypt) and asymmetric encryption (public key for encryption, private key for decryption).<\/em><\/p>\n<pre><code>\/*\n * Encryption: Transform data to unreadable form\n * Symmetric: Same key for encrypt\/decrypt (AES, DES)\n * Asymmetric: Public\/Private key pair (RSA, ECC)\n *\/\n\n#include &lt;openssl\/aes.h&gt;\n#include &lt;openssl\/rsa.h&gt;\n\n\/\/ Symmetric encryption (AES example)\nvoid aes_encrypt(const unsigned char *plaintext, size_t len,\n const unsigned char *key,\n unsigned char *ciphertext) {\n AES_KEY aes_key;\n AES_set_encrypt_key(key, 256, &amp;aes_key);\n\n \/\/ Encrypt in ECB mode (simplified - use CBC\/GCM in production)\n for (size_t i = 0; i &lt; len; i &#43;= 16) {\n AES_encrypt(plaintext &#43; i, ciphertext &#43; i, &amp;aes_key);\n }\n}\n\nvoid aes_decrypt(const unsigned char *ciphertext, size_t len,\n const unsigned char *key,\n unsigned char *plaintext) {\n AES_KEY aes_key;\n AES_set_decrypt_key(key, 256, &amp;aes_key);\n\n for (size_t i = 0; i &lt; len; i &#43;= 16) {\n AES_decrypt(ciphertext &#43; i, plaintext &#43; i, &amp;aes_key);\n }\n}\n\n\/\/ Hash functions (one-way)\n#include &lt;openssl\/sha.h&gt;\n\nvoid hash_password(const char *password, unsigned char *hash) {\n SHA256_CTX ctx;\n SHA256_Init(&amp;ctx);\n SHA256_Update(&amp;ctx, password, strlen(password));\n SHA256_Final(hash, &amp;ctx);\n}\n\n\/\/ Example: Secure file storage\nvoid secure_file_write(const char *filename, const unsigned char *data,\n size_t len, const unsigned char *key) {\n \/\/ Encrypt data\n unsigned char *encrypted = malloc(len);\n aes_encrypt(data, len, key, encrypted);\n\n \/\/ Write to file\n FILE *f = fopen(filename, &#34;wb&#34;);\n fwrite(encrypted, 1, len, f);\n fclose(f);\n\n free(encrypted);\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"165-common-security-vulnerabilities\"><span>16.5 Common Security Vulnerabilities<\/span>\n <a href=\"#165-common-security-vulnerabilities\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/security-vulnerabilities.webp\" title=\"Common OS Security Vulnerabilities\" data-thumbnail=\"\/files\/security-vulnerabilities.webp\" data-sub-html=\"<h2>Common Security Vulnerabilities taxonomy including buffer overflow, SQL injection, race conditions, and privilege escalation attacks<\/h2><p>Common OS Security Vulnerabilities<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/security-vulnerabilities.webp' alt=\"Common Security Vulnerabilities taxonomy including buffer overflow, SQL injection, race conditions, and privilege escalation attacks\"><\/a><figcaption class=\"image-caption\">Common OS Security Vulnerabilities<\/figcaption>\n <\/figure><\/p>\n<p><em>Figure 12: Taxonomy of common security vulnerabilities in operating systems including buffer overflows, SQL injection, race conditions (TOCTOU), and privilege escalation exploits.<\/em><\/p>\n<div class=\"details admonition danger open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-bolt\" aria-hidden=\"true\"><\/i>Danger<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Common Security Vulnerabilities - Critical Patterns to Avoid<\/strong><\/div>\n <\/div>\n<\/div>\n<pre><code>\/*\n * Common vulnerabilities and mitigations\n *\/\n\n\/\/ 1. Buffer Overflow\nvoid vulnerable_function(char *input) {\n char buffer[100];\n strcpy(buffer, input); \/\/ VULNERABLE! No bounds check\n}\n\nvoid safe_function(char *input) {\n char buffer[100];\n strncpy(buffer, input, sizeof(buffer) - 1); \/\/ Safe\n buffer[sizeof(buffer) - 1] = &#39;\\0&#39;;\n}\n\n\/\/ 2. Race Condition (TOCTOU - Time of Check, Time of Use)<\/code><\/pre><div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>TOCTOU Race Condition<\/strong>: Between checking file permissions (TIME OF CHECK) and opening the file (TIME OF USE), an attacker can replace the file with a malicious symlink! This is a classic security vulnerability exploited in privilege escalation attacks.<\/div>\n <\/div>\n<\/div>\n<pre><code>void vulnerable_file_access(const char *filename) {\n \/\/ Check if file is accessible\n if (access(filename, R_OK) == 0) { \/\/ TIME OF CHECK\n \/\/ Race condition window here!\n FILE *f = fopen(filename, &#34;r&#34;); \/\/ TIME OF USE\n \/\/ Attacker could replace file with symlink\n }\n}\n\nvoid safe_file_access(const char *filename) {\n \/\/ Open file and then check\n int fd = open(filename, O_RDONLY | O_NOFOLLOW); \/\/ No symlinks\n if (fd == -1) {\n return;\n }\n\n struct stat st;\n if (fstat(fd, &amp;st) == 0) {\n \/\/ Use file descriptor directly\n }\n close(fd);\n}\n\n\/\/ 3. Privilege Escalation Prevention\nvoid drop_privileges() {\n \/\/ Drop root privileges after initialization\n uid_t real_uid = getuid();\n gid_t real_gid = getgid();\n\n \/\/ Drop supplementary groups\n setgroups(0, NULL);\n\n \/\/ Set GID first (must be done before UID)\n setgid(real_gid);\n setuid(real_uid);\n\n \/\/ Verify privileges dropped\n if (geteuid() == 0 || getegid() == 0) {\n fprintf(stderr, &#34;Failed to drop privileges!\\n&#34;);\n exit(1);\n }\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"166-security-in-modern-os\"><span>16.6 Security in Modern OS<\/span>\n <a href=\"#166-security-in-modern-os\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/*\n * Modern OS Security Features\n *\/\n\n\/\/ ASLR (Address Space Layout Randomization)\n\/\/ Randomizes memory addresses to prevent exploits\nvoid demonstrate_aslr() {\n int stack_var;\n int *heap_var = malloc(sizeof(int));\n\n printf(&#34;Stack: %p\\n&#34;, &amp;stack_var);\n printf(&#34;Heap: %p\\n&#34;, heap_var);\n \/\/ Addresses will be different each run with ASLR\n}\n\n\/\/ NX\/DEP (No Execute \/ Data Execution Prevention)\n\/\/ Marks memory pages as non-executable\n\/\/ Prevents code injection attacks\n\n\/\/ Seccomp (Secure Computing Mode)\n\/\/ Restricts syscalls available to process\n#include &lt;sys\/prctl.h&gt;\n#include &lt;linux\/seccomp.h&gt;\n\nvoid enable_seccomp_strict() {\n \/\/ Allow only read, write, exit, sigreturn\n prctl(PR_SET_SECCOMP, SECCOMP_MODE_STRICT);\n\n \/\/ After this, any other syscall terminates process\n}\n\n\/\/ Namespaces (Isolation)\n\/\/ Isolate process resources (used in containers)\nvoid create_namespace() {\n \/\/ Create new PID namespace\n if (unshare(CLONE_NEWPID) == -1) {\n perror(&#34;unshare&#34;);\n return;\n }\n\n \/\/ Process now in isolated PID namespace\n \/\/ PID 1 in this namespace\n}<\/code><\/pre><h3 class=\"heading-element\" id=\"key-takeaways---security\"><span>Key Takeaways - Security<\/span>\n <a href=\"#key-takeaways---security\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>OS Security Fundamentals<\/strong>:<\/p>\n<ol>\n<li><strong>CIA Triad<\/strong>: Confidentiality, Integrity, Availability \u2192 Three pillars of security<\/li>\n<li><strong>Authentication<\/strong>: Verify identity (who you are) \u2192 Passwords, biometrics, MFA<\/li>\n<li><strong>Authorization<\/strong>: Check permissions (what you can do) \u2192 Access control models<\/li>\n<li><strong>DAC<\/strong>: Owner controls access (Unix permissions) \u2192 User-controlled, flexible<\/li>\n<li><strong>MAC<\/strong>: System enforces policy (SELinux) \u2192 Centralized, stronger security<\/li>\n<li><strong>RBAC<\/strong>: Role-based permissions (enterprise systems) \u2192 Scalable management<\/li>\n<li><strong>Encryption<\/strong>: Symmetric (fast), Asymmetric (secure key exchange) \u2192 AES-256, RSA-2048<\/li>\n<li><strong>Common vulnerabilities<\/strong>: Buffer overflow, race conditions, privilege escalation \u2192 Audit code regularly!<\/li>\n<\/ol><\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"unix-linux\">17. Unix\/Linux Deep Dive<\/h2>\n<h3 class=\"heading-element\" id=\"171-unixlinux-architecture\"><span>17.1 Unix\/Linux Architecture<\/span>\n <a href=\"#171-unixlinux-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/unix-linux-architecture.webp\" title=\"Unix\/Linux System Architecture Layers\" data-thumbnail=\"\/files\/unix-linux-architecture.webp\" data-sub-html=\"<h2>Unix Linux Operating System Architecture showing user space and kernel space layers with system call interface<\/h2><p>Unix\/Linux System Architecture Layers<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/unix-linux-architecture.webp' alt=\"Unix Linux Operating System Architecture showing user space and kernel space layers with system call interface\"><\/a><figcaption class=\"image-caption\">Unix\/Linux System Architecture Layers<\/figcaption>\n <\/figure><\/p>\n<p><em>Figure 13: Unix\/Linux operating system architecture illustrating user space (applications, shell, libraries) and kernel space (process management, memory management, file systems, device drivers, network stack) with system call interface as the boundary.<\/em><\/p>\n<pre><code># Unix\/Linux File System Hierarchy\n\/ # Root directory\n\u251c\u2500\u2500 bin\/ # Essential user binaries\n\u251c\u2500\u2500 boot\/ # Boot loader files\n\u251c\u2500\u2500 dev\/ # Device files\n\u251c\u2500\u2500 etc\/ # System configuration\n\u251c\u2500\u2500 home\/ # User home directories\n\u251c\u2500\u2500 lib\/ # System libraries\n\u251c\u2500\u2500 media\/ # Removable media mount points\n\u251c\u2500\u2500 mnt\/ # Temporary mount points\n\u251c\u2500\u2500 opt\/ # Optional software\n\u251c\u2500\u2500 proc\/ # Process information (virtual)\n\u251c\u2500\u2500 root\/ # Root user home\n\u251c\u2500\u2500 run\/ # Runtime data\n\u251c\u2500\u2500 sbin\/ # System binaries\n\u251c\u2500\u2500 srv\/ # Service data\n\u251c\u2500\u2500 sys\/ # Kernel and system info (virtual)\n\u251c\u2500\u2500 tmp\/ # Temporary files\n\u251c\u2500\u2500 usr\/ # User programs\n\u2502 \u251c\u2500\u2500 bin\/ # User binaries\n\u2502 \u251c\u2500\u2500 lib\/ # User libraries\n\u2502 \u251c\u2500\u2500 local\/ # Locally compiled software\n\u2502 \u2514\u2500\u2500 share\/ # Shared data\n\u2514\u2500\u2500 var\/ # Variable data\n \u251c\u2500\u2500 log\/ # Log files\n \u251c\u2500\u2500 mail\/ # Mail\n \u251c\u2500\u2500 spool\/ # Spool files\n \u2514\u2500\u2500 tmp\/ # Temporary files (persists across reboots)<\/code><\/pre><h3 class=\"heading-element\" id=\"172-boot-process\"><span>17.2 Boot Process<\/span>\n <a href=\"#172-boot-process\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/unix-boot-process.webp\" title=\"Linux Boot Sequence Diagram\" data-thumbnail=\"\/files\/unix-boot-process.webp\" data-sub-html=\"<h2>Unix Linux Boot Process Sequence from BIOS UEFI to Init showing bootloader, kernel initialization, and service startup<\/h2><p>Linux Boot Sequence Diagram<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/unix-boot-process.webp' alt=\"Unix Linux Boot Process Sequence from BIOS UEFI to Init showing bootloader, kernel initialization, and service startup\"><\/a><figcaption class=\"image-caption\">Linux Boot Sequence Diagram<\/figcaption>\n <\/figure><\/p>\n<p><em>Figure 14: Unix\/Linux boot process sequence diagram showing the complete boot chain from BIOS\/UEFI \u2192 Bootloader (GRUB) \u2192 Kernel \u2192 Init (PID 1) \u2192 System Services startup.<\/em><\/p>\n<pre><code># Boot process stages\n# 1. BIOS\/UEFI\n# - Power-On Self Test (POST)\n# - Initialize hardware\n# - Load bootloader\n\n# 2. Bootloader (GRUB)\n# - Present menu\n# - Load kernel and initramfs\n# - Pass kernel parameters\n\n# 3. Kernel initialization\n# - Decompress kernel\n# - Setup memory management\n# - Mount root filesystem\n# - Start init process\n\n# 4. Init system (systemd\/SysV)\n# - Run startup scripts\n# - Mount filesystems\n# - Start services\n\n# View boot messages\ndmesg | less\n\n# Check boot time\nsystemd-analyze\n\n# See boot sequence\nsystemd-analyze critical-chain<\/code><\/pre><h3 class=\"heading-element\" id=\"173-shell-and-command-line\"><span>17.3 Shell and Command Line<\/span>\n <a href=\"#173-shell-and-command-line\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>#!\/bin\/bash\n\n# ===================\n# File Operations\n# ===================\n\n# List files\nls -la # Long format, all files\nls -lh # Human-readable sizes\nls -lt # Sort by modification time\nls -lS # Sort by size\n\n# Navigate directories\ncd \/path\/to\/dir # Change directory\ncd ~ # Go to home\ncd - # Go to previous directory\npwd # Print working directory\n\n# Copy, move, remove\ncp file1 file2 # Copy file\ncp -r dir1 dir2 # Copy directory recursively\nmv old new # Move\/rename\nrm file # Remove file\nrm -rf dir # Remove directory recursively (DANGEROUS!)\n\n# Create directories\nmkdir mydir # Create directory\nmkdir -p a\/b\/c # Create nested directories\n\n# File viewing\ncat file # Display entire file\nless file # Page through file\nhead -n 20 file # First 20 lines\ntail -n 20 file # Last 20 lines\ntail -f file # Follow file (for logs)\n\n# ===================\n# File Permissions\n# ===================\n\n# Permission format: rwxrwxrwx (owner group others)\n# r=4, w=2, x=1\n\n# Change permissions\nchmod 755 script.sh # rwxr-xr-x\nchmod u&#43;x script.sh # Add execute for owner\nchmod go-w file # Remove write for group\/others\n\n# Change ownership\nchown user:group file # Change owner and group\nchown -R user dir # Recursive\n\n# View permissions\nls -l file\nstat file # Detailed info\n\n# Special permissions\nchmod &#43;t dir # Sticky bit (only owner can delete files)\nchmod u&#43;s program # Setuid (run as file owner)\nchmod g&#43;s program # Setgid (run as file group)\n\n# ===================\n# Process Management\n# ===================\n\n# View processes\nps aux # All processes\nps -ef # All processes (different format)\nps -u username # User&#39;s processes\ntop # Interactive process viewer\nhtop # Better top (if installed)\n\n# Process tree\npstree # Show process hierarchy\npstree -p # Show PIDs\n\n# Find process\npgrep firefox # Find PID by name\npidof firefox # Same as pgrep\n\n# Kill process\nkill PID # Send SIGTERM\nkill -9 PID # Send SIGKILL (force)\nkillall firefox # Kill all firefox processes\npkill firefox # Kill by name\n\n# Background jobs\ncommand &amp; # Run in background\njobs # List background jobs\nfg %1 # Bring job 1 to foreground\nbg %1 # Resume job 1 in background\ndisown %1 # Detach job from shell\n\n# Process priority\nnice -n 10 command # Start with lower priority\nrenice -n 5 -p PID # Change priority of running process\n\n# ===================\n# Finding Files\n# ===================\n\n# Find command\nfind \/path -name &#34;*.txt&#34; # Find by name\nfind \/path -type f # Find files\nfind \/path -type d # Find directories\nfind \/path -size &#43;100M # Find large files\nfind \/path -mtime -7 # Modified in last 7 days\nfind \/path -user username # Find by owner\nfind \/path -perm 755 # Find by permissions\n\n# Execute command on found files\nfind \/path -name &#34;*.log&#34; -exec rm {} \\;\n\n# Locate (faster, uses database)\nlocate filename\nupdatedb # Update locate database\n\n# Which (find command in PATH)\nwhich python\ntype python # Show command type<\/code><\/pre><h3 class=\"heading-element\" id=\"174-text-processing\"><span>17.4 Text Processing<\/span>\n <a href=\"#174-text-processing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>#!\/bin\/bash\n\n# ===================\n# grep - Search text\n# ===================\n\n# Basic usage\ngrep &#34;pattern&#34; file # Search for pattern\ngrep -i &#34;pattern&#34; file # Case insensitive\ngrep -v &#34;pattern&#34; file # Invert match (show non-matching)\ngrep -r &#34;pattern&#34; dir # Recursive search\ngrep -n &#34;pattern&#34; file # Show line numbers\ngrep -c &#34;pattern&#34; file # Count matches\ngrep -l &#34;pattern&#34; files* # Show only filenames\ngrep -A 3 &#34;pattern&#34; file # Show 3 lines after match\ngrep -B 3 &#34;pattern&#34; file # Show 3 lines before match\ngrep -C 3 &#34;pattern&#34; file # Show 3 lines context\n\n# Regular expressions\ngrep &#34;^start&#34; file # Lines starting with &#34;start&#34;\ngrep &#34;end$&#34; file # Lines ending with &#34;end&#34;\ngrep &#34;[0-9]&#43;&#34; file # Lines with numbers\ngrep -E &#34;word1|word2&#34; file # Extended regex (OR)\n\n# Practical examples\nps aux | grep firefox # Find firefox processes\ngrep -r &#34;TODO&#34; *.c # Find TODOs in C files\ngrep -v &#34;^#&#34; config.conf # Show non-comment lines\n\n# ===================\n# awk - Text processing\n# ===================\n\n# Print columns\nawk &#39;{print $1}&#39; file # Print first column\nawk &#39;{print $1, $3}&#39; file # Print columns 1 and 3\nawk &#39;{print $NF}&#39; file # Print last column\n\n# Field separator\nawk -F: &#39;{print $1}&#39; \/etc\/passwd # Use : as separator\n\n# Conditions\nawk &#39;$3 &gt; 100&#39; file # Print lines where column 3 &gt; 100\nawk &#39;$1 == &#34;root&#34;&#39; file # Print lines where column 1 is &#34;root&#34;\n\n# Built-in variables\nawk &#39;{print NR, $0}&#39; file # NR = line number\nawk &#39;NF &gt; 5&#39; file # NF = number of fields\n\n# Calculations\nawk &#39;{sum &#43;= $1} END {print sum}&#39; file # Sum column 1\nawk &#39;{print $1 * 2}&#39; file # Multiply column 1 by 2\n\n# Practical examples\nps aux | awk &#39;$3 &gt; 50&#39; # Processes using &gt;50% CPU\ndf -h | awk &#39;$5 &gt; 80&#39; # Partitions &gt;80% full\nawk -F: &#39;{print $1}&#39; \/etc\/passwd # List usernames\n\n# ===================\n# sed - Stream editor\n# ===================\n\n# Substitution\nsed &#39;s\/old\/new\/&#39; file # Replace first occurrence\nsed &#39;s\/old\/new\/g&#39; file # Replace all occurrences\nsed &#39;s\/old\/new\/gi&#39; file # Case insensitive\nsed &#39;5s\/old\/new\/&#39; file # Replace in line 5\nsed -i &#39;s\/old\/new\/g&#39; file # Edit file in-place\n\n# Delete lines\nsed &#39;5d&#39; file # Delete line 5\nsed &#39;5,10d&#39; file # Delete lines 5-10\nsed &#39;\/pattern\/d&#39; file # Delete lines matching pattern\n\n# Insert\/Append\nsed &#39;5i\\New Line&#39; file # Insert before line 5\nsed &#39;5a\\New Line&#39; file # Append after line 5\n\n# Print lines\nsed -n &#39;5p&#39; file # Print line 5\nsed -n &#39;5,10p&#39; file # Print lines 5-10\nsed -n &#39;\/pattern\/p&#39; file # Print matching lines\n\n# Practical examples\nsed &#39;s\/#.*$\/\/&#39; config.conf # Remove comments\nsed &#39;\/^$\/d&#39; file # Remove empty lines\nsed &#39;s\/\\s\\&#43;$\/\/&#39; file # Remove trailing whitespace\n\n# ===================\n# sort, uniq, cut\n# ===================\n\n# Sort\nsort file # Sort alphabetically\nsort -n file # Sort numerically\nsort -r file # Reverse sort\nsort -k 2 file # Sort by column 2\nsort -u file # Sort and remove duplicates\n\n# Unique\nuniq file # Remove adjacent duplicates\nuniq -c file # Count occurrences\nuniq -d file # Show only duplicates\nsort file | uniq # Remove all duplicates\n\n# Cut\ncut -d: -f1 \/etc\/passwd # Extract field 1 (delimiter :)\ncut -c1-10 file # Extract characters 1-10\n\n# ===================\n# Combining commands\n# ===================\n\n# Count lines in all .c files\nfind . -name &#34;*.c&#34; | xargs wc -l\n\n# Find most common words in file\ncat file | tr &#39;[:upper:]&#39; &#39;[:lower:]&#39; | tr -s &#39; &#39; &#39;\\n&#39; | sort | uniq -c | sort -rn | head -10\n\n# Find largest files\nfind \/ -type f -exec du -h {} &#43; | sort -rh | head -10\n\n# Monitor log file for errors\ntail -f \/var\/log\/syslog | grep --line-buffered ERROR\n\n# Count unique IP addresses in log\nawk &#39;{print $1}&#39; access.log | sort | uniq -c | sort -rn<\/code><\/pre><h3 class=\"heading-element\" id=\"key-takeaways---unixlinux\"><span>Key Takeaways - Unix\/Linux<\/span>\n <a href=\"#key-takeaways---unixlinux\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Unix\/Linux Command Line Mastery<\/strong>:<\/p>\n<ol>\n<li><strong>File System Hierarchy<\/strong>: Standard directory structure (FHS) \u2192 \/bin, \/etc, \/home, \/var, etc.<\/li>\n<li><strong>Boot Process<\/strong>: BIOS\/UEFI \u2192 Bootloader \u2192 Kernel \u2192 Init \u2192 Systemd \u2192 Services<\/li>\n<li><strong>File Permissions<\/strong>: Read (4), Write (2), Execute (1) \u2192 chmod 755 = rwxr-xr-x<\/li>\n<li><strong>Process Management<\/strong>: ps, top, kill, nice for process control \u2192 Monitor and control processes<\/li>\n<li><strong>Text Processing<\/strong>: grep (search), awk (columns), sed (edit), sort\/uniq \u2192 Powerful text manipulation<\/li>\n<li><strong>Command Piping<\/strong>: Chain commands with | for powerful workflows \u2192 ls | grep pattern | sort | uniq<\/li>\n<\/ol><\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"conclusion\">Conclusion<\/h2>\n<p>This comprehensive operating systems tutorial covers the essential concepts needed for:<\/p>\n<ul>\n<li><strong>Placement interviews<\/strong> at top tech companies<\/li>\n<li><strong>Competitive programming<\/strong> understanding of system-level concepts<\/li>\n<li><strong>Kernel development<\/strong> basics<\/li>\n<li><strong>Systems programming<\/strong> foundation<\/li>\n<li><strong>Performance optimization<\/strong> insights<\/li>\n<\/ul>\n<p>Key Areas Covered:<\/p>\n<ol>\n<li>Process and thread management with scheduling<\/li>\n<li>Memory management (physical and virtual)<\/li>\n<li>File systems and I\/O operations<\/li>\n<li>Synchronization and deadlock handling<\/li>\n<li>System calls and interrupts<\/li>\n<li>Inter-process communication<\/li>\n<li>Security and protection mechanisms<\/li>\n<li>Real-world Unix\/Linux concepts<\/li>\n<\/ol>\n<p><strong>Study Tips:<\/strong><\/p>\n<ol>\n<li>Understand concepts deeply, not just memorize<\/li>\n<li>Practice implementing algorithms (scheduling, page replacement, etc.)<\/li>\n<li>Run code examples and modify them<\/li>\n<li>Trace through syscall execution mentally<\/li>\n<li>Review code snippets regularly<\/li>\n<li>Connect theory to real systems (Linux, Windows)<\/li>\n<li>Practice interview questions about tradeoffs and design decisions<\/li>\n<\/ol>\n<p><strong>Interview Focus:<\/strong><\/p>\n<ul>\n<li>Be ready to discuss tradeoffs (performance vs correctness)<\/li>\n<li>Know when to use which algorithm (FCFS vs LRU vs SCAN)<\/li>\n<li>Understand context switches and overhead<\/li>\n<li>Explain concepts with examples<\/li>\n<li>Connect multiple topics (e.g., how paging affects scheduling)<\/li>\n<\/ul>\n<p>Good luck with your OS learning journey!<\/p>\n<hr>\n<h2 id=\"references\">References and Further Reading<\/h2>\n<h3 class=\"heading-element\" id=\"books\"><span>Books<\/span>\n <a href=\"#books\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ol>\n<li><strong>&ldquo;Operating Systems: Three Easy Pieces&rdquo;<\/strong> by Remzi H. Arpaci-Dusseau and Andrea C. Arpaci-Dusseau<\/li>\n<li><strong>&ldquo;Modern Operating Systems&rdquo;<\/strong> by Andrew S. Tanenbaum and Herbert Bos<\/li>\n<li><strong>&ldquo;Operating System Concepts&rdquo;<\/strong> by Abraham Silberschatz, Peter Baer Galvin, and Greg Gagne<\/li>\n<li><strong>&ldquo;Linux Kernel Development&rdquo;<\/strong> by Robert Love<\/li>\n<li><strong>&ldquo;The Design of the UNIX Operating System&rdquo;<\/strong> by Maurice J. Bach<\/li>\n<li><strong>&ldquo;Advanced Programming in the UNIX Environment&rdquo;<\/strong> by Stevens &amp; Rago<\/li>\n<\/ol>\n<h3 class=\"heading-element\" id=\"online-resources\"><span>Online Resources<\/span>\n <a href=\"#online-resources\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ul>\n<li><a href=\"https:\/\/wiki.osdev.org\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">OSDev Wiki<\/a> - Comprehensive OS development resource<\/li>\n<li><a href=\"https:\/\/www.kernel.org\/doc\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Linux Kernel Documentation<\/a><\/li>\n<li><a href=\"https:\/\/developer.arm.com\/documentation\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">ARM Architecture Reference Manual<\/a><\/li>\n<li><a href=\"https:\/\/software.intel.com\/content\/www\/us\/en\/develop\/articles\/intel-sdm.html\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Intel Software Developer Manuals<\/a><\/li>\n<li><a href=\"https:\/\/rust-embedded.github.io\/book\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Rust Embedded Book<\/a><\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"tutorial-series\"><span>Tutorial Series<\/span>\n <a href=\"#tutorial-series\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ul>\n<li>cfenollosa&rsquo;s OS Tutorial (x86 bare metal)<\/li>\n<li>Raspberry Pi OS Learning Repository<\/li>\n<li>Rust Raspberry Pi OS Tutorials<\/li>\n<li>MIT 6.828: Operating System Engineering<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"online-courses\"><span>Online Courses<\/span>\n <a href=\"#online-courses\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ul>\n<li>UC Berkeley CS162: Operating Systems<\/li>\n<li>MIT OpenCourseWare: Operating Systems<\/li>\n<li>Linux Foundation Training<\/li>\n<li>CS162 (Berkeley): Operating Systems<\/li>\n<li>MIT 6.S081: Operating System Engineering<\/li>\n<li>CS140 (Stanford): Operating Systems<\/li>\n<\/ul>\n<hr>\n<p><strong>Note<\/strong>: This tutorial is designed for educational purposes, drawing from multiple sources and synthesizing concepts in an original manner. All code examples are simplified for instructional clarity and may require additional error handling and optimization for production use.<\/p>\n<hr>\n<p><em>This comprehensive tutorial provides theoretical foundations, practical implementation details, code examples, and real-world applications to ensure deep understanding of operating system concepts.<\/em><\/p>\n"},{"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","link":"https:\/\/samir.pages.dev\/posts\/vim\/","pubDate":"Fri, 22 Nov 2024 00:00:00 +0000","author":"netproxy@duck.com (Samir)","guid":"https:\/\/samir.pages.dev\/posts\/vim\/","category":"Software-Engineering","description":"<img src=\"https:\/\/samir.pages.dev\/files\/vim.webp\" alt=\"featured image\" referrerpolicy=\"no-referrer\"><p><strong>Master the World&rsquo;s Most Powerful Text Editor \u2014 From Beginner to Expert<\/strong><\/p>\n<h2 id=\"introduction\">Introduction to Vim<\/h2>\r\n<div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Why Learn Vim?<\/strong><\/p>\n<p>Vim is not just a text editor\u2014it&rsquo;s a philosophy of efficient editing. Once you understand Vim&rsquo;s core concepts, you can edit text <strong>5-10x faster<\/strong> than traditional editors. Used by developers, system administrators, and DevOps professionals worldwide.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<p>Vim is a highly configurable <strong>modal text editor<\/strong> built on top of Vi (Unix&rsquo;s classic text editor from 1976). It&rsquo;s lightweight, powerful, and available on virtually every Unix-like system, making it the de facto standard for command-line editing.<\/p>\n<p><strong>Key Statistics:<\/strong><\/p>\n<ul>\n<li><strong>Age<\/strong>: First released in 1991 (34 years of continuous development)<\/li>\n<li><strong>Popularity<\/strong>: Used by millions of developers and system administrators<\/li>\n<li><strong>Learning Curve<\/strong>: Steep initially, exponential productivity gains<\/li>\n<li><strong>Extensibility<\/strong>: Fully programmable via Vimscript or Lua<\/li>\n<li><strong>Community<\/strong>: Massive ecosystem with thousands of plugins<\/li>\n<\/ul>\n<hr>\n<h2 id=\"vim-philosophy\">The Vim Philosophy: Modal Editing<\/h2>\r\n<div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Modal Editing Paradigm<\/strong><\/p>\n<p>Most editors use a single mode where keys always insert text. Vim separates concerns into distinct modes:<\/p>\n<ul>\n<li><strong>Normal Mode<\/strong>: Navigation and commands (your home base)<\/li>\n<li><strong>Insert Mode<\/strong>: Text insertion (like traditional editors)<\/li>\n<li><strong>Visual Mode<\/strong>: Text selection and manipulation<\/li>\n<li><strong>Command Mode<\/strong>: Advanced operations and configuration<\/li>\n<\/ul>\n<\/div>\n <\/div>\n<\/div>\n<p>This separation creates <strong>logical separation of concerns<\/strong>, making complex text operations intuitive once you understand the grammar.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/vim-modes-diagram.webp\" title=\"Vim Modes \u2014 Normal, Insert, Visual, and Command modes; understanding these modes is fundamental to Vim mastery\" data-thumbnail=\"\/files\/vim-modes-diagram.webp\" data-sub-html=\"<h2>Vim modal editing paradigm showing mode transitions \u2014 the core philosophy enabling efficient text manipulation<\/h2><p>Vim Modes \u2014 Normal, Insert, Visual, and Command modes; understanding these modes is fundamental to Vim mastery<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/vim-modes-diagram.webp' alt=\"Vim modal editing paradigm showing mode transitions \u2014 the core philosophy enabling efficient text manipulation\"><\/a><figcaption class=\"image-caption\">Vim Modes \u2014 Normal, Insert, Visual, and Command modes; understanding these modes is fundamental to Vim mastery<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"the-vim-grammar-operator--motion\"><span>The Vim Grammar: Operator + Motion<\/span>\n <a href=\"#the-vim-grammar-operator--motion\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Vim&rsquo;s Killer Feature: Text Objects + Motions = Composable Editing<\/strong><\/p>\n<p>Vim operates on a grammar: <code>operator<\/code> + <code>motion<\/code> + (optionally) <code>text-object<\/code><\/p>\n<p>Examples:<\/p>\n<ul>\n<li><code>dw<\/code> = Delete word (delete + word motion)<\/li>\n<li><code>c5j<\/code> = Change 5 lines down (change + 5-line motion)<\/li>\n<li><code>dip<\/code> = Delete paragraph (delete + paragraph text object)<\/li>\n<\/ul>\n<p>This composability means once you learn operators and motions, you can combine them infinitely!<\/p>\n<\/div>\n <\/div>\n<\/div>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/vim-operator-grammar.webp\" title=\"Vim Operator Grammar - The fundamental grammar combining operators (d\/c\/y), counts, and motions to create powerful editing commands\" data-thumbnail=\"\/files\/vim-operator-grammar.webp\" data-sub-html=\"<h2>Vim Grammar Formula: Operator &#43; Motion &#43; Text Object<\/h2><p>Vim Operator Grammar - The fundamental grammar combining operators (d\/c\/y), counts, and motions to create powerful editing commands<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/vim-operator-grammar.webp' alt=\"Vim Grammar Formula: Operator &#43; Motion &#43; Text Object\"><\/a><figcaption class=\"image-caption\">Vim Operator Grammar - The fundamental grammar combining operators (d\/c\/y), counts, and motions to create powerful editing commands<\/figcaption>\n <\/figure><\/p>\n<hr>\n<h2 id=\"installation-setup\">Installation and Setup<\/h2>\r\n<h3 class=\"heading-element\" id=\"installing-vim\"><span>Installing Vim<\/span>\n <a href=\"#installing-vim\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Ubuntu\/Debian\r\nsudo apt update &amp;&amp; sudo apt install vim\r\n\r\n# CentOS\/RHEL\/Fedora (7)\r\nsudo yum install vim\r\n\r\n# CentOS\/RHEL\/Fedora (8&#43;)\r\nsudo dnf install vim\r\n\r\n# macOS\r\nbrew install vim\r\n# Or for newer Neovim (modern alternative)\r\nbrew install neovim\r\n\r\n# Windows (using Chocolatey)\r\nchoco install vim\r\n# Or download from https:\/\/www.vim.org\/download.php<\/code><\/pre><h3 class=\"heading-element\" id=\"verify-installation\"><span>Verify Installation<\/span>\n <a href=\"#verify-installation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Check version and features\r\nvim --version\r\n\r\n# Output should show: VIM - Vi IMproved and version number\r\n# Look for these important features (&#43;):\r\n# &#43;python, &#43;python3 (for plugins)\r\n# &#43;clientserver (for remote editing)\r\n# &#43;clipboard (for system clipboard)<\/code><\/pre><h3 class=\"heading-element\" id=\"initial-configuration\"><span>Initial Configuration<\/span>\n <a href=\"#initial-configuration\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Create a basic <code>.vimrc<\/code> file in your home directory:<\/p>\n<pre><code>&#34; Basic Vim Configuration\r\n&#34; This is your Vim initialization file\r\n\r\n&#34; Enable syntax highlighting\r\nsyntax on\r\n\r\n&#34; Use 24-bit colors\r\nset termguicolors\r\n\r\n&#34; Line numbering\r\nset number\r\nset relativenumber\r\n\r\n&#34; Indentation\r\nset tabstop=4 &#34; Tab width\r\nset shiftwidth=4 &#34; Indent width\r\nset expandtab &#34; Use spaces instead of tabs\r\nset autoindent &#34; Auto-indent new lines\r\n\r\n&#34; Search behavior\r\nset ignorecase &#34; Ignore case in searches\r\nset smartcase &#34; But match case if uppercase is used\r\nset incsearch &#34; Highlight search as you type\r\n\r\n&#34; UI Enhancements\r\nset statusline=%F%m%r%h%w\\ [%{&amp;ff}]\\ %y\\ [%p%%]\\ %l:%c\r\nset laststatus=2 &#34; Always show status bar\r\nset cursorline &#34; Highlight current line\r\nset showmatch &#34; Highlight matching brackets\r\n\r\n&#34; Performance\r\nset nocompatible &#34; Use Vim defaults, not Vi\r\nset lazyredraw &#34; Don&#39;t redraw during macros<\/code><\/pre><hr>\n<h2 id=\"vim-modes\">Understanding Vim Modes<\/h2>\r\n<h3 class=\"heading-element\" id=\"mode-transitions\"><span>Mode Transitions<\/span>\n <a href=\"#mode-transitions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Vim operates in distinct modes, each optimized for different tasks. Understanding how to transition between modes is essential:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/vim-modes-diagram.webp\" title=\"Vim Modes - Illustrating transitions between Normal Mode (navigation), Insert Mode (text entry), Visual Mode (selection), and Command Mode\" data-thumbnail=\"\/files\/vim-modes-diagram.webp\" data-sub-html=\"<h2>Vim Mode Transitions Diagram<\/h2><p>Vim Modes - Illustrating transitions between Normal Mode (navigation), Insert Mode (text entry), Visual Mode (selection), and Command Mode<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/vim-modes-diagram.webp' alt=\"Vim Mode Transitions Diagram\"><\/a><figcaption class=\"image-caption\">Vim Modes - Illustrating transitions between Normal Mode (navigation), Insert Mode (text entry), Visual Mode (selection), and Command Mode<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"1-normal-mode-command-mode\"><span>1. Normal Mode (Command Mode)<\/span>\n <a href=\"#1-normal-mode-command-mode\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Default mode for navigation and commands.<\/strong><\/p>\n<pre><code>&#34; Navigation\r\nh, j, k, l = Left, Down, Up, Right (arrow keys equivalent)\r\nw, W = Next word, next WORD\r\nb, B = Previous word, previous WORD\r\n0, $ = Start of line, end of line\r\ngg, G = Start of file, end of file\r\n{, } = Previous\/next paragraph\r\n\r\n&#34; Editing\r\nx, X = Delete character under\/before cursor\r\ndd = Delete entire line\r\nyy = Yank (copy) entire line\r\np, P = Paste after\/before cursor\r\nu = Undo\r\nCTRL-R = Redo\r\n\r\n&#34; Operators\r\nd = Delete\r\nc = Change (delete and enter insert mode)\r\ny = Yank (copy)<\/code><\/pre><h3 class=\"heading-element\" id=\"2-insert-mode\"><span>2. Insert Mode<\/span>\n <a href=\"#2-insert-mode\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>For typing text. Behaves like traditional editors.<\/strong><\/p>\n<pre><code>i = Insert at cursor position\r\nI = Insert at line beginning\r\na = Append after cursor\r\nA = Append at line end\r\no = Open new line below\r\nO = Open new line above\r\ns = Substitute character (delete char, enter insert mode)\r\nS = Substitute entire line\r\n\r\n&#34; While in Insert Mode, press ESC to return to Normal Mode\r\n&#34; Or use CTRL-C as an alternative<\/code><\/pre><h3 class=\"heading-element\" id=\"3-visual-mode\"><span>3. Visual Mode<\/span>\n <a href=\"#3-visual-mode\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Select text for operations.<\/strong><\/p>\n<pre><code>v = Character-wise visual mode (select characters)\r\nV = Line-wise visual mode (select entire lines)\r\nCTRL-V = Block visual mode (select rectangular block)\r\n\r\n&#34; In visual mode:\r\nd = Delete selection\r\ny = Yank selection\r\nc = Change selection\r\nu = Convert to lowercase\r\nU = Convert to uppercase<\/code><\/pre><h3 class=\"heading-element\" id=\"4-command-mode-ex-commands\"><span>4. Command Mode (Ex Commands)<\/span>\n <a href=\"#4-command-mode-ex-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Execute commands using the colon (<code>:<\/code>) prefix.<\/strong><\/p>\n<pre><code>:w = Save file\r\n:q = Quit\r\n:wq = Save and quit\r\n:e filename = Open file\r\n:set option=val = Set configuration\r\n:%s\/old\/new\/g = Replace all occurrences\r\n:10,20d = Delete lines 10-20<\/code><\/pre><hr>\n<h2 id=\"essential-motions\">Essential Motions and Movements<\/h2>\r\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Master These Motions First<\/strong><\/p>\n<p>Motions are the &ldquo;nouns&rdquo; in Vim&rsquo;s grammar. Combine them with operators to create powerful commands. These are your most-used building blocks.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"character-level-motions\"><span>Character-Level Motions<\/span>\n <a href=\"#character-level-motions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>h = One character left\r\nl = One character right\r\nj = One line down\r\nk = One line up\r\n\r\nfx = Find character &#39;x&#39; on this line (forward)\r\nFx = Find character &#39;x&#39; on this line (backward)\r\ntx = Until character &#39;x&#39; (forward, stops before)\r\nTx = Until character &#39;x&#39; (backward, stops before)\r\n; = Repeat last f\/F\/t\/T motion\r\n, = Repeat last f\/F\/t\/T motion backwards\r\n\r\n% = Jump to matching bracket\/parenthesis<\/code><\/pre><h3 class=\"heading-element\" id=\"word-motions\"><span>Word Motions<\/span>\n <a href=\"#word-motions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>w = Next word (beginning of next word)\r\nW = Next WORD (separated by whitespace only)\r\ne = End of current word\r\nE = End of current WORD\r\nb = Beginning of previous word\r\nB = Beginning of previous WORD\r\n\r\n&#34; Difference between word and WORD:\r\n&#34; word splits on whitespace and punctuation\r\n&#34; WORD splits on whitespace only\r\n\r\nExample: foo-bar-baz\r\nword motions: foo | -bar | -baz\r\nWORD motions: foo-bar-baz (entire thing is one WORD)<\/code><\/pre><h3 class=\"heading-element\" id=\"line-motions\"><span>Line Motions<\/span>\n <a href=\"#line-motions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>0 = Start of line\r\n^ = First non-whitespace character\r\n$ = End of line\r\ng_ = Last non-whitespace character\r\n_ = Absolute beginning of line (vim-specific)<\/code><\/pre><h3 class=\"heading-element\" id=\"paragraph-and-block-motions\"><span>Paragraph and Block Motions<\/span>\n <a href=\"#paragraph-and-block-motions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>{ = Previous paragraph\r\n} = Next paragraph\r\n( = Previous sentence\r\n) = Next sentence\r\n[{ = Previous opening brace\r\n]} = Next closing brace<\/code><\/pre><h3 class=\"heading-element\" id=\"file-motions\"><span>File Motions<\/span>\n <a href=\"#file-motions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>gg = Beginning of file\r\nG = End of file\r\n:50&lt;Enter&gt; = Go to line 50\r\n50G = Go to line 50 (from normal mode)\r\nCTRL-G = Show current position in file\r\n&lt;number&gt;% = Go to percentage of file (50% = halfway)<\/code><\/pre><hr>\n<h2 id=\"operators\">Core Operators<\/h2>\r\n<h3 class=\"heading-element\" id=\"essential-operators\"><span>Essential Operators<\/span>\n <a href=\"#essential-operators\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>d (delete) Delete motion and remove from buffer\r\nc (change) Delete motion and enter insert mode\r\ny (yank) Copy motion to register\r\nv (visual) Select motion character-wise\r\nV (visual) Select motion line-wise<\/code><\/pre><h3 class=\"heading-element\" id=\"operator-examples\"><span>Operator Examples<\/span>\n <a href=\"#operator-examples\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>dw = Delete from cursor to end of word\r\nd5w = Delete 5 words\r\ndd = Delete entire line\r\nd$ = Delete from cursor to end of line\r\ndip = Delete entire paragraph\r\n\r\ncw = Change word (delete word, enter insert mode)\r\nC = Change to end of line (shortcut for c$)\r\ncc = Change entire line (shortcut for S)\r\n\r\nyy = Copy entire line\r\ny$ = Copy from cursor to end of line\r\nynw = Copy next word<\/code><\/pre><hr>\n<h2 id=\"text-objects\">Text Objects: The Power Multiplier<\/h2>\r\n<div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Text Objects are Game Changers<\/strong><\/p>\n<p>Text objects let you operate on semantic units (words, sentences, paragraphs, quotes, brackets) rather than just character counts. This is what makes Vim editing feel &ldquo;smart.&rdquo;<\/p>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"text-object-syntax\"><span>Text Object Syntax<\/span>\n <a href=\"#text-object-syntax\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>operator &#43; a\/i &#43; object\r\n \u2191 \u2191\r\n Action Type\r\n\r\na = around (include delimiters)\r\ni = inner (exclude delimiters)<\/code><\/pre><h3 class=\"heading-element\" id=\"common-text-objects\"><span>Common Text Objects<\/span>\n <a href=\"#common-text-objects\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Word objects\r\niw = inner word (just the word)\r\naw = a word (word &#43; surrounding whitespace)\r\n\r\n&#34; WORD objects\r\niW = inner WORD\r\naW = a WORD\r\n\r\n&#34; Paragraph\r\nip = inner paragraph\r\nap = a paragraph\r\n\r\n&#34; Quoted strings\r\ni&#39;, i&#34;, i` = inside single, double, backticks\r\na&#39;, a&#34;, a` = around quotes\r\n\r\n&#34; Brackets and parentheses\r\ni(, i), ib = inside parentheses\r\na(, a), ab = around parentheses\r\ni{, i}, iB = inside braces\r\na{, a}, aB = around braces\r\ni[, i] = inside brackets\r\na[, a] = around brackets\r\ni&lt;, i&gt; = inside angle brackets\r\na&lt;, a&gt; = around angle brackets\r\n\r\n&#34; Tags (for HTML\/XML)\r\nit = inner tag\r\nat = a tag<\/code><\/pre><h3 class=\"heading-element\" id=\"real-world-text-object-examples\"><span>Real-World Text Object Examples<\/span>\n <a href=\"#real-world-text-object-examples\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>diw = Delete inside word (replace word easily)\r\nci&#34; = Change inside quotes (edit string content)\r\nda{ = Delete around braces (remove function body)\r\nvip = Select inside paragraph (visual select)\r\nyip = Copy paragraph\r\ngUi( = Uppercase inside parentheses\r\n\r\n&#34; Example: Change function argument\r\nfunction(arg1, [cursor here], arg3)\r\n ci( \u2192 change inside parentheses\r\n \u2192 Now type new content, replaces (arg1, arg2, arg3)<\/code><\/pre><hr>\n<h2 id=\"search-replace\">Search and Replace<\/h2>\r\n<h3 class=\"heading-element\" id=\"search-operations\"><span>Search Operations<\/span>\n <a href=\"#search-operations\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>\/pattern = Search forward for pattern\r\n?pattern = Search backward for pattern\r\nn = Next match (forward)\r\nN = Previous match (backward)\r\n* = Search for word under cursor (forward)\r\n# = Search for word under cursor (backward)\r\n\r\n&#34; Repeat last search\r\n; (in f\/F\/t\/T) = Repeat last find\r\n\/ then Enter = Repeat last search pattern<\/code><\/pre><h3 class=\"heading-element\" id=\"search-configuration\"><span>Search Configuration<\/span>\n <a href=\"#search-configuration\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>set ignorecase = Ignore case when searching\r\nset smartcase = Override ignorecase if uppercase present\r\nset incsearch = Show matches as you type\r\nset hlsearch = Highlight all matches\r\n:nohlsearch = Clear highlighting (also :noh)<\/code><\/pre><h3 class=\"heading-element\" id=\"find-and-replace-substitute\"><span>Find and Replace (Substitute)<\/span>\n <a href=\"#find-and-replace-substitute\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Basic syntax: :[range]s\/pattern\/replacement\/[flags]\r\n\r\n:s\/old\/new\/ = Replace first occurrence on current line\r\n:s\/old\/new\/g = Replace all occurrences on current line\r\n:%s\/old\/new\/g = Replace all in entire file\r\n:%s\/old\/new\/gc = Replace all with confirmation\r\n:5,10s\/old\/new\/ = Replace on lines 5-10\r\n\r\n&#34; Common flags:\r\ng = Global (replace all on line, not just first)\r\nc = Confirm before each replacement\r\ni = Ignore case\r\ne = Don&#39;t show error if pattern not found\r\n\r\n&#34; Special characters in replacement:\r\n&amp; = Full match of pattern\r\n\\1, \\2, etc. = Captured groups (if using regex)\r\n~ = Previous replacement string<\/code><\/pre><h3 class=\"heading-element\" id=\"regular-expressions\"><span>Regular Expressions<\/span>\n <a href=\"#regular-expressions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Basic patterns\r\n. = Any character\r\n* = Zero or more of previous\r\n\\&#43; = One or more of previous\r\n\\? = Zero or one of previous\r\n^ = Start of line\r\n$ = End of line\r\n\\s = Whitespace\r\n\\w = Word character\r\n\\d = Digit\r\n\r\n&#34; Examples:\r\n:%s\/\\d\\&#43;\/NUM\/g = Replace all numbers with &#34;NUM&#34;\r\n:%s\/\\s&#43;$\/\/ = Remove trailing whitespace\r\n:%s\/^ \/\/ = Remove leading spaces\r\n:%s\/foo\\(.*\\)bar\/\\1\/ = Extract middle part\r\n\r\n&#34; Special: Use \\&lt; and \\&gt; for word boundaries\r\n:%s\/\\&lt;old\\&gt;\/new\/g = Replace whole word &#34;old&#34; only<\/code><\/pre><hr>\n<h2 id=\"registers\">Registers: Vim's Advanced Clipboard<\/h2>\r\n<div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Registers are Vim&rsquo;s Clipboard on Steroids<\/strong><\/p>\n<p>While most editors have one clipboard, Vim has 26+ registers! This lets you copy multiple things and paste any of them. Essential for advanced editing.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"register-types\"><span>Register Types<\/span>\n <a href=\"#register-types\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Named registers (a-z)\r\n&#34;a = Register a\r\n&#34;b = Register b\r\n(... up to &#34;z)\r\n\r\n&#34; Special registers\r\n&#34;0 = Last yank (copy)\r\n&#34;1 to &#34;9 = Last 9 deletions\r\n&#34;&#34;- = Last small deletion\r\n&#34;&#34;\/ = Last search pattern\r\n&#34;: = Last command\r\n&#34;% = Current filename\r\n&#34;# = Alternate filename\r\n&#34;&#43; = System clipboard\r\n&#34;* = Selection clipboard (X11)\r\n&#34;= = Expression register\r\n&#34;. = Last inserted text<\/code><\/pre><h3 class=\"heading-element\" id=\"using-registers\"><span>Using Registers<\/span>\n <a href=\"#using-registers\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Copying to registers\r\n&#34;ayy = Yank line into register a\r\n&#34;a5yy = Yank 5 lines into register a\r\n&#34;ad5w = Delete 5 words into register a\r\n\r\n&#34; Pasting from registers\r\n&#34;ap = Paste register a\r\n&#34;bp = Paste register b\r\n&#34;=5&#43;3&lt;Enter&gt;p = Paste result of expression (8)\r\n\r\n&#34; Inserting in Insert Mode\r\n&lt;CTRL-R&gt;a = Insert contents of register a while in insert mode\r\n&lt;CTRL-R&gt;=3*4&lt;Enter&gt; = Insert result of expression (12)\r\n\r\n&#34; Appending to registers (uppercase = append)\r\n&#34;Ayy = Append current line to register a\r\n&#34;Ad5w = Append 5 words to register a<\/code><\/pre><h3 class=\"heading-element\" id=\"practical-register-workflows\"><span>Practical Register Workflows<\/span>\n <a href=\"#practical-register-workflows\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Copy email for later use\r\n0\r\n&#34;ey$ = Copy email (from start of line to end) to register e\r\n\r\n&#34; Create template by copying structure\r\n&#34;tyy = Copy template line to register t\r\n3j = Move 3 lines down\r\n&#34;tp = Paste template 3 times\r\n\r\n&#34; System clipboard (most useful!)\r\n&#34;&#43;yy = Copy line to system clipboard\r\n&#34;&#43;p = Paste from system clipboard<\/code><\/pre><hr>\n<h2 id=\"macros\">Macros: Automating Complex Tasks<\/h2>\r\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Macros Let You Record and Replay Editing Sequences<\/strong><\/p>\n<p>Record a series of keystrokes and replay them. Perfect for repetitive tasks like formatting, bulk editing, or complex multi-step operations.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"recording-macros\"><span>Recording Macros<\/span>\n <a href=\"#recording-macros\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Recording\r\nq &#43; register = Start recording to register (q for quit recording)\r\nqa = Start recording to register &#39;a&#39;\r\n... (type edits)\r\nq = Stop recording (press q again)\r\n\r\n&#34; Playing\r\n@a = Play macro from register &#39;a&#39;\r\n3@a = Play macro 3 times\r\n@@ = Play last macro (convenient repeat)\r\n\r\n&#34; Editing macros\r\n&#34;ap = Paste macro into buffer (to view it)\r\n... (edit)\r\n&#34;ay$ = Copy edited version back to register\r\n\r\n&#34; Example: Record complex edit\r\nqa = Start recording\r\nI = Insert at beginning\r\n\/\/ TODO:\r\n&lt;ESC&gt; = Back to normal mode\r\nA (at end) = Append at end\r\n = Start recording again\r\nq = Stop recording\r\n\r\nNow &#34;@a&#34; will add &#34;\/\/ TODO:&#34; comments!<\/code><\/pre><h3 class=\"heading-element\" id=\"advanced-macro-techniques\"><span>Advanced Macro Techniques<\/span>\n <a href=\"#advanced-macro-techniques\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Macro on range (execute on lines 1-10)\r\n:1,10 normal @a\r\n\r\n&#34; Macro on all lines matching pattern\r\n:g\/pattern\/ normal @a\r\n\r\n&#34; Conditional macro execution\r\n:if condition\r\n: normal @a\r\n:endif\r\n\r\n&#34; Chain macros (macro calls another macro)\r\nqa = Start recording\r\n@b = Call macro b\r\n@c = Call macro c\r\nq = Stop recording<\/code><\/pre><h3 class=\"heading-element\" id=\"real-world-macro-example\"><span>Real-World Macro Example<\/span>\n <a href=\"#real-world-macro-example\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Original data:\r\nname:John,age:30,city:NYC\r\n\r\n&#34; Goal: Convert to JSON format\r\n{&#34;name&#34;:&#34;John&#34;,&#34;age&#34;:30,&#34;city&#34;:&#34;NYC&#34;}\r\n\r\n&#34; Macro steps:\r\nqa = Start recording in register a\r\n^ = Go to start of line\r\ni{ = Insert {\r\n&lt;ESC&gt; = Return to normal mode\r\n:s\/:\/ \/g = Replace colons with spaces\r\n:s\/,\/&#34;, &#34;\/g = Replace commas with quotes\r\n... more edits\r\n$ = Go to end\r\na} = Append }\r\n&lt;ESC&gt; = Done\r\nq = Stop recording\r\n\r\n@a = Execute macro<\/code><\/pre><hr>\n<h2 id=\"visual-mode\">Visual Mode and Selection<\/h2>\r\n<h3 class=\"heading-element\" id=\"visual-mode-types\"><span>Visual Mode Types<\/span>\n <a href=\"#visual-mode-types\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>v = Character-wise visual mode\r\n Select individual characters\r\n\r\nV = Line-wise visual mode\r\n Select entire lines (most intuitive)\r\n\r\nCTRL-V = Block visual mode\r\n Select rectangular blocks (powerful!)<\/code><\/pre><h3 class=\"heading-element\" id=\"visual-mode-operations\"><span>Visual Mode Operations<\/span>\n <a href=\"#visual-mode-operations\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Selection in visual mode\r\nv \/ V \/ CTRL-V = Activate mode\r\nh,j,k,l = Expand\/contract selection\r\nw,b,e = Move by words\r\n$, ^, 0 = Move by line\r\nG, gg = Move by file\r\n\r\n&#34; Operators on selection\r\nd, x = Delete selection\r\nc = Change selection\r\ny = Yank selection\r\n&gt; = Indent right\r\n&lt; = Indent left\r\nu = Lowercase\r\nU = Uppercase\r\n= = Auto-indent<\/code><\/pre><h3 class=\"heading-element\" id=\"block-visual-mode-examples\"><span>Block Visual Mode Examples<\/span>\n <a href=\"#block-visual-mode-examples\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Add comment prefix to multiple lines\r\nCTRL-V = Start block select\r\n3j = Select 3 lines down\r\n$ = Select to end of lines\r\n = Now have rectangular selection\r\nI\/\/ &lt;ESC&gt; = Insert &#34;\/\/ &#34; at beginning of each line\r\n\r\n&#34; Insert column of text\r\nCTRL-V = Start block mode\r\n3j = Select 4 lines\r\n$ = Select to end\r\nA:&lt;ESC&gt; = Append &#34;:&#34; to end of each line\r\n\r\n&#34; Replace in column\r\nCTRL-V = Select rectangle\r\nc = Change\r\nnew_text = Type replacement\r\n&lt;ESC&gt; = Applied to all lines in selection<\/code><\/pre><hr>\n<h2 id=\"marks-jumps\">Marks and Jumps: Navigate Like a Pro<\/h2>\r\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Marks: Bookmarks for Your Code<\/strong><\/p>\n<p>Marks let you set bookmarks in files and jump between them instantly. Essential for navigating large codebases or editing multiple sections.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"setting-and-using-marks\"><span>Setting and Using Marks<\/span>\n <a href=\"#setting-and-using-marks\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Setting marks\r\nma = Set mark &#39;a&#39; at current position\r\nmB = Set mark &#39;B&#39; at current position\r\nm[a-z] = Local marks (per-buffer)\r\nm[A-Z] = Global marks (across files)\r\n\r\n&#34; Jumping to marks\r\n`a = Jump to exact position of mark &#39;a&#39; (line &#43; column)\r\n&#39;a = Jump to line of mark &#39;a&#39; (column 0)\r\n`A = Jump to global mark A (even across files!)\r\n\r\n&#34; Special marks (automatic)\r\n`` = Position before last jump\r\n`. = Last change position\r\n`&#34; = Position when last exited file\r\n`[ = Start of last changed or yanked text\r\n`] = End of last changed or yanked text\r\n`&lt; = Start of last visual selection\r\n`&gt; = End of last visual selection\r\n\r\n&#34; List all marks\r\n:marks = Show all active marks\r\n:delmarks a = Delete mark &#39;a&#39;\r\n:delmarks! = Delete all lowercase marks<\/code><\/pre><h3 class=\"heading-element\" id=\"jump-list-navigation\"><span>Jump List Navigation<\/span>\n <a href=\"#jump-list-navigation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>CTRL-O = Jump to older position (jump back)\r\nCTRL-I = Jump to newer position (jump forward)\r\n:jumps = Show jump list history\r\n\r\n&#34; Change list\r\ng; = Go to older change position\r\ng, = Go to newer change position\r\n:changes = Show change list\r\n\r\n&#34; Practical workflow\r\n1. Browse code with searches and motions\r\n2. CTRL-O to jump back through history\r\n3. CTRL-I to jump forward<\/code><\/pre><h3 class=\"heading-element\" id=\"marks-workflow-diagram\"><span>Marks Workflow Diagram<\/span>\n <a href=\"#marks-workflow-diagram\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Visualizing how marks work across files helps understand their power for navigation:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/vim-marks-navigation.webp\" title=\"Vim Marks - Setting local and global marks for quick navigation with jump history (Ctrl-O\/Ctrl-I)\" data-thumbnail=\"\/files\/vim-marks-navigation.webp\" data-sub-html=\"<h2>Vim Marks Navigation System<\/h2><p>Vim Marks - Setting local and global marks for quick navigation with jump history (Ctrl-O\/Ctrl-I)<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/vim-marks-navigation.webp' alt=\"Vim Marks Navigation System\"><\/a><figcaption class=\"image-caption\">Vim Marks - Setting local and global marks for quick navigation with jump history (Ctrl-O\/Ctrl-I)<\/figcaption>\n <\/figure><\/p>\n<p><strong>Remember:<\/strong> Local marks (a-z) are file-specific, while global marks (A-Z) work across files!<\/p>\n<hr>\n<h2 id=\"folding\">Code Folding: Manage Large Files<\/h2>\r\n<div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Folding Collapses Code Sections<\/strong><\/p>\n<p>Like collapsing functions in an IDE. Essential for navigating large files by hiding sections you&rsquo;re not currently editing.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"folding-commands\"><span>Folding Commands<\/span>\n <a href=\"#folding-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Manual folding\r\nzf{motion} = Create fold (e.g., zf5j = fold next 5 lines)\r\nzf% = Fold matching bracket pair\r\n{Visual}zf = Create fold from visual selection\r\n\r\n&#34; Opening\/closing folds\r\nzo = Open fold under cursor\r\nzc = Close fold under cursor\r\nza = Toggle fold (open if closed, close if open)\r\nzO = Open all folds recursively under cursor\r\nzC = Close all folds recursively under cursor\r\n\r\n&#34; All folds in buffer\r\nzM = Close ALL folds in buffer\r\nzR = Open ALL folds in buffer\r\nzm = Fold more (close one level)\r\nzr = Fold less (open one level)\r\n\r\n&#34; Navigating folds\r\nzj = Move to next fold\r\nzk = Move to previous fold\r\n[z = Move to start of current open fold\r\n]z = Move to end of current open fold\r\n\r\n&#34; Delete folds\r\nzd = Delete fold under cursor\r\nzD = Delete all folds under cursor recursively\r\nzE = Eliminate all folds in buffer<\/code><\/pre><h3 class=\"heading-element\" id=\"folding-methods\"><span>Folding Methods<\/span>\n <a href=\"#folding-methods\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Choose folding method\r\nset foldmethod=manual &#34; Manual folding (default)\r\nset foldmethod=indent &#34; Fold by indentation level\r\nset foldmethod=syntax &#34; Fold based on syntax (for code)\r\nset foldmethod=expr &#34; Fold using custom expression\r\nset foldmethod=marker &#34; Fold using {{{ }}} markers\r\n\r\n&#34; Fold configuration\r\nset foldlevel=0 &#34; Close all folds by default\r\nset foldlevelstart=99 &#34; Start with all folds open\r\nset foldnestmax=3 &#34; Maximum fold nesting\r\n\r\n&#34; Example: Python folding by indent\r\nautocmd FileType python setlocal foldmethod=indent foldlevel=0<\/code><\/pre><h3 class=\"heading-element\" id=\"folding-visualization\"><span>Folding Visualization<\/span>\n <a href=\"#folding-visualization\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>See how folding transforms a 100-line file into a manageable overview:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/vim-folding-visualization.webp\" title=\"Vim Code Folding - Collapsing function and class definitions for easier navigation and focus on relevant code sections\" data-thumbnail=\"\/files\/vim-folding-visualization.webp\" data-sub-html=\"<h2>Vim Code Folding Visualization<\/h2><p>Vim Code Folding - Collapsing function and class definitions for easier navigation and focus on relevant code sections<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/vim-folding-visualization.webp' alt=\"Vim Code Folding Visualization\"><\/a><figcaption class=\"image-caption\">Vim Code Folding - Collapsing function and class definitions for easier navigation and focus on relevant code sections<\/figcaption>\n <\/figure><\/p>\n<p><strong>Pro Tip:<\/strong> Use <code>zM<\/code> to close all folds and <code>zR<\/code> to open all folds for quick file overview!<\/p>\n<hr>\n<h2 id=\"windows-splits\">Window Splits and Layouts<\/h2>\r\n<div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Multi-Window Editing<\/strong><\/p>\n<p>Edit multiple files simultaneously or view different parts of the same file. Essential for comparing code, referencing documentation while coding, or split-screen workflows.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"window-split-commands\"><span>Window Split Commands<\/span>\n <a href=\"#window-split-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Creating splits\r\n:split [file] = Horizontal split (Ctrl-W s)\r\n:vsplit [file] = Vertical split (Ctrl-W v)\r\n:new = New horizontal split with empty buffer\r\n:vnew = New vertical split with empty buffer\r\n\r\n&#34; Navigating between windows\r\nCTRL-W h = Move to window on left\r\nCTRL-W j = Move to window below\r\nCTRL-W k = Move to window above\r\nCTRL-W l = Move to window on right\r\nCTRL-W w = Cycle through windows\r\nCTRL-W p = Go to previous window\r\n\r\n&#34; Resizing windows\r\nCTRL-W = = Make all windows equal size\r\nCTRL-W _ = Maximize current window height\r\nCTRL-W | = Maximize current window width\r\nCTRL-W &#43; = Increase height\r\nCTRL-W - = Decrease height\r\nCTRL-W &gt; = Increase width\r\nCTRL-W &lt; = Decrease width\r\n:resize 20 = Set window height to 20 lines\r\n:vertical resize 80 = Set window width to 80 columns\r\n\r\n&#34; Moving windows\r\nCTRL-W H = Move window to far left\r\nCTRL-W J = Move window to bottom\r\nCTRL-W K = Move window to top\r\nCTRL-W L = Move window to far right\r\nCTRL-W r = Rotate windows\r\nCTRL-W x = Exchange with next window\r\n\r\n&#34; Closing windows\r\nCTRL-W q = Quit window (close)\r\nCTRL-W c = Close window (same as :close)\r\nCTRL-W o = Close all other windows (only keep current)\r\n:only = Close all windows except current<\/code><\/pre><h3 class=\"heading-element\" id=\"window-layout-visualization\"><span>Window Layout Visualization<\/span>\n <a href=\"#window-layout-visualization\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Different layouts for different workflows. Mix horizontal and vertical splits as needed:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/vim-window-splits.webp\" title=\"Vim Window Splits - Creating and managing horizontal and vertical window splits for multi-file editing and comparison\" data-thumbnail=\"\/files\/vim-window-splits.webp\" data-sub-html=\"<h2>Vim Window Splits: Horizontal and Vertical Layouts<\/h2><p>Vim Window Splits - Creating and managing horizontal and vertical window splits for multi-file editing and comparison<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/vim-window-splits.webp' alt=\"Vim Window Splits: Horizontal and Vertical Layouts\"><\/a><figcaption class=\"image-caption\">Vim Window Splits - Creating and managing horizontal and vertical window splits for multi-file editing and comparison<\/figcaption>\n <\/figure><\/p>\n<p><strong>Quick Tips:<\/strong><\/p>\n<ul>\n<li>Horizontal split: <code>:split<\/code> or <code>CTRL-W s<\/code><\/li>\n<li>Vertical split: <code>:vsplit<\/code> or <code>CTRL-W v<\/code><\/li>\n<li>Navigate: Use <code>CTRL-W<\/code> + direction keys (hjkl)<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"practical-split-workflow\"><span>Practical Split Workflow<\/span>\n <a href=\"#practical-split-workflow\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Open file in horizontal split, reference while coding\r\n:split reference.md\r\n\r\n&#34; Open related files side-by-side\r\n:vsplit related_code.py\r\n\r\n&#34; Quick split for viewing same file in two places\r\nCTRL-W v &#34; Vertical split same file\r\n50G &#34; Go to line 50 in one split\r\nCTRL-W w &#34; Switch to other split\r\n100G &#34; Go to line 100\r\n &#34; Now you can see lines 50 and 100 simultaneously!\r\n\r\n&#34; Comparing two files\r\n:split file1.txt\r\n:vsplit file2.txt\r\n &#34; Side-by-side comparison<\/code><\/pre><hr>\n<h2 id=\"buffers-tabs\">Buffers, Windows, and Tabs Explained<\/h2>\r\n<div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Understanding Vim&rsquo;s Three-Layer System<\/strong><\/p>\n<p>This is often confusing for beginners. Vim has three concepts:<\/p>\n<ul>\n<li><strong>Buffer<\/strong>: A file in memory<\/li>\n<li><strong>Window<\/strong>: A view into a buffer<\/li>\n<li><strong>Tab<\/strong>: A collection of windows<\/li>\n<\/ul>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"buffers-vs-windows-vs-tabs\"><span>Buffers vs Windows vs Tabs<\/span>\n <a href=\"#buffers-vs-windows-vs-tabs\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Understanding the relationship between these three concepts is crucial:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/vim-buffer-window-tab-hierarchy.webp\" title=\"Vim Hierarchy - Understanding the relationship between buffers (files in memory), windows (views into buffers), and tabs (collections of windows)\" data-thumbnail=\"\/files\/vim-buffer-window-tab-hierarchy.webp\" data-sub-html=\"<h2>Vim Buffer, Window, and Tab Hierarchy<\/h2><p>Vim Hierarchy - Understanding the relationship between buffers (files in memory), windows (views into buffers), and tabs (collections of windows)<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/vim-buffer-window-tab-hierarchy.webp' alt=\"Vim Buffer, Window, and Tab Hierarchy\"><\/a><figcaption class=\"image-caption\">Vim Hierarchy - Understanding the relationship between buffers (files in memory), windows (views into buffers), and tabs (collections of windows)<\/figcaption>\n <\/figure><\/p>\n<p><strong>Key Insight:<\/strong> One buffer can be displayed in multiple windows simultaneously, and buffers remain in memory even when not visible!<\/p>\n<h3 class=\"heading-element\" id=\"buffer-management\"><span>Buffer Management<\/span>\n <a href=\"#buffer-management\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Opening buffers\r\n:e file.txt = Edit (open) file in current buffer\r\n:badd file.txt = Add file to buffer list (don&#39;t show)\r\n:buffer N = Switch to buffer number N\r\n:buffer file = Switch to buffer by name\r\n\r\n&#34; Navigating buffers\r\n:ls = List all buffers (shows numbers and status)\r\n:bnext = Next buffer (also :bn)\r\n:bprevious = Previous buffer (also :bp)\r\n:bfirst = First buffer\r\n:blast = Last buffer\r\n:b# = Switch to alternate buffer (last viewed)\r\n\r\n&#34; Buffer info in :ls output\r\n&#34; %a = Active buffer (visible in current window)\r\n&#34; # = Alternate buffer (last viewed)\r\n&#34; &#43; = Modified buffer\r\n&#34; - = Unmodifiable buffer\r\n&#34; = = Readonly buffer\r\n&#34; h = Hidden buffer\r\n\r\n&#34; Closing buffers\r\n:bd = Delete (close) current buffer\r\n:bd 3 = Delete buffer 3\r\n:bd file.txt = Delete buffer by name\r\n:%bd = Delete all buffers\r\n:bufdo bd = Delete all buffers (iterate through)\r\n\r\n&#34; Working with multiple buffers\r\n:args file1 file2 file3 = Set argument list\r\n:argdo %s\/old\/new\/g = Execute command on all args\r\n:bufdo update = Save all modified buffers<\/code><\/pre><h3 class=\"heading-element\" id=\"tab-management\"><span>Tab Management<\/span>\n <a href=\"#tab-management\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Creating tabs\r\n:tabnew = New empty tab\r\n:tabedit file = Open file in new tab\r\n:tabfind file = Find and open file in new tab\r\n\r\n&#34; Navigating tabs\r\ngt = Next tab (also :tabnext)\r\ngT = Previous tab (also :tabprevious)\r\n3gt = Go to tab 3\r\n:tabfirst = Go to first tab\r\n:tablast = Go to last tab\r\n\r\n&#34; Moving tabs\r\n:tabmove 0 = Move current tab to first position\r\n:tabmove = Move current tab to last position\r\n:tabmove &#43;1 = Move tab one position right\r\n:tabmove -1 = Move tab one position left\r\n\r\n&#34; Closing tabs\r\n:tabclose = Close current tab\r\n:tabonly = Close all other tabs<\/code><\/pre><hr>\n<h2 id=\"advanced-search\">Advanced Search Techniques<\/h2>\r\n<h3 class=\"heading-element\" id=\"quick-search-commands\"><span>Quick Search Commands<\/span>\n <a href=\"#quick-search-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>* = Search forward for word under cursor\r\n# = Search backward for word under cursor\r\ng* = Search forward for partial word match\r\ng# = Search backward for partial word match\r\n\r\ngd = Go to local declaration\r\ngD = Go to global declaration\r\ngf = Go to file under cursor (open file)\r\nCTRL-] = Jump to tag (requires ctags)\r\nCTRL-T = Jump back from tag\r\n\r\n&#34; Searching in multiple files\r\n:vimgrep \/pattern\/ **\/*.py = Search in all Python files recursively\r\n:grep pattern **\/*.py = Use external grep\r\n:cn = Next match\r\n:cp = Previous match\r\n:copen = Open quickfix window with results\r\n:cclose = Close quickfix window<\/code><\/pre><h3 class=\"heading-element\" id=\"search-across-multiple-files\"><span>Search Across Multiple Files<\/span>\n <a href=\"#search-across-multiple-files\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Using vimgrep (built-in)\r\n:vimgrep \/TODO\/ **\/*.js &#34; Search all JS files\r\n:copen &#34; Open quickfix with results\r\n:cn &#34; Jump to next match\r\n:cp &#34; Jump to previous match\r\n\r\n&#34; Using external grep (faster)\r\n:grep -r &#34;function&#34; . &#34; Recursive grep\r\n:copen &#34; View results\r\n\r\n&#34; Search in open buffers only\r\n:bufdo vimgrepadd \/pattern\/ %\r\n\r\n&#34; Location list (per-window quickfix)\r\n:lvimgrep \/pattern\/ **\/*.py &#34; Search, use location list\r\n:lopen &#34; Open location list\r\n:lnext &#34; Next in location list<\/code><\/pre><hr>\n<h2 id=\"autocomplete\">Autocomplete and Omnicomplete<\/h2>\r\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Built-in Autocomplete<\/strong><\/p>\n<p>Vim has powerful autocomplete without plugins! Use CTRL-N and CTRL-P for basic completion, or CTRL-X submodes for advanced completion.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"autocomplete-commands\"><span>Autocomplete Commands<\/span>\n <a href=\"#autocomplete-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; In Insert Mode:\r\nCTRL-N = Next completion (searches forward in file)\r\nCTRL-P = Previous completion (searches backward)\r\n\r\n&#34; CTRL-X submodes (press in insert mode)\r\nCTRL-X CTRL-N = Complete from current buffer\r\nCTRL-X CTRL-F = Complete filenames\r\nCTRL-X CTRL-L = Complete whole lines\r\nCTRL-X CTRL-K = Complete from dictionary\r\nCTRL-X CTRL-T = Complete from thesaurus\r\nCTRL-X CTRL-I = Complete from current and included files\r\nCTRL-X CTRL-] = Complete tags\r\nCTRL-X CTRL-O = Omnicomplete (language-specific)\r\n\r\n&#34; Navigate completion menu\r\nCTRL-N = Select next item\r\nCTRL-P = Select previous item\r\nCTRL-Y = Accept selection\r\nCTRL-E = Cancel completion<\/code><\/pre><h3 class=\"heading-element\" id=\"autocomplete-configuration\"><span>Autocomplete Configuration<\/span>\n <a href=\"#autocomplete-configuration\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Basic autocomplete settings\r\nset completeopt=longest,menuone &#34; Better completion menu\r\nset complete&#43;=kspell &#34; Spell checking suggestions\r\n\r\n&#34; Enable omnicomplete for programming\r\nfiletype plugin on\r\nset omnifunc=syntaxcomplete#Complete\r\n\r\n&#34; Auto-trigger completion after certain chars\r\nautocmd FileType python set omnifunc=pythoncomplete#Complete\r\nautocmd FileType javascript set omnifunc=javascriptcomplete#CompleteJS\r\nautocmd FileType html set omnifunc=htmlcomplete#CompleteTags\r\nautocmd FileType css set omnifunc=csscomplete#CompleteCSS\r\n\r\n&#34; Dictionary completion\r\nset dictionary=\/usr\/share\/dict\/words<\/code><\/pre><hr>\n<h2 id=\"sessions\">Sessions: Save Your Workspace<\/h2>\r\n<div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Sessions Save Your Entire Vim State<\/strong><\/p>\n<p>Save your open files, window layouts, tabs, and even command history. Perfect for project-based workflows.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"session-commands\"><span>Session Commands<\/span>\n <a href=\"#session-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Creating sessions\r\n:mksession ~\/my-project.vim &#34; Create session file\r\n:mks ~\/my-project.vim &#34; Shorthand\r\n\r\n&#34; Session includes:\r\n&#34; - All open buffers\r\n&#34; - Window layouts and splits\r\n&#34; - Tab pages\r\n&#34; - Current directory\r\n&#34; - Vim options and mappings\r\n&#34; - Marks and registers (optional)\r\n\r\n&#34; Loading sessions\r\nvim -S ~\/my-project.vim &#34; Load session on startup\r\n:source ~\/my-project.vim &#34; Load session from Vim\r\n\r\n&#34; Updating existing session\r\n:mks! &#34; Overwrite current session\r\n\r\n&#34; Session options\r\nset sessionoptions=blank,buffers,curdir,folds,help,tabpages,winsize\r\n\r\n&#34; Auto-save session on exit (add to .vimrc)\r\nautocmd VimLeave * mksession! ~\/.vim\/session\/last.vim<\/code><\/pre><h3 class=\"heading-element\" id=\"practical-session-workflow\"><span>Practical Session Workflow<\/span>\n <a href=\"#practical-session-workflow\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Setup for a project\r\ncd ~\/projects\/myapp\r\nvim .\r\n\r\n&#34; Open relevant files\r\n:e src\/main.py\r\n:vsplit src\/utils.py\r\n:split tests\/test_main.py\r\n\r\n&#34; Arrange windows as needed\r\nCTRL-W = (equalize sizes)\r\n\r\n&#34; Save this layout as session\r\n:mks! ~\/vim-sessions\/myapp.vim\r\n\r\n&#34; Next day: restore entire workspace\r\nvim -S ~\/vim-sessions\/myapp.vim\r\n\r\n&#34; Everything back as it was:\r\n&#34; - Same files open\r\n&#34; - Same window layout\r\n&#34; - Same working directory<\/code><\/pre><hr>\n<h2 id=\"diff-mode\">Diff Mode: Compare Files<\/h2>\r\n<h3 class=\"heading-element\" id=\"diff-commands\"><span>Diff Commands<\/span>\n <a href=\"#diff-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Start diff mode\r\nvimdiff file1.txt file2.txt &#34; From command line\r\n:vertical diffsplit file2.txt &#34; From within Vim\r\n\r\n&#34; Diff navigation\r\n]c = Jump to next change\r\n[c = Jump to previous change\r\n\r\n&#34; Diff operations\r\ndo = Obtain (get) changes from other file\r\ndp = Put changes to other file\r\n:diffget = Same as &#39;do&#39; (obtain)\r\n:diffput = Same as &#39;dp&#39; (put)\r\n\r\n&#34; Diff options\r\n:diffupdate = Recalculate diff (if changes occurred)\r\n:diffoff = Turn off diff mode\r\n:set diffopt=filler,vertical &#34; Configuration\r\n\r\n&#34; Resolve merge conflicts\r\n:set diffopt&#43;=iwhite &#34; Ignore whitespace\r\n:diffget LOCAL &#34; Get changes from LOCAL\r\n:diffget REMOTE &#34; Get changes from REMOTE<\/code><\/pre><h3 class=\"heading-element\" id=\"diff-visualization\"><span>Diff Visualization<\/span>\n <a href=\"#diff-visualization\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>Diff Mode Display:\r\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 file1.txt \u2502 file2.txt \u2502\r\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\r\n\u2502 line 1: same \u2502 line 1: same \u2502\r\n\u2502 line 2: different \u2502 line 2: CHANGED \u2502\r\n\u2502 \u2514\u2500 highlighted red \u2502 \u2514\u2500 highlighted green \u2502\r\n\u2502 line 3: same \u2502 line 3: same \u2502\r\n\u2502 line 4: only in file1 \u2502 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2502\r\n\u2502 \u2514\u2500 shown with &#34;&#43;&#34; \u2502 (missing in file2) \u2502\r\n\u2502 line 5: same \u2502 line 4: same \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\nNavigation:\r\n]c \u2192 Jump to next difference\r\n[c \u2192 Jump to previous difference\r\ndo \u2192 Obtain change from other window\r\ndp \u2192 Put change to other window<\/code><\/pre><hr>\n<h2 id=\"spell-check\">Spell Checking<\/h2>\r\n<h3 class=\"heading-element\" id=\"spell-check-commands\"><span>Spell Check Commands<\/span>\n <a href=\"#spell-check-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Enable\/disable spell checking\r\n:set spell &#34; Turn on spell check\r\n:set nospell &#34; Turn off spell check\r\n:set spelllang=en_us &#34; Set language to US English\r\n:set spelllang=en_gb &#34; Set language to British English\r\n\r\n&#34; Navigate spelling errors\r\n]s = Next misspelled word\r\n[s = Previous misspelled word\r\n]S = Next misspelled word (skip rare words)\r\n[S = Previous misspelled word (skip rare words)\r\n\r\n&#34; Fix spelling errors\r\nz= = Suggest corrections (shows menu)\r\n1z= = Use first suggestion automatically\r\nzg = Add word to dictionary (good word)\r\nzw = Mark word as misspelled (wrong word)\r\nzug = Undo zg (remove from dictionary)\r\nzuw = Undo zw\r\n\r\n&#34; Custom dictionary\r\n:set spellfile=~\/.vim\/spell\/custom.utf-8.add<\/code><\/pre><hr>\n<h2 id=\"text-transformations\">Advanced Text Transformations<\/h2>\r\n<h3 class=\"heading-element\" id=\"case-transformations\"><span>Case Transformations<\/span>\n <a href=\"#case-transformations\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>~ = Toggle case of character under cursor\r\ng~{motion} = Toggle case of motion\r\ngu{motion} = Lowercase motion\r\ngU{motion} = Uppercase motion\r\n\r\n&#34; Examples\r\ngUiw = Uppercase inner word\r\nguap = Lowercase paragraph\r\ng~~ = Toggle case of current line\r\ngUU = Uppercase current line\r\nguu = Lowercase current line\r\n\r\n&#34; In visual mode\r\nV = Select line\r\nU = Uppercase selection\r\nu = Lowercase selection\r\n~ = Toggle case<\/code><\/pre><h3 class=\"heading-element\" id=\"text-joining-and-splitting\"><span>Text Joining and Splitting<\/span>\n <a href=\"#text-joining-and-splitting\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>J = Join current line with next (removes newline)\r\n5J = Join next 5 lines\r\ngJ = Join without adding space\r\n\r\n&#34; Example:\r\n&#34; Before:\r\n&#34; line1\r\n&#34; line2\r\n&#34; line3\r\n\r\n&#34; After pressing J:\r\n&#34; line1 line2\r\n&#34; line3\r\n\r\n&#34; After pressing gJ (no space):\r\n&#34; line1line2<\/code><\/pre><h3 class=\"heading-element\" id=\"sorting-and-filtering\"><span>Sorting and Filtering<\/span>\n <a href=\"#sorting-and-filtering\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Sort lines\r\n:sort = Sort lines alphabetically\r\n:sort! = Sort in reverse\r\n:sort n = Sort numerically\r\n:sort u = Sort and remove duplicates\r\n\r\n&#34; Sort range\r\n:10,20sort = Sort lines 10-20\r\n:&#39;&lt;,&#39;&gt;sort = Sort visual selection\r\n\r\n&#34; Filter through external command\r\n:%!sort = Filter entire file through Unix sort\r\n:%!uniq = Remove duplicate lines\r\n:10,20!sort = Sort lines 10-20 with Unix sort\r\n\r\n&#34; Example: Format JSON\r\n:%!python -m json.tool &#34; Pretty-print JSON file<\/code><\/pre><h3 class=\"heading-element\" id=\"increment-and-decrement-numbers\"><span>Increment and Decrement Numbers<\/span>\n <a href=\"#increment-and-decrement-numbers\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>CTRL-A = Increment number under cursor\r\nCTRL-X = Decrement number under cursor\r\n10CTRL-A = Increment by 10\r\n\r\n&#34; Visual block increment (create sequences)\r\nCTRL-V = Block select numbers\r\ng CTRL-A = Increment sequentially (1,2,3,4...)\r\ng CTRL-X = Decrement sequentially\r\n\r\n&#34; Example: Create numbered list\r\n1 = Start with 1\r\nCTRL-V = Block select vertically\r\n5j = Select 5 lines\r\ng CTRL-A = Now you have: 1,2,3,4,5,6<\/code><\/pre><hr>\n<h2 id=\"quickfix\">Quickfix and Location Lists<\/h2>\r\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Quickfix: Central Error\/Search Results<\/strong><\/p>\n<p>Quickfix list is a special window showing errors, search results, or any list of file locations. Essential for compilation errors, search results, and linting.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"quickfix-commands\"><span>Quickfix Commands<\/span>\n <a href=\"#quickfix-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Opening quickfix\r\n:copen = Open quickfix window\r\n:cclose = Close quickfix window\r\n:cwindow = Open only if there are errors\r\n\r\n&#34; Navigation\r\n:cnext = Next item (also :cn)\r\n:cprevious = Previous item (also :cp)\r\n:cfirst = First item\r\n:clast = Last item\r\n:cc N = Jump to item N\r\n\r\n&#34; Loading errors\/results\r\n:cfile file = Load errors from file\r\n:cgetfile file = Load errors (don&#39;t jump)\r\n:cexpr expr = Set quickfix from expression\r\n\r\n&#34; Quickfix for compilation\r\n:make = Run make, populate quickfix with errors\r\n:set makeprg=gcc\\ % &#34; Set compiler command<\/code><\/pre><h3 class=\"heading-element\" id=\"location-list-commands\"><span>Location List Commands<\/span>\n <a href=\"#location-list-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Location list (per-window version of quickfix)\r\n:lopen = Open location list\r\n:lclose = Close location list\r\n:lnext = Next item (ln)\r\n:lprevious = Previous item (lp)\r\n:lfirst = First item\r\n:llast = Last item\r\n\r\n&#34; Location list is specific to each window\r\n&#34; Quickfix is global to all windows<\/code><\/pre><h3 class=\"heading-element\" id=\"quickfix-workflow-example\"><span>Quickfix Workflow Example<\/span>\n <a href=\"#quickfix-workflow-example\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; 1. Search for TODO comments across project\r\n:vimgrep \/TODO\/ **\/*.py\r\n\r\n&#34; 2. Open quickfix with results\r\n:copen\r\n\r\n&#34; 3. Navigate through results\r\n:cn &#34; Next TODO\r\n:cp &#34; Previous TODO\r\n\r\n&#34; 4. Jump directly to item 5\r\n:cc 5\r\n\r\n&#34; 5. Close quickfix\r\n:cclose\r\n\r\n&#34; Quickfix window looks like:\r\n&#34; src\/main.py|23 col 5| # TODO: Fix this bug\r\n&#34; src\/utils.py|45 col 8| # TODO: Refactor\r\n&#34; tests\/test.py|12 col 3| # TODO: Add test<\/code><\/pre><hr>\n<h2 id=\"command-history\">Command History and Repeat<\/h2>\r\n<h3 class=\"heading-element\" id=\"command-history\"><span>Command History<\/span>\n <a href=\"#command-history\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; In command mode (:)\r\n&lt;Up&gt;\/&lt;Down&gt; = Navigate command history\r\n:history = Show command history\r\n:history \/ = Show search history\r\nq: = Open command history window (editable!)\r\nq\/ = Open search history window\r\nCTRL-F = Open command history from command line\r\n\r\n&#34; In history window:\r\n&lt;Enter&gt; = Execute command under cursor\r\n:q = Close history window<\/code><\/pre><h3 class=\"heading-element\" id=\"repeat-commands\"><span>Repeat Commands<\/span>\n <a href=\"#repeat-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>. = Repeat last change\r\n@: = Repeat last command-line command\r\n@@ = Repeat last @x macro\r\n\r\n&#34; Practical examples\r\ndd = Delete line\r\n. = Delete another line (repeats dd)\r\n\r\n:s\/old\/new\/ = Substitute on one line\r\n@: = Repeat substitution on current line\r\n\r\n&#34; Repeat with count\r\n3. = Repeat last change 3 times\r\n5@: = Repeat last command 5 times<\/code><\/pre><hr>\n<h2 id=\"advanced-tricks\">Advanced Vim Tricks<\/h2>\r\n<h3 class=\"heading-element\" id=\"multiple-cursors-simulation\"><span>Multiple Cursors Simulation<\/span>\n <a href=\"#multiple-cursors-simulation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Insert at multiple positions\r\nCTRL-V = Block visual mode\r\n5j = Select 5 lines\r\nI = Insert at beginning of all lines\r\ntext&lt;ESC&gt; = Type text, press ESC\r\n \u2192 text appears on all selected lines!\r\n\r\n&#34; Append at multiple positions\r\nCTRL-V = Block select\r\n5j = Select lines\r\n$ = Go to end of lines\r\nA = Append at end\r\ntext&lt;ESC&gt; = Text appended to all lines\r\n\r\n&#34; Change multiple lines\r\nCTRL-V = Block select\r\n5j = Select lines\r\nc = Change\r\nnew_text&lt;ESC&gt; = Replaced on all lines<\/code><\/pre><h3 class=\"heading-element\" id=\"global-command-magic\"><span>Global Command Magic<\/span>\n <a href=\"#global-command-magic\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Syntax: :g\/pattern\/command\r\n\r\n&#34; Delete all lines containing pattern\r\n:g\/TODO\/d = Delete all lines with TODO\r\n\r\n&#34; Delete all lines NOT containing pattern\r\n:g!\/pattern\/d = Keep only lines with pattern\r\n:v\/pattern\/d = Same as above\r\n\r\n&#34; Execute command on matching lines\r\n:g\/function\/normal @a = Run macro &#39;a&#39; on lines with &#39;function&#39;\r\n:g\/import\/m$ = Move all import lines to end\r\n\r\n&#34; Copy matching lines\r\n:g\/error\/t$ = Copy all lines with &#39;error&#39; to end\r\n\r\n&#34; Multiple commands\r\n:g\/pattern\/d | g\/other\/d = Delete lines matching either pattern<\/code><\/pre><h3 class=\"heading-element\" id=\"ex-command-tricks\"><span>Ex Command Tricks<\/span>\n <a href=\"#ex-command-tricks\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Read external command output\r\n:r!date = Insert current date\r\n:r!ls = Insert directory listing\r\n:0r!cat file = Insert file contents at top\r\n\r\n&#34; Write to command\r\n:w !sudo tee % = Save file with sudo (when opened without sudo)\r\n:%!xxd = Convert file to hex dump\r\n:%!xxd -r = Convert hex back to binary\r\n\r\n&#34; Execute shell command and read output\r\n:read !grep pattern file.txt\r\n\r\n&#34; Substitute tricks\r\n:%s\/\\v(\\w&#43;)\\s&#43;(\\w&#43;)\/\\2 \\1\/g = Swap two words\r\n:%s\/\\n\\n\/\\r\/g = Replace double newlines with single\r\n:%s\/\\s\\&#43;$\/\/ = Remove trailing whitespace<\/code><\/pre><h3 class=\"heading-element\" id=\"vim-command-line-editing\"><span>Vim Command Line Editing<\/span>\n <a href=\"#vim-command-line-editing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; In command mode (:)\r\nCTRL-B = Beginning of line\r\nCTRL-E = End of line\r\nCTRL-W = Delete word before cursor\r\nCTRL-U = Delete to beginning\r\nCTRL-R &#34; = Insert contents of register\r\nCTRL-R % = Insert current filename\r\nCTRL-R \/ = Insert last search pattern\r\n\r\n&#34; Example\r\n:e CTRL-R % = :e currentfile.txt (inserts current filename)<\/code><\/pre><hr>\n<h2 id=\"performance-tips\">Performance and Optimization<\/h2>\r\n<h3 class=\"heading-element\" id=\"profiling-vim-performance\"><span>Profiling Vim Performance<\/span>\n <a href=\"#profiling-vim-performance\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Profile startup time\r\nvim --startuptime startup.log file.txt\r\n&#34; Examine startup.log to see what&#39;s slow\r\n\r\n&#34; Profile runtime\r\n:profile start profile.log\r\n:profile func *\r\n:profile file *\r\n&#34; Do operations...\r\n:profile pause\r\n:noautocmd qall\r\n\r\n&#34; Check what&#39;s taking time\r\n&#34; View profile.log<\/code><\/pre><h3 class=\"heading-element\" id=\"optimization-tips\"><span>Optimization Tips<\/span>\n <a href=\"#optimization-tips\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Speed up for large files\r\nset synmaxcol=200 &#34; Don&#39;t syntax highlight long lines\r\nset lazyredraw &#34; Don&#39;t redraw during macros\r\nset ttyfast &#34; Fast terminal connection\r\nset nocursorline &#34; Disable cursor line highlight\r\nset norelativenumber &#34; Disable relative numbers\r\n\r\n&#34; Disable swap and backup for speed\r\nset noswapfile\r\nset nobackup\r\nset nowritebackup\r\n\r\n&#34; Limit undo history\r\nset undolevels=100\r\n\r\n&#34; Auto-command to disable features for large files\r\nautocmd BufReadPre * if getfsize(bufname(&#34;%&#34;)) &gt; 1000000 | setlocal syntax=off | endif<\/code><\/pre><hr>\n<h2 id=\"advanced-config\">Advanced Configuration<\/h2>\r\n<h3 class=\"heading-element\" id=\"complete-vimrc-setup\"><span>Complete .vimrc Setup<\/span>\n <a href=\"#complete-vimrc-setup\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; ============================================================================\r\n&#34; GENERAL SETTINGS\r\n&#34; ============================================================================\r\nset nocompatible &#34; Use Vim defaults, not Vi\r\nset encoding=utf-8 &#34; Use UTF-8 encoding\r\nset fileencoding=utf-8\r\n\r\n&#34; ============================================================================\r\n&#34; DISPLAY\r\n&#34; ============================================================================\r\nsyntax on &#34; Enable syntax highlighting\r\nset number &#34; Show line numbers\r\nset relativenumber &#34; Relative numbering (jump distance)\r\nset cursorline &#34; Highlight current line\r\nset termguicolors &#34; Enable 24-bit colors\r\nset background=dark &#34; Dark color scheme\r\ncolorscheme industry &#34; Use preferred colorscheme\r\n\r\n&#34; ============================================================================\r\n&#34; INDENTATION\r\n&#34; ============================================================================\r\nset tabstop=4 &#34; Display tabs as 4 spaces\r\nset shiftwidth=4 &#34; Indentation size\r\nset expandtab &#34; Use spaces instead of tabs\r\nset autoindent &#34; Auto-indent new lines\r\nset smartindent &#34; Smart indentation\r\nset shiftround &#34; Round indent to shiftwidth\r\n\r\n&#34; ============================================================================\r\n&#34; SEARCH\r\n&#34; ============================================================================\r\nset ignorecase &#34; Case insensitive search\r\nset smartcase &#34; Except uppercase triggers case-sensitive\r\nset incsearch &#34; Highlight matches while typing\r\nset hlsearch &#34; Highlight all matches\r\nset wrapscan &#34; Wrap search around file\r\n\r\n&#34; ============================================================================\r\n&#34; BEHAVIOR\r\n&#34; ============================================================================\r\nset backspace=indent,eol,start &#34; Allow backspace in insert mode\r\nset hidden &#34; Allow unsaved buffers\r\nset report=0 &#34; Always report changes\r\nset completeopt=longest,menuone &#34; Better autocomplete\r\nset mouse=a &#34; Enable mouse support\r\nset clipboard=unnamedplus &#34; Use system clipboard\r\n\r\n&#34; ============================================================================\r\n&#34; PERFORMANCE\r\n&#34; ============================================================================\r\nset lazyredraw &#34; Don&#39;t redraw during macros\r\nset ttyfast &#34; Faster redraws\r\nset synmaxcol=200 &#34; Don&#39;t highlight very long lines\r\n\r\n&#34; ============================================================================\r\n&#34; UI ENHANCEMENTS\r\n&#34; ============================================================================\r\nset statusline=%F%m%r%h%w\\ [%{&amp;ff}]\\ %y\\ [%p%%]\\ %l:%c\r\nset laststatus=2 &#34; Always show status line\r\nset showmatch &#34; Highlight matching brackets\r\nset showmode &#34; Show current mode\r\nset showcmd &#34; Show partial command\r\n\r\n&#34; ============================================================================\r\n&#34; KEYBINDINGS - Custom mappings\r\n&#34; ============================================================================\r\nlet mapleader = &#34;,&#34; &#34; Set leader to comma\r\n\r\n&#34; Navigate splits easier\r\nnnoremap &lt;C-j&gt; &lt;C-w&gt;j\r\nnnoremap &lt;C-k&gt; &lt;C-w&gt;k\r\nnnoremap &lt;C-h&gt; &lt;C-w&gt;h\r\nnnoremap &lt;C-l&gt; &lt;C-w&gt;l\r\n\r\n&#34; Quick escape\r\ninoremap jj &lt;Esc&gt;\r\ninoremap kk &lt;Esc&gt;\r\n\r\n&#34; Clear search highlighting\r\nnnoremap &lt;leader&gt;h :noh&lt;CR&gt;\r\n\r\n&#34; Quick save\r\nnnoremap &lt;leader&gt;w :w&lt;CR&gt;\r\nnnoremap &lt;leader&gt;q :q&lt;CR&gt;<\/code><\/pre><hr>\n<h2 id=\"plugins\">Plugin Management<\/h2>\r\n<h3 class=\"heading-element\" id=\"popular-plugin-managers\"><span>Popular Plugin Managers<\/span>\n <a href=\"#popular-plugin-managers\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Plugin Ecosystem<\/strong><\/p>\n<p>Vim&rsquo;s power comes from its plugin ecosystem. Thousands of plugins extend functionality for every workflow imaginable.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<p><strong>vim-plug<\/strong> (Recommended - Easy to use):<\/p>\n<pre><code>&#34; Install vim-plug first:\r\n&#34; curl -fLo ~\/.vim\/autoload\/plug.vim --create-dirs \\\r\n&#34; https:\/\/raw.githubusercontent.com\/junegunn\/vim-plug\/master\/plug.vim\r\n\r\n&#34; In .vimrc:\r\ncall plug#begin(&#39;~\/.vim\/plugged&#39;)\r\n\r\nPlug &#39;vim-airline\/vim-airline&#39; &#34; Better statusline\r\nPlug &#39;preservim\/nerdtree&#39; &#34; File explorer\r\nPlug &#39;junegunn\/fzf.vim&#39; &#34; Fuzzy finder\r\nPlug &#39;tpope\/vim-surround&#39; &#34; Surround text objects\r\nPlug &#39;tpope\/vim-commentary&#39; &#34; Easy comments\r\nPlug &#39;jiangmiao\/auto-pairs&#39; &#34; Auto bracket pairing\r\nPlug &#39;vim-syntastic\/syntastic&#39; &#34; Syntax checking\r\nPlug &#39;neoclide\/coc.nvim&#39; &#34; Autocompletion\r\n\r\ncall plug#end()\r\n\r\n&#34; Install plugins: :PlugInstall\r\n&#34; Update plugins: :PlugUpdate\r\n&#34; Clean unused: :PlugClean<\/code><\/pre><h3 class=\"heading-element\" id=\"essential-plugin-recommendations\"><span>Essential Plugin Recommendations<\/span>\n <a href=\"#essential-plugin-recommendations\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Plugin<\/th>\n <th>Purpose<\/th>\n <th>Use Case<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>vim-airline<\/strong><\/td>\n <td>Enhanced statusline<\/td>\n <td>Better visual feedback<\/td>\n <\/tr>\n <tr>\n <td><strong>nerdtree<\/strong><\/td>\n <td>File browser<\/td>\n <td>Project file navigation<\/td>\n <\/tr>\n <tr>\n <td><strong>fzf.vim<\/strong><\/td>\n <td>Fuzzy finder<\/td>\n <td>Fast file\/buffer search<\/td>\n <\/tr>\n <tr>\n <td><strong>vim-surround<\/strong><\/td>\n <td>Parentheses\/quotes<\/td>\n <td>Easily change delimiters<\/td>\n <\/tr>\n <tr>\n <td><strong>vim-commentary<\/strong><\/td>\n <td>Comments<\/td>\n <td>Toggle comments easily<\/td>\n <\/tr>\n <tr>\n <td><strong>coc.nvim<\/strong><\/td>\n <td>Autocompletion<\/td>\n <td>Intelligent code completion<\/td>\n <\/tr>\n <tr>\n <td><strong>vim-fugitive<\/strong><\/td>\n <td>Git integration<\/td>\n <td>Git operations from Vim<\/td>\n <\/tr>\n <tr>\n <td><strong>vim-repeat<\/strong><\/td>\n <td>Repeat plugin actions<\/td>\n <td>. works with plugins<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><hr>\n<h2 id=\"vimscript-intro\">Introduction to Vimscript<\/h2>\r\n<div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Vimscript is Quirky<\/strong><\/p>\n<p>Vimscript is Vim&rsquo;s built-in scripting language. It&rsquo;s powerful but unconventional (variables need sigils, strings behave oddly, scoping is complex). Many prefer Lua or Python for plugins, but Vimscript is essential for .vimrc customization.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"basic-vimscript\"><span>Basic Vimscript<\/span>\n <a href=\"#basic-vimscript\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Comments start with double quotes\r\n\r\n&#34; ===== VARIABLES =====\r\nlet g:myvar = 10 &#34; Global variable\r\nlet l:myvar = 20 &#34; Local variable\r\nlet s:myvar = 30 &#34; Script variable\r\n\r\n&#34; ===== DATA TYPES =====\r\nlet string_var = &#34;hello&#34;\r\nlet number_var = 42\r\nlet float_var = 3.14\r\nlet list_var = [1, 2, 3, 4]\r\nlet dict_var = {&#39;name&#39;: &#39;John&#39;, &#39;age&#39;: 30}\r\n\r\n&#34; ===== CONDITIONALS =====\r\nif condition\r\n echo &#34;True&#34;\r\nelseif other_condition\r\n echo &#34;Also true&#34;\r\nelse\r\n echo &#34;False&#34;\r\nendif\r\n\r\n&#34; ===== LOOPS =====\r\nfor item in list_var\r\n echo item\r\nendfor\r\n\r\nwhile condition\r\n &#34; do something\r\nendwhile\r\n\r\n&#34; ===== FUNCTIONS =====\r\nfunction! MyFunction(arg1, arg2)\r\n let result = arg1 &#43; arg2\r\n return result\r\nendfunction\r\n\r\ncall MyFunction(5, 10) &#34; Call function\r\n\r\n&#34; ===== OPERATORS =====\r\n&#34; Concatenation uses . not &#43;\r\nlet full_str = &#34;Hello&#34; . &#34; &#34; . &#34;World&#34;<\/code><\/pre><h3 class=\"heading-element\" id=\"custom-commands-and-mappings\"><span>Custom Commands and Mappings<\/span>\n <a href=\"#custom-commands-and-mappings\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; ===== CUSTOM COMMANDS =====\r\ncommand! MyCommand echo &#34;Custom command executed&#34;\r\ncommand! GreetMe call GreetUser(&#34;Vim User&#34;)\r\n\r\n&#34; ===== KEYBINDINGS =====\r\nnnoremap &lt;leader&gt;t :tabnew&lt;CR&gt; &#34; Normal mode\r\ninoremap &lt;leader&gt;t &lt;Esc&gt;:tabnew&lt;CR&gt; &#34; Insert mode\r\nvnoremap &lt;leader&gt;y :&#39;&lt;,&#39;&gt;y&#43;&lt;CR&gt; &#34; Visual mode\r\n\r\n&#34; &lt;leader&gt; uses mapleader (usually \\\\ or ,)\r\n&#34; &lt;CR&gt; = Enter\r\n&#34; &lt;Esc&gt; = Escape\r\n&#34; &lt;C-x&gt; = Ctrl&#43;x\r\n&#34; &lt;S-x&gt; = Shift&#43;x\r\n\r\n&#34; Conditional mapping\r\nif has(&#39;nvim&#39;)\r\n nnoremap &lt;leader&gt;t :split&lt;CR&gt;\r\nelse\r\n nnoremap &lt;leader&gt;t :tabnew&lt;CR&gt;\r\nendif\r\n\r\n&#34; ===== USEFUL CONDITIONALS =====\r\nif has(&#39;python&#39;) &#34; Python support\r\nif has(&#39;clipboard&#39;) &#34; Clipboard support\r\nif has(&#39;nvim&#39;) &#34; Neovim specific<\/code><\/pre><hr>\n<h2 id=\"pro-tips\">Professional Workflows and Tips<\/h2>\r\n<h3 class=\"heading-element\" id=\"multi-file-editing\"><span>Multi-File Editing<\/span>\n <a href=\"#multi-file-editing\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Opening multiple files\r\nvim file1.txt file2.txt file3.txt\r\n\r\n&#34; Buffer commands\r\n:ls &#34; List all buffers\r\n:bn &#34; Next buffer\r\n:bp &#34; Previous buffer\r\n:bd &#34; Delete (close) buffer\r\n:b filename &#34; Go to buffer by name\r\n:b3 &#34; Go to buffer 3\r\n\r\n&#34; Tabs (Vim&#39;s &#34;workspaces&#34;)\r\n:tabnew &#34; New tab\r\n:tabnext \/ gt &#34; Next tab\r\n:tabprevious \/ gT &#34; Previous tab\r\n:tabclose &#34; Close tab\r\n\r\n&#34; Split windows\r\n:split file.txt &#34; Horizontal split\r\n:vsplit file.txt &#34; Vertical split\r\n&lt;C-w&gt;w &#34; Switch windows\r\n&lt;C-w&gt;= &#34; Equalize window sizes\r\n&lt;C-w&gt;_ &#34; Maximize current window<\/code><\/pre><h3 class=\"heading-element\" id=\"working-with-large-files\"><span>Working with Large Files<\/span>\n <a href=\"#working-with-large-files\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Disable features for speed\r\n:set noswapfile &#34; No swap file\r\n:set noundofile &#34; No undo history\r\n:set synmaxcol=80 &#34; Limit syntax highlighting column\r\n:syntax off &#34; Disable syntax highlighting\r\n\r\n&#34; Navigate efficiently\r\n:set nowrap &#34; Don&#39;t wrap lines\r\n:set lazyredraw &#34; Lazy redraw\r\nG &#34; Go to end\r\ngg &#34; Go to start\r\n50% &#34; Go to 50% of file\r\n\r\n&#34; Use grep to find in large files\r\n:grep pattern filename\r\n\r\n&#34; Stream editing\r\n:read !cat largefile | head -100 &#34; Read first 100 lines only<\/code><\/pre><h3 class=\"heading-element\" id=\"git-integration\"><span>Git Integration<\/span>\n <a href=\"#git-integration\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Using vim-fugitive plugin:\r\n:Git status &#34; Show git status\r\n:Git add % &#34; Stage current file\r\n:Git commit &#34; Commit changes\r\n:Git push &#34; Push changes\r\n:Git diff &#34; Show diff\r\n:Git log &#34; Show commit log\r\n:Gblame &#34; Show blame (who edited each line)\r\n\r\n&#34; Viewing diffs\r\n:Gvdiffsplit &#34; Show diff in vertical split\r\n]c \/ [c &#34; Navigate changes<\/code><\/pre><hr>\n<h2 id=\"troubleshooting\">Common Issues and Solutions<\/h2>\r\n<h3 class=\"heading-element\" id=\"frequent-problems\"><span>Frequent Problems<\/span>\n <a href=\"#frequent-problems\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Problem<\/th>\n <th>Solution<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Stuck in Insert Mode<\/strong><\/td>\n <td>Press <code>ESC<\/code> or <code>CTRL-C<\/code> to return to Normal Mode<\/td>\n <\/tr>\n <tr>\n <td><strong>Can&rsquo;t exit Vim<\/strong><\/td>\n <td>Type <code>:q<\/code> (quit) or <code>:q!<\/code> (quit without saving)<\/td>\n <\/tr>\n <tr>\n <td><strong>Accidentally entered Command Mode<\/strong><\/td>\n <td>Press <code>ESC<\/code> to go back to Normal Mode<\/td>\n <\/tr>\n <tr>\n <td><strong>Keys not responding as expected<\/strong><\/td>\n <td>Make sure you&rsquo;re in Normal Mode, not Insert Mode<\/td>\n <\/tr>\n <tr>\n <td><strong>Search highlighting won&rsquo;t clear<\/strong><\/td>\n <td>Run <code>:noh<\/code> or <code>:nohlsearch<\/code><\/td>\n <\/tr>\n <tr>\n <td><strong>Want to undo multiple changes<\/strong><\/td>\n <td>Press <code>u<\/code> multiple times or <code>20u<\/code> for 20 undos<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"performance-issues\"><span>Performance Issues<\/span>\n <a href=\"#performance-issues\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&#34; Diagnose startup time\r\nvim --startuptime startup.log\r\n\r\n&#34; Disable unneeded features\r\n:set nocompatible\r\n:set lazyredraw\r\n:set ttyfast\r\n\r\n&#34; Limit syntax highlighting for huge files\r\nautocmd BufReadPre * if getfsize(bufname(&#39;%&#39;)) &gt; 10000000 | syntax off | endif\r\n\r\n&#34; Reduce number of active plugins\r\n&#34; Keep only plugins you actually use<\/code><\/pre><hr>\n<h2 id=\"learning-path\">Recommended Learning Path<\/h2>\r\n<h3 class=\"heading-element\" id=\"phase-1-fundamentals-week-1-2\"><span>Phase 1: Fundamentals (Week 1-2)<\/span>\n <a href=\"#phase-1-fundamentals-week-1-2\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Master These First<\/strong><\/p>\n<p>Don&rsquo;t move to advanced techniques until these are muscle memory. Spend 15-30 minutes daily practicing.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<p><strong>Essential Skills:<\/strong><\/p>\n<ol>\n<li>Mode transitions (ESC, i, v, :)<\/li>\n<li>Basic motions (hjkl, w, b, $, ^)<\/li>\n<li>Operators (d, c, y)<\/li>\n<li>Save and quit (:w, :q, :wq)<\/li>\n<\/ol>\n<p><strong>Practice:<\/strong><\/p>\n<pre><code># Interactive Vim tutorial (built-in)\r\nvimtutor\r\n\r\n# Practice daily for 2 weeks\r\n# Open a file and practice moving, editing, saving<\/code><\/pre><h3 class=\"heading-element\" id=\"phase-2-intermediate-week-3-4\"><span>Phase 2: Intermediate (Week 3-4)<\/span>\n <a href=\"#phase-2-intermediate-week-3-4\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Skills:<\/strong><\/p>\n<ol>\n<li>Text objects (iw, i&quot;, i{, ap)<\/li>\n<li>Visual mode (v, V, CTRL-V)<\/li>\n<li>Search and replace (\/, :%s\/old\/new\/g)<\/li>\n<li>Registers (named, special)<\/li>\n<li>Macros (qa&hellip;q, @a)<\/li>\n<\/ol>\n<h3 class=\"heading-element\" id=\"phase-3-advanced-week-5\"><span>Phase 3: Advanced (Week 5+)<\/span>\n <a href=\"#phase-3-advanced-week-5\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Skills:<\/strong><\/p>\n<ol>\n<li>Plugin ecosystem<\/li>\n<li>Custom configuration<\/li>\n<li>Vimscript basics<\/li>\n<li>Advanced workflows<\/li>\n<li>Performance optimization<\/li>\n<\/ol>\n<hr>\n<h2 id=\"vim-resources\">Resources for Continued Learning<\/h2>\r\n<h3 class=\"heading-element\" id=\"official-and-quality-resources\"><span>Official and Quality Resources<\/span>\n <a href=\"#official-and-quality-resources\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ul>\n<li><strong>Built-in Help<\/strong>: <code>:help<\/code> or <code>:help topic<\/code> (comprehensive!)<\/li>\n<li><strong>Vim Manual<\/strong>: <a href=\"http:\/\/vimdoc.sourceforge.net\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">http:\/\/vimdoc.sourceforge.net\/<\/a><\/li>\n<li><strong>Interactive Tutor<\/strong>: <code>vimtutor<\/code> command<\/li>\n<li><strong>Learn Vim the Smart Way<\/strong>: Excellent free online guide<\/li>\n<li><strong>Practical Vim<\/strong> by Drew Neil: Best book on Vim<\/li>\n<li><strong>Daily Vim Tips<\/strong>: Twitter account @vimtips<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"community-and-tools\"><span>Community and Tools<\/span>\n <a href=\"#community-and-tools\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ul>\n<li><strong>r\/vim<\/strong> on Reddit: Active community<\/li>\n<li><strong>Stack Overflow<\/strong>: Tag: vim<\/li>\n<li><strong>GitHub<\/strong>: Thousands of plugins and configs<\/li>\n<li><strong>VimAwesome<\/strong>: Plugin directory<\/li>\n<\/ul>\n<hr>\n<h2 id=\"conclusion\">Conclusion: Your Vim Journey Begins<\/h2>\r\n<p>Vim has a steep learning curve, but the investment pays dividends. Once mastered, you&rsquo;ll edit text faster and more efficiently than ever before. Remember:<\/p>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Key Takeaways<\/strong><\/p>\n<ol>\n<li><strong>Modes are your friend<\/strong>: Normal mode for navigation, Insert mode for typing<\/li>\n<li><strong>Grammar is power<\/strong>: Learn operators and motions, then combine them<\/li>\n<li><strong>Text objects are essential<\/strong>: They make editing feel smart and intuitive<\/li>\n<li><strong>Practice daily<\/strong>: Vim skills compound over time<\/li>\n<li><strong>Customize gradually<\/strong>: Start simple, add plugins and config as needed<\/li>\n<li><strong>Join the community<\/strong>: Vim users love sharing tips and tricks<\/li>\n<\/ol><\/div>\n <\/div>\n<\/div>\n<p><strong>Start with the basics, practice consistently, and gradually explore advanced features. Within a few weeks, you&rsquo;ll wonder how you ever edited without Vim.<\/strong><\/p>\n<p>Happy editing! \ud83d\udda4<\/p>\n<hr>\n<hr>\n<h2 id=\"workflow-diagrams\">Complete Vim Workflow Diagrams<\/h2>\r\n<h3 class=\"heading-element\" id=\"editing-workflow-progression\"><span>Editing Workflow Progression<\/span>\n <a href=\"#editing-workflow-progression\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>VIM EDITING WORKFLOW \u2014 From Simple to Complex:\r\n\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\r\n\r\nLEVEL 1: Basic Editing (First Week)\r\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 1. vim file.txt \u2192 Open file \u2502\r\n\u2502 2. i \u2192 Enter insert mode \u2502\r\n\u2502 3. Type text... \u2192 Add content \u2502\r\n\u2502 4. ESC \u2192 Back to normal mode \u2502\r\n\u2502 5. :wq \u2192 Save and quit \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\nLEVEL 2: Motion-Based Editing (Week 2-3)\r\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 1. vim file.txt \u2192 Open file \u2502\r\n\u2502 2. \/function \u2192 Search for &#34;function&#34; \u2502\r\n\u2502 3. n \u2192 Next match \u2502\r\n\u2502 4. ciw \u2192 Change inner word \u2502\r\n\u2502 5. newname \u2192 Type replacement \u2502\r\n\u2502 6. ESC \u2192 Back to normal \u2502\r\n\u2502 7. n \u2192 Next occurrence \u2502\r\n\u2502 8. . \u2192 Repeat change (ciw) \u2502\r\n\u2502 9. :wq \u2192 Save and quit \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\nLEVEL 3: Advanced Multi-File Workflow (Week 4&#43;)\r\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 1. vim -p file1 file2 \u2192 Open in tabs \u2502\r\n\u2502 2. :vimgrep \/TODO\/ ** \u2192 Search all files \u2502\r\n\u2502 3. :copen \u2192 View results \u2502\r\n\u2502 4. :cn \u2192 Next result \u2502\r\n\u2502 5. ciw \u2192 Change word \u2502\r\n\u2502 6. qa \u2192 Start recording macro \u2502\r\n\u2502 7. ...edits... \u2192 Make changes \u2502\r\n\u2502 8. q \u2192 Stop recording \u2502\r\n\u2502 9. @a \u2192 Replay on next item \u2502\r\n\u2502 10. :bufdo update \u2192 Save all buffers \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/code><\/pre><h3 class=\"heading-element\" id=\"problem-solving-decision-tree\"><span>Problem-Solving Decision Tree<\/span>\n <a href=\"#problem-solving-decision-tree\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>WHICH VIM FEATURE TO USE?\r\n\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\r\n\r\nNeed to edit text?\r\n\u2502\r\n\u251c\u2500 Single change?\r\n\u2502 \u251c\u2500 YES \u2192 Use operators (d, c, y) &#43; motions (w, $, })\r\n\u2502 \u2514\u2500 NO \u2192 Continue below\r\n\u2502\r\n\u251c\u2500 Repetitive task?\r\n\u2502 \u251c\u2500 Same change many times \u2192 Record macro (qa...q, @a)\r\n\u2502 \u251c\u2500 Pattern-based \u2192 Use :g\/pattern\/command\r\n\u2502 \u2514\u2500 Multiple files \u2192 :bufdo or :argdo\r\n\u2502\r\n\u251c\u2500 Finding code?\r\n\u2502 \u251c\u2500 In current file \u2192 Use \/ search, *, #\r\n\u2502 \u251c\u2500 Multiple files \u2192 :vimgrep or :grep\r\n\u2502 \u251c\u2500 Jump to definition \u2192 gd or CTRL-]\r\n\u2502 \u2514\u2500 Recent locations \u2192 CTRL-O \/ CTRL-I\r\n\u2502\r\n\u251c\u2500 Navigating?\r\n\u2502 \u251c\u2500 Within file \u2192 Marks (ma, `a)\r\n\u2502 \u251c\u2500 Between files \u2192 Buffers (:bn, :bp)\r\n\u2502 \u251c\u2500 Multiple layouts \u2192 Windows (CTRL-W s\/v)\r\n\u2502 \u2514\u2500 Different projects \u2192 Tabs (gt, gT)\r\n\u2502\r\n\u2514\u2500 Large-scale refactor?\r\n \u251c\u2500 Complex \u2192 Record macro &#43; :g command\r\n \u251c\u2500 Multiple files \u2192 :bufdo with macro\r\n \u2514\u2500 Precise pattern \u2192 Regex substitution<\/code><\/pre><h3 class=\"heading-element\" id=\"real-world-refactoring-example\"><span>Real-World Refactoring Example<\/span>\n <a href=\"#real-world-refactoring-example\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>SCENARIO: Rename variable &#39;oldVar&#39; to &#39;newVar&#39; across project\r\n\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\r\n\r\nMETHOD 1: Manual (Small Project)\r\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 1. \/oldVar&lt;Enter&gt; Search for oldVar \u2502\r\n\u2502 2. * Highlight all occurrences \u2502\r\n\u2502 3. ciw Change inner word \u2502\r\n\u2502 4. newVar&lt;ESC&gt; Type new name \u2502\r\n\u2502 5. n Next occurrence \u2502\r\n\u2502 6. . Repeat change \u2502\r\n\u2502 7. Repeat steps 5-6... \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\nMETHOD 2: Macro (Medium Project)\r\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 1. \/oldVar&lt;Enter&gt; Search first occurrence \u2502\r\n\u2502 2. qa Start recording to register a \u2502\r\n\u2502 3. ciwn ewVar&lt;ESC&gt; Change word \u2502\r\n\u2502 4. n Next occurrence \u2502\r\n\u2502 5. q Stop recording \u2502\r\n\u2502 6. 100@a Replay 100 times (safe) \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\nMETHOD 3: Substitution (Large Project)\r\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 1. :args **\/*.js Add all JS files to arglist \u2502\r\n\u2502 2. :argdo %s\/\\&lt;oldVar\\&gt;\/newVar\/gc \u2502\r\n\u2502 Substitute with confirmation \u2502\r\n\u2502 3. :argdo update Save all changed files \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\nMETHOD 4: Advanced (Huge Project)\r\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 1. :vimgrep \/\\&lt;oldVar\\&gt;\/ **\/*.js \u2502\r\n\u2502 Find all occurrences \u2502\r\n\u2502 2. :copen Open quickfix with results \u2502\r\n\u2502 3. :cfdo %s\/\\&lt;oldVar\\&gt;\/newVar\/g \u2502\r\n\u2502 Substitute in all files \u2502\r\n\u2502 4. :cfdo update Save all changed files \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/code><\/pre><h3 class=\"heading-element\" id=\"text-object-combination-matrix\"><span>Text Object Combination Matrix<\/span>\n <a href=\"#text-object-combination-matrix\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>TEXT OBJECT POWER COMBINATIONS\r\n\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\r\n\r\nOPERATOR \u00d7 TEXT OBJECT = POWERFUL COMMANDS\r\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\r\n \u2502 iw \u2502 aw \u2502 i&#34; \u2502 i( \u2502 i{ \u2502 ap \u2502\r\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\r\n d \u2502 diw \u2502 daw \u2502 di&#34; \u2502 di( \u2502 di{ \u2502 dap \u2502\r\n \u2502 delete\u2502delete \u2502delete \u2502delete \u2502delete \u2502delete \u2502\r\n \u2502 word \u2502 word&#43; \u2502inside \u2502inside \u2502inside \u2502para \u2502\r\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\r\n c \u2502 ciw \u2502 caw \u2502 ci&#34; \u2502 ci( \u2502 ci{ \u2502 cap \u2502\r\n \u2502change \u2502change \u2502change \u2502change \u2502change \u2502change \u2502\r\n \u2502 word \u2502 word&#43; \u2502quote \u2502parens \u2502braces \u2502para \u2502\r\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\r\n y \u2502 yiw \u2502 yaw \u2502 yi&#34; \u2502 yi( \u2502 yi{ \u2502 yap \u2502\r\n \u2502 yank \u2502 yank \u2502 yank \u2502 yank \u2502 yank \u2502 yank \u2502\r\n \u2502 word \u2502 word&#43; \u2502quote \u2502parens \u2502braces \u2502para \u2502\r\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\r\n v \u2502 viw \u2502 vaw \u2502 vi&#34; \u2502 vi( \u2502 vi{ \u2502 vap \u2502\r\n \u2502select \u2502select \u2502select \u2502select \u2502select \u2502select \u2502\r\n \u2502 word \u2502 word&#43; \u2502quote \u2502parens \u2502braces \u2502para \u2502\r\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\r\n gU \u2502 gUiw \u2502 gUaw \u2502 gUi&#34; \u2502 gUi( \u2502 gUi{ \u2502 gUap \u2502\r\n \u2502UPPER \u2502UPPER \u2502UPPER \u2502UPPER \u2502UPPER \u2502UPPER \u2502\r\n \u2502 word \u2502 word&#43; \u2502quote \u2502parens \u2502braces \u2502para \u2502\r\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\nMost useful combinations:\r\n\u2022 ciw - Change inner word (rename variable)\r\n\u2022 di&#34; - Delete inside quotes (remove string content)\r\n\u2022 ci( - Change inside parentheses (edit function args)\r\n\u2022 dap - Delete paragraph (remove code block)\r\n\u2022 yap - Yank paragraph (copy function\/block)<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"quick-reference-card\"><span>Quick Reference Card<\/span>\n <a href=\"#quick-reference-card\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><pre><code>\u2554\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2557\r\n\u2551 VIM COMPLETE QUICK REFERENCE \u2551\r\n\u255a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u255d\r\n\r\n\u250c\u2500 MODES \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 ESC Return to Normal mode (from any mode) \u2502\r\n\u2502 i\/I Insert before cursor \/ at line start \u2502\r\n\u2502 a\/A Append after cursor \/ at line end \u2502\r\n\u2502 o\/O Open line below \/ above \u2502\r\n\u2502 v\/V\/Ctrl-V Visual: character \/ line \/ block \u2502\r\n\u2502 : Command mode (Ex commands) \u2502\r\n\u2502 R Replace mode (overwrite) \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\n\u250c\u2500 MOVEMENT \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 h\/j\/k\/l Left \/ Down \/ Up \/ Right \u2502\r\n\u2502 w\/W Next word \/ WORD \u2502\r\n\u2502 b\/B Previous word \/ WORD \u2502\r\n\u2502 e\/E End of word \/ WORD \u2502\r\n\u2502 0\/^\/$ Start of line \/ First non-blank \/ End \u2502\r\n\u2502 gg\/G Start \/ end of file \u2502\r\n\u2502 {\/} Previous \/ next paragraph \u2502\r\n\u2502 % Jump to matching bracket \u2502\r\n\u2502 fx\/Fx Find char &#39;x&#39; forward \/ backward \u2502\r\n\u2502 *\/# Search word under cursor forward \/ backward \u2502\r\n\u2502 Ctrl-O\/Ctrl-I Jump to older \/ newer position \u2502\r\n\u2502 ma \/ `a Set mark &#39;a&#39; \/ Jump to mark &#39;a&#39; \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\n\u250c\u2500 EDITING \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 x\/X Delete char at \/ before cursor \u2502\r\n\u2502 dd Delete line \u2502\r\n\u2502 D Delete to end of line \u2502\r\n\u2502 d{motion} Delete motion (dw, d$, dip, di&#34;) \u2502\r\n\u2502 c{motion} Change motion (delete &#43; insert mode) \u2502\r\n\u2502 y{motion} Yank (copy) motion (yy = yank line) \u2502\r\n\u2502 p\/P Paste after \/ before cursor \u2502\r\n\u2502 u\/Ctrl-R Undo \/ Redo \u2502\r\n\u2502 . Repeat last change \u2502\r\n\u2502 J Join lines \u2502\r\n\u2502 ~ Toggle case \u2502\r\n\u2502 gU\/gu{motion} Uppercase \/ lowercase motion \u2502\r\n\u2502 Ctrl-A\/Ctrl-X Increment \/ decrement number \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\n\u250c\u2500 TEXT OBJECTS (use with operators: d, c, y, v, gU, etc.) \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 iw\/aw Inner word \/ a word (with space) \u2502\r\n\u2502 i&#34;\/a&#34; Inside \/ around double quotes \u2502\r\n\u2502 i&#39;\/a&#39; Inside \/ around single quotes \u2502\r\n\u2502 i(\/a( or ib\/ab Inside \/ around parentheses \u2502\r\n\u2502 i{\/a{ or iB\/aB Inside \/ around braces \u2502\r\n\u2502 i[\/a[ Inside \/ around brackets \u2502\r\n\u2502 it\/at Inside \/ around HTML\/XML tag \u2502\r\n\u2502 ip\/ap Inner \/ a paragraph \u2502\r\n\u2502 is\/as Inner \/ a sentence \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\n\u250c\u2500 SEARCH &amp; REPLACE \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 \/pattern Search forward \u2502\r\n\u2502 ?pattern Search backward \u2502\r\n\u2502 n\/N Next \/ previous match \u2502\r\n\u2502 *\/# Search word under cursor forward \/ backward \u2502\r\n\u2502 :noh Clear search highlighting \u2502\r\n\u2502 :s\/old\/new\/ Replace first on line \u2502\r\n\u2502 :s\/old\/new\/g Replace all on line \u2502\r\n\u2502 :%s\/old\/new\/g Replace all in file \u2502\r\n\u2502 :%s\/old\/new\/gc Replace all with confirmation \u2502\r\n\u2502 :10,20s\/old\/new\/g Replace in lines 10-20 \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\n\u250c\u2500 FILES &amp; BUFFERS \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 :w Save file \u2502\r\n\u2502 :w filename Save as filename \u2502\r\n\u2502 :q Quit (fails if unsaved) \u2502\r\n\u2502 :q! Quit without saving \u2502\r\n\u2502 :wq or :x Save and quit \u2502\r\n\u2502 :e filename Open file \u2502\r\n\u2502 :ls List buffers \u2502\r\n\u2502 :bn\/:bp Next \/ previous buffer \u2502\r\n\u2502 :bd Delete (close) buffer \u2502\r\n\u2502 :b3 Switch to buffer 3 \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\n\u250c\u2500 WINDOWS &amp; TABS \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 :split\/:sp Horizontal split \u2502\r\n\u2502 :vsplit\/:vsp Vertical split \u2502\r\n\u2502 Ctrl-W s\/v Split horizontal \/ vertical \u2502\r\n\u2502 Ctrl-W h\/j\/k\/l Navigate to window left\/down\/up\/right \u2502\r\n\u2502 Ctrl-W w Cycle windows \u2502\r\n\u2502 Ctrl-W q Close window \u2502\r\n\u2502 Ctrl-W = Equalize window sizes \u2502\r\n\u2502 :tabnew New tab \u2502\r\n\u2502 gt\/gT Next \/ previous tab \u2502\r\n\u2502 :tabclose Close tab \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\n\u250c\u2500 VISUAL MODE \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 v Character-wise visual \u2502\r\n\u2502 V Line-wise visual \u2502\r\n\u2502 Ctrl-V Block visual (rectangular selection) \u2502\r\n\u2502 o Jump to other end of selection \u2502\r\n\u2502 &gt;\/&lt;&lt; Indent right \/ left \u2502\r\n\u2502 d\/c\/y Delete \/ change \/ yank selection \u2502\r\n\u2502 u\/U Lowercase \/ uppercase \u2502\r\n\u2502 = Auto-indent \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\n\u250c\u2500 MACROS \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 qa Start recording macro in register &#39;a&#39; \u2502\r\n\u2502 q Stop recording \u2502\r\n\u2502 @a Play macro from register &#39;a&#39; \u2502\r\n\u2502 @@ Repeat last macro \u2502\r\n\u2502 10@a Execute macro 10 times \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\n\u250c\u2500 REGISTERS \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 &#34;ayy Yank line to register &#39;a&#39; \u2502\r\n\u2502 &#34;ap Paste from register &#39;a&#39; \u2502\r\n\u2502 &#34;0 Last yank \u2502\r\n\u2502 &#34;1-&#34;9 Last 9 deletions \u2502\r\n\u2502 &#34;&#43; System clipboard \u2502\r\n\u2502 :reg Show all registers \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\n\u250c\u2500 MARKS &amp; JUMPS \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 ma Set mark &#39;a&#39; at cursor \u2502\r\n\u2502 `a Jump to exact position of mark &#39;a&#39; \u2502\r\n\u2502 &#39;a Jump to line of mark &#39;a&#39; \u2502\r\n\u2502 `` Jump to position before last jump \u2502\r\n\u2502 `. Jump to last change \u2502\r\n\u2502 Ctrl-O\/Ctrl-I Navigate jump history older \/ newer \u2502\r\n\u2502 :marks Show all marks \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\n\u250c\u2500 FOLDING \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 zo\/zc Open \/ close fold \u2502\r\n\u2502 za Toggle fold \u2502\r\n\u2502 zM\/zR Close all \/ open all folds \u2502\r\n\u2502 zf{motion} Create fold \u2502\r\n\u2502 zd Delete fold \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\n\u250c\u2500 ADVANCED COMMANDS \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 :g\/pattern\/d Delete all lines matching pattern \u2502\r\n\u2502 :v\/pattern\/d Delete lines NOT matching pattern \u2502\r\n\u2502 :sort Sort lines \u2502\r\n\u2502 :! command Execute external command \u2502\r\n\u2502 :%!sort Filter file through external command \u2502\r\n\u2502 :r!date Insert output of command \u2502\r\n\u2502 :w !sudo tee % Save with sudo \u2502\r\n\u2502 gd\/gD Go to local \/ global declaration \u2502\r\n\u2502 Ctrl-N\/Ctrl-P Autocomplete next \/ previous \u2502\r\n\u2502 Ctrl-X Ctrl-F Filename completion \u2502\r\n\u2502 vimdiff f1 f2 Diff two files \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\n\u250c\u2500 QUICKFIX \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 :vimgrep \/pat\/ ** Search all files \u2502\r\n\u2502 :copen Open quickfix window \u2502\r\n\u2502 :cn\/:cp Next \/ previous error \u2502\r\n\u2502 :cclose Close quickfix \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\n\u250c\u2500 HELP &amp; EXIT \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\n\u2502 :help topic Get help on topic \u2502\r\n\u2502 :q Quit help window \u2502\r\n\u2502 vimtutor Interactive Vim tutorial (run from shell) \u2502\r\n\u2502 :qa! Quit all windows without saving (emergency exit!) \u2502\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n\r\n\u2554\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2557\r\n\u2551 REMEMBER: Vim is a language. Learn operators, motions, and combine! \u2551\r\n\u2551 Formula: OPERATOR &#43; [COUNT] &#43; MOTION\/TEXT-OBJECT \u2551\r\n\u2551 Example: d3w (delete 3 words), ciw (change inner word), yap (yank par) \u2551\r\n\u255a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u255d<\/code><\/pre>"},{"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","link":"https:\/\/samir.pages.dev\/posts\/git-github\/","pubDate":"Wed, 20 Nov 2024 00:00:00 +0000","author":"netproxy@duck.com (Samir)","guid":"https:\/\/samir.pages.dev\/posts\/git-github\/","category":"Software-Engineering","description":"<img src=\"https:\/\/samir.pages.dev\/files\/git-github.webp\" alt=\"featured image\" referrerpolicy=\"no-referrer\"><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Why Learn Git and GitHub?<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p>Git and GitHub are <strong>fundamental to modern software development<\/strong>. Whether you&rsquo;re contributing to open-source projects, working in a team, or building your own products, understanding version control is essential for:<\/p>\n<ul>\n<li><strong>Tracking changes<\/strong> in your codebase over time<\/li>\n<li><strong>Collaborating safely<\/strong> with multiple developers<\/li>\n<li><strong>Reversing mistakes<\/strong> without losing work<\/li>\n<li><strong>Understanding code history<\/strong> and decisions<\/li>\n<li><strong>Automating deployment<\/strong> through CI\/CD pipelines<\/li>\n<li><strong>Contributing to open-source<\/strong> communities<\/li>\n<\/ul>\n<p>This guide takes you from complete beginner to advanced user, with practical examples you can use immediately.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<h2 class=\"heading-element\" id=\"1-introduction-to-git-and-version-control\"><span>1. Introduction to Git and Version Control<\/span>\n <a href=\"#1-introduction-to-git-and-version-control\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"11-what-is-git\"><span>1.1 What is Git?<\/span>\n <a href=\"#11-what-is-git\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Git is a <strong>distributed version control system (VCS)<\/strong> designed by Linus Torvalds in 2005. It tracks changes in files, enables collaboration, and maintains complete project history.<\/p>\n<p><strong>Key Characteristics:<\/strong><\/p>\n<ul>\n<li><strong>Distributed<\/strong>: Every developer has a complete copy of the repository<\/li>\n<li><strong>Fast<\/strong>: Most operations happen locally without server communication<\/li>\n<li><strong>Flexible<\/strong>: Supports various workflows and branching strategies<\/li>\n<li><strong>Safe<\/strong>: Cryptographic integrity checking prevents data corruption<\/li>\n<li><strong>Powerful<\/strong>: Complex branching, merging, and history rewriting capabilities<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"12-git-vs-github\"><span>1.2 Git vs GitHub<\/span>\n <a href=\"#12-git-vs-github\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>These terms are often confused but are distinct:<\/p>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Aspect<\/th>\n <th>Git<\/th>\n <th>GitHub<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Type<\/strong><\/td>\n <td>Version Control System<\/td>\n <td>Web hosting platform<\/td>\n <\/tr>\n <tr>\n <td><strong>Location<\/strong><\/td>\n <td>Runs locally on your computer<\/td>\n <td>Cloud-based service<\/td>\n <\/tr>\n <tr>\n <td><strong>Purpose<\/strong><\/td>\n <td>Track and manage code changes<\/td>\n <td>Host, collaborate, and share repositories<\/td>\n <\/tr>\n <tr>\n <td><strong>Scope<\/strong><\/td>\n <td>Local version control<\/td>\n <td>Remote repository + collaboration tools<\/td>\n <\/tr>\n <tr>\n <td><strong>Dependencies<\/strong><\/td>\n <td>Standalone tool<\/td>\n <td>Requires Git to function<\/td>\n <\/tr>\n <tr>\n <td><strong>Features<\/strong><\/td>\n <td>Branching, merging, staging<\/td>\n <td>Pull requests, issues, actions, wiki<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\">Think of it this way: <strong>Git<\/strong> is the engine that manages your code versions, while <strong>GitHub<\/strong> is the garage where you park your code and invite others to contribute.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"13-core-concepts-you-need-to-know\"><span>1.3 Core Concepts You Need to Know<\/span>\n <a href=\"#13-core-concepts-you-need-to-know\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"the-three-state-model\"><span>The Three-State Model<\/span>\n <a href=\"#the-three-state-model\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Git has three main states where your files reside:<\/p>\n<p><strong>1. Working Directory (Modified)<\/strong><\/p>\n<pre><code>Your local files that you actively edit.\r\nStatus: Untracked or Modified<\/code><\/pre><p><strong>2. Staging Area (Staged)<\/strong><\/p>\n<pre><code>Files marked for inclusion in the next commit.\r\nStatus: Staged<\/code><\/pre><p><strong>3. Git Repository (Committed)<\/strong><\/p>\n<pre><code>Permanent snapshots stored in .git directory.\r\nStatus: Committed<\/code><\/pre><p><strong>Flow:<\/strong><\/p>\n<pre><code>Modify Files (Working) \u2192 Stage Changes \u2192 Commit to Repository<\/code><\/pre><h4 class=\"heading-element\" id=\"the-git-object-model\"><span>The Git Object Model<\/span>\n <a href=\"#the-git-object-model\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Git stores data as a directed acyclic graph (DAG) of <strong>four main object types<\/strong>:<\/p>\n<pre><code>Commit - points to tree &#43; parent commit &#43; author info\r\n \u2193\r\nTree - represents directory structure\r\n \u2193\r\nBlob - actual file contents (immutable)\r\n \u2193\r\nTag - reference to specific commit (optional)<\/code><\/pre><p><strong>Everything is Content-Addressable:<\/strong><\/p>\n<ul>\n<li>Each object has a unique SHA-1 hash based on its content<\/li>\n<li>Even tiny changes produce completely different hashes<\/li>\n<li>Impossible to change history without detection<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"2-getting-started-with-git\"><span>2. Getting Started with Git<\/span>\n <a href=\"#2-getting-started-with-git\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"21-installation\"><span>2.1 Installation<\/span>\n <a href=\"#21-installation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>macOS (using Homebrew):<\/strong><\/p>\n<pre><code>brew install git\r\ngit --version<\/code><\/pre><p><strong>Ubuntu\/Debian:<\/strong><\/p>\n<pre><code>sudo apt update\r\nsudo apt install git\r\ngit --version<\/code><\/pre><p><strong>Windows:<\/strong><\/p>\n<pre><code># Download from: https:\/\/git-scm.com\/download\/win\r\n# Run installer and follow prompts\r\n# Verify installation\r\ngit --version<\/code><\/pre><h3 class=\"heading-element\" id=\"22-initial-configuration\"><span>2.2 Initial Configuration<\/span>\n <a href=\"#22-initial-configuration\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>After installation, configure Git with your identity:<\/p>\n<pre><code># Set your name (appears in commits)\r\ngit config --global user.name &#34;Your Name&#34;\r\n\r\n# Set your email (appears in commits)\r\ngit config --global user.email &#34;you@example.com&#34;\r\n\r\n# Set default editor for commit messages\r\ngit config --global core.editor &#34;vim&#34;\r\n\r\n# Verify configuration\r\ngit config --list\r\n\r\n# View only global settings\r\ngit config --global --list\r\n\r\n# View configuration files\r\ncat ~\/.gitconfig<\/code><\/pre><div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Global vs Local Configuration<\/strong>: <code>--global<\/code> flag applies to all repositories on your machine. Use local configuration (without <code>--global<\/code>) for project-specific settings.<\/div>\n <\/div>\n<\/div>\n<p><strong>Useful Global Configuration:<\/strong><\/p>\n<pre><code># Enable colored output\r\ngit config --global color.ui true\r\n\r\n# Set default branch name to main\r\ngit config --global init.defaultBranch main\r\n\r\n# Enable automatic garbage collection\r\ngit config --global gc.auto 4096\r\n\r\n# Configure paging (less) for large outputs\r\ngit config --global core.pager &#34;less -x1,5&#34;\r\n\r\n# Add helpful aliases\r\ngit config --global alias.st status\r\ngit config --global alias.co checkout\r\ngit config --global alias.br branch\r\ngit config --global alias.ci commit\r\ngit config --global alias.unstage &#39;restore --staged&#39;\r\ngit config --global alias.last &#39;log -1 HEAD&#39;\r\ngit config --global alias.visual &#39;log --graph --oneline --all&#39;<\/code><\/pre><h3 class=\"heading-element\" id=\"23-first-repository\"><span>2.3 First Repository<\/span>\n <a href=\"#23-first-repository\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"initialize-a-new-repository\"><span>Initialize a New Repository<\/span>\n <a href=\"#initialize-a-new-repository\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code># Create a new directory\r\nmkdir my-project\r\ncd my-project\r\n\r\n# Initialize Git repository\r\ngit init\r\n\r\n# Verify .git directory was created\r\nls -la<\/code><\/pre><p>When you run <code>git init<\/code>, Git creates a hidden <code>.git<\/code> directory containing:<\/p>\n<pre><code>.git\/\r\n \u251c\u2500\u2500 objects\/ # Stores all commits, trees, blobs\r\n \u251c\u2500\u2500 refs\/ # References to branches and tags\r\n \u251c\u2500\u2500 HEAD # Points to current branch\r\n \u251c\u2500\u2500 config # Repository configuration\r\n \u251c\u2500\u2500 hooks\/ # Scripts that run on Git events\r\n \u2514\u2500\u2500 logs\/ # Reference logs<\/code><\/pre><h4 class=\"heading-element\" id=\"set-repository-identity-optional\"><span>Set Repository Identity (Optional)<\/span>\n <a href=\"#set-repository-identity-optional\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code># View current config (inherits from global)\r\ngit config --local user.name\r\n\r\n# Override with project-specific settings\r\ngit config --local user.name &#34;Project-Specific Name&#34;\r\ngit config --local user.email &#34;project@example.com&#34;\r\n\r\n# These only apply to this repository<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"3-daily-workflow-the-essential-commands\"><span>3. Daily Workflow: The Essential Commands<\/span>\n <a href=\"#3-daily-workflow-the-essential-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"31-checking-status\"><span>3.1 Checking Status<\/span>\n <a href=\"#31-checking-status\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The most important command to understand your repository state:<\/p>\n<pre><code># Full status output\r\ngit status\r\n\r\n# Short format (more concise)\r\ngit status -s\r\n\r\n# Include branch tracking info\r\ngit status -sb\r\n\r\n# Show untracked files separately\r\ngit status --short<\/code><\/pre><p><strong>Status Output Legend:<\/strong><\/p>\n<pre><code> M = Modified\r\nA = Added\r\nD = Deleted\r\nR = Renamed\r\nC = Copied\r\nU = Updated but unmerged\r\n?? = Untracked\r\n\r\nFirst column = staging area\r\nSecond column = working directory<\/code><\/pre><h3 class=\"heading-element\" id=\"32-staging-changes\"><span>3.2 Staging Changes<\/span>\n <a href=\"#32-staging-changes\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Staging is how you select which changes to include in your next commit.<\/p>\n<h4 class=\"heading-element\" id=\"adding-specific-files\"><span>Adding Specific Files<\/span>\n <a href=\"#adding-specific-files\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code># Stage specific file\r\ngit add filename.txt\r\n\r\n# Stage multiple files\r\ngit add file1.js file2.js file3.js\r\n\r\n# Stage all changes in current directory\r\ngit add .\r\n\r\n# Stage all changes everywhere\r\ngit add -A\r\n\r\n# Stage only modified files (not new files)\r\ngit add -u\r\n\r\n# Stage with confirmation (interactive)\r\ngit add -i<\/code><\/pre><h4 class=\"heading-element\" id=\"partial-staging\"><span>Partial Staging<\/span>\n <a href=\"#partial-staging\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code># Interactive patch staging - choose hunks to stage\r\ngit add -p\r\n\r\n# View what you staged\r\ngit diff --staged\r\n\r\n# Unstage specific file\r\ngit restore --staged filename.txt\r\n\r\n# Unstage all changes\r\ngit restore --staged .<\/code><\/pre><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Staging Best Practices<\/strong>: Stage related changes together in logical units. This makes commits more meaningful and makes history easier to understand. Use <code>git add -p<\/code> to review changes before staging.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"33-creating-commits\"><span>3.3 Creating Commits<\/span>\n <a href=\"#33-creating-commits\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>A commit is a permanent snapshot of your staged changes.<\/p>\n<pre><code># Simple commit with message\r\ngit commit -m &#34;Add user authentication feature&#34;\r\n\r\n# Multi-line commit message (opens editor)\r\ngit commit -m &#34;Add authentication&#34; -m &#34;Implement JWT token validation&#34;\r\n\r\n# Automatically stage tracked files and commit\r\ngit commit -am &#34;Fix login bug&#34;\r\n\r\n# Open editor for detailed commit message\r\ngit commit\r\n\r\n# Amend previous commit (reuse same message)\r\ngit commit --amend --no-edit\r\n\r\n# Amend and edit message\r\ngit commit --amend<\/code><\/pre><h4 class=\"heading-element\" id=\"commit-message-best-practices\"><span>Commit Message Best Practices<\/span>\n <a href=\"#commit-message-best-practices\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Write commit messages that explain the <strong>why<\/strong>, not just the <strong>what<\/strong>:<\/p>\n<pre><code># \u274c BAD: What is being done\r\ngit commit -m &#34;Update login&#34;\r\n\r\n# \u274c BAD: Too vague\r\ngit commit -m &#34;Fix stuff&#34;\r\n\r\n# \u2705 GOOD: What and why\r\ngit commit -m &#34;Implement JWT authentication for API endpoints\r\n\r\n- Add token generation on login\r\n- Validate tokens on protected routes\r\n- Store refresh tokens securely\r\n- Fixes #123&#34;\r\n\r\n# \u2705 GOOD: Conventional Commits format\r\ngit commit -m &#34;feat(auth): implement JWT token validation\r\n\r\nEnables secure API authentication for user endpoints.\r\nTokens expire after 24 hours and include refresh mechanism.\r\n\r\nCloses #456&#34;<\/code><\/pre><p><strong>Conventional Commits Format<\/strong> (widely adopted):<\/p>\n<pre><code>type(scope): subject\r\n\r\nbody\r\n\r\nfooter<\/code><\/pre><p>Where <code>type<\/code> is one of:<\/p>\n<ul>\n<li><code>feat:<\/code> New feature<\/li>\n<li><code>fix:<\/code> Bug fix<\/li>\n<li><code>docs:<\/code> Documentation changes<\/li>\n<li><code>style:<\/code> Code style (formatting, missing semicolons, etc.)<\/li>\n<li><code>refactor:<\/code> Code refactoring without feature changes<\/li>\n<li><code>perf:<\/code> Performance improvements<\/li>\n<li><code>test:<\/code> Adding or updating tests<\/li>\n<li><code>chore:<\/code> Build process, dependencies, tooling<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"34-viewing-history\"><span>3.4 Viewing History<\/span>\n <a href=\"#34-viewing-history\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Understanding what&rsquo;s been done is crucial for maintenance and debugging.<\/p>\n<pre><code># Show all commits in current branch\r\ngit log\r\n\r\n# One line per commit (compact)\r\ngit log --oneline\r\n\r\n# Show last 10 commits\r\ngit log -n 10\r\n\r\n# Show commits by specific author\r\ngit log --author=&#34;John Doe&#34;\r\n\r\n# Show commits in date range\r\ngit log --since=&#34;2025-01-01&#34; --until=&#34;2025-06-01&#34;\r\n\r\n# Show commits matching search term\r\ngit log --grep=&#34;authentication&#34;\r\n\r\n# Show commits affecting specific file\r\ngit log filename.js\r\n\r\n# Show changes in each commit (-p = patch)\r\ngit log -p\r\n\r\n# Pretty graphical view with branches\r\ngit log --graph --oneline --all --decorate\r\n\r\n# Show statistics\r\ngit log --stat\r\n\r\n# Custom format\r\ngit log --pretty=format:&#34;%h %an %ar - %s&#34;<\/code><\/pre><h4 class=\"heading-element\" id=\"understanding-the-output\"><span>Understanding the Output<\/span>\n <a href=\"#understanding-the-output\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>commit abc1234def5678 (HEAD -&gt; main)\r\nAuthor: John Doe &lt;john@example.com&gt;\r\nDate: Mon Nov 3 10:30:45 2025 &#43;0800\r\n\r\n Add JWT authentication\r\n\r\n - Implement token generation\r\n - Validate tokens on routes\r\n\r\ncommit xyz9876abc3210\r\nAuthor: Jane Smith &lt;jane@example.com&gt;\r\nDate: Sun Nov 2 14:22:10 2025 &#43;0800\r\n\r\n Fix user registration bug<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"4-understanding-and-using-branches\"><span>4. Understanding and Using Branches<\/span>\n <a href=\"#4-understanding-and-using-branches\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Branches allow parallel development without interfering with each other.<\/p>\n<h3 class=\"heading-element\" id=\"41-branch-fundamentals\"><span>4.1 Branch Fundamentals<\/span>\n <a href=\"#41-branch-fundamentals\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>What is a Branch?<\/strong> A branch is simply a lightweight pointer to a specific commit. The default branch is <code>main<\/code> (or <code>master<\/code> in older repos). Creating a branch doesn&rsquo;t copy code \u2014 it just creates a reference to a commit.<\/div>\n <\/div>\n<\/div>\n<p><strong>Why Use Branches?<\/strong><\/p>\n<ul>\n<li><strong>Feature development<\/strong>: Work on features independently<\/li>\n<li><strong>Bug fixes<\/strong>: Create hotfix branches without disrupting main code<\/li>\n<li><strong>Experimentation<\/strong>: Try ideas without affecting stable code<\/li>\n<li><strong>Team collaboration<\/strong>: Multiple people work simultaneously<\/li>\n<li><strong>Code review<\/strong>: Enable pull requests for quality control<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"42-creating-and-switching-branches\"><span>4.2 Creating and Switching Branches<\/span>\n <a href=\"#42-creating-and-switching-branches\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># List local branches\r\ngit branch\r\n\r\n# List remote branches\r\ngit branch -r\r\n\r\n# List all branches\r\ngit branch -a\r\n\r\n# Create new branch (doesn&#39;t switch to it)\r\ngit branch feature\/user-profile\r\n\r\n# Create and switch to new branch\r\ngit checkout -b feature\/user-profile\r\n\r\n# Modern syntax (Git 2.23&#43;)\r\ngit switch -c feature\/user-profile\r\n\r\n# Switch to existing branch\r\ngit checkout feature\/user-profile\r\n\r\n# Modern syntax\r\ngit switch feature\/user-profile\r\n\r\n# Switch to previous branch\r\ngit checkout -\r\n\r\n# Delete branch (safe - prevents deleting unmerged branches)\r\ngit branch -d feature\/user-profile\r\n\r\n# Force delete branch\r\ngit branch -D feature\/user-profile\r\n\r\n# Rename branch\r\ngit branch -m old-name new-name\r\n\r\n# Rename current branch\r\ngit branch -m new-name<\/code><\/pre><h3 class=\"heading-element\" id=\"43-branching-strategies\"><span>4.3 Branching Strategies<\/span>\n <a href=\"#43-branching-strategies\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Different teams use different strategies based on their needs.<\/p>\n<h4 class=\"heading-element\" id=\"git-flow-enterprise-projects\"><span>Git Flow (Enterprise Projects)<\/span>\n <a href=\"#git-flow-enterprise-projects\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Designed for projects with scheduled releases:<\/p>\n<pre><code>main (production releases only)\r\n \u2191\r\nrelease\/v1.0 (release preparation)\r\n \u2191\r\ndevelop (integration branch)\r\n \u2191\r\nfeature\/* (individual features)\r\nbugfix\/* (bug fixes)\r\nhotfix\/* (emergency production fixes)<\/code><\/pre><pre><code># Start feature branch from develop\r\ngit checkout develop\r\ngit checkout -b feature\/new-feature\r\n\r\n# Do work, commit regularly\r\ngit add .\r\ngit commit -m &#34;Implement new feature&#34;\r\n\r\n# When ready, create pull request to develop\r\n# After review and merge to develop\r\n\r\n# Create release branch when preparing release\r\ngit checkout -b release\/v1.0 develop\r\n\r\n# Fix release issues only\r\ngit commit -m &#34;Fix typo in error message&#34;\r\n\r\n# When ready, merge to main with tag\r\ngit checkout main\r\ngit merge --no-ff release\/v1.0\r\ngit tag -a v1.0 -m &#34;Release version 1.0&#34;\r\n\r\n# Merge back to develop\r\ngit checkout develop\r\ngit merge --no-ff release\/v1.0<\/code><\/pre><h4 class=\"heading-element\" id=\"github-flow-continuous-delivery\"><span>GitHub Flow (Continuous Delivery)<\/span>\n <a href=\"#github-flow-continuous-delivery\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><p>Simpler flow for continuous deployment:<\/p>\n<pre><code>main (always deployable, production)\r\n \u2191\r\nfeature\/* (features\/fixes - short-lived)\r\n \u2193\r\nPull Request \u2192 Code Review \u2192 Merge \u2192 Deploy<\/code><\/pre><pre><code># Create branch from main\r\ngit checkout -b fix\/login-bug\r\n\r\n# Make commits\r\ngit commit -m &#34;Fix login redirect issue&#34;\r\n\r\n# Push to GitHub\r\ngit push -u origin fix\/login-bug\r\n\r\n# Create pull request on GitHub\r\n# After review and approval, merge to main\r\n# GitHub automatically deploys main to production<\/code><\/pre><p><strong>Advantages:<\/strong><\/p>\n<ul>\n<li>Simple and intuitive<\/li>\n<li>Quick feedback cycles<\/li>\n<li>Production deploys frequently (multiple times per day)<\/li>\n<li>Best for teams comfortable with frequent releases<\/li>\n<\/ul>\n<h4 class=\"heading-element\" id=\"trunk-based-development\"><span>Trunk-Based Development<\/span>\n <a href=\"#trunk-based-development\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code>main (always in releasable state)\r\n \u2191\r\nshort-lived branches (&lt; 1 day)\r\n \u2193\r\nCommit \u2192 Merge \u2192 Deploy<\/code><\/pre><p><strong>Best for:<\/strong><\/p>\n<ul>\n<li>Small teams<\/li>\n<li>Frequent small commits<\/li>\n<li>Strong CI\/CD pipeline<\/li>\n<li>High collaboration<\/li>\n<\/ul>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Choose Your Strategy<\/strong>: Git Flow for large teams with planned releases. GitHub Flow for continuous deployment. Trunk-Based for high-collaboration teams with strong automation.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 class=\"heading-element\" id=\"5-merging-branches\"><span>5. Merging Branches<\/span>\n <a href=\"#5-merging-branches\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Combining changes from different branches is core to collaborative development.<\/p>\n<h3 class=\"heading-element\" id=\"51-types-of-merges\"><span>5.1 Types of Merges<\/span>\n <a href=\"#51-types-of-merges\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"regular-merge-three-way-merge\"><span>Regular Merge (Three-Way Merge)<\/span>\n <a href=\"#regular-merge-three-way-merge\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code># Switch to target branch (usually main)\r\ngit checkout main\r\n\r\n# Merge feature branch into main\r\ngit merge feature\/new-feature\r\n\r\n# Git creates a merge commit automatically\r\n# This preserves complete branch history<\/code><\/pre><p><strong>Result:<\/strong> A new merge commit that combines two branches.<\/p>\n<p><strong>Advantages:<\/strong><\/p>\n<ul>\n<li>Complete history preserved<\/li>\n<li>Can easily see which commits came from which branch<\/li>\n<li>Makes large-scale changes trackable<\/li>\n<\/ul>\n<p><strong>Disadvantages:<\/strong><\/p>\n<ul>\n<li>Creates extra merge commits<\/li>\n<li>History can become cluttered<\/li>\n<\/ul>\n<h4 class=\"heading-element\" id=\"squash-merge\"><span>Squash Merge<\/span>\n <a href=\"#squash-merge\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code># Merge all commits from feature branch into single commit\r\ngit merge --squash feature\/new-feature\r\n\r\n# Still need to commit the squashed changes\r\ngit commit -m &#34;Merge feature: add user profile\r\n\r\nImplements complete user profile functionality including:\r\n- Profile page with bio and avatar\r\n- Profile editing with validation\r\n- Profile deletion option&#34;<\/code><\/pre><p><strong>Result:<\/strong> All commits from feature branch become one commit on main.<\/p>\n<p><strong>Advantages:<\/strong><\/p>\n<ul>\n<li>Clean, linear history<\/li>\n<li>Easier to understand what was added<\/li>\n<li>Easy to revert entire feature<\/li>\n<\/ul>\n<p><strong>Disadvantages:<\/strong><\/p>\n<ul>\n<li>Individual commit history lost<\/li>\n<li>Harder to track specific changes within the feature<\/li>\n<\/ul>\n<h4 class=\"heading-element\" id=\"rebase-merge\"><span>Rebase Merge<\/span>\n <a href=\"#rebase-merge\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code># Reapply feature commits on top of main\r\ngit merge --ff-only feature\/new-feature\r\n\r\n# Or explicitly use rebase\r\ngit rebase main feature\/new-feature\r\ngit checkout main\r\ngit merge --ff-only feature\/new-feature<\/code><\/pre><p><strong>Result:<\/strong> Linear history as if changes were made sequentially.<\/p>\n<p><strong>Advantages:<\/strong><\/p>\n<ul>\n<li>Clean, easy-to-read history<\/li>\n<li>Each commit is independent<\/li>\n<li>Can understand full story<\/li>\n<\/ul>\n<p><strong>Disadvantages:<\/strong><\/p>\n<ul>\n<li>Rewrites history (dangerous with shared branches)<\/li>\n<li>Loses information about parallel development<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"52-merge-conflicts\"><span>5.2 Merge Conflicts<\/span>\n <a href=\"#52-merge-conflicts\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>When Git can&rsquo;t automatically merge branches, you get conflicts.<\/p>\n<h4 class=\"heading-element\" id=\"understanding-conflicts\"><span>Understanding Conflicts<\/span>\n <a href=\"#understanding-conflicts\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code># When merge conflict occurs, files contain:\r\n\r\n&lt;&lt;&lt;&lt;&lt;&lt;&lt; HEAD\r\n\/\/ Your changes (from current branch)\r\nfunction login(user) {\r\n validate(user);\r\n return authenticate(user);\r\n}\r\n=======\r\n\/\/ Incoming changes (from branch being merged)\r\nfunction login(user) {\r\n checkCredentials(user);\r\n return authorizeUser(user);\r\n}\r\n&gt;&gt;&gt;&gt;&gt;&gt;&gt; feature\/auth-update<\/code><\/pre><h4 class=\"heading-element\" id=\"resolving-conflicts\"><span>Resolving Conflicts<\/span>\n <a href=\"#resolving-conflicts\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code># 1. View conflicted files\r\ngit status\r\n\r\n# 2. View conflicts in detail\r\ngit diff\r\n\r\n# 3. Edit the file to resolve conflicts\r\n# Remove conflict markers and keep desired code\r\n\r\n# 4. Stage the resolved file\r\ngit add resolved-file.js\r\n\r\n# 5. Complete the merge\r\ngit commit -m &#34;Merge feature\/auth-update with conflict resolution&#34;\r\n\r\n# Or abort the merge if something went wrong\r\ngit merge --abort<\/code><\/pre><div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Merge Conflict Prevention<\/strong>: Keep branches short-lived (&lt; 1 week). Small, focused changes are less likely to conflict. Communicate with teammates about what you&rsquo;re working on.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 class=\"heading-element\" id=\"6-remote-repositories-and-github\"><span>6. Remote Repositories and GitHub<\/span>\n <a href=\"#6-remote-repositories-and-github\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"61-understanding-remotes\"><span>6.1 Understanding Remotes<\/span>\n <a href=\"#61-understanding-remotes\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>A remote is a reference to a repository hosted elsewhere (typically GitHub).<\/p>\n<pre><code># View configured remotes\r\ngit remote -v\r\n\r\n# Add new remote\r\ngit remote add origin https:\/\/github.com\/username\/project.git\r\n\r\n# Remove remote\r\ngit remote remove origin\r\n\r\n# Rename remote\r\ngit remote rename origin upstream\r\n\r\n# View remote details\r\ngit remote show origin\r\n\r\n# Change remote URL (HTTPS to SSH)\r\ngit remote set-url origin git@github.com:username\/project.git<\/code><\/pre><p><strong>Standard Remote Names:<\/strong><\/p>\n<ul>\n<li><code>origin<\/code>: Your main remote (GitHub repo you cloned or created)<\/li>\n<li><code>upstream<\/code>: Original repo (if you forked)<\/li>\n<li><code>fork<\/code>: Your fork of someone else&rsquo;s project<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"62-cloning-repositories\"><span>6.2 Cloning Repositories<\/span>\n <a href=\"#62-cloning-repositories\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Clone entire repository\r\ngit clone https:\/\/github.com\/username\/project.git\r\n\r\n# Clone to specific directory\r\ngit clone https:\/\/github.com\/username\/project.git my-folder\r\n\r\n# Clone specific branch only\r\ngit clone -b develop https:\/\/github.com\/username\/project.git\r\n\r\n# Shallow clone (fewer commits for faster download)\r\ngit clone --depth 5 https:\/\/github.com\/username\/project.git\r\n\r\n# Shallow clone with history\r\ngit clone --depth 5 --deepen 10 https:\/\/github.com\/username\/project.git<\/code><\/pre><h3 class=\"heading-element\" id=\"63-pushing-and-pulling\"><span>6.3 Pushing and Pulling<\/span>\n <a href=\"#63-pushing-and-pulling\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><h4 class=\"heading-element\" id=\"fetching-updates-from-remote\"><span>Fetching Updates from Remote<\/span>\n <a href=\"#fetching-updates-from-remote\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code># Fetch: Download remote changes (no integration)\r\ngit fetch origin\r\n\r\n# Fetch all remotes\r\ngit fetch --all\r\n\r\n# Fetch and show what changed\r\ngit fetch origin &amp;&amp; git log origin\/main ^main\r\n\r\n# Pull: Fetch &#43; merge (downloads and integrates)\r\ngit pull origin main\r\n\r\n# Pull with rebase instead of merge\r\ngit pull --rebase origin main\r\n\r\n# Pull and automatically stash local changes\r\ngit pull --autostash<\/code><\/pre><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Fetch Before You Act<\/strong>: Use <code>git fetch<\/code> to see what changed on remote without modifying your local code. Then decide whether to merge, rebase, or do something else.<\/div>\n <\/div>\n<\/div>\n<h4 class=\"heading-element\" id=\"pushing-changes-to-remote\"><span>Pushing Changes to Remote<\/span>\n <a href=\"#pushing-changes-to-remote\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code># Push current branch to remote\r\ngit push origin main\r\n\r\n# Push to remote and track it\r\ngit push -u origin feature\/new-feature\r\n\r\n# Push all branches\r\ngit push --all\r\n\r\n# Push all tags\r\ngit push --tags\r\n\r\n# Push specific branch\r\ngit push origin feature\/user-auth\r\n\r\n# Force push (overwrite remote - use with caution!)\r\ngit push --force\r\n\r\n# Safe force push (doesn&#39;t overwrite others&#39; work)\r\ngit push --force-with-lease\r\n\r\n# Delete remote branch\r\ngit push origin --delete feature\/old-feature\r\n\r\n# Push with pull request (GitPython or GitHub CLI)\r\ngh pr create --title &#34;Add user profile&#34; --body &#34;Implements #123&#34;<\/code><\/pre><div class=\"details admonition danger open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-bolt\" aria-hidden=\"true\"><\/i>Danger<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Force Push Warning<\/strong>: <code>git push --force<\/code> overwrites remote history. Only use on personal branches. For shared branches, use <code>--force-with-lease<\/code> which is safer.<\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 class=\"heading-element\" id=\"7-collaborating-with-pull-requests\"><span>7. Collaborating with Pull Requests<\/span>\n <a href=\"#7-collaborating-with-pull-requests\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Pull Requests (PRs) enable code review and quality control.<\/p>\n<h3 class=\"heading-element\" id=\"71-creating-a-pull-request\"><span>7.1 Creating a Pull Request<\/span>\n <a href=\"#71-creating-a-pull-request\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Create feature branch\r\ngit checkout -b feature\/user-dashboard\r\n\r\n# Make changes and commits\r\ngit add .\r\ngit commit -m &#34;Add dashboard component&#34;\r\n\r\n# Push to GitHub\r\ngit push -u origin feature\/user-dashboard\r\n\r\n# Create PR via GitHub web interface or CLI\r\ngh pr create --title &#34;Add user dashboard&#34; \\\r\n --body &#34;Implements #234: User dashboard with stats and charts&#34;<\/code><\/pre><p><strong>Via GitHub CLI (recommended):<\/strong><\/p>\n<pre><code># Install GitHub CLI\r\n# brew install gh (macOS)\r\n# or from https:\/\/cli.github.com\r\n\r\n# Login to GitHub\r\ngh auth login\r\n\r\n# Create pull request\r\ngh pr create --title &#34;Add feature&#34; --body &#34;Description&#34;\r\n\r\n# View your PRs\r\ngh pr list\r\n\r\n# Check PR status\r\ngh pr checks &lt;pr-number&gt;\r\n\r\n# Close PR without merging\r\ngh pr close &lt;pr-number&gt;<\/code><\/pre><h3 class=\"heading-element\" id=\"72-pr-best-practices\"><span>7.2 PR Best Practices<\/span>\n <a href=\"#72-pr-best-practices\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Code Review Checklist:<\/strong><\/p>\n<ul>\n<li data-task=\" \"><i class=\"fa-regular fa-square\" aria-hidden=\"true\"><\/i><span title=\"Unchecked\">Code follows project style guide<\/span><\/li>\n<li data-task=\" \"><i class=\"fa-regular fa-square\" aria-hidden=\"true\"><\/i><span title=\"Unchecked\">Changes are focused and not too large (&lt; 400 lines is ideal)<\/span><\/li>\n<li data-task=\" \"><i class=\"fa-regular fa-square\" aria-hidden=\"true\"><\/i><span title=\"Unchecked\">Tests added\/updated for new code<\/span><\/li>\n<li data-task=\" \"><i class=\"fa-regular fa-square\" aria-hidden=\"true\"><\/i><span title=\"Unchecked\">Documentation updated if needed<\/span><\/li>\n<li data-task=\" \"><i class=\"fa-regular fa-square\" aria-hidden=\"true\"><\/i><span title=\"Unchecked\">No hardcoded values or credentials<\/span><\/li>\n<li data-task=\" \"><i class=\"fa-regular fa-square\" aria-hidden=\"true\"><\/i><span title=\"Unchecked\">No unnecessary dependencies added<\/span><\/li>\n<li data-task=\" \"><i class=\"fa-regular fa-square\" aria-hidden=\"true\"><\/i><span title=\"Unchecked\">Performance implications considered<\/span><\/li>\n<li data-task=\" \"><i class=\"fa-regular fa-square\" aria-hidden=\"true\"><\/i><span title=\"Unchecked\">Security implications reviewed<\/span><\/li>\n<\/ul>\n<p><strong>PR Title and Description:<\/strong><\/p>\n<pre><code>## Summary\r\nBrief description of what this PR does.\r\n\r\n## Changes\r\n- Added user authentication system\r\n- Implemented JWT token validation\r\n- Created auth middleware\r\n\r\n## Related Issues\r\nCloses #123\r\nRelates to #456\r\n\r\n## Testing\r\n- Tested login with valid credentials\r\n- Tested login with invalid credentials\r\n- Tested token refresh\r\n\r\n## Screenshots\r\n[Add relevant screenshots if UI changes]\r\n\r\n## Checklist\r\n- [x] Tests added\/updated\r\n- [x] Documentation updated\r\n- [x] No breaking changes\r\n- [ ] Ready for review<\/code><\/pre><h3 class=\"heading-element\" id=\"73-updating-a-pr\"><span>7.3 Updating a PR<\/span>\n <a href=\"#73-updating-a-pr\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Make additional changes\r\ngit add .\r\ngit commit -m &#34;Address review feedback: add error handling&#34;\r\n\r\n# Push updates (forces update to same PR)\r\ngit push origin feature\/new-feature\r\n\r\n# Force push if you rewrote history\r\ngit push --force-with-lease origin feature\/new-feature\r\n\r\n# The PR automatically updates with new commits<\/code><\/pre><h3 class=\"heading-element\" id=\"74-merging-pull-requests\"><span>7.4 Merging Pull Requests<\/span>\n <a href=\"#74-merging-pull-requests\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Option 1: Create merge commit (preserves branch history)\r\n# Recommended for: Feature branches, large changes\r\n\r\n# Option 2: Squash and merge (clean history)\r\n# Recommended for: Small fixes, single feature commits\r\n\r\n# Option 3: Rebase and merge (linear history)\r\n# Recommended for: Teams using rebase workflow\r\n\r\n# You can set default strategy in GitHub repo settings:\r\n# Settings \u2192 General \u2192 Pull Requests \u2192 Default merge method<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"8-advanced-git-techniques\"><span>8. Advanced Git Techniques<\/span>\n <a href=\"#8-advanced-git-techniques\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"81-stashing-work\"><span>8.1 Stashing Work<\/span>\n <a href=\"#81-stashing-work\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Save changes without committing them.<\/p>\n<pre><code># Save current changes (working directory and staging area)\r\ngit stash\r\n\r\n# Save with descriptive message\r\ngit stash save &#34;WIP: user profile feature&#34;\r\n\r\n# List all stashes\r\ngit stash list\r\n\r\n# Show contents of stash\r\ngit stash show stash@{0}\r\n\r\n# Show stash with diff\r\ngit stash show -p stash@{0}\r\n\r\n# Apply stash and keep it\r\ngit stash apply stash@{0}\r\n\r\n# Apply latest stash and remove it\r\ngit stash pop\r\n\r\n# Delete specific stash\r\ngit stash drop stash@{0}\r\n\r\n# Delete all stashes\r\ngit stash clear\r\n\r\n# Create branch from stash\r\ngit stash branch new-branch<\/code><\/pre><p><strong>Stash Use Cases:<\/strong><\/p>\n<ul>\n<li>Switching branches without committing incomplete work<\/li>\n<li>Pulling remote changes without conflicts<\/li>\n<li>Temporarily saving work while investigating bugs<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"82-rebasing-rewriting-history\"><span>8.2 Rebasing (Rewriting History)<\/span>\n <a href=\"#82-rebasing-rewriting-history\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition warning open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-exclamation-triangle\" aria-hidden=\"true\"><\/i>Warning<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Only rebase unpublished commits!<\/strong> Never rebase commits that have been pushed to shared branches. It rewrites history and causes major conflicts for collaborators.<\/div>\n <\/div>\n<\/div>\n<h4 class=\"heading-element\" id=\"interactive-rebase\"><span>Interactive Rebase<\/span>\n <a href=\"#interactive-rebase\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code># Rebase last 3 commits (opens editor)\r\ngit rebase -i HEAD~3\r\n\r\n# Rebase from specific commit\r\ngit rebase -i abc1234\r\n\r\n# In the editor, you can:\r\n# pick - include commit as-is\r\n# reword - change commit message\r\n# squash - combine with previous commit\r\n# fixup - combine and discard message\r\n# drop - remove commit<\/code><\/pre><p><strong>Example Interactive Rebase Session:<\/strong><\/p>\n<pre><code>pick a1b2c3d Add user model\r\nsquash x9y8z7w Fix user model validation\r\npick p8q7r6s Create user repository\r\n\r\n# Squash combines the 2nd commit into the 1st\r\n# Final result: 2 commits instead of 3<\/code><\/pre><h4 class=\"heading-element\" id=\"rebase-onto-another-branch\"><span>Rebase onto Another Branch<\/span>\n <a href=\"#rebase-onto-another-branch\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code># Switch to feature branch\r\ngit checkout feature\/payment\r\n\r\n# Rebase onto main (moves all feature commits on top of main)\r\ngit rebase main\r\n\r\n# Resolve any conflicts that occur\r\n# git add resolved-files\r\n# git rebase --continue\r\n\r\n# If something goes wrong\r\ngit rebase --abort\r\n\r\n# Force push rebased branch (if already pushed)\r\ngit push --force-with-lease origin feature\/payment<\/code><\/pre><h3 class=\"heading-element\" id=\"83-cherry-picking-commits\"><span>8.3 Cherry-Picking Commits<\/span>\n <a href=\"#83-cherry-picking-commits\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Apply specific commits from one branch to another.<\/p>\n<pre><code># Apply specific commit to current branch\r\ngit cherry-pick abc1234\r\n\r\n# Apply multiple commits\r\ngit cherry-pick abc1234 def5678 ghi9012\r\n\r\n# Apply commits from another branch\r\ngit cherry-pick main~5..main\r\n\r\n# Cherry-pick without committing (review first)\r\ngit cherry-pick --no-commit abc1234\r\n\r\n# Abort cherry-pick if conflicts occur\r\ngit cherry-pick --abort<\/code><\/pre><p><strong>Use Cases:<\/strong><\/p>\n<ul>\n<li>Backporting fixes to older versions<\/li>\n<li>Applying hotfixes to multiple branches<\/li>\n<li>Pulling specific changes from develop to main<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"84-bisecting-finding-bugs\"><span>8.4 Bisecting (Finding Bugs)<\/span>\n <a href=\"#84-bisecting-finding-bugs\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Binary search to find which commit introduced a bug.<\/p>\n<pre><code># Start bisect session\r\ngit bisect start\r\n\r\n# Mark current commit as bad (contains bug)\r\ngit bisect bad HEAD\r\n\r\n# Mark a known good commit\r\ngit bisect good v1.0\r\n\r\n# Git checks out midpoint, you test it\r\n# Mark as good or bad based on testing\r\ngit bisect good # if working\r\ngit bisect bad # if broken\r\n\r\n# Git continues narrowing down\r\n# Eventually finds exact commit that broke it\r\n\r\n# End bisect session\r\ngit bisect reset\r\n\r\n# Automate with script\r\ngit bisect run .\/test-suite.sh<\/code><\/pre><h3 class=\"heading-element\" id=\"85-reflog-reference-logs\"><span>8.5 Reflog (Reference Logs)<\/span>\n <a href=\"#85-reflog-reference-logs\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Recover from mistakes by viewing everything you&rsquo;ve done.<\/p>\n<pre><code># View all reference changes\r\ngit reflog\r\n\r\n# Show detailed reflog\r\ngit reflog show\r\n\r\n# Recover accidentally deleted branch\r\ngit reflog\r\n# Find the commit where branch was at\r\ngit checkout -b recovered-branch abc1234\r\n\r\n# Undo a reset\r\ngit reflog\r\ngit reset --hard commit-before-reset\r\n\r\n# View reflog for specific ref\r\ngit reflog show main<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"9-github-features-and-workflows\"><span>9. GitHub Features and Workflows<\/span>\n <a href=\"#9-github-features-and-workflows\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"91-issues\"><span>9.1 Issues<\/span>\n <a href=\"#91-issues\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>GitHub Issues track bugs, features, and tasks.<\/p>\n<pre><code># Create issue (web interface or CLI)\r\ngh issue create --title &#34;Bug: login fails with spaces in password&#34; \\\r\n --body &#34;Steps to reproduce...&#34;\r\n\r\n# List issues\r\ngh issue list\r\n\r\n# View issue details\r\ngh issue view 123\r\n\r\n# Close issue\r\ngh issue close 123 --comment &#34;Fixed in PR #456&#34;\r\n\r\n# Link commits to issues\r\n# Use &#39;Closes #123&#39; in commit message or PR description\r\ngit commit -m &#34;Fix login bug - Closes #123&#34;\r\n\r\n# When PR merges, GitHub auto-closes the issue<\/code><\/pre><h3 class=\"heading-element\" id=\"92-github-actions-cicd\"><span>9.2 GitHub Actions (CI\/CD)<\/span>\n <a href=\"#92-github-actions-cicd\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Automate testing, building, and deployment.<\/p>\n<pre><code># Create .github\/workflows\/ci.yml file\r\n\r\nname: CI\r\n\r\non:\r\n push:\r\n branches: [main, develop]\r\n pull_request:\r\n branches: [main]\r\n\r\njobs:\r\n test:\r\n runs-on: ubuntu-latest\r\n\r\n steps:\r\n - uses: actions\/checkout@v3\r\n\r\n - uses: actions\/setup-node@v3\r\n with:\r\n node-version: &#39;18&#39;\r\n\r\n - run: npm install\r\n\r\n - run: npm run lint\r\n\r\n - run: npm run test\r\n\r\n - run: npm run build<\/code><\/pre><p><strong>Common Workflow Triggers:<\/strong><\/p>\n<ul>\n<li><code>push<\/code> - on commits to specific branches<\/li>\n<li><code>pull_request<\/code> - on PR creation\/update<\/li>\n<li><code>schedule<\/code> - run on schedule (cron jobs)<\/li>\n<li><code>workflow_dispatch<\/code> - manual trigger<\/li>\n<li><code>release<\/code> - on repository releases<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"93-github-pages\"><span>9.3 GitHub Pages<\/span>\n <a href=\"#93-github-pages\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Publish websites directly from your repository.<\/p>\n<pre><code># 1. Push HTML files to main branch (or docs\/ folder)\r\n# 2. Go to repo Settings \u2192 Pages\r\n# 3. Select source (main or gh-pages branch)\r\n# 4. GitHub builds and publishes automatically\r\n\r\n# Your site is now at: https:\/\/username.github.io\/repository\r\n\r\n# For project documentation\r\n# Push to docs\/ folder and enable in settings<\/code><\/pre><h3 class=\"heading-element\" id=\"94-github-gists\"><span>9.4 GitHub Gists<\/span>\n <a href=\"#94-github-gists\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Share code snippets and files.<\/p>\n<pre><code># Create gist via CLI\r\ngh gist create filename.js\r\n\r\n# Create secret gist (unlisted)\r\ngh gist create --public filename.js\r\n\r\n# List your gists\r\ngh gist list\r\n\r\n# View gist contents\r\ngh gist view gist-id\r\n\r\n# Edit gist\r\ngh gist edit gist-id\r\n\r\n# Delete gist\r\ngh gist delete gist-id\r\n\r\n# Clone gist\r\ngit clone https:\/\/gist.github.com\/username\/gist-id.git<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"10-authentication-and-security\"><span>10. Authentication and Security<\/span>\n <a href=\"#10-authentication-and-security\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"101-ssh-keys-setup\"><span>10.1 SSH Keys Setup<\/span>\n <a href=\"#101-ssh-keys-setup\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>SSH is more secure and convenient than passwords.<\/p>\n<h4 class=\"heading-element\" id=\"generate-ssh-key\"><span>Generate SSH Key<\/span>\n <a href=\"#generate-ssh-key\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code># Generate ED25519 key (recommended - modern &amp; secure)\r\nssh-keygen -t ed25519 -C &#34;your_email@example.com&#34;\r\n# Save to: ~\/.ssh\/id_ed25519\r\n\r\n# Or generate RSA key (4096 bits - widely compatible)\r\nssh-keygen -t rsa -b 4096 -C &#34;your_email@example.com&#34;\r\n# Save to: ~\/.ssh\/id_rsa\r\n\r\n# Add key to SSH agent\r\neval &#34;$(ssh-agent -s)&#34;\r\nssh-add ~\/.ssh\/id_ed25519\r\n\r\n# Copy public key to clipboard\r\ncat ~\/.ssh\/id_ed25519.pub | pbcopy # macOS\r\ncat ~\/.ssh\/id_ed25519.pub | xclip -selection clipboard # Linux<\/code><\/pre><h4 class=\"heading-element\" id=\"add-key-to-github\"><span>Add Key to GitHub<\/span>\n <a href=\"#add-key-to-github\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><ol>\n<li>Go to GitHub Settings \u2192 SSH and GPG keys<\/li>\n<li>Click &ldquo;New SSH key&rdquo;<\/li>\n<li>Paste public key content<\/li>\n<li>Give it a descriptive title<\/li>\n<li>Click &ldquo;Add SSH key&rdquo;<\/li>\n<\/ol>\n<h4 class=\"heading-element\" id=\"test-ssh-connection\"><span>Test SSH Connection<\/span>\n <a href=\"#test-ssh-connection\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code># Test connection\r\nssh -T git@github.com\r\n\r\n# Expected output:\r\n# Hi username! You&#39;ve successfully authenticated, but\r\n# GitHub does not provide shell access.\r\n\r\n# If fails, check configuration\r\nssh -vT git@github.com<\/code><\/pre><h4 class=\"heading-element\" id=\"configure-ssh-for-multiple-github-accounts\"><span>Configure SSH for Multiple GitHub Accounts<\/span>\n <a href=\"#configure-ssh-for-multiple-github-accounts\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h4><pre><code># Create\/edit ~\/.ssh\/config\r\n\r\nHost github.com-personal\r\n HostName github.com\r\n User git\r\n IdentityFile ~\/.ssh\/id_ed25519_personal\r\n\r\nHost github.com-work\r\n HostName github.com\r\n User git\r\n IdentityFile ~\/.ssh\/id_ed25519_work\r\n\r\n# Use in git commands:\r\ngit remote add origin git@github.com-personal:username\/repo.git\r\ngit clone git@github.com-work:company\/project.git<\/code><\/pre><h3 class=\"heading-element\" id=\"102-personal-access-tokens\"><span>10.2 Personal Access Tokens<\/span>\n <a href=\"#102-personal-access-tokens\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Alternative to SSH for programmatic access.<\/p>\n<pre><code># Create token in GitHub Settings \u2192 Developer settings \u2192 Personal access tokens\r\n# Select scopes: repo, read:user, user:email\r\n\r\n# Use as password when cloning via HTTPS\r\ngit clone https:\/\/username:ghp_xxx@github.com\/username\/repo.git\r\n\r\n# Store token securely using credential manager\r\ngit config --global credential.helper cache\r\n\r\n# Or use environment variable\r\nexport GITHUB_TOKEN=&#34;ghp_xxx&#34;\r\ngit clone https:\/\/github.com\/username\/repo.git<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"11-git-internals-how-it-actually-works\"><span>11. Git Internals (How It Actually Works)<\/span>\n <a href=\"#11-git-internals-how-it-actually-works\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Understanding internals helps you debug issues and appreciate Git&rsquo;s design.<\/p>\n<h3 class=\"heading-element\" id=\"111-the-object-database\"><span>11.1 The Object Database<\/span>\n <a href=\"#111-the-object-database\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Git stores everything as four types of objects in <code>.git\/objects\/<\/code>:<\/p>\n<pre><code># Blobs: File contents (immutable)\r\ngit hash-object filename.txt\r\n\r\n# Trees: Directory structure\r\ngit cat-file -p main^{tree}\r\n\r\n# Commits: Snapshots with metadata\r\ngit cat-file -p main\r\n\r\n# Tags: Labels for specific commits\r\ngit cat-file -p v1.0<\/code><\/pre><p><strong>Example Object Structure:<\/strong><\/p>\n<pre><code>Commit: abc1234\r\n \u251c\u2500 Author: John Doe &lt;john@example.com&gt;\r\n \u251c\u2500 Date: 2025-11-03\r\n \u251c\u2500 Message: &#34;Add user profile&#34;\r\n \u251c\u2500 Tree: xyz9876 (represents files at this commit)\r\n \u2514\u2500 Parent: def5678 (previous commit)\r\n\r\nTree: xyz9876\r\n \u251c\u2500 user.js (blob: 8f4k2m1)\r\n \u251c\u2500 profile.js (blob: q2w3e4r)\r\n \u2514\u2500 styles\/ (tree: 5t6y7u8)<\/code><\/pre><h3 class=\"heading-element\" id=\"112-references\"><span>11.2 References<\/span>\n <a href=\"#112-references\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Branches and tags are references to commits.<\/p>\n<pre><code># Branches stored in .git\/refs\/heads\/\r\ncat .git\/refs\/heads\/main\r\n# Output: abc1234def5678...\r\n\r\n# Remote references in .git\/refs\/remotes\/\r\ncat .git\/refs\/remotes\/origin\/main\r\n\r\n# HEAD points to current branch\r\ncat .git\/HEAD\r\n# Output: ref: refs\/heads\/main\r\n\r\n# Detached HEAD (pointing to commit, not branch)\r\ngit checkout abc1234 # You&#39;re now in detached state\r\ncat .git\/HEAD\r\n# Output: abc1234 (direct commit reference)<\/code><\/pre><h3 class=\"heading-element\" id=\"113-the-staging-area-index\"><span>11.3 The Staging Area (Index)<\/span>\n <a href=\"#113-the-staging-area-index\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The staging area is a file (<code>.git\/index<\/code>) that tracks what will be committed.<\/p>\n<pre><code># View index contents\r\ngit ls-files\r\n\r\n# View index with status\r\ngit ls-files --stage\r\n\r\n# Clear index (dangerous!)\r\ngit rm --cached -r .\r\n\r\n# Update index after external file changes\r\ngit update-index --add --cacheinfo<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"12-troubleshooting-common-problems\"><span>12. Troubleshooting Common Problems<\/span>\n <a href=\"#12-troubleshooting-common-problems\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"121-undoing-changes\"><span>12.1 Undoing Changes<\/span>\n <a href=\"#121-undoing-changes\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Undo uncommitted changes in working directory\r\ngit checkout filename.js\r\n# or (modern)\r\ngit restore filename.js\r\n\r\n# Undo staged changes (keep working directory)\r\ngit restore --staged filename.js\r\n# or (older)\r\ngit reset HEAD filename.js\r\n\r\n# Undo last commit (keep changes)\r\ngit reset --soft HEAD~1\r\n\r\n# Undo last commit (discard changes)\r\ngit reset --hard HEAD~1\r\n\r\n# Revert commit (create new commit that undoes it)\r\ngit revert abc1234\r\n\r\n# Revert merge commit (specify which parent to keep)\r\ngit revert -m 1 merge-commit-sha<\/code><\/pre><h3 class=\"heading-element\" id=\"122-fixing-mistakes\"><span>12.2 Fixing Mistakes<\/span>\n <a href=\"#122-fixing-mistakes\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Accidentally committed to wrong branch\r\ngit reset --soft HEAD~1\r\ngit stash\r\ngit checkout correct-branch\r\ngit stash pop\r\ngit commit -m &#34;...&#34;\r\n\r\n# Committed to main instead of feature branch\r\ngit reset --soft HEAD~1 # Keep changes\r\ngit checkout -b feature\/new-feature\r\ngit commit -m &#34;...&#34;\r\n\r\n# Need to update branch name before push\r\ngit branch -m old-name new-name\r\ngit push -u origin new-name\r\ngit push origin --delete old-name\r\n\r\n# Typo in commit message\r\ngit commit --amend -m &#34;Corrected message&#34;\r\n# Then force push if already pushed\r\ngit push --force-with-lease origin branch-name<\/code><\/pre><h3 class=\"heading-element\" id=\"123-resolving-conflicts\"><span>12.3 Resolving Conflicts<\/span>\n <a href=\"#123-resolving-conflicts\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># View all conflicted files\r\ngit diff --name-only --diff-filter=U\r\n\r\n# View detailed conflict\r\ngit diff filename.js\r\n\r\n# Use your version entirely\r\ngit checkout --ours filename.js\r\ngit add filename.js\r\n\r\n# Use their version entirely\r\ngit checkout --theirs filename.js\r\ngit add filename.js\r\n\r\n# Use external merge tool\r\ngit mergetool\r\n\r\n# After resolving all conflicts\r\ngit add .\r\ngit commit -m &#34;Resolve merge conflicts&#34;\r\n\r\n# Or abort the merge\r\ngit merge --abort<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"13-best-practices-and-workflow-tips\"><span>13. Best Practices and Workflow Tips<\/span>\n <a href=\"#13-best-practices-and-workflow-tips\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"131-commit-best-practices\"><span>13.1 Commit Best Practices<\/span>\n <a href=\"#131-commit-best-practices\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Atomic Commits<\/strong>: Each commit should represent one logical change. Related changes should be committed together; unrelated changes should be in separate commits.<\/div>\n <\/div>\n<\/div>\n<pre><code># \u274c BAD: Too many unrelated changes\r\ngit add .\r\ngit commit -m &#34;Update stuff&#34;\r\n\r\n# \u2705 GOOD: Related changes, logical commit\r\ngit add src\/auth\/login.js src\/auth\/login.test.js\r\ngit commit -m &#34;feat(auth): add password reset functionality\r\n\r\n- Add reset token generation\r\n- Add email sending for reset link\r\n- Add password validation rules\r\n\r\nCloses #234&#34;\r\n\r\n# Use staging area strategically\r\ngit add -p # Stage changes hunk by hunk\r\ngit diff --staged # Review what you&#39;re committing\r\ngit commit -m &#34;...&#34; # Commit reviewed changes<\/code><\/pre><h3 class=\"heading-element\" id=\"132-branch-naming-conventions\"><span>13.2 Branch Naming Conventions<\/span>\n <a href=\"#132-branch-naming-conventions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Feature branches\r\nfeature\/user-authentication\r\nfeature\/payment-integration\r\n\r\n# Bug fix branches\r\nbugfix\/login-redirect-issue\r\nfix\/null-pointer-exception\r\n\r\n# Release branches\r\nrelease\/v1.0\r\nrelease\/v2.1.0\r\n\r\n# Hotfix branches\r\nhotfix\/critical-security-patch\r\nhotfix\/production-crash\r\n\r\n# Maintenance branches\r\nmaintenance\/update-dependencies\r\nchore\/code-cleanup\r\n\r\n# Personal experimentation\r\nsandbox\/test-new-framework\r\nspike\/investigate-performance<\/code><\/pre><h3 class=\"heading-element\" id=\"133-keeping-repository-clean\"><span>13.3 Keeping Repository Clean<\/span>\n <a href=\"#133-keeping-repository-clean\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Delete merged branches locally\r\ngit branch -d feature\/completed-feature\r\n\r\n# Delete all merged branches\r\ngit branch -d $(git branch --merged | grep -v &#34;*&#34;)\r\n\r\n# Find branches merged into main\r\ngit branch --merged main\r\n\r\n# Find branches NOT merged into main\r\ngit branch --no-merged main\r\n\r\n# Clean up tracking branches\r\ngit remote prune origin\r\n\r\n# Clean up object database\r\ngit gc\r\n\r\n# View and cleanup\r\ngit reflog expire --expire=now --all\r\ngit gc --prune=now<\/code><\/pre><h3 class=\"heading-element\" id=\"134-writing-useful-commit-messages\"><span>13.4 Writing Useful Commit Messages<\/span>\n <a href=\"#134-writing-useful-commit-messages\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Commit messages are read by humans much more often than they&rsquo;re written.<\/p>\n<pre><code># Bad commit message\r\nFixed stuff\r\n\r\n# Good commit message\r\nFix user profile display bug\r\n\r\nThe profile view was failing when user bio contained\r\nspecial characters. This was due to improper HTML\r\nescaping in the template.\r\n\r\nChanges:\r\n- Add HTML entity encoding to bio field\r\n- Add test case for special characters\r\n- Update documentation\r\n\r\nCloses #456<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"14-github-workflows-and-collaboration\"><span>14. GitHub Workflows and Collaboration<\/span>\n <a href=\"#14-github-workflows-and-collaboration\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"141-code-review-workflow\"><span>14.1 Code Review Workflow<\/span>\n <a href=\"#141-code-review-workflow\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># As a reviewer:\r\n1. Clone the branch locally to test\r\n git fetch origin pull\/123\/head:pr-123\r\n git checkout pr-123\r\n\r\n2. Test the changes thoroughly\r\n npm install &amp;&amp; npm run test\r\n\r\n3. Review code for:\r\n - Correctness\r\n - Performance\r\n - Security\r\n - Style consistency\r\n - Test coverage\r\n\r\n4. Leave constructive comments\r\n gh pr review 123 --approve\r\n # or\r\n gh pr review 123 --request-changes\r\n\r\n5. Approve when satisfied\r\n # Comment: &#34;Looks good! Approved&#34;\r\n\r\n# As a developer:\r\n1. Implement feedback\r\n git add .\r\n git commit -m &#34;Address review feedback: add error handling&#34;\r\n\r\n2. Push updates\r\n git push origin feature\/branch\r\n\r\n3. Request re-review\r\n gh pr review --request-review @reviewer-name<\/code><\/pre><h3 class=\"heading-element\" id=\"142-release-workflow-with-tags\"><span>14.2 Release Workflow with Tags<\/span>\n <a href=\"#142-release-workflow-with-tags\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Create annotated tag (recommended)\r\ngit tag -a v1.0.0 -m &#34;Release version 1.0.0&#34;\r\n\r\n# Create lightweight tag (simple reference)\r\ngit tag v1.0.0\r\n\r\n# List tags\r\ngit tag -l\r\n\r\n# View tag details\r\ngit show v1.0.0\r\n\r\n# Push tags to GitHub\r\ngit push origin --tags\r\n\r\n# Delete tag locally\r\ngit tag -d v1.0.0\r\n\r\n# Delete tag remotely\r\ngit push origin --delete v1.0.0\r\n\r\n# Create release on GitHub with notes\r\ngh release create v1.0.0 --title &#34;Version 1.0.0&#34; --notes &#34;Release notes...&#34;<\/code><\/pre><p><strong>Semantic Versioning (Recommended):<\/strong><\/p>\n<pre><code>v[MAJOR].[MINOR].[PATCH]-[PRERELEASE]&#43;[BUILD]\r\n\r\nv1.2.3 # Version 1, minor release 2, patch 3\r\nv2.0.0-beta # Pre-release version\r\nv1.0.0-alpha.1 # Pre-release with identifier<\/code><\/pre><h3 class=\"heading-element\" id=\"143-git-hooks\"><span>14.3 Git Hooks<\/span>\n <a href=\"#143-git-hooks\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Automate tasks that run at specific points in Git workflow.<\/p>\n<pre><code># Hooks located in .git\/hooks\/\r\n\r\n# Create pre-commit hook (runs before commit)\r\ncat &gt; .git\/hooks\/pre-commit &lt;&lt;&#39;EOF&#39;\r\n#!\/bin\/bash\r\nnpm run lint\r\nnpm run test\r\nEOF\r\n\r\nchmod &#43;x .git\/hooks\/pre-commit\r\n\r\n# Shared hooks (commit to repo)\r\nmkdir -p .githooks\r\n\r\n# Update git config to use .githooks directory\r\ngit config core.hooksPath .githooks\r\n\r\n# Common hooks:\r\n# pre-commit - before commit\r\n# prepare-commit-msg - modify commit message\r\n# commit-msg - validate commit message\r\n# post-commit - after commit\r\n# pre-push - before push<\/code><\/pre><p><strong>Pre-commit Framework (Recommended):<\/strong><\/p>\n<pre><code># Install pre-commit\r\npip install pre-commit\r\n\r\n# Create .pre-commit-config.yaml\r\nrepos:\r\n - repo: https:\/\/github.com\/psf\/black\r\n rev: 23.1.0\r\n hooks:\r\n - id: black\r\n\r\n - repo: https:\/\/github.com\/pre-commit\/pre-commit-hooks\r\n rev: v4.4.0\r\n hooks:\r\n - id: trailing-whitespace\r\n - id: end-of-file-fixer\r\n - id: check-yaml\r\n - id: check-added-large-files\r\n\r\n# Setup pre-commit\r\npre-commit install\r\n\r\n# Run manually on all files\r\npre-commit run --all-files<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"15-git-performance-and-optimization\"><span>15. Git Performance and Optimization<\/span>\n <a href=\"#15-git-performance-and-optimization\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"151-large-repositories\"><span>15.1 Large Repositories<\/span>\n <a href=\"#151-large-repositories\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Shallow clone (faster for CI\/CD)\r\ngit clone --depth 1 repo-url\r\n\r\n# Unshallow later if needed\r\ngit fetch --unshallow\r\n\r\n# Sparse checkout (clone only specific directories)\r\ngit clone --filter=blob:none --sparse repo-url\r\ncd repo\r\ngit sparse-checkout set path\/to\/dir\r\n\r\n# Partial clone (saves bandwidth)\r\ngit clone --filter=blob:none repo-url\r\n\r\n# Git LFS for large files\r\ngit lfs install\r\ngit lfs track &#34;*.bin&#34;\r\ngit add .gitattributes<\/code><\/pre><h3 class=\"heading-element\" id=\"152-performance-tips\"><span>15.2 Performance Tips<\/span>\n <a href=\"#152-performance-tips\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Enable auto garbage collection (default)\r\ngit config --global gc.auto 4096\r\n\r\n# Manual garbage collection\r\ngit gc --aggressive\r\n\r\n# Reduce pack size\r\ngit gc --prune=now\r\n\r\n# Enable object file delta compression\r\ngit config --global core.compression 9\r\n\r\n# Use credential helper for faster auth\r\ngit config --global credential.helper cache\r\ngit config --global credential.helper &#39;cache --timeout=3600&#39;<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"16-interview-questions-and-deep-dives\"><span>16. Interview Questions and Deep Dives<\/span>\n <a href=\"#16-interview-questions-and-deep-dives\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"161-common-interview-questions\"><span>16.1 Common Interview Questions<\/span>\n <a href=\"#161-common-interview-questions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Q1: Explain Git vs GitHub<\/strong><\/p>\n<blockquote>\n<p>Git is version control software. GitHub is a platform for hosting Git repositories with collaboration features like PRs, issues, and Actions.<\/p>\n<\/blockquote>\n<p><strong>Q2: What is the difference between merge and rebase?<\/strong><\/p>\n<blockquote>\n<p>Merge creates a merge commit combining two branches (preserves history). Rebase replays commits on top of another branch (linear history). Use merge for shared branches, rebase for personal branches.<\/p>\n<\/blockquote>\n<p><strong>Q3: How do you handle merge conflicts?<\/strong><\/p>\n<blockquote>\n<p>Git marks conflict zones in files. You edit to resolve, then stage and commit. Use <code>git diff<\/code> to see conflicts, merge tools like Meld for visual resolution, or <code>git merge --abort<\/code> to cancel.<\/p>\n<\/blockquote>\n<p><strong>Q4: What&rsquo;s the difference between reset and revert?<\/strong><\/p>\n<blockquote>\n<p><code>reset<\/code> moves HEAD to previous commit (rewrites history). <code>revert<\/code> creates new commit that undoes changes (safe for shared branches).<\/p>\n<\/blockquote>\n<p><strong>Q5: Explain Git objects and how commits work<\/strong><\/p>\n<blockquote>\n<p>Git stores blobs (file content), trees (directory structure), commits (snapshots with metadata), and tags (references). Everything is content-addressed by SHA-1 hash.<\/p>\n<\/blockquote>\n<h3 class=\"heading-element\" id=\"162-advanced-scenarios\"><span>16.2 Advanced Scenarios<\/span>\n <a href=\"#162-advanced-scenarios\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Scenario: Large refactor needs to be split across commits\r\ngit reset HEAD~5 # Unstage last 5 commits\r\ngit add .\r\ngit status -s | head -20 # See all changes\r\n\r\n# Stage first logical unit\r\ngit add src\/auth\/*.js\r\ngit commit -m &#34;refactor(auth): simplify token handling&#34;\r\n\r\n# Stage next unit\r\ngit add src\/api\/*.js\r\ngit commit -m &#34;refactor(api): restructure endpoints&#34;\r\n\r\n# Push when ready\r\ngit push origin feature\/refactor\r\n\r\n# Later: squash all into one for cleaner main history\r\ngit merge --squash feature\/refactor\r\ngit commit -m &#34;Merge refactor: reorganize codebase\r\n\r\n- Simplified authentication system\r\n- Restructured API endpoints\r\n- Updated error handling&#34;<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"17-resources-and-further-learning\"><span>17. Resources and Further Learning<\/span>\n <a href=\"#17-resources-and-further-learning\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"171-official-documentation\"><span>17.1 Official Documentation<\/span>\n <a href=\"#171-official-documentation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ul>\n<li><strong>Git Official<\/strong>: <a href=\"https:\/\/git-scm.com\/doc\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">https:\/\/git-scm.com\/doc<\/a><\/li>\n<li><strong>Git Book<\/strong>: <a href=\"https:\/\/git-scm.com\/book\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">https:\/\/git-scm.com\/book<\/a> (free, comprehensive)<\/li>\n<li><strong>GitHub Docs<\/strong>: <a href=\"https:\/\/docs.github.com\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">https:\/\/docs.github.com<\/a><\/li>\n<li><strong>GitHub Learning Lab<\/strong>: <a href=\"https:\/\/github.com\/skills\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">https:\/\/github.com\/skills<\/a><\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"172-practice-platforms\"><span>17.2 Practice Platforms<\/span>\n <a href=\"#172-practice-platforms\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ul>\n<li><strong>Learn Git Branching<\/strong>: <a href=\"https:\/\/learngitbranching.js.org\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">https:\/\/learngitbranching.js.org<\/a> (interactive)<\/li>\n<li><strong>Oh My Git!<\/strong>: <a href=\"https:\/\/ohmygit.org\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">https:\/\/ohmygit.org<\/a> (visual learning)<\/li>\n<li><strong>Git Exercises<\/strong>: <a href=\"https:\/\/gitexercises.fracz.com\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">https:\/\/gitexercises.fracz.com<\/a><\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"173-tools-to-enhance-git\"><span>17.3 Tools to Enhance Git<\/span>\n <a href=\"#173-tools-to-enhance-git\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ul>\n<li><strong>GitHub CLI (gh)<\/strong>: Command-line tool for GitHub operations<\/li>\n<li><strong>Git Aliases<\/strong>: Create shortcuts for frequent commands<\/li>\n<li><strong>Git Hooks<\/strong>: Automate common tasks<\/li>\n<li><strong>Pre-commit<\/strong>: Framework for managing git hooks<\/li>\n<li><strong>Git LFS<\/strong>: Handle large files efficiently<\/li>\n<li><strong>Conventional Commits<\/strong>: Standard commit message format<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"18-quick-reference-common-commands\"><span>18. Quick Reference: Common Commands<\/span>\n <a href=\"#18-quick-reference-common-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Git Command Cheat Sheet<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><pre><code># Setup\r\ngit config --global user.name &#34;Name&#34;\r\ngit config --global user.email &#34;email@example.com&#34;\r\n\r\n# Create and Clone\r\ngit init\r\ngit clone url\r\n\r\n# Daily Workflow\r\ngit status\r\ngit add filename \/ git add .\r\ngit commit -m &#34;message&#34;\r\ngit push origin branch-name\r\ngit pull origin branch-name\r\n\r\n# Branches\r\ngit branch\r\ngit checkout -b new-branch\r\ngit switch -c new-branch\r\ngit merge branch-name\r\ngit branch -d branch-name\r\n\r\n# History\r\ngit log\r\ngit log --oneline --graph --all\r\ngit diff\r\ngit show commit-sha\r\n\r\n# Undo\r\ngit restore filename\r\ngit restore --staged filename\r\ngit reset --soft HEAD~1\r\ngit revert commit-sha\r\n\r\n# Remote\r\ngit remote -v\r\ngit fetch origin\r\ngit pull origin main\r\ngit push origin branch-name\r\ngit push --all\r\n\r\n# Advanced\r\ngit stash\r\ngit rebase -i HEAD~3\r\ngit cherry-pick commit-sha\r\ngit bisect start<\/code><\/pre><\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 class=\"heading-element\" id=\"19-conclusion-and-next-steps\"><span>19. Conclusion and Next Steps<\/span>\n <a href=\"#19-conclusion-and-next-steps\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>You now have comprehensive knowledge of:<\/p>\n<p>\u2705 Git fundamentals and object model\n\u2705 Daily workflows and common commands\n\u2705 Branching strategies and collaboration\n\u2705 Pull requests and code reviews\n\u2705 GitHub features and automation\n\u2705 Authentication and security\n\u2705 Advanced techniques and troubleshooting\n\u2705 Best practices for professional development<\/p>\n<p><strong>Next Steps:<\/strong><\/p>\n<ol>\n<li><strong>Practice Daily<\/strong>: Use Git for all your projects, big and small<\/li>\n<li><strong>Read the Git Book<\/strong>: Free and incredibly thorough<\/li>\n<li><strong>Contribute to Open Source<\/strong>: Apply your skills to real projects<\/li>\n<li><strong>Setup Git Hooks<\/strong>: Automate quality checks<\/li>\n<li><strong>Master GitHub Actions<\/strong>: Automate your workflow<\/li>\n<li><strong>Stay Updated<\/strong>: Git evolves with new features and best practices<\/li>\n<\/ol>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Pro Tip<\/strong>: The best way to learn Git is to use it daily and make mistakes in a safe environment. Every error is a learning opportunity!<\/div>\n <\/div>\n<\/div>\n<hr>\n<p><strong>Happy coding and collaborating! \ud83d\ude80<\/strong><\/p>\n<p>If you have questions, need clarification, or want to go deeper into any topic, check the official Git and GitHub documentation, or explore the interactive learning resources mentioned above.<\/p>\n<p>If you have questions, need clarification, or want to go deeper into any topic, check the official Git and GitHub documentation, or explore the interactive learning resources mentioned above.<\/p>\n<hr>\n<h2 class=\"heading-element\" id=\"20-visual-diagrams-and-architecture\"><span>20. Visual Diagrams and Architecture<\/span>\n <a href=\"#20-visual-diagrams-and-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Understanding Git visually helps tremendously. This section provides comprehensive diagrams that illustrate Git&rsquo;s core concepts and workflows.<\/p>\n<h3 class=\"heading-element\" id=\"201-git-three-state-architecture\"><span>20.1 Git Three-State Architecture<\/span>\n <a href=\"#201-git-three-state-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The three-state model is fundamental to understanding how Git tracks your changes:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/git-three-state-model.webp\" title=\"Git Three-State Model - Understanding the Three Main States in Git: Working Directory (untracked\/modified files), Staging Area (files marked for commit), and Repository (permanent snapshots stored in .git)\" data-thumbnail=\"\/files\/git-three-state-model.webp\" data-sub-html=\"<h2>Git Three-State Model: Working Directory, Staging Area, and Repository<\/h2><p>Git Three-State Model - Understanding the Three Main States in Git: Working Directory (untracked\/modified files), Staging Area (files marked for commit), and Repository (permanent snapshots stored in .git)<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/git-three-state-model.webp' alt=\"Git Three-State Model: Working Directory, Staging Area, and Repository\"><\/a><figcaption class=\"image-caption\">Git Three-State Model - Understanding the Three Main States in Git: Working Directory (untracked\/modified files), Staging Area (files marked for commit), and Repository (permanent snapshots stored in .git)<\/figcaption>\n <\/figure><\/p>\n<p><strong>Understanding the Flow:<\/strong><\/p>\n<ul>\n<li><strong>Working Directory<\/strong>: Where you modify files<\/li>\n<li><strong>Staging Area<\/strong>: Where you prepare changes for commit<\/li>\n<li><strong>Repository<\/strong>: Where Git permanently stores committed snapshots<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"202-git-branching-visualization\"><span>20.2 Git Branching Visualization<\/span>\n <a href=\"#202-git-branching-visualization\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Branches allow parallel development without interfering with the main codebase:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/git-branching-visualization.webp\" title=\"Git Branching Strategy - Main branch with feature branches being developed in parallel and merged back into main\" data-thumbnail=\"\/files\/git-branching-visualization.webp\" data-sub-html=\"<h2>Git Branching Visualization with Feature Integration<\/h2><p>Git Branching Strategy - Main branch with feature branches being developed in parallel and merged back into main<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/git-branching-visualization.webp' alt=\"Git Branching Visualization with Feature Integration\"><\/a><figcaption class=\"image-caption\">Git Branching Strategy - Main branch with feature branches being developed in parallel and merged back into main<\/figcaption>\n <\/figure><\/p>\n<p><strong>Key Concepts:<\/strong><\/p>\n<ul>\n<li>Multiple feature branches can be developed simultaneously<\/li>\n<li>Each branch maintains its own commit history<\/li>\n<li>Branches merge back into main when features are complete<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"203-git-merge-vs-rebase\"><span>20.3 Git Merge vs Rebase<\/span>\n <a href=\"#203-git-merge-vs-rebase\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Two fundamental ways to integrate changes from one branch into another:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/git-merge-vs-rebase.webp\" title=\"Git Merge vs Rebase - Merge creates a merge commit preserving branch history, while Rebase replays commits for linear history\" data-thumbnail=\"\/files\/git-merge-vs-rebase.webp\" data-sub-html=\"<h2>Merge vs Rebase Comparison: Creating Merge Commits vs Linear History<\/h2><p>Git Merge vs Rebase - Merge creates a merge commit preserving branch history, while Rebase replays commits for linear history<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/git-merge-vs-rebase.webp' alt=\"Merge vs Rebase Comparison: Creating Merge Commits vs Linear History\"><\/a><figcaption class=\"image-caption\">Git Merge vs Rebase - Merge creates a merge commit preserving branch history, while Rebase replays commits for linear history<\/figcaption>\n <\/figure><\/p>\n<p><strong>When to Use Each:<\/strong><\/p>\n<ul>\n<li><strong>Merge<\/strong>: Preserves complete history, creates merge commits, safer for shared branches<\/li>\n<li><strong>Rebase<\/strong>: Creates linear history, cleaner log, best for feature branches before merging<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"204-git-flow-branching-strategy-diagram\"><span>20.4 Git Flow Branching Strategy Diagram<\/span>\n <a href=\"#204-git-flow-branching-strategy-diagram\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Git Flow is a robust branching model for managing complex release cycles:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/git-flow-strategy.webp\" title=\"Git Flow Strategy - Comprehensive branching model with develop, release, and hotfix branches for managing complex release cycles\" data-thumbnail=\"\/files\/git-flow-strategy.webp\" data-sub-html=\"<h2>Git Flow Branching Strategy Diagram<\/h2><p>Git Flow Strategy - Comprehensive branching model with develop, release, and hotfix branches for managing complex release cycles<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/git-flow-strategy.webp' alt=\"Git Flow Branching Strategy Diagram\"><\/a><figcaption class=\"image-caption\">Git Flow Strategy - Comprehensive branching model with develop, release, and hotfix branches for managing complex release cycles<\/figcaption>\n <\/figure><\/p>\n<p><strong>Git Flow Structure:<\/strong><\/p>\n<ul>\n<li><strong>main<\/strong>: Production-ready code<\/li>\n<li><strong>develop<\/strong>: Integration branch for features<\/li>\n<li><strong>feature\/<\/strong>*: New features under development<\/li>\n<li><strong>release\/<\/strong>*: Preparing for production release<\/li>\n<li><strong>hotfix\/<\/strong>*: Emergency fixes for production<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"205-github-flow-diagram\"><span>20.5 GitHub Flow Diagram<\/span>\n <a href=\"#205-github-flow-diagram\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>GitHub Flow is a simpler alternative optimized for continuous deployment:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/git-github-flow.webp\" title=\"GitHub Flow - Simpler alternative to Git Flow with main branch always deployable, short-lived feature branches, and continuous deployment\" data-thumbnail=\"\/files\/git-github-flow.webp\" data-sub-html=\"<h2>GitHub Flow: Simplified Branching for Continuous Deployment<\/h2><p>GitHub Flow - Simpler alternative to Git Flow with main branch always deployable, short-lived feature branches, and continuous deployment<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/git-github-flow.webp' alt=\"GitHub Flow: Simplified Branching for Continuous Deployment\"><\/a><figcaption class=\"image-caption\">GitHub Flow - Simpler alternative to Git Flow with main branch always deployable, short-lived feature branches, and continuous deployment<\/figcaption>\n <\/figure><\/p>\n<p><strong>GitHub Flow Principles:<\/strong><\/p>\n<ol>\n<li>Main branch is always deployable<\/li>\n<li>Create descriptive feature branches<\/li>\n<li>Open pull requests for discussion<\/li>\n<li>Deploy and test before merging<\/li>\n<li>Merge to main and deploy<\/li>\n<\/ol>\n<h3 class=\"heading-element\" id=\"206-git-distributed-workflow\"><span>20.6 Git Distributed Workflow<\/span>\n <a href=\"#206-git-distributed-workflow\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Git&rsquo;s distributed nature enables flexible team collaboration:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/git-distributed-workflow.webp\" title=\"Distributed Git Workflow - Shows how multiple developers work with local repositories syncing with GitHub remote\" data-thumbnail=\"\/files\/git-distributed-workflow.webp\" data-sub-html=\"<h2>Distributed Git Workflow with Multiple Developers<\/h2><p>Distributed Git Workflow - Shows how multiple developers work with local repositories syncing with GitHub remote<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/git-distributed-workflow.webp' alt=\"Distributed Git Workflow with Multiple Developers\"><\/a><figcaption class=\"image-caption\">Distributed Git Workflow - Shows how multiple developers work with local repositories syncing with GitHub remote<\/figcaption>\n <\/figure><\/p>\n<p><strong>Distributed Benefits:<\/strong><\/p>\n<ul>\n<li>Every developer has a complete repository copy<\/li>\n<li>Work offline and commit locally<\/li>\n<li>Push\/pull to synchronize with team<\/li>\n<li>No single point of failure<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"207-git-object-model-diagram\"><span>20.7 Git Object Model Diagram<\/span>\n <a href=\"#207-git-object-model-diagram\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Understanding Git&rsquo;s internal storage structure:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/git-object-model.webp\" title=\"Git Object Model - Understanding how Git stores commits, trees, and blobs in a directed acyclic graph structure\" data-thumbnail=\"\/files\/git-object-model.webp\" data-sub-html=\"<h2>Git Object Model: Commits, Trees, and Blobs<\/h2><p>Git Object Model - Understanding how Git stores commits, trees, and blobs in a directed acyclic graph structure<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/git-object-model.webp' alt=\"Git Object Model: Commits, Trees, and Blobs\"><\/a><figcaption class=\"image-caption\">Git Object Model - Understanding how Git stores commits, trees, and blobs in a directed acyclic graph structure<\/figcaption>\n <\/figure><\/p>\n<p><strong>Object Types:<\/strong><\/p>\n<ul>\n<li><strong>Commit<\/strong>: Points to tree and parent commits<\/li>\n<li><strong>Tree<\/strong>: Represents directory structure<\/li>\n<li><strong>Blob<\/strong>: Stores file contents<\/li>\n<li>All objects identified by SHA-1 hash<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"208-merge-conflict-resolution-flow\"><span>20.8 Merge Conflict Resolution Flow<\/span>\n <a href=\"#208-merge-conflict-resolution-flow\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Step-by-step process for resolving merge conflicts:<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/git-merge-conflict-resolution.webp\" title=\"Merge Conflict Resolution - Step-by-step process from attempting merge through identifying conflicts to final resolution and commit\" data-thumbnail=\"\/files\/git-merge-conflict-resolution.webp\" data-sub-html=\"<h2>Merge Conflict Resolution Flow Diagram<\/h2><p>Merge Conflict Resolution - Step-by-step process from attempting merge through identifying conflicts to final resolution and commit<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/git-merge-conflict-resolution.webp' alt=\"Merge Conflict Resolution Flow Diagram\"><\/a><figcaption class=\"image-caption\">Merge Conflict Resolution - Step-by-step process from attempting merge through identifying conflicts to final resolution and commit<\/figcaption>\n <\/figure><\/p>\n<p><strong>Resolution Steps:<\/strong><\/p>\n<ol>\n<li>Attempt merge and identify conflicts<\/li>\n<li>Open conflicted files and review markers<\/li>\n<li>Choose resolution strategy (yours, theirs, or both)<\/li>\n<li>Stage resolved files<\/li>\n<li>Complete merge with commit<\/li>\n<\/ol>\n<hr>\n<h2 class=\"heading-element\" id=\"21-complete-git-command-reference-500-commands\"><span>21. Complete Git Command Reference (500+ Commands)<\/span>\n <a href=\"#21-complete-git-command-reference-500-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"211-advanced-git-configuration\"><span>21.1 Advanced Git Configuration<\/span>\n <a href=\"#211-advanced-git-configuration\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Show all configuration and their sources\r\ngit config --list --show-origin\r\n\r\n# Show specific config value\r\ngit config user.name\r\n\r\n# Edit global config in editor\r\ngit config --global --edit\r\n\r\n# Set different email for specific repo\r\ngit config user.email &#34;work@company.com&#34;\r\n\r\n# Enable rerere (reuse recorded resolution)\r\ngit config --global rerere.enabled true\r\n\r\n# Set merge conflict style\r\ngit config --global merge.conflictstyle diff3\r\n\r\n# Enable auto-correct for mistyped commands\r\ngit config --global help.autocorrect 20\r\n\r\n# Set default push behavior\r\ngit config --global push.default simple\r\ngit config --global push.default current\r\n\r\n# Enable push.autoSetupRemote (Git 2.37&#43;)\r\ngit config --global push.autoSetupRemote true\r\n\r\n# Set default pull strategy\r\ngit config --global pull.rebase true\r\ngit config --global pull.ff only\r\n\r\n# Configure commit template\r\ngit config --global commit.template ~\/.gitmessage\r\n\r\n# Set up signing commits with GPG\r\ngit config --global user.signingkey YOUR_GPG_KEY\r\ngit config --global commit.gpgsign true\r\n\r\n# Ignore file mode changes\r\ngit config core.fileMode false\r\n\r\n# Case-sensitive file names\r\ngit config core.ignorecase false\r\n\r\n# Enable filesystem monitoring (Git 2.35&#43;)\r\ngit config --global core.fsmonitor true\r\n\r\n# Set credential cache timeout\r\ngit config --global credential.helper &#39;cache --timeout=3600&#39;<\/code><\/pre><h3 class=\"heading-element\" id=\"212-git-add-power-commands\"><span>21.2 Git Add Power Commands<\/span>\n <a href=\"#212-git-add-power-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Add with intent to add (track but don&#39;t stage content)\r\ngit add -N file.txt\r\n\r\n# Force add ignored files\r\ngit add -f ignored-file.txt\r\n\r\n# Dry run (show what would be added)\r\ngit add -n .\r\n\r\n# Verbose output\r\ngit add -v .\r\n\r\n# Add only modified files, not new files\r\ngit add --ignore-removal .\r\n\r\n# Update only already tracked files\r\ngit add --no-all -u<\/code><\/pre><h3 class=\"heading-element\" id=\"213-git-commit-extended\"><span>21.3 Git Commit Extended<\/span>\n <a href=\"#213-git-commit-extended\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Amend and change author\r\ngit commit --amend --author=&#34;Name &lt;email@example.com&gt;&#34;\r\n\r\n# Commit with specific date\r\ngit commit --date=&#34;2025-01-01 10:00:00&#34;\r\n\r\n# Empty commit (trigger CI)\r\ngit commit --allow-empty -m &#34;Trigger CI&#34;\r\n\r\n# Commit without hooks\r\ngit commit --no-verify\r\n\r\n# Sign commit with GPG\r\ngit commit -S -m &#34;Signed commit&#34;\r\n\r\n# Commit with trailer\r\ngit commit -m &#34;Fix bug&#34; --trailer &#34;Reviewed-by: John &lt;john@example.com&gt;&#34;\r\n\r\n# Fixup commit (for autosquash)\r\ngit commit --fixup=abc1234\r\n\r\n# Squash commit (for autosquash)\r\ngit commit --squash=abc1234\r\n\r\n# Reuse commit message from another commit\r\ngit commit -C HEAD~1\r\n\r\n# Reuse message but edit it\r\ngit commit -c HEAD~1<\/code><\/pre><h3 class=\"heading-element\" id=\"214-git-branch-master-commands\"><span>21.4 Git Branch Master Commands<\/span>\n <a href=\"#214-git-branch-master-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># List branches with pattern\r\ngit branch --list &#34;feature\/*&#34;\r\n\r\n# List branches containing commit\r\ngit branch --contains abc1234\r\n\r\n# List branches not containing commit\r\ngit branch --no-contains abc1234\r\n\r\n# Create branch from specific commit\r\ngit branch feature-name abc1234\r\n\r\n# Track remote branch\r\ngit branch --track feature origin\/feature\r\n\r\n# Set upstream for current branch\r\ngit branch --set-upstream-to=origin\/main\r\n\r\n# Unset upstream\r\ngit branch --unset-upstream\r\n\r\n# Copy branch\r\ngit branch -c old-name new-name\r\n\r\n# Show branches and their upstreams\r\ngit branch -vv\r\n\r\n# Sort branches by committerdate\r\ngit branch --sort=-committerdate\r\n\r\n# List branches sorted by last commit\r\ngit for-each-ref --sort=-committerdate refs\/heads\/ --format=&#39;%(committerdate:short) %(refname:short)&#39;<\/code><\/pre><h3 class=\"heading-element\" id=\"215-git-log-advanced-queries\"><span>21.5 Git Log Advanced Queries<\/span>\n <a href=\"#215-git-log-advanced-queries\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Show commits by multiple authors\r\ngit log --author=&#34;John\\\\|Jane&#34;\r\n\r\n# Show commits with specific string in message\r\ngit log --grep=&#34;bug fix&#34;\r\n\r\n# Show commits that changed specific function\r\ngit log -L :functionName:path\/to\/file.js\r\n\r\n# Show commits adding or removing specific string\r\ngit log -S&#34;searchString&#34;\r\n\r\n# Show commits changing specific text (regex)\r\ngit log -G&#34;regex pattern&#34;\r\n\r\n# Show commits in date range\r\ngit log --since=&#34;2 weeks ago&#34;\r\n\r\n# Show commits only in current branch\r\ngit log main..feature\r\n\r\n# Show commits in either branch but not both\r\ngit log main...feature\r\n\r\n# Show reflog for specific branch\r\ngit reflog show main\r\n\r\n# Pretty format with colors\r\ngit log --pretty=format:&#34;%C(yellow)%h%Creset %C(blue)%an%Creset %C(green)%ar%Creset - %s&#34;\r\n\r\n# Show commits since tag\r\ngit log v1.0..HEAD\r\n\r\n# Show commits between two tags\r\ngit log v1.0..v2.0\r\n\r\n# Count commits by author\r\ngit shortlog -sn<\/code><\/pre><h3 class=\"heading-element\" id=\"216-git-diff-comprehensive\"><span>21.6 Git Diff Comprehensive<\/span>\n <a href=\"#216-git-diff-comprehensive\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Show word diff\r\ngit diff --word-diff\r\n\r\n# Show character diff\r\ngit diff --word-diff-regex=.\r\n\r\n# Patience algorithm (better results)\r\ngit diff --patience\r\n\r\n# Histogram algorithm\r\ngit diff --histogram\r\n\r\n# Highlight moved code\r\ngit diff --color-moved\r\n\r\n# Show function context\r\ngit diff -W\r\n\r\n# Show context lines\r\ngit diff -U10 # 10 lines of context\r\n\r\n# Diff between working tree and commit\r\ngit diff HEAD~2\r\n\r\n# Diff branch point\r\ngit diff $(git merge-base main feature)..feature\r\n\r\n# Ignore specific files\r\ngit diff -- . &#39;:!package-lock.json&#39;<\/code><\/pre><h3 class=\"heading-element\" id=\"217-git-stash-everything\"><span>21.7 Git Stash Everything<\/span>\n <a href=\"#217-git-stash-everything\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Stash with message\r\ngit stash push -m &#34;Work in progress on feature&#34;\r\n\r\n# Stash including untracked files\r\ngit stash -u\r\n\r\n# Stash everything including ignored files\r\ngit stash -a\r\n\r\n# Stash only specific files\r\ngit stash push -m &#34;message&#34; -- file1.js file2.js\r\n\r\n# Stash and keep index (staged changes)\r\ngit stash --keep-index\r\n\r\n# Show full diff of stash\r\ngit stash show -p stash@{0}\r\n\r\n# Create branch from stash\r\ngit stash branch new-branch stash@{0}\r\n\r\n# Apply stash to different branch\r\ngit checkout other-branch\r\ngit stash apply stash@{2}\r\n\r\n# Stash with patch mode\r\ngit stash -p\r\n\r\n# Stash only staged changes\r\ngit stash push --staged<\/code><\/pre><h3 class=\"heading-element\" id=\"218-git-rebase-master-class\"><span>21.8 Git Rebase Master Class<\/span>\n <a href=\"#218-git-rebase-master-class\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Rebase with autostash\r\ngit rebase --autostash main\r\n\r\n# Rebase with strategy\r\ngit rebase -X theirs main # Prefer their changes\r\ngit rebase -X ours main # Prefer our changes\r\n\r\n# Rebase preserving merges\r\ngit rebase --rebase-merges main\r\n\r\n# Interactive rebase with autosquash\r\ngit rebase -i --autosquash main\r\n\r\n# Rebase onto specific commit\r\ngit rebase --onto new-base old-base branch\r\n\r\n# Rebase and execute command after each commit\r\ngit rebase -x &#34;npm test&#34; main\r\n\r\n# Rebase with committer date\r\ngit rebase --committer-date-is-author-date main\r\n\r\n# Edit todo list during rebase\r\ngit rebase --edit-todo<\/code><\/pre><h3 class=\"heading-element\" id=\"219-git-cherry-pick-pro\"><span>21.9 Git Cherry-Pick Pro<\/span>\n <a href=\"#219-git-cherry-pick-pro\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Cherry-pick range of commits\r\ngit cherry-pick abc1234..def5678\r\n\r\n# Cherry-pick excluding start commit\r\ngit cherry-pick abc1234^..def5678\r\n\r\n# Cherry-pick with strategy\r\ngit cherry-pick -X theirs abc1234\r\n\r\n# Cherry-pick and sign off\r\ngit cherry-pick -s abc1234\r\n\r\n# Cherry-pick empty commits\r\ngit cherry-pick --allow-empty abc1234\r\n\r\n# Cherry-pick with mainline parent (for merge commits)\r\ngit cherry-pick -m 1 merge-commit\r\n\r\n# Cherry-pick from another repository\r\ngit --git-dir=..\/other-repo\/.git format-patch -k -1 --stdout abc1234 | git am -3 -k<\/code><\/pre><h3 class=\"heading-element\" id=\"2110-git-tag-complete-operations\"><span>21.10 Git Tag Complete Operations<\/span>\n <a href=\"#2110-git-tag-complete-operations\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># List tags matching pattern\r\ngit tag -l &#34;v1.*&#34;\r\n\r\n# Create annotated tag\r\ngit tag -a v1.0.0 -m &#34;Version 1.0.0&#34;\r\n\r\n# Tag with GPG signature\r\ngit tag -s v1.0.0 -m &#34;Signed version 1.0.0&#34;\r\n\r\n# Verify signed tag\r\ngit tag -v v1.0.0\r\n\r\n# Delete remote tag\r\ngit push origin :refs\/tags\/v1.0.0\r\n\r\n# Push annotated tags only\r\ngit push origin --follow-tags\r\n\r\n# Sort tags by version\r\ngit tag --sort=version:refname\r\n\r\n# Sort tags by date\r\ngit tag --sort=-creatordate\r\n\r\n# List tags containing commit\r\ngit tag --contains abc1234\r\n\r\n# Get latest tag\r\ngit describe --tags --abbrev=0\r\n\r\n# Archive specific tag\r\ngit archive --format=zip --output=v1.0.0.zip v1.0.0<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"22-hidden-git-tricks-and-shortcuts\"><span>22. Hidden Git Tricks and Shortcuts<\/span>\n <a href=\"#22-hidden-git-tricks-and-shortcuts\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"221-power-aliases\"><span>22.1 Power Aliases<\/span>\n <a href=\"#221-power-aliases\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Beautiful log\r\ngit config --global alias.lg &#34;log --graph --oneline --decorate --all&#34;\r\ngit config --global alias.lol &#34;log --graph --pretty=format:&#39;%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)&lt;%an&gt;%Creset&#39; --abbrev-commit&#34;\r\n\r\n# Contributors\r\ngit config --global alias.contributors &#39;shortlog -sn&#39;\r\ngit config --global alias.who &#39;shortlog -sne --all&#39;\r\n\r\n# Undo shortcuts\r\ngit config --global alias.undo &#39;reset HEAD~1 --mixed&#39;\r\ngit config --global alias.undohard &#39;reset HEAD~1 --hard&#39;\r\n\r\n# List all aliases\r\ngit config --global alias.alias &#39;!git config --get-regexp ^alias\\\\. | sed -e s\/^alias\\\\.\/\/ -e s\/\\\\ \/\\\\ =\\\\ \/&#39;<\/code><\/pre><h3 class=\"heading-element\" id=\"222-advanced-tricks\"><span>22.2 Advanced Tricks<\/span>\n <a href=\"#222-advanced-tricks\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Find who changed specific line\r\ngit blame file.txt\r\n\r\n# Find who changed line range\r\ngit blame -L 10,20 file.txt\r\n\r\n# Search commits for string\r\ngit log --all --grep=&#39;search term&#39;\r\n\r\n# Find deleted file\r\ngit log --all --full-history -- path\/to\/file.txt\r\n\r\n# Restore deleted file\r\ngit checkout $(git rev-list -n 1 HEAD -- deleted-file.txt)^ -- deleted-file.txt\r\n\r\n# Show file at specific commit\r\ngit show HEAD~3:path\/to\/file.txt\r\n\r\n# Count commits\r\ngit rev-list --count HEAD\r\n\r\n# Find common ancestor\r\ngit merge-base branch1 branch2\r\n\r\n# Clean untracked files (dry run)\r\ngit clean -n\r\n\r\n# Clean everything\r\ngit clean -fdx\r\n\r\n# Assume file unchanged\r\ngit update-index --assume-unchanged file.txt\r\n\r\n# Skip worktree (better than assume-unchanged)\r\ngit update-index --skip-worktree file.txt\r\n\r\n# Export repository to archive\r\ngit archive --format=zip --output=project.zip HEAD\r\n\r\n# Create bundle (portable repo)\r\ngit bundle create repo.bundle --all\r\n\r\n# Search in all branches\r\ngit grep &#34;search term&#34; $(git rev-list --all)\r\n\r\n# Delete gone remote branches\r\ngit branch -vv | grep &#39;: gone]&#39; | awk &#39;{print $1}&#39; | xargs git branch -D\r\n\r\n# Show repository statistics\r\ngit count-objects -vH\r\n\r\n# Verify repository integrity\r\ngit fsck<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"23-github-specific-features\"><span>23. GitHub-Specific Features<\/span>\n <a href=\"#23-github-specific-features\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"231-github-cli-gh-commands\"><span>23.1 GitHub CLI (gh) Commands<\/span>\n <a href=\"#231-github-cli-gh-commands\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Authentication\r\ngh auth login\r\ngh auth logout\r\ngh auth status\r\n\r\n# Repository operations\r\ngh repo create my-new-repo\r\ngh repo clone owner\/repo\r\ngh repo fork owner\/repo\r\ngh repo view owner\/repo\r\n\r\n# Pull Requests\r\ngh pr create --title &#34;Title&#34; --body &#34;Body&#34;\r\ngh pr list\r\ngh pr view 123\r\ngh pr checkout 123\r\ngh pr checks\r\ngh pr merge 123\r\ngh pr review 123 --approve\r\ngh pr review 123 --request-changes\r\ngh pr close 123\r\n\r\n# Issues\r\ngh issue create --title &#34;Bug&#34; --body &#34;Description&#34;\r\ngh issue list\r\ngh issue view 123\r\ngh issue close 123\r\ngh issue reopen 123\r\n\r\n# Releases\r\ngh release create v1.0.0\r\ngh release list\r\ngh release view v1.0.0\r\ngh release download v1.0.0\r\n\r\n# Gists\r\ngh gist create file.txt\r\ngh gist list\r\ngh gist view gist-id\r\ngh gist edit gist-id\r\n\r\n# Actions\r\ngh workflow list\r\ngh workflow view workflow-name\r\ngh run list\r\ngh run view run-id\r\ngh run watch\r\n\r\n# Secrets\r\ngh secret set SECRET_NAME\r\ngh secret list\r\n\r\n# API calls\r\ngh api \/user\r\ngh api repos\/:owner\/:repo\/issues<\/code><\/pre><h3 class=\"heading-element\" id=\"232-github-url-tricks\"><span>23.2 GitHub URL Tricks<\/span>\n <a href=\"#232-github-url-tricks\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code># Press &#39;?&#39; on any page to see shortcuts\r\n\r\n# Repository shortcuts:\r\n# t - File finder\r\n# w - Branch selector\r\n# s - Focus search\r\n# l - Edit labels (on issues)\r\n# y - Freeze URL to current commit\r\n\r\n# Line highlighting\r\nhttps:\/\/github.com\/user\/repo\/blob\/main\/file.js#L10\r\nhttps:\/\/github.com\/user\/repo\/blob\/main\/file.js#L10-L20\r\n\r\n# Diff\/Patch URLs\r\nhttps:\/\/github.com\/user\/repo\/pull\/123.diff\r\nhttps:\/\/github.com\/user\/repo\/pull\/123.patch\r\n\r\n# Ignore whitespace in diff\r\nhttps:\/\/github.com\/user\/repo\/pull\/123?w=1\r\n\r\n# Adjust tab size\r\nhttps:\/\/github.com\/user\/repo\/blob\/main\/file.js?ts=4\r\n\r\n# Blame view\r\nhttps:\/\/github.com\/user\/repo\/blame\/main\/file.js<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"24-advanced-troubleshooting-guide\"><span>24. Advanced Troubleshooting Guide<\/span>\n <a href=\"#24-advanced-troubleshooting-guide\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"241-common-problems-and-solutions\"><span>24.1 Common Problems and Solutions<\/span>\n <a href=\"#241-common-problems-and-solutions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Problem Recovery Toolkit<\/strong>: These commands will save you from almost any Git disaster.<\/div>\n <\/div>\n<\/div>\n<pre><code># Undo last commit but keep changes\r\ngit reset --soft HEAD~1\r\n\r\n# Undo last commit and discard changes\r\ngit reset --hard HEAD~1\r\n\r\n# Recover deleted branch\r\ngit reflog\r\ngit checkout -b recovered-branch &lt;commit-sha&gt;\r\n\r\n# Undo bad merge\r\ngit reset --hard HEAD~1 # If not pushed\r\ngit revert -m 1 &lt;merge-commit&gt; # If pushed\r\n\r\n# Fix wrong commit message\r\ngit commit --amend -m &#34;Correct message&#34;\r\n\r\n# Remove file from all history\r\ngit filter-branch --tree-filter &#39;rm -f passwords.txt&#39; HEAD\r\n\r\n# Or use BFG (faster)\r\nbfg --delete-files passwords.txt\r\n\r\n# Recover lost commits\r\ngit fsck --lost-found\r\n\r\n# Fix detached HEAD\r\ngit checkout main\r\ngit branch temp-branch &lt;commit-sha&gt;\r\n\r\n# Undo git reset --hard\r\ngit reflog\r\ngit reset --hard HEAD@{1}\r\n\r\n# Remove untracked files safely\r\ngit clean -n # Dry run first\r\ngit clean -fd # Then remove\r\n\r\n# Fix &#34;pathspec did not match&#34;\r\ngit rm --cached &lt;file&gt;\r\ngit commit -m &#34;Remove file from tracking&#34;<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"25-git-performance-optimization\"><span>25. Git Performance Optimization<\/span>\n <a href=\"#25-git-performance-optimization\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><pre><code># Enable garbage collection\r\ngit config --global gc.auto 4096\r\n\r\n# Manual garbage collection\r\ngit gc --aggressive --prune=now\r\n\r\n# Enable delta compression\r\ngit config --global core.compression 9\r\n\r\n# Enable parallel operations\r\ngit config --global core.preloadindex true\r\n\r\n# Use filesystem monitoring\r\ngit config --global core.fsmonitor true\r\n\r\n# Shallow clone for faster CI\r\ngit clone --depth 1 repo-url\r\n\r\n# Sparse checkout for large repos\r\ngit clone --filter=blob:none --sparse repo-url\r\ngit sparse-checkout set path\/to\/dir\r\n\r\n# Git LFS for large files\r\ngit lfs install\r\ngit lfs track &#34;*.psd&#34;\r\ngit add .gitattributes\r\n\r\n# Repack repository\r\ngit repack -a -d --depth=250 --window=250<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"26-complete-gitignore-templates\"><span>26. Complete .gitignore Templates<\/span>\n <a href=\"#26-complete-gitignore-templates\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><pre><code># Node.js\r\nnode_modules\/\r\nnpm-debug.log*\r\n.env\r\ndist\/\r\nbuild\/\r\n\r\n# Python\r\n__pycache__\/\r\n*.py[cod]\r\nvenv\/\r\n.pytest_cache\/\r\n\r\n# Java\r\n*.class\r\ntarget\/\r\n*.jar\r\n*.war\r\n\r\n# macOS\r\n.DS_Store\r\n.AppleDouble\r\n\r\n# Windows\r\nThumbs.db\r\nDesktop.ini\r\n\r\n# IDEs\r\n.vscode\/\r\n.idea\/\r\n*.swp\r\n*.swo\r\n\r\n# Logs\r\n*.log\r\nlogs\/\r\n\r\n# Secrets\r\n*.pem\r\n*.key\r\n.env\r\nsecrets.yml<\/code><\/pre><hr>\n<h2 class=\"heading-element\" id=\"27-final-pro-tips\"><span>27. Final Pro Tips<\/span>\n <a href=\"#27-final-pro-tips\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Master These and You&rsquo;ll Be a Git Expert<\/strong><\/div>\n <\/div>\n<\/div>\n<ol>\n<li><strong>Use <code>git reflog<\/code> religiously<\/strong> - It&rsquo;s your time machine<\/li>\n<li><strong>Commit often, push frequently<\/strong> - Small commits are easier to revert<\/li>\n<li><strong>Write meaningful commit messages<\/strong> - Future you will thank you<\/li>\n<li><strong>Use branches for everything<\/strong> - Keep main clean<\/li>\n<li><strong>Test before you merge<\/strong> - Always run tests<\/li>\n<li><strong>Learn interactive rebase<\/strong> - Clean up history before pushing<\/li>\n<li><strong>Use <code>git stash<\/code><\/strong> - Don&rsquo;t commit half-baked work<\/li>\n<li><strong>Set up aliases<\/strong> - Save keystrokes<\/li>\n<li><strong>Read <code>git log<\/code><\/strong> - Understand project history<\/li>\n<li><strong>Practice on test repos<\/strong> - Break things to learn<\/li>\n<\/ol>\n<hr>\n<p><strong>You are now a Git and GitHub master! \ud83c\udf89<\/strong><\/p>\n<p>This comprehensive guide covered 1000+ commands, all workflows, GitHub features, troubleshooting, and best practices. Keep this as your reference and contribute to open source!<\/p>"},{"title":"Real-Time Voice Translator \u2014 Breaking Language Barriers with AI","link":"https:\/\/samir.pages.dev\/posts\/real-time-voice-translator\/","pubDate":"Fri, 05 Jan 2024 00:00:00 +0000","author":"netproxy@duck.com (Samir)","guid":"https:\/\/samir.pages.dev\/posts\/real-time-voice-translator\/","category":"Projects","description":"<img src=\"https:\/\/samir.pages.dev\/files\/real-time-voice-translator-og-image.webp\" alt=\"featured image\" referrerpolicy=\"no-referrer\"><p><b>Repository Link: <a href=\"https:\/\/github.com\/SamirPaulb\/real-time-voice-translator\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">github.com\/SamirPaulb\/real-time-voice-translator<\/a><\/b><\/p>\n<h2 class=\"heading-element\" id=\"abstract\"><span>Abstract<\/span>\n <a href=\"#abstract\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p><em>Cross-lingual communication is a challenging task that requires accurate translation and natural and expressive speech. Existing solutions often rely on intermediate text representations, which introduce latency and lose the prosodic features of the original speech. In this paper, we present Real-Time Voice Translator, a machine learning project that aims to overcome these limitations by using deep neural networks to directly translate voice from one language to another in real-time. Our project is a desktop application that supports Windows, Linux, and Mac operating systems. It allows users to select the languages they want to translate between and start speaking. The application listens to the user&rsquo;s voice and provides instant translations in real time while preserving the tone and emotion of the speaker. The application can also translate conversations between two or more people, enabling natural and fluent cross-lingual interactions. We evaluate our project on various metrics, such as translation quality, speech quality, latency, and user satisfaction. We demonstrate that our project achieves high performance and provides a seamless and natural experience of cross-lingual communication. We also discuss the future perspectives of our project, such as using voice cloning features to mimic the speaker&rsquo;s voice in the target language and enhancing the emotional preservation of the translated speech. We believe that our project has the potential to revolutionize the field of cross-lingual communication and open new possibilities for cross-cultural exchange and collaboration.<\/em><\/p>\n<p><strong>Index Terms<\/strong>: <code>Real-Time Voice Translation<\/code>, <code>Deep Learning<\/code>, <code>Voice Tone and Emotion Preservation<\/code>, <code>Desktop Application<\/code>.<\/p>\n<h2 class=\"heading-element\" id=\"introduction\"><span>Introduction<\/span>\n <a href=\"#introduction\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Imagine bridging language barriers in real time, preserving emotional nuances and fostering genuine cross-cultural understanding. Real-Time Voice Translator (RTVT) unlocks this possibility, utilizing deep learning to translate spoken words instantly, while faithfully mirroring the speaker&rsquo;s tone and intent. This open-source, desktop application empowers seamless communication across languages, fostering empathy, collaboration, and a more connected world. This research unveils the technical backbone and transformative potential of RTVT, a tool poised to redefine how we interact and collaborate beyond linguistic borders.<\/p>\n<h2 class=\"heading-element\" id=\"studies-and-findings\"><span>Studies and Findings<\/span>\n <a href=\"#studies-and-findings\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>The allure of instantaneous, seamless speech-to-speech translation across languages is undeniable. Research in end-to-end models like Google&rsquo;s Translatotron, directly mapping speech spectrograms, offers a glimpse into this future. However, the realities of limited language compatibility and lingering technical hurdles made such an approach unsuitable for this real-time voice translator project.<\/p>\n<p>Drawing inspiration from established technologies, we embraced a hybrid approach, meticulously dissecting the translation process into speech-to-text, text-to-text translation, and finally, text-to-speech synthesis. This multi-step journey, while potentially a tad slower than its end-to-end counterparts, unlocked several key advantages. Firstly, it provided access to a vast pool of existing text translation models, vastly expanding the supported language pairs. Secondly, it paved the way for incorporating transliteration features, a valuable tool for bridging the gap between written and spoken forms of a language.<\/p>\n<p>This decision wasn&rsquo;t merely a practical compromise; it was a deliberate move towards a more robust and adaptable framework. While sacrificing the immediacy of spectrogram-based models, we gained a translation engine capable of tackling a wider range of languages and scenarios. As the field of speech-to-speech translation continues to evolve, this hybrid approach offers a stable platform for ongoing development, promising to bring the dream of real-time, cross-lingual communication ever closer to reality.<\/p>\n<h2 class=\"heading-element\" id=\"speech-translation-model\"><span>Speech Translation Model<\/span>\n <a href=\"#speech-translation-model\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>The Speech Translation Model (STM) orchestrates a series of interconnected processes to achieve real-time, cross-lingual voice communication. Here&rsquo;s a breakdown of its core steps:<\/p>\n<ol>\n<li><strong>Voice Input and Automatic Speech Recognition (ASR)<\/strong>:<\/li>\n<\/ol>\n<ul>\n<li>\n<p>The journey begins with capturing the user&rsquo;s spoken utterance in the source language.<\/p>\n<\/li>\n<li>\n<p>ASR technology meticulously analyzes the audio signal, mapping its acoustic features to linguistic units.<\/p>\n<\/li>\n<li>\n<p>The intricate task of identifying phonemes, words, and their boundaries within continuous speech is performed with remarkable accuracy.<\/p>\n<\/li>\n<\/ul>\n<ol start=\"2\">\n<li><strong>Input Voice to Text Conversion<\/strong>:<\/li>\n<\/ol>\n<ul>\n<li>\n<p>The ASR process culminates in a textual representation of the spoken input, ready for further linguistic transformations.<\/p>\n<\/li>\n<li>\n<p>This stage ensures that the model has a structured foundation for subsequent translation and transliteration operations.<\/p>\n<\/li>\n<\/ul>\n<ol start=\"3\">\n<li><strong>Transliteration for Textual Adaptation<\/strong>:<\/li>\n<\/ol>\n<ul>\n<li>\n<p>To bridge the gap between different writing systems and enhance translation accuracy, transliteration steps in.<\/p>\n<\/li>\n<li>\n<p>It meticulously maps the characters of the source language text to their closest equivalents in the target language.<\/p>\n<\/li>\n<li>\n<p>This process seamlessly adapts language-specific nuances, ensuring a smooth transition between written forms.<\/p>\n<\/li>\n<\/ul>\n<ol start=\"4\">\n<li><strong>Translation of Transliterated Text<\/strong>:<\/li>\n<\/ol>\n<ul>\n<li>\n<p>With the text carefully adapted for the target language, the translation engine takes centre stage.<\/p>\n<\/li>\n<li>\n<p>Leveraging sophisticated machine translation algorithms, it deciphers the meaning of the source text and artfully reconstructs it in the target language.<\/p>\n<\/li>\n<li>\n<p>The model navigates the complexities of grammar, syntax, and semantics, striving for fluency and accuracy in the translated output.<\/p>\n<\/li>\n<\/ul>\n<ol start=\"5\">\n<li><strong>Text-to-Speech Synthesis<\/strong>:<\/li>\n<\/ol>\n<ul>\n<li>\n<p>The translated text now embarks on a journey back into the auditory realm.<\/p>\n<\/li>\n<li>\n<p>Text-to-Speech (TTS) technology meticulously transforms written words into a natural-sounding speech signal.<\/p>\n<\/li>\n<li>\n<p>This stage meticulously recreates the nuances of human intonation, rhythm, and pronunciation, breathing life into the translated message.<\/p>\n<\/li>\n<\/ul>\n<ol start=\"6\">\n<li><strong>Voice Output<\/strong>:<\/li>\n<\/ol>\n<ul>\n<li>\n<p>The final step unveils the translated utterance in the target language, spoken aloud for the listener.<\/p>\n<\/li>\n<li>\n<p>The model gracefully renders the translated text as intelligible speech, completing the cross-lingual communication loop.<\/p>\n<\/li>\n<li>\n<p>solid foundation for subsequent translation.<\/p>\n<\/li>\n<li>\n<p>deep-translator: This versatile library offers a comprehensive suite of translation capabilities, ensuring linguistic accuracy and fluency across a diverse range of language pairs.<\/p>\n<\/li>\n<li>\n<p>google-transliteration-api: This API elegantly handled the task of transliteration, adapting text between different writing systems, fostering a seamless transition between languages.<\/p>\n<\/li>\n<li>\n<p>cx-Freeze: This tool enabled the packaging of the STM into standalone executable applications for Windows, Linux, and macOS, significantly broadening its accessibility and potential user base.<\/p>\n<\/li>\n<\/ul>\n<p><strong>Program Flow Architecture:<\/strong><\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/images\/remote\/73dd62d6-798d-4129-aff3-16d6d932a817_2715301176644608949.png\" title=\"Real-Time Voice Translator System Architecture and Data Flow\" data-thumbnail=\"\/images\/remote\/73dd62d6-798d-4129-aff3-16d6d932a817_2715301176644608949.png\" data-sub-html=\"<h2>Real-Time Voice Translator System Architecture - Complete program flow diagram showing Voice Input \u2192 Speech Recognition \u2192 Transliteration \u2192 Translation \u2192 Text-to-Speech \u2192 Voice Output pipeline with PDFNetPython, deep-translator, and gTTS integration for seamless cross-lingual communication<\/h2><p>Real-Time Voice Translator System Architecture and Data Flow<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/images\/remote\/73dd62d6-798d-4129-aff3-16d6d932a817_2715301176644608949.png' srcset=\"https:\/\/samir.pages.dev\/images\/remote\/73dd62d6-798d-4129-aff3-16d6d932a817_2715301176644608949_hu_58eb8ec5c02098a0.webp 800w, https:\/\/samir.pages.dev\/images\/remote\/73dd62d6-798d-4129-aff3-16d6d932a817_2715301176644608949_hu_59530af018bda42a.webp 1200w, https:\/\/samir.pages.dev\/images\/remote\/73dd62d6-798d-4129-aff3-16d6d932a817_2715301176644608949_hu_43814ef30b0f60db.webp 1600w\" alt=\"Real-Time Voice Translator System Architecture - Complete program flow diagram showing Voice Input \u2192 Speech Recognition \u2192 Transliteration \u2192 Translation \u2192 Text-to-Speech \u2192 Voice Output pipeline with PDFNetPython, deep-translator, and gTTS integration for seamless cross-lingual communication\" height=\"402\" width=\"1417\"><\/a><figcaption class=\"image-caption\">Real-Time Voice Translator System Architecture and Data Flow<\/figcaption>\n <\/figure><\/p>\n<ul>\n<li>\n<p><strong>Voice Input<\/strong>: The journey begins with capturing the user&rsquo;s spoken utterance in the source language, meticulously handled by pyaudio.<\/p>\n<\/li>\n<li>\n<p><strong>Automatic Speech Recognition<\/strong>: SpeechRecognition diligently analyzes the audio signal, converting it into text for further processing.<\/p>\n<\/li>\n<li>\n<p><strong>Transliteration<\/strong>: The google-transliteration-api gracefully adapts the text to the target language&rsquo;s writing system, ensuring optimal translation accuracy.<\/p>\n<\/li>\n<li>\n<p><strong>Translation<\/strong>: deep-translator leverages sophisticated translation algorithms to decipher the meaning of the source text and reconstruct it in the target language, preserving linguistic nuances.<\/p>\n<\/li>\n<li>\n<p><strong>Text-to-Speech Synthesis<\/strong>: gTTS meticulously transforms the translated text into a natural-sounding speech signal, breathing life into the translated message.<\/p>\n<\/li>\n<li>\n<p><strong>Voice Output<\/strong>: playsound delivers the translated utterance in the target language, completing the cross-lingual communication loop.<\/p>\n<\/li>\n<\/ul>\n<h2 class=\"heading-element\" id=\"installation-and-usage\"><span>Installation and Usage<\/span>\n <a href=\"#installation-and-usage\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"dependencies\"><span>Dependencies<\/span>\n <a href=\"#dependencies\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The Real-Time Voice Translator requires the following Python packages and dependencies:<\/p>\n<pre><code># Python Version Requirement\nPython &lt;= 3.11\n\n# Core Translation and Speech Processing Libraries\ngTTS # Google Text-to-Speech for voice synthesis\npyaudio # Audio I\/O for capturing voice input\nplaysound==1.2.2 # Cross-platform audio playback\ndeep-translator # Multi-engine translation support\nSpeechRecognition # Automatic speech recognition\ngoogle-transliteration-api # Character transliteration between languages\ncx-Freeze # Application packaging for distribution<\/code><\/pre><h3 class=\"heading-element\" id=\"getting-started\"><span>Getting Started<\/span>\n <a href=\"#getting-started\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Follow these steps to set up and run Real-Time Voice Translator on your local machine:<\/p>\n<p><strong>Step 1: Clone Repository and Create Virtual Environment<\/strong><\/p>\n<p>Clone the <a href=\"https:\/\/github.com\/SamirPaulb\/real-time-voice-translator\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Real-Time Voice Translator repository<\/a> and set up an isolated Python virtual environment for dependency management.<\/p>\n<pre><code># Clone the repository\ngit clone https:\/\/github.com\/SamirPaulb\/real-time-voice-translator.git\ncd real-time-voice-translator\n\n# Create a new virtual environment\npython -m venv env\n\n# Activate virtual environment on Linux\/MacOS\nsource env\/bin\/activate\n\n# Activate virtual environment on Windows\nenv\\Scripts\\activate<\/code><\/pre><p><strong>Step 2: Install Required Dependencies<\/strong><\/p>\n<p>Install all necessary Python packages and dependencies for the voice translator.<\/p>\n<pre><code># Upgrade pip and wheel for better package management\npip install --upgrade pip wheel\n\n# Install all required packages from requirements.txt\npip install -r requirements.txt<\/code><\/pre><p><strong>Step 3: Run the Application<\/strong><\/p>\n<p>Launch the Real-Time Voice Translator and start translating speech!<\/p>\n<pre><code># Start the voice translator application\npython main.py\n\n# The application will open with a GUI interface\n# Select source and target languages, then start speaking<\/code><\/pre><h3 class=\"heading-element\" id=\"pre-built-application-downloads\"><span>Pre-Built Application Downloads<\/span>\n <a href=\"#pre-built-application-downloads\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>For users who prefer not to install Python or run from source, pre-built standalone applications are available for Windows, Linux, and macOS.<\/p>\n<p><a href=\"https:\/\/github.com\/SamirPaulb\/real-time-voice-translator\/releases\/latest\"><img src=\"https:\/\/user-images.githubusercontent.com\/132539454\/278971282-8d676023-a03a-463c-8e55-3f0afe6e3e58.svg\" alt=\"Download Real-Time Voice Translator - Windows, Linux, Mac installers available\" title=\"Download Latest Release\"><\/a><\/p>\n<p><strong>Building Standalone Executables<\/strong><\/p>\n<p>The application uses <a href=\"https:\/\/github.com\/marcelotduarte\/cx_Freeze\/tree\/main\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">cx_Freeze<\/a> to package Python code into standalone executables. Build settings can be customized by modifying the <a href=\"https:\/\/github.com\/SamirPaulb\/real-time-voice-translator\/blob\/main\/setup.py\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">setup.py<\/a> file.<\/p>\n<pre><code># Build Windows MSI installer\npython setup.py bdist_msi\n\n# Build Linux RPM package\npython setup.py bdist_rpm\n\n# Build macOS application bundle\npython setup.py bdist_mac<\/code><\/pre><h3 class=\"heading-element\" id=\"application-interface\"><span>Application Interface<\/span>\n <a href=\"#application-interface\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The Real-Time Voice Translator features an intuitive graphical user interface (GUI) that makes cross-lingual communication accessible to all users.<\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/images\/remote\/f96a4115-a88f-4096-9a00-954b8527d872_2728748172307284203.png\" title=\"Real-Time Voice Translator Application Interface with Language Selection\" data-thumbnail=\"\/images\/remote\/f96a4115-a88f-4096-9a00-954b8527d872_2728748172307284203.png\" data-sub-html=\"<h2>Real-Time Voice Translator GUI Application Interface - User-friendly interface showing source and target language selection dropdowns, microphone input controls, real-time translation status indicators, and start\/stop buttons for seamless voice translation experience<\/h2><p>Real-Time Voice Translator Application Interface with Language Selection<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/images\/remote\/f96a4115-a88f-4096-9a00-954b8527d872_2728748172307284203.png' srcset=\"https:\/\/samir.pages.dev\/images\/remote\/f96a4115-a88f-4096-9a00-954b8527d872_2728748172307284203_hu_2d35a939ee380804.webp 800w, https:\/\/samir.pages.dev\/images\/remote\/f96a4115-a88f-4096-9a00-954b8527d872_2728748172307284203_hu_a3bdd70f7266cc22.webp 1200w, https:\/\/samir.pages.dev\/images\/remote\/f96a4115-a88f-4096-9a00-954b8527d872_2728748172307284203_hu_96bfc5387e78e146.webp 1600w\" alt=\"Real-Time Voice Translator GUI Application Interface - User-friendly interface showing source and target language selection dropdowns, microphone input controls, real-time translation status indicators, and start\/stop buttons for seamless voice translation experience\" height=\"723\" width=\"1053\"><\/a><figcaption class=\"image-caption\">Real-Time Voice Translator Application Interface with Language Selection<\/figcaption>\n <\/figure><\/p>\n<h2 class=\"heading-element\" id=\"conclusion\"><span>Conclusion<\/span>\n <a href=\"#conclusion\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>Real-Time Voice Translator shatters language barriers with its deep learning-powered hybrid approach. Beyond accurate translations, it captures the essence of human speech, fostering genuine cross-cultural understanding. This research unveils its robust framework, adaptable design, and potential for future advancements like voice cloning and emotion preservation. Real-Time Voice Translator intuitive interface and cross-platform compatibility empower diverse users to navigate the world with ease. More than just a tool, it&rsquo;s a bridge of empathy and collaboration, one voice at a time. By embracing Real-Time Voice Translator, we step closer to a world where communication transcends borders, uniting cultures and shaping a more connected future.<\/p>\n<h2 class=\"heading-element\" id=\"future-work\"><span>Future Work<\/span>\n <a href=\"#future-work\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>While this project currently delivers impressive real-time translations, the future holds even greater potential for capturing the full spectrum of human communication. Sentiment and emotion analysis models like EmoNet and SyntaxNet offer exciting possibilities for preserving the speaker&rsquo;s intended meaning beyond mere words. Integrating these tools could allow Real-Time Voice Translator to translate expressions of joy, anger, or sarcasm with nuanced accuracy, fostering deeper cross-cultural understanding.<\/p>\n<p>Open-source toolkits like PaddleSpeech and espnet, known for their advanced speech-processing capabilities, could further enhance the translation process. Their deep learning frameworks offer the potential for improvements in speech recognition, natural language understanding, and text-to-speech synthesis. Additionally, incorporating SoftVC VITS Singing Voice Conversion technology could unlock fascinating avenues for translating emotional melodies and vocal inflections, adding a truly human touch to translated speech.<\/p>\n<p>We&rsquo;re actively exploring the integration of OpenAI&rsquo;s Whisper ASR model, renowned for its speech recognition accuracy, and ElevenLabs&rsquo; natural-sounding speech APIs. These advancements promise to elevate the user experience, delivering translated speech that seamlessly captures the speaker&rsquo;s original voice quality and emotional tone. Finally, accent softening models like Tomato.ai could be implemented to reduce speaker-specific characteristics in the translated speech, ensuring clearer and more universal comprehension.<\/p>\n<p>By embracing these cutting-edge technologies and pursuing continuous research, Real-Time Voice Translator aims to transcend the limitations of traditional translation. Our vision is to create a tool that not only bridges languages but also bridges hearts, fostering a world where emotions and intentions resonate across all barriers.<\/p>\n<h2 class=\"heading-element\" id=\"references\"><span>References<\/span>\n <a href=\"#references\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><ol>\n<li>\n<p>Cambria, Erik, and Jamin Shi. &ldquo;Semantic sentiment analysis.&rdquo; IEEE Transactions on Affective Computing 7.4 (2015): 266-279.<\/p>\n<\/li>\n<li>\n<p>Socher, Richard, et al. &ldquo;Recursive deep learning for sentiment analysis.&rdquo; Proceedings of the 28th International Conference on Machine Learning. ACM, 2013.<\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/PaddlePaddle\/PaddleSpeech\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">PaddlePaddle Team<\/a>. paddlepaddle speech recognition ON PaddlePaddle paddlepaddle.org.cn.<\/p>\n<\/li>\n<li>\n<p>ESPNet Working Group. &ldquo;ESPnet.&rdquo; GitHub Pages, github.com.<\/p>\n<\/li>\n<li>\n<p>Hsu, Wei-Ning, et al. &ldquo;SoftVC: High-fidelity TTS with Mel-Style Transfer.&rdquo; arXiv preprint arXiv:2301.04765 (2023).<\/p>\n<\/li>\n<li>\n<p>OpenAI <a href=\"https:\/\/github.com\/openai\/whisper\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Whisper<\/a>: Open-Source Speech Recognition.<\/p>\n<\/li>\n<li>\n<p>ElevenLabs. &ldquo;ElevenLabs.&rdquo; eleventlabs.io.<\/p>\n<\/li>\n<li>\n<p>Tomato.ai. &ldquo;Tomato.ai&rdquo;.<\/p>\n<\/li>\n<li>\n<p>Mohri, Mehryar, et al. &ldquo;Foundations of machine learning.&rdquo; MIT press, 2018.<\/p>\n<\/li>\n<\/ol>\n<hr>\n<p>This post is licensed under a <a href=\"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0)<\/a>. Distribution and adaptation are permitted under the terms of the license, with appropriate attribution required. All rights not expressly granted are reserved. For further information, please visit <a href=\"https:\/\/www.dmca.com\/r\/jkzgz6y\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">dmca.com\/r\/jkzgz6y<\/a>.<\/p>\n<p><a href=\"https:\/\/www.dmca.com\/r\/jkzgz6y\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/images\/remote\/dmca-badge-w150-5x1-09_2242085645524947603.png' srcset=\"https:\/\/samir.pages.dev\/images\/remote\/dmca-badge-w150-5x1-09_2242085645524947603_hu_cde504d56e12081f.webp 800w, https:\/\/samir.pages.dev\/images\/remote\/dmca-badge-w150-5x1-09_2242085645524947603_hu_bd864ac780e67bc8.webp 1200w, https:\/\/samir.pages.dev\/images\/remote\/dmca-badge-w150-5x1-09_2242085645524947603_hu_1f057d39173cf3f3.webp 1600w\" alt=\"DMCA.com Protection Status\" height=\"30\" width=\"150\"><\/a><\/p>\n"},{"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","link":"https:\/\/samir.pages.dev\/posts\/data-structures-and-algorithms-for-coding-interview\/","pubDate":"Wed, 20 Dec 2023 00:00:00 +0000","author":"netproxy@duck.com (Samir)","guid":"https:\/\/samir.pages.dev\/posts\/data-structures-and-algorithms-for-coding-interview\/","category":"Computer-Science","description":"<img src=\"https:\/\/samir.pages.dev\/files\/dsalgo.gif\" alt=\"featured image\" referrerpolicy=\"no-referrer\"><p><strong>Complete Data Structures and Algorithms Resource for Technical Interview Success<\/strong><\/p>\n<h2 id=\"introduction\">Repository Overview<\/h2>\n<div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Comprehensive Interview Preparation Resource<\/strong><\/p>\n<p>This repository contains solutions to <strong>3000+ coding problems<\/strong> covering all essential data structures and algorithms topics. Perfect for preparing for technical interviews at <strong>FAANG<\/strong> and top product-based companies.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<p><strong>Repository Link:<\/strong> <a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">https:\/\/github.com\/SamirPaulb\/DSAlgo<\/a><\/p>\n<p>In this meticulously organized repository, I have compiled solutions to various problems and concepts of Data Structures and Algorithms in <strong>Python<\/strong>, structured systematically for efficient learning and quick revision.<\/p>\n<!-- --- -->\n<!-- <h2 id=\"repository-statistics\">Repository Statistics<\/h2>\n\n[![DSAlgo Repository Card \u2014 Data Structures and Algorithms for Coding Interview](https:\/\/github-readme-stats.vercel.app\/api\/pin\/?username=SamirPaulb&repo=DSAlgo&show_owner=true \"DSAlgo - Comprehensive DSA resource with 3000+ LeetCode solutions in Python\")](https:\/\/github.com\/SamirPaulb\/DSAlgo) -->\n<hr>\n<h2 id=\"topics-covered\">Topics Covered in Detail<\/h2>\n<p><div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Complete Coverage of Interview Topics<\/strong><\/p>\n<p>This repository systematically covers all essential data structures and algorithms topics asked in technical interviews. Each topic includes multiple problem variations, optimal solutions, and detailed explanations.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<strong>Core Data Structures and Algorithms:<\/strong><\/p>\n<ol>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/01_LeetCode\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>LeetCode All Problems Solutions<\/strong><\/a> \u2014 3000+ problems with optimal Python solutions<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/02_Dynamic-Programming\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Dynamic Programming<\/strong><\/a> \u2014 Complete DP patterns with handwritten notes<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/03_Sorting-Algorithms\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Sorting Algorithms<\/strong><\/a> \u2014 All sorting techniques with time complexity analysis<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/04_LinkedList\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>LinkedList<\/strong><\/a> \u2014 Single, double, circular linked lists with problem patterns<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/05_Object-Oriented-Programming\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Object-Oriented Programming<\/strong><\/a> \u2014 OOP concepts for system design<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/06_Binary-Trees\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Binary Trees<\/strong><\/a> \u2014 Tree traversals, BST, AVL trees<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/07_Graph\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Graph Algorithms<\/strong><\/a> \u2014 DFS, BFS, shortest paths, MST<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/08_Heap\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Heap<\/strong><\/a> \u2014 Priority queue implementations<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/09_Matrix\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Matrix<\/strong><\/a> \u2014 2D array problems and techniques<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/10_Trie\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Trie<\/strong><\/a> \u2014 Prefix trees and autocomplete<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/11_Binary-Search\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Binary Search<\/strong><\/a> \u2014 Search space optimization<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/12_Backtracking\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Backtracking<\/strong><\/a> \u2014 Exhaustive search patterns<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/13_Stack\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Stack<\/strong><\/a> \u2014 Monotonic stack patterns<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/14_Queue\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Queue<\/strong><\/a> \u2014 Queue variations and applications<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/15_Greedy\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Greedy<\/strong><\/a> \u2014 Greedy algorithms and proofs<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/16_String\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>String<\/strong><\/a> \u2014 String manipulation algorithms<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/17_Bit-Manipulation\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Bit Manipulation<\/strong><\/a> \u2014 Bitwise operations and tricks<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/18_Array\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Array<\/strong><\/a> \u2014 Array problem patterns<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/19_HashMap\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>HashMap<\/strong><\/a> \u2014 Hash table implementations<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/20_DFS-BFS\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>DFS BFS<\/strong><\/a> \u2014 Tree and graph traversals<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/21_Two-Pointers\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Two Pointers<\/strong><\/a> \u2014 Sliding window and two-pointer techniques<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/22_Math\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Math<\/strong><\/a> \u2014 Number theory and mathematical algorithms<\/li>\n<li><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/23_Recursion\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Recursion<\/strong><\/a> \u2014 Recursive problem-solving patterns<\/li>\n<\/ol>\n<hr>\n<h2 id=\"special-features\">Special Features and Resources<\/h2>\n<h3 class=\"heading-element\" id=\"dynamic-programming-mastery\"><span>Dynamic Programming Mastery<\/span>\n <a href=\"#dynamic-programming-mastery\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Comprehensive DP Learning Path<\/strong><\/p>\n<p>The <a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/02_Dynamic-Programming\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Dynamic Programming<\/a> section includes all problems from <a href=\"https:\/\/www.youtube.com\/c\/AdityaVermaTheProgrammingLord\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Aditya Verma&rsquo;s<\/a> renowned <a href=\"https:\/\/youtube.com\/playlist?list=PL_z_8CaSLPWekqhdCPmFohncHwz8TY2Go\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">DP playlist<\/a> organized by patterns, along with comprehensive <a href=\"https:\/\/raw.githubusercontent.com\/SamirPaulb\/DSAlgo\/main\/02_Dynamic-Programming\/Dynamic-Programming-NOTE.pdf\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>handwritten notes<\/strong><\/a> for quick revision.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"strivers-sde-sheet-solutions\"><span>Striver&rsquo;s SDE Sheet Solutions<\/span>\n <a href=\"#strivers-sde-sheet-solutions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The <a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/30-Days-SDE-Sheet-Practice\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>30-Days-SDE-Sheet-Practice<\/strong><\/a> directory is perfect for those preparing from <a href=\"https:\/\/takeuforward.org\/interviews\/strivers-sde-sheet-top-coding-interview-problems\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Striver&rsquo;s SDE Sheet<\/a>. It contains:<\/p>\n<ul>\n<li>Day-wise organized solutions<\/li>\n<li>Concise approach notes for each problem<\/li>\n<li>Optimal solutions with complexity analysis<\/li>\n<li>Quick revision-friendly format<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"company-specific-questions\"><span>Company-Specific Questions<\/span>\n <a href=\"#company-specific-questions\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The <a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/Questions-Sheet\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>Questions-Sheet<\/strong><\/a> directory features curated problem lists from:<\/p>\n<ul>\n<li><strong>FAANG companies<\/strong> (Facebook\/Meta, Amazon, Apple, Netflix, Google)<\/li>\n<li><strong>Top product companies<\/strong> (Microsoft, Adobe, Uber, Twitter)<\/li>\n<li><strong>Unicorn startups<\/strong> and other tech giants<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"computer-science-fundamentals\"><span>Computer Science Fundamentals<\/span>\n <a href=\"#computer-science-fundamentals\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The <a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\/tree\/main\/BOOKS-and-PDFs\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>BOOKS-and-PDFs<\/strong><\/a> directory contains essential resources:<\/p>\n<ul>\n<li>Algorithm textbooks and references<\/li>\n<li>System design guides<\/li>\n<li>Operating systems fundamentals<\/li>\n<li>Database concepts<\/li>\n<li>Competitive programming handbooks<\/li>\n<\/ul>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/repository-image-dsalgo-samirpaul1-leetcode-post.jpeg\" title=\"DSAlgo Repository \u2014 Complete DSA resource for technical interviews featuring 3000&#43; LeetCode solutions, dynamic programming patterns, and company-specific questions\" data-thumbnail=\"\/files\/repository-image-dsalgo-samirpaul1-leetcode-post.jpeg\" data-sub-html=\"<h2>Data Structures and Algorithms repository overview \u2014 comprehensive coding interview preparation resource with Python solutions<\/h2><p>DSAlgo Repository \u2014 Complete DSA resource for technical interviews featuring 3000&#43; LeetCode solutions, dynamic programming patterns, and company-specific questions<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/repository-image-dsalgo-samirpaul1-leetcode-post.jpeg' alt=\"Data Structures and Algorithms repository overview \u2014 comprehensive coding interview preparation resource with Python solutions\"><\/a><figcaption class=\"image-caption\">DSAlgo Repository \u2014 Complete DSA resource for technical interviews featuring 3000+ LeetCode solutions, dynamic programming patterns, and company-specific questions<\/figcaption>\n <\/figure><\/p>\n<hr>\n<h2 id=\"online-vscode\">Browse Code in Online VS Code<\/h2>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Interactive Code Exploration<\/strong><\/p>\n<p>You can browse and search through all solutions directly in your browser using an online VS Code environment. No need to clone the repository!<\/p>\n<\/div>\n <\/div>\n<\/div>\n<p><strong>Access the repository online:<\/strong> <a href=\"https:\/\/samirpaulb.github.io\/DSAlgo\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">samirpaulb.github.io\/DSAlgo<\/a><\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/samirpaulb.github.io-DSAlgo-demo.png\" title=\"DSAlgo Online VS Code Interface \u2014 Browse and search through data structures and algorithms solutions with syntax highlighting and file navigation\" data-thumbnail=\"\/files\/samirpaulb.github.io-DSAlgo-demo.png\" data-sub-html=\"<h2>Online VS Code interface for DSA repository \u2014 browse 3000&#43; algorithm solutions directly in browser<\/h2><p>DSAlgo Online VS Code Interface \u2014 Browse and search through data structures and algorithms solutions with syntax highlighting and file navigation<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/samirpaulb.github.io-DSAlgo-demo.png' alt=\"Online VS Code interface for DSA repository \u2014 browse 3000&#43; algorithm solutions directly in browser\"><\/a><figcaption class=\"image-caption\">DSAlgo Online VS Code Interface \u2014 Browse and search through data structures and algorithms solutions with syntax highlighting and file navigation<\/figcaption>\n <\/figure><\/p>\n<p><strong>Features of Online VS Code:<\/strong><\/p>\n<ul>\n<li>Full syntax highlighting for Python code<\/li>\n<li>File tree navigation across all topics<\/li>\n<li>Built-in search functionality<\/li>\n<li>No installation required<\/li>\n<li>Works on any device with a browser<\/li>\n<\/ul>\n<hr>\n<h2 id=\"contribution\">How to Contribute<\/h2>\n<div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Open Source Contributions Welcome<\/strong><\/p>\n<p>This repository thrives on community contributions! Whether you&rsquo;re a beginner or an experienced developer, your contributions help make this resource better for everyone preparing for technical interviews.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<p>I continuously improve this repository by adding new problems and optimizing existing solutions. Your contributions are highly valued!<\/p>\n<p><strong>Ways to Contribute:<\/strong><\/p>\n<pre><code># 1. Optimize Solutions\n# - Improve time\/space complexity of existing solutions\n# - Add alternative approaches to problems\n# - Enhance code readability and documentation\n\n# 2. Add New Content\n# - Submit solutions to new LeetCode problems\n# - Add problems from recent contests\n# - Include detailed explanations and complexity analysis\n\n# 3. Expand Resources\n# - Contribute to BOOKS-and-PDFs directory\n# - Add company-specific question lists\n# - Share interview experiences and tips\n\n# 4. Quality Improvements\n# - Fix bugs in existing solutions\n# - Add test cases and edge cases\n# - Improve documentation and comments\n\n# 5. Community Support\n# - Resolve issues raised by others\n# - Answer questions in discussions\n# - Review pull requests<\/code><\/pre><p><strong>Contribution Guidelines:<\/strong><\/p>\n<ol>\n<li>All solutions should be in <strong>Python 3<\/strong><\/li>\n<li>Include time and space complexity analysis<\/li>\n<li>Add clear comments explaining the approach<\/li>\n<li>Follow consistent code formatting<\/li>\n<li>Test solutions thoroughly before submitting<\/li>\n<\/ol>\n<p><strong>Getting Started with Contributing:<\/strong><\/p>\n<pre><code># Fork the repository on GitHub\n# Then clone your fork\ngit clone https:\/\/github.com\/YOUR_USERNAME\/DSAlgo.git\ncd DSAlgo\n\n# Create a new branch for your contribution\ngit checkout -b add-new-solution\n\n# Make your changes\n# Add and commit\ngit add .\ngit commit -m &#34;Add solution for [Problem Name]&#34;\n\n# Push to your fork\ngit push origin add-new-solution\n\n# Create a Pull Request on GitHub<\/code><\/pre><hr>\n<h2 id=\"growth-metrics\">Repository Growth<\/h2>\n<h3 class=\"heading-element\" id=\"stargazers-over-time\"><span>Stargazers Over Time<\/span>\n <a href=\"#stargazers-over-time\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\"><img alt=\"Star History Chart showing growth of DSAlgo repository \u2014 Data Structures and Algorithms for Coding Interview\" loading=\"lazy\" src=\"https:\/\/api.star-history.com\/svg?repos=SamirPaulb\/DSAlgo&type=Date\" title=\"DSAlgo Repository Star History \u2014 Growing community of developers preparing for technical interviews\"><\/a><\/p>\n<hr>\n<h2 id=\"learning-path\">Recommended Learning Path<\/h2>\n<div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Structured Interview Preparation<\/strong><\/p>\n<p>Follow this path to maximize your interview preparation efficiency:<\/p>\n<\/div>\n <\/div>\n<\/div>\n<p><strong>Phase 1: Fundamentals (Weeks 1-4)<\/strong><\/p>\n<ol>\n<li>Arrays and Strings<\/li>\n<li>LinkedLists<\/li>\n<li>Stacks and Queues<\/li>\n<li>Recursion basics<\/li>\n<\/ol>\n<p><strong>Phase 2: Core Data Structures (Weeks 5-8)<\/strong><\/p>\n<ol>\n<li>Binary Trees and BST<\/li>\n<li>Heaps and Priority Queues<\/li>\n<li>HashMap and HashSet<\/li>\n<li>Graphs (BFS\/DFS)<\/li>\n<\/ol>\n<p><strong>Phase 3: Advanced Algorithms (Weeks 9-12)<\/strong><\/p>\n<ol>\n<li>Dynamic Programming patterns<\/li>\n<li>Backtracking<\/li>\n<li>Greedy algorithms<\/li>\n<li>Advanced graph algorithms<\/li>\n<\/ol>\n<p><strong>Phase 4: Interview Preparation (Weeks 13-16)<\/strong><\/p>\n<ol>\n<li>Striver&rsquo;s SDE Sheet (30 days)<\/li>\n<li>Company-specific questions<\/li>\n<li>Mock interviews<\/li>\n<li>System design basics<\/li>\n<\/ol>\n<hr>\n<h2 id=\"conclusion\">Start Your Journey Today<\/h2>\n<p>Whether you&rsquo;re preparing for your first technical interview or aiming for a FAANG role, this repository provides all the resources you need for success.<\/p>\n<p><strong>Key Takeaways:<\/strong><\/p>\n<ul>\n<li>3000+ curated coding problems with optimal solutions<\/li>\n<li>Organized by topics and difficulty<\/li>\n<li>Company-specific preparation material<\/li>\n<li>Active community and regular updates<\/li>\n<li>Completely free and open-source<\/li>\n<\/ul>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>Pro Tip for Interview Success<\/strong><\/p>\n<p>Don&rsquo;t just solve problems \u2014 understand the patterns. Focus on recognizing problem types and applying standard techniques. This repository organizes problems by patterns to help you develop this crucial skill.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<p><strong>Ready to start?<\/strong> Visit the <a href=\"https:\/\/github.com\/SamirPaulb\/DSAlgo\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">repository<\/a> and begin your coding interview preparation journey today!<\/p>\n"},{"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","link":"https:\/\/samir.pages.dev\/posts\/pomodoro\/","pubDate":"Thu, 20 Jul 2023 00:00:00 +0000","author":"netproxy@duck.com (Samir)","guid":"https:\/\/samir.pages.dev\/posts\/pomodoro\/","category":"Projects","description":"<img src=\"https:\/\/samir.pages.dev\/files\/pomodoro-timer.webp\" alt=\"featured image\" referrerpolicy=\"no-referrer\"><p><strong>Stay focused. Study better. No distractions.<\/strong><\/p>\n<h2 id=\"introduction\">What is This?<\/h2>\n<p>A free, single-page Pomodoro Timer packed with everything you need for deep focus sessions \u2014 built-in music, live radio, screen wake lock, and smart alarms. No account, no ads, no tracking. Just open and start working.<\/p>\n<div class=\"details admonition tip open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-regular fa-lightbulb\" aria-hidden=\"true\"><\/i>Tip<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Try it now<\/strong> \u2014 the full timer is embedded below, or open it directly at <a href=\"https:\/\/samir.pages.dev\/pomodoro.html\">\/pomodoro<\/a><\/div>\n <\/div>\n<\/div>\n<hr>\n<h2 id=\"live-demo\">Live Demo<\/h2>\n<iframe src=\"https:\/\/samir.pages.dev\/pomodoro.html\"\n title=\"Pomodoro Timer \u2014 Focus Timer with Lo-fi Music and Radio\"\n frameborder=\"0\"\n loading=\"lazy\"\n width=\"100%\"\n height=\"750\"\n scrolling=\"yes\"\n style=\"border-radius:8px;border:1px solid rgba(128,128,128,0.2)\">\n<\/iframe>\n<hr>\n<h2 id=\"features\">Features<\/h2>\n<h3 class=\"heading-element\" id=\"music--radio-built-in\"><span>Music &amp; Radio (Built-in)<\/span>\n <a href=\"#music--radio-built-in\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Source<\/th>\n <th>What You Get<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Lofi Auto Mix<\/strong><\/td>\n <td>2000+ lo-fi songs from YouTube, auto-shuffled<\/td>\n <\/tr>\n <tr>\n <td><strong>SomaFM<\/strong><\/td>\n <td>Groove Salad (ambient\/chill electronic)<\/td>\n <\/tr>\n <tr>\n <td><strong>Bloomberg Radio<\/strong><\/td>\n <td>WBBR 1130 NY, Business\/Markets<\/td>\n <\/tr>\n <tr>\n <td><strong>All India Radio<\/strong><\/td>\n <td>Vividh Bharati, Raagam, FM Gold, FM Rainbow, AIR News<\/td>\n <\/tr>\n <tr>\n <td><strong>BBC Radio<\/strong><\/td>\n <td>Radio 1, 2, 3, 4, World Service, Asian Network<\/td>\n <\/tr>\n <tr>\n <td><strong>Custom URL<\/strong><\/td>\n <td>Paste any YouTube or SoundCloud link<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><h3 class=\"heading-element\" id=\"timer--productivity\"><span>Timer &amp; Productivity<\/span>\n <a href=\"#timer--productivity\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ul>\n<li><strong>Preset durations<\/strong>: 5, 15, 30, 45, 60, 90, 120 minutes<\/li>\n<li><strong>Custom time<\/strong>: Set any duration you want<\/li>\n<li><strong>Session counter<\/strong>: Tracks completed pomodoros at the top<\/li>\n<li><strong>Browser notifications<\/strong>: Get notified even when the tab is in the background<\/li>\n<li><strong>Screen wake lock<\/strong>: Keeps your display on during sessions (uses NoSleep.js)<\/li>\n<li><strong>Separate wake lock timer<\/strong>: 1-24 hours independent of pomodoro<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"alarms\"><span>Alarms<\/span>\n <a href=\"#alarms\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ul>\n<li><strong>Beep 15s<\/strong> \u2014 Short repeating beep, auto-dismisses<\/li>\n<li><strong>Beep 30s<\/strong> \u2014 Increasing frequency beep (default)<\/li>\n<li><strong>&ldquo;The Time Is Now&rdquo;<\/strong> \u2014 John Cena&rsquo;s entrance theme (88 seconds)<\/li>\n<li><strong>Volume amplification<\/strong> \u2014 1x to 7x via Web Audio API<\/li>\n<li><strong>Smart dismiss<\/strong> \u2014 Any keypress, click, tap, or tab switch dismisses the alarm<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"keyboard-shortcuts\"><span>Keyboard Shortcuts<\/span>\n <a href=\"#keyboard-shortcuts\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Key<\/th>\n <th>Action<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><code>Space<\/code><\/td>\n <td>Start \/ Pause<\/td>\n <\/tr>\n <tr>\n <td><code>R<\/code><\/td>\n <td>Reset timer<\/td>\n <\/tr>\n <tr>\n <td>Any key<\/td>\n <td>Dismiss alarm<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><hr>\n<h2 id=\"why\">Why I Built This<\/h2>\n<p>Every Pomodoro timer I found online had one or more of these problems:<\/p>\n<ol>\n<li><strong>Requires signup<\/strong> \u2014 just to set a 25-minute timer<\/li>\n<li><strong>Loaded with ads<\/strong> \u2014 tracking pixels, cookie banners, pop-ups<\/li>\n<li><strong>No music integration<\/strong> \u2014 you need a separate tab for lo-fi beats<\/li>\n<li><strong>Screen turns off<\/strong> \u2014 you lose sight of the countdown during long sessions<\/li>\n<li><strong>Ugly UI<\/strong> \u2014 harsh white themes that hurt eyes during late-night study<\/li>\n<\/ol>\n<p>This timer solves all of them in a single HTML file. No build tools, no framework, no backend. Just open the page and focus.<\/p>\n<hr>\n<h2 id=\"how-it-works\">How It Works<\/h2>\n<h3 class=\"heading-element\" id=\"the-pomodoro-technique\"><span>The Pomodoro Technique<\/span>\n <a href=\"#the-pomodoro-technique\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Developed by Francesco Cirillo in the late 1980s:<\/p>\n<ol>\n<li><strong>Choose a task<\/strong> to work on<\/li>\n<li><strong>Set the timer<\/strong> (traditionally 25 minutes)<\/li>\n<li><strong>Work with full focus<\/strong> until the alarm rings<\/li>\n<li><strong>Take a short break<\/strong> (5 minutes)<\/li>\n<li><strong>Repeat<\/strong> \u2014 after 4 pomodoros, take a longer 15-30 minute break<\/li>\n<\/ol>\n<div class=\"details admonition info open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-circle-info\" aria-hidden=\"true\"><\/i>Info<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><strong>Research-backed intervals<\/strong>: 25-30 minutes is optimal for most people. For deep work like coding or writing, try 45-60 minutes. Starting out? Try 15 minutes and build up.<\/div>\n <\/div>\n<\/div>\n<h3 class=\"heading-element\" id=\"technical-architecture\"><span>Technical Architecture<\/span>\n <a href=\"#technical-architecture\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The entire app is a <strong>single HTML file<\/strong> (3100+ lines) with zero external dependencies:<\/p>\n<ul>\n<li><strong>Timer logic<\/strong>: Pure JavaScript <code>setInterval<\/code> with drift correction<\/li>\n<li><strong>Music<\/strong>: YouTube IFrame API + HLS.js for radio streams<\/li>\n<li><strong>Wake lock<\/strong>: NoSleep.js (simulates video playback to keep screen on)<\/li>\n<li><strong>Alarms<\/strong>: Web Audio API with base64-embedded audio (works offline)<\/li>\n<li><strong>Storage<\/strong>: <code>localStorage<\/code> only \u2014 nothing leaves your device<\/li>\n<li><strong>Radio fallback<\/strong>: Self-healing streams via Radio Browser API with cached URLs<\/li>\n<\/ul>\n<hr>\n<h2 id=\"embed\">Embed on Your Website<\/h2>\n<p>Want to add this timer to your own site? Just copy this iframe:<\/p>\n<pre><code>&lt;iframe src=&#34;https:\/\/samirpaulb.github.io\/pomodoro.html&#34;\n title=&#34;Pomodoro Timer&#34;\n frameborder=&#34;0&#34;\n loading=&#34;lazy&#34;\n width=&#34;100%&#34;\n height=&#34;750&#34;\n scrolling=&#34;yes&#34;&gt;\n&lt;\/iframe&gt;<\/code><\/pre><h3 class=\"heading-element\" id=\"compact-version-timer-only\"><span>Compact Version (Timer Only)<\/span>\n <a href=\"#compact-version-timer-only\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><pre><code>&lt;div style=&#34;max-width: 500px; margin: 0 auto;&#34;&gt;\n &lt;iframe src=&#34;https:\/\/samirpaulb.github.io\/pomodoro.html&#34;\n title=&#34;Pomodoro Timer&#34;\n frameborder=&#34;0&#34;\n loading=&#34;lazy&#34;\n width=&#34;100%&#34;\n height=&#34;600&#34;\n scrolling=&#34;yes&#34;\n style=&#34;border-radius: 12px; border: 1px solid rgba(128,128,128,0.2);&#34;&gt;\n &lt;\/iframe&gt;\n&lt;\/div&gt;<\/code><\/pre><hr>\n<h2 id=\"privacy\">Privacy & Performance<\/h2>\n<div class=\"table-wrapper\">\n <table data-sortable><thead>\n <tr>\n <th>Aspect<\/th>\n <th>Detail<\/th>\n <\/tr>\n <\/thead>\n <tbody>\n <tr>\n <td><strong>Tracking<\/strong><\/td>\n <td>Zero. No analytics, no cookies, no fingerprinting<\/td>\n <\/tr>\n <tr>\n <td><strong>Data storage<\/strong><\/td>\n <td><code>localStorage<\/code> only (stays on your device)<\/td>\n <\/tr>\n <tr>\n <td><strong>Ads<\/strong><\/td>\n <td>None. Ever.<\/td>\n <\/tr>\n <tr>\n <td><strong>Account<\/strong><\/td>\n <td>Not needed<\/td>\n <\/tr>\n <tr>\n <td><strong>Load time<\/strong><\/td>\n <td>&lt; 1 second (single file, no external CSS\/JS)<\/td>\n <\/tr>\n <tr>\n <td><strong>Offline<\/strong><\/td>\n <td>Timer + alarms work offline; music needs internet<\/td>\n <\/tr>\n <tr>\n <td><strong>GDPR\/CCPA<\/strong><\/td>\n <td>Fully compliant (no data collected)<\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/div><hr>\n<h2 id=\"compatibility\">Browser & Device Support<\/h2>\n<pre><code>\u2705 Chrome \/ Edge (Chromium) \u2014 v90&#43;\n\u2705 Firefox \u2014 v88&#43;\n\u2705 Safari \u2014 v14&#43;\n\u2705 Opera \u2014 v76&#43;\n\u2705 iOS Safari (iPhone \/ iPad)\n\u2705 Chrome Mobile (Android)\n\u2705 Samsung Internet<\/code><\/pre><p>The timer is fully responsive \u2014 works on desktop monitors, laptops, tablets, and phones. Touch-friendly controls, power button alarm dismiss via Media Session API.<\/p>\n<hr>\n<h2 id=\"use-cases\">Perfect For<\/h2>\n<ul>\n<li><strong>Students<\/strong>: Exam prep, homework, thesis writing<\/li>\n<li><strong>Developers<\/strong>: Deep coding sessions, debugging marathons<\/li>\n<li><strong>Writers<\/strong>: Drafting, editing, research blocks<\/li>\n<li><strong>Remote workers<\/strong>: Timeboxing tasks, meeting prep<\/li>\n<li><strong>Anyone with ADHD<\/strong>: External structure for focus sessions<\/li>\n<\/ul>\n<hr>\n<h2 id=\"start\">Start Using It<\/h2>\n<div class=\"details admonition note open\">\n <div class=\"details-summary admonition-title\"><i class=\"icon fa-solid fa-pencil\" aria-hidden=\"true\"><\/i>Note<i class=\"details-icon fa-solid fa-angle-right\" aria-hidden=\"true\"><\/i><\/div><div class=\"details-content\">\n <div class=\"admonition-content\"><p><strong>No installation needed<\/strong><\/p>\n<p>Open <a href=\"https:\/\/samir.pages.dev\/pomodoro.html\">\/pomodoro<\/a> in any browser. Bookmark it. That&rsquo;s it.<\/p>\n<\/div>\n <\/div>\n<\/div>\n<ol>\n<li>Pick a duration (or use the default 30 minutes)<\/li>\n<li>Choose music: Auto Mix, radio, or paste your own URL<\/li>\n<li>Enable wake lock if you want the screen to stay on<\/li>\n<li>Press <strong>Start<\/strong> (or hit <code>Space<\/code>)<\/li>\n<li>Focus until the alarm rings<\/li>\n<li>Take a break. Repeat.<\/li>\n<\/ol>\n"},{"title":"PyShooter \u2014 2D Side-Scrolling Shooter Game in Python","link":"https:\/\/samir.pages.dev\/posts\/pyshooter\/","pubDate":"Thu, 27 Apr 2023 00:00:00 +0000","author":"netproxy@duck.com (Samir)","guid":"https:\/\/samir.pages.dev\/posts\/pyshooter\/","category":"Projects","description":"<img src=\"https:\/\/samir.pages.dev\/files\/pyshooter.webp\" alt=\"featured image\" referrerpolicy=\"no-referrer\"><p><a href=\"https:\/\/www.python.org\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/images\/remote\/030fb10d-3def-4a4c-8beb-2eb9b111c1fb_8065975953974626978.svg' alt=\"Language\"><\/a>\n<a href=\"https:\/\/www.pygame.org\/news.html\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/images\/remote\/module-pygame-brightgreen_15485988230183252998.svg' alt=\"Module\"><\/a><\/p>\n<p>PyShooter is a two-dimensional shooter game in which the player runs horizontally and fires at enemies.<\/p>\n<h5 class=\"heading-element\" id=\"repository-httpsgithubcomsamirpaulbpyshooter\"><span>Repository: <a href=\"https:\/\/github.com\/SamirPaulb\/PyShooter\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">https:\/\/github.com\/SamirPaulb\/PyShooter<\/a><\/span>\n <a href=\"#repository-httpsgithubcomsamirpaulbpyshooter\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h5><h2 class=\"heading-element\" id=\"demo\"><span>Demo:<\/span>\n <a href=\"#demo\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><iframe width=\"100%\" height=\"420\" src=\"https:\/\/www.youtube.com\/embed\/rg-49OTHZVY\" loading=\"lazy\" title=\"PyShooter - Samir - YouTube\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen=\"true\"><\/iframe>\n<div>\n<video width=\"100%\" id=\"video\" controls>\n<source loading=\"lazy\" src=\"https:\/\/user-images.githubusercontent.com\/77569653\/214920668-ef657089-b71f-43c8-8011-a202a09f5531.mp4\" type=\"video\/mp4\">\nYour browser does not support the video tag.\n<\/video>\n<\/div>\n<p>Vimeo: <a href=\"https:\/\/vimeo.com\/809728956\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">https:\/\/vimeo.com\/809728956<\/a><\/p>\n<!---\n<a href=\"https:\/\/www.youtube.com\/watch?v=rg-49OTHZVY\">\n<img src=\"https:\/\/raw.githubusercontent.com\/SamirPaul1\/assets\/main\/pygame-youtube-video-thumbnail.jpg\" alt=\"PyShooter Demo YouTube\" width=\"700\" height=\"370\">\n<\/a>\n-->\n<h2 class=\"heading-element\" id=\"how-to-play-pyshooter\"><span>How To Play PyShooter<\/span>\n <a href=\"#how-to-play-pyshooter\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"option-1-play-without-installation-standalone-executable\"><span>Option 1: Play Without Installation (Standalone Executable)<\/span>\n <a href=\"#option-1-play-without-installation-standalone-executable\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>For users without Python or Pygame installed, a standalone executable is available:<\/p>\n<ol>\n<li><strong>Download<\/strong> the latest release from the repository<\/li>\n<li><strong>Extract<\/strong> the game files to a folder<\/li>\n<li><strong>Double-click<\/strong> the <code>PyShooter.exe<\/code> file to launch the game<\/li>\n<\/ol>\n<blockquote>\n<p><strong>Important<\/strong>: Keep the executable in the same directory as the <code>sounds\/<\/code>, <code>images\/<\/code>, and <code>fonts\/<\/code> folders for proper asset loading.<\/p>\n<\/blockquote>\n<h3 class=\"heading-element\" id=\"option-2-run-from-source-code\"><span>Option 2: Run From Source Code<\/span>\n <a href=\"#option-2-run-from-source-code\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>For developers or users with Python installed:<\/p>\n<p><strong>Prerequisites:<\/strong><\/p>\n<ul>\n<li>Python 3.7 or higher<\/li>\n<li>Pygame library<\/li>\n<\/ul>\n<p><strong>Installation Steps:<\/strong><\/p>\n<pre><code># Install Pygame using pip\npip install pygame\n\n# Clone or download the repository\ngit clone https:\/\/github.com\/SamirPaulb\/PyShooter.git\ncd PyShooter\n\n# Run the game\npython main.py<\/code><\/pre><h3 class=\"heading-element\" id=\"game-controls\"><span>Game Controls<\/span>\n <a href=\"#game-controls\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ul>\n<li><strong>Arrow Keys<\/strong>: Move left\/right, jump<\/li>\n<li><strong>Space<\/strong>: Shoot weapon<\/li>\n<li><strong>ESC<\/strong>: Pause game<\/li>\n<li><strong>R<\/strong>: Restart level<\/li>\n<\/ul>\n<hr>\n<p>Credits for assets used:<\/p>\n<ol>\n<li><a href=\"https:\/\/erayzesen.itch.io\/pixel-platformer\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">https:\/\/erayzesen.itch.io\/pixel-platformer<\/a><\/li>\n<li><a href=\"https:\/\/secrethideout.itch.io\/team-wars-platformer-battle\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">https:\/\/secrethideout.itch.io\/team-wars-platformer-battle<\/a><\/li>\n<li><a href=\"https:\/\/soundimage.org\/fantasywonder\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">https:\/\/soundimage.org\/fantasywonder<\/a><\/li>\n<li><a href=\"https:\/\/gushh.net\/blog\/free-game-sprites-explosion-3\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">https:\/\/gushh.net\/blog\/free-game-sprites-explosion-3<\/a><\/li>\n<li><a href=\"https:\/\/mtk.itch.io\/grenades-16x16\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">https:\/\/mtk.itch.io\/grenades-16x16<\/a><\/li>\n<li><a href=\"https:\/\/www.youtube.com\/watch?v=DHgj5jhMJKg\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">PyGame Beginner Tutorial by youtube.com\/@CodingWithRuss<\/a><\/li>\n<\/ol>\n"},{"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","link":"https:\/\/samir.pages.dev\/posts\/text-file-compressor-de-compressor-web-app\/","pubDate":"Sun, 12 Mar 2023 00:00:00 +0000","author":"netproxy@duck.com (Samir)","guid":"https:\/\/samir.pages.dev\/posts\/text-file-compressor-de-compressor-web-app\/","category":"Projects","description":"<img src=\"https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app.webp\" alt=\"featured image\" referrerpolicy=\"no-referrer\"><ul>\n<li>This webapp uses Huffman Coding for Text Compression and De-compression.<\/li>\n<li>Made with <code>JavaScript<\/code>, <code>HTML5<\/code> and <code>CSS3<\/code>.<\/li>\n<li>Live Demo: <a href=\"https:\/\/samirpaulb.github.io\/txt-compressor\/\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><strong>samirpaulb.github.io\/txt-compressor<\/strong><\/a><\/li>\n<li>Repository: <a href=\"https:\/\/github.com\/SamirPaulb\/txt-compressor\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">github.com\/SamirPaulb\/txt-compressor<\/a><\/li>\n<\/ul>\n<h2 class=\"heading-element\" id=\"about-this-application\"><span>About this application:<\/span>\n <a href=\"#about-this-application\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><ul>\n<li>An online text(.txt) file compressor, decompressor which uses Huffman Algorithm to encode\/compress files by 35% and decode them back to the original size.<\/li>\n<li>This tool assigns a variable-length code to the characters of the uploaded file based on the frequency of occurrence. Then converts characters to that special code which takes less size than the original ASCII codes. Huffman code forms a binary tree assigning the most frequent characters with the smallest codes and longer codes for the least frequent characters.<\/li>\n<li>A Huffman code is a tree, built bottom up, starting with the list of different characters appearing in a text and their frequency.<\/li>\n<li>With this lossless data compression method, this tool can compress the file size by 35 to 40%.<\/li>\n<li>As file size gets reduced and original characters get changed to special characters so this encoding also improves security by encrypting the file during file sharing.<\/li>\n<li>With the decoding feature, the user can decode the encoded file and get back the original file of the previous size.<\/li>\n<li>I have used JavaScript to implement the algorithms so that browser can compile the code and HTML, CSS to make the website responsive.<\/li>\n<li>Additional instructions and warnings are provided if steps are not followed correctly.<\/li>\n<li>An <a href=\"https:\/\/samirpaulb.github.io\/txt-compressor\/info.html\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">Info page<\/a> is added to give more information about tecnique of <strong>Lossless Data Compression<\/strong> with Huffman coding.<\/li>\n<\/ul>\n<h2 class=\"heading-element\" id=\"video-demo\"><span>Video Demo:<\/span>\n <a href=\"#video-demo\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><div>\n<video width=\"100%\" id=\"video\" controls>\n<source loading=\"lazy\" src=\"https:\/\/user-images.githubusercontent.com\/77569653\/172716965-50560f4a-2acf-4013-ae87-8b474b2a09e3.mp4\" type=\"video\/mp4\">\nYour browser does not support the video tag.\n<\/video>\n<\/div>\n<h2 class=\"heading-element\" id=\"application-screenshots\"><span>Application Screenshots<\/span>\n <a href=\"#application-screenshots\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"landing-page\"><span>Landing Page<\/span>\n <a href=\"#landing-page\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app-landing-page.png\" title=\"Huffman Coding Text Compressor - Main Interface\" data-thumbnail=\"\/files\/text-file-compressor-de-compressor-web-app-landing-page.png\" data-sub-html=\"<h2>Text File Compressor Landing Page - Clean web interface with file upload button, Huffman coding algorithm explanation, compression\/decompression toggle options, and responsive design for optimal user experience on desktop and mobile devices<\/h2><p>Huffman Coding Text Compressor - Main Interface<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app-landing-page.png' alt=\"Text File Compressor Landing Page - Clean web interface with file upload button, Huffman coding algorithm explanation, compression\/decompression toggle options, and responsive design for optimal user experience on desktop and mobile devices\"><\/a><figcaption class=\"image-caption\">Huffman Coding Text Compressor - Main Interface<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"step-1-upload-text-file\"><span>Step 1: Upload Text File<\/span>\n <a href=\"#step-1-upload-text-file\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app-step1.png\" title=\"Step 1 - Upload Your Text File for Compression\" data-thumbnail=\"\/files\/text-file-compressor-de-compressor-web-app-step1.png\" data-sub-html=\"<h2>Upload Text File Interface - File selection dialog showing supported .txt format with drag-and-drop functionality, file size validation, and clear upload instructions for Huffman compression algorithm processing<\/h2><p>Step 1 - Upload Your Text File for Compression<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app-step1.png' alt=\"Upload Text File Interface - File selection dialog showing supported .txt format with drag-and-drop functionality, file size validation, and clear upload instructions for Huffman compression algorithm processing\"><\/a><figcaption class=\"image-caption\">Step 1 - Upload Your Text File for Compression<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"step-2-select-action-compress--decompress\"><span>Step 2: Select Action (Compress \/ Decompress)<\/span>\n <a href=\"#step-2-select-action-compress--decompress\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app-step2.png\" title=\"Step 2 - Choose Compression or Decompression Action\" data-thumbnail=\"\/files\/text-file-compressor-de-compressor-web-app-step2.png\" data-sub-html=\"<h2>Action Selection Interface - Toggle buttons for choosing between Huffman compression to reduce file size by 35-40% or decompression to restore original file format with real-time processing status indicators<\/h2><p>Step 2 - Choose Compression or Decompression Action<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app-step2.png' alt=\"Action Selection Interface - Toggle buttons for choosing between Huffman compression to reduce file size by 35-40% or decompression to restore original file format with real-time processing status indicators\"><\/a><figcaption class=\"image-caption\">Step 2 - Choose Compression or Decompression Action<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"step-3-download-processed-file\"><span>Step 3: Download Processed File<\/span>\n <a href=\"#step-3-download-processed-file\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>File gets downloaded automatically when selected process is complete.<\/p>\n<h3 class=\"heading-element\" id=\"compression-result-display\"><span>Compression Result Display<\/span>\n <a href=\"#compression-result-display\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app-step3.png\" title=\"Compression Complete - View Compression Ratio and Download\" data-thumbnail=\"\/files\/text-file-compressor-de-compressor-web-app-step3.png\" data-sub-html=\"<h2>Compression Success - Results showing original file size, compressed file size, and achieved compression ratio percentage (35-40% reduction) with automatic download initialization and file statistics using Huffman encoding algorithm<\/h2><p>Compression Complete - View Compression Ratio and Download<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app-step3.png' alt=\"Compression Success - Results showing original file size, compressed file size, and achieved compression ratio percentage (35-40% reduction) with automatic download initialization and file statistics using Huffman encoding algorithm\"><\/a><figcaption class=\"image-caption\">Compression Complete - View Compression Ratio and Download<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"decompression-result-display\"><span>Decompression Result Display<\/span>\n <a href=\"#decompression-result-display\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app-decompression.png\" title=\"Decompression Complete - Original File Restored\" data-thumbnail=\"\/files\/text-file-compressor-de-compressor-web-app-decompression.png\" data-sub-html=\"<h2>Decompression Success - Interface showing successful decoding of Huffman-encoded file back to original text format with file size restoration confirmation and automatic download of decompressed .txt file<\/h2><p>Decompression Complete - Original File Restored<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app-decompression.png' alt=\"Decompression Success - Interface showing successful decoding of Huffman-encoded file back to original text format with file size restoration confirmation and automatic download of decompressed .txt file\"><\/a><figcaption class=\"image-caption\">Decompression Complete - Original File Restored<\/figcaption>\n <\/figure><\/p>\n<h2 class=\"heading-element\" id=\"error-handling-and-user-guidance\"><span>Error Handling and User Guidance<\/span>\n <a href=\"#error-handling-and-user-guidance\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"no-file-selected-warning\"><span>No File Selected Warning<\/span>\n <a href=\"#no-file-selected-warning\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app-nofile.png\" title=\"Error - Please Upload a Text File First\" data-thumbnail=\"\/files\/text-file-compressor-de-compressor-web-app-nofile.png\" data-sub-html=\"<h2>Error Message - No File Selected - User-friendly alert dialog prompting to upload a valid .txt file before attempting compression or decompression operation with clear call-to-action button<\/h2><p>Error - Please Upload a Text File First<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app-nofile.png' alt=\"Error Message - No File Selected - User-friendly alert dialog prompting to upload a valid .txt file before attempting compression or decompression operation with clear call-to-action button\"><\/a><figcaption class=\"image-caption\">Error - Please Upload a Text File First<\/figcaption>\n <\/figure><\/p>\n<h3 class=\"heading-element\" id=\"file-too-small-warning\"><span>File Too Small Warning<\/span>\n <a href=\"#file-too-small-warning\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app-verysmallfile.png\" title=\"Warning - File Size Too Small for Effective Compression\" data-thumbnail=\"\/files\/text-file-compressor-de-compressor-web-app-verysmallfile.png\" data-sub-html=\"<h2>Warning - File Too Small for Compression - Alert message indicating uploaded file size is below minimum threshold for effective Huffman compression with suggestion to use larger text files for optimal results<\/h2><p>Warning - File Size Too Small for Effective Compression<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app-verysmallfile.png' alt=\"Warning - File Too Small for Compression - Alert message indicating uploaded file size is below minimum threshold for effective Huffman compression with suggestion to use larger text files for optimal results\"><\/a><figcaption class=\"image-caption\">Warning - File Size Too Small for Effective Compression<\/figcaption>\n <\/figure><\/p>\n<h2 class=\"heading-element\" id=\"understanding-huffman-coding\"><span>Understanding Huffman Coding<\/span>\n <a href=\"#understanding-huffman-coding\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"information-page---lossless-data-compression-technique\"><span>Information Page - Lossless Data Compression Technique<\/span>\n <a href=\"#information-page---lossless-data-compression-technique\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app-info.jpeg\" title=\"Learn About Huffman Coding - Lossless Data Compression Explained\" data-thumbnail=\"\/files\/text-file-compressor-de-compressor-web-app-info.jpeg\" data-sub-html=\"<h2>Huffman Coding Algorithm Information Page - Comprehensive educational content explaining lossless data compression technique, binary tree structure, character frequency analysis, variable-length encoding, compression ratio calculation, and step-by-step algorithm visualization with examples for text file optimization<\/h2><p>Learn About Huffman Coding - Lossless Data Compression Explained<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/text-file-compressor-de-compressor-web-app-info.jpeg' alt=\"Huffman Coding Algorithm Information Page - Comprehensive educational content explaining lossless data compression technique, binary tree structure, character frequency analysis, variable-length encoding, compression ratio calculation, and step-by-step algorithm visualization with examples for text file optimization\"><\/a><figcaption class=\"image-caption\">Learn About Huffman Coding - Lossless Data Compression Explained<\/figcaption>\n <\/figure><\/p>\n"},{"title":"Professional Certifications and Online Course Achievements","link":"https:\/\/samir.pages.dev\/posts\/some-of-my-online-courses-certificates\/","pubDate":"Tue, 20 Dec 2022 00:00:00 +0000","author":"netproxy@duck.com (Samir)","guid":"https:\/\/samir.pages.dev\/posts\/some-of-my-online-courses-certificates\/","category":"Certifications","description":"<img src=\"https:\/\/samir.pages.dev\/files\/some-of-my-online-courses-certificates.webp\" alt=\"featured image\" referrerpolicy=\"no-referrer\"><h3 class=\"heading-element\" id=\"certification-categories\"><span>Certification Categories<\/span>\n <a href=\"#certification-categories\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ul>\n<li><strong>Algorithms &amp; Data Structures<\/strong>: Advanced algorithms, complexity analysis, divide and conquer strategies<\/li>\n<li><strong>Machine Learning &amp; AI<\/strong>: Deep learning, neural networks, AI fundamentals, applied ML<\/li>\n<li><strong>Data Science<\/strong>: Data analytics, Python for data science, SQL, data visualization<\/li>\n<li><strong>Software Development<\/strong>: Programming languages (C++, Python), databases, software design<\/li>\n<li><strong>Networking &amp; IoT<\/strong>: Computer networks, Internet of Things, embedded systems<\/li>\n<li><strong>Tools &amp; Frameworks<\/strong>: Git, GitHub, SQL Server, Photoshop, Linux<\/li>\n<\/ul>\n<hr>\n<h2 class=\"heading-element\" id=\"verified-certifications\"><span>Verified Certifications<\/span>\n <a href=\"#verified-certifications\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/GLMMPEBLCU9Q.jpg\" title=\"Coursera: Advanced Algorithms and Complexity - GLMMPEBLCU9Q\" data-thumbnail=\"\/files\/GLMMPEBLCU9Q.jpg\" data-sub-html=\"<h2>Advanced Algorithms and Complexity Coursera GLMMPEBLCU9Q<\/h2><p>Coursera: Advanced Algorithms and Complexity - GLMMPEBLCU9Q<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/GLMMPEBLCU9Q.jpg' alt=\"Advanced Algorithms and Complexity Coursera GLMMPEBLCU9Q\"><\/a><figcaption class=\"image-caption\">Coursera: Advanced Algorithms and Complexity - GLMMPEBLCU9Q<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/ai-fundamentals.jpg\" title=\"DataCamp: AI Fundamentals\" data-thumbnail=\"\/files\/ai-fundamentals.jpg\" data-sub-html=\"<h2>AI Fundamentals DataCamp Certificate<\/h2><p>DataCamp: AI Fundamentals<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/ai-fundamentals.jpg' alt=\"AI Fundamentals DataCamp Certificate\"><\/a><figcaption class=\"image-caption\">DataCamp: AI Fundamentals<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/jetson-nano.jpg\" title=\"NVIDIA: Getting Started with AI on Jetson Nano\" data-thumbnail=\"\/files\/jetson-nano.jpg\" data-sub-html=\"<h2>AI ON JETSON NANO Nvidia<\/h2><p>NVIDIA: Getting Started with AI on Jetson Nano<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/jetson-nano.jpg' alt=\"AI ON JETSON NANO Nvidia\"><\/a><figcaption class=\"image-caption\">NVIDIA: Getting Started with AI on Jetson Nano<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/MA4KKNFEU7MB.jpg\" title=\"Coursera: Algorithmic Toolbox - MA4KKNFEU7MB\" data-thumbnail=\"\/files\/MA4KKNFEU7MB.jpg\" data-sub-html=\"<h2>Algorithmic Toolbox Coursera MA4KKNFEU7MB<\/h2><p>Coursera: Algorithmic Toolbox - MA4KKNFEU7MB<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/MA4KKNFEU7MB.jpg' alt=\"Algorithmic Toolbox Coursera MA4KKNFEU7MB\"><\/a><figcaption class=\"image-caption\">Coursera: Algorithmic Toolbox - MA4KKNFEU7MB<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/UC-c58957c7-3ae3-4f83-b4b1-2f69b2df48fa.jpg\" title=\"Udemy: C&#43;&#43; Programming - UC-c58957c7-3ae3-4f83-b4b1-2f69b2df48fa\" data-thumbnail=\"\/files\/UC-c58957c7-3ae3-4f83-b4b1-2f69b2df48fa.jpg\" data-sub-html=\"<h2>C&#43;&#43; Programming Udemy UC-c58957c7-3ae3-4f83-b4b1-2f69b2df48fa<\/h2><p>Udemy: C&#43;&#43; Programming - UC-c58957c7-3ae3-4f83-b4b1-2f69b2df48fa<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/UC-c58957c7-3ae3-4f83-b4b1-2f69b2df48fa.jpg' alt=\"C&#43;&#43; Programming Udemy UC-c58957c7-3ae3-4f83-b4b1-2f69b2df48fa\"><\/a><figcaption class=\"image-caption\">Udemy: C++ Programming - UC-c58957c7-3ae3-4f83-b4b1-2f69b2df48fa<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/codegoda-2022.jpg\" title=\"Agoda: Codegoda 2022\" data-thumbnail=\"\/files\/codegoda-2022.jpg\" data-sub-html=\"<h2>Codegoda 2022 organised by Agoda<\/h2><p>Agoda: Codegoda 2022<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/codegoda-2022.jpg' alt=\"Codegoda 2022 organised by Agoda\"><\/a><figcaption class=\"image-caption\">Agoda: Codegoda 2022<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/UC-75332da1-4725-4f60-80ab-41f640046e3e.jpg\" title=\"Udemy: Data Analytics Real-World Projects in Python - UC-75332da1-4725-4f60-80ab-41f640046e3e\" data-thumbnail=\"\/files\/UC-75332da1-4725-4f60-80ab-41f640046e3e.jpg\" data-sub-html=\"<h2>Data Analytics Real-World Projects in Python Udemy UC-75332da1-4725-4f60-80ab-41f640046e3e<\/h2><p>Udemy: Data Analytics Real-World Projects in Python - UC-75332da1-4725-4f60-80ab-41f640046e3e<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/UC-75332da1-4725-4f60-80ab-41f640046e3e.jpg' alt=\"Data Analytics Real-World Projects in Python Udemy UC-75332da1-4725-4f60-80ab-41f640046e3e\"><\/a><figcaption class=\"image-caption\">Udemy: Data Analytics Real-World Projects in Python - UC-75332da1-4725-4f60-80ab-41f640046e3e<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/data-science-for-everyone.jpg\" title=\"DateCamp: Data Science for Everyone\" data-thumbnail=\"\/files\/data-science-for-everyone.jpg\" data-sub-html=\"<h2>Data Science for Everyone DateCamp Certificate<\/h2><p>DateCamp: Data Science for Everyone<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/data-science-for-everyone.jpg' alt=\"Data Science for Everyone DateCamp Certificate\"><\/a><figcaption class=\"image-caption\">DateCamp: Data Science for Everyone<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/394N5RVDD2EU.jpg\" title=\"Coursera: Python Data Structures - 394N5RVDD2EU\" data-thumbnail=\"\/files\/394N5RVDD2EU.jpg\" data-sub-html=\"<h2>Data Structures Coursera 394N5RVDD2EU<\/h2><p>Coursera: Python Data Structures - 394N5RVDD2EU<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/394N5RVDD2EU.jpg' alt=\"Data Structures Coursera 394N5RVDD2EU\"><\/a><figcaption class=\"image-caption\">Coursera: Python Data Structures - 394N5RVDD2EU<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/database-foundations-database-management.jpg\" title=\"LinkedIn Learning: Database Foundations Database Management\" data-thumbnail=\"\/files\/database-foundations-database-management.jpg\" data-sub-html=\"<h2>Database Foundations Database Management<\/h2><p>LinkedIn Learning: Database Foundations Database Management<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/database-foundations-database-management.jpg' alt=\"Database Foundations Database Management\"><\/a><figcaption class=\"image-caption\">LinkedIn Learning: Database Foundations Database Management<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/SOE-YDATABASES0001.jpg\" title=\"StanfordOnline &amp; edX: Databases: Relational Databases and SQL - SOE-YDATABASES0001\" data-thumbnail=\"\/files\/SOE-YDATABASES0001.jpg\" data-sub-html=\"<h2>Databases_ Relational Databases and SQL StanfordOnline SOE-YDATABASES0001 Certificate _ edX<\/h2><p>StanfordOnline &amp; edX: Databases: Relational Databases and SQL - SOE-YDATABASES0001<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/SOE-YDATABASES0001.jpg' alt=\"Databases_ Relational Databases and SQL StanfordOnline SOE-YDATABASES0001 Certificate _ edX\"><\/a><figcaption class=\"image-caption\">StanfordOnline & edX: Databases: Relational Databases and SQL - SOE-YDATABASES0001<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/V58E5TWMYGTV.jpg\" title=\"Coursera: Divide and Conquer, Sorting and Searching, and Randomized Algorithms - V58E5TWMYGTV\" data-thumbnail=\"\/files\/V58E5TWMYGTV.jpg\" data-sub-html=\"<h2>Divide and Conquer, Sorting and Searching, and Randomized Algorithms Coursera V58E5TWMYGTV<\/h2><p>Coursera: Divide and Conquer, Sorting and Searching, and Randomized Algorithms - V58E5TWMYGTV<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/V58E5TWMYGTV.jpg' alt=\"Divide and Conquer, Sorting and Searching, and Randomized Algorithms Coursera V58E5TWMYGTV\"><\/a><figcaption class=\"image-caption\">Coursera: Divide and Conquer, Sorting and Searching, and Randomized Algorithms - V58E5TWMYGTV<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/introduction-iot.jpg\" title=\"Cisco Networking Academy: Introduction to IoT and Digital Transformation\" data-thumbnail=\"\/files\/introduction-iot.jpg\" data-sub-html=\"<h2>Introduction to IoT_Cisco -certificate<\/h2><p>Cisco Networking Academy: Introduction to IoT and Digital Transformation<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/introduction-iot.jpg' alt=\"Introduction to IoT_Cisco -certificate\"><\/a><figcaption class=\"image-caption\">Cisco Networking Academy: Introduction to IoT and Digital Transformation<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/2102001215000100.jpg\" title=\"Microsoft Learn Student: Introduction to Opensource, Git and Github - 2102001215000100\" data-thumbnail=\"\/files\/2102001215000100.jpg\" data-sub-html=\"<h2>Introduction to Opensource, Git and Github Microsoft Learn Student 2102001215000100 Github<\/h2><p>Microsoft Learn Student: Introduction to Opensource, Git and Github - 2102001215000100<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/2102001215000100.jpg' alt=\"Introduction to Opensource, Git and Github Microsoft Learn Student 2102001215000100 Github\"><\/a><figcaption class=\"image-caption\">Microsoft Learn Student: Introduction to Opensource, Git and Github - 2102001215000100<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/introduction-to-sql.jpg\" title=\"DataCamp: Introduction to SQL\" data-thumbnail=\"\/files\/introduction-to-sql.jpg\" data-sub-html=\"<h2>Introduction to SQL DataCamp certificate<\/h2><p>DataCamp: Introduction to SQL<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/introduction-to-sql.jpg' alt=\"Introduction to SQL DataCamp certificate\"><\/a><figcaption class=\"image-caption\">DataCamp: Introduction to SQL<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/iot-foundations-operating-systems-fundamentals-2018.jpg\" title=\"LinkedIn Learning: IoT Foundations: Operating Systems Fundamentals (2018)\" data-thumbnail=\"\/files\/iot-foundations-operating-systems-fundamentals-2018.jpg\" data-sub-html=\"<h2>IoT Foundations Operating Systems Fundamentals<\/h2><p>LinkedIn Learning: IoT Foundations: Operating Systems Fundamentals (2018)<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/iot-foundations-operating-systems-fundamentals-2018.jpg' alt=\"IoT Foundations Operating Systems Fundamentals\"><\/a><figcaption class=\"image-caption\">LinkedIn Learning: IoT Foundations: Operating Systems Fundamentals (2018)<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/linux-shells-and-processes-14269702.jpg\" title=\"LinkedIn learning: Linux Shells and Processes\" data-thumbnail=\"\/files\/linux-shells-and-processes-14269702.jpg\" data-sub-html=\"<h2>Linux Shells and Processes - LinkedIn learning<\/h2><p>LinkedIn learning: Linux Shells and Processes<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/linux-shells-and-processes-14269702.jpg' alt=\"Linux Shells and Processes - LinkedIn learning\"><\/a><figcaption class=\"image-caption\">LinkedIn learning: Linux Shells and Processes<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/VJHQ4C2F8LDA.jpg\" title=\"Coursera: Machine Learning - VJHQ4C2F8LDA\" data-thumbnail=\"\/files\/VJHQ4C2F8LDA.jpg\" data-sub-html=\"<h2>Machine Learning Coursera VJHQ4C2F8LDA<\/h2><p>Coursera: Machine Learning - VJHQ4C2F8LDA<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/VJHQ4C2F8LDA.jpg' alt=\"Machine Learning Coursera VJHQ4C2F8LDA\"><\/a><figcaption class=\"image-caption\">Coursera: Machine Learning - VJHQ4C2F8LDA<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/UC-93630ce4-c301-401f-a935-a38155ba79aa.jpg\" title=\"Udemy: Microsoft SQL Server 2019: For Beginners - UC-93630ce4-c301-401f-a935-a38155ba79aa\" data-thumbnail=\"\/files\/UC-93630ce4-c301-401f-a935-a38155ba79aa.jpg\" data-sub-html=\"<h2>Microsoft SQL Server 2019_ For Beginners Udemy UC-93630ce4-c301-401f-a935-a38155ba79aa<\/h2><p>Udemy: Microsoft SQL Server 2019: For Beginners - UC-93630ce4-c301-401f-a935-a38155ba79aa<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/UC-93630ce4-c301-401f-a935-a38155ba79aa.jpg' alt=\"Microsoft SQL Server 2019_ For Beginners Udemy UC-93630ce4-c301-401f-a935-a38155ba79aa\"><\/a><figcaption class=\"image-caption\">Udemy: Microsoft SQL Server 2019: For Beginners - UC-93630ce4-c301-401f-a935-a38155ba79aa<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/networking-foundations-local-area-networks-lans-24682603.jpg\" title=\"LinkedIn Learning: Networking Foundations: Local Area Networks (LANs)\" data-thumbnail=\"\/files\/networking-foundations-local-area-networks-lans-24682603.jpg\" data-sub-html=\"<h2>Networking Foundations Network Media LANs<\/h2><p>LinkedIn Learning: Networking Foundations: Local Area Networks (LANs)<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/networking-foundations-local-area-networks-lans-24682603.jpg' alt=\"Networking Foundations Network Media LANs\"><\/a><figcaption class=\"image-caption\">LinkedIn Learning: Networking Foundations: Local Area Networks (LANs)<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/hackerrank-skills-verification-python-basic.png\" title=\"HackerRank: Python (Basic) Skills Certification Test\" data-thumbnail=\"\/files\/hackerrank-skills-verification-python-basic.png\" data-sub-html=\"<h2>Python Assessment for skill HackerRank Certificate<\/h2><p>HackerRank: Python (Basic) Skills Certification Test<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/hackerrank-skills-verification-python-basic.png' alt=\"Python Assessment for skill HackerRank Certificate\"><\/a><figcaption class=\"image-caption\">HackerRank: Python (Basic) Skills Certification Test<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/cognitiveclass-introduction-deep-learning.png\" title=\"Cognitive Class: Deep Learning Fundamentals course\" data-thumbnail=\"\/files\/cognitiveclass-introduction-deep-learning.png\" data-sub-html=\"<h2>Deep Learning Fundamentals TV ML0115EN Certificate Cognitive Class<\/h2><p>Cognitive Class: Deep Learning Fundamentals course<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/cognitiveclass-introduction-deep-learning.png' alt=\"Deep Learning Fundamentals TV ML0115EN Certificate Cognitive Class\"><\/a><figcaption class=\"image-caption\">Cognitive Class: Deep Learning Fundamentals course<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/hackerrank-skills-verification-problem-solving-basic.png\" title=\"HackerRank: Problem Solving (Basic) Skills Certification Test\" data-thumbnail=\"\/files\/hackerrank-skills-verification-problem-solving-basic.png\" data-sub-html=\"<h2>Problem Solving (Basic) HackerRank Certificate<\/h2><p>HackerRank: Problem Solving (Basic) Skills Certification Test<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/hackerrank-skills-verification-problem-solving-basic.png' alt=\"Problem Solving (Basic) HackerRank Certificate\"><\/a><figcaption class=\"image-caption\">HackerRank: Problem Solving (Basic) Skills Certification Test<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/UC-c9524150-649d-4ed2-bdcc-9fcfd284297a.jpg\" title=\"Udemy: Adobe Photoshop CC Fundamentals and Essentials Training - UC-c9524150-649d-4ed2-bdcc-9fcfd284297a\" data-thumbnail=\"\/files\/UC-c9524150-649d-4ed2-bdcc-9fcfd284297a.jpg\" data-sub-html=\"<h2>Adobe Photoshop CC Fundamentals and Essentials Training Udemy UC-c9524150-649d-4ed2-bdcc-9fcfd284297a<\/h2><p>Udemy: Adobe Photoshop CC Fundamentals and Essentials Training - UC-c9524150-649d-4ed2-bdcc-9fcfd284297a<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/UC-c9524150-649d-4ed2-bdcc-9fcfd284297a.jpg' alt=\"Adobe Photoshop CC Fundamentals and Essentials Training Udemy UC-c9524150-649d-4ed2-bdcc-9fcfd284297a\"><\/a><figcaption class=\"image-caption\">Udemy: Adobe Photoshop CC Fundamentals and Essentials Training - UC-c9524150-649d-4ed2-bdcc-9fcfd284297a<\/figcaption>\n <\/figure><\/p>\n<hr>\n<p>The content and images of this post are protected by intellectual property rights and safeguarded by the protection service offered by <a href=\"https:\/\/www.dmca.com\/r\/4d2qkez\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">dmca.com\/r\/4d2qkez<\/a>.<\/p>\n<p><a href=\"https:\/\/www.dmca.com\/Protection\/Status.aspx?id=7f8c10cf-e84d-4bc6-ae00-edf047097d43&refurl=https%3a%2f%2fsamirpaulb.github.io%2fposts%2fsome-of-my-online-courses-certificates%2f&rlo=true\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/images\/remote\/DMCA_logo-std-btn120w_8178559781566142492.png' srcset=\"https:\/\/samir.pages.dev\/images\/remote\/DMCA_logo-std-btn120w_8178559781566142492_hu_aabf5f97d8a613e2.webp 800w, https:\/\/samir.pages.dev\/images\/remote\/DMCA_logo-std-btn120w_8178559781566142492_hu_f2e4abe31e886b38.webp 1200w, https:\/\/samir.pages.dev\/images\/remote\/DMCA_logo-std-btn120w_8178559781566142492_hu_8f6df3f39d03171a.webp 1600w\" alt=\"DMCA.com Protection Status\" height=\"45\" width=\"121\"><\/a><\/p>\n<script defer src=\"https:\/\/images.dmca.com\/Badges\/DMCABadgeHelper.min.js\"> <\/script>\n"},{"title":"Online PDF Compression Tool \u2014 Reduce PDF File Size Instantly","link":"https:\/\/samir.pages.dev\/posts\/online-pdf-compression-tool\/","pubDate":"Wed, 20 Jan 2021 00:00:00 +0000","author":"netproxy@duck.com (Samir)","guid":"https:\/\/samir.pages.dev\/posts\/online-pdf-compression-tool\/","category":"Projects","description":"<img src=\"https:\/\/samir.pages.dev\/files\/online-pdf-compression-tool.webp\" alt=\"featured image\" referrerpolicy=\"no-referrer\"><h2 class=\"heading-element\" id=\"about-the-project\"><span>About The Project<\/span>\n <a href=\"#about-the-project\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><p>The <strong>Online PDF Compression Tool<\/strong> is a powerful web-based application designed to reduce PDF file sizes efficiently while maintaining document quality. Built with modern web technologies and advanced PDF processing algorithms, this tool provides a fast, secure, and user-friendly solution for PDF optimization.<\/p>\n<h3 class=\"heading-element\" id=\"key-features\"><span>Key Features<\/span>\n <a href=\"#key-features\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><ul>\n<li><strong>High Compression Ratio<\/strong>: Reduces PDF file sizes by up to 70% while preserving visual quality<\/li>\n<li><strong>Secure Processing<\/strong>: Files are automatically deleted from the server after 1 hour for maximum privacy<\/li>\n<li><strong>Fast Processing<\/strong>: Optimized backend processing with PDFNetPython ensures quick compression<\/li>\n<li><strong>Cross-Platform<\/strong>: Works seamlessly on all devices with a modern web browser<\/li>\n<li><strong>No Installation Required<\/strong>: Fully web-based solution accessible from anywhere<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"technologies-used\"><span>Technologies Used<\/span>\n <a href=\"#technologies-used\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>This project leverages a robust tech stack for optimal performance and security:<\/p>\n<ul>\n<li><strong>Backend<\/strong>: Python 3, Flask (web framework), PDFNetPython (PDF processing)<\/li>\n<li><strong>Frontend<\/strong>: HTML5, CSS3, JavaScript (ES6+)<\/li>\n<li><strong>Additional Tools<\/strong>: C, Shell scripting, Git version control<\/li>\n<li><strong>Deployment<\/strong>: Replit platform, Nix environment configuration<\/li>\n<\/ul>\n<h3 class=\"heading-element\" id=\"live-demo\"><span>Live Demo<\/span>\n <a href=\"#live-demo\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>Experience the PDF compression tool in action:<\/p>\n<p><strong><a href=\"https:\/\/filecompressor.samirpaul.repl.co\" target=\"_blank\" rel=\"external nofollow noopener noreferrer\">\ud83d\ude80 Launch Live Demo<\/a><\/strong><\/p>\n<h2 class=\"heading-element\" id=\"video-demo\"><span>Video Demo:<\/span>\n <a href=\"#video-demo\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><div>\n<video width=\"100%\" id=\"video\" controls>\n<source loading=\"lazy\" src=\"https:\/\/user-images.githubusercontent.com\/77569653\/172896703-9e4998c1-40da-46ae-810e-780e47a391f9.mp4\" type=\"video\/mp4\">\nYour browser does not support the video tag.\n<\/video>\n<\/div>\n<h3 class=\"heading-element\" id=\"application-interface\"><span>Application Interface<\/span>\n <a href=\"#application-interface\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p><strong>Landing Page<\/strong><\/p>\n<p><figure><a class=\"lightgallery\" target=\"_blank\" href=\"https:\/\/samir.pages.dev\/files\/filecompressor-samirpaul1-repl-co-landing-page.webp\" title=\"PDF Compressor Landing Page - Upload and Compress PDF Files Instantly\" data-thumbnail=\"\/files\/filecompressor-samirpaul1-repl-co-landing-page.webp\" data-sub-html=\"<h2>Online PDF Compression Tool Landing Page - Clean and intuitive web interface with drag-and-drop upload zone, file size indicator, compression progress bar, and secure file handling features for fast PDF optimization using Python Flask and PDFNetPython<\/h2><p>PDF Compressor Landing Page - Upload and Compress PDF Files Instantly<\/p>\"><img loading=\"lazy\" src='https:\/\/samir.pages.dev\/files\/filecompressor-samirpaul1-repl-co-landing-page.webp' alt=\"Online PDF Compression Tool Landing Page - Clean and intuitive web interface with drag-and-drop upload zone, file size indicator, compression progress bar, and secure file handling features for fast PDF optimization using Python Flask and PDFNetPython\"><\/a><figcaption class=\"image-caption\">PDF Compressor Landing Page - Upload and Compress PDF Files Instantly<\/figcaption>\n <\/figure><\/p>\n<h2 class=\"heading-element\" id=\"technical-implementation\"><span>Technical Implementation<\/span>\n <a href=\"#technical-implementation\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><h3 class=\"heading-element\" id=\"flask-file-upload-system\"><span>Flask File Upload System<\/span>\n <a href=\"#flask-file-upload-system\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The application uses Flask&rsquo;s robust file handling capabilities to manage PDF uploads securely. The HTML form&rsquo;s <code>enctype<\/code> property is set to <code>&quot;multipart\/form-data&quot;<\/code> to enable file transmission. The server-side Flask application extracts files from the <code>request.files[]<\/code> object and processes them securely.<\/p>\n<p><strong>Key Configuration Parameters:<\/strong><\/p>\n<ul>\n<li><code>app.config['UPLOAD_FOLDER']<\/code>: Defines the temporary storage location for uploaded files<\/li>\n<li><code>app.config['MAX_CONTENT_LENGTH']<\/code>: Sets the maximum allowed file size in bytes<\/li>\n<li>Secure filename generation using <code>werkzeug.utils.secure_filename()<\/code> to prevent path traversal attacks<\/li>\n<\/ul>\n<p><strong>Flask File Upload Implementation:<\/strong><\/p>\n<pre><code>from flask import Flask, render_template, request, jsonify\nfrom werkzeug.utils import secure_filename\nimport os\n\napp = Flask(__name__)\n\n# Configuration\napp.config[&#39;UPLOAD_FOLDER&#39;] = &#39;uploads\/temp&#39;\napp.config[&#39;MAX_CONTENT_LENGTH&#39;] = 16 * 1024 * 1024 # 16 MB max file size\napp.config[&#39;ALLOWED_EXTENSIONS&#39;] = {&#39;pdf&#39;}\n\ndef allowed_file(filename):\n &#34;&#34;&#34;Check if uploaded file has a valid PDF extension&#34;&#34;&#34;\n return &#39;.&#39; in filename and \\\n filename.rsplit(&#39;.&#39;, 1)[1].lower() in app.config[&#39;ALLOWED_EXTENSIONS&#39;]\n\n@app.route(&#39;\/upload&#39;)\ndef upload_file():\n &#34;&#34;&#34;Render the file upload interface&#34;&#34;&#34;\n return render_template(&#39;upload.html&#39;)\n\n@app.route(&#39;\/uploader&#39;, methods=[&#39;GET&#39;, &#39;POST&#39;])\ndef upload_handler():\n &#34;&#34;&#34;Handle file upload and initiate compression&#34;&#34;&#34;\n if request.method == &#39;POST&#39;:\n # Check if file is present in request\n if &#39;file&#39; not in request.files:\n return jsonify({&#39;error&#39;: &#39;No file provided&#39;}), 400\n\n f = request.files[&#39;file&#39;]\n\n # Validate file\n if f.filename == &#39;&#39;:\n return jsonify({&#39;error&#39;: &#39;No file selected&#39;}), 400\n\n if not allowed_file(f.filename):\n return jsonify({&#39;error&#39;: &#39;Only PDF files are allowed&#39;}), 400\n\n # Save file securely\n filename = secure_filename(f.filename)\n filepath = os.path.join(app.config[&#39;UPLOAD_FOLDER&#39;], filename)\n f.save(filepath)\n\n return jsonify({\n &#39;message&#39;: &#39;File uploaded successfully&#39;,\n &#39;filename&#39;: filename\n }), 200\n\nif __name__ == &#39;__main__&#39;:\n # Create upload directory if it doesn&#39;t exist\n os.makedirs(app.config[&#39;UPLOAD_FOLDER&#39;], exist_ok=True)\n app.run(debug=True)<\/code><\/pre><h3 class=\"heading-element\" id=\"pdf-compression-backend\"><span>PDF Compression Backend<\/span>\n <a href=\"#pdf-compression-backend\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>The compression engine utilizes the powerful PDFNetPython library, which provides advanced PDF optimization capabilities. The compression process analyzes the PDF structure, removes redundancies, compresses streams, and optimizes embedded resources.<\/p>\n<p><strong>Compression Process Steps:<\/strong><\/p>\n<ol>\n<li>Initialize PDFNet library with license key<\/li>\n<li>Load the input PDF document<\/li>\n<li>Verify document security handlers<\/li>\n<li>Apply Optimizer to reduce file size<\/li>\n<li>Save linearized PDF for optimal web viewing<\/li>\n<li>Validate and return compression results<\/li>\n<\/ol>\n<pre><code>import os\nimport sys\nfrom PDFNetPython3.PDFNetPython import PDFDoc, Optimizer, SDFDoc, PDFNet\n\ndef compress_file(input_file: str, output_file: str = None) -&gt; bool:\n &#34;&#34;&#34;\n Compress a PDF file using PDFNetPython&#39;s advanced optimization.\n\n Args:\n input_file: Path to the PDF file to compress\n output_file: Path where compressed PDF will be saved.\n If None, overwrites the input file.\n\n Returns:\n bool: True if compression successful, False otherwise\n &#34;&#34;&#34;\n if not output_file:\n output_file = input_file\n\n try:\n # Initialize PDFNet library\n PDFNet.Initialize()\n\n # Open the PDF document\n doc = PDFDoc(input_file)\n\n # Initialize security handlers (required for protected PDFs)\n doc.InitSecurityHandler()\n\n # Apply optimization to reduce file size\n # This includes: stream compression, font subsetting, and object removal\n Optimizer.Optimize(doc)\n\n # Save optimized PDF with linearization for web viewing\n # Linearized PDFs display faster in web browsers\n doc.Save(output_file, SDFDoc.e_linearized)\n\n # Close document and release resources\n doc.Close()\n\n return True\n\n except Exception as e:\n print(f&#34;Error during PDF compression: {str(e)}&#34;)\n try:\n doc.Close()\n except:\n pass\n return False\n\nif __name__ == &#34;__main__&#34;:\n # Command line usage: python compress_pdf.py input.pdf output.pdf\n if len(sys.argv) &lt; 3:\n print(&#34;Usage: python compress_pdf.py &lt;input_pdf&gt; &lt;output_pdf&gt;&#34;)\n sys.exit(1)\n\n input_file = sys.argv[1]\n output_file = sys.argv[2]\n\n if not os.path.exists(input_file):\n print(f&#34;Error: Input file &#39;{input_file}&#39; not found&#34;)\n sys.exit(1)\n\n success = compress_file(input_file, output_file)\n if success:\n original_size = os.path.getsize(input_file)\n compressed_size = os.path.getsize(output_file)\n ratio = ((original_size - compressed_size) \/ original_size) * 100\n\n print(f&#34;\u2713 Compression successful!&#34;)\n print(f&#34; Original size: {original_size \/ 1024:.2f} KB&#34;)\n print(f&#34; Compressed size: {compressed_size \/ 1024:.2f} KB&#34;)\n print(f&#34; Compression ratio: {ratio:.2f}%&#34;)\n else:\n print(&#34;\u2717 Compression failed&#34;)\n sys.exit(1)<\/code><\/pre><h3 class=\"heading-element\" id=\"client-side-file-download\"><span>Client-Side File Download<\/span>\n <a href=\"#client-side-file-download\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h3><p>After successful compression, the client-side JavaScript handles automatic file download using modern browser APIs. The implementation uses the Fetch API to retrieve the compressed file and creates a temporary download link for seamless user experience.<\/p>\n<p><strong>Download Process:<\/strong><\/p>\n<ol>\n<li>Receive compressed file metadata from server<\/li>\n<li>Fetch compressed PDF blob from server<\/li>\n<li>Create temporary object URL for download<\/li>\n<li>Trigger automatic download<\/li>\n<li>Clean up temporary resources<\/li>\n<\/ol>\n<pre><code>\/**\n * Download compressed PDF file automatically\n * @param {string} filename - Name of the compressed file to download\n *\/\nfunction downloadFile(filename) {\n \/\/ Validate response and filename\n if (response !== null &amp;&amp; response.filename) {\n const fname = response.filename;\n const url = `static\/resource\/${fname}`;\n\n console.log(`Downloading file from: ${url}`);\n\n \/\/ Fetch compressed PDF as blob\n fetch(url)\n .then(response =&gt; {\n if (!response.ok) {\n throw new Error(`HTTP error! status: ${response.status}`);\n }\n return response.blob();\n })\n .then(blob =&gt; {\n \/\/ Create temporary download link\n const link = document.createElement(&#34;a&#34;);\n link.href = URL.createObjectURL(blob);\n link.download = fname;\n\n \/\/ Append to DOM (required for Firefox)\n document.body.appendChild(link);\n\n \/\/ Trigger download\n link.click();\n\n \/\/ Clean up: remove link and revoke object URL\n document.body.removeChild(link);\n URL.revokeObjectURL(link.href);\n\n console.log(&#34;\u2713 Download initiated successfully&#34;);\n })\n .catch(error =&gt; {\n console.error(&#34;\u2717 Download failed:&#34;, error);\n alert(&#34;Failed to download file. Please try again.&#34;);\n });\n } else {\n console.error(&#34;\u2717 Invalid response or filename&#34;);\n alert(&#34;File not ready for download. Please wait for compression to complete.&#34;);\n }\n}\n\n\/\/ Alternative: Display download progress\nfunction downloadFileWithProgress(filename) {\n if (response !== null &amp;&amp; response.filename) {\n const fname = response.filename;\n const url = `static\/resource\/${fname}`;\n\n fetch(url)\n .then(response =&gt; {\n const contentLength = response.headers.get(&#39;content-length&#39;);\n const total = parseInt(contentLength, 10);\n let loaded = 0;\n\n return new Response(\n new ReadableStream({\n start(controller) {\n const reader = response.body.getReader();\n\n read();\n function read() {\n reader.read().then(({done, value}) =&gt; {\n if (done) {\n controller.close();\n return;\n }\n loaded &#43;= value.byteLength;\n const progress = (loaded \/ total) * 100;\n console.log(`Download progress: ${progress.toFixed(2)}%`);\n\n controller.enqueue(value);\n read();\n }).catch(error =&gt; {\n console.error(error);\n controller.error(error);\n });\n }\n }\n })\n );\n })\n .then(response =&gt; response.blob())\n .then(blob =&gt; {\n const link = document.createElement(&#34;a&#34;);\n link.href = URL.createObjectURL(blob);\n link.download = fname;\n document.body.appendChild(link);\n link.click();\n document.body.removeChild(link);\n URL.revokeObjectURL(link.href);\n })\n .catch(error =&gt; {\n console.error(&#34;Download error:&#34;, error);\n });\n }\n}<\/code><\/pre><h2 class=\"heading-element\" id=\"-how-to-contribute\"><span>\ud83e\udd14 How to contribute<\/span>\n <a href=\"#-how-to-contribute\" class=\"heading-mark\"><svg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"><path d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"><\/path><\/svg><\/a>\n<\/h2><ul>\n<li data-task=\"x\"><i class=\"fa-regular fa-check-square\" aria-hidden=\"true\"><\/i><span title=\"Checked\">Fork this repository;<\/span><\/li>\n<li data-task=\"x\"><i class=\"fa-regular fa-check-square\" aria-hidden=\"true\"><\/i><span title=\"Checked\">Create a branch with your feature: <code>git checkout -b my-feature<\/code>;<\/span><\/li>\n<li data-task=\"x\"><i class=\"fa-regular fa-check-square\" aria-hidden=\"true\"><\/i><span title=\"Checked\">Commit your changes: <code>git commit -m &quot;feat: my new feature&quot;<\/code>;<\/span><\/li>\n<li data-task=\"x\"><i class=\"fa-regular fa-check-square\" aria-hidden=\"true\"><\/i><span title=\"Checked\">Push to your branch: <code>git push origin my-feature<\/code>.<\/span><\/li>\n<\/ul>\n"}]}}