[{"categories":["software-engineering"],"collections":null,"content":"Practical guide to database schema migrations using golang-migrate in microservices. Covers PostgreSQL, ClickHouse clusters with ZooKeeper, CI\/CD automation with GitHub Actions, advisory locks, dirty state recovery, zero-downtime expand-contract migrations, and distributed table patterns.","date":"03-Jul-2026","objectID":"\/posts\/database-schema-migrations\/","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/"},{"categories":["software-engineering"],"collections":null,"content":" Disclaimer I’m not endorsing or discouraging any specific tool in this post. This is purely from my personal experience working with microservices and different databases across various projects over the years. What worked best for me might not be the right fit for your team or stack. Evaluate based on your own constraints \u2014 team size, database engine, deployment model, and risk tolerance. Every tool mentioned here has trade-offs, and the “best” choice depends entirely on your context. ","date":"03-Jul-2026","heading":"","objectID":"\/posts\/database-schema-migrations\/:0:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#"},{"categories":["software-engineering"],"collections":null,"content":"Why Schema Migrations Matter Every application that talks to a database eventually needs to change its schema. New columns, new tables, renamed fields, dropped constraints. In a monolith with one database, this is manageable. In a microservices architecture with multiple databases (maybe even different engines \u2014 PostgreSQL for transactional data, ClickHouse for analytics), it gets messy fast. The problems I kept running into: Who applied what, and when? No audit trail of schema changes. Works on my machine. Dev database has the column, staging doesn’t. Two pipelines running at once. Race condition corrupts the migration state. Failed migration halfway through. Database stuck in a dirty state, all future migrations blocked. Shared schema, multiple services. Service A needs a new column, Service B still reads the old one. How do you not break either? Sharded clusters. DDL needs to propagate to 20+ nodes consistently. After trying a few approaches (hand-written SQL scripts, ORM-based migrations, custom tooling), I settled on golang-migrate \u2014 a database-agnostic migration tool written in Go that works as both a CLI and a library. ","date":"03-Jul-2026","heading":"Why Schema Migrations Matter","objectID":"\/posts\/database-schema-migrations\/:1:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#why-schema-migrations-matter"},{"categories":["software-engineering"],"collections":null,"content":"Why Not Flyway or Liquibase? Before landing on golang-migrate, I tried two of the most popular migration tools: Flyway and Liquibase. Both worked fine for PostgreSQL. But my stack also included a ClickHouse cluster with ZooKeeper coordination \u2014 and that’s where things fell apart. The ClickHouse + ZooKeeper Problem ClickHouse in clustered mode requires ON CLUSTER DDL statements that propagate schema changes across all shards via ZooKeeper. Neither Flyway nor Liquibase had proper support for this at the time I evaluated them: Flyway \u2014 The community ClickHouse driver was incomplete. ON CLUSTER DDL propagation wasn’t handled correctly, and the tool would report success while only one node received the schema change. Multi-statement migrations (needed for creating both _local and Distributed tables) required hacks. Liquibase \u2014 The ClickHouse extension existed but was buggy with ZooKeeper-coordinated DDL. The changelog tracking didn’t play well with ClickHouse’s eventual consistency model across shards. golang-migrate’s ClickHouse driver supports x-multi-statement=true in the connection URL, handles ON CLUSTER DDL natively, and works with both the native TCP protocol (port 9440 with TLS) and the standard ClickHouse cluster topology. If your stack is PostgreSQL-only or MySQL-only, Flyway and Liquibase are solid choices with larger ecosystems and excellent Java integration. But if you’re dealing with ClickHouse clusters, sharded analytical databases, or exotic engines \u2014 golang-migrate’s lightweight driver model handled it significantly better in my experience. ","date":"03-Jul-2026","heading":"Why Not Flyway or Liquibase?","objectID":"\/posts\/database-schema-migrations\/:2:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#why-not-flyway-or-liquibase"},{"categories":["software-engineering"],"collections":null,"content":"The Architecture \u2014 High-Level Overview Here’s the architecture I ended up with across projects: %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#4fc3f7', 'primaryTextColor': '#000', 'primaryBorderColor': '#0288d1', 'lineColor': '#455a64', 'secondaryColor': '#a5d6a7', 'tertiaryColor': '#fff9c4', 'background': '#fafafa'}}}%% graph TD DEV[\"Developer\"] GIT[\"Git Repository\"] WEBHOOK[\"Push Event\"] PIPELINE[\"GitHub Actions\"] PG[\"PostgreSQL\"] CH[\"ClickHouse Cluster\"] DEV -->|\"git push\"| GIT GIT -->|\"branch event\"| WEBHOOK WEBHOOK -->|\"triggers workflow\"| PIPELINE PIPELINE -->|\"migrate up\"| PG PIPELINE -->|\"migrate up\"| CH style DEV fill:#e3f2fd,stroke:#1565c0,stroke-width:2px,color:#000 style GIT fill:#fce4ec,stroke:#c62828,stroke-width:2px,color:#000 style WEBHOOK fill:#fff3e0,stroke:#e65100,stroke-width:2px,color:#000 style PIPELINE fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px,color:#000 style PG fill:#e8eaf6,stroke:#283593,stroke-width:2px,color:#000 style CH fill:#fff8e1,stroke:#f57f17,stroke-width:2px,color:#000 The key insight: migrations are just versioned SQL files in a git repository. The CI\/CD pipeline is the only thing that applies them to databases. No developer runs migrations manually against staging or production. ","date":"03-Jul-2026","heading":"The Architecture \u2014 High-Level Overview","objectID":"\/posts\/database-schema-migrations\/:3:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#the-architecture--high-level-overview"},{"categories":["software-engineering"],"collections":null,"content":"Repository Structure Here’s what a mature migration project looks like: db-migrations\/ clickhouse\/ 001_create_core_tables.up.sql 001_create_core_tables.down.sql 002_seed_reference_data.up.sql 002_seed_reference_data.down.sql 003_create_merge_tables.up.sql 003_create_merge_tables.down.sql 004_create_parameterized_views.up.sql 004_create_parameterized_views.down.sql 005_add_analytics_tables.up.sql 005_add_analytics_tables.down.sql postgresql\/ 001_create_users_table.up.sql 001_create_users_table.down.sql 002_create_api_keys_table.up.sql 002_create_api_keys_table.down.sql 003_create_feedback_table.up.sql 003_create_feedback_table.down.sql .github\/ workflows\/ migrate.yml Makefile .gitignore README.md Key decisions in this structure: Separate directories per database engine \u2014 Each has independent versioning. ClickHouse might be at version 005 while PostgreSQL is at version 003. .github\/workflows\/migrate.yml \u2014 GitHub Actions workflow that installs golang-migrate and runs migrations automatically on push. No secrets in the repo \u2014 Database credentials are stored as GitHub Actions secrets and injected as environment variables at runtime. ","date":"03-Jul-2026","heading":"Repository Structure","objectID":"\/posts\/database-schema-migrations\/:4:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#repository-structure"},{"categories":["software-engineering"],"collections":null,"content":"How golang-migrate Works ","date":"03-Jul-2026","heading":"How golang-migrate Works","objectID":"\/posts\/database-schema-migrations\/:5:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#how-golang-migrate-works"},{"categories":["software-engineering"],"collections":null,"content":"Migration Files Each migration is a pair of files \u2014 one to apply, one to reverse: .up.sql \u2014 applies the change (moving forward) .down.sql \u2014 reverses the change (rollback) The numeric prefix determines execution order. golang-migrate compares these numbers against what’s stored in the database and applies only what’s new. ","date":"03-Jul-2026","heading":"Migration Files","objectID":"\/posts\/database-schema-migrations\/:5:1","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#migration-files"},{"categories":["software-engineering"],"collections":null,"content":"The schema_migrations Table golang-migrate creates a single tracking table in each database: CREATE TABLE schema_migrations ( version BIGINT NOT NULL, dirty BOOLEAN NOT NULL ); Column Purpose version Highest applied migration number dirty true if a migration failed midway On each run, the tool checks: “What’s the current version? Are there SQL files with a higher number? If yes, apply them in order.” %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#bbdefb', 'primaryTextColor': '#000', 'lineColor': '#1565c0'}}}%% graph LR FILES[\"SQL Files: 001 to 005\"] TABLE[\"schema_migrations: version=3\"] DIFF[\"Diff: 004, 005 pending\"] APPLY[\"Apply 004 and 005\"] DONE[\"version=5, dirty=false\"] FILES --> DIFF TABLE --> DIFF DIFF --> APPLY --> DONE style FILES fill:#e3f2fd,stroke:#1565c0,stroke-width:2px style TABLE fill:#fff3e0,stroke:#e65100,stroke-width:2px style DIFF fill:#fce4ec,stroke:#c62828,stroke-width:2px style APPLY fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style DONE fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px ","date":"03-Jul-2026","heading":"The schema_migrations Table","objectID":"\/posts\/database-schema-migrations\/:5:2","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#the-schema_migrations-table"},{"categories":["software-engineering"],"collections":null,"content":"Core Commands Command What it does migrate up Apply all pending .up.sql files in order migrate up N Apply only the next N migrations migrate down Apply .down.sql to rollback (I rarely use this in production) migrate version Show current applied version number migrate force V Reset version to V, clears dirty flag Basic CLI usage: # Apply all pending migrations migrate -source file:\/\/.\/clickhouse \\ -database \"clickhouse:\/\/host:9440?database=mydb&username=user&password=pass&x-multi-statement=true&secure=true\" \\ up # Check current version migrate -source file:\/\/.\/postgresql \\ -database \"postgres:\/\/user:pass@host:5432\/mydb?sslmode=require\" \\ version ","date":"03-Jul-2026","heading":"Core Commands","objectID":"\/posts\/database-schema-migrations\/:5:3","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#core-commands"},{"categories":["software-engineering"],"collections":null,"content":"Branch-Based Environment Routing One pattern that worked well: the branch you push to determines which environment gets migrated. %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#e3f2fd', 'lineColor': '#37474f'}}}%% graph TD subgraph \"Git Branches\" DEV_BRANCH[\"dev branch\"] MAIN_BRANCH[\"main branch\"] end subgraph \"Pipeline Decision\" DEV_ENV[\"ENV=dev, dev secrets\"] PROD_ENV[\"ENV=prod, prod secrets\"] end subgraph \"Target Databases\" DEV_CH[\"ClickHouse Dev\"] DEV_PG[\"PostgreSQL Dev\"] PROD_CH[\"ClickHouse Prod\"] PROD_PG[\"PostgreSQL Prod\"] end DEV_BRANCH -->|\"push\"| DEV_ENV MAIN_BRANCH -->|\"merge\"| PROD_ENV DEV_ENV --> DEV_CH DEV_ENV --> DEV_PG PROD_ENV --> PROD_CH PROD_ENV --> PROD_PG style DEV_BRANCH fill:#e3f2fd,stroke:#1565c0,stroke-width:2px style MAIN_BRANCH fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style DEV_ENV fill:#e3f2fd,stroke:#1565c0,stroke-width:2px style PROD_ENV fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style DEV_CH fill:#fff3e0,stroke:#e65100,stroke-width:2px style DEV_PG fill:#e8eaf6,stroke:#283593,stroke-width:2px style PROD_CH fill:#fff3e0,stroke:#e65100,stroke-width:2px style PROD_PG fill:#e8eaf6,stroke:#283593,stroke-width:2px With GitHub Actions, this is implemented using branch-conditional secrets: # In your workflow: env: POSTGRES_URL: ${{ github.ref == 'refs\/heads\/main' && secrets.PG_PROD_URL || secrets.PG_DEV_URL }} CLICKHOUSE_URL: ${{ github.ref == 'refs\/heads\/main' && secrets.CH_PROD_URL || secrets.CH_DEV_URL }} Same migration SQL files, same workflow \u2014 only the connection URL changes based on which branch triggered the run. ","date":"03-Jul-2026","heading":"Branch-Based Environment Routing","objectID":"\/posts\/database-schema-migrations\/:6:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#branch-based-environment-routing"},{"categories":["software-engineering"],"collections":null,"content":"Secrets Management \u2014 Keep It Simple Database credentials should never be in your repository. With modern CI\/CD platforms like GitHub Actions, GitLab CI, or CircleCI, this is straightforward \u2014 store the full database URL as a secret and inject it at runtime. %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#e8f5e9', 'lineColor': '#2e7d32'}}}%% graph LR subgraph \"Your Repository\" CODE[\"SQL migration files\"] end subgraph \"CI\/CD Platform Secrets\" SEC[\"POSTGRES_URL, CLICKHOUSE_URL\"] end subgraph \"Runtime\" RUN[\"migrate -database $URL up\"] end CODE -->|\"push\"| RUN SEC -->|\"injected as env vars\"| RUN style CODE fill:#e3f2fd,stroke:#1565c0,stroke-width:2px style SEC fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px style RUN fill:#fff3e0,stroke:#e65100,stroke-width:2px Where to store secrets GitHub Actions Secrets \u2014 Encrypted, only exposed to workflows, never shown in logs HashiCorp Vault \u2014 For teams that need dynamic credentials, rotation, and audit trails AWS Secrets Manager \/ GCP Secret Manager \u2014 Cloud-native secret storage Doppler \u2014 Universal secrets manager that syncs across platforms The key rule: the database URL is assembled by the CI\/CD platform, not by your code. Your migration script just reads $DATABASE_URL from the environment. ","date":"03-Jul-2026","heading":"Secrets Management \u2014 Keep It Simple","objectID":"\/posts\/database-schema-migrations\/:7:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#secrets-management--keep-it-simple"},{"categories":["software-engineering"],"collections":null,"content":"The CI\/CD Pipeline \u2014 GitHub Actions Here’s a complete GitHub Actions workflow that runs migrations automatically on push: # .github\/workflows\/migrate.yml name: Database Migrations on: push: branches: [main, dev] paths: - 'clickhouse\/**' - 'postgresql\/**' env: MIGRATE_VERSION: \"v4.17.0\" jobs: migrate: runs-on: ubuntu-latest steps: - uses: actions\/checkout@v4 - name: Install golang-migrate run: | curl -L https:\/\/github.com\/golang-migrate\/migrate\/releases\/download\/$MIGRATE_VERSION\/migrate.linux-amd64.tar.gz | tar xz sudo mv migrate \/usr\/local\/bin\/ - name: Run ClickHouse migrations run: migrate -source file:\/\/.\/clickhouse -database \"$CLICKHOUSE_URL\" up env: CLICKHOUSE_URL: ${{ github.ref == 'refs\/heads\/main' && secrets.CH_PROD_URL || secrets.CH_DEV_URL }} - name: Run PostgreSQL migrations run: migrate -source file:\/\/.\/postgresql -database \"$POSTGRES_URL\" up env: POSTGRES_URL: ${{ github.ref == 'refs\/heads\/main' && secrets.PG_PROD_URL || secrets.PG_DEV_URL }} %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#f3e5f5', 'lineColor': '#4a148c'}}}%% graph TD subgraph \"Pipeline Trigger\" PUSH[\"Git push \/ merge\"] end subgraph \"Setup\" CHECKOUT[\"Checkout repository\"] INSTALL[\"Install golang-migrate\"] end subgraph \"Run Migrations\" CH_RUN[\"migrate -database $CLICKHOUSE_URL up\"] PG_RUN[\"migrate -database $POSTGRES_URL up\"] end subgraph \"Result\" SUCCESS[\"Workflow passes \u2713\"] FAILURE[\"Workflow fails \u2717\"] end PUSH --> CHECKOUT --> INSTALL INSTALL --> CH_RUN --> PG_RUN PG_RUN -->|\"exit 0\"| SUCCESS PG_RUN -->|\"exit != 0\"| FAILURE CH_RUN -->|\"exit != 0\"| FAILURE style PUSH fill:#e3f2fd,stroke:#1565c0,stroke-width:2px style CHECKOUT fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px style INSTALL fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px style CH_RUN fill:#fff3e0,stroke:#e65100,stroke-width:2px style PG_RUN fill:#e8eaf6,stroke:#283593,stroke-width:2px style SUCCESS fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px style FAILURE fill:#ffcdd2,stroke:#b71c1c,stroke-width:2px Key details golang-migrate is downloaded from GitHub Releases \u2014 fast, reliable, and you pin the exact version. Branch-based routing via GitHub expression: main \u2192 production secrets, everything else \u2192 dev secrets. Path filter (paths: ['clickhouse\/**', 'postgresql\/**']) \u2014 the workflow only runs when migration files actually change, not on every push. Database URLs stored as GitHub Secrets \u2014 never exposed in logs, encrypted at rest by GitHub. ","date":"03-Jul-2026","heading":"The CI\/CD Pipeline \u2014 GitHub Actions","objectID":"\/posts\/database-schema-migrations\/:8:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#the-cicd-pipeline--github-actions"},{"categories":["software-engineering"],"collections":null,"content":"ClickHouse Migration Patterns This section covers patterns specific to ClickHouse clustered deployments. These are the patterns that Flyway and Liquibase couldn’t handle properly. ","date":"03-Jul-2026","heading":"ClickHouse Migration Patterns","objectID":"\/posts\/database-schema-migrations\/:9:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#clickhouse-migration-patterns"},{"categories":["software-engineering"],"collections":null,"content":"Pattern 1: Local + Distributed Tables In a sharded ClickHouse cluster, you always create tables in pairs \u2014 a _local table on each shard, and a Distributed table that routes queries across shards: -- 001_create_events.up.sql -- Local table: exists on every shard, stores actual data CREATE TABLE IF NOT EXISTS analytics.events_local ON CLUSTER '{cluster}' ( event_id UUID DEFAULT generateUUIDv4(), event_time DateTime64(3), user_id UInt64, event_type LowCardinality(String), properties String, site_key LowCardinality(String) ) ENGINE = ReplicatedMergeTree('\/clickhouse\/tables\/{shard}\/events_local', '{replica}') PARTITION BY toYYYYMM(event_time) ORDER BY (site_key, event_time, user_id) TTL event_time + INTERVAL 2 YEAR; -- Distributed table: routes queries to all shards CREATE TABLE IF NOT EXISTS analytics.events ON CLUSTER '{cluster}' AS analytics.events_local ENGINE = Distributed('{cluster}', 'analytics', 'events_local', cityHash64(site_key)); %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#fff3e0', 'lineColor': '#e65100'}}}%% graph TD subgraph \"Distributed Table (Router)\" DIST[\"analytics.events - Distributed\"] end subgraph \"Shard 1\" L1[\"events_local\"] end subgraph \"Shard 2\" L2[\"events_local\"] end subgraph \"Shard 3\" L3[\"events_local\"] end subgraph \"Shard N...\" LN[\"events_local\"] end DIST -->|\"site_key hash \u2192 shard 1\"| L1 DIST -->|\"site_key hash \u2192 shard 2\"| L2 DIST -->|\"site_key hash \u2192 shard 3\"| L3 DIST -->|\"site_key hash \u2192 shard N\"| LN style DIST fill:#fff3e0,stroke:#e65100,stroke-width:2px style L1 fill:#e3f2fd,stroke:#1565c0,stroke-width:2px style L2 fill:#e3f2fd,stroke:#1565c0,stroke-width:2px style L3 fill:#e3f2fd,stroke:#1565c0,stroke-width:2px style LN fill:#e3f2fd,stroke:#1565c0,stroke-width:2px ","date":"03-Jul-2026","heading":"Pattern 1: Local + Distributed Tables","objectID":"\/posts\/database-schema-migrations\/:9:1","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#pattern-1-local--distributed-tables"},{"categories":["software-engineering"],"collections":null,"content":"Pattern 2: Merge Tables for Cross-Database Reads When you need to read from multiple source databases without copying data, ClickHouse’s Merge engine is incredibly powerful: -- 003_create_merge_tables.up.sql -- Read from ALL databases matching the pattern DROP TABLE IF EXISTS analytics.all_run_data ON CLUSTER '{cluster}'; CREATE TABLE analytics.all_run_data ON CLUSTER '{cluster}' ( run_id UUID, site_key LowCardinality(String), lot_id String, wafer_id String, program String, start_time DateTime64(3), end_time DateTime64(3), status LowCardinality(String) ) ENGINE = Merge(REGEXP('.*_source'), '^run_data$'); This creates a virtual table that reads from every database matching *_source and every table named run_data. No data is copied \u2014 it’s a live federated view across 30+ source databases. %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#e8f5e9', 'lineColor': '#2e7d32'}}}%% graph LR subgraph \"Merge Table (Virtual)\" MERGE[\"analytics.all_run_data - Merge Engine\"] end subgraph \"Source Databases (30+)\" DB1[\"site_a_source.run_data\"] DB2[\"site_b_source.run_data\"] DB3[\"site_c_source.run_data\"] DBN[\"site_n_source.run_data\"] end MERGE --> DB1 MERGE --> DB2 MERGE --> DB3 MERGE --> DBN style MERGE fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px style DB1 fill:#e3f2fd,stroke:#1565c0,stroke-width:1px style DB2 fill:#e3f2fd,stroke:#1565c0,stroke-width:1px style DB3 fill:#e3f2fd,stroke:#1565c0,stroke-width:1px style DBN fill:#e3f2fd,stroke:#1565c0,stroke-width:1px Merge engine: no data duplication The Merge engine stores zero data. It’s purely metadata \u2014 column definitions and a regex pattern. When queried, it fans out reads to all matching tables in real-time. If an upstream database adds a new table matching the pattern, it’s automatically included. This is perfect for multi-tenant or multi-site architectures where new data sources appear regularly. ","date":"03-Jul-2026","heading":"Pattern 2: Merge Tables for Cross-Database Reads","objectID":"\/posts\/database-schema-migrations\/:9:2","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#pattern-2-merge-tables-for-cross-database-reads"},{"categories":["software-engineering"],"collections":null,"content":"Pattern 3: Parameterized Views (ClickHouse 23.1+) Parameterized views let you create reusable query interfaces with runtime parameters: -- 004_create_parameterized_views.up.sql CREATE OR REPLACE VIEW analytics.run_summary_view ON CLUSTER '{cluster}' AS SELECT _database AS source_db, run_id, ifNull(lot_id, '') AS lot_id, ifNull(wafer_id, '') AS wafer_id, ifNull(program, '') AS program, start_time, end_time, dateDiff('second', start_time, end_time) AS duration_seconds, count(*) OVER (PARTITION BY lot_id) AS wafers_in_lot, row_number() OVER (PARTITION BY lot_id ORDER BY start_time) - 1 AS sequence_num FROM analytics.all_run_data WHERE _database = {site_db:String} ORDER BY start_time DESC; The application calls: SELECT * FROM analytics.run_summary_view(site_db='site_a_source') \u2014 clean API contract, all complexity hidden in the view. ","date":"03-Jul-2026","heading":"Pattern 3: Parameterized Views (ClickHouse 23.1+)","objectID":"\/posts\/database-schema-migrations\/:9:3","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#pattern-3-parameterized-views-clickhouse-231"},{"categories":["software-engineering"],"collections":null,"content":"Pattern 4: Seed Data Migrations Some migrations insert reference data rather than changing schema: -- 002_seed_reference_data.up.sql INSERT INTO analytics.site_registry (site_key, display_name, region, country, enabled) VALUES ('site_a', 'Austin Facility', 'americas', 'US', true), ('site_b', 'Munich Facility', 'emea', 'DE', true), ('site_c', 'Tokyo Facility', 'apac', 'JP', true), ('site_d', 'Bangalore Facility', 'apac', 'IN', true); INSERT INTO analytics.bin_config (bin_number, bin_name, color, quality) VALUES (0, 'PASS', '#4caf50', 'good'), (1, 'FAIL_ELECTRICAL', '#f44336', 'bad'), (2, 'FAIL_PARAMETRIC', '#ff9800', 'bad'), (3, 'RETEST', '#2196f3', 'retest'); -- ... 256 entries Seed data via Distributed tables When inserting seed data in a sharded cluster, always INSERT via the Distributed table (not _local). The Distributed engine automatically routes each row to the correct shard based on the sharding key. If you insert directly into _local, the data only exists on one shard. ","date":"03-Jul-2026","heading":"Pattern 4: Seed Data Migrations","objectID":"\/posts\/database-schema-migrations\/:9:4","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#pattern-4-seed-data-migrations"},{"categories":["software-engineering"],"collections":null,"content":"Multi-Shard Version Broadcast One challenge with ClickHouse clusters: the schema_migrations table uses a plain MergeTree engine (not replicated), so after a migration runs, only one shard knows the current version. Load-balanced queries hitting other shards might report the wrong version. The solution: after each successful migration, broadcast the version to all shards via parallel HTTP requests: %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#f3e5f5', 'lineColor': '#6a1b9a'}}}%% graph TD MIGRATE[\"migrate up\"] VERSION[\"Read new version\"] BROADCAST[\"Broadcast to all shards\"] S1[\"Shard 1 \u2713\"] S2[\"Shard 2 \u2713\"] S3[\"Shard 3 \u2713\"] SN[\"Shard 20 \u2713\"] MIGRATE --> VERSION --> BROADCAST BROADCAST --> S1 BROADCAST --> S2 BROADCAST --> S3 BROADCAST --> SN style MIGRATE fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style VERSION fill:#e3f2fd,stroke:#1565c0,stroke-width:2px style BROADCAST fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px style S1 fill:#c8e6c9,stroke:#1b5e20,stroke-width:1px style S2 fill:#c8e6c9,stroke:#1b5e20,stroke-width:1px style S3 fill:#c8e6c9,stroke:#1b5e20,stroke-width:1px style SN fill:#c8e6c9,stroke:#1b5e20,stroke-width:1px Enterprise ClickHouse note If you’re using ClickHouse Cloud or enterprise ClickHouse with ClickHouse Keeper, you can change the migrations table engine to ReplicatedMergeTree and delete the broadcast step entirely \u2014 replication handles it automatically. ","date":"03-Jul-2026","heading":"Multi-Shard Version Broadcast","objectID":"\/posts\/database-schema-migrations\/:9:5","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#multi-shard-version-broadcast"},{"categories":["software-engineering"],"collections":null,"content":"PostgreSQL Migration Patterns PostgreSQL is where migrations are most pleasant to work with, thanks to its unique support for transactional DDL. ","date":"03-Jul-2026","heading":"PostgreSQL Migration Patterns","objectID":"\/posts\/database-schema-migrations\/:10:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#postgresql-migration-patterns"},{"categories":["software-engineering"],"collections":null,"content":"Transactional DDL \u2014 PostgreSQL’s Superpower PostgreSQL Transactional DDL PostgreSQL is one of the few databases that supports transactional DDL \u2014 meaning CREATE TABLE, ALTER TABLE, DROP INDEX, and most other DDL statements can all run inside a transaction. If any statement fails, the entire migration rolls back atomically. You never end up in a dirty state. This is a major advantage over ClickHouse (no DDL transactions) and MySQL (implicit commits on DDL). ","date":"03-Jul-2026","heading":"Transactional DDL \u2014 PostgreSQL’s Superpower","objectID":"\/posts\/database-schema-migrations\/:10:1","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#transactional-ddl--postgresqls-superpower"},{"categories":["software-engineering"],"collections":null,"content":"Example: Users Table with Indexes -- 001_create_users_table.up.sql CREATE TABLE IF NOT EXISTS users ( user_id VARCHAR(50) PRIMARY KEY, display_name VARCHAR(255), first_name VARCHAR(100), last_name VARCHAR(100), email VARCHAR(255), country VARCHAR(5), -- ISO 3166-1 alpha-2 first_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), last_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ); CREATE INDEX IF NOT EXISTS idx_users_email ON users(email); CREATE INDEX IF NOT EXISTS idx_users_last_seen ON users(last_seen_at); COMMENT ON TABLE users IS 'Auto-populated on first JWT authentication'; COMMENT ON COLUMN users.user_id IS 'Unique identifier from JWT sub claim'; ","date":"03-Jul-2026","heading":"Example: Users Table with Indexes","objectID":"\/posts\/database-schema-migrations\/:10:2","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#example-users-table-with-indexes"},{"categories":["software-engineering"],"collections":null,"content":"Example: API Keys with Partial Indexes -- 002_create_api_keys_table.up.sql CREATE TABLE IF NOT EXISTS api_keys ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), team_name VARCHAR(100) NOT NULL, key_encrypted TEXT NOT NULL, -- AES-256-GCM encrypted key_prefix VARCHAR(8) NOT NULL, -- First 8 chars for fast lookup description TEXT, primary_owner VARCHAR(50) NOT NULL, secondary_owner VARCHAR(50), created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), expires_at TIMESTAMPTZ, -- NULL = never expires active BOOLEAN NOT NULL DEFAULT TRUE, CONSTRAINT uq_api_keys_prefix UNIQUE (key_prefix) ); -- Partial index: only index active keys (hot path for auth lookups) CREATE INDEX IF NOT EXISTS idx_api_keys_active_prefix ON api_keys(key_prefix) WHERE active = TRUE; CREATE INDEX IF NOT EXISTS idx_api_keys_team ON api_keys(team_name); CREATE INDEX IF NOT EXISTS idx_api_keys_owner ON api_keys(primary_owner); Partial indexes Partial indexes (WHERE active = TRUE) are a PostgreSQL feature that keeps the index small and fast. Only rows matching the condition are indexed \u2014 ideal for status columns where you almost always query one value. ","date":"03-Jul-2026","heading":"Example: API Keys with Partial Indexes","objectID":"\/posts\/database-schema-migrations\/:10:3","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#example-api-keys-with-partial-indexes"},{"categories":["software-engineering"],"collections":null,"content":"CREATE INDEX CONCURRENTLY For large tables in production, standard index creation locks the table. Use CREATE INDEX CONCURRENTLY to avoid this \u2014 but note it cannot run inside a transaction, so it needs its own migration file: -- 004_add_events_index_concurrently.up.sql -- NOTE: This migration cannot be wrapped in a transaction CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_events_timestamp ON events(created_at DESC); ","date":"03-Jul-2026","heading":"CREATE INDEX CONCURRENTLY","objectID":"\/posts\/database-schema-migrations\/:10:4","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#create-index-concurrently"},{"categories":["software-engineering"],"collections":null,"content":"Advisory Locks \u2014 Preventing Race Conditions In a CI\/CD environment, it’s possible for two pipelines to trigger simultaneously. If both try to run migrations at the same time, you get corrupted state. %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#e8eaf6', 'lineColor': '#283593', 'actorTextColor': '#000', 'actorBkg': '#e3f2fd', 'signalColor': '#1565c0'}}}%% sequenceDiagram participant P1 as Pipeline 1 participant DB as PostgreSQL participant P2 as Pipeline 2 rect rgb(232, 245, 233) P1->>DB: pg_advisory_lock(migrate_hash) DB-->>P1: Lock GRANTED \u2713 end rect rgb(255, 235, 238) P2->>DB: pg_advisory_lock(migrate_hash) DB-->>P2: BLOCKED (waiting...) end rect rgb(232, 245, 233) P1->>DB: Apply migration 005 P1->>DB: Apply migration 006 P1->>DB: pg_advisory_unlock() end rect rgb(227, 242, 253) DB-->>P2: Lock GRANTED (finally) P2->>DB: Check version \u2192 already 006 P2->>DB: Nothing to apply \u2713 P2->>DB: pg_advisory_unlock() end golang-migrate handles this natively for PostgreSQL using advisory locks. Only one migration instance can run at a time per database. The second one waits (or fails fast, depending on configuration). Key point: This is automatic for PostgreSQL. For ClickHouse and MySQL, you need to handle concurrency at the pipeline level (e.g., disallow concurrent builds in your CI\/CD config, or use a distributed lock via Redis \/ Consul). ","date":"03-Jul-2026","heading":"Advisory Locks \u2014 Preventing Race Conditions","objectID":"\/posts\/database-schema-migrations\/:11:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#advisory-locks--preventing-race-conditions"},{"categories":["software-engineering"],"collections":null,"content":"Dirty State Recovery This is the scenario that causes the most panic the first time you encounter it: %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#e8f5e9', 'lineColor': '#2e7d32'}}}%% stateDiagram-v2 [*] --> Clean: Normal operation Clean --> Migrating: migrate up triggered state Migrating { [*] --> Applying Applying --> Applied: SQL succeeds Applying --> Failed: SQL error } Migrating --> Clean: All migrations succeed Migrating --> Dirty: Migration fails midway state Dirty { [*] --> Blocked Blocked --> Blocked: All migrate up attempts FAIL } Dirty --> Recovery: Developer intervention Recovery --> Clean: migrate force N + fix SQL Clean --> Migrating: Re-run migrate up What happens: Migration 005 starts executing SQL statement fails (syntax error, constraint violation, missing column, etc.) The database is now in an indeterminate state \u2014 some changes applied, some didn’t golang-migrate sets dirty = true in schema_migrations All future migrate up commands refuse to run until you resolve this How to fix it: %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#fff3e0', 'lineColor': '#e65100'}}}%% graph LR A[\"1. Check state\"] B[\"2. Inspect DB\"] C[\"3. Fix SQL file\"] D[\"4. Force version\"] E[\"5. Re-run migrate up\"] A --> B --> C --> D --> E style A fill:#ffcdd2,stroke:#c62828,stroke-width:2px style B fill:#fff3e0,stroke:#e65100,stroke-width:2px style C fill:#fff9c4,stroke:#f57f17,stroke-width:2px style D fill:#e3f2fd,stroke:#1565c0,stroke-width:2px style E fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px # 1. Check current state migrate -source file:\/\/.\/migrations -database \"$URL\" version # Output: 5 (dirty) # 2-3. Fix the SQL file or manually clean up partial changes in the DB # 4. Force back to last good version migrate -source file:\/\/.\/migrations -database \"$URL\" force 4 # 5. Re-run (applies the fixed migration 005) migrate -source file:\/\/.\/migrations -database \"$URL\" up Prevention strategies PostgreSQL: Transactional DDL means a failed migration auto-rolls back. You rarely hit dirty state. ClickHouse: No DDL transactions. Keep migrations atomic \u2014 one DDL statement per file. If it fails, nothing partial applied. General: Always test migrations against a local Docker database before pushing. ","date":"03-Jul-2026","heading":"Dirty State Recovery","objectID":"\/posts\/database-schema-migrations\/:12:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#dirty-state-recovery"},{"categories":["software-engineering"],"collections":null,"content":"Zero-Downtime Migrations \u2014 The Expand-Contract Pattern This is the pattern I use when multiple services share a database and you need to change a column without downtime. It’s sometimes called “expand and contract” or “parallel change.” Scenario: Two services read\/write a full_name column. Service A now needs separate first_name and last_name columns. %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#e3f2fd', 'lineColor': '#1565c0'}}}%% timeline title Zero-Downtime Column Migration (Weeks, Not Minutes) section Phase 1 - Expand (Week 1) Migration 001 : ADD first_name, last_name (NULLABLE) Migration 002 : Backfill existing rows from full_name section Phase 2 - Transition (Week 2-3) Deploy Service A : Dual-write both old + new columns : Reads from new columns Service B unchanged : Still writes only full_name section Phase 3 - Contract (Week 4) Deploy Service B : Now uses first_name + last_name Migration 003 : Backfill NULLs from transition window Migration 004 : DROP full_name (only after ALL services migrated) %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#e8f5e9', 'lineColor': '#2e7d32'}}}%% graph TD subgraph \"Phase 1: EXPAND Schema\" M1[\"Migration 1: ADD new columns\"] M2[\"Migration 2: Backfill data\"] end subgraph \"Phase 2: TRANSITION (dual-write)\" S1[\"Deploy Service A: dual-write\"] S2[\"Service B unchanged: writes old column\"] end subgraph \"Phase 3: CONTRACT Schema\" S2B[\"Deploy Service B: uses new columns\"] M3[\"Migration 3: Backfill NULLs\"] M4[\"Migration 4: DROP old column\"] end M1 --> M2 --> S1 S1 --> S2 S2 --> S2B --> M3 --> M4 style M1 fill:#bbdefb,stroke:#0d47a1,stroke-width:2px style M2 fill:#bbdefb,stroke:#0d47a1,stroke-width:2px style S1 fill:#fff3e0,stroke:#e65100,stroke-width:2px style S2 fill:#fff9c4,stroke:#f57f17,stroke-width:2px style S2B fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px style M3 fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px style M4 fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px The mistake I made early on I once dropped a column in the same deployment that switched the service to new columns. During the rolling deployment window, old pods were still reading the dropped column. Instant 500s across the fleet. The expand-contract pattern exists specifically to avoid this \u2014 you only contract (drop old columns) after ALL consumers have migrated and been stable for days. ","date":"03-Jul-2026","heading":"Zero-Downtime Migrations \u2014 The Expand-Contract Pattern","objectID":"\/posts\/database-schema-migrations\/:13:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#zero-downtime-migrations--the-expand-contract-pattern"},{"categories":["software-engineering"],"collections":null,"content":"Forward-Only Migration Policy After a few production incidents, I adopted a strict forward-only policy: %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#ffebee', 'lineColor': '#c62828'}}}%% graph LR subgraph \"What I do\" FORWARD[\"Always move forward\"] end subgraph \"What I avoid\" ROLLBACK[\"Never roll back in prod\"] end FORWARD -.-|\"instead of\"| ROLLBACK style FORWARD fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px style ROLLBACK fill:#ffcdd2,stroke:#b71c1c,stroke-width:2px Rules I follow: I write .down.sql files for local dev\/testing only In production, I never run migrate down If a migration was wrong, I write a new forward migration that corrects it All DDL uses IF NOT EXISTS \/ IF EXISTS \/ CREATE OR REPLACE for idempotency Migrations can be re-run safely (important for ClickHouse shard broadcast scenarios) ","date":"03-Jul-2026","heading":"Forward-Only Migration Policy","objectID":"\/posts\/database-schema-migrations\/:14:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#forward-only-migration-policy"},{"categories":["software-engineering"],"collections":null,"content":"Practical Tips from Experience ","date":"03-Jul-2026","heading":"Practical Tips from Experience","objectID":"\/posts\/database-schema-migrations\/:15:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#practical-tips-from-experience"},{"categories":["software-engineering"],"collections":null,"content":"1. One Change Per Migration File Don’t pack multiple unrelated changes into one migration: 005_add_email_column.up.sql 005_add_email_column.down.sql 006_create_orders_table.up.sql 006_create_orders_table.down.sql 007_add_orders_user_index.up.sql 007_add_orders_user_index.down.sql If migration 005 has “add email column” AND “create orders table” and the orders creation fails, you can’t easily recover the email column part. ","date":"03-Jul-2026","heading":"1. One Change Per Migration File","objectID":"\/posts\/database-schema-migrations\/:15:1","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#1-one-change-per-migration-file"},{"categories":["software-engineering"],"collections":null,"content":"2. Make Migrations Idempotent -- GOOD: Idempotent (safe to re-run) CREATE TABLE IF NOT EXISTS users (...); CREATE INDEX IF NOT EXISTS idx_users_email ON users(email); DROP TABLE IF EXISTS old_events ON CLUSTER '{cluster}'; CREATE OR REPLACE VIEW analytics.summary_view AS ...; -- BAD: Non-idempotent (fails if already applied) CREATE TABLE users (...); CREATE INDEX idx_users_email ON users(email); ","date":"03-Jul-2026","heading":"2. Make Migrations Idempotent","objectID":"\/posts\/database-schema-migrations\/:15:2","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#2-make-migrations-idempotent"},{"categories":["software-engineering"],"collections":null,"content":"3. Test Against Real Databases Locally Don’t just lint your SQL. Run your migrations against real instances via Docker: # PostgreSQL docker run -d --name pg-test -e POSTGRES_PASSWORD=test -p 5433:5432 postgres:16 migrate -source file:\/\/.\/postgresql \\ -database \"postgres:\/\/postgres:test@localhost:5433\/postgres?sslmode=disable\" up # ClickHouse (single node, sufficient for DDL testing) docker run -d --name ch-test -p 9000:9000 -p 8123:8123 clickhouse\/clickhouse-server:24.3 migrate -source file:\/\/.\/clickhouse \\ -database \"clickhouse:\/\/localhost:9000\/default?x-multi-statement=true\" up ","date":"03-Jul-2026","heading":"3. Test Against Real Databases Locally","objectID":"\/posts\/database-schema-migrations\/:15:3","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#3-test-against-real-databases-locally"},{"categories":["software-engineering"],"collections":null,"content":"4. Version Numbers: Sequential Integers vs. Timestamps golang-migrate supports both. I prefer sequential integers (001, 002, 003) for smaller teams. Timestamps (20260703120000) are better when multiple developers create migrations on different branches \u2014 fewer merge conflicts. ","date":"03-Jul-2026","heading":"4. Version Numbers: Sequential Integers vs. Timestamps","objectID":"\/posts\/database-schema-migrations\/:15:4","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#4-version-numbers-sequential-integers-vs-timestamps"},{"categories":["software-engineering"],"collections":null,"content":"5. Document the “Why” in SQL Comments -- 003_add_partition_key.up.sql -- WHY: Queries by month are 50x faster with monthly partitions. -- Old data (>2 years) auto-drops via TTL without manual cleanup. ALTER TABLE analytics.events_local ON CLUSTER '{cluster}' MODIFY TTL event_time + INTERVAL 2 YEAR; ","date":"03-Jul-2026","heading":"5. Document the “Why” in SQL Comments","objectID":"\/posts\/database-schema-migrations\/:15:5","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#5-document-the-why-in-sql-comments"},{"categories":["software-engineering"],"collections":null,"content":"6. Use ON CLUSTER for EVERY DDL in ClickHouse If you forget ON CLUSTER on even one statement, you’ll have schema drift between shards that’s incredibly painful to debug. ","date":"03-Jul-2026","heading":"6. Use","objectID":"\/posts\/database-schema-migrations\/:15:6","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#6-use-on-cluster-for-every-ddl-in-clickhouse"},{"categories":["software-engineering"],"collections":null,"content":"Comparison with Other Tools Feature golang-migrate Flyway Liquibase Atlas goose Language Go (CLI + lib) Java Java\/XML Go Go Config overhead Zero (just SQL files) Medium High (XML\/YAML) Low (HCL) Low Database support 15+ 20+ 25+ 10+ 12+ ClickHouse clusters Works natively Incomplete Buggy Limited Works PostgreSQL support Excellent Excellent Excellent Excellent Excellent Dirty state handling force command repair command Manual Built-in Built-in Advisory locks Yes (PostgreSQL) Yes Yes Yes Yes Embedded in Go apps Native No No Yes Yes Docker image Yes (17MB) Yes (300MB+) Yes (400MB+) Yes Yes JVM required No Yes Yes No No Learning curve Low Medium High Medium Low ","date":"03-Jul-2026","heading":"Comparison with Other Tools","objectID":"\/posts\/database-schema-migrations\/:16:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#comparison-with-other-tools"},{"categories":["software-engineering"],"collections":null,"content":"Using golang-migrate as a Go Library Beyond the CLI, you can embed migrations directly in your Go application using Go’s embed package: package main import ( \"database\/sql\" \"embed\" \"log\" \"os\" \"github.com\/golang-migrate\/migrate\/v4\" \"github.com\/golang-migrate\/migrate\/v4\/database\/postgres\" \"github.com\/golang-migrate\/migrate\/v4\/source\/iofs\" _ \"github.com\/lib\/pq\" ) \/\/go:embed migrations\/postgresql\/*.sql var migrationFS embed.FS func main() { db, err := sql.Open(\"postgres\", os.Getenv(\"DATABASE_URL\")) if err != nil { log.Fatal(err) } driver, err := postgres.WithInstance(db, &postgres.Config{}) if err != nil { log.Fatal(err) } source, err := iofs.New(migrationFS, \"migrations\/postgresql\") if err != nil { log.Fatal(err) } m, err := migrate.NewWithInstance(\"iofs\", source, \"postgres\", driver) if err != nil { log.Fatal(err) } if err := m.Up(); err != nil && err != migrate.ErrNoChange { log.Fatal(\"migration failed: \", err) } log.Println(\"Migrations applied successfully\") } This is useful for self-migrating applications \u2014 the binary carries its own migrations embedded at compile time. Deploy a new version and it auto-migrates on startup. No separate migration step needed. ","date":"03-Jul-2026","heading":"Using golang-migrate as a Go Library","objectID":"\/posts\/database-schema-migrations\/:17:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#using-golang-migrate-as-a-go-library"},{"categories":["software-engineering"],"collections":null,"content":"The Abstraction Layer Pattern One architectural pattern I’ve found invaluable for analytics workloads: never let applications query source databases directly. Instead, build an abstraction layer via migrations: %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#e8f5e9', 'lineColor': '#2e7d32'}}}%% graph TD subgraph \"Applications (API Layer)\" APP[\"Your Service\"] end subgraph \"Abstraction Layer (Your Migrations)\" VIEWS[\"Parameterized Views\"] MERGE[\"Merge Tables\"] LOCAL[\"Owned Tables\"] end subgraph \"Source Systems (Not Yours)\" SRC1[\"Source DB 1\"] SRC2[\"Source DB 2\"] SRC3[\"Source DB N...\"] end APP --> VIEWS APP --> LOCAL VIEWS --> MERGE MERGE --> SRC1 MERGE --> SRC2 MERGE --> SRC3 style APP fill:#e3f2fd,stroke:#1565c0,stroke-width:2px style VIEWS fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px style MERGE fill:#fff3e0,stroke:#e65100,stroke-width:2px style LOCAL fill:#e8eaf6,stroke:#283593,stroke-width:2px style SRC1 fill:#f5f5f5,stroke:#757575,stroke-width:1px style SRC2 fill:#f5f5f5,stroke:#757575,stroke-width:1px style SRC3 fill:#f5f5f5,stroke:#757575,stroke-width:1px Why this matters: If an upstream team renames a column, you update ONE migration (the Merge table definition) \u2014 your views and app code stay unchanged. The parameterized views add NULL safety (ifNull(col, default)), computed fields, and deduplication logic that every consumer needs. New source databases are automatically included via the regex pattern in Merge tables. ","date":"03-Jul-2026","heading":"The Abstraction Layer Pattern","objectID":"\/posts\/database-schema-migrations\/:18:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#the-abstraction-layer-pattern"},{"categories":["software-engineering"],"collections":null,"content":"Full Pipeline \u2014 End to End %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#f5f5f5', 'lineColor': '#37474f'}}}%% graph TD subgraph \"Developer Workflow\" WRITE[\"Write SQL migration file\"] TEST[\"Test against Docker DB\"] PUSH[\"git push to dev\"] PR[\"Create PR to main\"] REVIEW[\"Code review\"] MERGE[\"Merge to main\"] end subgraph \"GitHub Actions Workflow\" TRIGGER[\"Workflow triggered\"] INSTALL[\"Install golang-migrate\"] CH_MIG[\"migrate -database $CH_URL up\"] PG_MIG[\"migrate -database $PG_URL up\"] RESULT[\"Pass \/ Fail notification\"] end subgraph \"Databases\" CH_CLUSTER[\"ClickHouse Cluster\"] PG_DB[\"PostgreSQL\"] end WRITE --> TEST --> PUSH PUSH -->|\"dev env\"| TRIGGER PUSH -.->|\"later\"| PR --> REVIEW --> MERGE MERGE -->|\"prod env\"| TRIGGER TRIGGER --> INSTALL INSTALL --> CH_MIG --> PG_MIG --> RESULT CH_MIG --> CH_CLUSTER PG_MIG --> PG_DB style WRITE fill:#e3f2fd,stroke:#1565c0,stroke-width:2px style TEST fill:#e3f2fd,stroke:#1565c0,stroke-width:2px style PUSH fill:#e3f2fd,stroke:#1565c0,stroke-width:2px style PR fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style REVIEW fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style MERGE fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style TRIGGER fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px style INSTALL fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px style CH_MIG fill:#fff3e0,stroke:#e65100,stroke-width:2px style PG_MIG fill:#e8eaf6,stroke:#283593,stroke-width:2px style RESULT fill:#e0f2f1,stroke:#004d40,stroke-width:2px style CH_CLUSTER fill:#fff3e0,stroke:#e65100,stroke-width:2px style PG_DB fill:#e8eaf6,stroke:#283593,stroke-width:2px ","date":"03-Jul-2026","heading":"Full Pipeline \u2014 End to End","objectID":"\/posts\/database-schema-migrations\/:19:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#full-pipeline--end-to-end"},{"categories":["software-engineering"],"collections":null,"content":"Key Takeaways Migrations are code. Treat them like application code \u2014 version controlled, reviewed, tested in CI. The pipeline is the only actor. No human runs migrations manually against shared environments. Advisory locks prevent races. Without them, concurrent pipelines will corrupt your migration state. Dirty state is recoverable. Don’t panic. Check the state, fix the SQL, force back, re-run. Expand-contract for shared schemas. Never drop a column that another service still reads. Never store credentials in the repo. Use your CI\/CD platform’s secrets (GitHub Secrets, Vault, etc.) to inject database URLs at runtime. One change per file. Granular migrations are easier to debug, easier to force-recover, easier to reason about. Forward-only in production. Never roll back \u2014 always write a new migration to correct mistakes. Test your migration tool against your actual database engines. What works for PostgreSQL might not work for ClickHouse clusters \u2014 I learned this the hard way with Flyway and Liquibase. Build an abstraction layer. Merge tables + parameterized views insulate your app from upstream schema changes. ","date":"03-Jul-2026","heading":"Key Takeaways","objectID":"\/posts\/database-schema-migrations\/:20:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#key-takeaways"},{"categories":["software-engineering"],"collections":null,"content":"Resources golang-migrate \u2014 18.7k stars, supports 15+ databases Migration Best Practices (golang-migrate docs) PostgreSQL \u2014 Transactional DDL makes migrations significantly safer PostgreSQL Advisory Locks PostgreSQL CREATE INDEX CONCURRENTLY \u2014 Non-blocking index creation for large tables PostgreSQL Partial Indexes Expand and Contract Pattern \u2014 Martin Fowler ClickHouse \u2014 Columnar analytics database ClickHouse Schema Migration Tools \u2014 Official comparison of migration tools for ClickHouse ClickHouse Merge Engine \u2014 Virtual tables for cross-database reads ClickHouse Distributed Engine \u2014 Sharded query routing ClickHouse Parameterized Views ClickHouse Keeper \u2014 ZooKeeper alternative for coordination Flyway \u2014 JVM-based migration tool (strong PostgreSQL\/MySQL support) Liquibase \u2014 XML\/YAML\/JSON changelog-based migrations Atlas \u2014 Declarative schema management for Go goose \u2014 Another Go migration tool (similar philosophy) GitHub Actions \u2014 CI\/CD platform for automating migration workflows GitHub Actions Secrets \u2014 Encrypted secret storage for database credentials HashiCorp Vault \u2014 Dynamic secrets management for enterprise teams Docker \u2014 Container runtime for testing migrations locally ","date":"03-Jul-2026","heading":"Resources","objectID":"\/posts\/database-schema-migrations\/:21:0","tags":["microservices","database","migration"],"title":"Database Schema Migrations with golang-migrate \u2014 What Actually Worked for Me","uri":"\/posts\/database-schema-migrations\/#resources"},{"categories":["software-engineering"],"collections":null,"content":"Hands-on comparison notes between the Grafana observability stack (Grafana, Loki, Mimir, Tempo, Alloy) and Elastic (Elasticsearch, Kibana, Elastic Agent) for self-hosted on-premises deployments. Covers dashboards, alerting, GitOps, log management, LLM integration, resource footprint, and cost.","date":"01-Apr-2026","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/"},{"categories":["software-engineering"],"collections":null,"content":"I needed to pick an observability platform for a self-hosted environment. Mix of modern and legacy hardware, on-prem CI\/CD, and a future requirement for self-hosted LLM integration. The two real contenders were the Grafana stack (Grafana + Loki + Mimir + Tempo + Alloy + Pyroscope) and the Elastic stack (Elasticsearch + Kibana + Elastic Agent). I ran both side by side for a few months. These are my notes. Scope Everything here is about self-managed, on-premises deployments. If you’re using Elastic Cloud, the cost and ops picture changes a lot \u2014 managed services take care of scaling, patching, and node management for you. ","date":"01-Apr-2026","heading":"","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:0:0","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#"},{"categories":["software-engineering"],"collections":null,"content":"Dashboards Grafana wins this one easily. The killer feature is multi-datasource dashboards \u2014 one panel pulling from Prometheus, another from Loki, another from Elasticsearch or a random REST API, all on the same dashboard. Kibana only talks to Elasticsearch. If you want a single pane of glass across multiple backends, Grafana is it. The chart library is also wider. Heatmaps, flame graphs (huge for profiling), geo maps, canvas panels (drag-and-drop layout), node graphs for topology views. Kibana has improved over the years, but it still feels more limited for custom visualization. The other big deal: Git Sync landed as GA in Grafana v13.0. The workflow is: edit a dashboard in the browser, hit save, it commits to a Git branch. Open a PR, get it reviewed, merge \u2014 Grafana auto-applies. Your entire dashboard history lives in Git with full diff capability. Roll back a bad change with git revert. Kibana has no equivalent \u2014 dashboards live in an internal Elasticsearch index with no version history you can meaningfully browse. Grafana Kibana Multi-datasource Yes \u2014 any backend No \u2014 Elasticsearch only Git Sync (bidirectional) GA in v13.0 None Community templates Thousands on grafana.com Limited Chart variety Flame graphs, geo maps, heatmaps, canvas, node graphs Standard charts Embeddable Yes, via iframes or public dashboards Limited ","date":"01-Apr-2026","heading":"Dashboards","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:1:0","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#dashboards"},{"categories":["software-engineering"],"collections":null,"content":"Telemetry Collection and Real-Time Monitoring Tie on both fronts. Both support OTLP natively, so you can use a single standard for logs, metrics, and traces regardless of which backend stores them. Prometheus + Mimir handles metrics on the Grafana side \u2014 Prometheus for scraping and short-term, Mimir for horizontal scaling and long-term retention. Elastic has TSDB mode since v8.7 for efficient time-series storage. For distributed tracing, Grafana has Tempo (stores traces cheaply in object storage), Elastic has its APM (mature, integrated with the rest of the Elastic stack). Both are solid \u2014 pick whichever query language your team already knows. If your devs think in PromQL, go Grafana. If they’re deep in Elasticsearch query DSL already, Elastic APM will feel natural. The real difference is in agent resource footprint: Alloy (Grafana): Single Go binary. Scrapes Prometheus endpoints, tails log files, receives OTLP, forwards everything. In my testing, idle memory sat around 50-100 MB. Elastic Agent: Java-based. Typically 512 MB to 1 GB+ idle. Same story for Logstash if you’re using it in your pipeline. On a server with 4 GB total RAM, one Logstash instance eats 25% of your memory before it processes a single log line. Alloy barely registers. On modern hardware with 32+ GB, the gap doesn’t matter. But if you’re deploying collectors on older machines (and in an on-prem environment, you almost certainly are), this is worth thinking about. ","date":"01-Apr-2026","heading":"Telemetry Collection and Real-Time Monitoring","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:2:0","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#telemetry-collection-and-real-time-monitoring"},{"categories":["software-engineering"],"collections":null,"content":"Log Management \u2014 The Tricky One This is where the two stacks make fundamentally different architectural bets. Neither is wrong \u2014 they just optimize for different things. ","date":"01-Apr-2026","heading":"Log Management \u2014 The Tricky One","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:3:0","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#log-management--the-tricky-one"},{"categories":["software-engineering"],"collections":null,"content":"Loki’s Approach Loki indexes by labels only \u2014 things like {host="web-01", app="nginx", env="prod"}. The actual log content is stored as compressed chunks but never indexed. That means: Storage overhead in my experience is roughly 0.5-1x raw volume (often smaller than raw because of compression) Queries on labels are lightning fast But searching inside log content is basically grep \u2014 |= "ERROR" works, but complex full-text queries across millions of lines get slow ","date":"01-Apr-2026","heading":"Loki’s Approach","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:3:1","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#lokis-approach"},{"categories":["software-engineering"],"collections":null,"content":"Elasticsearch’s Approach Elasticsearch builds a full inverted index on every field of every log line. Every token is searchable. That means: Best-in-class full-text search \u2014 complex regex, aggregations, field-level queries all instant But roughly 3-6x storage overhead from the indexing (varies with mapping and replica config) 8-16 GB JVM heap minimum for a production cluster Expensive to scale if you have high log volume ","date":"01-Apr-2026","heading":"Elasticsearch’s Approach","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:3:2","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#elasticsearchs-approach"},{"categories":["software-engineering"],"collections":null,"content":"What I Actually Did I ended up running both. Loki handles day-to-day ops \u2014 dashboards showing recent logs by service, alerting on error patterns, tailing live logs. Elasticsearch sits behind Grafana as a datasource for when I actually need to do forensic deep-dive searches \u2014 “find every request with this specific trace ID across all services from last Tuesday.” Grafana queries Elasticsearch natively, so you don’t need Kibana at all. One UI, two log backends, each playing to its strength. graph TD G[Grafana UI - Single Pane] --> L[Loki] G --> E[Elasticsearch] L -.- D[Daily Ops: dashboards, alerts, log tails] E -.- F[Forensic Search: full-text, deep-dive queries] ","date":"01-Apr-2026","heading":"What I Actually Did","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:3:3","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#what-i-actually-did"},{"categories":["software-engineering"],"collections":null,"content":"Alerting Grafana wins. The whole alerting system is provisionable as YAML or Terraform \u2014 alert rules, contact points, notification policies, silence rules. All of it. Drop a YAML file, hit the API, changes apply immediately. No restart. What makes this different from Elastic: Grafana Elastic Config as YAML\/Terraform Yes No Hot-reload (no restart) Yes No \u2014 restart needed Multi-datasource rules Yes No \u2014 Elasticsearch data only Alertmanager-compatible routing Yes No Native Webex webhook Yes No The multi-datasource alerting is the one that really changed things for me. You can write a single alert rule that says: “fire if CPU from Prometheus is above 90% for 5 minutes AND the word OOM appears in Loki logs from the same host.” Cross-signal correlation in one rule. Kibana can only alert on Elasticsearch data. ","date":"01-Apr-2026","heading":"Alerting","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:4:0","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#alerting"},{"categories":["software-engineering"],"collections":null,"content":"GitOps This is where Grafana’s lead is biggest. Here’s what you can store as code and manage through Git: Dashboards \u2014 JSON files, provisionable via Git Sync, ConfigMap, or Terraform Datasources \u2014 YAML files dropped into a provisioning directory Alert rules \u2014 YAML, same directory Contact points and routing \u2014 YAML Folders, teams, permissions \u2014 via the Terraform provider Everything is text. You can grep it, diff it, put it in a PR, run linting in CI. Standard software engineering workflow applied to your monitoring config. Kibana? Saved objects export exists, but it’s a manual JSON dump. The format is proprietary \u2014 good luck reading a Kibana dashboard export and figuring out what changed between two versions. There’s no community tooling ecosystem around it. No Terraform provider for Kibana dashboards. No Git sync. If you treat infrastructure-as-code as non-negotiable, the choice is clear. Lock-in Signal If you can’t git diff your monitoring config, you can’t review it, test it in staging, or roll it back safely. That’s a red flag for any production system. ","date":"01-Apr-2026","heading":"GitOps","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:5:0","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#gitops"},{"categories":["software-engineering"],"collections":null,"content":"ML and Anomaly Detection Elastic has built-in ML \u2014 anomaly detection on time-series data, log categorization (auto-grouping similar log messages), forecasting, and root cause analysis. It works out of the box, no external dependencies. The downside: ML jobs run on dedicated ML nodes that need Platinum license, plus separate servers with 8-16 GB RAM each. Those ML nodes don’t share resources with your data nodes. Grafana has no built-in ML at all. But if you already run MLflow (pretty common in teams doing any model tracking or experiment management), you can point Grafana’s HTTP datasource at MLflow’s REST API. Query experiment metrics, model accuracy over time, prediction outputs \u2014 all visualized in Grafana dashboards alongside your infrastructure metrics. No extra subscription, no dedicated nodes. Bottom line: if you already have MLflow or another ML platform in your stack, Grafana gives you unified visibility without additional cost. If you have nothing and want anomaly detection with zero setup, Elastic’s built-in stuff works well \u2014 just factor in the node cost and licensing. ","date":"01-Apr-2026","heading":"ML and Anomaly Detection","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:6:0","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#ml-and-anomaly-detection"},{"categories":["software-engineering"],"collections":null,"content":"On-Prem LLM Integration Grafana wins here for a simple reason: no license gating. The HTTP datasource queries any endpoint that speaks HTTP. Point it at Ollama, vLLM, a local OpenAI-compatible API, or a routing proxy like LiteLLM. Build whatever workflow you want \u2014 summarize alerts with an LLM, auto-generate runbooks from incident patterns, query knowledge bases. All in the OSS tier. Elastic’s AI features have been in flux. Their managed AI capabilities require Cloud Connect (an Elastic Cloud account), which defeats the purpose of an air-gapped on-prem setup. If you can’t phone home to Elastic’s cloud, those features aren’t available to you. Check their current docs for the latest on what’s available without cloud connectivity \u2014 this space is moving fast. ","date":"01-Apr-2026","heading":"On-Prem LLM Integration","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:7:0","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#on-prem-llm-integration"},{"categories":["software-engineering"],"collections":null,"content":"Automated Remediation The pattern that works well with Grafana for closing the “detect \u2192 act” loop: graph LR A[Alert Fires] --> B[Grafana Webhook] B --> C[Agent\/Script Diagnoses] C --> D[POST to Annotations API] D --> E[Annotation on Dashboard] Per-alert-policy webhooks fire to any HTTP endpoint. The agent can be a Python script, an n8n workflow, a custom FastAPI service \u2014 whatever you want. Results post back as annotations so your team can see what happened and when, right on the dashboard that triggered the alert. For the SSH execution layer (actually running remediation commands on servers), I looked at four tools: Tool Runtime Grafana integration Best for Rundeck Java (~512 MB) Native webhook receiver Ops teams, ad hoc tasks, simple runbooks Ansible + Semaphore Go (~100-200 MB) API trigger via webhook Complex multi-step playbooks StackStorm Multiple services (~4-8 GB) Native webhook sensor Event-driven automation at scale SaltStack Agent-based Manual integration Very large fleets (1000+ nodes) Rundeck is the easiest to get running. Single process, Apache 2.0 license, built-in web UI for building runbooks. For most teams starting out, this is the right answer. If your automation outgrows it (complex multi-host playbooks, conditional logic, rolling updates), Ansible + Semaphore UI gives you the richest module ecosystem with a lightweight Go-based web frontend. StackStorm is overkill unless you specifically need its event-rule engine. SaltStack’s community has been declining since the VMware acquisition \u2014 I wouldn’t start a new project with it today. ","date":"01-Apr-2026","heading":"Automated Remediation","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:8:0","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#automated-remediation"},{"categories":["software-engineering"],"collections":null,"content":"Resource Footprint The numbers that convinced me. For 100 GB\/day of log ingestion: Metric Loki Elasticsearch Storage per day 50-100 GB 300-600 GB Storage per year 18-37 TB 109-219 TB Indexing approach Label-only (compressed chunks) Full inverted index At one year retention, you’re looking at roughly 3-6x more disk on the Elastic side depending on your config. On commodity NVMe storage at ~$100\/TB, that’s a real budget line item. For the full stack RAM comparison: Component Grafana Stack Elastic Stack Metrics (Mimir) 2-4 GB \u2014 Logs (Loki) 2-4 GB \u2014 Traces (Tempo) 1-2 GB \u2014 Dashboard (Grafana) 512 MB-1 GB \u2014 Search + Storage (Elasticsearch) \u2014 8-16 GB JVM heap Dashboard (Kibana) \u2014 1-2 GB Total ~6-11 GB ~9-18 GB The RAM difference is smaller than the storage difference, but Elasticsearch’s JVM heap is fixed allocation \u2014 you’re paying that 8-16 GB whether you’re querying or idle. Loki and Mimir scale more linearly with actual load. ","date":"01-Apr-2026","heading":"Resource Footprint","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:9:0","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#resource-footprint"},{"categories":["software-engineering"],"collections":null,"content":"Pricing The core Grafana OSS stack (Grafana, Loki, Mimir, Tempo) is AGPLv3 \u2014 free. Alloy is Apache 2.0. All of it. Git Sync, multi-datasource alerting, Terraform provisioning, OTLP collection, distributed tracing, profiling. Enterprise only gates two things: SAML\/OIDC SSO and fine-grained folder-level RBAC. Elastic’s licensing is more layered: Feature Required License Core search and analytics Free (Basic) ML \/ Anomaly detection Platinum AIOps \/ AI Assistant Enterprise SAML SSO Platinum Field-level security Platinum Dedicated ML nodes Platinum (plus the server hardware) So if you need anomaly detection on Elastic, the real cost is: Platinum subscription + new servers for ML nodes (8-16 GB RAM each, separate from data nodes). On Grafana’s side, the same outcome costs: MLflow (free) + HTTP datasource (free) + your existing compute. ","date":"01-Apr-2026","heading":"Pricing","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:10:0","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#pricing"},{"categories":["software-engineering"],"collections":null,"content":"Vendor Lock-in This is the one people underestimate until they need to migrate. Grafana exit path: Run git clone on your dashboard\/alerting repos. Done. All dashboards are JSON, all config is YAML Downgrading from Enterprise to OSS only loses SAML SSO Your data stays in Prometheus\/Mimir\/Loki \u2014 none of which are Grafana-proprietary Elastic exit path: Kibana dashboards are stored in an internal .kibana index. The format is proprietary and deeply nested. No Git sync means no history, no diff, no audit trail Dropping from Platinum to Basic loses ML, AIOps, SAML SSO, and field-level security Rebuilding your Kibana dashboards in another tool means starting from scratch \u2014 there’s no export format that other tools understand That asymmetry is real. Grafana’s entire configuration surface is text files in version control. Elastic’s is locked in a database you can’t meaningfully read or port. ","date":"01-Apr-2026","heading":"Vendor Lock-in","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:11:0","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#vendor-lock-in"},{"categories":["software-engineering"],"collections":null,"content":"Scorecard Dimension Winner Dashboards Grafana Telemetry Collection Tie Log Search (full-text) Elastic Log Storage Efficiency Grafana Real-Time Monitoring Tie Alerting Grafana GitOps Grafana Built-in ML Elastic MLflow Integration Grafana On-Prem LLM Grafana Automation Grafana Resource Efficiency Grafana Pricing Grafana Vendor Lock-in Grafana Grafana 10 \u2014 Elastic 2 \u2014 Tie 2 ","date":"01-Apr-2026","heading":"Scorecard","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:12:0","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#scorecard"},{"categories":["software-engineering"],"collections":null,"content":"What I Went With Primary: Grafana + Prometheus + Mimir (metrics), Loki (logs), Tempo (traces), Alloy (collection), Pyroscope (profiling). Hybrid: Elasticsearch as a Grafana datasource for full-text log search. No Kibana. Automation: Rundeck for runbook automation, with a path to Ansible + Semaphore if the playbooks get complex enough. Elastic’s full-text search is still the best in the industry. I just don’t think it should be your primary observability UI anymore. Use it as a specialized backend that Grafana queries \u2014 you get the best of both worlds without the lock-in, the JVM overhead as your primary stack, or the Kibana maintenance burden. Disclaimer For educational purposes. Based on my own experience and publicly available docs. I have no affiliation with Grafana Labs, Elastic, or any vendor mentioned here. Your mileage may vary. ","date":"01-Apr-2026","heading":"What I Went With","objectID":"\/posts\/grafana-stack-vs-elastic-observability\/:13:0","tags":["observability","grafana","elastic","devops","monitoring"],"title":"Grafana Stack vs. Elastic Observability \u2014 Notes from Hands-On Experience","uri":"\/posts\/grafana-stack-vs-elastic-observability\/#what-i-went-with"},{"categories":["AI-ML"],"collections":null,"content":"Comprehensive guide to vector databases, RAG (Retrieval-Augmented Generation), embeddings, semantic search, hybrid search, and building production LLM applications with external knowledge.","date":"10-Dec-2025","objectID":"\/posts\/vector-databases-rag-llm\/","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/"},{"categories":["AI-ML"],"collections":null,"content":"A Practical Guide to Building Knowledge-Enhanced AI Applications ","date":"10-Dec-2025","heading":"","objectID":"\/posts\/vector-databases-rag-llm\/:0:0","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#"},{"categories":["AI-ML"],"collections":null,"content":"1. Introduction to Vector Databases\r","date":"10-Dec-2025","heading":"1. Introduction to Vector Databases","objectID":"\/posts\/vector-databases-rag-llm\/:1:0","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#introduction"},{"categories":["AI-ML"],"collections":null,"content":"What is a Vector Database? A vector database is a specialized database designed to store and query high-dimensional vector embeddings efficiently. Unlike traditional databases that excel at exact matches (“find users named John”), vector databases excel at similarity searches (“find documents semantically similar to this query”). Info Why This Matters: Vector databases are the backbone of modern AI applications\u2014enabling semantic search, recommendation systems, and knowledge-augmented LLMs. Key Characteristics: Similarity Search: Find the most similar items based on vector distance Scalability: Handle billions of vectors with sub-second query times AI-Native: Optimized for machine learning pipelines ","date":"10-Dec-2025","heading":"What is a Vector Database?","objectID":"\/posts\/vector-databases-rag-llm\/:1:1","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#what-is-a-vector-database"},{"categories":["AI-ML"],"collections":null,"content":"Why LLMs Need Vector Databases Large Language Models (GPT-4, Claude, Llama) have critical limitations that vector databases solve: LLM Limitation Vector DB Solution Knowledge Cutoff Store and retrieve current information Hallucinations Ground responses in factual data No Long-Term Memory Persist context across sessions Token Limits Retrieve only relevant information Lacks Domain Knowledge Inject specialized expertise ","date":"10-Dec-2025","heading":"Why LLMs Need Vector Databases","objectID":"\/posts\/vector-databases-rag-llm\/:1:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#why-llms-need-vector-databases"},{"categories":["AI-ML"],"collections":null,"content":"Real-World Applications Vector databases power many AI applications you use daily: Customer Support Bots: Retrieve relevant help articles to answer user questions Enterprise Search: Find documents by meaning, not just keywords Code Assistants: Search codebases for similar implementations Research Tools: Find related papers and citations Recommendation Systems: “Users who liked X also liked Y” Chatbots with Memory: Remember past conversations ","date":"10-Dec-2025","heading":"Real-World Applications","objectID":"\/posts\/vector-databases-rag-llm\/:1:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#real-world-applications"},{"categories":["AI-ML"],"collections":null,"content":"2. Understanding Embeddings\r","date":"10-Dec-2025","heading":"2. Understanding Embeddings","objectID":"\/posts\/vector-databases-rag-llm\/:2:0","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#embeddings"},{"categories":["AI-ML"],"collections":null,"content":"What are Embeddings? Embeddings are numerical representations of data (text, images, etc.) as dense vectors of floating-point numbers. These vectors capture semantic meaning in a way machines can process and compare. Example: The sentence “The quick brown fox” becomes a vector like: [0.0123, -0.0456, 0.0789, -0.0234, 0.0567, ..., 0.0891] (1536 dimensions) ","date":"10-Dec-2025","heading":"What are Embeddings?","objectID":"\/posts\/vector-databases-rag-llm\/:2:1","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#what-are-embeddings"},{"categories":["AI-ML"],"collections":null,"content":"How Embeddings Are Created Neural networks (specifically transformers) create embeddings through a training process: Input Processing: Text is broken into tokens (words or subwords) Contextual Understanding: The transformer processes all tokens together, understanding relationships Vector Output: The final layer produces a fixed-size vector representing the entire input Training Method - Contrastive Learning: The model sees pairs of similar texts (e.g., question and its answer) It learns to place similar texts close together in vector space Dissimilar texts are pushed apart After training on millions of pairs, the model understands semantic relationships Note You don’t need to train your own embedding model. Pre-trained models like OpenAI’s text-embedding-3-small already understand language well. You just use them via API. ","date":"10-Dec-2025","heading":"How Embeddings Are Created","objectID":"\/posts\/vector-databases-rag-llm\/:2:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#how-embeddings-are-created"},{"categories":["AI-ML"],"collections":null,"content":"Why Similar Meanings Cluster Together The training process creates a semantic space where: “Dog” and “Puppy” \u2192 Close together (similar meaning) “Dog” and “Airplane” \u2192 Far apart (unrelated) “King - Man + Woman \u2248 Queen” \u2192 Vector arithmetic captures relationships This is why searching for “automobile” can find documents about “cars”\u2014they occupy nearby regions in the embedding space. ","date":"10-Dec-2025","heading":"Why Similar Meanings Cluster Together","objectID":"\/posts\/vector-databases-rag-llm\/:2:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#why-similar-meanings-cluster-together"},{"categories":["AI-ML"],"collections":null,"content":"Semantic Space Visualization When text is converted to embeddings, semantically similar items cluster together in vector space: Semantic embedding visualization showing word clustering based on meaning Key Insight: A query for “Kitten” naturally finds related animal terms, not fruits\u2014because they’re close in vector space. ","date":"10-Dec-2025","heading":"Semantic Space Visualization","objectID":"\/posts\/vector-databases-rag-llm\/:2:4","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#semantic-space-visualization"},{"categories":["AI-ML"],"collections":null,"content":"Dense vs Sparse Embeddings Type Description Example Dense Every dimension has a value; captures semantic meaning [0.01, 0.74, 0.52, ...] Sparse Most dimensions are zero; captures keyword presence {"cat": 4, "dog": 1} Note Modern systems combine both in hybrid search for semantic understanding AND keyword precision. ","date":"10-Dec-2025","heading":"Dense vs Sparse Embeddings","objectID":"\/posts\/vector-databases-rag-llm\/:2:5","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#dense-vs-sparse-embeddings"},{"categories":["AI-ML"],"collections":null,"content":"Dimension Trade-offs More dimensions capture more nuance, but at a cost: Dimensions Speed Accuracy Memory Best For 384 Fastest Good ~1.5 KB\/vector Real-time apps, chatbots 768 Fast Better ~3 KB\/vector Balanced performance 1536 Medium Excellent ~6 KB\/vector Most production use cases 3072 Slower Best ~12 KB\/vector Maximum accuracy needed Rule of Thumb: Start with 1536 dimensions (OpenAI’s default). Only go larger if accuracy is critical and you have the infrastructure. ","date":"10-Dec-2025","heading":"Dimension Trade-offs","objectID":"\/posts\/vector-databases-rag-llm\/:2:6","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#dimension-trade-offs"},{"categories":["AI-ML"],"collections":null,"content":"Popular Embedding Models Model Dimensions Provider Best For text-embedding-3-small 1536 OpenAI Cost-effective general use text-embedding-3-large 3072 OpenAI Maximum accuracy all-MiniLM-L6-v2 384 HuggingFace Fast, open-source BGE-large-en-v1.5 1024 BAAI Top open-source performance embed-multilingual-v3.0 1024 Cohere 100+ languages ","date":"10-Dec-2025","heading":"Popular Embedding Models","objectID":"\/posts\/vector-databases-rag-llm\/:2:7","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#popular-embedding-models"},{"categories":["AI-ML"],"collections":null,"content":"3. Vector Search and Similarity\r","date":"10-Dec-2025","heading":"3. Vector Search and Similarity","objectID":"\/posts\/vector-databases-rag-llm\/:3:0","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#vector-search"},{"categories":["AI-ML"],"collections":null,"content":"Distance Metrics To find similar vectors, we measure “distance” between them: Cosine Similarity (most common for text): \ud835\udc2c\ud835\udc22\ud835\udc26\ud835\udc22\ud835\udc25\ud835\udc1a\ud835\udc2b\ud835\udc22\ud835\udc2d\ud835\udc32 = (\ud835\udc00 \u22c5 \ud835\udc01) \u00f7 (\u2016\ud835\udc00\u2016 \u2a2f \u2016\ud835\udc01\u2016) Score of 1.0 = Identical meaning Score of 0.0 = Completely unrelated Score of -1.0 = Opposite meaning Metric Best For Cosine Text embeddings Euclidean (L2) Image embeddings Dot Product Normalized vectors ","date":"10-Dec-2025","heading":"Distance Metrics","objectID":"\/posts\/vector-databases-rag-llm\/:3:1","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#distance-metrics"},{"categories":["AI-ML"],"collections":null,"content":"Similarity Scoring Example Query: “Is Windows 8 any good?” Semantic similarity scoring demonstration Warning Pitfall: “Windows 10 is good” scores 0.88\u2014as high as actual Windows 8 content! Embeddings treat version numbers as semantically similar. This is why hybrid search matters. ","date":"10-Dec-2025","heading":"Similarity Scoring Example","objectID":"\/posts\/vector-databases-rag-llm\/:3:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#similarity-scoring-example"},{"categories":["AI-ML"],"collections":null,"content":"Vector Indexing Algorithms Searching billions of vectors requires Approximate Nearest Neighbor (ANN) algorithms. Here’s how the main ones work: HNSW (Hierarchical Navigable Small World) The most popular algorithm for production systems. Think of it as a multi-level express train system: How it works: Creates multiple layers of connected nodes (vectors) Top layer: Few nodes, long-distance connections (express trains) Bottom layer: All nodes, short-distance connections (local stops) Search: Start at top, quickly narrow down, then refine at bottom Key Parameters: M: Connections per node (higher = better recall, more memory) ef: Search width (higher = more accurate, slower) Trade-off: Excellent speed and accuracy, but uses more memory than other methods. IVF (Inverted File Index) Uses clustering to organize vectors into buckets: How it works: Training: Use k-means to create cluster centers (centroids) Indexing: Assign each vector to its nearest centroid Search: Find nearest centroids to query, then search only those clusters Key Parameter: nprobe: How many clusters to search (higher = more accurate, slower) Trade-off: Fast for huge datasets, but requires training data and careful tuning. PQ (Product Quantization) Compression technique that shrinks vectors dramatically: How it works: Split each vector into sub-vectors (e.g., 1536D \u2192 8 chunks of 192D) Replace each sub-vector with a code pointing to a codebook entry Store only the codes (8 bytes instead of 6 KB!) Trade-off: Massive memory savings (32-64x), but loses some accuracy. Algorithm Comparison Algorithm How It Works Search Speed Memory Accuracy Best For HNSW Graph navigation O(log N) High Excellent Production systems IVF Cluster search O(\u221aN) Medium Good Large-scale search PQ Compressed codes O(N) Very Low Moderate Billions of vectors IVF-PQ Clusters + compression O(\u221aN) Low Good Balance of all factors Tip Most production systems use HNSW (Pinecone, Weaviate, Qdrant). It offers the best balance of speed and accuracy. Use IVF-PQ only when you have billions of vectors and limited memory. ","date":"10-Dec-2025","heading":"Vector Indexing Algorithms","objectID":"\/posts\/vector-databases-rag-llm\/:3:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#vector-indexing-algorithms"},{"categories":["AI-ML"],"collections":null,"content":"Vector Indexing Algorithms Searching billions of vectors requires Approximate Nearest Neighbor (ANN) algorithms. Here’s how the main ones work: HNSW (Hierarchical Navigable Small World) The most popular algorithm for production systems. Think of it as a multi-level express train system: How it works: Creates multiple layers of connected nodes (vectors) Top layer: Few nodes, long-distance connections (express trains) Bottom layer: All nodes, short-distance connections (local stops) Search: Start at top, quickly narrow down, then refine at bottom Key Parameters: M: Connections per node (higher = better recall, more memory) ef: Search width (higher = more accurate, slower) Trade-off: Excellent speed and accuracy, but uses more memory than other methods. IVF (Inverted File Index) Uses clustering to organize vectors into buckets: How it works: Training: Use k-means to create cluster centers (centroids) Indexing: Assign each vector to its nearest centroid Search: Find nearest centroids to query, then search only those clusters Key Parameter: nprobe: How many clusters to search (higher = more accurate, slower) Trade-off: Fast for huge datasets, but requires training data and careful tuning. PQ (Product Quantization) Compression technique that shrinks vectors dramatically: How it works: Split each vector into sub-vectors (e.g., 1536D \u2192 8 chunks of 192D) Replace each sub-vector with a code pointing to a codebook entry Store only the codes (8 bytes instead of 6 KB!) Trade-off: Massive memory savings (32-64x), but loses some accuracy. Algorithm Comparison Algorithm How It Works Search Speed Memory Accuracy Best For HNSW Graph navigation O(log N) High Excellent Production systems IVF Cluster search O(\u221aN) Medium Good Large-scale search PQ Compressed codes O(N) Very Low Moderate Billions of vectors IVF-PQ Clusters + compression O(\u221aN) Low Good Balance of all factors Tip Most production systems use HNSW (Pinecone, Weaviate, Qdrant). It offers the best balance of speed and accuracy. Use IVF-PQ only when you have billions of vectors and limited memory. ","date":"10-Dec-2025","heading":"HNSW (Hierarchical Navigable Small World)","objectID":"\/posts\/vector-databases-rag-llm\/:3:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#hnsw-hierarchical-navigable-small-world"},{"categories":["AI-ML"],"collections":null,"content":"Vector Indexing Algorithms Searching billions of vectors requires Approximate Nearest Neighbor (ANN) algorithms. Here’s how the main ones work: HNSW (Hierarchical Navigable Small World) The most popular algorithm for production systems. Think of it as a multi-level express train system: How it works: Creates multiple layers of connected nodes (vectors) Top layer: Few nodes, long-distance connections (express trains) Bottom layer: All nodes, short-distance connections (local stops) Search: Start at top, quickly narrow down, then refine at bottom Key Parameters: M: Connections per node (higher = better recall, more memory) ef: Search width (higher = more accurate, slower) Trade-off: Excellent speed and accuracy, but uses more memory than other methods. IVF (Inverted File Index) Uses clustering to organize vectors into buckets: How it works: Training: Use k-means to create cluster centers (centroids) Indexing: Assign each vector to its nearest centroid Search: Find nearest centroids to query, then search only those clusters Key Parameter: nprobe: How many clusters to search (higher = more accurate, slower) Trade-off: Fast for huge datasets, but requires training data and careful tuning. PQ (Product Quantization) Compression technique that shrinks vectors dramatically: How it works: Split each vector into sub-vectors (e.g., 1536D \u2192 8 chunks of 192D) Replace each sub-vector with a code pointing to a codebook entry Store only the codes (8 bytes instead of 6 KB!) Trade-off: Massive memory savings (32-64x), but loses some accuracy. Algorithm Comparison Algorithm How It Works Search Speed Memory Accuracy Best For HNSW Graph navigation O(log N) High Excellent Production systems IVF Cluster search O(\u221aN) Medium Good Large-scale search PQ Compressed codes O(N) Very Low Moderate Billions of vectors IVF-PQ Clusters + compression O(\u221aN) Low Good Balance of all factors Tip Most production systems use HNSW (Pinecone, Weaviate, Qdrant). It offers the best balance of speed and accuracy. Use IVF-PQ only when you have billions of vectors and limited memory. ","date":"10-Dec-2025","heading":"IVF (Inverted File Index)","objectID":"\/posts\/vector-databases-rag-llm\/:3:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#ivf-inverted-file-index"},{"categories":["AI-ML"],"collections":null,"content":"Vector Indexing Algorithms Searching billions of vectors requires Approximate Nearest Neighbor (ANN) algorithms. Here’s how the main ones work: HNSW (Hierarchical Navigable Small World) The most popular algorithm for production systems. Think of it as a multi-level express train system: How it works: Creates multiple layers of connected nodes (vectors) Top layer: Few nodes, long-distance connections (express trains) Bottom layer: All nodes, short-distance connections (local stops) Search: Start at top, quickly narrow down, then refine at bottom Key Parameters: M: Connections per node (higher = better recall, more memory) ef: Search width (higher = more accurate, slower) Trade-off: Excellent speed and accuracy, but uses more memory than other methods. IVF (Inverted File Index) Uses clustering to organize vectors into buckets: How it works: Training: Use k-means to create cluster centers (centroids) Indexing: Assign each vector to its nearest centroid Search: Find nearest centroids to query, then search only those clusters Key Parameter: nprobe: How many clusters to search (higher = more accurate, slower) Trade-off: Fast for huge datasets, but requires training data and careful tuning. PQ (Product Quantization) Compression technique that shrinks vectors dramatically: How it works: Split each vector into sub-vectors (e.g., 1536D \u2192 8 chunks of 192D) Replace each sub-vector with a code pointing to a codebook entry Store only the codes (8 bytes instead of 6 KB!) Trade-off: Massive memory savings (32-64x), but loses some accuracy. Algorithm Comparison Algorithm How It Works Search Speed Memory Accuracy Best For HNSW Graph navigation O(log N) High Excellent Production systems IVF Cluster search O(\u221aN) Medium Good Large-scale search PQ Compressed codes O(N) Very Low Moderate Billions of vectors IVF-PQ Clusters + compression O(\u221aN) Low Good Balance of all factors Tip Most production systems use HNSW (Pinecone, Weaviate, Qdrant). It offers the best balance of speed and accuracy. Use IVF-PQ only when you have billions of vectors and limited memory. ","date":"10-Dec-2025","heading":"PQ (Product Quantization)","objectID":"\/posts\/vector-databases-rag-llm\/:3:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#pq-product-quantization"},{"categories":["AI-ML"],"collections":null,"content":"Vector Indexing Algorithms Searching billions of vectors requires Approximate Nearest Neighbor (ANN) algorithms. Here’s how the main ones work: HNSW (Hierarchical Navigable Small World) The most popular algorithm for production systems. Think of it as a multi-level express train system: How it works: Creates multiple layers of connected nodes (vectors) Top layer: Few nodes, long-distance connections (express trains) Bottom layer: All nodes, short-distance connections (local stops) Search: Start at top, quickly narrow down, then refine at bottom Key Parameters: M: Connections per node (higher = better recall, more memory) ef: Search width (higher = more accurate, slower) Trade-off: Excellent speed and accuracy, but uses more memory than other methods. IVF (Inverted File Index) Uses clustering to organize vectors into buckets: How it works: Training: Use k-means to create cluster centers (centroids) Indexing: Assign each vector to its nearest centroid Search: Find nearest centroids to query, then search only those clusters Key Parameter: nprobe: How many clusters to search (higher = more accurate, slower) Trade-off: Fast for huge datasets, but requires training data and careful tuning. PQ (Product Quantization) Compression technique that shrinks vectors dramatically: How it works: Split each vector into sub-vectors (e.g., 1536D \u2192 8 chunks of 192D) Replace each sub-vector with a code pointing to a codebook entry Store only the codes (8 bytes instead of 6 KB!) Trade-off: Massive memory savings (32-64x), but loses some accuracy. Algorithm Comparison Algorithm How It Works Search Speed Memory Accuracy Best For HNSW Graph navigation O(log N) High Excellent Production systems IVF Cluster search O(\u221aN) Medium Good Large-scale search PQ Compressed codes O(N) Very Low Moderate Billions of vectors IVF-PQ Clusters + compression O(\u221aN) Low Good Balance of all factors Tip Most production systems use HNSW (Pinecone, Weaviate, Qdrant). It offers the best balance of speed and accuracy. Use IVF-PQ only when you have billions of vectors and limited memory. ","date":"10-Dec-2025","heading":"Algorithm Comparison","objectID":"\/posts\/vector-databases-rag-llm\/:3:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#algorithm-comparison"},{"categories":["AI-ML"],"collections":null,"content":"4. RAG (Retrieval-Augmented Generation)\r","date":"10-Dec-2025","heading":"4. RAG (Retrieval-Augmented Generation)","objectID":"\/posts\/vector-databases-rag-llm\/:4:0","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#rag-fundamentals"},{"categories":["AI-ML"],"collections":null,"content":"What is RAG? RAG enhances LLM responses by retrieving relevant information from external sources before generating an answer. Basic RAG pipeline architecture ","date":"10-Dec-2025","heading":"What is RAG?","objectID":"\/posts\/vector-databases-rag-llm\/:4:1","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#what-is-rag"},{"categories":["AI-ML"],"collections":null,"content":"RAG Pipeline Steps User Query \u2192 User asks a question Query Embedding \u2192 Convert query to vector Vector Search \u2192 Find similar documents Context Retrieval \u2192 Fetch document content Prompt Construction \u2192 Combine query + context LLM Generation \u2192 Generate grounded answer Info Why RAG Works: LLMs can’t read thousands of documents and remember them. Fine-tuning influences style, not knowledge. RAG retrieves relevant knowledge dynamically at runtime. ","date":"10-Dec-2025","heading":"RAG Pipeline Steps","objectID":"\/posts\/vector-databases-rag-llm\/:4:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#rag-pipeline-steps"},{"categories":["AI-ML"],"collections":null,"content":"RAG Benefits Benefit Description No Retraining Update knowledge by updating the database Verifiable Can cite sources for answers Cost-Effective Cheaper than fine-tuning Up-to-Date Add new information instantly ","date":"10-Dec-2025","heading":"RAG Benefits","objectID":"\/posts\/vector-databases-rag-llm\/:4:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#rag-benefits"},{"categories":["AI-ML"],"collections":null,"content":"Hallucination Prevention RAG combats hallucinations by: Providing factual context in the prompt Instructing the LLM to answer only from provided context Allowing “I don’t know” when context is insufficient ","date":"10-Dec-2025","heading":"Hallucination Prevention","objectID":"\/posts\/vector-databases-rag-llm\/:4:4","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#hallucination-prevention"},{"categories":["AI-ML"],"collections":null,"content":"5. Document Processing Pipeline\r","date":"10-Dec-2025","heading":"5. Document Processing Pipeline","objectID":"\/posts\/vector-databases-rag-llm\/:5:0","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#document-processing"},{"categories":["AI-ML"],"collections":null,"content":"Basic Chunking Pipeline Basic document chunking pipeline Steps: Extract raw text from documents (PDF, DOCX, HTML) Chunk into fixed-size pieces (500-1000 characters) Embed each chunk to a vector Store (vector, chunk) pairs in database ","date":"10-Dec-2025","heading":"Basic Chunking Pipeline","objectID":"\/posts\/vector-databases-rag-llm\/:5:1","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#basic-chunking-pipeline"},{"categories":["AI-ML"],"collections":null,"content":"Chunking Strategies Compared There are multiple ways to split documents. Each has trade-offs: Strategy How It Works Pros Cons Best For Fixed-size Split every N characters Simple, predictable May break mid-sentence Quick prototyping Sentence-based Split at sentence boundaries Grammatically correct Variable chunk sizes Articles, blogs Recursive Try paragraphs \u2192 sentences \u2192 words Balances size and meaning More complex Mixed documents Semantic Use embeddings to find topic shifts Best coherence Computationally expensive Technical docs Recursive Chunking (most recommended): First, try splitting by paragraph (\\n\\n) If chunks are still too large, split by sentence (. ) If still too large, split by words This preserves natural document structure ","date":"10-Dec-2025","heading":"Chunking Strategies Compared","objectID":"\/posts\/vector-databases-rag-llm\/:5:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#chunking-strategies-compared"},{"categories":["AI-ML"],"collections":null,"content":"Chunk Size Trade-offs Size Tokens Behavior Best For Small (~256) ~64 Precise retrieval, less context Specific fact lookup Medium (~512) ~128 Balanced approach General Q&A Large (~1024) ~256 More context, fewer chunks Complex explanations Very Large (~2048) ~512 Full paragraphs Broad topic summaries Rule of Thumb: Match chunk size to expected query length. Short questions \u2192 smaller chunks. Complex questions \u2192 larger chunks. Tip Always use chunk overlap (10-20%). This prevents losing information that spans chunk boundaries. If a key fact is at the edge of two chunks, overlap ensures it appears in at least one complete chunk. ","date":"10-Dec-2025","heading":"Chunk Size Trade-offs","objectID":"\/posts\/vector-databases-rag-llm\/:5:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#chunk-size-trade-offs"},{"categories":["AI-ML"],"collections":null,"content":"Advanced: Summarization Before Embedding Enhanced pipeline with LLM-based summarization Benefits: Removes filler text that confuses embeddings Creates denser, more meaningful representations Normalizes formatting across document types ","date":"10-Dec-2025","heading":"Advanced: Summarization Before Embedding","objectID":"\/posts\/vector-databases-rag-llm\/:5:4","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#advanced-summarization-before-embedding"},{"categories":["AI-ML"],"collections":null,"content":"Parent Document Retrieval For hierarchical documents (books, papers): Index smaller chunks (paragraphs) for precise retrieval When retrieved, also fetch the parent section for context Or use “windowed” approach\u2014retrieve neighboring chunks ","date":"10-Dec-2025","heading":"Parent Document Retrieval","objectID":"\/posts\/vector-databases-rag-llm\/:5:5","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#parent-document-retrieval"},{"categories":["AI-ML"],"collections":null,"content":"6. Prompt Engineering with RAG\r","date":"10-Dec-2025","heading":"6. Prompt Engineering with RAG","objectID":"\/posts\/vector-databases-rag-llm\/:6:0","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#prompt-engineering"},{"categories":["AI-ML"],"collections":null,"content":"Dynamic Few-Shot Prompting Instead of hardcoding examples, retrieve relevant examples from a vector database: Dynamic few-shot prompt construction Flow: User asks a question Vector DB retrieves similar Q&A examples Examples are injected into the prompt LLM generates answer using those examples as guidance ","date":"10-Dec-2025","heading":"Dynamic Few-Shot Prompting","objectID":"\/posts\/vector-databases-rag-llm\/:6:1","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#dynamic-few-shot-prompting"},{"categories":["AI-ML"],"collections":null,"content":"Temperature Settings Temperature Behavior Best For 0.0 - 0.3 Deterministic, factual RAG, Q&A, summarization 0.4 - 0.7 Balanced General conversation 0.8 - 1.0+ Creative, diverse Brainstorming, fiction Warning For RAG: Use low temperature (0.0-0.3). Higher temperatures increase hallucination risk\u2014defeating the purpose of retrieval-augmented generation. ","date":"10-Dec-2025","heading":"Temperature Settings","objectID":"\/posts\/vector-databases-rag-llm\/:6:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#temperature-settings"},{"categories":["AI-ML"],"collections":null,"content":"7. Hybrid Search Strategies\r","date":"10-Dec-2025","heading":"7. Hybrid Search Strategies","objectID":"\/posts\/vector-databases-rag-llm\/:7:0","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#hybrid-search"},{"categories":["AI-ML"],"collections":null,"content":"Limitations of Pure Semantic Search Semantic search can fail for: Entity-specific queries: “Windows 8” retrieves Windows 10\/11 content Exact terminology: Medical terms, legal citations, product SKUs Negations: “not Python” still retrieves Python content Rare terms: Words not well-represented in training data ","date":"10-Dec-2025","heading":"Limitations of Pure Semantic Search","objectID":"\/posts\/vector-databases-rag-llm\/:7:1","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#limitations-of-pure-semantic-search"},{"categories":["AI-ML"],"collections":null,"content":"Understanding Keyword Search (Sparse) Before combining approaches, understand how keyword search works: TF-IDF (Term Frequency - Inverse Document Frequency) A classic algorithm that scores documents based on keyword importance: TF (Term Frequency): How often does the word appear in this document? IDF (Inverse Document Frequency): How rare is this word across ALL documents? Score = TF \u00d7 IDF Intuition: If “quantum” appears 5 times in a document and is rare across your corpus, that document scores high for “quantum” queries. BM25 (Best Match 25) An improved version of TF-IDF used by most search engines: Adds document length normalization (short docs don’t unfairly win) Diminishing returns for repeated terms (10 mentions isn’t 10x better than 1) Tunable parameters (k1, b) for different use cases When BM25 Shines: Exact terminology, product codes, legal citations, medical terms. ","date":"10-Dec-2025","heading":"Understanding Keyword Search (Sparse)","objectID":"\/posts\/vector-databases-rag-llm\/:7:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#understanding-keyword-search-sparse"},{"categories":["AI-ML"],"collections":null,"content":"Understanding Keyword Search (Sparse) Before combining approaches, understand how keyword search works: TF-IDF (Term Frequency - Inverse Document Frequency) A classic algorithm that scores documents based on keyword importance: TF (Term Frequency): How often does the word appear in this document? IDF (Inverse Document Frequency): How rare is this word across ALL documents? Score = TF \u00d7 IDF Intuition: If “quantum” appears 5 times in a document and is rare across your corpus, that document scores high for “quantum” queries. BM25 (Best Match 25) An improved version of TF-IDF used by most search engines: Adds document length normalization (short docs don’t unfairly win) Diminishing returns for repeated terms (10 mentions isn’t 10x better than 1) Tunable parameters (k1, b) for different use cases When BM25 Shines: Exact terminology, product codes, legal citations, medical terms. ","date":"10-Dec-2025","heading":"TF-IDF (Term Frequency - Inverse Document Frequency)","objectID":"\/posts\/vector-databases-rag-llm\/:7:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#tf-idf-term-frequency---inverse-document-frequency"},{"categories":["AI-ML"],"collections":null,"content":"Understanding Keyword Search (Sparse) Before combining approaches, understand how keyword search works: TF-IDF (Term Frequency - Inverse Document Frequency) A classic algorithm that scores documents based on keyword importance: TF (Term Frequency): How often does the word appear in this document? IDF (Inverse Document Frequency): How rare is this word across ALL documents? Score = TF \u00d7 IDF Intuition: If “quantum” appears 5 times in a document and is rare across your corpus, that document scores high for “quantum” queries. BM25 (Best Match 25) An improved version of TF-IDF used by most search engines: Adds document length normalization (short docs don’t unfairly win) Diminishing returns for repeated terms (10 mentions isn’t 10x better than 1) Tunable parameters (k1, b) for different use cases When BM25 Shines: Exact terminology, product codes, legal citations, medical terms. ","date":"10-Dec-2025","heading":"BM25 (Best Match 25)","objectID":"\/posts\/vector-databases-rag-llm\/:7:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#bm25-best-match-25"},{"categories":["AI-ML"],"collections":null,"content":"Combining Dense and Sparse Vectors Hybrid search combining dense and sparse vectors How It Works: Dense Vector \u2192 Captures semantic meaning (“car” finds “automobile”) Sparse Vector \u2192 Captures exact keywords (BM25\/TF-IDF) Combined Query \u2192 Search both indexes simultaneously Score Fusion \u2192 Merge results using Reciprocal Rank Fusion (RRF) Reciprocal Rank Fusion (RRF) The standard method for combining search results: \ud835\udc11\ud835\udc11\ud835\udc05(\ud835\udc85) = \u2211 \ud835\udfcf \/ (\ud835\udc8c + \ud835\udc2b\ud835\udc1a\ud835\udc27\ud835\udc24\u1d63(\ud835\udc85)) for each \ud835\udc93 \u2208 \ud835\udc79 Where: d = document R = set of ranking methods (dense, sparse) k = constant (typically 60) rank_r(d) = position of document d in ranking r Why It Works: Documents appearing in BOTH dense and sparse results get boosted. A document ranked #3 in dense and #5 in sparse will outrank one that’s #1 in only one method. ","date":"10-Dec-2025","heading":"Combining Dense and Sparse Vectors","objectID":"\/posts\/vector-databases-rag-llm\/:7:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#combining-dense-and-sparse-vectors"},{"categories":["AI-ML"],"collections":null,"content":"Combining Dense and Sparse Vectors Hybrid search combining dense and sparse vectors How It Works: Dense Vector \u2192 Captures semantic meaning (“car” finds “automobile”) Sparse Vector \u2192 Captures exact keywords (BM25\/TF-IDF) Combined Query \u2192 Search both indexes simultaneously Score Fusion \u2192 Merge results using Reciprocal Rank Fusion (RRF) Reciprocal Rank Fusion (RRF) The standard method for combining search results: \ud835\udc11\ud835\udc11\ud835\udc05(\ud835\udc85) = \u2211 \ud835\udfcf \/ (\ud835\udc8c + \ud835\udc2b\ud835\udc1a\ud835\udc27\ud835\udc24\u1d63(\ud835\udc85)) for each \ud835\udc93 \u2208 \ud835\udc79 Where: d = document R = set of ranking methods (dense, sparse) k = constant (typically 60) rank_r(d) = position of document d in ranking r Why It Works: Documents appearing in BOTH dense and sparse results get boosted. A document ranked #3 in dense and #5 in sparse will outrank one that’s #1 in only one method. ","date":"10-Dec-2025","heading":"Reciprocal Rank Fusion (RRF)","objectID":"\/posts\/vector-databases-rag-llm\/:7:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#reciprocal-rank-fusion-rrf"},{"categories":["AI-ML"],"collections":null,"content":"When to Use Hybrid Search Scenario Approach Why General Q&A Dense (semantic) Users phrase questions differently than docs Technical docs Hybrid Need both concepts AND specific terms Legal\/Medical Hybrid (favor sparse) Exact terminology is critical Product search with SKUs Hybrid Must match exact product codes Conversational AI Dense Natural language varies widely ","date":"10-Dec-2025","heading":"When to Use Hybrid Search","objectID":"\/posts\/vector-databases-rag-llm\/:7:4","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#when-to-use-hybrid-search"},{"categories":["AI-ML"],"collections":null,"content":"Search Type Effectiveness Search Type Finds Misses Example Dense only Synonyms, paraphrases Exact codes, rare terms “Tell me about vehicles” \u2192 finds “car”, “automobile” Sparse only Exact matches Semantic variations “SKU-12345” \u2192 finds exact match Hybrid Both semantic AND exact Rarely misses anything “Windows 8 issues” \u2192 finds Windows 8 specifically Tip Rule of Thumb: If exact keywords matter, use hybrid search. Most modern vector databases (Pinecone, Qdrant, Weaviate) support it natively. Start with a 50\/50 weight, then tune based on your data. ","date":"10-Dec-2025","heading":"Search Type Effectiveness","objectID":"\/posts\/vector-databases-rag-llm\/:7:5","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#search-type-effectiveness"},{"categories":["AI-ML"],"collections":null,"content":"8. Reranking and Context Compression\r","date":"10-Dec-2025","heading":"8. Reranking and Context Compression","objectID":"\/posts\/vector-databases-rag-llm\/:8:0","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#reranking"},{"categories":["AI-ML"],"collections":null,"content":"Why Rerank? Initial vector search isn’t perfect: Relevant documents may rank low Too much context overwhelms the LLM Documents buried in the middle get ignored (“lost in the middle” problem) ","date":"10-Dec-2025","heading":"Why Rerank?","objectID":"\/posts\/vector-databases-rag-llm\/:8:1","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#why-rerank"},{"categories":["AI-ML"],"collections":null,"content":"How Reranking Works Search result reranking process Two-Stage Retrieval: Stage 1: Vector search retrieves broad candidates (fast, ~100 docs) Stage 2: Reranker re-scores top candidates by relevance (precise, ~20 docs) ","date":"10-Dec-2025","heading":"How Reranking Works","objectID":"\/posts\/vector-databases-rag-llm\/:8:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#how-reranking-works"},{"categories":["AI-ML"],"collections":null,"content":"Bi-Encoder vs Cross-Encoder The key to understanding reranking is understanding these two architectures: Bi-Encoder (Used for Initial Retrieval) Embeds query and document separately Compares pre-computed embeddings using cosine similarity Speed: Can search millions of docs in milliseconds Accuracy: Good, but misses nuanced query-document relationships Cross-Encoder (Used for Reranking) Processes query AND document together as one input Considers every word interaction between query and document Speed: Slow (must process each doc individually) Accuracy: Excellent (understands exact relevance) Type How It Works Speed Accuracy Stage Bi-Encoder Embed separately, compare Milliseconds for millions Good Initial retrieval Cross-Encoder Process together Seconds for dozens Excellent Reranking Analogy: Bi-encoders are like speed dating (quick impressions). Cross-encoders are like in-depth interviews (thorough evaluation). ","date":"10-Dec-2025","heading":"Bi-Encoder vs Cross-Encoder","objectID":"\/posts\/vector-databases-rag-llm\/:8:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#bi-encoder-vs-cross-encoder"},{"categories":["AI-ML"],"collections":null,"content":"Bi-Encoder vs Cross-Encoder The key to understanding reranking is understanding these two architectures: Bi-Encoder (Used for Initial Retrieval) Embeds query and document separately Compares pre-computed embeddings using cosine similarity Speed: Can search millions of docs in milliseconds Accuracy: Good, but misses nuanced query-document relationships Cross-Encoder (Used for Reranking) Processes query AND document together as one input Considers every word interaction between query and document Speed: Slow (must process each doc individually) Accuracy: Excellent (understands exact relevance) Type How It Works Speed Accuracy Stage Bi-Encoder Embed separately, compare Milliseconds for millions Good Initial retrieval Cross-Encoder Process together Seconds for dozens Excellent Reranking Analogy: Bi-encoders are like speed dating (quick impressions). Cross-encoders are like in-depth interviews (thorough evaluation). ","date":"10-Dec-2025","heading":"Bi-Encoder (Used for Initial Retrieval)","objectID":"\/posts\/vector-databases-rag-llm\/:8:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#bi-encoder-used-for-initial-retrieval"},{"categories":["AI-ML"],"collections":null,"content":"Bi-Encoder vs Cross-Encoder The key to understanding reranking is understanding these two architectures: Bi-Encoder (Used for Initial Retrieval) Embeds query and document separately Compares pre-computed embeddings using cosine similarity Speed: Can search millions of docs in milliseconds Accuracy: Good, but misses nuanced query-document relationships Cross-Encoder (Used for Reranking) Processes query AND document together as one input Considers every word interaction between query and document Speed: Slow (must process each doc individually) Accuracy: Excellent (understands exact relevance) Type How It Works Speed Accuracy Stage Bi-Encoder Embed separately, compare Milliseconds for millions Good Initial retrieval Cross-Encoder Process together Seconds for dozens Excellent Reranking Analogy: Bi-encoders are like speed dating (quick impressions). Cross-encoders are like in-depth interviews (thorough evaluation). ","date":"10-Dec-2025","heading":"Cross-Encoder (Used for Reranking)","objectID":"\/posts\/vector-databases-rag-llm\/:8:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#cross-encoder-used-for-reranking"},{"categories":["AI-ML"],"collections":null,"content":"Why Two Stages? You can’t use cross-encoders for initial search\u2014scoring 1 million documents would take hours. Instead: Stage 1 (Bi-Encoder): Cast a wide net, retrieve ~100 candidates fast Stage 2 (Cross-Encoder): Carefully evaluate top ~20 candidates Return the best ~5-10 to the LLM This gives you the best of both worlds: speed AND accuracy. ","date":"10-Dec-2025","heading":"Why Two Stages?","objectID":"\/posts\/vector-databases-rag-llm\/:8:4","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#why-two-stages"},{"categories":["AI-ML"],"collections":null,"content":"Popular Reranking Models Model Type Speed Accuracy Notes rerank-english-v3.0 Cohere API Fast Excellent Production-ready, paid bge-reranker-large Open Source Medium Excellent Best open-source option ms-marco-MiniLM Open Source Fast Good Lightweight, fast cross-encoder\/ms-marco-MiniLM-L-6-v2 HuggingFace Fast Good Easy to deploy ","date":"10-Dec-2025","heading":"Popular Reranking Models","objectID":"\/posts\/vector-databases-rag-llm\/:8:5","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#popular-reranking-models"},{"categories":["AI-ML"],"collections":null,"content":"Contextual Compression Use a smaller LLM to: Extract only relevant portions from each document Discard irrelevant context Reduce token usage and cost ","date":"10-Dec-2025","heading":"Contextual Compression","objectID":"\/posts\/vector-databases-rag-llm\/:8:6","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#contextual-compression"},{"categories":["AI-ML"],"collections":null,"content":"The “Lost in the Middle” Problem Research shows LLMs have a U-shaped attention pattern: Pay most attention to the beginning of context Pay good attention to the end of context Ignore or forget information in the middle Implications for RAG: Don’t just append retrieved docs in order of similarity score Put the MOST relevant document first Put the SECOND most relevant document last Less critical docs go in the middle Warning Lost in the Middle: If you retrieve 10 documents and the answer is in document #5, the LLM might miss it entirely. Reorder your context strategically! ","date":"10-Dec-2025","heading":"The “Lost in the Middle” Problem","objectID":"\/posts\/vector-databases-rag-llm\/:8:7","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#the-lost-in-the-middle-problem"},{"categories":["AI-ML"],"collections":null,"content":"9. Query Transformation Techniques\rSometimes the user’s query isn’t optimal for retrieval. Query transformation techniques improve retrieval by reformulating the query before searching. ","date":"10-Dec-2025","heading":"9. Query Transformation Techniques","objectID":"\/posts\/vector-databases-rag-llm\/:9:0","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#query-transformation"},{"categories":["AI-ML"],"collections":null,"content":"HyDE (Hypothetical Document Embeddings) Problem: User queries are short and may not match document vocabulary. Solution: Generate a hypothetical answer, then search for documents similar to that answer. How it works: User asks: “How do I fix memory leaks in Python?” LLM generates a hypothetical answer (even if imperfect) Embed the hypothetical answer (not the question) Search for real documents similar to this hypothetical Retrieved docs are often more relevant than direct query search When to use: Technical queries, specialized domains where users and documents use different vocabulary. ","date":"10-Dec-2025","heading":"HyDE (Hypothetical Document Embeddings)","objectID":"\/posts\/vector-databases-rag-llm\/:9:1","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#hyde-hypothetical-document-embeddings"},{"categories":["AI-ML"],"collections":null,"content":"Multi-Query Expansion Problem: A single query may miss relevant documents phrased differently. Solution: Generate multiple variations of the query and search with all of them. How it works: Original query: “Python web frameworks” Generate variations: “Django vs Flask comparison” “Best backend frameworks for Python” “Building web apps with Python” Search with each query Combine results (using RRF or union) When to use: Ambiguous queries, broad topics, when recall is more important than precision. ","date":"10-Dec-2025","heading":"Multi-Query Expansion","objectID":"\/posts\/vector-databases-rag-llm\/:9:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#multi-query-expansion"},{"categories":["AI-ML"],"collections":null,"content":"Step-Back Prompting Problem: Specific questions may miss broader context needed to answer well. Solution: Generate a more general version of the query first. How it works: Specific query: “What’s the boiling point of water at 2000m altitude?” Step-back query: “How does altitude affect boiling point?” Retrieve documents for BOTH queries Combine context (general + specific) When to use: Complex questions that require background knowledge. ","date":"10-Dec-2025","heading":"Step-Back Prompting","objectID":"\/posts\/vector-databases-rag-llm\/:9:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#step-back-prompting"},{"categories":["AI-ML"],"collections":null,"content":"Query Transformation Comparison Technique When to Use Latency Impact Recall Improvement HyDE Technical\/specialized queries +1 LLM call +15-25% Multi-Query Ambiguous queries +3-5 searches +10-20% Step-Back Complex questions +1 LLM call, +1 search Better context Query Rewriting Poor user queries +1 LLM call Variable Tip Start simple: Most RAG systems work fine without query transformation. Add these techniques only if you see retrieval quality issues. ","date":"10-Dec-2025","heading":"Query Transformation Comparison","objectID":"\/posts\/vector-databases-rag-llm\/:9:4","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#query-transformation-comparison"},{"categories":["AI-ML"],"collections":null,"content":"10. Advanced RAG Patterns\rBasic RAG (chunk \u2192 embed \u2192 retrieve \u2192 generate) works well for simple use cases. For more complex scenarios, consider these advanced patterns: ","date":"10-Dec-2025","heading":"10. Advanced RAG Patterns","objectID":"\/posts\/vector-databases-rag-llm\/:10:0","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#advanced-rag"},{"categories":["AI-ML"],"collections":null,"content":"CRAG (Corrective RAG) Problem: Retrieved documents might be irrelevant or outdated. Solution: Evaluate retrieval quality and self-correct if needed. How it works: Retrieve documents normally Use an LLM to evaluate: “Are these documents relevant to the query?” If confident \u2192 proceed to generation If uncertain \u2192 try alternative retrieval (e.g., web search) If irrelevant \u2192 fall back to web search or “I don’t know” When to use: Knowledge bases that may be incomplete, time-sensitive information. ","date":"10-Dec-2025","heading":"CRAG (Corrective RAG)","objectID":"\/posts\/vector-databases-rag-llm\/:10:1","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#crag-corrective-rag"},{"categories":["AI-ML"],"collections":null,"content":"Self-RAG (Self-Reflective RAG) Problem: Not every query needs retrieval. Basic RAG always retrieves, wasting resources. Solution: Let the LLM decide WHEN to retrieve and verify its own outputs. How it works: LLM evaluates: “Do I need external information for this query?” If yes \u2192 retrieve and generate with context If no \u2192 generate directly from knowledge After generation \u2192 LLM verifies: “Is this answer supported by the context?” When to use: Mixed query types (some factual, some conversational), cost-sensitive applications. ","date":"10-Dec-2025","heading":"Self-RAG (Self-Reflective RAG)","objectID":"\/posts\/vector-databases-rag-llm\/:10:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#self-rag-self-reflective-rag"},{"categories":["AI-ML"],"collections":null,"content":"Agentic RAG Problem: Complex questions require multiple retrieval steps and reasoning. Solution: LLM acts as an agent that can retrieve, reason, and retrieve again. How it works: LLM analyzes the question Breaks it into sub-questions if needed Retrieves information for each sub-question Reasons over retrieved information May retrieve again if gaps are found Synthesizes final answer Example: “Compare the economic policies of the last 3 US presidents” Agent retrieves info on President 1 Agent retrieves info on President 2 Agent retrieves info on President 3 Agent synthesizes comparison When to use: Research questions, multi-hop reasoning, complex analysis. ","date":"10-Dec-2025","heading":"Agentic RAG","objectID":"\/posts\/vector-databases-rag-llm\/:10:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#agentic-rag"},{"categories":["AI-ML"],"collections":null,"content":"RAG Pattern Selection Pattern Complexity Best For Key Benefit Basic RAG Simple Straightforward Q&A Easy to implement CRAG Medium Incomplete knowledge bases Handles retrieval failures Self-RAG Medium Mixed query types Efficient (skips unnecessary retrieval) Agentic RAG High Complex research Multi-step reasoning Info Start with Basic RAG. Only add complexity when you have evidence that basic RAG isn’t working. Each additional pattern adds latency and cost. ","date":"10-Dec-2025","heading":"RAG Pattern Selection","objectID":"\/posts\/vector-databases-rag-llm\/:10:4","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#rag-pattern-selection"},{"categories":["AI-ML"],"collections":null,"content":"11. RAG Evaluation Framework\rYou can’t improve what you don’t measure. RAG systems require evaluation at two stages: retrieval and generation. ","date":"10-Dec-2025","heading":"11. RAG Evaluation Framework","objectID":"\/posts\/vector-databases-rag-llm\/:11:0","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#evaluation"},{"categories":["AI-ML"],"collections":null,"content":"Retrieval Metrics These measure how well your system finds relevant documents: Metric What It Measures Formula Good Score Recall@k % of relevant docs in top k results relevant_in_k \/ total_relevant > 0.8 Precision@k % of top k results that are relevant relevant_in_k \/ k > 0.6 MRR Position of first relevant result 1 \/ rank_of_first_relevant > 0.7 NDCG Ranking quality (position matters) DCG \/ ideal_DCG > 0.7 Example: For query “What is RAG?”, if you retrieve 10 docs and 3 are relevant: If relevant docs are at positions 1, 2, 5 \u2192 Good (high MRR, good NDCG) If relevant docs are at positions 6, 8, 10 \u2192 Bad (low MRR, poor NDCG) ","date":"10-Dec-2025","heading":"Retrieval Metrics","objectID":"\/posts\/vector-databases-rag-llm\/:11:1","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#retrieval-metrics"},{"categories":["AI-ML"],"collections":null,"content":"Generation Metrics (RAGAS Framework) RAGAS is the standard framework for evaluating RAG generation quality: Metric What It Measures How It’s Calculated Faithfulness Is answer grounded in context? LLM checks if claims are supported Answer Relevancy Does answer address the question? LLM scores relevance Context Precision Are retrieved docs actually useful? % of context that contributed to answer Context Recall Did we retrieve all needed info? Can answer be derived from context alone? ","date":"10-Dec-2025","heading":"Generation Metrics (RAGAS Framework)","objectID":"\/posts\/vector-databases-rag-llm\/:11:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#generation-metrics-ragas-framework"},{"categories":["AI-ML"],"collections":null,"content":"Building an Evaluation Dataset To evaluate your RAG system, you need: Test Questions: 50-100 representative queries Ground Truth Answers: What the correct answer should be Relevant Documents: Which docs should be retrieved Quick Start: Extract real user questions from logs Have domain experts write answers Run evaluation weekly to catch regressions ","date":"10-Dec-2025","heading":"Building an Evaluation Dataset","objectID":"\/posts\/vector-databases-rag-llm\/:11:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#building-an-evaluation-dataset"},{"categories":["AI-ML"],"collections":null,"content":"Evaluation Best Practices Separate retrieval and generation evaluation - A bad answer might be due to poor retrieval OR poor generation Test edge cases - Queries with no answer, ambiguous queries, multi-hop questions Track metrics over time - Catch regressions early Use human evaluation for final quality assessment - Automated metrics don’t catch everything Tip Minimum viable evaluation: Start with 50 test questions and track Recall@10 for retrieval and Faithfulness for generation. Expand from there. ","date":"10-Dec-2025","heading":"Evaluation Best Practices","objectID":"\/posts\/vector-databases-rag-llm\/:11:4","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#evaluation-best-practices"},{"categories":["AI-ML"],"collections":null,"content":"12. Debugging RAG Failures\rWhen your RAG system gives wrong answers, use this systematic approach to find and fix the problem. ","date":"10-Dec-2025","heading":"12. Debugging RAG Failures","objectID":"\/posts\/vector-databases-rag-llm\/:12:0","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#debugging"},{"categories":["AI-ML"],"collections":null,"content":"Common Failure Modes Symptom Likely Cause Solution Returns irrelevant documents Embedding model mismatch Try domain-specific embeddings Misses obvious answers Chunks too small Increase chunk size + overlap “I don’t know” for known facts Document not indexed Check ingestion pipeline Contradictory answers Multiple conflicting sources Add source reliability scoring Slow responses Too many docs retrieved Reduce k, add reranking Hallucinations despite context LLM ignoring context Lower temperature, stronger instructions Wrong version info Semantic similarity ignores numbers Use hybrid search ","date":"10-Dec-2025","heading":"Common Failure Modes","objectID":"\/posts\/vector-databases-rag-llm\/:12:1","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#common-failure-modes"},{"categories":["AI-ML"],"collections":null,"content":"Debugging Checklist When a query fails, check these in order: 1. Is the document even in the index? Search for an exact phrase from the expected document If not found \u2192 ingestion problem 2. Is the document retrieved? Look at similarity scores of retrieved docs If expected doc has low score \u2192 embedding or chunking problem 3. Is the right chunk retrieved? Check if the answer spans multiple chunks If answer is split \u2192 adjust chunk size\/overlap 4. Is the LLM using the context? Check if answer matches retrieved context If LLM ignores context \u2192 adjust prompt, lower temperature 5. Is there conflicting information? Check for contradictory docs in results If present \u2192 add source filtering or recency scoring ","date":"10-Dec-2025","heading":"Debugging Checklist","objectID":"\/posts\/vector-databases-rag-llm\/:12:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#debugging-checklist"},{"categories":["AI-ML"],"collections":null,"content":"Retrieval Quality Debugging Test #1: Direct phrase search Search for exact text from a document you know exists If it doesn’t appear in top 10 \u2192 indexing problem Test #2: Synonym search Search using synonyms of known document content If it works \u2192 your embedding model is fine If it fails \u2192 consider different embedding model Test #3: Compare dense vs sparse Run query through dense search only Run query through sparse (keyword) search only Compare results \u2192 decide if hybrid search would help ","date":"10-Dec-2025","heading":"Retrieval Quality Debugging","objectID":"\/posts\/vector-databases-rag-llm\/:12:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#retrieval-quality-debugging"},{"categories":["AI-ML"],"collections":null,"content":"When to Suspect Each Component Component Suspect If… Chunking Answers are partially correct, missing context Embedding Model Synonyms don’t match, domain terms fail Index Config High recall but slow, or fast but missing results Retrieval k Good docs exist but not in top k Reranking Good docs retrieved but ranked low Prompt Correct context but wrong answer LLM Temperature Answers vary wildly, or include made-up facts Warning Don’t guess - measure! Log every query, retrieval result, and answer. Use these logs to identify patterns in failures. ","date":"10-Dec-2025","heading":"When to Suspect Each Component","objectID":"\/posts\/vector-databases-rag-llm\/:12:4","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#when-to-suspect-each-component"},{"categories":["AI-ML"],"collections":null,"content":"13. Cost Optimization\rRAG systems can get expensive at scale. Here’s how to optimize costs without sacrificing quality. ","date":"10-Dec-2025","heading":"13. Cost Optimization","objectID":"\/posts\/vector-databases-rag-llm\/:13:0","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#cost"},{"categories":["AI-ML"],"collections":null,"content":"Cost Components Component Cost Driver Typical Cost Embedding API Per token ~$0.02 per 1M tokens Vector Database Storage + queries $0.01-0.10 per 1K queries LLM API Input + output tokens $0.50-5.00 per 1K queries Reranking Per document scored ~$0.001 per document Key Insight: LLM calls dominate costs (often 80-90% of total). ","date":"10-Dec-2025","heading":"Cost Components","objectID":"\/posts\/vector-databases-rag-llm\/:13:1","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#cost-components"},{"categories":["AI-ML"],"collections":null,"content":"Cost Reduction Strategies 1. Query Routing (Biggest Impact) Not every query needs RAG. Route simple queries directly to the LLM. “What’s 2+2?” \u2192 Direct LLM (no retrieval needed) “What’s our refund policy?” \u2192 RAG (needs company docs) Savings: 30-50% reduction in RAG operations. 2. Aggressive Caching Cache at multiple levels: Query cache: Same query \u2192 same results Embedding cache: Same text \u2192 same embedding Answer cache: Frequent questions \u2192 cached answers Savings: 20-40% reduction in API calls. 3. Tiered Retrieval Only use expensive operations when needed: Fast vector search (always) Reranking (only if initial results are uncertain) Query transformation (only if initial retrieval fails) Savings: 15-25% reduction in compute. 4. Context Optimization Reduce tokens sent to LLM: Retrieve fewer documents (5 instead of 10) Use smaller chunks Compress context before sending Savings: 20-40% reduction in LLM costs. 5. Model Selection Use cheaper models for appropriate tasks: Task Expensive Option Cheaper Alternative Embedding text-embedding-3-large text-embedding-3-small Query routing GPT-4 GPT-3.5 or classifier Simple Q&A GPT-4 GPT-3.5 or Claude Haiku Complex reasoning GPT-4 Keep GPT-4 (worth the cost) ","date":"10-Dec-2025","heading":"Cost Reduction Strategies","objectID":"\/posts\/vector-databases-rag-llm\/:13:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#cost-reduction-strategies"},{"categories":["AI-ML"],"collections":null,"content":"Cost Reduction Strategies 1. Query Routing (Biggest Impact) Not every query needs RAG. Route simple queries directly to the LLM. “What’s 2+2?” \u2192 Direct LLM (no retrieval needed) “What’s our refund policy?” \u2192 RAG (needs company docs) Savings: 30-50% reduction in RAG operations. 2. Aggressive Caching Cache at multiple levels: Query cache: Same query \u2192 same results Embedding cache: Same text \u2192 same embedding Answer cache: Frequent questions \u2192 cached answers Savings: 20-40% reduction in API calls. 3. Tiered Retrieval Only use expensive operations when needed: Fast vector search (always) Reranking (only if initial results are uncertain) Query transformation (only if initial retrieval fails) Savings: 15-25% reduction in compute. 4. Context Optimization Reduce tokens sent to LLM: Retrieve fewer documents (5 instead of 10) Use smaller chunks Compress context before sending Savings: 20-40% reduction in LLM costs. 5. Model Selection Use cheaper models for appropriate tasks: Task Expensive Option Cheaper Alternative Embedding text-embedding-3-large text-embedding-3-small Query routing GPT-4 GPT-3.5 or classifier Simple Q&A GPT-4 GPT-3.5 or Claude Haiku Complex reasoning GPT-4 Keep GPT-4 (worth the cost) ","date":"10-Dec-2025","heading":"1. Query Routing (Biggest Impact)","objectID":"\/posts\/vector-databases-rag-llm\/:13:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#1-query-routing-biggest-impact"},{"categories":["AI-ML"],"collections":null,"content":"Cost Reduction Strategies 1. Query Routing (Biggest Impact) Not every query needs RAG. Route simple queries directly to the LLM. “What’s 2+2?” \u2192 Direct LLM (no retrieval needed) “What’s our refund policy?” \u2192 RAG (needs company docs) Savings: 30-50% reduction in RAG operations. 2. Aggressive Caching Cache at multiple levels: Query cache: Same query \u2192 same results Embedding cache: Same text \u2192 same embedding Answer cache: Frequent questions \u2192 cached answers Savings: 20-40% reduction in API calls. 3. Tiered Retrieval Only use expensive operations when needed: Fast vector search (always) Reranking (only if initial results are uncertain) Query transformation (only if initial retrieval fails) Savings: 15-25% reduction in compute. 4. Context Optimization Reduce tokens sent to LLM: Retrieve fewer documents (5 instead of 10) Use smaller chunks Compress context before sending Savings: 20-40% reduction in LLM costs. 5. Model Selection Use cheaper models for appropriate tasks: Task Expensive Option Cheaper Alternative Embedding text-embedding-3-large text-embedding-3-small Query routing GPT-4 GPT-3.5 or classifier Simple Q&A GPT-4 GPT-3.5 or Claude Haiku Complex reasoning GPT-4 Keep GPT-4 (worth the cost) ","date":"10-Dec-2025","heading":"2. Aggressive Caching","objectID":"\/posts\/vector-databases-rag-llm\/:13:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#2-aggressive-caching"},{"categories":["AI-ML"],"collections":null,"content":"Cost Reduction Strategies 1. Query Routing (Biggest Impact) Not every query needs RAG. Route simple queries directly to the LLM. “What’s 2+2?” \u2192 Direct LLM (no retrieval needed) “What’s our refund policy?” \u2192 RAG (needs company docs) Savings: 30-50% reduction in RAG operations. 2. Aggressive Caching Cache at multiple levels: Query cache: Same query \u2192 same results Embedding cache: Same text \u2192 same embedding Answer cache: Frequent questions \u2192 cached answers Savings: 20-40% reduction in API calls. 3. Tiered Retrieval Only use expensive operations when needed: Fast vector search (always) Reranking (only if initial results are uncertain) Query transformation (only if initial retrieval fails) Savings: 15-25% reduction in compute. 4. Context Optimization Reduce tokens sent to LLM: Retrieve fewer documents (5 instead of 10) Use smaller chunks Compress context before sending Savings: 20-40% reduction in LLM costs. 5. Model Selection Use cheaper models for appropriate tasks: Task Expensive Option Cheaper Alternative Embedding text-embedding-3-large text-embedding-3-small Query routing GPT-4 GPT-3.5 or classifier Simple Q&A GPT-4 GPT-3.5 or Claude Haiku Complex reasoning GPT-4 Keep GPT-4 (worth the cost) ","date":"10-Dec-2025","heading":"3. Tiered Retrieval","objectID":"\/posts\/vector-databases-rag-llm\/:13:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#3-tiered-retrieval"},{"categories":["AI-ML"],"collections":null,"content":"Cost Reduction Strategies 1. Query Routing (Biggest Impact) Not every query needs RAG. Route simple queries directly to the LLM. “What’s 2+2?” \u2192 Direct LLM (no retrieval needed) “What’s our refund policy?” \u2192 RAG (needs company docs) Savings: 30-50% reduction in RAG operations. 2. Aggressive Caching Cache at multiple levels: Query cache: Same query \u2192 same results Embedding cache: Same text \u2192 same embedding Answer cache: Frequent questions \u2192 cached answers Savings: 20-40% reduction in API calls. 3. Tiered Retrieval Only use expensive operations when needed: Fast vector search (always) Reranking (only if initial results are uncertain) Query transformation (only if initial retrieval fails) Savings: 15-25% reduction in compute. 4. Context Optimization Reduce tokens sent to LLM: Retrieve fewer documents (5 instead of 10) Use smaller chunks Compress context before sending Savings: 20-40% reduction in LLM costs. 5. Model Selection Use cheaper models for appropriate tasks: Task Expensive Option Cheaper Alternative Embedding text-embedding-3-large text-embedding-3-small Query routing GPT-4 GPT-3.5 or classifier Simple Q&A GPT-4 GPT-3.5 or Claude Haiku Complex reasoning GPT-4 Keep GPT-4 (worth the cost) ","date":"10-Dec-2025","heading":"4. Context Optimization","objectID":"\/posts\/vector-databases-rag-llm\/:13:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#4-context-optimization"},{"categories":["AI-ML"],"collections":null,"content":"Cost Reduction Strategies 1. Query Routing (Biggest Impact) Not every query needs RAG. Route simple queries directly to the LLM. “What’s 2+2?” \u2192 Direct LLM (no retrieval needed) “What’s our refund policy?” \u2192 RAG (needs company docs) Savings: 30-50% reduction in RAG operations. 2. Aggressive Caching Cache at multiple levels: Query cache: Same query \u2192 same results Embedding cache: Same text \u2192 same embedding Answer cache: Frequent questions \u2192 cached answers Savings: 20-40% reduction in API calls. 3. Tiered Retrieval Only use expensive operations when needed: Fast vector search (always) Reranking (only if initial results are uncertain) Query transformation (only if initial retrieval fails) Savings: 15-25% reduction in compute. 4. Context Optimization Reduce tokens sent to LLM: Retrieve fewer documents (5 instead of 10) Use smaller chunks Compress context before sending Savings: 20-40% reduction in LLM costs. 5. Model Selection Use cheaper models for appropriate tasks: Task Expensive Option Cheaper Alternative Embedding text-embedding-3-large text-embedding-3-small Query routing GPT-4 GPT-3.5 or classifier Simple Q&A GPT-4 GPT-3.5 or Claude Haiku Complex reasoning GPT-4 Keep GPT-4 (worth the cost) ","date":"10-Dec-2025","heading":"5. Model Selection","objectID":"\/posts\/vector-databases-rag-llm\/:13:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#5-model-selection"},{"categories":["AI-ML"],"collections":null,"content":"Cost Estimation Example Scenario: 10,000 queries\/month, 5 docs retrieved per query Approach Embedding Vector DB LLM Total\/Month Basic RAG $0.20 $1.00 $50.00 ~$51 With reranking $0.20 $1.00 $70.00 ~$71 Optimized $0.10 $0.50 $25.00 ~$26 Optimizations applied: Query routing (50%), caching (20%), context compression (30%). Tip Measure before optimizing. Log costs per query type. You’ll often find 80% of costs come from 20% of query types\u2014focus there first. ","date":"10-Dec-2025","heading":"Cost Estimation Example","objectID":"\/posts\/vector-databases-rag-llm\/:13:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#cost-estimation-example"},{"categories":["AI-ML"],"collections":null,"content":"14. Popular Vector Databases\rDatabase Type Key Features Best For Pinecone Managed SaaS Serverless, hybrid search, easy setup Production without DevOps Weaviate Open Source\/Cloud GraphQL API, built-in ML modules Flexible deployments Qdrant Open Source\/Cloud Rust-based, very fast, good filtering High performance needs Milvus Open Source Distributed, GPU support, massive scale Billions of vectors Chroma Open Source Simple API, in-memory option Prototyping, local dev pgvector Postgres Extension SQL integration, familiar tooling Teams using Postgres ","date":"10-Dec-2025","heading":"14. Popular Vector Databases","objectID":"\/posts\/vector-databases-rag-llm\/:14:0","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#implementation"},{"categories":["AI-ML"],"collections":null,"content":"Deployment Options Option Pros Cons Best For Managed SaaS (Pinecone) Zero DevOps, high reliability Higher cost, vendor lock-in Small teams, quick start Managed Cloud (Weaviate Cloud, Qdrant Cloud) Balance of control and convenience Medium cost Growing teams Self-Hosted (Milvus, Qdrant, Weaviate) Full control, lowest cost at scale DevOps required Large teams, enterprises ","date":"10-Dec-2025","heading":"Deployment Options","objectID":"\/posts\/vector-databases-rag-llm\/:14:1","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#deployment-options"},{"categories":["AI-ML"],"collections":null,"content":"Choosing a Vector Database Choose Pinecone if: You want zero infrastructure management and fastest time-to-production. Choose Weaviate if: You need flexibility and like GraphQL APIs. Choose Qdrant if: Performance is critical and you want both cloud and self-hosted options. Choose Milvus if: You have billions of vectors and need distributed architecture. Choose Chroma if: You’re prototyping or building local-first applications. Choose pgvector if: You already use PostgreSQL and want to add vector search without new infrastructure. ","date":"10-Dec-2025","heading":"Choosing a Vector Database","objectID":"\/posts\/vector-databases-rag-llm\/:14:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#choosing-a-vector-database"},{"categories":["AI-ML"],"collections":null,"content":"Production Considerations Warning Security Never expose API keys in client-side code Implement rate limiting Sanitize queries to prevent prompt injection Use row-level security for multi-tenant applications Performance: Batch embedding requests when indexing Use async operations for concurrent retrieval Cache frequently-asked queries Monitor query latency and adjust index parameters Reliability: Set up automated backups Use replicas for high availability Implement graceful degradation when DB is slow ","date":"10-Dec-2025","heading":"Production Considerations","objectID":"\/posts\/vector-databases-rag-llm\/:14:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#production-considerations"},{"categories":["AI-ML"],"collections":null,"content":"15. Best Practices Summary\r","date":"10-Dec-2025","heading":"15. Best Practices Summary","objectID":"\/posts\/vector-databases-rag-llm\/:15:0","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#best-practices"},{"categories":["AI-ML"],"collections":null,"content":"Chunking Checklist Tip Preserve context: Don’t split mid-sentence Use overlap: 10-20% prevents information loss Respect structure: Use headers as natural break points Include metadata: Store source, page number, section title Test empirically: Evaluate with different chunk sizes ","date":"10-Dec-2025","heading":"Chunking Checklist","objectID":"\/posts\/vector-databases-rag-llm\/:15:1","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#chunking-checklist"},{"categories":["AI-ML"],"collections":null,"content":"Embedding Model Selection Use Case Model Rationale General English text-embedding-3-small Cost-effective Maximum accuracy text-embedding-3-large Best quality Multi-language embed-multilingual-v3.0 100+ languages Open-source BGE-large-en-v1.5 Top OSS performance Low latency all-MiniLM-L6-v2 Fast, 384 dimensions ","date":"10-Dec-2025","heading":"Embedding Model Selection","objectID":"\/posts\/vector-databases-rag-llm\/:15:2","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#embedding-model-selection"},{"categories":["AI-ML"],"collections":null,"content":"Key Takeaways Embeddings transform text into searchable vectors capturing semantic meaning Vector databases enable fast similarity search at scale (HNSW is the standard) RAG grounds LLM responses in factual, retrievable knowledge Chunking significantly impacts quality\u2014use recursive chunking with overlap Hybrid search combines semantic + keyword for best coverage Reranking improves precision using cross-encoders Evaluation is essential\u2014track Recall@k and Faithfulness at minimum Cost optimization starts with query routing and caching ","date":"10-Dec-2025","heading":"Key Takeaways","objectID":"\/posts\/vector-databases-rag-llm\/:15:3","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#key-takeaways"},{"categories":["AI-ML"],"collections":null,"content":"Implementation Roadmap Phase 1: Basic RAG (Start here) Choose embedding model (start with text-embedding-3-small) Set up vector database (Pinecone for managed, Chroma for local) Implement basic chunking (512 tokens, 10% overlap) Build retrieve \u2192 generate pipeline Test with 20-30 sample questions Phase 2: Quality Improvements (When basic RAG isn’t enough) Add hybrid search if keyword matching matters Add reranking if relevant docs are retrieved but ranked low Adjust chunk sizes based on retrieval quality Build evaluation dataset, track metrics Phase 3: Advanced Features (For complex use cases) Query transformation (HyDE, multi-query) for better retrieval CRAG\/Self-RAG for handling failures Agentic RAG for multi-hop reasoning Cost optimization for scale Info Start Simple, Iterate Don’t add complexity until you have evidence it’s needed. Each additional component adds latency, cost, and maintenance burden. The best RAG system is the simplest one that meets your quality requirements. ","date":"10-Dec-2025","heading":"Implementation Roadmap","objectID":"\/posts\/vector-databases-rag-llm\/:15:4","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#implementation-roadmap"},{"categories":["AI-ML"],"collections":null,"content":"Quick Decision Guide Question If Yes \u2192 If No \u2192 Do exact keywords matter? Use hybrid search Dense search is fine Are relevant docs ranked low? Add reranking Skip reranking Do queries use different vocabulary than docs? Try HyDE Basic retrieval is fine Are some queries too complex? Consider Agentic RAG Basic RAG is fine Is cost a concern? Implement query routing + caching Optimize later Further Reading: Weaviate: What is a Vector Database? LangChain RAG Documentation Pinecone Learning Center RAG Paper (arXiv:2005.11401) RAGAS Evaluation Framework Lost in the Middle Paper ","date":"10-Dec-2025","heading":"Quick Decision Guide","objectID":"\/posts\/vector-databases-rag-llm\/:15:5","tags":["LLM","RAG","vector-databases"],"title":"Vector Databases and RAG Systems \u2014 Building Intelligent LLM Applications","uri":"\/posts\/vector-databases-rag-llm\/#quick-decision-guide"},{"categories":["projects"],"collections":null,"content":"Build a production-ready newsletter and contact management system using Cloudflare Workers, KV storage, and multiple email providers. Features automated RSS monitoring, intelligent bot protection, and infinite scalability at the edge.","date":"08-Nov-2025","objectID":"\/posts\/serverless-newsletter-and-contact-system\/","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/"},{"categories":["projects"],"collections":null,"content":" \ud83d\ude80 Live Demo & Source Code GitHub Repository: github.com\/SamirPaulb\/newsletter-and-contact-system \u2b50 Production Ready: Running on Cloudflare’s global edge network with 275+ locations worldwide Tech Stack: JavaScript, Cloudflare Workers, KV Storage, GitHub Actions, Multiple Email Providers ","date":"08-Nov-2025","heading":"","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:0:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#"},{"categories":["projects"],"collections":null,"content":"Introduction: The Power of Edge Computing Imagine building a newsletter system that can handle 1 subscriber or 1 million subscribers without changing a single line of infrastructure code. No servers to provision, no databases to scale, no load balancers to configure. This is the reality of edge computing with Cloudflare Workers. In this comprehensive guide, I’ll walk you through building a production-ready newsletter and contact management system that: Feature Description \ud83c\udf0d Global Distribution Runs on 275+ edge locations with <50ms latency worldwide \ud83d\udce7 Multi-Provider Support Gmail SMTP, MailerLite API, Cloudflare Email Routing \ud83d\udee1\ufe0f Intelligent Protection Progressive CAPTCHA that adapts to threat levels \ud83d\udcca Infinite Scalability Same code handles 1 to 1M+ subscribers \ud83d\udcb0 Cost Effective ~$5\/month for 1000 subscribers \ud83d\udd10 Enterprise Security Zero Trust, rate limiting, DDoS protection built-in \ud83e\udd16 Full Automation RSS monitoring, batch processing, auto-backups \ud83d\udcf0 Universal Feed Parser Supports RSS 2.0, RSS 1.0, Atom, JSON Feed \ud83d\udca1 Why Cloudflare Workers? Traditional server-based solutions require complex scaling strategies, database replication, and significant DevOps overhead. Cloudflare Workers eliminate all of this by running your code at the edge, automatically scaling to handle any load. ","date":"08-Nov-2025","heading":"Introduction: The Power of Edge Computing","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:1:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#introduction-the-power-of-edge-computing"},{"categories":["projects"],"collections":null,"content":"\ud83c\udfe0 System Homepage Clean and modern homepage interface showing the newsletter & contact management system with quick access to Subscribe, Unsubscribe, Contact, and Status pages. The interface displays system information including email provider configuration, RSS feed status, batch size settings, and bot protection status ","date":"08-Nov-2025","heading":"\ud83c\udfe0 System Homepage","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:1:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#-system-homepage"},{"categories":["projects"],"collections":null,"content":"How Serverless Works Under the Hood Before diving into our newsletter system, let’s understand the revolutionary technology powering it. Cloudflare Workers isn’t just “serverless” - it’s a fundamentally different approach to computing that enables true infinite scalability. ","date":"08-Nov-2025","heading":"How Serverless Works Under the Hood","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:2:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#how-serverless-works-under-the-hood"},{"categories":["projects"],"collections":null,"content":"The Evolution from Servers to Serverless The evolution of hosting infrastructure from physical servers (100% resource usage) through VMs, containers, FaaS, to edge computing with Cloudflare Workers (0.1% resource usage) ","date":"08-Nov-2025","heading":"The Evolution from Servers to Serverless","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:2:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#the-evolution-from-servers-to-serverless"},{"categories":["projects"],"collections":null,"content":"V8 Isolates: The Secret Sauce Unlike traditional serverless platforms that use containers or VMs, Cloudflare Workers uses V8 Isolates - the same technology that powers Chrome’s JavaScript engine. This is the key to achieving microsecond cold starts and infinite scalability. Comparing cold starts: AWS Lambda with 500-3000ms container spin-up vs Cloudflare Workers with 5 microsecond isolate creation What are V8 Isolates? V8 Isolates are lightweight JavaScript execution contexts that provide: Memory Isolation: Each worker runs in its own memory space CPU Isolation: Prevents one worker from affecting others Zero Cold Starts: New isolates spin up in microseconds Shared Runtime: Multiple isolates run in the same process \/\/ This is what happens under the hood when your Worker receives a request \/\/ 1. Request arrives at edge location \/\/ 2. V8 engine (already running) creates new isolate class WorkerIsolate { constructor(code) { this.context = new V8.Context(); \/\/ Microseconds this.globals = this.setupGlobals(); this.code = code; } async handleRequest(request) { \/\/ Your worker code executes here \/\/ Completely isolated from other workers return await this.code.fetch(request); } } \/\/ 3. Isolate destroyed after request completes \/\/ Total overhead: ~0.005ms vs 500-3000ms for containers ","date":"08-Nov-2025","heading":"V8 Isolates: The Secret Sauce","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:2:2","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#v8-isolates-the-secret-sauce"},{"categories":["projects"],"collections":null,"content":"V8 Isolates: The Secret Sauce Unlike traditional serverless platforms that use containers or VMs, Cloudflare Workers uses V8 Isolates - the same technology that powers Chrome’s JavaScript engine. This is the key to achieving microsecond cold starts and infinite scalability. Comparing cold starts: AWS Lambda with 500-3000ms container spin-up vs Cloudflare Workers with 5 microsecond isolate creation What are V8 Isolates? V8 Isolates are lightweight JavaScript execution contexts that provide: Memory Isolation: Each worker runs in its own memory space CPU Isolation: Prevents one worker from affecting others Zero Cold Starts: New isolates spin up in microseconds Shared Runtime: Multiple isolates run in the same process \/\/ This is what happens under the hood when your Worker receives a request \/\/ 1. Request arrives at edge location \/\/ 2. V8 engine (already running) creates new isolate class WorkerIsolate { constructor(code) { this.context = new V8.Context(); \/\/ Microseconds this.globals = this.setupGlobals(); this.code = code; } async handleRequest(request) { \/\/ Your worker code executes here \/\/ Completely isolated from other workers return await this.code.fetch(request); } } \/\/ 3. Isolate destroyed after request completes \/\/ Total overhead: ~0.005ms vs 500-3000ms for containers ","date":"08-Nov-2025","heading":"What are V8 Isolates?","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:2:2","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#what-are-v8-isolates"},{"categories":["projects"],"collections":null,"content":"The Edge Network Architecture Cloudflare Workers run on every one of Cloudflare’s 275+ edge locations worldwide. Here’s how a request flows through the system: Complete request flow through Cloudflare's edge network showing DNS resolution, V8 engine processing, isolate creation, KV storage access, and response generation in under 50ms ","date":"08-Nov-2025","heading":"The Edge Network Architecture","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:2:3","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#the-edge-network-architecture"},{"categories":["projects"],"collections":null,"content":"Request Handling Deep Dive Let’s examine exactly what happens when a request hits your Worker: Detailed request lifecycle showing TLS termination, WAF, rate limiting, caching, isolate creation, and resource limits (10-50ms CPU, 128MB memory, 50 subrequests) ","date":"08-Nov-2025","heading":"Request Handling Deep Dive","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:2:4","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#request-handling-deep-dive"},{"categories":["projects"],"collections":null,"content":"Memory Management in Isolates One of the most impressive aspects of V8 Isolates is their memory efficiency: Memory comparison: Traditional containers require 1.5GB for 3 concurrent requests vs V8 Isolates needing only 130MB for the same load ","date":"08-Nov-2025","heading":"Memory Management in Isolates","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:2:5","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#memory-management-in-isolates"},{"categories":["projects"],"collections":null,"content":"Global Distribution & Anycast Routing Cloudflare Workers leverage Anycast IP routing to automatically route users to the nearest edge location: Anycast routing automatically directs users to the nearest edge location with the same worker code running everywhere and KV storage providing eventual consistency ","date":"08-Nov-2025","heading":"Global Distribution & Anycast Routing","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:2:6","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#global-distribution--anycast-routing"},{"categories":["projects"],"collections":null,"content":"Cold Starts Comparison The elimination of cold starts is what makes Workers truly revolutionary: Gantt chart showing cold start times: AWS Lambda (850ms total), Azure Functions (880ms total), vs Cloudflare Workers (50ms total with 1ms isolate creation) ","date":"08-Nov-2025","heading":"Cold Starts Comparison","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:2:7","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#cold-starts-comparison"},{"categories":["projects"],"collections":null,"content":"The Magic of Zero Infrastructure Here’s what you DON’T need to manage with Cloudflare Workers: Comprehensive mindmap showing everything you don't need to manage with Cloudflare Workers: no servers, no operations, no complexity, with built-in features like auto-scaling and DDoS protection ","date":"08-Nov-2025","heading":"The Magic of Zero Infrastructure","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:2:8","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#the-magic-of-zero-infrastructure"},{"categories":["projects"],"collections":null,"content":"Resource Limits and Pricing Model Understanding the limits helps you design efficient Workers: Resource Free Tier Paid Plan Why It Matters Requests 100,000\/day 10M\/month + $0.50\/million Generous for most applications CPU Time 10ms\/request 50ms\/request Forces efficient code Memory 128MB 128MB Sufficient for most use cases Script Size 1MB 10MB Keeps workers fast Subrequests 50 50 Enough for complex workflows Environment Variables 64 64 Use KV for more config \u26a1 Performance Optimization Tips Minimize subrequests: Batch API calls when possible Use KV caching: Cache expensive computations Optimize script size: Use tree-shaking and minification Leverage cache API: Cache responses at the edge Avoid CPU-intensive operations: Offload to external services if needed ","date":"08-Nov-2025","heading":"Resource Limits and Pricing Model","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:2:9","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#resource-limits-and-pricing-model"},{"categories":["projects"],"collections":null,"content":"Why This Architecture Enables Infinite Scale The combination of V8 Isolates and edge distribution creates a system that can handle any load: No Shared State: Each request is independent Instant Scaling: New isolates created in microseconds Global Distribution: Load spread across 275+ locations Resource Isolation: One worker can’t affect others Automatic Failover: If one location fails, traffic routes elsewhere \/\/ This same code handles 1 request or 1 billion requests export default { async fetch(request, env) { \/\/ No difference in code for any scale \/\/ Cloudflare handles all the complexity return new Response(\"Hello World\"); } } \ud83c\udfaf The Bottom Line Cloudflare Workers represent a paradigm shift in computing. By leveraging V8 Isolates instead of containers, eliminating cold starts, and distributing execution globally, they achieve what was previously impossible: true infinite scalability with zero infrastructure management. ","date":"08-Nov-2025","heading":"Why This Architecture Enables Infinite Scale","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:2:10","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#why-this-architecture-enables-infinite-scale"},{"categories":["projects"],"collections":null,"content":"System Architecture Overview Our newsletter system leverages Cloudflare’s global edge network to achieve unprecedented scalability and performance. Here’s the complete architecture: Complete architecture showing global edge network, worker platform with KV storage, multiple email providers, scheduled jobs, GitHub backups, and security layers ","date":"08-Nov-2025","heading":"System Architecture Overview","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:3:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#system-architecture-overview"},{"categories":["projects"],"collections":null,"content":"Key Architectural Decisions Serverless at the Edge: No servers, containers, or VMs to manage Distributed KV Storage: Automatic replication across regions Multi-Provider Email: Failover capability and provider flexibility Progressive Security: Intelligent bot protection that adapts to threats Automated Operations: Self-maintaining with cleanup and backups ","date":"08-Nov-2025","heading":"Key Architectural Decisions","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:3:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#key-architectural-decisions"},{"categories":["projects"],"collections":null,"content":"Core Features Deep Dive ","date":"08-Nov-2025","heading":"Core Features Deep Dive","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:4:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#core-features-deep-dive"},{"categories":["projects"],"collections":null,"content":"1. Multi-Provider Email Architecture The system implements a factory pattern for seamless provider switching: \/\/ Email Provider Factory Pattern with Validation export class EmailFactory { static createProvider(config, env) { const provider = config.EMAIL_PROVIDER?.toLowerCase(); switch (provider) { case 'gmail': \/\/ Validate Gmail configuration const gmailValidation = GmailProvider.validateConfig(config); if (!gmailValidation.valid) { throw new Error(`Gmail config errors: ${gmailValidation.errors.join(', ')}`); } return new GmailProvider(config); case 'mailerlite': \/\/ Validate MailerLite configuration if (!config.MAILERLITE_API_TOKEN) { throw new Error('MailerLite API token required'); } return new MailerLiteProvider(config, env); case 'worker-email': \/\/ Validate Worker Email configuration const workerValidation = WorkerEmailProvider.validateConfig(config); if (!workerValidation.valid) { throw new Error(`Worker Email errors: ${workerValidation.errors.join(', ')}`); } return new WorkerEmailProvider(config, env); default: throw new Error(`Unknown provider: ${provider}`); } } \/\/ Send newsletter with automatic retry and fallback static async sendNewsletter(config, env, { recipients, post }) { const provider = this.createProvider(config, env); \/\/ Create responsive HTML email const html = this.createNewsletterHtml(post, config); const text = this.createNewsletterText(post, config); \/\/ Send with batch processing return await provider.sendBatchEmail({ recipients: recipients, subject: `New Post: ${post.title}`, html: html, text: text }); } } \ud83d\udce7 Email Provider Comparison Provider Best For Limits Cost Gmail SMTP Small lists 500\/day (free) Free MailerLite Growing lists 1000\/month free $10+ Worker Email Custom domains Unlimited* Pay-per-use ","date":"08-Nov-2025","heading":"1. Multi-Provider Email Architecture","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:4:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#1-multi-provider-email-architecture"},{"categories":["projects"],"collections":null,"content":"2. Intelligent Bot Protection Our system implements a progressive CAPTCHA system that only challenges suspicious users: State diagram showing intelligent bot protection that progressively challenges suspicious users with Turnstile CAPTCHA based on rate limiting and abuse detection ","date":"08-Nov-2025","heading":"2. Intelligent Bot Protection","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:4:2","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#2-intelligent-bot-protection"},{"categories":["projects"],"collections":null,"content":"3. Universal Feed Parser - RSS, Atom, JSON Feed The system includes a powerful universal feed parser that automatically detects and parses any feed format: \/\/ Universal Feed Parser - Supports multiple formats export function parseFeed(content, contentType = '') { \/\/ Auto-detect format const trimmed = content.trim(); \/\/ Check if it's JSON Feed if (trimmed.startsWith('{') || contentType.includes('json')) { return parseJsonFeed(trimmed); } \/\/ Otherwise, parse as XML-based feed return parseXmlFeed(trimmed); } \/\/ Intelligent format detection export function detectFeedType(content) { const trimmed = content.trim(); \/\/ JSON Feed detection if (trimmed.startsWith('{')) { const json = JSON.parse(trimmed); if (json.version?.includes('jsonfeed.org')) { return 'json-feed'; } } \/\/ Atom detection if (trimmed.includes('xmlns=\"http:\/\/www.w3.org\/2005\/Atom\"')) { return 'atom'; } \/\/ RSS 1.0\/RDF detection if (trimmed.includes(' \ud83d\udcf0 Supported Feed Formats Format Example Auto-Detected Features RSS 2.0 WordPress, Blogger \u2705 Most common, widely supported RSS 1.0\/RDF Older CMS systems \u2705 Namespace support Atom GitHub, modern blogs \u2705 Modern XML format JSON Feed Micro.blog, modern APIs \u2705 JSON-based, easy to parse Custom Namespaces Dublin Core, Content \u2705 dc:creator, content:encoded ","date":"08-Nov-2025","heading":"3. Universal Feed Parser - RSS, Atom, JSON Feed","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:4:3","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#3-universal-feed-parser---rss-atom-json-feed"},{"categories":["projects"],"collections":null,"content":"4. Automated RSS Processing Workflow \/\/ Complete RSS Processing Pipeline async function discoverFromRssAndQueue(env, config) { \/\/ 1. Validate feed URL if (!isValidFeedUrl(config.RSS_FEED_URL)) { console.error('Invalid feed URL'); return; } \/\/ 2. Fetch with proper headers const response = await fetch(config.RSS_FEED_URL, { headers: { 'User-Agent': config.USER_AGENT, 'Accept': 'application\/rss+xml, application\/atom+xml, application\/xml, text\/xml, application\/json, *\/*' } }); \/\/ 3. Parse with universal parser const feedContent = await response.text(); const feedType = detectFeedType(feedContent); console.log(`Detected feed type: ${feedType}`); const items = parseFeed(feedContent, response.headers.get('content-type')); \/\/ 4. Process new items for (const item of items) { \/\/ Check if already sent const already = await alreadySent(env, config, item.guid); if (already) continue; \/\/ Queue for sending await createEmailQueue({ post: { url: item.url, title: item.title, description: item.description, author: item.author, categories: item.categories, pubDate: item.pubDate }, subscribers: await getAllSubscribers(env, config), feedType: feedType }); } } ","date":"08-Nov-2025","heading":"4. Automated RSS Processing Workflow","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:4:4","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#4-automated-rss-processing-workflow"},{"categories":["projects"],"collections":null,"content":"Data Flow Diagrams ","date":"08-Nov-2025","heading":"Data Flow Diagrams","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:5:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#data-flow-diagrams"},{"categories":["projects"],"collections":null,"content":"Newsletter Subscription Flow Complete subscription flow showing form submission, Turnstile verification, KV storage check, welcome email sending, and GitHub backup queuing ","date":"08-Nov-2025","heading":"Newsletter Subscription Flow","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:5:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#newsletter-subscription-flow"},{"categories":["projects"],"collections":null,"content":"Batch Email Processing Email queue processing showing batch creation (100 emails each), sequential sending with 5-minute delays, retry logic, and dead letter queue for failed emails ","date":"08-Nov-2025","heading":"Batch Email Processing","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:5:2","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#batch-email-processing"},{"categories":["projects"],"collections":null,"content":"Quick Start Guide ","date":"08-Nov-2025","heading":"Quick Start Guide","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:6:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#quick-start-guide"},{"categories":["projects"],"collections":null,"content":"\ud83d\ude80 Deploy in 5 Minutes \u26a1 One-Click Deploy The fastest way to get started is to fork the repository and deploy with Wrangler CLI: # Quick deploy git clone https:\/\/github.com\/SamirPaulb\/newsletter-and-contact-system.git cd cloudflare-workers npm install wrangler deploy ","date":"08-Nov-2025","heading":"\ud83d\ude80 Deploy in 5 Minutes","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:6:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#-deploy-in-5-minutes"},{"categories":["projects"],"collections":null,"content":"Implementation Guide ","date":"08-Nov-2025","heading":"Implementation Guide","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:7:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#implementation-guide"},{"categories":["projects"],"collections":null,"content":"Step 1: Project Setup # Clone the repository git clone https:\/\/github.com\/SamirPaulb\/newsletter-and-contact-system.git cd cloudflare-workers # Install dependencies npm install # Configure wrangler.toml cp wrangler.example.toml wrangler.toml ","date":"08-Nov-2025","heading":"Step 1: Project Setup","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:7:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#step-1-project-setup"},{"categories":["projects"],"collections":null,"content":"Step 2: Configuration Edit wrangler.toml with your settings: name = \"newsletter-system\" main = \"src\/index.js\" compatibility_date = \"2024-12-31\" [triggers] crons = [ \"0 0 * * *\", # Daily at midnight UTC \"0 0 * * sat\" # Weekly on Saturday ] [[kv_namespaces]] binding = \"KV\" id = \"your-kv-namespace-id\" [vars] EMAIL_PROVIDER = \"gmail\" RSS_FEED_URL = \"https:\/\/your-site.com\/feed.xml\" BATCH_SIZE = 100 BATCH_WAIT_MINUTES = 5 ","date":"08-Nov-2025","heading":"Step 2: Configuration","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:7:2","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#step-2-configuration"},{"categories":["projects"],"collections":null,"content":"Step 3: Set Secrets # Required secrets wrangler secret put GMAIL_USER wrangler secret put GMAIL_PASSWORD wrangler secret put GITHUB_TOKEN wrangler secret put TURNSTILE_SITE_KEY wrangler secret put TURNSTILE_SECRET_KEY ","date":"08-Nov-2025","heading":"Step 3: Set Secrets","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:7:3","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#step-3-set-secrets"},{"categories":["projects"],"collections":null,"content":"Step 4: Deploy # Deploy to Cloudflare Workers wrangler deploy # Verify deployment curl https:\/\/your-worker.workers.dev\/health ","date":"08-Nov-2025","heading":"Step 4: Deploy","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:7:4","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#step-4-deploy"},{"categories":["projects"],"collections":null,"content":"Security Deep Dive ","date":"08-Nov-2025","heading":"Security Deep Dive","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:8:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#security-deep-dive"},{"categories":["projects"],"collections":null,"content":"Multi-Layer Defense Strategy Four-layer security architecture: Edge protection (DDoS, WAF), Application security (CORS, Turnstile), Data security (encryption, sanitization), and Access control (Zero Trust, RBAC) ","date":"08-Nov-2025","heading":"Multi-Layer Defense Strategy","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:8:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#multi-layer-defense-strategy"},{"categories":["projects"],"collections":null,"content":"Key Security Features Rate Limiting: Two-tier system (per-IP and global) CORS Protection: Restricted to your domain only Input Validation: Email validation with disposable domain blocking XSS Prevention: All user inputs sanitized Bot Protection: Progressive CAPTCHA with Turnstile Zero Trust Integration: Status page requires authentication \ud83d\udd12 Security Best Practices Never commit secrets to Git Use Cloudflare’s secret management Enable 2FA on all accounts Regularly rotate API keys Monitor rate limit logs ","date":"08-Nov-2025","heading":"Key Security Features","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:8:2","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#key-security-features"},{"categories":["projects"],"collections":null,"content":"Live Demo & User Interface ","date":"08-Nov-2025","heading":"Live Demo & User Interface","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:9:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#live-demo--user-interface"},{"categories":["projects"],"collections":null,"content":"\ud83c\udfa8 Interactive Forms Experience the system in action with these live, embedded forms: \ud83d\udcac Contact Form with Auto-Subscribe \ud83d\udcf8 Screenshot: Contact Form Comprehensive contact form interface with fields for name, email, phone, subject, and message. Features an auto-subscribe checkbox option and Cloudflare Turnstile CAPTCHA for security. The form provides instant validation feedback and confirmation emails upon submission \ud83d\udcec Newsletter Subscription Form \ud83d\udcf8 Screenshot: Subscription Form Clean newsletter subscription interface featuring an email input field and Cloudflare Turnstile CAPTCHA verification. The form ensures bot protection while maintaining user-friendly design with instant validation feedback \ud83d\udc4b Unsubscribe Page \ud83d\udcf8 Screenshot: Unsubscribe Page Simple unsubscribe interface with email input field and Turnstile CAPTCHA. Users can quickly opt-out from newsletters with a single click while the system maintains security through bot protection ","date":"08-Nov-2025","heading":"\ud83c\udfa8 Interactive Forms","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:9:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#-interactive-forms"},{"categories":["projects"],"collections":null,"content":"\ud83c\udfa8 Interactive Forms Experience the system in action with these live, embedded forms: \ud83d\udcac Contact Form with Auto-Subscribe \ud83d\udcf8 Screenshot: Contact Form Comprehensive contact form interface with fields for name, email, phone, subject, and message. Features an auto-subscribe checkbox option and Cloudflare Turnstile CAPTCHA for security. The form provides instant validation feedback and confirmation emails upon submission \ud83d\udcec Newsletter Subscription Form \ud83d\udcf8 Screenshot: Subscription Form Clean newsletter subscription interface featuring an email input field and Cloudflare Turnstile CAPTCHA verification. The form ensures bot protection while maintaining user-friendly design with instant validation feedback \ud83d\udc4b Unsubscribe Page \ud83d\udcf8 Screenshot: Unsubscribe Page Simple unsubscribe interface with email input field and Turnstile CAPTCHA. Users can quickly opt-out from newsletters with a single click while the system maintains security through bot protection ","date":"08-Nov-2025","heading":"\ud83d\udcac Contact Form with Auto-Subscribe","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:9:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#-contact-form-with-auto-subscribe"},{"categories":["projects"],"collections":null,"content":"\ud83c\udfa8 Interactive Forms Experience the system in action with these live, embedded forms: \ud83d\udcac Contact Form with Auto-Subscribe \ud83d\udcf8 Screenshot: Contact Form Comprehensive contact form interface with fields for name, email, phone, subject, and message. Features an auto-subscribe checkbox option and Cloudflare Turnstile CAPTCHA for security. The form provides instant validation feedback and confirmation emails upon submission \ud83d\udcec Newsletter Subscription Form \ud83d\udcf8 Screenshot: Subscription Form Clean newsletter subscription interface featuring an email input field and Cloudflare Turnstile CAPTCHA verification. The form ensures bot protection while maintaining user-friendly design with instant validation feedback \ud83d\udc4b Unsubscribe Page \ud83d\udcf8 Screenshot: Unsubscribe Page Simple unsubscribe interface with email input field and Turnstile CAPTCHA. Users can quickly opt-out from newsletters with a single click while the system maintains security through bot protection ","date":"08-Nov-2025","heading":"\ud83d\udcec Newsletter Subscription Form","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:9:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#-newsletter-subscription-form"},{"categories":["projects"],"collections":null,"content":"\ud83c\udfa8 Interactive Forms Experience the system in action with these live, embedded forms: \ud83d\udcac Contact Form with Auto-Subscribe \ud83d\udcf8 Screenshot: Contact Form Comprehensive contact form interface with fields for name, email, phone, subject, and message. Features an auto-subscribe checkbox option and Cloudflare Turnstile CAPTCHA for security. The form provides instant validation feedback and confirmation emails upon submission \ud83d\udcec Newsletter Subscription Form \ud83d\udcf8 Screenshot: Subscription Form Clean newsletter subscription interface featuring an email input field and Cloudflare Turnstile CAPTCHA verification. The form ensures bot protection while maintaining user-friendly design with instant validation feedback \ud83d\udc4b Unsubscribe Page \ud83d\udcf8 Screenshot: Unsubscribe Page Simple unsubscribe interface with email input field and Turnstile CAPTCHA. Users can quickly opt-out from newsletters with a single click while the system maintains security through bot protection ","date":"08-Nov-2025","heading":"\ud83d\udc4b Unsubscribe Page","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:9:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#-unsubscribe-page"},{"categories":["projects"],"collections":null,"content":"Performance & Scalability ","date":"08-Nov-2025","heading":"Performance & Scalability","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:10:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#performance--scalability"},{"categories":["projects"],"collections":null,"content":"Benchmark Results Load test results showing zero overhead scaling from 1 request\/second to 100,000 requests\/second with consistent performance \ud83d\udcca Performance Metrics Response Time: <50ms globally (P95) Throughput: 100,000+ requests\/second Availability: 99.99% SLA Storage: Unlimited KV pairs Email Sending: 500-2000\/day (provider dependent) ","date":"08-Nov-2025","heading":"Benchmark Results","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:10:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#benchmark-results"},{"categories":["projects"],"collections":null,"content":"Cost Analysis Subscribers Monthly Cost Cost per Subscriber 1,000 ~$5 $0.005 10,000 ~$20 $0.002 100,000 ~$75 $0.00075 1,000,000 ~$500 $0.0005 Based on Cloudflare Workers pricing with daily newsletter sends ","date":"08-Nov-2025","heading":"Cost Analysis","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:10:2","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#cost-analysis"},{"categories":["projects"],"collections":null,"content":"Advanced Features ","date":"08-Nov-2025","heading":"Advanced Features","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:11:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#advanced-features"},{"categories":["projects"],"collections":null,"content":"1. Webhook Integration \/\/ Webhook notification on new subscriber async function notifyWebhook(subscriber, config) { await fetch(config.WEBHOOK_URL, { method: 'POST', headers: { 'Content-Type': 'application\/json', 'X-Webhook-Secret': config.WEBHOOK_SECRET }, body: JSON.stringify({ event: 'subscriber.created', data: subscriber, timestamp: new Date().toISOString() }) }); } ","date":"08-Nov-2025","heading":"1. Webhook Integration","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:11:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#1-webhook-integration"},{"categories":["projects"],"collections":null,"content":"2. A\/B Testing Support \/\/ Split testing for email campaigns function selectEmailVariant(subscriber, variants) { const hash = cyrb53(subscriber.email); const bucketIndex = hash % variants.length; return variants[bucketIndex]; } ","date":"08-Nov-2025","heading":"2. A\/B Testing Support","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:11:2","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#2-ab-testing-support"},{"categories":["projects"],"collections":null,"content":"3. Analytics Integration \/\/ Track email opens with pixel tracking function generateTrackingPixel(emailId, subscriberId) { const trackingId = generateUUID(); return ``; } ","date":"08-Nov-2025","heading":"3. Analytics Integration","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:11:3","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#3-analytics-integration"},{"categories":["projects"],"collections":null,"content":"Monitoring & Maintenance ","date":"08-Nov-2025","heading":"Monitoring & Maintenance","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:12:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#monitoring--maintenance"},{"categories":["projects"],"collections":null,"content":"Automated Maintenance Tasks Saturday maintenance schedule showing 30 minutes for data cleanup, 20 minutes for GitHub backups, 10 minutes for report generation, and 10 minutes for integrity verification ","date":"08-Nov-2025","heading":"Automated Maintenance Tasks","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:12:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#automated-maintenance-tasks"},{"categories":["projects"],"collections":null,"content":"Health Monitoring Endpoints Endpoint Purpose Access \/health Basic health check Public \/status Detailed system status Zero Trust \/debug Configuration debug Zero Trust \/metrics Performance metrics Zero Trust \ud83d\udd0d Cloudflare Zero Trust Integration The status page is protected with Cloudflare Zero Trust for enterprise-grade security. Only authorized administrators can access system metrics and debug information. This prevents information disclosure while maintaining operational visibility. Cloudflare Zero Trust authentication interface protecting the admin status page. Shows the secure login prompt that prevents unauthorized access to sensitive system information while maintaining enterprise-grade security compliance Cloudflare Turnstile CAPTCHA to block unwanted bots Real-time system status dashboard showing KV storage statistics, subscriber count, email queue metrics, and system health indicators. Provides administrators with comprehensive monitoring of all newsletter system components and performance metrics ","date":"08-Nov-2025","heading":"Health Monitoring Endpoints","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:12:2","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#health-monitoring-endpoints"},{"categories":["projects"],"collections":null,"content":"Troubleshooting Guide ","date":"08-Nov-2025","heading":"Troubleshooting Guide","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:13:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#troubleshooting-guide"},{"categories":["projects"],"collections":null,"content":"Common Issues and Solutions Issue Cause Solution Emails not sending Invalid credentials Verify Gmail App Password Rate limit hit Too many requests Wait 24 hours or adjust limits Turnstile failing Invalid keys Check site\/secret key match KV write errors Quota exceeded Upgrade KV plan Backup failing GitHub token expired Regenerate PAT token ","date":"08-Nov-2025","heading":"Common Issues and Solutions","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:13:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#common-issues-and-solutions"},{"categories":["projects"],"collections":null,"content":"Debug Commands # Check worker logs wrangler tail # Test email sending curl -X POST https:\/\/your-worker.dev\/check-now # Verify configuration curl https:\/\/your-worker.dev\/debug # Run maintenance manually curl -X POST https:\/\/your-worker.dev\/maintenance ","date":"08-Nov-2025","heading":"Debug Commands","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:13:2","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#debug-commands"},{"categories":["projects"],"collections":null,"content":"Future Enhancements ","date":"08-Nov-2025","heading":"Future Enhancements","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:14:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#future-enhancements"},{"categories":["projects"],"collections":null,"content":"Roadmap GraphQL API for advanced integrations Multi-language support with i18n Template engine for custom email designs Segmentation for targeted campaigns Click tracking and engagement analytics Drip campaigns with automation workflows SMS notifications via Twilio integration Dashboard UI with Cloudflare Pages ","date":"08-Nov-2025","heading":"Roadmap","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:14:1","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#roadmap"},{"categories":["projects"],"collections":null,"content":"Conclusion Building a newsletter system with Cloudflare Workers demonstrates the power of edge computing. With zero infrastructure to manage, infinite scalability, and enterprise-grade security, this architecture represents the future of web applications. The combination of: \ud83c\udf0d Global edge deployment \ud83d\udce7 Multi-provider email support \ud83d\udee1\ufe0f Intelligent bot protection \ud83d\udcca Automated operations \ud83d\udcb0 Cost-effective scaling Makes this solution perfect for projects ranging from personal blogs to enterprise newsletters serving millions of subscribers. \ud83c\udfaf Key Takeaways Serverless is the future: No servers, no worries Edge computing wins: <50ms latency globally Security by default: Multiple layers of protection Infinitely scalable: Same code for 1 or 1M users Cost-effective: Pay only for what you use ","date":"08-Nov-2025","heading":"Conclusion","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:15:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#conclusion"},{"categories":["projects"],"collections":null,"content":"Get Started Today Ready to build your own infinitely scalable newsletter system? Fork the repository: github.com\/SamirPaulb\/newsletter-and-contact-system Follow the setup guide above Deploy in minutes with Wrangler Scale to millions without changing anything ","date":"08-Nov-2025","heading":"Get Started Today","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:16:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#get-started-today"},{"categories":["projects"],"collections":null,"content":"Share Your Feedback I’d love to hear about your experience building with Cloudflare Workers! Have you implemented similar systems? What challenges did you face? What features would you add? Please share your thoughts, questions, and suggestions in the comments below! Your feedback helps improve this project and guides future enhancements. If you found this guide helpful, please consider: \u2b50 Starring the GitHub repository \ud83d\udd04 Sharing with your network \ud83d\udcac Leaving a comment with your thoughts \ud83d\udc1b Reporting issues or suggesting features Happy building! \ud83d\ude80 ","date":"08-Nov-2025","heading":"Share Your Feedback","objectID":"\/posts\/serverless-newsletter-and-contact-system\/:17:0","tags":["cloudflare-workers","serverless","newsletter"],"title":"Building a Serverless Newsletter & Contact System That Scales to Millions","uri":"\/posts\/serverless-newsletter-and-contact-system\/#share-your-feedback"},{"categories":["software-engineering"],"collections":null,"content":"Master system design from basics to advanced. Learn to build scalable distributed systems with real-world examples \u2014 ideal for interviews or production-ready architecture.","date":"28-Jul-2025","objectID":"\/posts\/system-design\/","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/"},{"categories":["software-engineering"],"collections":null,"content":" About This Guide This comprehensive guide covers everything you need to know about system design, from basic concepts to advanced distributed systems. Perfect for software engineers preparing for system design interviews or architecting real-world applications. ","date":"28-Jul-2025","heading":"","objectID":"\/posts\/system-design\/:0:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#"},{"categories":["software-engineering"],"collections":null,"content":"1. Introduction to System Design System design is the process of defining the architecture, components, modules, interfaces, and data for a system to satisfy specified requirements. In the context of software engineering, it involves making high-level decisions about how to build scalable, reliable, and maintainable systems. ","date":"28-Jul-2025","heading":"1. Introduction to System Design","objectID":"\/posts\/system-design\/:1:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#1-introduction-to-system-design"},{"categories":["software-engineering"],"collections":null,"content":"Why System Design Matters Modern Application Requirements Modern applications serve millions or even billions of users simultaneously. These systems must be: Scalable: Handle growing amounts of work Reliable: Continue functioning under failures Available: Accessible when users need them Maintainable: Easy to update and modify Performant: Respond quickly to user requests Cost-effective: Operate within reasonable budgets ","date":"28-Jul-2025","heading":"Why System Design Matters","objectID":"\/posts\/system-design\/:1:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#why-system-design-matters"},{"categories":["software-engineering"],"collections":null,"content":"The Journey Ahead This guide takes you through a comprehensive exploration of system design, from fundamental concepts to complex real-world implementations. Each section builds upon previous knowledge, creating a complete understanding of how modern distributed systems work. ","date":"28-Jul-2025","heading":"The Journey Ahead","objectID":"\/posts\/system-design\/:1:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#the-journey-ahead"},{"categories":["software-engineering"],"collections":null,"content":"Learning Approach Throughout this guide, we’ll use: Visual diagrams (in Mermaid format) to illustrate architectures Real-world examples from companies like Netflix, Uber, and Twitter Code examples in Python to demonstrate concepts Trade-off analyses to understand decision-making Practical exercises to reinforce learning ","date":"28-Jul-2025","heading":"Learning Approach","objectID":"\/posts\/system-design\/:1:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#learning-approach"},{"categories":["software-engineering"],"collections":null,"content":"2. System Design Fundamentals ","date":"28-Jul-2025","heading":"2. System Design Fundamentals","objectID":"\/posts\/system-design\/:2:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#2-system-design-fundamentals"},{"categories":["software-engineering"],"collections":null,"content":"2.1 Scalability Scalability is the ability of a system to handle increased load by adding resources. There are two primary approaches: Vertical Scaling (Scale Up) Vertical scaling means upgrading the hardware of your existing server \u2014 adding more CPU cores, RAM, or faster disks. It is the simplest way to handle more load because your application code doesn’t change at all; you just run the same code on a more powerful machine. Why it works well initially: A single powerful machine can handle millions of operations per second. For most startups and early-stage products, a well-tuned single server (say, 64-core machine with 512 GB RAM) is far cheaper and simpler than distributing work across many machines. Airbnb and Twitter both started on a single database server. Why it eventually fails: Hardware has hard physical limits \u2014 you cannot buy a server with 1 million CPU cores. Beyond a certain point, the most powerful server available gets dramatically more expensive per unit of performance (the curve is non-linear). And critically, vertical scaling gives you a single point of failure \u2014 if that one machine dies, everything goes down. You also need downtime to upgrade hardware. Advantages: No application code changes \u2014 the simplest possible scale-up path Data consistency is trivial \u2014 everything is on one machine, no distributed coordination Lower operational complexity \u2014 one server to monitor, backup, and maintain Disadvantages: Hard upper limit \u2014 the most powerful server you can buy has finite capacity Single point of failure \u2014 one hardware failure takes down your entire system Expensive at the high end \u2014 doubling capacity can cost 4\u00d7 or 10\u00d7 the price Requires planned downtime to perform hardware upgrades Horizontal Scaling (Scale Out) Horizontal scaling means adding more machines and distributing the load across them. Instead of one big server, you have 10 (or 1000) commodity servers working together. This is the approach used by every major internet company at scale. Why it works at internet scale: You can add servers at any time without downtime. If one server fails, the others continue serving traffic. Commodity servers (standard AWS EC2 instances, etc.) follow normal economies of scale \u2014 doubling your server count roughly doubles your capacity at a predictable cost. The core challenge: Your application must be designed to run correctly across multiple machines. State (like user sessions) cannot live in one server’s memory \u2014 it must be stored externally (Redis, database). Requests from the same user might go to different servers. Keeping data consistent across nodes requires careful design (see \u00a713, \u00a720). Advantages: No theoretical upper limit \u2014 just keep adding machines Better fault tolerance \u2014 losing one of ten servers has minimal impact No downtime to scale \u2014 add servers without restarting anything Cost scales linearly \u2014 predictable cost as you grow Disadvantages: Application must be stateless \u2014 session data and state needs external storage Data consistency is harder \u2014 reads from different nodes may see different data Network overhead \u2014 inter-service calls add latency vs. in-process function calls Requires a load balancer \u2014 clients need a single endpoint that routes to any server Vertical vs Horizontal Scaling \u2014 Pros, cons, and use-cases for scaling strategies; essential for system design interviews ","date":"28-Jul-2025","heading":"2.1 Scalability","objectID":"\/posts\/system-design\/:2:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#21-scalability"},{"categories":["software-engineering"],"collections":null,"content":"2.1 Scalability Scalability is the ability of a system to handle increased load by adding resources. There are two primary approaches: Vertical Scaling (Scale Up) Vertical scaling means upgrading the hardware of your existing server \u2014 adding more CPU cores, RAM, or faster disks. It is the simplest way to handle more load because your application code doesn’t change at all; you just run the same code on a more powerful machine. Why it works well initially: A single powerful machine can handle millions of operations per second. For most startups and early-stage products, a well-tuned single server (say, 64-core machine with 512 GB RAM) is far cheaper and simpler than distributing work across many machines. Airbnb and Twitter both started on a single database server. Why it eventually fails: Hardware has hard physical limits \u2014 you cannot buy a server with 1 million CPU cores. Beyond a certain point, the most powerful server available gets dramatically more expensive per unit of performance (the curve is non-linear). And critically, vertical scaling gives you a single point of failure \u2014 if that one machine dies, everything goes down. You also need downtime to upgrade hardware. Advantages: No application code changes \u2014 the simplest possible scale-up path Data consistency is trivial \u2014 everything is on one machine, no distributed coordination Lower operational complexity \u2014 one server to monitor, backup, and maintain Disadvantages: Hard upper limit \u2014 the most powerful server you can buy has finite capacity Single point of failure \u2014 one hardware failure takes down your entire system Expensive at the high end \u2014 doubling capacity can cost 4\u00d7 or 10\u00d7 the price Requires planned downtime to perform hardware upgrades Horizontal Scaling (Scale Out) Horizontal scaling means adding more machines and distributing the load across them. Instead of one big server, you have 10 (or 1000) commodity servers working together. This is the approach used by every major internet company at scale. Why it works at internet scale: You can add servers at any time without downtime. If one server fails, the others continue serving traffic. Commodity servers (standard AWS EC2 instances, etc.) follow normal economies of scale \u2014 doubling your server count roughly doubles your capacity at a predictable cost. The core challenge: Your application must be designed to run correctly across multiple machines. State (like user sessions) cannot live in one server’s memory \u2014 it must be stored externally (Redis, database). Requests from the same user might go to different servers. Keeping data consistent across nodes requires careful design (see \u00a713, \u00a720). Advantages: No theoretical upper limit \u2014 just keep adding machines Better fault tolerance \u2014 losing one of ten servers has minimal impact No downtime to scale \u2014 add servers without restarting anything Cost scales linearly \u2014 predictable cost as you grow Disadvantages: Application must be stateless \u2014 session data and state needs external storage Data consistency is harder \u2014 reads from different nodes may see different data Network overhead \u2014 inter-service calls add latency vs. in-process function calls Requires a load balancer \u2014 clients need a single endpoint that routes to any server Vertical vs Horizontal Scaling \u2014 Pros, cons, and use-cases for scaling strategies; essential for system design interviews ","date":"28-Jul-2025","heading":"Vertical Scaling (Scale Up)","objectID":"\/posts\/system-design\/:2:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#vertical-scaling-scale-up"},{"categories":["software-engineering"],"collections":null,"content":"2.1 Scalability Scalability is the ability of a system to handle increased load by adding resources. There are two primary approaches: Vertical Scaling (Scale Up) Vertical scaling means upgrading the hardware of your existing server \u2014 adding more CPU cores, RAM, or faster disks. It is the simplest way to handle more load because your application code doesn’t change at all; you just run the same code on a more powerful machine. Why it works well initially: A single powerful machine can handle millions of operations per second. For most startups and early-stage products, a well-tuned single server (say, 64-core machine with 512 GB RAM) is far cheaper and simpler than distributing work across many machines. Airbnb and Twitter both started on a single database server. Why it eventually fails: Hardware has hard physical limits \u2014 you cannot buy a server with 1 million CPU cores. Beyond a certain point, the most powerful server available gets dramatically more expensive per unit of performance (the curve is non-linear). And critically, vertical scaling gives you a single point of failure \u2014 if that one machine dies, everything goes down. You also need downtime to upgrade hardware. Advantages: No application code changes \u2014 the simplest possible scale-up path Data consistency is trivial \u2014 everything is on one machine, no distributed coordination Lower operational complexity \u2014 one server to monitor, backup, and maintain Disadvantages: Hard upper limit \u2014 the most powerful server you can buy has finite capacity Single point of failure \u2014 one hardware failure takes down your entire system Expensive at the high end \u2014 doubling capacity can cost 4\u00d7 or 10\u00d7 the price Requires planned downtime to perform hardware upgrades Horizontal Scaling (Scale Out) Horizontal scaling means adding more machines and distributing the load across them. Instead of one big server, you have 10 (or 1000) commodity servers working together. This is the approach used by every major internet company at scale. Why it works at internet scale: You can add servers at any time without downtime. If one server fails, the others continue serving traffic. Commodity servers (standard AWS EC2 instances, etc.) follow normal economies of scale \u2014 doubling your server count roughly doubles your capacity at a predictable cost. The core challenge: Your application must be designed to run correctly across multiple machines. State (like user sessions) cannot live in one server’s memory \u2014 it must be stored externally (Redis, database). Requests from the same user might go to different servers. Keeping data consistent across nodes requires careful design (see \u00a713, \u00a720). Advantages: No theoretical upper limit \u2014 just keep adding machines Better fault tolerance \u2014 losing one of ten servers has minimal impact No downtime to scale \u2014 add servers without restarting anything Cost scales linearly \u2014 predictable cost as you grow Disadvantages: Application must be stateless \u2014 session data and state needs external storage Data consistency is harder \u2014 reads from different nodes may see different data Network overhead \u2014 inter-service calls add latency vs. in-process function calls Requires a load balancer \u2014 clients need a single endpoint that routes to any server Vertical vs Horizontal Scaling \u2014 Pros, cons, and use-cases for scaling strategies; essential for system design interviews ","date":"28-Jul-2025","heading":"Horizontal Scaling (Scale Out)","objectID":"\/posts\/system-design\/:2:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#horizontal-scaling-scale-out"},{"categories":["software-engineering"],"collections":null,"content":"2.2 Performance vs Scalability These concepts are often confused but have fundamentally different causes and solutions. Mixing them up leads to wasted engineering effort. Performance problem: Your system is slow even with a single user. The bottleneck is in the code itself \u2014 an inefficient algorithm, a missing database index, an N+1 query that fires 1000 SQL calls to build one response, or a blocking synchronous call that holds up the entire request. How to diagnose: Profile a single request in isolation. If it’s slow with zero concurrent users, you have a performance problem. Tools: EXPLAIN on SQL queries, profilers (cProfile, py-spy), APM tools (Datadog, New Relic). How to fix: Optimize the slow code \u2014 add database indexes, fix inefficient queries, cache computed results, upgrade to a faster algorithm. These fixes are often 10\u2013100\u00d7 improvements with no infrastructure changes. Scalability problem: Your system works fine for one user but degrades (or crashes) as concurrent users increase. A single request might take 50ms, but at 10,000 concurrent requests the average climbs to 5 seconds. The bottleneck is resource contention \u2014 too many requests competing for the same CPU, database connections, or memory. How to diagnose: Run a load test (k6, Locust, JMeter). If p50 latency is fine but p99 is terrible, or latency is fine at 100 RPS but terrible at 1,000 RPS, you have a scalability problem. How to fix: Add capacity \u2014 more servers, more database replicas, a caching layer to absorb reads, connection pooling to share database connections, or async processing to decouple slow operations from the critical path. Don’t Confuse the Two Adding servers to a system with a performance problem won’t help much \u2014 if each request takes 5 seconds because of a bad query, having 10 servers just means 10 servers each taking 5 seconds. Fix the performance problem first, then scale. ","date":"28-Jul-2025","heading":"2.2 Performance vs Scalability","objectID":"\/posts\/system-design\/:2:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#22-performance-vs-scalability"},{"categories":["software-engineering"],"collections":null,"content":"2.3 Latency vs Throughput Latency is how long a single operation takes from start to finish \u2014 the time one request waits before it gets a response. Measured in milliseconds (ms) or microseconds (\u00b5s). Example: a database query completes in 5ms. Throughput is how many operations a system can complete per unit of time \u2014 measured in requests per second (RPS) or queries per second (QPS). Example: a system handles 50,000 database reads per second. The key insight \u2014 they are often in tension with each other: To maximize throughput, you often want to batch operations: instead of writing each row to disk immediately, buffer 1,000 writes and flush them together. This dramatically increases throughput (fewer disk seeks, better compression), but it increases the latency of each individual write because it now has to wait in the buffer. Conversely, minimizing latency \u2014 responding to each request immediately \u2014 often means lower throughput, because you can’t take advantage of batching. Real examples: Kafka trades higher individual-message latency (messages are batched before sending) for enormous throughput (millions of messages\/second per broker) Databases trade latency for throughput via write-ahead logs and group commits CDN caching improves both \u2014 cache hits have lower latency AND the origin server gets less load (higher effective throughput) Design implication: You need to know your application’s priority. An online transaction (credit card charge) must have low latency \u2014 waiting 2 seconds is unacceptable. A data analytics pipeline processes terabytes of events \u2014 throughput is what matters, and 30-second batch latency is fine. Design accordingly. Amdahl’s Law Even if you parallelize 90% of a task, the serial 10% becomes the bottleneck. A system processing requests in parallel still has serial sections (database writes, locking) that cap throughput. Identifying and optimizing the serial bottleneck gives the biggest gains. ","date":"28-Jul-2025","heading":"2.3 Latency vs Throughput","objectID":"\/posts\/system-design\/:2:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#23-latency-vs-throughput"},{"categories":["software-engineering"],"collections":null,"content":"2.4 Key Latency Numbers Every Engineer Should Know Latency Reference Numbers Understanding these numbers helps make informed design decisions: Operation Latency Relative Scale L1 cache reference 0.5 ns 1x Branch mispredict 5 ns 10x L2 cache reference 7 ns 14x Mutex lock\/unlock 100 ns 200x Main memory reference 100 ns 200x Compress 1KB with Snappy 10,000 ns (10 \u00b5s) 20,000x Send 1KB over 1 Gbps network 10,000 ns (10 \u00b5s) 20,000x Read 1 MB sequentially from memory 250,000 ns (250 \u00b5s) 500,000x Round trip within same datacenter 500,000 ns (500 \u00b5s) 1,000,000x Read 1 MB sequentially from SSD 1,000,000 ns (1 ms) 2,000,000x Disk seek 10,000,000 ns (10 ms) 20,000,000x Read 1 MB sequentially from disk 30,000,000 ns (30 ms) 60,000,000x Send packet CA\u2192Netherlands\u2192CA 150,000,000 ns (150 ms) 300,000,000x Key Takeaways Memory is fast, disk is slow SSD is 20x faster than traditional disk Network calls within datacenter are relatively fast Cross-continent network calls are expensive Always prefer sequential reads over random seeks ","date":"28-Jul-2025","heading":"2.4 Key Latency Numbers Every Engineer Should Know","objectID":"\/posts\/system-design\/:2:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#24-key-latency-numbers-every-engineer-should-know"},{"categories":["software-engineering"],"collections":null,"content":"2.5 Availability Availability is the percentage of time a system is operational and accessible. Availability Calculation: Availability = (Total Time - Downtime) \/ Total Time \u00d7 100% Standard Availability Tiers (The Nines): Availability Downtime per Year Downtime per Month Downtime per Week 99% (two nines) 3.65 days 7.31 hours 1.68 hours 99.9% (three nines) 8.77 hours 43.83 minutes 10.08 minutes 99.99% (four nines) 52.60 minutes 4.38 minutes 1.01 minutes 99.999% (five nines) 5.26 minutes 26.30 seconds 6.05 seconds 99.9999% (six nines) 31.56 seconds 2.63 seconds 0.61 seconds Achieving High Availability: The core principle is: eliminate every single point of failure. A single point of failure (SPOF) is any component whose failure brings the entire system down. Every SPOF you identify and add redundancy to improves your availability. Eliminate Single Points of Failure (SPOF) \u2014 Run at least two instances of every critical component. If you have one database server and it crashes, everything breaks. If you have a primary + replica with automatic failover, the replica takes over in seconds. Apply this thinking everywhere: two load balancers, multiple web servers, replicated databases, redundant network links. For critical systems, deploy across multiple availability zones (AZs) or regions so a datacenter failure doesn’t take you down. Implement Health Checks \u2014 Load balancers and orchestration systems (Kubernetes) constantly probe your services with health check requests (e.g., GET \/health). When a service stops responding or returns errors, traffic is automatically stopped to that instance and rerouted to healthy ones. A service can fail silently (the process keeps running but stops serving correct responses) \u2014 health checks detect this. The response must come back in under a timeout (typically 2\u20135 seconds) or the check fails. Use Load Balancers \u2014 A load balancer sits in front of your application servers and distributes incoming requests across all healthy instances. If one server fails, the load balancer detects it via health checks and stops sending traffic there \u2014 users experience no disruption. Load balancers themselves can be a SPOF, so production systems run two load balancers (active-passive or active-active). Cloud providers (AWS ELB, GCP Load Balancer) handle this for you automatically. Design for Graceful Degradation \u2014 When a non-critical dependency is down, return a degraded but functional response rather than failing completely. Amazon’s product pages still show the product even if the “Customers who bought this also bought” recommendation service is down \u2014 they just hide that section. Circuit breakers (see \u00a715.2) implement this pattern automatically. Plan for Recovery, Not Just Prevention \u2014 No matter how redundant your system is, failures will occur. Define your RTO (Recovery Time Objective \u2014 how quickly must service be restored?) and RPO (Recovery Point Objective \u2014 how much data loss is acceptable?). Test your failover procedures regularly; a failover plan that has never been tested will fail when you need it. ","date":"28-Jul-2025","heading":"2.5 Availability","objectID":"\/posts\/system-design\/:2:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#25-availability"},{"categories":["software-engineering"],"collections":null,"content":"3. Back-of-the-Envelope Estimation Being able to quickly estimate system requirements is crucial for system design. This section covers the estimation techniques used in interviews and real-world planning. ","date":"28-Jul-2025","heading":"3. Back-of-the-Envelope Estimation","objectID":"\/posts\/system-design\/:3:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#3-back-of-the-envelope-estimation"},{"categories":["software-engineering"],"collections":null,"content":"3.1 Power of Two Understand data volume units: Power Exact Value Approximate Value Short Name 10 1,024 1 thousand 1 KB 20 1,048,576 1 million 1 MB 30 1,073,741,824 1 billion 1 GB 40 1,099,511,627,776 1 trillion 1 TB 50 1,125,899,906,842,624 1 quadrillion 1 PB ","date":"28-Jul-2025","heading":"3.1 Power of Two","objectID":"\/posts\/system-design\/:3:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#31-power-of-two"},{"categories":["software-engineering"],"collections":null,"content":"3.2 Common Performance Numbers These are rough but useful benchmarks to anchor your estimates. You don’t need to memorize exact numbers \u2014 the order of magnitude is what matters in an interview or capacity planning session. Resource Typical Capacity Why It Matters Single web server (8-core) 1,000\u20135,000 RPS Baseline for estimating how many servers you need Database connection pool 50\u2013100 connections per instance Connections are expensive; exceeding this causes queuing Single MySQL read (indexed) ~10,000 QPS How many servers you need for your read traffic Single MySQL write ~2,000\u20135,000 QPS Writes are slower; often the bottleneck Redis throughput 100,000\u2013200,000 ops\/sec Why Redis works as a cache in front of databases Average web page size 1\u20132 MB Used to estimate egress bandwidth cost Video streaming (1080p) 3\u20138 Mbps Used to size CDN and origin bandwidth Mobile API calls per session 5\u201310 requests Used to estimate total API traffic from DAU How to use these numbers: Say you’re designing a system with 10 million daily active users (DAU), each making 10 API calls per day. That’s 100 million API calls\/day \u00f7 86,400 seconds \u2248 1,160 RPS average. With a 3\u00d7 peak factor, you need to handle ~3,500 RPS peak. One well-tuned web server handles ~2,000\u20133,000 RPS, so you need 2\u20133 web servers as a baseline \u2014 before caching, before CDN. If 30% of those calls hit the database directly (the rest served from cache), that’s ~350 QPS to the database at peak. A single MySQL replica handles 5,000\u201310,000 reads\/sec, so one read replica with a cache layer is sufficient at this scale. This is the thinking interviewers look for \u2014 not exact numbers, but using rough benchmarks to reach reasonable architectural decisions. ","date":"28-Jul-2025","heading":"3.2 Common Performance Numbers","objectID":"\/posts\/system-design\/:3:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#32-common-performance-numbers"},{"categories":["software-engineering"],"collections":null,"content":"3.3 Example Estimation: Twitter-Like System Twitter System Requirements Given Requirements: 300 million monthly active users (MAU) 50% use Twitter daily Average user posts 2 tweets per day 10% of tweets contain media Data retained for 5 years Calculations: Step-by-Step Calculations Daily Active Users (DAU): DAU = 300M \u00d7 50% = 150M Tweet Traffic: Tweets per day = 150M \u00d7 2 = 300M Tweets per second (average) = 300M \/ 86400 \u2248 3,500 tweets\/second Peak QPS = 3,500 \u00d7 2 = 7,000 tweets\/second (assuming 2x peak factor) Storage Estimation: Average tweet size: - Tweet text: 280 chars \u00d7 2 bytes = 560 bytes - Metadata (ID, timestamp, user_id, etc.): 200 bytes - Total per tweet: ~800 bytes \u2248 1 KB Media tweets (10%): - Image average: 500 KB - Total per media tweet: 500 KB + 1 KB \u2248 500 KB Daily storage: - Text tweets: 270M \u00d7 1 KB = 270 GB - Media tweets: 30M \u00d7 500 KB = 15 TB - Total per day = 15.27 TB 5-year storage: 15.27 TB\/day \u00d7 365 days \u00d7 5 years \u2248 27.9 PB Bandwidth Estimation: Ingress (upload): 15.27 TB\/day \u00f7 86,400 seconds \u2248 177 MB\/second Egress (assume each tweet viewed 10 times): 177 MB\/s \u00d7 10 = 1.77 GB\/second Memory for Cache (80-20 rule): 20% of tweets generate 80% of traffic Daily cache: 15.27 TB \u00d7 0.2 = 3 TB ","date":"28-Jul-2025","heading":"3.3 Example Estimation: Twitter-Like System","objectID":"\/posts\/system-design\/:3:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#33-example-estimation-twitter-like-system"},{"categories":["software-engineering"],"collections":null,"content":"4. The System Design Interview Framework A systematic approach to solving any system design problem. System Design Interview Framework \u2014 4-step approach: requirements, high-level design, deep dive, and wrap-up for FAANG interviews ","date":"28-Jul-2025","heading":"4. The System Design Interview Framework","objectID":"\/posts\/system-design\/:4:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#4-the-system-design-interview-framework"},{"categories":["software-engineering"],"collections":null,"content":"4.1 The 4-Step Framework System Design Interview Steps Step 1: Understand the Problem and Establish Design Scope (3-10 minutes) Ask clarifying questions: Who are the users? How many users? What features are essential? What is the scale we’re designing for? What is the expected traffic pattern? What technologies does the company use? Step 2: Propose High-Level Design and Get Buy-In (10-15 minutes) Key actions: Draw initial architecture diagram Identify major components Explain data flow Get feedback from interviewer Step 3: Design Deep Dive (10-25 minutes) Focus areas: Dig into 2-3 components based on interviewer interest Discuss trade-offs Address bottlenecks Consider edge cases Step 4: Wrap Up (3-5 minutes) Final touches: Identify system bottlenecks Discuss potential improvements Recap design decisions Mention monitoring and operations ","date":"28-Jul-2025","heading":"4.1 The 4-Step Framework","objectID":"\/posts\/system-design\/:4:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#41-the-4-step-framework"},{"categories":["software-engineering"],"collections":null,"content":"4.1 The 4-Step Framework System Design Interview Steps Step 1: Understand the Problem and Establish Design Scope (3-10 minutes) Ask clarifying questions: Who are the users? How many users? What features are essential? What is the scale we’re designing for? What is the expected traffic pattern? What technologies does the company use? Step 2: Propose High-Level Design and Get Buy-In (10-15 minutes) Key actions: Draw initial architecture diagram Identify major components Explain data flow Get feedback from interviewer Step 3: Design Deep Dive (10-25 minutes) Focus areas: Dig into 2-3 components based on interviewer interest Discuss trade-offs Address bottlenecks Consider edge cases Step 4: Wrap Up (3-5 minutes) Final touches: Identify system bottlenecks Discuss potential improvements Recap design decisions Mention monitoring and operations ","date":"28-Jul-2025","heading":"Step 1: Understand the Problem and Establish Design Scope (3-10 minutes)","objectID":"\/posts\/system-design\/:4:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#step-1-understand-the-problem-and-establish-design-scope-3-10-minutes"},{"categories":["software-engineering"],"collections":null,"content":"4.1 The 4-Step Framework System Design Interview Steps Step 1: Understand the Problem and Establish Design Scope (3-10 minutes) Ask clarifying questions: Who are the users? How many users? What features are essential? What is the scale we’re designing for? What is the expected traffic pattern? What technologies does the company use? Step 2: Propose High-Level Design and Get Buy-In (10-15 minutes) Key actions: Draw initial architecture diagram Identify major components Explain data flow Get feedback from interviewer Step 3: Design Deep Dive (10-25 minutes) Focus areas: Dig into 2-3 components based on interviewer interest Discuss trade-offs Address bottlenecks Consider edge cases Step 4: Wrap Up (3-5 minutes) Final touches: Identify system bottlenecks Discuss potential improvements Recap design decisions Mention monitoring and operations ","date":"28-Jul-2025","heading":"Step 2: Propose High-Level Design and Get Buy-In (10-15 minutes)","objectID":"\/posts\/system-design\/:4:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#step-2-propose-high-level-design-and-get-buy-in-10-15-minutes"},{"categories":["software-engineering"],"collections":null,"content":"4.1 The 4-Step Framework System Design Interview Steps Step 1: Understand the Problem and Establish Design Scope (3-10 minutes) Ask clarifying questions: Who are the users? How many users? What features are essential? What is the scale we’re designing for? What is the expected traffic pattern? What technologies does the company use? Step 2: Propose High-Level Design and Get Buy-In (10-15 minutes) Key actions: Draw initial architecture diagram Identify major components Explain data flow Get feedback from interviewer Step 3: Design Deep Dive (10-25 minutes) Focus areas: Dig into 2-3 components based on interviewer interest Discuss trade-offs Address bottlenecks Consider edge cases Step 4: Wrap Up (3-5 minutes) Final touches: Identify system bottlenecks Discuss potential improvements Recap design decisions Mention monitoring and operations ","date":"28-Jul-2025","heading":"Step 3: Design Deep Dive (10-25 minutes)","objectID":"\/posts\/system-design\/:4:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#step-3-design-deep-dive-10-25-minutes"},{"categories":["software-engineering"],"collections":null,"content":"4.1 The 4-Step Framework System Design Interview Steps Step 1: Understand the Problem and Establish Design Scope (3-10 minutes) Ask clarifying questions: Who are the users? How many users? What features are essential? What is the scale we’re designing for? What is the expected traffic pattern? What technologies does the company use? Step 2: Propose High-Level Design and Get Buy-In (10-15 minutes) Key actions: Draw initial architecture diagram Identify major components Explain data flow Get feedback from interviewer Step 3: Design Deep Dive (10-25 minutes) Focus areas: Dig into 2-3 components based on interviewer interest Discuss trade-offs Address bottlenecks Consider edge cases Step 4: Wrap Up (3-5 minutes) Final touches: Identify system bottlenecks Discuss potential improvements Recap design decisions Mention monitoring and operations ","date":"28-Jul-2025","heading":"Step 4: Wrap Up (3-5 minutes)","objectID":"\/posts\/system-design\/:4:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#step-4-wrap-up-3-5-minutes"},{"categories":["software-engineering"],"collections":null,"content":"4.2 Key Principles Start Simple, Then Iterate Begin with a basic design Add complexity incrementally Justify each addition Focus on What Matters Prioritize based on requirements Don’t over-engineer Keep the user experience in mind Think About Trade-offs No perfect solution exists Discuss pros and cons Choose based on requirements Use Real Numbers Apply back-of-the-envelope calculations Validate your assumptions Show quantitative reasoning ","date":"28-Jul-2025","heading":"4.2 Key Principles","objectID":"\/posts\/system-design\/:4:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#42-key-principles"},{"categories":["software-engineering"],"collections":null,"content":"5. Networking and Communication Network Fundamentals Understanding how data moves through networks is fundamental to system design. This section covers essential networking concepts that impact system performance, reliability, and scalability. ","date":"28-Jul-2025","heading":"5. Networking and Communication","objectID":"\/posts\/system-design\/:5:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#5-networking-and-communication"},{"categories":["software-engineering"],"collections":null,"content":"5.1 The OSI Model and TCP\/IP Model Overview The OSI (Open Systems Interconnection) model has 7 layers, but for practical system design, focus on the TCP\/IP model, which is more pragmatic and widely used in modern applications. OSI and TCP\/IP Network Models \u2014 Understanding network layers for architecture design and optimization ","date":"28-Jul-2025","heading":"5.1 The OSI Model and TCP\/IP","objectID":"\/posts\/system-design\/:5:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#51-the-osi-model-and-tcpip"},{"categories":["software-engineering"],"collections":null,"content":"5.2 HTTP and HTTPS HTTP (Hypertext Transfer Protocol) is the foundation of data communication on the web. Common HTTP Methods: Method Purpose Idempotent Safe GET Retrieve data Yes Yes POST Create resource No No PUT Update\/Replace resource Yes No PATCH Partial update No No DELETE Remove resource Yes No HEAD Get headers only Yes Yes OPTIONS Query methods Yes Yes HTTP Status Codes: Common HTTP Status Codes 1xx: Informational 100 Continue 101 Switching Protocols 2xx: Success 200 OK 201 Created 202 Accepted 204 No Content 3xx: Redirection 301 Moved Permanently 302 Found (Temporary Redirect) 304 Not Modified 307 Temporary Redirect 308 Permanent Redirect 4xx: Client Errors 400 Bad Request 401 Unauthorized 403 Forbidden 404 Not Found 409 Conflict 429 Too Many Requests 5xx: Server Errors 500 Internal Server Error 502 Bad Gateway 503 Service Unavailable 504 Gateway Timeout HTTPS = HTTP + SSL\/TLS HTTPS adds encryption and authentication through SSL\/TLS: SSL\/TLS Handshake Process \u2014 How HTTPS secures data; critical for system security and authentication ","date":"28-Jul-2025","heading":"5.2 HTTP and HTTPS","objectID":"\/posts\/system-design\/:5:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#52-http-and-https"},{"categories":["software-engineering"],"collections":null,"content":"5.3 REST vs GraphQL vs gRPC vs WebSockets REST (Representational State Transfer) Characteristics: Resource-based (nouns in URLs) Stateless Uses standard HTTP methods Returns JSON or XML Example: GET \/api\/users\/123 GET \/api\/users\/123\/posts POST \/api\/users\/123\/posts PUT \/api\/users\/123 DELETE \/api\/users\/123 Pros: Simple and widely understood Cacheable Good tooling support Cons: Over-fetching or under-fetching data Multiple round trips for related data Versioning can be challenging GraphQL Characteristics: Client specifies exactly what data it needs Single endpoint Strongly typed schema Real-time updates via subscriptions Example: query { user(id: \"123\") { name email posts(limit: 10) { title createdAt } } } Pros: No over-fetching\/under-fetching Single request for complex data Self-documenting API Cons: More complex backend Caching is harder Potential for expensive queries gRPC (Google Remote Procedure Call) Characteristics: Uses Protocol Buffers (binary format) HTTP\/2 based Strongly typed Bi-directional streaming Example (protobuf): service UserService { rpc GetUser (UserRequest) returns (UserResponse); rpc ListUsers (ListUsersRequest) returns (stream UserResponse); } message UserRequest { string user_id = 1; } message UserResponse { string user_id = 1; string name = 2; string email = 3; } Pros: Very fast (binary protocol) Streaming support Strong typing Cons: Not browser-friendly Less human-readable Steeper learning curve WebSockets Characteristics: Full-duplex communication Persistent connection Real-time bidirectional data flow Use cases: Chat applications Live feeds Real-time gaming Collaborative editing WebSocket Connection Flow \u2014 Real-time communication protocol for live applications and low-latency messaging ","date":"28-Jul-2025","heading":"5.3 REST vs GraphQL vs gRPC vs WebSockets","objectID":"\/posts\/system-design\/:5:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#53-rest-vs-graphql-vs-grpc-vs-websockets"},{"categories":["software-engineering"],"collections":null,"content":"5.4 API Authentication 1. Basic Authentication Authorization: Basic base64(username:password) Simple but insecure (credentials in every request) Use only over HTTPS 2. API Keys X-API-Key: your-api-key-here Simple to implement Good for server-to-server Hard to revoke individual keys 3. OAuth 2.0 Most common flows: Authorization Code Flow (for web apps): OAuth 2.0 Authorization Flow \u2014 Secure user authentication and third-party app authorization pattern 4. JWT (JSON Web Tokens) Structure: header.payload.signature \/\/ Header { \"alg\": \"HS256\", \"typ\": \"JWT\" } \/\/ Payload { \"sub\": \"user123\", \"name\": \"John Doe\", \"iat\": 1516239022, \"exp\": 1516242622 } \/\/ Signature HMACSHA256( base64UrlEncode(header) + \".\" + base64UrlEncode(payload), secret ) Pros: Stateless (no server-side session storage) Portable across domains Contains user information Cons: Can’t revoke before expiration Token size can be large Vulnerable if secret is exposed ","date":"28-Jul-2025","heading":"5.4 API Authentication","objectID":"\/posts\/system-design\/:5:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#54-api-authentication"},{"categories":["software-engineering"],"collections":null,"content":"6. Load Balancing Load balancers distribute incoming traffic across multiple servers, improving availability and preventing any single server from becoming a bottleneck. ","date":"28-Jul-2025","heading":"6. Load Balancing","objectID":"\/posts\/system-design\/:6:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#6-load-balancing"},{"categories":["software-engineering"],"collections":null,"content":"6.1 Why Load Balancers? Benefits: High Availability: If one server fails, traffic routes to healthy servers Scalability: Add more servers to handle increased load Performance: Distribute load evenly Flexibility: Perform maintenance without downtime ","date":"28-Jul-2025","heading":"6.1 Why Load Balancers?","objectID":"\/posts\/system-design\/:6:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#61-why-load-balancers"},{"categories":["software-engineering"],"collections":null,"content":"6.2 Load Balancer Architecture Load Balancer Architecture \u2014 How to distribute traffic and ensure high availability in production systems ","date":"28-Jul-2025","heading":"6.2 Load Balancer Architecture","objectID":"\/posts\/system-design\/:6:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#62-load-balancer-architecture"},{"categories":["software-engineering"],"collections":null,"content":"6.3 Load Balancing Algorithms Tip Choosing the Right Algorithm: The best load balancing algorithm depends on your specific use case. For stateless applications with similar request complexity, Round Robin works well. For long-lived connections (like WebSockets), use Least Connections. For session affinity, use IP Hash or Consistent Hashing. 1. Round Robin Distributes requests sequentially Simple and fair distribution Doesn’t account for server load or capacity class RoundRobinLoadBalancer: def __init__(self, servers): self.servers = servers self.current = 0 def get_server(self): server = self.servers[self.current] self.current = (self.current + 1) % len(self.servers) return server 2. Weighted Round Robin Assigns weight to each server based on capacity Servers with higher capacity get more requests class WeightedRoundRobinLoadBalancer: def __init__(self, servers_with_weights): # servers_with_weights = [('server1', 5), ('server2', 3), ('server3', 2)] self.servers = [] for server, weight in servers_with_weights: self.servers.extend([server] * weight) self.current = 0 def get_server(self): server = self.servers[self.current] self.current = (self.current + 1) % len(self.servers) return server 3. Least Connections Routes to server with fewest active connections Good for long-lived connections class LeastConnectionsLoadBalancer: def __init__(self, servers): self.connections = {server: 0 for server in servers} def get_server(self): return min(self.connections, key=self.connections.get) def on_connection_established(self, server): self.connections[server] += 1 def on_connection_closed(self, server): self.connections[server] -= 1 4. Least Response Time Routes to server with lowest average response time Dynamically adapts to server performance 5. IP Hash Uses client IP to determine server Ensures same client always routes to same server Good for session persistence import hashlib class IPHashLoadBalancer: def __init__(self, servers): self.servers = servers def get_server(self, client_ip): hash_value = int(hashlib.md5(client_ip.encode()).hexdigest(), 16) server_index = hash_value % len(self.servers) return self.servers[server_index] 6. Random Randomly selects server Simple and works well with many servers ","date":"28-Jul-2025","heading":"6.3 Load Balancing Algorithms","objectID":"\/posts\/system-design\/:6:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#63-load-balancing-algorithms"},{"categories":["software-engineering"],"collections":null,"content":"6.4 Load Balancer Layers Note Layer 4 vs Layer 7 Trade-off: Layer 4 load balancing is faster (10-100 microseconds latency) but less intelligent. Layer 7 adds 1-10 milliseconds latency but enables content-based routing, URL rewriting, and application-aware decisions. Use Layer 4 for raw throughput, Layer 7 for microservices architectures. Layer 4 (Transport Layer) Load Balancing Operates at TCP\/UDP level Routes based on IP and port Fast (no packet inspection) Cannot make routing decisions based on content Layer 7 (Application Layer) Load Balancing Operates at HTTP level Can route based on URL, headers, cookies More intelligent routing Slower (needs to inspect packets) Can do SSL termination Layer 4 vs Layer 7 Load Balancing \u2014 Understanding transport vs application layer routing and performance trade-offs ","date":"28-Jul-2025","heading":"6.4 Load Balancer Layers","objectID":"\/posts\/system-design\/:6:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#64-load-balancer-layers"},{"categories":["software-engineering"],"collections":null,"content":"6.5 Health Checks Load balancers need to know which servers are healthy: import time import requests class HealthChecker: def __init__(self, servers, check_interval=10): self.servers = servers self.healthy_servers = set(servers) self.check_interval = check_interval def check_health(self, server): try: response = requests.get(f\"{server}\/health\", timeout=2) return response.status_code == 200 except: return False def monitor(self): while True: for server in self.servers: if self.check_health(server): self.healthy_servers.add(server) else: self.healthy_servers.discard(server) time.sleep(self.check_interval) def get_healthy_servers(self): return list(self.healthy_servers) ","date":"28-Jul-2025","heading":"6.5 Health Checks","objectID":"\/posts\/system-design\/:6:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#65-health-checks"},{"categories":["software-engineering"],"collections":null,"content":"6.6 Session Persistence (Sticky Sessions) Challenge: With load balancing, subsequent requests from the same user might go to different servers. Solutions: 1. Sticky Sessions at Load Balancer Load balancer remembers which server served a client Routes all requests from that client to same server Problem: If server dies, session is lost 2. Shared Session Storage Store sessions in Redis\/Memcached Any server can serve any request More scalable and fault-tolerant Shared Session Storage \u2014 How to scale stateless servers using centralized session management with Redis 3. Client-Side Sessions (JWT) Session data stored in token on client Stateless servers Token passed with each request ","date":"28-Jul-2025","heading":"6.6 Session Persistence (Sticky Sessions)","objectID":"\/posts\/system-design\/:6:6","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#66-session-persistence-sticky-sessions"},{"categories":["software-engineering"],"collections":null,"content":"7. Caching Strategies Performance Optimization Caching is one of the most effective ways to improve system performance by storing frequently accessed data in fast storage. Well-designed caching can reduce latency by 10-100x and significantly decrease database load. ","date":"28-Jul-2025","heading":"7. Caching Strategies","objectID":"\/posts\/system-design\/:7:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#7-caching-strategies"},{"categories":["software-engineering"],"collections":null,"content":"7.1 What to Cache? Caching Best Practices Good candidates for caching: Read-heavy data: Data that is read frequently but updated infrequently Expensive computations: Results that take significant CPU time to calculate Database query results: Frequently-run queries with stable results API responses: Third-party API calls with consistent data Rendered HTML pages: Static or semi-static page content Session data: User authentication and session information Static assets: Images, CSS, JavaScript files What NOT to Cache Avoid caching: Highly dynamic data that changes frequently User-specific sensitive data without proper isolation Data requiring strong consistency guarantees Data with low hit rates (rarely accessed) ","date":"28-Jul-2025","heading":"7.1 What to Cache?","objectID":"\/posts\/system-design\/:7:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#71-what-to-cache"},{"categories":["software-engineering"],"collections":null,"content":"7.2 Cache Levels Cache Levels in System \u2014 Comprehensive caching strategy: browser, CDN, application, and database caching 1. Client-Side Caching Browser cache LocalStorage \/ SessionStorage Service Workers 2. CDN Caching Caches static assets close to users Reduces origin server load 3. Application-Level Caching In-memory caches (Redis, Memcached) Application server cache 4. Database Caching Query result cache Buffer pool cache ","date":"28-Jul-2025","heading":"7.2 Cache Levels","objectID":"\/posts\/system-design\/:7:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#72-cache-levels"},{"categories":["software-engineering"],"collections":null,"content":"7.3 Caching Patterns Info Caching Pattern Selection: Choose your caching pattern based on read\/write ratios and consistency requirements: Cache-Aside: Best for read-heavy workloads with eventual consistency tolerance (90% of use cases) Write-Through: When strong consistency is required between cache and database Write-Behind: For write-heavy workloads where eventual consistency is acceptable Refresh-Ahead: For predictable access patterns with hot data 1. Cache-Aside (Lazy Loading) Most common pattern: def get_user(user_id): # Try cache first user = cache.get(f\"user:{user_id}\") if user is None: # Cache miss - load from database user = database.query(f\"SELECT * FROM users WHERE id = {user_id}\") # Store in cache for next time cache.set(f\"user:{user_id}\", user, ttl=3600) return user Flow: Application checks cache If data exists (cache hit), return it If data doesn’t exist (cache miss): Load from database Write to cache Return data Pros: Only requested data is cached Cache failure doesn’t break application Cons: Initial request is slow (cache miss) Stale data possible if cache doesn’t expire 2. Read-Through Cache Cache sits between application and database: class ReadThroughCache: def __init__(self, cache, database): self.cache = cache self.database = database def get(self, key): # Cache handles database lookup automatically return self.cache.get(key, loader=lambda: self.database.get(key)) Flow: Application requests data from cache Cache checks if data exists If not, cache loads from database automatically Cache returns data Pros: Cleaner application code Consistent caching logic Cons: Cache becomes critical dependency Potential bottleneck 3. Write-Through Cache Data written to cache and database simultaneously: def update_user(user_id, data): # Write to both cache and database cache.set(f\"user:{user_id}\", data) database.update(f\"UPDATE users SET ... WHERE id = {user_id}\") Pros: Cache always consistent with database No stale data Cons: Slower writes (two operations) Cache may contain rarely-read data 4. Write-Behind (Write-Back) Cache Data written to cache immediately, database updated asynchronously: def update_user(user_id, data): # Write to cache immediately cache.set(f\"user:{user_id}\", data) # Queue database write for later write_queue.push({ 'operation': 'update', 'table': 'users', 'id': user_id, 'data': data }) Pros: Very fast writes Can batch database writes Cons: Risk of data loss if cache fails More complex to implement Eventual consistency 5. Refresh-Ahead Proactively refresh cache before expiration: import time import threading def refresh_ahead_cache(key, ttl, refresh_threshold=0.8): data = cache.get(key) if data is None: # Cache miss - load and cache data = load_from_database(key) cache.set(key, data, ttl=ttl) else: # Check if approaching expiration remaining_ttl = cache.ttl(key) if remaining_ttl < (ttl * refresh_threshold): # Asynchronously refresh threading.Thread( target=lambda: cache.set(key, load_from_database(key), ttl=ttl) ).start() return data ","date":"28-Jul-2025","heading":"7.3 Caching Patterns","objectID":"\/posts\/system-design\/:7:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#73-caching-patterns"},{"categories":["software-engineering"],"collections":null,"content":"7.4 Cache Eviction Policies When cache is full, which data should be removed? 1. LRU (Least Recently Used) Evicts least recently accessed items Most commonly used Good for general-purpose caching 2. LFU (Least Frequently Used) Evicts items accessed least often Good for data with varying access patterns 3. FIFO (First In First Out) Evicts oldest items first Simple but not always optimal 4. TTL (Time To Live) Items expire after fixed time Good for time-sensitive data Implementation of LRU Cache: from collections import OrderedDict class LRUCache: def __init__(self, capacity): self.cache = OrderedDict() self.capacity = capacity def get(self, key): if key not in self.cache: return None # Move to end (most recently used) self.cache.move_to_end(key) return self.cache[key] def put(self, key, value): if key in self.cache: # Update and move to end self.cache.move_to_end(key) self.cache[key] = value if len(self.cache) > self.capacity: # Remove least recently used (first item) self.cache.popitem(last=False) # Usage cache = LRUCache(capacity=3) cache.put(\"user:1\", {\"name\": \"Alice\"}) cache.put(\"user:2\", {\"name\": \"Bob\"}) cache.put(\"user:3\", {\"name\": \"Charlie\"}) cache.put(\"user:4\", {\"name\": \"David\"}) # Evicts user:1 ","date":"28-Jul-2025","heading":"7.4 Cache Eviction Policies","objectID":"\/posts\/system-design\/:7:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#74-cache-eviction-policies"},{"categories":["software-engineering"],"collections":null,"content":"7.5 Cache Considerations 1. Cache Expiration (TTL) Warning TTL Strategy Pitfalls: Setting TTL too short causes cache misses and unnecessary database hits. Setting it too long risks serving stale data. Monitor cache hit ratios and adjust TTL dynamically based on actual data update frequency. For critical data, implement cache invalidation events rather than relying solely on TTL. Set appropriate TTL based on data characteristics: # Fast-changing data cache.set(\"stock_price:AAPL\", price, ttl=60) # 1 minute # Slow-changing data cache.set(\"user_profile:123\", profile, ttl=3600) # 1 hour # Rarely changing data cache.set(\"product_category\", categories, ttl=86400) # 24 hours 2. Cache Stampede Prevention Danger Cache Thundering Herd: When a popular cached item expires, hundreds or thousands of requests can simultaneously hit the database, causing a spike in load. This can bring down the database. Implement one of: (1) Locking (only first request loads), (2) Probabilistic early refresh (reload before expiration), or (3) Refresh-Ahead pattern. Problem: When cached item expires, multiple requests simultaneously hit database. Solution: Use locks or probabilistic early expiration: import threading import random lock_dict = {} def get_with_stampede_protection(key, ttl, load_function): data = cache.get(key) if data is None: # Get or create lock for this key if key not in lock_dict: lock_dict[key] = threading.Lock() lock = lock_dict[key] # Only one thread loads data with lock: # Double-check cache (another thread may have loaded it) data = cache.get(key) if data is None: data = load_function() cache.set(key, data, ttl=ttl) # Probabilistic early refresh remaining_ttl = cache.ttl(key) if remaining_ttl < ttl * 0.2: # Last 20% of TTL if random.random() < 0.1: # 10% chance threading.Thread( target=lambda: cache.set(key, load_function(), ttl=ttl) ).start() return data 3. Cache Warming Pre-populate cache with frequently accessed data: def warm_cache(): \"\"\"Run during application startup\"\"\" # Load popular users popular_users = database.query(\"SELECT * FROM users ORDER BY popularity DESC LIMIT 1000\") for user in popular_users: cache.set(f\"user:{user.id}\", user, ttl=3600) # Load popular products popular_products = database.query(\"SELECT * FROM products ORDER BY sales DESC LIMIT 5000\") for product in popular_products: cache.set(f\"product:{product.id}\", product, ttl=7200) 4. Distributed Caching For high availability and scalability: Redis Cluster Architecture \u2014 Distributed Redis deployment for scalability and fault tolerance in caching systems ","date":"28-Jul-2025","heading":"7.5 Cache Considerations","objectID":"\/posts\/system-design\/:7:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#75-cache-considerations"},{"categories":["software-engineering"],"collections":null,"content":"7.6 Redis vs Memcached Feature Redis Memcached Data Structures Strings, Lists, Sets, Sorted Sets, Hashes, Bitmaps Only strings Persistence Yes (RDB, AOF) No Replication Yes No (requires external tools) Transactions Yes No Pub\/Sub Yes No Lua Scripting Yes No Multi-threading Single-threaded (6.0+ has I\/O threads) Multi-threaded Memory Efficiency Good Slightly better Use Case Complex caching, session store, queues Simple key-value caching ","date":"28-Jul-2025","heading":"7.6 Redis vs Memcached","objectID":"\/posts\/system-design\/:7:6","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#76-redis-vs-memcached"},{"categories":["software-engineering"],"collections":null,"content":"8. Content Delivery Networks (CDN) CDNs are geographically distributed networks of servers that deliver content to users based on their location. ","date":"28-Jul-2025","heading":"8. Content Delivery Networks (CDN)","objectID":"\/posts\/system-design\/:8:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#8-content-delivery-networks-cdn"},{"categories":["software-engineering"],"collections":null,"content":"8.1 How CDN Works CDN Global Distribution \u2014 How content delivery networks reduce latency and improve performance worldwide CDN Request Flow: CDN Request Flow \u2014 Understanding cache hits, misses, and origin server interactions in CDN architecture ","date":"28-Jul-2025","heading":"8.1 How CDN Works","objectID":"\/posts\/system-design\/:8:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#81-how-cdn-works"},{"categories":["software-engineering"],"collections":null,"content":"8.2 Benefits of CDN Reduced Latency Content served from nearest location Fewer network hops Reduced Origin Load CDN handles most traffic Origin serves only cache misses Better Availability Multiple edge servers provide redundancy Can serve stale content if origin is down DDoS Protection Distributed architecture absorbs attacks Traffic filtered at edge Cost Savings Reduced bandwidth from origin Lower infrastructure costs ","date":"28-Jul-2025","heading":"8.2 Benefits of CDN","objectID":"\/posts\/system-design\/:8:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#82-benefits-of-cdn"},{"categories":["software-engineering"],"collections":null,"content":"8.3 What to Put on CDN? Static Assets (Best for CDN): Images CSS files JavaScript files Fonts Videos PDFs and downloadable files Dynamic Content (Possible with advanced CDNs): API responses with caching headers Personalized content (with edge computing) HTML pages ","date":"28-Jul-2025","heading":"8.3 What to Put on CDN?","objectID":"\/posts\/system-design\/:8:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#83-what-to-put-on-cdn"},{"categories":["software-engineering"],"collections":null,"content":"8.4 CDN Considerations 1. Cache Invalidation Challenge: How to update cached content? Solutions: a. Time-Based (TTL) Cache-Control: max-age=86400 # Cache for 24 hours b. Versioned URLs \/static\/app.js?v=1.2.3 \/static\/app.1.2.3.js \/static\/app.abc123hash.js c. Cache Purge API import requests def purge_cdn_cache(url): # Cloudflare example response = requests.post( 'https:\/\/api.cloudflare.com\/client\/v4\/zones\/{zone_id}\/purge_cache', headers={'Authorization': 'Bearer {api_token}'}, json={'files': [url]} ) return response.json() 2. Cache-Control Headers # Don't cache Cache-Control: no-store # Cache but revalidate Cache-Control: no-cache # Cache for 1 hour Cache-Control: max-age=3600 # Cache for 1 hour, revalidate after expiry Cache-Control: max-age=3600, must-revalidate # Private (browser only, not CDN) Cache-Control: private, max-age=3600 # Public (can be cached by CDN) Cache-Control: public, max-age=86400 # Immutable (never changes) Cache-Control: public, max-age=31536000, immutable 3. CDN Push vs Pull Pull CDN (Most Common): CDN fetches content from origin on first request Origin remains source of truth Lazy loading Push CDN: You upload content directly to CDN No origin server needed Manual or automated uploads ","date":"28-Jul-2025","heading":"8.4 CDN Considerations","objectID":"\/posts\/system-design\/:8:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#84-cdn-considerations"},{"categories":["software-engineering"],"collections":null,"content":"9. Database Systems Database Selection Choosing the right database is one of the most important system design decisions. The choice between SQL and NoSQL, along with specific database technologies, can significantly impact your system’s performance, scalability, and maintainability. ","date":"28-Jul-2025","heading":"9. Database Systems","objectID":"\/posts\/system-design\/:9:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#9-database-systems"},{"categories":["software-engineering"],"collections":null,"content":"9.1 SQL vs NoSQL Decision Framework The choice between SQL and NoSQL is not binary. Many modern systems use polyglot persistence, combining different database types for different use cases within the same application. Database Types Classification \u2014 SQL vs NoSQL: relational, document, key-value, graph, and column-family databases ","date":"28-Jul-2025","heading":"9.1 SQL vs NoSQL","objectID":"\/posts\/system-design\/:9:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#91-sql-vs-nosql"},{"categories":["software-engineering"],"collections":null,"content":"9.2 When to Use SQL? Info ACID Guarantees: SQL databases provide ACID properties (Atomicity, Consistency, Isolation, Durability) which are critical for financial transactions, inventory management, and any scenario where data integrity cannot be compromised. If a transaction fails midway, all changes are rolled back - ensuring your data never ends up in an inconsistent state. Use SQL (Relational Databases) When: Data is structured and relationships are important E-commerce (orders, customers, products) Financial systems (accounts, transactions) CRM systems ACID compliance is required Banking and financial applications Inventory management Any system where data consistency is critical Complex queries needed Reporting and analytics Joins across multiple tables Aggregations and complex filtering Data integrity constraints Foreign keys Unique constraints Check constraints Example Schema (E-commerce): CREATE TABLE users ( id BIGINT PRIMARY KEY AUTO_INCREMENT, email VARCHAR(255) UNIQUE NOT NULL, name VARCHAR(255) NOT NULL, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); CREATE TABLE products ( id BIGINT PRIMARY KEY AUTO_INCREMENT, name VARCHAR(255) NOT NULL, price DECIMAL(10, 2) NOT NULL, stock INT NOT NULL, category_id BIGINT, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, FOREIGN KEY (category_id) REFERENCES categories(id) ); CREATE TABLE orders ( id BIGINT PRIMARY KEY AUTO_INCREMENT, user_id BIGINT NOT NULL, total_amount DECIMAL(10, 2) NOT NULL, status ENUM('pending', 'paid', 'shipped', 'delivered') DEFAULT 'pending', created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, FOREIGN KEY (user_id) REFERENCES users(id) ); CREATE TABLE order_items ( id BIGINT PRIMARY KEY AUTO_INCREMENT, order_id BIGINT NOT NULL, product_id BIGINT NOT NULL, quantity INT NOT NULL, price DECIMAL(10, 2) NOT NULL, FOREIGN KEY (order_id) REFERENCES orders(id), FOREIGN KEY (product_id) REFERENCES products(id) ); ","date":"28-Jul-2025","heading":"9.2 When to Use SQL?","objectID":"\/posts\/system-design\/:9:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#92-when-to-use-sql"},{"categories":["software-engineering"],"collections":null,"content":"9.3 When to Use NoSQL? Use NoSQL When: Massive scale required Billions of rows Petabytes of data Millions of operations per second Schema flexibility needed Rapidly evolving data models Each record has different fields Semi-structured or unstructured data High availability over consistency Social media feeds Real-time analytics IoT data Denormalized data is acceptable Data duplication is okay Joins are rare NoSQL Types: 1. Key-Value Stores (Redis, DynamoDB) Simplest NoSQL model: # Redis example cache.set(\"user:123\", json.dumps({ \"name\": \"John Doe\", \"email\": \"john@example.com\" })) user = json.loads(cache.get(\"user:123\")) Use cases: Session storage Caching Real-time analytics Shopping carts 2. Document Databases (MongoDB, CouchDB) Store JSON-like documents: \/\/ MongoDB example { \"_id\": ObjectId(\"507f1f77bcf86cd799439011\"), \"name\": \"John Doe\", \"email\": \"john@example.com\", \"addresses\": [ { \"type\": \"home\", \"street\": \"123 Main St\", \"city\": \"San Francisco\", \"zip\": \"94102\" }, { \"type\": \"work\", \"street\": \"456 Market St\", \"city\": \"San Francisco\", \"zip\": \"94103\" } ], \"orders\": [ {\"order_id\": \"ORD001\", \"total\": 99.99}, {\"order_id\": \"ORD002\", \"total\": 149.99} ] } Use cases: Content management User profiles Product catalogs Real-time analytics 3. Column-Family Stores (Cassandra, HBase) Optimized for write-heavy workloads: Row Key: user_123 Column Family: profile name: \"John Doe\" email: \"john@example.com\" Column Family: activity 2024-01-15:10:30:00: \"login\" 2024-01-15:10:31:22: \"viewed_product_456\" 2024-01-15:10:35:45: \"added_to_cart_456\" Use cases: Time-series data Event logging IoT sensor data Message systems 4. Graph Databases (Neo4j, Amazon Neptune) Optimized for relationships: \/\/ Neo4j example (John:Person {name: \"John Doe\"}) -[:FRIENDS_WITH]->(Jane:Person {name: \"Jane Smith\"}) -[:WORKS_AT]->(Company:Company {name: \"Acme Corp\"}) <-[:WORKS_AT]-(Bob:Person {name: \"Bob Johnson\"}) -[:LIKES]->(Product:Product {name: \"Widget\"}) Use cases: Social networks Recommendation engines Fraud detection Knowledge graphs ","date":"28-Jul-2025","heading":"9.3 When to Use NoSQL?","objectID":"\/posts\/system-design\/:9:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#93-when-to-use-nosql"},{"categories":["software-engineering"],"collections":null,"content":"9.4 Database Performance Optimization Tip Performance Optimization Hierarchy: Follow this order for maximum impact: Add indexes on frequently queried columns (10-100x speedup) Optimize slow queries using EXPLAIN (5-50x speedup) Add caching layer for hot data (10-100x speedup) Scale reads with replicas when CPU-bound (linear scaling) Shard database only when vertical scaling exhausted (adds complexity) 1. Indexing Warning Index Trade-offs: Every index speeds up reads but slows down writes. An insert\/update\/delete must update all indexes on that table. A table with 10 indexes will have 10x slower writes. Only index columns that are frequently used in WHERE, JOIN, or ORDER BY clauses. Monitor index usage with EXPLAIN and remove unused indexes. Indexes speed up queries but slow down writes: -- Create index CREATE INDEX idx_users_email ON users(email); CREATE INDEX idx_products_category ON products(category_id); -- Composite index CREATE INDEX idx_orders_user_status ON orders(user_id, status); -- Full-text index CREATE FULLTEXT INDEX idx_products_search ON products(name, description); Types of Indexes: B-Tree Index (default): Good for equality and range queries Hash Index: Only for equality comparisons Full-Text Index: For text search Spatial Index: For geographic data Index Best Practices: Index columns used in WHERE, JOIN, ORDER BY Don’t over-index (slows writes) Use composite indexes for multi-column queries Monitor index usage and remove unused indexes 2. Query Optimization -- Bad: SELECT * SELECT * FROM users WHERE email = 'john@example.com'; -- Good: Select only needed columns SELECT id, name, email FROM users WHERE email = 'john@example.com'; -- Use EXPLAIN to analyze queries EXPLAIN SELECT * FROM orders WHERE user_id = 123 AND status = 'pending'; -- Avoid N+1 queries -- Bad: Query in loop for user_id in user_ids: orders = db.query(f\"SELECT * FROM orders WHERE user_id = {user_id}\") -- Good: Single query orders = db.query(f\"SELECT * FROM orders WHERE user_id IN ({','.join(user_ids)})\") 3. Connection Pooling import psycopg2 from psycopg2 import pool # Create connection pool connection_pool = pool.SimpleConnectionPool( minconn=5, maxconn=20, host=\"localhost\", database=\"mydb\", user=\"user\", password=\"password\" ) def execute_query(query): conn = connection_pool.getconn() try: cursor = conn.cursor() cursor.execute(query) result = cursor.fetchall() return result finally: connection_pool.putconn(conn) ","date":"28-Jul-2025","heading":"9.4 Database Performance Optimization","objectID":"\/posts\/system-design\/:9:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#94-database-performance-optimization"},{"categories":["software-engineering"],"collections":null,"content":"10. Database Sharding and Partitioning Info Sharding: Last Resort Scaling: Sharding adds significant complexity - cross-shard joins become application-level operations, distributed transactions are complex, and rebalancing data is challenging. Exhaust these options first: (1) Optimize queries and add indexes, (2) Add read replicas, (3) Vertical scaling (bigger machine), (4) Caching layer. Only shard when you absolutely must handle massive scale (billions of rows, millions of QPS). When a single database can’t handle the load, we need to split data across multiple databases. ","date":"28-Jul-2025","heading":"10. Database Sharding and Partitioning","objectID":"\/posts\/system-design\/:10:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#10-database-sharding-and-partitioning"},{"categories":["software-engineering"],"collections":null,"content":"10.1 Vertical vs Horizontal Partitioning Vertical Partitioning (Splitting by columns) Original Table: users: id, name, email, address, bio, profile_pic Split into: users_basic: id, name, email users_extended: id, address, bio, profile_pic Use when: Different columns accessed with different frequencies Some columns are rarely used Reduce I\/O for common queries Horizontal Partitioning\/Sharding (Splitting by rows) users table split by region: users_us: All US users users_eu: All EU users users_asia: All ASIA users ","date":"28-Jul-2025","heading":"10.1 Vertical vs Horizontal Partitioning","objectID":"\/posts\/system-design\/:10:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#101-vertical-vs-horizontal-partitioning"},{"categories":["software-engineering"],"collections":null,"content":"10.2 Sharding Strategies Tip Choosing a Sharding Strategy: Range-Based: Use for time-series data or when range queries are common (e.g., “get all orders from last month”) Hash-Based: Best for even distribution and single-key lookups (e.g., user profiles, sessions) Geographic: Essential for data residency compliance (GDPR, data sovereignty laws) Directory-Based: Most flexible but adds latency and single point of failure (shard directory) 1. Range-Based Sharding Split data based on a continuous range of the shard key \u2014 typically ID ranges, alphabetical ranges, or time ranges. def get_shard_by_range(user_id): if user_id < 1000000: return \"shard_1\" elif user_id < 2000000: return \"shard_2\" elif user_id < 3000000: return \"shard_3\" else: return \"shard_4\" Pros: Range queries are fast \u2014 “get all orders from last month” maps to a single shard (or a small set of consecutive shards) with no cross-shard fan-out. Easy to understand and operate. Cons: If your data grows non-uniformly, some shards get much more data than others \u2014 called a hot shard problem. For example, if most users were created recently, shard_4 (user_id > 3M) could be 10\u00d7 larger than shard_1. Adding new shards means splitting existing ranges, which requires migrating data. Best for: Time-series data (IoT, logs, financial ticks) where you always query recent ranges, and old shards naturally become read-only. 2. Hash-Based Sharding Apply a hash function to the shard key, then take the modulo to distribute rows evenly. import hashlib def get_shard_by_hash(user_id, num_shards): hash_value = int(hashlib.md5(str(user_id).encode()).hexdigest(), 16) shard_id = hash_value % num_shards return f\"shard_{shard_id}\" Pros: Hash functions distribute data very evenly \u2014 with a good hash function and many keys, each shard gets roughly equal load. No hot shard problem. Single-key lookups (get user by ID) are O(1) \u2014 just hash the key and query one shard. Cons: The biggest operational headache: when you add a shard, num_shards changes, so hash(key) % num_shards changes for most keys \u2014 meaning most data needs to be migrated to a different shard. This is why consistent hashing (\u00a718) was invented \u2014 it minimizes remapping when nodes are added\/removed, only ~1\/N keys need to move. Range queries are also impossible \u2014 “get all users created in January” would require querying all shards and merging. Best for: Any workload where you primarily access data by a single known key (user profiles, session data, product catalog). The go-to choice when you just need even distribution. 3. Geographic Sharding Route data to the shard closest to where it originates or is most often accessed. def get_shard_by_geo(user_location): if user_location in ['US', 'CA', 'MX']: return \"shard_americas\" elif user_location in ['GB', 'FR', 'DE']: return \"shard_europe\" elif user_location in ['CN', 'JP', 'IN']: return \"shard_asia\" else: return \"shard_global\" Pros: Data latency is minimized \u2014 a European user’s data lives in the European shard, physically co-located with the servers serving that user. Critical for regulatory compliance: GDPR requires EU user data to remain within the EU; geographic sharding is the standard way to achieve data residency requirements. Cons: User population is not evenly spread globally \u2014 the Americas shard will be far larger than, say, an Africa shard. If a user travels internationally, their requests may cross regions, adding latency. Cross-shard queries (e.g., global analytics) are expensive. Best for: Applications with legal data residency requirements, or where you need to minimize latency for geographically dispersed users (social networks, cloud storage, global SaaS). 4. Directory-Based Sharding Maintain a lookup table: # Shard directory shard_directory = { \"user:1\": \"shard_1\", \"user:2\": \"shard_1\", \"user:3\": \"shard_2\", \"user:4\": \"shard_3\", } def get_shard_by_directory(user_id): return shard_directory.get(f\"user:{user_id}\", \"shard_default\") Pros: You ha","date":"28-Jul-2025","heading":"10.2 Sharding Strategies","objectID":"\/posts\/system-design\/:10:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#102-sharding-strategies"},{"categories":["software-engineering"],"collections":null,"content":"10.3 Challenges with Sharding Danger Sharding Operational Complexity: Sharding introduces these hard problems: Cross-shard joins: Must fetch from multiple shards and join in application (slow) Distributed transactions: 2PC is slow and error-prone (avoid if possible) Data rebalancing: Adding shards requires massive data migration (hours\/days downtime) Hotspots: Uneven sharding key distribution causes some shards to be overloaded Query routing: Routing layer becomes complex and critical 1. Cross-Shard Joins Note Denormalization is Necessary: With sharding, you cannot do efficient cross-shard joins. Denormalize data by storing redundant copies across shards. Accept the write overhead and potential staleness. Example: Store user_name and user_email in orders table even though they’re in users table. Problem: Can’t join tables across different databases. Solution: Denormalize data or handle joins at application level: def get_user_with_orders(user_id): # Get shard for user user_shard = get_shard(user_id) user = user_shard.query(f\"SELECT * FROM users WHERE id = {user_id}\") # Orders might be on different shard order_shard = get_shard_for_orders(user_id) orders = order_shard.query(f\"SELECT * FROM orders WHERE user_id = {user_id}\") # Combine in application user['orders'] = orders return user 2. Distributed Transactions Warning 2PC Performance Penalty: Two-Phase Commit (2PC) requires locking all shards and coordinating across them. In MySQL, 2PC adds 20-50% latency overhead and can deadlock if timing is wrong. Better approach: Use Saga pattern - break transaction into compensating transactions, or ensure shard key is included in all transactions so data stays on one shard. Problem: Transaction spanning multiple shards. Solution: Two-Phase Commit (2PC) or avoid distributed transactions: def transfer_money(from_user_id, to_user_id, amount): from_shard = get_shard(from_user_id) to_shard = get_shard(to_user_id) if from_shard == to_shard: # Same shard - normal transaction with from_shard.transaction(): from_shard.execute(f\"UPDATE accounts SET balance = balance - {amount} WHERE user_id = {from_user_id}\") to_shard.execute(f\"UPDATE accounts SET balance = balance + {amount} WHERE user_id = {to_user_id}\") else: # Different shards - use 2PC or saga pattern # (More complex - see distributed transactions section) pass 3. Rebalancing Problem: Adding\/removing shards requires data migration. Solution: Consistent hashing or careful planning: def rebalance_shards(old_shards, new_shards): # Calculate which data needs to move for key in get_all_keys(): old_shard = hash(key) % old_shards new_shard = hash(key) % new_shards if old_shard != new_shard: # Migrate data data = get_data_from_shard(old_shard, key) write_data_to_shard(new_shard, key, data) delete_data_from_shard(old_shard, key) ","date":"28-Jul-2025","heading":"10.3 Challenges with Sharding","objectID":"\/posts\/system-design\/:10:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#103-challenges-with-sharding"},{"categories":["software-engineering"],"collections":null,"content":"10.4 Sharding Architecture Database Sharding \u2014 Horizontal partitioning strategies for scaling large databases; range, hash, geo, and directory-based sharding ","date":"28-Jul-2025","heading":"10.4 Sharding Architecture","objectID":"\/posts\/system-design\/:10:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#104-sharding-architecture"},{"categories":["software-engineering"],"collections":null,"content":"11. Database Replication Tip Replication Benefits: Database replication is the easiest way to scale read traffic. With 3 read replicas, you can handle 3x the read load. Replication also provides data redundancy (if master fails, promote a replica) and enables running analytics\/backups on replicas without impacting production. Database replication is the process of copying data from one database to another to ensure redundancy, improve availability, and enhance read performance. ","date":"28-Jul-2025","heading":"11. Database Replication","objectID":"\/posts\/system-design\/:11:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#11-database-replication"},{"categories":["software-engineering"],"collections":null,"content":"11.1 Master-Slave Replication The most common replication pattern: Database Replication \u2014 Master-slave and multi-master replication strategies for availability and performance How it works: All write operations go to the master database Master propagates changes to slave databases Read operations are distributed across slave databases Improves read performance through load distribution Advantages: Better read performance (distribute reads across multiple servers) Data backup (slaves serve as backups) Analytics without impacting production (run reports on slaves) High availability (promote slave if master fails) Disadvantages: Replication lag (slaves might be slightly behind master) Single point of failure for writes (only one master) Complexity in failover scenarios Warning Replication Lag Problem: Replication is usually asynchronous - slaves can be seconds or even minutes behind the master. If a user writes data and immediately reads from a replica, they might not see their own write! Solutions: (1) Read-after-write consistency (read from master for short time after write), (2) Sticky sessions (always read from same replica), or (3) Use timestamps\/version numbers to detect stale reads. ","date":"28-Jul-2025","heading":"11.1 Master-Slave Replication","objectID":"\/posts\/system-design\/:11:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#111-master-slave-replication"},{"categories":["software-engineering"],"collections":null,"content":"11.2 Master-Master Replication Both databases accept writes and sync with each other: High Availability Design \u2014 Redundancy, failover strategies, and fault tolerance for mission-critical systems Advantages: No single point of failure for writes Better write performance (distribute writes) Lower latency for geographically distributed systems Disadvantages: Complex conflict resolution Potential for inconsistency More difficult to manage Danger Multi-Master Conflicts: With master-master replication, two masters can accept conflicting writes. Example: User A updates email to “a@example.com” on Master 1, User B updates same user’s email to “b@example.com” on Master 2. Which wins? Last-Write-Wins (LWW) is simple but can lose data. Version vectors detect conflicts but require application-level resolution. Avoid multi-master if possible. Conflict Resolution Strategies: # Last Write Wins (LWW) def resolve_conflict_lww(value1, value2): \"\"\" Choose the value with the latest timestamp \"\"\" if value1.timestamp > value2.timestamp: return value1 return value2 # Version Vector class VersionVector: def __init__(self): self.vector = {} def increment(self, node_id): self.vector[node_id] = self.vector.get(node_id, 0) + 1 def merge(self, other): \"\"\"Merge two version vectors\"\"\" for node_id, version in other.vector.items(): self.vector[node_id] = max( self.vector.get(node_id, 0), version ) def compare(self, other): \"\"\" Returns: - 'before' if self < other - 'after' if self > other - 'concurrent' if conflicting \"\"\" self_greater = False other_greater = False all_nodes = set(self.vector.keys()) | set(other.vector.keys()) for node in all_nodes: self_ver = self.vector.get(node, 0) other_ver = other.vector.get(node, 0) if self_ver > other_ver: self_greater = True elif other_ver > self_ver: other_greater = True if self_greater and not other_greater: return 'after' elif other_greater and not self_greater: return 'before' else: return 'concurrent' ","date":"28-Jul-2025","heading":"11.2 Master-Master Replication","objectID":"\/posts\/system-design\/:11:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#112-master-master-replication"},{"categories":["software-engineering"],"collections":null,"content":"11.3 Replication Methods Synchronous Replication: Master waits for slave acknowledgment before confirming write Strong consistency Higher latency Used when data consistency is critical def synchronous_write(data): # Write to master master.write(data) # Wait for all slaves to acknowledge for slave in slaves: slave.write(data) slave.acknowledge() return \"Write successful\" Asynchronous Replication: Master confirms write immediately Slaves updated eventually Lower latency Risk of data loss if master fails def asynchronous_write(data): # Write to master master.write(data) # Queue replication (don't wait) for slave in slaves: replication_queue.enqueue({ 'slave': slave, 'data': data }) return \"Write successful\" Semi-Synchronous Replication: The master confirms a write only after at least one slave has acknowledged receiving the data \u2014 but doesn’t wait for all slaves. This is the default in MySQL 5.5+ and is a pragmatic middle ground used by most production systems. Why it’s better than purely asynchronous: If the master crashes immediately after confirming a write, at least one replica already has that data. You haven’t lost the write \u2014 you just need to promote the right replica. Pure async replication has a window where a confirmed write exists only in the (now-crashed) master’s memory. Why it’s better than fully synchronous: You only wait for one acknowledgment, not all. If you have 3 replicas and one is slow or in a different AZ, you don’t block on it. Typical added latency is just one extra network round-trip to the nearest replica (~0.5\u20131ms within the same datacenter). Practical usage: MySQL’s rpl_semi_sync_master_enabled implements this. Google Cloud SQL and AWS RDS use a form of semi-synchronous replication for their managed database services. For most applications, semi-synchronous replication gives you the durability guarantee you actually need without the full latency cost of synchronous replication. ","date":"28-Jul-2025","heading":"11.3 Replication Methods","objectID":"\/posts\/system-design\/:11:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#113-replication-methods"},{"categories":["software-engineering"],"collections":null,"content":"12. Distributed Systems Concepts Distributed Systems Foundation Understanding distributed systems is crucial for designing scalable applications. Modern applications rarely run on a single machine - they are distributed across multiple servers, data centers, and geographical regions. Distributed Systems Overview \u2014 Nodes, consensus algorithms (Raft\/Paxos), leader election, and replicated data stores in a fault-tolerant cluster ","date":"28-Jul-2025","heading":"12. Distributed Systems Concepts","objectID":"\/posts\/system-design\/:12:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#12-distributed-systems-concepts"},{"categories":["software-engineering"],"collections":null,"content":"12.1 What is a Distributed System? Definition A distributed system is a collection of independent computers that appears to users as a single coherent system. Characteristics: Multiple autonomous computers: Independent nodes with their own CPU, memory, and storage Connected through a network: Communication via TCP\/IP, message queues, or RPC Coordinate actions through message passing: No shared memory between nodes No shared memory: Each node has isolated memory space Failures are partial: Some components can fail while others continue working Why Distributed Systems? Benefits: Scalability: Handle more load by adding machines (horizontal scaling) Reliability: System continues working even if components fail Performance: Process data closer to users (reduced latency) Availability: No single point of failure with proper design Challenges: Network is unreliable: Messages can be lost or delayed Partial failures: Some components fail while others work Synchronization: Coordinating actions is difficult without shared memory Consistency: Keeping data consistent across nodes is complex ","date":"28-Jul-2025","heading":"12.1 What is a Distributed System?","objectID":"\/posts\/system-design\/:12:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#121-what-is-a-distributed-system"},{"categories":["software-engineering"],"collections":null,"content":"12.2 Fallacies of Distributed Computing First identified by Peter Deutsch and colleagues at Sun Microsystems in the 1990s, these eight assumptions are mistakes that developers who come from single-machine programming routinely make. Violating them causes outages, data corruption, and subtle bugs that only appear in production. The Eight Fallacies \u2014 and What to Do Instead 1. The network is reliable. Reality: Networks fail constantly. Packets are lost, connections time out, switches reboot, NICs malfunction. In a large distributed system, network events happen every day. What to do: Implement retries with exponential backoff and jitter. Use circuit breakers to stop hammering failing services. Design your protocol to be idempotent so retries don’t cause double-processing. 2. Latency is zero. Reality: Even within the same datacenter, a round-trip network call takes ~0.5ms. Cross-region calls take 100\u2013200ms. This sounds small, but if you make 100 sequential remote calls to render one page, that’s 50ms just in network overhead. What to do: Batch operations to reduce round trips. Cache frequently-needed remote data locally. Use async patterns so network calls don’t block the critical path. 3. Bandwidth is infinite. Reality: Network interfaces have finite capacity (typically 1\u201325 Gbps in cloud environments), and bandwidth costs money. Sending large payloads frequently saturates links and increases costs. What to do: Compress data in transit (gzip, snappy, protobuf instead of JSON). Paginate large responses. Use CDNs to offload static data delivery from your origin servers. 4. The network is secure. Reality: Traffic on internal networks can be intercepted by a compromised host within the same datacenter. Supply-chain attacks and insider threats are real. What to do: Encrypt all traffic, even on internal networks (mTLS via service mesh). Authenticate service-to-service calls, not just client-to-server. Adopt zero-trust networking \u2014 assume the internal network is hostile. 5. Topology doesn’t change. Reality: Servers are added and removed constantly. DNS entries change. Load balancers update their backends. Cloud auto-scaling adds\/removes instances in seconds. A hardcoded IP address will break. What to do: Use service discovery (Consul, Kubernetes DNS). Never hardcode IPs. Use connection pooling with health checks that detect topology changes. 6. There is one administrator. Reality: In a microservices system, each service is owned by a different team with different deployment schedules and on-call rotations. A change by one team can break another team’s service. What to do: Publish stable API contracts (OpenAPI specs). Use consumer-driven contract testing (Pact) so API changes that break consumers are caught before deployment. Communicate across teams before making breaking changes. 7. Transport cost is zero. Reality: Serializing data (JSON encoding, Protobuf marshalling), compressing it, and routing it through network stacks consumes CPU cycles on both ends. High-frequency, small-payload RPC calls can consume significant CPU even if individual calls are fast. What to do: Choose efficient serialization formats (Protocol Buffers instead of XML). Batch small messages into larger ones. Profile your serialization overhead \u2014 it’s often surprising how much CPU JSON encoding consumes at scale. 8. The network is homogeneous. Reality: Your system likely spans multiple cloud providers, on-premises datacenters, different operating systems, different programming languages, and different network hardware with varying capabilities. A packet from your Go service must be correctly interpreted by your Java service running on a different OS. What to do: Use language-neutral serialization (Protobuf, Avro, JSON). Version your APIs explicitly. Test across the actual environments you deploy to, not just your development machine. ","date":"28-Jul-2025","heading":"12.2 Fallacies of Distributed Computing","objectID":"\/posts\/system-design\/:12:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#122-fallacies-of-distributed-computing"},{"categories":["software-engineering"],"collections":null,"content":"13. CAP Theorem and Consistency Models Fundamental Trade-off The CAP theorem, proposed by Eric Brewer, is one of the most important concepts in distributed systems. It defines the trade-offs between consistency, availability, and partition tolerance. ","date":"28-Jul-2025","heading":"13. CAP Theorem and Consistency Models","objectID":"\/posts\/system-design\/:13:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#13-cap-theorem-and-consistency-models"},{"categories":["software-engineering"],"collections":null,"content":"13.1 CAP Theorem CAP Theorem Statement The CAP theorem states that a distributed system can provide at most TWO of the following three guarantees simultaneously: C (Consistency): All nodes see the same data at the same time A (Availability): Every request receives a response (success or failure) P (Partition Tolerance): System continues operating despite network partitions CAP Theorem \u2014 Consistency vs Availability vs Partition Tolerance; critical concept for distributed system design decisions In Reality: Network partitions will happen (P is mandatory) Must choose between C and A during partition When partition heals, system needs reconciliation strategy Info Partition Tolerance is Mandatory: In the real world, network partitions WILL happen - switches fail, network cables get cut, data centers lose connectivity. You cannot choose “CA” because you cannot avoid network partitions. The real choice is: During a partition, do you want Consistency (CP) or Availability (AP)? CP Systems (Consistency + Partition Tolerance): Wait for partition to heal Return errors during partition Example: Banking systems, MongoDB (with majority write concern) Note CP System Trade-off: CP systems (like banking systems) choose to be unavailable during network partitions rather than show inconsistent data. If you can’t reach a quorum of nodes, writes fail. This prevents showing incorrect account balances but means some operations fail during network issues. AP Systems (Availability + Partition Tolerance): Always accept reads\/writes Resolve conflicts later Example: Social media feeds, DNS, Cassandra Note AP System Trade-off: AP systems (like social media feeds) stay available during network partitions but accept eventual consistency. You might temporarily see old data or conflicting updates, but the system never returns errors. Facebook chooses AP - it’s better to show a slightly stale news feed than to be completely unavailable. ","date":"28-Jul-2025","heading":"13.1 CAP Theorem","objectID":"\/posts\/system-design\/:13:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#131-cap-theorem"},{"categories":["software-engineering"],"collections":null,"content":"13.2 Consistency Models Tip Consistency Model Selection: Choose based on your use case: Strong Consistency: Financial transactions, inventory systems (can’t sell same item twice) Eventual Consistency: Social media feeds, analytics, caching (temporary staleness is acceptable) Read-Your-Writes: User profiles, settings (users must see their own changes immediately) Causal Consistency: Comment threads, chat (replies must appear after original message) Strong Consistency: After a write, all reads see that value Highest consistency guarantee Higher latency def strong_consistency_read(): # Read from master or wait for replication return master.read() Eventual Consistency: If no new updates, eventually all reads return same value Lower latency Used in AP systems def eventual_consistency_read(): # Read from any replica # Might return stale data return random.choice(replicas).read() Read-Your-Writes Consistency: Users see their own writes immediately Others might see stale data session_writes = {} def write(user_id, data): master.write(data) session_writes[user_id] = time.now() def read(user_id): if user_id in session_writes: # Read from master for this user return master.read() else: # Can read from replica for others return replica.read() Monotonic Reads: If user reads value A, subsequent reads won’t return older values user_last_read_time = {} def monotonic_read(user_id): last_time = user_last_read_time.get(user_id, 0) # Only read from replicas caught up to last_time valid_replicas = [ r for r in replicas if r.last_sync_time >= last_time ] result = random.choice(valid_replicas).read() user_last_read_time[user_id] = time.now() return result Causal Consistency: Related operations are seen in correct order Independent operations can be seen in any order ","date":"28-Jul-2025","heading":"13.2 Consistency Models","objectID":"\/posts\/system-design\/:13:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#132-consistency-models"},{"categories":["software-engineering"],"collections":null,"content":"13.3 BASE Properties Alternative to ACID for distributed systems: Basically Available: System appears to work most of the time Soft state: State may change without input (due to eventual consistency) Eventual consistency: System becomes consistent over time ","date":"28-Jul-2025","heading":"13.3 BASE Properties","objectID":"\/posts\/system-design\/:13:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#133-base-properties"},{"categories":["software-engineering"],"collections":null,"content":"14. Message Queues and Event-Driven Architecture Asynchronous Communication Message queues are essential for building scalable, decoupled distributed systems. They enable asynchronous communication between services, improving system reliability and flexibility. ","date":"28-Jul-2025","heading":"14. Message Queues and Event-Driven Architecture","objectID":"\/posts\/system-design\/:14:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#14-message-queues-and-event-driven-architecture"},{"categories":["software-engineering"],"collections":null,"content":"14.1 Why Message Queues? Key Benefits Message queues enable asynchronous communication between services with these benefits: 1. Decoupling: Services don’t need to know about each other directly Producers and consumers are independent Can evolve separately without coordination 2. Reliability: Messages persisted until successfully processed Messages won’t be lost if consumer is down Retry mechanisms for failed processing 3. Scalability: Add more consumers to handle increased load Horizontal scaling of message processing Load distribution across multiple consumers 4. Buffering: Handle traffic spikes gracefully Queue absorbs burst traffic Consumers process at sustainable rate Message Queue System \u2014 Async communication pattern using Kafka, RabbitMQ for system scalability and resilience ","date":"28-Jul-2025","heading":"14.1 Why Message Queues?","objectID":"\/posts\/system-design\/:14:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#141-why-message-queues"},{"categories":["software-engineering"],"collections":null,"content":"14.2 Message Queue Patterns 1. Point-to-Point (Queue) Each message consumed by exactly one consumer: class MessageQueue: def __init__(self): self.queue = [] self.lock = threading.Lock() def send(self, message): with self.lock: self.queue.append(message) def receive(self): with self.lock: if self.queue: return self.queue.pop(0) return None # Usage queue = MessageQueue() # Producer queue.send({\"order_id\": 123, \"action\": \"process\"}) # Consumer message = queue.receive() if message: process_order(message) 2. Publish-Subscribe (Topic) Each message consumed by all subscribers: class PubSubTopic: def __init__(self): self.subscribers = [] self.messages = [] def subscribe(self, subscriber): self.subscribers.append(subscriber) def publish(self, message): for subscriber in self.subscribers: subscriber.receive(message) # Usage topic = PubSubTopic() # Subscribers email_service = EmailService() sms_service = SMSService() push_service = PushService() topic.subscribe(email_service) topic.subscribe(sms_service) topic.subscribe(push_service) # Publisher topic.publish({\"event\": \"order_placed\", \"user_id\": 123}) # All three services receive the message ","date":"28-Jul-2025","heading":"14.2 Message Queue Patterns","objectID":"\/posts\/system-design\/:14:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#142-message-queue-patterns"},{"categories":["software-engineering"],"collections":null,"content":"14.3 Popular Message Queue Systems Choosing the right message queue significantly impacts your system’s behaviour. Here is what each tool is actually good at and when to reach for it: RabbitMQ is a traditional message broker built around the AMQP protocol. It excels when you need flexible routing logic \u2014 messages can be sent to exchanges that route them to queues based on routing keys, patterns, or headers. It supports per-message acknowledgements, dead-letter queues (where failed messages land for inspection), message priorities, and request\/reply patterns. RabbitMQ stores messages only as long as consumers need them; once consumed, they are deleted. Use it when you need complex routing, task queues where each job should be processed exactly once, or request-reply workflows. Apache Kafka is fundamentally different from RabbitMQ \u2014 it is a distributed, persistent, append-only log, not a traditional queue. Messages are written to topics divided into partitions, and they stay there for a configurable retention period (days or weeks), not just until consumed. Multiple consumer groups can independently read the same messages at different offsets, enabling multiple services to process the same event stream. Kafka handles millions of events per second per broker with sequential disk I\/O (see \u00a744). Use it when you need event streaming, real-time data pipelines, audit logs, or when multiple independent consumers need to process the same events (e.g., analytics pipeline + notification service both consuming order events). Amazon SQS is a fully managed, serverless queue \u2014 there are no servers to provision, patch, or scale. It handles queues of virtually any size and integrates natively with Lambda, EC2, and other AWS services. It offers two flavors: Standard (at-least-once delivery, best-effort ordering) and FIFO (exactly-once processing, strict ordering). Use it when you’re already on AWS, want zero operational overhead, and need a simple work queue for decoupling services. Redis Pub\/Sub broadcasts messages to all current subscribers in real time \u2014 with sub-millisecond latency because everything is in memory. However, if no subscriber is listening when a message is published, the message is permanently lost. Redis does not persist pub\/sub messages. Use it when you need real-time notifications or live updates where some message loss is acceptable (e.g., live chat “user is typing” indicator, live score updates, real-time dashboard refreshes). RabbitMQ Kafka Amazon SQS Redis Pub\/Sub Throughput High Very high High Very high Message retention Until consumed Days\/weeks 14 days max None Message ordering Per-queue Per-partition FIFO mode only Broadcast order Multiple consumers Competing consumers Independent offsets Competing consumers All subscribers Persistence Yes (disk-backed) Yes (log) Yes (managed) No Complexity Medium High Low Very low Best for Task queues, routing Event streaming Simple AWS queues Real-time push ","date":"28-Jul-2025","heading":"14.3 Popular Message Queue Systems","objectID":"\/posts\/system-design\/:14:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#143-popular-message-queue-systems"},{"categories":["software-engineering"],"collections":null,"content":"14.4 Event-Driven Architecture Example # Event Bus class EventBus: def __init__(self): self.handlers = {} def subscribe(self, event_type, handler): if event_type not in self.handlers: self.handlers[event_type] = [] self.handlers[event_type].append(handler) def publish(self, event): event_type = event.get('type') if event_type in self.handlers: for handler in self.handlers[event_type]: try: handler(event) except Exception as e: print(f\"Handler error: {e}\") # Services def send_confirmation_email(event): user_id = event['user_id'] order_id = event['order_id'] print(f\"Sending email to user {user_id} for order {order_id}\") def update_inventory(event): items = event['items'] print(f\"Updating inventory for {items}\") def send_notification(event): user_id = event['user_id'] print(f\"Sending push notification to user {user_id}\") # Setup event_bus = EventBus() event_bus.subscribe('order_placed', send_confirmation_email) event_bus.subscribe('order_placed', update_inventory) event_bus.subscribe('order_placed', send_notification) # Trigger event event_bus.publish({ 'type': 'order_placed', 'order_id': 12345, 'user_id': 789, 'items': ['item1', 'item2'] }) ","date":"28-Jul-2025","heading":"14.4 Event-Driven Architecture Example","objectID":"\/posts\/system-design\/:14:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#144-event-driven-architecture-example"},{"categories":["software-engineering"],"collections":null,"content":"15. Microservices Architecture ","date":"28-Jul-2025","heading":"15. Microservices Architecture","objectID":"\/posts\/system-design\/:15:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#15-microservices-architecture"},{"categories":["software-engineering"],"collections":null,"content":"15.1 Monolith vs Microservices Microservices Architecture \u2014 Service decomposition, API gateway pattern, and inter-service communication for distributed systems Monolith Advantages: Simple to develop and test Easy deployment Good for small teams Lower latency (no network calls) Monolith Disadvantages: Difficult to scale Long deployment times One bug can bring down entire system Hard to adopt new technologies Microservices Advantages: Independent scaling Technology diversity Fault isolation Faster deployment Team autonomy Microservices Disadvantages: Increased complexity Network latency Data consistency challenges Testing is harder Operational overhead ","date":"28-Jul-2025","heading":"15.1 Monolith vs Microservices","objectID":"\/posts\/system-design\/:15:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#151-monolith-vs-microservices"},{"categories":["software-engineering"],"collections":null,"content":"15.2 Microservices Best Practices 1. Single Responsibility Each service should do one thing well: Good: - UserService: Manages users - OrderService: Manages orders - EmailService: Sends emails Bad: - UserOrderEmailService: Does everything 2. Database per Service Each service has its own database: # User Service class UserService: def __init__(self): self.db = UserDatabase() def create_user(self, data): return self.db.insert(data) def get_user(self, user_id): return self.db.query(user_id) # Order Service class OrderService: def __init__(self): self.db = OrderDatabase() # Different database self.user_service = UserServiceClient() def create_order(self, user_id, items): # Get user info via API (not direct DB access) user = self.user_service.get_user(user_id) order = { 'user_id': user_id, 'items': items, 'total': self.calculate_total(items) } return self.db.insert(order) 3. API Gateway Pattern Single entry point for all clients: class APIGateway: def __init__(self): self.user_service = UserServiceClient() self.order_service = OrderServiceClient() self.product_service = ProductServiceClient() def get_user_dashboard(self, user_id): # Aggregate data from multiple services user = self.user_service.get_user(user_id) orders = self.order_service.get_user_orders(user_id) recommendations = self.product_service.get_recommendations(user_id) return { 'user': user, 'recent_orders': orders[:5], 'recommended_products': recommendations } 4. Circuit Breaker Pattern Prevent cascading failures: from enum import Enum import time class CircuitState(Enum): CLOSED = \"closed\" # Normal operation OPEN = \"open\" # Failing, reject requests HALF_OPEN = \"half_open\" # Testing if recovered class CircuitBreaker: def __init__(self, failure_threshold=5, timeout=60): self.failure_count = 0 self.failure_threshold = failure_threshold self.timeout = timeout self.state = CircuitState.CLOSED self.last_failure_time = None def call(self, func, *args, **kwargs): if self.state == CircuitState.OPEN: if time.time() - self.last_failure_time > self.timeout: self.state = CircuitState.HALF_OPEN else: raise Exception(\"Circuit breaker is OPEN\") try: result = func(*args, **kwargs) self.on_success() return result except Exception as e: self.on_failure() raise e def on_success(self): self.failure_count = 0 self.state = CircuitState.CLOSED def on_failure(self): self.failure_count += 1 self.last_failure_time = time.time() if self.failure_count >= self.failure_threshold: self.state = CircuitState.OPEN # Usage payment_circuit = CircuitBreaker(failure_threshold=5, timeout=60) def process_payment(amount): return payment_circuit.call(payment_service.charge, amount) ","date":"28-Jul-2025","heading":"15.2 Microservices Best Practices","objectID":"\/posts\/system-design\/:15:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#152-microservices-best-practices"},{"categories":["software-engineering"],"collections":null,"content":"16. Service Mesh and API Gateway ","date":"28-Jul-2025","heading":"16. Service Mesh and API Gateway","objectID":"\/posts\/system-design\/:16:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#16-service-mesh-and-api-gateway"},{"categories":["software-engineering"],"collections":null,"content":"16.1 API Gateway An API Gateway is the single entry point for all client requests to a microservices system. It handles cross-cutting concerns so individual services don’t have to. API Gateway Pattern \u2014 Routing, rate limiting, authentication, and protocol translation for scalable microservices What an API Gateway does: Concern Description Authentication \/ Authorization Verify JWT tokens or API keys before routing Rate Limiting Enforce per-client request quotas Request Routing Map \/api\/users \u2192 UserService, \/api\/orders \u2192 OrderService SSL Termination Decrypt HTTPS at the gateway, use plain HTTP internally Load Balancing Distribute traffic across service instances Request\/Response Transformation Protocol translation (REST \u2192 gRPC), header injection Circuit Breaking Fail fast when downstream services are unhealthy Observability Centralized access logs, request tracing Popular API Gateways: Kong \u2014 Open source, plugin-based, high performance (built on NGINX\/OpenResty) AWS API Gateway \u2014 Fully managed, native Lambda integration NGINX \/ Caddy \u2014 Lightweight, often used as a reverse proxy with gateway functionality Traefik \u2014 Cloud-native, auto-discovers services in Kubernetes class APIGateway: def __init__(self): self.routes = {} self.rate_limiter = DistributedRateLimiter(redis_client) self.auth = JWTAuthenticator() def register_route(self, path_prefix, service_url): self.routes[path_prefix] = service_url def handle_request(self, request): # 1. Authentication token = request.headers.get('Authorization', '').replace('Bearer ', '') if not self.auth.verify(token): return Response(status=401, body='Unauthorized') # 2. Rate limiting client_id = self.auth.get_subject(token) if not self.rate_limiter.allow_request(client_id, limit=1000, window=3600): return Response(status=429, body='Too Many Requests') # 3. Route to downstream service for prefix, service_url in self.routes.items(): if request.path.startswith(prefix): downstream_path = request.path[len(prefix):] return self.forward(request, service_url, downstream_path) return Response(status=404, body='Not Found') def forward(self, request, service_url, path): \"\"\"Forward request to downstream service.\"\"\" import requests as http response = http.request( method=request.method, url=f\"{service_url}{path}\", headers=request.headers, data=request.body, timeout=5 ) return Response(status=response.status_code, body=response.text) ","date":"28-Jul-2025","heading":"16.1 API Gateway","objectID":"\/posts\/system-design\/:16:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#161-api-gateway"},{"categories":["software-engineering"],"collections":null,"content":"16.2 Service Mesh A service mesh manages service-to-service communication within a microservices cluster. Unlike an API Gateway (which handles north-south traffic from clients), a service mesh handles east-west traffic between services. Service Mesh \u2014 Istio and Envoy sidecar proxy architecture for microservices communication, observability, and security How it works \u2014 Sidecar Proxy Pattern: Each service pod gets a sidecar proxy (typically Envoy) All inbound\/outbound traffic passes through the proxy The control plane configures proxies centrally (Istio’s istiod) What a Service Mesh provides: Feature Description Mutual TLS (mTLS) Automatic encryption between services, certificate rotation Load Balancing Intelligent traffic splitting, canary deployments Circuit Breaking Automatic failure detection and request short-circuiting Observability Distributed tracing, metrics per service, traffic visualization Retry \/ Timeout Configurable retry policies without application code changes Traffic Management A\/B testing, blue-green deployments, fault injection Popular Service Meshes: Istio \u2014 Most feature-rich, uses Envoy sidecars + istiod control plane Linkerd \u2014 Lightweight, Rust-based data plane AWS App Mesh \u2014 AWS-native, integrates with ECS and EKS Consul Connect \u2014 HashiCorp, multi-platform ","date":"28-Jul-2025","heading":"16.2 Service Mesh","objectID":"\/posts\/system-design\/:16:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#162-service-mesh"},{"categories":["software-engineering"],"collections":null,"content":"16.3 API Gateway vs Service Mesh Concern API Gateway Service Mesh Traffic direction North-South (client \u2192 system) East-West (service \u2192 service) Authentication External clients (JWT, API keys) Internal mTLS between services Typical location Cluster edge \/ perimeter Inside the cluster (sidecar) Scope Single entry point Every service-to-service call Examples Kong, AWS API Gateway, NGINX Istio, Linkerd, Consul Use Both Together Production systems often use both: an API Gateway at the perimeter handles external clients and auth, while a Service Mesh manages internal service communication, mTLS, and observability. They are complementary, not alternatives. ","date":"28-Jul-2025","heading":"16.3 API Gateway vs Service Mesh","objectID":"\/posts\/system-design\/:16:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#163-api-gateway-vs-service-mesh"},{"categories":["software-engineering"],"collections":null,"content":"25. Scaling from Zero to Million Users Progressive Scaling Journey This section walks through building a system that scales from a single server to handling millions of concurrent users. Understanding this progression helps you make informed architecture decisions based on your current scale and future needs. ","date":"28-Jul-2025","heading":"25. Scaling from Zero to Million Users","objectID":"\/posts\/system-design\/:17:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#25-scaling-from-zero-to-million-users"},{"categories":["software-engineering"],"collections":null,"content":"25.1 Stage 1: Single Server Start Simple At the beginning, everything runs on one server. This is perfectly fine for early-stage startups and MVPs. Don’t over-engineer for scale you don’t have yet! Scaling from Zero to Million Users \u2014 Architecture evolution stages and strategies for rapid growth Architecture: All code in one place Single database Simple deployment No external services Configuration: 1 web server 1 database ~1,000-5,000 concurrent users Issues emerge when: Single point of failure Can’t scale beyond server capacity Database bottleneck ","date":"28-Jul-2025","heading":"25.1 Stage 1: Single Server","objectID":"\/posts\/system-design\/:17:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#251-stage-1-single-server"},{"categories":["software-engineering"],"collections":null,"content":"25.2 Stage 2: Separate Web and Database Tiers Scaling from Zero to Million Users \u2014 Architecture evolution stages and strategies for rapid growth Changes: Add load balancer (distribute traffic) Multiple web servers (can scale horizontally) Separate database (can upgrade independently) Capacity: ~100,000 concurrent users Load balancer: 1-2 Gbps throughput Web servers: 5-10 servers Database: Single instance (might be beefy) Implementation Example: # Load Balancer (Round Robin) class LoadBalancer: def __init__(self, servers): self.servers = servers self.current = 0 def route_request(self, request): server = self.servers[self.current] self.current = (self.current + 1) % len(self.servers) return server.handle(request) # Web Servers (Stateless) class WebServer: def __init__(self, db_connection): self.db = db_connection def handle(self, request): if request.path == '\/users\/123': user = self.db.query(f\"SELECT * FROM users WHERE id = 123\") return json.dumps(user) # Setup servers = [WebServer(db_connection), WebServer(db_connection)] lb = LoadBalancer(servers) ","date":"28-Jul-2025","heading":"25.2 Stage 2: Separate Web and Database Tiers","objectID":"\/posts\/system-design\/:17:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#252-stage-2-separate-web-and-database-tiers"},{"categories":["software-engineering"],"collections":null,"content":"25.3 Stage 3: Add Caching Layer Cache Levels in System \u2014 Comprehensive caching strategy: browser, CDN, application, and database caching Benefits: Reduce database load Faster response times Better user experience Implementation: from functools import wraps import time class CachingWebServer: def __init__(self, db, cache): self.db = db self.cache = cache def get_user(self, user_id): # Check cache first cache_key = f\"user:{user_id}\" cached_user = self.cache.get(cache_key) if cached_user: return cached_user # Cache miss - query database user = self.db.query(f\"SELECT * FROM users WHERE id = {user_id}\") # Store in cache for 1 hour self.cache.set(cache_key, user, ttl=3600) return user ","date":"28-Jul-2025","heading":"25.3 Stage 3: Add Caching Layer","objectID":"\/posts\/system-design\/:17:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#253-stage-3-add-caching-layer"},{"categories":["software-engineering"],"collections":null,"content":"25.4 Stage 4: Add CDN for Static Content CDN Global Distribution \u2014 How content delivery networks reduce latency and improve performance worldwide CDN Benefits: Serve content from location closest to user Reduce origin server load Faster page loads globally Lower bandwidth costs ","date":"28-Jul-2025","heading":"25.4 Stage 4: Add CDN for Static Content","objectID":"\/posts\/system-design\/:17:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#254-stage-4-add-cdn-for-static-content"},{"categories":["software-engineering"],"collections":null,"content":"25.5 Stage 5: Database Replication Database Replication \u2014 Master-slave and multi-master replication strategies for availability and performance Benefits: Master for writes Slaves for reads Better read performance High availability ","date":"28-Jul-2025","heading":"25.5 Stage 5: Database Replication","objectID":"\/posts\/system-design\/:17:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#255-stage-5-database-replication"},{"categories":["software-engineering"],"collections":null,"content":"25.6 Stage 6: Database Sharding Database Sharding \u2014 Horizontal partitioning strategies for scaling large databases; range, hash, geo, and directory-based sharding When to shard: Database size > 1TB Single master at capacity Need to scale beyond vertical limits Sharding key considerations: Even distribution Immutable after shard creation Avoid hot keys ","date":"28-Jul-2025","heading":"25.6 Stage 6: Database Sharding","objectID":"\/posts\/system-design\/:17:6","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#256-stage-6-database-sharding"},{"categories":["software-engineering"],"collections":null,"content":"25.7 Stage 7: Message Queue for Async Processing Message Queue System \u2014 Async communication pattern using Kafka, RabbitMQ for system scalability and resilience Benefits: Non-blocking operations Decouple producers and consumers Scale workers independently Handle traffic spikes ","date":"28-Jul-2025","heading":"25.7 Stage 7: Message Queue for Async Processing","objectID":"\/posts\/system-design\/:17:7","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#257-stage-7-message-queue-for-async-processing"},{"categories":["software-engineering"],"collections":null,"content":"19. Unique ID Generation in Distributed Systems Generating globally unique IDs is critical for distributed systems. IDs are used for order numbers, message IDs, user IDs, and any entity that must be uniquely addressable across all nodes without coordination. Distributed ID Generation \u2014 Snowflake ID, UUID, and timestamp-based strategies for generating unique identifiers at scale ","date":"28-Jul-2025","heading":"19. Unique ID Generation in Distributed Systems","objectID":"\/posts\/system-design\/:18:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#19-unique-id-generation-in-distributed-systems"},{"categories":["software-engineering"],"collections":null,"content":"19.1 Requirements Unique IDs must be: Unique across all servers Sortable (roughly ordered by time) Decentralized (no single point of failure) Fast generation Fit in 64 bits (for efficiency) ","date":"28-Jul-2025","heading":"19.1 Requirements","objectID":"\/posts\/system-design\/:18:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#191-requirements"},{"categories":["software-engineering"],"collections":null,"content":"19.2 Snowflake ID Algorithm Twitter’s Snowflake algorithm is industry standard: | 1 | 41 | 10 | 12 | |Unused |Timestamp |Machine ID |Sequence| |1 bit |ms since |Data center|Counter | | |epoch |+ Worker | | Structure (64 bits): Bit 0: Unused (ensures positive numbers) Bits 1-41: Timestamp (42 bits, milliseconds since epoch) Bits 42-51: Machine ID (10 bits, 1024 machines) Bits 52-63: Sequence number (12 bits, 4096 IDs per ms per machine) Capacity: 2^41 timestamps \u2248 69 years 2^10 machines = 1,024 machines 2^12 sequences = 4,096 IDs per millisecond per machine Total: ~4 million IDs per second per machine Implementation: import time import threading class SnowflakeIDGenerator: EPOCH = 1288834974657 # Twitter Epoch WORKER_ID_BITS = 10 DATACENTER_ID_BITS = 5 SEQUENCE_BITS = 12 MAX_WORKER_ID = (1 << WORKER_ID_BITS) - 1 MAX_DATACENTER_ID = (1 << DATACENTER_ID_BITS) - 1 MAX_SEQUENCE = (1 << SEQUENCE_BITS) - 1 WORKER_ID_SHIFT = SEQUENCE_BITS DATACENTER_ID_SHIFT = SEQUENCE_BITS + WORKER_ID_BITS TIMESTAMP_SHIFT = SEQUENCE_BITS + WORKER_ID_BITS + DATACENTER_ID_BITS def __init__(self, worker_id, datacenter_id): if worker_id > self.MAX_WORKER_ID or worker_id < 0: raise ValueError(f\"Worker ID must be <= {self.MAX_WORKER_ID}\") if datacenter_id > self.MAX_DATACENTER_ID or datacenter_id < 0: raise ValueError(f\"Datacenter ID must be <= {self.MAX_DATACENTER_ID}\") self.worker_id = worker_id self.datacenter_id = datacenter_id self.sequence = 0 self.last_timestamp = -1 self.lock = threading.Lock() def _current_timestamp(self): return int(time.time() * 1000) def generate_id(self): with self.lock: timestamp = self._current_timestamp() if timestamp < self.last_timestamp: raise Exception(\"Clock moved backwards\") if timestamp == self.last_timestamp: self.sequence += 1 if self.sequence > self.MAX_SEQUENCE: # Wait for next millisecond timestamp = self._current_timestamp() self.sequence = 0 else: self.sequence = 0 self.last_timestamp = timestamp id = ( (timestamp - self.EPOCH) << self.TIMESTAMP_SHIFT | self.datacenter_id << self.DATACENTER_ID_SHIFT | self.worker_id << self.WORKER_ID_SHIFT | self.sequence ) return id # Usage id_gen = SnowflakeIDGenerator(worker_id=1, datacenter_id=1) for _ in range(5): print(id_gen.generate_id()) ","date":"28-Jul-2025","heading":"19.2 Snowflake ID Algorithm","objectID":"\/posts\/system-design\/:18:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#192-snowflake-id-algorithm"},{"categories":["software-engineering"],"collections":null,"content":"19.3 Other ID Generation Methods UUID: 128 bits Guaranteed unique Not sortable Larger ID size Database Auto-Increment: Simple Requires network call Single point of failure potential Exposes data volume Timestamp + Random: Simple Sortable Risk of collision with multiple generators ","date":"28-Jul-2025","heading":"19.3 Other ID Generation Methods","objectID":"\/posts\/system-design\/:18:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#193-other-id-generation-methods"},{"categories":["software-engineering"],"collections":null,"content":"20. Distributed Transactions, Saga Pattern, and CQRS When Things Span Multiple Services In microservices, a single business operation (like placing an order) often touches multiple services and databases. Since each service owns its own database, you cannot use a standard database transaction across them. This section covers patterns for maintaining data consistency across service boundaries. ","date":"28-Jul-2025","heading":"20. Distributed Transactions, Saga Pattern, and CQRS","objectID":"\/posts\/system-design\/:19:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#20-distributed-transactions-saga-pattern-and-cqrs"},{"categories":["software-engineering"],"collections":null,"content":"20.1 The Problem: Distributed Transactions Classic ACID transactions work within a single database. Once you split data across services, you lose that guarantee: Monolith: BEGIN TRANSACTION UPDATE inventory SET stock = stock - 1 WHERE product_id = 123 INSERT INTO orders VALUES (...) UPDATE accounts SET balance = balance - 99.99 COMMIT \u2190 all or nothing Microservices: OrderService.createOrder() \u2192 writes to Order DB InventoryService.reserve() \u2192 writes to Inventory DB PaymentService.charge() \u2192 writes to Payment DB \u2190 NO global COMMIT, partial failures are real ","date":"28-Jul-2025","heading":"20.1 The Problem: Distributed Transactions","objectID":"\/posts\/system-design\/:19:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#201-the-problem-distributed-transactions"},{"categories":["software-engineering"],"collections":null,"content":"20.2 Two-Phase Commit (2PC) The traditional distributed transaction protocol. Rarely used in modern microservices due to performance and operational complexity. Two-Phase Commit (2PC) \u2014 Distributed transaction coordination across multiple services with prepare and commit phases How 2PC works: Phase 1 \u2014 Prepare: Coordinator sends PREPARE to all participants Each participant locks resources and writes to WAL, responds VOTE_COMMIT or VOTE_ABORT Phase 2 \u2014 Commit\/Abort: If all voted COMMIT \u2192 coordinator sends COMMIT to all If any voted ABORT \u2192 coordinator sends ROLLBACK to all class TwoPhaseCommitCoordinator: def __init__(self, participants): self.participants = participants def execute_transaction(self, operations): \"\"\"Execute distributed transaction across all participants.\"\"\" # Phase 1: Prepare votes = [] for participant, operation in zip(self.participants, operations): try: vote = participant.prepare(operation) votes.append(vote) except Exception: votes.append('ABORT') # Phase 2: Commit or Rollback if all(v == 'COMMIT' for v in votes): for participant in self.participants: participant.commit() return True else: for participant in self.participants: participant.rollback() return False Problems with 2PC: Blocking: If coordinator crashes after Phase 1, participants hold locks indefinitely Latency: Synchronous round-trips across services add 20\u201350% overhead Availability: System is unavailable during coordinator failure (CP, not AP) ","date":"28-Jul-2025","heading":"20.2 Two-Phase Commit (2PC)","objectID":"\/posts\/system-design\/:19:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#202-two-phase-commit-2pc"},{"categories":["software-engineering"],"collections":null,"content":"20.3 Saga Pattern The preferred approach for distributed transactions in microservices. Instead of one atomic transaction, a Saga is a sequence of local transactions where each step publishes an event that triggers the next. If any step fails, compensating transactions undo previous steps. Saga Pattern \u2014 Distributed transaction management through compensating transactions in microservices architectures Two Saga Implementation Styles: 1. Choreography (Event-driven) Each service reacts to events and triggers the next service. No central coordinator. # Order Service - publishes event class OrderService: def create_order(self, user_id, items, total): order = self.db.create_order(user_id, items, total, status='PENDING') # Publish event - Inventory Service will react self.kafka.publish('order.created', { 'order_id': order.id, 'items': items, 'total': total }) return order # Inventory Service - reacts to order.created class InventoryService: def on_order_created(self, event): try: self.reserve_items(event['items']) self.kafka.publish('inventory.reserved', {'order_id': event['order_id']}) except InsufficientStockError: # Trigger compensation self.kafka.publish('inventory.failed', {'order_id': event['order_id']}) # Order Service - compensates on failure class OrderService: def on_inventory_failed(self, event): self.db.update_order(event['order_id'], status='CANCELLED') self.kafka.publish('order.cancelled', {'order_id': event['order_id']}) 2. Orchestration (Central coordinator) A Saga orchestrator explicitly calls each service and handles failures. class OrderSagaOrchestrator: def execute(self, order_id, user_id, items, total): steps_completed = [] try: # Step 1: Reserve inventory self.inventory_service.reserve(items) steps_completed.append('inventory') # Step 2: Process payment self.payment_service.charge(user_id, total) steps_completed.append('payment') # Step 3: Confirm order self.order_service.confirm(order_id) steps_completed.append('order') return {'status': 'success', 'order_id': order_id} except Exception as e: # Compensate in reverse order self._compensate(steps_completed, order_id, items, user_id, total) return {'status': 'failed', 'reason': str(e)} def _compensate(self, steps, order_id, items, user_id, total): if 'payment' in steps: self.payment_service.refund(user_id, total) if 'inventory' in steps: self.inventory_service.release(items) if 'order' in steps: self.order_service.cancel(order_id) Choreography vs Orchestration: Choreography Orchestration Coordination Decentralized (events) Centralized (orchestrator) Coupling Loose Tighter (orchestrator knows all services) Visibility Hard to trace flow Easy to see whole saga in one place Failure handling Each service handles its own Orchestrator manages compensation Best for Simple flows, few steps Complex flows, many steps ","date":"28-Jul-2025","heading":"20.3 Saga Pattern","objectID":"\/posts\/system-design\/:19:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#203-saga-pattern"},{"categories":["software-engineering"],"collections":null,"content":"20.4 CQRS (Command Query Responsibility Segregation) Separate the write model (Commands) from the read model (Queries). This enables independent scaling, optimized read\/write schemas, and eventual consistency. CQRS and Event Sourcing \u2014 Command\/query separation and event-driven state reconstruction for scalable distributed architectures The Core Idea: Traditional: CQRS: \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Single Model Write Model (Commands) Read Model (Queries) \u2502 \u2502 \u2502 \u251c\u2500\u2500 GET users \u2500\u2500\u25ba Commands DB \u2500\u2500\u25ba Read DB \u2514\u2500\u2500 POST users \u2502 (normalized) \u2502 (denormalized, \u2514\u2500\u2500 publish events \u2500\u2500\u25ba optimized views) # Command side - handles writes class CreateOrderCommand: def __init__(self, user_id, items, total): self.user_id = user_id self.items = items self.total = total class OrderCommandHandler: def handle_create(self, command: CreateOrderCommand): order = self.orders_db.insert({ 'user_id': command.user_id, 'items': command.items, 'total': command.total, 'status': 'pending' }) # Publish event to update read models self.event_bus.publish('OrderCreated', order) return order.id # Query side - handles reads (separate optimized DB\/view) class OrderQueryHandler: def get_user_orders(self, user_id, limit=10): # Read from denormalized read model (pre-joined, pre-aggregated) return self.read_db.query( \"SELECT * FROM user_orders_view WHERE user_id = ? ORDER BY created_at DESC LIMIT ?\", [user_id, limit] ) ","date":"28-Jul-2025","heading":"20.4 CQRS (Command Query Responsibility Segregation)","objectID":"\/posts\/system-design\/:19:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#204-cqrs-command-query-responsibility-segregation"},{"categories":["software-engineering"],"collections":null,"content":"20.5 Event Sourcing Instead of storing the current state, store the sequence of events that led to it. State is reconstructed by replaying events. # Event store: append-only log of all events class EventStore: def append(self, aggregate_id, event): \"\"\"Append event to the store.\"\"\" self.store.append({ 'aggregate_id': aggregate_id, 'event_type': event['type'], 'data': event['data'], 'timestamp': time.time(), 'version': self.get_next_version(aggregate_id) }) def get_events(self, aggregate_id): \"\"\"Retrieve all events for an aggregate.\"\"\" return [e for e in self.store if e['aggregate_id'] == aggregate_id] # Reconstruct current state by replaying events class BankAccount: def __init__(self, account_id, event_store): self.account_id = account_id self.balance = 0 self._replay(event_store.get_events(account_id)) def _replay(self, events): for event in events: if event['event_type'] == 'MoneyDeposited': self.balance += event['data']['amount'] elif event['event_type'] == 'MoneyWithdrawn': self.balance -= event['data']['amount'] def deposit(self, amount, event_store): event_store.append(self.account_id, { 'type': 'MoneyDeposited', 'data': {'amount': amount} }) self.balance += amount # Benefits: # - Full audit trail (every change recorded) # - Time travel: reconstruct state at any point in time # - Event replay for debugging and analytics # - Natural fit with CQRS (events update read models) When to use Event Sourcing: Financial systems (audit trail required) Systems needing time-travel queries Complex domain models with many state transitions Systems that need to rebuild read models from scratch Event Sourcing Complexity Event sourcing adds significant complexity. Only use it when the benefits (audit trail, time travel, CQRS compatibility) justify the cost. Most CRUD applications do not need event sourcing. ","date":"28-Jul-2025","heading":"20.5 Event Sourcing","objectID":"\/posts\/system-design\/:19:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#205-event-sourcing"},{"categories":["software-engineering"],"collections":null,"content":"26. Design a Key-Value Store ","date":"28-Jul-2025","heading":"26. Design a Key-Value Store","objectID":"\/posts\/system-design\/:20:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#26-design-a-key-value-store"},{"categories":["software-engineering"],"collections":null,"content":"26.1 Requirements Functional: GET(key): retrieve value PUT(key, value): store value DELETE(key): remove value Non-Functional: Single node (vertical scalability only) O(1) time complexity for all operations Persistent storage Efficient memory usage Hardware-aware optimization ","date":"28-Jul-2025","heading":"26.1 Requirements","objectID":"\/posts\/system-design\/:20:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#261-requirements"},{"categories":["software-engineering"],"collections":null,"content":"26.2 Architecture Key-Value Store Architecture \u2014 Design patterns for distributed key-value systems and NoSQL storage optimization ","date":"28-Jul-2025","heading":"26.2 Architecture","objectID":"\/posts\/system-design\/:20:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#262-architecture"},{"categories":["software-engineering"],"collections":null,"content":"26.3 Implementation import os import json import time from threading import Lock class SimpleKVStore: def __init__(self, data_dir=\".\/data\"): self.data_dir = data_dir self.memory_index = {} # In-memory hash map self.wal_file = os.path.join(data_dir, \"wal.log\") # Write-ahead log self.data_file = os.path.join(data_dir, \"data.db\") # Actual data self.lock = Lock() os.makedirs(data_dir, exist_ok=True) self._load_from_disk() def _load_from_disk(self): \"\"\"Load index from disk on startup\"\"\" if os.path.exists(self.wal_file): with open(self.wal_file, 'r') as f: for line in f: entry = json.loads(line.strip()) if entry['op'] == 'PUT': self.memory_index[entry['key']] = entry['value'] elif entry['op'] == 'DELETE': self.memory_index.pop(entry['key'], None) def get(self, key): with self.lock: if key not in self.memory_index: return None return self.memory_index[key] def put(self, key, value): with self.lock: # Write-ahead log (WAL) entry = {'op': 'PUT', 'key': key, 'value': value, 'timestamp': time.time()} with open(self.wal_file, 'a') as f: f.write(json.dumps(entry) + '\\n') # Update in-memory index self.memory_index[key] = value # Periodic flush to disk (simplified) if len(self.memory_index) % 1000 == 0: self._flush_to_disk() def delete(self, key): with self.lock: if key not in self.memory_index: return False # Write deletion to WAL entry = {'op': 'DELETE', 'key': key, 'timestamp': time.time()} with open(self.wal_file, 'a') as f: f.write(json.dumps(entry) + '\\n') # Remove from memory del self.memory_index[key] return True def _flush_to_disk(self): \"\"\"Periodically flush in-memory index to disk\"\"\" with open(self.data_file, 'w') as f: for key, value in self.memory_index.items(): f.write(f\"{key}={json.dumps(value)}\\n\") # Usage store = SimpleKVStore() store.put(\"user:1\", {\"name\": \"Alice\", \"age\": 30}) store.put(\"user:2\", {\"name\": \"Bob\", \"age\": 25}) print(store.get(\"user:1\")) # {\"name\": \"Alice\", \"age\": 30} store.delete(\"user:2\") print(store.get(\"user:2\")) # None ","date":"28-Jul-2025","heading":"26.3 Implementation","objectID":"\/posts\/system-design\/:20:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#263-implementation"},{"categories":["software-engineering"],"collections":null,"content":"28. Design Twitter with Microservices ","date":"28-Jul-2025","heading":"28. Design Twitter with Microservices","objectID":"\/posts\/system-design\/:21:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#28-design-twitter-with-microservices"},{"categories":["software-engineering"],"collections":null,"content":"28.1 High-Level Architecture Twitter Microservices Design \u2014 Real-time social media system architecture for billions of users and interactions ","date":"28-Jul-2025","heading":"28.1 High-Level Architecture","objectID":"\/posts\/system-design\/:21:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#281-high-level-architecture"},{"categories":["software-engineering"],"collections":null,"content":"28.2 Core Services Explained Tweet Service: Create, read, update, delete tweets Validate tweet content Handle media uploads Publish tweet events Feed Service: Generate personalized feed Combine tweets from followed accounts Sort chronologically Cache frequently accessed feeds Search Service: Full-text search on tweets Handle hashtags Trending topics Use Elasticsearch or similar Follow Service: Manage follower\/following relationships Social graph Notification for new followers ","date":"28-Jul-2025","heading":"28.2 Core Services Explained","objectID":"\/posts\/system-design\/:21:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#282-core-services-explained"},{"categories":["software-engineering"],"collections":null,"content":"28.3 Handling the “Thundering Herd” Challenge: When celebrity tweets, millions access simultaneously. Solution: Fan-Out Cache # When a celebrity tweets def create_celebrity_tweet(user_id, content): tweet = TweetService.create(user_id, content) # Get all followers followers = FollowService.get_followers(user_id) # Pre-populate feeds for active followers for follower_id in followers[:100000]: # Top 100K followers feed_key = f\"feed:{follower_id}\" cache.prepend(feed_key, tweet) # Add to front of feed # Enqueue notification notification_queue.enqueue({ 'user_id': follower_id, 'event': 'new_tweet', 'from_user': user_id }) ","date":"28-Jul-2025","heading":"28.3 Handling the “Thundering Herd”","objectID":"\/posts\/system-design\/:21:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#283-handling-the-thundering-herd"},{"categories":["software-engineering"],"collections":null,"content":"29. Design Google Maps \/ Location-Based Services ","date":"28-Jul-2025","heading":"29. Design Google Maps \/ Location-Based Services","objectID":"\/posts\/system-design\/:22:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#29-design-google-maps--location-based-services"},{"categories":["software-engineering"],"collections":null,"content":"29.1 Architecture Overview Location-Based Services \u2014 Geospatial database design and efficient proximity search implementation ","date":"28-Jul-2025","heading":"29.1 Architecture Overview","objectID":"\/posts\/system-design\/:22:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#291-architecture-overview"},{"categories":["software-engineering"],"collections":null,"content":"29.2 Geospatial Indexing Storing location data efficiently: # Using Geohash for spatial indexing def geohash_encode(latitude, longitude, precision=5): \"\"\" Geohash encodes lat\/long into a short string Nearby locations have similar geohashes \"\"\" # Simplified geohash (actual implementation is more complex) lat_bits = bin(int((latitude + 90) \/ 180 * (2 ** 20)))[2:].zfill(20) lon_bits = bin(int((longitude + 180) \/ 360 * (2 ** 20)))[2:].zfill(20) # Interleave bits interleaved = ''.join(lon_bits[i] + lat_bits[i] for i in range(20)) # Convert to base32 base32 = \"0123456789bcdefghjkmnpqrstuvwxyz\" geohash = '' for i in range(0, len(interleaved), 5): chunk = interleaved[i:i+5] geohash += base32[int(chunk, 2)] return geohash[:precision] # Nearby search using geohash def find_nearby_places(latitude, longitude, radius_km=1): \"\"\"Find all places within radius\"\"\" center_geohash = geohash_encode(latitude, longitude, precision=7) # Get geohashes for surrounding cells nearby_geohashes = get_neighbors(center_geohash) places = [] for geohash in nearby_geohashes: # Query database for places with this geohash prefix db_places = PlaceDB.query(f\"geohash LIKE '{geohash}%'\") # Filter by distance for place in db_places: dist = haversine_distance(latitude, longitude, place.lat, place.lon) if dist <= radius_km: places.append(place) return places def haversine_distance(lat1, lon1, lat2, lon2): \"\"\"Calculate distance between two points in km\"\"\" import math R = 6371 # Earth radius in km dlat = math.radians(lat2 - lat1) dlon = math.radians(lon2 - lon1) a = math.sin(dlat\/2)**2 + math.cos(math.radians(lat1)) * math.cos(math.radians(lat2)) * math.sin(dlon\/2)**2 c = 2 * math.atan2(math.sqrt(a), math.sqrt(1-a)) return R * c ","date":"28-Jul-2025","heading":"29.2 Geospatial Indexing","objectID":"\/posts\/system-design\/:22:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#292-geospatial-indexing"},{"categories":["software-engineering"],"collections":null,"content":"29.3 Direction Finding (Shortest Path) # Simplified Dijkstra's algorithm for shortest path import heapq class DirectionFinder: def __init__(self, road_network): self.graph = road_network # Graph of intersections and roads def find_shortest_path(self, start_id, end_id): \"\"\"Find shortest path between two locations\"\"\" distances = {start_id: 0} previous = {} pq = [(0, start_id)] # (distance, node) visited = set() while pq: current_dist, current_node = heapq.heappop(pq) if current_node in visited: continue visited.add(current_node) # Check neighbors for neighbor, weight in self.graph[current_node]: if neighbor in visited: continue new_dist = current_dist + weight if neighbor not in distances or new_dist < distances[neighbor]: distances[neighbor] = new_dist previous[neighbor] = current_node heapq.heappush(pq, (new_dist, neighbor)) # Reconstruct path path = [] current = end_id while current in previous: path.append(current) current = previous[current] path.append(start_id) path.reverse() return path, distances.get(end_id, float('inf')) ","date":"28-Jul-2025","heading":"29.3 Direction Finding (Shortest Path)","objectID":"\/posts\/system-design\/:22:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#293-direction-finding-shortest-path"},{"categories":["software-engineering"],"collections":null,"content":"17. Rate Limiting Rate limiting controls how many requests a client can make within a time window, protecting services from abuse, DDoS attacks, and runaway consumers. Rate Limiting Algorithms \u2014 Token bucket, leaky bucket, sliding window counter implementations for API protection Where to Apply Rate Limiting API Gateway level: Protect all services uniformly before requests reach your application Per user \/ IP: Prevent single bad actor from degrading service for everyone Per endpoint: More aggressive limits on expensive operations (e.g. \/search vs \/ping) Distributed rate limiting: Use Redis so limits apply across all API server instances ","date":"28-Jul-2025","heading":"17. Rate Limiting","objectID":"\/posts\/system-design\/:23:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#17-rate-limiting"},{"categories":["software-engineering"],"collections":null,"content":"17.1 Rate Limiting Algorithms 1. Token Bucket Algorithm Pre-calculated tokens in a bucket, remove on each request: import time from threading import Lock class TokenBucket: def __init__(self, capacity, refill_rate): \"\"\" capacity: max tokens in bucket refill_rate: tokens added per second \"\"\" self.capacity = capacity self.refill_rate = refill_rate self.tokens = capacity self.last_refill = time.time() self.lock = Lock() def allow_request(self, tokens=1): with self.lock: self._refill() if self.tokens >= tokens: self.tokens -= tokens return True return False def _refill(self): now = time.time() elapsed = now - self.last_refill self.tokens = min( self.capacity, self.tokens + elapsed * self.refill_rate ) self.last_refill = now # Usage bucket = TokenBucket(capacity=10, refill_rate=1) # 10 tokens, 1 per second for i in range(15): if bucket.allow_request(): print(f\"Request {i+1}: Allowed\") else: print(f\"Request {i+1}: Rate limited\") 2. Leaky Bucket Algorithm Drain bucket at fixed rate: from collections import deque import time from threading import Lock class LeakyBucket: def __init__(self, capacity, outflow_rate): \"\"\" capacity: max requests in queue outflow_rate: requests processed per second \"\"\" self.capacity = capacity self.outflow_rate = outflow_rate self.queue = deque() self.last_leak = time.time() self.lock = Lock() def allow_request(self): with self.lock: self._leak() if len(self.queue) < self.capacity: self.queue.append(time.time()) return True return False def _leak(self): now = time.time() elapsed = now - self.last_leak leak_count = int(elapsed * self.outflow_rate) for _ in range(leak_count): if self.queue: self.queue.popleft() self.last_leak = now 3. Sliding Window Counter Hybrid approach combining fixed and sliding window: import time from threading import Lock class SlidingWindowCounter: def __init__(self, window_size, max_requests): \"\"\" window_size: time window in seconds max_requests: max requests allowed in window \"\"\" self.window_size = window_size self.max_requests = max_requests self.counters = {} # {window_start: count} self.lock = Lock() def allow_request(self, identifier): with self.lock: now = time.time() window_start = int(now \/ self.window_size) * self.window_size # Remove old windows for key in list(self.counters.keys()): if key < window_start - self.window_size: del self.counters[key] # Check current window key = f\"{identifier}:{window_start}\" if key not in self.counters: self.counters[key] = 0 if self.counters[key] < self.max_requests: self.counters[key] += 1 return True return False ","date":"28-Jul-2025","heading":"17.1 Rate Limiting Algorithms","objectID":"\/posts\/system-design\/:23:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#171-rate-limiting-algorithms"},{"categories":["software-engineering"],"collections":null,"content":"17.2 Distributed Rate Limiting For multi-server deployments, use Redis: import redis import time class DistributedRateLimiter: def __init__(self, redis_client, key_prefix=\"rate_limit\"): self.redis = redis_client self.key_prefix = key_prefix def allow_request(self, identifier, limit, window): \"\"\" identifier: user_id or IP limit: max requests window: time window in seconds \"\"\" key = f\"{self.key_prefix}:{identifier}\" now = time.time() window_start = now - window # Remove old entries self.redis.zremrangebyscore(key, 0, window_start) # Count requests in current window count = self.redis.zcard(key) if count < limit: # Add current request self.redis.zadd(key, {str(now): now}) # Set expiration self.redis.expire(key, window + 1) return True return False # Usage redis_client = redis.Redis(host='localhost', port=6379) limiter = DistributedRateLimiter(redis_client) for i in range(5): if limiter.allow_request('user:123', limit=3, window=60): print(f\"Request {i+1}: Allowed\") else: print(f\"Request {i+1}: Rate limited\") ","date":"28-Jul-2025","heading":"17.2 Distributed Rate Limiting","objectID":"\/posts\/system-design\/:23:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#172-distributed-rate-limiting"},{"categories":["software-engineering"],"collections":null,"content":"18. Consistent Hashing Consistent hashing solves the problem of routing requests to nodes in a dynamic cluster (cache servers, storage nodes) where nodes can be added or removed without remapping most keys. Consistent Hashing Ring \u2014 Virtual nodes and hash ring design for distributed cache and storage systems ","date":"28-Jul-2025","heading":"18. Consistent Hashing","objectID":"\/posts\/system-design\/:24:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#18-consistent-hashing"},{"categories":["software-engineering"],"collections":null,"content":"18.1 Problem Statement Standard hashing breaks when servers are added\/removed: # Standard hashing - doesn't work well with dynamic servers def standard_hash(key, num_servers): return hash(key) % num_servers When you add\/remove a server, most keys rehash to different locations! ","date":"28-Jul-2025","heading":"18.1 Problem Statement","objectID":"\/posts\/system-design\/:24:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#181-problem-statement"},{"categories":["software-engineering"],"collections":null,"content":"18.2 Consistent Hashing Solution Maps both keys and servers onto a circle: import hashlib from bisect import bisect_right class ConsistentHashing: def __init__(self, num_virtual_nodes=150): self.num_virtual_nodes = num_virtual_nodes self.ring = {} # hash -> server_id self.sorted_hashes = [] self.servers = set() def _hash(self, key): \"\"\"Hash function\"\"\" return int(hashlib.md5(str(key).encode()).hexdigest(), 16) def add_server(self, server_id): \"\"\"Add server to ring\"\"\" self.servers.add(server_id) for i in range(self.num_virtual_nodes): virtual_key = f\"{server_id}:{i}\" hash_value = self._hash(virtual_key) self.ring[hash_value] = server_id self.sorted_hashes.append(hash_value) self.sorted_hashes.sort() def remove_server(self, server_id): \"\"\"Remove server from ring\"\"\" self.servers.discard(server_id) hashes_to_remove = [] for i in range(self.num_virtual_nodes): virtual_key = f\"{server_id}:{i}\" hash_value = self._hash(virtual_key) hashes_to_remove.append(hash_value) del self.ring[hash_value] for h in hashes_to_remove: self.sorted_hashes.remove(h) def get_server(self, key): \"\"\"Get server for key\"\"\" if not self.servers: return None hash_value = self._hash(key) # Find the first server hash >= key hash idx = bisect_right(self.sorted_hashes, hash_value) if idx == len(self.sorted_hashes): idx = 0 server_hash = self.sorted_hashes[idx] return self.ring[server_hash] # Usage ch = ConsistentHashing() ch.add_server(\"server:1\") ch.add_server(\"server:2\") ch.add_server(\"server:3\") # Map keys to servers for key in [\"user:1\", \"user:2\", \"user:3\", \"user:4\", \"user:5\"]: server = ch.get_server(key) print(f\"{key} -> {server}\") # Add new server - only some keys rehash print(\"\\nAdding server:4...\") ch.add_server(\"server:4\") for key in [\"user:1\", \"user:2\", \"user:3\", \"user:4\", \"user:5\"]: server = ch.get_server(key) print(f\"{key} -> {server}\") Benefits: Adding\/removing server affects only 1\/N of keys Virtual nodes ensure better distribution Works well for cache distribution ","date":"28-Jul-2025","heading":"18.2 Consistent Hashing Solution","objectID":"\/posts\/system-design\/:24:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#182-consistent-hashing-solution"},{"categories":["software-engineering"],"collections":null,"content":"27. Design a URL Shortener ","date":"28-Jul-2025","heading":"27. Design a URL Shortener","objectID":"\/posts\/system-design\/:25:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#27-design-a-url-shortener"},{"categories":["software-engineering"],"collections":null,"content":"27.1 Requirements Functional: Shorten long URLs Redirect short URL to long URL Custom short URLs (optional) Non-Functional: High availability Low latency Shortened URL unique and not guessable ","date":"28-Jul-2025","heading":"27.1 Requirements","objectID":"\/posts\/system-design\/:25:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#271-requirements"},{"categories":["software-engineering"],"collections":null,"content":"27.2 Architecture URL Shortener System \u2014 System design for high-traffic URL shortening services with caching and analytics ","date":"28-Jul-2025","heading":"27.2 Architecture","objectID":"\/posts\/system-design\/:25:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#272-architecture"},{"categories":["software-engineering"],"collections":null,"content":"27.3 Implementation import string import random from datetime import datetime class URLShortener: def __init__(self, base_url=\"http:\/\/short.url\", cache=None, db=None): self.base_url = base_url self.cache = cache or {} self.db = db or {} self.characters = string.ascii_letters + string.digits def _generate_short_code(self, length=6): \"\"\"Generate random short code\"\"\" return ''.join(random.choice(self.characters) for _ in range(length)) def shorten(self, long_url, custom_code=None): \"\"\"Create shortened URL\"\"\" short_code = custom_code or self._generate_short_code() # Check if custom code exists if short_code in self.db: return None # Already exists url_record = { 'long_url': long_url, 'short_code': short_code, 'created_at': datetime.now(), 'expires_at': None, 'click_count': 0 } # Store in database self.db[short_code] = url_record # Cache it self.cache[short_code] = long_url return f\"{self.base_url}\/{short_code}\" def expand(self, short_code): \"\"\"Get original URL\"\"\" # Check cache first if short_code in self.cache: return self.cache[short_code] # Check database if short_code in self.db: long_url = self.db[short_code]['long_url'] # Update cache self.cache[short_code] = long_url # Increment click count self.db[short_code]['click_count'] += 1 return long_url return None def get_stats(self, short_code): \"\"\"Get URL statistics\"\"\" if short_code not in self.db: return None record = self.db[short_code] return { 'short_code': short_code, 'long_url': record['long_url'], 'created_at': record['created_at'], 'click_count': record['click_count'] } # Usage shortener = URLShortener() # Create shortened URL short = shortener.shorten(\"https:\/\/www.example.com\/very\/long\/url\/path\") print(f\"Shortened: {short}\") # Expand shortened URL long = shortener.expand(short.split('\/')[-1]) print(f\"Expanded: {long}\") # Get statistics code = short.split('\/')[-1] stats = shortener.get_stats(code) print(f\"Stats: {stats}\") ","date":"28-Jul-2025","heading":"27.3 Implementation","objectID":"\/posts\/system-design\/:25:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#273-implementation"},{"categories":["software-engineering"],"collections":null,"content":"30. Design an Object Storage System (S3) ","date":"28-Jul-2025","heading":"30. Design an Object Storage System (S3)","objectID":"\/posts\/system-design\/:26:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#30-design-an-object-storage-system-s3"},{"categories":["software-engineering"],"collections":null,"content":"30.1 High-Level Design Object Storage Architecture \u2014 S3-compatible distributed storage system design for petabyte-scale data ","date":"28-Jul-2025","heading":"30.1 High-Level Design","objectID":"\/posts\/system-design\/:26:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#301-high-level-design"},{"categories":["software-engineering"],"collections":null,"content":"30.2 Implementation Concepts from dataclasses import dataclass from datetime import datetime import hashlib import os @dataclass class ObjectMetadata: bucket: str key: str size: int content_type: str etag: str # hash for integrity created_at: datetime replicas: list # list of storage nodes with copies class S3LikeStorage: def __init__(self, storage_nodes=3): self.metadata = {} # bucket\/key -> metadata self.storage_nodes = [f\"node_{i}\" for i in range(storage_nodes)] self.object_store = {} # actual data storage def put_object(self, bucket, key, data, content_type=\"binary\"): \"\"\"Upload object\"\"\" # Calculate hash etag = hashlib.md5(data).hexdigest() # Store metadata metadata = ObjectMetadata( bucket=bucket, key=key, size=len(data), content_type=content_type, etag=etag, created_at=datetime.now(), replicas=self.storage_nodes.copy() ) full_key = f\"{bucket}\/{key}\" self.metadata[full_key] = metadata self.object_store[full_key] = data # Replicate to other nodes self._replicate(full_key, data) return etag def get_object(self, bucket, key): \"\"\"Download object\"\"\" full_key = f\"{bucket}\/{key}\" if full_key not in self.object_store: return None return self.object_store[full_key] def delete_object(self, bucket, key): \"\"\"Delete object\"\"\" full_key = f\"{bucket}\/{key}\" if full_key in self.metadata: del self.metadata[full_key] if full_key in self.object_store: del self.object_store[full_key] return True def list_objects(self, bucket, prefix=\"\"): \"\"\"List objects in bucket\"\"\" objects = [] for key in self.object_store: if key.startswith(f\"{bucket}\/\"): obj_key = key.replace(f\"{bucket}\/\", \"\") if obj_key.startswith(prefix): objects.append(obj_key) return objects def _replicate(self, key, data): \"\"\"Replicate data to storage nodes\"\"\" for node in self.storage_nodes: # In real system: write to remote storage node pass # Usage s3 = S3LikeStorage() # Upload etag = s3.put_object(\"my-bucket\", \"file.txt\", b\"Hello World\") print(f\"Uploaded with ETag: {etag}\") # Download data = s3.get_object(\"my-bucket\", \"file.txt\") print(f\"Downloaded: {data}\") # List objects = s3.list_objects(\"my-bucket\") print(f\"Objects: {objects}\") ","date":"28-Jul-2025","heading":"30.2 Implementation Concepts","objectID":"\/posts\/system-design\/:26:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#302-implementation-concepts"},{"categories":["software-engineering"],"collections":null,"content":"31. Design YouTube \/ Video Streaming Platform ","date":"28-Jul-2025","heading":"31. Design YouTube \/ Video Streaming Platform","objectID":"\/posts\/system-design\/:27:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#31-design-youtube--video-streaming-platform"},{"categories":["software-engineering"],"collections":null,"content":"31.1 Architecture Overview Video Streaming Platform \u2014 End-to-end video system design: upload, processing, storage, and adaptive delivery ","date":"28-Jul-2025","heading":"31.1 Architecture Overview","objectID":"\/posts\/system-design\/:27:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#311-architecture-overview"},{"categories":["software-engineering"],"collections":null,"content":"31.2 Video Processing Pipeline from enum import Enum class VideoQuality(Enum): LOW = \"360p\" MEDIUM = \"480p\" HIGH = \"720p\" HD = \"1080p\" FULL_HD = \"4K\" class VideoProcessingService: def __init__(self): self.processing_queue = [] self.storage = {} def upload_video(self, user_id, video_file): \"\"\"Handle video upload\"\"\" video_id = self._generate_id() # Store raw video raw_path = f\"raw\/{video_id}\/{video_file.name}\" self.storage[raw_path] = video_file # Queue for processing job = { 'video_id': video_id, 'user_id': user_id, 'raw_path': raw_path, 'status': 'queued' } self.processing_queue.append(job) return video_id def process_video(self, job): \"\"\"Process video: transcode to multiple qualities\"\"\" video_id = job['video_id'] raw_path = job['raw_path'] # Transcode to different qualities qualities = [VideoQuality.LOW, VideoQuality.MEDIUM, VideoQuality.HIGH, VideoQuality.HD] processed_paths = [] for quality in qualities: output_path = f\"processed\/{video_id}\/{quality.value}.mp4\" self._transcode(raw_path, output_path, quality) processed_paths.append(output_path) # Generate thumbnail thumbnail_path = f\"thumbnails\/{video_id}\/thumb.jpg\" self._generate_thumbnail(raw_path, thumbnail_path) # Update metadata self._update_metadata(video_id, { 'processed_paths': processed_paths, 'thumbnail': thumbnail_path, 'status': 'ready' }) return processed_paths def _transcode(self, input_path, output_path, quality): \"\"\"Transcode video to specified quality\"\"\" # Simplified - real implementation uses FFmpeg print(f\"Transcoding {input_path} to {quality.value}\") self.storage[output_path] = f\"transcoded_{quality.value}\" def _generate_thumbnail(self, video_path, output_path): \"\"\"Extract thumbnail from video\"\"\" print(f\"Generating thumbnail for {video_path}\") self.storage[output_path] = \"thumbnail_data\" def _generate_id(self): import uuid return str(uuid.uuid4()) def _update_metadata(self, video_id, metadata): \"\"\"Update video metadata\"\"\" print(f\"Updating metadata for {video_id}: {metadata}\") ","date":"28-Jul-2025","heading":"31.2 Video Processing Pipeline","objectID":"\/posts\/system-design\/:27:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#312-video-processing-pipeline"},{"categories":["software-engineering"],"collections":null,"content":"32. Design a Chat System ","date":"28-Jul-2025","heading":"32. Design a Chat System","objectID":"\/posts\/system-design\/:28:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#32-design-a-chat-system"},{"categories":["software-engineering"],"collections":null,"content":"32.1 Architecture Chat System Architecture \u2014 Real-time messaging design with persistence, notifications, and scalability ","date":"28-Jul-2025","heading":"32.1 Architecture","objectID":"\/posts\/system-design\/:28:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#321-architecture"},{"categories":["software-engineering"],"collections":null,"content":"32.2 Implementation import json from datetime import datetime from collections import defaultdict class ChatSystem: def __init__(self): self.connections = {} # user_id -> websocket self.messages = [] # message history self.online_users = set() self.typing_status = defaultdict(set) # chat_id -> set of typing users def connect_user(self, user_id, websocket): \"\"\"User connects to chat\"\"\" self.connections[user_id] = websocket self.online_users.add(user_id) # Notify others user is online self._broadcast_presence(user_id, 'online') return True def disconnect_user(self, user_id): \"\"\"User disconnects\"\"\" if user_id in self.connections: del self.connections[user_id] self.online_users.discard(user_id) # Notify others user is offline self._broadcast_presence(user_id, 'offline') def send_message(self, from_user, to_user, content): \"\"\"Send message from one user to another\"\"\" message = { 'id': len(self.messages) + 1, 'from': from_user, 'to': to_user, 'content': content, 'timestamp': datetime.now().isoformat(), 'read': False } # Store message self.messages.append(message) # Deliver if recipient online if to_user in self.connections: self._deliver_message(to_user, message) return message['id'] def send_group_message(self, from_user, group_id, content): \"\"\"Send message to group\"\"\" message = { 'id': len(self.messages) + 1, 'from': from_user, 'group_id': group_id, 'content': content, 'timestamp': datetime.now().isoformat() } self.messages.append(message) # Deliver to all group members group_members = self._get_group_members(group_id) for member in group_members: if member != from_user and member in self.connections: self._deliver_message(member, message) return message['id'] def mark_typing(self, user_id, chat_id, is_typing): \"\"\"Update typing status\"\"\" if is_typing: self.typing_status[chat_id].add(user_id) else: self.typing_status[chat_id].discard(user_id) # Notify other participants self._broadcast_typing(chat_id, user_id, is_typing) def get_message_history(self, user_id, other_user_id, limit=50): \"\"\"Get chat history between two users\"\"\" history = [] for msg in reversed(self.messages): if ((msg.get('from') == user_id and msg.get('to') == other_user_id) or (msg.get('from') == other_user_id and msg.get('to') == user_id)): history.append(msg) if len(history) >= limit: break return list(reversed(history)) def _deliver_message(self, user_id, message): \"\"\"Deliver message via WebSocket\"\"\" if user_id in self.connections: websocket = self.connections[user_id] # In real system: websocket.send(json.dumps(message)) print(f\"Delivered to {user_id}: {message}\") def _broadcast_presence(self, user_id, status): \"\"\"Notify all users about presence change\"\"\" for uid, ws in self.connections.items(): if uid != user_id: # Send presence notification print(f\"User {user_id} is {status}\") def _broadcast_typing(self, chat_id, user_id, is_typing): \"\"\"Notify typing status\"\"\" pass def _get_group_members(self, group_id): \"\"\"Get members of a group\"\"\" # Simplified - would query database return [] # Usage chat = ChatSystem() # Users connect chat.connect_user('alice', 'websocket_alice') chat.connect_user('bob', 'websocket_bob') # Send message msg_id = chat.send_message('alice', 'bob', 'Hello Bob!') # Get history history = chat.get_message_history('alice', 'bob') print(f\"Message history: {history}\") ","date":"28-Jul-2025","heading":"32.2 Implementation","objectID":"\/posts\/system-design\/:28:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#322-implementation"},{"categories":["software-engineering"],"collections":null,"content":"33. Design a News Feed System ","date":"28-Jul-2025","heading":"33. Design a News Feed System","objectID":"\/posts\/system-design\/:29:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#33-design-a-news-feed-system"},{"categories":["software-engineering"],"collections":null,"content":"33.1 Architecture News Feed System \u2014 Designing scalable social media feeds with ranking, caching, and real-time updates ","date":"28-Jul-2025","heading":"33.1 Architecture","objectID":"\/posts\/system-design\/:29:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#331-architecture"},{"categories":["software-engineering"],"collections":null,"content":"33.2 Feed Generation Strategies Fan-out on Write (Push Model): class FanoutOnWrite: def __init__(self): self.feeds = {} # user_id -> list of post_ids self.posts = {} # post_id -> post data self.followers = {} # user_id -> list of follower_ids def create_post(self, user_id, content): \"\"\"User creates a post\"\"\" post_id = len(self.posts) + 1 post = { 'id': post_id, 'user_id': user_id, 'content': content, 'created_at': datetime.now() } self.posts[post_id] = post # Fan-out: push to all followers' feeds followers = self.followers.get(user_id, []) for follower_id in followers: if follower_id not in self.feeds: self.feeds[follower_id] = [] self.feeds[follower_id].insert(0, post_id) # Add to front return post_id def get_feed(self, user_id, limit=10): \"\"\"Get user's news feed\"\"\" post_ids = self.feeds.get(user_id, [])[:limit] return [self.posts[pid] for pid in post_ids] Fan-out on Read (Pull Model): class FanoutOnRead: def __init__(self): self.posts = {} # user_id -> list of posts self.following = {} # user_id -> list of followed user_ids def create_post(self, user_id, content): \"\"\"User creates a post\"\"\" post_id = len(sum(self.posts.values(), [])) + 1 post = { 'id': post_id, 'user_id': user_id, 'content': content, 'created_at': datetime.now() } if user_id not in self.posts: self.posts[user_id] = [] self.posts[user_id].insert(0, post) return post_id def get_feed(self, user_id, limit=10): \"\"\"Aggregate feed on-the-fly\"\"\" followed_users = self.following.get(user_id, []) # Collect posts from all followed users all_posts = [] for followed_id in followed_users: all_posts.extend(self.posts.get(followed_id, [])) # Sort by timestamp and return top posts all_posts.sort(key=lambda p: p['created_at'], reverse=True) return all_posts[:limit] Hybrid Approach: class HybridFeedService: def __init__(self): self.fanout_write = FanoutOnWrite() self.fanout_read = FanoutOnRead() self.celebrity_threshold = 10000 # Users with >10K followers def create_post(self, user_id, content): \"\"\"Smart fan-out based on user popularity\"\"\" follower_count = len(self.fanout_write.followers.get(user_id, [])) if follower_count > self.celebrity_threshold: # Celebrity: use pull model return self.fanout_read.create_post(user_id, content) else: # Regular user: use push model return self.fanout_write.create_post(user_id, content) def get_feed(self, user_id, limit=10): \"\"\"Get personalized feed\"\"\" # Mix pre-computed and real-time feeds precomputed = self.fanout_write.get_feed(user_id, limit \/\/ 2) realtime = self.fanout_read.get_feed(user_id, limit \/\/ 2) # Merge and sort combined = precomputed + realtime combined.sort(key=lambda p: p['created_at'], reverse=True) return combined[:limit] ","date":"28-Jul-2025","heading":"33.2 Feed Generation Strategies","objectID":"\/posts\/system-design\/:29:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#332-feed-generation-strategies"},{"categories":["software-engineering"],"collections":null,"content":"34. Design a Search Autocomplete System ","date":"28-Jul-2025","heading":"34. Design a Search Autocomplete System","objectID":"\/posts\/system-design\/:30:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#34-design-a-search-autocomplete-system"},{"categories":["software-engineering"],"collections":null,"content":"34.1 Trie-Based Implementation Autocomplete System \u2014 Efficient search suggestion implementation using trie and ranking algorithms class TrieNode: def __init__(self): self.children = {} self.is_end = False self.frequency = 0 self.suggestions = [] # Top suggestions from this node class AutocompleteSystem: def __init__(self): self.root = TrieNode() def add_query(self, query, frequency=1): \"\"\"Add search query to trie\"\"\" node = self.root for char in query.lower(): if char not in node.children: node.children[char] = TrieNode() node = node.children[char] node.is_end = True node.frequency += frequency def search(self, prefix, limit=5): \"\"\"Get autocomplete suggestions\"\"\" node = self.root # Navigate to prefix for char in prefix.lower(): if char not in node.children: return [] node = node.children[char] # Collect all completions suggestions = [] self._collect_suggestions(node, prefix, suggestions) # Sort by frequency and return top results suggestions.sort(key=lambda x: x[1], reverse=True) return [s[0] for s in suggestions[:limit]] def _collect_suggestions(self, node, prefix, suggestions): \"\"\"DFS to collect all completions\"\"\" if node.is_end: suggestions.append((prefix, node.frequency)) for char, child_node in node.children.items(): self._collect_suggestions(child_node, prefix + char, suggestions) # Usage autocomplete = AutocompleteSystem() # Add popular queries queries = [ (\"system design\", 1000), (\"system design interview\", 800), (\"system design patterns\", 600), (\"system architecture\", 500), (\"distributed systems\", 900) ] for query, freq in queries: autocomplete.add_query(query, freq) # Search results = autocomplete.search(\"sys\") print(f\"Suggestions for 'sys': {results}\") ","date":"28-Jul-2025","heading":"34.1 Trie-Based Implementation","objectID":"\/posts\/system-design\/:30:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#341-trie-based-implementation"},{"categories":["software-engineering"],"collections":null,"content":"35. Design a Notification System ","date":"28-Jul-2025","heading":"35. Design a Notification System","objectID":"\/posts\/system-design\/:31:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#35-design-a-notification-system"},{"categories":["software-engineering"],"collections":null,"content":"35.1 Multi-Channel Notification System Notification System \u2014 Multi-channel messaging system design for reliable user notifications from abc import ABC, abstractmethod from enum import Enum class NotificationType(Enum): EMAIL = \"email\" SMS = \"sms\" PUSH = \"push\" IN_APP = \"in_app\" class NotificationChannel(ABC): @abstractmethod def send(self, recipient, message): pass class EmailChannel(NotificationChannel): def send(self, recipient, message): print(f\"Sending email to {recipient}: {message}\") # Implementation: SMTP, SendGrid, etc. class SMSChannel(NotificationChannel): def send(self, recipient, message): print(f\"Sending SMS to {recipient}: {message}\") # Implementation: Twilio, AWS SNS, etc. class PushChannel(NotificationChannel): def send(self, recipient, message): print(f\"Sending push to {recipient}: {message}\") # Implementation: FCM, APNS, etc. class InAppChannel(NotificationChannel): def send(self, recipient, message): print(f\"Sending in-app notification to {recipient}: {message}\") # Implementation: WebSocket, polling, etc. class NotificationService: def __init__(self): self.channels = { NotificationType.EMAIL: EmailChannel(), NotificationType.SMS: SMSChannel(), NotificationType.PUSH: PushChannel(), NotificationType.IN_APP: InAppChannel() } self.user_preferences = {} # user_id -> list of enabled channels self.notification_queue = [] def send_notification(self, user_id, message, channels=None): \"\"\"Send notification via specified channels\"\"\" if channels is None: channels = self.user_preferences.get(user_id, [NotificationType.PUSH, NotificationType.IN_APP]) for channel_type in channels: if channel_type in self.channels: channel = self.channels[channel_type] # Add to queue for async processing self.notification_queue.append({ 'user_id': user_id, 'channel': channel, 'message': message }) def process_queue(self): \"\"\"Process notification queue\"\"\" while self.notification_queue: notif = self.notification_queue.pop(0) try: notif['channel'].send(notif['user_id'], notif['message']) except Exception as e: print(f\"Failed to send notification: {e}\") # Retry logic here # Usage notification_service = NotificationService() # Send multi-channel notification notification_service.send_notification( user_id='user123', message='Your order has shipped!', channels=[NotificationType.EMAIL, NotificationType.PUSH] ) ","date":"28-Jul-2025","heading":"35.1 Multi-Channel Notification System","objectID":"\/posts\/system-design\/:31:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#351-multi-channel-notification-system"},{"categories":["software-engineering"],"collections":null,"content":"21. Common Design Patterns in System Design ","date":"28-Jul-2025","heading":"21. Common Design Patterns in System Design","objectID":"\/posts\/system-design\/:32:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#21-common-design-patterns-in-system-design"},{"categories":["software-engineering"],"collections":null,"content":"21.1 Creational Patterns Singleton Pattern Ensures only one instance of a class: class Database: _instance = None _lock = __import__('threading').Lock() def __new__(cls): if cls._instance is None: with cls._lock: if cls._instance is None: cls._instance = super().__new__(cls) cls._instance._initialized = False return cls._instance def __init__(self): if self._initialized: return self.connection = self._create_connection() self._initialized = True def _create_connection(self): print(\"Creating database connection\") return \"connection_object\" # Usage db1 = Database() db2 = Database() print(db1 is db2) # True - same instance Factory Pattern Create objects without specifying exact classes: from abc import ABC, abstractmethod class Cache(ABC): @abstractmethod def get(self, key): pass @abstractmethod def set(self, key, value): pass class RedisCache(Cache): def get(self, key): return f\"Redis get {key}\" def set(self, key, value): print(f\"Redis set {key}={value}\") class MemcachedCache(Cache): def get(self, key): return f\"Memcached get {key}\" def set(self, key, value): print(f\"Memcached set {key}={value}\") class CacheFactory: @staticmethod def create_cache(cache_type): if cache_type == \"redis\": return RedisCache() elif cache_type == \"memcached\": return MemcachedCache() raise ValueError(f\"Unknown cache type: {cache_type}\") # Usage cache = CacheFactory.create_cache(\"redis\") cache.set(\"key\", \"value\") Builder Pattern Construct complex objects step by step: class QueryBuilder: def __init__(self): self.select_fields = [] self.from_table = None self.where_clause = None self.order_by = None self.limit_count = None def select(self, *fields): self.select_fields = list(fields) return self def from_table(self, table): self.from_table = table return self def where(self, condition): self.where_clause = condition return self def order_by(self, field, direction=\"ASC\"): self.order_by = f\"{field} {direction}\" return self def limit(self, count): self.limit_count = count return self def build(self): query = f\"SELECT {','.join(self.select_fields)} FROM {self.from_table}\" if self.where_clause: query += f\" WHERE {self.where_clause}\" if self.order_by: query += f\" ORDER BY {self.order_by}\" if self.limit_count: query += f\" LIMIT {self.limit_count}\" return query # Usage query = (QueryBuilder() .select(\"id\", \"name\", \"email\") .from_table(\"users\") .where(\"age > 18\") .order_by(\"name\") .limit(10) .build()) print(query) ","date":"28-Jul-2025","heading":"21.1 Creational Patterns","objectID":"\/posts\/system-design\/:32:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#211-creational-patterns"},{"categories":["software-engineering"],"collections":null,"content":"21.2 Structural Patterns Adapter Pattern Convert one interface to another: class LegacyPaymentSystem: def make_payment(self, amount): return f\"Legacy system processed ${amount}\" class ModernPaymentInterface: def charge(self, amount): pass class PaymentAdapter(ModernPaymentInterface): def __init__(self, legacy_system): self.legacy = legacy_system def charge(self, amount): return self.legacy.make_payment(amount) # Usage legacy = LegacyPaymentSystem() adapter = PaymentAdapter(legacy) result = adapter.charge(100) print(result) Decorator Pattern Add functionality to objects dynamically: class Coffee: def cost(self): return 5.0 def description(self): return \"Coffee\" class CoffeeDecorator(Coffee): def __init__(self, coffee): self.coffee = coffee class Milk(CoffeeDecorator): def cost(self): return self.coffee.cost() + 1.0 def description(self): return self.coffee.description() + \", Milk\" class Espresso(CoffeeDecorator): def cost(self): return self.coffee.cost() + 2.0 def description(self): return self.coffee.description() + \", Espresso\" # Usage coffee = Coffee() coffee = Milk(coffee) coffee = Espresso(coffee) print(f\"{coffee.description()}: ${coffee.cost()}\") # Coffee, Milk, Espresso: $8.0 Facade Pattern Provide simplified interface to complex subsystem: class VideoEncoder: def encode(self, video): return f\"Encoded {video}\" class AudioProcessor: def process(self, audio): return f\"Processed {audio}\" class Subtitles: def generate(self, video): return f\"Generated subtitles for {video}\" class VideoConversionFacade: def __init__(self): self.encoder = VideoEncoder() self.audio = AudioProcessor() self.subtitles = Subtitles() def convert_video(self, video_file): \"\"\"Single method to convert video with all options\"\"\" encoded = self.encoder.encode(video_file) processed_audio = self.audio.process(video_file) subs = self.subtitles.generate(video_file) return f\"Converted: {encoded}, {processed_audio}, {subs}\" # Usage converter = VideoConversionFacade() result = converter.convert_video(\"movie.mp4\") print(result) ","date":"28-Jul-2025","heading":"21.2 Structural Patterns","objectID":"\/posts\/system-design\/:32:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#212-structural-patterns"},{"categories":["software-engineering"],"collections":null,"content":"21.3 Behavioral Patterns Observer Pattern Objects notify observers of state changes: from abc import ABC, abstractmethod class Observer(ABC): @abstractmethod def update(self, event): pass class EventEmitter: def __init__(self): self.observers = [] def subscribe(self, observer): self.observers.append(observer) def unsubscribe(self, observer): self.observers.remove(observer) def emit(self, event): for observer in self.observers: observer.update(event) class EmailNotifier(Observer): def update(self, event): print(f\"Sending email: {event}\") class LogNotifier(Observer): def update(self, event): print(f\"Logging: {event}\") # Usage emitter = EventEmitter() emitter.subscribe(EmailNotifier()) emitter.subscribe(LogNotifier()) emitter.emit(\"User registered\") Strategy Pattern Select algorithm at runtime: from abc import ABC, abstractmethod class PaymentStrategy(ABC): @abstractmethod def pay(self, amount): pass class CreditCardPayment(PaymentStrategy): def pay(self, amount): return f\"Paid ${amount} with credit card\" class PayPalPayment(PaymentStrategy): def pay(self, amount): return f\"Paid ${amount} with PayPal\" class CryptoCurrency(PaymentStrategy): def pay(self, amount): return f\"Paid ${amount} with cryptocurrency\" class ShoppingCart: def __init__(self, payment_strategy): self.strategy = payment_strategy def checkout(self, amount): return self.strategy.pay(amount) # Usage cart = ShoppingCart(CreditCardPayment()) print(cart.checkout(100)) cart.strategy = PayPalPayment() print(cart.checkout(50)) State Pattern Alter behavior when state changes: from abc import ABC, abstractmethod class State(ABC): @abstractmethod def process(self, order): pass class PendingState(State): def process(self, order): print(\"Order is pending\") order.state = ProcessingState() class ProcessingState(State): def process(self, order): print(\"Order is being processed\") order.state = ShippedState() class ShippedState(State): def process(self, order): print(\"Order has shipped\") order.state = DeliveredState() class DeliveredState(State): def process(self, order): print(\"Order delivered\") class Order: def __init__(self): self.state = PendingState() def process(self): self.state.process(self) # Usage order = Order() order.process() # Pending order.process() # Processing order.process() # Shipped order.process() # Delivered ","date":"28-Jul-2025","heading":"21.3 Behavioral Patterns","objectID":"\/posts\/system-design\/:32:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#213-behavioral-patterns"},{"categories":["software-engineering"],"collections":null,"content":"41. Netflix Architecture ","date":"28-Jul-2025","heading":"41. Netflix Architecture","objectID":"\/posts\/system-design\/:33:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#41-netflix-architecture"},{"categories":["software-engineering"],"collections":null,"content":"41.1 High-Level Overview Netflix serves 280+ million paid subscribers in 190 countries, streaming 100+ million hours of content daily. Their engineering choices are documented extensively on the Netflix Tech Blog. Key Components: Netflix Architecture \u2014 How Netflix built a massive-scale streaming platform using microservices and cloud computing Architecture Principles: 500+ microservices \u2014 each owned by a small team (“you build it, you run it”) Chaos Monkey \u2014 randomly terminates production instances to test resilience Open Connect CDN \u2014 Netflix’s own CDN deployed at ISPs worldwide, serves 99%+ of Netflix traffic Adaptive Bitrate Streaming \u2014 per-title encode optimization for 1000s of quality levels Multi-region active-active \u2014 traffic can shift regions automatically on failure Zuul API Gateway \u2014 handles 100+ billion edge requests per day Database & Storage: EVCache (Memcached + replication) \u2014 distributed caching layer for user\/content data Cassandra \u2014 distributed storage for viewing history and user state AWS S3 \u2014 masters and encoded video assets Hollow \u2014 in-memory read-only dataset for fast catalog data access ","date":"28-Jul-2025","heading":"41.1 High-Level Overview","objectID":"\/posts\/system-design\/:33:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#411-high-level-overview"},{"categories":["software-engineering"],"collections":null,"content":"41.2 Video Streaming Pipeline class NetflixStreamingArchitecture: \"\"\"Simplified Netflix streaming architecture\"\"\" def __init__(self): self.video_qualities = ['240p', '480p', '720p', '1080p', '4K'] self.cdn_locations = [] self.user_bandwidth_data = {} def select_video_quality(self, user_id): \"\"\"Adaptive bitrate streaming\"\"\" bandwidth = self.user_bandwidth_data.get(user_id, 5.0) # Mbps if bandwidth >= 25: return '4K' elif bandwidth >= 5: return '1080p' elif bandwidth >= 3: return '720p' elif bandwidth >= 1.5: return '480p' else: return '240p' def get_nearest_cdn(self, user_location): \"\"\"Find closest CDN server\"\"\" # Geographic routing min_distance = float('inf') nearest_cdn = None for cdn in self.cdn_locations: distance = self.calculate_distance(user_location, cdn.location) if distance < min_distance: min_distance = distance nearest_cdn = cdn return nearest_cdn Key Learnings: Use CDN for global content delivery Adaptive streaming based on network conditions Pre-positioning content close to users Chaos engineering for resilience testing ","date":"28-Jul-2025","heading":"41.2 Video Streaming Pipeline","objectID":"\/posts\/system-design\/:33:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#412-video-streaming-pipeline"},{"categories":["software-engineering"],"collections":null,"content":"42. Uber Tech Stack Uber’s platform handles millions of real-time ride requests globally, requiring sub-second driver matching, live GPS tracking, dynamic pricing, and payment processing at planetary scale. Uber Architecture \u2014 Real-time dispatch, geospatial driver matching, and trip lifecycle management for on-demand transportation Key Architectural Highlights: Dispatch service uses geospatial indexing (Google S2 library) to find nearby drivers in O(log n) Supply\/Demand service dynamically adjusts surge pricing based on driver\/rider ratios per hex cell Schemaless (NoSQL layer on MySQL) stores trip data with flexible schemas Apache Kafka powers the real-time event pipeline \u2014 location updates, trip events, payments Redis stores driver locations with 5-second TTL; lookup by geohash prefix ","date":"28-Jul-2025","heading":"42. Uber Tech Stack","objectID":"\/posts\/system-design\/:34:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#42-uber-tech-stack"},{"categories":["software-engineering"],"collections":null,"content":"42.1 Dispatch System Core of Uber’s platform matching riders with drivers: import heapq from dataclasses import dataclass from typing import Tuple @dataclass class Driver: id: str location: Tuple[float, float] # (lat, lon) available: bool @dataclass class RideRequest: rider_id: str pickup_location: Tuple[float, float] dropoff_location: Tuple[float, float] class UberDispatchSystem: def __init__(self): self.available_drivers = [] # Available drivers self.pending_requests = [] # Ride requests queue def add_driver(self, driver): \"\"\"Add driver to available pool\"\"\" if driver.available: self.available_drivers.append(driver) def request_ride(self, request): \"\"\"Find nearest available driver\"\"\" if not self.available_drivers: return None # Find k nearest drivers nearest_drivers = self._find_nearest_drivers( request.pickup_location, k=5 ) if not nearest_drivers: return None # Assign to nearest available driver driver = nearest_drivers[0] driver.available = False self.available_drivers.remove(driver) return { 'driver_id': driver.id, 'driver_location': driver.location, 'eta': self._calculate_eta(driver.location, request.pickup_location) } def _find_nearest_drivers(self, location, k=5): \"\"\"Find k nearest available drivers\"\"\" distances = [] for driver in self.available_drivers: dist = self._haversine_distance(location, driver.location) distances.append((dist, driver)) # Sort by distance and return top k distances.sort(key=lambda x: x[0]) return [driver for _, driver in distances[:k]] def _haversine_distance(self, loc1, loc2): \"\"\"Calculate distance between two points\"\"\" import math lat1, lon1 = loc1 lat2, lon2 = loc2 R = 6371 # Earth radius in km dlat = math.radians(lat2 - lat1) dlon = math.radians(lon2 - lon1) a = (math.sin(dlat\/2)**2 + math.cos(math.radians(lat1)) * math.cos(math.radians(lat2)) * math.sin(dlon\/2)**2) c = 2 * math.atan2(math.sqrt(a), math.sqrt(1-a)) return R * c def _calculate_eta(self, from_loc, to_loc): \"\"\"Estimate time to arrival\"\"\" distance = self._haversine_distance(from_loc, to_loc) avg_speed = 40 # km\/h in city return distance \/ avg_speed * 60 # minutes Key Technologies: Microservices architecture Cassandra for trip data Redis for real-time driver locations Apache Kafka for event streaming Geospatial indexing for driver matching ","date":"28-Jul-2025","heading":"42.1 Dispatch System","objectID":"\/posts\/system-design\/:34:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#421-dispatch-system"},{"categories":["software-engineering"],"collections":null,"content":"43. How Discord Stores Trillions of Messages Discord serves 19+ million active servers with hundreds of billions of messages. Their storage architecture evolved multiple times as scale grew \u2014 documented in their engineering blog posts: How Discord Stores Billions of Messages and How Discord Stores Trillions of Messages. Discord Message Storage \u2014 How Discord stores trillions of messages using Cassandra\/ScyllaDB with channel-based partitioning Evolution Timeline: 2015 \u2014 MongoDB (single replica set). Hit 100 million stored messages, queries slowing. 2017 \u2014 Migrated to Cassandra. Better write throughput, partition by channel_id, cluster by message_id (Snowflake ID \u2014 time-sorted). 2022 \u2014 Migrated from Cassandra to ScyllaDB. ScyllaDB is a C++ reimplementation of Cassandra with better tail latency and memory management. Key Design Insight: Snowflake IDs as the clustering key means messages are automatically time-ordered within each partition. Pagination (load older messages) is a simple WHERE message_id < $cursor range scan \u2014 extremely efficient. Hot Partition Problem: Popular channels generate far more writes than others. Discord solved this with data service workers that batch and coalesce writes. ","date":"28-Jul-2025","heading":"43. How Discord Stores Trillions of Messages","objectID":"\/posts\/system-design\/:35:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#43-how-discord-stores-trillions-of-messages"},{"categories":["software-engineering"],"collections":null,"content":"43.1 Data Model Evolution Discord evolved from MongoDB to Cassandra to handle scale: # Cassandra schema for messages class DiscordMessageStorage: \"\"\" Cassandra table design: CREATE TABLE messages ( channel_id bigint, message_id bigint, author_id bigint, content text, timestamp timestamp, PRIMARY KEY ((channel_id), message_id) ) WITH CLUSTERING ORDER BY (message_id DESC); \"\"\" def __init__(self): self.message_buckets = {} # Simplified storage def store_message(self, channel_id, message_id, author_id, content): \"\"\"Store message\"\"\" if channel_id not in self.message_buckets: self.message_buckets[channel_id] = [] message = { 'message_id': message_id, 'author_id': author_id, 'content': content, 'timestamp': __import__('time').time() } # Insert at beginning (newest first) self.message_buckets[channel_id].insert(0, message) def get_recent_messages(self, channel_id, limit=50): \"\"\"Get most recent messages\"\"\" if channel_id not in self.message_buckets: return [] return self.message_buckets[channel_id][:limit] def get_messages_before(self, channel_id, before_id, limit=50): \"\"\"Get messages before a message_id (pagination)\"\"\" if channel_id not in self.message_buckets: return [] messages = self.message_buckets[channel_id] # Find messages before the given message_id result = [] for msg in messages: if msg['message_id'] < before_id: result.append(msg) if len(result) >= limit: break return result Key Learnings: Use partition key = channel_id for efficient queries Clustering key = message_id for sorting Hot partition problem solved with proper sharding Cassandra chosen for write-heavy workload ","date":"28-Jul-2025","heading":"43.1 Data Model Evolution","objectID":"\/posts\/system-design\/:35:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#431-data-model-evolution"},{"categories":["software-engineering"],"collections":null,"content":"44. How Kafka Achieves High Throughput Apache Kafka was originally built at LinkedIn to handle their activity stream data, then open-sourced. The official Kafka design docs explain the architectural decisions that enable 2M+ writes\/second per broker. ","date":"28-Jul-2025","heading":"44. How Kafka Achieves High Throughput","objectID":"\/posts\/system-design\/:36:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#44-how-kafka-achieves-high-throughput"},{"categories":["software-engineering"],"collections":null,"content":"44.1 Architecture Principles Kafka Architecture \u2014 High-throughput distributed message streaming for real-time data processing class KafkaOptimizations: \"\"\" Key optimizations that make Kafka fast: \"\"\" def sequential_io(self): \"\"\" 1. Sequential I\/O - Kafka writes to disk sequentially - Much faster than random I\/O - Modern disks: sequential writes = memory writes \"\"\" pass def zero_copy(self): \"\"\" 2. Zero-Copy Technology - Data transferred directly from file to socket - No copying to application memory - Uses sendfile() system call \"\"\" # Traditional: Disk -> Kernel -> User Space -> Kernel -> Socket # Zero-copy: Disk -> Kernel -> Socket pass def batch_compression(self): \"\"\" 3. Batch Compression - Compress multiple messages together - Better compression ratio - Reduces network bandwidth \"\"\" messages = [\"msg1\", \"msg2\", \"msg3\"] compressed_batch = self.compress(messages) return compressed_batch def page_cache(self): \"\"\" 4. Page Cache - Relies on OS page cache - Recently written\/read data cached in memory - No need for application-level cache \"\"\" pass Performance Numbers: 2 million writes\/second per server Hundreds of MB\/s throughput Sub-millisecond latency ","date":"28-Jul-2025","heading":"44.1 Architecture Principles","objectID":"\/posts\/system-design\/:36:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#441-architecture-principles"},{"categories":["software-engineering"],"collections":null,"content":"45. Instagram’s Scaling Journey Instagram launched in 2010, hit 1 million users in 3 months, and was acquired by Facebook in 2012 with just 13 engineers. Its architecture is a textbook case in pragmatic scaling. Key posts: Sharding & IDs at Instagram, What Powers Instagram. Instagram Scaling \u2014 Architecture evolution from startup to 2 billion users: Django, PostgreSQL, Redis, Cassandra, and global CDN Key Architecture Decisions: Python\/Django throughout \u2014 optimized existing code rather than rewriting in a “faster” language (engineering post) PostgreSQL with custom sharding \u2014 12 logical shards, Snowflake-style IDs embedding shard number Redis for follower\/following graph, activity feeds, and session data \u2014 serving ~100M items from memory Cassandra for user relationship data at scale CloudFront CDN for photos \u2014 media delivery accounts for 99%+ of egress bandwidth Celery + RabbitMQ for async tasks (notifications, feed updates) ","date":"28-Jul-2025","heading":"45. Instagram’s Scaling Journey","objectID":"\/posts\/system-design\/:37:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#45-instagrams-scaling-journey"},{"categories":["software-engineering"],"collections":null,"content":"45.1 Feed Ranking Algorithm class InstagramFeedRanking: \"\"\" Instagram's feed ranking considers multiple signals \"\"\" def __init__(self): self.weights = { 'recency': 0.3, 'relationship': 0.25, 'engagement': 0.25, 'content_type': 0.2 } def calculate_score(self, post, user): \"\"\"Calculate post relevance score\"\"\" # Recency score time_decay = self._time_decay_score(post.created_at) # Relationship score relationship_score = self._relationship_strength(user, post.author) # Engagement score engagement_score = self._engagement_probability(post, user) # Content type preference content_score = self._content_type_score(post, user) # Weighted sum total_score = ( self.weights['recency'] * time_decay + self.weights['relationship'] * relationship_score + self.weights['engagement'] * engagement_score + self.weights['content_type'] * content_score ) return total_score def _time_decay_score(self, created_at): \"\"\"Recent posts score higher\"\"\" import time age_hours = (time.time() - created_at) \/ 3600 return 1.0 \/ (1.0 + age_hours) def _relationship_strength(self, user, author): \"\"\"How close is the user to the author\"\"\" # Factors: follows, DMs, likes, comments return 0.8 # Placeholder def _engagement_probability(self, post, user): \"\"\"Likelihood user will engage\"\"\" # ML model predicts like\/comment probability return 0.7 # Placeholder def _content_type_score(self, post, user): \"\"\"User preference for content type\"\"\" # Photo vs video vs carousel preference return 0.6 # Placeholder Scaling Techniques: PostgreSQL with custom sharding Redis for feed caching Cassandra for user relationships memcached for general caching Horizontal scaling of web servers ","date":"28-Jul-2025","heading":"45.1 Feed Ranking Algorithm","objectID":"\/posts\/system-design\/:37:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#451-feed-ranking-algorithm"},{"categories":["software-engineering"],"collections":null,"content":"22. SOLID Principles SOLID principles are five design principles that help make software more maintainable, flexible, and scalable. ","date":"28-Jul-2025","heading":"22. SOLID Principles","objectID":"\/posts\/system-design\/:38:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#22-solid-principles"},{"categories":["software-engineering"],"collections":null,"content":"22.1 Single Responsibility Principle (SRP) Definition: A class should have only one reason to change. Bad Example: # Violates SRP - User class handles too much class User: def __init__(self, name, email): self.name = name self.email = email def save_to_database(self): # Database logic here print(f\"Saving {self.name} to database\") def send_email(self): # Email logic here print(f\"Sending email to {self.email}\") Good Example: # Each class has ONE responsibility class User: def __init__(self, name, email): self.name = name self.email = email class UserRepository: def save(self, user): print(f\"Saving {user.name} to database\") class EmailService: def send(self, user): print(f\"Sending email to {user.email}\") # Usage user = User(\"Alice\", \"alice@example.com\") UserRepository().save(user) EmailService().send(user) Why it matters: Easy to test each class Changes in database don’t affect email logic Clear separation of concerns ","date":"28-Jul-2025","heading":"22.1 Single Responsibility Principle (SRP)","objectID":"\/posts\/system-design\/:38:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#221-single-responsibility-principle-srp"},{"categories":["software-engineering"],"collections":null,"content":"22.2 Open\/Closed Principle (OCP) Definition: Classes should be open for extension, closed for modification. Bad Example: # Violates OCP - must modify for new discounts class Discount: def calculate(self, customer_type, price): if customer_type == \"regular\": return price * 0.1 elif customer_type == \"vip\": return price * 0.2 elif customer_type == \"premium\": # Need to modify! return price * 0.3 Good Example: # Open for extension, closed for modification class Discount: def calculate(self, price): return price * 0.1 class VIPDiscount(Discount): def calculate(self, price): return price * 0.2 class PremiumDiscount(Discount): def calculate(self, price): return price * 0.3 # Usage - no modification needed def apply_discount(discount: Discount, price: float): return price - discount.calculate(price) print(apply_discount(VIPDiscount(), 100)) # 80 print(apply_discount(PremiumDiscount(), 100)) # 70 Why it matters: Add new features without changing existing code Reduces risk of breaking existing functionality Easy to extend with new discount types ","date":"28-Jul-2025","heading":"22.2 Open\/Closed Principle (OCP)","objectID":"\/posts\/system-design\/:38:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#222-openclosed-principle-ocp"},{"categories":["software-engineering"],"collections":null,"content":"22.3 Liskov Substitution Principle (LSP) Definition: Subclasses should be substitutable for their parent classes. Bad Example: # Violates LSP - Square changes Rectangle behavior class Rectangle: def __init__(self, width, height): self.width = width self.height = height def set_width(self, width): self.width = width def set_height(self, height): self.height = height def area(self): return self.width * self.height class Square(Rectangle): def set_width(self, width): self.width = width self.height = width # Changes both! def set_height(self, height): self.width = height # Changes both! # Problem: rect = Rectangle(5, 10) rect.set_width(3) print(rect.area()) # 30 \u2713 square = Square(5, 5) square.set_width(3) print(square.area()) # 9 (unexpected if treated as Rectangle) Good Example: # Correct - use composition instead class Shape: def area(self): raise NotImplementedError class Rectangle(Shape): def __init__(self, width, height): self.width = width self.height = height def area(self): return self.width * self.height class Square(Shape): def __init__(self, side): self.side = side def area(self): return self.side * self.side # Works correctly shapes = [Rectangle(5, 10), Square(5)] for shape in shapes: print(shape.area()) # 50, 25 Why it matters: Prevents unexpected behavior Code works correctly with any subclass Makes inheritance more reliable ","date":"28-Jul-2025","heading":"22.3 Liskov Substitution Principle (LSP)","objectID":"\/posts\/system-design\/:38:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#223-liskov-substitution-principle-lsp"},{"categories":["software-engineering"],"collections":null,"content":"22.4 Interface Segregation Principle (ISP) Definition: Clients shouldn’t depend on methods they don’t use. Bad Example: # Violates ISP - forces unnecessary methods class Worker: def work(self): raise NotImplementedError def eat(self): raise NotImplementedError class Human(Worker): def work(self): print(\"Working\") def eat(self): print(\"Eating lunch\") class Robot(Worker): def work(self): print(\"Working\") def eat(self): pass # Robots don't eat! Forced to implement Good Example: # Split into specific interfaces class Workable: def work(self): raise NotImplementedError class Eatable: def eat(self): raise NotImplementedError class Human(Workable, Eatable): def work(self): print(\"Working\") def eat(self): print(\"Eating lunch\") class Robot(Workable): def work(self): print(\"Working\") # No eat() method needed! Why it matters: Classes only implement what they need More flexible and maintainable Clear interface contracts ","date":"28-Jul-2025","heading":"22.4 Interface Segregation Principle (ISP)","objectID":"\/posts\/system-design\/:38:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#224-interface-segregation-principle-isp"},{"categories":["software-engineering"],"collections":null,"content":"22.5 Dependency Inversion Principle (DIP) Definition: Depend on abstractions, not concrete implementations. Bad Example: # Violates DIP - depends on concrete class class MySQLDatabase: def save(self, data): print(f\"Saving to MySQL: {data}\") class UserService: def __init__(self): self.db = MySQLDatabase() # Tightly coupled! def create_user(self, user): self.db.save(user) # Problem: Can't easily switch to PostgreSQL Good Example: # Depend on abstraction from abc import ABC, abstractmethod class Database(ABC): @abstractmethod def save(self, data): pass class MySQLDatabase(Database): def save(self, data): print(f\"Saving to MySQL: {data}\") class PostgreSQLDatabase(Database): def save(self, data): print(f\"Saving to PostgreSQL: {data}\") class UserService: def __init__(self, database: Database): self.db = database # Depends on abstraction def create_user(self, user): self.db.save(user) # Easy to switch implementations service1 = UserService(MySQLDatabase()) service2 = UserService(PostgreSQLDatabase()) Why it matters: Easy to swap implementations Better for testing (can use mock database) Loose coupling between components ","date":"28-Jul-2025","heading":"22.5 Dependency Inversion Principle (DIP)","objectID":"\/posts\/system-design\/:38:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#225-dependency-inversion-principle-dip"},{"categories":["software-engineering"],"collections":null,"content":"22.6 SOLID Principles Summary Principle Key Idea Benefit SRP One class, one job Easy to maintain OCP Extend, don’t modify Safe to add features LSP Subclass = Parent Reliable inheritance ISP Small, focused interfaces No unused methods DIP Depend on abstractions Flexible architecture ","date":"28-Jul-2025","heading":"22.6 SOLID Principles Summary","objectID":"\/posts\/system-design\/:38:6","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#226-solid-principles-summary"},{"categories":["software-engineering"],"collections":null,"content":"Real-World Application # Complete example using all SOLID principles from abc import ABC, abstractmethod # DIP - Abstract payment interface class PaymentProcessor(ABC): @abstractmethod def process(self, amount): pass # ISP - Separate interfaces class Refundable(ABC): @abstractmethod def refund(self, amount): pass # OCP - Extensible payment types class StripePayment(PaymentProcessor, Refundable): def process(self, amount): return f\"Stripe processed ${amount}\" def refund(self, amount): return f\"Stripe refunded ${amount}\" class PayPalPayment(PaymentProcessor): def process(self, amount): return f\"PayPal processed ${amount}\" # No refund - some payment methods don't support it # SRP - Each class has one responsibility class Order: def __init__(self, items, total): self.items = items self.total = total class OrderProcessor: def __init__(self, payment: PaymentProcessor): self.payment = payment def process_order(self, order: Order): result = self.payment.process(order.total) print(result) return True # LSP - Can substitute any PaymentProcessor processors = [ OrderProcessor(StripePayment()), OrderProcessor(PayPalPayment()) ] order = Order([\"item1\", \"item2\"], 100) for processor in processors: processor.process_order(order) # Works with any processor When to Use SOLID: \u2705 Building large applications \u2705 Code that will be maintained long-term \u2705 Teams working on same codebase \u2705 When you need flexibility When to Skip: \u26a0\ufe0f Small scripts or prototypes \u26a0\ufe0f One-time use code \u26a0\ufe0f Very simple applications Remember: SOLID principles guide good design but aren’t absolute rules. Use judgment based on your project’s needs. ","date":"28-Jul-2025","heading":"Real-World Application","objectID":"\/posts\/system-design\/:38:7","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#real-world-application"},{"categories":["software-engineering"],"collections":null,"content":"Additional Real-World System Designs ","date":"28-Jul-2025","heading":"Additional Real-World System Designs","objectID":"\/posts\/system-design\/:39:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#additional-real-world-system-designs"},{"categories":["software-engineering"],"collections":null,"content":"36. Design Pastebin (Text Sharing Service) Pastebin allows users to share text snippets via short URLs. ","date":"28-Jul-2025","heading":"36. Design Pastebin (Text Sharing Service)","objectID":"\/posts\/system-design\/:40:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#36-design-pastebin-text-sharing-service"},{"categories":["software-engineering"],"collections":null,"content":"36.1 Requirements Functional: Users paste text and get a unique short URL Users access URL to view content Optional expiration time Track view statistics Non-Functional: High availability Low latency for URL generation Handle 10M users, 10M writes\/month, 100M reads\/month 10:1 read-to-write ratio ","date":"28-Jul-2025","heading":"36.1 Requirements","objectID":"\/posts\/system-design\/:40:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#361-requirements"},{"categories":["software-engineering"],"collections":null,"content":"36.2 Capacity Estimation Storage: Average paste size: 1 KB Metadata: ~270 bytes (URL, expiration, timestamps) Total per paste: ~1.27 KB Monthly storage: 1.27 KB \u00d7 10M = 12.7 GB\/month 3-year storage: 12.7 GB \u00d7 36 = 457 GB Traffic: Write QPS: 10M\/month \u00f7 2.5M seconds = 4 writes\/second Read QPS: 100M\/month \u00f7 2.5M seconds = 40 reads\/second Peak: 2\u00d7 average = 8 writes\/sec, 80 reads\/sec ","date":"28-Jul-2025","heading":"36.2 Capacity Estimation","objectID":"\/posts\/system-design\/:40:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#362-capacity-estimation"},{"categories":["software-engineering"],"collections":null,"content":"36.3 High-Level Design Pastebin \/ URL Shortener Architecture \u2014 System design for high-traffic text sharing and link shortening with caching and analytics Core Components: Web Servers: Handle paste creation and retrieval requests. Stateless, horizontally scalable. Short ID Generator: Produces unique 7-character Base62 codes via MD5 hash of content+timestamp, avoiding collisions. Object Store (S3-compatible): Stores raw paste content (text blobs). Separate from metadata for cost efficiency at scale. Metadata Database (SQL): Stores short_id, object_key, expires_at, view_count, created_at. PostgreSQL with index on short_id and expires_at. Cache (Redis): LRU cache keyed by paste:<short_id> storing the content. Dramatically reduces S3 fetches for hot pastes. Expiration Worker: Background job periodically deletes expired paste metadata and S3 objects (avoid orphaned storage). Data Flow: Write path: Client \u2192 Web Server \u2192 generate short_id \u2192 store content in S3 \u2192 write metadata to DB \u2192 cache the content \u2192 return short URL. Read path: Client \u2192 Web Server \u2192 check Redis cache \u2192 on miss: query DB for object_key \u2192 fetch from S3 \u2192 update cache \u2192 return content. Scaling Considerations: 10 million users, 4 writes\/sec, 40 reads\/sec \u2014 a single PostgreSQL instance handles this comfortably. At 10\u00d7 scale, move to read replicas for the metadata DB. S3 provides virtually unlimited object storage at low cost. ","date":"28-Jul-2025","heading":"36.3 High-Level Design","objectID":"\/posts\/system-design\/:40:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#363-high-level-design"},{"categories":["software-engineering"],"collections":null,"content":"36.4 Implementation import hashlib import base64 from datetime import datetime, timedelta class PastebinService: def __init__(self, db, object_store, cache): self.db = db self.object_store = object_store self.cache = cache def create_paste(self, content, expiration_minutes=None): \"\"\"Create a new paste and return short URL\"\"\" # Generate unique ID using MD5 + Base62 unique_str = content + str(datetime.now().timestamp()) hash_value = hashlib.md5(unique_str.encode()).hexdigest() short_id = self._base62_encode(int(hash_value[:16], 16))[:7] # Store content in object store object_key = f\"pastes\/{short_id}\" self.object_store.put(object_key, content) # Store metadata in database paste_metadata = { 'short_id': short_id, 'object_key': object_key, 'created_at': datetime.now(), 'expires_at': datetime.now() + timedelta(minutes=expiration_minutes) if expiration_minutes else None, 'size_bytes': len(content) } self.db.insert('pastes', paste_metadata) return short_id def get_paste(self, short_id): \"\"\"Retrieve paste content\"\"\" # Check cache first cache_key = f\"paste:{short_id}\" cached = self.cache.get(cache_key) if cached: return cached # Get metadata from database metadata = self.db.query(f\"SELECT * FROM pastes WHERE short_id = '{short_id}'\") if not metadata: return None # Check if expired if metadata['expires_at'] and datetime.now() > metadata['expires_at']: return None # Fetch content from object store content = self.object_store.get(metadata['object_key']) # Cache for future requests self.cache.set(cache_key, content, ttl=3600) # Update view count asynchronously self.db.execute(f\"UPDATE pastes SET views = views + 1 WHERE short_id = '{short_id}'\") return content def _base62_encode(self, num): \"\"\"Encode number to Base62\"\"\" chars = \"0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ\" if num == 0: return chars[0] result = [] while num > 0: result.append(chars[num % 62]) num \/\/= 62 return ''.join(reversed(result)) # Usage service = PastebinService(db, s3, redis) url = service.create_paste(\"Hello World!\", expiration_minutes=60) content = service.get_paste(url) ","date":"28-Jul-2025","heading":"36.4 Implementation","objectID":"\/posts\/system-design\/:40:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#364-implementation"},{"categories":["software-engineering"],"collections":null,"content":"37. Design a Web Crawler A web crawler systematically browses the web to index pages for search engines. ","date":"28-Jul-2025","heading":"37. Design a Web Crawler","objectID":"\/posts\/system-design\/:41:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#37-design-a-web-crawler"},{"categories":["software-engineering"],"collections":null,"content":"37.1 Requirements Functional: Crawl 1 billion links Generate reverse index (words \u2192 pages) Extract titles and snippets Respect robots.txt Handle duplicate content Non-Functional: Scalable to billions of pages Fresh content (recrawl weekly) Avoid infinite loops Politeness (rate limiting per domain) ","date":"28-Jul-2025","heading":"37.1 Requirements","objectID":"\/posts\/system-design\/:41:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#371-requirements"},{"categories":["software-engineering"],"collections":null,"content":"37.2 Capacity Estimation 1 billion links to crawl Average page size: 500 KB Crawl frequency: Weekly Monthly crawls: 4 billion pages Monthly storage: 500 KB \u00d7 4B = 2 PB\/month Write QPS: 4B \/ 2.5M seconds = 1,600 writes\/second Search QPS: 100B \/ 2.5M seconds = 40,000 searches\/second ","date":"28-Jul-2025","heading":"37.2 Capacity Estimation","objectID":"\/posts\/system-design\/:41:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#372-capacity-estimation"},{"categories":["software-engineering"],"collections":null,"content":"37.3 Architecture Web Crawler Architecture \u2014 Distributed crawl pipeline: URL frontier, fetchers, parser, dedup filter, and reverse index store Core Components: URL Frontier: Priority queue of URLs to crawl. Implemented as a distributed queue (Kafka or Redis ZSET with priority score). Separates “to crawl” from “crawled” sets. Fetchers (Workers): Stateless worker processes that pull URLs from the frontier, issue HTTP GET, and pass raw HTML to the parser. Scale horizontally. Respect robots.txt and rate-limit per domain. HTML Parser \/ Link Extractor: Extracts <a href> links from fetched pages. Normalizes URLs (strip query params, resolve relative paths). Deduplication Filter: Uses SimHash or MD5 of content to detect near-duplicate pages. Bloom filter for fast URL deduplication (seen-URL check before fetching). Content Store (Object Storage): Raw HTML stored in S3-compatible store for reprocessing. Partition by domain hash for cost-efficient retrieval. Reverse Index Builder: Extracts words, computes TF-IDF scores, and writes to an inverted index (Elasticsearch or custom sharded index). Scheduler: Re-crawls popular pages more frequently (every day), less-popular pages weekly. Uses crawl frequency heuristics based on historical change rate. Design Decisions: Use BFS from seed URLs, with priority weighting by PageRank or domain authority. Respect Crawl-Delay from robots.txt \u2014 critical for politeness. Avoid crawl traps (infinite URL generation) by canonicalizing URLs and limiting depth. For 1 billion links at 1,600 writes\/second: distribute across 10\u201320 fetcher nodes, each handling ~100 QPS. ","date":"28-Jul-2025","heading":"37.3 Architecture","objectID":"\/posts\/system-design\/:41:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#373-architecture"},{"categories":["software-engineering"],"collections":null,"content":"37.4 Implementation import hashlib from collections import deque from urllib.parse import urljoin, urlparse import requests from bs4 import BeautifulSoup class WebCrawler: def __init__(self, seed_urls, max_pages=1000): self.to_crawl = deque(seed_urls) self.crawled = set() self.page_signatures = {} self.max_pages = max_pages self.crawled_count = 0 def crawl(self): \"\"\"Main crawling loop\"\"\" while self.to_crawl and self.crawled_count < self.max_pages: url = self.to_crawl.popleft() if url in self.crawled: continue try: page_data = self.fetch_page(url) if not page_data: continue # Check for duplicates signature = self.create_signature(page_data['content']) if signature in self.page_signatures: print(f\"Duplicate content: {url}\") continue self.page_signatures[signature] = url self.crawled.add(url) self.crawled_count += 1 # Extract and queue new URLs new_urls = self.extract_urls(url, page_data['content']) for new_url in new_urls: if new_url not in self.crawled: self.to_crawl.append(new_url) # Process page (index, store, etc.) self.process_page(url, page_data) print(f\"Crawled ({self.crawled_count}): {url}\") except Exception as e: print(f\"Error crawling {url}: {e}\") def fetch_page(self, url): \"\"\"Fetch page content\"\"\" try: response = requests.get(url, timeout=5) if response.status_code == 200: return { 'url': url, 'content': response.text, 'headers': dict(response.headers) } except: return None def create_signature(self, content): \"\"\"Create page signature for duplicate detection\"\"\" # Normalize and hash content normalized = content.lower().strip() return hashlib.md5(normalized.encode()).hexdigest() def extract_urls(self, base_url, html_content): \"\"\"Extract all URLs from page\"\"\" soup = BeautifulSoup(html_content, 'html.parser') urls = [] for link in soup.find_all('a', href=True): url = urljoin(base_url, link['href']) # Filter only HTTP(S) URLs if urlparse(url).scheme in ['http', 'https']: urls.append(url) return urls def process_page(self, url, page_data): \"\"\"Process crawled page - index, store, etc.\"\"\" # Extract title and snippet soup = BeautifulSoup(page_data['content'], 'html.parser') title = soup.find('title').text if soup.find('title') else url # Create reverse index (word -> URLs) words = self.tokenize(page_data['content']) # Store in reverse index... # Store document # Store in document service... print(f\" Title: {title}\") print(f\" Words: {len(words)}\") def tokenize(self, text): \"\"\"Simple tokenization\"\"\" soup = BeautifulSoup(text, 'html.parser') text = soup.get_text() words = text.lower().split() return [w.strip('.,!?;:()[]{}') for w in words if len(w) > 2] # Usage crawler = WebCrawler(['https:\/\/example.com'], max_pages=100) crawler.crawl() Advanced Features: class PoliteCrawler(WebCrawler): \"\"\"Crawler with politeness - rate limiting per domain\"\"\" def __init__(self, seed_urls, max_pages=1000, delay_seconds=1): super().__init__(seed_urls, max_pages) self.domain_last_crawl = {} self.delay_seconds = delay_seconds def can_crawl_domain(self, url): \"\"\"Check if enough time has passed since last crawl\"\"\" import time domain = urlparse(url).netloc last_crawl = self.domain_last_crawl.get(domain, 0) if time.time() - last_crawl < self.delay_seconds: return False self.domain_last_crawl[domain] = time.time() return True def fetch_page(self, url): \"\"\"Fetch with politeness check\"\"\" if not self.can_crawl_domain(url): return None return super().fetch_page(url) ","date":"28-Jul-2025","heading":"37.4 Implementation","objectID":"\/posts\/system-design\/:41:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#374-implementation"},{"categories":["software-engineering"],"collections":null,"content":"38. Design Mint.com (Personal Finance Tracker) Mint.com aggregates financial data from multiple sources and provides insights. ","date":"28-Jul-2025","heading":"38. Design Mint.com (Personal Finance Tracker)","objectID":"\/posts\/system-design\/:42:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#38-design-mintcom-personal-finance-tracker"},{"categories":["software-engineering"],"collections":null,"content":"38.1 Requirements Functional: Connect to multiple bank accounts Automatically categorize transactions Budget tracking and alerts Financial insights and trends Bill reminders Non-Functional: Highly secure (financial data) Near real-time sync Support millions of users 99.99% uptime ","date":"28-Jul-2025","heading":"38.1 Requirements","objectID":"\/posts\/system-design\/:42:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#381-requirements"},{"categories":["software-engineering"],"collections":null,"content":"38.2 Architecture Mint.com Architecture \u2014 Financial data aggregation pipeline: bank connectors, transaction sync, ML categorization, and budget alerting Core Components: Bank Aggregator Integration: Third-party services like Plaid or Yodlee act as intermediaries that connect to thousands of banks via OAuth \/ credential-based scraping. Mint never stores bank credentials directly. Transaction Sync Service: Scheduled pull (every 4\u20136 hours) from aggregator API. Stores new transactions in PostgreSQL partitioned by user_id and account_id. Transaction Categorization Engine: ML model (Naive Bayes \/ gradient boosting) classifies transactions based on merchant name and description. Falls back to user-defined rules. Budget Alerting Service: Pub\/Sub consumer that aggregates monthly spend per category and compares against user-defined budgets. Emits alert events when thresholds exceeded. Notification Service: Delivers alerts via push notification, email, or SMS (similar architecture to \u00a735). Read API: Serves transaction history, spending trends, and net worth calculations to the frontend. Uses Redis for caching aggregated summaries. Encryption Layer: All bank credentials (if stored at all) are encrypted at rest using AES-256. PCI DSS compliance required. Data Flow: User links bank account \u2192 OAuth flow with Plaid \u2192 Mint receives access token. Sync worker polls Plaid every 6 hours \u2192 new transactions land in DB. Categorization engine processes uncategorized transactions \u2192 writes category back. Budget checker runs after each sync \u2192 publishes alert events if budget exceeded. Notification service delivers alerts to user devices. ","date":"28-Jul-2025","heading":"38.2 Architecture","objectID":"\/posts\/system-design\/:42:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#382-architecture"},{"categories":["software-engineering"],"collections":null,"content":"38.3 Key Components Transaction Categorization (ML-based): from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.naive_bayes import MultinomialNB class TransactionCategorizer: def __init__(self): self.vectorizer = TfidfVectorizer(max_features=1000) self.classifier = MultinomialNB() self.categories = [ 'Groceries', 'Restaurants', 'Transportation', 'Entertainment', 'Bills', 'Shopping', 'Healthcare', 'Travel', 'Income', 'Other' ] def train(self, transactions, labels): \"\"\"Train categorizer on labeled data\"\"\" descriptions = [t['description'] for t in transactions] X = self.vectorizer.fit_transform(descriptions) self.classifier.fit(X, labels) def categorize(self, transaction): \"\"\"Predict category for new transaction\"\"\" description = transaction['description'] X = self.vectorizer.transform([description]) category_idx = self.classifier.predict(X)[0] confidence = max(self.classifier.predict_proba(X)[0]) return { 'category': self.categories[category_idx], 'confidence': confidence } class MintService: def __init__(self): self.categorizer = TransactionCategorizer() self.accounts = {} self.transactions = [] def sync_accounts(self, user_id): \"\"\"Sync transactions from all connected accounts\"\"\" accounts = self.get_user_accounts(user_id) new_transactions = [] for account in accounts: # Fetch from bank API (via Plaid\/Yodlee) recent_transactions = self.fetch_transactions(account) for txn in recent_transactions: # Categorize automatically category_info = self.categorizer.categorize(txn) txn['category'] = category_info['category'] txn['auto_categorized'] = True new_transactions.append(txn) # Store transactions self.store_transactions(user_id, new_transactions) # Check budget alerts self.check_budget_alerts(user_id) return new_transactions def get_user_accounts(self, user_id): \"\"\"Get all linked accounts for user\"\"\" return [] # Query from database def fetch_transactions(self, account): \"\"\"Fetch transactions from bank via aggregator\"\"\" return [] # Call Plaid\/Yodlee API def store_transactions(self, user_id, transactions): \"\"\"Store transactions in database\"\"\" pass def check_budget_alerts(self, user_id): \"\"\"Check if user exceeded budget categories\"\"\" budgets = self.get_user_budgets(user_id) spending = self.calculate_spending(user_id) alerts = [] for category, limit in budgets.items(): if spending.get(category, 0) > limit: alerts.append({ 'type': 'budget_exceeded', 'category': category, 'limit': limit, 'actual': spending[category] }) if alerts: self.send_alerts(user_id, alerts) return alerts def get_user_budgets(self, user_id): \"\"\"Get budget limits for user\"\"\" return { 'Groceries': 500, 'Restaurants': 300, 'Entertainment': 200 } def calculate_spending(self, user_id): \"\"\"Calculate spending by category this month\"\"\" return { 'Groceries': 450, 'Restaurants': 350, # Exceeded! 'Entertainment': 150 } def send_alerts(self, user_id, alerts): \"\"\"Send push notification\/email\"\"\" print(f\"Alerts for user {user_id}: {alerts}\") Security Considerations: Encrypt bank credentials at rest Use OAuth2 for authentication PCI DSS compliance End-to-end encryption Regular security audits ","date":"28-Jul-2025","heading":"38.3 Key Components","objectID":"\/posts\/system-design\/:42:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#383-key-components"},{"categories":["software-engineering"],"collections":null,"content":"39. Design a Social Network Graph Social graphs (friend networks, follower graphs) power features like friend suggestions, mutual connections, six degrees of separation, and social feed generation. Efficient graph traversal at scale (billions of users, trillions of edges) requires specialized storage and query strategies. Social Network Graph Architecture \u2014 Graph DB, adjacency list cache, BFS engine, and friend recommendation service at billion-user scale ","date":"28-Jul-2025","heading":"39. Design a Social Network Graph","objectID":"\/posts\/system-design\/:43:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#39-design-a-social-network-graph"},{"categories":["software-engineering"],"collections":null,"content":"39.1 Requirements Functional: Add\/remove friendships Find friends of user Find mutual friends Friend recommendations Degree of separation Non-Functional: Billions of users Millions of connections per user Fast friend queries (< 100ms) Scale horizontally ","date":"28-Jul-2025","heading":"39.1 Requirements","objectID":"\/posts\/system-design\/:43:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#391-requirements"},{"categories":["software-engineering"],"collections":null,"content":"39.2 Data Structures Adjacency List (Most Common): class SocialGraph: def __init__(self): self.adjacency_list = {} # user_id -> set of friend_ids def add_friendship(self, user1, user2): \"\"\"Add bidirectional friendship\"\"\" if user1 not in self.adjacency_list: self.adjacency_list[user1] = set() if user2 not in self.adjacency_list: self.adjacency_list[user2] = set() self.adjacency_list[user1].add(user2) self.adjacency_list[user2].add(user1) def remove_friendship(self, user1, user2): \"\"\"Remove friendship\"\"\" if user1 in self.adjacency_list: self.adjacency_list[user1].discard(user2) if user2 in self.adjacency_list: self.adjacency_list[user2].discard(user1) def get_friends(self, user_id): \"\"\"Get all friends of user\"\"\" return self.adjacency_list.get(user_id, set()) def get_mutual_friends(self, user1, user2): \"\"\"Find mutual friends between two users\"\"\" friends1 = self.get_friends(user1) friends2 = self.get_friends(user2) return friends1 & friends2 # Set intersection def are_friends(self, user1, user2): \"\"\"Check if two users are friends\"\"\" return user2 in self.adjacency_list.get(user1, set()) def get_friend_count(self, user_id): \"\"\"Get number of friends\"\"\" return len(self.adjacency_list.get(user_id, set())) def friend_recommendations(self, user_id, limit=10): \"\"\"Recommend friends (friends of friends)\"\"\" friends = self.get_friends(user_id) recommendations = {} # Find friends of friends for friend in friends: for friend_of_friend in self.get_friends(friend): if friend_of_friend == user_id: continue if friend_of_friend in friends: continue # Count mutual friends recommendations[friend_of_friend] = \\ recommendations.get(friend_of_friend, 0) + 1 # Sort by number of mutual friends sorted_recs = sorted( recommendations.items(), key=lambda x: x[1], reverse=True ) return [user for user, _ in sorted_recs[:limit]] def degrees_of_separation(self, user1, user2): \"\"\"Find shortest path between two users (BFS)\"\"\" if user1 == user2: return 0 from collections import deque queue = deque([(user1, 0)]) visited = {user1} while queue: current_user, distance = queue.popleft() for friend in self.get_friends(current_user): if friend == user2: return distance + 1 if friend not in visited: visited.add(friend) queue.append((friend, distance + 1)) return -1 # Not connected # Usage graph = SocialGraph() graph.add_friendship(\"alice\", \"bob\") graph.add_friendship(\"bob\", \"charlie\") graph.add_friendship(\"charlie\", \"david\") print(graph.degrees_of_separation(\"alice\", \"david\")) # 3 print(graph.friend_recommendations(\"alice\")) # ['charlie'] Database Schema for Scale: -- Users table CREATE TABLE users ( user_id BIGINT PRIMARY KEY, username VARCHAR(50) UNIQUE, created_at TIMESTAMP ); -- Friendships table (bidirectional) CREATE TABLE friendships ( user_id1 BIGINT, user_id2 BIGINT, created_at TIMESTAMP, PRIMARY KEY (user_id1, user_id2), INDEX idx_user1 (user_id1), INDEX idx_user2 (user_id2) ); -- Denormalized for faster queries CREATE TABLE user_friends_count ( user_id BIGINT PRIMARY KEY, friend_count INT, updated_at TIMESTAMP ); ","date":"28-Jul-2025","heading":"39.2 Data Structures","objectID":"\/posts\/system-design\/:43:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#392-data-structures"},{"categories":["software-engineering"],"collections":null,"content":"40. Design Amazon Sales Rank by Category Track best-selling products per category in real-time. ","date":"28-Jul-2025","heading":"40. Design Amazon Sales Rank by Category","objectID":"\/posts\/system-design\/:44:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#40-design-amazon-sales-rank-by-category"},{"categories":["software-engineering"],"collections":null,"content":"40.1 Requirements Functional: Track sales for millions of products Calculate rank per category Update ranks in near real-time Support historical rankings Non-Functional: Handle millions of sales\/hour Rankings updated within 1 hour Support 1000s of categories Scalable to billions of products ","date":"28-Jul-2025","heading":"40.1 Requirements","objectID":"\/posts\/system-design\/:44:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#401-requirements"},{"categories":["software-engineering"],"collections":null,"content":"40.2 Architecture Data Pipeline \u2014 ETL and stream processing architecture for big data analytics and real-time processing ","date":"28-Jul-2025","heading":"40.2 Architecture","objectID":"\/posts\/system-design\/:44:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#402-architecture"},{"categories":["software-engineering"],"collections":null,"content":"40.3 Implementation from collections import defaultdict import heapq class SalesRankingService: def __init__(self): self.sales_by_category = defaultdict(lambda: defaultdict(int)) # category -> {product_id -> sales_count} self.ranks_by_category = defaultdict(list) # category -> [(sales_count, product_id), ...] def record_sale(self, product_id, category_id, timestamp): \"\"\"Record a product sale\"\"\" self.sales_by_category[category_id][product_id] += 1 def calculate_ranks(self, category_id): \"\"\"Calculate rankings for category\"\"\" sales = self.sales_by_category[category_id] # Sort by sales count (descending) sorted_products = sorted( sales.items(), key=lambda x: x[1], reverse=True ) # Store ranks ranks = [] for rank, (product_id, sales_count) in enumerate(sorted_products, 1): ranks.append({ 'rank': rank, 'product_id': product_id, 'sales_count': sales_count }) return ranks def get_top_n(self, category_id, n=100): \"\"\"Get top N products in category\"\"\" ranks = self.calculate_ranks(category_id) return ranks[:n] def get_product_rank(self, product_id, category_id): \"\"\"Get rank of specific product\"\"\" ranks = self.calculate_ranks(category_id) for item in ranks: if item['product_id'] == product_id: return item['rank'] return None # For scale: Use sliding window aggregation class StreamingSalesRank: \"\"\"Real-time ranking using sliding window\"\"\" def __init__(self, window_hours=24): self.window_hours = window_hours self.hourly_buckets = defaultdict(lambda: defaultdict(lambda: defaultdict(int))) # timestamp_hour -> category -> product -> count def record_sale(self, product_id, category_id, timestamp): \"\"\"Record sale in time bucket\"\"\" import datetime hour_bucket = timestamp.replace(minute=0, second=0, microsecond=0) self.hourly_buckets[hour_bucket][category_id][product_id] += 1 def get_rankings(self, category_id): \"\"\"Get rankings for category using sliding window\"\"\" import datetime now = datetime.datetime.now() start_window = now - datetime.timedelta(hours=self.window_hours) # Aggregate sales in window total_sales = defaultdict(int) for hour_bucket, categories in self.hourly_buckets.items(): if hour_bucket >= start_window: for product_id, count in categories[category_id].items(): total_sales[product_id] += count # Calculate ranks sorted_products = sorted( total_sales.items(), key=lambda x: x[1], reverse=True ) return [(rank, pid, count) for rank, (pid, count) in enumerate(sorted_products, 1)] ","date":"28-Jul-2025","heading":"40.3 Implementation","objectID":"\/posts\/system-design\/:44:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#403-implementation"},{"categories":["software-engineering"],"collections":null,"content":"46. Design a System that Scales to Millions of Users on AWS ","date":"28-Jul-2025","heading":"46. Design a System that Scales to Millions of Users on AWS","objectID":"\/posts\/system-design\/:45:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#46-design-a-system-that-scales-to-millions-of-users-on-aws"},{"categories":["software-engineering"],"collections":null,"content":"Overview Scaling a web application from a single user to millions requires an iterative approach: Benchmark \u2192 Profile \u2192 Address Bottlenecks \u2192 Repeat. This section demonstrates progressive scaling strategies using AWS services, applicable to any cloud platform. ","date":"28-Jul-2025","heading":"Overview","objectID":"\/posts\/system-design\/:45:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#overview"},{"categories":["software-engineering"],"collections":null,"content":"Problem Requirements Use Cases: User makes read\/write requests Service processes requests and stores data Evolution from 1 user to 10 million users Maintain high availability throughout Traffic Assumptions: 10 million users 1 billion writes per month (~400 writes\/sec) 100 billion reads per month (~40,000 reads\/sec) 100:1 read-to-write ratio 1 KB per write 1 TB new content per month 36 TB in 3 years ","date":"28-Jul-2025","heading":"Problem Requirements","objectID":"\/posts\/system-design\/:45:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#problem-requirements"},{"categories":["software-engineering"],"collections":null,"content":"Evolution Stage 1: Single Server (1-100 Users) Architecture: Single EC2 instance running web server MySQL database on same instance Public static IP (Elastic IP) Route 53 for DNS mapping Characteristics: Vertical scaling only No redundancy Simple monitoring (CloudWatch, CPU, memory) Open ports: 80 (HTTP), 443 (HTTPS), 22 (SSH - whitelisted IPs only) # Simple single-server setup class SingleServerApp: def __init__(self): self.db = MySQLDatabase() def handle_request(self, user_id, data): # Process request result = self.process(data) # Store in local database self.db.save(user_id, result) return result ","date":"28-Jul-2025","heading":"Evolution Stage 1: Single Server (1-100 Users)","objectID":"\/posts\/system-design\/:45:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#evolution-stage-1-single-server-1-100-users"},{"categories":["software-engineering"],"collections":null,"content":"Evolution Stage 2: Separate Components (100-1000 Users) Key Changes: Move MySQL to RDS (managed database service) Store static content on S3 (object storage) Implement VPC with public\/private subnets Benefits: Independent scaling of components Reduced load on web server Automatic backups and multi-AZ replication (RDS) Server-side encryption at rest ","date":"28-Jul-2025","heading":"Evolution Stage 2: Separate Components (100-1000 Users)","objectID":"\/posts\/system-design\/:45:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#evolution-stage-2-separate-components-100-1000-users"},{"categories":["software-engineering"],"collections":null,"content":"Evolution Stage 3: Horizontal Scaling (1,000-10,000 Users) Key Changes: Add Elastic Load Balancer (ELB) Multiple web servers across availability zones Master-Slave MySQL replication Separate web and application tiers CloudFront CDN for static content Load Balancer Configuration: class LoadBalancedArchitecture: def __init__(self): self.load_balancer = ELB() self.web_servers = [ WebServer(az='us-east-1a'), WebServer(az='us-east-1b'), WebServer(az='us-east-1c') ] self.app_servers = [ AppServer(type='read_api'), AppServer(type='write_api') ] self.db_master = RDS_Master() self.db_slaves = [RDS_Slave(), RDS_Slave()] def route_request(self, request): # ELB distributes to web servers web_server = self.load_balancer.select_server() # Web server routes to appropriate app server if request.method == 'GET': app_server = self.select_read_server() else: app_server = self.select_write_server() return app_server.handle(request) def select_read_server(self): # Route reads to slave replicas return random.choice(self.app_servers) Architecture: Database Replication \u2014 Master-slave and multi-master replication strategies for availability and performance ","date":"28-Jul-2025","heading":"Evolution Stage 3: Horizontal Scaling (1,000-10,000 Users)","objectID":"\/posts\/system-design\/:45:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#evolution-stage-3-horizontal-scaling-1000-10000-users"},{"categories":["software-engineering"],"collections":null,"content":"Evolution Stage 4: Caching Layer (10,000-100,000 Users) Key Changes: Add ElastiCache (Redis\/Memcached) Cache frequently accessed data from MySQL Store session data in cache (enables stateless web servers) Add more read replicas Why Caching? Memory access: 250 microseconds SSD access: 1,000 microseconds (4x slower) Disk access: 20,000 microseconds (80x slower) class CachedArchitecture: def __init__(self): self.cache = Redis() # ElastiCache self.db_master = RDS_Master() self.db_slaves = [RDS_Slave() for _ in range(3)] def read_data(self, key): # Try cache first cached = self.cache.get(key) if cached: return cached # Cache miss - read from slave slave = random.choice(self.db_slaves) data = slave.query(key) # Update cache self.cache.set(key, data, ttl=3600) return data def write_data(self, key, value): # Write to master self.db_master.write(key, value) # Invalidate cache self.cache.delete(key) def store_session(self, session_id, session_data): # Store session in Redis for stateless servers self.cache.set(f\"session:{session_id}\", session_data, ttl=1800) ","date":"28-Jul-2025","heading":"Evolution Stage 4: Caching Layer (10,000-100,000 Users)","objectID":"\/posts\/system-design\/:45:6","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#evolution-stage-4-caching-layer-10000-100000-users"},{"categories":["software-engineering"],"collections":null,"content":"Evolution Stage 5: Autoscaling (100,000-1M Users) Key Changes: Auto Scaling Groups for web and app servers CloudWatch metrics trigger scaling Chef\/Puppet\/Ansible for configuration management Enhanced monitoring and alerting Autoscaling Configuration: class AutoScalingConfig: def __init__(self): self.web_server_group = AutoScalingGroup( min_instances=2, max_instances=20, desired_capacity=5, availability_zones=['us-east-1a', 'us-east-1b', 'us-east-1c'] ) self.app_server_group = AutoScalingGroup( min_instances=3, max_instances=30, desired_capacity=10 ) def configure_scaling_policies(self): # Scale up when CPU > 70% for 5 minutes scale_up_policy = ScalingPolicy( metric='CPUUtilization', threshold=70, comparison='GreaterThanThreshold', evaluation_periods=2, period=300, action='add', adjustment=2 # Add 2 instances ) # Scale down when CPU < 30% for 10 minutes scale_down_policy = ScalingPolicy( metric='CPUUtilization', threshold=30, comparison='LessThanThreshold', evaluation_periods=2, period=600, action='remove', adjustment=1 # Remove 1 instance ) return [scale_up_policy, scale_down_policy] Monitoring Stack: CloudWatch for metrics CloudTrail for API logging PagerDuty for incident management Sentry for error tracking New Relic for application performance ","date":"28-Jul-2025","heading":"Evolution Stage 5: Autoscaling (100,000-1M Users)","objectID":"\/posts\/system-design\/:45:7","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#evolution-stage-5-autoscaling-100000-1m-users"},{"categories":["software-engineering"],"collections":null,"content":"Evolution Stage 6: Advanced Scaling (1M-10M Users) Key Changes: Database sharding or federation NoSQL databases (DynamoDB) for specific use cases Data warehouse (Redshift) for analytics Asynchronous processing with SQS and Lambda When to Use NoSQL: High write throughput (> 400 writes\/sec) Simple key-value access patterns Denormalized data structures Geographic distribution needs Asynchronous Processing: class AsyncArchitecture: def __init__(self): self.sqs = SQS_Queue('image-processing-queue') self.lambda_function = Lambda('thumbnail-creator') self.s3 = S3_Bucket('user-images') self.dynamodb = DynamoDB_Table('image-metadata') def upload_image(self, user_id, image_data): # 1. Upload original to S3 image_key = f\"images\/{user_id}\/{uuid.uuid4()}.jpg\" self.s3.put_object(image_key, image_data) # 2. Send message to queue for processing message = { 'user_id': user_id, 'image_key': image_key, 'created_at': datetime.now().isoformat() } self.sqs.send_message(message) # 3. Return immediately (non-blocking) return {'image_key': image_key, 'status': 'processing'} def process_image_async(self, message): # Lambda function triggered by SQS image_key = message['image_key'] user_id = message['user_id'] # Download original original = self.s3.get_object(image_key) # Create thumbnail thumbnail = self.create_thumbnail(original) thumbnail_key = image_key.replace('.jpg', '_thumb.jpg') self.s3.put_object(thumbnail_key, thumbnail) # Update metadata self.dynamodb.put_item({ 'user_id': user_id, 'image_key': image_key, 'thumbnail_key': thumbnail_key, 'status': 'completed', 'processed_at': datetime.now().isoformat() }) Database Sharding Strategy: class ShardedDatabase: def __init__(self, num_shards=8): self.shards = [RDS_Instance(f'shard-{i}') for i in range(num_shards)] self.num_shards = num_shards def get_shard(self, user_id): # Hash-based sharding shard_id = hash(user_id) % self.num_shards return self.shards[shard_id] def write_user_data(self, user_id, data): shard = self.get_shard(user_id) shard.write(user_id, data) def read_user_data(self, user_id): shard = self.get_shard(user_id) return shard.read(user_id) ","date":"28-Jul-2025","heading":"Evolution Stage 6: Advanced Scaling (1M-10M Users)","objectID":"\/posts\/system-design\/:45:8","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#evolution-stage-6-advanced-scaling-1m-10m-users"},{"categories":["software-engineering"],"collections":null,"content":"Cost Optimization Strategies Right-sizing Instances: Use T3 instances for variable workloads (burstable CPU) Use C5 for compute-intensive tasks Use R5 for memory-intensive workloads Reserved Instances: 1-year or 3-year commitments for baseline capacity Save up to 75% vs on-demand pricing Spot Instances: Use for fault-tolerant, flexible workloads Save up to 90% vs on-demand class CostOptimizedScaling: def __init__(self): self.baseline_servers = [ EC2_Reserved('c5.large') for _ in range(5) # Reserved instances ] self.burst_servers = [] # Spot instances self.peak_servers = [] # On-demand instances def handle_traffic_spike(self, current_load): if current_load > self.baseline_capacity(): # Try to use spot instances first spot_instance = self.request_spot_instance('c5.large', max_price=0.05) if spot_instance: self.burst_servers.append(spot_instance) else: # Fall back to on-demand on_demand = self.launch_on_demand('c5.large') self.peak_servers.append(on_demand) ","date":"28-Jul-2025","heading":"Cost Optimization Strategies","objectID":"\/posts\/system-design\/:45:9","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#cost-optimization-strategies"},{"categories":["software-engineering"],"collections":null,"content":"Disaster Recovery and Multi-Region Strategies: Backup and Restore (RPO: hours, RTO: hours, cheapest) Pilot Light (RPO: minutes, RTO: 10s of minutes) Warm Standby (RPO: seconds, RTO: minutes) Multi-Region Active-Active (RPO: none, RTO: none, most expensive) class MultiRegionArchitecture: def __init__(self): self.primary_region = Region('us-east-1') self.secondary_region = Region('us-west-2') self.route53 = Route53_HealthCheck() def configure_failover(self): # Health check on primary self.route53.add_health_check( endpoint=self.primary_region.load_balancer, interval=30, failure_threshold=3 ) # Failover routing policy self.route53.add_record( name='api.example.com', type='A', primary=self.primary_region.load_balancer, secondary=self.secondary_region.load_balancer, failover=True ) def sync_data(self): # Cross-region replication for S3 self.primary_region.s3.enable_cross_region_replication( destination=self.secondary_region.s3 ) # Database replication self.primary_region.rds.create_read_replica( target_region='us-west-2' ) ","date":"28-Jul-2025","heading":"Disaster Recovery and Multi-Region","objectID":"\/posts\/system-design\/:45:10","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#disaster-recovery-and-multi-region"},{"categories":["software-engineering"],"collections":null,"content":"Key Takeaways Start Simple: Begin with single server, scale incrementally Monitor Everything: Use metrics to identify real bottlenecks Horizontal Scaling: Easier to scale than vertical Caching is Critical: Dramatically reduces database load Decouple Components: Use queues for asynchronous processing Plan for Failure: Multi-AZ, auto-healing, backups Cost vs Performance: Balance based on business needs Progressive Evolution: 1-100 users: Single server 100-1K users: Separate database, object storage 1K-10K users: Load balancing, horizontal scaling 10K-100K users: Caching layer, more replicas 100K-1M users: Autoscaling, configuration management 1M-10M users: Sharding, NoSQL, async processing, multi-region ","date":"28-Jul-2025","heading":"Key Takeaways","objectID":"\/posts\/system-design\/:45:11","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#key-takeaways"},{"categories":["software-engineering"],"collections":null,"content":"47. Design a Query Cache for Search Engine ","date":"28-Jul-2025","heading":"47. Design a Query Cache for Search Engine","objectID":"\/posts\/system-design\/:46:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#47-design-a-query-cache-for-search-engine"},{"categories":["software-engineering"],"collections":null,"content":"Overview Design a distributed key-value cache to save results of recent web search queries, reducing latency and load on backend search services. This pattern applies to any high-traffic query system (Google, Bing, e-commerce search). ","date":"28-Jul-2025","heading":"Overview","objectID":"\/posts\/system-design\/:46:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#overview-1"},{"categories":["software-engineering"],"collections":null,"content":"Problem Requirements Use Cases: User sends search request \u2192 cache hit (fast response) User sends search request \u2192 cache miss (query backend, cache result) High availability with low latency Traffic Assumptions: 10 million users 10 billion queries per month (~4,000 queries\/sec) Traffic not evenly distributed (popular queries dominate) Limited cache memory (cannot store everything) Capacity Estimation: Each cache entry: Query: 50 bytes Title: 20 bytes Snippet: 200 bytes Total: 270 bytes per entry If all 10 billion queries were unique: 270 bytes \u00d7 10 billion = 2.7 TB\/month Reality: Popular queries repeat, need LRU eviction QPS Calculation: 2.5 million seconds per month 10 billion queries \/ 2.5M seconds = 4,000 QPS ","date":"28-Jul-2025","heading":"Problem Requirements","objectID":"\/posts\/system-design\/:46:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#problem-requirements-1"},{"categories":["software-engineering"],"collections":null,"content":"High-Level Architecture API Gateway Pattern \u2014 Routing, rate limiting, authentication, and protocol translation for scalable APIs ","date":"28-Jul-2025","heading":"High-Level Architecture","objectID":"\/posts\/system-design\/:46:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#high-level-architecture"},{"categories":["software-engineering"],"collections":null,"content":"Component Design 1. Query API Server Responsibilities: Parse and normalize queries Check cache first Route to search backend on miss Update cache with results class QueryAPIServer: def __init__(self, cache, reverse_index_service, document_service): self.cache = cache self.reverse_index = reverse_index_service self.doc_service = document_service def parse_query(self, raw_query): \"\"\"Normalize query for consistent cache keys.\"\"\" # Remove HTML\/markup query = self.strip_html(raw_query) # Fix typos (simple example) query = self.spell_check(query) # Normalize: lowercase, trim whitespace query = query.lower().strip() # Remove stop words for better matching query = self.remove_stop_words(query) # Convert to boolean operations terms = query.split() boolean_query = ' AND '.join(terms) return boolean_query def process_query(self, raw_query): # 1. Parse and normalize query = self.parse_query(raw_query) # 2. Check cache results = self.cache.get(query) if results is not None: print(f\"Cache HIT for query: {query}\") return results print(f\"Cache MISS for query: {query}\") # 3. Query backend services doc_ids = self.reverse_index.search(query) results = self.doc_service.fetch_documents(doc_ids) # 4. Update cache self.cache.set(query, results) return results def strip_html(self, text): # Remove HTML tags import re return re.sub(r'<[^>]+>', '', text) def spell_check(self, query): # Simplified: use edit distance or ML model corrections = { 'gogle': 'google', 'amazn': 'amazon', 'facebok': 'facebook' } return corrections.get(query, query) def remove_stop_words(self, query): stop_words = {'the', 'a', 'an', 'and', 'or', 'but', 'in', 'on', 'at'} words = query.split() filtered = [w for w in words if w not in stop_words] return ' '.join(filtered) 2. LRU Cache Implementation Use doubly-linked list + hash map for O(1) operations: class Node: def __init__(self, query, results): self.query = query self.results = results self.prev = None self.next = None class DoublyLinkedList: def __init__(self): self.head = None self.tail = None def move_to_front(self, node): \"\"\"Move existing node to front (most recently used).\"\"\" if node == self.head: return # Already at front # Remove from current position if node.prev: node.prev.next = node.next if node.next: node.next.prev = node.prev if node == self.tail: self.tail = node.prev # Move to front node.prev = None node.next = self.head if self.head: self.head.prev = node self.head = node if self.tail is None: self.tail = node def append_to_front(self, node): \"\"\"Add new node to front.\"\"\" node.next = self.head node.prev = None if self.head: self.head.prev = node self.head = node if self.tail is None: self.tail = node def remove_from_tail(self): \"\"\"Remove least recently used node from tail.\"\"\" if self.tail is None: return None removed = self.tail if self.tail.prev: self.tail.prev.next = None self.tail = self.tail.prev else: # Only one node self.head = None self.tail = None return removed class LRUCache: def __init__(self, max_size): self.max_size = max_size self.size = 0 self.lookup = {} # query -> node self.linked_list = DoublyLinkedList() def get(self, query): \"\"\"Get cached results. O(1) time complexity.\"\"\" node = self.lookup.get(query) if node is None: return None # Move to front (mark as recently used) self.linked_list.move_to_front(node) return node.results def set(self, query, results): \"\"\"Cache query results. O(1) time complexity.\"\"\" node = self.lookup.get(query) if node is not None: # Update existing entry node.results = results self.linked_list.move_to_front(node) else: # New entry if self.size >= self.max_size: # Evict least recently used evicted = self.linked_list.remove_from_tail() if evicted: del self.lookup[evicted.query] self.size -= 1 # Add new node new_node = Node(query, results) self.linked_list.append_to_front(new_node) self.lookup[query] = new_node self.size += 1 def clear(self): \"\"\"Clear entire cache.\"\"\" self.lookup.clear() self.linked_list = DoublyLinkedList() self.size = 0 # Example usage cache = LRUCache(m","date":"28-Jul-2025","heading":"Component Design","objectID":"\/posts\/system-design\/:46:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#component-design"},{"categories":["software-engineering"],"collections":null,"content":"Component Design 1. Query API Server Responsibilities: Parse and normalize queries Check cache first Route to search backend on miss Update cache with results class QueryAPIServer: def __init__(self, cache, reverse_index_service, document_service): self.cache = cache self.reverse_index = reverse_index_service self.doc_service = document_service def parse_query(self, raw_query): \"\"\"Normalize query for consistent cache keys.\"\"\" # Remove HTML\/markup query = self.strip_html(raw_query) # Fix typos (simple example) query = self.spell_check(query) # Normalize: lowercase, trim whitespace query = query.lower().strip() # Remove stop words for better matching query = self.remove_stop_words(query) # Convert to boolean operations terms = query.split() boolean_query = ' AND '.join(terms) return boolean_query def process_query(self, raw_query): # 1. Parse and normalize query = self.parse_query(raw_query) # 2. Check cache results = self.cache.get(query) if results is not None: print(f\"Cache HIT for query: {query}\") return results print(f\"Cache MISS for query: {query}\") # 3. Query backend services doc_ids = self.reverse_index.search(query) results = self.doc_service.fetch_documents(doc_ids) # 4. Update cache self.cache.set(query, results) return results def strip_html(self, text): # Remove HTML tags import re return re.sub(r'<[^>]+>', '', text) def spell_check(self, query): # Simplified: use edit distance or ML model corrections = { 'gogle': 'google', 'amazn': 'amazon', 'facebok': 'facebook' } return corrections.get(query, query) def remove_stop_words(self, query): stop_words = {'the', 'a', 'an', 'and', 'or', 'but', 'in', 'on', 'at'} words = query.split() filtered = [w for w in words if w not in stop_words] return ' '.join(filtered) 2. LRU Cache Implementation Use doubly-linked list + hash map for O(1) operations: class Node: def __init__(self, query, results): self.query = query self.results = results self.prev = None self.next = None class DoublyLinkedList: def __init__(self): self.head = None self.tail = None def move_to_front(self, node): \"\"\"Move existing node to front (most recently used).\"\"\" if node == self.head: return # Already at front # Remove from current position if node.prev: node.prev.next = node.next if node.next: node.next.prev = node.prev if node == self.tail: self.tail = node.prev # Move to front node.prev = None node.next = self.head if self.head: self.head.prev = node self.head = node if self.tail is None: self.tail = node def append_to_front(self, node): \"\"\"Add new node to front.\"\"\" node.next = self.head node.prev = None if self.head: self.head.prev = node self.head = node if self.tail is None: self.tail = node def remove_from_tail(self): \"\"\"Remove least recently used node from tail.\"\"\" if self.tail is None: return None removed = self.tail if self.tail.prev: self.tail.prev.next = None self.tail = self.tail.prev else: # Only one node self.head = None self.tail = None return removed class LRUCache: def __init__(self, max_size): self.max_size = max_size self.size = 0 self.lookup = {} # query -> node self.linked_list = DoublyLinkedList() def get(self, query): \"\"\"Get cached results. O(1) time complexity.\"\"\" node = self.lookup.get(query) if node is None: return None # Move to front (mark as recently used) self.linked_list.move_to_front(node) return node.results def set(self, query, results): \"\"\"Cache query results. O(1) time complexity.\"\"\" node = self.lookup.get(query) if node is not None: # Update existing entry node.results = results self.linked_list.move_to_front(node) else: # New entry if self.size >= self.max_size: # Evict least recently used evicted = self.linked_list.remove_from_tail() if evicted: del self.lookup[evicted.query] self.size -= 1 # Add new node new_node = Node(query, results) self.linked_list.append_to_front(new_node) self.lookup[query] = new_node self.size += 1 def clear(self): \"\"\"Clear entire cache.\"\"\" self.lookup.clear() self.linked_list = DoublyLinkedList() self.size = 0 # Example usage cache = LRUCache(m","date":"28-Jul-2025","heading":"1. Query API Server","objectID":"\/posts\/system-design\/:46:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#1-query-api-server"},{"categories":["software-engineering"],"collections":null,"content":"Component Design 1. Query API Server Responsibilities: Parse and normalize queries Check cache first Route to search backend on miss Update cache with results class QueryAPIServer: def __init__(self, cache, reverse_index_service, document_service): self.cache = cache self.reverse_index = reverse_index_service self.doc_service = document_service def parse_query(self, raw_query): \"\"\"Normalize query for consistent cache keys.\"\"\" # Remove HTML\/markup query = self.strip_html(raw_query) # Fix typos (simple example) query = self.spell_check(query) # Normalize: lowercase, trim whitespace query = query.lower().strip() # Remove stop words for better matching query = self.remove_stop_words(query) # Convert to boolean operations terms = query.split() boolean_query = ' AND '.join(terms) return boolean_query def process_query(self, raw_query): # 1. Parse and normalize query = self.parse_query(raw_query) # 2. Check cache results = self.cache.get(query) if results is not None: print(f\"Cache HIT for query: {query}\") return results print(f\"Cache MISS for query: {query}\") # 3. Query backend services doc_ids = self.reverse_index.search(query) results = self.doc_service.fetch_documents(doc_ids) # 4. Update cache self.cache.set(query, results) return results def strip_html(self, text): # Remove HTML tags import re return re.sub(r'<[^>]+>', '', text) def spell_check(self, query): # Simplified: use edit distance or ML model corrections = { 'gogle': 'google', 'amazn': 'amazon', 'facebok': 'facebook' } return corrections.get(query, query) def remove_stop_words(self, query): stop_words = {'the', 'a', 'an', 'and', 'or', 'but', 'in', 'on', 'at'} words = query.split() filtered = [w for w in words if w not in stop_words] return ' '.join(filtered) 2. LRU Cache Implementation Use doubly-linked list + hash map for O(1) operations: class Node: def __init__(self, query, results): self.query = query self.results = results self.prev = None self.next = None class DoublyLinkedList: def __init__(self): self.head = None self.tail = None def move_to_front(self, node): \"\"\"Move existing node to front (most recently used).\"\"\" if node == self.head: return # Already at front # Remove from current position if node.prev: node.prev.next = node.next if node.next: node.next.prev = node.prev if node == self.tail: self.tail = node.prev # Move to front node.prev = None node.next = self.head if self.head: self.head.prev = node self.head = node if self.tail is None: self.tail = node def append_to_front(self, node): \"\"\"Add new node to front.\"\"\" node.next = self.head node.prev = None if self.head: self.head.prev = node self.head = node if self.tail is None: self.tail = node def remove_from_tail(self): \"\"\"Remove least recently used node from tail.\"\"\" if self.tail is None: return None removed = self.tail if self.tail.prev: self.tail.prev.next = None self.tail = self.tail.prev else: # Only one node self.head = None self.tail = None return removed class LRUCache: def __init__(self, max_size): self.max_size = max_size self.size = 0 self.lookup = {} # query -> node self.linked_list = DoublyLinkedList() def get(self, query): \"\"\"Get cached results. O(1) time complexity.\"\"\" node = self.lookup.get(query) if node is None: return None # Move to front (mark as recently used) self.linked_list.move_to_front(node) return node.results def set(self, query, results): \"\"\"Cache query results. O(1) time complexity.\"\"\" node = self.lookup.get(query) if node is not None: # Update existing entry node.results = results self.linked_list.move_to_front(node) else: # New entry if self.size >= self.max_size: # Evict least recently used evicted = self.linked_list.remove_from_tail() if evicted: del self.lookup[evicted.query] self.size -= 1 # Add new node new_node = Node(query, results) self.linked_list.append_to_front(new_node) self.lookup[query] = new_node self.size += 1 def clear(self): \"\"\"Clear entire cache.\"\"\" self.lookup.clear() self.linked_list = DoublyLinkedList() self.size = 0 # Example usage cache = LRUCache(m","date":"28-Jul-2025","heading":"2. LRU Cache Implementation","objectID":"\/posts\/system-design\/:46:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#2-lru-cache-implementation"},{"categories":["software-engineering"],"collections":null,"content":"Component Design 1. Query API Server Responsibilities: Parse and normalize queries Check cache first Route to search backend on miss Update cache with results class QueryAPIServer: def __init__(self, cache, reverse_index_service, document_service): self.cache = cache self.reverse_index = reverse_index_service self.doc_service = document_service def parse_query(self, raw_query): \"\"\"Normalize query for consistent cache keys.\"\"\" # Remove HTML\/markup query = self.strip_html(raw_query) # Fix typos (simple example) query = self.spell_check(query) # Normalize: lowercase, trim whitespace query = query.lower().strip() # Remove stop words for better matching query = self.remove_stop_words(query) # Convert to boolean operations terms = query.split() boolean_query = ' AND '.join(terms) return boolean_query def process_query(self, raw_query): # 1. Parse and normalize query = self.parse_query(raw_query) # 2. Check cache results = self.cache.get(query) if results is not None: print(f\"Cache HIT for query: {query}\") return results print(f\"Cache MISS for query: {query}\") # 3. Query backend services doc_ids = self.reverse_index.search(query) results = self.doc_service.fetch_documents(doc_ids) # 4. Update cache self.cache.set(query, results) return results def strip_html(self, text): # Remove HTML tags import re return re.sub(r'<[^>]+>', '', text) def spell_check(self, query): # Simplified: use edit distance or ML model corrections = { 'gogle': 'google', 'amazn': 'amazon', 'facebok': 'facebook' } return corrections.get(query, query) def remove_stop_words(self, query): stop_words = {'the', 'a', 'an', 'and', 'or', 'but', 'in', 'on', 'at'} words = query.split() filtered = [w for w in words if w not in stop_words] return ' '.join(filtered) 2. LRU Cache Implementation Use doubly-linked list + hash map for O(1) operations: class Node: def __init__(self, query, results): self.query = query self.results = results self.prev = None self.next = None class DoublyLinkedList: def __init__(self): self.head = None self.tail = None def move_to_front(self, node): \"\"\"Move existing node to front (most recently used).\"\"\" if node == self.head: return # Already at front # Remove from current position if node.prev: node.prev.next = node.next if node.next: node.next.prev = node.prev if node == self.tail: self.tail = node.prev # Move to front node.prev = None node.next = self.head if self.head: self.head.prev = node self.head = node if self.tail is None: self.tail = node def append_to_front(self, node): \"\"\"Add new node to front.\"\"\" node.next = self.head node.prev = None if self.head: self.head.prev = node self.head = node if self.tail is None: self.tail = node def remove_from_tail(self): \"\"\"Remove least recently used node from tail.\"\"\" if self.tail is None: return None removed = self.tail if self.tail.prev: self.tail.prev.next = None self.tail = self.tail.prev else: # Only one node self.head = None self.tail = None return removed class LRUCache: def __init__(self, max_size): self.max_size = max_size self.size = 0 self.lookup = {} # query -> node self.linked_list = DoublyLinkedList() def get(self, query): \"\"\"Get cached results. O(1) time complexity.\"\"\" node = self.lookup.get(query) if node is None: return None # Move to front (mark as recently used) self.linked_list.move_to_front(node) return node.results def set(self, query, results): \"\"\"Cache query results. O(1) time complexity.\"\"\" node = self.lookup.get(query) if node is not None: # Update existing entry node.results = results self.linked_list.move_to_front(node) else: # New entry if self.size >= self.max_size: # Evict least recently used evicted = self.linked_list.remove_from_tail() if evicted: del self.lookup[evicted.query] self.size -= 1 # Add new node new_node = Node(query, results) self.linked_list.append_to_front(new_node) self.lookup[query] = new_node self.size += 1 def clear(self): \"\"\"Clear entire cache.\"\"\" self.lookup.clear() self.linked_list = DoublyLinkedList() self.size = 0 # Example usage cache = LRUCache(m","date":"28-Jul-2025","heading":"3. Cache Update Strategies","objectID":"\/posts\/system-design\/:46:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#3-cache-update-strategies"},{"categories":["software-engineering"],"collections":null,"content":"Distributed Caching at Scale For 4,000 QPS and millions of entries, single-server cache won’t work. Option 1: Replicate Cache on Every Server Simple but inefficient memory usage Low cache hit rate (each server has different subset) Option 2: Dedicated Cache Servers (Recommended) Use consistent hashing to shard cache Multiple cache servers with replication import hashlib import bisect class ConsistentHashRing: def __init__(self, num_virtual_nodes=150): self.num_virtual_nodes = num_virtual_nodes self.ring = {} # hash_value -> server_id self.sorted_hashes = [] def add_server(self, server_id): for i in range(self.num_virtual_nodes): virtual_key = f\"{server_id}:{i}\" hash_value = int(hashlib.md5(virtual_key.encode()).hexdigest(), 16) self.ring[hash_value] = server_id bisect.insort(self.sorted_hashes, hash_value) def remove_server(self, server_id): for i in range(self.num_virtual_nodes): virtual_key = f\"{server_id}:{i}\" hash_value = int(hashlib.md5(virtual_key.encode()).hexdigest(), 16) del self.ring[hash_value] self.sorted_hashes.remove(hash_value) def get_server(self, query): query_hash = int(hashlib.md5(query.encode()).hexdigest(), 16) idx = bisect.bisect_right(self.sorted_hashes, query_hash) if idx == len(self.sorted_hashes): idx = 0 server_hash = self.sorted_hashes[idx] return self.ring[server_hash] class DistributedCache: def __init__(self, cache_servers): self.hash_ring = ConsistentHashRing() self.cache_servers = {} # server_id -> LRUCache instance for server_id in cache_servers: self.add_cache_server(server_id) def add_cache_server(self, server_id): self.hash_ring.add_server(server_id) self.cache_servers[server_id] = LRUCache(max_size=100000) def get(self, query): server_id = self.hash_ring.get_server(query) cache = self.cache_servers[server_id] return cache.get(query) def set(self, query, results): server_id = self.hash_ring.get_server(query) cache = self.cache_servers[server_id] cache.set(query, results) # Example: 5 cache servers distributed_cache = DistributedCache(['cache-1', 'cache-2', 'cache-3', 'cache-4', 'cache-5']) # Queries are automatically sharded distributed_cache.set(\"python tutorials\", [\"Result 1\", \"Result 2\"]) distributed_cache.set(\"java tutorials\", [\"Result 3\", \"Result 4\"]) results = distributed_cache.get(\"python tutorials\") ","date":"28-Jul-2025","heading":"Distributed Caching at Scale","objectID":"\/posts\/system-design\/:46:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#distributed-caching-at-scale"},{"categories":["software-engineering"],"collections":null,"content":"Monitoring and Metrics Monitoring Stack \u2014 Complete observability solution: logs, metrics, traces, and alerting for production systems Key Metrics: Cache Hit Rate: (Hits \/ Total Requests) \u00d7 100% Cache Miss Latency: Time to fetch from backend Memory Usage: Prevent OOM Eviction Rate: How often entries are removed class CacheMetrics: def __init__(self): self.hits = 0 self.misses = 0 self.total_latency = 0.0 self.evictions = 0 def record_hit(self): self.hits += 1 def record_miss(self, latency): self.misses += 1 self.total_latency += latency def record_eviction(self): self.evictions += 1 def get_stats(self): total_requests = self.hits + self.misses hit_rate = (self.hits \/ total_requests * 100) if total_requests > 0 else 0 avg_miss_latency = (self.total_latency \/ self.misses) if self.misses > 0 else 0 return { 'hit_rate': hit_rate, 'total_requests': total_requests, 'hits': self.hits, 'misses': self.misses, 'avg_miss_latency_ms': avg_miss_latency * 1000, 'evictions': self.evictions } ","date":"28-Jul-2025","heading":"Monitoring and Metrics","objectID":"\/posts\/system-design\/:46:6","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#monitoring-and-metrics"},{"categories":["software-engineering"],"collections":null,"content":"Complete System Integration class SearchEngineQueryCache: def __init__(self): self.distributed_cache = DistributedCache( ['cache-1', 'cache-2', 'cache-3', 'cache-4', 'cache-5'] ) self.metrics = CacheMetrics() self.reverse_index = ReverseIndexService() self.doc_service = DocumentService() def search(self, raw_query): # Parse query query = self.parse_query(raw_query) # Check distributed cache start_time = time.time() results = self.distributed_cache.get(query) if results is not None: # Cache hit self.metrics.record_hit() return results # Cache miss - query backend doc_ids = self.reverse_index.search(query) results = self.doc_service.fetch_documents(doc_ids) # Record metrics latency = time.time() - start_time self.metrics.record_miss(latency) # Update cache self.distributed_cache.set(query, results) return results def parse_query(self, raw_query): # Same as QueryAPIServer implementation return raw_query.lower().strip() ","date":"28-Jul-2025","heading":"Complete System Integration","objectID":"\/posts\/system-design\/:46:7","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#complete-system-integration"},{"categories":["software-engineering"],"collections":null,"content":"Trade-offs and Considerations Memory vs Latency: Larger cache = higher hit rate but more memory cost Optimal size depends on query distribution (Zipf’s law) TTL Selection: Short TTL (minutes): Fresh data, but more backend load Long TTL (hours): Less backend load, but stale data risk Adaptive TTL based on query popularity Consistency: Cache-aside pattern: Simple but can serve stale data Write-through: Always consistent but slower writes Eventual consistency acceptable for search (not financial data) Replication: Single cache node: Fast but single point of failure Replicated cache: High availability but consistency challenges ","date":"28-Jul-2025","heading":"Trade-offs and Considerations","objectID":"\/posts\/system-design\/:46:8","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#trade-offs-and-considerations"},{"categories":["software-engineering"],"collections":null,"content":"Key Takeaways LRU eviction balances memory usage and hit rate Consistent hashing enables horizontal scaling Cache-aside pattern works well for read-heavy workloads TTL prevents stale data without manual invalidation Monitoring hit rate guides cache size tuning Distributed caching required for high QPS (4,000+) Performance Impact: Memory access: 0.25 ms Cache miss + SSD: 1 ms Cache miss + HDD: 20 ms 80x speedup vs disk for popular queries! ","date":"28-Jul-2025","heading":"Key Takeaways","objectID":"\/posts\/system-design\/:46:9","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#key-takeaways-1"},{"categories":["software-engineering"],"collections":null,"content":"48. Design an E-Commerce Website (Amazon-like) ","date":"28-Jul-2025","heading":"48. Design an E-Commerce Website (Amazon-like)","objectID":"\/posts\/system-design\/:47:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#48-design-an-e-commerce-website-amazon-like"},{"categories":["software-engineering"],"collections":null,"content":"Overview Design a scalable e-commerce platform handling millions of products, orders, and users with features like product catalog, cart, checkout, inventory management, and order processing. ","date":"28-Jul-2025","heading":"Overview","objectID":"\/posts\/system-design\/:47:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#overview-2"},{"categories":["software-engineering"],"collections":null,"content":"Problem Requirements Functional Requirements: User registration\/authentication Product browsing and search Shopping cart management Checkout and payment processing Order tracking Inventory management Seller portal (for marketplace model) Reviews and ratings Non-Functional Requirements: 100 million active users 50 million products 1 million orders per day (~12 orders\/sec) 99.99% availability Sub-second search latency ACID transactions for payments PCI DSS compliance Capacity Estimation: Storage: 50M products \u00d7 10KB metadata = 500 GB Images: 50M products \u00d7 5 images \u00d7 500KB = 125 TB Orders: 1M\/day \u00d7 365 \u00d7 5 years \u00d7 5KB = 9 TB Peak traffic: 12 orders\/sec \u00d7 10 (peak multiplier) = 120 orders\/sec ","date":"28-Jul-2025","heading":"Problem Requirements","objectID":"\/posts\/system-design\/:47:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#problem-requirements-2"},{"categories":["software-engineering"],"collections":null,"content":"High-Level Architecture E-Commerce Architecture \u2014 Amazon-style microservices: product, cart, order, inventory, payment, and notification services behind an API gateway Core Services: Service Responsibility Database Product Service Catalog, search, recommendations PostgreSQL + Elasticsearch Inventory Service Stock levels, reservations PostgreSQL (ACID) Cart Service Shopping cart state Redis (fast, ephemeral) Order Service Order lifecycle, checkout PostgreSQL (ACID transactions) Payment Service Payment processing, refunds PostgreSQL + Payment gateway (Stripe\/Braintree) User Service Auth, profiles, addresses PostgreSQL Notification Service Email\/SMS\/Push confirmations Kafka + SendGrid\/Twilio Search Service Full-text product search, filtering Elasticsearch Traffic Flow: User requests hit the API Gateway (authentication, rate limiting, routing). Product browsing \u2192 Product Service \u2192 PostgreSQL (cached in Redis). Add to cart \u2192 Cart Service \u2192 Redis (7-day TTL). Checkout \u2192 Order Service \u2192 reserves inventory (Inventory Service) \u2192 processes payment \u2192 publishes order_confirmed to Kafka. Downstream consumers listen on Kafka: Inventory Service (deduct stock), Notification Service (send confirmation), Analytics Service. Key Design Decisions: Cart in Redis: ephemeral, fast, lost on cache flush but acceptable (users can rebuild cart). Inventory reservation uses DB-level row locking (SELECT ... FOR UPDATE) to prevent overselling. Payment service integrates with Stripe or Braintree \u2014 never store raw card data (PCI compliance). Order service uses Saga pattern for distributed transaction: if payment fails, compensation transactions release inventory. ","date":"28-Jul-2025","heading":"High-Level Architecture","objectID":"\/posts\/system-design\/:47:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#high-level-architecture-1"},{"categories":["software-engineering"],"collections":null,"content":"Database Schema # Product Schema class Product: id: UUID (PK) seller_id: UUID (FK) name: str description: text category_id: UUID (FK) price: decimal(10, 2) currency: str brand: str created_at: timestamp updated_at: timestamp is_active: boolean class ProductImage: id: UUID (PK) product_id: UUID (FK) image_url: str is_primary: boolean display_order: int class Inventory: id: UUID (PK) product_id: UUID (FK) warehouse_id: UUID (FK) quantity: int reserved_quantity: int # For pending orders updated_at: timestamp # Order Schema class Order: id: UUID (PK) user_id: UUID (FK) status: enum('pending', 'confirmed', 'shipped', 'delivered', 'cancelled') subtotal: decimal(10, 2) tax: decimal(10, 2) shipping: decimal(10, 2) total: decimal(10, 2) shipping_address_id: UUID (FK) billing_address_id: UUID (FK) created_at: timestamp updated_at: timestamp class OrderItem: id: UUID (PK) order_id: UUID (FK) product_id: UUID (FK) quantity: int unit_price: decimal(10, 2) total_price: decimal(10, 2) class Payment: id: UUID (PK) order_id: UUID (FK) payment_method: enum('credit_card', 'debit_card', 'paypal', 'crypto') amount: decimal(10, 2) status: enum('pending', 'completed', 'failed', 'refunded') transaction_id: str # From payment gateway created_at: timestamp ","date":"28-Jul-2025","heading":"Database Schema","objectID":"\/posts\/system-design\/:47:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#database-schema"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Product Service from flask import Flask, jsonify, request import redis import psycopg2 class ProductService: def __init__(self): self.db = psycopg2.connect(\"dbname=products user=admin\") self.cache = redis.Redis(host='localhost', port=6379) def get_product(self, product_id): # Try cache first cache_key = f\"product:{product_id}\" cached = self.cache.get(cache_key) if cached: return json.loads(cached) # Cache miss - query database cursor = self.db.cursor() cursor.execute(\"\"\" SELECT p.*, array_agg(pi.image_url) as images FROM products p LEFT JOIN product_images pi ON p.id = pi.product_id WHERE p.id = %s AND p.is_active = true GROUP BY p.id \"\"\", (product_id,)) product = cursor.fetchone() if product: product_data = self.serialize_product(product) # Cache for 1 hour self.cache.setex(cache_key, 3600, json.dumps(product_data)) return product_data return None def search_products(self, query, filters, page=1, page_size=20): # Delegate to Elasticsearch for full-text search es_query = { \"query\": { \"bool\": { \"must\": { \"multi_match\": { \"query\": query, \"fields\": [\"name^3\", \"description\", \"brand^2\"] } }, \"filter\": self.build_filters(filters) } }, \"from\": (page - 1) * page_size, \"size\": page_size, \"sort\": [{\"_score\": \"desc\"}, {\"created_at\": \"desc\"}] } results = self.elasticsearch.search(index=\"products\", body=es_query) return results['hits']['hits'] def build_filters(self, filters): filter_clauses = [] if filters.get('category'): filter_clauses.append({\"term\": {\"category_id\": filters['category']}}) if filters.get('price_range'): filter_clauses.append({ \"range\": { \"price\": { \"gte\": filters['price_range']['min'], \"lte\": filters['price_range']['max'] } } }) if filters.get('brand'): filter_clauses.append({\"terms\": {\"brand\": filters['brand']}}) return filter_clauses 2. Cart Service (Redis-based) import json from datetime import timedelta class CartService: def __init__(self): self.redis = redis.Redis(host='localhost', port=6379) def add_to_cart(self, user_id, product_id, quantity): cart_key = f\"cart:{user_id}\" # Get current cart cart = self.get_cart(user_id) # Add\/update item if product_id in cart: cart[product_id]['quantity'] += quantity else: # Fetch product details product = self.product_service.get_product(product_id) cart[product_id] = { 'product_id': product_id, 'name': product['name'], 'price': product['price'], 'quantity': quantity, 'image': product['images'][0] if product['images'] else None } # Save cart (expires in 7 days) self.redis.setex(cart_key, timedelta(days=7), json.dumps(cart)) return cart def get_cart(self, user_id): cart_key = f\"cart:{user_id}\" cart_data = self.redis.get(cart_key) return json.loads(cart_data) if cart_data else {} def remove_from_cart(self, user_id, product_id): cart_key = f\"cart:{user_id}\" cart = self.get_cart(user_id) if product_id in cart: del cart[product_id] self.redis.setex(cart_key, timedelta(days=7), json.dumps(cart)) return cart def clear_cart(self, user_id): cart_key = f\"cart:{user_id}\" self.redis.delete(cart_key) def calculate_totals(self, user_id): cart = self.get_cart(user_id) subtotal = sum( item['price'] * item['quantity'] for item in cart.values() ) tax = subtotal * 0.08 # 8% tax shipping = 10.00 if subtotal < 50 else 0.00 # Free shipping over $50 total = subtotal + tax + shipping return { 'subtotal': subtotal, 'tax': tax, 'shipping': shipping, 'total': total } 3. Order Service with Transaction Management from contextlib import contextmanager import uuid from datetime import datetime class OrderService: def __init__(self): self.db = psycopg2.connect(\"dbname=orders user=admin\") self.kafka_producer = KafkaProducer(bootstrap_servers=['localhost:9092']) @contextmanager def transaction(self): \"\"\"Context manager for database transactions.\"\"\" conn = self.db try: yield conn conn.commit() except Exception as e: conn.rollback() raise e def create_order(self, user_id, cart_items, shipping_address, payment_method): \"\"\"Create order with ACID guarantees.\"\"\" with self.transaction()","date":"28-Jul-2025","heading":"Core Service Implementations","objectID":"\/posts\/system-design\/:47:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#core-service-implementations"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Product Service from flask import Flask, jsonify, request import redis import psycopg2 class ProductService: def __init__(self): self.db = psycopg2.connect(\"dbname=products user=admin\") self.cache = redis.Redis(host='localhost', port=6379) def get_product(self, product_id): # Try cache first cache_key = f\"product:{product_id}\" cached = self.cache.get(cache_key) if cached: return json.loads(cached) # Cache miss - query database cursor = self.db.cursor() cursor.execute(\"\"\" SELECT p.*, array_agg(pi.image_url) as images FROM products p LEFT JOIN product_images pi ON p.id = pi.product_id WHERE p.id = %s AND p.is_active = true GROUP BY p.id \"\"\", (product_id,)) product = cursor.fetchone() if product: product_data = self.serialize_product(product) # Cache for 1 hour self.cache.setex(cache_key, 3600, json.dumps(product_data)) return product_data return None def search_products(self, query, filters, page=1, page_size=20): # Delegate to Elasticsearch for full-text search es_query = { \"query\": { \"bool\": { \"must\": { \"multi_match\": { \"query\": query, \"fields\": [\"name^3\", \"description\", \"brand^2\"] } }, \"filter\": self.build_filters(filters) } }, \"from\": (page - 1) * page_size, \"size\": page_size, \"sort\": [{\"_score\": \"desc\"}, {\"created_at\": \"desc\"}] } results = self.elasticsearch.search(index=\"products\", body=es_query) return results['hits']['hits'] def build_filters(self, filters): filter_clauses = [] if filters.get('category'): filter_clauses.append({\"term\": {\"category_id\": filters['category']}}) if filters.get('price_range'): filter_clauses.append({ \"range\": { \"price\": { \"gte\": filters['price_range']['min'], \"lte\": filters['price_range']['max'] } } }) if filters.get('brand'): filter_clauses.append({\"terms\": {\"brand\": filters['brand']}}) return filter_clauses 2. Cart Service (Redis-based) import json from datetime import timedelta class CartService: def __init__(self): self.redis = redis.Redis(host='localhost', port=6379) def add_to_cart(self, user_id, product_id, quantity): cart_key = f\"cart:{user_id}\" # Get current cart cart = self.get_cart(user_id) # Add\/update item if product_id in cart: cart[product_id]['quantity'] += quantity else: # Fetch product details product = self.product_service.get_product(product_id) cart[product_id] = { 'product_id': product_id, 'name': product['name'], 'price': product['price'], 'quantity': quantity, 'image': product['images'][0] if product['images'] else None } # Save cart (expires in 7 days) self.redis.setex(cart_key, timedelta(days=7), json.dumps(cart)) return cart def get_cart(self, user_id): cart_key = f\"cart:{user_id}\" cart_data = self.redis.get(cart_key) return json.loads(cart_data) if cart_data else {} def remove_from_cart(self, user_id, product_id): cart_key = f\"cart:{user_id}\" cart = self.get_cart(user_id) if product_id in cart: del cart[product_id] self.redis.setex(cart_key, timedelta(days=7), json.dumps(cart)) return cart def clear_cart(self, user_id): cart_key = f\"cart:{user_id}\" self.redis.delete(cart_key) def calculate_totals(self, user_id): cart = self.get_cart(user_id) subtotal = sum( item['price'] * item['quantity'] for item in cart.values() ) tax = subtotal * 0.08 # 8% tax shipping = 10.00 if subtotal < 50 else 0.00 # Free shipping over $50 total = subtotal + tax + shipping return { 'subtotal': subtotal, 'tax': tax, 'shipping': shipping, 'total': total } 3. Order Service with Transaction Management from contextlib import contextmanager import uuid from datetime import datetime class OrderService: def __init__(self): self.db = psycopg2.connect(\"dbname=orders user=admin\") self.kafka_producer = KafkaProducer(bootstrap_servers=['localhost:9092']) @contextmanager def transaction(self): \"\"\"Context manager for database transactions.\"\"\" conn = self.db try: yield conn conn.commit() except Exception as e: conn.rollback() raise e def create_order(self, user_id, cart_items, shipping_address, payment_method): \"\"\"Create order with ACID guarantees.\"\"\" with self.transaction()","date":"28-Jul-2025","heading":"1. Product Service","objectID":"\/posts\/system-design\/:47:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#1-product-service"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Product Service from flask import Flask, jsonify, request import redis import psycopg2 class ProductService: def __init__(self): self.db = psycopg2.connect(\"dbname=products user=admin\") self.cache = redis.Redis(host='localhost', port=6379) def get_product(self, product_id): # Try cache first cache_key = f\"product:{product_id}\" cached = self.cache.get(cache_key) if cached: return json.loads(cached) # Cache miss - query database cursor = self.db.cursor() cursor.execute(\"\"\" SELECT p.*, array_agg(pi.image_url) as images FROM products p LEFT JOIN product_images pi ON p.id = pi.product_id WHERE p.id = %s AND p.is_active = true GROUP BY p.id \"\"\", (product_id,)) product = cursor.fetchone() if product: product_data = self.serialize_product(product) # Cache for 1 hour self.cache.setex(cache_key, 3600, json.dumps(product_data)) return product_data return None def search_products(self, query, filters, page=1, page_size=20): # Delegate to Elasticsearch for full-text search es_query = { \"query\": { \"bool\": { \"must\": { \"multi_match\": { \"query\": query, \"fields\": [\"name^3\", \"description\", \"brand^2\"] } }, \"filter\": self.build_filters(filters) } }, \"from\": (page - 1) * page_size, \"size\": page_size, \"sort\": [{\"_score\": \"desc\"}, {\"created_at\": \"desc\"}] } results = self.elasticsearch.search(index=\"products\", body=es_query) return results['hits']['hits'] def build_filters(self, filters): filter_clauses = [] if filters.get('category'): filter_clauses.append({\"term\": {\"category_id\": filters['category']}}) if filters.get('price_range'): filter_clauses.append({ \"range\": { \"price\": { \"gte\": filters['price_range']['min'], \"lte\": filters['price_range']['max'] } } }) if filters.get('brand'): filter_clauses.append({\"terms\": {\"brand\": filters['brand']}}) return filter_clauses 2. Cart Service (Redis-based) import json from datetime import timedelta class CartService: def __init__(self): self.redis = redis.Redis(host='localhost', port=6379) def add_to_cart(self, user_id, product_id, quantity): cart_key = f\"cart:{user_id}\" # Get current cart cart = self.get_cart(user_id) # Add\/update item if product_id in cart: cart[product_id]['quantity'] += quantity else: # Fetch product details product = self.product_service.get_product(product_id) cart[product_id] = { 'product_id': product_id, 'name': product['name'], 'price': product['price'], 'quantity': quantity, 'image': product['images'][0] if product['images'] else None } # Save cart (expires in 7 days) self.redis.setex(cart_key, timedelta(days=7), json.dumps(cart)) return cart def get_cart(self, user_id): cart_key = f\"cart:{user_id}\" cart_data = self.redis.get(cart_key) return json.loads(cart_data) if cart_data else {} def remove_from_cart(self, user_id, product_id): cart_key = f\"cart:{user_id}\" cart = self.get_cart(user_id) if product_id in cart: del cart[product_id] self.redis.setex(cart_key, timedelta(days=7), json.dumps(cart)) return cart def clear_cart(self, user_id): cart_key = f\"cart:{user_id}\" self.redis.delete(cart_key) def calculate_totals(self, user_id): cart = self.get_cart(user_id) subtotal = sum( item['price'] * item['quantity'] for item in cart.values() ) tax = subtotal * 0.08 # 8% tax shipping = 10.00 if subtotal < 50 else 0.00 # Free shipping over $50 total = subtotal + tax + shipping return { 'subtotal': subtotal, 'tax': tax, 'shipping': shipping, 'total': total } 3. Order Service with Transaction Management from contextlib import contextmanager import uuid from datetime import datetime class OrderService: def __init__(self): self.db = psycopg2.connect(\"dbname=orders user=admin\") self.kafka_producer = KafkaProducer(bootstrap_servers=['localhost:9092']) @contextmanager def transaction(self): \"\"\"Context manager for database transactions.\"\"\" conn = self.db try: yield conn conn.commit() except Exception as e: conn.rollback() raise e def create_order(self, user_id, cart_items, shipping_address, payment_method): \"\"\"Create order with ACID guarantees.\"\"\" with self.transaction()","date":"28-Jul-2025","heading":"2. Cart Service (Redis-based)","objectID":"\/posts\/system-design\/:47:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#2-cart-service-redis-based"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Product Service from flask import Flask, jsonify, request import redis import psycopg2 class ProductService: def __init__(self): self.db = psycopg2.connect(\"dbname=products user=admin\") self.cache = redis.Redis(host='localhost', port=6379) def get_product(self, product_id): # Try cache first cache_key = f\"product:{product_id}\" cached = self.cache.get(cache_key) if cached: return json.loads(cached) # Cache miss - query database cursor = self.db.cursor() cursor.execute(\"\"\" SELECT p.*, array_agg(pi.image_url) as images FROM products p LEFT JOIN product_images pi ON p.id = pi.product_id WHERE p.id = %s AND p.is_active = true GROUP BY p.id \"\"\", (product_id,)) product = cursor.fetchone() if product: product_data = self.serialize_product(product) # Cache for 1 hour self.cache.setex(cache_key, 3600, json.dumps(product_data)) return product_data return None def search_products(self, query, filters, page=1, page_size=20): # Delegate to Elasticsearch for full-text search es_query = { \"query\": { \"bool\": { \"must\": { \"multi_match\": { \"query\": query, \"fields\": [\"name^3\", \"description\", \"brand^2\"] } }, \"filter\": self.build_filters(filters) } }, \"from\": (page - 1) * page_size, \"size\": page_size, \"sort\": [{\"_score\": \"desc\"}, {\"created_at\": \"desc\"}] } results = self.elasticsearch.search(index=\"products\", body=es_query) return results['hits']['hits'] def build_filters(self, filters): filter_clauses = [] if filters.get('category'): filter_clauses.append({\"term\": {\"category_id\": filters['category']}}) if filters.get('price_range'): filter_clauses.append({ \"range\": { \"price\": { \"gte\": filters['price_range']['min'], \"lte\": filters['price_range']['max'] } } }) if filters.get('brand'): filter_clauses.append({\"terms\": {\"brand\": filters['brand']}}) return filter_clauses 2. Cart Service (Redis-based) import json from datetime import timedelta class CartService: def __init__(self): self.redis = redis.Redis(host='localhost', port=6379) def add_to_cart(self, user_id, product_id, quantity): cart_key = f\"cart:{user_id}\" # Get current cart cart = self.get_cart(user_id) # Add\/update item if product_id in cart: cart[product_id]['quantity'] += quantity else: # Fetch product details product = self.product_service.get_product(product_id) cart[product_id] = { 'product_id': product_id, 'name': product['name'], 'price': product['price'], 'quantity': quantity, 'image': product['images'][0] if product['images'] else None } # Save cart (expires in 7 days) self.redis.setex(cart_key, timedelta(days=7), json.dumps(cart)) return cart def get_cart(self, user_id): cart_key = f\"cart:{user_id}\" cart_data = self.redis.get(cart_key) return json.loads(cart_data) if cart_data else {} def remove_from_cart(self, user_id, product_id): cart_key = f\"cart:{user_id}\" cart = self.get_cart(user_id) if product_id in cart: del cart[product_id] self.redis.setex(cart_key, timedelta(days=7), json.dumps(cart)) return cart def clear_cart(self, user_id): cart_key = f\"cart:{user_id}\" self.redis.delete(cart_key) def calculate_totals(self, user_id): cart = self.get_cart(user_id) subtotal = sum( item['price'] * item['quantity'] for item in cart.values() ) tax = subtotal * 0.08 # 8% tax shipping = 10.00 if subtotal < 50 else 0.00 # Free shipping over $50 total = subtotal + tax + shipping return { 'subtotal': subtotal, 'tax': tax, 'shipping': shipping, 'total': total } 3. Order Service with Transaction Management from contextlib import contextmanager import uuid from datetime import datetime class OrderService: def __init__(self): self.db = psycopg2.connect(\"dbname=orders user=admin\") self.kafka_producer = KafkaProducer(bootstrap_servers=['localhost:9092']) @contextmanager def transaction(self): \"\"\"Context manager for database transactions.\"\"\" conn = self.db try: yield conn conn.commit() except Exception as e: conn.rollback() raise e def create_order(self, user_id, cart_items, shipping_address, payment_method): \"\"\"Create order with ACID guarantees.\"\"\" with self.transaction()","date":"28-Jul-2025","heading":"3. Order Service with Transaction Management","objectID":"\/posts\/system-design\/:47:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#3-order-service-with-transaction-management"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Product Service from flask import Flask, jsonify, request import redis import psycopg2 class ProductService: def __init__(self): self.db = psycopg2.connect(\"dbname=products user=admin\") self.cache = redis.Redis(host='localhost', port=6379) def get_product(self, product_id): # Try cache first cache_key = f\"product:{product_id}\" cached = self.cache.get(cache_key) if cached: return json.loads(cached) # Cache miss - query database cursor = self.db.cursor() cursor.execute(\"\"\" SELECT p.*, array_agg(pi.image_url) as images FROM products p LEFT JOIN product_images pi ON p.id = pi.product_id WHERE p.id = %s AND p.is_active = true GROUP BY p.id \"\"\", (product_id,)) product = cursor.fetchone() if product: product_data = self.serialize_product(product) # Cache for 1 hour self.cache.setex(cache_key, 3600, json.dumps(product_data)) return product_data return None def search_products(self, query, filters, page=1, page_size=20): # Delegate to Elasticsearch for full-text search es_query = { \"query\": { \"bool\": { \"must\": { \"multi_match\": { \"query\": query, \"fields\": [\"name^3\", \"description\", \"brand^2\"] } }, \"filter\": self.build_filters(filters) } }, \"from\": (page - 1) * page_size, \"size\": page_size, \"sort\": [{\"_score\": \"desc\"}, {\"created_at\": \"desc\"}] } results = self.elasticsearch.search(index=\"products\", body=es_query) return results['hits']['hits'] def build_filters(self, filters): filter_clauses = [] if filters.get('category'): filter_clauses.append({\"term\": {\"category_id\": filters['category']}}) if filters.get('price_range'): filter_clauses.append({ \"range\": { \"price\": { \"gte\": filters['price_range']['min'], \"lte\": filters['price_range']['max'] } } }) if filters.get('brand'): filter_clauses.append({\"terms\": {\"brand\": filters['brand']}}) return filter_clauses 2. Cart Service (Redis-based) import json from datetime import timedelta class CartService: def __init__(self): self.redis = redis.Redis(host='localhost', port=6379) def add_to_cart(self, user_id, product_id, quantity): cart_key = f\"cart:{user_id}\" # Get current cart cart = self.get_cart(user_id) # Add\/update item if product_id in cart: cart[product_id]['quantity'] += quantity else: # Fetch product details product = self.product_service.get_product(product_id) cart[product_id] = { 'product_id': product_id, 'name': product['name'], 'price': product['price'], 'quantity': quantity, 'image': product['images'][0] if product['images'] else None } # Save cart (expires in 7 days) self.redis.setex(cart_key, timedelta(days=7), json.dumps(cart)) return cart def get_cart(self, user_id): cart_key = f\"cart:{user_id}\" cart_data = self.redis.get(cart_key) return json.loads(cart_data) if cart_data else {} def remove_from_cart(self, user_id, product_id): cart_key = f\"cart:{user_id}\" cart = self.get_cart(user_id) if product_id in cart: del cart[product_id] self.redis.setex(cart_key, timedelta(days=7), json.dumps(cart)) return cart def clear_cart(self, user_id): cart_key = f\"cart:{user_id}\" self.redis.delete(cart_key) def calculate_totals(self, user_id): cart = self.get_cart(user_id) subtotal = sum( item['price'] * item['quantity'] for item in cart.values() ) tax = subtotal * 0.08 # 8% tax shipping = 10.00 if subtotal < 50 else 0.00 # Free shipping over $50 total = subtotal + tax + shipping return { 'subtotal': subtotal, 'tax': tax, 'shipping': shipping, 'total': total } 3. Order Service with Transaction Management from contextlib import contextmanager import uuid from datetime import datetime class OrderService: def __init__(self): self.db = psycopg2.connect(\"dbname=orders user=admin\") self.kafka_producer = KafkaProducer(bootstrap_servers=['localhost:9092']) @contextmanager def transaction(self): \"\"\"Context manager for database transactions.\"\"\" conn = self.db try: yield conn conn.commit() except Exception as e: conn.rollback() raise e def create_order(self, user_id, cart_items, shipping_address, payment_method): \"\"\"Create order with ACID guarantees.\"\"\" with self.transaction()","date":"28-Jul-2025","heading":"4. Inventory Management","objectID":"\/posts\/system-design\/:47:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#4-inventory-management"},{"categories":["software-engineering"],"collections":null,"content":"Search Implementation with Elasticsearch from elasticsearch import Elasticsearch class SearchService: def __init__(self): self.es = Elasticsearch(['localhost:9200']) def index_product(self, product): \"\"\"Index product for search.\"\"\" self.es.index(index='products', id=product['id'], body={ 'name': product['name'], 'description': product['description'], 'category': product['category'], 'brand': product['brand'], 'price': product['price'], 'rating': product.get('rating', 0), 'num_reviews': product.get('num_reviews', 0), 'in_stock': product.get('in_stock', True), 'created_at': product['created_at'] }) def autocomplete(self, query): \"\"\"Suggest completions for search query.\"\"\" result = self.es.search(index='products', body={ 'suggest': { 'product-suggest': { 'prefix': query, 'completion': { 'field': 'name.completion', 'size': 10, 'skip_duplicates': True } } } }) suggestions = result['suggest']['product-suggest'][0]['options'] return [s['text'] for s in suggestions] ","date":"28-Jul-2025","heading":"Search Implementation with Elasticsearch","objectID":"\/posts\/system-design\/:47:6","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#search-implementation-with-elasticsearch"},{"categories":["software-engineering"],"collections":null,"content":"Scalability Optimizations 1. Database Sharding: Shard orders by user_id (user affinity) Shard products by category (hot products in separate shards) 2. Caching Strategy: Product details: Redis (1-hour TTL) Cart: Redis (7-day TTL) Search results: Redis (15-minute TTL) 3. CDN for Static Assets: Product images CSS\/JS bundles Category thumbnails 4. Async Processing: Order confirmation emails Inventory updates Analytics events Review indexing ","date":"28-Jul-2025","heading":"Scalability Optimizations","objectID":"\/posts\/system-design\/:47:7","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#scalability-optimizations"},{"categories":["software-engineering"],"collections":null,"content":"Key Takeaways Use Redis for cart - Fast, supports TTL, scales horizontally ACID for payments - PostgreSQL transactions ensure consistency Elasticsearch for search - Fast full-text search with facets Kafka for events - Decouple order processing Inventory reservation - Prevent overselling during checkout Idempotency keys - Prevent duplicate orders on retry ","date":"28-Jul-2025","heading":"Key Takeaways","objectID":"\/posts\/system-design\/:47:8","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#key-takeaways-2"},{"categories":["software-engineering"],"collections":null,"content":"49. Design an Online Bookstore (Amazon Books \/ Goodreads) ","date":"28-Jul-2025","heading":"49. Design an Online Bookstore (Amazon Books \/ Goodreads)","objectID":"\/posts\/system-design\/:48:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#49-design-an-online-bookstore-amazon-books--goodreads"},{"categories":["software-engineering"],"collections":null,"content":"Overview Design a comprehensive online bookstore with features for browsing, purchasing, reading reviews, creating reading lists, and personalized book recommendations. ","date":"28-Jul-2025","heading":"Overview","objectID":"\/posts\/system-design\/:48:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#overview-3"},{"categories":["software-engineering"],"collections":null,"content":"Problem Requirements Functional Requirements: Book catalog browsing and search Book details, reviews, and ratings Purchase and download (for e-books) Reading lists and bookshelves Book recommendations Author pages and following Book clubs and discussions Non-Functional Requirements: 50 million users 10 million books in catalog 1 billion reviews 100K concurrent users Sub-second search response 99.9% availability Personalized recommendations Capacity Estimation: Book metadata: 10M books \u00d7 5KB = 50 GB Book covers: 10M \u00d7 200KB = 2 TB Reviews: 1B \u00d7 500 bytes = 500 GB E-books: 1M \u00d7 5MB = 5 TB ","date":"28-Jul-2025","heading":"Problem Requirements","objectID":"\/posts\/system-design\/:48:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#problem-requirements-3"},{"categories":["software-engineering"],"collections":null,"content":"High-Level Architecture Online Bookstore Architecture \u2014 Microservices for book catalog, inventory, orders, recommendations, and reviews behind an API gateway ","date":"28-Jul-2025","heading":"High-Level Architecture","objectID":"\/posts\/system-design\/:48:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#high-level-architecture-2"},{"categories":["software-engineering"],"collections":null,"content":"Database Schema # Book Schema class Book: isbn: str (PK) title: str subtitle: str author_ids: UUID[] (FK) publisher: str publication_date: date language: str pages: int format: enum('hardcover', 'paperback', 'ebook', 'audiobook') description: text genre_ids: UUID[] price: decimal(10, 2) avg_rating: float num_ratings: int cover_image_url: str ebook_url: str # S3 link created_at: timestamp class Author: id: UUID (PK) name: str bio: text birth_date: date nationality: str photo_url: str num_followers: int class Review: id: UUID (PK) user_id: UUID (FK) book_isbn: str (FK) rating: int (1-5) review_text: text num_likes: int num_comments: int spoiler: boolean created_at: timestamp updated_at: timestamp class Bookshelf: id: UUID (PK) user_id: UUID (FK) name: str # \"Want to Read\", \"Currently Reading\", \"Read\" is_public: boolean created_at: timestamp class BookshelfItem: id: UUID (PK) bookshelf_id: UUID (FK) book_isbn: str (FK) added_at: timestamp reading_progress: int # Percentage read started_at: timestamp finished_at: timestamp ","date":"28-Jul-2025","heading":"Database Schema","objectID":"\/posts\/system-design\/:48:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#database-schema-1"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Book Service with Multi-Format Support class BookService: def __init__(self): self.db = PostgreSQL() self.cache = Redis() self.s3 = S3Client() def get_book(self, isbn): \"\"\"Get book details with caching.\"\"\" cache_key = f\"book:{isbn}\" cached = self.cache.get(cache_key) if cached: return json.loads(cached) book = self.db.query(\"\"\" SELECT b.*, array_agg(DISTINCT a.name) as authors, array_agg(DISTINCT g.name) as genres FROM books b LEFT JOIN book_authors ba ON b.isbn = ba.book_isbn LEFT JOIN authors a ON ba.author_id = a.id LEFT JOIN book_genres bg ON b.isbn = bg.book_isbn LEFT JOIN genres g ON bg.genre_id = g.id WHERE b.isbn = %s GROUP BY b.isbn \"\"\", (isbn,)) if book: book_data = self.serialize(book) self.cache.setex(cache_key, 3600, json.dumps(book_data)) return book_data return None def get_similar_books(self, isbn, limit=10): \"\"\"Find similar books using content-based filtering.\"\"\" book = self.get_book(isbn) if not book: return [] # Find books with similar genres and authors similar = self.db.query(\"\"\" WITH book_features AS ( SELECT b.isbn, array_agg(DISTINCT bg.genre_id) as genres, array_agg(DISTINCT ba.author_id) as authors FROM books b LEFT JOIN book_genres bg ON b.isbn = bg.book_isbn LEFT JOIN book_authors ba ON b.isbn = ba.book_isbn WHERE b.isbn = %s GROUP BY b.isbn ) SELECT b.isbn, b.title, b.avg_rating, -- Calculate similarity score ( -- Genre overlap cardinality(array_intersect( (SELECT genres FROM book_features), array_agg(DISTINCT bg2.genre_id) )) * 2 + -- Author overlap cardinality(array_intersect( (SELECT authors FROM book_features), array_agg(DISTINCT ba2.author_id) )) * 3 ) as similarity_score FROM books b LEFT JOIN book_genres bg2 ON b.isbn = bg2.book_isbn LEFT JOIN book_authors ba2 ON b.isbn = ba2.book_isbn WHERE b.isbn != %s GROUP BY b.isbn HAVING similarity_score > 0 ORDER BY similarity_score DESC, b.avg_rating DESC LIMIT %s \"\"\", (isbn, isbn, limit)) return similar def download_ebook(self, user_id, isbn): \"\"\"Generate pre-signed URL for e-book download.\"\"\" # Verify user owns the book if not self.verify_purchase(user_id, isbn): raise UnauthorizedError(\"User has not purchased this e-book\") book = self.get_book(isbn) if not book['ebook_url']: raise NotFoundError(\"E-book not available for this title\") # Generate pre-signed URL (valid for 1 hour) download_url = self.s3.generate_presigned_url( 'get_object', Params={'Bucket': 'ebooks', 'Key': book['ebook_url']}, ExpiresIn=3600 ) # Log download for analytics self.analytics.track('ebook_downloaded', { 'user_id': user_id, 'isbn': isbn, 'timestamp': datetime.now() }) return download_url 2. Review Service with Cassandra for Scale from cassandra.cluster import Cluster class ReviewService: def __init__(self): self.cluster = Cluster(['cassandra-node-1', 'cassandra-node-2']) self.session = self.cluster.connect('bookstore') self.cache = Redis() def create_review(self, user_id, book_isbn, rating, review_text, spoiler=False): \"\"\"Create a new book review.\"\"\" review_id = uuid.uuid4() # Insert into Cassandra (denormalized for fast reads) self.session.execute(\"\"\" INSERT INTO reviews_by_book (book_isbn, review_id, user_id, rating, review_text, spoiler, num_likes, created_at) VALUES (%s, %s, %s, %s, %s, %s, 0, %s) \"\"\", (book_isbn, review_id, user_id, rating, review_text, spoiler, datetime.now())) self.session.execute(\"\"\" INSERT INTO reviews_by_user (user_id, review_id, book_isbn, rating, review_text, created_at) VALUES (%s, %s, %s, %s, %s, %s) \"\"\", (user_id, review_id, book_isbn, rating, review_text, datetime.now())) # Update book's average rating (async via Kafka) self.kafka_producer.send('review-events', { 'event_type': 'review_created', 'book_isbn': book_isbn, 'rating': rating }) # Invalidate cache self.cache.delete(f\"reviews:book:{book_isbn}\") self.cache.delete(f\"book:{book_isbn}\") return review_id def get_reviews_for_book(self, book_isbn, sort_by='helpful', page=1, page_size=20): \"\"\"Get paginated reviews for a book.\"\"\" cache_key = f\"reviews:book:{boo","date":"28-Jul-2025","heading":"Core Service Implementations","objectID":"\/posts\/system-design\/:48:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#core-service-implementations-1"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Book Service with Multi-Format Support class BookService: def __init__(self): self.db = PostgreSQL() self.cache = Redis() self.s3 = S3Client() def get_book(self, isbn): \"\"\"Get book details with caching.\"\"\" cache_key = f\"book:{isbn}\" cached = self.cache.get(cache_key) if cached: return json.loads(cached) book = self.db.query(\"\"\" SELECT b.*, array_agg(DISTINCT a.name) as authors, array_agg(DISTINCT g.name) as genres FROM books b LEFT JOIN book_authors ba ON b.isbn = ba.book_isbn LEFT JOIN authors a ON ba.author_id = a.id LEFT JOIN book_genres bg ON b.isbn = bg.book_isbn LEFT JOIN genres g ON bg.genre_id = g.id WHERE b.isbn = %s GROUP BY b.isbn \"\"\", (isbn,)) if book: book_data = self.serialize(book) self.cache.setex(cache_key, 3600, json.dumps(book_data)) return book_data return None def get_similar_books(self, isbn, limit=10): \"\"\"Find similar books using content-based filtering.\"\"\" book = self.get_book(isbn) if not book: return [] # Find books with similar genres and authors similar = self.db.query(\"\"\" WITH book_features AS ( SELECT b.isbn, array_agg(DISTINCT bg.genre_id) as genres, array_agg(DISTINCT ba.author_id) as authors FROM books b LEFT JOIN book_genres bg ON b.isbn = bg.book_isbn LEFT JOIN book_authors ba ON b.isbn = ba.book_isbn WHERE b.isbn = %s GROUP BY b.isbn ) SELECT b.isbn, b.title, b.avg_rating, -- Calculate similarity score ( -- Genre overlap cardinality(array_intersect( (SELECT genres FROM book_features), array_agg(DISTINCT bg2.genre_id) )) * 2 + -- Author overlap cardinality(array_intersect( (SELECT authors FROM book_features), array_agg(DISTINCT ba2.author_id) )) * 3 ) as similarity_score FROM books b LEFT JOIN book_genres bg2 ON b.isbn = bg2.book_isbn LEFT JOIN book_authors ba2 ON b.isbn = ba2.book_isbn WHERE b.isbn != %s GROUP BY b.isbn HAVING similarity_score > 0 ORDER BY similarity_score DESC, b.avg_rating DESC LIMIT %s \"\"\", (isbn, isbn, limit)) return similar def download_ebook(self, user_id, isbn): \"\"\"Generate pre-signed URL for e-book download.\"\"\" # Verify user owns the book if not self.verify_purchase(user_id, isbn): raise UnauthorizedError(\"User has not purchased this e-book\") book = self.get_book(isbn) if not book['ebook_url']: raise NotFoundError(\"E-book not available for this title\") # Generate pre-signed URL (valid for 1 hour) download_url = self.s3.generate_presigned_url( 'get_object', Params={'Bucket': 'ebooks', 'Key': book['ebook_url']}, ExpiresIn=3600 ) # Log download for analytics self.analytics.track('ebook_downloaded', { 'user_id': user_id, 'isbn': isbn, 'timestamp': datetime.now() }) return download_url 2. Review Service with Cassandra for Scale from cassandra.cluster import Cluster class ReviewService: def __init__(self): self.cluster = Cluster(['cassandra-node-1', 'cassandra-node-2']) self.session = self.cluster.connect('bookstore') self.cache = Redis() def create_review(self, user_id, book_isbn, rating, review_text, spoiler=False): \"\"\"Create a new book review.\"\"\" review_id = uuid.uuid4() # Insert into Cassandra (denormalized for fast reads) self.session.execute(\"\"\" INSERT INTO reviews_by_book (book_isbn, review_id, user_id, rating, review_text, spoiler, num_likes, created_at) VALUES (%s, %s, %s, %s, %s, %s, 0, %s) \"\"\", (book_isbn, review_id, user_id, rating, review_text, spoiler, datetime.now())) self.session.execute(\"\"\" INSERT INTO reviews_by_user (user_id, review_id, book_isbn, rating, review_text, created_at) VALUES (%s, %s, %s, %s, %s, %s) \"\"\", (user_id, review_id, book_isbn, rating, review_text, datetime.now())) # Update book's average rating (async via Kafka) self.kafka_producer.send('review-events', { 'event_type': 'review_created', 'book_isbn': book_isbn, 'rating': rating }) # Invalidate cache self.cache.delete(f\"reviews:book:{book_isbn}\") self.cache.delete(f\"book:{book_isbn}\") return review_id def get_reviews_for_book(self, book_isbn, sort_by='helpful', page=1, page_size=20): \"\"\"Get paginated reviews for a book.\"\"\" cache_key = f\"reviews:book:{boo","date":"28-Jul-2025","heading":"1. Book Service with Multi-Format Support","objectID":"\/posts\/system-design\/:48:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#1-book-service-with-multi-format-support"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Book Service with Multi-Format Support class BookService: def __init__(self): self.db = PostgreSQL() self.cache = Redis() self.s3 = S3Client() def get_book(self, isbn): \"\"\"Get book details with caching.\"\"\" cache_key = f\"book:{isbn}\" cached = self.cache.get(cache_key) if cached: return json.loads(cached) book = self.db.query(\"\"\" SELECT b.*, array_agg(DISTINCT a.name) as authors, array_agg(DISTINCT g.name) as genres FROM books b LEFT JOIN book_authors ba ON b.isbn = ba.book_isbn LEFT JOIN authors a ON ba.author_id = a.id LEFT JOIN book_genres bg ON b.isbn = bg.book_isbn LEFT JOIN genres g ON bg.genre_id = g.id WHERE b.isbn = %s GROUP BY b.isbn \"\"\", (isbn,)) if book: book_data = self.serialize(book) self.cache.setex(cache_key, 3600, json.dumps(book_data)) return book_data return None def get_similar_books(self, isbn, limit=10): \"\"\"Find similar books using content-based filtering.\"\"\" book = self.get_book(isbn) if not book: return [] # Find books with similar genres and authors similar = self.db.query(\"\"\" WITH book_features AS ( SELECT b.isbn, array_agg(DISTINCT bg.genre_id) as genres, array_agg(DISTINCT ba.author_id) as authors FROM books b LEFT JOIN book_genres bg ON b.isbn = bg.book_isbn LEFT JOIN book_authors ba ON b.isbn = ba.book_isbn WHERE b.isbn = %s GROUP BY b.isbn ) SELECT b.isbn, b.title, b.avg_rating, -- Calculate similarity score ( -- Genre overlap cardinality(array_intersect( (SELECT genres FROM book_features), array_agg(DISTINCT bg2.genre_id) )) * 2 + -- Author overlap cardinality(array_intersect( (SELECT authors FROM book_features), array_agg(DISTINCT ba2.author_id) )) * 3 ) as similarity_score FROM books b LEFT JOIN book_genres bg2 ON b.isbn = bg2.book_isbn LEFT JOIN book_authors ba2 ON b.isbn = ba2.book_isbn WHERE b.isbn != %s GROUP BY b.isbn HAVING similarity_score > 0 ORDER BY similarity_score DESC, b.avg_rating DESC LIMIT %s \"\"\", (isbn, isbn, limit)) return similar def download_ebook(self, user_id, isbn): \"\"\"Generate pre-signed URL for e-book download.\"\"\" # Verify user owns the book if not self.verify_purchase(user_id, isbn): raise UnauthorizedError(\"User has not purchased this e-book\") book = self.get_book(isbn) if not book['ebook_url']: raise NotFoundError(\"E-book not available for this title\") # Generate pre-signed URL (valid for 1 hour) download_url = self.s3.generate_presigned_url( 'get_object', Params={'Bucket': 'ebooks', 'Key': book['ebook_url']}, ExpiresIn=3600 ) # Log download for analytics self.analytics.track('ebook_downloaded', { 'user_id': user_id, 'isbn': isbn, 'timestamp': datetime.now() }) return download_url 2. Review Service with Cassandra for Scale from cassandra.cluster import Cluster class ReviewService: def __init__(self): self.cluster = Cluster(['cassandra-node-1', 'cassandra-node-2']) self.session = self.cluster.connect('bookstore') self.cache = Redis() def create_review(self, user_id, book_isbn, rating, review_text, spoiler=False): \"\"\"Create a new book review.\"\"\" review_id = uuid.uuid4() # Insert into Cassandra (denormalized for fast reads) self.session.execute(\"\"\" INSERT INTO reviews_by_book (book_isbn, review_id, user_id, rating, review_text, spoiler, num_likes, created_at) VALUES (%s, %s, %s, %s, %s, %s, 0, %s) \"\"\", (book_isbn, review_id, user_id, rating, review_text, spoiler, datetime.now())) self.session.execute(\"\"\" INSERT INTO reviews_by_user (user_id, review_id, book_isbn, rating, review_text, created_at) VALUES (%s, %s, %s, %s, %s, %s) \"\"\", (user_id, review_id, book_isbn, rating, review_text, datetime.now())) # Update book's average rating (async via Kafka) self.kafka_producer.send('review-events', { 'event_type': 'review_created', 'book_isbn': book_isbn, 'rating': rating }) # Invalidate cache self.cache.delete(f\"reviews:book:{book_isbn}\") self.cache.delete(f\"book:{book_isbn}\") return review_id def get_reviews_for_book(self, book_isbn, sort_by='helpful', page=1, page_size=20): \"\"\"Get paginated reviews for a book.\"\"\" cache_key = f\"reviews:book:{boo","date":"28-Jul-2025","heading":"2. Review Service with Cassandra for Scale","objectID":"\/posts\/system-design\/:48:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#2-review-service-with-cassandra-for-scale"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Book Service with Multi-Format Support class BookService: def __init__(self): self.db = PostgreSQL() self.cache = Redis() self.s3 = S3Client() def get_book(self, isbn): \"\"\"Get book details with caching.\"\"\" cache_key = f\"book:{isbn}\" cached = self.cache.get(cache_key) if cached: return json.loads(cached) book = self.db.query(\"\"\" SELECT b.*, array_agg(DISTINCT a.name) as authors, array_agg(DISTINCT g.name) as genres FROM books b LEFT JOIN book_authors ba ON b.isbn = ba.book_isbn LEFT JOIN authors a ON ba.author_id = a.id LEFT JOIN book_genres bg ON b.isbn = bg.book_isbn LEFT JOIN genres g ON bg.genre_id = g.id WHERE b.isbn = %s GROUP BY b.isbn \"\"\", (isbn,)) if book: book_data = self.serialize(book) self.cache.setex(cache_key, 3600, json.dumps(book_data)) return book_data return None def get_similar_books(self, isbn, limit=10): \"\"\"Find similar books using content-based filtering.\"\"\" book = self.get_book(isbn) if not book: return [] # Find books with similar genres and authors similar = self.db.query(\"\"\" WITH book_features AS ( SELECT b.isbn, array_agg(DISTINCT bg.genre_id) as genres, array_agg(DISTINCT ba.author_id) as authors FROM books b LEFT JOIN book_genres bg ON b.isbn = bg.book_isbn LEFT JOIN book_authors ba ON b.isbn = ba.book_isbn WHERE b.isbn = %s GROUP BY b.isbn ) SELECT b.isbn, b.title, b.avg_rating, -- Calculate similarity score ( -- Genre overlap cardinality(array_intersect( (SELECT genres FROM book_features), array_agg(DISTINCT bg2.genre_id) )) * 2 + -- Author overlap cardinality(array_intersect( (SELECT authors FROM book_features), array_agg(DISTINCT ba2.author_id) )) * 3 ) as similarity_score FROM books b LEFT JOIN book_genres bg2 ON b.isbn = bg2.book_isbn LEFT JOIN book_authors ba2 ON b.isbn = ba2.book_isbn WHERE b.isbn != %s GROUP BY b.isbn HAVING similarity_score > 0 ORDER BY similarity_score DESC, b.avg_rating DESC LIMIT %s \"\"\", (isbn, isbn, limit)) return similar def download_ebook(self, user_id, isbn): \"\"\"Generate pre-signed URL for e-book download.\"\"\" # Verify user owns the book if not self.verify_purchase(user_id, isbn): raise UnauthorizedError(\"User has not purchased this e-book\") book = self.get_book(isbn) if not book['ebook_url']: raise NotFoundError(\"E-book not available for this title\") # Generate pre-signed URL (valid for 1 hour) download_url = self.s3.generate_presigned_url( 'get_object', Params={'Bucket': 'ebooks', 'Key': book['ebook_url']}, ExpiresIn=3600 ) # Log download for analytics self.analytics.track('ebook_downloaded', { 'user_id': user_id, 'isbn': isbn, 'timestamp': datetime.now() }) return download_url 2. Review Service with Cassandra for Scale from cassandra.cluster import Cluster class ReviewService: def __init__(self): self.cluster = Cluster(['cassandra-node-1', 'cassandra-node-2']) self.session = self.cluster.connect('bookstore') self.cache = Redis() def create_review(self, user_id, book_isbn, rating, review_text, spoiler=False): \"\"\"Create a new book review.\"\"\" review_id = uuid.uuid4() # Insert into Cassandra (denormalized for fast reads) self.session.execute(\"\"\" INSERT INTO reviews_by_book (book_isbn, review_id, user_id, rating, review_text, spoiler, num_likes, created_at) VALUES (%s, %s, %s, %s, %s, %s, 0, %s) \"\"\", (book_isbn, review_id, user_id, rating, review_text, spoiler, datetime.now())) self.session.execute(\"\"\" INSERT INTO reviews_by_user (user_id, review_id, book_isbn, rating, review_text, created_at) VALUES (%s, %s, %s, %s, %s, %s) \"\"\", (user_id, review_id, book_isbn, rating, review_text, datetime.now())) # Update book's average rating (async via Kafka) self.kafka_producer.send('review-events', { 'event_type': 'review_created', 'book_isbn': book_isbn, 'rating': rating }) # Invalidate cache self.cache.delete(f\"reviews:book:{book_isbn}\") self.cache.delete(f\"book:{book_isbn}\") return review_id def get_reviews_for_book(self, book_isbn, sort_by='helpful', page=1, page_size=20): \"\"\"Get paginated reviews for a book.\"\"\" cache_key = f\"reviews:book:{boo","date":"28-Jul-2025","heading":"3. Recommendation Engine","objectID":"\/posts\/system-design\/:48:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#3-recommendation-engine"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Book Service with Multi-Format Support class BookService: def __init__(self): self.db = PostgreSQL() self.cache = Redis() self.s3 = S3Client() def get_book(self, isbn): \"\"\"Get book details with caching.\"\"\" cache_key = f\"book:{isbn}\" cached = self.cache.get(cache_key) if cached: return json.loads(cached) book = self.db.query(\"\"\" SELECT b.*, array_agg(DISTINCT a.name) as authors, array_agg(DISTINCT g.name) as genres FROM books b LEFT JOIN book_authors ba ON b.isbn = ba.book_isbn LEFT JOIN authors a ON ba.author_id = a.id LEFT JOIN book_genres bg ON b.isbn = bg.book_isbn LEFT JOIN genres g ON bg.genre_id = g.id WHERE b.isbn = %s GROUP BY b.isbn \"\"\", (isbn,)) if book: book_data = self.serialize(book) self.cache.setex(cache_key, 3600, json.dumps(book_data)) return book_data return None def get_similar_books(self, isbn, limit=10): \"\"\"Find similar books using content-based filtering.\"\"\" book = self.get_book(isbn) if not book: return [] # Find books with similar genres and authors similar = self.db.query(\"\"\" WITH book_features AS ( SELECT b.isbn, array_agg(DISTINCT bg.genre_id) as genres, array_agg(DISTINCT ba.author_id) as authors FROM books b LEFT JOIN book_genres bg ON b.isbn = bg.book_isbn LEFT JOIN book_authors ba ON b.isbn = ba.book_isbn WHERE b.isbn = %s GROUP BY b.isbn ) SELECT b.isbn, b.title, b.avg_rating, -- Calculate similarity score ( -- Genre overlap cardinality(array_intersect( (SELECT genres FROM book_features), array_agg(DISTINCT bg2.genre_id) )) * 2 + -- Author overlap cardinality(array_intersect( (SELECT authors FROM book_features), array_agg(DISTINCT ba2.author_id) )) * 3 ) as similarity_score FROM books b LEFT JOIN book_genres bg2 ON b.isbn = bg2.book_isbn LEFT JOIN book_authors ba2 ON b.isbn = ba2.book_isbn WHERE b.isbn != %s GROUP BY b.isbn HAVING similarity_score > 0 ORDER BY similarity_score DESC, b.avg_rating DESC LIMIT %s \"\"\", (isbn, isbn, limit)) return similar def download_ebook(self, user_id, isbn): \"\"\"Generate pre-signed URL for e-book download.\"\"\" # Verify user owns the book if not self.verify_purchase(user_id, isbn): raise UnauthorizedError(\"User has not purchased this e-book\") book = self.get_book(isbn) if not book['ebook_url']: raise NotFoundError(\"E-book not available for this title\") # Generate pre-signed URL (valid for 1 hour) download_url = self.s3.generate_presigned_url( 'get_object', Params={'Bucket': 'ebooks', 'Key': book['ebook_url']}, ExpiresIn=3600 ) # Log download for analytics self.analytics.track('ebook_downloaded', { 'user_id': user_id, 'isbn': isbn, 'timestamp': datetime.now() }) return download_url 2. Review Service with Cassandra for Scale from cassandra.cluster import Cluster class ReviewService: def __init__(self): self.cluster = Cluster(['cassandra-node-1', 'cassandra-node-2']) self.session = self.cluster.connect('bookstore') self.cache = Redis() def create_review(self, user_id, book_isbn, rating, review_text, spoiler=False): \"\"\"Create a new book review.\"\"\" review_id = uuid.uuid4() # Insert into Cassandra (denormalized for fast reads) self.session.execute(\"\"\" INSERT INTO reviews_by_book (book_isbn, review_id, user_id, rating, review_text, spoiler, num_likes, created_at) VALUES (%s, %s, %s, %s, %s, %s, 0, %s) \"\"\", (book_isbn, review_id, user_id, rating, review_text, spoiler, datetime.now())) self.session.execute(\"\"\" INSERT INTO reviews_by_user (user_id, review_id, book_isbn, rating, review_text, created_at) VALUES (%s, %s, %s, %s, %s, %s) \"\"\", (user_id, review_id, book_isbn, rating, review_text, datetime.now())) # Update book's average rating (async via Kafka) self.kafka_producer.send('review-events', { 'event_type': 'review_created', 'book_isbn': book_isbn, 'rating': rating }) # Invalidate cache self.cache.delete(f\"reviews:book:{book_isbn}\") self.cache.delete(f\"book:{book_isbn}\") return review_id def get_reviews_for_book(self, book_isbn, sort_by='helpful', page=1, page_size=20): \"\"\"Get paginated reviews for a book.\"\"\" cache_key = f\"reviews:book:{boo","date":"28-Jul-2025","heading":"4. Reading List \/ Bookshelf Management","objectID":"\/posts\/system-design\/:48:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#4-reading-list--bookshelf-management"},{"categories":["software-engineering"],"collections":null,"content":"Scalability Optimizations 1. Caching Strategy: Book details: 1-hour TTL (rarely changes) Reviews: 5-minute TTL (frequently updated) User embeddings: 24-hour TTL Popular books: 1-hour TTL 2. Database Sharding: Books: Shard by ISBN prefix Reviews: Shard by book_isbn (Cassandra partition key) Users: Shard by user_id 3. Search Optimization: Elasticsearch with custom analyzers for book titles Autocomplete with edge n-grams Faceted search (genre, author, rating, price) 4. ML Model Serving: TensorFlow Serving for real-time recommendations Batch processing for embedding generation (nightly) A\/B testing framework for recommendation algorithms ","date":"28-Jul-2025","heading":"Scalability Optimizations","objectID":"\/posts\/system-design\/:48:6","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#scalability-optimizations-1"},{"categories":["software-engineering"],"collections":null,"content":"Key Takeaways Cassandra for reviews - Write-heavy, needs horizontal scaling Hybrid recommendations - Combine collaborative + content-based User embeddings - Cache in Redis for fast lookups Pre-signed URLs - Secure e-book downloads without exposing S3 Reading progress tracking - Engages users, improves recommendations Denormalization - Store avg_rating on book to avoid aggregation queries ","date":"28-Jul-2025","heading":"Key Takeaways","objectID":"\/posts\/system-design\/:48:7","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#key-takeaways-3"},{"categories":["software-engineering"],"collections":null,"content":"50. Design an Online Multiplayer Game Backend (Real-Time) ","date":"28-Jul-2025","heading":"50. Design an Online Multiplayer Game Backend (Real-Time)","objectID":"\/posts\/system-design\/:49:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#50-design-an-online-multiplayer-game-backend-real-time"},{"categories":["software-engineering"],"collections":null,"content":"Overview Design a scalable backend for a real-time multiplayer game (like Fortnite, Among Us, or League of Legends) supporting matchmaking, game state synchronization, chat, leaderboards, and anti-cheat. ","date":"28-Jul-2025","heading":"Overview","objectID":"\/posts\/system-design\/:49:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#overview-4"},{"categories":["software-engineering"],"collections":null,"content":"Problem Requirements Functional Requirements: Player authentication and profiles Matchmaking system Real-time game state synchronization In-game chat Leaderboards and rankings Match history Anti-cheat detection Game replay system Non-Functional Requirements: 10 million daily active users 500K concurrent players Sub-50ms latency for game actions 99.99% uptime during matches Support 100-player battle royale matches Global distribution (multiple regions) Capacity Estimation: 500K concurrent \/ 100 players per match = 5,000 concurrent matches Game state updates: 20 updates\/sec \u00d7 100 players \u00d7 5,000 matches = 10M updates\/sec Bandwidth: 10M updates \u00d7 200 bytes = 2 GB\/sec ","date":"28-Jul-2025","heading":"Problem Requirements","objectID":"\/posts\/system-design\/:49:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#problem-requirements-4"},{"categories":["software-engineering"],"collections":null,"content":"High-Level Architecture Multiplayer Game Architecture \u2014 High-level design for real-time online multiplayer games with matchmaking and regional servers ","date":"28-Jul-2025","heading":"High-Level Architecture","objectID":"\/posts\/system-design\/:49:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#high-level-architecture-3"},{"categories":["software-engineering"],"collections":null,"content":"Detailed Game Server Architecture Game Server Detail \u2014 Internal game server design: game loop, physics, state sync, and Kafka event streaming ","date":"28-Jul-2025","heading":"Detailed Game Server Architecture","objectID":"\/posts\/system-design\/:49:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#detailed-game-server-architecture"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Matchmaking Service import asyncio import redis from dataclasses import dataclass from typing import List import heapq @dataclass class Player: player_id: str skill_rating: int # ELO or MMR region: str queue_time: float preferred_roles: List[str] class MatchmakingService: def __init__(self): self.redis = redis.Redis(host='localhost', port=6379) self.queue_key_prefix = \"matchmaking:queue:\" self.min_players = 10 # For 10-player matches self.max_skill_diff = 200 # Maximum MMR difference async def add_to_queue(self, player: Player): \"\"\"Add player to matchmaking queue.\"\"\" queue_key = f\"{self.queue_key_prefix}{player.region}\" # Add to Redis sorted set (score = skill_rating) self.redis.zadd(queue_key, { json.dumps({ 'player_id': player.player_id, 'skill_rating': player.skill_rating, 'queue_time': time.time(), 'preferred_roles': player.preferred_roles }): player.skill_rating }) # Start matchmaking check asyncio.create_task(self.find_matches(player.region)) async def find_matches(self, region): \"\"\"Find suitable matches from queue.\"\"\" queue_key = f\"{self.queue_key_prefix}{region}\" # Get all players in queue players_data = self.redis.zrange(queue_key, 0, -1, withscores=True) players = [json.loads(p[0]) for p in players_data] if len(players) < self.min_players: return # Not enough players # Sort by skill rating players.sort(key=lambda p: p['skill_rating']) # Try to create balanced matches matches = [] current_match = [] for player in players: if not current_match: current_match.append(player) continue # Check skill difference avg_skill = sum(p['skill_rating'] for p in current_match) \/ len(current_match) skill_diff = abs(player['skill_rating'] - avg_skill) # Check queue time (expand search range over time) queue_time = time.time() - player['queue_time'] max_diff = self.max_skill_diff + (queue_time * 10) # +10 MMR per second if skill_diff <= max_diff: current_match.append(player) if len(current_match) == self.min_players: matches.append(current_match) current_match = [] # Create game sessions for matches for match in matches: await self.create_game_session(match, region) # Remove matched players from queue for player in match: self.redis.zrem(queue_key, json.dumps(player)) async def create_game_session(self, players: List[dict], region: str): \"\"\"Allocate game server and start match.\"\"\" # Find available game server in region game_server = await self.allocate_game_server(region) # Create game session session_id = str(uuid.uuid4()) # Store session info in Redis self.redis.hmset(f\"game_session:{session_id}\", { 'session_id': session_id, 'server_ip': game_server['ip'], 'server_port': game_server['port'], 'region': region, 'players': json.dumps([p['player_id'] for p in players]), 'status': 'waiting', 'created_at': time.time() }) # Notify players (via WebSocket or push notification) for player in players: await self.notify_player(player['player_id'], { 'event': 'match_found', 'session_id': session_id, 'server_ip': game_server['ip'], 'server_port': game_server['port'] }) return session_id async def allocate_game_server(self, region): \"\"\"Find or spin up game server instance.\"\"\" # Check for idle servers idle_servers_key = f\"game_servers:idle:{region}\" server_data = self.redis.lpop(idle_servers_key) if server_data: return json.loads(server_data) # No idle servers - request new instance from orchestrator new_server = await self.orchestrator.provision_server(region) return new_server 2. Real-Time Game Server import asyncio import websockets from dataclasses import dataclass from typing import Dict, Set import numpy as np @dataclass class GameState: players: Dict[str, 'PlayerState'] projectiles: List['Projectile'] game_time: float tick_number: int @dataclass class PlayerState: player_id: str position: np.ndarray # [x, y, z] velocity: np.ndarray rotation: float health: int score: int is_alive: bool class GameServer: def __init__(self, session_id, max_players=100): self.session_id = session_id self.max_players = max_players s","date":"28-Jul-2025","heading":"Core Service Implementations","objectID":"\/posts\/system-design\/:49:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#core-service-implementations-2"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Matchmaking Service import asyncio import redis from dataclasses import dataclass from typing import List import heapq @dataclass class Player: player_id: str skill_rating: int # ELO or MMR region: str queue_time: float preferred_roles: List[str] class MatchmakingService: def __init__(self): self.redis = redis.Redis(host='localhost', port=6379) self.queue_key_prefix = \"matchmaking:queue:\" self.min_players = 10 # For 10-player matches self.max_skill_diff = 200 # Maximum MMR difference async def add_to_queue(self, player: Player): \"\"\"Add player to matchmaking queue.\"\"\" queue_key = f\"{self.queue_key_prefix}{player.region}\" # Add to Redis sorted set (score = skill_rating) self.redis.zadd(queue_key, { json.dumps({ 'player_id': player.player_id, 'skill_rating': player.skill_rating, 'queue_time': time.time(), 'preferred_roles': player.preferred_roles }): player.skill_rating }) # Start matchmaking check asyncio.create_task(self.find_matches(player.region)) async def find_matches(self, region): \"\"\"Find suitable matches from queue.\"\"\" queue_key = f\"{self.queue_key_prefix}{region}\" # Get all players in queue players_data = self.redis.zrange(queue_key, 0, -1, withscores=True) players = [json.loads(p[0]) for p in players_data] if len(players) < self.min_players: return # Not enough players # Sort by skill rating players.sort(key=lambda p: p['skill_rating']) # Try to create balanced matches matches = [] current_match = [] for player in players: if not current_match: current_match.append(player) continue # Check skill difference avg_skill = sum(p['skill_rating'] for p in current_match) \/ len(current_match) skill_diff = abs(player['skill_rating'] - avg_skill) # Check queue time (expand search range over time) queue_time = time.time() - player['queue_time'] max_diff = self.max_skill_diff + (queue_time * 10) # +10 MMR per second if skill_diff <= max_diff: current_match.append(player) if len(current_match) == self.min_players: matches.append(current_match) current_match = [] # Create game sessions for matches for match in matches: await self.create_game_session(match, region) # Remove matched players from queue for player in match: self.redis.zrem(queue_key, json.dumps(player)) async def create_game_session(self, players: List[dict], region: str): \"\"\"Allocate game server and start match.\"\"\" # Find available game server in region game_server = await self.allocate_game_server(region) # Create game session session_id = str(uuid.uuid4()) # Store session info in Redis self.redis.hmset(f\"game_session:{session_id}\", { 'session_id': session_id, 'server_ip': game_server['ip'], 'server_port': game_server['port'], 'region': region, 'players': json.dumps([p['player_id'] for p in players]), 'status': 'waiting', 'created_at': time.time() }) # Notify players (via WebSocket or push notification) for player in players: await self.notify_player(player['player_id'], { 'event': 'match_found', 'session_id': session_id, 'server_ip': game_server['ip'], 'server_port': game_server['port'] }) return session_id async def allocate_game_server(self, region): \"\"\"Find or spin up game server instance.\"\"\" # Check for idle servers idle_servers_key = f\"game_servers:idle:{region}\" server_data = self.redis.lpop(idle_servers_key) if server_data: return json.loads(server_data) # No idle servers - request new instance from orchestrator new_server = await self.orchestrator.provision_server(region) return new_server 2. Real-Time Game Server import asyncio import websockets from dataclasses import dataclass from typing import Dict, Set import numpy as np @dataclass class GameState: players: Dict[str, 'PlayerState'] projectiles: List['Projectile'] game_time: float tick_number: int @dataclass class PlayerState: player_id: str position: np.ndarray # [x, y, z] velocity: np.ndarray rotation: float health: int score: int is_alive: bool class GameServer: def __init__(self, session_id, max_players=100): self.session_id = session_id self.max_players = max_players s","date":"28-Jul-2025","heading":"1. Matchmaking Service","objectID":"\/posts\/system-design\/:49:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#1-matchmaking-service"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Matchmaking Service import asyncio import redis from dataclasses import dataclass from typing import List import heapq @dataclass class Player: player_id: str skill_rating: int # ELO or MMR region: str queue_time: float preferred_roles: List[str] class MatchmakingService: def __init__(self): self.redis = redis.Redis(host='localhost', port=6379) self.queue_key_prefix = \"matchmaking:queue:\" self.min_players = 10 # For 10-player matches self.max_skill_diff = 200 # Maximum MMR difference async def add_to_queue(self, player: Player): \"\"\"Add player to matchmaking queue.\"\"\" queue_key = f\"{self.queue_key_prefix}{player.region}\" # Add to Redis sorted set (score = skill_rating) self.redis.zadd(queue_key, { json.dumps({ 'player_id': player.player_id, 'skill_rating': player.skill_rating, 'queue_time': time.time(), 'preferred_roles': player.preferred_roles }): player.skill_rating }) # Start matchmaking check asyncio.create_task(self.find_matches(player.region)) async def find_matches(self, region): \"\"\"Find suitable matches from queue.\"\"\" queue_key = f\"{self.queue_key_prefix}{region}\" # Get all players in queue players_data = self.redis.zrange(queue_key, 0, -1, withscores=True) players = [json.loads(p[0]) for p in players_data] if len(players) < self.min_players: return # Not enough players # Sort by skill rating players.sort(key=lambda p: p['skill_rating']) # Try to create balanced matches matches = [] current_match = [] for player in players: if not current_match: current_match.append(player) continue # Check skill difference avg_skill = sum(p['skill_rating'] for p in current_match) \/ len(current_match) skill_diff = abs(player['skill_rating'] - avg_skill) # Check queue time (expand search range over time) queue_time = time.time() - player['queue_time'] max_diff = self.max_skill_diff + (queue_time * 10) # +10 MMR per second if skill_diff <= max_diff: current_match.append(player) if len(current_match) == self.min_players: matches.append(current_match) current_match = [] # Create game sessions for matches for match in matches: await self.create_game_session(match, region) # Remove matched players from queue for player in match: self.redis.zrem(queue_key, json.dumps(player)) async def create_game_session(self, players: List[dict], region: str): \"\"\"Allocate game server and start match.\"\"\" # Find available game server in region game_server = await self.allocate_game_server(region) # Create game session session_id = str(uuid.uuid4()) # Store session info in Redis self.redis.hmset(f\"game_session:{session_id}\", { 'session_id': session_id, 'server_ip': game_server['ip'], 'server_port': game_server['port'], 'region': region, 'players': json.dumps([p['player_id'] for p in players]), 'status': 'waiting', 'created_at': time.time() }) # Notify players (via WebSocket or push notification) for player in players: await self.notify_player(player['player_id'], { 'event': 'match_found', 'session_id': session_id, 'server_ip': game_server['ip'], 'server_port': game_server['port'] }) return session_id async def allocate_game_server(self, region): \"\"\"Find or spin up game server instance.\"\"\" # Check for idle servers idle_servers_key = f\"game_servers:idle:{region}\" server_data = self.redis.lpop(idle_servers_key) if server_data: return json.loads(server_data) # No idle servers - request new instance from orchestrator new_server = await self.orchestrator.provision_server(region) return new_server 2. Real-Time Game Server import asyncio import websockets from dataclasses import dataclass from typing import Dict, Set import numpy as np @dataclass class GameState: players: Dict[str, 'PlayerState'] projectiles: List['Projectile'] game_time: float tick_number: int @dataclass class PlayerState: player_id: str position: np.ndarray # [x, y, z] velocity: np.ndarray rotation: float health: int score: int is_alive: bool class GameServer: def __init__(self, session_id, max_players=100): self.session_id = session_id self.max_players = max_players s","date":"28-Jul-2025","heading":"2. Real-Time Game Server","objectID":"\/posts\/system-design\/:49:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#2-real-time-game-server"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Matchmaking Service import asyncio import redis from dataclasses import dataclass from typing import List import heapq @dataclass class Player: player_id: str skill_rating: int # ELO or MMR region: str queue_time: float preferred_roles: List[str] class MatchmakingService: def __init__(self): self.redis = redis.Redis(host='localhost', port=6379) self.queue_key_prefix = \"matchmaking:queue:\" self.min_players = 10 # For 10-player matches self.max_skill_diff = 200 # Maximum MMR difference async def add_to_queue(self, player: Player): \"\"\"Add player to matchmaking queue.\"\"\" queue_key = f\"{self.queue_key_prefix}{player.region}\" # Add to Redis sorted set (score = skill_rating) self.redis.zadd(queue_key, { json.dumps({ 'player_id': player.player_id, 'skill_rating': player.skill_rating, 'queue_time': time.time(), 'preferred_roles': player.preferred_roles }): player.skill_rating }) # Start matchmaking check asyncio.create_task(self.find_matches(player.region)) async def find_matches(self, region): \"\"\"Find suitable matches from queue.\"\"\" queue_key = f\"{self.queue_key_prefix}{region}\" # Get all players in queue players_data = self.redis.zrange(queue_key, 0, -1, withscores=True) players = [json.loads(p[0]) for p in players_data] if len(players) < self.min_players: return # Not enough players # Sort by skill rating players.sort(key=lambda p: p['skill_rating']) # Try to create balanced matches matches = [] current_match = [] for player in players: if not current_match: current_match.append(player) continue # Check skill difference avg_skill = sum(p['skill_rating'] for p in current_match) \/ len(current_match) skill_diff = abs(player['skill_rating'] - avg_skill) # Check queue time (expand search range over time) queue_time = time.time() - player['queue_time'] max_diff = self.max_skill_diff + (queue_time * 10) # +10 MMR per second if skill_diff <= max_diff: current_match.append(player) if len(current_match) == self.min_players: matches.append(current_match) current_match = [] # Create game sessions for matches for match in matches: await self.create_game_session(match, region) # Remove matched players from queue for player in match: self.redis.zrem(queue_key, json.dumps(player)) async def create_game_session(self, players: List[dict], region: str): \"\"\"Allocate game server and start match.\"\"\" # Find available game server in region game_server = await self.allocate_game_server(region) # Create game session session_id = str(uuid.uuid4()) # Store session info in Redis self.redis.hmset(f\"game_session:{session_id}\", { 'session_id': session_id, 'server_ip': game_server['ip'], 'server_port': game_server['port'], 'region': region, 'players': json.dumps([p['player_id'] for p in players]), 'status': 'waiting', 'created_at': time.time() }) # Notify players (via WebSocket or push notification) for player in players: await self.notify_player(player['player_id'], { 'event': 'match_found', 'session_id': session_id, 'server_ip': game_server['ip'], 'server_port': game_server['port'] }) return session_id async def allocate_game_server(self, region): \"\"\"Find or spin up game server instance.\"\"\" # Check for idle servers idle_servers_key = f\"game_servers:idle:{region}\" server_data = self.redis.lpop(idle_servers_key) if server_data: return json.loads(server_data) # No idle servers - request new instance from orchestrator new_server = await self.orchestrator.provision_server(region) return new_server 2. Real-Time Game Server import asyncio import websockets from dataclasses import dataclass from typing import Dict, Set import numpy as np @dataclass class GameState: players: Dict[str, 'PlayerState'] projectiles: List['Projectile'] game_time: float tick_number: int @dataclass class PlayerState: player_id: str position: np.ndarray # [x, y, z] velocity: np.ndarray rotation: float health: int score: int is_alive: bool class GameServer: def __init__(self, session_id, max_players=100): self.session_id = session_id self.max_players = max_players s","date":"28-Jul-2025","heading":"3. Leaderboard Service","objectID":"\/posts\/system-design\/:49:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#3-leaderboard-service"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Matchmaking Service import asyncio import redis from dataclasses import dataclass from typing import List import heapq @dataclass class Player: player_id: str skill_rating: int # ELO or MMR region: str queue_time: float preferred_roles: List[str] class MatchmakingService: def __init__(self): self.redis = redis.Redis(host='localhost', port=6379) self.queue_key_prefix = \"matchmaking:queue:\" self.min_players = 10 # For 10-player matches self.max_skill_diff = 200 # Maximum MMR difference async def add_to_queue(self, player: Player): \"\"\"Add player to matchmaking queue.\"\"\" queue_key = f\"{self.queue_key_prefix}{player.region}\" # Add to Redis sorted set (score = skill_rating) self.redis.zadd(queue_key, { json.dumps({ 'player_id': player.player_id, 'skill_rating': player.skill_rating, 'queue_time': time.time(), 'preferred_roles': player.preferred_roles }): player.skill_rating }) # Start matchmaking check asyncio.create_task(self.find_matches(player.region)) async def find_matches(self, region): \"\"\"Find suitable matches from queue.\"\"\" queue_key = f\"{self.queue_key_prefix}{region}\" # Get all players in queue players_data = self.redis.zrange(queue_key, 0, -1, withscores=True) players = [json.loads(p[0]) for p in players_data] if len(players) < self.min_players: return # Not enough players # Sort by skill rating players.sort(key=lambda p: p['skill_rating']) # Try to create balanced matches matches = [] current_match = [] for player in players: if not current_match: current_match.append(player) continue # Check skill difference avg_skill = sum(p['skill_rating'] for p in current_match) \/ len(current_match) skill_diff = abs(player['skill_rating'] - avg_skill) # Check queue time (expand search range over time) queue_time = time.time() - player['queue_time'] max_diff = self.max_skill_diff + (queue_time * 10) # +10 MMR per second if skill_diff <= max_diff: current_match.append(player) if len(current_match) == self.min_players: matches.append(current_match) current_match = [] # Create game sessions for matches for match in matches: await self.create_game_session(match, region) # Remove matched players from queue for player in match: self.redis.zrem(queue_key, json.dumps(player)) async def create_game_session(self, players: List[dict], region: str): \"\"\"Allocate game server and start match.\"\"\" # Find available game server in region game_server = await self.allocate_game_server(region) # Create game session session_id = str(uuid.uuid4()) # Store session info in Redis self.redis.hmset(f\"game_session:{session_id}\", { 'session_id': session_id, 'server_ip': game_server['ip'], 'server_port': game_server['port'], 'region': region, 'players': json.dumps([p['player_id'] for p in players]), 'status': 'waiting', 'created_at': time.time() }) # Notify players (via WebSocket or push notification) for player in players: await self.notify_player(player['player_id'], { 'event': 'match_found', 'session_id': session_id, 'server_ip': game_server['ip'], 'server_port': game_server['port'] }) return session_id async def allocate_game_server(self, region): \"\"\"Find or spin up game server instance.\"\"\" # Check for idle servers idle_servers_key = f\"game_servers:idle:{region}\" server_data = self.redis.lpop(idle_servers_key) if server_data: return json.loads(server_data) # No idle servers - request new instance from orchestrator new_server = await self.orchestrator.provision_server(region) return new_server 2. Real-Time Game Server import asyncio import websockets from dataclasses import dataclass from typing import Dict, Set import numpy as np @dataclass class GameState: players: Dict[str, 'PlayerState'] projectiles: List['Projectile'] game_time: float tick_number: int @dataclass class PlayerState: player_id: str position: np.ndarray # [x, y, z] velocity: np.ndarray rotation: float health: int score: int is_alive: bool class GameServer: def __init__(self, session_id, max_players=100): self.session_id = session_id self.max_players = max_players s","date":"28-Jul-2025","heading":"4. Anti-Cheat Service","objectID":"\/posts\/system-design\/:49:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#4-anti-cheat-service"},{"categories":["software-engineering"],"collections":null,"content":"Scalability Optimizations 1. Regional Game Servers: Deploy servers in multiple regions (US, EU, Asia) Route players to nearest region for low latency 2. Game Server Fleet Management: Kubernetes for orchestration Auto-scaling based on queue size Warm pool of idle servers 3. State Synchronization: UDP for low-latency game state (tolerate packet loss) TCP for critical actions (weapon purchases, match results) 4. Database Strategy: Cassandra for match history (write-heavy) Redis for real-time leaderboards PostgreSQL for player profiles ","date":"28-Jul-2025","heading":"Scalability Optimizations","objectID":"\/posts\/system-design\/:49:6","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#scalability-optimizations-2"},{"categories":["software-engineering"],"collections":null,"content":"Key Takeaways Fixed tick rate - Ensures consistent gameplay across clients Client-side prediction - Reduces perceived latency Server reconciliation - Prevents cheating with authoritative server UDP for game state - Lower latency than TCP Regional distribution - Critical for low-latency global gameplay Anti-cheat ML - Detect anomalous behavior patterns Redis sorted sets - Perfect for leaderboards ","date":"28-Jul-2025","heading":"Key Takeaways","objectID":"\/posts\/system-design\/:49:7","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#key-takeaways-4"},{"categories":["software-engineering"],"collections":null,"content":"51. Design a Collaborative Document Editor (Google Docs) ","date":"28-Jul-2025","heading":"51. Design a Collaborative Document Editor (Google Docs)","objectID":"\/posts\/system-design\/:50:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#51-design-a-collaborative-document-editor-google-docs"},{"categories":["software-engineering"],"collections":null,"content":"Overview Design a real-time collaborative document editor where multiple users can simultaneously edit the same document with conflict-free synchronization, version history, comments, and rich text formatting. ","date":"28-Jul-2025","heading":"Overview","objectID":"\/posts\/system-design\/:50:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#overview-5"},{"categories":["software-engineering"],"collections":null,"content":"Problem Requirements Functional Requirements: Real-time collaborative editing Rich text formatting (bold, italic, headings, lists, etc.) Document sharing and permissions Comments and suggestions Version history Auto-save Offline editing with sync Non-Functional Requirements: 100 million users 50 million documents Support 100 concurrent editors per document Sub-100ms latency for edits 99.99% availability Conflict-free synchronization Capacity Estimation: Storage: 50M documents \u00d7 100KB average = 5 TB Versions: 50M documents \u00d7 10 versions \u00d7 100KB = 50 TB Concurrent editors: 100K active documents \u00d7 10 editors = 1M WebSocket connections ","date":"28-Jul-2025","heading":"Problem Requirements","objectID":"\/posts\/system-design\/:50:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#problem-requirements-5"},{"categories":["software-engineering"],"collections":null,"content":"High-Level Architecture Collaborative Editor Architecture \u2014 WebSocket-based real-time synchronization for concurrent document editing with CRDT conflict resolution Core Services: Service Responsibility Key Technology Auth Service JWT token issuance and validation PostgreSQL + Redis sessions Document Service CRUD for documents, versions, metadata MongoDB (flexible schema for document content) Collaboration Service WebSocket server, CRDT operation broadcast Node.js + WebSockets + Redis pub\/sub Presence Service Track online users, cursor positions Redis (TTL-based) Comment Service Inline comments and suggestions PostgreSQL Notification Service Share\/mention notifications Kafka + FCM Export Service PDF\/DOCX export Background job workers Traffic Flow: User opens document \u2192 Auth Service validates JWT. Client establishes WebSocket to Collaboration Service (sticky session via consistent hash on doc_id). User types \u2192 local CRDT operation applied immediately \u2192 operation sent over WebSocket. Collaboration Service broadcasts to all other editors of same document. Document Service auto-saves every 30 seconds to MongoDB. Version snapshots taken on explicit save or every 100 operations. Scaling WebSockets: Each Collaboration Service node handles ~10K concurrent WebSocket connections. Redis Pub\/Sub used to relay operations between nodes (editor A on Node 1, editor B on Node 2 \u2014 both receive updates via Redis channel doc:<doc_id>). At 1M connections \u2192 ~100 Collaboration Service nodes. ","date":"28-Jul-2025","heading":"High-Level Architecture","objectID":"\/posts\/system-design\/:50:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#high-level-architecture-4"},{"categories":["software-engineering"],"collections":null,"content":"CRDT (Conflict-free Replicated Data Type) Architecture Eventual Consistency \u2014 Distributed system pattern balancing availability and consistency in CAP theorem ","date":"28-Jul-2025","heading":"CRDT (Conflict-free Replicated Data Type) Architecture","objectID":"\/posts\/system-design\/:50:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#crdt-conflict-free-replicated-data-type-architecture"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Operational Transformation (OT) \/ CRDT for Sync from dataclasses import dataclass from typing import List, Tuple import uuid @dataclass class Character: char: str char_id: str # Unique ID for each character site_id: str # Client\/user who created it position: List[int] # Fractional indexing position class CRDTDocument: \"\"\" CRDT-based document using fractional indexing. Enables conflict-free concurrent editing. \"\"\" def __init__(self, doc_id): self.doc_id = doc_id self.characters: List[Character] = [] self.tombstones = set() # Deleted character IDs self.version = 0 def local_insert(self, char, position, site_id): \"\"\"Insert character at position (local operation).\"\"\" # Generate unique fractional position frac_pos = self.generate_position(position) char_id = f\"{site_id}:{uuid.uuid4()}\" character = Character( char=char, char_id=char_id, site_id=site_id, position=frac_pos ) # Insert into sorted array self.characters.insert(position, character) self.version += 1 # Return operation to broadcast return { 'type': 'insert', 'char_id': char_id, 'char': char, 'position': frac_pos, 'site_id': site_id, 'version': self.version } def local_delete(self, position, site_id): \"\"\"Delete character at position (local operation).\"\"\" if position >= len(self.characters): return None character = self.characters[position] char_id = character.char_id # Mark as tombstone (don't actually delete for consistency) self.tombstones.add(char_id) self.characters.pop(position) self.version += 1 return { 'type': 'delete', 'char_id': char_id, 'site_id': site_id, 'version': self.version } def remote_insert(self, operation): \"\"\"Apply remote insert operation.\"\"\" char_id = operation['char_id'] # Check if already applied (idempotency) if any(c.char_id == char_id for c in self.characters): return False # Check if previously deleted if char_id in self.tombstones: return False character = Character( char=operation['char'], char_id=char_id, site_id=operation['site_id'], position=operation['position'] ) # Find insertion point based on fractional position insert_idx = self.find_insert_position(character.position) self.characters.insert(insert_idx, character) return True def remote_delete(self, operation): \"\"\"Apply remote delete operation.\"\"\" char_id = operation['char_id'] # Find and remove character for i, char in enumerate(self.characters): if char.char_id == char_id: self.characters.pop(i) self.tombstones.add(char_id) return True # Already deleted self.tombstones.add(char_id) return False def generate_position(self, index): \"\"\"Generate fractional position for insertion.\"\"\" if index == 0: if not self.characters: return [0.5] else: prev_pos = [0.0] next_pos = self.characters[0].position elif index >= len(self.characters): prev_pos = self.characters[-1].position next_pos = [1.0] else: prev_pos = self.characters[index - 1].position next_pos = self.characters[index].position # Generate position between prev and next return self.between(prev_pos, next_pos) def between(self, prev, next): \"\"\"Generate position between two fractional positions.\"\"\" # Simplified fractional indexing if len(prev) < len(next): prev = prev + [0] * (len(next) - len(prev)) elif len(next) < len(prev): next = next + [0] * (len(prev) - len(next)) result = [] for i in range(len(prev)): if prev[i] < next[i]: result.append((prev[i] + next[i]) \/ 2) break result.append(prev[i]) return result def to_string(self): \"\"\"Convert document to string.\"\"\" return ''.join(char.char for char in self.characters) def find_insert_position(self, position): \"\"\"Binary search for insertion position.\"\"\" left, right = 0, len(self.characters) while left < right: mid = (left + right) \/\/ 2 if self.characters[mid].position < position: left = mid + 1 else: right = mid return left 2. Collaboration Service import asyncio import websockets from collections import defaultdict class CollaborationService: def __init__(self): self.documents: Dict[str, CRDTDocument] = {} self.connections: Dict[str, Set[websockets.WebSocket","date":"28-Jul-2025","heading":"Core Service Implementations","objectID":"\/posts\/system-design\/:50:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#core-service-implementations-3"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Operational Transformation (OT) \/ CRDT for Sync from dataclasses import dataclass from typing import List, Tuple import uuid @dataclass class Character: char: str char_id: str # Unique ID for each character site_id: str # Client\/user who created it position: List[int] # Fractional indexing position class CRDTDocument: \"\"\" CRDT-based document using fractional indexing. Enables conflict-free concurrent editing. \"\"\" def __init__(self, doc_id): self.doc_id = doc_id self.characters: List[Character] = [] self.tombstones = set() # Deleted character IDs self.version = 0 def local_insert(self, char, position, site_id): \"\"\"Insert character at position (local operation).\"\"\" # Generate unique fractional position frac_pos = self.generate_position(position) char_id = f\"{site_id}:{uuid.uuid4()}\" character = Character( char=char, char_id=char_id, site_id=site_id, position=frac_pos ) # Insert into sorted array self.characters.insert(position, character) self.version += 1 # Return operation to broadcast return { 'type': 'insert', 'char_id': char_id, 'char': char, 'position': frac_pos, 'site_id': site_id, 'version': self.version } def local_delete(self, position, site_id): \"\"\"Delete character at position (local operation).\"\"\" if position >= len(self.characters): return None character = self.characters[position] char_id = character.char_id # Mark as tombstone (don't actually delete for consistency) self.tombstones.add(char_id) self.characters.pop(position) self.version += 1 return { 'type': 'delete', 'char_id': char_id, 'site_id': site_id, 'version': self.version } def remote_insert(self, operation): \"\"\"Apply remote insert operation.\"\"\" char_id = operation['char_id'] # Check if already applied (idempotency) if any(c.char_id == char_id for c in self.characters): return False # Check if previously deleted if char_id in self.tombstones: return False character = Character( char=operation['char'], char_id=char_id, site_id=operation['site_id'], position=operation['position'] ) # Find insertion point based on fractional position insert_idx = self.find_insert_position(character.position) self.characters.insert(insert_idx, character) return True def remote_delete(self, operation): \"\"\"Apply remote delete operation.\"\"\" char_id = operation['char_id'] # Find and remove character for i, char in enumerate(self.characters): if char.char_id == char_id: self.characters.pop(i) self.tombstones.add(char_id) return True # Already deleted self.tombstones.add(char_id) return False def generate_position(self, index): \"\"\"Generate fractional position for insertion.\"\"\" if index == 0: if not self.characters: return [0.5] else: prev_pos = [0.0] next_pos = self.characters[0].position elif index >= len(self.characters): prev_pos = self.characters[-1].position next_pos = [1.0] else: prev_pos = self.characters[index - 1].position next_pos = self.characters[index].position # Generate position between prev and next return self.between(prev_pos, next_pos) def between(self, prev, next): \"\"\"Generate position between two fractional positions.\"\"\" # Simplified fractional indexing if len(prev) < len(next): prev = prev + [0] * (len(next) - len(prev)) elif len(next) < len(prev): next = next + [0] * (len(prev) - len(next)) result = [] for i in range(len(prev)): if prev[i] < next[i]: result.append((prev[i] + next[i]) \/ 2) break result.append(prev[i]) return result def to_string(self): \"\"\"Convert document to string.\"\"\" return ''.join(char.char for char in self.characters) def find_insert_position(self, position): \"\"\"Binary search for insertion position.\"\"\" left, right = 0, len(self.characters) while left < right: mid = (left + right) \/\/ 2 if self.characters[mid].position < position: left = mid + 1 else: right = mid return left 2. Collaboration Service import asyncio import websockets from collections import defaultdict class CollaborationService: def __init__(self): self.documents: Dict[str, CRDTDocument] = {} self.connections: Dict[str, Set[websockets.WebSocket","date":"28-Jul-2025","heading":"1. Operational Transformation (OT) \/ CRDT for Sync","objectID":"\/posts\/system-design\/:50:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#1-operational-transformation-ot--crdt-for-sync"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Operational Transformation (OT) \/ CRDT for Sync from dataclasses import dataclass from typing import List, Tuple import uuid @dataclass class Character: char: str char_id: str # Unique ID for each character site_id: str # Client\/user who created it position: List[int] # Fractional indexing position class CRDTDocument: \"\"\" CRDT-based document using fractional indexing. Enables conflict-free concurrent editing. \"\"\" def __init__(self, doc_id): self.doc_id = doc_id self.characters: List[Character] = [] self.tombstones = set() # Deleted character IDs self.version = 0 def local_insert(self, char, position, site_id): \"\"\"Insert character at position (local operation).\"\"\" # Generate unique fractional position frac_pos = self.generate_position(position) char_id = f\"{site_id}:{uuid.uuid4()}\" character = Character( char=char, char_id=char_id, site_id=site_id, position=frac_pos ) # Insert into sorted array self.characters.insert(position, character) self.version += 1 # Return operation to broadcast return { 'type': 'insert', 'char_id': char_id, 'char': char, 'position': frac_pos, 'site_id': site_id, 'version': self.version } def local_delete(self, position, site_id): \"\"\"Delete character at position (local operation).\"\"\" if position >= len(self.characters): return None character = self.characters[position] char_id = character.char_id # Mark as tombstone (don't actually delete for consistency) self.tombstones.add(char_id) self.characters.pop(position) self.version += 1 return { 'type': 'delete', 'char_id': char_id, 'site_id': site_id, 'version': self.version } def remote_insert(self, operation): \"\"\"Apply remote insert operation.\"\"\" char_id = operation['char_id'] # Check if already applied (idempotency) if any(c.char_id == char_id for c in self.characters): return False # Check if previously deleted if char_id in self.tombstones: return False character = Character( char=operation['char'], char_id=char_id, site_id=operation['site_id'], position=operation['position'] ) # Find insertion point based on fractional position insert_idx = self.find_insert_position(character.position) self.characters.insert(insert_idx, character) return True def remote_delete(self, operation): \"\"\"Apply remote delete operation.\"\"\" char_id = operation['char_id'] # Find and remove character for i, char in enumerate(self.characters): if char.char_id == char_id: self.characters.pop(i) self.tombstones.add(char_id) return True # Already deleted self.tombstones.add(char_id) return False def generate_position(self, index): \"\"\"Generate fractional position for insertion.\"\"\" if index == 0: if not self.characters: return [0.5] else: prev_pos = [0.0] next_pos = self.characters[0].position elif index >= len(self.characters): prev_pos = self.characters[-1].position next_pos = [1.0] else: prev_pos = self.characters[index - 1].position next_pos = self.characters[index].position # Generate position between prev and next return self.between(prev_pos, next_pos) def between(self, prev, next): \"\"\"Generate position between two fractional positions.\"\"\" # Simplified fractional indexing if len(prev) < len(next): prev = prev + [0] * (len(next) - len(prev)) elif len(next) < len(prev): next = next + [0] * (len(prev) - len(next)) result = [] for i in range(len(prev)): if prev[i] < next[i]: result.append((prev[i] + next[i]) \/ 2) break result.append(prev[i]) return result def to_string(self): \"\"\"Convert document to string.\"\"\" return ''.join(char.char for char in self.characters) def find_insert_position(self, position): \"\"\"Binary search for insertion position.\"\"\" left, right = 0, len(self.characters) while left < right: mid = (left + right) \/\/ 2 if self.characters[mid].position < position: left = mid + 1 else: right = mid return left 2. Collaboration Service import asyncio import websockets from collections import defaultdict class CollaborationService: def __init__(self): self.documents: Dict[str, CRDTDocument] = {} self.connections: Dict[str, Set[websockets.WebSocket","date":"28-Jul-2025","heading":"2. Collaboration Service","objectID":"\/posts\/system-design\/:50:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#2-collaboration-service"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Operational Transformation (OT) \/ CRDT for Sync from dataclasses import dataclass from typing import List, Tuple import uuid @dataclass class Character: char: str char_id: str # Unique ID for each character site_id: str # Client\/user who created it position: List[int] # Fractional indexing position class CRDTDocument: \"\"\" CRDT-based document using fractional indexing. Enables conflict-free concurrent editing. \"\"\" def __init__(self, doc_id): self.doc_id = doc_id self.characters: List[Character] = [] self.tombstones = set() # Deleted character IDs self.version = 0 def local_insert(self, char, position, site_id): \"\"\"Insert character at position (local operation).\"\"\" # Generate unique fractional position frac_pos = self.generate_position(position) char_id = f\"{site_id}:{uuid.uuid4()}\" character = Character( char=char, char_id=char_id, site_id=site_id, position=frac_pos ) # Insert into sorted array self.characters.insert(position, character) self.version += 1 # Return operation to broadcast return { 'type': 'insert', 'char_id': char_id, 'char': char, 'position': frac_pos, 'site_id': site_id, 'version': self.version } def local_delete(self, position, site_id): \"\"\"Delete character at position (local operation).\"\"\" if position >= len(self.characters): return None character = self.characters[position] char_id = character.char_id # Mark as tombstone (don't actually delete for consistency) self.tombstones.add(char_id) self.characters.pop(position) self.version += 1 return { 'type': 'delete', 'char_id': char_id, 'site_id': site_id, 'version': self.version } def remote_insert(self, operation): \"\"\"Apply remote insert operation.\"\"\" char_id = operation['char_id'] # Check if already applied (idempotency) if any(c.char_id == char_id for c in self.characters): return False # Check if previously deleted if char_id in self.tombstones: return False character = Character( char=operation['char'], char_id=char_id, site_id=operation['site_id'], position=operation['position'] ) # Find insertion point based on fractional position insert_idx = self.find_insert_position(character.position) self.characters.insert(insert_idx, character) return True def remote_delete(self, operation): \"\"\"Apply remote delete operation.\"\"\" char_id = operation['char_id'] # Find and remove character for i, char in enumerate(self.characters): if char.char_id == char_id: self.characters.pop(i) self.tombstones.add(char_id) return True # Already deleted self.tombstones.add(char_id) return False def generate_position(self, index): \"\"\"Generate fractional position for insertion.\"\"\" if index == 0: if not self.characters: return [0.5] else: prev_pos = [0.0] next_pos = self.characters[0].position elif index >= len(self.characters): prev_pos = self.characters[-1].position next_pos = [1.0] else: prev_pos = self.characters[index - 1].position next_pos = self.characters[index].position # Generate position between prev and next return self.between(prev_pos, next_pos) def between(self, prev, next): \"\"\"Generate position between two fractional positions.\"\"\" # Simplified fractional indexing if len(prev) < len(next): prev = prev + [0] * (len(next) - len(prev)) elif len(next) < len(prev): next = next + [0] * (len(prev) - len(next)) result = [] for i in range(len(prev)): if prev[i] < next[i]: result.append((prev[i] + next[i]) \/ 2) break result.append(prev[i]) return result def to_string(self): \"\"\"Convert document to string.\"\"\" return ''.join(char.char for char in self.characters) def find_insert_position(self, position): \"\"\"Binary search for insertion position.\"\"\" left, right = 0, len(self.characters) while left < right: mid = (left + right) \/\/ 2 if self.characters[mid].position < position: left = mid + 1 else: right = mid return left 2. Collaboration Service import asyncio import websockets from collections import defaultdict class CollaborationService: def __init__(self): self.documents: Dict[str, CRDTDocument] = {} self.connections: Dict[str, Set[websockets.WebSocket","date":"28-Jul-2025","heading":"3. Document Service","objectID":"\/posts\/system-design\/:50:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#3-document-service"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Operational Transformation (OT) \/ CRDT for Sync from dataclasses import dataclass from typing import List, Tuple import uuid @dataclass class Character: char: str char_id: str # Unique ID for each character site_id: str # Client\/user who created it position: List[int] # Fractional indexing position class CRDTDocument: \"\"\" CRDT-based document using fractional indexing. Enables conflict-free concurrent editing. \"\"\" def __init__(self, doc_id): self.doc_id = doc_id self.characters: List[Character] = [] self.tombstones = set() # Deleted character IDs self.version = 0 def local_insert(self, char, position, site_id): \"\"\"Insert character at position (local operation).\"\"\" # Generate unique fractional position frac_pos = self.generate_position(position) char_id = f\"{site_id}:{uuid.uuid4()}\" character = Character( char=char, char_id=char_id, site_id=site_id, position=frac_pos ) # Insert into sorted array self.characters.insert(position, character) self.version += 1 # Return operation to broadcast return { 'type': 'insert', 'char_id': char_id, 'char': char, 'position': frac_pos, 'site_id': site_id, 'version': self.version } def local_delete(self, position, site_id): \"\"\"Delete character at position (local operation).\"\"\" if position >= len(self.characters): return None character = self.characters[position] char_id = character.char_id # Mark as tombstone (don't actually delete for consistency) self.tombstones.add(char_id) self.characters.pop(position) self.version += 1 return { 'type': 'delete', 'char_id': char_id, 'site_id': site_id, 'version': self.version } def remote_insert(self, operation): \"\"\"Apply remote insert operation.\"\"\" char_id = operation['char_id'] # Check if already applied (idempotency) if any(c.char_id == char_id for c in self.characters): return False # Check if previously deleted if char_id in self.tombstones: return False character = Character( char=operation['char'], char_id=char_id, site_id=operation['site_id'], position=operation['position'] ) # Find insertion point based on fractional position insert_idx = self.find_insert_position(character.position) self.characters.insert(insert_idx, character) return True def remote_delete(self, operation): \"\"\"Apply remote delete operation.\"\"\" char_id = operation['char_id'] # Find and remove character for i, char in enumerate(self.characters): if char.char_id == char_id: self.characters.pop(i) self.tombstones.add(char_id) return True # Already deleted self.tombstones.add(char_id) return False def generate_position(self, index): \"\"\"Generate fractional position for insertion.\"\"\" if index == 0: if not self.characters: return [0.5] else: prev_pos = [0.0] next_pos = self.characters[0].position elif index >= len(self.characters): prev_pos = self.characters[-1].position next_pos = [1.0] else: prev_pos = self.characters[index - 1].position next_pos = self.characters[index].position # Generate position between prev and next return self.between(prev_pos, next_pos) def between(self, prev, next): \"\"\"Generate position between two fractional positions.\"\"\" # Simplified fractional indexing if len(prev) < len(next): prev = prev + [0] * (len(next) - len(prev)) elif len(next) < len(prev): next = next + [0] * (len(prev) - len(next)) result = [] for i in range(len(prev)): if prev[i] < next[i]: result.append((prev[i] + next[i]) \/ 2) break result.append(prev[i]) return result def to_string(self): \"\"\"Convert document to string.\"\"\" return ''.join(char.char for char in self.characters) def find_insert_position(self, position): \"\"\"Binary search for insertion position.\"\"\" left, right = 0, len(self.characters) while left < right: mid = (left + right) \/\/ 2 if self.characters[mid].position < position: left = mid + 1 else: right = mid return left 2. Collaboration Service import asyncio import websockets from collections import defaultdict class CollaborationService: def __init__(self): self.documents: Dict[str, CRDTDocument] = {} self.connections: Dict[str, Set[websockets.WebSocket","date":"28-Jul-2025","heading":"4. Permission Service","objectID":"\/posts\/system-design\/:50:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#4-permission-service"},{"categories":["software-engineering"],"collections":null,"content":"Client-Side Implementation (JavaScript) class CollaborativeEditor { constructor(docId, userId) { this.docId = docId; this.userId = userId; this.ws = null; this.crdt = new CRDTDocument(docId); this.editor = document.getElementById('editor'); this.connect(); } connect() { this.ws = new WebSocket(`wss:\/\/api.docs.com\/ws`); this.ws.onopen = () => { \/\/ Authenticate this.ws.send(JSON.stringify({ doc_id: this.docId, user_id: this.userId, token: localStorage.getItem('auth_token') })); }; this.ws.onmessage = (event) => { const message = JSON.parse(event.data); this.handleMessage(message); }; \/\/ Listen for local edits this.editor.addEventListener('input', (e) => { this.handleLocalEdit(e); }); } handleLocalEdit(event) { const operation = this.crdt.local_insert( event.data, this.editor.selectionStart, this.userId ); \/\/ Send to server this.ws.send(JSON.stringify(operation)); \/\/ Update UI optimistically \/\/ (already done by browser) } handleMessage(message) { if (message.type === 'insert') { \/\/ Apply remote insert if (this.crdt.remote_insert(message)) { \/\/ Update editor content this.updateEditor(); } } else if (message.type === 'delete') { if (this.crdt.remote_delete(message)) { this.updateEditor(); } } else if (message.type === 'cursor_update') { \/\/ Show other user's cursor this.showRemoteCursor(message.user_id, message.position); } } updateEditor() { const currentCursor = this.editor.selectionStart; this.editor.value = this.crdt.to_string(); \/\/ Restore cursor position this.editor.setSelectionRange(currentCursor, currentCursor); } } ","date":"28-Jul-2025","heading":"Client-Side Implementation (JavaScript)","objectID":"\/posts\/system-design\/:50:6","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#client-side-implementation-javascript"},{"categories":["software-engineering"],"collections":null,"content":"Scalability Optimizations 1. Connection Pooling: WebSocket gateway with connection pooling Multiple servers behind load balancer with sticky sessions 2. Document Sharding: Shard documents by doc_id Co-locate related data (document + operations) 3. Caching: Recent documents in Redis Version snapshots in S3 (cold storage) 4. Compression: Compress operations in transit Store compressed versions ","date":"28-Jul-2025","heading":"Scalability Optimizations","objectID":"\/posts\/system-design\/:50:7","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#scalability-optimizations-3"},{"categories":["software-engineering"],"collections":null,"content":"Key Takeaways CRDT for conflict resolution - Guarantees eventual consistency Fractional indexing - Enables efficient position tracking WebSockets for real-time - Bi-directional communication Debounced auto-save - Reduce database writes Version snapshots - S3 for cost-efficient storage Cursor broadcasting - Enhance collaboration awareness Permission hierarchy - Fine-grained access control ","date":"28-Jul-2025","heading":"Key Takeaways","objectID":"\/posts\/system-design\/:50:8","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#key-takeaways-5"},{"categories":["software-engineering"],"collections":null,"content":"52. Design an Online Food Delivery System (Uber Eats \/ DoorDash) ","date":"28-Jul-2025","heading":"52. Design an Online Food Delivery System (Uber Eats \/ DoorDash)","objectID":"\/posts\/system-design\/:51:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#52-design-an-online-food-delivery-system-uber-eats--doordash"},{"categories":["software-engineering"],"collections":null,"content":"Overview Design a scalable food delivery platform connecting restaurants, delivery partners, and customers with real-time order tracking, optimized delivery routing, and dynamic pricing. ","date":"28-Jul-2025","heading":"Overview","objectID":"\/posts\/system-design\/:51:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#overview-6"},{"categories":["software-engineering"],"collections":null,"content":"Problem Requirements Functional Requirements: Restaurant browsing and menu search Order placement and tracking Real-time delivery tracking Payment processing Ratings and reviews Push notifications Delivery optimization Non-Functional Requirements: 50 million users 500K restaurants 100K concurrent orders Sub-second search latency Real-time location tracking with < 5-second updates 99.9% availability Support 50 geographic regions Capacity Estimation: 100K concurrent orders \u00d7 50 states = 2M orders\/day Each order: Order details (1KB) + Location data (500B) = 1.5KB 2M orders \u00d7 1.5KB \u00d7 30 days = 90 GB\/month (metadata) Location updates: 100K active deliveries \u00d7 1 update\/5sec = 20K updates\/sec ","date":"28-Jul-2025","heading":"Problem Requirements","objectID":"\/posts\/system-design\/:51:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#problem-requirements-6"},{"categories":["software-engineering"],"collections":null,"content":"High-Level Architecture Food Delivery Architecture \u2014 Uber Eats\/DoorDash-style microservices: restaurant catalog, order management, real-time driver dispatch, and ETA service Core Services: Service Responsibility Key Technology Restaurant Service Menu catalog, availability, ratings PostgreSQL + Elasticsearch Order Service Order lifecycle (placed \u2192 picked up \u2192 delivered) PostgreSQL (ACID) + Kafka User Service Customer profiles, addresses, payment methods PostgreSQL Driver Service Driver onboarding, availability status, ratings PostgreSQL Matching Service Assign nearest available driver to order Redis Geo + Kafka Location Service Real-time GPS tracking (drivers), ETA Redis (TTL 30s) + WebSockets Payment Service Charge customer, pay restaurant + driver Stripe + scheduled payouts Notification Service Order status push notifications FCM\/APNs + WebSockets ETA Service Estimate delivery time based on distance + traffic Road network graph (Dijkstra \/ A*) Traffic Flow: Customer places order \u2192 Order Service creates order, charges payment, publishes order_placed to Kafka. Matching Service finds nearest available driver using geospatial query (Redis GEO or PostGIS). Driver accepts \u2192 location updates stream every 5 seconds via WebSocket \u2192 Location Service stores in Redis. Customer app subscribes to real-time order updates (WebSocket or SSE). On delivery \u2192 Order Service marks delivered, triggers payment disbursement to restaurant and driver. Key Design Challenges: Real-time location: 20K GPS updates\/sec \u2014 Redis GEOADD with 30-second TTL per driver. Driver matching: Must be fast (<200ms). Use geohash to narrow candidates, then rank by proximity + rating. Surge pricing: Adjust prices dynamically based on demand\/supply ratio per geo zone. Reliability: If driver goes offline mid-delivery, re-dispatch to another driver within SLA. ","date":"28-Jul-2025","heading":"High-Level Architecture","objectID":"\/posts\/system-design\/:51:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#high-level-architecture-5"},{"categories":["software-engineering"],"collections":null,"content":"Detailed Location & Delivery Service Architecture Uber Architecture \u2014 Real-time matching, routing, and geospatial system design for on-demand transportation ","date":"28-Jul-2025","heading":"Detailed Location & Delivery Service Architecture","objectID":"\/posts\/system-design\/:51:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#detailed-location--delivery-service-architecture"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Order Service from enum import Enum from datetime import datetime, timedelta class OrderStatus(Enum): PENDING = \"pending\" CONFIRMED = \"confirmed\" PREPARING = \"preparing\" READY = \"ready\" PICKED_UP = \"picked_up\" IN_TRANSIT = \"in_transit\" DELIVERED = \"delivered\" CANCELLED = \"cancelled\" class OrderService: def __init__(self): self.db = PostgreSQL() self.cache = redis.Redis() self.kafka_producer = KafkaProducer() self.payment_service = PaymentService() def create_order(self, customer_id, restaurant_id, items, delivery_address): \"\"\"Create new order.\"\"\" order_id = str(uuid.uuid4()) # Calculate order total total = self.calculate_total(restaurant_id, items) # Create order in database order = { 'id': order_id, 'customer_id': customer_id, 'restaurant_id': restaurant_id, 'items': items, 'status': OrderStatus.PENDING.value, 'subtotal': total['subtotal'], 'tax': total['tax'], 'delivery_fee': total['delivery_fee'], 'total': total['total'], 'delivery_address': delivery_address, 'created_at': datetime.now(), 'estimated_delivery': datetime.now() + timedelta(minutes=45) } self.db.execute(\"\"\" INSERT INTO orders (id, customer_id, restaurant_id, items, total, status, created_at) VALUES (%s, %s, %s, %s, %s, %s, %s) \"\"\", (order_id, customer_id, restaurant_id, json.dumps(items), total['total'], OrderStatus.PENDING.value, datetime.now())) # Process payment payment_result = self.payment_service.charge( customer_id, total['total'], order_id ) if not payment_result['success']: self.db.execute(\"UPDATE orders SET status = %s WHERE id = %s\", (OrderStatus.CANCELLED.value, order_id)) raise PaymentFailedError(\"Payment failed\") # Update status self.update_order_status(order_id, OrderStatus.CONFIRMED) # Publish event to Kafka self.kafka_producer.send('order-events', { 'event_type': 'order_created', 'order_id': order_id, 'restaurant_id': restaurant_id, 'total': total['total'], 'timestamp': datetime.now().isoformat() }) return order_id def calculate_total(self, restaurant_id, items): \"\"\"Calculate order total with taxes and fees.\"\"\" # Get menu items and prices prices = self.get_menu_prices(restaurant_id, [item['menu_item_id'] for item in items]) subtotal = sum( prices[item['menu_item_id']] * item['quantity'] for item in items ) # Tax tax_rate = 0.08 # 8% tax = subtotal * tax_rate # Delivery fee (base + distance-based) delivery_distance = self.get_delivery_distance(restaurant_id, None) # Will add address delivery_fee = 2.99 + (delivery_distance * 0.5) # Promotions\/discounts discount = self.apply_promotions(restaurant_id, customer_id, subtotal) total = subtotal + tax + delivery_fee - discount return { 'subtotal': subtotal, 'tax': tax, 'delivery_fee': delivery_fee, 'discount': discount, 'total': total } def update_order_status(self, order_id, new_status): \"\"\"Update order status and notify parties.\"\"\" old_status = self.get_order_status(order_id) if new_status == old_status: return self.db.execute(\"\"\" UPDATE orders SET status = %s, updated_at = NOW() WHERE id = %s \"\"\", (new_status.value, order_id)) # Cache invalidation self.cache.delete(f\"order:{order_id}\") # Publish status change event order = self.get_order(order_id) self.kafka_producer.send('order-status-updates', { 'order_id': order_id, 'status': new_status.value, 'customer_id': order['customer_id'], 'restaurant_id': order['restaurant_id'], 'timestamp': datetime.now().isoformat() }) # Send notifications asyncio.create_task(self.notify_status_change(order, new_status)) 2. Delivery Service with Real-Time Tracking import math class DeliveryService: def __init__(self): self.redis = redis.Redis() self.db = PostgreSQL() self.route_optimizer = RouteOptimizer() def assign_delivery(self, order_id, restaurant_location, delivery_address): \"\"\"Assign order to nearest available delivery partner.\"\"\" # Find nearby available delivery partners (within 5 km) partners = self.find_nearby_partners( restaurant_location, radius_km=5 ) if not partners: raise NoAvailableDeliveryError(\"No delivery partners avail","date":"28-Jul-2025","heading":"Core Service Implementations","objectID":"\/posts\/system-design\/:51:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#core-service-implementations-4"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Order Service from enum import Enum from datetime import datetime, timedelta class OrderStatus(Enum): PENDING = \"pending\" CONFIRMED = \"confirmed\" PREPARING = \"preparing\" READY = \"ready\" PICKED_UP = \"picked_up\" IN_TRANSIT = \"in_transit\" DELIVERED = \"delivered\" CANCELLED = \"cancelled\" class OrderService: def __init__(self): self.db = PostgreSQL() self.cache = redis.Redis() self.kafka_producer = KafkaProducer() self.payment_service = PaymentService() def create_order(self, customer_id, restaurant_id, items, delivery_address): \"\"\"Create new order.\"\"\" order_id = str(uuid.uuid4()) # Calculate order total total = self.calculate_total(restaurant_id, items) # Create order in database order = { 'id': order_id, 'customer_id': customer_id, 'restaurant_id': restaurant_id, 'items': items, 'status': OrderStatus.PENDING.value, 'subtotal': total['subtotal'], 'tax': total['tax'], 'delivery_fee': total['delivery_fee'], 'total': total['total'], 'delivery_address': delivery_address, 'created_at': datetime.now(), 'estimated_delivery': datetime.now() + timedelta(minutes=45) } self.db.execute(\"\"\" INSERT INTO orders (id, customer_id, restaurant_id, items, total, status, created_at) VALUES (%s, %s, %s, %s, %s, %s, %s) \"\"\", (order_id, customer_id, restaurant_id, json.dumps(items), total['total'], OrderStatus.PENDING.value, datetime.now())) # Process payment payment_result = self.payment_service.charge( customer_id, total['total'], order_id ) if not payment_result['success']: self.db.execute(\"UPDATE orders SET status = %s WHERE id = %s\", (OrderStatus.CANCELLED.value, order_id)) raise PaymentFailedError(\"Payment failed\") # Update status self.update_order_status(order_id, OrderStatus.CONFIRMED) # Publish event to Kafka self.kafka_producer.send('order-events', { 'event_type': 'order_created', 'order_id': order_id, 'restaurant_id': restaurant_id, 'total': total['total'], 'timestamp': datetime.now().isoformat() }) return order_id def calculate_total(self, restaurant_id, items): \"\"\"Calculate order total with taxes and fees.\"\"\" # Get menu items and prices prices = self.get_menu_prices(restaurant_id, [item['menu_item_id'] for item in items]) subtotal = sum( prices[item['menu_item_id']] * item['quantity'] for item in items ) # Tax tax_rate = 0.08 # 8% tax = subtotal * tax_rate # Delivery fee (base + distance-based) delivery_distance = self.get_delivery_distance(restaurant_id, None) # Will add address delivery_fee = 2.99 + (delivery_distance * 0.5) # Promotions\/discounts discount = self.apply_promotions(restaurant_id, customer_id, subtotal) total = subtotal + tax + delivery_fee - discount return { 'subtotal': subtotal, 'tax': tax, 'delivery_fee': delivery_fee, 'discount': discount, 'total': total } def update_order_status(self, order_id, new_status): \"\"\"Update order status and notify parties.\"\"\" old_status = self.get_order_status(order_id) if new_status == old_status: return self.db.execute(\"\"\" UPDATE orders SET status = %s, updated_at = NOW() WHERE id = %s \"\"\", (new_status.value, order_id)) # Cache invalidation self.cache.delete(f\"order:{order_id}\") # Publish status change event order = self.get_order(order_id) self.kafka_producer.send('order-status-updates', { 'order_id': order_id, 'status': new_status.value, 'customer_id': order['customer_id'], 'restaurant_id': order['restaurant_id'], 'timestamp': datetime.now().isoformat() }) # Send notifications asyncio.create_task(self.notify_status_change(order, new_status)) 2. Delivery Service with Real-Time Tracking import math class DeliveryService: def __init__(self): self.redis = redis.Redis() self.db = PostgreSQL() self.route_optimizer = RouteOptimizer() def assign_delivery(self, order_id, restaurant_location, delivery_address): \"\"\"Assign order to nearest available delivery partner.\"\"\" # Find nearby available delivery partners (within 5 km) partners = self.find_nearby_partners( restaurant_location, radius_km=5 ) if not partners: raise NoAvailableDeliveryError(\"No delivery partners avail","date":"28-Jul-2025","heading":"1. Order Service","objectID":"\/posts\/system-design\/:51:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#1-order-service"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Order Service from enum import Enum from datetime import datetime, timedelta class OrderStatus(Enum): PENDING = \"pending\" CONFIRMED = \"confirmed\" PREPARING = \"preparing\" READY = \"ready\" PICKED_UP = \"picked_up\" IN_TRANSIT = \"in_transit\" DELIVERED = \"delivered\" CANCELLED = \"cancelled\" class OrderService: def __init__(self): self.db = PostgreSQL() self.cache = redis.Redis() self.kafka_producer = KafkaProducer() self.payment_service = PaymentService() def create_order(self, customer_id, restaurant_id, items, delivery_address): \"\"\"Create new order.\"\"\" order_id = str(uuid.uuid4()) # Calculate order total total = self.calculate_total(restaurant_id, items) # Create order in database order = { 'id': order_id, 'customer_id': customer_id, 'restaurant_id': restaurant_id, 'items': items, 'status': OrderStatus.PENDING.value, 'subtotal': total['subtotal'], 'tax': total['tax'], 'delivery_fee': total['delivery_fee'], 'total': total['total'], 'delivery_address': delivery_address, 'created_at': datetime.now(), 'estimated_delivery': datetime.now() + timedelta(minutes=45) } self.db.execute(\"\"\" INSERT INTO orders (id, customer_id, restaurant_id, items, total, status, created_at) VALUES (%s, %s, %s, %s, %s, %s, %s) \"\"\", (order_id, customer_id, restaurant_id, json.dumps(items), total['total'], OrderStatus.PENDING.value, datetime.now())) # Process payment payment_result = self.payment_service.charge( customer_id, total['total'], order_id ) if not payment_result['success']: self.db.execute(\"UPDATE orders SET status = %s WHERE id = %s\", (OrderStatus.CANCELLED.value, order_id)) raise PaymentFailedError(\"Payment failed\") # Update status self.update_order_status(order_id, OrderStatus.CONFIRMED) # Publish event to Kafka self.kafka_producer.send('order-events', { 'event_type': 'order_created', 'order_id': order_id, 'restaurant_id': restaurant_id, 'total': total['total'], 'timestamp': datetime.now().isoformat() }) return order_id def calculate_total(self, restaurant_id, items): \"\"\"Calculate order total with taxes and fees.\"\"\" # Get menu items and prices prices = self.get_menu_prices(restaurant_id, [item['menu_item_id'] for item in items]) subtotal = sum( prices[item['menu_item_id']] * item['quantity'] for item in items ) # Tax tax_rate = 0.08 # 8% tax = subtotal * tax_rate # Delivery fee (base + distance-based) delivery_distance = self.get_delivery_distance(restaurant_id, None) # Will add address delivery_fee = 2.99 + (delivery_distance * 0.5) # Promotions\/discounts discount = self.apply_promotions(restaurant_id, customer_id, subtotal) total = subtotal + tax + delivery_fee - discount return { 'subtotal': subtotal, 'tax': tax, 'delivery_fee': delivery_fee, 'discount': discount, 'total': total } def update_order_status(self, order_id, new_status): \"\"\"Update order status and notify parties.\"\"\" old_status = self.get_order_status(order_id) if new_status == old_status: return self.db.execute(\"\"\" UPDATE orders SET status = %s, updated_at = NOW() WHERE id = %s \"\"\", (new_status.value, order_id)) # Cache invalidation self.cache.delete(f\"order:{order_id}\") # Publish status change event order = self.get_order(order_id) self.kafka_producer.send('order-status-updates', { 'order_id': order_id, 'status': new_status.value, 'customer_id': order['customer_id'], 'restaurant_id': order['restaurant_id'], 'timestamp': datetime.now().isoformat() }) # Send notifications asyncio.create_task(self.notify_status_change(order, new_status)) 2. Delivery Service with Real-Time Tracking import math class DeliveryService: def __init__(self): self.redis = redis.Redis() self.db = PostgreSQL() self.route_optimizer = RouteOptimizer() def assign_delivery(self, order_id, restaurant_location, delivery_address): \"\"\"Assign order to nearest available delivery partner.\"\"\" # Find nearby available delivery partners (within 5 km) partners = self.find_nearby_partners( restaurant_location, radius_km=5 ) if not partners: raise NoAvailableDeliveryError(\"No delivery partners avail","date":"28-Jul-2025","heading":"2. Delivery Service with Real-Time Tracking","objectID":"\/posts\/system-design\/:51:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#2-delivery-service-with-real-time-tracking"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Order Service from enum import Enum from datetime import datetime, timedelta class OrderStatus(Enum): PENDING = \"pending\" CONFIRMED = \"confirmed\" PREPARING = \"preparing\" READY = \"ready\" PICKED_UP = \"picked_up\" IN_TRANSIT = \"in_transit\" DELIVERED = \"delivered\" CANCELLED = \"cancelled\" class OrderService: def __init__(self): self.db = PostgreSQL() self.cache = redis.Redis() self.kafka_producer = KafkaProducer() self.payment_service = PaymentService() def create_order(self, customer_id, restaurant_id, items, delivery_address): \"\"\"Create new order.\"\"\" order_id = str(uuid.uuid4()) # Calculate order total total = self.calculate_total(restaurant_id, items) # Create order in database order = { 'id': order_id, 'customer_id': customer_id, 'restaurant_id': restaurant_id, 'items': items, 'status': OrderStatus.PENDING.value, 'subtotal': total['subtotal'], 'tax': total['tax'], 'delivery_fee': total['delivery_fee'], 'total': total['total'], 'delivery_address': delivery_address, 'created_at': datetime.now(), 'estimated_delivery': datetime.now() + timedelta(minutes=45) } self.db.execute(\"\"\" INSERT INTO orders (id, customer_id, restaurant_id, items, total, status, created_at) VALUES (%s, %s, %s, %s, %s, %s, %s) \"\"\", (order_id, customer_id, restaurant_id, json.dumps(items), total['total'], OrderStatus.PENDING.value, datetime.now())) # Process payment payment_result = self.payment_service.charge( customer_id, total['total'], order_id ) if not payment_result['success']: self.db.execute(\"UPDATE orders SET status = %s WHERE id = %s\", (OrderStatus.CANCELLED.value, order_id)) raise PaymentFailedError(\"Payment failed\") # Update status self.update_order_status(order_id, OrderStatus.CONFIRMED) # Publish event to Kafka self.kafka_producer.send('order-events', { 'event_type': 'order_created', 'order_id': order_id, 'restaurant_id': restaurant_id, 'total': total['total'], 'timestamp': datetime.now().isoformat() }) return order_id def calculate_total(self, restaurant_id, items): \"\"\"Calculate order total with taxes and fees.\"\"\" # Get menu items and prices prices = self.get_menu_prices(restaurant_id, [item['menu_item_id'] for item in items]) subtotal = sum( prices[item['menu_item_id']] * item['quantity'] for item in items ) # Tax tax_rate = 0.08 # 8% tax = subtotal * tax_rate # Delivery fee (base + distance-based) delivery_distance = self.get_delivery_distance(restaurant_id, None) # Will add address delivery_fee = 2.99 + (delivery_distance * 0.5) # Promotions\/discounts discount = self.apply_promotions(restaurant_id, customer_id, subtotal) total = subtotal + tax + delivery_fee - discount return { 'subtotal': subtotal, 'tax': tax, 'delivery_fee': delivery_fee, 'discount': discount, 'total': total } def update_order_status(self, order_id, new_status): \"\"\"Update order status and notify parties.\"\"\" old_status = self.get_order_status(order_id) if new_status == old_status: return self.db.execute(\"\"\" UPDATE orders SET status = %s, updated_at = NOW() WHERE id = %s \"\"\", (new_status.value, order_id)) # Cache invalidation self.cache.delete(f\"order:{order_id}\") # Publish status change event order = self.get_order(order_id) self.kafka_producer.send('order-status-updates', { 'order_id': order_id, 'status': new_status.value, 'customer_id': order['customer_id'], 'restaurant_id': order['restaurant_id'], 'timestamp': datetime.now().isoformat() }) # Send notifications asyncio.create_task(self.notify_status_change(order, new_status)) 2. Delivery Service with Real-Time Tracking import math class DeliveryService: def __init__(self): self.redis = redis.Redis() self.db = PostgreSQL() self.route_optimizer = RouteOptimizer() def assign_delivery(self, order_id, restaurant_location, delivery_address): \"\"\"Assign order to nearest available delivery partner.\"\"\" # Find nearby available delivery partners (within 5 km) partners = self.find_nearby_partners( restaurant_location, radius_km=5 ) if not partners: raise NoAvailableDeliveryError(\"No delivery partners avail","date":"28-Jul-2025","heading":"3. Search Service with Filtering","objectID":"\/posts\/system-design\/:51:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#3-search-service-with-filtering"},{"categories":["software-engineering"],"collections":null,"content":"Core Service Implementations 1. Order Service from enum import Enum from datetime import datetime, timedelta class OrderStatus(Enum): PENDING = \"pending\" CONFIRMED = \"confirmed\" PREPARING = \"preparing\" READY = \"ready\" PICKED_UP = \"picked_up\" IN_TRANSIT = \"in_transit\" DELIVERED = \"delivered\" CANCELLED = \"cancelled\" class OrderService: def __init__(self): self.db = PostgreSQL() self.cache = redis.Redis() self.kafka_producer = KafkaProducer() self.payment_service = PaymentService() def create_order(self, customer_id, restaurant_id, items, delivery_address): \"\"\"Create new order.\"\"\" order_id = str(uuid.uuid4()) # Calculate order total total = self.calculate_total(restaurant_id, items) # Create order in database order = { 'id': order_id, 'customer_id': customer_id, 'restaurant_id': restaurant_id, 'items': items, 'status': OrderStatus.PENDING.value, 'subtotal': total['subtotal'], 'tax': total['tax'], 'delivery_fee': total['delivery_fee'], 'total': total['total'], 'delivery_address': delivery_address, 'created_at': datetime.now(), 'estimated_delivery': datetime.now() + timedelta(minutes=45) } self.db.execute(\"\"\" INSERT INTO orders (id, customer_id, restaurant_id, items, total, status, created_at) VALUES (%s, %s, %s, %s, %s, %s, %s) \"\"\", (order_id, customer_id, restaurant_id, json.dumps(items), total['total'], OrderStatus.PENDING.value, datetime.now())) # Process payment payment_result = self.payment_service.charge( customer_id, total['total'], order_id ) if not payment_result['success']: self.db.execute(\"UPDATE orders SET status = %s WHERE id = %s\", (OrderStatus.CANCELLED.value, order_id)) raise PaymentFailedError(\"Payment failed\") # Update status self.update_order_status(order_id, OrderStatus.CONFIRMED) # Publish event to Kafka self.kafka_producer.send('order-events', { 'event_type': 'order_created', 'order_id': order_id, 'restaurant_id': restaurant_id, 'total': total['total'], 'timestamp': datetime.now().isoformat() }) return order_id def calculate_total(self, restaurant_id, items): \"\"\"Calculate order total with taxes and fees.\"\"\" # Get menu items and prices prices = self.get_menu_prices(restaurant_id, [item['menu_item_id'] for item in items]) subtotal = sum( prices[item['menu_item_id']] * item['quantity'] for item in items ) # Tax tax_rate = 0.08 # 8% tax = subtotal * tax_rate # Delivery fee (base + distance-based) delivery_distance = self.get_delivery_distance(restaurant_id, None) # Will add address delivery_fee = 2.99 + (delivery_distance * 0.5) # Promotions\/discounts discount = self.apply_promotions(restaurant_id, customer_id, subtotal) total = subtotal + tax + delivery_fee - discount return { 'subtotal': subtotal, 'tax': tax, 'delivery_fee': delivery_fee, 'discount': discount, 'total': total } def update_order_status(self, order_id, new_status): \"\"\"Update order status and notify parties.\"\"\" old_status = self.get_order_status(order_id) if new_status == old_status: return self.db.execute(\"\"\" UPDATE orders SET status = %s, updated_at = NOW() WHERE id = %s \"\"\", (new_status.value, order_id)) # Cache invalidation self.cache.delete(f\"order:{order_id}\") # Publish status change event order = self.get_order(order_id) self.kafka_producer.send('order-status-updates', { 'order_id': order_id, 'status': new_status.value, 'customer_id': order['customer_id'], 'restaurant_id': order['restaurant_id'], 'timestamp': datetime.now().isoformat() }) # Send notifications asyncio.create_task(self.notify_status_change(order, new_status)) 2. Delivery Service with Real-Time Tracking import math class DeliveryService: def __init__(self): self.redis = redis.Redis() self.db = PostgreSQL() self.route_optimizer = RouteOptimizer() def assign_delivery(self, order_id, restaurant_location, delivery_address): \"\"\"Assign order to nearest available delivery partner.\"\"\" # Find nearby available delivery partners (within 5 km) partners = self.find_nearby_partners( restaurant_location, radius_km=5 ) if not partners: raise NoAvailableDeliveryError(\"No delivery partners avail","date":"28-Jul-2025","heading":"4. Dynamic Pricing Service","objectID":"\/posts\/system-design\/:51:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#4-dynamic-pricing-service"},{"categories":["software-engineering"],"collections":null,"content":"Scalability Optimizations 1. Geographic Sharding: Shard data by region\/city Deploy services in each region for low latency 2. Redis Geospatial Index: Track all active delivery partners and their locations Fast nearest-neighbor queries 3. Event-Driven Architecture: Kafka for order events Asynchronous updates to analytics, ratings, etc. 4. Caching Strategy: Restaurant data: 1-hour TTL Menu data: 2-hour TTL Delivery locations: Real-time (Redis) 5. Database Optimization: Orders: PostgreSQL (strong consistency needed) Delivery history: Cassandra (time-series data) Locations: Redis (real-time) ","date":"28-Jul-2025","heading":"Scalability Optimizations","objectID":"\/posts\/system-design\/:51:6","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#scalability-optimizations-4"},{"categories":["software-engineering"],"collections":null,"content":"Key Takeaways Geospatial indexing - Critical for location-based matching Real-time tracking - WebSocket for live delivery updates Dynamic pricing - ML-based demand estimation Intelligent assignment - Score-based delivery partner matching Regional deployment - Minimize latency for critical operations Event sourcing - Track all state changes for auditing Eventual consistency - Between regions is acceptable ","date":"28-Jul-2025","heading":"Key Takeaways","objectID":"\/posts\/system-design\/:51:7","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#key-takeaways-6"},{"categories":["software-engineering"],"collections":null,"content":"53. Advanced System Design Topics ","date":"28-Jul-2025","heading":"53. Advanced System Design Topics","objectID":"\/posts\/system-design\/:52:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#53-advanced-system-design-topics"},{"categories":["software-engineering"],"collections":null,"content":"Testing Distributed Systems Challenges in Testing Distributed Systems: Non-deterministic behavior Network partitions Race conditions Timing issues Testing Strategies: # Example: Chaos Engineering Test class ChaosTest: def __init__(self, system): self.system = system def test_network_partition(self): \"\"\"Simulate network partition between nodes.\"\"\" # Isolate node from cluster self.system.isolate_node('node-2') # Verify system continues operating assert self.system.is_available() # Restore network self.system.restore_node('node-2') # Verify data consistency assert self.system.verify_consistency() def test_node_failure(self): \"\"\"Test system behavior when node crashes.\"\"\" self.system.kill_node('node-1') # Verify failover assert self.system.leader_elected() assert self.system.is_available() def test_latency_injection(self): \"\"\"Inject random latencies to find race conditions.\"\"\" self.system.inject_latency(min_ms=100, max_ms=500) # Run concurrent operations results = self.system.concurrent_writes(count=1000) # Verify eventual consistency self.system.wait_for_convergence() assert self.system.verify_consistency() Tools: Jepsen - Distributed systems testing framework TLA+ - Formal specification language (by Leslie Lamport) Chaos Monkey (Netflix) - Random failure injection ","date":"28-Jul-2025","heading":"Testing Distributed Systems","objectID":"\/posts\/system-design\/:52:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#testing-distributed-systems"},{"categories":["software-engineering"],"collections":null,"content":"Distributed Consensus Algorithms Paxos Algorithm: class PaxosNode: def __init__(self, node_id): self.node_id = node_id self.promised_id = None self.accepted_id = None self.accepted_value = None # Phase 1a: Proposer sends Prepare(n) def prepare(self, proposal_id): if proposal_id > (self.promised_id or 0): self.promised_id = proposal_id return { 'promised': True, 'accepted_id': self.accepted_id, 'accepted_value': self.accepted_value } return {'promised': False} # Phase 2a: Proposer sends Accept(n, value) def accept(self, proposal_id, value): if proposal_id >= (self.promised_id or 0): self.promised_id = proposal_id self.accepted_id = proposal_id self.accepted_value = value return {'accepted': True} return {'accepted': False} Raft Consensus (Simplified): Leader election Log replication Safety guarantees ","date":"28-Jul-2025","heading":"Distributed Consensus Algorithms","objectID":"\/posts\/system-design\/:52:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#distributed-consensus-algorithms"},{"categories":["software-engineering"],"collections":null,"content":"Time Series Databases Use Cases: Metrics monitoring IoT sensor data Financial market data Application performance monitoring (APM) Design Considerations: class TimeSeriesDB: def __init__(self): self.segments = {} # time_bucket -> data self.compression_enabled = True def write(self, metric_name, timestamp, value, tags=None): \"\"\"Write time series data point.\"\"\" # Determine time bucket (e.g., hourly) time_bucket = self.get_time_bucket(timestamp) # Create segment if not exists if time_bucket not in self.segments: self.segments[time_bucket] = TimeSeriesSegment(time_bucket) # Append to segment self.segments[time_bucket].append({ 'metric': metric_name, 'timestamp': timestamp, 'value': value, 'tags': tags or {} }) # Compress old segments if len(self.segments) > 24: # Keep 24 hours hot self.compress_old_segments() def query(self, metric_name, start_time, end_time, aggregation='avg'): \"\"\"Query time series data with aggregation.\"\"\" results = [] # Find relevant segments segments = self.get_segments_in_range(start_time, end_time) for segment in segments: data_points = segment.get_metric(metric_name) results.extend(data_points) # Aggregate if aggregation == 'avg': return sum(r['value'] for r in results) \/ len(results) elif aggregation == 'max': return max(r['value'] for r in results) elif aggregation == 'min': return min(r['value'] for r in results) return results def compress_old_segments(self): \"\"\"Compress segments older than 24 hours.\"\"\" cutoff = datetime.now() - timedelta(hours=24) for time_bucket, segment in list(self.segments.items()): if segment.timestamp < cutoff: # Downsample and compress compressed = segment.downsample(factor=10) self.segments[time_bucket] = compressed Popular Time Series DBs: InfluxDB Prometheus TimescaleDB (PostgreSQL extension) Facebook Gorilla ","date":"28-Jul-2025","heading":"Time Series Databases","objectID":"\/posts\/system-design\/:52:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#time-series-databases"},{"categories":["software-engineering"],"collections":null,"content":"Authorization Systems Role-Based Access Control (RBAC): class RBACSystem: def __init__(self): self.roles = {} # role_name -> permissions self.user_roles = {} # user_id -> [role_names] def create_role(self, role_name, permissions): \"\"\"Create role with permissions.\"\"\" self.roles[role_name] = set(permissions) def assign_role(self, user_id, role_name): \"\"\"Assign role to user.\"\"\" if user_id not in self.user_roles: self.user_roles[user_id] = [] self.user_roles[user_id].append(role_name) def check_permission(self, user_id, resource, action): \"\"\"Check if user has permission.\"\"\" user_roles = self.user_roles.get(user_id, []) for role in user_roles: permissions = self.roles.get(role, set()) if f\"{resource}:{action}\" in permissions: return True return False # Usage rbac = RBACSystem() rbac.create_role('admin', ['user:create', 'user:read', 'user:update', 'user:delete']) rbac.create_role('viewer', ['user:read']) rbac.assign_role('user123', 'admin') rbac.check_permission('user123', 'user', 'delete') # True Attribute-Based Access Control (ABAC): More flexible than RBAC Uses attributes (user, resource, environment) Example: “Allow if user.department == resource.department” ","date":"28-Jul-2025","heading":"Authorization Systems","objectID":"\/posts\/system-design\/:52:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#authorization-systems"},{"categories":["software-engineering"],"collections":null,"content":"Batch vs Stream Processing Batch Processing (MapReduce): # Map function def map_word_count(document): words = document.split() return [(word, 1) for word in words] # Reduce function def reduce_word_count(word, counts): return (word, sum(counts)) # MapReduce framework class MapReduce: def __init__(self, num_workers=4): self.num_workers = num_workers def run(self, documents, map_fn, reduce_fn): # Map phase (parallel) mapped = [] with ThreadPoolExecutor(max_workers=self.num_workers) as executor: futures = [executor.submit(map_fn, doc) for doc in documents] for future in futures: mapped.extend(future.result()) # Shuffle phase (group by key) shuffled = {} for key, value in mapped: if key not in shuffled: shuffled[key] = [] shuffled[key].append(value) # Reduce phase (parallel) results = [] with ThreadPoolExecutor(max_workers=self.num_workers) as executor: futures = [ executor.submit(reduce_fn, key, values) for key, values in shuffled.items() ] results = [future.result() for future in futures] return results Stream Processing (Kafka Streams): from kafka import KafkaConsumer, KafkaProducer import json class StreamProcessor: def __init__(self, input_topic, output_topic): self.consumer = KafkaConsumer( input_topic, bootstrap_servers=['localhost:9092'], value_deserializer=lambda m: json.loads(m.decode('utf-8')) ) self.producer = KafkaProducer( bootstrap_servers=['localhost:9092'], value_serializer=lambda m: json.dumps(m).encode('utf-8') ) self.output_topic = output_topic # Windowed aggregation state self.windows = {} # window_key -> aggregate def process(self): \"\"\"Process stream with windowed aggregation.\"\"\" for message in self.consumer: event = message.value # Extract timestamp and create window key timestamp = event['timestamp'] window_key = self.get_window_key(timestamp, window_size=60) # Update window aggregate if window_key not in self.windows: self.windows[window_key] = {'count': 0, 'sum': 0} self.windows[window_key]['count'] += 1 self.windows[window_key]['sum'] += event['value'] # Emit windowed aggregate self.producer.send(self.output_topic, { 'window': window_key, 'avg': self.windows[window_key]['sum'] \/ self.windows[window_key]['count'], 'count': self.windows[window_key]['count'] }) def get_window_key(self, timestamp, window_size): \"\"\"Get window key for tumbling window.\"\"\" return int(timestamp \/ window_size) * window_size ","date":"28-Jul-2025","heading":"Batch vs Stream Processing","objectID":"\/posts\/system-design\/:52:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#batch-vs-stream-processing"},{"categories":["software-engineering"],"collections":null,"content":"Service Mesh What is Service Mesh? Infrastructure layer for service-to-service communication Handles service discovery, load balancing, encryption, authentication Popular: Istio, Linkerd, Consul Key Components: Service Mesh \u2014 Istio and service mesh architecture for microservices communication and observability Benefits: Automatic retries and circuit breaking Mutual TLS encryption Distributed tracing Traffic shaping and routing ","date":"28-Jul-2025","heading":"Service Mesh","objectID":"\/posts\/system-design\/:52:6","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#service-mesh"},{"categories":["software-engineering"],"collections":null,"content":"Distributed Logging and Tracing Distributed Tracing Architecture: import uuid from datetime import datetime class Span: def __init__(self, trace_id, span_id, parent_span_id, operation_name): self.trace_id = trace_id self.span_id = span_id self.parent_span_id = parent_span_id self.operation_name = operation_name self.start_time = datetime.now() self.end_time = None self.tags = {} self.logs = [] def set_tag(self, key, value): self.tags[key] = value def log(self, message): self.logs.append({ 'timestamp': datetime.now(), 'message': message }) def finish(self): self.end_time = datetime.now() def duration_ms(self): if self.end_time: return (self.end_time - self.start_time).total_seconds() * 1000 return None class Tracer: def __init__(self, service_name): self.service_name = service_name self.spans = [] def start_span(self, operation_name, parent_span=None): \"\"\"Start a new span.\"\"\" trace_id = parent_span.trace_id if parent_span else str(uuid.uuid4()) span_id = str(uuid.uuid4()) parent_span_id = parent_span.span_id if parent_span else None span = Span(trace_id, span_id, parent_span_id, operation_name) span.set_tag('service.name', self.service_name) self.spans.append(span) return span def inject(self, span, carrier): \"\"\"Inject span context into carrier (e.g., HTTP headers).\"\"\" carrier['X-Trace-Id'] = span.trace_id carrier['X-Span-Id'] = span.span_id def extract(self, carrier): \"\"\"Extract span context from carrier.\"\"\" return { 'trace_id': carrier.get('X-Trace-Id'), 'parent_span_id': carrier.get('X-Span-Id') } # Usage example tracer = Tracer('api-service') # In API handler def handle_request(request): # Extract parent span context parent_context = tracer.extract(request.headers) # Start span span = tracer.start_span('handle_api_request') if parent_context['parent_span_id']: span.parent_span_id = parent_context['parent_span_id'] span.trace_id = parent_context['trace_id'] span.set_tag('http.method', request.method) span.set_tag('http.url', request.url) try: # Process request result = process_request(request, span) span.set_tag('http.status_code', 200) return result except Exception as e: span.set_tag('error', True) span.log(f\"Error: {str(e)}\") raise finally: span.finish() Popular Tracing Systems: Jaeger (Uber) Zipkin (Twitter) AWS X-Ray Google Cloud Trace ","date":"28-Jul-2025","heading":"Distributed Logging and Tracing","objectID":"\/posts\/system-design\/:52:7","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#distributed-logging-and-tracing"},{"categories":["software-engineering"],"collections":null,"content":"Anomaly Detection Systems Statistical Anomaly Detection: import numpy as np from sklearn.ensemble import IsolationForest class AnomalyDetector: def __init__(self): self.model = IsolationForest(contamination=0.01) # 1% anomalies self.trained = False def train(self, historical_data): \"\"\"Train on historical metrics.\"\"\" # historical_data: array of [cpu, memory, latency, error_rate] self.model.fit(historical_data) self.trained = True def detect(self, current_metrics): \"\"\"Detect if current metrics are anomalous.\"\"\" if not self.trained: raise ValueError(\"Model not trained\") prediction = self.model.predict([current_metrics]) # -1 = anomaly, 1 = normal return prediction[0] == -1 def get_anomaly_score(self, metrics): \"\"\"Get anomaly score (lower = more anomalous).\"\"\" return self.model.score_samples([metrics])[0] # Usage detector = AnomalyDetector() # Train on normal data historical_data = np.array([ [50, 60, 100, 0.1], # [CPU%, Memory%, Latency ms, Error%] [55, 62, 105, 0.12], [52, 58, 98, 0.09], # ... more normal samples ]) detector.train(historical_data) # Check current metrics current = [95, 90, 500, 5.0] # High CPU, memory, latency, errors if detector.detect(current): print(\"Anomaly detected! Alert the team.\") ","date":"28-Jul-2025","heading":"Anomaly Detection Systems","objectID":"\/posts\/system-design\/:52:8","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#anomaly-detection-systems"},{"categories":["software-engineering"],"collections":null,"content":"54. References and Further Reading ","date":"28-Jul-2025","heading":"54. References and Further Reading","objectID":"\/posts\/system-design\/:53:0","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#54-references-and-further-reading"},{"categories":["software-engineering"],"collections":null,"content":"\ud83d\udcda Essential Books “Designing Data-Intensive Applications” by Martin Kleppmann The definitive guide to distributed systems Covers replication, partitioning, transactions, consensus “System Design Interview” by Alex Xu (Vol 1 & 2) Practical interview preparation Real-world system designs “Database Internals” by Alex Petrov Deep dive into database storage engines LSM trees, B-trees, replication “Designing Distributed Systems” by Brendan Burns Patterns for Kubernetes and containers Microservices architectures “Site Reliability Engineering” by Google SRE practices from Google Monitoring, incident response, automation ","date":"28-Jul-2025","heading":"\ud83d\udcda Essential Books","objectID":"\/posts\/system-design\/:53:1","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#-essential-books"},{"categories":["software-engineering"],"collections":null,"content":"\ud83c\udf10 Engineering Blogs Recommended Reading: Netflix Tech Blog \u2014 Microservices, chaos engineering, CDN, Open Connect Uber Engineering \u2014 Geospatial systems, real-time data processing, Kafka Meta Engineering \u2014 Scale challenges, distributed systems, React Twitter Engineering \u2014 Real-time systems, graph databases, timeline service LinkedIn Engineering \u2014 Data infrastructure, Kafka (invented here), Voldemort Airbnb Engineering \u2014 Microservices, ML infrastructure, data quality Pinterest Engineering \u2014 Visual search, recommendation systems, Kraken Dropbox Engineering \u2014 Distributed file systems, sync, Magic Pocket Spotify Engineering \u2014 Event-driven architecture, ML, Backstage Instagram Engineering \u2014 Photo storage, feed ranking, Django at scale Discord Engineering \u2014 Message storage, ScyllaDB migration, Rust Cloudflare Blog \u2014 Edge computing, DDoS mitigation, Workers ByteByteGo \u2014 System design visual explainers (Alex Xu) High Scalability \u2014 Architecture case studies from real companies ","date":"28-Jul-2025","heading":"\ud83c\udf10 Engineering Blogs","objectID":"\/posts\/system-design\/:53:2","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#-engineering-blogs"},{"categories":["software-engineering"],"collections":null,"content":"\ud83d\udcdd Important Papers “The Google File System” (GFS) Distributed file system fundamentals, chunk servers, master node “MapReduce: Simplified Data Processing on Large Clusters” Batch processing framework, parallel computation model “Bigtable: A Distributed Storage System” Wide-column store design, basis for HBase and Cassandra “Dynamo: Amazon’s Highly Available Key-value Store” Eventual consistency, consistent hashing, vector clocks “The Chubby Lock Service for Loosely-Coupled Distributed Systems” Distributed locking and coordination, leader election “Spanner: Google’s Globally-Distributed Database” Globally consistent transactions using TrueTime API “Kafka: a Distributed Messaging System for Log Processing” Log-based message queue, sequential I\/O, consumer groups “Raft Consensus Algorithm” Understandable distributed consensus ","date":"28-Jul-2025","heading":"\ud83d\udcdd Important Papers","objectID":"\/posts\/system-design\/:53:3","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#-important-papers"},{"categories":["software-engineering"],"collections":null,"content":"\ud83c\udf93 Online Courses MIT 6.824 Distributed Systems (Free on YouTube) Grokking the System Design Interview (educative.io) System Design for Tech Interviews (Udemy) Distributed Systems by Chris Colohan (Coursera) ","date":"28-Jul-2025","heading":"\ud83c\udf93 Online Courses","objectID":"\/posts\/system-design\/:53:4","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#-online-courses"},{"categories":["software-engineering"],"collections":null,"content":"\ud83d\udee0\ufe0f Tools and Frameworks Databases: PostgreSQL, MySQL (SQL) MongoDB, Cassandra, DynamoDB (NoSQL) Redis, Memcached (Cache) Elasticsearch (Search) Message Queues: Apache Kafka RabbitMQ AWS SQS\/SNS Google Pub\/Sub Infrastructure: Kubernetes (Container orchestration) Docker (Containerization) Terraform (Infrastructure as Code) Prometheus + Grafana (Monitoring) Load Balancers: NGINX HAProxy AWS ELB\/ALB Google Cloud Load Balancing ","date":"28-Jul-2025","heading":"\ud83d\udee0\ufe0f Tools and Frameworks","objectID":"\/posts\/system-design\/:53:5","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#-tools-and-frameworks"},{"categories":["software-engineering"],"collections":null,"content":"\ud83d\udca1 System Design Patterns Summary Pattern Use Case Example Load Balancing Distribute traffic NGINX, HAProxy Caching Reduce latency Redis, CDN Sharding Scale database Hash-based, Range-based Replication High availability Master-Slave, Multi-Master Rate Limiting Prevent abuse Token bucket, Sliding window Circuit Breaker Fault tolerance Netflix Hystrix CQRS Separate read\/write Event sourcing Event Sourcing Audit trail Kafka, EventStore Saga Distributed transactions Choreography, Orchestration Bulkhead Isolation Thread pools, Process isolation ","date":"28-Jul-2025","heading":"\ud83d\udca1 System Design Patterns Summary","objectID":"\/posts\/system-design\/:53:6","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#-system-design-patterns-summary"},{"categories":["software-engineering"],"collections":null,"content":"\ud83c\udfaf Interview Preparation Checklist Fundamentals: \u2705 CAP theorem \u2705 Consistency models \u2705 Load balancing algorithms \u2705 Caching strategies \u2705 Database indexing \u2705 Sharding and partitioning Advanced Topics: \u2705 Distributed transactions \u2705 Consensus algorithms (Paxos, Raft) \u2705 Consistent hashing \u2705 Rate limiting \u2705 Monitoring and alerting \u2705 Security best practices System Designs: \u2705 URL shortener \u2705 Twitter\/social media \u2705 Chat system \u2705 Video streaming (YouTube) \u2705 E-commerce platform \u2705 Ride-sharing service \u2705 Food delivery system \u2705 Search engine \u2705 Notification system \u2705 Collaborative editor ","date":"28-Jul-2025","heading":"\ud83c\udfaf Interview Preparation Checklist","objectID":"\/posts\/system-design\/:53:7","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#-interview-preparation-checklist"},{"categories":["software-engineering"],"collections":null,"content":"\ud83d\udd17 Online Resources System Design Primer \u2014 270K+ star GitHub repo with diagrams, flashcards, and mock interviews High Scalability Blog \u2014 Real architecture case studies from companies ByteByteGo \u2014 System design visual explainers by Alex Xu InterviewReady \u2014 Structured system design interview prep MIT 6.824 Distributed Systems \u2014 Free lecture notes and videos from MIT Martin Kleppmann’s Blog \u2014 Deep dives on distributed systems concepts ","date":"28-Jul-2025","heading":"\ud83d\udd17 Online Resources","objectID":"\/posts\/system-design\/:53:8","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#-online-resources"},{"categories":["software-engineering"],"collections":null,"content":"\ud83d\udcca Key Metrics to Know Metric Value Context L1 cache reference 0.5 ns CPU cache L2 cache reference 7 ns CPU cache RAM reference 100 ns Main memory SSD random read 150 \u03bcs Storage HDD seek 10 ms Storage Network within datacenter 0.5 ms Internal Network between continents 150 ms Global Read 1MB from RAM 250 \u03bcs Sequential Read 1MB from SSD 1,000 \u03bcs Sequential Read 1MB from HDD 20,000 \u03bcs Sequential End of System Design Complete Guide Remember: System design is about trade-offs. There’s no single “right” answer\u2014only solutions that better fit specific requirements and constraints. Good luck with your interviews and building amazing systems! \ud83d\ude80 ","date":"28-Jul-2025","heading":"\ud83d\udcca Key Metrics to Know","objectID":"\/posts\/system-design\/:53:9","tags":["system-design","distributed-systems","interview-prep"],"title":"System Design Notes","uri":"\/posts\/system-design\/#-key-metrics-to-know"},{"categories":["computer-science"],"collections":null,"content":"Deep, practical Operating Systems guide covering processes, threads, scheduling, virtual memory, file systems, I\/O, security and kernel development \u2014 ideal for interviews and OS engineers.","date":"29-Nov-2024","objectID":"\/posts\/operating-system\/","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/"},{"categories":["computer-science"],"collections":null,"content":"A Comprehensive Journey Through Operating System Concepts, Architecture, and Implementation ","date":"29-Nov-2024","heading":"","objectID":"\/posts\/operating-system\/:0:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#"},{"categories":["computer-science"],"collections":null,"content":"1. Introduction to Operating Systems ","date":"29-Nov-2024","heading":"1. Introduction to Operating Systems","objectID":"\/posts\/operating-system\/:1:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#introduction"},{"categories":["computer-science"],"collections":null,"content":"What is an Operating System? An Operating System (OS) is sophisticated system software that manages computer hardware resources and provides an environment for application programs to execute efficiently. It serves as the critical intermediary layer between the bare metal hardware and user applications, abstracting complex hardware details while providing essential services. Core Responsibilities: Resource Management: Orchestrates CPU time, memory allocation, storage access, and I\/O device operations Hardware Abstraction: Shields applications from hardware complexity through standardized interfaces Process Isolation: Ensures programs run independently without interfering with each other Security Enforcement: Implements access controls and protection mechanisms User Interface: Provides means for users to interact with the system ","date":"29-Nov-2024","heading":"What is an Operating System?","objectID":"\/posts\/operating-system\/:1:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#what-is-an-operating-system"},{"categories":["computer-science"],"collections":null,"content":"Why Study Operating Systems? Understanding operating systems is fundamental for several critical reasons: Info Foundation of All Computing Nearly every application executes atop an operating system OS behavior directly impacts application performance, security, and reliability Modern cloud infrastructure, mobile devices, and embedded systems all rely on OS principles Tip Performance Optimization Deep OS knowledge enables writing high-performance code Understanding scheduling, caching, and memory management leads to better algorithmic choices Ability to diagnose and resolve performance bottlenecks Note Security and Reliability OS provides the trust boundary for system security Knowledge of OS internals helps identify and prevent vulnerabilities Understanding process isolation and memory protection prevents common exploits Tip Career Advancement Essential knowledge for system programming, embedded development, and infrastructure roles Required for roles in kernel development, device drivers, and low-level software Valuable for cloud architecture, distributed systems, and high-performance computing ","date":"29-Nov-2024","heading":"Why Study Operating Systems?","objectID":"\/posts\/operating-system\/:1:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#why-study-operating-systems"},{"categories":["computer-science"],"collections":null,"content":"Historical Evolution First Generation (1945-1955): Vacuum Tubes and Plugboards Characteristics: No operating system concept; direct hardware programming Programming Method: Machine language with plugboards for connections Key Limitation: One program at a time; no multiprogramming Example Systems: ENIAC, UNIVAC I Second Generation (1955-1965): Transistors and Batch Systems Key Innovation: Introduction of batch processing Batch System Concept: Group similar jobs together; process sequentially Advantages: Reduced setup time; improved resource utilization Programming: Assembly language and early compilers (FORTRAN, COBOL) Notable Systems: IBM 7094, Atlas Computer Third Generation (1965-1980): Integrated Circuits and Multiprogramming Breakthrough: Integrated circuits enabled smaller, faster, and cheaper computers Multiprogramming: Multiple jobs in memory simultaneously; CPU switches between them Time-Sharing: Interactive computing with multiple users Key Concepts: Spooling, virtual memory, file systems Landmark Systems: IBM System\/360, UNIX (1969), MULTICS Fourth Generation (1980-Present): Personal Computers Revolution: Microprocessors enabled personal computers GUI Introduction: Graphical User Interfaces (Windows, Mac OS, X Window System) Network Operating Systems: Client-server architectures Notable Systems: MS-DOS, Windows, Mac OS, Linux (1991) Fifth Generation (1990-Present): Mobile and Distributed Computing Mobile OS: iOS (2007), Android (2008) Cloud Computing: Virtualization, containerization Distributed Systems: Microservices architecture IoT: Embedded and real-time operating systems Modern Trends: Rust-based OS development, RISC-V architecture ","date":"29-Nov-2024","heading":"Historical Evolution","objectID":"\/posts\/operating-system\/:1:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#historical-evolution"},{"categories":["computer-science"],"collections":null,"content":"Historical Evolution First Generation (1945-1955): Vacuum Tubes and Plugboards Characteristics: No operating system concept; direct hardware programming Programming Method: Machine language with plugboards for connections Key Limitation: One program at a time; no multiprogramming Example Systems: ENIAC, UNIVAC I Second Generation (1955-1965): Transistors and Batch Systems Key Innovation: Introduction of batch processing Batch System Concept: Group similar jobs together; process sequentially Advantages: Reduced setup time; improved resource utilization Programming: Assembly language and early compilers (FORTRAN, COBOL) Notable Systems: IBM 7094, Atlas Computer Third Generation (1965-1980): Integrated Circuits and Multiprogramming Breakthrough: Integrated circuits enabled smaller, faster, and cheaper computers Multiprogramming: Multiple jobs in memory simultaneously; CPU switches between them Time-Sharing: Interactive computing with multiple users Key Concepts: Spooling, virtual memory, file systems Landmark Systems: IBM System\/360, UNIX (1969), MULTICS Fourth Generation (1980-Present): Personal Computers Revolution: Microprocessors enabled personal computers GUI Introduction: Graphical User Interfaces (Windows, Mac OS, X Window System) Network Operating Systems: Client-server architectures Notable Systems: MS-DOS, Windows, Mac OS, Linux (1991) Fifth Generation (1990-Present): Mobile and Distributed Computing Mobile OS: iOS (2007), Android (2008) Cloud Computing: Virtualization, containerization Distributed Systems: Microservices architecture IoT: Embedded and real-time operating systems Modern Trends: Rust-based OS development, RISC-V architecture ","date":"29-Nov-2024","heading":"First Generation (1945-1955): Vacuum Tubes and Plugboards","objectID":"\/posts\/operating-system\/:1:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#first-generation-1945-1955-vacuum-tubes-and-plugboards"},{"categories":["computer-science"],"collections":null,"content":"Historical Evolution First Generation (1945-1955): Vacuum Tubes and Plugboards Characteristics: No operating system concept; direct hardware programming Programming Method: Machine language with plugboards for connections Key Limitation: One program at a time; no multiprogramming Example Systems: ENIAC, UNIVAC I Second Generation (1955-1965): Transistors and Batch Systems Key Innovation: Introduction of batch processing Batch System Concept: Group similar jobs together; process sequentially Advantages: Reduced setup time; improved resource utilization Programming: Assembly language and early compilers (FORTRAN, COBOL) Notable Systems: IBM 7094, Atlas Computer Third Generation (1965-1980): Integrated Circuits and Multiprogramming Breakthrough: Integrated circuits enabled smaller, faster, and cheaper computers Multiprogramming: Multiple jobs in memory simultaneously; CPU switches between them Time-Sharing: Interactive computing with multiple users Key Concepts: Spooling, virtual memory, file systems Landmark Systems: IBM System\/360, UNIX (1969), MULTICS Fourth Generation (1980-Present): Personal Computers Revolution: Microprocessors enabled personal computers GUI Introduction: Graphical User Interfaces (Windows, Mac OS, X Window System) Network Operating Systems: Client-server architectures Notable Systems: MS-DOS, Windows, Mac OS, Linux (1991) Fifth Generation (1990-Present): Mobile and Distributed Computing Mobile OS: iOS (2007), Android (2008) Cloud Computing: Virtualization, containerization Distributed Systems: Microservices architecture IoT: Embedded and real-time operating systems Modern Trends: Rust-based OS development, RISC-V architecture ","date":"29-Nov-2024","heading":"Second Generation (1955-1965): Transistors and Batch Systems","objectID":"\/posts\/operating-system\/:1:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#second-generation-1955-1965-transistors-and-batch-systems"},{"categories":["computer-science"],"collections":null,"content":"Historical Evolution First Generation (1945-1955): Vacuum Tubes and Plugboards Characteristics: No operating system concept; direct hardware programming Programming Method: Machine language with plugboards for connections Key Limitation: One program at a time; no multiprogramming Example Systems: ENIAC, UNIVAC I Second Generation (1955-1965): Transistors and Batch Systems Key Innovation: Introduction of batch processing Batch System Concept: Group similar jobs together; process sequentially Advantages: Reduced setup time; improved resource utilization Programming: Assembly language and early compilers (FORTRAN, COBOL) Notable Systems: IBM 7094, Atlas Computer Third Generation (1965-1980): Integrated Circuits and Multiprogramming Breakthrough: Integrated circuits enabled smaller, faster, and cheaper computers Multiprogramming: Multiple jobs in memory simultaneously; CPU switches between them Time-Sharing: Interactive computing with multiple users Key Concepts: Spooling, virtual memory, file systems Landmark Systems: IBM System\/360, UNIX (1969), MULTICS Fourth Generation (1980-Present): Personal Computers Revolution: Microprocessors enabled personal computers GUI Introduction: Graphical User Interfaces (Windows, Mac OS, X Window System) Network Operating Systems: Client-server architectures Notable Systems: MS-DOS, Windows, Mac OS, Linux (1991) Fifth Generation (1990-Present): Mobile and Distributed Computing Mobile OS: iOS (2007), Android (2008) Cloud Computing: Virtualization, containerization Distributed Systems: Microservices architecture IoT: Embedded and real-time operating systems Modern Trends: Rust-based OS development, RISC-V architecture ","date":"29-Nov-2024","heading":"Third Generation (1965-1980): Integrated Circuits and Multiprogramming","objectID":"\/posts\/operating-system\/:1:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#third-generation-1965-1980-integrated-circuits-and-multiprogramming"},{"categories":["computer-science"],"collections":null,"content":"Historical Evolution First Generation (1945-1955): Vacuum Tubes and Plugboards Characteristics: No operating system concept; direct hardware programming Programming Method: Machine language with plugboards for connections Key Limitation: One program at a time; no multiprogramming Example Systems: ENIAC, UNIVAC I Second Generation (1955-1965): Transistors and Batch Systems Key Innovation: Introduction of batch processing Batch System Concept: Group similar jobs together; process sequentially Advantages: Reduced setup time; improved resource utilization Programming: Assembly language and early compilers (FORTRAN, COBOL) Notable Systems: IBM 7094, Atlas Computer Third Generation (1965-1980): Integrated Circuits and Multiprogramming Breakthrough: Integrated circuits enabled smaller, faster, and cheaper computers Multiprogramming: Multiple jobs in memory simultaneously; CPU switches between them Time-Sharing: Interactive computing with multiple users Key Concepts: Spooling, virtual memory, file systems Landmark Systems: IBM System\/360, UNIX (1969), MULTICS Fourth Generation (1980-Present): Personal Computers Revolution: Microprocessors enabled personal computers GUI Introduction: Graphical User Interfaces (Windows, Mac OS, X Window System) Network Operating Systems: Client-server architectures Notable Systems: MS-DOS, Windows, Mac OS, Linux (1991) Fifth Generation (1990-Present): Mobile and Distributed Computing Mobile OS: iOS (2007), Android (2008) Cloud Computing: Virtualization, containerization Distributed Systems: Microservices architecture IoT: Embedded and real-time operating systems Modern Trends: Rust-based OS development, RISC-V architecture ","date":"29-Nov-2024","heading":"Fourth Generation (1980-Present): Personal Computers","objectID":"\/posts\/operating-system\/:1:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#fourth-generation-1980-present-personal-computers"},{"categories":["computer-science"],"collections":null,"content":"Historical Evolution First Generation (1945-1955): Vacuum Tubes and Plugboards Characteristics: No operating system concept; direct hardware programming Programming Method: Machine language with plugboards for connections Key Limitation: One program at a time; no multiprogramming Example Systems: ENIAC, UNIVAC I Second Generation (1955-1965): Transistors and Batch Systems Key Innovation: Introduction of batch processing Batch System Concept: Group similar jobs together; process sequentially Advantages: Reduced setup time; improved resource utilization Programming: Assembly language and early compilers (FORTRAN, COBOL) Notable Systems: IBM 7094, Atlas Computer Third Generation (1965-1980): Integrated Circuits and Multiprogramming Breakthrough: Integrated circuits enabled smaller, faster, and cheaper computers Multiprogramming: Multiple jobs in memory simultaneously; CPU switches between them Time-Sharing: Interactive computing with multiple users Key Concepts: Spooling, virtual memory, file systems Landmark Systems: IBM System\/360, UNIX (1969), MULTICS Fourth Generation (1980-Present): Personal Computers Revolution: Microprocessors enabled personal computers GUI Introduction: Graphical User Interfaces (Windows, Mac OS, X Window System) Network Operating Systems: Client-server architectures Notable Systems: MS-DOS, Windows, Mac OS, Linux (1991) Fifth Generation (1990-Present): Mobile and Distributed Computing Mobile OS: iOS (2007), Android (2008) Cloud Computing: Virtualization, containerization Distributed Systems: Microservices architecture IoT: Embedded and real-time operating systems Modern Trends: Rust-based OS development, RISC-V architecture ","date":"29-Nov-2024","heading":"Fifth Generation (1990-Present): Mobile and Distributed Computing","objectID":"\/posts\/operating-system\/:1:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#fifth-generation-1990-present-mobile-and-distributed-computing"},{"categories":["computer-science"],"collections":null,"content":"2. Computer Hardware Fundamentals Understanding operating systems requires solid knowledge of underlying hardware components. ","date":"29-Nov-2024","heading":"2. Computer Hardware Fundamentals","objectID":"\/posts\/operating-system\/:2:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#hardware-fundamentals"},{"categories":["computer-science"],"collections":null,"content":"The Transistor: Building Block of Computing A transistor is a semiconductor device functioning as an electronic switch with two states: conducting (ON) or non-conducting (OFF). These binary states represent the fundamental 0s and 1s of digital computing. Key Facts: Modern CPUs contain billions of transistors (Apple M2: ~20 billion; AMD EPYC: ~40+ billion) Transistor density follows Moore’s Law: doubling approximately every two years Advanced manufacturing processes now reach 3nm and smaller Transistor Evolution: 1947: First transistor (Bell Labs) - Point-contact type 1959: Planar silicon transistor - Foundation for ICs 1971: Intel 4004 - 2,300 transistors 2023: AMD EPYC Genoa - 57 billion transistors ","date":"29-Nov-2024","heading":"The Transistor: Building Block of Computing","objectID":"\/posts\/operating-system\/:2:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#the-transistor-building-block-of-computing"},{"categories":["computer-science"],"collections":null,"content":"Integrated Circuits (ICs) An Integrated Circuit is a complete electronic circuit fabricated on a single semiconductor wafer (typically silicon). It contains millions or billions of interconnected transistors, resistors, and capacitors forming complex circuitry. Types: SSI (Small-Scale Integration): < 100 components MSI (Medium-Scale Integration): 100-1000 components LSI (Large-Scale Integration): 1000-100,000 components VLSI (Very Large-Scale Integration): > 100,000 components ULSI (Ultra Large-Scale Integration): > 1 million components ","date":"29-Nov-2024","heading":"Integrated Circuits (ICs)","objectID":"\/posts\/operating-system\/:2:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#integrated-circuits-ics"},{"categories":["computer-science"],"collections":null,"content":"Central Processing Unit (CPU) The CPU is the primary computational engine, often called the “brain” of the computer. It fetches instructions from memory, decodes them, executes operations, and stores results. CPU Architecture Components 1. Arithmetic Logic Unit (ALU) Functions: \u251c\u2500\u2500 Arithmetic Operations \u2502 \u251c\u2500\u2500 Addition, Subtraction \u2502 \u251c\u2500\u2500 Multiplication, Division \u2502 \u2514\u2500\u2500 Modulo operations \u2514\u2500\u2500 Logical Operations \u251c\u2500\u2500 AND, OR, XOR, NOT \u251c\u2500\u2500 Bit shifting (Left\/Right) \u2514\u2500\u2500 Comparison operations Implementation Concept: \/\/ Simplified ALU operation representation uint32_t alu_execute(uint8_t operation, uint32_t operand_a, uint32_t operand_b) { switch(operation) { case ALU_ADD: return operand_a + operand_b; case ALU_SUB: return operand_a - operand_b; case ALU_AND: return operand_a & operand_b; case ALU_OR: return operand_a | operand_b; case ALU_XOR: return operand_a ^ operand_b; case ALU_SHL: return operand_a << operand_b; case ALU_SHR: return operand_a >> operand_b; default: return 0; \/\/ Invalid operation } } 2. Control Unit (CU) The Control Unit orchestrates CPU operations by: Fetching instructions from memory Decoding instruction opcodes Generating control signals for other components Managing instruction sequencing Instruction Cycle (Fetch-Decode-Execute): 1. FETCH: Retrieve instruction from memory[PC] 2. DECODE: Interpret opcode and extract operands 3. EXECUTE: Perform operation via ALU or other units 4. STORE: Write results back to registers\/memory 5. INCREMENT: Update Program Counter (PC) 3. Registers Registers are ultra-fast storage locations within the CPU for immediate data access. Essential Register Types: Register Purpose Description Program Counter (PC) Next Instruction Address Points to memory address of next instruction to execute Instruction Register (IR) Current Instruction Holds instruction currently being executed Accumulator (ACC) ALU Results Stores intermediate arithmetic\/logic results Memory Address Register (MAR) Memory Address Contains address for memory read\/write operations Memory Data Register (MDR) Memory Data Buffer for data read from or written to memory Stack Pointer (SP) Stack Top Points to top of the runtime stack Status Register (FLAGS) Processor State Contains condition codes (Zero, Carry, Negative, Overflow) General Purpose Registers Data Storage R0-R15 (ARM), RAX-R15 (x86-64) for various operations Register Example (x86-64 Architecture): ; Example: Adding two numbers using registers mov rax, 10 ; Load 10 into RAX register mov rbx, 20 ; Load 20 into RBX register add rax, rbx ; Add RBX to RAX (result: RAX = 30) mov [result], rax ; Store result in memory 4. Cache Memory Hierarchy Modern CPUs employ multi-level caching to bridge the speed gap between processor and main memory. Cache Levels: CPU Registers (< 1 KB) \u2193 Access: 0 cycles L1 Cache (32-64 KB per core) \u2193 Access: 1-4 cycles L2 Cache (256-512 KB per core) \u2193 Access: 10-20 cycles L3 Cache (8-64 MB shared) \u2193 Access: 40-70 cycles Main Memory (4-256 GB) \u2193 Access: 100-300 cycles SSD Storage (128 GB - 4 TB) \u2193 Access: ~50,000 cycles HDD Storage (500 GB - 20 TB) \u2193 Access: ~10,000,000 cycles Cache Properties: Cache Line: Typically 64 bytes; unit of transfer between cache levels Cache Hit: Data found in cache (fast access) Cache Miss: Data not in cache, must fetch from lower level (slow) Cache Coherence: Maintaining consistency across multiple caches in multicore systems Cache Coherence Protocols: MESI Protocol (Modified, Exclusive, Shared, Invalid) MOESI Protocol (adds Owned state) Ensures all CPUs see consistent memory values ","date":"29-Nov-2024","heading":"Central Processing Unit (CPU)","objectID":"\/posts\/operating-system\/:2:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#central-processing-unit-cpu"},{"categories":["computer-science"],"collections":null,"content":"Central Processing Unit (CPU) The CPU is the primary computational engine, often called the “brain” of the computer. It fetches instructions from memory, decodes them, executes operations, and stores results. CPU Architecture Components 1. Arithmetic Logic Unit (ALU) Functions: \u251c\u2500\u2500 Arithmetic Operations \u2502 \u251c\u2500\u2500 Addition, Subtraction \u2502 \u251c\u2500\u2500 Multiplication, Division \u2502 \u2514\u2500\u2500 Modulo operations \u2514\u2500\u2500 Logical Operations \u251c\u2500\u2500 AND, OR, XOR, NOT \u251c\u2500\u2500 Bit shifting (Left\/Right) \u2514\u2500\u2500 Comparison operations Implementation Concept: \/\/ Simplified ALU operation representation uint32_t alu_execute(uint8_t operation, uint32_t operand_a, uint32_t operand_b) { switch(operation) { case ALU_ADD: return operand_a + operand_b; case ALU_SUB: return operand_a - operand_b; case ALU_AND: return operand_a & operand_b; case ALU_OR: return operand_a | operand_b; case ALU_XOR: return operand_a ^ operand_b; case ALU_SHL: return operand_a << operand_b; case ALU_SHR: return operand_a >> operand_b; default: return 0; \/\/ Invalid operation } } 2. Control Unit (CU) The Control Unit orchestrates CPU operations by: Fetching instructions from memory Decoding instruction opcodes Generating control signals for other components Managing instruction sequencing Instruction Cycle (Fetch-Decode-Execute): 1. FETCH: Retrieve instruction from memory[PC] 2. DECODE: Interpret opcode and extract operands 3. EXECUTE: Perform operation via ALU or other units 4. STORE: Write results back to registers\/memory 5. INCREMENT: Update Program Counter (PC) 3. Registers Registers are ultra-fast storage locations within the CPU for immediate data access. Essential Register Types: Register Purpose Description Program Counter (PC) Next Instruction Address Points to memory address of next instruction to execute Instruction Register (IR) Current Instruction Holds instruction currently being executed Accumulator (ACC) ALU Results Stores intermediate arithmetic\/logic results Memory Address Register (MAR) Memory Address Contains address for memory read\/write operations Memory Data Register (MDR) Memory Data Buffer for data read from or written to memory Stack Pointer (SP) Stack Top Points to top of the runtime stack Status Register (FLAGS) Processor State Contains condition codes (Zero, Carry, Negative, Overflow) General Purpose Registers Data Storage R0-R15 (ARM), RAX-R15 (x86-64) for various operations Register Example (x86-64 Architecture): ; Example: Adding two numbers using registers mov rax, 10 ; Load 10 into RAX register mov rbx, 20 ; Load 20 into RBX register add rax, rbx ; Add RBX to RAX (result: RAX = 30) mov [result], rax ; Store result in memory 4. Cache Memory Hierarchy Modern CPUs employ multi-level caching to bridge the speed gap between processor and main memory. Cache Levels: CPU Registers (< 1 KB) \u2193 Access: 0 cycles L1 Cache (32-64 KB per core) \u2193 Access: 1-4 cycles L2 Cache (256-512 KB per core) \u2193 Access: 10-20 cycles L3 Cache (8-64 MB shared) \u2193 Access: 40-70 cycles Main Memory (4-256 GB) \u2193 Access: 100-300 cycles SSD Storage (128 GB - 4 TB) \u2193 Access: ~50,000 cycles HDD Storage (500 GB - 20 TB) \u2193 Access: ~10,000,000 cycles Cache Properties: Cache Line: Typically 64 bytes; unit of transfer between cache levels Cache Hit: Data found in cache (fast access) Cache Miss: Data not in cache, must fetch from lower level (slow) Cache Coherence: Maintaining consistency across multiple caches in multicore systems Cache Coherence Protocols: MESI Protocol (Modified, Exclusive, Shared, Invalid) MOESI Protocol (adds Owned state) Ensures all CPUs see consistent memory values ","date":"29-Nov-2024","heading":"CPU Architecture Components","objectID":"\/posts\/operating-system\/:2:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#cpu-architecture-components"},{"categories":["computer-science"],"collections":null,"content":"Graphics Processing Unit (GPU) GPUs are specialized processors optimized for parallel computation, originally designed for graphics rendering but now widely used for general-purpose computing. Key Differences from CPUs: Aspect CPU GPU Core Count 4-64 high-performance cores 1000s of simpler cores Design Philosophy Optimized for latency Optimized for throughput Task Suitability Complex, sequential tasks Highly parallel, repetitive tasks Memory Large cache hierarchy High-bandwidth memory Control Flow Complex branching SIMT (Single Instruction, Multiple Threads) GPU Applications: Graphics rendering and ray tracing Machine learning and deep neural networks Scientific simulations (computational fluid dynamics, molecular modeling) Cryptocurrency mining Video encoding\/decoding ","date":"29-Nov-2024","heading":"Graphics Processing Unit (GPU)","objectID":"\/posts\/operating-system\/:2:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#graphics-processing-unit-gpu"},{"categories":["computer-science"],"collections":null,"content":"Memory Systems Memory Hierarchy Goals: Speed: Fast access for frequently used data Capacity: Large storage for all program data Cost: Balance between performance and affordability Primary Memory (Volatile) Random Access Memory (RAM) DRAM (Dynamic RAM): Main memory; needs periodic refresh; slower but denser DDR4: 2133-3200 MT\/s, 16-64 GB typical DDR5: 4800-8400 MT\/s, 16-128 GB typical SRAM (Static RAM): Cache memory; no refresh needed; faster but more expensive Used in L1\/L2\/L3 caches Memory Access Patterns: \/\/ Demonstrating cache-friendly vs cache-unfriendly access \/\/ Cache-friendly: Sequential access (spatial locality) void sum_array_row_major(int arr[N][N]) { for (int i = 0; i < N; i++) { for (int j = 0; j < N; j++) { sum += arr[i][j]; \/\/ Sequential memory access } } } \/\/ Cache-unfriendly: Column-major in row-major storage void sum_array_column_major(int arr[N][N]) { for (int j = 0; j < N; j++) { for (int i = 0; i < N; i++) { sum += arr[i][j]; \/\/ Jumps across cache lines } } } Secondary Storage (Non-Volatile) Solid State Drives (SSDs) Technology: NAND flash memory Speed: 500-7000 MB\/s (SATA to NVMe) Latency: 50-100 microseconds Wear Leveling: Distributes writes to prevent cell degradation Hard Disk Drives (HDDs) Technology: Magnetic storage on rotating platters Speed: 100-200 MB\/s Latency: 5-15 milliseconds (seek time + rotational delay) Capacity: Up to 20 TB per drive Storage Technologies Comparison: Technology Speed Latency Cost per GB Durability Use Case RAM 20+ GB\/s 50-100 ns High Volatile Active data NVMe SSD 3-7 GB\/s 50-100 \u03bcs Medium Limited writes OS, applications SATA SSD 500 MB\/s 50-100 \u03bcs Medium-Low Limited writes General storage HDD 100-200 MB\/s 5-15 ms Low High Archival, bulk storage ","date":"29-Nov-2024","heading":"Memory Systems","objectID":"\/posts\/operating-system\/:2:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#memory-systems"},{"categories":["computer-science"],"collections":null,"content":"System Bus Architecture The System Bus is a collection of parallel electrical conductors connecting CPU, memory, and I\/O devices. Bus Types: 1. Data Bus Carries actual data between components Width determines data transfer rate (32-bit, 64-bit) Bidirectional 2. Address Bus Carries memory addresses Width determines addressable memory space 32-bit: 4 GB (2\u00b3\u00b2 bytes) 64-bit: 16 Exabytes (2\u2076\u2074 bytes) Unidirectional (CPU to memory\/devices) 3. Control Bus Carries control signals (read, write, interrupt, clock) Coordinates bus operations Bidirectional Bus Operation Example: CPU wants to read memory location 0x1000: 1. CPU places 0x1000 on Address Bus 2. CPU asserts READ control signal on Control Bus 3. Memory responds by placing data at 0x1000 onto Data Bus 4. CPU reads data from Data Bus 5. CPU de-asserts READ signal ","date":"29-Nov-2024","heading":"System Bus Architecture","objectID":"\/posts\/operating-system\/:2:6","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#system-bus-architecture"},{"categories":["computer-science"],"collections":null,"content":"3. Operating System Architecture ","date":"29-Nov-2024","heading":"3. Operating System Architecture","objectID":"\/posts\/operating-system\/:3:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#os-architecture"},{"categories":["computer-science"],"collections":null,"content":"OS Design Principles Separation of Mechanism and Policy Mechanism: How to do something (implementation) Policy: What should be done (decision-making) Benefit: Flexibility; policies can change without altering mechanisms Example: Mechanism: Context switching between processes Policy: Which process to run next (scheduling algorithm) Optimization for Common Case Design decisions should favor frequently occurring scenarios Identify workload characteristics and usage patterns Balance generality with specialized optimization ","date":"29-Nov-2024","heading":"OS Design Principles","objectID":"\/posts\/operating-system\/:3:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#os-design-principles"},{"categories":["computer-science"],"collections":null,"content":"Kernel Architectures 1. Monolithic Kernel Concept: All OS services run in kernel space with full hardware access. Structure: User Space \u2502 \u251c\u2500 Application 1 \u251c\u2500 Application 2 \u2514\u2500 Application N \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 System Call Interface \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space (Single Address Space) \u2502 \u251c\u2500 Process Management \u251c\u2500 Memory Management \u251c\u2500 File System \u251c\u2500 Device Drivers \u251c\u2500 Network Stack \u2514\u2500 Scheduling \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Tip Advantages of Monolithic Kernel: Performance: Direct function calls within kernel; no context switching overhead Simplicity: Straightforward design and implementation Efficiency: Minimal message passing overhead Warning Disadvantages of Monolithic Kernel: Reliability: Bug in any component can crash entire system Maintainability: Large, complex codebase difficult to maintain Security: Large attack surface; any vulnerability affects whole kernel Examples: Linux, traditional UNIX, Windows 9x Linux Kernel Structure: \/\/ Simplified representation of kernel subsystems struct linux_kernel { struct process_manager { struct task_struct *process_list; struct scheduler *sched; } pm; struct memory_manager { struct page_table *pgd; struct vm_area *vma_list; } mm; struct file_system { struct super_block *sb; struct inode_cache *icache; } fs; struct network_stack { struct socket *socket_list; struct protocol_handler *protocols; } net; }; 2. Microkernel Architecture Concept: Minimal kernel with essential services only; other services run in user space. Structure: User Space \u2502 \u251c\u2500 File Server \u251c\u2500 Device Drivers \u251c\u2500 Network Server \u251c\u2500 Applications \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 IPC (Message Passing) \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space (Minimal) \u2502 \u251c\u2500 IPC Mechanism \u251c\u2500 Basic Scheduling \u251c\u2500 Low-level Memory Management \u2514\u2500 Hardware Abstraction \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Essential Kernel Services: Inter-Process Communication (IPC) Basic thread scheduling Low-level address space management Interrupt handling Tip Advantages of Microkernel: Fault Isolation: Service crash doesn’t affect kernel Modularity: Easy to add\/remove services Security: Smaller trusted computing base (TCB) Portability: Hardware-specific code isolated Danger Disadvantages of Microkernel: Performance: IPC overhead for service communication Complexity: More complex system structure Development Effort: Requires careful IPC design Examples: MINIX 3, L4 microkernel family, QNX, seL4 Microkernel IPC Example: \/\/ Simplified IPC message passing struct message { int src_pid; int dst_pid; int msg_type; char data[1024]; }; \/\/ Send message to file server int read_file(const char *path, char *buffer, size_t size) { struct message msg; msg.src_pid = getpid(); msg.dst_pid = FILE_SERVER_PID; msg.msg_type = MSG_READ_FILE; strcpy(msg.data, path); \/\/ Send to microkernel IPC send_message(&msg); \/\/ Wait for reply receive_message(&msg); \/\/ Copy data from reply memcpy(buffer, msg.data, size); return msg.msg_type; \/\/ Return value } 3. Hybrid Kernel Concept: Combines monolithic and microkernel approaches; some services in kernel for performance, others in user space. Structure: User Space \u2502 \u251c\u2500 User-mode Services \u251c\u2500 Device Drivers (some) \u2514\u2500 Applications \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Mixed Interface \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space \u2502 \u251c\u2500 Core Kernel (Microkernel-like) \u2502 \u251c\u2500 IPC \u2502 \u251c\u2500 Scheduling \u2502 \u2514\u2500 Low-level MM \u2502 \u2514\u2500 Integrated Services (Monolithic-like) \u251c\u2500 File System \u251c\u2500 Device Drivers (critical) \u2514\u2500 Network Stack \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Advantages: Balances performance and modularity Critical services in kernel for speed Non-critical services in user space for safety Examples: Windows NT\/modern Windows: NT kernel (hybrid) macOS XNU: Mach microkernel + BSD monolithic components XNU (macOS\/iOS Kernel) Architecture: \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 User Space Applications \u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 System Libraries \u2502 \u2502 (libSystem, Foundation, etc.) \u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 XNU Kernel \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502 \u2502 BSD Layer (Monolithic) \u2502 \u2502 \u2502 \u2502 - POSIX comp","date":"29-Nov-2024","heading":"Kernel Architectures","objectID":"\/posts\/operating-system\/:3:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#kernel-architectures"},{"categories":["computer-science"],"collections":null,"content":"Kernel Architectures 1. Monolithic Kernel Concept: All OS services run in kernel space with full hardware access. Structure: User Space \u2502 \u251c\u2500 Application 1 \u251c\u2500 Application 2 \u2514\u2500 Application N \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 System Call Interface \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space (Single Address Space) \u2502 \u251c\u2500 Process Management \u251c\u2500 Memory Management \u251c\u2500 File System \u251c\u2500 Device Drivers \u251c\u2500 Network Stack \u2514\u2500 Scheduling \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Tip Advantages of Monolithic Kernel: Performance: Direct function calls within kernel; no context switching overhead Simplicity: Straightforward design and implementation Efficiency: Minimal message passing overhead Warning Disadvantages of Monolithic Kernel: Reliability: Bug in any component can crash entire system Maintainability: Large, complex codebase difficult to maintain Security: Large attack surface; any vulnerability affects whole kernel Examples: Linux, traditional UNIX, Windows 9x Linux Kernel Structure: \/\/ Simplified representation of kernel subsystems struct linux_kernel { struct process_manager { struct task_struct *process_list; struct scheduler *sched; } pm; struct memory_manager { struct page_table *pgd; struct vm_area *vma_list; } mm; struct file_system { struct super_block *sb; struct inode_cache *icache; } fs; struct network_stack { struct socket *socket_list; struct protocol_handler *protocols; } net; }; 2. Microkernel Architecture Concept: Minimal kernel with essential services only; other services run in user space. Structure: User Space \u2502 \u251c\u2500 File Server \u251c\u2500 Device Drivers \u251c\u2500 Network Server \u251c\u2500 Applications \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 IPC (Message Passing) \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space (Minimal) \u2502 \u251c\u2500 IPC Mechanism \u251c\u2500 Basic Scheduling \u251c\u2500 Low-level Memory Management \u2514\u2500 Hardware Abstraction \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Essential Kernel Services: Inter-Process Communication (IPC) Basic thread scheduling Low-level address space management Interrupt handling Tip Advantages of Microkernel: Fault Isolation: Service crash doesn’t affect kernel Modularity: Easy to add\/remove services Security: Smaller trusted computing base (TCB) Portability: Hardware-specific code isolated Danger Disadvantages of Microkernel: Performance: IPC overhead for service communication Complexity: More complex system structure Development Effort: Requires careful IPC design Examples: MINIX 3, L4 microkernel family, QNX, seL4 Microkernel IPC Example: \/\/ Simplified IPC message passing struct message { int src_pid; int dst_pid; int msg_type; char data[1024]; }; \/\/ Send message to file server int read_file(const char *path, char *buffer, size_t size) { struct message msg; msg.src_pid = getpid(); msg.dst_pid = FILE_SERVER_PID; msg.msg_type = MSG_READ_FILE; strcpy(msg.data, path); \/\/ Send to microkernel IPC send_message(&msg); \/\/ Wait for reply receive_message(&msg); \/\/ Copy data from reply memcpy(buffer, msg.data, size); return msg.msg_type; \/\/ Return value } 3. Hybrid Kernel Concept: Combines monolithic and microkernel approaches; some services in kernel for performance, others in user space. Structure: User Space \u2502 \u251c\u2500 User-mode Services \u251c\u2500 Device Drivers (some) \u2514\u2500 Applications \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Mixed Interface \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space \u2502 \u251c\u2500 Core Kernel (Microkernel-like) \u2502 \u251c\u2500 IPC \u2502 \u251c\u2500 Scheduling \u2502 \u2514\u2500 Low-level MM \u2502 \u2514\u2500 Integrated Services (Monolithic-like) \u251c\u2500 File System \u251c\u2500 Device Drivers (critical) \u2514\u2500 Network Stack \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Advantages: Balances performance and modularity Critical services in kernel for speed Non-critical services in user space for safety Examples: Windows NT\/modern Windows: NT kernel (hybrid) macOS XNU: Mach microkernel + BSD monolithic components XNU (macOS\/iOS Kernel) Architecture: \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 User Space Applications \u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 System Libraries \u2502 \u2502 (libSystem, Foundation, etc.) \u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 XNU Kernel \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502 \u2502 BSD Layer (Monolithic) \u2502 \u2502 \u2502 \u2502 - POSIX comp","date":"29-Nov-2024","heading":"1. Monolithic Kernel","objectID":"\/posts\/operating-system\/:3:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1-monolithic-kernel"},{"categories":["computer-science"],"collections":null,"content":"Kernel Architectures 1. Monolithic Kernel Concept: All OS services run in kernel space with full hardware access. Structure: User Space \u2502 \u251c\u2500 Application 1 \u251c\u2500 Application 2 \u2514\u2500 Application N \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 System Call Interface \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space (Single Address Space) \u2502 \u251c\u2500 Process Management \u251c\u2500 Memory Management \u251c\u2500 File System \u251c\u2500 Device Drivers \u251c\u2500 Network Stack \u2514\u2500 Scheduling \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Tip Advantages of Monolithic Kernel: Performance: Direct function calls within kernel; no context switching overhead Simplicity: Straightforward design and implementation Efficiency: Minimal message passing overhead Warning Disadvantages of Monolithic Kernel: Reliability: Bug in any component can crash entire system Maintainability: Large, complex codebase difficult to maintain Security: Large attack surface; any vulnerability affects whole kernel Examples: Linux, traditional UNIX, Windows 9x Linux Kernel Structure: \/\/ Simplified representation of kernel subsystems struct linux_kernel { struct process_manager { struct task_struct *process_list; struct scheduler *sched; } pm; struct memory_manager { struct page_table *pgd; struct vm_area *vma_list; } mm; struct file_system { struct super_block *sb; struct inode_cache *icache; } fs; struct network_stack { struct socket *socket_list; struct protocol_handler *protocols; } net; }; 2. Microkernel Architecture Concept: Minimal kernel with essential services only; other services run in user space. Structure: User Space \u2502 \u251c\u2500 File Server \u251c\u2500 Device Drivers \u251c\u2500 Network Server \u251c\u2500 Applications \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 IPC (Message Passing) \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space (Minimal) \u2502 \u251c\u2500 IPC Mechanism \u251c\u2500 Basic Scheduling \u251c\u2500 Low-level Memory Management \u2514\u2500 Hardware Abstraction \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Essential Kernel Services: Inter-Process Communication (IPC) Basic thread scheduling Low-level address space management Interrupt handling Tip Advantages of Microkernel: Fault Isolation: Service crash doesn’t affect kernel Modularity: Easy to add\/remove services Security: Smaller trusted computing base (TCB) Portability: Hardware-specific code isolated Danger Disadvantages of Microkernel: Performance: IPC overhead for service communication Complexity: More complex system structure Development Effort: Requires careful IPC design Examples: MINIX 3, L4 microkernel family, QNX, seL4 Microkernel IPC Example: \/\/ Simplified IPC message passing struct message { int src_pid; int dst_pid; int msg_type; char data[1024]; }; \/\/ Send message to file server int read_file(const char *path, char *buffer, size_t size) { struct message msg; msg.src_pid = getpid(); msg.dst_pid = FILE_SERVER_PID; msg.msg_type = MSG_READ_FILE; strcpy(msg.data, path); \/\/ Send to microkernel IPC send_message(&msg); \/\/ Wait for reply receive_message(&msg); \/\/ Copy data from reply memcpy(buffer, msg.data, size); return msg.msg_type; \/\/ Return value } 3. Hybrid Kernel Concept: Combines monolithic and microkernel approaches; some services in kernel for performance, others in user space. Structure: User Space \u2502 \u251c\u2500 User-mode Services \u251c\u2500 Device Drivers (some) \u2514\u2500 Applications \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Mixed Interface \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space \u2502 \u251c\u2500 Core Kernel (Microkernel-like) \u2502 \u251c\u2500 IPC \u2502 \u251c\u2500 Scheduling \u2502 \u2514\u2500 Low-level MM \u2502 \u2514\u2500 Integrated Services (Monolithic-like) \u251c\u2500 File System \u251c\u2500 Device Drivers (critical) \u2514\u2500 Network Stack \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Advantages: Balances performance and modularity Critical services in kernel for speed Non-critical services in user space for safety Examples: Windows NT\/modern Windows: NT kernel (hybrid) macOS XNU: Mach microkernel + BSD monolithic components XNU (macOS\/iOS Kernel) Architecture: \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 User Space Applications \u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 System Libraries \u2502 \u2502 (libSystem, Foundation, etc.) \u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 XNU Kernel \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502 \u2502 BSD Layer (Monolithic) \u2502 \u2502 \u2502 \u2502 - POSIX comp","date":"29-Nov-2024","heading":"2. Microkernel Architecture","objectID":"\/posts\/operating-system\/:3:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#2-microkernel-architecture"},{"categories":["computer-science"],"collections":null,"content":"Kernel Architectures 1. Monolithic Kernel Concept: All OS services run in kernel space with full hardware access. Structure: User Space \u2502 \u251c\u2500 Application 1 \u251c\u2500 Application 2 \u2514\u2500 Application N \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 System Call Interface \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space (Single Address Space) \u2502 \u251c\u2500 Process Management \u251c\u2500 Memory Management \u251c\u2500 File System \u251c\u2500 Device Drivers \u251c\u2500 Network Stack \u2514\u2500 Scheduling \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Tip Advantages of Monolithic Kernel: Performance: Direct function calls within kernel; no context switching overhead Simplicity: Straightforward design and implementation Efficiency: Minimal message passing overhead Warning Disadvantages of Monolithic Kernel: Reliability: Bug in any component can crash entire system Maintainability: Large, complex codebase difficult to maintain Security: Large attack surface; any vulnerability affects whole kernel Examples: Linux, traditional UNIX, Windows 9x Linux Kernel Structure: \/\/ Simplified representation of kernel subsystems struct linux_kernel { struct process_manager { struct task_struct *process_list; struct scheduler *sched; } pm; struct memory_manager { struct page_table *pgd; struct vm_area *vma_list; } mm; struct file_system { struct super_block *sb; struct inode_cache *icache; } fs; struct network_stack { struct socket *socket_list; struct protocol_handler *protocols; } net; }; 2. Microkernel Architecture Concept: Minimal kernel with essential services only; other services run in user space. Structure: User Space \u2502 \u251c\u2500 File Server \u251c\u2500 Device Drivers \u251c\u2500 Network Server \u251c\u2500 Applications \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 IPC (Message Passing) \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space (Minimal) \u2502 \u251c\u2500 IPC Mechanism \u251c\u2500 Basic Scheduling \u251c\u2500 Low-level Memory Management \u2514\u2500 Hardware Abstraction \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Essential Kernel Services: Inter-Process Communication (IPC) Basic thread scheduling Low-level address space management Interrupt handling Tip Advantages of Microkernel: Fault Isolation: Service crash doesn’t affect kernel Modularity: Easy to add\/remove services Security: Smaller trusted computing base (TCB) Portability: Hardware-specific code isolated Danger Disadvantages of Microkernel: Performance: IPC overhead for service communication Complexity: More complex system structure Development Effort: Requires careful IPC design Examples: MINIX 3, L4 microkernel family, QNX, seL4 Microkernel IPC Example: \/\/ Simplified IPC message passing struct message { int src_pid; int dst_pid; int msg_type; char data[1024]; }; \/\/ Send message to file server int read_file(const char *path, char *buffer, size_t size) { struct message msg; msg.src_pid = getpid(); msg.dst_pid = FILE_SERVER_PID; msg.msg_type = MSG_READ_FILE; strcpy(msg.data, path); \/\/ Send to microkernel IPC send_message(&msg); \/\/ Wait for reply receive_message(&msg); \/\/ Copy data from reply memcpy(buffer, msg.data, size); return msg.msg_type; \/\/ Return value } 3. Hybrid Kernel Concept: Combines monolithic and microkernel approaches; some services in kernel for performance, others in user space. Structure: User Space \u2502 \u251c\u2500 User-mode Services \u251c\u2500 Device Drivers (some) \u2514\u2500 Applications \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Mixed Interface \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space \u2502 \u251c\u2500 Core Kernel (Microkernel-like) \u2502 \u251c\u2500 IPC \u2502 \u251c\u2500 Scheduling \u2502 \u2514\u2500 Low-level MM \u2502 \u2514\u2500 Integrated Services (Monolithic-like) \u251c\u2500 File System \u251c\u2500 Device Drivers (critical) \u2514\u2500 Network Stack \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Advantages: Balances performance and modularity Critical services in kernel for speed Non-critical services in user space for safety Examples: Windows NT\/modern Windows: NT kernel (hybrid) macOS XNU: Mach microkernel + BSD monolithic components XNU (macOS\/iOS Kernel) Architecture: \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 User Space Applications \u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 System Libraries \u2502 \u2502 (libSystem, Foundation, etc.) \u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 XNU Kernel \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502 \u2502 BSD Layer (Monolithic) \u2502 \u2502 \u2502 \u2502 - POSIX comp","date":"29-Nov-2024","heading":"3. Hybrid Kernel","objectID":"\/posts\/operating-system\/:3:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#3-hybrid-kernel"},{"categories":["computer-science"],"collections":null,"content":"Kernel Architectures 1. Monolithic Kernel Concept: All OS services run in kernel space with full hardware access. Structure: User Space \u2502 \u251c\u2500 Application 1 \u251c\u2500 Application 2 \u2514\u2500 Application N \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 System Call Interface \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space (Single Address Space) \u2502 \u251c\u2500 Process Management \u251c\u2500 Memory Management \u251c\u2500 File System \u251c\u2500 Device Drivers \u251c\u2500 Network Stack \u2514\u2500 Scheduling \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Tip Advantages of Monolithic Kernel: Performance: Direct function calls within kernel; no context switching overhead Simplicity: Straightforward design and implementation Efficiency: Minimal message passing overhead Warning Disadvantages of Monolithic Kernel: Reliability: Bug in any component can crash entire system Maintainability: Large, complex codebase difficult to maintain Security: Large attack surface; any vulnerability affects whole kernel Examples: Linux, traditional UNIX, Windows 9x Linux Kernel Structure: \/\/ Simplified representation of kernel subsystems struct linux_kernel { struct process_manager { struct task_struct *process_list; struct scheduler *sched; } pm; struct memory_manager { struct page_table *pgd; struct vm_area *vma_list; } mm; struct file_system { struct super_block *sb; struct inode_cache *icache; } fs; struct network_stack { struct socket *socket_list; struct protocol_handler *protocols; } net; }; 2. Microkernel Architecture Concept: Minimal kernel with essential services only; other services run in user space. Structure: User Space \u2502 \u251c\u2500 File Server \u251c\u2500 Device Drivers \u251c\u2500 Network Server \u251c\u2500 Applications \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 IPC (Message Passing) \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space (Minimal) \u2502 \u251c\u2500 IPC Mechanism \u251c\u2500 Basic Scheduling \u251c\u2500 Low-level Memory Management \u2514\u2500 Hardware Abstraction \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Essential Kernel Services: Inter-Process Communication (IPC) Basic thread scheduling Low-level address space management Interrupt handling Tip Advantages of Microkernel: Fault Isolation: Service crash doesn’t affect kernel Modularity: Easy to add\/remove services Security: Smaller trusted computing base (TCB) Portability: Hardware-specific code isolated Danger Disadvantages of Microkernel: Performance: IPC overhead for service communication Complexity: More complex system structure Development Effort: Requires careful IPC design Examples: MINIX 3, L4 microkernel family, QNX, seL4 Microkernel IPC Example: \/\/ Simplified IPC message passing struct message { int src_pid; int dst_pid; int msg_type; char data[1024]; }; \/\/ Send message to file server int read_file(const char *path, char *buffer, size_t size) { struct message msg; msg.src_pid = getpid(); msg.dst_pid = FILE_SERVER_PID; msg.msg_type = MSG_READ_FILE; strcpy(msg.data, path); \/\/ Send to microkernel IPC send_message(&msg); \/\/ Wait for reply receive_message(&msg); \/\/ Copy data from reply memcpy(buffer, msg.data, size); return msg.msg_type; \/\/ Return value } 3. Hybrid Kernel Concept: Combines monolithic and microkernel approaches; some services in kernel for performance, others in user space. Structure: User Space \u2502 \u251c\u2500 User-mode Services \u251c\u2500 Device Drivers (some) \u2514\u2500 Applications \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Mixed Interface \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space \u2502 \u251c\u2500 Core Kernel (Microkernel-like) \u2502 \u251c\u2500 IPC \u2502 \u251c\u2500 Scheduling \u2502 \u2514\u2500 Low-level MM \u2502 \u2514\u2500 Integrated Services (Monolithic-like) \u251c\u2500 File System \u251c\u2500 Device Drivers (critical) \u2514\u2500 Network Stack \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Advantages: Balances performance and modularity Critical services in kernel for speed Non-critical services in user space for safety Examples: Windows NT\/modern Windows: NT kernel (hybrid) macOS XNU: Mach microkernel + BSD monolithic components XNU (macOS\/iOS Kernel) Architecture: \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 User Space Applications \u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 System Libraries \u2502 \u2502 (libSystem, Foundation, etc.) \u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 XNU Kernel \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502 \u2502 BSD Layer (Monolithic) \u2502 \u2502 \u2502 \u2502 - POSIX comp","date":"29-Nov-2024","heading":"4. Exokernel","objectID":"\/posts\/operating-system\/:3:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#4-exokernel"},{"categories":["computer-science"],"collections":null,"content":"Kernel Architectures 1. Monolithic Kernel Concept: All OS services run in kernel space with full hardware access. Structure: User Space \u2502 \u251c\u2500 Application 1 \u251c\u2500 Application 2 \u2514\u2500 Application N \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 System Call Interface \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space (Single Address Space) \u2502 \u251c\u2500 Process Management \u251c\u2500 Memory Management \u251c\u2500 File System \u251c\u2500 Device Drivers \u251c\u2500 Network Stack \u2514\u2500 Scheduling \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Tip Advantages of Monolithic Kernel: Performance: Direct function calls within kernel; no context switching overhead Simplicity: Straightforward design and implementation Efficiency: Minimal message passing overhead Warning Disadvantages of Monolithic Kernel: Reliability: Bug in any component can crash entire system Maintainability: Large, complex codebase difficult to maintain Security: Large attack surface; any vulnerability affects whole kernel Examples: Linux, traditional UNIX, Windows 9x Linux Kernel Structure: \/\/ Simplified representation of kernel subsystems struct linux_kernel { struct process_manager { struct task_struct *process_list; struct scheduler *sched; } pm; struct memory_manager { struct page_table *pgd; struct vm_area *vma_list; } mm; struct file_system { struct super_block *sb; struct inode_cache *icache; } fs; struct network_stack { struct socket *socket_list; struct protocol_handler *protocols; } net; }; 2. Microkernel Architecture Concept: Minimal kernel with essential services only; other services run in user space. Structure: User Space \u2502 \u251c\u2500 File Server \u251c\u2500 Device Drivers \u251c\u2500 Network Server \u251c\u2500 Applications \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 IPC (Message Passing) \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space (Minimal) \u2502 \u251c\u2500 IPC Mechanism \u251c\u2500 Basic Scheduling \u251c\u2500 Low-level Memory Management \u2514\u2500 Hardware Abstraction \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Essential Kernel Services: Inter-Process Communication (IPC) Basic thread scheduling Low-level address space management Interrupt handling Tip Advantages of Microkernel: Fault Isolation: Service crash doesn’t affect kernel Modularity: Easy to add\/remove services Security: Smaller trusted computing base (TCB) Portability: Hardware-specific code isolated Danger Disadvantages of Microkernel: Performance: IPC overhead for service communication Complexity: More complex system structure Development Effort: Requires careful IPC design Examples: MINIX 3, L4 microkernel family, QNX, seL4 Microkernel IPC Example: \/\/ Simplified IPC message passing struct message { int src_pid; int dst_pid; int msg_type; char data[1024]; }; \/\/ Send message to file server int read_file(const char *path, char *buffer, size_t size) { struct message msg; msg.src_pid = getpid(); msg.dst_pid = FILE_SERVER_PID; msg.msg_type = MSG_READ_FILE; strcpy(msg.data, path); \/\/ Send to microkernel IPC send_message(&msg); \/\/ Wait for reply receive_message(&msg); \/\/ Copy data from reply memcpy(buffer, msg.data, size); return msg.msg_type; \/\/ Return value } 3. Hybrid Kernel Concept: Combines monolithic and microkernel approaches; some services in kernel for performance, others in user space. Structure: User Space \u2502 \u251c\u2500 User-mode Services \u251c\u2500 Device Drivers (some) \u2514\u2500 Applications \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Mixed Interface \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kernel Space \u2502 \u251c\u2500 Core Kernel (Microkernel-like) \u2502 \u251c\u2500 IPC \u2502 \u251c\u2500 Scheduling \u2502 \u2514\u2500 Low-level MM \u2502 \u2514\u2500 Integrated Services (Monolithic-like) \u251c\u2500 File System \u251c\u2500 Device Drivers (critical) \u2514\u2500 Network Stack \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Hardware \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Advantages: Balances performance and modularity Critical services in kernel for speed Non-critical services in user space for safety Examples: Windows NT\/modern Windows: NT kernel (hybrid) macOS XNU: Mach microkernel + BSD monolithic components XNU (macOS\/iOS Kernel) Architecture: \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 User Space Applications \u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 System Libraries \u2502 \u2502 (libSystem, Foundation, etc.) \u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 XNU Kernel \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502 \u2502 BSD Layer (Monolithic) \u2502 \u2502 \u2502 \u2502 - POSIX comp","date":"29-Nov-2024","heading":"5. Unikernel","objectID":"\/posts\/operating-system\/:3:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#5-unikernel"},{"categories":["computer-science"],"collections":null,"content":"System Layers User Mode vs Kernel Mode Modern processors provide privilege levels (protection rings) to enforce security and stability. x86 Protection Rings: Ring 3 (Lowest Privilege) \u2500\u2500\u2500 User Applications Ring 2 (Device Drivers - rarely used) Ring 1 (Device Drivers - rarely used) Ring 0 (Highest Privilege) \u2500\u2500\u2500 Operating System Kernel ARM Exception Levels (ARMv8): EL0 \u2500\u2500\u2500 User Applications EL1 \u2500\u2500\u2500 Operating System Kernel EL2 \u2500\u2500\u2500 Hypervisor EL3 \u2500\u2500\u2500 Secure Monitor (TrustZone) Mode Transition: User programs cannot directly access hardware or privileged instructions. They must request OS services via system calls, which trigger a mode switch. Mode Switch Process: 1. Application executes syscall instruction (INT 0x80, SYSCALL, SVC) 2. CPU switches from User Mode to Kernel Mode 3. CPU saves user context (registers, program counter) 4. CPU jumps to kernel's system call handler 5. Kernel validates request and executes service 6. Kernel prepares return value 7. CPU restores user context 8. CPU switches back to User Mode 9. Application resumes with result Protection Benefits: Isolation: Faulty applications can’t crash the system Security: Malicious code can’t directly manipulate hardware Stability: Critical OS data structures protected from corruption ","date":"29-Nov-2024","heading":"System Layers","objectID":"\/posts\/operating-system\/:3:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#system-layers"},{"categories":["computer-science"],"collections":null,"content":"4. Boot Process and System Initialization ","date":"29-Nov-2024","heading":"4. Boot Process and System Initialization","objectID":"\/posts\/operating-system\/:4:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#boot-process"},{"categories":["computer-science"],"collections":null,"content":"x86 Boot Sequence 1. Power-On Self-Test (POST) Power On \u2192 CPU Reset \u2192 POST \u2502 \u251c\u2500 Test CPU registers \u251c\u2500 Check memory \u251c\u2500 Initialize chipset \u251c\u2500 Detect hardware \u2514\u2500 Display boot screen 2. BIOS\/UEFI Initialization Legacy BIOS (Basic Input\/Output System): Stored in ROM\/flash on motherboard Executes in 16-bit real mode Loads Master Boot Record (MBR) from boot device MBR Structure (512 bytes): Offset Size Description 0x000 446 Boot code 0x1BE 16 Partition entry 1 0x1CE 16 Partition entry 2 0x1DE 16 Partition entry 3 0x1EE 16 Partition entry 4 0x1FE 2 Boot signature (0x55AA) Simple MBR Boot Code (x86 Assembly): ; boot.asm - Minimal bootloader BITS 16 ; 16-bit real mode ORG 0x7C00 ; BIOS loads us at 0x7C00 start: ; Setup segments xor ax, ax mov ds, ax mov es, ax mov ss, ax mov sp, 0x7C00 ; Display message mov si, message call print_string ; Hang jmp $ print_string: lodsb ; Load byte from [SI] into AL or al, al ; Check if zero (end of string) jz done mov ah, 0x0E ; BIOS teletype function int 0x10 ; Call BIOS video interrupt jmp print_string done: ret message db 'Booting...', 0x0D, 0x0A, 0 times 510-($-$$) db 0 ; Pad to 510 bytes dw 0xAA55 ; Boot signature UEFI (Unified Extensible Firmware Interface): Modern replacement for BIOS Supports 32-bit and 64-bit modes GPT (GUID Partition Table) instead of MBR More secure (Secure Boot) Includes a boot manager UEFI Boot Process: Power On \u2192 UEFI Firmware \u2192 Boot Manager \u2192 Boot Loader \u2192 OS Kernel 3. Bootloader Execution GRUB (Grand Unified Bootloader): Stage 1: Bootstrap code (in MBR or GPT) \u2193 Stage 1.5: Filesystem drivers (if needed) \u2193 Stage 2: Full GRUB with menu \u2193 Kernel Loading: Load kernel image into memory \u2193 Kernel Execution: Jump to kernel entry point Multiboot Specification: \/\/ Multiboot header (must be in first 8KB of kernel) #define MULTIBOOT_HEADER_MAGIC 0x1BADB002 #define MULTIBOOT_HEADER_FLAGS 0x00000003 .section .multiboot .align 4 multiboot_header: .long MULTIBOOT_HEADER_MAGIC .long MULTIBOOT_HEADER_FLAGS .long -(MULTIBOOT_HEADER_MAGIC + MULTIBOOT_HEADER_FLAGS) 4. Protected Mode Transition Real mode has limitations: Only 1 MB addressable memory No memory protection 16-bit operations Entering Protected Mode: ; Disable interrupts cli ; Load Global Descriptor Table (GDT) lgdt [gdt_descriptor] ; Set PE (Protection Enable) bit in CR0 mov eax, cr0 or eax, 0x1 mov cr0, eax ; Far jump to load CS with code segment selector jmp 0x08:protected_mode [BITS 32] protected_mode: ; Setup data segments mov ax, 0x10 ; Data segment selector mov ds, ax mov es, ax mov fs, ax mov gs, ax mov ss, ax ; Setup stack mov esp, 0x90000 ; Jump to kernel call kernel_main Global Descriptor Table (GDT): struct gdt_entry { uint16_t limit_low; uint16_t base_low; uint8_t base_middle; uint8_t access; uint8_t granularity; uint8_t base_high; } __attribute__((packed)); struct gdt_ptr { uint16_t limit; uint32_t base; } __attribute__((packed)); struct gdt_entry gdt[5]; void gdt_set_gate(int num, uint32_t base, uint32_t limit, uint8_t access, uint8_t gran) { gdt[num].base_low = (base & 0xFFFF); gdt[num].base_middle = (base >> 16) & 0xFF; gdt[num].base_high = (base >> 24) & 0xFF; gdt[num].limit_low = (limit & 0xFFFF); gdt[num].granularity = ((limit >> 16) & 0x0F) | (gran & 0xF0); gdt[num].access = access; } void gdt_install() { struct gdt_ptr gp; gp.limit = (sizeof(struct gdt_entry) * 5) - 1; gp.base = (uint32_t)&gdt; gdt_set_gate(0, 0, 0, 0, 0); \/\/ Null segment gdt_set_gate(1, 0, 0xFFFFFFFF, 0x9A, 0xCF); \/\/ Code segment gdt_set_gate(2, 0, 0xFFFFFFFF, 0x92, 0xCF); \/\/ Data segment gdt_set_gate(3, 0, 0xFFFFFFFF, 0xFA, 0xCF); \/\/ User code gdt_set_gate(4, 0, 0xFFFFFFFF, 0xF2, 0xCF); \/\/ User data gdt_flush(&gp); \/\/ Assembly function to load GDT } ","date":"29-Nov-2024","heading":"x86 Boot Sequence","objectID":"\/posts\/operating-system\/:4:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#x86-boot-sequence"},{"categories":["computer-science"],"collections":null,"content":"ARM Boot Process 1. ROM Bootloader (Primary Bootloader) Hardcoded in SoC ROM Minimal functionality Loads secondary bootloader from boot media (SD, eMMC, SPI) 2. SPL (Secondary Program Loader) \/ U-Boot Initializes DRAM controller Sets up clocks and power management Loads main bootloader or kernel U-Boot Commands: # Load kernel from SD card fatload mmc 0:1 ${kernel_addr_r} zImage # Load device tree fatload mmc 0:1 ${fdt_addr_r} bcm2710-rpi-3-b.dtb # Boot kernel bootz ${kernel_addr_r} - ${fdt_addr_r} 3. Device Tree (DTB) Hardware description format Describes hardware topology, addresses, interrupts Passed to kernel during boot Device Tree Example (Simplified): \/dts-v1\/; \/ { compatible = \"raspberrypi,3-model-b\", \"brcm,bcm2837\"; model = \"Raspberry Pi 3 Model B\"; memory@0 { device_type = \"memory\"; reg = <0x00000000 0x40000000>; \/\/ 1GB }; cpus { #address-cells = <1>; #size-cells = <0>; cpu@0 { device_type = \"cpu\"; compatible = \"arm,cortex-a53\"; reg = <0>; }; \/\/ ... more CPUs }; soc { uart0: serial@7e201000 { compatible = \"brcm,bcm2835-uart\"; reg = <0x7e201000 0x1000>; interrupts = <2 25>; }; }; }; 4. Kernel Initialization Linux Kernel Boot (ARM64): 1. Decompress kernel (if compressed) 2. Setup initial page tables 3. Enable MMU and caches 4. Parse device tree 5. Initialize memory subsystem 6. Mount initial ramdisk (initramfs) 7. Initialize drivers 8. Start init process (PID 1) ","date":"29-Nov-2024","heading":"ARM Boot Process","objectID":"\/posts\/operating-system\/:4:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#arm-boot-process"},{"categories":["computer-science"],"collections":null,"content":"Kernel Initialization Deep Dive Early Boot Process: \/\/ Simplified Linux kernel initialization (kernel\/init\/main.c) asmlinkage __visible void __init start_kernel(void) { char *command_line; \/\/ 1. Setup architecture-specific components setup_arch(&command_line); \/\/ 2. Initialize memory management mm_init(); \/\/ 3. Setup scheduler sched_init(); \/\/ 4. Initialize IRQs and timers init_IRQ(); init_timers(); \/\/ 5. Initialize console for early output console_init(); \/\/ 6. Initialize rest of kernel subsystems vfs_caches_init(); signals_init(); \/\/ 7. Rest of initialization (drivers, etc.) rest_init(); } static noinline void __init rest_init(void) { \/\/ Create kernel threads kernel_thread(kernel_init, NULL, CLONE_FS); kernel_thread(kthreadd, NULL, CLONE_FS | CLONE_FILES); \/\/ CPU idle loop cpu_startup_entry(CPUHP_ONLINE); } static int __init kernel_init(void *unused) { \/\/ Try to run init program if (!try_to_run_init_process(\"\/sbin\/init\") || !try_to_run_init_process(\"\/etc\/init\") || !try_to_run_init_process(\"\/bin\/init\") || !try_to_run_init_process(\"\/bin\/sh\")) return 0; panic(\"No working init found.\"); } ","date":"29-Nov-2024","heading":"Kernel Initialization Deep Dive","objectID":"\/posts\/operating-system\/:4:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#kernel-initialization-deep-dive"},{"categories":["computer-science"],"collections":null,"content":"5. Process Management ","date":"29-Nov-2024","heading":"5. Process Management","objectID":"\/posts\/operating-system\/:5:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#process-management"},{"categories":["computer-science"],"collections":null,"content":"Process Concept A process is an instance of a program in execution. It represents the dynamic, active entity, as opposed to a program which is a passive, static collection of instructions. Process Components: Program Code (Text Section): Executable instructions Program Counter: Current instruction position Processor Registers: Register contents Stack: Temporary data (function parameters, return addresses, local variables) Data Section: Global variables Heap: Dynamically allocated memory Process vs Program: Program: Passive entity (file on disk containing instructions) Process: Active entity (program loaded in memory and executing) Example: - \"notepad.exe\" is a program - When you run it, each window is a separate process ","date":"29-Nov-2024","heading":"Process Concept","objectID":"\/posts\/operating-system\/:5:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#process-concept"},{"categories":["computer-science"],"collections":null,"content":"Process Memory Layout High Memory (0xFFFFFFFF) \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 Kernel Space \u2502 \u2190 OS kernel (1GB on 32-bit Linux) \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 Stack \u2502 \u2190 Grows downward \u2502 \u2193 \u2502 Function calls, local variables \u2502 \u2502 \u2502 (Unused) \u2502 \u2502 \u2502 \u2502 \u2191 \u2502 \u2502 Heap \u2502 \u2190 Grows upward \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 Dynamic allocation (malloc\/new) \u2502 BSS Segment \u2502 \u2190 Uninitialized data \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 Data Segment \u2502 \u2190 Initialized data \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 Text Segment \u2502 \u2190 Program code (read-only) \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 Low Memory (0x00000000) Example: #include int global_initialized = 42; \/\/ Data segment int global_uninitialized; \/\/ BSS segment const char *text_data = \"Hello\"; \/\/ Text\/Read-only segment void function() { int stack_var = 10; \/\/ Stack int *heap_var = malloc(sizeof(int)); \/\/ Heap *heap_var = 20; free(heap_var); } ","date":"29-Nov-2024","heading":"Process Memory Layout","objectID":"\/posts\/operating-system\/:5:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#process-memory-layout"},{"categories":["computer-science"],"collections":null,"content":"Process Control Block (PCB) The PCB (also called Task Control Block or TCB) is the kernel data structure representing a process. PCB Contents: struct task_struct { \/\/ Linux representation (simplified) \/\/ 1. Process State volatile long state; \/\/ Running, Ready, Waiting, etc. \/\/ 2. Process ID pid_t pid; \/\/ Process ID pid_t tgid; \/\/ Thread group ID struct task_struct *parent; \/\/ Parent process \/\/ 3. CPU Scheduling Information int prio; \/\/ Priority int static_prio; \/\/ Static priority int normal_prio; \/\/ Normal priority unsigned int policy; \/\/ Scheduling policy struct sched_entity se; \/\/ Scheduling entity \/\/ 4. Memory Management struct mm_struct *mm; \/\/ Memory descriptor \/\/ 5. File Descriptors struct files_struct *files; \/\/ Open file information \/\/ 6. CPU Context struct thread_struct thread; \/\/ CPU registers, PC, SP \/\/ 7. Signal Handling struct signal_struct *signal; \/\/ 8. Process Times u64 utime; \/\/ User mode time u64 stime; \/\/ Kernel mode time \/\/ 9. Process Credentials uid_t uid, euid; \/\/ User ID, Effective UID gid_t gid, egid; \/\/ Group ID, Effective GID \/\/ 10. List Management struct list_head tasks; \/\/ Process list struct list_head sibling; \/\/ Sibling list struct list_head children; \/\/ Children list }; ","date":"29-Nov-2024","heading":"Process Control Block (PCB)","objectID":"\/posts\/operating-system\/:5:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#process-control-block-pcb"},{"categories":["computer-science"],"collections":null,"content":"Process States A process transitions through various states during its lifetime. Five-State Model: \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 New \u2502 \u2190 Process being created \u2514\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2518 \u2502 \u2193 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 Scheduler \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2192\u2502 Ready \u2502 \u2190 \u2500 \u2500 \u2500 \u2500 \u2500 \u2500 \u2500 \u2500 \u2524 Running \u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2518 Dispatch \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502 \u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502 \u2502 \u2502 I\/O or \u2502 I\/O Complete \u2502 Event Wait \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502 \u2502 \u2193 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2514\u2500\u2500\u2500\u2500\u2524 Waiting \u2502 \u2502 Terminated \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 State Descriptions: New: Process is being created; PCB allocated but not ready Ready: Process is waiting to be assigned to CPU Running: Instructions are being executed Waiting (Blocked): Process waiting for event (I\/O completion, signal) Terminated (Zombie): Process finished execution; waiting for parent to read exit status State Transitions Example (Linux): \/\/ Process states in Linux (linux\/sched.h) #define TASK_RUNNING 0 #define TASK_INTERRUPTIBLE 1 \/\/ Can be woken by signals #define TASK_UNINTERRUPTIBLE 2 \/\/ Cannot be interrupted #define __TASK_STOPPED 4 #define __TASK_TRACED 8 #define EXIT_ZOMBIE 16 #define EXIT_DEAD 32 \/\/ State transition example void state_transition() { set_current_state(TASK_INTERRUPTIBLE); \/\/ Ready \u2192 Waiting schedule(); \/\/ Yield CPU \/\/ After wake-up \/\/ Waiting \u2192 Ready \u2192 Running (when scheduled) } ","date":"29-Nov-2024","heading":"Process States","objectID":"\/posts\/operating-system\/:5:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#process-states"},{"categories":["computer-science"],"collections":null,"content":"Process Creation UNIX: fork() System Call fork() creates a new process by duplicating the calling process. Characteristics: Child process gets a copy of parent’s address space (Copy-on-Write) Child has unique PID fork() returns twice: 0 in child, child’s PID in parent Tip Copy-on-Write Optimization: fork() doesn’t immediately copy all memory! Instead, parent and child share the same pages until one tries to write. This makes fork() extremely fast and memory-efficient, even for large processes. Fork Example: #include #include #include int main() { pid_t pid; int x = 10; printf(\"Before fork: x = %d\\n\", x); pid = fork(); if (pid < 0) { \/\/ Fork failed fprintf(stderr, \"Fork failed\\n\"); return 1; } else if (pid == 0) { \/\/ Child process x = 20; printf(\"Child: x = %d, PID = %d\\n\", x, getpid()); } else { \/\/ Parent process x = 30; printf(\"Parent: x = %d, Child PID = %d\\n\", x, pid); } return 0; } \/* Output: Before fork: x = 10 Child: x = 20, PID = 1235 Parent: x = 30, Child PID = 1235 *\/ exec() Family exec() replaces the current process image with a new program. Common exec variants: execl(): List of arguments execv(): Vector (array) of arguments execle(): List + environment execve(): Vector + environment Fork + Exec Pattern: #include #include #include int main() { pid_t pid = fork(); if (pid == 0) { \/\/ Child: Replace with \"\/bin\/ls\" program char *args[] = {\"\/bin\/ls\", \"-l\", \"\/tmp\", NULL}; execv(\"\/bin\/ls\", args); \/\/ If execv returns, it failed perror(\"execv failed\"); return 1; } else { \/\/ Parent: Wait for child to finish int status; waitpid(pid, &status, 0); printf(\"Child exited with status %d\\n\", WEXITSTATUS(status)); } return 0; } Windows: CreateProcess() Windows doesn’t use fork(); it creates processes differently. #include #include int main() { STARTUPINFO si; PROCESS_INFORMATION pi; ZeroMemory(&si, sizeof(si)); si.cb = sizeof(si); ZeroMemory(&pi, sizeof(pi)); \/\/ Create child process if (!CreateProcess( NULL, \/\/ Application name \"notepad.exe\", \/\/ Command line NULL, \/\/ Process attributes NULL, \/\/ Thread attributes FALSE, \/\/ Inherit handles 0, \/\/ Creation flags NULL, \/\/ Environment NULL, \/\/ Current directory &si, \/\/ Startup info &pi \/\/ Process info )) { printf(\"CreateProcess failed (%d)\\n\", GetLastError()); return 1; } \/\/ Wait for child WaitForSingleObject(pi.hProcess, INFINITE); \/\/ Close handles CloseHandle(pi.hProcess); CloseHandle(pi.hThread); return 0; } ","date":"29-Nov-2024","heading":"Process Creation","objectID":"\/posts\/operating-system\/:5:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#process-creation"},{"categories":["computer-science"],"collections":null,"content":"Process Termination Normal Termination: Program executes last statement Calls exit() system call Returns from main() Abnormal Termination: Fatal error (division by zero, segmentation fault) Killed by another process (kill command) Parent terminates (orphan processes) Exit Status: #include int main() { \/\/ exit(0) - Success \/\/ exit(1) - General error \/\/ exit(127) - Command not found if (error_occurred) { exit(EXIT_FAILURE); } return EXIT_SUCCESS; } Zombie and Orphan Processes: Zombie Process: Process terminated but PCB still exists Waiting for parent to read exit status via wait() Shows as <defunct> in ps output Warning Zombie Processes: If a parent never calls wait(), the child’s PCB remains allocated even after the child terminates. This wastes kernel resources. Long-running parent processes should always wait() for their children or ignore SIGCHLD to automatically reap zombies. \/\/ Creating a zombie #include int main() { if (fork() == 0) { \/\/ Child exits immediately exit(0); } \/\/ Parent doesn't call wait() sleep(60); \/\/ Child becomes zombie for 60 seconds return 0; } Orphan Process: Parent terminates before child Child is adopted by init (PID 1) or systemd Init\/systemd periodically calls wait() to reap orphans Process Wait Example: #include #include #include int main() { pid_t pid = fork(); if (pid == 0) { printf(\"Child process\\n\"); sleep(2); exit(42); \/\/ Exit with status 42 } else { int status; pid_t child = wait(&status); \/\/ Wait for any child if (WIFEXITED(status)) { printf(\"Child %d exited with status %d\\n\", child, WEXITSTATUS(status)); } \/\/ Alternative: waitpid() for specific child \/\/ waitpid(pid, &status, 0); } return 0; } ","date":"29-Nov-2024","heading":"Process Termination","objectID":"\/posts\/operating-system\/:5:6","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#process-termination"},{"categories":["computer-science"],"collections":null,"content":"Context Switching Context Switch: Saving the state of current process and loading the state of the next process. Context Switch Steps: 1. Save current process state (registers, PC, SP) to PCB 2. Update PCB (state, timing information) 3. Move PCB to appropriate queue (ready, waiting) 4. Select next process (scheduling algorithm) 5. Update new process PCB (state = RUNNING) 6. Restore new process context from PCB 7. Switch memory management (page tables) 8. Resume execution of new process Info Context Switch Overhead - A Hidden Cost: Direct Cost: Time to save\/restore registers (~microseconds) Indirect Cost: Cache flush \u2192 huge performance hit! Modern CPUs with large caches can lose 1-10% performance per context switch TLB Flush: Virtual memory translation caches invalidated Pipeline Flush: CPU instruction pipeline emptied Practical Impact: High context switch rates (>1000\/sec) cause significant throughput degradation even though each individual switch is “fast” Minimizing Context Switches: Use threads instead of processes (shared address space) Increase time slices (but hurts responsiveness) CPU affinity (keep process on same CPU) Reduce I\/O operations (batch I\/O requests) Linux Context Switch Implementation (Simplified): \/\/ arch\/x86\/include\/asm\/switch_to.h #define switch_to(prev, next, last) do { \\ \/* Save previous task's context *\/ \\ asm volatile( \\ \"pushfl\\n\\t\" \/* Save flags *\/ \\ \"pushl %%ebp\\n\\t\" \/* Save base pointer *\/\\ \"movl %%esp,%[prev_sp]\\n\\t\" \/* Save stack ptr *\/ \\ \"movl %[next_sp],%%esp\\n\\t\" \/* Load next stack*\/\\ \"movl $1f,%[prev_ip]\\n\\t\" \/* Save return IP *\/ \\ \"pushl %[next_ip]\\n\\t\" \/* Push next IP *\/ \\ \"jmp __switch_to\\n\\t\" \/* Context switch *\/ \\ \"1:\\t\" \\ \"popl %%ebp\\n\\t\" \/* Restore base ptr *\/ \\ \"popfl\\n\" \/* Restore flags *\/ \\ : [prev_sp] \"=m\" (prev->thread.sp), \\ [prev_ip] \"=m\" (prev->thread.ip) \\ : [next_sp] \"m\" (next->thread.sp), \\ [next_ip] \"m\" (next->thread.ip), \\ [prev] \"a\" (prev), \\ [next] \"d\" (next) \\ : \"memory\", \"cc\" \\ ); \\ } while (0) ","date":"29-Nov-2024","heading":"Context Switching","objectID":"\/posts\/operating-system\/:5:7","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#context-switching"},{"categories":["computer-science"],"collections":null,"content":"Process Hierarchies UNIX Process Tree: init (PID 1) or systemd \u251c\u2500 sshd \u2502 \u2514\u2500 bash (login shell) \u2502 \u2514\u2500 vim \u251c\u2500 cron \u2502 \u2514\u2500 backup_script \u251c\u2500 Apache httpd \u2502 \u251c\u2500 httpd (worker 1) \u2502 \u251c\u2500 httpd (worker 2) \u2502 \u2514\u2500 httpd (worker N) \u2514\u2500 X Window System \u2514\u2500 window_manager \u251c\u2500 terminal_emulator \u2514\u2500 browser Viewing Process Tree (Linux): $ pstree init\u2500\u252c\u2500acpid \u251c\u2500cron \u251c\u2500dbus-daemon \u251c\u2500dhclient \u251c\u2500login\u2500\u2500\u2500bash\u2500\u2500\u2500pstree \u251c\u2500rsyslogd\u2500\u2500\u25003*[{rsyslogd}] \u251c\u2500sshd\u2500\u2500\u2500sshd\u2500\u2500\u2500bash \u2514\u2500udevd\u2500\u2500\u25002*[udevd] $ ps -ef # Full process list $ ps aux # BSD-style format ","date":"29-Nov-2024","heading":"Process Hierarchies","objectID":"\/posts\/operating-system\/:5:8","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#process-hierarchies"},{"categories":["computer-science"],"collections":null,"content":"6. Thread Management and Concurrency ","date":"29-Nov-2024","heading":"6. Thread Management and Concurrency","objectID":"\/posts\/operating-system\/:6:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#thread-management"},{"categories":["computer-science"],"collections":null,"content":"Thread Concept A thread is the smallest unit of execution within a process. Unlike processes which have separate address spaces, threads share the same memory space and resources, making them lightweight and efficient. Key Differences: Process vs Thread Aspect Process Thread Memory Space Separate address space Shared address space Resource Overhead High (memory, file descriptors) Low (shares with process) Creation Time Slow (milliseconds) Fast (microseconds) Context Switch Expensive (page tables, cache misses) Cheaper (same page tables) Communication IPC mechanisms needed Direct memory access Isolation High (protected) Low (shared memory) Thread Components: Each thread has its own: \u251c\u2500\u2500 Program Counter (PC) \u251c\u2500\u2500 Stack (local variables, function calls) \u251c\u2500\u2500 CPU Registers \u251c\u2500\u2500 Thread-local storage \u2514\u2500\u2500 Stack pointer Shared by all threads in process: \u251c\u2500\u2500 Process memory (heap, code, globals) \u251c\u2500\u2500 File descriptors \u251c\u2500\u2500 Signal handlers \u2514\u2500\u2500 Current working directory ","date":"29-Nov-2024","heading":"Thread Concept","objectID":"\/posts\/operating-system\/:6:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#thread-concept"},{"categories":["computer-science"],"collections":null,"content":"User-Level vs Kernel-Level Threads User-Level Threads Managed by thread library (not OS) No kernel involvement Fast context switching Cannot utilize multiprocessors If one thread blocks, entire process blocked \/\/ User-level thread example (POSIX Threads) #include #include void* thread_function(void *arg) { printf(\"Thread ID: %ld\\n\", (long)pthread_self()); return NULL; } int main() { pthread_t thread; \/\/ Create user-level thread pthread_create(&thread, NULL, thread_function, NULL); \/\/ Wait for thread completion pthread_join(thread, NULL); return 0; } Kernel-Level Threads Managed by OS kernel Full kernel support Can run on multiple processors Slower context switching (syscall overhead) Better for I\/O-bound operations ","date":"29-Nov-2024","heading":"User-Level vs Kernel-Level Threads","objectID":"\/posts\/operating-system\/:6:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#user-level-vs-kernel-level-threads"},{"categories":["computer-science"],"collections":null,"content":"Thread Models 1. One-to-One Model User Thread 1 \u2194 Kernel Thread 1 User Thread 2 \u2194 Kernel Thread 2 User Thread 3 \u2194 Kernel Thread 3 Advantages: - Full parallelism - Blocking doesn't stop other threads - OS sees all threads Disadvantages: - High kernel overhead - Limited number of threads - Resource consumption 2. Many-to-One Model User Thread 1 \u2510 User Thread 2 \u251c\u2192 Kernel Thread 1 User Thread 3 \u2518 Advantages: - Low overhead - Fast context switching - No syscalls needed Disadvantages: - No true parallelism - One blocking thread blocks all - Cannot use multiprocessors 3. Many-to-Many Model User Thread 1 \u2510 \u250c\u2192 Kernel Thread 1 User Thread 2 \u251c\u2500\u2192 \u251c\u2192 Kernel Thread 2 User Thread 3 \u2524 \u2514\u2192 Kernel Thread 3 User Thread 4 \u2518 Best of both worlds: - Multiple kernel threads - Efficient thread management - True parallelism with low overhead ","date":"29-Nov-2024","heading":"Thread Models","objectID":"\/posts\/operating-system\/:6:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#thread-models"},{"categories":["computer-science"],"collections":null,"content":"Thread Synchronization Critical Section: Code segment that accesses shared resources Mutual Exclusion Problem: Ensuring only one thread enters critical section at a time Warning Race Condition Danger: Without synchronization, the statement shared_counter++ (which consists of 3 operations: READ \u2192 INCREMENT \u2192 WRITE) can interleave between threads, causing lost updates. For example: Thread 1: READ (10) \u2192 INCREMENT (11) Thread 2: READ (10) \u2192 INCREMENT (11) \u2190 Oops! Both read 10 Thread 1: WRITE (11) Thread 2: WRITE (11) \u2190 Expected 12, got 11! Solution 1: Mutex (Mutual Exclusion Lock) #include #include pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER; int shared_counter = 0; void* increment_counter(void *arg) { for (int i = 0; i < 1000000; i++) { pthread_mutex_lock(&mutex); shared_counter++; \/\/ Critical section pthread_mutex_unlock(&mutex); } return NULL; } int main() { pthread_t threads[2]; pthread_create(&threads[0], NULL, increment_counter, NULL); pthread_create(&threads[1], NULL, increment_counter, NULL); pthread_join(threads[0], NULL); pthread_join(threads[1], NULL); printf(\"Counter value: %d (expected: 2000000)\\n\", shared_counter); pthread_mutex_destroy(&mutex); return 0; } Solution 2: Semaphore #include #include #include sem_t semaphore; int shared_resource = 0; void* worker(void *arg) { for (int i = 0; i < 5; i++) { sem_wait(&semaphore); \/\/ Acquire (P operation) printf(\"Thread %ld accessing resource\\n\", (long)pthread_self()); shared_resource++; sem_post(&semaphore); \/\/ Release (V operation) } return NULL; } int main() { sem_init(&semaphore, 0, 1); \/\/ Binary semaphore pthread_t threads[3]; for (int i = 0; i < 3; i++) { pthread_create(&threads[i], NULL, worker, NULL); } for (int i = 0; i < 3; i++) { pthread_join(threads[i], NULL); } sem_destroy(&semaphore); return 0; } Solution 3: Condition Variables #include #include pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER; pthread_cond_t condition = PTHREAD_COND_INITIALIZER; int ready = 0; void* waiter_thread(void *arg) { pthread_mutex_lock(&mutex); while (!ready) { \/\/ Always use while, not if printf(\"Waiting...\\n\"); pthread_cond_wait(&condition, &mutex); } printf(\"Proceeding!\\n\"); pthread_mutex_unlock(&mutex); return NULL; } void* signaler_thread(void *arg) { sleep(2); pthread_mutex_lock(&mutex); ready = 1; pthread_cond_signal(&condition); \/\/ Wake one waiter pthread_mutex_unlock(&mutex); return NULL; } int main() { pthread_t t1, t2; pthread_create(&t1, NULL, waiter_thread, NULL); pthread_create(&t2, NULL, signaler_thread, NULL); pthread_join(t1, NULL); pthread_join(t2, NULL); return 0; } ","date":"29-Nov-2024","heading":"Thread Synchronization","objectID":"\/posts\/operating-system\/:6:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#thread-synchronization"},{"categories":["computer-science"],"collections":null,"content":"Producer-Consumer Problem Scenario: Producer generates data, Consumer uses data, bounded buffer between them Note Classic Synchronization Problem: The Producer-Consumer problem demonstrates critical synchronization concepts: Mutual Exclusion: Only one thread accesses buffer at a time Resource Counting: Track empty\/full slots Deadlock Avoidance: Proper ordering of semaphore operations Real-world Applications: Message queues, thread pools, I\/O buffering #include #include #include #include #define BUFFER_SIZE 5 int buffer[BUFFER_SIZE]; int in = 0, out = 0; sem_t empty; \/\/ Counts empty slots sem_t full; \/\/ Counts full slots pthread_mutex_t mutex; void* producer(void *arg) { for (int i = 0; i < 10; i++) { sem_wait(&empty); \/\/ Wait for empty slot pthread_mutex_lock(&mutex); buffer[in] = i * 10; printf(\"Producer: Added %d at %d\\n\", buffer[in], in); in = (in + 1) % BUFFER_SIZE; pthread_mutex_unlock(&mutex); sem_post(&full); \/\/ Signal full slot } return NULL; } void* consumer(void *arg) { for (int i = 0; i < 10; i++) { sem_wait(&full); \/\/ Wait for full slot pthread_mutex_lock(&mutex); int value = buffer[out]; printf(\"Consumer: Consumed %d from %d\\n\", value, out); out = (out + 1) % BUFFER_SIZE; pthread_mutex_unlock(&mutex); sem_post(&empty); \/\/ Signal empty slot } return NULL; } int main() { pthread_t prod, cons; sem_init(&empty, 0, BUFFER_SIZE); sem_init(&full, 0, 0); pthread_mutex_init(&mutex, NULL); pthread_create(&prod, NULL, producer, NULL); pthread_create(&cons, NULL, consumer, NULL); pthread_join(prod, NULL); pthread_join(cons, NULL); sem_destroy(&empty); sem_destroy(&full); pthread_mutex_destroy(&mutex); return 0; } ","date":"29-Nov-2024","heading":"Producer-Consumer Problem","objectID":"\/posts\/operating-system\/:6:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#producer-consumer-problem"},{"categories":["computer-science"],"collections":null,"content":"7. CPU Scheduling Algorithms ","date":"29-Nov-2024","heading":"7. CPU Scheduling Algorithms","objectID":"\/posts\/operating-system\/:7:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#cpu-scheduling"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Criteria Maximize: CPU Utilization: Keep CPU as busy as possible (40-90% is reasonable) Throughput: Number of processes completed per unit time Minimize: Turnaround Time: Time from submission to completion Waiting Time: Time spent in ready queue Response Time: Time from submission to first response ","date":"29-Nov-2024","heading":"Scheduling Criteria","objectID":"\/posts\/operating-system\/:7:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#scheduling-criteria"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Algorithms (Complete Analysis) 1. First-Come-First-Served (FCFS) Algorithm: Process executes for entire duration, no preemption \/\/ FCFS Scheduler Implementation struct process { int pid; int arrival_time; int burst_time; int start_time; int end_time; }; void fcfs_schedule(struct process procs[], int n) { int current_time = 0; double total_turnaround = 0, total_wait = 0; for (int i = 0; i < n; i++) { procs[i].start_time = current_time; procs[i].end_time = current_time + procs[i].burst_time; current_time = procs[i].end_time; int turnaround = procs[i].end_time - procs[i].arrival_time; int wait = procs[i].start_time - procs[i].arrival_time; total_turnaround += turnaround; total_wait += wait; printf(\"P%d: Wait=%d, Turnaround=%d\\n\", procs[i].pid, wait, turnaround); } printf(\"Average Wait Time: %.2f\\n\", total_wait \/ n); printf(\"Average Turnaround: %.2f\\n\", total_turnaround \/ n); } \/* Example: Processes with burst times [24, 3, 3] FCFS: P1_____________________ P2__ P3__ 0 24 27 30 Waiting times: P1=0, P2=24, P3=27 Average wait: 17ms *\/ Warning Convoy Effect Problem: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation. 2. Shortest Job First (SJF) Non-preemptive: Each process runs completely void sjf_schedule(struct process procs[], int n) { \/\/ Sort by burst time (ascending) for (int i = 0; i < n - 1; i++) { for (int j = 0; j < n - i - 1; j++) { if (procs[j].burst_time > procs[j+1].burst_time) { struct process temp = procs[j]; procs[j] = procs[j+1]; procs[j+1] = temp; } } } fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list } \/* Example: Processes [24, 3, 3] Sorted: [3, 3, 24] Gantt Chart: P2_ P3_ P1_____________________ 0 3 6 30 Average wait: (0 + 3 + 6) \/ 3 = 3ms (vs 17ms for FCFS!) *\/ Tip SJF Optimization: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an 82% reduction! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously. Warning Starvation Risk: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms. Preemptive SJF (Shortest Remaining Time First - SRTF) void srtf_schedule(struct process procs[], int n) { int remaining[n]; for (int i = 0; i < n; i++) { remaining[i] = procs[i].burst_time; } int time = 0; int total_processes = n; while (total_processes > 0) { \/\/ Find process with shortest remaining time int min_index = -1; int min_remaining = INT_MAX; for (int i = 0; i < n; i++) { if (remaining[i] > 0 && remaining[i] < min_remaining && procs[i].arrival_time <= time) { min_remaining = remaining[i]; min_index = i; } } if (min_index == -1) { time++; continue; } \/\/ Execute for 1 time unit remaining[min_index]--; time++; if (remaining[min_index] == 0) { total_processes--; } } } 3. Round Robin (RR) Scheduling Overview: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It’s designed specifically for time-sharing systems where fairness and responsiveness are critical. Key Concept: Time Quantum (Time Slice) - Fixed time period allocated to each process Info Time Quantum Selection is Critical: Too small (1-10ms): Excessive context switching overhead \u2192 CPU thrashing Too large (100ms+): Behaves like FCFS \u2192 poor response time Sweet spot: 80% of CPU bursts should be shorter than the quantum Linux default: ~100ms (configurable via sched_latency_ns) Characteristics: Preemptive: CPU is preempted after time quantum expires Fair: All processes get equal CPU time in rotation FIFO Queue: Processes are arranged in circular queue No Starvation: Every process eventually gets CPU time Round Robin Algorithm Flow Round Robin Algorithm Flow Diagram - CPU Sch","date":"29-Nov-2024","heading":"Scheduling Algorithms (Complete Analysis)","objectID":"\/posts\/operating-system\/:7:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#scheduling-algorithms-complete-analysis"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Algorithms (Complete Analysis) 1. First-Come-First-Served (FCFS) Algorithm: Process executes for entire duration, no preemption \/\/ FCFS Scheduler Implementation struct process { int pid; int arrival_time; int burst_time; int start_time; int end_time; }; void fcfs_schedule(struct process procs[], int n) { int current_time = 0; double total_turnaround = 0, total_wait = 0; for (int i = 0; i < n; i++) { procs[i].start_time = current_time; procs[i].end_time = current_time + procs[i].burst_time; current_time = procs[i].end_time; int turnaround = procs[i].end_time - procs[i].arrival_time; int wait = procs[i].start_time - procs[i].arrival_time; total_turnaround += turnaround; total_wait += wait; printf(\"P%d: Wait=%d, Turnaround=%d\\n\", procs[i].pid, wait, turnaround); } printf(\"Average Wait Time: %.2f\\n\", total_wait \/ n); printf(\"Average Turnaround: %.2f\\n\", total_turnaround \/ n); } \/* Example: Processes with burst times [24, 3, 3] FCFS: P1_____________________ P2__ P3__ 0 24 27 30 Waiting times: P1=0, P2=24, P3=27 Average wait: 17ms *\/ Warning Convoy Effect Problem: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation. 2. Shortest Job First (SJF) Non-preemptive: Each process runs completely void sjf_schedule(struct process procs[], int n) { \/\/ Sort by burst time (ascending) for (int i = 0; i < n - 1; i++) { for (int j = 0; j < n - i - 1; j++) { if (procs[j].burst_time > procs[j+1].burst_time) { struct process temp = procs[j]; procs[j] = procs[j+1]; procs[j+1] = temp; } } } fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list } \/* Example: Processes [24, 3, 3] Sorted: [3, 3, 24] Gantt Chart: P2_ P3_ P1_____________________ 0 3 6 30 Average wait: (0 + 3 + 6) \/ 3 = 3ms (vs 17ms for FCFS!) *\/ Tip SJF Optimization: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an 82% reduction! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously. Warning Starvation Risk: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms. Preemptive SJF (Shortest Remaining Time First - SRTF) void srtf_schedule(struct process procs[], int n) { int remaining[n]; for (int i = 0; i < n; i++) { remaining[i] = procs[i].burst_time; } int time = 0; int total_processes = n; while (total_processes > 0) { \/\/ Find process with shortest remaining time int min_index = -1; int min_remaining = INT_MAX; for (int i = 0; i < n; i++) { if (remaining[i] > 0 && remaining[i] < min_remaining && procs[i].arrival_time <= time) { min_remaining = remaining[i]; min_index = i; } } if (min_index == -1) { time++; continue; } \/\/ Execute for 1 time unit remaining[min_index]--; time++; if (remaining[min_index] == 0) { total_processes--; } } } 3. Round Robin (RR) Scheduling Overview: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It’s designed specifically for time-sharing systems where fairness and responsiveness are critical. Key Concept: Time Quantum (Time Slice) - Fixed time period allocated to each process Info Time Quantum Selection is Critical: Too small (1-10ms): Excessive context switching overhead \u2192 CPU thrashing Too large (100ms+): Behaves like FCFS \u2192 poor response time Sweet spot: 80% of CPU bursts should be shorter than the quantum Linux default: ~100ms (configurable via sched_latency_ns) Characteristics: Preemptive: CPU is preempted after time quantum expires Fair: All processes get equal CPU time in rotation FIFO Queue: Processes are arranged in circular queue No Starvation: Every process eventually gets CPU time Round Robin Algorithm Flow Round Robin Algorithm Flow Diagram - CPU Sch","date":"29-Nov-2024","heading":"1. First-Come-First-Served (FCFS)","objectID":"\/posts\/operating-system\/:7:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1-first-come-first-served-fcfs"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Algorithms (Complete Analysis) 1. First-Come-First-Served (FCFS) Algorithm: Process executes for entire duration, no preemption \/\/ FCFS Scheduler Implementation struct process { int pid; int arrival_time; int burst_time; int start_time; int end_time; }; void fcfs_schedule(struct process procs[], int n) { int current_time = 0; double total_turnaround = 0, total_wait = 0; for (int i = 0; i < n; i++) { procs[i].start_time = current_time; procs[i].end_time = current_time + procs[i].burst_time; current_time = procs[i].end_time; int turnaround = procs[i].end_time - procs[i].arrival_time; int wait = procs[i].start_time - procs[i].arrival_time; total_turnaround += turnaround; total_wait += wait; printf(\"P%d: Wait=%d, Turnaround=%d\\n\", procs[i].pid, wait, turnaround); } printf(\"Average Wait Time: %.2f\\n\", total_wait \/ n); printf(\"Average Turnaround: %.2f\\n\", total_turnaround \/ n); } \/* Example: Processes with burst times [24, 3, 3] FCFS: P1_____________________ P2__ P3__ 0 24 27 30 Waiting times: P1=0, P2=24, P3=27 Average wait: 17ms *\/ Warning Convoy Effect Problem: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation. 2. Shortest Job First (SJF) Non-preemptive: Each process runs completely void sjf_schedule(struct process procs[], int n) { \/\/ Sort by burst time (ascending) for (int i = 0; i < n - 1; i++) { for (int j = 0; j < n - i - 1; j++) { if (procs[j].burst_time > procs[j+1].burst_time) { struct process temp = procs[j]; procs[j] = procs[j+1]; procs[j+1] = temp; } } } fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list } \/* Example: Processes [24, 3, 3] Sorted: [3, 3, 24] Gantt Chart: P2_ P3_ P1_____________________ 0 3 6 30 Average wait: (0 + 3 + 6) \/ 3 = 3ms (vs 17ms for FCFS!) *\/ Tip SJF Optimization: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an 82% reduction! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously. Warning Starvation Risk: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms. Preemptive SJF (Shortest Remaining Time First - SRTF) void srtf_schedule(struct process procs[], int n) { int remaining[n]; for (int i = 0; i < n; i++) { remaining[i] = procs[i].burst_time; } int time = 0; int total_processes = n; while (total_processes > 0) { \/\/ Find process with shortest remaining time int min_index = -1; int min_remaining = INT_MAX; for (int i = 0; i < n; i++) { if (remaining[i] > 0 && remaining[i] < min_remaining && procs[i].arrival_time <= time) { min_remaining = remaining[i]; min_index = i; } } if (min_index == -1) { time++; continue; } \/\/ Execute for 1 time unit remaining[min_index]--; time++; if (remaining[min_index] == 0) { total_processes--; } } } 3. Round Robin (RR) Scheduling Overview: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It’s designed specifically for time-sharing systems where fairness and responsiveness are critical. Key Concept: Time Quantum (Time Slice) - Fixed time period allocated to each process Info Time Quantum Selection is Critical: Too small (1-10ms): Excessive context switching overhead \u2192 CPU thrashing Too large (100ms+): Behaves like FCFS \u2192 poor response time Sweet spot: 80% of CPU bursts should be shorter than the quantum Linux default: ~100ms (configurable via sched_latency_ns) Characteristics: Preemptive: CPU is preempted after time quantum expires Fair: All processes get equal CPU time in rotation FIFO Queue: Processes are arranged in circular queue No Starvation: Every process eventually gets CPU time Round Robin Algorithm Flow Round Robin Algorithm Flow Diagram - CPU Sch","date":"29-Nov-2024","heading":"2. Shortest Job First (SJF)","objectID":"\/posts\/operating-system\/:7:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#2-shortest-job-first-sjf"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Algorithms (Complete Analysis) 1. First-Come-First-Served (FCFS) Algorithm: Process executes for entire duration, no preemption \/\/ FCFS Scheduler Implementation struct process { int pid; int arrival_time; int burst_time; int start_time; int end_time; }; void fcfs_schedule(struct process procs[], int n) { int current_time = 0; double total_turnaround = 0, total_wait = 0; for (int i = 0; i < n; i++) { procs[i].start_time = current_time; procs[i].end_time = current_time + procs[i].burst_time; current_time = procs[i].end_time; int turnaround = procs[i].end_time - procs[i].arrival_time; int wait = procs[i].start_time - procs[i].arrival_time; total_turnaround += turnaround; total_wait += wait; printf(\"P%d: Wait=%d, Turnaround=%d\\n\", procs[i].pid, wait, turnaround); } printf(\"Average Wait Time: %.2f\\n\", total_wait \/ n); printf(\"Average Turnaround: %.2f\\n\", total_turnaround \/ n); } \/* Example: Processes with burst times [24, 3, 3] FCFS: P1_____________________ P2__ P3__ 0 24 27 30 Waiting times: P1=0, P2=24, P3=27 Average wait: 17ms *\/ Warning Convoy Effect Problem: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation. 2. Shortest Job First (SJF) Non-preemptive: Each process runs completely void sjf_schedule(struct process procs[], int n) { \/\/ Sort by burst time (ascending) for (int i = 0; i < n - 1; i++) { for (int j = 0; j < n - i - 1; j++) { if (procs[j].burst_time > procs[j+1].burst_time) { struct process temp = procs[j]; procs[j] = procs[j+1]; procs[j+1] = temp; } } } fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list } \/* Example: Processes [24, 3, 3] Sorted: [3, 3, 24] Gantt Chart: P2_ P3_ P1_____________________ 0 3 6 30 Average wait: (0 + 3 + 6) \/ 3 = 3ms (vs 17ms for FCFS!) *\/ Tip SJF Optimization: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an 82% reduction! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously. Warning Starvation Risk: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms. Preemptive SJF (Shortest Remaining Time First - SRTF) void srtf_schedule(struct process procs[], int n) { int remaining[n]; for (int i = 0; i < n; i++) { remaining[i] = procs[i].burst_time; } int time = 0; int total_processes = n; while (total_processes > 0) { \/\/ Find process with shortest remaining time int min_index = -1; int min_remaining = INT_MAX; for (int i = 0; i < n; i++) { if (remaining[i] > 0 && remaining[i] < min_remaining && procs[i].arrival_time <= time) { min_remaining = remaining[i]; min_index = i; } } if (min_index == -1) { time++; continue; } \/\/ Execute for 1 time unit remaining[min_index]--; time++; if (remaining[min_index] == 0) { total_processes--; } } } 3. Round Robin (RR) Scheduling Overview: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It’s designed specifically for time-sharing systems where fairness and responsiveness are critical. Key Concept: Time Quantum (Time Slice) - Fixed time period allocated to each process Info Time Quantum Selection is Critical: Too small (1-10ms): Excessive context switching overhead \u2192 CPU thrashing Too large (100ms+): Behaves like FCFS \u2192 poor response time Sweet spot: 80% of CPU bursts should be shorter than the quantum Linux default: ~100ms (configurable via sched_latency_ns) Characteristics: Preemptive: CPU is preempted after time quantum expires Fair: All processes get equal CPU time in rotation FIFO Queue: Processes are arranged in circular queue No Starvation: Every process eventually gets CPU time Round Robin Algorithm Flow Round Robin Algorithm Flow Diagram - CPU Sch","date":"29-Nov-2024","heading":"3. Round Robin (RR) Scheduling","objectID":"\/posts\/operating-system\/:7:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#3-round-robin-rr-scheduling"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Algorithms (Complete Analysis) 1. First-Come-First-Served (FCFS) Algorithm: Process executes for entire duration, no preemption \/\/ FCFS Scheduler Implementation struct process { int pid; int arrival_time; int burst_time; int start_time; int end_time; }; void fcfs_schedule(struct process procs[], int n) { int current_time = 0; double total_turnaround = 0, total_wait = 0; for (int i = 0; i < n; i++) { procs[i].start_time = current_time; procs[i].end_time = current_time + procs[i].burst_time; current_time = procs[i].end_time; int turnaround = procs[i].end_time - procs[i].arrival_time; int wait = procs[i].start_time - procs[i].arrival_time; total_turnaround += turnaround; total_wait += wait; printf(\"P%d: Wait=%d, Turnaround=%d\\n\", procs[i].pid, wait, turnaround); } printf(\"Average Wait Time: %.2f\\n\", total_wait \/ n); printf(\"Average Turnaround: %.2f\\n\", total_turnaround \/ n); } \/* Example: Processes with burst times [24, 3, 3] FCFS: P1_____________________ P2__ P3__ 0 24 27 30 Waiting times: P1=0, P2=24, P3=27 Average wait: 17ms *\/ Warning Convoy Effect Problem: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation. 2. Shortest Job First (SJF) Non-preemptive: Each process runs completely void sjf_schedule(struct process procs[], int n) { \/\/ Sort by burst time (ascending) for (int i = 0; i < n - 1; i++) { for (int j = 0; j < n - i - 1; j++) { if (procs[j].burst_time > procs[j+1].burst_time) { struct process temp = procs[j]; procs[j] = procs[j+1]; procs[j+1] = temp; } } } fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list } \/* Example: Processes [24, 3, 3] Sorted: [3, 3, 24] Gantt Chart: P2_ P3_ P1_____________________ 0 3 6 30 Average wait: (0 + 3 + 6) \/ 3 = 3ms (vs 17ms for FCFS!) *\/ Tip SJF Optimization: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an 82% reduction! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously. Warning Starvation Risk: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms. Preemptive SJF (Shortest Remaining Time First - SRTF) void srtf_schedule(struct process procs[], int n) { int remaining[n]; for (int i = 0; i < n; i++) { remaining[i] = procs[i].burst_time; } int time = 0; int total_processes = n; while (total_processes > 0) { \/\/ Find process with shortest remaining time int min_index = -1; int min_remaining = INT_MAX; for (int i = 0; i < n; i++) { if (remaining[i] > 0 && remaining[i] < min_remaining && procs[i].arrival_time <= time) { min_remaining = remaining[i]; min_index = i; } } if (min_index == -1) { time++; continue; } \/\/ Execute for 1 time unit remaining[min_index]--; time++; if (remaining[min_index] == 0) { total_processes--; } } } 3. Round Robin (RR) Scheduling Overview: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It’s designed specifically for time-sharing systems where fairness and responsiveness are critical. Key Concept: Time Quantum (Time Slice) - Fixed time period allocated to each process Info Time Quantum Selection is Critical: Too small (1-10ms): Excessive context switching overhead \u2192 CPU thrashing Too large (100ms+): Behaves like FCFS \u2192 poor response time Sweet spot: 80% of CPU bursts should be shorter than the quantum Linux default: ~100ms (configurable via sched_latency_ns) Characteristics: Preemptive: CPU is preempted after time quantum expires Fair: All processes get equal CPU time in rotation FIFO Queue: Processes are arranged in circular queue No Starvation: Every process eventually gets CPU time Round Robin Algorithm Flow Round Robin Algorithm Flow Diagram - CPU Sch","date":"29-Nov-2024","heading":"Round Robin Algorithm Flow","objectID":"\/posts\/operating-system\/:7:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#round-robin-algorithm-flow"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Algorithms (Complete Analysis) 1. First-Come-First-Served (FCFS) Algorithm: Process executes for entire duration, no preemption \/\/ FCFS Scheduler Implementation struct process { int pid; int arrival_time; int burst_time; int start_time; int end_time; }; void fcfs_schedule(struct process procs[], int n) { int current_time = 0; double total_turnaround = 0, total_wait = 0; for (int i = 0; i < n; i++) { procs[i].start_time = current_time; procs[i].end_time = current_time + procs[i].burst_time; current_time = procs[i].end_time; int turnaround = procs[i].end_time - procs[i].arrival_time; int wait = procs[i].start_time - procs[i].arrival_time; total_turnaround += turnaround; total_wait += wait; printf(\"P%d: Wait=%d, Turnaround=%d\\n\", procs[i].pid, wait, turnaround); } printf(\"Average Wait Time: %.2f\\n\", total_wait \/ n); printf(\"Average Turnaround: %.2f\\n\", total_turnaround \/ n); } \/* Example: Processes with burst times [24, 3, 3] FCFS: P1_____________________ P2__ P3__ 0 24 27 30 Waiting times: P1=0, P2=24, P3=27 Average wait: 17ms *\/ Warning Convoy Effect Problem: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation. 2. Shortest Job First (SJF) Non-preemptive: Each process runs completely void sjf_schedule(struct process procs[], int n) { \/\/ Sort by burst time (ascending) for (int i = 0; i < n - 1; i++) { for (int j = 0; j < n - i - 1; j++) { if (procs[j].burst_time > procs[j+1].burst_time) { struct process temp = procs[j]; procs[j] = procs[j+1]; procs[j+1] = temp; } } } fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list } \/* Example: Processes [24, 3, 3] Sorted: [3, 3, 24] Gantt Chart: P2_ P3_ P1_____________________ 0 3 6 30 Average wait: (0 + 3 + 6) \/ 3 = 3ms (vs 17ms for FCFS!) *\/ Tip SJF Optimization: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an 82% reduction! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously. Warning Starvation Risk: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms. Preemptive SJF (Shortest Remaining Time First - SRTF) void srtf_schedule(struct process procs[], int n) { int remaining[n]; for (int i = 0; i < n; i++) { remaining[i] = procs[i].burst_time; } int time = 0; int total_processes = n; while (total_processes > 0) { \/\/ Find process with shortest remaining time int min_index = -1; int min_remaining = INT_MAX; for (int i = 0; i < n; i++) { if (remaining[i] > 0 && remaining[i] < min_remaining && procs[i].arrival_time <= time) { min_remaining = remaining[i]; min_index = i; } } if (min_index == -1) { time++; continue; } \/\/ Execute for 1 time unit remaining[min_index]--; time++; if (remaining[min_index] == 0) { total_processes--; } } } 3. Round Robin (RR) Scheduling Overview: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It’s designed specifically for time-sharing systems where fairness and responsiveness are critical. Key Concept: Time Quantum (Time Slice) - Fixed time period allocated to each process Info Time Quantum Selection is Critical: Too small (1-10ms): Excessive context switching overhead \u2192 CPU thrashing Too large (100ms+): Behaves like FCFS \u2192 poor response time Sweet spot: 80% of CPU bursts should be shorter than the quantum Linux default: ~100ms (configurable via sched_latency_ns) Characteristics: Preemptive: CPU is preempted after time quantum expires Fair: All processes get equal CPU time in rotation FIFO Queue: Processes are arranged in circular queue No Starvation: Every process eventually gets CPU time Round Robin Algorithm Flow Round Robin Algorithm Flow Diagram - CPU Sch","date":"29-Nov-2024","heading":"Process State Transitions in Round Robin","objectID":"\/posts\/operating-system\/:7:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#process-state-transitions-in-round-robin"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Algorithms (Complete Analysis) 1. First-Come-First-Served (FCFS) Algorithm: Process executes for entire duration, no preemption \/\/ FCFS Scheduler Implementation struct process { int pid; int arrival_time; int burst_time; int start_time; int end_time; }; void fcfs_schedule(struct process procs[], int n) { int current_time = 0; double total_turnaround = 0, total_wait = 0; for (int i = 0; i < n; i++) { procs[i].start_time = current_time; procs[i].end_time = current_time + procs[i].burst_time; current_time = procs[i].end_time; int turnaround = procs[i].end_time - procs[i].arrival_time; int wait = procs[i].start_time - procs[i].arrival_time; total_turnaround += turnaround; total_wait += wait; printf(\"P%d: Wait=%d, Turnaround=%d\\n\", procs[i].pid, wait, turnaround); } printf(\"Average Wait Time: %.2f\\n\", total_wait \/ n); printf(\"Average Turnaround: %.2f\\n\", total_turnaround \/ n); } \/* Example: Processes with burst times [24, 3, 3] FCFS: P1_____________________ P2__ P3__ 0 24 27 30 Waiting times: P1=0, P2=24, P3=27 Average wait: 17ms *\/ Warning Convoy Effect Problem: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation. 2. Shortest Job First (SJF) Non-preemptive: Each process runs completely void sjf_schedule(struct process procs[], int n) { \/\/ Sort by burst time (ascending) for (int i = 0; i < n - 1; i++) { for (int j = 0; j < n - i - 1; j++) { if (procs[j].burst_time > procs[j+1].burst_time) { struct process temp = procs[j]; procs[j] = procs[j+1]; procs[j+1] = temp; } } } fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list } \/* Example: Processes [24, 3, 3] Sorted: [3, 3, 24] Gantt Chart: P2_ P3_ P1_____________________ 0 3 6 30 Average wait: (0 + 3 + 6) \/ 3 = 3ms (vs 17ms for FCFS!) *\/ Tip SJF Optimization: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an 82% reduction! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously. Warning Starvation Risk: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms. Preemptive SJF (Shortest Remaining Time First - SRTF) void srtf_schedule(struct process procs[], int n) { int remaining[n]; for (int i = 0; i < n; i++) { remaining[i] = procs[i].burst_time; } int time = 0; int total_processes = n; while (total_processes > 0) { \/\/ Find process with shortest remaining time int min_index = -1; int min_remaining = INT_MAX; for (int i = 0; i < n; i++) { if (remaining[i] > 0 && remaining[i] < min_remaining && procs[i].arrival_time <= time) { min_remaining = remaining[i]; min_index = i; } } if (min_index == -1) { time++; continue; } \/\/ Execute for 1 time unit remaining[min_index]--; time++; if (remaining[min_index] == 0) { total_processes--; } } } 3. Round Robin (RR) Scheduling Overview: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It’s designed specifically for time-sharing systems where fairness and responsiveness are critical. Key Concept: Time Quantum (Time Slice) - Fixed time period allocated to each process Info Time Quantum Selection is Critical: Too small (1-10ms): Excessive context switching overhead \u2192 CPU thrashing Too large (100ms+): Behaves like FCFS \u2192 poor response time Sweet spot: 80% of CPU bursts should be shorter than the quantum Linux default: ~100ms (configurable via sched_latency_ns) Characteristics: Preemptive: CPU is preempted after time quantum expires Fair: All processes get equal CPU time in rotation FIFO Queue: Processes are arranged in circular queue No Starvation: Every process eventually gets CPU time Round Robin Algorithm Flow Round Robin Algorithm Flow Diagram - CPU Sch","date":"29-Nov-2024","heading":"Circular Queue Mechanism","objectID":"\/posts\/operating-system\/:7:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#circular-queue-mechanism"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Algorithms (Complete Analysis) 1. First-Come-First-Served (FCFS) Algorithm: Process executes for entire duration, no preemption \/\/ FCFS Scheduler Implementation struct process { int pid; int arrival_time; int burst_time; int start_time; int end_time; }; void fcfs_schedule(struct process procs[], int n) { int current_time = 0; double total_turnaround = 0, total_wait = 0; for (int i = 0; i < n; i++) { procs[i].start_time = current_time; procs[i].end_time = current_time + procs[i].burst_time; current_time = procs[i].end_time; int turnaround = procs[i].end_time - procs[i].arrival_time; int wait = procs[i].start_time - procs[i].arrival_time; total_turnaround += turnaround; total_wait += wait; printf(\"P%d: Wait=%d, Turnaround=%d\\n\", procs[i].pid, wait, turnaround); } printf(\"Average Wait Time: %.2f\\n\", total_wait \/ n); printf(\"Average Turnaround: %.2f\\n\", total_turnaround \/ n); } \/* Example: Processes with burst times [24, 3, 3] FCFS: P1_____________________ P2__ P3__ 0 24 27 30 Waiting times: P1=0, P2=24, P3=27 Average wait: 17ms *\/ Warning Convoy Effect Problem: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation. 2. Shortest Job First (SJF) Non-preemptive: Each process runs completely void sjf_schedule(struct process procs[], int n) { \/\/ Sort by burst time (ascending) for (int i = 0; i < n - 1; i++) { for (int j = 0; j < n - i - 1; j++) { if (procs[j].burst_time > procs[j+1].burst_time) { struct process temp = procs[j]; procs[j] = procs[j+1]; procs[j+1] = temp; } } } fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list } \/* Example: Processes [24, 3, 3] Sorted: [3, 3, 24] Gantt Chart: P2_ P3_ P1_____________________ 0 3 6 30 Average wait: (0 + 3 + 6) \/ 3 = 3ms (vs 17ms for FCFS!) *\/ Tip SJF Optimization: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an 82% reduction! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously. Warning Starvation Risk: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms. Preemptive SJF (Shortest Remaining Time First - SRTF) void srtf_schedule(struct process procs[], int n) { int remaining[n]; for (int i = 0; i < n; i++) { remaining[i] = procs[i].burst_time; } int time = 0; int total_processes = n; while (total_processes > 0) { \/\/ Find process with shortest remaining time int min_index = -1; int min_remaining = INT_MAX; for (int i = 0; i < n; i++) { if (remaining[i] > 0 && remaining[i] < min_remaining && procs[i].arrival_time <= time) { min_remaining = remaining[i]; min_index = i; } } if (min_index == -1) { time++; continue; } \/\/ Execute for 1 time unit remaining[min_index]--; time++; if (remaining[min_index] == 0) { total_processes--; } } } 3. Round Robin (RR) Scheduling Overview: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It’s designed specifically for time-sharing systems where fairness and responsiveness are critical. Key Concept: Time Quantum (Time Slice) - Fixed time period allocated to each process Info Time Quantum Selection is Critical: Too small (1-10ms): Excessive context switching overhead \u2192 CPU thrashing Too large (100ms+): Behaves like FCFS \u2192 poor response time Sweet spot: 80% of CPU bursts should be shorter than the quantum Linux default: ~100ms (configurable via sched_latency_ns) Characteristics: Preemptive: CPU is preempted after time quantum expires Fair: All processes get equal CPU time in rotation FIFO Queue: Processes are arranged in circular queue No Starvation: Every process eventually gets CPU time Round Robin Algorithm Flow Round Robin Algorithm Flow Diagram - CPU Sch","date":"29-Nov-2024","heading":"Detailed Implementation with Comments","objectID":"\/posts\/operating-system\/:7:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#detailed-implementation-with-comments"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Algorithms (Complete Analysis) 1. First-Come-First-Served (FCFS) Algorithm: Process executes for entire duration, no preemption \/\/ FCFS Scheduler Implementation struct process { int pid; int arrival_time; int burst_time; int start_time; int end_time; }; void fcfs_schedule(struct process procs[], int n) { int current_time = 0; double total_turnaround = 0, total_wait = 0; for (int i = 0; i < n; i++) { procs[i].start_time = current_time; procs[i].end_time = current_time + procs[i].burst_time; current_time = procs[i].end_time; int turnaround = procs[i].end_time - procs[i].arrival_time; int wait = procs[i].start_time - procs[i].arrival_time; total_turnaround += turnaround; total_wait += wait; printf(\"P%d: Wait=%d, Turnaround=%d\\n\", procs[i].pid, wait, turnaround); } printf(\"Average Wait Time: %.2f\\n\", total_wait \/ n); printf(\"Average Turnaround: %.2f\\n\", total_turnaround \/ n); } \/* Example: Processes with burst times [24, 3, 3] FCFS: P1_____________________ P2__ P3__ 0 24 27 30 Waiting times: P1=0, P2=24, P3=27 Average wait: 17ms *\/ Warning Convoy Effect Problem: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation. 2. Shortest Job First (SJF) Non-preemptive: Each process runs completely void sjf_schedule(struct process procs[], int n) { \/\/ Sort by burst time (ascending) for (int i = 0; i < n - 1; i++) { for (int j = 0; j < n - i - 1; j++) { if (procs[j].burst_time > procs[j+1].burst_time) { struct process temp = procs[j]; procs[j] = procs[j+1]; procs[j+1] = temp; } } } fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list } \/* Example: Processes [24, 3, 3] Sorted: [3, 3, 24] Gantt Chart: P2_ P3_ P1_____________________ 0 3 6 30 Average wait: (0 + 3 + 6) \/ 3 = 3ms (vs 17ms for FCFS!) *\/ Tip SJF Optimization: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an 82% reduction! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously. Warning Starvation Risk: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms. Preemptive SJF (Shortest Remaining Time First - SRTF) void srtf_schedule(struct process procs[], int n) { int remaining[n]; for (int i = 0; i < n; i++) { remaining[i] = procs[i].burst_time; } int time = 0; int total_processes = n; while (total_processes > 0) { \/\/ Find process with shortest remaining time int min_index = -1; int min_remaining = INT_MAX; for (int i = 0; i < n; i++) { if (remaining[i] > 0 && remaining[i] < min_remaining && procs[i].arrival_time <= time) { min_remaining = remaining[i]; min_index = i; } } if (min_index == -1) { time++; continue; } \/\/ Execute for 1 time unit remaining[min_index]--; time++; if (remaining[min_index] == 0) { total_processes--; } } } 3. Round Robin (RR) Scheduling Overview: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It’s designed specifically for time-sharing systems where fairness and responsiveness are critical. Key Concept: Time Quantum (Time Slice) - Fixed time period allocated to each process Info Time Quantum Selection is Critical: Too small (1-10ms): Excessive context switching overhead \u2192 CPU thrashing Too large (100ms+): Behaves like FCFS \u2192 poor response time Sweet spot: 80% of CPU bursts should be shorter than the quantum Linux default: ~100ms (configurable via sched_latency_ns) Characteristics: Preemptive: CPU is preempted after time quantum expires Fair: All processes get equal CPU time in rotation FIFO Queue: Processes are arranged in circular queue No Starvation: Every process eventually gets CPU time Round Robin Algorithm Flow Round Robin Algorithm Flow Diagram - CPU Sch","date":"29-Nov-2024","heading":"Gantt Chart Example","objectID":"\/posts\/operating-system\/:7:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#gantt-chart-example"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Algorithms (Complete Analysis) 1. First-Come-First-Served (FCFS) Algorithm: Process executes for entire duration, no preemption \/\/ FCFS Scheduler Implementation struct process { int pid; int arrival_time; int burst_time; int start_time; int end_time; }; void fcfs_schedule(struct process procs[], int n) { int current_time = 0; double total_turnaround = 0, total_wait = 0; for (int i = 0; i < n; i++) { procs[i].start_time = current_time; procs[i].end_time = current_time + procs[i].burst_time; current_time = procs[i].end_time; int turnaround = procs[i].end_time - procs[i].arrival_time; int wait = procs[i].start_time - procs[i].arrival_time; total_turnaround += turnaround; total_wait += wait; printf(\"P%d: Wait=%d, Turnaround=%d\\n\", procs[i].pid, wait, turnaround); } printf(\"Average Wait Time: %.2f\\n\", total_wait \/ n); printf(\"Average Turnaround: %.2f\\n\", total_turnaround \/ n); } \/* Example: Processes with burst times [24, 3, 3] FCFS: P1_____________________ P2__ P3__ 0 24 27 30 Waiting times: P1=0, P2=24, P3=27 Average wait: 17ms *\/ Warning Convoy Effect Problem: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation. 2. Shortest Job First (SJF) Non-preemptive: Each process runs completely void sjf_schedule(struct process procs[], int n) { \/\/ Sort by burst time (ascending) for (int i = 0; i < n - 1; i++) { for (int j = 0; j < n - i - 1; j++) { if (procs[j].burst_time > procs[j+1].burst_time) { struct process temp = procs[j]; procs[j] = procs[j+1]; procs[j+1] = temp; } } } fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list } \/* Example: Processes [24, 3, 3] Sorted: [3, 3, 24] Gantt Chart: P2_ P3_ P1_____________________ 0 3 6 30 Average wait: (0 + 3 + 6) \/ 3 = 3ms (vs 17ms for FCFS!) *\/ Tip SJF Optimization: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an 82% reduction! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously. Warning Starvation Risk: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms. Preemptive SJF (Shortest Remaining Time First - SRTF) void srtf_schedule(struct process procs[], int n) { int remaining[n]; for (int i = 0; i < n; i++) { remaining[i] = procs[i].burst_time; } int time = 0; int total_processes = n; while (total_processes > 0) { \/\/ Find process with shortest remaining time int min_index = -1; int min_remaining = INT_MAX; for (int i = 0; i < n; i++) { if (remaining[i] > 0 && remaining[i] < min_remaining && procs[i].arrival_time <= time) { min_remaining = remaining[i]; min_index = i; } } if (min_index == -1) { time++; continue; } \/\/ Execute for 1 time unit remaining[min_index]--; time++; if (remaining[min_index] == 0) { total_processes--; } } } 3. Round Robin (RR) Scheduling Overview: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It’s designed specifically for time-sharing systems where fairness and responsiveness are critical. Key Concept: Time Quantum (Time Slice) - Fixed time period allocated to each process Info Time Quantum Selection is Critical: Too small (1-10ms): Excessive context switching overhead \u2192 CPU thrashing Too large (100ms+): Behaves like FCFS \u2192 poor response time Sweet spot: 80% of CPU bursts should be shorter than the quantum Linux default: ~100ms (configurable via sched_latency_ns) Characteristics: Preemptive: CPU is preempted after time quantum expires Fair: All processes get equal CPU time in rotation FIFO Queue: Processes are arranged in circular queue No Starvation: Every process eventually gets CPU time Round Robin Algorithm Flow Round Robin Algorithm Flow Diagram - CPU Sch","date":"29-Nov-2024","heading":"Impact of Time Quantum","objectID":"\/posts\/operating-system\/:7:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#impact-of-time-quantum"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Algorithms (Complete Analysis) 1. First-Come-First-Served (FCFS) Algorithm: Process executes for entire duration, no preemption \/\/ FCFS Scheduler Implementation struct process { int pid; int arrival_time; int burst_time; int start_time; int end_time; }; void fcfs_schedule(struct process procs[], int n) { int current_time = 0; double total_turnaround = 0, total_wait = 0; for (int i = 0; i < n; i++) { procs[i].start_time = current_time; procs[i].end_time = current_time + procs[i].burst_time; current_time = procs[i].end_time; int turnaround = procs[i].end_time - procs[i].arrival_time; int wait = procs[i].start_time - procs[i].arrival_time; total_turnaround += turnaround; total_wait += wait; printf(\"P%d: Wait=%d, Turnaround=%d\\n\", procs[i].pid, wait, turnaround); } printf(\"Average Wait Time: %.2f\\n\", total_wait \/ n); printf(\"Average Turnaround: %.2f\\n\", total_turnaround \/ n); } \/* Example: Processes with burst times [24, 3, 3] FCFS: P1_____________________ P2__ P3__ 0 24 27 30 Waiting times: P1=0, P2=24, P3=27 Average wait: 17ms *\/ Warning Convoy Effect Problem: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation. 2. Shortest Job First (SJF) Non-preemptive: Each process runs completely void sjf_schedule(struct process procs[], int n) { \/\/ Sort by burst time (ascending) for (int i = 0; i < n - 1; i++) { for (int j = 0; j < n - i - 1; j++) { if (procs[j].burst_time > procs[j+1].burst_time) { struct process temp = procs[j]; procs[j] = procs[j+1]; procs[j+1] = temp; } } } fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list } \/* Example: Processes [24, 3, 3] Sorted: [3, 3, 24] Gantt Chart: P2_ P3_ P1_____________________ 0 3 6 30 Average wait: (0 + 3 + 6) \/ 3 = 3ms (vs 17ms for FCFS!) *\/ Tip SJF Optimization: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an 82% reduction! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously. Warning Starvation Risk: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms. Preemptive SJF (Shortest Remaining Time First - SRTF) void srtf_schedule(struct process procs[], int n) { int remaining[n]; for (int i = 0; i < n; i++) { remaining[i] = procs[i].burst_time; } int time = 0; int total_processes = n; while (total_processes > 0) { \/\/ Find process with shortest remaining time int min_index = -1; int min_remaining = INT_MAX; for (int i = 0; i < n; i++) { if (remaining[i] > 0 && remaining[i] < min_remaining && procs[i].arrival_time <= time) { min_remaining = remaining[i]; min_index = i; } } if (min_index == -1) { time++; continue; } \/\/ Execute for 1 time unit remaining[min_index]--; time++; if (remaining[min_index] == 0) { total_processes--; } } } 3. Round Robin (RR) Scheduling Overview: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It’s designed specifically for time-sharing systems where fairness and responsiveness are critical. Key Concept: Time Quantum (Time Slice) - Fixed time period allocated to each process Info Time Quantum Selection is Critical: Too small (1-10ms): Excessive context switching overhead \u2192 CPU thrashing Too large (100ms+): Behaves like FCFS \u2192 poor response time Sweet spot: 80% of CPU bursts should be shorter than the quantum Linux default: ~100ms (configurable via sched_latency_ns) Characteristics: Preemptive: CPU is preempted after time quantum expires Fair: All processes get equal CPU time in rotation FIFO Queue: Processes are arranged in circular queue No Starvation: Every process eventually gets CPU time Round Robin Algorithm Flow Round Robin Algorithm Flow Diagram - CPU Sch","date":"29-Nov-2024","heading":"Quantum Comparison Example","objectID":"\/posts\/operating-system\/:7:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#quantum-comparison-example"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Algorithms (Complete Analysis) 1. First-Come-First-Served (FCFS) Algorithm: Process executes for entire duration, no preemption \/\/ FCFS Scheduler Implementation struct process { int pid; int arrival_time; int burst_time; int start_time; int end_time; }; void fcfs_schedule(struct process procs[], int n) { int current_time = 0; double total_turnaround = 0, total_wait = 0; for (int i = 0; i < n; i++) { procs[i].start_time = current_time; procs[i].end_time = current_time + procs[i].burst_time; current_time = procs[i].end_time; int turnaround = procs[i].end_time - procs[i].arrival_time; int wait = procs[i].start_time - procs[i].arrival_time; total_turnaround += turnaround; total_wait += wait; printf(\"P%d: Wait=%d, Turnaround=%d\\n\", procs[i].pid, wait, turnaround); } printf(\"Average Wait Time: %.2f\\n\", total_wait \/ n); printf(\"Average Turnaround: %.2f\\n\", total_turnaround \/ n); } \/* Example: Processes with burst times [24, 3, 3] FCFS: P1_____________________ P2__ P3__ 0 24 27 30 Waiting times: P1=0, P2=24, P3=27 Average wait: 17ms *\/ Warning Convoy Effect Problem: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation. 2. Shortest Job First (SJF) Non-preemptive: Each process runs completely void sjf_schedule(struct process procs[], int n) { \/\/ Sort by burst time (ascending) for (int i = 0; i < n - 1; i++) { for (int j = 0; j < n - i - 1; j++) { if (procs[j].burst_time > procs[j+1].burst_time) { struct process temp = procs[j]; procs[j] = procs[j+1]; procs[j+1] = temp; } } } fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list } \/* Example: Processes [24, 3, 3] Sorted: [3, 3, 24] Gantt Chart: P2_ P3_ P1_____________________ 0 3 6 30 Average wait: (0 + 3 + 6) \/ 3 = 3ms (vs 17ms for FCFS!) *\/ Tip SJF Optimization: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an 82% reduction! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously. Warning Starvation Risk: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms. Preemptive SJF (Shortest Remaining Time First - SRTF) void srtf_schedule(struct process procs[], int n) { int remaining[n]; for (int i = 0; i < n; i++) { remaining[i] = procs[i].burst_time; } int time = 0; int total_processes = n; while (total_processes > 0) { \/\/ Find process with shortest remaining time int min_index = -1; int min_remaining = INT_MAX; for (int i = 0; i < n; i++) { if (remaining[i] > 0 && remaining[i] < min_remaining && procs[i].arrival_time <= time) { min_remaining = remaining[i]; min_index = i; } } if (min_index == -1) { time++; continue; } \/\/ Execute for 1 time unit remaining[min_index]--; time++; if (remaining[min_index] == 0) { total_processes--; } } } 3. Round Robin (RR) Scheduling Overview: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It’s designed specifically for time-sharing systems where fairness and responsiveness are critical. Key Concept: Time Quantum (Time Slice) - Fixed time period allocated to each process Info Time Quantum Selection is Critical: Too small (1-10ms): Excessive context switching overhead \u2192 CPU thrashing Too large (100ms+): Behaves like FCFS \u2192 poor response time Sweet spot: 80% of CPU bursts should be shorter than the quantum Linux default: ~100ms (configurable via sched_latency_ns) Characteristics: Preemptive: CPU is preempted after time quantum expires Fair: All processes get equal CPU time in rotation FIFO Queue: Processes are arranged in circular queue No Starvation: Every process eventually gets CPU time Round Robin Algorithm Flow Round Robin Algorithm Flow Diagram - CPU Sch","date":"29-Nov-2024","heading":"Advantages and Disadvantages","objectID":"\/posts\/operating-system\/:7:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#advantages-and-disadvantages"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Algorithms (Complete Analysis) 1. First-Come-First-Served (FCFS) Algorithm: Process executes for entire duration, no preemption \/\/ FCFS Scheduler Implementation struct process { int pid; int arrival_time; int burst_time; int start_time; int end_time; }; void fcfs_schedule(struct process procs[], int n) { int current_time = 0; double total_turnaround = 0, total_wait = 0; for (int i = 0; i < n; i++) { procs[i].start_time = current_time; procs[i].end_time = current_time + procs[i].burst_time; current_time = procs[i].end_time; int turnaround = procs[i].end_time - procs[i].arrival_time; int wait = procs[i].start_time - procs[i].arrival_time; total_turnaround += turnaround; total_wait += wait; printf(\"P%d: Wait=%d, Turnaround=%d\\n\", procs[i].pid, wait, turnaround); } printf(\"Average Wait Time: %.2f\\n\", total_wait \/ n); printf(\"Average Turnaround: %.2f\\n\", total_turnaround \/ n); } \/* Example: Processes with burst times [24, 3, 3] FCFS: P1_____________________ P2__ P3__ 0 24 27 30 Waiting times: P1=0, P2=24, P3=27 Average wait: 17ms *\/ Warning Convoy Effect Problem: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation. 2. Shortest Job First (SJF) Non-preemptive: Each process runs completely void sjf_schedule(struct process procs[], int n) { \/\/ Sort by burst time (ascending) for (int i = 0; i < n - 1; i++) { for (int j = 0; j < n - i - 1; j++) { if (procs[j].burst_time > procs[j+1].burst_time) { struct process temp = procs[j]; procs[j] = procs[j+1]; procs[j+1] = temp; } } } fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list } \/* Example: Processes [24, 3, 3] Sorted: [3, 3, 24] Gantt Chart: P2_ P3_ P1_____________________ 0 3 6 30 Average wait: (0 + 3 + 6) \/ 3 = 3ms (vs 17ms for FCFS!) *\/ Tip SJF Optimization: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an 82% reduction! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously. Warning Starvation Risk: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms. Preemptive SJF (Shortest Remaining Time First - SRTF) void srtf_schedule(struct process procs[], int n) { int remaining[n]; for (int i = 0; i < n; i++) { remaining[i] = procs[i].burst_time; } int time = 0; int total_processes = n; while (total_processes > 0) { \/\/ Find process with shortest remaining time int min_index = -1; int min_remaining = INT_MAX; for (int i = 0; i < n; i++) { if (remaining[i] > 0 && remaining[i] < min_remaining && procs[i].arrival_time <= time) { min_remaining = remaining[i]; min_index = i; } } if (min_index == -1) { time++; continue; } \/\/ Execute for 1 time unit remaining[min_index]--; time++; if (remaining[min_index] == 0) { total_processes--; } } } 3. Round Robin (RR) Scheduling Overview: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It’s designed specifically for time-sharing systems where fairness and responsiveness are critical. Key Concept: Time Quantum (Time Slice) - Fixed time period allocated to each process Info Time Quantum Selection is Critical: Too small (1-10ms): Excessive context switching overhead \u2192 CPU thrashing Too large (100ms+): Behaves like FCFS \u2192 poor response time Sweet spot: 80% of CPU bursts should be shorter than the quantum Linux default: ~100ms (configurable via sched_latency_ns) Characteristics: Preemptive: CPU is preempted after time quantum expires Fair: All processes get equal CPU time in rotation FIFO Queue: Processes are arranged in circular queue No Starvation: Every process eventually gets CPU time Round Robin Algorithm Flow Round Robin Algorithm Flow Diagram - CPU Sch","date":"29-Nov-2024","heading":"Real-World Usage","objectID":"\/posts\/operating-system\/:7:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#real-world-usage"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Algorithms (Complete Analysis) 1. First-Come-First-Served (FCFS) Algorithm: Process executes for entire duration, no preemption \/\/ FCFS Scheduler Implementation struct process { int pid; int arrival_time; int burst_time; int start_time; int end_time; }; void fcfs_schedule(struct process procs[], int n) { int current_time = 0; double total_turnaround = 0, total_wait = 0; for (int i = 0; i < n; i++) { procs[i].start_time = current_time; procs[i].end_time = current_time + procs[i].burst_time; current_time = procs[i].end_time; int turnaround = procs[i].end_time - procs[i].arrival_time; int wait = procs[i].start_time - procs[i].arrival_time; total_turnaround += turnaround; total_wait += wait; printf(\"P%d: Wait=%d, Turnaround=%d\\n\", procs[i].pid, wait, turnaround); } printf(\"Average Wait Time: %.2f\\n\", total_wait \/ n); printf(\"Average Turnaround: %.2f\\n\", total_turnaround \/ n); } \/* Example: Processes with burst times [24, 3, 3] FCFS: P1_____________________ P2__ P3__ 0 24 27 30 Waiting times: P1=0, P2=24, P3=27 Average wait: 17ms *\/ Warning Convoy Effect Problem: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation. 2. Shortest Job First (SJF) Non-preemptive: Each process runs completely void sjf_schedule(struct process procs[], int n) { \/\/ Sort by burst time (ascending) for (int i = 0; i < n - 1; i++) { for (int j = 0; j < n - i - 1; j++) { if (procs[j].burst_time > procs[j+1].burst_time) { struct process temp = procs[j]; procs[j] = procs[j+1]; procs[j+1] = temp; } } } fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list } \/* Example: Processes [24, 3, 3] Sorted: [3, 3, 24] Gantt Chart: P2_ P3_ P1_____________________ 0 3 6 30 Average wait: (0 + 3 + 6) \/ 3 = 3ms (vs 17ms for FCFS!) *\/ Tip SJF Optimization: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an 82% reduction! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously. Warning Starvation Risk: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms. Preemptive SJF (Shortest Remaining Time First - SRTF) void srtf_schedule(struct process procs[], int n) { int remaining[n]; for (int i = 0; i < n; i++) { remaining[i] = procs[i].burst_time; } int time = 0; int total_processes = n; while (total_processes > 0) { \/\/ Find process with shortest remaining time int min_index = -1; int min_remaining = INT_MAX; for (int i = 0; i < n; i++) { if (remaining[i] > 0 && remaining[i] < min_remaining && procs[i].arrival_time <= time) { min_remaining = remaining[i]; min_index = i; } } if (min_index == -1) { time++; continue; } \/\/ Execute for 1 time unit remaining[min_index]--; time++; if (remaining[min_index] == 0) { total_processes--; } } } 3. Round Robin (RR) Scheduling Overview: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It’s designed specifically for time-sharing systems where fairness and responsiveness are critical. Key Concept: Time Quantum (Time Slice) - Fixed time period allocated to each process Info Time Quantum Selection is Critical: Too small (1-10ms): Excessive context switching overhead \u2192 CPU thrashing Too large (100ms+): Behaves like FCFS \u2192 poor response time Sweet spot: 80% of CPU bursts should be shorter than the quantum Linux default: ~100ms (configurable via sched_latency_ns) Characteristics: Preemptive: CPU is preempted after time quantum expires Fair: All processes get equal CPU time in rotation FIFO Queue: Processes are arranged in circular queue No Starvation: Every process eventually gets CPU time Round Robin Algorithm Flow Round Robin Algorithm Flow Diagram - CPU Sch","date":"29-Nov-2024","heading":"Practice Problems","objectID":"\/posts\/operating-system\/:7:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#practice-problems"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Algorithms (Complete Analysis) 1. First-Come-First-Served (FCFS) Algorithm: Process executes for entire duration, no preemption \/\/ FCFS Scheduler Implementation struct process { int pid; int arrival_time; int burst_time; int start_time; int end_time; }; void fcfs_schedule(struct process procs[], int n) { int current_time = 0; double total_turnaround = 0, total_wait = 0; for (int i = 0; i < n; i++) { procs[i].start_time = current_time; procs[i].end_time = current_time + procs[i].burst_time; current_time = procs[i].end_time; int turnaround = procs[i].end_time - procs[i].arrival_time; int wait = procs[i].start_time - procs[i].arrival_time; total_turnaround += turnaround; total_wait += wait; printf(\"P%d: Wait=%d, Turnaround=%d\\n\", procs[i].pid, wait, turnaround); } printf(\"Average Wait Time: %.2f\\n\", total_wait \/ n); printf(\"Average Turnaround: %.2f\\n\", total_turnaround \/ n); } \/* Example: Processes with burst times [24, 3, 3] FCFS: P1_____________________ P2__ P3__ 0 24 27 30 Waiting times: P1=0, P2=24, P3=27 Average wait: 17ms *\/ Warning Convoy Effect Problem: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation. 2. Shortest Job First (SJF) Non-preemptive: Each process runs completely void sjf_schedule(struct process procs[], int n) { \/\/ Sort by burst time (ascending) for (int i = 0; i < n - 1; i++) { for (int j = 0; j < n - i - 1; j++) { if (procs[j].burst_time > procs[j+1].burst_time) { struct process temp = procs[j]; procs[j] = procs[j+1]; procs[j+1] = temp; } } } fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list } \/* Example: Processes [24, 3, 3] Sorted: [3, 3, 24] Gantt Chart: P2_ P3_ P1_____________________ 0 3 6 30 Average wait: (0 + 3 + 6) \/ 3 = 3ms (vs 17ms for FCFS!) *\/ Tip SJF Optimization: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an 82% reduction! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously. Warning Starvation Risk: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms. Preemptive SJF (Shortest Remaining Time First - SRTF) void srtf_schedule(struct process procs[], int n) { int remaining[n]; for (int i = 0; i < n; i++) { remaining[i] = procs[i].burst_time; } int time = 0; int total_processes = n; while (total_processes > 0) { \/\/ Find process with shortest remaining time int min_index = -1; int min_remaining = INT_MAX; for (int i = 0; i < n; i++) { if (remaining[i] > 0 && remaining[i] < min_remaining && procs[i].arrival_time <= time) { min_remaining = remaining[i]; min_index = i; } } if (min_index == -1) { time++; continue; } \/\/ Execute for 1 time unit remaining[min_index]--; time++; if (remaining[min_index] == 0) { total_processes--; } } } 3. Round Robin (RR) Scheduling Overview: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It’s designed specifically for time-sharing systems where fairness and responsiveness are critical. Key Concept: Time Quantum (Time Slice) - Fixed time period allocated to each process Info Time Quantum Selection is Critical: Too small (1-10ms): Excessive context switching overhead \u2192 CPU thrashing Too large (100ms+): Behaves like FCFS \u2192 poor response time Sweet spot: 80% of CPU bursts should be shorter than the quantum Linux default: ~100ms (configurable via sched_latency_ns) Characteristics: Preemptive: CPU is preempted after time quantum expires Fair: All processes get equal CPU time in rotation FIFO Queue: Processes are arranged in circular queue No Starvation: Every process eventually gets CPU time Round Robin Algorithm Flow Round Robin Algorithm Flow Diagram - CPU Sch","date":"29-Nov-2024","heading":"4. Priority Scheduling","objectID":"\/posts\/operating-system\/:7:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#4-priority-scheduling"},{"categories":["computer-science"],"collections":null,"content":"Scheduling Algorithms (Complete Analysis) 1. First-Come-First-Served (FCFS) Algorithm: Process executes for entire duration, no preemption \/\/ FCFS Scheduler Implementation struct process { int pid; int arrival_time; int burst_time; int start_time; int end_time; }; void fcfs_schedule(struct process procs[], int n) { int current_time = 0; double total_turnaround = 0, total_wait = 0; for (int i = 0; i < n; i++) { procs[i].start_time = current_time; procs[i].end_time = current_time + procs[i].burst_time; current_time = procs[i].end_time; int turnaround = procs[i].end_time - procs[i].arrival_time; int wait = procs[i].start_time - procs[i].arrival_time; total_turnaround += turnaround; total_wait += wait; printf(\"P%d: Wait=%d, Turnaround=%d\\n\", procs[i].pid, wait, turnaround); } printf(\"Average Wait Time: %.2f\\n\", total_wait \/ n); printf(\"Average Turnaround: %.2f\\n\", total_turnaround \/ n); } \/* Example: Processes with burst times [24, 3, 3] FCFS: P1_____________________ P2__ P3__ 0 24 27 30 Waiting times: P1=0, P2=24, P3=27 Average wait: 17ms *\/ Warning Convoy Effect Problem: Short processes become stuck behind long ones in the queue, causing massive average waiting time degradation. In this example, P2 and P3 must wait 24 and 27 time units respectively just for P1 to finish - this is highly inefficient! FCFS is rarely used in practice due to this critical limitation. 2. Shortest Job First (SJF) Non-preemptive: Each process runs completely void sjf_schedule(struct process procs[], int n) { \/\/ Sort by burst time (ascending) for (int i = 0; i < n - 1; i++) { for (int j = 0; j < n - i - 1; j++) { if (procs[j].burst_time > procs[j+1].burst_time) { struct process temp = procs[j]; procs[j] = procs[j+1]; procs[j+1] = temp; } } } fcfs_schedule(procs, n); \/\/ Then apply FCFS on sorted list } \/* Example: Processes [24, 3, 3] Sorted: [3, 3, 24] Gantt Chart: P2_ P3_ P1_____________________ 0 3 6 30 Average wait: (0 + 3 + 6) \/ 3 = 3ms (vs 17ms for FCFS!) *\/ Tip SJF Optimization: Notice the dramatic improvement! Average waiting time dropped from 17ms (FCFS) to just 3ms (SJF) - an 82% reduction! SJF is provably optimal for minimizing average waiting time when all processes arrive simultaneously. Warning Starvation Risk: Long processes may never execute if short processes keep arriving. Not practical in real systems without aging mechanisms. Preemptive SJF (Shortest Remaining Time First - SRTF) void srtf_schedule(struct process procs[], int n) { int remaining[n]; for (int i = 0; i < n; i++) { remaining[i] = procs[i].burst_time; } int time = 0; int total_processes = n; while (total_processes > 0) { \/\/ Find process with shortest remaining time int min_index = -1; int min_remaining = INT_MAX; for (int i = 0; i < n; i++) { if (remaining[i] > 0 && remaining[i] < min_remaining && procs[i].arrival_time <= time) { min_remaining = remaining[i]; min_index = i; } } if (min_index == -1) { time++; continue; } \/\/ Execute for 1 time unit remaining[min_index]--; time++; if (remaining[min_index] == 0) { total_processes--; } } } 3. Round Robin (RR) Scheduling Overview: Round Robin is a preemptive scheduling algorithm that assigns a fixed time slice (quantum) to each process in the ready queue. It’s designed specifically for time-sharing systems where fairness and responsiveness are critical. Key Concept: Time Quantum (Time Slice) - Fixed time period allocated to each process Info Time Quantum Selection is Critical: Too small (1-10ms): Excessive context switching overhead \u2192 CPU thrashing Too large (100ms+): Behaves like FCFS \u2192 poor response time Sweet spot: 80% of CPU bursts should be shorter than the quantum Linux default: ~100ms (configurable via sched_latency_ns) Characteristics: Preemptive: CPU is preempted after time quantum expires Fair: All processes get equal CPU time in rotation FIFO Queue: Processes are arranged in circular queue No Starvation: Every process eventually gets CPU time Round Robin Algorithm Flow Round Robin Algorithm Flow Diagram - CPU Sch","date":"29-Nov-2024","heading":"5. Linux CFS (Completely Fair Scheduler)","objectID":"\/posts\/operating-system\/:7:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#5-linux-cfs-completely-fair-scheduler"},{"categories":["computer-science"],"collections":null,"content":"Context Switching Overhead Costs of Context Switching: Direct Cost: ~1-10 microseconds - Save\/restore registers - Switch address space (if different process) Indirect Cost: ~milliseconds - Cache misses (cold cache) - TLB misses - Increased memory bandwidth Example: - Context switch: 10 \u03bcs - CPU stall for cache miss: 100 ns - 100+ misses per switch = 10 \u03bcs + 10 \u03bcs = 20 \u03bcs visible cost ","date":"29-Nov-2024","heading":"Context Switching Overhead","objectID":"\/posts\/operating-system\/:7:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#context-switching-overhead"},{"categories":["computer-science"],"collections":null,"content":"8. Synchronization Primitives ","date":"29-Nov-2024","heading":"8. Synchronization Primitives","objectID":"\/posts\/operating-system\/:8:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#synchronization"},{"categories":["computer-science"],"collections":null,"content":"Race Conditions and Critical Sections Race Condition: Multiple threads\/processes access shared data, result depends on timing \/\/ Classic race condition example int counter = 0; \/\/ Without synchronization (WRONG!) void increment_unsafe() { counter++; \/\/ Not atomic! Translates to: \/\/ 1. Load counter into register \/\/ 2. Increment register \/\/ 3. Store register back to counter } \/\/ With synchronization (CORRECT) pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; void increment_safe() { pthread_mutex_lock(&lock); counter++; \/\/ Now atomic pthread_mutex_unlock(&lock); } \/* Without locks, with 1000 increments per thread (2 threads): Expected: 2000 Actual: Random (often 1200, 1500, etc.) Reason: Interleaved execution: Thread 1: Load (0) Increment (1) ... Thread 2: Load (0) Increment (1) Store (1) ... Thread 1: ... Store (1) Result: Both stored 1, lost an increment *\/ ","date":"29-Nov-2024","heading":"Race Conditions and Critical Sections","objectID":"\/posts\/operating-system\/:8:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#race-conditions-and-critical-sections"},{"categories":["computer-science"],"collections":null,"content":"Semaphores Detailed Binary Semaphore (value: 0 or 1) #include sem_t mutex; sem_init(&mutex, 0, 1); \/\/ Initial value = 1 \/\/ Usage sem_wait(&mutex); \/\/ P() operation - decrement \/\/ Critical section sem_post(&mutex); \/\/ V() operation - increment Counting Semaphore (value: 0 to N) sem_t available_resources; sem_init(&available_resources, 0, 5); \/\/ 5 resources available void* worker(void *arg) { sem_wait(&available_resources); \/\/ Acquire resource \/\/ Use resource printf(\"Using resource\\n\"); sleep(1); sem_post(&available_resources); \/\/ Release resource return NULL; } \/* Example with 3 threads and 2 resources: Thread 1: sem_wait() \u2192 success (available=1) Thread 2: sem_wait() \u2192 success (available=0) Thread 3: sem_wait() \u2192 blocks (available=0) Thread 1: sem_post() \u2192 (available=1), wakes Thread 3 *\/ ","date":"29-Nov-2024","heading":"Semaphores Detailed","objectID":"\/posts\/operating-system\/:8:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#semaphores-detailed"},{"categories":["computer-science"],"collections":null,"content":"Readers-Writers Problem Requirement: Multiple readers can access resource simultaneously, but writers need exclusive access #include #include #include int read_count = 0; int shared_data = 0; pthread_mutex_t read_count_lock = PTHREAD_MUTEX_INITIALIZER; sem_t write_lock; void* reader(void *arg) { while (1) { \/\/ Entry section pthread_mutex_lock(&read_count_lock); read_count++; if (read_count == 1) { sem_wait(&write_lock); \/\/ First reader blocks writers } pthread_mutex_unlock(&read_count_lock); \/\/ Critical section (read) printf(\"Reader %ld: %d\\n\", (long)pthread_self(), shared_data); \/\/ Exit section pthread_mutex_lock(&read_count_lock); read_count--; if (read_count == 0) { sem_post(&write_lock); \/\/ Last reader allows writers } pthread_mutex_unlock(&read_count_lock); } return NULL; } void* writer(void *arg) { while (1) { sem_wait(&write_lock); \/\/ Exclusive access \/\/ Critical section (write) shared_data++; printf(\"Writer %ld: Updated to %d\\n\", (long)pthread_self(), shared_data); sem_post(&write_lock); } return NULL; } ","date":"29-Nov-2024","heading":"Readers-Writers Problem","objectID":"\/posts\/operating-system\/:8:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#readers-writers-problem"},{"categories":["computer-science"],"collections":null,"content":"Atomic Operations and Memory Barriers #include #include \/\/ Atomic variable atomic_int counter = ATOMIC_VAR_INIT(0); void increment_atomic() { atomic_fetch_add(&counter, 1); \/\/ Guaranteed atomic } \/\/ Memory barriers void process_data(int *data) { *data = 42; atomic_thread_fence(memory_order_release); \/\/ Release barrier \/\/ All writes before this point visible to other threads } void read_data(int *data) { atomic_thread_fence(memory_order_acquire); \/\/ Acquire barrier \/\/ All reads after this point see changes made before int value = *data; } \/* Memory Ordering Options: - memory_order_relaxed: No synchronization - memory_order_release: Release semantics - memory_order_acquire: Acquire semantics - memory_order_seq_cst: Sequential consistency (default, most expensive) *\/ ","date":"29-Nov-2024","heading":"Atomic Operations and Memory Barriers","objectID":"\/posts\/operating-system\/:8:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#atomic-operations-and-memory-barriers"},{"categories":["computer-science"],"collections":null,"content":"Lock-Free Data Structures #include \/\/ Lock-free counter typedef struct { atomic_long value; } LockFreeCounter; void increment(LockFreeCounter *counter) { atomic_fetch_add(&counter->value, 1); } \/\/ Lock-free stack using Compare-And-Swap (CAS) typedef struct Node { int data; struct Node *next; } Node; typedef struct { atomic_ptr_t top; } LockFreeStack; void push(LockFreeStack *stack, int value) { Node *new_node = malloc(sizeof(Node)); new_node->data = value; Node *old_top; do { old_top = atomic_load(&stack->top); new_node->next = old_top; } while (!atomic_compare_exchange_strong(&stack->top, &old_top, new_node)); } \/* CAS (Compare-And-Swap): - Atomically compare value at address with expected - If equal, store new value - Return whether operation succeeded - No explicit locks needed - Wait-free for many scenarios *\/ ","date":"29-Nov-2024","heading":"Lock-Free Data Structures","objectID":"\/posts\/operating-system\/:8:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#lock-free-data-structures"},{"categories":["computer-science"],"collections":null,"content":"9. Deadlock Handling ","date":"29-Nov-2024","heading":"9. Deadlock Handling","objectID":"\/posts\/operating-system\/:9:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#deadlock-handling"},{"categories":["computer-science"],"collections":null,"content":"9.1 Understanding Deadlocks A deadlock is a situation where a set of processes are blocked because each process is holding a resource and waiting for another resource held by another process in the set. Coffman Conditions (Necessary Conditions for Deadlock) For a deadlock to occur, all four conditions must hold simultaneously: Mutual Exclusion: At least one resource must be held in a non-shareable mode Hold and Wait: A process must be holding at least one resource and waiting for additional resources No Preemption: Resources cannot be forcibly removed from processes Circular Wait: A circular chain of processes exists where each process holds resources needed by the next \/* * Example: Deadlock Scenario * Process P1 holds Resource R1, needs R2 * Process P2 holds Resource R2, needs R1 *\/ #include #include #include pthread_mutex_t resource1 = PTHREAD_MUTEX_INITIALIZER; pthread_mutex_t resource2 = PTHREAD_MUTEX_INITIALIZER; void* process1(void *arg) { printf(\"P1: Acquiring Resource 1...\\n\"); pthread_mutex_lock(&resource1); printf(\"P1: Got Resource 1\\n\"); sleep(1); \/\/ Simulate some work printf(\"P1: Acquiring Resource 2...\\n\"); pthread_mutex_lock(&resource2); \/\/ DEADLOCK! P2 holds R2 printf(\"P1: Got Resource 2\\n\"); \/\/ Critical section printf(\"P1: Working with both resources\\n\"); pthread_mutex_unlock(&resource2); pthread_mutex_unlock(&resource1); return NULL; } void* process2(void *arg) { printf(\"P2: Acquiring Resource 2...\\n\"); pthread_mutex_lock(&resource2); printf(\"P2: Got Resource 2\\n\"); sleep(1); \/\/ Simulate some work printf(\"P2: Acquiring Resource 1...\\n\"); pthread_mutex_lock(&resource1); \/\/ DEADLOCK! P1 holds R1 printf(\"P2: Got Resource 1\\n\"); \/\/ Critical section printf(\"P2: Working with both resources\\n\"); pthread_mutex_unlock(&resource1); pthread_mutex_unlock(&resource2); return NULL; } int main() { pthread_t t1, t2; pthread_create(&t1, NULL, process1, NULL); pthread_create(&t2, NULL, process2, NULL); pthread_join(t1, NULL); pthread_join(t2, NULL); return 0; } ","date":"29-Nov-2024","heading":"9.1 Understanding Deadlocks","objectID":"\/posts\/operating-system\/:9:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#91-understanding-deadlocks"},{"categories":["computer-science"],"collections":null,"content":"9.1 Understanding Deadlocks A deadlock is a situation where a set of processes are blocked because each process is holding a resource and waiting for another resource held by another process in the set. Coffman Conditions (Necessary Conditions for Deadlock) For a deadlock to occur, all four conditions must hold simultaneously: Mutual Exclusion: At least one resource must be held in a non-shareable mode Hold and Wait: A process must be holding at least one resource and waiting for additional resources No Preemption: Resources cannot be forcibly removed from processes Circular Wait: A circular chain of processes exists where each process holds resources needed by the next \/* * Example: Deadlock Scenario * Process P1 holds Resource R1, needs R2 * Process P2 holds Resource R2, needs R1 *\/ #include #include #include pthread_mutex_t resource1 = PTHREAD_MUTEX_INITIALIZER; pthread_mutex_t resource2 = PTHREAD_MUTEX_INITIALIZER; void* process1(void *arg) { printf(\"P1: Acquiring Resource 1...\\n\"); pthread_mutex_lock(&resource1); printf(\"P1: Got Resource 1\\n\"); sleep(1); \/\/ Simulate some work printf(\"P1: Acquiring Resource 2...\\n\"); pthread_mutex_lock(&resource2); \/\/ DEADLOCK! P2 holds R2 printf(\"P1: Got Resource 2\\n\"); \/\/ Critical section printf(\"P1: Working with both resources\\n\"); pthread_mutex_unlock(&resource2); pthread_mutex_unlock(&resource1); return NULL; } void* process2(void *arg) { printf(\"P2: Acquiring Resource 2...\\n\"); pthread_mutex_lock(&resource2); printf(\"P2: Got Resource 2\\n\"); sleep(1); \/\/ Simulate some work printf(\"P2: Acquiring Resource 1...\\n\"); pthread_mutex_lock(&resource1); \/\/ DEADLOCK! P1 holds R1 printf(\"P2: Got Resource 1\\n\"); \/\/ Critical section printf(\"P2: Working with both resources\\n\"); pthread_mutex_unlock(&resource1); pthread_mutex_unlock(&resource2); return NULL; } int main() { pthread_t t1, t2; pthread_create(&t1, NULL, process1, NULL); pthread_create(&t2, NULL, process2, NULL); pthread_join(t1, NULL); pthread_join(t2, NULL); return 0; } ","date":"29-Nov-2024","heading":"Coffman Conditions (Necessary Conditions for Deadlock)","objectID":"\/posts\/operating-system\/:9:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#coffman-conditions-necessary-conditions-for-deadlock"},{"categories":["computer-science"],"collections":null,"content":"9.2 Deadlock Prevention Strategy: Ensure at least one Coffman condition cannot hold. 9.2.1 Breaking Mutual Exclusion Make resources shareable (not always possible) Use read-write locks for read-heavy workloads #include pthread_rwlock_t shared_resource = PTHREAD_RWLOCK_INITIALIZER; void* reader(void *arg) { pthread_rwlock_rdlock(&shared_resource); \/\/ Multiple readers OK \/\/ Read data printf(\"Reader %ld reading\\n\", (long)pthread_self()); pthread_rwlock_unlock(&shared_resource); return NULL; } void* writer(void *arg) { pthread_rwlock_wrlock(&shared_resource); \/\/ Exclusive write \/\/ Write data printf(\"Writer %ld writing\\n\", (long)pthread_self()); pthread_rwlock_unlock(&shared_resource); return NULL; } 9.2.2 Breaking Hold and Wait Require processes to request all resources at once #include #include typedef struct { pthread_mutex_t *locks; int num_locks; pthread_mutex_t allocation_lock; } ResourceManager; \/\/ Acquire all resources atomically bool acquire_all_resources(ResourceManager *rm, int *resource_ids, int count) { pthread_mutex_lock(&rm->allocation_lock); \/\/ Try to acquire all locks for (int i = 0; i < count; i++) { if (pthread_mutex_trylock(&rm->locks[resource_ids[i]]) != 0) { \/\/ Failed to acquire, release all acquired locks for (int j = 0; j < i; j++) { pthread_mutex_unlock(&rm->locks[resource_ids[j]]); } pthread_mutex_unlock(&rm->allocation_lock); return false; } } pthread_mutex_unlock(&rm->allocation_lock); return true; } 9.2.3 Breaking No Preemption Allow resources to be preempted (forcibly removed) Save process state and restore later \/\/ Conceptual example - actual implementation is OS-specific void preemptable_resource_request(Process *p, Resource *r) { if (!try_acquire(r)) { \/\/ Release all currently held resources release_all_resources(p); \/\/ Save process state save_state(p); \/\/ Add to waiting queue add_to_queue(p, r); \/\/ Context switch to another process schedule_next_process(); } } 9.2.4 Breaking Circular Wait Resource Ordering: Impose total ordering on all resources \/* * Deadlock Prevention: Resource Ordering * Always acquire resources in increasing order of their IDs *\/ #include #include pthread_mutex_t resource1 = PTHREAD_MUTEX_INITIALIZER; \/\/ ID: 1 pthread_mutex_t resource2 = PTHREAD_MUTEX_INITIALIZER; \/\/ ID: 2 void* safe_process1(void *arg) { \/\/ Always acquire in order: R1 before R2 pthread_mutex_lock(&resource1); printf(\"P1: Got Resource 1\\n\"); pthread_mutex_lock(&resource2); printf(\"P1: Got Resource 2\\n\"); \/\/ Critical section - NO DEADLOCK! printf(\"P1: Working safely\\n\"); pthread_mutex_unlock(&resource2); pthread_mutex_unlock(&resource1); return NULL; } void* safe_process2(void *arg) { \/\/ Also acquire in order: R1 before R2 pthread_mutex_lock(&resource1); printf(\"P2: Got Resource 1\\n\"); pthread_mutex_lock(&resource2); printf(\"P2: Got Resource 2\\n\"); \/\/ Critical section printf(\"P2: Working safely\\n\"); pthread_mutex_unlock(&resource2); pthread_mutex_unlock(&resource1); return NULL; } ","date":"29-Nov-2024","heading":"9.2 Deadlock Prevention","objectID":"\/posts\/operating-system\/:9:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#92-deadlock-prevention"},{"categories":["computer-science"],"collections":null,"content":"9.2 Deadlock Prevention Strategy: Ensure at least one Coffman condition cannot hold. 9.2.1 Breaking Mutual Exclusion Make resources shareable (not always possible) Use read-write locks for read-heavy workloads #include pthread_rwlock_t shared_resource = PTHREAD_RWLOCK_INITIALIZER; void* reader(void *arg) { pthread_rwlock_rdlock(&shared_resource); \/\/ Multiple readers OK \/\/ Read data printf(\"Reader %ld reading\\n\", (long)pthread_self()); pthread_rwlock_unlock(&shared_resource); return NULL; } void* writer(void *arg) { pthread_rwlock_wrlock(&shared_resource); \/\/ Exclusive write \/\/ Write data printf(\"Writer %ld writing\\n\", (long)pthread_self()); pthread_rwlock_unlock(&shared_resource); return NULL; } 9.2.2 Breaking Hold and Wait Require processes to request all resources at once #include #include typedef struct { pthread_mutex_t *locks; int num_locks; pthread_mutex_t allocation_lock; } ResourceManager; \/\/ Acquire all resources atomically bool acquire_all_resources(ResourceManager *rm, int *resource_ids, int count) { pthread_mutex_lock(&rm->allocation_lock); \/\/ Try to acquire all locks for (int i = 0; i < count; i++) { if (pthread_mutex_trylock(&rm->locks[resource_ids[i]]) != 0) { \/\/ Failed to acquire, release all acquired locks for (int j = 0; j < i; j++) { pthread_mutex_unlock(&rm->locks[resource_ids[j]]); } pthread_mutex_unlock(&rm->allocation_lock); return false; } } pthread_mutex_unlock(&rm->allocation_lock); return true; } 9.2.3 Breaking No Preemption Allow resources to be preempted (forcibly removed) Save process state and restore later \/\/ Conceptual example - actual implementation is OS-specific void preemptable_resource_request(Process *p, Resource *r) { if (!try_acquire(r)) { \/\/ Release all currently held resources release_all_resources(p); \/\/ Save process state save_state(p); \/\/ Add to waiting queue add_to_queue(p, r); \/\/ Context switch to another process schedule_next_process(); } } 9.2.4 Breaking Circular Wait Resource Ordering: Impose total ordering on all resources \/* * Deadlock Prevention: Resource Ordering * Always acquire resources in increasing order of their IDs *\/ #include #include pthread_mutex_t resource1 = PTHREAD_MUTEX_INITIALIZER; \/\/ ID: 1 pthread_mutex_t resource2 = PTHREAD_MUTEX_INITIALIZER; \/\/ ID: 2 void* safe_process1(void *arg) { \/\/ Always acquire in order: R1 before R2 pthread_mutex_lock(&resource1); printf(\"P1: Got Resource 1\\n\"); pthread_mutex_lock(&resource2); printf(\"P1: Got Resource 2\\n\"); \/\/ Critical section - NO DEADLOCK! printf(\"P1: Working safely\\n\"); pthread_mutex_unlock(&resource2); pthread_mutex_unlock(&resource1); return NULL; } void* safe_process2(void *arg) { \/\/ Also acquire in order: R1 before R2 pthread_mutex_lock(&resource1); printf(\"P2: Got Resource 1\\n\"); pthread_mutex_lock(&resource2); printf(\"P2: Got Resource 2\\n\"); \/\/ Critical section printf(\"P2: Working safely\\n\"); pthread_mutex_unlock(&resource2); pthread_mutex_unlock(&resource1); return NULL; } ","date":"29-Nov-2024","heading":"9.2.1 Breaking Mutual Exclusion","objectID":"\/posts\/operating-system\/:9:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#921-breaking-mutual-exclusion"},{"categories":["computer-science"],"collections":null,"content":"9.2 Deadlock Prevention Strategy: Ensure at least one Coffman condition cannot hold. 9.2.1 Breaking Mutual Exclusion Make resources shareable (not always possible) Use read-write locks for read-heavy workloads #include pthread_rwlock_t shared_resource = PTHREAD_RWLOCK_INITIALIZER; void* reader(void *arg) { pthread_rwlock_rdlock(&shared_resource); \/\/ Multiple readers OK \/\/ Read data printf(\"Reader %ld reading\\n\", (long)pthread_self()); pthread_rwlock_unlock(&shared_resource); return NULL; } void* writer(void *arg) { pthread_rwlock_wrlock(&shared_resource); \/\/ Exclusive write \/\/ Write data printf(\"Writer %ld writing\\n\", (long)pthread_self()); pthread_rwlock_unlock(&shared_resource); return NULL; } 9.2.2 Breaking Hold and Wait Require processes to request all resources at once #include #include typedef struct { pthread_mutex_t *locks; int num_locks; pthread_mutex_t allocation_lock; } ResourceManager; \/\/ Acquire all resources atomically bool acquire_all_resources(ResourceManager *rm, int *resource_ids, int count) { pthread_mutex_lock(&rm->allocation_lock); \/\/ Try to acquire all locks for (int i = 0; i < count; i++) { if (pthread_mutex_trylock(&rm->locks[resource_ids[i]]) != 0) { \/\/ Failed to acquire, release all acquired locks for (int j = 0; j < i; j++) { pthread_mutex_unlock(&rm->locks[resource_ids[j]]); } pthread_mutex_unlock(&rm->allocation_lock); return false; } } pthread_mutex_unlock(&rm->allocation_lock); return true; } 9.2.3 Breaking No Preemption Allow resources to be preempted (forcibly removed) Save process state and restore later \/\/ Conceptual example - actual implementation is OS-specific void preemptable_resource_request(Process *p, Resource *r) { if (!try_acquire(r)) { \/\/ Release all currently held resources release_all_resources(p); \/\/ Save process state save_state(p); \/\/ Add to waiting queue add_to_queue(p, r); \/\/ Context switch to another process schedule_next_process(); } } 9.2.4 Breaking Circular Wait Resource Ordering: Impose total ordering on all resources \/* * Deadlock Prevention: Resource Ordering * Always acquire resources in increasing order of their IDs *\/ #include #include pthread_mutex_t resource1 = PTHREAD_MUTEX_INITIALIZER; \/\/ ID: 1 pthread_mutex_t resource2 = PTHREAD_MUTEX_INITIALIZER; \/\/ ID: 2 void* safe_process1(void *arg) { \/\/ Always acquire in order: R1 before R2 pthread_mutex_lock(&resource1); printf(\"P1: Got Resource 1\\n\"); pthread_mutex_lock(&resource2); printf(\"P1: Got Resource 2\\n\"); \/\/ Critical section - NO DEADLOCK! printf(\"P1: Working safely\\n\"); pthread_mutex_unlock(&resource2); pthread_mutex_unlock(&resource1); return NULL; } void* safe_process2(void *arg) { \/\/ Also acquire in order: R1 before R2 pthread_mutex_lock(&resource1); printf(\"P2: Got Resource 1\\n\"); pthread_mutex_lock(&resource2); printf(\"P2: Got Resource 2\\n\"); \/\/ Critical section printf(\"P2: Working safely\\n\"); pthread_mutex_unlock(&resource2); pthread_mutex_unlock(&resource1); return NULL; } ","date":"29-Nov-2024","heading":"9.2.2 Breaking Hold and Wait","objectID":"\/posts\/operating-system\/:9:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#922-breaking-hold-and-wait"},{"categories":["computer-science"],"collections":null,"content":"9.2 Deadlock Prevention Strategy: Ensure at least one Coffman condition cannot hold. 9.2.1 Breaking Mutual Exclusion Make resources shareable (not always possible) Use read-write locks for read-heavy workloads #include pthread_rwlock_t shared_resource = PTHREAD_RWLOCK_INITIALIZER; void* reader(void *arg) { pthread_rwlock_rdlock(&shared_resource); \/\/ Multiple readers OK \/\/ Read data printf(\"Reader %ld reading\\n\", (long)pthread_self()); pthread_rwlock_unlock(&shared_resource); return NULL; } void* writer(void *arg) { pthread_rwlock_wrlock(&shared_resource); \/\/ Exclusive write \/\/ Write data printf(\"Writer %ld writing\\n\", (long)pthread_self()); pthread_rwlock_unlock(&shared_resource); return NULL; } 9.2.2 Breaking Hold and Wait Require processes to request all resources at once #include #include typedef struct { pthread_mutex_t *locks; int num_locks; pthread_mutex_t allocation_lock; } ResourceManager; \/\/ Acquire all resources atomically bool acquire_all_resources(ResourceManager *rm, int *resource_ids, int count) { pthread_mutex_lock(&rm->allocation_lock); \/\/ Try to acquire all locks for (int i = 0; i < count; i++) { if (pthread_mutex_trylock(&rm->locks[resource_ids[i]]) != 0) { \/\/ Failed to acquire, release all acquired locks for (int j = 0; j < i; j++) { pthread_mutex_unlock(&rm->locks[resource_ids[j]]); } pthread_mutex_unlock(&rm->allocation_lock); return false; } } pthread_mutex_unlock(&rm->allocation_lock); return true; } 9.2.3 Breaking No Preemption Allow resources to be preempted (forcibly removed) Save process state and restore later \/\/ Conceptual example - actual implementation is OS-specific void preemptable_resource_request(Process *p, Resource *r) { if (!try_acquire(r)) { \/\/ Release all currently held resources release_all_resources(p); \/\/ Save process state save_state(p); \/\/ Add to waiting queue add_to_queue(p, r); \/\/ Context switch to another process schedule_next_process(); } } 9.2.4 Breaking Circular Wait Resource Ordering: Impose total ordering on all resources \/* * Deadlock Prevention: Resource Ordering * Always acquire resources in increasing order of their IDs *\/ #include #include pthread_mutex_t resource1 = PTHREAD_MUTEX_INITIALIZER; \/\/ ID: 1 pthread_mutex_t resource2 = PTHREAD_MUTEX_INITIALIZER; \/\/ ID: 2 void* safe_process1(void *arg) { \/\/ Always acquire in order: R1 before R2 pthread_mutex_lock(&resource1); printf(\"P1: Got Resource 1\\n\"); pthread_mutex_lock(&resource2); printf(\"P1: Got Resource 2\\n\"); \/\/ Critical section - NO DEADLOCK! printf(\"P1: Working safely\\n\"); pthread_mutex_unlock(&resource2); pthread_mutex_unlock(&resource1); return NULL; } void* safe_process2(void *arg) { \/\/ Also acquire in order: R1 before R2 pthread_mutex_lock(&resource1); printf(\"P2: Got Resource 1\\n\"); pthread_mutex_lock(&resource2); printf(\"P2: Got Resource 2\\n\"); \/\/ Critical section printf(\"P2: Working safely\\n\"); pthread_mutex_unlock(&resource2); pthread_mutex_unlock(&resource1); return NULL; } ","date":"29-Nov-2024","heading":"9.2.3 Breaking No Preemption","objectID":"\/posts\/operating-system\/:9:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#923-breaking-no-preemption"},{"categories":["computer-science"],"collections":null,"content":"9.2 Deadlock Prevention Strategy: Ensure at least one Coffman condition cannot hold. 9.2.1 Breaking Mutual Exclusion Make resources shareable (not always possible) Use read-write locks for read-heavy workloads #include pthread_rwlock_t shared_resource = PTHREAD_RWLOCK_INITIALIZER; void* reader(void *arg) { pthread_rwlock_rdlock(&shared_resource); \/\/ Multiple readers OK \/\/ Read data printf(\"Reader %ld reading\\n\", (long)pthread_self()); pthread_rwlock_unlock(&shared_resource); return NULL; } void* writer(void *arg) { pthread_rwlock_wrlock(&shared_resource); \/\/ Exclusive write \/\/ Write data printf(\"Writer %ld writing\\n\", (long)pthread_self()); pthread_rwlock_unlock(&shared_resource); return NULL; } 9.2.2 Breaking Hold and Wait Require processes to request all resources at once #include #include typedef struct { pthread_mutex_t *locks; int num_locks; pthread_mutex_t allocation_lock; } ResourceManager; \/\/ Acquire all resources atomically bool acquire_all_resources(ResourceManager *rm, int *resource_ids, int count) { pthread_mutex_lock(&rm->allocation_lock); \/\/ Try to acquire all locks for (int i = 0; i < count; i++) { if (pthread_mutex_trylock(&rm->locks[resource_ids[i]]) != 0) { \/\/ Failed to acquire, release all acquired locks for (int j = 0; j < i; j++) { pthread_mutex_unlock(&rm->locks[resource_ids[j]]); } pthread_mutex_unlock(&rm->allocation_lock); return false; } } pthread_mutex_unlock(&rm->allocation_lock); return true; } 9.2.3 Breaking No Preemption Allow resources to be preempted (forcibly removed) Save process state and restore later \/\/ Conceptual example - actual implementation is OS-specific void preemptable_resource_request(Process *p, Resource *r) { if (!try_acquire(r)) { \/\/ Release all currently held resources release_all_resources(p); \/\/ Save process state save_state(p); \/\/ Add to waiting queue add_to_queue(p, r); \/\/ Context switch to another process schedule_next_process(); } } 9.2.4 Breaking Circular Wait Resource Ordering: Impose total ordering on all resources \/* * Deadlock Prevention: Resource Ordering * Always acquire resources in increasing order of their IDs *\/ #include #include pthread_mutex_t resource1 = PTHREAD_MUTEX_INITIALIZER; \/\/ ID: 1 pthread_mutex_t resource2 = PTHREAD_MUTEX_INITIALIZER; \/\/ ID: 2 void* safe_process1(void *arg) { \/\/ Always acquire in order: R1 before R2 pthread_mutex_lock(&resource1); printf(\"P1: Got Resource 1\\n\"); pthread_mutex_lock(&resource2); printf(\"P1: Got Resource 2\\n\"); \/\/ Critical section - NO DEADLOCK! printf(\"P1: Working safely\\n\"); pthread_mutex_unlock(&resource2); pthread_mutex_unlock(&resource1); return NULL; } void* safe_process2(void *arg) { \/\/ Also acquire in order: R1 before R2 pthread_mutex_lock(&resource1); printf(\"P2: Got Resource 1\\n\"); pthread_mutex_lock(&resource2); printf(\"P2: Got Resource 2\\n\"); \/\/ Critical section printf(\"P2: Working safely\\n\"); pthread_mutex_unlock(&resource2); pthread_mutex_unlock(&resource1); return NULL; } ","date":"29-Nov-2024","heading":"9.2.4 Breaking Circular Wait","objectID":"\/posts\/operating-system\/:9:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#924-breaking-circular-wait"},{"categories":["computer-science"],"collections":null,"content":"9.3 Deadlock Avoidance Strategy: Make informed decisions about resource allocation to ensure the system never enters an unsafe state. 9.3.1 Safe State A state is safe if there exists a sequence of all processes <P1, P2, …, Pn> such that for each Pi, the resources that Pi can still request can be satisfied by: Currently available resources Resources held by all Pj where j < i 9.3.2 Banker’s Algorithm Developed by Edsger Dijkstra, simulates allocation to test for safe state. #include #include #include #define NUM_PROCESSES 5 #define NUM_RESOURCES 3 typedef struct { int available[NUM_RESOURCES]; \/\/ Available resources int maximum[NUM_PROCESSES][NUM_RESOURCES]; \/\/ Max demand int allocation[NUM_PROCESSES][NUM_RESOURCES]; \/\/ Currently allocated int need[NUM_PROCESSES][NUM_RESOURCES]; \/\/ Remaining need } BankersState; \/\/ Calculate need matrix: Need = Maximum - Allocation void calculate_need(BankersState *state) { for (int i = 0; i < NUM_PROCESSES; i++) { for (int j = 0; j < NUM_RESOURCES; j++) { state->need[i][j] = state->maximum[i][j] - state->allocation[i][j]; } } } \/\/ Check if system is in safe state bool is_safe_state(BankersState *state) { int work[NUM_RESOURCES]; bool finish[NUM_PROCESSES] = {false}; int safe_sequence[NUM_PROCESSES]; int count = 0; \/\/ Initialize work = available memcpy(work, state->available, sizeof(work)); \/\/ Find a process that can finish while (count < NUM_PROCESSES) { bool found = false; for (int i = 0; i < NUM_PROCESSES; i++) { if (!finish[i]) { \/\/ Check if process i can finish bool can_finish = true; for (int j = 0; j < NUM_RESOURCES; j++) { if (state->need[i][j] > work[j]) { can_finish = false; break; } } if (can_finish) { \/\/ Process i can finish \/\/ Add its allocated resources back to work for (int j = 0; j < NUM_RESOURCES; j++) { work[j] += state->allocation[i][j]; } safe_sequence[count++] = i; finish[i] = true; found = true; } } } if (!found) { \/\/ No process could finish - not in safe state printf(\"System is NOT in safe state!\\n\"); return false; } } \/\/ Found safe sequence printf(\"System is in SAFE state.\\nSafe sequence: \"); for (int i = 0; i < NUM_PROCESSES; i++) { printf(\"P%d \", safe_sequence[i]); } printf(\"\\n\"); return true; } \/\/ Request resources using Banker's Algorithm bool request_resources(BankersState *state, int process_id, int request[]) { printf(\"\\nProcess P%d requesting: \", process_id); for (int i = 0; i < NUM_RESOURCES; i++) { printf(\"%d \", request[i]); } printf(\"\\n\"); \/\/ Check if request <= need for (int i = 0; i < NUM_RESOURCES; i++) { if (request[i] > state->need[process_id][i]) { printf(\"Error: Process exceeded its maximum claim!\\n\"); return false; } } \/\/ Check if request <= available for (int i = 0; i < NUM_RESOURCES; i++) { if (request[i] > state->available[i]) { printf(\"Process must wait - resources not available\\n\"); return false; } } \/\/ Pretend to allocate resources BankersState test_state = *state; for (int i = 0; i < NUM_RESOURCES; i++) { test_state.available[i] -= request[i]; test_state.allocation[process_id][i] += request[i]; test_state.need[process_id][i] -= request[i]; } \/\/ Check if resulting state is safe if (is_safe_state(&test_state)) { \/\/ Safe - commit the allocation *state = test_state; printf(\"Request GRANTED\\n\"); return true; } else { \/\/ Unsafe - rollback printf(\"Request DENIED (would lead to unsafe state)\\n\"); return false; } } \/\/ Example usage int main() { BankersState state = { .available = {3, 3, 2}, .maximum = { {7, 5, 3}, \/\/ P0 {3, 2, 2}, \/\/ P1 {9, 0, 2}, \/\/ P2 {2, 2, 2}, \/\/ P3 {4, 3, 3} \/\/ P4 }, .allocation = { {0, 1, 0}, \/\/ P0 {2, 0, 0}, \/\/ P1 {3, 0, 2}, \/\/ P2 {2, 1, 1}, \/\/ P3 {0, 0, 2} \/\/ P4 } }; calculate_need(&state); printf(\"Initial state:\\n\"); printf(\"Available: \"); for (int i = 0; i < NUM_RESOURCES; i++) { printf(\"%d \", state.available[i]); } printf(\"\\n\\n\"); is_safe_state(&state); \/\/ P1 requests (1, 0, 2) int request1[] = {1, 0, 2}; request_resources(&state, 1, request1); return 0; } \/* Output Analysis: - Algorithm checks if grantin","date":"29-Nov-2024","heading":"9.3 Deadlock Avoidance","objectID":"\/posts\/operating-system\/:9:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#93-deadlock-avoidance"},{"categories":["computer-science"],"collections":null,"content":"9.3 Deadlock Avoidance Strategy: Make informed decisions about resource allocation to ensure the system never enters an unsafe state. 9.3.1 Safe State A state is safe if there exists a sequence of all processes <P1, P2, …, Pn> such that for each Pi, the resources that Pi can still request can be satisfied by: Currently available resources Resources held by all Pj where j < i 9.3.2 Banker’s Algorithm Developed by Edsger Dijkstra, simulates allocation to test for safe state. #include #include #include #define NUM_PROCESSES 5 #define NUM_RESOURCES 3 typedef struct { int available[NUM_RESOURCES]; \/\/ Available resources int maximum[NUM_PROCESSES][NUM_RESOURCES]; \/\/ Max demand int allocation[NUM_PROCESSES][NUM_RESOURCES]; \/\/ Currently allocated int need[NUM_PROCESSES][NUM_RESOURCES]; \/\/ Remaining need } BankersState; \/\/ Calculate need matrix: Need = Maximum - Allocation void calculate_need(BankersState *state) { for (int i = 0; i < NUM_PROCESSES; i++) { for (int j = 0; j < NUM_RESOURCES; j++) { state->need[i][j] = state->maximum[i][j] - state->allocation[i][j]; } } } \/\/ Check if system is in safe state bool is_safe_state(BankersState *state) { int work[NUM_RESOURCES]; bool finish[NUM_PROCESSES] = {false}; int safe_sequence[NUM_PROCESSES]; int count = 0; \/\/ Initialize work = available memcpy(work, state->available, sizeof(work)); \/\/ Find a process that can finish while (count < NUM_PROCESSES) { bool found = false; for (int i = 0; i < NUM_PROCESSES; i++) { if (!finish[i]) { \/\/ Check if process i can finish bool can_finish = true; for (int j = 0; j < NUM_RESOURCES; j++) { if (state->need[i][j] > work[j]) { can_finish = false; break; } } if (can_finish) { \/\/ Process i can finish \/\/ Add its allocated resources back to work for (int j = 0; j < NUM_RESOURCES; j++) { work[j] += state->allocation[i][j]; } safe_sequence[count++] = i; finish[i] = true; found = true; } } } if (!found) { \/\/ No process could finish - not in safe state printf(\"System is NOT in safe state!\\n\"); return false; } } \/\/ Found safe sequence printf(\"System is in SAFE state.\\nSafe sequence: \"); for (int i = 0; i < NUM_PROCESSES; i++) { printf(\"P%d \", safe_sequence[i]); } printf(\"\\n\"); return true; } \/\/ Request resources using Banker's Algorithm bool request_resources(BankersState *state, int process_id, int request[]) { printf(\"\\nProcess P%d requesting: \", process_id); for (int i = 0; i < NUM_RESOURCES; i++) { printf(\"%d \", request[i]); } printf(\"\\n\"); \/\/ Check if request <= need for (int i = 0; i < NUM_RESOURCES; i++) { if (request[i] > state->need[process_id][i]) { printf(\"Error: Process exceeded its maximum claim!\\n\"); return false; } } \/\/ Check if request <= available for (int i = 0; i < NUM_RESOURCES; i++) { if (request[i] > state->available[i]) { printf(\"Process must wait - resources not available\\n\"); return false; } } \/\/ Pretend to allocate resources BankersState test_state = *state; for (int i = 0; i < NUM_RESOURCES; i++) { test_state.available[i] -= request[i]; test_state.allocation[process_id][i] += request[i]; test_state.need[process_id][i] -= request[i]; } \/\/ Check if resulting state is safe if (is_safe_state(&test_state)) { \/\/ Safe - commit the allocation *state = test_state; printf(\"Request GRANTED\\n\"); return true; } else { \/\/ Unsafe - rollback printf(\"Request DENIED (would lead to unsafe state)\\n\"); return false; } } \/\/ Example usage int main() { BankersState state = { .available = {3, 3, 2}, .maximum = { {7, 5, 3}, \/\/ P0 {3, 2, 2}, \/\/ P1 {9, 0, 2}, \/\/ P2 {2, 2, 2}, \/\/ P3 {4, 3, 3} \/\/ P4 }, .allocation = { {0, 1, 0}, \/\/ P0 {2, 0, 0}, \/\/ P1 {3, 0, 2}, \/\/ P2 {2, 1, 1}, \/\/ P3 {0, 0, 2} \/\/ P4 } }; calculate_need(&state); printf(\"Initial state:\\n\"); printf(\"Available: \"); for (int i = 0; i < NUM_RESOURCES; i++) { printf(\"%d \", state.available[i]); } printf(\"\\n\\n\"); is_safe_state(&state); \/\/ P1 requests (1, 0, 2) int request1[] = {1, 0, 2}; request_resources(&state, 1, request1); return 0; } \/* Output Analysis: - Algorithm checks if grantin","date":"29-Nov-2024","heading":"9.3.1 Safe State","objectID":"\/posts\/operating-system\/:9:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#931-safe-state"},{"categories":["computer-science"],"collections":null,"content":"9.3 Deadlock Avoidance Strategy: Make informed decisions about resource allocation to ensure the system never enters an unsafe state. 9.3.1 Safe State A state is safe if there exists a sequence of all processes <P1, P2, …, Pn> such that for each Pi, the resources that Pi can still request can be satisfied by: Currently available resources Resources held by all Pj where j < i 9.3.2 Banker’s Algorithm Developed by Edsger Dijkstra, simulates allocation to test for safe state. #include #include #include #define NUM_PROCESSES 5 #define NUM_RESOURCES 3 typedef struct { int available[NUM_RESOURCES]; \/\/ Available resources int maximum[NUM_PROCESSES][NUM_RESOURCES]; \/\/ Max demand int allocation[NUM_PROCESSES][NUM_RESOURCES]; \/\/ Currently allocated int need[NUM_PROCESSES][NUM_RESOURCES]; \/\/ Remaining need } BankersState; \/\/ Calculate need matrix: Need = Maximum - Allocation void calculate_need(BankersState *state) { for (int i = 0; i < NUM_PROCESSES; i++) { for (int j = 0; j < NUM_RESOURCES; j++) { state->need[i][j] = state->maximum[i][j] - state->allocation[i][j]; } } } \/\/ Check if system is in safe state bool is_safe_state(BankersState *state) { int work[NUM_RESOURCES]; bool finish[NUM_PROCESSES] = {false}; int safe_sequence[NUM_PROCESSES]; int count = 0; \/\/ Initialize work = available memcpy(work, state->available, sizeof(work)); \/\/ Find a process that can finish while (count < NUM_PROCESSES) { bool found = false; for (int i = 0; i < NUM_PROCESSES; i++) { if (!finish[i]) { \/\/ Check if process i can finish bool can_finish = true; for (int j = 0; j < NUM_RESOURCES; j++) { if (state->need[i][j] > work[j]) { can_finish = false; break; } } if (can_finish) { \/\/ Process i can finish \/\/ Add its allocated resources back to work for (int j = 0; j < NUM_RESOURCES; j++) { work[j] += state->allocation[i][j]; } safe_sequence[count++] = i; finish[i] = true; found = true; } } } if (!found) { \/\/ No process could finish - not in safe state printf(\"System is NOT in safe state!\\n\"); return false; } } \/\/ Found safe sequence printf(\"System is in SAFE state.\\nSafe sequence: \"); for (int i = 0; i < NUM_PROCESSES; i++) { printf(\"P%d \", safe_sequence[i]); } printf(\"\\n\"); return true; } \/\/ Request resources using Banker's Algorithm bool request_resources(BankersState *state, int process_id, int request[]) { printf(\"\\nProcess P%d requesting: \", process_id); for (int i = 0; i < NUM_RESOURCES; i++) { printf(\"%d \", request[i]); } printf(\"\\n\"); \/\/ Check if request <= need for (int i = 0; i < NUM_RESOURCES; i++) { if (request[i] > state->need[process_id][i]) { printf(\"Error: Process exceeded its maximum claim!\\n\"); return false; } } \/\/ Check if request <= available for (int i = 0; i < NUM_RESOURCES; i++) { if (request[i] > state->available[i]) { printf(\"Process must wait - resources not available\\n\"); return false; } } \/\/ Pretend to allocate resources BankersState test_state = *state; for (int i = 0; i < NUM_RESOURCES; i++) { test_state.available[i] -= request[i]; test_state.allocation[process_id][i] += request[i]; test_state.need[process_id][i] -= request[i]; } \/\/ Check if resulting state is safe if (is_safe_state(&test_state)) { \/\/ Safe - commit the allocation *state = test_state; printf(\"Request GRANTED\\n\"); return true; } else { \/\/ Unsafe - rollback printf(\"Request DENIED (would lead to unsafe state)\\n\"); return false; } } \/\/ Example usage int main() { BankersState state = { .available = {3, 3, 2}, .maximum = { {7, 5, 3}, \/\/ P0 {3, 2, 2}, \/\/ P1 {9, 0, 2}, \/\/ P2 {2, 2, 2}, \/\/ P3 {4, 3, 3} \/\/ P4 }, .allocation = { {0, 1, 0}, \/\/ P0 {2, 0, 0}, \/\/ P1 {3, 0, 2}, \/\/ P2 {2, 1, 1}, \/\/ P3 {0, 0, 2} \/\/ P4 } }; calculate_need(&state); printf(\"Initial state:\\n\"); printf(\"Available: \"); for (int i = 0; i < NUM_RESOURCES; i++) { printf(\"%d \", state.available[i]); } printf(\"\\n\\n\"); is_safe_state(&state); \/\/ P1 requests (1, 0, 2) int request1[] = {1, 0, 2}; request_resources(&state, 1, request1); return 0; } \/* Output Analysis: - Algorithm checks if grantin","date":"29-Nov-2024","heading":"9.3.2 Banker’s Algorithm","objectID":"\/posts\/operating-system\/:9:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#932-bankers-algorithm"},{"categories":["computer-science"],"collections":null,"content":"9.4 Deadlock Detection Strategy: Allow deadlocks to occur, detect them, and recover. 9.4.1 Resource Allocation Graph (RAG) #include #include #define MAX_PROCESSES 10 #define MAX_RESOURCES 10 typedef struct { int num_processes; int num_resources; \/\/ Adjacency matrix: graph[i][j] = 1 if edge from i to j int graph[MAX_PROCESSES + MAX_RESOURCES][MAX_PROCESSES + MAX_RESOURCES]; } RAG; \/\/ Detect cycle using DFS bool has_cycle_util(RAG *rag, int v, bool visited[], bool rec_stack[]) { if (!visited[v]) { visited[v] = true; rec_stack[v] = true; int total_nodes = rag->num_processes + rag->num_resources; for (int i = 0; i < total_nodes; i++) { if (rag->graph[v][i]) { if (!visited[i] && has_cycle_util(rag, i, visited, rec_stack)) { return true; } else if (rec_stack[i]) { return true; } } } } rec_stack[v] = false; return false; } bool detect_deadlock(RAG *rag) { int total_nodes = rag->num_processes + rag->num_resources; bool visited[MAX_PROCESSES + MAX_RESOURCES] = {false}; bool rec_stack[MAX_PROCESSES + MAX_RESOURCES] = {false}; for (int i = 0; i < total_nodes; i++) { if (has_cycle_util(rag, i, visited, rec_stack)) { printf(\"DEADLOCK DETECTED! Cycle found in RAG\\n\"); return true; } } printf(\"No deadlock detected\\n\"); return false; } 9.4.2 Wait-For Graph (for single-instance resources) \/* * Wait-For Graph: Variant of RAG for single-instance resources * Process Pi waits for Pj if Pi is waiting for a resource held by Pj * Deadlock exists iff WFG contains a cycle *\/ typedef struct { int num_processes; bool wait_for[MAX_PROCESSES][MAX_PROCESSES]; \/\/ wait_for[i][j] = true if Pi waits for Pj } WaitForGraph; \/\/ Detect cycle in wait-for graph bool detect_deadlock_wfg(WaitForGraph *wfg) { for (int start = 0; start < wfg->num_processes; start++) { bool visited[MAX_PROCESSES] = {false}; int current = start; while (true) { if (visited[current]) { if (current == start) { printf(\"Deadlock: Cycle detected starting at P%d\\n\", start); return true; } break; } visited[current] = true; \/\/ Find next process in potential cycle bool found_next = false; for (int next = 0; next < wfg->num_processes; next++) { if (wfg->wait_for[current][next]) { current = next; found_next = true; break; } } if (!found_next) break; } } return false; } ","date":"29-Nov-2024","heading":"9.4 Deadlock Detection","objectID":"\/posts\/operating-system\/:9:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#94-deadlock-detection"},{"categories":["computer-science"],"collections":null,"content":"9.4 Deadlock Detection Strategy: Allow deadlocks to occur, detect them, and recover. 9.4.1 Resource Allocation Graph (RAG) #include #include #define MAX_PROCESSES 10 #define MAX_RESOURCES 10 typedef struct { int num_processes; int num_resources; \/\/ Adjacency matrix: graph[i][j] = 1 if edge from i to j int graph[MAX_PROCESSES + MAX_RESOURCES][MAX_PROCESSES + MAX_RESOURCES]; } RAG; \/\/ Detect cycle using DFS bool has_cycle_util(RAG *rag, int v, bool visited[], bool rec_stack[]) { if (!visited[v]) { visited[v] = true; rec_stack[v] = true; int total_nodes = rag->num_processes + rag->num_resources; for (int i = 0; i < total_nodes; i++) { if (rag->graph[v][i]) { if (!visited[i] && has_cycle_util(rag, i, visited, rec_stack)) { return true; } else if (rec_stack[i]) { return true; } } } } rec_stack[v] = false; return false; } bool detect_deadlock(RAG *rag) { int total_nodes = rag->num_processes + rag->num_resources; bool visited[MAX_PROCESSES + MAX_RESOURCES] = {false}; bool rec_stack[MAX_PROCESSES + MAX_RESOURCES] = {false}; for (int i = 0; i < total_nodes; i++) { if (has_cycle_util(rag, i, visited, rec_stack)) { printf(\"DEADLOCK DETECTED! Cycle found in RAG\\n\"); return true; } } printf(\"No deadlock detected\\n\"); return false; } 9.4.2 Wait-For Graph (for single-instance resources) \/* * Wait-For Graph: Variant of RAG for single-instance resources * Process Pi waits for Pj if Pi is waiting for a resource held by Pj * Deadlock exists iff WFG contains a cycle *\/ typedef struct { int num_processes; bool wait_for[MAX_PROCESSES][MAX_PROCESSES]; \/\/ wait_for[i][j] = true if Pi waits for Pj } WaitForGraph; \/\/ Detect cycle in wait-for graph bool detect_deadlock_wfg(WaitForGraph *wfg) { for (int start = 0; start < wfg->num_processes; start++) { bool visited[MAX_PROCESSES] = {false}; int current = start; while (true) { if (visited[current]) { if (current == start) { printf(\"Deadlock: Cycle detected starting at P%d\\n\", start); return true; } break; } visited[current] = true; \/\/ Find next process in potential cycle bool found_next = false; for (int next = 0; next < wfg->num_processes; next++) { if (wfg->wait_for[current][next]) { current = next; found_next = true; break; } } if (!found_next) break; } } return false; } ","date":"29-Nov-2024","heading":"9.4.1 Resource Allocation Graph (RAG)","objectID":"\/posts\/operating-system\/:9:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#941-resource-allocation-graph-rag"},{"categories":["computer-science"],"collections":null,"content":"9.4 Deadlock Detection Strategy: Allow deadlocks to occur, detect them, and recover. 9.4.1 Resource Allocation Graph (RAG) #include #include #define MAX_PROCESSES 10 #define MAX_RESOURCES 10 typedef struct { int num_processes; int num_resources; \/\/ Adjacency matrix: graph[i][j] = 1 if edge from i to j int graph[MAX_PROCESSES + MAX_RESOURCES][MAX_PROCESSES + MAX_RESOURCES]; } RAG; \/\/ Detect cycle using DFS bool has_cycle_util(RAG *rag, int v, bool visited[], bool rec_stack[]) { if (!visited[v]) { visited[v] = true; rec_stack[v] = true; int total_nodes = rag->num_processes + rag->num_resources; for (int i = 0; i < total_nodes; i++) { if (rag->graph[v][i]) { if (!visited[i] && has_cycle_util(rag, i, visited, rec_stack)) { return true; } else if (rec_stack[i]) { return true; } } } } rec_stack[v] = false; return false; } bool detect_deadlock(RAG *rag) { int total_nodes = rag->num_processes + rag->num_resources; bool visited[MAX_PROCESSES + MAX_RESOURCES] = {false}; bool rec_stack[MAX_PROCESSES + MAX_RESOURCES] = {false}; for (int i = 0; i < total_nodes; i++) { if (has_cycle_util(rag, i, visited, rec_stack)) { printf(\"DEADLOCK DETECTED! Cycle found in RAG\\n\"); return true; } } printf(\"No deadlock detected\\n\"); return false; } 9.4.2 Wait-For Graph (for single-instance resources) \/* * Wait-For Graph: Variant of RAG for single-instance resources * Process Pi waits for Pj if Pi is waiting for a resource held by Pj * Deadlock exists iff WFG contains a cycle *\/ typedef struct { int num_processes; bool wait_for[MAX_PROCESSES][MAX_PROCESSES]; \/\/ wait_for[i][j] = true if Pi waits for Pj } WaitForGraph; \/\/ Detect cycle in wait-for graph bool detect_deadlock_wfg(WaitForGraph *wfg) { for (int start = 0; start < wfg->num_processes; start++) { bool visited[MAX_PROCESSES] = {false}; int current = start; while (true) { if (visited[current]) { if (current == start) { printf(\"Deadlock: Cycle detected starting at P%d\\n\", start); return true; } break; } visited[current] = true; \/\/ Find next process in potential cycle bool found_next = false; for (int next = 0; next < wfg->num_processes; next++) { if (wfg->wait_for[current][next]) { current = next; found_next = true; break; } } if (!found_next) break; } } return false; } ","date":"29-Nov-2024","heading":"9.4.2 Wait-For Graph (for single-instance resources)","objectID":"\/posts\/operating-system\/:9:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#942-wait-for-graph-for-single-instance-resources"},{"categories":["computer-science"],"collections":null,"content":"9.5 Deadlock Recovery Once deadlock is detected, recovery strategies include: 9.5.1 Process Termination \/* * Recovery Strategy 1: Abort processes to break cycle *\/ \/\/ Abort all deadlocked processes void abort_all_deadlocked() { \/\/ Terminate all processes in deadlock \/\/ Restart them from beginning \/\/ Simple but expensive } \/\/ Abort processes one at a time void abort_one_at_a_time(int *deadlocked_processes, int count) { \/\/ Selection criteria (choose minimum cost): \/\/ 1. Process priority \/\/ 2. How long process has computed \/\/ 3. Resources used \/\/ 4. Resources needed to complete \/\/ 5. Number of processes that need to be terminated int min_cost_process = select_min_cost_process(deadlocked_processes, count); terminate_process(min_cost_process); \/\/ Check if deadlock still exists if (detect_deadlock()) { abort_one_at_a_time(deadlocked_processes, count - 1); } } 9.5.2 Resource Preemption \/* * Recovery Strategy 2: Preempt resources from processes *\/ typedef struct { int process_id; int resources_held[MAX_RESOURCES]; int execution_time; int priority; } ProcessInfo; \/\/ Select victim process for preemption int select_victim(ProcessInfo *processes, int count) { int victim = 0; int min_cost = INT_MAX; for (int i = 0; i < count; i++) { \/\/ Cost factors: \/\/ - Number of resources held \/\/ - Execution time (prefer younger processes) \/\/ - Priority (prefer low priority) int resource_count = 0; for (int j = 0; j < MAX_RESOURCES; j++) { resource_count += processes[i].resources_held[j]; } int cost = resource_count * 10 + processes[i].execution_time - processes[i].priority * 5; if (cost < min_cost) { min_cost = cost; victim = i; } } return victim; } \/\/ Rollback victim to safe state void rollback_and_preempt(int victim_id) { \/\/ 1. Save state (checkpoint) save_checkpoint(victim_id); \/\/ 2. Preempt resources release_all_resources(victim_id); \/\/ 3. Rollback to previous checkpoint restore_checkpoint(victim_id); \/\/ To avoid starvation: use aging or limits on preemption count } ","date":"29-Nov-2024","heading":"9.5 Deadlock Recovery","objectID":"\/posts\/operating-system\/:9:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#95-deadlock-recovery"},{"categories":["computer-science"],"collections":null,"content":"9.5 Deadlock Recovery Once deadlock is detected, recovery strategies include: 9.5.1 Process Termination \/* * Recovery Strategy 1: Abort processes to break cycle *\/ \/\/ Abort all deadlocked processes void abort_all_deadlocked() { \/\/ Terminate all processes in deadlock \/\/ Restart them from beginning \/\/ Simple but expensive } \/\/ Abort processes one at a time void abort_one_at_a_time(int *deadlocked_processes, int count) { \/\/ Selection criteria (choose minimum cost): \/\/ 1. Process priority \/\/ 2. How long process has computed \/\/ 3. Resources used \/\/ 4. Resources needed to complete \/\/ 5. Number of processes that need to be terminated int min_cost_process = select_min_cost_process(deadlocked_processes, count); terminate_process(min_cost_process); \/\/ Check if deadlock still exists if (detect_deadlock()) { abort_one_at_a_time(deadlocked_processes, count - 1); } } 9.5.2 Resource Preemption \/* * Recovery Strategy 2: Preempt resources from processes *\/ typedef struct { int process_id; int resources_held[MAX_RESOURCES]; int execution_time; int priority; } ProcessInfo; \/\/ Select victim process for preemption int select_victim(ProcessInfo *processes, int count) { int victim = 0; int min_cost = INT_MAX; for (int i = 0; i < count; i++) { \/\/ Cost factors: \/\/ - Number of resources held \/\/ - Execution time (prefer younger processes) \/\/ - Priority (prefer low priority) int resource_count = 0; for (int j = 0; j < MAX_RESOURCES; j++) { resource_count += processes[i].resources_held[j]; } int cost = resource_count * 10 + processes[i].execution_time - processes[i].priority * 5; if (cost < min_cost) { min_cost = cost; victim = i; } } return victim; } \/\/ Rollback victim to safe state void rollback_and_preempt(int victim_id) { \/\/ 1. Save state (checkpoint) save_checkpoint(victim_id); \/\/ 2. Preempt resources release_all_resources(victim_id); \/\/ 3. Rollback to previous checkpoint restore_checkpoint(victim_id); \/\/ To avoid starvation: use aging or limits on preemption count } ","date":"29-Nov-2024","heading":"9.5.1 Process Termination","objectID":"\/posts\/operating-system\/:9:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#951-process-termination"},{"categories":["computer-science"],"collections":null,"content":"9.5 Deadlock Recovery Once deadlock is detected, recovery strategies include: 9.5.1 Process Termination \/* * Recovery Strategy 1: Abort processes to break cycle *\/ \/\/ Abort all deadlocked processes void abort_all_deadlocked() { \/\/ Terminate all processes in deadlock \/\/ Restart them from beginning \/\/ Simple but expensive } \/\/ Abort processes one at a time void abort_one_at_a_time(int *deadlocked_processes, int count) { \/\/ Selection criteria (choose minimum cost): \/\/ 1. Process priority \/\/ 2. How long process has computed \/\/ 3. Resources used \/\/ 4. Resources needed to complete \/\/ 5. Number of processes that need to be terminated int min_cost_process = select_min_cost_process(deadlocked_processes, count); terminate_process(min_cost_process); \/\/ Check if deadlock still exists if (detect_deadlock()) { abort_one_at_a_time(deadlocked_processes, count - 1); } } 9.5.2 Resource Preemption \/* * Recovery Strategy 2: Preempt resources from processes *\/ typedef struct { int process_id; int resources_held[MAX_RESOURCES]; int execution_time; int priority; } ProcessInfo; \/\/ Select victim process for preemption int select_victim(ProcessInfo *processes, int count) { int victim = 0; int min_cost = INT_MAX; for (int i = 0; i < count; i++) { \/\/ Cost factors: \/\/ - Number of resources held \/\/ - Execution time (prefer younger processes) \/\/ - Priority (prefer low priority) int resource_count = 0; for (int j = 0; j < MAX_RESOURCES; j++) { resource_count += processes[i].resources_held[j]; } int cost = resource_count * 10 + processes[i].execution_time - processes[i].priority * 5; if (cost < min_cost) { min_cost = cost; victim = i; } } return victim; } \/\/ Rollback victim to safe state void rollback_and_preempt(int victim_id) { \/\/ 1. Save state (checkpoint) save_checkpoint(victim_id); \/\/ 2. Preempt resources release_all_resources(victim_id); \/\/ 3. Rollback to previous checkpoint restore_checkpoint(victim_id); \/\/ To avoid starvation: use aging or limits on preemption count } ","date":"29-Nov-2024","heading":"9.5.2 Resource Preemption","objectID":"\/posts\/operating-system\/:9:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#952-resource-preemption"},{"categories":["computer-science"],"collections":null,"content":"9.6 Practical Deadlock Handling in Modern Systems Linux Approach \/* * Linux uses multiple strategies: * 1. Timeouts on lock acquisition (trylock with timeout) * 2. Lock ordering conventions * 3. Lockdep: Runtime deadlock detector (detects potential deadlocks) *\/ #include #include #include \/\/ Trylock with timeout bool acquire_with_timeout(pthread_mutex_t *lock, int timeout_ms) { struct timespec timeout; clock_gettime(CLOCK_REALTIME, &timeout); timeout.tv_sec += timeout_ms \/ 1000; timeout.tv_nsec += (timeout_ms % 1000) * 1000000; if (timeout.tv_nsec >= 1000000000) { timeout.tv_sec++; timeout.tv_nsec -= 1000000000; } int result = pthread_mutex_timedlock(lock, &timeout); if (result == ETIMEDOUT) { printf(\"Lock acquisition timeout - potential deadlock\\n\"); return false; } return (result == 0); } Database Systems \/* * Database Deadlock Handling: * - Wait-Die: Older transaction waits, younger dies * - Wound-Wait: Older transaction wounds (preempts) younger *\/ typedef struct { int transaction_id; long timestamp; \/\/ Transaction start time } Transaction; \/\/ Wait-Die Scheme void wait_die(Transaction *requester, Transaction *holder) { if (requester->timestamp < holder->timestamp) { \/\/ Older transaction waits wait_for_resource(requester); } else { \/\/ Younger transaction dies (aborts) abort_transaction(requester); restart_transaction(requester); } } \/\/ Wound-Wait Scheme void wound_wait(Transaction *requester, Transaction *holder) { if (requester->timestamp < holder->timestamp) { \/\/ Older wounds younger (preempts) abort_transaction(holder); grant_resource(requester); } else { \/\/ Younger waits wait_for_resource(requester); } } ","date":"29-Nov-2024","heading":"9.6 Practical Deadlock Handling in Modern Systems","objectID":"\/posts\/operating-system\/:9:6","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#96-practical-deadlock-handling-in-modern-systems"},{"categories":["computer-science"],"collections":null,"content":"9.6 Practical Deadlock Handling in Modern Systems Linux Approach \/* * Linux uses multiple strategies: * 1. Timeouts on lock acquisition (trylock with timeout) * 2. Lock ordering conventions * 3. Lockdep: Runtime deadlock detector (detects potential deadlocks) *\/ #include #include #include \/\/ Trylock with timeout bool acquire_with_timeout(pthread_mutex_t *lock, int timeout_ms) { struct timespec timeout; clock_gettime(CLOCK_REALTIME, &timeout); timeout.tv_sec += timeout_ms \/ 1000; timeout.tv_nsec += (timeout_ms % 1000) * 1000000; if (timeout.tv_nsec >= 1000000000) { timeout.tv_sec++; timeout.tv_nsec -= 1000000000; } int result = pthread_mutex_timedlock(lock, &timeout); if (result == ETIMEDOUT) { printf(\"Lock acquisition timeout - potential deadlock\\n\"); return false; } return (result == 0); } Database Systems \/* * Database Deadlock Handling: * - Wait-Die: Older transaction waits, younger dies * - Wound-Wait: Older transaction wounds (preempts) younger *\/ typedef struct { int transaction_id; long timestamp; \/\/ Transaction start time } Transaction; \/\/ Wait-Die Scheme void wait_die(Transaction *requester, Transaction *holder) { if (requester->timestamp < holder->timestamp) { \/\/ Older transaction waits wait_for_resource(requester); } else { \/\/ Younger transaction dies (aborts) abort_transaction(requester); restart_transaction(requester); } } \/\/ Wound-Wait Scheme void wound_wait(Transaction *requester, Transaction *holder) { if (requester->timestamp < holder->timestamp) { \/\/ Older wounds younger (preempts) abort_transaction(holder); grant_resource(requester); } else { \/\/ Younger waits wait_for_resource(requester); } } ","date":"29-Nov-2024","heading":"Linux Approach","objectID":"\/posts\/operating-system\/:9:6","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#linux-approach"},{"categories":["computer-science"],"collections":null,"content":"9.6 Practical Deadlock Handling in Modern Systems Linux Approach \/* * Linux uses multiple strategies: * 1. Timeouts on lock acquisition (trylock with timeout) * 2. Lock ordering conventions * 3. Lockdep: Runtime deadlock detector (detects potential deadlocks) *\/ #include #include #include \/\/ Trylock with timeout bool acquire_with_timeout(pthread_mutex_t *lock, int timeout_ms) { struct timespec timeout; clock_gettime(CLOCK_REALTIME, &timeout); timeout.tv_sec += timeout_ms \/ 1000; timeout.tv_nsec += (timeout_ms % 1000) * 1000000; if (timeout.tv_nsec >= 1000000000) { timeout.tv_sec++; timeout.tv_nsec -= 1000000000; } int result = pthread_mutex_timedlock(lock, &timeout); if (result == ETIMEDOUT) { printf(\"Lock acquisition timeout - potential deadlock\\n\"); return false; } return (result == 0); } Database Systems \/* * Database Deadlock Handling: * - Wait-Die: Older transaction waits, younger dies * - Wound-Wait: Older transaction wounds (preempts) younger *\/ typedef struct { int transaction_id; long timestamp; \/\/ Transaction start time } Transaction; \/\/ Wait-Die Scheme void wait_die(Transaction *requester, Transaction *holder) { if (requester->timestamp < holder->timestamp) { \/\/ Older transaction waits wait_for_resource(requester); } else { \/\/ Younger transaction dies (aborts) abort_transaction(requester); restart_transaction(requester); } } \/\/ Wound-Wait Scheme void wound_wait(Transaction *requester, Transaction *holder) { if (requester->timestamp < holder->timestamp) { \/\/ Older wounds younger (preempts) abort_transaction(holder); grant_resource(requester); } else { \/\/ Younger waits wait_for_resource(requester); } } ","date":"29-Nov-2024","heading":"Database Systems","objectID":"\/posts\/operating-system\/:9:6","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#database-systems"},{"categories":["computer-science"],"collections":null,"content":"9.7 Comparison of Deadlock Handling Methods Method Prevention Avoidance Detection + Recovery Approach Prevent one Coffman condition Don’t enter unsafe state Allow deadlock, then fix Resource Utilization Low (conservative) Medium High (optimistic) Runtime Overhead Low Medium (safety check) Low (periodic detection) Implementation Easiest Complex (Banker’s) Medium Used In Simple systems Real-time systems Databases, OSes ","date":"29-Nov-2024","heading":"9.7 Comparison of Deadlock Handling Methods","objectID":"\/posts\/operating-system\/:9:7","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#97-comparison-of-deadlock-handling-methods"},{"categories":["computer-science"],"collections":null,"content":"Key Takeaways Deadlock requires all 4 Coffman conditions simultaneously Prevention: Break at least one condition (most common: resource ordering) Avoidance: Use Banker’s algorithm to stay in safe state Detection: Use RAG or WFG to find cycles Recovery: Terminate processes or preempt resources Real systems use hybrid approaches: Prevention + detection with timeouts ","date":"29-Nov-2024","heading":"Key Takeaways","objectID":"\/posts\/operating-system\/:9:8","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#key-takeaways"},{"categories":["computer-science"],"collections":null,"content":"10. Memory Management ","date":"29-Nov-2024","heading":"10. Memory Management","objectID":"\/posts\/operating-system\/:10:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#memory-management"},{"categories":["computer-science"],"collections":null,"content":"10.1 Memory Management Basics Memory management is responsible for: Keeping track of which parts of memory are in use Allocating and deallocating memory for processes Managing the swap between main memory and disk Protecting memory space between processes Memory Hierarchy Registers (CPU) <-- Fastest, smallest, most expensive \u2193 Cache (L1, L2, L3) \u2193 Main Memory (RAM) \u2193 Secondary Storage (SSD) \u2193 Tertiary Storage (HDD) <-- Slowest, largest, cheapest ","date":"29-Nov-2024","heading":"10.1 Memory Management Basics","objectID":"\/posts\/operating-system\/:10:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#101-memory-management-basics"},{"categories":["computer-science"],"collections":null,"content":"10.1 Memory Management Basics Memory management is responsible for: Keeping track of which parts of memory are in use Allocating and deallocating memory for processes Managing the swap between main memory and disk Protecting memory space between processes Memory Hierarchy Registers (CPU) <-- Fastest, smallest, most expensive \u2193 Cache (L1, L2, L3) \u2193 Main Memory (RAM) \u2193 Secondary Storage (SSD) \u2193 Tertiary Storage (HDD) <-- Slowest, largest, cheapest ","date":"29-Nov-2024","heading":"Memory Hierarchy","objectID":"\/posts\/operating-system\/:10:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#memory-hierarchy"},{"categories":["computer-science"],"collections":null,"content":"10.2 Address Binding Address binding is the process of mapping logical addresses to physical addresses. \/* * Types of Addresses: * 1. Symbolic address: Variable names in source code (e.g., \"x\") * 2. Relocatable address: Relative address from program start (e.g., \"14 bytes from start\") * 3. Absolute address: Physical memory address (e.g., \"0x400014\") *\/ \/\/ Example: Compile-time binding (rarely used) \/\/ Address is fixed at compile time int x = 10; \/\/ Compiler assigns absolute address 0x1000 \/\/ Run-time binding (most common in modern systems) \/\/ Address determined when process loads int *ptr = malloc(sizeof(int)); \/\/ Address assigned at runtime Binding Times Compile time: If memory location known a priori (obsolete) Load time: Addresses generated when program loaded Execution time: Addresses can change during execution (requires hardware support - MMU) ","date":"29-Nov-2024","heading":"10.2 Address Binding","objectID":"\/posts\/operating-system\/:10:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#102-address-binding"},{"categories":["computer-science"],"collections":null,"content":"10.2 Address Binding Address binding is the process of mapping logical addresses to physical addresses. \/* * Types of Addresses: * 1. Symbolic address: Variable names in source code (e.g., \"x\") * 2. Relocatable address: Relative address from program start (e.g., \"14 bytes from start\") * 3. Absolute address: Physical memory address (e.g., \"0x400014\") *\/ \/\/ Example: Compile-time binding (rarely used) \/\/ Address is fixed at compile time int x = 10; \/\/ Compiler assigns absolute address 0x1000 \/\/ Run-time binding (most common in modern systems) \/\/ Address determined when process loads int *ptr = malloc(sizeof(int)); \/\/ Address assigned at runtime Binding Times Compile time: If memory location known a priori (obsolete) Load time: Addresses generated when program loaded Execution time: Addresses can change during execution (requires hardware support - MMU) ","date":"29-Nov-2024","heading":"Binding Times","objectID":"\/posts\/operating-system\/:10:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#binding-times"},{"categories":["computer-science"],"collections":null,"content":"10.3 Logical vs Physical Address Space \/* * Logical Address (Virtual Address): * - Generated by CPU * - Used by user programs * - Address space from 0 to MAX * * Physical Address: * - Actual location in memory * - Seen by memory unit * - Mapped by MMU (Memory Management Unit) *\/ \/\/ Example: MMU Translation typedef struct { uint32_t logical_address; uint32_t physical_address; uint32_t base_register; \/\/ Relocation register uint32_t limit_register; \/\/ Size of process memory } AddressTranslation; uint32_t mmu_translate(AddressTranslation *at, uint32_t logical_addr) { \/\/ Check if address is within bounds if (logical_addr >= at->limit_register) { printf(\"ERROR: Segmentation fault - address out of bounds\\n\"); return 0xFFFFFFFF; \/\/ Error indicator } \/\/ Simple relocation: Physical = Logical + Base at->physical_address = logical_addr + at->base_register; printf(\"Logical: 0x%X \u2192 Physical: 0x%X\\n\", logical_addr, at->physical_address); return at->physical_address; } \/\/ Example usage int main() { AddressTranslation at = { .base_register = 0x10000, \/\/ Process starts at 0x10000 .limit_register = 0x5000 \/\/ Process size is 0x5000 bytes }; \/\/ Translate logical address 0x1234 mmu_translate(&at, 0x1234); \/\/ \u2192 Physical: 0x11234 \/\/ Attempt to access beyond limit mmu_translate(&at, 0x6000); \/\/ \u2192 ERROR! return 0; } ","date":"29-Nov-2024","heading":"10.3 Logical vs Physical Address Space","objectID":"\/posts\/operating-system\/:10:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#103-logical-vs-physical-address-space"},{"categories":["computer-science"],"collections":null,"content":"10.4 Contiguous Memory Allocation Fixed Partitioning (Static) #define NUM_PARTITIONS 4 typedef struct { uint32_t start_address; uint32_t size; bool is_allocated; int process_id; } Partition; Partition partitions[NUM_PARTITIONS] = { {0x00000, 0x10000, false, -1}, \/\/ 64KB {0x10000, 0x20000, false, -1}, \/\/ 128KB {0x30000, 0x30000, false, -1}, \/\/ 192KB {0x60000, 0x40000, false, -1} \/\/ 256KB }; \/* * Problems with fixed partitioning: * 1. Internal fragmentation: Unused space within partition * 2. Limited number of processes * 3. Inflexible partition sizes *\/ Dynamic Partitioning (Variable) typedef struct MemoryBlock { uint32_t start_address; uint32_t size; bool is_free; int process_id; struct MemoryBlock *next; } MemoryBlock; \/\/ Global memory list MemoryBlock *memory_list = NULL; void init_memory(uint32_t total_size) { memory_list = malloc(sizeof(MemoryBlock)); memory_list->start_address = 0; memory_list->size = total_size; memory_list->is_free = true; memory_list->process_id = -1; memory_list->next = NULL; } \/\/ Display current memory state void display_memory() { MemoryBlock *current = memory_list; printf(\"\\n=== Memory Map ===\\n\"); while (current != NULL) { printf(\"[0x%06X - 0x%06X] Size: %6d bytes | \", current->start_address, current->start_address + current->size - 1, current->size); if (current->is_free) { printf(\"FREE\\n\"); } else { printf(\"Process %d\\n\", current->process_id); } current = current->next; } printf(\"==================\\n\\n\"); } ","date":"29-Nov-2024","heading":"10.4 Contiguous Memory Allocation","objectID":"\/posts\/operating-system\/:10:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#104-contiguous-memory-allocation"},{"categories":["computer-science"],"collections":null,"content":"10.4 Contiguous Memory Allocation Fixed Partitioning (Static) #define NUM_PARTITIONS 4 typedef struct { uint32_t start_address; uint32_t size; bool is_allocated; int process_id; } Partition; Partition partitions[NUM_PARTITIONS] = { {0x00000, 0x10000, false, -1}, \/\/ 64KB {0x10000, 0x20000, false, -1}, \/\/ 128KB {0x30000, 0x30000, false, -1}, \/\/ 192KB {0x60000, 0x40000, false, -1} \/\/ 256KB }; \/* * Problems with fixed partitioning: * 1. Internal fragmentation: Unused space within partition * 2. Limited number of processes * 3. Inflexible partition sizes *\/ Dynamic Partitioning (Variable) typedef struct MemoryBlock { uint32_t start_address; uint32_t size; bool is_free; int process_id; struct MemoryBlock *next; } MemoryBlock; \/\/ Global memory list MemoryBlock *memory_list = NULL; void init_memory(uint32_t total_size) { memory_list = malloc(sizeof(MemoryBlock)); memory_list->start_address = 0; memory_list->size = total_size; memory_list->is_free = true; memory_list->process_id = -1; memory_list->next = NULL; } \/\/ Display current memory state void display_memory() { MemoryBlock *current = memory_list; printf(\"\\n=== Memory Map ===\\n\"); while (current != NULL) { printf(\"[0x%06X - 0x%06X] Size: %6d bytes | \", current->start_address, current->start_address + current->size - 1, current->size); if (current->is_free) { printf(\"FREE\\n\"); } else { printf(\"Process %d\\n\", current->process_id); } current = current->next; } printf(\"==================\\n\\n\"); } ","date":"29-Nov-2024","heading":"Fixed Partitioning (Static)","objectID":"\/posts\/operating-system\/:10:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#fixed-partitioning-static"},{"categories":["computer-science"],"collections":null,"content":"10.4 Contiguous Memory Allocation Fixed Partitioning (Static) #define NUM_PARTITIONS 4 typedef struct { uint32_t start_address; uint32_t size; bool is_allocated; int process_id; } Partition; Partition partitions[NUM_PARTITIONS] = { {0x00000, 0x10000, false, -1}, \/\/ 64KB {0x10000, 0x20000, false, -1}, \/\/ 128KB {0x30000, 0x30000, false, -1}, \/\/ 192KB {0x60000, 0x40000, false, -1} \/\/ 256KB }; \/* * Problems with fixed partitioning: * 1. Internal fragmentation: Unused space within partition * 2. Limited number of processes * 3. Inflexible partition sizes *\/ Dynamic Partitioning (Variable) typedef struct MemoryBlock { uint32_t start_address; uint32_t size; bool is_free; int process_id; struct MemoryBlock *next; } MemoryBlock; \/\/ Global memory list MemoryBlock *memory_list = NULL; void init_memory(uint32_t total_size) { memory_list = malloc(sizeof(MemoryBlock)); memory_list->start_address = 0; memory_list->size = total_size; memory_list->is_free = true; memory_list->process_id = -1; memory_list->next = NULL; } \/\/ Display current memory state void display_memory() { MemoryBlock *current = memory_list; printf(\"\\n=== Memory Map ===\\n\"); while (current != NULL) { printf(\"[0x%06X - 0x%06X] Size: %6d bytes | \", current->start_address, current->start_address + current->size - 1, current->size); if (current->is_free) { printf(\"FREE\\n\"); } else { printf(\"Process %d\\n\", current->process_id); } current = current->next; } printf(\"==================\\n\\n\"); } ","date":"29-Nov-2024","heading":"Dynamic Partitioning (Variable)","objectID":"\/posts\/operating-system\/:10:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#dynamic-partitioning-variable"},{"categories":["computer-science"],"collections":null,"content":"10.5 Memory Allocation Algorithms 10.5.1 First Fit Allocate the first hole that is large enough. \/* * First Fit: Fast, but may leave small unusable holes at beginning *\/ bool first_fit(int process_id, uint32_t required_size) { MemoryBlock *current = memory_list; while (current != NULL) { if (current->is_free && current->size >= required_size) { \/\/ Found suitable block if (current->size > required_size) { \/\/ Split the block MemoryBlock *new_block = malloc(sizeof(MemoryBlock)); new_block->start_address = current->start_address + required_size; new_block->size = current->size - required_size; new_block->is_free = true; new_block->process_id = -1; new_block->next = current->next; current->next = new_block; current->size = required_size; } current->is_free = false; current->process_id = process_id; printf(\"First Fit: Allocated %d bytes for Process %d at 0x%X\\n\", required_size, process_id, current->start_address); return true; } current = current->next; } printf(\"First Fit: Allocation failed for Process %d\\n\", process_id); return false; } 10.5.2 Best Fit Allocate the smallest hole that is large enough. \/* * Best Fit: Minimizes wasted space but slower (searches entire list) * May create many tiny unusable holes *\/ bool best_fit(int process_id, uint32_t required_size) { MemoryBlock *current = memory_list; MemoryBlock *best_block = NULL; uint32_t min_leftover = UINT32_MAX; \/\/ Search for best fitting block while (current != NULL) { if (current->is_free && current->size >= required_size) { uint32_t leftover = current->size - required_size; if (leftover < min_leftover) { min_leftover = leftover; best_block = current; } } current = current->next; } if (best_block == NULL) { printf(\"Best Fit: Allocation failed for Process %d\\n\", process_id); return false; } \/\/ Allocate the best block if (best_block->size > required_size) { MemoryBlock *new_block = malloc(sizeof(MemoryBlock)); new_block->start_address = best_block->start_address + required_size; new_block->size = best_block->size - required_size; new_block->is_free = true; new_block->process_id = -1; new_block->next = best_block->next; best_block->next = new_block; best_block->size = required_size; } best_block->is_free = false; best_block->process_id = process_id; printf(\"Best Fit: Allocated %d bytes for Process %d at 0x%X (leftover: %d)\\n\", required_size, process_id, best_block->start_address, min_leftover); return true; } 10.5.3 Worst Fit Allocate the largest hole. \/* * Worst Fit: Leaves largest leftover (may be more usable) * Slowest (searches entire list) *\/ bool worst_fit(int process_id, uint32_t required_size) { MemoryBlock *current = memory_list; MemoryBlock *worst_block = NULL; uint32_t max_size = 0; \/\/ Search for largest block while (current != NULL) { if (current->is_free && current->size >= required_size) { if (current->size > max_size) { max_size = current->size; worst_block = current; } } current = current->next; } if (worst_block == NULL) { printf(\"Worst Fit: Allocation failed for Process %d\\n\", process_id); return false; } \/\/ Allocate the worst (largest) block if (worst_block->size > required_size) { MemoryBlock *new_block = malloc(sizeof(MemoryBlock)); new_block->start_address = worst_block->start_address + required_size; new_block->size = worst_block->size - required_size; new_block->is_free = true; new_block->process_id = -1; new_block->next = worst_block->next; worst_block->next = new_block; worst_block->size = required_size; } worst_block->is_free = false; worst_block->process_id = process_id; printf(\"Worst Fit: Allocated %d bytes for Process %d at 0x%X\\n\", required_size, process_id, worst_block->start_address); return true; } 10.5.4 Next Fit Similar to First Fit, but starts search from last allocation point. \/* * Next Fit: Variant of First Fit that continues from last allocation * Distributes holes more evenly across memory *\/ MemoryBlock *last_allocated = NULL; bool next_fit(int process_id, uint32_t required_size) { if (last_allocated == NULL) { last_allocated = memory","date":"29-Nov-2024","heading":"10.5 Memory Allocation Algorithms","objectID":"\/posts\/operating-system\/:10:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#105-memory-allocation-algorithms"},{"categories":["computer-science"],"collections":null,"content":"10.5 Memory Allocation Algorithms 10.5.1 First Fit Allocate the first hole that is large enough. \/* * First Fit: Fast, but may leave small unusable holes at beginning *\/ bool first_fit(int process_id, uint32_t required_size) { MemoryBlock *current = memory_list; while (current != NULL) { if (current->is_free && current->size >= required_size) { \/\/ Found suitable block if (current->size > required_size) { \/\/ Split the block MemoryBlock *new_block = malloc(sizeof(MemoryBlock)); new_block->start_address = current->start_address + required_size; new_block->size = current->size - required_size; new_block->is_free = true; new_block->process_id = -1; new_block->next = current->next; current->next = new_block; current->size = required_size; } current->is_free = false; current->process_id = process_id; printf(\"First Fit: Allocated %d bytes for Process %d at 0x%X\\n\", required_size, process_id, current->start_address); return true; } current = current->next; } printf(\"First Fit: Allocation failed for Process %d\\n\", process_id); return false; } 10.5.2 Best Fit Allocate the smallest hole that is large enough. \/* * Best Fit: Minimizes wasted space but slower (searches entire list) * May create many tiny unusable holes *\/ bool best_fit(int process_id, uint32_t required_size) { MemoryBlock *current = memory_list; MemoryBlock *best_block = NULL; uint32_t min_leftover = UINT32_MAX; \/\/ Search for best fitting block while (current != NULL) { if (current->is_free && current->size >= required_size) { uint32_t leftover = current->size - required_size; if (leftover < min_leftover) { min_leftover = leftover; best_block = current; } } current = current->next; } if (best_block == NULL) { printf(\"Best Fit: Allocation failed for Process %d\\n\", process_id); return false; } \/\/ Allocate the best block if (best_block->size > required_size) { MemoryBlock *new_block = malloc(sizeof(MemoryBlock)); new_block->start_address = best_block->start_address + required_size; new_block->size = best_block->size - required_size; new_block->is_free = true; new_block->process_id = -1; new_block->next = best_block->next; best_block->next = new_block; best_block->size = required_size; } best_block->is_free = false; best_block->process_id = process_id; printf(\"Best Fit: Allocated %d bytes for Process %d at 0x%X (leftover: %d)\\n\", required_size, process_id, best_block->start_address, min_leftover); return true; } 10.5.3 Worst Fit Allocate the largest hole. \/* * Worst Fit: Leaves largest leftover (may be more usable) * Slowest (searches entire list) *\/ bool worst_fit(int process_id, uint32_t required_size) { MemoryBlock *current = memory_list; MemoryBlock *worst_block = NULL; uint32_t max_size = 0; \/\/ Search for largest block while (current != NULL) { if (current->is_free && current->size >= required_size) { if (current->size > max_size) { max_size = current->size; worst_block = current; } } current = current->next; } if (worst_block == NULL) { printf(\"Worst Fit: Allocation failed for Process %d\\n\", process_id); return false; } \/\/ Allocate the worst (largest) block if (worst_block->size > required_size) { MemoryBlock *new_block = malloc(sizeof(MemoryBlock)); new_block->start_address = worst_block->start_address + required_size; new_block->size = worst_block->size - required_size; new_block->is_free = true; new_block->process_id = -1; new_block->next = worst_block->next; worst_block->next = new_block; worst_block->size = required_size; } worst_block->is_free = false; worst_block->process_id = process_id; printf(\"Worst Fit: Allocated %d bytes for Process %d at 0x%X\\n\", required_size, process_id, worst_block->start_address); return true; } 10.5.4 Next Fit Similar to First Fit, but starts search from last allocation point. \/* * Next Fit: Variant of First Fit that continues from last allocation * Distributes holes more evenly across memory *\/ MemoryBlock *last_allocated = NULL; bool next_fit(int process_id, uint32_t required_size) { if (last_allocated == NULL) { last_allocated = memory","date":"29-Nov-2024","heading":"10.5.1 First Fit","objectID":"\/posts\/operating-system\/:10:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1051-first-fit"},{"categories":["computer-science"],"collections":null,"content":"10.5 Memory Allocation Algorithms 10.5.1 First Fit Allocate the first hole that is large enough. \/* * First Fit: Fast, but may leave small unusable holes at beginning *\/ bool first_fit(int process_id, uint32_t required_size) { MemoryBlock *current = memory_list; while (current != NULL) { if (current->is_free && current->size >= required_size) { \/\/ Found suitable block if (current->size > required_size) { \/\/ Split the block MemoryBlock *new_block = malloc(sizeof(MemoryBlock)); new_block->start_address = current->start_address + required_size; new_block->size = current->size - required_size; new_block->is_free = true; new_block->process_id = -1; new_block->next = current->next; current->next = new_block; current->size = required_size; } current->is_free = false; current->process_id = process_id; printf(\"First Fit: Allocated %d bytes for Process %d at 0x%X\\n\", required_size, process_id, current->start_address); return true; } current = current->next; } printf(\"First Fit: Allocation failed for Process %d\\n\", process_id); return false; } 10.5.2 Best Fit Allocate the smallest hole that is large enough. \/* * Best Fit: Minimizes wasted space but slower (searches entire list) * May create many tiny unusable holes *\/ bool best_fit(int process_id, uint32_t required_size) { MemoryBlock *current = memory_list; MemoryBlock *best_block = NULL; uint32_t min_leftover = UINT32_MAX; \/\/ Search for best fitting block while (current != NULL) { if (current->is_free && current->size >= required_size) { uint32_t leftover = current->size - required_size; if (leftover < min_leftover) { min_leftover = leftover; best_block = current; } } current = current->next; } if (best_block == NULL) { printf(\"Best Fit: Allocation failed for Process %d\\n\", process_id); return false; } \/\/ Allocate the best block if (best_block->size > required_size) { MemoryBlock *new_block = malloc(sizeof(MemoryBlock)); new_block->start_address = best_block->start_address + required_size; new_block->size = best_block->size - required_size; new_block->is_free = true; new_block->process_id = -1; new_block->next = best_block->next; best_block->next = new_block; best_block->size = required_size; } best_block->is_free = false; best_block->process_id = process_id; printf(\"Best Fit: Allocated %d bytes for Process %d at 0x%X (leftover: %d)\\n\", required_size, process_id, best_block->start_address, min_leftover); return true; } 10.5.3 Worst Fit Allocate the largest hole. \/* * Worst Fit: Leaves largest leftover (may be more usable) * Slowest (searches entire list) *\/ bool worst_fit(int process_id, uint32_t required_size) { MemoryBlock *current = memory_list; MemoryBlock *worst_block = NULL; uint32_t max_size = 0; \/\/ Search for largest block while (current != NULL) { if (current->is_free && current->size >= required_size) { if (current->size > max_size) { max_size = current->size; worst_block = current; } } current = current->next; } if (worst_block == NULL) { printf(\"Worst Fit: Allocation failed for Process %d\\n\", process_id); return false; } \/\/ Allocate the worst (largest) block if (worst_block->size > required_size) { MemoryBlock *new_block = malloc(sizeof(MemoryBlock)); new_block->start_address = worst_block->start_address + required_size; new_block->size = worst_block->size - required_size; new_block->is_free = true; new_block->process_id = -1; new_block->next = worst_block->next; worst_block->next = new_block; worst_block->size = required_size; } worst_block->is_free = false; worst_block->process_id = process_id; printf(\"Worst Fit: Allocated %d bytes for Process %d at 0x%X\\n\", required_size, process_id, worst_block->start_address); return true; } 10.5.4 Next Fit Similar to First Fit, but starts search from last allocation point. \/* * Next Fit: Variant of First Fit that continues from last allocation * Distributes holes more evenly across memory *\/ MemoryBlock *last_allocated = NULL; bool next_fit(int process_id, uint32_t required_size) { if (last_allocated == NULL) { last_allocated = memory","date":"29-Nov-2024","heading":"10.5.2 Best Fit","objectID":"\/posts\/operating-system\/:10:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1052-best-fit"},{"categories":["computer-science"],"collections":null,"content":"10.5 Memory Allocation Algorithms 10.5.1 First Fit Allocate the first hole that is large enough. \/* * First Fit: Fast, but may leave small unusable holes at beginning *\/ bool first_fit(int process_id, uint32_t required_size) { MemoryBlock *current = memory_list; while (current != NULL) { if (current->is_free && current->size >= required_size) { \/\/ Found suitable block if (current->size > required_size) { \/\/ Split the block MemoryBlock *new_block = malloc(sizeof(MemoryBlock)); new_block->start_address = current->start_address + required_size; new_block->size = current->size - required_size; new_block->is_free = true; new_block->process_id = -1; new_block->next = current->next; current->next = new_block; current->size = required_size; } current->is_free = false; current->process_id = process_id; printf(\"First Fit: Allocated %d bytes for Process %d at 0x%X\\n\", required_size, process_id, current->start_address); return true; } current = current->next; } printf(\"First Fit: Allocation failed for Process %d\\n\", process_id); return false; } 10.5.2 Best Fit Allocate the smallest hole that is large enough. \/* * Best Fit: Minimizes wasted space but slower (searches entire list) * May create many tiny unusable holes *\/ bool best_fit(int process_id, uint32_t required_size) { MemoryBlock *current = memory_list; MemoryBlock *best_block = NULL; uint32_t min_leftover = UINT32_MAX; \/\/ Search for best fitting block while (current != NULL) { if (current->is_free && current->size >= required_size) { uint32_t leftover = current->size - required_size; if (leftover < min_leftover) { min_leftover = leftover; best_block = current; } } current = current->next; } if (best_block == NULL) { printf(\"Best Fit: Allocation failed for Process %d\\n\", process_id); return false; } \/\/ Allocate the best block if (best_block->size > required_size) { MemoryBlock *new_block = malloc(sizeof(MemoryBlock)); new_block->start_address = best_block->start_address + required_size; new_block->size = best_block->size - required_size; new_block->is_free = true; new_block->process_id = -1; new_block->next = best_block->next; best_block->next = new_block; best_block->size = required_size; } best_block->is_free = false; best_block->process_id = process_id; printf(\"Best Fit: Allocated %d bytes for Process %d at 0x%X (leftover: %d)\\n\", required_size, process_id, best_block->start_address, min_leftover); return true; } 10.5.3 Worst Fit Allocate the largest hole. \/* * Worst Fit: Leaves largest leftover (may be more usable) * Slowest (searches entire list) *\/ bool worst_fit(int process_id, uint32_t required_size) { MemoryBlock *current = memory_list; MemoryBlock *worst_block = NULL; uint32_t max_size = 0; \/\/ Search for largest block while (current != NULL) { if (current->is_free && current->size >= required_size) { if (current->size > max_size) { max_size = current->size; worst_block = current; } } current = current->next; } if (worst_block == NULL) { printf(\"Worst Fit: Allocation failed for Process %d\\n\", process_id); return false; } \/\/ Allocate the worst (largest) block if (worst_block->size > required_size) { MemoryBlock *new_block = malloc(sizeof(MemoryBlock)); new_block->start_address = worst_block->start_address + required_size; new_block->size = worst_block->size - required_size; new_block->is_free = true; new_block->process_id = -1; new_block->next = worst_block->next; worst_block->next = new_block; worst_block->size = required_size; } worst_block->is_free = false; worst_block->process_id = process_id; printf(\"Worst Fit: Allocated %d bytes for Process %d at 0x%X\\n\", required_size, process_id, worst_block->start_address); return true; } 10.5.4 Next Fit Similar to First Fit, but starts search from last allocation point. \/* * Next Fit: Variant of First Fit that continues from last allocation * Distributes holes more evenly across memory *\/ MemoryBlock *last_allocated = NULL; bool next_fit(int process_id, uint32_t required_size) { if (last_allocated == NULL) { last_allocated = memory","date":"29-Nov-2024","heading":"10.5.3 Worst Fit","objectID":"\/posts\/operating-system\/:10:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1053-worst-fit"},{"categories":["computer-science"],"collections":null,"content":"10.5 Memory Allocation Algorithms 10.5.1 First Fit Allocate the first hole that is large enough. \/* * First Fit: Fast, but may leave small unusable holes at beginning *\/ bool first_fit(int process_id, uint32_t required_size) { MemoryBlock *current = memory_list; while (current != NULL) { if (current->is_free && current->size >= required_size) { \/\/ Found suitable block if (current->size > required_size) { \/\/ Split the block MemoryBlock *new_block = malloc(sizeof(MemoryBlock)); new_block->start_address = current->start_address + required_size; new_block->size = current->size - required_size; new_block->is_free = true; new_block->process_id = -1; new_block->next = current->next; current->next = new_block; current->size = required_size; } current->is_free = false; current->process_id = process_id; printf(\"First Fit: Allocated %d bytes for Process %d at 0x%X\\n\", required_size, process_id, current->start_address); return true; } current = current->next; } printf(\"First Fit: Allocation failed for Process %d\\n\", process_id); return false; } 10.5.2 Best Fit Allocate the smallest hole that is large enough. \/* * Best Fit: Minimizes wasted space but slower (searches entire list) * May create many tiny unusable holes *\/ bool best_fit(int process_id, uint32_t required_size) { MemoryBlock *current = memory_list; MemoryBlock *best_block = NULL; uint32_t min_leftover = UINT32_MAX; \/\/ Search for best fitting block while (current != NULL) { if (current->is_free && current->size >= required_size) { uint32_t leftover = current->size - required_size; if (leftover < min_leftover) { min_leftover = leftover; best_block = current; } } current = current->next; } if (best_block == NULL) { printf(\"Best Fit: Allocation failed for Process %d\\n\", process_id); return false; } \/\/ Allocate the best block if (best_block->size > required_size) { MemoryBlock *new_block = malloc(sizeof(MemoryBlock)); new_block->start_address = best_block->start_address + required_size; new_block->size = best_block->size - required_size; new_block->is_free = true; new_block->process_id = -1; new_block->next = best_block->next; best_block->next = new_block; best_block->size = required_size; } best_block->is_free = false; best_block->process_id = process_id; printf(\"Best Fit: Allocated %d bytes for Process %d at 0x%X (leftover: %d)\\n\", required_size, process_id, best_block->start_address, min_leftover); return true; } 10.5.3 Worst Fit Allocate the largest hole. \/* * Worst Fit: Leaves largest leftover (may be more usable) * Slowest (searches entire list) *\/ bool worst_fit(int process_id, uint32_t required_size) { MemoryBlock *current = memory_list; MemoryBlock *worst_block = NULL; uint32_t max_size = 0; \/\/ Search for largest block while (current != NULL) { if (current->is_free && current->size >= required_size) { if (current->size > max_size) { max_size = current->size; worst_block = current; } } current = current->next; } if (worst_block == NULL) { printf(\"Worst Fit: Allocation failed for Process %d\\n\", process_id); return false; } \/\/ Allocate the worst (largest) block if (worst_block->size > required_size) { MemoryBlock *new_block = malloc(sizeof(MemoryBlock)); new_block->start_address = worst_block->start_address + required_size; new_block->size = worst_block->size - required_size; new_block->is_free = true; new_block->process_id = -1; new_block->next = worst_block->next; worst_block->next = new_block; worst_block->size = required_size; } worst_block->is_free = false; worst_block->process_id = process_id; printf(\"Worst Fit: Allocated %d bytes for Process %d at 0x%X\\n\", required_size, process_id, worst_block->start_address); return true; } 10.5.4 Next Fit Similar to First Fit, but starts search from last allocation point. \/* * Next Fit: Variant of First Fit that continues from last allocation * Distributes holes more evenly across memory *\/ MemoryBlock *last_allocated = NULL; bool next_fit(int process_id, uint32_t required_size) { if (last_allocated == NULL) { last_allocated = memory","date":"29-Nov-2024","heading":"10.5.4 Next Fit","objectID":"\/posts\/operating-system\/:10:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1054-next-fit"},{"categories":["computer-science"],"collections":null,"content":"10.6 Fragmentation External Fragmentation Total memory space exists to satisfy request, but it’s not contiguous. \/* * Example of External Fragmentation: * * Before: * [P1: 10KB] [FREE: 5KB] [P2: 10KB] [FREE: 5KB] [P3: 10KB] * * Request: 8KB process * Problem: Total free = 10KB, but largest contiguous = 5KB * * Solution: Compaction *\/ void compact_memory() { MemoryBlock *current = memory_list; uint32_t compact_address = 0; printf(\"Starting memory compaction...\\n\"); \/\/ Move all allocated blocks to beginning while (current != NULL) { if (!current->is_free) { if (current->start_address != compact_address) { printf(\"Moving Process %d from 0x%X to 0x%X\\n\", current->process_id, current->start_address, compact_address); \/\/ In real system, copy memory content here current->start_address = compact_address; } compact_address += current->size; } current = current->next; } \/\/ Merge all free blocks at the end current = memory_list; MemoryBlock *prev = NULL; uint32_t total_free = 0; while (current != NULL) { if (current->is_free) { total_free += current->size; MemoryBlock *temp = current; current = current->next; if (prev != NULL) { prev->next = current; } free(temp); } else { prev = current; current = current->next; } } \/\/ Create single large free block if (total_free > 0) { MemoryBlock *free_block = malloc(sizeof(MemoryBlock)); free_block->start_address = compact_address; free_block->size = total_free; free_block->is_free = true; free_block->process_id = -1; free_block->next = NULL; if (prev != NULL) { prev->next = free_block; } else { memory_list = free_block; } } printf(\"Compaction complete. Free space: %d bytes at 0x%X\\n\", total_free, compact_address); } Internal Fragmentation Allocated memory is larger than requested, wasted space within allocated block. \/* * Example: Internal Fragmentation * Process requests 1 byte, system allocates minimum block of 64 bytes * Wasted: 63 bytes (internal fragmentation) *\/ #define MIN_ALLOCATION_SIZE 64 void* allocate_with_internal_frag(size_t requested_size) { size_t actual_size = requested_size; if (actual_size < MIN_ALLOCATION_SIZE) { actual_size = MIN_ALLOCATION_SIZE; printf(\"Internal Fragmentation: Requested %zu, allocated %zu, wasted %zu bytes\\n\", requested_size, actual_size, actual_size - requested_size); } return malloc(actual_size); } ","date":"29-Nov-2024","heading":"10.6 Fragmentation","objectID":"\/posts\/operating-system\/:10:6","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#106-fragmentation"},{"categories":["computer-science"],"collections":null,"content":"10.6 Fragmentation External Fragmentation Total memory space exists to satisfy request, but it’s not contiguous. \/* * Example of External Fragmentation: * * Before: * [P1: 10KB] [FREE: 5KB] [P2: 10KB] [FREE: 5KB] [P3: 10KB] * * Request: 8KB process * Problem: Total free = 10KB, but largest contiguous = 5KB * * Solution: Compaction *\/ void compact_memory() { MemoryBlock *current = memory_list; uint32_t compact_address = 0; printf(\"Starting memory compaction...\\n\"); \/\/ Move all allocated blocks to beginning while (current != NULL) { if (!current->is_free) { if (current->start_address != compact_address) { printf(\"Moving Process %d from 0x%X to 0x%X\\n\", current->process_id, current->start_address, compact_address); \/\/ In real system, copy memory content here current->start_address = compact_address; } compact_address += current->size; } current = current->next; } \/\/ Merge all free blocks at the end current = memory_list; MemoryBlock *prev = NULL; uint32_t total_free = 0; while (current != NULL) { if (current->is_free) { total_free += current->size; MemoryBlock *temp = current; current = current->next; if (prev != NULL) { prev->next = current; } free(temp); } else { prev = current; current = current->next; } } \/\/ Create single large free block if (total_free > 0) { MemoryBlock *free_block = malloc(sizeof(MemoryBlock)); free_block->start_address = compact_address; free_block->size = total_free; free_block->is_free = true; free_block->process_id = -1; free_block->next = NULL; if (prev != NULL) { prev->next = free_block; } else { memory_list = free_block; } } printf(\"Compaction complete. Free space: %d bytes at 0x%X\\n\", total_free, compact_address); } Internal Fragmentation Allocated memory is larger than requested, wasted space within allocated block. \/* * Example: Internal Fragmentation * Process requests 1 byte, system allocates minimum block of 64 bytes * Wasted: 63 bytes (internal fragmentation) *\/ #define MIN_ALLOCATION_SIZE 64 void* allocate_with_internal_frag(size_t requested_size) { size_t actual_size = requested_size; if (actual_size < MIN_ALLOCATION_SIZE) { actual_size = MIN_ALLOCATION_SIZE; printf(\"Internal Fragmentation: Requested %zu, allocated %zu, wasted %zu bytes\\n\", requested_size, actual_size, actual_size - requested_size); } return malloc(actual_size); } ","date":"29-Nov-2024","heading":"External Fragmentation","objectID":"\/posts\/operating-system\/:10:6","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#external-fragmentation"},{"categories":["computer-science"],"collections":null,"content":"10.6 Fragmentation External Fragmentation Total memory space exists to satisfy request, but it’s not contiguous. \/* * Example of External Fragmentation: * * Before: * [P1: 10KB] [FREE: 5KB] [P2: 10KB] [FREE: 5KB] [P3: 10KB] * * Request: 8KB process * Problem: Total free = 10KB, but largest contiguous = 5KB * * Solution: Compaction *\/ void compact_memory() { MemoryBlock *current = memory_list; uint32_t compact_address = 0; printf(\"Starting memory compaction...\\n\"); \/\/ Move all allocated blocks to beginning while (current != NULL) { if (!current->is_free) { if (current->start_address != compact_address) { printf(\"Moving Process %d from 0x%X to 0x%X\\n\", current->process_id, current->start_address, compact_address); \/\/ In real system, copy memory content here current->start_address = compact_address; } compact_address += current->size; } current = current->next; } \/\/ Merge all free blocks at the end current = memory_list; MemoryBlock *prev = NULL; uint32_t total_free = 0; while (current != NULL) { if (current->is_free) { total_free += current->size; MemoryBlock *temp = current; current = current->next; if (prev != NULL) { prev->next = current; } free(temp); } else { prev = current; current = current->next; } } \/\/ Create single large free block if (total_free > 0) { MemoryBlock *free_block = malloc(sizeof(MemoryBlock)); free_block->start_address = compact_address; free_block->size = total_free; free_block->is_free = true; free_block->process_id = -1; free_block->next = NULL; if (prev != NULL) { prev->next = free_block; } else { memory_list = free_block; } } printf(\"Compaction complete. Free space: %d bytes at 0x%X\\n\", total_free, compact_address); } Internal Fragmentation Allocated memory is larger than requested, wasted space within allocated block. \/* * Example: Internal Fragmentation * Process requests 1 byte, system allocates minimum block of 64 bytes * Wasted: 63 bytes (internal fragmentation) *\/ #define MIN_ALLOCATION_SIZE 64 void* allocate_with_internal_frag(size_t requested_size) { size_t actual_size = requested_size; if (actual_size < MIN_ALLOCATION_SIZE) { actual_size = MIN_ALLOCATION_SIZE; printf(\"Internal Fragmentation: Requested %zu, allocated %zu, wasted %zu bytes\\n\", requested_size, actual_size, actual_size - requested_size); } return malloc(actual_size); } ","date":"29-Nov-2024","heading":"Internal Fragmentation","objectID":"\/posts\/operating-system\/:10:6","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#internal-fragmentation"},{"categories":["computer-science"],"collections":null,"content":"10.7 Buddy System Allocator Efficient compromise between fixed and variable partitioning. #include #define MAX_ORDER 10 \/\/ 2^10 = 1024 units #define MEMORY_SIZE (1 << MAX_ORDER) typedef struct BuddyBlock { uint32_t address; int order; \/\/ Block size = 2^order bool is_free; struct BuddyBlock *next; } BuddyBlock; BuddyBlock *free_lists[MAX_ORDER + 1]; void init_buddy_system() { for (int i = 0; i <= MAX_ORDER; i++) { free_lists[i] = NULL; } \/\/ Initially one large free block free_lists[MAX_ORDER] = malloc(sizeof(BuddyBlock)); free_lists[MAX_ORDER]->address = 0; free_lists[MAX_ORDER]->order = MAX_ORDER; free_lists[MAX_ORDER]->is_free = true; free_lists[MAX_ORDER]->next = NULL; } \/\/ Calculate buddy address uint32_t get_buddy_address(uint32_t address, int order) { return address ^ (1 << order); } void* buddy_alloc(size_t size) { \/\/ Find minimum order that fits request int order = 0; while ((1 << order) < size) { order++; } if (order > MAX_ORDER) { printf(\"Buddy: Request too large\\n\"); return NULL; } \/\/ Find available block int current_order = order; while (current_order <= MAX_ORDER && free_lists[current_order] == NULL) { current_order++; } if (current_order > MAX_ORDER) { printf(\"Buddy: No memory available\\n\"); return NULL; } \/\/ Remove block from free list BuddyBlock *block = free_lists[current_order]; free_lists[current_order] = block->next; \/\/ Split block until we reach desired order while (current_order > order) { current_order--; \/\/ Create buddy block BuddyBlock *buddy = malloc(sizeof(BuddyBlock)); buddy->address = block->address + (1 << current_order); buddy->order = current_order; buddy->is_free = true; buddy->next = free_lists[current_order]; free_lists[current_order] = buddy; printf(\"Buddy: Split block at 0x%X (order %d) into 0x%X and 0x%X\\n\", block->address, current_order + 1, block->address, buddy->address); } block->order = order; block->is_free = false; printf(\"Buddy: Allocated %d bytes (2^%d) at 0x%X\\n\", 1 << order, order, block->address); return (void *)(uintptr_t)block->address; } void buddy_free(void *ptr, int order) { uint32_t address = (uint32_t)(uintptr_t)ptr; printf(\"Buddy: Freeing block at 0x%X (order %d)\\n\", address, order); \/\/ Try to coalesce with buddy while (order < MAX_ORDER) { uint32_t buddy_addr = get_buddy_address(address, order); \/\/ Search for free buddy BuddyBlock *prev = NULL; BuddyBlock *current = free_lists[order]; bool found_buddy = false; while (current != NULL) { if (current->address == buddy_addr && current->is_free) { \/\/ Found free buddy - coalesce if (prev != NULL) { prev->next = current->next; } else { free_lists[order] = current->next; } free(current); found_buddy = true; printf(\"Buddy: Coalescing 0x%X and 0x%X into order %d\\n\", address, buddy_addr, order + 1); \/\/ Move to parent block address = address < buddy_addr ? address : buddy_addr; order++; break; } prev = current; current = current->next; } if (!found_buddy) { break; } } \/\/ Add to free list BuddyBlock *new_block = malloc(sizeof(BuddyBlock)); new_block->address = address; new_block->order = order; new_block->is_free = true; new_block->next = free_lists[order]; free_lists[order] = new_block; } \/* * Buddy System Advantages: * - Fast allocation\/deallocation (O(log n)) * - Automatic coalescing * - Reduces external fragmentation * * Disadvantages: * - Internal fragmentation (power of 2 sizes) * - Complex implementation *\/ ","date":"29-Nov-2024","heading":"10.7 Buddy System Allocator","objectID":"\/posts\/operating-system\/:10:7","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#107-buddy-system-allocator"},{"categories":["computer-science"],"collections":null,"content":"10.8 Slab Allocator (Linux Kernel) Efficient allocator for kernel objects. \/* * Slab Allocator: Cache objects for quick reuse * Used in Linux kernel for allocating kernel structures *\/ #define SLAB_SIZE 4096 \/\/ One page #define OBJECTS_PER_SLAB 64 typedef struct SlabObject { bool is_free; void *data; struct SlabObject *next; } SlabObject; typedef struct Slab { void *memory; \/\/ Actual slab memory SlabObject *free_list; \/\/ Free objects in this slab int free_count; struct Slab *next; } Slab; typedef struct SlabCache { size_t object_size; Slab *partial_slabs; \/\/ Slabs with some free objects Slab *full_slabs; \/\/ Slabs with no free objects Slab *empty_slabs; \/\/ Slabs with all objects free void (*constructor)(void *); \/\/ Object constructor void (*destructor)(void *); \/\/ Object destructor } SlabCache; SlabCache* slab_cache_create(size_t object_size, void (*ctor)(void *), void (*dtor)(void *)) { SlabCache *cache = malloc(sizeof(SlabCache)); cache->object_size = object_size; cache->partial_slabs = NULL; cache->full_slabs = NULL; cache->empty_slabs = NULL; cache->constructor = ctor; cache->destructor = dtor; printf(\"Slab: Created cache for %zu-byte objects\\n\", object_size); return cache; } Slab* slab_create(size_t object_size) { Slab *slab = malloc(sizeof(Slab)); slab->memory = malloc(SLAB_SIZE); slab->free_count = OBJECTS_PER_SLAB; slab->next = NULL; \/\/ Initialize free list slab->free_list = NULL; for (int i = 0; i < OBJECTS_PER_SLAB; i++) { SlabObject *obj = malloc(sizeof(SlabObject)); obj->is_free = true; obj->data = (char *)slab->memory + (i * object_size); obj->next = slab->free_list; slab->free_list = obj; } printf(\"Slab: Created new slab with %d objects\\n\", OBJECTS_PER_SLAB); return slab; } void* slab_alloc(SlabCache *cache) { Slab *slab = cache->partial_slabs; \/\/ No partial slabs, try empty or create new if (slab == NULL) { slab = cache->empty_slabs; if (slab != NULL) { \/\/ Move from empty to partial cache->empty_slabs = slab->next; slab->next = cache->partial_slabs; cache->partial_slabs = slab; } else { \/\/ Create new slab slab = slab_create(cache->object_size); slab->next = cache->partial_slabs; cache->partial_slabs = slab; } } \/\/ Allocate object from slab SlabObject *obj = slab->free_list; slab->free_list = obj->next; obj->is_free = false; slab->free_count--; \/\/ Call constructor if present if (cache->constructor) { cache->constructor(obj->data); } \/\/ Move to full list if no more free objects if (slab->free_count == 0) { cache->partial_slabs = slab->next; slab->next = cache->full_slabs; cache->full_slabs = slab; } printf(\"Slab: Allocated object at %p (free in slab: %d)\\n\", obj->data, slab->free_count); return obj->data; } void slab_free(SlabCache *cache, void *ptr) { \/\/ Find which slab contains this object \/\/ (In real implementation, use more efficient lookup) \/\/ Simplified: assume we know the slab \/\/ In reality, calculate from ptr and SLAB_SIZE printf(\"Slab: Freed object at %p\\n\", ptr); \/\/ Call destructor if present if (cache->destructor) { cache->destructor(ptr); } \/\/ Return object to slab's free list \/\/ Move slab between lists as needed } \/* * Slab Allocator Benefits: * 1. No repeated initialization overhead * 2. Cache-friendly (objects close in memory) * 3. Minimal fragmentation * 4. Fast allocation (pre-allocated slabs) * * Used in Linux for: * - task_struct (process descriptors) * - file descriptors * - inode caching * - dentry caching *\/ ","date":"29-Nov-2024","heading":"10.8 Slab Allocator (Linux Kernel)","objectID":"\/posts\/operating-system\/:10:8","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#108-slab-allocator-linux-kernel"},{"categories":["computer-science"],"collections":null,"content":"10.9 Memory Allocation Comparison Algorithm Speed Fragmentation Use Case First Fit Fast Medium external General purpose Best Fit Slow High external (tiny holes) Memory constrained Worst Fit Slow Low external Large requests Next Fit Fast Medium external Balanced distribution Buddy System Medium Medium internal Kernel memory Slab Allocator Very Fast Low Kernel objects ","date":"29-Nov-2024","heading":"10.9 Memory Allocation Comparison","objectID":"\/posts\/operating-system\/:10:9","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#109-memory-allocation-comparison"},{"categories":["computer-science"],"collections":null,"content":"10.10 Practical Example: Complete Memory Manager \/\/ Complete example demonstrating multiple allocation strategies int main() { printf(\"=== Memory Management Demo ===\\n\\n\"); \/\/ Initialize memory: 1MB init_memory(0x100000); printf(\"--- First Fit Allocation ---\\n\"); first_fit(1, 0x10000); \/\/ P1: 64KB first_fit(2, 0x20000); \/\/ P2: 128KB first_fit(3, 0x15000); \/\/ P3: 84KB display_memory(); printf(\"--- Deallocation ---\\n\"); \/\/ Free P2 (creates hole in middle) MemoryBlock *current = memory_list; while (current != NULL) { if (current->process_id == 2) { current->is_free = true; current->process_id = -1; printf(\"Freed Process 2\\n\"); break; } current = current->next; } display_memory(); printf(\"--- Compaction ---\\n\"); compact_memory(); display_memory(); printf(\"--- Buddy System Demo ---\\n\"); init_buddy_system(); void *p1 = buddy_alloc(100); \/\/ Will allocate 128 (2^7) void *p2 = buddy_alloc(200); \/\/ Will allocate 256 (2^8) void *p3 = buddy_alloc(50); \/\/ Will allocate 64 (2^6) buddy_free(p2, 8); \/\/ Free and coalesce buddy_free(p1, 7); buddy_free(p3, 6); return 0; } ","date":"29-Nov-2024","heading":"10.10 Practical Example: Complete Memory Manager","objectID":"\/posts\/operating-system\/:10:10","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1010-practical-example-complete-memory-manager"},{"categories":["computer-science"],"collections":null,"content":"Key Takeaways Memory management handles allocation, deallocation, and protection First Fit is fastest, Best Fit minimizes waste, Worst Fit leaves larger holes External fragmentation: Free memory is scattered (solution: compaction) Internal fragmentation: Allocated block larger than needed Buddy System: Power-of-2 allocation with fast coalescing Slab Allocator: Pre-allocates objects for fast reuse (Linux kernel) Modern systems use hybrid approaches combining multiple strategies ","date":"29-Nov-2024","heading":"Key Takeaways","objectID":"\/posts\/operating-system\/:10:11","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#key-takeaways-1"},{"categories":["computer-science"],"collections":null,"content":"11. Virtual Memory Systems ","date":"29-Nov-2024","heading":"11. Virtual Memory Systems","objectID":"\/posts\/operating-system\/:11:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#virtual-memory"},{"categories":["computer-science"],"collections":null,"content":"11.1 Virtual Memory Overview Virtual Memory separates logical memory from physical memory, allowing: Larger address space than physical RAM Efficient use of physical memory Process isolation and protection Memory sharing between processes Simplified programming model \/* * Key Benefits: * 1. Programs larger than physical RAM can run * 2. Each process has own address space (0 to MAX) * 3. Protection: process cannot access others' memory * 4. Shared libraries occupy physical memory once *\/ \/\/ Example: 32-bit system #define VIRTUAL_ADDRESS_SPACE 0xFFFFFFFF \/\/ 4GB per process \/\/ Even if physical RAM is only 1GB! ","date":"29-Nov-2024","heading":"11.1 Virtual Memory Overview","objectID":"\/posts\/operating-system\/:11:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#111-virtual-memory-overview"},{"categories":["computer-science"],"collections":null,"content":"11.2 Demand Paging Demand paging loads pages into memory only when needed. \/* * Demand Paging Process: * 1. CPU generates virtual address * 2. MMU looks up page table * 3. If page present: translate to physical address * 4. If page not present: PAGE FAULT * a. OS handles page fault * b. Load page from disk * c. Update page table * d. Restart instruction *\/ #include #include #define PAGE_SIZE 4096 \/\/ 4KB pages #define NUM_PAGES 1024 \/\/ 1024 pages = 4MB address space #define NUM_FRAMES 256 \/\/ 256 frames = 1MB physical RAM typedef struct { bool present; \/\/ Is page in memory? uint32_t frame; \/\/ Physical frame number bool dirty; \/\/ Modified since loaded? bool accessed; \/\/ Recently accessed? (for LRU) uint32_t protection; \/\/ Read\/Write\/Execute bits } PageTableEntry; typedef struct { PageTableEntry entries[NUM_PAGES]; } PageTable; \/\/ Simulate page fault handler void handle_page_fault(PageTable *pt, uint32_t page_number) { printf(\"Page Fault! Loading page %d from disk...\\n\", page_number); \/\/ Find free frame (or evict a page) uint32_t free_frame = find_free_frame(); if (free_frame == UINT32_MAX) { \/\/ No free frames - need to evict free_frame = page_replacement_algorithm(pt); } \/\/ Load page from disk (simulated) load_page_from_disk(page_number, free_frame); \/\/ Update page table pt->entries[page_number].present = true; pt->entries[page_number].frame = free_frame; pt->entries[page_number].dirty = false; pt->entries[page_number].accessed = true; printf(\"Page %d loaded into frame %d\\n\", page_number, free_frame); } \/\/ Virtual to physical address translation uint32_t translate_address(PageTable *pt, uint32_t virtual_address) { uint32_t page_number = virtual_address \/ PAGE_SIZE; uint32_t offset = virtual_address % PAGE_SIZE; if (page_number >= NUM_PAGES) { printf(\"ERROR: Invalid virtual address 0x%X\\n\", virtual_address); return UINT32_MAX; } PageTableEntry *pte = &pt->entries[page_number]; if (!pte->present) { \/\/ Page fault! handle_page_fault(pt, page_number); } \/\/ Mark as accessed (for page replacement) pte->accessed = true; \/\/ Calculate physical address uint32_t physical_address = (pte->frame * PAGE_SIZE) + offset; printf(\"Virtual 0x%X (page %d, offset %d) \u2192 Physical 0x%X (frame %d)\\n\", virtual_address, page_number, offset, physical_address, pte->frame); return physical_address; } ","date":"29-Nov-2024","heading":"11.2 Demand Paging","objectID":"\/posts\/operating-system\/:11:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#112-demand-paging"},{"categories":["computer-science"],"collections":null,"content":"11.3 Page Replacement Algorithms When all frames are occupied, must choose a victim page to evict. 11.3.1 FIFO (First-In-First-Out) \/* * FIFO: Replace oldest page in memory * Simple but suffers from Belady's Anomaly *\/ #include #define MAX_FRAMES 3 typedef struct { uint32_t frames[MAX_FRAMES]; int front; \/\/ Oldest page index int count; } FIFOQueue; void init_fifo(FIFOQueue *q) { q->front = 0; q->count = 0; for (int i = 0; i < MAX_FRAMES; i++) { q->frames[i] = UINT32_MAX; } } bool is_page_in_memory(FIFOQueue *q, uint32_t page) { for (int i = 0; i < MAX_FRAMES; i++) { if (q->frames[i] == page) { return true; } } return false; } int fifo_replace(FIFOQueue *q, uint32_t page) { if (is_page_in_memory(q, page)) { printf(\"Page %d already in memory\\n\", page); return 0; \/\/ No fault } if (q->count < MAX_FRAMES) { \/\/ Still have free frames q->frames[q->count] = page; q->count++; printf(\"FIFO: Loaded page %d into frame %d\\n\", page, q->count - 1); } else { \/\/ Replace oldest page uint32_t victim = q->frames[q->front]; q->frames[q->front] = page; q->front = (q->front + 1) % MAX_FRAMES; printf(\"FIFO: Replaced page %d with page %d\\n\", victim, page); } return 1; \/\/ Page fault occurred } \/\/ Example: Belady's Anomaly void demonstrate_belady_anomaly() { uint32_t reference_string[] = {1, 2, 3, 4, 1, 2, 5, 1, 2, 3, 4, 5}; int length = sizeof(reference_string) \/ sizeof(reference_string[0]); FIFOQueue queue; init_fifo(&queue); int page_faults = 0; printf(\"=== FIFO Page Replacement ===\\n\"); for (int i = 0; i < length; i++) { printf(\"Accessing page %d: \", reference_string[i]); page_faults += fifo_replace(&queue, reference_string[i]); } printf(\"Total page faults: %d\\n\\n\", page_faults); } 11.3.2 LRU (Least Recently Used) \/* * LRU: Replace page that hasn't been used for longest time * Optimal but expensive to implement exactly *\/ typedef struct { uint32_t page; uint64_t last_access_time; } LRUFrame; typedef struct { LRUFrame frames[MAX_FRAMES]; int count; uint64_t current_time; } LRUCache; void init_lru(LRUCache *cache) { cache->count = 0; cache->current_time = 0; for (int i = 0; i < MAX_FRAMES; i++) { cache->frames[i].page = UINT32_MAX; cache->frames[i].last_access_time = 0; } } int find_lru_victim(LRUCache *cache) { int victim_idx = 0; uint64_t oldest_time = cache->frames[0].last_access_time; for (int i = 1; i < MAX_FRAMES; i++) { if (cache->frames[i].last_access_time < oldest_time) { oldest_time = cache->frames[i].last_access_time; victim_idx = i; } } return victim_idx; } int lru_access(LRUCache *cache, uint32_t page) { cache->current_time++; \/\/ Check if page already in memory for (int i = 0; i < cache->count; i++) { if (cache->frames[i].page == page) { cache->frames[i].last_access_time = cache->current_time; printf(\"LRU: Page %d hit (updated access time)\\n\", page); return 0; \/\/ No fault } } \/\/ Page fault if (cache->count < MAX_FRAMES) { \/\/ Free frame available cache->frames[cache->count].page = page; cache->frames[cache->count].last_access_time = cache->current_time; cache->count++; printf(\"LRU: Loaded page %d into frame %d\\n\", page, cache->count - 1); } else { \/\/ Replace LRU page int victim_idx = find_lru_victim(cache); uint32_t victim_page = cache->frames[victim_idx].page; cache->frames[victim_idx].page = page; cache->frames[victim_idx].last_access_time = cache->current_time; printf(\"LRU: Replaced page %d (LRU) with page %d\\n\", victim_page, page); } return 1; \/\/ Page fault } \/\/ Approximate LRU using Clock algorithm (Second Chance) typedef struct { uint32_t page; bool reference_bit; } ClockFrame; typedef struct { ClockFrame frames[MAX_FRAMES]; int hand; \/\/ Clock hand position int count; } ClockAlgorithm; void init_clock(ClockAlgorithm *clock) { clock->hand = 0; clock->count = 0; for (int i = 0; i < MAX_FRAMES; i++) { clock->frames[i].page = UINT32_MAX; clock->frames[i].reference_bit = false; } } int clock_replace(ClockAlgorithm *clock, uint32_t page) { \/\/ Check if already in memory for (int i = 0; i < clock->count; i++) { if (clock->frames[i].page == p","date":"29-Nov-2024","heading":"11.3 Page Replacement Algorithms","objectID":"\/posts\/operating-system\/:11:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#113-page-replacement-algorithms"},{"categories":["computer-science"],"collections":null,"content":"11.3 Page Replacement Algorithms When all frames are occupied, must choose a victim page to evict. 11.3.1 FIFO (First-In-First-Out) \/* * FIFO: Replace oldest page in memory * Simple but suffers from Belady's Anomaly *\/ #include #define MAX_FRAMES 3 typedef struct { uint32_t frames[MAX_FRAMES]; int front; \/\/ Oldest page index int count; } FIFOQueue; void init_fifo(FIFOQueue *q) { q->front = 0; q->count = 0; for (int i = 0; i < MAX_FRAMES; i++) { q->frames[i] = UINT32_MAX; } } bool is_page_in_memory(FIFOQueue *q, uint32_t page) { for (int i = 0; i < MAX_FRAMES; i++) { if (q->frames[i] == page) { return true; } } return false; } int fifo_replace(FIFOQueue *q, uint32_t page) { if (is_page_in_memory(q, page)) { printf(\"Page %d already in memory\\n\", page); return 0; \/\/ No fault } if (q->count < MAX_FRAMES) { \/\/ Still have free frames q->frames[q->count] = page; q->count++; printf(\"FIFO: Loaded page %d into frame %d\\n\", page, q->count - 1); } else { \/\/ Replace oldest page uint32_t victim = q->frames[q->front]; q->frames[q->front] = page; q->front = (q->front + 1) % MAX_FRAMES; printf(\"FIFO: Replaced page %d with page %d\\n\", victim, page); } return 1; \/\/ Page fault occurred } \/\/ Example: Belady's Anomaly void demonstrate_belady_anomaly() { uint32_t reference_string[] = {1, 2, 3, 4, 1, 2, 5, 1, 2, 3, 4, 5}; int length = sizeof(reference_string) \/ sizeof(reference_string[0]); FIFOQueue queue; init_fifo(&queue); int page_faults = 0; printf(\"=== FIFO Page Replacement ===\\n\"); for (int i = 0; i < length; i++) { printf(\"Accessing page %d: \", reference_string[i]); page_faults += fifo_replace(&queue, reference_string[i]); } printf(\"Total page faults: %d\\n\\n\", page_faults); } 11.3.2 LRU (Least Recently Used) \/* * LRU: Replace page that hasn't been used for longest time * Optimal but expensive to implement exactly *\/ typedef struct { uint32_t page; uint64_t last_access_time; } LRUFrame; typedef struct { LRUFrame frames[MAX_FRAMES]; int count; uint64_t current_time; } LRUCache; void init_lru(LRUCache *cache) { cache->count = 0; cache->current_time = 0; for (int i = 0; i < MAX_FRAMES; i++) { cache->frames[i].page = UINT32_MAX; cache->frames[i].last_access_time = 0; } } int find_lru_victim(LRUCache *cache) { int victim_idx = 0; uint64_t oldest_time = cache->frames[0].last_access_time; for (int i = 1; i < MAX_FRAMES; i++) { if (cache->frames[i].last_access_time < oldest_time) { oldest_time = cache->frames[i].last_access_time; victim_idx = i; } } return victim_idx; } int lru_access(LRUCache *cache, uint32_t page) { cache->current_time++; \/\/ Check if page already in memory for (int i = 0; i < cache->count; i++) { if (cache->frames[i].page == page) { cache->frames[i].last_access_time = cache->current_time; printf(\"LRU: Page %d hit (updated access time)\\n\", page); return 0; \/\/ No fault } } \/\/ Page fault if (cache->count < MAX_FRAMES) { \/\/ Free frame available cache->frames[cache->count].page = page; cache->frames[cache->count].last_access_time = cache->current_time; cache->count++; printf(\"LRU: Loaded page %d into frame %d\\n\", page, cache->count - 1); } else { \/\/ Replace LRU page int victim_idx = find_lru_victim(cache); uint32_t victim_page = cache->frames[victim_idx].page; cache->frames[victim_idx].page = page; cache->frames[victim_idx].last_access_time = cache->current_time; printf(\"LRU: Replaced page %d (LRU) with page %d\\n\", victim_page, page); } return 1; \/\/ Page fault } \/\/ Approximate LRU using Clock algorithm (Second Chance) typedef struct { uint32_t page; bool reference_bit; } ClockFrame; typedef struct { ClockFrame frames[MAX_FRAMES]; int hand; \/\/ Clock hand position int count; } ClockAlgorithm; void init_clock(ClockAlgorithm *clock) { clock->hand = 0; clock->count = 0; for (int i = 0; i < MAX_FRAMES; i++) { clock->frames[i].page = UINT32_MAX; clock->frames[i].reference_bit = false; } } int clock_replace(ClockAlgorithm *clock, uint32_t page) { \/\/ Check if already in memory for (int i = 0; i < clock->count; i++) { if (clock->frames[i].page == p","date":"29-Nov-2024","heading":"11.3.1 FIFO (First-In-First-Out)","objectID":"\/posts\/operating-system\/:11:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1131-fifo-first-in-first-out"},{"categories":["computer-science"],"collections":null,"content":"11.3 Page Replacement Algorithms When all frames are occupied, must choose a victim page to evict. 11.3.1 FIFO (First-In-First-Out) \/* * FIFO: Replace oldest page in memory * Simple but suffers from Belady's Anomaly *\/ #include #define MAX_FRAMES 3 typedef struct { uint32_t frames[MAX_FRAMES]; int front; \/\/ Oldest page index int count; } FIFOQueue; void init_fifo(FIFOQueue *q) { q->front = 0; q->count = 0; for (int i = 0; i < MAX_FRAMES; i++) { q->frames[i] = UINT32_MAX; } } bool is_page_in_memory(FIFOQueue *q, uint32_t page) { for (int i = 0; i < MAX_FRAMES; i++) { if (q->frames[i] == page) { return true; } } return false; } int fifo_replace(FIFOQueue *q, uint32_t page) { if (is_page_in_memory(q, page)) { printf(\"Page %d already in memory\\n\", page); return 0; \/\/ No fault } if (q->count < MAX_FRAMES) { \/\/ Still have free frames q->frames[q->count] = page; q->count++; printf(\"FIFO: Loaded page %d into frame %d\\n\", page, q->count - 1); } else { \/\/ Replace oldest page uint32_t victim = q->frames[q->front]; q->frames[q->front] = page; q->front = (q->front + 1) % MAX_FRAMES; printf(\"FIFO: Replaced page %d with page %d\\n\", victim, page); } return 1; \/\/ Page fault occurred } \/\/ Example: Belady's Anomaly void demonstrate_belady_anomaly() { uint32_t reference_string[] = {1, 2, 3, 4, 1, 2, 5, 1, 2, 3, 4, 5}; int length = sizeof(reference_string) \/ sizeof(reference_string[0]); FIFOQueue queue; init_fifo(&queue); int page_faults = 0; printf(\"=== FIFO Page Replacement ===\\n\"); for (int i = 0; i < length; i++) { printf(\"Accessing page %d: \", reference_string[i]); page_faults += fifo_replace(&queue, reference_string[i]); } printf(\"Total page faults: %d\\n\\n\", page_faults); } 11.3.2 LRU (Least Recently Used) \/* * LRU: Replace page that hasn't been used for longest time * Optimal but expensive to implement exactly *\/ typedef struct { uint32_t page; uint64_t last_access_time; } LRUFrame; typedef struct { LRUFrame frames[MAX_FRAMES]; int count; uint64_t current_time; } LRUCache; void init_lru(LRUCache *cache) { cache->count = 0; cache->current_time = 0; for (int i = 0; i < MAX_FRAMES; i++) { cache->frames[i].page = UINT32_MAX; cache->frames[i].last_access_time = 0; } } int find_lru_victim(LRUCache *cache) { int victim_idx = 0; uint64_t oldest_time = cache->frames[0].last_access_time; for (int i = 1; i < MAX_FRAMES; i++) { if (cache->frames[i].last_access_time < oldest_time) { oldest_time = cache->frames[i].last_access_time; victim_idx = i; } } return victim_idx; } int lru_access(LRUCache *cache, uint32_t page) { cache->current_time++; \/\/ Check if page already in memory for (int i = 0; i < cache->count; i++) { if (cache->frames[i].page == page) { cache->frames[i].last_access_time = cache->current_time; printf(\"LRU: Page %d hit (updated access time)\\n\", page); return 0; \/\/ No fault } } \/\/ Page fault if (cache->count < MAX_FRAMES) { \/\/ Free frame available cache->frames[cache->count].page = page; cache->frames[cache->count].last_access_time = cache->current_time; cache->count++; printf(\"LRU: Loaded page %d into frame %d\\n\", page, cache->count - 1); } else { \/\/ Replace LRU page int victim_idx = find_lru_victim(cache); uint32_t victim_page = cache->frames[victim_idx].page; cache->frames[victim_idx].page = page; cache->frames[victim_idx].last_access_time = cache->current_time; printf(\"LRU: Replaced page %d (LRU) with page %d\\n\", victim_page, page); } return 1; \/\/ Page fault } \/\/ Approximate LRU using Clock algorithm (Second Chance) typedef struct { uint32_t page; bool reference_bit; } ClockFrame; typedef struct { ClockFrame frames[MAX_FRAMES]; int hand; \/\/ Clock hand position int count; } ClockAlgorithm; void init_clock(ClockAlgorithm *clock) { clock->hand = 0; clock->count = 0; for (int i = 0; i < MAX_FRAMES; i++) { clock->frames[i].page = UINT32_MAX; clock->frames[i].reference_bit = false; } } int clock_replace(ClockAlgorithm *clock, uint32_t page) { \/\/ Check if already in memory for (int i = 0; i < clock->count; i++) { if (clock->frames[i].page == p","date":"29-Nov-2024","heading":"11.3.2 LRU (Least Recently Used)","objectID":"\/posts\/operating-system\/:11:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1132-lru-least-recently-used"},{"categories":["computer-science"],"collections":null,"content":"11.3 Page Replacement Algorithms When all frames are occupied, must choose a victim page to evict. 11.3.1 FIFO (First-In-First-Out) \/* * FIFO: Replace oldest page in memory * Simple but suffers from Belady's Anomaly *\/ #include #define MAX_FRAMES 3 typedef struct { uint32_t frames[MAX_FRAMES]; int front; \/\/ Oldest page index int count; } FIFOQueue; void init_fifo(FIFOQueue *q) { q->front = 0; q->count = 0; for (int i = 0; i < MAX_FRAMES; i++) { q->frames[i] = UINT32_MAX; } } bool is_page_in_memory(FIFOQueue *q, uint32_t page) { for (int i = 0; i < MAX_FRAMES; i++) { if (q->frames[i] == page) { return true; } } return false; } int fifo_replace(FIFOQueue *q, uint32_t page) { if (is_page_in_memory(q, page)) { printf(\"Page %d already in memory\\n\", page); return 0; \/\/ No fault } if (q->count < MAX_FRAMES) { \/\/ Still have free frames q->frames[q->count] = page; q->count++; printf(\"FIFO: Loaded page %d into frame %d\\n\", page, q->count - 1); } else { \/\/ Replace oldest page uint32_t victim = q->frames[q->front]; q->frames[q->front] = page; q->front = (q->front + 1) % MAX_FRAMES; printf(\"FIFO: Replaced page %d with page %d\\n\", victim, page); } return 1; \/\/ Page fault occurred } \/\/ Example: Belady's Anomaly void demonstrate_belady_anomaly() { uint32_t reference_string[] = {1, 2, 3, 4, 1, 2, 5, 1, 2, 3, 4, 5}; int length = sizeof(reference_string) \/ sizeof(reference_string[0]); FIFOQueue queue; init_fifo(&queue); int page_faults = 0; printf(\"=== FIFO Page Replacement ===\\n\"); for (int i = 0; i < length; i++) { printf(\"Accessing page %d: \", reference_string[i]); page_faults += fifo_replace(&queue, reference_string[i]); } printf(\"Total page faults: %d\\n\\n\", page_faults); } 11.3.2 LRU (Least Recently Used) \/* * LRU: Replace page that hasn't been used for longest time * Optimal but expensive to implement exactly *\/ typedef struct { uint32_t page; uint64_t last_access_time; } LRUFrame; typedef struct { LRUFrame frames[MAX_FRAMES]; int count; uint64_t current_time; } LRUCache; void init_lru(LRUCache *cache) { cache->count = 0; cache->current_time = 0; for (int i = 0; i < MAX_FRAMES; i++) { cache->frames[i].page = UINT32_MAX; cache->frames[i].last_access_time = 0; } } int find_lru_victim(LRUCache *cache) { int victim_idx = 0; uint64_t oldest_time = cache->frames[0].last_access_time; for (int i = 1; i < MAX_FRAMES; i++) { if (cache->frames[i].last_access_time < oldest_time) { oldest_time = cache->frames[i].last_access_time; victim_idx = i; } } return victim_idx; } int lru_access(LRUCache *cache, uint32_t page) { cache->current_time++; \/\/ Check if page already in memory for (int i = 0; i < cache->count; i++) { if (cache->frames[i].page == page) { cache->frames[i].last_access_time = cache->current_time; printf(\"LRU: Page %d hit (updated access time)\\n\", page); return 0; \/\/ No fault } } \/\/ Page fault if (cache->count < MAX_FRAMES) { \/\/ Free frame available cache->frames[cache->count].page = page; cache->frames[cache->count].last_access_time = cache->current_time; cache->count++; printf(\"LRU: Loaded page %d into frame %d\\n\", page, cache->count - 1); } else { \/\/ Replace LRU page int victim_idx = find_lru_victim(cache); uint32_t victim_page = cache->frames[victim_idx].page; cache->frames[victim_idx].page = page; cache->frames[victim_idx].last_access_time = cache->current_time; printf(\"LRU: Replaced page %d (LRU) with page %d\\n\", victim_page, page); } return 1; \/\/ Page fault } \/\/ Approximate LRU using Clock algorithm (Second Chance) typedef struct { uint32_t page; bool reference_bit; } ClockFrame; typedef struct { ClockFrame frames[MAX_FRAMES]; int hand; \/\/ Clock hand position int count; } ClockAlgorithm; void init_clock(ClockAlgorithm *clock) { clock->hand = 0; clock->count = 0; for (int i = 0; i < MAX_FRAMES; i++) { clock->frames[i].page = UINT32_MAX; clock->frames[i].reference_bit = false; } } int clock_replace(ClockAlgorithm *clock, uint32_t page) { \/\/ Check if already in memory for (int i = 0; i < clock->count; i++) { if (clock->frames[i].page == p","date":"29-Nov-2024","heading":"11.3.3 Optimal (MIN)","objectID":"\/posts\/operating-system\/:11:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1133-optimal-min"},{"categories":["computer-science"],"collections":null,"content":"11.4 Translation Lookaside Buffer (TLB) TLB is a hardware cache for page table entries. \/* * TLB (Translation Lookaside Buffer): * Fast cache for page table entries * Typically 64-128 entries, fully associative * 95%+ hit rate in practice *\/ #define TLB_SIZE 16 typedef struct { uint32_t page_number; uint32_t frame_number; bool valid; } TLBEntry; typedef struct { TLBEntry entries[TLB_SIZE]; int next_replacement; \/\/ For FIFO replacement uint64_t hits; uint64_t misses; } TLB; void init_tlb(TLB *tlb) { for (int i = 0; i < TLB_SIZE; i++) { tlb->entries[i].valid = false; } tlb->next_replacement = 0; tlb->hits = 0; tlb->misses = 0; } bool tlb_lookup(TLB *tlb, uint32_t page_number, uint32_t *frame_number) { for (int i = 0; i < TLB_SIZE; i++) { if (tlb->entries[i].valid && tlb->entries[i].page_number == page_number) { *frame_number = tlb->entries[i].frame_number; tlb->hits++; printf(\"TLB HIT: Page %d \u2192 Frame %d\\n\", page_number, *frame_number); return true; } } tlb->misses++; printf(\"TLB MISS: Page %d\\n\", page_number); return false; } void tlb_insert(TLB *tlb, uint32_t page_number, uint32_t frame_number) { \/\/ FIFO replacement in TLB tlb->entries[tlb->next_replacement].page_number = page_number; tlb->entries[tlb->next_replacement].frame_number = frame_number; tlb->entries[tlb->next_replacement].valid = true; tlb->next_replacement = (tlb->next_replacement + 1) % TLB_SIZE; printf(\"TLB INSERT: Page %d \u2192 Frame %d\\n\", page_number, frame_number); } void tlb_flush(TLB *tlb) { \/\/ Called on context switch for (int i = 0; i < TLB_SIZE; i++) { tlb->entries[i].valid = false; } printf(\"TLB FLUSHED (context switch)\\n\"); } \/\/ Complete address translation with TLB uint32_t translate_with_tlb(TLB *tlb, PageTable *pt, uint32_t virtual_addr) { uint32_t page_number = virtual_addr \/ PAGE_SIZE; uint32_t offset = virtual_addr % PAGE_SIZE; uint32_t frame_number; \/\/ Step 1: Check TLB if (tlb_lookup(tlb, page_number, &frame_number)) { \/\/ TLB hit - fast path! return (frame_number * PAGE_SIZE) + offset; } \/\/ Step 2: TLB miss - check page table if (!pt->entries[page_number].present) { \/\/ Page fault handle_page_fault(pt, page_number); } frame_number = pt->entries[page_number].frame; \/\/ Step 3: Update TLB tlb_insert(tlb, page_number, frame_number); return (frame_number * PAGE_SIZE) + offset; } void print_tlb_stats(TLB *tlb) { uint64_t total = tlb->hits + tlb->misses; double hit_rate = (double)tlb->hits \/ total * 100.0; printf(\"\\n=== TLB Statistics ===\\n\"); printf(\"Hits: %lu\\n\", tlb->hits); printf(\"Misses: %lu\\n\", tlb->misses); printf(\"Hit Rate: %.2f%%\\n\", hit_rate); } ","date":"29-Nov-2024","heading":"11.4 Translation Lookaside Buffer (TLB)","objectID":"\/posts\/operating-system\/:11:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#114-translation-lookaside-buffer-tlb"},{"categories":["computer-science"],"collections":null,"content":"11.5 Multi-Level Page Tables Reduce memory overhead of large page tables. \/* * Problem: Single-level page table for 32-bit address space * 4KB pages: 2^32 \/ 2^12 = 2^20 (1 million) entries * If each entry = 4 bytes: 4MB per process! * * Solution: Multi-level (hierarchical) page tables *\/ \/\/ Two-level page table for 32-bit address space #define PAGE_DIR_SIZE 1024 \/\/ 10 bits #define PAGE_TABLE_SIZE 1024 \/\/ 10 bits \/\/ Offset: 12 bits \/\/ Total: 10 + 10 + 12 = 32 bits typedef struct { PageTableEntry entries[PAGE_TABLE_SIZE]; } SecondLevelPageTable; typedef struct { SecondLevelPageTable *tables[PAGE_DIR_SIZE]; bool present[PAGE_DIR_SIZE]; } PageDirectory; void init_page_directory(PageDirectory *pd) { for (int i = 0; i < PAGE_DIR_SIZE; i++) { pd->tables[i] = NULL; pd->present[i] = false; } } uint32_t translate_two_level(PageDirectory *pd, uint32_t virtual_addr) { \/\/ Extract indices uint32_t dir_index = (virtual_addr >> 22) & 0x3FF; \/\/ Bits 31-22 uint32_t table_index = (virtual_addr >> 12) & 0x3FF; \/\/ Bits 21-12 uint32_t offset = virtual_addr & 0xFFF; \/\/ Bits 11-0 printf(\"Virtual 0x%08X: dir=%d, table=%d, offset=%d\\n\", virtual_addr, dir_index, table_index, offset); \/\/ Check if page table exists if (!pd->present[dir_index]) { printf(\"Page directory entry not present - allocate page table\\n\"); pd->tables[dir_index] = malloc(sizeof(SecondLevelPageTable)); pd->present[dir_index] = true; } SecondLevelPageTable *page_table = pd->tables[dir_index]; PageTableEntry *pte = &page_table->entries[table_index]; if (!pte->present) { \/\/ Page fault printf(\"Page fault at dir=%d, table=%d\\n\", dir_index, table_index); return UINT32_MAX; } uint32_t physical_addr = (pte->frame * PAGE_SIZE) + offset; return physical_addr; } \/* * Advantages of Multi-Level Page Tables: * 1. Only allocate page tables for used address space * 2. Typical process uses small fraction of address space * 3. Example: Process using 8MB of 4GB space * - Single level: 4MB page table * - Two level: ~8KB page tables * 4. Can extend to 3, 4, or more levels (x86-64 uses 4 levels) *\/ ","date":"29-Nov-2024","heading":"11.5 Multi-Level Page Tables","objectID":"\/posts\/operating-system\/:11:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#115-multi-level-page-tables"},{"categories":["computer-science"],"collections":null,"content":"11.6 Page Table Entry Structure \/* * Typical Page Table Entry (x86) *\/ typedef struct { unsigned int present : 1; \/\/ 1 = page in memory unsigned int rw : 1; \/\/ 1 = read\/write, 0 = read-only unsigned int user : 1; \/\/ 1 = user mode can access unsigned int pwt : 1; \/\/ Page write-through unsigned int pcd : 1; \/\/ Page cache disabled unsigned int accessed : 1; \/\/ Has page been accessed? unsigned int dirty : 1; \/\/ Has page been written to? unsigned int pat : 1; \/\/ Page attribute table unsigned int global : 1; \/\/ Don't flush on context switch unsigned int available : 3; \/\/ Available for OS use unsigned int frame : 20; \/\/ Physical frame number (4KB aligned) } x86_PTE; void print_pte(x86_PTE *pte) { printf(\"PTE: Frame=%d, P=%d, R\/W=%d, U\/S=%d, A=%d, D=%d\\n\", pte->frame, pte->present, pte->rw, pte->user, pte->accessed, pte->dirty); } \/\/ Check page permissions bool check_access(x86_PTE *pte, bool is_write, bool is_user_mode) { if (!pte->present) { printf(\"Page fault: Page not present\\n\"); return false; } if (is_write && !pte->rw) { printf(\"Protection fault: Write to read-only page\\n\"); return false; } if (is_user_mode && !pte->user) { printf(\"Protection fault: User mode accessing supervisor page\\n\"); return false; } return true; } ","date":"29-Nov-2024","heading":"11.6 Page Table Entry Structure","objectID":"\/posts\/operating-system\/:11:6","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#116-page-table-entry-structure"},{"categories":["computer-science"],"collections":null,"content":"11.7 Inverted Page Tables Alternative approach for large address spaces. \/* * Inverted Page Table: * One entry per physical frame (not per virtual page) * Saves space for 64-bit systems *\/ #define NUM_PHYSICAL_FRAMES 256 typedef struct { uint32_t pid; \/\/ Process ID uint32_t virtual_page; \/\/ Virtual page number bool valid; } InvertedPageTableEntry; typedef struct { InvertedPageTableEntry entries[NUM_PHYSICAL_FRAMES]; } InvertedPageTable; uint32_t inverted_translate(InvertedPageTable *ipt, uint32_t pid, uint32_t virtual_addr) { uint32_t virtual_page = virtual_addr \/ PAGE_SIZE; uint32_t offset = virtual_addr % PAGE_SIZE; \/\/ Search for matching entry (linear search or use hash table) for (int frame = 0; frame < NUM_PHYSICAL_FRAMES; frame++) { if (ipt->entries[frame].valid && ipt->entries[frame].pid == pid && ipt->entries[frame].virtual_page == virtual_page) { uint32_t physical_addr = (frame * PAGE_SIZE) + offset; printf(\"Inverted PT: Found frame %d for PID=%d, VPage=%d\\n\", frame, pid, virtual_page); return physical_addr; } } printf(\"Page fault: No mapping found\\n\"); return UINT32_MAX; } \/* * Pros: * - Fixed size regardless of address space size * - One table for entire system * * Cons: * - Slower lookup (need to search) * - Difficult to implement shared memory * * Used in: IBM PowerPC, HP PA-RISC *\/ ","date":"29-Nov-2024","heading":"11.7 Inverted Page Tables","objectID":"\/posts\/operating-system\/:11:7","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#117-inverted-page-tables"},{"categories":["computer-science"],"collections":null,"content":"11.8 Memory-Mapped Files Map files directly into virtual address space. \/* * Memory-Mapped Files: * Access file contents as if they were in memory * OS handles loading pages from file on demand *\/ #include #include #include #include void example_mmap() { \/\/ Open file int fd = open(\"example.dat\", O_RDWR); if (fd == -1) { perror(\"open\"); return; } \/\/ Get file size struct stat sb; if (fstat(fd, &sb) == -1) { perror(\"fstat\"); close(fd); return; } \/\/ Map file into memory char *mapped = mmap(NULL, \/\/ Let OS choose address sb.st_size, \/\/ Map entire file PROT_READ | PROT_WRITE, \/\/ Read\/write access MAP_SHARED, \/\/ Changes written to file fd, 0); \/\/ Offset 0 if (mapped == MAP_FAILED) { perror(\"mmap\"); close(fd); return; } printf(\"File mapped at %p, size %ld bytes\\n\", mapped, sb.st_size); \/\/ Access file like normal memory! mapped[0] = 'X'; \/\/ Modifies file printf(\"First byte: %c\\n\", mapped[0]); \/\/ Unmap when done munmap(mapped, sb.st_size); close(fd); } \/* * Benefits of mmap: * 1. Simplified file I\/O (no read\/write calls) * 2. Automatic caching by OS * 3. Shared memory between processes * 4. Efficient for large files (load on demand) *\/ ","date":"29-Nov-2024","heading":"11.8 Memory-Mapped Files","objectID":"\/posts\/operating-system\/:11:8","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#118-memory-mapped-files"},{"categories":["computer-science"],"collections":null,"content":"11.9 Copy-on-Write (COW) Optimize fork() by deferring copying. \/* * Copy-on-Write: * When fork() called, don't copy pages immediately * Parent and child share pages (marked read-only) * On write attempt, copy page at that moment *\/ void demonstrate_cow() { int *data = malloc(1000000 * sizeof(int)); \/\/ Fill with data... pid_t pid = fork(); if (pid == 0) { \/\/ Child process printf(\"Child: data pointer = %p\\n\", data); \/\/ Reading is fine (pages shared) printf(\"Child reading: data[0] = %d\\n\", data[0]); \/\/ Writing triggers copy printf(\"Child writing to data[0]...\\n\"); data[0] = 999; \/\/ COW: Page copied now! printf(\"Child: data[0] = %d\\n\", data[0]); } else { \/\/ Parent process sleep(1); printf(\"Parent: data[0] = %d\\n\", data[0]); \/\/ Still original value } } \/* * COW Implementation: * 1. Mark all pages as read-only in child's page table * 2. Set COW bit in page table entries * 3. On write, page fault handler: * a. Check if COW page * b. Allocate new frame * c. Copy data * d. Update page table (mark writable) * e. Resume execution *\/ ","date":"29-Nov-2024","heading":"11.9 Copy-on-Write (COW)","objectID":"\/posts\/operating-system\/:11:9","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#119-copy-on-write-cow"},{"categories":["computer-science"],"collections":null,"content":"11.10 Thrashing When system spends more time paging than executing. \/* * Thrashing occurs when: * - Too many processes competing for limited frames * - Each process doesn't have enough frames * - Constant page faults * - CPU utilization drops dramatically *\/ \/\/ Working Set Model typedef struct { uint32_t *pages; \/\/ Pages in working set int size; \/\/ Current size int max_size; \/\/ Window size uint64_t time_window; \/\/ Time window (e.g., last 1000 refs) } WorkingSet; \/\/ Calculate working set size int calculate_working_set(uint32_t reference_string[], int length, int window) { bool in_working_set[NUM_PAGES] = {false}; int working_set_size = 0; \/\/ Consider last 'window' references int start = (length - window > 0) ? length - window : 0; for (int i = start; i < length; i++) { uint32_t page = reference_string[i]; if (!in_working_set[page]) { in_working_set[page] = true; working_set_size++; } } return working_set_size; } \/* * Solutions to Thrashing: * 1. Working Set Model: Allocate frames based on working set size * 2. Page Fault Frequency: Adjust allocation based on fault rate * 3. Suspend processes if insufficient memory * 4. Increase physical memory *\/ ","date":"29-Nov-2024","heading":"11.10 Thrashing","objectID":"\/posts\/operating-system\/:11:10","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1110-thrashing"},{"categories":["computer-science"],"collections":null,"content":"11.11 Page Replacement Algorithm Comparison Algorithm Advantages Disadvantages Use Case FIFO Simple, low overhead Belady’s anomaly Rarely used alone Optimal Best possible (fewest faults) Impossible to implement Benchmark only LRU Good performance Expensive to implement exactly Common approximation target Clock (Second Chance) Good approximation of LRU Slightly more complex than FIFO Linux, Windows LFU Protects frequently used pages Difficult to implement Specialized cases ","date":"29-Nov-2024","heading":"11.11 Page Replacement Algorithm Comparison","objectID":"\/posts\/operating-system\/:11:11","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1111-page-replacement-algorithm-comparison"},{"categories":["computer-science"],"collections":null,"content":"Key Takeaways Note Virtual Memory Fundamentals - Master These Concepts: Virtual memory separates logical from physical address space \u2192 Programs can use more memory than physically available Demand paging loads pages only when accessed (lazy loading) \u2192 Saves memory, improves startup time TLB caches page table entries for fast translation (95%+ hit rate) \u2192 Critical for performance! Multi-level page tables save space for sparse address spaces \u2192 4-level on x86-64 Page replacement algorithms: FIFO (simple), LRU (optimal), Clock (practical) \u2192 Choose based on workload Copy-on-Write optimizes fork() by deferring page copying \u2192 Makes process creation fast Thrashing occurs when system spends more time paging than executing \u2192 Monitor page fault rate! Memory-mapped files provide efficient file I\/O \u2192 Zero-copy I\/O ","date":"29-Nov-2024","heading":"Key Takeaways","objectID":"\/posts\/operating-system\/:11:12","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#key-takeaways-2"},{"categories":["computer-science"],"collections":null,"content":"12. File Systems ","date":"29-Nov-2024","heading":"12. File Systems","objectID":"\/posts\/operating-system\/:12:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#file-systems"},{"categories":["computer-science"],"collections":null,"content":"12.1 File System Basics A file system organizes and stores data on storage devices, providing: Persistent storage Hierarchical namespace Access control and permissions Efficient space management \/* * File: Named collection of related information * Directory: Container for files and other directories * Path: Location of file in hierarchy *\/ \/\/ File attributes (metadata) typedef struct { char name[256]; uint64_t size; \/\/ Size in bytes uint32_t inode_number; \/\/ Unique identifier uint16_t mode; \/\/ Permissions uint32_t uid; \/\/ Owner user ID uint32_t gid; \/\/ Owner group ID time_t created; \/\/ Creation time time_t modified; \/\/ Last modification time_t accessed; \/\/ Last access uint32_t link_count; \/\/ Number of hard links } FileMetadata; ","date":"29-Nov-2024","heading":"12.1 File System Basics","objectID":"\/posts\/operating-system\/:12:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#121-file-system-basics"},{"categories":["computer-science"],"collections":null,"content":"12.2 File Operations \/* * Core File Operations (System Calls) *\/ #include #include #include #include \/\/ Open file: Get file descriptor int fd = open(\"file.txt\", O_RDWR | O_CREAT, 0644); if (fd == -1) { perror(\"open\"); return -1; } \/\/ Read from file char buffer[1024]; ssize_t bytes_read = read(fd, buffer, sizeof(buffer)); printf(\"Read %zd bytes\\n\", bytes_read); \/\/ Write to file const char *data = \"Hello, World!\"; ssize_t bytes_written = write(fd, data, strlen(data)); \/\/ Move file pointer (seek) off_t new_pos = lseek(fd, 0, SEEK_SET); \/\/ Seek to beginning \/\/ SEEK_CUR: relative to current position \/\/ SEEK_END: relative to end of file \/\/ Get file information struct stat file_info; if (fstat(fd, &file_info) == 0) { printf(\"File size: %ld bytes\\n\", file_info.st_size); printf(\"Inode: %lu\\n\", file_info.st_ino); printf(\"Permissions: %o\\n\", file_info.st_mode & 0777); } \/\/ Close file close(fd); \/* * File Descriptor Table: * Process maintains table mapping fd to open file structures * fd 0: stdin, fd 1: stdout, fd 2: stderr * New files get lowest available fd *\/ ","date":"29-Nov-2024","heading":"12.2 File Operations","objectID":"\/posts\/operating-system\/:12:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#122-file-operations"},{"categories":["computer-science"],"collections":null,"content":"12.3 Directory Structure Unix\/Linux File System Hierarchy \/* * Unix\/Linux Filesystem Hierarchy Standard (FHS) * * \/ Root directory * \u251c\u2500\u2500 bin\/ Essential user binaries (ls, cp, mv) * \u251c\u2500\u2500 boot\/ Boot loader files (kernel, initrd) * \u251c\u2500\u2500 dev\/ Device files (disks, terminals) * \u251c\u2500\u2500 etc\/ System configuration files * \u251c\u2500\u2500 home\/ User home directories * \u251c\u2500\u2500 lib\/ Shared libraries * \u251c\u2500\u2500 mnt\/ Temporary mount points * \u251c\u2500\u2500 opt\/ Optional application packages * \u251c\u2500\u2500 proc\/ Process information (virtual filesystem) * \u251c\u2500\u2500 root\/ Root user's home directory * \u251c\u2500\u2500 sbin\/ System binaries (fsck, init) * \u251c\u2500\u2500 sys\/ Kernel and system information (virtual) * \u251c\u2500\u2500 tmp\/ Temporary files * \u251c\u2500\u2500 usr\/ User programs and data * \u2502 \u251c\u2500\u2500 bin\/ User binaries * \u2502 \u251c\u2500\u2500 lib\/ Libraries * \u2502 \u2514\u2500\u2500 share\/ Shared data * \u2514\u2500\u2500 var\/ Variable data (logs, mail, temp files) * \u251c\u2500\u2500 log\/ System logs * \u2514\u2500\u2500 tmp\/ Temporary files *\/ \/\/ Directory operations #include void list_directory(const char *path) { DIR *dir = opendir(path); if (!dir) { perror(\"opendir\"); return; } struct dirent *entry; while ((entry = readdir(dir)) != NULL) { printf(\"%s\", entry->d_name); \/\/ Determine file type switch (entry->d_type) { case DT_REG: printf(\" (file)\\n\"); break; case DT_DIR: printf(\" (directory)\\n\"); break; case DT_LNK: printf(\" (symlink)\\n\"); break; case DT_CHR: printf(\" (char device)\\n\"); break; case DT_BLK: printf(\" (block device)\\n\"); break; case DT_FIFO: printf(\" (FIFO)\\n\"); break; case DT_SOCK: printf(\" (socket)\\n\"); break; default: printf(\" (unknown)\\n\"); break; } } closedir(dir); } ","date":"29-Nov-2024","heading":"12.3 Directory Structure","objectID":"\/posts\/operating-system\/:12:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#123-directory-structure"},{"categories":["computer-science"],"collections":null,"content":"12.3 Directory Structure Unix\/Linux File System Hierarchy \/* * Unix\/Linux Filesystem Hierarchy Standard (FHS) * * \/ Root directory * \u251c\u2500\u2500 bin\/ Essential user binaries (ls, cp, mv) * \u251c\u2500\u2500 boot\/ Boot loader files (kernel, initrd) * \u251c\u2500\u2500 dev\/ Device files (disks, terminals) * \u251c\u2500\u2500 etc\/ System configuration files * \u251c\u2500\u2500 home\/ User home directories * \u251c\u2500\u2500 lib\/ Shared libraries * \u251c\u2500\u2500 mnt\/ Temporary mount points * \u251c\u2500\u2500 opt\/ Optional application packages * \u251c\u2500\u2500 proc\/ Process information (virtual filesystem) * \u251c\u2500\u2500 root\/ Root user's home directory * \u251c\u2500\u2500 sbin\/ System binaries (fsck, init) * \u251c\u2500\u2500 sys\/ Kernel and system information (virtual) * \u251c\u2500\u2500 tmp\/ Temporary files * \u251c\u2500\u2500 usr\/ User programs and data * \u2502 \u251c\u2500\u2500 bin\/ User binaries * \u2502 \u251c\u2500\u2500 lib\/ Libraries * \u2502 \u2514\u2500\u2500 share\/ Shared data * \u2514\u2500\u2500 var\/ Variable data (logs, mail, temp files) * \u251c\u2500\u2500 log\/ System logs * \u2514\u2500\u2500 tmp\/ Temporary files *\/ \/\/ Directory operations #include void list_directory(const char *path) { DIR *dir = opendir(path); if (!dir) { perror(\"opendir\"); return; } struct dirent *entry; while ((entry = readdir(dir)) != NULL) { printf(\"%s\", entry->d_name); \/\/ Determine file type switch (entry->d_type) { case DT_REG: printf(\" (file)\\n\"); break; case DT_DIR: printf(\" (directory)\\n\"); break; case DT_LNK: printf(\" (symlink)\\n\"); break; case DT_CHR: printf(\" (char device)\\n\"); break; case DT_BLK: printf(\" (block device)\\n\"); break; case DT_FIFO: printf(\" (FIFO)\\n\"); break; case DT_SOCK: printf(\" (socket)\\n\"); break; default: printf(\" (unknown)\\n\"); break; } } closedir(dir); } ","date":"29-Nov-2024","heading":"Unix\/Linux File System Hierarchy","objectID":"\/posts\/operating-system\/:12:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#unixlinux-file-system-hierarchy"},{"categories":["computer-science"],"collections":null,"content":"12.4 File Allocation Methods 12.4.1 Contiguous Allocation \/* * Contiguous Allocation: * File occupies consecutive blocks on disk * * Pros: * - Simple * - Fast sequential access * - Easy random access * * Cons: * - External fragmentation * - Difficult to grow files * - Need to know file size in advance *\/ typedef struct { char filename[256]; uint32_t start_block; \/\/ Starting block number uint32_t length; \/\/ Number of blocks } ContiguousFile; void read_contiguous_file(ContiguousFile *file, uint32_t offset) { uint32_t block_size = 4096; uint32_t block_number = file->start_block + (offset \/ block_size); uint32_t block_offset = offset % block_size; printf(\"Reading from block %d, offset %d\\n\", block_number, block_offset); \/\/ Read from disk block... } 12.4.2 Linked Allocation \/* * Linked Allocation: * Each block contains pointer to next block * * Pros: * - No external fragmentation * - File can grow easily * - No need to know size in advance * * Cons: * - Slow random access * - Space overhead (pointers) * - Poor reliability (broken link loses rest of file) *\/ #define BLOCK_SIZE 4096 #define DATA_SIZE (BLOCK_SIZE - sizeof(uint32_t)) typedef struct FileBlock { uint8_t data[DATA_SIZE]; uint32_t next_block; \/\/ Next block number (or -1 for end) } FileBlock; typedef struct { char filename[256]; uint32_t first_block; } LinkedFile; void read_linked_file(LinkedFile *file, uint32_t offset) { uint32_t current_block = file->first_block; uint32_t bytes_to_skip = offset; \/\/ Follow chain to find correct block while (bytes_to_skip > DATA_SIZE) { FileBlock block; \/\/ Read block from disk... current_block = block.next_block; if (current_block == (uint32_t)-1) { printf(\"Offset beyond end of file\\n\"); return; } bytes_to_skip -= DATA_SIZE; } printf(\"Reading from block %d, offset %d\\n\", current_block, bytes_to_skip); } \/\/ FAT (File Allocation Table) Variant #define NUM_BLOCKS 10000 typedef struct { uint32_t fat[NUM_BLOCKS]; \/\/ fat[i] = next block after block i } FAT; \/* * FAT Advantages: * - Random access improved (FAT can be cached in memory) * - Only need to traverse FAT, not actual disk blocks * Used in: MS-DOS, FAT32 *\/ 12.4.3 Indexed Allocation \/* * Indexed Allocation: * Index block contains pointers to all data blocks * * Pros: * - Fast random access * - No external fragmentation * - Supports direct access * * Cons: * - Index block overhead * - Maximum file size limited by index block size *\/ #define POINTERS_PER_BLOCK (BLOCK_SIZE \/ sizeof(uint32_t)) \/\/ 1024 pointers typedef struct { uint32_t block_pointers[POINTERS_PER_BLOCK]; } IndexBlock; typedef struct { char filename[256]; uint32_t index_block_number; uint32_t file_size; } IndexedFile; void read_indexed_file(IndexedFile *file, uint32_t offset) { uint32_t block_num = offset \/ BLOCK_SIZE; uint32_t block_offset = offset % BLOCK_SIZE; if (block_num >= POINTERS_PER_BLOCK) { printf(\"File too large for single index block\\n\"); return; } \/\/ Read index block IndexBlock index; \/\/ read_disk_block(file->index_block_number, &index); uint32_t data_block = index.block_pointers[block_num]; printf(\"Reading from block %d, offset %d\\n\", data_block, block_offset); } \/\/ Multi-level Indexed (used in Unix inode) typedef struct { uint32_t direct[12]; \/\/ Direct block pointers uint32_t single_indirect; \/\/ Points to block of pointers uint32_t double_indirect; \/\/ Points to block of indirect blocks uint32_t triple_indirect; \/\/ Three levels of indirection } MultiLevelIndex; \/* * Unix inode addressing: * - Direct: 12 * 4KB = 48KB * - Single indirect: 1024 * 4KB = 4MB * - Double indirect: 1024 * 1024 * 4KB = 4GB * - Triple indirect: 1024 * 1024 * 1024 * 4KB = 4TB * Total: ~4TB max file size *\/ ","date":"29-Nov-2024","heading":"12.4 File Allocation Methods","objectID":"\/posts\/operating-system\/:12:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#124-file-allocation-methods"},{"categories":["computer-science"],"collections":null,"content":"12.4 File Allocation Methods 12.4.1 Contiguous Allocation \/* * Contiguous Allocation: * File occupies consecutive blocks on disk * * Pros: * - Simple * - Fast sequential access * - Easy random access * * Cons: * - External fragmentation * - Difficult to grow files * - Need to know file size in advance *\/ typedef struct { char filename[256]; uint32_t start_block; \/\/ Starting block number uint32_t length; \/\/ Number of blocks } ContiguousFile; void read_contiguous_file(ContiguousFile *file, uint32_t offset) { uint32_t block_size = 4096; uint32_t block_number = file->start_block + (offset \/ block_size); uint32_t block_offset = offset % block_size; printf(\"Reading from block %d, offset %d\\n\", block_number, block_offset); \/\/ Read from disk block... } 12.4.2 Linked Allocation \/* * Linked Allocation: * Each block contains pointer to next block * * Pros: * - No external fragmentation * - File can grow easily * - No need to know size in advance * * Cons: * - Slow random access * - Space overhead (pointers) * - Poor reliability (broken link loses rest of file) *\/ #define BLOCK_SIZE 4096 #define DATA_SIZE (BLOCK_SIZE - sizeof(uint32_t)) typedef struct FileBlock { uint8_t data[DATA_SIZE]; uint32_t next_block; \/\/ Next block number (or -1 for end) } FileBlock; typedef struct { char filename[256]; uint32_t first_block; } LinkedFile; void read_linked_file(LinkedFile *file, uint32_t offset) { uint32_t current_block = file->first_block; uint32_t bytes_to_skip = offset; \/\/ Follow chain to find correct block while (bytes_to_skip > DATA_SIZE) { FileBlock block; \/\/ Read block from disk... current_block = block.next_block; if (current_block == (uint32_t)-1) { printf(\"Offset beyond end of file\\n\"); return; } bytes_to_skip -= DATA_SIZE; } printf(\"Reading from block %d, offset %d\\n\", current_block, bytes_to_skip); } \/\/ FAT (File Allocation Table) Variant #define NUM_BLOCKS 10000 typedef struct { uint32_t fat[NUM_BLOCKS]; \/\/ fat[i] = next block after block i } FAT; \/* * FAT Advantages: * - Random access improved (FAT can be cached in memory) * - Only need to traverse FAT, not actual disk blocks * Used in: MS-DOS, FAT32 *\/ 12.4.3 Indexed Allocation \/* * Indexed Allocation: * Index block contains pointers to all data blocks * * Pros: * - Fast random access * - No external fragmentation * - Supports direct access * * Cons: * - Index block overhead * - Maximum file size limited by index block size *\/ #define POINTERS_PER_BLOCK (BLOCK_SIZE \/ sizeof(uint32_t)) \/\/ 1024 pointers typedef struct { uint32_t block_pointers[POINTERS_PER_BLOCK]; } IndexBlock; typedef struct { char filename[256]; uint32_t index_block_number; uint32_t file_size; } IndexedFile; void read_indexed_file(IndexedFile *file, uint32_t offset) { uint32_t block_num = offset \/ BLOCK_SIZE; uint32_t block_offset = offset % BLOCK_SIZE; if (block_num >= POINTERS_PER_BLOCK) { printf(\"File too large for single index block\\n\"); return; } \/\/ Read index block IndexBlock index; \/\/ read_disk_block(file->index_block_number, &index); uint32_t data_block = index.block_pointers[block_num]; printf(\"Reading from block %d, offset %d\\n\", data_block, block_offset); } \/\/ Multi-level Indexed (used in Unix inode) typedef struct { uint32_t direct[12]; \/\/ Direct block pointers uint32_t single_indirect; \/\/ Points to block of pointers uint32_t double_indirect; \/\/ Points to block of indirect blocks uint32_t triple_indirect; \/\/ Three levels of indirection } MultiLevelIndex; \/* * Unix inode addressing: * - Direct: 12 * 4KB = 48KB * - Single indirect: 1024 * 4KB = 4MB * - Double indirect: 1024 * 1024 * 4KB = 4GB * - Triple indirect: 1024 * 1024 * 1024 * 4KB = 4TB * Total: ~4TB max file size *\/ ","date":"29-Nov-2024","heading":"12.4.1 Contiguous Allocation","objectID":"\/posts\/operating-system\/:12:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1241-contiguous-allocation"},{"categories":["computer-science"],"collections":null,"content":"12.4 File Allocation Methods 12.4.1 Contiguous Allocation \/* * Contiguous Allocation: * File occupies consecutive blocks on disk * * Pros: * - Simple * - Fast sequential access * - Easy random access * * Cons: * - External fragmentation * - Difficult to grow files * - Need to know file size in advance *\/ typedef struct { char filename[256]; uint32_t start_block; \/\/ Starting block number uint32_t length; \/\/ Number of blocks } ContiguousFile; void read_contiguous_file(ContiguousFile *file, uint32_t offset) { uint32_t block_size = 4096; uint32_t block_number = file->start_block + (offset \/ block_size); uint32_t block_offset = offset % block_size; printf(\"Reading from block %d, offset %d\\n\", block_number, block_offset); \/\/ Read from disk block... } 12.4.2 Linked Allocation \/* * Linked Allocation: * Each block contains pointer to next block * * Pros: * - No external fragmentation * - File can grow easily * - No need to know size in advance * * Cons: * - Slow random access * - Space overhead (pointers) * - Poor reliability (broken link loses rest of file) *\/ #define BLOCK_SIZE 4096 #define DATA_SIZE (BLOCK_SIZE - sizeof(uint32_t)) typedef struct FileBlock { uint8_t data[DATA_SIZE]; uint32_t next_block; \/\/ Next block number (or -1 for end) } FileBlock; typedef struct { char filename[256]; uint32_t first_block; } LinkedFile; void read_linked_file(LinkedFile *file, uint32_t offset) { uint32_t current_block = file->first_block; uint32_t bytes_to_skip = offset; \/\/ Follow chain to find correct block while (bytes_to_skip > DATA_SIZE) { FileBlock block; \/\/ Read block from disk... current_block = block.next_block; if (current_block == (uint32_t)-1) { printf(\"Offset beyond end of file\\n\"); return; } bytes_to_skip -= DATA_SIZE; } printf(\"Reading from block %d, offset %d\\n\", current_block, bytes_to_skip); } \/\/ FAT (File Allocation Table) Variant #define NUM_BLOCKS 10000 typedef struct { uint32_t fat[NUM_BLOCKS]; \/\/ fat[i] = next block after block i } FAT; \/* * FAT Advantages: * - Random access improved (FAT can be cached in memory) * - Only need to traverse FAT, not actual disk blocks * Used in: MS-DOS, FAT32 *\/ 12.4.3 Indexed Allocation \/* * Indexed Allocation: * Index block contains pointers to all data blocks * * Pros: * - Fast random access * - No external fragmentation * - Supports direct access * * Cons: * - Index block overhead * - Maximum file size limited by index block size *\/ #define POINTERS_PER_BLOCK (BLOCK_SIZE \/ sizeof(uint32_t)) \/\/ 1024 pointers typedef struct { uint32_t block_pointers[POINTERS_PER_BLOCK]; } IndexBlock; typedef struct { char filename[256]; uint32_t index_block_number; uint32_t file_size; } IndexedFile; void read_indexed_file(IndexedFile *file, uint32_t offset) { uint32_t block_num = offset \/ BLOCK_SIZE; uint32_t block_offset = offset % BLOCK_SIZE; if (block_num >= POINTERS_PER_BLOCK) { printf(\"File too large for single index block\\n\"); return; } \/\/ Read index block IndexBlock index; \/\/ read_disk_block(file->index_block_number, &index); uint32_t data_block = index.block_pointers[block_num]; printf(\"Reading from block %d, offset %d\\n\", data_block, block_offset); } \/\/ Multi-level Indexed (used in Unix inode) typedef struct { uint32_t direct[12]; \/\/ Direct block pointers uint32_t single_indirect; \/\/ Points to block of pointers uint32_t double_indirect; \/\/ Points to block of indirect blocks uint32_t triple_indirect; \/\/ Three levels of indirection } MultiLevelIndex; \/* * Unix inode addressing: * - Direct: 12 * 4KB = 48KB * - Single indirect: 1024 * 4KB = 4MB * - Double indirect: 1024 * 1024 * 4KB = 4GB * - Triple indirect: 1024 * 1024 * 1024 * 4KB = 4TB * Total: ~4TB max file size *\/ ","date":"29-Nov-2024","heading":"12.4.2 Linked Allocation","objectID":"\/posts\/operating-system\/:12:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1242-linked-allocation"},{"categories":["computer-science"],"collections":null,"content":"12.4 File Allocation Methods 12.4.1 Contiguous Allocation \/* * Contiguous Allocation: * File occupies consecutive blocks on disk * * Pros: * - Simple * - Fast sequential access * - Easy random access * * Cons: * - External fragmentation * - Difficult to grow files * - Need to know file size in advance *\/ typedef struct { char filename[256]; uint32_t start_block; \/\/ Starting block number uint32_t length; \/\/ Number of blocks } ContiguousFile; void read_contiguous_file(ContiguousFile *file, uint32_t offset) { uint32_t block_size = 4096; uint32_t block_number = file->start_block + (offset \/ block_size); uint32_t block_offset = offset % block_size; printf(\"Reading from block %d, offset %d\\n\", block_number, block_offset); \/\/ Read from disk block... } 12.4.2 Linked Allocation \/* * Linked Allocation: * Each block contains pointer to next block * * Pros: * - No external fragmentation * - File can grow easily * - No need to know size in advance * * Cons: * - Slow random access * - Space overhead (pointers) * - Poor reliability (broken link loses rest of file) *\/ #define BLOCK_SIZE 4096 #define DATA_SIZE (BLOCK_SIZE - sizeof(uint32_t)) typedef struct FileBlock { uint8_t data[DATA_SIZE]; uint32_t next_block; \/\/ Next block number (or -1 for end) } FileBlock; typedef struct { char filename[256]; uint32_t first_block; } LinkedFile; void read_linked_file(LinkedFile *file, uint32_t offset) { uint32_t current_block = file->first_block; uint32_t bytes_to_skip = offset; \/\/ Follow chain to find correct block while (bytes_to_skip > DATA_SIZE) { FileBlock block; \/\/ Read block from disk... current_block = block.next_block; if (current_block == (uint32_t)-1) { printf(\"Offset beyond end of file\\n\"); return; } bytes_to_skip -= DATA_SIZE; } printf(\"Reading from block %d, offset %d\\n\", current_block, bytes_to_skip); } \/\/ FAT (File Allocation Table) Variant #define NUM_BLOCKS 10000 typedef struct { uint32_t fat[NUM_BLOCKS]; \/\/ fat[i] = next block after block i } FAT; \/* * FAT Advantages: * - Random access improved (FAT can be cached in memory) * - Only need to traverse FAT, not actual disk blocks * Used in: MS-DOS, FAT32 *\/ 12.4.3 Indexed Allocation \/* * Indexed Allocation: * Index block contains pointers to all data blocks * * Pros: * - Fast random access * - No external fragmentation * - Supports direct access * * Cons: * - Index block overhead * - Maximum file size limited by index block size *\/ #define POINTERS_PER_BLOCK (BLOCK_SIZE \/ sizeof(uint32_t)) \/\/ 1024 pointers typedef struct { uint32_t block_pointers[POINTERS_PER_BLOCK]; } IndexBlock; typedef struct { char filename[256]; uint32_t index_block_number; uint32_t file_size; } IndexedFile; void read_indexed_file(IndexedFile *file, uint32_t offset) { uint32_t block_num = offset \/ BLOCK_SIZE; uint32_t block_offset = offset % BLOCK_SIZE; if (block_num >= POINTERS_PER_BLOCK) { printf(\"File too large for single index block\\n\"); return; } \/\/ Read index block IndexBlock index; \/\/ read_disk_block(file->index_block_number, &index); uint32_t data_block = index.block_pointers[block_num]; printf(\"Reading from block %d, offset %d\\n\", data_block, block_offset); } \/\/ Multi-level Indexed (used in Unix inode) typedef struct { uint32_t direct[12]; \/\/ Direct block pointers uint32_t single_indirect; \/\/ Points to block of pointers uint32_t double_indirect; \/\/ Points to block of indirect blocks uint32_t triple_indirect; \/\/ Three levels of indirection } MultiLevelIndex; \/* * Unix inode addressing: * - Direct: 12 * 4KB = 48KB * - Single indirect: 1024 * 4KB = 4MB * - Double indirect: 1024 * 1024 * 4KB = 4GB * - Triple indirect: 1024 * 1024 * 1024 * 4KB = 4TB * Total: ~4TB max file size *\/ ","date":"29-Nov-2024","heading":"12.4.3 Indexed Allocation","objectID":"\/posts\/operating-system\/:12:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1243-indexed-allocation"},{"categories":["computer-science"],"collections":null,"content":"12.5 Unix\/Linux Inode Structure \/* * Inode: Index Node * Contains all file metadata except name * Name stored in directory entry *\/ typedef struct { uint16_t mode; \/\/ File type and permissions uint16_t uid; \/\/ User ID uint32_t size; \/\/ File size in bytes uint32_t atime; \/\/ Access time uint32_t ctime; \/\/ Change time (metadata) uint32_t mtime; \/\/ Modification time (content) uint32_t dtime; \/\/ Deletion time uint16_t gid; \/\/ Group ID uint16_t links_count; \/\/ Hard link count uint32_t blocks; \/\/ Number of 512-byte blocks uint32_t flags; \/\/ File flags uint32_t block[15]; \/\/ Block pointers (12 direct + 3 indirect) } Inode; \/\/ Directory entry typedef struct { uint32_t inode_number; uint16_t rec_len; \/\/ Length of this directory entry uint8_t name_len; \/\/ Length of filename uint8_t file_type; \/\/ File type char name[255]; \/\/ Filename (variable length) } DirEntry; \/* * Hard Link: Multiple directory entries point to same inode * Symbolic Link: File containing path to another file *\/ \/\/ Create hard link link(\"original.txt\", \"hardlink.txt\"); \/\/ Both point to same inode \/\/ Create symbolic link symlink(\"original.txt\", \"symlink.txt\"); \/\/ Contains path string \/\/ Differences: \/\/ - Hard link: Same inode, survives if original deleted \/\/ - Sym link: Different inode, breaks if original deleted ","date":"29-Nov-2024","heading":"12.5 Unix\/Linux Inode Structure","objectID":"\/posts\/operating-system\/:12:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#125-unixlinux-inode-structure"},{"categories":["computer-science"],"collections":null,"content":"12.6 Virtual File System (VFS) \/* * VFS: Abstract layer above file systems * Provides uniform interface for different FS types * Allows ext4, FAT, NFS, etc. to coexist *\/ \/\/ VFS abstraction typedef struct FileSystemOperations { int (*open)(const char *path, int flags); ssize_t (*read)(int fd, void *buf, size_t count); ssize_t (*write)(int fd, const void *buf, size_t count); int (*close)(int fd); int (*mkdir)(const char *path, mode_t mode); \/\/ ... more operations } FSOperations; \/\/ Different file systems implement interface FSOperations ext4_ops = { .open = ext4_open, .read = ext4_read, .write = ext4_write, \/\/ ... }; FSOperations fat32_ops = { .open = fat32_open, .read = fat32_read, .write = fat32_write, \/\/ ... }; \/* * VFS Components: * 1. Superblock: Filesystem metadata * 2. Inode: File metadata * 3. Dentry: Directory entry cache * 4. File: Open file representation *\/ typedef struct VFS_Superblock { uint32_t magic; \/\/ Filesystem type uint32_t block_size; uint64_t total_blocks; uint64_t free_blocks; FSOperations *ops; \/\/ Filesystem-specific operations } VFS_Superblock; ","date":"29-Nov-2024","heading":"12.6 Virtual File System (VFS)","objectID":"\/posts\/operating-system\/:12:6","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#126-virtual-file-system-vfs"},{"categories":["computer-science"],"collections":null,"content":"12.7 Free Space Management 12.7.1 Bit Vector (Bitmap) \/* * Bitmap: One bit per block * 0 = free, 1 = allocated * * Pros: Simple, efficient to find free blocks * Cons: Extra space overhead *\/ #define TOTAL_BLOCKS 100000 #define BITMAP_SIZE (TOTAL_BLOCKS \/ 8) \/\/ Bytes needed uint8_t bitmap[BITMAP_SIZE]; \/\/ Set bit (mark block as allocated) void mark_allocated(uint32_t block_num) { uint32_t byte_index = block_num \/ 8; uint8_t bit_offset = block_num % 8; bitmap[byte_index] |= (1 << bit_offset); } \/\/ Clear bit (mark block as free) void mark_free(uint32_t block_num) { uint32_t byte_index = block_num \/ 8; uint8_t bit_offset = block_num % 8; bitmap[byte_index] &= ~(1 << bit_offset); } \/\/ Check if block is free bool is_free(uint32_t block_num) { uint32_t byte_index = block_num \/ 8; uint8_t bit_offset = block_num % 8; return (bitmap[byte_index] & (1 << bit_offset)) == 0; } \/\/ Find first free block uint32_t find_free_block() { for (uint32_t i = 0; i < BITMAP_SIZE; i++) { if (bitmap[i] != 0xFF) { \/\/ Not all bits set \/\/ Find first zero bit for (uint8_t bit = 0; bit < 8; bit++) { if ((bitmap[i] & (1 << bit)) == 0) { return (i * 8) + bit; } } } } return UINT32_MAX; \/\/ No free blocks } 12.7.2 Linked List \/* * Linked List of Free Blocks * * Pros: No extra space needed (use free blocks themselves) * Cons: Cannot efficiently find contiguous space *\/ typedef struct FreeBlock { uint32_t next_free; \/\/ Stored in free block itself } FreeBlock; typedef struct { uint32_t head; \/\/ First free block } FreeList; FreeList free_list = {.head = 0}; uint32_t allocate_block(FreeList *list) { if (list->head == UINT32_MAX) { return UINT32_MAX; \/\/ No free blocks } uint32_t allocated = list->head; \/\/ Read free block to get next pointer FreeBlock block; \/\/ read_disk_block(allocated, &block); list->head = block.next_free; printf(\"Allocated block %d\\n\", allocated); return allocated; } void free_block(FreeList *list, uint32_t block_num) { FreeBlock block; block.next_free = list->head; \/\/ Write block back with updated pointer \/\/ write_disk_block(block_num, &block); list->head = block_num; printf(\"Freed block %d\\n\", block_num); } 12.7.3 Grouping \/* * Grouping: Store addresses of n free blocks in first free block * Last address points to another block with n free addresses *\/ #define ADDRESSES_PER_BLOCK 512 typedef struct FreeBlockGroup { uint32_t count; uint32_t blocks[ADDRESSES_PER_BLOCK]; } FreeBlockGroup; ","date":"29-Nov-2024","heading":"12.7 Free Space Management","objectID":"\/posts\/operating-system\/:12:7","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#127-free-space-management"},{"categories":["computer-science"],"collections":null,"content":"12.7 Free Space Management 12.7.1 Bit Vector (Bitmap) \/* * Bitmap: One bit per block * 0 = free, 1 = allocated * * Pros: Simple, efficient to find free blocks * Cons: Extra space overhead *\/ #define TOTAL_BLOCKS 100000 #define BITMAP_SIZE (TOTAL_BLOCKS \/ 8) \/\/ Bytes needed uint8_t bitmap[BITMAP_SIZE]; \/\/ Set bit (mark block as allocated) void mark_allocated(uint32_t block_num) { uint32_t byte_index = block_num \/ 8; uint8_t bit_offset = block_num % 8; bitmap[byte_index] |= (1 << bit_offset); } \/\/ Clear bit (mark block as free) void mark_free(uint32_t block_num) { uint32_t byte_index = block_num \/ 8; uint8_t bit_offset = block_num % 8; bitmap[byte_index] &= ~(1 << bit_offset); } \/\/ Check if block is free bool is_free(uint32_t block_num) { uint32_t byte_index = block_num \/ 8; uint8_t bit_offset = block_num % 8; return (bitmap[byte_index] & (1 << bit_offset)) == 0; } \/\/ Find first free block uint32_t find_free_block() { for (uint32_t i = 0; i < BITMAP_SIZE; i++) { if (bitmap[i] != 0xFF) { \/\/ Not all bits set \/\/ Find first zero bit for (uint8_t bit = 0; bit < 8; bit++) { if ((bitmap[i] & (1 << bit)) == 0) { return (i * 8) + bit; } } } } return UINT32_MAX; \/\/ No free blocks } 12.7.2 Linked List \/* * Linked List of Free Blocks * * Pros: No extra space needed (use free blocks themselves) * Cons: Cannot efficiently find contiguous space *\/ typedef struct FreeBlock { uint32_t next_free; \/\/ Stored in free block itself } FreeBlock; typedef struct { uint32_t head; \/\/ First free block } FreeList; FreeList free_list = {.head = 0}; uint32_t allocate_block(FreeList *list) { if (list->head == UINT32_MAX) { return UINT32_MAX; \/\/ No free blocks } uint32_t allocated = list->head; \/\/ Read free block to get next pointer FreeBlock block; \/\/ read_disk_block(allocated, &block); list->head = block.next_free; printf(\"Allocated block %d\\n\", allocated); return allocated; } void free_block(FreeList *list, uint32_t block_num) { FreeBlock block; block.next_free = list->head; \/\/ Write block back with updated pointer \/\/ write_disk_block(block_num, &block); list->head = block_num; printf(\"Freed block %d\\n\", block_num); } 12.7.3 Grouping \/* * Grouping: Store addresses of n free blocks in first free block * Last address points to another block with n free addresses *\/ #define ADDRESSES_PER_BLOCK 512 typedef struct FreeBlockGroup { uint32_t count; uint32_t blocks[ADDRESSES_PER_BLOCK]; } FreeBlockGroup; ","date":"29-Nov-2024","heading":"12.7.1 Bit Vector (Bitmap)","objectID":"\/posts\/operating-system\/:12:7","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1271-bit-vector-bitmap"},{"categories":["computer-science"],"collections":null,"content":"12.7 Free Space Management 12.7.1 Bit Vector (Bitmap) \/* * Bitmap: One bit per block * 0 = free, 1 = allocated * * Pros: Simple, efficient to find free blocks * Cons: Extra space overhead *\/ #define TOTAL_BLOCKS 100000 #define BITMAP_SIZE (TOTAL_BLOCKS \/ 8) \/\/ Bytes needed uint8_t bitmap[BITMAP_SIZE]; \/\/ Set bit (mark block as allocated) void mark_allocated(uint32_t block_num) { uint32_t byte_index = block_num \/ 8; uint8_t bit_offset = block_num % 8; bitmap[byte_index] |= (1 << bit_offset); } \/\/ Clear bit (mark block as free) void mark_free(uint32_t block_num) { uint32_t byte_index = block_num \/ 8; uint8_t bit_offset = block_num % 8; bitmap[byte_index] &= ~(1 << bit_offset); } \/\/ Check if block is free bool is_free(uint32_t block_num) { uint32_t byte_index = block_num \/ 8; uint8_t bit_offset = block_num % 8; return (bitmap[byte_index] & (1 << bit_offset)) == 0; } \/\/ Find first free block uint32_t find_free_block() { for (uint32_t i = 0; i < BITMAP_SIZE; i++) { if (bitmap[i] != 0xFF) { \/\/ Not all bits set \/\/ Find first zero bit for (uint8_t bit = 0; bit < 8; bit++) { if ((bitmap[i] & (1 << bit)) == 0) { return (i * 8) + bit; } } } } return UINT32_MAX; \/\/ No free blocks } 12.7.2 Linked List \/* * Linked List of Free Blocks * * Pros: No extra space needed (use free blocks themselves) * Cons: Cannot efficiently find contiguous space *\/ typedef struct FreeBlock { uint32_t next_free; \/\/ Stored in free block itself } FreeBlock; typedef struct { uint32_t head; \/\/ First free block } FreeList; FreeList free_list = {.head = 0}; uint32_t allocate_block(FreeList *list) { if (list->head == UINT32_MAX) { return UINT32_MAX; \/\/ No free blocks } uint32_t allocated = list->head; \/\/ Read free block to get next pointer FreeBlock block; \/\/ read_disk_block(allocated, &block); list->head = block.next_free; printf(\"Allocated block %d\\n\", allocated); return allocated; } void free_block(FreeList *list, uint32_t block_num) { FreeBlock block; block.next_free = list->head; \/\/ Write block back with updated pointer \/\/ write_disk_block(block_num, &block); list->head = block_num; printf(\"Freed block %d\\n\", block_num); } 12.7.3 Grouping \/* * Grouping: Store addresses of n free blocks in first free block * Last address points to another block with n free addresses *\/ #define ADDRESSES_PER_BLOCK 512 typedef struct FreeBlockGroup { uint32_t count; uint32_t blocks[ADDRESSES_PER_BLOCK]; } FreeBlockGroup; ","date":"29-Nov-2024","heading":"12.7.2 Linked List","objectID":"\/posts\/operating-system\/:12:7","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1272-linked-list"},{"categories":["computer-science"],"collections":null,"content":"12.7 Free Space Management 12.7.1 Bit Vector (Bitmap) \/* * Bitmap: One bit per block * 0 = free, 1 = allocated * * Pros: Simple, efficient to find free blocks * Cons: Extra space overhead *\/ #define TOTAL_BLOCKS 100000 #define BITMAP_SIZE (TOTAL_BLOCKS \/ 8) \/\/ Bytes needed uint8_t bitmap[BITMAP_SIZE]; \/\/ Set bit (mark block as allocated) void mark_allocated(uint32_t block_num) { uint32_t byte_index = block_num \/ 8; uint8_t bit_offset = block_num % 8; bitmap[byte_index] |= (1 << bit_offset); } \/\/ Clear bit (mark block as free) void mark_free(uint32_t block_num) { uint32_t byte_index = block_num \/ 8; uint8_t bit_offset = block_num % 8; bitmap[byte_index] &= ~(1 << bit_offset); } \/\/ Check if block is free bool is_free(uint32_t block_num) { uint32_t byte_index = block_num \/ 8; uint8_t bit_offset = block_num % 8; return (bitmap[byte_index] & (1 << bit_offset)) == 0; } \/\/ Find first free block uint32_t find_free_block() { for (uint32_t i = 0; i < BITMAP_SIZE; i++) { if (bitmap[i] != 0xFF) { \/\/ Not all bits set \/\/ Find first zero bit for (uint8_t bit = 0; bit < 8; bit++) { if ((bitmap[i] & (1 << bit)) == 0) { return (i * 8) + bit; } } } } return UINT32_MAX; \/\/ No free blocks } 12.7.2 Linked List \/* * Linked List of Free Blocks * * Pros: No extra space needed (use free blocks themselves) * Cons: Cannot efficiently find contiguous space *\/ typedef struct FreeBlock { uint32_t next_free; \/\/ Stored in free block itself } FreeBlock; typedef struct { uint32_t head; \/\/ First free block } FreeList; FreeList free_list = {.head = 0}; uint32_t allocate_block(FreeList *list) { if (list->head == UINT32_MAX) { return UINT32_MAX; \/\/ No free blocks } uint32_t allocated = list->head; \/\/ Read free block to get next pointer FreeBlock block; \/\/ read_disk_block(allocated, &block); list->head = block.next_free; printf(\"Allocated block %d\\n\", allocated); return allocated; } void free_block(FreeList *list, uint32_t block_num) { FreeBlock block; block.next_free = list->head; \/\/ Write block back with updated pointer \/\/ write_disk_block(block_num, &block); list->head = block_num; printf(\"Freed block %d\\n\", block_num); } 12.7.3 Grouping \/* * Grouping: Store addresses of n free blocks in first free block * Last address points to another block with n free addresses *\/ #define ADDRESSES_PER_BLOCK 512 typedef struct FreeBlockGroup { uint32_t count; uint32_t blocks[ADDRESSES_PER_BLOCK]; } FreeBlockGroup; ","date":"29-Nov-2024","heading":"12.7.3 Grouping","objectID":"\/posts\/operating-system\/:12:7","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1273-grouping"},{"categories":["computer-science"],"collections":null,"content":"12.8 Disk Layout Examples ext4 Filesystem Layout \/* * ext4 (Fourth Extended Filesystem) - Linux default * * Structure: * [Boot Block][Block Group 0][Block Group 1]...[Block Group N] * * Each Block Group: * - Superblock (backup in some groups) * - Group Descriptors * - Block Bitmap * - Inode Bitmap * - Inode Table * - Data Blocks *\/ typedef struct ext4_superblock { uint32_t s_inodes_count; \/\/ Total inodes uint32_t s_blocks_count; \/\/ Total blocks uint32_t s_r_blocks_count; \/\/ Reserved blocks uint32_t s_free_blocks_count; \/\/ Free blocks uint32_t s_free_inodes_count; \/\/ Free inodes uint32_t s_first_data_block; \/\/ First data block uint32_t s_log_block_size; \/\/ Block size = 1024 << s_log_block_size uint32_t s_blocks_per_group; \/\/ Blocks per group uint32_t s_inodes_per_group; \/\/ Inodes per group uint32_t s_magic; \/\/ Magic signature (0xEF53) \/\/ ... many more fields } ext4_superblock_t; #define EXT4_SUPER_MAGIC 0xEF53 ","date":"29-Nov-2024","heading":"12.8 Disk Layout Examples","objectID":"\/posts\/operating-system\/:12:8","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#128-disk-layout-examples"},{"categories":["computer-science"],"collections":null,"content":"12.8 Disk Layout Examples ext4 Filesystem Layout \/* * ext4 (Fourth Extended Filesystem) - Linux default * * Structure: * [Boot Block][Block Group 0][Block Group 1]...[Block Group N] * * Each Block Group: * - Superblock (backup in some groups) * - Group Descriptors * - Block Bitmap * - Inode Bitmap * - Inode Table * - Data Blocks *\/ typedef struct ext4_superblock { uint32_t s_inodes_count; \/\/ Total inodes uint32_t s_blocks_count; \/\/ Total blocks uint32_t s_r_blocks_count; \/\/ Reserved blocks uint32_t s_free_blocks_count; \/\/ Free blocks uint32_t s_free_inodes_count; \/\/ Free inodes uint32_t s_first_data_block; \/\/ First data block uint32_t s_log_block_size; \/\/ Block size = 1024 << s_log_block_size uint32_t s_blocks_per_group; \/\/ Blocks per group uint32_t s_inodes_per_group; \/\/ Inodes per group uint32_t s_magic; \/\/ Magic signature (0xEF53) \/\/ ... many more fields } ext4_superblock_t; #define EXT4_SUPER_MAGIC 0xEF53 ","date":"29-Nov-2024","heading":"ext4 Filesystem Layout","objectID":"\/posts\/operating-system\/:12:8","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#ext4-filesystem-layout"},{"categories":["computer-science"],"collections":null,"content":"12.9 File System Operations Implementation \/* * Create a new file *\/ int create_file(const char *path, mode_t mode) { \/\/ 1. Parse path to find parent directory char parent_path[PATH_MAX]; char filename[256]; \/\/ split_path(path, parent_path, filename); \/\/ 2. Allocate new inode uint32_t new_inode_num = allocate_inode(); if (new_inode_num == UINT32_MAX) { return -ENOSPC; \/\/ No space } \/\/ 3. Initialize inode Inode *inode = get_inode(new_inode_num); inode->mode = mode | S_IFREG; \/\/ Regular file inode->uid = getuid(); inode->gid = getgid(); inode->size = 0; inode->atime = inode->mtime = inode->ctime = time(NULL); inode->links_count = 1; inode->blocks = 0; \/\/ 4. Add directory entry in parent DirEntry entry = { .inode_number = new_inode_num, .name_len = strlen(filename), .file_type = DT_REG, }; strcpy(entry.name, filename); \/\/ add_dir_entry(parent_path, &entry); printf(\"Created file: %s (inode %d)\\n\", path, new_inode_num); return 0; } \/* * Delete a file *\/ int delete_file(const char *path) { \/\/ 1. Find inode uint32_t inode_num = path_to_inode(path); if (inode_num == UINT32_MAX) { return -ENOENT; \/\/ File not found } Inode *inode = get_inode(inode_num); \/\/ 2. Decrement link count inode->links_count--; if (inode->links_count == 0) { \/\/ 3. Free data blocks for (int i = 0; i < 12 && inode->block[i] != 0; i++) { free_block(&free_list, inode->block[i]); } \/\/ Handle indirect blocks... \/\/ 4. Free inode free_inode(inode_num); printf(\"Deleted file: %s\\n\", path); } else { printf(\"Decremented link count for: %s (links remaining: %d)\\n\", path, inode->links_count); } \/\/ 5. Remove directory entry \/\/ remove_dir_entry(path); return 0; } ","date":"29-Nov-2024","heading":"12.9 File System Operations Implementation","objectID":"\/posts\/operating-system\/:12:9","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#129-file-system-operations-implementation"},{"categories":["computer-science"],"collections":null,"content":"12.10 File Permissions (Unix\/Linux) \/* * Unix File Permissions: * 9 bits: rwx rwx rwx (owner, group, others) * 3 special bits: setuid, setgid, sticky *\/ \/\/ Permission bits #define S_IRUSR 0400 \/\/ User read #define S_IWUSR 0200 \/\/ User write #define S_IXUSR 0100 \/\/ User execute #define S_IRGRP 0040 \/\/ Group read #define S_IWGRP 0020 \/\/ Group write #define S_IXGRP 0010 \/\/ Group execute #define S_IROTH 0004 \/\/ Others read #define S_IWOTH 0002 \/\/ Others write #define S_IXOTH 0001 \/\/ Others execute \/\/ Check permissions bool can_access(Inode *inode, uid_t uid, gid_t gid, int required_perms) { mode_t mode = inode->mode; \/\/ Root can do anything if (uid == 0) { return true; } \/\/ Check owner permissions if (uid == inode->uid) { mode >>= 6; \/\/ Shift to owner bits } \/\/ Check group permissions else if (gid == inode->gid) { mode >>= 3; \/\/ Shift to group bits } \/\/ Others permissions (no shift needed) return (mode & required_perms) == required_perms; } \/\/ Change permissions int chmod_file(const char *path, mode_t new_mode) { uint32_t inode_num = path_to_inode(path); Inode *inode = get_inode(inode_num); \/\/ Only owner or root can change permissions if (getuid() != 0 && getuid() != inode->uid) { return -EPERM; } inode->mode = (inode->mode & ~0777) | (new_mode & 0777); printf(\"Changed permissions of %s to %o\\n\", path, new_mode); return 0; } \/\/ Example usage chmod(\"file.txt\", 0644); \/\/ rw-r--r-- chmod(\"script.sh\", 0755); \/\/ rwxr-xr-x ","date":"29-Nov-2024","heading":"12.10 File Permissions (Unix\/Linux)","objectID":"\/posts\/operating-system\/:12:10","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1210-file-permissions-unixlinux"},{"categories":["computer-science"],"collections":null,"content":"12.11 Journaling File Systems \/* * Journaling: Log changes before applying them * Ensures consistency after crash * * Types: * 1. Data journaling: Log both metadata and data * 2. Metadata journaling: Log only metadata (most common) * 3. Ordered: Write data before metadata *\/ typedef struct JournalEntry { uint64_t transaction_id; uint32_t operation; \/\/ CREATE, DELETE, MODIFY uint32_t inode_num; time_t timestamp; \/\/ ... operation-specific data } JournalEntry; \/\/ Write operation with journaling void journaled_write(const char *path, const void *data, size_t size) { \/\/ 1. Begin transaction - write to journal JournalEntry entry = { .transaction_id = get_next_txn_id(), .operation = OP_WRITE, .timestamp = time(NULL), }; write_to_journal(&entry); \/\/ 2. Write data to journal write_data_to_journal(data, size); \/\/ 3. Commit transaction in journal commit_journal_transaction(entry.transaction_id); \/\/ 4. Apply changes to filesystem (checkpoint) apply_to_filesystem(path, data, size); \/\/ 5. Mark transaction as complete in journal complete_journal_transaction(entry.transaction_id); printf(\"Journaled write completed\\n\"); } \/* * Recovery after crash: * 1. Scan journal for incomplete transactions * 2. Replay committed transactions * 3. Discard uncommitted transactions *\/ \/\/ Used in: ext4, XFS, NTFS, HFS+ ","date":"29-Nov-2024","heading":"12.11 Journaling File Systems","objectID":"\/posts\/operating-system\/:12:11","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1211-journaling-file-systems"},{"categories":["computer-science"],"collections":null,"content":"12.12 File System Performance \/* * Performance Optimizations: * * 1. Block Caching (Buffer Cache) * 2. Read-ahead (Prefetching) * 3. Write-behind (Delayed writes) * 4. File System Check (fsck) minimization via journaling *\/ \/\/ Simple cache implementation #define CACHE_SIZE 1024 typedef struct CacheEntry { uint32_t block_num; uint8_t data[BLOCK_SIZE]; bool dirty; time_t last_access; } CacheEntry; CacheEntry cache[CACHE_SIZE]; uint8_t* read_block_cached(uint32_t block_num) { \/\/ Check cache first for (int i = 0; i < CACHE_SIZE; i++) { if (cache[i].block_num == block_num) { cache[i].last_access = time(NULL); printf(\"Cache HIT: block %d\\n\", block_num); return cache[i].data; } } \/\/ Cache miss - read from disk printf(\"Cache MISS: block %d\\n\", block_num); \/\/ Find LRU entry to evict int lru_idx = find_lru_cache_entry(); \/\/ Write back if dirty if (cache[lru_idx].dirty) { write_block_to_disk(cache[lru_idx].block_num, cache[lru_idx].data); } \/\/ Load new block read_block_from_disk(block_num, cache[lru_idx].data); cache[lru_idx].block_num = block_num; cache[lru_idx].dirty = false; cache[lru_idx].last_access = time(NULL); return cache[lru_idx].data; } ","date":"29-Nov-2024","heading":"12.12 File System Performance","objectID":"\/posts\/operating-system\/:12:12","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1212-file-system-performance"},{"categories":["computer-science"],"collections":null,"content":"Key Takeaways Note File System Essentials: File systems organize data on persistent storage with hierarchical structure \u2192 Essential for data management Unix inode separates metadata from filename (allows hard links) \u2192 Enables flexible file organization Allocation methods: Contiguous (fast but fragmented), Linked (flexible but slow), Indexed (balanced) \u2192 Choose based on access patterns VFS provides uniform interface for different filesystem types \u2192 Mount ext4, FAT32, NFS seamlessly Free space management: Bitmap (fast), Linked list (space-efficient), Grouping (hybrid) \u2192 ext4 uses bitmaps Unix permissions: 9 bits (rwx for owner\/group\/others)** \u2192 Foundational security model Journaling ensures filesystem consistency after crashes \u2192 Metadata or full journaling Caching critical for performance (buffer cache, read-ahead) \u2192 Reduces disk I\/O dramatically ","date":"29-Nov-2024","heading":"Key Takeaways","objectID":"\/posts\/operating-system\/:12:13","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#key-takeaways-3"},{"categories":["computer-science"],"collections":null,"content":"13. I\/O Management ","date":"29-Nov-2024","heading":"13. I\/O Management","objectID":"\/posts\/operating-system\/:13:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#io-management"},{"categories":["computer-science"],"collections":null,"content":"13.1 I\/O Subsystem Overview The I\/O subsystem manages communication between CPU and peripheral devices. \/* * I\/O Components: * 1. CPU: Initiates I\/O operations * 2. Controller: Device-specific interface * 3. Device: Physical peripheral * 4. Bus\/Channel: Communication pathway *\/ typedef struct { uint8_t status; \/\/ Device status uint16_t control; \/\/ Control signals uint32_t data_register; \/\/ Data buffer } IODevice; \/* * Device Controller: * - Handles protocol specifics * - Manages data transfers * - Signals completion to CPU *\/ typedef struct { uint16_t status_register; \/\/ Device status uint16_t control_register; \/\/ Commands uint32_t data_port; \/\/ Data transfers uint32_t address_register; \/\/ Memory address for DMA } DeviceController; ","date":"29-Nov-2024","heading":"13.1 I\/O Subsystem Overview","objectID":"\/posts\/operating-system\/:13:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#131-io-subsystem-overview"},{"categories":["computer-science"],"collections":null,"content":"13.2 I\/O Techniques 13.2.1 Programmed I\/O (Polling) \/* * Programmed I\/O (Polling): * CPU continuously checks device status * * Pros: Simple implementation * Cons: CPU wastes cycles waiting (busy-waiting) *\/ #define DEVICE_STATUS 0x1000 #define DEVICE_DATA 0x1004 #define DEVICE_BUSY 0x01 #define DEVICE_READY 0x02 \/\/ Polling approach void read_data_polling() { \/\/ Wait for device to be ready while ((inb(DEVICE_STATUS) & DEVICE_READY) == 0) { \/\/ Busy wait - CPU does nothing useful } \/\/ Read data uint8_t data = inb(DEVICE_DATA); printf(\"Read: 0x%02X\\n\", data); } void write_data_polling(uint8_t data) { \/\/ Wait for device to not be busy while ((inb(DEVICE_STATUS) & DEVICE_BUSY) != 0) { \/\/ Busy wait } \/\/ Write data outb(DEVICE_DATA, data); printf(\"Wrote: 0x%02X\\n\", data); } \/* * Polling pseudocode: * while (device_busy) ; \/\/ Wait (wastes CPU cycles) * read_data() \/\/ Get data from port *\/ 13.2.2 Interrupt-Driven I\/O \/* * Interrupt-Driven I\/O: * Device signals CPU when ready (via interrupt) * CPU context switches to handler * * Pros: CPU not wasting cycles * Cons: Higher overhead per interrupt *\/ volatile uint8_t data_received = 0; \/\/ Interrupt handler (Interrupt Service Routine - ISR) void device_interrupt_handler() { printf(\"Interrupt received!\\n\"); \/\/ Read data from device data_received = inb(DEVICE_DATA); \/\/ Clear interrupt flag outb(DEVICE_CONTROL, 0x00); printf(\"Data: 0x%02X\\n\", data_received); \/\/ Return to interrupted code } void read_data_interrupt() { \/\/ Enable interrupts for this device outb(DEVICE_CONTROL, 0x01); \/\/ Wait for data (using sleep or event) sleep(5); \/\/ Timeout after 5 seconds \/\/ Data now available in data_received printf(\"Data available: 0x%02X\\n\", data_received); } \/* * Interrupt Processing: * 1. Device signals interrupt * 2. CPU saves current state * 3. CPU jumps to interrupt handler * 4. Handler processes device * 5. Handler clears interrupt * 6. CPU restores state and continues *\/ 13.2.3 Direct Memory Access (DMA) \/* * DMA: Device transfers data directly to memory * CPU not involved in each byte transfer * * Pros: Minimal CPU overhead, fast transfers * Cons: Complex implementation, requires DMA controller *\/ typedef struct { uint32_t source_address; \/\/ Device address uint32_t dest_address; \/\/ Memory address uint32_t transfer_count; \/\/ Number of bytes uint8_t direction; \/\/ Read\/Write uint8_t control; \/\/ Enable, interrupt on complete } DMARequest; \/\/ DMA Channel descriptor typedef struct { uint32_t memory_address; uint32_t transfer_count; uint8_t control; \/\/ Enable, interrupt flag uint8_t status; \/\/ Complete flag } DMAChannel; \/\/ Initialize DMA transfer void setup_dma_read(uint32_t device_addr, uint32_t mem_addr, uint32_t bytes) { \/\/ Set up DMA channel DMAChannel *dma = (DMAChannel *)0x8000; \/\/ DMA controller base address dma->memory_address = mem_addr; dma->transfer_count = bytes; dma->control = 0x01; \/\/ Enable DMA \/\/ Device starts transfer, DMA controller handles it printf(\"DMA transfer initiated: %d bytes from 0x%X to 0x%X\\n\", bytes, device_addr, mem_addr); } \/\/ DMA completion interrupt void dma_completion_handler() { DMAChannel *dma = (DMAChannel *)0x8000; if (dma->status & 0x01) { printf(\"DMA transfer complete!\\n\"); \/\/ Process transferred data... } } \/* * DMA Advantages: * - CPU free to do other work * - Fast bulk transfers * - Multiple DMA channels for concurrent transfers *\/ ","date":"29-Nov-2024","heading":"13.2 I\/O Techniques","objectID":"\/posts\/operating-system\/:13:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#132-io-techniques"},{"categories":["computer-science"],"collections":null,"content":"13.2 I\/O Techniques 13.2.1 Programmed I\/O (Polling) \/* * Programmed I\/O (Polling): * CPU continuously checks device status * * Pros: Simple implementation * Cons: CPU wastes cycles waiting (busy-waiting) *\/ #define DEVICE_STATUS 0x1000 #define DEVICE_DATA 0x1004 #define DEVICE_BUSY 0x01 #define DEVICE_READY 0x02 \/\/ Polling approach void read_data_polling() { \/\/ Wait for device to be ready while ((inb(DEVICE_STATUS) & DEVICE_READY) == 0) { \/\/ Busy wait - CPU does nothing useful } \/\/ Read data uint8_t data = inb(DEVICE_DATA); printf(\"Read: 0x%02X\\n\", data); } void write_data_polling(uint8_t data) { \/\/ Wait for device to not be busy while ((inb(DEVICE_STATUS) & DEVICE_BUSY) != 0) { \/\/ Busy wait } \/\/ Write data outb(DEVICE_DATA, data); printf(\"Wrote: 0x%02X\\n\", data); } \/* * Polling pseudocode: * while (device_busy) ; \/\/ Wait (wastes CPU cycles) * read_data() \/\/ Get data from port *\/ 13.2.2 Interrupt-Driven I\/O \/* * Interrupt-Driven I\/O: * Device signals CPU when ready (via interrupt) * CPU context switches to handler * * Pros: CPU not wasting cycles * Cons: Higher overhead per interrupt *\/ volatile uint8_t data_received = 0; \/\/ Interrupt handler (Interrupt Service Routine - ISR) void device_interrupt_handler() { printf(\"Interrupt received!\\n\"); \/\/ Read data from device data_received = inb(DEVICE_DATA); \/\/ Clear interrupt flag outb(DEVICE_CONTROL, 0x00); printf(\"Data: 0x%02X\\n\", data_received); \/\/ Return to interrupted code } void read_data_interrupt() { \/\/ Enable interrupts for this device outb(DEVICE_CONTROL, 0x01); \/\/ Wait for data (using sleep or event) sleep(5); \/\/ Timeout after 5 seconds \/\/ Data now available in data_received printf(\"Data available: 0x%02X\\n\", data_received); } \/* * Interrupt Processing: * 1. Device signals interrupt * 2. CPU saves current state * 3. CPU jumps to interrupt handler * 4. Handler processes device * 5. Handler clears interrupt * 6. CPU restores state and continues *\/ 13.2.3 Direct Memory Access (DMA) \/* * DMA: Device transfers data directly to memory * CPU not involved in each byte transfer * * Pros: Minimal CPU overhead, fast transfers * Cons: Complex implementation, requires DMA controller *\/ typedef struct { uint32_t source_address; \/\/ Device address uint32_t dest_address; \/\/ Memory address uint32_t transfer_count; \/\/ Number of bytes uint8_t direction; \/\/ Read\/Write uint8_t control; \/\/ Enable, interrupt on complete } DMARequest; \/\/ DMA Channel descriptor typedef struct { uint32_t memory_address; uint32_t transfer_count; uint8_t control; \/\/ Enable, interrupt flag uint8_t status; \/\/ Complete flag } DMAChannel; \/\/ Initialize DMA transfer void setup_dma_read(uint32_t device_addr, uint32_t mem_addr, uint32_t bytes) { \/\/ Set up DMA channel DMAChannel *dma = (DMAChannel *)0x8000; \/\/ DMA controller base address dma->memory_address = mem_addr; dma->transfer_count = bytes; dma->control = 0x01; \/\/ Enable DMA \/\/ Device starts transfer, DMA controller handles it printf(\"DMA transfer initiated: %d bytes from 0x%X to 0x%X\\n\", bytes, device_addr, mem_addr); } \/\/ DMA completion interrupt void dma_completion_handler() { DMAChannel *dma = (DMAChannel *)0x8000; if (dma->status & 0x01) { printf(\"DMA transfer complete!\\n\"); \/\/ Process transferred data... } } \/* * DMA Advantages: * - CPU free to do other work * - Fast bulk transfers * - Multiple DMA channels for concurrent transfers *\/ ","date":"29-Nov-2024","heading":"13.2.1 Programmed I\/O (Polling)","objectID":"\/posts\/operating-system\/:13:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1321-programmed-io-polling"},{"categories":["computer-science"],"collections":null,"content":"13.2 I\/O Techniques 13.2.1 Programmed I\/O (Polling) \/* * Programmed I\/O (Polling): * CPU continuously checks device status * * Pros: Simple implementation * Cons: CPU wastes cycles waiting (busy-waiting) *\/ #define DEVICE_STATUS 0x1000 #define DEVICE_DATA 0x1004 #define DEVICE_BUSY 0x01 #define DEVICE_READY 0x02 \/\/ Polling approach void read_data_polling() { \/\/ Wait for device to be ready while ((inb(DEVICE_STATUS) & DEVICE_READY) == 0) { \/\/ Busy wait - CPU does nothing useful } \/\/ Read data uint8_t data = inb(DEVICE_DATA); printf(\"Read: 0x%02X\\n\", data); } void write_data_polling(uint8_t data) { \/\/ Wait for device to not be busy while ((inb(DEVICE_STATUS) & DEVICE_BUSY) != 0) { \/\/ Busy wait } \/\/ Write data outb(DEVICE_DATA, data); printf(\"Wrote: 0x%02X\\n\", data); } \/* * Polling pseudocode: * while (device_busy) ; \/\/ Wait (wastes CPU cycles) * read_data() \/\/ Get data from port *\/ 13.2.2 Interrupt-Driven I\/O \/* * Interrupt-Driven I\/O: * Device signals CPU when ready (via interrupt) * CPU context switches to handler * * Pros: CPU not wasting cycles * Cons: Higher overhead per interrupt *\/ volatile uint8_t data_received = 0; \/\/ Interrupt handler (Interrupt Service Routine - ISR) void device_interrupt_handler() { printf(\"Interrupt received!\\n\"); \/\/ Read data from device data_received = inb(DEVICE_DATA); \/\/ Clear interrupt flag outb(DEVICE_CONTROL, 0x00); printf(\"Data: 0x%02X\\n\", data_received); \/\/ Return to interrupted code } void read_data_interrupt() { \/\/ Enable interrupts for this device outb(DEVICE_CONTROL, 0x01); \/\/ Wait for data (using sleep or event) sleep(5); \/\/ Timeout after 5 seconds \/\/ Data now available in data_received printf(\"Data available: 0x%02X\\n\", data_received); } \/* * Interrupt Processing: * 1. Device signals interrupt * 2. CPU saves current state * 3. CPU jumps to interrupt handler * 4. Handler processes device * 5. Handler clears interrupt * 6. CPU restores state and continues *\/ 13.2.3 Direct Memory Access (DMA) \/* * DMA: Device transfers data directly to memory * CPU not involved in each byte transfer * * Pros: Minimal CPU overhead, fast transfers * Cons: Complex implementation, requires DMA controller *\/ typedef struct { uint32_t source_address; \/\/ Device address uint32_t dest_address; \/\/ Memory address uint32_t transfer_count; \/\/ Number of bytes uint8_t direction; \/\/ Read\/Write uint8_t control; \/\/ Enable, interrupt on complete } DMARequest; \/\/ DMA Channel descriptor typedef struct { uint32_t memory_address; uint32_t transfer_count; uint8_t control; \/\/ Enable, interrupt flag uint8_t status; \/\/ Complete flag } DMAChannel; \/\/ Initialize DMA transfer void setup_dma_read(uint32_t device_addr, uint32_t mem_addr, uint32_t bytes) { \/\/ Set up DMA channel DMAChannel *dma = (DMAChannel *)0x8000; \/\/ DMA controller base address dma->memory_address = mem_addr; dma->transfer_count = bytes; dma->control = 0x01; \/\/ Enable DMA \/\/ Device starts transfer, DMA controller handles it printf(\"DMA transfer initiated: %d bytes from 0x%X to 0x%X\\n\", bytes, device_addr, mem_addr); } \/\/ DMA completion interrupt void dma_completion_handler() { DMAChannel *dma = (DMAChannel *)0x8000; if (dma->status & 0x01) { printf(\"DMA transfer complete!\\n\"); \/\/ Process transferred data... } } \/* * DMA Advantages: * - CPU free to do other work * - Fast bulk transfers * - Multiple DMA channels for concurrent transfers *\/ ","date":"29-Nov-2024","heading":"13.2.2 Interrupt-Driven I\/O","objectID":"\/posts\/operating-system\/:13:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1322-interrupt-driven-io"},{"categories":["computer-science"],"collections":null,"content":"13.2 I\/O Techniques 13.2.1 Programmed I\/O (Polling) \/* * Programmed I\/O (Polling): * CPU continuously checks device status * * Pros: Simple implementation * Cons: CPU wastes cycles waiting (busy-waiting) *\/ #define DEVICE_STATUS 0x1000 #define DEVICE_DATA 0x1004 #define DEVICE_BUSY 0x01 #define DEVICE_READY 0x02 \/\/ Polling approach void read_data_polling() { \/\/ Wait for device to be ready while ((inb(DEVICE_STATUS) & DEVICE_READY) == 0) { \/\/ Busy wait - CPU does nothing useful } \/\/ Read data uint8_t data = inb(DEVICE_DATA); printf(\"Read: 0x%02X\\n\", data); } void write_data_polling(uint8_t data) { \/\/ Wait for device to not be busy while ((inb(DEVICE_STATUS) & DEVICE_BUSY) != 0) { \/\/ Busy wait } \/\/ Write data outb(DEVICE_DATA, data); printf(\"Wrote: 0x%02X\\n\", data); } \/* * Polling pseudocode: * while (device_busy) ; \/\/ Wait (wastes CPU cycles) * read_data() \/\/ Get data from port *\/ 13.2.2 Interrupt-Driven I\/O \/* * Interrupt-Driven I\/O: * Device signals CPU when ready (via interrupt) * CPU context switches to handler * * Pros: CPU not wasting cycles * Cons: Higher overhead per interrupt *\/ volatile uint8_t data_received = 0; \/\/ Interrupt handler (Interrupt Service Routine - ISR) void device_interrupt_handler() { printf(\"Interrupt received!\\n\"); \/\/ Read data from device data_received = inb(DEVICE_DATA); \/\/ Clear interrupt flag outb(DEVICE_CONTROL, 0x00); printf(\"Data: 0x%02X\\n\", data_received); \/\/ Return to interrupted code } void read_data_interrupt() { \/\/ Enable interrupts for this device outb(DEVICE_CONTROL, 0x01); \/\/ Wait for data (using sleep or event) sleep(5); \/\/ Timeout after 5 seconds \/\/ Data now available in data_received printf(\"Data available: 0x%02X\\n\", data_received); } \/* * Interrupt Processing: * 1. Device signals interrupt * 2. CPU saves current state * 3. CPU jumps to interrupt handler * 4. Handler processes device * 5. Handler clears interrupt * 6. CPU restores state and continues *\/ 13.2.3 Direct Memory Access (DMA) \/* * DMA: Device transfers data directly to memory * CPU not involved in each byte transfer * * Pros: Minimal CPU overhead, fast transfers * Cons: Complex implementation, requires DMA controller *\/ typedef struct { uint32_t source_address; \/\/ Device address uint32_t dest_address; \/\/ Memory address uint32_t transfer_count; \/\/ Number of bytes uint8_t direction; \/\/ Read\/Write uint8_t control; \/\/ Enable, interrupt on complete } DMARequest; \/\/ DMA Channel descriptor typedef struct { uint32_t memory_address; uint32_t transfer_count; uint8_t control; \/\/ Enable, interrupt flag uint8_t status; \/\/ Complete flag } DMAChannel; \/\/ Initialize DMA transfer void setup_dma_read(uint32_t device_addr, uint32_t mem_addr, uint32_t bytes) { \/\/ Set up DMA channel DMAChannel *dma = (DMAChannel *)0x8000; \/\/ DMA controller base address dma->memory_address = mem_addr; dma->transfer_count = bytes; dma->control = 0x01; \/\/ Enable DMA \/\/ Device starts transfer, DMA controller handles it printf(\"DMA transfer initiated: %d bytes from 0x%X to 0x%X\\n\", bytes, device_addr, mem_addr); } \/\/ DMA completion interrupt void dma_completion_handler() { DMAChannel *dma = (DMAChannel *)0x8000; if (dma->status & 0x01) { printf(\"DMA transfer complete!\\n\"); \/\/ Process transferred data... } } \/* * DMA Advantages: * - CPU free to do other work * - Fast bulk transfers * - Multiple DMA channels for concurrent transfers *\/ ","date":"29-Nov-2024","heading":"13.2.3 Direct Memory Access (DMA)","objectID":"\/posts\/operating-system\/:13:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1323-direct-memory-access-dma"},{"categories":["computer-science"],"collections":null,"content":"13.3 Disk Scheduling Algorithms When multiple I\/O requests queued, choose order to minimize seek time. 13.3.1 FCFS (First-Come-First-Served) \/* * FCFS: Service requests in order received * Pros: Fair, simple * Cons: High average seek time (disk may jump around) *\/ #define DISK_TRACKS 200 #define QUEUE_SIZE 32 typedef struct { int track; int id; } DiskRequest; void disk_schedule_fcfs(DiskRequest queue[], int queue_size) { int current_track = 0; int total_seek = 0; printf(\"=== FCFS Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) { int seek = abs(queue[i].track - current_track); total_seek += seek; printf(\"Request %d: Track %d (seek: %d)\\n\", queue[i].id, queue[i].track, seek); current_track = queue[i].track; } printf(\"Total head movement: %d tracks\\n\", total_seek); } \/\/ Example: Queue = [98, 183, 37, 122, 14, 165, 30, 85] \/\/ FCFS movement: 0->98 (98) ->183 (85) ->37 (146) ->122 (85) ->14 (108) ->165 (151) ->30 (135) ->85 (55) = 863 13.3.2 SSTF (Shortest Seek Time First) \/* * SSTF: Serve closest request next * Pros: Lower seek time than FCFS * Cons: Starvation possible (outer tracks if inner always busy) *\/ void disk_schedule_sstf(DiskRequest queue[], int queue_size) { bool served[QUEUE_SIZE] = {false}; int current_track = 0; int total_seek = 0; printf(\"=== SSTF Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) { \/\/ Find closest unserved request int next_idx = -1; int min_seek = INT_MAX; for (int j = 0; j < queue_size; j++) { if (!served[j]) { int seek = abs(queue[j].track - current_track); if (seek < min_seek) { min_seek = seek; next_idx = j; } } } served[next_idx] = true; total_seek += min_seek; printf(\"Request %d: Track %d (seek: %d)\\n\", queue[next_idx].id, queue[next_idx].track, min_seek); current_track = queue[next_idx].track; } printf(\"Total head movement: %d tracks\\n\", total_seek); } \/\/ Same queue: [98, 183, 37, 122, 14, 165, 30, 85] \/\/ SSTF movement: 0->30 (30) ->37 (7) ->85 (48) ->98 (13) ->122 (24) ->165 (43) ->183 (18) ->14 (169) = 352 13.3.3 SCAN (Elevator Algorithm) \/* * SCAN: Head moves in one direction, serves requests, then reverses * Pros: Fair, prevents starvation, good for heavy load * Cons: Slightly higher seek time than SSTF *\/ void disk_schedule_scan(DiskRequest queue[], int queue_size) { bool served[QUEUE_SIZE] = {false}; int current_track = 0; int total_seek = 0; int direction = 1; \/\/ 1 = towards higher tracks, -1 = towards lower printf(\"=== SCAN (Elevator) Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) { \/\/ Find closest request in current direction int next_idx = -1; int best_track = -1; for (int j = 0; j < queue_size; j++) { if (!served[j]) { int track = queue[j].track; \/\/ Check if in current direction if ((direction > 0 && track >= current_track) || (direction < 0 && track <= current_track)) { \/\/ Find closest in direction if (best_track == -1 || (direction > 0 && track < best_track) || (direction < 0 && track > best_track)) { best_track = track; next_idx = j; } } } } \/\/ If no request in current direction, reverse if (next_idx == -1) { direction *= -1; \/\/ Reverse direction \/\/ Find closest in new direction for (int j = 0; j < queue_size; j++) { if (!served[j]) { best_track = queue[j].track; next_idx = j; break; } } } served[next_idx] = true; int seek = abs(queue[next_idx].track - current_track); total_seek += seek; printf(\"Request %d: Track %d (seek: %d, direction: %s)\\n\", queue[next_idx].id, queue[next_idx].track, seek, direction > 0 ? \"up\" : \"down\"); current_track = queue[next_idx].track; } printf(\"Total head movement: %d tracks\\n\", total_seek); } 13.3.4 C-SCAN (Circular SCAN) \/* * C-SCAN: After reaching end, jump to start and scan again * Pros: More uniform seek time distribution * Cons: Higher seek time for first few tracks after reversal *\/ void disk_schedule_c_scan(DiskRequest queue[], int queue_size) { bool served[QUEUE_SIZE] = {false}; int current_track = 0; int total_seek = 0; printf(\"=== C-SCAN Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) ","date":"29-Nov-2024","heading":"13.3 Disk Scheduling Algorithms","objectID":"\/posts\/operating-system\/:13:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#133-disk-scheduling-algorithms"},{"categories":["computer-science"],"collections":null,"content":"13.3 Disk Scheduling Algorithms When multiple I\/O requests queued, choose order to minimize seek time. 13.3.1 FCFS (First-Come-First-Served) \/* * FCFS: Service requests in order received * Pros: Fair, simple * Cons: High average seek time (disk may jump around) *\/ #define DISK_TRACKS 200 #define QUEUE_SIZE 32 typedef struct { int track; int id; } DiskRequest; void disk_schedule_fcfs(DiskRequest queue[], int queue_size) { int current_track = 0; int total_seek = 0; printf(\"=== FCFS Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) { int seek = abs(queue[i].track - current_track); total_seek += seek; printf(\"Request %d: Track %d (seek: %d)\\n\", queue[i].id, queue[i].track, seek); current_track = queue[i].track; } printf(\"Total head movement: %d tracks\\n\", total_seek); } \/\/ Example: Queue = [98, 183, 37, 122, 14, 165, 30, 85] \/\/ FCFS movement: 0->98 (98) ->183 (85) ->37 (146) ->122 (85) ->14 (108) ->165 (151) ->30 (135) ->85 (55) = 863 13.3.2 SSTF (Shortest Seek Time First) \/* * SSTF: Serve closest request next * Pros: Lower seek time than FCFS * Cons: Starvation possible (outer tracks if inner always busy) *\/ void disk_schedule_sstf(DiskRequest queue[], int queue_size) { bool served[QUEUE_SIZE] = {false}; int current_track = 0; int total_seek = 0; printf(\"=== SSTF Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) { \/\/ Find closest unserved request int next_idx = -1; int min_seek = INT_MAX; for (int j = 0; j < queue_size; j++) { if (!served[j]) { int seek = abs(queue[j].track - current_track); if (seek < min_seek) { min_seek = seek; next_idx = j; } } } served[next_idx] = true; total_seek += min_seek; printf(\"Request %d: Track %d (seek: %d)\\n\", queue[next_idx].id, queue[next_idx].track, min_seek); current_track = queue[next_idx].track; } printf(\"Total head movement: %d tracks\\n\", total_seek); } \/\/ Same queue: [98, 183, 37, 122, 14, 165, 30, 85] \/\/ SSTF movement: 0->30 (30) ->37 (7) ->85 (48) ->98 (13) ->122 (24) ->165 (43) ->183 (18) ->14 (169) = 352 13.3.3 SCAN (Elevator Algorithm) \/* * SCAN: Head moves in one direction, serves requests, then reverses * Pros: Fair, prevents starvation, good for heavy load * Cons: Slightly higher seek time than SSTF *\/ void disk_schedule_scan(DiskRequest queue[], int queue_size) { bool served[QUEUE_SIZE] = {false}; int current_track = 0; int total_seek = 0; int direction = 1; \/\/ 1 = towards higher tracks, -1 = towards lower printf(\"=== SCAN (Elevator) Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) { \/\/ Find closest request in current direction int next_idx = -1; int best_track = -1; for (int j = 0; j < queue_size; j++) { if (!served[j]) { int track = queue[j].track; \/\/ Check if in current direction if ((direction > 0 && track >= current_track) || (direction < 0 && track <= current_track)) { \/\/ Find closest in direction if (best_track == -1 || (direction > 0 && track < best_track) || (direction < 0 && track > best_track)) { best_track = track; next_idx = j; } } } } \/\/ If no request in current direction, reverse if (next_idx == -1) { direction *= -1; \/\/ Reverse direction \/\/ Find closest in new direction for (int j = 0; j < queue_size; j++) { if (!served[j]) { best_track = queue[j].track; next_idx = j; break; } } } served[next_idx] = true; int seek = abs(queue[next_idx].track - current_track); total_seek += seek; printf(\"Request %d: Track %d (seek: %d, direction: %s)\\n\", queue[next_idx].id, queue[next_idx].track, seek, direction > 0 ? \"up\" : \"down\"); current_track = queue[next_idx].track; } printf(\"Total head movement: %d tracks\\n\", total_seek); } 13.3.4 C-SCAN (Circular SCAN) \/* * C-SCAN: After reaching end, jump to start and scan again * Pros: More uniform seek time distribution * Cons: Higher seek time for first few tracks after reversal *\/ void disk_schedule_c_scan(DiskRequest queue[], int queue_size) { bool served[QUEUE_SIZE] = {false}; int current_track = 0; int total_seek = 0; printf(\"=== C-SCAN Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) ","date":"29-Nov-2024","heading":"13.3.1 FCFS (First-Come-First-Served)","objectID":"\/posts\/operating-system\/:13:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1331-fcfs-first-come-first-served"},{"categories":["computer-science"],"collections":null,"content":"13.3 Disk Scheduling Algorithms When multiple I\/O requests queued, choose order to minimize seek time. 13.3.1 FCFS (First-Come-First-Served) \/* * FCFS: Service requests in order received * Pros: Fair, simple * Cons: High average seek time (disk may jump around) *\/ #define DISK_TRACKS 200 #define QUEUE_SIZE 32 typedef struct { int track; int id; } DiskRequest; void disk_schedule_fcfs(DiskRequest queue[], int queue_size) { int current_track = 0; int total_seek = 0; printf(\"=== FCFS Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) { int seek = abs(queue[i].track - current_track); total_seek += seek; printf(\"Request %d: Track %d (seek: %d)\\n\", queue[i].id, queue[i].track, seek); current_track = queue[i].track; } printf(\"Total head movement: %d tracks\\n\", total_seek); } \/\/ Example: Queue = [98, 183, 37, 122, 14, 165, 30, 85] \/\/ FCFS movement: 0->98 (98) ->183 (85) ->37 (146) ->122 (85) ->14 (108) ->165 (151) ->30 (135) ->85 (55) = 863 13.3.2 SSTF (Shortest Seek Time First) \/* * SSTF: Serve closest request next * Pros: Lower seek time than FCFS * Cons: Starvation possible (outer tracks if inner always busy) *\/ void disk_schedule_sstf(DiskRequest queue[], int queue_size) { bool served[QUEUE_SIZE] = {false}; int current_track = 0; int total_seek = 0; printf(\"=== SSTF Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) { \/\/ Find closest unserved request int next_idx = -1; int min_seek = INT_MAX; for (int j = 0; j < queue_size; j++) { if (!served[j]) { int seek = abs(queue[j].track - current_track); if (seek < min_seek) { min_seek = seek; next_idx = j; } } } served[next_idx] = true; total_seek += min_seek; printf(\"Request %d: Track %d (seek: %d)\\n\", queue[next_idx].id, queue[next_idx].track, min_seek); current_track = queue[next_idx].track; } printf(\"Total head movement: %d tracks\\n\", total_seek); } \/\/ Same queue: [98, 183, 37, 122, 14, 165, 30, 85] \/\/ SSTF movement: 0->30 (30) ->37 (7) ->85 (48) ->98 (13) ->122 (24) ->165 (43) ->183 (18) ->14 (169) = 352 13.3.3 SCAN (Elevator Algorithm) \/* * SCAN: Head moves in one direction, serves requests, then reverses * Pros: Fair, prevents starvation, good for heavy load * Cons: Slightly higher seek time than SSTF *\/ void disk_schedule_scan(DiskRequest queue[], int queue_size) { bool served[QUEUE_SIZE] = {false}; int current_track = 0; int total_seek = 0; int direction = 1; \/\/ 1 = towards higher tracks, -1 = towards lower printf(\"=== SCAN (Elevator) Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) { \/\/ Find closest request in current direction int next_idx = -1; int best_track = -1; for (int j = 0; j < queue_size; j++) { if (!served[j]) { int track = queue[j].track; \/\/ Check if in current direction if ((direction > 0 && track >= current_track) || (direction < 0 && track <= current_track)) { \/\/ Find closest in direction if (best_track == -1 || (direction > 0 && track < best_track) || (direction < 0 && track > best_track)) { best_track = track; next_idx = j; } } } } \/\/ If no request in current direction, reverse if (next_idx == -1) { direction *= -1; \/\/ Reverse direction \/\/ Find closest in new direction for (int j = 0; j < queue_size; j++) { if (!served[j]) { best_track = queue[j].track; next_idx = j; break; } } } served[next_idx] = true; int seek = abs(queue[next_idx].track - current_track); total_seek += seek; printf(\"Request %d: Track %d (seek: %d, direction: %s)\\n\", queue[next_idx].id, queue[next_idx].track, seek, direction > 0 ? \"up\" : \"down\"); current_track = queue[next_idx].track; } printf(\"Total head movement: %d tracks\\n\", total_seek); } 13.3.4 C-SCAN (Circular SCAN) \/* * C-SCAN: After reaching end, jump to start and scan again * Pros: More uniform seek time distribution * Cons: Higher seek time for first few tracks after reversal *\/ void disk_schedule_c_scan(DiskRequest queue[], int queue_size) { bool served[QUEUE_SIZE] = {false}; int current_track = 0; int total_seek = 0; printf(\"=== C-SCAN Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) ","date":"29-Nov-2024","heading":"13.3.2 SSTF (Shortest Seek Time First)","objectID":"\/posts\/operating-system\/:13:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1332-sstf-shortest-seek-time-first"},{"categories":["computer-science"],"collections":null,"content":"13.3 Disk Scheduling Algorithms When multiple I\/O requests queued, choose order to minimize seek time. 13.3.1 FCFS (First-Come-First-Served) \/* * FCFS: Service requests in order received * Pros: Fair, simple * Cons: High average seek time (disk may jump around) *\/ #define DISK_TRACKS 200 #define QUEUE_SIZE 32 typedef struct { int track; int id; } DiskRequest; void disk_schedule_fcfs(DiskRequest queue[], int queue_size) { int current_track = 0; int total_seek = 0; printf(\"=== FCFS Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) { int seek = abs(queue[i].track - current_track); total_seek += seek; printf(\"Request %d: Track %d (seek: %d)\\n\", queue[i].id, queue[i].track, seek); current_track = queue[i].track; } printf(\"Total head movement: %d tracks\\n\", total_seek); } \/\/ Example: Queue = [98, 183, 37, 122, 14, 165, 30, 85] \/\/ FCFS movement: 0->98 (98) ->183 (85) ->37 (146) ->122 (85) ->14 (108) ->165 (151) ->30 (135) ->85 (55) = 863 13.3.2 SSTF (Shortest Seek Time First) \/* * SSTF: Serve closest request next * Pros: Lower seek time than FCFS * Cons: Starvation possible (outer tracks if inner always busy) *\/ void disk_schedule_sstf(DiskRequest queue[], int queue_size) { bool served[QUEUE_SIZE] = {false}; int current_track = 0; int total_seek = 0; printf(\"=== SSTF Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) { \/\/ Find closest unserved request int next_idx = -1; int min_seek = INT_MAX; for (int j = 0; j < queue_size; j++) { if (!served[j]) { int seek = abs(queue[j].track - current_track); if (seek < min_seek) { min_seek = seek; next_idx = j; } } } served[next_idx] = true; total_seek += min_seek; printf(\"Request %d: Track %d (seek: %d)\\n\", queue[next_idx].id, queue[next_idx].track, min_seek); current_track = queue[next_idx].track; } printf(\"Total head movement: %d tracks\\n\", total_seek); } \/\/ Same queue: [98, 183, 37, 122, 14, 165, 30, 85] \/\/ SSTF movement: 0->30 (30) ->37 (7) ->85 (48) ->98 (13) ->122 (24) ->165 (43) ->183 (18) ->14 (169) = 352 13.3.3 SCAN (Elevator Algorithm) \/* * SCAN: Head moves in one direction, serves requests, then reverses * Pros: Fair, prevents starvation, good for heavy load * Cons: Slightly higher seek time than SSTF *\/ void disk_schedule_scan(DiskRequest queue[], int queue_size) { bool served[QUEUE_SIZE] = {false}; int current_track = 0; int total_seek = 0; int direction = 1; \/\/ 1 = towards higher tracks, -1 = towards lower printf(\"=== SCAN (Elevator) Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) { \/\/ Find closest request in current direction int next_idx = -1; int best_track = -1; for (int j = 0; j < queue_size; j++) { if (!served[j]) { int track = queue[j].track; \/\/ Check if in current direction if ((direction > 0 && track >= current_track) || (direction < 0 && track <= current_track)) { \/\/ Find closest in direction if (best_track == -1 || (direction > 0 && track < best_track) || (direction < 0 && track > best_track)) { best_track = track; next_idx = j; } } } } \/\/ If no request in current direction, reverse if (next_idx == -1) { direction *= -1; \/\/ Reverse direction \/\/ Find closest in new direction for (int j = 0; j < queue_size; j++) { if (!served[j]) { best_track = queue[j].track; next_idx = j; break; } } } served[next_idx] = true; int seek = abs(queue[next_idx].track - current_track); total_seek += seek; printf(\"Request %d: Track %d (seek: %d, direction: %s)\\n\", queue[next_idx].id, queue[next_idx].track, seek, direction > 0 ? \"up\" : \"down\"); current_track = queue[next_idx].track; } printf(\"Total head movement: %d tracks\\n\", total_seek); } 13.3.4 C-SCAN (Circular SCAN) \/* * C-SCAN: After reaching end, jump to start and scan again * Pros: More uniform seek time distribution * Cons: Higher seek time for first few tracks after reversal *\/ void disk_schedule_c_scan(DiskRequest queue[], int queue_size) { bool served[QUEUE_SIZE] = {false}; int current_track = 0; int total_seek = 0; printf(\"=== C-SCAN Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) ","date":"29-Nov-2024","heading":"13.3.3 SCAN (Elevator Algorithm)","objectID":"\/posts\/operating-system\/:13:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1333-scan-elevator-algorithm"},{"categories":["computer-science"],"collections":null,"content":"13.3 Disk Scheduling Algorithms When multiple I\/O requests queued, choose order to minimize seek time. 13.3.1 FCFS (First-Come-First-Served) \/* * FCFS: Service requests in order received * Pros: Fair, simple * Cons: High average seek time (disk may jump around) *\/ #define DISK_TRACKS 200 #define QUEUE_SIZE 32 typedef struct { int track; int id; } DiskRequest; void disk_schedule_fcfs(DiskRequest queue[], int queue_size) { int current_track = 0; int total_seek = 0; printf(\"=== FCFS Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) { int seek = abs(queue[i].track - current_track); total_seek += seek; printf(\"Request %d: Track %d (seek: %d)\\n\", queue[i].id, queue[i].track, seek); current_track = queue[i].track; } printf(\"Total head movement: %d tracks\\n\", total_seek); } \/\/ Example: Queue = [98, 183, 37, 122, 14, 165, 30, 85] \/\/ FCFS movement: 0->98 (98) ->183 (85) ->37 (146) ->122 (85) ->14 (108) ->165 (151) ->30 (135) ->85 (55) = 863 13.3.2 SSTF (Shortest Seek Time First) \/* * SSTF: Serve closest request next * Pros: Lower seek time than FCFS * Cons: Starvation possible (outer tracks if inner always busy) *\/ void disk_schedule_sstf(DiskRequest queue[], int queue_size) { bool served[QUEUE_SIZE] = {false}; int current_track = 0; int total_seek = 0; printf(\"=== SSTF Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) { \/\/ Find closest unserved request int next_idx = -1; int min_seek = INT_MAX; for (int j = 0; j < queue_size; j++) { if (!served[j]) { int seek = abs(queue[j].track - current_track); if (seek < min_seek) { min_seek = seek; next_idx = j; } } } served[next_idx] = true; total_seek += min_seek; printf(\"Request %d: Track %d (seek: %d)\\n\", queue[next_idx].id, queue[next_idx].track, min_seek); current_track = queue[next_idx].track; } printf(\"Total head movement: %d tracks\\n\", total_seek); } \/\/ Same queue: [98, 183, 37, 122, 14, 165, 30, 85] \/\/ SSTF movement: 0->30 (30) ->37 (7) ->85 (48) ->98 (13) ->122 (24) ->165 (43) ->183 (18) ->14 (169) = 352 13.3.3 SCAN (Elevator Algorithm) \/* * SCAN: Head moves in one direction, serves requests, then reverses * Pros: Fair, prevents starvation, good for heavy load * Cons: Slightly higher seek time than SSTF *\/ void disk_schedule_scan(DiskRequest queue[], int queue_size) { bool served[QUEUE_SIZE] = {false}; int current_track = 0; int total_seek = 0; int direction = 1; \/\/ 1 = towards higher tracks, -1 = towards lower printf(\"=== SCAN (Elevator) Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) { \/\/ Find closest request in current direction int next_idx = -1; int best_track = -1; for (int j = 0; j < queue_size; j++) { if (!served[j]) { int track = queue[j].track; \/\/ Check if in current direction if ((direction > 0 && track >= current_track) || (direction < 0 && track <= current_track)) { \/\/ Find closest in direction if (best_track == -1 || (direction > 0 && track < best_track) || (direction < 0 && track > best_track)) { best_track = track; next_idx = j; } } } } \/\/ If no request in current direction, reverse if (next_idx == -1) { direction *= -1; \/\/ Reverse direction \/\/ Find closest in new direction for (int j = 0; j < queue_size; j++) { if (!served[j]) { best_track = queue[j].track; next_idx = j; break; } } } served[next_idx] = true; int seek = abs(queue[next_idx].track - current_track); total_seek += seek; printf(\"Request %d: Track %d (seek: %d, direction: %s)\\n\", queue[next_idx].id, queue[next_idx].track, seek, direction > 0 ? \"up\" : \"down\"); current_track = queue[next_idx].track; } printf(\"Total head movement: %d tracks\\n\", total_seek); } 13.3.4 C-SCAN (Circular SCAN) \/* * C-SCAN: After reaching end, jump to start and scan again * Pros: More uniform seek time distribution * Cons: Higher seek time for first few tracks after reversal *\/ void disk_schedule_c_scan(DiskRequest queue[], int queue_size) { bool served[QUEUE_SIZE] = {false}; int current_track = 0; int total_seek = 0; printf(\"=== C-SCAN Disk Scheduling ===\\n\"); for (int i = 0; i < queue_size; i++) ","date":"29-Nov-2024","heading":"13.3.4 C-SCAN (Circular SCAN)","objectID":"\/posts\/operating-system\/:13:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1334-c-scan-circular-scan"},{"categories":["computer-science"],"collections":null,"content":"13.4 Disk Scheduling Comparison Tip Disk Scheduling Trade-offs: FCFS: Fair but causes excessive head movement (thrashing) \u2192 Poor for workstations SSTF: Minimizes seeks but starves outer tracks \u2192 Risky for critical data SCAN\/C-SCAN: Balanced fairness and performance \u2192 Best for most systems (recommended) LOOK: Practical SCAN variant \u2192 Similar performance to SCAN Algorithm Avg Seek Time Starvation Implementation Notes FCFS High No Simple Fair but inefficient SSTF Low-Medium Yes Medium Good but can starve outer tracks SCAN Medium No Complex Fair and good under heavy load C-SCAN Medium No Complex More uniform service LOOK Low-Medium No Complex Like SCAN but doesn’t go to end ","date":"29-Nov-2024","heading":"13.4 Disk Scheduling Comparison","objectID":"\/posts\/operating-system\/:13:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#134-disk-scheduling-comparison"},{"categories":["computer-science"],"collections":null,"content":"13.5 Interrupt Handling \/* * Interrupt Levels (Interrupt Request - IRQ) * x86 PC: 16 IRQs (IRQ0-15) * - IRQ0: System timer * - IRQ1: Keyboard * - IRQ8: Real-time clock * - IRQ14-15: IDE disk controllers *\/ \/\/ Interrupt Descriptor Table (IDT) entry typedef struct { uint16_t offset_low; \/\/ Lower 16 bits of handler address uint16_t selector; \/\/ Code segment selector uint8_t reserved; uint8_t type; \/\/ Gate type (trap, interrupt) uint16_t offset_high; \/\/ Upper 16 bits of handler address } IDTEntry; #define IDT_SIZE 256 IDTEntry idt[IDT_SIZE]; \/\/ Register interrupt handler void register_interrupt_handler(int irq, void (*handler)(void)) { uint32_t handler_addr = (uint32_t)handler; idt[irq].offset_low = handler_addr & 0xFFFF; idt[irq].offset_high = (handler_addr >> 16) & 0xFFFF; idt[irq].selector = 0x08; \/\/ Kernel code segment idt[irq].type = 0x8E; \/\/ 32-bit interrupt gate printf(\"Registered handler for IRQ%d at 0x%X\\n\", irq, handler_addr); } \/\/ Interrupt context (saved CPU state) typedef struct { uint32_t edi, esi, ebp, esp; uint32_t ebx, edx, ecx, eax; uint32_t ds, es, fs, gs; uint32_t error_code; uint32_t eip; uint32_t cs; uint32_t eflags; } InterruptContext; \/\/ Generic interrupt handler wrapper void generic_interrupt_handler(InterruptContext *ctx) { printf(\"Interrupt at 0x%X, Error: 0x%X\\n\", ctx->eip, ctx->error_code); \/\/ Acknowledge interrupt to PIC (Programmable Interrupt Controller) outb(0x20, 0x20); \/\/ Send EOI to master PIC \/\/ Call appropriate handler based on interrupt number... } ","date":"29-Nov-2024","heading":"13.5 Interrupt Handling","objectID":"\/posts\/operating-system\/:13:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#135-interrupt-handling"},{"categories":["computer-science"],"collections":null,"content":"Key Takeaways Tip I\/O Management Principles: I\/O Techniques: Polling (simple, wasteful), Interrupts (efficient), DMA (best for bulk transfers)** \u2192 Use DMA for high throughput Disk Scheduling: FCFS (fair, high seek), SSTF (low seek, starvation), SCAN (fair, medium seek)** \u2192 SCAN is production standard Interrupt Handling: Device signals CPU via interrupt, handler processes request** \u2192 Minimize handler time Performance depends on workload and device characteristics \u2192 Profile before optimizing ","date":"29-Nov-2024","heading":"Key Takeaways","objectID":"\/posts\/operating-system\/:13:6","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#key-takeaways-4"},{"categories":["computer-science"],"collections":null,"content":"14. System Calls and Interrupts ","date":"29-Nov-2024","heading":"14. System Calls and Interrupts","objectID":"\/posts\/operating-system\/:14:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#system-calls"},{"categories":["computer-science"],"collections":null,"content":"14.1 System Calls Basics System calls are the interface between user programs and the OS kernel, providing controlled access to hardware and kernel services. \/* * System Call Execution Flow: * 1. User program calls system call wrapper * 2. Arguments loaded into CPU registers\/stack * 3. Trap instruction (int 0x80 on x86) * 4. CPU switches to kernel mode * 5. Kernel handles request * 6. Return to user mode with result * * Context Switch Cost: Very expensive! *\/ #include #include #include \/\/ System calls frequently used in Unix\/Linux int fd = open(\"file.txt\", O_RDONLY); \/\/ open() syscall ssize_t n = read(fd, buf, 100); \/\/ read() syscall ssize_t m = write(1, \"hello\", 5); \/\/ write() syscall close(fd); \/\/ close() syscall pid_t pid = fork(); \/\/ fork() syscall execve(\"\/bin\/ls\", argv, env); \/\/ execve() syscall int status; wait(&status); \/\/ wait() syscall ","date":"29-Nov-2024","heading":"14.1 System Calls Basics","objectID":"\/posts\/operating-system\/:14:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#141-system-calls-basics"},{"categories":["computer-science"],"collections":null,"content":"14.2 Mode Switching: User to Kernel \/* * User Mode vs Kernel Mode * * User Mode: * - Limited privileges * - Can't access hardware directly * - Can't modify memory protection * - Must use syscalls for privileged operations * * Kernel Mode: * - Full privileges * - Direct hardware access * - Can modify CPU registers * - Can access all memory *\/ \/\/ x86 Privilege Levels (Rings): \/\/ Ring 0: Kernel (highest privilege) \/\/ Ring 1, 2: Rarely used \/\/ Ring 3: User (lowest privilege) typedef struct { uint32_t eax, ebx, ecx, edx; uint32_t esi, edi, ebp, esp; uint32_t eip; \/\/ Instruction pointer uint32_t cs; \/\/ Code segment (determines privilege) uint32_t eflags; \/\/ Flags register uint32_t ss; \/\/ Stack segment } CPUState; \/* * Switching from User to Kernel: * 1. User executes \"int 0x80\" (software interrupt on x86) * 2. CPU changes to kernel mode * 3. Kernel trap handler takes over * 4. Kernel executes requested syscall * 5. \"iret\" instruction returns to user mode *\/ ","date":"29-Nov-2024","heading":"14.2 Mode Switching: User to Kernel","objectID":"\/posts\/operating-system\/:14:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#142-mode-switching-user-to-kernel"},{"categories":["computer-science"],"collections":null,"content":"14.3 Exception and Trap Handling \/* * Exceptions: Internal CPU events (errors, traps) * Examples: Division by zero, page fault, invalid opcode * * Traps: Intentional exceptions used for syscalls *\/ \/\/ x86 Exception Types #define EXC_DIVIDE_BY_ZERO 0 \/\/ Division by zero #define EXC_DEBUG 1 \/\/ Debugger breakpoint #define EXC_NMI 2 \/\/ Non-maskable interrupt #define EXC_BREAKPOINT 3 \/\/ Breakpoint trap #define EXC_OVERFLOW 4 \/\/ Signed overflow #define EXC_BOUNDS 5 \/\/ Bounds check #define EXC_INVALID_OP 6 \/\/ Invalid opcode #define EXC_NO_MATH_COPROC 7 \/\/ Math coprocessor not available #define EXC_DOUBLE_FAULT 8 \/\/ Exception during exception #define EXC_SEGMENT_OVERRUN 9 \/\/ Segment overrun #define EXC_BAD_TSS 10 \/\/ Invalid Task State Segment #define EXC_SEGMENT_NOT_PRES 11 \/\/ Segment not present #define EXC_STACK_FAULT 12 \/\/ Stack segment fault #define EXC_GEN_PROTECT_FAULT 13 \/\/ General protection fault #define EXC_PAGE_FAULT 14 \/\/ Page fault #define EXC_RESERVED 15 \/\/ Reserved #define EXC_MATH_FAULT 16 \/\/ Floating point error #define EXC_ALIGNMENT_CHECK 17 \/\/ Alignment check \/\/ Exception handler void exception_handler(int exc_number, InterruptContext *ctx) { switch (exc_number) { case EXC_DIVIDE_BY_ZERO: printf(\"Division by zero at 0x%X\\n\", ctx->eip); signal_process(ctx->process_id, SIGFPE); \/\/ Floating point exception break; case EXC_PAGE_FAULT: printf(\"Page fault at 0x%X\\n\", ctx->eip); \/\/ Try to handle page fault if (handle_page_fault(ctx->cr2)) { \/\/ CR2 contains faulting address return; \/\/ Handled successfully } signal_process(ctx->process_id, SIGSEGV); \/\/ Segmentation violation break; case EXC_GEN_PROTECT_FAULT: printf(\"General protection fault at 0x%X\\n\", ctx->eip); signal_process(ctx->process_id, SIGSEGV); break; default: printf(\"Unhandled exception %d at 0x%X\\n\", exc_number, ctx->eip); signal_process(ctx->process_id, SIGABRT); \/\/ Abort } } ","date":"29-Nov-2024","heading":"14.3 Exception and Trap Handling","objectID":"\/posts\/operating-system\/:14:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#143-exception-and-trap-handling"},{"categories":["computer-science"],"collections":null,"content":"14.4 System Call Implementation \/* * System Call Dispatch * Kernel maintains table of system call handlers *\/ #define MAX_SYSCALLS 512 typedef long (*syscall_handler_t)(long arg1, long arg2, long arg3, long arg4, long arg5, long arg6); syscall_handler_t syscall_table[MAX_SYSCALLS] = { [0] = sys_read, [1] = sys_write, [2] = sys_open, [3] = sys_close, [4] = sys_stat, \/\/ ... many more syscalls [11] = sys_execve, [20] = sys_getpid, [39] = sys_mkdir, \/\/ ... up to 512 syscalls on modern Linux }; \/\/ Syscall entry point (assembly calls this) long syscall_dispatcher(long syscall_number, long arg1, long arg2, long arg3, long arg4, long arg5, long arg6) { if (syscall_number >= MAX_SYSCALLS) { return -ENOSYS; \/\/ Function not implemented } syscall_handler_t handler = syscall_table[syscall_number]; if (!handler) { return -ENOSYS; } \/\/ Call the actual syscall handler return handler(arg1, arg2, arg3, arg4, arg5, arg6); } \/\/ Example: sys_write implementation long sys_write(long fd, long buffer, long count, long arg4, long arg5, long arg6) { \/\/ Validate file descriptor if (fd < 0 || fd >= MAX_FDS) { return -EBADF; \/\/ Bad file descriptor } \/\/ Validate buffer is in user space if (!is_user_memory((void *)buffer, count)) { return -EFAULT; \/\/ Bad address } OpenFile *file = current_process->file_descriptors[fd]; if (!file) { return -EBADF; } \/\/ Check write permissions if (!file->writable) { return -EACCES; \/\/ Permission denied } \/\/ Perform the write ssize_t written = file->write(file, (void *)buffer, count); return written; } \/\/ Example: sys_read implementation long sys_read(long fd, long buffer, long count, long arg4, long arg5, long arg6) { if (fd < 0 || fd >= MAX_FDS) { return -EBADF; } if (!is_user_memory((void *)buffer, count)) { return -EFAULT; } OpenFile *file = current_process->file_descriptors[fd]; if (!file) { return -EBADF; } if (!file->readable) { return -EACCES; } ssize_t bytes_read = file->read(file, (void *)buffer, count); return bytes_read; } \/\/ Example: sys_fork implementation long sys_fork(long arg1, long arg2, long arg3, long arg4, long arg5, long arg6) { Process *current = get_current_process(); \/\/ Allocate new process structure Process *child = allocate_process(); if (!child) { return -ENOMEM; \/\/ No memory } \/\/ Copy process image (or use COW) copy_process_image(current, child); \/\/ Set parent-child relationship child->parent_pid = current->pid; child->ppid = current->pid; \/\/ Add to process list add_to_process_list(child); \/\/ In parent: return child PID \/\/ In child: this is handled by returning 0 to child return child->pid; } \/\/ Example: sys_exit implementation long sys_exit(long exit_code, long arg2, long arg3, long arg4, long arg5, long arg6) { Process *current = get_current_process(); \/\/ Store exit code current->exit_code = exit_code; current->state = PROCESS_ZOMBIE; \/\/ Close all file descriptors for (int i = 0; i < MAX_FDS; i++) { if (current->file_descriptors[i]) { sys_close(i, 0, 0, 0, 0, 0); } } \/\/ Free memory pages free_process_memory(current); \/\/ Wake up parent if waiting if (current->parent_pid != 0) { Process *parent = get_process_by_pid(current->parent_pid); if (parent && parent->state == PROCESS_WAITING) { parent->state = PROCESS_READY; } } \/\/ Schedule next process schedule_next(); return 0; \/\/ Never returns } ","date":"29-Nov-2024","heading":"14.4 System Call Implementation","objectID":"\/posts\/operating-system\/:14:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#144-system-call-implementation"},{"categories":["computer-science"],"collections":null,"content":"14.5 Virtual System Call (vsyscall) \/* * vsyscall: Virtual system call (fast path for common syscalls) * Avoids context switch overhead for frequently-used syscalls * Mapped to user space (read-only) * Used for: gettimeofday, clock_gettime, getcpu *\/ \/\/ vsyscall page mapped at fixed address (typically 0xFFFFE000) #define VSYSCALL_ADDR 0xFFFFE000 typedef struct { int (*gettimeofday)(struct timeval *tv, struct timezone *tz); long (*clock_gettime)(clockid_t clock_id, struct timespec *tp); int (*getcpu)(unsigned *cpu, unsigned *node); } vsyscall_functions_t; \/\/ User program can call without context switch time_t fast_gettimeofday() { vsyscall_functions_t *vsyscall = (vsyscall_functions_t *)VSYSCALL_ADDR; struct timeval tv; vsyscall->gettimeofday(&tv, NULL); return tv.tv_sec; } \/* * Replaced by vDSO (Virtual Dynamic Shared Object) in modern Linux * vDSO: Kernel-provided shared library * Functions: clock_gettime, gettimeofday, clock_nanosleep * Advantages: * - Faster than vsyscall * - More secure (randomized) * - Easy to upgrade *\/ ","date":"29-Nov-2024","heading":"14.5 Virtual System Call (vsyscall)","objectID":"\/posts\/operating-system\/:14:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#145-virtual-system-call-vsyscall"},{"categories":["computer-science"],"collections":null,"content":"14.6 Comparison: User vs Kernel Mode Operations Operation User Mode Kernel Mode Direct memory access Limited (own pages only) Unrestricted (all pages) I\/O operations Via syscalls Direct CPU register access Limited Full Mode switching No (need syscall) Can switch to user Performance Slower (syscall overhead) Faster (direct) Safety Protected (isolated) Privileged (risky) ","date":"29-Nov-2024","heading":"14.6 Comparison: User vs Kernel Mode Operations","objectID":"\/posts\/operating-system\/:14:6","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#146-comparison-user-vs-kernel-mode-operations"},{"categories":["computer-science"],"collections":null,"content":"Key Takeaways - System Calls Info System Calls Bridge User and Kernel Worlds: System calls provide controlled interface to kernel services \u2192 Sandboxed access to hardware Mode switch (user to kernel) involves context save\/restore \u2192 Expensive! (100s of cycles) Syscall dispatch uses lookup table indexed by syscall number \u2192 Fast lookup via syscall table vDSO\/vsyscall optimize frequent low-overhead syscalls \u2192 gettimeofday(), clock_gettime() avoid mode switch Exceptions handle errors and traps within kernel \u2192 Division by zero, page faults, etc. ","date":"29-Nov-2024","heading":"Key Takeaways - System Calls","objectID":"\/posts\/operating-system\/:14:7","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#key-takeaways---system-calls"},{"categories":["computer-science"],"collections":null,"content":"15. Inter-Process Communication (IPC) ","date":"29-Nov-2024","heading":"15. Inter-Process Communication (IPC)","objectID":"\/posts\/operating-system\/:15:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#ipc"},{"categories":["computer-science"],"collections":null,"content":"15.1 IPC Overview Processes need to communicate and synchronize. IPC mechanisms provide this. \/* * IPC Types: * 1. Pipes & FIFOs * 2. Message Queues * 3. Shared Memory * 4. Sockets * 5. Signals * 6. Semaphores *\/ ","date":"29-Nov-2024","heading":"15.1 IPC Overview","objectID":"\/posts\/operating-system\/:15:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#151-ipc-overview"},{"categories":["computer-science"],"collections":null,"content":"15.2 Pipes and Named Pipes (FIFOs) \/* * Pipe: Unidirectional communication channel * - Created by pipe() syscall * - Buffered (4096 bytes typically) * - Used for parent-child communication *\/ #include void pipe_example() { int pipefd[2]; \/\/ pipefd[0] = read end, pipefd[1] = write end if (pipe(pipefd) == -1) { perror(\"pipe\"); return; } pid_t pid = fork(); if (pid == 0) { \/\/ Child process close(pipefd[1]); \/\/ Close write end char buffer[100]; ssize_t n = read(pipefd[0], buffer, sizeof(buffer)); printf(\"Child received: %.*s\\n\", (int)n, buffer); close(pipefd[0]); exit(0); } else { \/\/ Parent process close(pipefd[0]); \/\/ Close read end const char *msg = \"Hello from parent\"; write(pipefd[1], msg, strlen(msg)); close(pipefd[1]); wait(NULL); } } \/* * Named Pipe (FIFO): * - Persistent named pipe * - Created with mkfifo() * - Can be opened by unrelated processes *\/ #include #include void fifo_example() { \/\/ Create FIFO mkfifo(\"\/tmp\/myfifo\", 0666); \/\/ Writer process if (fork() == 0) { int fd = open(\"\/tmp\/myfifo\", O_WRONLY); write(fd, \"Data\", 4); close(fd); exit(0); } \/\/ Reader process int fd = open(\"\/tmp\/myfifo\", O_RDONLY); char buffer[10]; read(fd, buffer, 4); printf(\"Received: %.4s\\n\", buffer); close(fd); unlink(\"\/tmp\/myfifo\"); } ","date":"29-Nov-2024","heading":"15.2 Pipes and Named Pipes (FIFOs)","objectID":"\/posts\/operating-system\/:15:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#152-pipes-and-named-pipes-fifos"},{"categories":["computer-science"],"collections":null,"content":"15.3 Message Queues \/* * Message Queue: POSIX IPC mechanism * - Persistent (survives process termination) * - Multiple senders\/receivers * - Messages have types *\/ #include #include typedef struct { long mtype; char mtext[256]; } Message; void message_queue_example() { struct mq_attr attr = { .mq_flags = 0, .mq_maxmsg = 10, .mq_msgsize = sizeof(Message), }; \/\/ Create message queue mqd_t mq = mq_open(\"\/myqueue\", O_CREAT | O_RDWR, 0666, &attr); if (mq == (mqd_t)-1) { perror(\"mq_open\"); return; } \/\/ Producer: Send message if (fork() == 0) { Message msg = { .mtype = 1, }; strcpy(msg.mtext, \"Hello from producer\"); if (mq_send(mq, (const char *)&msg, sizeof(msg), 0) == -1) { perror(\"mq_send\"); } mq_close(mq); exit(0); } \/\/ Consumer: Receive message Message msg; if (mq_receive(mq, (char *)&msg, sizeof(msg), NULL) == -1) { perror(\"mq_receive\"); } printf(\"Received: type=%ld, text=%s\\n\", msg.mtype, msg.mtext); mq_close(mq); mq_unlink(\"\/myqueue\"); } ","date":"29-Nov-2024","heading":"15.3 Message Queues","objectID":"\/posts\/operating-system\/:15:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#153-message-queues"},{"categories":["computer-science"],"collections":null,"content":"15.4 Shared Memory \/* * Shared Memory: Fast IPC (no copying) * - Multiple processes access same memory region * - Requires synchronization (mutex\/semaphore) * - Linux: System V IPC or POSIX shm *\/ #include #include #include typedef struct { int counter; char data[256]; pthread_mutex_t lock; } SharedData; void shared_memory_example() { key_t key = ftok(\"\/tmp\", 'S'); \/\/ Create unique key \/\/ Create or get shared memory segment int shmid = shmget(key, sizeof(SharedData), IPC_CREAT | 0666); if (shmid == -1) { perror(\"shmget\"); return; } \/\/ Attach to process address space SharedData *shared = (SharedData *)shmat(shmid, NULL, 0); if (shared == (void *)-1) { perror(\"shmat\"); return; } \/\/ Initialize on first use static int initialized = 0; if (!initialized) { memset(shared, 0, sizeof(SharedData)); pthread_mutex_init(&shared->lock, NULL); initialized = 1; } \/\/ Multiple processes can access pthread_mutex_lock(&shared->lock); shared->counter++; strcpy(shared->data, \"Shared data\"); pthread_mutex_unlock(&shared->lock); printf(\"Counter: %d, Data: %s\\n\", shared->counter, shared->data); \/\/ Detach from memory shmdt(shared); \/\/ Only last process to detach should delete \/\/ shmctl(shmid, IPC_RMID, NULL); } ","date":"29-Nov-2024","heading":"15.4 Shared Memory","objectID":"\/posts\/operating-system\/:15:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#154-shared-memory"},{"categories":["computer-science"],"collections":null,"content":"15.5 Sockets \/* * Sockets: Network IPC (local and remote) * - Unix domain sockets: Local machine only * - Network sockets: TCP\/IP over network *\/ #include #include #include #include \/\/ Unix domain socket server void unix_socket_server() { int sockfd = socket(AF_UNIX, SOCK_STREAM, 0); if (sockfd == -1) { perror(\"socket\"); return; } struct sockaddr_un addr = { .sun_family = AF_UNIX, }; strcpy(addr.sun_path, \"\/tmp\/server.sock\"); \/\/ Remove old socket file if exists unlink(addr.sun_path); if (bind(sockfd, (struct sockaddr *)&addr, sizeof(addr)) == -1) { perror(\"bind\"); return; } if (listen(sockfd, 1) == -1) { perror(\"listen\"); return; } printf(\"Server listening on \/tmp\/server.sock\\n\"); struct sockaddr_un peer_addr; socklen_t peer_addr_len = sizeof(peer_addr); int connfd = accept(sockfd, (struct sockaddr *)&peer_addr, &peer_addr_len); if (connfd == -1) { perror(\"accept\"); return; } char buffer[256]; ssize_t n = recv(connfd, buffer, sizeof(buffer), 0); printf(\"Received: %.*s\\n\", (int)n, buffer); close(connfd); close(sockfd); } \/\/ Unix domain socket client void unix_socket_client() { int sockfd = socket(AF_UNIX, SOCK_STREAM, 0); if (sockfd == -1) { perror(\"socket\"); return; } struct sockaddr_un addr = { .sun_family = AF_UNIX, }; strcpy(addr.sun_path, \"\/tmp\/server.sock\"); if (connect(sockfd, (struct sockaddr *)&addr, sizeof(addr)) == -1) { perror(\"connect\"); return; } const char *msg = \"Hello from client\"; send(sockfd, msg, strlen(msg), 0); close(sockfd); } \/\/ TCP socket (network) void tcp_socket_server(int port) { int sockfd = socket(AF_INET, SOCK_STREAM, 0); struct sockaddr_in addr = { .sin_family = AF_INET, .sin_port = htons(port), .sin_addr.s_addr = htonl(INADDR_ANY), }; int opt = 1; setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)); bind(sockfd, (struct sockaddr *)&addr, sizeof(addr)); listen(sockfd, 5); printf(\"Server listening on port %d\\n\", port); struct sockaddr_in client_addr; socklen_t client_addr_len = sizeof(client_addr); int connfd = accept(sockfd, (struct sockaddr *)&client_addr, &client_addr_len); char buffer[256]; recv(connfd, buffer, sizeof(buffer), 0); printf(\"Received: %s\\n\", buffer); close(connfd); close(sockfd); } ","date":"29-Nov-2024","heading":"15.5 Sockets","objectID":"\/posts\/operating-system\/:15:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#155-sockets"},{"categories":["computer-science"],"collections":null,"content":"15.6 Signals \/* * Signals: Asynchronous notifications * - Limited information (only signal number) * - Can interrupt process execution * - Common signals: SIGTERM, SIGKILL, SIGCHLD *\/ #include void signal_handler(int sig) { switch (sig) { case SIGTERM: printf(\"Received SIGTERM (terminate)\\n\"); exit(0); case SIGINT: printf(\"Received SIGINT (interrupt, Ctrl+C)\\n\"); exit(0); case SIGCHLD: printf(\"Child process terminated\\n\"); break; case SIGUSR1: printf(\"Received SIGUSR1 (user-defined)\\n\"); break; } } void signal_example() { \/\/ Register signal handler signal(SIGTERM, signal_handler); signal(SIGINT, signal_handler); signal(SIGCHLD, signal_handler); signal(SIGUSR1, signal_handler); printf(\"Process %d waiting for signals...\\n\", getpid()); \/\/ Infinite loop, waiting for signals while (1) { pause(); \/\/ Wait for signal } } \/\/ Send signal to process void send_signal_example() { pid_t target_pid = 1234; \/\/ Send SIGUSR1 if (kill(target_pid, SIGUSR1) == -1) { perror(\"kill\"); } } \/\/ Signal mask (block\/unblock signals) void signal_mask_example() { sigset_t set; \/\/ Create set with SIGTERM sigemptyset(&set); sigaddset(&set, SIGTERM); \/\/ Block SIGTERM if (sigprocmask(SIG_BLOCK, &set, NULL) == -1) { perror(\"sigprocmask\"); } printf(\"SIGTERM is now blocked\\n\"); \/\/ Unblock SIGTERM if (sigprocmask(SIG_UNBLOCK, &set, NULL) == -1) { perror(\"sigprocmask\"); } printf(\"SIGTERM is now unblocked\\n\"); } ","date":"29-Nov-2024","heading":"15.6 Signals","objectID":"\/posts\/operating-system\/:15:6","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#156-signals"},{"categories":["computer-science"],"collections":null,"content":"15.7 IPC Comparison Mechanism Speed Persistence Relationship Use Case Pipe Very Fast No Parent-child Simple stream FIFO Fast Yes Unrelated Named pipe Message Queue Medium Yes Any Asynchronous messages Shared Memory Fastest Yes Any High-speed data sharing Socket Medium No Any (network) Network communication Signal Very Fast No Any Event notification ","date":"29-Nov-2024","heading":"15.7 IPC Comparison","objectID":"\/posts\/operating-system\/:15:7","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#157-ipc-comparison"},{"categories":["computer-science"],"collections":null,"content":"Key Takeaways - IPC Tip Choosing the Right IPC Mechanism: Pipes: Simple, unidirectional, parent-child only \u2192 Use for process pipelines (ls | grep) Message Queues: Asynchronous, persistent, typed messages \u2192 Use for decoupled communication Shared Memory: Fastest but needs synchronization \u2192 Use for high-throughput data sharing Sockets: Flexible, supports local and network communication \u2192 Use for network services Signals: Lightweight but limited information \u2192 Use for event notifications Choose based on performance, persistence, and relationship needs \u2192 Profile your use case! ","date":"29-Nov-2024","heading":"Key Takeaways - IPC","objectID":"\/posts\/operating-system\/:15:8","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#key-takeaways---ipc"},{"categories":["computer-science"],"collections":null,"content":"16. Security and Protection ","date":"29-Nov-2024","heading":"16. Security and Protection","objectID":"\/posts\/operating-system\/:16:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#security"},{"categories":["computer-science"],"collections":null,"content":"16.1 Security Fundamentals CIA Triad - Information Security Fundamentals Figure 6: CIA Triad security model illustrating the three pillars of information security: Confidentiality (encryption, access control), Integrity (checksums, digital signatures), and Availability (redundancy, load balancing). \/* * CIA Triad: * - Confidentiality: Prevent unauthorized access * - Integrity: Prevent unauthorized modification * - Availability: Ensure authorized access *\/ \/\/ Security levels in operating systems typedef enum { SECURITY_NONE = 0, SECURITY_LOW = 1, SECURITY_MEDIUM = 2, SECURITY_HIGH = 3, SECURITY_TOP_SECRET = 4 } SecurityLevel; ","date":"29-Nov-2024","heading":"16.1 Security Fundamentals","objectID":"\/posts\/operating-system\/:16:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#161-security-fundamentals"},{"categories":["computer-science"],"collections":null,"content":"16.2 Authentication and Authorization Authentication vs Authorization Process Flow Figure 7: Authentication and authorization sequence diagram demonstrating user login, credential verification, session token generation, and resource access control. \/* * Authentication: Verify identity (who are you?) * Authorization: Check permissions (what can you do?) *\/ #include #include typedef struct { char username[32]; char password_hash[128]; uid_t uid; gid_t gid; time_t last_login; } UserAccount; \/\/ Password authentication bool authenticate_user(const char *username, const char *password) { \/\/ Read user from \/etc\/shadow (simplified) UserAccount user; if (!get_user_account(username, &user)) { return false; } \/\/ Hash the provided password char *hash = crypt(password, user.password_hash); \/\/ Compare with stored hash if (strcmp(hash, user.password_hash) == 0) { user.last_login = time(NULL); update_user_account(&user); return true; } return false; } \/\/ Multi-factor authentication bool mfa_authenticate(const char *username, const char *password, const char *token) { \/\/ First factor: password if (!authenticate_user(username, password)) { return false; } \/\/ Second factor: OTP token if (!verify_otp_token(username, token)) { return false; } return true; } ","date":"29-Nov-2024","heading":"16.2 Authentication and Authorization","objectID":"\/posts\/operating-system\/:16:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#162-authentication-and-authorization"},{"categories":["computer-science"],"collections":null,"content":"16.3 Access Control Models 16.3.1 Discretionary Access Control (DAC) DAC - Owner-Controlled Access Figure 8: Discretionary Access Control (DAC) model where resource owners set permissions (rwx) for users, groups, and others, as implemented in Unix\/Linux file systems. \/* * DAC: Owner controls access to their resources * Used in Unix\/Linux file permissions *\/ \/\/ Unix permission structure typedef struct { uid_t owner_uid; gid_t owner_gid; mode_t permissions; \/\/ rwxrwxrwx } DACPermissions; bool check_dac_access(DACPermissions *perm, uid_t uid, gid_t gid, int access_mode) { mode_t mode = perm->permissions; \/\/ Root has all permissions if (uid == 0) { return true; } \/\/ Owner permissions if (uid == perm->owner_uid) { mode >>= 6; \/\/ Get owner bits return (mode & access_mode) == access_mode; } \/\/ Group permissions if (gid == perm->owner_gid) { mode >>= 3; \/\/ Get group bits mode &= 0x7; return (mode & access_mode) == access_mode; } \/\/ Others permissions mode &= 0x7; return (mode & access_mode) == access_mode; } \/\/ Example: chmod command implementation void change_permissions(const char *path, mode_t new_mode) { \/\/ Verify caller is owner or root struct stat st; stat(path, &st); if (getuid() != 0 && getuid() != st.st_uid) { printf(\"Permission denied: not owner\\n\"); return; } \/\/ Change permissions chmod(path, new_mode); printf(\"Changed permissions to %o\\n\", new_mode); } 16.3.2 Mandatory Access Control (MAC) MAC - System-Enforced Security Policy Figure 9: Mandatory Access Control (MAC) system with hierarchical security labels (Top Secret, Secret, Confidential, Unclassified) and Bell-LaPadula model enforcing “No Read Up, No Write Down” rules. \/* * MAC: System enforces security policy * User cannot override (unlike DAC) * Used in: SELinux, AppArmor *\/ typedef struct { SecurityLevel classification; char category[32]; bool need_to_know; } MACLabel; typedef struct { SecurityLevel clearance; char categories[10][32]; int num_categories; } UserClearance; \/\/ Bell-LaPadula Model: No read up, no write down bool mac_can_read(UserClearance *user, MACLabel *object) { \/\/ Can read if clearance >= classification return user->clearance >= object->classification; } bool mac_can_write(UserClearance *user, MACLabel *object) { \/\/ Can write if clearance <= classification return user->clearance <= object->classification; } \/\/ SELinux-style context typedef struct { char user[32]; \/\/ SELinux user char role[32]; \/\/ Role char type[32]; \/\/ Type\/Domain char level[32]; \/\/ MLS level } SELinuxContext; bool selinux_check_access(SELinuxContext *subject, SELinuxContext *object, const char *permission) { \/\/ Check against SELinux policy \/\/ Simplified: check type enforcement return selinux_policy_allows(subject->type, object->type, permission); } 16.3.3 Role-Based Access Control (RBAC) RBAC - Role-Based Permissions Figure 10: Role-Based Access Control (RBAC) architecture where users are assigned roles (Admin, Developer, Viewer) that determine their permissions to access organizational resources. \/* * RBAC: Access based on user's role * Simplifies management for large organizations *\/ typedef enum { PERM_READ = 0x01, PERM_WRITE = 0x02, PERM_EXECUTE = 0x04, PERM_DELETE = 0x08, PERM_ADMIN = 0x10 } Permission; typedef struct { char role_name[32]; Permission permissions; char resources[10][64]; int num_resources; } Role; typedef struct { char username[32]; Role *roles[5]; int num_roles; } User; bool rbac_check_access(User *user, const char *resource, Permission required) { \/\/ Check if any of user's roles grant the permission for (int i = 0; i < user->num_roles; i++) { Role *role = user->roles[i]; \/\/ Check if role has permission if ((role->permissions & required) != required) { continue; } \/\/ Check if role applies to this resource for (int j = 0; j < role->num_resources; j++) { if (strcmp(role->resources[j], resource) == 0) { return true; } } } return false; } \/\/ Example: Database RBAC void create_role_example() { Role admin_role = { .role_name = \"Administrator\", .permissio","date":"29-Nov-2024","heading":"16.3 Access Control Models","objectID":"\/posts\/operating-system\/:16:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#163-access-control-models"},{"categories":["computer-science"],"collections":null,"content":"16.3 Access Control Models 16.3.1 Discretionary Access Control (DAC) DAC - Owner-Controlled Access Figure 8: Discretionary Access Control (DAC) model where resource owners set permissions (rwx) for users, groups, and others, as implemented in Unix\/Linux file systems. \/* * DAC: Owner controls access to their resources * Used in Unix\/Linux file permissions *\/ \/\/ Unix permission structure typedef struct { uid_t owner_uid; gid_t owner_gid; mode_t permissions; \/\/ rwxrwxrwx } DACPermissions; bool check_dac_access(DACPermissions *perm, uid_t uid, gid_t gid, int access_mode) { mode_t mode = perm->permissions; \/\/ Root has all permissions if (uid == 0) { return true; } \/\/ Owner permissions if (uid == perm->owner_uid) { mode >>= 6; \/\/ Get owner bits return (mode & access_mode) == access_mode; } \/\/ Group permissions if (gid == perm->owner_gid) { mode >>= 3; \/\/ Get group bits mode &= 0x7; return (mode & access_mode) == access_mode; } \/\/ Others permissions mode &= 0x7; return (mode & access_mode) == access_mode; } \/\/ Example: chmod command implementation void change_permissions(const char *path, mode_t new_mode) { \/\/ Verify caller is owner or root struct stat st; stat(path, &st); if (getuid() != 0 && getuid() != st.st_uid) { printf(\"Permission denied: not owner\\n\"); return; } \/\/ Change permissions chmod(path, new_mode); printf(\"Changed permissions to %o\\n\", new_mode); } 16.3.2 Mandatory Access Control (MAC) MAC - System-Enforced Security Policy Figure 9: Mandatory Access Control (MAC) system with hierarchical security labels (Top Secret, Secret, Confidential, Unclassified) and Bell-LaPadula model enforcing “No Read Up, No Write Down” rules. \/* * MAC: System enforces security policy * User cannot override (unlike DAC) * Used in: SELinux, AppArmor *\/ typedef struct { SecurityLevel classification; char category[32]; bool need_to_know; } MACLabel; typedef struct { SecurityLevel clearance; char categories[10][32]; int num_categories; } UserClearance; \/\/ Bell-LaPadula Model: No read up, no write down bool mac_can_read(UserClearance *user, MACLabel *object) { \/\/ Can read if clearance >= classification return user->clearance >= object->classification; } bool mac_can_write(UserClearance *user, MACLabel *object) { \/\/ Can write if clearance <= classification return user->clearance <= object->classification; } \/\/ SELinux-style context typedef struct { char user[32]; \/\/ SELinux user char role[32]; \/\/ Role char type[32]; \/\/ Type\/Domain char level[32]; \/\/ MLS level } SELinuxContext; bool selinux_check_access(SELinuxContext *subject, SELinuxContext *object, const char *permission) { \/\/ Check against SELinux policy \/\/ Simplified: check type enforcement return selinux_policy_allows(subject->type, object->type, permission); } 16.3.3 Role-Based Access Control (RBAC) RBAC - Role-Based Permissions Figure 10: Role-Based Access Control (RBAC) architecture where users are assigned roles (Admin, Developer, Viewer) that determine their permissions to access organizational resources. \/* * RBAC: Access based on user's role * Simplifies management for large organizations *\/ typedef enum { PERM_READ = 0x01, PERM_WRITE = 0x02, PERM_EXECUTE = 0x04, PERM_DELETE = 0x08, PERM_ADMIN = 0x10 } Permission; typedef struct { char role_name[32]; Permission permissions; char resources[10][64]; int num_resources; } Role; typedef struct { char username[32]; Role *roles[5]; int num_roles; } User; bool rbac_check_access(User *user, const char *resource, Permission required) { \/\/ Check if any of user's roles grant the permission for (int i = 0; i < user->num_roles; i++) { Role *role = user->roles[i]; \/\/ Check if role has permission if ((role->permissions & required) != required) { continue; } \/\/ Check if role applies to this resource for (int j = 0; j < role->num_resources; j++) { if (strcmp(role->resources[j], resource) == 0) { return true; } } } return false; } \/\/ Example: Database RBAC void create_role_example() { Role admin_role = { .role_name = \"Administrator\", .permissio","date":"29-Nov-2024","heading":"16.3.1 Discretionary Access Control (DAC)","objectID":"\/posts\/operating-system\/:16:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1631-discretionary-access-control-dac"},{"categories":["computer-science"],"collections":null,"content":"16.3 Access Control Models 16.3.1 Discretionary Access Control (DAC) DAC - Owner-Controlled Access Figure 8: Discretionary Access Control (DAC) model where resource owners set permissions (rwx) for users, groups, and others, as implemented in Unix\/Linux file systems. \/* * DAC: Owner controls access to their resources * Used in Unix\/Linux file permissions *\/ \/\/ Unix permission structure typedef struct { uid_t owner_uid; gid_t owner_gid; mode_t permissions; \/\/ rwxrwxrwx } DACPermissions; bool check_dac_access(DACPermissions *perm, uid_t uid, gid_t gid, int access_mode) { mode_t mode = perm->permissions; \/\/ Root has all permissions if (uid == 0) { return true; } \/\/ Owner permissions if (uid == perm->owner_uid) { mode >>= 6; \/\/ Get owner bits return (mode & access_mode) == access_mode; } \/\/ Group permissions if (gid == perm->owner_gid) { mode >>= 3; \/\/ Get group bits mode &= 0x7; return (mode & access_mode) == access_mode; } \/\/ Others permissions mode &= 0x7; return (mode & access_mode) == access_mode; } \/\/ Example: chmod command implementation void change_permissions(const char *path, mode_t new_mode) { \/\/ Verify caller is owner or root struct stat st; stat(path, &st); if (getuid() != 0 && getuid() != st.st_uid) { printf(\"Permission denied: not owner\\n\"); return; } \/\/ Change permissions chmod(path, new_mode); printf(\"Changed permissions to %o\\n\", new_mode); } 16.3.2 Mandatory Access Control (MAC) MAC - System-Enforced Security Policy Figure 9: Mandatory Access Control (MAC) system with hierarchical security labels (Top Secret, Secret, Confidential, Unclassified) and Bell-LaPadula model enforcing “No Read Up, No Write Down” rules. \/* * MAC: System enforces security policy * User cannot override (unlike DAC) * Used in: SELinux, AppArmor *\/ typedef struct { SecurityLevel classification; char category[32]; bool need_to_know; } MACLabel; typedef struct { SecurityLevel clearance; char categories[10][32]; int num_categories; } UserClearance; \/\/ Bell-LaPadula Model: No read up, no write down bool mac_can_read(UserClearance *user, MACLabel *object) { \/\/ Can read if clearance >= classification return user->clearance >= object->classification; } bool mac_can_write(UserClearance *user, MACLabel *object) { \/\/ Can write if clearance <= classification return user->clearance <= object->classification; } \/\/ SELinux-style context typedef struct { char user[32]; \/\/ SELinux user char role[32]; \/\/ Role char type[32]; \/\/ Type\/Domain char level[32]; \/\/ MLS level } SELinuxContext; bool selinux_check_access(SELinuxContext *subject, SELinuxContext *object, const char *permission) { \/\/ Check against SELinux policy \/\/ Simplified: check type enforcement return selinux_policy_allows(subject->type, object->type, permission); } 16.3.3 Role-Based Access Control (RBAC) RBAC - Role-Based Permissions Figure 10: Role-Based Access Control (RBAC) architecture where users are assigned roles (Admin, Developer, Viewer) that determine their permissions to access organizational resources. \/* * RBAC: Access based on user's role * Simplifies management for large organizations *\/ typedef enum { PERM_READ = 0x01, PERM_WRITE = 0x02, PERM_EXECUTE = 0x04, PERM_DELETE = 0x08, PERM_ADMIN = 0x10 } Permission; typedef struct { char role_name[32]; Permission permissions; char resources[10][64]; int num_resources; } Role; typedef struct { char username[32]; Role *roles[5]; int num_roles; } User; bool rbac_check_access(User *user, const char *resource, Permission required) { \/\/ Check if any of user's roles grant the permission for (int i = 0; i < user->num_roles; i++) { Role *role = user->roles[i]; \/\/ Check if role has permission if ((role->permissions & required) != required) { continue; } \/\/ Check if role applies to this resource for (int j = 0; j < role->num_resources; j++) { if (strcmp(role->resources[j], resource) == 0) { return true; } } } return false; } \/\/ Example: Database RBAC void create_role_example() { Role admin_role = { .role_name = \"Administrator\", .permissio","date":"29-Nov-2024","heading":"16.3.2 Mandatory Access Control (MAC)","objectID":"\/posts\/operating-system\/:16:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1632-mandatory-access-control-mac"},{"categories":["computer-science"],"collections":null,"content":"16.3 Access Control Models 16.3.1 Discretionary Access Control (DAC) DAC - Owner-Controlled Access Figure 8: Discretionary Access Control (DAC) model where resource owners set permissions (rwx) for users, groups, and others, as implemented in Unix\/Linux file systems. \/* * DAC: Owner controls access to their resources * Used in Unix\/Linux file permissions *\/ \/\/ Unix permission structure typedef struct { uid_t owner_uid; gid_t owner_gid; mode_t permissions; \/\/ rwxrwxrwx } DACPermissions; bool check_dac_access(DACPermissions *perm, uid_t uid, gid_t gid, int access_mode) { mode_t mode = perm->permissions; \/\/ Root has all permissions if (uid == 0) { return true; } \/\/ Owner permissions if (uid == perm->owner_uid) { mode >>= 6; \/\/ Get owner bits return (mode & access_mode) == access_mode; } \/\/ Group permissions if (gid == perm->owner_gid) { mode >>= 3; \/\/ Get group bits mode &= 0x7; return (mode & access_mode) == access_mode; } \/\/ Others permissions mode &= 0x7; return (mode & access_mode) == access_mode; } \/\/ Example: chmod command implementation void change_permissions(const char *path, mode_t new_mode) { \/\/ Verify caller is owner or root struct stat st; stat(path, &st); if (getuid() != 0 && getuid() != st.st_uid) { printf(\"Permission denied: not owner\\n\"); return; } \/\/ Change permissions chmod(path, new_mode); printf(\"Changed permissions to %o\\n\", new_mode); } 16.3.2 Mandatory Access Control (MAC) MAC - System-Enforced Security Policy Figure 9: Mandatory Access Control (MAC) system with hierarchical security labels (Top Secret, Secret, Confidential, Unclassified) and Bell-LaPadula model enforcing “No Read Up, No Write Down” rules. \/* * MAC: System enforces security policy * User cannot override (unlike DAC) * Used in: SELinux, AppArmor *\/ typedef struct { SecurityLevel classification; char category[32]; bool need_to_know; } MACLabel; typedef struct { SecurityLevel clearance; char categories[10][32]; int num_categories; } UserClearance; \/\/ Bell-LaPadula Model: No read up, no write down bool mac_can_read(UserClearance *user, MACLabel *object) { \/\/ Can read if clearance >= classification return user->clearance >= object->classification; } bool mac_can_write(UserClearance *user, MACLabel *object) { \/\/ Can write if clearance <= classification return user->clearance <= object->classification; } \/\/ SELinux-style context typedef struct { char user[32]; \/\/ SELinux user char role[32]; \/\/ Role char type[32]; \/\/ Type\/Domain char level[32]; \/\/ MLS level } SELinuxContext; bool selinux_check_access(SELinuxContext *subject, SELinuxContext *object, const char *permission) { \/\/ Check against SELinux policy \/\/ Simplified: check type enforcement return selinux_policy_allows(subject->type, object->type, permission); } 16.3.3 Role-Based Access Control (RBAC) RBAC - Role-Based Permissions Figure 10: Role-Based Access Control (RBAC) architecture where users are assigned roles (Admin, Developer, Viewer) that determine their permissions to access organizational resources. \/* * RBAC: Access based on user's role * Simplifies management for large organizations *\/ typedef enum { PERM_READ = 0x01, PERM_WRITE = 0x02, PERM_EXECUTE = 0x04, PERM_DELETE = 0x08, PERM_ADMIN = 0x10 } Permission; typedef struct { char role_name[32]; Permission permissions; char resources[10][64]; int num_resources; } Role; typedef struct { char username[32]; Role *roles[5]; int num_roles; } User; bool rbac_check_access(User *user, const char *resource, Permission required) { \/\/ Check if any of user's roles grant the permission for (int i = 0; i < user->num_roles; i++) { Role *role = user->roles[i]; \/\/ Check if role has permission if ((role->permissions & required) != required) { continue; } \/\/ Check if role applies to this resource for (int j = 0; j < role->num_resources; j++) { if (strcmp(role->resources[j], resource) == 0) { return true; } } } return false; } \/\/ Example: Database RBAC void create_role_example() { Role admin_role = { .role_name = \"Administrator\", .permissio","date":"29-Nov-2024","heading":"16.3.3 Role-Based Access Control (RBAC)","objectID":"\/posts\/operating-system\/:16:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#1633-role-based-access-control-rbac"},{"categories":["computer-science"],"collections":null,"content":"16.4 Encryption Basics Symmetric vs Asymmetric Encryption Figure 11: Encryption fundamentals comparing symmetric encryption (same key for encrypt\/decrypt) and asymmetric encryption (public key for encryption, private key for decryption). \/* * Encryption: Transform data to unreadable form * Symmetric: Same key for encrypt\/decrypt (AES, DES) * Asymmetric: Public\/Private key pair (RSA, ECC) *\/ #include #include \/\/ Symmetric encryption (AES example) void aes_encrypt(const unsigned char *plaintext, size_t len, const unsigned char *key, unsigned char *ciphertext) { AES_KEY aes_key; AES_set_encrypt_key(key, 256, &aes_key); \/\/ Encrypt in ECB mode (simplified - use CBC\/GCM in production) for (size_t i = 0; i < len; i += 16) { AES_encrypt(plaintext + i, ciphertext + i, &aes_key); } } void aes_decrypt(const unsigned char *ciphertext, size_t len, const unsigned char *key, unsigned char *plaintext) { AES_KEY aes_key; AES_set_decrypt_key(key, 256, &aes_key); for (size_t i = 0; i < len; i += 16) { AES_decrypt(ciphertext + i, plaintext + i, &aes_key); } } \/\/ Hash functions (one-way) #include void hash_password(const char *password, unsigned char *hash) { SHA256_CTX ctx; SHA256_Init(&ctx); SHA256_Update(&ctx, password, strlen(password)); SHA256_Final(hash, &ctx); } \/\/ Example: Secure file storage void secure_file_write(const char *filename, const unsigned char *data, size_t len, const unsigned char *key) { \/\/ Encrypt data unsigned char *encrypted = malloc(len); aes_encrypt(data, len, key, encrypted); \/\/ Write to file FILE *f = fopen(filename, \"wb\"); fwrite(encrypted, 1, len, f); fclose(f); free(encrypted); } ","date":"29-Nov-2024","heading":"16.4 Encryption Basics","objectID":"\/posts\/operating-system\/:16:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#164-encryption-basics"},{"categories":["computer-science"],"collections":null,"content":"16.5 Common Security Vulnerabilities Common OS Security Vulnerabilities Figure 12: Taxonomy of common security vulnerabilities in operating systems including buffer overflows, SQL injection, race conditions (TOCTOU), and privilege escalation exploits. Danger Common Security Vulnerabilities - Critical Patterns to Avoid \/* * Common vulnerabilities and mitigations *\/ \/\/ 1. Buffer Overflow void vulnerable_function(char *input) { char buffer[100]; strcpy(buffer, input); \/\/ VULNERABLE! No bounds check } void safe_function(char *input) { char buffer[100]; strncpy(buffer, input, sizeof(buffer) - 1); \/\/ Safe buffer[sizeof(buffer) - 1] = '\\0'; } \/\/ 2. Race Condition (TOCTOU - Time of Check, Time of Use) Warning TOCTOU Race Condition: Between checking file permissions (TIME OF CHECK) and opening the file (TIME OF USE), an attacker can replace the file with a malicious symlink! This is a classic security vulnerability exploited in privilege escalation attacks. void vulnerable_file_access(const char *filename) { \/\/ Check if file is accessible if (access(filename, R_OK) == 0) { \/\/ TIME OF CHECK \/\/ Race condition window here! FILE *f = fopen(filename, \"r\"); \/\/ TIME OF USE \/\/ Attacker could replace file with symlink } } void safe_file_access(const char *filename) { \/\/ Open file and then check int fd = open(filename, O_RDONLY | O_NOFOLLOW); \/\/ No symlinks if (fd == -1) { return; } struct stat st; if (fstat(fd, &st) == 0) { \/\/ Use file descriptor directly } close(fd); } \/\/ 3. Privilege Escalation Prevention void drop_privileges() { \/\/ Drop root privileges after initialization uid_t real_uid = getuid(); gid_t real_gid = getgid(); \/\/ Drop supplementary groups setgroups(0, NULL); \/\/ Set GID first (must be done before UID) setgid(real_gid); setuid(real_uid); \/\/ Verify privileges dropped if (geteuid() == 0 || getegid() == 0) { fprintf(stderr, \"Failed to drop privileges!\\n\"); exit(1); } } ","date":"29-Nov-2024","heading":"16.5 Common Security Vulnerabilities","objectID":"\/posts\/operating-system\/:16:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#165-common-security-vulnerabilities"},{"categories":["computer-science"],"collections":null,"content":"16.6 Security in Modern OS \/* * Modern OS Security Features *\/ \/\/ ASLR (Address Space Layout Randomization) \/\/ Randomizes memory addresses to prevent exploits void demonstrate_aslr() { int stack_var; int *heap_var = malloc(sizeof(int)); printf(\"Stack: %p\\n\", &stack_var); printf(\"Heap: %p\\n\", heap_var); \/\/ Addresses will be different each run with ASLR } \/\/ NX\/DEP (No Execute \/ Data Execution Prevention) \/\/ Marks memory pages as non-executable \/\/ Prevents code injection attacks \/\/ Seccomp (Secure Computing Mode) \/\/ Restricts syscalls available to process #include #include void enable_seccomp_strict() { \/\/ Allow only read, write, exit, sigreturn prctl(PR_SET_SECCOMP, SECCOMP_MODE_STRICT); \/\/ After this, any other syscall terminates process } \/\/ Namespaces (Isolation) \/\/ Isolate process resources (used in containers) void create_namespace() { \/\/ Create new PID namespace if (unshare(CLONE_NEWPID) == -1) { perror(\"unshare\"); return; } \/\/ Process now in isolated PID namespace \/\/ PID 1 in this namespace } ","date":"29-Nov-2024","heading":"16.6 Security in Modern OS","objectID":"\/posts\/operating-system\/:16:6","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#166-security-in-modern-os"},{"categories":["computer-science"],"collections":null,"content":"Key Takeaways - Security Info OS Security Fundamentals: CIA Triad: Confidentiality, Integrity, Availability \u2192 Three pillars of security Authentication: Verify identity (who you are) \u2192 Passwords, biometrics, MFA Authorization: Check permissions (what you can do) \u2192 Access control models DAC: Owner controls access (Unix permissions) \u2192 User-controlled, flexible MAC: System enforces policy (SELinux) \u2192 Centralized, stronger security RBAC: Role-based permissions (enterprise systems) \u2192 Scalable management Encryption: Symmetric (fast), Asymmetric (secure key exchange) \u2192 AES-256, RSA-2048 Common vulnerabilities: Buffer overflow, race conditions, privilege escalation \u2192 Audit code regularly! ","date":"29-Nov-2024","heading":"Key Takeaways - Security","objectID":"\/posts\/operating-system\/:16:7","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#key-takeaways---security"},{"categories":["computer-science"],"collections":null,"content":"17. Unix\/Linux Deep Dive ","date":"29-Nov-2024","heading":"17. Unix\/Linux Deep Dive","objectID":"\/posts\/operating-system\/:17:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#unix-linux"},{"categories":["computer-science"],"collections":null,"content":"17.1 Unix\/Linux Architecture Unix\/Linux System Architecture Layers Figure 13: Unix\/Linux operating system architecture illustrating user space (applications, shell, libraries) and kernel space (process management, memory management, file systems, device drivers, network stack) with system call interface as the boundary. # Unix\/Linux File System Hierarchy \/ # Root directory \u251c\u2500\u2500 bin\/ # Essential user binaries \u251c\u2500\u2500 boot\/ # Boot loader files \u251c\u2500\u2500 dev\/ # Device files \u251c\u2500\u2500 etc\/ # System configuration \u251c\u2500\u2500 home\/ # User home directories \u251c\u2500\u2500 lib\/ # System libraries \u251c\u2500\u2500 media\/ # Removable media mount points \u251c\u2500\u2500 mnt\/ # Temporary mount points \u251c\u2500\u2500 opt\/ # Optional software \u251c\u2500\u2500 proc\/ # Process information (virtual) \u251c\u2500\u2500 root\/ # Root user home \u251c\u2500\u2500 run\/ # Runtime data \u251c\u2500\u2500 sbin\/ # System binaries \u251c\u2500\u2500 srv\/ # Service data \u251c\u2500\u2500 sys\/ # Kernel and system info (virtual) \u251c\u2500\u2500 tmp\/ # Temporary files \u251c\u2500\u2500 usr\/ # User programs \u2502 \u251c\u2500\u2500 bin\/ # User binaries \u2502 \u251c\u2500\u2500 lib\/ # User libraries \u2502 \u251c\u2500\u2500 local\/ # Locally compiled software \u2502 \u2514\u2500\u2500 share\/ # Shared data \u2514\u2500\u2500 var\/ # Variable data \u251c\u2500\u2500 log\/ # Log files \u251c\u2500\u2500 mail\/ # Mail \u251c\u2500\u2500 spool\/ # Spool files \u2514\u2500\u2500 tmp\/ # Temporary files (persists across reboots) ","date":"29-Nov-2024","heading":"17.1 Unix\/Linux Architecture","objectID":"\/posts\/operating-system\/:17:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#171-unixlinux-architecture"},{"categories":["computer-science"],"collections":null,"content":"17.2 Boot Process Linux Boot Sequence Diagram Figure 14: Unix\/Linux boot process sequence diagram showing the complete boot chain from BIOS\/UEFI \u2192 Bootloader (GRUB) \u2192 Kernel \u2192 Init (PID 1) \u2192 System Services startup. # Boot process stages # 1. BIOS\/UEFI # - Power-On Self Test (POST) # - Initialize hardware # - Load bootloader # 2. Bootloader (GRUB) # - Present menu # - Load kernel and initramfs # - Pass kernel parameters # 3. Kernel initialization # - Decompress kernel # - Setup memory management # - Mount root filesystem # - Start init process # 4. Init system (systemd\/SysV) # - Run startup scripts # - Mount filesystems # - Start services # View boot messages dmesg | less # Check boot time systemd-analyze # See boot sequence systemd-analyze critical-chain ","date":"29-Nov-2024","heading":"17.2 Boot Process","objectID":"\/posts\/operating-system\/:17:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#172-boot-process"},{"categories":["computer-science"],"collections":null,"content":"17.3 Shell and Command Line #!\/bin\/bash # =================== # File Operations # =================== # List files ls -la # Long format, all files ls -lh # Human-readable sizes ls -lt # Sort by modification time ls -lS # Sort by size # Navigate directories cd \/path\/to\/dir # Change directory cd ~ # Go to home cd - # Go to previous directory pwd # Print working directory # Copy, move, remove cp file1 file2 # Copy file cp -r dir1 dir2 # Copy directory recursively mv old new # Move\/rename rm file # Remove file rm -rf dir # Remove directory recursively (DANGEROUS!) # Create directories mkdir mydir # Create directory mkdir -p a\/b\/c # Create nested directories # File viewing cat file # Display entire file less file # Page through file head -n 20 file # First 20 lines tail -n 20 file # Last 20 lines tail -f file # Follow file (for logs) # =================== # File Permissions # =================== # Permission format: rwxrwxrwx (owner group others) # r=4, w=2, x=1 # Change permissions chmod 755 script.sh # rwxr-xr-x chmod u+x script.sh # Add execute for owner chmod go-w file # Remove write for group\/others # Change ownership chown user:group file # Change owner and group chown -R user dir # Recursive # View permissions ls -l file stat file # Detailed info # Special permissions chmod +t dir # Sticky bit (only owner can delete files) chmod u+s program # Setuid (run as file owner) chmod g+s program # Setgid (run as file group) # =================== # Process Management # =================== # View processes ps aux # All processes ps -ef # All processes (different format) ps -u username # User's processes top # Interactive process viewer htop # Better top (if installed) # Process tree pstree # Show process hierarchy pstree -p # Show PIDs # Find process pgrep firefox # Find PID by name pidof firefox # Same as pgrep # Kill process kill PID # Send SIGTERM kill -9 PID # Send SIGKILL (force) killall firefox # Kill all firefox processes pkill firefox # Kill by name # Background jobs command & # Run in background jobs # List background jobs fg %1 # Bring job 1 to foreground bg %1 # Resume job 1 in background disown %1 # Detach job from shell # Process priority nice -n 10 command # Start with lower priority renice -n 5 -p PID # Change priority of running process # =================== # Finding Files # =================== # Find command find \/path -name \"*.txt\" # Find by name find \/path -type f # Find files find \/path -type d # Find directories find \/path -size +100M # Find large files find \/path -mtime -7 # Modified in last 7 days find \/path -user username # Find by owner find \/path -perm 755 # Find by permissions # Execute command on found files find \/path -name \"*.log\" -exec rm {} \\; # Locate (faster, uses database) locate filename updatedb # Update locate database # Which (find command in PATH) which python type python # Show command type ","date":"29-Nov-2024","heading":"17.3 Shell and Command Line","objectID":"\/posts\/operating-system\/:17:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#173-shell-and-command-line"},{"categories":["computer-science"],"collections":null,"content":"17.4 Text Processing #!\/bin\/bash # =================== # grep - Search text # =================== # Basic usage grep \"pattern\" file # Search for pattern grep -i \"pattern\" file # Case insensitive grep -v \"pattern\" file # Invert match (show non-matching) grep -r \"pattern\" dir # Recursive search grep -n \"pattern\" file # Show line numbers grep -c \"pattern\" file # Count matches grep -l \"pattern\" files* # Show only filenames grep -A 3 \"pattern\" file # Show 3 lines after match grep -B 3 \"pattern\" file # Show 3 lines before match grep -C 3 \"pattern\" file # Show 3 lines context # Regular expressions grep \"^start\" file # Lines starting with \"start\" grep \"end$\" file # Lines ending with \"end\" grep \"[0-9]+\" file # Lines with numbers grep -E \"word1|word2\" file # Extended regex (OR) # Practical examples ps aux | grep firefox # Find firefox processes grep -r \"TODO\" *.c # Find TODOs in C files grep -v \"^#\" config.conf # Show non-comment lines # =================== # awk - Text processing # =================== # Print columns awk '{print $1}' file # Print first column awk '{print $1, $3}' file # Print columns 1 and 3 awk '{print $NF}' file # Print last column # Field separator awk -F: '{print $1}' \/etc\/passwd # Use : as separator # Conditions awk '$3 > 100' file # Print lines where column 3 > 100 awk '$1 == \"root\"' file # Print lines where column 1 is \"root\" # Built-in variables awk '{print NR, $0}' file # NR = line number awk 'NF > 5' file # NF = number of fields # Calculations awk '{sum += $1} END {print sum}' file # Sum column 1 awk '{print $1 * 2}' file # Multiply column 1 by 2 # Practical examples ps aux | awk '$3 > 50' # Processes using >50% CPU df -h | awk '$5 > 80' # Partitions >80% full awk -F: '{print $1}' \/etc\/passwd # List usernames # =================== # sed - Stream editor # =================== # Substitution sed 's\/old\/new\/' file # Replace first occurrence sed 's\/old\/new\/g' file # Replace all occurrences sed 's\/old\/new\/gi' file # Case insensitive sed '5s\/old\/new\/' file # Replace in line 5 sed -i 's\/old\/new\/g' file # Edit file in-place # Delete lines sed '5d' file # Delete line 5 sed '5,10d' file # Delete lines 5-10 sed '\/pattern\/d' file # Delete lines matching pattern # Insert\/Append sed '5i\\New Line' file # Insert before line 5 sed '5a\\New Line' file # Append after line 5 # Print lines sed -n '5p' file # Print line 5 sed -n '5,10p' file # Print lines 5-10 sed -n '\/pattern\/p' file # Print matching lines # Practical examples sed 's\/#.*$\/\/' config.conf # Remove comments sed '\/^$\/d' file # Remove empty lines sed 's\/\\s\\+$\/\/' file # Remove trailing whitespace # =================== # sort, uniq, cut # =================== # Sort sort file # Sort alphabetically sort -n file # Sort numerically sort -r file # Reverse sort sort -k 2 file # Sort by column 2 sort -u file # Sort and remove duplicates # Unique uniq file # Remove adjacent duplicates uniq -c file # Count occurrences uniq -d file # Show only duplicates sort file | uniq # Remove all duplicates # Cut cut -d: -f1 \/etc\/passwd # Extract field 1 (delimiter :) cut -c1-10 file # Extract characters 1-10 # =================== # Combining commands # =================== # Count lines in all .c files find . -name \"*.c\" | xargs wc -l # Find most common words in file cat file | tr '[:upper:]' '[:lower:]' | tr -s ' ' '\\n' | sort | uniq -c | sort -rn | head -10 # Find largest files find \/ -type f -exec du -h {} + | sort -rh | head -10 # Monitor log file for errors tail -f \/var\/log\/syslog | grep --line-buffered ERROR # Count unique IP addresses in log awk '{print $1}' access.log | sort | uniq -c | sort -rn ","date":"29-Nov-2024","heading":"17.4 Text Processing","objectID":"\/posts\/operating-system\/:17:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#174-text-processing"},{"categories":["computer-science"],"collections":null,"content":"Key Takeaways - Unix\/Linux Note Unix\/Linux Command Line Mastery: File System Hierarchy: Standard directory structure (FHS) \u2192 \/bin, \/etc, \/home, \/var, etc. Boot Process: BIOS\/UEFI \u2192 Bootloader \u2192 Kernel \u2192 Init \u2192 Systemd \u2192 Services File Permissions: Read (4), Write (2), Execute (1) \u2192 chmod 755 = rwxr-xr-x Process Management: ps, top, kill, nice for process control \u2192 Monitor and control processes Text Processing: grep (search), awk (columns), sed (edit), sort\/uniq \u2192 Powerful text manipulation Command Piping: Chain commands with | for powerful workflows \u2192 ls | grep pattern | sort | uniq ","date":"29-Nov-2024","heading":"Key Takeaways - Unix\/Linux","objectID":"\/posts\/operating-system\/:17:5","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#key-takeaways---unixlinux"},{"categories":["computer-science"],"collections":null,"content":"Conclusion This comprehensive operating systems tutorial covers the essential concepts needed for: Placement interviews at top tech companies Competitive programming understanding of system-level concepts Kernel development basics Systems programming foundation Performance optimization insights Key Areas Covered: Process and thread management with scheduling Memory management (physical and virtual) File systems and I\/O operations Synchronization and deadlock handling System calls and interrupts Inter-process communication Security and protection mechanisms Real-world Unix\/Linux concepts Study Tips: Understand concepts deeply, not just memorize Practice implementing algorithms (scheduling, page replacement, etc.) Run code examples and modify them Trace through syscall execution mentally Review code snippets regularly Connect theory to real systems (Linux, Windows) Practice interview questions about tradeoffs and design decisions Interview Focus: Be ready to discuss tradeoffs (performance vs correctness) Know when to use which algorithm (FCFS vs LRU vs SCAN) Understand context switches and overhead Explain concepts with examples Connect multiple topics (e.g., how paging affects scheduling) Good luck with your OS learning journey! ","date":"29-Nov-2024","heading":"Conclusion","objectID":"\/posts\/operating-system\/:18:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#conclusion"},{"categories":["computer-science"],"collections":null,"content":"References and Further Reading ","date":"29-Nov-2024","heading":"References and Further Reading","objectID":"\/posts\/operating-system\/:19:0","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#references"},{"categories":["computer-science"],"collections":null,"content":"Books “Operating Systems: Three Easy Pieces” by Remzi H. Arpaci-Dusseau and Andrea C. Arpaci-Dusseau “Modern Operating Systems” by Andrew S. Tanenbaum and Herbert Bos “Operating System Concepts” by Abraham Silberschatz, Peter Baer Galvin, and Greg Gagne “Linux Kernel Development” by Robert Love “The Design of the UNIX Operating System” by Maurice J. Bach “Advanced Programming in the UNIX Environment” by Stevens & Rago ","date":"29-Nov-2024","heading":"Books","objectID":"\/posts\/operating-system\/:19:1","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#books"},{"categories":["computer-science"],"collections":null,"content":"Online Resources OSDev Wiki - Comprehensive OS development resource Linux Kernel Documentation ARM Architecture Reference Manual Intel Software Developer Manuals Rust Embedded Book ","date":"29-Nov-2024","heading":"Online Resources","objectID":"\/posts\/operating-system\/:19:2","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#online-resources"},{"categories":["computer-science"],"collections":null,"content":"Tutorial Series cfenollosa’s OS Tutorial (x86 bare metal) Raspberry Pi OS Learning Repository Rust Raspberry Pi OS Tutorials MIT 6.828: Operating System Engineering ","date":"29-Nov-2024","heading":"Tutorial Series","objectID":"\/posts\/operating-system\/:19:3","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#tutorial-series"},{"categories":["computer-science"],"collections":null,"content":"Online Courses UC Berkeley CS162: Operating Systems MIT OpenCourseWare: Operating Systems Linux Foundation Training CS162 (Berkeley): Operating Systems MIT 6.S081: Operating System Engineering CS140 (Stanford): Operating Systems Note: This tutorial is designed for educational purposes, drawing from multiple sources and synthesizing concepts in an original manner. All code examples are simplified for instructional clarity and may require additional error handling and optimization for production use. This comprehensive tutorial provides theoretical foundations, practical implementation details, code examples, and real-world applications to ensure deep understanding of operating system concepts. ","date":"29-Nov-2024","heading":"Online Courses","objectID":"\/posts\/operating-system\/:19:4","tags":["operating-systems","kernel-development","system-programming"],"title":"Operating Systems Notes \u2014 Fundamentals to Kernel Programming","uri":"\/posts\/operating-system\/#online-courses"},{"categories":["software-engineering"],"collections":null,"content":"Comprehensive Vim reference covering modal editing concepts, efficient navigation, text manipulation with operators and motions, macros, registers, customization, and advanced workflows for productive text editing.","date":"22-Nov-2024","objectID":"\/posts\/vim\/","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/"},{"categories":["software-engineering"],"collections":null,"content":"Master the World’s Most Powerful Text Editor \u2014 From Beginner to Expert ","date":"22-Nov-2024","heading":"","objectID":"\/posts\/vim\/:0:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#"},{"categories":["software-engineering"],"collections":null,"content":"Introduction to Vim\rInfo Why Learn Vim? Vim is not just a text editor\u2014it’s a philosophy of efficient editing. Once you understand Vim’s core concepts, you can edit text 5-10x faster than traditional editors. Used by developers, system administrators, and DevOps professionals worldwide. Vim is a highly configurable modal text editor built on top of Vi (Unix’s classic text editor from 1976). It’s lightweight, powerful, and available on virtually every Unix-like system, making it the de facto standard for command-line editing. Key Statistics: Age: First released in 1991 (34 years of continuous development) Popularity: Used by millions of developers and system administrators Learning Curve: Steep initially, exponential productivity gains Extensibility: Fully programmable via Vimscript or Lua Community: Massive ecosystem with thousands of plugins ","date":"22-Nov-2024","heading":"Introduction to Vim","objectID":"\/posts\/vim\/:1:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#introduction"},{"categories":["software-engineering"],"collections":null,"content":"The Vim Philosophy: Modal Editing\rNote Modal Editing Paradigm Most editors use a single mode where keys always insert text. Vim separates concerns into distinct modes: Normal Mode: Navigation and commands (your home base) Insert Mode: Text insertion (like traditional editors) Visual Mode: Text selection and manipulation Command Mode: Advanced operations and configuration This separation creates logical separation of concerns, making complex text operations intuitive once you understand the grammar. Vim Modes \u2014 Normal, Insert, Visual, and Command modes; understanding these modes is fundamental to Vim mastery ","date":"22-Nov-2024","heading":"The Vim Philosophy: Modal Editing","objectID":"\/posts\/vim\/:2:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#vim-philosophy"},{"categories":["software-engineering"],"collections":null,"content":"The Vim Grammar: Operator + Motion Tip Vim’s Killer Feature: Text Objects + Motions = Composable Editing Vim operates on a grammar: operator + motion + (optionally) text-object Examples: dw = Delete word (delete + word motion) c5j = Change 5 lines down (change + 5-line motion) dip = Delete paragraph (delete + paragraph text object) This composability means once you learn operators and motions, you can combine them infinitely! Vim Operator Grammar - The fundamental grammar combining operators (d\/c\/y), counts, and motions to create powerful editing commands ","date":"22-Nov-2024","heading":"The Vim Grammar: Operator + Motion","objectID":"\/posts\/vim\/:2:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#the-vim-grammar-operator--motion"},{"categories":["software-engineering"],"collections":null,"content":"Installation and Setup\r","date":"22-Nov-2024","heading":"Installation and Setup","objectID":"\/posts\/vim\/:3:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#installation-setup"},{"categories":["software-engineering"],"collections":null,"content":"Installing Vim # Ubuntu\/Debian\rsudo apt update && sudo apt install vim\r# CentOS\/RHEL\/Fedora (7)\rsudo yum install vim\r# CentOS\/RHEL\/Fedora (8+)\rsudo dnf install vim\r# macOS\rbrew install vim\r# Or for newer Neovim (modern alternative)\rbrew install neovim\r# Windows (using Chocolatey)\rchoco install vim\r# Or download from https:\/\/www.vim.org\/download.php ","date":"22-Nov-2024","heading":"Installing Vim","objectID":"\/posts\/vim\/:3:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#installing-vim"},{"categories":["software-engineering"],"collections":null,"content":"Verify Installation # Check version and features\rvim --version\r# Output should show: VIM - Vi IMproved and version number\r# Look for these important features (+):\r# +python, +python3 (for plugins)\r# +clientserver (for remote editing)\r# +clipboard (for system clipboard) ","date":"22-Nov-2024","heading":"Verify Installation","objectID":"\/posts\/vim\/:3:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#verify-installation"},{"categories":["software-engineering"],"collections":null,"content":"Initial Configuration Create a basic .vimrc file in your home directory: \" Basic Vim Configuration\r\" This is your Vim initialization file\r\" Enable syntax highlighting\rsyntax on\r\" Use 24-bit colors\rset termguicolors\r\" Line numbering\rset number\rset relativenumber\r\" Indentation\rset tabstop=4 \" Tab width\rset shiftwidth=4 \" Indent width\rset expandtab \" Use spaces instead of tabs\rset autoindent \" Auto-indent new lines\r\" Search behavior\rset ignorecase \" Ignore case in searches\rset smartcase \" But match case if uppercase is used\rset incsearch \" Highlight search as you type\r\" UI Enhancements\rset statusline=%F%m%r%h%w\\ [%{&ff}]\\ %y\\ [%p%%]\\ %l:%c\rset laststatus=2 \" Always show status bar\rset cursorline \" Highlight current line\rset showmatch \" Highlight matching brackets\r\" Performance\rset nocompatible \" Use Vim defaults, not Vi\rset lazyredraw \" Don't redraw during macros ","date":"22-Nov-2024","heading":"Initial Configuration","objectID":"\/posts\/vim\/:3:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#initial-configuration"},{"categories":["software-engineering"],"collections":null,"content":"Understanding Vim Modes\r","date":"22-Nov-2024","heading":"Understanding Vim Modes","objectID":"\/posts\/vim\/:4:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#vim-modes"},{"categories":["software-engineering"],"collections":null,"content":"Mode Transitions Vim operates in distinct modes, each optimized for different tasks. Understanding how to transition between modes is essential: Vim Modes - Illustrating transitions between Normal Mode (navigation), Insert Mode (text entry), Visual Mode (selection), and Command Mode ","date":"22-Nov-2024","heading":"Mode Transitions","objectID":"\/posts\/vim\/:4:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#mode-transitions"},{"categories":["software-engineering"],"collections":null,"content":"1. Normal Mode (Command Mode) Default mode for navigation and commands. \" Navigation\rh, j, k, l = Left, Down, Up, Right (arrow keys equivalent)\rw, W = Next word, next WORD\rb, B = Previous word, previous WORD\r0, $ = Start of line, end of line\rgg, G = Start of file, end of file\r{, } = Previous\/next paragraph\r\" Editing\rx, X = Delete character under\/before cursor\rdd = Delete entire line\ryy = Yank (copy) entire line\rp, P = Paste after\/before cursor\ru = Undo\rCTRL-R = Redo\r\" Operators\rd = Delete\rc = Change (delete and enter insert mode)\ry = Yank (copy) ","date":"22-Nov-2024","heading":"1. Normal Mode (Command Mode)","objectID":"\/posts\/vim\/:4:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#1-normal-mode-command-mode"},{"categories":["software-engineering"],"collections":null,"content":"2. Insert Mode For typing text. Behaves like traditional editors. i = Insert at cursor position\rI = Insert at line beginning\ra = Append after cursor\rA = Append at line end\ro = Open new line below\rO = Open new line above\rs = Substitute character (delete char, enter insert mode)\rS = Substitute entire line\r\" While in Insert Mode, press ESC to return to Normal Mode\r\" Or use CTRL-C as an alternative ","date":"22-Nov-2024","heading":"2. Insert Mode","objectID":"\/posts\/vim\/:4:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#2-insert-mode"},{"categories":["software-engineering"],"collections":null,"content":"3. Visual Mode Select text for operations. v = Character-wise visual mode (select characters)\rV = Line-wise visual mode (select entire lines)\rCTRL-V = Block visual mode (select rectangular block)\r\" In visual mode:\rd = Delete selection\ry = Yank selection\rc = Change selection\ru = Convert to lowercase\rU = Convert to uppercase ","date":"22-Nov-2024","heading":"3. Visual Mode","objectID":"\/posts\/vim\/:4:4","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#3-visual-mode"},{"categories":["software-engineering"],"collections":null,"content":"4. Command Mode (Ex Commands) Execute commands using the colon (:) prefix. :w = Save file\r:q = Quit\r:wq = Save and quit\r:e filename = Open file\r:set option=val = Set configuration\r:%s\/old\/new\/g = Replace all occurrences\r:10,20d = Delete lines 10-20 ","date":"22-Nov-2024","heading":"4. Command Mode (Ex Commands)","objectID":"\/posts\/vim\/:4:5","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#4-command-mode-ex-commands"},{"categories":["software-engineering"],"collections":null,"content":"Essential Motions and Movements\rTip Master These Motions First Motions are the “nouns” in Vim’s grammar. Combine them with operators to create powerful commands. These are your most-used building blocks. ","date":"22-Nov-2024","heading":"Essential Motions and Movements","objectID":"\/posts\/vim\/:5:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#essential-motions"},{"categories":["software-engineering"],"collections":null,"content":"Character-Level Motions h = One character left\rl = One character right\rj = One line down\rk = One line up\rfx = Find character 'x' on this line (forward)\rFx = Find character 'x' on this line (backward)\rtx = Until character 'x' (forward, stops before)\rTx = Until character 'x' (backward, stops before)\r; = Repeat last f\/F\/t\/T motion\r, = Repeat last f\/F\/t\/T motion backwards\r% = Jump to matching bracket\/parenthesis ","date":"22-Nov-2024","heading":"Character-Level Motions","objectID":"\/posts\/vim\/:5:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#character-level-motions"},{"categories":["software-engineering"],"collections":null,"content":"Word Motions w = Next word (beginning of next word)\rW = Next WORD (separated by whitespace only)\re = End of current word\rE = End of current WORD\rb = Beginning of previous word\rB = Beginning of previous WORD\r\" Difference between word and WORD:\r\" word splits on whitespace and punctuation\r\" WORD splits on whitespace only\rExample: foo-bar-baz\rword motions: foo | -bar | -baz\rWORD motions: foo-bar-baz (entire thing is one WORD) ","date":"22-Nov-2024","heading":"Word Motions","objectID":"\/posts\/vim\/:5:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#word-motions"},{"categories":["software-engineering"],"collections":null,"content":"Line Motions 0 = Start of line\r^ = First non-whitespace character\r$ = End of line\rg_ = Last non-whitespace character\r_ = Absolute beginning of line (vim-specific) ","date":"22-Nov-2024","heading":"Line Motions","objectID":"\/posts\/vim\/:5:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#line-motions"},{"categories":["software-engineering"],"collections":null,"content":"Paragraph and Block Motions { = Previous paragraph\r} = Next paragraph\r( = Previous sentence\r) = Next sentence\r[{ = Previous opening brace\r]} = Next closing brace ","date":"22-Nov-2024","heading":"Paragraph and Block Motions","objectID":"\/posts\/vim\/:5:4","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#paragraph-and-block-motions"},{"categories":["software-engineering"],"collections":null,"content":"File Motions gg = Beginning of file\rG = End of file\r:50 = Go to line 50\r50G = Go to line 50 (from normal mode)\rCTRL-G = Show current position in file\r% = Go to percentage of file (50% = halfway) ","date":"22-Nov-2024","heading":"File Motions","objectID":"\/posts\/vim\/:5:5","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#file-motions"},{"categories":["software-engineering"],"collections":null,"content":"Core Operators\r","date":"22-Nov-2024","heading":"Core Operators","objectID":"\/posts\/vim\/:6:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#operators"},{"categories":["software-engineering"],"collections":null,"content":"Essential Operators d (delete) Delete motion and remove from buffer\rc (change) Delete motion and enter insert mode\ry (yank) Copy motion to register\rv (visual) Select motion character-wise\rV (visual) Select motion line-wise ","date":"22-Nov-2024","heading":"Essential Operators","objectID":"\/posts\/vim\/:6:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#essential-operators"},{"categories":["software-engineering"],"collections":null,"content":"Operator Examples dw = Delete from cursor to end of word\rd5w = Delete 5 words\rdd = Delete entire line\rd$ = Delete from cursor to end of line\rdip = Delete entire paragraph\rcw = Change word (delete word, enter insert mode)\rC = Change to end of line (shortcut for c$)\rcc = Change entire line (shortcut for S)\ryy = Copy entire line\ry$ = Copy from cursor to end of line\rynw = Copy next word ","date":"22-Nov-2024","heading":"Operator Examples","objectID":"\/posts\/vim\/:6:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#operator-examples"},{"categories":["software-engineering"],"collections":null,"content":"Text Objects: The Power Multiplier\rInfo Text Objects are Game Changers Text objects let you operate on semantic units (words, sentences, paragraphs, quotes, brackets) rather than just character counts. This is what makes Vim editing feel “smart.” ","date":"22-Nov-2024","heading":"Text Objects: The Power Multiplier","objectID":"\/posts\/vim\/:7:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#text-objects"},{"categories":["software-engineering"],"collections":null,"content":"Text Object Syntax operator + a\/i + object\r\u2191 \u2191\rAction Type\ra = around (include delimiters)\ri = inner (exclude delimiters) ","date":"22-Nov-2024","heading":"Text Object Syntax","objectID":"\/posts\/vim\/:7:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#text-object-syntax"},{"categories":["software-engineering"],"collections":null,"content":"Common Text Objects \" Word objects\riw = inner word (just the word)\raw = a word (word + surrounding whitespace)\r\" WORD objects\riW = inner WORD\raW = a WORD\r\" Paragraph\rip = inner paragraph\rap = a paragraph\r\" Quoted strings\ri', i\", i` = inside single, double, backticks\ra', a\", a` = around quotes\r\" Brackets and parentheses\ri(, i), ib = inside parentheses\ra(, a), ab = around parentheses\ri{, i}, iB = inside braces\ra{, a}, aB = around braces\ri[, i] = inside brackets\ra[, a] = around brackets\ri<, i> = inside angle brackets\ra<, a> = around angle brackets\r\" Tags (for HTML\/XML)\rit = inner tag\rat = a tag ","date":"22-Nov-2024","heading":"Common Text Objects","objectID":"\/posts\/vim\/:7:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#common-text-objects"},{"categories":["software-engineering"],"collections":null,"content":"Real-World Text Object Examples diw = Delete inside word (replace word easily)\rci\" = Change inside quotes (edit string content)\rda{ = Delete around braces (remove function body)\rvip = Select inside paragraph (visual select)\ryip = Copy paragraph\rgUi( = Uppercase inside parentheses\r\" Example: Change function argument\rfunction(arg1, [cursor here], arg3)\rci( \u2192 change inside parentheses\r\u2192 Now type new content, replaces (arg1, arg2, arg3) ","date":"22-Nov-2024","heading":"Real-World Text Object Examples","objectID":"\/posts\/vim\/:7:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#real-world-text-object-examples"},{"categories":["software-engineering"],"collections":null,"content":"Search and Replace\r","date":"22-Nov-2024","heading":"Search and Replace","objectID":"\/posts\/vim\/:8:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#search-replace"},{"categories":["software-engineering"],"collections":null,"content":"Search Operations \/pattern = Search forward for pattern\r?pattern = Search backward for pattern\rn = Next match (forward)\rN = Previous match (backward)\r* = Search for word under cursor (forward)\r# = Search for word under cursor (backward)\r\" Repeat last search\r; (in f\/F\/t\/T) = Repeat last find\r\/ then Enter = Repeat last search pattern ","date":"22-Nov-2024","heading":"Search Operations","objectID":"\/posts\/vim\/:8:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#search-operations"},{"categories":["software-engineering"],"collections":null,"content":"Search Configuration set ignorecase = Ignore case when searching\rset smartcase = Override ignorecase if uppercase present\rset incsearch = Show matches as you type\rset hlsearch = Highlight all matches\r:nohlsearch = Clear highlighting (also :noh) ","date":"22-Nov-2024","heading":"Search Configuration","objectID":"\/posts\/vim\/:8:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#search-configuration"},{"categories":["software-engineering"],"collections":null,"content":"Find and Replace (Substitute) \" Basic syntax: :[range]s\/pattern\/replacement\/[flags]\r:s\/old\/new\/ = Replace first occurrence on current line\r:s\/old\/new\/g = Replace all occurrences on current line\r:%s\/old\/new\/g = Replace all in entire file\r:%s\/old\/new\/gc = Replace all with confirmation\r:5,10s\/old\/new\/ = Replace on lines 5-10\r\" Common flags:\rg = Global (replace all on line, not just first)\rc = Confirm before each replacement\ri = Ignore case\re = Don't show error if pattern not found\r\" Special characters in replacement:\r& = Full match of pattern\r\\1, \\2, etc. = Captured groups (if using regex)\r~ = Previous replacement string ","date":"22-Nov-2024","heading":"Find and Replace (Substitute)","objectID":"\/posts\/vim\/:8:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#find-and-replace-substitute"},{"categories":["software-engineering"],"collections":null,"content":"Regular Expressions \" Basic patterns\r. = Any character\r* = Zero or more of previous\r\\+ = One or more of previous\r\\? = Zero or one of previous\r^ = Start of line\r$ = End of line\r\\s = Whitespace\r\\w = Word character\r\\d = Digit\r\" Examples:\r:%s\/\\d\\+\/NUM\/g = Replace all numbers with \"NUM\"\r:%s\/\\s+$\/\/ = Remove trailing whitespace\r:%s\/^ \/\/ = Remove leading spaces\r:%s\/foo\\(.*\\)bar\/\\1\/ = Extract middle part\r\" Special: Use \\< and \\> for word boundaries\r:%s\/\\\/new\/g = Replace whole word \"old\" only ","date":"22-Nov-2024","heading":"Regular Expressions","objectID":"\/posts\/vim\/:8:4","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#regular-expressions"},{"categories":["software-engineering"],"collections":null,"content":"Registers: Vim's Advanced Clipboard\rNote Registers are Vim’s Clipboard on Steroids While most editors have one clipboard, Vim has 26+ registers! This lets you copy multiple things and paste any of them. Essential for advanced editing. ","date":"22-Nov-2024","heading":"Registers: Vim's Advanced Clipboard","objectID":"\/posts\/vim\/:9:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#registers"},{"categories":["software-engineering"],"collections":null,"content":"Register Types \" Named registers (a-z)\r\"a = Register a\r\"b = Register b\r(... up to \"z)\r\" Special registers\r\"0 = Last yank (copy)\r\"1 to \"9 = Last 9 deletions\r\"\"- = Last small deletion\r\"\"\/ = Last search pattern\r\": = Last command\r\"% = Current filename\r\"# = Alternate filename\r\"+ = System clipboard\r\"* = Selection clipboard (X11)\r\"= = Expression register\r\". = Last inserted text ","date":"22-Nov-2024","heading":"Register Types","objectID":"\/posts\/vim\/:9:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#register-types"},{"categories":["software-engineering"],"collections":null,"content":"Using Registers \" Copying to registers\r\"ayy = Yank line into register a\r\"a5yy = Yank 5 lines into register a\r\"ad5w = Delete 5 words into register a\r\" Pasting from registers\r\"ap = Paste register a\r\"bp = Paste register b\r\"=5+3p = Paste result of expression (8)\r\" Inserting in Insert Mode\ra = Insert contents of register a while in insert mode\r=3*4 = Insert result of expression (12)\r\" Appending to registers (uppercase = append)\r\"Ayy = Append current line to register a\r\"Ad5w = Append 5 words to register a ","date":"22-Nov-2024","heading":"Using Registers","objectID":"\/posts\/vim\/:9:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#using-registers"},{"categories":["software-engineering"],"collections":null,"content":"Practical Register Workflows \" Copy email for later use\r0\r\"ey$ = Copy email (from start of line to end) to register e\r\" Create template by copying structure\r\"tyy = Copy template line to register t\r3j = Move 3 lines down\r\"tp = Paste template 3 times\r\" System clipboard (most useful!)\r\"+yy = Copy line to system clipboard\r\"+p = Paste from system clipboard ","date":"22-Nov-2024","heading":"Practical Register Workflows","objectID":"\/posts\/vim\/:9:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#practical-register-workflows"},{"categories":["software-engineering"],"collections":null,"content":"Macros: Automating Complex Tasks\rTip Macros Let You Record and Replay Editing Sequences Record a series of keystrokes and replay them. Perfect for repetitive tasks like formatting, bulk editing, or complex multi-step operations. ","date":"22-Nov-2024","heading":"Macros: Automating Complex Tasks","objectID":"\/posts\/vim\/:10:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#macros"},{"categories":["software-engineering"],"collections":null,"content":"Recording Macros \" Recording\rq + register = Start recording to register (q for quit recording)\rqa = Start recording to register 'a'\r... (type edits)\rq = Stop recording (press q again)\r\" Playing\r@a = Play macro from register 'a'\r3@a = Play macro 3 times\r@@ = Play last macro (convenient repeat)\r\" Editing macros\r\"ap = Paste macro into buffer (to view it)\r... (edit)\r\"ay$ = Copy edited version back to register\r\" Example: Record complex edit\rqa = Start recording\rI = Insert at beginning\r\/\/ TODO:\r= Back to normal mode\rA (at end) = Append at end\r= Start recording again\rq = Stop recording\rNow \"@a\" will add \"\/\/ TODO:\" comments! ","date":"22-Nov-2024","heading":"Recording Macros","objectID":"\/posts\/vim\/:10:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#recording-macros"},{"categories":["software-engineering"],"collections":null,"content":"Advanced Macro Techniques \" Macro on range (execute on lines 1-10)\r:1,10 normal @a\r\" Macro on all lines matching pattern\r:g\/pattern\/ normal @a\r\" Conditional macro execution\r:if condition\r: normal @a\r:endif\r\" Chain macros (macro calls another macro)\rqa = Start recording\r@b = Call macro b\r@c = Call macro c\rq = Stop recording ","date":"22-Nov-2024","heading":"Advanced Macro Techniques","objectID":"\/posts\/vim\/:10:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#advanced-macro-techniques"},{"categories":["software-engineering"],"collections":null,"content":"Real-World Macro Example \" Original data:\rname:John,age:30,city:NYC\r\" Goal: Convert to JSON format\r{\"name\":\"John\",\"age\":30,\"city\":\"NYC\"}\r\" Macro steps:\rqa = Start recording in register a\r^ = Go to start of line\ri{ = Insert {\r= Return to normal mode\r:s\/:\/ \/g = Replace colons with spaces\r:s\/,\/\", \"\/g = Replace commas with quotes\r... more edits\r$ = Go to end\ra} = Append }\r= Done\rq = Stop recording\r@a = Execute macro ","date":"22-Nov-2024","heading":"Real-World Macro Example","objectID":"\/posts\/vim\/:10:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#real-world-macro-example"},{"categories":["software-engineering"],"collections":null,"content":"Visual Mode and Selection\r","date":"22-Nov-2024","heading":"Visual Mode and Selection","objectID":"\/posts\/vim\/:11:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#visual-mode"},{"categories":["software-engineering"],"collections":null,"content":"Visual Mode Types v = Character-wise visual mode\rSelect individual characters\rV = Line-wise visual mode\rSelect entire lines (most intuitive)\rCTRL-V = Block visual mode\rSelect rectangular blocks (powerful!) ","date":"22-Nov-2024","heading":"Visual Mode Types","objectID":"\/posts\/vim\/:11:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#visual-mode-types"},{"categories":["software-engineering"],"collections":null,"content":"Visual Mode Operations \" Selection in visual mode\rv \/ V \/ CTRL-V = Activate mode\rh,j,k,l = Expand\/contract selection\rw,b,e = Move by words\r$, ^, 0 = Move by line\rG, gg = Move by file\r\" Operators on selection\rd, x = Delete selection\rc = Change selection\ry = Yank selection\r> = Indent right\r< = Indent left\ru = Lowercase\rU = Uppercase\r= = Auto-indent ","date":"22-Nov-2024","heading":"Visual Mode Operations","objectID":"\/posts\/vim\/:11:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#visual-mode-operations"},{"categories":["software-engineering"],"collections":null,"content":"Block Visual Mode Examples \" Add comment prefix to multiple lines\rCTRL-V = Start block select\r3j = Select 3 lines down\r$ = Select to end of lines\r= Now have rectangular selection\rI\/\/ = Insert \"\/\/ \" at beginning of each line\r\" Insert column of text\rCTRL-V = Start block mode\r3j = Select 4 lines\r$ = Select to end\rA: = Append \":\" to end of each line\r\" Replace in column\rCTRL-V = Select rectangle\rc = Change\rnew_text = Type replacement\r= Applied to all lines in selection ","date":"22-Nov-2024","heading":"Block Visual Mode Examples","objectID":"\/posts\/vim\/:11:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#block-visual-mode-examples"},{"categories":["software-engineering"],"collections":null,"content":"Marks and Jumps: Navigate Like a Pro\rTip Marks: Bookmarks for Your Code Marks let you set bookmarks in files and jump between them instantly. Essential for navigating large codebases or editing multiple sections. ","date":"22-Nov-2024","heading":"Marks and Jumps: Navigate Like a Pro","objectID":"\/posts\/vim\/:12:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#marks-jumps"},{"categories":["software-engineering"],"collections":null,"content":"Setting and Using Marks \" Setting marks\rma = Set mark 'a' at current position\rmB = Set mark 'B' at current position\rm[a-z] = Local marks (per-buffer)\rm[A-Z] = Global marks (across files)\r\" Jumping to marks\r`a = Jump to exact position of mark 'a' (line + column)\r'a = Jump to line of mark 'a' (column 0)\r`A = Jump to global mark A (even across files!)\r\" Special marks (automatic)\r`` = Position before last jump\r`. = Last change position\r`\" = Position when last exited file\r`[ = Start of last changed or yanked text\r`] = End of last changed or yanked text\r`< = Start of last visual selection\r`> = End of last visual selection\r\" List all marks\r:marks = Show all active marks\r:delmarks a = Delete mark 'a'\r:delmarks! = Delete all lowercase marks ","date":"22-Nov-2024","heading":"Setting and Using Marks","objectID":"\/posts\/vim\/:12:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#setting-and-using-marks"},{"categories":["software-engineering"],"collections":null,"content":"Jump List Navigation CTRL-O = Jump to older position (jump back)\rCTRL-I = Jump to newer position (jump forward)\r:jumps = Show jump list history\r\" Change list\rg; = Go to older change position\rg, = Go to newer change position\r:changes = Show change list\r\" Practical workflow\r1. Browse code with searches and motions\r2. CTRL-O to jump back through history\r3. CTRL-I to jump forward ","date":"22-Nov-2024","heading":"Jump List Navigation","objectID":"\/posts\/vim\/:12:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#jump-list-navigation"},{"categories":["software-engineering"],"collections":null,"content":"Marks Workflow Diagram Visualizing how marks work across files helps understand their power for navigation: Vim Marks - Setting local and global marks for quick navigation with jump history (Ctrl-O\/Ctrl-I) Remember: Local marks (a-z) are file-specific, while global marks (A-Z) work across files! ","date":"22-Nov-2024","heading":"Marks Workflow Diagram","objectID":"\/posts\/vim\/:12:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#marks-workflow-diagram"},{"categories":["software-engineering"],"collections":null,"content":"Code Folding: Manage Large Files\rNote Folding Collapses Code Sections Like collapsing functions in an IDE. Essential for navigating large files by hiding sections you’re not currently editing. ","date":"22-Nov-2024","heading":"Code Folding: Manage Large Files","objectID":"\/posts\/vim\/:13:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#folding"},{"categories":["software-engineering"],"collections":null,"content":"Folding Commands \" Manual folding\rzf{motion} = Create fold (e.g., zf5j = fold next 5 lines)\rzf% = Fold matching bracket pair\r{Visual}zf = Create fold from visual selection\r\" Opening\/closing folds\rzo = Open fold under cursor\rzc = Close fold under cursor\rza = Toggle fold (open if closed, close if open)\rzO = Open all folds recursively under cursor\rzC = Close all folds recursively under cursor\r\" All folds in buffer\rzM = Close ALL folds in buffer\rzR = Open ALL folds in buffer\rzm = Fold more (close one level)\rzr = Fold less (open one level)\r\" Navigating folds\rzj = Move to next fold\rzk = Move to previous fold\r[z = Move to start of current open fold\r]z = Move to end of current open fold\r\" Delete folds\rzd = Delete fold under cursor\rzD = Delete all folds under cursor recursively\rzE = Eliminate all folds in buffer ","date":"22-Nov-2024","heading":"Folding Commands","objectID":"\/posts\/vim\/:13:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#folding-commands"},{"categories":["software-engineering"],"collections":null,"content":"Folding Methods \" Choose folding method\rset foldmethod=manual \" Manual folding (default)\rset foldmethod=indent \" Fold by indentation level\rset foldmethod=syntax \" Fold based on syntax (for code)\rset foldmethod=expr \" Fold using custom expression\rset foldmethod=marker \" Fold using {{{ }}} markers\r\" Fold configuration\rset foldlevel=0 \" Close all folds by default\rset foldlevelstart=99 \" Start with all folds open\rset foldnestmax=3 \" Maximum fold nesting\r\" Example: Python folding by indent\rautocmd FileType python setlocal foldmethod=indent foldlevel=0 ","date":"22-Nov-2024","heading":"Folding Methods","objectID":"\/posts\/vim\/:13:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#folding-methods"},{"categories":["software-engineering"],"collections":null,"content":"Folding Visualization See how folding transforms a 100-line file into a manageable overview: Vim Code Folding - Collapsing function and class definitions for easier navigation and focus on relevant code sections Pro Tip: Use zM to close all folds and zR to open all folds for quick file overview! ","date":"22-Nov-2024","heading":"Folding Visualization","objectID":"\/posts\/vim\/:13:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#folding-visualization"},{"categories":["software-engineering"],"collections":null,"content":"Window Splits and Layouts\rInfo Multi-Window Editing Edit multiple files simultaneously or view different parts of the same file. Essential for comparing code, referencing documentation while coding, or split-screen workflows. ","date":"22-Nov-2024","heading":"Window Splits and Layouts","objectID":"\/posts\/vim\/:14:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#windows-splits"},{"categories":["software-engineering"],"collections":null,"content":"Window Split Commands \" Creating splits\r:split [file] = Horizontal split (Ctrl-W s)\r:vsplit [file] = Vertical split (Ctrl-W v)\r:new = New horizontal split with empty buffer\r:vnew = New vertical split with empty buffer\r\" Navigating between windows\rCTRL-W h = Move to window on left\rCTRL-W j = Move to window below\rCTRL-W k = Move to window above\rCTRL-W l = Move to window on right\rCTRL-W w = Cycle through windows\rCTRL-W p = Go to previous window\r\" Resizing windows\rCTRL-W = = Make all windows equal size\rCTRL-W _ = Maximize current window height\rCTRL-W | = Maximize current window width\rCTRL-W + = Increase height\rCTRL-W - = Decrease height\rCTRL-W > = Increase width\rCTRL-W < = Decrease width\r:resize 20 = Set window height to 20 lines\r:vertical resize 80 = Set window width to 80 columns\r\" Moving windows\rCTRL-W H = Move window to far left\rCTRL-W J = Move window to bottom\rCTRL-W K = Move window to top\rCTRL-W L = Move window to far right\rCTRL-W r = Rotate windows\rCTRL-W x = Exchange with next window\r\" Closing windows\rCTRL-W q = Quit window (close)\rCTRL-W c = Close window (same as :close)\rCTRL-W o = Close all other windows (only keep current)\r:only = Close all windows except current ","date":"22-Nov-2024","heading":"Window Split Commands","objectID":"\/posts\/vim\/:14:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#window-split-commands"},{"categories":["software-engineering"],"collections":null,"content":"Window Layout Visualization Different layouts for different workflows. Mix horizontal and vertical splits as needed: Vim Window Splits - Creating and managing horizontal and vertical window splits for multi-file editing and comparison Quick Tips: Horizontal split: :split or CTRL-W s Vertical split: :vsplit or CTRL-W v Navigate: Use CTRL-W + direction keys (hjkl) ","date":"22-Nov-2024","heading":"Window Layout Visualization","objectID":"\/posts\/vim\/:14:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#window-layout-visualization"},{"categories":["software-engineering"],"collections":null,"content":"Practical Split Workflow \" Open file in horizontal split, reference while coding\r:split reference.md\r\" Open related files side-by-side\r:vsplit related_code.py\r\" Quick split for viewing same file in two places\rCTRL-W v \" Vertical split same file\r50G \" Go to line 50 in one split\rCTRL-W w \" Switch to other split\r100G \" Go to line 100\r\" Now you can see lines 50 and 100 simultaneously!\r\" Comparing two files\r:split file1.txt\r:vsplit file2.txt\r\" Side-by-side comparison ","date":"22-Nov-2024","heading":"Practical Split Workflow","objectID":"\/posts\/vim\/:14:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#practical-split-workflow"},{"categories":["software-engineering"],"collections":null,"content":"Buffers, Windows, and Tabs Explained\rNote Understanding Vim’s Three-Layer System This is often confusing for beginners. Vim has three concepts: Buffer: A file in memory Window: A view into a buffer Tab: A collection of windows ","date":"22-Nov-2024","heading":"Buffers, Windows, and Tabs Explained","objectID":"\/posts\/vim\/:15:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#buffers-tabs"},{"categories":["software-engineering"],"collections":null,"content":"Buffers vs Windows vs Tabs Understanding the relationship between these three concepts is crucial: Vim Hierarchy - Understanding the relationship between buffers (files in memory), windows (views into buffers), and tabs (collections of windows) Key Insight: One buffer can be displayed in multiple windows simultaneously, and buffers remain in memory even when not visible! ","date":"22-Nov-2024","heading":"Buffers vs Windows vs Tabs","objectID":"\/posts\/vim\/:15:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#buffers-vs-windows-vs-tabs"},{"categories":["software-engineering"],"collections":null,"content":"Buffer Management \" Opening buffers\r:e file.txt = Edit (open) file in current buffer\r:badd file.txt = Add file to buffer list (don't show)\r:buffer N = Switch to buffer number N\r:buffer file = Switch to buffer by name\r\" Navigating buffers\r:ls = List all buffers (shows numbers and status)\r:bnext = Next buffer (also :bn)\r:bprevious = Previous buffer (also :bp)\r:bfirst = First buffer\r:blast = Last buffer\r:b# = Switch to alternate buffer (last viewed)\r\" Buffer info in :ls output\r\" %a = Active buffer (visible in current window)\r\" # = Alternate buffer (last viewed)\r\" + = Modified buffer\r\" - = Unmodifiable buffer\r\" = = Readonly buffer\r\" h = Hidden buffer\r\" Closing buffers\r:bd = Delete (close) current buffer\r:bd 3 = Delete buffer 3\r:bd file.txt = Delete buffer by name\r:%bd = Delete all buffers\r:bufdo bd = Delete all buffers (iterate through)\r\" Working with multiple buffers\r:args file1 file2 file3 = Set argument list\r:argdo %s\/old\/new\/g = Execute command on all args\r:bufdo update = Save all modified buffers ","date":"22-Nov-2024","heading":"Buffer Management","objectID":"\/posts\/vim\/:15:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#buffer-management"},{"categories":["software-engineering"],"collections":null,"content":"Tab Management \" Creating tabs\r:tabnew = New empty tab\r:tabedit file = Open file in new tab\r:tabfind file = Find and open file in new tab\r\" Navigating tabs\rgt = Next tab (also :tabnext)\rgT = Previous tab (also :tabprevious)\r3gt = Go to tab 3\r:tabfirst = Go to first tab\r:tablast = Go to last tab\r\" Moving tabs\r:tabmove 0 = Move current tab to first position\r:tabmove = Move current tab to last position\r:tabmove +1 = Move tab one position right\r:tabmove -1 = Move tab one position left\r\" Closing tabs\r:tabclose = Close current tab\r:tabonly = Close all other tabs ","date":"22-Nov-2024","heading":"Tab Management","objectID":"\/posts\/vim\/:15:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#tab-management"},{"categories":["software-engineering"],"collections":null,"content":"Advanced Search Techniques\r","date":"22-Nov-2024","heading":"Advanced Search Techniques","objectID":"\/posts\/vim\/:16:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#advanced-search"},{"categories":["software-engineering"],"collections":null,"content":"Quick Search Commands * = Search forward for word under cursor\r# = Search backward for word under cursor\rg* = Search forward for partial word match\rg# = Search backward for partial word match\rgd = Go to local declaration\rgD = Go to global declaration\rgf = Go to file under cursor (open file)\rCTRL-] = Jump to tag (requires ctags)\rCTRL-T = Jump back from tag\r\" Searching in multiple files\r:vimgrep \/pattern\/ **\/*.py = Search in all Python files recursively\r:grep pattern **\/*.py = Use external grep\r:cn = Next match\r:cp = Previous match\r:copen = Open quickfix window with results\r:cclose = Close quickfix window ","date":"22-Nov-2024","heading":"Quick Search Commands","objectID":"\/posts\/vim\/:16:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#quick-search-commands"},{"categories":["software-engineering"],"collections":null,"content":"Search Across Multiple Files \" Using vimgrep (built-in)\r:vimgrep \/TODO\/ **\/*.js \" Search all JS files\r:copen \" Open quickfix with results\r:cn \" Jump to next match\r:cp \" Jump to previous match\r\" Using external grep (faster)\r:grep -r \"function\" . \" Recursive grep\r:copen \" View results\r\" Search in open buffers only\r:bufdo vimgrepadd \/pattern\/ %\r\" Location list (per-window quickfix)\r:lvimgrep \/pattern\/ **\/*.py \" Search, use location list\r:lopen \" Open location list\r:lnext \" Next in location list ","date":"22-Nov-2024","heading":"Search Across Multiple Files","objectID":"\/posts\/vim\/:16:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#search-across-multiple-files"},{"categories":["software-engineering"],"collections":null,"content":"Autocomplete and Omnicomplete\rTip Built-in Autocomplete Vim has powerful autocomplete without plugins! Use CTRL-N and CTRL-P for basic completion, or CTRL-X submodes for advanced completion. ","date":"22-Nov-2024","heading":"Autocomplete and Omnicomplete","objectID":"\/posts\/vim\/:17:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#autocomplete"},{"categories":["software-engineering"],"collections":null,"content":"Autocomplete Commands \" In Insert Mode:\rCTRL-N = Next completion (searches forward in file)\rCTRL-P = Previous completion (searches backward)\r\" CTRL-X submodes (press in insert mode)\rCTRL-X CTRL-N = Complete from current buffer\rCTRL-X CTRL-F = Complete filenames\rCTRL-X CTRL-L = Complete whole lines\rCTRL-X CTRL-K = Complete from dictionary\rCTRL-X CTRL-T = Complete from thesaurus\rCTRL-X CTRL-I = Complete from current and included files\rCTRL-X CTRL-] = Complete tags\rCTRL-X CTRL-O = Omnicomplete (language-specific)\r\" Navigate completion menu\rCTRL-N = Select next item\rCTRL-P = Select previous item\rCTRL-Y = Accept selection\rCTRL-E = Cancel completion ","date":"22-Nov-2024","heading":"Autocomplete Commands","objectID":"\/posts\/vim\/:17:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#autocomplete-commands"},{"categories":["software-engineering"],"collections":null,"content":"Autocomplete Configuration \" Basic autocomplete settings\rset completeopt=longest,menuone \" Better completion menu\rset complete+=kspell \" Spell checking suggestions\r\" Enable omnicomplete for programming\rfiletype plugin on\rset omnifunc=syntaxcomplete#Complete\r\" Auto-trigger completion after certain chars\rautocmd FileType python set omnifunc=pythoncomplete#Complete\rautocmd FileType javascript set omnifunc=javascriptcomplete#CompleteJS\rautocmd FileType html set omnifunc=htmlcomplete#CompleteTags\rautocmd FileType css set omnifunc=csscomplete#CompleteCSS\r\" Dictionary completion\rset dictionary=\/usr\/share\/dict\/words ","date":"22-Nov-2024","heading":"Autocomplete Configuration","objectID":"\/posts\/vim\/:17:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#autocomplete-configuration"},{"categories":["software-engineering"],"collections":null,"content":"Sessions: Save Your Workspace\rNote Sessions Save Your Entire Vim State Save your open files, window layouts, tabs, and even command history. Perfect for project-based workflows. ","date":"22-Nov-2024","heading":"Sessions: Save Your Workspace","objectID":"\/posts\/vim\/:18:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#sessions"},{"categories":["software-engineering"],"collections":null,"content":"Session Commands \" Creating sessions\r:mksession ~\/my-project.vim \" Create session file\r:mks ~\/my-project.vim \" Shorthand\r\" Session includes:\r\" - All open buffers\r\" - Window layouts and splits\r\" - Tab pages\r\" - Current directory\r\" - Vim options and mappings\r\" - Marks and registers (optional)\r\" Loading sessions\rvim -S ~\/my-project.vim \" Load session on startup\r:source ~\/my-project.vim \" Load session from Vim\r\" Updating existing session\r:mks! \" Overwrite current session\r\" Session options\rset sessionoptions=blank,buffers,curdir,folds,help,tabpages,winsize\r\" Auto-save session on exit (add to .vimrc)\rautocmd VimLeave * mksession! ~\/.vim\/session\/last.vim ","date":"22-Nov-2024","heading":"Session Commands","objectID":"\/posts\/vim\/:18:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#session-commands"},{"categories":["software-engineering"],"collections":null,"content":"Practical Session Workflow \" Setup for a project\rcd ~\/projects\/myapp\rvim .\r\" Open relevant files\r:e src\/main.py\r:vsplit src\/utils.py\r:split tests\/test_main.py\r\" Arrange windows as needed\rCTRL-W = (equalize sizes)\r\" Save this layout as session\r:mks! ~\/vim-sessions\/myapp.vim\r\" Next day: restore entire workspace\rvim -S ~\/vim-sessions\/myapp.vim\r\" Everything back as it was:\r\" - Same files open\r\" - Same window layout\r\" - Same working directory ","date":"22-Nov-2024","heading":"Practical Session Workflow","objectID":"\/posts\/vim\/:18:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#practical-session-workflow"},{"categories":["software-engineering"],"collections":null,"content":"Diff Mode: Compare Files\r","date":"22-Nov-2024","heading":"Diff Mode: Compare Files","objectID":"\/posts\/vim\/:19:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#diff-mode"},{"categories":["software-engineering"],"collections":null,"content":"Diff Commands \" Start diff mode\rvimdiff file1.txt file2.txt \" From command line\r:vertical diffsplit file2.txt \" From within Vim\r\" Diff navigation\r]c = Jump to next change\r[c = Jump to previous change\r\" Diff operations\rdo = Obtain (get) changes from other file\rdp = Put changes to other file\r:diffget = Same as 'do' (obtain)\r:diffput = Same as 'dp' (put)\r\" Diff options\r:diffupdate = Recalculate diff (if changes occurred)\r:diffoff = Turn off diff mode\r:set diffopt=filler,vertical \" Configuration\r\" Resolve merge conflicts\r:set diffopt+=iwhite \" Ignore whitespace\r:diffget LOCAL \" Get changes from LOCAL\r:diffget REMOTE \" Get changes from REMOTE ","date":"22-Nov-2024","heading":"Diff Commands","objectID":"\/posts\/vim\/:19:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#diff-commands"},{"categories":["software-engineering"],"collections":null,"content":"Diff Visualization Diff Mode Display:\r\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\u2502 file1.txt \u2502 file2.txt \u2502\r\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\r\u2502 line 1: same \u2502 line 1: same \u2502\r\u2502 line 2: different \u2502 line 2: CHANGED \u2502\r\u2502 \u2514\u2500 highlighted red \u2502 \u2514\u2500 highlighted green \u2502\r\u2502 line 3: same \u2502 line 3: same \u2502\r\u2502 line 4: only in file1 \u2502 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2502\r\u2502 \u2514\u2500 shown with \"+\" \u2502 (missing in file2) \u2502\r\u2502 line 5: same \u2502 line 4: same \u2502\r\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\rNavigation:\r]c \u2192 Jump to next difference\r[c \u2192 Jump to previous difference\rdo \u2192 Obtain change from other window\rdp \u2192 Put change to other window ","date":"22-Nov-2024","heading":"Diff Visualization","objectID":"\/posts\/vim\/:19:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#diff-visualization"},{"categories":["software-engineering"],"collections":null,"content":"Spell Checking\r","date":"22-Nov-2024","heading":"Spell Checking","objectID":"\/posts\/vim\/:20:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#spell-check"},{"categories":["software-engineering"],"collections":null,"content":"Spell Check Commands \" Enable\/disable spell checking\r:set spell \" Turn on spell check\r:set nospell \" Turn off spell check\r:set spelllang=en_us \" Set language to US English\r:set spelllang=en_gb \" Set language to British English\r\" Navigate spelling errors\r]s = Next misspelled word\r[s = Previous misspelled word\r]S = Next misspelled word (skip rare words)\r[S = Previous misspelled word (skip rare words)\r\" Fix spelling errors\rz= = Suggest corrections (shows menu)\r1z= = Use first suggestion automatically\rzg = Add word to dictionary (good word)\rzw = Mark word as misspelled (wrong word)\rzug = Undo zg (remove from dictionary)\rzuw = Undo zw\r\" Custom dictionary\r:set spellfile=~\/.vim\/spell\/custom.utf-8.add ","date":"22-Nov-2024","heading":"Spell Check Commands","objectID":"\/posts\/vim\/:20:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#spell-check-commands"},{"categories":["software-engineering"],"collections":null,"content":"Advanced Text Transformations\r","date":"22-Nov-2024","heading":"Advanced Text Transformations","objectID":"\/posts\/vim\/:21:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#text-transformations"},{"categories":["software-engineering"],"collections":null,"content":"Case Transformations ~ = Toggle case of character under cursor\rg~{motion} = Toggle case of motion\rgu{motion} = Lowercase motion\rgU{motion} = Uppercase motion\r\" Examples\rgUiw = Uppercase inner word\rguap = Lowercase paragraph\rg~~ = Toggle case of current line\rgUU = Uppercase current line\rguu = Lowercase current line\r\" In visual mode\rV = Select line\rU = Uppercase selection\ru = Lowercase selection\r~ = Toggle case ","date":"22-Nov-2024","heading":"Case Transformations","objectID":"\/posts\/vim\/:21:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#case-transformations"},{"categories":["software-engineering"],"collections":null,"content":"Text Joining and Splitting J = Join current line with next (removes newline)\r5J = Join next 5 lines\rgJ = Join without adding space\r\" Example:\r\" Before:\r\" line1\r\" line2\r\" line3\r\" After pressing J:\r\" line1 line2\r\" line3\r\" After pressing gJ (no space):\r\" line1line2 ","date":"22-Nov-2024","heading":"Text Joining and Splitting","objectID":"\/posts\/vim\/:21:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#text-joining-and-splitting"},{"categories":["software-engineering"],"collections":null,"content":"Sorting and Filtering \" Sort lines\r:sort = Sort lines alphabetically\r:sort! = Sort in reverse\r:sort n = Sort numerically\r:sort u = Sort and remove duplicates\r\" Sort range\r:10,20sort = Sort lines 10-20\r:'<,'>sort = Sort visual selection\r\" Filter through external command\r:%!sort = Filter entire file through Unix sort\r:%!uniq = Remove duplicate lines\r:10,20!sort = Sort lines 10-20 with Unix sort\r\" Example: Format JSON\r:%!python -m json.tool \" Pretty-print JSON file ","date":"22-Nov-2024","heading":"Sorting and Filtering","objectID":"\/posts\/vim\/:21:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#sorting-and-filtering"},{"categories":["software-engineering"],"collections":null,"content":"Increment and Decrement Numbers CTRL-A = Increment number under cursor\rCTRL-X = Decrement number under cursor\r10CTRL-A = Increment by 10\r\" Visual block increment (create sequences)\rCTRL-V = Block select numbers\rg CTRL-A = Increment sequentially (1,2,3,4...)\rg CTRL-X = Decrement sequentially\r\" Example: Create numbered list\r1 = Start with 1\rCTRL-V = Block select vertically\r5j = Select 5 lines\rg CTRL-A = Now you have: 1,2,3,4,5,6 ","date":"22-Nov-2024","heading":"Increment and Decrement Numbers","objectID":"\/posts\/vim\/:21:4","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#increment-and-decrement-numbers"},{"categories":["software-engineering"],"collections":null,"content":"Quickfix and Location Lists\rTip Quickfix: Central Error\/Search Results Quickfix list is a special window showing errors, search results, or any list of file locations. Essential for compilation errors, search results, and linting. ","date":"22-Nov-2024","heading":"Quickfix and Location Lists","objectID":"\/posts\/vim\/:22:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#quickfix"},{"categories":["software-engineering"],"collections":null,"content":"Quickfix Commands \" Opening quickfix\r:copen = Open quickfix window\r:cclose = Close quickfix window\r:cwindow = Open only if there are errors\r\" Navigation\r:cnext = Next item (also :cn)\r:cprevious = Previous item (also :cp)\r:cfirst = First item\r:clast = Last item\r:cc N = Jump to item N\r\" Loading errors\/results\r:cfile file = Load errors from file\r:cgetfile file = Load errors (don't jump)\r:cexpr expr = Set quickfix from expression\r\" Quickfix for compilation\r:make = Run make, populate quickfix with errors\r:set makeprg=gcc\\ % \" Set compiler command ","date":"22-Nov-2024","heading":"Quickfix Commands","objectID":"\/posts\/vim\/:22:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#quickfix-commands"},{"categories":["software-engineering"],"collections":null,"content":"Location List Commands \" Location list (per-window version of quickfix)\r:lopen = Open location list\r:lclose = Close location list\r:lnext = Next item (ln)\r:lprevious = Previous item (lp)\r:lfirst = First item\r:llast = Last item\r\" Location list is specific to each window\r\" Quickfix is global to all windows ","date":"22-Nov-2024","heading":"Location List Commands","objectID":"\/posts\/vim\/:22:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#location-list-commands"},{"categories":["software-engineering"],"collections":null,"content":"Quickfix Workflow Example \" 1. Search for TODO comments across project\r:vimgrep \/TODO\/ **\/*.py\r\" 2. Open quickfix with results\r:copen\r\" 3. Navigate through results\r:cn \" Next TODO\r:cp \" Previous TODO\r\" 4. Jump directly to item 5\r:cc 5\r\" 5. Close quickfix\r:cclose\r\" Quickfix window looks like:\r\" src\/main.py|23 col 5| # TODO: Fix this bug\r\" src\/utils.py|45 col 8| # TODO: Refactor\r\" tests\/test.py|12 col 3| # TODO: Add test ","date":"22-Nov-2024","heading":"Quickfix Workflow Example","objectID":"\/posts\/vim\/:22:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#quickfix-workflow-example"},{"categories":["software-engineering"],"collections":null,"content":"Command History and Repeat\r","date":"22-Nov-2024","heading":"Command History and Repeat","objectID":"\/posts\/vim\/:23:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#command-history"},{"categories":["software-engineering"],"collections":null,"content":"Command History \" In command mode (:)\r\/ = Navigate command history\r:history = Show command history\r:history \/ = Show search history\rq: = Open command history window (editable!)\rq\/ = Open search history window\rCTRL-F = Open command history from command line\r\" In history window:\r= Execute command under cursor\r:q = Close history window ","date":"22-Nov-2024","heading":"Command History","objectID":"\/posts\/vim\/:23:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#command-history"},{"categories":["software-engineering"],"collections":null,"content":"Repeat Commands . = Repeat last change\r@: = Repeat last command-line command\r@@ = Repeat last @x macro\r\" Practical examples\rdd = Delete line\r. = Delete another line (repeats dd)\r:s\/old\/new\/ = Substitute on one line\r@: = Repeat substitution on current line\r\" Repeat with count\r3. = Repeat last change 3 times\r5@: = Repeat last command 5 times ","date":"22-Nov-2024","heading":"Repeat Commands","objectID":"\/posts\/vim\/:23:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#repeat-commands"},{"categories":["software-engineering"],"collections":null,"content":"Advanced Vim Tricks\r","date":"22-Nov-2024","heading":"Advanced Vim Tricks","objectID":"\/posts\/vim\/:24:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#advanced-tricks"},{"categories":["software-engineering"],"collections":null,"content":"Multiple Cursors Simulation \" Insert at multiple positions\rCTRL-V = Block visual mode\r5j = Select 5 lines\rI = Insert at beginning of all lines\rtext = Type text, press ESC\r\u2192 text appears on all selected lines!\r\" Append at multiple positions\rCTRL-V = Block select\r5j = Select lines\r$ = Go to end of lines\rA = Append at end\rtext = Text appended to all lines\r\" Change multiple lines\rCTRL-V = Block select\r5j = Select lines\rc = Change\rnew_text = Replaced on all lines ","date":"22-Nov-2024","heading":"Multiple Cursors Simulation","objectID":"\/posts\/vim\/:24:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#multiple-cursors-simulation"},{"categories":["software-engineering"],"collections":null,"content":"Global Command Magic \" Syntax: :g\/pattern\/command\r\" Delete all lines containing pattern\r:g\/TODO\/d = Delete all lines with TODO\r\" Delete all lines NOT containing pattern\r:g!\/pattern\/d = Keep only lines with pattern\r:v\/pattern\/d = Same as above\r\" Execute command on matching lines\r:g\/function\/normal @a = Run macro 'a' on lines with 'function'\r:g\/import\/m$ = Move all import lines to end\r\" Copy matching lines\r:g\/error\/t$ = Copy all lines with 'error' to end\r\" Multiple commands\r:g\/pattern\/d | g\/other\/d = Delete lines matching either pattern ","date":"22-Nov-2024","heading":"Global Command Magic","objectID":"\/posts\/vim\/:24:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#global-command-magic"},{"categories":["software-engineering"],"collections":null,"content":"Ex Command Tricks \" Read external command output\r:r!date = Insert current date\r:r!ls = Insert directory listing\r:0r!cat file = Insert file contents at top\r\" Write to command\r:w !sudo tee % = Save file with sudo (when opened without sudo)\r:%!xxd = Convert file to hex dump\r:%!xxd -r = Convert hex back to binary\r\" Execute shell command and read output\r:read !grep pattern file.txt\r\" Substitute tricks\r:%s\/\\v(\\w+)\\s+(\\w+)\/\\2 \\1\/g = Swap two words\r:%s\/\\n\\n\/\\r\/g = Replace double newlines with single\r:%s\/\\s\\+$\/\/ = Remove trailing whitespace ","date":"22-Nov-2024","heading":"Ex Command Tricks","objectID":"\/posts\/vim\/:24:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#ex-command-tricks"},{"categories":["software-engineering"],"collections":null,"content":"Vim Command Line Editing \" In command mode (:)\rCTRL-B = Beginning of line\rCTRL-E = End of line\rCTRL-W = Delete word before cursor\rCTRL-U = Delete to beginning\rCTRL-R \" = Insert contents of register\rCTRL-R % = Insert current filename\rCTRL-R \/ = Insert last search pattern\r\" Example\r:e CTRL-R % = :e currentfile.txt (inserts current filename) ","date":"22-Nov-2024","heading":"Vim Command Line Editing","objectID":"\/posts\/vim\/:24:4","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#vim-command-line-editing"},{"categories":["software-engineering"],"collections":null,"content":"Performance and Optimization\r","date":"22-Nov-2024","heading":"Performance and Optimization","objectID":"\/posts\/vim\/:25:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#performance-tips"},{"categories":["software-engineering"],"collections":null,"content":"Profiling Vim Performance \" Profile startup time\rvim --startuptime startup.log file.txt\r\" Examine startup.log to see what's slow\r\" Profile runtime\r:profile start profile.log\r:profile func *\r:profile file *\r\" Do operations...\r:profile pause\r:noautocmd qall\r\" Check what's taking time\r\" View profile.log ","date":"22-Nov-2024","heading":"Profiling Vim Performance","objectID":"\/posts\/vim\/:25:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#profiling-vim-performance"},{"categories":["software-engineering"],"collections":null,"content":"Optimization Tips \" Speed up for large files\rset synmaxcol=200 \" Don't syntax highlight long lines\rset lazyredraw \" Don't redraw during macros\rset ttyfast \" Fast terminal connection\rset nocursorline \" Disable cursor line highlight\rset norelativenumber \" Disable relative numbers\r\" Disable swap and backup for speed\rset noswapfile\rset nobackup\rset nowritebackup\r\" Limit undo history\rset undolevels=100\r\" Auto-command to disable features for large files\rautocmd BufReadPre * if getfsize(bufname(\"%\")) > 1000000 | setlocal syntax=off | endif ","date":"22-Nov-2024","heading":"Optimization Tips","objectID":"\/posts\/vim\/:25:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#optimization-tips"},{"categories":["software-engineering"],"collections":null,"content":"Advanced Configuration\r","date":"22-Nov-2024","heading":"Advanced Configuration","objectID":"\/posts\/vim\/:26:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#advanced-config"},{"categories":["software-engineering"],"collections":null,"content":"Complete .vimrc Setup \" ============================================================================\r\" GENERAL SETTINGS\r\" ============================================================================\rset nocompatible \" Use Vim defaults, not Vi\rset encoding=utf-8 \" Use UTF-8 encoding\rset fileencoding=utf-8\r\" ============================================================================\r\" DISPLAY\r\" ============================================================================\rsyntax on \" Enable syntax highlighting\rset number \" Show line numbers\rset relativenumber \" Relative numbering (jump distance)\rset cursorline \" Highlight current line\rset termguicolors \" Enable 24-bit colors\rset background=dark \" Dark color scheme\rcolorscheme industry \" Use preferred colorscheme\r\" ============================================================================\r\" INDENTATION\r\" ============================================================================\rset tabstop=4 \" Display tabs as 4 spaces\rset shiftwidth=4 \" Indentation size\rset expandtab \" Use spaces instead of tabs\rset autoindent \" Auto-indent new lines\rset smartindent \" Smart indentation\rset shiftround \" Round indent to shiftwidth\r\" ============================================================================\r\" SEARCH\r\" ============================================================================\rset ignorecase \" Case insensitive search\rset smartcase \" Except uppercase triggers case-sensitive\rset incsearch \" Highlight matches while typing\rset hlsearch \" Highlight all matches\rset wrapscan \" Wrap search around file\r\" ============================================================================\r\" BEHAVIOR\r\" ============================================================================\rset backspace=indent,eol,start \" Allow backspace in insert mode\rset hidden \" Allow unsaved buffers\rset report=0 \" Always report changes\rset completeopt=longest,menuone \" Better autocomplete\rset mouse=a \" Enable mouse support\rset clipboard=unnamedplus \" Use system clipboard\r\" ============================================================================\r\" PERFORMANCE\r\" ============================================================================\rset lazyredraw \" Don't redraw during macros\rset ttyfast \" Faster redraws\rset synmaxcol=200 \" Don't highlight very long lines\r\" ============================================================================\r\" UI ENHANCEMENTS\r\" ============================================================================\rset statusline=%F%m%r%h%w\\ [%{&ff}]\\ %y\\ [%p%%]\\ %l:%c\rset laststatus=2 \" Always show status line\rset showmatch \" Highlight matching brackets\rset showmode \" Show current mode\rset showcmd \" Show partial command\r\" ============================================================================\r\" KEYBINDINGS - Custom mappings\r\" ============================================================================\rlet mapleader = \",\" \" Set leader to comma\r\" Navigate splits easier\rnnoremap j\rnnoremap k\rnnoremap h\rnnoremap l\r\" Quick escape\rinoremap jj inoremap kk \" Clear search highlighting\rnnoremap h :noh\r\" Quick save\rnnoremap w :w\rnnoremap q :q ","date":"22-Nov-2024","heading":"Complete .vimrc Setup","objectID":"\/posts\/vim\/:26:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#complete-vimrc-setup"},{"categories":["software-engineering"],"collections":null,"content":"Plugin Management\r","date":"22-Nov-2024","heading":"Plugin Management","objectID":"\/posts\/vim\/:27:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#plugins"},{"categories":["software-engineering"],"collections":null,"content":"Popular Plugin Managers Note Plugin Ecosystem Vim’s power comes from its plugin ecosystem. Thousands of plugins extend functionality for every workflow imaginable. vim-plug (Recommended - Easy to use): \" Install vim-plug first:\r\" curl -fLo ~\/.vim\/autoload\/plug.vim --create-dirs \\\r\" https:\/\/raw.githubusercontent.com\/junegunn\/vim-plug\/master\/plug.vim\r\" In .vimrc:\rcall plug#begin('~\/.vim\/plugged')\rPlug 'vim-airline\/vim-airline' \" Better statusline\rPlug 'preservim\/nerdtree' \" File explorer\rPlug 'junegunn\/fzf.vim' \" Fuzzy finder\rPlug 'tpope\/vim-surround' \" Surround text objects\rPlug 'tpope\/vim-commentary' \" Easy comments\rPlug 'jiangmiao\/auto-pairs' \" Auto bracket pairing\rPlug 'vim-syntastic\/syntastic' \" Syntax checking\rPlug 'neoclide\/coc.nvim' \" Autocompletion\rcall plug#end()\r\" Install plugins: :PlugInstall\r\" Update plugins: :PlugUpdate\r\" Clean unused: :PlugClean ","date":"22-Nov-2024","heading":"Popular Plugin Managers","objectID":"\/posts\/vim\/:27:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#popular-plugin-managers"},{"categories":["software-engineering"],"collections":null,"content":"Essential Plugin Recommendations Plugin Purpose Use Case vim-airline Enhanced statusline Better visual feedback nerdtree File browser Project file navigation fzf.vim Fuzzy finder Fast file\/buffer search vim-surround Parentheses\/quotes Easily change delimiters vim-commentary Comments Toggle comments easily coc.nvim Autocompletion Intelligent code completion vim-fugitive Git integration Git operations from Vim vim-repeat Repeat plugin actions . works with plugins ","date":"22-Nov-2024","heading":"Essential Plugin Recommendations","objectID":"\/posts\/vim\/:27:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#essential-plugin-recommendations"},{"categories":["software-engineering"],"collections":null,"content":"Introduction to Vimscript\rWarning Vimscript is Quirky Vimscript is Vim’s built-in scripting language. It’s powerful but unconventional (variables need sigils, strings behave oddly, scoping is complex). Many prefer Lua or Python for plugins, but Vimscript is essential for .vimrc customization. ","date":"22-Nov-2024","heading":"Introduction to Vimscript","objectID":"\/posts\/vim\/:28:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#vimscript-intro"},{"categories":["software-engineering"],"collections":null,"content":"Basic Vimscript \" Comments start with double quotes\r\" ===== VARIABLES =====\rlet g:myvar = 10 \" Global variable\rlet l:myvar = 20 \" Local variable\rlet s:myvar = 30 \" Script variable\r\" ===== DATA TYPES =====\rlet string_var = \"hello\"\rlet number_var = 42\rlet float_var = 3.14\rlet list_var = [1, 2, 3, 4]\rlet dict_var = {'name': 'John', 'age': 30}\r\" ===== CONDITIONALS =====\rif condition\recho \"True\"\relseif other_condition\recho \"Also true\"\relse\recho \"False\"\rendif\r\" ===== LOOPS =====\rfor item in list_var\recho item\rendfor\rwhile condition\r\" do something\rendwhile\r\" ===== FUNCTIONS =====\rfunction! MyFunction(arg1, arg2)\rlet result = arg1 + arg2\rreturn result\rendfunction\rcall MyFunction(5, 10) \" Call function\r\" ===== OPERATORS =====\r\" Concatenation uses . not +\rlet full_str = \"Hello\" . \" \" . \"World\" ","date":"22-Nov-2024","heading":"Basic Vimscript","objectID":"\/posts\/vim\/:28:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#basic-vimscript"},{"categories":["software-engineering"],"collections":null,"content":"Custom Commands and Mappings \" ===== CUSTOM COMMANDS =====\rcommand! MyCommand echo \"Custom command executed\"\rcommand! GreetMe call GreetUser(\"Vim User\")\r\" ===== KEYBINDINGS =====\rnnoremap t :tabnew \" Normal mode\rinoremap t :tabnew \" Insert mode\rvnoremap y :'<,'>y+ \" Visual mode\r\" uses mapleader (usually \\\\ or ,)\r\" = Enter\r\" = Escape\r\" = Ctrl+x\r\" = Shift+x\r\" Conditional mapping\rif has('nvim')\rnnoremap t :split\relse\rnnoremap t :tabnew\rendif\r\" ===== USEFUL CONDITIONALS =====\rif has('python') \" Python support\rif has('clipboard') \" Clipboard support\rif has('nvim') \" Neovim specific ","date":"22-Nov-2024","heading":"Custom Commands and Mappings","objectID":"\/posts\/vim\/:28:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#custom-commands-and-mappings"},{"categories":["software-engineering"],"collections":null,"content":"Professional Workflows and Tips\r","date":"22-Nov-2024","heading":"Professional Workflows and Tips","objectID":"\/posts\/vim\/:29:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#pro-tips"},{"categories":["software-engineering"],"collections":null,"content":"Multi-File Editing \" Opening multiple files\rvim file1.txt file2.txt file3.txt\r\" Buffer commands\r:ls \" List all buffers\r:bn \" Next buffer\r:bp \" Previous buffer\r:bd \" Delete (close) buffer\r:b filename \" Go to buffer by name\r:b3 \" Go to buffer 3\r\" Tabs (Vim's \"workspaces\")\r:tabnew \" New tab\r:tabnext \/ gt \" Next tab\r:tabprevious \/ gT \" Previous tab\r:tabclose \" Close tab\r\" Split windows\r:split file.txt \" Horizontal split\r:vsplit file.txt \" Vertical split\rw \" Switch windows\r= \" Equalize window sizes\r_ \" Maximize current window ","date":"22-Nov-2024","heading":"Multi-File Editing","objectID":"\/posts\/vim\/:29:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#multi-file-editing"},{"categories":["software-engineering"],"collections":null,"content":"Working with Large Files \" Disable features for speed\r:set noswapfile \" No swap file\r:set noundofile \" No undo history\r:set synmaxcol=80 \" Limit syntax highlighting column\r:syntax off \" Disable syntax highlighting\r\" Navigate efficiently\r:set nowrap \" Don't wrap lines\r:set lazyredraw \" Lazy redraw\rG \" Go to end\rgg \" Go to start\r50% \" Go to 50% of file\r\" Use grep to find in large files\r:grep pattern filename\r\" Stream editing\r:read !cat largefile | head -100 \" Read first 100 lines only ","date":"22-Nov-2024","heading":"Working with Large Files","objectID":"\/posts\/vim\/:29:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#working-with-large-files"},{"categories":["software-engineering"],"collections":null,"content":"Git Integration \" Using vim-fugitive plugin:\r:Git status \" Show git status\r:Git add % \" Stage current file\r:Git commit \" Commit changes\r:Git push \" Push changes\r:Git diff \" Show diff\r:Git log \" Show commit log\r:Gblame \" Show blame (who edited each line)\r\" Viewing diffs\r:Gvdiffsplit \" Show diff in vertical split\r]c \/ [c \" Navigate changes ","date":"22-Nov-2024","heading":"Git Integration","objectID":"\/posts\/vim\/:29:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#git-integration"},{"categories":["software-engineering"],"collections":null,"content":"Common Issues and Solutions\r","date":"22-Nov-2024","heading":"Common Issues and Solutions","objectID":"\/posts\/vim\/:30:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#troubleshooting"},{"categories":["software-engineering"],"collections":null,"content":"Frequent Problems Problem Solution Stuck in Insert Mode Press ESC or CTRL-C to return to Normal Mode Can’t exit Vim Type :q (quit) or :q! (quit without saving) Accidentally entered Command Mode Press ESC to go back to Normal Mode Keys not responding as expected Make sure you’re in Normal Mode, not Insert Mode Search highlighting won’t clear Run :noh or :nohlsearch Want to undo multiple changes Press u multiple times or 20u for 20 undos ","date":"22-Nov-2024","heading":"Frequent Problems","objectID":"\/posts\/vim\/:30:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#frequent-problems"},{"categories":["software-engineering"],"collections":null,"content":"Performance Issues \" Diagnose startup time\rvim --startuptime startup.log\r\" Disable unneeded features\r:set nocompatible\r:set lazyredraw\r:set ttyfast\r\" Limit syntax highlighting for huge files\rautocmd BufReadPre * if getfsize(bufname('%')) > 10000000 | syntax off | endif\r\" Reduce number of active plugins\r\" Keep only plugins you actually use ","date":"22-Nov-2024","heading":"Performance Issues","objectID":"\/posts\/vim\/:30:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#performance-issues"},{"categories":["software-engineering"],"collections":null,"content":"Recommended Learning Path\r","date":"22-Nov-2024","heading":"Recommended Learning Path","objectID":"\/posts\/vim\/:31:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#learning-path"},{"categories":["software-engineering"],"collections":null,"content":"Phase 1: Fundamentals (Week 1-2) Info Master These First Don’t move to advanced techniques until these are muscle memory. Spend 15-30 minutes daily practicing. Essential Skills: Mode transitions (ESC, i, v, :) Basic motions (hjkl, w, b, $, ^) Operators (d, c, y) Save and quit (:w, :q, :wq) Practice: # Interactive Vim tutorial (built-in)\rvimtutor\r# Practice daily for 2 weeks\r# Open a file and practice moving, editing, saving ","date":"22-Nov-2024","heading":"Phase 1: Fundamentals (Week 1-2)","objectID":"\/posts\/vim\/:31:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#phase-1-fundamentals-week-1-2"},{"categories":["software-engineering"],"collections":null,"content":"Phase 2: Intermediate (Week 3-4) Skills: Text objects (iw, i", i{, ap) Visual mode (v, V, CTRL-V) Search and replace (\/, :%s\/old\/new\/g) Registers (named, special) Macros (qa…q, @a) ","date":"22-Nov-2024","heading":"Phase 2: Intermediate (Week 3-4)","objectID":"\/posts\/vim\/:31:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#phase-2-intermediate-week-3-4"},{"categories":["software-engineering"],"collections":null,"content":"Phase 3: Advanced (Week 5+) Skills: Plugin ecosystem Custom configuration Vimscript basics Advanced workflows Performance optimization ","date":"22-Nov-2024","heading":"Phase 3: Advanced (Week 5+)","objectID":"\/posts\/vim\/:31:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#phase-3-advanced-week-5"},{"categories":["software-engineering"],"collections":null,"content":"Resources for Continued Learning\r","date":"22-Nov-2024","heading":"Resources for Continued Learning","objectID":"\/posts\/vim\/:32:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#vim-resources"},{"categories":["software-engineering"],"collections":null,"content":"Official and Quality Resources Built-in Help: :help or :help topic (comprehensive!) Vim Manual: http:\/\/vimdoc.sourceforge.net\/ Interactive Tutor: vimtutor command Learn Vim the Smart Way: Excellent free online guide Practical Vim by Drew Neil: Best book on Vim Daily Vim Tips: Twitter account @vimtips ","date":"22-Nov-2024","heading":"Official and Quality Resources","objectID":"\/posts\/vim\/:32:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#official-and-quality-resources"},{"categories":["software-engineering"],"collections":null,"content":"Community and Tools r\/vim on Reddit: Active community Stack Overflow: Tag: vim GitHub: Thousands of plugins and configs VimAwesome: Plugin directory ","date":"22-Nov-2024","heading":"Community and Tools","objectID":"\/posts\/vim\/:32:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#community-and-tools"},{"categories":["software-engineering"],"collections":null,"content":"Conclusion: Your Vim Journey Begins\rVim has a steep learning curve, but the investment pays dividends. Once mastered, you’ll edit text faster and more efficiently than ever before. Remember: Tip Key Takeaways Modes are your friend: Normal mode for navigation, Insert mode for typing Grammar is power: Learn operators and motions, then combine them Text objects are essential: They make editing feel smart and intuitive Practice daily: Vim skills compound over time Customize gradually: Start simple, add plugins and config as needed Join the community: Vim users love sharing tips and tricks Start with the basics, practice consistently, and gradually explore advanced features. Within a few weeks, you’ll wonder how you ever edited without Vim. Happy editing! \ud83d\udda4 ","date":"22-Nov-2024","heading":"Conclusion: Your Vim Journey Begins","objectID":"\/posts\/vim\/:33:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#conclusion"},{"categories":["software-engineering"],"collections":null,"content":"Complete Vim Workflow Diagrams\r","date":"22-Nov-2024","heading":"Complete Vim Workflow Diagrams","objectID":"\/posts\/vim\/:34:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#workflow-diagrams"},{"categories":["software-engineering"],"collections":null,"content":"Editing Workflow Progression VIM EDITING WORKFLOW \u2014 From Simple to Complex:\r\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\rLEVEL 1: Basic Editing (First Week)\r\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\u2502 1. vim file.txt \u2192 Open file \u2502\r\u2502 2. i \u2192 Enter insert mode \u2502\r\u2502 3. Type text... \u2192 Add content \u2502\r\u2502 4. ESC \u2192 Back to normal mode \u2502\r\u2502 5. :wq \u2192 Save and quit \u2502\r\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\rLEVEL 2: Motion-Based Editing (Week 2-3)\r\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\u2502 1. vim file.txt \u2192 Open file \u2502\r\u2502 2. \/function \u2192 Search for \"function\" \u2502\r\u2502 3. n \u2192 Next match \u2502\r\u2502 4. ciw \u2192 Change inner word \u2502\r\u2502 5. newname \u2192 Type replacement \u2502\r\u2502 6. ESC \u2192 Back to normal \u2502\r\u2502 7. n \u2192 Next occurrence \u2502\r\u2502 8. . \u2192 Repeat change (ciw) \u2502\r\u2502 9. :wq \u2192 Save and quit \u2502\r\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\rLEVEL 3: Advanced Multi-File Workflow (Week 4+)\r\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\u2502 1. vim -p file1 file2 \u2192 Open in tabs \u2502\r\u2502 2. :vimgrep \/TODO\/ ** \u2192 Search all files \u2502\r\u2502 3. :copen \u2192 View results \u2502\r\u2502 4. :cn \u2192 Next result \u2502\r\u2502 5. ciw \u2192 Change word \u2502\r\u2502 6. qa \u2192 Start recording macro \u2502\r\u2502 7. ...edits... \u2192 Make changes \u2502\r\u2502 8. q \u2192 Stop recording \u2502\r\u2502 9. @a \u2192 Replay on next item \u2502\r\u2502 10. :bufdo update \u2192 Save all buffers \u2502\r\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 ","date":"22-Nov-2024","heading":"Editing Workflow Progression","objectID":"\/posts\/vim\/:34:1","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#editing-workflow-progression"},{"categories":["software-engineering"],"collections":null,"content":"Problem-Solving Decision Tree WHICH VIM FEATURE TO USE?\r\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\rNeed to edit text?\r\u2502\r\u251c\u2500 Single change?\r\u2502 \u251c\u2500 YES \u2192 Use operators (d, c, y) + motions (w, $, })\r\u2502 \u2514\u2500 NO \u2192 Continue below\r\u2502\r\u251c\u2500 Repetitive task?\r\u2502 \u251c\u2500 Same change many times \u2192 Record macro (qa...q, @a)\r\u2502 \u251c\u2500 Pattern-based \u2192 Use :g\/pattern\/command\r\u2502 \u2514\u2500 Multiple files \u2192 :bufdo or :argdo\r\u2502\r\u251c\u2500 Finding code?\r\u2502 \u251c\u2500 In current file \u2192 Use \/ search, *, #\r\u2502 \u251c\u2500 Multiple files \u2192 :vimgrep or :grep\r\u2502 \u251c\u2500 Jump to definition \u2192 gd or CTRL-]\r\u2502 \u2514\u2500 Recent locations \u2192 CTRL-O \/ CTRL-I\r\u2502\r\u251c\u2500 Navigating?\r\u2502 \u251c\u2500 Within file \u2192 Marks (ma, `a)\r\u2502 \u251c\u2500 Between files \u2192 Buffers (:bn, :bp)\r\u2502 \u251c\u2500 Multiple layouts \u2192 Windows (CTRL-W s\/v)\r\u2502 \u2514\u2500 Different projects \u2192 Tabs (gt, gT)\r\u2502\r\u2514\u2500 Large-scale refactor?\r\u251c\u2500 Complex \u2192 Record macro + :g command\r\u251c\u2500 Multiple files \u2192 :bufdo with macro\r\u2514\u2500 Precise pattern \u2192 Regex substitution ","date":"22-Nov-2024","heading":"Problem-Solving Decision Tree","objectID":"\/posts\/vim\/:34:2","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#problem-solving-decision-tree"},{"categories":["software-engineering"],"collections":null,"content":"Real-World Refactoring Example SCENARIO: Rename variable 'oldVar' to 'newVar' across project\r\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\rMETHOD 1: Manual (Small Project)\r\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\u2502 1. \/oldVar Search for oldVar \u2502\r\u2502 2. * Highlight all occurrences \u2502\r\u2502 3. ciw Change inner word \u2502\r\u2502 4. newVar Type new name \u2502\r\u2502 5. n Next occurrence \u2502\r\u2502 6. . Repeat change \u2502\r\u2502 7. Repeat steps 5-6... \u2502\r\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\rMETHOD 2: Macro (Medium Project)\r\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\u2502 1. \/oldVar Search first occurrence \u2502\r\u2502 2. qa Start recording to register a \u2502\r\u2502 3. ciwn ewVar Change word \u2502\r\u2502 4. n Next occurrence \u2502\r\u2502 5. q Stop recording \u2502\r\u2502 6. 100@a Replay 100 times (safe) \u2502\r\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\rMETHOD 3: Substitution (Large Project)\r\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\u2502 1. :args **\/*.js Add all JS files to arglist \u2502\r\u2502 2. :argdo %s\/\\\/newVar\/gc \u2502\r\u2502 Substitute with confirmation \u2502\r\u2502 3. :argdo update Save all changed files \u2502\r\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\rMETHOD 4: Advanced (Huge Project)\r\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\u2502 1. :vimgrep \/\\\/ **\/*.js \u2502\r\u2502 Find all occurrences \u2502\r\u2502 2. :copen Open quickfix with results \u2502\r\u2502 3. :cfdo %s\/\\\/newVar\/g \u2502\r\u2502 Substitute in all files \u2502\r\u2502 4. :cfdo update Save all changed files \u2502\r\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 ","date":"22-Nov-2024","heading":"Real-World Refactoring Example","objectID":"\/posts\/vim\/:34:3","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#real-world-refactoring-example"},{"categories":["software-engineering"],"collections":null,"content":"Text Object Combination Matrix TEXT OBJECT POWER COMBINATIONS\r\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\rOPERATOR \u00d7 TEXT OBJECT = POWERFUL COMMANDS\r\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\r\u2502 iw \u2502 aw \u2502 i\" \u2502 i( \u2502 i{ \u2502 ap \u2502\r\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\rd \u2502 diw \u2502 daw \u2502 di\" \u2502 di( \u2502 di{ \u2502 dap \u2502\r\u2502 delete\u2502delete \u2502delete \u2502delete \u2502delete \u2502delete \u2502\r\u2502 word \u2502 word+ \u2502inside \u2502inside \u2502inside \u2502para \u2502\r\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\rc \u2502 ciw \u2502 caw \u2502 ci\" \u2502 ci( \u2502 ci{ \u2502 cap \u2502\r\u2502change \u2502change \u2502change \u2502change \u2502change \u2502change \u2502\r\u2502 word \u2502 word+ \u2502quote \u2502parens \u2502braces \u2502para \u2502\r\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\ry \u2502 yiw \u2502 yaw \u2502 yi\" \u2502 yi( \u2502 yi{ \u2502 yap \u2502\r\u2502 yank \u2502 yank \u2502 yank \u2502 yank \u2502 yank \u2502 yank \u2502\r\u2502 word \u2502 word+ \u2502quote \u2502parens \u2502braces \u2502para \u2502\r\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\rv \u2502 viw \u2502 vaw \u2502 vi\" \u2502 vi( \u2502 vi{ \u2502 vap \u2502\r\u2502select \u2502select \u2502select \u2502select \u2502select \u2502select \u2502\r\u2502 word \u2502 word+ \u2502quote \u2502parens \u2502braces \u2502para \u2502\r\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\rgU \u2502 gUiw \u2502 gUaw \u2502 gUi\" \u2502 gUi( \u2502 gUi{ \u2502 gUap \u2502\r\u2502UPPER \u2502UPPER \u2502UPPER \u2502UPPER \u2502UPPER \u2502UPPER \u2502\r\u2502 word \u2502 word+ \u2502quote \u2502parens \u2502braces \u2502para \u2502\r\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\rMost useful combinations:\r\u2022 ciw - Change inner word (rename variable)\r\u2022 di\" - Delete inside quotes (remove string content)\r\u2022 ci( - Change inside parentheses (edit function args)\r\u2022 dap - Delete paragraph (remove code block)\r\u2022 yap - Yank paragraph (copy function\/block) ","date":"22-Nov-2024","heading":"Text Object Combination Matrix","objectID":"\/posts\/vim\/:34:4","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#text-object-combination-matrix"},{"categories":["software-engineering"],"collections":null,"content":"Quick Reference Card \u2554\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2557\r\u2551 VIM COMPLETE QUICK REFERENCE \u2551\r\u255a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u255d\r\u250c\u2500 MODES \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\u2502 ESC Return to Normal mode (from any mode) \u2502\r\u2502 i\/I Insert before cursor \/ at line start \u2502\r\u2502 a\/A Append after cursor \/ at line end \u2502\r\u2502 o\/O Open line below \/ above \u2502\r\u2502 v\/V\/Ctrl-V Visual: character \/ line \/ block \u2502\r\u2502 : Command mode (Ex commands) \u2502\r\u2502 R Replace mode (overwrite) \u2502\r\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\u250c\u2500 MOVEMENT \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\u2502 h\/j\/k\/l Left \/ Down \/ Up \/ Right \u2502\r\u2502 w\/W Next word \/ WORD \u2502\r\u2502 b\/B Previous word \/ WORD \u2502\r\u2502 e\/E End of word \/ WORD \u2502\r\u2502 0\/^\/$ Start of line \/ First non-blank \/ End \u2502\r\u2502 gg\/G Start \/ end of file \u2502\r\u2502 {\/} Previous \/ next paragraph \u2502\r\u2502 % Jump to matching bracket \u2502\r\u2502 fx\/Fx Find char 'x' forward \/ backward \u2502\r\u2502 *\/# Search word under cursor forward \/ backward \u2502\r\u2502 Ctrl-O\/Ctrl-I Jump to older \/ newer position \u2502\r\u2502 ma \/ `a Set mark 'a' \/ Jump to mark 'a' \u2502\r\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\u250c\u2500 EDITING \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\u2502 x\/X Delete char at \/ before cursor \u2502\r\u2502 dd Delete line \u2502\r\u2502 D Delete to end of line \u2502\r\u2502 d{motion} Delete motion (dw, d$, dip, di\") \u2502\r\u2502 c{motion} Change motion (delete + insert mode) \u2502\r\u2502 y{motion} Yank (copy) motion (yy = yank line) \u2502\r\u2502 p\/P Paste after \/ before cursor \u2502\r\u2502 u\/Ctrl-R Undo \/ Redo \u2502\r\u2502 . Repeat last change \u2502\r\u2502 J Join lines \u2502\r\u2502 ~ Toggle case \u2502\r\u2502 gU\/gu{motion} Uppercase \/ lowercase motion \u2502\r\u2502 Ctrl-A\/Ctrl-X Increment \/ decrement number \u2502\r\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\u250c\u2500 TEXT OBJECTS (use with operators: d, c, y, v, gU, etc.) \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\u2502 iw\/aw Inner word \/ a word (with space) \u2502\r\u2502 i\"\/a\" Inside \/ around double quotes \u2502\r\u2502 i'\/a' Inside \/ around single quotes \u2502\r\u2502 i(\/a( or ib\/ab Inside \/ around parentheses \u2502\r\u2502 i{\/a{ or iB\/aB Inside \/ around braces \u2502\r\u2502 i[\/a[ Inside \/ around brackets \u2502\r\u2502 it\/at Inside \/ around HTML\/XML tag \u2502\r\u2502 ip\/ap Inner \/ a paragraph \u2502\r\u2502 is\/as Inner \/ a sentence \u2502\r\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\u250c\u2500 SEARCH & REPLACE \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\u2502 \/pattern Search forward \u2502\r\u2502 ?pattern Search backward \u2502\r\u2502 n\/N Next \/ previous match \u2502\r\u2502 *\/# Search word under cursor forward \/ backward \u2502\r\u2502 :noh Clear search highlighting \u2502\r\u2502 :s\/old\/new\/ Replace first on line \u2502\r\u2502 :s\/old\/new\/g Replace all on line \u2502\r\u2502 :%s\/old\/new\/g Replace all in file \u2502\r\u2502 :%s\/old\/new\/gc Replace all with confirmation \u2502\r\u2502 :10,20s\/old\/new\/g Replace in lines 10-20 \u2502\r\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\u250c\u2500 FILES & BUFFERS \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\u2502 :w Save file \u2502\r\u2502 :w filename Save as filename \u2502\r\u2502 :q Quit (fails if unsaved) \u2502\r\u2502 :q! Quit without saving \u2502\r\u2502 :wq or :x Save and quit \u2502\r\u2502 :e filename Open file \u2502\r\u2502 :ls List buffers \u2502\r\u2502 :bn\/:bp Next \/ previous buffer \u2502\r\u2502 :bd Delete (close) buffer \u2502\r\u2502 :b3 Switch to buffer 3 \u2502\r\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\u250c\u2500 WINDOWS & TABS \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\u2502 :split\/:sp Horizontal split \u2502\r\u2502 :vsplit\/:vsp Vertical split \u2502\r\u2502 Ctrl-W s\/v Split horizontal \/ vertical \u2502\r\u2502 Ctrl-W h\/j\/k\/l Navigate to window left\/down\/up\/right \u2502\r\u2502 Ctrl-W w Cycle windows \u2502\r\u2502 Ctrl-W q Close window \u2502\r\u2502 Ctrl-W = Equalize window sizes \u2502\r\u2502 :tabnew New tab \u2502\r\u2502 gt\/gT Next \/ previous tab \u2502\r\u2502 :tabclose Close tab \u2502\r\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\u250c\u2500 VISUAL MODE \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\r\u2502 v Character-wise visual \u2502\r\u2502 V Line-wise visual \u2502\r\u2502 Ctrl-V Block visual (rectangular selection) \u2502\r\u2502 o Jump to other end of selection \u2502\r\u2502 >\/<< Indent right \/ left \u2502\r\u2502 d\/c\/y Delete ","date":"22-Nov-2024","heading":"Quick Reference Card","objectID":"\/posts\/vim\/:35:0","tags":["vim","terminal","productivity"],"title":"Vim Notes \u2014 Modal Text Editing and Efficient Navigation","uri":"\/posts\/vim\/#quick-reference-card"},{"categories":["software-engineering"],"collections":null,"content":"Comprehensive Git and GitHub reference covering version control fundamentals, branching strategies, pull requests, GitHub Actions, and team collaboration workflows with practical examples and real-world scenarios.","date":"20-Nov-2024","objectID":"\/posts\/git-github\/","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/"},{"categories":["software-engineering"],"collections":null,"content":" Why Learn Git and GitHub? Git and GitHub are fundamental to modern software development. Whether you’re contributing to open-source projects, working in a team, or building your own products, understanding version control is essential for: Tracking changes in your codebase over time Collaborating safely with multiple developers Reversing mistakes without losing work Understanding code history and decisions Automating deployment through CI\/CD pipelines Contributing to open-source communities This guide takes you from complete beginner to advanced user, with practical examples you can use immediately. ","date":"20-Nov-2024","heading":"","objectID":"\/posts\/git-github\/:0:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#"},{"categories":["software-engineering"],"collections":null,"content":"1. Introduction to Git and Version Control ","date":"20-Nov-2024","heading":"1. Introduction to Git and Version Control","objectID":"\/posts\/git-github\/:1:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#1-introduction-to-git-and-version-control"},{"categories":["software-engineering"],"collections":null,"content":"1.1 What is Git? Git is a distributed version control system (VCS) designed by Linus Torvalds in 2005. It tracks changes in files, enables collaboration, and maintains complete project history. Key Characteristics: Distributed: Every developer has a complete copy of the repository Fast: Most operations happen locally without server communication Flexible: Supports various workflows and branching strategies Safe: Cryptographic integrity checking prevents data corruption Powerful: Complex branching, merging, and history rewriting capabilities ","date":"20-Nov-2024","heading":"1.1 What is Git?","objectID":"\/posts\/git-github\/:1:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#11-what-is-git"},{"categories":["software-engineering"],"collections":null,"content":"1.2 Git vs GitHub These terms are often confused but are distinct: Aspect Git GitHub Type Version Control System Web hosting platform Location Runs locally on your computer Cloud-based service Purpose Track and manage code changes Host, collaborate, and share repositories Scope Local version control Remote repository + collaboration tools Dependencies Standalone tool Requires Git to function Features Branching, merging, staging Pull requests, issues, actions, wiki Tip Think of it this way: Git is the engine that manages your code versions, while GitHub is the garage where you park your code and invite others to contribute. ","date":"20-Nov-2024","heading":"1.2 Git vs GitHub","objectID":"\/posts\/git-github\/:1:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#12-git-vs-github"},{"categories":["software-engineering"],"collections":null,"content":"1.3 Core Concepts You Need to Know The Three-State Model Git has three main states where your files reside: 1. Working Directory (Modified) Your local files that you actively edit.\rStatus: Untracked or Modified 2. Staging Area (Staged) Files marked for inclusion in the next commit.\rStatus: Staged 3. Git Repository (Committed) Permanent snapshots stored in .git directory.\rStatus: Committed Flow: Modify Files (Working) \u2192 Stage Changes \u2192 Commit to Repository The Git Object Model Git stores data as a directed acyclic graph (DAG) of four main object types: Commit - points to tree + parent commit + author info\r\u2193\rTree - represents directory structure\r\u2193\rBlob - actual file contents (immutable)\r\u2193\rTag - reference to specific commit (optional) Everything is Content-Addressable: Each object has a unique SHA-1 hash based on its content Even tiny changes produce completely different hashes Impossible to change history without detection ","date":"20-Nov-2024","heading":"1.3 Core Concepts You Need to Know","objectID":"\/posts\/git-github\/:1:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#13-core-concepts-you-need-to-know"},{"categories":["software-engineering"],"collections":null,"content":"1.3 Core Concepts You Need to Know The Three-State Model Git has three main states where your files reside: 1. Working Directory (Modified) Your local files that you actively edit.\rStatus: Untracked or Modified 2. Staging Area (Staged) Files marked for inclusion in the next commit.\rStatus: Staged 3. Git Repository (Committed) Permanent snapshots stored in .git directory.\rStatus: Committed Flow: Modify Files (Working) \u2192 Stage Changes \u2192 Commit to Repository The Git Object Model Git stores data as a directed acyclic graph (DAG) of four main object types: Commit - points to tree + parent commit + author info\r\u2193\rTree - represents directory structure\r\u2193\rBlob - actual file contents (immutable)\r\u2193\rTag - reference to specific commit (optional) Everything is Content-Addressable: Each object has a unique SHA-1 hash based on its content Even tiny changes produce completely different hashes Impossible to change history without detection ","date":"20-Nov-2024","heading":"The Three-State Model","objectID":"\/posts\/git-github\/:1:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#the-three-state-model"},{"categories":["software-engineering"],"collections":null,"content":"1.3 Core Concepts You Need to Know The Three-State Model Git has three main states where your files reside: 1. Working Directory (Modified) Your local files that you actively edit.\rStatus: Untracked or Modified 2. Staging Area (Staged) Files marked for inclusion in the next commit.\rStatus: Staged 3. Git Repository (Committed) Permanent snapshots stored in .git directory.\rStatus: Committed Flow: Modify Files (Working) \u2192 Stage Changes \u2192 Commit to Repository The Git Object Model Git stores data as a directed acyclic graph (DAG) of four main object types: Commit - points to tree + parent commit + author info\r\u2193\rTree - represents directory structure\r\u2193\rBlob - actual file contents (immutable)\r\u2193\rTag - reference to specific commit (optional) Everything is Content-Addressable: Each object has a unique SHA-1 hash based on its content Even tiny changes produce completely different hashes Impossible to change history without detection ","date":"20-Nov-2024","heading":"The Git Object Model","objectID":"\/posts\/git-github\/:1:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#the-git-object-model"},{"categories":["software-engineering"],"collections":null,"content":"2. Getting Started with Git ","date":"20-Nov-2024","heading":"2. Getting Started with Git","objectID":"\/posts\/git-github\/:2:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#2-getting-started-with-git"},{"categories":["software-engineering"],"collections":null,"content":"2.1 Installation macOS (using Homebrew): brew install git\rgit --version Ubuntu\/Debian: sudo apt update\rsudo apt install git\rgit --version Windows: # Download from: https:\/\/git-scm.com\/download\/win\r# Run installer and follow prompts\r# Verify installation\rgit --version ","date":"20-Nov-2024","heading":"2.1 Installation","objectID":"\/posts\/git-github\/:2:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#21-installation"},{"categories":["software-engineering"],"collections":null,"content":"2.2 Initial Configuration After installation, configure Git with your identity: # Set your name (appears in commits)\rgit config --global user.name \"Your Name\"\r# Set your email (appears in commits)\rgit config --global user.email \"you@example.com\"\r# Set default editor for commit messages\rgit config --global core.editor \"vim\"\r# Verify configuration\rgit config --list\r# View only global settings\rgit config --global --list\r# View configuration files\rcat ~\/.gitconfig Note Global vs Local Configuration: --global flag applies to all repositories on your machine. Use local configuration (without --global) for project-specific settings. Useful Global Configuration: # Enable colored output\rgit config --global color.ui true\r# Set default branch name to main\rgit config --global init.defaultBranch main\r# Enable automatic garbage collection\rgit config --global gc.auto 4096\r# Configure paging (less) for large outputs\rgit config --global core.pager \"less -x1,5\"\r# Add helpful aliases\rgit config --global alias.st status\rgit config --global alias.co checkout\rgit config --global alias.br branch\rgit config --global alias.ci commit\rgit config --global alias.unstage 'restore --staged'\rgit config --global alias.last 'log -1 HEAD'\rgit config --global alias.visual 'log --graph --oneline --all' ","date":"20-Nov-2024","heading":"2.2 Initial Configuration","objectID":"\/posts\/git-github\/:2:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#22-initial-configuration"},{"categories":["software-engineering"],"collections":null,"content":"2.3 First Repository Initialize a New Repository # Create a new directory\rmkdir my-project\rcd my-project\r# Initialize Git repository\rgit init\r# Verify .git directory was created\rls -la When you run git init, Git creates a hidden .git directory containing: .git\/\r\u251c\u2500\u2500 objects\/ # Stores all commits, trees, blobs\r\u251c\u2500\u2500 refs\/ # References to branches and tags\r\u251c\u2500\u2500 HEAD # Points to current branch\r\u251c\u2500\u2500 config # Repository configuration\r\u251c\u2500\u2500 hooks\/ # Scripts that run on Git events\r\u2514\u2500\u2500 logs\/ # Reference logs Set Repository Identity (Optional) # View current config (inherits from global)\rgit config --local user.name\r# Override with project-specific settings\rgit config --local user.name \"Project-Specific Name\"\rgit config --local user.email \"project@example.com\"\r# These only apply to this repository ","date":"20-Nov-2024","heading":"2.3 First Repository","objectID":"\/posts\/git-github\/:2:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#23-first-repository"},{"categories":["software-engineering"],"collections":null,"content":"2.3 First Repository Initialize a New Repository # Create a new directory\rmkdir my-project\rcd my-project\r# Initialize Git repository\rgit init\r# Verify .git directory was created\rls -la When you run git init, Git creates a hidden .git directory containing: .git\/\r\u251c\u2500\u2500 objects\/ # Stores all commits, trees, blobs\r\u251c\u2500\u2500 refs\/ # References to branches and tags\r\u251c\u2500\u2500 HEAD # Points to current branch\r\u251c\u2500\u2500 config # Repository configuration\r\u251c\u2500\u2500 hooks\/ # Scripts that run on Git events\r\u2514\u2500\u2500 logs\/ # Reference logs Set Repository Identity (Optional) # View current config (inherits from global)\rgit config --local user.name\r# Override with project-specific settings\rgit config --local user.name \"Project-Specific Name\"\rgit config --local user.email \"project@example.com\"\r# These only apply to this repository ","date":"20-Nov-2024","heading":"Initialize a New Repository","objectID":"\/posts\/git-github\/:2:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#initialize-a-new-repository"},{"categories":["software-engineering"],"collections":null,"content":"2.3 First Repository Initialize a New Repository # Create a new directory\rmkdir my-project\rcd my-project\r# Initialize Git repository\rgit init\r# Verify .git directory was created\rls -la When you run git init, Git creates a hidden .git directory containing: .git\/\r\u251c\u2500\u2500 objects\/ # Stores all commits, trees, blobs\r\u251c\u2500\u2500 refs\/ # References to branches and tags\r\u251c\u2500\u2500 HEAD # Points to current branch\r\u251c\u2500\u2500 config # Repository configuration\r\u251c\u2500\u2500 hooks\/ # Scripts that run on Git events\r\u2514\u2500\u2500 logs\/ # Reference logs Set Repository Identity (Optional) # View current config (inherits from global)\rgit config --local user.name\r# Override with project-specific settings\rgit config --local user.name \"Project-Specific Name\"\rgit config --local user.email \"project@example.com\"\r# These only apply to this repository ","date":"20-Nov-2024","heading":"Set Repository Identity (Optional)","objectID":"\/posts\/git-github\/:2:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#set-repository-identity-optional"},{"categories":["software-engineering"],"collections":null,"content":"3. Daily Workflow: The Essential Commands ","date":"20-Nov-2024","heading":"3. Daily Workflow: The Essential Commands","objectID":"\/posts\/git-github\/:3:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#3-daily-workflow-the-essential-commands"},{"categories":["software-engineering"],"collections":null,"content":"3.1 Checking Status The most important command to understand your repository state: # Full status output\rgit status\r# Short format (more concise)\rgit status -s\r# Include branch tracking info\rgit status -sb\r# Show untracked files separately\rgit status --short Status Output Legend: M = Modified\rA = Added\rD = Deleted\rR = Renamed\rC = Copied\rU = Updated but unmerged\r?? = Untracked\rFirst column = staging area\rSecond column = working directory ","date":"20-Nov-2024","heading":"3.1 Checking Status","objectID":"\/posts\/git-github\/:3:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#31-checking-status"},{"categories":["software-engineering"],"collections":null,"content":"3.2 Staging Changes Staging is how you select which changes to include in your next commit. Adding Specific Files # Stage specific file\rgit add filename.txt\r# Stage multiple files\rgit add file1.js file2.js file3.js\r# Stage all changes in current directory\rgit add .\r# Stage all changes everywhere\rgit add -A\r# Stage only modified files (not new files)\rgit add -u\r# Stage with confirmation (interactive)\rgit add -i Partial Staging # Interactive patch staging - choose hunks to stage\rgit add -p\r# View what you staged\rgit diff --staged\r# Unstage specific file\rgit restore --staged filename.txt\r# Unstage all changes\rgit restore --staged . Tip Staging Best Practices: Stage related changes together in logical units. This makes commits more meaningful and makes history easier to understand. Use git add -p to review changes before staging. ","date":"20-Nov-2024","heading":"3.2 Staging Changes","objectID":"\/posts\/git-github\/:3:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#32-staging-changes"},{"categories":["software-engineering"],"collections":null,"content":"3.2 Staging Changes Staging is how you select which changes to include in your next commit. Adding Specific Files # Stage specific file\rgit add filename.txt\r# Stage multiple files\rgit add file1.js file2.js file3.js\r# Stage all changes in current directory\rgit add .\r# Stage all changes everywhere\rgit add -A\r# Stage only modified files (not new files)\rgit add -u\r# Stage with confirmation (interactive)\rgit add -i Partial Staging # Interactive patch staging - choose hunks to stage\rgit add -p\r# View what you staged\rgit diff --staged\r# Unstage specific file\rgit restore --staged filename.txt\r# Unstage all changes\rgit restore --staged . Tip Staging Best Practices: Stage related changes together in logical units. This makes commits more meaningful and makes history easier to understand. Use git add -p to review changes before staging. ","date":"20-Nov-2024","heading":"Adding Specific Files","objectID":"\/posts\/git-github\/:3:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#adding-specific-files"},{"categories":["software-engineering"],"collections":null,"content":"3.2 Staging Changes Staging is how you select which changes to include in your next commit. Adding Specific Files # Stage specific file\rgit add filename.txt\r# Stage multiple files\rgit add file1.js file2.js file3.js\r# Stage all changes in current directory\rgit add .\r# Stage all changes everywhere\rgit add -A\r# Stage only modified files (not new files)\rgit add -u\r# Stage with confirmation (interactive)\rgit add -i Partial Staging # Interactive patch staging - choose hunks to stage\rgit add -p\r# View what you staged\rgit diff --staged\r# Unstage specific file\rgit restore --staged filename.txt\r# Unstage all changes\rgit restore --staged . Tip Staging Best Practices: Stage related changes together in logical units. This makes commits more meaningful and makes history easier to understand. Use git add -p to review changes before staging. ","date":"20-Nov-2024","heading":"Partial Staging","objectID":"\/posts\/git-github\/:3:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#partial-staging"},{"categories":["software-engineering"],"collections":null,"content":"3.3 Creating Commits A commit is a permanent snapshot of your staged changes. # Simple commit with message\rgit commit -m \"Add user authentication feature\"\r# Multi-line commit message (opens editor)\rgit commit -m \"Add authentication\" -m \"Implement JWT token validation\"\r# Automatically stage tracked files and commit\rgit commit -am \"Fix login bug\"\r# Open editor for detailed commit message\rgit commit\r# Amend previous commit (reuse same message)\rgit commit --amend --no-edit\r# Amend and edit message\rgit commit --amend Commit Message Best Practices Write commit messages that explain the why, not just the what: # \u274c BAD: What is being done\rgit commit -m \"Update login\"\r# \u274c BAD: Too vague\rgit commit -m \"Fix stuff\"\r# \u2705 GOOD: What and why\rgit commit -m \"Implement JWT authentication for API endpoints\r- Add token generation on login\r- Validate tokens on protected routes\r- Store refresh tokens securely\r- Fixes #123\"\r# \u2705 GOOD: Conventional Commits format\rgit commit -m \"feat(auth): implement JWT token validation\rEnables secure API authentication for user endpoints.\rTokens expire after 24 hours and include refresh mechanism.\rCloses #456\" Conventional Commits Format (widely adopted): type(scope): subject\rbody\rfooter Where type is one of: feat: New feature fix: Bug fix docs: Documentation changes style: Code style (formatting, missing semicolons, etc.) refactor: Code refactoring without feature changes perf: Performance improvements test: Adding or updating tests chore: Build process, dependencies, tooling ","date":"20-Nov-2024","heading":"3.3 Creating Commits","objectID":"\/posts\/git-github\/:3:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#33-creating-commits"},{"categories":["software-engineering"],"collections":null,"content":"3.3 Creating Commits A commit is a permanent snapshot of your staged changes. # Simple commit with message\rgit commit -m \"Add user authentication feature\"\r# Multi-line commit message (opens editor)\rgit commit -m \"Add authentication\" -m \"Implement JWT token validation\"\r# Automatically stage tracked files and commit\rgit commit -am \"Fix login bug\"\r# Open editor for detailed commit message\rgit commit\r# Amend previous commit (reuse same message)\rgit commit --amend --no-edit\r# Amend and edit message\rgit commit --amend Commit Message Best Practices Write commit messages that explain the why, not just the what: # \u274c BAD: What is being done\rgit commit -m \"Update login\"\r# \u274c BAD: Too vague\rgit commit -m \"Fix stuff\"\r# \u2705 GOOD: What and why\rgit commit -m \"Implement JWT authentication for API endpoints\r- Add token generation on login\r- Validate tokens on protected routes\r- Store refresh tokens securely\r- Fixes #123\"\r# \u2705 GOOD: Conventional Commits format\rgit commit -m \"feat(auth): implement JWT token validation\rEnables secure API authentication for user endpoints.\rTokens expire after 24 hours and include refresh mechanism.\rCloses #456\" Conventional Commits Format (widely adopted): type(scope): subject\rbody\rfooter Where type is one of: feat: New feature fix: Bug fix docs: Documentation changes style: Code style (formatting, missing semicolons, etc.) refactor: Code refactoring without feature changes perf: Performance improvements test: Adding or updating tests chore: Build process, dependencies, tooling ","date":"20-Nov-2024","heading":"Commit Message Best Practices","objectID":"\/posts\/git-github\/:3:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#commit-message-best-practices"},{"categories":["software-engineering"],"collections":null,"content":"3.4 Viewing History Understanding what’s been done is crucial for maintenance and debugging. # Show all commits in current branch\rgit log\r# One line per commit (compact)\rgit log --oneline\r# Show last 10 commits\rgit log -n 10\r# Show commits by specific author\rgit log --author=\"John Doe\"\r# Show commits in date range\rgit log --since=\"2025-01-01\" --until=\"2025-06-01\"\r# Show commits matching search term\rgit log --grep=\"authentication\"\r# Show commits affecting specific file\rgit log filename.js\r# Show changes in each commit (-p = patch)\rgit log -p\r# Pretty graphical view with branches\rgit log --graph --oneline --all --decorate\r# Show statistics\rgit log --stat\r# Custom format\rgit log --pretty=format:\"%h %an %ar - %s\" Understanding the Output commit abc1234def5678 (HEAD -> main)\rAuthor: John Doe Date: Mon Nov 3 10:30:45 2025 +0800\rAdd JWT authentication\r- Implement token generation\r- Validate tokens on routes\rcommit xyz9876abc3210\rAuthor: Jane Smith Date: Sun Nov 2 14:22:10 2025 +0800\rFix user registration bug ","date":"20-Nov-2024","heading":"3.4 Viewing History","objectID":"\/posts\/git-github\/:3:4","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#34-viewing-history"},{"categories":["software-engineering"],"collections":null,"content":"3.4 Viewing History Understanding what’s been done is crucial for maintenance and debugging. # Show all commits in current branch\rgit log\r# One line per commit (compact)\rgit log --oneline\r# Show last 10 commits\rgit log -n 10\r# Show commits by specific author\rgit log --author=\"John Doe\"\r# Show commits in date range\rgit log --since=\"2025-01-01\" --until=\"2025-06-01\"\r# Show commits matching search term\rgit log --grep=\"authentication\"\r# Show commits affecting specific file\rgit log filename.js\r# Show changes in each commit (-p = patch)\rgit log -p\r# Pretty graphical view with branches\rgit log --graph --oneline --all --decorate\r# Show statistics\rgit log --stat\r# Custom format\rgit log --pretty=format:\"%h %an %ar - %s\" Understanding the Output commit abc1234def5678 (HEAD -> main)\rAuthor: John Doe Date: Mon Nov 3 10:30:45 2025 +0800\rAdd JWT authentication\r- Implement token generation\r- Validate tokens on routes\rcommit xyz9876abc3210\rAuthor: Jane Smith Date: Sun Nov 2 14:22:10 2025 +0800\rFix user registration bug ","date":"20-Nov-2024","heading":"Understanding the Output","objectID":"\/posts\/git-github\/:3:4","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#understanding-the-output"},{"categories":["software-engineering"],"collections":null,"content":"4. Understanding and Using Branches Branches allow parallel development without interfering with each other. ","date":"20-Nov-2024","heading":"4. Understanding and Using Branches","objectID":"\/posts\/git-github\/:4:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#4-understanding-and-using-branches"},{"categories":["software-engineering"],"collections":null,"content":"4.1 Branch Fundamentals Info What is a Branch? A branch is simply a lightweight pointer to a specific commit. The default branch is main (or master in older repos). Creating a branch doesn’t copy code \u2014 it just creates a reference to a commit. Why Use Branches? Feature development: Work on features independently Bug fixes: Create hotfix branches without disrupting main code Experimentation: Try ideas without affecting stable code Team collaboration: Multiple people work simultaneously Code review: Enable pull requests for quality control ","date":"20-Nov-2024","heading":"4.1 Branch Fundamentals","objectID":"\/posts\/git-github\/:4:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#41-branch-fundamentals"},{"categories":["software-engineering"],"collections":null,"content":"4.2 Creating and Switching Branches # List local branches\rgit branch\r# List remote branches\rgit branch -r\r# List all branches\rgit branch -a\r# Create new branch (doesn't switch to it)\rgit branch feature\/user-profile\r# Create and switch to new branch\rgit checkout -b feature\/user-profile\r# Modern syntax (Git 2.23+)\rgit switch -c feature\/user-profile\r# Switch to existing branch\rgit checkout feature\/user-profile\r# Modern syntax\rgit switch feature\/user-profile\r# Switch to previous branch\rgit checkout -\r# Delete branch (safe - prevents deleting unmerged branches)\rgit branch -d feature\/user-profile\r# Force delete branch\rgit branch -D feature\/user-profile\r# Rename branch\rgit branch -m old-name new-name\r# Rename current branch\rgit branch -m new-name ","date":"20-Nov-2024","heading":"4.2 Creating and Switching Branches","objectID":"\/posts\/git-github\/:4:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#42-creating-and-switching-branches"},{"categories":["software-engineering"],"collections":null,"content":"4.3 Branching Strategies Different teams use different strategies based on their needs. Git Flow (Enterprise Projects) Designed for projects with scheduled releases: main (production releases only)\r\u2191\rrelease\/v1.0 (release preparation)\r\u2191\rdevelop (integration branch)\r\u2191\rfeature\/* (individual features)\rbugfix\/* (bug fixes)\rhotfix\/* (emergency production fixes) # Start feature branch from develop\rgit checkout develop\rgit checkout -b feature\/new-feature\r# Do work, commit regularly\rgit add .\rgit commit -m \"Implement new feature\"\r# When ready, create pull request to develop\r# After review and merge to develop\r# Create release branch when preparing release\rgit checkout -b release\/v1.0 develop\r# Fix release issues only\rgit commit -m \"Fix typo in error message\"\r# When ready, merge to main with tag\rgit checkout main\rgit merge --no-ff release\/v1.0\rgit tag -a v1.0 -m \"Release version 1.0\"\r# Merge back to develop\rgit checkout develop\rgit merge --no-ff release\/v1.0 GitHub Flow (Continuous Delivery) Simpler flow for continuous deployment: main (always deployable, production)\r\u2191\rfeature\/* (features\/fixes - short-lived)\r\u2193\rPull Request \u2192 Code Review \u2192 Merge \u2192 Deploy # Create branch from main\rgit checkout -b fix\/login-bug\r# Make commits\rgit commit -m \"Fix login redirect issue\"\r# Push to GitHub\rgit push -u origin fix\/login-bug\r# Create pull request on GitHub\r# After review and approval, merge to main\r# GitHub automatically deploys main to production Advantages: Simple and intuitive Quick feedback cycles Production deploys frequently (multiple times per day) Best for teams comfortable with frequent releases Trunk-Based Development main (always in releasable state)\r\u2191\rshort-lived branches (< 1 day)\r\u2193\rCommit \u2192 Merge \u2192 Deploy Best for: Small teams Frequent small commits Strong CI\/CD pipeline High collaboration Tip Choose Your Strategy: Git Flow for large teams with planned releases. GitHub Flow for continuous deployment. Trunk-Based for high-collaboration teams with strong automation. ","date":"20-Nov-2024","heading":"4.3 Branching Strategies","objectID":"\/posts\/git-github\/:4:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#43-branching-strategies"},{"categories":["software-engineering"],"collections":null,"content":"4.3 Branching Strategies Different teams use different strategies based on their needs. Git Flow (Enterprise Projects) Designed for projects with scheduled releases: main (production releases only)\r\u2191\rrelease\/v1.0 (release preparation)\r\u2191\rdevelop (integration branch)\r\u2191\rfeature\/* (individual features)\rbugfix\/* (bug fixes)\rhotfix\/* (emergency production fixes) # Start feature branch from develop\rgit checkout develop\rgit checkout -b feature\/new-feature\r# Do work, commit regularly\rgit add .\rgit commit -m \"Implement new feature\"\r# When ready, create pull request to develop\r# After review and merge to develop\r# Create release branch when preparing release\rgit checkout -b release\/v1.0 develop\r# Fix release issues only\rgit commit -m \"Fix typo in error message\"\r# When ready, merge to main with tag\rgit checkout main\rgit merge --no-ff release\/v1.0\rgit tag -a v1.0 -m \"Release version 1.0\"\r# Merge back to develop\rgit checkout develop\rgit merge --no-ff release\/v1.0 GitHub Flow (Continuous Delivery) Simpler flow for continuous deployment: main (always deployable, production)\r\u2191\rfeature\/* (features\/fixes - short-lived)\r\u2193\rPull Request \u2192 Code Review \u2192 Merge \u2192 Deploy # Create branch from main\rgit checkout -b fix\/login-bug\r# Make commits\rgit commit -m \"Fix login redirect issue\"\r# Push to GitHub\rgit push -u origin fix\/login-bug\r# Create pull request on GitHub\r# After review and approval, merge to main\r# GitHub automatically deploys main to production Advantages: Simple and intuitive Quick feedback cycles Production deploys frequently (multiple times per day) Best for teams comfortable with frequent releases Trunk-Based Development main (always in releasable state)\r\u2191\rshort-lived branches (< 1 day)\r\u2193\rCommit \u2192 Merge \u2192 Deploy Best for: Small teams Frequent small commits Strong CI\/CD pipeline High collaboration Tip Choose Your Strategy: Git Flow for large teams with planned releases. GitHub Flow for continuous deployment. Trunk-Based for high-collaboration teams with strong automation. ","date":"20-Nov-2024","heading":"Git Flow (Enterprise Projects)","objectID":"\/posts\/git-github\/:4:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#git-flow-enterprise-projects"},{"categories":["software-engineering"],"collections":null,"content":"4.3 Branching Strategies Different teams use different strategies based on their needs. Git Flow (Enterprise Projects) Designed for projects with scheduled releases: main (production releases only)\r\u2191\rrelease\/v1.0 (release preparation)\r\u2191\rdevelop (integration branch)\r\u2191\rfeature\/* (individual features)\rbugfix\/* (bug fixes)\rhotfix\/* (emergency production fixes) # Start feature branch from develop\rgit checkout develop\rgit checkout -b feature\/new-feature\r# Do work, commit regularly\rgit add .\rgit commit -m \"Implement new feature\"\r# When ready, create pull request to develop\r# After review and merge to develop\r# Create release branch when preparing release\rgit checkout -b release\/v1.0 develop\r# Fix release issues only\rgit commit -m \"Fix typo in error message\"\r# When ready, merge to main with tag\rgit checkout main\rgit merge --no-ff release\/v1.0\rgit tag -a v1.0 -m \"Release version 1.0\"\r# Merge back to develop\rgit checkout develop\rgit merge --no-ff release\/v1.0 GitHub Flow (Continuous Delivery) Simpler flow for continuous deployment: main (always deployable, production)\r\u2191\rfeature\/* (features\/fixes - short-lived)\r\u2193\rPull Request \u2192 Code Review \u2192 Merge \u2192 Deploy # Create branch from main\rgit checkout -b fix\/login-bug\r# Make commits\rgit commit -m \"Fix login redirect issue\"\r# Push to GitHub\rgit push -u origin fix\/login-bug\r# Create pull request on GitHub\r# After review and approval, merge to main\r# GitHub automatically deploys main to production Advantages: Simple and intuitive Quick feedback cycles Production deploys frequently (multiple times per day) Best for teams comfortable with frequent releases Trunk-Based Development main (always in releasable state)\r\u2191\rshort-lived branches (< 1 day)\r\u2193\rCommit \u2192 Merge \u2192 Deploy Best for: Small teams Frequent small commits Strong CI\/CD pipeline High collaboration Tip Choose Your Strategy: Git Flow for large teams with planned releases. GitHub Flow for continuous deployment. Trunk-Based for high-collaboration teams with strong automation. ","date":"20-Nov-2024","heading":"GitHub Flow (Continuous Delivery)","objectID":"\/posts\/git-github\/:4:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#github-flow-continuous-delivery"},{"categories":["software-engineering"],"collections":null,"content":"4.3 Branching Strategies Different teams use different strategies based on their needs. Git Flow (Enterprise Projects) Designed for projects with scheduled releases: main (production releases only)\r\u2191\rrelease\/v1.0 (release preparation)\r\u2191\rdevelop (integration branch)\r\u2191\rfeature\/* (individual features)\rbugfix\/* (bug fixes)\rhotfix\/* (emergency production fixes) # Start feature branch from develop\rgit checkout develop\rgit checkout -b feature\/new-feature\r# Do work, commit regularly\rgit add .\rgit commit -m \"Implement new feature\"\r# When ready, create pull request to develop\r# After review and merge to develop\r# Create release branch when preparing release\rgit checkout -b release\/v1.0 develop\r# Fix release issues only\rgit commit -m \"Fix typo in error message\"\r# When ready, merge to main with tag\rgit checkout main\rgit merge --no-ff release\/v1.0\rgit tag -a v1.0 -m \"Release version 1.0\"\r# Merge back to develop\rgit checkout develop\rgit merge --no-ff release\/v1.0 GitHub Flow (Continuous Delivery) Simpler flow for continuous deployment: main (always deployable, production)\r\u2191\rfeature\/* (features\/fixes - short-lived)\r\u2193\rPull Request \u2192 Code Review \u2192 Merge \u2192 Deploy # Create branch from main\rgit checkout -b fix\/login-bug\r# Make commits\rgit commit -m \"Fix login redirect issue\"\r# Push to GitHub\rgit push -u origin fix\/login-bug\r# Create pull request on GitHub\r# After review and approval, merge to main\r# GitHub automatically deploys main to production Advantages: Simple and intuitive Quick feedback cycles Production deploys frequently (multiple times per day) Best for teams comfortable with frequent releases Trunk-Based Development main (always in releasable state)\r\u2191\rshort-lived branches (< 1 day)\r\u2193\rCommit \u2192 Merge \u2192 Deploy Best for: Small teams Frequent small commits Strong CI\/CD pipeline High collaboration Tip Choose Your Strategy: Git Flow for large teams with planned releases. GitHub Flow for continuous deployment. Trunk-Based for high-collaboration teams with strong automation. ","date":"20-Nov-2024","heading":"Trunk-Based Development","objectID":"\/posts\/git-github\/:4:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#trunk-based-development"},{"categories":["software-engineering"],"collections":null,"content":"5. Merging Branches Combining changes from different branches is core to collaborative development. ","date":"20-Nov-2024","heading":"5. Merging Branches","objectID":"\/posts\/git-github\/:5:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#5-merging-branches"},{"categories":["software-engineering"],"collections":null,"content":"5.1 Types of Merges Regular Merge (Three-Way Merge) # Switch to target branch (usually main)\rgit checkout main\r# Merge feature branch into main\rgit merge feature\/new-feature\r# Git creates a merge commit automatically\r# This preserves complete branch history Result: A new merge commit that combines two branches. Advantages: Complete history preserved Can easily see which commits came from which branch Makes large-scale changes trackable Disadvantages: Creates extra merge commits History can become cluttered Squash Merge # Merge all commits from feature branch into single commit\rgit merge --squash feature\/new-feature\r# Still need to commit the squashed changes\rgit commit -m \"Merge feature: add user profile\rImplements complete user profile functionality including:\r- Profile page with bio and avatar\r- Profile editing with validation\r- Profile deletion option\" Result: All commits from feature branch become one commit on main. Advantages: Clean, linear history Easier to understand what was added Easy to revert entire feature Disadvantages: Individual commit history lost Harder to track specific changes within the feature Rebase Merge # Reapply feature commits on top of main\rgit merge --ff-only feature\/new-feature\r# Or explicitly use rebase\rgit rebase main feature\/new-feature\rgit checkout main\rgit merge --ff-only feature\/new-feature Result: Linear history as if changes were made sequentially. Advantages: Clean, easy-to-read history Each commit is independent Can understand full story Disadvantages: Rewrites history (dangerous with shared branches) Loses information about parallel development ","date":"20-Nov-2024","heading":"5.1 Types of Merges","objectID":"\/posts\/git-github\/:5:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#51-types-of-merges"},{"categories":["software-engineering"],"collections":null,"content":"5.1 Types of Merges Regular Merge (Three-Way Merge) # Switch to target branch (usually main)\rgit checkout main\r# Merge feature branch into main\rgit merge feature\/new-feature\r# Git creates a merge commit automatically\r# This preserves complete branch history Result: A new merge commit that combines two branches. Advantages: Complete history preserved Can easily see which commits came from which branch Makes large-scale changes trackable Disadvantages: Creates extra merge commits History can become cluttered Squash Merge # Merge all commits from feature branch into single commit\rgit merge --squash feature\/new-feature\r# Still need to commit the squashed changes\rgit commit -m \"Merge feature: add user profile\rImplements complete user profile functionality including:\r- Profile page with bio and avatar\r- Profile editing with validation\r- Profile deletion option\" Result: All commits from feature branch become one commit on main. Advantages: Clean, linear history Easier to understand what was added Easy to revert entire feature Disadvantages: Individual commit history lost Harder to track specific changes within the feature Rebase Merge # Reapply feature commits on top of main\rgit merge --ff-only feature\/new-feature\r# Or explicitly use rebase\rgit rebase main feature\/new-feature\rgit checkout main\rgit merge --ff-only feature\/new-feature Result: Linear history as if changes were made sequentially. Advantages: Clean, easy-to-read history Each commit is independent Can understand full story Disadvantages: Rewrites history (dangerous with shared branches) Loses information about parallel development ","date":"20-Nov-2024","heading":"Regular Merge (Three-Way Merge)","objectID":"\/posts\/git-github\/:5:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#regular-merge-three-way-merge"},{"categories":["software-engineering"],"collections":null,"content":"5.1 Types of Merges Regular Merge (Three-Way Merge) # Switch to target branch (usually main)\rgit checkout main\r# Merge feature branch into main\rgit merge feature\/new-feature\r# Git creates a merge commit automatically\r# This preserves complete branch history Result: A new merge commit that combines two branches. Advantages: Complete history preserved Can easily see which commits came from which branch Makes large-scale changes trackable Disadvantages: Creates extra merge commits History can become cluttered Squash Merge # Merge all commits from feature branch into single commit\rgit merge --squash feature\/new-feature\r# Still need to commit the squashed changes\rgit commit -m \"Merge feature: add user profile\rImplements complete user profile functionality including:\r- Profile page with bio and avatar\r- Profile editing with validation\r- Profile deletion option\" Result: All commits from feature branch become one commit on main. Advantages: Clean, linear history Easier to understand what was added Easy to revert entire feature Disadvantages: Individual commit history lost Harder to track specific changes within the feature Rebase Merge # Reapply feature commits on top of main\rgit merge --ff-only feature\/new-feature\r# Or explicitly use rebase\rgit rebase main feature\/new-feature\rgit checkout main\rgit merge --ff-only feature\/new-feature Result: Linear history as if changes were made sequentially. Advantages: Clean, easy-to-read history Each commit is independent Can understand full story Disadvantages: Rewrites history (dangerous with shared branches) Loses information about parallel development ","date":"20-Nov-2024","heading":"Squash Merge","objectID":"\/posts\/git-github\/:5:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#squash-merge"},{"categories":["software-engineering"],"collections":null,"content":"5.1 Types of Merges Regular Merge (Three-Way Merge) # Switch to target branch (usually main)\rgit checkout main\r# Merge feature branch into main\rgit merge feature\/new-feature\r# Git creates a merge commit automatically\r# This preserves complete branch history Result: A new merge commit that combines two branches. Advantages: Complete history preserved Can easily see which commits came from which branch Makes large-scale changes trackable Disadvantages: Creates extra merge commits History can become cluttered Squash Merge # Merge all commits from feature branch into single commit\rgit merge --squash feature\/new-feature\r# Still need to commit the squashed changes\rgit commit -m \"Merge feature: add user profile\rImplements complete user profile functionality including:\r- Profile page with bio and avatar\r- Profile editing with validation\r- Profile deletion option\" Result: All commits from feature branch become one commit on main. Advantages: Clean, linear history Easier to understand what was added Easy to revert entire feature Disadvantages: Individual commit history lost Harder to track specific changes within the feature Rebase Merge # Reapply feature commits on top of main\rgit merge --ff-only feature\/new-feature\r# Or explicitly use rebase\rgit rebase main feature\/new-feature\rgit checkout main\rgit merge --ff-only feature\/new-feature Result: Linear history as if changes were made sequentially. Advantages: Clean, easy-to-read history Each commit is independent Can understand full story Disadvantages: Rewrites history (dangerous with shared branches) Loses information about parallel development ","date":"20-Nov-2024","heading":"Rebase Merge","objectID":"\/posts\/git-github\/:5:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#rebase-merge"},{"categories":["software-engineering"],"collections":null,"content":"5.2 Merge Conflicts When Git can’t automatically merge branches, you get conflicts. Understanding Conflicts # When merge conflict occurs, files contain:\r<<<<<<< HEAD\r\/\/ Your changes (from current branch)\rfunction login(user) {\rvalidate(user);\rreturn authenticate(user);\r}\r=======\r\/\/ Incoming changes (from branch being merged)\rfunction login(user) {\rcheckCredentials(user);\rreturn authorizeUser(user);\r}\r>>>>>>> feature\/auth-update Resolving Conflicts # 1. View conflicted files\rgit status\r# 2. View conflicts in detail\rgit diff\r# 3. Edit the file to resolve conflicts\r# Remove conflict markers and keep desired code\r# 4. Stage the resolved file\rgit add resolved-file.js\r# 5. Complete the merge\rgit commit -m \"Merge feature\/auth-update with conflict resolution\"\r# Or abort the merge if something went wrong\rgit merge --abort Warning Merge Conflict Prevention: Keep branches short-lived (< 1 week). Small, focused changes are less likely to conflict. Communicate with teammates about what you’re working on. ","date":"20-Nov-2024","heading":"5.2 Merge Conflicts","objectID":"\/posts\/git-github\/:5:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#52-merge-conflicts"},{"categories":["software-engineering"],"collections":null,"content":"5.2 Merge Conflicts When Git can’t automatically merge branches, you get conflicts. Understanding Conflicts # When merge conflict occurs, files contain:\r<<<<<<< HEAD\r\/\/ Your changes (from current branch)\rfunction login(user) {\rvalidate(user);\rreturn authenticate(user);\r}\r=======\r\/\/ Incoming changes (from branch being merged)\rfunction login(user) {\rcheckCredentials(user);\rreturn authorizeUser(user);\r}\r>>>>>>> feature\/auth-update Resolving Conflicts # 1. View conflicted files\rgit status\r# 2. View conflicts in detail\rgit diff\r# 3. Edit the file to resolve conflicts\r# Remove conflict markers and keep desired code\r# 4. Stage the resolved file\rgit add resolved-file.js\r# 5. Complete the merge\rgit commit -m \"Merge feature\/auth-update with conflict resolution\"\r# Or abort the merge if something went wrong\rgit merge --abort Warning Merge Conflict Prevention: Keep branches short-lived (< 1 week). Small, focused changes are less likely to conflict. Communicate with teammates about what you’re working on. ","date":"20-Nov-2024","heading":"Understanding Conflicts","objectID":"\/posts\/git-github\/:5:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#understanding-conflicts"},{"categories":["software-engineering"],"collections":null,"content":"5.2 Merge Conflicts When Git can’t automatically merge branches, you get conflicts. Understanding Conflicts # When merge conflict occurs, files contain:\r<<<<<<< HEAD\r\/\/ Your changes (from current branch)\rfunction login(user) {\rvalidate(user);\rreturn authenticate(user);\r}\r=======\r\/\/ Incoming changes (from branch being merged)\rfunction login(user) {\rcheckCredentials(user);\rreturn authorizeUser(user);\r}\r>>>>>>> feature\/auth-update Resolving Conflicts # 1. View conflicted files\rgit status\r# 2. View conflicts in detail\rgit diff\r# 3. Edit the file to resolve conflicts\r# Remove conflict markers and keep desired code\r# 4. Stage the resolved file\rgit add resolved-file.js\r# 5. Complete the merge\rgit commit -m \"Merge feature\/auth-update with conflict resolution\"\r# Or abort the merge if something went wrong\rgit merge --abort Warning Merge Conflict Prevention: Keep branches short-lived (< 1 week). Small, focused changes are less likely to conflict. Communicate with teammates about what you’re working on. ","date":"20-Nov-2024","heading":"Resolving Conflicts","objectID":"\/posts\/git-github\/:5:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#resolving-conflicts"},{"categories":["software-engineering"],"collections":null,"content":"6. Remote Repositories and GitHub ","date":"20-Nov-2024","heading":"6. Remote Repositories and GitHub","objectID":"\/posts\/git-github\/:6:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#6-remote-repositories-and-github"},{"categories":["software-engineering"],"collections":null,"content":"6.1 Understanding Remotes A remote is a reference to a repository hosted elsewhere (typically GitHub). # View configured remotes\rgit remote -v\r# Add new remote\rgit remote add origin https:\/\/github.com\/username\/project.git\r# Remove remote\rgit remote remove origin\r# Rename remote\rgit remote rename origin upstream\r# View remote details\rgit remote show origin\r# Change remote URL (HTTPS to SSH)\rgit remote set-url origin git@github.com:username\/project.git Standard Remote Names: origin: Your main remote (GitHub repo you cloned or created) upstream: Original repo (if you forked) fork: Your fork of someone else’s project ","date":"20-Nov-2024","heading":"6.1 Understanding Remotes","objectID":"\/posts\/git-github\/:6:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#61-understanding-remotes"},{"categories":["software-engineering"],"collections":null,"content":"6.2 Cloning Repositories # Clone entire repository\rgit clone https:\/\/github.com\/username\/project.git\r# Clone to specific directory\rgit clone https:\/\/github.com\/username\/project.git my-folder\r# Clone specific branch only\rgit clone -b develop https:\/\/github.com\/username\/project.git\r# Shallow clone (fewer commits for faster download)\rgit clone --depth 5 https:\/\/github.com\/username\/project.git\r# Shallow clone with history\rgit clone --depth 5 --deepen 10 https:\/\/github.com\/username\/project.git ","date":"20-Nov-2024","heading":"6.2 Cloning Repositories","objectID":"\/posts\/git-github\/:6:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#62-cloning-repositories"},{"categories":["software-engineering"],"collections":null,"content":"6.3 Pushing and Pulling Fetching Updates from Remote # Fetch: Download remote changes (no integration)\rgit fetch origin\r# Fetch all remotes\rgit fetch --all\r# Fetch and show what changed\rgit fetch origin && git log origin\/main ^main\r# Pull: Fetch + merge (downloads and integrates)\rgit pull origin main\r# Pull with rebase instead of merge\rgit pull --rebase origin main\r# Pull and automatically stash local changes\rgit pull --autostash Tip Fetch Before You Act: Use git fetch to see what changed on remote without modifying your local code. Then decide whether to merge, rebase, or do something else. Pushing Changes to Remote # Push current branch to remote\rgit push origin main\r# Push to remote and track it\rgit push -u origin feature\/new-feature\r# Push all branches\rgit push --all\r# Push all tags\rgit push --tags\r# Push specific branch\rgit push origin feature\/user-auth\r# Force push (overwrite remote - use with caution!)\rgit push --force\r# Safe force push (doesn't overwrite others' work)\rgit push --force-with-lease\r# Delete remote branch\rgit push origin --delete feature\/old-feature\r# Push with pull request (GitPython or GitHub CLI)\rgh pr create --title \"Add user profile\" --body \"Implements #123\" Danger Force Push Warning: git push --force overwrites remote history. Only use on personal branches. For shared branches, use --force-with-lease which is safer. ","date":"20-Nov-2024","heading":"6.3 Pushing and Pulling","objectID":"\/posts\/git-github\/:6:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#63-pushing-and-pulling"},{"categories":["software-engineering"],"collections":null,"content":"6.3 Pushing and Pulling Fetching Updates from Remote # Fetch: Download remote changes (no integration)\rgit fetch origin\r# Fetch all remotes\rgit fetch --all\r# Fetch and show what changed\rgit fetch origin && git log origin\/main ^main\r# Pull: Fetch + merge (downloads and integrates)\rgit pull origin main\r# Pull with rebase instead of merge\rgit pull --rebase origin main\r# Pull and automatically stash local changes\rgit pull --autostash Tip Fetch Before You Act: Use git fetch to see what changed on remote without modifying your local code. Then decide whether to merge, rebase, or do something else. Pushing Changes to Remote # Push current branch to remote\rgit push origin main\r# Push to remote and track it\rgit push -u origin feature\/new-feature\r# Push all branches\rgit push --all\r# Push all tags\rgit push --tags\r# Push specific branch\rgit push origin feature\/user-auth\r# Force push (overwrite remote - use with caution!)\rgit push --force\r# Safe force push (doesn't overwrite others' work)\rgit push --force-with-lease\r# Delete remote branch\rgit push origin --delete feature\/old-feature\r# Push with pull request (GitPython or GitHub CLI)\rgh pr create --title \"Add user profile\" --body \"Implements #123\" Danger Force Push Warning: git push --force overwrites remote history. Only use on personal branches. For shared branches, use --force-with-lease which is safer. ","date":"20-Nov-2024","heading":"Fetching Updates from Remote","objectID":"\/posts\/git-github\/:6:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#fetching-updates-from-remote"},{"categories":["software-engineering"],"collections":null,"content":"6.3 Pushing and Pulling Fetching Updates from Remote # Fetch: Download remote changes (no integration)\rgit fetch origin\r# Fetch all remotes\rgit fetch --all\r# Fetch and show what changed\rgit fetch origin && git log origin\/main ^main\r# Pull: Fetch + merge (downloads and integrates)\rgit pull origin main\r# Pull with rebase instead of merge\rgit pull --rebase origin main\r# Pull and automatically stash local changes\rgit pull --autostash Tip Fetch Before You Act: Use git fetch to see what changed on remote without modifying your local code. Then decide whether to merge, rebase, or do something else. Pushing Changes to Remote # Push current branch to remote\rgit push origin main\r# Push to remote and track it\rgit push -u origin feature\/new-feature\r# Push all branches\rgit push --all\r# Push all tags\rgit push --tags\r# Push specific branch\rgit push origin feature\/user-auth\r# Force push (overwrite remote - use with caution!)\rgit push --force\r# Safe force push (doesn't overwrite others' work)\rgit push --force-with-lease\r# Delete remote branch\rgit push origin --delete feature\/old-feature\r# Push with pull request (GitPython or GitHub CLI)\rgh pr create --title \"Add user profile\" --body \"Implements #123\" Danger Force Push Warning: git push --force overwrites remote history. Only use on personal branches. For shared branches, use --force-with-lease which is safer. ","date":"20-Nov-2024","heading":"Pushing Changes to Remote","objectID":"\/posts\/git-github\/:6:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#pushing-changes-to-remote"},{"categories":["software-engineering"],"collections":null,"content":"7. Collaborating with Pull Requests Pull Requests (PRs) enable code review and quality control. ","date":"20-Nov-2024","heading":"7. Collaborating with Pull Requests","objectID":"\/posts\/git-github\/:7:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#7-collaborating-with-pull-requests"},{"categories":["software-engineering"],"collections":null,"content":"7.1 Creating a Pull Request # Create feature branch\rgit checkout -b feature\/user-dashboard\r# Make changes and commits\rgit add .\rgit commit -m \"Add dashboard component\"\r# Push to GitHub\rgit push -u origin feature\/user-dashboard\r# Create PR via GitHub web interface or CLI\rgh pr create --title \"Add user dashboard\" \\\r--body \"Implements #234: User dashboard with stats and charts\" Via GitHub CLI (recommended): # Install GitHub CLI\r# brew install gh (macOS)\r# or from https:\/\/cli.github.com\r# Login to GitHub\rgh auth login\r# Create pull request\rgh pr create --title \"Add feature\" --body \"Description\"\r# View your PRs\rgh pr list\r# Check PR status\rgh pr checks # Close PR without merging\rgh pr close ","date":"20-Nov-2024","heading":"7.1 Creating a Pull Request","objectID":"\/posts\/git-github\/:7:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#71-creating-a-pull-request"},{"categories":["software-engineering"],"collections":null,"content":"7.2 PR Best Practices Code Review Checklist: Code follows project style guide Changes are focused and not too large (< 400 lines is ideal) Tests added\/updated for new code Documentation updated if needed No hardcoded values or credentials No unnecessary dependencies added Performance implications considered Security implications reviewed PR Title and Description: ## Summary\rBrief description of what this PR does.\r## Changes\r- Added user authentication system\r- Implemented JWT token validation\r- Created auth middleware\r## Related Issues\rCloses #123\rRelates to #456\r## Testing\r- Tested login with valid credentials\r- Tested login with invalid credentials\r- Tested token refresh\r## Screenshots\r[Add relevant screenshots if UI changes]\r## Checklist\r- [x] Tests added\/updated\r- [x] Documentation updated\r- [x] No breaking changes\r- [ ] Ready for review ","date":"20-Nov-2024","heading":"7.2 PR Best Practices","objectID":"\/posts\/git-github\/:7:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#72-pr-best-practices"},{"categories":["software-engineering"],"collections":null,"content":"7.3 Updating a PR # Make additional changes\rgit add .\rgit commit -m \"Address review feedback: add error handling\"\r# Push updates (forces update to same PR)\rgit push origin feature\/new-feature\r# Force push if you rewrote history\rgit push --force-with-lease origin feature\/new-feature\r# The PR automatically updates with new commits ","date":"20-Nov-2024","heading":"7.3 Updating a PR","objectID":"\/posts\/git-github\/:7:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#73-updating-a-pr"},{"categories":["software-engineering"],"collections":null,"content":"7.4 Merging Pull Requests # Option 1: Create merge commit (preserves branch history)\r# Recommended for: Feature branches, large changes\r# Option 2: Squash and merge (clean history)\r# Recommended for: Small fixes, single feature commits\r# Option 3: Rebase and merge (linear history)\r# Recommended for: Teams using rebase workflow\r# You can set default strategy in GitHub repo settings:\r# Settings \u2192 General \u2192 Pull Requests \u2192 Default merge method ","date":"20-Nov-2024","heading":"7.4 Merging Pull Requests","objectID":"\/posts\/git-github\/:7:4","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#74-merging-pull-requests"},{"categories":["software-engineering"],"collections":null,"content":"8. Advanced Git Techniques ","date":"20-Nov-2024","heading":"8. Advanced Git Techniques","objectID":"\/posts\/git-github\/:8:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#8-advanced-git-techniques"},{"categories":["software-engineering"],"collections":null,"content":"8.1 Stashing Work Save changes without committing them. # Save current changes (working directory and staging area)\rgit stash\r# Save with descriptive message\rgit stash save \"WIP: user profile feature\"\r# List all stashes\rgit stash list\r# Show contents of stash\rgit stash show stash@{0}\r# Show stash with diff\rgit stash show -p stash@{0}\r# Apply stash and keep it\rgit stash apply stash@{0}\r# Apply latest stash and remove it\rgit stash pop\r# Delete specific stash\rgit stash drop stash@{0}\r# Delete all stashes\rgit stash clear\r# Create branch from stash\rgit stash branch new-branch Stash Use Cases: Switching branches without committing incomplete work Pulling remote changes without conflicts Temporarily saving work while investigating bugs ","date":"20-Nov-2024","heading":"8.1 Stashing Work","objectID":"\/posts\/git-github\/:8:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#81-stashing-work"},{"categories":["software-engineering"],"collections":null,"content":"8.2 Rebasing (Rewriting History) Warning Only rebase unpublished commits! Never rebase commits that have been pushed to shared branches. It rewrites history and causes major conflicts for collaborators. Interactive Rebase # Rebase last 3 commits (opens editor)\rgit rebase -i HEAD~3\r# Rebase from specific commit\rgit rebase -i abc1234\r# In the editor, you can:\r# pick - include commit as-is\r# reword - change commit message\r# squash - combine with previous commit\r# fixup - combine and discard message\r# drop - remove commit Example Interactive Rebase Session: pick a1b2c3d Add user model\rsquash x9y8z7w Fix user model validation\rpick p8q7r6s Create user repository\r# Squash combines the 2nd commit into the 1st\r# Final result: 2 commits instead of 3 Rebase onto Another Branch # Switch to feature branch\rgit checkout feature\/payment\r# Rebase onto main (moves all feature commits on top of main)\rgit rebase main\r# Resolve any conflicts that occur\r# git add resolved-files\r# git rebase --continue\r# If something goes wrong\rgit rebase --abort\r# Force push rebased branch (if already pushed)\rgit push --force-with-lease origin feature\/payment ","date":"20-Nov-2024","heading":"8.2 Rebasing (Rewriting History)","objectID":"\/posts\/git-github\/:8:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#82-rebasing-rewriting-history"},{"categories":["software-engineering"],"collections":null,"content":"8.2 Rebasing (Rewriting History) Warning Only rebase unpublished commits! Never rebase commits that have been pushed to shared branches. It rewrites history and causes major conflicts for collaborators. Interactive Rebase # Rebase last 3 commits (opens editor)\rgit rebase -i HEAD~3\r# Rebase from specific commit\rgit rebase -i abc1234\r# In the editor, you can:\r# pick - include commit as-is\r# reword - change commit message\r# squash - combine with previous commit\r# fixup - combine and discard message\r# drop - remove commit Example Interactive Rebase Session: pick a1b2c3d Add user model\rsquash x9y8z7w Fix user model validation\rpick p8q7r6s Create user repository\r# Squash combines the 2nd commit into the 1st\r# Final result: 2 commits instead of 3 Rebase onto Another Branch # Switch to feature branch\rgit checkout feature\/payment\r# Rebase onto main (moves all feature commits on top of main)\rgit rebase main\r# Resolve any conflicts that occur\r# git add resolved-files\r# git rebase --continue\r# If something goes wrong\rgit rebase --abort\r# Force push rebased branch (if already pushed)\rgit push --force-with-lease origin feature\/payment ","date":"20-Nov-2024","heading":"Interactive Rebase","objectID":"\/posts\/git-github\/:8:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#interactive-rebase"},{"categories":["software-engineering"],"collections":null,"content":"8.2 Rebasing (Rewriting History) Warning Only rebase unpublished commits! Never rebase commits that have been pushed to shared branches. It rewrites history and causes major conflicts for collaborators. Interactive Rebase # Rebase last 3 commits (opens editor)\rgit rebase -i HEAD~3\r# Rebase from specific commit\rgit rebase -i abc1234\r# In the editor, you can:\r# pick - include commit as-is\r# reword - change commit message\r# squash - combine with previous commit\r# fixup - combine and discard message\r# drop - remove commit Example Interactive Rebase Session: pick a1b2c3d Add user model\rsquash x9y8z7w Fix user model validation\rpick p8q7r6s Create user repository\r# Squash combines the 2nd commit into the 1st\r# Final result: 2 commits instead of 3 Rebase onto Another Branch # Switch to feature branch\rgit checkout feature\/payment\r# Rebase onto main (moves all feature commits on top of main)\rgit rebase main\r# Resolve any conflicts that occur\r# git add resolved-files\r# git rebase --continue\r# If something goes wrong\rgit rebase --abort\r# Force push rebased branch (if already pushed)\rgit push --force-with-lease origin feature\/payment ","date":"20-Nov-2024","heading":"Rebase onto Another Branch","objectID":"\/posts\/git-github\/:8:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#rebase-onto-another-branch"},{"categories":["software-engineering"],"collections":null,"content":"8.3 Cherry-Picking Commits Apply specific commits from one branch to another. # Apply specific commit to current branch\rgit cherry-pick abc1234\r# Apply multiple commits\rgit cherry-pick abc1234 def5678 ghi9012\r# Apply commits from another branch\rgit cherry-pick main~5..main\r# Cherry-pick without committing (review first)\rgit cherry-pick --no-commit abc1234\r# Abort cherry-pick if conflicts occur\rgit cherry-pick --abort Use Cases: Backporting fixes to older versions Applying hotfixes to multiple branches Pulling specific changes from develop to main ","date":"20-Nov-2024","heading":"8.3 Cherry-Picking Commits","objectID":"\/posts\/git-github\/:8:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#83-cherry-picking-commits"},{"categories":["software-engineering"],"collections":null,"content":"8.4 Bisecting (Finding Bugs) Binary search to find which commit introduced a bug. # Start bisect session\rgit bisect start\r# Mark current commit as bad (contains bug)\rgit bisect bad HEAD\r# Mark a known good commit\rgit bisect good v1.0\r# Git checks out midpoint, you test it\r# Mark as good or bad based on testing\rgit bisect good # if working\rgit bisect bad # if broken\r# Git continues narrowing down\r# Eventually finds exact commit that broke it\r# End bisect session\rgit bisect reset\r# Automate with script\rgit bisect run .\/test-suite.sh ","date":"20-Nov-2024","heading":"8.4 Bisecting (Finding Bugs)","objectID":"\/posts\/git-github\/:8:4","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#84-bisecting-finding-bugs"},{"categories":["software-engineering"],"collections":null,"content":"8.5 Reflog (Reference Logs) Recover from mistakes by viewing everything you’ve done. # View all reference changes\rgit reflog\r# Show detailed reflog\rgit reflog show\r# Recover accidentally deleted branch\rgit reflog\r# Find the commit where branch was at\rgit checkout -b recovered-branch abc1234\r# Undo a reset\rgit reflog\rgit reset --hard commit-before-reset\r# View reflog for specific ref\rgit reflog show main ","date":"20-Nov-2024","heading":"8.5 Reflog (Reference Logs)","objectID":"\/posts\/git-github\/:8:5","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#85-reflog-reference-logs"},{"categories":["software-engineering"],"collections":null,"content":"9. GitHub Features and Workflows ","date":"20-Nov-2024","heading":"9. GitHub Features and Workflows","objectID":"\/posts\/git-github\/:9:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#9-github-features-and-workflows"},{"categories":["software-engineering"],"collections":null,"content":"9.1 Issues GitHub Issues track bugs, features, and tasks. # Create issue (web interface or CLI)\rgh issue create --title \"Bug: login fails with spaces in password\" \\\r--body \"Steps to reproduce...\"\r# List issues\rgh issue list\r# View issue details\rgh issue view 123\r# Close issue\rgh issue close 123 --comment \"Fixed in PR #456\"\r# Link commits to issues\r# Use 'Closes #123' in commit message or PR description\rgit commit -m \"Fix login bug - Closes #123\"\r# When PR merges, GitHub auto-closes the issue ","date":"20-Nov-2024","heading":"9.1 Issues","objectID":"\/posts\/git-github\/:9:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#91-issues"},{"categories":["software-engineering"],"collections":null,"content":"9.2 GitHub Actions (CI\/CD) Automate testing, building, and deployment. # Create .github\/workflows\/ci.yml file\rname: CI\ron:\rpush:\rbranches: [main, develop]\rpull_request:\rbranches: [main]\rjobs:\rtest:\rruns-on: ubuntu-latest\rsteps:\r- uses: actions\/checkout@v3\r- uses: actions\/setup-node@v3\rwith:\rnode-version: '18'\r- run: npm install\r- run: npm run lint\r- run: npm run test\r- run: npm run build Common Workflow Triggers: push - on commits to specific branches pull_request - on PR creation\/update schedule - run on schedule (cron jobs) workflow_dispatch - manual trigger release - on repository releases ","date":"20-Nov-2024","heading":"9.2 GitHub Actions (CI\/CD)","objectID":"\/posts\/git-github\/:9:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#92-github-actions-cicd"},{"categories":["software-engineering"],"collections":null,"content":"9.3 GitHub Pages Publish websites directly from your repository. # 1. Push HTML files to main branch (or docs\/ folder)\r# 2. Go to repo Settings \u2192 Pages\r# 3. Select source (main or gh-pages branch)\r# 4. GitHub builds and publishes automatically\r# Your site is now at: https:\/\/username.github.io\/repository\r# For project documentation\r# Push to docs\/ folder and enable in settings ","date":"20-Nov-2024","heading":"9.3 GitHub Pages","objectID":"\/posts\/git-github\/:9:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#93-github-pages"},{"categories":["software-engineering"],"collections":null,"content":"9.4 GitHub Gists Share code snippets and files. # Create gist via CLI\rgh gist create filename.js\r# Create secret gist (unlisted)\rgh gist create --public filename.js\r# List your gists\rgh gist list\r# View gist contents\rgh gist view gist-id\r# Edit gist\rgh gist edit gist-id\r# Delete gist\rgh gist delete gist-id\r# Clone gist\rgit clone https:\/\/gist.github.com\/username\/gist-id.git ","date":"20-Nov-2024","heading":"9.4 GitHub Gists","objectID":"\/posts\/git-github\/:9:4","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#94-github-gists"},{"categories":["software-engineering"],"collections":null,"content":"10. Authentication and Security ","date":"20-Nov-2024","heading":"10. Authentication and Security","objectID":"\/posts\/git-github\/:10:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#10-authentication-and-security"},{"categories":["software-engineering"],"collections":null,"content":"10.1 SSH Keys Setup SSH is more secure and convenient than passwords. Generate SSH Key # Generate ED25519 key (recommended - modern & secure)\rssh-keygen -t ed25519 -C \"your_email@example.com\"\r# Save to: ~\/.ssh\/id_ed25519\r# Or generate RSA key (4096 bits - widely compatible)\rssh-keygen -t rsa -b 4096 -C \"your_email@example.com\"\r# Save to: ~\/.ssh\/id_rsa\r# Add key to SSH agent\reval \"$(ssh-agent -s)\"\rssh-add ~\/.ssh\/id_ed25519\r# Copy public key to clipboard\rcat ~\/.ssh\/id_ed25519.pub | pbcopy # macOS\rcat ~\/.ssh\/id_ed25519.pub | xclip -selection clipboard # Linux Add Key to GitHub Go to GitHub Settings \u2192 SSH and GPG keys Click “New SSH key” Paste public key content Give it a descriptive title Click “Add SSH key” Test SSH Connection # Test connection\rssh -T git@github.com\r# Expected output:\r# Hi username! You've successfully authenticated, but\r# GitHub does not provide shell access.\r# If fails, check configuration\rssh -vT git@github.com Configure SSH for Multiple GitHub Accounts # Create\/edit ~\/.ssh\/config\rHost github.com-personal\rHostName github.com\rUser git\rIdentityFile ~\/.ssh\/id_ed25519_personal\rHost github.com-work\rHostName github.com\rUser git\rIdentityFile ~\/.ssh\/id_ed25519_work\r# Use in git commands:\rgit remote add origin git@github.com-personal:username\/repo.git\rgit clone git@github.com-work:company\/project.git ","date":"20-Nov-2024","heading":"10.1 SSH Keys Setup","objectID":"\/posts\/git-github\/:10:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#101-ssh-keys-setup"},{"categories":["software-engineering"],"collections":null,"content":"10.1 SSH Keys Setup SSH is more secure and convenient than passwords. Generate SSH Key # Generate ED25519 key (recommended - modern & secure)\rssh-keygen -t ed25519 -C \"your_email@example.com\"\r# Save to: ~\/.ssh\/id_ed25519\r# Or generate RSA key (4096 bits - widely compatible)\rssh-keygen -t rsa -b 4096 -C \"your_email@example.com\"\r# Save to: ~\/.ssh\/id_rsa\r# Add key to SSH agent\reval \"$(ssh-agent -s)\"\rssh-add ~\/.ssh\/id_ed25519\r# Copy public key to clipboard\rcat ~\/.ssh\/id_ed25519.pub | pbcopy # macOS\rcat ~\/.ssh\/id_ed25519.pub | xclip -selection clipboard # Linux Add Key to GitHub Go to GitHub Settings \u2192 SSH and GPG keys Click “New SSH key” Paste public key content Give it a descriptive title Click “Add SSH key” Test SSH Connection # Test connection\rssh -T git@github.com\r# Expected output:\r# Hi username! You've successfully authenticated, but\r# GitHub does not provide shell access.\r# If fails, check configuration\rssh -vT git@github.com Configure SSH for Multiple GitHub Accounts # Create\/edit ~\/.ssh\/config\rHost github.com-personal\rHostName github.com\rUser git\rIdentityFile ~\/.ssh\/id_ed25519_personal\rHost github.com-work\rHostName github.com\rUser git\rIdentityFile ~\/.ssh\/id_ed25519_work\r# Use in git commands:\rgit remote add origin git@github.com-personal:username\/repo.git\rgit clone git@github.com-work:company\/project.git ","date":"20-Nov-2024","heading":"Generate SSH Key","objectID":"\/posts\/git-github\/:10:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#generate-ssh-key"},{"categories":["software-engineering"],"collections":null,"content":"10.1 SSH Keys Setup SSH is more secure and convenient than passwords. Generate SSH Key # Generate ED25519 key (recommended - modern & secure)\rssh-keygen -t ed25519 -C \"your_email@example.com\"\r# Save to: ~\/.ssh\/id_ed25519\r# Or generate RSA key (4096 bits - widely compatible)\rssh-keygen -t rsa -b 4096 -C \"your_email@example.com\"\r# Save to: ~\/.ssh\/id_rsa\r# Add key to SSH agent\reval \"$(ssh-agent -s)\"\rssh-add ~\/.ssh\/id_ed25519\r# Copy public key to clipboard\rcat ~\/.ssh\/id_ed25519.pub | pbcopy # macOS\rcat ~\/.ssh\/id_ed25519.pub | xclip -selection clipboard # Linux Add Key to GitHub Go to GitHub Settings \u2192 SSH and GPG keys Click “New SSH key” Paste public key content Give it a descriptive title Click “Add SSH key” Test SSH Connection # Test connection\rssh -T git@github.com\r# Expected output:\r# Hi username! You've successfully authenticated, but\r# GitHub does not provide shell access.\r# If fails, check configuration\rssh -vT git@github.com Configure SSH for Multiple GitHub Accounts # Create\/edit ~\/.ssh\/config\rHost github.com-personal\rHostName github.com\rUser git\rIdentityFile ~\/.ssh\/id_ed25519_personal\rHost github.com-work\rHostName github.com\rUser git\rIdentityFile ~\/.ssh\/id_ed25519_work\r# Use in git commands:\rgit remote add origin git@github.com-personal:username\/repo.git\rgit clone git@github.com-work:company\/project.git ","date":"20-Nov-2024","heading":"Add Key to GitHub","objectID":"\/posts\/git-github\/:10:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#add-key-to-github"},{"categories":["software-engineering"],"collections":null,"content":"10.1 SSH Keys Setup SSH is more secure and convenient than passwords. Generate SSH Key # Generate ED25519 key (recommended - modern & secure)\rssh-keygen -t ed25519 -C \"your_email@example.com\"\r# Save to: ~\/.ssh\/id_ed25519\r# Or generate RSA key (4096 bits - widely compatible)\rssh-keygen -t rsa -b 4096 -C \"your_email@example.com\"\r# Save to: ~\/.ssh\/id_rsa\r# Add key to SSH agent\reval \"$(ssh-agent -s)\"\rssh-add ~\/.ssh\/id_ed25519\r# Copy public key to clipboard\rcat ~\/.ssh\/id_ed25519.pub | pbcopy # macOS\rcat ~\/.ssh\/id_ed25519.pub | xclip -selection clipboard # Linux Add Key to GitHub Go to GitHub Settings \u2192 SSH and GPG keys Click “New SSH key” Paste public key content Give it a descriptive title Click “Add SSH key” Test SSH Connection # Test connection\rssh -T git@github.com\r# Expected output:\r# Hi username! You've successfully authenticated, but\r# GitHub does not provide shell access.\r# If fails, check configuration\rssh -vT git@github.com Configure SSH for Multiple GitHub Accounts # Create\/edit ~\/.ssh\/config\rHost github.com-personal\rHostName github.com\rUser git\rIdentityFile ~\/.ssh\/id_ed25519_personal\rHost github.com-work\rHostName github.com\rUser git\rIdentityFile ~\/.ssh\/id_ed25519_work\r# Use in git commands:\rgit remote add origin git@github.com-personal:username\/repo.git\rgit clone git@github.com-work:company\/project.git ","date":"20-Nov-2024","heading":"Test SSH Connection","objectID":"\/posts\/git-github\/:10:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#test-ssh-connection"},{"categories":["software-engineering"],"collections":null,"content":"10.1 SSH Keys Setup SSH is more secure and convenient than passwords. Generate SSH Key # Generate ED25519 key (recommended - modern & secure)\rssh-keygen -t ed25519 -C \"your_email@example.com\"\r# Save to: ~\/.ssh\/id_ed25519\r# Or generate RSA key (4096 bits - widely compatible)\rssh-keygen -t rsa -b 4096 -C \"your_email@example.com\"\r# Save to: ~\/.ssh\/id_rsa\r# Add key to SSH agent\reval \"$(ssh-agent -s)\"\rssh-add ~\/.ssh\/id_ed25519\r# Copy public key to clipboard\rcat ~\/.ssh\/id_ed25519.pub | pbcopy # macOS\rcat ~\/.ssh\/id_ed25519.pub | xclip -selection clipboard # Linux Add Key to GitHub Go to GitHub Settings \u2192 SSH and GPG keys Click “New SSH key” Paste public key content Give it a descriptive title Click “Add SSH key” Test SSH Connection # Test connection\rssh -T git@github.com\r# Expected output:\r# Hi username! You've successfully authenticated, but\r# GitHub does not provide shell access.\r# If fails, check configuration\rssh -vT git@github.com Configure SSH for Multiple GitHub Accounts # Create\/edit ~\/.ssh\/config\rHost github.com-personal\rHostName github.com\rUser git\rIdentityFile ~\/.ssh\/id_ed25519_personal\rHost github.com-work\rHostName github.com\rUser git\rIdentityFile ~\/.ssh\/id_ed25519_work\r# Use in git commands:\rgit remote add origin git@github.com-personal:username\/repo.git\rgit clone git@github.com-work:company\/project.git ","date":"20-Nov-2024","heading":"Configure SSH for Multiple GitHub Accounts","objectID":"\/posts\/git-github\/:10:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#configure-ssh-for-multiple-github-accounts"},{"categories":["software-engineering"],"collections":null,"content":"10.2 Personal Access Tokens Alternative to SSH for programmatic access. # Create token in GitHub Settings \u2192 Developer settings \u2192 Personal access tokens\r# Select scopes: repo, read:user, user:email\r# Use as password when cloning via HTTPS\rgit clone https:\/\/username:ghp_xxx@github.com\/username\/repo.git\r# Store token securely using credential manager\rgit config --global credential.helper cache\r# Or use environment variable\rexport GITHUB_TOKEN=\"ghp_xxx\"\rgit clone https:\/\/github.com\/username\/repo.git ","date":"20-Nov-2024","heading":"10.2 Personal Access Tokens","objectID":"\/posts\/git-github\/:10:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#102-personal-access-tokens"},{"categories":["software-engineering"],"collections":null,"content":"11. Git Internals (How It Actually Works) Understanding internals helps you debug issues and appreciate Git’s design. ","date":"20-Nov-2024","heading":"11. Git Internals (How It Actually Works)","objectID":"\/posts\/git-github\/:11:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#11-git-internals-how-it-actually-works"},{"categories":["software-engineering"],"collections":null,"content":"11.1 The Object Database Git stores everything as four types of objects in .git\/objects\/: # Blobs: File contents (immutable)\rgit hash-object filename.txt\r# Trees: Directory structure\rgit cat-file -p main^{tree}\r# Commits: Snapshots with metadata\rgit cat-file -p main\r# Tags: Labels for specific commits\rgit cat-file -p v1.0 Example Object Structure: Commit: abc1234\r\u251c\u2500 Author: John Doe \u251c\u2500 Date: 2025-11-03\r\u251c\u2500 Message: \"Add user profile\"\r\u251c\u2500 Tree: xyz9876 (represents files at this commit)\r\u2514\u2500 Parent: def5678 (previous commit)\rTree: xyz9876\r\u251c\u2500 user.js (blob: 8f4k2m1)\r\u251c\u2500 profile.js (blob: q2w3e4r)\r\u2514\u2500 styles\/ (tree: 5t6y7u8) ","date":"20-Nov-2024","heading":"11.1 The Object Database","objectID":"\/posts\/git-github\/:11:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#111-the-object-database"},{"categories":["software-engineering"],"collections":null,"content":"11.2 References Branches and tags are references to commits. # Branches stored in .git\/refs\/heads\/\rcat .git\/refs\/heads\/main\r# Output: abc1234def5678...\r# Remote references in .git\/refs\/remotes\/\rcat .git\/refs\/remotes\/origin\/main\r# HEAD points to current branch\rcat .git\/HEAD\r# Output: ref: refs\/heads\/main\r# Detached HEAD (pointing to commit, not branch)\rgit checkout abc1234 # You're now in detached state\rcat .git\/HEAD\r# Output: abc1234 (direct commit reference) ","date":"20-Nov-2024","heading":"11.2 References","objectID":"\/posts\/git-github\/:11:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#112-references"},{"categories":["software-engineering"],"collections":null,"content":"11.3 The Staging Area (Index) The staging area is a file (.git\/index) that tracks what will be committed. # View index contents\rgit ls-files\r# View index with status\rgit ls-files --stage\r# Clear index (dangerous!)\rgit rm --cached -r .\r# Update index after external file changes\rgit update-index --add --cacheinfo ","date":"20-Nov-2024","heading":"11.3 The Staging Area (Index)","objectID":"\/posts\/git-github\/:11:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#113-the-staging-area-index"},{"categories":["software-engineering"],"collections":null,"content":"12. Troubleshooting Common Problems ","date":"20-Nov-2024","heading":"12. Troubleshooting Common Problems","objectID":"\/posts\/git-github\/:12:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#12-troubleshooting-common-problems"},{"categories":["software-engineering"],"collections":null,"content":"12.1 Undoing Changes # Undo uncommitted changes in working directory\rgit checkout filename.js\r# or (modern)\rgit restore filename.js\r# Undo staged changes (keep working directory)\rgit restore --staged filename.js\r# or (older)\rgit reset HEAD filename.js\r# Undo last commit (keep changes)\rgit reset --soft HEAD~1\r# Undo last commit (discard changes)\rgit reset --hard HEAD~1\r# Revert commit (create new commit that undoes it)\rgit revert abc1234\r# Revert merge commit (specify which parent to keep)\rgit revert -m 1 merge-commit-sha ","date":"20-Nov-2024","heading":"12.1 Undoing Changes","objectID":"\/posts\/git-github\/:12:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#121-undoing-changes"},{"categories":["software-engineering"],"collections":null,"content":"12.2 Fixing Mistakes # Accidentally committed to wrong branch\rgit reset --soft HEAD~1\rgit stash\rgit checkout correct-branch\rgit stash pop\rgit commit -m \"...\"\r# Committed to main instead of feature branch\rgit reset --soft HEAD~1 # Keep changes\rgit checkout -b feature\/new-feature\rgit commit -m \"...\"\r# Need to update branch name before push\rgit branch -m old-name new-name\rgit push -u origin new-name\rgit push origin --delete old-name\r# Typo in commit message\rgit commit --amend -m \"Corrected message\"\r# Then force push if already pushed\rgit push --force-with-lease origin branch-name ","date":"20-Nov-2024","heading":"12.2 Fixing Mistakes","objectID":"\/posts\/git-github\/:12:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#122-fixing-mistakes"},{"categories":["software-engineering"],"collections":null,"content":"12.3 Resolving Conflicts # View all conflicted files\rgit diff --name-only --diff-filter=U\r# View detailed conflict\rgit diff filename.js\r# Use your version entirely\rgit checkout --ours filename.js\rgit add filename.js\r# Use their version entirely\rgit checkout --theirs filename.js\rgit add filename.js\r# Use external merge tool\rgit mergetool\r# After resolving all conflicts\rgit add .\rgit commit -m \"Resolve merge conflicts\"\r# Or abort the merge\rgit merge --abort ","date":"20-Nov-2024","heading":"12.3 Resolving Conflicts","objectID":"\/posts\/git-github\/:12:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#123-resolving-conflicts"},{"categories":["software-engineering"],"collections":null,"content":"13. Best Practices and Workflow Tips ","date":"20-Nov-2024","heading":"13. Best Practices and Workflow Tips","objectID":"\/posts\/git-github\/:13:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#13-best-practices-and-workflow-tips"},{"categories":["software-engineering"],"collections":null,"content":"13.1 Commit Best Practices Info Atomic Commits: Each commit should represent one logical change. Related changes should be committed together; unrelated changes should be in separate commits. # \u274c BAD: Too many unrelated changes\rgit add .\rgit commit -m \"Update stuff\"\r# \u2705 GOOD: Related changes, logical commit\rgit add src\/auth\/login.js src\/auth\/login.test.js\rgit commit -m \"feat(auth): add password reset functionality\r- Add reset token generation\r- Add email sending for reset link\r- Add password validation rules\rCloses #234\"\r# Use staging area strategically\rgit add -p # Stage changes hunk by hunk\rgit diff --staged # Review what you're committing\rgit commit -m \"...\" # Commit reviewed changes ","date":"20-Nov-2024","heading":"13.1 Commit Best Practices","objectID":"\/posts\/git-github\/:13:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#131-commit-best-practices"},{"categories":["software-engineering"],"collections":null,"content":"13.2 Branch Naming Conventions # Feature branches\rfeature\/user-authentication\rfeature\/payment-integration\r# Bug fix branches\rbugfix\/login-redirect-issue\rfix\/null-pointer-exception\r# Release branches\rrelease\/v1.0\rrelease\/v2.1.0\r# Hotfix branches\rhotfix\/critical-security-patch\rhotfix\/production-crash\r# Maintenance branches\rmaintenance\/update-dependencies\rchore\/code-cleanup\r# Personal experimentation\rsandbox\/test-new-framework\rspike\/investigate-performance ","date":"20-Nov-2024","heading":"13.2 Branch Naming Conventions","objectID":"\/posts\/git-github\/:13:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#132-branch-naming-conventions"},{"categories":["software-engineering"],"collections":null,"content":"13.3 Keeping Repository Clean # Delete merged branches locally\rgit branch -d feature\/completed-feature\r# Delete all merged branches\rgit branch -d $(git branch --merged | grep -v \"*\")\r# Find branches merged into main\rgit branch --merged main\r# Find branches NOT merged into main\rgit branch --no-merged main\r# Clean up tracking branches\rgit remote prune origin\r# Clean up object database\rgit gc\r# View and cleanup\rgit reflog expire --expire=now --all\rgit gc --prune=now ","date":"20-Nov-2024","heading":"13.3 Keeping Repository Clean","objectID":"\/posts\/git-github\/:13:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#133-keeping-repository-clean"},{"categories":["software-engineering"],"collections":null,"content":"13.4 Writing Useful Commit Messages Commit messages are read by humans much more often than they’re written. # Bad commit message\rFixed stuff\r# Good commit message\rFix user profile display bug\rThe profile view was failing when user bio contained\rspecial characters. This was due to improper HTML\rescaping in the template.\rChanges:\r- Add HTML entity encoding to bio field\r- Add test case for special characters\r- Update documentation\rCloses #456 ","date":"20-Nov-2024","heading":"13.4 Writing Useful Commit Messages","objectID":"\/posts\/git-github\/:13:4","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#134-writing-useful-commit-messages"},{"categories":["software-engineering"],"collections":null,"content":"14. GitHub Workflows and Collaboration ","date":"20-Nov-2024","heading":"14. GitHub Workflows and Collaboration","objectID":"\/posts\/git-github\/:14:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#14-github-workflows-and-collaboration"},{"categories":["software-engineering"],"collections":null,"content":"14.1 Code Review Workflow # As a reviewer:\r1. Clone the branch locally to test\rgit fetch origin pull\/123\/head:pr-123\rgit checkout pr-123\r2. Test the changes thoroughly\rnpm install && npm run test\r3. Review code for:\r- Correctness\r- Performance\r- Security\r- Style consistency\r- Test coverage\r4. Leave constructive comments\rgh pr review 123 --approve\r# or\rgh pr review 123 --request-changes\r5. Approve when satisfied\r# Comment: \"Looks good! Approved\"\r# As a developer:\r1. Implement feedback\rgit add .\rgit commit -m \"Address review feedback: add error handling\"\r2. Push updates\rgit push origin feature\/branch\r3. Request re-review\rgh pr review --request-review @reviewer-name ","date":"20-Nov-2024","heading":"14.1 Code Review Workflow","objectID":"\/posts\/git-github\/:14:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#141-code-review-workflow"},{"categories":["software-engineering"],"collections":null,"content":"14.2 Release Workflow with Tags # Create annotated tag (recommended)\rgit tag -a v1.0.0 -m \"Release version 1.0.0\"\r# Create lightweight tag (simple reference)\rgit tag v1.0.0\r# List tags\rgit tag -l\r# View tag details\rgit show v1.0.0\r# Push tags to GitHub\rgit push origin --tags\r# Delete tag locally\rgit tag -d v1.0.0\r# Delete tag remotely\rgit push origin --delete v1.0.0\r# Create release on GitHub with notes\rgh release create v1.0.0 --title \"Version 1.0.0\" --notes \"Release notes...\" Semantic Versioning (Recommended): v[MAJOR].[MINOR].[PATCH]-[PRERELEASE]+[BUILD]\rv1.2.3 # Version 1, minor release 2, patch 3\rv2.0.0-beta # Pre-release version\rv1.0.0-alpha.1 # Pre-release with identifier ","date":"20-Nov-2024","heading":"14.2 Release Workflow with Tags","objectID":"\/posts\/git-github\/:14:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#142-release-workflow-with-tags"},{"categories":["software-engineering"],"collections":null,"content":"14.3 Git Hooks Automate tasks that run at specific points in Git workflow. # Hooks located in .git\/hooks\/\r# Create pre-commit hook (runs before commit)\rcat > .git\/hooks\/pre-commit <<'EOF'\r#!\/bin\/bash\rnpm run lint\rnpm run test\rEOF\rchmod +x .git\/hooks\/pre-commit\r# Shared hooks (commit to repo)\rmkdir -p .githooks\r# Update git config to use .githooks directory\rgit config core.hooksPath .githooks\r# Common hooks:\r# pre-commit - before commit\r# prepare-commit-msg - modify commit message\r# commit-msg - validate commit message\r# post-commit - after commit\r# pre-push - before push Pre-commit Framework (Recommended): # Install pre-commit\rpip install pre-commit\r# Create .pre-commit-config.yaml\rrepos:\r- repo: https:\/\/github.com\/psf\/black\rrev: 23.1.0\rhooks:\r- id: black\r- repo: https:\/\/github.com\/pre-commit\/pre-commit-hooks\rrev: v4.4.0\rhooks:\r- id: trailing-whitespace\r- id: end-of-file-fixer\r- id: check-yaml\r- id: check-added-large-files\r# Setup pre-commit\rpre-commit install\r# Run manually on all files\rpre-commit run --all-files ","date":"20-Nov-2024","heading":"14.3 Git Hooks","objectID":"\/posts\/git-github\/:14:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#143-git-hooks"},{"categories":["software-engineering"],"collections":null,"content":"15. Git Performance and Optimization ","date":"20-Nov-2024","heading":"15. Git Performance and Optimization","objectID":"\/posts\/git-github\/:15:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#15-git-performance-and-optimization"},{"categories":["software-engineering"],"collections":null,"content":"15.1 Large Repositories # Shallow clone (faster for CI\/CD)\rgit clone --depth 1 repo-url\r# Unshallow later if needed\rgit fetch --unshallow\r# Sparse checkout (clone only specific directories)\rgit clone --filter=blob:none --sparse repo-url\rcd repo\rgit sparse-checkout set path\/to\/dir\r# Partial clone (saves bandwidth)\rgit clone --filter=blob:none repo-url\r# Git LFS for large files\rgit lfs install\rgit lfs track \"*.bin\"\rgit add .gitattributes ","date":"20-Nov-2024","heading":"15.1 Large Repositories","objectID":"\/posts\/git-github\/:15:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#151-large-repositories"},{"categories":["software-engineering"],"collections":null,"content":"15.2 Performance Tips # Enable auto garbage collection (default)\rgit config --global gc.auto 4096\r# Manual garbage collection\rgit gc --aggressive\r# Reduce pack size\rgit gc --prune=now\r# Enable object file delta compression\rgit config --global core.compression 9\r# Use credential helper for faster auth\rgit config --global credential.helper cache\rgit config --global credential.helper 'cache --timeout=3600' ","date":"20-Nov-2024","heading":"15.2 Performance Tips","objectID":"\/posts\/git-github\/:15:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#152-performance-tips"},{"categories":["software-engineering"],"collections":null,"content":"16. Interview Questions and Deep Dives ","date":"20-Nov-2024","heading":"16. Interview Questions and Deep Dives","objectID":"\/posts\/git-github\/:16:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#16-interview-questions-and-deep-dives"},{"categories":["software-engineering"],"collections":null,"content":"16.1 Common Interview Questions Q1: Explain Git vs GitHub Git is version control software. GitHub is a platform for hosting Git repositories with collaboration features like PRs, issues, and Actions. Q2: What is the difference between merge and rebase? Merge creates a merge commit combining two branches (preserves history). Rebase replays commits on top of another branch (linear history). Use merge for shared branches, rebase for personal branches. Q3: How do you handle merge conflicts? Git marks conflict zones in files. You edit to resolve, then stage and commit. Use git diff to see conflicts, merge tools like Meld for visual resolution, or git merge --abort to cancel. Q4: What’s the difference between reset and revert? reset moves HEAD to previous commit (rewrites history). revert creates new commit that undoes changes (safe for shared branches). Q5: Explain Git objects and how commits work Git stores blobs (file content), trees (directory structure), commits (snapshots with metadata), and tags (references). Everything is content-addressed by SHA-1 hash. ","date":"20-Nov-2024","heading":"16.1 Common Interview Questions","objectID":"\/posts\/git-github\/:16:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#161-common-interview-questions"},{"categories":["software-engineering"],"collections":null,"content":"16.2 Advanced Scenarios # Scenario: Large refactor needs to be split across commits\rgit reset HEAD~5 # Unstage last 5 commits\rgit add .\rgit status -s | head -20 # See all changes\r# Stage first logical unit\rgit add src\/auth\/*.js\rgit commit -m \"refactor(auth): simplify token handling\"\r# Stage next unit\rgit add src\/api\/*.js\rgit commit -m \"refactor(api): restructure endpoints\"\r# Push when ready\rgit push origin feature\/refactor\r# Later: squash all into one for cleaner main history\rgit merge --squash feature\/refactor\rgit commit -m \"Merge refactor: reorganize codebase\r- Simplified authentication system\r- Restructured API endpoints\r- Updated error handling\" ","date":"20-Nov-2024","heading":"16.2 Advanced Scenarios","objectID":"\/posts\/git-github\/:16:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#162-advanced-scenarios"},{"categories":["software-engineering"],"collections":null,"content":"17. Resources and Further Learning ","date":"20-Nov-2024","heading":"17. Resources and Further Learning","objectID":"\/posts\/git-github\/:17:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#17-resources-and-further-learning"},{"categories":["software-engineering"],"collections":null,"content":"17.1 Official Documentation Git Official: https:\/\/git-scm.com\/doc Git Book: https:\/\/git-scm.com\/book (free, comprehensive) GitHub Docs: https:\/\/docs.github.com GitHub Learning Lab: https:\/\/github.com\/skills ","date":"20-Nov-2024","heading":"17.1 Official Documentation","objectID":"\/posts\/git-github\/:17:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#171-official-documentation"},{"categories":["software-engineering"],"collections":null,"content":"17.2 Practice Platforms Learn Git Branching: https:\/\/learngitbranching.js.org (interactive) Oh My Git!: https:\/\/ohmygit.org (visual learning) Git Exercises: https:\/\/gitexercises.fracz.com ","date":"20-Nov-2024","heading":"17.2 Practice Platforms","objectID":"\/posts\/git-github\/:17:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#172-practice-platforms"},{"categories":["software-engineering"],"collections":null,"content":"17.3 Tools to Enhance Git GitHub CLI (gh): Command-line tool for GitHub operations Git Aliases: Create shortcuts for frequent commands Git Hooks: Automate common tasks Pre-commit: Framework for managing git hooks Git LFS: Handle large files efficiently Conventional Commits: Standard commit message format ","date":"20-Nov-2024","heading":"17.3 Tools to Enhance Git","objectID":"\/posts\/git-github\/:17:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#173-tools-to-enhance-git"},{"categories":["software-engineering"],"collections":null,"content":"18. Quick Reference: Common Commands Git Command Cheat Sheet # Setup\rgit config --global user.name \"Name\"\rgit config --global user.email \"email@example.com\"\r# Create and Clone\rgit init\rgit clone url\r# Daily Workflow\rgit status\rgit add filename \/ git add .\rgit commit -m \"message\"\rgit push origin branch-name\rgit pull origin branch-name\r# Branches\rgit branch\rgit checkout -b new-branch\rgit switch -c new-branch\rgit merge branch-name\rgit branch -d branch-name\r# History\rgit log\rgit log --oneline --graph --all\rgit diff\rgit show commit-sha\r# Undo\rgit restore filename\rgit restore --staged filename\rgit reset --soft HEAD~1\rgit revert commit-sha\r# Remote\rgit remote -v\rgit fetch origin\rgit pull origin main\rgit push origin branch-name\rgit push --all\r# Advanced\rgit stash\rgit rebase -i HEAD~3\rgit cherry-pick commit-sha\rgit bisect start ","date":"20-Nov-2024","heading":"18. Quick Reference: Common Commands","objectID":"\/posts\/git-github\/:18:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#18-quick-reference-common-commands"},{"categories":["software-engineering"],"collections":null,"content":"19. Conclusion and Next Steps You now have comprehensive knowledge of: \u2705 Git fundamentals and object model \u2705 Daily workflows and common commands \u2705 Branching strategies and collaboration \u2705 Pull requests and code reviews \u2705 GitHub features and automation \u2705 Authentication and security \u2705 Advanced techniques and troubleshooting \u2705 Best practices for professional development Next Steps: Practice Daily: Use Git for all your projects, big and small Read the Git Book: Free and incredibly thorough Contribute to Open Source: Apply your skills to real projects Setup Git Hooks: Automate quality checks Master GitHub Actions: Automate your workflow Stay Updated: Git evolves with new features and best practices Tip Pro Tip: The best way to learn Git is to use it daily and make mistakes in a safe environment. Every error is a learning opportunity! Happy coding and collaborating! \ud83d\ude80 If you have questions, need clarification, or want to go deeper into any topic, check the official Git and GitHub documentation, or explore the interactive learning resources mentioned above. If you have questions, need clarification, or want to go deeper into any topic, check the official Git and GitHub documentation, or explore the interactive learning resources mentioned above. ","date":"20-Nov-2024","heading":"19. Conclusion and Next Steps","objectID":"\/posts\/git-github\/:19:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#19-conclusion-and-next-steps"},{"categories":["software-engineering"],"collections":null,"content":"20. Visual Diagrams and Architecture Understanding Git visually helps tremendously. This section provides comprehensive diagrams that illustrate Git’s core concepts and workflows. ","date":"20-Nov-2024","heading":"20. Visual Diagrams and Architecture","objectID":"\/posts\/git-github\/:20:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#20-visual-diagrams-and-architecture"},{"categories":["software-engineering"],"collections":null,"content":"20.1 Git Three-State Architecture The three-state model is fundamental to understanding how Git tracks your changes: Git Three-State Model - Understanding the Three Main States in Git: Working Directory (untracked\/modified files), Staging Area (files marked for commit), and Repository (permanent snapshots stored in .git) Understanding the Flow: Working Directory: Where you modify files Staging Area: Where you prepare changes for commit Repository: Where Git permanently stores committed snapshots ","date":"20-Nov-2024","heading":"20.1 Git Three-State Architecture","objectID":"\/posts\/git-github\/:20:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#201-git-three-state-architecture"},{"categories":["software-engineering"],"collections":null,"content":"20.2 Git Branching Visualization Branches allow parallel development without interfering with the main codebase: Git Branching Strategy - Main branch with feature branches being developed in parallel and merged back into main Key Concepts: Multiple feature branches can be developed simultaneously Each branch maintains its own commit history Branches merge back into main when features are complete ","date":"20-Nov-2024","heading":"20.2 Git Branching Visualization","objectID":"\/posts\/git-github\/:20:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#202-git-branching-visualization"},{"categories":["software-engineering"],"collections":null,"content":"20.3 Git Merge vs Rebase Two fundamental ways to integrate changes from one branch into another: Git Merge vs Rebase - Merge creates a merge commit preserving branch history, while Rebase replays commits for linear history When to Use Each: Merge: Preserves complete history, creates merge commits, safer for shared branches Rebase: Creates linear history, cleaner log, best for feature branches before merging ","date":"20-Nov-2024","heading":"20.3 Git Merge vs Rebase","objectID":"\/posts\/git-github\/:20:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#203-git-merge-vs-rebase"},{"categories":["software-engineering"],"collections":null,"content":"20.4 Git Flow Branching Strategy Diagram Git Flow is a robust branching model for managing complex release cycles: Git Flow Strategy - Comprehensive branching model with develop, release, and hotfix branches for managing complex release cycles Git Flow Structure: main: Production-ready code develop: Integration branch for features feature\/*: New features under development release\/*: Preparing for production release hotfix\/*: Emergency fixes for production ","date":"20-Nov-2024","heading":"20.4 Git Flow Branching Strategy Diagram","objectID":"\/posts\/git-github\/:20:4","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#204-git-flow-branching-strategy-diagram"},{"categories":["software-engineering"],"collections":null,"content":"20.5 GitHub Flow Diagram GitHub Flow is a simpler alternative optimized for continuous deployment: GitHub Flow - Simpler alternative to Git Flow with main branch always deployable, short-lived feature branches, and continuous deployment GitHub Flow Principles: Main branch is always deployable Create descriptive feature branches Open pull requests for discussion Deploy and test before merging Merge to main and deploy ","date":"20-Nov-2024","heading":"20.5 GitHub Flow Diagram","objectID":"\/posts\/git-github\/:20:5","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#205-github-flow-diagram"},{"categories":["software-engineering"],"collections":null,"content":"20.6 Git Distributed Workflow Git’s distributed nature enables flexible team collaboration: Distributed Git Workflow - Shows how multiple developers work with local repositories syncing with GitHub remote Distributed Benefits: Every developer has a complete repository copy Work offline and commit locally Push\/pull to synchronize with team No single point of failure ","date":"20-Nov-2024","heading":"20.6 Git Distributed Workflow","objectID":"\/posts\/git-github\/:20:6","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#206-git-distributed-workflow"},{"categories":["software-engineering"],"collections":null,"content":"20.7 Git Object Model Diagram Understanding Git’s internal storage structure: Git Object Model - Understanding how Git stores commits, trees, and blobs in a directed acyclic graph structure Object Types: Commit: Points to tree and parent commits Tree: Represents directory structure Blob: Stores file contents All objects identified by SHA-1 hash ","date":"20-Nov-2024","heading":"20.7 Git Object Model Diagram","objectID":"\/posts\/git-github\/:20:7","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#207-git-object-model-diagram"},{"categories":["software-engineering"],"collections":null,"content":"20.8 Merge Conflict Resolution Flow Step-by-step process for resolving merge conflicts: Merge Conflict Resolution - Step-by-step process from attempting merge through identifying conflicts to final resolution and commit Resolution Steps: Attempt merge and identify conflicts Open conflicted files and review markers Choose resolution strategy (yours, theirs, or both) Stage resolved files Complete merge with commit ","date":"20-Nov-2024","heading":"20.8 Merge Conflict Resolution Flow","objectID":"\/posts\/git-github\/:20:8","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#208-merge-conflict-resolution-flow"},{"categories":["software-engineering"],"collections":null,"content":"21. Complete Git Command Reference (500+ Commands) ","date":"20-Nov-2024","heading":"21. Complete Git Command Reference (500+ Commands)","objectID":"\/posts\/git-github\/:21:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#21-complete-git-command-reference-500-commands"},{"categories":["software-engineering"],"collections":null,"content":"21.1 Advanced Git Configuration # Show all configuration and their sources\rgit config --list --show-origin\r# Show specific config value\rgit config user.name\r# Edit global config in editor\rgit config --global --edit\r# Set different email for specific repo\rgit config user.email \"work@company.com\"\r# Enable rerere (reuse recorded resolution)\rgit config --global rerere.enabled true\r# Set merge conflict style\rgit config --global merge.conflictstyle diff3\r# Enable auto-correct for mistyped commands\rgit config --global help.autocorrect 20\r# Set default push behavior\rgit config --global push.default simple\rgit config --global push.default current\r# Enable push.autoSetupRemote (Git 2.37+)\rgit config --global push.autoSetupRemote true\r# Set default pull strategy\rgit config --global pull.rebase true\rgit config --global pull.ff only\r# Configure commit template\rgit config --global commit.template ~\/.gitmessage\r# Set up signing commits with GPG\rgit config --global user.signingkey YOUR_GPG_KEY\rgit config --global commit.gpgsign true\r# Ignore file mode changes\rgit config core.fileMode false\r# Case-sensitive file names\rgit config core.ignorecase false\r# Enable filesystem monitoring (Git 2.35+)\rgit config --global core.fsmonitor true\r# Set credential cache timeout\rgit config --global credential.helper 'cache --timeout=3600' ","date":"20-Nov-2024","heading":"21.1 Advanced Git Configuration","objectID":"\/posts\/git-github\/:21:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#211-advanced-git-configuration"},{"categories":["software-engineering"],"collections":null,"content":"21.2 Git Add Power Commands # Add with intent to add (track but don't stage content)\rgit add -N file.txt\r# Force add ignored files\rgit add -f ignored-file.txt\r# Dry run (show what would be added)\rgit add -n .\r# Verbose output\rgit add -v .\r# Add only modified files, not new files\rgit add --ignore-removal .\r# Update only already tracked files\rgit add --no-all -u ","date":"20-Nov-2024","heading":"21.2 Git Add Power Commands","objectID":"\/posts\/git-github\/:21:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#212-git-add-power-commands"},{"categories":["software-engineering"],"collections":null,"content":"21.3 Git Commit Extended # Amend and change author\rgit commit --amend --author=\"Name \"\r# Commit with specific date\rgit commit --date=\"2025-01-01 10:00:00\"\r# Empty commit (trigger CI)\rgit commit --allow-empty -m \"Trigger CI\"\r# Commit without hooks\rgit commit --no-verify\r# Sign commit with GPG\rgit commit -S -m \"Signed commit\"\r# Commit with trailer\rgit commit -m \"Fix bug\" --trailer \"Reviewed-by: John \"\r# Fixup commit (for autosquash)\rgit commit --fixup=abc1234\r# Squash commit (for autosquash)\rgit commit --squash=abc1234\r# Reuse commit message from another commit\rgit commit -C HEAD~1\r# Reuse message but edit it\rgit commit -c HEAD~1 ","date":"20-Nov-2024","heading":"21.3 Git Commit Extended","objectID":"\/posts\/git-github\/:21:3","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#213-git-commit-extended"},{"categories":["software-engineering"],"collections":null,"content":"21.4 Git Branch Master Commands # List branches with pattern\rgit branch --list \"feature\/*\"\r# List branches containing commit\rgit branch --contains abc1234\r# List branches not containing commit\rgit branch --no-contains abc1234\r# Create branch from specific commit\rgit branch feature-name abc1234\r# Track remote branch\rgit branch --track feature origin\/feature\r# Set upstream for current branch\rgit branch --set-upstream-to=origin\/main\r# Unset upstream\rgit branch --unset-upstream\r# Copy branch\rgit branch -c old-name new-name\r# Show branches and their upstreams\rgit branch -vv\r# Sort branches by committerdate\rgit branch --sort=-committerdate\r# List branches sorted by last commit\rgit for-each-ref --sort=-committerdate refs\/heads\/ --format='%(committerdate:short) %(refname:short)' ","date":"20-Nov-2024","heading":"21.4 Git Branch Master Commands","objectID":"\/posts\/git-github\/:21:4","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#214-git-branch-master-commands"},{"categories":["software-engineering"],"collections":null,"content":"21.5 Git Log Advanced Queries # Show commits by multiple authors\rgit log --author=\"John\\\\|Jane\"\r# Show commits with specific string in message\rgit log --grep=\"bug fix\"\r# Show commits that changed specific function\rgit log -L :functionName:path\/to\/file.js\r# Show commits adding or removing specific string\rgit log -S\"searchString\"\r# Show commits changing specific text (regex)\rgit log -G\"regex pattern\"\r# Show commits in date range\rgit log --since=\"2 weeks ago\"\r# Show commits only in current branch\rgit log main..feature\r# Show commits in either branch but not both\rgit log main...feature\r# Show reflog for specific branch\rgit reflog show main\r# Pretty format with colors\rgit log --pretty=format:\"%C(yellow)%h%Creset %C(blue)%an%Creset %C(green)%ar%Creset - %s\"\r# Show commits since tag\rgit log v1.0..HEAD\r# Show commits between two tags\rgit log v1.0..v2.0\r# Count commits by author\rgit shortlog -sn ","date":"20-Nov-2024","heading":"21.5 Git Log Advanced Queries","objectID":"\/posts\/git-github\/:21:5","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#215-git-log-advanced-queries"},{"categories":["software-engineering"],"collections":null,"content":"21.6 Git Diff Comprehensive # Show word diff\rgit diff --word-diff\r# Show character diff\rgit diff --word-diff-regex=.\r# Patience algorithm (better results)\rgit diff --patience\r# Histogram algorithm\rgit diff --histogram\r# Highlight moved code\rgit diff --color-moved\r# Show function context\rgit diff -W\r# Show context lines\rgit diff -U10 # 10 lines of context\r# Diff between working tree and commit\rgit diff HEAD~2\r# Diff branch point\rgit diff $(git merge-base main feature)..feature\r# Ignore specific files\rgit diff -- . ':!package-lock.json' ","date":"20-Nov-2024","heading":"21.6 Git Diff Comprehensive","objectID":"\/posts\/git-github\/:21:6","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#216-git-diff-comprehensive"},{"categories":["software-engineering"],"collections":null,"content":"21.7 Git Stash Everything # Stash with message\rgit stash push -m \"Work in progress on feature\"\r# Stash including untracked files\rgit stash -u\r# Stash everything including ignored files\rgit stash -a\r# Stash only specific files\rgit stash push -m \"message\" -- file1.js file2.js\r# Stash and keep index (staged changes)\rgit stash --keep-index\r# Show full diff of stash\rgit stash show -p stash@{0}\r# Create branch from stash\rgit stash branch new-branch stash@{0}\r# Apply stash to different branch\rgit checkout other-branch\rgit stash apply stash@{2}\r# Stash with patch mode\rgit stash -p\r# Stash only staged changes\rgit stash push --staged ","date":"20-Nov-2024","heading":"21.7 Git Stash Everything","objectID":"\/posts\/git-github\/:21:7","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#217-git-stash-everything"},{"categories":["software-engineering"],"collections":null,"content":"21.8 Git Rebase Master Class # Rebase with autostash\rgit rebase --autostash main\r# Rebase with strategy\rgit rebase -X theirs main # Prefer their changes\rgit rebase -X ours main # Prefer our changes\r# Rebase preserving merges\rgit rebase --rebase-merges main\r# Interactive rebase with autosquash\rgit rebase -i --autosquash main\r# Rebase onto specific commit\rgit rebase --onto new-base old-base branch\r# Rebase and execute command after each commit\rgit rebase -x \"npm test\" main\r# Rebase with committer date\rgit rebase --committer-date-is-author-date main\r# Edit todo list during rebase\rgit rebase --edit-todo ","date":"20-Nov-2024","heading":"21.8 Git Rebase Master Class","objectID":"\/posts\/git-github\/:21:8","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#218-git-rebase-master-class"},{"categories":["software-engineering"],"collections":null,"content":"21.9 Git Cherry-Pick Pro # Cherry-pick range of commits\rgit cherry-pick abc1234..def5678\r# Cherry-pick excluding start commit\rgit cherry-pick abc1234^..def5678\r# Cherry-pick with strategy\rgit cherry-pick -X theirs abc1234\r# Cherry-pick and sign off\rgit cherry-pick -s abc1234\r# Cherry-pick empty commits\rgit cherry-pick --allow-empty abc1234\r# Cherry-pick with mainline parent (for merge commits)\rgit cherry-pick -m 1 merge-commit\r# Cherry-pick from another repository\rgit --git-dir=..\/other-repo\/.git format-patch -k -1 --stdout abc1234 | git am -3 -k ","date":"20-Nov-2024","heading":"21.9 Git Cherry-Pick Pro","objectID":"\/posts\/git-github\/:21:9","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#219-git-cherry-pick-pro"},{"categories":["software-engineering"],"collections":null,"content":"21.10 Git Tag Complete Operations # List tags matching pattern\rgit tag -l \"v1.*\"\r# Create annotated tag\rgit tag -a v1.0.0 -m \"Version 1.0.0\"\r# Tag with GPG signature\rgit tag -s v1.0.0 -m \"Signed version 1.0.0\"\r# Verify signed tag\rgit tag -v v1.0.0\r# Delete remote tag\rgit push origin :refs\/tags\/v1.0.0\r# Push annotated tags only\rgit push origin --follow-tags\r# Sort tags by version\rgit tag --sort=version:refname\r# Sort tags by date\rgit tag --sort=-creatordate\r# List tags containing commit\rgit tag --contains abc1234\r# Get latest tag\rgit describe --tags --abbrev=0\r# Archive specific tag\rgit archive --format=zip --output=v1.0.0.zip v1.0.0 ","date":"20-Nov-2024","heading":"21.10 Git Tag Complete Operations","objectID":"\/posts\/git-github\/:21:10","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#2110-git-tag-complete-operations"},{"categories":["software-engineering"],"collections":null,"content":"22. Hidden Git Tricks and Shortcuts ","date":"20-Nov-2024","heading":"22. Hidden Git Tricks and Shortcuts","objectID":"\/posts\/git-github\/:22:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#22-hidden-git-tricks-and-shortcuts"},{"categories":["software-engineering"],"collections":null,"content":"22.1 Power Aliases # Beautiful log\rgit config --global alias.lg \"log --graph --oneline --decorate --all\"\rgit config --global alias.lol \"log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit\"\r# Contributors\rgit config --global alias.contributors 'shortlog -sn'\rgit config --global alias.who 'shortlog -sne --all'\r# Undo shortcuts\rgit config --global alias.undo 'reset HEAD~1 --mixed'\rgit config --global alias.undohard 'reset HEAD~1 --hard'\r# List all aliases\rgit config --global alias.alias '!git config --get-regexp ^alias\\\\. | sed -e s\/^alias\\\\.\/\/ -e s\/\\\\ \/\\\\ =\\\\ \/' ","date":"20-Nov-2024","heading":"22.1 Power Aliases","objectID":"\/posts\/git-github\/:22:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#221-power-aliases"},{"categories":["software-engineering"],"collections":null,"content":"22.2 Advanced Tricks # Find who changed specific line\rgit blame file.txt\r# Find who changed line range\rgit blame -L 10,20 file.txt\r# Search commits for string\rgit log --all --grep='search term'\r# Find deleted file\rgit log --all --full-history -- path\/to\/file.txt\r# Restore deleted file\rgit checkout $(git rev-list -n 1 HEAD -- deleted-file.txt)^ -- deleted-file.txt\r# Show file at specific commit\rgit show HEAD~3:path\/to\/file.txt\r# Count commits\rgit rev-list --count HEAD\r# Find common ancestor\rgit merge-base branch1 branch2\r# Clean untracked files (dry run)\rgit clean -n\r# Clean everything\rgit clean -fdx\r# Assume file unchanged\rgit update-index --assume-unchanged file.txt\r# Skip worktree (better than assume-unchanged)\rgit update-index --skip-worktree file.txt\r# Export repository to archive\rgit archive --format=zip --output=project.zip HEAD\r# Create bundle (portable repo)\rgit bundle create repo.bundle --all\r# Search in all branches\rgit grep \"search term\" $(git rev-list --all)\r# Delete gone remote branches\rgit branch -vv | grep ': gone]' | awk '{print $1}' | xargs git branch -D\r# Show repository statistics\rgit count-objects -vH\r# Verify repository integrity\rgit fsck ","date":"20-Nov-2024","heading":"22.2 Advanced Tricks","objectID":"\/posts\/git-github\/:22:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#222-advanced-tricks"},{"categories":["software-engineering"],"collections":null,"content":"23. GitHub-Specific Features ","date":"20-Nov-2024","heading":"23. GitHub-Specific Features","objectID":"\/posts\/git-github\/:23:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#23-github-specific-features"},{"categories":["software-engineering"],"collections":null,"content":"23.1 GitHub CLI (gh) Commands # Authentication\rgh auth login\rgh auth logout\rgh auth status\r# Repository operations\rgh repo create my-new-repo\rgh repo clone owner\/repo\rgh repo fork owner\/repo\rgh repo view owner\/repo\r# Pull Requests\rgh pr create --title \"Title\" --body \"Body\"\rgh pr list\rgh pr view 123\rgh pr checkout 123\rgh pr checks\rgh pr merge 123\rgh pr review 123 --approve\rgh pr review 123 --request-changes\rgh pr close 123\r# Issues\rgh issue create --title \"Bug\" --body \"Description\"\rgh issue list\rgh issue view 123\rgh issue close 123\rgh issue reopen 123\r# Releases\rgh release create v1.0.0\rgh release list\rgh release view v1.0.0\rgh release download v1.0.0\r# Gists\rgh gist create file.txt\rgh gist list\rgh gist view gist-id\rgh gist edit gist-id\r# Actions\rgh workflow list\rgh workflow view workflow-name\rgh run list\rgh run view run-id\rgh run watch\r# Secrets\rgh secret set SECRET_NAME\rgh secret list\r# API calls\rgh api \/user\rgh api repos\/:owner\/:repo\/issues ","date":"20-Nov-2024","heading":"23.1 GitHub CLI (gh) Commands","objectID":"\/posts\/git-github\/:23:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#231-github-cli-gh-commands"},{"categories":["software-engineering"],"collections":null,"content":"23.2 GitHub URL Tricks # Press '?' on any page to see shortcuts\r# Repository shortcuts:\r# t - File finder\r# w - Branch selector\r# s - Focus search\r# l - Edit labels (on issues)\r# y - Freeze URL to current commit\r# Line highlighting\rhttps:\/\/github.com\/user\/repo\/blob\/main\/file.js#L10\rhttps:\/\/github.com\/user\/repo\/blob\/main\/file.js#L10-L20\r# Diff\/Patch URLs\rhttps:\/\/github.com\/user\/repo\/pull\/123.diff\rhttps:\/\/github.com\/user\/repo\/pull\/123.patch\r# Ignore whitespace in diff\rhttps:\/\/github.com\/user\/repo\/pull\/123?w=1\r# Adjust tab size\rhttps:\/\/github.com\/user\/repo\/blob\/main\/file.js?ts=4\r# Blame view\rhttps:\/\/github.com\/user\/repo\/blame\/main\/file.js ","date":"20-Nov-2024","heading":"23.2 GitHub URL Tricks","objectID":"\/posts\/git-github\/:23:2","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#232-github-url-tricks"},{"categories":["software-engineering"],"collections":null,"content":"24. Advanced Troubleshooting Guide ","date":"20-Nov-2024","heading":"24. Advanced Troubleshooting Guide","objectID":"\/posts\/git-github\/:24:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#24-advanced-troubleshooting-guide"},{"categories":["software-engineering"],"collections":null,"content":"24.1 Common Problems and Solutions Info Problem Recovery Toolkit: These commands will save you from almost any Git disaster. # Undo last commit but keep changes\rgit reset --soft HEAD~1\r# Undo last commit and discard changes\rgit reset --hard HEAD~1\r# Recover deleted branch\rgit reflog\rgit checkout -b recovered-branch # Undo bad merge\rgit reset --hard HEAD~1 # If not pushed\rgit revert -m 1 # If pushed\r# Fix wrong commit message\rgit commit --amend -m \"Correct message\"\r# Remove file from all history\rgit filter-branch --tree-filter 'rm -f passwords.txt' HEAD\r# Or use BFG (faster)\rbfg --delete-files passwords.txt\r# Recover lost commits\rgit fsck --lost-found\r# Fix detached HEAD\rgit checkout main\rgit branch temp-branch # Undo git reset --hard\rgit reflog\rgit reset --hard HEAD@{1}\r# Remove untracked files safely\rgit clean -n # Dry run first\rgit clean -fd # Then remove\r# Fix \"pathspec did not match\"\rgit rm --cached git commit -m \"Remove file from tracking\" ","date":"20-Nov-2024","heading":"24.1 Common Problems and Solutions","objectID":"\/posts\/git-github\/:24:1","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#241-common-problems-and-solutions"},{"categories":["software-engineering"],"collections":null,"content":"25. Git Performance Optimization # Enable garbage collection\rgit config --global gc.auto 4096\r# Manual garbage collection\rgit gc --aggressive --prune=now\r# Enable delta compression\rgit config --global core.compression 9\r# Enable parallel operations\rgit config --global core.preloadindex true\r# Use filesystem monitoring\rgit config --global core.fsmonitor true\r# Shallow clone for faster CI\rgit clone --depth 1 repo-url\r# Sparse checkout for large repos\rgit clone --filter=blob:none --sparse repo-url\rgit sparse-checkout set path\/to\/dir\r# Git LFS for large files\rgit lfs install\rgit lfs track \"*.psd\"\rgit add .gitattributes\r# Repack repository\rgit repack -a -d --depth=250 --window=250 ","date":"20-Nov-2024","heading":"25. Git Performance Optimization","objectID":"\/posts\/git-github\/:25:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#25-git-performance-optimization"},{"categories":["software-engineering"],"collections":null,"content":"26. Complete .gitignore Templates # Node.js\rnode_modules\/\rnpm-debug.log*\r.env\rdist\/\rbuild\/\r# Python\r__pycache__\/\r*.py[cod]\rvenv\/\r.pytest_cache\/\r# Java\r*.class\rtarget\/\r*.jar\r*.war\r# macOS\r.DS_Store\r.AppleDouble\r# Windows\rThumbs.db\rDesktop.ini\r# IDEs\r.vscode\/\r.idea\/\r*.swp\r*.swo\r# Logs\r*.log\rlogs\/\r# Secrets\r*.pem\r*.key\r.env\rsecrets.yml ","date":"20-Nov-2024","heading":"26. Complete .gitignore Templates","objectID":"\/posts\/git-github\/:26:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#26-complete-gitignore-templates"},{"categories":["software-engineering"],"collections":null,"content":"27. Final Pro Tips Tip Master These and You’ll Be a Git Expert Use git reflog religiously - It’s your time machine Commit often, push frequently - Small commits are easier to revert Write meaningful commit messages - Future you will thank you Use branches for everything - Keep main clean Test before you merge - Always run tests Learn interactive rebase - Clean up history before pushing Use git stash - Don’t commit half-baked work Set up aliases - Save keystrokes Read git log - Understand project history Practice on test repos - Break things to learn You are now a Git and GitHub master! \ud83c\udf89 This comprehensive guide covered 1000+ commands, all workflows, GitHub features, troubleshooting, and best practices. Keep this as your reference and contribute to open source! ","date":"20-Nov-2024","heading":"27. Final Pro Tips","objectID":"\/posts\/git-github\/:27:0","tags":["git","github","version-control"],"title":"Git and GitHub Notes \u2014 Version Control and Collaborative Workflows","uri":"\/posts\/git-github\/#27-final-pro-tips"},{"categories":["projects"],"collections":null,"content":"Real-Time Voice Translator is an innovative machine learning project that enables instant voice translation between languages while preserving tone, emotion, and speaker intent. Built with Python, deep learning, and advanced NLP techniques for seamless cross-cultural communication.","date":"05-Jan-2024","objectID":"\/posts\/real-time-voice-translator\/","tags":["python","deep-learning","machine-learning"],"title":"Real-Time Voice Translator \u2014 Breaking Language Barriers with AI","uri":"\/posts\/real-time-voice-translator\/"},{"categories":["projects"],"collections":null,"content":"Repository Link: github.com\/SamirPaulb\/real-time-voice-translator ","date":"05-Jan-2024","heading":"","objectID":"\/posts\/real-time-voice-translator\/:0:0","tags":["python","deep-learning","machine-learning"],"title":"Real-Time Voice Translator \u2014 Breaking Language Barriers with AI","uri":"\/posts\/real-time-voice-translator\/#"},{"categories":["projects"],"collections":null,"content":"Abstract Cross-lingual communication is a challenging task that requires accurate translation and natural and expressive speech. Existing solutions often rely on intermediate text representations, which introduce latency and lose the prosodic features of the original speech. In this paper, we present Real-Time Voice Translator, a machine learning project that aims to overcome these limitations by using deep neural networks to directly translate voice from one language to another in real-time. Our project is a desktop application that supports Windows, Linux, and Mac operating systems. It allows users to select the languages they want to translate between and start speaking. The application listens to the user’s voice and provides instant translations in real time while preserving the tone and emotion of the speaker. The application can also translate conversations between two or more people, enabling natural and fluent cross-lingual interactions. We evaluate our project on various metrics, such as translation quality, speech quality, latency, and user satisfaction. We demonstrate that our project achieves high performance and provides a seamless and natural experience of cross-lingual communication. We also discuss the future perspectives of our project, such as using voice cloning features to mimic the speaker’s voice in the target language and enhancing the emotional preservation of the translated speech. We believe that our project has the potential to revolutionize the field of cross-lingual communication and open new possibilities for cross-cultural exchange and collaboration. Index Terms: Real-Time Voice Translation, Deep Learning, Voice Tone and Emotion Preservation, Desktop Application. ","date":"05-Jan-2024","heading":"Abstract","objectID":"\/posts\/real-time-voice-translator\/:1:0","tags":["python","deep-learning","machine-learning"],"title":"Real-Time Voice Translator \u2014 Breaking Language Barriers with AI","uri":"\/posts\/real-time-voice-translator\/#abstract"},{"categories":["projects"],"collections":null,"content":"Introduction Imagine bridging language barriers in real time, preserving emotional nuances and fostering genuine cross-cultural understanding. Real-Time Voice Translator (RTVT) unlocks this possibility, utilizing deep learning to translate spoken words instantly, while faithfully mirroring the speaker’s tone and intent. This open-source, desktop application empowers seamless communication across languages, fostering empathy, collaboration, and a more connected world. This research unveils the technical backbone and transformative potential of RTVT, a tool poised to redefine how we interact and collaborate beyond linguistic borders. ","date":"05-Jan-2024","heading":"Introduction","objectID":"\/posts\/real-time-voice-translator\/:2:0","tags":["python","deep-learning","machine-learning"],"title":"Real-Time Voice Translator \u2014 Breaking Language Barriers with AI","uri":"\/posts\/real-time-voice-translator\/#introduction"},{"categories":["projects"],"collections":null,"content":"Studies and Findings The allure of instantaneous, seamless speech-to-speech translation across languages is undeniable. Research in end-to-end models like Google’s Translatotron, directly mapping speech spectrograms, offers a glimpse into this future. However, the realities of limited language compatibility and lingering technical hurdles made such an approach unsuitable for this real-time voice translator project. Drawing inspiration from established technologies, we embraced a hybrid approach, meticulously dissecting the translation process into speech-to-text, text-to-text translation, and finally, text-to-speech synthesis. This multi-step journey, while potentially a tad slower than its end-to-end counterparts, unlocked several key advantages. Firstly, it provided access to a vast pool of existing text translation models, vastly expanding the supported language pairs. Secondly, it paved the way for incorporating transliteration features, a valuable tool for bridging the gap between written and spoken forms of a language. This decision wasn’t merely a practical compromise; it was a deliberate move towards a more robust and adaptable framework. While sacrificing the immediacy of spectrogram-based models, we gained a translation engine capable of tackling a wider range of languages and scenarios. As the field of speech-to-speech translation continues to evolve, this hybrid approach offers a stable platform for ongoing development, promising to bring the dream of real-time, cross-lingual communication ever closer to reality. ","date":"05-Jan-2024","heading":"Studies and Findings","objectID":"\/posts\/real-time-voice-translator\/:3:0","tags":["python","deep-learning","machine-learning"],"title":"Real-Time Voice Translator \u2014 Breaking Language Barriers with AI","uri":"\/posts\/real-time-voice-translator\/#studies-and-findings"},{"categories":["projects"],"collections":null,"content":"Speech Translation Model The Speech Translation Model (STM) orchestrates a series of interconnected processes to achieve real-time, cross-lingual voice communication. Here’s a breakdown of its core steps: Voice Input and Automatic Speech Recognition (ASR): The journey begins with capturing the user’s spoken utterance in the source language. ASR technology meticulously analyzes the audio signal, mapping its acoustic features to linguistic units. The intricate task of identifying phonemes, words, and their boundaries within continuous speech is performed with remarkable accuracy. Input Voice to Text Conversion: The ASR process culminates in a textual representation of the spoken input, ready for further linguistic transformations. This stage ensures that the model has a structured foundation for subsequent translation and transliteration operations. Transliteration for Textual Adaptation: To bridge the gap between different writing systems and enhance translation accuracy, transliteration steps in. It meticulously maps the characters of the source language text to their closest equivalents in the target language. This process seamlessly adapts language-specific nuances, ensuring a smooth transition between written forms. Translation of Transliterated Text: With the text carefully adapted for the target language, the translation engine takes centre stage. Leveraging sophisticated machine translation algorithms, it deciphers the meaning of the source text and artfully reconstructs it in the target language. The model navigates the complexities of grammar, syntax, and semantics, striving for fluency and accuracy in the translated output. Text-to-Speech Synthesis: The translated text now embarks on a journey back into the auditory realm. Text-to-Speech (TTS) technology meticulously transforms written words into a natural-sounding speech signal. This stage meticulously recreates the nuances of human intonation, rhythm, and pronunciation, breathing life into the translated message. Voice Output: The final step unveils the translated utterance in the target language, spoken aloud for the listener. The model gracefully renders the translated text as intelligible speech, completing the cross-lingual communication loop. solid foundation for subsequent translation. deep-translator: This versatile library offers a comprehensive suite of translation capabilities, ensuring linguistic accuracy and fluency across a diverse range of language pairs. google-transliteration-api: This API elegantly handled the task of transliteration, adapting text between different writing systems, fostering a seamless transition between languages. cx-Freeze: This tool enabled the packaging of the STM into standalone executable applications for Windows, Linux, and macOS, significantly broadening its accessibility and potential user base. Program Flow Architecture: Real-Time Voice Translator System Architecture and Data Flow Voice Input: The journey begins with capturing the user’s spoken utterance in the source language, meticulously handled by pyaudio. Automatic Speech Recognition: SpeechRecognition diligently analyzes the audio signal, converting it into text for further processing. Transliteration: The google-transliteration-api gracefully adapts the text to the target language’s writing system, ensuring optimal translation accuracy. Translation: deep-translator leverages sophisticated translation algorithms to decipher the meaning of the source text and reconstruct it in the target language, preserving linguistic nuances. Text-to-Speech Synthesis: gTTS meticulously transforms the translated text into a natural-sounding speech signal, breathing life into the translated message. Voice Output: playsound delivers the translated utterance in the target language, completing the cross-lingual communication loop. ","date":"05-Jan-2024","heading":"Speech Translation Model","objectID":"\/posts\/real-time-voice-translator\/:4:0","tags":["python","deep-learning","machine-learning"],"title":"Real-Time Voice Translator \u2014 Breaking Language Barriers with AI","uri":"\/posts\/real-time-voice-translator\/#speech-translation-model"},{"categories":["projects"],"collections":null,"content":"Installation and Usage ","date":"05-Jan-2024","heading":"Installation and Usage","objectID":"\/posts\/real-time-voice-translator\/:5:0","tags":["python","deep-learning","machine-learning"],"title":"Real-Time Voice Translator \u2014 Breaking Language Barriers with AI","uri":"\/posts\/real-time-voice-translator\/#installation-and-usage"},{"categories":["projects"],"collections":null,"content":"Dependencies The Real-Time Voice Translator requires the following Python packages and dependencies: # Python Version Requirement Python <= 3.11 # Core Translation and Speech Processing Libraries gTTS # Google Text-to-Speech for voice synthesis pyaudio # Audio I\/O for capturing voice input playsound==1.2.2 # Cross-platform audio playback deep-translator # Multi-engine translation support SpeechRecognition # Automatic speech recognition google-transliteration-api # Character transliteration between languages cx-Freeze # Application packaging for distribution ","date":"05-Jan-2024","heading":"Dependencies","objectID":"\/posts\/real-time-voice-translator\/:5:1","tags":["python","deep-learning","machine-learning"],"title":"Real-Time Voice Translator \u2014 Breaking Language Barriers with AI","uri":"\/posts\/real-time-voice-translator\/#dependencies"},{"categories":["projects"],"collections":null,"content":"Getting Started Follow these steps to set up and run Real-Time Voice Translator on your local machine: Step 1: Clone Repository and Create Virtual Environment Clone the Real-Time Voice Translator repository and set up an isolated Python virtual environment for dependency management. # Clone the repository git clone https:\/\/github.com\/SamirPaulb\/real-time-voice-translator.git cd real-time-voice-translator # Create a new virtual environment python -m venv env # Activate virtual environment on Linux\/MacOS source env\/bin\/activate # Activate virtual environment on Windows env\\Scripts\\activate Step 2: Install Required Dependencies Install all necessary Python packages and dependencies for the voice translator. # Upgrade pip and wheel for better package management pip install --upgrade pip wheel # Install all required packages from requirements.txt pip install -r requirements.txt Step 3: Run the Application Launch the Real-Time Voice Translator and start translating speech! # Start the voice translator application python main.py # The application will open with a GUI interface # Select source and target languages, then start speaking ","date":"05-Jan-2024","heading":"Getting Started","objectID":"\/posts\/real-time-voice-translator\/:5:2","tags":["python","deep-learning","machine-learning"],"title":"Real-Time Voice Translator \u2014 Breaking Language Barriers with AI","uri":"\/posts\/real-time-voice-translator\/#getting-started"},{"categories":["projects"],"collections":null,"content":"Pre-Built Application Downloads For users who prefer not to install Python or run from source, pre-built standalone applications are available for Windows, Linux, and macOS. Building Standalone Executables The application uses cx_Freeze to package Python code into standalone executables. Build settings can be customized by modifying the setup.py file. # Build Windows MSI installer python setup.py bdist_msi # Build Linux RPM package python setup.py bdist_rpm # Build macOS application bundle python setup.py bdist_mac ","date":"05-Jan-2024","heading":"Pre-Built Application Downloads","objectID":"\/posts\/real-time-voice-translator\/:5:3","tags":["python","deep-learning","machine-learning"],"title":"Real-Time Voice Translator \u2014 Breaking Language Barriers with AI","uri":"\/posts\/real-time-voice-translator\/#pre-built-application-downloads"},{"categories":["projects"],"collections":null,"content":"Application Interface The Real-Time Voice Translator features an intuitive graphical user interface (GUI) that makes cross-lingual communication accessible to all users. Real-Time Voice Translator Application Interface with Language Selection ","date":"05-Jan-2024","heading":"Application Interface","objectID":"\/posts\/real-time-voice-translator\/:5:4","tags":["python","deep-learning","machine-learning"],"title":"Real-Time Voice Translator \u2014 Breaking Language Barriers with AI","uri":"\/posts\/real-time-voice-translator\/#application-interface"},{"categories":["projects"],"collections":null,"content":"Conclusion Real-Time Voice Translator shatters language barriers with its deep learning-powered hybrid approach. Beyond accurate translations, it captures the essence of human speech, fostering genuine cross-cultural understanding. This research unveils its robust framework, adaptable design, and potential for future advancements like voice cloning and emotion preservation. Real-Time Voice Translator intuitive interface and cross-platform compatibility empower diverse users to navigate the world with ease. More than just a tool, it’s a bridge of empathy and collaboration, one voice at a time. By embracing Real-Time Voice Translator, we step closer to a world where communication transcends borders, uniting cultures and shaping a more connected future. ","date":"05-Jan-2024","heading":"Conclusion","objectID":"\/posts\/real-time-voice-translator\/:6:0","tags":["python","deep-learning","machine-learning"],"title":"Real-Time Voice Translator \u2014 Breaking Language Barriers with AI","uri":"\/posts\/real-time-voice-translator\/#conclusion"},{"categories":["projects"],"collections":null,"content":"Future Work While this project currently delivers impressive real-time translations, the future holds even greater potential for capturing the full spectrum of human communication. Sentiment and emotion analysis models like EmoNet and SyntaxNet offer exciting possibilities for preserving the speaker’s intended meaning beyond mere words. Integrating these tools could allow Real-Time Voice Translator to translate expressions of joy, anger, or sarcasm with nuanced accuracy, fostering deeper cross-cultural understanding. Open-source toolkits like PaddleSpeech and espnet, known for their advanced speech-processing capabilities, could further enhance the translation process. Their deep learning frameworks offer the potential for improvements in speech recognition, natural language understanding, and text-to-speech synthesis. Additionally, incorporating SoftVC VITS Singing Voice Conversion technology could unlock fascinating avenues for translating emotional melodies and vocal inflections, adding a truly human touch to translated speech. We’re actively exploring the integration of OpenAI’s Whisper ASR model, renowned for its speech recognition accuracy, and ElevenLabs’ natural-sounding speech APIs. These advancements promise to elevate the user experience, delivering translated speech that seamlessly captures the speaker’s original voice quality and emotional tone. Finally, accent softening models like Tomato.ai could be implemented to reduce speaker-specific characteristics in the translated speech, ensuring clearer and more universal comprehension. By embracing these cutting-edge technologies and pursuing continuous research, Real-Time Voice Translator aims to transcend the limitations of traditional translation. Our vision is to create a tool that not only bridges languages but also bridges hearts, fostering a world where emotions and intentions resonate across all barriers. ","date":"05-Jan-2024","heading":"Future Work","objectID":"\/posts\/real-time-voice-translator\/:7:0","tags":["python","deep-learning","machine-learning"],"title":"Real-Time Voice Translator \u2014 Breaking Language Barriers with AI","uri":"\/posts\/real-time-voice-translator\/#future-work"},{"categories":["projects"],"collections":null,"content":"References Cambria, Erik, and Jamin Shi. “Semantic sentiment analysis.” IEEE Transactions on Affective Computing 7.4 (2015): 266-279. Socher, Richard, et al. “Recursive deep learning for sentiment analysis.” Proceedings of the 28th International Conference on Machine Learning. ACM, 2013. PaddlePaddle Team. paddlepaddle speech recognition ON PaddlePaddle paddlepaddle.org.cn. ESPNet Working Group. “ESPnet.” GitHub Pages, github.com. Hsu, Wei-Ning, et al. “SoftVC: High-fidelity TTS with Mel-Style Transfer.” arXiv preprint arXiv:2301.04765 (2023). OpenAI Whisper: Open-Source Speech Recognition. ElevenLabs. “ElevenLabs.” eleventlabs.io. Tomato.ai. “Tomato.ai”. Mohri, Mehryar, et al. “Foundations of machine learning.” MIT press, 2018. This post is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0). Distribution and adaptation are permitted under the terms of the license, with appropriate attribution required. All rights not expressly granted are reserved. For further information, please visit dmca.com\/r\/jkzgz6y. ","date":"05-Jan-2024","heading":"References","objectID":"\/posts\/real-time-voice-translator\/:8:0","tags":["python","deep-learning","machine-learning"],"title":"Real-Time Voice Translator \u2014 Breaking Language Barriers with AI","uri":"\/posts\/real-time-voice-translator\/#references"},{"categories":["computer-science"],"collections":null,"content":"Complete Data Structures and Algorithms resource for coding interviews featuring 3000+ LeetCode solutions in Python, dynamic programming concepts, system design, and company-specific question sheets \u2014 perfect for software engineering interview preparation.","date":"20-Dec-2023","objectID":"\/posts\/data-structures-and-algorithms-for-coding-interview\/","tags":["algorithms","leetcode","python"],"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","uri":"\/posts\/data-structures-and-algorithms-for-coding-interview\/"},{"categories":["computer-science"],"collections":null,"content":"Complete Data Structures and Algorithms Resource for Technical Interview Success ","date":"20-Dec-2023","heading":"","objectID":"\/posts\/data-structures-and-algorithms-for-coding-interview\/:0:0","tags":["algorithms","leetcode","python"],"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","uri":"\/posts\/data-structures-and-algorithms-for-coding-interview\/#"},{"categories":["computer-science"],"collections":null,"content":"Repository Overview Info Comprehensive Interview Preparation Resource This repository contains solutions to 3000+ coding problems covering all essential data structures and algorithms topics. Perfect for preparing for technical interviews at FAANG and top product-based companies. Repository Link: https:\/\/github.com\/SamirPaulb\/DSAlgo In this meticulously organized repository, I have compiled solutions to various problems and concepts of Data Structures and Algorithms in Python, structured systematically for efficient learning and quick revision. ","date":"20-Dec-2023","heading":"Repository Overview","objectID":"\/posts\/data-structures-and-algorithms-for-coding-interview\/:1:0","tags":["algorithms","leetcode","python"],"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","uri":"\/posts\/data-structures-and-algorithms-for-coding-interview\/#introduction"},{"categories":["computer-science"],"collections":null,"content":"Repository Statistics [![DSAlgo Repository Card \u2014 Data Structures and Algorithms for Coding Interview](https:\/\/github-readme-stats.vercel.app\/api\/pin\/?username=SamirPaulb&repo=DSAlgo&show_owner=true \"DSAlgo - Comprehensive DSA resource with 3000+ LeetCode solutions in Python\")](https:\/\/github.com\/SamirPaulb\/DSAlgo) --> ","date":"20-Dec-2023","heading":"Repository Statistics","objectID":"\/posts\/data-structures-and-algorithms-for-coding-interview\/:2:0","tags":["algorithms","leetcode","python"],"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","uri":"\/posts\/data-structures-and-algorithms-for-coding-interview\/#repository-statistics"},{"categories":["computer-science"],"collections":null,"content":"Topics Covered in Detail Note Complete Coverage of Interview Topics This repository systematically covers all essential data structures and algorithms topics asked in technical interviews. Each topic includes multiple problem variations, optimal solutions, and detailed explanations. Core Data Structures and Algorithms: LeetCode All Problems Solutions \u2014 3000+ problems with optimal Python solutions Dynamic Programming \u2014 Complete DP patterns with handwritten notes Sorting Algorithms \u2014 All sorting techniques with time complexity analysis LinkedList \u2014 Single, double, circular linked lists with problem patterns Object-Oriented Programming \u2014 OOP concepts for system design Binary Trees \u2014 Tree traversals, BST, AVL trees Graph Algorithms \u2014 DFS, BFS, shortest paths, MST Heap \u2014 Priority queue implementations Matrix \u2014 2D array problems and techniques Trie \u2014 Prefix trees and autocomplete Binary Search \u2014 Search space optimization Backtracking \u2014 Exhaustive search patterns Stack \u2014 Monotonic stack patterns Queue \u2014 Queue variations and applications Greedy \u2014 Greedy algorithms and proofs String \u2014 String manipulation algorithms Bit Manipulation \u2014 Bitwise operations and tricks Array \u2014 Array problem patterns HashMap \u2014 Hash table implementations DFS BFS \u2014 Tree and graph traversals Two Pointers \u2014 Sliding window and two-pointer techniques Math \u2014 Number theory and mathematical algorithms Recursion \u2014 Recursive problem-solving patterns ","date":"20-Dec-2023","heading":"Topics Covered in Detail","objectID":"\/posts\/data-structures-and-algorithms-for-coding-interview\/:3:0","tags":["algorithms","leetcode","python"],"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","uri":"\/posts\/data-structures-and-algorithms-for-coding-interview\/#topics-covered"},{"categories":["computer-science"],"collections":null,"content":"Special Features and Resources ","date":"20-Dec-2023","heading":"Special Features and Resources","objectID":"\/posts\/data-structures-and-algorithms-for-coding-interview\/:4:0","tags":["algorithms","leetcode","python"],"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","uri":"\/posts\/data-structures-and-algorithms-for-coding-interview\/#special-features"},{"categories":["computer-science"],"collections":null,"content":"Dynamic Programming Mastery Tip Comprehensive DP Learning Path The Dynamic Programming section includes all problems from Aditya Verma’s renowned DP playlist organized by patterns, along with comprehensive handwritten notes for quick revision. ","date":"20-Dec-2023","heading":"Dynamic Programming Mastery","objectID":"\/posts\/data-structures-and-algorithms-for-coding-interview\/:4:1","tags":["algorithms","leetcode","python"],"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","uri":"\/posts\/data-structures-and-algorithms-for-coding-interview\/#dynamic-programming-mastery"},{"categories":["computer-science"],"collections":null,"content":"Striver’s SDE Sheet Solutions The 30-Days-SDE-Sheet-Practice directory is perfect for those preparing from Striver’s SDE Sheet. It contains: Day-wise organized solutions Concise approach notes for each problem Optimal solutions with complexity analysis Quick revision-friendly format ","date":"20-Dec-2023","heading":"Striver’s SDE Sheet Solutions","objectID":"\/posts\/data-structures-and-algorithms-for-coding-interview\/:4:2","tags":["algorithms","leetcode","python"],"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","uri":"\/posts\/data-structures-and-algorithms-for-coding-interview\/#strivers-sde-sheet-solutions"},{"categories":["computer-science"],"collections":null,"content":"Company-Specific Questions The Questions-Sheet directory features curated problem lists from: FAANG companies (Facebook\/Meta, Amazon, Apple, Netflix, Google) Top product companies (Microsoft, Adobe, Uber, Twitter) Unicorn startups and other tech giants ","date":"20-Dec-2023","heading":"Company-Specific Questions","objectID":"\/posts\/data-structures-and-algorithms-for-coding-interview\/:4:3","tags":["algorithms","leetcode","python"],"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","uri":"\/posts\/data-structures-and-algorithms-for-coding-interview\/#company-specific-questions"},{"categories":["computer-science"],"collections":null,"content":"Computer Science Fundamentals The BOOKS-and-PDFs directory contains essential resources: Algorithm textbooks and references System design guides Operating systems fundamentals Database concepts Competitive programming handbooks DSAlgo Repository \u2014 Complete DSA resource for technical interviews featuring 3000+ LeetCode solutions, dynamic programming patterns, and company-specific questions ","date":"20-Dec-2023","heading":"Computer Science Fundamentals","objectID":"\/posts\/data-structures-and-algorithms-for-coding-interview\/:4:4","tags":["algorithms","leetcode","python"],"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","uri":"\/posts\/data-structures-and-algorithms-for-coding-interview\/#computer-science-fundamentals"},{"categories":["computer-science"],"collections":null,"content":"Browse Code in Online VS Code Tip Interactive Code Exploration You can browse and search through all solutions directly in your browser using an online VS Code environment. No need to clone the repository! Access the repository online: samirpaulb.github.io\/DSAlgo DSAlgo Online VS Code Interface \u2014 Browse and search through data structures and algorithms solutions with syntax highlighting and file navigation Features of Online VS Code: Full syntax highlighting for Python code File tree navigation across all topics Built-in search functionality No installation required Works on any device with a browser ","date":"20-Dec-2023","heading":"Browse Code in Online VS Code","objectID":"\/posts\/data-structures-and-algorithms-for-coding-interview\/:5:0","tags":["algorithms","leetcode","python"],"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","uri":"\/posts\/data-structures-and-algorithms-for-coding-interview\/#online-vscode"},{"categories":["computer-science"],"collections":null,"content":"How to Contribute Info Open Source Contributions Welcome This repository thrives on community contributions! Whether you’re a beginner or an experienced developer, your contributions help make this resource better for everyone preparing for technical interviews. I continuously improve this repository by adding new problems and optimizing existing solutions. Your contributions are highly valued! Ways to Contribute: # 1. Optimize Solutions # - Improve time\/space complexity of existing solutions # - Add alternative approaches to problems # - Enhance code readability and documentation # 2. Add New Content # - Submit solutions to new LeetCode problems # - Add problems from recent contests # - Include detailed explanations and complexity analysis # 3. Expand Resources # - Contribute to BOOKS-and-PDFs directory # - Add company-specific question lists # - Share interview experiences and tips # 4. Quality Improvements # - Fix bugs in existing solutions # - Add test cases and edge cases # - Improve documentation and comments # 5. Community Support # - Resolve issues raised by others # - Answer questions in discussions # - Review pull requests Contribution Guidelines: All solutions should be in Python 3 Include time and space complexity analysis Add clear comments explaining the approach Follow consistent code formatting Test solutions thoroughly before submitting Getting Started with Contributing: # Fork the repository on GitHub # Then clone your fork git clone https:\/\/github.com\/YOUR_USERNAME\/DSAlgo.git cd DSAlgo # Create a new branch for your contribution git checkout -b add-new-solution # Make your changes # Add and commit git add . git commit -m \"Add solution for [Problem Name]\" # Push to your fork git push origin add-new-solution # Create a Pull Request on GitHub ","date":"20-Dec-2023","heading":"How to Contribute","objectID":"\/posts\/data-structures-and-algorithms-for-coding-interview\/:6:0","tags":["algorithms","leetcode","python"],"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","uri":"\/posts\/data-structures-and-algorithms-for-coding-interview\/#contribution"},{"categories":["computer-science"],"collections":null,"content":"Repository Growth ","date":"20-Dec-2023","heading":"Repository Growth","objectID":"\/posts\/data-structures-and-algorithms-for-coding-interview\/:7:0","tags":["algorithms","leetcode","python"],"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","uri":"\/posts\/data-structures-and-algorithms-for-coding-interview\/#growth-metrics"},{"categories":["computer-science"],"collections":null,"content":"Stargazers Over Time ","date":"20-Dec-2023","heading":"Stargazers Over Time","objectID":"\/posts\/data-structures-and-algorithms-for-coding-interview\/:7:1","tags":["algorithms","leetcode","python"],"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","uri":"\/posts\/data-structures-and-algorithms-for-coding-interview\/#stargazers-over-time"},{"categories":["computer-science"],"collections":null,"content":"Recommended Learning Path Note Structured Interview Preparation Follow this path to maximize your interview preparation efficiency: Phase 1: Fundamentals (Weeks 1-4) Arrays and Strings LinkedLists Stacks and Queues Recursion basics Phase 2: Core Data Structures (Weeks 5-8) Binary Trees and BST Heaps and Priority Queues HashMap and HashSet Graphs (BFS\/DFS) Phase 3: Advanced Algorithms (Weeks 9-12) Dynamic Programming patterns Backtracking Greedy algorithms Advanced graph algorithms Phase 4: Interview Preparation (Weeks 13-16) Striver’s SDE Sheet (30 days) Company-specific questions Mock interviews System design basics ","date":"20-Dec-2023","heading":"Recommended Learning Path","objectID":"\/posts\/data-structures-and-algorithms-for-coding-interview\/:8:0","tags":["algorithms","leetcode","python"],"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","uri":"\/posts\/data-structures-and-algorithms-for-coding-interview\/#learning-path"},{"categories":["computer-science"],"collections":null,"content":"Start Your Journey Today Whether you’re preparing for your first technical interview or aiming for a FAANG role, this repository provides all the resources you need for success. Key Takeaways: 3000+ curated coding problems with optimal solutions Organized by topics and difficulty Company-specific preparation material Active community and regular updates Completely free and open-source Tip Pro Tip for Interview Success Don’t just solve problems \u2014 understand the patterns. Focus on recognizing problem types and applying standard techniques. This repository organizes problems by patterns to help you develop this crucial skill. Ready to start? Visit the repository and begin your coding interview preparation journey today! ","date":"20-Dec-2023","heading":"Start Your Journey Today","objectID":"\/posts\/data-structures-and-algorithms-for-coding-interview\/:9:0","tags":["algorithms","leetcode","python"],"title":"Data Structures and Algorithms for Coding Interview \u2014 Complete Python Resource","uri":"\/posts\/data-structures-and-algorithms-for-coding-interview\/#conclusion"},{"categories":["projects"],"collections":null,"content":"Free online Pomodoro Timer with built-in lo-fi music player (2000+ songs), All India Radio, BBC Radio, SomaFM, screen wake lock, customizable alarms, browser notifications, and dark aesthetic theme. No ads, no login, no tracking.","date":"20-Jul-2023","objectID":"\/posts\/pomodoro\/","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/"},{"categories":["projects"],"collections":null,"content":"Stay focused. Study better. No distractions. ","date":"20-Jul-2023","heading":"","objectID":"\/posts\/pomodoro\/:0:0","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#"},{"categories":["projects"],"collections":null,"content":"What is This? A free, single-page Pomodoro Timer packed with everything you need for deep focus sessions \u2014 built-in music, live radio, screen wake lock, and smart alarms. No account, no ads, no tracking. Just open and start working. Tip Try it now \u2014 the full timer is embedded below, or open it directly at \/pomodoro ","date":"20-Jul-2023","heading":"What is This?","objectID":"\/posts\/pomodoro\/:1:0","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#introduction"},{"categories":["projects"],"collections":null,"content":"Live Demo ","date":"20-Jul-2023","heading":"Live Demo","objectID":"\/posts\/pomodoro\/:2:0","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#live-demo"},{"categories":["projects"],"collections":null,"content":"Features ","date":"20-Jul-2023","heading":"Features","objectID":"\/posts\/pomodoro\/:3:0","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#features"},{"categories":["projects"],"collections":null,"content":"Music & Radio (Built-in) Source What You Get Lofi Auto Mix 2000+ lo-fi songs from YouTube, auto-shuffled SomaFM Groove Salad (ambient\/chill electronic) Bloomberg Radio WBBR 1130 NY, Business\/Markets All India Radio Vividh Bharati, Raagam, FM Gold, FM Rainbow, AIR News BBC Radio Radio 1, 2, 3, 4, World Service, Asian Network Custom URL Paste any YouTube or SoundCloud link ","date":"20-Jul-2023","heading":"Music & Radio (Built-in)","objectID":"\/posts\/pomodoro\/:3:1","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#music--radio-built-in"},{"categories":["projects"],"collections":null,"content":"Timer & Productivity Preset durations: 5, 15, 30, 45, 60, 90, 120 minutes Custom time: Set any duration you want Session counter: Tracks completed pomodoros at the top Browser notifications: Get notified even when the tab is in the background Screen wake lock: Keeps your display on during sessions (uses NoSleep.js) Separate wake lock timer: 1-24 hours independent of pomodoro ","date":"20-Jul-2023","heading":"Timer & Productivity","objectID":"\/posts\/pomodoro\/:3:2","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#timer--productivity"},{"categories":["projects"],"collections":null,"content":"Alarms Beep 15s \u2014 Short repeating beep, auto-dismisses Beep 30s \u2014 Increasing frequency beep (default) “The Time Is Now” \u2014 John Cena’s entrance theme (88 seconds) Volume amplification \u2014 1x to 7x via Web Audio API Smart dismiss \u2014 Any keypress, click, tap, or tab switch dismisses the alarm ","date":"20-Jul-2023","heading":"Alarms","objectID":"\/posts\/pomodoro\/:3:3","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#alarms"},{"categories":["projects"],"collections":null,"content":"Keyboard Shortcuts Key Action Space Start \/ Pause R Reset timer Any key Dismiss alarm ","date":"20-Jul-2023","heading":"Keyboard Shortcuts","objectID":"\/posts\/pomodoro\/:3:4","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#keyboard-shortcuts"},{"categories":["projects"],"collections":null,"content":"Why I Built This Every Pomodoro timer I found online had one or more of these problems: Requires signup \u2014 just to set a 25-minute timer Loaded with ads \u2014 tracking pixels, cookie banners, pop-ups No music integration \u2014 you need a separate tab for lo-fi beats Screen turns off \u2014 you lose sight of the countdown during long sessions Ugly UI \u2014 harsh white themes that hurt eyes during late-night study This timer solves all of them in a single HTML file. No build tools, no framework, no backend. Just open the page and focus. ","date":"20-Jul-2023","heading":"Why I Built This","objectID":"\/posts\/pomodoro\/:4:0","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#why"},{"categories":["projects"],"collections":null,"content":"How It Works ","date":"20-Jul-2023","heading":"How It Works","objectID":"\/posts\/pomodoro\/:5:0","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#how-it-works"},{"categories":["projects"],"collections":null,"content":"The Pomodoro Technique Developed by Francesco Cirillo in the late 1980s: Choose a task to work on Set the timer (traditionally 25 minutes) Work with full focus until the alarm rings Take a short break (5 minutes) Repeat \u2014 after 4 pomodoros, take a longer 15-30 minute break Info Research-backed intervals: 25-30 minutes is optimal for most people. For deep work like coding or writing, try 45-60 minutes. Starting out? Try 15 minutes and build up. ","date":"20-Jul-2023","heading":"The Pomodoro Technique","objectID":"\/posts\/pomodoro\/:5:1","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#the-pomodoro-technique"},{"categories":["projects"],"collections":null,"content":"Technical Architecture The entire app is a single HTML file (3100+ lines) with zero external dependencies: Timer logic: Pure JavaScript setInterval with drift correction Music: YouTube IFrame API + HLS.js for radio streams Wake lock: NoSleep.js (simulates video playback to keep screen on) Alarms: Web Audio API with base64-embedded audio (works offline) Storage: localStorage only \u2014 nothing leaves your device Radio fallback: Self-healing streams via Radio Browser API with cached URLs ","date":"20-Jul-2023","heading":"Technical Architecture","objectID":"\/posts\/pomodoro\/:5:2","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#technical-architecture"},{"categories":["projects"],"collections":null,"content":"Embed on Your Website Want to add this timer to your own site? Just copy this iframe: ","date":"20-Jul-2023","heading":"Embed on Your Website","objectID":"\/posts\/pomodoro\/:6:0","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#embed"},{"categories":["projects"],"collections":null,"content":"Compact Version (Timer Only) ","date":"20-Jul-2023","heading":"Compact Version (Timer Only)","objectID":"\/posts\/pomodoro\/:6:1","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#compact-version-timer-only"},{"categories":["projects"],"collections":null,"content":"Privacy & Performance Aspect Detail Tracking Zero. No analytics, no cookies, no fingerprinting Data storage localStorage only (stays on your device) Ads None. Ever. Account Not needed Load time < 1 second (single file, no external CSS\/JS) Offline Timer + alarms work offline; music needs internet GDPR\/CCPA Fully compliant (no data collected) ","date":"20-Jul-2023","heading":"Privacy & Performance","objectID":"\/posts\/pomodoro\/:7:0","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#privacy"},{"categories":["projects"],"collections":null,"content":"Browser & Device Support \u2705 Chrome \/ Edge (Chromium) \u2014 v90+ \u2705 Firefox \u2014 v88+ \u2705 Safari \u2014 v14+ \u2705 Opera \u2014 v76+ \u2705 iOS Safari (iPhone \/ iPad) \u2705 Chrome Mobile (Android) \u2705 Samsung Internet The timer is fully responsive \u2014 works on desktop monitors, laptops, tablets, and phones. Touch-friendly controls, power button alarm dismiss via Media Session API. ","date":"20-Jul-2023","heading":"Browser & Device Support","objectID":"\/posts\/pomodoro\/:8:0","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#compatibility"},{"categories":["projects"],"collections":null,"content":"Perfect For Students: Exam prep, homework, thesis writing Developers: Deep coding sessions, debugging marathons Writers: Drafting, editing, research blocks Remote workers: Timeboxing tasks, meeting prep Anyone with ADHD: External structure for focus sessions ","date":"20-Jul-2023","heading":"Perfect For","objectID":"\/posts\/pomodoro\/:9:0","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#use-cases"},{"categories":["projects"],"collections":null,"content":"Start Using It Note No installation needed Open \/pomodoro in any browser. Bookmark it. That’s it. Pick a duration (or use the default 30 minutes) Choose music: Auto Mix, radio, or paste your own URL Enable wake lock if you want the screen to stay on Press Start (or hit Space) Focus until the alarm rings Take a break. Repeat. ","date":"20-Jul-2023","heading":"Start Using It","objectID":"\/posts\/pomodoro\/:10:0","tags":["productivity","web-development","javascript"],"title":"Pomodoro Timer \u2014 Free Focus Timer with Lo-fi Music, Radio & Screen Wake Lock","uri":"\/posts\/pomodoro\/#start"},{"categories":["projects"],"collections":null,"content":"PyShooter is a complete 2D platform shooter game built with Python and Pygame. Run, jump, and shoot through challenging levels with smooth animations, engaging gameplay mechanics, and professional sound design. Features enemy AI, collision detection, health system, and multiple weapons. Download as standalone .exe or run from source code.","date":"27-Apr-2023","objectID":"\/posts\/pyshooter\/","tags":["python","pygame","game-development"],"title":"PyShooter \u2014 2D Side-Scrolling Shooter Game in Python","uri":"\/posts\/pyshooter\/"},{"categories":["projects"],"collections":null,"content":" PyShooter is a two-dimensional shooter game in which the player runs horizontally and fires at enemies. Repository: https:\/\/github.com\/SamirPaulb\/PyShooter ","date":"27-Apr-2023","heading":"","objectID":"\/posts\/pyshooter\/:0:0","tags":["python","pygame","game-development"],"title":"PyShooter \u2014 2D Side-Scrolling Shooter Game in Python","uri":"\/posts\/pyshooter\/#"},{"categories":["projects"],"collections":null,"content":" PyShooter is a two-dimensional shooter game in which the player runs horizontally and fires at enemies. Repository: https:\/\/github.com\/SamirPaulb\/PyShooter ","date":"27-Apr-2023","heading":"Repository:","objectID":"\/posts\/pyshooter\/:0:0","tags":["python","pygame","game-development"],"title":"PyShooter \u2014 2D Side-Scrolling Shooter Game in Python","uri":"\/posts\/pyshooter\/#repository-httpsgithubcomsamirpaulbpyshooter"},{"categories":["projects"],"collections":null,"content":"Demo: Your browser does not support the video tag. Vimeo: https:\/\/vimeo.com\/809728956 ","date":"27-Apr-2023","heading":"Demo:","objectID":"\/posts\/pyshooter\/:1:0","tags":["python","pygame","game-development"],"title":"PyShooter \u2014 2D Side-Scrolling Shooter Game in Python","uri":"\/posts\/pyshooter\/#demo"},{"categories":["projects"],"collections":null,"content":"How To Play PyShooter ","date":"27-Apr-2023","heading":"How To Play PyShooter","objectID":"\/posts\/pyshooter\/:2:0","tags":["python","pygame","game-development"],"title":"PyShooter \u2014 2D Side-Scrolling Shooter Game in Python","uri":"\/posts\/pyshooter\/#how-to-play-pyshooter"},{"categories":["projects"],"collections":null,"content":"Option 1: Play Without Installation (Standalone Executable) For users without Python or Pygame installed, a standalone executable is available: Download the latest release from the repository Extract the game files to a folder Double-click the PyShooter.exe file to launch the game Important: Keep the executable in the same directory as the sounds\/, images\/, and fonts\/ folders for proper asset loading. ","date":"27-Apr-2023","heading":"Option 1: Play Without Installation (Standalone Executable)","objectID":"\/posts\/pyshooter\/:2:1","tags":["python","pygame","game-development"],"title":"PyShooter \u2014 2D Side-Scrolling Shooter Game in Python","uri":"\/posts\/pyshooter\/#option-1-play-without-installation-standalone-executable"},{"categories":["projects"],"collections":null,"content":"Option 2: Run From Source Code For developers or users with Python installed: Prerequisites: Python 3.7 or higher Pygame library Installation Steps: # Install Pygame using pip pip install pygame # Clone or download the repository git clone https:\/\/github.com\/SamirPaulb\/PyShooter.git cd PyShooter # Run the game python main.py ","date":"27-Apr-2023","heading":"Option 2: Run From Source Code","objectID":"\/posts\/pyshooter\/:2:2","tags":["python","pygame","game-development"],"title":"PyShooter \u2014 2D Side-Scrolling Shooter Game in Python","uri":"\/posts\/pyshooter\/#option-2-run-from-source-code"},{"categories":["projects"],"collections":null,"content":"Game Controls Arrow Keys: Move left\/right, jump Space: Shoot weapon ESC: Pause game R: Restart level Credits for assets used: https:\/\/erayzesen.itch.io\/pixel-platformer https:\/\/secrethideout.itch.io\/team-wars-platformer-battle https:\/\/soundimage.org\/fantasywonder https:\/\/gushh.net\/blog\/free-game-sprites-explosion-3 https:\/\/mtk.itch.io\/grenades-16x16 PyGame Beginner Tutorial by youtube.com\/@CodingWithRuss ","date":"27-Apr-2023","heading":"Game Controls","objectID":"\/posts\/pyshooter\/:2:3","tags":["python","pygame","game-development"],"title":"PyShooter \u2014 2D Side-Scrolling Shooter Game in Python","uri":"\/posts\/pyshooter\/#game-controls"},{"categories":["projects"],"collections":null,"content":"Text File Compressor & Decompressor is a sophisticated web application implementing Huffman Coding algorithm for lossless data compression. Compress .txt files by 35-40%, encode for security, and decode back to original format. Built with pure JavaScript, HTML5, and CSS3 for client-side processing. Features include compression ratio display, automatic downloads, and comprehensive user guidance.","date":"12-Mar-2023","objectID":"\/posts\/text-file-compressor-de-compressor-web-app\/","tags":["huffman-coding","javascript","algorithms"],"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","uri":"\/posts\/text-file-compressor-de-compressor-web-app\/"},{"categories":["projects"],"collections":null,"content":" This webapp uses Huffman Coding for Text Compression and De-compression. Made with JavaScript, HTML5 and CSS3. Live Demo: samirpaulb.github.io\/txt-compressor Repository: github.com\/SamirPaulb\/txt-compressor ","date":"12-Mar-2023","heading":"","objectID":"\/posts\/text-file-compressor-de-compressor-web-app\/:0:0","tags":["huffman-coding","javascript","algorithms"],"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","uri":"\/posts\/text-file-compressor-de-compressor-web-app\/#"},{"categories":["projects"],"collections":null,"content":"About this application: An online text(.txt) file compressor, decompressor which uses Huffman Algorithm to encode\/compress files by 35% and decode them back to the original size. This tool assigns a variable-length code to the characters of the uploaded file based on the frequency of occurrence. Then converts characters to that special code which takes less size than the original ASCII codes. Huffman code forms a binary tree assigning the most frequent characters with the smallest codes and longer codes for the least frequent characters. A Huffman code is a tree, built bottom up, starting with the list of different characters appearing in a text and their frequency. With this lossless data compression method, this tool can compress the file size by 35 to 40%. As file size gets reduced and original characters get changed to special characters so this encoding also improves security by encrypting the file during file sharing. With the decoding feature, the user can decode the encoded file and get back the original file of the previous size. I have used JavaScript to implement the algorithms so that browser can compile the code and HTML, CSS to make the website responsive. Additional instructions and warnings are provided if steps are not followed correctly. An Info page is added to give more information about tecnique of Lossless Data Compression with Huffman coding. ","date":"12-Mar-2023","heading":"About this application:","objectID":"\/posts\/text-file-compressor-de-compressor-web-app\/:1:0","tags":["huffman-coding","javascript","algorithms"],"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","uri":"\/posts\/text-file-compressor-de-compressor-web-app\/#about-this-application"},{"categories":["projects"],"collections":null,"content":"Video Demo: Your browser does not support the video tag. ","date":"12-Mar-2023","heading":"Video Demo:","objectID":"\/posts\/text-file-compressor-de-compressor-web-app\/:2:0","tags":["huffman-coding","javascript","algorithms"],"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","uri":"\/posts\/text-file-compressor-de-compressor-web-app\/#video-demo"},{"categories":["projects"],"collections":null,"content":"Application Screenshots ","date":"12-Mar-2023","heading":"Application Screenshots","objectID":"\/posts\/text-file-compressor-de-compressor-web-app\/:3:0","tags":["huffman-coding","javascript","algorithms"],"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","uri":"\/posts\/text-file-compressor-de-compressor-web-app\/#application-screenshots"},{"categories":["projects"],"collections":null,"content":"Landing Page Huffman Coding Text Compressor - Main Interface ","date":"12-Mar-2023","heading":"Landing Page","objectID":"\/posts\/text-file-compressor-de-compressor-web-app\/:3:1","tags":["huffman-coding","javascript","algorithms"],"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","uri":"\/posts\/text-file-compressor-de-compressor-web-app\/#landing-page"},{"categories":["projects"],"collections":null,"content":"Step 1: Upload Text File Step 1 - Upload Your Text File for Compression ","date":"12-Mar-2023","heading":"Step 1: Upload Text File","objectID":"\/posts\/text-file-compressor-de-compressor-web-app\/:3:2","tags":["huffman-coding","javascript","algorithms"],"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","uri":"\/posts\/text-file-compressor-de-compressor-web-app\/#step-1-upload-text-file"},{"categories":["projects"],"collections":null,"content":"Step 2: Select Action (Compress \/ Decompress) Step 2 - Choose Compression or Decompression Action ","date":"12-Mar-2023","heading":"Step 2: Select Action (Compress \/ Decompress)","objectID":"\/posts\/text-file-compressor-de-compressor-web-app\/:3:3","tags":["huffman-coding","javascript","algorithms"],"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","uri":"\/posts\/text-file-compressor-de-compressor-web-app\/#step-2-select-action-compress--decompress"},{"categories":["projects"],"collections":null,"content":"Step 3: Download Processed File File gets downloaded automatically when selected process is complete. ","date":"12-Mar-2023","heading":"Step 3: Download Processed File","objectID":"\/posts\/text-file-compressor-de-compressor-web-app\/:3:4","tags":["huffman-coding","javascript","algorithms"],"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","uri":"\/posts\/text-file-compressor-de-compressor-web-app\/#step-3-download-processed-file"},{"categories":["projects"],"collections":null,"content":"Compression Result Display Compression Complete - View Compression Ratio and Download ","date":"12-Mar-2023","heading":"Compression Result Display","objectID":"\/posts\/text-file-compressor-de-compressor-web-app\/:3:5","tags":["huffman-coding","javascript","algorithms"],"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","uri":"\/posts\/text-file-compressor-de-compressor-web-app\/#compression-result-display"},{"categories":["projects"],"collections":null,"content":"Decompression Result Display Decompression Complete - Original File Restored ","date":"12-Mar-2023","heading":"Decompression Result Display","objectID":"\/posts\/text-file-compressor-de-compressor-web-app\/:3:6","tags":["huffman-coding","javascript","algorithms"],"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","uri":"\/posts\/text-file-compressor-de-compressor-web-app\/#decompression-result-display"},{"categories":["projects"],"collections":null,"content":"Error Handling and User Guidance ","date":"12-Mar-2023","heading":"Error Handling and User Guidance","objectID":"\/posts\/text-file-compressor-de-compressor-web-app\/:4:0","tags":["huffman-coding","javascript","algorithms"],"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","uri":"\/posts\/text-file-compressor-de-compressor-web-app\/#error-handling-and-user-guidance"},{"categories":["projects"],"collections":null,"content":"No File Selected Warning Error - Please Upload a Text File First ","date":"12-Mar-2023","heading":"No File Selected Warning","objectID":"\/posts\/text-file-compressor-de-compressor-web-app\/:4:1","tags":["huffman-coding","javascript","algorithms"],"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","uri":"\/posts\/text-file-compressor-de-compressor-web-app\/#no-file-selected-warning"},{"categories":["projects"],"collections":null,"content":"File Too Small Warning Warning - File Size Too Small for Effective Compression ","date":"12-Mar-2023","heading":"File Too Small Warning","objectID":"\/posts\/text-file-compressor-de-compressor-web-app\/:4:2","tags":["huffman-coding","javascript","algorithms"],"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","uri":"\/posts\/text-file-compressor-de-compressor-web-app\/#file-too-small-warning"},{"categories":["projects"],"collections":null,"content":"Understanding Huffman Coding ","date":"12-Mar-2023","heading":"Understanding Huffman Coding","objectID":"\/posts\/text-file-compressor-de-compressor-web-app\/:5:0","tags":["huffman-coding","javascript","algorithms"],"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","uri":"\/posts\/text-file-compressor-de-compressor-web-app\/#understanding-huffman-coding"},{"categories":["projects"],"collections":null,"content":"Information Page - Lossless Data Compression Technique Learn About Huffman Coding - Lossless Data Compression Explained ","date":"12-Mar-2023","heading":"Information Page - Lossless Data Compression Technique","objectID":"\/posts\/text-file-compressor-de-compressor-web-app\/:5:1","tags":["huffman-coding","javascript","algorithms"],"title":"Text File Compressor & Decompressor \u2014 Huffman Coding Web Application","uri":"\/posts\/text-file-compressor-de-compressor-web-app\/#information-page---lossless-data-compression-technique"},{"categories":["certifications"],"collections":null,"content":"Browse my verified professional certifications and online course achievements covering Computer Science fundamentals, Data Structures and Algorithms, Machine Learning, Artificial Intelligence, Data Science, Python Programming, SQL, IoT, Networking, and more from top platforms like Coursera, Stanford, Udemy, DataCamp, edX, Cisco, Microsoft, and LinkedIn Learning.","date":"20-Dec-2022","objectID":"\/posts\/some-of-my-online-courses-certificates\/","tags":["certifications","online-courses","machine-learning"],"title":"Professional Certifications and Online Course Achievements","uri":"\/posts\/some-of-my-online-courses-certificates\/"},{"categories":["certifications"],"collections":null,"content":"Certification Categories Algorithms & Data Structures: Advanced algorithms, complexity analysis, divide and conquer strategies Machine Learning & AI: Deep learning, neural networks, AI fundamentals, applied ML Data Science: Data analytics, Python for data science, SQL, data visualization Software Development: Programming languages (C++, Python), databases, software design Networking & IoT: Computer networks, Internet of Things, embedded systems Tools & Frameworks: Git, GitHub, SQL Server, Photoshop, Linux ","date":"20-Dec-2022","heading":"Certification Categories","objectID":"\/posts\/some-of-my-online-courses-certificates\/:0:1","tags":["certifications","online-courses","machine-learning"],"title":"Professional Certifications and Online Course Achievements","uri":"\/posts\/some-of-my-online-courses-certificates\/#certification-categories"},{"categories":["certifications"],"collections":null,"content":"Verified Certifications Coursera: Advanced Algorithms and Complexity - GLMMPEBLCU9Q DataCamp: AI Fundamentals NVIDIA: Getting Started with AI on Jetson Nano Coursera: Algorithmic Toolbox - MA4KKNFEU7MB Udemy: C++ Programming - UC-c58957c7-3ae3-4f83-b4b1-2f69b2df48fa Agoda: Codegoda 2022 Udemy: Data Analytics Real-World Projects in Python - UC-75332da1-4725-4f60-80ab-41f640046e3e DateCamp: Data Science for Everyone Coursera: Python Data Structures - 394N5RVDD2EU LinkedIn Learning: Database Foundations Database Management StanfordOnline & edX: Databases: Relational Databases and SQL - SOE-YDATABASES0001 Coursera: Divide and Conquer, Sorting and Searching, and Randomized Algorithms - V58E5TWMYGTV Cisco Networking Academy: Introduction to IoT and Digital Transformation Microsoft Learn Student: Introduction to Opensource, Git and Github - 2102001215000100 DataCamp: Introduction to SQL LinkedIn Learning: IoT Foundations: Operating Systems Fundamentals (2018) LinkedIn learning: Linux Shells and Processes Coursera: Machine Learning - VJHQ4C2F8LDA Udemy: Microsoft SQL Server 2019: For Beginners - UC-93630ce4-c301-401f-a935-a38155ba79aa LinkedIn Learning: Networking Foundations: Local Area Networks (LANs) HackerRank: Python (Basic) Skills Certification Test Cognitive Class: Deep Learning Fundamentals course HackerRank: Problem Solving (Basic) Skills Certification Test Udemy: Adobe Photoshop CC Fundamentals and Essentials Training - UC-c9524150-649d-4ed2-bdcc-9fcfd284297a The content and images of this post are protected by intellectual property rights and safeguarded by the protection service offered by dmca.com\/r\/4d2qkez. ","date":"20-Dec-2022","heading":"Verified Certifications","objectID":"\/posts\/some-of-my-online-courses-certificates\/:1:0","tags":["certifications","online-courses","machine-learning"],"title":"Professional Certifications and Online Course Achievements","uri":"\/posts\/some-of-my-online-courses-certificates\/#verified-certifications"},{"categories":["projects"],"collections":null,"content":"Online PDF Compression Tool is a web-based application that compresses PDF files efficiently using advanced algorithms. Built with Python Flask and PDFNetPython library, it reduces file sizes significantly while preserving document quality and automatically deletes files after 1 hour for security.","date":"20-Jan-2021","objectID":"\/posts\/online-pdf-compression-tool\/","tags":["python","flask","pdf-tools"],"title":"Online PDF Compression Tool \u2014 Reduce PDF File Size Instantly","uri":"\/posts\/online-pdf-compression-tool\/"},{"categories":["projects"],"collections":null,"content":"About The Project The Online PDF Compression Tool is a powerful web-based application designed to reduce PDF file sizes efficiently while maintaining document quality. Built with modern web technologies and advanced PDF processing algorithms, this tool provides a fast, secure, and user-friendly solution for PDF optimization. ","date":"20-Jan-2021","heading":"About The Project","objectID":"\/posts\/online-pdf-compression-tool\/:1:0","tags":["python","flask","pdf-tools"],"title":"Online PDF Compression Tool \u2014 Reduce PDF File Size Instantly","uri":"\/posts\/online-pdf-compression-tool\/#about-the-project"},{"categories":["projects"],"collections":null,"content":"Key Features High Compression Ratio: Reduces PDF file sizes by up to 70% while preserving visual quality Secure Processing: Files are automatically deleted from the server after 1 hour for maximum privacy Fast Processing: Optimized backend processing with PDFNetPython ensures quick compression Cross-Platform: Works seamlessly on all devices with a modern web browser No Installation Required: Fully web-based solution accessible from anywhere ","date":"20-Jan-2021","heading":"Key Features","objectID":"\/posts\/online-pdf-compression-tool\/:1:1","tags":["python","flask","pdf-tools"],"title":"Online PDF Compression Tool \u2014 Reduce PDF File Size Instantly","uri":"\/posts\/online-pdf-compression-tool\/#key-features"},{"categories":["projects"],"collections":null,"content":"Technologies Used This project leverages a robust tech stack for optimal performance and security: Backend: Python 3, Flask (web framework), PDFNetPython (PDF processing) Frontend: HTML5, CSS3, JavaScript (ES6+) Additional Tools: C, Shell scripting, Git version control Deployment: Replit platform, Nix environment configuration ","date":"20-Jan-2021","heading":"Technologies Used","objectID":"\/posts\/online-pdf-compression-tool\/:1:2","tags":["python","flask","pdf-tools"],"title":"Online PDF Compression Tool \u2014 Reduce PDF File Size Instantly","uri":"\/posts\/online-pdf-compression-tool\/#technologies-used"},{"categories":["projects"],"collections":null,"content":"Live Demo Experience the PDF compression tool in action: \ud83d\ude80 Launch Live Demo ","date":"20-Jan-2021","heading":"Live Demo","objectID":"\/posts\/online-pdf-compression-tool\/:1:3","tags":["python","flask","pdf-tools"],"title":"Online PDF Compression Tool \u2014 Reduce PDF File Size Instantly","uri":"\/posts\/online-pdf-compression-tool\/#live-demo"},{"categories":["projects"],"collections":null,"content":"Video Demo: Your browser does not support the video tag. ","date":"20-Jan-2021","heading":"Video Demo:","objectID":"\/posts\/online-pdf-compression-tool\/:2:0","tags":["python","flask","pdf-tools"],"title":"Online PDF Compression Tool \u2014 Reduce PDF File Size Instantly","uri":"\/posts\/online-pdf-compression-tool\/#video-demo"},{"categories":["projects"],"collections":null,"content":"Application Interface Landing Page PDF Compressor Landing Page - Upload and Compress PDF Files Instantly ","date":"20-Jan-2021","heading":"Application Interface","objectID":"\/posts\/online-pdf-compression-tool\/:2:1","tags":["python","flask","pdf-tools"],"title":"Online PDF Compression Tool \u2014 Reduce PDF File Size Instantly","uri":"\/posts\/online-pdf-compression-tool\/#application-interface"},{"categories":["projects"],"collections":null,"content":"Technical Implementation ","date":"20-Jan-2021","heading":"Technical Implementation","objectID":"\/posts\/online-pdf-compression-tool\/:3:0","tags":["python","flask","pdf-tools"],"title":"Online PDF Compression Tool \u2014 Reduce PDF File Size Instantly","uri":"\/posts\/online-pdf-compression-tool\/#technical-implementation"},{"categories":["projects"],"collections":null,"content":"Flask File Upload System The application uses Flask’s robust file handling capabilities to manage PDF uploads securely. The HTML form’s enctype property is set to "multipart\/form-data" to enable file transmission. The server-side Flask application extracts files from the request.files[] object and processes them securely. Key Configuration Parameters: app.config['UPLOAD_FOLDER']: Defines the temporary storage location for uploaded files app.config['MAX_CONTENT_LENGTH']: Sets the maximum allowed file size in bytes Secure filename generation using werkzeug.utils.secure_filename() to prevent path traversal attacks Flask File Upload Implementation: from flask import Flask, render_template, request, jsonify from werkzeug.utils import secure_filename import os app = Flask(__name__) # Configuration app.config['UPLOAD_FOLDER'] = 'uploads\/temp' app.config['MAX_CONTENT_LENGTH'] = 16 * 1024 * 1024 # 16 MB max file size app.config['ALLOWED_EXTENSIONS'] = {'pdf'} def allowed_file(filename): \"\"\"Check if uploaded file has a valid PDF extension\"\"\" return '.' in filename and \\ filename.rsplit('.', 1)[1].lower() in app.config['ALLOWED_EXTENSIONS'] @app.route('\/upload') def upload_file(): \"\"\"Render the file upload interface\"\"\" return render_template('upload.html') @app.route('\/uploader', methods=['GET', 'POST']) def upload_handler(): \"\"\"Handle file upload and initiate compression\"\"\" if request.method == 'POST': # Check if file is present in request if 'file' not in request.files: return jsonify({'error': 'No file provided'}), 400 f = request.files['file'] # Validate file if f.filename == '': return jsonify({'error': 'No file selected'}), 400 if not allowed_file(f.filename): return jsonify({'error': 'Only PDF files are allowed'}), 400 # Save file securely filename = secure_filename(f.filename) filepath = os.path.join(app.config['UPLOAD_FOLDER'], filename) f.save(filepath) return jsonify({ 'message': 'File uploaded successfully', 'filename': filename }), 200 if __name__ == '__main__': # Create upload directory if it doesn't exist os.makedirs(app.config['UPLOAD_FOLDER'], exist_ok=True) app.run(debug=True) ","date":"20-Jan-2021","heading":"Flask File Upload System","objectID":"\/posts\/online-pdf-compression-tool\/:3:1","tags":["python","flask","pdf-tools"],"title":"Online PDF Compression Tool \u2014 Reduce PDF File Size Instantly","uri":"\/posts\/online-pdf-compression-tool\/#flask-file-upload-system"},{"categories":["projects"],"collections":null,"content":"PDF Compression Backend The compression engine utilizes the powerful PDFNetPython library, which provides advanced PDF optimization capabilities. The compression process analyzes the PDF structure, removes redundancies, compresses streams, and optimizes embedded resources. Compression Process Steps: Initialize PDFNet library with license key Load the input PDF document Verify document security handlers Apply Optimizer to reduce file size Save linearized PDF for optimal web viewing Validate and return compression results import os import sys from PDFNetPython3.PDFNetPython import PDFDoc, Optimizer, SDFDoc, PDFNet def compress_file(input_file: str, output_file: str = None) -> bool: \"\"\" Compress a PDF file using PDFNetPython's advanced optimization. Args: input_file: Path to the PDF file to compress output_file: Path where compressed PDF will be saved. If None, overwrites the input file. Returns: bool: True if compression successful, False otherwise \"\"\" if not output_file: output_file = input_file try: # Initialize PDFNet library PDFNet.Initialize() # Open the PDF document doc = PDFDoc(input_file) # Initialize security handlers (required for protected PDFs) doc.InitSecurityHandler() # Apply optimization to reduce file size # This includes: stream compression, font subsetting, and object removal Optimizer.Optimize(doc) # Save optimized PDF with linearization for web viewing # Linearized PDFs display faster in web browsers doc.Save(output_file, SDFDoc.e_linearized) # Close document and release resources doc.Close() return True except Exception as e: print(f\"Error during PDF compression: {str(e)}\") try: doc.Close() except: pass return False if __name__ == \"__main__\": # Command line usage: python compress_pdf.py input.pdf output.pdf if len(sys.argv) < 3: print(\"Usage: python compress_pdf.py \") sys.exit(1) input_file = sys.argv[1] output_file = sys.argv[2] if not os.path.exists(input_file): print(f\"Error: Input file '{input_file}' not found\") sys.exit(1) success = compress_file(input_file, output_file) if success: original_size = os.path.getsize(input_file) compressed_size = os.path.getsize(output_file) ratio = ((original_size - compressed_size) \/ original_size) * 100 print(f\"\u2713 Compression successful!\") print(f\" Original size: {original_size \/ 1024:.2f} KB\") print(f\" Compressed size: {compressed_size \/ 1024:.2f} KB\") print(f\" Compression ratio: {ratio:.2f}%\") else: print(\"\u2717 Compression failed\") sys.exit(1) ","date":"20-Jan-2021","heading":"PDF Compression Backend","objectID":"\/posts\/online-pdf-compression-tool\/:3:2","tags":["python","flask","pdf-tools"],"title":"Online PDF Compression Tool \u2014 Reduce PDF File Size Instantly","uri":"\/posts\/online-pdf-compression-tool\/#pdf-compression-backend"},{"categories":["projects"],"collections":null,"content":"Client-Side File Download After successful compression, the client-side JavaScript handles automatic file download using modern browser APIs. The implementation uses the Fetch API to retrieve the compressed file and creates a temporary download link for seamless user experience. Download Process: Receive compressed file metadata from server Fetch compressed PDF blob from server Create temporary object URL for download Trigger automatic download Clean up temporary resources \/** * Download compressed PDF file automatically * @param {string} filename - Name of the compressed file to download *\/ function downloadFile(filename) { \/\/ Validate response and filename if (response !== null && response.filename) { const fname = response.filename; const url = `static\/resource\/${fname}`; console.log(`Downloading file from: ${url}`); \/\/ Fetch compressed PDF as blob fetch(url) .then(response => { if (!response.ok) { throw new Error(`HTTP error! status: ${response.status}`); } return response.blob(); }) .then(blob => { \/\/ Create temporary download link const link = document.createElement(\"a\"); link.href = URL.createObjectURL(blob); link.download = fname; \/\/ Append to DOM (required for Firefox) document.body.appendChild(link); \/\/ Trigger download link.click(); \/\/ Clean up: remove link and revoke object URL document.body.removeChild(link); URL.revokeObjectURL(link.href); console.log(\"\u2713 Download initiated successfully\"); }) .catch(error => { console.error(\"\u2717 Download failed:\", error); alert(\"Failed to download file. Please try again.\"); }); } else { console.error(\"\u2717 Invalid response or filename\"); alert(\"File not ready for download. Please wait for compression to complete.\"); } } \/\/ Alternative: Display download progress function downloadFileWithProgress(filename) { if (response !== null && response.filename) { const fname = response.filename; const url = `static\/resource\/${fname}`; fetch(url) .then(response => { const contentLength = response.headers.get('content-length'); const total = parseInt(contentLength, 10); let loaded = 0; return new Response( new ReadableStream({ start(controller) { const reader = response.body.getReader(); read(); function read() { reader.read().then(({done, value}) => { if (done) { controller.close(); return; } loaded += value.byteLength; const progress = (loaded \/ total) * 100; console.log(`Download progress: ${progress.toFixed(2)}%`); controller.enqueue(value); read(); }).catch(error => { console.error(error); controller.error(error); }); } } }) ); }) .then(response => response.blob()) .then(blob => { const link = document.createElement(\"a\"); link.href = URL.createObjectURL(blob); link.download = fname; document.body.appendChild(link); link.click(); document.body.removeChild(link); URL.revokeObjectURL(link.href); }) .catch(error => { console.error(\"Download error:\", error); }); } } ","date":"20-Jan-2021","heading":"Client-Side File Download","objectID":"\/posts\/online-pdf-compression-tool\/:3:3","tags":["python","flask","pdf-tools"],"title":"Online PDF Compression Tool \u2014 Reduce PDF File Size Instantly","uri":"\/posts\/online-pdf-compression-tool\/#client-side-file-download"},{"categories":["projects"],"collections":null,"content":"\ud83e\udd14 How to contribute Fork this repository; Create a branch with your feature: git checkout -b my-feature; Commit your changes: git commit -m "feat: my new feature"; Push to your branch: git push origin my-feature. ","date":"20-Jan-2021","heading":"\ud83e\udd14 How to contribute","objectID":"\/posts\/online-pdf-compression-tool\/:4:0","tags":["python","flask","pdf-tools"],"title":"Online PDF Compression Tool \u2014 Reduce PDF File Size Instantly","uri":"\/posts\/online-pdf-compression-tool\/#-how-to-contribute"},{"categories":null,"collections":null,"content":"Hi, I\u2019m Samir. I design and build scalable software systems, mostly on the backend, with a strong interest in AI, automation, and sharing what I learn through code and occasional blog posts.","date":"01-Jan-0001","objectID":"\/about\/","tags":null,"title":"About Me","uri":"\/about\/"},{"categories":null,"collections":null,"content":" Hi, I\u2019m Samir \u2014 a software engineer who enjoys designing and building scalable software systems, especially on the backend, and experimenting with applied AI when it actually solves real problems. \ud83d\udcbb\ud83e\udd16 Most of what I work on lives around APIs, data, infrastructure, and automation, with a bias toward clean, maintainable designs that can grow without falling apart. \u2699\ufe0f\ud83d\udcc8 I don\u2019t update this website very often \ud83d\ude42 so for a more up-to-date picture of what I\u2019m doing, please check: LinkedIn.com\/in\/SamirPaul Some things I\u2019m generally interested in: \ud83e\uddd1\u200d\ud83d\udcbb Backend engineering, distributed systems, and building scalable, reliable services \ud83e\udd16 Applied AI and ML (RAG, models in production, and practical workflows around them) \u2699\ufe0f Automation, scripting, and developer tooling to remove repetitive work \ud83d\udcda Data structures, algorithms, and system design as the backbone of everything above If you want to say hi, collaborate, or just leave a note, you can drop me a message here: \/contact or reach out directly at Copy \u2709\ufe0f If you\u2019d like new posts delivered to your inbox, you can subscribe here: \/newsletter \ud83d\udcf0 ","date":"01-Jan-0001","heading":"","objectID":"\/about\/:0:0","tags":null,"title":"About Me","uri":"\/about\/#"},{"categories":null,"collections":null,"content":"Contact Samir for collaboration, career opportunities, freelance projects, or simply to connect. Multiple ways to reach out including LinkedIn, GitHub, Twitter, email, and calendar scheduling.","date":"01-Jan-0001","objectID":"\/contact\/","tags":null,"title":"Contact Me","uri":"\/contact\/"},{"categories":null,"collections":null,"content":"Feel free to reach out through: LinkedIn. Email: Copyor schedule a meeting on Calendly or Book a 1:1 Topmate call. ","date":"01-Jan-0001","heading":"","objectID":"\/contact\/:0:0","tags":null,"title":"Contact Me","uri":"\/contact\/#"},{"categories":null,"collections":null,"content":"Disclaimer for samirpaulb.github.io covering content accuracy, AI-generated content, code examples, copyright, fair use, and limitation of liability.","date":"01-Jan-0001","objectID":"\/disclaimer\/","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/"},{"categories":null,"collections":null,"content":"Website Purpose This website (samirpaulb.github.io \/ samir.pages.dev) is a personal learning blog and digital notebook maintained by Samir. It contains personal notes, technical articles, code examples, and curated content shared for educational purposes. ","date":"01-Jan-0001","heading":"Website Purpose","objectID":"\/disclaimer\/:1:0","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/#website-purpose"},{"categories":null,"collections":null,"content":"Content Accuracy All content is provided for informational and educational purposes only. It may contain errors, be incomplete, or become outdated. Nothing on this website constitutes professional advice of any kind, including legal, medical, financial, engineering, or tax advice. Always verify information from authoritative primary sources before making decisions or implementations based on content found here. ","date":"01-Jan-0001","heading":"Content Accuracy","objectID":"\/disclaimer\/:2:0","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/#content-accuracy"},{"categories":null,"collections":null,"content":"AI-Generated Content ","date":"01-Jan-0001","heading":"AI-Generated Content","objectID":"\/disclaimer\/:3:0","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/#ai-generated-content"},{"categories":null,"collections":null,"content":"Daily Digest The Daily Digest section displays AI-generated news summaries that are not hosted on this website. This content is fetched from a separate repository at github.com\/SamirPaulb\/daily-digest. These summaries: Are generated by AI language models and may contain factual errors Are not independently verified for accuracy Should not be relied upon as a primary news source May become outdated quickly ","date":"01-Jan-0001","heading":"Daily Digest","objectID":"\/disclaimer\/:3:1","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/#daily-digest"},{"categories":null,"collections":null,"content":"Other Content Some blog posts may be written with AI assistance. Where applicable, this is noted in the article. AI-assisted content has not been independently verified and may contain inaccuracies. ","date":"01-Jan-0001","heading":"Other Content","objectID":"\/disclaimer\/:3:2","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/#other-content"},{"categories":null,"collections":null,"content":"Code and Technical Content Code examples, configurations, and technical implementations are provided “as-is” without any warranties. They: May contain bugs, errors, or security vulnerabilities Are simplified for learning purposes Have not been tested for production environments May not follow current best practices Your responsibility: Review, understand, test, and audit any code before use. I accept no liability for damages resulting from the use of code found on this website. ","date":"01-Jan-0001","heading":"Code and Technical Content","objectID":"\/disclaimer\/:4:0","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/#code-and-technical-content"},{"categories":null,"collections":null,"content":"Copyright and Licensing ","date":"01-Jan-0001","heading":"Copyright and Licensing","objectID":"\/disclaimer\/:5:0","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/#copyright-and-licensing"},{"categories":null,"collections":null,"content":"Original Content Original articles and blog posts are licensed under Creative Commons CC BY-NC-ND 4.0 unless otherwise stated in the article footer. This means you may share the content with attribution, but may not modify it or use it commercially. ","date":"01-Jan-0001","heading":"Original Content","objectID":"\/disclaimer\/:5:1","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/#original-content"},{"categories":null,"collections":null,"content":"Educational Summaries Some content consists of summaries, notes, and commentary on publicly available materials. This constitutes fair use under 17 U.S.C. Section 107 and equivalent provisions in other jurisdictions, as the use is: Non-commercial and educational Transformative (summarized and annotated) Limited in portion used Non-competitive with the original works ","date":"01-Jan-0001","heading":"Educational Summaries","objectID":"\/disclaimer\/:5:2","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/#educational-summaries"},{"categories":null,"collections":null,"content":"Third-Party Content Content from third-party sources remains the intellectual property of their respective owners. If you believe any content infringes your copyright, please contact me. ","date":"01-Jan-0001","heading":"Third-Party Content","objectID":"\/disclaimer\/:5:3","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/#third-party-content"},{"categories":null,"collections":null,"content":"DMCA and Takedown Requests For copyright concerns, contact me via the contact form or netproxy@duck.com with: Identification of the copyrighted work URL of the allegedly infringing content Your contact information A statement of good faith belief that the use is not authorized A statement that the information is accurate, under penalty of perjury I will respond to valid requests promptly. ","date":"01-Jan-0001","heading":"DMCA and Takedown Requests","objectID":"\/disclaimer\/:5:4","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/#dmca-and-takedown-requests"},{"categories":null,"collections":null,"content":"External Links This website contains links to external websites. I have no control over and accept no responsibility for the content, privacy practices, or availability of third-party websites. ","date":"01-Jan-0001","heading":"External Links","objectID":"\/disclaimer\/:6:0","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/#external-links"},{"categories":null,"collections":null,"content":"Limitation of Liability To the maximum extent permitted by applicable law, I shall not be liable for any direct, indirect, incidental, or consequential damages arising from your use of this website or reliance on its content. This limitation does not exclude liability for fraud, gross negligence, or any other liability that cannot be excluded under applicable law. Use of this website and its content is entirely at your own risk. ","date":"01-Jan-0001","heading":"Limitation of Liability","objectID":"\/disclaimer\/:7:0","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/#limitation-of-liability"},{"categories":null,"collections":null,"content":"Governing Law This disclaimer is governed by the laws of India. Any disputes arising from the use of this website shall be subject to the jurisdiction of courts in India. ","date":"01-Jan-0001","heading":"Governing Law","objectID":"\/disclaimer\/:8:0","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/#governing-law"},{"categories":null,"collections":null,"content":"Changes This disclaimer may be updated at any time. The date below indicates the most recent revision. Last Updated: May 2026 ","date":"01-Jan-0001","heading":"Changes","objectID":"\/disclaimer\/:9:0","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/#changes"},{"categories":null,"collections":null,"content":"Contact For questions, copyright concerns, or takedown requests: Contact: Contact Form or netproxy@duck.com ","date":"01-Jan-0001","heading":"Contact","objectID":"\/disclaimer\/:10:0","tags":null,"title":"Disclaimer","uri":"\/disclaimer\/#contact"},{"categories":null,"collections":null,"content":"Follow Samir's activity across all platforms. Get updates on GitHub projects, Twitter posts, LinkedIn articles, YouTube videos, and current Spotify playlist in real-time.","date":"01-Jan-0001","objectID":"\/feed\/","tags":null,"title":"Feeds","uri":"\/feed\/"},{"categories":null,"collections":null,"content":"Website RSS Feeds ","date":"01-Jan-0001","heading":"Website","objectID":"\/feed\/:0:1","tags":null,"title":"Feeds","uri":"\/feed\/#website"},{"categories":null,"collections":null,"content":"GitHub ","date":"01-Jan-0001","heading":"GitHub","objectID":"\/feed\/:0:2","tags":null,"title":"Feeds","uri":"\/feed\/#github"},{"categories":null,"collections":null,"content":"X\/Twitter ","date":"01-Jan-0001","heading":"X\/Twitter","objectID":"\/feed\/:0:3","tags":null,"title":"Feeds","uri":"\/feed\/#xtwitter"},{"categories":null,"collections":null,"content":"LinkedIn ","date":"01-Jan-0001","heading":"LinkedIn","objectID":"\/feed\/:0:4","tags":null,"title":"Feeds","uri":"\/feed\/#linkedin"},{"categories":null,"collections":null,"content":"YouTube ","date":"01-Jan-0001","heading":"YouTube","objectID":"\/feed\/:0:5","tags":null,"title":"Feeds","uri":"\/feed\/#youtube"},{"categories":null,"collections":null,"content":"Subscribe to Samir\u2019s newsletter to receive new posts and occasional updates by email. No spam\u2014unsubscribe anytime.","date":"01-Jan-0001","objectID":"\/newsletter\/","tags":null,"title":"Newsletter","uri":"\/newsletter\/"},{"categories":null,"collections":null,"content":"Want new posts by email? Join the newsletter for concise updates from projects and everyday learnings. No spam \u2014 unsubscribe anytime. ","date":"01-Jan-0001","heading":"","objectID":"\/newsletter\/:0:0","tags":null,"title":"Newsletter","uri":"\/newsletter\/#"},{"categories":null,"collections":null,"content":"Privacy policy for samirpaulb.github.io explaining Google Analytics, Google AdSense, cookies, data collection, consent management, and your rights under GDPR, CCPA, DPDPA, and global data protection laws.","date":"01-Jan-0001","objectID":"\/privacy\/","tags":null,"title":"Privacy Policy","uri":"\/privacy\/"},{"categories":null,"collections":null,"content":"This privacy policy applies to samirpaulb.github.io, samir.pages.dev, and all its mirrors (collectively, “this website”). This is a personal blog maintained by Samir (“I”, “me”, “my”). By using this website, you acknowledge this privacy policy. If you disagree with any part, please stop using this website. ","date":"01-Jan-0001","heading":"","objectID":"\/privacy\/:0:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#"},{"categories":null,"collections":null,"content":"Information We Collect ","date":"01-Jan-0001","heading":"Information We Collect","objectID":"\/privacy\/:1:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#information-we-collect"},{"categories":null,"collections":null,"content":"Automatically Collected Data When you visit this website, the hosting infrastructure collects: IP address (used for routing and security; not stored by me) Browser type and operating system Pages visited and timestamps Referring URL (how you arrived) This data is processed by GitHub Pages and Cloudflare Pages for delivery and security. I do not have access to raw server logs. ","date":"01-Jan-0001","heading":"Automatically Collected Data","objectID":"\/privacy\/:1:1","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#automatically-collected-data"},{"categories":null,"collections":null,"content":"Analytics This website uses Google Analytics 4 (“GA4”) to understand how visitors use the site. When you grant consent, GA4 collects: Pages visited and time spent Device type, browser, and operating system Approximate geographic location (country\/city level) Referral source Scroll depth and outbound link clicks Privacy measures: IP anonymization is enabled \u2014 your full IP address is never stored by Google Analytics Data retention is set to 14 months (automatically deleted after) Google Signals is disabled \u2014 no cross-device tracking Consent Mode v2 is implemented \u2014 cookies are only set after you explicitly grant consent Ad data redaction is enabled \u2014 ad identifiers are redacted when consent is not granted If you decline consent, GA4 operates in cookieless mode \u2014 no cookies are set, and Google uses behavioral modeling to provide aggregate statistics. ","date":"01-Jan-0001","heading":"Analytics","objectID":"\/privacy\/:1:2","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#analytics"},{"categories":null,"collections":null,"content":"Advertising This website may display advertisements served by Google AdSense. When advertising is active and you have granted consent: Google and its certified ad networks may use cookies to serve ads based on your prior visits to this or other websites You can opt out of personalized advertising at Google Ad Settings You can opt out of third-party vendor cookies at aboutads.info or networkadvertising.org Without consent: No advertising cookies are set. If ads are displayed, they will be non-personalized (contextual only). ","date":"01-Jan-0001","heading":"Advertising","objectID":"\/privacy\/:1:3","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#advertising"},{"categories":null,"collections":null,"content":"View Counter A lightweight, privacy-friendly view counter powered by Cloudflare Workers tracks anonymous page view counts. It uses no cookies, stores no personal data, and cannot identify individual visitors. ","date":"01-Jan-0001","heading":"View Counter","objectID":"\/privacy\/:1:4","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#view-counter"},{"categories":null,"collections":null,"content":"Comments Comments are powered by Giscus, which uses GitHub Discussions. When you post a comment, you interact directly with GitHub under their Privacy Statement. I do not separately collect or store comment data. ","date":"01-Jan-0001","heading":"Comments","objectID":"\/privacy\/:1:5","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#comments"},{"categories":null,"collections":null,"content":"Contact Form and Newsletter When you submit the contact form or subscribe to the newsletter, the following is collected: Name and email address Message content (contact form only) IP address and timestamp (spam prevention) This data is encrypted using AES-256 and stored on Cloudflare’s infrastructure. Only I hold the decryption key. ","date":"01-Jan-0001","heading":"Contact Form and Newsletter","objectID":"\/privacy\/:1:6","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#contact-form-and-newsletter"},{"categories":null,"collections":null,"content":"Consent Management This website uses Google Consent Mode v2 with a consent banner shown to all visitors worldwide. How it works: On your first visit, a small consent popup appears Accept \u2014 analytics and advertising cookies are enabled Decline \u2014 no cookies are set; analytics runs in cookieless mode; ads (if present) are non-personalized Your choice is stored in your browser’s localStorage (not a cookie) and remembered for future visits You can change your choice at any time (see How to Opt Out below) No cookies are ever set before you make a choice. The consent popup itself uses zero cookies and makes zero network requests. ","date":"01-Jan-0001","heading":"Consent Management","objectID":"\/privacy\/:2:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#consent-management"},{"categories":null,"collections":null,"content":"Cookies ","date":"01-Jan-0001","heading":"Cookies","objectID":"\/privacy\/:3:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#cookies"},{"categories":null,"collections":null,"content":"Analytics Cookies (set only after consent) Cookie Provider Purpose Duration _ga Google Analytics Distinguishes unique visitors 14 months _ga_* Google Analytics Stores session state 14 months ","date":"01-Jan-0001","heading":"Analytics Cookies (set only after consent)","objectID":"\/privacy\/:3:1","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#analytics-cookies-set-only-after-consent"},{"categories":null,"collections":null,"content":"Advertising Cookies (set only after consent, when ads are active) Cookie Provider Purpose Duration Various Google AdSense Serves and measures ads Variable You can review all Google cookies at Google’s Cookie Policy. ","date":"01-Jan-0001","heading":"Advertising Cookies (set only after consent, when ads are active)","objectID":"\/privacy\/:3:2","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#advertising-cookies-set-only-after-consent-when-ads-are-active"},{"categories":null,"collections":null,"content":"Security Cookies (strictly necessary \u2014 no consent required) Cookie Provider Purpose Duration __cf_bm Cloudflare Bot management 30 minutes cf_clearance Cloudflare Security challenge passed Variable These are exempt from consent requirements under the ePrivacy Directive as they are strictly necessary for the website to function. ","date":"01-Jan-0001","heading":"Security Cookies (strictly necessary \u2014 no consent required)","objectID":"\/privacy\/:3:3","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#security-cookies-strictly-necessary--no-consent-required"},{"categories":null,"collections":null,"content":"How to Opt Out or Withdraw Consent ","date":"01-Jan-0001","heading":"How to Opt Out or Withdraw Consent","objectID":"\/privacy\/:4:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#how-to-opt-out-or-withdraw-consent"},{"categories":null,"collections":null,"content":"Decline via the consent popup On your first visit, click “Decline” to prevent all analytics and advertising cookies. ","date":"01-Jan-0001","heading":"Decline via the consent popup","objectID":"\/privacy\/:4:1","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#decline-via-the-consent-popup"},{"categories":null,"collections":null,"content":"Withdraw consent after accepting Open your browser’s developer tools (F12) Go to Application > Local Storage > this website’s domain Delete the site_consent entry Refresh the page \u2014 the consent popup will reappear ","date":"01-Jan-0001","heading":"Withdraw consent after accepting","objectID":"\/privacy\/:4:2","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#withdraw-consent-after-accepting"},{"categories":null,"collections":null,"content":"Additional opt-out methods Method How Google Analytics Install the GA Opt-out Add-on Personalized Ads Visit Google Ad Settings Third-party Ad Cookies Visit DAA Opt-out or NAI Opt-out Browser built-in Use Firefox, Brave, or Safari (block trackers by default) Ad blockers uBlock Origin, AdGuard, or Privacy Badger Clear cookies Delete cookies for this domain in your browser settings Disable JavaScript Prevents all analytics and ad scripts from running Opting out does not affect website functionality. All content remains fully accessible. ","date":"01-Jan-0001","heading":"Additional opt-out methods","objectID":"\/privacy\/:4:3","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#additional-opt-out-methods"},{"categories":null,"collections":null,"content":"Legal Basis for Processing Purpose Legal Basis Reference Website delivery and security Legitimate interest GDPR Art. 6(1)(f) Spam\/DDoS protection Legitimate interest GDPR Art. 6(1)(f) Analytics Your consent GDPR Art. 6(1)(a) Advertising Your consent GDPR Art. 6(1)(a) Contact form \/ newsletter Your consent (by submitting) GDPR Art. 6(1)(a) View counter Legitimate interest (anonymous) GDPR Art. 6(1)(f) ","date":"01-Jan-0001","heading":"Legal Basis for Processing","objectID":"\/privacy\/:5:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#legal-basis-for-processing"},{"categories":null,"collections":null,"content":"Third-Party Services Service Purpose Privacy Policy GitHub Pages Hosting GitHub Privacy Cloudflare Pages Hosting, CDN, security Cloudflare Privacy Google Analytics 4 Analytics Google Privacy Google AdSense Advertising Google Privacy Giscus Comments (via GitHub) GitHub Privacy Cloudflare Workers View counter Cloudflare Privacy Google processes analytics data under their Data Processing Terms. ","date":"01-Jan-0001","heading":"Third-Party Services","objectID":"\/privacy\/:6:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#third-party-services"},{"categories":null,"collections":null,"content":"International Data Transfers This website uses services based in the United States. Data transfers from the EU\/EEA\/UK to the US are protected by: EU-US Data Privacy Framework \u2014 Google, GitHub, and Cloudflare are certified (adequacy decision adopted 10 July 2023) Standard Contractual Clauses \u2014 maintained by all providers as a fallback mechanism UK Extension \u2014 the UK Extension to the EU-US DPF covers UK transfers ","date":"01-Jan-0001","heading":"International Data Transfers","objectID":"\/privacy\/:7:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#international-data-transfers"},{"categories":null,"collections":null,"content":"Data Retention Data Retention Deletion Analytics data 14 months Automatic by Google Analytics cookies 14 months from last visit Automatic expiry or manual Advertising cookies Variable (see Google Cookie Policy) Automatic expiry or manual Contact form submissions Until deletion requested On request Newsletter subscriptions Until unsubscribe Automatic on unsubscribe Cloudflare logs ~72 hours Automatic by Cloudflare ","date":"01-Jan-0001","heading":"Data Retention","objectID":"\/privacy\/:8:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#data-retention"},{"categories":null,"collections":null,"content":"Data Security All connections use HTTPS (TLS 1.2+) Form data encrypted at rest (AES-256) Cloudflare WAF and DDoS protection active Google Analytics data protected by Google’s infrastructure security Decryption keys held only by the website owner No system is 100% secure. While I take reasonable measures, I cannot guarantee absolute security. ","date":"01-Jan-0001","heading":"Data Security","objectID":"\/privacy\/:9:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#data-security"},{"categories":null,"collections":null,"content":"Your Rights ","date":"01-Jan-0001","heading":"Your Rights","objectID":"\/privacy\/:10:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#your-rights"},{"categories":null,"collections":null,"content":"EU\/EEA\/UK \u2014 GDPR and UK GDPR Access your data (Art. 15) Rectify inaccurate data (Art. 16) Erase your data (Art. 17) Restrict processing (Art. 18) Object to processing (Art. 21) Data portability (Art. 20) Withdraw consent at any time (Art. 7) Lodge a complaint with your supervisory authority (Art. 77) ","date":"01-Jan-0001","heading":"EU\/EEA\/UK \u2014","objectID":"\/privacy\/:10:1","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#eueeauk--gdpr-and-uk-gdpr"},{"categories":null,"collections":null,"content":"California \u2014 CCPA\/CPRA Right to know what data is collected Right to delete your data Right to opt out of sale\/sharing \u2014 I do not sell or share personal information Right to non-discrimination for exercising rights ","date":"01-Jan-0001","heading":"California \u2014","objectID":"\/privacy\/:10:2","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#california--ccpacpra"},{"categories":null,"collections":null,"content":"India \u2014 DPDPA 2023 Access and correction of your data Grievance redressal \u2014 contact me via the details below Withdraw consent at any time ","date":"01-Jan-0001","heading":"India \u2014","objectID":"\/privacy\/:10:3","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#india--dpdpa-2023"},{"categories":null,"collections":null,"content":"China \u2014 PIPL Google services are generally inaccessible in mainland China. If you access this website from China, no data transfer to Google occurs when Google’s servers are unreachable. ","date":"01-Jan-0001","heading":"China \u2014","objectID":"\/privacy\/:10:4","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#china--pipl"},{"categories":null,"collections":null,"content":"Brazil \u2014 LGPD Rights to access, correct, delete, and object to processing apply. ","date":"01-Jan-0001","heading":"Brazil \u2014","objectID":"\/privacy\/:10:5","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#brazil--lgpd"},{"categories":null,"collections":null,"content":"All other jurisdictions Regardless of location, I will honor reasonable requests to disclose, delete, or cease processing your data. ","date":"01-Jan-0001","heading":"All other jurisdictions","objectID":"\/privacy\/:10:6","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#all-other-jurisdictions"},{"categories":null,"collections":null,"content":"Children This website is not directed at children under 13 (or the applicable age of digital consent in your jurisdiction). I do not knowingly collect data from children. If you believe a child has submitted personal data, contact me for immediate removal. ","date":"01-Jan-0001","heading":"Children","objectID":"\/privacy\/:11:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#children"},{"categories":null,"collections":null,"content":"Do Not Sell or Share This website does not sell, rent, or share personal data with third parties for their own marketing purposes. Analytics data is used solely to understand content performance. Advertising (if active) is served by Google under their policies and your consent choices. ","date":"01-Jan-0001","heading":"Do Not Sell or Share","objectID":"\/privacy\/:12:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#do-not-sell-or-share"},{"categories":null,"collections":null,"content":"Limitation of Liability To the maximum extent permitted by applicable law: This website is provided “as is” without warranties of any kind, express or implied I shall not be liable for any direct, indirect, incidental, consequential, or punitive damages arising from your use of this website or the processing of your data by third-party services (including Google, Cloudflare, and GitHub) My total liability for any claim related to this privacy policy shall not exceed the amount you have paid to use this website (which is zero, as this is a free, non-commercial website) I am not responsible for the privacy practices, data handling, or security of third-party services linked from or integrated into this website \u2014 their respective privacy policies govern their processing This limitation does not exclude liability for fraud, gross negligence, or any liability that cannot be excluded under the mandatory laws of your jurisdiction. ","date":"01-Jan-0001","heading":"Limitation of Liability","objectID":"\/privacy\/:13:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#limitation-of-liability"},{"categories":null,"collections":null,"content":"Disclaimer Regarding Third-Party Services Google Analytics, Google AdSense, GitHub, and Cloudflare are independent third-party services. I do not control their data processing practices. By using this website, you acknowledge that: Data processed by these services is governed by their privacy policies (linked above), not solely by this policy I act in good faith to configure these services with privacy-protective settings (IP anonymization, consent mode, data retention limits), but cannot guarantee their compliance with every law in every jurisdiction Any claim regarding data processed by these third parties should be directed to them under their respective data processing agreements and privacy policies ","date":"01-Jan-0001","heading":"Disclaimer Regarding Third-Party Services","objectID":"\/privacy\/:14:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#disclaimer-regarding-third-party-services"},{"categories":null,"collections":null,"content":"Changes to This Policy This policy is based on applicable laws as of the date below. This website is a personal project maintained on a best-effort basis. While I strive to keep this policy accurate, I am not obligated to update it on a fixed schedule. The website’s actual practices (consent-first, no data sales, opt-out available) are designed to remain protective regardless of evolving legal requirements. Continued use after changes constitutes acceptance of the revised policy. Last Updated: June 2026 ","date":"01-Jan-0001","heading":"Changes to This Policy","objectID":"\/privacy\/:15:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#changes-to-this-policy"},{"categories":null,"collections":null,"content":"Dispute Resolution If you have a concern about this website’s privacy practices: Contact me first: Contact Form or netproxy@duck.com. I will respond within 30 days. Most concerns can be resolved informally. Supervisory authority: If unsatisfied, you may lodge a complaint with your local DPA (EU\/EEA), the UK ICO, or equivalent authority in your jurisdiction. Governing law: This policy is governed by the laws of India. Any disputes shall be subject to the exclusive jurisdiction of courts in India, except where mandatory local law grants you the right to bring proceedings in your own jurisdiction. Statutory rights preserved: Nothing in this policy limits or waives your statutory rights under the data protection laws of your country of residence (including GDPR rights for EU\/EEA\/UK residents, CCPA rights for California residents, or DPDPA rights for Indian residents). Severability: If any provision of this policy is found to be unenforceable or invalid, the remaining provisions shall continue in full force and effect. ","date":"01-Jan-0001","heading":"Dispute Resolution","objectID":"\/privacy\/:16:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#dispute-resolution"},{"categories":null,"collections":null,"content":"Contact Email: netproxy@duck.com Form: \/contact Response time: Within 30 days ","date":"01-Jan-0001","heading":"Contact","objectID":"\/privacy\/:17:0","tags":null,"title":"Privacy Policy","uri":"\/privacy\/#contact"},{"categories":null,"collections":null,"content":"Unsubscribe from Samir\u2019s newsletter to stop receiving new posts and updates by email. You can resubscribe anytime.","date":"01-Jan-0001","objectID":"\/unsubscribe-newsletter\/","tags":null,"title":"Unsubscribe Newsletter","uri":"\/unsubscribe-newsletter\/"},{"categories":null,"collections":null,"content":"Want to stop receiving emails? You can easily unsubscribe below. You\u2019ll no longer get new posts or updates \u2014 but you can rejoin anytime if you change your mind. ","date":"01-Jan-0001","heading":"","objectID":"\/unsubscribe-newsletter\/:0:0","tags":null,"title":"Unsubscribe Newsletter","uri":"\/unsubscribe-newsletter\/#"}]