[{"content":"Working for the leading AI company, people tend to ask you what AI means for their careers. Usually, they want to hear one of two extremes: “this changes nothing” or “we’re all doomed”. To be honest, I appreciate both. A lot of what I have spent learning over two plus decades is now just a prompt away. Hard-earned debugging skills and deep technical knowledge can be generated by a model in seconds. It’s uncomfortable to watch. But having your knowledge become cheap doesn’t mean you become irrelevant. There is a huge gap between those two statements.\nThe Shelf-Life of Expertise This isn’t actually new. Throughout my 20+ year career, the specific skills I fought hard to master have repeatedly stopped mattering. What kept me moving forward wasn’t the specific knowledge; it was my ability to be perspicacious in an ambiguous situation \u2014 learn fast. AI didn’t start this trend, it just turned the dial up to 11. What’s actually disappearing? Memorizing syntax, looking up basic facts, and writing boilerplate code. Tech has always marched toward automating the boring stuff. Real engineering has never been about memorizing commands; it’s about defining intent, making trade-offs, and managing systems.\nThe New Engineering Bottleneck When anyone can generate code for free, the bottleneck shifts from writing code to defining and verifying it. If you operate outside your comfort zone, you lose the ability to audit what the AI gives you. You need deep experience to spot the subtle, sometimes breaking mistakes an AI makes with total confidence. And trust me, it does make mistakes. Furthermore, unconstrained AI code creates a mess and bloat. We need engineers more than ever to clean up that architecture, manage dependencies, and keep the system sane. Code generation is a commodity; system orchestration is not.\nIt’s Okay to Admit This Hurts I’m not going to give you fake optimism. This shift genuinely hurts, and the anxiety people feel is completely justified:\nFewer entry-level spots \u2014 It’s harder for junior engineers to build judgment when they can’t do the basic, manual tasks that built mine. The middle squeeze \u2014 When one person can manage a fleet of AI agents, you simply need fewer people to do the same work. Loss of craft \u2014 The “flow state” of building things with your own hands is being replaced by reviewing and directing. “You’ll be more productive” is a pretty hollow answer to that loss. The Playbook for Staying Relevant Still, at every tech shift I’ve lived through, the winning strategy has been the same four moves:\nObserve \u2014 Use the tech or tools enough to know exactly where they shine and where they break. Learn \u2014 Understand the underlying primitives and how it all works. Adopt \u2014 Use it in new ways, apply it on the problems nobody has written about. Thrive \u2014 Stay close to the tech and take accountability for the final output. What Do We Do Now? Nobody actually knows how this ends, and anyone who claims they do is selling you something. But the answer isn’t to pretend it’s not happening, or to quit tech and take up woodworking. Get your hands on the tools. Find out exactly where they break. Be the person who can tell the difference between work that is actually done, and work that just looks done. My expertise has been eroding for over 20 years now. And honestly, I wouldn’t want it any other way.\n","permalink":"https:\/\/blog.chmarny.com\/posts\/staying-relevant-through-inflection-points\/","summary":"

Working for the leading AI company, people tend to ask you what AI means for their careers. Usually, they want to hear one of two extremes: “this changes nothing” or “we’re all doomed”. To be honest, I appreciate both. A lot of what I have spent learning over two plus decades is now just a prompt away. Hard-earned debugging skills and deep technical knowledge can be generated by a model in seconds. It’s uncomfortable to watch. But having your knowledge become cheap doesn’t mean you<\/em> become irrelevant. There is a huge gap between those two statements.<\/p>","title":"Staying Relevant Through Inflection Points"},{"content":"If you maintain an open source project, you’ve probably wondered: who’s actually contributing? Are we retaining contributors or burning through a revolving door? Is one person carrying the whole project? These questions matter.\nCommercial platforms answer these questions well. They do come with trade-offs though. Hosted analytics means sending your potentially private repo contributor data to a third party. Pricing models often assume enterprise budgets. And for a maintainer running a project with a handful of repos and no funding, the friction of setting up yet another SaaS integration is enough to never bother.\nDevPulse takes a different approach. It’s a single binary that imports GitHub data into a embedded database and serves a browser-based dashboard from your machine. No dependencies, no cloud account, no subscriptions. Your data stays on your machine.\nDevPulse supports external DBs like PostgreSQL so when the time comes, migration is easy. See Scaling Up section below for details.\nWhat you get DevPulse organizes project insights into six areas, each mapped to a dashboard tab:\nHealth \u2014 Bus factor and pony factor tell you how concentrated your project risk is. Bus factor counts the minimum number of developers producing half your contributions. Pony factor does the same for organizations. A bus factor of one means your project has a single point of failure.\nActivity \u2014 Monthly event volume broken down by type: pull requests, reviews, issues, comments, forks. A trend line shows whether activity is growing or declining. PR size distribution reveals whether your project culture favors small, reviewable changes or massive drops.\nVelocity \u2014 Lead time from PR creation to merge, change failure rate, and release cadence. These are DORA metrics \u2014 the same signals that elite engineering teams track \u2014 computed automatically from your GitHub data. You also get release download counts and container image publication frequency if you publish to ghcr.io.\nQuality \u2014 PR-to-review ratio shows whether code is being reviewed at the rate it’s being written. Review latency measures how long contributors wait for their first review. Time-to-close tracks issue responsiveness. Contributor reputation scoring surfaces identity confidence signals for people contributing to your project.\nCommunity \u2014 New contributors versus returning contributors per month. A rolling three-month active contributor count with momentum indicators. First-time contributor milestones (first comment, first PR, first merged PR). Top contributing organizations and individuals. This is the tab that tells you whether your community is actually a community or just a mailing list.\nEvents \u2014 Searchable, filterable event log. Click any data point in the charts above and the Events tab populates with the underlying records, each linked back to GitHub.\nGetting started Install via Homebrew:\nbrew tap mchmarny\/tap && brew install devpulse Or download a binary from the releases page. Binaries are available for macOS, Linux, and Windows across amd64 and arm64. Every release is signed with Sigstore, ships with an SBOM, and includes GitHub build provenance attestations \u2014 so you can verify exactly what you’re running.\nThen, three simple commands:\n# Authenticate with GitHub (stores token in your OS keychain) devpulse auth # Import data for your project devpulse import --org your-org --repo your-repo # Start the dashboard devpulse server Your browser opens to http:\/\/127.0.0.1:8080 (port and auto-open configurable) with a full analytics dashboard. The whole process takes a few minutes depending on how much history you’re importing. By default, DevPulse pulls six months of data. Adjust with --months if you want more.\nYour data, your machine The default storage backend is SQLite. Your database lives at ~\/.devpulse\/data.db \u2014 a single file you can back up, move, or delete. The dashboard reads from this local database. No data leaves your machine once the initial GitHub API import is complete.\nThis matters if you maintain a project where contributor data is sensitive. Maybe you’re tracking contributions across a corporate open source portfolio and don’t want that mapping exported to a third party. Maybe you’re a solo maintainer and just don’t want another account to manage. Either way, the data stays where you put it.\nBeyond the dashboard The CLI isn’t just a launcher for the web UI. It’s a complete data management toolkit:\nQuery \u2014 Pull structured JSON from your local database. List developers, search events by type and date range, explore entity affiliations. Pipe the output to jq for scripting:\ndevpulse query events --org my-org --repo my-repo \\ --type pr --author username --since 2026-01-01 Score \u2014 Enrich contributor profiles with deep reputation signals from the GitHub API. Account age, organization membership, commit signing history, profile completeness, activity patterns:\ndevpulse score --org my-org --count 20 Substitute \u2014 Normalize messy entity data. GitHub profile company fields are free-text, so “Google”, “Google LLC”, “Google Inc.”, and “google” all show up as separate organizations. Fix that in one command:\ndevpulse substitute --type entity --old "Google LLC" --new "GOOGLE" Sync \u2014 Automate imports on a schedule. Point it at a YAML config file listing your orgs and repos, and run it hourly via cron. DevPulse rotates through repos round-robin style to stay within GitHub API rate limits:\ndevpulse sync --config sync.yaml Scaling up when you need to SQLite works well for most use cases \u2014 a maintainer tracking a handful of repos, or a small team monitoring their project portfolio. But if you’re running DevPulse across a large organization with dozens of repos and thousands of contributors, you can switch to PostgreSQL:\ndevpulse import --org large-org --repo repo-1 \\ --db "postgres:\/\/user:pass@host:5432\/devpulse" The --db flag (or DEVPULSE_DB environment variable) accepts either a file path for SQLite or a postgres:\/\/ URI. Same CLI, same dashboard, same queries \u2014 just a different backend. This also opens the door to running DevPulse as a hosted service behind Cloud Run service or any container platform, with a managed database and scheduled sync jobs.\nThe point is that you start local and self-contained. If your needs grow, the path to a shared deployment is a flag change, not a rewrite.\nTry it DevPulse is open source under Apache 2.0. Install it, point it at your project, and see what the data says about your community. The questions it answers \u2014 who’s contributing, are we retaining people, how concentrated is our risk \u2014 are the ones that determine whether a project is sustainable. Having answers shouldn’t require an enterprise contract.\ngithub.com\/mchmarny\/devpulse\n","permalink":"https:\/\/blog.chmarny.com\/posts\/devpulse-community-health-analytics-for-the-rest-of-us\/","summary":"

If you maintain an open source project, you’ve probably wondered: who’s actually contributing? Are we retaining contributors or burning through a revolving door? Is one person carrying the whole project? These questions matter.<\/p>\n

Commercial platforms answer these questions well. They do come with trade-offs though. Hosted analytics means sending your potentially private repo contributor data to a third party. Pricing models often assume enterprise budgets. And for a maintainer running a project with a handful of repos and no funding, the friction of setting up yet another SaaS integration is enough to never bother.<\/p>","title":"DevPulse: Community Health Analytics for the Rest of Us"},{"content":"GPU Kubernetes is hard. Aligning kernels, drivers, container runtimes, operators, and Kubernetes versions is a version compatibility minefield. A single misconfigured component can take down an entire GPU fleet, and root cause analysis can take days. Typically, these known-good configurations live as tribal knowledge in “runbooks” and internal pipelines, not as portable, reproducible artifacts.\nThe recently open-sourced AI Cluster Runtime (AICR) project is designed to solve that friction. It provides optimized, validated, and reproducible configurations for a given GPU-accelerated Kubernetes cluster.\nThe Workflow AICR utilizes a composable, standards-based approach using YAML recipes and Helm bundles. The workflow is broken down into four stages:\nSnapshot (optional): Captures the current state of an existing GPU cluster Recipe: Generates an optimized, version-locked config for your environment Bundle: Converts the recipe into deployment-ready artifacts for a given deployer (e.g. Helm or ArgoCD, more too come soon) Deploy: Applies the bundle using your existing CD pipeline \u2014 this step is yours, no new tooling required Validate: Verifies deployment, performance, and conformance against the cluster AICR is about the optimized configuration for, and validation of your cluster, not the deployment or management of your cluster where it leans on the wealth of existing open source options.\nGetting Started The AICR CLI can be installed on Linux, macOS, or Windows (via WSL). You can use Homebrew:\nbrew tap NVIDIA\/aicr brew install aicr Alternatively, you can use the install script which does the implicit SLSA Level 3 provenance verification:\ncurl -sfL https:\/\/raw.githubusercontent.com\/NVIDIA\/aicr\/main\/install | bash -s Generating a Recipe Recipe captures the exact constraints and component versions required for your target state. You can specify your environment explicitly:\naicr recipe \\ --service eks \\ --accelerator h100 \\ --intent training \\ --os ubuntu \\ --platform kubeflow \\ --output recipe.yaml Or, you can point aicr at any existing GPU-accelerated Kubernetes cluster (EKS, GKE, AKS, or self-managed) to auto-discover that state:\naicr snapshot \\ --namespace aicr-validation \\ --node-selector nodeGroup=gpu-worker \\ --output snapshot.yaml aicr recipe \\ --snapshot snapshot.yaml \\ --intent training \\ --node-selector nodeGroup=gpu-worker \\ --output recipe.yaml Bundling for Deployment Once your recipe.yaml is generated, you convert it into deployable artifacts. For modern GitOps workflows, you can output directly to an OCI registry, or optionally include bundle cryptographic attestation using Sigstore:\naicr bundle \\ --recipe recipe.yaml \\ --deployer argocd \\ --output oci:\/\/ghcr.io\/nvidia\/bundle \\ --attest Supply chain security is built into every layer. Every bundle includes checksums for all generated artifacts end-to-end, and the project itself ships with SLSA Level 3 provenance, SPDX SBOMs, and cosign image attestations.\nValidate the Cluster Once deployed, AICR validates the cluster against the same recipe used to create the bundle. Validation runs across three phases:\nPhase What it proves deployment Fabric is properly configured and operational performance Synthetic workloads meet expected throughput conformance Overall cluster conformance is verified These phases can be run individually or all at once. Validation reports are output in Common Test Report Format (CTRF) to enable programmatic post-processing.\naicr validate \\ --recipe recipe.yaml \\ --phase all \\ --output report.json To verify CNCF AI Conformance, you can also add --evidence-dir to export evidence artifacts for CNCF submission.\nContributing The matrix of potential platform, GPU, OS, intent, and service configurations is bigger than any one team to manage. We built the framework, but we do need the community to help fill it in. If you’ve already validated and optimized a new configuration combination, run performance benchmarks, or written new deployment tests, please contribute to this project by opening a PR.\nDo star the repo, try it on your cluster, and open an issue with your findings: github.com\/NVIDIA\/aicr.\n","permalink":"https:\/\/blog.chmarny.com\/posts\/ai-cluster-runtime-reproducible-configs-for-gpu-accelerated-kubernetes\/","summary":"

GPU Kubernetes is hard. Aligning kernels, drivers, container runtimes, operators, and Kubernetes versions is a version compatibility minefield. A single misconfigured component can take down an entire GPU fleet, and root cause analysis can take days. Typically, these known-good configurations live as tribal knowledge in “runbooks” and internal pipelines, not as portable, reproducible artifacts.<\/p>","title":"AI Cluster Runtime: Reproducible Configs for GPU-Accelerated Kubernetes Clusters"},{"content":"Every dependency you install, every pull request you merge, carries an implicit trust decision. You trust that the person behind the commit is who they claim to be, that their account hasn’t been compromised, and that their contribution is genuine. Most of the time, that trust is warranted. But supply chain attacks like the xz utils backdoor remind us that trust without verification is a vulnerability.\nreputer is a CLI tool that calculates contributor reputation scores from Git provider APIs. It doesn’t claim to detect malicious intent. Instead, it answers a narrower question: based on observable signals from GitHub or GitLab (other signal providers coming soon), how confident should you be in a contributor’s identity and engagement? The score is best understood as an identity confidence indicator, not a judgment of code quality.\nWhat it measures reputer evaluates seven signals grouped into four threat-model-prioritized categories. Each category carries a weight reflecting its relative importance to supply chain risk:\nCode Provenance \u2014 Are commits cryptographically signed? Is two-factor authentication enabled? Unsigned commits from accounts without 2FA represent the highest-risk vector. An account with 2FA acts as a multiplier on the verification score, while accounts without it are penalized up to 50% on core reputation.\nIdentity Authenticity \u2014 How old is the account, and does the contributor belong to the repository owner’s organization? Freshly created accounts are inherently higher risk. Account age follows a logarithmic curve where early days matter most, with a ceiling at two years. Organization membership is a binary signal that provides strong identity corroboration.\nEngagement Depth \u2014 What proportion of the repository’s commits belong to this contributor, and how recently did they contribute? A long-time contributor with recent activity is more trustworthy than an account that appeared once. Commit proportion adapts to the contributor count, and recency uses exponential decay with a 90-day half-life.\nCommunity Standing \u2014 What does the contributor’s broader GitHub presence look like? Follower-to-following ratio and repository count provide weak but useful signals. These carry the lowest weight because they’re the easiest to fabricate.\nHow scoring works reputer uses a graduated proportional model. Each signal contributes linearly between zero and its weight ceiling. The final score sums all signal contributions to produce a value between 0.0 and 1.0.\nThree mathematical curves shape the signals:\nLinear clamping for ratios like commit proportion, where more is proportionally better up to a ceiling Logarithmic scaling for signals like account age and follower ratio, where early growth matters more than late growth (diminishing returns) Exponential decay for recency, where the signal halves roughly every 90 days of inactivity A suspended account always scores zero, regardless of other signals. The model version is included in every report, so consumers can track scoring methodology changes over time.\nArchitecture The tool follows a layered design:\nCLI \u2192 Reporter \u2192 Provider \u2192 Scoring Engine The CLI layer parses flags and authenticates against the Git provider. The Reporter orchestrates the query and serializes output. The Provider layer abstracts the Git hosting API \u2014 GitHub is fully implemented, GitLab is mostly done. The Scoring Engine in pkg\/score is a standalone package with no provider dependencies, making it independently testable.\nWhen evaluating a repository, reputer paginates through commits, then concurrently fetches user profiles (up to 10 at a time) with built-in rate-limit awareness. It pauses when API calls run low and waits for the rate limit window to reset, so it won’t burn through your token quota.\nThe output is structured JSON that includes repository metadata, the scoring model version, category weight definitions, and per-contributor scores with optional detailed stats.\nUsing the CLI Install via Homebrew, Go, or download a binary from the releases page:\nbrew tap mchmarny\/reputer && brew install reputer Then point it at a repository:\nreputer --repo github.com\/owner\/repo Add --stats to see the raw signals behind each score, or --file report.json to write output to a file. The JSON report includes every contributor’s reputation score alongside the model metadata:\n{ "repo": "github.com\/owner\/repo", "total_commits": 338, "total_contributors": 4, "meta": { "model_version": "2.0.0", "categories": [ { "name": "code_provenance", "weight": 0.35 }, { "name": "identity", "weight": 0.25 }, { "name": "engagement", "weight": 0.25 }, { "name": "community", "weight": 0.15 } ] }, "contributors": [ { "username": "contributor-a", "reputation": 0.92, "context": { "created": "2016-03-12T10:22:08Z", "name": "Alice Example", "company": "ExampleOrg" }, "stats": { "verified_commits": true, "strong_auth": true, "age_days": 3634, "commits": 295, "unverified_commits": 0, "public_repos": 47, "private_repos": 12, "followers": 183, "following": 22, "last_commit_days": 3, "org_member": true } }, { "username": "contributor-b", "reputation": 0.41, "context": { "created": "2024-11-08T15:44:31Z", "name": "Bob NewDev" }, "stats": { "age_days": 470, "commits": 12, "unverified_commits": 12, "public_repos": 3, "followers": 2, "following": 15, "last_commit_days": 38 } } ] } Using the GitHub Action reputer also ships as a GitHub Action that automatically welcomes first-time pull request contributors with their reputation stats. Add a workflow file to your repository:\nname: welcome on: pull_request_target: types: [opened] permissions: pull-requests: write contents: read jobs: welcome: runs-on: ubuntu-latest steps: - uses: mchmarny\/reputer@main When a new contributor opens their first PR, the action posts a comment with a reputation summary table \u2014 account age, 2FA status, commit verification, follower count, and the overall score. It gives maintainers a quick signal without requiring them to manually investigate the contributor’s profile.\nThe action uses pull_request_target rather than pull_request, meaning it runs in the context of the base repository and never checks out untrusted PR code.\nConclusion No single tool solves supply chain security. But having a quantified, reproducible signal about contributor identity confidence is one more data point for maintainers making trust decisions. reputer is open source under Apache 2.0 \u2014 try it on your own repositories, and decide for yourself whether the signals it surfaces are useful for your threat model.\ngithub.com\/mchmarny\/reputer\n","permalink":"https:\/\/blog.chmarny.com\/posts\/reputer-scoring-contributor-reputation\/","summary":"

Every dependency you install, every pull request you merge, carries an implicit trust decision. You trust that the person behind the commit is who they claim to be, that their account hasn’t been compromised, and that their contribution is genuine. Most of the time, that trust is warranted. But supply chain attacks like the xz utils backdoor<\/a> remind us that trust without verification is a vulnerability.<\/p>","title":"Reputation scoring for open source contributors: what reputer measures and why"},{"content":"Builder, architect, and strategist dedicated to simplifying how developers interact with complex systems. Currently serving as a Principal Cloud Architect in NVIDIA\u2019s DGX Cloud organization, Mark focuses on bridging the gap between cutting-edge AI hardware and the cloud-native ecosystem. His mission is to enable optimized, validated, and reproducible GPU-accelerated infrastructure in Kubernetes for everyone.\nMark\u2019s work spans the entire lifecycle of large-scale AI\/ML platforms, including GPU runtime, networking, storage, and performance validation across heterogeneous, multi-cloud environments used for training and inference. Throughout a distinguished career at Google Cloud, Apple, and Microsoft Azure OCTO, Mark has championed “developer-first” platforms, leading engineering efforts in Kubernetes control-planes, software supply chain security, and distributed workload management.\nCore Expertise:\nAI\/ML Infrastructure: Architecting large-scale Kubernetes-based platforms for GPU clusters, including Blackwell-class compute infrastructure. Led GB200 enablement efforts on Kubernetes across AWS, GCP, and OCI, working closely with each CSP to ensure an optimized experience for their users. Cloud-Native Strategy: Steering the technical direction for serverless computing, multi-cloud deployment frameworks, CI\/CD, and secure software supply chains. Platform Engineering: Transforming internal technical expertise into public-facing, machine-consumable runtime data and validated recipes to reduce duplication and improve consistency across the ecosystem. Open Source Leadership: Deeply committed to the open-source community through leadership in projects like Knative, Tekton, and Dapr, as well as broader industry initiatives like NVSentinel to improve GPU fleet health and remediation. You can find me on Bluesky (@chmarny.com), Mastodon (@mchmarny@fosstodon.org), and my open source contributions on GitHub (@mchmarny)\nProfile picture\nNote: All posts in this blog represent Mark’s personal opinions.\n","permalink":"https:\/\/blog.chmarny.com\/about\/","summary":"

Builder, architect, and strategist dedicated to simplifying how developers interact with complex systems. Currently serving as a Principal Cloud Architect in NVIDIA\u2019s DGX Cloud organization, Mark focuses on bridging the gap between cutting-edge AI hardware and the cloud-native ecosystem. His mission is to enable optimized, validated, and reproducible GPU-accelerated infrastructure in Kubernetes for everyone.<\/p>","title":"About Mark Chmarny"},{"content":"Today marks a bittersweet moment as I say goodbye to Cruise. When I joined the company seven months ago, my mission was to scale the AV services worldwide and to modernize the AV and Cloud service developer platforms. Despite the unexpected challenges following the October incident, my journey at Cruise has been incredibly enriching, teaching me the true essence of resilience, adaptability, and commitment to excellence.\nThis experience has also deepened my appreciation for the pivotal role of platforms in AI development, emphasizing the need for seamless creation, deployment, and scaling of AI applications. My previous work with platforms like Cloud Foundry, Knative, and Dapr has shown me the power of simplifying complexity and how it can fuel innovation. The AI field is ripe with opportunities for these advancements. I’m excited to dive deeper into this space.\nLooking ahead, I’m thrilled to embark on a new chapter focused on improved access to datasets, computational resources, and pre-built models. I’m eager to share more about this journey next week.\n","permalink":"https:\/\/blog.chmarny.com\/posts\/leaving-cruise\/","summary":"

Today marks a bittersweet moment as I say goodbye to Cruise<\/a>. When I joined the company seven months ago, my mission was to scale the AV services worldwide and to modernize the AV and Cloud service developer platforms. Despite the unexpected challenges following the October incident<\/a>, my journey at Cruise has been incredibly enriching, teaching me the true essence of resilience, adaptability, and commitment to excellence.<\/p>","title":"Leaving Cruise; why I'm still excited about AI platforms"},{"content":"The practice of Secure Software Supply Chain (S3C) can get complex at times. Fortunately though, a large portion of the key things we can do to secure our software delivery pipelines are actually pretty easy. This post covers three concepts you can implement today:\nprovenance - Where and how was the software created? Using what source code, tools, and build system? Who were the individuals involved in its development? transparency - What\u2019s in the software artifact? What are the packages, libraries, and their versions? Who provided them? And, what are the relationships between them? (what depends on, uses, what) context - What are the vulnerabilities, and what is their potential exploitability in that specific artifact? Let’s review each one of these and provide some options on how to implement them in your software release pipeline.\nProvenance The open source community under OpenSSF has created a security framework for how we can think of build provenance. This framework is called Supply Chain Levels for Software Artifacts (SLSA). it provides a standardized check-list of controls to improve the integrity, prevent tampering, and secure software artifacts and build infrastructure.\nAt its core, SLSA is based on the builders producing conformant and signed provenance documents (aka build attestation). The provenance document includes Build Definition (e.g. parameters, dependencies), and Run Details (e.g. builder, metadata). And, to make it easier for us to reason about it, the key characteristics of the build provenance are grouped into SLSA Levels (1-3). The downstream consumers of the produced artifacts can inspect the provenance, and base their policy decisions on the inferred level of SLSA. More about SLSA levels on slsa.dev.\nAdd SLSA-based build provenance to your release pipeline:\nGoogle Cloud Build - Level 3 assurance automatically generated in Artifact Analysis for all images pushed to Artifact Registry GitHub Actions - Reusable workflow, which when added to your release pipeline generates Level 3 build provenance GitLab CI\/CD - Runner which can generate artifact metadata meeting SLSA Level 2 with support for external code signer to potentially deliver Level 3. Regardless of the means by which SLSA provenance was generated, you can use the SLSA Verifier CLI to ensure its validity.\nTransparency While SLSA is about how the software was produced, Software Bill of Materials (SBOM) is about what’s in the software. Think of SBOM like an ingredients label on a can of food you buy in the grocery store. It’s an inventory of all the components that make up the software. SBOMs are used during the production of software (e.g. license compliance), during process of software selection (e.g. policy compliance), or during operations (e.g. risk management). SBOMs can be also used in all three of these use-cases for vulnerability monitoring as new vulnerabilities are discovered all the time. More about SBOM and the related US regulation on cisa.gov\/sbom. Similar regulations apply in the EU and the UK. While there is many formats\/flavors of SBOM, the most common open standards today are:\nSPDX (Software Package Data Exchange) - Provides a structured format for documenting software components, licenses, copyrights, and security vulnerabilities, including information about packages, files, and relationships. It’s maintained by the Linux Foundation (LF), and it has a broader ecosystem of tools and libraries which can make it easier to integrate into your pipeline. Given the extensive set of fields and options, SPDX can get complex for humans to reason about though. CycloneDX - Provides a lightweight and flexible representation of the components and dependencies, including details about packages, versions, licenses, and vulnerability information, allowing for interoperability between different tools and platforms. CycloneDX allows for extensibility (see VEX section below). However, unlike SPDX, CycloneDX is not an industry-standard format, and while it has gained traction and adoption, it may not enjoy the same level of standardization and industry-wide acceptance as SPDX. Ultimately, the choice between SPDX and CycloneDX depends on your specific needs, the ecosystem you operate in, and the level of standardization and extensibility required for your software supply chain management.\nYou can add SBOM to your release pipeline in a number of ways:\nUsing Lockfile - These SBOMs are generated using the information in language specific lockfile (e.g. go.mod, gradle.lockfile, or pom.xml). Sometimes they are generated by your Source Code Management System (e.g. GitHub SBOM). While easy, this approach does not account only for the user defined code, and lacks information about the base image on which this code will be deployed. Lockfile-based SBOMs don’t work well also for repos which build to more than one artifact. During Build - Generating SBOMs during build is not very common yet. For the Go ecosystem, there is ko, and for Java Maven using the CycloneDX plugin. The benefit of this approach is build tools have a deep understanding of the dependencies, and the SBOM generation becomes this way an inherent part of the build workflow, ensuring consistency with each build (e.g. format) From Container - These SBOMs are generated from the final product. This is the most common approach to generating SBOMs today (e.g. using OSS tools like syft or snyk). The reliance on the layered file systems or container image may result in an incomplete view of all the dependencies, especially with transitive dependencies and in situations when not installed using package manager. Context Unless you are building a simple hello world application, chances are high you will have some vulnerabilities in your software and\/or its dependencies. The thing that’s important to realize though is that software can contain components with vulnerabilities and yet not be vulnerable itself. This is where Vulnerability Exploitability eXchange (VEX) comes in. It’s a machine-readable file that can be used by producers to convey information about the actual “exploitability” of each vulnerability and prioritize remediation, and by the consumers as an input into policy controls driving their release process.\nThe file itself is based on set of assertions (impact statements) about the status of a vulnerability in specific artifact and deployment context:\nNot affected \u2013 No remediation is required regarding this vulnerability. Affected \u2013 Actions are recommended to remediate or address this vulnerability. Fixed \u2013 Represents that these product versions contain a fix for the vulnerability. Under Investigation \u2013 Still looking whether this artifact is affected by this vulnerability. Somewhat early days for VEX, so there isn’t yet a well-formalized standard or many OSS tools. Still, A couple of interesting initiatives in this space worth keeping an eye on:\nCycloneDX 1.4 - This version of the CycloneDX SBOM format adds support for VEX’d vulnerabilities. This embedded VEX provides a single artifact that describes both inventory and VEX data. OpenVEX is a minimal (but compliant) implementation of the VEX format optimized for interoperability. OpenVex includes spec, and a growing ecosystem of tools (Go, .NET, Rest). Conclusion Hope this post helped you identify at least one thing you can do in your software release pipeline to increase its supply chain security.\n","permalink":"https:\/\/blog.chmarny.com\/posts\/provenance-transparency-context-3-aspects-of-s3c-you-can-implement-today\/","summary":"

The practice of Secure Software Supply Chain (S3C) can get complex at times. Fortunately though, a large portion of the key things we can do to secure our software delivery pipelines are actually pretty easy. This post covers three concepts you can implement today:<\/p>","title":"Provenance, transparency, and context, the three aspects of software supply chain security you can implement today"},{"content":"If you are doing any vulnerability detection in your software release pipeline today, you are already familiar with the volumes of data these scanners can generate. That dataset gets significantly larger when you add things like license scanning and Software Bill of Materials (SBOM) generation. That volume of data gets further compounded with each highly-automated pipeline you operate. This can quickly lead to what I refer to as a Software Supply Chain Security (S3C) data fatigue, as many vulnerabilities you\u2019ll discover you simply can\u2019t do anything about. There is an actionable signal in there actually, it\u2019s just hard to find it in the midst of all the noise.\nOver the last year, there has been a growing number of existing and new security focused ISVs who are starting to now provide integrated products to help you management of all this data and offer algorithms to automate the discovery (e.g. Chainguard, Mend, Synopsys, Sonatype, Snyk). In GCP, we\u2019ve also added Container Analysis API to enable metadata management.\nTo learn more about S3C data, and to better understand the challenges in this space, I\u2019ve put together a simple solution called disco to:\nContinuously discover container images used in my workloads across multiple GCP projects, regions, and runtimes Easily plug different open source image scanners (e.g. syft and trivy for SBOM, grype, osv-scanner, snyk, and trivy for vulnerabilities, and some combination of each of these for licenses) Automatically manage data exports for: Raw scanner reports into GCS bucket Key data feature metrics into Cloud Monitoring time-series Synthesized data into BigQuery tables Here is a high-level topology of the disco solution:\nThe four challenges I\u2019ve been thinking about in relation to the above-described S3C data noise, along with a simple pragmatic approaches I wanted to evaluate are:\nLarge number of data sources (i.e. many images with many versions in container registry, which ones are actually used?) - Scope sources down to only what\u2019s actually being used at any given time to underpin the live services. Ideally the discovery shifts left and integrates into release pipeline, but, if the release frequency is not high (order of days, or more), this may be valuable addition, as new vulnerabilities are discovered all the time (i.e. Day 0). Multiple data formats (i.e. SPDX, CycloneDX, which version?) - Normalize the data into a consistent set that maps the key identities (e.g. packages, files, relationships, licenses, vulnerabilities, etc.), across multiple formats. Hard to parse signal in raw data (hard to reason over values in JSON or YAML file) - Identify key features and automate metering to create a change detection system to focus on significant events using threshold rules scoped to a project, deployment, runtime, image, or even single package. Point-in-time perspective (hard to compare multiple sources or capture deviation over time) - Enable forensic (historical) analysis over data spanning longer periods of time to detect trends and potential gaps using commonly known skills and technologies (e.g. SQL) to enable a broader number of reporting tools and audiences. Here is how I\u2019ve implemented each one of these approaches and what I have learned in the process:\nImage Discovery To find images that are being actively used in live services I\u2019ve written a simple Go client. That client queries the GCP API for projects, and traverses the active deployments in each runtime to find the specific image digest. This is basically equivalent to this gcloud commands like this:\ncurl -H "Content-Type: application\/json; charset=utf-8" \\ -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \\ "https:\/\/cloudresourcemanager.googleapis.com\/v1\/projects" The project, regions, runtimes, and service that client discovers are automatically scoped to the identity under which the disco service operates. This makes it easy to manage cross-project discovery using IAM roles.\nNew services are deployed all the time, so the disco service uses a cron job in Cloud Scheduler to continuously execute the discovery service (defaults to hourly). Technically, each new deployment in these runtimes has a corresponding event in Eventarc, so ad-hoc scans for newly updated services are possible. Still, to keep things simple, disco operates in batches.\nThe actual digest that underpins the live service in Cloud Run is stored on the revision, and Cloud Run can split traffic across multiple revisions. To deal with this, disco uses API filters and traverses the revisions to identify all of the \u201cactive\u201d ones. Also, while somewhat counter-intuitive, Cloud Functions does actually build images behind the scene when you deploy a new function. These are managed by GCP, so the GCF API doesn\u2019t expose the actual digest, but it does provide the revision ID, which can be used against the Cloud Run API to discover the actual digest.\nData Normalization To extract data from each one of the discovered images, disco uses one of the preconfigured open source scanners. These scanners generate reports which disco then uses to extract three main entities:\nPackages SBOM file generated by scanner (default: SPDX v2.2 using trivy although I have experimented with the other formats I found in the set the Chainguard team maintains in their (s)bom-shelter). The disco mapping is mostly flattened \u2018.packages\u2019 data with file level metadata. I\u2019ve started experimenting with traversing the \u2018.relationships\u2019 graph but this turned out to be a lot more complicated than I expected, and didn\u2019t really add that much value to the simple use-case I was after. (I\u2019d like to integrate GUAC here in the future to normalizing entity identities via mapping standard relationships using graph DB)\nVulnerabilities From a vulnerability report generated by one of the OSS scanners (default: Trivy) for all layers in each image (SPDX v3.0 will include vulnerability data, so this will remove the need for additional scans and different parsers. For now though, the subset of packages with vulnerabilities need to be mapped to all of the packages found in the image. I\u2019ve also started to look into the CVSS scores from different vendors, but still have much to learn about the lifecycle and implications of these so for now just using the \u201csimple\u201d severity\nLicenses From OS and package license report used in each image using an OSS scanner (default: trivy). Technically, you could pull the license information from SBOM, but the dataset in the explicit report is more complete and I was able to avoid the whole \u201cdeclared\u201d (by vendor) vs \u201cconcluded\u201d dance. Trivy also provides a \u201cconfidence\u201d indicator which helps in downstream analysis\nOnce extracted, in addition to all the other data, disco also normalizes the shape of the key elements for all of these identities to enable cross-reference:\nBatchID - single ID for the entire batch to make deduplication easier ImageURI - image URI (without tag or sha) for queries spanning multiple versions ImageSHA - image sha to ensure unique identity for specific image version Package - fully qualified package name PackageVersion - canonical version of the package Updated - timestamp when the data element was extracted Metrics Export\nEven with only a small number of services across a couple of runtimes, disco will quickly generate thousands of data points. To ensure that the meaningful signal is detected early, disco exports custom metrics from each scan, for example:\ndisco\/vulnerability\/severity - vulnerability count for each severity (labels: severity value, project, service, runtime, image) disco\/license\/image - licenses per image (labels: license kind, project, service, runtime) disco\/package\/image - packages per image (labels: project, service, runtime) Once exported, I was able to use Metric Explorer to review the generated metrics:\nI was also able to create notifications policies based on metric thresholds to alert using one of the supported channels (e.g. Mobile device app notification, Slack, Email, SMS, etc), and route the events to another process downstream using Pub\/Sub (or Webhook).\nAt the end, only the severity metric turned out to be of significant value here. Packages, and licenses have way too high cardinality for individual metrics, and generic point metrics are fun for graphs, but I\u2019m not sure anyone would actuate something on that (even when generalized, because there are at least ten diff ways to spell Artistic-2).\nForensic Analysis Each of the data sources in disco (image, package, license, and vulnerability), has many data points, so for broader analysis over a longer period of time, disco also batch exports its data to BigQuery tables. The schema of each one of the tables is available here. To ensure the queries are performant over time regardless of volumes, disco partitions each table per day. More on partitioned tables here.\nOnce in BigQuery, I was able to use standard SQL to query over that data, using different joins across all the three main identities (license, package, vulnerability). I haven\u2019t had the time to do it yet, but it would be interesting to combine the reports across different CVE sources like NVD or VULDB, to see how each reports CVEs over time. Using this setup, I was however able to identify a few interesting deviations over time. For example:\nPackage version changes between versions, and cross-service distribution Vulnerability (CVE) severity and score changes Package license changes (yep, those do happen), see FOSSA And because BigQuery has a rich ecosystem of visualization tools, I can also easily create reports from all this collected data like this one in Looker Studio which actually allows for drill-downs (select image and only its vulnerabilities, licenses, and package distribution).\nIn Summary If nothing else, building disco was an educational experience. I gained appreciation for some of the idiosyncrasies in SBOM formats, I\u2019ve learned about the different vulnerability data sources, and evaluated the tradeoffs in data aggregation and synthesis. This already happens in companies with more advanced SecOps, I do think, eventually, this data will more broadly drive policy decisions, be instrumental in audits, and used to manage risk. This data could also potentially drive better development practices. All in all, this is an exciting space with much, much more room for innovation. I look forward to digging deeper.\nIf you are interested, the disco source code, and Terraform-based deployment with prebuilt images, is available in my repo in github.com\/mchmarny\/disco. There also is a CLI you can use locally (installation instructions here). Hope you found this helpful.\n","permalink":"https:\/\/blog.chmarny.com\/posts\/automating-software-supply-chain-security\/","summary":"

If you are doing any vulnerability detection in your software release pipeline today, you are already familiar with the volumes of data these scanners can generate. That dataset gets significantly larger when you add things like license scanning and Software Bill of Materials<\/a> (SBOM) generation. That volume of data gets further compounded with each highly-automated pipeline you operate. This can quickly lead to what I refer to as a Software Supply Chain Security (S3C) data fatigue, as many vulnerabilities you\u2019ll discover you simply can\u2019t do anything about. There is an actionable signal in there actually, it\u2019s just hard to find it in the midst of all the noise.<\/p>","title":"Software supply chain data fatigue and what I\u2019ve learned from SBOM, vulnerability reports"},{"content":"Twitter does provide notifications for when new users start following you. It does now however provide any notifications when users stop following you. Now, there is an ample of web sites out there who do provide that service, in most cases though, they cost money, and ask you for a complete access to your Twitter account.\nThis is the Twitter prompt to authorize access for these kind of sites to your account. Do you feel comfortable agreeing to this after reading what that application actually will be able to do?\nNow, it’s hard to blame these sites for asking for such extensive access though. Twitter app configuration settings allow only two modes of access request, Read Only, and Full Access like the one above. I’ve tweeted before about the need for more granular access permissions in Twitter here, so who knows, maybe one day.\nPersonally, I wasn’t willing to give some site out there a full access to my Twitter account. So, I did what most technologists do in these kinds of situations… I’ve built my own. It’s called TweeThingz and you use it for free at twee.thingz.io.\nTweeThingz provides visualization of the follower trends, reports of daily follower events, and optional on-change email notifications. Best of all, TweeThingz only needs a read access to your Twitter account!\nYou can also get email notifications whenever there are daily changes to your follower, friend, or list stats. The notifications can be disabled in the Account preferences.\nFinally, TweeThingz is very much a personal project. There is no ads or tracking. I’m running this service under personal account in Google Cloud, using Cloud Run and use Cloud SQL for state management, which until this grows to thousands (millions?) of users, makes TweeThingz operations pretty inexpensive. Enjoy it, and let me know what you think.\n","permalink":"https:\/\/blog.chmarny.com\/posts\/twitter-follower-activity-monitoring-using-tweethingz\/","summary":"

Twitter does provide notifications for when new users start following you. It does now however provide any notifications when users stop following you. Now, there is an ample of web sites out there who do provide that service, in most cases though, they cost money, and ask you for a complete access to your Twitter account.<\/p>","title":"Twitter follower status monitoring made easy using TweeThingz"},{"content":"The recently introduced by GitHub support for OpenID Connect (OIDC) tokens in GitHub Actions allows workflows to mint new tokens and then exchange those tokens for short-lived OAuth 2.0 or JWT tokens. These tokens can be used to access and manage Cloud resources. This is all done without the need to store the traditional long-lived service account keys in a form of GitHub secrets.\nOn Google Cloud Platform (GCP) this process uses Workload Identity Federation and enable your GitHub Actions a fine-grained access scoping using attribute mappings between the OIDC token, which by default, automatically expires one hour after it was created, to impersonate a GCP service account and access resources on Google Cloud using the designated GCP permissions.\nOIDC Complexity The problem is that setting up identity federation for GitHub Actions in any of the Cloud Service Providers (CSP) requires executing number of CLI commands, each with endless number of flags to implement the required attribute mappings between the provider access permissions and GitHub repository or username.\nIn this post I’ll outline a reproducible process of setting up Identity Federation in GCP for one of the most common use-cases in GitHub Actions, building and publishing container images.\nReproducible Setup Each execution of this process applies to a single GCP project and GitHub repository combination. To make this setup reproducible we’re going to use Terraform. By providing different variables during execution, we\u2019ll be able to apply that same Terraform configuration to any other GitHub repo and GCP project combination.\nPrerequisites The prerequisites to executing this setup include:\nTerraform CLI GCP Project gcloud CLI Good how-to on using terraform with GCP is located here.\nOne-time Setup To acquire the reproducible Github Workflow OpenID Connect setup for GCP you can either clone the Repo using SSH:\ngit clone git@github.com:mchmarny\/oidc-for-gcp-using-terraform.git or using HTTP:\ngit clone https:\/\/github.com\/mchmarny\/oidc-for-gcp-using-terraform.git Once you’ve cloned the setup repo, navigate inside of that cloned directory and initialize Terraform\nMake sure to authenticate to GCP using gcloud auth application-default login if you haven’t done it already.\nterraform init Note, this flow uses the default, local terraform state. Make sure you do not check the state files into your source control (see .gitignore), or consider using a persistent state provider like GCS.\nExecuting Configuration To configure Github Workflow OpenID Connect setup for GCP apply the cloned configuration:\nterraform apply When promoted, provide the 2 required variables:\nproject_id is the GCP project ID (not the name) which you want to target from your GitHub Action. git_repo is the username\/repo combination in which you GitHub Actions will be executing What Included You can review each one of the *.tf files for content. When you confirm yes at the final prompt, the main artifacts created by this setup in the GCP project defined by the project_id variable include:\nEnablement of the required GCP APIs servicecontrol.googleapis.com containerregistry.googleapis.com iam.googleapis.com iamcredentials.googleapis.com servicemanagement.googleapis.com storage-api.googleapis.com Creation of github-actions-user service account which the GitHub Action will impersonate when publishing images into GCR, and binding that account to the two required role: roles\/storage.objectCreator roles\/storage.objectViewer Creation of the workload identity pool: github-pool, and GitHub repo-level pool provider: github-provider Finally, creation of the IAM policy bindings to the service account resources created by GitHub identify for the specific GitHub repository defined by the git_repo variable Repo Configuration The result each execution of the above defined configuration will include 3 GitHub repo configuration properties:\nPROJECT_ID which is the project ID in which you setup the workload identity federation SERVICE_ACCOUNT which is the IAM service account your GitHub Action workflows will use to push images into GCR (e.g. github-action-publisher@<project_id>.iam.gserviceaccount.com) IDENTITY_PROVIDER which is the workflow identity provider ID you must use lng with the above service account to connect to GCP (e.g. projects\/<project_number>\/locations\/global\/workloadIdentityPools\/github-pool\/providers\/github-provider) Depending on your tolerance, you may be OK using all 3 of these parameters in your GitHub Actions workflow in plain-text. In most cases, however, you will probably create GitHubsecrets in your repository to inject them into your workflow at runtime.\nGitHub Workflow Configuration With the Workload Identity Federation configured yur workflow can now establish delegated trust relationships to the narrowly scoped set of permissions in GCP. The google-github-actions\/auth includes many examples using gcloud in your workflow.\nIn this post I’m going to focus on Go-specific configuration using ko, (a super simple and fast container image builder for Go apps) to build and publish images into GCR. The full workflow is available here. The key steps include:\nPush Job First, in order to create OIDC tokens, the GitHub Actions will need additional permissions. In addition to regular content read, the workflow will also id-token write.\njobs: push: runs-on: ubuntu-latest permissions: contents: read id-token: write steps: GCP Authentication In order to push images to GCR, the workflow will need to first authenticate to GCP. Google has an action just for that that can be configured to generate OAuth 2.0 Access Token. To do this you will need to set the token_format to access_token. Additionally, this step will use the workload identity provider and service account secrets we configured above:\n- id: auth name: Get GCP token uses: google-github-actions\/auth@v0.5.0 with: token_format: "access_token" workload_identity_provider: ${{ secrets.IDENTITY_PROVIDER }} service_account: ${{ secrets.SERVICE_ACCOUNT }} Install And Login Ko Ko is the fastest way of creating container images in Go without Docker. All we need to do is install it and login to GCR with the access token created by the auth step above:\n- name: Install Ko uses: imjasonh\/setup-ko@v0.4 with: version: tip - name: Login With ko run: | ko login gcr.io --username=oauth2accesstoken --password=${{ steps.auth.outputs.access_token }} Publish Image With ko logged in, now you can build and publish the image. A few things to highlight here. ko build (pka publish) will build and publish container images from the given path. The --image-refs flag will output the digest of the published image to the provided file, and the --bare allows us to define the full image URL using the KO_DOCKER_REPO environment variable.\nIn addition to this we will set the previously exported RELEASE_VERSION environment variable to both version field in the main.go file and set it as a tag on the image.\n- name: Publish Image run: | ko build .\/cmd\/ --image-refs .\/image-digest --bare --tags ${{ env.RELEASE_VERSION }},latest env: KO_DOCKER_REPO: gcr.io\/${{ secrets.PROJECT_ID }}\/restme GOFLAGS: "-ldflags=-X=main.version=${{ env.RELEASE_VERSION }}" Sign Image Once the image is published, we can also sign and verify the published image in GCR using cosign.\n- name: Install Cosign uses: sigstore\/cosign-installer@main with: cosign-release: v1.4.1 The benefit of combining ko and cosign is that we can use the image digest output into a local file by ko by providing its path using --force flag in the cosign sign command.\nWith the v1.4 release of cosign, you set th COSIGN_EXPERIMENTAL variable to push the data into GCR.\n- name: Sign Image run: | cosign sign --force $(cat .\/image-digest) env: COSIGN_EXPERIMENTAL: 1 Summary Hope you find this helpful. The Terraform configuration is available in my oidc-for-gcp-using-terraform repo, and you can find a fully functional example fo the image-on-git-tag GitHub Actions workflow in my restme repo.\n","permalink":"https:\/\/blog.chmarny.com\/posts\/reproducible-github-workflow-openid-connect-for-gcp-using-terraform\/","summary":"

The recently introduced by GitHub support for OpenID Connect<\/a> (OIDC) tokens in GitHub Actions<\/a> allows workflows to mint new tokens and then exchange those tokens for short-lived OAuth 2.0 or JWT tokens. These tokens can be used to access and manage Cloud resources. This is all done without the need to store the traditional long-lived service account keys in a form of GitHub secrets.<\/p>","title":"Reproducible OpenID connect to GCR for GitHub Actions workflows using Terraform"},{"content":"I learn best by doing. And recently, most of the projects I’ve been building are either REST or gRPC-base services deployed as container images into Cloud Run on GCP. That means that I increasingly find myself recreating a lot of the same infra and app deployment flows.\nOver time, the stack for these services started getting also more complex. Multi-region deployment in Cloud Run, fronted by Load Balancer with custom domain and SSL cert, throttled by Cloud Armor policies, and pushing images to GCR from GitHub action without the need to provision service account keys.\nSo, over the holidays, I’ve decided to automate the provisioning of the entire stack in a GitHub template project so I can quickly bootstrap new services, in a more reproducible fashion, and focus on actually writing the code, not tweaking the infra.\nThe result, restme. It’s a Terraform-based configuration to provision into your GCP project a fully functional, multi-region, REST service with a secure developer release pipeline in GitHub.\nHere is what’s included:\nCloud Run service provisioning Configured in an n number of regions with Custom identity (service account) Sample Secret Manager-based variable Configurable capacity and autoscaling strategy Accessible only by Internal and Load Balancer traffic (no external access) HTTPS Load Balancer with external IP and Custom domain with SSL certificate Cloud Armor policies for throttling and Canary CVE Serverless NEGs config for Cloud Run service in each region Dev and Ops Configuration for Container registry (GCR) Service uptime and SSL cert expiration alerts IODC-based Workload Identity Pool short-lived credentials provider for GitHub Actions Service logging with GCS bucket sink Local test, lint, and validate actions using Makefile GitHub Actions to test each PR and container image build\/publish to GCR on git tag The template project also includes Go source code for very rudimentary REST services based on gin framework to expedite bootstrapping new app dev:\n[GET] Request info - client request, headers and environment variables [POST] Echo message - simple echo message This entire setup is available as a template with prerequisites and deployment instructions my GitHub repo. I hope you find it as helpful as I do.\n","permalink":"https:\/\/blog.chmarny.com\/posts\/multi-region-rest-service-on-gcp\/","summary":"

I learn best by doing. And recently, most of the projects I’ve been building are either REST or gRPC-base services deployed as container images into Cloud Run on GCP. That means that I increasingly find myself recreating a lot of the same infra and app deployment flows.<\/p>","title":"Multi Region REST Service on GCP using GitHub Template"},{"content":"Why not Medium My main reason for migrating off Medium was the paywall Medium introduced while back. I actually understand why they did it. The unlimited access price: $5\/month ($50\/year) is too high, but still, I get it.\nFor me though, the objective was to allow readers to easily discover and read my posts. I don’t want my readers to experience any friction. Forcing the reader to deal with the frustrating Medium up-sell pop-ups just to read my post was just unnecessary.\nWhy Hugo on GitHub I chose Hugo because it was the simplest, most flexible, and fastest static site generator I could find. It allowed me to use the technologies I already was familiar with (e.g. Markdown). And, it can be easily deployed to any number of hosting targets (e.g. including GitHub Pages for free!).\nThe other properties of Hugo that contributed to my decision were:\nOpen source (Apache v2) with a thriving community and many free themes to choose from Written in Go with well-documented customization\/contribution process (github.com\/gohugoio\/hugo) Multilingual support (one of these days I’ll write in Polish; one of these days…) How I Migrated content The process of migrating form Medium to Hugo was a lot more complex than I expected. It was mainly related to some idiosyncrasies of how Medium exports images and formats code samples in its HTML files. Hope my learnings captured in this post are helpful.\nMedium provides a quick way to export all of your personal content as HTML files in a single .zip file. The entire process for either desktop of mobile is documented here.\nOnce I had my content downloaded locally, I had to convert the posts into Markdown (it’s how posts are defined in Hugo). I’ve tried a few tools and the medium2md CLI did the best job. After the conversion, I had to manually modify the Hugo metadata generated by my HTML to Markdown conversion tool. Here is an example of the metadata for this post:\n--- title: Medium to GitHub using Hugo - Why and How I Migrated my Blog date: '2022-01-03T13:51:00.022Z' categories: [dev] tags: [hugo, github, blog] thumbnail: "\/images\/hugo.png" draft: true --- Note: medium2md requires Node. The only other migration tool I found that wasn’t Node based was medium-to-hugo but it hasn’t been updated for over 3 years.\nOnce I had my posts in Markdown format and all the images exported, I’ve used the Hugo Quick Start docs to create my site. Most of the steps were easy to follow. I’ve spent over two hours selecting the theme for my blog, but that has to do more with the amount of free themes that are available and my inability to make a decision.\nThe one thing that made the whole customization a lot easier was the live reload feature in Hugo. When started as a server in the developer mode (hugo server -D), Hugo allowed me to edit the different configuration options as well as post content and instantaneously view them in the browser without reload or refreshing of the pages. This made the whole experience a lot easier and faster.\nOnce I was satisfied with how my blog looked, I moved on to deploying the blog to GitHub. Hugo has a doc on how to publish your blog to GitHib pages which keeps everything in a single, public, repo. I’ve opted to keep my Hugo project in a private repo called blog, and use GitHub Action to publish only the generated content to another repo (mchmarny.github.io).\nTo do this I had to edit the Hugo workflow to provide parameters for the external_repository and configuring a deploy key on the repo. The standard GITHUB_TOKEN has no permission to access to external repositories.\n- name: Deploy Blog uses: peaceiris\/actions-gh-pages@v3 if: github.ref == 'refs\/heads\/main' with: deploy_key: ${{ secrets.ACTIONS_DEPLOY_KEY }} external_repository: mchmarny\/mchmarny.github.io publish_branch: main publish_dir: .\/public commit_message: ${{ github.event.head_commit.message }} cname: blog.chmarny.com allow_empty_commit: true Notice the cname setting in the above YAML. It’s required to setup the SSL cert for the custom domain in my blog. More on setting up custom domain in GitHub pages here.\nAs a result of this configuration:\nSite generation workflow is triggered with each commit to the blog repo I can write and edit posts on my iPad just by creating a new Markdown file (*.md) New posts in draft mode are not being discoverable in public site (draft: true in hugo prevents the public content from being generated) Static content (no DB) means more secure site (i.e. how often do you hear about Wordpress vulnerabilities) I hope you found my experience of migrating from Medium to Hugo helpful.\n","permalink":"https:\/\/blog.chmarny.com\/posts\/medium-to-hugo-why-and-how\/","summary":"

Why not Medium<\/h2>\n

My main reason for migrating off Medium was the paywall<\/a> Medium introduced while back. I actually understand why they did it. The unlimited access price: $5\/month ($50\/year) is too high, but still, I get it.<\/p>\n

For me though, the objective was to allow readers to easily discover and read my posts. I don’t want my readers to experience any friction. Forcing the reader to deal with the frustrating Medium up-sell pop-ups just to read my post was just unnecessary.<\/p>","title":"Medium to GitHub using Hugo - Why and How I Migrated my Blog"},{"content":"Increasing large amount of technical news I read come from the posts shared on Hacker News or on Twitter. While both of these services have search options, neither of these seem to be advanced enough to narrow my searches at the desired level or to setup automatic delivery.\nI decided to write my own using a few of the on-line services that come with generous free-tier option:\nFirst, the runtime. I\u2019ve been impressed with the flexibility of GitHub Actions. While actions are normally written in JavaScript, with its support for pre-built Docker images, I decided to write my service in Go. To persist the state between each execution (GitHub Actions are ephemeral by nature) I\u2019ve used Firestore (part of Firebase service), which has been my goto serverless “database” for a while now due to its easy API, schema-less design, and generous free plan (1GB, 50K reads and 20K writes per day). Finally, to deliver the matching results, I\u2019ve chosen the e-mail sending service offered by SendGrid. Again, simple API and super easy configuration with up to 100 free emails per day. With the services selected, I wrote a simple application that builds the desired queries against the Hacker News and Twitter API, filters out results based on provided criteria, and captures the final state of each query for subsequent queries. The application builds into a Docker image using, you guest it, GitHub Actions. The entire source code and the automation around the project are located here.\nDelivery schedule You can setup your searches in a private repository, this keeps the topics to which you subscribe and your email address hidden. Start by creating an action with on schedule matching your desired delivery times. For example (15 min after noon, 5 and 11PM GMT):\non: schedule: - cron: "15 12,17,23 \\* \\* \\*" Next, define any number of steps in your job, each with the source type parameter and the query which you want to schedule. For example, to search for most recent not retweets in English on Kubernetes, DevOps, or CI\/CD that have links, were shared by people who themselves posted more than 200 tweets, have at least 500 followers, and have a low following (friend) to follower ratio the schedule would look something like this:\n- name: Twitter Search uses: docker:\/\/ghcr.io\/mchmarny\/gaw:v0.2.9 with: source: twitter query: kubernetes OR devops OR ci\/cd to: you@your-domain.com subject: Tweets about k8s, devops, or ci\/cd email-config: ${{ secrets.EMAIL_CONFIG }} state-config: ${{ secrets.FIRESTORE_CONFIG }} twitter-config: ${{ secrets.TWITTER_CONFIG }} lang: en # language (en == English) latest: true # most recent popular links: true # must have links followers: 500 # min number of followers retweets: false # Include retweets posts: 200 # min number of posts ratio: 0.1 # number of friends\/number of followers Notice that the API tokens and other secrets are using encrypted environment variables provided by GitHub secrets. The instructions on how to configure these are located here.\nThe full examples of schedules are available in the repo:\nHacker News search example Twitter search example Delivery dispatch In addition to scheduled queries, you can also set up on-demand searches for specific topic. In this case we combine the Twitter and Hacker News sources into a single workflow with an on action set to repo dispatch:\non: repository_dispatch: types: [search] The full example of this workflow is available here. When configured, the specific topic (e.g. golang) for which you want to search can be defined in a command line:\ncurl -H "Accept: application\/vnd.github.v3+json" \\ -H "Authorization: token ${GITHUB_TOKEN}" \\ https:\/\/api.github.com\/repos\/<username>\/<repo>\/dispatches \\ -d '{ "event_type": "search", "client_payload": { "topic":"golang" }}' To execute the dispatches using HTTP API you will need GitHub personal access token. The instructions how to create one are located here.\nEmail Notifications Whether you executed the workflow on schedule or using the HTTP dispatch, the delivered email looks something like this:\nHope you found this helpful,\n","permalink":"https:\/\/blog.chmarny.com\/posts\/custom-news-scraper-using-free-services-from-github-firebase-sengrid\/","summary":"

Increasing large amount of technical news I read come from the posts shared on Hacker News or on Twitter. While both of these services have search options<\/a>, neither of these seem to be advanced enough to narrow my searches at the desired level or to setup automatic delivery.<\/p>","title":"Custom news scraper using free services from GitHub, Firebase, and Sengrid"},{"content":"All complexity needs to be abstracted, right? This reductionist statements misses nuance around the inherent cost\/benefit tradeoffs, especially when you consider these over time.\nDon\u2019t get me wrong, there often are good reasons for additional layers to make things simpler (grow adoption, lowering toil, removing friction, etc.). Still, these layers come at the long-term cost that\u2019s often is not a part of the evaluation process.\nTake for example the complexity around Kubernetes (it seems to be an easy target nowadays).\nWhile improving the user experience in specific use-case there, you are de-facto taking on the responsibility for maintenance of that abstraction over time.\nIs that value (ease of use for your specific use-case) higher than that responsibility? If your use-case is more common, you may be thinking about open-sourcing that abstraction to offset that cost. Just ask those who created OSS projects before, that cost is usually multiple of that associated with managing your own “custom solution”.\nMore importantly, perhaps, you have to ask yourself if that compounded cost is lower than the cognitive load required to learn how to use the current implementation.\nIs that a one time cost that users, once learn, apply over time? Or is that just replacement of one thing the user has to learn with another? The Kubernetes landscape is full of carcasses of well-intended projects aiming to solve its complexity. Their relevance usually ends by either being outpaced by the community, or acquiring an even higher cognitive load. Another cost which perhaps is not always consider when hiding complexity is the potential for decreased portability. This can be due to the fact that the resulting implementation, lacking broad adoption, is simply not available in the target platform. Or, the decreased portability may also be related to the implementation “leaking” into the user space, making that application now dependent on that abstraction.\nAgain, there are many examples where abstractions of complexity delivers a lot of value. Often that value is broadly recognized, and it becomes either upstreamed to address the root complexity (e.g. kustomize in kubectl), or it becomes part fo the broader community cannon (e.g. CNCF projects) around the project.\nThe points I\u2019m trying to make here is this:\nEvaluate the broader and long-term cost\/value trade-offs of abstracting complexity Consider addressing that complexity at its root, rather than abstraction And, most importantly, don\u2019t underestimate users\u2019 ability to learn! ","permalink":"https:\/\/blog.chmarny.com\/posts\/complexity-can-be-learned-but-abstractions-come-at-long-term-cost\/","summary":"

All complexity needs to be abstracted, right? This reductionist statements misses nuance around the inherent cost\/benefit tradeoffs, especially when you consider these over time.<\/p>\n

Don\u2019t get me wrong, there often are good reasons for additional layers to make things simpler (grow adoption, lowering toil, removing friction, etc.). Still, these layers come at the long-term cost that\u2019s often is not a part of the evaluation process.<\/p>","title":"Complexity can be learned but abstractions come at a long-term cost"},{"content":"I recently joined the Office of CTO in Azure at Microsoft and wanted to ramp up on one of the open source projects the team has built there called Dapr. Dapr describes itself as:\nA portable, event-driven runtime that makes it easy for developers to build resilient, microservice stateless and stateful applications that run on the cloud and edge and embraces the diversity of languages and developer frameworks.\nI learn technology best by using it, so I decided to build a simple tweet sentiment processing pipeline and see how easy and portable it really is. The resulting project with source code, which you can run both locally and on Kubernetes, is available here.\nA few takeaways from my experience:\nDapr was easy to start with, thanks mainly to the very thorough documentation and its dead simple CLI. The local development experience in Dapr also felt natural. I was able to use my existing skills and get something going within 10\u201315 minutes. I did like that Dapr didn\u2019t have any “invasive” libraries or required SDKs. There are client libraries in most languages, but its API is well structured and has feature parity in both HTTP and gRPC. I, actually, ended up writing my own HTTP client (godapr), just to get a better “feel” for the API. The opt-in abstractions for state and pub\/sub enable easy portability. I used Redis for both during local development and then swapped for Azure Storage Table and Azure Service Bus when deploying to Kubernetes without any code changes. The growing list of bindings allow developers to quickly integrate backing services with minimal effort. Everything from generic ones like Kafka or MQTT to Cloud-provider specific ones like DynamoDB or S3 for AWS, Cloud Pub\/Sub and GCS for GCP, or EventGrid and EventHubs for Azure. Building these extensions is pretty easy too. I contributed the Twitter input binding, which went out in the recent Dapr v0.7 release. When building apps based on multiple microservices like my pipeline, Dapr provides a consistent service discovery and invocation mechanism, whether locally or on Kubernetes. Dapr seems to strike the right balance between helping developers with primitives, like persisting state or eventing though pub\/sub, without being too overbearing.\nI look forward to working on Dapr, contributing to the project, and yes, developing with on Dapr myself.\n","permalink":"https:\/\/blog.chmarny.com\/posts\/how-i-learned-dapr-building-tweet-sentiment-processing-pipeline\/","summary":"

I recently joined the Office of CTO in Azure at Microsoft and wanted to ramp up on one of the open source projects the team has built there called Dapr. Dapr describes itself<\/a> as:<\/p>\n

\n

A portable, event-driven runtime that makes it easy for developers to build resilient, microservice stateless and stateful applications that run on the cloud and edge and embraces the diversity of languages and developer frameworks.<\/p>","title":"How I learned Dapr building tweet sentiment processing pipeline"},{"content":"We are entering a period where custom, highly-optimized, vertical solutions are becoming viable option again. This is a good news for ISVs with proven domain expertise and skilled development resources.\nWhy do I think so? We now have:\nPlethora of feature-rich developer frameworks, message queues, scalable data stores, and even lower-level components in the OSS community with great documentation and a large number of use-case validation Growing number of custom solution companies (more than just ISVs) with existing deep vertical\/domain expertise who are also increasingly now investing in hiring and training strong development teams Virtually every Cloud provider offering either a raw Kubernetes service or managed container execution platform which (regardless how you feel about these technologies) creates ubiquitous surface area that can be addressed with a single solution Yes, there still are many ways in which these custom development efforts can fail. Still, as one who has started their professional career developing custom software, I\u2019m glad to see how these kinds of efforts are becoming cost effective again and increasingly represent a viable option for differentiation and real business value delivery.\nKeep an eye on vertically-focused companies like (just to name a few) NCR or Manhattan Associates in retail, Schlumberger or OSI Soft in energy, or Siemens and Rockwell Automation in manufacturing. The renaissance of custom vertical solution is coming.\n","permalink":"https:\/\/blog.chmarny.com\/posts\/renaissance-of-custom-vertical-solution\/","summary":"

We are entering a period where custom, highly-optimized, vertical solutions are becoming viable option again. This is a good news for ISVs with proven domain expertise and skilled development resources.<\/p>\n

Why do I think so? We now have:<\/p>\n

    \n
  1. Plethora of feature-rich developer frameworks, message queues, scalable data stores, and even lower-level components in the OSS community with great documentation and a large number of use-case validation<\/li>\n
  2. Growing number of custom solution companies (more than just ISVs) with existing deep vertical\/domain expertise who are also increasingly now investing in hiring and training strong development teams<\/li>\n
  3. Virtually every Cloud provider offering either a raw Kubernetes service or managed container execution platform which (regardless how you feel about these technologies) creates ubiquitous surface area that can be addressed with a single solution<\/li>\n<\/ol>\n

    Yes, there still are many ways in which these custom development efforts can fail. Still, as one who has started their professional career developing custom software, I\u2019m glad to see how these kinds of efforts are becoming cost effective again and increasingly represent a viable option for differentiation and real business value delivery.<\/p>","title":"Renaissance of custom vertical solution"},{"content":"When dealing with file permissions in a non-root image or building apps that include static content (like css or templates), I sometime get an error resulting from the final image content mismatch with my expectations.\nMost of the time the errors are pretty obvious, simple fix and rebuild will do. Sometimes though, you want to take a look into the image and understand what the actual layout looks like in there.\nI\u2019m sure there is a few valid ways of doing this, here are the simplest steps I found to be consistently reliable for me\nLet’s assume I build my app (myapp) locally using this command:\ndocker build -t myapp . To start, I first need to save the image into an archive:\ndocker save -o myapp.tar myapp Now I can list the archive layers by piping the image manifest to some pretty json tool like jQuery:\ntar xf myapp.tar -O manifest.json | jq "." To peek into a specific layer, I can output its content using one of the layer\u2019s ID. (Unless you are doing something creative, the content of your app will be the last item in “Layers”):\ntar xf myapp.tar -O 093015edb4b7f6f64a3bcb3a4646146d0476fe59b4f47924700b66b779cb7330\/layer.tar | tar tv\nDon\u2019t @ me for .DS_Store files or file permissions in the above image ;)\nHope this helps.\nPS. Massive thanks to mattomata@ for clarifying some of these things for me.\n","permalink":"https:\/\/blog.chmarny.com\/posts\/how-to-debug-container-image-content\/","summary":"

    When dealing with file permissions in a non-root image or building apps that include static content (like css or templates), I sometime get an error resulting from the final image content mismatch with my expectations.<\/p>\n

    Most of the time the errors are pretty obvious, simple fix and rebuild will do. Sometimes though, you want to take a look into the image and understand what the actual layout looks like in there.<\/p>","title":"How to debug container image content"},{"content":"While the idea of a serverless platform and long running workloads does seem somewhat “unnatural” at first, smart people are already working on that (looking at you @Knative community). In the meantime, a simple approach is sometimes all you may need.\nIn this post I will illustrating how to use Google Compute Engine (GCE) container execution option to run variable duration jobs. This approach supports custom VMs, GPU\/TPU accelerators and VPC networks\u2026 so may be handy alternative to other compute options on GCP. I\u2019ll also demo how to auto-terminate the created VM on container completion, so you won\u2019t have to pay for idle VM time.\nFinally, in this example I\u2019ll parses small gzip file from Google Cloud Storage (GCS), but since this approach is not limited by client timeouts you can use it to do pretty much anything\u2026 transformations on bigger files, lightweight ETL, or media format encoding. You can even combine it with GCP Task Queue for more complex pipelines.\nPre-requirements If you don\u2019t have one already, create new GCP project and configuring Google Cloud SDK.\nSetup To start, clone this repo, and navigate into that directory:\ngit clone [https:\/\/github.com\/mchmarny\/long-running-job.git](https:\/\/github.com\/mchmarny\/long-running-job.git) cd long-running-job Also, to streamline the subsequent commands, we are going to export a few variables:\nexport PROJECT=$(gcloud config get-value project) export APP_NAME="my-long-job" export SA_NAME="${APP_NAME}@${PROJECT}.iam.gserviceaccount.com" Service Account To execute this sample you will need a GCP service account. You can do that either in UI or using gcloud SDK.\ngcloud iam service-accounts create $APP_NAME \\ --display-name "Service Invoker Account for ${APP_NAME}" We also will have to assign this account the necessary IAM roles:\ngcloud projects add-iam-policy-binding $PROJECT \\ --member "serviceAccount:${SA_NAME}" \\ --role roles\/logging.logWriter gcloud projects add-iam-policy-binding $PROJECT \\ --member "serviceAccount:${SA_NAME}" \\ --role roles\/cloudtrace.agent gcloud projects add-iam-policy-binding $PROJECT \\ --member "serviceAccount:${SA_NAME}" \\ --role roles\/monitoring.metricWriter gcloud projects add-iam-policy-binding $PROJECT \\ --member "serviceAccount:${SA_NAME}" \\ --role roles\/storage.objectViewer gcloud projects add-iam-policy-binding $PROJECT \\ --member "serviceAccount:${SA_NAME}" \\ --role roles\/pubsub.editor gcloud projects add-iam-policy-binding $PROJECT \\ --member "serviceAccount:${SA_NAME}" \\ --role roles\/compute.instanceAdmin Finally, to enable our demo to impersonate that account, we will need to provision a key for that account which will be saved in your home directory. You should protect that key or just delete it after this demo.\ngcloud iam service-accounts keys create \\ --iam-account $SA_NAME "${HOME}\/.${APP_NAME}-sa.json" Container Image The unit of workload in this example is container image. To create an image from the source code in this demo, first vendor all the go dependencies:\ngo mod tidy go mod vendor And then submit the image build request to Cloud Build:\ngcloud builds submit --tag "gcr.io\/${PROJECT}\/${APP_NAME}:0.0.1" If successful, you should see a confirmation with the fully qualified URI to the created image.\nDeploy Container to VM Now to start a new VM and configure it to run the above built image. This sample will use gzipped CSV file (100-Sales-Records.csv.gz) located in publicly available GCS bucket (long-running-job-src-files). If you want you can replace these with your own files.\ngcloud compute instances create-with-container $APP_NAME \\ --container-image="gcr.io\/${PROJECT}\/${APP_NAME}:0.0.1" \\ --machine-type=n1-standard-1 \\ --zone=us-central1-c \\ --image-family=cos-stable \\ --image-project=cos-cloud \\ --maintenance-policy=MIGRATE \\ --scopes=cloud-platform \\ --container-privileged \\ --container-env="GOOGLE_APPLICATION_CREDENTIALS=\/tmp\/sa.pem,BUCKET=long-running-job-src-files,OBJECT=100-Sales-Records.csv.gz,TOPIC=${APP_NAME}" \\ --container-mount-host-path=mount-path=\/tmp,host-path=\/tmp,mode=rw There is no way right now to upload files in the same command, so right after, you need copy the service account key created above to the new VM. The container is set to restart on failure so it will automatically use the key when it becomes available in the mounted volume attached to our VM.\ngcloud compute scp "${HOME}\/.${APP_NAME}-sa.json" \\ "${APP_NAME}:\/tmp\/sa.pem" --zone=us-central1-c Container Logs Once the VM has started you can monitor the logs output from the container to Stackdriver. To do that we need to first capture the VM instance Id:\nexport VM_INSTANCE_ID=$(gcloud compute instances describe \\ $APP_NAME --zone=us-central1-c --format="value(id)") Once we have captured the VM instance ID we can now query the Stackdriver for logs output by the container:\ngcloud logging read "resource.type=gce_instance AND \\ logName=projects\/${PROJECT}\/logs\/cos_containers AND \\ resource.labels.instance_id=${VM_INSTANCE_ID} AND \\ jsonPayload.message:\\"\\[LRJ\\]\\"" \\ --order="asc" Note, this command will print only the logs that are output by the user code in the container (the logger inside prefixes all log entries with “[LRJ]”). You can see the complete list of log entries by removing the__jsonPayload.message:"[LRJ]"_ filter.\nAfter the container exists, the VM will automatically shutdown but the logs should still be still available in Stackdriver for forensic analyses.\nConclusion I hope you find this helpful addition to your GCP compute toolbox. You can find the source code along with scripts for easier execution at https:\/\/github.com\/mchmarny\/long-running-job\n","permalink":"https:\/\/blog.chmarny.com\/posts\/how-to-run-containerized-workloads-in-gce-vm\/","summary":"

    While the idea of a serverless platform and long running workloads does seem somewhat “unnatural” at first, smart people are already working on that (looking at you @Knative<\/a> community). In the meantime, a simple approach is sometimes all you may need.<\/p>","title":"How to run containerized workloads in GCE VM"},{"content":"A co-worker recently told me about flic.io buttons. These button caught my attention because they can include triggers for single, double, or hold click and can be easily wired up to all kinds of actions.\nI instantly thought of of a few really interesting applications. To start with though, I wanted to create a simple service that would allow me to push the custom data defined on each button over HTTP to Cloud PubSub. That in turn would then connect me to the many more actuation options through GCP APIs and services that connect to Cloud PugSub.\nI went ahead and ordered 4-pack of Flic buttons and chose Cloud Run to implement my buttons service. Here is a short demo on to:\nDeploy Cloud Run service that will persist sent data to Cloud PubSub and, how to configure Flic buttons to send data to Cloud Run service https:\/\/github.com\/mchmarny\/buttons\n","permalink":"https:\/\/blog.chmarny.com\/posts\/using-flick-buttons-with-cloud-run-on-gcp\/","summary":"

    A co-worker recently told me about flic.io<\/a> buttons. These button caught my attention because they can include triggers for single, double, or hold click and can be easily wired up to all kinds of actions.<\/p>\n

    I instantly thought of of a few really interesting applications. To start with though, I wanted to create a simple service that would allow me to push the custom data defined on each button over HTTP to Cloud PubSub. That in turn would then connect me to the many more actuation options through GCP APIs and services that connect to Cloud PugSub.<\/p>","title":"Using Flick buttons with Cloud Run on GCP"},{"content":"Next week, April 9\u201311, Google will be hosting this year\u2019s Cloud Next Conference in San Francisco. The conference is already sold out, but there will be a livestream from keynotes and video available shortly after the sessions.\nThis year, we have a lot of content to share, and I have the privilege of presenting in four sessions\u200a\u2014\u200aand hope to do at least six live demos.\nOn Tuesday, April 10 at 12:35 PM, Donna Malayeri and I will be overviewing the integration options developers have on GCP when building serverless solutions on Kubernetes:\nKubernetes provides a great platform for operators, but what about your developers? Dive into the details about a new GCP product powered by Knative and Kubernetes Engine that enables enterprise developers to build and deploy stateless containerized apps into a cluster without writing any YAML.\nThen on Thursday, April 11 at 9 AM, Tomas Isdal and I will give an update on Knative and the fascinating momentum this project has had since we announced it at this same event the previous year:\nKnative, which provides the technology for a serverless developer experience on Kubernetes, has seen remarkable growth and uptake since launching in 2018. Join Google to hear how we are delivering Knative-based products to customers in 2019.\nAt 11:40 AM, Ryan Gregg and I will host a session overving the consistent developer experience enabled by Google on Kubernetes:\nCome learn about the latest Google Cloud offering that brings the best of serverless to your GKE cluster and anywhere you can run Kubernetes. Learn how Knative and GCP provide a consistent developer experience for new apps across cloud, on-prem and hybrid. Where you choose to run your services should be your choice!\nAnd last, but not least, at 1:15 PM, Ville Aikas and I will overview the new Knative Eventing features and demonstrate how to leverage them in your solutions.\nTechnical deep dive into how to write an event source to connect your legacy systems with modern compute primitives in Knative.\nI\u2019m super excited for Google Cloud Next, and am looking forward to these sessions. If you are in San Francisco next week, I hope to see you there.\n","permalink":"https:\/\/blog.chmarny.com\/posts\/my-sessions-at-google-next-2019\/","summary":"

    Next week, April 9\u201311, Google will be hosting this year\u2019s Cloud Next Conference<\/strong><\/a> in San Francisco. The conference is already sold out, but there will be a livestream<\/a> from keynotes and video available shortly after the sessions.<\/p>\n

    This year, we have a lot of content to share, and I have the privilege of presenting in four sessions<\/a>\u200a\u2014\u200aand hope to do at least six live demos.<\/p>","title":"My Sessions at Google Next 2019"},{"content":"I had a chance to speak at the Cloud Conf 2019 in Turin, Italy. The conference has double its audience from last year, had a spectacular venue, and large selection of topics. I spoke in the #serverless track on using Knative as a means to serverless where you want it and on your own terms. Stressing the importance of portability and demonstrating the key features that the Kantive community has already delivered.\nServing Knative Serving provides higher level abstraction for Kubernetes and, by extension, the underlying infrastructure, so that the developers can focus on writing code.\nIt exposes easy to reason about object model and enables seamless autoscaling based on requests. It also integrates networking and service mesh automatically so devs can gradually rollout new revisions or label some services as internal only so that they can\u2019t be access directly in a microservice based solution.\nBuild Knative Build provides configurable and flexible approach to building source code into containers.\nIt can leverage Dockerfiles or built templates in case of building directly from sources without the developers needing to worry about cross-compiling or even installing the build tools locally. And because the build happens on cluster, it support artifact caching for faster builds\nEventing Knative Universal provides a subscription, delivery, and management framework for events both on and off the cluster.\nIt allows developer to create a declarative binding between event producers and event consuming services. And, because it integrates with serving, it\u2019s scalable from just a few events to live event streams. And it\u2019s flexible, so you can build simple trigger to one service binding or custom event pipelines to connect with your existing systems.\nServerless on your own terms So, yes, there is Kubernetes, yes there are containers, and yes, there are even servers underneath it all\u2026 but you do you, developer. You write code and the Knative operators do their job well, you don\u2019t have care about none of it\u2026 until you want to.\n","permalink":"https:\/\/blog.chmarny.com\/posts\/knative-serverless-on-your-own-terms\/","summary":"

    I had a chance to speak at the Cloud Conf 2019 in Turin, Italy. The conference has double its audience from last year, had a spectacular venue, and large selection of topics. I spoke in the #serverless track on using Knative as a means to serverless where you want it and on your own terms. Stressing the importance of portability and demonstrating the key features that the Kantive community has already delivered.<\/p>","title":"Knative - Serverless on Your Own Terms"},{"content":" I wrote a new post on Google blog on the momentum behind the Knative project. How it the community reached another adoption milestone, doubling the number of its contributors. Also, another data point underscoring the Knative momentum is the month-over-month contributions which have increased over 45% since the 0.1 release, now representing more than a dozen of different companies.\nThis post was released to showcase Google’s OSS contributions at KubeCon North America. Full post at https:\/\/opensource.googleblog.com\/2018\/12\/knative-momentum-continues.html\n","permalink":"https:\/\/blog.chmarny.com\/posts\/knative-momentum-continues\/","summary":"\n

    I wrote a new post<\/a> on Google blog on the momentum behind the Knative<\/a> project. How it the community reached another adoption milestone, doubling the number of its contributors. Also, another data point underscoring the Knative momentum is the month-over-month contributions which have increased over 45% since the 0.1 release, now representing more than a dozen of different companies.<\/p>","title":"Knative momentum continues\u2026"},{"content":"I had an opportunity to keynote at this year\u2019s SpringOne conference in DC on Serverless, Kubernetes, and more specifically Knative.\nI also covered the great work our open source team at Google been doing, making Spring 1st class citizen on Google Cloud Platform. On a personal level, it was good to reconnect with some old friends from Pivotal, and, community at large.\n","permalink":"https:\/\/blog.chmarny.com\/posts\/my-keynote-at-springone\/","summary":"

    I had an opportunity to keynote at this year\u2019s SpringOne<\/a> conference in DC on Serverless, Kubernetes<\/a>, and more specifically Knative<\/a>.<\/p>\n